2024-04-05T21:03:07.920 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-05T21:03:07.920 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --owner scheduled_yuriw@teuthology --name yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641605 --verbose -- /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641605/orig.config.yaml 2024-04-05T21:03:07.977 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-05T21:03:08.076 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641605 branch: wip-yuri4-testing-2024-04-04-1128-reef description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7641605' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/v18.2.1 ' name: yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-04-04-1128-reef ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2024-04-04-1128-reef sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 5779 sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri4-testing-2024-04-04-1128-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e targets: smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO58RlMZFByVT5ZRBanllmMbC9CSawxqmhUpRvHjVWyUXy5xZEoiBB5gELQFV7WNcoMcdiVG87UfgWfN2t6CfHk= smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEIbE86l8eTwgRWHd5p9CW3jojPfoughoMo3/8pqxotWgoG4si8oBR41hhjMMH+sWlEujqcu+S9EdAxqzfUZ0pU= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: exclude_packages: - ceph-volume tag: v18.2.1 - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.io/ceph/ceph:v18.2.1 roleless: true - print: '**** done end installing v18.2.1 cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-05_18:13:21 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-04-05T21:03:08.076 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa; will attempt to use it 2024-04-05T21:03:08.077 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks 2024-04-05T21:03:08.077 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-05T21:03:08.079 INFO:teuthology.task.internal:Checking packages... 2024-04-05T21:03:08.103 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8919605b0a5dc1f1650c8aec3051c77c0e3e240e' 2024-04-05T21:03:08.103 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-05T21:03:08.104 INFO:teuthology.packaging:ref: None 2024-04-05T21:03:08.104 INFO:teuthology.packaging:tag: None 2024-04-05T21:03:08.104 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-04-04-1128-reef 2024-04-05T21:03:08.104 INFO:teuthology.packaging:sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:03:08.104 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2024-04-04-1128-reef 2024-04-05T21:03:08.329 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1173.g8919605b 2024-04-05T21:03:08.331 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-05T21:03:08.337 INFO:teuthology.task.internal:no buildpackages task found 2024-04-05T21:03:08.337 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-05T21:03:08.355 INFO:teuthology.task.internal:Saving configuration 2024-04-05T21:03:08.376 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-05T21:03:08.382 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-05T21:03:08.407 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641605', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 20:58:00.114737', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO58RlMZFByVT5ZRBanllmMbC9CSawxqmhUpRvHjVWyUXy5xZEoiBB5gELQFV7WNcoMcdiVG87UfgWfN2t6CfHk='} 2024-04-05T21:03:08.430 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641605', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 20:58:00.112903', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEIbE86l8eTwgRWHd5p9CW3jojPfoughoMo3/8pqxotWgoG4si8oBR41hhjMMH+sWlEujqcu+S9EdAxqzfUZ0pU='} 2024-04-05T21:03:08.430 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-05T21:03:08.438 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-05T21:03:08.439 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-05T21:03:08.439 INFO:teuthology.run_tasks:Running task console_log... 2024-04-05T21:03:08.519 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9b75cd9160>, signals=[15]) 2024-04-05T21:03:08.520 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-05T21:03:08.526 INFO:teuthology.task.internal:Opening connections... 2024-04-05T21:03:08.526 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T21:03:08.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T21:03:08.603 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2024-04-05T21:03:08.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T21:03:08.679 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-05T21:03:08.685 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-04-05T21:03:08.707 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-04-05T21:03:08.708 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-04-05T21:03:08.768 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-04-05T21:03:08.769 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8" 2024-04-05T21:03:08.769 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-04-05T21:03:08.769 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-04-05T21:03:08.769 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8" 2024-04-05T21:03:08.769 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8" 2024-04-05T21:03:08.769 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-05T21:03:08.769 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-04-05T21:03:08.769 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-05T21:03:08.769 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-04-05T21:03:08.769 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-05T21:03:08.769 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-05T21:03:08.769 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-05T21:03:08.770 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-04-05T21:03:08.793 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-04-05T21:03:08.815 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-04-05T21:03:08.815 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-04-05T21:03:08.877 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2024-04-05T21:03:08.877 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8" 2024-04-05T21:03:08.877 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2024-04-05T21:03:08.877 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2024-04-05T21:03:08.877 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8" 2024-04-05T21:03:08.877 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2024-04-05T21:03:08.877 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-05T21:03:08.877 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-04-05T21:03:08.877 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-05T21:03:08.878 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2024-04-05T21:03:08.878 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-05T21:03:08.878 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-05T21:03:08.878 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-05T21:03:08.878 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-04-05T21:03:08.902 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-05T21:03:08.911 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-05T21:03:08.916 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-05T21:03:08.917 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-04-05T21:03:08.919 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2024-04-05T21:03:08.940 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-05T21:03:08.946 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-05T21:03:08.946 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-04-05T21:03:08.980 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2024-04-05T21:03:09.009 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-05T21:03:09.063 INFO:teuthology.run_tasks:Running task kernel... 2024-04-05T21:03:09.079 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-05T21:03:09.079 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-05T21:03:09.080 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-05T21:03:09.080 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-05T21:03:09.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T21:03:09.101 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-04-05T21:03:09.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T21:03:09.102 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-04-05T21:03:09.160 INFO:teuthology.orchestra.run.smithi121.stdout:4.18.0-489.el8.x86_64 2024-04-05T21:03:09.161 INFO:teuthology.task.kernel:Running kernel on smithi121: 4.18.0-489.el8.x86_64 2024-04-05T21:03:09.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-04-05T21:03:09.163 INFO:teuthology.orchestra.run.smithi188.stdout:4.18.0-489.el8.x86_64 2024-04-05T21:03:09.163 INFO:teuthology.task.kernel:Running kernel on smithi188: 4.18.0-489.el8.x86_64 2024-04-05T21:03:09.163 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum install -y kernel 2024-04-05T21:03:14.144 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - AppStream 11 MB/s | 28 MB 00:02 2024-04-05T21:03:14.169 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - AppStream 11 MB/s | 28 MB 00:02 2024-04-05T21:03:19.911 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-04-05T21:03:19.911 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-04-05T21:03:21.809 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras 58 kB/s | 18 kB 00:00 2024-04-05T21:03:21.836 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras 59 kB/s | 18 kB 00:00 2024-04-05T21:03:22.312 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 7.7 kB 00:00 2024-04-05T21:03:22.320 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 7.7 kB 00:00 2024-04-05T21:03:22.514 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 31 kB/s | 3.5 kB 00:00 2024-04-05T21:03:22.546 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-04-05T21:03:23.180 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 30 MB/s | 16 MB 00:00 2024-04-05T21:03:26.914 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 413 kB/s | 24 kB 00:00 2024-04-05T21:03:29.927 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-05T21:03:29.927 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-05T21:03:30.056 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:03:30.057 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:03:30.057 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-05T21:03:30.057 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:03:30.057 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:03:30.057 INFO:teuthology.orchestra.run.smithi188.stdout: kernel x86_64 4.18.0-548.el8 baseos 10 M 2024-04-05T21:03:30.057 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-05T21:03:30.058 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-core x86_64 4.18.0-548.el8 baseos 44 M 2024-04-05T21:03:30.058 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules x86_64 4.18.0-548.el8 baseos 36 M 2024-04-05T21:03:30.058 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:03:30.058 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:03:30.058 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:03:30.058 INFO:teuthology.orchestra.run.smithi188.stdout:Install 3 Packages 2024-04-05T21:03:30.058 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:03:30.059 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 90 M 2024-04-05T21:03:30.059 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 96 M 2024-04-05T21:03:30.059 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:03:32.306 INFO:teuthology.orchestra.run.smithi188.stdout:(1/3): kernel-4.18.0-548.el8.x86_64.rpm 4.7 MB/s | 10 MB 00:02 2024-04-05T21:03:34.314 INFO:teuthology.orchestra.run.smithi188.stdout:(2/3): kernel-modules-4.18.0-548.el8.x86_64.rpm 8.7 MB/s | 36 MB 00:04 2024-04-05T21:03:35.182 INFO:teuthology.orchestra.run.smithi188.stdout:(3/3): kernel-core-4.18.0-548.el8.x86_64.rpm 8.6 MB/s | 44 MB 00:05 2024-04-05T21:03:35.182 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:03:35.182 INFO:teuthology.orchestra.run.smithi188.stdout:Total 18 MB/s | 90 MB 00:05 2024-04-05T21:03:35.862 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:03:36.272 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:03:36.272 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:03:37.512 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:03:37.538 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:03:39.798 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 969 kB/s | 16 MB 00:17 2024-04-05T21:03:40.339 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:03:40.846 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-core-4.18.0-548.el8.x86_64 1/3 2024-04-05T21:03:43.544 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 466 kB/s | 24 kB 00:00 2024-04-05T21:03:44.449 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-4.18.0-548.el8.x86_64 1/3 2024-04-05T21:03:44.854 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-modules-4.18.0-548.el8.x86_64 2/3 2024-04-05T21:03:46.565 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-05T21:03:46.566 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-05T21:03:46.689 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:03:46.690 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:03:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-05T21:03:46.690 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:03:46.690 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:03:46.690 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 4.18.0-548.el8 baseos 10 M 2024-04-05T21:03:46.691 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-05T21:03:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 4.18.0-548.el8 baseos 44 M 2024-04-05T21:03:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 4.18.0-548.el8 baseos 36 M 2024-04-05T21:03:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:03:46.691 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:03:46.691 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:03:46.691 INFO:teuthology.orchestra.run.smithi121.stdout:Install 3 Packages 2024-04-05T21:03:46.691 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:03:46.692 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 90 M 2024-04-05T21:03:46.692 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 96 M 2024-04-05T21:03:46.692 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:03:48.070 INFO:teuthology.orchestra.run.smithi121.stdout:(1/3): kernel-4.18.0-548.el8.x86_64.rpm 7.8 MB/s | 10 MB 00:01 2024-04-05T21:03:48.285 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-4.18.0-548.el8.x86_64 2/3 2024-04-05T21:03:48.577 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-4.18.0-548.el8.x86_64 3/3 2024-04-05T21:03:49.287 INFO:teuthology.orchestra.run.smithi121.stdout:(2/3): kernel-modules-4.18.0-548.el8.x86_64.rpm 14 MB/s | 36 MB 00:02 2024-04-05T21:03:49.795 INFO:teuthology.orchestra.run.smithi121.stdout:(3/3): kernel-core-4.18.0-548.el8.x86_64.rpm 14 MB/s | 44 MB 00:03 2024-04-05T21:03:49.796 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:03:49.796 INFO:teuthology.orchestra.run.smithi121.stdout:Total 29 MB/s | 90 MB 00:03 2024-04-05T21:03:50.483 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:03:50.926 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:03:50.926 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:03:52.190 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:03:52.353 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:03:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:03:55.706 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-4.18.0-548.el8.x86_64 1/3 2024-04-05T21:03:59.246 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-4.18.0-548.el8.x86_64 1/3 2024-04-05T21:03:59.655 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-4.18.0-548.el8.x86_64 2/3 2024-04-05T21:04:03.135 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-4.18.0-548.el8.x86_64 2/3 2024-04-05T21:04:03.452 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-4.18.0-548.el8.x86_64 3/3 2024-04-05T21:04:10.382 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-4.18.0-548.el8.x86_64 3/3 2024-04-05T21:04:10.405 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-4.18.0-548.el8.x86_64 3/3 2024-04-05T21:04:10.749 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-4.18.0-548.el8.x86_64 3/3 2024-04-05T21:04:10.749 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-4.18.0-548.el8.x86_64 1/3 2024-04-05T21:04:10.749 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-core-4.18.0-548.el8.x86_64 2/3 2024-04-05T21:04:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-modules-4.18.0-548.el8.x86_64 3/3 2024-04-05T21:04:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:04:11.215 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:04:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-4.18.0-548.el8.x86_64 kernel-core-4.18.0-548.el8.x86_64 2024-04-05T21:04:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules-4.18.0-548.el8.x86_64 2024-04-05T21:04:11.215 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:04:11.215 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:04:11.369 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2024-04-05T21:04:11.586 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-4.18.0-548.el8.x86_64 2024-04-05T21:04:11.586 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-548.el8.x86_64 2024-04-05T21:04:11.587 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-548.el8.x86_64 2024-04-05T21:04:11.587 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-05T21:04:11.587 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-05T21:04:11.587 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-05T21:04:11.587 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi188.front.sepia.ceph.com, path=None, version=distro) 2024-04-05T21:04:11.587 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2024-04-05T21:04:11.797 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-4.18.0-548.el8.x86_64 2024-04-05T21:04:11.797 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-548.el8.x86_64 2024-04-05T21:04:11.798 DEBUG:teuthology.orchestra.run.smithi188:> sudo rpm -qi grub2-tools 2024-04-05T21:04:11.831 INFO:teuthology.orchestra.run.smithi188.stdout:Name : grub2-tools 2024-04-05T21:04:11.831 INFO:teuthology.orchestra.run.smithi188.stdout:Epoch : 1 2024-04-05T21:04:11.832 INFO:teuthology.orchestra.run.smithi188.stdout:Version : 2.02 2024-04-05T21:04:11.832 INFO:teuthology.orchestra.run.smithi188.stdout:Release : 129.el8 2024-04-05T21:04:11.832 INFO:teuthology.orchestra.run.smithi188.stdout:Architecture: x86_64 2024-04-05T21:04:11.832 INFO:teuthology.orchestra.run.smithi188.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-05T21:04:11.832 INFO:teuthology.orchestra.run.smithi188.stdout:Group : System Environment/Base 2024-04-05T21:04:11.832 INFO:teuthology.orchestra.run.smithi188.stdout:Size : 9265586 2024-04-05T21:04:11.833 INFO:teuthology.orchestra.run.smithi188.stdout:License : GPLv3+ 2024-04-05T21:04:11.833 INFO:teuthology.orchestra.run.smithi188.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-05T21:04:11.833 INFO:teuthology.orchestra.run.smithi188.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-05T21:04:11.833 INFO:teuthology.orchestra.run.smithi188.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-05T21:04:11.833 INFO:teuthology.orchestra.run.smithi188.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-05T21:04:11.833 INFO:teuthology.orchestra.run.smithi188.stdout:Relocations : (not relocatable) 2024-04-05T21:04:11.834 INFO:teuthology.orchestra.run.smithi188.stdout:Packager : CentOS BuildSystem 2024-04-05T21:04:11.834 INFO:teuthology.orchestra.run.smithi188.stdout:Vendor : CentOS 2024-04-05T21:04:11.834 INFO:teuthology.orchestra.run.smithi188.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-05T21:04:11.834 INFO:teuthology.orchestra.run.smithi188.stdout:Summary : Support tools for GRUB. 2024-04-05T21:04:11.834 INFO:teuthology.orchestra.run.smithi188.stdout:Description : 2024-04-05T21:04:11.834 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:04:11.834 INFO:teuthology.orchestra.run.smithi188.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-05T21:04:11.835 INFO:teuthology.orchestra.run.smithi188.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-05T21:04:11.835 INFO:teuthology.orchestra.run.smithi188.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-05T21:04:11.835 INFO:teuthology.orchestra.run.smithi188.stdout:hardware devices. 2024-04-05T21:04:11.835 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:04:11.835 INFO:teuthology.orchestra.run.smithi188.stdout:This subpackage provides tools for support of all platforms. 2024-04-05T21:04:11.837 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-05T21:04:11.837 INFO:teuthology.task.kernel:Updating grub on smithi188 to boot 4.18.0-548.el8.x86_64 2024-04-05T21:04:11.838 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-05T21:04:13.471 INFO:teuthology.orchestra.run.smithi188.stderr:Generating grub configuration file ... 2024-04-05T21:04:15.532 INFO:teuthology.orchestra.run.smithi188.stderr:done 2024-04-05T21:04:15.534 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-04-05T21:04:15.563 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.skoUWQJ1Xg 2024-04-05T21:04:15.563 DEBUG:teuthology.orchestra.run.smithi188:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.skoUWQJ1Xg 2024-04-05T21:04:15.637 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.skoUWQJ1Xg 2024-04-05T21:04:15.780 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.skoUWQJ1Xg is 6KB 2024-04-05T21:04:15.831 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.skoUWQJ1Xg 2024-04-05T21:04:15.848 DEBUG:teuthology.orchestra.run.smithi188:> sudo /bin/ls /boot/loader/entries || true 2024-04-05T21:04:15.913 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-05T21:04:15.913 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-05T21:04:15.913 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-05T21:04:15.913 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-548.el8.x86_64.conf 2024-04-05T21:04:15.914 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-548.el8.x86_64 2024-04-05T21:04:16.084 DEBUG:teuthology.orchestra.run.smithi188:> sudo shutdown -r now 2024-04-05T21:04:25.521 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-4.18.0-548.el8.x86_64 3/3 2024-04-05T21:04:25.543 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-4.18.0-548.el8.x86_64 3/3 2024-04-05T21:04:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-4.18.0-548.el8.x86_64 3/3 2024-04-05T21:04:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-4.18.0-548.el8.x86_64 1/3 2024-04-05T21:04:25.881 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-4.18.0-548.el8.x86_64 2/3 2024-04-05T21:04:26.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-4.18.0-548.el8.x86_64 3/3 2024-04-05T21:04:26.368 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:04:26.368 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:04:26.368 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-4.18.0-548.el8.x86_64 kernel-core-4.18.0-548.el8.x86_64 2024-04-05T21:04:26.368 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-4.18.0-548.el8.x86_64 2024-04-05T21:04:26.368 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:04:26.368 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:04:26.540 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-04-05T21:04:26.758 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-4.18.0-548.el8.x86_64 2024-04-05T21:04:26.758 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-548.el8.x86_64 2024-04-05T21:04:26.759 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-548.el8.x86_64 2024-04-05T21:04:26.759 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-05T21:04:26.759 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-05T21:04:26.759 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-05T21:04:26.759 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-04-05T21:04:26.759 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-04-05T21:04:26.968 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-4.18.0-548.el8.x86_64 2024-04-05T21:04:26.969 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-548.el8.x86_64 2024-04-05T21:04:26.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-04-05T21:04:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-04-05T21:04:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-04-05T21:04:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.02 2024-04-05T21:04:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 129.el8 2024-04-05T21:04:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Group : System Environment/Base 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 9265586 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Relocations : (not relocatable) 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : CentOS BuildSystem 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-04-05T21:04:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-04-05T21:04:27.004 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:04:27.004 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-05T21:04:27.004 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-05T21:04:27.004 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-05T21:04:27.004 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-04-05T21:04:27.004 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:04:27.004 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-04-05T21:04:27.005 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-05T21:04:27.005 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 4.18.0-548.el8.x86_64 2024-04-05T21:04:27.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-05T21:04:28.810 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-04-05T21:04:31.156 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-04-05T21:04:31.157 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-04-05T21:04:31.229 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.0mUovTqjjU 2024-04-05T21:04:31.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0mUovTqjjU 2024-04-05T21:04:31.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.0mUovTqjjU 2024-04-05T21:04:31.463 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.0mUovTqjjU is 6KB 2024-04-05T21:04:31.511 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.0mUovTqjjU 2024-04-05T21:04:31.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-04-05T21:04:31.606 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-05T21:04:31.607 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-05T21:04:31.607 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-05T21:04:31.607 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-548.el8.x86_64.conf 2024-04-05T21:04:31.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-548.el8.x86_64 2024-04-05T21:04:31.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-04-05T21:04:46.098 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-05T21:04:46.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-05T21:04:46.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T21:05:01.786 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-05T21:05:01.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-05T21:05:01.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T21:05:46.147 DEBUG:teuthology.orchestra.remote:timed out 2024-04-05T21:05:55.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-05T21:05:55.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T21:05:55.613 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-04-05T21:05:55.935 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-05T21:05:55.935 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-548.el8.x86_64"... 2024-04-05T21:05:55.935 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-04-05T21:05:55.953 INFO:teuthology.orchestra.run.smithi188.stdout:4.18.0-548.el8.x86_64 2024-04-05T21:05:55.953 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-548.el8.x86_64 vs 4.18.0-548.el8.x86_64 2024-04-05T21:05:55.953 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-05T21:05:55.953 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-05T21:05:56.953 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-05T21:05:56.954 DEBUG:teuthology.orchestra.run.smithi188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-05T21:05:57.022 INFO:teuthology.orchestra.run.smithi188.stdout:ttyS1 2024-04-05T21:05:57.037 DEBUG:teuthology.parallel:result is None 2024-04-05T21:06:01.789 DEBUG:teuthology.orchestra.remote:timed out 2024-04-05T21:06:10.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-05T21:06:10.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T21:06:11.432 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-04-05T21:06:11.799 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-05T21:06:11.799 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-548.el8.x86_64"... 2024-04-05T21:06:11.800 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-04-05T21:06:11.816 INFO:teuthology.orchestra.run.smithi121.stdout:4.18.0-548.el8.x86_64 2024-04-05T21:06:11.816 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-548.el8.x86_64 vs 4.18.0-548.el8.x86_64 2024-04-05T21:06:11.816 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-05T21:06:11.817 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-05T21:06:12.817 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-05T21:06:12.817 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-05T21:06:12.898 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-04-05T21:06:12.913 DEBUG:teuthology.parallel:result is None 2024-04-05T21:06:12.913 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-05T21:06:12.922 INFO:teuthology.task.internal:Creating test directory... 2024-04-05T21:06:12.922 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-05T21:06:12.924 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-05T21:06:12.949 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-05T21:06:12.958 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-05T21:06:12.968 INFO:teuthology.task.internal:Creating archive directory... 2024-04-05T21:06:12.968 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-05T21:06:12.982 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-05T21:06:13.023 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-05T21:06:13.031 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-05T21:06:13.031 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-05T21:06:13.042 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-05T21:06:13.084 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T21:06:13.095 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T21:06:13.120 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T21:06:13.130 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-05T21:06:13.132 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-05T21:06:13.139 INFO:teuthology.task.internal:Configuring sudo... 2024-04-05T21:06:13.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-05T21:06:13.142 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-05T21:06:13.171 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-05T21:06:13.203 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-05T21:06:13.203 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-05T21:06:13.210 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-05T21:06:13.230 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T21:06:13.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T21:06:13.366 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T21:06:13.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T21:06:13.492 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:06:13.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-05T21:06:13.559 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T21:06:13.608 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-05T21:06:13.659 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T21:06:13.747 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-05T21:06:13.789 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-05T21:06:13.789 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-05T21:06:13.858 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-04-05T21:06:13.860 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2024-04-05T21:06:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T21:06:13.986 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T21:06:14.394 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-05T21:06:14.402 INFO:teuthology.task.internal:Starting timer... 2024-04-05T21:06:14.402 INFO:teuthology.run_tasks:Running task pcp... 2024-04-05T21:06:14.422 INFO:teuthology.run_tasks:Running task selinux... 2024-04-05T21:06:14.431 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-05T21:06:14.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-04-05T21:06:14.505 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-04-05T21:06:14.507 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2024-04-05T21:06:14.594 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2024-04-05T21:06:14.596 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-05T21:06:14.596 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-04-05T21:06:14.622 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-04-05T21:06:14.622 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2024-04-05T21:06:14.649 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2024-04-05T21:06:14.650 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi121.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2024-04-05T21:06:14.650 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-05T21:06:14.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T21:06:14.689 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-05T21:06:14.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T21:06:14.718 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-05T21:06:14.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-04-05T21:06:14.756 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2024-04-05T21:06:14.785 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-05T21:06:14.795 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-05T21:06:14.813 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-05T21:06:14.814 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi121.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-05T21:12:06.127 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2024-04-05T21:12:06.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-05T21:12:06.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T21:12:06.207 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-04-05T21:12:06.277 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-05T21:12:06.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-05T21:12:06.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-05T21:12:06.353 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-04-05T21:12:06.421 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-05T21:12:06.421 INFO:teuthology.run_tasks:Running task clock... 2024-04-05T21:12:06.432 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-05T21:12:06.432 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-05T21:12:06.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T21:12:06.435 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-05T21:12:06.435 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T21:12:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-05T21:12:06.478 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-05T21:12:06.494 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-05T21:12:06.508 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-05T21:12:06.543 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-04-05T21:12:06.554 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-04-05T21:12:06.567 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2024-04-05T21:12:06.567 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-05T21:12:06.578 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2024-04-05T21:12:06.580 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-05T21:12:06.592 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-05T21:12:06.605 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-05T21:12:06.649 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2024-04-05T21:12:06.653 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T21:12:06.653 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-04-05T21:12:06.653 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T21:12:06.653 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T21:12:06.653 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T21:12:06.653 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T21:12:06.678 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2024-04-05T21:12:06.682 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T21:12:06.682 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-05T21:12:06.682 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T21:12:06.682 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T21:12:06.683 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T21:12:06.683 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T21:12:06.684 INFO:teuthology.run_tasks:Running task pexec... 2024-04-05T21:12:06.693 INFO:teuthology.task.pexec:Executing custom commands... 2024-04-05T21:12:06.693 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T21:12:06.693 DEBUG:teuthology.orchestra.run.smithi121:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-05T21:12:06.694 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi188.front.sepia.ceph.com 2024-04-05T21:12:06.694 DEBUG:teuthology.orchestra.run.smithi188:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-04-05T21:12:07.449 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:24 ago on Fri 05 Apr 2024 09:10:43 PM UTC. 2024-04-05T21:12:07.482 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:23 ago on Fri 05 Apr 2024 09:10:44 PM UTC. 2024-04-05T21:12:09.582 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:12:09.583 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:12:09.583 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-05T21:12:09.583 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:12:09.583 INFO:teuthology.orchestra.run.smithi121.stdout:Resetting modules: 2024-04-05T21:12:09.583 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools 2024-04-05T21:12:09.583 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:12:09.583 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:12:09.583 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:12:09.583 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:12:09.629 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:12:09.630 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:12:09.630 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-05T21:12:09.630 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:12:09.630 INFO:teuthology.orchestra.run.smithi188.stdout:Resetting modules: 2024-04-05T21:12:09.631 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools 2024-04-05T21:12:09.631 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:12:09.631 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:12:09.631 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:12:09.631 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:12:09.867 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:12:09.874 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:12:10.684 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:27 ago on Fri 05 Apr 2024 09:10:43 PM UTC. 2024-04-05T21:12:10.691 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:26 ago on Fri 05 Apr 2024 09:10:44 PM UTC. 2024-04-05T21:12:12.817 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:12:12.832 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:12:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:12:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-05T21:12:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:12:12.854 INFO:teuthology.orchestra.run.smithi121.stdout:Installing group/module packages: 2024-04-05T21:12:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-05T21:12:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-05T21:12:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-05T21:12:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-05T21:12:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-05T21:12:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-05T21:12:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-05T21:12:12.854 INFO:teuthology.orchestra.run.smithi121.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout:Installing module profiles: 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools/common 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling module streams: 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools rhel8 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:12:12.855 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:12:12.856 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:12:12.856 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2024-04-05T21:12:12.856 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:12:12.856 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 27 M 2024-04-05T21:12:12.856 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 91 M 2024-04-05T21:12:12.856 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:12:12.869 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:12:12.869 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-05T21:12:12.869 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:12:12.869 INFO:teuthology.orchestra.run.smithi188.stdout:Installing group/module packages: 2024-04-05T21:12:12.869 INFO:teuthology.orchestra.run.smithi188.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout:Installing weak dependencies: 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-04-05T21:12:12.870 INFO:teuthology.orchestra.run.smithi188.stdout:Installing module profiles: 2024-04-05T21:12:12.871 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools/common 2024-04-05T21:12:12.871 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:12:12.871 INFO:teuthology.orchestra.run.smithi188.stdout:Enabling module streams: 2024-04-05T21:12:12.871 INFO:teuthology.orchestra.run.smithi188.stdout: container-tools rhel8 2024-04-05T21:12:12.871 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:12:12.871 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:12:12.871 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:12:12.871 INFO:teuthology.orchestra.run.smithi188.stdout:Install 13 Packages 2024-04-05T21:12:12.871 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:12:12.872 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 27 M 2024-04-05T21:12:12.872 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 91 M 2024-04-05T21:12:12.872 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:12:13.302 INFO:teuthology.orchestra.run.smithi188.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 1.8 MB/s | 689 kB 00:00 2024-04-05T21:12:13.354 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 2.1 MB/s | 689 kB 00:00 2024-04-05T21:12:13.360 INFO:teuthology.orchestra.run.smithi188.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 650 kB/s | 38 kB 00:00 2024-04-05T21:12:13.427 INFO:teuthology.orchestra.run.smithi188.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.0 MB/s | 1.0 MB 00:00 2024-04-05T21:12:13.445 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 413 kB/s | 38 kB 00:00 2024-04-05T21:12:13.486 INFO:teuthology.orchestra.run.smithi188.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.0 MB/s | 258 kB 00:00 2024-04-05T21:12:13.538 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.7 MB/s | 258 kB 00:00 2024-04-05T21:12:13.578 INFO:teuthology.orchestra.run.smithi188.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.9 MB/s | 180 kB 00:00 2024-04-05T21:12:13.636 INFO:teuthology.orchestra.run.smithi188.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 424 kB/s | 25 kB 00:00 2024-04-05T21:12:13.680 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.6 MB/s | 1.0 MB 00:00 2024-04-05T21:12:13.703 INFO:teuthology.orchestra.run.smithi188.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2024-04-05T21:12:13.788 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.6 MB/s | 180 kB 00:00 2024-04-05T21:12:13.847 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 425 kB/s | 25 kB 00:00 2024-04-05T21:12:13.905 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2024-04-05T21:12:14.103 INFO:teuthology.orchestra.run.smithi188.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 6.1 MB/s | 4.1 MB 00:00 2024-04-05T21:12:14.156 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 6.6 MB/s | 4.1 MB 00:00 2024-04-05T21:12:14.481 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): buildah-1.33.6-2.module_el8+976+1136073 6.7 MB/s | 9.6 MB 00:01 2024-04-05T21:12:14.537 INFO:teuthology.orchestra.run.smithi188.stdout:(9/13): buildah-1.33.6-2.module_el8+976+1136073 6.0 MB/s | 9.6 MB 00:01 2024-04-05T21:12:14.582 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 526 kB/s | 53 kB 00:00 2024-04-05T21:12:14.595 INFO:teuthology.orchestra.run.smithi188.stdout:(10/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 5.1 MB/s | 2.5 MB 00:00 2024-04-05T21:12:14.621 INFO:teuthology.orchestra.run.smithi188.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 631 kB/s | 53 kB 00:00 2024-04-05T21:12:14.640 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 5.2 MB/s | 2.5 MB 00:00 2024-04-05T21:12:14.654 INFO:teuthology.orchestra.run.smithi188.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 701 kB/s | 41 kB 00:00 2024-04-05T21:12:14.665 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 492 kB/s | 41 kB 00:00 2024-04-05T21:12:14.897 INFO:teuthology.orchestra.run.smithi188.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 7.4 MB/s | 8.8 MB 00:01 2024-04-05T21:12:14.897 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:12:14.897 INFO:teuthology.orchestra.run.smithi188.stdout:Total 14 MB/s | 27 MB 00:02 2024-04-05T21:12:15.167 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:12:15.188 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:12:15.188 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:12:15.399 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 5.9 MB/s | 8.8 MB 00:01 2024-04-05T21:12:15.399 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:12:15.400 INFO:teuthology.orchestra.run.smithi121.stdout:Total 11 MB/s | 27 MB 00:02 2024-04-05T21:12:15.461 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:12:15.464 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:12:15.673 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:12:15.694 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:12:15.694 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:12:15.953 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:12:15.956 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:12:16.176 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:12:16.666 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:12:17.139 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-05T21:12:17.356 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-05T21:12:17.543 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-05T21:12:17.632 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-04-05T21:12:17.755 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-05T21:12:17.851 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-04-05T21:12:17.989 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-05T21:12:18.029 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-04-05T21:12:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-04-05T21:12:18.466 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-04-05T21:12:18.545 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-05T21:12:19.032 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-04-05T21:12:19.509 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-05T21:12:19.760 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-05T21:12:19.989 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-05T21:12:20.024 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-04-05T21:12:20.275 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-04-05T21:12:20.460 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-05T21:12:20.504 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-04-05T21:12:20.685 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-05T21:12:20.921 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-05T21:12:20.958 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-04-05T21:12:21.100 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-05T21:12:21.175 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-05T21:12:21.375 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-05T21:12:21.547 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-05T21:12:21.547 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-05T21:12:21.547 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-05T21:12:21.547 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-05T21:12:21.547 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-05T21:12:21.547 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-05T21:12:21.547 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-05T21:12:21.547 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-05T21:12:21.548 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-05T21:12:21.548 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-05T21:12:21.548 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-05T21:12:21.548 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-05T21:12:21.548 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-05T21:12:21.553 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-05T21:12:21.986 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-04-05T21:12:21.986 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-04-05T21:12:21.986 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-04-05T21:12:21.986 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-04-05T21:12:21.986 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-04-05T21:12:21.986 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-04-05T21:12:21.986 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-04-05T21:12:21.986 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-04-05T21:12:21.986 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-04-05T21:12:21.986 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-04-05T21:12:21.987 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-04-05T21:12:21.987 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-04-05T21:12:21.987 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-04-05T21:12:22.138 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-05T21:12:22.139 INFO:teuthology.orchestra.run.smithi188.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-05T21:12:22.140 INFO:teuthology.orchestra.run.smithi188.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-05T21:12:22.140 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:12:22.140 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:12:22.446 DEBUG:teuthology.parallel:result is None 2024-04-05T21:12:22.516 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-04-05T21:12:22.516 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:12:22.516 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:12:22.516 INFO:teuthology.orchestra.run.smithi121.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-04-05T21:12:22.516 INFO:teuthology.orchestra.run.smithi121.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-04-05T21:12:22.516 INFO:teuthology.orchestra.run.smithi121.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-04-05T21:12:22.517 INFO:teuthology.orchestra.run.smithi121.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-04-05T21:12:22.517 INFO:teuthology.orchestra.run.smithi121.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-04-05T21:12:22.517 INFO:teuthology.orchestra.run.smithi121.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-04-05T21:12:22.517 INFO:teuthology.orchestra.run.smithi121.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-04-05T21:12:22.517 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-04-05T21:12:22.517 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-04-05T21:12:22.517 INFO:teuthology.orchestra.run.smithi121.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-04-05T21:12:22.517 INFO:teuthology.orchestra.run.smithi121.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-04-05T21:12:22.517 INFO:teuthology.orchestra.run.smithi121.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-04-05T21:12:22.517 INFO:teuthology.orchestra.run.smithi121.stdout: yajl-2.1.0-12.el8.x86_64 2024-04-05T21:12:22.517 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:12:22.517 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:12:22.788 DEBUG:teuthology.parallel:result is None 2024-04-05T21:12:22.788 INFO:teuthology.run_tasks:Running task install... 2024-04-05T21:12:22.798 DEBUG:teuthology.task.install:project ceph 2024-04-05T21:12:22.798 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8919605b0a5dc1f1650c8aec3051c77c0e3e240e'}} 2024-04-05T21:12:22.798 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v18.2.1', 'flavor': 'default', 'sha1': '8919605b0a5dc1f1650c8aec3051c77c0e3e240e'} 2024-04-05T21:12:22.798 INFO:teuthology.task.install:Using flavor: default 2024-04-05T21:12:22.806 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-05T21:12:22.807 INFO:teuthology.task.install:extra packages: [] 2024-04-05T21:12:22.807 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8919605b0a5dc1f1650c8aec3051c77c0e3e240e', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-04-05T21:12:22.807 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-05T21:12:22.807 INFO:teuthology.packaging:ref: None 2024-04-05T21:12:22.807 INFO:teuthology.packaging:tag: v18.2.1 2024-04-05T21:12:22.807 INFO:teuthology.packaging:branch: None 2024-04-05T21:12:22.807 INFO:teuthology.packaging:sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:12:22.998 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v18.2.1^{} -> None 2024-04-05T21:12:22.999 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://github.com/ceph/ceph-ci; will also look in https://github.com/ceph/ceph 2024-04-05T21:12:23.303 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph v18.2.1^{} -> 7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-05T21:12:23.304 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-05T21:12:23.306 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8919605b0a5dc1f1650c8aec3051c77c0e3e240e', 'tag': 'v18.2.1', 'wait_for_package': False} 2024-04-05T21:12:23.307 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-05T21:12:23.307 INFO:teuthology.packaging:ref: None 2024-04-05T21:12:23.307 INFO:teuthology.packaging:tag: v18.2.1 2024-04-05T21:12:23.307 INFO:teuthology.packaging:branch: None 2024-04-05T21:12:23.307 INFO:teuthology.packaging:sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:12:23.307 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://github.com/ceph/ceph-ci; will also look in https://github.com/ceph/ceph 2024-04-05T21:12:23.307 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-05T21:12:23.453 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/8/flavors/default/ 2024-04-05T21:12:23.454 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-04-05T21:12:23.484 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-05T21:12:23.484 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:12:23.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-05T21:12:23.514 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-05T21:12:23.514 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-05T21:12:23.514 INFO:teuthology.packaging:ref: None 2024-04-05T21:12:23.514 INFO:teuthology.packaging:tag: v18.2.1 2024-04-05T21:12:23.514 INFO:teuthology.packaging:branch: None 2024-04-05T21:12:23.514 INFO:teuthology.packaging:sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:12:23.515 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-05T21:12:23.536 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/8/flavors/default/ 2024-04-05T21:12:23.537 INFO:teuthology.task.install.rpm:Package version is 18.2.1-0 2024-04-05T21:12:23.568 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/reef/7fe91d5d5842e04be3b4f514d6dd990c54b29c76/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-05T21:12:23.568 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-05T21:12:23.568 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-05T21:12:23.583 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-05T21:12:23.602 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-05T21:12:23.602 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-05T21:12:23.602 INFO:teuthology.packaging:ref: None 2024-04-05T21:12:23.602 INFO:teuthology.packaging:tag: v18.2.1 2024-04-05T21:12:23.602 INFO:teuthology.packaging:branch: None 2024-04-05T21:12:23.602 INFO:teuthology.packaging:sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:12:23.602 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v18.2.1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-05T21:12:23.662 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-05T21:12:23.672 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-05T21:12:23.729 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-04-05T21:12:23.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-04-05T21:12:23.752 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-05T21:12:23.820 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2024-04-05T21:12:23.821 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2024-04-05T21:12:23.993 INFO:teuthology.orchestra.run.smithi121.stdout:76 files removed 2024-04-05T21:12:24.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2024-04-05T21:12:24.092 INFO:teuthology.orchestra.run.smithi188.stdout:76 files removed 2024-04-05T21:12:24.120 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw 2024-04-05T21:12:24.548 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 639 kB/s | 79 kB 00:00 2024-04-05T21:12:24.606 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 978 kB/s | 79 kB 00:00 2024-04-05T21:12:24.707 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 152 kB/s | 9.4 kB 00:00 2024-04-05T21:12:24.708 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 78 kB/s | 9.4 kB 00:00 2024-04-05T21:12:24.833 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 18 kB/s | 1.8 kB 00:00 2024-04-05T21:12:25.138 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 4.4 kB/s | 1.8 kB 00:00 2024-04-05T21:12:26.321 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - AppStream 19 MB/s | 28 MB 00:01 2024-04-05T21:12:27.084 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - AppStream 14 MB/s | 28 MB 00:01 2024-04-05T21:12:31.943 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - BaseOS 17 MB/s | 10 MB 00:00 2024-04-05T21:12:32.726 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - BaseOS 16 MB/s | 10 MB 00:00 2024-04-05T21:12:33.827 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras 61 kB/s | 18 kB 00:00 2024-04-05T21:12:34.269 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 7.7 kB 00:00 2024-04-05T21:12:34.584 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras 66 kB/s | 18 kB 00:00 2024-04-05T21:12:34.902 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras common packages 37 kB/s | 7.7 kB 00:00 2024-04-05T21:12:35.437 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - PowerTools 4.6 MB/s | 4.9 MB 00:01 2024-04-05T21:12:36.036 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - PowerTools 4.7 MB/s | 4.9 MB 00:01 2024-04-05T21:12:36.530 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2024-04-05T21:12:37.088 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-04-05T21:12:38.517 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 16 MB 00:01 2024-04-05T21:12:39.074 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 16 MB 00:01 2024-04-05T21:12:42.389 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 390 kB/s | 24 kB 00:00 2024-04-05T21:12:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 473 kB/s | 24 kB 00:00 2024-04-05T21:12:46.015 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:12:46.017 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:12:46.018 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-05T21:12:46.018 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:12:46.018 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:12:46.018 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:18.2.1-0.el8 ceph 13 M 2024-04-05T21:12:46.018 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2024-04-05T21:12:46.018 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:18.2.1-0.el8 ceph 3.7 M 2024-04-05T21:12:46.018 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:18.2.1-0.el8 ceph 3.6 M 2024-04-05T21:12:46.018 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-05T21:12:46.018 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:18.2.1-0.el8 ceph 5.4 M 2024-04-05T21:12:46.018 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:18.2.1-0.el8 ceph 22 M 2024-04-05T21:12:46.018 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:18.2.1-0.el8 ceph 24 k 2024-04-05T21:12:46.018 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:18.2.1-0.el8 ceph 774 k 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:18.2.1-0.el8 ceph 474 k 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:18.2.1-0.el8 ceph 6.5 M 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el8 ceph 45 k 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:18.2.1-0.el8 ceph 123 k 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:18.2.1-0.el8 ceph 197 k 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:18.2.1-0.el8 ceph 371 k 2024-04-05T21:12:46.019 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:18.2.1-0.el8 ceph 365 k 2024-04-05T21:12:46.020 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:18.2.1-0.el8 ceph 107 k 2024-04-05T21:12:46.020 INFO:teuthology.orchestra.run.smithi188.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-05T21:12:46.020 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:12:46.020 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:12:46.020 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:12:46.020 INFO:teuthology.orchestra.run.smithi188.stdout:Install 21 Packages 2024-04-05T21:12:46.020 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 2 Packages 2024-04-05T21:12:46.020 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:12:46.020 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 59 M 2024-04-05T21:12:46.021 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:12:46.449 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:12:46.452 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:12:46.452 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-05T21:12:46.452 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:12:46.452 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:12:46.452 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:18.2.1-0.el8 ceph 13 M 2024-04-05T21:12:46.452 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-04-05T21:12:46.452 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:18.2.1-0.el8 ceph 3.7 M 2024-04-05T21:12:46.452 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:18.2.1-0.el8 ceph 3.6 M 2024-04-05T21:12:46.452 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-05T21:12:46.452 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:18.2.1-0.el8 ceph 5.4 M 2024-04-05T21:12:46.452 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:18.2.1-0.el8 ceph 22 M 2024-04-05T21:12:46.452 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:18.2.1-0.el8 ceph 24 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:18.2.1-0.el8 ceph 774 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:18.2.1-0.el8 ceph 474 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:18.2.1-0.el8 ceph 6.5 M 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:18.2.1-0.el8 ceph 45 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:18.2.1-0.el8 ceph 123 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:18.2.1-0.el8 ceph 197 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:18.2.1-0.el8 ceph 371 k 2024-04-05T21:12:46.453 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:18.2.1-0.el8 ceph 365 k 2024-04-05T21:12:46.454 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:18.2.1-0.el8 ceph 107 k 2024-04-05T21:12:46.454 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-05T21:12:46.454 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:12:46.454 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:12:46.454 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:12:46.454 INFO:teuthology.orchestra.run.smithi121.stdout:Install 21 Packages 2024-04-05T21:12:46.454 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2024-04-05T21:12:46.454 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:12:46.454 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 59 M 2024-04-05T21:12:46.455 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:12:47.081 INFO:teuthology.orchestra.run.smithi121.stdout:(1/23): ceph-base-18.2.1-0.el8.x86_64.rpm 16 MB/s | 5.4 MB 00:00 2024-04-05T21:12:47.090 INFO:teuthology.orchestra.run.smithi188.stdout:(1/23): ceph-base-18.2.1-0.el8.x86_64.rpm 7.9 MB/s | 5.4 MB 00:00 2024-04-05T21:12:47.157 INFO:teuthology.orchestra.run.smithi188.stdout:(2/23): ceph-selinux-18.2.1-0.el8.x86_64.rpm 364 kB/s | 24 kB 00:00 2024-04-05T21:12:47.214 INFO:teuthology.orchestra.run.smithi121.stdout:(2/23): ceph-radosgw-18.2.1-0.el8.x86_64.rpm 28 MB/s | 13 MB 00:00 2024-04-05T21:12:47.224 INFO:teuthology.orchestra.run.smithi188.stdout:(3/23): libcephfs2-18.2.1-0.el8.x86_64.rpm 11 MB/s | 774 kB 00:00 2024-04-05T21:12:47.231 INFO:teuthology.orchestra.run.smithi121.stdout:(3/23): ceph-selinux-18.2.1-0.el8.x86_64.rpm 162 kB/s | 24 kB 00:00 2024-04-05T21:12:47.265 INFO:teuthology.orchestra.run.smithi121.stdout:(4/23): libcephfs2-18.2.1-0.el8.x86_64.rpm 15 MB/s | 774 kB 00:00 2024-04-05T21:12:47.282 INFO:teuthology.orchestra.run.smithi188.stdout:(4/23): libradosstriper1-18.2.1-0.el8.x86_64.rp 8.0 MB/s | 474 kB 00:00 2024-04-05T21:12:47.290 INFO:teuthology.orchestra.run.smithi121.stdout:(5/23): libradosstriper1-18.2.1-0.el8.x86_64.rp 7.9 MB/s | 474 kB 00:00 2024-04-05T21:12:47.441 INFO:teuthology.orchestra.run.smithi188.stdout:(5/23): ceph-radosgw-18.2.1-0.el8.x86_64.rpm 13 MB/s | 13 MB 00:01 2024-04-05T21:12:47.466 INFO:teuthology.orchestra.run.smithi188.stdout:(6/23): python3-ceph-argparse-18.2.1-0.el8.x86_ 1.8 MB/s | 45 kB 00:00 2024-04-05T21:12:47.498 INFO:teuthology.orchestra.run.smithi121.stdout:(6/23): librgw2-18.2.1-0.el8.x86_64.rpm 28 MB/s | 6.5 MB 00:00 2024-04-05T21:12:47.515 INFO:teuthology.orchestra.run.smithi121.stdout:(7/23): python3-ceph-argparse-18.2.1-0.el8.x86_ 198 kB/s | 45 kB 00:00 2024-04-05T21:12:47.541 INFO:teuthology.orchestra.run.smithi121.stdout:(8/23): python3-ceph-common-18.2.1-0.el8.x86_64 2.9 MB/s | 123 kB 00:00 2024-04-05T21:12:47.566 INFO:teuthology.orchestra.run.smithi121.stdout:(9/23): python3-cephfs-18.2.1-0.el8.x86_64.rpm 3.8 MB/s | 197 kB 00:00 2024-04-05T21:12:47.608 INFO:teuthology.orchestra.run.smithi121.stdout:(10/23): python3-rados-18.2.1-0.el8.x86_64.rpm 5.4 MB/s | 371 kB 00:00 2024-04-05T21:12:47.650 INFO:teuthology.orchestra.run.smithi121.stdout:(11/23): python3-rbd-18.2.1-0.el8.x86_64.rpm 4.3 MB/s | 365 kB 00:00 2024-04-05T21:12:47.675 INFO:teuthology.orchestra.run.smithi121.stdout:(12/23): python3-rgw-18.2.1-0.el8.x86_64.rpm 1.6 MB/s | 107 kB 00:00 2024-04-05T21:12:47.700 INFO:teuthology.orchestra.run.smithi188.stdout:(7/23): ceph-common-18.2.1-0.el8.x86_64.rpm 17 MB/s | 22 MB 00:01 2024-04-05T21:12:47.726 INFO:teuthology.orchestra.run.smithi188.stdout:(8/23): python3-ceph-common-18.2.1-0.el8.x86_64 477 kB/s | 123 kB 00:00 2024-04-05T21:12:47.751 INFO:teuthology.orchestra.run.smithi188.stdout:(9/23): python3-cephfs-18.2.1-0.el8.x86_64.rpm 3.8 MB/s | 197 kB 00:00 2024-04-05T21:12:47.776 INFO:teuthology.orchestra.run.smithi188.stdout:(10/23): python3-rados-18.2.1-0.el8.x86_64.rpm 7.2 MB/s | 371 kB 00:00 2024-04-05T21:12:47.810 INFO:teuthology.orchestra.run.smithi188.stdout:(11/23): python3-rbd-18.2.1-0.el8.x86_64.rpm 6.1 MB/s | 365 kB 00:00 2024-04-05T21:12:47.835 INFO:teuthology.orchestra.run.smithi188.stdout:(12/23): python3-rgw-18.2.1-0.el8.x86_64.rpm 1.8 MB/s | 107 kB 00:00 2024-04-05T21:12:47.909 INFO:teuthology.orchestra.run.smithi121.stdout:(13/23): ceph-common-18.2.1-0.el8.x86_64.rpm 19 MB/s | 22 MB 00:01 2024-04-05T21:12:47.960 INFO:teuthology.orchestra.run.smithi188.stdout:(13/23): librgw2-18.2.1-0.el8.x86_64.rpm 9.7 MB/s | 6.5 MB 00:00 2024-04-05T21:12:48.085 INFO:teuthology.orchestra.run.smithi121.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 251 kB/s | 44 kB 00:00 2024-04-05T21:12:48.136 INFO:teuthology.orchestra.run.smithi188.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 250 kB/s | 44 kB 00:00 2024-04-05T21:12:48.143 INFO:teuthology.orchestra.run.smithi121.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 555 kB/s | 259 kB 00:00 2024-04-05T21:12:48.169 INFO:teuthology.orchestra.run.smithi121.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 563 kB/s | 47 kB 00:00 2024-04-05T21:12:48.219 INFO:teuthology.orchestra.run.smithi121.stdout:(17/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.2 MB/s | 677 kB 00:00 2024-04-05T21:12:48.222 INFO:teuthology.orchestra.run.smithi188.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 550 kB/s | 47 kB 00:00 2024-04-05T21:12:48.253 INFO:teuthology.orchestra.run.smithi188.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 621 kB/s | 259 kB 00:00 2024-04-05T21:12:48.261 INFO:teuthology.orchestra.run.smithi121.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 640 kB/s | 59 kB 00:00 2024-04-05T21:12:48.311 INFO:teuthology.orchestra.run.smithi121.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2024-04-05T21:12:48.337 INFO:teuthology.orchestra.run.smithi121.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 640 kB/s | 75 kB 00:00 2024-04-05T21:12:48.462 INFO:teuthology.orchestra.run.smithi121.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 8.7 MB/s | 1.7 MB 00:00 2024-04-05T21:12:48.493 INFO:teuthology.orchestra.run.smithi188.stdout:(17/23): librdkafka-1.6.1-1.el8.x86_64.rpm 992 kB/s | 677 kB 00:00 2024-04-05T21:12:48.553 INFO:teuthology.orchestra.run.smithi121.stdout:(22/23): librados2-18.2.1-0.el8.x86_64.rpm 15 MB/s | 3.7 MB 00:00 2024-04-05T21:12:48.604 INFO:teuthology.orchestra.run.smithi121.stdout:(23/23): librbd1-18.2.1-0.el8.x86_64.rpm 13 MB/s | 3.6 MB 00:00 2024-04-05T21:12:48.606 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:12:48.606 INFO:teuthology.orchestra.run.smithi121.stdout:Total 28 MB/s | 59 MB 00:02 2024-04-05T21:12:48.646 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:12:48.691 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:12:48.692 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:12:49.037 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:12:49.040 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:12:49.303 INFO:teuthology.orchestra.run.smithi188.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 93 kB/s | 75 kB 00:00 2024-04-05T21:12:49.404 INFO:teuthology.orchestra.run.smithi188.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 51 kB/s | 59 kB 00:01 2024-04-05T21:12:49.429 INFO:teuthology.orchestra.run.smithi188.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 254 kB/s | 306 kB 00:01 2024-04-05T21:12:49.521 INFO:teuthology.orchestra.run.smithi188.stdout:(21/23): librados2-18.2.1-0.el8.x86_64.rpm 31 MB/s | 3.7 MB 00:00 2024-04-05T21:12:49.613 INFO:teuthology.orchestra.run.smithi188.stdout:(22/23): librbd1-18.2.1-0.el8.x86_64.rpm 19 MB/s | 3.6 MB 00:00 2024-04-05T21:12:49.738 INFO:teuthology.orchestra.run.smithi188.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 4.0 MB/s | 1.7 MB 00:00 2024-04-05T21:12:49.741 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:12:49.741 INFO:teuthology.orchestra.run.smithi188.stdout:Total 16 MB/s | 59 MB 00:03 2024-04-05T21:12:49.782 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:12:49.828 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:12:49.829 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:12:50.184 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:12:50.187 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:12:50.207 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:12:50.404 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-05T21:12:50.626 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2024-04-05T21:12:50.798 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2024-04-05T21:12:51.211 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2024-04-05T21:12:51.408 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:12:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:18.2.1-0.el8.x86_64 4/25 2024-04-05T21:12:51.625 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2024-04-05T21:12:51.864 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2024-04-05T21:12:52.089 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:18.2.1-0.el8.x86_64 4/25 2024-04-05T21:12:52.102 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2024-04-05T21:12:52.531 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2024-04-05T21:12:52.681 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:18.2.1-0.el8.x86_64 5/25 2024-04-05T21:12:52.864 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:18.2.1-0.el8.x86_64 6/25 2024-04-05T21:12:53.154 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:18.2.1-0.el8.x86_64 4/25 2024-04-05T21:12:53.249 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:18.2.1-0.el8.x86_64 4/25 2024-04-05T21:12:53.255 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el8.x86_64 6/25 2024-04-05T21:12:53.439 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:18.2.1-0.el8.x86_64 7/25 2024-04-05T21:12:53.485 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:18.2.1-0.el8.x86_64 7/25 2024-04-05T21:12:53.510 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:18.2.1-0.el8.x86_64 5/25 2024-04-05T21:12:53.715 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-04-05T21:12:53.719 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:18.2.1-0.el8.x86_64 6/25 2024-04-05T21:12:53.904 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-04-05T21:12:54.058 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el8.x86_64 10/25 2024-04-05T21:12:54.117 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:18.2.1-0.el8.x86_64 6/25 2024-04-05T21:12:54.240 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:18.2.1-0.el8.x86_64 11/25 2024-04-05T21:12:54.343 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:18.2.1-0.el8.x86_64 7/25 2024-04-05T21:12:54.404 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:18.2.1-0.el8.x86_64 7/25 2024-04-05T21:12:54.464 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:18.2.1-0.el8.x86_64 12/25 2024-04-05T21:12:54.614 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:18.2.1-0.el8.x86_64 13/25 2024-04-05T21:12:54.654 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-04-05T21:12:54.677 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el8.x86_64 13/25 2024-04-05T21:12:54.825 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-04-05T21:12:54.881 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2024-04-05T21:12:55.013 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:18.2.1-0.el8.x86_64 10/25 2024-04-05T21:12:55.196 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:18.2.1-0.el8.x86_64 11/25 2024-04-05T21:12:55.411 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:18.2.1-0.el8.x86_64 12/25 2024-04-05T21:12:55.576 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:18.2.1-0.el8.x86_64 13/25 2024-04-05T21:12:55.645 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:18.2.1-0.el8.x86_64 13/25 2024-04-05T21:12:55.673 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2024-04-05T21:12:55.814 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:18.2.1-0.el8.x86_64 16/25 2024-04-05T21:12:55.869 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2024-04-05T21:12:55.889 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:18.2.1-0.el8.x86_64 16/25 2024-04-05T21:12:56.105 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:18.2.1-0.el8.x86_64 17/25 2024-04-05T21:12:56.288 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-05T21:12:56.514 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:18.2.1-0.el8.x86_64 19/25 2024-04-05T21:12:56.648 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2024-04-05T21:12:56.793 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:18.2.1-0.el8.x86_64 16/25 2024-04-05T21:12:56.853 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:18.2.1-0.el8.x86_64 16/25 2024-04-05T21:12:57.042 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:18.2.1-0.el8.x86_64 17/25 2024-04-05T21:12:57.252 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-05T21:12:57.468 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:18.2.1-0.el8.x86_64 19/25 2024-04-05T21:12:59.065 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el8.x86_64 20/25 2024-04-05T21:12:59.330 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:18.2.1-0.el8.x86_64 20/25 2024-04-05T21:12:59.937 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el8.x86_64 20/25 2024-04-05T21:13:00.012 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el8.x86_64 20/25 2024-04-05T21:13:00.246 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:18.2.1-0.el8.x86_64 21/25 2024-04-05T21:13:00.268 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:18.2.1-0.el8.x86_64 20/25 2024-04-05T21:13:00.338 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el8.x86_64 21/25 2024-04-05T21:13:00.538 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:18.2.1-0.el8.x86_64 22/25 2024-04-05T21:13:00.875 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:18.2.1-0.el8.x86_64 20/25 2024-04-05T21:13:01.163 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:18.2.1-0.el8.x86_64 21/25 2024-04-05T21:13:01.274 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:18.2.1-0.el8.x86_64 21/25 2024-04-05T21:13:01.475 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:18.2.1-0.el8.x86_64 22/25 2024-04-05T21:13:19.804 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el8.x86_64 22/25 2024-04-05T21:13:19.804 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-04-05T21:13:19.804 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-04-05T21:13:19.804 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2024-04-05T21:13:19.804 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /run 2024-04-05T21:13:19.804 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-04-05T21:13:19.804 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-04-05T21:13:19.804 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-04-05T21:13:19.804 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-04-05T21:13:19.805 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2024-04-05T21:13:19.805 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:20.740 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:18.2.1-0.el8.x86_64 22/25 2024-04-05T21:13:20.740 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2024-04-05T21:13:20.740 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2024-04-05T21:13:20.740 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2024-04-05T21:13:20.740 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /run 2024-04-05T21:13:20.740 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2024-04-05T21:13:20.740 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2024-04-05T21:13:20.740 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2024-04-05T21:13:20.741 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2024-04-05T21:13:20.741 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2024-04-05T21:13:20.741 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:21.175 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:18.2.1-0.el8.x86_64 23/25 2024-04-05T21:13:21.195 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el8.x86_64 23/25 2024-04-05T21:13:21.195 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-05T21:13:21.393 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-05T21:13:21.394 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-05T21:13:22.109 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:18.2.1-0.el8.x86_64 23/25 2024-04-05T21:13:22.116 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-05T21:13:22.116 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:18.2.1-0.el8.x86_64 1/25 2024-04-05T21:13:22.116 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:18.2.1-0.el8.x86_64 2/25 2024-04-05T21:13:22.116 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el8.x86_64 3/25 2024-04-05T21:13:22.116 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:18.2.1-0.el8.x86_64 4/25 2024-04-05T21:13:22.116 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:18.2.1-0.el8.x86_64 5/25 2024-04-05T21:13:22.116 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:18.2.1-0.el8.x86_64 6/25 2024-04-05T21:13:22.116 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:18.2.1-0.el8.x86_64 7/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el8.x86_64 8/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el8.x86_64 9/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:18.2.1-0.el8.x86_64 10/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:18.2.1-0.el8.x86_64 11/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:18.2.1-0.el8.x86_64 12/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:18.2.1-0.el8.x86_64 13/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:18.2.1-0.el8.x86_64 22/25 2024-04-05T21:13:22.117 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-05T21:13:22.118 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:18.2.1-0.el8.x86_64 24/25 2024-04-05T21:13:22.130 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-0.el8.x86_64 23/25 2024-04-05T21:13:22.130 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-05T21:13:22.389 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-05T21:13:22.389 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-05T21:13:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-05T21:13:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:22.946 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-04-05T21:13:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:18.2.1-0.el8.x86_64 librbd1-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.946 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:13:22.946 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-05T21:13:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-05T21:13:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-05T21:13:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-05T21:13:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-05T21:13:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-05T21:13:22.947 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-05T21:13:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-05T21:13:22.948 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:22.948 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:13:23.142 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-05T21:13:23.142 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:18.2.1-0.el8.x86_64 1/25 2024-04-05T21:13:23.142 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:18.2.1-0.el8.x86_64 2/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:18.2.1-0.el8.x86_64 3/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:18.2.1-0.el8.x86_64 4/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:18.2.1-0.el8.x86_64 5/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:18.2.1-0.el8.x86_64 6/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:18.2.1-0.el8.x86_64 7/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:18.2.1-0.el8.x86_64 8/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:18.2.1-0.el8.x86_64 9/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:18.2.1-0.el8.x86_64 10/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:18.2.1-0.el8.x86_64 11/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:18.2.1-0.el8.x86_64 12/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:18.2.1-0.el8.x86_64 13/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-04-05T21:13:23.143 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-05T21:13:23.144 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-04-05T21:13:23.144 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-05T21:13:23.144 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-04-05T21:13:23.144 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-04-05T21:13:23.144 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-04-05T21:13:23.144 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:18.2.1-0.el8.x86_64 22/25 2024-04-05T21:13:23.144 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-05T21:13:23.144 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:18.2.1-0.el8.x86_64 24/25 2024-04-05T21:13:23.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2024-04-05T21:13:24.032 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:42 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:13:24.119 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-05T21:13:24.119 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:24.119 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2024-04-05T21:13:24.119 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:18.2.1-0.el8.x86_64 librbd1-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.119 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-05T21:13:24.120 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.121 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.121 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.121 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-05T21:13:24.121 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.121 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.121 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:24.121 INFO:teuthology.orchestra.run.smithi188.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-05T21:13:24.121 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:24.121 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:13:24.443 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-test 2024-04-05T21:13:25.175 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:43 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:13:25.665 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:13:25.666 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:13:25.666 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-05T21:13:25.666 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:13:25.666 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:13:25.666 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:18.2.1-0.el8 ceph 47 M 2024-04-05T21:13:25.666 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-05T21:13:25.667 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-05T21:13:25.667 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:18.2.1-0.el8 ceph 141 k 2024-04-05T21:13:25.667 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-05T21:13:25.667 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-05T21:13:25.667 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-05T21:13:25.667 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:25.667 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:13:25.667 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:13:25.667 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-04-05T21:13:25.667 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:25.668 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 48 M 2024-04-05T21:13:25.668 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 212 M 2024-04-05T21:13:25.668 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:13:26.159 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 697 kB/s | 203 kB 00:00 2024-04-05T21:13:26.193 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): libcephsqlite-18.2.1-0.el8.x86_64.rpm 433 kB/s | 141 kB 00:00 2024-04-05T21:13:26.246 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-05T21:13:26.327 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 849 kB/s | 71 kB 00:00 2024-04-05T21:13:26.544 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 922 kB/s | 323 kB 00:00 2024-04-05T21:13:26.815 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:13:26.816 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:13:26.816 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-05T21:13:26.817 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:13:26.817 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:13:26.817 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:18.2.1-0.el8 ceph 47 M 2024-04-05T21:13:26.817 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-05T21:13:26.817 INFO:teuthology.orchestra.run.smithi188.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-05T21:13:26.817 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:18.2.1-0.el8 ceph 141 k 2024-04-05T21:13:26.817 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-05T21:13:26.818 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-05T21:13:26.818 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-05T21:13:26.818 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:26.818 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:13:26.818 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:13:26.818 INFO:teuthology.orchestra.run.smithi188.stdout:Install 6 Packages 2024-04-05T21:13:26.818 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:26.819 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 48 M 2024-04-05T21:13:26.901 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 212 M 2024-04-05T21:13:26.901 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:13:27.111 INFO:teuthology.orchestra.run.smithi188.stdout:(1/6): libcephsqlite-18.2.1-0.el8.x86_64.rpm 2.7 MB/s | 141 kB 00:00 2024-04-05T21:13:27.144 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): ceph-test-18.2.1-0.el8.x86_64.rpm 37 MB/s | 47 MB 00:01 2024-04-05T21:13:27.146 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:13:27.146 INFO:teuthology.orchestra.run.smithi121.stdout:Total 32 MB/s | 48 MB 00:01 2024-04-05T21:13:27.187 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:13:27.208 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:13:27.208 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:13:27.287 INFO:teuthology.orchestra.run.smithi188.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 895 kB/s | 203 kB 00:00 2024-04-05T21:13:27.321 INFO:teuthology.orchestra.run.smithi188.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 901 kB/s | 188 kB 00:00 2024-04-05T21:13:27.501 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:13:27.504 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:13:27.804 INFO:teuthology.orchestra.run.smithi188.stdout:(4/6): ceph-test-18.2.1-0.el8.x86_64.rpm 63 MB/s | 47 MB 00:00 2024-04-05T21:13:27.830 INFO:teuthology.orchestra.run.smithi188.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 139 kB/s | 71 kB 00:00 2024-04-05T21:13:27.855 INFO:teuthology.orchestra.run.smithi188.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 569 kB/s | 323 kB 00:00 2024-04-05T21:13:27.857 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:13:27.857 INFO:teuthology.orchestra.run.smithi188.stdout:Total 46 MB/s | 48 MB 00:01 2024-04-05T21:13:27.899 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:13:27.920 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:13:27.920 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:13:27.975 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:13:28.159 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-04-05T21:13:28.214 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:13:28.217 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:13:28.342 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-04-05T21:13:28.448 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-05T21:13:28.565 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-05T21:13:28.684 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:13:28.752 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-04-05T21:13:28.948 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:18.2.1-0.el8.x86_64 5/6 2024-04-05T21:13:28.949 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-04-05T21:13:29.154 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-04-05T21:13:29.313 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-05T21:13:29.376 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-05T21:13:29.573 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-04-05T21:13:29.710 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:18.2.1-0.el8.x86_64 5/6 2024-04-05T21:13:33.410 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el8.x86_64 5/6 2024-04-05T21:13:33.679 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:18.2.1-0.el8.x86_64 6/6 2024-04-05T21:13:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el8.x86_64 6/6 2024-04-05T21:13:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:18.2.1-0.el8.x86_64 1/6 2024-04-05T21:13:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:18.2.1-0.el8.x86_64 2/6 2024-04-05T21:13:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-04-05T21:13:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-04-05T21:13:33.950 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-04-05T21:13:34.223 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:18.2.1-0.el8.x86_64 5/6 2024-04-05T21:13:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-04-05T21:13:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:34.383 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:13:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:18.2.1-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-05T21:13:34.383 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:18.2.1-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-04-05T21:13:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-04-05T21:13:34.384 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:34.384 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:13:34.466 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:18.2.1-0.el8.x86_64 6/6 2024-04-05T21:13:34.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2024-04-05T21:13:34.755 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-test-2:18.2.1-0.el8.x86_64 6/6 2024-04-05T21:13:34.756 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:18.2.1-0.el8.x86_64 1/6 2024-04-05T21:13:34.756 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:18.2.1-0.el8.x86_64 2/6 2024-04-05T21:13:34.756 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-04-05T21:13:34.756 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-04-05T21:13:34.756 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-04-05T21:13:35.247 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-04-05T21:13:35.247 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:35.247 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:13:35.247 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:18.2.1-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-05T21:13:35.247 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:18.2.1-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-04-05T21:13:35.247 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-04-05T21:13:35.247 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:35.247 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:13:35.366 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:53 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:13:35.469 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph 2024-04-05T21:13:36.203 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:54 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:13:37.011 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:13:37.014 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:13:37.014 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-05T21:13:37.014 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:13:37.014 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:13:37.015 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:18.2.1-0.el8 ceph 6.3 k 2024-04-05T21:13:37.015 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-05T21:13:37.015 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:18.2.1-0.el8 ceph 2.3 M 2024-04-05T21:13:37.015 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:18.2.1-0.el8 ceph 1.6 M 2024-04-05T21:13:37.015 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el8 ceph-noarch 242 k 2024-04-05T21:13:37.015 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:18.2.1-0.el8 ceph 4.6 M 2024-04-05T21:13:37.015 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:18.2.1-0.el8 ceph 18 M 2024-04-05T21:13:37.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-05T21:13:37.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-05T21:13:37.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-05T21:13:37.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-05T21:13:37.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-05T21:13:37.016 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-05T21:13:37.016 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-05T21:13:37.016 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-05T21:13:37.016 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-05T21:13:37.016 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-05T21:13:37.016 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-05T21:13:37.016 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-05T21:13:37.016 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-05T21:13:37.016 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-05T21:13:37.016 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-05T21:13:37.017 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-05T21:13:37.017 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-05T21:13:37.017 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-05T21:13:37.017 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-05T21:13:37.017 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-05T21:13:37.017 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-05T21:13:37.017 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-05T21:13:37.017 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-05T21:13:37.017 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-05T21:13:37.017 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-05T21:13:37.018 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-05T21:13:37.018 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:37.018 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:13:37.018 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:13:37.018 INFO:teuthology.orchestra.run.smithi121.stdout:Install 32 Packages 2024-04-05T21:13:37.018 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:37.019 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 30 M 2024-04-05T21:13:37.019 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 119 M 2024-04-05T21:13:37.019 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:13:37.388 INFO:teuthology.orchestra.run.smithi121.stdout:(1/32): ceph-18.2.1-0.el8.x86_64.rpm 85 kB/s | 6.3 kB 00:00 2024-04-05T21:13:37.463 INFO:teuthology.orchestra.run.smithi121.stdout:(2/32): ceph-mds-18.2.1-0.el8.x86_64.rpm 15 MB/s | 2.3 MB 00:00 2024-04-05T21:13:37.589 INFO:teuthology.orchestra.run.smithi121.stdout:(3/32): ceph-mon-18.2.1-0.el8.x86_64.rpm 23 MB/s | 4.6 MB 00:00 2024-04-05T21:13:37.622 INFO:teuthology.orchestra.run.smithi121.stdout:(4/32): ceph-mgr-18.2.1-0.el8.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-04-05T21:13:37.648 INFO:teuthology.orchestra.run.smithi121.stdout:(5/32): ceph-mgr-modules-core-18.2.1-0.el8.noar 4.0 MB/s | 242 kB 00:00 2024-04-05T21:13:37.798 INFO:teuthology.orchestra.run.smithi121.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 259 kB/s | 39 kB 00:00 2024-04-05T21:13:37.820 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:13:37.824 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:13:37.824 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-05T21:13:37.824 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:13:37.824 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:13:37.824 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:18.2.1-0.el8 ceph 6.3 k 2024-04-05T21:13:37.824 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-05T21:13:37.824 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:18.2.1-0.el8 ceph 2.3 M 2024-04-05T21:13:37.824 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:18.2.1-0.el8 ceph 1.6 M 2024-04-05T21:13:37.824 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:18.2.1-0.el8 ceph-noarch 242 k 2024-04-05T21:13:37.824 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:18.2.1-0.el8 ceph 4.6 M 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:18.2.1-0.el8 ceph 18 M 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-05T21:13:37.825 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-05T21:13:37.826 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:37.827 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:13:37.827 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:13:37.827 INFO:teuthology.orchestra.run.smithi188.stdout:Install 32 Packages 2024-04-05T21:13:37.827 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:37.827 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 30 M 2024-04-05T21:13:37.827 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 119 M 2024-04-05T21:13:37.827 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:13:37.898 INFO:teuthology.orchestra.run.smithi121.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 569 kB/s | 157 kB 00:00 2024-04-05T21:13:38.041 INFO:teuthology.orchestra.run.smithi188.stdout:(1/32): ceph-18.2.1-0.el8.x86_64.rpm 139 kB/s | 6.3 kB 00:00 2024-04-05T21:13:38.073 INFO:teuthology.orchestra.run.smithi121.stdout:(8/32): ceph-osd-18.2.1-0.el8.x86_64.rpm 29 MB/s | 18 MB 00:00 2024-04-05T21:13:38.099 INFO:teuthology.orchestra.run.smithi121.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 344 kB/s | 103 kB 00:00 2024-04-05T21:13:38.108 INFO:teuthology.orchestra.run.smithi188.stdout:(2/32): ceph-mgr-18.2.1-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-05T21:13:38.124 INFO:teuthology.orchestra.run.smithi121.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 253 kB/s | 57 kB 00:00 2024-04-05T21:13:38.158 INFO:teuthology.orchestra.run.smithi188.stdout:(3/32): ceph-mds-18.2.1-0.el8.x86_64.rpm 14 MB/s | 2.3 MB 00:00 2024-04-05T21:13:38.200 INFO:teuthology.orchestra.run.smithi188.stdout:(4/32): ceph-mgr-modules-core-18.2.1-0.el8.noar 5.7 MB/s | 242 kB 00:00 2024-04-05T21:13:38.242 INFO:teuthology.orchestra.run.smithi121.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.7 MB/s | 249 kB 00:00 2024-04-05T21:13:38.275 INFO:teuthology.orchestra.run.smithi121.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.2 MB/s | 457 kB 00:00 2024-04-05T21:13:38.275 INFO:teuthology.orchestra.run.smithi188.stdout:(5/32): ceph-mon-18.2.1-0.el8.x86_64.rpm 20 MB/s | 4.6 MB 00:00 2024-04-05T21:13:38.308 INFO:teuthology.orchestra.run.smithi121.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.6 MB/s | 109 kB 00:00 2024-04-05T21:13:38.333 INFO:teuthology.orchestra.run.smithi121.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 3.0 MB/s | 641 kB 00:00 2024-04-05T21:13:38.501 INFO:teuthology.orchestra.run.smithi188.stdout:(6/32): ceph-osd-18.2.1-0.el8.x86_64.rpm 45 MB/s | 18 MB 00:00 2024-04-05T21:13:38.526 INFO:teuthology.orchestra.run.smithi188.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 155 kB/s | 39 kB 00:00 2024-04-05T21:13:38.552 INFO:teuthology.orchestra.run.smithi188.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 447 kB/s | 157 kB 00:00 2024-04-05T21:13:38.625 INFO:teuthology.orchestra.run.smithi121.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 124 kB/s | 44 kB 00:00 2024-04-05T21:13:38.627 INFO:teuthology.orchestra.run.smithi188.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 566 kB/s | 57 kB 00:00 2024-04-05T21:13:38.653 INFO:teuthology.orchestra.run.smithi188.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.4 MB/s | 457 kB 00:00 2024-04-05T21:13:38.703 INFO:teuthology.orchestra.run.smithi188.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 514 kB/s | 103 kB 00:00 2024-04-05T21:13:38.717 INFO:teuthology.orchestra.run.smithi121.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 450 kB/s | 173 kB 00:00 2024-04-05T21:13:38.736 INFO:teuthology.orchestra.run.smithi188.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.2 MB/s | 249 kB 00:00 2024-04-05T21:13:38.743 INFO:teuthology.orchestra.run.smithi121.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 427 kB/s | 185 kB 00:00 2024-04-05T21:13:38.770 INFO:teuthology.orchestra.run.smithi188.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 5.3 MB/s | 641 kB 00:00 2024-04-05T21:13:38.795 INFO:teuthology.orchestra.run.smithi188.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 1.2 MB/s | 109 kB 00:00 2024-04-05T21:13:38.818 INFO:teuthology.orchestra.run.smithi121.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-04-05T21:13:38.843 INFO:teuthology.orchestra.run.smithi121.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2024-04-05T21:13:38.885 INFO:teuthology.orchestra.run.smithi121.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2024-04-05T21:13:38.910 INFO:teuthology.orchestra.run.smithi121.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 529 kB/s | 49 kB 00:00 2024-04-05T21:13:38.961 INFO:teuthology.orchestra.run.smithi121.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 501 kB/s | 59 kB 00:00 2024-04-05T21:13:38.994 INFO:teuthology.orchestra.run.smithi121.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2024-04-05T21:13:38.996 INFO:teuthology.orchestra.run.smithi188.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 168 kB/s | 44 kB 00:00 2024-04-05T21:13:39.011 INFO:teuthology.orchestra.run.smithi121.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 154 kB/s | 16 kB 00:00 2024-04-05T21:13:39.054 INFO:teuthology.orchestra.run.smithi121.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 202 kB/s | 19 kB 00:00 2024-04-05T21:13:39.079 INFO:teuthology.orchestra.run.smithi121.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 285 kB/s | 24 kB 00:00 2024-04-05T21:13:39.096 INFO:teuthology.orchestra.run.smithi121.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 348 kB/s | 29 kB 00:00 2024-04-05T21:13:39.154 INFO:teuthology.orchestra.run.smithi121.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2024-04-05T21:13:39.163 INFO:teuthology.orchestra.run.smithi188.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 472 kB/s | 185 kB 00:00 2024-04-05T21:13:39.188 INFO:teuthology.orchestra.run.smithi188.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 445 kB/s | 173 kB 00:00 2024-04-05T21:13:39.188 INFO:teuthology.orchestra.run.smithi121.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2024-04-05T21:13:39.230 INFO:teuthology.orchestra.run.smithi188.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2024-04-05T21:13:39.255 INFO:teuthology.orchestra.run.smithi121.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 851 kB/s | 85 kB 00:00 2024-04-05T21:13:39.263 INFO:teuthology.orchestra.run.smithi188.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2024-04-05T21:13:39.280 INFO:teuthology.orchestra.run.smithi188.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 191 kB/s | 18 kB 00:00 2024-04-05T21:13:39.306 INFO:teuthology.orchestra.run.smithi188.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 644 kB/s | 49 kB 00:00 2024-04-05T21:13:39.313 INFO:teuthology.orchestra.run.smithi121.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.1 MB/s | 251 kB 00:00 2024-04-05T21:13:39.331 INFO:teuthology.orchestra.run.smithi188.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 874 kB/s | 59 kB 00:00 2024-04-05T21:13:39.339 INFO:teuthology.orchestra.run.smithi121.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 152 kB/s | 23 kB 00:00 2024-04-05T21:13:39.340 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:13:39.340 INFO:teuthology.orchestra.run.smithi121.stdout:Total 13 MB/s | 30 MB 00:02 2024-04-05T21:13:39.401 INFO:teuthology.orchestra.run.smithi188.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 164 kB/s | 16 kB 00:00 2024-04-05T21:13:39.421 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:13:39.424 INFO:teuthology.orchestra.run.smithi188.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 203 kB/s | 19 kB 00:00 2024-04-05T21:13:39.449 INFO:teuthology.orchestra.run.smithi188.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2024-04-05T21:13:39.458 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:13:39.458 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:13:39.474 INFO:teuthology.orchestra.run.smithi188.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 329 kB/s | 24 kB 00:00 2024-04-05T21:13:39.499 INFO:teuthology.orchestra.run.smithi188.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 387 kB/s | 29 kB 00:00 2024-04-05T21:13:39.525 INFO:teuthology.orchestra.run.smithi188.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 364 kB/s | 27 kB 00:00 2024-04-05T21:13:39.600 INFO:teuthology.orchestra.run.smithi188.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2024-04-05T21:13:39.625 INFO:teuthology.orchestra.run.smithi188.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.0 MB/s | 251 kB 00:00 2024-04-05T21:13:39.679 INFO:teuthology.orchestra.run.smithi188.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 554 kB/s | 85 kB 00:00 2024-04-05T21:13:39.709 INFO:teuthology.orchestra.run.smithi188.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 211 kB/s | 23 kB 00:00 2024-04-05T21:13:39.713 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:13:39.713 INFO:teuthology.orchestra.run.smithi188.stdout:Total 16 MB/s | 30 MB 00:01 2024-04-05T21:13:39.770 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:13:39.773 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:13:39.793 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:13:39.831 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:13:39.831 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:13:40.149 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:13:40.152 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:13:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:13:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-05T21:13:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-05T21:13:41.347 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:13:41.569 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-05T21:13:41.583 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-05T21:13:41.835 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-05T21:13:41.840 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-05T21:13:42.041 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-05T21:13:42.073 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-05T21:13:42.247 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-05T21:13:42.329 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-05T21:13:42.431 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-05T21:13:42.520 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-05T21:13:42.634 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-05T21:13:42.750 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-05T21:13:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-05T21:13:42.960 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-05T21:13:43.089 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-05T21:13:43.171 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-05T21:13:43.336 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-05T21:13:43.380 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-05T21:13:43.576 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-05T21:13:43.601 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-05T21:13:43.855 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-05T21:13:43.989 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-05T21:13:44.129 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-05T21:13:44.239 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-05T21:13:44.452 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-05T21:13:44.535 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-05T21:13:44.720 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-05T21:13:44.818 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-05T21:13:44.937 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-05T21:13:45.039 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-05T21:13:45.290 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-05T21:13:45.340 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-05T21:13:45.534 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-05T21:13:45.547 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-05T21:13:45.689 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-05T21:13:45.899 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-05T21:13:45.900 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-05T21:13:46.151 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-05T21:13:46.153 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-05T21:13:46.308 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-05T21:13:46.382 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-05T21:13:46.536 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-05T21:13:46.624 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-05T21:13:46.792 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-05T21:13:46.959 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-05T21:13:47.035 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-05T21:13:47.257 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-05T21:13:47.266 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-05T21:13:47.578 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-05T21:13:47.616 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el8.noarch 27/32 2024-04-05T21:13:47.819 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:18.2.1-0.el8.x86_64 28/32 2024-04-05T21:13:47.895 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-05T21:13:48.315 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-0.el8.noarch 27/32 2024-04-05T21:13:48.553 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:18.2.1-0.el8.x86_64 28/32 2024-04-05T21:13:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el8.x86_64 28/32 2024-04-05T21:13:49.610 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:18.2.1-0.el8.x86_64 29/32 2024-04-05T21:13:50.070 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el8.x86_64 29/32 2024-04-05T21:13:50.128 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:18.2.1-0.el8.x86_64 28/32 2024-04-05T21:13:50.293 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:18.2.1-0.el8.x86_64 30/32 2024-04-05T21:13:50.294 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:18.2.1-0.el8.x86_64 29/32 2024-04-05T21:13:50.524 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el8.x86_64 30/32 2024-04-05T21:13:50.735 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:18.2.1-0.el8.x86_64 31/32 2024-04-05T21:13:50.753 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:18.2.1-0.el8.x86_64 29/32 2024-04-05T21:13:50.753 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el8.x86_64 31/32 2024-04-05T21:13:50.918 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:18.2.1-0.el8.x86_64 32/32 2024-04-05T21:13:50.964 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:18.2.1-0.el8.x86_64 30/32 2024-04-05T21:13:51.178 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:18.2.1-0.el8.x86_64 30/32 2024-04-05T21:13:51.308 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:18.2.1-0.el8.x86_64 32/32 2024-04-05T21:13:51.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:18.2.1-0.el8.x86_64 1/32 2024-04-05T21:13:51.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:18.2.1-0.el8.x86_64 2/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:18.2.1-0.el8.x86_64 3/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:18.2.1-0.el8.x86_64 4/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:18.2.1-0.el8.x86_64 5/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el8.noarch 6/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-05T21:13:51.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-05T21:13:51.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-05T21:13:51.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-05T21:13:51.356 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:18.2.1-0.el8.x86_64 31/32 2024-04-05T21:13:51.373 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:18.2.1-0.el8.x86_64 31/32 2024-04-05T21:13:51.547 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:18.2.1-0.el8.x86_64 32/32 2024-04-05T21:13:51.953 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-2:18.2.1-0.el8.x86_64 32/32 2024-04-05T21:13:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:18.2.1-0.el8.x86_64 1/32 2024-04-05T21:13:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:18.2.1-0.el8.x86_64 2/32 2024-04-05T21:13:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:18.2.1-0.el8.x86_64 3/32 2024-04-05T21:13:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:18.2.1-0.el8.x86_64 4/32 2024-04-05T21:13:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:18.2.1-0.el8.x86_64 5/32 2024-04-05T21:13:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-0.el8.noarch 6/32 2024-04-05T21:13:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-05T21:13:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-05T21:13:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-05T21:13:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-05T21:13:51.954 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-05T21:13:51.955 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-05T21:13:51.956 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-05T21:13:51.956 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-05T21:13:51.956 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-05T21:13:51.956 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-05T21:13:51.956 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-05T21:13:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-05T21:13:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:52.176 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:13:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:52.176 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:18.2.1-0.el8.noarch 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-05T21:13:52.177 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-05T21:13:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-05T21:13:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-05T21:13:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-2.6-1.el8.noarch 2024-04-05T21:13:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-05T21:13:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-05T21:13:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-05T21:13:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-05T21:13:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-05T21:13:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-05T21:13:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-05T21:13:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-05T21:13:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-05T21:13:52.178 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-05T21:13:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-05T21:13:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-05T21:13:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:52.179 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:13:52.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2024-04-05T21:13:52.946 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-05T21:13:52.946 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:52.946 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:13:52.946 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:52.946 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:52.946 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:52.946 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:18.2.1-0.el8.noarch 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:18.2.1-0.el8.x86_64 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-05T21:13:52.947 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-2.6-1.el8.noarch 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:52.948 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:13:53.236 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:11 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:13:53.244 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-base 2024-04-05T21:13:53.998 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:11 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:13:54.824 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:13:54.872 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:13:54.873 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-05T21:13:54.873 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:13:54.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install cephadm 2024-04-05T21:13:55.592 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-base-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:13:55.640 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:13:55.642 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-05T21:13:55.642 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:13:55.741 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:13 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:13:56.027 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install cephadm 2024-04-05T21:13:56.777 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:14 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:13:57.366 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:13:57.367 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:13:57.367 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-05T21:13:57.367 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:13:57.367 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:13:57.367 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:18.2.1-0.el8 ceph-noarch 171 k 2024-04-05T21:13:57.367 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:57.367 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:13:57.367 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:13:57.367 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-05T21:13:57.367 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:57.367 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 171 k 2024-04-05T21:13:57.368 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 680 k 2024-04-05T21:13:57.368 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:13:57.455 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm-18.2.1-0.el8.noarch.rpm 1.9 MB/s | 171 kB 00:00 2024-04-05T21:13:57.455 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:13:57.455 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.9 MB/s | 171 kB 00:00 2024-04-05T21:13:57.456 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:13:57.463 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:13:57.463 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:13:57.465 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:13:57.468 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:13:57.514 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:13:58.051 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:18.2.1-0.el8.noarch 1/1 2024-04-05T21:13:58.255 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:18.2.1-0.el8.noarch 1/1 2024-04-05T21:13:58.403 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:13:58.403 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:13:58.403 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-05T21:13:58.403 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:13:58.403 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:13:58.403 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:18.2.1-0.el8 ceph-noarch 171 k 2024-04-05T21:13:58.403 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:58.404 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:13:58.404 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:13:58.404 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-04-05T21:13:58.404 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:58.404 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 171 k 2024-04-05T21:13:58.418 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 680 k 2024-04-05T21:13:58.418 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:13:58.460 INFO:teuthology.orchestra.run.smithi188.stdout:cephadm-18.2.1-0.el8.noarch.rpm 4.0 MB/s | 171 kB 00:00 2024-04-05T21:13:58.461 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:13:58.461 INFO:teuthology.orchestra.run.smithi188.stdout:Total 3.9 MB/s | 171 kB 00:00 2024-04-05T21:13:58.461 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:13:58.469 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:13:58.469 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:13:58.471 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:13:58.474 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:13:58.477 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:18.2.1-0.el8.noarch 1/1 2024-04-05T21:13:58.519 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:13:58.843 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:18.2.1-0.el8.noarch 1/1 2024-04-05T21:13:58.844 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:58.844 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:13:58.844 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:18.2.1-0.el8.noarch 2024-04-05T21:13:58.844 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:13:58.844 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:13:59.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache 2024-04-05T21:13:59.113 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:18.2.1-0.el8.noarch 1/1 2024-04-05T21:13:59.263 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:18.2.1-0.el8.noarch 1/1 2024-04-05T21:13:59.489 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:18.2.1-0.el8.noarch 1/1 2024-04-05T21:13:59.852 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:17 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:13:59.875 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:18.2.1-0.el8.noarch 1/1 2024-04-05T21:13:59.875 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:59.875 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:13:59.875 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:18.2.1-0.el8.noarch 2024-04-05T21:13:59.875 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:13:59.875 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:14:00.073 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-immutable-object-cache 2024-04-05T21:14:00.842 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:18 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:14:01.508 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:01.508 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-04-05T21:14:01.508 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:01.508 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:14:01.508 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el8 ceph 165 k 2024-04-05T21:14:01.508 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:01.508 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:14:01.508 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:01.508 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-05T21:14:01.509 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:01.509 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 165 k 2024-04-05T21:14:01.509 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 503 k 2024-04-05T21:14:01.509 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:14:01.577 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-18.2.1-0.el8.x86_64 2.4 MB/s | 165 kB 00:00 2024-04-05T21:14:01.577 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:01.577 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.4 MB/s | 165 kB 00:00 2024-04-05T21:14:01.578 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:14:01.589 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:14:01.589 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:14:01.637 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:14:01.640 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:14:01.754 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:14:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:02.382 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:02.443 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:14:02.444 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-04-05T21:14:02.445 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:02.445 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:14:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-0.el8 ceph 165 k 2024-04-05T21:14:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:02.445 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:14:02.445 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:02.445 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-04-05T21:14:02.445 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:02.445 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 165 k 2024-04-05T21:14:02.445 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 503 k 2024-04-05T21:14:02.445 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:14:02.492 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-immutable-object-cache-18.2.1-0.el8.x86_64 3.5 MB/s | 165 kB 00:00 2024-04-05T21:14:02.492 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:02.493 INFO:teuthology.orchestra.run.smithi188.stdout:Total 3.4 MB/s | 165 kB 00:00 2024-04-05T21:14:02.493 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:14:02.504 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:14:02.504 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:14:02.550 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:14:02.553 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:14:02.701 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:14:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:02.752 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:14:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:18.2.1-0.el8.x86_64 2024-04-05T21:14:02.752 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:02.752 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:14:02.914 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:03.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2024-04-05T21:14:03.308 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:03.682 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:03.683 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:03.683 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:14:03.683 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:18.2.1-0.el8.x86_64 2024-04-05T21:14:03.683 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:03.683 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:14:03.801 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:21 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:03.933 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr 2024-04-05T21:14:04.701 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:22 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:05.378 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:14:05.426 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:14:05.427 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-05T21:14:05.427 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:14:05.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-dashboard 2024-04-05T21:14:06.284 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-mgr-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:14:06.302 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:24 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:06.332 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:14:06.333 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-05T21:14:06.334 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:14:06.468 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-dashboard 2024-04-05T21:14:07.216 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:25 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:07.898 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:14:07.899 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-05T21:14:07.899 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:07.899 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:14:07.899 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el8 ceph-noarch 1.8 M 2024-04-05T21:14:07.899 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-05T21:14:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el8 ceph-noarch 23 k 2024-04-05T21:14:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el8 ceph-noarch 14 k 2024-04-05T21:14:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-05T21:14:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-05T21:14:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-05T21:14:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:07.900 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:14:07.900 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:07.900 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-04-05T21:14:07.900 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:07.901 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 2.1 M 2024-04-05T21:14:07.901 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 8.5 M 2024-04-05T21:14:07.901 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:14:08.205 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): ceph-grafana-dashboards-18.2.1-0.el8.noa 314 kB/s | 23 kB 00:00 2024-04-05T21:14:08.247 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): ceph-prometheus-alerts-18.2.1-0.el8.noar 126 kB/s | 14 kB 00:00 2024-04-05T21:14:08.272 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 635 kB/s | 43 kB 00:00 2024-04-05T21:14:08.306 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): ceph-mgr-dashboard-18.2.1-0.el8.noarch.r 10 MB/s | 1.8 MB 00:00 2024-04-05T21:14:08.427 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 187 kB/s | 34 kB 00:00 2024-04-05T21:14:08.506 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 841 kB/s | 196 kB 00:00 2024-04-05T21:14:08.508 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:08.508 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.4 MB/s | 2.1 MB 00:00 2024-04-05T21:14:08.518 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:14:08.526 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:14:08.527 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:14:08.601 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:14:08.604 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:14:08.843 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:14:08.844 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:08.844 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-05T21:14:08.844 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:08.844 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:14:08.844 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:18.2.1-0.el8 ceph-noarch 1.8 M 2024-04-05T21:14:08.844 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-05T21:14:08.845 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:18.2.1-0.el8 ceph-noarch 23 k 2024-04-05T21:14:08.845 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:18.2.1-0.el8 ceph-noarch 14 k 2024-04-05T21:14:08.845 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-05T21:14:08.845 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-05T21:14:08.845 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-05T21:14:08.845 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:08.845 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:14:08.845 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:08.845 INFO:teuthology.orchestra.run.smithi188.stdout:Install 6 Packages 2024-04-05T21:14:08.845 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:08.846 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 2.1 M 2024-04-05T21:14:08.846 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 8.5 M 2024-04-05T21:14:08.846 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:14:08.885 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:14:09.086 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-04-05T21:14:09.121 INFO:teuthology.orchestra.run.smithi188.stdout:(1/6): ceph-prometheus-alerts-18.2.1-0.el8.noar 247 kB/s | 14 kB 00:00 2024-04-05T21:14:09.146 INFO:teuthology.orchestra.run.smithi188.stdout:(2/6): ceph-grafana-dashboards-18.2.1-0.el8.noa 274 kB/s | 23 kB 00:00 2024-04-05T21:14:09.188 INFO:teuthology.orchestra.run.smithi188.stdout:(3/6): ceph-mgr-dashboard-18.2.1-0.el8.noarch.r 14 MB/s | 1.8 MB 00:00 2024-04-05T21:14:09.238 INFO:teuthology.orchestra.run.smithi188.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 364 kB/s | 43 kB 00:00 2024-04-05T21:14:09.365 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-04-05T21:14:09.430 INFO:teuthology.orchestra.run.smithi188.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 118 kB/s | 34 kB 00:00 2024-04-05T21:14:09.600 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-04-05T21:14:09.630 INFO:teuthology.orchestra.run.smithi188.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 445 kB/s | 196 kB 00:00 2024-04-05T21:14:09.633 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:09.633 INFO:teuthology.orchestra.run.smithi188.stdout:Total 2.6 MB/s | 2.1 MB 00:00 2024-04-05T21:14:09.643 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:14:09.651 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:14:09.651 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:14:09.695 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el8.noarch 4/6 2024-04-05T21:14:09.730 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:14:09.733 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:14:10.026 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:14:10.040 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el8.noarch 5/6 2024-04-05T21:14:10.240 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-04-05T21:14:10.289 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el8.noarch 6/6 2024-04-05T21:14:10.502 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-04-05T21:14:10.569 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el8.noarch 6/6 2024-04-05T21:14:10.569 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el8.noarch 1/6 2024-04-05T21:14:10.569 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el8.noarch 2/6 2024-04-05T21:14:10.569 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el8.noarch 3/6 2024-04-05T21:14:10.569 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-04-05T21:14:10.569 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-04-05T21:14:10.720 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-04-05T21:14:10.840 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-0.el8.noarch 4/6 2024-04-05T21:14:10.965 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-04-05T21:14:10.965 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:10.965 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:14:10.965 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:18.2.1-0.el8.noarch 2024-04-05T21:14:10.965 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:18.2.1-0.el8.noarch 2024-04-05T21:14:10.965 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:18.2.1-0.el8.noarch 2024-04-05T21:14:10.965 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-05T21:14:10.965 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-05T21:14:10.965 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-05T21:14:10.965 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:10.966 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:14:11.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-05T21:14:11.214 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-0.el8.noarch 5/6 2024-04-05T21:14:11.464 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-0.el8.noarch 6/6 2024-04-05T21:14:11.753 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-0.el8.noarch 6/6 2024-04-05T21:14:11.753 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-0.el8.noarch 1/6 2024-04-05T21:14:11.753 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-0.el8.noarch 2/6 2024-04-05T21:14:11.753 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-0.el8.noarch 3/6 2024-04-05T21:14:11.754 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-04-05T21:14:11.754 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-04-05T21:14:11.846 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:29 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:12.165 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-04-05T21:14:12.165 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:12.165 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:14:12.165 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:18.2.1-0.el8.noarch 2024-04-05T21:14:12.165 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:18.2.1-0.el8.noarch 2024-04-05T21:14:12.165 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:18.2.1-0.el8.noarch 2024-04-05T21:14:12.165 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-05T21:14:12.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-05T21:14:12.166 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-05T21:14:12.166 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:12.166 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:14:12.299 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-05T21:14:13.044 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:31 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:13.454 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:14:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-04-05T21:14:13.455 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-05T21:14:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-04-05T21:14:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:14:13.455 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el8 ceph-noarch 7.4 M 2024-04-05T21:14:13.455 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-05T21:14:13.456 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-05T21:14:13.456 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-05T21:14:13.456 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-05T21:14:13.456 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-05T21:14:13.456 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-05T21:14:13.456 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-05T21:14:13.456 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-05T21:14:13.456 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:13.456 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:14:13.456 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-04-05T21:14:13.456 INFO:teuthology.orchestra.run.smithi121.stdout:Install 8 Packages 2024-04-05T21:14:13.456 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:13.457 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 36 M 2024-04-05T21:14:13.457 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 210 M 2024-04-05T21:14:13.457 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:14:14.074 INFO:teuthology.orchestra.run.smithi121.stdout:(1/8): ceph-mgr-diskprediction-local-18.2.1-0.e 20 MB/s | 7.4 MB 00:00 2024-04-05T21:14:14.230 INFO:teuthology.orchestra.run.smithi121.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.3 MB/s | 4.9 MB 00:00 2024-04-05T21:14:14.305 INFO:teuthology.orchestra.run.smithi121.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 7.9 MB/s | 4.7 MB 00:00 2024-04-05T21:14:14.330 INFO:teuthology.orchestra.run.smithi121.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2024-04-05T21:14:14.581 INFO:teuthology.orchestra.run.smithi121.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.5 MB/s | 645 kB 00:00 2024-04-05T21:14:14.664 INFO:teuthology.orchestra.run.smithi121.stdout:(6/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 6.3 MB/s | 3.7 MB 00:00 2024-04-05T21:14:14.673 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:14:14.674 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-04-05T21:14:14.674 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-05T21:14:14.674 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-0.el8 ceph-noarch 7.4 M 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout:Install 8 Packages 2024-04-05T21:14:14.675 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:14.676 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 36 M 2024-04-05T21:14:14.676 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 210 M 2024-04-05T21:14:14.676 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:14:14.690 INFO:teuthology.orchestra.run.smithi121.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.5 MB/s | 172 kB 00:00 2024-04-05T21:14:14.832 INFO:teuthology.orchestra.run.smithi121.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 27 MB/s | 14 MB 00:00 2024-04-05T21:14:14.832 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:14.832 INFO:teuthology.orchestra.run.smithi121.stdout:Total 26 MB/s | 36 MB 00:01 2024-04-05T21:14:14.937 INFO:teuthology.orchestra.run.smithi188.stdout:(1/8): ceph-mgr-diskprediction-local-18.2.1-0.e 47 MB/s | 7.4 MB 00:00 2024-04-05T21:14:15.061 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:14:15.079 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:14:15.079 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:14:15.237 INFO:teuthology.orchestra.run.smithi188.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 10 MB/s | 4.7 MB 00:00 2024-04-05T21:14:15.304 INFO:teuthology.orchestra.run.smithi188.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.3 MB/s | 4.9 MB 00:00 2024-04-05T21:14:15.338 INFO:teuthology.orchestra.run.smithi188.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2024-04-05T21:14:15.478 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:14:15.481 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:14:15.488 INFO:teuthology.orchestra.run.smithi188.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 4.2 MB/s | 645 kB 00:00 2024-04-05T21:14:15.530 INFO:teuthology.orchestra.run.smithi188.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 4.0 MB/s | 172 kB 00:00 2024-04-05T21:14:15.789 INFO:teuthology.orchestra.run.smithi188.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 29 MB/s | 14 MB 00:00 2024-04-05T21:14:16.060 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:14:16.064 INFO:teuthology.orchestra.run.smithi188.stdout:(8/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.3 MB/s | 3.7 MB 00:01 2024-04-05T21:14:16.065 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:16.065 INFO:teuthology.orchestra.run.smithi188.stdout:Total 26 MB/s | 36 MB 00:01 2024-04-05T21:14:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-05T21:14:16.295 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:14:16.313 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:14:16.313 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:14:16.332 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-05T21:14:16.517 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-05T21:14:16.697 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:14:16.700 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:14:17.098 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-05T21:14:17.287 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:14:17.304 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-05T21:14:17.466 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-05T21:14:17.598 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-04-05T21:14:17.749 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-05T21:14:17.949 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-05T21:14:18.110 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-05T21:14:18.345 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-04-05T21:14:18.574 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-04-05T21:14:18.804 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-05T21:14:19.166 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-05T21:14:19.190 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-04-05T21:14:19.357 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-05T21:14:19.404 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-05T21:14:20.062 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-04-05T21:14:20.411 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-04-05T21:14:20.665 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-05T21:14:21.169 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-05T21:14:22.434 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-05T21:14:22.442 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-04-05T21:14:22.994 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el8.noarch 8/8 2024-04-05T21:14:23.327 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el8.noarch 8/8 2024-04-05T21:14:23.328 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el8.noarch 1/8 2024-04-05T21:14:23.328 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-05T21:14:23.328 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-05T21:14:23.328 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-05T21:14:23.328 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-05T21:14:23.328 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-05T21:14:23.328 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-05T21:14:23.681 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-04-05T21:14:23.748 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-05T21:14:23.748 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:23.748 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:14:23.748 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el8.noarch 2024-04-05T21:14:23.748 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-05T21:14:23.748 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-05T21:14:23.748 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-05T21:14:23.749 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-05T21:14:23.749 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-05T21:14:23.749 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-05T21:14:23.749 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-05T21:14:23.749 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:23.749 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:14:23.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-rook 2024-04-05T21:14:24.539 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-0.el8.noarch 8/8 2024-04-05T21:14:24.626 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:42 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:24.875 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-0.el8.noarch 8/8 2024-04-05T21:14:24.875 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-0.el8.noarch 1/8 2024-04-05T21:14:24.875 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-04-05T21:14:24.875 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-04-05T21:14:24.875 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-04-05T21:14:24.875 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-04-05T21:14:24.875 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-04-05T21:14:24.875 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-04-05T21:14:25.288 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-04-05T21:14:25.288 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:25.288 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:14:25.288 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:18.2.1-0.el8.noarch 2024-04-05T21:14:25.288 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-05T21:14:25.288 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-05T21:14:25.288 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-05T21:14:25.288 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-05T21:14:25.289 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-05T21:14:25.289 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-05T21:14:25.289 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-05T21:14:25.289 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:25.289 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:14:25.421 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-rook 2024-04-05T21:14:26.172 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:44 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:26.257 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:14:26.259 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:26.259 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-05T21:14:26.259 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:26.259 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:14:26.259 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el8 ceph-noarch 50 k 2024-04-05T21:14:26.259 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-05T21:14:26.259 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-05T21:14:26.259 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2024-04-05T21:14:26.260 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:26.261 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 1.9 M 2024-04-05T21:14:26.261 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 25 M 2024-04-05T21:14:26.261 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:14:26.899 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el8.noarch.rpm 642 kB/s | 50 kB 00:00 2024-04-05T21:14:27.024 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 131 kB/s | 27 kB 00:00 2024-04-05T21:14:27.049 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 87 kB/s | 20 kB 00:00 2024-04-05T21:14:27.091 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 658 kB/s | 126 kB 00:00 2024-04-05T21:14:27.125 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 2.0 MB/s | 155 kB 00:00 2024-04-05T21:14:27.150 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 872 kB/s | 110 kB 00:00 2024-04-05T21:14:27.175 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 507 kB/s | 43 kB 00:00 2024-04-05T21:14:27.276 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 219 kB/s | 33 kB 00:00 2024-04-05T21:14:27.293 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 115 kB/s | 16 kB 00:00 2024-04-05T21:14:27.310 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 651 kB/s | 81 kB 00:00 2024-04-05T21:14:27.352 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 1.1 MB/s | 64 kB 00:00 2024-04-05T21:14:27.368 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2024-04-05T21:14:27.585 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.8 MB/s | 1.2 MB 00:00 2024-04-05T21:14:27.588 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:27.588 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.5 MB/s | 1.9 MB 00:01 2024-04-05T21:14:27.646 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:14:27.662 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:14:27.662 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:14:27.750 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:14:27.753 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:14:27.817 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:14:27.819 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:27.819 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-05T21:14:27.819 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:27.819 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:14:27.819 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:18.2.1-0.el8 ceph-noarch 50 k 2024-04-05T21:14:27.819 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-05T21:14:27.819 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-05T21:14:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-05T21:14:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-05T21:14:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-05T21:14:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-05T21:14:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-05T21:14:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-05T21:14:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-05T21:14:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-05T21:14:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-05T21:14:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-05T21:14:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-05T21:14:27.820 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:27.821 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:14:27.821 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:27.821 INFO:teuthology.orchestra.run.smithi188.stdout:Install 13 Packages 2024-04-05T21:14:27.821 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:27.821 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 1.9 M 2024-04-05T21:14:27.821 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 25 M 2024-04-05T21:14:27.821 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:14:28.223 INFO:teuthology.orchestra.run.smithi188.stdout:(1/13): ceph-mgr-rook-18.2.1-0.el8.noarch.rpm 1.3 MB/s | 50 kB 00:00 2024-04-05T21:14:28.308 INFO:teuthology.orchestra.run.smithi188.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 163 kB/s | 20 kB 00:00 2024-04-05T21:14:28.315 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:14:28.324 INFO:teuthology.orchestra.run.smithi188.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 190 kB/s | 27 kB 00:00 2024-04-05T21:14:28.435 INFO:teuthology.orchestra.run.smithi188.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.4 MB/s | 155 kB 00:00 2024-04-05T21:14:28.466 INFO:teuthology.orchestra.run.smithi188.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 521 kB/s | 126 kB 00:00 2024-04-05T21:14:28.491 INFO:teuthology.orchestra.run.smithi188.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 761 kB/s | 43 kB 00:00 2024-04-05T21:14:28.517 INFO:teuthology.orchestra.run.smithi188.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 524 kB/s | 110 kB 00:00 2024-04-05T21:14:28.742 INFO:teuthology.orchestra.run.smithi188.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 65 kB/s | 16 kB 00:00 2024-04-05T21:14:28.767 INFO:teuthology.orchestra.run.smithi188.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 110 kB/s | 33 kB 00:00 2024-04-05T21:14:28.779 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-05T21:14:28.926 INFO:teuthology.orchestra.run.smithi188.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 199 kB/s | 81 kB 00:00 2024-04-05T21:14:28.967 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-04-05T21:14:28.985 INFO:teuthology.orchestra.run.smithi188.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 296 kB/s | 64 kB 00:00 2024-04-05T21:14:29.135 INFO:teuthology.orchestra.run.smithi188.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 293 kB/s | 61 kB 00:00 2024-04-05T21:14:29.177 INFO:teuthology.orchestra.run.smithi188.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.7 MB/s | 1.2 MB 00:00 2024-04-05T21:14:29.177 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:29.177 INFO:teuthology.orchestra.run.smithi188.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2024-04-05T21:14:29.211 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-04-05T21:14:29.236 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:14:29.252 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:14:29.252 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:14:29.340 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:14:29.343 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:14:29.467 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-04-05T21:14:29.657 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-04-05T21:14:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-04-05T21:14:30.033 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:14:30.165 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-04-05T21:14:30.274 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-04-05T21:14:30.389 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-04-05T21:14:30.487 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-04-05T21:14:30.732 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-04-05T21:14:30.986 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-04-05T21:14:31.202 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-04-05T21:14:31.422 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-04-05T21:14:31.425 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-04-05T21:14:31.699 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-04-05T21:14:31.709 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-04-05T21:14:31.888 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-04-05T21:14:31.925 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-04-05T21:14:32.111 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-04-05T21:14:32.284 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el8.noarch 13/13 2024-04-05T21:14:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el8.noarch 13/13 2024-04-05T21:14:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el8.noarch 1/13 2024-04-05T21:14:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-04-05T21:14:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-04-05T21:14:32.579 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-04-05T21:14:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-04-05T21:14:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-04-05T21:14:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-04-05T21:14:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-04-05T21:14:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-05T21:14:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-04-05T21:14:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-04-05T21:14:32.580 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-04-05T21:14:32.961 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-04-05T21:14:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-04-05T21:14:33.071 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:33.071 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:14:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:18.2.1-0.el8.noarch 2024-04-05T21:14:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-05T21:14:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-05T21:14:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-05T21:14:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-05T21:14:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-05T21:14:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-05T21:14:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-05T21:14:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-05T21:14:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-05T21:14:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-05T21:14:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-05T21:14:33.072 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-05T21:14:33.073 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:33.073 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:14:33.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-cephadm 2024-04-05T21:14:33.252 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-04-05T21:14:33.478 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-04-05T21:14:33.716 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-04-05T21:14:33.929 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:18.2.1-0.el8.noarch 13/13 2024-04-05T21:14:33.940 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:51 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:34.238 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-0.el8.noarch 13/13 2024-04-05T21:14:34.238 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:18.2.1-0.el8.noarch 1/13 2024-04-05T21:14:34.238 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-04-05T21:14:34.238 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-04-05T21:14:34.238 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-04-05T21:14:34.238 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-04-05T21:14:34.238 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-04-05T21:14:34.239 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-04-05T21:14:34.239 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-04-05T21:14:34.239 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-04-05T21:14:34.239 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-04-05T21:14:34.239 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-04-05T21:14:34.239 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-04-05T21:14:34.710 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-04-05T21:14:34.710 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:34.710 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:14:34.710 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:18.2.1-0.el8.noarch 2024-04-05T21:14:34.710 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-05T21:14:34.710 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-05T21:14:34.710 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-05T21:14:34.710 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-05T21:14:34.710 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-05T21:14:34.710 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-05T21:14:34.710 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-05T21:14:34.711 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-05T21:14:34.711 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-05T21:14:34.711 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-05T21:14:34.711 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-05T21:14:34.711 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-05T21:14:34.711 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:34.711 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:14:34.851 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-cephadm 2024-04-05T21:14:35.565 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:14:35.566 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:35.566 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-05T21:14:35.567 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:35.567 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:14:35.567 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el8 ceph-noarch 132 k 2024-04-05T21:14:35.567 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-05T21:14:35.567 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-05T21:14:35.567 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-05T21:14:35.567 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-05T21:14:35.567 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-05T21:14:35.567 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:35.567 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:14:35.567 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:35.567 INFO:teuthology.orchestra.run.smithi121.stdout:Install 5 Packages 2024-04-05T21:14:35.567 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:35.568 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 6.0 M 2024-04-05T21:14:35.568 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2024-04-05T21:14:35.568 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:14:35.606 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:53 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:35.710 INFO:teuthology.orchestra.run.smithi121.stdout:(1/5): ceph-mgr-cephadm-18.2.1-0.el8.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-04-05T21:14:36.052 INFO:teuthology.orchestra.run.smithi121.stdout:(2/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.4 MB/s | 578 kB 00:00 2024-04-05T21:14:36.202 INFO:teuthology.orchestra.run.smithi121.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 967 kB/s | 476 kB 00:00 2024-04-05T21:14:36.286 INFO:teuthology.orchestra.run.smithi121.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 258 kB/s | 60 kB 00:00 2024-04-05T21:14:36.770 INFO:teuthology.orchestra.run.smithi121.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 4.3 MB/s | 4.8 MB 00:01 2024-04-05T21:14:36.772 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:36.772 INFO:teuthology.orchestra.run.smithi121.stdout:Total 5.0 MB/s | 6.0 MB 00:01 2024-04-05T21:14:36.822 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:14:36.833 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:14:36.833 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:14:36.936 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:14:36.939 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:14:37.202 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:14:37.238 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:14:37.239 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:37.239 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-05T21:14:37.239 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:37.239 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:14:37.239 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:18.2.1-0.el8 ceph-noarch 132 k 2024-04-05T21:14:37.239 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-05T21:14:37.239 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-05T21:14:37.239 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-05T21:14:37.239 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-05T21:14:37.239 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-05T21:14:37.239 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:37.239 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:14:37.239 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:37.240 INFO:teuthology.orchestra.run.smithi188.stdout:Install 5 Packages 2024-04-05T21:14:37.240 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:37.240 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 6.0 M 2024-04-05T21:14:37.240 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 26 M 2024-04-05T21:14:37.240 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:14:37.519 INFO:teuthology.orchestra.run.smithi188.stdout:(1/5): ceph-mgr-cephadm-18.2.1-0.el8.noarch.rpm 3.5 MB/s | 132 kB 00:00 2024-04-05T21:14:37.538 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-04-05T21:14:37.711 INFO:teuthology.orchestra.run.smithi188.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.4 MB/s | 476 kB 00:00 2024-04-05T21:14:37.753 INFO:teuthology.orchestra.run.smithi188.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 1.4 MB/s | 60 kB 00:00 2024-04-05T21:14:37.778 INFO:teuthology.orchestra.run.smithi188.stdout:(4/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.9 MB/s | 578 kB 00:00 2024-04-05T21:14:38.311 INFO:teuthology.orchestra.run.smithi188.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 5.8 MB/s | 4.8 MB 00:00 2024-04-05T21:14:38.312 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:38.312 INFO:teuthology.orchestra.run.smithi188.stdout:Total 5.6 MB/s | 6.0 MB 00:01 2024-04-05T21:14:38.364 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:14:38.375 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:14:38.375 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:14:38.480 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:14:38.483 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:14:38.509 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-05T21:14:38.783 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:14:38.860 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-04-05T21:14:39.099 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-04-05T21:14:39.173 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-04-05T21:14:39.358 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el8.noarch 5/5 2024-04-05T21:14:39.638 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el8.noarch 5/5 2024-04-05T21:14:39.639 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el8.noarch 1/5 2024-04-05T21:14:39.639 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-04-05T21:14:39.639 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-04-05T21:14:39.639 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-05T21:14:40.018 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-04-05T21:14:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:40.019 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:14:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:18.2.1-0.el8.noarch python3-asyncssh-2.7.0-2.el8.noarch 2024-04-05T21:14:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.5.1-7.el8.noarch python3-jinja2-2.10.1-4.el8.noarch 2024-04-05T21:14:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-05T21:14:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:40.019 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:14:40.075 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-04-05T21:14:40.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2024-04-05T21:14:40.475 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-04-05T21:14:40.768 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-04-05T21:14:40.905 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:58 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:40.986 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-0.el8.noarch 5/5 2024-04-05T21:14:41.296 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-0.el8.noarch 5/5 2024-04-05T21:14:41.296 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-0.el8.noarch 1/5 2024-04-05T21:14:41.296 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-04-05T21:14:41.296 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-04-05T21:14:41.296 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-04-05T21:14:41.722 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-04-05T21:14:41.722 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:41.722 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:14:41.722 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:18.2.1-0.el8.noarch python3-asyncssh-2.7.0-2.el8.noarch 2024-04-05T21:14:41.723 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.5.1-7.el8.noarch python3-jinja2-2.10.1-4.el8.noarch 2024-04-05T21:14:41.723 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-05T21:14:41.723 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:41.723 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:14:41.851 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-fuse 2024-04-05T21:14:42.479 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:14:42.480 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:42.480 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-05T21:14:42.480 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:42.480 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:14:42.480 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:18.2.1-0.el8 ceph 851 k 2024-04-05T21:14:42.480 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-05T21:14:42.480 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-05T21:14:42.480 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:42.480 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:14:42.480 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:42.480 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-04-05T21:14:42.480 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:42.481 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 934 k 2024-04-05T21:14:42.481 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 3.0 M 2024-04-05T21:14:42.481 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:14:42.599 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:00 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:42.619 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): ceph-fuse-18.2.1-0.el8.x86_64.rpm 9.4 MB/s | 851 kB 00:00 2024-04-05T21:14:42.644 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 734 kB/s | 83 kB 00:00 2024-04-05T21:14:42.645 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:42.645 INFO:teuthology.orchestra.run.smithi121.stdout:Total 5.6 MB/s | 934 kB 00:00 2024-04-05T21:14:42.655 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:14:42.676 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:14:42.677 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:14:42.761 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:14:42.764 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:14:42.942 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:14:43.240 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-05T21:14:43.465 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:18.2.1-0.el8.x86_64 2/2 2024-04-05T21:14:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el8.x86_64 2/2 2024-04-05T21:14:43.863 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:18.2.1-0.el8.x86_64 1/2 2024-04-05T21:14:44.227 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:14:44.228 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:44.228 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-05T21:14:44.228 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:44.228 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:14:44.228 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:18.2.1-0.el8 ceph 851 k 2024-04-05T21:14:44.228 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-05T21:14:44.229 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-05T21:14:44.229 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:44.229 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:14:44.229 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:44.229 INFO:teuthology.orchestra.run.smithi188.stdout:Install 2 Packages 2024-04-05T21:14:44.229 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:44.229 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 934 k 2024-04-05T21:14:44.229 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 3.0 M 2024-04-05T21:14:44.229 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:14:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-05T21:14:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:44.331 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:14:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:18.2.1-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-05T21:14:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:44.331 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:14:44.479 INFO:teuthology.orchestra.run.smithi188.stdout:(1/2): ceph-fuse-18.2.1-0.el8.x86_64.rpm 16 MB/s | 851 kB 00:00 2024-04-05T21:14:44.530 INFO:teuthology.orchestra.run.smithi188.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 815 kB/s | 83 kB 00:00 2024-04-05T21:14:44.530 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:44.530 INFO:teuthology.orchestra.run.smithi188.stdout:Total 3.0 MB/s | 934 kB 00:00 2024-04-05T21:14:44.540 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:14:44.562 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:14:44.562 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:14:44.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2024-04-05T21:14:44.650 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:14:44.653 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:14:44.833 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:14:45.106 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-05T21:14:45.342 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:03 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:45.347 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:18.2.1-0.el8.x86_64 2/2 2024-04-05T21:14:45.747 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-fuse-2:18.2.1-0.el8.x86_64 2/2 2024-04-05T21:14:45.747 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:18.2.1-0.el8.x86_64 1/2 2024-04-05T21:14:46.137 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-05T21:14:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:46.138 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:14:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:18.2.1-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-05T21:14:46.138 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:46.138 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:14:46.384 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados-devel 2024-04-05T21:14:46.958 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:14:46.959 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:46.959 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-05T21:14:46.959 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:46.959 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:14:46.959 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:18.2.1-0.el8 ceph 139 k 2024-04-05T21:14:46.959 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:46.959 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:14:46.959 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:46.959 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-05T21:14:46.960 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:46.960 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 139 k 2024-04-05T21:14:46.960 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 513 k 2024-04-05T21:14:46.960 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:14:47.040 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-18.2.1-0.el8.x86_64.rpm 1.7 MB/s | 139 kB 00:00 2024-04-05T21:14:47.041 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:47.041 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.7 MB/s | 139 kB 00:00 2024-04-05T21:14:47.041 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:14:47.046 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:14:47.046 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:14:47.095 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:14:47.099 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:14:47.146 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:05 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:47.213 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:14:47.488 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:47.784 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:48.176 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:48.176 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:48.176 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:14:48.176 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:18.2.1-0.el8.x86_64 2024-04-05T21:14:48.177 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:48.177 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:14:48.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2024-04-05T21:14:48.763 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:14:48.764 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:48.764 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-05T21:14:48.764 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:48.764 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:14:48.764 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:18.2.1-0.el8 ceph 139 k 2024-04-05T21:14:48.764 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:48.764 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:14:48.764 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:48.764 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-04-05T21:14:48.765 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:48.765 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 139 k 2024-04-05T21:14:48.765 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 513 k 2024-04-05T21:14:48.765 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:14:48.818 INFO:teuthology.orchestra.run.smithi188.stdout:librados-devel-18.2.1-0.el8.x86_64.rpm 2.6 MB/s | 139 kB 00:00 2024-04-05T21:14:48.819 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:48.819 INFO:teuthology.orchestra.run.smithi188.stdout:Total 2.5 MB/s | 139 kB 00:00 2024-04-05T21:14:48.819 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:14:48.824 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:14:48.824 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:14:48.874 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:14:48.876 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:14:48.985 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:14:49.130 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:07 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:49.233 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:49.521 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados-devel-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:49.960 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:49.960 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:49.960 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:14:49.960 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:18.2.1-0.el8.x86_64 2024-04-05T21:14:49.960 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:49.960 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:14:50.227 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs2 2024-04-05T21:14:50.711 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:14:50.759 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:14:50.760 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-05T21:14:50.760 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:14:50.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2024-04-05T21:14:50.992 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:08 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:51.603 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:09 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:52.559 INFO:teuthology.orchestra.run.smithi188.stdout:Package libcephfs2-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:14:52.608 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:14:52.609 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-05T21:14:52.609 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:14:52.779 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs-devel 2024-04-05T21:14:53.242 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:14:53.242 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:53.242 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-05T21:14:53.242 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:53.242 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:14:53.242 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:18.2.1-0.el8 ceph 31 k 2024-04-05T21:14:53.243 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:53.243 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:14:53.243 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:14:53.243 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-05T21:14:53.243 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:53.243 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 31 k 2024-04-05T21:14:53.243 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 139 k 2024-04-05T21:14:53.243 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:14:53.312 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-18.2.1-0.el8.x86_64.rpm 450 kB/s | 31 kB 00:00 2024-04-05T21:14:53.312 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:53.313 INFO:teuthology.orchestra.run.smithi121.stdout:Total 442 kB/s | 31 kB 00:00 2024-04-05T21:14:53.313 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:14:53.316 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:14:53.316 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:14:53.342 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:14:53.345 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:14:53.404 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:14:53.530 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:11 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:53.587 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:53.865 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:54.215 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:54.215 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:54.215 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:14:54.215 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:18.2.1-0.el8.x86_64 2024-04-05T21:14:54.215 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:14:54.215 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:14:54.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2024-04-05T21:14:55.102 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:13 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:55.168 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:14:55.169 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:55.169 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-05T21:14:55.169 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:55.169 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:14:55.169 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:18.2.1-0.el8 ceph 31 k 2024-04-05T21:14:55.169 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:55.169 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:14:55.169 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:14:55.169 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-04-05T21:14:55.169 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:55.170 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 31 k 2024-04-05T21:14:55.170 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 139 k 2024-04-05T21:14:55.170 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:14:55.226 INFO:teuthology.orchestra.run.smithi188.stdout:libcephfs-devel-18.2.1-0.el8.x86_64.rpm 548 kB/s | 31 kB 00:00 2024-04-05T21:14:55.227 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:14:55.227 INFO:teuthology.orchestra.run.smithi188.stdout:Total 536 kB/s | 31 kB 00:00 2024-04-05T21:14:55.227 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:14:55.230 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:14:55.230 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:14:55.255 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:14:55.258 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:14:55.342 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:14:55.514 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:55.807 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs-devel-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:56.230 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:14:56.231 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:56.231 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:14:56.231 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:18.2.1-0.el8.x86_64 2024-04-05T21:14:56.231 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:14:56.231 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:14:56.364 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados2 2024-04-05T21:14:56.671 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:14:56.719 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:14:56.721 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-05T21:14:56.721 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:14:56.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2024-04-05T21:14:57.120 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:15 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:57.587 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:15 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:14:58.711 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:14:58.759 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:14:58.760 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-05T21:14:58.760 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:14:58.897 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librbd1 2024-04-05T21:14:59.169 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:14:59.235 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:14:59.236 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-05T21:14:59.236 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:14:59.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados 2024-04-05T21:14:59.652 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:17 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:00.089 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:18 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:01.218 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:15:01.268 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:15:01.269 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-05T21:15:01.269 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:15:01.373 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rados 2024-04-05T21:15:01.655 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:15:01.703 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:15:01.704 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-05T21:15:01.704 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:15:01.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw 2024-04-05T21:15:02.137 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:20 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:02.556 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:20 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:03.726 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rados-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:15:03.775 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:15:03.776 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-05T21:15:03.776 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:15:04.040 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rgw 2024-04-05T21:15:04.108 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:15:04.155 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:15:04.156 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-05T21:15:04.157 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:15:04.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs 2024-04-05T21:15:04.800 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:22 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:05.022 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:23 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:06.385 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rgw-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:15:06.433 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:15:06.434 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-05T21:15:06.435 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:15:06.544 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-cephfs 2024-04-05T21:15:06.579 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:15:06.627 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:15:06.628 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-05T21:15:06.628 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:15:06.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd 2024-04-05T21:15:07.298 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:25 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:07.525 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:25 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:08.867 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-cephfs-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:15:08.916 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:15:08.917 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-05T21:15:08.917 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:15:09.027 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rbd 2024-04-05T21:15:09.097 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:15:09.145 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:15:09.146 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-05T21:15:09.146 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:15:09.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2024-04-05T21:15:09.783 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:27 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:09.995 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:27 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:11.360 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rbd-2:18.2.1-0.el8.x86_64 is already installed. 2024-04-05T21:15:11.410 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:15:11.411 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-04-05T21:15:11.411 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:15:11.534 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-fuse 2024-04-05T21:15:11.591 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:15:11.591 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:15:11.592 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-05T21:15:11.592 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:15:11.592 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:15:11.592 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:18.2.1-0.el8 ceph 94 k 2024-04-05T21:15:11.592 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:15:11.592 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:15:11.592 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:15:11.592 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-05T21:15:11.592 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:15:11.593 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 94 k 2024-04-05T21:15:11.593 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 276 k 2024-04-05T21:15:11.593 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:15:11.667 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-18.2.1-0.el8.x86_64.rpm 1.2 MB/s | 94 kB 00:00 2024-04-05T21:15:11.668 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:15:11.668 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.2 MB/s | 94 kB 00:00 2024-04-05T21:15:11.668 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:15:11.679 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:15:11.679 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:15:11.728 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:15:11.731 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:15:12.124 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:15:12.290 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:30 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:12.333 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:12.658 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:13.069 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:13.069 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:15:13.069 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:15:13.069 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:18.2.1-0.el8.x86_64 2024-04-05T21:15:13.069 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:15:13.069 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:15:13.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2024-04-05T21:15:13.922 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:15:13.923 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:15:13.923 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-05T21:15:13.923 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:15:13.923 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:15:13.923 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:18.2.1-0.el8 ceph 94 k 2024-04-05T21:15:13.924 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:15:13.924 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:15:13.924 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:15:13.924 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-04-05T21:15:13.924 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:15:13.924 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 94 k 2024-04-05T21:15:13.924 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 276 k 2024-04-05T21:15:13.924 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:15:13.981 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-fuse-18.2.1-0.el8.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-05T21:15:13.981 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:15:13.981 INFO:teuthology.orchestra.run.smithi188.stdout:Total 1.6 MB/s | 94 kB 00:00 2024-04-05T21:15:13.982 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:15:13.992 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:15:13.992 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:15:14.041 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:15:14.044 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:15:14.062 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:32 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:14.157 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:15:14.686 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:14.985 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-fuse-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:15.354 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:15.354 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:15:15.354 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:15:15.354 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:18.2.1-0.el8.x86_64 2024-04-05T21:15:15.355 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:15:15.355 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:15:15.575 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-mirror 2024-04-05T21:15:15.693 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:15:15.694 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:15:15.694 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-05T21:15:15.694 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:15:15.694 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:15:15.694 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:18.2.1-0.el8 ceph 3.4 M 2024-04-05T21:15:15.694 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:15:15.694 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:15:15.694 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:15:15.694 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-05T21:15:15.695 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:15:15.695 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.4 M 2024-04-05T21:15:15.695 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 16 M 2024-04-05T21:15:15.695 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:15:15.830 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-18.2.1-0.el8.x86_64.rpm 26 MB/s | 3.4 MB 00:00 2024-04-05T21:15:15.831 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:15:15.831 INFO:teuthology.orchestra.run.smithi121.stdout:Total 25 MB/s | 3.4 MB 00:00 2024-04-05T21:15:15.832 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:15:15.844 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:15:15.844 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:15:15.906 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:15:15.909 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:15:16.340 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:34 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:16.347 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:15:16.541 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:16.951 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:17.338 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:15:17.339 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:15:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:18.2.1-0.el8.x86_64 2024-04-05T21:15:17.339 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:15:17.339 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:15:17.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2024-04-05T21:15:17.977 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:15:17.978 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:15:17.978 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-05T21:15:17.978 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:15:17.978 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:15:17.978 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:18.2.1-0.el8 ceph 3.4 M 2024-04-05T21:15:17.978 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:15:17.978 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:15:17.978 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:15:17.978 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-04-05T21:15:17.978 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:15:17.979 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 3.4 M 2024-04-05T21:15:17.979 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 16 M 2024-04-05T21:15:17.979 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:15:18.094 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-mirror-18.2.1-0.el8.x86_64.rpm 30 MB/s | 3.4 MB 00:00 2024-04-05T21:15:18.094 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:15:18.095 INFO:teuthology.orchestra.run.smithi188.stdout:Total 30 MB/s | 3.4 MB 00:00 2024-04-05T21:15:18.095 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:15:18.106 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:15:18.107 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:15:18.168 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:15:18.171 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:15:18.364 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:36 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:18.610 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:15:18.802 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:19.242 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:19.647 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:15:19.648 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:15:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:18.2.1-0.el8.x86_64 2024-04-05T21:15:19.648 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:15:19.648 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:15:19.887 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-nbd 2024-04-05T21:15:19.997 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-05T21:15:19.998 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:15:19.998 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-04-05T21:15:19.998 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:15:19.998 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-05T21:15:19.998 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:18.2.1-0.el8 ceph 181 k 2024-04-05T21:15:19.998 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:15:19.998 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-05T21:15:19.998 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-04-05T21:15:19.999 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-04-05T21:15:19.999 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:15:19.999 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 181 k 2024-04-05T21:15:19.999 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 568 k 2024-04-05T21:15:19.999 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-05T21:15:20.085 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-18.2.1-0.el8.x86_64.rpm 2.1 MB/s | 181 kB 00:00 2024-04-05T21:15:20.086 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:15:20.086 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.0 MB/s | 181 kB 00:00 2024-04-05T21:15:20.086 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-05T21:15:20.097 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-05T21:15:20.097 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-05T21:15:20.148 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-05T21:15:20.151 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-05T21:15:20.269 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-05T21:15:20.481 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:20.652 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:02:38 ago on Fri 05 Apr 2024 09:12:42 PM UTC. 2024-04-05T21:15:20.772 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:21.129 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:21.129 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:15:21.129 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-05T21:15:21.129 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:18.2.1-0.el8.x86_64 2024-04-05T21:15:21.129 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:15:21.129 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-05T21:15:21.344 DEBUG:teuthology.parallel:result is None 2024-04-05T21:15:22.263 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-05T21:15:22.265 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:15:22.265 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-04-05T21:15:22.265 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:15:22.265 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-05T21:15:22.265 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:18.2.1-0.el8 ceph 181 k 2024-04-05T21:15:22.265 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:15:22.265 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-05T21:15:22.265 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-05T21:15:22.265 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-04-05T21:15:22.265 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:15:22.265 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 181 k 2024-04-05T21:15:22.265 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 568 k 2024-04-05T21:15:22.266 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-05T21:15:22.328 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-nbd-18.2.1-0.el8.x86_64.rpm 2.8 MB/s | 181 kB 00:00 2024-04-05T21:15:22.328 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:15:22.329 INFO:teuthology.orchestra.run.smithi188.stdout:Total 2.8 MB/s | 181 kB 00:00 2024-04-05T21:15:22.329 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-05T21:15:22.340 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-05T21:15:22.340 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-05T21:15:22.390 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-05T21:15:22.393 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-05T21:15:22.507 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-05T21:15:22.725 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:23.013 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-nbd-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:23.406 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:18.2.1-0.el8.x86_64 1/1 2024-04-05T21:15:23.406 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:15:23.406 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-05T21:15:23.407 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:18.2.1-0.el8.x86_64 2024-04-05T21:15:23.407 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:15:23.407 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-05T21:15:23.642 DEBUG:teuthology.parallel:result is None 2024-04-05T21:15:23.642 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-05T21:15:23.642 INFO:teuthology.packaging:ref: None 2024-04-05T21:15:23.642 INFO:teuthology.packaging:tag: v18.2.1 2024-04-05T21:15:23.643 INFO:teuthology.packaging:branch: None 2024-04-05T21:15:23.643 INFO:teuthology.packaging:sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:15:23.643 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://github.com/ceph/ceph-ci; will also look in https://github.com/ceph/ceph 2024-04-05T21:15:23.643 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-05T21:15:23.862 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-05T21:15:23.889 INFO:teuthology.orchestra.run.smithi121.stdout:18.2.1-0.el8 2024-04-05T21:15:23.890 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el8 2024-04-05T21:15:23.890 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-04-05T21:15:23.892 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-04-05T21:15:23.892 INFO:teuthology.packaging:ref: None 2024-04-05T21:15:23.892 INFO:teuthology.packaging:tag: v18.2.1 2024-04-05T21:15:23.892 INFO:teuthology.packaging:branch: None 2024-04-05T21:15:23.892 INFO:teuthology.packaging:sha1: 8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:15:23.892 INFO:teuthology.packaging:Tag 'v18.2.1' not found in https://github.com/ceph/ceph-ci; will also look in https://github.com/ceph/ceph 2024-04-05T21:15:23.892 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=7fe91d5d5842e04be3b4f514d6dd990c54b29c76 2024-04-05T21:15:24.105 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-05T21:15:24.128 INFO:teuthology.orchestra.run.smithi188.stdout:18.2.1-0.el8 2024-04-05T21:15:24.128 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-0.el8 2024-04-05T21:15:24.128 INFO:teuthology.task.install:The correct ceph version 18.2.1-0 is installed. 2024-04-05T21:15:24.130 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-05T21:15:24.130 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:15:24.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-05T21:15:24.163 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-05T21:15:24.164 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-05T21:15:24.195 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-05T21:15:24.195 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:15:24.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-04-05T21:15:24.234 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-05T21:15:24.304 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-05T21:15:24.305 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2024-04-05T21:15:24.332 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-05T21:15:24.398 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-05T21:15:24.399 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:15:24.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-05T21:15:24.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-05T21:15:24.497 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-05T21:15:24.497 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-05T21:15:24.524 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-05T21:15:24.590 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-05T21:15:24.590 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:15:24.590 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-04-05T21:15:24.619 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-05T21:15:24.687 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-05T21:15:24.687 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2024-04-05T21:15:24.713 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-05T21:15:24.780 INFO:teuthology.run_tasks:Running task print... 2024-04-05T21:15:24.791 INFO:teuthology.task.print:**** done install task... 2024-04-05T21:15:24.791 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-05T21:15:24.886 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.io/ceph/ceph:v18.2.1', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '8919605b0a5dc1f1650c8aec3051c77c0e3e240e'} 2024-04-05T21:15:24.886 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v18.2.1 2024-04-05T21:15:24.887 INFO:tasks.cephadm:Cluster fsid is 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:15:24.887 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-05T21:15:24.887 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-05T21:15:24.887 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi121': '172.21.15.121', 'mon.smithi188': '172.21.15.188'} 2024-04-05T21:15:24.887 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-05T21:15:24.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname $(hostname -s) 2024-04-05T21:15:24.917 DEBUG:teuthology.orchestra.run.smithi188:> sudo hostname $(hostname -s) 2024-04-05T21:15:24.944 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-05T21:15:24.944 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:15:25.078 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '8919605b0a5dc1f1650c8aec3051c77c0e3e240e', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/78290/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.2-1173-g8919605b', 'node_name': '172.21.2.7+braggi07', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.2-1173.g8919605b'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-04-05 14:10:52.055812', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri4-testing-2024-04-04-1128-reef', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-05T21:15:25.288 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-yuri4-testing-2024-04-04-1128-reef, sha1 8919605b0a5dc1f1650c8aec3051c77c0e3e240e from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:15:25.289 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/x86_64/flavors/default/cephadm 2024-04-05T21:15:25.290 DEBUG:teuthology.orchestra.run.smithi121:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-05T21:15:25.712 INFO:teuthology.orchestra.run.smithi121.stdout:-rw-rw-r--. 1 ubuntu ubuntu 703022 Apr 5 21:15 /home/ubuntu/cephtest/cephadm 2024-04-05T21:15:25.712 DEBUG:teuthology.orchestra.run.smithi188:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-yuri4-testing-2024-04-04-1128-reef/8919605b0a5dc1f1650c8aec3051c77c0e3e240e/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-05T21:15:26.145 INFO:teuthology.orchestra.run.smithi188.stdout:-rw-rw-r--. 1 ubuntu ubuntu 703022 Apr 5 21:15 /home/ubuntu/cephtest/cephadm 2024-04-05T21:15:26.145 DEBUG:teuthology.orchestra.run.smithi121:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-05T21:15:26.168 DEBUG:teuthology.orchestra.run.smithi188:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-05T21:15:26.202 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v18.2.1 on all hosts... 2024-04-05T21:15:26.203 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-04-05T21:15:26.211 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 pull 2024-04-05T21:15:27.434 INFO:teuthology.orchestra.run.smithi188.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-05T21:15:27.484 INFO:teuthology.orchestra.run.smithi121.stderr:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-05T21:15:47.825 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:15:47.825 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-05T21:15:47.825 INFO:teuthology.orchestra.run.smithi121.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-04-05T21:15:47.825 INFO:teuthology.orchestra.run.smithi121.stdout: "repo_digests": [ 2024-04-05T21:15:47.825 INFO:teuthology.orchestra.run.smithi121.stdout: "quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3", 2024-04-05T21:15:47.825 INFO:teuthology.orchestra.run.smithi121.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-04-05T21:15:47.825 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-04-05T21:15:47.826 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:15:47.941 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-05T21:15:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-05T21:15:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: "image_id": "5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf", 2024-04-05T21:15:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: "repo_digests": [ 2024-04-05T21:15:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: "quay.io/ceph/ceph@sha256:9f35728f6070a596500c0804814a12ab6b98e05067316dc64876fb4b28d04af3", 2024-04-05T21:15:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: "quay.io/ceph/ceph@sha256:e8e55db8b4fd270dbec25bc764437a2a3abb707971c4dba5f559fb83018049dc" 2024-04-05T21:15:47.941 INFO:teuthology.orchestra.run.smithi188.stdout: ] 2024-04-05T21:15:47.941 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-05T21:15:47.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph 2024-04-05T21:15:47.990 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph 2024-04-05T21:15:48.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /etc/ceph 2024-04-05T21:15:48.057 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /etc/ceph 2024-04-05T21:15:48.085 INFO:tasks.cephadm:Writing seed config... 2024-04-05T21:15:48.086 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-05T21:15:48.086 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-05T21:15:48.086 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-05T21:15:48.086 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-05T21:15:48.086 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-05T21:15:48.087 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-05T21:15:48.088 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-05T21:15:48.089 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-05T21:15:48.089 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-05T21:15:48.089 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-05T21:15:48.089 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:15:48.089 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-05T21:15:48.115 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-05T21:15:48.116 DEBUG:teuthology.orchestra.run.smithi121:mon.smithi121> sudo journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mon.smithi121.service 2024-04-05T21:15:48.158 INFO:tasks.cephadm:Bootstrapping... 2024-04-05T21:15:48.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 -v bootstrap --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.121 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-05T21:15:48.184 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:-- Logs begin at Fri 2024-04-05 21:05:55 UTC. -- 2024-04-05T21:15:48.317 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-05T21:15:48.317 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm ['--image', 'quay.io/ceph/ceph:v18.2.1', '-v', 'bootstrap', '--fsid', '9e6dd7f6-f391-11ee-b649-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.121', '--skip-admin-label'] 2024-04-05T21:15:48.348 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-05T21:15:48.349 INFO:teuthology.orchestra.run.smithi121.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-05T21:15:48.349 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying podman|docker is present... 2024-04-05T21:15:48.378 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-05T21:15:48.378 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying lvm2 is present... 2024-04-05T21:15:48.378 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying time synchronization is in place... 2024-04-05T21:15:48.385 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-05T21:15:48.385 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-05T21:15:48.391 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-05T21:15:48.391 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-04-05T21:15:48.396 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout enabled 2024-04-05T21:15:48.402 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout active 2024-04-05T21:15:48.402 INFO:teuthology.orchestra.run.smithi121.stdout:Unit chronyd.service is enabled and running 2024-04-05T21:15:48.402 INFO:teuthology.orchestra.run.smithi121.stdout:Repeating the final host check... 2024-04-05T21:15:48.431 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-05T21:15:48.431 INFO:teuthology.orchestra.run.smithi121.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-05T21:15:48.431 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl is present 2024-04-05T21:15:48.431 INFO:teuthology.orchestra.run.smithi121.stdout:lvcreate is present 2024-04-05T21:15:48.437 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-05T21:15:48.437 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-05T21:15:48.442 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-05T21:15:48.442 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-04-05T21:15:48.448 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout enabled 2024-04-05T21:15:48.454 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout active 2024-04-05T21:15:48.454 INFO:teuthology.orchestra.run.smithi121.stdout:Unit chronyd.service is enabled and running 2024-04-05T21:15:48.454 INFO:teuthology.orchestra.run.smithi121.stdout:Host looks OK 2024-04-05T21:15:48.454 INFO:teuthology.orchestra.run.smithi121.stdout:Cluster fsid: 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:15:48.454 INFO:teuthology.orchestra.run.smithi121.stdout:Acquiring lock 140125914257896 on /run/cephadm/9e6dd7f6-f391-11ee-b649-cb9ed24678a4.lock 2024-04-05T21:15:48.454 INFO:teuthology.orchestra.run.smithi121.stdout:Lock 140125914257896 acquired on /run/cephadm/9e6dd7f6-f391-11ee-b649-cb9ed24678a4.lock 2024-04-05T21:15:48.455 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying IP 172.21.15.121 port 3300 ... 2024-04-05T21:15:48.455 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying IP 172.21.15.121 port 6789 ... 2024-04-05T21:15:48.455 INFO:teuthology.orchestra.run.smithi121.stdout:Base mon IP(s) is [172.21.15.121:3300, 172.21.15.121:6789], mon addrv is [v2:172.21.15.121:3300,v1:172.21.15.121:6789] 2024-04-05T21:15:48.459 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.121 metric 100 2024-04-05T21:15:48.459 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.121 metric 100 2024-04-05T21:15:48.461 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-05T21:15:48.461 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-05T21:15:48.462 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1741sec hoplimit 64 pref medium 2024-04-05T21:15:48.464 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-05T21:15:48.464 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-05T21:15:48.464 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-05T21:15:48.464 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-05T21:15:48.464 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cba3/64 scope link 2024-04-05T21:15:48.464 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-05T21:15:48.465 INFO:teuthology.orchestra.run.smithi121.stdout:Mon IP `172.21.15.121` is in CIDR network `172.21.0.0/20` 2024-04-05T21:15:48.465 INFO:teuthology.orchestra.run.smithi121.stdout:Mon IP `172.21.15.121` is in CIDR network `172.21.0.0/20` 2024-04-05T21:15:48.465 INFO:teuthology.orchestra.run.smithi121.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-05T21:15:48.465 INFO:teuthology.orchestra.run.smithi121.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-05T21:15:48.466 INFO:teuthology.orchestra.run.smithi121.stdout:Pulling container image quay.io/ceph/ceph:v18.2.1... 2024-04-05T21:15:49.410 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stdout 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-04-05T21:15:49.410 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Trying to pull quay.io/ceph/ceph:v18.2.1... 2024-04-05T21:15:49.410 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Getting image source signatures 2024-04-05T21:15:49.410 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Copying blob sha256:a733d3c618b71f19c168ebecd1953429dce2c1631835ca182e9551c36dce5989 2024-04-05T21:15:49.410 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-04-05T21:15:49.410 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Copying config sha256:5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf 2024-04-05T21:15:49.410 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-05T21:15:50.604 INFO:teuthology.orchestra.run.smithi121.stdout:ceph: stdout ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-04-05T21:15:50.604 INFO:teuthology.orchestra.run.smithi121.stdout:Ceph version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 2024-04-05T21:15:50.604 INFO:teuthology.orchestra.run.smithi121.stdout:Extracting ceph user uid/gid from container image... 2024-04-05T21:15:51.792 INFO:teuthology.orchestra.run.smithi121.stdout:stat: stdout 167 167 2024-04-05T21:15:51.793 INFO:teuthology.orchestra.run.smithi121.stdout:Creating initial keys... 2024-04-05T21:15:53.088 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph-authtool: stdout AQAIahBm17xlFRAAeo2AYt20x9TsyqgXGjVVXA== 2024-04-05T21:15:54.247 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph-authtool: stdout AQAJahBmyGlaIBAAlFisqkhk8OCuOyJI/hOdWw== 2024-04-05T21:15:55.446 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph-authtool: stdout AQAKahBm3CuqKRAAuaFwSSJ7WmwHtLdviJP+nA== 2024-04-05T21:15:55.447 INFO:teuthology.orchestra.run.smithi121.stdout:Creating initial monmap... 2024-04-05T21:15:56.591 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-05T21:15:56.591 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-04-05T21:15:56.591 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:15:56.591 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-05T21:15:56.591 INFO:teuthology.orchestra.run.smithi121.stdout:monmaptool for smithi121 [v2:172.21.15.121:3300,v1:172.21.15.121:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-05T21:15:56.591 INFO:teuthology.orchestra.run.smithi121.stdout:setting min_mon_release = pacific 2024-04-05T21:15:56.591 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: set fsid to 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:15:56.591 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-05T21:15:56.592 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:15:56.592 INFO:teuthology.orchestra.run.smithi121.stdout:Creating mon... 2024-04-05T21:15:57.616 INFO:teuthology.orchestra.run.smithi121.stdout:create mon.smithi121 on 2024-04-05T21:15:57.746 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-04-05T21:15:57.835 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-05T21:15:57.951 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4.target. 2024-04-05T21:15:57.951 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4.target. 2024-04-05T21:15:58.120 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mon.smithi121 2024-04-05T21:15:58.121 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mon.smithi121.service: Unit ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mon.smithi121.service not loaded. 2024-04-05T21:15:58.213 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4.target.wants/ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mon.smithi121.service → /etc/systemd/system/ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@.service. 2024-04-05T21:15:58.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:15:58 smithi121 systemd[1]: Starting Ceph mon.smithi121 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:15:58.871 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-05T21:15:58.871 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout disabled 2024-04-05T21:15:58.876 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-05T21:15:58.876 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-04-05T21:15:58.876 INFO:teuthology.orchestra.run.smithi121.stdout:firewalld.service is not enabled 2024-04-05T21:15:58.876 INFO:teuthology.orchestra.run.smithi121.stdout:Not possible to enable service . firewalld.service is not available 2024-04-05T21:15:58.876 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mon to start... 2024-04-05T21:15:58.877 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mon... 2024-04-05T21:15:59.099 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:15:58 smithi121 ceph-mon[39233]: mkfs 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:15:59.100 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:15:58 smithi121 ceph-mon[39233]: mon.smithi121 is new leader, mons smithi121 in quorum (ranks 0) 2024-04-05T21:15:59.100 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:15:58 smithi121 bash[39196]: 693ca0d40e07f6a73464dbdefa9ffdb4bb38f2fd567c3eb6acdfdc3176e3cdc5 2024-04-05T21:15:59.100 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:15:58 smithi121 systemd[1]: Started Ceph mon.smithi121 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:16:00.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout cluster: 2024-04-05T21:16:00.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout id: 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:16:00.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-05T21:16:00.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-04-05T21:16:00.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout services: 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi121 (age 0.683449s) 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout data: 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout pgs: 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.403+0000 7fbd71732700 1 Processor -- start 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.403+0000 7fbd71732700 1 -- start start 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.403+0000 7fbd71732700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 0x7fbd6c106bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.403+0000 7fbd71732700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c1071c0 con 0x7fbd6c1067d0 2024-04-05T21:16:00.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.404+0000 7fbd6affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 0x7fbd6c106bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:00.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.404+0000 7fbd6affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 0x7fbd6c106bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49128/0 (socket says 172.21.15.121:49128) 2024-04-05T21:16:00.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.404+0000 7fbd6affd700 1 -- 172.21.15.121:0/174175196 learned_addr learned my addr 172.21.15.121:0/174175196 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:00.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.404+0000 7fbd6affd700 1 -- 172.21.15.121:0/174175196 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6c105430 con 0x7fbd6c1067d0 2024-04-05T21:16:00.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.404+0000 7fbd6affd700 1 --2- 172.21.15.121:0/174175196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 0x7fbd6c106bf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0098a0 tx=0x7fbd5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5198fa1cde6e651 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:00.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.404+0000 7fbd69ffb700 1 -- 172.21.15.121:0/174175196 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd5c004180 con 0x7fbd6c1067d0 2024-04-05T21:16:00.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.404+0000 7fbd69ffb700 1 -- 172.21.15.121:0/174175196 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5c0042e0 con 0x7fbd6c1067d0 2024-04-05T21:16:00.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.404+0000 7fbd69ffb700 1 -- 172.21.15.121:0/174175196 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd5c0036a0 con 0x7fbd6c1067d0 2024-04-05T21:16:00.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.405+0000 7fbd71732700 1 -- 172.21.15.121:0/174175196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 msgr2=0x7fbd6c106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:00.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.405+0000 7fbd71732700 1 --2- 172.21.15.121:0/174175196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 0x7fbd6c106bf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0098a0 tx=0x7fbd5c009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:00.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.405+0000 7fbd71732700 1 -- 172.21.15.121:0/174175196 shutdown_connections 2024-04-05T21:16:00.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.405+0000 7fbd71732700 1 --2- 172.21.15.121:0/174175196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 0x7fbd6c106bf0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:00.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.405+0000 7fbd71732700 1 -- 172.21.15.121:0/174175196 >> 172.21.15.121:0/174175196 conn(0x7fbd6c0fff40 msgr2=0x7fbd6c102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.405+0000 7fbd71732700 1 -- 172.21.15.121:0/174175196 shutdown_connections 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.405+0000 7fbd71732700 1 -- 172.21.15.121:0/174175196 wait complete. 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.405+0000 7fbd71732700 1 Processor -- start 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.406+0000 7fbd71732700 1 -- start start 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.406+0000 7fbd71732700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 0x7fbd6c19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.406+0000 7fbd71732700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c19bf60 con 0x7fbd6c1067d0 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.406+0000 7fbd6affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 0x7fbd6c19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.406+0000 7fbd6affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 0x7fbd6c19ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49132/0 (socket says 172.21.15.121:49132) 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.406+0000 7fbd6affd700 1 -- 172.21.15.121:0/2585007617 learned_addr learned my addr 172.21.15.121:0/2585007617 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.406+0000 7fbd6affd700 1 -- 172.21.15.121:0/2585007617 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c009550 con 0x7fbd6c1067d0 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.406+0000 7fbd6affd700 1 --2- 172.21.15.121:0/2585007617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 0x7fbd6c19ba20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c073950 tx=0x7fbd5c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.406+0000 7fbd53fff700 1 -- 172.21.15.121:0/2585007617 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd5c003ce0 con 0x7fbd6c1067d0 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.406+0000 7fbd53fff700 1 -- 172.21.15.121:0/2585007617 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5c020070 con 0x7fbd6c1067d0 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.406+0000 7fbd71732700 1 -- 172.21.15.121:0/2585007617 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6c19c160 con 0x7fbd6c1067d0 2024-04-05T21:16:00.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.406+0000 7fbd53fff700 1 -- 172.21.15.121:0/2585007617 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd5c011630 con 0x7fbd6c1067d0 2024-04-05T21:16:00.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.407+0000 7fbd71732700 1 -- 172.21.15.121:0/2585007617 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6c19c580 con 0x7fbd6c1067d0 2024-04-05T21:16:00.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.407+0000 7fbd53fff700 1 -- 172.21.15.121:0/2585007617 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbd5c022030 con 0x7fbd6c1067d0 2024-04-05T21:16:00.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.407+0000 7fbd53fff700 1 -- 172.21.15.121:0/2585007617 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbd5c025d00 con 0x7fbd6c1067d0 2024-04-05T21:16:00.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.407+0000 7fbd71732700 1 -- 172.21.15.121:0/2585007617 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6c0623c0 con 0x7fbd6c1067d0 2024-04-05T21:16:00.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.410+0000 7fbd53fff700 1 -- 172.21.15.121:0/2585007617 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fbd5c025480 con 0x7fbd6c1067d0 2024-04-05T21:16:00.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.478+0000 7fbd71732700 1 -- 172.21.15.121:0/2585007617 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fbd6c19f0b0 con 0x7fbd6c1067d0 2024-04-05T21:16:00.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.479+0000 7fbd53fff700 1 -- 172.21.15.121:0/2585007617 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fbd5c011c20 con 0x7fbd6c1067d0 2024-04-05T21:16:00.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.479+0000 7fbd71732700 1 -- 172.21.15.121:0/2585007617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 msgr2=0x7fbd6c19ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:00.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.479+0000 7fbd71732700 1 --2- 172.21.15.121:0/2585007617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 0x7fbd6c19ba20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c073950 tx=0x7fbd5c004750 comp rx=0 tx=0).stop 2024-04-05T21:16:00.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.479+0000 7fbd71732700 1 -- 172.21.15.121:0/2585007617 shutdown_connections 2024-04-05T21:16:00.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.479+0000 7fbd71732700 1 --2- 172.21.15.121:0/2585007617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd6c1067d0 0x7fbd6c19ba20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:00.121 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:15:59 smithi121 ceph-mon[39233]: mon.smithi121 is new leader, mons smithi121 in quorum (ranks 0) 2024-04-05T21:16:00.121 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:15:59 smithi121 ceph-mon[39233]: monmap e1: 1 mons at {smithi121=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T21:16:00.121 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:15:59 smithi121 ceph-mon[39233]: fsmap 2024-04-05T21:16:00.121 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:15:59 smithi121 ceph-mon[39233]: osdmap e1: 0 total, 0 up, 0 in 2024-04-05T21:16:00.121 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:15:59 smithi121 ceph-mon[39233]: mgrmap e1: no daemons active 2024-04-05T21:16:00.121 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:15:59 smithi121 ceph-mon[39233]: from='client.? 172.21.15.121:0/2585007617' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-05T21:16:00.122 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.479+0000 7fbd71732700 1 -- 172.21.15.121:0/2585007617 >> 172.21.15.121:0/2585007617 conn(0x7fbd6c0fff40 msgr2=0x7fbd6c104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:00.122 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.479+0000 7fbd71732700 1 -- 172.21.15.121:0/2585007617 shutdown_connections 2024-04-05T21:16:00.122 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:15:59.479+0000 7fbd71732700 1 -- 172.21.15.121:0/2585007617 wait complete. 2024-04-05T21:16:00.122 INFO:teuthology.orchestra.run.smithi121.stdout:mon is available 2024-04-05T21:16:00.122 INFO:teuthology.orchestra.run.smithi121.stdout:Assimilating anything we can from ceph.conf... 2024-04-05T21:16:01.193 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:00 smithi121 ceph-mon[39233]: from='client.? 172.21.15.121:0/1100117100' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-05T21:16:01.193 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:00 smithi121 ceph-mon[39233]: from='client.? 172.21.15.121:0/1100117100' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-05T21:16:01.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-04-05T21:16:01.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [global] 2024-04-05T21:16:01.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout fsid = 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:16:01.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.121:3300,v1:172.21.15.121:6789] 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [osd] 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-05T21:16:01.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.709+0000 7fb63e636700 1 Processor -- start 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.709+0000 7fb63e636700 1 -- start start 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.709+0000 7fb63e636700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 0x7fb638106750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.709+0000 7fb63e636700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb638106d20 con 0x7fb638106330 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.709+0000 7fb637fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 0x7fb638106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.709+0000 7fb637fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 0x7fb638106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49146/0 (socket says 172.21.15.121:49146) 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.709+0000 7fb637fff700 1 -- 172.21.15.121:0/724516620 learned_addr learned my addr 172.21.15.121:0/724516620 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.709+0000 7fb637fff700 1 -- 172.21.15.121:0/724516620 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb638107530 con 0x7fb638106330 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.710+0000 7fb637fff700 1 --2- 172.21.15.121:0/724516620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 0x7fb638106750 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb6280098a0 tx=0x7fb628009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a625aa5f4edf9b server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.710+0000 7fb636ffd700 1 -- 172.21.15.121:0/724516620 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb628004180 con 0x7fb638106330 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.710+0000 7fb636ffd700 1 -- 172.21.15.121:0/724516620 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6280042e0 con 0x7fb638106330 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.710+0000 7fb636ffd700 1 -- 172.21.15.121:0/724516620 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6280036a0 con 0x7fb638106330 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.710+0000 7fb63e636700 1 -- 172.21.15.121:0/724516620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 msgr2=0x7fb638106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.710+0000 7fb63e636700 1 --2- 172.21.15.121:0/724516620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 0x7fb638106750 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb6280098a0 tx=0x7fb628009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:01.505 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.710+0000 7fb63e636700 1 -- 172.21.15.121:0/724516620 shutdown_connections 2024-04-05T21:16:01.506 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.710+0000 7fb63e636700 1 --2- 172.21.15.121:0/724516620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 0x7fb638106750 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:01.506 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.710+0000 7fb63e636700 1 -- 172.21.15.121:0/724516620 >> 172.21.15.121:0/724516620 conn(0x7fb6381018d0 msgr2=0x7fb638103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:01.506 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.710+0000 7fb63e636700 1 -- 172.21.15.121:0/724516620 shutdown_connections 2024-04-05T21:16:01.506 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.710+0000 7fb63e636700 1 -- 172.21.15.121:0/724516620 wait complete. 2024-04-05T21:16:01.506 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.711+0000 7fb63e636700 1 Processor -- start 2024-04-05T21:16:01.506 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.711+0000 7fb63e636700 1 -- start start 2024-04-05T21:16:01.506 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.711+0000 7fb63e636700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 0x7fb63807aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:01.506 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.711+0000 7fb63e636700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63807afb0 con 0x7fb638106330 2024-04-05T21:16:01.506 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.711+0000 7fb637fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 0x7fb63807aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:01.506 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.711+0000 7fb637fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 0x7fb63807aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49150/0 (socket says 172.21.15.121:49150) 2024-04-05T21:16:01.506 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.711+0000 7fb637fff700 1 -- 172.21.15.121:0/1100117100 learned_addr learned my addr 172.21.15.121:0/1100117100 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:01.506 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.711+0000 7fb637fff700 1 -- 172.21.15.121:0/1100117100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb628009550 con 0x7fb638106330 2024-04-05T21:16:01.506 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.712+0000 7fb637fff700 1 --2- 172.21.15.121:0/1100117100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 0x7fb63807aa70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb628004000 tx=0x7fb628004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.712+0000 7fb6357fa700 1 -- 172.21.15.121:0/1100117100 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb628003ce0 con 0x7fb638106330 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.712+0000 7fb6357fa700 1 -- 172.21.15.121:0/1100117100 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb628020070 con 0x7fb638106330 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.712+0000 7fb63e636700 1 -- 172.21.15.121:0/1100117100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6380790c0 con 0x7fb638106330 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.712+0000 7fb6357fa700 1 -- 172.21.15.121:0/1100117100 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb628011630 con 0x7fb638106330 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.712+0000 7fb63e636700 1 -- 172.21.15.121:0/1100117100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6380794e0 con 0x7fb638106330 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.712+0000 7fb6357fa700 1 -- 172.21.15.121:0/1100117100 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb628022030 con 0x7fb638106330 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.712+0000 7fb6357fa700 1 -- 172.21.15.121:0/1100117100 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb628025d00 con 0x7fb638106330 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.713+0000 7fb63e636700 1 -- 172.21.15.121:0/1100117100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb638079ef0 con 0x7fb638106330 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.715+0000 7fb6357fa700 1 -- 172.21.15.121:0/1100117100 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fb628025480 con 0x7fb638106330 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.782+0000 7fb63e636700 1 -- 172.21.15.121:0/1100117100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb638193120 con 0x7fb638106330 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.785+0000 7fb6357fa700 1 -- 172.21.15.121:0/1100117100 <== mon.0 v2:172.21.15.121:3300/0 7 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7fb628011c20 con 0x7fb638106330 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.785+0000 7fb6357fa700 1 -- 172.21.15.121:0/1100117100 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7fb638193120 con 0x7fb638106330 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.786+0000 7fb63e636700 1 -- 172.21.15.121:0/1100117100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 msgr2=0x7fb63807aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:01.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.786+0000 7fb63e636700 1 --2- 172.21.15.121:0/1100117100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 0x7fb63807aa70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb628004000 tx=0x7fb628004750 comp rx=0 tx=0).stop 2024-04-05T21:16:01.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.786+0000 7fb63e636700 1 -- 172.21.15.121:0/1100117100 shutdown_connections 2024-04-05T21:16:01.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.786+0000 7fb63e636700 1 --2- 172.21.15.121:0/1100117100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb638106330 0x7fb63807aa70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:01.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.786+0000 7fb63e636700 1 -- 172.21.15.121:0/1100117100 >> 172.21.15.121:0/1100117100 conn(0x7fb6381018d0 msgr2=0x7fb63818dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:01.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.786+0000 7fb63e636700 1 -- 172.21.15.121:0/1100117100 shutdown_connections 2024-04-05T21:16:01.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:00.787+0000 7fb63e636700 1 -- 172.21.15.121:0/1100117100 wait complete. 2024-04-05T21:16:01.508 INFO:teuthology.orchestra.run.smithi121.stdout:Generating new minimal ceph.conf... 2024-04-05T21:16:02.468 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:02 smithi121 ceph-mon[39233]: from='client.? 172.21.15.121:0/2426564514' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:16:02.756 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.007+0000 7fb4ad1b8700 1 Processor -- start 2024-04-05T21:16:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.007+0000 7fb4ad1b8700 1 -- start start 2024-04-05T21:16:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.007+0000 7fb4ad1b8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 0x7fb4a8106730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.007+0000 7fb4ad1b8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a8106d00 con 0x7fb4a8106310 2024-04-05T21:16:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.007+0000 7fb4a6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 0x7fb4a8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.007+0000 7fb4a6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 0x7fb4a8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49162/0 (socket says 172.21.15.121:49162) 2024-04-05T21:16:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.007+0000 7fb4a6d9d700 1 -- 172.21.15.121:0/965611047 learned_addr learned my addr 172.21.15.121:0/965611047 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.007+0000 7fb4a6d9d700 1 -- 172.21.15.121:0/965611047 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a8107510 con 0x7fb4a8106310 2024-04-05T21:16:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.008+0000 7fb4a6d9d700 1 --2- 172.21.15.121:0/965611047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 0x7fb4a8106730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb4980098a0 tx=0x7fb498009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff84f53ee61abf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.008+0000 7fb4a5d9b700 1 -- 172.21.15.121:0/965611047 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb498004180 con 0x7fb4a8106310 2024-04-05T21:16:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.008+0000 7fb4a5d9b700 1 -- 172.21.15.121:0/965611047 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7fb4980042e0 con 0x7fb4a8106310 2024-04-05T21:16:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.008+0000 7fb4a5d9b700 1 -- 172.21.15.121:0/965611047 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4980119d0 con 0x7fb4a8106310 2024-04-05T21:16:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.008+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/965611047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 msgr2=0x7fb4a8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:02.757 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.008+0000 7fb4ad1b8700 1 --2- 172.21.15.121:0/965611047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 0x7fb4a8106730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb4980098a0 tx=0x7fb498009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:02.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.008+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/965611047 shutdown_connections 2024-04-05T21:16:02.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.008+0000 7fb4ad1b8700 1 --2- 172.21.15.121:0/965611047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 0x7fb4a8106730 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:02.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.008+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/965611047 >> 172.21.15.121:0/965611047 conn(0x7fb4a81018d0 msgr2=0x7fb4a8103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:02.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.008+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/965611047 shutdown_connections 2024-04-05T21:16:02.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.008+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/965611047 wait complete. 2024-04-05T21:16:02.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.009+0000 7fb4ad1b8700 1 Processor -- start 2024-04-05T21:16:02.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.009+0000 7fb4ad1b8700 1 -- start start 2024-04-05T21:16:02.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.010+0000 7fb4ad1b8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 0x7fb4a819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:02.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.010+0000 7fb4ad1b8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a819a550 con 0x7fb4a8106310 2024-04-05T21:16:02.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.010+0000 7fb4a6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 0x7fb4a819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:02.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.010+0000 7fb4a6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 0x7fb4a819a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49166/0 (socket says 172.21.15.121:49166) 2024-04-05T21:16:02.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.010+0000 7fb4a6d9d700 1 -- 172.21.15.121:0/2426564514 learned_addr learned my addr 172.21.15.121:0/2426564514 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:02.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.010+0000 7fb4a6d9d700 1 -- 172.21.15.121:0/2426564514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb498009550 con 0x7fb4a8106310 2024-04-05T21:16:02.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.010+0000 7fb4a6d9d700 1 --2- 172.21.15.121:0/2426564514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 0x7fb4a819a010 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb498006af0 tx=0x7fb498011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:02.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.010+0000 7fb48ffff700 1 -- 172.21.15.121:0/2426564514 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb498003b40 con 0x7fb4a8106310 2024-04-05T21:16:02.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.010+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/2426564514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a819a750 con 0x7fb4a8106310 2024-04-05T21:16:02.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.010+0000 7fb48ffff700 1 -- 172.21.15.121:0/2426564514 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7fb49801a880 con 0x7fb4a8106310 2024-04-05T21:16:02.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.010+0000 7fb48ffff700 1 -- 172.21.15.121:0/2426564514 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb498019890 con 0x7fb4a8106310 2024-04-05T21:16:02.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.010+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/2426564514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a819ab70 con 0x7fb4a8106310 2024-04-05T21:16:02.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.011+0000 7fb48ffff700 1 -- 172.21.15.121:0/2426564514 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb49802d440 con 0x7fb4a8106310 2024-04-05T21:16:02.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.011+0000 7fb48ffff700 1 -- 172.21.15.121:0/2426564514 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb498024820 con 0x7fb4a8106310 2024-04-05T21:16:02.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.011+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/2426564514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a80623c0 con 0x7fb4a8106310 2024-04-05T21:16:02.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.014+0000 7fb48ffff700 1 -- 172.21.15.121:0/2426564514 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fb49801adf0 con 0x7fb4a8106310 2024-04-05T21:16:02.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.079+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/2426564514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fb4a810a710 con 0x7fb4a8106310 2024-04-05T21:16:02.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.080+0000 7fb48ffff700 1 -- 172.21.15.121:0/2426564514 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fb49801a9f0 con 0x7fb4a8106310 2024-04-05T21:16:02.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.080+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/2426564514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 msgr2=0x7fb4a819a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:02.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.080+0000 7fb4ad1b8700 1 --2- 172.21.15.121:0/2426564514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 0x7fb4a819a010 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb498006af0 tx=0x7fb498011a30 comp rx=0 tx=0).stop 2024-04-05T21:16:02.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.080+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/2426564514 shutdown_connections 2024-04-05T21:16:02.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.080+0000 7fb4ad1b8700 1 --2- 172.21.15.121:0/2426564514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4a8106310 0x7fb4a819a010 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:02.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.080+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/2426564514 >> 172.21.15.121:0/2426564514 conn(0x7fb4a81018d0 msgr2=0x7fb4a8102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:02.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.080+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/2426564514 shutdown_connections 2024-04-05T21:16:02.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:02.080+0000 7fb4ad1b8700 1 -- 172.21.15.121:0/2426564514 wait complete. 2024-04-05T21:16:02.760 INFO:teuthology.orchestra.run.smithi121.stdout:Restarting the monitor... 2024-04-05T21:16:03.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:02 smithi121 systemd[1]: Stopping Ceph mon.smithi121 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:16:03.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:02 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[39210]: 2024-04-05T21:16:02.990+0000 7fbc751ff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi121 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T21:16:03.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:02 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[39210]: 2024-04-05T21:16:02.990+0000 7fbc751ff700 -1 mon.smithi121@0(leader) e1 *** Got Signal Terminated *** 2024-04-05T21:16:04.423 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:04 smithi121 bash[39566]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121 2024-04-05T21:16:04.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:04 smithi121 systemd[1]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mon.smithi121.service: Succeeded. 2024-04-05T21:16:04.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:04 smithi121 systemd[1]: Stopped Ceph mon.smithi121 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:16:04.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:04 smithi121 systemd[1]: Starting Ceph mon.smithi121 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:16:05.255 INFO:teuthology.orchestra.run.smithi121.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-05T21:16:05.469 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: set uid:gid to 167:167 (ceph:ceph) 2024-04-05T21:16:05.470 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable), process ceph-mon, pid 7 2024-04-05T21:16:05.470 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: pidfile_write: ignore empty --pid-file 2024-04-05T21:16:05.470 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: load: jerasure load: lrc 2024-04-05T21:16:05.470 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: RocksDB version: 7.9.2 2024-04-05T21:16:05.470 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Git sha 0 2024-04-05T21:16:05.470 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Compile date 2023-12-11 22:07:34 2024-04-05T21:16:05.470 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: DB SUMMARY 2024-04-05T21:16:05.470 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: DB Session ID: DS74E6TO5RV6L01LXWTH 2024-04-05T21:16:05.470 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: CURRENT file: CURRENT 2024-04-05T21:16:05.470 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: IDENTITY file: IDENTITY 2024-04-05T21:16:05.470 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-05T21:16:05.470 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi121/store.db dir, Total Num: 1, files: 000008.sst 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi121/store.db: 000009.log size: 96966 ; 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.error_if_exists: 0 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.create_if_missing: 0 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.paranoid_checks: 1 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.env: 0x5558dc994720 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.fs: PosixFileSystem 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.info_log: 0x5558df4952e0 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_file_opening_threads: 16 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.statistics: (nil) 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.use_fsync: 0 2024-04-05T21:16:05.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_log_file_size: 0 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.keep_log_file_num: 1000 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.recycle_log_file_num: 0 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.allow_fallocate: 1 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.allow_mmap_reads: 0 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.allow_mmap_writes: 0 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.use_direct_reads: 0 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.create_missing_column_families: 0 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.db_log_dir: 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.wal_dir: 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-05T21:16:05.472 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.advise_random_on_open: 1 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.db_write_buffer_size: 0 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.write_buffer_manager: 0x5558de706140 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.rate_limiter: (nil) 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.wal_recovery_mode: 2 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.enable_thread_tracking: 0 2024-04-05T21:16:05.473 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.enable_pipelined_write: 0 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.unordered_write: 0 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.row_cache: None 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.wal_filter: None 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.allow_ingest_behind: 0 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.two_write_queues: 0 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.manual_wal_flush: 0 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.wal_compression: 0 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.atomic_flush: 0 2024-04-05T21:16:05.474 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-05T21:16:05.475 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-05T21:16:05.475 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-05T21:16:05.475 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.log_readahead_size: 0 2024-04-05T21:16:05.475 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-05T21:16:05.475 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.best_efforts_recovery: 0 2024-04-05T21:16:05.475 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-05T21:16:05.475 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-05T21:16:05.475 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.allow_data_in_errors: 0 2024-04-05T21:16:05.475 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.db_host_id: __hostname__ 2024-04-05T21:16:05.475 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-05T21:16:05.476 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_background_jobs: 2 2024-04-05T21:16:05.476 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_background_compactions: -1 2024-04-05T21:16:05.476 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_subcompactions: 1 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_total_wal_size: 0 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_open_files: -1 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.bytes_per_sync: 0 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compaction_readahead_size: 0 2024-04-05T21:16:05.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_background_flushes: -1 2024-04-05T21:16:05.478 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Compression algorithms supported: 2024-04-05T21:16:05.478 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-05T21:16:05.478 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: kZSTD supported: 0 2024-04-05T21:16:05.478 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: kXpressCompression supported: 0 2024-04-05T21:16:05.478 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: kLZ4HCCompression supported: 1 2024-04-05T21:16:05.478 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: kZlibCompression supported: 1 2024-04-05T21:16:05.478 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: kSnappyCompression supported: 1 2024-04-05T21:16:05.478 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: kLZ4Compression supported: 1 2024-04-05T21:16:05.478 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: kBZip2Compression supported: 0 2024-04-05T21:16:05.478 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-05T21:16:05.478 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-05T21:16:05.478 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi121/store.db/MANIFEST-000010 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.merge_operator: 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compaction_filter: None 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compaction_filter_factory: None 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.sst_partitioner_factory: None 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5558df4953e0) 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: cache_index_and_filter_blocks: 1 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: pin_top_level_index_and_filter: 1 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: index_type: 0 2024-04-05T21:16:05.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: data_block_index_type: 0 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: index_shortening: 1 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: checksum: 4 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: no_block_cache: 0 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_cache: 0x5558de7a7350 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_cache_name: BinnedLRUCache 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_cache_options: 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: capacity : 536870912 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: num_shard_bits : 4 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: strict_capacity_limit : 0 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: high_pri_pool_ratio: 0.000 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_cache_compressed: (nil) 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: persistent_cache: (nil) 2024-04-05T21:16:05.480 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_size: 4096 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_size_deviation: 10 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_restart_interval: 16 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: index_block_restart_interval: 1 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: metadata_block_size: 4096 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: partition_filters: 0 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: use_delta_encoding: 1 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: filter_policy: bloomfilter 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: whole_key_filtering: 1 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: verify_compression: 0 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: read_amp_bytes_per_bit: 0 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: format_version: 5 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: enable_index_compression: 1 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_align: 0 2024-04-05T21:16:05.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: max_auto_readahead_size: 262144 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: prepopulate_block_cache: 0 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: initial_auto_readahead_size: 8192 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: num_file_reads_for_auto_readahead: 2 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.write_buffer_size: 33554432 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_write_buffer_number: 2 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compression: NoCompression 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.bottommost_compression: Disabled 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.prefix_extractor: nullptr 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.num_levels: 7 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-05T21:16:05.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compression_opts.level: 32767 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compression_opts.strategy: 0 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-05T21:16:05.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compression_opts.enabled: false 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.target_file_size_base: 67108864 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-05T21:16:05.484 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.arena_block_size: 1048576 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.disable_auto_compactions: 0 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-05T21:16:05.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.table_properties_collectors: 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.inplace_update_support: 0 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.bloom_locality: 0 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.max_successive_merges: 0 2024-04-05T21:16:05.486 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.paranoid_file_checks: 0 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.force_consistency_checks: 1 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.report_bg_io_stats: 0 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.ttl: 2592000 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.enable_blob_files: false 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.min_blob_size: 0 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.blob_file_size: 268435456 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-05T21:16:05.487 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.blob_file_starting_level: 0 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi121/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 31f70f03-b4f1-420b-ad02-7abefe01d6b5 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712351765151128, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712351765152079, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92413, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 308, "table_properties": {"data_size": 90430, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13225, "raw_average_key_size": 49, "raw_value_size": 83169, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 267, "num_filter_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712351765, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "31f70f03-b4f1-420b-ad02-7abefe01d6b5", "db_session_id": "DS74E6TO5RV6L01LXWTH", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712351765152160, "job": 1, "event": "recovery_finished"} 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi121/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5558de844000 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: DB pointer 0x5558de830000 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-05T21:16:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: ** DB Stats ** 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: ** Compaction Stats [default] ** 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: L0 2/0 92.12 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-05T21:16:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Sum 2/0 92.12 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-05T21:16:05.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-05T21:16:05.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:16:05.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: ** Compaction Stats [default] ** 2024-04-05T21:16:05.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T21:16:05.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T21:16:05.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 142.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-05T21:16:05.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:16:05.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-05T21:16:05.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:16:05.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Cumulative compaction: 0.00 GB write, 30.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Interval compaction: 0.00 GB write, 30.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Block cache BinnedLRUCache@0x5558de7a7350#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: starting mon.smithi121 rank 0 at public addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] at bind addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi121 fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: mon.smithi121@-1(???) e1 preinit fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:16:05.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: mon.smithi121@-1(???).mds e1 new map 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: mon.smithi121@-1(???).mds e1 print_map 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: e1 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: legacy client fscid: -1 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: No filesystems configured 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: mon.smithi121@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: mon.smithi121@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: mon.smithi121@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: mon.smithi121@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: mon.smithi121@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: mon.smithi121 is new leader, mons smithi121 in quorum (ranks 0) 2024-04-05T21:16:05.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: monmap e1: 1 mons at {smithi121=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T21:16:05.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: fsmap 2024-04-05T21:16:05.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: osdmap e1: 0 total, 0 up, 0 in 2024-04-05T21:16:05.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 ceph-mon[39723]: mgrmap e1: no daemons active 2024-04-05T21:16:05.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 bash[39685]: 3c30ce99c0062d89b3556b7201875a326e93541e73c40cba212548460a739080 2024-04-05T21:16:05.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:05 smithi121 systemd[1]: Started Ceph mon.smithi121 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:16:06.520 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.783+0000 7f995a6c2700 1 Processor -- start 2024-04-05T21:16:06.526 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.783+0000 7f995a6c2700 1 -- start start 2024-04-05T21:16:06.526 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.783+0000 7f995a6c2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 0x7f99541091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:06.526 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.783+0000 7f995a6c2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99540744f0 con 0x7f9954106de0 2024-04-05T21:16:06.526 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.784+0000 7f9953fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 0x7f99541091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:06.526 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.784+0000 7f9953fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 0x7f99541091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55970/0 (socket says 172.21.15.121:55970) 2024-04-05T21:16:06.526 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.784+0000 7f9953fff700 1 -- 172.21.15.121:0/3260226512 learned_addr learned my addr 172.21.15.121:0/3260226512 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:06.526 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.784+0000 7f9953fff700 1 -- 172.21.15.121:0/3260226512 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9954109710 con 0x7f9954106de0 2024-04-05T21:16:06.526 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.784+0000 7f9953fff700 1 --2- 172.21.15.121:0/3260226512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 0x7f99541091d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f993c0098a0 tx=0x7f993c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff85673f20814a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:06.526 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.784+0000 7f9952ffd700 1 -- 172.21.15.121:0/3260226512 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f993c004180 con 0x7f9954106de0 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.784+0000 7f9952ffd700 1 -- 172.21.15.121:0/3260226512 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f993c0042e0 con 0x7f9954106de0 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.784+0000 7f9952ffd700 1 -- 172.21.15.121:0/3260226512 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f993c0119d0 con 0x7f9954106de0 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.785+0000 7f995a6c2700 1 -- 172.21.15.121:0/3260226512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 msgr2=0x7f99541091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.785+0000 7f995a6c2700 1 --2- 172.21.15.121:0/3260226512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 0x7f99541091d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f993c0098a0 tx=0x7f993c009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.785+0000 7f995a6c2700 1 -- 172.21.15.121:0/3260226512 shutdown_connections 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.785+0000 7f995a6c2700 1 --2- 172.21.15.121:0/3260226512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 0x7f99541091d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.785+0000 7f995a6c2700 1 -- 172.21.15.121:0/3260226512 >> 172.21.15.121:0/3260226512 conn(0x7f99541007f0 msgr2=0x7f9954102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.785+0000 7f995a6c2700 1 -- 172.21.15.121:0/3260226512 shutdown_connections 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.785+0000 7f995a6c2700 1 -- 172.21.15.121:0/3260226512 wait complete. 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.785+0000 7f995a6c2700 1 Processor -- start 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.786+0000 7f995a6c2700 1 -- start start 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.786+0000 7f995a6c2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 0x7f995419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.786+0000 7f995a6c2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995419a570 con 0x7f9954106de0 2024-04-05T21:16:06.527 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.786+0000 7f9953fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 0x7f995419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.786+0000 7f9953fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 0x7f995419a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55976/0 (socket says 172.21.15.121:55976) 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.786+0000 7f9953fff700 1 -- 172.21.15.121:0/13159947 learned_addr learned my addr 172.21.15.121:0/13159947 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.786+0000 7f9953fff700 1 -- 172.21.15.121:0/13159947 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c009550 con 0x7f9954106de0 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.786+0000 7f9953fff700 1 --2- 172.21.15.121:0/13159947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 0x7f995419a030 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f993c009130 tx=0x7f993c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.787+0000 7f99517fa700 1 -- 172.21.15.121:0/13159947 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f993c003900 con 0x7f9954106de0 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.787+0000 7f99517fa700 1 -- 172.21.15.121:0/13159947 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f993c003a60 con 0x7f9954106de0 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.787+0000 7f99517fa700 1 -- 172.21.15.121:0/13159947 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f993c0246e0 con 0x7f9954106de0 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.787+0000 7f995a6c2700 1 -- 172.21.15.121:0/13159947 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995419a770 con 0x7f9954106de0 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.787+0000 7f995a6c2700 1 -- 172.21.15.121:0/13159947 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995419ab90 con 0x7f9954106de0 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.787+0000 7f99517fa700 1 -- 172.21.15.121:0/13159947 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f993c024ca0 con 0x7f9954106de0 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.787+0000 7f99517fa700 1 -- 172.21.15.121:0/13159947 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f993c0193c0 con 0x7f9954106de0 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.788+0000 7f995a6c2700 1 -- 172.21.15.121:0/13159947 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99540623c0 con 0x7f9954106de0 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.790+0000 7f99517fa700 1 -- 172.21.15.121:0/13159947 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f993c01ab80 con 0x7f9954106de0 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.856+0000 7f995a6c2700 1 -- 172.21.15.121:0/13159947 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f995419d5d0 con 0x7f9954106de0 2024-04-05T21:16:06.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.858+0000 7f99517fa700 1 -- 172.21.15.121:0/13159947 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f993c01a430 con 0x7f9954106de0 2024-04-05T21:16:06.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.859+0000 7f995a6c2700 1 -- 172.21.15.121:0/13159947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 msgr2=0x7f995419a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:06.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.859+0000 7f995a6c2700 1 --2- 172.21.15.121:0/13159947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 0x7f995419a030 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f993c009130 tx=0x7f993c011a30 comp rx=0 tx=0).stop 2024-04-05T21:16:06.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.859+0000 7f995a6c2700 1 -- 172.21.15.121:0/13159947 shutdown_connections 2024-04-05T21:16:06.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.859+0000 7f995a6c2700 1 --2- 172.21.15.121:0/13159947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9954106de0 0x7f995419a030 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:06.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.859+0000 7f995a6c2700 1 -- 172.21.15.121:0/13159947 >> 172.21.15.121:0/13159947 conn(0x7f99541007f0 msgr2=0x7f9954101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:06.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.859+0000 7f995a6c2700 1 -- 172.21.15.121:0/13159947 shutdown_connections 2024-04-05T21:16:06.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:05.859+0000 7f995a6c2700 1 -- 172.21.15.121:0/13159947 wait complete. 2024-04-05T21:16:06.529 INFO:teuthology.orchestra.run.smithi121.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-05T21:16:06.551 INFO:teuthology.orchestra.run.smithi121.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-05T21:16:06.551 INFO:teuthology.orchestra.run.smithi121.stdout:Creating mgr... 2024-04-05T21:16:06.551 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-05T21:16:06.552 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-05T21:16:06.552 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying port 0.0.0.0:8443 ... 2024-04-05T21:16:06.780 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mgr.smithi121.etbquh 2024-04-05T21:16:06.781 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mgr.smithi121.etbquh.service: Unit ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mgr.smithi121.etbquh.service not loaded. 2024-04-05T21:16:06.871 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4.target.wants/ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mgr.smithi121.etbquh.service → /etc/systemd/system/ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@.service. 2024-04-05T21:16:07.190 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:06 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/13159947' entity='client.admin' 2024-04-05T21:16:07.503 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-05T21:16:07.504 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout disabled 2024-04-05T21:16:07.513 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-05T21:16:07.513 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-04-05T21:16:07.513 INFO:teuthology.orchestra.run.smithi121.stdout:firewalld.service is not enabled 2024-04-05T21:16:07.513 INFO:teuthology.orchestra.run.smithi121.stdout:Not possible to enable service . firewalld.service is not available 2024-04-05T21:16:07.522 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-05T21:16:07.523 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout disabled 2024-04-05T21:16:07.531 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-05T21:16:07.532 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-04-05T21:16:07.532 INFO:teuthology.orchestra.run.smithi121.stdout:firewalld.service is not enabled 2024-04-05T21:16:07.532 INFO:teuthology.orchestra.run.smithi121.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-04-05T21:16:07.532 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mgr to start... 2024-04-05T21:16:07.532 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mgr... 2024-04-05T21:16:08.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:08 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3675013469' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-05T21:16:08.989 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-04-05T21:16:09.002 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-04-05T21:16:09.002 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsid": "9e6dd7f6-f391-11ee-b649-cb9ed24678a4", 2024-04-05T21:16:09.002 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "health": { 2024-04-05T21:16:09.002 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-05T21:16:09.002 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-05T21:16:09.002 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-05T21:16:09.002 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:09.002 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-05T21:16:09.002 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-05T21:16:09.002 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 0 2024-04-05T21:16:09.003 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-04-05T21:16:09.003 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-05T21:16:09.003 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "smithi121" 2024-04-05T21:16:09.003 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-04-05T21:16:09.003 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-05T21:16:09.003 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-05T21:16:09.003 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T21:16:09.003 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-05T21:16:09.004 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-05T21:16:09.004 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:09.004 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-05T21:16:09.004 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T21:16:09.004 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-05T21:16:09.004 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-05T21:16:09.004 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-05T21:16:09.004 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-05T21:16:09.004 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-05T21:16:09.004 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-05T21:16:09.004 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:09.005 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-05T21:16:09.005 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-05T21:16:09.005 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-05T21:16:09.005 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-05T21:16:09.005 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-05T21:16:09.005 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-05T21:16:09.005 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-05T21:16:09.005 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-05T21:16:09.005 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-05T21:16:09.005 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:09.006 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-05T21:16:09.006 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T21:16:09.006 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-05T21:16:09.006 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-05T21:16:09.006 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:09.006 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-05T21:16:09.006 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-05T21:16:09.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-05T21:16:09.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-05T21:16:09.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-05T21:16:09.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-05T21:16:09.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "restful" 2024-04-05T21:16:09.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-04-05T21:16:09.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T21:16:09.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:09.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-05T21:16:09.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T21:16:09.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modified": "2024-04-05T21:15:58.797545+0000", 2024-04-05T21:16:09.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T21:16:09.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:09.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-05T21:16:09.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-04-05T21:16:09.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.189+0000 7f2490ba1700 1 Processor -- start 2024-04-05T21:16:09.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.189+0000 7f2490ba1700 1 -- start start 2024-04-05T21:16:09.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.189+0000 7f2490ba1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 0x7f248c106580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:09.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.189+0000 7f2490ba1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c106b50 con 0x7f248c106160 2024-04-05T21:16:09.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.190+0000 7f248b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 0x7f248c106580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:09.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.190+0000 7f248b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 0x7f248c106580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56020/0 (socket says 172.21.15.121:56020) 2024-04-05T21:16:09.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.190+0000 7f248b7fe700 1 -- 172.21.15.121:0/3007251045 learned_addr learned my addr 172.21.15.121:0/3007251045 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:09.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.190+0000 7f248b7fe700 1 -- 172.21.15.121:0/3007251045 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c107360 con 0x7f248c106160 2024-04-05T21:16:09.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.190+0000 7f248b7fe700 1 --2- 172.21.15.121:0/3007251045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 0x7f248c106580 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f247c0098a0 tx=0x7f247c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58c3afffcbd7989 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:09.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.190+0000 7f248a7fc700 1 -- 172.21.15.121:0/3007251045 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f247c004180 con 0x7f248c106160 2024-04-05T21:16:09.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.190+0000 7f248a7fc700 1 -- 172.21.15.121:0/3007251045 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f247c0042e0 con 0x7f248c106160 2024-04-05T21:16:09.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.190+0000 7f248a7fc700 1 -- 172.21.15.121:0/3007251045 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f247c0119d0 con 0x7f248c106160 2024-04-05T21:16:09.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.191+0000 7f2490ba1700 1 -- 172.21.15.121:0/3007251045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 msgr2=0x7f248c106580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:09.011 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.191+0000 7f2490ba1700 1 --2- 172.21.15.121:0/3007251045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 0x7f248c106580 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f247c0098a0 tx=0x7f247c009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:09.011 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.191+0000 7f2490ba1700 1 -- 172.21.15.121:0/3007251045 shutdown_connections 2024-04-05T21:16:09.011 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.191+0000 7f2490ba1700 1 --2- 172.21.15.121:0/3007251045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 0x7f248c106580 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:09.011 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.191+0000 7f2490ba1700 1 -- 172.21.15.121:0/3007251045 >> 172.21.15.121:0/3007251045 conn(0x7f248c101700 msgr2=0x7f248c103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:09.011 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.191+0000 7f2490ba1700 1 -- 172.21.15.121:0/3007251045 shutdown_connections 2024-04-05T21:16:09.011 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.191+0000 7f2490ba1700 1 -- 172.21.15.121:0/3007251045 wait complete. 2024-04-05T21:16:09.011 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.192+0000 7f2490ba1700 1 Processor -- start 2024-04-05T21:16:09.011 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.192+0000 7f2490ba1700 1 -- start start 2024-04-05T21:16:09.011 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.192+0000 7f2490ba1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 0x7f248c19c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:09.011 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.192+0000 7f2490ba1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c19c5f0 con 0x7f248c106160 2024-04-05T21:16:09.011 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.192+0000 7f248b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 0x7f248c19c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:09.011 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.192+0000 7f248b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 0x7f248c19c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56036/0 (socket says 172.21.15.121:56036) 2024-04-05T21:16:09.012 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.192+0000 7f248b7fe700 1 -- 172.21.15.121:0/3675013469 learned_addr learned my addr 172.21.15.121:0/3675013469 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:09.012 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.193+0000 7f248b7fe700 1 -- 172.21.15.121:0/3675013469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c009550 con 0x7f248c106160 2024-04-05T21:16:09.012 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.193+0000 7f248b7fe700 1 --2- 172.21.15.121:0/3675013469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 0x7f248c19c0b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f247c009130 tx=0x7f247c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:09.012 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.193+0000 7f2488ff9700 1 -- 172.21.15.121:0/3675013469 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f247c003900 con 0x7f248c106160 2024-04-05T21:16:09.012 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.193+0000 7f2490ba1700 1 -- 172.21.15.121:0/3675013469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248c19c7f0 con 0x7f248c106160 2024-04-05T21:16:09.012 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.193+0000 7f2488ff9700 1 -- 172.21.15.121:0/3675013469 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f247c003a60 con 0x7f248c106160 2024-04-05T21:16:09.012 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.193+0000 7f2490ba1700 1 -- 172.21.15.121:0/3675013469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248c19cc10 con 0x7f248c106160 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.193+0000 7f2488ff9700 1 -- 172.21.15.121:0/3675013469 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f247c0246e0 con 0x7f248c106160 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.193+0000 7f2488ff9700 1 -- 172.21.15.121:0/3675013469 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f247c024cd0 con 0x7f248c106160 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.193+0000 7f2488ff9700 1 -- 172.21.15.121:0/3675013469 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f247c0193c0 con 0x7f248c106160 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.194+0000 7f2490ba1700 1 -- 172.21.15.121:0/3675013469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2478005320 con 0x7f248c106160 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.198+0000 7f2488ff9700 1 -- 172.21.15.121:0/3675013469 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f247c01ab80 con 0x7f248c106160 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.270+0000 7f2490ba1700 1 -- 172.21.15.121:0/3675013469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2478005cc0 con 0x7f248c106160 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.270+0000 7f2488ff9700 1 -- 172.21.15.121:0/3675013469 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f247c01a430 con 0x7f248c106160 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.271+0000 7f2490ba1700 1 -- 172.21.15.121:0/3675013469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 msgr2=0x7f248c19c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.271+0000 7f2490ba1700 1 --2- 172.21.15.121:0/3675013469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 0x7f248c19c0b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f247c009130 tx=0x7f247c011a30 comp rx=0 tx=0).stop 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.271+0000 7f2490ba1700 1 -- 172.21.15.121:0/3675013469 shutdown_connections 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.271+0000 7f2490ba1700 1 --2- 172.21.15.121:0/3675013469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c106160 0x7f248c19c0b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.271+0000 7f2490ba1700 1 -- 172.21.15.121:0/3675013469 >> 172.21.15.121:0/3675013469 conn(0x7f248c101700 msgr2=0x7f248c102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.271+0000 7f2490ba1700 1 -- 172.21.15.121:0/3675013469 shutdown_connections 2024-04-05T21:16:09.013 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:08.271+0000 7f2490ba1700 1 -- 172.21.15.121:0/3675013469 wait complete. 2024-04-05T21:16:09.014 INFO:teuthology.orchestra.run.smithi121.stdout:mgr not available, waiting (1/15)... 2024-04-05T21:16:14.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:14 smithi121 ceph-mon[39723]: Activating manager daemon smithi121.etbquh 2024-04-05T21:16:14.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:14 smithi121 ceph-mon[39723]: mgrmap e2: smithi121.etbquh(active, starting, since 0.00282941s) 2024-04-05T21:16:14.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:14 smithi121 ceph-mon[39723]: from='mgr.14100 172.21.15.121:0/1981011216' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T21:16:14.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:14 smithi121 ceph-mon[39723]: from='mgr.14100 172.21.15.121:0/1981011216' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T21:16:14.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:14 smithi121 ceph-mon[39723]: from='mgr.14100 172.21.15.121:0/1981011216' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T21:16:14.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:14 smithi121 ceph-mon[39723]: from='mgr.14100 172.21.15.121:0/1981011216' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:16:14.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:14 smithi121 ceph-mon[39723]: from='mgr.14100 172.21.15.121:0/1981011216' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr metadata", "who": "smithi121.etbquh", "id": "smithi121.etbquh"}]: dispatch 2024-04-05T21:16:14.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:14 smithi121 ceph-mon[39723]: Manager daemon smithi121.etbquh is now available 2024-04-05T21:16:14.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:14 smithi121 ceph-mon[39723]: from='mgr.14100 172.21.15.121:0/1981011216' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.etbquh/mirror_snapshot_schedule"}]: dispatch 2024-04-05T21:16:14.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:14 smithi121 ceph-mon[39723]: from='mgr.14100 172.21.15.121:0/1981011216' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.etbquh/trash_purge_schedule"}]: dispatch 2024-04-05T21:16:14.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:14 smithi121 ceph-mon[39723]: from='mgr.14100 172.21.15.121:0/1981011216' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:14.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:14 smithi121 ceph-mon[39723]: from='mgr.14100 172.21.15.121:0/1981011216' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:14.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:14 smithi121 ceph-mon[39723]: from='mgr.14100 172.21.15.121:0/1981011216' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:15 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/4081135449' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-05T21:16:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:15 smithi121 ceph-mon[39723]: mgrmap e3: smithi121.etbquh(active, since 1.00717s) 2024-04-05T21:16:17.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:17 smithi121 ceph-mon[39723]: mgrmap e4: smithi121.etbquh(active, since 2s) 2024-04-05T21:16:20.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-04-05T21:16:20.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-04-05T21:16:20.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsid": "9e6dd7f6-f391-11ee-b649-cb9ed24678a4", 2024-04-05T21:16:20.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "health": { 2024-04-05T21:16:20.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-05T21:16:20.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-05T21:16:20.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 0 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "smithi121" 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:20.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-05T21:16:20.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T21:16:20.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-05T21:16:20.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-05T21:16:20.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-05T21:16:20.544 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-05T21:16:20.544 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-05T21:16:20.544 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-05T21:16:20.544 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:20.544 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-05T21:16:20.544 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-05T21:16:20.544 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-05T21:16:20.544 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-05T21:16:20.544 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-05T21:16:20.544 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-05T21:16:20.545 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-05T21:16:20.545 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-05T21:16:20.545 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-05T21:16:20.545 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:20.545 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-05T21:16:20.545 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T21:16:20.545 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-05T21:16:20.545 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-05T21:16:20.545 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:20.545 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-05T21:16:20.545 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-05T21:16:20.545 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-05T21:16:20.546 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-05T21:16:20.546 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-05T21:16:20.546 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-05T21:16:20.546 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "restful" 2024-04-05T21:16:20.546 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-04-05T21:16:20.546 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T21:16:20.546 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:20.546 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modified": "2024-04-05T21:15:58.797545+0000", 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.392+0000 7fd08dafb700 1 Processor -- start 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.392+0000 7fd08dafb700 1 -- start start 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.392+0000 7fd08dafb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 0x7fd088106730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.392+0000 7fd08dafb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd088106d00 con 0x7fd088106310 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.392+0000 7fd0877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 0x7fd088106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.392+0000 7fd0877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 0x7fd088106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39118/0 (socket says 172.21.15.121:39118) 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.392+0000 7fd0877fe700 1 -- 172.21.15.121:0/1321333400 learned_addr learned my addr 172.21.15.121:0/1321333400 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.392+0000 7fd0877fe700 1 -- 172.21.15.121:0/1321333400 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd088107510 con 0x7fd088106310 2024-04-05T21:16:20.547 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.392+0000 7fd0877fe700 1 --2- 172.21.15.121:0/1321333400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 0x7fd088106730 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd0780098a0 tx=0x7fd078009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193babfbd4420661 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:20.548 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.392+0000 7fd0867fc700 1 -- 172.21.15.121:0/1321333400 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd078004180 con 0x7fd088106310 2024-04-05T21:16:20.548 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.393+0000 7fd0867fc700 1 -- 172.21.15.121:0/1321333400 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7fd0780042e0 con 0x7fd088106310 2024-04-05T21:16:20.548 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.393+0000 7fd0867fc700 1 -- 172.21.15.121:0/1321333400 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0780119d0 con 0x7fd088106310 2024-04-05T21:16:20.548 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.393+0000 7fd08dafb700 1 -- 172.21.15.121:0/1321333400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 msgr2=0x7fd088106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:20.548 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.393+0000 7fd08dafb700 1 --2- 172.21.15.121:0/1321333400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 0x7fd088106730 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd0780098a0 tx=0x7fd078009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:20.548 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.393+0000 7fd08dafb700 1 -- 172.21.15.121:0/1321333400 shutdown_connections 2024-04-05T21:16:20.548 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.393+0000 7fd08dafb700 1 --2- 172.21.15.121:0/1321333400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 0x7fd088106730 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:20.548 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.393+0000 7fd08dafb700 1 -- 172.21.15.121:0/1321333400 >> 172.21.15.121:0/1321333400 conn(0x7fd0881018d0 msgr2=0x7fd088103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:20.548 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.393+0000 7fd08dafb700 1 -- 172.21.15.121:0/1321333400 shutdown_connections 2024-04-05T21:16:20.548 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.393+0000 7fd08dafb700 1 -- 172.21.15.121:0/1321333400 wait complete. 2024-04-05T21:16:20.548 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.394+0000 7fd08dafb700 1 Processor -- start 2024-04-05T21:16:20.549 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.394+0000 7fd08dafb700 1 -- start start 2024-04-05T21:16:20.549 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.394+0000 7fd08dafb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 0x7fd08819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:20.549 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.394+0000 7fd08dafb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08819a550 con 0x7fd088106310 2024-04-05T21:16:20.549 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.394+0000 7fd0877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 0x7fd08819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:20.549 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.394+0000 7fd0877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 0x7fd08819a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39124/0 (socket says 172.21.15.121:39124) 2024-04-05T21:16:20.549 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.394+0000 7fd0877fe700 1 -- 172.21.15.121:0/4081135449 learned_addr learned my addr 172.21.15.121:0/4081135449 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:20.549 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.394+0000 7fd0877fe700 1 -- 172.21.15.121:0/4081135449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd078009550 con 0x7fd088106310 2024-04-05T21:16:20.549 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.394+0000 7fd0877fe700 1 --2- 172.21.15.121:0/4081135449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 0x7fd08819a010 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd078006af0 tx=0x7fd078011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:20.549 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.394+0000 7fd084ff9700 1 -- 172.21.15.121:0/4081135449 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd078024430 con 0x7fd088106310 2024-04-05T21:16:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.395+0000 7fd084ff9700 1 -- 172.21.15.121:0/4081135449 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7fd0780249e0 con 0x7fd088106310 2024-04-05T21:16:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.395+0000 7fd08dafb700 1 -- 172.21.15.121:0/4081135449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd08819a750 con 0x7fd088106310 2024-04-05T21:16:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.395+0000 7fd084ff9700 1 -- 172.21.15.121:0/4081135449 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd07801a970 con 0x7fd088106310 2024-04-05T21:16:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.395+0000 7fd08dafb700 1 -- 172.21.15.121:0/4081135449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08819ab70 con 0x7fd088106310 2024-04-05T21:16:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.395+0000 7fd084ff9700 1 -- 172.21.15.121:0/4081135449 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 2) v1 ==== 45039+0+0 (secure 0 0 0) 0x7fd07801ac30 con 0x7fd088106310 2024-04-05T21:16:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.395+0000 7fd084ff9700 1 -- 172.21.15.121:0/4081135449 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd07804bfb0 con 0x7fd088106310 2024-04-05T21:16:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.395+0000 7fd08dafb700 1 -- 172.21.15.121:0/4081135449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0880623c0 con 0x7fd088106310 2024-04-05T21:16:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.398+0000 7fd084ff9700 1 -- 172.21.15.121:0/4081135449 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fd0780193c0 con 0x7fd088106310 2024-04-05T21:16:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.469+0000 7fd08dafb700 1 -- 172.21.15.121:0/4081135449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd08810a710 con 0x7fd088106310 2024-04-05T21:16:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.469+0000 7fd084ff9700 1 -- 172.21.15.121:0/4081135449 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fd078029210 con 0x7fd088106310 2024-04-05T21:16:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.470+0000 7fd08dafb700 1 -- 172.21.15.121:0/4081135449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 msgr2=0x7fd08819a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:20.550 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.470+0000 7fd08dafb700 1 --2- 172.21.15.121:0/4081135449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 0x7fd08819a010 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd078006af0 tx=0x7fd078011a30 comp rx=0 tx=0).stop 2024-04-05T21:16:20.551 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.470+0000 7fd08dafb700 1 -- 172.21.15.121:0/4081135449 shutdown_connections 2024-04-05T21:16:20.551 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.470+0000 7fd08dafb700 1 --2- 172.21.15.121:0/4081135449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd088106310 0x7fd08819a010 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:20.551 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.470+0000 7fd08dafb700 1 -- 172.21.15.121:0/4081135449 >> 172.21.15.121:0/4081135449 conn(0x7fd0881018d0 msgr2=0x7fd088102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:20.551 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.470+0000 7fd08dafb700 1 -- 172.21.15.121:0/4081135449 shutdown_connections 2024-04-05T21:16:20.551 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:14.470+0000 7fd08dafb700 1 -- 172.21.15.121:0/4081135449 wait complete. 2024-04-05T21:16:20.551 INFO:teuthology.orchestra.run.smithi121.stdout:mgr not available, waiting (2/15)... 2024-04-05T21:16:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:23 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/373574829' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-05T21:16:24.101 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-04-05T21:16:24.102 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-04-05T21:16:24.102 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsid": "9e6dd7f6-f391-11ee-b649-cb9ed24678a4", 2024-04-05T21:16:24.102 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "health": { 2024-04-05T21:16:24.102 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-05T21:16:24.102 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-05T21:16:24.102 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-05T21:16:24.102 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:24.102 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-05T21:16:24.102 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-05T21:16:24.102 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 0 2024-04-05T21:16:24.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-04-05T21:16:24.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-05T21:16:24.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "smithi121" 2024-04-05T21:16:24.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-04-05T21:16:24.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_age": 18, 2024-04-05T21:16:24.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-05T21:16:24.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T21:16:24.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-04-05T21:16:24.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-05T21:16:24.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:24.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-05T21:16:24.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T21:16:24.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-05T21:16:24.104 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-05T21:16:24.104 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-05T21:16:24.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-05T21:16:24.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-05T21:16:24.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-05T21:16:24.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:24.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-05T21:16:24.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-05T21:16:24.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-05T21:16:24.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-05T21:16:24.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-05T21:16:24.116 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-05T21:16:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-05T21:16:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-05T21:16:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-05T21:16:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-05T21:16:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T21:16:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-05T21:16:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-05T21:16:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-05T21:16:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-05T21:16:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-05T21:16:24.117 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-05T21:16:24.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-05T21:16:24.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-05T21:16:24.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "restful" 2024-04-05T21:16:24.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-04-05T21:16:24.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T21:16:24.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:24.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-05T21:16:24.118 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-05T21:16:24.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modified": "2024-04-05T21:15:58.797545+0000", 2024-04-05T21:16:24.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-05T21:16:24.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-04-05T21:16:24.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-05T21:16:24.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-04-05T21:16:24.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.178+0000 7f2b7e383700 1 Processor -- start 2024-04-05T21:16:24.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.178+0000 7f2b7e383700 1 -- start start 2024-04-05T21:16:24.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.178+0000 7f2b7e383700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 0x7f2b781091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:24.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.178+0000 7f2b7e383700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b780744d0 con 0x7f2b78106e00 2024-04-05T21:16:24.119 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.178+0000 7f2b77fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 0x7f2b781091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:24.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.178+0000 7f2b77fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 0x7f2b781091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39132/0 (socket says 172.21.15.121:39132) 2024-04-05T21:16:24.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.178+0000 7f2b77fff700 1 -- 172.21.15.121:0/1159780876 learned_addr learned my addr 172.21.15.121:0/1159780876 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:24.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.179+0000 7f2b77fff700 1 -- 172.21.15.121:0/1159780876 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b78109730 con 0x7f2b78106e00 2024-04-05T21:16:24.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.179+0000 7f2b77fff700 1 --2- 172.21.15.121:0/1159780876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 0x7f2b781091f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b600098a0 tx=0x7f2b60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6af190a68971e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:24.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.179+0000 7f2b76ffd700 1 -- 172.21.15.121:0/1159780876 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b60004180 con 0x7f2b78106e00 2024-04-05T21:16:24.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.179+0000 7f2b76ffd700 1 -- 172.21.15.121:0/1159780876 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f2b600042e0 con 0x7f2b78106e00 2024-04-05T21:16:24.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.179+0000 7f2b76ffd700 1 -- 172.21.15.121:0/1159780876 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b600119d0 con 0x7f2b78106e00 2024-04-05T21:16:24.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.179+0000 7f2b7e383700 1 -- 172.21.15.121:0/1159780876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 msgr2=0x7f2b781091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:24.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.179+0000 7f2b7e383700 1 --2- 172.21.15.121:0/1159780876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 0x7f2b781091f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b600098a0 tx=0x7f2b60009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:24.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.179+0000 7f2b7e383700 1 -- 172.21.15.121:0/1159780876 shutdown_connections 2024-04-05T21:16:24.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.179+0000 7f2b7e383700 1 --2- 172.21.15.121:0/1159780876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 0x7f2b781091f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:24.120 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.180+0000 7f2b7e383700 1 -- 172.21.15.121:0/1159780876 >> 172.21.15.121:0/1159780876 conn(0x7f2b781007f0 msgr2=0x7f2b78102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:24.121 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.180+0000 7f2b7e383700 1 -- 172.21.15.121:0/1159780876 shutdown_connections 2024-04-05T21:16:24.121 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.180+0000 7f2b7e383700 1 -- 172.21.15.121:0/1159780876 wait complete. 2024-04-05T21:16:24.121 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.180+0000 7f2b7e383700 1 Processor -- start 2024-04-05T21:16:24.121 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.180+0000 7f2b7e383700 1 -- start start 2024-04-05T21:16:24.121 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.180+0000 7f2b7e383700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 0x7f2b7819a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:24.121 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.180+0000 7f2b7e383700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7819a560 con 0x7f2b78106e00 2024-04-05T21:16:24.121 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.181+0000 7f2b77fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 0x7f2b7819a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:24.121 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.181+0000 7f2b77fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 0x7f2b7819a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39142/0 (socket says 172.21.15.121:39142) 2024-04-05T21:16:24.121 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.181+0000 7f2b77fff700 1 -- 172.21.15.121:0/373574829 learned_addr learned my addr 172.21.15.121:0/373574829 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:24.121 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.181+0000 7f2b77fff700 1 -- 172.21.15.121:0/373574829 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b60009550 con 0x7f2b78106e00 2024-04-05T21:16:24.121 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.181+0000 7f2b77fff700 1 --2- 172.21.15.121:0/373574829 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 0x7f2b7819a020 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009130 tx=0x7f2b60011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:24.122 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.181+0000 7f2b757fa700 1 -- 172.21.15.121:0/373574829 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b60003900 con 0x7f2b78106e00 2024-04-05T21:16:24.122 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.181+0000 7f2b757fa700 1 -- 172.21.15.121:0/373574829 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f2b60003a60 con 0x7f2b78106e00 2024-04-05T21:16:24.122 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.181+0000 7f2b7e383700 1 -- 172.21.15.121:0/373574829 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7819a760 con 0x7f2b78106e00 2024-04-05T21:16:24.122 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.181+0000 7f2b757fa700 1 -- 172.21.15.121:0/373574829 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b600246e0 con 0x7f2b78106e00 2024-04-05T21:16:24.122 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.181+0000 7f2b7e383700 1 -- 172.21.15.121:0/373574829 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7819ab80 con 0x7f2b78106e00 2024-04-05T21:16:24.122 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.182+0000 7f2b757fa700 1 -- 172.21.15.121:0/373574829 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f2b6002d3f0 con 0x7f2b78106e00 2024-04-05T21:16:24.122 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.182+0000 7f2b757fa700 1 --2- 172.21.15.121:0/373574829 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f2b640382d0 0x7f2b6403a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:24.122 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.182+0000 7f2b757fa700 1 -- 172.21.15.121:0/373574829 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2b6004c190 con 0x7f2b78106e00 2024-04-05T21:16:24.122 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.182+0000 7f2b777fe700 1 --2- 172.21.15.121:0/373574829 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f2b640382d0 0x7f2b6403a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:24.122 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.182+0000 7f2b7e383700 1 -- 172.21.15.121:0/373574829 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b780623c0 con 0x7f2b78106e00 2024-04-05T21:16:24.123 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.183+0000 7f2b777fe700 1 --2- 172.21.15.121:0/373574829 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f2b640382d0 0x7f2b6403a790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2b68006fd0 tx=0x7f2b68006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:24.123 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.187+0000 7f2b757fa700 1 -- 172.21.15.121:0/373574829 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2b60010970 con 0x7f2b78106e00 2024-04-05T21:16:24.123 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.451+0000 7f2b7e383700 1 -- 172.21.15.121:0/373574829 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2b7819d6b0 con 0x7f2b78106e00 2024-04-05T21:16:24.123 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.452+0000 7f2b757fa700 1 -- 172.21.15.121:0/373574829 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f2b6000b3e0 con 0x7f2b78106e00 2024-04-05T21:16:24.123 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.454+0000 7f2b7e383700 1 -- 172.21.15.121:0/373574829 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f2b640382d0 msgr2=0x7f2b6403a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:24.123 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.454+0000 7f2b7e383700 1 --2- 172.21.15.121:0/373574829 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f2b640382d0 0x7f2b6403a790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2b68006fd0 tx=0x7f2b68006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:24.123 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.454+0000 7f2b7e383700 1 -- 172.21.15.121:0/373574829 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 msgr2=0x7f2b7819a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:24.123 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.454+0000 7f2b7e383700 1 --2- 172.21.15.121:0/373574829 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 0x7f2b7819a020 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009130 tx=0x7f2b60011a30 comp rx=0 tx=0).stop 2024-04-05T21:16:24.123 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.454+0000 7f2b7e383700 1 -- 172.21.15.121:0/373574829 shutdown_connections 2024-04-05T21:16:24.123 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.454+0000 7f2b7e383700 1 --2- 172.21.15.121:0/373574829 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f2b640382d0 0x7f2b6403a790 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:24.123 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.454+0000 7f2b7e383700 1 --2- 172.21.15.121:0/373574829 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b78106e00 0x7f2b7819a020 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:24.123 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.454+0000 7f2b7e383700 1 -- 172.21.15.121:0/373574829 >> 172.21.15.121:0/373574829 conn(0x7f2b781007f0 msgr2=0x7f2b78101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:24.124 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.454+0000 7f2b7e383700 1 -- 172.21.15.121:0/373574829 shutdown_connections 2024-04-05T21:16:24.124 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:23.454+0000 7f2b7e383700 1 -- 172.21.15.121:0/373574829 wait complete. 2024-04-05T21:16:24.124 INFO:teuthology.orchestra.run.smithi121.stdout:mgr is available 2024-04-05T21:16:25.165 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:24 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2642857458' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-05T21:16:25.453 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-04-05T21:16:25.453 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [global] 2024-04-05T21:16:25.453 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout fsid = 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:16:25.453 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-05T21:16:25.454 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-05T21:16:25.454 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-05T21:16:25.454 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-05T21:16:25.454 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-04-05T21:16:25.454 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-05T21:16:25.454 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-05T21:16:25.454 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-04-05T21:16:25.454 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [osd] 2024-04-05T21:16:25.454 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-05T21:16:25.454 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-05T21:16:25.454 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-05T21:16:25.454 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-05T21:16:25.454 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.609+0000 7fa8f9389700 1 Processor -- start 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.609+0000 7fa8f9389700 1 -- start start 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.609+0000 7fa8f9389700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 0x7fa8f41091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.609+0000 7fa8f9389700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4079da0 con 0x7fa8f4106de0 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.609+0000 7fa8f2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 0x7fa8f41091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.609+0000 7fa8f2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 0x7fa8f41091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52524/0 (socket says 172.21.15.121:52524) 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.609+0000 7fa8f2ffd700 1 -- 172.21.15.121:0/1755488074 learned_addr learned my addr 172.21.15.121:0/1755488074 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.609+0000 7fa8f2ffd700 1 -- 172.21.15.121:0/1755488074 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f4109710 con 0x7fa8f4106de0 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.609+0000 7fa8f2ffd700 1 --2- 172.21.15.121:0/1755488074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 0x7fa8f41091d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa8e40098a0 tx=0x7fa8e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf2e173b396026d server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.610+0000 7fa8f1ffb700 1 -- 172.21.15.121:0/1755488074 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8e4004180 con 0x7fa8f4106de0 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.610+0000 7fa8f1ffb700 1 -- 172.21.15.121:0/1755488074 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7fa8e40042e0 con 0x7fa8f4106de0 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.610+0000 7fa8f1ffb700 1 -- 172.21.15.121:0/1755488074 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8e40119d0 con 0x7fa8f4106de0 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.610+0000 7fa8f9389700 1 -- 172.21.15.121:0/1755488074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 msgr2=0x7fa8f41091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:25.455 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.610+0000 7fa8f9389700 1 --2- 172.21.15.121:0/1755488074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 0x7fa8f41091d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa8e40098a0 tx=0x7fa8e4009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.610+0000 7fa8f9389700 1 -- 172.21.15.121:0/1755488074 shutdown_connections 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.610+0000 7fa8f9389700 1 --2- 172.21.15.121:0/1755488074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 0x7fa8f41091d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.610+0000 7fa8f9389700 1 -- 172.21.15.121:0/1755488074 >> 172.21.15.121:0/1755488074 conn(0x7fa8f41007d0 msgr2=0x7fa8f4102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.610+0000 7fa8f9389700 1 -- 172.21.15.121:0/1755488074 shutdown_connections 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.610+0000 7fa8f9389700 1 -- 172.21.15.121:0/1755488074 wait complete. 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.611+0000 7fa8f9389700 1 Processor -- start 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.611+0000 7fa8f9389700 1 -- start start 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.611+0000 7fa8f9389700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 0x7fa8f4106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.611+0000 7fa8f9389700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4106c80 con 0x7fa8f4106de0 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.611+0000 7fa8f2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 0x7fa8f4106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.611+0000 7fa8f2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 0x7fa8f4106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52532/0 (socket says 172.21.15.121:52532) 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.611+0000 7fa8f2ffd700 1 -- 172.21.15.121:0/2642857458 learned_addr learned my addr 172.21.15.121:0/2642857458 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.611+0000 7fa8f2ffd700 1 -- 172.21.15.121:0/2642857458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e4009550 con 0x7fa8f4106de0 2024-04-05T21:16:25.456 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.612+0000 7fa8f2ffd700 1 --2- 172.21.15.121:0/2642857458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 0x7fa8f4106740 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4011980 tx=0x7fa8e4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.612+0000 7fa8dbfff700 1 -- 172.21.15.121:0/2642857458 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8e4003b40 con 0x7fa8f4106de0 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.612+0000 7fa8dbfff700 1 -- 172.21.15.121:0/2642857458 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7fa8e401a880 con 0x7fa8f4106de0 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.612+0000 7fa8f9389700 1 -- 172.21.15.121:0/2642857458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f4104d90 con 0x7fa8f4106de0 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.612+0000 7fa8dbfff700 1 -- 172.21.15.121:0/2642857458 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8e4019910 con 0x7fa8f4106de0 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.612+0000 7fa8f9389700 1 -- 172.21.15.121:0/2642857458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f41051b0 con 0x7fa8f4106de0 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.612+0000 7fa8dbfff700 1 -- 172.21.15.121:0/2642857458 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7fa8e4019b80 con 0x7fa8f4106de0 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.613+0000 7fa8dbfff700 1 --2- 172.21.15.121:0/2642857458 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fa8dc0382e0 0x7fa8dc03a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.613+0000 7fa8dbfff700 1 -- 172.21.15.121:0/2642857458 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa8e404c690 con 0x7fa8f4106de0 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.613+0000 7fa8f9389700 1 -- 172.21.15.121:0/2642857458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f410a400 con 0x7fa8f4106de0 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.613+0000 7fa8f27fc700 1 --2- 172.21.15.121:0/2642857458 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fa8dc0382e0 0x7fa8dc03a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.613+0000 7fa8f27fc700 1 --2- 172.21.15.121:0/2642857458 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fa8dc0382e0 0x7fa8dc03a7a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8006fd0 tx=0x7fa8e8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.618+0000 7fa8dbfff700 1 -- 172.21.15.121:0/2642857458 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa8e402c3a0 con 0x7fa8f4106de0 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.801+0000 7fa8f9389700 1 -- 172.21.15.121:0/2642857458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa8f410a5f0 con 0x7fa8f4106de0 2024-04-05T21:16:25.457 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.802+0000 7fa8dbfff700 1 -- 172.21.15.121:0/2642857458 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fa8e40172b0 con 0x7fa8f4106de0 2024-04-05T21:16:25.458 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.804+0000 7fa8f9389700 1 -- 172.21.15.121:0/2642857458 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fa8dc0382e0 msgr2=0x7fa8dc03a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:25.458 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.804+0000 7fa8f9389700 1 --2- 172.21.15.121:0/2642857458 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fa8dc0382e0 0x7fa8dc03a7a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8006fd0 tx=0x7fa8e8006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:25.458 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.804+0000 7fa8f9389700 1 -- 172.21.15.121:0/2642857458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 msgr2=0x7fa8f4106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:25.458 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.804+0000 7fa8f9389700 1 --2- 172.21.15.121:0/2642857458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 0x7fa8f4106740 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4011980 tx=0x7fa8e4011a60 comp rx=0 tx=0).stop 2024-04-05T21:16:25.458 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.804+0000 7fa8f9389700 1 -- 172.21.15.121:0/2642857458 shutdown_connections 2024-04-05T21:16:25.458 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.804+0000 7fa8f9389700 1 --2- 172.21.15.121:0/2642857458 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fa8dc0382e0 0x7fa8dc03a7a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:25.458 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.804+0000 7fa8f9389700 1 --2- 172.21.15.121:0/2642857458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8f4106de0 0x7fa8f4106740 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:25.458 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.804+0000 7fa8f9389700 1 -- 172.21.15.121:0/2642857458 >> 172.21.15.121:0/2642857458 conn(0x7fa8f41007d0 msgr2=0x7fa8f4101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:25.458 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.804+0000 7fa8f9389700 1 -- 172.21.15.121:0/2642857458 shutdown_connections 2024-04-05T21:16:25.458 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:24.804+0000 7fa8f9389700 1 -- 172.21.15.121:0/2642857458 wait complete. 2024-04-05T21:16:25.458 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling cephadm module... 2024-04-05T21:16:26.703 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:26 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3733794967' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-05T21:16:27.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.075+0000 7fa367650700 1 Processor -- start 2024-04-05T21:16:27.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.075+0000 7fa367650700 1 -- start start 2024-04-05T21:16:27.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.075+0000 7fa367650700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 0x7fa3601091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:27.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.075+0000 7fa367650700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa360079da0 con 0x7fa360106de0 2024-04-05T21:16:27.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.075+0000 7fa3653ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 0x7fa3601091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:27.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.075+0000 7fa3653ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 0x7fa3601091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52548/0 (socket says 172.21.15.121:52548) 2024-04-05T21:16:27.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.075+0000 7fa3653ec700 1 -- 172.21.15.121:0/539999958 learned_addr learned my addr 172.21.15.121:0/539999958 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:27.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.076+0000 7fa3653ec700 1 -- 172.21.15.121:0/539999958 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa360109710 con 0x7fa360106de0 2024-04-05T21:16:27.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.076+0000 7fa3653ec700 1 --2- 172.21.15.121:0/539999958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 0x7fa3601091d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0098a0 tx=0x7fa35c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d17768a8a7465a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:27.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.076+0000 7fa357fff700 1 -- 172.21.15.121:0/539999958 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa35c004180 con 0x7fa360106de0 2024-04-05T21:16:27.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.076+0000 7fa357fff700 1 -- 172.21.15.121:0/539999958 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7fa35c0042e0 con 0x7fa360106de0 2024-04-05T21:16:27.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.076+0000 7fa357fff700 1 -- 172.21.15.121:0/539999958 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa35c0119d0 con 0x7fa360106de0 2024-04-05T21:16:27.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.076+0000 7fa367650700 1 -- 172.21.15.121:0/539999958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 msgr2=0x7fa3601091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:27.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.076+0000 7fa367650700 1 --2- 172.21.15.121:0/539999958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 0x7fa3601091d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0098a0 tx=0x7fa35c009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:27.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.076+0000 7fa367650700 1 -- 172.21.15.121:0/539999958 shutdown_connections 2024-04-05T21:16:27.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.076+0000 7fa367650700 1 --2- 172.21.15.121:0/539999958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 0x7fa3601091d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:27.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.076+0000 7fa367650700 1 -- 172.21.15.121:0/539999958 >> 172.21.15.121:0/539999958 conn(0x7fa3601007d0 msgr2=0x7fa360102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:27.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.076+0000 7fa367650700 1 -- 172.21.15.121:0/539999958 shutdown_connections 2024-04-05T21:16:27.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.077+0000 7fa367650700 1 -- 172.21.15.121:0/539999958 wait complete. 2024-04-05T21:16:27.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.077+0000 7fa367650700 1 Processor -- start 2024-04-05T21:16:27.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.077+0000 7fa367650700 1 -- start start 2024-04-05T21:16:27.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.077+0000 7fa367650700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 0x7fa360199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:27.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.077+0000 7fa367650700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36019a520 con 0x7fa360106de0 2024-04-05T21:16:27.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.078+0000 7fa3653ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 0x7fa360199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:27.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.078+0000 7fa3653ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 0x7fa360199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52562/0 (socket says 172.21.15.121:52562) 2024-04-05T21:16:27.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.078+0000 7fa3653ec700 1 -- 172.21.15.121:0/3733794967 learned_addr learned my addr 172.21.15.121:0/3733794967 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:27.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.078+0000 7fa3653ec700 1 -- 172.21.15.121:0/3733794967 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c009550 con 0x7fa360106de0 2024-04-05T21:16:27.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.078+0000 7fa3653ec700 1 --2- 172.21.15.121:0/3733794967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 0x7fa360199fe0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa35c006af0 tx=0x7fa35c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:27.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.078+0000 7fa3567fc700 1 -- 172.21.15.121:0/3733794967 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa35c024430 con 0x7fa360106de0 2024-04-05T21:16:27.063 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.078+0000 7fa3567fc700 1 -- 172.21.15.121:0/3733794967 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7fa35c0249e0 con 0x7fa360106de0 2024-04-05T21:16:27.063 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.078+0000 7fa367650700 1 -- 172.21.15.121:0/3733794967 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36019a720 con 0x7fa360106de0 2024-04-05T21:16:27.063 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.078+0000 7fa3567fc700 1 -- 172.21.15.121:0/3733794967 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa35c01aa00 con 0x7fa360106de0 2024-04-05T21:16:27.063 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.078+0000 7fa367650700 1 -- 172.21.15.121:0/3733794967 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36019ab40 con 0x7fa360106de0 2024-04-05T21:16:27.063 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.079+0000 7fa3567fc700 1 -- 172.21.15.121:0/3733794967 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) v1 ==== 45272+0+0 (secure 0 0 0) 0x7fa35c01acc0 con 0x7fa360106de0 2024-04-05T21:16:27.063 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.079+0000 7fa3567fc700 1 --2- 172.21.15.121:0/3733794967 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fa34c0382d0 0x7fa34c03a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:27.063 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.079+0000 7fa3567fc700 1 -- 172.21.15.121:0/3733794967 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa35c04c3f0 con 0x7fa360106de0 2024-04-05T21:16:27.063 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.079+0000 7fa367650700 1 -- 172.21.15.121:0/3733794967 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3600623c0 con 0x7fa360106de0 2024-04-05T21:16:27.063 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.079+0000 7fa364beb700 1 --2- 172.21.15.121:0/3733794967 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fa34c0382d0 0x7fa34c03a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:27.063 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.079+0000 7fa364beb700 1 --2- 172.21.15.121:0/3733794967 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fa34c0382d0 0x7fa34c03a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa350006fd0 tx=0x7fa350006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:27.063 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.084+0000 7fa3567fc700 1 -- 172.21.15.121:0/3733794967 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa35c0193c0 con 0x7fa360106de0 2024-04-05T21:16:27.063 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.322+0000 7fa367650700 1 -- 172.21.15.121:0/3733794967 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fa36019d580 con 0x7fa360106de0 2024-04-05T21:16:27.064 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.375+0000 7fa3567fc700 1 -- 172.21.15.121:0/3733794967 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fa35c016d60 con 0x7fa360106de0 2024-04-05T21:16:27.064 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.376+0000 7fa3567fc700 1 -- 172.21.15.121:0/3733794967 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fa35c04be10 con 0x7fa360106de0 2024-04-05T21:16:27.064 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.378+0000 7fa367650700 1 -- 172.21.15.121:0/3733794967 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fa34c0382d0 msgr2=0x7fa34c03a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:27.064 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.378+0000 7fa367650700 1 --2- 172.21.15.121:0/3733794967 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fa34c0382d0 0x7fa34c03a790 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa350006fd0 tx=0x7fa350006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:27.064 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.378+0000 7fa367650700 1 -- 172.21.15.121:0/3733794967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 msgr2=0x7fa360199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:27.064 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.378+0000 7fa367650700 1 --2- 172.21.15.121:0/3733794967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 0x7fa360199fe0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa35c006af0 tx=0x7fa35c011a30 comp rx=0 tx=0).stop 2024-04-05T21:16:27.064 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.378+0000 7fa367650700 1 -- 172.21.15.121:0/3733794967 shutdown_connections 2024-04-05T21:16:27.064 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.378+0000 7fa367650700 1 --2- 172.21.15.121:0/3733794967 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fa34c0382d0 0x7fa34c03a790 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:27.064 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.378+0000 7fa367650700 1 --2- 172.21.15.121:0/3733794967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa360106de0 0x7fa360199fe0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:27.064 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.378+0000 7fa367650700 1 -- 172.21.15.121:0/3733794967 >> 172.21.15.121:0/3733794967 conn(0x7fa3601007d0 msgr2=0x7fa360101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:27.064 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.378+0000 7fa367650700 1 -- 172.21.15.121:0/3733794967 shutdown_connections 2024-04-05T21:16:27.064 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:26.378+0000 7fa367650700 1 -- 172.21.15.121:0/3733794967 wait complete. 2024-04-05T21:16:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:27 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3733794967' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-05T21:16:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:27 smithi121 ceph-mon[39723]: mgrmap e5: smithi121.etbquh(active, since 12s) 2024-04-05T21:16:28.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:28 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/361011534' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-05T21:16:28.507 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "active_name": "smithi121.etbquh", 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.545+0000 7f7e14859700 1 Processor -- start 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.545+0000 7f7e14859700 1 -- start start 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.546+0000 7f7e14859700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 0x7f7e101091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.546+0000 7f7e14859700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e10079da0 con 0x7f7e10106de0 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.546+0000 7f7e0e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 0x7f7e101091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.546+0000 7f7e0e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 0x7f7e101091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52596/0 (socket says 172.21.15.121:52596) 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.546+0000 7f7e0e59c700 1 -- 172.21.15.121:0/557850084 learned_addr learned my addr 172.21.15.121:0/557850084 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.546+0000 7f7e0e59c700 1 -- 172.21.15.121:0/557850084 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e10109710 con 0x7f7e10106de0 2024-04-05T21:16:28.508 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.546+0000 7f7e0e59c700 1 --2- 172.21.15.121:0/557850084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 0x7f7e101091d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e000098a0 tx=0x7f7e00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93846933661e263f server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.547+0000 7f7e0d59a700 1 -- 172.21.15.121:0/557850084 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e00004180 con 0x7f7e10106de0 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.547+0000 7f7e0d59a700 1 -- 172.21.15.121:0/557850084 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f7e000042e0 con 0x7f7e10106de0 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.547+0000 7f7e0d59a700 1 -- 172.21.15.121:0/557850084 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e000119d0 con 0x7f7e10106de0 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.547+0000 7f7e14859700 1 -- 172.21.15.121:0/557850084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 msgr2=0x7f7e101091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.547+0000 7f7e14859700 1 --2- 172.21.15.121:0/557850084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 0x7f7e101091d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e000098a0 tx=0x7f7e00009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.547+0000 7f7e14859700 1 -- 172.21.15.121:0/557850084 shutdown_connections 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.547+0000 7f7e14859700 1 --2- 172.21.15.121:0/557850084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 0x7f7e101091d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.547+0000 7f7e14859700 1 -- 172.21.15.121:0/557850084 >> 172.21.15.121:0/557850084 conn(0x7f7e101007d0 msgr2=0x7f7e10102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.547+0000 7f7e14859700 1 -- 172.21.15.121:0/557850084 shutdown_connections 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.547+0000 7f7e14859700 1 -- 172.21.15.121:0/557850084 wait complete. 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.548+0000 7f7e14859700 1 Processor -- start 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.548+0000 7f7e14859700 1 -- start start 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.548+0000 7f7e14859700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 0x7f7e10106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.548+0000 7f7e14859700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e10106c80 con 0x7f7e10106de0 2024-04-05T21:16:28.509 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.548+0000 7f7e0e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 0x7f7e10106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:28.510 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.548+0000 7f7e0e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 0x7f7e10106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52598/0 (socket says 172.21.15.121:52598) 2024-04-05T21:16:28.510 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.548+0000 7f7e0e59c700 1 -- 172.21.15.121:0/361011534 learned_addr learned my addr 172.21.15.121:0/361011534 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:28.510 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.549+0000 7f7e0e59c700 1 -- 172.21.15.121:0/361011534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e00009550 con 0x7f7e10106de0 2024-04-05T21:16:28.510 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.549+0000 7f7e0e59c700 1 --2- 172.21.15.121:0/361011534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 0x7f7e10106740 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7e00006af0 tx=0x7f7e00011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:28.510 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.549+0000 7f7dff7fe700 1 -- 172.21.15.121:0/361011534 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e00024430 con 0x7f7e10106de0 2024-04-05T21:16:28.510 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.549+0000 7f7e14859700 1 -- 172.21.15.121:0/361011534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e10104d90 con 0x7f7e10106de0 2024-04-05T21:16:28.510 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.549+0000 7f7dff7fe700 1 -- 172.21.15.121:0/361011534 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f7e000249e0 con 0x7f7e10106de0 2024-04-05T21:16:28.510 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.549+0000 7f7dff7fe700 1 -- 172.21.15.121:0/361011534 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e0001aa00 con 0x7f7e10106de0 2024-04-05T21:16:28.510 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.549+0000 7f7e14859700 1 -- 172.21.15.121:0/361011534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e101051b0 con 0x7f7e10106de0 2024-04-05T21:16:28.510 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.550+0000 7f7dff7fe700 1 -- 172.21.15.121:0/361011534 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f7e0001acc0 con 0x7f7e10106de0 2024-04-05T21:16:28.510 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.550+0000 7f7dff7fe700 1 --2- 172.21.15.121:0/361011534 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f7df8038320 0x7f7df803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:28.510 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.550+0000 7f7dff7fe700 1 -- 172.21.15.121:0/361011534 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7e0004d3a0 con 0x7f7e10106de0 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.550+0000 7f7e0dd9b700 1 -- 172.21.15.121:0/361011534 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f7df8038320 msgr2=0x7f7df803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/169707344 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.550+0000 7f7e0dd9b700 1 --2- 172.21.15.121:0/361011534 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f7df8038320 0x7f7df803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.550+0000 7f7e14859700 1 -- 172.21.15.121:0/361011534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1010a400 con 0x7f7e10106de0 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.555+0000 7f7dff7fe700 1 -- 172.21.15.121:0/361011534 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7e0002d3f0 con 0x7f7e10106de0 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.751+0000 7f7e0dd9b700 1 -- 172.21.15.121:0/361011534 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f7df8038320 msgr2=0x7f7df803a7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/169707344 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.751+0000 7f7e0dd9b700 1 --2- 172.21.15.121:0/361011534 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f7df8038320 0x7f7df803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.805+0000 7f7e14859700 1 -- 172.21.15.121:0/361011534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7e10105c20 con 0x7f7e10106de0 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.806+0000 7f7dff7fe700 1 -- 172.21.15.121:0/361011534 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f7e00023390 con 0x7f7e10106de0 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.808+0000 7f7e14859700 1 -- 172.21.15.121:0/361011534 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f7df8038320 msgr2=0x7f7df803a7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.808+0000 7f7e14859700 1 --2- 172.21.15.121:0/361011534 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f7df8038320 0x7f7df803a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.808+0000 7f7e14859700 1 -- 172.21.15.121:0/361011534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 msgr2=0x7f7e10106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.808+0000 7f7e14859700 1 --2- 172.21.15.121:0/361011534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 0x7f7e10106740 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7e00006af0 tx=0x7f7e00011a30 comp rx=0 tx=0).stop 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.808+0000 7f7e14859700 1 -- 172.21.15.121:0/361011534 shutdown_connections 2024-04-05T21:16:28.511 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.808+0000 7f7e14859700 1 --2- 172.21.15.121:0/361011534 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7f7df8038320 0x7f7df803a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:28.512 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.808+0000 7f7e14859700 1 --2- 172.21.15.121:0/361011534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e10106de0 0x7f7e10106740 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:28.512 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.808+0000 7f7e14859700 1 -- 172.21.15.121:0/361011534 >> 172.21.15.121:0/361011534 conn(0x7f7e101007d0 msgr2=0x7f7e10101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:28.512 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.808+0000 7f7e14859700 1 -- 172.21.15.121:0/361011534 shutdown_connections 2024-04-05T21:16:28.512 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:27.808+0000 7f7e14859700 1 -- 172.21.15.121:0/361011534 wait complete. 2024-04-05T21:16:28.512 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for the mgr to restart... 2024-04-05T21:16:28.512 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mgr epoch 5... 2024-04-05T21:16:33.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: Active manager daemon smithi121.etbquh restarted 2024-04-05T21:16:33.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: Activating manager daemon smithi121.etbquh 2024-04-05T21:16:33.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: osdmap e2: 0 total, 0 up, 0 in 2024-04-05T21:16:33.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: mgrmap e6: smithi121.etbquh(active, starting, since 0.0031698s) 2024-04-05T21:16:33.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:16:33.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr metadata", "who": "smithi121.etbquh", "id": "smithi121.etbquh"}]: dispatch 2024-04-05T21:16:33.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T21:16:33.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T21:16:33.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T21:16:33.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: Manager daemon smithi121.etbquh is now available 2024-04-05T21:16:33.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:33.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:33.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:16:33.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:16:33.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:33 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:16:34.330 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:34 smithi121 ceph-mon[39723]: Found migration_current of "None". Setting to last migration. 2024-04-05T21:16:34.330 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:34 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.etbquh/mirror_snapshot_schedule"}]: dispatch 2024-04-05T21:16:34.330 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:34 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.etbquh/trash_purge_schedule"}]: dispatch 2024-04-05T21:16:34.331 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:34 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:34.331 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:34 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:34.331 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:34 smithi121 ceph-mon[39723]: mgrmap e7: smithi121.etbquh(active, since 1.00538s) 2024-04-05T21:16:34.649 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-04-05T21:16:34.649 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-05T21:16:34.649 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-05T21:16:34.650 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-04-05T21:16:34.650 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.058+0000 7fccbb4db700 1 Processor -- start 2024-04-05T21:16:34.651 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.058+0000 7fccbb4db700 1 -- start start 2024-04-05T21:16:34.651 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.058+0000 7fccbb4db700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb4104c10 0x7fccb4105030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:34.651 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.058+0000 7fccbb4db700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb4079d60 con 0x7fccb4104c10 2024-04-05T21:16:34.651 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.058+0000 7fccb9277700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb4104c10 0x7fccb4105030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:34.651 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.058+0000 7fccb9277700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb4104c10 0x7fccb4105030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52610/0 (socket says 172.21.15.121:52610) 2024-04-05T21:16:34.651 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.058+0000 7fccb9277700 1 -- 172.21.15.121:0/993084300 learned_addr learned my addr 172.21.15.121:0/993084300 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:34.651 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.059+0000 7fccb9277700 1 -- 172.21.15.121:0/993084300 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb4105570 con 0x7fccb4104c10 2024-04-05T21:16:34.651 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.059+0000 7fccb9277700 1 --2- 172.21.15.121:0/993084300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb4104c10 0x7fccb4105030 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fccb00098a0 tx=0x7fccb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da2850c3b0ff14ec server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:34.652 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.059+0000 7fccabfff700 1 -- 172.21.15.121:0/993084300 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccb0004180 con 0x7fccb4104c10 2024-04-05T21:16:34.652 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.059+0000 7fccabfff700 1 -- 172.21.15.121:0/993084300 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7fccb00042e0 con 0x7fccb4104c10 2024-04-05T21:16:34.652 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.059+0000 7fccabfff700 1 -- 172.21.15.121:0/993084300 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccb00119d0 con 0x7fccb4104c10 2024-04-05T21:16:34.652 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.060+0000 7fccbb4db700 1 -- 172.21.15.121:0/993084300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb4104c10 msgr2=0x7fccb4105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:34.652 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.060+0000 7fccbb4db700 1 --2- 172.21.15.121:0/993084300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb4104c10 0x7fccb4105030 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fccb00098a0 tx=0x7fccb0009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:34.652 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.060+0000 7fccbb4db700 1 -- 172.21.15.121:0/993084300 shutdown_connections 2024-04-05T21:16:34.652 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.060+0000 7fccbb4db700 1 --2- 172.21.15.121:0/993084300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb4104c10 0x7fccb4105030 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:34.652 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.060+0000 7fccbb4db700 1 -- 172.21.15.121:0/993084300 >> 172.21.15.121:0/993084300 conn(0x7fccb41007d0 msgr2=0x7fccb4102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:34.652 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.060+0000 7fccbb4db700 1 -- 172.21.15.121:0/993084300 shutdown_connections 2024-04-05T21:16:34.652 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.060+0000 7fccbb4db700 1 -- 172.21.15.121:0/993084300 wait complete. 2024-04-05T21:16:34.652 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.061+0000 7fccbb4db700 1 Processor -- start 2024-04-05T21:16:34.652 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.061+0000 7fccbb4db700 1 -- start start 2024-04-05T21:16:34.652 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.061+0000 7fccbb4db700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb419c3f0 0x7fccb419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.061+0000 7fccbb4db700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb419cd50 con 0x7fccb419c3f0 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.061+0000 7fccb9277700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb419c3f0 0x7fccb419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.061+0000 7fccb9277700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb419c3f0 0x7fccb419c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52626/0 (socket says 172.21.15.121:52626) 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.061+0000 7fccb9277700 1 -- 172.21.15.121:0/2508171288 learned_addr learned my addr 172.21.15.121:0/2508171288 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.062+0000 7fccb9277700 1 -- 172.21.15.121:0/2508171288 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb0009550 con 0x7fccb419c3f0 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.062+0000 7fccb9277700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb419c3f0 0x7fccb419c810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fccb0006af0 tx=0x7fccb0012ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.062+0000 7fccaa7fc700 1 -- 172.21.15.121:0/2508171288 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccb00037f0 con 0x7fccb419c3f0 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.062+0000 7fccaa7fc700 1 -- 172.21.15.121:0/2508171288 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7fccb001f070 con 0x7fccb419c3f0 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.062+0000 7fccbb4db700 1 -- 172.21.15.121:0/2508171288 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb419cf50 con 0x7fccb419c3f0 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.062+0000 7fccaa7fc700 1 -- 172.21.15.121:0/2508171288 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccb001ae80 con 0x7fccb419c3f0 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.062+0000 7fccbb4db700 1 -- 172.21.15.121:0/2508171288 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb419fba0 con 0x7fccb419c3f0 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.063+0000 7fccaa7fc700 1 -- 172.21.15.121:0/2508171288 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fccb001b440 con 0x7fccb419c3f0 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.063+0000 7fccaa7fc700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 0x7fcca003a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:34.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.063+0000 7fccaa7fc700 1 -- 172.21.15.121:0/2508171288 --> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcca003ae60 con 0x7fcca0038290 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.063+0000 7fccaa7fc700 1 -- 172.21.15.121:0/2508171288 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fccb0021070 con 0x7fccb419c3f0 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.063+0000 7fccb8a76700 1 -- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 msgr2=0x7fcca003a750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/169707344 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.063+0000 7fccb8a76700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 0x7fcca003a750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.264+0000 7fccb8a76700 1 -- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 msgr2=0x7fcca003a750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/169707344 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.264+0000 7fccb8a76700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 0x7fcca003a750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.665+0000 7fccb8a76700 1 -- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 msgr2=0x7fcca003a750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/169707344 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:29.665+0000 7fccb8a76700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 0x7fcca003a750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:30.466+0000 7fccb8a76700 1 -- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 msgr2=0x7fcca003a750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/169707344 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:30.466+0000 7fccb8a76700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 0x7fcca003a750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:32.067+0000 7fccb8a76700 1 -- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 msgr2=0x7fcca003a750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/169707344 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:32.067+0000 7fccb8a76700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 0x7fcca003a750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:32.953+0000 7fccaa7fc700 1 -- 172.21.15.121:0/2508171288 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mgrmap(e 6) v1 ==== 45050+0+0 (secure 0 0 0) 0x7fccb001b790 con 0x7fccb419c3f0 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:32.953+0000 7fccaa7fc700 1 -- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 msgr2=0x7fcca003a750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T21:16:34.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:32.953+0000 7fccaa7fc700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 0x7fcca003a750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.954+0000 7fccaa7fc700 1 -- 172.21.15.121:0/2508171288 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 7) v1 ==== 45177+0+0 (secure 0 0 0) 0x7fccb004d140 con 0x7fccb419c3f0 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.954+0000 7fccaa7fc700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fcca003bc60 0x7fcca003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.954+0000 7fccaa7fc700 1 -- 172.21.15.121:0/2508171288 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcca003ae60 con 0x7fcca003bc60 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.955+0000 7fccb8a76700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fcca003bc60 0x7fcca003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.955+0000 7fccb8a76700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fcca003bc60 0x7fcca003e050 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcca4003a10 tx=0x7fcca4009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.956+0000 7fccaa7fc700 1 -- 172.21.15.121:0/2508171288 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fcca003ae60 con 0x7fcca003bc60 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.962+0000 7fccbb4db700 1 -- 172.21.15.121:0/2508171288 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fccb419d770 con 0x7fcca003bc60 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.962+0000 7fccaa7fc700 1 -- 172.21.15.121:0/2508171288 <== mgr.14118 v2:172.21.15.121:6800/1004235250 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fccb419d770 con 0x7fcca003bc60 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.962+0000 7fccbb4db700 1 -- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fcca003bc60 msgr2=0x7fcca003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.962+0000 7fccbb4db700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fcca003bc60 0x7fcca003e050 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcca4003a10 tx=0x7fcca4009290 comp rx=0 tx=0).stop 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.962+0000 7fccbb4db700 1 -- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb419c3f0 msgr2=0x7fccb419c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.962+0000 7fccbb4db700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb419c3f0 0x7fccb419c810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fccb0006af0 tx=0x7fccb0012ff0 comp rx=0 tx=0).stop 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.962+0000 7fccbb4db700 1 -- 172.21.15.121:0/2508171288 shutdown_connections 2024-04-05T21:16:34.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.962+0000 7fccbb4db700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fcca003bc60 0x7fcca003e050 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:34.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.962+0000 7fccbb4db700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb419c3f0 0x7fccb419c810 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:34.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.962+0000 7fccbb4db700 1 --2- 172.21.15.121:0/2508171288 >> [v2:172.21.15.121:6800/169707344,v1:172.21.15.121:6801/169707344] conn(0x7fcca0038290 0x7fcca003a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:34.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.962+0000 7fccbb4db700 1 -- 172.21.15.121:0/2508171288 >> 172.21.15.121:0/2508171288 conn(0x7fccb41007d0 msgr2=0x7fccb41907d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:34.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.963+0000 7fccbb4db700 1 -- 172.21.15.121:0/2508171288 shutdown_connections 2024-04-05T21:16:34.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:33.963+0000 7fccbb4db700 1 -- 172.21.15.121:0/2508171288 wait complete. 2024-04-05T21:16:34.656 INFO:teuthology.orchestra.run.smithi121.stdout:mgr epoch 5 is available 2024-04-05T21:16:34.656 INFO:teuthology.orchestra.run.smithi121.stdout:Setting orchestrator backend to cephadm... 2024-04-05T21:16:35.376 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:35 smithi121 ceph-mon[39723]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-05T21:16:35.376 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:35 smithi121 ceph-mon[39723]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-05T21:16:35.376 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:35 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:16:35.945 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.133+0000 7f81871e3700 1 Processor -- start 2024-04-05T21:16:35.946 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.133+0000 7f81871e3700 1 -- start start 2024-04-05T21:16:35.946 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.133+0000 7f81871e3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 0x7f8180106750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:35.946 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.133+0000 7f81871e3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8180106d20 con 0x7f8180106330 2024-04-05T21:16:35.946 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.133+0000 7f8184f7f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 0x7f8180106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:35.946 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.133+0000 7f8184f7f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 0x7f8180106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39512/0 (socket says 172.21.15.121:39512) 2024-04-05T21:16:35.946 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.133+0000 7f8184f7f700 1 -- 172.21.15.121:0/1769457363 learned_addr learned my addr 172.21.15.121:0/1769457363 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:35.946 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.133+0000 7f8184f7f700 1 -- 172.21.15.121:0/1769457363 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8180107530 con 0x7f8180106330 2024-04-05T21:16:35.946 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.133+0000 7f8184f7f700 1 --2- 172.21.15.121:0/1769457363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 0x7f8180106750 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f81740098a0 tx=0x7f8174009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa570332e3ee6a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:35.946 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.134+0000 7f817f7fe700 1 -- 172.21.15.121:0/1769457363 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8174004180 con 0x7f8180106330 2024-04-05T21:16:35.946 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.134+0000 7f817f7fe700 1 -- 172.21.15.121:0/1769457363 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f81740042e0 con 0x7f8180106330 2024-04-05T21:16:35.946 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.134+0000 7f817f7fe700 1 -- 172.21.15.121:0/1769457363 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81740119d0 con 0x7f8180106330 2024-04-05T21:16:35.946 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.134+0000 7f81871e3700 1 -- 172.21.15.121:0/1769457363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 msgr2=0x7f8180106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:35.946 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.134+0000 7f81871e3700 1 --2- 172.21.15.121:0/1769457363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 0x7f8180106750 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f81740098a0 tx=0x7f8174009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.134+0000 7f81871e3700 1 -- 172.21.15.121:0/1769457363 shutdown_connections 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.134+0000 7f81871e3700 1 --2- 172.21.15.121:0/1769457363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 0x7f8180106750 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.134+0000 7f81871e3700 1 -- 172.21.15.121:0/1769457363 >> 172.21.15.121:0/1769457363 conn(0x7f81801018d0 msgr2=0x7f8180103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.134+0000 7f81871e3700 1 -- 172.21.15.121:0/1769457363 shutdown_connections 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.134+0000 7f81871e3700 1 -- 172.21.15.121:0/1769457363 wait complete. 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.135+0000 7f81871e3700 1 Processor -- start 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.135+0000 7f81871e3700 1 -- start start 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.135+0000 7f81871e3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 0x7f818019a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.135+0000 7f81871e3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818019a550 con 0x7f8180106330 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.135+0000 7f8184f7f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 0x7f818019a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.135+0000 7f8184f7f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 0x7f818019a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39514/0 (socket says 172.21.15.121:39514) 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.135+0000 7f8184f7f700 1 -- 172.21.15.121:0/1914711536 learned_addr learned my addr 172.21.15.121:0/1914711536 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.135+0000 7f8184f7f700 1 -- 172.21.15.121:0/1914711536 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8174009550 con 0x7f8180106330 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.135+0000 7f8184f7f700 1 --2- 172.21.15.121:0/1914711536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 0x7f818019a010 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8174006af0 tx=0x7f8174011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:35.947 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.136+0000 7f817dffb700 1 -- 172.21.15.121:0/1914711536 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8174024430 con 0x7f8180106330 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.136+0000 7f817dffb700 1 -- 172.21.15.121:0/1914711536 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f81740249e0 con 0x7f8180106330 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.136+0000 7f81871e3700 1 -- 172.21.15.121:0/1914711536 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f818019a750 con 0x7f8180106330 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.136+0000 7f817dffb700 1 -- 172.21.15.121:0/1914711536 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f817401a970 con 0x7f8180106330 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.136+0000 7f81871e3700 1 -- 172.21.15.121:0/1914711536 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f818019ab70 con 0x7f8180106330 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.136+0000 7f817dffb700 1 -- 172.21.15.121:0/1914711536 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f817401abe0 con 0x7f8180106330 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.137+0000 7f817dffb700 1 --2- 172.21.15.121:0/1914711536 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f8170038370 0x7f817003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.137+0000 7f817dffb700 1 -- 172.21.15.121:0/1914711536 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f817404d390 con 0x7f8180106330 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.137+0000 7f817ffff700 1 --2- 172.21.15.121:0/1914711536 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f8170038370 0x7f817003a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.137+0000 7f81871e3700 1 -- 172.21.15.121:0/1914711536 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81800623c0 con 0x7f8180106330 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.137+0000 7f817ffff700 1 --2- 172.21.15.121:0/1914711536 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f8170038370 0x7f817003a830 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f816c006fd0 tx=0x7f816c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.142+0000 7f817dffb700 1 -- 172.21.15.121:0/1914711536 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f817402d3f0 con 0x7f8180106330 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.343+0000 7f81871e3700 1 -- 172.21.15.121:0/1914711536 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f8180103ba0 con 0x7f8170038370 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.349+0000 7f817dffb700 1 -- 172.21.15.121:0/1914711536 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8180103ba0 con 0x7f8170038370 2024-04-05T21:16:35.948 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.351+0000 7f81871e3700 1 -- 172.21.15.121:0/1914711536 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f8170038370 msgr2=0x7f817003a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:35.949 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.351+0000 7f81871e3700 1 --2- 172.21.15.121:0/1914711536 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f8170038370 0x7f817003a830 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f816c006fd0 tx=0x7f816c006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:35.949 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.351+0000 7f81871e3700 1 -- 172.21.15.121:0/1914711536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 msgr2=0x7f818019a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:35.949 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.351+0000 7f81871e3700 1 --2- 172.21.15.121:0/1914711536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 0x7f818019a010 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8174006af0 tx=0x7f8174011a30 comp rx=0 tx=0).stop 2024-04-05T21:16:35.949 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.351+0000 7f81871e3700 1 -- 172.21.15.121:0/1914711536 shutdown_connections 2024-04-05T21:16:35.949 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.351+0000 7f81871e3700 1 --2- 172.21.15.121:0/1914711536 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f8170038370 0x7f817003a830 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:35.949 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.351+0000 7f81871e3700 1 --2- 172.21.15.121:0/1914711536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8180106330 0x7f818019a010 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:35.949 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.351+0000 7f81871e3700 1 -- 172.21.15.121:0/1914711536 >> 172.21.15.121:0/1914711536 conn(0x7f81801018d0 msgr2=0x7f8180102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:35.949 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.352+0000 7f81871e3700 1 -- 172.21.15.121:0/1914711536 shutdown_connections 2024-04-05T21:16:35.949 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:35.352+0000 7f81871e3700 1 -- 172.21.15.121:0/1914711536 wait complete. 2024-04-05T21:16:36.194 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:36 smithi121 ceph-mon[39723]: [05/Apr/2024:21:16:34] ENGINE Bus STARTING 2024-04-05T21:16:36.194 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:36 smithi121 ceph-mon[39723]: [05/Apr/2024:21:16:34] ENGINE Serving on https://172.21.15.121:7150 2024-04-05T21:16:36.194 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:36 smithi121 ceph-mon[39723]: [05/Apr/2024:21:16:34] ENGINE Serving on http://172.21.15.121:8765 2024-04-05T21:16:36.194 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:36 smithi121 ceph-mon[39723]: [05/Apr/2024:21:16:34] ENGINE Bus STARTED 2024-04-05T21:16:36.194 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:36 smithi121 ceph-mon[39723]: mgrmap e8: smithi121.etbquh(active, since 2s) 2024-04-05T21:16:36.194 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:36 smithi121 ceph-mon[39723]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:16:36.194 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:36 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:36.194 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:36 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:16:37.597 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-05T21:16:37.597 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.475+0000 7f6bc8143700 1 Processor -- start 2024-04-05T21:16:37.597 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.475+0000 7f6bc8143700 1 -- start start 2024-04-05T21:16:37.597 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.475+0000 7f6bc8143700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 0x7f6bc01091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:37.601 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.475+0000 7f6bc8143700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc00744f0 con 0x7f6bc0106de0 2024-04-05T21:16:37.602 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.475+0000 7f6bc5edf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 0x7f6bc01091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:37.602 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.475+0000 7f6bc5edf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 0x7f6bc01091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39522/0 (socket says 172.21.15.121:39522) 2024-04-05T21:16:37.602 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.475+0000 7f6bc5edf700 1 -- 172.21.15.121:0/2258696813 learned_addr learned my addr 172.21.15.121:0/2258696813 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:37.602 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.476+0000 7f6bc5edf700 1 -- 172.21.15.121:0/2258696813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc0109710 con 0x7f6bc0106de0 2024-04-05T21:16:37.602 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.476+0000 7f6bc5edf700 1 --2- 172.21.15.121:0/2258696813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 0x7f6bc01091d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00098a0 tx=0x7f6bb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3d85a2912b250a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:37.602 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.476+0000 7f6bc4edd700 1 -- 172.21.15.121:0/2258696813 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bb0004180 con 0x7f6bc0106de0 2024-04-05T21:16:37.602 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.476+0000 7f6bc4edd700 1 -- 172.21.15.121:0/2258696813 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f6bb00042e0 con 0x7f6bc0106de0 2024-04-05T21:16:37.602 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.476+0000 7f6bc4edd700 1 -- 172.21.15.121:0/2258696813 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bb00119d0 con 0x7f6bc0106de0 2024-04-05T21:16:37.602 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.476+0000 7f6bc8143700 1 -- 172.21.15.121:0/2258696813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 msgr2=0x7f6bc01091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:37.602 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.476+0000 7f6bc8143700 1 --2- 172.21.15.121:0/2258696813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 0x7f6bc01091d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00098a0 tx=0x7f6bb0009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.477+0000 7f6bc8143700 1 -- 172.21.15.121:0/2258696813 shutdown_connections 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.477+0000 7f6bc8143700 1 --2- 172.21.15.121:0/2258696813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 0x7f6bc01091d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.477+0000 7f6bc8143700 1 -- 172.21.15.121:0/2258696813 >> 172.21.15.121:0/2258696813 conn(0x7f6bc01007f0 msgr2=0x7f6bc0102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.477+0000 7f6bc8143700 1 -- 172.21.15.121:0/2258696813 shutdown_connections 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.477+0000 7f6bc8143700 1 -- 172.21.15.121:0/2258696813 wait complete. 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.477+0000 7f6bc8143700 1 Processor -- start 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.477+0000 7f6bc8143700 1 -- start start 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.477+0000 7f6bc8143700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 0x7f6bc019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.477+0000 7f6bc8143700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc019a540 con 0x7f6bc0106de0 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.477+0000 7f6bc5edf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 0x7f6bc019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.478+0000 7f6bc5edf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 0x7f6bc019a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39528/0 (socket says 172.21.15.121:39528) 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.478+0000 7f6bc5edf700 1 -- 172.21.15.121:0/3327064648 learned_addr learned my addr 172.21.15.121:0/3327064648 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.478+0000 7f6bc5edf700 1 -- 172.21.15.121:0/3327064648 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb0009550 con 0x7f6bc0106de0 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.478+0000 7f6bc5edf700 1 --2- 172.21.15.121:0/3327064648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 0x7f6bc019a000 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0009130 tx=0x7f6bb0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:37.603 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.478+0000 7f6bb6ffd700 1 -- 172.21.15.121:0/3327064648 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bb0003900 con 0x7f6bc0106de0 2024-04-05T21:16:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.478+0000 7f6bb6ffd700 1 -- 172.21.15.121:0/3327064648 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f6bb0003a60 con 0x7f6bc0106de0 2024-04-05T21:16:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.478+0000 7f6bc8143700 1 -- 172.21.15.121:0/3327064648 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc019a740 con 0x7f6bc0106de0 2024-04-05T21:16:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.478+0000 7f6bb6ffd700 1 -- 172.21.15.121:0/3327064648 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bb00246e0 con 0x7f6bc0106de0 2024-04-05T21:16:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.478+0000 7f6bc8143700 1 -- 172.21.15.121:0/3327064648 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc019ab60 con 0x7f6bc0106de0 2024-04-05T21:16:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.479+0000 7f6bb6ffd700 1 -- 172.21.15.121:0/3327064648 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f6bb002d3f0 con 0x7f6bc0106de0 2024-04-05T21:16:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.479+0000 7f6bc8143700 1 -- 172.21.15.121:0/3327064648 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc00623c0 con 0x7f6bc0106de0 2024-04-05T21:16:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.479+0000 7f6bb6ffd700 1 --2- 172.21.15.121:0/3327064648 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f6bac038370 0x7f6bac03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.479+0000 7f6bb6ffd700 1 -- 172.21.15.121:0/3327064648 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6bb004c3a0 con 0x7f6bc0106de0 2024-04-05T21:16:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.479+0000 7f6bc56de700 1 --2- 172.21.15.121:0/3327064648 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f6bac038370 0x7f6bac03a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.479+0000 7f6bc56de700 1 --2- 172.21.15.121:0/3327064648 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f6bac038370 0x7f6bac03a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc006fd0 tx=0x7f6bbc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.484+0000 7f6bb6ffd700 1 -- 172.21.15.121:0/3327064648 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6bb0010970 con 0x7f6bc0106de0 2024-04-05T21:16:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.673+0000 7f6bc8143700 1 -- 172.21.15.121:0/3327064648 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bc0102b00 con 0x7f6bac038370 2024-04-05T21:16:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.673+0000 7f6bb6ffd700 1 -- 172.21.15.121:0/3327064648 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f6bc0102b00 con 0x7f6bac038370 2024-04-05T21:16:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.675+0000 7f6bc8143700 1 -- 172.21.15.121:0/3327064648 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f6bac038370 msgr2=0x7f6bac03a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:37.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:37 smithi121 ceph-mon[39723]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:16:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.675+0000 7f6bc8143700 1 --2- 172.21.15.121:0/3327064648 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f6bac038370 0x7f6bac03a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc006fd0 tx=0x7f6bbc006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.675+0000 7f6bc8143700 1 -- 172.21.15.121:0/3327064648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 msgr2=0x7f6bc019a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.675+0000 7f6bc8143700 1 --2- 172.21.15.121:0/3327064648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 0x7f6bc019a000 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0009130 tx=0x7f6bb0011a30 comp rx=0 tx=0).stop 2024-04-05T21:16:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.675+0000 7f6bc8143700 1 -- 172.21.15.121:0/3327064648 shutdown_connections 2024-04-05T21:16:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.675+0000 7f6bc8143700 1 --2- 172.21.15.121:0/3327064648 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f6bac038370 0x7f6bac03a830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.675+0000 7f6bc8143700 1 --2- 172.21.15.121:0/3327064648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bc0106de0 0x7f6bc019a000 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.675+0000 7f6bc8143700 1 -- 172.21.15.121:0/3327064648 >> 172.21.15.121:0/3327064648 conn(0x7f6bc01007f0 msgr2=0x7f6bc0101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:37.607 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.676+0000 7f6bc8143700 1 -- 172.21.15.121:0/3327064648 shutdown_connections 2024-04-05T21:16:37.607 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:36.676+0000 7f6bc8143700 1 -- 172.21.15.121:0/3327064648 wait complete. 2024-04-05T21:16:37.607 INFO:teuthology.orchestra.run.smithi121.stdout:Generating ssh key... 2024-04-05T21:16:38.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:38 smithi121 ceph-mon[39723]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:16:38.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:38 smithi121 ceph-mon[39723]: Generating ssh key... 2024-04-05T21:16:39.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.189+0000 7f6e726b6700 1 Processor -- start 2024-04-05T21:16:39.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.189+0000 7f6e726b6700 1 -- start start 2024-04-05T21:16:39.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.189+0000 7f6e726b6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 0x7f6e6c104a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:39.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.189+0000 7f6e726b6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c104ff0 con 0x7f6e6c104600 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.189+0000 7f6e6bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 0x7f6e6c104a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.189+0000 7f6e6bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 0x7f6e6c104a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39540/0 (socket says 172.21.15.121:39540) 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.189+0000 7f6e6bfff700 1 -- 172.21.15.121:0/1552634065 learned_addr learned my addr 172.21.15.121:0/1552634065 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.189+0000 7f6e6bfff700 1 -- 172.21.15.121:0/1552634065 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c105130 con 0x7f6e6c104600 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.189+0000 7f6e6bfff700 1 --2- 172.21.15.121:0/1552634065 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 0x7f6e6c104a20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0098a0 tx=0x7f6e5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2574e158a31a16cd server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.190+0000 7f6e6affd700 1 -- 172.21.15.121:0/1552634065 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e5c004180 con 0x7f6e6c104600 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.190+0000 7f6e6affd700 1 -- 172.21.15.121:0/1552634065 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f6e5c0042e0 con 0x7f6e6c104600 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.190+0000 7f6e6affd700 1 -- 172.21.15.121:0/1552634065 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e5c0119d0 con 0x7f6e6c104600 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.190+0000 7f6e726b6700 1 -- 172.21.15.121:0/1552634065 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 msgr2=0x7f6e6c104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.190+0000 7f6e726b6700 1 --2- 172.21.15.121:0/1552634065 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 0x7f6e6c104a20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c0098a0 tx=0x7f6e5c009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.190+0000 7f6e726b6700 1 -- 172.21.15.121:0/1552634065 shutdown_connections 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.190+0000 7f6e726b6700 1 --2- 172.21.15.121:0/1552634065 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 0x7f6e6c104a20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.190+0000 7f6e726b6700 1 -- 172.21.15.121:0/1552634065 >> 172.21.15.121:0/1552634065 conn(0x7f6e6c0fff40 msgr2=0x7f6e6c102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:39.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.190+0000 7f6e726b6700 1 -- 172.21.15.121:0/1552634065 shutdown_connections 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.190+0000 7f6e726b6700 1 -- 172.21.15.121:0/1552634065 wait complete. 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.191+0000 7f6e726b6700 1 Processor -- start 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.191+0000 7f6e726b6700 1 -- start start 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.191+0000 7f6e726b6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 0x7f6e6c199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.191+0000 7f6e726b6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c19a4f0 con 0x7f6e6c104600 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.191+0000 7f6e6bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 0x7f6e6c199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.191+0000 7f6e6bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 0x7f6e6c199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39546/0 (socket says 172.21.15.121:39546) 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.191+0000 7f6e6bfff700 1 -- 172.21.15.121:0/1387428861 learned_addr learned my addr 172.21.15.121:0/1387428861 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.191+0000 7f6e6bfff700 1 -- 172.21.15.121:0/1387428861 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c009550 con 0x7f6e6c104600 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.192+0000 7f6e6bfff700 1 --2- 172.21.15.121:0/1387428861 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 0x7f6e6c199fb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c006af0 tx=0x7f6e5c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.192+0000 7f6e697fa700 1 -- 172.21.15.121:0/1387428861 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e5c024430 con 0x7f6e6c104600 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.192+0000 7f6e697fa700 1 -- 172.21.15.121:0/1387428861 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f6e5c0249e0 con 0x7f6e6c104600 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.192+0000 7f6e726b6700 1 -- 172.21.15.121:0/1387428861 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c19a750 con 0x7f6e6c104600 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.192+0000 7f6e697fa700 1 -- 172.21.15.121:0/1387428861 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e5c01a970 con 0x7f6e6c104600 2024-04-05T21:16:39.084 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.192+0000 7f6e726b6700 1 -- 172.21.15.121:0/1387428861 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6c19abd0 con 0x7f6e6c104600 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.193+0000 7f6e697fa700 1 -- 172.21.15.121:0/1387428861 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f6e5c01abe0 con 0x7f6e6c104600 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.193+0000 7f6e726b6700 1 -- 172.21.15.121:0/1387428861 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6c1081f0 con 0x7f6e6c104600 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.193+0000 7f6e697fa700 1 --2- 172.21.15.121:0/1387428861 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f6e54038370 0x7f6e5403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.193+0000 7f6e697fa700 1 -- 172.21.15.121:0/1387428861 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6e5c04c400 con 0x7f6e6c104600 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.193+0000 7f6e6b7fe700 1 --2- 172.21.15.121:0/1387428861 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f6e54038370 0x7f6e5403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.193+0000 7f6e6b7fe700 1 --2- 172.21.15.121:0/1387428861 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f6e54038370 0x7f6e5403a830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6e60006fd0 tx=0x7f6e60006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.197+0000 7f6e697fa700 1 -- 172.21.15.121:0/1387428861 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6e5c02d3f0 con 0x7f6e6c104600 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.384+0000 7f6e726b6700 1 -- 172.21.15.121:0/1387428861 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e6c1022a0 con 0x7f6e54038370 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.480+0000 7f6e697fa700 1 -- 172.21.15.121:0/1387428861 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6e6c1022a0 con 0x7f6e54038370 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.482+0000 7f6e726b6700 1 -- 172.21.15.121:0/1387428861 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f6e54038370 msgr2=0x7f6e5403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.482+0000 7f6e726b6700 1 --2- 172.21.15.121:0/1387428861 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f6e54038370 0x7f6e5403a830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6e60006fd0 tx=0x7f6e60006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.482+0000 7f6e726b6700 1 -- 172.21.15.121:0/1387428861 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 msgr2=0x7f6e6c199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.482+0000 7f6e726b6700 1 --2- 172.21.15.121:0/1387428861 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 0x7f6e6c199fb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c006af0 tx=0x7f6e5c011a30 comp rx=0 tx=0).stop 2024-04-05T21:16:39.085 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.483+0000 7f6e726b6700 1 -- 172.21.15.121:0/1387428861 shutdown_connections 2024-04-05T21:16:39.086 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.483+0000 7f6e726b6700 1 --2- 172.21.15.121:0/1387428861 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f6e54038370 0x7f6e5403a830 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:39.086 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.483+0000 7f6e726b6700 1 --2- 172.21.15.121:0/1387428861 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e6c104600 0x7f6e6c199fb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:39.086 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.483+0000 7f6e726b6700 1 -- 172.21.15.121:0/1387428861 >> 172.21.15.121:0/1387428861 conn(0x7f6e6c0fff40 msgr2=0x7f6e6c101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:39.086 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.483+0000 7f6e726b6700 1 -- 172.21.15.121:0/1387428861 shutdown_connections 2024-04-05T21:16:39.086 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:38.483+0000 7f6e726b6700 1 -- 172.21.15.121:0/1387428861 wait complete. 2024-04-05T21:16:39.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:39 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:39.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:39 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZEkPkm9R+lyo91WlIkAk6OEW7qkCToqF+TI4I5zWm6+TfxBRjkEpByiYhK6pDrxxtGOlvx1yWZm/6Wzl67bgIzviXJmnZKbePJIzIY6TKqZGlOUTYPqKNAs9Fv+zjkwKAl7z4suLH5BPfEzQJt5Fs5dxTkP+XgNheoyq0Vegbt0ui5JfxrQIA5tZIqb1awAbnJAKcQepBArocGNmDgUem/Y312GXkqqvSxBOscDoe2y7DpoDSCvneouZ2R4kVHeRmoJUHuV7Nz6oMWcAUP6S5yyUidRb5kFAc2kKOpXlqQSNcnlPf7rwksHqwcLCc4/vvNWAIE7uwJMQM9f24LstyrUTK8EzyCQWNoiFpEY+fYe8JDutCwFTyBL6ZBeKhg0gANDov3owtxXyJKTyiCA9aBtm6/MKhNArnWIibH7e1PiYZLYOHveGhM6lz4ov++iUzWNBjOONDgL20Ws3lBGiT4cYpEeLcFq0AEWVl05oqYYlEqKYKvrP7eJUt3KIvWcU= ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:16:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.640+0000 7f5f6bb2f700 1 Processor -- start 2024-04-05T21:16:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.640+0000 7f5f6bb2f700 1 -- start start 2024-04-05T21:16:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.640+0000 7f5f6bb2f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 0x7f5f6407a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.640+0000 7f5f6bb2f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6407a820 con 0x7f5f6407bdf0 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.640+0000 7f5f698cb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 0x7f5f6407a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.640+0000 7f5f698cb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 0x7f5f6407a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39558/0 (socket says 172.21.15.121:39558) 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.640+0000 7f5f698cb700 1 -- 172.21.15.121:0/3746941924 learned_addr learned my addr 172.21.15.121:0/3746941924 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.640+0000 7f5f698cb700 1 -- 172.21.15.121:0/3746941924 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6407a960 con 0x7f5f6407bdf0 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.640+0000 7f5f698cb700 1 --2- 172.21.15.121:0/3746941924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 0x7f5f6407a250 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5f540098a0 tx=0x7f5f54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ad827a75ee2bf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.641+0000 7f5f688c9700 1 -- 172.21.15.121:0/3746941924 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f54004180 con 0x7f5f6407bdf0 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.641+0000 7f5f688c9700 1 -- 172.21.15.121:0/3746941924 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f5f540042e0 con 0x7f5f6407bdf0 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.641+0000 7f5f688c9700 1 -- 172.21.15.121:0/3746941924 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f540119d0 con 0x7f5f6407bdf0 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.641+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3746941924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 msgr2=0x7f5f6407a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.641+0000 7f5f6bb2f700 1 --2- 172.21.15.121:0/3746941924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 0x7f5f6407a250 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5f540098a0 tx=0x7f5f54009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.641+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3746941924 shutdown_connections 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.641+0000 7f5f6bb2f700 1 --2- 172.21.15.121:0/3746941924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 0x7f5f6407a250 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.641+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3746941924 >> 172.21.15.121:0/3746941924 conn(0x7f5f64101910 msgr2=0x7f5f64103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.641+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3746941924 shutdown_connections 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.641+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3746941924 wait complete. 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.642+0000 7f5f6bb2f700 1 Processor -- start 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.642+0000 7f5f6bb2f700 1 -- start start 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.642+0000 7f5f6bb2f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 0x7f5f6419c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.642+0000 7f5f6bb2f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6419c7b0 con 0x7f5f6407bdf0 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.642+0000 7f5f698cb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 0x7f5f6419c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.642+0000 7f5f698cb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 0x7f5f6419c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39566/0 (socket says 172.21.15.121:39566) 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.642+0000 7f5f698cb700 1 -- 172.21.15.121:0/3667207186 learned_addr learned my addr 172.21.15.121:0/3667207186 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.642+0000 7f5f698cb700 1 -- 172.21.15.121:0/3667207186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f54009550 con 0x7f5f6407bdf0 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.643+0000 7f5f698cb700 1 --2- 172.21.15.121:0/3667207186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 0x7f5f6419c270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009130 tx=0x7f5f54011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.643+0000 7f5f5affd700 1 -- 172.21.15.121:0/3667207186 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f54003a00 con 0x7f5f6407bdf0 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.643+0000 7f5f5affd700 1 -- 172.21.15.121:0/3667207186 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f5f5401a880 con 0x7f5f6407bdf0 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.643+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3667207186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6419c9b0 con 0x7f5f6407bdf0 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.643+0000 7f5f5affd700 1 -- 172.21.15.121:0/3667207186 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f54019760 con 0x7f5f6407bdf0 2024-04-05T21:16:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.643+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3667207186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6419cdd0 con 0x7f5f6407bdf0 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.643+0000 7f5f5affd700 1 -- 172.21.15.121:0/3667207186 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f5f540199d0 con 0x7f5f6407bdf0 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.644+0000 7f5f5affd700 1 --2- 172.21.15.121:0/3667207186 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5f50038330 0x7f5f5003a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.644+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3667207186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f640623c0 con 0x7f5f6407bdf0 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.644+0000 7f5f5affd700 1 -- 172.21.15.121:0/3667207186 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5f5404c6a0 con 0x7f5f6407bdf0 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.644+0000 7f5f690ca700 1 --2- 172.21.15.121:0/3667207186 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5f50038330 0x7f5f5003a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.644+0000 7f5f690ca700 1 --2- 172.21.15.121:0/3667207186 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5f50038330 0x7f5f5003a7f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5f60006fd0 tx=0x7f5f60006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.648+0000 7f5f5affd700 1 -- 172.21.15.121:0/3667207186 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5f5402b750 con 0x7f5f6407bdf0 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.837+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3667207186 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f6419f810 con 0x7f5f50038330 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.838+0000 7f5f5affd700 1 -- 172.21.15.121:0/3667207186 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f5f6419f810 con 0x7f5f50038330 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.840+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3667207186 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5f50038330 msgr2=0x7f5f5003a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.840+0000 7f5f6bb2f700 1 --2- 172.21.15.121:0/3667207186 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5f50038330 0x7f5f5003a7f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5f60006fd0 tx=0x7f5f60006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.840+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3667207186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 msgr2=0x7f5f6419c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.840+0000 7f5f6bb2f700 1 --2- 172.21.15.121:0/3667207186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 0x7f5f6419c270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009130 tx=0x7f5f54011a30 comp rx=0 tx=0).stop 2024-04-05T21:16:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.840+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3667207186 shutdown_connections 2024-04-05T21:16:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.840+0000 7f5f6bb2f700 1 --2- 172.21.15.121:0/3667207186 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5f50038330 0x7f5f5003a7f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.840+0000 7f5f6bb2f700 1 --2- 172.21.15.121:0/3667207186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6407bdf0 0x7f5f6419c270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.840+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3667207186 >> 172.21.15.121:0/3667207186 conn(0x7f5f64101910 msgr2=0x7f5f64103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.840+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3667207186 shutdown_connections 2024-04-05T21:16:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:39.840+0000 7f5f6bb2f700 1 -- 172.21.15.121:0/3667207186 wait complete. 2024-04-05T21:16:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-05T21:16:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:Adding key to root@localhost authorized_keys... 2024-04-05T21:16:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:Adding host smithi121... 2024-04-05T21:16:41.416 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:41 smithi121 ceph-mon[39723]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:16:42.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:42 smithi121 ceph-mon[39723]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi121", "addr": "172.21.15.121", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:16:43.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:43 smithi121 ceph-mon[39723]: Deploying cephadm binary to smithi121 2024-04-05T21:16:44.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Added host 'smithi121' with addr '172.21.15.121' 2024-04-05T21:16:44.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.096+0000 7ff4f1e01700 1 Processor -- start 2024-04-05T21:16:44.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.096+0000 7ff4f1e01700 1 -- start start 2024-04-05T21:16:44.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.096+0000 7ff4f1e01700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 0x7ff4ec1091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:44.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.096+0000 7ff4f1e01700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec079d80 con 0x7ff4ec106e00 2024-04-05T21:16:44.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.096+0000 7ff4eb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 0x7ff4ec1091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:44.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.096+0000 7ff4eb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 0x7ff4ec1091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39578/0 (socket says 172.21.15.121:39578) 2024-04-05T21:16:44.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.096+0000 7ff4eb7fe700 1 -- 172.21.15.121:0/4070088730 learned_addr learned my addr 172.21.15.121:0/4070088730 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:44.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.097+0000 7ff4eb7fe700 1 -- 172.21.15.121:0/4070088730 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ec109730 con 0x7ff4ec106e00 2024-04-05T21:16:44.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.097+0000 7ff4eb7fe700 1 --2- 172.21.15.121:0/4070088730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 0x7ff4ec1091f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff4e00098a0 tx=0x7ff4e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45586962497cea0e server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:44.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.097+0000 7ff4ea7fc700 1 -- 172.21.15.121:0/4070088730 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4e0004180 con 0x7ff4ec106e00 2024-04-05T21:16:44.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.097+0000 7ff4ea7fc700 1 -- 172.21.15.121:0/4070088730 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7ff4e00042e0 con 0x7ff4ec106e00 2024-04-05T21:16:44.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.097+0000 7ff4ea7fc700 1 -- 172.21.15.121:0/4070088730 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4e00119d0 con 0x7ff4ec106e00 2024-04-05T21:16:44.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.097+0000 7ff4f1e01700 1 -- 172.21.15.121:0/4070088730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 msgr2=0x7ff4ec1091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:44.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.097+0000 7ff4f1e01700 1 --2- 172.21.15.121:0/4070088730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 0x7ff4ec1091f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff4e00098a0 tx=0x7ff4e0009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:44.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.098+0000 7ff4f1e01700 1 -- 172.21.15.121:0/4070088730 shutdown_connections 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.098+0000 7ff4f1e01700 1 --2- 172.21.15.121:0/4070088730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 0x7ff4ec1091f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.098+0000 7ff4f1e01700 1 -- 172.21.15.121:0/4070088730 >> 172.21.15.121:0/4070088730 conn(0x7ff4ec1007d0 msgr2=0x7ff4ec102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.098+0000 7ff4f1e01700 1 -- 172.21.15.121:0/4070088730 shutdown_connections 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.098+0000 7ff4f1e01700 1 -- 172.21.15.121:0/4070088730 wait complete. 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.098+0000 7ff4f1e01700 1 Processor -- start 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.098+0000 7ff4f1e01700 1 -- start start 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.099+0000 7ff4f1e01700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 0x7ff4ec199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.099+0000 7ff4f1e01700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec19a530 con 0x7ff4ec106e00 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.099+0000 7ff4eb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 0x7ff4ec199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.099+0000 7ff4eb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 0x7ff4ec199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39580/0 (socket says 172.21.15.121:39580) 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.099+0000 7ff4eb7fe700 1 -- 172.21.15.121:0/668506216 learned_addr learned my addr 172.21.15.121:0/668506216 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.099+0000 7ff4eb7fe700 1 -- 172.21.15.121:0/668506216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e0009550 con 0x7ff4ec106e00 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.099+0000 7ff4eb7fe700 1 --2- 172.21.15.121:0/668506216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 0x7ff4ec199ff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0006af0 tx=0x7ff4e0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.099+0000 7ff4e8ff9700 1 -- 172.21.15.121:0/668506216 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4e0024430 con 0x7ff4ec106e00 2024-04-05T21:16:44.159 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.099+0000 7ff4f1e01700 1 -- 172.21.15.121:0/668506216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ec19a730 con 0x7ff4ec106e00 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.099+0000 7ff4f1e01700 1 -- 172.21.15.121:0/668506216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ec19ab50 con 0x7ff4ec106e00 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.099+0000 7ff4e8ff9700 1 -- 172.21.15.121:0/668506216 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7ff4e00249e0 con 0x7ff4ec106e00 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.099+0000 7ff4e8ff9700 1 -- 172.21.15.121:0/668506216 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4e001a850 con 0x7ff4ec106e00 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.100+0000 7ff4e8ff9700 1 -- 172.21.15.121:0/668506216 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7ff4e001a9b0 con 0x7ff4ec106e00 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.100+0000 7ff4e8ff9700 1 --2- 172.21.15.121:0/668506216 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7ff4d403c730 0x7ff4d403ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.100+0000 7ff4e8ff9700 1 -- 172.21.15.121:0/668506216 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff4e004d270 con 0x7ff4ec106e00 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.100+0000 7ff4eaffd700 1 --2- 172.21.15.121:0/668506216 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7ff4d403c730 0x7ff4d403ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.100+0000 7ff4f1e01700 1 -- 172.21.15.121:0/668506216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d8005320 con 0x7ff4ec106e00 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.101+0000 7ff4eaffd700 1 --2- 172.21.15.121:0/668506216 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7ff4d403c730 0x7ff4d403ebf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc006fd0 tx=0x7ff4dc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.105+0000 7ff4e8ff9700 1 -- 172.21.15.121:0/668506216 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff4e0023390 con 0x7ff4ec106e00 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:41.295+0000 7ff4f1e01700 1 -- 172.21.15.121:0/668506216 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi121", "addr": "172.21.15.121", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4d8000bf0 con 0x7ff4d403c730 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:43.407+0000 7ff4e8ff9700 1 -- 172.21.15.121:0/668506216 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff4d8000bf0 con 0x7ff4d403c730 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:43.410+0000 7ff4f1e01700 1 -- 172.21.15.121:0/668506216 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7ff4d403c730 msgr2=0x7ff4d403ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:43.410+0000 7ff4f1e01700 1 --2- 172.21.15.121:0/668506216 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7ff4d403c730 0x7ff4d403ebf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc006fd0 tx=0x7ff4dc006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:44.161 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:43.410+0000 7ff4f1e01700 1 -- 172.21.15.121:0/668506216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 msgr2=0x7ff4ec199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:44.161 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:43.410+0000 7ff4f1e01700 1 --2- 172.21.15.121:0/668506216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 0x7ff4ec199ff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0006af0 tx=0x7ff4e0011a30 comp rx=0 tx=0).stop 2024-04-05T21:16:44.161 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:43.410+0000 7ff4f1e01700 1 -- 172.21.15.121:0/668506216 shutdown_connections 2024-04-05T21:16:44.161 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:43.410+0000 7ff4f1e01700 1 --2- 172.21.15.121:0/668506216 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7ff4d403c730 0x7ff4d403ebf0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:44.161 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:43.410+0000 7ff4f1e01700 1 --2- 172.21.15.121:0/668506216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ec106e00 0x7ff4ec199ff0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:44.161 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:43.410+0000 7ff4f1e01700 1 -- 172.21.15.121:0/668506216 >> 172.21.15.121:0/668506216 conn(0x7ff4ec1007d0 msgr2=0x7ff4ec101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:44.161 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:43.411+0000 7ff4f1e01700 1 -- 172.21.15.121:0/668506216 shutdown_connections 2024-04-05T21:16:44.161 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:43.411+0000 7ff4f1e01700 1 -- 172.21.15.121:0/668506216 wait complete. 2024-04-05T21:16:44.161 INFO:teuthology.orchestra.run.smithi121.stdout:Deploying mon service with default placement... 2024-04-05T21:16:44.666 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:44 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:44.666 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:44 smithi121 ceph-mon[39723]: Added host smithi121 2024-04-05T21:16:44.666 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:44 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:16:46.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:46 smithi121 ceph-mon[39723]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:16:46.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:46 smithi121 ceph-mon[39723]: Saving service mon spec with placement count:5 2024-04-05T21:16:46.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:46 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:46.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:46 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:46.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-05T21:16:46.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.851+0000 7f60b7fd9700 1 Processor -- start 2024-04-05T21:16:46.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.851+0000 7f60b7fd9700 1 -- start start 2024-04-05T21:16:46.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.851+0000 7f60b7fd9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 0x7f60b0104dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:46.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.851+0000 7f60b7fd9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b0105390 con 0x7f60b01049a0 2024-04-05T21:16:46.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.852+0000 7f60b5d75700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 0x7f60b0104dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:46.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.852+0000 7f60b5d75700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 0x7f60b0104dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36034/0 (socket says 172.21.15.121:36034) 2024-04-05T21:16:46.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.852+0000 7f60b5d75700 1 -- 172.21.15.121:0/1870420559 learned_addr learned my addr 172.21.15.121:0/1870420559 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:46.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.852+0000 7f60b5d75700 1 -- 172.21.15.121:0/1870420559 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b0105ba0 con 0x7f60b01049a0 2024-04-05T21:16:46.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.852+0000 7f60b5d75700 1 --2- 172.21.15.121:0/1870420559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 0x7f60b0104dc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0098a0 tx=0x7f60ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24bf8c2da41a76a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:46.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.852+0000 7f60b4d73700 1 -- 172.21.15.121:0/1870420559 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ac004180 con 0x7f60b01049a0 2024-04-05T21:16:46.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.852+0000 7f60b4d73700 1 -- 172.21.15.121:0/1870420559 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f60ac0042e0 con 0x7f60b01049a0 2024-04-05T21:16:46.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.852+0000 7f60b4d73700 1 -- 172.21.15.121:0/1870420559 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ac0119d0 con 0x7f60b01049a0 2024-04-05T21:16:46.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.853+0000 7f60b7fd9700 1 -- 172.21.15.121:0/1870420559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 msgr2=0x7f60b0104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:46.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.853+0000 7f60b7fd9700 1 --2- 172.21.15.121:0/1870420559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 0x7f60b0104dc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0098a0 tx=0x7f60ac009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:46.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.853+0000 7f60b7fd9700 1 -- 172.21.15.121:0/1870420559 shutdown_connections 2024-04-05T21:16:46.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.853+0000 7f60b7fd9700 1 --2- 172.21.15.121:0/1870420559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 0x7f60b0104dc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:46.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.853+0000 7f60b7fd9700 1 -- 172.21.15.121:0/1870420559 >> 172.21.15.121:0/1870420559 conn(0x7f60b00fff40 msgr2=0x7f60b0102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:46.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.853+0000 7f60b7fd9700 1 -- 172.21.15.121:0/1870420559 shutdown_connections 2024-04-05T21:16:46.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.853+0000 7f60b7fd9700 1 -- 172.21.15.121:0/1870420559 wait complete. 2024-04-05T21:16:46.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.853+0000 7f60b7fd9700 1 Processor -- start 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.854+0000 7f60b7fd9700 1 -- start start 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.854+0000 7f60b7fd9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 0x7f60b0199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.854+0000 7f60b7fd9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b019a530 con 0x7f60b01049a0 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.854+0000 7f60b5d75700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 0x7f60b0199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.854+0000 7f60b5d75700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 0x7f60b0199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36040/0 (socket says 172.21.15.121:36040) 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.854+0000 7f60b5d75700 1 -- 172.21.15.121:0/3347097970 learned_addr learned my addr 172.21.15.121:0/3347097970 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.854+0000 7f60b5d75700 1 -- 172.21.15.121:0/3347097970 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac009550 con 0x7f60b01049a0 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.854+0000 7f60b5d75700 1 --2- 172.21.15.121:0/3347097970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 0x7f60b0199ff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f60ac006af0 tx=0x7f60ac011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.854+0000 7f60a6ffd700 1 -- 172.21.15.121:0/3347097970 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ac024430 con 0x7f60b01049a0 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.854+0000 7f60a6ffd700 1 -- 172.21.15.121:0/3347097970 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1002+0+0 (secure 0 0 0) 0x7f60ac0249e0 con 0x7f60b01049a0 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.854+0000 7f60b7fd9700 1 -- 172.21.15.121:0/3347097970 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b019a730 con 0x7f60b01049a0 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.854+0000 7f60a6ffd700 1 -- 172.21.15.121:0/3347097970 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ac01a970 con 0x7f60b01049a0 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.854+0000 7f60b7fd9700 1 -- 172.21.15.121:0/3347097970 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b019ab50 con 0x7f60b01049a0 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.855+0000 7f60a6ffd700 1 -- 172.21.15.121:0/3347097970 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f60ac01abe0 con 0x7f60b01049a0 2024-04-05T21:16:46.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.855+0000 7f60b7fd9700 1 -- 172.21.15.121:0/3347097970 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b00623c0 con 0x7f60b01049a0 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.855+0000 7f60a6ffd700 1 --2- 172.21.15.121:0/3347097970 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f609c038370 0x7f609c03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.855+0000 7f60a6ffd700 1 -- 172.21.15.121:0/3347097970 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f60ac024b50 con 0x7f60b01049a0 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.855+0000 7f60b5574700 1 --2- 172.21.15.121:0/3347097970 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f609c038370 0x7f609c03a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.856+0000 7f60b5574700 1 --2- 172.21.15.121:0/3347097970 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f609c038370 0x7f609c03a830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f60a0006fd0 tx=0x7f60a0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:44.860+0000 7f60a6ffd700 1 -- 172.21.15.121:0/3347097970 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f60ac04ee60 con 0x7f60b01049a0 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:45.066+0000 7f60b7fd9700 1 -- 172.21.15.121:0/3347097970 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f60b01013b0 con 0x7f609c038370 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:45.071+0000 7f60a6ffd700 1 -- 172.21.15.121:0/3347097970 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f60b01013b0 con 0x7f609c038370 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:45.073+0000 7f60b7fd9700 1 -- 172.21.15.121:0/3347097970 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f609c038370 msgr2=0x7f609c03a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:45.073+0000 7f60b7fd9700 1 --2- 172.21.15.121:0/3347097970 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f609c038370 0x7f609c03a830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f60a0006fd0 tx=0x7f60a0006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:45.073+0000 7f60b7fd9700 1 -- 172.21.15.121:0/3347097970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 msgr2=0x7f60b0199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:45.073+0000 7f60b7fd9700 1 --2- 172.21.15.121:0/3347097970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 0x7f60b0199ff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f60ac006af0 tx=0x7f60ac011a30 comp rx=0 tx=0).stop 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:45.073+0000 7f60b7fd9700 1 -- 172.21.15.121:0/3347097970 shutdown_connections 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:45.073+0000 7f60b7fd9700 1 --2- 172.21.15.121:0/3347097970 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f609c038370 0x7f609c03a830 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:46.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:45.073+0000 7f60b7fd9700 1 --2- 172.21.15.121:0/3347097970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60b01049a0 0x7f60b0199ff0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:46.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:45.073+0000 7f60b7fd9700 1 -- 172.21.15.121:0/3347097970 >> 172.21.15.121:0/3347097970 conn(0x7f60b00fff40 msgr2=0x7f60b0100ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:46.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:45.073+0000 7f60b7fd9700 1 -- 172.21.15.121:0/3347097970 shutdown_connections 2024-04-05T21:16:46.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:45.073+0000 7f60b7fd9700 1 -- 172.21.15.121:0/3347097970 wait complete. 2024-04-05T21:16:46.349 INFO:teuthology.orchestra.run.smithi121.stdout:Deploying mgr service with default placement... 2024-04-05T21:16:47.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:47 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:47.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:47 smithi121 ceph-mon[39723]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:16:47.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:47 smithi121 ceph-mon[39723]: Saving service mgr spec with placement count:2 2024-04-05T21:16:47.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:47 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:48.196 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-05T21:16:48.196 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.952+0000 7f903633b700 1 Processor -- start 2024-04-05T21:16:48.196 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.952+0000 7f903633b700 1 -- start start 2024-04-05T21:16:48.196 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.952+0000 7f903633b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 0x7f9030105650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:48.196 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.952+0000 7f903633b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9030105c20 con 0x7f9030105230 2024-04-05T21:16:48.196 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.952+0000 7f902ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 0x7f9030105650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:48.196 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.953+0000 7f902ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 0x7f9030105650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36052/0 (socket says 172.21.15.121:36052) 2024-04-05T21:16:48.196 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.953+0000 7f902ffff700 1 -- 172.21.15.121:0/445093411 learned_addr learned my addr 172.21.15.121:0/445093411 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:48.196 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.953+0000 7f902ffff700 1 -- 172.21.15.121:0/445093411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9030106430 con 0x7f9030105230 2024-04-05T21:16:48.196 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.953+0000 7f902ffff700 1 --2- 172.21.15.121:0/445093411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 0x7f9030105650 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f90200098a0 tx=0x7f9020009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cf480c33c9f5a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.953+0000 7f902f7fe700 1 -- 172.21.15.121:0/445093411 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9020004180 con 0x7f9030105230 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.953+0000 7f902f7fe700 1 -- 172.21.15.121:0/445093411 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f90200042e0 con 0x7f9030105230 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.953+0000 7f902f7fe700 1 -- 172.21.15.121:0/445093411 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9020011a30 con 0x7f9030105230 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.954+0000 7f903633b700 1 -- 172.21.15.121:0/445093411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 msgr2=0x7f9030105650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.954+0000 7f903633b700 1 --2- 172.21.15.121:0/445093411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 0x7f9030105650 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f90200098a0 tx=0x7f9020009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.954+0000 7f903633b700 1 -- 172.21.15.121:0/445093411 shutdown_connections 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.954+0000 7f903633b700 1 --2- 172.21.15.121:0/445093411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 0x7f9030105650 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.954+0000 7f903633b700 1 -- 172.21.15.121:0/445093411 >> 172.21.15.121:0/445093411 conn(0x7f90301007d0 msgr2=0x7f9030102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.954+0000 7f903633b700 1 -- 172.21.15.121:0/445093411 shutdown_connections 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.954+0000 7f903633b700 1 -- 172.21.15.121:0/445093411 wait complete. 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.955+0000 7f903633b700 1 Processor -- start 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.955+0000 7f903633b700 1 -- start start 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.955+0000 7f903633b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 0x7f9030198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.955+0000 7f903633b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90301994a0 con 0x7f9030105230 2024-04-05T21:16:48.197 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.955+0000 7f902ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 0x7f9030198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:48.198 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.955+0000 7f902ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 0x7f9030198f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36066/0 (socket says 172.21.15.121:36066) 2024-04-05T21:16:48.198 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.955+0000 7f902ffff700 1 -- 172.21.15.121:0/3156663581 learned_addr learned my addr 172.21.15.121:0/3156663581 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:48.198 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.956+0000 7f902ffff700 1 -- 172.21.15.121:0/3156663581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9020009550 con 0x7f9030105230 2024-04-05T21:16:48.198 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.956+0000 7f902ffff700 1 --2- 172.21.15.121:0/3156663581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 0x7f9030198f60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9020006af0 tx=0x7f9020011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:48.198 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.956+0000 7f902dffb700 1 -- 172.21.15.121:0/3156663581 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9020003bc0 con 0x7f9030105230 2024-04-05T21:16:48.198 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.956+0000 7f903633b700 1 -- 172.21.15.121:0/3156663581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90301996a0 con 0x7f9030105230 2024-04-05T21:16:48.198 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.956+0000 7f902dffb700 1 -- 172.21.15.121:0/3156663581 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f902001a8c0 con 0x7f9030105230 2024-04-05T21:16:48.198 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.956+0000 7f902dffb700 1 -- 172.21.15.121:0/3156663581 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9020019820 con 0x7f9030105230 2024-04-05T21:16:48.198 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.956+0000 7f903633b700 1 -- 172.21.15.121:0/3156663581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9030199ac0 con 0x7f9030105230 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.957+0000 7f902dffb700 1 -- 172.21.15.121:0/3156663581 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f9020019aa0 con 0x7f9030105230 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.957+0000 7f902dffb700 1 --2- 172.21.15.121:0/3156663581 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f901c0382e0 0x7f901c03a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.957+0000 7f902dffb700 1 -- 172.21.15.121:0/3156663581 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f902004c550 con 0x7f9030105230 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.957+0000 7f903633b700 1 -- 172.21.15.121:0/3156663581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90300623c0 con 0x7f9030105230 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.957+0000 7f9027fff700 1 --2- 172.21.15.121:0/3156663581 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f901c0382e0 0x7f901c03a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.958+0000 7f9027fff700 1 --2- 172.21.15.121:0/3156663581 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f901c0382e0 0x7f901c03a7a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f901800cd30 tx=0x7f901800b3f0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:46.963+0000 7f902dffb700 1 -- 172.21.15.121:0/3156663581 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f90200243f0 con 0x7f9030105230 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:47.185+0000 7f903633b700 1 -- 172.21.15.121:0/3156663581 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f9030102b10 con 0x7f901c0382e0 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:47.189+0000 7f902dffb700 1 -- 172.21.15.121:0/3156663581 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9030102b10 con 0x7f901c0382e0 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:47.191+0000 7f903633b700 1 -- 172.21.15.121:0/3156663581 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f901c0382e0 msgr2=0x7f901c03a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:47.191+0000 7f903633b700 1 --2- 172.21.15.121:0/3156663581 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f901c0382e0 0x7f901c03a7a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f901800cd30 tx=0x7f901800b3f0 comp rx=0 tx=0).stop 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:47.191+0000 7f903633b700 1 -- 172.21.15.121:0/3156663581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 msgr2=0x7f9030198f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:47.191+0000 7f903633b700 1 --2- 172.21.15.121:0/3156663581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 0x7f9030198f60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9020006af0 tx=0x7f9020011ab0 comp rx=0 tx=0).stop 2024-04-05T21:16:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:47.192+0000 7f903633b700 1 -- 172.21.15.121:0/3156663581 shutdown_connections 2024-04-05T21:16:48.200 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:47.192+0000 7f903633b700 1 --2- 172.21.15.121:0/3156663581 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f901c0382e0 0x7f901c03a7a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:48.200 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:47.192+0000 7f903633b700 1 --2- 172.21.15.121:0/3156663581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9030105230 0x7f9030198f60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:48.200 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:47.192+0000 7f903633b700 1 -- 172.21.15.121:0/3156663581 >> 172.21.15.121:0/3156663581 conn(0x7f90301007d0 msgr2=0x7f9030101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:48.200 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:47.192+0000 7f903633b700 1 -- 172.21.15.121:0/3156663581 shutdown_connections 2024-04-05T21:16:48.200 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:47.192+0000 7f903633b700 1 -- 172.21.15.121:0/3156663581 wait complete. 2024-04-05T21:16:48.200 INFO:teuthology.orchestra.run.smithi121.stdout:Deploying crash service with default placement... 2024-04-05T21:16:49.878 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-05T21:16:49.879 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.937+0000 7fe564e70700 1 Processor -- start 2024-04-05T21:16:49.879 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.937+0000 7fe564e70700 1 -- start start 2024-04-05T21:16:49.879 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.937+0000 7fe564e70700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 0x7fe560104a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:49.879 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.937+0000 7fe564e70700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe560104fd0 con 0x7fe5601045e0 2024-04-05T21:16:49.879 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.937+0000 7fe55e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 0x7fe560104a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:49.879 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.938+0000 7fe55e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 0x7fe560104a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36082/0 (socket says 172.21.15.121:36082) 2024-04-05T21:16:49.879 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.938+0000 7fe55e59c700 1 -- 172.21.15.121:0/1012447702 learned_addr learned my addr 172.21.15.121:0/1012447702 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:49.879 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.938+0000 7fe55e59c700 1 -- 172.21.15.121:0/1012447702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe560105110 con 0x7fe5601045e0 2024-04-05T21:16:49.879 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.938+0000 7fe55e59c700 1 --2- 172.21.15.121:0/1012447702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 0x7fe560104a00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe5540098a0 tx=0x7fe554009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c36657bdde6aa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:49.879 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.938+0000 7fe55d59a700 1 -- 172.21.15.121:0/1012447702 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe554004180 con 0x7fe5601045e0 2024-04-05T21:16:49.880 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.938+0000 7fe55d59a700 1 -- 172.21.15.121:0/1012447702 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fe5540042e0 con 0x7fe5601045e0 2024-04-05T21:16:49.880 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.938+0000 7fe55d59a700 1 -- 172.21.15.121:0/1012447702 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe554011a30 con 0x7fe5601045e0 2024-04-05T21:16:49.880 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.939+0000 7fe564e70700 1 -- 172.21.15.121:0/1012447702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 msgr2=0x7fe560104a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:49.880 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.939+0000 7fe564e70700 1 --2- 172.21.15.121:0/1012447702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 0x7fe560104a00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe5540098a0 tx=0x7fe554009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:49.880 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.939+0000 7fe564e70700 1 -- 172.21.15.121:0/1012447702 shutdown_connections 2024-04-05T21:16:49.880 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.939+0000 7fe564e70700 1 --2- 172.21.15.121:0/1012447702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 0x7fe560104a00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:49.880 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.939+0000 7fe564e70700 1 -- 172.21.15.121:0/1012447702 >> 172.21.15.121:0/1012447702 conn(0x7fe5600fff40 msgr2=0x7fe560102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:49.880 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.939+0000 7fe564e70700 1 -- 172.21.15.121:0/1012447702 shutdown_connections 2024-04-05T21:16:49.880 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.939+0000 7fe564e70700 1 -- 172.21.15.121:0/1012447702 wait complete. 2024-04-05T21:16:49.880 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.940+0000 7fe564e70700 1 Processor -- start 2024-04-05T21:16:49.880 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.940+0000 7fe564e70700 1 -- start start 2024-04-05T21:16:49.880 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.940+0000 7fe564e70700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 0x7fe56019a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:49.880 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.940+0000 7fe564e70700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56019a580 con 0x7fe5601045e0 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.940+0000 7fe55e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 0x7fe56019a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.940+0000 7fe55e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 0x7fe56019a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36092/0 (socket says 172.21.15.121:36092) 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.940+0000 7fe55e59c700 1 -- 172.21.15.121:0/1736272787 learned_addr learned my addr 172.21.15.121:0/1736272787 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.940+0000 7fe55e59c700 1 -- 172.21.15.121:0/1736272787 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe554009550 con 0x7fe5601045e0 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.940+0000 7fe55e59c700 1 --2- 172.21.15.121:0/1736272787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 0x7fe56019a040 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe554000c00 tx=0x7fe554011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.940+0000 7fe54f7fe700 1 -- 172.21.15.121:0/1736272787 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe554003bc0 con 0x7fe5601045e0 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.941+0000 7fe54f7fe700 1 -- 172.21.15.121:0/1736272787 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fe5540248d0 con 0x7fe5601045e0 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.941+0000 7fe564e70700 1 -- 172.21.15.121:0/1736272787 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56019a780 con 0x7fe5601045e0 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.941+0000 7fe54f7fe700 1 -- 172.21.15.121:0/1736272787 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe55401a890 con 0x7fe5601045e0 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.941+0000 7fe564e70700 1 -- 172.21.15.121:0/1736272787 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56019aba0 con 0x7fe5601045e0 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.941+0000 7fe54f7fe700 1 -- 172.21.15.121:0/1736272787 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fe55401ab00 con 0x7fe5601045e0 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.941+0000 7fe564e70700 1 -- 172.21.15.121:0/1736272787 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5600623c0 con 0x7fe5601045e0 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.942+0000 7fe54f7fe700 1 --2- 172.21.15.121:0/1736272787 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe548038370 0x7fe54803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:49.881 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.942+0000 7fe54f7fe700 1 -- 172.21.15.121:0/1736272787 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe55404c530 con 0x7fe5601045e0 2024-04-05T21:16:49.882 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.942+0000 7fe55dd9b700 1 --2- 172.21.15.121:0/1736272787 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe548038370 0x7fe54803a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:49.882 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.942+0000 7fe55dd9b700 1 --2- 172.21.15.121:0/1736272787 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe548038370 0x7fe54803a830 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe550006fd0 tx=0x7fe550006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:49.882 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:48.946+0000 7fe54f7fe700 1 -- 172.21.15.121:0/1736272787 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe55401adb0 con 0x7fe5601045e0 2024-04-05T21:16:49.882 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:49.171+0000 7fe564e70700 1 -- 172.21.15.121:0/1736272787 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fe560102260 con 0x7fe548038370 2024-04-05T21:16:49.882 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:49.178+0000 7fe54f7fe700 1 -- 172.21.15.121:0/1736272787 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fe560102260 con 0x7fe548038370 2024-04-05T21:16:49.882 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:49.180+0000 7fe564e70700 1 -- 172.21.15.121:0/1736272787 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe548038370 msgr2=0x7fe54803a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:49.882 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:49.180+0000 7fe564e70700 1 --2- 172.21.15.121:0/1736272787 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe548038370 0x7fe54803a830 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe550006fd0 tx=0x7fe550006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:49.882 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:49.180+0000 7fe564e70700 1 -- 172.21.15.121:0/1736272787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 msgr2=0x7fe56019a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:49.882 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:49.180+0000 7fe564e70700 1 --2- 172.21.15.121:0/1736272787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 0x7fe56019a040 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe554000c00 tx=0x7fe554011ab0 comp rx=0 tx=0).stop 2024-04-05T21:16:49.882 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:49.181+0000 7fe564e70700 1 -- 172.21.15.121:0/1736272787 shutdown_connections 2024-04-05T21:16:49.882 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:49.181+0000 7fe564e70700 1 --2- 172.21.15.121:0/1736272787 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe548038370 0x7fe54803a830 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:49.882 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:49.181+0000 7fe564e70700 1 --2- 172.21.15.121:0/1736272787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5601045e0 0x7fe56019a040 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:49.882 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:49.181+0000 7fe564e70700 1 -- 172.21.15.121:0/1736272787 >> 172.21.15.121:0/1736272787 conn(0x7fe5600fff40 msgr2=0x7fe560101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:49.883 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:49.181+0000 7fe564e70700 1 -- 172.21.15.121:0/1736272787 shutdown_connections 2024-04-05T21:16:49.883 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:49.181+0000 7fe564e70700 1 -- 172.21.15.121:0/1736272787 wait complete. 2024-04-05T21:16:49.883 INFO:teuthology.orchestra.run.smithi121.stdout:Deploying ceph-exporter service with default placement... 2024-04-05T21:16:50.156 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:50 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:50.156 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:50 smithi121 ceph-mon[39723]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:16:50.156 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:50 smithi121 ceph-mon[39723]: Saving service crash spec with placement * 2024-04-05T21:16:50.156 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:50 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:50.156 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:50 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:51.395 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-04-05T21:16:51.395 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.469+0000 7fe7cb703700 1 Processor -- start 2024-04-05T21:16:51.395 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.470+0000 7fe7cb703700 1 -- start start 2024-04-05T21:16:51.396 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.470+0000 7fe7cb703700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 0x7fe7c4105010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:51.396 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.470+0000 7fe7cb703700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c4079d80 con 0x7fe7c4104bf0 2024-04-05T21:16:51.396 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.470+0000 7fe7c949f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 0x7fe7c4105010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:51.396 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.471+0000 7fe7c949f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 0x7fe7c4105010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36102/0 (socket says 172.21.15.121:36102) 2024-04-05T21:16:51.396 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.471+0000 7fe7c949f700 1 -- 172.21.15.121:0/3463293265 learned_addr learned my addr 172.21.15.121:0/3463293265 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:51.396 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.471+0000 7fe7c949f700 1 -- 172.21.15.121:0/3463293265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c4105550 con 0x7fe7c4104bf0 2024-04-05T21:16:51.396 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.471+0000 7fe7c949f700 1 --2- 172.21.15.121:0/3463293265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 0x7fe7c4105010 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00098a0 tx=0x7fe7c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ff7a6a6f7e3666 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:51.396 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.471+0000 7fe7bbfff700 1 -- 172.21.15.121:0/3463293265 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7c0004180 con 0x7fe7c4104bf0 2024-04-05T21:16:51.396 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.471+0000 7fe7bbfff700 1 -- 172.21.15.121:0/3463293265 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fe7c00042e0 con 0x7fe7c4104bf0 2024-04-05T21:16:51.396 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.471+0000 7fe7bbfff700 1 -- 172.21.15.121:0/3463293265 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7c0011a30 con 0x7fe7c4104bf0 2024-04-05T21:16:51.396 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.472+0000 7fe7cb703700 1 -- 172.21.15.121:0/3463293265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 msgr2=0x7fe7c4105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:51.396 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.472+0000 7fe7cb703700 1 --2- 172.21.15.121:0/3463293265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 0x7fe7c4105010 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00098a0 tx=0x7fe7c0009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:51.396 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.472+0000 7fe7cb703700 1 -- 172.21.15.121:0/3463293265 shutdown_connections 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.472+0000 7fe7cb703700 1 --2- 172.21.15.121:0/3463293265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 0x7fe7c4105010 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.472+0000 7fe7cb703700 1 -- 172.21.15.121:0/3463293265 >> 172.21.15.121:0/3463293265 conn(0x7fe7c41007d0 msgr2=0x7fe7c4102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.472+0000 7fe7cb703700 1 -- 172.21.15.121:0/3463293265 shutdown_connections 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.472+0000 7fe7cb703700 1 -- 172.21.15.121:0/3463293265 wait complete. 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7cb703700 1 Processor -- start 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7cb703700 1 -- start start 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7cb703700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 0x7fe7c419c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7cb703700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c419c760 con 0x7fe7c4104bf0 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7c949f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 0x7fe7c419c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7c949f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 0x7fe7c419c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36114/0 (socket says 172.21.15.121:36114) 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7c949f700 1 -- 172.21.15.121:0/2339463596 learned_addr learned my addr 172.21.15.121:0/2339463596 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7c949f700 1 -- 172.21.15.121:0/2339463596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c0009550 con 0x7fe7c4104bf0 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7c949f700 1 --2- 172.21.15.121:0/2339463596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 0x7fe7c419c220 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0006af0 tx=0x7fe7c0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7ba7fc700 1 -- 172.21.15.121:0/2339463596 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7c0003bc0 con 0x7fe7c4104bf0 2024-04-05T21:16:51.397 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7cb703700 1 -- 172.21.15.121:0/2339463596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c419c960 con 0x7fe7c4104bf0 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7ba7fc700 1 -- 172.21.15.121:0/2339463596 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fe7c001a8c0 con 0x7fe7c4104bf0 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7ba7fc700 1 -- 172.21.15.121:0/2339463596 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7c0019860 con 0x7fe7c4104bf0 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.473+0000 7fe7cb703700 1 -- 172.21.15.121:0/2339463596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c419cd80 con 0x7fe7c4104bf0 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.474+0000 7fe7ba7fc700 1 -- 172.21.15.121:0/2339463596 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fe7c0019ad0 con 0x7fe7c4104bf0 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.474+0000 7fe7ba7fc700 1 --2- 172.21.15.121:0/2339463596 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe7b0038330 0x7fe7b003a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.475+0000 7fe7ba7fc700 1 -- 172.21.15.121:0/2339463596 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe7c004c8d0 con 0x7fe7c4104bf0 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.475+0000 7fe7c8c9e700 1 --2- 172.21.15.121:0/2339463596 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe7b0038330 0x7fe7b003a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.475+0000 7fe7cb703700 1 -- 172.21.15.121:0/2339463596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c4195b30 con 0x7fe7c4104bf0 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.475+0000 7fe7c8c9e700 1 --2- 172.21.15.121:0/2339463596 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe7b0038330 0x7fe7b003a7f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4006fd0 tx=0x7fe7b4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.481+0000 7fe7ba7fc700 1 -- 172.21.15.121:0/2339463596 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe7c0019dd0 con 0x7fe7c4104bf0 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.697+0000 7fe7cb703700 1 -- 172.21.15.121:0/2339463596 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7c40611d0 con 0x7fe7b0038330 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.702+0000 7fe7ba7fc700 1 -- 172.21.15.121:0/2339463596 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fe7c40611d0 con 0x7fe7b0038330 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.704+0000 7fe7cb703700 1 -- 172.21.15.121:0/2339463596 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe7b0038330 msgr2=0x7fe7b003a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.704+0000 7fe7cb703700 1 --2- 172.21.15.121:0/2339463596 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe7b0038330 0x7fe7b003a7f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4006fd0 tx=0x7fe7b4006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:51.399 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.704+0000 7fe7cb703700 1 -- 172.21.15.121:0/2339463596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 msgr2=0x7fe7c419c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:51.399 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.704+0000 7fe7cb703700 1 --2- 172.21.15.121:0/2339463596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 0x7fe7c419c220 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0006af0 tx=0x7fe7c0011ab0 comp rx=0 tx=0).stop 2024-04-05T21:16:51.399 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.704+0000 7fe7cb703700 1 -- 172.21.15.121:0/2339463596 shutdown_connections 2024-04-05T21:16:51.399 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.704+0000 7fe7cb703700 1 --2- 172.21.15.121:0/2339463596 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe7b0038330 0x7fe7b003a7f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:51.399 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.704+0000 7fe7cb703700 1 --2- 172.21.15.121:0/2339463596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7c4104bf0 0x7fe7c419c220 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:51.399 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.704+0000 7fe7cb703700 1 -- 172.21.15.121:0/2339463596 >> 172.21.15.121:0/2339463596 conn(0x7fe7c41007d0 msgr2=0x7fe7c41907b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:51.399 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.704+0000 7fe7cb703700 1 -- 172.21.15.121:0/2339463596 shutdown_connections 2024-04-05T21:16:51.399 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:50.704+0000 7fe7cb703700 1 -- 172.21.15.121:0/2339463596 wait complete. 2024-04-05T21:16:51.399 INFO:teuthology.orchestra.run.smithi121.stdout:Deploying prometheus service with default placement... 2024-04-05T21:16:51.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:51 smithi121 ceph-mon[39723]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:16:51.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:51 smithi121 ceph-mon[39723]: Saving service ceph-exporter spec with placement * 2024-04-05T21:16:51.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:51 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:53.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-05T21:16:53.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.078+0000 7f3d84d2e700 1 Processor -- start 2024-04-05T21:16:53.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.078+0000 7f3d84d2e700 1 -- start start 2024-04-05T21:16:53.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.079+0000 7f3d84d2e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 0x7f3d801091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:53.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.079+0000 7f3d84d2e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d800744f0 con 0x7f3d80106de0 2024-04-05T21:16:53.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.079+0000 7f3d7e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 0x7f3d801091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:53.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.079+0000 7f3d7e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 0x7f3d801091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36122/0 (socket says 172.21.15.121:36122) 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.079+0000 7f3d7e59c700 1 -- 172.21.15.121:0/2217118033 learned_addr learned my addr 172.21.15.121:0/2217118033 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.079+0000 7f3d7e59c700 1 -- 172.21.15.121:0/2217118033 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d80109710 con 0x7f3d80106de0 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.079+0000 7f3d7e59c700 1 --2- 172.21.15.121:0/2217118033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 0x7f3d801091d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3d740098a0 tx=0x7f3d74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437ef6c0faf9629f server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.079+0000 7f3d7d59a700 1 -- 172.21.15.121:0/2217118033 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d74004180 con 0x7f3d80106de0 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.079+0000 7f3d7d59a700 1 -- 172.21.15.121:0/2217118033 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f3d740042e0 con 0x7f3d80106de0 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.079+0000 7f3d7d59a700 1 -- 172.21.15.121:0/2217118033 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d74011a30 con 0x7f3d80106de0 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.080+0000 7f3d84d2e700 1 -- 172.21.15.121:0/2217118033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 msgr2=0x7f3d801091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.080+0000 7f3d84d2e700 1 --2- 172.21.15.121:0/2217118033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 0x7f3d801091d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3d740098a0 tx=0x7f3d74009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.080+0000 7f3d84d2e700 1 -- 172.21.15.121:0/2217118033 shutdown_connections 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.080+0000 7f3d84d2e700 1 --2- 172.21.15.121:0/2217118033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 0x7f3d801091d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.080+0000 7f3d84d2e700 1 -- 172.21.15.121:0/2217118033 >> 172.21.15.121:0/2217118033 conn(0x7f3d801007f0 msgr2=0x7f3d80102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.080+0000 7f3d84d2e700 1 -- 172.21.15.121:0/2217118033 shutdown_connections 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.080+0000 7f3d84d2e700 1 -- 172.21.15.121:0/2217118033 wait complete. 2024-04-05T21:16:53.021 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.081+0000 7f3d84d2e700 1 Processor -- start 2024-04-05T21:16:53.022 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.081+0000 7f3d84d2e700 1 -- start start 2024-04-05T21:16:53.022 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.081+0000 7f3d84d2e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 0x7f3d8019a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:53.022 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.081+0000 7f3d84d2e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8019a580 con 0x7f3d80106de0 2024-04-05T21:16:53.022 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.081+0000 7f3d7e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 0x7f3d8019a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:53.022 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.081+0000 7f3d7e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 0x7f3d8019a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36128/0 (socket says 172.21.15.121:36128) 2024-04-05T21:16:53.022 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.081+0000 7f3d7e59c700 1 -- 172.21.15.121:0/3712445115 learned_addr learned my addr 172.21.15.121:0/3712445115 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:53.022 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.081+0000 7f3d7e59c700 1 -- 172.21.15.121:0/3712445115 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d74009550 con 0x7f3d80106de0 2024-04-05T21:16:53.022 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.081+0000 7f3d7e59c700 1 --2- 172.21.15.121:0/3712445115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 0x7f3d8019a040 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3d74006af0 tx=0x7f3d74011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:53.022 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.081+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/3712445115 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d74003a80 con 0x7f3d80106de0 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.081+0000 7f3d84d2e700 1 -- 172.21.15.121:0/3712445115 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8019a780 con 0x7f3d80106de0 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.081+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/3712445115 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f3d74003be0 con 0x7f3d80106de0 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.081+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/3712445115 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d74019770 con 0x7f3d80106de0 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.082+0000 7f3d84d2e700 1 -- 172.21.15.121:0/3712445115 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8019aba0 con 0x7f3d80106de0 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.082+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/3712445115 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f3d74019a30 con 0x7f3d80106de0 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.082+0000 7f3d84d2e700 1 -- 172.21.15.121:0/3712445115 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d800623c0 con 0x7f3d80106de0 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.082+0000 7f3d6f7fe700 1 --2- 172.21.15.121:0/3712445115 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f3d680382e0 0x7f3d6803a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.082+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/3712445115 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3d7404c6d0 con 0x7f3d80106de0 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.083+0000 7f3d7dd9b700 1 --2- 172.21.15.121:0/3712445115 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f3d680382e0 0x7f3d6803a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.083+0000 7f3d7dd9b700 1 --2- 172.21.15.121:0/3712445115 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f3d680382e0 0x7f3d6803a7a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3d70006fd0 tx=0x7f3d70006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.087+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/3712445115 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3d7402b750 con 0x7f3d80106de0 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.288+0000 7f3d84d2e700 1 -- 172.21.15.121:0/3712445115 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d80102b00 con 0x7f3d680382e0 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.292+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/3712445115 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f3d80102b00 con 0x7f3d680382e0 2024-04-05T21:16:53.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.294+0000 7f3d84d2e700 1 -- 172.21.15.121:0/3712445115 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f3d680382e0 msgr2=0x7f3d6803a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:53.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.294+0000 7f3d84d2e700 1 --2- 172.21.15.121:0/3712445115 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f3d680382e0 0x7f3d6803a7a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3d70006fd0 tx=0x7f3d70006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:53.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.294+0000 7f3d84d2e700 1 -- 172.21.15.121:0/3712445115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 msgr2=0x7f3d8019a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:53.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.294+0000 7f3d84d2e700 1 --2- 172.21.15.121:0/3712445115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 0x7f3d8019a040 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3d74006af0 tx=0x7f3d74011b20 comp rx=0 tx=0).stop 2024-04-05T21:16:53.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.294+0000 7f3d84d2e700 1 -- 172.21.15.121:0/3712445115 shutdown_connections 2024-04-05T21:16:53.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.294+0000 7f3d84d2e700 1 --2- 172.21.15.121:0/3712445115 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f3d680382e0 0x7f3d6803a7a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:53.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.295+0000 7f3d84d2e700 1 --2- 172.21.15.121:0/3712445115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d80106de0 0x7f3d8019a040 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:53.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.295+0000 7f3d84d2e700 1 -- 172.21.15.121:0/3712445115 >> 172.21.15.121:0/3712445115 conn(0x7f3d801007f0 msgr2=0x7f3d80101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:53.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.295+0000 7f3d84d2e700 1 -- 172.21.15.121:0/3712445115 shutdown_connections 2024-04-05T21:16:53.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:52.295+0000 7f3d84d2e700 1 -- 172.21.15.121:0/3712445115 wait complete. 2024-04-05T21:16:53.024 INFO:teuthology.orchestra.run.smithi121.stdout:Deploying grafana service with default placement... 2024-04-05T21:16:53.580 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:53 smithi121 ceph-mon[39723]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:16:53.581 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:53 smithi121 ceph-mon[39723]: Saving service prometheus spec with placement count:1 2024-04-05T21:16:53.581 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:53 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:54.451 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:54 smithi121 ceph-mon[39723]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:16:54.451 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:54 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:54.745 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-05T21:16:54.745 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.921+0000 7f89d957d700 1 Processor -- start 2024-04-05T21:16:54.745 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.921+0000 7f89d957d700 1 -- start start 2024-04-05T21:16:54.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.921+0000 7f89d957d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 0x7f89d4109250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:54.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.921+0000 7f89d957d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d4079d20 con 0x7f89d4106e60 2024-04-05T21:16:54.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.921+0000 7f89d2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 0x7f89d4109250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:54.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.921+0000 7f89d2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 0x7f89d4109250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36140/0 (socket says 172.21.15.121:36140) 2024-04-05T21:16:54.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.921+0000 7f89d2ffd700 1 -- 172.21.15.121:0/2470678034 learned_addr learned my addr 172.21.15.121:0/2470678034 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:54.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.921+0000 7f89d2ffd700 1 -- 172.21.15.121:0/2470678034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d4079e60 con 0x7f89d4106e60 2024-04-05T21:16:54.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.922+0000 7f89d2ffd700 1 --2- 172.21.15.121:0/2470678034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 0x7f89d4109250 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f89c40098a0 tx=0x7f89c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d07497a1eb652ef server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:54.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.922+0000 7f89d1ffb700 1 -- 172.21.15.121:0/2470678034 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89c4004180 con 0x7f89d4106e60 2024-04-05T21:16:54.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.922+0000 7f89d1ffb700 1 -- 172.21.15.121:0/2470678034 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f89c40042e0 con 0x7f89d4106e60 2024-04-05T21:16:54.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.922+0000 7f89d1ffb700 1 -- 172.21.15.121:0/2470678034 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89c4011a30 con 0x7f89d4106e60 2024-04-05T21:16:54.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.922+0000 7f89d957d700 1 -- 172.21.15.121:0/2470678034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 msgr2=0x7f89d4109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:54.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.922+0000 7f89d957d700 1 --2- 172.21.15.121:0/2470678034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 0x7f89d4109250 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f89c40098a0 tx=0x7f89c4009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:54.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.922+0000 7f89d957d700 1 -- 172.21.15.121:0/2470678034 shutdown_connections 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.922+0000 7f89d957d700 1 --2- 172.21.15.121:0/2470678034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 0x7f89d4109250 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.922+0000 7f89d957d700 1 -- 172.21.15.121:0/2470678034 >> 172.21.15.121:0/2470678034 conn(0x7f89d41007d0 msgr2=0x7f89d4102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.922+0000 7f89d957d700 1 -- 172.21.15.121:0/2470678034 shutdown_connections 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.922+0000 7f89d957d700 1 -- 172.21.15.121:0/2470678034 wait complete. 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.923+0000 7f89d957d700 1 Processor -- start 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.923+0000 7f89d957d700 1 -- start start 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.923+0000 7f89d957d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 0x7f89d41067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.923+0000 7f89d957d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d4106d00 con 0x7f89d4106e60 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.923+0000 7f89d2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 0x7f89d41067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.923+0000 7f89d2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 0x7f89d41067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36148/0 (socket says 172.21.15.121:36148) 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.923+0000 7f89d2ffd700 1 -- 172.21.15.121:0/2930688195 learned_addr learned my addr 172.21.15.121:0/2930688195 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.924+0000 7f89d2ffd700 1 -- 172.21.15.121:0/2930688195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c4009550 con 0x7f89d4106e60 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.924+0000 7f89d2ffd700 1 --2- 172.21.15.121:0/2930688195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 0x7f89d41067c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f89c4011a00 tx=0x7f89c4011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:54.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.924+0000 7f89bbfff700 1 -- 172.21.15.121:0/2930688195 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89c4003bc0 con 0x7f89d4106e60 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.924+0000 7f89bbfff700 1 -- 172.21.15.121:0/2930688195 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f89c40248c0 con 0x7f89d4106e60 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.924+0000 7f89bbfff700 1 -- 172.21.15.121:0/2930688195 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89c401a850 con 0x7f89d4106e60 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.924+0000 7f89d957d700 1 -- 172.21.15.121:0/2930688195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89d4104e10 con 0x7f89d4106e60 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.924+0000 7f89d957d700 1 -- 172.21.15.121:0/2930688195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89d4105230 con 0x7f89d4106e60 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.925+0000 7f89bbfff700 1 -- 172.21.15.121:0/2930688195 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f89c401ab10 con 0x7f89d4106e60 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.925+0000 7f89bbfff700 1 --2- 172.21.15.121:0/2930688195 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f89bc038320 0x7f89bc03a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.925+0000 7f89d957d700 1 -- 172.21.15.121:0/2930688195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d410a450 con 0x7f89d4106e60 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.925+0000 7f89bbfff700 1 -- 172.21.15.121:0/2930688195 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f89c404c890 con 0x7f89d4106e60 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.925+0000 7f89d27fc700 1 --2- 172.21.15.121:0/2930688195 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f89bc038320 0x7f89bc03a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.925+0000 7f89d27fc700 1 --2- 172.21.15.121:0/2930688195 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f89bc038320 0x7f89bc03a7e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f89c8006fd0 tx=0x7f89c8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:53.930+0000 7f89bbfff700 1 -- 172.21.15.121:0/2930688195 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f89c4022330 con 0x7f89d4106e60 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:54.128+0000 7f89d957d700 1 -- 172.21.15.121:0/2930688195 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f89d4102b10 con 0x7f89bc038320 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:54.133+0000 7f89bbfff700 1 -- 172.21.15.121:0/2930688195 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f89d4102b10 con 0x7f89bc038320 2024-04-05T21:16:54.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:54.135+0000 7f89d957d700 1 -- 172.21.15.121:0/2930688195 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f89bc038320 msgr2=0x7f89bc03a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:54.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:54.135+0000 7f89d957d700 1 --2- 172.21.15.121:0/2930688195 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f89bc038320 0x7f89bc03a7e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f89c8006fd0 tx=0x7f89c8006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:54.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:54.135+0000 7f89d957d700 1 -- 172.21.15.121:0/2930688195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 msgr2=0x7f89d41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:54.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:54.135+0000 7f89d957d700 1 --2- 172.21.15.121:0/2930688195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 0x7f89d41067c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f89c4011a00 tx=0x7f89c4011ae0 comp rx=0 tx=0).stop 2024-04-05T21:16:54.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:54.135+0000 7f89d957d700 1 -- 172.21.15.121:0/2930688195 shutdown_connections 2024-04-05T21:16:54.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:54.135+0000 7f89d957d700 1 --2- 172.21.15.121:0/2930688195 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f89bc038320 0x7f89bc03a7e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:54.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:54.135+0000 7f89d957d700 1 --2- 172.21.15.121:0/2930688195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d4106e60 0x7f89d41067c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:54.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:54.135+0000 7f89d957d700 1 -- 172.21.15.121:0/2930688195 >> 172.21.15.121:0/2930688195 conn(0x7f89d41007d0 msgr2=0x7f89d4101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:54.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:54.135+0000 7f89d957d700 1 -- 172.21.15.121:0/2930688195 shutdown_connections 2024-04-05T21:16:54.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:54.135+0000 7f89d957d700 1 -- 172.21.15.121:0/2930688195 wait complete. 2024-04-05T21:16:54.749 INFO:teuthology.orchestra.run.smithi121.stdout:Deploying node-exporter service with default placement... 2024-04-05T21:16:55.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:55 smithi121 ceph-mon[39723]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:16:55.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:55 smithi121 ceph-mon[39723]: Saving service grafana spec with placement count:1 2024-04-05T21:16:56.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-05T21:16:56.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.322+0000 7f891c76e700 1 Processor -- start 2024-04-05T21:16:56.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.323+0000 7f891c76e700 1 -- start start 2024-04-05T21:16:56.103 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.323+0000 7f891c76e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 0x7f8914109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:56.104 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.323+0000 7f891c76e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8914074530 con 0x7f8914106da0 2024-04-05T21:16:56.104 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.323+0000 7f891a50a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 0x7f8914109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:56.104 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.323+0000 7f891a50a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 0x7f8914109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59068/0 (socket says 172.21.15.121:59068) 2024-04-05T21:16:56.104 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.323+0000 7f891a50a700 1 -- 172.21.15.121:0/370535329 learned_addr learned my addr 172.21.15.121:0/370535329 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:56.104 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.323+0000 7f891a50a700 1 -- 172.21.15.121:0/370535329 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89141096d0 con 0x7f8914106da0 2024-04-05T21:16:56.104 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.323+0000 7f891a50a700 1 --2- 172.21.15.121:0/370535329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 0x7f8914109190 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f89080098a0 tx=0x7f8908009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ce41a18d375680 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:56.105 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.324+0000 7f8919508700 1 -- 172.21.15.121:0/370535329 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8908004180 con 0x7f8914106da0 2024-04-05T21:16:56.105 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.324+0000 7f8919508700 1 -- 172.21.15.121:0/370535329 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f89080042e0 con 0x7f8914106da0 2024-04-05T21:16:56.105 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.324+0000 7f8919508700 1 -- 172.21.15.121:0/370535329 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8908011a30 con 0x7f8914106da0 2024-04-05T21:16:56.105 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.324+0000 7f891c76e700 1 -- 172.21.15.121:0/370535329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 msgr2=0x7f8914109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:56.105 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.324+0000 7f891c76e700 1 --2- 172.21.15.121:0/370535329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 0x7f8914109190 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f89080098a0 tx=0x7f8908009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:56.105 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.324+0000 7f891c76e700 1 -- 172.21.15.121:0/370535329 shutdown_connections 2024-04-05T21:16:56.105 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.324+0000 7f891c76e700 1 --2- 172.21.15.121:0/370535329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 0x7f8914109190 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:56.105 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.324+0000 7f891c76e700 1 -- 172.21.15.121:0/370535329 >> 172.21.15.121:0/370535329 conn(0x7f89141007f0 msgr2=0x7f8914102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:56.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.324+0000 7f891c76e700 1 -- 172.21.15.121:0/370535329 shutdown_connections 2024-04-05T21:16:56.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.324+0000 7f891c76e700 1 -- 172.21.15.121:0/370535329 wait complete. 2024-04-05T21:16:56.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.325+0000 7f891c76e700 1 Processor -- start 2024-04-05T21:16:56.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.325+0000 7f891c76e700 1 -- start start 2024-04-05T21:16:56.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.325+0000 7f891c76e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 0x7f891419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:56.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.325+0000 7f891c76e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891419a570 con 0x7f8914106da0 2024-04-05T21:16:56.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.325+0000 7f891a50a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 0x7f891419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:56.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.325+0000 7f891a50a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 0x7f891419a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59078/0 (socket says 172.21.15.121:59078) 2024-04-05T21:16:56.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.325+0000 7f891a50a700 1 -- 172.21.15.121:0/1928617099 learned_addr learned my addr 172.21.15.121:0/1928617099 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:56.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.325+0000 7f891a50a700 1 -- 172.21.15.121:0/1928617099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8908009550 con 0x7f8914106da0 2024-04-05T21:16:56.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.325+0000 7f891a50a700 1 --2- 172.21.15.121:0/1928617099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 0x7f891419a030 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8908006af0 tx=0x7f8908011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:56.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.326+0000 7f89077fe700 1 -- 172.21.15.121:0/1928617099 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8908003a80 con 0x7f8914106da0 2024-04-05T21:16:56.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.326+0000 7f89077fe700 1 -- 172.21.15.121:0/1928617099 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f8908003be0 con 0x7f8914106da0 2024-04-05T21:16:56.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.326+0000 7f89077fe700 1 -- 172.21.15.121:0/1928617099 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8908019770 con 0x7f8914106da0 2024-04-05T21:16:56.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.326+0000 7f891c76e700 1 -- 172.21.15.121:0/1928617099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891419a770 con 0x7f8914106da0 2024-04-05T21:16:56.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.326+0000 7f891c76e700 1 -- 172.21.15.121:0/1928617099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f891419ab90 con 0x7f8914106da0 2024-04-05T21:16:56.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.326+0000 7f89077fe700 1 -- 172.21.15.121:0/1928617099 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f89080199e0 con 0x7f8914106da0 2024-04-05T21:16:56.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.327+0000 7f891c76e700 1 -- 172.21.15.121:0/1928617099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89140623c0 con 0x7f8914106da0 2024-04-05T21:16:56.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.327+0000 7f89077fe700 1 --2- 172.21.15.121:0/1928617099 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f8900038330 0x7f890003a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:56.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.327+0000 7f89077fe700 1 -- 172.21.15.121:0/1928617099 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f890804c550 con 0x7f8914106da0 2024-04-05T21:16:56.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.327+0000 7f8919d09700 1 --2- 172.21.15.121:0/1928617099 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f8900038330 0x7f890003a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:56.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.327+0000 7f8919d09700 1 --2- 172.21.15.121:0/1928617099 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f8900038330 0x7f890003a7f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8910006fd0 tx=0x7f8910006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:56.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.331+0000 7f89077fe700 1 -- 172.21.15.121:0/1928617099 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8908023d60 con 0x7f8914106da0 2024-04-05T21:16:56.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.533+0000 7f891c76e700 1 -- 172.21.15.121:0/1928617099 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f8914102af0 con 0x7f8900038330 2024-04-05T21:16:56.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.539+0000 7f89077fe700 1 -- 172.21.15.121:0/1928617099 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f8914102af0 con 0x7f8900038330 2024-04-05T21:16:56.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.541+0000 7f891c76e700 1 -- 172.21.15.121:0/1928617099 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f8900038330 msgr2=0x7f890003a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:56.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.541+0000 7f891c76e700 1 --2- 172.21.15.121:0/1928617099 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f8900038330 0x7f890003a7f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8910006fd0 tx=0x7f8910006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:56.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.541+0000 7f891c76e700 1 -- 172.21.15.121:0/1928617099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 msgr2=0x7f891419a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:56.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.541+0000 7f891c76e700 1 --2- 172.21.15.121:0/1928617099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 0x7f891419a030 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8908006af0 tx=0x7f8908011b20 comp rx=0 tx=0).stop 2024-04-05T21:16:56.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.542+0000 7f891c76e700 1 -- 172.21.15.121:0/1928617099 shutdown_connections 2024-04-05T21:16:56.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.542+0000 7f891c76e700 1 --2- 172.21.15.121:0/1928617099 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f8900038330 0x7f890003a7f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:56.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.542+0000 7f891c76e700 1 --2- 172.21.15.121:0/1928617099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8914106da0 0x7f891419a030 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:56.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.542+0000 7f891c76e700 1 -- 172.21.15.121:0/1928617099 >> 172.21.15.121:0/1928617099 conn(0x7f89141007f0 msgr2=0x7f8914101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:56.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.542+0000 7f891c76e700 1 -- 172.21.15.121:0/1928617099 shutdown_connections 2024-04-05T21:16:56.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:55.542+0000 7f891c76e700 1 -- 172.21.15.121:0/1928617099 wait complete. 2024-04-05T21:16:56.109 INFO:teuthology.orchestra.run.smithi121.stdout:Deploying alertmanager service with default placement... 2024-04-05T21:16:56.369 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:56 smithi121 ceph-mon[39723]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:16:56.369 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:56 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:57.610 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-05T21:16:57.610 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.665+0000 7f77f0c16700 1 Processor -- start 2024-04-05T21:16:57.610 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.665+0000 7f77f0c16700 1 -- start start 2024-04-05T21:16:57.610 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.665+0000 7f77f0c16700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 0x7f77ec109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:57.610 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.665+0000 7f77f0c16700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec079de0 con 0x7f77ec106da0 2024-04-05T21:16:57.610 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.665+0000 7f77ea59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 0x7f77ec109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:57.610 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.665+0000 7f77ea59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 0x7f77ec109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59082/0 (socket says 172.21.15.121:59082) 2024-04-05T21:16:57.610 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.665+0000 7f77ea59c700 1 -- 172.21.15.121:0/4157613167 learned_addr learned my addr 172.21.15.121:0/4157613167 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:57.610 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.665+0000 7f77ea59c700 1 -- 172.21.15.121:0/4157613167 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec1096d0 con 0x7f77ec106da0 2024-04-05T21:16:57.610 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.665+0000 7f77ea59c700 1 --2- 172.21.15.121:0/4157613167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 0x7f77ec109190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77dc0098a0 tx=0x7f77dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3132b5d92be2667 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:57.611 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.666+0000 7f77e959a700 1 -- 172.21.15.121:0/4157613167 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77dc004180 con 0x7f77ec106da0 2024-04-05T21:16:57.611 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.666+0000 7f77e959a700 1 -- 172.21.15.121:0/4157613167 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f77dc0042e0 con 0x7f77ec106da0 2024-04-05T21:16:57.611 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.666+0000 7f77e959a700 1 -- 172.21.15.121:0/4157613167 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77dc011a30 con 0x7f77ec106da0 2024-04-05T21:16:57.611 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.666+0000 7f77f0c16700 1 -- 172.21.15.121:0/4157613167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 msgr2=0x7f77ec109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:57.611 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.666+0000 7f77f0c16700 1 --2- 172.21.15.121:0/4157613167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 0x7f77ec109190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77dc0098a0 tx=0x7f77dc009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:57.611 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.666+0000 7f77f0c16700 1 -- 172.21.15.121:0/4157613167 shutdown_connections 2024-04-05T21:16:57.611 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.666+0000 7f77f0c16700 1 --2- 172.21.15.121:0/4157613167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 0x7f77ec109190 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:57.611 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.666+0000 7f77f0c16700 1 -- 172.21.15.121:0/4157613167 >> 172.21.15.121:0/4157613167 conn(0x7f77ec1007d0 msgr2=0x7f77ec102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:57.611 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.666+0000 7f77f0c16700 1 -- 172.21.15.121:0/4157613167 shutdown_connections 2024-04-05T21:16:57.611 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.666+0000 7f77f0c16700 1 -- 172.21.15.121:0/4157613167 wait complete. 2024-04-05T21:16:57.611 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.667+0000 7f77f0c16700 1 Processor -- start 2024-04-05T21:16:57.612 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.667+0000 7f77f0c16700 1 -- start start 2024-04-05T21:16:57.612 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.667+0000 7f77f0c16700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 0x7f77ec199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:57.612 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.667+0000 7f77f0c16700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec19a530 con 0x7f77ec106da0 2024-04-05T21:16:57.612 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.667+0000 7f77ea59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 0x7f77ec199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:57.612 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.667+0000 7f77ea59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 0x7f77ec199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59086/0 (socket says 172.21.15.121:59086) 2024-04-05T21:16:57.612 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.667+0000 7f77ea59c700 1 -- 172.21.15.121:0/1172705660 learned_addr learned my addr 172.21.15.121:0/1172705660 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:57.612 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.668+0000 7f77ea59c700 1 -- 172.21.15.121:0/1172705660 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77dc009550 con 0x7f77ec106da0 2024-04-05T21:16:57.612 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.668+0000 7f77ea59c700 1 --2- 172.21.15.121:0/1172705660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 0x7f77ec199ff0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f77dc011a00 tx=0x7f77dc011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:57.612 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.668+0000 7f77db7fe700 1 -- 172.21.15.121:0/1172705660 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77dc003bc0 con 0x7f77ec106da0 2024-04-05T21:16:57.612 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.668+0000 7f77db7fe700 1 -- 172.21.15.121:0/1172705660 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f77dc0248c0 con 0x7f77ec106da0 2024-04-05T21:16:57.612 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.668+0000 7f77f0c16700 1 -- 172.21.15.121:0/1172705660 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ec19a730 con 0x7f77ec106da0 2024-04-05T21:16:57.612 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.668+0000 7f77db7fe700 1 -- 172.21.15.121:0/1172705660 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77dc01a850 con 0x7f77ec106da0 2024-04-05T21:16:57.612 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.668+0000 7f77f0c16700 1 -- 172.21.15.121:0/1172705660 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ec19ab50 con 0x7f77ec106da0 2024-04-05T21:16:57.613 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.669+0000 7f77db7fe700 1 -- 172.21.15.121:0/1172705660 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f77dc01aac0 con 0x7f77ec106da0 2024-04-05T21:16:57.613 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.669+0000 7f77f0c16700 1 -- 172.21.15.121:0/1172705660 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ec0623c0 con 0x7f77ec106da0 2024-04-05T21:16:57.613 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.669+0000 7f77db7fe700 1 --2- 172.21.15.121:0/1172705660 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f77d4038370 0x7f77d403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:57.613 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.669+0000 7f77db7fe700 1 -- 172.21.15.121:0/1172705660 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f77dc04c7b0 con 0x7f77ec106da0 2024-04-05T21:16:57.613 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.669+0000 7f77e9d9b700 1 --2- 172.21.15.121:0/1172705660 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f77d4038370 0x7f77d403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:57.613 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.669+0000 7f77e9d9b700 1 --2- 172.21.15.121:0/1172705660 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f77d4038370 0x7f77d403a830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f77e0006fd0 tx=0x7f77e0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:57.613 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.673+0000 7f77db7fe700 1 -- 172.21.15.121:0/1172705660 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f77dc01ad70 con 0x7f77ec106da0 2024-04-05T21:16:57.613 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.873+0000 7f77f0c16700 1 -- 172.21.15.121:0/1172705660 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f77ec102ac0 con 0x7f77d4038370 2024-04-05T21:16:57.613 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.877+0000 7f77db7fe700 1 -- 172.21.15.121:0/1172705660 <== mgr.14118 v2:172.21.15.121:6800/1004235250 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f77ec102ac0 con 0x7f77d4038370 2024-04-05T21:16:57.613 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.879+0000 7f77f0c16700 1 -- 172.21.15.121:0/1172705660 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f77d4038370 msgr2=0x7f77d403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:57.613 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.879+0000 7f77f0c16700 1 --2- 172.21.15.121:0/1172705660 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f77d4038370 0x7f77d403a830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f77e0006fd0 tx=0x7f77e0006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:57.614 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.879+0000 7f77f0c16700 1 -- 172.21.15.121:0/1172705660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 msgr2=0x7f77ec199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:57.614 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.879+0000 7f77f0c16700 1 --2- 172.21.15.121:0/1172705660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 0x7f77ec199ff0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f77dc011a00 tx=0x7f77dc011ae0 comp rx=0 tx=0).stop 2024-04-05T21:16:57.614 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.879+0000 7f77f0c16700 1 -- 172.21.15.121:0/1172705660 shutdown_connections 2024-04-05T21:16:57.614 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.879+0000 7f77f0c16700 1 --2- 172.21.15.121:0/1172705660 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f77d4038370 0x7f77d403a830 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:57.614 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.879+0000 7f77f0c16700 1 --2- 172.21.15.121:0/1172705660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec106da0 0x7f77ec199ff0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:57.614 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.879+0000 7f77f0c16700 1 -- 172.21.15.121:0/1172705660 >> 172.21.15.121:0/1172705660 conn(0x7f77ec1007d0 msgr2=0x7f77ec101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:57.614 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.879+0000 7f77f0c16700 1 -- 172.21.15.121:0/1172705660 shutdown_connections 2024-04-05T21:16:57.614 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:56.879+0000 7f77f0c16700 1 -- 172.21.15.121:0/1172705660 wait complete. 2024-04-05T21:16:57.614 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:57 smithi121 ceph-mon[39723]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:16:57.614 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:57 smithi121 ceph-mon[39723]: Saving service node-exporter spec with placement * 2024-04-05T21:16:57.615 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:57 smithi121 ceph-mon[39723]: from='mgr.14118 172.21.15.121:0/2846834337' entity='mgr.smithi121.etbquh' 2024-04-05T21:16:58.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:58 smithi121 ceph-mon[39723]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:16:58.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:58 smithi121 ceph-mon[39723]: Saving service alertmanager spec with placement count:1 2024-04-05T21:16:58.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:58 smithi121 ceph-mon[39723]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:16:59.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.223+0000 7fe2ba29e700 1 Processor -- start 2024-04-05T21:16:59.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.223+0000 7fe2ba29e700 1 -- start start 2024-04-05T21:16:59.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.223+0000 7fe2ba29e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 0x7fe2b4106c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.223+0000 7fe2ba29e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b41071f0 con 0x7fe2b4106800 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.223+0000 7fe2b3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 0x7fe2b4106c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.223+0000 7fe2b3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 0x7fe2b4106c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59090/0 (socket says 172.21.15.121:59090) 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.223+0000 7fe2b3fff700 1 -- 172.21.15.121:0/1055960587 learned_addr learned my addr 172.21.15.121:0/1055960587 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.224+0000 7fe2b3fff700 1 -- 172.21.15.121:0/1055960587 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b4105410 con 0x7fe2b4106800 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.224+0000 7fe2b3fff700 1 --2- 172.21.15.121:0/1055960587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 0x7fe2b4106c20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe2a40098a0 tx=0x7fe2a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad831241ba3d8a9a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.224+0000 7fe2b2ffd700 1 -- 172.21.15.121:0/1055960587 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a4004180 con 0x7fe2b4106800 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.224+0000 7fe2b2ffd700 1 -- 172.21.15.121:0/1055960587 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fe2a40042e0 con 0x7fe2b4106800 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.224+0000 7fe2b2ffd700 1 -- 172.21.15.121:0/1055960587 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a4011a30 con 0x7fe2b4106800 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.224+0000 7fe2ba29e700 1 -- 172.21.15.121:0/1055960587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 msgr2=0x7fe2b4106c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.224+0000 7fe2ba29e700 1 --2- 172.21.15.121:0/1055960587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 0x7fe2b4106c20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe2a40098a0 tx=0x7fe2a4009bb0 comp rx=0 tx=0).stop 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.225+0000 7fe2ba29e700 1 -- 172.21.15.121:0/1055960587 shutdown_connections 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.225+0000 7fe2ba29e700 1 --2- 172.21.15.121:0/1055960587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 0x7fe2b4106c20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:59.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.225+0000 7fe2ba29e700 1 -- 172.21.15.121:0/1055960587 >> 172.21.15.121:0/1055960587 conn(0x7fe2b40fff40 msgr2=0x7fe2b4102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.225+0000 7fe2ba29e700 1 -- 172.21.15.121:0/1055960587 shutdown_connections 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.225+0000 7fe2ba29e700 1 -- 172.21.15.121:0/1055960587 wait complete. 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.225+0000 7fe2ba29e700 1 Processor -- start 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.225+0000 7fe2ba29e700 1 -- start start 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.225+0000 7fe2ba29e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 0x7fe2b419c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.225+0000 7fe2ba29e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b419c790 con 0x7fe2b4106800 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.226+0000 7fe2b3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 0x7fe2b419c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.226+0000 7fe2b3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 0x7fe2b419c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59106/0 (socket says 172.21.15.121:59106) 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.226+0000 7fe2b3fff700 1 -- 172.21.15.121:0/2263835980 learned_addr learned my addr 172.21.15.121:0/2263835980 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.226+0000 7fe2b3fff700 1 -- 172.21.15.121:0/2263835980 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a4009550 con 0x7fe2b4106800 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.226+0000 7fe2b3fff700 1 --2- 172.21.15.121:0/2263835980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 0x7fe2b419c250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4006af0 tx=0x7fe2a4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.226+0000 7fe2b17fa700 1 -- 172.21.15.121:0/2263835980 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a4003bc0 con 0x7fe2b4106800 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.226+0000 7fe2b17fa700 1 -- 172.21.15.121:0/2263835980 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fe2a401a8c0 con 0x7fe2b4106800 2024-04-05T21:16:59.091 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.226+0000 7fe2b17fa700 1 -- 172.21.15.121:0/2263835980 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a4019820 con 0x7fe2b4106800 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.226+0000 7fe2ba29e700 1 -- 172.21.15.121:0/2263835980 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b419c990 con 0x7fe2b4106800 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.226+0000 7fe2ba29e700 1 -- 172.21.15.121:0/2263835980 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b419cdb0 con 0x7fe2b4106800 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.227+0000 7fe2b17fa700 1 -- 172.21.15.121:0/2263835980 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fe2a4019a50 con 0x7fe2b4106800 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.227+0000 7fe2ba29e700 1 -- 172.21.15.121:0/2263835980 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b4191ab0 con 0x7fe2b4106800 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.227+0000 7fe2b17fa700 1 --2- 172.21.15.121:0/2263835980 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe2a0038320 0x7fe2a003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.227+0000 7fe2b17fa700 1 -- 172.21.15.121:0/2263835980 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe2a404c9c0 con 0x7fe2b4106800 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.227+0000 7fe2b37fe700 1 --2- 172.21.15.121:0/2263835980 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe2a0038320 0x7fe2a003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.228+0000 7fe2b37fe700 1 --2- 172.21.15.121:0/2263835980 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe2a0038320 0x7fe2a003a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe29c006fd0 tx=0x7fe29c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.233+0000 7fe2b17fa700 1 -- 172.21.15.121:0/2263835980 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe2a402b750 con 0x7fe2b4106800 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.420+0000 7fe2ba29e700 1 -- 172.21.15.121:0/2263835980 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fe2b40619a0 con 0x7fe2b4106800 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.424+0000 7fe2b17fa700 1 -- 172.21.15.121:0/2263835980 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fe2a4018350 con 0x7fe2b4106800 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.426+0000 7fe2ba29e700 1 -- 172.21.15.121:0/2263835980 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe2a0038320 msgr2=0x7fe2a003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.426+0000 7fe2ba29e700 1 --2- 172.21.15.121:0/2263835980 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe2a0038320 0x7fe2a003a7e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe29c006fd0 tx=0x7fe29c006e40 comp rx=0 tx=0).stop 2024-04-05T21:16:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.426+0000 7fe2ba29e700 1 -- 172.21.15.121:0/2263835980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 msgr2=0x7fe2b419c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:16:59.093 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.426+0000 7fe2ba29e700 1 --2- 172.21.15.121:0/2263835980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 0x7fe2b419c250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4006af0 tx=0x7fe2a4011ab0 comp rx=0 tx=0).stop 2024-04-05T21:16:59.093 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.426+0000 7fe2ba29e700 1 -- 172.21.15.121:0/2263835980 shutdown_connections 2024-04-05T21:16:59.093 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.426+0000 7fe2ba29e700 1 --2- 172.21.15.121:0/2263835980 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fe2a0038320 0x7fe2a003a7e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:59.093 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.426+0000 7fe2ba29e700 1 --2- 172.21.15.121:0/2263835980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4106800 0x7fe2b419c250 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:16:59.093 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.426+0000 7fe2ba29e700 1 -- 172.21.15.121:0/2263835980 >> 172.21.15.121:0/2263835980 conn(0x7fe2b40fff40 msgr2=0x7fe2b418ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:16:59.093 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.426+0000 7fe2ba29e700 1 -- 172.21.15.121:0/2263835980 shutdown_connections 2024-04-05T21:16:59.093 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:58.426+0000 7fe2ba29e700 1 -- 172.21.15.121:0/2263835980 wait complete. 2024-04-05T21:16:59.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:59 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2263835980' entity='client.admin' 2024-04-05T21:16:59.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:16:59 smithi121 ceph-mon[39723]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:00.578 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.730+0000 7fb8f5391700 1 Processor -- start 2024-04-05T21:17:00.579 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.730+0000 7fb8f5391700 1 -- start start 2024-04-05T21:17:00.579 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.730+0000 7fb8f5391700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 0x7fb8f0105030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:00.579 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.730+0000 7fb8f5391700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f0079d60 con 0x7fb8f0104c10 2024-04-05T21:17:00.579 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.730+0000 7fb8eeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 0x7fb8f0105030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:00.579 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.730+0000 7fb8eeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 0x7fb8f0105030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59108/0 (socket says 172.21.15.121:59108) 2024-04-05T21:17:00.579 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.730+0000 7fb8eeffd700 1 -- 172.21.15.121:0/230242701 learned_addr learned my addr 172.21.15.121:0/230242701 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:00.579 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.731+0000 7fb8eeffd700 1 -- 172.21.15.121:0/230242701 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f0105570 con 0x7fb8f0104c10 2024-04-05T21:17:00.579 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.731+0000 7fb8eeffd700 1 --2- 172.21.15.121:0/230242701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 0x7fb8f0105030 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40098a0 tx=0x7fb8e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f2d8c0851eee7a server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:00.579 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.731+0000 7fb8edffb700 1 -- 172.21.15.121:0/230242701 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e4004180 con 0x7fb8f0104c10 2024-04-05T21:17:00.579 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.731+0000 7fb8edffb700 1 -- 172.21.15.121:0/230242701 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fb8e40042e0 con 0x7fb8f0104c10 2024-04-05T21:17:00.579 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.731+0000 7fb8edffb700 1 -- 172.21.15.121:0/230242701 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e4011a30 con 0x7fb8f0104c10 2024-04-05T21:17:00.579 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.731+0000 7fb8f5391700 1 -- 172.21.15.121:0/230242701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 msgr2=0x7fb8f0105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.731+0000 7fb8f5391700 1 --2- 172.21.15.121:0/230242701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 0x7fb8f0105030 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40098a0 tx=0x7fb8e4009bb0 comp rx=0 tx=0).stop 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.731+0000 7fb8f5391700 1 -- 172.21.15.121:0/230242701 shutdown_connections 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.731+0000 7fb8f5391700 1 --2- 172.21.15.121:0/230242701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 0x7fb8f0105030 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.731+0000 7fb8f5391700 1 -- 172.21.15.121:0/230242701 >> 172.21.15.121:0/230242701 conn(0x7fb8f01007d0 msgr2=0x7fb8f0102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.731+0000 7fb8f5391700 1 -- 172.21.15.121:0/230242701 shutdown_connections 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.732+0000 7fb8f5391700 1 -- 172.21.15.121:0/230242701 wait complete. 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.732+0000 7fb8f5391700 1 Processor -- start 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.732+0000 7fb8f5391700 1 -- start start 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.733+0000 7fb8f5391700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 0x7fb8f019c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.733+0000 7fb8f5391700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f019c750 con 0x7fb8f0104c10 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.733+0000 7fb8eeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 0x7fb8f019c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.733+0000 7fb8eeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 0x7fb8f019c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59110/0 (socket says 172.21.15.121:59110) 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.733+0000 7fb8eeffd700 1 -- 172.21.15.121:0/3685310612 learned_addr learned my addr 172.21.15.121:0/3685310612 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.733+0000 7fb8eeffd700 1 -- 172.21.15.121:0/3685310612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e4009550 con 0x7fb8f0104c10 2024-04-05T21:17:00.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.733+0000 7fb8eeffd700 1 --2- 172.21.15.121:0/3685310612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 0x7fb8f019c210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4011a00 tx=0x7fb8e4011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.733+0000 7fb8d7fff700 1 -- 172.21.15.121:0/3685310612 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e4003bc0 con 0x7fb8f0104c10 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.733+0000 7fb8d7fff700 1 -- 172.21.15.121:0/3685310612 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fb8e40248c0 con 0x7fb8f0104c10 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.733+0000 7fb8f5391700 1 -- 172.21.15.121:0/3685310612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f019c950 con 0x7fb8f0104c10 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.733+0000 7fb8d7fff700 1 -- 172.21.15.121:0/3685310612 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e401a850 con 0x7fb8f0104c10 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.733+0000 7fb8f5391700 1 -- 172.21.15.121:0/3685310612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f019cd70 con 0x7fb8f0104c10 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.734+0000 7fb8d7fff700 1 -- 172.21.15.121:0/3685310612 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fb8e401ab50 con 0x7fb8f0104c10 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.734+0000 7fb8f5391700 1 -- 172.21.15.121:0/3685310612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f0195a90 con 0x7fb8f0104c10 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.734+0000 7fb8d7fff700 1 --2- 172.21.15.121:0/3685310612 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fb8d8038320 0x7fb8d803a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.735+0000 7fb8d7fff700 1 -- 172.21.15.121:0/3685310612 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb8e404c880 con 0x7fb8f0104c10 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.735+0000 7fb8ee7fc700 1 --2- 172.21.15.121:0/3685310612 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fb8d8038320 0x7fb8d803a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.735+0000 7fb8ee7fc700 1 --2- 172.21.15.121:0/3685310612 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fb8d8038320 0x7fb8d803a7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0006fd0 tx=0x7fb8e0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.739+0000 7fb8d7fff700 1 -- 172.21.15.121:0/3685310612 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb8e401ae00 con 0x7fb8f0104c10 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.926+0000 7fb8f5391700 1 -- 172.21.15.121:0/3685310612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb8f002cf20 con 0x7fb8f0104c10 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.930+0000 7fb8d7fff700 1 -- 172.21.15.121:0/3685310612 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb8e4018ca0 con 0x7fb8f0104c10 2024-04-05T21:17:00.581 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.932+0000 7fb8f5391700 1 -- 172.21.15.121:0/3685310612 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fb8d8038320 msgr2=0x7fb8d803a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:00.582 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.932+0000 7fb8f5391700 1 --2- 172.21.15.121:0/3685310612 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fb8d8038320 0x7fb8d803a7e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0006fd0 tx=0x7fb8e0006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:00.582 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.932+0000 7fb8f5391700 1 -- 172.21.15.121:0/3685310612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 msgr2=0x7fb8f019c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:00.582 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.932+0000 7fb8f5391700 1 --2- 172.21.15.121:0/3685310612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 0x7fb8f019c210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4011a00 tx=0x7fb8e4011ae0 comp rx=0 tx=0).stop 2024-04-05T21:17:00.582 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.932+0000 7fb8f5391700 1 -- 172.21.15.121:0/3685310612 shutdown_connections 2024-04-05T21:17:00.582 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.932+0000 7fb8f5391700 1 --2- 172.21.15.121:0/3685310612 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fb8d8038320 0x7fb8d803a7e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:00.582 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.932+0000 7fb8f5391700 1 --2- 172.21.15.121:0/3685310612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0104c10 0x7fb8f019c210 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:00.582 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.932+0000 7fb8f5391700 1 -- 172.21.15.121:0/3685310612 >> 172.21.15.121:0/3685310612 conn(0x7fb8f01007d0 msgr2=0x7fb8f01907d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:00.582 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.933+0000 7fb8f5391700 1 -- 172.21.15.121:0/3685310612 shutdown_connections 2024-04-05T21:17:00.582 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:16:59.933+0000 7fb8f5391700 1 -- 172.21.15.121:0/3685310612 wait complete. 2024-04-05T21:17:00.582 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling the dashboard module... 2024-04-05T21:17:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:00 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3685310612' entity='client.admin' 2024-04-05T21:17:02.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:01 smithi121 ceph-mon[39723]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:02.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:01 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1401811092' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-05T21:17:02.711 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.173+0000 7fdf528cf700 1 Processor -- start 2024-04-05T21:17:02.711 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.173+0000 7fdf528cf700 1 -- start start 2024-04-05T21:17:02.711 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.173+0000 7fdf528cf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 0x7fdf4c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:02.711 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.173+0000 7fdf528cf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0744f0 con 0x7fdf4c106de0 2024-04-05T21:17:02.711 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.173+0000 7fdf4bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 0x7fdf4c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.173+0000 7fdf4bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 0x7fdf4c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59116/0 (socket says 172.21.15.121:59116) 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.174+0000 7fdf4bfff700 1 -- 172.21.15.121:0/2996267053 learned_addr learned my addr 172.21.15.121:0/2996267053 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.174+0000 7fdf4bfff700 1 -- 172.21.15.121:0/2996267053 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c109710 con 0x7fdf4c106de0 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.174+0000 7fdf4bfff700 1 --2- 172.21.15.121:0/2996267053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 0x7fdf4c1091d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdf340098a0 tx=0x7fdf34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15463bdd1717ff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.174+0000 7fdf4affd700 1 -- 172.21.15.121:0/2996267053 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf34004180 con 0x7fdf4c106de0 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.174+0000 7fdf4affd700 1 -- 172.21.15.121:0/2996267053 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fdf340042e0 con 0x7fdf4c106de0 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.174+0000 7fdf4affd700 1 -- 172.21.15.121:0/2996267053 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf34011a30 con 0x7fdf4c106de0 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.174+0000 7fdf528cf700 1 -- 172.21.15.121:0/2996267053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 msgr2=0x7fdf4c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.174+0000 7fdf528cf700 1 --2- 172.21.15.121:0/2996267053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 0x7fdf4c1091d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdf340098a0 tx=0x7fdf34009bb0 comp rx=0 tx=0).stop 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.175+0000 7fdf528cf700 1 -- 172.21.15.121:0/2996267053 shutdown_connections 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.175+0000 7fdf528cf700 1 --2- 172.21.15.121:0/2996267053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 0x7fdf4c1091d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.175+0000 7fdf528cf700 1 -- 172.21.15.121:0/2996267053 >> 172.21.15.121:0/2996267053 conn(0x7fdf4c1007f0 msgr2=0x7fdf4c102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.175+0000 7fdf528cf700 1 -- 172.21.15.121:0/2996267053 shutdown_connections 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.175+0000 7fdf528cf700 1 -- 172.21.15.121:0/2996267053 wait complete. 2024-04-05T21:17:02.712 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.175+0000 7fdf528cf700 1 Processor -- start 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.175+0000 7fdf528cf700 1 -- start start 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.175+0000 7fdf528cf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 0x7fdf4c19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.175+0000 7fdf528cf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c19a5b0 con 0x7fdf4c106de0 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.176+0000 7fdf4bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 0x7fdf4c19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.176+0000 7fdf4bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 0x7fdf4c19a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59120/0 (socket says 172.21.15.121:59120) 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.176+0000 7fdf4bfff700 1 -- 172.21.15.121:0/1401811092 learned_addr learned my addr 172.21.15.121:0/1401811092 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.176+0000 7fdf4bfff700 1 -- 172.21.15.121:0/1401811092 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf34009550 con 0x7fdf4c106de0 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.176+0000 7fdf4bfff700 1 --2- 172.21.15.121:0/1401811092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 0x7fdf4c19a070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdf34011a00 tx=0x7fdf34011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.176+0000 7fdf497fa700 1 -- 172.21.15.121:0/1401811092 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf34003bd0 con 0x7fdf4c106de0 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.176+0000 7fdf497fa700 1 -- 172.21.15.121:0/1401811092 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fdf340248d0 con 0x7fdf4c106de0 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.176+0000 7fdf497fa700 1 -- 172.21.15.121:0/1401811092 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf3401a7e0 con 0x7fdf4c106de0 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.176+0000 7fdf528cf700 1 -- 172.21.15.121:0/1401811092 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4c19a7b0 con 0x7fdf4c106de0 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.176+0000 7fdf528cf700 1 -- 172.21.15.121:0/1401811092 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4c19abd0 con 0x7fdf4c106de0 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.177+0000 7fdf497fa700 1 -- 172.21.15.121:0/1401811092 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fdf3401a9a0 con 0x7fdf4c106de0 2024-04-05T21:17:02.713 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.177+0000 7fdf528cf700 1 -- 172.21.15.121:0/1401811092 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4c0623c0 con 0x7fdf4c106de0 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.177+0000 7fdf497fa700 1 --2- 172.21.15.121:0/1401811092 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fdf38038370 0x7fdf3803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.177+0000 7fdf497fa700 1 -- 172.21.15.121:0/1401811092 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdf3404c450 con 0x7fdf4c106de0 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.177+0000 7fdf4b7fe700 1 --2- 172.21.15.121:0/1401811092 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fdf38038370 0x7fdf3803a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.178+0000 7fdf4b7fe700 1 --2- 172.21.15.121:0/1401811092 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fdf38038370 0x7fdf3803a830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c006fd0 tx=0x7fdf3c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.182+0000 7fdf497fa700 1 -- 172.21.15.121:0/1401811092 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdf3401ac50 con 0x7fdf4c106de0 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.421+0000 7fdf528cf700 1 -- 172.21.15.121:0/1401811092 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fdf4c19dcd0 con 0x7fdf4c106de0 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.934+0000 7fdf497fa700 1 -- 172.21.15.121:0/1401811092 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7fdf3402da50 con 0x7fdf4c106de0 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.935+0000 7fdf497fa700 1 -- 172.21.15.121:0/1401811092 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fdf34024dd0 con 0x7fdf4c106de0 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.937+0000 7fdf528cf700 1 -- 172.21.15.121:0/1401811092 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fdf38038370 msgr2=0x7fdf3803a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.937+0000 7fdf528cf700 1 --2- 172.21.15.121:0/1401811092 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fdf38038370 0x7fdf3803a830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c006fd0 tx=0x7fdf3c006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.938+0000 7fdf528cf700 1 -- 172.21.15.121:0/1401811092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 msgr2=0x7fdf4c19a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.938+0000 7fdf528cf700 1 --2- 172.21.15.121:0/1401811092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 0x7fdf4c19a070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdf34011a00 tx=0x7fdf34011ae0 comp rx=0 tx=0).stop 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.938+0000 7fdf528cf700 1 -- 172.21.15.121:0/1401811092 shutdown_connections 2024-04-05T21:17:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.938+0000 7fdf528cf700 1 --2- 172.21.15.121:0/1401811092 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7fdf38038370 0x7fdf3803a830 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:02.715 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.938+0000 7fdf528cf700 1 --2- 172.21.15.121:0/1401811092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf4c106de0 0x7fdf4c19a070 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:02.715 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.938+0000 7fdf528cf700 1 -- 172.21.15.121:0/1401811092 >> 172.21.15.121:0/1401811092 conn(0x7fdf4c1007f0 msgr2=0x7fdf4c101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:02.715 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.938+0000 7fdf528cf700 1 -- 172.21.15.121:0/1401811092 shutdown_connections 2024-04-05T21:17:02.715 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:01.938+0000 7fdf528cf700 1 -- 172.21.15.121:0/1401811092 wait complete. 2024-04-05T21:17:03.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:02 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1401811092' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-05T21:17:03.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:02 smithi121 ceph-mon[39723]: mgrmap e9: smithi121.etbquh(active, since 28s) 2024-04-05T21:17:04.170 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-04-05T21:17:04.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-05T21:17:04.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-05T21:17:04.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "active_name": "smithi121.etbquh", 2024-04-05T21:17:04.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-05T21:17:04.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-04-05T21:17:04.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.351+0000 7f09f7393700 1 Processor -- start 2024-04-05T21:17:04.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.351+0000 7f09f7393700 1 -- start start 2024-04-05T21:17:04.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.351+0000 7f09f7393700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 0x7f09f0105650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:04.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.351+0000 7f09f7393700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f0105c20 con 0x7f09f0105230 2024-04-05T21:17:04.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.352+0000 7f09f512f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 0x7f09f0105650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:04.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.352+0000 7f09f512f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 0x7f09f0105650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59136/0 (socket says 172.21.15.121:59136) 2024-04-05T21:17:04.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.352+0000 7f09f512f700 1 -- 172.21.15.121:0/2491057827 learned_addr learned my addr 172.21.15.121:0/2491057827 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:04.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.352+0000 7f09f512f700 1 -- 172.21.15.121:0/2491057827 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f0106430 con 0x7f09f0105230 2024-04-05T21:17:04.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.352+0000 7f09f512f700 1 --2- 172.21.15.121:0/2491057827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 0x7f09f0105650 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f09ec0098a0 tx=0x7f09ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2dadf3f13847973 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:04.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.352+0000 7f09e7fff700 1 -- 172.21.15.121:0/2491057827 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09ec004180 con 0x7f09f0105230 2024-04-05T21:17:04.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.352+0000 7f09e7fff700 1 -- 172.21.15.121:0/2491057827 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f09ec0042e0 con 0x7f09f0105230 2024-04-05T21:17:04.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.352+0000 7f09e7fff700 1 -- 172.21.15.121:0/2491057827 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09ec011a30 con 0x7f09f0105230 2024-04-05T21:17:04.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.353+0000 7f09f7393700 1 -- 172.21.15.121:0/2491057827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 msgr2=0x7f09f0105650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:04.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.353+0000 7f09f7393700 1 --2- 172.21.15.121:0/2491057827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 0x7f09f0105650 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f09ec0098a0 tx=0x7f09ec009bb0 comp rx=0 tx=0).stop 2024-04-05T21:17:04.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.353+0000 7f09f7393700 1 -- 172.21.15.121:0/2491057827 shutdown_connections 2024-04-05T21:17:04.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.353+0000 7f09f7393700 1 --2- 172.21.15.121:0/2491057827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 0x7f09f0105650 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:04.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.353+0000 7f09f7393700 1 -- 172.21.15.121:0/2491057827 >> 172.21.15.121:0/2491057827 conn(0x7f09f01007f0 msgr2=0x7f09f0102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:04.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.353+0000 7f09f7393700 1 -- 172.21.15.121:0/2491057827 shutdown_connections 2024-04-05T21:17:04.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.353+0000 7f09f7393700 1 -- 172.21.15.121:0/2491057827 wait complete. 2024-04-05T21:17:04.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.353+0000 7f09f7393700 1 Processor -- start 2024-04-05T21:17:04.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.353+0000 7f09f7393700 1 -- start start 2024-04-05T21:17:04.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.354+0000 7f09f7393700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 0x7f09f01903c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:04.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.354+0000 7f09f7393700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f0190900 con 0x7f09f0105230 2024-04-05T21:17:04.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.354+0000 7f09f512f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 0x7f09f01903c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:04.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.354+0000 7f09f512f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 0x7f09f01903c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59150/0 (socket says 172.21.15.121:59150) 2024-04-05T21:17:04.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.354+0000 7f09f512f700 1 -- 172.21.15.121:0/3575851897 learned_addr learned my addr 172.21.15.121:0/3575851897 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:04.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.354+0000 7f09f512f700 1 -- 172.21.15.121:0/3575851897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec009550 con 0x7f09f0105230 2024-04-05T21:17:04.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.354+0000 7f09f512f700 1 --2- 172.21.15.121:0/3575851897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 0x7f09f01903c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f09ec006af0 tx=0x7f09ec011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:04.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.354+0000 7f09e67fc700 1 -- 172.21.15.121:0/3575851897 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09ec003a80 con 0x7f09f0105230 2024-04-05T21:17:04.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.354+0000 7f09e67fc700 1 -- 172.21.15.121:0/3575851897 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f09ec003be0 con 0x7f09f0105230 2024-04-05T21:17:04.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.354+0000 7f09f7393700 1 -- 172.21.15.121:0/3575851897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f018ea10 con 0x7f09f0105230 2024-04-05T21:17:04.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.354+0000 7f09e67fc700 1 -- 172.21.15.121:0/3575851897 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09ec019770 con 0x7f09f0105230 2024-04-05T21:17:04.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.354+0000 7f09f7393700 1 -- 172.21.15.121:0/3575851897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f018ee30 con 0x7f09f0105230 2024-04-05T21:17:04.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.355+0000 7f09f7393700 1 -- 172.21.15.121:0/3575851897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f00623c0 con 0x7f09f0105230 2024-04-05T21:17:04.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.356+0000 7f09e67fc700 1 -- 172.21.15.121:0/3575851897 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f09ec019a30 con 0x7f09f0105230 2024-04-05T21:17:04.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.356+0000 7f09e67fc700 1 --2- 172.21.15.121:0/3575851897 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f09dc038330 0x7f09dc03a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:04.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.356+0000 7f09e67fc700 1 -- 172.21.15.121:0/3575851897 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f09ec04c550 con 0x7f09f0105230 2024-04-05T21:17:04.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.356+0000 7f09f492e700 1 -- 172.21.15.121:0/3575851897 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f09dc038330 msgr2=0x7f09dc03a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1004235250 2024-04-05T21:17:04.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.356+0000 7f09f492e700 1 --2- 172.21.15.121:0/3575851897 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f09dc038330 0x7f09dc03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T21:17:04.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.360+0000 7f09e67fc700 1 -- 172.21.15.121:0/3575851897 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f09ec01acd0 con 0x7f09f0105230 2024-04-05T21:17:04.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.557+0000 7f09f492e700 1 -- 172.21.15.121:0/3575851897 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f09dc038330 msgr2=0x7f09dc03a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1004235250 2024-04-05T21:17:04.183 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.557+0000 7f09f492e700 1 --2- 172.21.15.121:0/3575851897 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f09dc038330 0x7f09dc03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T21:17:04.183 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.610+0000 7f09f7393700 1 -- 172.21.15.121:0/3575851897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f09f018f840 con 0x7f09f0105230 2024-04-05T21:17:04.183 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.611+0000 7f09e67fc700 1 -- 172.21.15.121:0/3575851897 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f09ec018350 con 0x7f09f0105230 2024-04-05T21:17:04.183 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.613+0000 7f09f7393700 1 -- 172.21.15.121:0/3575851897 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f09dc038330 msgr2=0x7f09dc03a7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T21:17:04.183 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.613+0000 7f09f7393700 1 --2- 172.21.15.121:0/3575851897 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f09dc038330 0x7f09dc03a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:04.183 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.613+0000 7f09f7393700 1 -- 172.21.15.121:0/3575851897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 msgr2=0x7f09f01903c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:04.183 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.613+0000 7f09f7393700 1 --2- 172.21.15.121:0/3575851897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 0x7f09f01903c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f09ec006af0 tx=0x7f09ec011b20 comp rx=0 tx=0).stop 2024-04-05T21:17:04.183 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.613+0000 7f09f7393700 1 -- 172.21.15.121:0/3575851897 shutdown_connections 2024-04-05T21:17:04.183 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.613+0000 7f09f7393700 1 --2- 172.21.15.121:0/3575851897 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f09dc038330 0x7f09dc03a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:04.183 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.613+0000 7f09f7393700 1 --2- 172.21.15.121:0/3575851897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09f0105230 0x7f09f01903c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:04.184 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.613+0000 7f09f7393700 1 -- 172.21.15.121:0/3575851897 >> 172.21.15.121:0/3575851897 conn(0x7f09f01007f0 msgr2=0x7f09f0101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:04.184 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.613+0000 7f09f7393700 1 -- 172.21.15.121:0/3575851897 shutdown_connections 2024-04-05T21:17:04.184 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:03.613+0000 7f09f7393700 1 -- 172.21.15.121:0/3575851897 wait complete. 2024-04-05T21:17:04.184 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for the mgr to restart... 2024-04-05T21:17:04.184 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mgr epoch 9... 2024-04-05T21:17:04.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:03 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3575851897' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-05T21:17:08.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:08 smithi121 ceph-mon[39723]: Active manager daemon smithi121.etbquh restarted 2024-04-05T21:17:08.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:08 smithi121 ceph-mon[39723]: Activating manager daemon smithi121.etbquh 2024-04-05T21:17:08.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:08 smithi121 ceph-mon[39723]: osdmap e3: 0 total, 0 up, 0 in 2024-04-05T21:17:08.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:08 smithi121 ceph-mon[39723]: mgrmap e10: smithi121.etbquh(active, starting, since 0.00310613s) 2024-04-05T21:17:08.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:08 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:17:08.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:08 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr metadata", "who": "smithi121.etbquh", "id": "smithi121.etbquh"}]: dispatch 2024-04-05T21:17:08.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:08 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T21:17:08.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:08 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T21:17:08.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:08 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T21:17:08.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:08 smithi121 ceph-mon[39723]: Manager daemon smithi121.etbquh is now available 2024-04-05T21:17:08.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:08 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:17:08.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:08 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:17:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:09 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.etbquh/mirror_snapshot_schedule"}]: dispatch 2024-04-05T21:17:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:09 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.etbquh/trash_purge_schedule"}]: dispatch 2024-04-05T21:17:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:09 smithi121 ceph-mon[39723]: [05/Apr/2024:21:17:08] ENGINE Bus STARTING 2024-04-05T21:17:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:09 smithi121 ceph-mon[39723]: mgrmap e11: smithi121.etbquh(active, since 1.00536s) 2024-04-05T21:17:10.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-04-05T21:17:10.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-05T21:17:10.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-05T21:17:10.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-04-05T21:17:10.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.722+0000 7f5b70ca6700 1 Processor -- start 2024-04-05T21:17:10.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.723+0000 7f5b70ca6700 1 -- start start 2024-04-05T21:17:10.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.723+0000 7f5b70ca6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 0x7f5b6c105050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:10.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.723+0000 7f5b70ca6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c074490 con 0x7f5b6c104c30 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.723+0000 7f5b6a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 0x7f5b6c105050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.723+0000 7f5b6a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 0x7f5b6c105050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39482/0 (socket says 172.21.15.121:39482) 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.723+0000 7f5b6a59c700 1 -- 172.21.15.121:0/3318469487 learned_addr learned my addr 172.21.15.121:0/3318469487 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.723+0000 7f5b6a59c700 1 -- 172.21.15.121:0/3318469487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c0745d0 con 0x7f5b6c104c30 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.724+0000 7f5b6a59c700 1 --2- 172.21.15.121:0/3318469487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 0x7f5b6c105050 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c0098a0 tx=0x7f5b5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dda91f479dfb50b server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.724+0000 7f5b6959a700 1 -- 172.21.15.121:0/3318469487 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b5c004180 con 0x7f5b6c104c30 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.724+0000 7f5b6959a700 1 -- 172.21.15.121:0/3318469487 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f5b5c0042e0 con 0x7f5b6c104c30 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.724+0000 7f5b6959a700 1 -- 172.21.15.121:0/3318469487 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b5c011a30 con 0x7f5b6c104c30 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.725+0000 7f5b70ca6700 1 -- 172.21.15.121:0/3318469487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 msgr2=0x7f5b6c105050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.725+0000 7f5b70ca6700 1 --2- 172.21.15.121:0/3318469487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 0x7f5b6c105050 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c0098a0 tx=0x7f5b5c009bb0 comp rx=0 tx=0).stop 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.725+0000 7f5b70ca6700 1 -- 172.21.15.121:0/3318469487 shutdown_connections 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.725+0000 7f5b70ca6700 1 --2- 172.21.15.121:0/3318469487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 0x7f5b6c105050 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.725+0000 7f5b70ca6700 1 -- 172.21.15.121:0/3318469487 >> 172.21.15.121:0/3318469487 conn(0x7f5b6c1007f0 msgr2=0x7f5b6c102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:10.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.725+0000 7f5b70ca6700 1 -- 172.21.15.121:0/3318469487 shutdown_connections 2024-04-05T21:17:10.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.725+0000 7f5b70ca6700 1 -- 172.21.15.121:0/3318469487 wait complete. 2024-04-05T21:17:10.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.725+0000 7f5b70ca6700 1 Processor -- start 2024-04-05T21:17:10.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.725+0000 7f5b70ca6700 1 -- start start 2024-04-05T21:17:10.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.726+0000 7f5b70ca6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 0x7f5b6c10a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:10.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.726+0000 7f5b70ca6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c10acf0 con 0x7f5b6c104c30 2024-04-05T21:17:10.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.726+0000 7f5b6a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 0x7f5b6c10a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:10.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.726+0000 7f5b6a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 0x7f5b6c10a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39488/0 (socket says 172.21.15.121:39488) 2024-04-05T21:17:10.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.726+0000 7f5b6a59c700 1 -- 172.21.15.121:0/1400600997 learned_addr learned my addr 172.21.15.121:0/1400600997 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:10.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.726+0000 7f5b6a59c700 1 -- 172.21.15.121:0/1400600997 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5c009550 con 0x7f5b6c104c30 2024-04-05T21:17:10.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.726+0000 7f5b6a59c700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 0x7f5b6c10a7b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c006af0 tx=0x7f5b5c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:10.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.726+0000 7f5b5b7fe700 1 -- 172.21.15.121:0/1400600997 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b5c003a80 con 0x7f5b6c104c30 2024-04-05T21:17:10.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.726+0000 7f5b70ca6700 1 -- 172.21.15.121:0/1400600997 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6c10aef0 con 0x7f5b6c104c30 2024-04-05T21:17:10.316 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.726+0000 7f5b5b7fe700 1 -- 172.21.15.121:0/1400600997 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f5b5c003be0 con 0x7f5b6c104c30 2024-04-05T21:17:10.316 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.726+0000 7f5b5b7fe700 1 -- 172.21.15.121:0/1400600997 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b5c019770 con 0x7f5b6c104c30 2024-04-05T21:17:10.316 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.726+0000 7f5b70ca6700 1 -- 172.21.15.121:0/1400600997 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6c1073e0 con 0x7f5b6c104c30 2024-04-05T21:17:10.316 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.727+0000 7f5b5b7fe700 1 -- 172.21.15.121:0/1400600997 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 9) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f5b5c019a70 con 0x7f5b6c104c30 2024-04-05T21:17:10.316 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.727+0000 7f5b5b7fe700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 0x7f5b5403a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:10.316 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.727+0000 7f5b5b7fe700 1 -- 172.21.15.121:0/1400600997 --> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b5403af00 con 0x7f5b54038330 2024-04-05T21:17:10.316 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.728+0000 7f5b69d9b700 1 -- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 msgr2=0x7f5b5403a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1004235250 2024-04-05T21:17:10.316 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.728+0000 7f5b5b7fe700 1 -- 172.21.15.121:0/1400600997 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5b5c04c5a0 con 0x7f5b6c104c30 2024-04-05T21:17:10.316 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.728+0000 7f5b69d9b700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 0x7f5b5403a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T21:17:10.316 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.929+0000 7f5b69d9b700 1 -- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 msgr2=0x7f5b5403a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1004235250 2024-04-05T21:17:10.316 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:04.929+0000 7f5b69d9b700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 0x7f5b5403a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T21:17:10.316 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:05.330+0000 7f5b69d9b700 1 -- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 msgr2=0x7f5b5403a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1004235250 2024-04-05T21:17:10.316 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:05.330+0000 7f5b69d9b700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 0x7f5b5403a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-05T21:17:10.317 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:06.131+0000 7f5b69d9b700 1 -- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 msgr2=0x7f5b5403a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1004235250 2024-04-05T21:17:10.317 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:06.131+0000 7f5b69d9b700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 0x7f5b5403a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-05T21:17:10.317 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:07.732+0000 7f5b69d9b700 1 -- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 msgr2=0x7f5b5403a7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1004235250 2024-04-05T21:17:10.317 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:07.732+0000 7f5b69d9b700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 0x7f5b5403a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-05T21:17:10.317 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:08.627+0000 7f5b5b7fe700 1 -- 172.21.15.121:0/1400600997 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mgrmap(e 10) v1 ==== 45063+0+0 (secure 0 0 0) 0x7f5b5c016d40 con 0x7f5b6c104c30 2024-04-05T21:17:10.317 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:08.627+0000 7f5b5b7fe700 1 -- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 msgr2=0x7f5b5403a7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T21:17:10.317 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:08.627+0000 7f5b5b7fe700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 0x7f5b5403a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:10.317 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.628+0000 7f5b5b7fe700 1 -- 172.21.15.121:0/1400600997 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7f5b5c04ded0 con 0x7f5b6c104c30 2024-04-05T21:17:10.317 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.629+0000 7f5b5b7fe700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f5b5403bdf0 0x7f5b5403e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:10.317 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.629+0000 7f5b5b7fe700 1 -- 172.21.15.121:0/1400600997 --> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b5403af00 con 0x7f5b5403bdf0 2024-04-05T21:17:10.317 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.629+0000 7f5b69d9b700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f5b5403bdf0 0x7f5b5403e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:10.317 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.629+0000 7f5b69d9b700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f5b5403bdf0 0x7f5b5403e1e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5b60003a10 tx=0x7f5b600092b0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.630+0000 7f5b5b7fe700 1 -- 172.21.15.121:0/1400600997 <== mgr.14162 v2:172.21.15.121:6800/1237774071 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f5b5403af00 con 0x7f5b5403bdf0 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.636+0000 7f5b70ca6700 1 -- 172.21.15.121:0/1400600997 --> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5b6c10b080 con 0x7f5b5403bdf0 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.636+0000 7f5b5b7fe700 1 -- 172.21.15.121:0/1400600997 <== mgr.14162 v2:172.21.15.121:6800/1237774071 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f5b6c10b080 con 0x7f5b5403bdf0 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.636+0000 7f5b70ca6700 1 -- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f5b5403bdf0 msgr2=0x7f5b5403e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.636+0000 7f5b70ca6700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f5b5403bdf0 0x7f5b5403e1e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5b60003a10 tx=0x7f5b600092b0 comp rx=0 tx=0).stop 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.636+0000 7f5b70ca6700 1 -- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 msgr2=0x7f5b6c10a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.636+0000 7f5b70ca6700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 0x7f5b6c10a7b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c006af0 tx=0x7f5b5c011b20 comp rx=0 tx=0).stop 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.636+0000 7f5b70ca6700 1 -- 172.21.15.121:0/1400600997 shutdown_connections 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.636+0000 7f5b70ca6700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f5b5403bdf0 0x7f5b5403e1e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.636+0000 7f5b70ca6700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b6c104c30 0x7f5b6c10a7b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.636+0000 7f5b70ca6700 1 --2- 172.21.15.121:0/1400600997 >> [v2:172.21.15.121:6800/1004235250,v1:172.21.15.121:6801/1004235250] conn(0x7f5b54038330 0x7f5b5403a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.636+0000 7f5b70ca6700 1 -- 172.21.15.121:0/1400600997 >> 172.21.15.121:0/1400600997 conn(0x7f5b6c1007f0 msgr2=0x7f5b6c1907f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.637+0000 7f5b70ca6700 1 -- 172.21.15.121:0/1400600997 shutdown_connections 2024-04-05T21:17:10.318 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:09.637+0000 7f5b70ca6700 1 -- 172.21.15.121:0/1400600997 wait complete. 2024-04-05T21:17:10.319 INFO:teuthology.orchestra.run.smithi121.stdout:mgr epoch 9 is available 2024-04-05T21:17:10.319 INFO:teuthology.orchestra.run.smithi121.stdout:Generating a dashboard self-signed certificate... 2024-04-05T21:17:11.231 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:10 smithi121 ceph-mon[39723]: [05/Apr/2024:21:17:09] ENGINE Serving on http://172.21.15.121:8765 2024-04-05T21:17:11.231 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:10 smithi121 ceph-mon[39723]: [05/Apr/2024:21:17:09] ENGINE Serving on https://172.21.15.121:7150 2024-04-05T21:17:11.231 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:10 smithi121 ceph-mon[39723]: [05/Apr/2024:21:17:09] ENGINE Bus STARTED 2024-04-05T21:17:11.231 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:10 smithi121 ceph-mon[39723]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-05T21:17:11.231 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:10 smithi121 ceph-mon[39723]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-05T21:17:11.231 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:10 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:11.231 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:10 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-05T21:17:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.906+0000 7f811fde9700 1 Processor -- start 2024-04-05T21:17:11.919 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.907+0000 7f811fde9700 1 -- start start 2024-04-05T21:17:11.920 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.907+0000 7f811fde9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 0x7f81181091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:11.920 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.907+0000 7f811fde9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8118079da0 con 0x7f8118106de0 2024-04-05T21:17:11.920 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.907+0000 7f811db85700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 0x7f81181091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:11.920 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.907+0000 7f811db85700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 0x7f81181091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39564/0 (socket says 172.21.15.121:39564) 2024-04-05T21:17:11.920 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.907+0000 7f811db85700 1 -- 172.21.15.121:0/1815909234 learned_addr learned my addr 172.21.15.121:0/1815909234 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:11.920 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.907+0000 7f811db85700 1 -- 172.21.15.121:0/1815909234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8118109710 con 0x7f8118106de0 2024-04-05T21:17:11.920 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.907+0000 7f811db85700 1 --2- 172.21.15.121:0/1815909234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 0x7f81181091d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f81140098a0 tx=0x7f8114009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3a70163a430b5c server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:11.920 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.908+0000 7f811cb83700 1 -- 172.21.15.121:0/1815909234 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8114004180 con 0x7f8118106de0 2024-04-05T21:17:11.920 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.908+0000 7f811cb83700 1 -- 172.21.15.121:0/1815909234 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f81140042e0 con 0x7f8118106de0 2024-04-05T21:17:11.920 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.908+0000 7f811cb83700 1 -- 172.21.15.121:0/1815909234 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8114011a30 con 0x7f8118106de0 2024-04-05T21:17:11.921 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.909+0000 7f811fde9700 1 -- 172.21.15.121:0/1815909234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 msgr2=0x7f81181091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:11.921 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.909+0000 7f811fde9700 1 --2- 172.21.15.121:0/1815909234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 0x7f81181091d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f81140098a0 tx=0x7f8114009bb0 comp rx=0 tx=0).stop 2024-04-05T21:17:11.921 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.909+0000 7f811fde9700 1 -- 172.21.15.121:0/1815909234 shutdown_connections 2024-04-05T21:17:11.921 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.909+0000 7f811fde9700 1 --2- 172.21.15.121:0/1815909234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 0x7f81181091d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:11.921 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.909+0000 7f811fde9700 1 -- 172.21.15.121:0/1815909234 >> 172.21.15.121:0/1815909234 conn(0x7f81181007d0 msgr2=0x7f8118102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:11.921 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.909+0000 7f811fde9700 1 -- 172.21.15.121:0/1815909234 shutdown_connections 2024-04-05T21:17:11.921 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.909+0000 7f811fde9700 1 -- 172.21.15.121:0/1815909234 wait complete. 2024-04-05T21:17:11.921 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.909+0000 7f811fde9700 1 Processor -- start 2024-04-05T21:17:11.921 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.909+0000 7f811fde9700 1 -- start start 2024-04-05T21:17:11.921 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.910+0000 7f811fde9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 0x7f811819c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.910+0000 7f811fde9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811819c760 con 0x7f8118106de0 2024-04-05T21:17:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.910+0000 7f811db85700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 0x7f811819c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.910+0000 7f811db85700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 0x7f811819c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39582/0 (socket says 172.21.15.121:39582) 2024-04-05T21:17:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.910+0000 7f811db85700 1 -- 172.21.15.121:0/3364441208 learned_addr learned my addr 172.21.15.121:0/3364441208 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.910+0000 7f811db85700 1 -- 172.21.15.121:0/3364441208 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8114009550 con 0x7f8118106de0 2024-04-05T21:17:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.910+0000 7f811db85700 1 --2- 172.21.15.121:0/3364441208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 0x7f811819c220 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8114006af0 tx=0x7f8114011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.910+0000 7f810effd700 1 -- 172.21.15.121:0/3364441208 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8114003bc0 con 0x7f8118106de0 2024-04-05T21:17:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.910+0000 7f810effd700 1 -- 172.21.15.121:0/3364441208 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f811401a8c0 con 0x7f8118106de0 2024-04-05T21:17:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.910+0000 7f810effd700 1 -- 172.21.15.121:0/3364441208 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8114019860 con 0x7f8118106de0 2024-04-05T21:17:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.910+0000 7f811fde9700 1 -- 172.21.15.121:0/3364441208 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811819c960 con 0x7f8118106de0 2024-04-05T21:17:11.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.911+0000 7f811fde9700 1 -- 172.21.15.121:0/3364441208 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811819cd80 con 0x7f8118106de0 2024-04-05T21:17:11.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.911+0000 7f810effd700 1 -- 172.21.15.121:0/3364441208 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 11) v1 ==== 45190+0+0 (secure 0 0 0) 0x7f8114019ad0 con 0x7f8118106de0 2024-04-05T21:17:11.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.911+0000 7f810effd700 1 --2- 172.21.15.121:0/3364441208 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f8104038260 0x7f810403a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:11.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.911+0000 7f810effd700 1 -- 172.21.15.121:0/3364441208 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f811404c3f0 con 0x7f8118106de0 2024-04-05T21:17:11.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.912+0000 7f811d384700 1 --2- 172.21.15.121:0/3364441208 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f8104038260 0x7f810403a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:11.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.912+0000 7f811fde9700 1 -- 172.21.15.121:0/3364441208 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81180623c0 con 0x7f8118106de0 2024-04-05T21:17:11.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.912+0000 7f811d384700 1 --2- 172.21.15.121:0/3364441208 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f8104038260 0x7f810403a720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8108006fd0 tx=0x7f8108006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:11.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:10.917+0000 7f810effd700 1 -- 172.21.15.121:0/3364441208 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8114019e20 con 0x7f8118106de0 2024-04-05T21:17:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:11.121+0000 7f811fde9700 1 -- 172.21.15.121:0/3364441208 --> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f8118102ae0 con 0x7f8104038260 2024-04-05T21:17:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:11.168+0000 7f810effd700 1 -- 172.21.15.121:0/3364441208 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f811402cc90 con 0x7f8118106de0 2024-04-05T21:17:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:11.277+0000 7f810effd700 1 -- 172.21.15.121:0/3364441208 <== mgr.14162 v2:172.21.15.121:6800/1237774071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f8118102ae0 con 0x7f8104038260 2024-04-05T21:17:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:11.279+0000 7f811fde9700 1 -- 172.21.15.121:0/3364441208 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f8104038260 msgr2=0x7f810403a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:11.279+0000 7f811fde9700 1 --2- 172.21.15.121:0/3364441208 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f8104038260 0x7f810403a720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8108006fd0 tx=0x7f8108006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:11.279+0000 7f811fde9700 1 -- 172.21.15.121:0/3364441208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 msgr2=0x7f811819c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:11.279+0000 7f811fde9700 1 --2- 172.21.15.121:0/3364441208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 0x7f811819c220 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8114006af0 tx=0x7f8114011ab0 comp rx=0 tx=0).stop 2024-04-05T21:17:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:11.279+0000 7f811fde9700 1 -- 172.21.15.121:0/3364441208 shutdown_connections 2024-04-05T21:17:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:11.279+0000 7f811fde9700 1 --2- 172.21.15.121:0/3364441208 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f8104038260 0x7f810403a720 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:11.279+0000 7f811fde9700 1 --2- 172.21.15.121:0/3364441208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8118106de0 0x7f811819c220 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:11.279+0000 7f811fde9700 1 -- 172.21.15.121:0/3364441208 >> 172.21.15.121:0/3364441208 conn(0x7f81181007d0 msgr2=0x7f8118101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:11.279+0000 7f811fde9700 1 -- 172.21.15.121:0/3364441208 shutdown_connections 2024-04-05T21:17:11.927 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:11.279+0000 7f811fde9700 1 -- 172.21.15.121:0/3364441208 wait complete. 2024-04-05T21:17:11.927 INFO:teuthology.orchestra.run.smithi121.stdout:Creating initial admin user... 2024-04-05T21:17:12.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:12 smithi121 ceph-mon[39723]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:17:12.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:12 smithi121 ceph-mon[39723]: mgrmap e12: smithi121.etbquh(active, since 2s) 2024-04-05T21:17:12.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:12 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:12.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:12 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:14.006 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$V37V0ulVLEZWoCdIKsT78e3sgXyJ/EXkzLfMAYe8X5/AGotyL.SQa", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712351832, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-05T21:17:14.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.503+0000 7f3f28ed9700 1 Processor -- start 2024-04-05T21:17:14.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.503+0000 7f3f28ed9700 1 -- start start 2024-04-05T21:17:14.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.503+0000 7f3f28ed9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 0x7f3f24106730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:14.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.503+0000 7f3f28ed9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f24106d00 con 0x7f3f24106310 2024-04-05T21:17:14.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.504+0000 7f3f2259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 0x7f3f24106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:14.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.504+0000 7f3f2259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 0x7f3f24106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39584/0 (socket says 172.21.15.121:39584) 2024-04-05T21:17:14.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.504+0000 7f3f2259c700 1 -- 172.21.15.121:0/2246019067 learned_addr learned my addr 172.21.15.121:0/2246019067 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:14.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.504+0000 7f3f2259c700 1 -- 172.21.15.121:0/2246019067 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f24107510 con 0x7f3f24106310 2024-04-05T21:17:14.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.504+0000 7f3f2259c700 1 --2- 172.21.15.121:0/2246019067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 0x7f3f24106730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c0098a0 tx=0x7f3f0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67eedce4c6167a67 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:14.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.504+0000 7f3f2159a700 1 -- 172.21.15.121:0/2246019067 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f0c004180 con 0x7f3f24106310 2024-04-05T21:17:14.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.504+0000 7f3f2159a700 1 -- 172.21.15.121:0/2246019067 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f3f0c0042e0 con 0x7f3f24106310 2024-04-05T21:17:14.007 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.504+0000 7f3f2159a700 1 -- 172.21.15.121:0/2246019067 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f0c011a30 con 0x7f3f24106310 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.505+0000 7f3f28ed9700 1 -- 172.21.15.121:0/2246019067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 msgr2=0x7f3f24106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.505+0000 7f3f28ed9700 1 --2- 172.21.15.121:0/2246019067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 0x7f3f24106730 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c0098a0 tx=0x7f3f0c009bb0 comp rx=0 tx=0).stop 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.505+0000 7f3f28ed9700 1 -- 172.21.15.121:0/2246019067 shutdown_connections 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.505+0000 7f3f28ed9700 1 --2- 172.21.15.121:0/2246019067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 0x7f3f24106730 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.505+0000 7f3f28ed9700 1 -- 172.21.15.121:0/2246019067 >> 172.21.15.121:0/2246019067 conn(0x7f3f241018d0 msgr2=0x7f3f24103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.505+0000 7f3f28ed9700 1 -- 172.21.15.121:0/2246019067 shutdown_connections 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.505+0000 7f3f28ed9700 1 -- 172.21.15.121:0/2246019067 wait complete. 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.506+0000 7f3f28ed9700 1 Processor -- start 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.506+0000 7f3f28ed9700 1 -- start start 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.506+0000 7f3f28ed9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 0x7f3f2419a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.506+0000 7f3f28ed9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2419a5c0 con 0x7f3f24106310 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.506+0000 7f3f2259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 0x7f3f2419a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.506+0000 7f3f2259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 0x7f3f2419a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39590/0 (socket says 172.21.15.121:39590) 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.506+0000 7f3f2259c700 1 -- 172.21.15.121:0/1619850425 learned_addr learned my addr 172.21.15.121:0/1619850425 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:14.008 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.506+0000 7f3f2259c700 1 -- 172.21.15.121:0/1619850425 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c009550 con 0x7f3f24106310 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.506+0000 7f3f2259c700 1 --2- 172.21.15.121:0/1619850425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 0x7f3f2419a080 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c000c00 tx=0x7f3f0c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.507+0000 7f3f1b7fe700 1 -- 172.21.15.121:0/1619850425 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f0c003bc0 con 0x7f3f24106310 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.507+0000 7f3f1b7fe700 1 -- 172.21.15.121:0/1619850425 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f3f0c0248d0 con 0x7f3f24106310 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.507+0000 7f3f1b7fe700 1 -- 172.21.15.121:0/1619850425 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f0c01a890 con 0x7f3f24106310 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.507+0000 7f3f28ed9700 1 -- 172.21.15.121:0/1619850425 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2419a7c0 con 0x7f3f24106310 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.507+0000 7f3f28ed9700 1 -- 172.21.15.121:0/1619850425 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2419abe0 con 0x7f3f24106310 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.507+0000 7f3f1b7fe700 1 -- 172.21.15.121:0/1619850425 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f3f0c01ab00 con 0x7f3f24106310 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.508+0000 7f3f1b7fe700 1 --2- 172.21.15.121:0/1619850425 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f3f100383c0 0x7f3f1003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.508+0000 7f3f28ed9700 1 -- 172.21.15.121:0/1619850425 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f241940f0 con 0x7f3f24106310 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.508+0000 7f3f1b7fe700 1 -- 172.21.15.121:0/1619850425 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3f0c04c770 con 0x7f3f24106310 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.508+0000 7f3f21d9b700 1 --2- 172.21.15.121:0/1619850425 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f3f100383c0 0x7f3f1003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.508+0000 7f3f21d9b700 1 --2- 172.21.15.121:0/1619850425 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f3f100383c0 0x7f3f1003a880 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3f14006fd0 tx=0x7f3f14006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.512+0000 7f3f1b7fe700 1 -- 172.21.15.121:0/1619850425 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3f0c01ae00 con 0x7f3f24106310 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.719+0000 7f3f28ed9700 1 -- 172.21.15.121:0/1619850425 --> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3f240621e0 con 0x7f3f100383c0 2024-04-05T21:17:14.009 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.938+0000 7f3f1b7fe700 1 -- 172.21.15.121:0/1619850425 <== mgr.14162 v2:172.21.15.121:6800/1237774071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f3f240621e0 con 0x7f3f100383c0 2024-04-05T21:17:14.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.940+0000 7f3f28ed9700 1 -- 172.21.15.121:0/1619850425 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f3f100383c0 msgr2=0x7f3f1003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:14.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.940+0000 7f3f28ed9700 1 --2- 172.21.15.121:0/1619850425 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f3f100383c0 0x7f3f1003a880 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3f14006fd0 tx=0x7f3f14006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:14.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.940+0000 7f3f28ed9700 1 -- 172.21.15.121:0/1619850425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 msgr2=0x7f3f2419a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:14.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.940+0000 7f3f28ed9700 1 --2- 172.21.15.121:0/1619850425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 0x7f3f2419a080 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c000c00 tx=0x7f3f0c011ab0 comp rx=0 tx=0).stop 2024-04-05T21:17:14.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.940+0000 7f3f28ed9700 1 -- 172.21.15.121:0/1619850425 shutdown_connections 2024-04-05T21:17:14.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.940+0000 7f3f28ed9700 1 --2- 172.21.15.121:0/1619850425 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f3f100383c0 0x7f3f1003a880 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:14.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.940+0000 7f3f28ed9700 1 --2- 172.21.15.121:0/1619850425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f24106310 0x7f3f2419a080 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:14.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.940+0000 7f3f28ed9700 1 -- 172.21.15.121:0/1619850425 >> 172.21.15.121:0/1619850425 conn(0x7f3f241018d0 msgr2=0x7f3f24102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:14.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.940+0000 7f3f28ed9700 1 -- 172.21.15.121:0/1619850425 shutdown_connections 2024-04-05T21:17:14.010 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:12.940+0000 7f3f28ed9700 1 -- 172.21.15.121:0/1619850425 wait complete. 2024-04-05T21:17:14.010 INFO:teuthology.orchestra.run.smithi121.stdout:Fetching dashboard port number... 2024-04-05T21:17:14.215 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:13 smithi121 ceph-mon[39723]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:17:14.215 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:13 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:15.044 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:14 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/102572665' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-05T21:17:15.305 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 8443 2024-04-05T21:17:15.305 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.539+0000 7f3527367700 1 Processor -- start 2024-04-05T21:17:15.305 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.539+0000 7f3527367700 1 -- start start 2024-04-05T21:17:15.305 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.539+0000 7f3527367700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 0x7f3520104a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:15.305 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.539+0000 7f3527367700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3520105000 con 0x7f3520104610 2024-04-05T21:17:15.305 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.539+0000 7f3525103700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 0x7f3520104a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:15.305 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.539+0000 7f3525103700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 0x7f3520104a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54434/0 (socket says 172.21.15.121:54434) 2024-04-05T21:17:15.305 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.539+0000 7f3525103700 1 -- 172.21.15.121:0/3923563284 learned_addr learned my addr 172.21.15.121:0/3923563284 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:15.305 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.540+0000 7f3525103700 1 -- 172.21.15.121:0/3923563284 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35201075a0 con 0x7f3520104610 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.540+0000 7f3525103700 1 --2- 172.21.15.121:0/3923563284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 0x7f3520104a30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f351c0098a0 tx=0x7f351c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5036c61b37366dae server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.540+0000 7f3517fff700 1 -- 172.21.15.121:0/3923563284 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f351c004180 con 0x7f3520104610 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.540+0000 7f3517fff700 1 -- 172.21.15.121:0/3923563284 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f351c0042e0 con 0x7f3520104610 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.540+0000 7f3517fff700 1 -- 172.21.15.121:0/3923563284 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f351c011a30 con 0x7f3520104610 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.540+0000 7f3527367700 1 -- 172.21.15.121:0/3923563284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 msgr2=0x7f3520104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.540+0000 7f3527367700 1 --2- 172.21.15.121:0/3923563284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 0x7f3520104a30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f351c0098a0 tx=0x7f351c009bb0 comp rx=0 tx=0).stop 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.540+0000 7f3527367700 1 -- 172.21.15.121:0/3923563284 shutdown_connections 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.540+0000 7f3527367700 1 --2- 172.21.15.121:0/3923563284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 0x7f3520104a30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.540+0000 7f3527367700 1 -- 172.21.15.121:0/3923563284 >> 172.21.15.121:0/3923563284 conn(0x7f35200fff40 msgr2=0x7f3520102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.541+0000 7f3527367700 1 -- 172.21.15.121:0/3923563284 shutdown_connections 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.541+0000 7f3527367700 1 -- 172.21.15.121:0/3923563284 wait complete. 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.541+0000 7f3527367700 1 Processor -- start 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.541+0000 7f3527367700 1 -- start start 2024-04-05T21:17:15.306 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.541+0000 7f3527367700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 0x7f3520106f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.541+0000 7f3527367700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3520107440 con 0x7f3520104610 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.541+0000 7f3525103700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 0x7f3520106f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.541+0000 7f3525103700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 0x7f3520106f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54446/0 (socket says 172.21.15.121:54446) 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.541+0000 7f3525103700 1 -- 172.21.15.121:0/102572665 learned_addr learned my addr 172.21.15.121:0/102572665 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.542+0000 7f3525103700 1 -- 172.21.15.121:0/102572665 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c009550 con 0x7f3520104610 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.542+0000 7f3525103700 1 --2- 172.21.15.121:0/102572665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 0x7f3520106f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f351c000c00 tx=0x7f351c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.542+0000 7f35167fc700 1 -- 172.21.15.121:0/102572665 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f351c003bc0 con 0x7f3520104610 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.542+0000 7f3527367700 1 -- 172.21.15.121:0/102572665 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3520105550 con 0x7f3520104610 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.542+0000 7f35167fc700 1 -- 172.21.15.121:0/102572665 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f351c0248d0 con 0x7f3520104610 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.542+0000 7f35167fc700 1 -- 172.21.15.121:0/102572665 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f351c01a890 con 0x7f3520104610 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.542+0000 7f3527367700 1 -- 172.21.15.121:0/102572665 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3520105970 con 0x7f3520104610 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.543+0000 7f35167fc700 1 -- 172.21.15.121:0/102572665 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 12) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f351c01ab50 con 0x7f3520104610 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.543+0000 7f3527367700 1 -- 172.21.15.121:0/102572665 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35200623c0 con 0x7f3520104610 2024-04-05T21:17:15.307 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.543+0000 7f35167fc700 1 --2- 172.21.15.121:0/102572665 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f350c038370 0x7f350c03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.543+0000 7f35167fc700 1 -- 172.21.15.121:0/102572665 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f351c04c800 con 0x7f3520104610 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.543+0000 7f3524902700 1 --2- 172.21.15.121:0/102572665 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f350c038370 0x7f350c03a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.543+0000 7f3524902700 1 --2- 172.21.15.121:0/102572665 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f350c038370 0x7f350c03a830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3510006fd0 tx=0x7f3510006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.547+0000 7f35167fc700 1 -- 172.21.15.121:0/102572665 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f351c0185d0 con 0x7f3520104610 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.736+0000 7f3527367700 1 -- 172.21.15.121:0/102572665 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f352010a7d0 con 0x7f3520104610 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.737+0000 7f35167fc700 1 -- 172.21.15.121:0/102572665 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f351c0185d0 con 0x7f3520104610 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.739+0000 7f3527367700 1 -- 172.21.15.121:0/102572665 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f350c038370 msgr2=0x7f350c03a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.739+0000 7f3527367700 1 --2- 172.21.15.121:0/102572665 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f350c038370 0x7f350c03a830 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3510006fd0 tx=0x7f3510006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.739+0000 7f3527367700 1 -- 172.21.15.121:0/102572665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 msgr2=0x7f3520106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.739+0000 7f3527367700 1 --2- 172.21.15.121:0/102572665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 0x7f3520106f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f351c000c00 tx=0x7f351c011ab0 comp rx=0 tx=0).stop 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.739+0000 7f3527367700 1 -- 172.21.15.121:0/102572665 shutdown_connections 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.739+0000 7f3527367700 1 --2- 172.21.15.121:0/102572665 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f350c038370 0x7f350c03a830 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.739+0000 7f3527367700 1 --2- 172.21.15.121:0/102572665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3520104610 0x7f3520106f00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:15.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.739+0000 7f3527367700 1 -- 172.21.15.121:0/102572665 >> 172.21.15.121:0/102572665 conn(0x7f35200fff40 msgr2=0x7f35201015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:15.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.739+0000 7f3527367700 1 -- 172.21.15.121:0/102572665 shutdown_connections 2024-04-05T21:17:15.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:14.739+0000 7f3527367700 1 -- 172.21.15.121:0/102572665 wait complete. 2024-04-05T21:17:15.312 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-05T21:17:15.312 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout disabled 2024-04-05T21:17:15.319 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-05T21:17:15.320 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-04-05T21:17:15.320 INFO:teuthology.orchestra.run.smithi121.stdout:firewalld.service is not enabled 2024-04-05T21:17:15.320 INFO:teuthology.orchestra.run.smithi121.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-05T21:17:15.322 INFO:teuthology.orchestra.run.smithi121.stdout:Ceph Dashboard is now available at: 2024-04-05T21:17:15.322 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:17:15.322 INFO:teuthology.orchestra.run.smithi121.stdout: URL: https://smithi121.front.sepia.ceph.com:8443/ 2024-04-05T21:17:15.322 INFO:teuthology.orchestra.run.smithi121.stdout: User: admin 2024-04-05T21:17:15.322 INFO:teuthology.orchestra.run.smithi121.stdout: Password: hzekg79i1g 2024-04-05T21:17:15.322 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:17:15.322 INFO:teuthology.orchestra.run.smithi121.stdout:Saving cluster configuration to /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config directory 2024-04-05T21:17:15.323 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling autotune for osd_memory_target 2024-04-05T21:17:16.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:15 smithi121 ceph-mon[39723]: mgrmap e13: smithi121.etbquh(active, since 6s) 2024-04-05T21:17:16.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.928+0000 7f069c6e0700 1 Processor -- start 2024-04-05T21:17:16.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.928+0000 7f069c6e0700 1 -- start start 2024-04-05T21:17:16.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.928+0000 7f069c6e0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 0x7f0694105650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:16.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.928+0000 7f069c6e0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0694105c20 con 0x7f0694105230 2024-04-05T21:17:16.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.929+0000 7f069a47c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 0x7f0694105650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:16.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.929+0000 7f069a47c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 0x7f0694105650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54456/0 (socket says 172.21.15.121:54456) 2024-04-05T21:17:16.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.929+0000 7f069a47c700 1 -- 172.21.15.121:0/1769631916 learned_addr learned my addr 172.21.15.121:0/1769631916 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:16.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.929+0000 7f069a47c700 1 -- 172.21.15.121:0/1769631916 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0694106430 con 0x7f0694105230 2024-04-05T21:17:16.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.929+0000 7f069a47c700 1 --2- 172.21.15.121:0/1769631916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 0x7f0694105650 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f06840098a0 tx=0x7f0684009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236281d922ec3b52 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.929+0000 7f069947a700 1 -- 172.21.15.121:0/1769631916 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0684004180 con 0x7f0694105230 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.929+0000 7f069947a700 1 -- 172.21.15.121:0/1769631916 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f06840042e0 con 0x7f0694105230 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.929+0000 7f069947a700 1 -- 172.21.15.121:0/1769631916 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0684011a30 con 0x7f0694105230 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.930+0000 7f069c6e0700 1 -- 172.21.15.121:0/1769631916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 msgr2=0x7f0694105650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.930+0000 7f069c6e0700 1 --2- 172.21.15.121:0/1769631916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 0x7f0694105650 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f06840098a0 tx=0x7f0684009bb0 comp rx=0 tx=0).stop 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.930+0000 7f069c6e0700 1 -- 172.21.15.121:0/1769631916 shutdown_connections 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.930+0000 7f069c6e0700 1 --2- 172.21.15.121:0/1769631916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 0x7f0694105650 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.930+0000 7f069c6e0700 1 -- 172.21.15.121:0/1769631916 >> 172.21.15.121:0/1769631916 conn(0x7f06941007f0 msgr2=0x7f0694102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.930+0000 7f069c6e0700 1 -- 172.21.15.121:0/1769631916 shutdown_connections 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.930+0000 7f069c6e0700 1 -- 172.21.15.121:0/1769631916 wait complete. 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.931+0000 7f069c6e0700 1 Processor -- start 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.931+0000 7f069c6e0700 1 -- start start 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.931+0000 7f069c6e0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 0x7f0694074210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.931+0000 7f069c6e0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069419cd80 con 0x7f0694105230 2024-04-05T21:17:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.931+0000 7f069a47c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 0x7f0694074210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.931+0000 7f069a47c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 0x7f0694074210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54460/0 (socket says 172.21.15.121:54460) 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.931+0000 7f069a47c700 1 -- 172.21.15.121:0/1025035747 learned_addr learned my addr 172.21.15.121:0/1025035747 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.931+0000 7f069a47c700 1 -- 172.21.15.121:0/1025035747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684009550 con 0x7f0694105230 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.931+0000 7f069a47c700 1 --2- 172.21.15.121:0/1025035747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 0x7f0694074210 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0684006af0 tx=0x7f0684011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.931+0000 7f068b7fe700 1 -- 172.21.15.121:0/1025035747 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0684003a80 con 0x7f0694105230 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.932+0000 7f068b7fe700 1 -- 172.21.15.121:0/1025035747 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f0684003be0 con 0x7f0694105230 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.932+0000 7f069c6e0700 1 -- 172.21.15.121:0/1025035747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069419cec0 con 0x7f0694105230 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.932+0000 7f068b7fe700 1 -- 172.21.15.121:0/1025035747 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0684019770 con 0x7f0694105230 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.932+0000 7f069c6e0700 1 -- 172.21.15.121:0/1025035747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069419d2e0 con 0x7f0694105230 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.932+0000 7f068b7fe700 1 -- 172.21.15.121:0/1025035747 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f068401a430 con 0x7f0694105230 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.932+0000 7f069c6e0700 1 -- 172.21.15.121:0/1025035747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0694193010 con 0x7f0694105230 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.932+0000 7f068b7fe700 1 --2- 172.21.15.121:0/1025035747 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f0680038420 0x7f068003a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.933+0000 7f068b7fe700 1 -- 172.21.15.121:0/1025035747 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f068404c6d0 con 0x7f0694105230 2024-04-05T21:17:16.788 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.933+0000 7f0699c7b700 1 --2- 172.21.15.121:0/1025035747 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f0680038420 0x7f068003a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.933+0000 7f0699c7b700 1 --2- 172.21.15.121:0/1025035747 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f0680038420 0x7f068003a8e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0690006fd0 tx=0x7f0690006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:15.937+0000 7f068b7fe700 1 -- 172.21.15.121:0/1025035747 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f0684019a20 con 0x7f0694105230 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:16.122+0000 7f069c6e0700 1 -- 172.21.15.121:0/1025035747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f069402d090 con 0x7f0694105230 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:16.123+0000 7f068b7fe700 1 -- 172.21.15.121:0/1025035747 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f06840212a0 con 0x7f0694105230 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:16.125+0000 7f069c6e0700 1 -- 172.21.15.121:0/1025035747 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f0680038420 msgr2=0x7f068003a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:16.125+0000 7f069c6e0700 1 --2- 172.21.15.121:0/1025035747 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f0680038420 0x7f068003a8e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0690006fd0 tx=0x7f0690006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:16.125+0000 7f069c6e0700 1 -- 172.21.15.121:0/1025035747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 msgr2=0x7f0694074210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:16.125+0000 7f069c6e0700 1 --2- 172.21.15.121:0/1025035747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 0x7f0694074210 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0684006af0 tx=0x7f0684011b20 comp rx=0 tx=0).stop 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:16.125+0000 7f069c6e0700 1 -- 172.21.15.121:0/1025035747 shutdown_connections 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:16.125+0000 7f069c6e0700 1 --2- 172.21.15.121:0/1025035747 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f0680038420 0x7f068003a8e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:16.125+0000 7f069c6e0700 1 --2- 172.21.15.121:0/1025035747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0694105230 0x7f0694074210 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:16.125+0000 7f069c6e0700 1 -- 172.21.15.121:0/1025035747 >> 172.21.15.121:0/1025035747 conn(0x7f06941007f0 msgr2=0x7f0694101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:16.125+0000 7f069c6e0700 1 -- 172.21.15.121:0/1025035747 shutdown_connections 2024-04-05T21:17:16.789 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:16.125+0000 7f069c6e0700 1 -- 172.21.15.121:0/1025035747 wait complete. 2024-04-05T21:17:18.339 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.414+0000 7f20e704e700 1 Processor -- start 2024-04-05T21:17:18.339 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.414+0000 7f20e704e700 1 -- start start 2024-04-05T21:17:18.339 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.415+0000 7f20e704e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 0x7f20e0105050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:18.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.415+0000 7f20e704e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e0074490 con 0x7f20e0104c30 2024-04-05T21:17:18.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.415+0000 7f20e4dea700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 0x7f20e0105050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:18.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.415+0000 7f20e4dea700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 0x7f20e0105050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54468/0 (socket says 172.21.15.121:54468) 2024-04-05T21:17:18.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.415+0000 7f20e4dea700 1 -- 172.21.15.121:0/1701306719 learned_addr learned my addr 172.21.15.121:0/1701306719 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:18.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.415+0000 7f20e4dea700 1 -- 172.21.15.121:0/1701306719 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e00745d0 con 0x7f20e0104c30 2024-04-05T21:17:18.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.415+0000 7f20e4dea700 1 --2- 172.21.15.121:0/1701306719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 0x7f20e0105050 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f20d40098a0 tx=0x7f20d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d0412ad38ef33b server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:18.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.415+0000 7f20df7fe700 1 -- 172.21.15.121:0/1701306719 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20d4004180 con 0x7f20e0104c30 2024-04-05T21:17:18.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.415+0000 7f20df7fe700 1 -- 172.21.15.121:0/1701306719 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f20d40042e0 con 0x7f20e0104c30 2024-04-05T21:17:18.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.415+0000 7f20df7fe700 1 -- 172.21.15.121:0/1701306719 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20d4011a30 con 0x7f20e0104c30 2024-04-05T21:17:18.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.416+0000 7f20e704e700 1 -- 172.21.15.121:0/1701306719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 msgr2=0x7f20e0105050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:18.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.416+0000 7f20e704e700 1 --2- 172.21.15.121:0/1701306719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 0x7f20e0105050 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f20d40098a0 tx=0x7f20d4009bb0 comp rx=0 tx=0).stop 2024-04-05T21:17:18.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.416+0000 7f20e704e700 1 -- 172.21.15.121:0/1701306719 shutdown_connections 2024-04-05T21:17:18.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.416+0000 7f20e704e700 1 --2- 172.21.15.121:0/1701306719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 0x7f20e0105050 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.416+0000 7f20e704e700 1 -- 172.21.15.121:0/1701306719 >> 172.21.15.121:0/1701306719 conn(0x7f20e01007f0 msgr2=0x7f20e0102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.416+0000 7f20e704e700 1 -- 172.21.15.121:0/1701306719 shutdown_connections 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.416+0000 7f20e704e700 1 -- 172.21.15.121:0/1701306719 wait complete. 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.417+0000 7f20e704e700 1 Processor -- start 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.417+0000 7f20e704e700 1 -- start start 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.417+0000 7f20e704e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 0x7f20e019c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.417+0000 7f20e704e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e019c780 con 0x7f20e0104c30 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.417+0000 7f20e4dea700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 0x7f20e019c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.417+0000 7f20e4dea700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 0x7f20e019c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54482/0 (socket says 172.21.15.121:54482) 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.417+0000 7f20e4dea700 1 -- 172.21.15.121:0/490678593 learned_addr learned my addr 172.21.15.121:0/490678593 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.417+0000 7f20e4dea700 1 -- 172.21.15.121:0/490678593 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d4009550 con 0x7f20e0104c30 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.417+0000 7f20e4dea700 1 --2- 172.21.15.121:0/490678593 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 0x7f20e019c240 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20d4006af0 tx=0x7f20d4011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.418+0000 7f20ddffb700 1 -- 172.21.15.121:0/490678593 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20d4003a80 con 0x7f20e0104c30 2024-04-05T21:17:18.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.418+0000 7f20ddffb700 1 -- 172.21.15.121:0/490678593 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f20d4003be0 con 0x7f20e0104c30 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.418+0000 7f20e704e700 1 -- 172.21.15.121:0/490678593 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e019c980 con 0x7f20e0104c30 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.418+0000 7f20ddffb700 1 -- 172.21.15.121:0/490678593 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20d4019770 con 0x7f20e0104c30 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.418+0000 7f20e704e700 1 -- 172.21.15.121:0/490678593 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e019cda0 con 0x7f20e0104c30 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.419+0000 7f20ddffb700 1 -- 172.21.15.121:0/490678593 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f20d40199e0 con 0x7f20e0104c30 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.419+0000 7f20e704e700 1 -- 172.21.15.121:0/490678593 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e0195b10 con 0x7f20e0104c30 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.419+0000 7f20ddffb700 1 --2- 172.21.15.121:0/490678593 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f20d0038420 0x7f20d003a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.419+0000 7f20ddffb700 1 -- 172.21.15.121:0/490678593 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f20d404c710 con 0x7f20e0104c30 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.419+0000 7f20dffff700 1 --2- 172.21.15.121:0/490678593 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f20d0038420 0x7f20d003a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.419+0000 7f20dffff700 1 --2- 172.21.15.121:0/490678593 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f20d0038420 0x7f20d003a8e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f20cc006fd0 tx=0x7f20cc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.424+0000 7f20ddffb700 1 -- 172.21.15.121:0/490678593 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f20d402b750 con 0x7f20e0104c30 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.713+0000 7f20e704e700 1 -- 172.21.15.121:0/490678593 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f20e002cf20 con 0x7f20e0104c30 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.715+0000 7f20ddffb700 1 -- 172.21.15.121:0/490678593 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f20d40172b0 con 0x7f20e0104c30 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.717+0000 7f20e704e700 1 -- 172.21.15.121:0/490678593 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f20d0038420 msgr2=0x7f20d003a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:18.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.717+0000 7f20e704e700 1 --2- 172.21.15.121:0/490678593 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f20d0038420 0x7f20d003a8e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f20cc006fd0 tx=0x7f20cc006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.717+0000 7f20e704e700 1 -- 172.21.15.121:0/490678593 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 msgr2=0x7f20e019c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.717+0000 7f20e704e700 1 --2- 172.21.15.121:0/490678593 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 0x7f20e019c240 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20d4006af0 tx=0x7f20d4011b20 comp rx=0 tx=0).stop 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.717+0000 7f20e704e700 1 -- 172.21.15.121:0/490678593 shutdown_connections 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.718+0000 7f20e704e700 1 --2- 172.21.15.121:0/490678593 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f20d0038420 0x7f20d003a8e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.718+0000 7f20e704e700 1 --2- 172.21.15.121:0/490678593 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e0104c30 0x7f20e019c240 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.718+0000 7f20e704e700 1 -- 172.21.15.121:0/490678593 >> 172.21.15.121:0/490678593 conn(0x7f20e01007f0 msgr2=0x7f20e01907f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.718+0000 7f20e704e700 1 -- 172.21.15.121:0/490678593 shutdown_connections 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-04-05T21:17:17.718+0000 7f20e704e700 1 -- 172.21.15.121:0/490678593 wait complete. 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout:Or, if you are only running a single cluster on this host: 2024-04-05T21:17:18.343 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:17:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-05T21:17:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:17:18.344 INFO:teuthology.orchestra.run.smithi121.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-05T21:17:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:17:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: ceph telemetry on 2024-04-05T21:17:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:17:18.344 INFO:teuthology.orchestra.run.smithi121.stdout:For more information see: 2024-04-05T21:17:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:17:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-05T21:17:18.344 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:17:18.344 INFO:teuthology.orchestra.run.smithi121.stdout:Bootstrap complete. 2024-04-05T21:17:18.372 INFO:tasks.cephadm:Fetching config... 2024-04-05T21:17:18.373 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:17:18.373 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-05T21:17:18.432 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-05T21:17:18.432 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:17:18.432 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-05T21:17:18.490 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-05T21:17:18.490 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:17:18.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/keyring of=/dev/stdout 2024-04-05T21:17:18.557 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-05T21:17:18.557 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:17:18.557 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-05T21:17:18.615 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-05T21:17:18.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZEkPkm9R+lyo91WlIkAk6OEW7qkCToqF+TI4I5zWm6+TfxBRjkEpByiYhK6pDrxxtGOlvx1yWZm/6Wzl67bgIzviXJmnZKbePJIzIY6TKqZGlOUTYPqKNAs9Fv+zjkwKAl7z4suLH5BPfEzQJt5Fs5dxTkP+XgNheoyq0Vegbt0ui5JfxrQIA5tZIqb1awAbnJAKcQepBArocGNmDgUem/Y312GXkqqvSxBOscDoe2y7DpoDSCvneouZ2R4kVHeRmoJUHuV7Nz6oMWcAUP6S5yyUidRb5kFAc2kKOpXlqQSNcnlPf7rwksHqwcLCc4/vvNWAIE7uwJMQM9f24LstyrUTK8EzyCQWNoiFpEY+fYe8JDutCwFTyBL6ZBeKhg0gANDov3owtxXyJKTyiCA9aBtm6/MKhNArnWIibH7e1PiYZLYOHveGhM6lz4ov++iUzWNBjOONDgL20Ws3lBGiT4cYpEeLcFq0AEWVl05oqYYlEqKYKvrP7eJUt3KIvWcU= ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-05T21:17:18.697 INFO:teuthology.orchestra.run.smithi121.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZEkPkm9R+lyo91WlIkAk6OEW7qkCToqF+TI4I5zWm6+TfxBRjkEpByiYhK6pDrxxtGOlvx1yWZm/6Wzl67bgIzviXJmnZKbePJIzIY6TKqZGlOUTYPqKNAs9Fv+zjkwKAl7z4suLH5BPfEzQJt5Fs5dxTkP+XgNheoyq0Vegbt0ui5JfxrQIA5tZIqb1awAbnJAKcQepBArocGNmDgUem/Y312GXkqqvSxBOscDoe2y7DpoDSCvneouZ2R4kVHeRmoJUHuV7Nz6oMWcAUP6S5yyUidRb5kFAc2kKOpXlqQSNcnlPf7rwksHqwcLCc4/vvNWAIE7uwJMQM9f24LstyrUTK8EzyCQWNoiFpEY+fYe8JDutCwFTyBL6ZBeKhg0gANDov3owtxXyJKTyiCA9aBtm6/MKhNArnWIibH7e1PiYZLYOHveGhM6lz4ov++iUzWNBjOONDgL20Ws3lBGiT4cYpEeLcFq0AEWVl05oqYYlEqKYKvrP7eJUt3KIvWcU= ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:17:18.707 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZEkPkm9R+lyo91WlIkAk6OEW7qkCToqF+TI4I5zWm6+TfxBRjkEpByiYhK6pDrxxtGOlvx1yWZm/6Wzl67bgIzviXJmnZKbePJIzIY6TKqZGlOUTYPqKNAs9Fv+zjkwKAl7z4suLH5BPfEzQJt5Fs5dxTkP+XgNheoyq0Vegbt0ui5JfxrQIA5tZIqb1awAbnJAKcQepBArocGNmDgUem/Y312GXkqqvSxBOscDoe2y7DpoDSCvneouZ2R4kVHeRmoJUHuV7Nz6oMWcAUP6S5yyUidRb5kFAc2kKOpXlqQSNcnlPf7rwksHqwcLCc4/vvNWAIE7uwJMQM9f24LstyrUTK8EzyCQWNoiFpEY+fYe8JDutCwFTyBL6ZBeKhg0gANDov3owtxXyJKTyiCA9aBtm6/MKhNArnWIibH7e1PiYZLYOHveGhM6lz4ov++iUzWNBjOONDgL20Ws3lBGiT4cYpEeLcFq0AEWVl05oqYYlEqKYKvrP7eJUt3KIvWcU= ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-05T21:17:18.752 INFO:teuthology.orchestra.run.smithi188.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZEkPkm9R+lyo91WlIkAk6OEW7qkCToqF+TI4I5zWm6+TfxBRjkEpByiYhK6pDrxxtGOlvx1yWZm/6Wzl67bgIzviXJmnZKbePJIzIY6TKqZGlOUTYPqKNAs9Fv+zjkwKAl7z4suLH5BPfEzQJt5Fs5dxTkP+XgNheoyq0Vegbt0ui5JfxrQIA5tZIqb1awAbnJAKcQepBArocGNmDgUem/Y312GXkqqvSxBOscDoe2y7DpoDSCvneouZ2R4kVHeRmoJUHuV7Nz6oMWcAUP6S5yyUidRb5kFAc2kKOpXlqQSNcnlPf7rwksHqwcLCc4/vvNWAIE7uwJMQM9f24LstyrUTK8EzyCQWNoiFpEY+fYe8JDutCwFTyBL6ZBeKhg0gANDov3owtxXyJKTyiCA9aBtm6/MKhNArnWIibH7e1PiYZLYOHveGhM6lz4ov++iUzWNBjOONDgL20Ws3lBGiT4cYpEeLcFq0AEWVl05oqYYlEqKYKvrP7eJUt3KIvWcU= ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:17:18.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-05T21:17:18.983 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:17:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:18 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/490678593' entity='client.admin' 2024-04-05T21:17:21.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.624+0000 7efc21504700 1 -- 172.21.15.121:0/540600970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c0ff7a0 msgr2=0x7efc1c0ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:21.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.624+0000 7efc21504700 1 --2- 172.21.15.121:0/540600970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c0ff7a0 0x7efc1c0ffbc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efc04009910 tx=0x7efc04009c20 comp rx=0 tx=0).stop 2024-04-05T21:17:21.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.624+0000 7efc21504700 1 -- 172.21.15.121:0/540600970 shutdown_connections 2024-04-05T21:17:21.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.624+0000 7efc21504700 1 --2- 172.21.15.121:0/540600970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c0ff7a0 0x7efc1c0ffbc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:21.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.624+0000 7efc21504700 1 -- 172.21.15.121:0/540600970 >> 172.21.15.121:0/540600970 conn(0x7efc1c0fad20 msgr2=0x7efc1c0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:21.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.624+0000 7efc21504700 1 -- 172.21.15.121:0/540600970 shutdown_connections 2024-04-05T21:17:21.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.624+0000 7efc21504700 1 -- 172.21.15.121:0/540600970 wait complete. 2024-04-05T21:17:21.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.625+0000 7efc21504700 1 Processor -- start 2024-04-05T21:17:21.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.625+0000 7efc21504700 1 -- start start 2024-04-05T21:17:21.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.625+0000 7efc21504700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c0ff7a0 0x7efc1c0693f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:21.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.625+0000 7efc21504700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c1985e0 con 0x7efc1c0ff7a0 2024-04-05T21:17:21.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.625+0000 7efc1affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c0ff7a0 0x7efc1c0693f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:21.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.625+0000 7efc1affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c0ff7a0 0x7efc1c0693f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54512/0 (socket says 172.21.15.121:54512) 2024-04-05T21:17:21.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.625+0000 7efc1affd700 1 -- 172.21.15.121:0/2872841105 learned_addr learned my addr 172.21.15.121:0/2872841105 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:21.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.625+0000 7efc1affd700 1 -- 172.21.15.121:0/2872841105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc040095f0 con 0x7efc1c0ff7a0 2024-04-05T21:17:21.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.625+0000 7efc1affd700 1 --2- 172.21.15.121:0/2872841105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c0ff7a0 0x7efc1c0693f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efc04006010 tx=0x7efc040057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:21.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.626+0000 7efc13fff700 1 -- 172.21.15.121:0/2872841105 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc0401d070 con 0x7efc1c0ff7a0 2024-04-05T21:17:21.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.626+0000 7efc21504700 1 -- 172.21.15.121:0/2872841105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c198720 con 0x7efc1c0ff7a0 2024-04-05T21:17:21.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.626+0000 7efc13fff700 1 -- 172.21.15.121:0/2872841105 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7efc0400bdf0 con 0x7efc1c0ff7a0 2024-04-05T21:17:21.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.626+0000 7efc13fff700 1 -- 172.21.15.121:0/2872841105 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc04022e50 con 0x7efc1c0ff7a0 2024-04-05T21:17:21.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.626+0000 7efc21504700 1 -- 172.21.15.121:0/2872841105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c198bc0 con 0x7efc1c0ff7a0 2024-04-05T21:17:21.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.626+0000 7efc13fff700 1 -- 172.21.15.121:0/2872841105 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7efc0400f4b0 con 0x7efc1c0ff7a0 2024-04-05T21:17:21.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.627+0000 7efc13fff700 1 --2- 172.21.15.121:0/2872841105 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7efc08038370 0x7efc0803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:21.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.627+0000 7efc13fff700 1 -- 172.21.15.121:0/2872841105 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efc0404c3a0 con 0x7efc1c0ff7a0 2024-04-05T21:17:21.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.627+0000 7efc1a7fc700 1 --2- 172.21.15.121:0/2872841105 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7efc08038370 0x7efc0803a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:21.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.627+0000 7efc21504700 1 -- 172.21.15.121:0/2872841105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c18e280 con 0x7efc1c0ff7a0 2024-04-05T21:17:21.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.627+0000 7efc1a7fc700 1 --2- 172.21.15.121:0/2872841105 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7efc08038370 0x7efc0803a830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efc0c006fd0 tx=0x7efc0c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:21.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.631+0000 7efc13fff700 1 -- 172.21.15.121:0/2872841105 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7efc04027070 con 0x7efc1c0ff7a0 2024-04-05T21:17:21.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.823+0000 7efc21504700 1 -- 172.21.15.121:0/2872841105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7efc1c02cc70 con 0x7efc1c0ff7a0 2024-04-05T21:17:21.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.826+0000 7efc13fff700 1 -- 172.21.15.121:0/2872841105 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7efc04034070 con 0x7efc1c0ff7a0 2024-04-05T21:17:21.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.828+0000 7efc21504700 1 -- 172.21.15.121:0/2872841105 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7efc08038370 msgr2=0x7efc0803a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:21.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.828+0000 7efc21504700 1 --2- 172.21.15.121:0/2872841105 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7efc08038370 0x7efc0803a830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efc0c006fd0 tx=0x7efc0c006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:21.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.828+0000 7efc21504700 1 -- 172.21.15.121:0/2872841105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c0ff7a0 msgr2=0x7efc1c0693f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:21.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.828+0000 7efc21504700 1 --2- 172.21.15.121:0/2872841105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c0ff7a0 0x7efc1c0693f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efc04006010 tx=0x7efc040057b0 comp rx=0 tx=0).stop 2024-04-05T21:17:21.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.829+0000 7efc21504700 1 -- 172.21.15.121:0/2872841105 shutdown_connections 2024-04-05T21:17:21.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.829+0000 7efc21504700 1 --2- 172.21.15.121:0/2872841105 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7efc08038370 0x7efc0803a830 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:21.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.829+0000 7efc21504700 1 --2- 172.21.15.121:0/2872841105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c0ff7a0 0x7efc1c0693f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:21.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.829+0000 7efc21504700 1 -- 172.21.15.121:0/2872841105 >> 172.21.15.121:0/2872841105 conn(0x7efc1c0fad20 msgr2=0x7efc1c0fba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:21.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.829+0000 7efc21504700 1 -- 172.21.15.121:0/2872841105 shutdown_connections 2024-04-05T21:17:21.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:21.829+0000 7efc21504700 1 -- 172.21.15.121:0/2872841105 wait complete. 2024-04-05T21:17:22.559 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-05T21:17:22.559 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-05T21:17:23.217 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:22 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2872841105' entity='client.admin' 2024-04-05T21:17:23.345 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:17:24.650 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:24 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:24.651 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:24 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:24.651 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:24 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi121", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:17:24.651 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:24 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:24.651 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:24 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi121", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-05T21:17:24.651 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:24 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi121", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-05T21:17:24.651 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:24 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:17:24.652 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:24 smithi121 ceph-mon[39723]: Deploying daemon ceph-exporter.smithi121 on smithi121 2024-04-05T21:17:25.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.973+0000 7f83dc7a4700 1 -- 172.21.15.121:0/3076689740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d40ff7a0 msgr2=0x7f83d40ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:25.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.973+0000 7f83dc7a4700 1 --2- 172.21.15.121:0/3076689740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d40ff7a0 0x7f83d40ffbc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009910 tx=0x7f83d0009c20 comp rx=0 tx=0).stop 2024-04-05T21:17:25.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.973+0000 7f83dc7a4700 1 -- 172.21.15.121:0/3076689740 shutdown_connections 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.973+0000 7f83dc7a4700 1 --2- 172.21.15.121:0/3076689740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d40ff7a0 0x7f83d40ffbc0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.973+0000 7f83dc7a4700 1 -- 172.21.15.121:0/3076689740 >> 172.21.15.121:0/3076689740 conn(0x7f83d40fad20 msgr2=0x7f83d40fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.974+0000 7f83dc7a4700 1 -- 172.21.15.121:0/3076689740 shutdown_connections 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.974+0000 7f83dc7a4700 1 -- 172.21.15.121:0/3076689740 wait complete. 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.974+0000 7f83dc7a4700 1 Processor -- start 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.974+0000 7f83dc7a4700 1 -- start start 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.974+0000 7f83dc7a4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d40ff7a0 0x7f83d4194800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.975+0000 7f83dc7a4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d4194d40 con 0x7f83d40ff7a0 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.975+0000 7f83da540700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d40ff7a0 0x7f83d4194800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.975+0000 7f83da540700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d40ff7a0 0x7f83d4194800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51528/0 (socket says 172.21.15.121:51528) 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.975+0000 7f83da540700 1 -- 172.21.15.121:0/4289847167 learned_addr learned my addr 172.21.15.121:0/4289847167 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.975+0000 7f83da540700 1 -- 172.21.15.121:0/4289847167 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d00095f0 con 0x7f83d40ff7a0 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.975+0000 7f83da540700 1 --2- 172.21.15.121:0/4289847167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d40ff7a0 0x7f83d4194800 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f83d0006010 tx=0x7f83d00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:25.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.975+0000 7f83cb7fe700 1 -- 172.21.15.121:0/4289847167 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83d001d070 con 0x7f83d40ff7a0 2024-04-05T21:17:25.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.975+0000 7f83cb7fe700 1 -- 172.21.15.121:0/4289847167 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f83d000bdf0 con 0x7f83d40ff7a0 2024-04-05T21:17:25.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.975+0000 7f83dc7a4700 1 -- 172.21.15.121:0/4289847167 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d4194f40 con 0x7f83d40ff7a0 2024-04-05T21:17:25.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.975+0000 7f83cb7fe700 1 -- 172.21.15.121:0/4289847167 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83d0022e50 con 0x7f83d40ff7a0 2024-04-05T21:17:25.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.975+0000 7f83dc7a4700 1 -- 172.21.15.121:0/4289847167 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d41953e0 con 0x7f83d40ff7a0 2024-04-05T21:17:25.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.976+0000 7f83dc7a4700 1 -- 172.21.15.121:0/4289847167 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d418e280 con 0x7f83d40ff7a0 2024-04-05T21:17:25.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.976+0000 7f83cb7fe700 1 -- 172.21.15.121:0/4289847167 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f83d0022700 con 0x7f83d40ff7a0 2024-04-05T21:17:25.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.976+0000 7f83cb7fe700 1 --2- 172.21.15.121:0/4289847167 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f83c0038370 0x7f83c003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:25.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.977+0000 7f83cb7fe700 1 -- 172.21.15.121:0/4289847167 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f83d004c350 con 0x7f83d40ff7a0 2024-04-05T21:17:25.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.977+0000 7f83d9d3f700 1 --2- 172.21.15.121:0/4289847167 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f83c0038370 0x7f83c003a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:25.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.977+0000 7f83d9d3f700 1 --2- 172.21.15.121:0/4289847167 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f83c0038370 0x7f83c003a830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f83c4006fd0 tx=0x7f83c4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:25.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:25.981+0000 7f83cb7fe700 1 -- 172.21.15.121:0/4289847167 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f83d000b960 con 0x7f83d40ff7a0 2024-04-05T21:17:26.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:26.201+0000 7f83dc7a4700 1 -- 172.21.15.121:0/4289847167 --> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f83d4061260 con 0x7f83c0038370 2024-04-05T21:17:26.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:26.205+0000 7f83cb7fe700 1 -- 172.21.15.121:0/4289847167 <== mgr.14162 v2:172.21.15.121:6800/1237774071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f83d4061260 con 0x7f83c0038370 2024-04-05T21:17:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:26.207+0000 7f83dc7a4700 1 -- 172.21.15.121:0/4289847167 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f83c0038370 msgr2=0x7f83c003a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:26.207+0000 7f83dc7a4700 1 --2- 172.21.15.121:0/4289847167 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f83c0038370 0x7f83c003a830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f83c4006fd0 tx=0x7f83c4006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:26.207+0000 7f83dc7a4700 1 -- 172.21.15.121:0/4289847167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d40ff7a0 msgr2=0x7f83d4194800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:26.207+0000 7f83dc7a4700 1 --2- 172.21.15.121:0/4289847167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d40ff7a0 0x7f83d4194800 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f83d0006010 tx=0x7f83d00057b0 comp rx=0 tx=0).stop 2024-04-05T21:17:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:26.207+0000 7f83dc7a4700 1 -- 172.21.15.121:0/4289847167 shutdown_connections 2024-04-05T21:17:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:26.207+0000 7f83dc7a4700 1 --2- 172.21.15.121:0/4289847167 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f83c0038370 0x7f83c003a830 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:26.207+0000 7f83dc7a4700 1 --2- 172.21.15.121:0/4289847167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d40ff7a0 0x7f83d4194800 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:26.207+0000 7f83dc7a4700 1 -- 172.21.15.121:0/4289847167 >> 172.21.15.121:0/4289847167 conn(0x7f83d40fad20 msgr2=0x7f83d40fba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:26.207+0000 7f83dc7a4700 1 -- 172.21.15.121:0/4289847167 shutdown_connections 2024-04-05T21:17:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:26.207+0000 7f83dc7a4700 1 -- 172.21.15.121:0/4289847167 wait complete. 2024-04-05T21:17:27.294 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi188 2024-04-05T21:17:27.294 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-05T21:17:27.294 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.conf 2024-04-05T21:17:27.316 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-05T21:17:27.316 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-05T21:17:27.376 INFO:tasks.cephadm:Adding host smithi188 to orchestrator... 2024-04-05T21:17:27.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph orch host add smithi188 2024-04-05T21:17:27.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:27 smithi121 ceph-mon[39723]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:17:27.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:27 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:27.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:27 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:27.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:27 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:27.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:27 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:27.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:27 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:27.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:27 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi121", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T21:17:27.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:27 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi121", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-05T21:17:27.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:27 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:17:27.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:27 smithi121 ceph-mon[39723]: Deploying daemon crash.smithi121 on smithi121 2024-04-05T21:17:28.124 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:17:29.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:29 smithi121 ceph-mon[39723]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:29.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:29 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:31.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.200+0000 7f1ce977f700 1 -- 172.21.15.121:0/3795362960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce41012b0 msgr2=0x7f1ce41036a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:31.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.200+0000 7f1ce977f700 1 --2- 172.21.15.121:0/3795362960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce41012b0 0x7f1ce41036a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4009910 tx=0x7f1cd4009c20 comp rx=0 tx=0).stop 2024-04-05T21:17:31.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.200+0000 7f1ce977f700 1 -- 172.21.15.121:0/3795362960 shutdown_connections 2024-04-05T21:17:31.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.200+0000 7f1ce977f700 1 --2- 172.21.15.121:0/3795362960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce41012b0 0x7f1ce41036a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:31.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.200+0000 7f1ce977f700 1 -- 172.21.15.121:0/3795362960 >> 172.21.15.121:0/3795362960 conn(0x7f1ce40facc0 msgr2=0x7f1ce40fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:31.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.200+0000 7f1ce977f700 1 -- 172.21.15.121:0/3795362960 shutdown_connections 2024-04-05T21:17:31.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.200+0000 7f1ce977f700 1 -- 172.21.15.121:0/3795362960 wait complete. 2024-04-05T21:17:31.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.201+0000 7f1ce977f700 1 Processor -- start 2024-04-05T21:17:31.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.201+0000 7f1ce977f700 1 -- start start 2024-04-05T21:17:31.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.201+0000 7f1ce977f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce41012b0 0x7f1ce4195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.201+0000 7f1ce977f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce4195d70 con 0x7f1ce41012b0 2024-04-05T21:17:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.202+0000 7f1ce2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce41012b0 0x7f1ce4195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.202+0000 7f1ce2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce41012b0 0x7f1ce4195830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51548/0 (socket says 172.21.15.121:51548) 2024-04-05T21:17:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.202+0000 7f1ce2ffd700 1 -- 172.21.15.121:0/4232860021 learned_addr learned my addr 172.21.15.121:0/4232860021 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.202+0000 7f1ce2ffd700 1 -- 172.21.15.121:0/4232860021 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd40095f0 con 0x7f1ce41012b0 2024-04-05T21:17:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.202+0000 7f1ce2ffd700 1 --2- 172.21.15.121:0/4232860021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce41012b0 0x7f1ce4195830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4004f40 tx=0x7f1cd40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.202+0000 7f1ccbfff700 1 -- 172.21.15.121:0/4232860021 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cd401d070 con 0x7f1ce41012b0 2024-04-05T21:17:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.202+0000 7f1ccbfff700 1 -- 172.21.15.121:0/4232860021 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f1cd400bdf0 con 0x7f1ce41012b0 2024-04-05T21:17:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.202+0000 7f1ce977f700 1 -- 172.21.15.121:0/4232860021 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce4195f70 con 0x7f1ce41012b0 2024-04-05T21:17:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.202+0000 7f1ccbfff700 1 -- 172.21.15.121:0/4232860021 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cd4022e00 con 0x7f1ce41012b0 2024-04-05T21:17:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.202+0000 7f1ce977f700 1 -- 172.21.15.121:0/4232860021 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce4196410 con 0x7f1ce41012b0 2024-04-05T21:17:31.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.203+0000 7f1ccbfff700 1 -- 172.21.15.121:0/4232860021 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f1cd400f460 con 0x7f1ce41012b0 2024-04-05T21:17:31.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.204+0000 7f1cc9ffb700 1 -- 172.21.15.121:0/4232860021 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc40052f0 con 0x7f1ce41012b0 2024-04-05T21:17:31.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.204+0000 7f1ccbfff700 1 --2- 172.21.15.121:0/4232860021 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f1ccc038410 0x7f1ccc03a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:31.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.204+0000 7f1ccbfff700 1 -- 172.21.15.121:0/4232860021 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1cd404d3e0 con 0x7f1ce41012b0 2024-04-05T21:17:31.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.204+0000 7f1ce27fc700 1 --2- 172.21.15.121:0/4232860021 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f1ccc038410 0x7f1ccc03a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:31.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.205+0000 7f1ce27fc700 1 --2- 172.21.15.121:0/4232860021 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f1ccc038410 0x7f1ccc03a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8006fd0 tx=0x7f1cd8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:31.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.209+0000 7f1ccbfff700 1 -- 172.21.15.121:0/4232860021 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1cd402a950 con 0x7f1ce41012b0 2024-04-05T21:17:31.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:31.414+0000 7f1cc9ffb700 1 -- 172.21.15.121:0/4232860021 --> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cc4000bc0 con 0x7f1ccc038410 2024-04-05T21:17:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:31 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:31.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:31 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:31.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:31 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:31.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:31 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:31.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:31 smithi121 ceph-mon[39723]: Deploying daemon node-exporter.smithi121 on smithi121 2024-04-05T21:17:31.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:31 smithi121 ceph-mon[39723]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:32 smithi121 ceph-mon[39723]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:17:32.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:32 smithi121 ceph-mon[39723]: Deploying cephadm binary to smithi188 2024-04-05T21:17:33.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:33 smithi121 ceph-mon[39723]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:33.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:33.645+0000 7f1ccbfff700 1 -- 172.21.15.121:0/4232860021 <== mgr.14162 v2:172.21.15.121:6800/1237774071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1cc4000bc0 con 0x7f1ccc038410 2024-04-05T21:17:33.647 INFO:teuthology.orchestra.run.smithi121.stdout:Added host 'smithi188' with addr '172.21.15.188' 2024-04-05T21:17:33.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:33.647+0000 7f1cc9ffb700 1 -- 172.21.15.121:0/4232860021 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f1ccc038410 msgr2=0x7f1ccc03a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:33.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:33.647+0000 7f1cc9ffb700 1 --2- 172.21.15.121:0/4232860021 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f1ccc038410 0x7f1ccc03a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8006fd0 tx=0x7f1cd8006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:33.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:33.647+0000 7f1cc9ffb700 1 -- 172.21.15.121:0/4232860021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce41012b0 msgr2=0x7f1ce4195830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:33.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:33.647+0000 7f1cc9ffb700 1 --2- 172.21.15.121:0/4232860021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce41012b0 0x7f1ce4195830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4004f40 tx=0x7f1cd40057b0 comp rx=0 tx=0).stop 2024-04-05T21:17:33.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:33.647+0000 7f1cc9ffb700 1 -- 172.21.15.121:0/4232860021 shutdown_connections 2024-04-05T21:17:33.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:33.647+0000 7f1cc9ffb700 1 --2- 172.21.15.121:0/4232860021 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f1ccc038410 0x7f1ccc03a8d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:33.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:33.647+0000 7f1cc9ffb700 1 --2- 172.21.15.121:0/4232860021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce41012b0 0x7f1ce4195830 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:33.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:33.647+0000 7f1cc9ffb700 1 -- 172.21.15.121:0/4232860021 >> 172.21.15.121:0/4232860021 conn(0x7f1ce40facc0 msgr2=0x7f1ce40fb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:33.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:33.648+0000 7f1cc9ffb700 1 -- 172.21.15.121:0/4232860021 shutdown_connections 2024-04-05T21:17:33.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:33.648+0000 7f1cc9ffb700 1 -- 172.21.15.121:0/4232860021 wait complete. 2024-04-05T21:17:34.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:34 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:34.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:34 smithi121 ceph-mon[39723]: Added host smithi188 2024-04-05T21:17:34.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:34 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:34.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:34 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:34.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:34 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:34.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:34 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:34.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:34 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:34.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-05T21:17:35.094 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:17:35.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:35 smithi121 ceph-mon[39723]: Deploying daemon alertmanager.smithi121 on smithi121 2024-04-05T21:17:35.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:35 smithi121 ceph-mon[39723]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:37.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.363+0000 7f71cf2c5700 1 -- 172.21.15.121:0/346336027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71c81013b0 msgr2=0x7f71c81037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:37.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.363+0000 7f71cf2c5700 1 --2- 172.21.15.121:0/346336027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71c81013b0 0x7f71c81037a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f71c4009910 tx=0x7f71c4009c20 comp rx=0 tx=0).stop 2024-04-05T21:17:37.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.363+0000 7f71cf2c5700 1 -- 172.21.15.121:0/346336027 shutdown_connections 2024-04-05T21:17:37.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.363+0000 7f71cf2c5700 1 --2- 172.21.15.121:0/346336027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71c81013b0 0x7f71c81037a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:37.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.363+0000 7f71cf2c5700 1 -- 172.21.15.121:0/346336027 >> 172.21.15.121:0/346336027 conn(0x7f71c80fad20 msgr2=0x7f71c80fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:37.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.364+0000 7f71cf2c5700 1 -- 172.21.15.121:0/346336027 shutdown_connections 2024-04-05T21:17:37.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.364+0000 7f71cf2c5700 1 -- 172.21.15.121:0/346336027 wait complete. 2024-04-05T21:17:37.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.364+0000 7f71cf2c5700 1 Processor -- start 2024-04-05T21:17:37.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.364+0000 7f71cf2c5700 1 -- start start 2024-04-05T21:17:37.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.365+0000 7f71cf2c5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71c81013b0 0x7f71c81958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:37.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.365+0000 7f71cf2c5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8195df0 con 0x7f71c81013b0 2024-04-05T21:17:37.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.367+0000 7f71cd061700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71c81013b0 0x7f71c81958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:37.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.367+0000 7f71cd061700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71c81013b0 0x7f71c81958b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44002/0 (socket says 172.21.15.121:44002) 2024-04-05T21:17:37.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.367+0000 7f71cd061700 1 -- 172.21.15.121:0/3630553500 learned_addr learned my addr 172.21.15.121:0/3630553500 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:37.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.367+0000 7f71cd061700 1 -- 172.21.15.121:0/3630553500 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c40095f0 con 0x7f71c81013b0 2024-04-05T21:17:37.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.367+0000 7f71cd061700 1 --2- 172.21.15.121:0/3630553500 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71c81013b0 0x7f71c81958b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f71c4004f40 tx=0x7f71c40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:37.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.370+0000 7f71be7fc700 1 -- 172.21.15.121:0/3630553500 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71c401d070 con 0x7f71c81013b0 2024-04-05T21:17:37.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.370+0000 7f71be7fc700 1 -- 172.21.15.121:0/3630553500 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f71c400bdf0 con 0x7f71c81013b0 2024-04-05T21:17:37.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.370+0000 7f71be7fc700 1 -- 172.21.15.121:0/3630553500 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71c4022e00 con 0x7f71c81013b0 2024-04-05T21:17:37.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.370+0000 7f71cf2c5700 1 -- 172.21.15.121:0/3630553500 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c8195ff0 con 0x7f71c81013b0 2024-04-05T21:17:37.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.370+0000 7f71cf2c5700 1 -- 172.21.15.121:0/3630553500 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c8196490 con 0x7f71c81013b0 2024-04-05T21:17:37.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.371+0000 7f71cf2c5700 1 -- 172.21.15.121:0/3630553500 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c818f380 con 0x7f71c81013b0 2024-04-05T21:17:37.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.372+0000 7f71be7fc700 1 -- 172.21.15.121:0/3630553500 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f71c40226c0 con 0x7f71c81013b0 2024-04-05T21:17:37.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.372+0000 7f71be7fc700 1 --2- 172.21.15.121:0/3630553500 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f71b403c860 0x7f71b403ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:37.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.372+0000 7f71be7fc700 1 -- 172.21.15.121:0/3630553500 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f71c404c0d0 con 0x7f71c81013b0 2024-04-05T21:17:37.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.374+0000 7f71cc860700 1 --2- 172.21.15.121:0/3630553500 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f71b403c860 0x7f71b403ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:37.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.378+0000 7f71be7fc700 1 -- 172.21.15.121:0/3630553500 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f71c407f0e0 con 0x7f71c81013b0 2024-04-05T21:17:37.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.381+0000 7f71cc860700 1 --2- 172.21.15.121:0/3630553500 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f71b403c860 0x7f71b403ed20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f71b8006fd0 tx=0x7f71b8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.634+0000 7f71cf2c5700 1 -- 172.21.15.121:0/3630553500 --> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71c80611d0 con 0x7f71b403c860 2024-04-05T21:17:37.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.636+0000 7f71be7fc700 1 -- 172.21.15.121:0/3630553500 <== mgr.14162 v2:172.21.15.121:6800/1237774071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f71c80611d0 con 0x7f71b403c860 2024-04-05T21:17:37.637 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:17:37.637 INFO:teuthology.orchestra.run.smithi121.stdout:[{"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}, {"addr": "172.21.15.188", "hostname": "smithi188", "labels": [], "status": ""}] 2024-04-05T21:17:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.638+0000 7f71cf2c5700 1 -- 172.21.15.121:0/3630553500 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f71b403c860 msgr2=0x7f71b403ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.638+0000 7f71cf2c5700 1 --2- 172.21.15.121:0/3630553500 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f71b403c860 0x7f71b403ed20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f71b8006fd0 tx=0x7f71b8006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.638+0000 7f71cf2c5700 1 -- 172.21.15.121:0/3630553500 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71c81013b0 msgr2=0x7f71c81958b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.638+0000 7f71cf2c5700 1 --2- 172.21.15.121:0/3630553500 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71c81013b0 0x7f71c81958b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f71c4004f40 tx=0x7f71c40057b0 comp rx=0 tx=0).stop 2024-04-05T21:17:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.638+0000 7f71cf2c5700 1 -- 172.21.15.121:0/3630553500 shutdown_connections 2024-04-05T21:17:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.638+0000 7f71cf2c5700 1 --2- 172.21.15.121:0/3630553500 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f71b403c860 0x7f71b403ed20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.638+0000 7f71cf2c5700 1 --2- 172.21.15.121:0/3630553500 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71c81013b0 0x7f71c81958b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.638+0000 7f71cf2c5700 1 -- 172.21.15.121:0/3630553500 >> 172.21.15.121:0/3630553500 conn(0x7f71c80fad20 msgr2=0x7f71c80fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.638+0000 7f71cf2c5700 1 -- 172.21.15.121:0/3630553500 shutdown_connections 2024-04-05T21:17:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:37.638+0000 7f71cf2c5700 1 -- 172.21.15.121:0/3630553500 wait complete. 2024-04-05T21:17:37.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:37 smithi121 ceph-mon[39723]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:38.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:38 smithi121 ceph-mon[39723]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T21:17:39.023 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-05T21:17:39.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph osd crush tunables default 2024-04-05T21:17:39.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:39 smithi121 ceph-mon[39723]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:39.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:39 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:40.270 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:17:41.417 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:41 smithi121 ceph-mon[39723]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:43.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:43 smithi121 ceph-mon[39723]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:44.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.644+0000 7f420028b700 1 -- 172.21.15.121:0/150334421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41f8100890 msgr2=0x7f41f8100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:44.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.644+0000 7f420028b700 1 --2- 172.21.15.121:0/150334421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41f8100890 0x7f41f8100cb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009910 tx=0x7f41e8009c20 comp rx=0 tx=0).stop 2024-04-05T21:17:44.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.644+0000 7f420028b700 1 -- 172.21.15.121:0/150334421 shutdown_connections 2024-04-05T21:17:44.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.644+0000 7f420028b700 1 --2- 172.21.15.121:0/150334421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41f8100890 0x7f41f8100cb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.644+0000 7f420028b700 1 -- 172.21.15.121:0/150334421 >> 172.21.15.121:0/150334421 conn(0x7f41f80fbe30 msgr2=0x7f41f80fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.644+0000 7f420028b700 1 -- 172.21.15.121:0/150334421 shutdown_connections 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.644+0000 7f420028b700 1 -- 172.21.15.121:0/150334421 wait complete. 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.645+0000 7f420028b700 1 Processor -- start 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.645+0000 7f420028b700 1 -- start start 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.645+0000 7f420028b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41f8100890 0x7f41f8074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.645+0000 7f420028b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f8075070 con 0x7f41f8100890 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.645+0000 7f41fe027700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41f8100890 0x7f41f8074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.645+0000 7f41fe027700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41f8100890 0x7f41f8074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43306/0 (socket says 172.21.15.121:43306) 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.645+0000 7f41fe027700 1 -- 172.21.15.121:0/1849205578 learned_addr learned my addr 172.21.15.121:0/1849205578 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.645+0000 7f41fe027700 1 -- 172.21.15.121:0/1849205578 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e80095f0 con 0x7f41f8100890 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.646+0000 7f41fe027700 1 --2- 172.21.15.121:0/1849205578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41f8100890 0x7f41f8074b30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f41e8006010 tx=0x7f41e80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.646+0000 7f41ef7fe700 1 -- 172.21.15.121:0/1849205578 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41e801d070 con 0x7f41f8100890 2024-04-05T21:17:44.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.646+0000 7f420028b700 1 -- 172.21.15.121:0/1849205578 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f8073180 con 0x7f41f8100890 2024-04-05T21:17:44.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.646+0000 7f41ef7fe700 1 -- 172.21.15.121:0/1849205578 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f41e800bdf0 con 0x7f41f8100890 2024-04-05T21:17:44.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.646+0000 7f41ef7fe700 1 -- 172.21.15.121:0/1849205578 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41e8022e50 con 0x7f41f8100890 2024-04-05T21:17:44.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.646+0000 7f420028b700 1 -- 172.21.15.121:0/1849205578 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f8073620 con 0x7f41f8100890 2024-04-05T21:17:44.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.647+0000 7f41ef7fe700 1 -- 172.21.15.121:0/1849205578 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f41e8022700 con 0x7f41f8100890 2024-04-05T21:17:44.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.647+0000 7f420028b700 1 -- 172.21.15.121:0/1849205578 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f80623c0 con 0x7f41f8100890 2024-04-05T21:17:44.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.647+0000 7f41ef7fe700 1 --2- 172.21.15.121:0/1849205578 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f41e4038370 0x7f41e403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:44.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.647+0000 7f41ef7fe700 1 -- 172.21.15.121:0/1849205578 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f41e804c350 con 0x7f41f8100890 2024-04-05T21:17:44.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.647+0000 7f41fd826700 1 --2- 172.21.15.121:0/1849205578 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f41e4038370 0x7f41e403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:44.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.647+0000 7f41fd826700 1 --2- 172.21.15.121:0/1849205578 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f41e4038370 0x7f41e403a830 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f41f4006fd0 tx=0x7f41f4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:44.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.652+0000 7f41ef7fe700 1 -- 172.21.15.121:0/1849205578 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f41e800b960 con 0x7f41f8100890 2024-04-05T21:17:44.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:44.859+0000 7f420028b700 1 -- 172.21.15.121:0/1849205578 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f41f818ed50 con 0x7f41f8100890 2024-04-05T21:17:45.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:45.691+0000 7f41ef7fe700 1 -- 172.21.15.121:0/1849205578 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f41e8027030 con 0x7f41f8100890 2024-04-05T21:17:45.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:45.693+0000 7f420028b700 1 -- 172.21.15.121:0/1849205578 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f41e4038370 msgr2=0x7f41e403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:45.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:45.693+0000 7f420028b700 1 --2- 172.21.15.121:0/1849205578 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f41e4038370 0x7f41e403a830 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f41f4006fd0 tx=0x7f41f4006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:45.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:45.693+0000 7f420028b700 1 -- 172.21.15.121:0/1849205578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41f8100890 msgr2=0x7f41f8074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:45.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:45.693+0000 7f420028b700 1 --2- 172.21.15.121:0/1849205578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41f8100890 0x7f41f8074b30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f41e8006010 tx=0x7f41e80057b0 comp rx=0 tx=0).stop 2024-04-05T21:17:45.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:45.693+0000 7f420028b700 1 -- 172.21.15.121:0/1849205578 shutdown_connections 2024-04-05T21:17:45.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:45.693+0000 7f420028b700 1 --2- 172.21.15.121:0/1849205578 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f41e4038370 0x7f41e403a830 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:45.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:45.693+0000 7f420028b700 1 --2- 172.21.15.121:0/1849205578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41f8100890 0x7f41f8074b30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:45.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:45.693+0000 7f420028b700 1 -- 172.21.15.121:0/1849205578 >> 172.21.15.121:0/1849205578 conn(0x7f41f80fbe30 msgr2=0x7f41f80fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:45.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:45.693+0000 7f420028b700 1 -- 172.21.15.121:0/1849205578 shutdown_connections 2024-04-05T21:17:45.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:17:45.693+0000 7f420028b700 1 -- 172.21.15.121:0/1849205578 wait complete. 2024-04-05T21:17:45.695 INFO:teuthology.orchestra.run.smithi121.stderr:adjusted tunables profile to default 2024-04-05T21:17:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:45 smithi121 ceph-mon[39723]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:45 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1849205578' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-05T21:17:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:45 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:45 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:45 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:45 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:45 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:45 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:45 smithi121 ceph-mon[39723]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-05T21:17:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:45 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:45 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:45 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-05T21:17:45.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:45 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:46.464 INFO:tasks.cephadm:Adding mon.smithi121 on smithi121 2024-04-05T21:17:46.464 INFO:tasks.cephadm:Adding mon.smithi188 on smithi188 2024-04-05T21:17:46.464 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph orch apply mon '2;smithi121:172.21.15.121=smithi121;smithi188:172.21.15.188=smithi188' 2024-04-05T21:17:46.722 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:17:46.785 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:17:46.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:46 smithi121 ceph-mon[39723]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-05T21:17:46.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:46 smithi121 ceph-mon[39723]: Deploying daemon grafana.smithi121 on smithi121 2024-04-05T21:17:46.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:46 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1849205578' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-05T21:17:46.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:46 smithi121 ceph-mon[39723]: osdmap e4: 0 total, 0 up, 0 in 2024-04-05T21:17:47.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:47 smithi121 ceph-mon[39723]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:48.513 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.511+0000 7fde9be53700 1 -- 172.21.15.188:0/2915321274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde941013b0 msgr2=0x7fde941037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:48.513 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.511+0000 7fde9be53700 1 --2- 172.21.15.188:0/2915321274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde941013b0 0x7fde941037a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fde84009910 tx=0x7fde84009c20 comp rx=0 tx=0).stop 2024-04-05T21:17:48.513 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.511+0000 7fde9be53700 1 -- 172.21.15.188:0/2915321274 shutdown_connections 2024-04-05T21:17:48.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.511+0000 7fde9be53700 1 --2- 172.21.15.188:0/2915321274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde941013b0 0x7fde941037a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:48.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.511+0000 7fde9be53700 1 -- 172.21.15.188:0/2915321274 >> 172.21.15.188:0/2915321274 conn(0x7fde940fad20 msgr2=0x7fde940fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:48.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.511+0000 7fde9be53700 1 -- 172.21.15.188:0/2915321274 shutdown_connections 2024-04-05T21:17:48.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.511+0000 7fde9be53700 1 -- 172.21.15.188:0/2915321274 wait complete. 2024-04-05T21:17:48.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.511+0000 7fde9be53700 1 Processor -- start 2024-04-05T21:17:48.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.511+0000 7fde9be53700 1 -- start start 2024-04-05T21:17:48.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.511+0000 7fde9be53700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde941013b0 0x7fde941958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:48.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.511+0000 7fde9be53700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde94195df0 con 0x7fde941013b0 2024-04-05T21:17:48.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.512+0000 7fde99bef700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde941013b0 0x7fde941958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:48.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.512+0000 7fde99bef700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde941013b0 0x7fde941958b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:46402/0 (socket says 172.21.15.188:46402) 2024-04-05T21:17:48.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.512+0000 7fde99bef700 1 -- 172.21.15.188:0/496382688 learned_addr learned my addr 172.21.15.188:0/496382688 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:17:48.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.512+0000 7fde99bef700 1 -- 172.21.15.188:0/496382688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde840095f0 con 0x7fde941013b0 2024-04-05T21:17:48.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.512+0000 7fde99bef700 1 --2- 172.21.15.188:0/496382688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde941013b0 0x7fde941958b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fde84004f40 tx=0x7fde840057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:48.515 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.512+0000 7fde8affd700 1 -- 172.21.15.188:0/496382688 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde8401d070 con 0x7fde941013b0 2024-04-05T21:17:48.515 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.512+0000 7fde8affd700 1 -- 172.21.15.188:0/496382688 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fde8400bdf0 con 0x7fde941013b0 2024-04-05T21:17:48.515 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.512+0000 7fde9be53700 1 -- 172.21.15.188:0/496382688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde94195ff0 con 0x7fde941013b0 2024-04-05T21:17:48.515 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.512+0000 7fde8affd700 1 -- 172.21.15.188:0/496382688 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde84022e30 con 0x7fde941013b0 2024-04-05T21:17:48.515 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.512+0000 7fde9be53700 1 -- 172.21.15.188:0/496382688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde94196490 con 0x7fde941013b0 2024-04-05T21:17:48.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.513+0000 7fde8affd700 1 -- 172.21.15.188:0/496382688 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fde8400b960 con 0x7fde941013b0 2024-04-05T21:17:48.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.513+0000 7fde9be53700 1 -- 172.21.15.188:0/496382688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde9418f380 con 0x7fde941013b0 2024-04-05T21:17:48.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.513+0000 7fde8affd700 1 --2- 172.21.15.188:0/496382688 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fde80038460 0x7fde8003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:48.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.513+0000 7fde8affd700 1 -- 172.21.15.188:0/496382688 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fde8404c3a0 con 0x7fde941013b0 2024-04-05T21:17:48.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.514+0000 7fde993ee700 1 --2- 172.21.15.188:0/496382688 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fde80038460 0x7fde8003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:48.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.514+0000 7fde993ee700 1 --2- 172.21.15.188:0/496382688 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fde80038460 0x7fde8003a920 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fde90006fd0 tx=0x7fde90006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:48.523 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.518+0000 7fde8affd700 1 -- 172.21.15.188:0/496382688 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fde8402bd40 con 0x7fde941013b0 2024-04-05T21:17:48.719 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.717+0000 7fde9be53700 1 -- 172.21.15.188:0/496382688 --> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi121:172.21.15.121=smithi121;smithi188:172.21.15.188=smithi188", "target": ["mon-mgr", ""]}) v1 -- 0x7fde94061340 con 0x7fde80038460 2024-04-05T21:17:48.724 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.723+0000 7fde8affd700 1 -- 172.21.15.188:0/496382688 <== mgr.14162 v2:172.21.15.121:6800/1237774071 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fde94061340 con 0x7fde80038460 2024-04-05T21:17:48.724 INFO:teuthology.orchestra.run.smithi188.stdout:Scheduled mon update... 2024-04-05T21:17:48.726 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.725+0000 7fde9be53700 1 -- 172.21.15.188:0/496382688 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fde80038460 msgr2=0x7fde8003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:48.726 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.725+0000 7fde9be53700 1 --2- 172.21.15.188:0/496382688 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fde80038460 0x7fde8003a920 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fde90006fd0 tx=0x7fde90006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:48.726 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.725+0000 7fde9be53700 1 -- 172.21.15.188:0/496382688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde941013b0 msgr2=0x7fde941958b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:48.726 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.725+0000 7fde9be53700 1 --2- 172.21.15.188:0/496382688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde941013b0 0x7fde941958b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fde84004f40 tx=0x7fde840057b0 comp rx=0 tx=0).stop 2024-04-05T21:17:48.727 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.725+0000 7fde9be53700 1 -- 172.21.15.188:0/496382688 shutdown_connections 2024-04-05T21:17:48.727 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.725+0000 7fde9be53700 1 --2- 172.21.15.188:0/496382688 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fde80038460 0x7fde8003a920 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:48.727 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.725+0000 7fde9be53700 1 --2- 172.21.15.188:0/496382688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde941013b0 0x7fde941958b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:48.727 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.725+0000 7fde9be53700 1 -- 172.21.15.188:0/496382688 >> 172.21.15.188:0/496382688 conn(0x7fde940fad20 msgr2=0x7fde940fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:48.727 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.725+0000 7fde9be53700 1 -- 172.21.15.188:0/496382688 shutdown_connections 2024-04-05T21:17:48.727 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:48.725+0000 7fde9be53700 1 -- 172.21.15.188:0/496382688 wait complete. 2024-04-05T21:17:49.379 DEBUG:teuthology.orchestra.run.smithi188:mon.smithi188> sudo journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mon.smithi188.service 2024-04-05T21:17:49.381 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:17:49.381 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:17:49.412 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:-- Logs begin at Fri 2024-04-05 21:05:37 UTC. -- 2024-04-05T21:17:49.636 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:17:49.702 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:17:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:49 smithi121 ceph-mon[39723]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:49 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:49 smithi121 ceph-mon[39723]: from='client.14195 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi121:172.21.15.121=smithi121;smithi188:172.21.15.188=smithi188", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:17:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:49 smithi121 ceph-mon[39723]: Saving service mon spec with placement smithi121:172.21.15.121=smithi121;smithi188:172.21.15.188=smithi188;count:2 2024-04-05T21:17:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:49 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:17:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:51 smithi121 ceph-mon[39723]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:51.547 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.546+0000 7f31348d5700 1 -- 172.21.15.188:0/203434674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31300ff7a0 msgr2=0x7f31300ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:51.548 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.546+0000 7f31348d5700 1 --2- 172.21.15.188:0/203434674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31300ff7a0 0x7f31300ffbc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3120009910 tx=0x7f3120009c20 comp rx=0 tx=0).stop 2024-04-05T21:17:51.548 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.546+0000 7f31348d5700 1 -- 172.21.15.188:0/203434674 shutdown_connections 2024-04-05T21:17:51.548 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.546+0000 7f31348d5700 1 --2- 172.21.15.188:0/203434674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31300ff7a0 0x7f31300ffbc0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:51.548 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.546+0000 7f31348d5700 1 -- 172.21.15.188:0/203434674 >> 172.21.15.188:0/203434674 conn(0x7f31300fad20 msgr2=0x7f31300fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:51.548 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.546+0000 7f31348d5700 1 -- 172.21.15.188:0/203434674 shutdown_connections 2024-04-05T21:17:51.548 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.546+0000 7f31348d5700 1 -- 172.21.15.188:0/203434674 wait complete. 2024-04-05T21:17:51.549 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.547+0000 7f31348d5700 1 Processor -- start 2024-04-05T21:17:51.549 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.547+0000 7f31348d5700 1 -- start start 2024-04-05T21:17:51.549 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.547+0000 7f31348d5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31300ff7a0 0x7f3130068b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:51.550 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.547+0000 7f31348d5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3130198650 con 0x7f31300ff7a0 2024-04-05T21:17:51.550 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.547+0000 7f312e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31300ff7a0 0x7f3130068b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:51.550 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.547+0000 7f312e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31300ff7a0 0x7f3130068b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:46412/0 (socket says 172.21.15.188:46412) 2024-04-05T21:17:51.550 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.547+0000 7f312e59c700 1 -- 172.21.15.188:0/2003930549 learned_addr learned my addr 172.21.15.188:0/2003930549 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:17:51.550 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.548+0000 7f312e59c700 1 -- 172.21.15.188:0/2003930549 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31200095f0 con 0x7f31300ff7a0 2024-04-05T21:17:51.550 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.548+0000 7f312e59c700 1 --2- 172.21.15.188:0/2003930549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31300ff7a0 0x7f3130068b00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3120006010 tx=0x7f31200057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:51.550 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.548+0000 7f311f7fe700 1 -- 172.21.15.188:0/2003930549 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f312001d070 con 0x7f31300ff7a0 2024-04-05T21:17:51.550 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.548+0000 7f311f7fe700 1 -- 172.21.15.188:0/2003930549 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f312000bdf0 con 0x7f31300ff7a0 2024-04-05T21:17:51.550 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.548+0000 7f311f7fe700 1 -- 172.21.15.188:0/2003930549 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3120022e50 con 0x7f31300ff7a0 2024-04-05T21:17:51.550 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.548+0000 7f31348d5700 1 -- 172.21.15.188:0/2003930549 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3130198790 con 0x7f31300ff7a0 2024-04-05T21:17:51.550 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.548+0000 7f31348d5700 1 -- 172.21.15.188:0/2003930549 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3130198c30 con 0x7f31300ff7a0 2024-04-05T21:17:51.551 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.549+0000 7f311f7fe700 1 -- 172.21.15.188:0/2003930549 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f3120022700 con 0x7f31300ff7a0 2024-04-05T21:17:51.551 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.549+0000 7f31348d5700 1 -- 172.21.15.188:0/2003930549 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313018e280 con 0x7f31300ff7a0 2024-04-05T21:17:51.551 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.549+0000 7f311f7fe700 1 --2- 172.21.15.188:0/2003930549 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f3118038370 0x7f311803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:51.551 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.549+0000 7f311f7fe700 1 -- 172.21.15.188:0/2003930549 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f312004c410 con 0x7f31300ff7a0 2024-04-05T21:17:51.551 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.549+0000 7f312dd9b700 1 --2- 172.21.15.188:0/2003930549 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f3118038370 0x7f311803a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:51.551 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.550+0000 7f312dd9b700 1 --2- 172.21.15.188:0/2003930549 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f3118038370 0x7f311803a830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3124006fd0 tx=0x7f3124006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:51.555 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.554+0000 7f311f7fe700 1 -- 172.21.15.188:0/2003930549 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f312000b960 con 0x7f31300ff7a0 2024-04-05T21:17:51.816 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.814+0000 7f31348d5700 1 -- 172.21.15.188:0/2003930549 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f313002cc70 con 0x7f31300ff7a0 2024-04-05T21:17:51.816 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.815+0000 7f311f7fe700 1 -- 172.21.15.188:0/2003930549 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3120027020 con 0x7f31300ff7a0 2024-04-05T21:17:51.817 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:17:51.817 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:17:51.819 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.817+0000 7f31348d5700 1 -- 172.21.15.188:0/2003930549 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f3118038370 msgr2=0x7f311803a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:51.819 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.817+0000 7f31348d5700 1 --2- 172.21.15.188:0/2003930549 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f3118038370 0x7f311803a830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3124006fd0 tx=0x7f3124006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:51.819 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.817+0000 7f31348d5700 1 -- 172.21.15.188:0/2003930549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31300ff7a0 msgr2=0x7f3130068b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:51.819 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.817+0000 7f31348d5700 1 --2- 172.21.15.188:0/2003930549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31300ff7a0 0x7f3130068b00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3120006010 tx=0x7f31200057b0 comp rx=0 tx=0).stop 2024-04-05T21:17:51.819 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.817+0000 7f31348d5700 1 -- 172.21.15.188:0/2003930549 shutdown_connections 2024-04-05T21:17:51.819 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.817+0000 7f31348d5700 1 --2- 172.21.15.188:0/2003930549 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f3118038370 0x7f311803a830 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:51.819 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.817+0000 7f31348d5700 1 --2- 172.21.15.188:0/2003930549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31300ff7a0 0x7f3130068b00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:51.819 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.817+0000 7f31348d5700 1 -- 172.21.15.188:0/2003930549 >> 172.21.15.188:0/2003930549 conn(0x7f31300fad20 msgr2=0x7f31300fba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:51.819 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.817+0000 7f31348d5700 1 -- 172.21.15.188:0/2003930549 shutdown_connections 2024-04-05T21:17:51.819 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:51.817+0000 7f31348d5700 1 -- 172.21.15.188:0/2003930549 wait complete. 2024-04-05T21:17:51.820 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:17:52.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:52 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/2003930549' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:17:53.415 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:17:53.416 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:17:53.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:53 smithi121 ceph-mon[39723]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:53.665 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:17:53.727 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:17:55.563 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.538+0000 7f7cd0220700 1 -- 172.21.15.188:0/3350325482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc8100890 msgr2=0x7f7cc8100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:55.563 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.538+0000 7f7cd0220700 1 --2- 172.21.15.188:0/3350325482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc8100890 0x7f7cc8100cb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009910 tx=0x7f7cc4009c20 comp rx=0 tx=0).stop 2024-04-05T21:17:55.563 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.538+0000 7f7cd0220700 1 -- 172.21.15.188:0/3350325482 shutdown_connections 2024-04-05T21:17:55.563 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.538+0000 7f7cd0220700 1 --2- 172.21.15.188:0/3350325482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc8100890 0x7f7cc8100cb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:55.563 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.538+0000 7f7cd0220700 1 -- 172.21.15.188:0/3350325482 >> 172.21.15.188:0/3350325482 conn(0x7f7cc80fbe30 msgr2=0x7f7cc80fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:55.563 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.538+0000 7f7cd0220700 1 -- 172.21.15.188:0/3350325482 shutdown_connections 2024-04-05T21:17:55.563 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.538+0000 7f7cd0220700 1 -- 172.21.15.188:0/3350325482 wait complete. 2024-04-05T21:17:55.563 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.539+0000 7f7cd0220700 1 Processor -- start 2024-04-05T21:17:55.563 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.539+0000 7f7cd0220700 1 -- start start 2024-04-05T21:17:55.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.539+0000 7f7cd0220700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc8100890 0x7f7cc8074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:55.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.539+0000 7f7cd0220700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc8075070 con 0x7f7cc8100890 2024-04-05T21:17:55.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.539+0000 7f7ccdfbc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc8100890 0x7f7cc8074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:55.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.539+0000 7f7ccdfbc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc8100890 0x7f7cc8074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:46438/0 (socket says 172.21.15.188:46438) 2024-04-05T21:17:55.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.539+0000 7f7ccdfbc700 1 -- 172.21.15.188:0/2326857443 learned_addr learned my addr 172.21.15.188:0/2326857443 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:17:55.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.539+0000 7f7ccdfbc700 1 -- 172.21.15.188:0/2326857443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc40095f0 con 0x7f7cc8100890 2024-04-05T21:17:55.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.539+0000 7f7ccdfbc700 1 --2- 172.21.15.188:0/2326857443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc8100890 0x7f7cc8074b30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4006010 tx=0x7f7cc40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:55.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.540+0000 7f7cbeffd700 1 -- 172.21.15.188:0/2326857443 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cc401d070 con 0x7f7cc8100890 2024-04-05T21:17:55.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.540+0000 7f7cd0220700 1 -- 172.21.15.188:0/2326857443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc8073180 con 0x7f7cc8100890 2024-04-05T21:17:55.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.540+0000 7f7cbeffd700 1 -- 172.21.15.188:0/2326857443 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f7cc400bdf0 con 0x7f7cc8100890 2024-04-05T21:17:55.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.540+0000 7f7cbeffd700 1 -- 172.21.15.188:0/2326857443 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cc4022e50 con 0x7f7cc8100890 2024-04-05T21:17:55.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.540+0000 7f7cd0220700 1 -- 172.21.15.188:0/2326857443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc8073620 con 0x7f7cc8100890 2024-04-05T21:17:55.564 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.541+0000 7f7cd0220700 1 -- 172.21.15.188:0/2326857443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc80623c0 con 0x7f7cc8100890 2024-04-05T21:17:55.565 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.541+0000 7f7cbeffd700 1 -- 172.21.15.188:0/2326857443 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f7cc4022700 con 0x7f7cc8100890 2024-04-05T21:17:55.565 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.541+0000 7f7cbeffd700 1 --2- 172.21.15.188:0/2326857443 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f7cb4038370 0x7f7cb403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:55.565 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.541+0000 7f7cbeffd700 1 -- 172.21.15.188:0/2326857443 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7cc404c300 con 0x7f7cc8100890 2024-04-05T21:17:55.565 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.541+0000 7f7ccd7bb700 1 --2- 172.21.15.188:0/2326857443 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f7cb4038370 0x7f7cb403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:55.565 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.541+0000 7f7ccd7bb700 1 --2- 172.21.15.188:0/2326857443 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f7cb4038370 0x7f7cb403a830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8006fd0 tx=0x7f7cb8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:55.565 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.546+0000 7f7cbeffd700 1 -- 172.21.15.188:0/2326857443 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7cc400b960 con 0x7f7cc8100890 2024-04-05T21:17:55.810 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.809+0000 7f7cd0220700 1 -- 172.21.15.188:0/2326857443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7cc818ed50 con 0x7f7cc8100890 2024-04-05T21:17:55.811 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.810+0000 7f7cbeffd700 1 -- 172.21.15.188:0/2326857443 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f7cc4027020 con 0x7f7cc8100890 2024-04-05T21:17:55.811 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:17:55.811 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:17:55.813 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.812+0000 7f7cd0220700 1 -- 172.21.15.188:0/2326857443 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f7cb4038370 msgr2=0x7f7cb403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:55.813 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.812+0000 7f7cd0220700 1 --2- 172.21.15.188:0/2326857443 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f7cb4038370 0x7f7cb403a830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8006fd0 tx=0x7f7cb8006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:55.813 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.812+0000 7f7cd0220700 1 -- 172.21.15.188:0/2326857443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc8100890 msgr2=0x7f7cc8074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:55.813 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.812+0000 7f7cd0220700 1 --2- 172.21.15.188:0/2326857443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc8100890 0x7f7cc8074b30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4006010 tx=0x7f7cc40057b0 comp rx=0 tx=0).stop 2024-04-05T21:17:55.813 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.812+0000 7f7cd0220700 1 -- 172.21.15.188:0/2326857443 shutdown_connections 2024-04-05T21:17:55.813 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.812+0000 7f7cd0220700 1 --2- 172.21.15.188:0/2326857443 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f7cb4038370 0x7f7cb403a830 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:55.813 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.812+0000 7f7cd0220700 1 --2- 172.21.15.188:0/2326857443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc8100890 0x7f7cc8074b30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:55.813 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.812+0000 7f7cd0220700 1 -- 172.21.15.188:0/2326857443 >> 172.21.15.188:0/2326857443 conn(0x7f7cc80fbe30 msgr2=0x7f7cc80fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:55.814 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.812+0000 7f7cd0220700 1 -- 172.21.15.188:0/2326857443 shutdown_connections 2024-04-05T21:17:55.814 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:55.812+0000 7f7cd0220700 1 -- 172.21.15.188:0/2326857443 wait complete. 2024-04-05T21:17:55.814 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:17:55.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:55 smithi121 ceph-mon[39723]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:56.885 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:56 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/2326857443' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:17:57.470 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:17:57.471 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:17:57.723 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:17:57.785 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:17:57.905 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:57 smithi121 ceph-mon[39723]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:17:59.431 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.428+0000 7f79178d9700 1 -- 172.21.15.188:0/3801568400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79100ff7a0 msgr2=0x7f79100ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:59.431 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.428+0000 7f79178d9700 1 --2- 172.21.15.188:0/3801568400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79100ff7a0 0x7f79100ffbc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7900009910 tx=0x7f7900009c20 comp rx=0 tx=0).stop 2024-04-05T21:17:59.431 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.428+0000 7f79178d9700 1 -- 172.21.15.188:0/3801568400 shutdown_connections 2024-04-05T21:17:59.431 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.428+0000 7f79178d9700 1 --2- 172.21.15.188:0/3801568400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79100ff7a0 0x7f79100ffbc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:59.431 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.428+0000 7f79178d9700 1 -- 172.21.15.188:0/3801568400 >> 172.21.15.188:0/3801568400 conn(0x7f79100fad20 msgr2=0x7f79100fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.428+0000 7f79178d9700 1 -- 172.21.15.188:0/3801568400 shutdown_connections 2024-04-05T21:17:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.428+0000 7f79178d9700 1 -- 172.21.15.188:0/3801568400 wait complete. 2024-04-05T21:17:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.429+0000 7f79178d9700 1 Processor -- start 2024-04-05T21:17:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.429+0000 7f79178d9700 1 -- start start 2024-04-05T21:17:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.429+0000 7f79178d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79100ff7a0 0x7f79100fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.429+0000 7f79178d9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79100fd220 con 0x7f79100ff7a0 2024-04-05T21:17:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.429+0000 7f7915675700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79100ff7a0 0x7f79100fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.429+0000 7f7915675700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79100ff7a0 0x7f79100fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:37412/0 (socket says 172.21.15.188:37412) 2024-04-05T21:17:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.429+0000 7f7915675700 1 -- 172.21.15.188:0/3567385005 learned_addr learned my addr 172.21.15.188:0/3567385005 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:17:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.429+0000 7f7915675700 1 -- 172.21.15.188:0/3567385005 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79000095f0 con 0x7f79100ff7a0 2024-04-05T21:17:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.430+0000 7f7915675700 1 --2- 172.21.15.188:0/3567385005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79100ff7a0 0x7f79100fcce0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7900006010 tx=0x7f79000057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.430+0000 7f79067fc700 1 -- 172.21.15.188:0/3567385005 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f790001d070 con 0x7f79100ff7a0 2024-04-05T21:17:59.432 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.430+0000 7f79178d9700 1 -- 172.21.15.188:0/3567385005 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79100fb330 con 0x7f79100ff7a0 2024-04-05T21:17:59.434 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.430+0000 7f79067fc700 1 -- 172.21.15.188:0/3567385005 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f790000bdf0 con 0x7f79100ff7a0 2024-04-05T21:17:59.434 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.430+0000 7f79067fc700 1 -- 172.21.15.188:0/3567385005 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7900022e50 con 0x7f79100ff7a0 2024-04-05T21:17:59.434 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.430+0000 7f79178d9700 1 -- 172.21.15.188:0/3567385005 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79100fb7d0 con 0x7f79100ff7a0 2024-04-05T21:17:59.434 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.431+0000 7f79067fc700 1 -- 172.21.15.188:0/3567385005 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f790000b960 con 0x7f79100ff7a0 2024-04-05T21:17:59.434 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.431+0000 7f79178d9700 1 -- 172.21.15.188:0/3567385005 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79100623c0 con 0x7f79100ff7a0 2024-04-05T21:17:59.434 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.431+0000 7f79067fc700 1 --2- 172.21.15.188:0/3567385005 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f78fc0383c0 0x7f78fc03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:17:59.435 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.431+0000 7f79067fc700 1 -- 172.21.15.188:0/3567385005 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f790004c490 con 0x7f79100ff7a0 2024-04-05T21:17:59.435 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.431+0000 7f7914e74700 1 --2- 172.21.15.188:0/3567385005 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f78fc0383c0 0x7f78fc03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:17:59.435 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.431+0000 7f7914e74700 1 --2- 172.21.15.188:0/3567385005 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f78fc0383c0 0x7f78fc03a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f790c006fd0 tx=0x7f790c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:17:59.438 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.436+0000 7f79067fc700 1 -- 172.21.15.188:0/3567385005 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f790000f460 con 0x7f79100ff7a0 2024-04-05T21:17:59.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.697+0000 7f79178d9700 1 -- 172.21.15.188:0/3567385005 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f791018ec70 con 0x7f79100ff7a0 2024-04-05T21:17:59.699 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.698+0000 7f79067fc700 1 -- 172.21.15.188:0/3567385005 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f7900027030 con 0x7f79100ff7a0 2024-04-05T21:17:59.699 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:17:59.700 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:17:59.702 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.700+0000 7f79178d9700 1 -- 172.21.15.188:0/3567385005 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f78fc0383c0 msgr2=0x7f78fc03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:59.702 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.700+0000 7f79178d9700 1 --2- 172.21.15.188:0/3567385005 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f78fc0383c0 0x7f78fc03a880 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f790c006fd0 tx=0x7f790c006e40 comp rx=0 tx=0).stop 2024-04-05T21:17:59.702 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.700+0000 7f79178d9700 1 -- 172.21.15.188:0/3567385005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79100ff7a0 msgr2=0x7f79100fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:17:59.702 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.700+0000 7f79178d9700 1 --2- 172.21.15.188:0/3567385005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79100ff7a0 0x7f79100fcce0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7900006010 tx=0x7f79000057b0 comp rx=0 tx=0).stop 2024-04-05T21:17:59.702 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.700+0000 7f79178d9700 1 -- 172.21.15.188:0/3567385005 shutdown_connections 2024-04-05T21:17:59.702 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.700+0000 7f79178d9700 1 --2- 172.21.15.188:0/3567385005 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f78fc0383c0 0x7f78fc03a880 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:59.702 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.700+0000 7f79178d9700 1 --2- 172.21.15.188:0/3567385005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79100ff7a0 0x7f79100fcce0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:17:59.702 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.700+0000 7f79178d9700 1 -- 172.21.15.188:0/3567385005 >> 172.21.15.188:0/3567385005 conn(0x7f79100fad20 msgr2=0x7f79100690a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:17:59.703 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.700+0000 7f79178d9700 1 -- 172.21.15.188:0/3567385005 shutdown_connections 2024-04-05T21:17:59.703 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:17:59.700+0000 7f79178d9700 1 -- 172.21.15.188:0/3567385005 wait complete. 2024-04-05T21:17:59.703 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:18:00.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:59 smithi121 ceph-mon[39723]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:00.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:59 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:00.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:59 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:00.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:59 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:00.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:59 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:00.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:59 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:00.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:59 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:00.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:59 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:00.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:59 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:00.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:17:59 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/3567385005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:18:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:00 smithi121 ceph-mon[39723]: Deploying daemon prometheus.smithi121 on smithi121 2024-04-05T21:18:01.387 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:18:01.387 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:18:01.633 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:01.697 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:02.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:01 smithi121 ceph-mon[39723]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:03.591 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.589+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/2495891335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f941013b0 msgr2=0x7f6f941037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:03.591 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.589+0000 7f6f9a0a0700 1 --2- 172.21.15.188:0/2495891335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f941013b0 0x7f6f941037a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009910 tx=0x7f6f84009c20 comp rx=0 tx=0).stop 2024-04-05T21:18:03.592 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.589+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/2495891335 shutdown_connections 2024-04-05T21:18:03.592 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.589+0000 7f6f9a0a0700 1 --2- 172.21.15.188:0/2495891335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f941013b0 0x7f6f941037a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:03.592 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.589+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/2495891335 >> 172.21.15.188:0/2495891335 conn(0x7f6f940fad20 msgr2=0x7f6f940fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:03.592 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.589+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/2495891335 shutdown_connections 2024-04-05T21:18:03.592 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.589+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/2495891335 wait complete. 2024-04-05T21:18:03.592 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.590+0000 7f6f9a0a0700 1 Processor -- start 2024-04-05T21:18:03.592 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.590+0000 7f6f9a0a0700 1 -- start start 2024-04-05T21:18:03.592 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.590+0000 7f6f9a0a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f941013b0 0x7f6f941958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:03.592 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.590+0000 7f6f9a0a0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f94195df0 con 0x7f6f941013b0 2024-04-05T21:18:03.593 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.590+0000 7f6f937fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f941013b0 0x7f6f941958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:03.593 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.590+0000 7f6f937fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f941013b0 0x7f6f941958b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:37432/0 (socket says 172.21.15.188:37432) 2024-04-05T21:18:03.593 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.590+0000 7f6f937fe700 1 -- 172.21.15.188:0/3086819669 learned_addr learned my addr 172.21.15.188:0/3086819669 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:18:03.595 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.591+0000 7f6f937fe700 1 -- 172.21.15.188:0/3086819669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f840095f0 con 0x7f6f941013b0 2024-04-05T21:18:03.595 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.591+0000 7f6f937fe700 1 --2- 172.21.15.188:0/3086819669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f941013b0 0x7f6f941958b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6f84004f40 tx=0x7f6f840057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:03.595 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.591+0000 7f6f90ff9700 1 -- 172.21.15.188:0/3086819669 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f8401d070 con 0x7f6f941013b0 2024-04-05T21:18:03.595 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.591+0000 7f6f90ff9700 1 -- 172.21.15.188:0/3086819669 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f6f8400bdf0 con 0x7f6f941013b0 2024-04-05T21:18:03.595 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.591+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/3086819669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f94195ff0 con 0x7f6f941013b0 2024-04-05T21:18:03.596 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.591+0000 7f6f90ff9700 1 -- 172.21.15.188:0/3086819669 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f84022e30 con 0x7f6f941013b0 2024-04-05T21:18:03.596 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.591+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/3086819669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f94196490 con 0x7f6f941013b0 2024-04-05T21:18:03.596 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.592+0000 7f6f90ff9700 1 -- 172.21.15.188:0/3086819669 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f6f8400f4f0 con 0x7f6f941013b0 2024-04-05T21:18:03.596 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.592+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/3086819669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9418f380 con 0x7f6f941013b0 2024-04-05T21:18:03.596 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.592+0000 7f6f90ff9700 1 --2- 172.21.15.188:0/3086819669 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f6f7c0383c0 0x7f6f7c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:03.596 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.592+0000 7f6f90ff9700 1 -- 172.21.15.188:0/3086819669 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6f8404c540 con 0x7f6f941013b0 2024-04-05T21:18:03.596 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.592+0000 7f6f92ffd700 1 --2- 172.21.15.188:0/3086819669 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f6f7c0383c0 0x7f6f7c03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:03.596 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.593+0000 7f6f92ffd700 1 --2- 172.21.15.188:0/3086819669 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f6f7c0383c0 0x7f6f7c03a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6f88006fd0 tx=0x7f6f88006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:03.598 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.597+0000 7f6f90ff9700 1 -- 172.21.15.188:0/3086819669 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6f8400bb40 con 0x7f6f941013b0 2024-04-05T21:18:03.866 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.864+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/3086819669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f9402cc70 con 0x7f6f941013b0 2024-04-05T21:18:03.866 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.864+0000 7f6f90ff9700 1 -- 172.21.15.188:0/3086819669 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f6f84027030 con 0x7f6f941013b0 2024-04-05T21:18:03.866 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:18:03.866 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:18:03.868 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.866+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/3086819669 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f6f7c0383c0 msgr2=0x7f6f7c03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:03.868 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.867+0000 7f6f9a0a0700 1 --2- 172.21.15.188:0/3086819669 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f6f7c0383c0 0x7f6f7c03a880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6f88006fd0 tx=0x7f6f88006e40 comp rx=0 tx=0).stop 2024-04-05T21:18:03.868 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.867+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/3086819669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f941013b0 msgr2=0x7f6f941958b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:03.868 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.867+0000 7f6f9a0a0700 1 --2- 172.21.15.188:0/3086819669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f941013b0 0x7f6f941958b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6f84004f40 tx=0x7f6f840057b0 comp rx=0 tx=0).stop 2024-04-05T21:18:03.868 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.867+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/3086819669 shutdown_connections 2024-04-05T21:18:03.868 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.867+0000 7f6f9a0a0700 1 --2- 172.21.15.188:0/3086819669 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f6f7c0383c0 0x7f6f7c03a880 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:03.868 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.867+0000 7f6f9a0a0700 1 --2- 172.21.15.188:0/3086819669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f941013b0 0x7f6f941958b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:03.869 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.867+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/3086819669 >> 172.21.15.188:0/3086819669 conn(0x7f6f940fad20 msgr2=0x7f6f940fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:03.869 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.867+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/3086819669 shutdown_connections 2024-04-05T21:18:03.869 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:03.867+0000 7f6f9a0a0700 1 -- 172.21.15.188:0/3086819669 wait complete. 2024-04-05T21:18:03.869 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:18:04.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:03 smithi121 ceph-mon[39723]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:04.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:03 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:04.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:03 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/3086819669' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:18:05.458 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:18:05.458 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:18:05.715 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:05.772 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:06.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:05 smithi121 ceph-mon[39723]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:07.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.511+0000 7fea09944700 1 -- 172.21.15.188:0/4173138377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea040ff7a0 msgr2=0x7fea040ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:07.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.511+0000 7fea09944700 1 --2- 172.21.15.188:0/4173138377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea040ff7a0 0x7fea040ffbc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009910 tx=0x7fe9f8009c20 comp rx=0 tx=0).stop 2024-04-05T21:18:07.512 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.511+0000 7fea09944700 1 -- 172.21.15.188:0/4173138377 shutdown_connections 2024-04-05T21:18:07.513 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.511+0000 7fea09944700 1 --2- 172.21.15.188:0/4173138377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea040ff7a0 0x7fea040ffbc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:07.513 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.511+0000 7fea09944700 1 -- 172.21.15.188:0/4173138377 >> 172.21.15.188:0/4173138377 conn(0x7fea040fad20 msgr2=0x7fea040fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:07.513 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.511+0000 7fea09944700 1 -- 172.21.15.188:0/4173138377 shutdown_connections 2024-04-05T21:18:07.513 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.511+0000 7fea09944700 1 -- 172.21.15.188:0/4173138377 wait complete. 2024-04-05T21:18:07.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.512+0000 7fea09944700 1 Processor -- start 2024-04-05T21:18:07.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.512+0000 7fea09944700 1 -- start start 2024-04-05T21:18:07.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.512+0000 7fea09944700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea040ff7a0 0x7fea041947b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:07.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.512+0000 7fea09944700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea04194cf0 con 0x7fea040ff7a0 2024-04-05T21:18:07.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.512+0000 7fea02ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea040ff7a0 0x7fea041947b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:07.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.512+0000 7fea02ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea040ff7a0 0x7fea041947b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:49926/0 (socket says 172.21.15.188:49926) 2024-04-05T21:18:07.515 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.512+0000 7fea02ffd700 1 -- 172.21.15.188:0/3925294929 learned_addr learned my addr 172.21.15.188:0/3925294929 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:18:07.515 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.513+0000 7fea02ffd700 1 -- 172.21.15.188:0/3925294929 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f80095f0 con 0x7fea040ff7a0 2024-04-05T21:18:07.515 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.513+0000 7fea02ffd700 1 --2- 172.21.15.188:0/3925294929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea040ff7a0 0x7fea041947b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8006010 tx=0x7fe9f80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:07.515 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.513+0000 7fea08942700 1 -- 172.21.15.188:0/3925294929 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9f801d070 con 0x7fea040ff7a0 2024-04-05T21:18:07.515 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.513+0000 7fea08942700 1 -- 172.21.15.188:0/3925294929 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fe9f800bdf0 con 0x7fea040ff7a0 2024-04-05T21:18:07.515 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.513+0000 7fea09944700 1 -- 172.21.15.188:0/3925294929 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea04194ef0 con 0x7fea040ff7a0 2024-04-05T21:18:07.515 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.513+0000 7fea08942700 1 -- 172.21.15.188:0/3925294929 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9f8022e50 con 0x7fea040ff7a0 2024-04-05T21:18:07.515 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.513+0000 7fea09944700 1 -- 172.21.15.188:0/3925294929 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea04195390 con 0x7fea040ff7a0 2024-04-05T21:18:07.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.514+0000 7fea09944700 1 -- 172.21.15.188:0/3925294929 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea0418e280 con 0x7fea040ff7a0 2024-04-05T21:18:07.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.514+0000 7fea08942700 1 -- 172.21.15.188:0/3925294929 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7fe9f8022700 con 0x7fea040ff7a0 2024-04-05T21:18:07.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.514+0000 7fea08942700 1 --2- 172.21.15.188:0/3925294929 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fe9ec038370 0x7fe9ec03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:07.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.514+0000 7fea08942700 1 -- 172.21.15.188:0/3925294929 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe9f804c350 con 0x7fea040ff7a0 2024-04-05T21:18:07.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.514+0000 7fea027fc700 1 --2- 172.21.15.188:0/3925294929 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fe9ec038370 0x7fe9ec03a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:07.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.515+0000 7fea027fc700 1 --2- 172.21.15.188:0/3925294929 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fe9ec038370 0x7fe9ec03a830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4006fd0 tx=0x7fe9f4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:07.520 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.519+0000 7fea08942700 1 -- 172.21.15.188:0/3925294929 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe9f800b960 con 0x7fea040ff7a0 2024-04-05T21:18:07.786 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.784+0000 7fea09944700 1 -- 172.21.15.188:0/3925294929 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea0402cc70 con 0x7fea040ff7a0 2024-04-05T21:18:07.786 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.785+0000 7fea08942700 1 -- 172.21.15.188:0/3925294929 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fe9f8027020 con 0x7fea040ff7a0 2024-04-05T21:18:07.786 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:18:07.786 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:18:07.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.787+0000 7fea09944700 1 -- 172.21.15.188:0/3925294929 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fe9ec038370 msgr2=0x7fe9ec03a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:07.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.787+0000 7fea09944700 1 --2- 172.21.15.188:0/3925294929 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fe9ec038370 0x7fe9ec03a830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4006fd0 tx=0x7fe9f4006e40 comp rx=0 tx=0).stop 2024-04-05T21:18:07.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.787+0000 7fea09944700 1 -- 172.21.15.188:0/3925294929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea040ff7a0 msgr2=0x7fea041947b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:07.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.787+0000 7fea09944700 1 --2- 172.21.15.188:0/3925294929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea040ff7a0 0x7fea041947b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8006010 tx=0x7fe9f80057b0 comp rx=0 tx=0).stop 2024-04-05T21:18:07.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.787+0000 7fea09944700 1 -- 172.21.15.188:0/3925294929 shutdown_connections 2024-04-05T21:18:07.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.787+0000 7fea09944700 1 --2- 172.21.15.188:0/3925294929 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fe9ec038370 0x7fe9ec03a830 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:07.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.787+0000 7fea09944700 1 --2- 172.21.15.188:0/3925294929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea040ff7a0 0x7fea041947b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:07.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.787+0000 7fea09944700 1 -- 172.21.15.188:0/3925294929 >> 172.21.15.188:0/3925294929 conn(0x7fea040fad20 msgr2=0x7fea040fba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:07.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.787+0000 7fea09944700 1 -- 172.21.15.188:0/3925294929 shutdown_connections 2024-04-05T21:18:07.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:07.787+0000 7fea09944700 1 -- 172.21.15.188:0/3925294929 wait complete. 2024-04-05T21:18:07.790 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:18:08.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:07 smithi121 ceph-mon[39723]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:08.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:07 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/3925294929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:18:09.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:18:09.545 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:18:09.793 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:09.860 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:10.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:09 smithi121 ceph-mon[39723]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:11.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:11 smithi121 ceph-mon[39723]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:11.513 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.511+0000 7f222073e700 1 -- 172.21.15.188:0/3455128627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22180ff7a0 msgr2=0x7f22180ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:11.513 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.511+0000 7f222073e700 1 --2- 172.21.15.188:0/3455128627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22180ff7a0 0x7f22180ffbc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2208009910 tx=0x7f2208009c20 comp rx=0 tx=0).stop 2024-04-05T21:18:11.513 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.511+0000 7f222073e700 1 -- 172.21.15.188:0/3455128627 shutdown_connections 2024-04-05T21:18:11.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.511+0000 7f222073e700 1 --2- 172.21.15.188:0/3455128627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22180ff7a0 0x7f22180ffbc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:11.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.511+0000 7f222073e700 1 -- 172.21.15.188:0/3455128627 >> 172.21.15.188:0/3455128627 conn(0x7f22180fad20 msgr2=0x7f22180fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:11.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.511+0000 7f222073e700 1 -- 172.21.15.188:0/3455128627 shutdown_connections 2024-04-05T21:18:11.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.511+0000 7f222073e700 1 -- 172.21.15.188:0/3455128627 wait complete. 2024-04-05T21:18:11.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.512+0000 7f222073e700 1 Processor -- start 2024-04-05T21:18:11.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.512+0000 7f222073e700 1 -- start start 2024-04-05T21:18:11.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.512+0000 7f222073e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22180ff7a0 0x7f2218068b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:11.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.512+0000 7f222073e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2218196440 con 0x7f22180ff7a0 2024-04-05T21:18:11.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.512+0000 7f221e4da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22180ff7a0 0x7f2218068b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:11.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.512+0000 7f221e4da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22180ff7a0 0x7f2218068b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:49942/0 (socket says 172.21.15.188:49942) 2024-04-05T21:18:11.514 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.512+0000 7f221e4da700 1 -- 172.21.15.188:0/874447813 learned_addr learned my addr 172.21.15.188:0/874447813 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.513+0000 7f221e4da700 1 -- 172.21.15.188:0/874447813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22080095f0 con 0x7f22180ff7a0 2024-04-05T21:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.513+0000 7f221e4da700 1 --2- 172.21.15.188:0/874447813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22180ff7a0 0x7f2218068b00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2208006010 tx=0x7f22080057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.513+0000 7f220f7fe700 1 -- 172.21.15.188:0/874447813 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f220801d070 con 0x7f22180ff7a0 2024-04-05T21:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.513+0000 7f222073e700 1 -- 172.21.15.188:0/874447813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221819aa50 con 0x7f22180ff7a0 2024-04-05T21:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.513+0000 7f220f7fe700 1 -- 172.21.15.188:0/874447813 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f220800bdf0 con 0x7f22180ff7a0 2024-04-05T21:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.513+0000 7f220f7fe700 1 -- 172.21.15.188:0/874447813 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2208022e50 con 0x7f22180ff7a0 2024-04-05T21:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.513+0000 7f222073e700 1 -- 172.21.15.188:0/874447813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f221819adb0 con 0x7f22180ff7a0 2024-04-05T21:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.514+0000 7f222073e700 1 -- 172.21.15.188:0/874447813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f221818e280 con 0x7f22180ff7a0 2024-04-05T21:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.514+0000 7f220f7fe700 1 -- 172.21.15.188:0/874447813 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f2208022700 con 0x7f22180ff7a0 2024-04-05T21:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.514+0000 7f220f7fe700 1 --2- 172.21.15.188:0/874447813 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f22040383c0 0x7f220403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.514+0000 7f220f7fe700 1 -- 172.21.15.188:0/874447813 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f220804c330 con 0x7f22180ff7a0 2024-04-05T21:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.514+0000 7f221dcd9700 1 --2- 172.21.15.188:0/874447813 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f22040383c0 0x7f220403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:11.516 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.515+0000 7f221dcd9700 1 --2- 172.21.15.188:0/874447813 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f22040383c0 0x7f220403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2214006fd0 tx=0x7f2214006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:11.520 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.519+0000 7f220f7fe700 1 -- 172.21.15.188:0/874447813 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f220800b960 con 0x7f22180ff7a0 2024-04-05T21:18:11.786 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.784+0000 7f222073e700 1 -- 172.21.15.188:0/874447813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f221802cc70 con 0x7f22180ff7a0 2024-04-05T21:18:11.786 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.785+0000 7f220f7fe700 1 -- 172.21.15.188:0/874447813 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f2208027020 con 0x7f22180ff7a0 2024-04-05T21:18:11.787 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:18:11.787 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:18:11.788 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.787+0000 7f222073e700 1 -- 172.21.15.188:0/874447813 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f22040383c0 msgr2=0x7f220403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:11.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.787+0000 7f222073e700 1 --2- 172.21.15.188:0/874447813 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f22040383c0 0x7f220403a880 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2214006fd0 tx=0x7f2214006e40 comp rx=0 tx=0).stop 2024-04-05T21:18:11.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.787+0000 7f222073e700 1 -- 172.21.15.188:0/874447813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22180ff7a0 msgr2=0x7f2218068b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:11.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.787+0000 7f222073e700 1 --2- 172.21.15.188:0/874447813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22180ff7a0 0x7f2218068b00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2208006010 tx=0x7f22080057b0 comp rx=0 tx=0).stop 2024-04-05T21:18:11.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.787+0000 7f222073e700 1 -- 172.21.15.188:0/874447813 shutdown_connections 2024-04-05T21:18:11.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.787+0000 7f222073e700 1 --2- 172.21.15.188:0/874447813 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f22040383c0 0x7f220403a880 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:11.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.787+0000 7f222073e700 1 --2- 172.21.15.188:0/874447813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22180ff7a0 0x7f2218068b00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:11.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.787+0000 7f222073e700 1 -- 172.21.15.188:0/874447813 >> 172.21.15.188:0/874447813 conn(0x7f22180fad20 msgr2=0x7f22180fba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:11.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.787+0000 7f222073e700 1 -- 172.21.15.188:0/874447813 shutdown_connections 2024-04-05T21:18:11.789 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:11.787+0000 7f222073e700 1 -- 172.21.15.188:0/874447813 wait complete. 2024-04-05T21:18:11.790 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:18:12.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:12 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/874447813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:18:13.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:13 smithi121 ceph-mon[39723]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:13.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:18:13.530 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:18:13.771 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:13.839 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:15 smithi121 ceph-mon[39723]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:15.797 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.795+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/2743703067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4100890 msgr2=0x7f5ae4100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:15.797 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.795+0000 7f5ae9c7a700 1 --2- 172.21.15.188:0/2743703067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4100890 0x7f5ae4100cb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8009910 tx=0x7f5ad8009c20 comp rx=0 tx=0).stop 2024-04-05T21:18:15.797 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.796+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/2743703067 shutdown_connections 2024-04-05T21:18:15.797 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.796+0000 7f5ae9c7a700 1 --2- 172.21.15.188:0/2743703067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4100890 0x7f5ae4100cb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:15.797 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.796+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/2743703067 >> 172.21.15.188:0/2743703067 conn(0x7f5ae40fbe30 msgr2=0x7f5ae40fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:15.797 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.796+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/2743703067 shutdown_connections 2024-04-05T21:18:15.797 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.796+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/2743703067 wait complete. 2024-04-05T21:18:15.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.796+0000 7f5ae9c7a700 1 Processor -- start 2024-04-05T21:18:15.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.796+0000 7f5ae9c7a700 1 -- start start 2024-04-05T21:18:15.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.796+0000 7f5ae9c7a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4100890 0x7f5ae4074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:15.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.796+0000 7f5ae9c7a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae4075070 con 0x7f5ae4100890 2024-04-05T21:18:15.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.797+0000 7f5ae37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4100890 0x7f5ae4074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:15.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.797+0000 7f5ae37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4100890 0x7f5ae4074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:49962/0 (socket says 172.21.15.188:49962) 2024-04-05T21:18:15.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.797+0000 7f5ae37fe700 1 -- 172.21.15.188:0/793626916 learned_addr learned my addr 172.21.15.188:0/793626916 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:18:15.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.797+0000 7f5ae37fe700 1 -- 172.21.15.188:0/793626916 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad80095f0 con 0x7f5ae4100890 2024-04-05T21:18:15.799 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.797+0000 7f5ae37fe700 1 --2- 172.21.15.188:0/793626916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4100890 0x7f5ae4074b30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8004f70 tx=0x7f5ad8005ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:15.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.797+0000 7f5ae0ff9700 1 -- 172.21.15.188:0/793626916 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ad801d070 con 0x7f5ae4100890 2024-04-05T21:18:15.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.798+0000 7f5ae0ff9700 1 -- 172.21.15.188:0/793626916 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f5ad800bdf0 con 0x7f5ae4100890 2024-04-05T21:18:15.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.798+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/793626916 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae4073180 con 0x7f5ae4100890 2024-04-05T21:18:15.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.798+0000 7f5ae0ff9700 1 -- 172.21.15.188:0/793626916 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ad8022e40 con 0x7f5ae4100890 2024-04-05T21:18:15.800 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.798+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/793626916 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae4073620 con 0x7f5ae4100890 2024-04-05T21:18:15.801 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.798+0000 7f5ae0ff9700 1 -- 172.21.15.188:0/793626916 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45340+0+0 (secure 0 0 0) 0x7f5ad800b960 con 0x7f5ae4100890 2024-04-05T21:18:15.801 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.799+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/793626916 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae40623c0 con 0x7f5ae4100890 2024-04-05T21:18:15.801 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.799+0000 7f5ae0ff9700 1 --2- 172.21.15.188:0/793626916 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f5acc038410 0x7f5acc03a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:15.801 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.799+0000 7f5ae0ff9700 1 -- 172.21.15.188:0/793626916 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5ad804c400 con 0x7f5ae4100890 2024-04-05T21:18:15.801 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.799+0000 7f5ae2ffd700 1 --2- 172.21.15.188:0/793626916 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f5acc038410 0x7f5acc03a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:15.801 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.800+0000 7f5ae2ffd700 1 --2- 172.21.15.188:0/793626916 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f5acc038410 0x7f5acc03a8d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4006fd0 tx=0x7f5ad4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:15.805 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:15.804+0000 7f5ae0ff9700 1 -- 172.21.15.188:0/793626916 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5ad800f460 con 0x7f5ae4100890 2024-04-05T21:18:16.066 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:16.065+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/793626916 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ae418ed50 con 0x7f5ae4100890 2024-04-05T21:18:16.067 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:16.066+0000 7f5ae0ff9700 1 -- 172.21.15.188:0/793626916 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f5ad8027030 con 0x7f5ae4100890 2024-04-05T21:18:16.067 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:18:16.067 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:18:16.069 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:16.068+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/793626916 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f5acc038410 msgr2=0x7f5acc03a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:16.069 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:16.068+0000 7f5ae9c7a700 1 --2- 172.21.15.188:0/793626916 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f5acc038410 0x7f5acc03a8d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4006fd0 tx=0x7f5ad4006e40 comp rx=0 tx=0).stop 2024-04-05T21:18:16.069 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:16.068+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/793626916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4100890 msgr2=0x7f5ae4074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:16.069 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:16.068+0000 7f5ae9c7a700 1 --2- 172.21.15.188:0/793626916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4100890 0x7f5ae4074b30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8004f70 tx=0x7f5ad8005ee0 comp rx=0 tx=0).stop 2024-04-05T21:18:16.069 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:16.068+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/793626916 shutdown_connections 2024-04-05T21:18:16.069 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:16.068+0000 7f5ae9c7a700 1 --2- 172.21.15.188:0/793626916 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7f5acc038410 0x7f5acc03a8d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:16.069 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:16.068+0000 7f5ae9c7a700 1 --2- 172.21.15.188:0/793626916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4100890 0x7f5ae4074b30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:16.070 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:16.068+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/793626916 >> 172.21.15.188:0/793626916 conn(0x7f5ae40fbe30 msgr2=0x7f5ae40fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:16.070 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:16.068+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/793626916 shutdown_connections 2024-04-05T21:18:16.070 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:16.068+0000 7f5ae9c7a700 1 -- 172.21.15.188:0/793626916 wait complete. 2024-04-05T21:18:16.070 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:18:16.817 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:16 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/793626916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:18:17.845 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:18:17.846 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:18:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:17 smithi121 ceph-mon[39723]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:17 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:17 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:17 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:17.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:17 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-05T21:18:18.089 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:18.149 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:19.670 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:19 smithi121 ceph-mon[39723]: from='mgr.14162 172.21.15.121:0/352952062' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-05T21:18:19.671 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:19 smithi121 ceph-mon[39723]: mgrmap e14: smithi121.etbquh(active, since 69s) 2024-04-05T21:18:19.931 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.930+0000 7ff489326700 1 -- 172.21.15.188:0/2492345240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4840ff790 msgr2=0x7ff4840ffbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:19.931 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.930+0000 7ff489326700 1 --2- 172.21.15.188:0/2492345240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4840ff790 0x7ff4840ffbb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff478009910 tx=0x7ff478009c20 comp rx=0 tx=0).stop 2024-04-05T21:18:19.932 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.930+0000 7ff489326700 1 -- 172.21.15.188:0/2492345240 shutdown_connections 2024-04-05T21:18:19.932 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.930+0000 7ff489326700 1 --2- 172.21.15.188:0/2492345240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4840ff790 0x7ff4840ffbb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:19.932 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.930+0000 7ff489326700 1 -- 172.21.15.188:0/2492345240 >> 172.21.15.188:0/2492345240 conn(0x7ff4840fad30 msgr2=0x7ff4840fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:19.932 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.930+0000 7ff489326700 1 -- 172.21.15.188:0/2492345240 shutdown_connections 2024-04-05T21:18:19.932 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.930+0000 7ff489326700 1 -- 172.21.15.188:0/2492345240 wait complete. 2024-04-05T21:18:19.932 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.930+0000 7ff489326700 1 Processor -- start 2024-04-05T21:18:19.932 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.931+0000 7ff489326700 1 -- start start 2024-04-05T21:18:19.934 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.931+0000 7ff489326700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4840ff790 0x7ff484197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:19.934 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.931+0000 7ff489326700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff484198000 con 0x7ff4840ff790 2024-04-05T21:18:19.934 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.931+0000 7ff482ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4840ff790 0x7ff484197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:19.934 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.931+0000 7ff482ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4840ff790 0x7ff484197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:38702/0 (socket says 172.21.15.188:38702) 2024-04-05T21:18:19.934 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.931+0000 7ff482ffd700 1 -- 172.21.15.188:0/1626455987 learned_addr learned my addr 172.21.15.188:0/1626455987 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:18:19.934 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.931+0000 7ff482ffd700 1 -- 172.21.15.188:0/1626455987 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4780095f0 con 0x7ff4840ff790 2024-04-05T21:18:19.934 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.931+0000 7ff482ffd700 1 --2- 172.21.15.188:0/1626455987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4840ff790 0x7ff484197ac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff478004f70 tx=0x7ff478005ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:19.934 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.931+0000 7ff46bfff700 1 -- 172.21.15.188:0/1626455987 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff47801d070 con 0x7ff4840ff790 2024-04-05T21:18:19.935 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.932+0000 7ff489326700 1 -- 172.21.15.188:0/1626455987 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff484198200 con 0x7ff4840ff790 2024-04-05T21:18:19.935 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.932+0000 7ff46bfff700 1 -- 172.21.15.188:0/1626455987 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7ff47800bdf0 con 0x7ff4840ff790 2024-04-05T21:18:19.935 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.932+0000 7ff46bfff700 1 -- 172.21.15.188:0/1626455987 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff478022e40 con 0x7ff4840ff790 2024-04-05T21:18:19.935 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.932+0000 7ff489326700 1 -- 172.21.15.188:0/1626455987 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4841986a0 con 0x7ff4840ff790 2024-04-05T21:18:19.935 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.933+0000 7ff46bfff700 1 -- 172.21.15.188:0/1626455987 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 14) v1 ==== 45354+0+0 (secure 0 0 0) 0x7ff47800b960 con 0x7ff4840ff790 2024-04-05T21:18:19.935 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.933+0000 7ff489326700 1 -- 172.21.15.188:0/1626455987 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff48418d160 con 0x7ff4840ff790 2024-04-05T21:18:19.935 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.933+0000 7ff46bfff700 1 --2- 172.21.15.188:0/1626455987 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7ff46c0383f0 0x7ff46c03a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:19.935 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.933+0000 7ff46bfff700 1 -- 172.21.15.188:0/1626455987 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff47804c430 con 0x7ff4840ff790 2024-04-05T21:18:19.935 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.933+0000 7ff4827fc700 1 -- 172.21.15.188:0/1626455987 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7ff46c0383f0 msgr2=0x7ff46c03a8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1237774071 2024-04-05T21:18:19.935 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.933+0000 7ff4827fc700 1 --2- 172.21.15.188:0/1626455987 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7ff46c0383f0 0x7ff46c03a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T21:18:19.939 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:19.938+0000 7ff46bfff700 1 -- 172.21.15.188:0/1626455987 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff47802bd80 con 0x7ff4840ff790 2024-04-05T21:18:20.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.134+0000 7ff4827fc700 1 -- 172.21.15.188:0/1626455987 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7ff46c0383f0 msgr2=0x7ff46c03a8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1237774071 2024-04-05T21:18:20.135 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.134+0000 7ff4827fc700 1 --2- 172.21.15.188:0/1626455987 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7ff46c0383f0 0x7ff46c03a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T21:18:20.202 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.201+0000 7ff489326700 1 -- 172.21.15.188:0/1626455987 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff48402cc70 con 0x7ff4840ff790 2024-04-05T21:18:20.203 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.202+0000 7ff46bfff700 1 -- 172.21.15.188:0/1626455987 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ff478027030 con 0x7ff4840ff790 2024-04-05T21:18:20.203 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:18:20.204 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:18:20.205 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.204+0000 7ff489326700 1 -- 172.21.15.188:0/1626455987 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7ff46c0383f0 msgr2=0x7ff46c03a8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T21:18:20.205 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.204+0000 7ff489326700 1 --2- 172.21.15.188:0/1626455987 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7ff46c0383f0 0x7ff46c03a8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:20.205 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.204+0000 7ff489326700 1 -- 172.21.15.188:0/1626455987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4840ff790 msgr2=0x7ff484197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:20.205 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.204+0000 7ff489326700 1 --2- 172.21.15.188:0/1626455987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4840ff790 0x7ff484197ac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff478004f70 tx=0x7ff478005ee0 comp rx=0 tx=0).stop 2024-04-05T21:18:20.205 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.204+0000 7ff489326700 1 -- 172.21.15.188:0/1626455987 shutdown_connections 2024-04-05T21:18:20.205 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.204+0000 7ff489326700 1 --2- 172.21.15.188:0/1626455987 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7ff46c0383f0 0x7ff46c03a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:20.205 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.204+0000 7ff489326700 1 --2- 172.21.15.188:0/1626455987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4840ff790 0x7ff484197ac0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:20.206 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.204+0000 7ff489326700 1 -- 172.21.15.188:0/1626455987 >> 172.21.15.188:0/1626455987 conn(0x7ff4840fad30 msgr2=0x7ff4840fba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:20.206 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.204+0000 7ff489326700 1 -- 172.21.15.188:0/1626455987 shutdown_connections 2024-04-05T21:18:20.206 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:20.204+0000 7ff489326700 1 -- 172.21.15.188:0/1626455987 wait complete. 2024-04-05T21:18:20.206 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:18:21.450 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:21 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/1626455987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:18:21.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:18:21.883 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:18:22.129 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:22.190 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:23.919 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.918+0000 7fa9a37eb700 1 -- 172.21.15.188:0/3908366816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c0ffcc0 msgr2=0x7fa99c1000e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:23.919 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.918+0000 7fa9a37eb700 1 --2- 172.21.15.188:0/3908366816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c0ffcc0 0x7fa99c1000e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa98c009910 tx=0x7fa98c009c20 comp rx=0 tx=0).stop 2024-04-05T21:18:23.919 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.918+0000 7fa9a37eb700 1 -- 172.21.15.188:0/3908366816 shutdown_connections 2024-04-05T21:18:23.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.918+0000 7fa9a37eb700 1 --2- 172.21.15.188:0/3908366816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c0ffcc0 0x7fa99c1000e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:23.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.918+0000 7fa9a37eb700 1 -- 172.21.15.188:0/3908366816 >> 172.21.15.188:0/3908366816 conn(0x7fa99c0fa4a0 msgr2=0x7fa99c0fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:23.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.918+0000 7fa9a37eb700 1 -- 172.21.15.188:0/3908366816 shutdown_connections 2024-04-05T21:18:23.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.918+0000 7fa9a37eb700 1 -- 172.21.15.188:0/3908366816 wait complete. 2024-04-05T21:18:23.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.919+0000 7fa9a37eb700 1 Processor -- start 2024-04-05T21:18:23.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.919+0000 7fa9a37eb700 1 -- start start 2024-04-05T21:18:23.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.919+0000 7fa9a37eb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c0ffcc0 0x7fa99c194830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:23.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.919+0000 7fa9a37eb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c194d70 con 0x7fa99c0ffcc0 2024-04-05T21:18:23.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.919+0000 7fa9a1587700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c0ffcc0 0x7fa99c194830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:23.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.919+0000 7fa9a1587700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c0ffcc0 0x7fa99c194830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:38714/0 (socket says 172.21.15.188:38714) 2024-04-05T21:18:23.920 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.919+0000 7fa9a1587700 1 -- 172.21.15.188:0/2513352577 learned_addr learned my addr 172.21.15.188:0/2513352577 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:18:23.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.919+0000 7fa9a1587700 1 -- 172.21.15.188:0/2513352577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c0095f0 con 0x7fa99c0ffcc0 2024-04-05T21:18:23.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.920+0000 7fa9a1587700 1 --2- 172.21.15.188:0/2513352577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c0ffcc0 0x7fa99c194830 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa98c006010 tx=0x7fa98c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:23.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.920+0000 7fa9927fc700 1 -- 172.21.15.188:0/2513352577 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa98c01d070 con 0x7fa99c0ffcc0 2024-04-05T21:18:23.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.920+0000 7fa9927fc700 1 -- 172.21.15.188:0/2513352577 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fa98c00bdf0 con 0x7fa99c0ffcc0 2024-04-05T21:18:23.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.920+0000 7fa9a37eb700 1 -- 172.21.15.188:0/2513352577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c194f70 con 0x7fa99c0ffcc0 2024-04-05T21:18:23.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.920+0000 7fa9927fc700 1 -- 172.21.15.188:0/2513352577 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa98c022e50 con 0x7fa99c0ffcc0 2024-04-05T21:18:23.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.920+0000 7fa9a37eb700 1 -- 172.21.15.188:0/2513352577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c195410 con 0x7fa99c0ffcc0 2024-04-05T21:18:23.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.921+0000 7fa9927fc700 1 -- 172.21.15.188:0/2513352577 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 14) v1 ==== 45354+0+0 (secure 0 0 0) 0x7fa98c00f460 con 0x7fa99c0ffcc0 2024-04-05T21:18:23.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.921+0000 7fa9a37eb700 1 -- 172.21.15.188:0/2513352577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99c18e290 con 0x7fa99c0ffcc0 2024-04-05T21:18:23.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.921+0000 7fa9927fc700 1 --2- 172.21.15.188:0/2513352577 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fa988038410 0x7fa98803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:23.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.921+0000 7fa9927fc700 1 -- 172.21.15.188:0/2513352577 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa98c04c510 con 0x7fa99c0ffcc0 2024-04-05T21:18:23.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.921+0000 7fa9a0d86700 1 -- 172.21.15.188:0/2513352577 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fa988038410 msgr2=0x7fa98803a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1237774071 2024-04-05T21:18:23.922 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.921+0000 7fa9a0d86700 1 --2- 172.21.15.188:0/2513352577 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fa988038410 0x7fa98803a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T21:18:23.926 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:23.926+0000 7fa9927fc700 1 -- 172.21.15.188:0/2513352577 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fa98c017440 con 0x7fa99c0ffcc0 2024-04-05T21:18:24.122 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.122+0000 7fa9a0d86700 1 -- 172.21.15.188:0/2513352577 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fa988038410 msgr2=0x7fa98803a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1237774071 2024-04-05T21:18:24.122 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.122+0000 7fa9a0d86700 1 --2- 172.21.15.188:0/2513352577 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fa988038410 0x7fa98803a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-05T21:18:24.190 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.190+0000 7fa9a37eb700 1 -- 172.21.15.188:0/2513352577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa99c02cc70 con 0x7fa99c0ffcc0 2024-04-05T21:18:24.191 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.191+0000 7fa9927fc700 1 -- 172.21.15.188:0/2513352577 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa98c027020 con 0x7fa99c0ffcc0 2024-04-05T21:18:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:18:24.191 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:18:24.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.193+0000 7fa9a37eb700 1 -- 172.21.15.188:0/2513352577 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fa988038410 msgr2=0x7fa98803a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T21:18:24.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.193+0000 7fa9a37eb700 1 --2- 172.21.15.188:0/2513352577 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fa988038410 0x7fa98803a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:24.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.193+0000 7fa9a37eb700 1 -- 172.21.15.188:0/2513352577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c0ffcc0 msgr2=0x7fa99c194830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:24.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.193+0000 7fa9a37eb700 1 --2- 172.21.15.188:0/2513352577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c0ffcc0 0x7fa99c194830 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa98c006010 tx=0x7fa98c0057b0 comp rx=0 tx=0).stop 2024-04-05T21:18:24.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.193+0000 7fa9a37eb700 1 -- 172.21.15.188:0/2513352577 shutdown_connections 2024-04-05T21:18:24.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.193+0000 7fa9a37eb700 1 --2- 172.21.15.188:0/2513352577 >> [v2:172.21.15.121:6800/1237774071,v1:172.21.15.121:6801/1237774071] conn(0x7fa988038410 0x7fa98803a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:24.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.193+0000 7fa9a37eb700 1 --2- 172.21.15.188:0/2513352577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c0ffcc0 0x7fa99c194830 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:24.193 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.193+0000 7fa9a37eb700 1 -- 172.21.15.188:0/2513352577 >> 172.21.15.188:0/2513352577 conn(0x7fa99c0fa4a0 msgr2=0x7fa99c0fb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:24.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.193+0000 7fa9a37eb700 1 -- 172.21.15.188:0/2513352577 shutdown_connections 2024-04-05T21:18:24.194 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:24.193+0000 7fa9a37eb700 1 -- 172.21.15.188:0/2513352577 wait complete. 2024-04-05T21:18:24.194 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:18:24.642 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:24 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/2513352577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:18:25.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: Active manager daemon smithi121.etbquh restarted 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: Activating manager daemon smithi121.etbquh 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: osdmap e5: 0 total, 0 up, 0 in 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: mgrmap e15: smithi121.etbquh(active, starting, since 0.00326437s) 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr metadata", "who": "smithi121.etbquh", "id": "smithi121.etbquh"}]: dispatch 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: Manager daemon smithi121.etbquh is now available 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:18:25.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.etbquh/mirror_snapshot_schedule"}]: dispatch 2024-04-05T21:18:25.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.etbquh/trash_purge_schedule"}]: dispatch 2024-04-05T21:18:25.896 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:18:25.897 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:18:26.153 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:26.214 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:26.733 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:26 smithi121 ceph-mon[39723]: [05/Apr/2024:21:18:25] ENGINE Bus STARTING 2024-04-05T21:18:26.733 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:26 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:26.734 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:26 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:26.734 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:26 smithi121 ceph-mon[39723]: mgrmap e16: smithi121.etbquh(active, since 1.00775s) 2024-04-05T21:18:27.570 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:27 smithi121 ceph-mon[39723]: [05/Apr/2024:21:18:25] ENGINE Serving on https://172.21.15.121:7150 2024-04-05T21:18:27.570 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:27 smithi121 ceph-mon[39723]: [05/Apr/2024:21:18:25] ENGINE Serving on http://172.21.15.121:8765 2024-04-05T21:18:27.570 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:27 smithi121 ceph-mon[39723]: [05/Apr/2024:21:18:25] ENGINE Bus STARTED 2024-04-05T21:18:27.570 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:27 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:28.307 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.182+0000 7fdf4191d700 1 -- 172.21.15.188:0/338720995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf3c0ff7a0 msgr2=0x7fdf3c0ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:28.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.182+0000 7fdf4191d700 1 --2- 172.21.15.188:0/338720995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf3c0ff7a0 0x7fdf3c0ffbc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400b0a0 tx=0x7fdf3400b3b0 comp rx=0 tx=0).stop 2024-04-05T21:18:28.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.183+0000 7fdf4191d700 1 -- 172.21.15.188:0/338720995 shutdown_connections 2024-04-05T21:18:28.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.183+0000 7fdf4191d700 1 --2- 172.21.15.188:0/338720995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf3c0ff7a0 0x7fdf3c0ffbc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:28.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.183+0000 7fdf4191d700 1 -- 172.21.15.188:0/338720995 >> 172.21.15.188:0/338720995 conn(0x7fdf3c0fad20 msgr2=0x7fdf3c0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:28.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.183+0000 7fdf4191d700 1 -- 172.21.15.188:0/338720995 shutdown_connections 2024-04-05T21:18:28.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.183+0000 7fdf4191d700 1 -- 172.21.15.188:0/338720995 wait complete. 2024-04-05T21:18:28.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.183+0000 7fdf4191d700 1 Processor -- start 2024-04-05T21:18:28.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.183+0000 7fdf4191d700 1 -- start start 2024-04-05T21:18:28.308 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.183+0000 7fdf4191d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf3c0ff7a0 0x7fdf3c194800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:28.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.183+0000 7fdf4191d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c194d40 con 0x7fdf3c0ff7a0 2024-04-05T21:18:28.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.184+0000 7fdf3affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf3c0ff7a0 0x7fdf3c194800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:28.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.184+0000 7fdf3affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf3c0ff7a0 0x7fdf3c194800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:41742/0 (socket says 172.21.15.188:41742) 2024-04-05T21:18:28.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.184+0000 7fdf3affd700 1 -- 172.21.15.188:0/2206679694 learned_addr learned my addr 172.21.15.188:0/2206679694 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:18:28.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.184+0000 7fdf3affd700 1 -- 172.21.15.188:0/2206679694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf34009d00 con 0x7fdf3c0ff7a0 2024-04-05T21:18:28.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.184+0000 7fdf3affd700 1 --2- 172.21.15.188:0/2206679694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf3c0ff7a0 0x7fdf3c194800 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdf340062a0 tx=0x7fdf3400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:28.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.184+0000 7fdf4091b700 1 -- 172.21.15.188:0/2206679694 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf3400e040 con 0x7fdf3c0ff7a0 2024-04-05T21:18:28.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.184+0000 7fdf4091b700 1 -- 172.21.15.188:0/2206679694 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fdf34012ad0 con 0x7fdf3c0ff7a0 2024-04-05T21:18:28.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.184+0000 7fdf4191d700 1 -- 172.21.15.188:0/2206679694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3c194f40 con 0x7fdf3c0ff7a0 2024-04-05T21:18:28.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.184+0000 7fdf4091b700 1 -- 172.21.15.188:0/2206679694 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf3401c860 con 0x7fdf3c0ff7a0 2024-04-05T21:18:28.309 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.184+0000 7fdf4191d700 1 -- 172.21.15.188:0/2206679694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3c1953e0 con 0x7fdf3c0ff7a0 2024-04-05T21:18:28.310 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.185+0000 7fdf4091b700 1 -- 172.21.15.188:0/2206679694 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7fdf3401cb20 con 0x7fdf3c0ff7a0 2024-04-05T21:18:28.310 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.185+0000 7fdf4191d700 1 -- 172.21.15.188:0/2206679694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3c18e280 con 0x7fdf3c0ff7a0 2024-04-05T21:18:28.310 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.186+0000 7fdf4091b700 1 --2- 172.21.15.188:0/2206679694 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fdf240384b0 0x7fdf2403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:28.310 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.186+0000 7fdf4091b700 1 -- 172.21.15.188:0/2206679694 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fdf3404c740 con 0x7fdf3c0ff7a0 2024-04-05T21:18:28.310 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.186+0000 7fdf3a7fc700 1 --2- 172.21.15.188:0/2206679694 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fdf240384b0 0x7fdf2403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:28.310 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.186+0000 7fdf3a7fc700 1 --2- 172.21.15.188:0/2206679694 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fdf240384b0 0x7fdf2403a970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdf30006fd0 tx=0x7fdf30006e40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:28.310 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.191+0000 7fdf4091b700 1 -- 172.21.15.188:0/2206679694 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fdf3401b950 con 0x7fdf3c0ff7a0 2024-04-05T21:18:28.459 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.459+0000 7fdf4191d700 1 -- 172.21.15.188:0/2206679694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf3c02cc70 con 0x7fdf3c0ff7a0 2024-04-05T21:18:28.459 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.459+0000 7fdf4091b700 1 -- 172.21.15.188:0/2206679694 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fdf3401b360 con 0x7fdf3c0ff7a0 2024-04-05T21:18:28.460 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:18:28.460 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:18:28.461 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.461+0000 7fdf4191d700 1 -- 172.21.15.188:0/2206679694 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fdf240384b0 msgr2=0x7fdf2403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:28.462 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.461+0000 7fdf4191d700 1 --2- 172.21.15.188:0/2206679694 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fdf240384b0 0x7fdf2403a970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdf30006fd0 tx=0x7fdf30006e40 comp rx=0 tx=0).stop 2024-04-05T21:18:28.462 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.461+0000 7fdf4191d700 1 -- 172.21.15.188:0/2206679694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf3c0ff7a0 msgr2=0x7fdf3c194800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:28.462 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.461+0000 7fdf4191d700 1 --2- 172.21.15.188:0/2206679694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf3c0ff7a0 0x7fdf3c194800 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdf340062a0 tx=0x7fdf3400bd40 comp rx=0 tx=0).stop 2024-04-05T21:18:28.462 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.461+0000 7fdf4191d700 1 -- 172.21.15.188:0/2206679694 shutdown_connections 2024-04-05T21:18:28.462 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.461+0000 7fdf4191d700 1 --2- 172.21.15.188:0/2206679694 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fdf240384b0 0x7fdf2403a970 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:28.462 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.461+0000 7fdf4191d700 1 --2- 172.21.15.188:0/2206679694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf3c0ff7a0 0x7fdf3c194800 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:28.462 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.461+0000 7fdf4191d700 1 -- 172.21.15.188:0/2206679694 >> 172.21.15.188:0/2206679694 conn(0x7fdf3c0fad20 msgr2=0x7fdf3c0fba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:28.462 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.461+0000 7fdf4191d700 1 -- 172.21.15.188:0/2206679694 shutdown_connections 2024-04-05T21:18:28.462 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:28.461+0000 7fdf4191d700 1 -- 172.21.15.188:0/2206679694 wait complete. 2024-04-05T21:18:28.463 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:18:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:28 smithi121 ceph-mon[39723]: mgrmap e17: smithi121.etbquh(active, since 2s) 2024-04-05T21:18:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:28 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/2206679694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:18:30.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:30 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:30.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:30 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:30.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:30 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:30.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:30 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:30.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:30 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi121", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:18:31.258 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:18:31.259 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:18:31.500 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:31.570 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-05T21:18:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:32 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:32 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:32 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:32 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:32.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:32 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd/host:smithi188", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:18:32.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:32 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:18:32.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:32 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:18:32.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:32 smithi121 ceph-mon[39723]: Updating smithi121:/etc/ceph/ceph.conf 2024-04-05T21:18:32.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:32 smithi121 ceph-mon[39723]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-05T21:18:32.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:32 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:32.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:32 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:33.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:33 smithi121 ceph-mon[39723]: Updating smithi121:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:18:33.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:33 smithi121 ceph-mon[39723]: Updating smithi188:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:18:33.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:33 smithi121 ceph-mon[39723]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-04-05T21:18:33.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:33 smithi121 ceph-mon[39723]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-05T21:18:33.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:33 smithi121 ceph-mon[39723]: Updating smithi121:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T21:18:33.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:33 smithi121 ceph-mon[39723]: Updating smithi188:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T21:18:33.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:33 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:33.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:33 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:33.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:33 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:33.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:33 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi188", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-05T21:18:33.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:33 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi188", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-04-05T21:18:33.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:33 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:18:33.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:33 smithi121 ceph-mon[39723]: Deploying daemon ceph-exporter.smithi188 on smithi188 2024-04-05T21:18:34.050 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.048+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2680561613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b48106810 msgr2=0x7f4b48106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:34.050 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.048+0000 7f4b4f4d7700 1 --2- 172.21.15.188:0/2680561613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b48106810 0x7f4b48106bf0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4b38009910 tx=0x7f4b38009c20 comp rx=0 tx=0).stop 2024-04-05T21:18:34.050 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.048+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2680561613 shutdown_connections 2024-04-05T21:18:34.050 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.048+0000 7f4b4f4d7700 1 --2- 172.21.15.188:0/2680561613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b48106810 0x7f4b48106bf0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:34.050 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.048+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2680561613 >> 172.21.15.188:0/2680561613 conn(0x7f4b480fc320 msgr2=0x7f4b480fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:34.050 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.048+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2680561613 shutdown_connections 2024-04-05T21:18:34.050 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.048+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2680561613 wait complete. 2024-04-05T21:18:34.050 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.049+0000 7f4b4f4d7700 1 Processor -- start 2024-04-05T21:18:34.051 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.049+0000 7f4b4f4d7700 1 -- start start 2024-04-05T21:18:34.052 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.049+0000 7f4b4f4d7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b48106810 0x7f4b4819ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:34.052 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.049+0000 7f4b4f4d7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4819bf90 con 0x7f4b48106810 2024-04-05T21:18:34.052 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.050+0000 7f4b4d273700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b48106810 0x7f4b4819ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:34.052 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.050+0000 7f4b4d273700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b48106810 0x7f4b4819ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:41752/0 (socket says 172.21.15.188:41752) 2024-04-05T21:18:34.053 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.050+0000 7f4b4d273700 1 -- 172.21.15.188:0/2072892347 learned_addr learned my addr 172.21.15.188:0/2072892347 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:18:34.053 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.050+0000 7f4b4d273700 1 -- 172.21.15.188:0/2072892347 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b380095f0 con 0x7f4b48106810 2024-04-05T21:18:34.053 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.050+0000 7f4b4d273700 1 --2- 172.21.15.188:0/2072892347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b48106810 0x7f4b4819ba50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4b38004f40 tx=0x7f4b38005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:34.053 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.050+0000 7f4b3e7fc700 1 -- 172.21.15.188:0/2072892347 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b3801d070 con 0x7f4b48106810 2024-04-05T21:18:34.053 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.050+0000 7f4b3e7fc700 1 -- 172.21.15.188:0/2072892347 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f4b38005dc0 con 0x7f4b48106810 2024-04-05T21:18:34.053 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.050+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2072892347 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b4819c1f0 con 0x7f4b48106810 2024-04-05T21:18:34.053 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.050+0000 7f4b3e7fc700 1 -- 172.21.15.188:0/2072892347 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b38022d70 con 0x7f4b48106810 2024-04-05T21:18:34.053 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.050+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2072892347 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b48195a40 con 0x7f4b48106810 2024-04-05T21:18:34.053 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.051+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2072892347 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b480623c0 con 0x7f4b48106810 2024-04-05T21:18:34.053 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.051+0000 7f4b3e7fc700 1 -- 172.21.15.188:0/2072892347 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f4b3800b960 con 0x7f4b48106810 2024-04-05T21:18:34.053 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.051+0000 7f4b3e7fc700 1 --2- 172.21.15.188:0/2072892347 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4b340384b0 0x7f4b3403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:34.053 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.051+0000 7f4b3e7fc700 1 -- 172.21.15.188:0/2072892347 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f4b3804c600 con 0x7f4b48106810 2024-04-05T21:18:34.054 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.051+0000 7f4b4ca72700 1 --2- 172.21.15.188:0/2072892347 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4b340384b0 0x7f4b3403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:34.054 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.052+0000 7f4b4ca72700 1 --2- 172.21.15.188:0/2072892347 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4b340384b0 0x7f4b3403a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4b44006fd0 tx=0x7f4b44006e40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:34.057 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.056+0000 7f4b3e7fc700 1 -- 172.21.15.188:0/2072892347 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4b3802bba0 con 0x7f4b48106810 2024-04-05T21:18:34.321 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.320+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2072892347 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b481964d0 con 0x7f4b48106810 2024-04-05T21:18:34.322 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.320+0000 7f4b3e7fc700 1 -- 172.21.15.188:0/2072892347 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f4b38027030 con 0x7f4b48106810 2024-04-05T21:18:34.322 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:18:34.322 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:18:34.324 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.323+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2072892347 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4b340384b0 msgr2=0x7f4b3403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:34.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.323+0000 7f4b4f4d7700 1 --2- 172.21.15.188:0/2072892347 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4b340384b0 0x7f4b3403a970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4b44006fd0 tx=0x7f4b44006e40 comp rx=0 tx=0).stop 2024-04-05T21:18:34.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.323+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2072892347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b48106810 msgr2=0x7f4b4819ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:34.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.323+0000 7f4b4f4d7700 1 --2- 172.21.15.188:0/2072892347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b48106810 0x7f4b4819ba50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4b38004f40 tx=0x7f4b38005020 comp rx=0 tx=0).stop 2024-04-05T21:18:34.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.323+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2072892347 shutdown_connections 2024-04-05T21:18:34.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.323+0000 7f4b4f4d7700 1 --2- 172.21.15.188:0/2072892347 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4b340384b0 0x7f4b3403a970 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:34.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.323+0000 7f4b4f4d7700 1 --2- 172.21.15.188:0/2072892347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b48106810 0x7f4b4819ba50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:34.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.323+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2072892347 >> 172.21.15.188:0/2072892347 conn(0x7f4b480fc320 msgr2=0x7f4b48108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:34.325 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.323+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2072892347 shutdown_connections 2024-04-05T21:18:34.326 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:34.323+0000 7f4b4f4d7700 1 -- 172.21.15.188:0/2072892347 wait complete. 2024-04-05T21:18:34.326 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:18:34.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:34 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/2072892347' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:18:36.234 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:18:36.235 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:18:37.013 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:18:37.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:37 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:37.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:37 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:37.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:37 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:37.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:37 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:37.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:37 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi188", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T21:18:37.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:37 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi188", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-05T21:18:37.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:37 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:18:37.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:37 smithi121 ceph-mon[39723]: Deploying daemon crash.smithi188 on smithi188 2024-04-05T21:18:39.466 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.385+0000 7efc8e4db700 1 -- 172.21.15.188:0/4236204395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc88106810 msgr2=0x7efc88106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:39.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.385+0000 7efc8e4db700 1 --2- 172.21.15.188:0/4236204395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc88106810 0x7efc88106bf0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efc78009910 tx=0x7efc78009c20 comp rx=0 tx=0).stop 2024-04-05T21:18:39.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.385+0000 7efc8e4db700 1 -- 172.21.15.188:0/4236204395 shutdown_connections 2024-04-05T21:18:39.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.385+0000 7efc8e4db700 1 --2- 172.21.15.188:0/4236204395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc88106810 0x7efc88106bf0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:39.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.385+0000 7efc8e4db700 1 -- 172.21.15.188:0/4236204395 >> 172.21.15.188:0/4236204395 conn(0x7efc880fc320 msgr2=0x7efc880fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:39.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.385+0000 7efc8e4db700 1 -- 172.21.15.188:0/4236204395 shutdown_connections 2024-04-05T21:18:39.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.385+0000 7efc8e4db700 1 -- 172.21.15.188:0/4236204395 wait complete. 2024-04-05T21:18:39.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.385+0000 7efc8e4db700 1 Processor -- start 2024-04-05T21:18:39.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.385+0000 7efc8e4db700 1 -- start start 2024-04-05T21:18:39.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.386+0000 7efc8e4db700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc88106810 0x7efc88073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:39.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.386+0000 7efc8e4db700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc88075060 con 0x7efc88106810 2024-04-05T21:18:39.467 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.386+0000 7efc87fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc88106810 0x7efc88073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:39.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.386+0000 7efc87fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc88106810 0x7efc88073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:45850/0 (socket says 172.21.15.188:45850) 2024-04-05T21:18:39.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.386+0000 7efc87fff700 1 -- 172.21.15.188:0/2953438095 learned_addr learned my addr 172.21.15.188:0/2953438095 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:18:39.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.386+0000 7efc87fff700 1 -- 172.21.15.188:0/2953438095 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc780095f0 con 0x7efc88106810 2024-04-05T21:18:39.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.386+0000 7efc87fff700 1 --2- 172.21.15.188:0/2953438095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc88106810 0x7efc88073090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efc78004f40 tx=0x7efc78005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:39.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.387+0000 7efc857fa700 1 -- 172.21.15.188:0/2953438095 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc7801d070 con 0x7efc88106810 2024-04-05T21:18:39.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.387+0000 7efc8e4db700 1 -- 172.21.15.188:0/2953438095 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc88073690 con 0x7efc88106810 2024-04-05T21:18:39.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.387+0000 7efc8e4db700 1 -- 172.21.15.188:0/2953438095 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc88073b10 con 0x7efc88106810 2024-04-05T21:18:39.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.387+0000 7efc857fa700 1 -- 172.21.15.188:0/2953438095 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7efc78005dc0 con 0x7efc88106810 2024-04-05T21:18:39.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.387+0000 7efc857fa700 1 -- 172.21.15.188:0/2953438095 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc78022d20 con 0x7efc88106810 2024-04-05T21:18:39.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.388+0000 7efc857fa700 1 -- 172.21.15.188:0/2953438095 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7efc7800f460 con 0x7efc88106810 2024-04-05T21:18:39.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.388+0000 7efc857fa700 1 --2- 172.21.15.188:0/2953438095 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7efc700384b0 0x7efc7003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:39.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.388+0000 7efc857fa700 1 -- 172.21.15.188:0/2953438095 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7efc7804c3b0 con 0x7efc88106810 2024-04-05T21:18:39.468 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.388+0000 7efc8e4db700 1 -- 172.21.15.188:0/2953438095 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc74005320 con 0x7efc88106810 2024-04-05T21:18:39.469 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.388+0000 7efc877fe700 1 --2- 172.21.15.188:0/2953438095 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7efc700384b0 0x7efc7003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:39.469 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.389+0000 7efc877fe700 1 --2- 172.21.15.188:0/2953438095 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7efc700384b0 0x7efc7003a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efc7c006fd0 tx=0x7efc7c006e40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:39.469 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.393+0000 7efc857fa700 1 -- 172.21.15.188:0/2953438095 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7efc78027080 con 0x7efc88106810 2024-04-05T21:18:39.671 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.670+0000 7efc8e4db700 1 -- 172.21.15.188:0/2953438095 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc74005190 con 0x7efc88106810 2024-04-05T21:18:39.671 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.670+0000 7efc857fa700 1 -- 172.21.15.188:0/2953438095 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7efc7800fe40 con 0x7efc88106810 2024-04-05T21:18:39.672 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:18:39.672 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:18:39.673 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.672+0000 7efc8e4db700 1 -- 172.21.15.188:0/2953438095 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7efc700384b0 msgr2=0x7efc7003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:39.673 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.672+0000 7efc8e4db700 1 --2- 172.21.15.188:0/2953438095 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7efc700384b0 0x7efc7003a970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efc7c006fd0 tx=0x7efc7c006e40 comp rx=0 tx=0).stop 2024-04-05T21:18:39.674 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.672+0000 7efc8e4db700 1 -- 172.21.15.188:0/2953438095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc88106810 msgr2=0x7efc88073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:39.674 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.672+0000 7efc8e4db700 1 --2- 172.21.15.188:0/2953438095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc88106810 0x7efc88073090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efc78004f40 tx=0x7efc78005020 comp rx=0 tx=0).stop 2024-04-05T21:18:39.674 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.672+0000 7efc8e4db700 1 -- 172.21.15.188:0/2953438095 shutdown_connections 2024-04-05T21:18:39.674 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.672+0000 7efc8e4db700 1 --2- 172.21.15.188:0/2953438095 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7efc700384b0 0x7efc7003a970 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:39.674 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.672+0000 7efc8e4db700 1 --2- 172.21.15.188:0/2953438095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc88106810 0x7efc88073090 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:39.674 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.672+0000 7efc8e4db700 1 -- 172.21.15.188:0/2953438095 >> 172.21.15.188:0/2953438095 conn(0x7efc880fc320 msgr2=0x7efc88108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:39.674 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.672+0000 7efc8e4db700 1 -- 172.21.15.188:0/2953438095 shutdown_connections 2024-04-05T21:18:39.674 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:39.673+0000 7efc8e4db700 1 -- 172.21.15.188:0/2953438095 wait complete. 2024-04-05T21:18:39.674 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:18:40.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:40.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:40.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:40.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:40.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:40 smithi121 ceph-mon[39723]: Deploying daemon node-exporter.smithi188 on smithi188 2024-04-05T21:18:40.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:40 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/2953438095' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:18:40.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:40.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:18:41.352 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:18:41.352 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:18:41.586 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:18:44.483 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.480+0000 7f85d85cd700 1 -- 172.21.15.188:0/4240292501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85d0106820 msgr2=0x7f85d0106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:44.483 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.480+0000 7f85d85cd700 1 --2- 172.21.15.188:0/4240292501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85d0106820 0x7f85d0106c00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f85c0009910 tx=0x7f85c0009c20 comp rx=0 tx=0).stop 2024-04-05T21:18:44.483 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.481+0000 7f85d85cd700 1 -- 172.21.15.188:0/4240292501 shutdown_connections 2024-04-05T21:18:44.483 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.481+0000 7f85d85cd700 1 --2- 172.21.15.188:0/4240292501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85d0106820 0x7f85d0106c00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:44.483 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.481+0000 7f85d85cd700 1 -- 172.21.15.188:0/4240292501 >> 172.21.15.188:0/4240292501 conn(0x7f85d00752b0 msgr2=0x7f85d00756c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:44.483 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.481+0000 7f85d85cd700 1 -- 172.21.15.188:0/4240292501 shutdown_connections 2024-04-05T21:18:44.484 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.481+0000 7f85d85cd700 1 -- 172.21.15.188:0/4240292501 wait complete. 2024-04-05T21:18:44.484 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.481+0000 7f85d85cd700 1 Processor -- start 2024-04-05T21:18:44.484 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.481+0000 7f85d85cd700 1 -- start start 2024-04-05T21:18:44.484 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.482+0000 7f85d85cd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85d0106820 0x7f85d019ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:44.484 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.482+0000 7f85d85cd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d019bf60 con 0x7f85d0106820 2024-04-05T21:18:44.484 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.482+0000 7f85d6369700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85d0106820 0x7f85d019ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:44.484 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.482+0000 7f85d6369700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85d0106820 0x7f85d019ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:45872/0 (socket says 172.21.15.188:45872) 2024-04-05T21:18:44.484 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.482+0000 7f85d6369700 1 -- 172.21.15.188:0/804115966 learned_addr learned my addr 172.21.15.188:0/804115966 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:18:44.485 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.482+0000 7f85d6369700 1 -- 172.21.15.188:0/804115966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c00095f0 con 0x7f85d0106820 2024-04-05T21:18:44.486 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.482+0000 7f85d6369700 1 --2- 172.21.15.188:0/804115966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85d0106820 0x7f85d019ba20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f85c0004f40 tx=0x7f85c0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:44.486 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.483+0000 7f85c77fe700 1 -- 172.21.15.188:0/804115966 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85c001d070 con 0x7f85d0106820 2024-04-05T21:18:44.486 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.483+0000 7f85d85cd700 1 -- 172.21.15.188:0/804115966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d019c1c0 con 0x7f85d0106820 2024-04-05T21:18:44.486 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.483+0000 7f85c77fe700 1 -- 172.21.15.188:0/804115966 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f85c0005dc0 con 0x7f85d0106820 2024-04-05T21:18:44.486 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.483+0000 7f85c77fe700 1 -- 172.21.15.188:0/804115966 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85c0022d70 con 0x7f85d0106820 2024-04-05T21:18:44.486 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.483+0000 7f85d85cd700 1 -- 172.21.15.188:0/804115966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d0195a10 con 0x7f85d0106820 2024-04-05T21:18:44.486 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.484+0000 7f85d85cd700 1 -- 172.21.15.188:0/804115966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d00623c0 con 0x7f85d0106820 2024-04-05T21:18:44.486 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.484+0000 7f85c77fe700 1 -- 172.21.15.188:0/804115966 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f85c000b960 con 0x7f85d0106820 2024-04-05T21:18:44.486 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.484+0000 7f85c77fe700 1 --2- 172.21.15.188:0/804115966 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f85bc0384b0 0x7f85bc03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:44.486 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.484+0000 7f85c77fe700 1 -- 172.21.15.188:0/804115966 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f85c004c600 con 0x7f85d0106820 2024-04-05T21:18:44.486 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.484+0000 7f85d5b68700 1 --2- 172.21.15.188:0/804115966 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f85bc0384b0 0x7f85bc03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:44.486 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.484+0000 7f85d5b68700 1 --2- 172.21.15.188:0/804115966 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f85bc0384b0 0x7f85bc03a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f85cc006fd0 tx=0x7f85cc006e40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:44.490 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.489+0000 7f85c77fe700 1 -- 172.21.15.188:0/804115966 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f85c002bba0 con 0x7f85d0106820 2024-04-05T21:18:44.784 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.782+0000 7f85d85cd700 1 -- 172.21.15.188:0/804115966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85d01964a0 con 0x7f85d0106820 2024-04-05T21:18:44.784 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.783+0000 7f85c77fe700 1 -- 172.21.15.188:0/804115966 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f85c0027030 con 0x7f85d0106820 2024-04-05T21:18:44.784 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:18:44.784 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:18:44.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.785+0000 7f85d85cd700 1 -- 172.21.15.188:0/804115966 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f85bc0384b0 msgr2=0x7f85bc03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:44.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.785+0000 7f85d85cd700 1 --2- 172.21.15.188:0/804115966 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f85bc0384b0 0x7f85bc03a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f85cc006fd0 tx=0x7f85cc006e40 comp rx=0 tx=0).stop 2024-04-05T21:18:44.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.785+0000 7f85d85cd700 1 -- 172.21.15.188:0/804115966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85d0106820 msgr2=0x7f85d019ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:44.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.785+0000 7f85d85cd700 1 --2- 172.21.15.188:0/804115966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85d0106820 0x7f85d019ba20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f85c0004f40 tx=0x7f85c0005020 comp rx=0 tx=0).stop 2024-04-05T21:18:44.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.785+0000 7f85d85cd700 1 -- 172.21.15.188:0/804115966 shutdown_connections 2024-04-05T21:18:44.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.785+0000 7f85d85cd700 1 --2- 172.21.15.188:0/804115966 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f85bc0384b0 0x7f85bc03a970 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:44.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.785+0000 7f85d85cd700 1 --2- 172.21.15.188:0/804115966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85d0106820 0x7f85d019ba20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:44.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.785+0000 7f85d85cd700 1 -- 172.21.15.188:0/804115966 >> 172.21.15.188:0/804115966 conn(0x7f85d00752b0 msgr2=0x7f85d00fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:44.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.786+0000 7f85d85cd700 1 -- 172.21.15.188:0/804115966 shutdown_connections 2024-04-05T21:18:44.787 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:44.786+0000 7f85d85cd700 1 -- 172.21.15.188:0/804115966 wait complete. 2024-04-05T21:18:44.788 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:18:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:45 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:45 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:45 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:45 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:45 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi188.tgcuir", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T21:18:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:45 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi188.tgcuir", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-05T21:18:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:45 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T21:18:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:45 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:18:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:45 smithi121 ceph-mon[39723]: Deploying daemon mgr.smithi188.tgcuir on smithi188 2024-04-05T21:18:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:45 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/804115966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:18:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:45 smithi121 ceph-mon[39723]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:45.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:45 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:46.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:18:46.651 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:18:46.874 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:18:48.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:48 smithi121 ceph-mon[39723]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:49.036 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.998+0000 7f8a942cf700 1 -- 172.21.15.188:0/2092989290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a8c106820 msgr2=0x7f8a8c106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:49.036 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.998+0000 7f8a942cf700 1 --2- 172.21.15.188:0/2092989290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a8c106820 0x7f8a8c106c00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8a88009910 tx=0x7f8a88009c20 comp rx=0 tx=0).stop 2024-04-05T21:18:49.037 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.998+0000 7f8a942cf700 1 -- 172.21.15.188:0/2092989290 shutdown_connections 2024-04-05T21:18:49.037 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.998+0000 7f8a942cf700 1 --2- 172.21.15.188:0/2092989290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a8c106820 0x7f8a8c106c00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:49.037 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.998+0000 7f8a942cf700 1 -- 172.21.15.188:0/2092989290 >> 172.21.15.188:0/2092989290 conn(0x7f8a8c0752b0 msgr2=0x7f8a8c0756c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:49.037 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.998+0000 7f8a942cf700 1 -- 172.21.15.188:0/2092989290 shutdown_connections 2024-04-05T21:18:49.037 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.998+0000 7f8a942cf700 1 -- 172.21.15.188:0/2092989290 wait complete. 2024-04-05T21:18:49.037 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.999+0000 7f8a942cf700 1 Processor -- start 2024-04-05T21:18:49.037 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.999+0000 7f8a942cf700 1 -- start start 2024-04-05T21:18:49.037 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.999+0000 7f8a942cf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a8c106820 0x7f8a8c19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:49.037 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.999+0000 7f8a942cf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c19bf60 con 0x7f8a8c106820 2024-04-05T21:18:49.037 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.999+0000 7f8a9206b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a8c106820 0x7f8a8c19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:49.037 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.999+0000 7f8a9206b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a8c106820 0x7f8a8c19ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.188:42144/0 (socket says 172.21.15.188:42144) 2024-04-05T21:18:49.037 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:48.999+0000 7f8a9206b700 1 -- 172.21.15.188:0/2813188962 learned_addr learned my addr 172.21.15.188:0/2813188962 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:18:49.037 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.000+0000 7f8a9206b700 1 -- 172.21.15.188:0/2813188962 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a880095f0 con 0x7f8a8c106820 2024-04-05T21:18:49.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.000+0000 7f8a9206b700 1 --2- 172.21.15.188:0/2813188962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a8c106820 0x7f8a8c19ba20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8a88004f40 tx=0x7f8a88005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:49.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.000+0000 7f8a837fe700 1 -- 172.21.15.188:0/2813188962 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a8801d070 con 0x7f8a8c106820 2024-04-05T21:18:49.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.000+0000 7f8a837fe700 1 -- 172.21.15.188:0/2813188962 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f8a88005dc0 con 0x7f8a8c106820 2024-04-05T21:18:49.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.000+0000 7f8a837fe700 1 -- 172.21.15.188:0/2813188962 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a88022d70 con 0x7f8a8c106820 2024-04-05T21:18:49.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.000+0000 7f8a942cf700 1 -- 172.21.15.188:0/2813188962 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8c19c1c0 con 0x7f8a8c106820 2024-04-05T21:18:49.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.000+0000 7f8a942cf700 1 -- 172.21.15.188:0/2813188962 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a8c195a10 con 0x7f8a8c106820 2024-04-05T21:18:49.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.001+0000 7f8a837fe700 1 -- 172.21.15.188:0/2813188962 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 45398+0+0 (secure 0 0 0) 0x7f8a8800f460 con 0x7f8a8c106820 2024-04-05T21:18:49.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.001+0000 7f8a942cf700 1 -- 172.21.15.188:0/2813188962 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8c0623c0 con 0x7f8a8c106820 2024-04-05T21:18:49.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.001+0000 7f8a837fe700 1 --2- 172.21.15.188:0/2813188962 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f8a780384b0 0x7f8a7803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:18:49.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.001+0000 7f8a837fe700 1 -- 172.21.15.188:0/2813188962 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f8a8804c740 con 0x7f8a8c106820 2024-04-05T21:18:49.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.001+0000 7f8a9186a700 1 --2- 172.21.15.188:0/2813188962 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f8a780384b0 0x7f8a7803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:18:49.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.002+0000 7f8a9186a700 1 --2- 172.21.15.188:0/2813188962 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f8a780384b0 0x7f8a7803a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c006fd0 tx=0x7f8a7c006e40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:18:49.038 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.006+0000 7f8a837fe700 1 -- 172.21.15.188:0/2813188962 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8a88020dd0 con 0x7f8a8c106820 2024-04-05T21:18:49.293 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.292+0000 7f8a942cf700 1 -- 172.21.15.188:0/2813188962 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a8c1964a0 con 0x7f8a8c106820 2024-04-05T21:18:49.294 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:18:49.294 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:15:55.960602Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-05T21:18:49.294 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.292+0000 7f8a837fe700 1 -- 172.21.15.188:0/2813188962 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f8a88027020 con 0x7f8a8c106820 2024-04-05T21:18:49.296 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.294+0000 7f8a942cf700 1 -- 172.21.15.188:0/2813188962 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f8a780384b0 msgr2=0x7f8a7803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:49.296 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.294+0000 7f8a942cf700 1 --2- 172.21.15.188:0/2813188962 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f8a780384b0 0x7f8a7803a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c006fd0 tx=0x7f8a7c006e40 comp rx=0 tx=0).stop 2024-04-05T21:18:49.296 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.294+0000 7f8a942cf700 1 -- 172.21.15.188:0/2813188962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a8c106820 msgr2=0x7f8a8c19ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:18:49.296 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.294+0000 7f8a942cf700 1 --2- 172.21.15.188:0/2813188962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a8c106820 0x7f8a8c19ba20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8a88004f40 tx=0x7f8a88005020 comp rx=0 tx=0).stop 2024-04-05T21:18:49.296 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.294+0000 7f8a942cf700 1 -- 172.21.15.188:0/2813188962 shutdown_connections 2024-04-05T21:18:49.296 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.295+0000 7f8a942cf700 1 --2- 172.21.15.188:0/2813188962 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f8a780384b0 0x7f8a7803a970 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:49.296 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.295+0000 7f8a942cf700 1 --2- 172.21.15.188:0/2813188962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a8c106820 0x7f8a8c19ba20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:18:49.296 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.295+0000 7f8a942cf700 1 -- 172.21.15.188:0/2813188962 >> 172.21.15.188:0/2813188962 conn(0x7f8a8c0752b0 msgr2=0x7f8a8c0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:18:49.296 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.295+0000 7f8a942cf700 1 -- 172.21.15.188:0/2813188962 shutdown_connections 2024-04-05T21:18:49.297 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:18:49.295+0000 7f8a942cf700 1 -- 172.21.15.188:0/2813188962 wait complete. 2024-04-05T21:18:49.297 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 1 2024-04-05T21:18:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T21:18:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:18:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:49 smithi121 ceph-mon[39723]: Deploying daemon mon.smithi188 on smithi188 2024-04-05T21:18:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:49 smithi121 ceph-mon[39723]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:49.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:49 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/2813188962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:18:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:51 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:51.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-05T21:18:51.942 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-05T21:18:52.364 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:18:52.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:52 smithi121 ceph-mon[39723]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:54.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:53 smithi188 ceph-mon[49715]: mon.smithi188@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2024-04-05T21:18:54.321 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:54 smithi188 bash[49621]: 8ecc9636670e88f9112ed4f4a5e88ee0b153fd1b8eff968b3201b2e8ae792300 2024-04-05T21:18:54.322 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:54 smithi188 systemd[1]: Started Ceph mon.smithi188 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:18:59.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:18:59.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: mon.smithi121 calling monitor election 2024-04-05T21:18:59.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:18:59.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/2515986856' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/crt"}]: dispatch 2024-04-05T21:18:59.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:59.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:18:59.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:18:59.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:18:59.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: mon.smithi188 calling monitor election 2024-04-05T21:18:59.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:59.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:18:59.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:18:59.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: mon.smithi121 is new leader, mons smithi121,smithi188 in quorum (ranks 0,1) 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: monmap e2: 2 mons at {smithi121=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],smithi188=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: fsmap 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: osdmap e5: 0 total, 0 up, 0 in 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: mgrmap e17: smithi121.etbquh(active, since 34s) 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: Standby manager daemon smithi188.tgcuir started 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/2515986856' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: overall HEALTH_OK 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/2515986856' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/key"}]: dispatch 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/2515986856' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:59.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:18:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:59.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:18:59.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: mon.smithi121 calling monitor election 2024-04-05T21:18:59.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:18:59.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/2515986856' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/crt"}]: dispatch 2024-04-05T21:18:59.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:59.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:18:59.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:18:59.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:18:59.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: mon.smithi188 calling monitor election 2024-04-05T21:18:59.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:18:59.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:18:59.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:18:59.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:18:59.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: mon.smithi121 is new leader, mons smithi121,smithi188 in quorum (ranks 0,1) 2024-04-05T21:18:59.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: monmap e2: 2 mons at {smithi121=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],smithi188=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T21:18:59.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: fsmap 2024-04-05T21:18:59.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: osdmap e5: 0 total, 0 up, 0 in 2024-04-05T21:18:59.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: mgrmap e17: smithi121.etbquh(active, since 34s) 2024-04-05T21:18:59.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: Standby manager daemon smithi188.tgcuir started 2024-04-05T21:18:59.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/2515986856' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T21:18:59.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: overall HEALTH_OK 2024-04-05T21:18:59.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/2515986856' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/key"}]: dispatch 2024-04-05T21:18:59.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/2515986856' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T21:18:59.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:59.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:59.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:59.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:18:59.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:18:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:00 smithi188 ceph-mon[49715]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:00 smithi188 ceph-mon[49715]: mgrmap e18: smithi121.etbquh(active, since 34s), standbys: smithi188.tgcuir 2024-04-05T21:19:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:00 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr metadata", "who": "smithi188.tgcuir", "id": "smithi188.tgcuir"}]: dispatch 2024-04-05T21:19:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:00 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:19:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:00 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:19:00.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:00 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:00 smithi121 ceph-mon[39723]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:00 smithi121 ceph-mon[39723]: mgrmap e18: smithi121.etbquh(active, since 34s), standbys: smithi188.tgcuir 2024-04-05T21:19:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:00 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr metadata", "who": "smithi188.tgcuir", "id": "smithi188.tgcuir"}]: dispatch 2024-04-05T21:19:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:00 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:19:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:00 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:19:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:00 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:00.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.574+0000 7f94c9506700 1 -- 172.21.15.188:0/2939517212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94ac0062a0 msgr2=0x7f94ac008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:19:00.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.574+0000 7f94c9506700 1 --2- 172.21.15.188:0/2939517212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94ac0062a0 0x7f94ac008710 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f94c4069500 tx=0x7f94b400fe90 comp rx=0 tx=0).stop 2024-04-05T21:19:00.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.574+0000 7f94c9506700 1 -- 172.21.15.188:0/2939517212 shutdown_connections 2024-04-05T21:19:00.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.574+0000 7f94c9506700 1 --2- 172.21.15.188:0/2939517212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94ac0062a0 0x7f94ac008710 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:00.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.574+0000 7f94c9506700 1 -- 172.21.15.188:0/2939517212 >> 172.21.15.188:0/2939517212 conn(0x7f94c4075340 msgr2=0x7f94c4075750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:19:00.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.574+0000 7f94c9506700 1 -- 172.21.15.188:0/2939517212 shutdown_connections 2024-04-05T21:19:00.576 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.574+0000 7f94c9506700 1 -- 172.21.15.188:0/2939517212 wait complete. 2024-04-05T21:19:00.579 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.575+0000 7f94c9506700 1 Processor -- start 2024-04-05T21:19:00.579 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.575+0000 7f94c9506700 1 -- start start 2024-04-05T21:19:00.579 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c9506700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac0062a0 0x7f94c4195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:19:00.579 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c9506700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94c41964a0 0x7f94c419a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:19:00.579 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c9506700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c4196ac0 con 0x7f94c41964a0 2024-04-05T21:19:00.579 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c9506700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c4196c30 con 0x7f94ac0062a0 2024-04-05T21:19:00.579 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c2ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac0062a0 0x7f94c4195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:19:00.579 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c2ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac0062a0 0x7f94c4195f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:53524/0 (socket says 172.21.15.188:53524) 2024-04-05T21:19:00.579 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c2ffd700 1 -- 172.21.15.188:0/1630900549 learned_addr learned my addr 172.21.15.188:0/1630900549 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:19:00.579 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c27fc700 1 --2- 172.21.15.188:0/1630900549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94c41964a0 0x7f94c419a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:19:00.579 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c2ffd700 1 -- 172.21.15.188:0/1630900549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac0062a0 msgr2=0x7f94c4195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-05T21:19:00.579 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c2ffd700 1 -- 172.21.15.188:0/1630900549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac0062a0 msgr2=0x7f94c4195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-05T21:19:00.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c2ffd700 1 --2- 172.21.15.188:0/1630900549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac0062a0 0x7f94c4195f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-05T21:19:00.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c2ffd700 1 --2- 172.21.15.188:0/1630900549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac0062a0 0x7f94c4195f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T21:19:00.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c27fc700 1 -- 172.21.15.188:0/1630900549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac0062a0 msgr2=0x7f94c4195f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T21:19:00.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c27fc700 1 --2- 172.21.15.188:0/1630900549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac0062a0 0x7f94c4195f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:00.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c27fc700 1 -- 172.21.15.188:0/1630900549 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b400f8e0 con 0x7f94c41964a0 2024-04-05T21:19:00.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94c27fc700 1 --2- 172.21.15.188:0/1630900549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94c41964a0 0x7f94c419a910 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f94b800b4d0 tx=0x7f94b800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:19:00.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.576+0000 7f94abfff700 1 -- 172.21.15.188:0/1630900549 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94b8011800 con 0x7f94c41964a0 2024-04-05T21:19:00.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.577+0000 7f94c9506700 1 -- 172.21.15.188:0/1630900549 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c419af10 con 0x7f94c41964a0 2024-04-05T21:19:00.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.577+0000 7f94abfff700 1 -- 172.21.15.188:0/1630900549 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f94b8011df0 con 0x7f94c41964a0 2024-04-05T21:19:00.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.577+0000 7f94abfff700 1 -- 172.21.15.188:0/1630900549 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94b800f2e0 con 0x7f94c41964a0 2024-04-05T21:19:00.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.577+0000 7f94c9506700 1 -- 172.21.15.188:0/1630900549 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c419b460 con 0x7f94c41964a0 2024-04-05T21:19:00.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.577+0000 7f94c9506700 1 -- 172.21.15.188:0/1630900549 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c419b730 con 0x7f94c41964a0 2024-04-05T21:19:00.581 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.578+0000 7f94abfff700 1 -- 172.21.15.188:0/1630900549 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f94b8010380 con 0x7f94c41964a0 2024-04-05T21:19:00.581 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.578+0000 7f94abfff700 1 --2- 172.21.15.188:0/1630900549 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f94b006c560 0x7f94b006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:19:00.581 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.578+0000 7f94abfff700 1 -- 172.21.15.188:0/1630900549 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f94b808a2d0 con 0x7f94c41964a0 2024-04-05T21:19:00.581 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.579+0000 7f94c2ffd700 1 --2- 172.21.15.188:0/1630900549 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f94b006c560 0x7f94b006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:19:00.581 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.579+0000 7f94c2ffd700 1 --2- 172.21.15.188:0/1630900549 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f94b006c560 0x7f94b006ea20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f94b4001eb0 tx=0x7f94b4012a30 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:19:00.586 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.584+0000 7f94abfff700 1 -- 172.21.15.188:0/1630900549 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f94b8055b30 con 0x7f94c41964a0 2024-04-05T21:19:00.860 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.859+0000 7f94c9506700 1 -- 172.21.15.188:0/1630900549 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94c41a0bb0 con 0x7f94c41964a0 2024-04-05T21:19:00.861 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.860+0000 7f94abfff700 1 -- 172.21.15.188:0/1630900549 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1026 (secure 0 0 0) 0x7f94b8015420 con 0x7f94c41964a0 2024-04-05T21:19:00.861 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-05T21:19:00.861 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":2,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","modified":"2024-04-05T21:18:53.993220Z","created":"2024-04-05T21:15:55.960602Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi188","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:3300","nonce":0},{"type":"v1","addr":"172.21.15.188:6789","nonce":0}]},"addr":"172.21.15.188:6789/0","public_addr":"172.21.15.188:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-05T21:19:00.863 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.862+0000 7f94c9506700 1 -- 172.21.15.188:0/1630900549 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f94b006c560 msgr2=0x7f94b006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:19:00.863 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.862+0000 7f94c9506700 1 --2- 172.21.15.188:0/1630900549 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f94b006c560 0x7f94b006ea20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f94b4001eb0 tx=0x7f94b4012a30 comp rx=0 tx=0).stop 2024-04-05T21:19:00.864 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.862+0000 7f94c9506700 1 -- 172.21.15.188:0/1630900549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94c41964a0 msgr2=0x7f94c419a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:19:00.864 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.862+0000 7f94c9506700 1 --2- 172.21.15.188:0/1630900549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94c41964a0 0x7f94c419a910 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f94b800b4d0 tx=0x7f94b800b890 comp rx=0 tx=0).stop 2024-04-05T21:19:00.864 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.862+0000 7f94c9506700 1 -- 172.21.15.188:0/1630900549 shutdown_connections 2024-04-05T21:19:00.864 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.862+0000 7f94c9506700 1 --2- 172.21.15.188:0/1630900549 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f94b006c560 0x7f94b006ea20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:00.864 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.862+0000 7f94c9506700 1 --2- 172.21.15.188:0/1630900549 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94ac0062a0 0x7f94c4195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:00.864 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.862+0000 7f94c9506700 1 --2- 172.21.15.188:0/1630900549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94c41964a0 0x7f94c419a910 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:00.864 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.862+0000 7f94c9506700 1 -- 172.21.15.188:0/1630900549 >> 172.21.15.188:0/1630900549 conn(0x7f94c4075340 msgr2=0x7f94c4102460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:19:00.864 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.862+0000 7f94c9506700 1 -- 172.21.15.188:0/1630900549 shutdown_connections 2024-04-05T21:19:00.865 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:19:00.862+0000 7f94c9506700 1 -- 172.21.15.188:0/1630900549 wait complete. 2024-04-05T21:19:00.865 INFO:teuthology.orchestra.run.smithi188.stderr:dumped monmap epoch 2 2024-04-05T21:19:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:01 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/1630900549' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:19:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:01 smithi188 ceph-mon[49715]: from='client.? 172.21.15.188:0/1630900549' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-05T21:19:01.605 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-05T21:19:01.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-05T21:19:01.839 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:19:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:02 smithi121 ceph-mon[39723]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:02 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:02 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:02 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:02 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:19:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:02 smithi188 ceph-mon[49715]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:02 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:02 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:02 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:02 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:19:03.463 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:03 smithi121 ceph-mon[39723]: Updating smithi121:/etc/ceph/ceph.conf 2024-04-05T21:19:03.463 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:03 smithi121 ceph-mon[39723]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-05T21:19:03.463 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:03 smithi121 ceph-mon[39723]: Updating smithi188:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:19:03.463 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:03 smithi121 ceph-mon[39723]: Updating smithi121:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:19:03.463 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:03 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:03.463 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:03 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:03.463 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:03 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:03.463 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:03 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:03.463 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:03 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:03.463 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:03 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T21:19:03.463 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:03 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T21:19:03.463 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:03 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:03.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:03 smithi188 ceph-mon[49715]: Updating smithi121:/etc/ceph/ceph.conf 2024-04-05T21:19:03.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:03 smithi188 ceph-mon[49715]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-05T21:19:03.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:03 smithi188 ceph-mon[49715]: Updating smithi188:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:19:03.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:03 smithi188 ceph-mon[49715]: Updating smithi121:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:19:03.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:03 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:03.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:03 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:03.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:03 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:03.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:03 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:03.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:03 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:03.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:03 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T21:19:03.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:03 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T21:19:03.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:03 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:04.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.926+0000 7f7320caa700 1 -- 172.21.15.121:0/1623773535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f731c106810 msgr2=0x7f731c106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:19:04.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.926+0000 7f7320caa700 1 --2- 172.21.15.121:0/1623773535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f731c106810 0x7f731c106bf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7310009910 tx=0x7f7310009c20 comp rx=0 tx=0).stop 2024-04-05T21:19:04.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.926+0000 7f7320caa700 1 -- 172.21.15.121:0/1623773535 shutdown_connections 2024-04-05T21:19:04.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.926+0000 7f7320caa700 1 --2- 172.21.15.121:0/1623773535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f731c106810 0x7f731c106bf0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:04.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.926+0000 7f7320caa700 1 -- 172.21.15.121:0/1623773535 >> 172.21.15.121:0/1623773535 conn(0x7f731c0fc320 msgr2=0x7f731c0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:19:04.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.926+0000 7f7320caa700 1 -- 172.21.15.121:0/1623773535 shutdown_connections 2024-04-05T21:19:04.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.926+0000 7f7320caa700 1 -- 172.21.15.121:0/1623773535 wait complete. 2024-04-05T21:19:04.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.927+0000 7f7320caa700 1 Processor -- start 2024-04-05T21:19:04.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.927+0000 7f7320caa700 1 -- start start 2024-04-05T21:19:04.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.927+0000 7f7320caa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f731c106810 0x7f731c195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:19:04.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.927+0000 7f7320caa700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f731c196250 0x7f731c19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:19:04.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.927+0000 7f7320caa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c196870 con 0x7f731c106810 2024-04-05T21:19:04.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.927+0000 7f7320caa700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c1969e0 con 0x7f731c196250 2024-04-05T21:19:04.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.927+0000 7f731a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f731c106810 0x7f731c195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:19:04.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.927+0000 7f731a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f731c106810 0x7f731c195d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51482/0 (socket says 172.21.15.121:51482) 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.927+0000 7f731a59c700 1 -- 172.21.15.121:0/1757988313 learned_addr learned my addr 172.21.15.121:0/1757988313 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.927+0000 7f731a59c700 1 -- 172.21.15.121:0/1757988313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f731c196250 msgr2=0x7f731c19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.927+0000 7f731a59c700 1 --2- 172.21.15.121:0/1757988313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f731c196250 0x7f731c19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.927+0000 7f731a59c700 1 -- 172.21.15.121:0/1757988313 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73100095f0 con 0x7f731c106810 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.928+0000 7f731a59c700 1 --2- 172.21.15.121:0/1757988313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f731c106810 0x7f731c195d10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7310005b40 tx=0x7f7310004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.928+0000 7f730b7fe700 1 -- 172.21.15.121:0/1757988313 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f731001d070 con 0x7f731c106810 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.928+0000 7f7320caa700 1 -- 172.21.15.121:0/1757988313 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c19acc0 con 0x7f731c106810 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.928+0000 7f730b7fe700 1 -- 172.21.15.121:0/1757988313 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f7310005dc0 con 0x7f731c106810 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.928+0000 7f730b7fe700 1 -- 172.21.15.121:0/1757988313 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f731000f460 con 0x7f731c106810 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.928+0000 7f7320caa700 1 -- 172.21.15.121:0/1757988313 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731c19b210 con 0x7f731c106810 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.929+0000 7f7320caa700 1 -- 172.21.15.121:0/1757988313 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731c0623c0 con 0x7f731c106810 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.929+0000 7f730b7fe700 1 -- 172.21.15.121:0/1757988313 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f7310022470 con 0x7f731c106810 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.929+0000 7f730b7fe700 1 --2- 172.21.15.121:0/1757988313 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f730406c5c0 0x7f730406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:19:04.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.930+0000 7f730b7fe700 1 -- 172.21.15.121:0/1757988313 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f731008be30 con 0x7f731c106810 2024-04-05T21:19:04.270 INFO:teuthology.orchestra.run.smithi121.stdout:# minimal ceph.conf for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:19:04.270 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-04-05T21:19:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:19:04.270 INFO:teuthology.orchestra.run.smithi121.stdout: mon_host = [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] 2024-04-05T21:19:04.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.930+0000 7f7319d9b700 1 --2- 172.21.15.121:0/1757988313 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f730406c5c0 0x7f730406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:19:04.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.930+0000 7f7319d9b700 1 --2- 172.21.15.121:0/1757988313 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f730406c5c0 0x7f730406ea80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f730c0098e0 tx=0x7f730c006dd0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:19:04.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:03.934+0000 7f730b7fe700 1 -- 172.21.15.121:0/1757988313 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f73100577a0 con 0x7f731c106810 2024-04-05T21:19:04.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:04.132+0000 7f7320caa700 1 -- 172.21.15.121:0/1757988313 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f731c1a0960 con 0x7f731c106810 2024-04-05T21:19:04.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:04.133+0000 7f730b7fe700 1 -- 172.21.15.121:0/1757988313 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f7310027100 con 0x7f731c106810 2024-04-05T21:19:04.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:04.135+0000 7f7320caa700 1 -- 172.21.15.121:0/1757988313 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f730406c5c0 msgr2=0x7f730406ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:19:04.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:04.135+0000 7f7320caa700 1 --2- 172.21.15.121:0/1757988313 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f730406c5c0 0x7f730406ea80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f730c0098e0 tx=0x7f730c006dd0 comp rx=0 tx=0).stop 2024-04-05T21:19:04.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:04.135+0000 7f7320caa700 1 -- 172.21.15.121:0/1757988313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f731c106810 msgr2=0x7f731c195d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:19:04.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:04.135+0000 7f7320caa700 1 --2- 172.21.15.121:0/1757988313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f731c106810 0x7f731c195d10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7310005b40 tx=0x7f7310004d60 comp rx=0 tx=0).stop 2024-04-05T21:19:04.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:04.135+0000 7f7320caa700 1 -- 172.21.15.121:0/1757988313 shutdown_connections 2024-04-05T21:19:04.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:04.135+0000 7f7320caa700 1 --2- 172.21.15.121:0/1757988313 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f730406c5c0 0x7f730406ea80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:04.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:04.135+0000 7f7320caa700 1 --2- 172.21.15.121:0/1757988313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f731c106810 0x7f731c195d10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:04.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:04.135+0000 7f7320caa700 1 --2- 172.21.15.121:0/1757988313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f731c196250 0x7f731c19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:04.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:04.135+0000 7f7320caa700 1 -- 172.21.15.121:0/1757988313 >> 172.21.15.121:0/1757988313 conn(0x7f731c0fc320 msgr2=0x7f731c102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:19:04.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:04.135+0000 7f7320caa700 1 -- 172.21.15.121:0/1757988313 shutdown_connections 2024-04-05T21:19:04.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:04.135+0000 7f7320caa700 1 -- 172.21.15.121:0/1757988313 wait complete. 2024-04-05T21:19:04.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:04 smithi121 ceph-mon[39723]: Reconfiguring mon.smithi121 (unknown last config time)... 2024-04-05T21:19:04.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:04 smithi121 ceph-mon[39723]: Reconfiguring daemon mon.smithi121 on smithi121 2024-04-05T21:19:04.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:04 smithi121 ceph-mon[39723]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:04.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:04 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1757988313' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:04 smithi188 ceph-mon[49715]: Reconfiguring mon.smithi121 (unknown last config time)... 2024-04-05T21:19:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:04 smithi188 ceph-mon[49715]: Reconfiguring daemon mon.smithi121 on smithi121 2024-04-05T21:19:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:04 smithi188 ceph-mon[49715]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:04 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1757988313' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:05.383 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-05T21:19:05.384 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:19:05.384 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.conf 2024-04-05T21:19:05.414 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:19:05.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-05T21:19:05.480 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-05T21:19:05.481 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.conf 2024-04-05T21:19:05.519 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-05T21:19:05.519 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-05T21:19:05.590 INFO:tasks.cephadm:Deploying OSDs... 2024-04-05T21:19:05.590 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:19:05.590 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-04-05T21:19:05.608 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-05T21:19:05.608 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-04-05T21:19:05.665 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-05T21:19:05.665 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T21:19:05.665 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 164631 Links: 1 2024-04-05T21:19:05.665 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T21:19:05.666 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T21:19:05.666 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-05 21:18:16.515796314 +0000 2024-04-05T21:19:05.666 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-05 21:11:33.340937380 +0000 2024-04-05T21:19:05.666 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-05 21:11:33.340937380 +0000 2024-04-05T21:19:05.666 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2024-04-05T21:19:05.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-05T21:19:05.733 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-05T21:19:05.733 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-05T21:19:05.733 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000255631 s, 2.0 MB/s 2024-04-05T21:19:05.734 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-05T21:19:05.794 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-04-05T21:19:05.852 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-05T21:19:05.852 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T21:19:05.852 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 168371 Links: 1 2024-04-05T21:19:05.852 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T21:19:05.852 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T21:19:05.852 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-05 21:18:16.515796314 +0000 2024-04-05T21:19:05.852 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-05 21:11:33.687932973 +0000 2024-04-05T21:19:05.852 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-05 21:11:33.687932973 +0000 2024-04-05T21:19:05.852 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2024-04-05T21:19:05.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-05T21:19:05.917 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-05T21:19:05.917 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-05T21:19:05.918 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000218844 s, 2.3 MB/s 2024-04-05T21:19:05.918 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-05T21:19:05.969 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:05 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:05.969 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:05 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:05.969 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:05 smithi121 ceph-mon[39723]: Reconfiguring mgr.smithi121.etbquh (unknown last config time)... 2024-04-05T21:19:05.969 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:05 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi121.etbquh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T21:19:05.969 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:05 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T21:19:05.969 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:05 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:05.970 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:05 smithi121 ceph-mon[39723]: Reconfiguring daemon mgr.smithi121.etbquh on smithi121 2024-04-05T21:19:05.970 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:05 smithi121 ceph-mon[39723]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:05.977 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-04-05T21:19:06.035 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-05T21:19:06.035 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T21:19:06.035 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 161520 Links: 1 2024-04-05T21:19:06.035 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T21:19:06.036 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T21:19:06.036 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-05 21:18:16.515796314 +0000 2024-04-05T21:19:06.036 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-05 21:11:34.026928668 +0000 2024-04-05T21:19:06.036 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-05 21:11:34.026928668 +0000 2024-04-05T21:19:06.036 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2024-04-05T21:19:06.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-05T21:19:06.101 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-05T21:19:06.101 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-05T21:19:06.102 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000164478 s, 3.1 MB/s 2024-04-05T21:19:06.102 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-05T21:19:06.161 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-04-05T21:19:06.219 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-05T21:19:06.220 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T21:19:06.220 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 161563 Links: 1 2024-04-05T21:19:06.220 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T21:19:06.220 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T21:19:06.220 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-05 21:18:16.515796314 +0000 2024-04-05T21:19:06.220 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-05 21:11:34.363924388 +0000 2024-04-05T21:19:06.220 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-05 21:11:34.363924388 +0000 2024-04-05T21:19:06.220 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2024-04-05T21:19:06.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-05T21:19:06.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:05 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:06.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:05 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:06.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:05 smithi188 ceph-mon[49715]: Reconfiguring mgr.smithi121.etbquh (unknown last config time)... 2024-04-05T21:19:06.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:05 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi121.etbquh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T21:19:06.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:05 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T21:19:06.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:05 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:06.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:05 smithi188 ceph-mon[49715]: Reconfiguring daemon mgr.smithi121.etbquh on smithi121 2024-04-05T21:19:06.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:05 smithi188 ceph-mon[49715]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:06.285 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-05T21:19:06.285 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-05T21:19:06.285 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000233369 s, 2.2 MB/s 2024-04-05T21:19:06.286 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-05T21:19:06.344 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-05T21:19:06.344 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2024-04-05T21:19:06.367 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-05T21:19:06.367 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2024-04-05T21:19:06.428 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-05T21:19:06.428 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T21:19:06.428 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 166829 Links: 1 2024-04-05T21:19:06.428 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T21:19:06.428 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T21:19:06.428 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-05 21:18:52.540572327 +0000 2024-04-05T21:19:06.428 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-05 21:11:34.215699228 +0000 2024-04-05T21:19:06.428 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-05 21:11:34.215699228 +0000 2024-04-05T21:19:06.428 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-04-05T21:19:06.429 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-05T21:19:06.497 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-05T21:19:06.497 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-05T21:19:06.497 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000208616 s, 2.5 MB/s 2024-04-05T21:19:06.498 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-05T21:19:06.558 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2024-04-05T21:19:06.618 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-05T21:19:06.619 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T21:19:06.619 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 165323 Links: 1 2024-04-05T21:19:06.619 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T21:19:06.619 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T21:19:06.619 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-05 21:18:52.540572327 +0000 2024-04-05T21:19:06.619 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-05 21:11:34.563692989 +0000 2024-04-05T21:19:06.619 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-05 21:11:34.563692989 +0000 2024-04-05T21:19:06.619 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-04-05T21:19:06.619 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-05T21:19:06.686 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-05T21:19:06.687 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-05T21:19:06.687 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.0002218 s, 2.3 MB/s 2024-04-05T21:19:06.688 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-05T21:19:06.748 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2024-04-05T21:19:06.809 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-05T21:19:06.809 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T21:19:06.809 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 164658 Links: 1 2024-04-05T21:19:06.809 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T21:19:06.809 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T21:19:06.809 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-05 21:18:52.540572327 +0000 2024-04-05T21:19:06.809 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-05 21:11:34.915686678 +0000 2024-04-05T21:19:06.809 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-05 21:11:34.915686678 +0000 2024-04-05T21:19:06.809 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-04-05T21:19:06.809 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-05T21:19:06.877 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-05T21:19:06.877 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-05T21:19:06.877 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000172917 s, 3.0 MB/s 2024-04-05T21:19:06.878 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-05T21:19:06.938 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2024-04-05T21:19:06.997 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-05T21:19:06.997 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-05T21:19:06.997 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 170196 Links: 1 2024-04-05T21:19:06.997 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T21:19:06.997 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-05T21:19:06.997 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-05 21:18:52.540572327 +0000 2024-04-05T21:19:06.998 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-05 21:11:35.267680367 +0000 2024-04-05T21:19:06.998 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-05 21:11:35.267680367 +0000 2024-04-05T21:19:06.998 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-04-05T21:19:06.998 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-05T21:19:07.065 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-05T21:19:07.065 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-05T21:19:07.065 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000216413 s, 2.4 MB/s 2024-04-05T21:19:07.066 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-05T21:19:07.127 INFO:tasks.cephadm:Deploying osd.0 on smithi121 with /dev/vg_nvme/lv_4... 2024-04-05T21:19:07.128 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-05T21:19:07.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:07 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:07.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:07 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:07.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:07 smithi121 ceph-mon[39723]: Reconfiguring ceph-exporter.smithi121 (monmap changed)... 2024-04-05T21:19:07.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:07 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi121", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-05T21:19:07.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:07 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:07.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:07 smithi121 ceph-mon[39723]: Reconfiguring daemon ceph-exporter.smithi121 on smithi121 2024-04-05T21:19:07.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:07 smithi121 ceph-mon[39723]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:08.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:07 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:08.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:07 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:08.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:07 smithi188 ceph-mon[49715]: Reconfiguring ceph-exporter.smithi121 (monmap changed)... 2024-04-05T21:19:08.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:07 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi121", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-05T21:19:08.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:07 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:08.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:07 smithi188 ceph-mon[49715]: Reconfiguring daemon ceph-exporter.smithi121 on smithi121 2024-04-05T21:19:08.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:07 smithi188 ceph-mon[49715]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:08.476 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:19:09.654 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:09.654 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:09.654 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:09 smithi121 ceph-mon[39723]: Reconfiguring crash.smithi121 (monmap changed)... 2024-04-05T21:19:09.654 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi121", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T21:19:09.654 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:09.654 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:09 smithi121 ceph-mon[39723]: Reconfiguring daemon crash.smithi121 on smithi121 2024-04-05T21:19:09.655 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:09 smithi121 ceph-mon[39723]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:09 smithi188 ceph-mon[49715]: Reconfiguring crash.smithi121 (monmap changed)... 2024-04-05T21:19:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi121", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T21:19:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:09 smithi188 ceph-mon[49715]: Reconfiguring daemon crash.smithi121 on smithi121 2024-04-05T21:19:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:09 smithi188 ceph-mon[49715]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:10.687 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:10 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:19:10.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:10 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:19:11.961 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-05T21:19:11.961 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-05T21:19:11.961 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-04-05T21:19:11.961 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-04-05T21:19:11.962 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110348 s, 950 MB/s 2024-04-05T21:19:11.962 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-04-05T21:19:11.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:11 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:11.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:11 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:11.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:11 smithi121 ceph-mon[39723]: Reconfiguring alertmanager.smithi121 (dependencies changed)... 2024-04-05T21:19:11.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:11 smithi121 ceph-mon[39723]: Reconfiguring daemon alertmanager.smithi121 on smithi121 2024-04-05T21:19:11.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:11 smithi121 ceph-mon[39723]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:11 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:11 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:11 smithi188 ceph-mon[49715]: Reconfiguring alertmanager.smithi121 (dependencies changed)... 2024-04-05T21:19:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:11 smithi188 ceph-mon[49715]: Reconfiguring daemon alertmanager.smithi121 on smithi121 2024-04-05T21:19:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:11 smithi188 ceph-mon[49715]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:12.876 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi121:vg_nvme/lv_4 2024-04-05T21:19:14.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:14 smithi188 ceph-mon[49715]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:14.321 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:19:14.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:14 smithi121 ceph-mon[39723]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:16.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:16 smithi121 ceph-mon[39723]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:16.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:16 smithi188 ceph-mon[49715]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:17.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.686+0000 7ff3b36ca700 1 -- 172.21.15.121:0/329370699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3ac101e00 msgr2=0x7ff3ac102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:19:17.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.686+0000 7ff3b36ca700 1 --2- 172.21.15.121:0/329370699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3ac101e00 0x7ff3ac102260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff39c009910 tx=0x7ff39c009c20 comp rx=0 tx=0).stop 2024-04-05T21:19:17.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.686+0000 7ff3b36ca700 1 -- 172.21.15.121:0/329370699 shutdown_connections 2024-04-05T21:19:17.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.686+0000 7ff3b36ca700 1 --2- 172.21.15.121:0/329370699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3ac101e00 0x7ff3ac102260 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:17.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.686+0000 7ff3b36ca700 1 --2- 172.21.15.121:0/329370699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3ac100c00 0x7ff3ac101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:17.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.686+0000 7ff3b36ca700 1 -- 172.21.15.121:0/329370699 >> 172.21.15.121:0/329370699 conn(0x7ff3ac0fc180 msgr2=0x7ff3ac0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:19:17.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.686+0000 7ff3b36ca700 1 -- 172.21.15.121:0/329370699 shutdown_connections 2024-04-05T21:19:17.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.686+0000 7ff3b36ca700 1 -- 172.21.15.121:0/329370699 wait complete. 2024-04-05T21:19:17.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b36ca700 1 Processor -- start 2024-04-05T21:19:17.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b36ca700 1 -- start start 2024-04-05T21:19:17.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b36ca700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3ac100c00 0x7ff3ac074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:19:17.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b36ca700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3ac101e00 0x7ff3ac073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:19:17.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b36ca700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac0736c0 con 0x7ff3ac101e00 2024-04-05T21:19:17.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b36ca700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac073800 con 0x7ff3ac100c00 2024-04-05T21:19:17.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b0c65700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3ac101e00 0x7ff3ac073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:19:17.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b0c65700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3ac101e00 0x7ff3ac073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49974/0 (socket says 172.21.15.121:49974) 2024-04-05T21:19:17.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b0c65700 1 -- 172.21.15.121:0/3877106655 learned_addr learned my addr 172.21.15.121:0/3877106655 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:19:17.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b1466700 1 --2- 172.21.15.121:0/3877106655 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3ac100c00 0x7ff3ac074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:19:17.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b0c65700 1 -- 172.21.15.121:0/3877106655 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3ac100c00 msgr2=0x7ff3ac074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:19:17.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b0c65700 1 --2- 172.21.15.121:0/3877106655 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3ac100c00 0x7ff3ac074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:17.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b0c65700 1 -- 172.21.15.121:0/3877106655 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39c0095f0 con 0x7ff3ac101e00 2024-04-05T21:19:17.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3b0c65700 1 --2- 172.21.15.121:0/3877106655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3ac101e00 0x7ff3ac073180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff39c006010 tx=0x7ff39c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:19:17.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.687+0000 7ff3a27fc700 1 -- 172.21.15.121:0/3877106655 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff39c01d070 con 0x7ff3ac101e00 2024-04-05T21:19:17.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.688+0000 7ff3a27fc700 1 -- 172.21.15.121:0/3877106655 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7ff39c00f460 con 0x7ff3ac101e00 2024-04-05T21:19:17.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.688+0000 7ff3b36ca700 1 -- 172.21.15.121:0/3877106655 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ac073a80 con 0x7ff3ac101e00 2024-04-05T21:19:17.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.688+0000 7ff3a27fc700 1 -- 172.21.15.121:0/3877106655 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff39c021640 con 0x7ff3ac101e00 2024-04-05T21:19:17.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.688+0000 7ff3b36ca700 1 -- 172.21.15.121:0/3877106655 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ac073f70 con 0x7ff3ac101e00 2024-04-05T21:19:17.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.688+0000 7ff3b36ca700 1 -- 172.21.15.121:0/3877106655 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ac109620 con 0x7ff3ac101e00 2024-04-05T21:19:17.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.689+0000 7ff3a27fc700 1 -- 172.21.15.121:0/3877106655 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7ff39c00f9b0 con 0x7ff3ac101e00 2024-04-05T21:19:17.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.689+0000 7ff3a27fc700 1 --2- 172.21.15.121:0/3877106655 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff39806c460 0x7ff39806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:19:17.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.689+0000 7ff3a27fc700 1 -- 172.21.15.121:0/3877106655 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7ff39c08b890 con 0x7ff3ac101e00 2024-04-05T21:19:17.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.689+0000 7ff3b1466700 1 --2- 172.21.15.121:0/3877106655 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff39806c460 0x7ff39806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:19:17.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.690+0000 7ff3b1466700 1 --2- 172.21.15.121:0/3877106655 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff39806c460 0x7ff39806e920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac101c60 tx=0x7ff3a400b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:19:17.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.694+0000 7ff3a27fc700 1 -- 172.21.15.121:0/3877106655 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff39c05ac60 con 0x7ff3ac101e00 2024-04-05T21:19:17.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:17.906+0000 7ff3b36ca700 1 -- 172.21.15.121:0/3877106655 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3ac106750 con 0x7ff39806c460 2024-04-05T21:19:18.446 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:18 smithi121 ceph-mon[39723]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:18.446 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T21:19:18.446 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T21:19:18.446 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:18.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:18 smithi188 ceph-mon[49715]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:18.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T21:19:18.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T21:19:18.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:19.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:19 smithi121 ceph-mon[39723]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:19:19.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:19 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:19.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:19 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:19.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:19 smithi188 ceph-mon[49715]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:19:19.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:19 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:19.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:19 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:20.446 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:20 smithi121 ceph-mon[39723]: Reconfiguring grafana.smithi121 (dependencies changed)... 2024-04-05T21:19:20.446 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:20 smithi121 ceph-mon[39723]: Reconfiguring daemon grafana.smithi121 on smithi121 2024-04-05T21:19:20.446 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:20 smithi121 ceph-mon[39723]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:20.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:20 smithi188 ceph-mon[49715]: Reconfiguring grafana.smithi121 (dependencies changed)... 2024-04-05T21:19:20.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:20 smithi188 ceph-mon[49715]: Reconfiguring daemon grafana.smithi121 on smithi121 2024-04-05T21:19:20.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:20 smithi188 ceph-mon[49715]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:22.409 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:22 smithi121 ceph-mon[39723]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:22.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:22 smithi188 ceph-mon[49715]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:24.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:24 smithi121 ceph-mon[39723]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:24.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:24 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:24.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:24 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:24.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:24 smithi188 ceph-mon[49715]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:24.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:24 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:24.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:24 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:25.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:25 smithi121 ceph-mon[39723]: Reconfiguring prometheus.smithi121 (dependencies changed)... 2024-04-05T21:19:25.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:19:25.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:25 smithi188 ceph-mon[49715]: Reconfiguring prometheus.smithi121 (dependencies changed)... 2024-04-05T21:19:25.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:25 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:19:26.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:26 smithi188 ceph-mon[49715]: Reconfiguring daemon prometheus.smithi121 on smithi121 2024-04-05T21:19:26.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:26 smithi188 ceph-mon[49715]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:26.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:26 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2206914651' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "133652f8-9fe2-45e9-90b6-36f89191114b"}]: dispatch 2024-04-05T21:19:26.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:26 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2206914651' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "133652f8-9fe2-45e9-90b6-36f89191114b"}]': finished 2024-04-05T21:19:26.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:26 smithi188 ceph-mon[49715]: osdmap e6: 1 total, 0 up, 1 in 2024-04-05T21:19:26.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:26 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:19:26.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:26 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/587568580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T21:19:26.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:26 smithi121 ceph-mon[39723]: Reconfiguring daemon prometheus.smithi121 on smithi121 2024-04-05T21:19:26.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:26 smithi121 ceph-mon[39723]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:26.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:26 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2206914651' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "133652f8-9fe2-45e9-90b6-36f89191114b"}]: dispatch 2024-04-05T21:19:26.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:26 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2206914651' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "133652f8-9fe2-45e9-90b6-36f89191114b"}]': finished 2024-04-05T21:19:26.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:26 smithi121 ceph-mon[39723]: osdmap e6: 1 total, 0 up, 1 in 2024-04-05T21:19:26.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:26 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:19:26.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:26 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/587568580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T21:19:28.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:28 smithi188 ceph-mon[49715]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:28 smithi121 ceph-mon[39723]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:30.525 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:30 smithi121 ceph-mon[39723]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:30.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:30 smithi188 ceph-mon[49715]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:32 smithi121 ceph-mon[39723]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:32.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:32 smithi188 ceph-mon[49715]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:34.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:34 smithi121 ceph-mon[39723]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:34.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:34 smithi188 ceph-mon[49715]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:36.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:36 smithi188 ceph-mon[49715]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:36.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:36 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:36.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:36 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:36.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:36 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi188", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-05T21:19:36.596 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:36 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:36.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:36 smithi121 ceph-mon[39723]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:36.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:36 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:36.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:36 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:36.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:36 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi188", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-05T21:19:36.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:36 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:37.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:37 smithi121 ceph-mon[39723]: Reconfiguring ceph-exporter.smithi188 (monmap changed)... 2024-04-05T21:19:37.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:37 smithi121 ceph-mon[39723]: Reconfiguring daemon ceph-exporter.smithi188 on smithi188 2024-04-05T21:19:37.640 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:37 smithi188 ceph-mon[49715]: Reconfiguring ceph-exporter.smithi188 (monmap changed)... 2024-04-05T21:19:37.640 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:37 smithi188 ceph-mon[49715]: Reconfiguring daemon ceph-exporter.smithi188 on smithi188 2024-04-05T21:19:38.499 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:38 smithi188 ceph-mon[49715]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:38.499 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:38 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-05T21:19:38.500 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:38 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:38.500 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:38 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:38.500 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:38 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:38.500 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:38 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi188", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T21:19:38.500 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:38 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:38.714 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:38 smithi121 ceph-mon[39723]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:38.714 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:38 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-05T21:19:38.714 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:38 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:38.714 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:38 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:38.714 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:38 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:38.714 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:38 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi188", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T21:19:38.715 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:38 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:39.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:39 smithi121 ceph-mon[39723]: Deploying daemon osd.0 on smithi121 2024-04-05T21:19:39.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:39 smithi121 ceph-mon[39723]: Reconfiguring crash.smithi188 (monmap changed)... 2024-04-05T21:19:39.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:39 smithi121 ceph-mon[39723]: Reconfiguring daemon crash.smithi188 on smithi188 2024-04-05T21:19:39.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:39 smithi188 ceph-mon[49715]: Deploying daemon osd.0 on smithi121 2024-04-05T21:19:39.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:39 smithi188 ceph-mon[49715]: Reconfiguring crash.smithi188 (monmap changed)... 2024-04-05T21:19:39.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:39 smithi188 ceph-mon[49715]: Reconfiguring daemon crash.smithi188 on smithi188 2024-04-05T21:19:40.415 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:40 smithi121 ceph-mon[39723]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:40.415 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:40.415 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:40.415 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi188.tgcuir", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T21:19:40.415 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T21:19:40.415 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:40.415 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:19:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:40 smithi188 ceph-mon[49715]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:40 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:40 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:40 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi188.tgcuir", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T21:19:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:40 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T21:19:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:40 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:40 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:19:41.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:41 smithi121 ceph-mon[39723]: Reconfiguring mgr.smithi188.tgcuir (monmap changed)... 2024-04-05T21:19:41.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:41 smithi121 ceph-mon[39723]: Reconfiguring daemon mgr.smithi188.tgcuir on smithi188 2024-04-05T21:19:41.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:41 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:41.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:41 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:41.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:41 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T21:19:41.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:41 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T21:19:41.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:41 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:41.749 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:41 smithi188 ceph-mon[49715]: Reconfiguring mgr.smithi188.tgcuir (monmap changed)... 2024-04-05T21:19:41.811 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:41 smithi188 ceph-mon[49715]: Reconfiguring daemon mgr.smithi188.tgcuir on smithi188 2024-04-05T21:19:41.811 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:41 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:41.812 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:41 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:41.812 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:41 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T21:19:41.812 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:41 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T21:19:41.812 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:41 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:42.468 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:42 smithi188 ceph-mon[49715]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:42.469 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:42 smithi188 ceph-mon[49715]: Reconfiguring mon.smithi188 (monmap changed)... 2024-04-05T21:19:42.469 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:42 smithi188 ceph-mon[49715]: Reconfiguring daemon mon.smithi188 on smithi188 2024-04-05T21:19:42.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:42 smithi121 ceph-mon[39723]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:42.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:42 smithi121 ceph-mon[39723]: Reconfiguring mon.smithi188 (monmap changed)... 2024-04-05T21:19:42.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:42 smithi121 ceph-mon[39723]: Reconfiguring daemon mon.smithi188 on smithi188 2024-04-05T21:19:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T21:19:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T21:19:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi121.front.sepia.ceph.com:9093"}]: dispatch 2024-04-05T21:19:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi121.front.sepia.ceph.com:9093"}]: dispatch 2024-04-05T21:19:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T21:19:43.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T21:19:43.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi121.front.sepia.ceph.com:3000"}]: dispatch 2024-04-05T21:19:43.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi121.front.sepia.ceph.com:3000"}]: dispatch 2024-04-05T21:19:43.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:43.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:19:43.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:19:43.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi121.front.sepia.ceph.com:9095"}]: dispatch 2024-04-05T21:19:43.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi121.front.sepia.ceph.com:9095"}]: dispatch 2024-04-05T21:19:43.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:43.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:19:43.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:43.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:43.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:43.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:43.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:43.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T21:19:43.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T21:19:43.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi121.front.sepia.ceph.com:9093"}]: dispatch 2024-04-05T21:19:43.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi121.front.sepia.ceph.com:9093"}]: dispatch 2024-04-05T21:19:43.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:43.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T21:19:43.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T21:19:43.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi121.front.sepia.ceph.com:3000"}]: dispatch 2024-04-05T21:19:43.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi121.front.sepia.ceph.com:3000"}]: dispatch 2024-04-05T21:19:43.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:43.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:19:43.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:19:43.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi121.front.sepia.ceph.com:9095"}]: dispatch 2024-04-05T21:19:43.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi121.front.sepia.ceph.com:9095"}]: dispatch 2024-04-05T21:19:43.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:43.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:19:43.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:43.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:43.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:45.974 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:45 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:45.974 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:45 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:45.974 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:45 smithi188 ceph-mon[49715]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:45 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:45 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:45 smithi121 ceph-mon[39723]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:48.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:48 smithi188 ceph-mon[49715]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:48.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:48 smithi188 ceph-mon[49715]: from='osd.0 [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-05T21:19:48.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:48.292+0000 7ff3a27fc700 1 -- 172.21.15.121:0/3877106655 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff3ac106750 con 0x7ff39806c460 2024-04-05T21:19:48.293 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 0 on host 'smithi121' 2024-04-05T21:19:48.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:48.294+0000 7ff3b36ca700 1 -- 172.21.15.121:0/3877106655 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff39806c460 msgr2=0x7ff39806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:19:48.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:48.294+0000 7ff3b36ca700 1 --2- 172.21.15.121:0/3877106655 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff39806c460 0x7ff39806e920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac101c60 tx=0x7ff3a400b410 comp rx=0 tx=0).stop 2024-04-05T21:19:48.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:48.294+0000 7ff3b36ca700 1 -- 172.21.15.121:0/3877106655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3ac101e00 msgr2=0x7ff3ac073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:19:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:48.294+0000 7ff3b36ca700 1 --2- 172.21.15.121:0/3877106655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3ac101e00 0x7ff3ac073180 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff39c006010 tx=0x7ff39c00bd70 comp rx=0 tx=0).stop 2024-04-05T21:19:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:48.294+0000 7ff3b36ca700 1 -- 172.21.15.121:0/3877106655 shutdown_connections 2024-04-05T21:19:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:48.294+0000 7ff3b36ca700 1 --2- 172.21.15.121:0/3877106655 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff39806c460 0x7ff39806e920 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:48.294+0000 7ff3b36ca700 1 --2- 172.21.15.121:0/3877106655 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3ac100c00 0x7ff3ac074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:48.294+0000 7ff3b36ca700 1 --2- 172.21.15.121:0/3877106655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3ac101e00 0x7ff3ac073180 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:48.294+0000 7ff3b36ca700 1 -- 172.21.15.121:0/3877106655 >> 172.21.15.121:0/3877106655 conn(0x7ff3ac0fc180 msgr2=0x7ff3ac105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:19:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:48.294+0000 7ff3b36ca700 1 -- 172.21.15.121:0/3877106655 shutdown_connections 2024-04-05T21:19:48.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:48.294+0000 7ff3b36ca700 1 -- 172.21.15.121:0/3877106655 wait complete. 2024-04-05T21:19:48.323 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:48 smithi121 ceph-mon[39723]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:48.323 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:48 smithi121 ceph-mon[39723]: from='osd.0 [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-05T21:19:49.034 DEBUG:teuthology.orchestra.run.smithi121:osd.0> sudo journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.0.service 2024-04-05T21:19:49.037 INFO:tasks.cephadm:Deploying osd.1 on smithi121 with /dev/vg_nvme/lv_3... 2024-04-05T21:19:49.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-05T21:19:49.070 INFO:journalctl@ceph.osd.0.smithi121.stdout:-- Logs begin at Fri 2024-04-05 21:05:55 UTC. -- 2024-04-05T21:19:49.375 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:19:49.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:49 smithi121 ceph-mon[39723]: from='osd.0 [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-05T21:19:49.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:49 smithi121 ceph-mon[39723]: osdmap e7: 1 total, 0 up, 1 in 2024-04-05T21:19:49.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:19:49.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:49 smithi121 ceph-mon[39723]: from='osd.0 [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-05T21:19:49.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:49.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:49.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:49 smithi188 ceph-mon[49715]: from='osd.0 [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-05T21:19:49.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:49 smithi188 ceph-mon[49715]: osdmap e7: 1 total, 0 up, 1 in 2024-04-05T21:19:49.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:49 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:19:49.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:49 smithi188 ceph-mon[49715]: from='osd.0 [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-05T21:19:49.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:49 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:49.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:49 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:50.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:50 smithi121 ceph-mon[39723]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:50.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:50 smithi121 ceph-mon[39723]: from='osd.0 [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-05T21:19:50.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:50 smithi121 ceph-mon[39723]: osdmap e8: 1 total, 0 up, 1 in 2024-04-05T21:19:50.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:50 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:19:50.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:50 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:19:50.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:50 smithi188 ceph-mon[49715]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:50.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:50 smithi188 ceph-mon[49715]: from='osd.0 [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-05T21:19:50.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:50 smithi188 ceph-mon[49715]: osdmap e8: 1 total, 0 up, 1 in 2024-04-05T21:19:50.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:50 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:19:50.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:50 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:19:51.227 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:51 smithi121 ceph-mon[39723]: purged_snaps scrub starts 2024-04-05T21:19:51.227 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:51 smithi121 ceph-mon[39723]: purged_snaps scrub ok 2024-04-05T21:19:51.227 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:51 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:19:51.227 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:51 smithi121 ceph-mon[39723]: osd.0 [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890] boot 2024-04-05T21:19:51.227 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:51 smithi121 ceph-mon[39723]: osdmap e9: 1 total, 1 up, 1 in 2024-04-05T21:19:51.227 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:51 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:19:51.227 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:51 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:51.227 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:51 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:51.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:51 smithi188 ceph-mon[49715]: purged_snaps scrub starts 2024-04-05T21:19:51.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:51 smithi188 ceph-mon[49715]: purged_snaps scrub ok 2024-04-05T21:19:51.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:51 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:19:51.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:51 smithi188 ceph-mon[49715]: osd.0 [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890] boot 2024-04-05T21:19:51.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:51 smithi188 ceph-mon[49715]: osdmap e9: 1 total, 1 up, 1 in 2024-04-05T21:19:51.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:51 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:19:51.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:51 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:51.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:51 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:52.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:52 smithi121 ceph-mon[39723]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:52.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:52 smithi121 ceph-mon[39723]: osdmap e10: 1 total, 1 up, 1 in 2024-04-05T21:19:52.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:52 smithi188 ceph-mon[49715]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-05T21:19:52.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:52 smithi188 ceph-mon[49715]: osdmap e10: 1 total, 1 up, 1 in 2024-04-05T21:19:52.587 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-05T21:19:52.587 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-05T21:19:52.587 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-04-05T21:19:52.587 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-04-05T21:19:52.587 INFO:teuthology.orchestra.run.smithi121.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111565 s, 940 MB/s 2024-04-05T21:19:52.587 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 2024-04-05T21:19:52.587 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-04-05T21:19:53.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi121:vg_nvme/lv_3 2024-04-05T21:19:53.790 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:19:54.269 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:54 smithi121 ceph-mon[39723]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:19:54.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:54 smithi188 ceph-mon[49715]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:19:55.417 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:55 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:19:55.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:55 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:19:56.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:56 smithi121 ceph-mon[39723]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:19:56.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:56 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:56.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:56 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:56.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:56 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:19:56.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:56 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:56.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:56 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:56.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:56 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:19:56.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:56 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:56.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:56 smithi188 ceph-mon[49715]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:19:56.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:56 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:56.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:56 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:56.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:56 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:19:56.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:56 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:56.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:56 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:56.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:56 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:19:56.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:56 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:56.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.604+0000 7fea5026b700 1 -- 172.21.15.121:0/1387020874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea48101e60 msgr2=0x7fea481022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:19:56.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.604+0000 7fea5026b700 1 --2- 172.21.15.121:0/1387020874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea48101e60 0x7fea481022c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fea38009910 tx=0x7fea38009c20 comp rx=0 tx=0).stop 2024-04-05T21:19:56.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.604+0000 7fea5026b700 1 -- 172.21.15.121:0/1387020874 shutdown_connections 2024-04-05T21:19:56.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.604+0000 7fea5026b700 1 --2- 172.21.15.121:0/1387020874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea48101e60 0x7fea481022c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:56.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.604+0000 7fea5026b700 1 --2- 172.21.15.121:0/1387020874 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea480fee80 0x7fea480ff2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:56.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.604+0000 7fea5026b700 1 -- 172.21.15.121:0/1387020874 >> 172.21.15.121:0/1387020874 conn(0x7fea480fa7c0 msgr2=0x7fea480fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:19:56.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.604+0000 7fea5026b700 1 -- 172.21.15.121:0/1387020874 shutdown_connections 2024-04-05T21:19:56.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.604+0000 7fea5026b700 1 -- 172.21.15.121:0/1387020874 wait complete. 2024-04-05T21:19:56.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea5026b700 1 Processor -- start 2024-04-05T21:19:56.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea5026b700 1 -- start start 2024-04-05T21:19:56.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea5026b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea480fee80 0x7fea481962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:19:56.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea5026b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea48101e60 0x7fea481967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:19:56.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea5026b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea48196d80 con 0x7fea48101e60 2024-04-05T21:19:56.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea5026b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea48196ec0 con 0x7fea480fee80 2024-04-05T21:19:56.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea4d806700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea48101e60 0x7fea481967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:19:56.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea4d806700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea48101e60 0x7fea481967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44324/0 (socket says 172.21.15.121:44324) 2024-04-05T21:19:56.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea4d806700 1 -- 172.21.15.121:0/168251147 learned_addr learned my addr 172.21.15.121:0/168251147 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:19:56.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea4e007700 1 --2- 172.21.15.121:0/168251147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea480fee80 0x7fea481962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:19:56.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea4d806700 1 -- 172.21.15.121:0/168251147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea480fee80 msgr2=0x7fea481962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:19:56.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea4d806700 1 --2- 172.21.15.121:0/168251147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea480fee80 0x7fea481962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:19:56.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea4d806700 1 -- 172.21.15.121:0/168251147 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea380095f0 con 0x7fea48101e60 2024-04-05T21:19:56.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea4e007700 1 --2- 172.21.15.121:0/168251147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea480fee80 0x7fea481962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:19:56.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea4d806700 1 --2- 172.21.15.121:0/168251147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea48101e60 0x7fea481967f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fea3800bd70 tx=0x7fea3800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:19:56.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.605+0000 7fea3f7fe700 1 -- 172.21.15.121:0/168251147 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea3801d070 con 0x7fea48101e60 2024-04-05T21:19:56.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.606+0000 7fea3f7fe700 1 -- 172.21.15.121:0/168251147 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fea38005940 con 0x7fea48101e60 2024-04-05T21:19:56.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.606+0000 7fea5026b700 1 -- 172.21.15.121:0/168251147 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4819b920 con 0x7fea48101e60 2024-04-05T21:19:56.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.606+0000 7fea3f7fe700 1 -- 172.21.15.121:0/168251147 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea3800f460 con 0x7fea48101e60 2024-04-05T21:19:56.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.606+0000 7fea5026b700 1 -- 172.21.15.121:0/168251147 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4819be60 con 0x7fea48101e60 2024-04-05T21:19:56.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.606+0000 7fea5026b700 1 -- 172.21.15.121:0/168251147 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea481095e0 con 0x7fea48101e60 2024-04-05T21:19:56.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.607+0000 7fea3f7fe700 1 -- 172.21.15.121:0/168251147 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7fea38022800 con 0x7fea48101e60 2024-04-05T21:19:56.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.607+0000 7fea3f7fe700 1 --2- 172.21.15.121:0/168251147 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fea3406c460 0x7fea3406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:19:56.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.607+0000 7fea4e007700 1 --2- 172.21.15.121:0/168251147 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fea3406c460 0x7fea3406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:19:56.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.607+0000 7fea3f7fe700 1 -- 172.21.15.121:0/168251147 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7fea3808bef0 con 0x7fea48101e60 2024-04-05T21:19:56.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.608+0000 7fea4e007700 1 --2- 172.21.15.121:0/168251147 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fea3406c460 0x7fea3406e920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fea48068210 tx=0x7fea40009040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:19:56.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.612+0000 7fea3f7fe700 1 -- 172.21.15.121:0/168251147 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fea3805b090 con 0x7fea48101e60 2024-04-05T21:19:56.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:19:56.827+0000 7fea5026b700 1 -- 172.21.15.121:0/168251147 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fea481067b0 con 0x7fea3406c460 2024-04-05T21:19:57.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:57 smithi121 ceph-mon[39723]: Detected new or changed devices on smithi121 2024-04-05T21:19:57.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:57 smithi121 ceph-mon[39723]: Adjusting osd_memory_target on smithi121 to 11874M 2024-04-05T21:19:57.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:19:57.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T21:19:57.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T21:19:57.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:57 smithi188 ceph-mon[49715]: Detected new or changed devices on smithi121 2024-04-05T21:19:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:57 smithi188 ceph-mon[49715]: Adjusting osd_memory_target on smithi121 to 11874M 2024-04-05T21:19:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:19:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T21:19:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T21:19:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:19:58.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:58 smithi121 ceph-mon[39723]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:19:58.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:58 smithi121 ceph-mon[39723]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:19:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:58 smithi188 ceph-mon[49715]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:19:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:58 smithi188 ceph-mon[49715]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:19:59.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:59.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:19:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:59.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:19:59.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:19:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:00 smithi121 ceph-mon[39723]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:00 smithi121 ceph-mon[39723]: overall HEALTH_OK 2024-04-05T21:20:00.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:00 smithi188 ceph-mon[49715]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:00.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:00 smithi188 ceph-mon[49715]: overall HEALTH_OK 2024-04-05T21:20:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:01 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/3070034989' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6695518f-c4a5-416a-8af1-2899fdebefd1"}]: dispatch 2024-04-05T21:20:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:01 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/3070034989' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6695518f-c4a5-416a-8af1-2899fdebefd1"}]': finished 2024-04-05T21:20:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:01 smithi188 ceph-mon[49715]: osdmap e11: 2 total, 1 up, 2 in 2024-04-05T21:20:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:01 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:20:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:01 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3070034989' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6695518f-c4a5-416a-8af1-2899fdebefd1"}]: dispatch 2024-04-05T21:20:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:01 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3070034989' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6695518f-c4a5-416a-8af1-2899fdebefd1"}]': finished 2024-04-05T21:20:01.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:01 smithi121 ceph-mon[39723]: osdmap e11: 2 total, 1 up, 2 in 2024-04-05T21:20:01.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:01 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:20:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:02 smithi188 ceph-mon[49715]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:02 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2431466420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T21:20:02.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:02 smithi121 ceph-mon[39723]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:02.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:02 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2431466420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T21:20:03.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:03 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:03.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:03 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:03.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:03 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:03.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:03 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:20:03.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:03 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:03 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:03 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:03 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:03 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:20:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:03 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:04.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:04 smithi121 ceph-mon[39723]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:04.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:04 smithi188 ceph-mon[49715]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:06.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:06 smithi121 ceph-mon[39723]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:06.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:06 smithi188 ceph-mon[49715]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:08.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:08 smithi121 ceph-mon[39723]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:08.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:08 smithi188 ceph-mon[49715]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:09.343 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-05T21:20:09.344 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-05T21:20:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:10.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:10 smithi121 ceph-mon[39723]: Deploying daemon osd.1 on smithi121 2024-04-05T21:20:10.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:10 smithi121 ceph-mon[39723]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:10.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:10 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:20:10.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:10 smithi188 ceph-mon[49715]: Deploying daemon osd.1 on smithi121 2024-04-05T21:20:10.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:10 smithi188 ceph-mon[49715]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:10.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:10 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:20:12.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:12 smithi121 ceph-mon[39723]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:12.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:12 smithi188 ceph-mon[49715]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:14.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:14 smithi121 ceph-mon[39723]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:14.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:14 smithi188 ceph-mon[49715]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:15 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:15.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:15 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:15.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:15 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:20:15.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:15 smithi121 ceph-mon[39723]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:15 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:15 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:15 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:20:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:15 smithi188 ceph-mon[49715]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:17 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:17 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:17 smithi121 ceph-mon[39723]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:17.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:17 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:17.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:17 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:17.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:17 smithi188 ceph-mon[49715]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:18.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:18.393+0000 7fea3f7fe700 1 -- 172.21.15.121:0/168251147 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fea481067b0 con 0x7fea3406c460 2024-04-05T21:20:18.394 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 1 on host 'smithi121' 2024-04-05T21:20:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:18.396+0000 7fea5026b700 1 -- 172.21.15.121:0/168251147 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fea3406c460 msgr2=0x7fea3406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:20:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:18.396+0000 7fea5026b700 1 --2- 172.21.15.121:0/168251147 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fea3406c460 0x7fea3406e920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fea48068210 tx=0x7fea40009040 comp rx=0 tx=0).stop 2024-04-05T21:20:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:18.396+0000 7fea5026b700 1 -- 172.21.15.121:0/168251147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea48101e60 msgr2=0x7fea481967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:20:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:18.396+0000 7fea5026b700 1 --2- 172.21.15.121:0/168251147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea48101e60 0x7fea481967f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fea3800bd70 tx=0x7fea3800be50 comp rx=0 tx=0).stop 2024-04-05T21:20:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:18.396+0000 7fea5026b700 1 -- 172.21.15.121:0/168251147 shutdown_connections 2024-04-05T21:20:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:18.396+0000 7fea5026b700 1 --2- 172.21.15.121:0/168251147 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fea3406c460 0x7fea3406e920 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:20:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:18.396+0000 7fea5026b700 1 --2- 172.21.15.121:0/168251147 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea480fee80 0x7fea481962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:20:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:18.396+0000 7fea5026b700 1 --2- 172.21.15.121:0/168251147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea48101e60 0x7fea481967f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:20:18.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:18.396+0000 7fea5026b700 1 -- 172.21.15.121:0/168251147 >> 172.21.15.121:0/168251147 conn(0x7fea480fa7c0 msgr2=0x7fea48105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:20:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:18.396+0000 7fea5026b700 1 -- 172.21.15.121:0/168251147 shutdown_connections 2024-04-05T21:20:18.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:18.396+0000 7fea5026b700 1 -- 172.21.15.121:0/168251147 wait complete. 2024-04-05T21:20:18.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:18 smithi121 ceph-mon[39723]: from='osd.1 [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T21:20:18.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:18.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:18.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:18 smithi188 ceph-mon[49715]: from='osd.1 [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T21:20:18.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:18.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:19.341 DEBUG:teuthology.orchestra.run.smithi121:osd.1> sudo journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.1.service 2024-04-05T21:20:19.343 INFO:tasks.cephadm:Deploying osd.2 on smithi121 with /dev/vg_nvme/lv_2... 2024-04-05T21:20:19.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-05T21:20:19.375 INFO:journalctl@ceph.osd.1.smithi121.stdout:-- Logs begin at Fri 2024-04-05 21:05:55 UTC. -- 2024-04-05T21:20:19.694 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:20:19 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1[73558]: 2024-04-05T21:20:19.473+0000 7fd9c75af700 -1 osd.1 0 waiting for initial osdmap 2024-04-05T21:20:19.695 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:20:19 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1[73558]: 2024-04-05T21:20:19.480+0000 7fd9c1ba1700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T21:20:19.695 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:19 smithi121 ceph-mon[39723]: from='osd.1 [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-05T21:20:19.695 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:19 smithi121 ceph-mon[39723]: osdmap e12: 2 total, 1 up, 2 in 2024-04-05T21:20:19.695 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:19 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:20:19.695 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:19 smithi121 ceph-mon[39723]: from='osd.1 [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-05T21:20:19.695 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:19 smithi121 ceph-mon[39723]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:19.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:19 smithi188 ceph-mon[49715]: from='osd.1 [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-05T21:20:19.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:19 smithi188 ceph-mon[49715]: osdmap e12: 2 total, 1 up, 2 in 2024-04-05T21:20:19.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:19 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:20:19.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:19 smithi188 ceph-mon[49715]: from='osd.1 [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-05T21:20:19.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:19 smithi188 ceph-mon[49715]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:20.218 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:20:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:20 smithi121 ceph-mon[39723]: from='osd.1 [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-05T21:20:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:20 smithi121 ceph-mon[39723]: osdmap e13: 2 total, 1 up, 2 in 2024-04-05T21:20:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:20:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:20:20.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:20 smithi188 ceph-mon[49715]: from='osd.1 [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-05T21:20:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:20 smithi188 ceph-mon[49715]: osdmap e13: 2 total, 1 up, 2 in 2024-04-05T21:20:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:20:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:20:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:21 smithi121 ceph-mon[39723]: purged_snaps scrub starts 2024-04-05T21:20:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:21 smithi121 ceph-mon[39723]: purged_snaps scrub ok 2024-04-05T21:20:21.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:21 smithi121 ceph-mon[39723]: osd.1 [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215] boot 2024-04-05T21:20:21.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:21 smithi121 ceph-mon[39723]: osdmap e14: 2 total, 2 up, 2 in 2024-04-05T21:20:21.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:21 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:20:21.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:21 smithi121 ceph-mon[39723]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:21 smithi188 ceph-mon[49715]: purged_snaps scrub starts 2024-04-05T21:20:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:21 smithi188 ceph-mon[49715]: purged_snaps scrub ok 2024-04-05T21:20:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:21 smithi188 ceph-mon[49715]: osd.1 [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215] boot 2024-04-05T21:20:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:21 smithi188 ceph-mon[49715]: osdmap e14: 2 total, 2 up, 2 in 2024-04-05T21:20:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:21 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:20:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:21 smithi188 ceph-mon[49715]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-05T21:20:22.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:22 smithi121 ceph-mon[39723]: osdmap e15: 2 total, 2 up, 2 in 2024-04-05T21:20:22.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:22 smithi188 ceph-mon[49715]: osdmap e15: 2 total, 2 up, 2 in 2024-04-05T21:20:23.446 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-05T21:20:23.446 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-05T21:20:23.446 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-04-05T21:20:23.447 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-04-05T21:20:23.447 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114596 s, 915 MB/s 2024-04-05T21:20:23.447 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-04-05T21:20:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:23 smithi121 ceph-mon[39723]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:23.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:23 smithi188 ceph-mon[49715]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:24.142 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi121:vg_nvme/lv_2 2024-04-05T21:20:24.967 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:20:25.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:24 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:25.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:24 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:25.099 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:24 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:25.099 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:24 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:26.039 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:25 smithi121 ceph-mon[39723]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:26.039 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:20:26.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:25 smithi188 ceph-mon[49715]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:26.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:25 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:20:28.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:28 smithi121 ceph-mon[39723]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:28.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:28 smithi188 ceph-mon[49715]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.561+0000 7f289cd5e700 1 -- 172.21.15.121:0/2358166451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2898103f70 msgr2=0x7f2898106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:20:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.561+0000 7f289cd5e700 1 --2- 172.21.15.121:0/2358166451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2898103f70 0x7f2898106360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2880009910 tx=0x7f2880009c20 comp rx=0 tx=0).stop 2024-04-05T21:20:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.561+0000 7f289cd5e700 1 -- 172.21.15.121:0/2358166451 shutdown_connections 2024-04-05T21:20:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.561+0000 7f289cd5e700 1 --2- 172.21.15.121:0/2358166451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2898103f70 0x7f2898106360 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:20:28.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.561+0000 7f289cd5e700 1 --2- 172.21.15.121:0/2358166451 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2898101640 0x7f2898103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:20:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.561+0000 7f289cd5e700 1 -- 172.21.15.121:0/2358166451 >> 172.21.15.121:0/2358166451 conn(0x7f28980fb030 msgr2=0x7f28980fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:20:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.561+0000 7f289cd5e700 1 -- 172.21.15.121:0/2358166451 shutdown_connections 2024-04-05T21:20:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.561+0000 7f289cd5e700 1 -- 172.21.15.121:0/2358166451 wait complete. 2024-04-05T21:20:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.562+0000 7f289cd5e700 1 Processor -- start 2024-04-05T21:20:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.562+0000 7f289cd5e700 1 -- start start 2024-04-05T21:20:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.562+0000 7f289cd5e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2898101640 0x7f2898196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:20:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.562+0000 7f289cd5e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2898103f70 0x7f2898196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:20:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.562+0000 7f289cd5e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2898196db0 con 0x7f2898101640 2024-04-05T21:20:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.562+0000 7f289cd5e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2898196ef0 con 0x7f2898103f70 2024-04-05T21:20:28.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.562+0000 7f289659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2898101640 0x7f2898196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:20:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.562+0000 7f289659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2898101640 0x7f2898196250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52446/0 (socket says 172.21.15.121:52446) 2024-04-05T21:20:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.562+0000 7f289659c700 1 -- 172.21.15.121:0/899577740 learned_addr learned my addr 172.21.15.121:0/899577740 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:20:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.562+0000 7f2895d9b700 1 --2- 172.21.15.121:0/899577740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2898103f70 0x7f2898196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:20:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.562+0000 7f289659c700 1 -- 172.21.15.121:0/899577740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2898103f70 msgr2=0x7f2898196790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:20:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.562+0000 7f289659c700 1 --2- 172.21.15.121:0/899577740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2898103f70 0x7f2898196790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:20:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.562+0000 7f289659c700 1 -- 172.21.15.121:0/899577740 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28800095f0 con 0x7f2898101640 2024-04-05T21:20:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.563+0000 7f289659c700 1 --2- 172.21.15.121:0/899577740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2898101640 0x7f2898196250 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f288800d8d0 tx=0x7f288800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:20:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.563+0000 7f288f7fe700 1 -- 172.21.15.121:0/899577740 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28880098a0 con 0x7f2898101640 2024-04-05T21:20:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.563+0000 7f288f7fe700 1 -- 172.21.15.121:0/899577740 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f2888009e90 con 0x7f2898101640 2024-04-05T21:20:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.563+0000 7f289cd5e700 1 -- 172.21.15.121:0/899577740 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289819b9a0 con 0x7f2898101640 2024-04-05T21:20:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.563+0000 7f288f7fe700 1 -- 172.21.15.121:0/899577740 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f288800f5f0 con 0x7f2898101640 2024-04-05T21:20:28.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.563+0000 7f289cd5e700 1 -- 172.21.15.121:0/899577740 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f289819bf70 con 0x7f2898101640 2024-04-05T21:20:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.564+0000 7f289cd5e700 1 -- 172.21.15.121:0/899577740 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28981903e0 con 0x7f2898101640 2024-04-05T21:20:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.564+0000 7f288f7fe700 1 -- 172.21.15.121:0/899577740 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) v1 ==== 90258+0+0 (secure 0 0 0) 0x7f288800f820 con 0x7f2898101640 2024-04-05T21:20:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.564+0000 7f288f7fe700 1 --2- 172.21.15.121:0/899577740 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f288406c460 0x7f288406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:20:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.564+0000 7f288f7fe700 1 -- 172.21.15.121:0/899577740 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f288808aa20 con 0x7f2898101640 2024-04-05T21:20:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.564+0000 7f2895d9b700 1 --2- 172.21.15.121:0/899577740 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f288406c460 0x7f288406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:20:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.565+0000 7f2895d9b700 1 --2- 172.21.15.121:0/899577740 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f288406c460 0x7f288406e920 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f288000b5c0 tx=0x7f2880005ee0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:20:28.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.569+0000 7f288f7fe700 1 -- 172.21.15.121:0/899577740 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f28880599d0 con 0x7f2898101640 2024-04-05T21:20:28.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:28.791+0000 7f289cd5e700 1 -- 172.21.15.121:0/899577740 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f28980611d0 con 0x7f288406c460 2024-04-05T21:20:29.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: Detected new or changed devices on smithi121 2024-04-05T21:20:29.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:29.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:29.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:20:29.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:20:29.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: Adjusting osd_memory_target on smithi121 to 5937M 2024-04-05T21:20:29.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:29.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:29.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:20:29.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:29.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:20:29.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T21:20:29.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T21:20:29.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:29 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:29.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: Detected new or changed devices on smithi121 2024-04-05T21:20:29.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:29.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:29.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:20:29.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:20:29.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: Adjusting osd_memory_target on smithi121 to 5937M 2024-04-05T21:20:29.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:29.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:29.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:20:29.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:29.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:20:29.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T21:20:29.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T21:20:29.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:29 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:30 smithi121 ceph-mon[39723]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:20:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:30 smithi121 ceph-mon[39723]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:30.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:30 smithi188 ceph-mon[49715]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:20:30.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:30 smithi188 ceph-mon[49715]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:31.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:31 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:31.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:31 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:31.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:31 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:31.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:31 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:32 smithi121 ceph-mon[39723]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:32.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:32 smithi188 ceph-mon[49715]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:33 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/111111088' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3b6290b-ef26-4cda-96e2-8bb5649219a5"}]: dispatch 2024-04-05T21:20:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:33 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/111111088' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3b6290b-ef26-4cda-96e2-8bb5649219a5"}]': finished 2024-04-05T21:20:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:33 smithi121 ceph-mon[39723]: osdmap e16: 3 total, 2 up, 3 in 2024-04-05T21:20:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:33 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:20:33.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:33 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/894976140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T21:20:33.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:33 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/111111088' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3b6290b-ef26-4cda-96e2-8bb5649219a5"}]: dispatch 2024-04-05T21:20:33.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:33 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/111111088' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3b6290b-ef26-4cda-96e2-8bb5649219a5"}]': finished 2024-04-05T21:20:33.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:33 smithi188 ceph-mon[49715]: osdmap e16: 3 total, 2 up, 3 in 2024-04-05T21:20:33.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:33 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:20:33.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:33 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/894976140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T21:20:34.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:34 smithi121 ceph-mon[39723]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:34.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:34 smithi188 ceph-mon[49715]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:35.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:35 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:35.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:35 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:35.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:35 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:35.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:35 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:20:35.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:35 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:35.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:35 smithi121 ceph-mon[39723]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:35 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:35 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:35 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:35 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:20:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:35 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:35 smithi188 ceph-mon[49715]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:38.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:38 smithi121 ceph-mon[39723]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:38.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:38 smithi188 ceph-mon[49715]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:40.406 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:40 smithi121 ceph-mon[39723]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:40.407 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:20:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:40 smithi188 ceph-mon[49715]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:40 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:20:41.230 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:41 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-05T21:20:41.230 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:41 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:41.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:41 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-05T21:20:41.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:41 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:42.386 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:42 smithi121 ceph-mon[39723]: Deploying daemon osd.2 on smithi121 2024-04-05T21:20:42.387 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:42 smithi121 ceph-mon[39723]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:42.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:42 smithi188 ceph-mon[49715]: Deploying daemon osd.2 on smithi121 2024-04-05T21:20:42.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:42 smithi188 ceph-mon[49715]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:44.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:44 smithi121 ceph-mon[39723]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:44.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:44 smithi188 ceph-mon[49715]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:46.273 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:46 smithi121 ceph-mon[39723]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:46.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:46 smithi188 ceph-mon[49715]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:47.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:47 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:47.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:47 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:47.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:47 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:20:47.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:47 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:47.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:47 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:47.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:47 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:20:48.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:48 smithi121 ceph-mon[39723]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:48.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:48 smithi188 ceph-mon[49715]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:50.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:50 smithi121 ceph-mon[39723]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:50.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:50 smithi121 ceph-mon[39723]: from='osd.2 [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T21:20:50.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:50.654+0000 7f288f7fe700 1 -- 172.21.15.121:0/899577740 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f28980611d0 con 0x7f288406c460 2024-04-05T21:20:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 2 on host 'smithi121' 2024-04-05T21:20:50.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:50.656+0000 7f289cd5e700 1 -- 172.21.15.121:0/899577740 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f288406c460 msgr2=0x7f288406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:20:50.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:50.656+0000 7f289cd5e700 1 --2- 172.21.15.121:0/899577740 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f288406c460 0x7f288406e920 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f288000b5c0 tx=0x7f2880005ee0 comp rx=0 tx=0).stop 2024-04-05T21:20:50.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:50.656+0000 7f289cd5e700 1 -- 172.21.15.121:0/899577740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2898101640 msgr2=0x7f2898196250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:20:50.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:50.656+0000 7f289cd5e700 1 --2- 172.21.15.121:0/899577740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2898101640 0x7f2898196250 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f288800d8d0 tx=0x7f288800dbe0 comp rx=0 tx=0).stop 2024-04-05T21:20:50.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:50.657+0000 7f289cd5e700 1 -- 172.21.15.121:0/899577740 shutdown_connections 2024-04-05T21:20:50.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:50.657+0000 7f289cd5e700 1 --2- 172.21.15.121:0/899577740 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f288406c460 0x7f288406e920 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:20:50.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:50.657+0000 7f289cd5e700 1 --2- 172.21.15.121:0/899577740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2898101640 0x7f2898196250 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:20:50.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:50.657+0000 7f289cd5e700 1 --2- 172.21.15.121:0/899577740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2898103f70 0x7f2898196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:20:50.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:50.657+0000 7f289cd5e700 1 -- 172.21.15.121:0/899577740 >> 172.21.15.121:0/899577740 conn(0x7f28980fb030 msgr2=0x7f28980fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:20:50.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:50.657+0000 7f289cd5e700 1 -- 172.21.15.121:0/899577740 shutdown_connections 2024-04-05T21:20:50.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:20:50.657+0000 7f289cd5e700 1 -- 172.21.15.121:0/899577740 wait complete. 2024-04-05T21:20:50.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:50 smithi188 ceph-mon[49715]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:50.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:50 smithi188 ceph-mon[49715]: from='osd.2 [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T21:20:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:51 smithi121 ceph-mon[39723]: from='osd.2 [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-05T21:20:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:51 smithi121 ceph-mon[39723]: osdmap e17: 3 total, 2 up, 3 in 2024-04-05T21:20:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:51 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:20:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:51 smithi121 ceph-mon[39723]: from='osd.2 [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-05T21:20:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:51 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:51 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:51.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:51 smithi188 ceph-mon[49715]: from='osd.2 [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-05T21:20:51.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:51 smithi188 ceph-mon[49715]: osdmap e17: 3 total, 2 up, 3 in 2024-04-05T21:20:51.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:51 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:20:51.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:51 smithi188 ceph-mon[49715]: from='osd.2 [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-05T21:20:51.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:51 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:51.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:51 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:51.902 DEBUG:teuthology.orchestra.run.smithi121:osd.2> sudo journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.2.service 2024-04-05T21:20:51.904 INFO:tasks.cephadm:Deploying osd.3 on smithi188 with /dev/vg_nvme/lv_4... 2024-04-05T21:20:51.904 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-05T21:20:51.934 INFO:journalctl@ceph.osd.2.smithi121.stdout:-- Logs begin at Fri 2024-04-05 21:05:55 UTC. -- 2024-04-05T21:20:52.136 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:20:52.695 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:52 smithi188 ceph-mon[49715]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:52.695 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:52 smithi188 ceph-mon[49715]: from='osd.2 [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-05T21:20:52.696 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:52 smithi188 ceph-mon[49715]: osdmap e18: 3 total, 2 up, 3 in 2024-04-05T21:20:52.696 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:52 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:20:52.696 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:52 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:20:52.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:52 smithi121 ceph-mon[39723]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:52.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:52 smithi121 ceph-mon[39723]: from='osd.2 [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-04-05T21:20:52.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:52 smithi121 ceph-mon[39723]: osdmap e18: 3 total, 2 up, 3 in 2024-04-05T21:20:52.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:52 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:20:52.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:52 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:20:53.494 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:53 smithi188 ceph-mon[49715]: purged_snaps scrub starts 2024-04-05T21:20:53.495 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:53 smithi188 ceph-mon[49715]: purged_snaps scrub ok 2024-04-05T21:20:53.495 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:53 smithi188 ceph-mon[49715]: osd.2 [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010] boot 2024-04-05T21:20:53.495 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:53 smithi188 ceph-mon[49715]: osdmap e19: 3 total, 3 up, 3 in 2024-04-05T21:20:53.495 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:53 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:20:53.495 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:53 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-05T21:20:53.683 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:53 smithi121 ceph-mon[39723]: purged_snaps scrub starts 2024-04-05T21:20:53.683 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:53 smithi121 ceph-mon[39723]: purged_snaps scrub ok 2024-04-05T21:20:53.683 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:53 smithi121 ceph-mon[39723]: osd.2 [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010] boot 2024-04-05T21:20:53.683 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:53 smithi121 ceph-mon[39723]: osdmap e19: 3 total, 3 up, 3 in 2024-04-05T21:20:53.683 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:53 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:20:53.683 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:53 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-05T21:20:54.401 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-05T21:20:54.401 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-05T21:20:54.401 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-05T21:20:54.402 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-05T21:20:54.402 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144373 s, 726 MB/s 2024-04-05T21:20:54.402 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-05T21:20:54.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:54 smithi121 ceph-mon[39723]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:54.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-05T21:20:54.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:54 smithi121 ceph-mon[39723]: osdmap e20: 3 total, 3 up, 3 in 2024-04-05T21:20:54.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-05T21:20:54.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:54.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:54.778 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:54 smithi188 ceph-mon[49715]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-05T21:20:54.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-05T21:20:54.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:54 smithi188 ceph-mon[49715]: osdmap e20: 3 total, 3 up, 3 in 2024-04-05T21:20:54.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-05T21:20:54.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:54.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:55.087 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi188:vg_nvme/lv_4 2024-04-05T21:20:55.306 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:20:55.358 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85449]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T21:20:55.358 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85449]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T21:20:55.358 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85449]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T21:20:55.358 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85449]: pam_unix(sudo:session): session closed for user root 2024-04-05T21:20:55.611 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:55 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-05T21:20:55.611 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:55 smithi188 ceph-mon[49715]: osdmap e21: 3 total, 3 up, 3 in 2024-04-05T21:20:55.611 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:55 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:20:55.624 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:55 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-05T21:20:55.625 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:55 smithi121 ceph-mon[39723]: osdmap e21: 3 total, 3 up, 3 in 2024-04-05T21:20:55.625 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:55 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:20:55.625 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85470]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T21:20:55.625 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85470]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T21:20:55.625 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85470]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T21:20:55.625 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85470]: pam_unix(sudo:session): session closed for user root 2024-04-05T21:20:55.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85493]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T21:20:55.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85493]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T21:20:55.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85493]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T21:20:55.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85493]: pam_unix(sudo:session): session closed for user root 2024-04-05T21:20:55.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85496]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T21:20:55.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85496]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T21:20:55.990 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85496]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T21:20:55.990 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:20:55 smithi121 sudo[85496]: pam_unix(sudo:session): session closed for user root 2024-04-05T21:20:56.358 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85506]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T21:20:56.359 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85506]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T21:20:56.359 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85506]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T21:20:56.359 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85506]: pam_unix(sudo:session): session closed for user root 2024-04-05T21:20:56.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:56 smithi188 ceph-mon[49715]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:20:56.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:56 smithi188 ceph-mon[49715]: osdmap e22: 3 total, 3 up, 3 in 2024-04-05T21:20:56.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:56 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:56.617 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85509]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T21:20:56.618 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85509]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T21:20:56.618 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85509]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T21:20:56.618 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85509]: pam_unix(sudo:session): session closed for user root 2024-04-05T21:20:56.618 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:56 smithi121 ceph-mon[39723]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:20:56.619 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:56 smithi121 ceph-mon[39723]: osdmap e22: 3 total, 3 up, 3 in 2024-04-05T21:20:56.619 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:56 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:56.619 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85536]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T21:20:56.619 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85536]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T21:20:56.619 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85536]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T21:20:56.619 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85536]: pam_unix(sudo:session): session closed for user root 2024-04-05T21:20:56.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85559]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T21:20:56.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85559]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T21:20:56.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85559]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T21:20:56.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:56 smithi121 sudo[85559]: pam_unix(sudo:session): session closed for user root 2024-04-05T21:20:57.140 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.137+0000 7fb33c51f700 1 -- 172.21.15.188:0/1175097070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb334103f70 msgr2=0x7fb334106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:20:57.140 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.137+0000 7fb33c51f700 1 --2- 172.21.15.188:0/1175097070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb334103f70 0x7fb334106360 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb324009910 tx=0x7fb324009c20 comp rx=0 tx=0).stop 2024-04-05T21:20:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.137+0000 7fb33c51f700 1 -- 172.21.15.188:0/1175097070 shutdown_connections 2024-04-05T21:20:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.137+0000 7fb33c51f700 1 --2- 172.21.15.188:0/1175097070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb334103f70 0x7fb334106360 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:20:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.137+0000 7fb33c51f700 1 --2- 172.21.15.188:0/1175097070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb334101640 0x7fb334103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:20:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.137+0000 7fb33c51f700 1 -- 172.21.15.188:0/1175097070 >> 172.21.15.188:0/1175097070 conn(0x7fb3340fb030 msgr2=0x7fb3340fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:20:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.137+0000 7fb33c51f700 1 -- 172.21.15.188:0/1175097070 shutdown_connections 2024-04-05T21:20:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.137+0000 7fb33c51f700 1 -- 172.21.15.188:0/1175097070 wait complete. 2024-04-05T21:20:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.138+0000 7fb33c51f700 1 Processor -- start 2024-04-05T21:20:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.138+0000 7fb33c51f700 1 -- start start 2024-04-05T21:20:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.138+0000 7fb33c51f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb334101640 0x7fb334196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:20:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.138+0000 7fb33c51f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb334103f70 0x7fb334196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:20:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.138+0000 7fb33c51f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb334196db0 con 0x7fb334101640 2024-04-05T21:20:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.138+0000 7fb33c51f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb334196ef0 con 0x7fb334103f70 2024-04-05T21:20:57.141 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.138+0000 7fb339aba700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb334103f70 0x7fb334196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:20:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.138+0000 7fb33a2bb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb334101640 0x7fb334196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:20:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.138+0000 7fb339aba700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb334103f70 0x7fb334196790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:40992/0 (socket says 172.21.15.188:40992) 2024-04-05T21:20:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.138+0000 7fb339aba700 1 -- 172.21.15.188:0/3456374851 learned_addr learned my addr 172.21.15.188:0/3456374851 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:20:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.139+0000 7fb33a2bb700 1 -- 172.21.15.188:0/3456374851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb334103f70 msgr2=0x7fb334196790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:20:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.139+0000 7fb33a2bb700 1 --2- 172.21.15.188:0/3456374851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb334103f70 0x7fb334196790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:20:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.139+0000 7fb33a2bb700 1 -- 172.21.15.188:0/3456374851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3240095f0 con 0x7fb334101640 2024-04-05T21:20:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.139+0000 7fb339aba700 1 --2- 172.21.15.188:0/3456374851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb334103f70 0x7fb334196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T21:20:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.139+0000 7fb33a2bb700 1 --2- 172.21.15.188:0/3456374851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb334101640 0x7fb334196250 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb33000e970 tx=0x7fb33000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:20:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.139+0000 7fb32b7fe700 1 -- 172.21.15.188:0/3456374851 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb33000c9d0 con 0x7fb334101640 2024-04-05T21:20:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.139+0000 7fb32b7fe700 1 -- 172.21.15.188:0/3456374851 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fb330010430 con 0x7fb334101640 2024-04-05T21:20:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.139+0000 7fb33c51f700 1 -- 172.21.15.188:0/3456374851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33419b9a0 con 0x7fb334101640 2024-04-05T21:20:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.139+0000 7fb32b7fe700 1 -- 172.21.15.188:0/3456374851 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb330018630 con 0x7fb334101640 2024-04-05T21:20:57.142 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.139+0000 7fb33c51f700 1 -- 172.21.15.188:0/3456374851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33419bf70 con 0x7fb334101640 2024-04-05T21:20:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.140+0000 7fb33c51f700 1 -- 172.21.15.188:0/3456374851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3341903e0 con 0x7fb334101640 2024-04-05T21:20:57.143 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.140+0000 7fb32b7fe700 1 -- 172.21.15.188:0/3456374851 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fb3300105a0 con 0x7fb334101640 2024-04-05T21:20:57.144 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.141+0000 7fb32b7fe700 1 --2- 172.21.15.188:0/3456374851 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fb32006c4e0 0x7fb32006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:20:57.144 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.141+0000 7fb32b7fe700 1 -- 172.21.15.188:0/3456374851 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3232+0+0 (secure 0 0 0) 0x7fb330014070 con 0x7fb334101640 2024-04-05T21:20:57.144 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.141+0000 7fb339aba700 1 --2- 172.21.15.188:0/3456374851 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fb32006c4e0 0x7fb32006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:20:57.144 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.141+0000 7fb339aba700 1 --2- 172.21.15.188:0/3456374851 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fb32006c4e0 0x7fb32006e9a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb324009fd0 tx=0x7fb32400b5a0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:20:57.146 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.146+0000 7fb32b7fe700 1 -- 172.21.15.188:0/3456374851 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fb3300520e0 con 0x7fb334101640 2024-04-05T21:20:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 sudo[53945]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-05T21:20:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 sudo[53945]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T21:20:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 sudo[53945]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T21:20:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 sudo[53945]: pam_unix(sudo:session): session closed for user root 2024-04-05T21:20:57.349 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:20:57.348+0000 7fb33c51f700 1 -- 172.21.15.188:0/3456374851 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3340611d0 con 0x7fb32006c4e0 2024-04-05T21:20:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:57 smithi121 ceph-mon[39723]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T21:20:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:57 smithi121 ceph-mon[39723]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T21:20:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:20:57.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:20:57.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:57 smithi121 ceph-mon[39723]: mgrmap e19: smithi121.etbquh(active, since 2m), standbys: smithi188.tgcuir 2024-04-05T21:20:57.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:20:57.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:20:57.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T21:20:57.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T21:20:57.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 sudo[53971]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-05T21:20:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 sudo[53971]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-05T21:20:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 sudo[53971]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-05T21:20:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 sudo[53971]: pam_unix(sudo:session): session closed for user root 2024-04-05T21:20:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 ceph-mon[49715]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T21:20:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 ceph-mon[49715]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T21:20:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:20:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:20:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 ceph-mon[49715]: mgrmap e19: smithi121.etbquh(active, since 2m), standbys: smithi188.tgcuir 2024-04-05T21:20:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:20:57.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:20:57.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T21:20:57.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T21:20:57.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:58.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:58 smithi121 ceph-mon[39723]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T21:20:58.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:58 smithi121 ceph-mon[39723]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:20:58.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:58 smithi121 ceph-mon[39723]: from='client.14306 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:20:58.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:58 smithi121 ceph-mon[39723]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T21:20:58.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:58 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:58.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:58 smithi188 ceph-mon[49715]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-05T21:20:58.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:58 smithi188 ceph-mon[49715]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:20:58.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:58 smithi188 ceph-mon[49715]: from='client.14306 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:20:58.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:58 smithi188 ceph-mon[49715]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-05T21:20:58.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:58 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:59.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:59 smithi121 ceph-mon[39723]: Detected new or changed devices on smithi121 2024-04-05T21:20:59.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:59.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:59.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:20:59.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:20:59.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:20:59.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:59 smithi121 ceph-mon[39723]: Adjusting osd_memory_target on smithi121 to 3958M 2024-04-05T21:20:59.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:59.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:20:59.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:20:59.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:20:59.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:59 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:20:59.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:20:59 smithi121 ceph-mon[39723]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:21:00.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:59 smithi188 ceph-mon[49715]: Detected new or changed devices on smithi121 2024-04-05T21:21:00.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:00.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:00.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:21:00.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:21:00.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:21:00.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:59 smithi188 ceph-mon[49715]: Adjusting osd_memory_target on smithi121 to 3958M 2024-04-05T21:21:00.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:00.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:00.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:21:00.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:00.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:59 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:21:00.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:20:59 smithi188 ceph-mon[49715]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:21:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:00 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/594217576' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77fa8da5-d202-45f1-b6fa-5af969e9300c"}]: dispatch 2024-04-05T21:21:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:00 smithi121 ceph-mon[39723]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77fa8da5-d202-45f1-b6fa-5af969e9300c"}]: dispatch 2024-04-05T21:21:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:00 smithi121 ceph-mon[39723]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77fa8da5-d202-45f1-b6fa-5af969e9300c"}]': finished 2024-04-05T21:21:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:00 smithi121 ceph-mon[39723]: osdmap e23: 4 total, 3 up, 4 in 2024-04-05T21:21:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:00 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:21:00.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:00 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/2596444450' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T21:21:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:00 smithi188 ceph-mon[49715]: from='client.? 172.21.15.188:0/594217576' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77fa8da5-d202-45f1-b6fa-5af969e9300c"}]: dispatch 2024-04-05T21:21:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:00 smithi188 ceph-mon[49715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77fa8da5-d202-45f1-b6fa-5af969e9300c"}]: dispatch 2024-04-05T21:21:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:00 smithi188 ceph-mon[49715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77fa8da5-d202-45f1-b6fa-5af969e9300c"}]': finished 2024-04-05T21:21:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:00 smithi188 ceph-mon[49715]: osdmap e23: 4 total, 3 up, 4 in 2024-04-05T21:21:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:00 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:21:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:00 smithi188 ceph-mon[49715]: from='client.? 172.21.15.188:0/2596444450' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T21:21:01.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:01 smithi121 ceph-mon[39723]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:21:02.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:01 smithi188 ceph-mon[49715]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:21:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:04 smithi188 ceph-mon[49715]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-05T21:21:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:04 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:04 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:04 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:04 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:21:04.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:04 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:04.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:04 smithi121 ceph-mon[39723]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-05T21:21:04.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:04 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:04.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:04 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:04.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:04 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:04.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:04 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:21:04.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:04 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:06.354 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:06 smithi188 ceph-mon[49715]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-05T21:21:06.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:06 smithi121 ceph-mon[39723]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-05T21:21:08.451 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:08 smithi188 ceph-mon[49715]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-05T21:21:08.451 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:08 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-05T21:21:08.451 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:08 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:08.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:08 smithi121 ceph-mon[39723]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-05T21:21:08.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:08 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-05T21:21:08.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:08 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:09.385 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:09 smithi188 ceph-mon[49715]: Deploying daemon osd.3 on smithi188 2024-04-05T21:21:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:09 smithi121 ceph-mon[39723]: Deploying daemon osd.3 on smithi188 2024-04-05T21:21:10.453 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:10 smithi188 ceph-mon[49715]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-05T21:21:10.453 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:10 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:21:10.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:10 smithi121 ceph-mon[39723]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-05T21:21:10.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:10 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:21:12.430 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:12 smithi188 ceph-mon[49715]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 71 KiB/s wr, 5 op/s 2024-04-05T21:21:12.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:12 smithi121 ceph-mon[39723]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 71 KiB/s wr, 5 op/s 2024-04-05T21:21:14.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:14 smithi121 ceph-mon[39723]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-05T21:21:14.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:14 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:14.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:14 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:14.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:14 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:21:14.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:14 smithi188 ceph-mon[49715]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-05T21:21:14.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:14 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:14.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:14 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:14.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:14 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:21:16.627 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:16 smithi188 ceph-mon[49715]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:21:16.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:16 smithi121 ceph-mon[39723]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:21:17.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:17 smithi188 ceph-mon[49715]: from='osd.3 [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T21:21:17.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:17 smithi188 ceph-mon[49715]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T21:21:17.715 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:17.713+0000 7fb32b7fe700 1 -- 172.21.15.188:0/3456374851 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb3340611d0 con 0x7fb32006c4e0 2024-04-05T21:21:17.715 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 3 on host 'smithi188' 2024-04-05T21:21:17.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:17.715+0000 7fb33c51f700 1 -- 172.21.15.188:0/3456374851 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fb32006c4e0 msgr2=0x7fb32006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:21:17.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:17.715+0000 7fb33c51f700 1 --2- 172.21.15.188:0/3456374851 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fb32006c4e0 0x7fb32006e9a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb324009fd0 tx=0x7fb32400b5a0 comp rx=0 tx=0).stop 2024-04-05T21:21:17.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:17.715+0000 7fb33c51f700 1 -- 172.21.15.188:0/3456374851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb334101640 msgr2=0x7fb334196250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:21:17.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:17.715+0000 7fb33c51f700 1 --2- 172.21.15.188:0/3456374851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb334101640 0x7fb334196250 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb33000e970 tx=0x7fb33000ec80 comp rx=0 tx=0).stop 2024-04-05T21:21:17.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:17.715+0000 7fb33c51f700 1 -- 172.21.15.188:0/3456374851 shutdown_connections 2024-04-05T21:21:17.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:17.715+0000 7fb33c51f700 1 --2- 172.21.15.188:0/3456374851 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fb32006c4e0 0x7fb32006e9a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:21:17.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:17.715+0000 7fb33c51f700 1 --2- 172.21.15.188:0/3456374851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb334101640 0x7fb334196250 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:21:17.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:17.715+0000 7fb33c51f700 1 --2- 172.21.15.188:0/3456374851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb334103f70 0x7fb334196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:21:17.717 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:17.715+0000 7fb33c51f700 1 -- 172.21.15.188:0/3456374851 >> 172.21.15.188:0/3456374851 conn(0x7fb3340fb030 msgr2=0x7fb3340fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:21:17.718 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:17.716+0000 7fb33c51f700 1 -- 172.21.15.188:0/3456374851 shutdown_connections 2024-04-05T21:21:17.718 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:17.716+0000 7fb33c51f700 1 -- 172.21.15.188:0/3456374851 wait complete. 2024-04-05T21:21:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:17 smithi121 ceph-mon[39723]: from='osd.3 [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T21:21:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:17 smithi121 ceph-mon[39723]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T21:21:18.383 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:18 smithi188 ceph-mon[49715]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:21:18.383 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:18 smithi188 ceph-mon[49715]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-05T21:21:18.384 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:18 smithi188 ceph-mon[49715]: osdmap e24: 4 total, 3 up, 4 in 2024-04-05T21:21:18.384 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:21:18.384 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:18 smithi188 ceph-mon[49715]: from='osd.3 [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:21:18.384 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:18 smithi188 ceph-mon[49715]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:21:18.384 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:18.384 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:18.551 DEBUG:teuthology.orchestra.run.smithi188:osd.3> sudo journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.3.service 2024-04-05T21:21:18.553 INFO:tasks.cephadm:Deploying osd.4 on smithi188 with /dev/vg_nvme/lv_3... 2024-04-05T21:21:18.554 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-05T21:21:18.583 INFO:journalctl@ceph.osd.3.smithi188.stdout:-- Logs begin at Fri 2024-04-05 21:05:37 UTC. -- 2024-04-05T21:21:18.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:18 smithi121 ceph-mon[39723]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:21:18.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:18 smithi121 ceph-mon[39723]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-05T21:21:18.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:18 smithi121 ceph-mon[39723]: osdmap e24: 4 total, 3 up, 4 in 2024-04-05T21:21:18.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:21:18.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:18 smithi121 ceph-mon[39723]: from='osd.3 [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:21:18.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:18 smithi121 ceph-mon[39723]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:21:18.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:18.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:18.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:18.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:18.778 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:18.778 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:19.221 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:21:19.715 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:19 smithi188 ceph-mon[49715]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-05T21:21:19.715 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:19 smithi188 ceph-mon[49715]: osdmap e25: 4 total, 3 up, 4 in 2024-04-05T21:21:19.715 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:19 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:21:19.716 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:19 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:21:19.716 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:19 smithi188 ceph-mon[49715]: osd.3 [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079] boot 2024-04-05T21:21:19.716 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:19 smithi188 ceph-mon[49715]: osdmap e26: 4 total, 4 up, 4 in 2024-04-05T21:21:19.716 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:19 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:21:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:19 smithi121 ceph-mon[39723]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-05T21:21:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:19 smithi121 ceph-mon[39723]: osdmap e25: 4 total, 3 up, 4 in 2024-04-05T21:21:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:19 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:21:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:19 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:21:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:19 smithi121 ceph-mon[39723]: osd.3 [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079] boot 2024-04-05T21:21:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:19 smithi121 ceph-mon[39723]: osdmap e26: 4 total, 4 up, 4 in 2024-04-05T21:21:19.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:19 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:21:20.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:20 smithi188 ceph-mon[49715]: purged_snaps scrub starts 2024-04-05T21:21:20.629 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:20 smithi188 ceph-mon[49715]: purged_snaps scrub ok 2024-04-05T21:21:20.629 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:20 smithi188 ceph-mon[49715]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:21:20.629 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:20 smithi188 ceph-mon[49715]: osdmap e27: 4 total, 4 up, 4 in 2024-04-05T21:21:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:20 smithi121 ceph-mon[39723]: purged_snaps scrub starts 2024-04-05T21:21:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:20 smithi121 ceph-mon[39723]: purged_snaps scrub ok 2024-04-05T21:21:20.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:20 smithi121 ceph-mon[39723]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-05T21:21:20.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:20 smithi121 ceph-mon[39723]: osdmap e27: 4 total, 4 up, 4 in 2024-04-05T21:21:22.399 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:22 smithi188 ceph-mon[49715]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:22.399 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:22 smithi188 ceph-mon[49715]: osdmap e28: 4 total, 4 up, 4 in 2024-04-05T21:21:22.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:22 smithi121 ceph-mon[39723]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:22.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:22 smithi121 ceph-mon[39723]: osdmap e28: 4 total, 4 up, 4 in 2024-04-05T21:21:22.993 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-05T21:21:22.993 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-05T21:21:22.993 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-05T21:21:22.993 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-05T21:21:22.993 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109797 s, 955 MB/s 2024-04-05T21:21:22.993 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-05T21:21:23.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:23 smithi188 ceph-mon[49715]: osdmap e29: 4 total, 4 up, 4 in 2024-04-05T21:21:23.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:23.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:23.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:21:23.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:23.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:23.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:21:23.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:23.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:21:23.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:23.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:21:23.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:23.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:23 smithi121 ceph-mon[39723]: osdmap e29: 4 total, 4 up, 4 in 2024-04-05T21:21:23.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:23.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:23.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:21:23.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:23.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:23.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:21:23.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:23.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:21:23.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:23.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:21:23.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:23.665 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi188:vg_nvme/lv_3 2024-04-05T21:21:23.885 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:21:24.461 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:24 smithi188 ceph-mon[49715]: Detected new or changed devices on smithi188 2024-04-05T21:21:24.461 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:24 smithi188 ceph-mon[49715]: Adjusting osd_memory_target on smithi188 to 14946M 2024-04-05T21:21:24.461 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:24 smithi188 ceph-mon[49715]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-05T21:21:24.461 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:24 smithi188 ceph-mon[49715]: osdmap e30: 4 total, 4 up, 4 in 2024-04-05T21:21:24.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:24 smithi121 ceph-mon[39723]: Detected new or changed devices on smithi188 2024-04-05T21:21:24.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:24 smithi121 ceph-mon[39723]: Adjusting osd_memory_target on smithi188 to 14946M 2024-04-05T21:21:24.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:24 smithi121 ceph-mon[39723]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-05T21:21:24.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:24 smithi121 ceph-mon[39723]: osdmap e30: 4 total, 4 up, 4 in 2024-04-05T21:21:25.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:21:25.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:25 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:21:25.825 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.818+0000 7f2aaf587700 1 -- 172.21.15.188:0/877516232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aa8100bd0 msgr2=0x7f2aa8100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:21:25.826 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.818+0000 7f2aaf587700 1 --2- 172.21.15.188:0/877516232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aa8100bd0 0x7f2aa8100ff0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009910 tx=0x7f2aa4009c20 comp rx=0 tx=0).stop 2024-04-05T21:21:25.826 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.818+0000 7f2aaf587700 1 -- 172.21.15.188:0/877516232 shutdown_connections 2024-04-05T21:21:25.826 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.818+0000 7f2aaf587700 1 --2- 172.21.15.188:0/877516232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aa8101dd0 0x7f2aa8102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:21:25.826 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.818+0000 7f2aaf587700 1 --2- 172.21.15.188:0/877516232 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aa8100bd0 0x7f2aa8100ff0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:21:25.826 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.818+0000 7f2aaf587700 1 -- 172.21.15.188:0/877516232 >> 172.21.15.188:0/877516232 conn(0x7f2aa80fc190 msgr2=0x7f2aa80fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:21:25.826 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.819+0000 7f2aaf587700 1 -- 172.21.15.188:0/877516232 shutdown_connections 2024-04-05T21:21:25.826 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.819+0000 7f2aaf587700 1 -- 172.21.15.188:0/877516232 wait complete. 2024-04-05T21:21:25.826 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.819+0000 7f2aaf587700 1 Processor -- start 2024-04-05T21:21:25.826 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.819+0000 7f2aaf587700 1 -- start start 2024-04-05T21:21:25.826 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.819+0000 7f2aaf587700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aa8100bd0 0x7f2aa81963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:21:25.827 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.819+0000 7f2aaf587700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aa8101dd0 0x7f2aa81968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:21:25.827 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2aaf587700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa8196f10 con 0x7f2aa8100bd0 2024-04-05T21:21:25.827 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2aaf587700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa8197050 con 0x7f2aa8101dd0 2024-04-05T21:21:25.827 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2aacb22700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aa8101dd0 0x7f2aa81968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:21:25.827 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2aacb22700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aa8101dd0 0x7f2aa81968f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:43220/0 (socket says 172.21.15.188:43220) 2024-04-05T21:21:25.827 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2aacb22700 1 -- 172.21.15.188:0/3507392741 learned_addr learned my addr 172.21.15.188:0/3507392741 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:21:25.827 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2aad323700 1 --2- 172.21.15.188:0/3507392741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aa8100bd0 0x7f2aa81963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:21:25.827 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2aacb22700 1 -- 172.21.15.188:0/3507392741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aa8100bd0 msgr2=0x7f2aa81963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:21:25.827 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2aacb22700 1 --2- 172.21.15.188:0/3507392741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aa8100bd0 0x7f2aa81963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:21:25.827 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2aacb22700 1 -- 172.21.15.188:0/3507392741 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa40095f0 con 0x7f2aa8101dd0 2024-04-05T21:21:25.827 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2aacb22700 1 --2- 172.21.15.188:0/3507392741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aa8101dd0 0x7f2aa81968f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800b4b0 tx=0x7f2a9800b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:21:25.827 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2a9e7fc700 1 -- 172.21.15.188:0/3507392741 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a98011800 con 0x7f2aa8101dd0 2024-04-05T21:21:25.828 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2aaf587700 1 -- 172.21.15.188:0/3507392741 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa819bb00 con 0x7f2aa8101dd0 2024-04-05T21:21:25.828 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2a9e7fc700 1 -- 172.21.15.188:0/3507392741 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f2a98011df0 con 0x7f2aa8101dd0 2024-04-05T21:21:25.828 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2a9e7fc700 1 -- 172.21.15.188:0/3507392741 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a98010e50 con 0x7f2aa8101dd0 2024-04-05T21:21:25.828 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.820+0000 7f2aaf587700 1 -- 172.21.15.188:0/3507392741 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa819c0d0 con 0x7f2aa8101dd0 2024-04-05T21:21:25.828 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.822+0000 7f2aaf587700 1 -- 172.21.15.188:0/3507392741 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa8109670 con 0x7f2aa8101dd0 2024-04-05T21:21:25.828 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.822+0000 7f2a9e7fc700 1 -- 172.21.15.188:0/3507392741 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2a98010380 con 0x7f2aa8101dd0 2024-04-05T21:21:25.828 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.822+0000 7f2a9e7fc700 1 --2- 172.21.15.188:0/3507392741 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2a9406c4e0 0x7f2a9406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:21:25.828 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.822+0000 7f2a9e7fc700 1 -- 172.21.15.188:0/3507392741 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f2a9808aca0 con 0x7f2aa8101dd0 2024-04-05T21:21:25.828 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.822+0000 7f2aad323700 1 --2- 172.21.15.188:0/3507392741 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2a9406c4e0 0x7f2a9406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:21:25.828 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.823+0000 7f2aad323700 1 --2- 172.21.15.188:0/3507392741 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2a9406c4e0 0x7f2a9406e9a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4006010 tx=0x7f2aa400b540 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:21:25.829 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:25.827+0000 7f2a9e7fc700 1 -- 172.21.15.188:0/3507392741 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2a9808f070 con 0x7f2aa8101dd0 2024-04-05T21:21:26.023 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:26.022+0000 7f2aaf587700 1 -- 172.21.15.188:0/3507392741 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f2aa8106720 con 0x7f2a9406c4e0 2024-04-05T21:21:26.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:26 smithi121 ceph-mon[39723]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-04-05T21:21:26.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:26 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T21:21:26.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:26 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T21:21:26.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:26 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:26.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:26 smithi188 ceph-mon[49715]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-04-05T21:21:26.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:26 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T21:21:26.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:26 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T21:21:26.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:26 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:27.487 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:27 smithi188 ceph-mon[49715]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:21:27.546 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:27 smithi121 ceph-mon[39723]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:21:28.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:28 smithi188 ceph-mon[49715]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-05T21:21:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:28 smithi121 ceph-mon[39723]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-05T21:21:29.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:29 smithi188 ceph-mon[49715]: from='client.? 172.21.15.188:0/3276287005' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abf81f28-7473-45ef-a716-5ef16e038b6e"}]: dispatch 2024-04-05T21:21:29.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:29 smithi188 ceph-mon[49715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abf81f28-7473-45ef-a716-5ef16e038b6e"}]: dispatch 2024-04-05T21:21:29.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:29 smithi188 ceph-mon[49715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abf81f28-7473-45ef-a716-5ef16e038b6e"}]': finished 2024-04-05T21:21:29.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:29 smithi188 ceph-mon[49715]: osdmap e31: 5 total, 4 up, 5 in 2024-04-05T21:21:29.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:29 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:21:29.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:29 smithi188 ceph-mon[49715]: from='client.? 172.21.15.188:0/1783097529' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T21:21:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:29 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/3276287005' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abf81f28-7473-45ef-a716-5ef16e038b6e"}]: dispatch 2024-04-05T21:21:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:29 smithi121 ceph-mon[39723]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abf81f28-7473-45ef-a716-5ef16e038b6e"}]: dispatch 2024-04-05T21:21:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:29 smithi121 ceph-mon[39723]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abf81f28-7473-45ef-a716-5ef16e038b6e"}]': finished 2024-04-05T21:21:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:29 smithi121 ceph-mon[39723]: osdmap e31: 5 total, 4 up, 5 in 2024-04-05T21:21:29.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:29 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:21:29.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:29 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/1783097529' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T21:21:30.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:30 smithi188 ceph-mon[49715]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:30 smithi121 ceph-mon[39723]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:32.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:32 smithi188 ceph-mon[49715]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:32.585 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:32 smithi121 ceph-mon[39723]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:34.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:34 smithi188 ceph-mon[49715]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:34.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:34 smithi121 ceph-mon[39723]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:36 smithi188 ceph-mon[49715]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:36.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:36 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-05T21:21:36.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:36 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:36.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:36 smithi121 ceph-mon[39723]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:36.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:36 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-05T21:21:36.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:36 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:37.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:37 smithi188 ceph-mon[49715]: Deploying daemon osd.4 on smithi188 2024-04-05T21:21:37.624 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:37 smithi121 ceph-mon[39723]: Deploying daemon osd.4 on smithi188 2024-04-05T21:21:38.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:38 smithi188 ceph-mon[49715]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:38.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:38 smithi121 ceph-mon[39723]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:40 smithi188 ceph-mon[49715]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:40 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:21:40.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:40 smithi121 ceph-mon[39723]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:40.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:21:42.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:42 smithi188 ceph-mon[49715]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:42.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:42 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:42.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:42 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:42.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:42 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:21:42.663 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:42 smithi121 ceph-mon[39723]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:42.663 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:42 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:42.663 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:42 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:42.663 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:42 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:21:44.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:44 smithi188 ceph-mon[49715]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:44.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:44 smithi121 ceph-mon[39723]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:45.493 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 4 on host 'smithi188' 2024-04-05T21:21:45.493 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:45.491+0000 7f2a9e7fc700 1 -- 172.21.15.188:0/3507392741 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2aa8106720 con 0x7f2a9406c4e0 2024-04-05T21:21:45.495 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:45.494+0000 7f2aaf587700 1 -- 172.21.15.188:0/3507392741 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2a9406c4e0 msgr2=0x7f2a9406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:21:45.495 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:45.494+0000 7f2aaf587700 1 --2- 172.21.15.188:0/3507392741 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2a9406c4e0 0x7f2a9406e9a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4006010 tx=0x7f2aa400b540 comp rx=0 tx=0).stop 2024-04-05T21:21:45.495 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:45.494+0000 7f2aaf587700 1 -- 172.21.15.188:0/3507392741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aa8101dd0 msgr2=0x7f2aa81968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:21:45.495 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:45.494+0000 7f2aaf587700 1 --2- 172.21.15.188:0/3507392741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aa8101dd0 0x7f2aa81968f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800b4b0 tx=0x7f2a9800b7c0 comp rx=0 tx=0).stop 2024-04-05T21:21:45.495 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:45.494+0000 7f2aaf587700 1 -- 172.21.15.188:0/3507392741 shutdown_connections 2024-04-05T21:21:45.495 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:45.494+0000 7f2aaf587700 1 --2- 172.21.15.188:0/3507392741 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2a9406c4e0 0x7f2a9406e9a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:21:45.495 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:45.494+0000 7f2aaf587700 1 --2- 172.21.15.188:0/3507392741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aa8100bd0 0x7f2aa81963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:21:45.496 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:45.494+0000 7f2aaf587700 1 --2- 172.21.15.188:0/3507392741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aa8101dd0 0x7f2aa81968f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:21:45.496 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:45.494+0000 7f2aaf587700 1 -- 172.21.15.188:0/3507392741 >> 172.21.15.188:0/3507392741 conn(0x7f2aa80fc190 msgr2=0x7f2aa8105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:21:45.496 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:45.494+0000 7f2aaf587700 1 -- 172.21.15.188:0/3507392741 shutdown_connections 2024-04-05T21:21:45.496 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:45.494+0000 7f2aaf587700 1 -- 172.21.15.188:0/3507392741 wait complete. 2024-04-05T21:21:46.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:46 smithi188 ceph-mon[49715]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:46.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:46 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:46.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:46 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:46.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:46 smithi188 ceph-mon[49715]: from='osd.4 [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T21:21:46.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:46 smithi188 ceph-mon[49715]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T21:21:46.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:46 smithi121 ceph-mon[39723]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:46.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:46 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:46.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:46 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:46.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:46 smithi121 ceph-mon[39723]: from='osd.4 [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T21:21:46.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:46 smithi121 ceph-mon[39723]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T21:21:46.878 DEBUG:teuthology.orchestra.run.smithi188:osd.4> sudo journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.4.service 2024-04-05T21:21:46.880 INFO:tasks.cephadm:Deploying osd.5 on smithi188 with /dev/vg_nvme/lv_2... 2024-04-05T21:21:46.880 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-05T21:21:46.910 INFO:journalctl@ceph.osd.4.smithi188.stdout:-- Logs begin at Fri 2024-04-05 21:05:37 UTC. -- 2024-04-05T21:21:47.138 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:21:47.650 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:21:47 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[65356]: 2024-04-05T21:21:47.465+0000 7f3fbf6df700 -1 osd.4 0 waiting for initial osdmap 2024-04-05T21:21:47.650 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:21:47 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[65356]: 2024-04-05T21:21:47.472+0000 7f3fba4d2700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T21:21:47.650 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:47 smithi188 ceph-mon[49715]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-05T21:21:47.650 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:47 smithi188 ceph-mon[49715]: osdmap e32: 5 total, 4 up, 5 in 2024-04-05T21:21:47.650 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:47 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:21:47.650 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:47 smithi188 ceph-mon[49715]: from='osd.4 [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:21:47.651 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:47 smithi188 ceph-mon[49715]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:21:47.651 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:47 smithi188 ceph-mon[49715]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:47.651 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:47 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:47.651 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:47 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:47 smithi121 ceph-mon[39723]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-05T21:21:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:47 smithi121 ceph-mon[39723]: osdmap e32: 5 total, 4 up, 5 in 2024-04-05T21:21:47.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:47 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:21:47.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:47 smithi121 ceph-mon[39723]: from='osd.4 [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:21:47.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:47 smithi121 ceph-mon[39723]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:21:47.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:47 smithi121 ceph-mon[39723]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-05T21:21:47.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:47 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:47.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:47 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:48.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:48 smithi188 ceph-mon[49715]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-05T21:21:48.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:48 smithi188 ceph-mon[49715]: osdmap e33: 5 total, 4 up, 5 in 2024-04-05T21:21:48.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:48 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:21:48.597 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:48 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:21:48.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:48 smithi121 ceph-mon[39723]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-05T21:21:48.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:48 smithi121 ceph-mon[39723]: osdmap e33: 5 total, 4 up, 5 in 2024-04-05T21:21:48.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:48 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:21:48.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:48 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:21:49.553 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-05T21:21:49.553 INFO:teuthology.orchestra.run.smithi188.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-05T21:21:49.554 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10+0 records in 2024-04-05T21:21:49.554 INFO:teuthology.orchestra.run.smithi188.stderr:10+0 records out 2024-04-05T21:21:49.554 INFO:teuthology.orchestra.run.smithi188.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110982 s, 945 MB/s 2024-04-05T21:21:49.554 INFO:teuthology.orchestra.run.smithi188.stderr:--> Zapping successful for: 2024-04-05T21:21:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:49 smithi121 ceph-mon[39723]: purged_snaps scrub starts 2024-04-05T21:21:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:49 smithi121 ceph-mon[39723]: purged_snaps scrub ok 2024-04-05T21:21:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:49 smithi121 ceph-mon[39723]: osd.4 [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110] boot 2024-04-05T21:21:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:49 smithi121 ceph-mon[39723]: osdmap e34: 5 total, 5 up, 5 in 2024-04-05T21:21:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:21:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:49 smithi121 ceph-mon[39723]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:21:49.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:49 smithi188 ceph-mon[49715]: purged_snaps scrub starts 2024-04-05T21:21:49.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:49 smithi188 ceph-mon[49715]: purged_snaps scrub ok 2024-04-05T21:21:49.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:49 smithi188 ceph-mon[49715]: osd.4 [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110] boot 2024-04-05T21:21:49.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:49 smithi188 ceph-mon[49715]: osdmap e34: 5 total, 5 up, 5 in 2024-04-05T21:21:49.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:49 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:21:49.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:49 smithi188 ceph-mon[49715]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:21:50.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:50 smithi188 ceph-mon[49715]: osdmap e35: 5 total, 5 up, 5 in 2024-04-05T21:21:50.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:50 smithi121 ceph-mon[39723]: osdmap e35: 5 total, 5 up, 5 in 2024-04-05T21:21:50.801 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi188:vg_nvme/lv_2 2024-04-05T21:21:51.401 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:21:51.738 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:51 smithi188 ceph-mon[49715]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:21:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:51 smithi121 ceph-mon[39723]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:21:53.987 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.956+0000 7f67df896700 1 -- 172.21.15.188:0/391722536 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67d80ffab0 msgr2=0x7f67d80ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:21:53.988 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.956+0000 7f67df896700 1 --2- 172.21.15.188:0/391722536 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67d80ffab0 0x7f67d80ffed0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f67d4009910 tx=0x7f67d4009c20 comp rx=0 tx=0).stop 2024-04-05T21:21:53.988 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67df896700 1 -- 172.21.15.188:0/391722536 shutdown_connections 2024-04-05T21:21:53.988 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67df896700 1 --2- 172.21.15.188:0/391722536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67d8100cb0 0x7f67d8101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:21:53.988 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67df896700 1 --2- 172.21.15.188:0/391722536 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67d80ffab0 0x7f67d80ffed0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:21:53.988 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67df896700 1 -- 172.21.15.188:0/391722536 >> 172.21.15.188:0/391722536 conn(0x7f67d80fb050 msgr2=0x7f67d80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:21:53.988 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67df896700 1 -- 172.21.15.188:0/391722536 shutdown_connections 2024-04-05T21:21:53.988 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67df896700 1 -- 172.21.15.188:0/391722536 wait complete. 2024-04-05T21:21:53.988 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67df896700 1 Processor -- start 2024-04-05T21:21:53.988 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67df896700 1 -- start start 2024-04-05T21:21:53.988 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67df896700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67d80ffab0 0x7f67d8196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:21:53.988 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67df896700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67d8100cb0 0x7f67d81967c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:21:53.988 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67df896700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d8196de0 con 0x7f67d8100cb0 2024-04-05T21:21:53.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67df896700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d8196f20 con 0x7f67d80ffab0 2024-04-05T21:21:53.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67dd632700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67d80ffab0 0x7f67d8196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:21:53.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67dd632700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67d80ffab0 0x7f67d8196280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:43980/0 (socket says 172.21.15.188:43980) 2024-04-05T21:21:53.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.957+0000 7f67dd632700 1 -- 172.21.15.188:0/3203409031 learned_addr learned my addr 172.21.15.188:0/3203409031 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:21:53.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.958+0000 7f67dce31700 1 --2- 172.21.15.188:0/3203409031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67d8100cb0 0x7f67d81967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:21:53.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.958+0000 7f67dd632700 1 -- 172.21.15.188:0/3203409031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67d8100cb0 msgr2=0x7f67d81967c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:21:53.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.958+0000 7f67dd632700 1 --2- 172.21.15.188:0/3203409031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67d8100cb0 0x7f67d81967c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:21:53.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.958+0000 7f67dd632700 1 -- 172.21.15.188:0/3203409031 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d40095f0 con 0x7f67d80ffab0 2024-04-05T21:21:53.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.958+0000 7f67dd632700 1 --2- 172.21.15.188:0/3203409031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67d80ffab0 0x7f67d8196280 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f67d4000c00 tx=0x7f67d400bde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:21:53.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.958+0000 7f67ca7fc700 1 -- 172.21.15.188:0/3203409031 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67d401d070 con 0x7f67d80ffab0 2024-04-05T21:21:53.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.958+0000 7f67df896700 1 -- 172.21.15.188:0/3203409031 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d819b970 con 0x7f67d80ffab0 2024-04-05T21:21:53.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.958+0000 7f67ca7fc700 1 -- 172.21.15.188:0/3203409031 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f67d40059e0 con 0x7f67d80ffab0 2024-04-05T21:21:53.989 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.958+0000 7f67df896700 1 -- 172.21.15.188:0/3203409031 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d819be80 con 0x7f67d80ffab0 2024-04-05T21:21:53.990 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.958+0000 7f67ca7fc700 1 -- 172.21.15.188:0/3203409031 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67d4003980 con 0x7f67d80ffab0 2024-04-05T21:21:53.990 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.959+0000 7f67df896700 1 -- 172.21.15.188:0/3203409031 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d81084d0 con 0x7f67d80ffab0 2024-04-05T21:21:53.990 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.959+0000 7f67ca7fc700 1 -- 172.21.15.188:0/3203409031 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f67d40041f0 con 0x7f67d80ffab0 2024-04-05T21:21:53.990 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.960+0000 7f67ca7fc700 1 --2- 172.21.15.188:0/3203409031 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f67c406c4e0 0x7f67c406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:21:53.990 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.960+0000 7f67ca7fc700 1 -- 172.21.15.188:0/3203409031 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f67d408ca20 con 0x7f67d80ffab0 2024-04-05T21:21:53.990 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.960+0000 7f67dce31700 1 --2- 172.21.15.188:0/3203409031 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f67c406c4e0 0x7f67c406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:21:53.990 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.960+0000 7f67dce31700 1 --2- 172.21.15.188:0/3203409031 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f67c406c4e0 0x7f67c406e9a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009c60 tx=0x7f67cc009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:21:53.990 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:53.964+0000 7f67ca7fc700 1 -- 172.21.15.188:0/3203409031 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f67d405b180 con 0x7f67d80ffab0 2024-04-05T21:21:54.161 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:21:54.160+0000 7f67df896700 1 -- 172.21.15.188:0/3203409031 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f67d819c5d0 con 0x7f67c406c4e0 2024-04-05T21:21:54.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:54 smithi121 ceph-mon[39723]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:21:54.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:54.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:54.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:21:54.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:21:54.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:54.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:54.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:21:54.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:54.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:21:54.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:54.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:21:54.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:54 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:54.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:54 smithi188 ceph-mon[49715]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:21:54.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:54.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:54.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:21:54.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:21:54.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:54.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:54.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:21:54.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:54.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:21:54.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:54.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:21:54.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:54 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:21:55.369 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:55 smithi188 ceph-mon[49715]: Detected new or changed devices on smithi188 2024-04-05T21:21:55.369 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:55 smithi188 ceph-mon[49715]: Adjusting osd_memory_target on smithi188 to 7473M 2024-04-05T21:21:55.369 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:55 smithi188 ceph-mon[49715]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:21:55.369 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:55 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T21:21:55.369 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:55 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T21:21:55.369 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:55 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:55.369 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:55 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:21:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:55 smithi121 ceph-mon[39723]: Detected new or changed devices on smithi188 2024-04-05T21:21:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:55 smithi121 ceph-mon[39723]: Adjusting osd_memory_target on smithi188 to 7473M 2024-04-05T21:21:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:55 smithi121 ceph-mon[39723]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi188:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:21:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:55 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-05T21:21:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:55 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-05T21:21:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:55 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:21:55.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:55 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:21:56.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:56 smithi121 ceph-mon[39723]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:21:56.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:56 smithi188 ceph-mon[49715]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:21:57.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:57 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/4032083992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "700a347b-e303-447d-bf03-d6677955d140"}]: dispatch 2024-04-05T21:21:57.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:57 smithi121 ceph-mon[39723]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "700a347b-e303-447d-bf03-d6677955d140"}]: dispatch 2024-04-05T21:21:57.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:57 smithi121 ceph-mon[39723]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "700a347b-e303-447d-bf03-d6677955d140"}]': finished 2024-04-05T21:21:57.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:57 smithi121 ceph-mon[39723]: osdmap e36: 6 total, 5 up, 6 in 2024-04-05T21:21:57.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:21:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:57 smithi188 ceph-mon[49715]: from='client.? 172.21.15.188:0/4032083992' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "700a347b-e303-447d-bf03-d6677955d140"}]: dispatch 2024-04-05T21:21:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:57 smithi188 ceph-mon[49715]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "700a347b-e303-447d-bf03-d6677955d140"}]: dispatch 2024-04-05T21:21:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:57 smithi188 ceph-mon[49715]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "700a347b-e303-447d-bf03-d6677955d140"}]': finished 2024-04-05T21:21:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:57 smithi188 ceph-mon[49715]: osdmap e36: 6 total, 5 up, 6 in 2024-04-05T21:21:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:21:58.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:58 smithi121 ceph-mon[39723]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:21:58.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:21:58 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/2107668371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T21:21:58.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:58 smithi188 ceph-mon[49715]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:21:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:21:58 smithi188 ceph-mon[49715]: from='client.? 172.21.15.188:0/2107668371' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-05T21:22:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:00 smithi121 ceph-mon[39723]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:00.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:00 smithi188 ceph-mon[49715]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:02.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:02 smithi121 ceph-mon[39723]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:02 smithi188 ceph-mon[49715]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:04.480 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:04 smithi188 ceph-mon[49715]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:04.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:04 smithi121 ceph-mon[39723]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:05.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:05 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-05T21:22:05.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:05 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:22:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:05 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-05T21:22:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:05 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:22:06.316 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:06 smithi188 ceph-mon[49715]: Deploying daemon osd.5 on smithi188 2024-04-05T21:22:06.316 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:06 smithi188 ceph-mon[49715]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:06.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:06 smithi121 ceph-mon[39723]: Deploying daemon osd.5 on smithi188 2024-04-05T21:22:06.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:06 smithi121 ceph-mon[39723]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:08.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:08 smithi188 ceph-mon[49715]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:08.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:08 smithi121 ceph-mon[39723]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:10.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:10 smithi188 ceph-mon[49715]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:10.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:10 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:22:10.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:10 smithi121 ceph-mon[39723]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:10.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:10 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:22:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:11 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:11 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:11 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:22:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:11 smithi121 ceph-mon[39723]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:11.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:11 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:11.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:11 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:11.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:11 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:22:11.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:11 smithi188 ceph-mon[49715]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:14.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:14 smithi121 ceph-mon[39723]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:14.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:14 smithi188 ceph-mon[49715]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:15.219 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:15 smithi188 ceph-mon[49715]: from='osd.5 [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T21:22:15.220 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:15 smithi188 ceph-mon[49715]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T21:22:15.284 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:15.283+0000 7f67ca7fc700 1 -- 172.21.15.188:0/3203409031 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f67d819c5d0 con 0x7f67c406c4e0 2024-04-05T21:22:15.285 INFO:teuthology.orchestra.run.smithi188.stdout:Created osd(s) 5 on host 'smithi188' 2024-04-05T21:22:15.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:15.285+0000 7f67df896700 1 -- 172.21.15.188:0/3203409031 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f67c406c4e0 msgr2=0x7f67c406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:15.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:15.285+0000 7f67df896700 1 --2- 172.21.15.188:0/3203409031 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f67c406c4e0 0x7f67c406e9a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f67cc009c60 tx=0x7f67cc009260 comp rx=0 tx=0).stop 2024-04-05T21:22:15.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:15.285+0000 7f67df896700 1 -- 172.21.15.188:0/3203409031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67d80ffab0 msgr2=0x7f67d8196280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:15.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:15.285+0000 7f67df896700 1 --2- 172.21.15.188:0/3203409031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67d80ffab0 0x7f67d8196280 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f67d4000c00 tx=0x7f67d400bde0 comp rx=0 tx=0).stop 2024-04-05T21:22:15.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:15.285+0000 7f67df896700 1 -- 172.21.15.188:0/3203409031 shutdown_connections 2024-04-05T21:22:15.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:15.285+0000 7f67df896700 1 --2- 172.21.15.188:0/3203409031 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f67c406c4e0 0x7f67c406e9a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:15.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:15.285+0000 7f67df896700 1 --2- 172.21.15.188:0/3203409031 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67d80ffab0 0x7f67d8196280 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:15.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:15.285+0000 7f67df896700 1 --2- 172.21.15.188:0/3203409031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67d8100cb0 0x7f67d81967c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:15.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:15.285+0000 7f67df896700 1 -- 172.21.15.188:0/3203409031 >> 172.21.15.188:0/3203409031 conn(0x7f67d80fb050 msgr2=0x7f67d8068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:15.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:15.285+0000 7f67df896700 1 -- 172.21.15.188:0/3203409031 shutdown_connections 2024-04-05T21:22:15.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:15.285+0000 7f67df896700 1 -- 172.21.15.188:0/3203409031 wait complete. 2024-04-05T21:22:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:15 smithi121 ceph-mon[39723]: from='osd.5 [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T21:22:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:15 smithi121 ceph-mon[39723]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T21:22:16.107 DEBUG:teuthology.orchestra.run.smithi188:osd.5> sudo journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.5.service 2024-04-05T21:22:16.109 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-05T21:22:16.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph osd stat -f json 2024-04-05T21:22:16.137 INFO:journalctl@ceph.osd.5.smithi188.stdout:-- Logs begin at Fri 2024-04-05 21:05:37 UTC. -- 2024-04-05T21:22:16.346 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:16.377 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:16 smithi188 ceph-mon[49715]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:16.377 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:16 smithi188 ceph-mon[49715]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-05T21:22:16.378 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:16 smithi188 ceph-mon[49715]: osdmap e37: 6 total, 5 up, 6 in 2024-04-05T21:22:16.378 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:16 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:22:16.378 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:16 smithi188 ceph-mon[49715]: from='osd.5 [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:22:16.378 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:16 smithi188 ceph-mon[49715]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:22:16.378 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:16 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:16.378 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:16 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:16.378 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:16 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:16.378 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:16 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:16.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:16 smithi121 ceph-mon[39723]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:16.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:16 smithi121 ceph-mon[39723]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-05T21:22:16.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:16 smithi121 ceph-mon[39723]: osdmap e37: 6 total, 5 up, 6 in 2024-04-05T21:22:16.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:16 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:22:16.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:16 smithi121 ceph-mon[39723]: from='osd.5 [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:22:16.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:16 smithi121 ceph-mon[39723]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:22:16.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:16 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:16.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:16 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:16.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:16 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:16.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:16 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:17.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:17 smithi121 ceph-mon[39723]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-05T21:22:17.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:17 smithi121 ceph-mon[39723]: osdmap e38: 6 total, 5 up, 6 in 2024-04-05T21:22:17.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:17 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:22:17.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:17 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:22:17.506 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:17 smithi188 ceph-mon[49715]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]': finished 2024-04-05T21:22:17.506 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:17 smithi188 ceph-mon[49715]: osdmap e38: 6 total, 5 up, 6 in 2024-04-05T21:22:17.506 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:17 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:22:17.506 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:17 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:22:18.179 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:18 smithi121 ceph-mon[39723]: purged_snaps scrub starts 2024-04-05T21:22:18.179 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:18 smithi121 ceph-mon[39723]: purged_snaps scrub ok 2024-04-05T21:22:18.179 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:18 smithi121 ceph-mon[39723]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:18.179 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:18 smithi121 ceph-mon[39723]: osd.5 [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975] boot 2024-04-05T21:22:18.179 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:18 smithi121 ceph-mon[39723]: osdmap e39: 6 total, 6 up, 6 in 2024-04-05T21:22:18.179 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:22:18.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:18 smithi188 ceph-mon[49715]: purged_snaps scrub starts 2024-04-05T21:22:18.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:18 smithi188 ceph-mon[49715]: purged_snaps scrub ok 2024-04-05T21:22:18.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:18 smithi188 ceph-mon[49715]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-05T21:22:18.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:18 smithi188 ceph-mon[49715]: osd.5 [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975] boot 2024-04-05T21:22:18.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:18 smithi188 ceph-mon[49715]: osdmap e39: 6 total, 6 up, 6 in 2024-04-05T21:22:18.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:22:18.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.483+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1630276876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ebc103f70 msgr2=0x7f2ebc106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.483+0000 7f2ec0d5e700 1 --2- 172.21.15.121:0/1630276876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ebc103f70 0x7f2ebc106360 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009910 tx=0x7f2eb0009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.483+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1630276876 shutdown_connections 2024-04-05T21:22:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.483+0000 7f2ec0d5e700 1 --2- 172.21.15.121:0/1630276876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ebc103f70 0x7f2ebc106360 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.483+0000 7f2ec0d5e700 1 --2- 172.21.15.121:0/1630276876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ebc101640 0x7f2ebc103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.483+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1630276876 >> 172.21.15.121:0/1630276876 conn(0x7f2ebc0fb030 msgr2=0x7f2ebc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.483+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1630276876 shutdown_connections 2024-04-05T21:22:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.483+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1630276876 wait complete. 2024-04-05T21:22:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.483+0000 7f2ec0d5e700 1 Processor -- start 2024-04-05T21:22:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.483+0000 7f2ec0d5e700 1 -- start start 2024-04-05T21:22:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2ec0d5e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ebc101640 0x7f2ebc196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:18.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2ec0d5e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ebc103f70 0x7f2ebc196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2ec0d5e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc196db0 con 0x7f2ebc103f70 2024-04-05T21:22:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2ec0d5e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc196ef0 con 0x7f2ebc101640 2024-04-05T21:22:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2eb9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ebc103f70 0x7f2ebc196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2eb9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ebc103f70 0x7f2ebc196790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39026/0 (socket says 172.21.15.121:39026) 2024-04-05T21:22:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2eba59c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ebc101640 0x7f2ebc196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2eb9d9b700 1 -- 172.21.15.121:0/1911853723 learned_addr learned my addr 172.21.15.121:0/1911853723 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:22:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2eb9d9b700 1 -- 172.21.15.121:0/1911853723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ebc101640 msgr2=0x7f2ebc196250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2eb9d9b700 1 --2- 172.21.15.121:0/1911853723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ebc101640 0x7f2ebc196250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2eb9d9b700 1 -- 172.21.15.121:0/1911853723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb00095f0 con 0x7f2ebc103f70 2024-04-05T21:22:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2eb9d9b700 1 --2- 172.21.15.121:0/1911853723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ebc103f70 0x7f2ebc196790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000bd70 tx=0x7f2eb000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2eab7fe700 1 -- 172.21.15.121:0/1911853723 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2eb001d070 con 0x7f2ebc103f70 2024-04-05T21:22:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1911853723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ebc19b940 con 0x7f2ebc103f70 2024-04-05T21:22:18.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2eab7fe700 1 -- 172.21.15.121:0/1911853723 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f2eb000f460 con 0x7f2ebc103f70 2024-04-05T21:22:18.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2eab7fe700 1 -- 172.21.15.121:0/1911853723 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2eb0021640 con 0x7f2ebc103f70 2024-04-05T21:22:18.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.484+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1911853723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ebc19be30 con 0x7f2ebc103f70 2024-04-05T21:22:18.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.485+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1911853723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ebc1903e0 con 0x7f2ebc103f70 2024-04-05T21:22:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.487+0000 7f2eab7fe700 1 -- 172.21.15.121:0/1911853723 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2eb000fa20 con 0x7f2ebc103f70 2024-04-05T21:22:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.487+0000 7f2eab7fe700 1 --2- 172.21.15.121:0/1911853723 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2ea406c4e0 0x7f2ea406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.487+0000 7f2eab7fe700 1 -- 172.21.15.121:0/1911853723 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2eb008cb20 con 0x7f2ebc103f70 2024-04-05T21:22:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.487+0000 7f2eba59c700 1 --2- 172.21.15.121:0/1911853723 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2ea406c4e0 0x7f2ea406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:18.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.487+0000 7f2eba59c700 1 --2- 172.21.15.121:0/1911853723 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2ea406c4e0 0x7f2ea406e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2eac007c60 tx=0x7f2eac0073d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:18.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.491+0000 7f2eab7fe700 1 -- 172.21.15.121:0/1911853723 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2eb005b150 con 0x7f2ebc103f70 2024-04-05T21:22:18.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.682+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1911853723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f2ebc02cd60 con 0x7f2ebc103f70 2024-04-05T21:22:18.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.683+0000 7f2eab7fe700 1 -- 172.21.15.121:0/1911853723 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f2eb005ace0 con 0x7f2ebc103f70 2024-04-05T21:22:18.684 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:22:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.685+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1911853723 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2ea406c4e0 msgr2=0x7f2ea406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.685+0000 7f2ec0d5e700 1 --2- 172.21.15.121:0/1911853723 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2ea406c4e0 0x7f2ea406e9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2eac007c60 tx=0x7f2eac0073d0 comp rx=0 tx=0).stop 2024-04-05T21:22:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.685+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1911853723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ebc103f70 msgr2=0x7f2ebc196790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.685+0000 7f2ec0d5e700 1 --2- 172.21.15.121:0/1911853723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ebc103f70 0x7f2ebc196790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2eb000bd70 tx=0x7f2eb000bda0 comp rx=0 tx=0).stop 2024-04-05T21:22:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.685+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1911853723 shutdown_connections 2024-04-05T21:22:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.685+0000 7f2ec0d5e700 1 --2- 172.21.15.121:0/1911853723 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2ea406c4e0 0x7f2ea406e9a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:18.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.685+0000 7f2ec0d5e700 1 --2- 172.21.15.121:0/1911853723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2ebc101640 0x7f2ebc196250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.685+0000 7f2ec0d5e700 1 --2- 172.21.15.121:0/1911853723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ebc103f70 0x7f2ebc196790 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.685+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1911853723 >> 172.21.15.121:0/1911853723 conn(0x7f2ebc0fb030 msgr2=0x7f2ebc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.685+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1911853723 shutdown_connections 2024-04-05T21:22:18.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:18.685+0000 7f2ec0d5e700 1 -- 172.21.15.121:0/1911853723 wait complete. 2024-04-05T21:22:19.454 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:19 smithi121 ceph-mon[39723]: osdmap e40: 6 total, 6 up, 6 in 2024-04-05T21:22:19.454 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:19 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1911853723' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-05T21:22:19.474 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1712352137,"num_in_osds":6,"osd_in_since":1712352116,"num_remapped_pgs":0} 2024-04-05T21:22:19.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-05T21:22:19.488 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:19 smithi188 ceph-mon[49715]: osdmap e40: 6 total, 6 up, 6 in 2024-04-05T21:22:19.488 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:19 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1911853723' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-05T21:22:19.712 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:20.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:20.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:20.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:20.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:22:20.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:22:20.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:22:20.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:20.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:22:20.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:22:20.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:20.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:22:20.243 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:22:20.243 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:22:20.243 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:20.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:20.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:20.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:20.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:22:20.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:22:20.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:22:20.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:20.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:22:20.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:22:20.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:20.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:22:20.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:22:20.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:22:20.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:22:21.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:21 smithi188 ceph-mon[49715]: Detected new or changed devices on smithi188 2024-04-05T21:22:21.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:21 smithi188 ceph-mon[49715]: Adjusting osd_memory_target on smithi188 to 4982M 2024-04-05T21:22:21.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.577+0000 7ffa887ef700 1 -- 172.21.15.121:0/226080932 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa800ffbd0 msgr2=0x7ffa80100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.577+0000 7ffa887ef700 1 --2- 172.21.15.121:0/226080932 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa800ffbd0 0x7ffa80100050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009910 tx=0x7ffa7c009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.577+0000 7ffa887ef700 1 -- 172.21.15.121:0/226080932 shutdown_connections 2024-04-05T21:22:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.577+0000 7ffa887ef700 1 --2- 172.21.15.121:0/226080932 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa800ffbd0 0x7ffa80100050 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.577+0000 7ffa887ef700 1 --2- 172.21.15.121:0/226080932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa80105a40 0x7ffa80105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.577+0000 7ffa887ef700 1 -- 172.21.15.121:0/226080932 >> 172.21.15.121:0/226080932 conn(0x7ffa800755a0 msgr2=0x7ffa800759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.577+0000 7ffa887ef700 1 -- 172.21.15.121:0/226080932 shutdown_connections 2024-04-05T21:22:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.577+0000 7ffa887ef700 1 -- 172.21.15.121:0/226080932 wait complete. 2024-04-05T21:22:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.578+0000 7ffa887ef700 1 Processor -- start 2024-04-05T21:22:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.578+0000 7ffa887ef700 1 -- start start 2024-04-05T21:22:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.578+0000 7ffa887ef700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa800ffbd0 0x7ffa80195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.578+0000 7ffa887ef700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa80105a40 0x7ffa80195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.578+0000 7ffa887ef700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa80196160 con 0x7ffa800ffbd0 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.578+0000 7ffa887ef700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa80199ef0 con 0x7ffa80105a40 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.578+0000 7ffa8658b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa800ffbd0 0x7ffa80195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.578+0000 7ffa8658b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa800ffbd0 0x7ffa80195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39038/0 (socket says 172.21.15.121:39038) 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.578+0000 7ffa8658b700 1 -- 172.21.15.121:0/2575256884 learned_addr learned my addr 172.21.15.121:0/2575256884 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.579+0000 7ffa8658b700 1 -- 172.21.15.121:0/2575256884 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa80105a40 msgr2=0x7ffa80195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.579+0000 7ffa8658b700 1 --2- 172.21.15.121:0/2575256884 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa80105a40 0x7ffa80195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.579+0000 7ffa8658b700 1 -- 172.21.15.121:0/2575256884 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7c0095f0 con 0x7ffa800ffbd0 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.579+0000 7ffa8658b700 1 --2- 172.21.15.121:0/2575256884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa800ffbd0 0x7ffa80195540 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffa7000c8a0 tx=0x7ffa7000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.579+0000 7ffa777fe700 1 -- 172.21.15.121:0/2575256884 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa7000cea0 con 0x7ffa800ffbd0 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.579+0000 7ffa887ef700 1 -- 172.21.15.121:0/2575256884 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8019a1d0 con 0x7ffa800ffbd0 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.579+0000 7ffa777fe700 1 -- 172.21.15.121:0/2575256884 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7ffa70004830 con 0x7ffa800ffbd0 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.579+0000 7ffa777fe700 1 -- 172.21.15.121:0/2575256884 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa70005540 con 0x7ffa800ffbd0 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.579+0000 7ffa887ef700 1 -- 172.21.15.121:0/2575256884 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa8019a720 con 0x7ffa800ffbd0 2024-04-05T21:22:21.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.580+0000 7ffa887ef700 1 -- 172.21.15.121:0/2575256884 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa80107fd0 con 0x7ffa800ffbd0 2024-04-05T21:22:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.580+0000 7ffa777fe700 1 -- 172.21.15.121:0/2575256884 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ffa700049a0 con 0x7ffa800ffbd0 2024-04-05T21:22:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.581+0000 7ffa777fe700 1 --2- 172.21.15.121:0/2575256884 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffa6c06c4e0 0x7ffa6c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.581+0000 7ffa85d8a700 1 --2- 172.21.15.121:0/2575256884 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffa6c06c4e0 0x7ffa6c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.581+0000 7ffa777fe700 1 -- 172.21.15.121:0/2575256884 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ffa7008b1d0 con 0x7ffa800ffbd0 2024-04-05T21:22:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.581+0000 7ffa85d8a700 1 --2- 172.21.15.121:0/2575256884 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffa6c06c4e0 0x7ffa6c06e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00b5c0 tx=0x7ffa7c005960 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.585+0000 7ffa777fe700 1 -- 172.21.15.121:0/2575256884 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ffa70055cf0 con 0x7ffa800ffbd0 2024-04-05T21:22:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:21 smithi121 ceph-mon[39723]: Detected new or changed devices on smithi188 2024-04-05T21:22:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:21 smithi121 ceph-mon[39723]: Adjusting osd_memory_target on smithi188 to 4982M 2024-04-05T21:22:21.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.776+0000 7ffa887ef700 1 -- 172.21.15.121:0/2575256884 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa8004ea90 con 0x7ffa800ffbd0 2024-04-05T21:22:21.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.776+0000 7ffa777fe700 1 -- 172.21.15.121:0/2575256884 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11385 (secure 0 0 0) 0x7ffa70059310 con 0x7ffa800ffbd0 2024-04-05T21:22:21.777 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:22:21.778 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":40,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","created":"2024-04-05T21:15:58.797109+0000","modified":"2024-04-05T21:22:18.067824+0000","last_up_change":"2024-04-05T21:22:17.062975+0000","last_in_change":"2024-04-05T21:21:56.806570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T21:20:53.291734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"133652f8-9fe2-45e9-90b6-36f89191114b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6803","nonce":2285715890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6805","nonce":2285715890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6809","nonce":2285715890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6807","nonce":2285715890}]},"public_addr":"172.21.15.121:6803/2285715890","cluster_addr":"172.21.15.121:6805/2285715890","heartbeat_back_addr":"172.21.15.121:6809/2285715890","heartbeat_front_addr":"172.21.15.121:6807/2285715890","state":["exists","up"]},{"osd":1,"uuid":"6695518f-c4a5-416a-8af1-2899fdebefd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6811","nonce":3573482215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6813","nonce":3573482215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6817","nonce":3573482215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6815","nonce":3573482215}]},"public_addr":"172.21.15.121:6811/3573482215","cluster_addr":"172.21.15.121:6813/3573482215","heartbeat_back_addr":"172.21.15.121:6817/3573482215","heartbeat_front_addr":"172.21.15.121:6815/3573482215","state":["exists","up"]},{"osd":2,"uuid":"e3b6290b-ef26-4cda-96e2-8bb5649219a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6819","nonce":179235010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6821","nonce":179235010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6825","nonce":179235010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6823","nonce":179235010}]},"public_addr":"172.21.15.121:6819/179235010","cluster_addr":"172.21.15.121:6821/179235010","heartbeat_back_addr":"172.21.15.121:6825/179235010","heartbeat_front_addr":"172.21.15.121:6823/179235010","state":["exists","up"]},{"osd":3,"uuid":"77fa8da5-d202-45f1-b6fa-5af969e9300c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6801","nonce":2760393079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6803","nonce":2760393079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6807","nonce":2760393079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6805","nonce":2760393079}]},"public_addr":"172.21.15.188:6801/2760393079","cluster_addr":"172.21.15.188:6803/2760393079","heartbeat_back_addr":"172.21.15.188:6807/2760393079","heartbeat_front_addr":"172.21.15.188:6805/2760393079","state":["exists","up"]},{"osd":4,"uuid":"abf81f28-7473-45ef-a716-5ef16e038b6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6809","nonce":1828363110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6811","nonce":1828363110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6815","nonce":1828363110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6813","nonce":1828363110}]},"public_addr":"172.21.15.188:6809/1828363110","cluster_addr":"172.21.15.188:6811/1828363110","heartbeat_back_addr":"172.21.15.188:6815/1828363110","heartbeat_front_addr":"172.21.15.188:6813/1828363110","state":["exists","up"]},{"osd":5,"uuid":"700a347b-e303-447d-bf03-d6677955d140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6817","nonce":3569223975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6819","nonce":3569223975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6823","nonce":3569223975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6821","nonce":3569223975}]},"public_addr":"172.21.15.188:6817/3569223975","cluster_addr":"172.21.15.188:6819/3569223975","heartbeat_back_addr":"172.21.15.188:6823/3569223975","heartbeat_front_addr":"172.21.15.188:6821/3569223975","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:19:48.038490+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:19.269394+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:51.162437+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:18.203341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:46.805653+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:22:15.296548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6801/1237774071":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6800/1237774071":"2024-04-06T21:18:24.954707+0000","172.21.15.121:0/3235459786":"2024-04-06T21:18:24.954707+0000","172.21.15.121:0/3247654758":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6800/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/341396693":"2024-04-06T21:17:08.626094+0000","172.21.15.121:6801/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/2214582362":"2024-04-06T21:17:08.626094+0000","172.21.15.121:6801/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/3762940011":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6800/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/1364585923":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/2536941662":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/2383002272":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/42058318":"2024-04-06T21:17:08.626094+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T21:22:21.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.778+0000 7ffa887ef700 1 -- 172.21.15.121:0/2575256884 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffa6c06c4e0 msgr2=0x7ffa6c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:21.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.778+0000 7ffa887ef700 1 --2- 172.21.15.121:0/2575256884 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffa6c06c4e0 0x7ffa6c06e9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00b5c0 tx=0x7ffa7c005960 comp rx=0 tx=0).stop 2024-04-05T21:22:21.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.778+0000 7ffa887ef700 1 -- 172.21.15.121:0/2575256884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa800ffbd0 msgr2=0x7ffa80195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:21.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.778+0000 7ffa887ef700 1 --2- 172.21.15.121:0/2575256884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa800ffbd0 0x7ffa80195540 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffa7000c8a0 tx=0x7ffa7000cc60 comp rx=0 tx=0).stop 2024-04-05T21:22:21.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.778+0000 7ffa887ef700 1 -- 172.21.15.121:0/2575256884 shutdown_connections 2024-04-05T21:22:21.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.778+0000 7ffa887ef700 1 --2- 172.21.15.121:0/2575256884 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffa6c06c4e0 0x7ffa6c06e9a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:21.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.778+0000 7ffa887ef700 1 --2- 172.21.15.121:0/2575256884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa800ffbd0 0x7ffa80195540 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:21.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.778+0000 7ffa887ef700 1 --2- 172.21.15.121:0/2575256884 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffa80105a40 0x7ffa80195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:21.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.778+0000 7ffa887ef700 1 -- 172.21.15.121:0/2575256884 >> 172.21.15.121:0/2575256884 conn(0x7ffa800755a0 msgr2=0x7ffa800fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:21.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.779+0000 7ffa887ef700 1 -- 172.21.15.121:0/2575256884 shutdown_connections 2024-04-05T21:22:21.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:21.779+0000 7ffa887ef700 1 -- 172.21.15.121:0/2575256884 wait complete. 2024-04-05T21:22:22.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:22 smithi121 ceph-mon[39723]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:22.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:22 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2575256884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T21:22:22.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:22 smithi188 ceph-mon[49715]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:22.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:22 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2575256884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T21:22:22.682 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T21:20:53.291734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T21:22:22.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-05T21:22:22.911 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:24.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:24 smithi188 ceph-mon[49715]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:24.695 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:24 smithi121 ceph-mon[39723]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:25.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:24.999+0000 7f3eb7202700 1 -- 172.21.15.121:0/621084902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3eb0100dc0 msgr2=0x7f3eb01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:25.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:24.999+0000 7f3eb7202700 1 --2- 172.21.15.121:0/621084902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3eb0100dc0 0x7f3eb01011a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009910 tx=0x7f3ea4009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:25.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.000+0000 7f3eb7202700 1 -- 172.21.15.121:0/621084902 shutdown_connections 2024-04-05T21:22:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.000+0000 7f3eb7202700 1 --2- 172.21.15.121:0/621084902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3eb0101770 0x7f3eb01057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.000+0000 7f3eb7202700 1 --2- 172.21.15.121:0/621084902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3eb0100dc0 0x7f3eb01011a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.000+0000 7f3eb7202700 1 -- 172.21.15.121:0/621084902 >> 172.21.15.121:0/621084902 conn(0x7f3eb00fc670 msgr2=0x7f3eb00fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.000+0000 7f3eb7202700 1 -- 172.21.15.121:0/621084902 shutdown_connections 2024-04-05T21:22:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.000+0000 7f3eb7202700 1 -- 172.21.15.121:0/621084902 wait complete. 2024-04-05T21:22:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.000+0000 7f3eb7202700 1 Processor -- start 2024-04-05T21:22:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.000+0000 7f3eb7202700 1 -- start start 2024-04-05T21:22:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.000+0000 7f3eb7202700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3eb0100dc0 0x7f3eb0196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.000+0000 7f3eb7202700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3eb0101770 0x7f3eb0196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.000+0000 7f3eb7202700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb01972e0 con 0x7f3eb0100dc0 2024-04-05T21:22:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.000+0000 7f3eb7202700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb019a010 con 0x7f3eb0101770 2024-04-05T21:22:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.000+0000 7f3eb4f9e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3eb0100dc0 0x7f3eb0196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.001+0000 7f3eb4f9e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3eb0100dc0 0x7f3eb0196710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35878/0 (socket says 172.21.15.121:35878) 2024-04-05T21:22:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.001+0000 7f3eb4f9e700 1 -- 172.21.15.121:0/2382150959 learned_addr learned my addr 172.21.15.121:0/2382150959 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:22:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.001+0000 7f3eaffff700 1 --2- 172.21.15.121:0/2382150959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3eb0101770 0x7f3eb0196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.001+0000 7f3eb4f9e700 1 -- 172.21.15.121:0/2382150959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3eb0101770 msgr2=0x7f3eb0196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.001+0000 7f3eb4f9e700 1 --2- 172.21.15.121:0/2382150959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3eb0101770 0x7f3eb0196c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.001+0000 7f3eb4f9e700 1 -- 172.21.15.121:0/2382150959 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea40095f0 con 0x7f3eb0100dc0 2024-04-05T21:22:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.001+0000 7f3eb4f9e700 1 --2- 172.21.15.121:0/2382150959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3eb0100dc0 0x7f3eb0196710 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4006010 tx=0x7f3ea400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.001+0000 7f3eadffb700 1 -- 172.21.15.121:0/2382150959 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ea401d070 con 0x7f3eb0100dc0 2024-04-05T21:22:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.001+0000 7f3eadffb700 1 -- 172.21.15.121:0/2382150959 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f3ea4005710 con 0x7f3eb0100dc0 2024-04-05T21:22:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.001+0000 7f3eb7202700 1 -- 172.21.15.121:0/2382150959 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb019a290 con 0x7f3eb0100dc0 2024-04-05T21:22:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.001+0000 7f3eadffb700 1 -- 172.21.15.121:0/2382150959 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ea400fdc0 con 0x7f3eb0100dc0 2024-04-05T21:22:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.001+0000 7f3eb7202700 1 -- 172.21.15.121:0/2382150959 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb019a780 con 0x7f3eb0100dc0 2024-04-05T21:22:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.002+0000 7f3eb7202700 1 -- 172.21.15.121:0/2382150959 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb0109160 con 0x7f3eb0100dc0 2024-04-05T21:22:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.003+0000 7f3eadffb700 1 -- 172.21.15.121:0/2382150959 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3ea400f5f0 con 0x7f3eb0100dc0 2024-04-05T21:22:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.003+0000 7f3eadffb700 1 --2- 172.21.15.121:0/2382150959 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3ea006c4e0 0x7f3ea006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.003+0000 7f3eaffff700 1 --2- 172.21.15.121:0/2382150959 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3ea006c4e0 0x7f3ea006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:25.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.003+0000 7f3eadffb700 1 -- 172.21.15.121:0/2382150959 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3ea408c6e0 con 0x7f3eb0100dc0 2024-04-05T21:22:25.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.004+0000 7f3eaffff700 1 --2- 172.21.15.121:0/2382150959 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3ea006c4e0 0x7f3ea006e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0197d30 tx=0x7f3e9c00b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:25.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.008+0000 7f3eadffb700 1 -- 172.21.15.121:0/2382150959 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3ea4057200 con 0x7f3eb0100dc0 2024-04-05T21:22:25.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.197+0000 7f3eb7202700 1 -- 172.21.15.121:0/2382150959 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3eb0197b40 con 0x7f3eb0100dc0 2024-04-05T21:22:25.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.197+0000 7f3eadffb700 1 -- 172.21.15.121:0/2382150959 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3ea405a820 con 0x7f3eb0100dc0 2024-04-05T21:22:25.198 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2024-04-05T21:22:25.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.199+0000 7f3eb7202700 1 -- 172.21.15.121:0/2382150959 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3ea006c4e0 msgr2=0x7f3ea006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:25.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.199+0000 7f3eb7202700 1 --2- 172.21.15.121:0/2382150959 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3ea006c4e0 0x7f3ea006e9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3eb0197d30 tx=0x7f3e9c00b410 comp rx=0 tx=0).stop 2024-04-05T21:22:25.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.199+0000 7f3eb7202700 1 -- 172.21.15.121:0/2382150959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3eb0100dc0 msgr2=0x7f3eb0196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:25.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.199+0000 7f3eb7202700 1 --2- 172.21.15.121:0/2382150959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3eb0100dc0 0x7f3eb0196710 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4006010 tx=0x7f3ea400bd70 comp rx=0 tx=0).stop 2024-04-05T21:22:25.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.199+0000 7f3eb7202700 1 -- 172.21.15.121:0/2382150959 shutdown_connections 2024-04-05T21:22:25.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.199+0000 7f3eb7202700 1 --2- 172.21.15.121:0/2382150959 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3ea006c4e0 0x7f3ea006e9a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:25.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.199+0000 7f3eb7202700 1 --2- 172.21.15.121:0/2382150959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3eb0100dc0 0x7f3eb0196710 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:25.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.199+0000 7f3eb7202700 1 --2- 172.21.15.121:0/2382150959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3eb0101770 0x7f3eb0196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:25.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.199+0000 7f3eb7202700 1 -- 172.21.15.121:0/2382150959 >> 172.21.15.121:0/2382150959 conn(0x7f3eb00fc670 msgr2=0x7f3eb00fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:25.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.199+0000 7f3eb7202700 1 -- 172.21.15.121:0/2382150959 shutdown_connections 2024-04-05T21:22:25.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:25.199+0000 7f3eb7202700 1 -- 172.21.15.121:0/2382150959 wait complete. 2024-04-05T21:22:25.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:22:25.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:25 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2382150959' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T21:22:25.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:25 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:22:25.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:25 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2382150959' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T21:22:26.008 INFO:tasks.cephadm:Setting up client nodes... 2024-04-05T21:22:26.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-05T21:22:26.235 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:26.516 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:26 smithi121 ceph-mon[39723]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:26.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:26 smithi188 ceph-mon[49715]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:28.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.157+0000 7ff676bc5700 1 -- 172.21.15.121:0/22296932 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff670101db0 msgr2=0x7ff670102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:28.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.157+0000 7ff676bc5700 1 --2- 172.21.15.121:0/22296932 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff670101db0 0x7ff670102210 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff65c009910 tx=0x7ff65c009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:28.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.157+0000 7ff676bc5700 1 -- 172.21.15.121:0/22296932 shutdown_connections 2024-04-05T21:22:28.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.157+0000 7ff676bc5700 1 --2- 172.21.15.121:0/22296932 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff670101db0 0x7ff670102210 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:28.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.157+0000 7ff676bc5700 1 --2- 172.21.15.121:0/22296932 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff670100bb0 0x7ff670100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:28.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.157+0000 7ff676bc5700 1 -- 172.21.15.121:0/22296932 >> 172.21.15.121:0/22296932 conn(0x7ff6700fc110 msgr2=0x7ff6700fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:28.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.157+0000 7ff676bc5700 1 -- 172.21.15.121:0/22296932 shutdown_connections 2024-04-05T21:22:28.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.157+0000 7ff676bc5700 1 -- 172.21.15.121:0/22296932 wait complete. 2024-04-05T21:22:28.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff676bc5700 1 Processor -- start 2024-04-05T21:22:28.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff676bc5700 1 -- start start 2024-04-05T21:22:28.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff676bc5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff670100bb0 0x7ff670074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:28.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff676bc5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff670101db0 0x7ff670073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:28.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff676bc5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6700736c0 con 0x7ff670100bb0 2024-04-05T21:22:28.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff676bc5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff670073800 con 0x7ff670101db0 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff674961700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff670100bb0 0x7ff670074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff674961700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff670100bb0 0x7ff670074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35896/0 (socket says 172.21.15.121:35896) 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff674961700 1 -- 172.21.15.121:0/4148439184 learned_addr learned my addr 172.21.15.121:0/4148439184 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff66ffff700 1 --2- 172.21.15.121:0/4148439184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff670101db0 0x7ff670073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff674961700 1 -- 172.21.15.121:0/4148439184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff670101db0 msgr2=0x7ff670073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff674961700 1 --2- 172.21.15.121:0/4148439184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff670101db0 0x7ff670073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff674961700 1 -- 172.21.15.121:0/4148439184 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c0095f0 con 0x7ff670100bb0 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff66ffff700 1 --2- 172.21.15.121:0/4148439184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff670101db0 0x7ff670073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff674961700 1 --2- 172.21.15.121:0/4148439184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff670100bb0 0x7ff670074b30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff66400c8f0 tx=0x7ff66400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.158+0000 7ff66dffb700 1 -- 172.21.15.121:0/4148439184 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff664007a70 con 0x7ff670100bb0 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.159+0000 7ff676bc5700 1 -- 172.21.15.121:0/4148439184 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff670073ae0 con 0x7ff670100bb0 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.159+0000 7ff66dffb700 1 -- 172.21.15.121:0/4148439184 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7ff664007bd0 con 0x7ff670100bb0 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.159+0000 7ff66dffb700 1 -- 172.21.15.121:0/4148439184 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff664018630 con 0x7ff670100bb0 2024-04-05T21:22:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.159+0000 7ff676bc5700 1 -- 172.21.15.121:0/4148439184 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67019e300 con 0x7ff670100bb0 2024-04-05T21:22:28.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.160+0000 7ff676bc5700 1 -- 172.21.15.121:0/4148439184 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6701095d0 con 0x7ff670100bb0 2024-04-05T21:22:28.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.160+0000 7ff66dffb700 1 -- 172.21.15.121:0/4148439184 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff66400f450 con 0x7ff670100bb0 2024-04-05T21:22:28.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.160+0000 7ff66dffb700 1 --2- 172.21.15.121:0/4148439184 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff66006c4e0 0x7ff66006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:28.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.160+0000 7ff66dffb700 1 -- 172.21.15.121:0/4148439184 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff66408b4a0 con 0x7ff670100bb0 2024-04-05T21:22:28.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.160+0000 7ff66ffff700 1 --2- 172.21.15.121:0/4148439184 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff66006c4e0 0x7ff66006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:28.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.161+0000 7ff66ffff700 1 --2- 172.21.15.121:0/4148439184 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff66006c4e0 0x7ff66006e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00b5c0 tx=0x7ff65c005e20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:28.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.165+0000 7ff66dffb700 1 -- 172.21.15.121:0/4148439184 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff664059a50 con 0x7ff670100bb0 2024-04-05T21:22:28.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.432+0000 7ff676bc5700 1 -- 172.21.15.121:0/4148439184 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7ff67019e8b0 con 0x7ff670100bb0 2024-04-05T21:22:28.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.435+0000 7ff66dffb700 1 -- 172.21.15.121:0/4148439184 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7ff6640595e0 con 0x7ff670100bb0 2024-04-05T21:22:28.437 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2024-04-05T21:22:28.437 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQCUaxBmePXTGRAAa7Lo2puQk7Gng+A1a2FjGQ== 2024-04-05T21:22:28.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.437+0000 7ff676bc5700 1 -- 172.21.15.121:0/4148439184 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff66006c4e0 msgr2=0x7ff66006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:28.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.437+0000 7ff676bc5700 1 --2- 172.21.15.121:0/4148439184 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff66006c4e0 0x7ff66006e9a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00b5c0 tx=0x7ff65c005e20 comp rx=0 tx=0).stop 2024-04-05T21:22:28.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.437+0000 7ff676bc5700 1 -- 172.21.15.121:0/4148439184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff670100bb0 msgr2=0x7ff670074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:28.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.437+0000 7ff676bc5700 1 --2- 172.21.15.121:0/4148439184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff670100bb0 0x7ff670074b30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff66400c8f0 tx=0x7ff66400ccb0 comp rx=0 tx=0).stop 2024-04-05T21:22:28.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.437+0000 7ff676bc5700 1 -- 172.21.15.121:0/4148439184 shutdown_connections 2024-04-05T21:22:28.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.437+0000 7ff676bc5700 1 --2- 172.21.15.121:0/4148439184 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff66006c4e0 0x7ff66006e9a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:28.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.437+0000 7ff676bc5700 1 --2- 172.21.15.121:0/4148439184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff670100bb0 0x7ff670074b30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:28.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.437+0000 7ff676bc5700 1 --2- 172.21.15.121:0/4148439184 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff670101db0 0x7ff670073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:28.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.437+0000 7ff676bc5700 1 -- 172.21.15.121:0/4148439184 >> 172.21.15.121:0/4148439184 conn(0x7ff6700fc110 msgr2=0x7ff670104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:28.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.438+0000 7ff676bc5700 1 -- 172.21.15.121:0/4148439184 shutdown_connections 2024-04-05T21:22:28.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:28.438+0000 7ff676bc5700 1 -- 172.21.15.121:0/4148439184 wait complete. 2024-04-05T21:22:28.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:28 smithi188 ceph-mon[49715]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:28 smithi121 ceph-mon[39723]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:29.200 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:22:29.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-05T21:22:29.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-05T21:22:29.243 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-05T21:22:29.482 INFO:teuthology.orchestra.run.smithi188.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:22:29.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:29 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/4148439184' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T21:22:29.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:29 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/4148439184' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-05T21:22:29.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:29 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/4148439184' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T21:22:29.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:29 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/4148439184' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-05T21:22:30.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:30 smithi188 ceph-mon[49715]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:30 smithi121 ceph-mon[39723]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:31.286 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.280+0000 7ffaaca48700 1 -- 172.21.15.188:0/2460351882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa80ffab0 msgr2=0x7ffaa80ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:31.286 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.280+0000 7ffaaca48700 1 --2- 172.21.15.188:0/2460351882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa80ffab0 0x7ffaa80ffed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009910 tx=0x7ffa9c009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:31.286 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.280+0000 7ffaaca48700 1 -- 172.21.15.188:0/2460351882 shutdown_connections 2024-04-05T21:22:31.286 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.280+0000 7ffaaca48700 1 --2- 172.21.15.188:0/2460351882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffaa8100cb0 0x7ffaa8101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:31.286 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.280+0000 7ffaaca48700 1 --2- 172.21.15.188:0/2460351882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa80ffab0 0x7ffaa80ffed0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:31.286 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.280+0000 7ffaaca48700 1 -- 172.21.15.188:0/2460351882 >> 172.21.15.188:0/2460351882 conn(0x7ffaa80fb050 msgr2=0x7ffaa80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:31.286 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.280+0000 7ffaaca48700 1 -- 172.21.15.188:0/2460351882 shutdown_connections 2024-04-05T21:22:31.286 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.280+0000 7ffaaca48700 1 -- 172.21.15.188:0/2460351882 wait complete. 2024-04-05T21:22:31.286 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.281+0000 7ffaaca48700 1 Processor -- start 2024-04-05T21:22:31.286 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.281+0000 7ffaaca48700 1 -- start start 2024-04-05T21:22:31.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.281+0000 7ffaaca48700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffaa80ffab0 0x7ffaa81962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:31.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.281+0000 7ffaaca48700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa8100cb0 0x7ffaa81967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:31.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.281+0000 7ffaaca48700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa8196e00 con 0x7ffaa80ffab0 2024-04-05T21:22:31.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.281+0000 7ffaaca48700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa8196f40 con 0x7ffaa8100cb0 2024-04-05T21:22:31.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.281+0000 7ffaa5d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa8100cb0 0x7ffaa81967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:31.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.281+0000 7ffaa5d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa8100cb0 0x7ffaa81967e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.188:59766/0 (socket says 172.21.15.188:59766) 2024-04-05T21:22:31.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.281+0000 7ffaa5d9b700 1 -- 172.21.15.188:0/2529062586 learned_addr learned my addr 172.21.15.188:0/2529062586 (peer_addr_for_me v2:172.21.15.188:0/0) 2024-04-05T21:22:31.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.281+0000 7ffaa659c700 1 --2- 172.21.15.188:0/2529062586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffaa80ffab0 0x7ffaa81962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:31.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.282+0000 7ffaa5d9b700 1 -- 172.21.15.188:0/2529062586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffaa80ffab0 msgr2=0x7ffaa81962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:31.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.282+0000 7ffaa5d9b700 1 --2- 172.21.15.188:0/2529062586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffaa80ffab0 0x7ffaa81962a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:31.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.282+0000 7ffaa5d9b700 1 -- 172.21.15.188:0/2529062586 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c0095f0 con 0x7ffaa8100cb0 2024-04-05T21:22:31.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.282+0000 7ffaa5d9b700 1 --2- 172.21.15.188:0/2529062586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa8100cb0 0x7ffaa81967e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800e940 tx=0x7ffa9800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:31.287 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.282+0000 7ffa977fe700 1 -- 172.21.15.188:0/2529062586 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa980097b0 con 0x7ffaa8100cb0 2024-04-05T21:22:31.288 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.282+0000 7ffa977fe700 1 -- 172.21.15.188:0/2529062586 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7ffa98004830 con 0x7ffaa8100cb0 2024-04-05T21:22:31.288 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.282+0000 7ffaaca48700 1 -- 172.21.15.188:0/2529062586 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa819b9f0 con 0x7ffaa8100cb0 2024-04-05T21:22:31.288 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.282+0000 7ffa977fe700 1 -- 172.21.15.188:0/2529062586 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa980055b0 con 0x7ffaa8100cb0 2024-04-05T21:22:31.288 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.282+0000 7ffaaca48700 1 -- 172.21.15.188:0/2529062586 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa819bfc0 con 0x7ffaa8100cb0 2024-04-05T21:22:31.288 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.283+0000 7ffaaca48700 1 -- 172.21.15.188:0/2529062586 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa81084f0 con 0x7ffaa8100cb0 2024-04-05T21:22:31.288 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.283+0000 7ffa977fe700 1 -- 172.21.15.188:0/2529062586 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ffa980049a0 con 0x7ffaa8100cb0 2024-04-05T21:22:31.288 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.284+0000 7ffa977fe700 1 --2- 172.21.15.188:0/2529062586 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffa9006c4e0 0x7ffa9006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:31.288 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.284+0000 7ffa977fe700 1 -- 172.21.15.188:0/2529062586 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ffa98014070 con 0x7ffaa8100cb0 2024-04-05T21:22:31.288 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.284+0000 7ffaa659c700 1 --2- 172.21.15.188:0/2529062586 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffa9006c4e0 0x7ffa9006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:31.288 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.284+0000 7ffaa659c700 1 --2- 172.21.15.188:0/2529062586 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffa9006c4e0 0x7ffa9006e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c00b600 tx=0x7ffa9c009f70 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:31.289 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.288+0000 7ffa977fe700 1 -- 172.21.15.188:0/2529062586 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ffa98059e60 con 0x7ffaa8100cb0 2024-04-05T21:22:31.573 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.572+0000 7ffaaca48700 1 -- 172.21.15.188:0/2529062586 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7ffaa819c7b0 con 0x7ffaa8100cb0 2024-04-05T21:22:31.578 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.577+0000 7ffa977fe700 1 -- 172.21.15.188:0/2529062586 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7ffa980599f0 con 0x7ffaa8100cb0 2024-04-05T21:22:31.578 INFO:teuthology.orchestra.run.smithi188.stdout:[client.1] 2024-04-05T21:22:31.578 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQCXaxBmOSQ5IhAAODDWwUzJTFbNGV7fMkDl4g== 2024-04-05T21:22:31.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.579+0000 7ffaaca48700 1 -- 172.21.15.188:0/2529062586 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffa9006c4e0 msgr2=0x7ffa9006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:31.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.579+0000 7ffaaca48700 1 --2- 172.21.15.188:0/2529062586 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffa9006c4e0 0x7ffa9006e9a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c00b600 tx=0x7ffa9c009f70 comp rx=0 tx=0).stop 2024-04-05T21:22:31.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.579+0000 7ffaaca48700 1 -- 172.21.15.188:0/2529062586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa8100cb0 msgr2=0x7ffaa81967e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:31.580 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.579+0000 7ffaaca48700 1 --2- 172.21.15.188:0/2529062586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa8100cb0 0x7ffaa81967e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffa9800e940 tx=0x7ffa9800ed00 comp rx=0 tx=0).stop 2024-04-05T21:22:31.581 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.579+0000 7ffaaca48700 1 -- 172.21.15.188:0/2529062586 shutdown_connections 2024-04-05T21:22:31.581 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.579+0000 7ffaaca48700 1 --2- 172.21.15.188:0/2529062586 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffa9006c4e0 0x7ffa9006e9a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:31.581 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.579+0000 7ffaaca48700 1 --2- 172.21.15.188:0/2529062586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffaa80ffab0 0x7ffaa81962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:31.581 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.579+0000 7ffaaca48700 1 --2- 172.21.15.188:0/2529062586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffaa8100cb0 0x7ffaa81967e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:31.581 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.579+0000 7ffaaca48700 1 -- 172.21.15.188:0/2529062586 >> 172.21.15.188:0/2529062586 conn(0x7ffaa80fb050 msgr2=0x7ffaa8068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:31.581 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.579+0000 7ffaaca48700 1 -- 172.21.15.188:0/2529062586 shutdown_connections 2024-04-05T21:22:31.581 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-05T21:22:31.579+0000 7ffaaca48700 1 -- 172.21.15.188:0/2529062586 wait complete. 2024-04-05T21:22:32.289 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-05T21:22:32.289 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-05T21:22:32.289 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-05T21:22:32.331 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-05T21:22:32.331 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-05T21:22:32.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-05T21:22:32.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:32 smithi188 ceph-mon[49715]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:32.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:32 smithi188 ceph-mon[49715]: from='client.? 172.21.15.188:0/2529062586' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T21:22:32.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:32 smithi188 ceph-mon[49715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T21:22:32.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:32 smithi188 ceph-mon[49715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-05T21:22:32.567 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:32 smithi121 ceph-mon[39723]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:32 smithi121 ceph-mon[39723]: from='client.? 172.21.15.188:0/2529062586' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T21:22:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:32 smithi121 ceph-mon[39723]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-05T21:22:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:32 smithi121 ceph-mon[39723]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-05T21:22:34.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:34 smithi188 ceph-mon[49715]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:34.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.543+0000 7ff3390af700 1 -- 172.21.15.121:0/4264539529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3340ffa40 msgr2=0x7ff3340ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:34.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.543+0000 7ff3390af700 1 --2- 172.21.15.121:0/4264539529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3340ffa40 0x7ff3340ffec0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff324009910 tx=0x7ff324009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:34.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.543+0000 7ff3390af700 1 -- 172.21.15.121:0/4264539529 shutdown_connections 2024-04-05T21:22:34.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.543+0000 7ff3390af700 1 --2- 172.21.15.121:0/4264539529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3340ffa40 0x7ff3340ffec0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:34.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.543+0000 7ff3390af700 1 --2- 172.21.15.121:0/4264539529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff334105a40 0x7ff334105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:34.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.543+0000 7ff3390af700 1 -- 172.21.15.121:0/4264539529 >> 172.21.15.121:0/4264539529 conn(0x7ff3340fb570 msgr2=0x7ff3340fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:34.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.543+0000 7ff3390af700 1 -- 172.21.15.121:0/4264539529 shutdown_connections 2024-04-05T21:22:34.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.543+0000 7ff3390af700 1 -- 172.21.15.121:0/4264539529 wait complete. 2024-04-05T21:22:34.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.544+0000 7ff3390af700 1 Processor -- start 2024-04-05T21:22:34.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.544+0000 7ff3390af700 1 -- start start 2024-04-05T21:22:34.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.544+0000 7ff3390af700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3340ffa40 0x7ff3340fbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:34.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.544+0000 7ff3390af700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff334105a40 0x7ff3340fc480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:34.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.544+0000 7ff3390af700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3340fc9c0 con 0x7ff334105a40 2024-04-05T21:22:34.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.544+0000 7ff3390af700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3340fcb00 con 0x7ff3340ffa40 2024-04-05T21:22:34.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.544+0000 7ff33259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff334105a40 0x7ff3340fc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:34.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.544+0000 7ff33259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff334105a40 0x7ff3340fc480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43028/0 (socket says 172.21.15.121:43028) 2024-04-05T21:22:34.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.544+0000 7ff33259c700 1 -- 172.21.15.121:0/3246655335 learned_addr learned my addr 172.21.15.121:0/3246655335 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:22:34.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.545+0000 7ff332d9d700 1 --2- 172.21.15.121:0/3246655335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3340ffa40 0x7ff3340fbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:34.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.545+0000 7ff33259c700 1 -- 172.21.15.121:0/3246655335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3340ffa40 msgr2=0x7ff3340fbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:34.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.545+0000 7ff33259c700 1 --2- 172.21.15.121:0/3246655335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3340ffa40 0x7ff3340fbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:34.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.545+0000 7ff33259c700 1 -- 172.21.15.121:0/3246655335 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3240095f0 con 0x7ff334105a40 2024-04-05T21:22:34.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.545+0000 7ff33259c700 1 --2- 172.21.15.121:0/3246655335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff334105a40 0x7ff3340fc480 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff324006010 tx=0x7ff32400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:34.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.545+0000 7ff31bfff700 1 -- 172.21.15.121:0/3246655335 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff32401d070 con 0x7ff334105a40 2024-04-05T21:22:34.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.545+0000 7ff31bfff700 1 -- 172.21.15.121:0/3246655335 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7ff32400f460 con 0x7ff334105a40 2024-04-05T21:22:34.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.545+0000 7ff3390af700 1 -- 172.21.15.121:0/3246655335 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3340fcd80 con 0x7ff334105a40 2024-04-05T21:22:34.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.545+0000 7ff31bfff700 1 -- 172.21.15.121:0/3246655335 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff324021640 con 0x7ff334105a40 2024-04-05T21:22:34.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.545+0000 7ff3390af700 1 -- 172.21.15.121:0/3246655335 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3341a2b10 con 0x7ff334105a40 2024-04-05T21:22:34.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.546+0000 7ff3390af700 1 -- 172.21.15.121:0/3246655335 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff334108fe0 con 0x7ff334105a40 2024-04-05T21:22:34.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.546+0000 7ff31bfff700 1 -- 172.21.15.121:0/3246655335 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff32400fa60 con 0x7ff334105a40 2024-04-05T21:22:34.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.547+0000 7ff31bfff700 1 --2- 172.21.15.121:0/3246655335 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff31c06c4e0 0x7ff31c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:34.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.547+0000 7ff31bfff700 1 -- 172.21.15.121:0/3246655335 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ff32408c820 con 0x7ff334105a40 2024-04-05T21:22:34.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.547+0000 7ff332d9d700 1 --2- 172.21.15.121:0/3246655335 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff31c06c4e0 0x7ff31c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:34.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.547+0000 7ff332d9d700 1 --2- 172.21.15.121:0/3246655335 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff31c06c4e0 0x7ff31c06e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff328005fd0 tx=0x7ff32800a560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:34.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.551+0000 7ff31bfff700 1 -- 172.21.15.121:0/3246655335 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff324057340 con 0x7ff334105a40 2024-04-05T21:22:34.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:34 smithi121 ceph-mon[39723]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:34.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.794+0000 7ff3390af700 1 -- 172.21.15.121:0/3246655335 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff33404ea90 con 0x7ff334105a40 2024-04-05T21:22:34.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.797+0000 7ff31bfff700 1 -- 172.21.15.121:0/3246655335 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+173046 (secure 0 0 0) 0x7ff32405a960 con 0x7ff334105a40 2024-04-05T21:22:34.800 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:22:34.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.799+0000 7ff3390af700 1 -- 172.21.15.121:0/3246655335 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff31c06c4e0 msgr2=0x7ff31c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:34.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.799+0000 7ff3390af700 1 --2- 172.21.15.121:0/3246655335 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff31c06c4e0 0x7ff31c06e9a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff328005fd0 tx=0x7ff32800a560 comp rx=0 tx=0).stop 2024-04-05T21:22:34.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.799+0000 7ff3390af700 1 -- 172.21.15.121:0/3246655335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff334105a40 msgr2=0x7ff3340fc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:34.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.799+0000 7ff3390af700 1 --2- 172.21.15.121:0/3246655335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff334105a40 0x7ff3340fc480 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff324006010 tx=0x7ff32400be20 comp rx=0 tx=0).stop 2024-04-05T21:22:34.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.799+0000 7ff3390af700 1 -- 172.21.15.121:0/3246655335 shutdown_connections 2024-04-05T21:22:34.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.799+0000 7ff3390af700 1 --2- 172.21.15.121:0/3246655335 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff31c06c4e0 0x7ff31c06e9a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:34.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.799+0000 7ff3390af700 1 --2- 172.21.15.121:0/3246655335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3340ffa40 0x7ff3340fbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:34.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.800+0000 7ff3390af700 1 --2- 172.21.15.121:0/3246655335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff334105a40 0x7ff3340fc480 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:34.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.800+0000 7ff3390af700 1 -- 172.21.15.121:0/3246655335 >> 172.21.15.121:0/3246655335 conn(0x7ff3340fb570 msgr2=0x7ff334068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:34.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.800+0000 7ff3390af700 1 -- 172.21.15.121:0/3246655335 shutdown_connections 2024-04-05T21:22:34.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:34.800+0000 7ff3390af700 1 -- 172.21.15.121:0/3246655335 wait complete. 2024-04-05T21:22:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:35 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3246655335' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-05T21:22:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:35 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/3246655335' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-05T21:22:35.692 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":19,"active_gid":14211,"active_name":"smithi121.etbquh","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2735531747},{"type":"v1","addr":"172.21.15.121:6801","nonce":2735531747}]},"active_addr":"172.21.15.121:6801/2735531747","active_change":"2024-04-05T21:18:24.954815+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14232,"name":"smithi188.tgcuir","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.121:8443/","prometheus":"http://172.21.15.121:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":566857212}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":4016372676}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":1060423936}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":846460013}]}]} 2024-04-05T21:22:35.700 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-05T21:22:35.700 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-05T21:22:35.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-05T21:22:35.929 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:36 smithi121 ceph-mon[39723]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:36 smithi188 ceph-mon[49715]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:37.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.770+0000 7f06b1fca700 1 -- 172.21.15.121:0/716685800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06ac101da0 msgr2=0x7f06ac102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.770+0000 7f06b1fca700 1 --2- 172.21.15.121:0/716685800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06ac101da0 0x7f06ac102180 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009910 tx=0x7f06a0009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.770+0000 7f06b1fca700 1 -- 172.21.15.121:0/716685800 shutdown_connections 2024-04-05T21:22:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.770+0000 7f06b1fca700 1 --2- 172.21.15.121:0/716685800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06ac1026c0 0x7f06ac10abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.770+0000 7f06b1fca700 1 --2- 172.21.15.121:0/716685800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06ac101da0 0x7f06ac102180 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.770+0000 7f06b1fca700 1 -- 172.21.15.121:0/716685800 >> 172.21.15.121:0/716685800 conn(0x7f06ac0fb570 msgr2=0x7f06ac0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.770+0000 7f06b1fca700 1 -- 172.21.15.121:0/716685800 shutdown_connections 2024-04-05T21:22:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.770+0000 7f06b1fca700 1 -- 172.21.15.121:0/716685800 wait complete. 2024-04-05T21:22:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.770+0000 7f06b1fca700 1 Processor -- start 2024-04-05T21:22:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06b1fca700 1 -- start start 2024-04-05T21:22:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06b1fca700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06ac101da0 0x7f06ac1966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06b1fca700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06ac1026c0 0x7f06ac196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06b1fca700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac197270 con 0x7f06ac101da0 2024-04-05T21:22:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06b1fca700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac199fa0 con 0x7f06ac1026c0 2024-04-05T21:22:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06ab7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06ac101da0 0x7f06ac1966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06ab7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06ac101da0 0x7f06ac1966a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43044/0 (socket says 172.21.15.121:43044) 2024-04-05T21:22:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06ab7fe700 1 -- 172.21.15.121:0/2717752064 learned_addr learned my addr 172.21.15.121:0/2717752064 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:22:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06aaffd700 1 --2- 172.21.15.121:0/2717752064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06ac1026c0 0x7f06ac196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06ab7fe700 1 -- 172.21.15.121:0/2717752064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06ac1026c0 msgr2=0x7f06ac196be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06ab7fe700 1 --2- 172.21.15.121:0/2717752064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06ac1026c0 0x7f06ac196be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06ab7fe700 1 -- 172.21.15.121:0/2717752064 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a00095f0 con 0x7f06ac101da0 2024-04-05T21:22:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06aaffd700 1 --2- 172.21.15.121:0/2717752064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06ac1026c0 0x7f06ac196be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:22:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06ab7fe700 1 --2- 172.21.15.121:0/2717752064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06ac101da0 0x7f06ac1966a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f06a0006010 tx=0x7f06a000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06a8ff9700 1 -- 172.21.15.121:0/2717752064 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06a001d070 con 0x7f06ac101da0 2024-04-05T21:22:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06a8ff9700 1 -- 172.21.15.121:0/2717752064 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f06a000f460 con 0x7f06ac101da0 2024-04-05T21:22:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06b1fca700 1 -- 172.21.15.121:0/2717752064 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ac19a220 con 0x7f06ac101da0 2024-04-05T21:22:37.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06a8ff9700 1 -- 172.21.15.121:0/2717752064 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06a0021620 con 0x7f06ac101da0 2024-04-05T21:22:37.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.771+0000 7f06b1fca700 1 -- 172.21.15.121:0/2717752064 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ac19a710 con 0x7f06ac101da0 2024-04-05T21:22:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.772+0000 7f06b1fca700 1 -- 172.21.15.121:0/2717752064 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ac1082b0 con 0x7f06ac101da0 2024-04-05T21:22:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.773+0000 7f06a8ff9700 1 -- 172.21.15.121:0/2717752064 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f06a000f5d0 con 0x7f06ac101da0 2024-04-05T21:22:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.773+0000 7f06a8ff9700 1 --2- 172.21.15.121:0/2717752064 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f069406c4e0 0x7f069406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.773+0000 7f06a8ff9700 1 -- 172.21.15.121:0/2717752064 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f06a008c7b0 con 0x7f06ac101da0 2024-04-05T21:22:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.773+0000 7f06aaffd700 1 --2- 172.21.15.121:0/2717752064 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f069406c4e0 0x7f069406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.774+0000 7f06aaffd700 1 --2- 172.21.15.121:0/2717752064 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f069406c4e0 0x7f069406e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f069c005fd0 tx=0x7f069c005dc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:37.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.778+0000 7f06a8ff9700 1 -- 172.21.15.121:0/2717752064 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f06a00572d0 con 0x7f06ac101da0 2024-04-05T21:22:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.962+0000 7f06b1fca700 1 -- 172.21.15.121:0/2717752064 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06ac04ea90 con 0x7f06ac101da0 2024-04-05T21:22:37.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.963+0000 7f06a8ff9700 1 -- 172.21.15.121:0/2717752064 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11385 (secure 0 0 0) 0x7f06a005a8f0 con 0x7f06ac101da0 2024-04-05T21:22:37.965 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:22:37.965 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":40,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","created":"2024-04-05T21:15:58.797109+0000","modified":"2024-04-05T21:22:18.067824+0000","last_up_change":"2024-04-05T21:22:17.062975+0000","last_in_change":"2024-04-05T21:21:56.806570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T21:20:53.291734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"133652f8-9fe2-45e9-90b6-36f89191114b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6803","nonce":2285715890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6805","nonce":2285715890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6809","nonce":2285715890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6807","nonce":2285715890}]},"public_addr":"172.21.15.121:6803/2285715890","cluster_addr":"172.21.15.121:6805/2285715890","heartbeat_back_addr":"172.21.15.121:6809/2285715890","heartbeat_front_addr":"172.21.15.121:6807/2285715890","state":["exists","up"]},{"osd":1,"uuid":"6695518f-c4a5-416a-8af1-2899fdebefd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6811","nonce":3573482215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6813","nonce":3573482215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6817","nonce":3573482215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6815","nonce":3573482215}]},"public_addr":"172.21.15.121:6811/3573482215","cluster_addr":"172.21.15.121:6813/3573482215","heartbeat_back_addr":"172.21.15.121:6817/3573482215","heartbeat_front_addr":"172.21.15.121:6815/3573482215","state":["exists","up"]},{"osd":2,"uuid":"e3b6290b-ef26-4cda-96e2-8bb5649219a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6819","nonce":179235010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6821","nonce":179235010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6825","nonce":179235010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6823","nonce":179235010}]},"public_addr":"172.21.15.121:6819/179235010","cluster_addr":"172.21.15.121:6821/179235010","heartbeat_back_addr":"172.21.15.121:6825/179235010","heartbeat_front_addr":"172.21.15.121:6823/179235010","state":["exists","up"]},{"osd":3,"uuid":"77fa8da5-d202-45f1-b6fa-5af969e9300c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6801","nonce":2760393079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6803","nonce":2760393079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6807","nonce":2760393079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6805","nonce":2760393079}]},"public_addr":"172.21.15.188:6801/2760393079","cluster_addr":"172.21.15.188:6803/2760393079","heartbeat_back_addr":"172.21.15.188:6807/2760393079","heartbeat_front_addr":"172.21.15.188:6805/2760393079","state":["exists","up"]},{"osd":4,"uuid":"abf81f28-7473-45ef-a716-5ef16e038b6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6809","nonce":1828363110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6811","nonce":1828363110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6815","nonce":1828363110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6813","nonce":1828363110}]},"public_addr":"172.21.15.188:6809/1828363110","cluster_addr":"172.21.15.188:6811/1828363110","heartbeat_back_addr":"172.21.15.188:6815/1828363110","heartbeat_front_addr":"172.21.15.188:6813/1828363110","state":["exists","up"]},{"osd":5,"uuid":"700a347b-e303-447d-bf03-d6677955d140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6817","nonce":3569223975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6819","nonce":3569223975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6823","nonce":3569223975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6821","nonce":3569223975}]},"public_addr":"172.21.15.188:6817/3569223975","cluster_addr":"172.21.15.188:6819/3569223975","heartbeat_back_addr":"172.21.15.188:6823/3569223975","heartbeat_front_addr":"172.21.15.188:6821/3569223975","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:19:48.038490+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:19.269394+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:51.162437+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:18.203341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:46.805653+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:22:15.296548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6801/1237774071":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6800/1237774071":"2024-04-06T21:18:24.954707+0000","172.21.15.121:0/3235459786":"2024-04-06T21:18:24.954707+0000","172.21.15.121:0/3247654758":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6800/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/341396693":"2024-04-06T21:17:08.626094+0000","172.21.15.121:6801/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/2214582362":"2024-04-06T21:17:08.626094+0000","172.21.15.121:6801/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/3762940011":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6800/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/1364585923":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/2536941662":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/2383002272":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/42058318":"2024-04-06T21:17:08.626094+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T21:22:37.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.965+0000 7f06b1fca700 1 -- 172.21.15.121:0/2717752064 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f069406c4e0 msgr2=0x7f069406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:37.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.965+0000 7f06b1fca700 1 --2- 172.21.15.121:0/2717752064 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f069406c4e0 0x7f069406e9a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f069c005fd0 tx=0x7f069c005dc0 comp rx=0 tx=0).stop 2024-04-05T21:22:37.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.965+0000 7f06b1fca700 1 -- 172.21.15.121:0/2717752064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06ac101da0 msgr2=0x7f06ac1966a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:37.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.965+0000 7f06b1fca700 1 --2- 172.21.15.121:0/2717752064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06ac101da0 0x7f06ac1966a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f06a0006010 tx=0x7f06a000be20 comp rx=0 tx=0).stop 2024-04-05T21:22:37.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.965+0000 7f06b1fca700 1 -- 172.21.15.121:0/2717752064 shutdown_connections 2024-04-05T21:22:37.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.965+0000 7f06b1fca700 1 --2- 172.21.15.121:0/2717752064 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f069406c4e0 0x7f069406e9a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:37.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.965+0000 7f06b1fca700 1 --2- 172.21.15.121:0/2717752064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06ac101da0 0x7f06ac1966a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:37.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.965+0000 7f06b1fca700 1 --2- 172.21.15.121:0/2717752064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06ac1026c0 0x7f06ac196be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:37.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.965+0000 7f06b1fca700 1 -- 172.21.15.121:0/2717752064 >> 172.21.15.121:0/2717752064 conn(0x7f06ac0fb570 msgr2=0x7f06ac1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:37.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.965+0000 7f06b1fca700 1 -- 172.21.15.121:0/2717752064 shutdown_connections 2024-04-05T21:22:37.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:37.965+0000 7f06b1fca700 1 -- 172.21.15.121:0/2717752064 wait complete. 2024-04-05T21:22:38.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:38 smithi188 ceph-mon[49715]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:38.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:38 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2717752064' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T21:22:38.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:38 smithi121 ceph-mon[39723]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:38.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:38 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2717752064' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T21:22:38.796 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-05T21:22:38.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-05T21:22:39.022 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:40.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:40 smithi121 ceph-mon[39723]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:40.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:22:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:40 smithi188 ceph-mon[49715]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:40 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:22:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.092+0000 7f6184191700 1 -- 172.21.15.121:0/1642684842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c100dc0 msgr2=0x7f617c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.092+0000 7f6184191700 1 --2- 172.21.15.121:0/1642684842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c100dc0 0x7f617c1011a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f616c009910 tx=0x7f616c009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.092+0000 7f6184191700 1 -- 172.21.15.121:0/1642684842 shutdown_connections 2024-04-05T21:22:41.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.092+0000 7f6184191700 1 --2- 172.21.15.121:0/1642684842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f617c101770 0x7f617c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.092+0000 7f6184191700 1 --2- 172.21.15.121:0/1642684842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c100dc0 0x7f617c1011a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.092+0000 7f6184191700 1 -- 172.21.15.121:0/1642684842 >> 172.21.15.121:0/1642684842 conn(0x7f617c0fc650 msgr2=0x7f617c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.092+0000 7f6184191700 1 -- 172.21.15.121:0/1642684842 shutdown_connections 2024-04-05T21:22:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.092+0000 7f6184191700 1 -- 172.21.15.121:0/1642684842 wait complete. 2024-04-05T21:22:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f6184191700 1 Processor -- start 2024-04-05T21:22:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f6184191700 1 -- start start 2024-04-05T21:22:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f6184191700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f617c100dc0 0x7f617c196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f6184191700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c101770 0x7f617c196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f6184191700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c197280 con 0x7f617c101770 2024-04-05T21:22:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f6184191700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c19b010 con 0x7f617c100dc0 2024-04-05T21:22:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f618172c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c101770 0x7f617c196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:41.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f618172c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c101770 0x7f617c196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43056/0 (socket says 172.21.15.121:43056) 2024-04-05T21:22:41.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f618172c700 1 -- 172.21.15.121:0/3802496295 learned_addr learned my addr 172.21.15.121:0/3802496295 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:22:41.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f6181f2d700 1 --2- 172.21.15.121:0/3802496295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f617c100dc0 0x7f617c196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:41.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f618172c700 1 -- 172.21.15.121:0/3802496295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f617c100dc0 msgr2=0x7f617c196660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:41.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f618172c700 1 --2- 172.21.15.121:0/3802496295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f617c100dc0 0x7f617c196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:41.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f618172c700 1 -- 172.21.15.121:0/3802496295 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0095f0 con 0x7f617c101770 2024-04-05T21:22:41.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f618172c700 1 --2- 172.21.15.121:0/3802496295 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c101770 0x7f617c196ba0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f617800c8f0 tx=0x7f617800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:41.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f6172ffd700 1 -- 172.21.15.121:0/3802496295 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6178007a50 con 0x7f617c101770 2024-04-05T21:22:41.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f6172ffd700 1 -- 172.21.15.121:0/3802496295 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f6178007bb0 con 0x7f617c101770 2024-04-05T21:22:41.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f6184191700 1 -- 172.21.15.121:0/3802496295 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617c19b2f0 con 0x7f617c101770 2024-04-05T21:22:41.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f6172ffd700 1 -- 172.21.15.121:0/3802496295 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f617800fea0 con 0x7f617c101770 2024-04-05T21:22:41.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.093+0000 7f6184191700 1 -- 172.21.15.121:0/3802496295 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617c19b840 con 0x7f617c101770 2024-04-05T21:22:41.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.094+0000 7f6184191700 1 -- 172.21.15.121:0/3802496295 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617c109160 con 0x7f617c101770 2024-04-05T21:22:41.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.095+0000 7f6172ffd700 1 -- 172.21.15.121:0/3802496295 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f6178007d20 con 0x7f617c101770 2024-04-05T21:22:41.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.095+0000 7f6172ffd700 1 --2- 172.21.15.121:0/3802496295 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f616806c4e0 0x7f616806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:41.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.095+0000 7f6172ffd700 1 -- 172.21.15.121:0/3802496295 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f617808b490 con 0x7f617c101770 2024-04-05T21:22:41.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.095+0000 7f6181f2d700 1 --2- 172.21.15.121:0/3802496295 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f616806c4e0 0x7f616806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:41.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.096+0000 7f6181f2d700 1 --2- 172.21.15.121:0/3802496295 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f616806c4e0 0x7f616806e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f616c00b5c0 tx=0x7f616c005fd0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:41.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.100+0000 7f6172ffd700 1 -- 172.21.15.121:0/3802496295 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6178055fb0 con 0x7f617c101770 2024-04-05T21:22:41.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.288+0000 7f6184191700 1 -- 172.21.15.121:0/3802496295 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f617c04ea90 con 0x7f617c101770 2024-04-05T21:22:41.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.289+0000 7f6172ffd700 1 -- 172.21.15.121:0/3802496295 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11385 (secure 0 0 0) 0x7f61780595d0 con 0x7f617c101770 2024-04-05T21:22:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:22:41.290 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":40,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","created":"2024-04-05T21:15:58.797109+0000","modified":"2024-04-05T21:22:18.067824+0000","last_up_change":"2024-04-05T21:22:17.062975+0000","last_in_change":"2024-04-05T21:21:56.806570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T21:20:53.291734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"133652f8-9fe2-45e9-90b6-36f89191114b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6803","nonce":2285715890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6805","nonce":2285715890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6809","nonce":2285715890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6807","nonce":2285715890}]},"public_addr":"172.21.15.121:6803/2285715890","cluster_addr":"172.21.15.121:6805/2285715890","heartbeat_back_addr":"172.21.15.121:6809/2285715890","heartbeat_front_addr":"172.21.15.121:6807/2285715890","state":["exists","up"]},{"osd":1,"uuid":"6695518f-c4a5-416a-8af1-2899fdebefd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6811","nonce":3573482215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6813","nonce":3573482215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6817","nonce":3573482215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6815","nonce":3573482215}]},"public_addr":"172.21.15.121:6811/3573482215","cluster_addr":"172.21.15.121:6813/3573482215","heartbeat_back_addr":"172.21.15.121:6817/3573482215","heartbeat_front_addr":"172.21.15.121:6815/3573482215","state":["exists","up"]},{"osd":2,"uuid":"e3b6290b-ef26-4cda-96e2-8bb5649219a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6819","nonce":179235010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6821","nonce":179235010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6825","nonce":179235010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6823","nonce":179235010}]},"public_addr":"172.21.15.121:6819/179235010","cluster_addr":"172.21.15.121:6821/179235010","heartbeat_back_addr":"172.21.15.121:6825/179235010","heartbeat_front_addr":"172.21.15.121:6823/179235010","state":["exists","up"]},{"osd":3,"uuid":"77fa8da5-d202-45f1-b6fa-5af969e9300c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6801","nonce":2760393079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6803","nonce":2760393079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6807","nonce":2760393079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6805","nonce":2760393079}]},"public_addr":"172.21.15.188:6801/2760393079","cluster_addr":"172.21.15.188:6803/2760393079","heartbeat_back_addr":"172.21.15.188:6807/2760393079","heartbeat_front_addr":"172.21.15.188:6805/2760393079","state":["exists","up"]},{"osd":4,"uuid":"abf81f28-7473-45ef-a716-5ef16e038b6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6809","nonce":1828363110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6811","nonce":1828363110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6815","nonce":1828363110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6813","nonce":1828363110}]},"public_addr":"172.21.15.188:6809/1828363110","cluster_addr":"172.21.15.188:6811/1828363110","heartbeat_back_addr":"172.21.15.188:6815/1828363110","heartbeat_front_addr":"172.21.15.188:6813/1828363110","state":["exists","up"]},{"osd":5,"uuid":"700a347b-e303-447d-bf03-d6677955d140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6817","nonce":3569223975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6819","nonce":3569223975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6823","nonce":3569223975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6821","nonce":3569223975}]},"public_addr":"172.21.15.188:6817/3569223975","cluster_addr":"172.21.15.188:6819/3569223975","heartbeat_back_addr":"172.21.15.188:6823/3569223975","heartbeat_front_addr":"172.21.15.188:6821/3569223975","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:19:48.038490+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:19.269394+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:51.162437+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:18.203341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:46.805653+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:22:15.296548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6801/1237774071":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6800/1237774071":"2024-04-06T21:18:24.954707+0000","172.21.15.121:0/3235459786":"2024-04-06T21:18:24.954707+0000","172.21.15.121:0/3247654758":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6800/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/341396693":"2024-04-06T21:17:08.626094+0000","172.21.15.121:6801/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/2214582362":"2024-04-06T21:17:08.626094+0000","172.21.15.121:6801/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/3762940011":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6800/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/1364585923":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/2536941662":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/2383002272":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/42058318":"2024-04-06T21:17:08.626094+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T21:22:41.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.291+0000 7f6184191700 1 -- 172.21.15.121:0/3802496295 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f616806c4e0 msgr2=0x7f616806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:41.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.291+0000 7f6184191700 1 --2- 172.21.15.121:0/3802496295 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f616806c4e0 0x7f616806e9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f616c00b5c0 tx=0x7f616c005fd0 comp rx=0 tx=0).stop 2024-04-05T21:22:41.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.291+0000 7f6184191700 1 -- 172.21.15.121:0/3802496295 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c101770 msgr2=0x7f617c196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:41.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.291+0000 7f6184191700 1 --2- 172.21.15.121:0/3802496295 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c101770 0x7f617c196ba0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f617800c8f0 tx=0x7f617800ccb0 comp rx=0 tx=0).stop 2024-04-05T21:22:41.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.291+0000 7f6184191700 1 -- 172.21.15.121:0/3802496295 shutdown_connections 2024-04-05T21:22:41.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.291+0000 7f6184191700 1 --2- 172.21.15.121:0/3802496295 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f616806c4e0 0x7f616806e9a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:41.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.291+0000 7f6184191700 1 --2- 172.21.15.121:0/3802496295 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f617c100dc0 0x7f617c196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:41.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.291+0000 7f6184191700 1 --2- 172.21.15.121:0/3802496295 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c101770 0x7f617c196ba0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:41.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.291+0000 7f6184191700 1 -- 172.21.15.121:0/3802496295 >> 172.21.15.121:0/3802496295 conn(0x7f617c0fc650 msgr2=0x7f617c0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:41.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.291+0000 7f6184191700 1 -- 172.21.15.121:0/3802496295 shutdown_connections 2024-04-05T21:22:41.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:41.291+0000 7f6184191700 1 -- 172.21.15.121:0/3802496295 wait complete. 2024-04-05T21:22:42.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-05T21:22:42.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-05T21:22:42.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-05T21:22:42.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-05T21:22:42.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-05T21:22:42.004 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-05T21:22:42.267 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:42.580 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:42 smithi121 ceph-mon[39723]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:42.580 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:42 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3802496295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T21:22:42.653 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:42.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:42 smithi188 ceph-mon[49715]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:42.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:42 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/3802496295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T21:22:44.258 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:44.705 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:44 smithi121 ceph-mon[39723]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:44.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:44 smithi188 ceph-mon[49715]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:45.327 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:45.604 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:46.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:46 smithi121 ceph-mon[39723]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:46.622 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:46 smithi188 ceph-mon[49715]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:48.403 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:48 smithi121 ceph-mon[39723]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:48.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:48 smithi188 ceph-mon[49715]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:49.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 -- 172.21.15.121:0/1473142020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8988105a40 msgr2=0x7f8988105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:49.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 --2- 172.21.15.121:0/1473142020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8988105a40 0x7f8988105e20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8978009910 tx=0x7f8978009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:49.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 -- 172.21.15.121:0/1473142020 shutdown_connections 2024-04-05T21:22:49.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 --2- 172.21.15.121:0/1473142020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f89880ffbd0 0x7f8988100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:49.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 --2- 172.21.15.121:0/1473142020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8988105a40 0x7f8988105e20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:49.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 -- 172.21.15.121:0/1473142020 >> 172.21.15.121:0/1473142020 conn(0x7f89880755a0 msgr2=0x7f89880759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:49.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 -- 172.21.15.121:0/1473142020 shutdown_connections 2024-04-05T21:22:49.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 -- 172.21.15.121:0/1473142020 wait complete. 2024-04-05T21:22:49.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 Processor -- start 2024-04-05T21:22:49.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 -- start start 2024-04-05T21:22:49.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89880ffbd0 0x7f898819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:49.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8988105a40 0x7f898819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:49.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898819d0d0 con 0x7f89880ffbd0 2024-04-05T21:22:49.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.192+0000 7f898f95f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8988196580 con 0x7f8988105a40 2024-04-05T21:22:49.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f898d6fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89880ffbd0 0x7f898819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:49.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f898d6fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89880ffbd0 0x7f898819c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44002/0 (socket says 172.21.15.121:44002) 2024-04-05T21:22:49.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f898d6fb700 1 -- 172.21.15.121:0/177527272 learned_addr learned my addr 172.21.15.121:0/177527272 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:22:49.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f898cefa700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8988105a40 0x7f898819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:49.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f898d6fb700 1 -- 172.21.15.121:0/177527272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8988105a40 msgr2=0x7f898819ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:49.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f898d6fb700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8988105a40 0x7f898819ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:49.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f898d6fb700 1 -- 172.21.15.121:0/177527272 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89780095f0 con 0x7f89880ffbd0 2024-04-05T21:22:49.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f898cefa700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8988105a40 0x7f898819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:22:49.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f898d6fb700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89880ffbd0 0x7f898819c500 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8978006010 tx=0x7f897800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:49.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f897e7fc700 1 -- 172.21.15.121:0/177527272 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f897801d070 con 0x7f89880ffbd0 2024-04-05T21:22:49.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f897e7fc700 1 -- 172.21.15.121:0/177527272 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f897800f460 con 0x7f89880ffbd0 2024-04-05T21:22:49.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f898f95f700 1 -- 172.21.15.121:0/177527272 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8988196800 con 0x7f89880ffbd0 2024-04-05T21:22:49.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f897e7fc700 1 -- 172.21.15.121:0/177527272 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8978021640 con 0x7f89880ffbd0 2024-04-05T21:22:49.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.193+0000 7f898f95f700 1 -- 172.21.15.121:0/177527272 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8988196cf0 con 0x7f89880ffbd0 2024-04-05T21:22:49.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.194+0000 7f898f95f700 1 -- 172.21.15.121:0/177527272 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f898810ce80 con 0x7f89880ffbd0 2024-04-05T21:22:49.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.194+0000 7f897e7fc700 1 -- 172.21.15.121:0/177527272 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f897800fa60 con 0x7f89880ffbd0 2024-04-05T21:22:49.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.195+0000 7f897e7fc700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f897406c4e0 0x7f897406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:49.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.195+0000 7f897e7fc700 1 -- 172.21.15.121:0/177527272 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f897808c800 con 0x7f89880ffbd0 2024-04-05T21:22:49.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.195+0000 7f898cefa700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f897406c4e0 0x7f897406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:49.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.195+0000 7f897e7fc700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975] conn(0x7f89740721c0 0x7f89740745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:49.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.195+0000 7f897e7fc700 1 -- 172.21.15.121:0/177527272 --> [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8974074c90 con 0x7f89740721c0 2024-04-05T21:22:49.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.195+0000 7f897e7fc700 1 -- 172.21.15.121:0/177527272 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89780912a0 con 0x7f89880ffbd0 2024-04-05T21:22:49.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.195+0000 7f898cefa700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f897406c4e0 0x7f897406e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8988104e40 tx=0x7f8984008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:49.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.195+0000 7f898defc700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975] conn(0x7f89740721c0 0x7f89740745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:49.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.196+0000 7f898defc700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975] conn(0x7f89740721c0 0x7f89740745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:49.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.197+0000 7f897e7fc700 1 -- 172.21.15.121:0/177527272 <== osd.5 v2:172.21.15.188:6816/3569223975 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8974074c90 con 0x7f89740721c0 2024-04-05T21:22:49.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.212+0000 7f898f95f700 1 -- 172.21.15.121:0/177527272 --> [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f898804ea90 con 0x7f89740721c0 2024-04-05T21:22:49.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.212+0000 7f897e7fc700 1 -- 172.21.15.121:0/177527272 <== osd.5 v2:172.21.15.188:6816/3569223975 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f898804ea90 con 0x7f89740721c0 2024-04-05T21:22:49.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 -- 172.21.15.121:0/177527272 >> [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975] conn(0x7f89740721c0 msgr2=0x7f89740745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:49.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975] conn(0x7f89740721c0 0x7f89740745e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:49.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 -- 172.21.15.121:0/177527272 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f897406c4e0 msgr2=0x7f897406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:49.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f897406c4e0 0x7f897406e9a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8988104e40 tx=0x7f8984008040 comp rx=0 tx=0).stop 2024-04-05T21:22:49.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 -- 172.21.15.121:0/177527272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89880ffbd0 msgr2=0x7f898819c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:49.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89880ffbd0 0x7f898819c500 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8978006010 tx=0x7f897800be20 comp rx=0 tx=0).stop 2024-04-05T21:22:49.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 -- 172.21.15.121:0/177527272 shutdown_connections 2024-04-05T21:22:49.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.188:6816/3569223975,v1:172.21.15.188:6817/3569223975] conn(0x7f89740721c0 0x7f89740745e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:49.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f897406c4e0 0x7f897406e9a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:49.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89880ffbd0 0x7f898819c500 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:49.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 --2- 172.21.15.121:0/177527272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8988105a40 0x7f898819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:49.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 -- 172.21.15.121:0/177527272 >> 172.21.15.121:0/177527272 conn(0x7f89880755a0 msgr2=0x7f8988103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:49.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 -- 172.21.15.121:0/177527272 shutdown_connections 2024-04-05T21:22:49.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:49.213+0000 7f898f95f700 1 -- 172.21.15.121:0/177527272 wait complete. 2024-04-05T21:22:50.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:50 smithi121 ceph-mon[39723]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:50.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.149+0000 7f10f09bc700 1 -- 172.21.15.121:0/1599370926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec0ff050 msgr2=0x7f10ec0ff4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:50.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.149+0000 7f10f09bc700 1 --2- 172.21.15.121:0/1599370926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec0ff050 0x7f10ec0ff4d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f10e0009910 tx=0x7f10e0009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:50.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.149+0000 7f10f09bc700 1 -- 172.21.15.121:0/1599370926 shutdown_connections 2024-04-05T21:22:50.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.149+0000 7f10f09bc700 1 --2- 172.21.15.121:0/1599370926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec0ff050 0x7f10ec0ff4d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:50.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.149+0000 7f10f09bc700 1 --2- 172.21.15.121:0/1599370926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10ec105b20 0x7f10ec105f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:50.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.149+0000 7f10f09bc700 1 -- 172.21.15.121:0/1599370926 >> 172.21.15.121:0/1599370926 conn(0x7f10ec075e30 msgr2=0x7f10ec0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:50.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.149+0000 7f10f09bc700 1 -- 172.21.15.121:0/1599370926 shutdown_connections 2024-04-05T21:22:50.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.149+0000 7f10f09bc700 1 -- 172.21.15.121:0/1599370926 wait complete. 2024-04-05T21:22:50.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.149+0000 7f10f09bc700 1 Processor -- start 2024-04-05T21:22:50.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10f09bc700 1 -- start start 2024-04-05T21:22:50.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10f09bc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10ec0ff050 0x7f10ec1966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:50.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10f09bc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec105b20 0x7f10ec196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:50.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10f09bc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec197290 con 0x7f10ec105b20 2024-04-05T21:22:50.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10f09bc700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ec199fc0 con 0x7f10ec0ff050 2024-04-05T21:22:50.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10e9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec105b20 0x7f10ec196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:50.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10e9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec105b20 0x7f10ec196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44024/0 (socket says 172.21.15.121:44024) 2024-04-05T21:22:50.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10e9d9b700 1 -- 172.21.15.121:0/582234651 learned_addr learned my addr 172.21.15.121:0/582234651 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:22:50.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10ea59c700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10ec0ff050 0x7f10ec1966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:50.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10e9d9b700 1 -- 172.21.15.121:0/582234651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10ec0ff050 msgr2=0x7f10ec1966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:50.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10e9d9b700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10ec0ff050 0x7f10ec1966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:50.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10e9d9b700 1 -- 172.21.15.121:0/582234651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e00095f0 con 0x7f10ec105b20 2024-04-05T21:22:50.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10e9d9b700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec105b20 0x7f10ec196c00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f10e0006010 tx=0x7f10e000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:50.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10db7fe700 1 -- 172.21.15.121:0/582234651 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e001d070 con 0x7f10ec105b20 2024-04-05T21:22:50.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10f09bc700 1 -- 172.21.15.121:0/582234651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ec19a240 con 0x7f10ec105b20 2024-04-05T21:22:50.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10db7fe700 1 -- 172.21.15.121:0/582234651 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f10e0005dc0 con 0x7f10ec105b20 2024-04-05T21:22:50.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10db7fe700 1 -- 172.21.15.121:0/582234651 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e0021410 con 0x7f10ec105b20 2024-04-05T21:22:50.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.150+0000 7f10f09bc700 1 -- 172.21.15.121:0/582234651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ec19a730 con 0x7f10ec105b20 2024-04-05T21:22:50.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.151+0000 7f10f09bc700 1 -- 172.21.15.121:0/582234651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f10ec10df60 con 0x7f10ec105b20 2024-04-05T21:22:50.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.152+0000 7f10db7fe700 1 -- 172.21.15.121:0/582234651 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f10e0003a60 con 0x7f10ec105b20 2024-04-05T21:22:50.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.152+0000 7f10db7fe700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f10d406c4e0 0x7f10d406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:50.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.152+0000 7f10db7fe700 1 -- 172.21.15.121:0/582234651 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f10e008c740 con 0x7f10ec105b20 2024-04-05T21:22:50.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.152+0000 7f10ea59c700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f10d406c4e0 0x7f10d406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:50.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.153+0000 7f10db7fe700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110] conn(0x7f10d40721c0 0x7f10d40745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:50.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.153+0000 7f10db7fe700 1 -- 172.21.15.121:0/582234651 --> [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10d4074c90 con 0x7f10d40721c0 2024-04-05T21:22:50.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.153+0000 7f10db7fe700 1 -- 172.21.15.121:0/582234651 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10e008ee10 con 0x7f10ec105b20 2024-04-05T21:22:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.153+0000 7f10ea59c700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f10d406c4e0 0x7f10d406e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f10dc005950 tx=0x7f10dc00b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.153+0000 7f10ead9d700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110] conn(0x7f10d40721c0 0x7f10d40745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.153+0000 7f10ead9d700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110] conn(0x7f10d40721c0 0x7f10d40745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.154+0000 7f10db7fe700 1 -- 172.21.15.121:0/582234651 <== osd.4 v2:172.21.15.188:6808/1828363110 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f10d4074c90 con 0x7f10d40721c0 2024-04-05T21:22:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.169+0000 7f10f09bc700 1 -- 172.21.15.121:0/582234651 --> [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f10ec04ea90 con 0x7f10d40721c0 2024-04-05T21:22:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.170+0000 7f10db7fe700 1 -- 172.21.15.121:0/582234651 <== osd.4 v2:172.21.15.188:6808/1828363110 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f10ec04ea90 con 0x7f10d40721c0 2024-04-05T21:22:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.170+0000 7f10f09bc700 1 -- 172.21.15.121:0/582234651 >> [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110] conn(0x7f10d40721c0 msgr2=0x7f10d40745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.170+0000 7f10f09bc700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110] conn(0x7f10d40721c0 0x7f10d40745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.170+0000 7f10f09bc700 1 -- 172.21.15.121:0/582234651 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f10d406c4e0 msgr2=0x7f10d406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.170+0000 7f10f09bc700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f10d406c4e0 0x7f10d406e9a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f10dc005950 tx=0x7f10dc00b410 comp rx=0 tx=0).stop 2024-04-05T21:22:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.170+0000 7f10f09bc700 1 -- 172.21.15.121:0/582234651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec105b20 msgr2=0x7f10ec196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.170+0000 7f10f09bc700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec105b20 0x7f10ec196c00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f10e0006010 tx=0x7f10e000bd70 comp rx=0 tx=0).stop 2024-04-05T21:22:50.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.170+0000 7f10f09bc700 1 -- 172.21.15.121:0/582234651 shutdown_connections 2024-04-05T21:22:50.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.170+0000 7f10f09bc700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.188:6808/1828363110,v1:172.21.15.188:6809/1828363110] conn(0x7f10d40721c0 0x7f10d40745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:50.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.170+0000 7f10f09bc700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f10d406c4e0 0x7f10d406e9a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:50.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.170+0000 7f10f09bc700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f10ec0ff050 0x7f10ec1966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:50.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.170+0000 7f10f09bc700 1 --2- 172.21.15.121:0/582234651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec105b20 0x7f10ec196c00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:50.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.171+0000 7f10f09bc700 1 -- 172.21.15.121:0/582234651 >> 172.21.15.121:0/582234651 conn(0x7f10ec075e30 msgr2=0x7f10ec104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:50.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.171+0000 7f10f09bc700 1 -- 172.21.15.121:0/582234651 shutdown_connections 2024-04-05T21:22:50.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:50.171+0000 7f10f09bc700 1 -- 172.21.15.121:0/582234651 wait complete. 2024-04-05T21:22:50.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:50 smithi188 ceph-mon[49715]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:51.346 INFO:teuthology.orchestra.run.smithi121.stdout:146028888079 2024-04-05T21:22:51.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-05T21:22:51.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.471+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3425232189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b44105a40 msgr2=0x7f6b44105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:51.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.471+0000 7f6b4aeff700 1 --2- 172.21.15.121:0/3425232189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b44105a40 0x7f6b44105e20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009910 tx=0x7f6b38009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:51.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.471+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3425232189 shutdown_connections 2024-04-05T21:22:51.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.471+0000 7f6b4aeff700 1 --2- 172.21.15.121:0/3425232189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b440ffbd0 0x7f6b44100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:51.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.471+0000 7f6b4aeff700 1 --2- 172.21.15.121:0/3425232189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b44105a40 0x7f6b44105e20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:51.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.471+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3425232189 >> 172.21.15.121:0/3425232189 conn(0x7f6b440755a0 msgr2=0x7f6b440759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:51.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.471+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3425232189 shutdown_connections 2024-04-05T21:22:51.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.471+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3425232189 wait complete. 2024-04-05T21:22:51.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.471+0000 7f6b4aeff700 1 Processor -- start 2024-04-05T21:22:51.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.471+0000 7f6b4aeff700 1 -- start start 2024-04-05T21:22:51.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b4aeff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b440ffbd0 0x7f6b441975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:51.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b4aeff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b44105a40 0x7f6b44197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:51.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b4aeff700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b44198100 con 0x7f6b440ffbd0 2024-04-05T21:22:51.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b4aeff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b44198270 con 0x7f6b44105a40 2024-04-05T21:22:51.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b48c9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b440ffbd0 0x7f6b441975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:51.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b48c9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b440ffbd0 0x7f6b441975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43988/0 (socket says 172.21.15.121:43988) 2024-04-05T21:22:51.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b48c9b700 1 -- 172.21.15.121:0/3639727270 learned_addr learned my addr 172.21.15.121:0/3639727270 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:22:51.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b43fff700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b44105a40 0x7f6b44197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:51.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b48c9b700 1 -- 172.21.15.121:0/3639727270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b44105a40 msgr2=0x7f6b44197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:51.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b48c9b700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b44105a40 0x7f6b44197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:51.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b48c9b700 1 -- 172.21.15.121:0/3639727270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b380095f0 con 0x7f6b440ffbd0 2024-04-05T21:22:51.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b48c9b700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b440ffbd0 0x7f6b441975a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6b38006010 tx=0x7f6b3800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:51.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b41ffb700 1 -- 172.21.15.121:0/3639727270 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b3801d070 con 0x7f6b440ffbd0 2024-04-05T21:22:51.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b41ffb700 1 -- 172.21.15.121:0/3639727270 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f6b3800f460 con 0x7f6b440ffbd0 2024-04-05T21:22:51.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3639727270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b441a0610 con 0x7f6b440ffbd0 2024-04-05T21:22:51.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b41ffb700 1 -- 172.21.15.121:0/3639727270 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b38021640 con 0x7f6b440ffbd0 2024-04-05T21:22:51.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.472+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3639727270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b441a0a80 con 0x7f6b440ffbd0 2024-04-05T21:22:51.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.473+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3639727270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6b4410ce80 con 0x7f6b440ffbd0 2024-04-05T21:22:51.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.474+0000 7f6b41ffb700 1 -- 172.21.15.121:0/3639727270 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f6b3800fa60 con 0x7f6b440ffbd0 2024-04-05T21:22:51.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.474+0000 7f6b41ffb700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f6b3406c4e0 0x7f6b3406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:51.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.474+0000 7f6b41ffb700 1 -- 172.21.15.121:0/3639727270 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6b3808c800 con 0x7f6b440ffbd0 2024-04-05T21:22:51.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.474+0000 7f6b43fff700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f6b3406c4e0 0x7f6b3406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:51.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.474+0000 7f6b41ffb700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215] conn(0x7f6b340721c0 0x7f6b340745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:51.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.474+0000 7f6b41ffb700 1 -- 172.21.15.121:0/3639727270 --> [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b34074c90 con 0x7f6b340721c0 2024-04-05T21:22:51.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.474+0000 7f6b41ffb700 1 -- 172.21.15.121:0/3639727270 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b380912a0 con 0x7f6b440ffbd0 2024-04-05T21:22:51.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.474+0000 7f6b4949c700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215] conn(0x7f6b340721c0 0x7f6b340745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:51.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.475+0000 7f6b43fff700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f6b3406c4e0 0x7f6b3406e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6b30006fd0 tx=0x7f6b30008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:51.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.475+0000 7f6b4949c700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215] conn(0x7f6b340721c0 0x7f6b340745e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:51.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.475+0000 7f6b41ffb700 1 -- 172.21.15.121:0/3639727270 <== osd.1 v2:172.21.15.121:6810/3573482215 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6b34074c90 con 0x7f6b340721c0 2024-04-05T21:22:51.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3639727270 --> [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6b4404ea90 con 0x7f6b340721c0 2024-04-05T21:22:51.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b41ffb700 1 -- 172.21.15.121:0/3639727270 <== osd.1 v2:172.21.15.121:6810/3573482215 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6b4404ea90 con 0x7f6b340721c0 2024-04-05T21:22:51.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215] conn(0x7f6b340721c0 msgr2=0x7f6b340745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:51.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215] conn(0x7f6b340721c0 0x7f6b340745e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:51.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f6b3406c4e0 msgr2=0x7f6b3406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:51.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f6b3406c4e0 0x7f6b3406e9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6b30006fd0 tx=0x7f6b30008040 comp rx=0 tx=0).stop 2024-04-05T21:22:51.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b440ffbd0 msgr2=0x7f6b441975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:51.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b440ffbd0 0x7f6b441975a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6b38006010 tx=0x7f6b3800be20 comp rx=0 tx=0).stop 2024-04-05T21:22:51.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3639727270 shutdown_connections 2024-04-05T21:22:51.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:6810/3573482215,v1:172.21.15.121:6811/3573482215] conn(0x7f6b340721c0 0x7f6b340745e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:51.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f6b3406c4e0 0x7f6b3406e9a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:51.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b440ffbd0 0x7f6b441975a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:51.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 --2- 172.21.15.121:0/3639727270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b44105a40 0x7f6b44197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:51.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3639727270 >> 172.21.15.121:0/3639727270 conn(0x7f6b440755a0 msgr2=0x7f6b440fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:51.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3639727270 shutdown_connections 2024-04-05T21:22:51.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:47.491+0000 7f6b4aeff700 1 -- 172.21.15.121:0/3639727270 wait complete. 2024-04-05T21:22:52.187 INFO:teuthology.orchestra.run.smithi121.stdout:167503724554 2024-04-05T21:22:52.187 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-05T21:22:52.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.079+0000 7fe9d5c16700 1 -- 172.21.15.121:0/371691917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9d00ffbd0 msgr2=0x7fe9d0100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:52.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.079+0000 7fe9d5c16700 1 --2- 172.21.15.121:0/371691917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9d00ffbd0 0x7fe9d0100050 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009910 tx=0x7fe9c0009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:52.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.079+0000 7fe9d5c16700 1 -- 172.21.15.121:0/371691917 shutdown_connections 2024-04-05T21:22:52.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.079+0000 7fe9d5c16700 1 --2- 172.21.15.121:0/371691917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9d00ffbd0 0x7fe9d0100050 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:52.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.079+0000 7fe9d5c16700 1 --2- 172.21.15.121:0/371691917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9d0105a40 0x7fe9d0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:52.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.079+0000 7fe9d5c16700 1 -- 172.21.15.121:0/371691917 >> 172.21.15.121:0/371691917 conn(0x7fe9d00755a0 msgr2=0x7fe9d00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:52.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.079+0000 7fe9d5c16700 1 -- 172.21.15.121:0/371691917 shutdown_connections 2024-04-05T21:22:52.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.079+0000 7fe9d5c16700 1 -- 172.21.15.121:0/371691917 wait complete. 2024-04-05T21:22:52.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.080+0000 7fe9d5c16700 1 Processor -- start 2024-04-05T21:22:52.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.080+0000 7fe9d5c16700 1 -- start start 2024-04-05T21:22:52.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.080+0000 7fe9d5c16700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9d00ffbd0 0x7fe9d0101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:52.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.080+0000 7fe9d5c16700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9d0105a40 0x7fe9d0101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:52.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.080+0000 7fe9d5c16700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d0102090 con 0x7fe9d00ffbd0 2024-04-05T21:22:52.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.080+0000 7fe9d5c16700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d01021d0 con 0x7fe9d0105a40 2024-04-05T21:22:52.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.080+0000 7fe9cf7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9d00ffbd0 0x7fe9d0101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.080+0000 7fe9cf7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9d00ffbd0 0x7fe9d0101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43976/0 (socket says 172.21.15.121:43976) 2024-04-05T21:22:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.080+0000 7fe9cf7fe700 1 -- 172.21.15.121:0/313578416 learned_addr learned my addr 172.21.15.121:0/313578416 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:22:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.080+0000 7fe9cf7fe700 1 -- 172.21.15.121:0/313578416 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9d0105a40 msgr2=0x7fe9d0101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.080+0000 7fe9cf7fe700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9d0105a40 0x7fe9d0101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.080+0000 7fe9cf7fe700 1 -- 172.21.15.121:0/313578416 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c00095f0 con 0x7fe9d00ffbd0 2024-04-05T21:22:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.081+0000 7fe9cf7fe700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9d00ffbd0 0x7fe9d0101610 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400b4b0 tx=0x7fe9c400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.081+0000 7fe9ccff9700 1 -- 172.21.15.121:0/313578416 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c4011800 con 0x7fe9d00ffbd0 2024-04-05T21:22:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.081+0000 7fe9ccff9700 1 -- 172.21.15.121:0/313578416 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fe9c4011df0 con 0x7fe9d00ffbd0 2024-04-05T21:22:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.081+0000 7fe9d5c16700 1 -- 172.21.15.121:0/313578416 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d01a2900 con 0x7fe9d00ffbd0 2024-04-05T21:22:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.081+0000 7fe9ccff9700 1 -- 172.21.15.121:0/313578416 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9c4010e40 con 0x7fe9d00ffbd0 2024-04-05T21:22:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.081+0000 7fe9d5c16700 1 -- 172.21.15.121:0/313578416 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d01a2d10 con 0x7fe9d00ffbd0 2024-04-05T21:22:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.081+0000 7fe9d5c16700 1 -- 172.21.15.121:0/313578416 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe9d010ce80 con 0x7fe9d00ffbd0 2024-04-05T21:22:52.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.082+0000 7fe9ccff9700 1 -- 172.21.15.121:0/313578416 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fe9c4010380 con 0x7fe9d00ffbd0 2024-04-05T21:22:52.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.082+0000 7fe9ccff9700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fe9b806c490 0x7fe9b806e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:52.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.083+0000 7fe9ceffd700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fe9b806c490 0x7fe9b806e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:52.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.083+0000 7fe9ccff9700 1 -- 172.21.15.121:0/313578416 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fe9c408b740 con 0x7fe9d00ffbd0 2024-04-05T21:22:52.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.083+0000 7fe9ccff9700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890] conn(0x7fe9b8072170 0x7fe9b8074590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:52.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.083+0000 7fe9ccff9700 1 -- 172.21.15.121:0/313578416 --> [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9b8074c40 con 0x7fe9b8072170 2024-04-05T21:22:52.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.083+0000 7fe9ccff9700 1 -- 172.21.15.121:0/313578416 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9c408bba0 con 0x7fe9d00ffbd0 2024-04-05T21:22:52.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.083+0000 7fe9cffff700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890] conn(0x7fe9b8072170 0x7fe9b8074590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:52.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.083+0000 7fe9ceffd700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fe9b806c490 0x7fe9b806e950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000b5c0 tx=0x7fe9c0005960 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:52.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.083+0000 7fe9cffff700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890] conn(0x7fe9b8072170 0x7fe9b8074590 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:52.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.084+0000 7fe9ccff9700 1 -- 172.21.15.121:0/313578416 <== osd.0 v2:172.21.15.121:6802/2285715890 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe9b8074c40 con 0x7fe9b8072170 2024-04-05T21:22:52.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.099+0000 7fe9d5c16700 1 -- 172.21.15.121:0/313578416 --> [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe9d004ea90 con 0x7fe9b8072170 2024-04-05T21:22:52.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.099+0000 7fe9ccff9700 1 -- 172.21.15.121:0/313578416 <== osd.0 v2:172.21.15.121:6802/2285715890 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe9d004ea90 con 0x7fe9b8072170 2024-04-05T21:22:52.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.099+0000 7fe9d5c16700 1 -- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890] conn(0x7fe9b8072170 msgr2=0x7fe9b8074590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:52.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.099+0000 7fe9d5c16700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890] conn(0x7fe9b8072170 0x7fe9b8074590 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:52.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.099+0000 7fe9d5c16700 1 -- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fe9b806c490 msgr2=0x7fe9b806e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:52.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.099+0000 7fe9d5c16700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fe9b806c490 0x7fe9b806e950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe9c000b5c0 tx=0x7fe9c0005960 comp rx=0 tx=0).stop 2024-04-05T21:22:52.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.099+0000 7fe9d5c16700 1 -- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9d00ffbd0 msgr2=0x7fe9d0101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:52.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.099+0000 7fe9d5c16700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9d00ffbd0 0x7fe9d0101610 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400b4b0 tx=0x7fe9c400b870 comp rx=0 tx=0).stop 2024-04-05T21:22:52.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.100+0000 7fe9d5c16700 1 -- 172.21.15.121:0/313578416 shutdown_connections 2024-04-05T21:22:52.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.100+0000 7fe9d5c16700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:6802/2285715890,v1:172.21.15.121:6803/2285715890] conn(0x7fe9b8072170 0x7fe9b8074590 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:52.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.100+0000 7fe9d5c16700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fe9b806c490 0x7fe9b806e950 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:52.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.100+0000 7fe9d5c16700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9d00ffbd0 0x7fe9d0101610 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:52.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.100+0000 7fe9d5c16700 1 --2- 172.21.15.121:0/313578416 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9d0105a40 0x7fe9d0101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:52.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.100+0000 7fe9d5c16700 1 -- 172.21.15.121:0/313578416 >> 172.21.15.121:0/313578416 conn(0x7fe9d00755a0 msgr2=0x7fe9d0103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:52.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.100+0000 7fe9d5c16700 1 -- 172.21.15.121:0/313578416 shutdown_connections 2024-04-05T21:22:52.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:46.100+0000 7fe9d5c16700 1 -- 172.21.15.121:0/313578416 wait complete. 2024-04-05T21:22:52.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:52 smithi121 ceph-mon[39723]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:52.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:52 smithi188 ceph-mon[49715]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:53.285 INFO:teuthology.orchestra.run.smithi121.stdout:60129542177 2024-04-05T21:22:53.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-05T21:22:53.826 INFO:teuthology.orchestra.run.smithi121.stdout:38654705703 2024-04-05T21:22:53.826 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-05T21:22:54.156 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:54.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:54 smithi121 ceph-mon[39723]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:54.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.705+0000 7fedf63d4700 1 -- 172.21.15.121:0/2360237118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedf0105a40 msgr2=0x7fedf0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.705+0000 7fedf63d4700 1 --2- 172.21.15.121:0/2360237118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedf0105a40 0x7fedf0105e20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fede0009910 tx=0x7fede0009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.706+0000 7fedf63d4700 1 -- 172.21.15.121:0/2360237118 shutdown_connections 2024-04-05T21:22:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.706+0000 7fedf63d4700 1 --2- 172.21.15.121:0/2360237118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedf00ffbd0 0x7fedf0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.706+0000 7fedf63d4700 1 --2- 172.21.15.121:0/2360237118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedf0105a40 0x7fedf0105e20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.706+0000 7fedf63d4700 1 -- 172.21.15.121:0/2360237118 >> 172.21.15.121:0/2360237118 conn(0x7fedf00755a0 msgr2=0x7fedf00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.706+0000 7fedf63d4700 1 -- 172.21.15.121:0/2360237118 shutdown_connections 2024-04-05T21:22:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.706+0000 7fedf63d4700 1 -- 172.21.15.121:0/2360237118 wait complete. 2024-04-05T21:22:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.706+0000 7fedf63d4700 1 Processor -- start 2024-04-05T21:22:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.706+0000 7fedf63d4700 1 -- start start 2024-04-05T21:22:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.706+0000 7fedf63d4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedf00ffbd0 0x7fedf0195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:54.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedf63d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedf0105a40 0x7fedf0195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:54.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedf63d4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf0196170 con 0x7fedf0105a40 2024-04-05T21:22:54.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedf63d4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf0199f00 con 0x7fedf00ffbd0 2024-04-05T21:22:54.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedef7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedf0105a40 0x7fedf0195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:54.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedef7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedf0105a40 0x7fedf0195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38364/0 (socket says 172.21.15.121:38364) 2024-04-05T21:22:54.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedef7fe700 1 -- 172.21.15.121:0/3632271268 learned_addr learned my addr 172.21.15.121:0/3632271268 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:22:54.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedeffff700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedf00ffbd0 0x7fedf0195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:54.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedef7fe700 1 -- 172.21.15.121:0/3632271268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedf00ffbd0 msgr2=0x7fedf0195550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:54.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedef7fe700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedf00ffbd0 0x7fedf0195550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:54.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedef7fe700 1 -- 172.21.15.121:0/3632271268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede00095f0 con 0x7fedf0105a40 2024-04-05T21:22:54.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedeffff700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedf00ffbd0 0x7fedf0195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:22:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedef7fe700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedf0105a40 0x7fedf0195a90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fede400c8a0 tx=0x7fede400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7feded7fa700 1 -- 172.21.15.121:0/3632271268 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fede400cea0 con 0x7fedf0105a40 2024-04-05T21:22:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7feded7fa700 1 -- 172.21.15.121:0/3632271268 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fede4004830 con 0x7fedf0105a40 2024-04-05T21:22:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7feded7fa700 1 -- 172.21.15.121:0/3632271268 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fede4005540 con 0x7fedf0105a40 2024-04-05T21:22:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedf63d4700 1 -- 172.21.15.121:0/3632271268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf019a1e0 con 0x7fedf0105a40 2024-04-05T21:22:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.707+0000 7fedf63d4700 1 -- 172.21.15.121:0/3632271268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf019a730 con 0x7fedf0105a40 2024-04-05T21:22:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.708+0000 7fedf63d4700 1 -- 172.21.15.121:0/3632271268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fedf010ce00 con 0x7fedf0105a40 2024-04-05T21:22:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.709+0000 7feded7fa700 1 -- 172.21.15.121:0/3632271268 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fede40049a0 con 0x7fedf0105a40 2024-04-05T21:22:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.709+0000 7feded7fa700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fedd806c4e0 0x7fedd806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.709+0000 7feded7fa700 1 -- 172.21.15.121:0/3632271268 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fede408b0d0 con 0x7fedf0105a40 2024-04-05T21:22:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.709+0000 7fedeffff700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fedd806c4e0 0x7fedd806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.709+0000 7feded7fa700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079] conn(0x7fedd80721c0 0x7fedd80745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:54.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.709+0000 7feded7fa700 1 -- 172.21.15.121:0/3632271268 --> [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fedd8074c90 con 0x7fedd80721c0 2024-04-05T21:22:54.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.709+0000 7feded7fa700 1 -- 172.21.15.121:0/3632271268 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fede408d7a0 con 0x7fedf0105a40 2024-04-05T21:22:54.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.709+0000 7fedeffff700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fedd806c4e0 0x7fedd806e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fede000b5c0 tx=0x7fede0005960 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:54.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.709+0000 7fedf4971700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079] conn(0x7fedd80721c0 0x7fedd80745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:54.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.710+0000 7fedf4971700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079] conn(0x7fedd80721c0 0x7fedd80745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:54.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.711+0000 7feded7fa700 1 -- 172.21.15.121:0/3632271268 <== osd.3 v2:172.21.15.188:6800/2760393079 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fedd8074c90 con 0x7fedd80721c0 2024-04-05T21:22:54.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.726+0000 7fedf63d4700 1 -- 172.21.15.121:0/3632271268 --> [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fedf004ea90 con 0x7fedd80721c0 2024-04-05T21:22:54.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7feded7fa700 1 -- 172.21.15.121:0/3632271268 <== osd.3 v2:172.21.15.188:6800/2760393079 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fedf004ea90 con 0x7fedd80721c0 2024-04-05T21:22:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 -- 172.21.15.121:0/3632271268 >> [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079] conn(0x7fedd80721c0 msgr2=0x7fedd80745e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079] conn(0x7fedd80721c0 0x7fedd80745e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 -- 172.21.15.121:0/3632271268 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fedd806c4e0 msgr2=0x7fedd806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fedd806c4e0 0x7fedd806e9a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fede000b5c0 tx=0x7fede0005960 comp rx=0 tx=0).stop 2024-04-05T21:22:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 -- 172.21.15.121:0/3632271268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedf0105a40 msgr2=0x7fedf0195a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedf0105a40 0x7fedf0195a90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fede400c8a0 tx=0x7fede400cc60 comp rx=0 tx=0).stop 2024-04-05T21:22:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 -- 172.21.15.121:0/3632271268 shutdown_connections 2024-04-05T21:22:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fedd806c4e0 0x7fedd806e9a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fedf00ffbd0 0x7fedf0195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.188:6800/2760393079,v1:172.21.15.188:6801/2760393079] conn(0x7fedd80721c0 0x7fedd80745e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 --2- 172.21.15.121:0/3632271268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedf0105a40 0x7fedf0195a90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 -- 172.21.15.121:0/3632271268 >> 172.21.15.121:0/3632271268 conn(0x7fedf00755a0 msgr2=0x7fedf00fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 -- 172.21.15.121:0/3632271268 shutdown_connections 2024-04-05T21:22:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:54.727+0000 7fedf63d4700 1 -- 172.21.15.121:0/3632271268 wait complete. 2024-04-05T21:22:54.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:54 smithi188 ceph-mon[49715]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:55.374 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:55 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:22:55.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:55 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:22:56.357 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:56.358 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:22:56.384 INFO:teuthology.orchestra.run.smithi121.stdout:111669149718 2024-04-05T21:22:56.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-05T21:22:56.677 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:56 smithi121 ceph-mon[39723]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:56.699 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:56 smithi188 ceph-mon[49715]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:57.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.989+0000 7f01f8bb4700 1 -- 172.21.15.121:0/2766551131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4101750 msgr2=0x7f01f41057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:57.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.989+0000 7f01f8bb4700 1 --2- 172.21.15.121:0/2766551131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4101750 0x7f01f41057a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009910 tx=0x7f01e8009c20 comp rx=0 tx=0).stop 2024-04-05T21:22:57.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.989+0000 7f01f8bb4700 1 -- 172.21.15.121:0/2766551131 shutdown_connections 2024-04-05T21:22:57.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.989+0000 7f01f8bb4700 1 --2- 172.21.15.121:0/2766551131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4101750 0x7f01f41057a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:57.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.989+0000 7f01f8bb4700 1 --2- 172.21.15.121:0/2766551131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01f4100da0 0x7f01f4101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:57.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.989+0000 7f01f8bb4700 1 -- 172.21.15.121:0/2766551131 >> 172.21.15.121:0/2766551131 conn(0x7f01f40fc630 msgr2=0x7f01f40fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:57.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.989+0000 7f01f8bb4700 1 -- 172.21.15.121:0/2766551131 shutdown_connections 2024-04-05T21:22:57.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.989+0000 7f01f8bb4700 1 -- 172.21.15.121:0/2766551131 wait complete. 2024-04-05T21:22:57.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.990+0000 7f01f8bb4700 1 Processor -- start 2024-04-05T21:22:57.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.990+0000 7f01f8bb4700 1 -- start start 2024-04-05T21:22:57.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.990+0000 7f01f8bb4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4100da0 0x7f01f4196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:57.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.990+0000 7f01f8bb4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01f4101750 0x7f01f4196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:57.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.990+0000 7f01f8bb4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f4197270 con 0x7f01f4100da0 2024-04-05T21:22:57.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.990+0000 7f01f8bb4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f419b000 con 0x7f01f4101750 2024-04-05T21:22:57.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.990+0000 7f01f259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4100da0 0x7f01f4196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.990+0000 7f01f259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4100da0 0x7f01f4196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38388/0 (socket says 172.21.15.121:38388) 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.990+0000 7f01f259c700 1 -- 172.21.15.121:0/3095538867 learned_addr learned my addr 172.21.15.121:0/3095538867 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.990+0000 7f01f1d9b700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01f4101750 0x7f01f4196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.990+0000 7f01f259c700 1 -- 172.21.15.121:0/3095538867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01f4101750 msgr2=0x7f01f4196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.990+0000 7f01f259c700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01f4101750 0x7f01f4196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.990+0000 7f01f259c700 1 -- 172.21.15.121:0/3095538867 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e80095f0 con 0x7f01f4100da0 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.991+0000 7f01f259c700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4100da0 0x7f01f4196650 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f01e400d900 tx=0x7f01e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.991+0000 7f01e37fe700 1 -- 172.21.15.121:0/3095538867 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01e40041f0 con 0x7f01f4100da0 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.991+0000 7f01f8bb4700 1 -- 172.21.15.121:0/3095538867 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f419b2e0 con 0x7f01f4100da0 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.991+0000 7f01e37fe700 1 -- 172.21.15.121:0/3095538867 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f01e4004cc0 con 0x7f01f4100da0 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.991+0000 7f01e37fe700 1 -- 172.21.15.121:0/3095538867 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01e40056d0 con 0x7f01f4100da0 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.991+0000 7f01f8bb4700 1 -- 172.21.15.121:0/3095538867 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f419b830 con 0x7f01f4100da0 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.991+0000 7f01f8bb4700 1 -- 172.21.15.121:0/3095538867 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f01f410df00 con 0x7f01f4100da0 2024-04-05T21:22:57.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.992+0000 7f01e37fe700 1 -- 172.21.15.121:0/3095538867 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f01e4004750 con 0x7f01f4100da0 2024-04-05T21:22:57.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.992+0000 7f01e37fe700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f01dc06c480 0x7f01dc06e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:57.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.992+0000 7f01e37fe700 1 -- 172.21.15.121:0/3095538867 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f01e408b710 con 0x7f01f4100da0 2024-04-05T21:22:57.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.992+0000 7f01f1d9b700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f01dc06c480 0x7f01dc06e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:57.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.993+0000 7f01e37fe700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010] conn(0x7f01dc072160 0x7f01dc074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:22:57.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.993+0000 7f01e37fe700 1 -- 172.21.15.121:0/3095538867 --> [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f01dc074c30 con 0x7f01dc072160 2024-04-05T21:22:57.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.993+0000 7f01e37fe700 1 -- 172.21.15.121:0/3095538867 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01e408dd80 con 0x7f01f4100da0 2024-04-05T21:22:57.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.993+0000 7f01f2d9d700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010] conn(0x7f01dc072160 0x7f01dc074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.993+0000 7f01f1d9b700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f01dc06c480 0x7f01dc06e940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f01f4197c70 tx=0x7f01e801a040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.993+0000 7f01f2d9d700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010] conn(0x7f01dc072160 0x7f01dc074580 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:55.994+0000 7f01e37fe700 1 -- 172.21.15.121:0/3095538867 <== osd.2 v2:172.21.15.121:6818/179235010 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f01dc074c30 con 0x7f01dc072160 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 -- 172.21.15.121:0/3095538867 --> [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f01f404ea90 con 0x7f01dc072160 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01e37fe700 1 -- 172.21.15.121:0/3095538867 <== osd.2 v2:172.21.15.121:6818/179235010 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f01f404ea90 con 0x7f01dc072160 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 -- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010] conn(0x7f01dc072160 msgr2=0x7f01dc074580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010] conn(0x7f01dc072160 0x7f01dc074580 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 -- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f01dc06c480 msgr2=0x7f01dc06e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f01dc06c480 0x7f01dc06e940 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f01f4197c70 tx=0x7f01e801a040 comp rx=0 tx=0).stop 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 -- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4100da0 msgr2=0x7f01f4196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4100da0 0x7f01f4196650 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f01e400d900 tx=0x7f01e400dcc0 comp rx=0 tx=0).stop 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 -- 172.21.15.121:0/3095538867 shutdown_connections 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:6818/179235010,v1:172.21.15.121:6819/179235010] conn(0x7f01dc072160 0x7f01dc074580 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:57.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f01dc06c480 0x7f01dc06e940 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:57.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f4100da0 0x7f01f4196650 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:57.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 --2- 172.21.15.121:0/3095538867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f01f4101750 0x7f01f4196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:22:57.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 -- 172.21.15.121:0/3095538867 >> 172.21.15.121:0/3095538867 conn(0x7f01f40fc630 msgr2=0x7f01f40fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:22:57.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 -- 172.21.15.121:0/3095538867 shutdown_connections 2024-04-05T21:22:57.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:22:56.009+0000 7f01f8bb4700 1 -- 172.21.15.121:0/3095538867 wait complete. 2024-04-05T21:22:58.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:22:58 smithi121 ceph-mon[39723]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:22:58.552 INFO:teuthology.orchestra.run.smithi121.stdout:81604378652 2024-04-05T21:22:58.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-05T21:22:58.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:22:58 smithi188 ceph-mon[49715]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:00 smithi121 ceph-mon[39723]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:00.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:00 smithi188 ceph-mon[49715]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:02.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.156+0000 7fc7c4459700 1 -- 172.21.15.121:0/1548413352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7bc1017c0 msgr2=0x7fc7bc105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:02.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.156+0000 7fc7c4459700 1 --2- 172.21.15.121:0/1548413352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7bc1017c0 0x7fc7bc105810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8009910 tx=0x7fc7b8009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:02.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.156+0000 7fc7c4459700 1 -- 172.21.15.121:0/1548413352 shutdown_connections 2024-04-05T21:23:02.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.156+0000 7fc7c4459700 1 --2- 172.21.15.121:0/1548413352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7bc1017c0 0x7fc7bc105810 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:02.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.156+0000 7fc7c4459700 1 --2- 172.21.15.121:0/1548413352 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7bc100e10 0x7fc7bc1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:02.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.156+0000 7fc7c4459700 1 -- 172.21.15.121:0/1548413352 >> 172.21.15.121:0/1548413352 conn(0x7fc7bc0fc6c0 msgr2=0x7fc7bc0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:02.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.156+0000 7fc7c4459700 1 -- 172.21.15.121:0/1548413352 shutdown_connections 2024-04-05T21:23:02.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.156+0000 7fc7c4459700 1 -- 172.21.15.121:0/1548413352 wait complete. 2024-04-05T21:23:02.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.156+0000 7fc7c4459700 1 Processor -- start 2024-04-05T21:23:02.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.156+0000 7fc7c4459700 1 -- start start 2024-04-05T21:23:02.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c4459700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7bc100e10 0x7fc7bc196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:02.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c4459700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7bc1017c0 0x7fc7bc196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:02.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c4459700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc197260 con 0x7fc7bc100e10 2024-04-05T21:23:02.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c4459700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc199f90 con 0x7fc7bc1017c0 2024-04-05T21:23:02.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c21f5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7bc100e10 0x7fc7bc196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:02.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c21f5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7bc100e10 0x7fc7bc196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38412/0 (socket says 172.21.15.121:38412) 2024-04-05T21:23:02.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c21f5700 1 -- 172.21.15.121:0/467809465 learned_addr learned my addr 172.21.15.121:0/467809465 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:02.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c19f4700 1 --2- 172.21.15.121:0/467809465 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7bc1017c0 0x7fc7bc196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:02.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c21f5700 1 -- 172.21.15.121:0/467809465 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7bc1017c0 msgr2=0x7fc7bc196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:02.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c21f5700 1 --2- 172.21.15.121:0/467809465 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7bc1017c0 0x7fc7bc196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:02.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c21f5700 1 -- 172.21.15.121:0/467809465 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b80095f0 con 0x7fc7bc100e10 2024-04-05T21:23:02.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c19f4700 1 --2- 172.21.15.121:0/467809465 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7bc1017c0 0x7fc7bc196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:23:02.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c21f5700 1 --2- 172.21.15.121:0/467809465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7bc100e10 0x7fc7bc196690 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00c8a0 tx=0x7fc7ac00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:02.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7b37fe700 1 -- 172.21.15.121:0/467809465 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7ac00cea0 con 0x7fc7bc100e10 2024-04-05T21:23:02.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7b37fe700 1 -- 172.21.15.121:0/467809465 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fc7ac004830 con 0x7fc7bc100e10 2024-04-05T21:23:02.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c4459700 1 -- 172.21.15.121:0/467809465 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7bc19a270 con 0x7fc7bc100e10 2024-04-05T21:23:02.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7b37fe700 1 -- 172.21.15.121:0/467809465 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7ac005540 con 0x7fc7bc100e10 2024-04-05T21:23:02.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.157+0000 7fc7c4459700 1 -- 172.21.15.121:0/467809465 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7bc19a790 con 0x7fc7bc100e10 2024-04-05T21:23:02.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.158+0000 7fc7c4459700 1 -- 172.21.15.121:0/467809465 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7bc109210 con 0x7fc7bc100e10 2024-04-05T21:23:02.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.159+0000 7fc7b37fe700 1 -- 172.21.15.121:0/467809465 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc7ac007ad0 con 0x7fc7bc100e10 2024-04-05T21:23:02.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.159+0000 7fc7b37fe700 1 --2- 172.21.15.121:0/467809465 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fc7a806c4e0 0x7fc7a806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:02.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.159+0000 7fc7b37fe700 1 -- 172.21.15.121:0/467809465 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc7ac08b290 con 0x7fc7bc100e10 2024-04-05T21:23:02.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.159+0000 7fc7c19f4700 1 --2- 172.21.15.121:0/467809465 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fc7a806c4e0 0x7fc7a806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:02.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.160+0000 7fc7c19f4700 1 --2- 172.21.15.121:0/467809465 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fc7a806c4e0 0x7fc7a806e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc197db0 tx=0x7fc7b801a040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:02.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.164+0000 7fc7b37fe700 1 -- 172.21.15.121:0/467809465 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc7ac055d10 con 0x7fc7bc100e10 2024-04-05T21:23:02.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.359+0000 7fc7c4459700 1 -- 172.21.15.121:0/467809465 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc7bc04ea90 con 0x7fc7bc100e10 2024-04-05T21:23:02.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.359+0000 7fc7b37fe700 1 -- 172.21.15.121:0/467809465 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc7ac059330 con 0x7fc7bc100e10 2024-04-05T21:23:02.360 INFO:teuthology.orchestra.run.smithi121.stdout:146028888081 2024-04-05T21:23:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.361+0000 7fc7c4459700 1 -- 172.21.15.121:0/467809465 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fc7a806c4e0 msgr2=0x7fc7a806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.361+0000 7fc7c4459700 1 --2- 172.21.15.121:0/467809465 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fc7a806c4e0 0x7fc7a806e9a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc197db0 tx=0x7fc7b801a040 comp rx=0 tx=0).stop 2024-04-05T21:23:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.361+0000 7fc7c4459700 1 -- 172.21.15.121:0/467809465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7bc100e10 msgr2=0x7fc7bc196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.361+0000 7fc7c4459700 1 --2- 172.21.15.121:0/467809465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7bc100e10 0x7fc7bc196690 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00c8a0 tx=0x7fc7ac00cc60 comp rx=0 tx=0).stop 2024-04-05T21:23:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.361+0000 7fc7c4459700 1 -- 172.21.15.121:0/467809465 shutdown_connections 2024-04-05T21:23:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.361+0000 7fc7c4459700 1 --2- 172.21.15.121:0/467809465 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fc7a806c4e0 0x7fc7a806e9a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.361+0000 7fc7c4459700 1 --2- 172.21.15.121:0/467809465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7bc100e10 0x7fc7bc196690 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.361+0000 7fc7c4459700 1 --2- 172.21.15.121:0/467809465 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc7bc1017c0 0x7fc7bc196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.361+0000 7fc7c4459700 1 -- 172.21.15.121:0/467809465 >> 172.21.15.121:0/467809465 conn(0x7fc7bc0fc6c0 msgr2=0x7fc7bc0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.361+0000 7fc7c4459700 1 -- 172.21.15.121:0/467809465 shutdown_connections 2024-04-05T21:23:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.361+0000 7fc7c4459700 1 -- 172.21.15.121:0/467809465 wait complete. 2024-04-05T21:23:02.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:02 smithi121 ceph-mon[39723]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:02.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:02 smithi188 ceph-mon[49715]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:02.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.588+0000 7f3f1a567700 1 -- 172.21.15.121:0/442854980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f14101790 msgr2=0x7f3f141057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:02.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.588+0000 7f3f1a567700 1 --2- 172.21.15.121:0/442854980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f14101790 0x7f3f141057e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009910 tx=0x7f3efc009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:02.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.588+0000 7f3f1a567700 1 -- 172.21.15.121:0/442854980 shutdown_connections 2024-04-05T21:23:02.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.588+0000 7f3f1a567700 1 --2- 172.21.15.121:0/442854980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f14101790 0x7f3f141057e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:02.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.588+0000 7f3f1a567700 1 --2- 172.21.15.121:0/442854980 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f14100de0 0x7f3f141011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:02.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.588+0000 7f3f1a567700 1 -- 172.21.15.121:0/442854980 >> 172.21.15.121:0/442854980 conn(0x7f3f140fc670 msgr2=0x7f3f140fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:02.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.588+0000 7f3f1a567700 1 -- 172.21.15.121:0/442854980 shutdown_connections 2024-04-05T21:23:02.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.588+0000 7f3f1a567700 1 -- 172.21.15.121:0/442854980 wait complete. 2024-04-05T21:23:02.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.589+0000 7f3f1a567700 1 Processor -- start 2024-04-05T21:23:02.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.589+0000 7f3f1a567700 1 -- start start 2024-04-05T21:23:02.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.589+0000 7f3f1a567700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f14100de0 0x7f3f141966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:02.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.589+0000 7f3f1a567700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f14101790 0x7f3f14196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:02.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.589+0000 7f3f1a567700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f141972a0 con 0x7f3f14100de0 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.589+0000 7f3f1a567700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f14199fd0 con 0x7f3f14101790 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.589+0000 7f3f13fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f14100de0 0x7f3f141966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.589+0000 7f3f13fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f14100de0 0x7f3f141966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38440/0 (socket says 172.21.15.121:38440) 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.589+0000 7f3f13fff700 1 -- 172.21.15.121:0/3887117479 learned_addr learned my addr 172.21.15.121:0/3887117479 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.590+0000 7f3f137fe700 1 --2- 172.21.15.121:0/3887117479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f14101790 0x7f3f14196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.590+0000 7f3f13fff700 1 -- 172.21.15.121:0/3887117479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f14101790 msgr2=0x7f3f14196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.590+0000 7f3f13fff700 1 --2- 172.21.15.121:0/3887117479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f14101790 0x7f3f14196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.590+0000 7f3f13fff700 1 -- 172.21.15.121:0/3887117479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3efc0095f0 con 0x7f3f14100de0 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.590+0000 7f3f13fff700 1 --2- 172.21.15.121:0/3887117479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f14100de0 0x7f3f141966d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400c8a0 tx=0x7f3f0400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.590+0000 7f3f117fa700 1 -- 172.21.15.121:0/3887117479 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f0400cea0 con 0x7f3f14100de0 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.590+0000 7f3f1a567700 1 -- 172.21.15.121:0/3887117479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1419a2b0 con 0x7f3f14100de0 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.590+0000 7f3f117fa700 1 -- 172.21.15.121:0/3887117479 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f3f04004830 con 0x7f3f14100de0 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.590+0000 7f3f117fa700 1 -- 172.21.15.121:0/3887117479 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f04005540 con 0x7f3f14100de0 2024-04-05T21:23:02.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.590+0000 7f3f1a567700 1 -- 172.21.15.121:0/3887117479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1419a800 con 0x7f3f14100de0 2024-04-05T21:23:02.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.591+0000 7f3f1a567700 1 -- 172.21.15.121:0/3887117479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f141091e0 con 0x7f3f14100de0 2024-04-05T21:23:02.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.591+0000 7f3f117fa700 1 -- 172.21.15.121:0/3887117479 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3f040049a0 con 0x7f3f14100de0 2024-04-05T21:23:02.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.592+0000 7f3f117fa700 1 --2- 172.21.15.121:0/3887117479 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3f0006c4e0 0x7f3f0006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:02.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.592+0000 7f3f117fa700 1 -- 172.21.15.121:0/3887117479 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3f0408b1c0 con 0x7f3f14100de0 2024-04-05T21:23:02.834 INFO:teuthology.orchestra.run.smithi121.stdout:38654705705 2024-04-05T21:23:02.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.592+0000 7f3f137fe700 1 --2- 172.21.15.121:0/3887117479 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3f0006c4e0 0x7f3f0006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:02.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.592+0000 7f3f137fe700 1 --2- 172.21.15.121:0/3887117479 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3f0006c4e0 0x7f3f0006e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00b5c0 tx=0x7f3efc005fd0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:02.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.596+0000 7f3f117fa700 1 -- 172.21.15.121:0/3887117479 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3f04055ce0 con 0x7f3f14100de0 2024-04-05T21:23:02.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.788+0000 7f3f1a567700 1 -- 172.21.15.121:0/3887117479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3f1404ea90 con 0x7f3f14100de0 2024-04-05T21:23:02.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.788+0000 7f3f117fa700 1 -- 172.21.15.121:0/3887117479 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f04059300 con 0x7f3f14100de0 2024-04-05T21:23:02.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.790+0000 7f3f1a567700 1 -- 172.21.15.121:0/3887117479 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3f0006c4e0 msgr2=0x7f3f0006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:02.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.790+0000 7f3f1a567700 1 --2- 172.21.15.121:0/3887117479 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3f0006c4e0 0x7f3f0006e9a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00b5c0 tx=0x7f3efc005fd0 comp rx=0 tx=0).stop 2024-04-05T21:23:02.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.790+0000 7f3f1a567700 1 -- 172.21.15.121:0/3887117479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f14100de0 msgr2=0x7f3f141966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:02.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.790+0000 7f3f1a567700 1 --2- 172.21.15.121:0/3887117479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f14100de0 0x7f3f141966d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400c8a0 tx=0x7f3f0400cc60 comp rx=0 tx=0).stop 2024-04-05T21:23:02.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.790+0000 7f3f1a567700 1 -- 172.21.15.121:0/3887117479 shutdown_connections 2024-04-05T21:23:02.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.790+0000 7f3f1a567700 1 --2- 172.21.15.121:0/3887117479 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3f0006c4e0 0x7f3f0006e9a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:02.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.790+0000 7f3f1a567700 1 --2- 172.21.15.121:0/3887117479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f14100de0 0x7f3f141966d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:02.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.790+0000 7f3f1a567700 1 --2- 172.21.15.121:0/3887117479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f14101790 0x7f3f14196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:02.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.790+0000 7f3f1a567700 1 -- 172.21.15.121:0/3887117479 >> 172.21.15.121:0/3887117479 conn(0x7f3f140fc670 msgr2=0x7f3f140fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:02.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.790+0000 7f3f1a567700 1 -- 172.21.15.121:0/3887117479 shutdown_connections 2024-04-05T21:23:02.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:02.790+0000 7f3f1a567700 1 -- 172.21.15.121:0/3887117479 wait complete. 2024-04-05T21:23:03.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:03 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/467809465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-05T21:23:03.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:03 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3887117479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-05T21:23:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:03 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/467809465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-05T21:23:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:03 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/3887117479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-05T21:23:03.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.770+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2406613085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5c8105a40 msgr2=0x7fd5c8105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:03.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.770+0000 7fd5cd55e700 1 --2- 172.21.15.121:0/2406613085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5c8105a40 0x7fd5c8105e20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc009910 tx=0x7fd5bc009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:03.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.770+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2406613085 shutdown_connections 2024-04-05T21:23:03.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.770+0000 7fd5cd55e700 1 --2- 172.21.15.121:0/2406613085 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5c80ffbd0 0x7fd5c8100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:03.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.770+0000 7fd5cd55e700 1 --2- 172.21.15.121:0/2406613085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5c8105a40 0x7fd5c8105e20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:03.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.770+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2406613085 >> 172.21.15.121:0/2406613085 conn(0x7fd5c80755a0 msgr2=0x7fd5c80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.770+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2406613085 shutdown_connections 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.770+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2406613085 wait complete. 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.770+0000 7fd5cd55e700 1 Processor -- start 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5cd55e700 1 -- start start 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5cd55e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5c80ffbd0 0x7fd5c819c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5cd55e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5c8105a40 0x7fd5c819cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5cd55e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c819d150 con 0x7fd5c80ffbd0 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5cd55e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c8196600 con 0x7fd5c8105a40 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5c6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5c80ffbd0 0x7fd5c819c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5c6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5c80ffbd0 0x7fd5c819c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38470/0 (socket says 172.21.15.121:38470) 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5c6ffd700 1 -- 172.21.15.121:0/2579064801 learned_addr learned my addr 172.21.15.121:0/2579064801 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5c67fc700 1 --2- 172.21.15.121:0/2579064801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5c8105a40 0x7fd5c819cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5c6ffd700 1 -- 172.21.15.121:0/2579064801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5c8105a40 msgr2=0x7fd5c819cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5c6ffd700 1 --2- 172.21.15.121:0/2579064801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5c8105a40 0x7fd5c819cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:03.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5c6ffd700 1 -- 172.21.15.121:0/2579064801 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5bc0095f0 con 0x7fd5c80ffbd0 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5c67fc700 1 --2- 172.21.15.121:0/2579064801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5c8105a40 0x7fd5c819cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5c6ffd700 1 --2- 172.21.15.121:0/2579064801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5c80ffbd0 0x7fd5c819c580 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc006010 tx=0x7fd5bc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5affff700 1 -- 172.21.15.121:0/2579064801 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5bc01d070 con 0x7fd5c80ffbd0 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5affff700 1 -- 172.21.15.121:0/2579064801 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fd5bc00f460 con 0x7fd5c80ffbd0 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2579064801 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c8196880 con 0x7fd5c80ffbd0 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5affff700 1 -- 172.21.15.121:0/2579064801 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5bc021640 con 0x7fd5c80ffbd0 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.771+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2579064801 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c8196d70 con 0x7fd5c80ffbd0 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.772+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2579064801 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c8108040 con 0x7fd5c80ffbd0 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.773+0000 7fd5affff700 1 -- 172.21.15.121:0/2579064801 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fd5bc00fa60 con 0x7fd5c80ffbd0 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.773+0000 7fd5affff700 1 --2- 172.21.15.121:0/2579064801 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fd5b006c4e0 0x7fd5b006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.773+0000 7fd5affff700 1 -- 172.21.15.121:0/2579064801 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd5bc08c800 con 0x7fd5c80ffbd0 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.773+0000 7fd5c67fc700 1 --2- 172.21.15.121:0/2579064801 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fd5b006c4e0 0x7fd5b006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.774+0000 7fd5c67fc700 1 --2- 172.21.15.121:0/2579064801 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fd5b006c4e0 0x7fd5b006e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8104df0 tx=0x7fd5b8008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:03.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.778+0000 7fd5affff700 1 -- 172.21.15.121:0/2579064801 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd5bc05adb0 con 0x7fd5c80ffbd0 2024-04-05T21:23:03.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.970+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2579064801 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd5c804ea90 con 0x7fd5c80ffbd0 2024-04-05T21:23:03.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.971+0000 7fd5affff700 1 -- 172.21.15.121:0/2579064801 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd5bc05a940 con 0x7fd5c80ffbd0 2024-04-05T21:23:03.972 INFO:teuthology.orchestra.run.smithi121.stdout:60129542180 2024-04-05T21:23:03.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.973+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2579064801 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fd5b006c4e0 msgr2=0x7fd5b006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:03.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.973+0000 7fd5cd55e700 1 --2- 172.21.15.121:0/2579064801 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fd5b006c4e0 0x7fd5b006e9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8104df0 tx=0x7fd5b8008040 comp rx=0 tx=0).stop 2024-04-05T21:23:03.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.973+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2579064801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5c80ffbd0 msgr2=0x7fd5c819c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:03.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.973+0000 7fd5cd55e700 1 --2- 172.21.15.121:0/2579064801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5c80ffbd0 0x7fd5c819c580 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc006010 tx=0x7fd5bc00be20 comp rx=0 tx=0).stop 2024-04-05T21:23:03.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.973+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2579064801 shutdown_connections 2024-04-05T21:23:03.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.973+0000 7fd5cd55e700 1 --2- 172.21.15.121:0/2579064801 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fd5b006c4e0 0x7fd5b006e9a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:03.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.973+0000 7fd5cd55e700 1 --2- 172.21.15.121:0/2579064801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5c80ffbd0 0x7fd5c819c580 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:03.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.973+0000 7fd5cd55e700 1 --2- 172.21.15.121:0/2579064801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5c8105a40 0x7fd5c819cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:03.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.973+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2579064801 >> 172.21.15.121:0/2579064801 conn(0x7fd5c80755a0 msgr2=0x7fd5c80fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:03.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.973+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2579064801 shutdown_connections 2024-04-05T21:23:03.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:03.973+0000 7fd5cd55e700 1 -- 172.21.15.121:0/2579064801 wait complete. 2024-04-05T21:23:04.238 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705703 got 38654705705 for osd.0 2024-04-05T21:23:04.238 DEBUG:teuthology.parallel:result is None 2024-04-05T21:23:04.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:04 smithi121 ceph-mon[39723]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:04.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:04 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2579064801' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-05T21:23:04.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:04 smithi188 ceph-mon[49715]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:04.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:04 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2579064801' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-05T21:23:04.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.886+0000 7f1f8befd700 1 -- 172.21.15.121:0/1829037621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f84068d30 msgr2=0x7f1f8410d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:04.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.886+0000 7f1f8befd700 1 --2- 172.21.15.121:0/1829037621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f84068d30 0x7f1f8410d240 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009910 tx=0x7f1f80009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:04.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.886+0000 7f1f8befd700 1 -- 172.21.15.121:0/1829037621 shutdown_connections 2024-04-05T21:23:04.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.886+0000 7f1f8befd700 1 --2- 172.21.15.121:0/1829037621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f84068d30 0x7f1f8410d240 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.886+0000 7f1f8befd700 1 --2- 172.21.15.121:0/1829037621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f84068410 0x7f1f840687f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.886+0000 7f1f8befd700 1 -- 172.21.15.121:0/1829037621 >> 172.21.15.121:0/1829037621 conn(0x7f1f840755a0 msgr2=0x7f1f840759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.886+0000 7f1f8befd700 1 -- 172.21.15.121:0/1829037621 shutdown_connections 2024-04-05T21:23:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.886+0000 7f1f8befd700 1 -- 172.21.15.121:0/1829037621 wait complete. 2024-04-05T21:23:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.886+0000 7f1f8befd700 1 Processor -- start 2024-04-05T21:23:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.886+0000 7f1f8befd700 1 -- start start 2024-04-05T21:23:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f8befd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f84068410 0x7f1f8419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f8befd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f84068d30 0x7f1f8419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f8befd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8419f1f0 con 0x7f1f84068d30 2024-04-05T21:23:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f8befd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f841986a0 con 0x7f1f84068410 2024-04-05T21:23:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f89498700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f84068d30 0x7f1f8419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f89498700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f84068d30 0x7f1f8419eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57986/0 (socket says 172.21.15.121:57986) 2024-04-05T21:23:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f89498700 1 -- 172.21.15.121:0/1360011739 learned_addr learned my addr 172.21.15.121:0/1360011739 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:04.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f89c99700 1 --2- 172.21.15.121:0/1360011739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f84068410 0x7f1f8419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:04.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f89498700 1 -- 172.21.15.121:0/1360011739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f84068410 msgr2=0x7f1f8419e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:04.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f89498700 1 --2- 172.21.15.121:0/1360011739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f84068410 0x7f1f8419e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:04.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f89498700 1 -- 172.21.15.121:0/1360011739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f800095f0 con 0x7f1f84068d30 2024-04-05T21:23:04.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f89498700 1 --2- 172.21.15.121:0/1360011739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f84068d30 0x7f1f8419eb60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1f80006010 tx=0x7f1f8000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:04.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f7affd700 1 -- 172.21.15.121:0/1360011739 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f8001d070 con 0x7f1f84068d30 2024-04-05T21:23:04.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f7affd700 1 -- 172.21.15.121:0/1360011739 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f1f80005710 con 0x7f1f84068d30 2024-04-05T21:23:04.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f8befd700 1 -- 172.21.15.121:0/1360011739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f84198980 con 0x7f1f84068d30 2024-04-05T21:23:04.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f7affd700 1 -- 172.21.15.121:0/1360011739 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f8000fdc0 con 0x7f1f84068d30 2024-04-05T21:23:04.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.887+0000 7f1f8befd700 1 -- 172.21.15.121:0/1360011739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f84198ed0 con 0x7f1f84068d30 2024-04-05T21:23:04.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.888+0000 7f1f8befd700 1 -- 172.21.15.121:0/1360011739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8410a9b0 con 0x7f1f84068d30 2024-04-05T21:23:04.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.889+0000 7f1f7affd700 1 -- 172.21.15.121:0/1360011739 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f1f8000f5f0 con 0x7f1f84068d30 2024-04-05T21:23:04.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.889+0000 7f1f7affd700 1 --2- 172.21.15.121:0/1360011739 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1f7006c4e0 0x7f1f7006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:04.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.889+0000 7f1f89c99700 1 --2- 172.21.15.121:0/1360011739 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1f7006c4e0 0x7f1f7006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:04.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.889+0000 7f1f7affd700 1 -- 172.21.15.121:0/1360011739 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1f8008c650 con 0x7f1f84068d30 2024-04-05T21:23:04.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.890+0000 7f1f89c99700 1 --2- 172.21.15.121:0/1360011739 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1f7006c4e0 0x7f1f7006e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f84076b00 tx=0x7f1f7400b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:04.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:04.894+0000 7f1f7affd700 1 -- 172.21.15.121:0/1360011739 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1f8005ac00 con 0x7f1f84068d30 2024-04-05T21:23:05.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:05.086+0000 7f1f8befd700 1 -- 172.21.15.121:0/1360011739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1f8404ea90 con 0x7f1f84068d30 2024-04-05T21:23:05.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:05.086+0000 7f1f7affd700 1 -- 172.21.15.121:0/1360011739 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1f8005a790 con 0x7f1f84068d30 2024-04-05T21:23:05.087 INFO:teuthology.orchestra.run.smithi121.stdout:167503724556 2024-04-05T21:23:05.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:05.088+0000 7f1f8befd700 1 -- 172.21.15.121:0/1360011739 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1f7006c4e0 msgr2=0x7f1f7006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:05.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:05.088+0000 7f1f8befd700 1 --2- 172.21.15.121:0/1360011739 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1f7006c4e0 0x7f1f7006e9a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f84076b00 tx=0x7f1f7400b410 comp rx=0 tx=0).stop 2024-04-05T21:23:05.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:05.088+0000 7f1f8befd700 1 -- 172.21.15.121:0/1360011739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f84068d30 msgr2=0x7f1f8419eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:05.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:05.088+0000 7f1f8befd700 1 --2- 172.21.15.121:0/1360011739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f84068d30 0x7f1f8419eb60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1f80006010 tx=0x7f1f8000bd70 comp rx=0 tx=0).stop 2024-04-05T21:23:05.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:05.088+0000 7f1f8befd700 1 -- 172.21.15.121:0/1360011739 shutdown_connections 2024-04-05T21:23:05.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:05.088+0000 7f1f8befd700 1 --2- 172.21.15.121:0/1360011739 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1f7006c4e0 0x7f1f7006e9a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:05.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:05.088+0000 7f1f8befd700 1 --2- 172.21.15.121:0/1360011739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f84068410 0x7f1f8419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:05.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:05.088+0000 7f1f8befd700 1 --2- 172.21.15.121:0/1360011739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f84068d30 0x7f1f8419eb60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:05.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:05.088+0000 7f1f8befd700 1 -- 172.21.15.121:0/1360011739 >> 172.21.15.121:0/1360011739 conn(0x7f1f840755a0 msgr2=0x7f1f840fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:05.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:05.088+0000 7f1f8befd700 1 -- 172.21.15.121:0/1360011739 shutdown_connections 2024-04-05T21:23:05.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:05.088+0000 7f1f8befd700 1 -- 172.21.15.121:0/1360011739 wait complete. 2024-04-05T21:23:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:05 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1360011739' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-05T21:23:05.521 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542177 got 60129542180 for osd.1 2024-04-05T21:23:05.521 DEBUG:teuthology.parallel:result is None 2024-04-05T21:23:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:05 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1360011739' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-05T21:23:05.859 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:05.859 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:05.883 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888079 got 146028888081 for osd.4 2024-04-05T21:23:05.883 DEBUG:teuthology.parallel:result is None 2024-04-05T21:23:06.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:06 smithi121 ceph-mon[39723]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:06.530 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724554 got 167503724556 for osd.5 2024-04-05T21:23:06.530 DEBUG:teuthology.parallel:result is None 2024-04-05T21:23:06.776 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:06 smithi188 ceph-mon[49715]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:08.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:08 smithi121 ceph-mon[39723]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:08.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:08 smithi188 ceph-mon[49715]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:09.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.599+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/4245191506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb00ffa40 msgr2=0x7f1cb00ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:09.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.599+0000 7f1cb7ea0700 1 --2- 172.21.15.121:0/4245191506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb00ffa40 0x7f1cb00ffec0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1cac009910 tx=0x7f1cac009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:09.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.599+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/4245191506 shutdown_connections 2024-04-05T21:23:09.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.599+0000 7f1cb7ea0700 1 --2- 172.21.15.121:0/4245191506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb00ffa40 0x7f1cb00ffec0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:09.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.599+0000 7f1cb7ea0700 1 --2- 172.21.15.121:0/4245191506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cb0105a40 0x7f1cb0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:09.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.599+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/4245191506 >> 172.21.15.121:0/4245191506 conn(0x7f1cb00fb570 msgr2=0x7f1cb00fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.599+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/4245191506 shutdown_connections 2024-04-05T21:23:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.599+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/4245191506 wait complete. 2024-04-05T21:23:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.600+0000 7f1cb7ea0700 1 Processor -- start 2024-04-05T21:23:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.600+0000 7f1cb7ea0700 1 -- start start 2024-04-05T21:23:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.600+0000 7f1cb7ea0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb00ffa40 0x7f1cb019c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.600+0000 7f1cb7ea0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cb0105a40 0x7f1cb019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.600+0000 7f1cb7ea0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb019d100 con 0x7f1cb00ffa40 2024-04-05T21:23:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.600+0000 7f1cb7ea0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb01965b0 con 0x7f1cb0105a40 2024-04-05T21:23:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.600+0000 7f1cb5c3c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb00ffa40 0x7f1cb019c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.600+0000 7f1cb5c3c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb00ffa40 0x7f1cb019c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57998/0 (socket says 172.21.15.121:57998) 2024-04-05T21:23:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.600+0000 7f1cb5c3c700 1 -- 172.21.15.121:0/1392271724 learned_addr learned my addr 172.21.15.121:0/1392271724 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:09.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.601+0000 7f1cb5c3c700 1 -- 172.21.15.121:0/1392271724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cb0105a40 msgr2=0x7f1cb019ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.601+0000 7f1cb5c3c700 1 --2- 172.21.15.121:0/1392271724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cb0105a40 0x7f1cb019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.601+0000 7f1cb5c3c700 1 -- 172.21.15.121:0/1392271724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac0095f0 con 0x7f1cb00ffa40 2024-04-05T21:23:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.601+0000 7f1cb5c3c700 1 --2- 172.21.15.121:0/1392271724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb00ffa40 0x7f1cb019c530 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000b4b0 tx=0x7f1ca000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.601+0000 7f1ca6ffd700 1 -- 172.21.15.121:0/1392271724 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ca0011800 con 0x7f1cb00ffa40 2024-04-05T21:23:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.601+0000 7f1ca6ffd700 1 -- 172.21.15.121:0/1392271724 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f1ca0011df0 con 0x7f1cb00ffa40 2024-04-05T21:23:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.601+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/1392271724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb0196890 con 0x7f1cb00ffa40 2024-04-05T21:23:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.601+0000 7f1ca6ffd700 1 -- 172.21.15.121:0/1392271724 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ca0010e40 con 0x7f1cb00ffa40 2024-04-05T21:23:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.601+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/1392271724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb0196de0 con 0x7f1cb00ffa40 2024-04-05T21:23:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.602+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/1392271724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb0109070 con 0x7f1cb00ffa40 2024-04-05T21:23:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.602+0000 7f1ca6ffd700 1 -- 172.21.15.121:0/1392271724 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f1ca0010380 con 0x7f1cb00ffa40 2024-04-05T21:23:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.603+0000 7f1ca6ffd700 1 --2- 172.21.15.121:0/1392271724 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1c9c06c490 0x7f1c9c06e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.603+0000 7f1ca6ffd700 1 -- 172.21.15.121:0/1392271724 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1ca008b0e0 con 0x7f1cb00ffa40 2024-04-05T21:23:09.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.603+0000 7f1cb543b700 1 --2- 172.21.15.121:0/1392271724 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1c9c06c490 0x7f1c9c06e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:09.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.603+0000 7f1cb543b700 1 --2- 172.21.15.121:0/1392271724 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1c9c06c490 0x7f1c9c06e950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00b5c0 tx=0x7f1cac005960 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:09.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.607+0000 7f1ca6ffd700 1 -- 172.21.15.121:0/1392271724 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1ca0059690 con 0x7f1cb00ffa40 2024-04-05T21:23:09.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.798+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/1392271724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1cb004ea90 con 0x7f1cb00ffa40 2024-04-05T21:23:09.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.798+0000 7f1ca6ffd700 1 -- 172.21.15.121:0/1392271724 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1ca0089e60 con 0x7f1cb00ffa40 2024-04-05T21:23:09.800 INFO:teuthology.orchestra.run.smithi121.stdout:81604378655 2024-04-05T21:23:09.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.800+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/1392271724 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1c9c06c490 msgr2=0x7f1c9c06e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:09.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.800+0000 7f1cb7ea0700 1 --2- 172.21.15.121:0/1392271724 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1c9c06c490 0x7f1c9c06e950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00b5c0 tx=0x7f1cac005960 comp rx=0 tx=0).stop 2024-04-05T21:23:09.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.800+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/1392271724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb00ffa40 msgr2=0x7f1cb019c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:09.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.800+0000 7f1cb7ea0700 1 --2- 172.21.15.121:0/1392271724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb00ffa40 0x7f1cb019c530 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000b4b0 tx=0x7f1ca000b870 comp rx=0 tx=0).stop 2024-04-05T21:23:09.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.800+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/1392271724 shutdown_connections 2024-04-05T21:23:09.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.800+0000 7f1cb7ea0700 1 --2- 172.21.15.121:0/1392271724 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1c9c06c490 0x7f1c9c06e950 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:09.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.800+0000 7f1cb7ea0700 1 --2- 172.21.15.121:0/1392271724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb00ffa40 0x7f1cb019c530 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:09.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.800+0000 7f1cb7ea0700 1 --2- 172.21.15.121:0/1392271724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cb0105a40 0x7f1cb019ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:09.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.800+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/1392271724 >> 172.21.15.121:0/1392271724 conn(0x7f1cb00fb570 msgr2=0x7f1cb0103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:09.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.800+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/1392271724 shutdown_connections 2024-04-05T21:23:09.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.800+0000 7f1cb7ea0700 1 -- 172.21.15.121:0/1392271724 wait complete. 2024-04-05T21:23:09.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.952+0000 7f4b1fd04700 1 -- 172.21.15.121:0/3758335532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b18101750 msgr2=0x7f4b181057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:09.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.952+0000 7f4b1fd04700 1 --2- 172.21.15.121:0/3758335532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b18101750 0x7f4b181057a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4b14009910 tx=0x7f4b14009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:09.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.952+0000 7f4b1fd04700 1 -- 172.21.15.121:0/3758335532 shutdown_connections 2024-04-05T21:23:09.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.952+0000 7f4b1fd04700 1 --2- 172.21.15.121:0/3758335532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b18101750 0x7f4b181057a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:09.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.952+0000 7f4b1fd04700 1 --2- 172.21.15.121:0/3758335532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b18100da0 0x7f4b18101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:09.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.952+0000 7f4b1fd04700 1 -- 172.21.15.121:0/3758335532 >> 172.21.15.121:0/3758335532 conn(0x7f4b180fc630 msgr2=0x7f4b180fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:09.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.952+0000 7f4b1fd04700 1 -- 172.21.15.121:0/3758335532 shutdown_connections 2024-04-05T21:23:09.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.952+0000 7f4b1fd04700 1 -- 172.21.15.121:0/3758335532 wait complete. 2024-04-05T21:23:09.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1fd04700 1 Processor -- start 2024-04-05T21:23:09.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1fd04700 1 -- start start 2024-04-05T21:23:09.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1fd04700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b18100da0 0x7f4b1819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:09.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1fd04700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b18101750 0x7f4b1819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:09.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1fd04700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1819f230 con 0x7f4b18100da0 2024-04-05T21:23:09.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1fd04700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b181986e0 con 0x7f4b18101750 2024-04-05T21:23:09.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1daa0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b18100da0 0x7f4b1819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:09.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1daa0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b18100da0 0x7f4b1819e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58024/0 (socket says 172.21.15.121:58024) 2024-04-05T21:23:09.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1daa0700 1 -- 172.21.15.121:0/659461411 learned_addr learned my addr 172.21.15.121:0/659461411 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:09.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1d29f700 1 --2- 172.21.15.121:0/659461411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b18101750 0x7f4b1819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:09.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1daa0700 1 -- 172.21.15.121:0/659461411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b18101750 msgr2=0x7f4b1819eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:09.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1daa0700 1 --2- 172.21.15.121:0/659461411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b18101750 0x7f4b1819eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:09.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1daa0700 1 -- 172.21.15.121:0/659461411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b140095f0 con 0x7f4b18100da0 2024-04-05T21:23:09.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1d29f700 1 --2- 172.21.15.121:0/659461411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b18101750 0x7f4b1819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:23:09.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.953+0000 7f4b1daa0700 1 --2- 172.21.15.121:0/659461411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b18100da0 0x7f4b1819e660 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800d8d0 tx=0x7f4b0800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.954+0000 7f4b0effd700 1 -- 172.21.15.121:0/659461411 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b08009940 con 0x7f4b18100da0 2024-04-05T21:23:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.954+0000 7f4b0effd700 1 -- 172.21.15.121:0/659461411 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f4b08010460 con 0x7f4b18100da0 2024-04-05T21:23:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.954+0000 7f4b1fd04700 1 -- 172.21.15.121:0/659461411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b181989c0 con 0x7f4b18100da0 2024-04-05T21:23:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.954+0000 7f4b0effd700 1 -- 172.21.15.121:0/659461411 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b0800f5f0 con 0x7f4b18100da0 2024-04-05T21:23:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.954+0000 7f4b1fd04700 1 -- 172.21.15.121:0/659461411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b18198f10 con 0x7f4b18100da0 2024-04-05T21:23:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.955+0000 7f4b1fd04700 1 -- 172.21.15.121:0/659461411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b18104f20 con 0x7f4b18100da0 2024-04-05T21:23:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.955+0000 7f4b0effd700 1 -- 172.21.15.121:0/659461411 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4b080105d0 con 0x7f4b18100da0 2024-04-05T21:23:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.955+0000 7f4b0effd700 1 --2- 172.21.15.121:0/659461411 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4b0406c4e0 0x7f4b0406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.955+0000 7f4b0effd700 1 -- 172.21.15.121:0/659461411 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4b0808b420 con 0x7f4b18100da0 2024-04-05T21:23:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.955+0000 7f4b1d29f700 1 --2- 172.21.15.121:0/659461411 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4b0406c4e0 0x7f4b0406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.956+0000 7f4b1d29f700 1 --2- 172.21.15.121:0/659461411 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4b0406c4e0 0x7f4b0406e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4b18199b50 tx=0x7f4b1401a040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:09.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:09.960+0000 7f4b0effd700 1 -- 172.21.15.121:0/659461411 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4b08059a50 con 0x7f4b18100da0 2024-04-05T21:23:10.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:10.153+0000 7f4b1fd04700 1 -- 172.21.15.121:0/659461411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f4b1804ea90 con 0x7f4b18100da0 2024-04-05T21:23:10.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:10.153+0000 7f4b0effd700 1 -- 172.21.15.121:0/659461411 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4b08010880 con 0x7f4b18100da0 2024-04-05T21:23:10.155 INFO:teuthology.orchestra.run.smithi121.stdout:111669149720 2024-04-05T21:23:10.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:10.155+0000 7f4b1fd04700 1 -- 172.21.15.121:0/659461411 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4b0406c4e0 msgr2=0x7f4b0406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:10.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:10.155+0000 7f4b1fd04700 1 --2- 172.21.15.121:0/659461411 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4b0406c4e0 0x7f4b0406e9a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4b18199b50 tx=0x7f4b1401a040 comp rx=0 tx=0).stop 2024-04-05T21:23:10.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:10.155+0000 7f4b1fd04700 1 -- 172.21.15.121:0/659461411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b18100da0 msgr2=0x7f4b1819e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:10.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:10.155+0000 7f4b1fd04700 1 --2- 172.21.15.121:0/659461411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b18100da0 0x7f4b1819e660 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800d8d0 tx=0x7f4b0800dc90 comp rx=0 tx=0).stop 2024-04-05T21:23:10.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:10.155+0000 7f4b1fd04700 1 -- 172.21.15.121:0/659461411 shutdown_connections 2024-04-05T21:23:10.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:10.155+0000 7f4b1fd04700 1 --2- 172.21.15.121:0/659461411 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4b0406c4e0 0x7f4b0406e9a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:10.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:10.155+0000 7f4b1fd04700 1 --2- 172.21.15.121:0/659461411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b18100da0 0x7f4b1819e660 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:10.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:10.155+0000 7f4b1fd04700 1 --2- 172.21.15.121:0/659461411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4b18101750 0x7f4b1819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:10.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:10.155+0000 7f4b1fd04700 1 -- 172.21.15.121:0/659461411 >> 172.21.15.121:0/659461411 conn(0x7f4b180fc630 msgr2=0x7f4b180fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:10.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:10.155+0000 7f4b1fd04700 1 -- 172.21.15.121:0/659461411 shutdown_connections 2024-04-05T21:23:10.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:10.155+0000 7f4b1fd04700 1 -- 172.21.15.121:0/659461411 wait complete. 2024-04-05T21:23:10.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:10 smithi121 ceph-mon[39723]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:10.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:10 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1392271724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-05T21:23:10.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:10 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:23:10.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:10 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/659461411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-05T21:23:10.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:10 smithi188 ceph-mon[49715]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:10.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:10 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1392271724' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-05T21:23:10.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:10 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:23:10.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:10 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/659461411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-05T21:23:10.877 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378652 got 81604378655 for osd.2 2024-04-05T21:23:10.877 DEBUG:teuthology.parallel:result is None 2024-04-05T21:23:11.319 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149718 got 111669149720 for osd.3 2024-04-05T21:23:11.319 DEBUG:teuthology.parallel:result is None 2024-04-05T21:23:11.319 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-05T21:23:11.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-05T21:23:11.552 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:12.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:12 smithi121 ceph-mon[39723]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:12.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:12 smithi188 ceph-mon[49715]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:13.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.300+0000 7f2f439d3700 1 -- 172.21.15.121:0/3764069754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f3c0fef70 msgr2=0x7f2f3c0ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.300+0000 7f2f439d3700 1 --2- 172.21.15.121:0/3764069754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f3c0fef70 0x7f2f3c0ff3f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2f38009910 tx=0x7f2f38009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.300+0000 7f2f439d3700 1 -- 172.21.15.121:0/3764069754 shutdown_connections 2024-04-05T21:23:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.300+0000 7f2f439d3700 1 --2- 172.21.15.121:0/3764069754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f3c0fef70 0x7f2f3c0ff3f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.300+0000 7f2f439d3700 1 --2- 172.21.15.121:0/3764069754 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f3c105b50 0x7f2f3c105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.300+0000 7f2f439d3700 1 -- 172.21.15.121:0/3764069754 >> 172.21.15.121:0/3764069754 conn(0x7f2f3c075e30 msgr2=0x7f2f3c0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.300+0000 7f2f439d3700 1 -- 172.21.15.121:0/3764069754 shutdown_connections 2024-04-05T21:23:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.300+0000 7f2f439d3700 1 -- 172.21.15.121:0/3764069754 wait complete. 2024-04-05T21:23:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f439d3700 1 Processor -- start 2024-04-05T21:23:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f439d3700 1 -- start start 2024-04-05T21:23:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f439d3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f3c0fef70 0x7f2f3c19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:13.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f439d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f3c105b50 0x7f2f3c19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f439d3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c19d0e0 con 0x7f2f3c105b50 2024-04-05T21:23:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f439d3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c196590 con 0x7f2f3c0fef70 2024-04-05T21:23:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f40f6e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f3c105b50 0x7f2f3c19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f40f6e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f3c105b50 0x7f2f3c19ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58048/0 (socket says 172.21.15.121:58048) 2024-04-05T21:23:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f40f6e700 1 -- 172.21.15.121:0/2080768845 learned_addr learned my addr 172.21.15.121:0/2080768845 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f4176f700 1 --2- 172.21.15.121:0/2080768845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f3c0fef70 0x7f2f3c19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f40f6e700 1 -- 172.21.15.121:0/2080768845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f3c0fef70 msgr2=0x7f2f3c19c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f40f6e700 1 --2- 172.21.15.121:0/2080768845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f3c0fef70 0x7f2f3c19c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f40f6e700 1 -- 172.21.15.121:0/2080768845 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f380095f0 con 0x7f2f3c105b50 2024-04-05T21:23:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.301+0000 7f2f4176f700 1 --2- 172.21.15.121:0/2080768845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f3c0fef70 0x7f2f3c19c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:23:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.302+0000 7f2f40f6e700 1 --2- 172.21.15.121:0/2080768845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f3c105b50 0x7f2f3c19ca50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f38006010 tx=0x7f2f3800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.302+0000 7f2f2e7fc700 1 -- 172.21.15.121:0/2080768845 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f3801d070 con 0x7f2f3c105b50 2024-04-05T21:23:13.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.302+0000 7f2f2e7fc700 1 -- 172.21.15.121:0/2080768845 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f2f3800f460 con 0x7f2f3c105b50 2024-04-05T21:23:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.302+0000 7f2f439d3700 1 -- 172.21.15.121:0/2080768845 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f3c196810 con 0x7f2f3c105b50 2024-04-05T21:23:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.302+0000 7f2f2e7fc700 1 -- 172.21.15.121:0/2080768845 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f38021640 con 0x7f2f3c105b50 2024-04-05T21:23:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.302+0000 7f2f439d3700 1 -- 172.21.15.121:0/2080768845 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3c196d00 con 0x7f2f3c105b50 2024-04-05T21:23:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.303+0000 7f2f439d3700 1 -- 172.21.15.121:0/2080768845 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3c109140 con 0x7f2f3c105b50 2024-04-05T21:23:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.303+0000 7f2f2e7fc700 1 -- 172.21.15.121:0/2080768845 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2f3800fa60 con 0x7f2f3c105b50 2024-04-05T21:23:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.304+0000 7f2f2e7fc700 1 --2- 172.21.15.121:0/2080768845 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2f2806c4e0 0x7f2f2806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.304+0000 7f2f4176f700 1 --2- 172.21.15.121:0/2080768845 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2f2806c4e0 0x7f2f2806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.304+0000 7f2f2e7fc700 1 -- 172.21.15.121:0/2080768845 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2f3808c6b0 con 0x7f2f3c105b50 2024-04-05T21:23:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.304+0000 7f2f4176f700 1 --2- 172.21.15.121:0/2080768845 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2f2806c4e0 0x7f2f2806e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2f30007900 tx=0x7f2f30008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:13.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.308+0000 7f2f2e7fc700 1 -- 172.21.15.121:0/2080768845 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2f3805ace0 con 0x7f2f3c105b50 2024-04-05T21:23:13.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.500+0000 7f2f439d3700 1 -- 172.21.15.121:0/2080768845 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f3c02ccf0 con 0x7f2f2806c4e0 2024-04-05T21:23:13.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.501+0000 7f2f2e7fc700 1 -- 172.21.15.121:0/2080768845 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27902 (secure 0 0 0) 0x7f2f3c02ccf0 con 0x7f2f2806c4e0 2024-04-05T21:23:13.503 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:23:13.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.503+0000 7f2f439d3700 1 -- 172.21.15.121:0/2080768845 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2f2806c4e0 msgr2=0x7f2f2806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:13.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.503+0000 7f2f439d3700 1 --2- 172.21.15.121:0/2080768845 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2f2806c4e0 0x7f2f2806e9a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2f30007900 tx=0x7f2f30008040 comp rx=0 tx=0).stop 2024-04-05T21:23:13.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.503+0000 7f2f439d3700 1 -- 172.21.15.121:0/2080768845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f3c105b50 msgr2=0x7f2f3c19ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:13.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.503+0000 7f2f439d3700 1 --2- 172.21.15.121:0/2080768845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f3c105b50 0x7f2f3c19ca50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f38006010 tx=0x7f2f3800be20 comp rx=0 tx=0).stop 2024-04-05T21:23:13.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.504+0000 7f2f439d3700 1 -- 172.21.15.121:0/2080768845 shutdown_connections 2024-04-05T21:23:13.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.504+0000 7f2f439d3700 1 --2- 172.21.15.121:0/2080768845 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2f2806c4e0 0x7f2f2806e9a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:13.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.504+0000 7f2f439d3700 1 --2- 172.21.15.121:0/2080768845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f3c0fef70 0x7f2f3c19c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:13.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.504+0000 7f2f439d3700 1 --2- 172.21.15.121:0/2080768845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f3c105b50 0x7f2f3c19ca50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:13.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.504+0000 7f2f439d3700 1 -- 172.21.15.121:0/2080768845 >> 172.21.15.121:0/2080768845 conn(0x7f2f3c075e30 msgr2=0x7f2f3c104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:13.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.504+0000 7f2f439d3700 1 -- 172.21.15.121:0/2080768845 shutdown_connections 2024-04-05T21:23:13.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:13.504+0000 7f2f439d3700 1 -- 172.21.15.121:0/2080768845 wait complete. 2024-04-05T21:23:13.506 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-04-05T21:23:14.211 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-04-05T21:23:13.009618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165588,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305324,"statfs":{"total":575970213888,"available":575800651776,"internally_reserved":0,"allocated":3969024,"data_stored":2609166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002374"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":187,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-05T21:22:17.066119+0000","last_change":"2024-04-05T21:21:48.463255+0000","last_active":"2024-04-05T21:22:17.066119+0000","last_peered":"2024-04-05T21:22:17.066119+0000","last_clean":"2024-04-05T21:22:17.066119+0000","last_became_active":"2024-04-05T21:21:48.462941+0000","last_became_peered":"2024-04-05T21:21:48.462941+0000","last_unstale":"2024-04-05T21:22:17.066119+0000","last_undegraded":"2024-04-05T21:22:17.066119+0000","last_fullsized":"2024-04-05T21:22:17.066119+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T21:20:53.341841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T21:20:53.341841+0000","last_clean_scrub_stamp":"2024-04-05T21:20:53.341841+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T22:07:33.116638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 21:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.438}]},{"osd":1,"last update":"Fri Apr 5 21:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.48999999999999999}]},{"osd":2,"last update":"Fri Apr 5 21:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.40200000000000002}]},{"osd":3,"last update":"Fri Apr 5 21:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.377}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]}]},{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 21:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.375},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.40500000000000003}]},{"osd":1,"last update":"Fri Apr 5 21:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.496},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.42899999999999999}]},{"osd":2,"last update":"Fri Apr 5 21:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.39000000000000001}]},{"osd":4,"last update":"Fri Apr 5 21:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.46999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378655,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 21:22:57 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.35399999999999998,"15min":0.36399999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.51800000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.251},{"interface":"front","average":{"1min":0.30099999999999999,"5min":0.35599999999999998,"15min":0.36499999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.45800000000000002,"5min":0.5,"15min":0.5},"last":0.313}]},{"osd":1,"last update":"Fri Apr 5 21:22:57 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.38200000000000001,"15min":0.39100000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.502,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.29699999999999999}]},{"osd":3,"last update":"Fri Apr 5 21:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.443}]},{"osd":4,"last update":"Fri Apr 5 21:22:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705708,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 5 21:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.314,"15min":0.31},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.502,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.31900000000000001,"15min":0.311},"min":{"1min":0.25900000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.246}]},{"osd":2,"last update":"Fri Apr 5 21:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.35299999999999998,"15min":0.35799999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.318,"5min":0.34200000000000003,"15min":0.34599999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.254}]},{"osd":3,"last update":"Fri Apr 5 21:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.63100000000000001}]},{"osd":4,"last update":"Fri Apr 5 21:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.46899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542182,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 21:22:25 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.308,"15min":0.30099999999999999},"min":{"1min":0.23100000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.312},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.308,"15min":0.29799999999999999},"min":{"1min":0.26100000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.34000000000000002}]},{"osd":2,"last update":"Fri Apr 5 21:22:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.35699999999999998,"15min":0.35299999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.35899999999999999,"15min":0.35599999999999998},"min":{"1min":0.27400000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.39600000000000002}]},{"osd":3,"last update":"Fri Apr 5 21:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.47299999999999998}]},{"osd":4,"last update":"Fri Apr 5 21:22:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.54700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-05T21:23:14.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-05T21:23:14.438 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:14.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:14 smithi121 ceph-mon[39723]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:14.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:14 smithi188 ceph-mon[49715]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:15.485 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:15 smithi121 ceph-mon[39723]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T21:23:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:15 smithi188 ceph-mon[49715]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T21:23:16.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.498+0000 7f19e4d50700 1 -- 172.21.15.121:0/4009906697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e00ffa40 msgr2=0x7f19e00ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:16.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.498+0000 7f19e4d50700 1 --2- 172.21.15.121:0/4009906697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e00ffa40 0x7f19e00ffec0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009910 tx=0x7f19d4009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:16.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.499+0000 7f19e4d50700 1 -- 172.21.15.121:0/4009906697 shutdown_connections 2024-04-05T21:23:16.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.499+0000 7f19e4d50700 1 --2- 172.21.15.121:0/4009906697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e00ffa40 0x7f19e00ffec0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:16.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.499+0000 7f19e4d50700 1 --2- 172.21.15.121:0/4009906697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f19e0105a40 0x7f19e0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:16.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.499+0000 7f19e4d50700 1 -- 172.21.15.121:0/4009906697 >> 172.21.15.121:0/4009906697 conn(0x7f19e00fb570 msgr2=0x7f19e00fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:16.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.499+0000 7f19e4d50700 1 -- 172.21.15.121:0/4009906697 shutdown_connections 2024-04-05T21:23:16.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.499+0000 7f19e4d50700 1 -- 172.21.15.121:0/4009906697 wait complete. 2024-04-05T21:23:16.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.499+0000 7f19e4d50700 1 Processor -- start 2024-04-05T21:23:16.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.499+0000 7f19e4d50700 1 -- start start 2024-04-05T21:23:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19e4d50700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f19e00ffa40 0x7f19e0196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19e4d50700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e0105a40 0x7f19e0196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19e4d50700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e0197230 con 0x7f19e0105a40 2024-04-05T21:23:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19e4d50700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e019afc0 con 0x7f19e00ffa40 2024-04-05T21:23:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19ddd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e0105a40 0x7f19e0196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19ddd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e0105a40 0x7f19e0196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54908/0 (socket says 172.21.15.121:54908) 2024-04-05T21:23:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19ddd9b700 1 -- 172.21.15.121:0/960028136 learned_addr learned my addr 172.21.15.121:0/960028136 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19de59c700 1 --2- 172.21.15.121:0/960028136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f19e00ffa40 0x7f19e0196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19ddd9b700 1 -- 172.21.15.121:0/960028136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f19e00ffa40 msgr2=0x7f19e0196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19ddd9b700 1 --2- 172.21.15.121:0/960028136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f19e00ffa40 0x7f19e0196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19ddd9b700 1 -- 172.21.15.121:0/960028136 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d40095f0 con 0x7f19e0105a40 2024-04-05T21:23:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19de59c700 1 --2- 172.21.15.121:0/960028136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f19e00ffa40 0x7f19e0196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:23:16.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19ddd9b700 1 --2- 172.21.15.121:0/960028136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e0105a40 0x7f19e0196b50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f19d4006010 tx=0x7f19d400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:16.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19cf7fe700 1 -- 172.21.15.121:0/960028136 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19d401d070 con 0x7f19e0105a40 2024-04-05T21:23:16.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19cf7fe700 1 -- 172.21.15.121:0/960028136 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f19d400f460 con 0x7f19e0105a40 2024-04-05T21:23:16.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19e4d50700 1 -- 172.21.15.121:0/960028136 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e019b240 con 0x7f19e0105a40 2024-04-05T21:23:16.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19cf7fe700 1 -- 172.21.15.121:0/960028136 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19d4021620 con 0x7f19e0105a40 2024-04-05T21:23:16.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.500+0000 7f19e4d50700 1 -- 172.21.15.121:0/960028136 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e019b730 con 0x7f19e0105a40 2024-04-05T21:23:16.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.501+0000 7f19e4d50700 1 -- 172.21.15.121:0/960028136 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e0108fe0 con 0x7f19e0105a40 2024-04-05T21:23:16.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.502+0000 7f19cf7fe700 1 -- 172.21.15.121:0/960028136 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f19d4003a60 con 0x7f19e0105a40 2024-04-05T21:23:16.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.502+0000 7f19cf7fe700 1 --2- 172.21.15.121:0/960028136 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f19c806c4e0 0x7f19c806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:16.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.502+0000 7f19cf7fe700 1 -- 172.21.15.121:0/960028136 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f19d408c710 con 0x7f19e0105a40 2024-04-05T21:23:16.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.502+0000 7f19de59c700 1 --2- 172.21.15.121:0/960028136 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f19c806c4e0 0x7f19c806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:16.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.502+0000 7f19de59c700 1 --2- 172.21.15.121:0/960028136 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f19c806c4e0 0x7f19c806e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f19d0005fd0 tx=0x7f19d0005ee0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:16.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.507+0000 7f19cf7fe700 1 -- 172.21.15.121:0/960028136 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f19d4057230 con 0x7f19e0105a40 2024-04-05T21:23:16.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.689+0000 7f19e4d50700 1 -- 172.21.15.121:0/960028136 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19e002cd60 con 0x7f19c806c4e0 2024-04-05T21:23:16.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.691+0000 7f19cf7fe700 1 -- 172.21.15.121:0/960028136 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27930 (secure 0 0 0) 0x7f19e002cd60 con 0x7f19c806c4e0 2024-04-05T21:23:16.693 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:23:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.693+0000 7f19e4d50700 1 -- 172.21.15.121:0/960028136 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f19c806c4e0 msgr2=0x7f19c806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.693+0000 7f19e4d50700 1 --2- 172.21.15.121:0/960028136 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f19c806c4e0 0x7f19c806e9a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f19d0005fd0 tx=0x7f19d0005ee0 comp rx=0 tx=0).stop 2024-04-05T21:23:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.693+0000 7f19e4d50700 1 -- 172.21.15.121:0/960028136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e0105a40 msgr2=0x7f19e0196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.693+0000 7f19e4d50700 1 --2- 172.21.15.121:0/960028136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e0105a40 0x7f19e0196b50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f19d4006010 tx=0x7f19d400be20 comp rx=0 tx=0).stop 2024-04-05T21:23:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.693+0000 7f19e4d50700 1 -- 172.21.15.121:0/960028136 shutdown_connections 2024-04-05T21:23:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.693+0000 7f19e4d50700 1 --2- 172.21.15.121:0/960028136 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f19c806c4e0 0x7f19c806e9a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.693+0000 7f19e4d50700 1 --2- 172.21.15.121:0/960028136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f19e00ffa40 0x7f19e0196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.693+0000 7f19e4d50700 1 --2- 172.21.15.121:0/960028136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e0105a40 0x7f19e0196b50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.693+0000 7f19e4d50700 1 -- 172.21.15.121:0/960028136 >> 172.21.15.121:0/960028136 conn(0x7f19e00fb570 msgr2=0x7f19e0068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.693+0000 7f19e4d50700 1 -- 172.21.15.121:0/960028136 shutdown_connections 2024-04-05T21:23:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:16.693+0000 7f19e4d50700 1 -- 172.21.15.121:0/960028136 wait complete. 2024-04-05T21:23:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-04-05T21:23:16.721 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:16 smithi121 ceph-mon[39723]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:16.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:16 smithi188 ceph-mon[49715]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:17.738 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-04-05T21:23:15.009988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":165588,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":161664,"kb_avail":562305324,"statfs":{"total":575970213888,"available":575800651776,"internally_reserved":0,"allocated":3969024,"data_stored":2609166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":165543936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002390"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":187,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-05T21:22:17.066119+0000","last_change":"2024-04-05T21:21:48.463255+0000","last_active":"2024-04-05T21:22:17.066119+0000","last_peered":"2024-04-05T21:22:17.066119+0000","last_clean":"2024-04-05T21:22:17.066119+0000","last_became_active":"2024-04-05T21:21:48.462941+0000","last_became_peered":"2024-04-05T21:21:48.462941+0000","last_unstale":"2024-04-05T21:22:17.066119+0000","last_undegraded":"2024-04-05T21:22:17.066119+0000","last_fullsized":"2024-04-05T21:22:17.066119+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-05T21:20:53.341841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-05T21:20:53.341841+0000","last_clean_scrub_stamp":"2024-04-05T21:20:53.341841+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-06T22:07:33.116638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724559,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888084,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 21:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.435}]},{"osd":1,"last update":"Fri Apr 5 21:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.52000000000000002}]},{"osd":2,"last update":"Fri Apr 5 21:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.499}]},{"osd":3,"last update":"Fri Apr 5 21:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.41499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 21:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.48999999999999999}]},{"osd":1,"last update":"Fri Apr 5 21:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Apr 5 21:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.66000000000000003}]},{"osd":4,"last update":"Fri Apr 5 21:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.379}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378656,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27308,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717844,"statfs":{"total":95995035648,"available":95967072256,"internally_reserved":0,"allocated":364544,"data_stored":139677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 21:22:57 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.35399999999999998,"15min":0.36399999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.51800000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.252},{"interface":"front","average":{"1min":0.30099999999999999,"5min":0.35599999999999998,"15min":0.36499999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.45800000000000002,"5min":0.5,"15min":0.5},"last":0.26700000000000002}]},{"osd":1,"last update":"Fri Apr 5 21:22:57 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.38200000000000001,"15min":0.39100000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.502,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.23599999999999999}]},{"osd":3,"last update":"Fri Apr 5 21:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52100000000000002}]},{"osd":4,"last update":"Fri Apr 5 21:22:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.503},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705708,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 5 21:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.314,"15min":0.31},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.502,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.31900000000000001,"15min":0.311},"min":{"1min":0.25900000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.246}]},{"osd":2,"last update":"Fri Apr 5 21:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.35299999999999998,"15min":0.35799999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.318,"5min":0.34200000000000003,"15min":0.34599999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.254}]},{"osd":3,"last update":"Fri Apr 5 21:22:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.63100000000000001}]},{"osd":4,"last update":"Fri Apr 5 21:22:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.46899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542182,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":730045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 5 21:22:25 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.308,"15min":0.30099999999999999},"min":{"1min":0.23100000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.312},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.308,"15min":0.29799999999999999},"min":{"1min":0.26100000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.34000000000000002}]},{"osd":2,"last update":"Fri Apr 5 21:22:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.35699999999999998,"15min":0.35299999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.35899999999999999,"15min":0.35599999999999998},"min":{"1min":0.27400000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.39600000000000002}]},{"osd":3,"last update":"Fri Apr 5 21:22:19 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.47299999999999998}]},{"osd":4,"last update":"Fri Apr 5 21:22:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.54700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-05T21:23:17.740 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-05T21:23:17.740 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-05T21:23:17.741 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-05T21:23:17.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph health --format=json 2024-04-05T21:23:17.968 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:18.465 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:18 smithi121 ceph-mon[39723]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T21:23:18.465 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:18 smithi121 ceph-mon[39723]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:18.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:18 smithi188 ceph-mon[49715]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-05T21:23:18.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:18 smithi188 ceph-mon[49715]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:19.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.748+0000 7fea37c64700 1 -- 172.21.15.121:0/1176932568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea300ffbd0 msgr2=0x7fea30100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:19.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.748+0000 7fea37c64700 1 --2- 172.21.15.121:0/1176932568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea300ffbd0 0x7fea30100050 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fea20009910 tx=0x7fea20009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:19.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.748+0000 7fea37c64700 1 -- 172.21.15.121:0/1176932568 shutdown_connections 2024-04-05T21:23:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.748+0000 7fea37c64700 1 --2- 172.21.15.121:0/1176932568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea300ffbd0 0x7fea30100050 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.748+0000 7fea37c64700 1 --2- 172.21.15.121:0/1176932568 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea30105a40 0x7fea30105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.748+0000 7fea37c64700 1 -- 172.21.15.121:0/1176932568 >> 172.21.15.121:0/1176932568 conn(0x7fea300755a0 msgr2=0x7fea300759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.749+0000 7fea37c64700 1 -- 172.21.15.121:0/1176932568 shutdown_connections 2024-04-05T21:23:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.749+0000 7fea37c64700 1 -- 172.21.15.121:0/1176932568 wait complete. 2024-04-05T21:23:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.749+0000 7fea37c64700 1 Processor -- start 2024-04-05T21:23:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.749+0000 7fea37c64700 1 -- start start 2024-04-05T21:23:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.749+0000 7fea37c64700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea300ffbd0 0x7fea301975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.749+0000 7fea37c64700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea30105a40 0x7fea30197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.749+0000 7fea37c64700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea301981e0 con 0x7fea30105a40 2024-04-05T21:23:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.749+0000 7fea37c64700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea30198320 con 0x7fea300ffbd0 2024-04-05T21:23:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.749+0000 7fea351ff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea30105a40 0x7fea30197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.749+0000 7fea351ff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea30105a40 0x7fea30197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54934/0 (socket says 172.21.15.121:54934) 2024-04-05T21:23:19.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.749+0000 7fea351ff700 1 -- 172.21.15.121:0/4200973221 learned_addr learned my addr 172.21.15.121:0/4200973221 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:19.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.749+0000 7fea35a00700 1 --2- 172.21.15.121:0/4200973221 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea300ffbd0 0x7fea301975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:19.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.750+0000 7fea351ff700 1 -- 172.21.15.121:0/4200973221 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea300ffbd0 msgr2=0x7fea301975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:19.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.750+0000 7fea351ff700 1 --2- 172.21.15.121:0/4200973221 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea300ffbd0 0x7fea301975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:19.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.750+0000 7fea351ff700 1 -- 172.21.15.121:0/4200973221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea200095f0 con 0x7fea30105a40 2024-04-05T21:23:19.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.750+0000 7fea351ff700 1 --2- 172.21.15.121:0/4200973221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea30105a40 0x7fea30197b00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fea20006010 tx=0x7fea2000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:19.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.750+0000 7fea26ffd700 1 -- 172.21.15.121:0/4200973221 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea2001d070 con 0x7fea30105a40 2024-04-05T21:23:19.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.750+0000 7fea26ffd700 1 -- 172.21.15.121:0/4200973221 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fea2000f460 con 0x7fea30105a40 2024-04-05T21:23:19.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.750+0000 7fea26ffd700 1 -- 172.21.15.121:0/4200973221 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea20021640 con 0x7fea30105a40 2024-04-05T21:23:19.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.750+0000 7fea37c64700 1 -- 172.21.15.121:0/4200973221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea301a0630 con 0x7fea30105a40 2024-04-05T21:23:19.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.750+0000 7fea37c64700 1 -- 172.21.15.121:0/4200973221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea301a0aa0 con 0x7fea30105a40 2024-04-05T21:23:19.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.751+0000 7fea37c64700 1 -- 172.21.15.121:0/4200973221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea30108060 con 0x7fea30105a40 2024-04-05T21:23:19.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.751+0000 7fea26ffd700 1 -- 172.21.15.121:0/4200973221 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fea2000fa20 con 0x7fea30105a40 2024-04-05T21:23:19.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.752+0000 7fea26ffd700 1 --2- 172.21.15.121:0/4200973221 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fea1c06c4e0 0x7fea1c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:19.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.752+0000 7fea35a00700 1 --2- 172.21.15.121:0/4200973221 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fea1c06c4e0 0x7fea1c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:19.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.752+0000 7fea26ffd700 1 -- 172.21.15.121:0/4200973221 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fea2008c6e0 con 0x7fea30105a40 2024-04-05T21:23:19.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.752+0000 7fea35a00700 1 --2- 172.21.15.121:0/4200973221 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fea1c06c4e0 0x7fea1c06e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fea2c006fd0 tx=0x7fea2c008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:19.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:19.756+0000 7fea26ffd700 1 -- 172.21.15.121:0/4200973221 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fea20057280 con 0x7fea30105a40 2024-04-05T21:23:20.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:20.000+0000 7fea37c64700 1 -- 172.21.15.121:0/4200973221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fea3004ea90 con 0x7fea30105a40 2024-04-05T21:23:20.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:20.000+0000 7fea26ffd700 1 -- 172.21.15.121:0/4200973221 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fea2005a8a0 con 0x7fea30105a40 2024-04-05T21:23:20.002 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:23:20.002 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-05T21:23:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:20.002+0000 7fea37c64700 1 -- 172.21.15.121:0/4200973221 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fea1c06c4e0 msgr2=0x7fea1c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:20.002+0000 7fea37c64700 1 --2- 172.21.15.121:0/4200973221 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fea1c06c4e0 0x7fea1c06e9a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fea2c006fd0 tx=0x7fea2c008040 comp rx=0 tx=0).stop 2024-04-05T21:23:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:20.002+0000 7fea37c64700 1 -- 172.21.15.121:0/4200973221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea30105a40 msgr2=0x7fea30197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:20.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:20.002+0000 7fea37c64700 1 --2- 172.21.15.121:0/4200973221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea30105a40 0x7fea30197b00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fea20006010 tx=0x7fea2000be20 comp rx=0 tx=0).stop 2024-04-05T21:23:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:20.002+0000 7fea37c64700 1 -- 172.21.15.121:0/4200973221 shutdown_connections 2024-04-05T21:23:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:20.002+0000 7fea37c64700 1 --2- 172.21.15.121:0/4200973221 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fea1c06c4e0 0x7fea1c06e9a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:20.002+0000 7fea37c64700 1 --2- 172.21.15.121:0/4200973221 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea300ffbd0 0x7fea301975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:20.002+0000 7fea37c64700 1 --2- 172.21.15.121:0/4200973221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea30105a40 0x7fea30197b00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:20.002+0000 7fea37c64700 1 -- 172.21.15.121:0/4200973221 >> 172.21.15.121:0/4200973221 conn(0x7fea300755a0 msgr2=0x7fea300fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:20.002+0000 7fea37c64700 1 -- 172.21.15.121:0/4200973221 shutdown_connections 2024-04-05T21:23:20.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:20.002+0000 7fea37c64700 1 -- 172.21.15.121:0/4200973221 wait complete. 2024-04-05T21:23:20.716 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:20 smithi121 ceph-mon[39723]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:20.716 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:23:20.717 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:20 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/4200973221' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-05T21:23:20.717 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:20.717 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:23:20.717 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:20.735 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-05T21:23:20.735 INFO:tasks.cephadm:Setup complete, yielding 2024-04-05T21:23:20.735 INFO:teuthology.run_tasks:Running task print... 2024-04-05T21:23:20.745 INFO:teuthology.task.print:**** done end installing v18.2.1 cephadm ... 2024-04-05T21:23:20.745 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T21:23:20.767 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T21:23:20.768 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-05T21:23:20.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:20 smithi188 ceph-mon[49715]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:20.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:23:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:20 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/4200973221' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-05T21:23:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:23:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:20.993 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:22.663 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:22 smithi121 ceph-mon[39723]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:22.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:22 smithi188 ceph-mon[49715]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:22.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.938+0000 7fb008be5700 1 -- 172.21.15.121:0/1764348868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0040ffab0 msgr2=0x7fb0040ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:22.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.938+0000 7fb008be5700 1 --2- 172.21.15.121:0/1764348868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0040ffab0 0x7fb0040ffed0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7faff4009910 tx=0x7faff4009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:22.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.938+0000 7fb008be5700 1 -- 172.21.15.121:0/1764348868 shutdown_connections 2024-04-05T21:23:22.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.938+0000 7fb008be5700 1 --2- 172.21.15.121:0/1764348868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb004100cb0 0x7fb004101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:22.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.938+0000 7fb008be5700 1 --2- 172.21.15.121:0/1764348868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0040ffab0 0x7fb0040ffed0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:22.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.938+0000 7fb008be5700 1 -- 172.21.15.121:0/1764348868 >> 172.21.15.121:0/1764348868 conn(0x7fb0040fb030 msgr2=0x7fb0040fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:22.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.938+0000 7fb008be5700 1 -- 172.21.15.121:0/1764348868 shutdown_connections 2024-04-05T21:23:22.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.938+0000 7fb008be5700 1 -- 172.21.15.121:0/1764348868 wait complete. 2024-04-05T21:23:22.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.938+0000 7fb008be5700 1 Processor -- start 2024-04-05T21:23:22.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.938+0000 7fb008be5700 1 -- start start 2024-04-05T21:23:22.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb008be5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0040ffab0 0x7fb004107a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:22.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb008be5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb004100cb0 0x7fb0041060b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:22.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb008be5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0041065f0 con 0x7fb0040ffab0 2024-04-05T21:23:22.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb008be5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb004106730 con 0x7fb004100cb0 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb00259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0040ffab0 0x7fb004107a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb00259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0040ffab0 0x7fb004107a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54954/0 (socket says 172.21.15.121:54954) 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb00259c700 1 -- 172.21.15.121:0/1149982479 learned_addr learned my addr 172.21.15.121:0/1149982479 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb001d9b700 1 --2- 172.21.15.121:0/1149982479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb004100cb0 0x7fb0041060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb00259c700 1 -- 172.21.15.121:0/1149982479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb004100cb0 msgr2=0x7fb0041060b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb00259c700 1 --2- 172.21.15.121:0/1149982479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb004100cb0 0x7fb0041060b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb00259c700 1 -- 172.21.15.121:0/1149982479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff40095f0 con 0x7fb0040ffab0 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb001d9b700 1 --2- 172.21.15.121:0/1149982479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb004100cb0 0x7fb0041060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb00259c700 1 --2- 172.21.15.121:0/1149982479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0040ffab0 0x7fb004107a60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7faff400bda0 tx=0x7faff400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7faff37fe700 1 -- 172.21.15.121:0/1149982479 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faff401d070 con 0x7fb0040ffab0 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7faff37fe700 1 -- 172.21.15.121:0/1149982479 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7faff400f460 con 0x7fb0040ffab0 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb008be5700 1 -- 172.21.15.121:0/1149982479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0041069b0 con 0x7fb0040ffab0 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7faff37fe700 1 -- 172.21.15.121:0/1149982479 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faff4021640 con 0x7fb0040ffab0 2024-04-05T21:23:22.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.939+0000 7fb008be5700 1 -- 172.21.15.121:0/1149982479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb004106ea0 con 0x7fb0040ffab0 2024-04-05T21:23:22.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.940+0000 7fb008be5700 1 -- 172.21.15.121:0/1149982479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0041095f0 con 0x7fb0040ffab0 2024-04-05T21:23:22.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.941+0000 7faff37fe700 1 -- 172.21.15.121:0/1149982479 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7faff400f9b0 con 0x7fb0040ffab0 2024-04-05T21:23:22.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.941+0000 7faff37fe700 1 --2- 172.21.15.121:0/1149982479 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fafec06c4e0 0x7fafec06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:22.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.941+0000 7faff37fe700 1 -- 172.21.15.121:0/1149982479 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7faff408c800 con 0x7fb0040ffab0 2024-04-05T21:23:22.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.941+0000 7fb001d9b700 1 --2- 172.21.15.121:0/1149982479 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fafec06c4e0 0x7fafec06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:22.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.942+0000 7fb001d9b700 1 --2- 172.21.15.121:0/1149982479 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fafec06c4e0 0x7fafec06e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faff8005fd0 tx=0x7faff8005f60 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:22.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:22.946+0000 7faff37fe700 1 -- 172.21.15.121:0/1149982479 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7faff405adb0 con 0x7fb0040ffab0 2024-04-05T21:23:23.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:23.134+0000 7fb008be5700 1 -- 172.21.15.121:0/1149982479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7fb0041097b0 con 0x7fb0040ffab0 2024-04-05T21:23:23.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:23.138+0000 7faff37fe700 1 -- 172.21.15.121:0/1149982479 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7faff405a940 con 0x7fb0040ffab0 2024-04-05T21:23:23.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:23.140+0000 7fb008be5700 1 -- 172.21.15.121:0/1149982479 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fafec06c4e0 msgr2=0x7fafec06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:23.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:23.140+0000 7fb008be5700 1 --2- 172.21.15.121:0/1149982479 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fafec06c4e0 0x7fafec06e9a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faff8005fd0 tx=0x7faff8005f60 comp rx=0 tx=0).stop 2024-04-05T21:23:23.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:23.140+0000 7fb008be5700 1 -- 172.21.15.121:0/1149982479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0040ffab0 msgr2=0x7fb004107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:23.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:23.140+0000 7fb008be5700 1 --2- 172.21.15.121:0/1149982479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0040ffab0 0x7fb004107a60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7faff400bda0 tx=0x7faff400be80 comp rx=0 tx=0).stop 2024-04-05T21:23:23.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:23.140+0000 7fb008be5700 1 -- 172.21.15.121:0/1149982479 shutdown_connections 2024-04-05T21:23:23.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:23.140+0000 7fb008be5700 1 --2- 172.21.15.121:0/1149982479 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fafec06c4e0 0x7fafec06e9a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:23.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:23.140+0000 7fb008be5700 1 --2- 172.21.15.121:0/1149982479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0040ffab0 0x7fb004107a60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:23.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:23.140+0000 7fb008be5700 1 --2- 172.21.15.121:0/1149982479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb004100cb0 0x7fb0041060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:23.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:23.140+0000 7fb008be5700 1 -- 172.21.15.121:0/1149982479 >> 172.21.15.121:0/1149982479 conn(0x7fb0040fb030 msgr2=0x7fb004103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:23.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:23.140+0000 7fb008be5700 1 -- 172.21.15.121:0/1149982479 shutdown_connections 2024-04-05T21:23:23.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:23.141+0000 7fb008be5700 1 -- 172.21.15.121:0/1149982479 wait complete. 2024-04-05T21:23:23.865 INFO:teuthology.run_tasks:Running task print... 2024-04-05T21:23:23.875 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-05T21:23:23.875 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T21:23:23.989 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T21:23:23.989 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-04-05T21:23:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:24.420 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:24 smithi121 ceph-mon[39723]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:24.420 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:24 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1149982479' entity='client.admin' 2024-04-05T21:23:24.420 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:24 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:23:24.420 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:24 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:24.421 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:24 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:23:24.421 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:24 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:24.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:24 smithi188 ceph-mon[49715]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:24.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:24 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1149982479' entity='client.admin' 2024-04-05T21:23:24.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:24 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:23:24.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:24 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:24.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:24 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:23:24.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:24 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:25.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:23:25.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:25 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:23:26.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.083+0000 7fabee31e700 1 -- 172.21.15.121:0/776976955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe8100cb0 msgr2=0x7fabe8101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:26.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.083+0000 7fabee31e700 1 --2- 172.21.15.121:0/776976955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe8100cb0 0x7fabe8101110 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fabdc009910 tx=0x7fabdc009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:26.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.083+0000 7fabee31e700 1 -- 172.21.15.121:0/776976955 shutdown_connections 2024-04-05T21:23:26.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.083+0000 7fabee31e700 1 --2- 172.21.15.121:0/776976955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe8100cb0 0x7fabe8101110 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:26.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.083+0000 7fabee31e700 1 --2- 172.21.15.121:0/776976955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabe80ffab0 0x7fabe80ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:26.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.083+0000 7fabee31e700 1 -- 172.21.15.121:0/776976955 >> 172.21.15.121:0/776976955 conn(0x7fabe80fb030 msgr2=0x7fabe80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:26.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.083+0000 7fabee31e700 1 -- 172.21.15.121:0/776976955 shutdown_connections 2024-04-05T21:23:26.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.083+0000 7fabee31e700 1 -- 172.21.15.121:0/776976955 wait complete. 2024-04-05T21:23:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabee31e700 1 Processor -- start 2024-04-05T21:23:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabee31e700 1 -- start start 2024-04-05T21:23:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabee31e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe80ffab0 0x7fabe8068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabee31e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabe8100cb0 0x7fabe819b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabee31e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe819b6b0 con 0x7fabe80ffab0 2024-04-05T21:23:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabee31e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe819b7f0 con 0x7fabe8100cb0 2024-04-05T21:23:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabe7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe80ffab0 0x7fabe8068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabe7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe80ffab0 0x7fabe8068f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38630/0 (socket says 172.21.15.121:38630) 2024-04-05T21:23:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabe7fff700 1 -- 172.21.15.121:0/3576173404 learned_addr learned my addr 172.21.15.121:0/3576173404 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabe77fe700 1 --2- 172.21.15.121:0/3576173404 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabe8100cb0 0x7fabe819b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabe7fff700 1 -- 172.21.15.121:0/3576173404 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabe8100cb0 msgr2=0x7fabe819b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabe7fff700 1 --2- 172.21.15.121:0/3576173404 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabe8100cb0 0x7fabe819b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabe7fff700 1 -- 172.21.15.121:0/3576173404 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc0095f0 con 0x7fabe80ffab0 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.084+0000 7fabe77fe700 1 --2- 172.21.15.121:0/3576173404 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabe8100cb0 0x7fabe819b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.085+0000 7fabe7fff700 1 --2- 172.21.15.121:0/3576173404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe80ffab0 0x7fabe8068f70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fabd800b480 tx=0x7fabd800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.085+0000 7fabe57fa700 1 -- 172.21.15.121:0/3576173404 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabd80118c0 con 0x7fabe80ffab0 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.085+0000 7fabe57fa700 1 -- 172.21.15.121:0/3576173404 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fabd8010380 con 0x7fabe80ffab0 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.085+0000 7fabee31e700 1 -- 172.21.15.121:0/3576173404 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe819bad0 con 0x7fabe80ffab0 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.085+0000 7fabe57fa700 1 -- 172.21.15.121:0/3576173404 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabd800f510 con 0x7fabe80ffab0 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.085+0000 7fabee31e700 1 -- 172.21.15.121:0/3576173404 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe819c0a0 con 0x7fabe80ffab0 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.086+0000 7fabee31e700 1 -- 172.21.15.121:0/3576173404 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe8108570 con 0x7fabe80ffab0 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.086+0000 7fabe57fa700 1 -- 172.21.15.121:0/3576173404 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fabd8011a20 con 0x7fabe80ffab0 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.086+0000 7fabe57fa700 1 --2- 172.21.15.121:0/3576173404 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fabd006c4e0 0x7fabd006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.087+0000 7fabe57fa700 1 -- 172.21.15.121:0/3576173404 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fabd808b180 con 0x7fabe80ffab0 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.087+0000 7fabe77fe700 1 --2- 172.21.15.121:0/3576173404 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fabd006c4e0 0x7fabd006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.087+0000 7fabe77fe700 1 --2- 172.21.15.121:0/3576173404 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fabd006c4e0 0x7fabd006e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fabdc006010 tx=0x7fabdc005960 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.091+0000 7fabe57fa700 1 -- 172.21.15.121:0/3576173404 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fabd8059730 con 0x7fabe80ffab0 2024-04-05T21:23:26.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.293+0000 7fabee31e700 1 -- 172.21.15.121:0/3576173404 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fabe8105600 con 0x7fabd006c4e0 2024-04-05T21:23:26.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.294+0000 7fabe57fa700 1 -- 172.21.15.121:0/3576173404 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fabe8105600 con 0x7fabd006c4e0 2024-04-05T21:23:26.295 INFO:teuthology.orchestra.run.smithi121.stdout:Backend: cephadm 2024-04-05T21:23:26.295 INFO:teuthology.orchestra.run.smithi121.stdout:Available: Yes 2024-04-05T21:23:26.295 INFO:teuthology.orchestra.run.smithi121.stdout:Paused: No 2024-04-05T21:23:26.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.296+0000 7fabee31e700 1 -- 172.21.15.121:0/3576173404 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fabd006c4e0 msgr2=0x7fabd006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:26.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.296+0000 7fabee31e700 1 --2- 172.21.15.121:0/3576173404 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fabd006c4e0 0x7fabd006e9a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fabdc006010 tx=0x7fabdc005960 comp rx=0 tx=0).stop 2024-04-05T21:23:26.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.296+0000 7fabee31e700 1 -- 172.21.15.121:0/3576173404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe80ffab0 msgr2=0x7fabe8068f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:26.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.296+0000 7fabee31e700 1 --2- 172.21.15.121:0/3576173404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe80ffab0 0x7fabe8068f70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fabd800b480 tx=0x7fabd800b840 comp rx=0 tx=0).stop 2024-04-05T21:23:26.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.296+0000 7fabee31e700 1 -- 172.21.15.121:0/3576173404 shutdown_connections 2024-04-05T21:23:26.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.296+0000 7fabee31e700 1 --2- 172.21.15.121:0/3576173404 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fabd006c4e0 0x7fabd006e9a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:26.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.296+0000 7fabee31e700 1 --2- 172.21.15.121:0/3576173404 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabe80ffab0 0x7fabe8068f70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:26.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.296+0000 7fabee31e700 1 --2- 172.21.15.121:0/3576173404 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabe8100cb0 0x7fabe819b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:26.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.296+0000 7fabee31e700 1 -- 172.21.15.121:0/3576173404 >> 172.21.15.121:0/3576173404 conn(0x7fabe80fb030 msgr2=0x7fabe8103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:26.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.296+0000 7fabee31e700 1 -- 172.21.15.121:0/3576173404 shutdown_connections 2024-04-05T21:23:26.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:26.296+0000 7fabee31e700 1 -- 172.21.15.121:0/3576173404 wait complete. 2024-04-05T21:23:26.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:26 smithi188 ceph-mon[49715]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:26.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:26 smithi121 ceph-mon[39723]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:27.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-04-05T21:23:27.342 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:27.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:27 smithi121 ceph-mon[39723]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:23:27.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:27 smithi188 ceph-mon[49715]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:23:28.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:28 smithi188 ceph-mon[49715]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:28 smithi121 ceph-mon[39723]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:29.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.395+0000 7f0564867700 1 -- 172.21.15.121:0/1307049019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0560100cb0 msgr2=0x7f0560101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:29.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.395+0000 7f0564867700 1 --2- 172.21.15.121:0/1307049019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0560100cb0 0x7f0560101110 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0548009910 tx=0x7f0548009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:29.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.395+0000 7f0564867700 1 -- 172.21.15.121:0/1307049019 shutdown_connections 2024-04-05T21:23:29.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.395+0000 7f0564867700 1 --2- 172.21.15.121:0/1307049019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0560100cb0 0x7f0560101110 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:29.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.395+0000 7f0564867700 1 --2- 172.21.15.121:0/1307049019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05600ffab0 0x7f05600ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:29.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.395+0000 7f0564867700 1 -- 172.21.15.121:0/1307049019 >> 172.21.15.121:0/1307049019 conn(0x7f05600fb050 msgr2=0x7f05600fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:29.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.395+0000 7f0564867700 1 -- 172.21.15.121:0/1307049019 shutdown_connections 2024-04-05T21:23:29.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.395+0000 7f0564867700 1 -- 172.21.15.121:0/1307049019 wait complete. 2024-04-05T21:23:29.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.396+0000 7f0564867700 1 Processor -- start 2024-04-05T21:23:29.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.396+0000 7f0564867700 1 -- start start 2024-04-05T21:23:29.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.396+0000 7f0564867700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05600ffab0 0x7f0560195170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:29.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.396+0000 7f0564867700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0560100cb0 0x7f05601956b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:29.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.396+0000 7f0564867700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0560195cd0 con 0x7f0560100cb0 2024-04-05T21:23:29.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.396+0000 7f0564867700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0560195e10 con 0x7f05600ffab0 2024-04-05T21:23:29.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.396+0000 7f055dd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0560100cb0 0x7f05601956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:29.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.396+0000 7f055dd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0560100cb0 0x7f05601956b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38646/0 (socket says 172.21.15.121:38646) 2024-04-05T21:23:29.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.396+0000 7f055dd9b700 1 -- 172.21.15.121:0/191001045 learned_addr learned my addr 172.21.15.121:0/191001045 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:29.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.397+0000 7f055e59c700 1 --2- 172.21.15.121:0/191001045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05600ffab0 0x7f0560195170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:29.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.397+0000 7f055dd9b700 1 -- 172.21.15.121:0/191001045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05600ffab0 msgr2=0x7f0560195170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:29.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.397+0000 7f055dd9b700 1 --2- 172.21.15.121:0/191001045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05600ffab0 0x7f0560195170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:29.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.397+0000 7f055dd9b700 1 -- 172.21.15.121:0/191001045 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05480095f0 con 0x7f0560100cb0 2024-04-05T21:23:29.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.397+0000 7f055dd9b700 1 --2- 172.21.15.121:0/191001045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0560100cb0 0x7f05601956b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f054800b5c0 tx=0x7f054800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:29.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.397+0000 7f05577fe700 1 -- 172.21.15.121:0/191001045 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f054801d070 con 0x7f0560100cb0 2024-04-05T21:23:29.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.397+0000 7f05577fe700 1 -- 172.21.15.121:0/191001045 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f054800f460 con 0x7f0560100cb0 2024-04-05T21:23:29.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.397+0000 7f0564867700 1 -- 172.21.15.121:0/191001045 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f056019a860 con 0x7f0560100cb0 2024-04-05T21:23:29.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.397+0000 7f05577fe700 1 -- 172.21.15.121:0/191001045 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0548021640 con 0x7f0560100cb0 2024-04-05T21:23:29.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.397+0000 7f0564867700 1 -- 172.21.15.121:0/191001045 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f056019add0 con 0x7f0560100cb0 2024-04-05T21:23:29.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.398+0000 7f0564867700 1 -- 172.21.15.121:0/191001045 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05601084d0 con 0x7f0560100cb0 2024-04-05T21:23:29.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.398+0000 7f05577fe700 1 -- 172.21.15.121:0/191001045 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f054800fa20 con 0x7f0560100cb0 2024-04-05T21:23:29.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.399+0000 7f05577fe700 1 --2- 172.21.15.121:0/191001045 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f054c06c4e0 0x7f054c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:29.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.399+0000 7f05577fe700 1 -- 172.21.15.121:0/191001045 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f054808c6e0 con 0x7f0560100cb0 2024-04-05T21:23:29.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.399+0000 7f055e59c700 1 --2- 172.21.15.121:0/191001045 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f054c06c4e0 0x7f054c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:29.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.399+0000 7f055e59c700 1 --2- 172.21.15.121:0/191001045 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f054c06c4e0 0x7f054c06e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0560100b10 tx=0x7f055000b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:29.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.403+0000 7f05577fe700 1 -- 172.21.15.121:0/191001045 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f054805ad10 con 0x7f0560100cb0 2024-04-05T21:23:29.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.599+0000 7f0564867700 1 -- 172.21.15.121:0/191001045 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0560105600 con 0x7f054c06c4e0 2024-04-05T21:23:29.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.608+0000 7f05577fe700 1 -- 172.21.15.121:0/191001045 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2860 (secure 0 0 0) 0x7f0560105600 con 0x7f054c06c4e0 2024-04-05T21:23:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:23:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (4m) 2m ago 5m 26.2M - 0.25.0 c8568f914cd2 6594a12b2bc8 2024-04-05T21:23:29.609 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (6m) 2m ago 6m 7079k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (4m) 73s ago 4m 6949k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (6m) 2m ago 5m 6651k - 18.2.1 5be31c24972a 5bf35fe2ace4 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (4m) 73s ago 4m 6651k - 18.2.1 5be31c24972a dcdfa6ae17bf 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (4m) 2m ago 5m 78.4M - 9.4.7 954c08fa6188 ee38366c66e3 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:9283,8765,8443 running (7m) 2m ago 7m 482M - 18.2.1 5be31c24972a d8cb1548d844 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (4m) 73s ago 4m 439M - 18.2.1 5be31c24972a a38c0d6743cb 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (7m) 2m ago 7m 46.2M 2048M 18.2.1 5be31c24972a 3c30ce99c006 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (4m) 73s ago 4m 40.3M 2048M 18.2.1 5be31c24972a 8ecc9636670e 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (5m) 2m ago 5m 17.4M - 1.5.0 0da6a335fe13 5ac35283f3e9 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (4m) 73s ago 4m 16.5M - 1.5.0 0da6a335fe13 bc2ad104140f 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (3m) 2m ago 3m 40.5M 3958M 18.2.1 5be31c24972a f6f3a45429a2 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (3m) 2m ago 3m 42.9M 3958M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (2m) 2m ago 2m 40.0M 3958M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:23:29.610 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (2m) 73s ago 2m 43.5M 4982M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:23:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (108s) 73s ago 107s 38.8M 4982M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:23:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (79s) 73s ago 79s 11.4M 4982M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:23:29.611 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (3m) 2m ago 5m 41.2M - 2.43.0 a07b618ecd1d 9395525b1c3e 2024-04-05T21:23:29.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.610+0000 7f0564867700 1 -- 172.21.15.121:0/191001045 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f054c06c4e0 msgr2=0x7f054c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:29.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.610+0000 7f0564867700 1 --2- 172.21.15.121:0/191001045 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f054c06c4e0 0x7f054c06e9a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0560100b10 tx=0x7f055000b410 comp rx=0 tx=0).stop 2024-04-05T21:23:29.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.610+0000 7f0564867700 1 -- 172.21.15.121:0/191001045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0560100cb0 msgr2=0x7f05601956b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:29.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.610+0000 7f0564867700 1 --2- 172.21.15.121:0/191001045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0560100cb0 0x7f05601956b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f054800b5c0 tx=0x7f054800bd70 comp rx=0 tx=0).stop 2024-04-05T21:23:29.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.610+0000 7f0564867700 1 -- 172.21.15.121:0/191001045 shutdown_connections 2024-04-05T21:23:29.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.610+0000 7f0564867700 1 --2- 172.21.15.121:0/191001045 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f054c06c4e0 0x7f054c06e9a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:29.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.610+0000 7f0564867700 1 --2- 172.21.15.121:0/191001045 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05600ffab0 0x7f0560195170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:29.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.610+0000 7f0564867700 1 --2- 172.21.15.121:0/191001045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0560100cb0 0x7f05601956b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:29.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.610+0000 7f0564867700 1 -- 172.21.15.121:0/191001045 >> 172.21.15.121:0/191001045 conn(0x7f05600fb050 msgr2=0x7f0560103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:29.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.611+0000 7f0564867700 1 -- 172.21.15.121:0/191001045 shutdown_connections 2024-04-05T21:23:29.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:29.611+0000 7f0564867700 1 -- 172.21.15.121:0/191001045 wait complete. 2024-04-05T21:23:30.305 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-04-05T21:23:30.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:30 smithi188 ceph-mon[49715]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:30.532 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:30 smithi121 ceph-mon[39723]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:31 smithi188 ceph-mon[49715]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:23:31.569 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:31 smithi121 ceph-mon[39723]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:23:32.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.466+0000 7f8149c2e700 1 -- 172.21.15.121:0/855527666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8144101db0 msgr2=0x7f8144102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.466+0000 7f8149c2e700 1 --2- 172.21.15.121:0/855527666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8144101db0 0x7f8144102210 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8138009910 tx=0x7f8138009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.466+0000 7f8149c2e700 1 -- 172.21.15.121:0/855527666 shutdown_connections 2024-04-05T21:23:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.466+0000 7f8149c2e700 1 --2- 172.21.15.121:0/855527666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8144101db0 0x7f8144102210 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.466+0000 7f8149c2e700 1 --2- 172.21.15.121:0/855527666 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8144100bb0 0x7f8144100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.466+0000 7f8149c2e700 1 -- 172.21.15.121:0/855527666 >> 172.21.15.121:0/855527666 conn(0x7f81440fc110 msgr2=0x7f81440fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.466+0000 7f8149c2e700 1 -- 172.21.15.121:0/855527666 shutdown_connections 2024-04-05T21:23:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.466+0000 7f8149c2e700 1 -- 172.21.15.121:0/855527666 wait complete. 2024-04-05T21:23:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.466+0000 7f8149c2e700 1 Processor -- start 2024-04-05T21:23:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.467+0000 7f8149c2e700 1 -- start start 2024-04-05T21:23:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.467+0000 7f8149c2e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8144100bb0 0x7f81441962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.467+0000 7f8149c2e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8144101db0 0x7f81441967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.467+0000 7f8149c2e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8144196e00 con 0x7f8144101db0 2024-04-05T21:23:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.467+0000 7f8149c2e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8144196f40 con 0x7f8144100bb0 2024-04-05T21:23:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.467+0000 7f8142ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8144101db0 0x7f81441967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.467+0000 7f8142ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8144101db0 0x7f81441967e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38668/0 (socket says 172.21.15.121:38668) 2024-04-05T21:23:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.467+0000 7f8142ffd700 1 -- 172.21.15.121:0/2440014444 learned_addr learned my addr 172.21.15.121:0/2440014444 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.467+0000 7f81437fe700 1 --2- 172.21.15.121:0/2440014444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8144100bb0 0x7f81441962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.467+0000 7f8142ffd700 1 -- 172.21.15.121:0/2440014444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8144100bb0 msgr2=0x7f81441962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.467+0000 7f8142ffd700 1 --2- 172.21.15.121:0/2440014444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8144100bb0 0x7f81441962a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.467+0000 7f8142ffd700 1 -- 172.21.15.121:0/2440014444 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81380095f0 con 0x7f8144101db0 2024-04-05T21:23:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.467+0000 7f8142ffd700 1 --2- 172.21.15.121:0/2440014444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8144101db0 0x7f81441967e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8138006010 tx=0x7f813800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.468+0000 7f8140ff9700 1 -- 172.21.15.121:0/2440014444 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f813801d070 con 0x7f8144101db0 2024-04-05T21:23:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.468+0000 7f8140ff9700 1 -- 172.21.15.121:0/2440014444 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f813800f460 con 0x7f8144101db0 2024-04-05T21:23:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.468+0000 7f8149c2e700 1 -- 172.21.15.121:0/2440014444 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f814419b990 con 0x7f8144101db0 2024-04-05T21:23:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.468+0000 7f8140ff9700 1 -- 172.21.15.121:0/2440014444 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8138021640 con 0x7f8144101db0 2024-04-05T21:23:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.468+0000 7f8149c2e700 1 -- 172.21.15.121:0/2440014444 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f814419bf00 con 0x7f8144101db0 2024-04-05T21:23:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.469+0000 7f8140ff9700 1 -- 172.21.15.121:0/2440014444 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f813800f9b0 con 0x7f8144101db0 2024-04-05T21:23:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.469+0000 7f8149c2e700 1 -- 172.21.15.121:0/2440014444 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81441095d0 con 0x7f8144101db0 2024-04-05T21:23:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.469+0000 7f8140ff9700 1 --2- 172.21.15.121:0/2440014444 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f812c06c4e0 0x7f812c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.469+0000 7f8140ff9700 1 -- 172.21.15.121:0/2440014444 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f813808c730 con 0x7f8144101db0 2024-04-05T21:23:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.469+0000 7f81437fe700 1 --2- 172.21.15.121:0/2440014444 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f812c06c4e0 0x7f812c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.470+0000 7f81437fe700 1 --2- 172.21.15.121:0/2440014444 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f812c06c4e0 0x7f812c06e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8144101c10 tx=0x7f813400b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:32.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.474+0000 7f8140ff9700 1 -- 172.21.15.121:0/2440014444 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f813805ace0 con 0x7f8144101db0 2024-04-05T21:23:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:32 smithi121 ceph-mon[39723]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:32.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:32 smithi188 ceph-mon[49715]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:32.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.672+0000 7f8149c2e700 1 -- 172.21.15.121:0/2440014444 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8144106700 con 0x7f812c06c4e0 2024-04-05T21:23:32.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.676+0000 7f8140ff9700 1 -- 172.21.15.121:0/2440014444 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1310 (secure 0 0 0) 0x7f8144106700 con 0x7f812c06c4e0 2024-04-05T21:23:32.677 INFO:teuthology.orchestra.run.smithi121.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-05T21:23:32.677 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-05T21:23:32.678 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter 2/2 2m ago 6m * 2024-04-05T21:23:32.678 INFO:teuthology.orchestra.run.smithi121.stdout:crash 2/2 2m ago 6m * 2024-04-05T21:23:32.678 INFO:teuthology.orchestra.run.smithi121.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-05T21:23:32.678 INFO:teuthology.orchestra.run.smithi121.stdout:mgr 2/2 2m ago 6m count:2 2024-04-05T21:23:32.678 INFO:teuthology.orchestra.run.smithi121.stdout:mon 2/2 2m ago 5m smithi121:172.21.15.121=smithi121;smithi188:172.21.15.188=smithi188;count:2 2024-04-05T21:23:32.678 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-05T21:23:32.678 INFO:teuthology.orchestra.run.smithi121.stdout:osd 6 2m ago - 2024-04-05T21:23:32.678 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-05T21:23:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.678+0000 7f8149c2e700 1 -- 172.21.15.121:0/2440014444 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f812c06c4e0 msgr2=0x7f812c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.678+0000 7f8149c2e700 1 --2- 172.21.15.121:0/2440014444 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f812c06c4e0 0x7f812c06e9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8144101c10 tx=0x7f813400b410 comp rx=0 tx=0).stop 2024-04-05T21:23:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.678+0000 7f8149c2e700 1 -- 172.21.15.121:0/2440014444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8144101db0 msgr2=0x7f81441967e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.678+0000 7f8149c2e700 1 --2- 172.21.15.121:0/2440014444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8144101db0 0x7f81441967e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8138006010 tx=0x7f813800bd70 comp rx=0 tx=0).stop 2024-04-05T21:23:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.678+0000 7f8149c2e700 1 -- 172.21.15.121:0/2440014444 shutdown_connections 2024-04-05T21:23:32.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.678+0000 7f8149c2e700 1 --2- 172.21.15.121:0/2440014444 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f812c06c4e0 0x7f812c06e9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:32.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.678+0000 7f8149c2e700 1 --2- 172.21.15.121:0/2440014444 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8144100bb0 0x7f81441962a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:32.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.678+0000 7f8149c2e700 1 --2- 172.21.15.121:0/2440014444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8144101db0 0x7f81441967e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:32.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.678+0000 7f8149c2e700 1 -- 172.21.15.121:0/2440014444 >> 172.21.15.121:0/2440014444 conn(0x7f81440fc110 msgr2=0x7f8144104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:32.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.678+0000 7f8149c2e700 1 -- 172.21.15.121:0/2440014444 shutdown_connections 2024-04-05T21:23:32.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:32.678+0000 7f8149c2e700 1 -- 172.21.15.121:0/2440014444 wait complete. 2024-04-05T21:23:33.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-04-05T21:23:33.685 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:34.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:34 smithi121 ceph-mon[39723]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:23:34.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:34 smithi121 ceph-mon[39723]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:34.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:34 smithi188 ceph-mon[49715]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:23:34.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:34 smithi188 ceph-mon[49715]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:35.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.558+0000 7fceea578700 1 -- 172.21.15.121:0/264209347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100440 msgr2=0x7fcee41008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:35.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.558+0000 7fceea578700 1 --2- 172.21.15.121:0/264209347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100440 0x7fcee41008a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fced8009910 tx=0x7fced8009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.558+0000 7fceea578700 1 -- 172.21.15.121:0/264209347 shutdown_connections 2024-04-05T21:23:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.558+0000 7fceea578700 1 --2- 172.21.15.121:0/264209347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100440 0x7fcee41008a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.558+0000 7fceea578700 1 --2- 172.21.15.121:0/264209347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcee40ff240 0x7fcee40ff660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.558+0000 7fceea578700 1 -- 172.21.15.121:0/264209347 >> 172.21.15.121:0/264209347 conn(0x7fcee40fa7c0 msgr2=0x7fcee40fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.558+0000 7fceea578700 1 -- 172.21.15.121:0/264209347 shutdown_connections 2024-04-05T21:23:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.558+0000 7fceea578700 1 -- 172.21.15.121:0/264209347 wait complete. 2024-04-05T21:23:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.558+0000 7fceea578700 1 Processor -- start 2024-04-05T21:23:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.558+0000 7fceea578700 1 -- start start 2024-04-05T21:23:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fceea578700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcee40ff240 0x7fcee4196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fceea578700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100440 0x7fcee4196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fceea578700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee4196eb0 con 0x7fcee4100440 2024-04-05T21:23:35.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fceea578700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee4196ff0 con 0x7fcee40ff240 2024-04-05T21:23:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fcee37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100440 0x7fcee4196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fcee37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100440 0x7fcee4196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35246/0 (socket says 172.21.15.121:35246) 2024-04-05T21:23:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fcee37fe700 1 -- 172.21.15.121:0/4271017969 learned_addr learned my addr 172.21.15.121:0/4271017969 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fcee3fff700 1 --2- 172.21.15.121:0/4271017969 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcee40ff240 0x7fcee4196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fcee37fe700 1 -- 172.21.15.121:0/4271017969 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcee40ff240 msgr2=0x7fcee4196350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fcee37fe700 1 --2- 172.21.15.121:0/4271017969 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcee40ff240 0x7fcee4196350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fcee37fe700 1 -- 172.21.15.121:0/4271017969 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced80095f0 con 0x7fcee4100440 2024-04-05T21:23:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fcee3fff700 1 --2- 172.21.15.121:0/4271017969 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcee40ff240 0x7fcee4196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:23:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fcee37fe700 1 --2- 172.21.15.121:0/4271017969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100440 0x7fcee4196890 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fced8006010 tx=0x7fced800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fcee17fa700 1 -- 172.21.15.121:0/4271017969 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fced801d070 con 0x7fcee4100440 2024-04-05T21:23:35.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fcee17fa700 1 -- 172.21.15.121:0/4271017969 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fced800f460 con 0x7fcee4100440 2024-04-05T21:23:35.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fceea578700 1 -- 172.21.15.121:0/4271017969 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee419ba40 con 0x7fcee4100440 2024-04-05T21:23:35.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fcee17fa700 1 -- 172.21.15.121:0/4271017969 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fced8021640 con 0x7fcee4100440 2024-04-05T21:23:35.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.559+0000 7fceea578700 1 -- 172.21.15.121:0/4271017969 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee419bf30 con 0x7fcee4100440 2024-04-05T21:23:35.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.560+0000 7fceea578700 1 -- 172.21.15.121:0/4271017969 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee4190210 con 0x7fcee4100440 2024-04-05T21:23:35.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.561+0000 7fcee17fa700 1 -- 172.21.15.121:0/4271017969 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fced800f940 con 0x7fcee4100440 2024-04-05T21:23:35.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.561+0000 7fcee17fa700 1 --2- 172.21.15.121:0/4271017969 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fcecc06c4e0 0x7fcecc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:35.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.561+0000 7fcee17fa700 1 -- 172.21.15.121:0/4271017969 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fced808c7e0 con 0x7fcee4100440 2024-04-05T21:23:35.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.561+0000 7fcee3fff700 1 --2- 172.21.15.121:0/4271017969 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fcecc06c4e0 0x7fcecc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:35.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.561+0000 7fcee3fff700 1 --2- 172.21.15.121:0/4271017969 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fcecc06c4e0 0x7fcecc06e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fced4005fd0 tx=0x7fced4005f00 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:35.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.565+0000 7fcee17fa700 1 -- 172.21.15.121:0/4271017969 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fced805ad90 con 0x7fcee4100440 2024-04-05T21:23:35.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.759+0000 7fceea578700 1 -- 172.21.15.121:0/4271017969 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fcee40611d0 con 0x7fcecc06c4e0 2024-04-05T21:23:35.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.760+0000 7fcee17fa700 1 -- 172.21.15.121:0/4271017969 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fcee40611d0 con 0x7fcecc06c4e0 2024-04-05T21:23:35.761 INFO:teuthology.orchestra.run.smithi121.stdout:HOST ADDR LABELS STATUS 2024-04-05T21:23:35.761 INFO:teuthology.orchestra.run.smithi121.stdout:smithi121 172.21.15.121 2024-04-05T21:23:35.761 INFO:teuthology.orchestra.run.smithi121.stdout:smithi188 172.21.15.188 2024-04-05T21:23:35.762 INFO:teuthology.orchestra.run.smithi121.stdout:2 hosts in cluster 2024-04-05T21:23:35.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.762+0000 7fceea578700 1 -- 172.21.15.121:0/4271017969 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fcecc06c4e0 msgr2=0x7fcecc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:35.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.762+0000 7fceea578700 1 --2- 172.21.15.121:0/4271017969 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fcecc06c4e0 0x7fcecc06e9a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fced4005fd0 tx=0x7fced4005f00 comp rx=0 tx=0).stop 2024-04-05T21:23:35.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.762+0000 7fceea578700 1 -- 172.21.15.121:0/4271017969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100440 msgr2=0x7fcee4196890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:35.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.762+0000 7fceea578700 1 --2- 172.21.15.121:0/4271017969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100440 0x7fcee4196890 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fced8006010 tx=0x7fced800be20 comp rx=0 tx=0).stop 2024-04-05T21:23:35.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.763+0000 7fceea578700 1 -- 172.21.15.121:0/4271017969 shutdown_connections 2024-04-05T21:23:35.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.763+0000 7fceea578700 1 --2- 172.21.15.121:0/4271017969 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fcecc06c4e0 0x7fcecc06e9a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:35.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.763+0000 7fceea578700 1 --2- 172.21.15.121:0/4271017969 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcee40ff240 0x7fcee4196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:35.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.763+0000 7fceea578700 1 --2- 172.21.15.121:0/4271017969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100440 0x7fcee4196890 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:35.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.763+0000 7fceea578700 1 -- 172.21.15.121:0/4271017969 >> 172.21.15.121:0/4271017969 conn(0x7fcee40fa7c0 msgr2=0x7fcee40fc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:35.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.763+0000 7fceea578700 1 -- 172.21.15.121:0/4271017969 shutdown_connections 2024-04-05T21:23:35.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:35.763+0000 7fceea578700 1 -- 172.21.15.121:0/4271017969 wait complete. 2024-04-05T21:23:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:36 smithi121 ceph-mon[39723]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:36.525 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-04-05T21:23:36.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:36 smithi188 ceph-mon[49715]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:36.751 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:37.319 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:37 smithi121 ceph-mon[39723]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:23:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:37 smithi188 ceph-mon[49715]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:23:38.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:38 smithi121 ceph-mon[39723]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:38.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:38 smithi188 ceph-mon[49715]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:38.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.775+0000 7f7548963700 1 -- 172.21.15.121:0/2779623918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7544101db0 msgr2=0x7f7544102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:38.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.775+0000 7f7548963700 1 --2- 172.21.15.121:0/2779623918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7544101db0 0x7f7544102210 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7534009910 tx=0x7f7534009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:38.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.775+0000 7f7548963700 1 -- 172.21.15.121:0/2779623918 shutdown_connections 2024-04-05T21:23:38.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.775+0000 7f7548963700 1 --2- 172.21.15.121:0/2779623918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7544101db0 0x7f7544102210 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:38.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.775+0000 7f7548963700 1 --2- 172.21.15.121:0/2779623918 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7544100bb0 0x7f7544100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:38.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.775+0000 7f7548963700 1 -- 172.21.15.121:0/2779623918 >> 172.21.15.121:0/2779623918 conn(0x7f75440fc110 msgr2=0x7f75440fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:38.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.775+0000 7f7548963700 1 -- 172.21.15.121:0/2779623918 shutdown_connections 2024-04-05T21:23:38.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.775+0000 7f7548963700 1 -- 172.21.15.121:0/2779623918 wait complete. 2024-04-05T21:23:38.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f7548963700 1 Processor -- start 2024-04-05T21:23:38.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f7548963700 1 -- start start 2024-04-05T21:23:38.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f7548963700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7544100bb0 0x7f75441962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:38.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f7548963700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7544101db0 0x7f75441967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:38.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f7548963700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7544196e00 con 0x7f7544100bb0 2024-04-05T21:23:38.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f7548963700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7544196f40 con 0x7f7544101db0 2024-04-05T21:23:38.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f754259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7544100bb0 0x7f75441962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:38.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f754259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7544100bb0 0x7f75441962a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35268/0 (socket says 172.21.15.121:35268) 2024-04-05T21:23:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f754259c700 1 -- 172.21.15.121:0/3881213315 learned_addr learned my addr 172.21.15.121:0/3881213315 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f7541d9b700 1 --2- 172.21.15.121:0/3881213315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7544101db0 0x7f75441967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f754259c700 1 -- 172.21.15.121:0/3881213315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7544101db0 msgr2=0x7f75441967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f754259c700 1 --2- 172.21.15.121:0/3881213315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7544101db0 0x7f75441967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f754259c700 1 -- 172.21.15.121:0/3881213315 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75340095f0 con 0x7f7544100bb0 2024-04-05T21:23:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f7541d9b700 1 --2- 172.21.15.121:0/3881213315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7544101db0 0x7f75441967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:23:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f754259c700 1 --2- 172.21.15.121:0/3881213315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7544100bb0 0x7f75441962a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f753800d900 tx=0x7f753800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f75337fe700 1 -- 172.21.15.121:0/3881213315 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75380041f0 con 0x7f7544100bb0 2024-04-05T21:23:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f75337fe700 1 -- 172.21.15.121:0/3881213315 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f7538004350 con 0x7f7544100bb0 2024-04-05T21:23:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f7548963700 1 -- 172.21.15.121:0/3881213315 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754419b9f0 con 0x7f7544100bb0 2024-04-05T21:23:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f75337fe700 1 -- 172.21.15.121:0/3881213315 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7538003cf0 con 0x7f7544100bb0 2024-04-05T21:23:38.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.776+0000 7f7548963700 1 -- 172.21.15.121:0/3881213315 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754419bfc0 con 0x7f7544100bb0 2024-04-05T21:23:38.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.777+0000 7f7548963700 1 -- 172.21.15.121:0/3881213315 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75441095d0 con 0x7f7544100bb0 2024-04-05T21:23:38.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.778+0000 7f75337fe700 1 -- 172.21.15.121:0/3881213315 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f7538009730 con 0x7f7544100bb0 2024-04-05T21:23:38.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.778+0000 7f75337fe700 1 --2- 172.21.15.121:0/3881213315 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f752c06c4e0 0x7f752c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:38.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.778+0000 7f7541d9b700 1 --2- 172.21.15.121:0/3881213315 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f752c06c4e0 0x7f752c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:38.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.778+0000 7f75337fe700 1 -- 172.21.15.121:0/3881213315 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f753808b5d0 con 0x7f7544100bb0 2024-04-05T21:23:38.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.779+0000 7f7541d9b700 1 --2- 172.21.15.121:0/3881213315 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f752c06c4e0 0x7f752c06e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7534005b40 tx=0x7f7534005a90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:38.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.783+0000 7f75337fe700 1 -- 172.21.15.121:0/3881213315 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f7538059c00 con 0x7f7544100bb0 2024-04-05T21:23:38.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.977+0000 7f7548963700 1 -- 172.21.15.121:0/3881213315 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7544106700 con 0x7f752c06c4e0 2024-04-05T21:23:38.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.978+0000 7f75337fe700 1 -- 172.21.15.121:0/3881213315 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f7544106700 con 0x7f752c06c4e0 2024-04-05T21:23:38.979 INFO:teuthology.orchestra.run.smithi121.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-05T21:23:38.979 INFO:teuthology.orchestra.run.smithi121.stdout:smithi121 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400ZF400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-05T21:23:38.979 INFO:teuthology.orchestra.run.smithi121.stdout:smithi188 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500MP400BGN 372G No 79s ago Has a FileSystem, LVM detected 2024-04-05T21:23:38.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.980+0000 7f7548963700 1 -- 172.21.15.121:0/3881213315 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f752c06c4e0 msgr2=0x7f752c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:38.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.980+0000 7f7548963700 1 --2- 172.21.15.121:0/3881213315 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f752c06c4e0 0x7f752c06e9a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7534005b40 tx=0x7f7534005a90 comp rx=0 tx=0).stop 2024-04-05T21:23:38.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.980+0000 7f7548963700 1 -- 172.21.15.121:0/3881213315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7544100bb0 msgr2=0x7f75441962a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:38.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.980+0000 7f7548963700 1 --2- 172.21.15.121:0/3881213315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7544100bb0 0x7f75441962a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f753800d900 tx=0x7f753800dcc0 comp rx=0 tx=0).stop 2024-04-05T21:23:38.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.980+0000 7f7548963700 1 -- 172.21.15.121:0/3881213315 shutdown_connections 2024-04-05T21:23:38.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.980+0000 7f7548963700 1 --2- 172.21.15.121:0/3881213315 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f752c06c4e0 0x7f752c06e9a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:38.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.980+0000 7f7548963700 1 --2- 172.21.15.121:0/3881213315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7544100bb0 0x7f75441962a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:38.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.980+0000 7f7548963700 1 --2- 172.21.15.121:0/3881213315 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7544101db0 0x7f75441967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:38.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.980+0000 7f7548963700 1 -- 172.21.15.121:0/3881213315 >> 172.21.15.121:0/3881213315 conn(0x7f75440fc110 msgr2=0x7f7544104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:38.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.980+0000 7f7548963700 1 -- 172.21.15.121:0/3881213315 shutdown_connections 2024-04-05T21:23:38.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:38.980+0000 7f7548963700 1 -- 172.21.15.121:0/3881213315 wait complete. 2024-04-05T21:23:39.748 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T21:23:39.759 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T21:23:39.759 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-05T21:23:39.984 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:40.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:40 smithi121 ceph-mon[39723]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:23:40.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:40 smithi121 ceph-mon[39723]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:40.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:23:40.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:40 smithi188 ceph-mon[49715]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:23:40.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:40 smithi188 ceph-mon[49715]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:40.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:40 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:23:41.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.925+0000 7f2d1edb2700 1 -- 172.21.15.121:0/1306320265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d18100cb0 msgr2=0x7f2d18101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:41.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.925+0000 7f2d1edb2700 1 --2- 172.21.15.121:0/1306320265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d18100cb0 0x7f2d18101110 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2d04009910 tx=0x7f2d04009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:41.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.925+0000 7f2d1edb2700 1 -- 172.21.15.121:0/1306320265 shutdown_connections 2024-04-05T21:23:41.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.925+0000 7f2d1edb2700 1 --2- 172.21.15.121:0/1306320265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d18100cb0 0x7f2d18101110 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:41.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.925+0000 7f2d1edb2700 1 --2- 172.21.15.121:0/1306320265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d180ffab0 0x7f2d180ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:41.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.925+0000 7f2d1edb2700 1 -- 172.21.15.121:0/1306320265 >> 172.21.15.121:0/1306320265 conn(0x7f2d180fb050 msgr2=0x7f2d180fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:41.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.925+0000 7f2d1edb2700 1 -- 172.21.15.121:0/1306320265 shutdown_connections 2024-04-05T21:23:41.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.925+0000 7f2d1edb2700 1 -- 172.21.15.121:0/1306320265 wait complete. 2024-04-05T21:23:41.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d1edb2700 1 Processor -- start 2024-04-05T21:23:41.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d1edb2700 1 -- start start 2024-04-05T21:23:41.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d1edb2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d180ffab0 0x7f2d18196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:41.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d1edb2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d18100cb0 0x7f2d18196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:41.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d1edb2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d18196db0 con 0x7f2d18100cb0 2024-04-05T21:23:41.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d1edb2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d18196ef0 con 0x7f2d180ffab0 2024-04-05T21:23:41.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d17fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d18100cb0 0x7f2d18196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:41.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d17fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d18100cb0 0x7f2d18196790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35274/0 (socket says 172.21.15.121:35274) 2024-04-05T21:23:41.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d17fff700 1 -- 172.21.15.121:0/459594892 learned_addr learned my addr 172.21.15.121:0/459594892 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:41.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d1cb4e700 1 --2- 172.21.15.121:0/459594892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d180ffab0 0x7f2d18196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:41.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d17fff700 1 -- 172.21.15.121:0/459594892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d180ffab0 msgr2=0x7f2d18196250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:41.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d17fff700 1 --2- 172.21.15.121:0/459594892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d180ffab0 0x7f2d18196250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:41.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d17fff700 1 -- 172.21.15.121:0/459594892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d040095f0 con 0x7f2d18100cb0 2024-04-05T21:23:41.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.926+0000 7f2d17fff700 1 --2- 172.21.15.121:0/459594892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d18100cb0 0x7f2d18196790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400b5c0 tx=0x7f2d0400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:41.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.927+0000 7f2d15ffb700 1 -- 172.21.15.121:0/459594892 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d0401d070 con 0x7f2d18100cb0 2024-04-05T21:23:41.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.927+0000 7f2d1edb2700 1 -- 172.21.15.121:0/459594892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1819b940 con 0x7f2d18100cb0 2024-04-05T21:23:41.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.927+0000 7f2d15ffb700 1 -- 172.21.15.121:0/459594892 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f2d0400f460 con 0x7f2d18100cb0 2024-04-05T21:23:41.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.927+0000 7f2d15ffb700 1 -- 172.21.15.121:0/459594892 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d04021640 con 0x7f2d18100cb0 2024-04-05T21:23:41.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.927+0000 7f2d1edb2700 1 -- 172.21.15.121:0/459594892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1819beb0 con 0x7f2d18100cb0 2024-04-05T21:23:41.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.928+0000 7f2d1edb2700 1 -- 172.21.15.121:0/459594892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d181084d0 con 0x7f2d18100cb0 2024-04-05T21:23:41.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.928+0000 7f2d15ffb700 1 -- 172.21.15.121:0/459594892 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f2d0400fa20 con 0x7f2d18100cb0 2024-04-05T21:23:41.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.929+0000 7f2d15ffb700 1 --2- 172.21.15.121:0/459594892 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2d0806c4e0 0x7f2d0806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:41.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.929+0000 7f2d15ffb700 1 -- 172.21.15.121:0/459594892 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f2d0408c780 con 0x7f2d18100cb0 2024-04-05T21:23:41.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.929+0000 7f2d1cb4e700 1 --2- 172.21.15.121:0/459594892 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2d0806c4e0 0x7f2d0806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:41.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.929+0000 7f2d1cb4e700 1 --2- 172.21.15.121:0/459594892 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2d0806c4e0 0x7f2d0806e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2d18100b10 tx=0x7f2d0c00b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:41.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:41.933+0000 7f2d15ffb700 1 -- 172.21.15.121:0/459594892 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f2d0405ad30 con 0x7f2d18100cb0 2024-04-05T21:23:42.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:42.152+0000 7f2d1edb2700 1 -- 172.21.15.121:0/459594892 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d180693b0 con 0x7f2d0806c4e0 2024-04-05T21:23:42.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:42 smithi121 ceph-mon[39723]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:42.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:42 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-05T21:23:42.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:42 smithi188 ceph-mon[49715]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:42.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:42 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-05T21:23:43.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:43.465+0000 7f2d15ffb700 1 -- 172.21.15.121:0/459594892 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2d180693b0 con 0x7f2d0806c4e0 2024-04-05T21:23:43.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:43.467+0000 7f2d1edb2700 1 -- 172.21.15.121:0/459594892 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2d0806c4e0 msgr2=0x7f2d0806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:43.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:43.467+0000 7f2d1edb2700 1 --2- 172.21.15.121:0/459594892 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2d0806c4e0 0x7f2d0806e9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2d18100b10 tx=0x7f2d0c00b410 comp rx=0 tx=0).stop 2024-04-05T21:23:43.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:43.467+0000 7f2d1edb2700 1 -- 172.21.15.121:0/459594892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d18100cb0 msgr2=0x7f2d18196790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:43.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:43.467+0000 7f2d1edb2700 1 --2- 172.21.15.121:0/459594892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d18100cb0 0x7f2d18196790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400b5c0 tx=0x7f2d0400bd70 comp rx=0 tx=0).stop 2024-04-05T21:23:43.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:43.468+0000 7f2d1edb2700 1 -- 172.21.15.121:0/459594892 shutdown_connections 2024-04-05T21:23:43.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:43.468+0000 7f2d1edb2700 1 --2- 172.21.15.121:0/459594892 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f2d0806c4e0 0x7f2d0806e9a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:43.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:43.468+0000 7f2d1edb2700 1 --2- 172.21.15.121:0/459594892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d180ffab0 0x7f2d18196250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:43.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:43.468+0000 7f2d1edb2700 1 --2- 172.21.15.121:0/459594892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d18100cb0 0x7f2d18196790 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:43.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:43.468+0000 7f2d1edb2700 1 -- 172.21.15.121:0/459594892 >> 172.21.15.121:0/459594892 conn(0x7f2d180fb050 msgr2=0x7f2d18068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:43.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:43.468+0000 7f2d1edb2700 1 -- 172.21.15.121:0/459594892 shutdown_connections 2024-04-05T21:23:43.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:43.468+0000 7f2d1edb2700 1 -- 172.21.15.121:0/459594892 wait complete. 2024-04-05T21:23:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:43 smithi121 ceph-mon[39723]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:23:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-05T21:23:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:43 smithi121 ceph-mon[39723]: osdmap e41: 6 total, 6 up, 6 in 2024-04-05T21:23:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:43 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-05T21:23:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:43 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[39700]: 2024-04-05T21:23:43.437+0000 7f9952833700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T21:23:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:43 smithi188 ceph-mon[49715]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:23:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-05T21:23:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:43 smithi188 ceph-mon[49715]: osdmap e41: 6 total, 6 up, 6 in 2024-04-05T21:23:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:43 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-05T21:23:44.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-05T21:23:44.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: pgmap v188: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:44.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-05T21:23:44.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-05T21:23:44.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: osdmap e42: 6 total, 6 up, 6 in 2024-04-05T21:23:44.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-05T21:23:44.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T21:23:44.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-05T21:23:44.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: osdmap e43: 6 total, 6 up, 6 in 2024-04-05T21:23:44.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-05T21:23:44.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: fsmap cephfs:0 2024-04-05T21:23:44.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:44.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:23:44.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:44.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:23:44.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:44.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi121.gxheqt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:23:44.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi121.gxheqt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T21:23:44.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:44 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:44.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: pgmap v188: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:44.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-05T21:23:44.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-05T21:23:44.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: osdmap e42: 6 total, 6 up, 6 in 2024-04-05T21:23:44.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-05T21:23:44.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T21:23:44.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-05T21:23:44.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: osdmap e43: 6 total, 6 up, 6 in 2024-04-05T21:23:44.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-05T21:23:44.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: fsmap cephfs:0 2024-04-05T21:23:44.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:44.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:23:44.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:44.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:23:44.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:44.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi121.gxheqt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:23:44.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi121.gxheqt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T21:23:44.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:44 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:45.130 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:45 smithi121 ceph-mon[39723]: Saving service mds.cephfs spec with placement count:4 2024-04-05T21:23:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:45 smithi121 ceph-mon[39723]: Deploying daemon mds.cephfs.smithi121.gxheqt on smithi121 2024-04-05T21:23:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:45 smithi121 ceph-mon[39723]: osdmap e44: 6 total, 6 up, 6 in 2024-04-05T21:23:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:45 smithi188 ceph-mon[49715]: Saving service mds.cephfs spec with placement count:4 2024-04-05T21:23:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:45 smithi188 ceph-mon[49715]: Deploying daemon mds.cephfs.smithi121.gxheqt on smithi121 2024-04-05T21:23:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:45 smithi188 ceph-mon[49715]: osdmap e44: 6 total, 6 up, 6 in 2024-04-05T21:23:46.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:46 smithi121 ceph-mon[39723]: pgmap v192: 65 pgs: 14 active+clean, 2 creating+activating, 16 creating+peering, 33 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:46.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:46 smithi121 ceph-mon[39723]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-05T21:23:46.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:46 smithi121 ceph-mon[39723]: osdmap e45: 6 total, 6 up, 6 in 2024-04-05T21:23:46.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:46 smithi188 ceph-mon[49715]: pgmap v192: 65 pgs: 14 active+clean, 2 creating+activating, 16 creating+peering, 33 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:46.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:46 smithi188 ceph-mon[49715]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-05T21:23:46.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:46 smithi188 ceph-mon[49715]: osdmap e45: 6 total, 6 up, 6 in 2024-04-05T21:23:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.275+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/3773647432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ba4101db0 msgr2=0x7f9ba4102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.275+0000 7f9ba8c1e700 1 --2- 172.21.15.121:0/3773647432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ba4101db0 0x7f9ba4102210 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009910 tx=0x7f9b8c009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.275+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/3773647432 shutdown_connections 2024-04-05T21:23:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.275+0000 7f9ba8c1e700 1 --2- 172.21.15.121:0/3773647432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ba4101db0 0x7f9ba4102210 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:47.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.275+0000 7f9ba8c1e700 1 --2- 172.21.15.121:0/3773647432 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ba4100bb0 0x7f9ba4100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.275+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/3773647432 >> 172.21.15.121:0/3773647432 conn(0x7f9ba40fc110 msgr2=0x7f9ba40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.275+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/3773647432 shutdown_connections 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.275+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/3773647432 wait complete. 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.276+0000 7f9ba8c1e700 1 Processor -- start 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.276+0000 7f9ba8c1e700 1 -- start start 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.276+0000 7f9ba8c1e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ba4100bb0 0x7f9ba4074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.276+0000 7f9ba8c1e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ba4101db0 0x7f9ba4073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.276+0000 7f9ba8c1e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba40736c0 con 0x7f9ba4100bb0 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.276+0000 7f9ba8c1e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba4073800 con 0x7f9ba4101db0 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.276+0000 7f9ba259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ba4100bb0 0x7f9ba4074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.277+0000 7f9ba259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ba4100bb0 0x7f9ba4074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44136/0 (socket says 172.21.15.121:44136) 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.277+0000 7f9ba259c700 1 -- 172.21.15.121:0/848602211 learned_addr learned my addr 172.21.15.121:0/848602211 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.277+0000 7f9ba1d9b700 1 --2- 172.21.15.121:0/848602211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ba4101db0 0x7f9ba4073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:47.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.277+0000 7f9ba259c700 1 -- 172.21.15.121:0/848602211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ba4101db0 msgr2=0x7f9ba4073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.277+0000 7f9ba259c700 1 --2- 172.21.15.121:0/848602211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ba4101db0 0x7f9ba4073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.277+0000 7f9ba259c700 1 -- 172.21.15.121:0/848602211 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c0095f0 con 0x7f9ba4100bb0 2024-04-05T21:23:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.277+0000 7f9ba259c700 1 --2- 172.21.15.121:0/848602211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ba4100bb0 0x7f9ba4074b30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400b480 tx=0x7f9b9400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.277+0000 7f9b9b7fe700 1 -- 172.21.15.121:0/848602211 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b940118c0 con 0x7f9ba4100bb0 2024-04-05T21:23:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.277+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/848602211 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba4073ae0 con 0x7f9ba4100bb0 2024-04-05T21:23:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.277+0000 7f9b9b7fe700 1 -- 172.21.15.121:0/848602211 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f9b94010380 con 0x7f9ba4100bb0 2024-04-05T21:23:47.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.277+0000 7f9b9b7fe700 1 -- 172.21.15.121:0/848602211 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b9400f510 con 0x7f9ba4100bb0 2024-04-05T21:23:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.277+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/848602211 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba419e3e0 con 0x7f9ba4100bb0 2024-04-05T21:23:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.278+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/848602211 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba4109670 con 0x7f9ba4100bb0 2024-04-05T21:23:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.279+0000 7f9b9b7fe700 1 -- 172.21.15.121:0/848602211 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f9b940104f0 con 0x7f9ba4100bb0 2024-04-05T21:23:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.279+0000 7f9b9b7fe700 1 --2- 172.21.15.121:0/848602211 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f9b9006c4e0 0x7f9b9006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.279+0000 7f9b9b7fe700 1 -- 172.21.15.121:0/848602211 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f9b9408b910 con 0x7f9ba4100bb0 2024-04-05T21:23:47.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.279+0000 7f9ba1d9b700 1 --2- 172.21.15.121:0/848602211 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f9b9006c4e0 0x7f9b9006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:47.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.280+0000 7f9ba1d9b700 1 --2- 172.21.15.121:0/848602211 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f9b9006c4e0 0x7f9b9006e9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0059d0 tx=0x7f9b8c005920 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:47.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.283+0000 7f9b9b7fe700 1 -- 172.21.15.121:0/848602211 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f9b94059b60 con 0x7f9ba4100bb0 2024-04-05T21:23:47.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.518+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/848602211 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9ba4109860 con 0x7f9ba4100bb0 2024-04-05T21:23:47.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.518+0000 7f9b9b7fe700 1 -- 172.21.15.121:0/848602211 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1093 (secure 0 0 0) 0x7f9b94015460 con 0x7f9ba4100bb0 2024-04-05T21:23:47.520 INFO:teuthology.orchestra.run.smithi121.stdout:e2 2024-04-05T21:23:47.520 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:23:47.520 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:23:47.520 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:23:47.520 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:23:47.520 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:23:47.520 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:23:47.520 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 2 2024-04-05T21:23:47.520 INFO:teuthology.orchestra.run.smithi121.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-05T21:23:47.520 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:23:47.520 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:23:43.437942+0000 2024-04-05T21:23:47.520 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:23:47.521 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:23:47.521 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:23:47.521 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:23:47.521 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:23:47.521 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:23:47.521 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:23:47.521 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 0 2024-04-05T21:23:47.521 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:23:47.521 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 1 2024-04-05T21:23:47.521 INFO:teuthology.orchestra.run.smithi121.stdout:in 2024-04-05T21:23:47.521 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:23:47.522 INFO:teuthology.orchestra.run.smithi121.stdout:failed 2024-04-05T21:23:47.522 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:23:47.522 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:23:47.522 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:23:47.522 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:23:47.522 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:23:47.522 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:23:47.522 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:23:47.522 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 0 2024-04-05T21:23:47.523 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:23:47.523 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:23:47.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.520+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/848602211 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f9b9006c4e0 msgr2=0x7f9b9006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:47.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.520+0000 7f9ba8c1e700 1 --2- 172.21.15.121:0/848602211 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f9b9006c4e0 0x7f9b9006e9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0059d0 tx=0x7f9b8c005920 comp rx=0 tx=0).stop 2024-04-05T21:23:47.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.520+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/848602211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ba4100bb0 msgr2=0x7f9ba4074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:47.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.520+0000 7f9ba8c1e700 1 --2- 172.21.15.121:0/848602211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ba4100bb0 0x7f9ba4074b30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400b480 tx=0x7f9b9400b840 comp rx=0 tx=0).stop 2024-04-05T21:23:47.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.520+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/848602211 shutdown_connections 2024-04-05T21:23:47.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.520+0000 7f9ba8c1e700 1 --2- 172.21.15.121:0/848602211 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f9b9006c4e0 0x7f9b9006e9a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:47.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.520+0000 7f9ba8c1e700 1 --2- 172.21.15.121:0/848602211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ba4100bb0 0x7f9ba4074b30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:47.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.520+0000 7f9ba8c1e700 1 --2- 172.21.15.121:0/848602211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ba4101db0 0x7f9ba4073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:47.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.520+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/848602211 >> 172.21.15.121:0/848602211 conn(0x7f9ba40fc110 msgr2=0x7f9ba4104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:47.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.520+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/848602211 shutdown_connections 2024-04-05T21:23:47.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:47.520+0000 7f9ba8c1e700 1 -- 172.21.15.121:0/848602211 wait complete. 2024-04-05T21:23:47.524 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 2 2024-04-05T21:23:48.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:48 smithi121 ceph-mon[39723]: pgmap v194: 65 pgs: 35 active+clean, 2 creating+activating, 16 creating+peering, 12 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:48.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:48 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/848602211' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:23:48.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:48 smithi188 ceph-mon[49715]: pgmap v194: 65 pgs: 35 active+clean, 2 creating+activating, 16 creating+peering, 12 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-05T21:23:48.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:48 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/848602211' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:23:48.879 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T21:23:48.893 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T21:23:48.894 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-04-05T21:23:49.122 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:49.714 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.121:6826/1615448980,v1:172.21.15.121:6827/1615448980] up:boot 2024-04-05T21:23:49.714 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: daemon mds.cephfs.smithi121.gxheqt assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-05T21:23:49.714 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-05T21:23:49.714 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-05T21:23:49.715 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: Cluster is now healthy 2024-04-05T21:23:49.715 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: fsmap cephfs:0 1 up:standby 2024-04-05T21:23:49.715 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: fsmap cephfs:1 {0=cephfs.smithi121.gxheqt=up:creating} 2024-04-05T21:23:49.715 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:23:49.715 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: daemon mds.cephfs.smithi121.gxheqt is now active in filesystem cephfs as rank 0 2024-04-05T21:23:49.715 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:49.715 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:49.715 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:49.715 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi188.bjmisu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:23:49.715 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi188.bjmisu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T21:23:49.715 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:49 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:49.741 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.121:6826/1615448980,v1:172.21.15.121:6827/1615448980] up:boot 2024-04-05T21:23:49.741 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: daemon mds.cephfs.smithi121.gxheqt assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-05T21:23:49.741 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-05T21:23:49.742 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-05T21:23:49.742 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: Cluster is now healthy 2024-04-05T21:23:49.742 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: fsmap cephfs:0 1 up:standby 2024-04-05T21:23:49.742 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: fsmap cephfs:1 {0=cephfs.smithi121.gxheqt=up:creating} 2024-04-05T21:23:49.742 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:23:49.742 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: daemon mds.cephfs.smithi121.gxheqt is now active in filesystem cephfs as rank 0 2024-04-05T21:23:49.742 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:49.742 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:49.742 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:49.742 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi188.bjmisu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:23:49.742 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi188.bjmisu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T21:23:49.742 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:49 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:50.510 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:50 smithi188 ceph-mon[49715]: Deploying daemon mds.cephfs.smithi188.bjmisu on smithi188 2024-04-05T21:23:50.510 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:50 smithi188 ceph-mon[49715]: pgmap v195: 65 pgs: 59 active+clean, 6 creating+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 183 B/s wr, 0 op/s 2024-04-05T21:23:50.511 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:50 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.121:6826/1615448980,v1:172.21.15.121:6827/1615448980] up:active 2024-04-05T21:23:50.511 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:50 smithi188 ceph-mon[49715]: fsmap cephfs:1 {0=cephfs.smithi121.gxheqt=up:active} 2024-04-05T21:23:50.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:50 smithi121 ceph-mon[39723]: Deploying daemon mds.cephfs.smithi188.bjmisu on smithi188 2024-04-05T21:23:50.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:50 smithi121 ceph-mon[39723]: pgmap v195: 65 pgs: 59 active+clean, 6 creating+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 183 B/s wr, 0 op/s 2024-04-05T21:23:50.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:50 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.121:6826/1615448980,v1:172.21.15.121:6827/1615448980] up:active 2024-04-05T21:23:50.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:50 smithi121 ceph-mon[39723]: fsmap cephfs:1 {0=cephfs.smithi121.gxheqt=up:active} 2024-04-05T21:23:51.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.080+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/2373933345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe0100cb0 msgr2=0x7fbfe0101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:51.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.080+0000 7fbfe4e9c700 1 --2- 172.21.15.121:0/2373933345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe0100cb0 0x7fbfe0101110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009910 tx=0x7fbfd0009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:51.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.080+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/2373933345 shutdown_connections 2024-04-05T21:23:51.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.080+0000 7fbfe4e9c700 1 --2- 172.21.15.121:0/2373933345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe0100cb0 0x7fbfe0101110 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:51.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.080+0000 7fbfe4e9c700 1 --2- 172.21.15.121:0/2373933345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfe00ffab0 0x7fbfe00ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:51.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.080+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/2373933345 >> 172.21.15.121:0/2373933345 conn(0x7fbfe00fb030 msgr2=0x7fbfe00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:51.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.080+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/2373933345 shutdown_connections 2024-04-05T21:23:51.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.080+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/2373933345 wait complete. 2024-04-05T21:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfe4e9c700 1 Processor -- start 2024-04-05T21:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfe4e9c700 1 -- start start 2024-04-05T21:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfe4e9c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe00ffab0 0x7fbfe0068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfe4e9c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfe0100cb0 0x7fbfe019afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfe4e9c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe019b580 con 0x7fbfe00ffab0 2024-04-05T21:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfe4e9c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe019b6c0 con 0x7fbfe0100cb0 2024-04-05T21:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfde59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe00ffab0 0x7fbfe0068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfde59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe00ffab0 0x7fbfe0068f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44188/0 (socket says 172.21.15.121:44188) 2024-04-05T21:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfde59c700 1 -- 172.21.15.121:0/3616745017 learned_addr learned my addr 172.21.15.121:0/3616745017 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfddd9b700 1 --2- 172.21.15.121:0/3616745017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfe0100cb0 0x7fbfe019afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfde59c700 1 -- 172.21.15.121:0/3616745017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfe0100cb0 msgr2=0x7fbfe019afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfde59c700 1 --2- 172.21.15.121:0/3616745017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfe0100cb0 0x7fbfe019afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:51.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfde59c700 1 -- 172.21.15.121:0/3616745017 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd00095f0 con 0x7fbfe00ffab0 2024-04-05T21:23:51.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.081+0000 7fbfde59c700 1 --2- 172.21.15.121:0/3616745017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe00ffab0 0x7fbfe0068f70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400d900 tx=0x7fbfd400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:51.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.082+0000 7fbfcf7fe700 1 -- 172.21.15.121:0/3616745017 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd40041f0 con 0x7fbfe00ffab0 2024-04-05T21:23:51.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.082+0000 7fbfcf7fe700 1 -- 172.21.15.121:0/3616745017 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fbfd4004350 con 0x7fbfe00ffab0 2024-04-05T21:23:51.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.082+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/3616745017 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe019b9a0 con 0x7fbfe00ffab0 2024-04-05T21:23:51.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.082+0000 7fbfcf7fe700 1 -- 172.21.15.121:0/3616745017 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd4003cb0 con 0x7fbfe00ffab0 2024-04-05T21:23:51.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.082+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/3616745017 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe019bf70 con 0x7fbfe00ffab0 2024-04-05T21:23:51.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.082+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/3616745017 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe01084d0 con 0x7fbfe00ffab0 2024-04-05T21:23:51.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.083+0000 7fbfcf7fe700 1 -- 172.21.15.121:0/3616745017 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fbfd4009730 con 0x7fbfe00ffab0 2024-04-05T21:23:51.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.084+0000 7fbfcf7fe700 1 --2- 172.21.15.121:0/3616745017 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fbfc806c4e0 0x7fbfc806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:51.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.084+0000 7fbfddd9b700 1 --2- 172.21.15.121:0/3616745017 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fbfc806c4e0 0x7fbfc806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:51.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.084+0000 7fbfcf7fe700 1 -- 172.21.15.121:0/3616745017 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fbfd408bae0 con 0x7fbfe00ffab0 2024-04-05T21:23:51.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.084+0000 7fbfddd9b700 1 --2- 172.21.15.121:0/3616745017 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fbfc806c4e0 0x7fbfc806e9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbfe00694f0 tx=0x7fbfd000b540 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:51.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.088+0000 7fbfcf7fe700 1 -- 172.21.15.121:0/3616745017 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fbfd4059db0 con 0x7fbfe00ffab0 2024-04-05T21:23:51.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.327+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/3616745017 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fbfe019c670 con 0x7fbfe00ffab0 2024-04-05T21:23:51.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.406+0000 7fbfcf7fe700 1 -- 172.21.15.121:0/3616745017 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7fbfd4059940 con 0x7fbfe00ffab0 2024-04-05T21:23:51.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.409+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/3616745017 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fbfc806c4e0 msgr2=0x7fbfc806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:51.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.409+0000 7fbfe4e9c700 1 --2- 172.21.15.121:0/3616745017 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fbfc806c4e0 0x7fbfc806e9a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbfe00694f0 tx=0x7fbfd000b540 comp rx=0 tx=0).stop 2024-04-05T21:23:51.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.409+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/3616745017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe00ffab0 msgr2=0x7fbfe0068f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:51.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.409+0000 7fbfe4e9c700 1 --2- 172.21.15.121:0/3616745017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe00ffab0 0x7fbfe0068f70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400d900 tx=0x7fbfd400dc10 comp rx=0 tx=0).stop 2024-04-05T21:23:51.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.409+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/3616745017 shutdown_connections 2024-04-05T21:23:51.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.409+0000 7fbfe4e9c700 1 --2- 172.21.15.121:0/3616745017 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fbfc806c4e0 0x7fbfc806e9a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:51.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.409+0000 7fbfe4e9c700 1 --2- 172.21.15.121:0/3616745017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe00ffab0 0x7fbfe0068f70 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:51.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.409+0000 7fbfe4e9c700 1 --2- 172.21.15.121:0/3616745017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfe0100cb0 0x7fbfe019afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:51.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.409+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/3616745017 >> 172.21.15.121:0/3616745017 conn(0x7fbfe00fb030 msgr2=0x7fbfe0103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:51.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.409+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/3616745017 shutdown_connections 2024-04-05T21:23:51.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:51.409+0000 7fbfe4e9c700 1 -- 172.21.15.121:0/3616745017 wait complete. 2024-04-05T21:23:52.184 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T21:23:52.194 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T21:23:52.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-04-05T21:23:52.436 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: pgmap v196: 65 pgs: 65 active+clean; 578 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 541 B/s wr, 2 op/s 2024-04-05T21:23:52.436 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:52.436 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:52.436 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3616745017' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-05T21:23:52.437 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:52.437 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi121.ifoxmw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:23:52.437 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi121.ifoxmw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T21:23:52.437 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:52.437 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: Deploying daemon mds.cephfs.smithi121.ifoxmw on smithi121 2024-04-05T21:23:52.437 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-05T21:23:52.437 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] up:boot 2024-04-05T21:23:52.437 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3616745017' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-05T21:23:52.437 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: daemon mds.cephfs.smithi188.bjmisu assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-05T21:23:52.437 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-05T21:23:52.438 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-05T21:23:52.438 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: fsmap cephfs:1 {0=cephfs.smithi121.gxheqt=up:active} 1 up:standby 2024-04-05T21:23:52.438 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T21:23:52.438 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi188.bjmisu=up:creating} 2024-04-05T21:23:52.438 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:52 smithi121 ceph-mon[39723]: daemon mds.cephfs.smithi188.bjmisu is now active in filesystem cephfs as rank 1 2024-04-05T21:23:52.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: pgmap v196: 65 pgs: 65 active+clean; 578 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 541 B/s wr, 2 op/s 2024-04-05T21:23:52.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:52.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:52.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/3616745017' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-04-05T21:23:52.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:52.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi121.ifoxmw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:23:52.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi121.ifoxmw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T21:23:52.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:52.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: Deploying daemon mds.cephfs.smithi121.ifoxmw on smithi121 2024-04-05T21:23:52.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-05T21:23:52.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] up:boot 2024-04-05T21:23:52.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/3616745017' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-04-05T21:23:52.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: daemon mds.cephfs.smithi188.bjmisu assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-05T21:23:52.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-05T21:23:52.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-05T21:23:52.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: fsmap cephfs:1 {0=cephfs.smithi121.gxheqt=up:active} 1 up:standby 2024-04-05T21:23:52.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T21:23:52.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi188.bjmisu=up:creating} 2024-04-05T21:23:52.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:52 smithi188 ceph-mon[49715]: daemon mds.cephfs.smithi188.bjmisu is now active in filesystem cephfs as rank 1 2024-04-05T21:23:53.028 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:53.632 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:53 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] up:active 2024-04-05T21:23:53.633 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:53 smithi121 ceph-mon[39723]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi188.bjmisu=up:active} 2024-04-05T21:23:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:53 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] up:active 2024-04-05T21:23:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:53 smithi188 ceph-mon[49715]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi188.bjmisu=up:active} 2024-04-05T21:23:54.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:54 smithi121 ceph-mon[39723]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s wr, 4 op/s 2024-04-05T21:23:54.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:54 smithi188 ceph-mon[49715]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s wr, 4 op/s 2024-04-05T21:23:55.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.587+0000 7f5de6dc8700 1 -- 172.21.15.121:0/2908820899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5de0100cb0 msgr2=0x7f5de0101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:55.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.587+0000 7f5de6dc8700 1 --2- 172.21.15.121:0/2908820899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5de0100cb0 0x7f5de0101110 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc009910 tx=0x7f5dcc009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:55.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.587+0000 7f5de6dc8700 1 -- 172.21.15.121:0/2908820899 shutdown_connections 2024-04-05T21:23:55.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.587+0000 7f5de6dc8700 1 --2- 172.21.15.121:0/2908820899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5de0100cb0 0x7f5de0101110 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.587+0000 7f5de6dc8700 1 --2- 172.21.15.121:0/2908820899 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5de00ffab0 0x7f5de00ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.587+0000 7f5de6dc8700 1 -- 172.21.15.121:0/2908820899 >> 172.21.15.121:0/2908820899 conn(0x7f5de00fb050 msgr2=0x7f5de00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.587+0000 7f5de6dc8700 1 -- 172.21.15.121:0/2908820899 shutdown_connections 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.587+0000 7f5de6dc8700 1 -- 172.21.15.121:0/2908820899 wait complete. 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.587+0000 7f5de6dc8700 1 Processor -- start 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.587+0000 7f5de6dc8700 1 -- start start 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.587+0000 7f5de6dc8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5de00ffab0 0x7f5de0103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5de6dc8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5de0100cb0 0x7f5de0102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5de6dc8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de01025d0 con 0x7f5de00ffab0 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5de6dc8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de0102710 con 0x7f5de0100cb0 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5de4b64700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5de00ffab0 0x7f5de0103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5de4b64700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5de00ffab0 0x7f5de0103a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35514/0 (socket says 172.21.15.121:35514) 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5de4b64700 1 -- 172.21.15.121:0/1184763791 learned_addr learned my addr 172.21.15.121:0/1184763791 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5ddffff700 1 --2- 172.21.15.121:0/1184763791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5de0100cb0 0x7f5de0102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:55.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5de4b64700 1 -- 172.21.15.121:0/1184763791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5de0100cb0 msgr2=0x7f5de0102090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:55.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5de4b64700 1 --2- 172.21.15.121:0/1184763791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5de0100cb0 0x7f5de0102090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:55.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5de4b64700 1 -- 172.21.15.121:0/1184763791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc0095f0 con 0x7f5de00ffab0 2024-04-05T21:23:55.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5ddffff700 1 --2- 172.21.15.121:0/1184763791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5de0100cb0 0x7f5de0102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:23:55.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5de4b64700 1 --2- 172.21.15.121:0/1184763791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5de00ffab0 0x7f5de0103a40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400e8f0 tx=0x7f5dd400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:55.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5dddffb700 1 -- 172.21.15.121:0/1184763791 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dd4009790 con 0x7f5de00ffab0 2024-04-05T21:23:55.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5dddffb700 1 -- 172.21.15.121:0/1184763791 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f5dd40044b0 con 0x7f5de00ffab0 2024-04-05T21:23:55.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5de6dc8700 1 -- 172.21.15.121:0/1184763791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de01029f0 con 0x7f5de00ffab0 2024-04-05T21:23:55.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5dddffb700 1 -- 172.21.15.121:0/1184763791 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dd4003d30 con 0x7f5de00ffab0 2024-04-05T21:23:55.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.588+0000 7f5de6dc8700 1 -- 172.21.15.121:0/1184763791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de019e2b0 con 0x7f5de00ffab0 2024-04-05T21:23:55.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.589+0000 7f5de6dc8700 1 -- 172.21.15.121:0/1184763791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de01084d0 con 0x7f5de00ffab0 2024-04-05T21:23:55.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.590+0000 7f5dddffb700 1 -- 172.21.15.121:0/1184763791 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f5dd4003fe0 con 0x7f5de00ffab0 2024-04-05T21:23:55.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.590+0000 7f5dddffb700 1 --2- 172.21.15.121:0/1184763791 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f5dd006c4e0 0x7f5dd006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:55.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.590+0000 7f5dddffb700 1 -- 172.21.15.121:0/1184763791 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f5dd4014070 con 0x7f5de00ffab0 2024-04-05T21:23:55.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.590+0000 7f5ddffff700 1 --2- 172.21.15.121:0/1184763791 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f5dd006c4e0 0x7f5dd006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:55.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.590+0000 7f5ddffff700 1 --2- 172.21.15.121:0/1184763791 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f5dd006c4e0 0x7f5dd006e9a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00b600 tx=0x7f5dcc005fb0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:55.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.594+0000 7f5dddffb700 1 -- 172.21.15.121:0/1184763791 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5dd4059db0 con 0x7f5de00ffab0 2024-04-05T21:23:55.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:55.838+0000 7f5de6dc8700 1 -- 172.21.15.121:0/1184763791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f5de019e860 con 0x7f5de00ffab0 2024-04-05T21:23:56.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:56 smithi188 ceph-mon[49715]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s wr, 7 op/s 2024-04-05T21:23:56.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:56 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:56.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:56 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:23:56.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:56 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:56.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:56 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1184763791' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-05T21:23:56.291 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:56 smithi121 ceph-mon[39723]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s wr, 7 op/s 2024-04-05T21:23:56.291 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:56 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:56.291 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:56 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:23:56.291 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:56 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:56.291 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:56 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1184763791' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-05T21:23:56.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:56.419+0000 7f5dddffb700 1 -- 172.21.15.121:0/1184763791 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v9) v1 ==== 121+0+0 (secure 0 0 0) 0x7f5dd4059940 con 0x7f5de00ffab0 2024-04-05T21:23:56.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:56.422+0000 7f5de6dc8700 1 -- 172.21.15.121:0/1184763791 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f5dd006c4e0 msgr2=0x7f5dd006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:56.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:56.422+0000 7f5de6dc8700 1 --2- 172.21.15.121:0/1184763791 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f5dd006c4e0 0x7f5dd006e9a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00b600 tx=0x7f5dcc005fb0 comp rx=0 tx=0).stop 2024-04-05T21:23:56.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:56.422+0000 7f5de6dc8700 1 -- 172.21.15.121:0/1184763791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5de00ffab0 msgr2=0x7f5de0103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:56.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:56.422+0000 7f5de6dc8700 1 --2- 172.21.15.121:0/1184763791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5de00ffab0 0x7f5de0103a40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400e8f0 tx=0x7f5dd400ecb0 comp rx=0 tx=0).stop 2024-04-05T21:23:56.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:56.423+0000 7f5de6dc8700 1 -- 172.21.15.121:0/1184763791 shutdown_connections 2024-04-05T21:23:56.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:56.423+0000 7f5de6dc8700 1 --2- 172.21.15.121:0/1184763791 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f5dd006c4e0 0x7f5dd006e9a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:56.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:56.423+0000 7f5de6dc8700 1 --2- 172.21.15.121:0/1184763791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5de00ffab0 0x7f5de0103a40 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:56.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:56.423+0000 7f5de6dc8700 1 --2- 172.21.15.121:0/1184763791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5de0100cb0 0x7f5de0102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:56.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:56.423+0000 7f5de6dc8700 1 -- 172.21.15.121:0/1184763791 >> 172.21.15.121:0/1184763791 conn(0x7f5de00fb050 msgr2=0x7f5de0068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:56.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:56.423+0000 7f5de6dc8700 1 -- 172.21.15.121:0/1184763791 shutdown_connections 2024-04-05T21:23:56.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:56.423+0000 7f5de6dc8700 1 -- 172.21.15.121:0/1184763791 wait complete. 2024-04-05T21:23:57.128 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T21:23:57.139 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T21:23:57.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs set cephfs inline_data false' 2024-04-05T21:23:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:23:57.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:57.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:57.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:57.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi188.latnft", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:23:57.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi188.latnft", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T21:23:57.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:57.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: Deploying daemon mds.cephfs.smithi188.latnft on smithi188 2024-04-05T21:23:57.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-05T21:23:57.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: Cluster is now healthy 2024-04-05T21:23:57.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1184763791' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-05T21:23:57.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] up:boot 2024-04-05T21:23:57.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby 2024-04-05T21:23:57.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T21:23:57.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:57 smithi121 ceph-mon[39723]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 2024-04-05T21:23:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:23:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi188.latnft", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:23:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi188.latnft", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-05T21:23:57.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:23:57.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: Deploying daemon mds.cephfs.smithi188.latnft on smithi188 2024-04-05T21:23:57.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-05T21:23:57.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: Cluster is now healthy 2024-04-05T21:23:57.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1184763791' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-05T21:23:57.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] up:boot 2024-04-05T21:23:57.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby 2024-04-05T21:23:57.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T21:23:57.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:57 smithi188 ceph-mon[49715]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 2024-04-05T21:23:58.438 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:23:58 smithi188 ceph-mon[49715]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 88 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-05T21:23:58.666 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:23:58 smithi121 ceph-mon[39723]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 88 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-05T21:23:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.303+0000 7f3db9dc3700 1 -- 172.21.15.121:0/2843336857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db4100cb0 msgr2=0x7f3db4101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.303+0000 7f3db9dc3700 1 --2- 172.21.15.121:0/2843336857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db4100cb0 0x7f3db4101110 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009910 tx=0x7f3da8009c20 comp rx=0 tx=0).stop 2024-04-05T21:23:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.303+0000 7f3db9dc3700 1 -- 172.21.15.121:0/2843336857 shutdown_connections 2024-04-05T21:23:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.303+0000 7f3db9dc3700 1 --2- 172.21.15.121:0/2843336857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db4100cb0 0x7f3db4101110 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.303+0000 7f3db9dc3700 1 --2- 172.21.15.121:0/2843336857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3db40ffab0 0x7f3db40ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.303+0000 7f3db9dc3700 1 -- 172.21.15.121:0/2843336857 >> 172.21.15.121:0/2843336857 conn(0x7f3db40fb030 msgr2=0x7f3db40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:23:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.303+0000 7f3db9dc3700 1 -- 172.21.15.121:0/2843336857 shutdown_connections 2024-04-05T21:23:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.303+0000 7f3db9dc3700 1 -- 172.21.15.121:0/2843336857 wait complete. 2024-04-05T21:23:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.303+0000 7f3db9dc3700 1 Processor -- start 2024-04-05T21:23:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.303+0000 7f3db9dc3700 1 -- start start 2024-04-05T21:23:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db9dc3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3db40ffab0 0x7f3db4103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db9dc3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db4100cb0 0x7f3db4102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db9dc3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db41025d0 con 0x7f3db4100cb0 2024-04-05T21:23:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db9dc3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db4102710 con 0x7f3db40ffab0 2024-04-05T21:23:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db4100cb0 0x7f3db4102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db4100cb0 0x7f3db4102090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35540/0 (socket says 172.21.15.121:35540) 2024-04-05T21:23:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db2ffd700 1 -- 172.21.15.121:0/926182801 learned_addr learned my addr 172.21.15.121:0/926182801 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:23:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db37fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3db40ffab0 0x7f3db4103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db37fe700 1 -- 172.21.15.121:0/926182801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db4100cb0 msgr2=0x7f3db4102090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:23:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db37fe700 1 --2- 172.21.15.121:0/926182801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db4100cb0 0x7f3db4102090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:23:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db37fe700 1 -- 172.21.15.121:0/926182801 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da80095f0 con 0x7f3db40ffab0 2024-04-05T21:23:59.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db37fe700 1 --2- 172.21.15.121:0/926182801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3db40ffab0 0x7f3db4103a40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3da400e940 tx=0x7f3da400ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:59.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db0ff9700 1 -- 172.21.15.121:0/926182801 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3da40041f0 con 0x7f3db40ffab0 2024-04-05T21:23:59.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db9dc3700 1 -- 172.21.15.121:0/926182801 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db41029f0 con 0x7f3db40ffab0 2024-04-05T21:23:59.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db0ff9700 1 -- 172.21.15.121:0/926182801 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f3da4004350 con 0x7f3db40ffab0 2024-04-05T21:23:59.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.304+0000 7f3db0ff9700 1 -- 172.21.15.121:0/926182801 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3da4003c10 con 0x7f3db40ffab0 2024-04-05T21:23:59.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.305+0000 7f3db9dc3700 1 -- 172.21.15.121:0/926182801 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db419e2b0 con 0x7f3db40ffab0 2024-04-05T21:23:59.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.305+0000 7f3db9dc3700 1 -- 172.21.15.121:0/926182801 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db41084d0 con 0x7f3db40ffab0 2024-04-05T21:23:59.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.306+0000 7f3db0ff9700 1 -- 172.21.15.121:0/926182801 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3da400c7c0 con 0x7f3db40ffab0 2024-04-05T21:23:59.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.306+0000 7f3db0ff9700 1 --2- 172.21.15.121:0/926182801 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3d9c06c4e0 0x7f3d9c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:23:59.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.306+0000 7f3db0ff9700 1 -- 172.21.15.121:0/926182801 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f3da4014070 con 0x7f3db40ffab0 2024-04-05T21:23:59.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.307+0000 7f3db2ffd700 1 --2- 172.21.15.121:0/926182801 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3d9c06c4e0 0x7f3d9c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:23:59.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.307+0000 7f3db2ffd700 1 --2- 172.21.15.121:0/926182801 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3d9c06c4e0 0x7f3d9c06e9a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009fd0 tx=0x7f3da800b5a0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:23:59.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.311+0000 7f3db0ff9700 1 -- 172.21.15.121:0/926182801 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3da4059dd0 con 0x7f3db40ffab0 2024-04-05T21:23:59.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:23:59.561+0000 7f3db9dc3700 1 -- 172.21.15.121:0/926182801 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7f3db4108660 con 0x7f3db40ffab0 2024-04-05T21:24:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:00.260+0000 7f3db0ff9700 1 -- 172.21.15.121:0/926182801 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v13) v1 ==== 133+0+0 (secure 0 0 0) 0x7f3da4059960 con 0x7f3db40ffab0 2024-04-05T21:24:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:00.262+0000 7f3db9dc3700 1 -- 172.21.15.121:0/926182801 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3d9c06c4e0 msgr2=0x7f3d9c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:00.262+0000 7f3db9dc3700 1 --2- 172.21.15.121:0/926182801 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3d9c06c4e0 0x7f3d9c06e9a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009fd0 tx=0x7f3da800b5a0 comp rx=0 tx=0).stop 2024-04-05T21:24:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:00.262+0000 7f3db9dc3700 1 -- 172.21.15.121:0/926182801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3db40ffab0 msgr2=0x7f3db4103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:00.262+0000 7f3db9dc3700 1 --2- 172.21.15.121:0/926182801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3db40ffab0 0x7f3db4103a40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3da400e940 tx=0x7f3da400ec50 comp rx=0 tx=0).stop 2024-04-05T21:24:00.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:00.262+0000 7f3db9dc3700 1 -- 172.21.15.121:0/926182801 shutdown_connections 2024-04-05T21:24:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:00.262+0000 7f3db9dc3700 1 --2- 172.21.15.121:0/926182801 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3d9c06c4e0 0x7f3d9c06e9a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:00.262+0000 7f3db9dc3700 1 --2- 172.21.15.121:0/926182801 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3db40ffab0 0x7f3db4103a40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:00.262+0000 7f3db9dc3700 1 --2- 172.21.15.121:0/926182801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3db4100cb0 0x7f3db4102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:00.262+0000 7f3db9dc3700 1 -- 172.21.15.121:0/926182801 >> 172.21.15.121:0/926182801 conn(0x7f3db40fb030 msgr2=0x7f3db40683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:00.262+0000 7f3db9dc3700 1 -- 172.21.15.121:0/926182801 shutdown_connections 2024-04-05T21:24:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:00.262+0000 7f3db9dc3700 1 -- 172.21.15.121:0/926182801 wait complete. 2024-04-05T21:24:00.264 INFO:teuthology.orchestra.run.smithi121.stderr:inline data disabled 2024-04-05T21:24:00.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: pgmap v200: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-05T21:24:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:24:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] up:boot 2024-04-05T21:24:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T21:24:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.latnft"}]: dispatch 2024-04-05T21:24:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:24:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/926182801' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-05T21:24:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-05T21:24:00.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:00 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:00.284 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: pgmap v200: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 170 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-05T21:24:00.284 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:00.284 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:00.284 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:00.284 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:00.284 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:00.284 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:24:00.284 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] up:boot 2024-04-05T21:24:00.284 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T21:24:00.285 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.latnft"}]: dispatch 2024-04-05T21:24:00.285 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:24:00.285 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/926182801' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-05T21:24:00.285 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-05T21:24:00.285 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:00 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:01.113 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T21:24:01.123 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T21:24:01.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-05T21:24:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:01 smithi121 ceph-mon[39723]: Dropping low affinity active daemon mds.cephfs.smithi121.gxheqt in favor of higher affinity standby. 2024-04-05T21:24:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:01 smithi121 ceph-mon[39723]: Replacing daemon mds.cephfs.smithi121.gxheqt as rank 0 with standby daemon mds.cephfs.smithi121.ifoxmw 2024-04-05T21:24:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:01 smithi121 ceph-mon[39723]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-05T21:24:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:01 smithi121 ceph-mon[39723]: osdmap e46: 6 total, 6 up, 6 in 2024-04-05T21:24:01.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:01 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] up:active 2024-04-05T21:24:01.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:01 smithi121 ceph-mon[39723]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-05T21:24:01.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:01 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] up:standby-replay 2024-04-05T21:24:01.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:01 smithi121 ceph-mon[39723]: fsmap cephfs:2/2 {0=cephfs.smithi121.ifoxmw=up:replay,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 2024-04-05T21:24:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:01 smithi188 ceph-mon[49715]: Dropping low affinity active daemon mds.cephfs.smithi121.gxheqt in favor of higher affinity standby. 2024-04-05T21:24:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:01 smithi188 ceph-mon[49715]: Replacing daemon mds.cephfs.smithi121.gxheqt as rank 0 with standby daemon mds.cephfs.smithi121.ifoxmw 2024-04-05T21:24:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:01 smithi188 ceph-mon[49715]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-05T21:24:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:01 smithi188 ceph-mon[49715]: osdmap e46: 6 total, 6 up, 6 in 2024-04-05T21:24:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:01 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] up:active 2024-04-05T21:24:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:01 smithi188 ceph-mon[49715]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-05T21:24:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:01 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] up:standby-replay 2024-04-05T21:24:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:01 smithi188 ceph-mon[49715]: fsmap cephfs:2/2 {0=cephfs.smithi121.ifoxmw=up:replay,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 2024-04-05T21:24:01.715 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:24:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:02 smithi188 ceph-mon[49715]: pgmap v202: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-05T21:24:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:02 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.121:6826/4279084130,v1:172.21.15.121:6827/4279084130] up:boot 2024-04-05T21:24:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:02 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] up:resolve 2024-04-05T21:24:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:02 smithi188 ceph-mon[49715]: fsmap cephfs:2/2 {0=cephfs.smithi121.ifoxmw=up:resolve,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T21:24:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:02 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:24:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:02 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:02 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:02.631 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:02 smithi121 ceph-mon[39723]: pgmap v202: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 8 op/s 2024-04-05T21:24:02.631 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:02 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.121:6826/4279084130,v1:172.21.15.121:6827/4279084130] up:boot 2024-04-05T21:24:02.631 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:02 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] up:resolve 2024-04-05T21:24:02.631 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:02 smithi121 ceph-mon[39723]: fsmap cephfs:2/2 {0=cephfs.smithi121.ifoxmw=up:resolve,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T21:24:02.631 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:02 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:24:02.631 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:02 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:02.631 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:02 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:03.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:03 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] up:reconnect 2024-04-05T21:24:03.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:03 smithi188 ceph-mon[49715]: fsmap cephfs:2/2 {0=cephfs.smithi121.ifoxmw=up:reconnect,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T21:24:03.577 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:03 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] up:reconnect 2024-04-05T21:24:03.578 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:03 smithi121 ceph-mon[39723]: fsmap cephfs:2/2 {0=cephfs.smithi121.ifoxmw=up:reconnect,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T21:24:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:04 smithi188 ceph-mon[49715]: pgmap v203: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 819 B/s wr, 7 op/s 2024-04-05T21:24:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:04 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] up:rejoin 2024-04-05T21:24:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:04 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] up:standby-replay 2024-04-05T21:24:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:04 smithi188 ceph-mon[49715]: fsmap cephfs:2/2 {0=cephfs.smithi121.ifoxmw=up:rejoin,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T21:24:04.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:04 smithi188 ceph-mon[49715]: daemon mds.cephfs.smithi121.ifoxmw is now active in filesystem cephfs as rank 0 2024-04-05T21:24:04.531 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:04 smithi121 ceph-mon[39723]: pgmap v203: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 819 B/s wr, 7 op/s 2024-04-05T21:24:04.531 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:04 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] up:rejoin 2024-04-05T21:24:04.531 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:04 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] up:standby-replay 2024-04-05T21:24:04.531 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:04 smithi121 ceph-mon[39723]: fsmap cephfs:2/2 {0=cephfs.smithi121.ifoxmw=up:rejoin,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T21:24:04.531 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:04 smithi121 ceph-mon[39723]: daemon mds.cephfs.smithi121.ifoxmw is now active in filesystem cephfs as rank 0 2024-04-05T21:24:04.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.852+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/355521582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4103f70 msgr2=0x7fe2b4106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:04.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.852+0000 7fe2ba4dd700 1 --2- 172.21.15.121:0/355521582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4103f70 0x7fe2b4106360 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009910 tx=0x7fe2a4009c20 comp rx=0 tx=0).stop 2024-04-05T21:24:04.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.852+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/355521582 shutdown_connections 2024-04-05T21:24:04.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.852+0000 7fe2ba4dd700 1 --2- 172.21.15.121:0/355521582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4103f70 0x7fe2b4106360 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:04.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.852+0000 7fe2ba4dd700 1 --2- 172.21.15.121:0/355521582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2b4101640 0x7fe2b4103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:04.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.852+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/355521582 >> 172.21.15.121:0/355521582 conn(0x7fe2b40fb030 msgr2=0x7fe2b40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:04.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.852+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/355521582 shutdown_connections 2024-04-05T21:24:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.852+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/355521582 wait complete. 2024-04-05T21:24:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.852+0000 7fe2ba4dd700 1 Processor -- start 2024-04-05T21:24:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.852+0000 7fe2ba4dd700 1 -- start start 2024-04-05T21:24:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2ba4dd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2b4101640 0x7fe2b4196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2ba4dd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4103f70 0x7fe2b4196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2ba4dd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4196ea0 con 0x7fe2b4103f70 2024-04-05T21:24:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2ba4dd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4196fe0 con 0x7fe2b4101640 2024-04-05T21:24:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2b37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4103f70 0x7fe2b4196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2b37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4103f70 0x7fe2b4196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60824/0 (socket says 172.21.15.121:60824) 2024-04-05T21:24:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2b37fe700 1 -- 172.21.15.121:0/1533196020 learned_addr learned my addr 172.21.15.121:0/1533196020 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2b3fff700 1 --2- 172.21.15.121:0/1533196020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2b4101640 0x7fe2b4196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:04.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2b37fe700 1 -- 172.21.15.121:0/1533196020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2b4101640 msgr2=0x7fe2b4196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:04.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2b37fe700 1 --2- 172.21.15.121:0/1533196020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2b4101640 0x7fe2b4196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:04.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2b37fe700 1 -- 172.21.15.121:0/1533196020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a40095f0 con 0x7fe2b4103f70 2024-04-05T21:24:04.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2b3fff700 1 --2- 172.21.15.121:0/1533196020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2b4101640 0x7fe2b4196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:24:04.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2b37fe700 1 --2- 172.21.15.121:0/1533196020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4103f70 0x7fe2b4196880 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4006010 tx=0x7fe2a400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:04.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2b17fa700 1 -- 172.21.15.121:0/1533196020 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2a401d070 con 0x7fe2b4103f70 2024-04-05T21:24:04.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2b17fa700 1 -- 172.21.15.121:0/1533196020 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fe2a4022470 con 0x7fe2b4103f70 2024-04-05T21:24:04.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/1533196020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b419ba30 con 0x7fe2b4103f70 2024-04-05T21:24:04.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2b17fa700 1 -- 172.21.15.121:0/1533196020 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2a400f690 con 0x7fe2b4103f70 2024-04-05T21:24:04.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.853+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/1533196020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b419bfa0 con 0x7fe2b4103f70 2024-04-05T21:24:04.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.854+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/1533196020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b4190460 con 0x7fe2b4103f70 2024-04-05T21:24:04.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.855+0000 7fe2b17fa700 1 -- 172.21.15.121:0/1533196020 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fe2a4022980 con 0x7fe2b4103f70 2024-04-05T21:24:04.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.855+0000 7fe2b17fa700 1 --2- 172.21.15.121:0/1533196020 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fe29c06c4e0 0x7fe29c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:04.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.855+0000 7fe2b17fa700 1 -- 172.21.15.121:0/1533196020 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe2a408cee0 con 0x7fe2b4103f70 2024-04-05T21:24:04.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.855+0000 7fe2b3fff700 1 --2- 172.21.15.121:0/1533196020 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fe29c06c4e0 0x7fe29c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:04.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.855+0000 7fe2b3fff700 1 --2- 172.21.15.121:0/1533196020 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fe29c06c4e0 0x7fe29c06e9a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80099d0 tx=0x7fe2a8008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:04.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:04.859+0000 7fe2b17fa700 1 -- 172.21.15.121:0/1533196020 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fe2a405b290 con 0x7fe2b4103f70 2024-04-05T21:24:05.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:05.092+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/1533196020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe2b402cc70 con 0x7fe2b4103f70 2024-04-05T21:24:05.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:05.093+0000 7fe2b17fa700 1 -- 172.21.15.121:0/1533196020 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1853 (secure 0 0 0) 0x7fe2a4027070 con 0x7fe2b4103f70 2024-04-05T21:24:05.094 INFO:teuthology.orchestra.run.smithi121.stdout:e18 2024-04-05T21:24:05.094 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:24:05.094 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:24:05.094 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:24:05.094 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:24:05.094 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:24:05.094 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:24:05.094 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 18 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:24:04.283403+0000 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 46 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:24:05.095 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:up {0=14476,1=24255} 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:failed 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{0:14476} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{0:14488} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/4279084130,v1:172.21.15.121:6827/4279084130] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{1:24255} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{1:24265} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:24:05.096 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:24:05.097 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:24:05.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:05.094+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/1533196020 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fe29c06c4e0 msgr2=0x7fe29c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:05.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:05.094+0000 7fe2ba4dd700 1 --2- 172.21.15.121:0/1533196020 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fe29c06c4e0 0x7fe29c06e9a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe2a80099d0 tx=0x7fe2a8008040 comp rx=0 tx=0).stop 2024-04-05T21:24:05.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:05.094+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/1533196020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4103f70 msgr2=0x7fe2b4196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:05.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:05.094+0000 7fe2ba4dd700 1 --2- 172.21.15.121:0/1533196020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4103f70 0x7fe2b4196880 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4006010 tx=0x7fe2a400be20 comp rx=0 tx=0).stop 2024-04-05T21:24:05.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:05.095+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/1533196020 shutdown_connections 2024-04-05T21:24:05.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:05.095+0000 7fe2ba4dd700 1 --2- 172.21.15.121:0/1533196020 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fe29c06c4e0 0x7fe29c06e9a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:05.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:05.095+0000 7fe2ba4dd700 1 --2- 172.21.15.121:0/1533196020 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2b4101640 0x7fe2b4196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:05.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:05.095+0000 7fe2ba4dd700 1 --2- 172.21.15.121:0/1533196020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b4103f70 0x7fe2b4196880 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:05.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:05.095+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/1533196020 >> 172.21.15.121:0/1533196020 conn(0x7fe2b40fb030 msgr2=0x7fe2b40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:05.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:05.095+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/1533196020 shutdown_connections 2024-04-05T21:24:05.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:05.095+0000 7fe2ba4dd700 1 -- 172.21.15.121:0/1533196020 wait complete. 2024-04-05T21:24:05.098 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 18 2024-04-05T21:24:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:05 smithi121 ceph-mon[39723]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-05T21:24:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:05 smithi121 ceph-mon[39723]: Cluster is now healthy 2024-04-05T21:24:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:05 smithi121 ceph-mon[39723]: mds.? [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] up:active 2024-04-05T21:24:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:05 smithi121 ceph-mon[39723]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T21:24:05.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:05 smithi121 ceph-mon[39723]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:24:05.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:05 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1533196020' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:24:05.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:05 smithi188 ceph-mon[49715]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-05T21:24:05.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:05 smithi188 ceph-mon[49715]: Cluster is now healthy 2024-04-05T21:24:05.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:05 smithi188 ceph-mon[49715]: mds.? [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] up:active 2024-04-05T21:24:05.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:05 smithi188 ceph-mon[49715]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 1 up:standby-replay 1 up:standby 2024-04-05T21:24:05.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:05 smithi188 ceph-mon[49715]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:24:05.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:05 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1533196020' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:24:06.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-05T21:24:06.425 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:06 smithi121 ceph-mon[39723]: pgmap v204: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-05T21:24:06.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:06 smithi188 ceph-mon[49715]: pgmap v204: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-05T21:24:06.977 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:24:08.691 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:08 smithi121 ceph-mon[39723]: pgmap v205: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-05T21:24:08.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:08 smithi188 ceph-mon[49715]: pgmap v205: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-05T21:24:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.994+0000 7f3c5d159700 1 -- 172.21.15.121:0/285361007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c580ffab0 msgr2=0x7f3c580ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.994+0000 7f3c5d159700 1 --2- 172.21.15.121:0/285361007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c580ffab0 0x7f3c580ffed0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3c40009910 tx=0x7f3c40009c20 comp rx=0 tx=0).stop 2024-04-05T21:24:08.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.994+0000 7f3c5d159700 1 -- 172.21.15.121:0/285361007 shutdown_connections 2024-04-05T21:24:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.994+0000 7f3c5d159700 1 --2- 172.21.15.121:0/285361007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c58100cb0 0x7f3c58101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.994+0000 7f3c5d159700 1 --2- 172.21.15.121:0/285361007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c580ffab0 0x7f3c580ffed0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.994+0000 7f3c5d159700 1 -- 172.21.15.121:0/285361007 >> 172.21.15.121:0/285361007 conn(0x7f3c580fb050 msgr2=0x7f3c580fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.994+0000 7f3c5d159700 1 -- 172.21.15.121:0/285361007 shutdown_connections 2024-04-05T21:24:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.994+0000 7f3c5d159700 1 -- 172.21.15.121:0/285361007 wait complete. 2024-04-05T21:24:08.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.995+0000 7f3c5d159700 1 Processor -- start 2024-04-05T21:24:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.995+0000 7f3c5d159700 1 -- start start 2024-04-05T21:24:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.995+0000 7f3c5d159700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c580ffab0 0x7f3c581951b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.995+0000 7f3c5d159700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c58100cb0 0x7f3c581956f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.995+0000 7f3c5d159700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58195d10 con 0x7f3c58100cb0 2024-04-05T21:24:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.995+0000 7f3c5d159700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58195e50 con 0x7f3c580ffab0 2024-04-05T21:24:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.995+0000 7f3c5659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c58100cb0 0x7f3c581956f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.995+0000 7f3c5659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c58100cb0 0x7f3c581956f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60856/0 (socket says 172.21.15.121:60856) 2024-04-05T21:24:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.995+0000 7f3c5659c700 1 -- 172.21.15.121:0/1890921233 learned_addr learned my addr 172.21.15.121:0/1890921233 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:08.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.995+0000 7f3c56d9d700 1 --2- 172.21.15.121:0/1890921233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c580ffab0 0x7f3c581951b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.995+0000 7f3c5659c700 1 -- 172.21.15.121:0/1890921233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c580ffab0 msgr2=0x7f3c581951b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.995+0000 7f3c5659c700 1 --2- 172.21.15.121:0/1890921233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c580ffab0 0x7f3c581951b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.995+0000 7f3c5659c700 1 -- 172.21.15.121:0/1890921233 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c400095f0 con 0x7f3c58100cb0 2024-04-05T21:24:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.996+0000 7f3c5659c700 1 --2- 172.21.15.121:0/1890921233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c58100cb0 0x7f3c581956f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800d8d0 tx=0x7f3c4800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:08.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.996+0000 7f3c4ffff700 1 -- 172.21.15.121:0/1890921233 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c48009940 con 0x7f3c58100cb0 2024-04-05T21:24:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.996+0000 7f3c5d159700 1 -- 172.21.15.121:0/1890921233 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5819a900 con 0x7f3c58100cb0 2024-04-05T21:24:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.996+0000 7f3c4ffff700 1 -- 172.21.15.121:0/1890921233 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f3c48010460 con 0x7f3c58100cb0 2024-04-05T21:24:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.996+0000 7f3c4ffff700 1 -- 172.21.15.121:0/1890921233 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c4800f5f0 con 0x7f3c58100cb0 2024-04-05T21:24:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.996+0000 7f3c5d159700 1 -- 172.21.15.121:0/1890921233 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5819aed0 con 0x7f3c58100cb0 2024-04-05T21:24:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.997+0000 7f3c4ffff700 1 -- 172.21.15.121:0/1890921233 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3c480105d0 con 0x7f3c58100cb0 2024-04-05T21:24:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.997+0000 7f3c5d159700 1 -- 172.21.15.121:0/1890921233 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c581084d0 con 0x7f3c58100cb0 2024-04-05T21:24:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.998+0000 7f3c4ffff700 1 --2- 172.21.15.121:0/1890921233 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3c4406c4e0 0x7f3c4406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.998+0000 7f3c4ffff700 1 -- 172.21.15.121:0/1890921233 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f3c4808b790 con 0x7f3c58100cb0 2024-04-05T21:24:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.998+0000 7f3c56d9d700 1 --2- 172.21.15.121:0/1890921233 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3c4406c4e0 0x7f3c4406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:08.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:08.998+0000 7f3c56d9d700 1 --2- 172.21.15.121:0/1890921233 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3c4406c4e0 0x7f3c4406e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3c400059d0 tx=0x7f3c40005960 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:09.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.006+0000 7f3c4ffff700 1 -- 172.21.15.121:0/1890921233 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3c48059a20 con 0x7f3c58100cb0 2024-04-05T21:24:09.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.254+0000 7f3c5d159700 1 -- 172.21.15.121:0/1890921233 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c5819b6f0 con 0x7f3c58100cb0 2024-04-05T21:24:09.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.254+0000 7f3c4ffff700 1 -- 172.21.15.121:0/1890921233 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4779 (secure 0 0 0) 0x7f3c48016090 con 0x7f3c58100cb0 2024-04-05T21:24:09.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.256+0000 7f3c5d159700 1 -- 172.21.15.121:0/1890921233 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3c4406c4e0 msgr2=0x7f3c4406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:09.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.256+0000 7f3c5d159700 1 --2- 172.21.15.121:0/1890921233 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3c4406c4e0 0x7f3c4406e9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3c400059d0 tx=0x7f3c40005960 comp rx=0 tx=0).stop 2024-04-05T21:24:09.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.256+0000 7f3c5d159700 1 -- 172.21.15.121:0/1890921233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c58100cb0 msgr2=0x7f3c581956f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:09.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.256+0000 7f3c5d159700 1 --2- 172.21.15.121:0/1890921233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c58100cb0 0x7f3c581956f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800d8d0 tx=0x7f3c4800dc90 comp rx=0 tx=0).stop 2024-04-05T21:24:09.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.256+0000 7f3c5d159700 1 -- 172.21.15.121:0/1890921233 shutdown_connections 2024-04-05T21:24:09.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.256+0000 7f3c5d159700 1 --2- 172.21.15.121:0/1890921233 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3c4406c4e0 0x7f3c4406e9a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:09.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.256+0000 7f3c5d159700 1 --2- 172.21.15.121:0/1890921233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c580ffab0 0x7f3c581951b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:09.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.256+0000 7f3c5d159700 1 --2- 172.21.15.121:0/1890921233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c58100cb0 0x7f3c581956f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:09.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.256+0000 7f3c5d159700 1 -- 172.21.15.121:0/1890921233 >> 172.21.15.121:0/1890921233 conn(0x7f3c580fb050 msgr2=0x7f3c58103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:09.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.256+0000 7f3c5d159700 1 -- 172.21.15.121:0/1890921233 shutdown_connections 2024-04-05T21:24:09.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:09.256+0000 7f3c5d159700 1 -- 172.21.15.121:0/1890921233 wait complete. 2024-04-05T21:24:09.258 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 18 2024-04-05T21:24:09.267 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:24:09.710 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:09.711 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:09.711 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:24:09.711 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:24:09.711 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:09.711 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:24:09.711 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:09 smithi121 ceph-mon[39723]: pgmap v206: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 11 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-05T21:24:09.711 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:09 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1890921233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T21:24:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:24:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:24:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:24:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:09 smithi188 ceph-mon[49715]: pgmap v206: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 11 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-05T21:24:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:09 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1890921233' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T21:24:10.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-05T21:24:10.664 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:24:11.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:11 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:11.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:11 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:24:11.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:11 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:11.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:11 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:11 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:11.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:11 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:24:11.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:11 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:11.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:11 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:12.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:12 smithi121 ceph-mon[39723]: pgmap v207: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-05T21:24:12.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:12 smithi188 ceph-mon[49715]: pgmap v207: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-05T21:24:13.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.898+0000 7f1db42c9700 1 -- 172.21.15.121:0/927591926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dac100cb0 msgr2=0x7f1dac101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:13.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.898+0000 7f1db42c9700 1 --2- 172.21.15.121:0/927591926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dac100cb0 0x7f1dac101110 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1da8009910 tx=0x7f1da8009c20 comp rx=0 tx=0).stop 2024-04-05T21:24:13.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.898+0000 7f1db42c9700 1 -- 172.21.15.121:0/927591926 shutdown_connections 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.898+0000 7f1db42c9700 1 --2- 172.21.15.121:0/927591926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dac100cb0 0x7f1dac101110 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.898+0000 7f1db42c9700 1 --2- 172.21.15.121:0/927591926 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1dac0ffab0 0x7f1dac0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.898+0000 7f1db42c9700 1 -- 172.21.15.121:0/927591926 >> 172.21.15.121:0/927591926 conn(0x7f1dac0fb030 msgr2=0x7f1dac0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.898+0000 7f1db42c9700 1 -- 172.21.15.121:0/927591926 shutdown_connections 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.898+0000 7f1db42c9700 1 -- 172.21.15.121:0/927591926 wait complete. 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db42c9700 1 Processor -- start 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db42c9700 1 -- start start 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db42c9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1dac0ffab0 0x7f1dac068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db42c9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dac100cb0 0x7f1dac19d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db42c9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac196d90 con 0x7f1dac100cb0 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db42c9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac19d8f0 con 0x7f1dac0ffab0 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db1864700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dac100cb0 0x7f1dac19d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db1864700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dac100cb0 0x7f1dac19d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60870/0 (socket says 172.21.15.121:60870) 2024-04-05T21:24:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db1864700 1 -- 172.21.15.121:0/3338667486 learned_addr learned my addr 172.21.15.121:0/3338667486 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db2065700 1 --2- 172.21.15.121:0/3338667486 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1dac0ffab0 0x7f1dac068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db1864700 1 -- 172.21.15.121:0/3338667486 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1dac0ffab0 msgr2=0x7f1dac068f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db1864700 1 --2- 172.21.15.121:0/3338667486 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1dac0ffab0 0x7f1dac068f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db1864700 1 -- 172.21.15.121:0/3338667486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da80095f0 con 0x7f1dac100cb0 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db1864700 1 --2- 172.21.15.121:0/3338667486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dac100cb0 0x7f1dac19d3b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1da800bfa0 tx=0x7f1da800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1d9f7fe700 1 -- 172.21.15.121:0/3338667486 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1da801d070 con 0x7f1dac100cb0 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1d9f7fe700 1 -- 172.21.15.121:0/3338667486 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f1da800f460 con 0x7f1dac100cb0 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1db42c9700 1 -- 172.21.15.121:0/3338667486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac19daf0 con 0x7f1dac100cb0 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.899+0000 7f1d9f7fe700 1 -- 172.21.15.121:0/3338667486 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1da8017630 con 0x7f1dac100cb0 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.900+0000 7f1db42c9700 1 -- 172.21.15.121:0/3338667486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac19dfb0 con 0x7f1dac100cb0 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.901+0000 7f1db42c9700 1 -- 172.21.15.121:0/3338667486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac1084d0 con 0x7f1dac100cb0 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.901+0000 7f1d9f7fe700 1 -- 172.21.15.121:0/3338667486 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f1da800f9b0 con 0x7f1dac100cb0 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.901+0000 7f1d9f7fe700 1 --2- 172.21.15.121:0/3338667486 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1d9806c4e0 0x7f1d9806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:13.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.901+0000 7f1d9f7fe700 1 -- 172.21.15.121:0/3338667486 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1da808cdd0 con 0x7f1dac100cb0 2024-04-05T21:24:13.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.901+0000 7f1db2065700 1 --2- 172.21.15.121:0/3338667486 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1d9806c4e0 0x7f1d9806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:13.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.902+0000 7f1db2065700 1 --2- 172.21.15.121:0/3338667486 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1d9806c4e0 0x7f1d9806e9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1dac100b10 tx=0x7f1da000a3b0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:13.233 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:24:13.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:12.906+0000 7f1d9f7fe700 1 -- 172.21.15.121:0/3338667486 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1da805afe0 con 0x7f1dac100cb0 2024-04-05T21:24:13.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:13.181+0000 7f1db42c9700 1 -- 172.21.15.121:0/3338667486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f1dac108690 con 0x7f1dac100cb0 2024-04-05T21:24:13.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:13.181+0000 7f1d9f7fe700 1 -- 172.21.15.121:0/3338667486 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v18) v1 ==== 78+0+83 (secure 0 0 0) 0x7f1da805ab70 con 0x7f1dac100cb0 2024-04-05T21:24:13.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:13.183+0000 7f1db42c9700 1 -- 172.21.15.121:0/3338667486 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1d9806c4e0 msgr2=0x7f1d9806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:13.183+0000 7f1db42c9700 1 --2- 172.21.15.121:0/3338667486 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1d9806c4e0 0x7f1d9806e9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1dac100b10 tx=0x7f1da000a3b0 comp rx=0 tx=0).stop 2024-04-05T21:24:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:13.183+0000 7f1db42c9700 1 -- 172.21.15.121:0/3338667486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dac100cb0 msgr2=0x7f1dac19d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:13.183+0000 7f1db42c9700 1 --2- 172.21.15.121:0/3338667486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dac100cb0 0x7f1dac19d3b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1da800bfa0 tx=0x7f1da800bd70 comp rx=0 tx=0).stop 2024-04-05T21:24:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:13.183+0000 7f1db42c9700 1 -- 172.21.15.121:0/3338667486 shutdown_connections 2024-04-05T21:24:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:13.183+0000 7f1db42c9700 1 --2- 172.21.15.121:0/3338667486 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1d9806c4e0 0x7f1d9806e9a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:13.183+0000 7f1db42c9700 1 --2- 172.21.15.121:0/3338667486 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1dac0ffab0 0x7f1dac068f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:13.183+0000 7f1db42c9700 1 --2- 172.21.15.121:0/3338667486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1dac100cb0 0x7f1dac19d3b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:13.183+0000 7f1db42c9700 1 -- 172.21.15.121:0/3338667486 >> 172.21.15.121:0/3338667486 conn(0x7f1dac0fb030 msgr2=0x7f1dac103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:13.183+0000 7f1db42c9700 1 -- 172.21.15.121:0/3338667486 shutdown_connections 2024-04-05T21:24:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:13.183+0000 7f1db42c9700 1 -- 172.21.15.121:0/3338667486 wait complete. 2024-04-05T21:24:13.896 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-05T21:24:13.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T21:24:14.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.055+0000 7f4f08e3d700 1 -- 172.21.15.121:0/3603733202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f04100c10 msgr2=0x7f4f04101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:14.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.055+0000 7f4f08e3d700 1 --2- 172.21.15.121:0/3603733202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f04100c10 0x7f4f04101070 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009910 tx=0x7f4ef4009c20 comp rx=0 tx=0).stop 2024-04-05T21:24:14.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.055+0000 7f4f08e3d700 1 -- 172.21.15.121:0/3603733202 shutdown_connections 2024-04-05T21:24:14.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.055+0000 7f4f08e3d700 1 --2- 172.21.15.121:0/3603733202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f04100c10 0x7f4f04101070 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.055+0000 7f4f08e3d700 1 --2- 172.21.15.121:0/3603733202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f040ffa20 0x7f4f040ffe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.055+0000 7f4f08e3d700 1 -- 172.21.15.121:0/3603733202 >> 172.21.15.121:0/3603733202 conn(0x7f4f040faf50 msgr2=0x7f4f040fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:14.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.055+0000 7f4f08e3d700 1 -- 172.21.15.121:0/3603733202 shutdown_connections 2024-04-05T21:24:14.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.055+0000 7f4f08e3d700 1 -- 172.21.15.121:0/3603733202 wait complete. 2024-04-05T21:24:14.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.055+0000 7f4f08e3d700 1 Processor -- start 2024-04-05T21:24:14.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.055+0000 7f4f08e3d700 1 -- start start 2024-04-05T21:24:14.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f08e3d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f040ffa20 0x7f4f04195070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:14.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f08e3d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f04100c10 0x7f4f041955b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:14.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f08e3d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04195bd0 con 0x7f4f04100c10 2024-04-05T21:24:14.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f08e3d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04195d10 con 0x7f4f040ffa20 2024-04-05T21:24:14.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f01d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f04100c10 0x7f4f041955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:14.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f01d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f04100c10 0x7f4f041955b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60888/0 (socket says 172.21.15.121:60888) 2024-04-05T21:24:14.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f01d9b700 1 -- 172.21.15.121:0/289663105 learned_addr learned my addr 172.21.15.121:0/289663105 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:14.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f0259c700 1 --2- 172.21.15.121:0/289663105 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f040ffa20 0x7f4f04195070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:14.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f01d9b700 1 -- 172.21.15.121:0/289663105 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f040ffa20 msgr2=0x7f4f04195070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:14.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f01d9b700 1 --2- 172.21.15.121:0/289663105 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f040ffa20 0x7f4f04195070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f01d9b700 1 -- 172.21.15.121:0/289663105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef40095f0 con 0x7f4f04100c10 2024-04-05T21:24:14.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f01d9b700 1 --2- 172.21.15.121:0/289663105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f04100c10 0x7f4f041955b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400b5c0 tx=0x7f4ef400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:14.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4efb7fe700 1 -- 172.21.15.121:0/289663105 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ef401d070 con 0x7f4f04100c10 2024-04-05T21:24:14.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f08e3d700 1 -- 172.21.15.121:0/289663105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0419a750 con 0x7f4f04100c10 2024-04-05T21:24:14.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4efb7fe700 1 -- 172.21.15.121:0/289663105 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f4ef4022470 con 0x7f4f04100c10 2024-04-05T21:24:14.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4efb7fe700 1 -- 172.21.15.121:0/289663105 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ef400f670 con 0x7f4f04100c10 2024-04-05T21:24:14.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.056+0000 7f4f08e3d700 1 -- 172.21.15.121:0/289663105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0419acd0 con 0x7f4f04100c10 2024-04-05T21:24:14.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.057+0000 7f4f08e3d700 1 -- 172.21.15.121:0/289663105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f04108450 con 0x7f4f04100c10 2024-04-05T21:24:14.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.058+0000 7f4efb7fe700 1 -- 172.21.15.121:0/289663105 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4ef4004af0 con 0x7f4f04100c10 2024-04-05T21:24:14.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.058+0000 7f4efb7fe700 1 --2- 172.21.15.121:0/289663105 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4ef006c5a0 0x7f4ef006ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:14.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.058+0000 7f4efb7fe700 1 -- 172.21.15.121:0/289663105 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4ef408cf00 con 0x7f4f04100c10 2024-04-05T21:24:14.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.058+0000 7f4f0259c700 1 --2- 172.21.15.121:0/289663105 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4ef006c5a0 0x7f4ef006ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:14.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.058+0000 7f4f0259c700 1 --2- 172.21.15.121:0/289663105 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4ef006c5a0 0x7f4ef006ea60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4eec005fd0 tx=0x7f4eec005dd0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:14.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.062+0000 7f4efb7fe700 1 -- 172.21.15.121:0/289663105 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f4ef405b1c0 con 0x7f4f04100c10 2024-04-05T21:24:14.220 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:14 smithi121 ceph-mon[39723]: pgmap v208: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-05T21:24:14.220 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:14 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3338667486' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-05T21:24:14.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.278+0000 7f4f08e3d700 1 -- 172.21.15.121:0/289663105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f0419b310 con 0x7f4f04100c10 2024-04-05T21:24:14.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:14 smithi188 ceph-mon[49715]: pgmap v208: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 8.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-05T21:24:14.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:14 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/3338667486' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-05T21:24:14.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.280+0000 7f4efb7fe700 1 -- 172.21.15.121:0/289663105 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+15190 (secure 0 0 0) 0x7f4ef405ad50 con 0x7f4f04100c10 2024-04-05T21:24:14.281 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:24:14.281 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":46,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","created":"2024-04-05T21:15:58.797109+0000","modified":"2024-04-05T21:24:00.251529+0000","last_up_change":"2024-04-05T21:22:17.062975+0000","last_in_change":"2024-04-05T21:21:56.806570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T21:20:53.291734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-05T21:23:42.376376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-05T21:23:42.658469+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"133652f8-9fe2-45e9-90b6-36f89191114b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6803","nonce":2285715890}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6805","nonce":2285715890}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6809","nonce":2285715890}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2285715890},{"type":"v1","addr":"172.21.15.121:6807","nonce":2285715890}]},"public_addr":"172.21.15.121:6803/2285715890","cluster_addr":"172.21.15.121:6805/2285715890","heartbeat_back_addr":"172.21.15.121:6809/2285715890","heartbeat_front_addr":"172.21.15.121:6807/2285715890","state":["exists","up"]},{"osd":1,"uuid":"6695518f-c4a5-416a-8af1-2899fdebefd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6811","nonce":3573482215}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6813","nonce":3573482215}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6817","nonce":3573482215}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3573482215},{"type":"v1","addr":"172.21.15.121:6815","nonce":3573482215}]},"public_addr":"172.21.15.121:6811/3573482215","cluster_addr":"172.21.15.121:6813/3573482215","heartbeat_back_addr":"172.21.15.121:6817/3573482215","heartbeat_front_addr":"172.21.15.121:6815/3573482215","state":["exists","up"]},{"osd":2,"uuid":"e3b6290b-ef26-4cda-96e2-8bb5649219a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6819","nonce":179235010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6821","nonce":179235010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6825","nonce":179235010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":179235010},{"type":"v1","addr":"172.21.15.121:6823","nonce":179235010}]},"public_addr":"172.21.15.121:6819/179235010","cluster_addr":"172.21.15.121:6821/179235010","heartbeat_back_addr":"172.21.15.121:6825/179235010","heartbeat_front_addr":"172.21.15.121:6823/179235010","state":["exists","up"]},{"osd":3,"uuid":"77fa8da5-d202-45f1-b6fa-5af969e9300c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6801","nonce":2760393079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6803","nonce":2760393079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6807","nonce":2760393079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":2760393079},{"type":"v1","addr":"172.21.15.188:6805","nonce":2760393079}]},"public_addr":"172.21.15.188:6801/2760393079","cluster_addr":"172.21.15.188:6803/2760393079","heartbeat_back_addr":"172.21.15.188:6807/2760393079","heartbeat_front_addr":"172.21.15.188:6805/2760393079","state":["exists","up"]},{"osd":4,"uuid":"abf81f28-7473-45ef-a716-5ef16e038b6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6809","nonce":1828363110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6811","nonce":1828363110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6815","nonce":1828363110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":1828363110},{"type":"v1","addr":"172.21.15.188:6813","nonce":1828363110}]},"public_addr":"172.21.15.188:6809/1828363110","cluster_addr":"172.21.15.188:6811/1828363110","heartbeat_back_addr":"172.21.15.188:6815/1828363110","heartbeat_front_addr":"172.21.15.188:6813/1828363110","state":["exists","up"]},{"osd":5,"uuid":"700a347b-e303-447d-bf03-d6677955d140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6817","nonce":3569223975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6819","nonce":3569223975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6823","nonce":3569223975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3569223975},{"type":"v1","addr":"172.21.15.188:6821","nonce":3569223975}]},"public_addr":"172.21.15.188:6817/3569223975","cluster_addr":"172.21.15.188:6819/3569223975","heartbeat_back_addr":"172.21.15.188:6823/3569223975","heartbeat_front_addr":"172.21.15.188:6821/3569223975","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:19:48.038490+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:19.269394+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:51.162437+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:18.203341+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:46.805653+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:22:15.296548+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/42058318":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/2383002272":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/2536941662":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/1364585923":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/2214582362":"2024-04-06T21:17:08.626094+0000","172.21.15.121:6827/1615448980":"2024-04-06T21:24:00.251505+0000","172.21.15.121:0/341396693":"2024-04-06T21:17:08.626094+0000","172.21.15.121:6800/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:6801/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/3247654758":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6801/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/3235459786":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6826/1615448980":"2024-04-06T21:24:00.251505+0000","172.21.15.121:6800/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/3762940011":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6800/1237774071":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6801/1237774071":"2024-04-06T21:18:24.954707+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T21:24:14.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.282+0000 7f4f08e3d700 1 -- 172.21.15.121:0/289663105 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4ef006c5a0 msgr2=0x7f4ef006ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:14.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.282+0000 7f4f08e3d700 1 --2- 172.21.15.121:0/289663105 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4ef006c5a0 0x7f4ef006ea60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4eec005fd0 tx=0x7f4eec005dd0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.282+0000 7f4f08e3d700 1 -- 172.21.15.121:0/289663105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f04100c10 msgr2=0x7f4f041955b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:14.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.282+0000 7f4f08e3d700 1 --2- 172.21.15.121:0/289663105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f04100c10 0x7f4f041955b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400b5c0 tx=0x7f4ef400be20 comp rx=0 tx=0).stop 2024-04-05T21:24:14.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.282+0000 7f4f08e3d700 1 -- 172.21.15.121:0/289663105 shutdown_connections 2024-04-05T21:24:14.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.282+0000 7f4f08e3d700 1 --2- 172.21.15.121:0/289663105 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f4ef006c5a0 0x7f4ef006ea60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.282+0000 7f4f08e3d700 1 --2- 172.21.15.121:0/289663105 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4f040ffa20 0x7f4f04195070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.282+0000 7f4f08e3d700 1 --2- 172.21.15.121:0/289663105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f04100c10 0x7f4f041955b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.282+0000 7f4f08e3d700 1 -- 172.21.15.121:0/289663105 >> 172.21.15.121:0/289663105 conn(0x7f4f040faf50 msgr2=0x7f4f04103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:14.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.282+0000 7f4f08e3d700 1 -- 172.21.15.121:0/289663105 shutdown_connections 2024-04-05T21:24:14.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.282+0000 7f4f08e3d700 1 -- 172.21.15.121:0/289663105 wait complete. 2024-04-05T21:24:14.296 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T21:20:53.291734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-05T21:23:42.376376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-05T21:23:42.658469+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T21:24:14.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T21:24:14.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.422+0000 7fc6fff78700 1 -- 172.21.15.121:0/1765781348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f80ffc30 msgr2=0x7fc6f81000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:14.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.422+0000 7fc6fff78700 1 --2- 172.21.15.121:0/1765781348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f80ffc30 0x7fc6f81000b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8009910 tx=0x7fc6e8009c20 comp rx=0 tx=0).stop 2024-04-05T21:24:14.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.422+0000 7fc6fff78700 1 -- 172.21.15.121:0/1765781348 shutdown_connections 2024-04-05T21:24:14.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.422+0000 7fc6fff78700 1 --2- 172.21.15.121:0/1765781348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f80ffc30 0x7fc6f81000b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.422+0000 7fc6fff78700 1 --2- 172.21.15.121:0/1765781348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6f8100fe0 0x7fc6f8101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.422+0000 7fc6fff78700 1 -- 172.21.15.121:0/1765781348 >> 172.21.15.121:0/1765781348 conn(0x7fc6f80fa700 msgr2=0x7fc6f80fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:14.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.422+0000 7fc6fff78700 1 -- 172.21.15.121:0/1765781348 shutdown_connections 2024-04-05T21:24:14.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.422+0000 7fc6fff78700 1 -- 172.21.15.121:0/1765781348 wait complete. 2024-04-05T21:24:14.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fff78700 1 Processor -- start 2024-04-05T21:24:14.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fff78700 1 -- start start 2024-04-05T21:24:14.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fff78700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6f80ffc30 0x7fc6f80ff690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:14.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fff78700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f8100fe0 0x7fc6f819af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:14.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fff78700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f819b550 con 0x7fc6f8100fe0 2024-04-05T21:24:14.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fff78700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f819b690 con 0x7fc6f80ffc30 2024-04-05T21:24:14.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fd513700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f8100fe0 0x7fc6f819af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:14.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fd513700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f8100fe0 0x7fc6f819af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51074/0 (socket says 172.21.15.121:51074) 2024-04-05T21:24:14.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fd513700 1 -- 172.21.15.121:0/2835159078 learned_addr learned my addr 172.21.15.121:0/2835159078 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:14.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fdd14700 1 --2- 172.21.15.121:0/2835159078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6f80ffc30 0x7fc6f80ff690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:14.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fd513700 1 -- 172.21.15.121:0/2835159078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6f80ffc30 msgr2=0x7fc6f80ff690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:14.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fd513700 1 --2- 172.21.15.121:0/2835159078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6f80ffc30 0x7fc6f80ff690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fd513700 1 -- 172.21.15.121:0/2835159078 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e80095f0 con 0x7fc6f8100fe0 2024-04-05T21:24:14.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.423+0000 7fc6fd513700 1 --2- 172.21.15.121:0/2835159078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f8100fe0 0x7fc6f819af80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc6e800bda0 tx=0x7fc6e800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:14.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.424+0000 7fc6eeffd700 1 -- 172.21.15.121:0/2835159078 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6e801d070 con 0x7fc6f8100fe0 2024-04-05T21:24:14.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.424+0000 7fc6eeffd700 1 -- 172.21.15.121:0/2835159078 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fc6e8022470 con 0x7fc6f8100fe0 2024-04-05T21:24:14.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.424+0000 7fc6eeffd700 1 -- 172.21.15.121:0/2835159078 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6e800f690 con 0x7fc6f8100fe0 2024-04-05T21:24:14.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.424+0000 7fc6fff78700 1 -- 172.21.15.121:0/2835159078 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f819b910 con 0x7fc6f8100fe0 2024-04-05T21:24:14.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.424+0000 7fc6fff78700 1 -- 172.21.15.121:0/2835159078 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f819be90 con 0x7fc6f8100fe0 2024-04-05T21:24:14.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.425+0000 7fc6fff78700 1 -- 172.21.15.121:0/2835159078 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f8108450 con 0x7fc6f8100fe0 2024-04-05T21:24:14.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.425+0000 7fc6eeffd700 1 -- 172.21.15.121:0/2835159078 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fc6e80229c0 con 0x7fc6f8100fe0 2024-04-05T21:24:14.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.426+0000 7fc6eeffd700 1 --2- 172.21.15.121:0/2835159078 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fc6e406c4e0 0x7fc6e406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:14.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.426+0000 7fc6fdd14700 1 --2- 172.21.15.121:0/2835159078 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fc6e406c4e0 0x7fc6e406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:14.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.426+0000 7fc6eeffd700 1 -- 172.21.15.121:0/2835159078 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc6e808cf40 con 0x7fc6f8100fe0 2024-04-05T21:24:14.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.426+0000 7fc6fdd14700 1 --2- 172.21.15.121:0/2835159078 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fc6e406c4e0 0x7fc6e406e9a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0005fd0 tx=0x7fc6f0005dd0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:14.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.430+0000 7fc6eeffd700 1 -- 172.21.15.121:0/2835159078 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fc6e80c00e0 con 0x7fc6f8100fe0 2024-04-05T21:24:14.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.624+0000 7fc6fff78700 1 -- 172.21.15.121:0/2835159078 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc6f819c500 con 0x7fc6f8100fe0 2024-04-05T21:24:14.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.625+0000 7fc6eeffd700 1 -- 172.21.15.121:0/2835159078 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc6e805b200 con 0x7fc6f8100fe0 2024-04-05T21:24:14.626 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2024-04-05T21:24:14.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.627+0000 7fc6fff78700 1 -- 172.21.15.121:0/2835159078 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fc6e406c4e0 msgr2=0x7fc6e406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:14.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.627+0000 7fc6fff78700 1 --2- 172.21.15.121:0/2835159078 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fc6e406c4e0 0x7fc6e406e9a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc6f0005fd0 tx=0x7fc6f0005dd0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.627+0000 7fc6fff78700 1 -- 172.21.15.121:0/2835159078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f8100fe0 msgr2=0x7fc6f819af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:14.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.627+0000 7fc6fff78700 1 --2- 172.21.15.121:0/2835159078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f8100fe0 0x7fc6f819af80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc6e800bda0 tx=0x7fc6e800be80 comp rx=0 tx=0).stop 2024-04-05T21:24:14.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.627+0000 7fc6fff78700 1 -- 172.21.15.121:0/2835159078 shutdown_connections 2024-04-05T21:24:14.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.627+0000 7fc6fff78700 1 --2- 172.21.15.121:0/2835159078 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fc6e406c4e0 0x7fc6e406e9a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.627+0000 7fc6fff78700 1 --2- 172.21.15.121:0/2835159078 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6f80ffc30 0x7fc6f80ff690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.627+0000 7fc6fff78700 1 --2- 172.21.15.121:0/2835159078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f8100fe0 0x7fc6f819af80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.627+0000 7fc6fff78700 1 -- 172.21.15.121:0/2835159078 >> 172.21.15.121:0/2835159078 conn(0x7fc6f80fa700 msgr2=0x7fc6f8103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:14.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.627+0000 7fc6fff78700 1 -- 172.21.15.121:0/2835159078 shutdown_connections 2024-04-05T21:24:14.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.627+0000 7fc6fff78700 1 -- 172.21.15.121:0/2835159078 wait complete. 2024-04-05T21:24:14.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-05T21:24:14.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.771+0000 7fd2a685e700 1 -- 172.21.15.121:0/36282253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a0100b10 msgr2=0x7fd2a0100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:14.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.771+0000 7fd2a685e700 1 --2- 172.21.15.121:0/36282253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a0100b10 0x7fd2a0100f30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd290009870 tx=0x7fd290009b80 comp rx=0 tx=0).stop 2024-04-05T21:24:14.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.771+0000 7fd2a685e700 1 -- 172.21.15.121:0/36282253 shutdown_connections 2024-04-05T21:24:14.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.771+0000 7fd2a685e700 1 --2- 172.21.15.121:0/36282253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a0101d00 0x7fd2a0102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.771+0000 7fd2a685e700 1 --2- 172.21.15.121:0/36282253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a0100b10 0x7fd2a0100f30 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.771+0000 7fd2a685e700 1 -- 172.21.15.121:0/36282253 >> 172.21.15.121:0/36282253 conn(0x7fd2a00fc060 msgr2=0x7fd2a00fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:14.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.771+0000 7fd2a685e700 1 -- 172.21.15.121:0/36282253 shutdown_connections 2024-04-05T21:24:14.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.771+0000 7fd2a685e700 1 -- 172.21.15.121:0/36282253 wait complete. 2024-04-05T21:24:14.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.772+0000 7fd2a685e700 1 Processor -- start 2024-04-05T21:24:14.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.772+0000 7fd2a685e700 1 -- start start 2024-04-05T21:24:14.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.772+0000 7fd2a685e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a0100b10 0x7fd2a0196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:14.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.772+0000 7fd2a685e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a0101d00 0x7fd2a01967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:14.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd2a685e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a0196dd0 con 0x7fd2a0101d00 2024-04-05T21:24:14.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd2a685e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a0196f10 con 0x7fd2a0100b10 2024-04-05T21:24:14.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd29f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a0101d00 0x7fd2a01967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:14.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd29f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a0101d00 0x7fd2a01967b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51100/0 (socket says 172.21.15.121:51100) 2024-04-05T21:24:14.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd29f7fe700 1 -- 172.21.15.121:0/1529913296 learned_addr learned my addr 172.21.15.121:0/1529913296 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:14.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd29ffff700 1 --2- 172.21.15.121:0/1529913296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a0100b10 0x7fd2a0196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:14.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd29f7fe700 1 -- 172.21.15.121:0/1529913296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a0100b10 msgr2=0x7fd2a0196270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:14.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd29f7fe700 1 --2- 172.21.15.121:0/1529913296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a0100b10 0x7fd2a0196270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd29f7fe700 1 -- 172.21.15.121:0/1529913296 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd290009520 con 0x7fd2a0101d00 2024-04-05T21:24:14.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd29f7fe700 1 --2- 172.21.15.121:0/1529913296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a0101d00 0x7fd2a01967b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd29400d900 tx=0x7fd29400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:14.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd29d7fa700 1 -- 172.21.15.121:0/1529913296 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd294009900 con 0x7fd2a0101d00 2024-04-05T21:24:14.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd2a685e700 1 -- 172.21.15.121:0/1529913296 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a019b9b0 con 0x7fd2a0101d00 2024-04-05T21:24:14.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd2a685e700 1 -- 172.21.15.121:0/1529913296 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a019bf90 con 0x7fd2a0101d00 2024-04-05T21:24:14.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd29d7fa700 1 -- 172.21.15.121:0/1529913296 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fd294010460 con 0x7fd2a0101d00 2024-04-05T21:24:14.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.773+0000 7fd29d7fa700 1 -- 172.21.15.121:0/1529913296 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd29400f5f0 con 0x7fd2a0101d00 2024-04-05T21:24:14.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.774+0000 7fd2a685e700 1 -- 172.21.15.121:0/1529913296 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28c005310 con 0x7fd2a0101d00 2024-04-05T21:24:14.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.775+0000 7fd29d7fa700 1 -- 172.21.15.121:0/1529913296 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7fd294009a60 con 0x7fd2a0101d00 2024-04-05T21:24:14.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.775+0000 7fd29d7fa700 1 --2- 172.21.15.121:0/1529913296 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fd28806c4e0 0x7fd28806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:14.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.775+0000 7fd29d7fa700 1 -- 172.21.15.121:0/1529913296 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fd29408b790 con 0x7fd2a0101d00 2024-04-05T21:24:14.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.775+0000 7fd29ffff700 1 --2- 172.21.15.121:0/1529913296 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fd28806c4e0 0x7fd28806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:14.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.776+0000 7fd29ffff700 1 --2- 172.21.15.121:0/1529913296 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fd28806c4e0 0x7fd28806e9a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd290009230 tx=0x7fd290019040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:14.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.780+0000 7fd29d7fa700 1 -- 172.21.15.121:0/1529913296 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7fd2940599a0 con 0x7fd2a0101d00 2024-04-05T21:24:14.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.976+0000 7fd2a685e700 1 -- 172.21.15.121:0/1529913296 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fd28c005f60 con 0x7fd2a0101d00 2024-04-05T21:24:14.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.976+0000 7fd29d7fa700 1 -- 172.21.15.121:0/1529913296 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fd29408b020 con 0x7fd2a0101d00 2024-04-05T21:24:14.977 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 32 2024-04-05T21:24:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.978+0000 7fd2a685e700 1 -- 172.21.15.121:0/1529913296 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fd28806c4e0 msgr2=0x7fd28806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.978+0000 7fd2a685e700 1 --2- 172.21.15.121:0/1529913296 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fd28806c4e0 0x7fd28806e9a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd290009230 tx=0x7fd290019040 comp rx=0 tx=0).stop 2024-04-05T21:24:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.978+0000 7fd2a685e700 1 -- 172.21.15.121:0/1529913296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a0101d00 msgr2=0x7fd2a01967b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.978+0000 7fd2a685e700 1 --2- 172.21.15.121:0/1529913296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a0101d00 0x7fd2a01967b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd29400d900 tx=0x7fd29400dcc0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.978+0000 7fd2a685e700 1 -- 172.21.15.121:0/1529913296 shutdown_connections 2024-04-05T21:24:14.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.978+0000 7fd2a685e700 1 --2- 172.21.15.121:0/1529913296 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7fd28806c4e0 0x7fd28806e9a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.978+0000 7fd2a685e700 1 --2- 172.21.15.121:0/1529913296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd2a0100b10 0x7fd2a0196270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.978+0000 7fd2a685e700 1 --2- 172.21.15.121:0/1529913296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a0101d00 0x7fd2a01967b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:14.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.978+0000 7fd2a685e700 1 -- 172.21.15.121:0/1529913296 >> 172.21.15.121:0/1529913296 conn(0x7fd2a00fc060 msgr2=0x7fd2a0104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:14.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.978+0000 7fd2a685e700 1 -- 172.21.15.121:0/1529913296 shutdown_connections 2024-04-05T21:24:14.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:14.978+0000 7fd2a685e700 1 -- 172.21.15.121:0/1529913296 wait complete. 2024-04-05T21:24:14.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-05T21:24:15.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.125+0000 7ffb7e746700 1 -- 172.21.15.121:0/258386360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb78101d00 msgr2=0x7ffb78102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:15.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.125+0000 7ffb7e746700 1 --2- 172.21.15.121:0/258386360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb78101d00 0x7ffb78102160 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009910 tx=0x7ffb60009c20 comp rx=0 tx=0).stop 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.125+0000 7ffb7e746700 1 -- 172.21.15.121:0/258386360 shutdown_connections 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.125+0000 7ffb7e746700 1 --2- 172.21.15.121:0/258386360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb78101d00 0x7ffb78102160 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.125+0000 7ffb7e746700 1 --2- 172.21.15.121:0/258386360 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb78100b10 0x7ffb78100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.125+0000 7ffb7e746700 1 -- 172.21.15.121:0/258386360 >> 172.21.15.121:0/258386360 conn(0x7ffb780fc0a0 msgr2=0x7ffb780fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.125+0000 7ffb7e746700 1 -- 172.21.15.121:0/258386360 shutdown_connections 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.125+0000 7ffb7e746700 1 -- 172.21.15.121:0/258386360 wait complete. 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.125+0000 7ffb7e746700 1 Processor -- start 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.125+0000 7ffb7e746700 1 -- start start 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb7e746700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb78100b10 0x7ffb78074b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb7e746700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb78101d00 0x7ffb78073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb7e746700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb780736c0 con 0x7ffb78101d00 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb7e746700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb78073800 con 0x7ffb78100b10 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb6ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb78101d00 0x7ffb78073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:15.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb6ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb78101d00 0x7ffb78073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51112/0 (socket says 172.21.15.121:51112) 2024-04-05T21:24:15.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb6ffff700 1 -- 172.21.15.121:0/2239817981 learned_addr learned my addr 172.21.15.121:0/2239817981 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:15.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb77fff700 1 --2- 172.21.15.121:0/2239817981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb78100b10 0x7ffb78074b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:15.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb6ffff700 1 -- 172.21.15.121:0/2239817981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb78100b10 msgr2=0x7ffb78074b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:15.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb6ffff700 1 --2- 172.21.15.121:0/2239817981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb78100b10 0x7ffb78074b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb6ffff700 1 -- 172.21.15.121:0/2239817981 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb600095f0 con 0x7ffb78101d00 2024-04-05T21:24:15.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb77fff700 1 --2- 172.21.15.121:0/2239817981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb78100b10 0x7ffb78074b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:24:15.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb6ffff700 1 --2- 172.21.15.121:0/2239817981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb78101d00 0x7ffb78073180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000b5c0 tx=0x7ffb6000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:15.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb75ffb700 1 -- 172.21.15.121:0/2239817981 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb6001d070 con 0x7ffb78101d00 2024-04-05T21:24:15.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb75ffb700 1 -- 172.21.15.121:0/2239817981 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7ffb60022470 con 0x7ffb78101d00 2024-04-05T21:24:15.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb7e746700 1 -- 172.21.15.121:0/2239817981 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb78073a80 con 0x7ffb78101d00 2024-04-05T21:24:15.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb75ffb700 1 -- 172.21.15.121:0/2239817981 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb6000f670 con 0x7ffb78101d00 2024-04-05T21:24:15.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.126+0000 7ffb7e746700 1 -- 172.21.15.121:0/2239817981 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb78074000 con 0x7ffb78101d00 2024-04-05T21:24:15.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.127+0000 7ffb7e746700 1 -- 172.21.15.121:0/2239817981 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb78109540 con 0x7ffb78101d00 2024-04-05T21:24:15.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.128+0000 7ffb75ffb700 1 -- 172.21.15.121:0/2239817981 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ffb60004af0 con 0x7ffb78101d00 2024-04-05T21:24:15.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.128+0000 7ffb75ffb700 1 --2- 172.21.15.121:0/2239817981 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffb6406c4e0 0x7ffb6406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:15.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.128+0000 7ffb75ffb700 1 -- 172.21.15.121:0/2239817981 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ffb6008cf00 con 0x7ffb78101d00 2024-04-05T21:24:15.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.128+0000 7ffb77fff700 1 --2- 172.21.15.121:0/2239817981 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffb6406c4e0 0x7ffb6406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:15.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.129+0000 7ffb77fff700 1 --2- 172.21.15.121:0/2239817981 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffb6406c4e0 0x7ffb6406e9a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffb68005fd0 tx=0x7ffb68005dc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:15.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.133+0000 7ffb75ffb700 1 -- 172.21.15.121:0/2239817981 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ffb6005b1c0 con 0x7ffb78101d00 2024-04-05T21:24:15.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:15 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/289663105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T21:24:15.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:15 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2835159078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T21:24:15.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:15 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1529913296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T21:24:15.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.330+0000 7ffb7e746700 1 -- 172.21.15.121:0/2239817981 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7ffb7819e6e0 con 0x7ffb78101d00 2024-04-05T21:24:15.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.331+0000 7ffb75ffb700 1 -- 172.21.15.121:0/2239817981 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7ffb6005ad50 con 0x7ffb78101d00 2024-04-05T21:24:15.332 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 32 2024-04-05T21:24:15.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.333+0000 7ffb7e746700 1 -- 172.21.15.121:0/2239817981 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffb6406c4e0 msgr2=0x7ffb6406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:15.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.333+0000 7ffb7e746700 1 --2- 172.21.15.121:0/2239817981 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffb6406c4e0 0x7ffb6406e9a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffb68005fd0 tx=0x7ffb68005dc0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.333+0000 7ffb7e746700 1 -- 172.21.15.121:0/2239817981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb78101d00 msgr2=0x7ffb78073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:15.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.333+0000 7ffb7e746700 1 --2- 172.21.15.121:0/2239817981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb78101d00 0x7ffb78073180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000b5c0 tx=0x7ffb6000be20 comp rx=0 tx=0).stop 2024-04-05T21:24:15.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.333+0000 7ffb7e746700 1 -- 172.21.15.121:0/2239817981 shutdown_connections 2024-04-05T21:24:15.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.333+0000 7ffb7e746700 1 --2- 172.21.15.121:0/2239817981 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ffb6406c4e0 0x7ffb6406e9a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.333+0000 7ffb7e746700 1 --2- 172.21.15.121:0/2239817981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb78100b10 0x7ffb78074b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.333+0000 7ffb7e746700 1 --2- 172.21.15.121:0/2239817981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb78101d00 0x7ffb78073180 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.333+0000 7ffb7e746700 1 -- 172.21.15.121:0/2239817981 >> 172.21.15.121:0/2239817981 conn(0x7ffb780fc0a0 msgr2=0x7ffb78104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:15.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.333+0000 7ffb7e746700 1 -- 172.21.15.121:0/2239817981 shutdown_connections 2024-04-05T21:24:15.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.333+0000 7ffb7e746700 1 -- 172.21.15.121:0/2239817981 wait complete. 2024-04-05T21:24:15.345 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:15 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/289663105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T21:24:15.345 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:15 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2835159078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T21:24:15.345 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:15 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1529913296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T21:24:15.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T21:24:15.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.468+0000 7f39b2368700 1 -- 172.21.15.121:0/4025708584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac0ff9e0 msgr2=0x7f39ac0ffe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:15.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.468+0000 7f39b2368700 1 --2- 172.21.15.121:0/4025708584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac0ff9e0 0x7f39ac0ffe00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f399c009910 tx=0x7f399c009c20 comp rx=0 tx=0).stop 2024-04-05T21:24:15.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.468+0000 7f39b2368700 1 -- 172.21.15.121:0/4025708584 shutdown_connections 2024-04-05T21:24:15.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.468+0000 7f39b2368700 1 --2- 172.21.15.121:0/4025708584 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39ac100bd0 0x7f39ac101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.468+0000 7f39b2368700 1 --2- 172.21.15.121:0/4025708584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac0ff9e0 0x7f39ac0ffe00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.468+0000 7f39b2368700 1 -- 172.21.15.121:0/4025708584 >> 172.21.15.121:0/4025708584 conn(0x7f39ac0faf30 msgr2=0x7f39ac0fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:15.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.468+0000 7f39b2368700 1 -- 172.21.15.121:0/4025708584 shutdown_connections 2024-04-05T21:24:15.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.468+0000 7f39b2368700 1 -- 172.21.15.121:0/4025708584 wait complete. 2024-04-05T21:24:15.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.469+0000 7f39b2368700 1 Processor -- start 2024-04-05T21:24:15.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.469+0000 7f39b2368700 1 -- start start 2024-04-05T21:24:15.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.470+0000 7f39b2368700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39ac0ff9e0 0x7f39ac195080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:15.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.470+0000 7f39b2368700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac100bd0 0x7f39ac1955c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:15.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.470+0000 7f39b2368700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac195be0 con 0x7f39ac100bd0 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.470+0000 7f39b2368700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ac195d20 con 0x7f39ac0ff9e0 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.470+0000 7f39ab7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac100bd0 0x7f39ac1955c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.470+0000 7f39ab7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac100bd0 0x7f39ac1955c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51120/0 (socket says 172.21.15.121:51120) 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.470+0000 7f39ab7fe700 1 -- 172.21.15.121:0/3504238935 learned_addr learned my addr 172.21.15.121:0/3504238935 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.470+0000 7f39abfff700 1 --2- 172.21.15.121:0/3504238935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39ac0ff9e0 0x7f39ac195080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.470+0000 7f39ab7fe700 1 -- 172.21.15.121:0/3504238935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39ac0ff9e0 msgr2=0x7f39ac195080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.470+0000 7f39ab7fe700 1 --2- 172.21.15.121:0/3504238935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39ac0ff9e0 0x7f39ac195080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.470+0000 7f39ab7fe700 1 -- 172.21.15.121:0/3504238935 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c0095f0 con 0x7f39ac100bd0 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.470+0000 7f39ab7fe700 1 --2- 172.21.15.121:0/3504238935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac100bd0 0x7f39ac1955c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f39a000c8a0 tx=0x7f39a000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.471+0000 7f39a97fa700 1 -- 172.21.15.121:0/3504238935 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39a00078e0 con 0x7f39ac100bd0 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.471+0000 7f39a97fa700 1 -- 172.21.15.121:0/3504238935 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f39a000f450 con 0x7f39ac100bd0 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.471+0000 7f39b2368700 1 -- 172.21.15.121:0/3504238935 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39ac19a7c0 con 0x7f39ac100bd0 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.471+0000 7f39a97fa700 1 -- 172.21.15.121:0/3504238935 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39a000e5e0 con 0x7f39ac100bd0 2024-04-05T21:24:15.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.471+0000 7f39b2368700 1 -- 172.21.15.121:0/3504238935 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ac19ada0 con 0x7f39ac100bd0 2024-04-05T21:24:15.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.471+0000 7f39b2368700 1 -- 172.21.15.121:0/3504238935 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39ac108410 con 0x7f39ac100bd0 2024-04-05T21:24:15.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.472+0000 7f39a97fa700 1 -- 172.21.15.121:0/3504238935 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f39a000f5c0 con 0x7f39ac100bd0 2024-04-05T21:24:15.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.472+0000 7f39a97fa700 1 --2- 172.21.15.121:0/3504238935 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f399406c5a0 0x7f399406ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:15.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.473+0000 7f39a97fa700 1 -- 172.21.15.121:0/3504238935 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f39a008b530 con 0x7f39ac100bd0 2024-04-05T21:24:15.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.473+0000 7f39abfff700 1 --2- 172.21.15.121:0/3504238935 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f399406c5a0 0x7f399406ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:15.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.473+0000 7f39abfff700 1 --2- 172.21.15.121:0/3504238935 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f399406c5a0 0x7f399406ea60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f399c006010 tx=0x7f399c00b540 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:15.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.477+0000 7f39a97fa700 1 -- 172.21.15.121:0/3504238935 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f39a00596a0 con 0x7f39ac100bd0 2024-04-05T21:24:15.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.735+0000 7f39b2368700 1 -- 172.21.15.121:0/3504238935 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39ac19b440 con 0x7f39ac100bd0 2024-04-05T21:24:15.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.736+0000 7f39a97fa700 1 -- 172.21.15.121:0/3504238935 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4779 (secure 0 0 0) 0x7f39a0059230 con 0x7f39ac100bd0 2024-04-05T21:24:15.737 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:24:15.737 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:24:04.283403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14476,"mds_1":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14476":{"gid":14476,"name":"cephfs.smithi121.ifoxmw","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.121:6829/3760584056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3760584056},{"type":"v1","addr":"172.21.15.121:6829","nonce":3760584056}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14488":{"gid":14488,"name":"cephfs.smithi121.gxheqt","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6827/4279084130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":4279084130},{"type":"v1","addr":"172.21.15.121:6827","nonce":4279084130}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24255":{"gid":24255,"name":"cephfs.smithi188.bjmisu","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.188:6825/213881583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":213881583},{"type":"v1","addr":"172.21.15.188:6825","nonce":213881583}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24265":{"gid":24265,"name":"cephfs.smithi188.latnft","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.188:6827/3730442150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3730442150},{"type":"v1","addr":"172.21.15.188:6827","nonce":3730442150}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T21:24:15.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.738+0000 7f39b2368700 1 -- 172.21.15.121:0/3504238935 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f399406c5a0 msgr2=0x7f399406ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:15.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.738+0000 7f39b2368700 1 --2- 172.21.15.121:0/3504238935 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f399406c5a0 0x7f399406ea60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f399c006010 tx=0x7f399c00b540 comp rx=0 tx=0).stop 2024-04-05T21:24:15.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.738+0000 7f39b2368700 1 -- 172.21.15.121:0/3504238935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac100bd0 msgr2=0x7f39ac1955c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:15.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.738+0000 7f39b2368700 1 --2- 172.21.15.121:0/3504238935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac100bd0 0x7f39ac1955c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f39a000c8a0 tx=0x7f39a000cbb0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.738+0000 7f39b2368700 1 -- 172.21.15.121:0/3504238935 shutdown_connections 2024-04-05T21:24:15.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.738+0000 7f39b2368700 1 --2- 172.21.15.121:0/3504238935 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f399406c5a0 0x7f399406ea60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.738+0000 7f39b2368700 1 --2- 172.21.15.121:0/3504238935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f39ac0ff9e0 0x7f39ac195080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.738+0000 7f39b2368700 1 --2- 172.21.15.121:0/3504238935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39ac100bd0 0x7f39ac1955c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:15.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.738+0000 7f39b2368700 1 -- 172.21.15.121:0/3504238935 >> 172.21.15.121:0/3504238935 conn(0x7f39ac0faf30 msgr2=0x7f39ac103e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:15.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.738+0000 7f39b2368700 1 -- 172.21.15.121:0/3504238935 shutdown_connections 2024-04-05T21:24:15.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:15.738+0000 7f39b2368700 1 -- 172.21.15.121:0/3504238935 wait complete. 2024-04-05T21:24:15.740 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 18 2024-04-05T21:24:15.751 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 50} 2024-04-05T21:24:15.751 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-05T21:24:15.775 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-05T21:24:15.775 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-05T21:24:15.776 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-05T21:24:15.776 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-05T21:24:15.776 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-05T21:24:15.776 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-05T21:24:15.776 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-05T21:24:15.776 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-05T21:24:15.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:15.776 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-04-05T21:24:15.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:15.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link delete ceph-brx 2024-04-05T21:24:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:Cannot find device "ceph-brx" 2024-04-05T21:24:15.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T21:24:15.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:15.870 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-04-05T21:24:15.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:15.893 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link delete ceph-brx 2024-04-05T21:24:15.973 INFO:teuthology.orchestra.run.smithi188.stderr:Cannot find device "ceph-brx" 2024-04-05T21:24:15.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T21:24:15.975 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-05T21:24:15.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph fs ls 2024-04-05T21:24:16.336 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:24:16.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:16 smithi188 ceph-mon[49715]: pgmap v209: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-05T21:24:16.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:16 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2239817981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T21:24:16.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:16 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/3504238935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T21:24:16.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:16 smithi121 ceph-mon[39723]: pgmap v209: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 9.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-05T21:24:16.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:16 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2239817981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T21:24:16.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:16 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3504238935' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T21:24:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:17 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:17 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:17 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:24:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:17 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:24:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:17 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:17 smithi188 ceph-mon[49715]: pgmap v210: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-05T21:24:18.115 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:17 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:18.115 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:17 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:18.115 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:17 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:24:18.116 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:17 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:24:18.116 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:17 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:24:18.116 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:17 smithi121 ceph-mon[39723]: pgmap v210: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-05T21:24:18.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.428+0000 7f27e1f5e700 1 -- 172.21.15.121:0/154122686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27dc0ffbd0 msgr2=0x7f27dc100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:18.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.428+0000 7f27e1f5e700 1 --2- 172.21.15.121:0/154122686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27dc0ffbd0 0x7f27dc100050 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f27d0009910 tx=0x7f27d0009c20 comp rx=0 tx=0).stop 2024-04-05T21:24:18.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.428+0000 7f27e1f5e700 1 -- 172.21.15.121:0/154122686 shutdown_connections 2024-04-05T21:24:18.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.428+0000 7f27e1f5e700 1 --2- 172.21.15.121:0/154122686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27dc0ffbd0 0x7f27dc100050 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:18.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.428+0000 7f27e1f5e700 1 --2- 172.21.15.121:0/154122686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27dc105a40 0x7f27dc105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:18.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.428+0000 7f27e1f5e700 1 -- 172.21.15.121:0/154122686 >> 172.21.15.121:0/154122686 conn(0x7f27dc0755a0 msgr2=0x7f27dc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:18.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.428+0000 7f27e1f5e700 1 -- 172.21.15.121:0/154122686 shutdown_connections 2024-04-05T21:24:18.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.428+0000 7f27e1f5e700 1 -- 172.21.15.121:0/154122686 wait complete. 2024-04-05T21:24:18.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.429+0000 7f27e1f5e700 1 Processor -- start 2024-04-05T21:24:18.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.429+0000 7f27e1f5e700 1 -- start start 2024-04-05T21:24:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.429+0000 7f27e1f5e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27dc0ffbd0 0x7f27dc101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.429+0000 7f27e1f5e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27dc105a40 0x7f27dc101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.429+0000 7f27e1f5e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc102090 con 0x7f27dc0ffbd0 2024-04-05T21:24:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.429+0000 7f27e1f5e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc1021d0 con 0x7f27dc105a40 2024-04-05T21:24:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.429+0000 7f27db7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27dc0ffbd0 0x7f27dc101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.429+0000 7f27db7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27dc0ffbd0 0x7f27dc101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51134/0 (socket says 172.21.15.121:51134) 2024-04-05T21:24:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.429+0000 7f27db7fe700 1 -- 172.21.15.121:0/2221716417 learned_addr learned my addr 172.21.15.121:0/2221716417 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.430+0000 7f27db7fe700 1 -- 172.21.15.121:0/2221716417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27dc105a40 msgr2=0x7f27dc101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.430+0000 7f27db7fe700 1 --2- 172.21.15.121:0/2221716417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27dc105a40 0x7f27dc101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.430+0000 7f27db7fe700 1 -- 172.21.15.121:0/2221716417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d00095f0 con 0x7f27dc0ffbd0 2024-04-05T21:24:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.430+0000 7f27db7fe700 1 --2- 172.21.15.121:0/2221716417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27dc0ffbd0 0x7f27dc101610 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00d8d0 tx=0x7f27cc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:18.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.430+0000 7f27d8ff9700 1 -- 172.21.15.121:0/2221716417 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27cc009940 con 0x7f27dc0ffbd0 2024-04-05T21:24:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.430+0000 7f27e1f5e700 1 -- 172.21.15.121:0/2221716417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27dc1a2910 con 0x7f27dc0ffbd0 2024-04-05T21:24:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.430+0000 7f27d8ff9700 1 -- 172.21.15.121:0/2221716417 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f27cc010460 con 0x7f27dc0ffbd0 2024-04-05T21:24:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.430+0000 7f27d8ff9700 1 -- 172.21.15.121:0/2221716417 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27cc00f5f0 con 0x7f27dc0ffbd0 2024-04-05T21:24:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.430+0000 7f27e1f5e700 1 -- 172.21.15.121:0/2221716417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27dc1a2d20 con 0x7f27dc0ffbd0 2024-04-05T21:24:18.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.431+0000 7f27e1f5e700 1 -- 172.21.15.121:0/2221716417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27dc108fe0 con 0x7f27dc0ffbd0 2024-04-05T21:24:18.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.431+0000 7f27d8ff9700 1 -- 172.21.15.121:0/2221716417 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f27cc0105d0 con 0x7f27dc0ffbd0 2024-04-05T21:24:18.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.432+0000 7f27d8ff9700 1 --2- 172.21.15.121:0/2221716417 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f27c406c4e0 0x7f27c406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:18.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.432+0000 7f27daffd700 1 --2- 172.21.15.121:0/2221716417 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f27c406c4e0 0x7f27c406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:18.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.432+0000 7f27d8ff9700 1 -- 172.21.15.121:0/2221716417 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f27cc08b8c0 con 0x7f27dc0ffbd0 2024-04-05T21:24:18.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.432+0000 7f27daffd700 1 --2- 172.21.15.121:0/2221716417 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f27c406c4e0 0x7f27c406e9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f27d000b5c0 tx=0x7f27d0005960 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.436+0000 7f27d8ff9700 1 -- 172.21.15.121:0/2221716417 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f27cc056040 con 0x7f27dc0ffbd0 2024-04-05T21:24:18.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.667+0000 7f27e1f5e700 1 -- 172.21.15.121:0/2221716417 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f27dc1089c0 con 0x7f27dc0ffbd0 2024-04-05T21:24:18.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.667+0000 7f27d8ff9700 1 -- 172.21.15.121:0/2221716417 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f27cc010880 con 0x7f27dc0ffbd0 2024-04-05T21:24:18.668 INFO:teuthology.orchestra.run.smithi121.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-05T21:24:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.669+0000 7f27e1f5e700 1 -- 172.21.15.121:0/2221716417 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f27c406c4e0 msgr2=0x7f27c406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.669+0000 7f27e1f5e700 1 --2- 172.21.15.121:0/2221716417 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f27c406c4e0 0x7f27c406e9a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f27d000b5c0 tx=0x7f27d0005960 comp rx=0 tx=0).stop 2024-04-05T21:24:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.669+0000 7f27e1f5e700 1 -- 172.21.15.121:0/2221716417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27dc0ffbd0 msgr2=0x7f27dc101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.669+0000 7f27e1f5e700 1 --2- 172.21.15.121:0/2221716417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27dc0ffbd0 0x7f27dc101610 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f27cc00d8d0 tx=0x7f27cc00dc90 comp rx=0 tx=0).stop 2024-04-05T21:24:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.669+0000 7f27e1f5e700 1 -- 172.21.15.121:0/2221716417 shutdown_connections 2024-04-05T21:24:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.669+0000 7f27e1f5e700 1 --2- 172.21.15.121:0/2221716417 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f27c406c4e0 0x7f27c406e9a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.669+0000 7f27e1f5e700 1 --2- 172.21.15.121:0/2221716417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27dc0ffbd0 0x7f27dc101610 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.669+0000 7f27e1f5e700 1 --2- 172.21.15.121:0/2221716417 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f27dc105a40 0x7f27dc101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.669+0000 7f27e1f5e700 1 -- 172.21.15.121:0/2221716417 >> 172.21.15.121:0/2221716417 conn(0x7f27dc0755a0 msgr2=0x7f27dc103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.669+0000 7f27e1f5e700 1 -- 172.21.15.121:0/2221716417 shutdown_connections 2024-04-05T21:24:18.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:18.669+0000 7f27e1f5e700 1 -- 172.21.15.121:0/2221716417 wait complete. 2024-04-05T21:24:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:18 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2221716417' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-05T21:24:19.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:18 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2221716417' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-05T21:24:19.886 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-05T21:24:19.886 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-05T21:24:19.886 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi121.front.sepia.ceph.com 2024-04-05T21:24:19.887 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-05T21:24:19.887 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-05T21:24:19.887 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-05T21:24:19.887 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-05T21:24:19.887 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-05T21:24:19.887 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-05T21:24:19.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:19.887 DEBUG:teuthology.orchestra.run.smithi121:> ip addr 2024-04-05T21:24:19.906 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-05T21:24:19.906 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-05T21:24:19.906 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:82 brd ff:ff:ff:ff:ff:ff 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f0 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:83 brd ff:ff:ff:ff:ff:ff 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f1 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a2 brd ff:ff:ff:ff:ff:ff 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a3 brd ff:ff:ff:ff:ff:ff 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout: inet 172.21.15.121/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-05T21:24:19.907 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft 42110sec preferred_lft 42110sec 2024-04-05T21:24:19.908 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::ec4:7aff:fe8f:cba3/64 scope link 2024-04-05T21:24:19.908 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-05T21:24:19.908 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-05T21:24:19.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:19.908 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2024-04-05T21:24:19.908 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-05T21:24:19.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add name ceph-brx type bridge 2024-04-05T21:24:19.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip addr flush dev ceph-brx 2024-04-05T21:24:19.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set ceph-brx up 2024-04-05T21:24:19.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-05T21:24:19.909 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-05T21:24:20.052 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:19 smithi121 ceph-mon[39723]: pgmap v211: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.2 KiB/s rd, 6 op/s 2024-04-05T21:24:20.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:20.089 DEBUG:teuthology.orchestra.run.smithi121:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-05T21:24:20.115 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-04-05T21:24:20.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:20.117 DEBUG:teuthology.orchestra.run.smithi121:> route 2024-04-05T21:24:20.206 INFO:teuthology.orchestra.run.smithi121.stdout:Kernel IP routing table 2024-04-05T21:24:20.206 INFO:teuthology.orchestra.run.smithi121.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-05T21:24:20.206 INFO:teuthology.orchestra.run.smithi121.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-05T21:24:20.206 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-05T21:24:20.206 INFO:teuthology.orchestra.run.smithi121.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-05T21:24:20.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:20.207 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2024-04-05T21:24:20.207 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-05T21:24:20.207 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-05T21:24:20.207 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-05T21:24:20.207 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-05T21:24:20.207 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-05T21:24:20.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:19 smithi188 ceph-mon[49715]: pgmap v211: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 5.2 KiB/s rd, 6 op/s 2024-04-05T21:24:20.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:20.450 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-04-05T21:24:20.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:20.469 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list-id 2024-04-05T21:24:20.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:20.527 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2024-04-05T21:24:20.527 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-05T21:24:20.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-05T21:24:20.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-05T21:24:20.527 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-05T21:24:20.622 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-05T21:24:20.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:20.622 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2024-04-05T21:24:20.622 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-05T21:24:20.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-05T21:24:20.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-05T21:24:20.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-05T21:24:20.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-05T21:24:20.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-05T21:24:20.622 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-05T21:24:20.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:20.823 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2024-04-05T21:24:20.823 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-05T21:24:20.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set brx.0 up 2024-04-05T21:24:20.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set dev brx.0 master ceph-brx 2024-04-05T21:24:20.824 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-05T21:24:20.920 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-05T21:24:20.920 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-05T21:24:20.921 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-05T21:24:20.938 INFO:teuthology.orchestra.run.smithi121.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-05T21:24:20.939 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-05T21:24:20.939 DEBUG:teuthology.orchestra.run.smithi121:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-05T21:24:20.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2024-04-05T21:24:21.065 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-05T21:24:21.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-05T21:24:21.131 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-05T21:24:21.132 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-05T21:24:21.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:21.132 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2024-04-05T21:24:21.190 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-05T21:24:21.191 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-05T21:24:21.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2024-04-05T21:24:21.259 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-05T21:24:21.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-05T21:24:21.329 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-05T21:24:21.330 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-05T21:24:21.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:21.330 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2024-04-05T21:24:21.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2024-04-05T21:24:21.339+0000 7f8f7b49c5c0 -1 init, newargv = 0x560bb41f9300 newargc=15 2024-04-05T21:24:21.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[102066]: starting ceph client 2024-04-05T21:24:21.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[102066]: starting fuse 2024-04-05T21:24:21.358 INFO:teuthology.orchestra.run.smithi121.stdout:107 2024-04-05T21:24:21.359 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [107] 2024-04-05T21:24:21.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-04-05T21:24:21.359 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2024-04-05T21:24:21.359 DEBUG:teuthology.orchestra.run.smithi121:> import re 2024-04-05T21:24:21.359 DEBUG:teuthology.orchestra.run.smithi121:> import os 2024-04-05T21:24:21.359 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2024-04-05T21:24:21.359 DEBUG:teuthology.orchestra.run.smithi121:> 2024-04-05T21:24:21.359 DEBUG:teuthology.orchestra.run.smithi121:> def _find_admin_socket(client_name): 2024-04-05T21:24:21.359 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-05T21:24:21.359 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2024-04-05T21:24:21.359 DEBUG:teuthology.orchestra.run.smithi121:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> contents = proc_f.read() 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> if mountpoint in contents: 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> return f 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> print(_find_admin_socket("client.0")) 2024-04-05T21:24:21.360 DEBUG:teuthology.orchestra.run.smithi121:> ' 2024-04-05T21:24:21.481 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.102066.asok 2024-04-05T21:24:21.664 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.102066.asok 2024-04-05T21:24:21.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:21.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.102066.asok status 2024-04-05T21:24:21.821 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:24:21.822 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2024-04-05T21:24:21.822 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "7fe91d5d5842e04be3b4f514d6dd990c54b29c76", 2024-04-05T21:24:21.822 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-05T21:24:21.822 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2024-04-05T21:24:21.822 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2024-04-05T21:24:21.822 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-05T21:24:21.822 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "102066", 2024-04-05T21:24:21.822 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2024-04-05T21:24:21.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:24:21.822 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2024-04-05T21:24:21.822 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2024-04-05T21:24:21.822 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 14528, 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 14528 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "192.168.0.1:0", 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1232807809 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-04-05T21:24:21.823 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "192.168.0.1:0", 2024-04-05T21:24:21.824 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1232807809 2024-04-05T21:24:21.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:24:21.824 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.14528 192.168.0.1:0/1232807809", 2024-04-05T21:24:21.824 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "192.168.0.1:0/1232807809", 2024-04-05T21:24:21.824 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2024-04-05T21:24:21.824 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 18, 2024-04-05T21:24:21.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 46, 2024-04-05T21:24:21.824 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2024-04-05T21:24:21.824 INFO:teuthology.orchestra.run.smithi121.stdout: "blocklisted": false, 2024-04-05T21:24:21.824 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2024-04-05T21:24:21.824 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:24:21.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- ceph fs ls 2024-04-05T21:24:22.055 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:24:22.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:22 smithi188 ceph-mon[49715]: pgmap v212: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:24:22.565 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:22 smithi121 ceph-mon[39723]: pgmap v212: 65 pgs: 65 active+clean; 580 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:24:23.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.979+0000 7f8ee7d19700 1 -- 172.21.15.121:0/531532068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee0101750 msgr2=0x7f8ee01057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:23.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.979+0000 7f8ee7d19700 1 --2- 172.21.15.121:0/531532068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee0101750 0x7f8ee01057a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009910 tx=0x7f8ed0009c20 comp rx=0 tx=0).stop 2024-04-05T21:24:23.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.979+0000 7f8ee7d19700 1 -- 172.21.15.121:0/531532068 shutdown_connections 2024-04-05T21:24:23.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.979+0000 7f8ee7d19700 1 --2- 172.21.15.121:0/531532068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee0101750 0x7f8ee01057a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:23.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.979+0000 7f8ee7d19700 1 --2- 172.21.15.121:0/531532068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ee0100da0 0x7f8ee0101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:23.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.979+0000 7f8ee7d19700 1 -- 172.21.15.121:0/531532068 >> 172.21.15.121:0/531532068 conn(0x7f8ee00fc630 msgr2=0x7f8ee00fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:23.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.979+0000 7f8ee7d19700 1 -- 172.21.15.121:0/531532068 shutdown_connections 2024-04-05T21:24:23.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.979+0000 7f8ee7d19700 1 -- 172.21.15.121:0/531532068 wait complete. 2024-04-05T21:24:23.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.980+0000 7f8ee7d19700 1 Processor -- start 2024-04-05T21:24:23.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.980+0000 7f8ee7d19700 1 -- start start 2024-04-05T21:24:23.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.980+0000 7f8ee7d19700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ee0100da0 0x7f8ee0198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:23.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.980+0000 7f8ee7d19700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee0101750 0x7f8ee0198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.980+0000 7f8ee7d19700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee0199520 con 0x7f8ee0101750 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.980+0000 7f8ee7d19700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee019d330 con 0x7f8ee0100da0 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.980+0000 7f8ee52b4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee0101750 0x7f8ee0198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.980+0000 7f8ee52b4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee0101750 0x7f8ee0198e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51146/0 (socket says 172.21.15.121:51146) 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.980+0000 7f8ee52b4700 1 -- 172.21.15.121:0/1974950097 learned_addr learned my addr 172.21.15.121:0/1974950097 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.981+0000 7f8ee5ab5700 1 --2- 172.21.15.121:0/1974950097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ee0100da0 0x7f8ee0198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.981+0000 7f8ee52b4700 1 -- 172.21.15.121:0/1974950097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ee0100da0 msgr2=0x7f8ee0198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.981+0000 7f8ee52b4700 1 --2- 172.21.15.121:0/1974950097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ee0100da0 0x7f8ee0198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.981+0000 7f8ee52b4700 1 -- 172.21.15.121:0/1974950097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed00095f0 con 0x7f8ee0101750 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.981+0000 7f8ee52b4700 1 --2- 172.21.15.121:0/1974950097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee0101750 0x7f8ee0198e90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0005f50 tx=0x7f8ed000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.981+0000 7f8ed6ffd700 1 -- 172.21.15.121:0/1974950097 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ed001d070 con 0x7f8ee0101750 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.981+0000 7f8ed6ffd700 1 -- 172.21.15.121:0/1974950097 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f8ed000f460 con 0x7f8ee0101750 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.981+0000 7f8ee7d19700 1 -- 172.21.15.121:0/1974950097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee019d5b0 con 0x7f8ee0101750 2024-04-05T21:24:23.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.981+0000 7f8ed6ffd700 1 -- 172.21.15.121:0/1974950097 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ed0021640 con 0x7f8ee0101750 2024-04-05T21:24:23.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.981+0000 7f8ee7d19700 1 -- 172.21.15.121:0/1974950097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee019daa0 con 0x7f8ee0101750 2024-04-05T21:24:23.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.982+0000 7f8ee7d19700 1 -- 172.21.15.121:0/1974950097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee0104f20 con 0x7f8ee0101750 2024-04-05T21:24:23.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.983+0000 7f8ed6ffd700 1 -- 172.21.15.121:0/1974950097 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f8ed000f9e0 con 0x7f8ee0101750 2024-04-05T21:24:23.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.983+0000 7f8ed6ffd700 1 --2- 172.21.15.121:0/1974950097 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f8ecc06c4e0 0x7f8ecc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:23.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.983+0000 7f8ed6ffd700 1 -- 172.21.15.121:0/1974950097 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f8ed008cba0 con 0x7f8ee0101750 2024-04-05T21:24:23.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.983+0000 7f8ee5ab5700 1 --2- 172.21.15.121:0/1974950097 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f8ecc06c4e0 0x7f8ecc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:23.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.984+0000 7f8ee5ab5700 1 --2- 172.21.15.121:0/1974950097 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f8ecc06c4e0 0x7f8ecc06e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8edc006fd0 tx=0x7f8edc008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:23.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:23.987+0000 7f8ed6ffd700 1 -- 172.21.15.121:0/1974950097 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8ed005adb0 con 0x7f8ee0101750 2024-04-05T21:24:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:24.221+0000 7f8ee7d19700 1 -- 172.21.15.121:0/1974950097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8ee004ea90 con 0x7f8ee0101750 2024-04-05T21:24:24.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:24.221+0000 7f8ed6ffd700 1 -- 172.21.15.121:0/1974950097 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f8ed005a940 con 0x7f8ee0101750 2024-04-05T21:24:24.223 INFO:teuthology.orchestra.run.smithi121.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-05T21:24:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:24.224+0000 7f8ee7d19700 1 -- 172.21.15.121:0/1974950097 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f8ecc06c4e0 msgr2=0x7f8ecc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:24.224+0000 7f8ee7d19700 1 --2- 172.21.15.121:0/1974950097 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f8ecc06c4e0 0x7f8ecc06e9a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8edc006fd0 tx=0x7f8edc008040 comp rx=0 tx=0).stop 2024-04-05T21:24:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:24.224+0000 7f8ee7d19700 1 -- 172.21.15.121:0/1974950097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee0101750 msgr2=0x7f8ee0198e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:24.224+0000 7f8ee7d19700 1 --2- 172.21.15.121:0/1974950097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee0101750 0x7f8ee0198e90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0005f50 tx=0x7f8ed000be20 comp rx=0 tx=0).stop 2024-04-05T21:24:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:24.224+0000 7f8ee7d19700 1 -- 172.21.15.121:0/1974950097 shutdown_connections 2024-04-05T21:24:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:24.224+0000 7f8ee7d19700 1 --2- 172.21.15.121:0/1974950097 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f8ecc06c4e0 0x7f8ecc06e9a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:24.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:24.224+0000 7f8ee7d19700 1 --2- 172.21.15.121:0/1974950097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ee0100da0 0x7f8ee0198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:24.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:24.224+0000 7f8ee7d19700 1 --2- 172.21.15.121:0/1974950097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ee0101750 0x7f8ee0198e90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:24.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:24.224+0000 7f8ee7d19700 1 -- 172.21.15.121:0/1974950097 >> 172.21.15.121:0/1974950097 conn(0x7f8ee00fc630 msgr2=0x7f8ee00fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:24.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:24.224+0000 7f8ee7d19700 1 -- 172.21.15.121:0/1974950097 shutdown_connections 2024-04-05T21:24:24.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:24.224+0000 7f8ee7d19700 1 -- 172.21.15.121:0/1974950097 wait complete. 2024-04-05T21:24:24.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:24 smithi188 ceph-mon[49715]: pgmap v213: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-04-05T21:24:24.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:24 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1974950097' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-05T21:24:24.619 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:24 smithi121 ceph-mon[39723]: pgmap v213: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-04-05T21:24:24.619 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:24 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1974950097' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-05T21:24:25.005 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-05T21:24:25.005 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-05T21:24:25.005 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi188.front.sepia.ceph.com 2024-04-05T21:24:25.005 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-05T21:24:25.005 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-05T21:24:25.005 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-05T21:24:25.005 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-05T21:24:25.005 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-05T21:24:25.005 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-05T21:24:25.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:25.006 DEBUG:teuthology.orchestra.run.smithi188:> ip addr 2024-04-05T21:24:25.028 INFO:teuthology.orchestra.run.smithi188.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-05T21:24:25.028 INFO:teuthology.orchestra.run.smithi188.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-05T21:24:25.028 INFO:teuthology.orchestra.run.smithi188.stdout: inet 127.0.0.1/8 scope host lo 2024-04-05T21:24:25.028 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-04-05T21:24:25.028 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 ::1/128 scope host 2024-04-05T21:24:25.028 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:48 brd ff:ff:ff:ff:ff:ff 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f0 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:d9:37:49 brd ff:ff:ff:ff:ff:ff 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout: altname enp7s0f1 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:38 brd ff:ff:ff:ff:ff:ff 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout: link/ether 0c:c4:7a:88:80:39 brd ff:ff:ff:ff:ff:ff 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout: inet 172.21.15.188/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft 42086sec preferred_lft 42086sec 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout: inet6 fe80::ec4:7aff:fe88:8039/64 scope link 2024-04-05T21:24:25.029 INFO:teuthology.orchestra.run.smithi188.stdout: valid_lft forever preferred_lft forever 2024-04-05T21:24:25.030 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-05T21:24:25.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:25.030 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-04-05T21:24:25.030 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-04-05T21:24:25.030 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link add name ceph-brx type bridge 2024-04-05T21:24:25.030 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip addr flush dev ceph-brx 2024-04-05T21:24:25.030 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set ceph-brx up 2024-04-05T21:24:25.030 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-05T21:24:25.030 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-04-05T21:24:25.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:25.211 DEBUG:teuthology.orchestra.run.smithi188:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-05T21:24:25.240 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-04-05T21:24:25.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:25.242 DEBUG:teuthology.orchestra.run.smithi188:> route 2024-04-05T21:24:25.344 INFO:teuthology.orchestra.run.smithi188.stdout:Kernel IP routing table 2024-04-05T21:24:25.344 INFO:teuthology.orchestra.run.smithi188.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-05T21:24:25.344 INFO:teuthology.orchestra.run.smithi188.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-05T21:24:25.345 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-05T21:24:25.345 INFO:teuthology.orchestra.run.smithi188.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-05T21:24:25.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:25.345 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-04-05T21:24:25.345 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-04-05T21:24:25.345 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-05T21:24:25.345 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-05T21:24:25.345 DEBUG:teuthology.orchestra.run.smithi188:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-05T21:24:25.345 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-04-05T21:24:25.384 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:25 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:24:25.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:25.578 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list 2024-04-05T21:24:25.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:25.636 DEBUG:teuthology.orchestra.run.smithi188:> ip netns list-id 2024-04-05T21:24:25.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:25.694 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-04-05T21:24:25.694 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-04-05T21:24:25.694 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-05T21:24:25.694 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-05T21:24:25.694 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-04-05T21:24:25.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:24:25.788 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-05T21:24:25.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:25.789 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-04-05T21:24:25.789 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-04-05T21:24:25.789 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-05T21:24:25.789 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-05T21:24:25.789 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-05T21:24:25.789 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-05T21:24:25.789 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-05T21:24:25.789 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-04-05T21:24:26.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:26.006 DEBUG:teuthology.orchestra.run.smithi188:> (cd / && exec bash -c ' 2024-04-05T21:24:26.007 DEBUG:teuthology.orchestra.run.smithi188:> set -e 2024-04-05T21:24:26.007 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set brx.0 up 2024-04-05T21:24:26.007 DEBUG:teuthology.orchestra.run.smithi188:> sudo ip link set dev brx.0 master ceph-brx 2024-04-05T21:24:26.007 DEBUG:teuthology.orchestra.run.smithi188:> ') 2024-04-05T21:24:26.103 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-05T21:24:26.104 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-05T21:24:26.104 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-05T21:24:26.121 INFO:teuthology.orchestra.run.smithi188.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-05T21:24:26.121 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-05T21:24:26.121 DEBUG:teuthology.orchestra.run.smithi188:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-05T21:24:26.179 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-04-05T21:24:26.246 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-05T21:24:26.246 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-05T21:24:26.310 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:26 smithi188 ceph-mon[49715]: pgmap v214: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-05T21:24:26.312 INFO:teuthology.orchestra.run.smithi188.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-05T21:24:26.313 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-05T21:24:26.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:26.314 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-04-05T21:24:26.372 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-05T21:24:26.372 DEBUG:teuthology.orchestra.run.smithi188:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-05T21:24:26.414 DEBUG:teuthology.orchestra.run.smithi188:> sudo modprobe fuse 2024-04-05T21:24:26.440 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-05T21:24:26.440 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-04-05T21:24:26.507 INFO:teuthology.orchestra.run.smithi188.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-04-05T21:24:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-04-05T21:24:26.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:26.509 DEBUG:teuthology.orchestra.run.smithi188:> ls /sys/fs/fuse/connections 2024-04-05T21:24:26.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:2024-04-05T21:24:26.524+0000 7f6ca91d05c0 -1 init, newargv = 0x55948119b200 newargc=15 2024-04-05T21:24:26.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[82326]: starting ceph client 2024-04-05T21:24:26.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi188.stderr:ceph-fuse[82326]: starting fuse 2024-04-05T21:24:26.543 INFO:teuthology.orchestra.run.smithi188.stdout:137 2024-04-05T21:24:26.543 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [137] 2024-04-05T21:24:26.543 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-04-05T21:24:26.543 DEBUG:teuthology.orchestra.run.smithi188:> import glob 2024-04-05T21:24:26.543 DEBUG:teuthology.orchestra.run.smithi188:> import re 2024-04-05T21:24:26.543 DEBUG:teuthology.orchestra.run.smithi188:> import os 2024-04-05T21:24:26.543 DEBUG:teuthology.orchestra.run.smithi188:> import subprocess 2024-04-05T21:24:26.543 DEBUG:teuthology.orchestra.run.smithi188:> 2024-04-05T21:24:26.543 DEBUG:teuthology.orchestra.run.smithi188:> def _find_admin_socket(client_name): 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> files = glob.glob(asok_path) 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> # Given a non-glob path, it better be there 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> if "*" not in asok_path: 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> assert(len(files) == 1) 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> return files[0] 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> for f in files: 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> contents = proc_f.read() 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> if mountpoint in contents: 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> return f 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-05T21:24:26.544 DEBUG:teuthology.orchestra.run.smithi188:> 2024-04-05T21:24:26.545 DEBUG:teuthology.orchestra.run.smithi188:> print(_find_admin_socket("client.1")) 2024-04-05T21:24:26.545 DEBUG:teuthology.orchestra.run.smithi188:> ' 2024-04-05T21:24:26.665 INFO:teuthology.orchestra.run.smithi188.stdout:/var/run/ceph/ceph-client.1.82326.asok 2024-04-05T21:24:26.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:26 smithi121 ceph-mon[39723]: pgmap v214: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-05T21:24:26.847 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.82326.asok 2024-04-05T21:24:26.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:26.848 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.82326.asok status 2024-04-05T21:24:27.004 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-05T21:24:27.004 INFO:teuthology.orchestra.run.smithi188.stdout: "metadata": { 2024-04-05T21:24:27.004 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_sha1": "7fe91d5d5842e04be3b4f514d6dd990c54b29c76", 2024-04-05T21:24:27.004 INFO:teuthology.orchestra.run.smithi188.stdout: "ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)", 2024-04-05T21:24:27.004 INFO:teuthology.orchestra.run.smithi188.stdout: "entity_id": "1", 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: "hostname": "smithi188", 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: "pid": "82326", 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: "root": "/" 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_count": 0, 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: "dentry_pinned_count": 0, 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: "id": 24293, 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: "inst": { 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: "name": { 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "client", 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: "num": 24293 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-04-05T21:24:27.005 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 220997234 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: } 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": { 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: "type": "v1", 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: "addr": "192.168.0.1:0", 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: "nonce": 220997234 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: }, 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: "inst_str": "client.24293 192.168.0.1:0/220997234", 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: "addr_str": "192.168.0.1:0/220997234", 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: "inode_count": 1, 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: "mds_epoch": 18, 2024-04-05T21:24:27.006 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch": 46, 2024-04-05T21:24:27.007 INFO:teuthology.orchestra.run.smithi188.stdout: "osd_epoch_barrier": 0, 2024-04-05T21:24:27.007 INFO:teuthology.orchestra.run.smithi188.stdout: "blocklisted": false, 2024-04-05T21:24:27.007 INFO:teuthology.orchestra.run.smithi188.stdout: "fs_name": "cephfs" 2024-04-05T21:24:27.007 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-05T21:24:27.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:27.014 DEBUG:teuthology.orchestra.run.smithi121:> stat --file-system '--printf=%T 2024-04-05T21:24:27.014 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-05T21:24:27.038 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2024-04-05T21:24:27.038 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-05T21:24:27.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:27.038 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-05T21:24:27.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:27.115 DEBUG:teuthology.orchestra.run.smithi188:> stat --file-system '--printf=%T 2024-04-05T21:24:27.115 DEBUG:teuthology.orchestra.run.smithi188:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-05T21:24:27.134 INFO:teuthology.orchestra.run.smithi188.stdout:fuseblk 2024-04-05T21:24:27.134 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-05T21:24:27.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T21:24:27.134 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-05T21:24:28.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:28 smithi188 ceph-mon[49715]: pgmap v215: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-05T21:24:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:28 smithi121 ceph-mon[39723]: pgmap v215: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-05T21:24:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:30 smithi121 ceph-mon[39723]: pgmap v216: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-05T21:24:30.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:30 smithi188 ceph-mon[49715]: pgmap v216: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-04-05T21:24:31.194 INFO:teuthology.run_tasks:Running task print... 2024-04-05T21:24:31.204 INFO:teuthology.task.print:**** done client 2024-04-05T21:24:31.204 INFO:teuthology.run_tasks:Running task parallel... 2024-04-05T21:24:31.256 INFO:teuthology.task.parallel:starting parallel... 2024-04-05T21:24:31.257 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-05T21:24:31.257 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-05T21:24:31.258 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T21:24:31.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-05T21:24:31.258 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-05T21:24:31.258 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-04-05T21:24:31.262 INFO:tasks.workunit:Pulling workunits from ref 8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:24:31.262 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-05T21:24:31.262 INFO:tasks.workunit:timeout=3h 2024-04-05T21:24:31.262 INFO:tasks.workunit:cleanup=True 2024-04-05T21:24:31.263 DEBUG:teuthology.orchestra.run.smithi121:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-05T21:24:31.287 INFO:teuthology.orchestra.run.smithi121.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-05T21:24:31.288 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-05T21:24:31.288 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6bh/107d Inode: 1 Links: 2 2024-04-05T21:24:31.288 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T21:24:31.288 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-05T21:24:31.288 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-05T21:24:31.288 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-05 21:23:48.396246121 +0000 2024-04-05T21:24:31.288 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-05 21:24:27.111944855 +0000 2024-04-05T21:24:31.288 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2024-04-05T21:24:31.288 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-05T21:24:31.288 DEBUG:teuthology.orchestra.run.smithi121:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-05T21:24:31.363 DEBUG:teuthology.orchestra.run.smithi188:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-05T21:24:31.386 INFO:teuthology.orchestra.run.smithi188.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-05T21:24:31.387 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-05T21:24:31.387 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 89h/137d Inode: 1 Links: 3 2024-04-05T21:24:31.387 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-05T21:24:31.387 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-05T21:24:31.387 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-05T21:24:31.387 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-05 21:24:31.356923587 +0000 2024-04-05T21:24:31.387 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-05 21:24:31.356923587 +0000 2024-04-05T21:24:31.387 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: - 2024-04-05T21:24:31.387 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-05T21:24:31.387 DEBUG:teuthology.orchestra.run.smithi188:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-05T21:24:31.467 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:24:31.467 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:24:31.486 INFO:tasks.workunit.client.0.smithi121.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-05T21:24:31.510 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:24:31.527 INFO:tasks.workunit.client.1.smithi188.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-05T21:24:32.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:32 smithi188 ceph-mon[49715]: pgmap v217: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-05T21:24:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:32 smithi121 ceph-mon[39723]: pgmap v217: 65 pgs: 65 active+clean; 581 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-05T21:24:33.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.617+0000 7faeff127700 1 -- 172.21.15.121:0/3947161868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faef8100bd0 msgr2=0x7faef8100ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:33.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.617+0000 7faeff127700 1 --2- 172.21.15.121:0/3947161868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faef8100bd0 0x7faef8100ff0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7faeec009910 tx=0x7faeec009c20 comp rx=0 tx=0).stop 2024-04-05T21:24:33.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.617+0000 7faeff127700 1 -- 172.21.15.121:0/3947161868 shutdown_connections 2024-04-05T21:24:33.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.617+0000 7faeff127700 1 --2- 172.21.15.121:0/3947161868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faef8101dd0 0x7faef8102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:33.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.617+0000 7faeff127700 1 --2- 172.21.15.121:0/3947161868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faef8100bd0 0x7faef8100ff0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:33.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.617+0000 7faeff127700 1 -- 172.21.15.121:0/3947161868 >> 172.21.15.121:0/3947161868 conn(0x7faef80fc170 msgr2=0x7faef80fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:33.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.617+0000 7faeff127700 1 -- 172.21.15.121:0/3947161868 shutdown_connections 2024-04-05T21:24:33.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.617+0000 7faeff127700 1 -- 172.21.15.121:0/3947161868 wait complete. 2024-04-05T21:24:33.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.618+0000 7faeff127700 1 Processor -- start 2024-04-05T21:24:33.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.618+0000 7faeff127700 1 -- start start 2024-04-05T21:24:33.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.618+0000 7faeff127700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faef8100bd0 0x7faef81963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:33.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.618+0000 7faeff127700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faef8101dd0 0x7faef81968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:33.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.618+0000 7faeff127700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef8196f10 con 0x7faef8100bd0 2024-04-05T21:24:33.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.618+0000 7faeff127700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef8197050 con 0x7faef8101dd0 2024-04-05T21:24:33.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.618+0000 7faefcec3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faef8100bd0 0x7faef81963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.618+0000 7faefcec3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faef8100bd0 0x7faef81963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53068/0 (socket says 172.21.15.121:53068) 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.618+0000 7faefcec3700 1 -- 172.21.15.121:0/1796185231 learned_addr learned my addr 172.21.15.121:0/1796185231 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.618+0000 7faef7fff700 1 --2- 172.21.15.121:0/1796185231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faef8101dd0 0x7faef81968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.618+0000 7faefcec3700 1 -- 172.21.15.121:0/1796185231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faef8101dd0 msgr2=0x7faef81968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.618+0000 7faefcec3700 1 --2- 172.21.15.121:0/1796185231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faef8101dd0 0x7faef81968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.618+0000 7faefcec3700 1 -- 172.21.15.121:0/1796185231 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec0095f0 con 0x7faef8100bd0 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.619+0000 7faefcec3700 1 --2- 172.21.15.121:0/1796185231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faef8100bd0 0x7faef81963b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7faeec00bda0 tx=0x7faeec00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.619+0000 7faef5ffb700 1 -- 172.21.15.121:0/1796185231 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeec01d070 con 0x7faef8100bd0 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.619+0000 7faef5ffb700 1 -- 172.21.15.121:0/1796185231 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7faeec022470 con 0x7faef8100bd0 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.619+0000 7faef5ffb700 1 -- 172.21.15.121:0/1796185231 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeec00f690 con 0x7faef8100bd0 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.619+0000 7faeff127700 1 -- 172.21.15.121:0/1796185231 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef819baa0 con 0x7faef8100bd0 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.619+0000 7faeff127700 1 -- 172.21.15.121:0/1796185231 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef819c010 con 0x7faef8100bd0 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.620+0000 7faeff127700 1 -- 172.21.15.121:0/1796185231 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef8109670 con 0x7faef8100bd0 2024-04-05T21:24:33.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.620+0000 7faef5ffb700 1 -- 172.21.15.121:0/1796185231 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7faeec0229c0 con 0x7faef8100bd0 2024-04-05T21:24:33.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.621+0000 7faef5ffb700 1 --2- 172.21.15.121:0/1796185231 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7faee806c4e0 0x7faee806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:33.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.621+0000 7faef5ffb700 1 -- 172.21.15.121:0/1796185231 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7faeec08cf20 con 0x7faef8100bd0 2024-04-05T21:24:33.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.621+0000 7faef7fff700 1 --2- 172.21.15.121:0/1796185231 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7faee806c4e0 0x7faee806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:33.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.621+0000 7faef7fff700 1 --2- 172.21.15.121:0/1796185231 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7faee806c4e0 0x7faee806e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faee4006fd0 tx=0x7faee4008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:33.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.625+0000 7faef5ffb700 1 -- 172.21.15.121:0/1796185231 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7faeec0c00e0 con 0x7faef8100bd0 2024-04-05T21:24:33.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.815+0000 7faeff127700 1 -- 172.21.15.121:0/1796185231 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7faef8109830 con 0x7faef8100bd0 2024-04-05T21:24:33.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.816+0000 7faef5ffb700 1 -- 172.21.15.121:0/1796185231 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7faeec05b1e0 con 0x7faef8100bd0 2024-04-05T21:24:33.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.817+0000 7faeff127700 1 -- 172.21.15.121:0/1796185231 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7faee806c4e0 msgr2=0x7faee806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:33.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.817+0000 7faeff127700 1 --2- 172.21.15.121:0/1796185231 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7faee806c4e0 0x7faee806e9a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faee4006fd0 tx=0x7faee4008040 comp rx=0 tx=0).stop 2024-04-05T21:24:33.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.817+0000 7faeff127700 1 -- 172.21.15.121:0/1796185231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faef8100bd0 msgr2=0x7faef81963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:33.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.817+0000 7faeff127700 1 --2- 172.21.15.121:0/1796185231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faef8100bd0 0x7faef81963b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7faeec00bda0 tx=0x7faeec00be80 comp rx=0 tx=0).stop 2024-04-05T21:24:33.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.818+0000 7faeff127700 1 -- 172.21.15.121:0/1796185231 shutdown_connections 2024-04-05T21:24:33.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.818+0000 7faeff127700 1 --2- 172.21.15.121:0/1796185231 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7faee806c4e0 0x7faee806e9a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:33.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.818+0000 7faeff127700 1 --2- 172.21.15.121:0/1796185231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faef8100bd0 0x7faef81963b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:33.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.818+0000 7faeff127700 1 --2- 172.21.15.121:0/1796185231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faef8101dd0 0x7faef81968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:33.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.818+0000 7faeff127700 1 -- 172.21.15.121:0/1796185231 >> 172.21.15.121:0/1796185231 conn(0x7faef80fc170 msgr2=0x7faef8105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:33.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.818+0000 7faeff127700 1 -- 172.21.15.121:0/1796185231 shutdown_connections 2024-04-05T21:24:33.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:33.818+0000 7faeff127700 1 -- 172.21.15.121:0/1796185231 wait complete. 2024-04-05T21:24:34.682 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:34 smithi121 ceph-mon[39723]: pgmap v218: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-05T21:24:34.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-05T21:24:34.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:34 smithi188 ceph-mon[49715]: pgmap v218: 65 pgs: 65 active+clean; 582 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 682 B/s wr, 4 op/s 2024-04-05T21:24:34.982 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:24:36.566 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:36 smithi121 ceph-mon[39723]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-05T21:24:36.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:36 smithi188 ceph-mon[49715]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-05T21:24:37.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.199+0000 7ff6e9872700 1 -- 172.21.15.121:0/970682289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6e4100ce0 msgr2=0x7ff6e4101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:37.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.199+0000 7ff6e9872700 1 --2- 172.21.15.121:0/970682289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6e4100ce0 0x7ff6e4101140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009910 tx=0x7ff6cc009c20 comp rx=0 tx=0).stop 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.199+0000 7ff6e9872700 1 -- 172.21.15.121:0/970682289 shutdown_connections 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.199+0000 7ff6e9872700 1 --2- 172.21.15.121:0/970682289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6e4100ce0 0x7ff6e4101140 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.199+0000 7ff6e9872700 1 --2- 172.21.15.121:0/970682289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6e40ffae0 0x7ff6e40fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.199+0000 7ff6e9872700 1 -- 172.21.15.121:0/970682289 >> 172.21.15.121:0/970682289 conn(0x7ff6e40fb080 msgr2=0x7ff6e40fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.199+0000 7ff6e9872700 1 -- 172.21.15.121:0/970682289 shutdown_connections 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.200+0000 7ff6e9872700 1 -- 172.21.15.121:0/970682289 wait complete. 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.200+0000 7ff6e9872700 1 Processor -- start 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.200+0000 7ff6e9872700 1 -- start start 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.200+0000 7ff6e9872700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6e40ffae0 0x7ff6e4069950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.200+0000 7ff6e9872700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6e4100ce0 0x7ff6e419b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.200+0000 7ff6e9872700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e419b5e0 con 0x7ff6e4100ce0 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.200+0000 7ff6e9872700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e419b750 con 0x7ff6e40ffae0 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.201+0000 7ff6e27fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6e4100ce0 0x7ff6e419b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:37.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.201+0000 7ff6e2ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6e40ffae0 0x7ff6e4069950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.201+0000 7ff6e27fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6e4100ce0 0x7ff6e419b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38404/0 (socket says 172.21.15.121:38404) 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.201+0000 7ff6e27fc700 1 -- 172.21.15.121:0/3272641163 learned_addr learned my addr 172.21.15.121:0/3272641163 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.201+0000 7ff6e27fc700 1 -- 172.21.15.121:0/3272641163 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6e40ffae0 msgr2=0x7ff6e4069950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.201+0000 7ff6e27fc700 1 --2- 172.21.15.121:0/3272641163 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6e40ffae0 0x7ff6e4069950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.201+0000 7ff6e27fc700 1 -- 172.21.15.121:0/3272641163 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc0095f0 con 0x7ff6e4100ce0 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.201+0000 7ff6e27fc700 1 --2- 172.21.15.121:0/3272641163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6e4100ce0 0x7ff6e419b010 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc00bde0 tx=0x7ff6cc00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.201+0000 7ff6e8870700 1 -- 172.21.15.121:0/3272641163 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6cc01d070 con 0x7ff6e4100ce0 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.201+0000 7ff6e9872700 1 -- 172.21.15.121:0/3272641163 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e419b9d0 con 0x7ff6e4100ce0 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.201+0000 7ff6e8870700 1 -- 172.21.15.121:0/3272641163 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7ff6cc005dc0 con 0x7ff6e4100ce0 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.201+0000 7ff6e8870700 1 -- 172.21.15.121:0/3272641163 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6cc003ad0 con 0x7ff6e4100ce0 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.201+0000 7ff6e9872700 1 -- 172.21.15.121:0/3272641163 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e419bee0 con 0x7ff6e4100ce0 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.202+0000 7ff6e9872700 1 -- 172.21.15.121:0/3272641163 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e4108500 con 0x7ff6e4100ce0 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.203+0000 7ff6e8870700 1 -- 172.21.15.121:0/3272641163 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff6cc003d40 con 0x7ff6e4100ce0 2024-04-05T21:24:37.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.203+0000 7ff6e8870700 1 --2- 172.21.15.121:0/3272641163 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff6d006c4e0 0x7ff6d006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:37.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.203+0000 7ff6e2ffd700 1 --2- 172.21.15.121:0/3272641163 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff6d006c4e0 0x7ff6d006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:37.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.203+0000 7ff6e8870700 1 -- 172.21.15.121:0/3272641163 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff6cc08d050 con 0x7ff6e4100ce0 2024-04-05T21:24:37.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.204+0000 7ff6e2ffd700 1 --2- 172.21.15.121:0/3272641163 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff6d006c4e0 0x7ff6d006e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4100b40 tx=0x7ff6dc009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:37.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.208+0000 7ff6e8870700 1 -- 172.21.15.121:0/3272641163 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff6cc05b320 con 0x7ff6e4100ce0 2024-04-05T21:24:37.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.419+0000 7ff6e9872700 1 -- 172.21.15.121:0/3272641163 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7ff6e419c580 con 0x7ff6e4100ce0 2024-04-05T21:24:37.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.419+0000 7ff6e8870700 1 -- 172.21.15.121:0/3272641163 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7ff6cc05aeb0 con 0x7ff6e4100ce0 2024-04-05T21:24:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.422+0000 7ff6e9872700 1 -- 172.21.15.121:0/3272641163 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff6d006c4e0 msgr2=0x7ff6d006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.422+0000 7ff6e9872700 1 --2- 172.21.15.121:0/3272641163 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff6d006c4e0 0x7ff6d006e9a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4100b40 tx=0x7ff6dc009260 comp rx=0 tx=0).stop 2024-04-05T21:24:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.422+0000 7ff6e9872700 1 -- 172.21.15.121:0/3272641163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6e4100ce0 msgr2=0x7ff6e419b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.422+0000 7ff6e9872700 1 --2- 172.21.15.121:0/3272641163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6e4100ce0 0x7ff6e419b010 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc00bde0 tx=0x7ff6cc00be10 comp rx=0 tx=0).stop 2024-04-05T21:24:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.422+0000 7ff6e9872700 1 -- 172.21.15.121:0/3272641163 shutdown_connections 2024-04-05T21:24:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.422+0000 7ff6e9872700 1 --2- 172.21.15.121:0/3272641163 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff6d006c4e0 0x7ff6d006e9a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.422+0000 7ff6e9872700 1 --2- 172.21.15.121:0/3272641163 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6e40ffae0 0x7ff6e4069950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.422+0000 7ff6e9872700 1 --2- 172.21.15.121:0/3272641163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6e4100ce0 0x7ff6e419b010 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.422+0000 7ff6e9872700 1 -- 172.21.15.121:0/3272641163 >> 172.21.15.121:0/3272641163 conn(0x7ff6e40fb080 msgr2=0x7ff6e4103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.422+0000 7ff6e9872700 1 -- 172.21.15.121:0/3272641163 shutdown_connections 2024-04-05T21:24:37.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:37.422+0000 7ff6e9872700 1 -- 172.21.15.121:0/3272641163 wait complete. 2024-04-05T21:24:38.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-05T21:24:38.681 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:24:38.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:38 smithi121 ceph-mon[39723]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-05T21:24:38.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:38 smithi188 ceph-mon[49715]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-05T21:24:40.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:40 smithi121 ceph-mon[39723]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-05T21:24:40.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:24:40.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:40 smithi188 ceph-mon[49715]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-05T21:24:40.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:40 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:24:40.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.858+0000 7f278a236700 1 -- 172.21.15.121:0/3158761730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27840ff3a0 msgr2=0x7f27840ff7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:40.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.858+0000 7f278a236700 1 --2- 172.21.15.121:0/3158761730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27840ff3a0 0x7f27840ff7c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f277400aa20 tx=0x7f277400ad30 comp rx=0 tx=0).stop 2024-04-05T21:24:40.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.858+0000 7f278a236700 1 -- 172.21.15.121:0/3158761730 shutdown_connections 2024-04-05T21:24:40.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.858+0000 7f278a236700 1 --2- 172.21.15.121:0/3158761730 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2784100460 0x7f27841008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:40.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.858+0000 7f278a236700 1 --2- 172.21.15.121:0/3158761730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27840ff3a0 0x7f27840ff7c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:40.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.858+0000 7f278a236700 1 -- 172.21.15.121:0/3158761730 >> 172.21.15.121:0/3158761730 conn(0x7f27840fa380 msgr2=0x7f27840fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:40.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.858+0000 7f278a236700 1 -- 172.21.15.121:0/3158761730 shutdown_connections 2024-04-05T21:24:40.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.858+0000 7f278a236700 1 -- 172.21.15.121:0/3158761730 wait complete. 2024-04-05T21:24:40.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.859+0000 7f278a236700 1 Processor -- start 2024-04-05T21:24:40.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.859+0000 7f278a236700 1 -- start start 2024-04-05T21:24:40.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.859+0000 7f278a236700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27840ff3a0 0x7f2784196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:40.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.859+0000 7f278a236700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2784100460 0x7f27841967d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:40.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.859+0000 7f278a236700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2784196df0 con 0x7f27840ff3a0 2024-04-05T21:24:40.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.859+0000 7f278a236700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2784196f30 con 0x7f2784100460 2024-04-05T21:24:40.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.859+0000 7f277b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27840ff3a0 0x7f2784196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:40.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.859+0000 7f277b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27840ff3a0 0x7f2784196290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38422/0 (socket says 172.21.15.121:38422) 2024-04-05T21:24:40.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.859+0000 7f277b7fe700 1 -- 172.21.15.121:0/478416224 learned_addr learned my addr 172.21.15.121:0/478416224 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:40.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.859+0000 7f27837fe700 1 --2- 172.21.15.121:0/478416224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2784100460 0x7f27841967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:40.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.860+0000 7f277b7fe700 1 -- 172.21.15.121:0/478416224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2784100460 msgr2=0x7f27841967d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:40.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.860+0000 7f277b7fe700 1 --2- 172.21.15.121:0/478416224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2784100460 0x7f27841967d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:40.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.860+0000 7f277b7fe700 1 -- 172.21.15.121:0/478416224 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f277400a700 con 0x7f27840ff3a0 2024-04-05T21:24:40.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.860+0000 7f277b7fe700 1 --2- 172.21.15.121:0/478416224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27840ff3a0 0x7f2784196290 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2774006010 tx=0x7f277400af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:40.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.860+0000 7f27817fa700 1 -- 172.21.15.121:0/478416224 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f277400ccc0 con 0x7f27840ff3a0 2024-04-05T21:24:40.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.860+0000 7f27817fa700 1 -- 172.21.15.121:0/478416224 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f2774005000 con 0x7f27840ff3a0 2024-04-05T21:24:40.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.860+0000 7f278a236700 1 -- 172.21.15.121:0/478416224 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278419b980 con 0x7f27840ff3a0 2024-04-05T21:24:40.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.860+0000 7f27817fa700 1 -- 172.21.15.121:0/478416224 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2774016640 con 0x7f27840ff3a0 2024-04-05T21:24:40.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.860+0000 7f278a236700 1 -- 172.21.15.121:0/478416224 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278419bef0 con 0x7f27840ff3a0 2024-04-05T21:24:40.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.861+0000 7f278a236700 1 -- 172.21.15.121:0/478416224 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27841073e0 con 0x7f27840ff3a0 2024-04-05T21:24:40.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.861+0000 7f27817fa700 1 -- 172.21.15.121:0/478416224 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f277400e450 con 0x7f27840ff3a0 2024-04-05T21:24:40.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.862+0000 7f27817fa700 1 --2- 172.21.15.121:0/478416224 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f277006c4e0 0x7f277006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:40.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.862+0000 7f27817fa700 1 -- 172.21.15.121:0/478416224 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2774012070 con 0x7f27840ff3a0 2024-04-05T21:24:40.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.862+0000 7f27837fe700 1 --2- 172.21.15.121:0/478416224 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f277006c4e0 0x7f277006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:40.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.862+0000 7f27837fe700 1 --2- 172.21.15.121:0/478416224 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f277006c4e0 0x7f277006e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f276c009c60 tx=0x7f276c009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:40.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:40.866+0000 7f27817fa700 1 -- 172.21.15.121:0/478416224 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f277405bb00 con 0x7f27840ff3a0 2024-04-05T21:24:41.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:41.074+0000 7f278a236700 1 -- 172.21.15.121:0/478416224 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f278419c560 con 0x7f27840ff3a0 2024-04-05T21:24:41.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:41.075+0000 7f27817fa700 1 -- 172.21.15.121:0/478416224 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f277405b690 con 0x7f27840ff3a0 2024-04-05T21:24:41.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:41.077+0000 7f278a236700 1 -- 172.21.15.121:0/478416224 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f277006c4e0 msgr2=0x7f277006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:41.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:41.077+0000 7f278a236700 1 --2- 172.21.15.121:0/478416224 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f277006c4e0 0x7f277006e9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f276c009c60 tx=0x7f276c009260 comp rx=0 tx=0).stop 2024-04-05T21:24:41.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:41.077+0000 7f278a236700 1 -- 172.21.15.121:0/478416224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27840ff3a0 msgr2=0x7f2784196290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:41.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:41.077+0000 7f278a236700 1 --2- 172.21.15.121:0/478416224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27840ff3a0 0x7f2784196290 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2774006010 tx=0x7f277400af60 comp rx=0 tx=0).stop 2024-04-05T21:24:41.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:41.077+0000 7f278a236700 1 -- 172.21.15.121:0/478416224 shutdown_connections 2024-04-05T21:24:41.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:41.077+0000 7f278a236700 1 --2- 172.21.15.121:0/478416224 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f277006c4e0 0x7f277006e9a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:41.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:41.077+0000 7f278a236700 1 --2- 172.21.15.121:0/478416224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27840ff3a0 0x7f2784196290 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:41.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:41.077+0000 7f278a236700 1 --2- 172.21.15.121:0/478416224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2784100460 0x7f27841967d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:41.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:41.077+0000 7f278a236700 1 -- 172.21.15.121:0/478416224 >> 172.21.15.121:0/478416224 conn(0x7f27840fa380 msgr2=0x7f2784103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:41.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:41.077+0000 7f278a236700 1 -- 172.21.15.121:0/478416224 shutdown_connections 2024-04-05T21:24:41.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:41.077+0000 7f278a236700 1 -- 172.21.15.121:0/478416224 wait complete. 2024-04-05T21:24:42.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-05T21:24:42.277 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:24:42.553 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:42 smithi121 ceph-mon[39723]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-05T21:24:42.641 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:42 smithi188 ceph-mon[49715]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-05T21:24:44.296 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:44 smithi121 ceph-mon[39723]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-05T21:24:44.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.322+0000 7f65e67d9700 1 -- 172.21.15.121:0/2112100089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e01016e0 msgr2=0x7f65e0103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:44.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.322+0000 7f65e67d9700 1 --2- 172.21.15.121:0/2112100089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e01016e0 0x7f65e0103ad0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009910 tx=0x7f65d0009c20 comp rx=0 tx=0).stop 2024-04-05T21:24:44.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.322+0000 7f65e67d9700 1 -- 172.21.15.121:0/2112100089 shutdown_connections 2024-04-05T21:24:44.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.322+0000 7f65e67d9700 1 --2- 172.21.15.121:0/2112100089 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65e0104010 0x7f65e0106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:44.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.322+0000 7f65e67d9700 1 --2- 172.21.15.121:0/2112100089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e01016e0 0x7f65e0103ad0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:44.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.322+0000 7f65e67d9700 1 -- 172.21.15.121:0/2112100089 >> 172.21.15.121:0/2112100089 conn(0x7f65e00fb050 msgr2=0x7f65e00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:24:44.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.322+0000 7f65e67d9700 1 -- 172.21.15.121:0/2112100089 shutdown_connections 2024-04-05T21:24:44.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.322+0000 7f65e67d9700 1 -- 172.21.15.121:0/2112100089 wait complete. 2024-04-05T21:24:44.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.323+0000 7f65e67d9700 1 Processor -- start 2024-04-05T21:24:44.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.323+0000 7f65e67d9700 1 -- start start 2024-04-05T21:24:44.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.323+0000 7f65e67d9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65e01016e0 0x7f65e0196260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:44.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.323+0000 7f65e67d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e0104010 0x7f65e01967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:44.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.323+0000 7f65e67d9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e0196dc0 con 0x7f65e0104010 2024-04-05T21:24:44.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.323+0000 7f65e67d9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e0196f00 con 0x7f65e01016e0 2024-04-05T21:24:44.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.323+0000 7f65d7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e0104010 0x7f65e01967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:44.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.323+0000 7f65d7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e0104010 0x7f65e01967a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60510/0 (socket says 172.21.15.121:60510) 2024-04-05T21:24:44.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.323+0000 7f65d7fff700 1 -- 172.21.15.121:0/3176078374 learned_addr learned my addr 172.21.15.121:0/3176078374 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:24:44.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.324+0000 7f65dffff700 1 --2- 172.21.15.121:0/3176078374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65e01016e0 0x7f65e0196260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:44.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.324+0000 7f65d7fff700 1 -- 172.21.15.121:0/3176078374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65e01016e0 msgr2=0x7f65e0196260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:24:44.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.324+0000 7f65d7fff700 1 --2- 172.21.15.121:0/3176078374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65e01016e0 0x7f65e0196260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:24:44.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.324+0000 7f65d7fff700 1 -- 172.21.15.121:0/3176078374 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d00095f0 con 0x7f65e0104010 2024-04-05T21:24:44.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.324+0000 7f65d7fff700 1 --2- 172.21.15.121:0/3176078374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e0104010 0x7f65e01967a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f65c800d8d0 tx=0x7f65c800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:44.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.324+0000 7f65ddffb700 1 -- 172.21.15.121:0/3176078374 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c80098a0 con 0x7f65e0104010 2024-04-05T21:24:44.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.324+0000 7f65ddffb700 1 -- 172.21.15.121:0/3176078374 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f65c8009e90 con 0x7f65e0104010 2024-04-05T21:24:44.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.324+0000 7f65e67d9700 1 -- 172.21.15.121:0/3176078374 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e019b9b0 con 0x7f65e0104010 2024-04-05T21:24:44.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.324+0000 7f65ddffb700 1 -- 172.21.15.121:0/3176078374 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c800f5f0 con 0x7f65e0104010 2024-04-05T21:24:44.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.324+0000 7f65e67d9700 1 -- 172.21.15.121:0/3176078374 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e019bf80 con 0x7f65e0104010 2024-04-05T21:24:44.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.325+0000 7f65e67d9700 1 -- 172.21.15.121:0/3176078374 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e01903f0 con 0x7f65e0104010 2024-04-05T21:24:44.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.326+0000 7f65ddffb700 1 -- 172.21.15.121:0/3176078374 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f65c8010460 con 0x7f65e0104010 2024-04-05T21:24:44.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.326+0000 7f65ddffb700 1 --2- 172.21.15.121:0/3176078374 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f65cc06c4e0 0x7f65cc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:24:44.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.326+0000 7f65ddffb700 1 -- 172.21.15.121:0/3176078374 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f65c808b670 con 0x7f65e0104010 2024-04-05T21:24:44.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.326+0000 7f65dffff700 1 --2- 172.21.15.121:0/3176078374 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f65cc06c4e0 0x7f65cc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:24:44.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.327+0000 7f65dffff700 1 --2- 172.21.15.121:0/3176078374 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f65cc06c4e0 0x7f65cc06e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f65d0006010 tx=0x7f65d000b540 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:24:44.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.331+0000 7f65ddffb700 1 -- 172.21.15.121:0/3176078374 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f65c8059900 con 0x7f65e0104010 2024-04-05T21:24:44.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:24:44.572+0000 7f65e67d9700 1 -- 172.21.15.121:0/3176078374 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f65e00611d0 con 0x7f65cc06c4e0 2024-04-05T21:24:44.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:44 smithi188 ceph-mon[49715]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-05T21:24:46.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:46 smithi121 ceph-mon[39723]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:24:46.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:46 smithi121 ceph-mon[39723]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-05T21:24:46.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:46 smithi188 ceph-mon[49715]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:24:46.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:46 smithi188 ceph-mon[49715]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-05T21:24:48.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:48 smithi121 ceph-mon[39723]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-05T21:24:48.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:48 smithi188 ceph-mon[49715]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 597 B/s wr, 4 op/s 2024-04-05T21:24:50.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:50 smithi121 ceph-mon[39723]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T21:24:50.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:50 smithi188 ceph-mon[49715]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T21:24:52.747 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:52 smithi121 ceph-mon[39723]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T21:24:52.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:52 smithi188 ceph-mon[49715]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T21:24:54.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:54 smithi121 ceph-mon[39723]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T21:24:54.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:54 smithi188 ceph-mon[49715]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T21:24:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:55 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:24:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:55 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:24:56.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:56 smithi121 ceph-mon[39723]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-05T21:24:56.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:56 smithi188 ceph-mon[49715]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 426 B/s wr, 4 op/s 2024-04-05T21:24:58.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:24:58 smithi121 ceph-mon[39723]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:24:58.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:24:58 smithi188 ceph-mon[49715]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:25:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:00 smithi121 ceph-mon[39723]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T21:25:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:00 smithi188 ceph-mon[49715]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T21:25:02.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:02 smithi121 ceph-mon[39723]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:25:02.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:02 smithi188 ceph-mon[49715]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:25:04.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:04 smithi121 ceph-mon[39723]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:25:04.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:04 smithi188 ceph-mon[49715]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:25:06.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:06 smithi121 ceph-mon[39723]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:25:06.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:06 smithi188 ceph-mon[49715]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:25:08.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:08 smithi121 ceph-mon[39723]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:25:08.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:08 smithi188 ceph-mon[49715]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:25:10.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:10 smithi121 ceph-mon[39723]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T21:25:10.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:10 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:25:10.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:10 smithi188 ceph-mon[49715]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T21:25:10.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:10 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:25:12.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:12 smithi121 ceph-mon[39723]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:25:12.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:12 smithi188 ceph-mon[49715]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:25:14.682 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:14 smithi121 ceph-mon[39723]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:25:14.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:14 smithi188 ceph-mon[49715]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:25:16.490 INFO:tasks.workunit.client.1.smithi188.stderr:Note: switching to '8919605b0a5dc1f1650c8aec3051c77c0e3e240e'. 2024-04-05T21:25:16.493 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-04-05T21:25:16.493 INFO:tasks.workunit.client.1.smithi188.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-05T21:25:16.493 INFO:tasks.workunit.client.1.smithi188.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-05T21:25:16.494 INFO:tasks.workunit.client.1.smithi188.stderr:state without impacting any branches by switching back to a branch. 2024-04-05T21:25:16.494 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-04-05T21:25:16.494 INFO:tasks.workunit.client.1.smithi188.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-05T21:25:16.494 INFO:tasks.workunit.client.1.smithi188.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-05T21:25:16.494 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-04-05T21:25:16.494 INFO:tasks.workunit.client.1.smithi188.stderr: git switch -c 2024-04-05T21:25:16.495 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-04-05T21:25:16.495 INFO:tasks.workunit.client.1.smithi188.stderr:Or undo this operation with: 2024-04-05T21:25:16.495 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-04-05T21:25:16.495 INFO:tasks.workunit.client.1.smithi188.stderr: git switch - 2024-04-05T21:25:16.495 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-04-05T21:25:16.495 INFO:tasks.workunit.client.1.smithi188.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-05T21:25:16.495 INFO:tasks.workunit.client.1.smithi188.stderr: 2024-04-05T21:25:16.496 INFO:tasks.workunit.client.1.smithi188.stderr:HEAD is now at 8919605b0a5 Merge branch 'wip-65294-reef' of https://github.com/batrick/ceph into wip-yuri4-testing-2024-04-04-1128-reef 2024-04-05T21:25:16.498 DEBUG:teuthology.orchestra.run.smithi188:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-05T21:25:16.566 INFO:tasks.workunit.client.1.smithi188.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-05T21:25:16.567 INFO:tasks.workunit.client.1.smithi188.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-05T21:25:16.567 INFO:tasks.workunit.client.1.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-05T21:25:16.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:16 smithi121 ceph-mon[39723]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:25:16.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:16 smithi188 ceph-mon[49715]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:25:17.275 INFO:tasks.workunit.client.1.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-05T21:25:17.344 INFO:tasks.workunit.client.1.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-05T21:25:17.381 INFO:tasks.workunit.client.1.smithi188.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-05T21:25:17.383 INFO:tasks.workunit.client.1.smithi188.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-05T21:25:17.383 INFO:tasks.workunit.client.1.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-05T21:25:17.420 INFO:tasks.workunit.client.1.smithi188.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-05T21:25:17.423 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-05T21:25:17.423 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-05T21:25:17.481 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-04-05T21:25:17.483 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-05T21:25:17.483 DEBUG:teuthology.orchestra.run.smithi188:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8919605b0a5dc1f1650c8aec3051c77c0e3e240e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-04-05T21:25:17.550 INFO:tasks.workunit.client.1.smithi188.stderr:+ mkdir -p fsstress 2024-04-05T21:25:17.553 INFO:tasks.workunit.client.1.smithi188.stderr:+ pushd fsstress 2024-04-05T21:25:17.554 INFO:tasks.workunit.client.1.smithi188.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-05T21:25:17.554 INFO:tasks.workunit.client.1.smithi188.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-05T21:25:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:17 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:25:17.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:17 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:25:18.091 INFO:tasks.workunit.client.1.smithi188.stderr:+ tar xzf ltp-full.tgz 2024-04-05T21:25:18.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:18 smithi188 ceph-mon[49715]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:25:18.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:25:18.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:25:18.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:18 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:18.721 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:18 smithi121 ceph-mon[39723]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:25:18.722 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:25:18.722 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:25:18.722 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:18 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:20.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:20.040+0000 7f65ddffb700 1 -- 172.21.15.121:0/3176078374 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f65e00611d0 con 0x7f65cc06c4e0 2024-04-05T21:25:20.042 INFO:teuthology.orchestra.run.smithi121.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:25:20.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:20.043+0000 7f65e67d9700 1 -- 172.21.15.121:0/3176078374 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f65cc06c4e0 msgr2=0x7f65cc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:20.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:20.043+0000 7f65e67d9700 1 --2- 172.21.15.121:0/3176078374 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f65cc06c4e0 0x7f65cc06e9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f65d0006010 tx=0x7f65d000b540 comp rx=0 tx=0).stop 2024-04-05T21:25:20.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:20.043+0000 7f65e67d9700 1 -- 172.21.15.121:0/3176078374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e0104010 msgr2=0x7f65e01967a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:20.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:20.043+0000 7f65e67d9700 1 --2- 172.21.15.121:0/3176078374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e0104010 0x7f65e01967a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f65c800d8d0 tx=0x7f65c800dbe0 comp rx=0 tx=0).stop 2024-04-05T21:25:20.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:20.043+0000 7f65e67d9700 1 -- 172.21.15.121:0/3176078374 shutdown_connections 2024-04-05T21:25:20.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:20.043+0000 7f65e67d9700 1 --2- 172.21.15.121:0/3176078374 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f65cc06c4e0 0x7f65cc06e9a0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f65d0006010 tx=0x7f65d000b540 comp rx=0 tx=0).stop 2024-04-05T21:25:20.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:20.043+0000 7f65e67d9700 1 --2- 172.21.15.121:0/3176078374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f65e01016e0 0x7f65e0196260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:20.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:20.043+0000 7f65e67d9700 1 --2- 172.21.15.121:0/3176078374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e0104010 0x7f65e01967a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:20.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:20.043+0000 7f65e67d9700 1 -- 172.21.15.121:0/3176078374 >> 172.21.15.121:0/3176078374 conn(0x7f65e00fb050 msgr2=0x7f65e00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:20.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:20.044+0000 7f65e67d9700 1 -- 172.21.15.121:0/3176078374 shutdown_connections 2024-04-05T21:25:20.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:20.046+0000 7f65e67d9700 1 -- 172.21.15.121:0/3176078374 wait complete. 2024-04-05T21:25:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:20 smithi121 ceph-mon[39723]: pgmap v241: 65 pgs: 65 active+clean; 782 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 17 KiB/s wr, 5 op/s 2024-04-05T21:25:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:25:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:25:20.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:25:20.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:20 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:20.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:20 smithi188 ceph-mon[49715]: pgmap v241: 65 pgs: 65 active+clean; 782 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 17 KiB/s wr, 5 op/s 2024-04-05T21:25:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:25:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:25:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:25:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:20 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:20.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-05T21:25:21.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:21 smithi121 ceph-mon[39723]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:25:21.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:21 smithi121 ceph-mon[39723]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:25:21.598 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:25:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:21 smithi188 ceph-mon[49715]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:25:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:21 smithi188 ceph-mon[49715]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:25:22.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:22 smithi121 ceph-mon[39723]: pgmap v242: 65 pgs: 65 active+clean; 1.9 MiB data, 166 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 121 KiB/s wr, 12 op/s 2024-04-05T21:25:22.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:22 smithi188 ceph-mon[49715]: pgmap v242: 65 pgs: 65 active+clean; 1.9 MiB data, 166 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 121 KiB/s wr, 12 op/s 2024-04-05T21:25:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:23 smithi121 ceph-mon[39723]: Upgrade: Target is version 18.2.2-1173-g8919605b (reef) 2024-04-05T21:25:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:23 smithi121 ceph-mon[39723]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da'] 2024-04-05T21:25:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:25:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:23 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:23 smithi121 ceph-mon[39723]: Upgrade: Need to upgrade myself (mgr.smithi121.etbquh) 2024-04-05T21:25:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:23 smithi121 ceph-mon[39723]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da on smithi188 2024-04-05T21:25:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:23 smithi121 ceph-mon[39723]: pgmap v243: 65 pgs: 65 active+clean; 9.3 MiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 773 KiB/s wr, 39 op/s 2024-04-05T21:25:24.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:24.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:23 smithi188 ceph-mon[49715]: Upgrade: Target is version 18.2.2-1173-g8919605b (reef) 2024-04-05T21:25:24.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:23 smithi188 ceph-mon[49715]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da'] 2024-04-05T21:25:24.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:25:24.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:23 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:24.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:23 smithi188 ceph-mon[49715]: Upgrade: Need to upgrade myself (mgr.smithi121.etbquh) 2024-04-05T21:25:24.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:23 smithi188 ceph-mon[49715]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da on smithi188 2024-04-05T21:25:24.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:23 smithi188 ceph-mon[49715]: pgmap v243: 65 pgs: 65 active+clean; 9.3 MiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2.8 KiB/s rd, 773 KiB/s wr, 39 op/s 2024-04-05T21:25:25.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:25 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:25:25.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:25 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:25:25.903 INFO:tasks.workunit.client.0.smithi121.stderr:Updating files: 82% (9955/12088) Updating files: 83% (10034/12088) Updating files: 84% (10154/12088) Updating files: 85% (10275/12088) Updating files: 86% (10396/12088) Updating files: 87% (10517/12088) Updating files: 88% (10638/12088) Updating files: 89% (10759/12088) Updating files: 90% (10880/12088) Updating files: 91% (11001/12088) Updating files: 92% (11121/12088) Updating files: 93% (11242/12088) Updating files: 94% (11363/12088) Updating files: 95% (11484/12088) Updating files: 96% (11605/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-04-05T21:25:26.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:26 smithi188 ceph-mon[49715]: pgmap v244: 65 pgs: 65 active+clean; 19 MiB data, 233 MiB used, 536 GiB / 536 GiB avail; 273 KiB/s rd, 1.6 MiB/s wr, 118 op/s 2024-04-05T21:25:26.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:26 smithi121 ceph-mon[39723]: pgmap v244: 65 pgs: 65 active+clean; 19 MiB data, 233 MiB used, 536 GiB / 536 GiB avail; 273 KiB/s rd, 1.6 MiB/s wr, 118 op/s 2024-04-05T21:25:26.915 INFO:tasks.workunit.client.0.smithi121.stderr:Note: switching to '8919605b0a5dc1f1650c8aec3051c77c0e3e240e'. 2024-04-05T21:25:26.915 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-04-05T21:25:26.915 INFO:tasks.workunit.client.0.smithi121.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr:state without impacting any branches by switching back to a branch. 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr: git switch -c 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr:Or undo this operation with: 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr: git switch - 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-04-05T21:25:26.916 INFO:tasks.workunit.client.0.smithi121.stderr:HEAD is now at 8919605b0a5 Merge branch 'wip-65294-reef' of https://github.com/batrick/ceph into wip-yuri4-testing-2024-04-04-1128-reef 2024-04-05T21:25:26.921 DEBUG:teuthology.orchestra.run.smithi121:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-05T21:25:27.062 INFO:tasks.workunit.client.0.smithi121.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-05T21:25:27.067 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-05T21:25:27.067 INFO:tasks.workunit.client.0.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-05T21:25:27.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.287+0000 7f01490ed700 1 -- 172.21.15.121:0/924953603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01440fee80 msgr2=0x7f01440ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:27.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.287+0000 7f01490ed700 1 --2- 172.21.15.121:0/924953603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01440fee80 0x7f01440ff2a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f0134009910 tx=0x7f0134009c20 comp rx=0 tx=0).stop 2024-04-05T21:25:27.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.287+0000 7f01490ed700 1 -- 172.21.15.121:0/924953603 shutdown_connections 2024-04-05T21:25:27.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.287+0000 7f01490ed700 1 --2- 172.21.15.121:0/924953603 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0144101e60 0x7f01441022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:27.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.287+0000 7f01490ed700 1 --2- 172.21.15.121:0/924953603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01440fee80 0x7f01440ff2a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:27.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.287+0000 7f01490ed700 1 -- 172.21.15.121:0/924953603 >> 172.21.15.121:0/924953603 conn(0x7f01440fa7c0 msgr2=0x7f01440fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:27.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.287+0000 7f01490ed700 1 -- 172.21.15.121:0/924953603 shutdown_connections 2024-04-05T21:25:27.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.288+0000 7f01490ed700 1 -- 172.21.15.121:0/924953603 wait complete. 2024-04-05T21:25:27.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.288+0000 7f01490ed700 1 Processor -- start 2024-04-05T21:25:27.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.289+0000 7f01490ed700 1 -- start start 2024-04-05T21:25:27.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.289+0000 7f01490ed700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01440fee80 0x7f01441962d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:27.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.289+0000 7f01490ed700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0144101e60 0x7f0144196810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:27.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.289+0000 7f01490ed700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0144196e30 con 0x7f01440fee80 2024-04-05T21:25:27.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.289+0000 7f01490ed700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0144196f70 con 0x7f0144101e60 2024-04-05T21:25:27.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.289+0000 7f0142d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01440fee80 0x7f01441962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:27.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.289+0000 7f014259c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0144101e60 0x7f0144196810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:27.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.289+0000 7f0142d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01440fee80 0x7f01441962d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48272/0 (socket says 172.21.15.121:48272) 2024-04-05T21:25:27.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.289+0000 7f0142d9d700 1 -- 172.21.15.121:0/1305968336 learned_addr learned my addr 172.21.15.121:0/1305968336 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:25:27.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.289+0000 7f0142d9d700 1 -- 172.21.15.121:0/1305968336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0144101e60 msgr2=0x7f0144196810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:27.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.289+0000 7f0142d9d700 1 --2- 172.21.15.121:0/1305968336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0144101e60 0x7f0144196810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:27.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.289+0000 7f0142d9d700 1 -- 172.21.15.121:0/1305968336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01340095f0 con 0x7f01440fee80 2024-04-05T21:25:27.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.289+0000 7f014259c700 1 --2- 172.21.15.121:0/1305968336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0144101e60 0x7f0144196810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T21:25:27.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.290+0000 7f0142d9d700 1 --2- 172.21.15.121:0/1305968336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01440fee80 0x7f01441962d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f013400be10 tx=0x7f013400be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:27.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.290+0000 7f012bfff700 1 -- 172.21.15.121:0/1305968336 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f013401d070 con 0x7f01440fee80 2024-04-05T21:25:27.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.290+0000 7f01490ed700 1 -- 172.21.15.121:0/1305968336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014419b9c0 con 0x7f01440fee80 2024-04-05T21:25:27.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.290+0000 7f01490ed700 1 -- 172.21.15.121:0/1305968336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014419bed0 con 0x7f01440fee80 2024-04-05T21:25:27.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.290+0000 7f012bfff700 1 -- 172.21.15.121:0/1305968336 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f0134005dc0 con 0x7f01440fee80 2024-04-05T21:25:27.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.290+0000 7f012bfff700 1 -- 172.21.15.121:0/1305968336 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0134003a90 con 0x7f01440fee80 2024-04-05T21:25:27.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.291+0000 7f01490ed700 1 -- 172.21.15.121:0/1305968336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0130005320 con 0x7f01440fee80 2024-04-05T21:25:27.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.292+0000 7f012bfff700 1 -- 172.21.15.121:0/1305968336 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f0134003bf0 con 0x7f01440fee80 2024-04-05T21:25:27.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.292+0000 7f012bfff700 1 --2- 172.21.15.121:0/1305968336 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f012c06c4e0 0x7f012c06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:27.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.292+0000 7f012bfff700 1 -- 172.21.15.121:0/1305968336 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f013408d020 con 0x7f01440fee80 2024-04-05T21:25:27.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.292+0000 7f014259c700 1 --2- 172.21.15.121:0/1305968336 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f012c06c4e0 0x7f012c06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:27.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.293+0000 7f014259c700 1 --2- 172.21.15.121:0/1305968336 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f012c06c4e0 0x7f012c06e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0138011490 tx=0x7f0138009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:27.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.297+0000 7f012bfff700 1 -- 172.21.15.121:0/1305968336 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f013405b2b0 con 0x7f01440fee80 2024-04-05T21:25:27.722 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:25:27.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.544+0000 7f01490ed700 1 -- 172.21.15.121:0/1305968336 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0130000bf0 con 0x7f012c06c4e0 2024-04-05T21:25:27.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.545+0000 7f012bfff700 1 -- 172.21.15.121:0/1305968336 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f0130000bf0 con 0x7f012c06c4e0 2024-04-05T21:25:27.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.547+0000 7f01490ed700 1 -- 172.21.15.121:0/1305968336 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f012c06c4e0 msgr2=0x7f012c06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:27.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.547+0000 7f01490ed700 1 --2- 172.21.15.121:0/1305968336 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f012c06c4e0 0x7f012c06e9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0138011490 tx=0x7f0138009260 comp rx=0 tx=0).stop 2024-04-05T21:25:27.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.547+0000 7f01490ed700 1 -- 172.21.15.121:0/1305968336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01440fee80 msgr2=0x7f01441962d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:27.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.547+0000 7f01490ed700 1 --2- 172.21.15.121:0/1305968336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01440fee80 0x7f01441962d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f013400be10 tx=0x7f013400be40 comp rx=0 tx=0).stop 2024-04-05T21:25:27.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.547+0000 7f01490ed700 1 -- 172.21.15.121:0/1305968336 shutdown_connections 2024-04-05T21:25:27.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.547+0000 7f01490ed700 1 --2- 172.21.15.121:0/1305968336 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f012c06c4e0 0x7f012c06e9a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:27.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.547+0000 7f01490ed700 1 --2- 172.21.15.121:0/1305968336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01440fee80 0x7f01441962d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:27.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.547+0000 7f01490ed700 1 --2- 172.21.15.121:0/1305968336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0144101e60 0x7f0144196810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:27.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.547+0000 7f01490ed700 1 -- 172.21.15.121:0/1305968336 >> 172.21.15.121:0/1305968336 conn(0x7f01440fa7c0 msgr2=0x7f0144105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:27.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.547+0000 7f01490ed700 1 -- 172.21.15.121:0/1305968336 shutdown_connections 2024-04-05T21:25:27.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.547+0000 7f01490ed700 1 -- 172.21.15.121:0/1305968336 wait complete. 2024-04-05T21:25:27.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.662+0000 7f642a940700 1 -- 172.21.15.121:0/3019283349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64240ffab0 msgr2=0x7f64240ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:27.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.662+0000 7f642a940700 1 --2- 172.21.15.121:0/3019283349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64240ffab0 0x7f64240ffed0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6414009910 tx=0x7f6414009c20 comp rx=0 tx=0).stop 2024-04-05T21:25:27.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.662+0000 7f642a940700 1 -- 172.21.15.121:0/3019283349 shutdown_connections 2024-04-05T21:25:27.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.662+0000 7f642a940700 1 --2- 172.21.15.121:0/3019283349 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6424100cb0 0x7f6424101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:27.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.662+0000 7f642a940700 1 --2- 172.21.15.121:0/3019283349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64240ffab0 0x7f64240ffed0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:27.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.662+0000 7f642a940700 1 -- 172.21.15.121:0/3019283349 >> 172.21.15.121:0/3019283349 conn(0x7f64240fb050 msgr2=0x7f64240fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:27.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.662+0000 7f642a940700 1 -- 172.21.15.121:0/3019283349 shutdown_connections 2024-04-05T21:25:27.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.662+0000 7f642a940700 1 -- 172.21.15.121:0/3019283349 wait complete. 2024-04-05T21:25:27.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.663+0000 7f642a940700 1 Processor -- start 2024-04-05T21:25:27.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.663+0000 7f642a940700 1 -- start start 2024-04-05T21:25:27.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.663+0000 7f642a940700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64240ffab0 0x7f6424069800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:27.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.663+0000 7f642a940700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6424100cb0 0x7f642419b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:27.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.663+0000 7f642a940700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642419b5e0 con 0x7f6424100cb0 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.663+0000 7f642a940700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642419b720 con 0x7f64240ffab0 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.663+0000 7f64237fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6424100cb0 0x7f642419b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.664+0000 7f64237fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6424100cb0 0x7f642419b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48294/0 (socket says 172.21.15.121:48294) 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.664+0000 7f64237fe700 1 -- 172.21.15.121:0/3119065353 learned_addr learned my addr 172.21.15.121:0/3119065353 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.664+0000 7f6423fff700 1 --2- 172.21.15.121:0/3119065353 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64240ffab0 0x7f6424069800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.664+0000 7f64237fe700 1 -- 172.21.15.121:0/3119065353 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64240ffab0 msgr2=0x7f6424069800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.664+0000 7f64237fe700 1 --2- 172.21.15.121:0/3119065353 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64240ffab0 0x7f6424069800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.664+0000 7f64237fe700 1 -- 172.21.15.121:0/3119065353 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64140095f0 con 0x7f6424100cb0 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.664+0000 7f64237fe700 1 --2- 172.21.15.121:0/3119065353 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6424100cb0 0x7f642419b010 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f641800d900 tx=0x7f641800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.664+0000 7f64217fa700 1 -- 172.21.15.121:0/3119065353 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64180041f0 con 0x7f6424100cb0 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.664+0000 7f642a940700 1 -- 172.21.15.121:0/3119065353 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642419ba00 con 0x7f6424100cb0 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.664+0000 7f64217fa700 1 -- 172.21.15.121:0/3119065353 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f6418004cc0 con 0x7f6424100cb0 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.664+0000 7f64217fa700 1 -- 172.21.15.121:0/3119065353 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64180056c0 con 0x7f6424100cb0 2024-04-05T21:25:27.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.664+0000 7f642a940700 1 -- 172.21.15.121:0/3119065353 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642419bf50 con 0x7f6424100cb0 2024-04-05T21:25:27.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.665+0000 7f642a940700 1 -- 172.21.15.121:0/3119065353 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64241084d0 con 0x7f6424100cb0 2024-04-05T21:25:27.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.666+0000 7f64217fa700 1 -- 172.21.15.121:0/3119065353 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f6418004750 con 0x7f6424100cb0 2024-04-05T21:25:27.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.666+0000 7f64217fa700 1 --2- 172.21.15.121:0/3119065353 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f640c06c490 0x7f640c06e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:27.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.666+0000 7f64217fa700 1 -- 172.21.15.121:0/3119065353 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f641808bad0 con 0x7f6424100cb0 2024-04-05T21:25:27.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.666+0000 7f6423fff700 1 --2- 172.21.15.121:0/3119065353 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f640c06c490 0x7f640c06e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:27.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.667+0000 7f6423fff700 1 --2- 172.21.15.121:0/3119065353 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f640c06c490 0x7f640c06e950 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6414006010 tx=0x7f641400b580 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:27.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.672+0000 7f64217fa700 1 -- 172.21.15.121:0/3119065353 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f6418059ce0 con 0x7f6424100cb0 2024-04-05T21:25:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:27 smithi121 ceph-mon[39723]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:25:27.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:27 smithi188 ceph-mon[49715]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:25:27.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.918+0000 7f642a940700 1 -- 172.21.15.121:0/3119065353 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6424105600 con 0x7f640c06c490 2024-04-05T21:25:27.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.919+0000 7f64217fa700 1 -- 172.21.15.121:0/3119065353 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f6424105600 con 0x7f640c06c490 2024-04-05T21:25:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.922+0000 7f642a940700 1 -- 172.21.15.121:0/3119065353 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f640c06c490 msgr2=0x7f640c06e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.922+0000 7f642a940700 1 --2- 172.21.15.121:0/3119065353 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f640c06c490 0x7f640c06e950 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6414006010 tx=0x7f641400b580 comp rx=0 tx=0).stop 2024-04-05T21:25:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.922+0000 7f642a940700 1 -- 172.21.15.121:0/3119065353 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6424100cb0 msgr2=0x7f642419b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.922+0000 7f642a940700 1 --2- 172.21.15.121:0/3119065353 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6424100cb0 0x7f642419b010 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f641800d900 tx=0x7f641800dcc0 comp rx=0 tx=0).stop 2024-04-05T21:25:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.922+0000 7f642a940700 1 -- 172.21.15.121:0/3119065353 shutdown_connections 2024-04-05T21:25:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.922+0000 7f642a940700 1 --2- 172.21.15.121:0/3119065353 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f640c06c490 0x7f640c06e950 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.922+0000 7f642a940700 1 --2- 172.21.15.121:0/3119065353 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64240ffab0 0x7f6424069800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.922+0000 7f642a940700 1 --2- 172.21.15.121:0/3119065353 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6424100cb0 0x7f642419b010 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.922+0000 7f642a940700 1 -- 172.21.15.121:0/3119065353 >> 172.21.15.121:0/3119065353 conn(0x7f64240fb050 msgr2=0x7f6424103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:27.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.922+0000 7f642a940700 1 -- 172.21.15.121:0/3119065353 shutdown_connections 2024-04-05T21:25:27.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:27.922+0000 7f642a940700 1 -- 172.21.15.121:0/3119065353 wait complete. 2024-04-05T21:25:28.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.041+0000 7f47654d8700 1 -- 172.21.15.121:0/1717865749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47600ffa90 msgr2=0x7f47600ffeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:28.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.041+0000 7f47654d8700 1 --2- 172.21.15.121:0/1717865749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47600ffa90 0x7f47600ffeb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4750009910 tx=0x7f4750009c20 comp rx=0 tx=0).stop 2024-04-05T21:25:28.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.041+0000 7f47654d8700 1 -- 172.21.15.121:0/1717865749 shutdown_connections 2024-04-05T21:25:28.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.041+0000 7f47654d8700 1 --2- 172.21.15.121:0/1717865749 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4760100c90 0x7f47601010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.041+0000 7f47654d8700 1 --2- 172.21.15.121:0/1717865749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47600ffa90 0x7f47600ffeb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.041+0000 7f47654d8700 1 -- 172.21.15.121:0/1717865749 >> 172.21.15.121:0/1717865749 conn(0x7f47600fb030 msgr2=0x7f47600fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:28.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.041+0000 7f47654d8700 1 -- 172.21.15.121:0/1717865749 shutdown_connections 2024-04-05T21:25:28.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.041+0000 7f47654d8700 1 -- 172.21.15.121:0/1717865749 wait complete. 2024-04-05T21:25:28.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.042+0000 7f47654d8700 1 Processor -- start 2024-04-05T21:25:28.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.042+0000 7f47654d8700 1 -- start start 2024-04-05T21:25:28.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.042+0000 7f47654d8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47600ffa90 0x7f4760195130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:28.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.042+0000 7f47654d8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4760100c90 0x7f4760195670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:28.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.042+0000 7f47654d8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4760195c90 con 0x7f4760100c90 2024-04-05T21:25:28.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.042+0000 7f47654d8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4760195dd0 con 0x7f47600ffa90 2024-04-05T21:25:28.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.042+0000 7f475effd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47600ffa90 0x7f4760195130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:28.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.042+0000 7f475effd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47600ffa90 0x7f4760195130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.121:47832/0 (socket says 172.21.15.121:47832) 2024-04-05T21:25:28.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.042+0000 7f475effd700 1 -- 172.21.15.121:0/2539361993 learned_addr learned my addr 172.21.15.121:0/2539361993 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:25:28.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.043+0000 7f475effd700 1 -- 172.21.15.121:0/2539361993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4760100c90 msgr2=0x7f4760195670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:28.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.043+0000 7f475effd700 1 --2- 172.21.15.121:0/2539361993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4760100c90 0x7f4760195670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.043+0000 7f475effd700 1 -- 172.21.15.121:0/2539361993 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47500095f0 con 0x7f47600ffa90 2024-04-05T21:25:28.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.043+0000 7f475effd700 1 --2- 172.21.15.121:0/2539361993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47600ffa90 0x7f4760195130 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4750000c00 tx=0x7f475000be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:28.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.043+0000 7f4747fff700 1 -- 172.21.15.121:0/2539361993 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f475001d070 con 0x7f47600ffa90 2024-04-05T21:25:28.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.043+0000 7f47654d8700 1 -- 172.21.15.121:0/2539361993 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476019a820 con 0x7f47600ffa90 2024-04-05T21:25:28.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.043+0000 7f4747fff700 1 -- 172.21.15.121:0/2539361993 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f4750005dc0 con 0x7f47600ffa90 2024-04-05T21:25:28.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.043+0000 7f4747fff700 1 -- 172.21.15.121:0/2539361993 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4750003b10 con 0x7f47600ffa90 2024-04-05T21:25:28.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.043+0000 7f47654d8700 1 -- 172.21.15.121:0/2539361993 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f476019ad30 con 0x7f47600ffa90 2024-04-05T21:25:28.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.044+0000 7f47654d8700 1 -- 172.21.15.121:0/2539361993 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47601084b0 con 0x7f47600ffa90 2024-04-05T21:25:28.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.045+0000 7f4747fff700 1 -- 172.21.15.121:0/2539361993 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f4750003c70 con 0x7f47600ffa90 2024-04-05T21:25:28.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.045+0000 7f4747fff700 1 --2- 172.21.15.121:0/2539361993 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f474806c4e0 0x7f474806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:28.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.045+0000 7f475e7fc700 1 --2- 172.21.15.121:0/2539361993 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f474806c4e0 0x7f474806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:28.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.045+0000 7f4747fff700 1 -- 172.21.15.121:0/2539361993 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f475008d0e0 con 0x7f47600ffa90 2024-04-05T21:25:28.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.046+0000 7f475e7fc700 1 --2- 172.21.15.121:0/2539361993 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f474806c4e0 0x7f474806e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4754005950 tx=0x7f47540058e0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:28.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.054+0000 7f4747fff700 1 -- 172.21.15.121:0/2539361993 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f475005b290 con 0x7f47600ffa90 2024-04-05T21:25:28.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.300+0000 7f47654d8700 1 -- 172.21.15.121:0/2539361993 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4760068e70 con 0x7f474806c4e0 2024-04-05T21:25:28.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.311+0000 7f4747fff700 1 -- 172.21.15.121:0/2539361993 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3504 (secure 0 0 0) 0x7f4760068e70 con 0x7f474806c4e0 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (6m) 71s ago 7m 26.2M - 0.25.0 c8568f914cd2 6594a12b2bc8 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (8m) 71s ago 8m 7520k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (6m) 77s ago 6m 7470k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (7m) 71s ago 7m 6651k - 18.2.1 5be31c24972a 5bf35fe2ace4 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (6m) 77s ago 6m 6651k - 18.2.1 5be31c24972a dcdfa6ae17bf 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (6m) 71s ago 7m 84.5M - 9.4.7 954c08fa6188 ee38366c66e3 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (100s) 71s ago 99s 12.9M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (92s) 71s ago 92s 16.4M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (97s) 77s ago 97s 16.6M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (89s) 77s ago 89s 16.8M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:9283,8765,8443 running (9m) 71s ago 9m 489M - 18.2.1 5be31c24972a d8cb1548d844 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (6m) 77s ago 6m 440M - 18.2.1 5be31c24972a a38c0d6743cb 2024-04-05T21:25:28.313 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (9m) 71s ago 9m 46.8M 2048M 18.2.1 5be31c24972a 3c30ce99c006 2024-04-05T21:25:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (6m) 77s ago 6m 36.3M 2048M 18.2.1 5be31c24972a 8ecc9636670e 2024-04-05T21:25:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (7m) 71s ago 7m 18.4M - 1.5.0 0da6a335fe13 5ac35283f3e9 2024-04-05T21:25:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (6m) 77s ago 6m 16.3M - 1.5.0 0da6a335fe13 bc2ad104140f 2024-04-05T21:25:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (5m) 71s ago 5m 48.1M 3958M 18.2.1 5be31c24972a f6f3a45429a2 2024-04-05T21:25:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (5m) 71s ago 5m 50.7M 3958M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:25:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (4m) 71s ago 4m 47.7M 3958M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:25:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (4m) 77s ago 4m 49.0M 4982M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:25:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (3m) 77s ago 3m 43.6M 4982M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:25:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (3m) 77s ago 3m 44.3M 4982M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:25:28.314 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (5m) 71s ago 7m 56.4M - 2.43.0 a07b618ecd1d 9395525b1c3e 2024-04-05T21:25:28.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.314+0000 7f47654d8700 1 -- 172.21.15.121:0/2539361993 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f474806c4e0 msgr2=0x7f474806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:28.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.314+0000 7f47654d8700 1 --2- 172.21.15.121:0/2539361993 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f474806c4e0 0x7f474806e9a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4754005950 tx=0x7f47540058e0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.314+0000 7f47654d8700 1 -- 172.21.15.121:0/2539361993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47600ffa90 msgr2=0x7f4760195130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:28.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.314+0000 7f47654d8700 1 --2- 172.21.15.121:0/2539361993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47600ffa90 0x7f4760195130 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4750000c00 tx=0x7f475000be90 comp rx=0 tx=0).stop 2024-04-05T21:25:28.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.314+0000 7f47654d8700 1 -- 172.21.15.121:0/2539361993 shutdown_connections 2024-04-05T21:25:28.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.314+0000 7f47654d8700 1 --2- 172.21.15.121:0/2539361993 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f474806c4e0 0x7f474806e9a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.314+0000 7f47654d8700 1 --2- 172.21.15.121:0/2539361993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47600ffa90 0x7f4760195130 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.314+0000 7f47654d8700 1 --2- 172.21.15.121:0/2539361993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4760100c90 0x7f4760195670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.314+0000 7f47654d8700 1 -- 172.21.15.121:0/2539361993 >> 172.21.15.121:0/2539361993 conn(0x7f47600fb030 msgr2=0x7f4760103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:28.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.314+0000 7f47654d8700 1 -- 172.21.15.121:0/2539361993 shutdown_connections 2024-04-05T21:25:28.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.314+0000 7f47654d8700 1 -- 172.21.15.121:0/2539361993 wait complete. 2024-04-05T21:25:28.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.454+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/3870125680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce4100cb0 msgr2=0x7f1ce4101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:28.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.454+0000 7f1ceb8bf700 1 --2- 172.21.15.121:0/3870125680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce4100cb0 0x7f1ce4101110 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4009910 tx=0x7f1cd4009c20 comp rx=0 tx=0).stop 2024-04-05T21:25:28.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.454+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/3870125680 shutdown_connections 2024-04-05T21:25:28.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.454+0000 7f1ceb8bf700 1 --2- 172.21.15.121:0/3870125680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce4100cb0 0x7f1ce4101110 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.454+0000 7f1ceb8bf700 1 --2- 172.21.15.121:0/3870125680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ce40ffab0 0x7f1ce40ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.454+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/3870125680 >> 172.21.15.121:0/3870125680 conn(0x7f1ce40fb030 msgr2=0x7f1ce40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:28.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.454+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/3870125680 shutdown_connections 2024-04-05T21:25:28.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.454+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/3870125680 wait complete. 2024-04-05T21:25:28.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.455+0000 7f1ceb8bf700 1 Processor -- start 2024-04-05T21:25:28.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.455+0000 7f1ceb8bf700 1 -- start start 2024-04-05T21:25:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.455+0000 7f1ceb8bf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce40ffab0 0x7f1ce4068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.455+0000 7f1ceb8bf700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ce4100cb0 0x7f1ce419afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.455+0000 7f1ceb8bf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce419b580 con 0x7f1ce40ffab0 2024-04-05T21:25:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.455+0000 7f1ceb8bf700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce419b6c0 con 0x7f1ce4100cb0 2024-04-05T21:25:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.455+0000 7f1ce965b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce40ffab0 0x7f1ce4068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.455+0000 7f1ce965b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce40ffab0 0x7f1ce4068f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48328/0 (socket says 172.21.15.121:48328) 2024-04-05T21:25:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.455+0000 7f1ce965b700 1 -- 172.21.15.121:0/177027403 learned_addr learned my addr 172.21.15.121:0/177027403 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:25:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.456+0000 7f1ce965b700 1 -- 172.21.15.121:0/177027403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ce4100cb0 msgr2=0x7f1ce419afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.456+0000 7f1ce965b700 1 --2- 172.21.15.121:0/177027403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ce4100cb0 0x7f1ce419afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.456+0000 7f1ce965b700 1 -- 172.21.15.121:0/177027403 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd40095f0 con 0x7f1ce40ffab0 2024-04-05T21:25:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.456+0000 7f1ce965b700 1 --2- 172.21.15.121:0/177027403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce40ffab0 0x7f1ce4068f70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000d900 tx=0x7f1ce000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:28.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.456+0000 7f1cda7fc700 1 -- 172.21.15.121:0/177027403 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ce000b920 con 0x7f1ce40ffab0 2024-04-05T21:25:28.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.456+0000 7f1cda7fc700 1 -- 172.21.15.121:0/177027403 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f1ce0010460 con 0x7f1ce40ffab0 2024-04-05T21:25:28.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.456+0000 7f1cda7fc700 1 -- 172.21.15.121:0/177027403 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ce000f5b0 con 0x7f1ce40ffab0 2024-04-05T21:25:28.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.456+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/177027403 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce419b9a0 con 0x7f1ce40ffab0 2024-04-05T21:25:28.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.456+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/177027403 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce419bef0 con 0x7f1ce40ffab0 2024-04-05T21:25:28.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.458+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/177027403 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce41084d0 con 0x7f1ce40ffab0 2024-04-05T21:25:28.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.458+0000 7f1cda7fc700 1 -- 172.21.15.121:0/177027403 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f1ce00105d0 con 0x7f1ce40ffab0 2024-04-05T21:25:28.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.459+0000 7f1cda7fc700 1 --2- 172.21.15.121:0/177027403 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1cd006c4e0 0x7f1cd006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:28.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.459+0000 7f1cda7fc700 1 -- 172.21.15.121:0/177027403 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f1ce008b6f0 con 0x7f1ce40ffab0 2024-04-05T21:25:28.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.459+0000 7f1ce8e5a700 1 --2- 172.21.15.121:0/177027403 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1cd006c4e0 0x7f1cd006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:28.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.459+0000 7f1ce8e5a700 1 --2- 172.21.15.121:0/177027403 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1cd006c4e0 0x7f1cd006e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1cd400b5c0 tx=0x7f1cd4005fd0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:28.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.464+0000 7f1cda7fc700 1 -- 172.21.15.121:0/177027403 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f1ce0059980 con 0x7f1ce40ffab0 2024-04-05T21:25:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:28 smithi121 ceph-mon[39723]: pgmap v245: 65 pgs: 65 active+clean; 24 MiB data, 251 MiB used, 536 GiB / 536 GiB avail; 397 KiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-04-05T21:25:28.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:28 smithi188 ceph-mon[49715]: pgmap v245: 65 pgs: 65 active+clean; 24 MiB data, 251 MiB used, 536 GiB / 536 GiB avail; 397 KiB/s rd, 2.1 MiB/s wr, 162 op/s 2024-04-05T21:25:28.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.782+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/177027403 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1ce419ca00 con 0x7f1ce40ffab0 2024-04-05T21:25:28.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.783+0000 7f1cda7fc700 1 -- 172.21.15.121:0/177027403 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f1ce0059510 con 0x7f1ce40ffab0 2024-04-05T21:25:28.786 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:25:28.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:25:28.786 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-05T21:25:28.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:25:28.786 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:25:28.787 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-05T21:25:28.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:25:28.787 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:25:28.787 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-05T21:25:28.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:25:28.787 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:25:28.787 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:25:28.787 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:25:28.787 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:25:28.787 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-04-05T21:25:28.787 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:25:28.787 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:25:28.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.785+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/177027403 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1cd006c4e0 msgr2=0x7f1cd006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:28.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.785+0000 7f1ceb8bf700 1 --2- 172.21.15.121:0/177027403 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1cd006c4e0 0x7f1cd006e9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1cd400b5c0 tx=0x7f1cd4005fd0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.785+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/177027403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce40ffab0 msgr2=0x7f1ce4068f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:28.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.785+0000 7f1ceb8bf700 1 --2- 172.21.15.121:0/177027403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce40ffab0 0x7f1ce4068f70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000d900 tx=0x7f1ce000dcc0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.785+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/177027403 shutdown_connections 2024-04-05T21:25:28.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.785+0000 7f1ceb8bf700 1 --2- 172.21.15.121:0/177027403 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f1cd006c4e0 0x7f1cd006e9a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.785+0000 7f1ceb8bf700 1 --2- 172.21.15.121:0/177027403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ce40ffab0 0x7f1ce4068f70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.785+0000 7f1ceb8bf700 1 --2- 172.21.15.121:0/177027403 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ce4100cb0 0x7f1ce419afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.785+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/177027403 >> 172.21.15.121:0/177027403 conn(0x7f1ce40fb030 msgr2=0x7f1ce4103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:28.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.785+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/177027403 shutdown_connections 2024-04-05T21:25:28.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.785+0000 7f1ceb8bf700 1 -- 172.21.15.121:0/177027403 wait complete. 2024-04-05T21:25:28.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.902+0000 7f99f8c61700 1 -- 172.21.15.121:0/3661675533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99f4101dd0 msgr2=0x7f99f4102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:28.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.902+0000 7f99f8c61700 1 --2- 172.21.15.121:0/3661675533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99f4101dd0 0x7f99f4102230 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f99e8009910 tx=0x7f99e8009c20 comp rx=0 tx=0).stop 2024-04-05T21:25:28.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.903+0000 7f99f8c61700 1 -- 172.21.15.121:0/3661675533 shutdown_connections 2024-04-05T21:25:28.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.903+0000 7f99f8c61700 1 --2- 172.21.15.121:0/3661675533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99f4101dd0 0x7f99f4102230 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.903+0000 7f99f8c61700 1 --2- 172.21.15.121:0/3661675533 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99f4100bd0 0x7f99f4100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.903+0000 7f99f8c61700 1 -- 172.21.15.121:0/3661675533 >> 172.21.15.121:0/3661675533 conn(0x7f99f40fc170 msgr2=0x7f99f40fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:28.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.903+0000 7f99f8c61700 1 -- 172.21.15.121:0/3661675533 shutdown_connections 2024-04-05T21:25:28.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.903+0000 7f99f8c61700 1 -- 172.21.15.121:0/3661675533 wait complete. 2024-04-05T21:25:28.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.904+0000 7f99f8c61700 1 Processor -- start 2024-04-05T21:25:28.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.904+0000 7f99f8c61700 1 -- start start 2024-04-05T21:25:28.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.904+0000 7f99f8c61700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99f4100bd0 0x7f99f419c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:28.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.904+0000 7f99f8c61700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99f4101dd0 0x7f99f419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:28.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.904+0000 7f99f8c61700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f419d500 con 0x7f99f4100bd0 2024-04-05T21:25:28.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.904+0000 7f99f8c61700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f419d640 con 0x7f99f4101dd0 2024-04-05T21:25:28.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.904+0000 7f99f259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99f4100bd0 0x7f99f419c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:28.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.904+0000 7f99f1d9b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99f4101dd0 0x7f99f419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:28.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.904+0000 7f99f259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99f4100bd0 0x7f99f419c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48350/0 (socket says 172.21.15.121:48350) 2024-04-05T21:25:28.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.904+0000 7f99f259c700 1 -- 172.21.15.121:0/3429311856 learned_addr learned my addr 172.21.15.121:0/3429311856 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:25:28.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.905+0000 7f99f259c700 1 -- 172.21.15.121:0/3429311856 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99f4101dd0 msgr2=0x7f99f419cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:28.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.905+0000 7f99f259c700 1 --2- 172.21.15.121:0/3429311856 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99f4101dd0 0x7f99f419cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:28.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.905+0000 7f99f259c700 1 -- 172.21.15.121:0/3429311856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e80095f0 con 0x7f99f4100bd0 2024-04-05T21:25:28.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.905+0000 7f99f259c700 1 --2- 172.21.15.121:0/3429311856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99f4100bd0 0x7f99f419c9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f99e400b480 tx=0x7f99e400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:28.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.905+0000 7f99e37fe700 1 -- 172.21.15.121:0/3429311856 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99e4011800 con 0x7f99f4100bd0 2024-04-05T21:25:28.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.905+0000 7f99f8c61700 1 -- 172.21.15.121:0/3429311856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f406a950 con 0x7f99f4100bd0 2024-04-05T21:25:28.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.905+0000 7f99e37fe700 1 -- 172.21.15.121:0/3429311856 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f99e4011df0 con 0x7f99f4100bd0 2024-04-05T21:25:28.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.905+0000 7f99e37fe700 1 -- 172.21.15.121:0/3429311856 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99e4010e30 con 0x7f99f4100bd0 2024-04-05T21:25:28.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.905+0000 7f99f8c61700 1 -- 172.21.15.121:0/3429311856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f406af20 con 0x7f99f4100bd0 2024-04-05T21:25:28.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.906+0000 7f99f8c61700 1 -- 172.21.15.121:0/3429311856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d4005320 con 0x7f99f4100bd0 2024-04-05T21:25:28.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.907+0000 7f99e37fe700 1 -- 172.21.15.121:0/3429311856 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f99e401a400 con 0x7f99f4100bd0 2024-04-05T21:25:28.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.907+0000 7f99e37fe700 1 --2- 172.21.15.121:0/3429311856 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f99dc06c4e0 0x7f99dc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:28.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.908+0000 7f99e37fe700 1 -- 172.21.15.121:0/3429311856 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f99e408b2f0 con 0x7f99f4100bd0 2024-04-05T21:25:28.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.908+0000 7f99f1d9b700 1 --2- 172.21.15.121:0/3429311856 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f99dc06c4e0 0x7f99dc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:28.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.908+0000 7f99f1d9b700 1 --2- 172.21.15.121:0/3429311856 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f99dc06c4e0 0x7f99dc06e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f99e800b5c0 tx=0x7f99e80051d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:28.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:28.916+0000 7f99e37fe700 1 -- 172.21.15.121:0/3429311856 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f99e4059570 con 0x7f99f4100bd0 2024-04-05T21:25:29.024 INFO:tasks.workunit.client.0.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-05T21:25:29.103 INFO:tasks.workunit.client.0.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-05T21:25:29.145 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-05T21:25:29.146 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-05T21:25:29.146 INFO:tasks.workunit.client.0.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-05T21:25:29.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:29.162+0000 7f99f8c61700 1 -- 172.21.15.121:0/3429311856 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99d4000bf0 con 0x7f99dc06c4e0 2024-04-05T21:25:29.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:29.163+0000 7f99e37fe700 1 -- 172.21.15.121:0/3429311856 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f99d4000bf0 con 0x7f99dc06c4e0 2024-04-05T21:25:29.164 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:25:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:25:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:25:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T21:25:29.164 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [], 2024-04-05T21:25:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "0/2 daemons upgraded", 2024-04-05T21:25:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da image on host smithi188", 2024-04-05T21:25:29.165 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:25:29.165 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:25:29.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:29.165+0000 7f99f8c61700 1 -- 172.21.15.121:0/3429311856 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f99dc06c4e0 msgr2=0x7f99dc06e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:29.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:29.165+0000 7f99f8c61700 1 --2- 172.21.15.121:0/3429311856 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f99dc06c4e0 0x7f99dc06e9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f99e800b5c0 tx=0x7f99e80051d0 comp rx=0 tx=0).stop 2024-04-05T21:25:29.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:29.165+0000 7f99f8c61700 1 -- 172.21.15.121:0/3429311856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99f4100bd0 msgr2=0x7f99f419c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:29.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:29.165+0000 7f99f8c61700 1 --2- 172.21.15.121:0/3429311856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99f4100bd0 0x7f99f419c9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f99e400b480 tx=0x7f99e400b790 comp rx=0 tx=0).stop 2024-04-05T21:25:29.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:29.165+0000 7f99f8c61700 1 -- 172.21.15.121:0/3429311856 shutdown_connections 2024-04-05T21:25:29.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:29.165+0000 7f99f8c61700 1 --2- 172.21.15.121:0/3429311856 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f99dc06c4e0 0x7f99dc06e9a0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f99e800b5c0 tx=0x7f99e80051d0 comp rx=0 tx=0).stop 2024-04-05T21:25:29.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:29.165+0000 7f99f8c61700 1 --2- 172.21.15.121:0/3429311856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99f4100bd0 0x7f99f419c9a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:29.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:29.165+0000 7f99f8c61700 1 --2- 172.21.15.121:0/3429311856 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99f4101dd0 0x7f99f419cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:29.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:29.165+0000 7f99f8c61700 1 -- 172.21.15.121:0/3429311856 >> 172.21.15.121:0/3429311856 conn(0x7f99f40fc170 msgr2=0x7f99f4102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:29.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:29.166+0000 7f99f8c61700 1 -- 172.21.15.121:0/3429311856 shutdown_connections 2024-04-05T21:25:29.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:29.166+0000 7f99f8c61700 1 -- 172.21.15.121:0/3429311856 wait complete. 2024-04-05T21:25:29.187 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-05T21:25:29.191 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T21:25:29.191 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-05T21:25:29.210 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-04-05T21:25:29.211 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-04-05T21:25:29.211 DEBUG:teuthology.orchestra.run.smithi121:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8919605b0a5dc1f1650c8aec3051c77c0e3e240e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-04-05T21:25:29.281 INFO:tasks.workunit.client.0.smithi121.stderr:+ mkdir -p fsstress 2024-04-05T21:25:29.283 INFO:tasks.workunit.client.0.smithi121.stderr:+ pushd fsstress 2024-04-05T21:25:29.284 INFO:tasks.workunit.client.0.smithi121.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-05T21:25:29.285 INFO:tasks.workunit.client.0.smithi121.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-04-05T21:25:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:29 smithi121 ceph-mon[39723]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:25:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:29 smithi121 ceph-mon[39723]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:25:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:29 smithi121 ceph-mon[39723]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:25:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:29 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/177027403' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:25:29.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:29 smithi188 ceph-mon[49715]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:25:29.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:29 smithi188 ceph-mon[49715]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:25:29.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:29 smithi188 ceph-mon[49715]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:25:29.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:29 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/177027403' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:25:30.375 INFO:tasks.workunit.client.0.smithi121.stderr:+ tar xzf ltp-full.tgz 2024-04-05T21:25:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:30 smithi121 ceph-mon[39723]: pgmap v246: 65 pgs: 65 active+clean; 27 MiB data, 275 MiB used, 536 GiB / 536 GiB avail; 397 KiB/s rd, 2.3 MiB/s wr, 214 op/s 2024-04-05T21:25:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:30 smithi121 ceph-mon[39723]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:25:30.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:30 smithi188 ceph-mon[49715]: pgmap v246: 65 pgs: 65 active+clean; 27 MiB data, 275 MiB used, 536 GiB / 536 GiB avail; 397 KiB/s rd, 2.3 MiB/s wr, 214 op/s 2024-04-05T21:25:30.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:30 smithi188 ceph-mon[49715]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:25:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:32 smithi121 ceph-mon[39723]: pgmap v247: 65 pgs: 65 active+clean; 28 MiB data, 283 MiB used, 536 GiB / 536 GiB avail; 468 KiB/s rd, 2.4 MiB/s wr, 232 op/s 2024-04-05T21:25:32.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:32 smithi188 ceph-mon[49715]: pgmap v247: 65 pgs: 65 active+clean; 28 MiB data, 283 MiB used, 536 GiB / 536 GiB avail; 468 KiB/s rd, 2.4 MiB/s wr, 232 op/s 2024-04-05T21:25:34.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:34 smithi121 ceph-mon[39723]: pgmap v248: 65 pgs: 65 active+clean; 41 MiB data, 325 MiB used, 536 GiB / 536 GiB avail; 781 KiB/s rd, 3.5 MiB/s wr, 279 op/s 2024-04-05T21:25:34.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:34 smithi188 ceph-mon[49715]: pgmap v248: 65 pgs: 65 active+clean; 41 MiB data, 325 MiB used, 536 GiB / 536 GiB avail; 781 KiB/s rd, 3.5 MiB/s wr, 279 op/s 2024-04-05T21:25:36.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:36 smithi121 ceph-mon[39723]: pgmap v249: 65 pgs: 65 active+clean; 62 MiB data, 416 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 384 op/s 2024-04-05T21:25:36.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:36 smithi188 ceph-mon[49715]: pgmap v249: 65 pgs: 65 active+clean; 62 MiB data, 416 MiB used, 536 GiB / 536 GiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 384 op/s 2024-04-05T21:25:38.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:38 smithi121 ceph-mon[39723]: pgmap v250: 65 pgs: 65 active+clean; 68 MiB data, 454 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.3 MiB/s wr, 373 op/s 2024-04-05T21:25:38.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:38 smithi188 ceph-mon[49715]: pgmap v250: 65 pgs: 65 active+clean; 68 MiB data, 454 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.3 MiB/s wr, 373 op/s 2024-04-05T21:25:40.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:40 smithi121 ceph-mon[39723]: pgmap v251: 65 pgs: 65 active+clean; 72 MiB data, 494 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.2 MiB/s wr, 410 op/s 2024-04-05T21:25:40.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:40 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:25:40.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:40 smithi188 ceph-mon[49715]: pgmap v251: 65 pgs: 65 active+clean; 72 MiB data, 494 MiB used, 536 GiB / 536 GiB avail; 1.4 MiB/s rd, 4.2 MiB/s wr, 410 op/s 2024-04-05T21:25:40.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:40 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:25:42.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:42 smithi121 ceph-mon[39723]: pgmap v252: 65 pgs: 65 active+clean; 77 MiB data, 496 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 392 op/s 2024-04-05T21:25:42.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:42 smithi188 ceph-mon[49715]: pgmap v252: 65 pgs: 65 active+clean; 77 MiB data, 496 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 392 op/s 2024-04-05T21:25:44.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:44 smithi121 ceph-mon[39723]: pgmap v253: 65 pgs: 65 active+clean; 79 MiB data, 503 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 432 op/s 2024-04-05T21:25:44.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:44 smithi188 ceph-mon[49715]: pgmap v253: 65 pgs: 65 active+clean; 79 MiB data, 503 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 432 op/s 2024-04-05T21:25:46.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:46 smithi121 ceph-mon[39723]: pgmap v254: 65 pgs: 65 active+clean; 95 MiB data, 513 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 504 op/s 2024-04-05T21:25:46.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:46 smithi188 ceph-mon[49715]: pgmap v254: 65 pgs: 65 active+clean; 95 MiB data, 513 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 504 op/s 2024-04-05T21:25:48.595 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:48 smithi188 ceph-mon[49715]: pgmap v255: 65 pgs: 65 active+clean; 99 MiB data, 544 MiB used, 536 GiB / 536 GiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 430 op/s 2024-04-05T21:25:48.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:48 smithi121 ceph-mon[39723]: pgmap v255: 65 pgs: 65 active+clean; 99 MiB data, 544 MiB used, 536 GiB / 536 GiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 430 op/s 2024-04-05T21:25:50.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:50 smithi121 ceph-mon[39723]: pgmap v256: 65 pgs: 65 active+clean; 106 MiB data, 590 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 441 op/s 2024-04-05T21:25:50.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:50 smithi188 ceph-mon[49715]: pgmap v256: 65 pgs: 65 active+clean; 106 MiB data, 590 MiB used, 536 GiB / 536 GiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 441 op/s 2024-04-05T21:25:52.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:52 smithi121 ceph-mon[39723]: pgmap v257: 65 pgs: 65 active+clean; 113 MiB data, 609 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 393 op/s 2024-04-05T21:25:52.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:52 smithi188 ceph-mon[49715]: pgmap v257: 65 pgs: 65 active+clean; 113 MiB data, 609 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 393 op/s 2024-04-05T21:25:54.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:53 smithi188 ceph-mon[49715]: Upgrade: Updating mgr.smithi188.tgcuir 2024-04-05T21:25:54.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:53 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:54.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:53 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi188.tgcuir", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T21:25:54.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:53 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T21:25:54.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:53 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:25:54.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:53 smithi188 ceph-mon[49715]: Deploying daemon mgr.smithi188.tgcuir on smithi188 2024-04-05T21:25:54.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:53 smithi188 ceph-mon[49715]: pgmap v258: 65 pgs: 65 active+clean; 121 MiB data, 643 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 427 op/s 2024-04-05T21:25:54.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:53 smithi121 ceph-mon[39723]: Upgrade: Updating mgr.smithi188.tgcuir 2024-04-05T21:25:54.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:53 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:54.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:53 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi188.tgcuir", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T21:25:54.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:53 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T21:25:54.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:53 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:25:54.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:53 smithi121 ceph-mon[39723]: Deploying daemon mgr.smithi188.tgcuir on smithi188 2024-04-05T21:25:54.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:53 smithi121 ceph-mon[39723]: pgmap v258: 65 pgs: 65 active+clean; 121 MiB data, 643 MiB used, 536 GiB / 536 GiB avail; 1.6 MiB/s rd, 3.9 MiB/s wr, 427 op/s 2024-04-05T21:25:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:55 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:25:55.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:55 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:25:56.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:56 smithi188 ceph-mon[49715]: pgmap v259: 65 pgs: 65 active+clean; 135 MiB data, 703 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 491 op/s 2024-04-05T21:25:56.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:56 smithi121 ceph-mon[39723]: pgmap v259: 65 pgs: 65 active+clean; 135 MiB data, 703 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 4.9 MiB/s wr, 491 op/s 2024-04-05T21:25:58.650 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:58 smithi188 ceph-mon[49715]: pgmap v260: 65 pgs: 65 active+clean; 141 MiB data, 724 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 428 op/s 2024-04-05T21:25:58.650 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:58 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:58.650 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:58 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:58.650 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:25:58 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:25:58.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:58 smithi121 ceph-mon[39723]: pgmap v260: 65 pgs: 65 active+clean; 141 MiB data, 724 MiB used, 536 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 428 op/s 2024-04-05T21:25:58.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:58 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:58.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:58 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:25:58.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:25:58 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:25:59.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.299+0000 7f3dd099a700 1 -- 172.21.15.121:0/2654455832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dcc101050 msgr2=0x7f3dcc101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:59.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.299+0000 7f3dd099a700 1 --2- 172.21.15.121:0/2654455832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dcc101050 0x7f3dcc101470 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc009910 tx=0x7f3dbc009c20 comp rx=0 tx=0).stop 2024-04-05T21:25:59.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.300+0000 7f3dd099a700 1 -- 172.21.15.121:0/2654455832 shutdown_connections 2024-04-05T21:25:59.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.300+0000 7f3dd099a700 1 --2- 172.21.15.121:0/2654455832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3dcc0ffd90 0x7f3dcc1001f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:59.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.300+0000 7f3dd099a700 1 --2- 172.21.15.121:0/2654455832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dcc101050 0x7f3dcc101470 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:59.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.300+0000 7f3dd099a700 1 -- 172.21.15.121:0/2654455832 >> 172.21.15.121:0/2654455832 conn(0x7f3dcc0fa7c0 msgr2=0x7f3dcc0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.300+0000 7f3dd099a700 1 -- 172.21.15.121:0/2654455832 shutdown_connections 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.300+0000 7f3dd099a700 1 -- 172.21.15.121:0/2654455832 wait complete. 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.300+0000 7f3dd099a700 1 Processor -- start 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.300+0000 7f3dd099a700 1 -- start start 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dd099a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3dcc0ffd90 0x7f3dcc1951d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dd099a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dcc101050 0x7f3dcc195710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dd099a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc195d30 con 0x7f3dcc101050 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dd099a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc195e70 con 0x7f3dcc0ffd90 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dc9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dcc101050 0x7f3dcc195710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dc9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dcc101050 0x7f3dcc195710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40834/0 (socket says 172.21.15.121:40834) 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dc9d9b700 1 -- 172.21.15.121:0/3462539250 learned_addr learned my addr 172.21.15.121:0/3462539250 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dca59c700 1 --2- 172.21.15.121:0/3462539250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3dcc0ffd90 0x7f3dcc1951d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dc9d9b700 1 -- 172.21.15.121:0/3462539250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3dcc0ffd90 msgr2=0x7f3dcc1951d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dc9d9b700 1 --2- 172.21.15.121:0/3462539250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3dcc0ffd90 0x7f3dcc1951d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dc9d9b700 1 -- 172.21.15.121:0/3462539250 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc0095f0 con 0x7f3dcc101050 2024-04-05T21:25:59.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dc9d9b700 1 --2- 172.21.15.121:0/3462539250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dcc101050 0x7f3dcc195710 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000e920 tx=0x7f3dc000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:59.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dbb7fe700 1 -- 172.21.15.121:0/3462539250 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dc000c9d0 con 0x7f3dcc101050 2024-04-05T21:25:59.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dbb7fe700 1 -- 172.21.15.121:0/3462539250 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f3dc0010430 con 0x7f3dcc101050 2024-04-05T21:25:59.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dd099a700 1 -- 172.21.15.121:0/3462539250 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dcc19a920 con 0x7f3dcc101050 2024-04-05T21:25:59.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dbb7fe700 1 -- 172.21.15.121:0/3462539250 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dc0018630 con 0x7f3dcc101050 2024-04-05T21:25:59.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.301+0000 7f3dd099a700 1 -- 172.21.15.121:0/3462539250 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dcc19aef0 con 0x7f3dcc101050 2024-04-05T21:25:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.303+0000 7f3dd099a700 1 -- 172.21.15.121:0/3462539250 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dcc1084f0 con 0x7f3dcc101050 2024-04-05T21:25:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.303+0000 7f3dbb7fe700 1 -- 172.21.15.121:0/3462539250 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f3dc00105a0 con 0x7f3dcc101050 2024-04-05T21:25:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.303+0000 7f3dbb7fe700 1 --2- 172.21.15.121:0/3462539250 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3db406c530 0x7f3db406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.303+0000 7f3dbb7fe700 1 -- 172.21.15.121:0/3462539250 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f3dc0014070 con 0x7f3dcc101050 2024-04-05T21:25:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.303+0000 7f3dca59c700 1 --2- 172.21.15.121:0/3462539250 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3db406c530 0x7f3db406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:59.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.304+0000 7f3dca59c700 1 --2- 172.21.15.121:0/3462539250 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3db406c530 0x7f3db406e9f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc009fd0 tx=0x7f3dbc00b5a0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:59.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.309+0000 7f3dbb7fe700 1 -- 172.21.15.121:0/3462539250 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f3dc00520e0 con 0x7f3dcc101050 2024-04-05T21:25:59.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.596+0000 7f3dd099a700 1 -- 172.21.15.121:0/3462539250 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3dcc1056f0 con 0x7f3db406c530 2024-04-05T21:25:59.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.598+0000 7f3dbb7fe700 1 -- 172.21.15.121:0/3462539250 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f3dcc1056f0 con 0x7f3db406c530 2024-04-05T21:25:59.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.600+0000 7f3dd099a700 1 -- 172.21.15.121:0/3462539250 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3db406c530 msgr2=0x7f3db406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:59.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.600+0000 7f3dd099a700 1 --2- 172.21.15.121:0/3462539250 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3db406c530 0x7f3db406e9f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc009fd0 tx=0x7f3dbc00b5a0 comp rx=0 tx=0).stop 2024-04-05T21:25:59.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.600+0000 7f3dd099a700 1 -- 172.21.15.121:0/3462539250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dcc101050 msgr2=0x7f3dcc195710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:59.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.600+0000 7f3dd099a700 1 --2- 172.21.15.121:0/3462539250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dcc101050 0x7f3dcc195710 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000e920 tx=0x7f3dc000ec30 comp rx=0 tx=0).stop 2024-04-05T21:25:59.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.601+0000 7f3dd099a700 1 -- 172.21.15.121:0/3462539250 shutdown_connections 2024-04-05T21:25:59.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.601+0000 7f3dd099a700 1 --2- 172.21.15.121:0/3462539250 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f3db406c530 0x7f3db406e9f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:59.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.601+0000 7f3dd099a700 1 --2- 172.21.15.121:0/3462539250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3dcc0ffd90 0x7f3dcc1951d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:59.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.601+0000 7f3dd099a700 1 --2- 172.21.15.121:0/3462539250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dcc101050 0x7f3dcc195710 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:59.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.601+0000 7f3dd099a700 1 -- 172.21.15.121:0/3462539250 >> 172.21.15.121:0/3462539250 conn(0x7f3dcc0fa7c0 msgr2=0x7f3dcc103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:59.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.601+0000 7f3dd099a700 1 -- 172.21.15.121:0/3462539250 shutdown_connections 2024-04-05T21:25:59.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.601+0000 7f3dd099a700 1 -- 172.21.15.121:0/3462539250 wait complete. 2024-04-05T21:25:59.614 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:25:59.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.740+0000 7f5c9eba6700 1 -- 172.21.15.121:0/4080460728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c98100cb0 msgr2=0x7f5c98101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:59.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.740+0000 7f5c9eba6700 1 --2- 172.21.15.121:0/4080460728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c98100cb0 0x7f5c98101110 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009910 tx=0x7f5c84009c20 comp rx=0 tx=0).stop 2024-04-05T21:25:59.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.740+0000 7f5c9eba6700 1 -- 172.21.15.121:0/4080460728 shutdown_connections 2024-04-05T21:25:59.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.740+0000 7f5c9eba6700 1 --2- 172.21.15.121:0/4080460728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c98100cb0 0x7f5c98101110 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:59.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.740+0000 7f5c9eba6700 1 --2- 172.21.15.121:0/4080460728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c980ffab0 0x7f5c980ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:59.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.740+0000 7f5c9eba6700 1 -- 172.21.15.121:0/4080460728 >> 172.21.15.121:0/4080460728 conn(0x7f5c980fb030 msgr2=0x7f5c980fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:25:59.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.740+0000 7f5c9eba6700 1 -- 172.21.15.121:0/4080460728 shutdown_connections 2024-04-05T21:25:59.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.740+0000 7f5c9eba6700 1 -- 172.21.15.121:0/4080460728 wait complete. 2024-04-05T21:25:59.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.740+0000 7f5c9eba6700 1 Processor -- start 2024-04-05T21:25:59.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.740+0000 7f5c9eba6700 1 -- start start 2024-04-05T21:25:59.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.741+0000 7f5c9eba6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c980ffab0 0x7f5c98198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:59.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.741+0000 7f5c9eba6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c98100cb0 0x7f5c98198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:59.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.741+0000 7f5c9eba6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98198fb0 con 0x7f5c980ffab0 2024-04-05T21:25:59.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.741+0000 7f5c9eba6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c981990f0 con 0x7f5c98100cb0 2024-04-05T21:25:59.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.741+0000 7f5c97fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c98100cb0 0x7f5c98198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:59.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.741+0000 7f5c97fff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c98100cb0 0x7f5c98198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.121:32974/0 (socket says 172.21.15.121:32974) 2024-04-05T21:25:59.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.741+0000 7f5c97fff700 1 -- 172.21.15.121:0/2779573781 learned_addr learned my addr 172.21.15.121:0/2779573781 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:25:59.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.741+0000 7f5c9c942700 1 --2- 172.21.15.121:0/2779573781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c980ffab0 0x7f5c98198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:59.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.741+0000 7f5c9c942700 1 -- 172.21.15.121:0/2779573781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c98100cb0 msgr2=0x7f5c98198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:25:59.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.741+0000 7f5c9c942700 1 --2- 172.21.15.121:0/2779573781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c98100cb0 0x7f5c98198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:25:59.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.741+0000 7f5c9c942700 1 -- 172.21.15.121:0/2779573781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c840095f0 con 0x7f5c980ffab0 2024-04-05T21:25:59.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.741+0000 7f5c97fff700 1 --2- 172.21.15.121:0/2779573781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c98100cb0 0x7f5c98198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T21:25:59.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.741+0000 7f5c9c942700 1 --2- 172.21.15.121:0/2779573781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c980ffab0 0x7f5c98198450 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00e940 tx=0x7f5c8c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:59.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.742+0000 7f5c95ffb700 1 -- 172.21.15.121:0/2779573781 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c8c0041f0 con 0x7f5c980ffab0 2024-04-05T21:25:59.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.742+0000 7f5c9eba6700 1 -- 172.21.15.121:0/2779573781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c9819dba0 con 0x7f5c980ffab0 2024-04-05T21:25:59.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.742+0000 7f5c95ffb700 1 -- 172.21.15.121:0/2779573781 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f5c8c004350 con 0x7f5c980ffab0 2024-04-05T21:25:59.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.742+0000 7f5c95ffb700 1 -- 172.21.15.121:0/2779573781 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c8c003c10 con 0x7f5c980ffab0 2024-04-05T21:25:59.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.742+0000 7f5c9eba6700 1 -- 172.21.15.121:0/2779573781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c9819e0f0 con 0x7f5c980ffab0 2024-04-05T21:25:59.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.743+0000 7f5c9eba6700 1 -- 172.21.15.121:0/2779573781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c981084d0 con 0x7f5c980ffab0 2024-04-05T21:25:59.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.744+0000 7f5c95ffb700 1 -- 172.21.15.121:0/2779573781 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f5c8c00c7c0 con 0x7f5c980ffab0 2024-04-05T21:25:59.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.744+0000 7f5c95ffb700 1 --2- 172.21.15.121:0/2779573781 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f5c8806c4e0 0x7f5c8806e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:25:59.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.744+0000 7f5c95ffb700 1 -- 172.21.15.121:0/2779573781 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f5c8c014070 con 0x7f5c980ffab0 2024-04-05T21:25:59.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.744+0000 7f5c97fff700 1 --2- 172.21.15.121:0/2779573781 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f5c8806c4e0 0x7f5c8806e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:25:59.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.745+0000 7f5c97fff700 1 --2- 172.21.15.121:0/2779573781 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f5c8806c4e0 0x7f5c8806e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009fd0 tx=0x7f5c8400b5a0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:25:59.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:25:59.750+0000 7f5c95ffb700 1 -- 172.21.15.121:0/2779573781 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f5c8c059df0 con 0x7f5c980ffab0 2024-04-05T21:26:00.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.048+0000 7f5c9eba6700 1 -- 172.21.15.121:0/2779573781 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c98068ae0 con 0x7f5c8806c4e0 2024-04-05T21:26:00.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.050+0000 7f5c95ffb700 1 -- 172.21.15.121:0/2779573781 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f5c98068ae0 con 0x7f5c8806c4e0 2024-04-05T21:26:00.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.053+0000 7f5c9eba6700 1 -- 172.21.15.121:0/2779573781 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f5c8806c4e0 msgr2=0x7f5c8806e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:00.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.053+0000 7f5c9eba6700 1 --2- 172.21.15.121:0/2779573781 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f5c8806c4e0 0x7f5c8806e9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009fd0 tx=0x7f5c8400b5a0 comp rx=0 tx=0).stop 2024-04-05T21:26:00.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.053+0000 7f5c9eba6700 1 -- 172.21.15.121:0/2779573781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c980ffab0 msgr2=0x7f5c98198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:00.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.053+0000 7f5c9eba6700 1 --2- 172.21.15.121:0/2779573781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c980ffab0 0x7f5c98198450 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00e940 tx=0x7f5c8c00ec50 comp rx=0 tx=0).stop 2024-04-05T21:26:00.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.053+0000 7f5c9eba6700 1 -- 172.21.15.121:0/2779573781 shutdown_connections 2024-04-05T21:26:00.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.053+0000 7f5c9eba6700 1 --2- 172.21.15.121:0/2779573781 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f5c8806c4e0 0x7f5c8806e9a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:00.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.053+0000 7f5c9eba6700 1 --2- 172.21.15.121:0/2779573781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c980ffab0 0x7f5c98198450 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:00.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.053+0000 7f5c9eba6700 1 --2- 172.21.15.121:0/2779573781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5c98100cb0 0x7f5c98198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:00.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.053+0000 7f5c9eba6700 1 -- 172.21.15.121:0/2779573781 >> 172.21.15.121:0/2779573781 conn(0x7f5c980fb030 msgr2=0x7f5c980683d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:00.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.053+0000 7f5c9eba6700 1 -- 172.21.15.121:0/2779573781 shutdown_connections 2024-04-05T21:26:00.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.053+0000 7f5c9eba6700 1 -- 172.21.15.121:0/2779573781 wait complete. 2024-04-05T21:26:00.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.192+0000 7f26cabef700 1 -- 172.21.15.121:0/2709497998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26c4100cb0 msgr2=0x7f26c4101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:00.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.192+0000 7f26cabef700 1 --2- 172.21.15.121:0/2709497998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26c4100cb0 0x7f26c4101110 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f26b0009910 tx=0x7f26b0009c20 comp rx=0 tx=0).stop 2024-04-05T21:26:00.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.192+0000 7f26cabef700 1 -- 172.21.15.121:0/2709497998 shutdown_connections 2024-04-05T21:26:00.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.192+0000 7f26cabef700 1 --2- 172.21.15.121:0/2709497998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26c4100cb0 0x7f26c4101110 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:00.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.192+0000 7f26cabef700 1 --2- 172.21.15.121:0/2709497998 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26c40ffab0 0x7f26c40ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:00.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.192+0000 7f26cabef700 1 -- 172.21.15.121:0/2709497998 >> 172.21.15.121:0/2709497998 conn(0x7f26c40fb030 msgr2=0x7f26c40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:00.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.192+0000 7f26cabef700 1 -- 172.21.15.121:0/2709497998 shutdown_connections 2024-04-05T21:26:00.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.192+0000 7f26cabef700 1 -- 172.21.15.121:0/2709497998 wait complete. 2024-04-05T21:26:00.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.193+0000 7f26cabef700 1 Processor -- start 2024-04-05T21:26:00.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.193+0000 7f26cabef700 1 -- start start 2024-04-05T21:26:00.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.193+0000 7f26cabef700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26c40ffab0 0x7f26c4068fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:00.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.193+0000 7f26cabef700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26c4100cb0 0x7f26c419d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:00.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.193+0000 7f26cabef700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c419d890 con 0x7f26c4100cb0 2024-04-05T21:26:00.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.193+0000 7f26cabef700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c419d9d0 con 0x7f26c40ffab0 2024-04-05T21:26:00.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.193+0000 7f26c898b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26c40ffab0 0x7f26c4068fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:00.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.193+0000 7f26c3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26c4100cb0 0x7f26c419d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:00.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.193+0000 7f26c898b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26c40ffab0 0x7f26c4068fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.121:32996/0 (socket says 172.21.15.121:32996) 2024-04-05T21:26:00.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.193+0000 7f26c3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26c4100cb0 0x7f26c419d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40870/0 (socket says 172.21.15.121:40870) 2024-04-05T21:26:00.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.193+0000 7f26c898b700 1 -- 172.21.15.121:0/2645604433 learned_addr learned my addr 172.21.15.121:0/2645604433 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:26:00.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.194+0000 7f26c3fff700 1 -- 172.21.15.121:0/2645604433 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26c40ffab0 msgr2=0x7f26c4068fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:00.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.194+0000 7f26c3fff700 1 --2- 172.21.15.121:0/2645604433 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26c40ffab0 0x7f26c4068fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:00.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.194+0000 7f26c3fff700 1 -- 172.21.15.121:0/2645604433 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b00095f0 con 0x7f26c4100cb0 2024-04-05T21:26:00.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.194+0000 7f26c3fff700 1 --2- 172.21.15.121:0/2645604433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26c4100cb0 0x7f26c419d350 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f26b0009910 tx=0x7f26b000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:00.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.194+0000 7f26c1ffb700 1 -- 172.21.15.121:0/2645604433 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26b001d070 con 0x7f26c4100cb0 2024-04-05T21:26:00.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.194+0000 7f26cabef700 1 -- 172.21.15.121:0/2645604433 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c4196ef0 con 0x7f26c4100cb0 2024-04-05T21:26:00.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.194+0000 7f26c1ffb700 1 -- 172.21.15.121:0/2645604433 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f26b0005dc0 con 0x7f26c4100cb0 2024-04-05T21:26:00.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.194+0000 7f26c1ffb700 1 -- 172.21.15.121:0/2645604433 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26b0021410 con 0x7f26c4100cb0 2024-04-05T21:26:00.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.194+0000 7f26cabef700 1 -- 172.21.15.121:0/2645604433 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c4197460 con 0x7f26c4100cb0 2024-04-05T21:26:00.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.195+0000 7f26cabef700 1 -- 172.21.15.121:0/2645604433 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c41084d0 con 0x7f26c4100cb0 2024-04-05T21:26:00.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.196+0000 7f26c1ffb700 1 -- 172.21.15.121:0/2645604433 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f26b000f460 con 0x7f26c4100cb0 2024-04-05T21:26:00.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.197+0000 7f26c1ffb700 1 --2- 172.21.15.121:0/2645604433 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f26b406c4e0 0x7f26b406e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:00.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.197+0000 7f26c1ffb700 1 -- 172.21.15.121:0/2645604433 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f26b008c950 con 0x7f26c4100cb0 2024-04-05T21:26:00.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.197+0000 7f26c898b700 1 --2- 172.21.15.121:0/2645604433 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f26b406c4e0 0x7f26b406e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:00.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.198+0000 7f26c898b700 1 --2- 172.21.15.121:0/2645604433 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f26b406c4e0 0x7f26b406e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f26b8015fd0 tx=0x7f26b8017040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:00.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.203+0000 7f26c1ffb700 1 -- 172.21.15.121:0/2645604433 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f26b005abe0 con 0x7f26c4100cb0 2024-04-05T21:26:00.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.490+0000 7f26cabef700 1 -- 172.21.15.121:0/2645604433 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f26c4105600 con 0x7f26b406c4e0 2024-04-05T21:26:00.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.503+0000 7f26c1ffb700 1 -- 172.21.15.121:0/2645604433 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3504 (secure 0 0 0) 0x7f26c4105600 con 0x7f26b406c4e0 2024-04-05T21:26:00.504 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:26:00.504 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (6m) 103s ago 8m 26.2M - 0.25.0 c8568f914cd2 6594a12b2bc8 2024-04-05T21:26:00.505 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (8m) 103s ago 8m 7520k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:26:00.505 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (7m) 109s ago 7m 7470k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:26:00.505 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (8m) 103s ago 8m 6651k - 18.2.1 5be31c24972a 5bf35fe2ace4 2024-04-05T21:26:00.505 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (7m) 109s ago 7m 6651k - 18.2.1 5be31c24972a dcdfa6ae17bf 2024-04-05T21:26:00.505 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (6m) 103s ago 8m 84.5M - 9.4.7 954c08fa6188 ee38366c66e3 2024-04-05T21:26:00.505 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (2m) 103s ago 2m 12.9M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:26:00.505 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (2m) 103s ago 2m 16.4M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:26:00.505 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (2m) 109s ago 2m 16.6M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:26:00.505 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (2m) 109s ago 2m 16.8M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:26:00.505 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:9283,8765,8443 running (9m) 103s ago 9m 489M - 18.2.1 5be31c24972a d8cb1548d844 2024-04-05T21:26:00.505 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 starting - - - - 2024-04-05T21:26:00.505 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (9m) 103s ago 10m 46.8M 2048M 18.2.1 5be31c24972a 3c30ce99c006 2024-04-05T21:26:00.505 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (7m) 109s ago 7m 36.3M 2048M 18.2.1 5be31c24972a 8ecc9636670e 2024-04-05T21:26:00.506 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (8m) 103s ago 8m 18.4M - 1.5.0 0da6a335fe13 5ac35283f3e9 2024-04-05T21:26:00.506 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (7m) 109s ago 7m 16.3M - 1.5.0 0da6a335fe13 bc2ad104140f 2024-04-05T21:26:00.506 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (6m) 103s ago 6m 48.1M 3958M 18.2.1 5be31c24972a f6f3a45429a2 2024-04-05T21:26:00.506 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (5m) 103s ago 5m 50.7M 3958M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:26:00.506 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (5m) 103s ago 5m 47.7M 3958M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:26:00.506 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (4m) 109s ago 4m 49.0M 4982M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:26:00.506 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (4m) 109s ago 4m 43.6M 4982M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:26:00.506 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (3m) 109s ago 3m 44.3M 4982M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:26:00.506 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (6m) 103s ago 7m 56.4M - 2.43.0 a07b618ecd1d 9395525b1c3e 2024-04-05T21:26:00.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.506+0000 7f26cabef700 1 -- 172.21.15.121:0/2645604433 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f26b406c4e0 msgr2=0x7f26b406e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:00.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.506+0000 7f26cabef700 1 --2- 172.21.15.121:0/2645604433 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f26b406c4e0 0x7f26b406e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f26b8015fd0 tx=0x7f26b8017040 comp rx=0 tx=0).stop 2024-04-05T21:26:00.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.506+0000 7f26cabef700 1 -- 172.21.15.121:0/2645604433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26c4100cb0 msgr2=0x7f26c419d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:00.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.506+0000 7f26cabef700 1 --2- 172.21.15.121:0/2645604433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26c4100cb0 0x7f26c419d350 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f26b0009910 tx=0x7f26b000bd70 comp rx=0 tx=0).stop 2024-04-05T21:26:00.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.506+0000 7f26cabef700 1 -- 172.21.15.121:0/2645604433 shutdown_connections 2024-04-05T21:26:00.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.506+0000 7f26cabef700 1 --2- 172.21.15.121:0/2645604433 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f26b406c4e0 0x7f26b406e9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:00.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.506+0000 7f26cabef700 1 --2- 172.21.15.121:0/2645604433 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f26c40ffab0 0x7f26c4068fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:00.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.506+0000 7f26cabef700 1 --2- 172.21.15.121:0/2645604433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26c4100cb0 0x7f26c419d350 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:00.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.506+0000 7f26cabef700 1 -- 172.21.15.121:0/2645604433 >> 172.21.15.121:0/2645604433 conn(0x7f26c40fb030 msgr2=0x7f26c4103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:00.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.506+0000 7f26cabef700 1 -- 172.21.15.121:0/2645604433 shutdown_connections 2024-04-05T21:26:00.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.506+0000 7f26cabef700 1 -- 172.21.15.121:0/2645604433 wait complete. 2024-04-05T21:26:00.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:00 smithi188 ceph-mon[49715]: pgmap v261: 65 pgs: 65 active+clean; 148 MiB data, 746 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 4.3 MiB/s wr, 440 op/s 2024-04-05T21:26:00.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.641+0000 7f858358c700 1 -- 172.21.15.121:0/1257480854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f857c100cb0 msgr2=0x7f857c101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:00.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.641+0000 7f858358c700 1 --2- 172.21.15.121:0/1257480854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f857c100cb0 0x7f857c101110 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f856c009910 tx=0x7f856c009c20 comp rx=0 tx=0).stop 2024-04-05T21:26:00.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.641+0000 7f858358c700 1 -- 172.21.15.121:0/1257480854 shutdown_connections 2024-04-05T21:26:00.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.641+0000 7f858358c700 1 --2- 172.21.15.121:0/1257480854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f857c100cb0 0x7f857c101110 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:00.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.641+0000 7f858358c700 1 --2- 172.21.15.121:0/1257480854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f857c0ffab0 0x7f857c0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:00.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.641+0000 7f858358c700 1 -- 172.21.15.121:0/1257480854 >> 172.21.15.121:0/1257480854 conn(0x7f857c0fb050 msgr2=0x7f857c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:00.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.641+0000 7f858358c700 1 -- 172.21.15.121:0/1257480854 shutdown_connections 2024-04-05T21:26:00.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.641+0000 7f858358c700 1 -- 172.21.15.121:0/1257480854 wait complete. 2024-04-05T21:26:00.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.642+0000 7f858358c700 1 Processor -- start 2024-04-05T21:26:00.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.642+0000 7f858358c700 1 -- start start 2024-04-05T21:26:00.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.642+0000 7f858358c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f857c0ffab0 0x7f857c196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:00.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.642+0000 7f858358c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f857c100cb0 0x7f857c1993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:00.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.642+0000 7f858358c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857c069980 con 0x7f857c0ffab0 2024-04-05T21:26:00.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.642+0000 7f858358c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857c069ac0 con 0x7f857c100cb0 2024-04-05T21:26:00.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.643+0000 7f8581328700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f857c0ffab0 0x7f857c196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:00.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.643+0000 7f8581328700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f857c0ffab0 0x7f857c196e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40880/0 (socket says 172.21.15.121:40880) 2024-04-05T21:26:00.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.643+0000 7f8581328700 1 -- 172.21.15.121:0/3235561439 learned_addr learned my addr 172.21.15.121:0/3235561439 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:26:00.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.643+0000 7f8581328700 1 -- 172.21.15.121:0/3235561439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f857c100cb0 msgr2=0x7f857c1993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:00.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.643+0000 7f8581328700 1 --2- 172.21.15.121:0/3235561439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f857c100cb0 0x7f857c1993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:00.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.643+0000 7f8581328700 1 -- 172.21.15.121:0/3235561439 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c0095f0 con 0x7f857c0ffab0 2024-04-05T21:26:00.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.643+0000 7f8581328700 1 --2- 172.21.15.121:0/3235561439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f857c0ffab0 0x7f857c196e50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f857800b480 tx=0x7f857800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:00.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.643+0000 7f85727fc700 1 -- 172.21.15.121:0/3235561439 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85780118c0 con 0x7f857c0ffab0 2024-04-05T21:26:00.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.644+0000 7f85727fc700 1 -- 172.21.15.121:0/3235561439 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f8578010380 con 0x7f857c0ffab0 2024-04-05T21:26:00.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.644+0000 7f858358c700 1 -- 172.21.15.121:0/3235561439 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857c1998e0 con 0x7f857c0ffab0 2024-04-05T21:26:00.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.644+0000 7f85727fc700 1 -- 172.21.15.121:0/3235561439 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f857800f510 con 0x7f857c0ffab0 2024-04-05T21:26:00.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.644+0000 7f858358c700 1 -- 172.21.15.121:0/3235561439 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857c199eb0 con 0x7f857c0ffab0 2024-04-05T21:26:00.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.645+0000 7f858358c700 1 -- 172.21.15.121:0/3235561439 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f857c108570 con 0x7f857c0ffab0 2024-04-05T21:26:00.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.646+0000 7f85727fc700 1 -- 172.21.15.121:0/3235561439 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7f85780104f0 con 0x7f857c0ffab0 2024-04-05T21:26:00.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.646+0000 7f85727fc700 1 --2- 172.21.15.121:0/3235561439 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f856806c530 0x7f856806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:00.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.646+0000 7f85727fc700 1 -- 172.21.15.121:0/3235561439 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f857808b970 con 0x7f857c0ffab0 2024-04-05T21:26:00.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.646+0000 7f8580b27700 1 --2- 172.21.15.121:0/3235561439 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f856806c530 0x7f856806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:00.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.647+0000 7f8580b27700 1 --2- 172.21.15.121:0/3235561439 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f856806c530 0x7f856806e9f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f856c00b5c0 tx=0x7f856c005960 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:00.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:00.652+0000 7f85727fc700 1 -- 172.21.15.121:0/3235561439 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7f8578059b60 con 0x7f857c0ffab0 2024-04-05T21:26:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:00 smithi121 ceph-mon[39723]: pgmap v261: 65 pgs: 65 active+clean; 148 MiB data, 746 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 4.3 MiB/s wr, 440 op/s 2024-04-05T21:26:01.079 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:26:01.079 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:26:01.079 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-05T21:26:01.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:26:01.080 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:26:01.080 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-05T21:26:01.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:26:01.080 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:26:01.080 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-05T21:26:01.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:26:01.080 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:26:01.080 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:26:01.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:26:01.080 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:26:01.080 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 14 2024-04-05T21:26:01.080 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:26:01.081 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:26:01.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.077+0000 7f858358c700 1 -- 172.21.15.121:0/3235561439 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f857c108730 con 0x7f857c0ffab0 2024-04-05T21:26:01.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.078+0000 7f85727fc700 1 -- 172.21.15.121:0/3235561439 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+558 (secure 0 0 0) 0x7f85780107a0 con 0x7f857c0ffab0 2024-04-05T21:26:01.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.082+0000 7f858358c700 1 -- 172.21.15.121:0/3235561439 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f856806c530 msgr2=0x7f856806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:01.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.082+0000 7f858358c700 1 --2- 172.21.15.121:0/3235561439 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f856806c530 0x7f856806e9f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f856c00b5c0 tx=0x7f856c005960 comp rx=0 tx=0).stop 2024-04-05T21:26:01.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.082+0000 7f858358c700 1 -- 172.21.15.121:0/3235561439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f857c0ffab0 msgr2=0x7f857c196e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:01.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.082+0000 7f858358c700 1 --2- 172.21.15.121:0/3235561439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f857c0ffab0 0x7f857c196e50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f857800b480 tx=0x7f857800b840 comp rx=0 tx=0).stop 2024-04-05T21:26:01.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.082+0000 7f858358c700 1 -- 172.21.15.121:0/3235561439 shutdown_connections 2024-04-05T21:26:01.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.082+0000 7f858358c700 1 --2- 172.21.15.121:0/3235561439 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7f856806c530 0x7f856806e9f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:01.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.082+0000 7f858358c700 1 --2- 172.21.15.121:0/3235561439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f857c0ffab0 0x7f857c196e50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:01.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.082+0000 7f858358c700 1 --2- 172.21.15.121:0/3235561439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f857c100cb0 0x7f857c1993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:01.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.082+0000 7f858358c700 1 -- 172.21.15.121:0/3235561439 >> 172.21.15.121:0/3235561439 conn(0x7f857c0fb050 msgr2=0x7f857c103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:01.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.085+0000 7f858358c700 1 -- 172.21.15.121:0/3235561439 shutdown_connections 2024-04-05T21:26:01.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.085+0000 7f858358c700 1 -- 172.21.15.121:0/3235561439 wait complete. 2024-04-05T21:26:01.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.234+0000 7ff641553700 1 -- 172.21.15.121:0/2414356915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff63c101640 msgr2=0x7ff63c103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:01.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.234+0000 7ff641553700 1 --2- 172.21.15.121:0/2414356915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff63c101640 0x7ff63c103a30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff62c009910 tx=0x7ff62c009c20 comp rx=0 tx=0).stop 2024-04-05T21:26:01.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.234+0000 7ff641553700 1 -- 172.21.15.121:0/2414356915 shutdown_connections 2024-04-05T21:26:01.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.234+0000 7ff641553700 1 --2- 172.21.15.121:0/2414356915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff63c103f70 0x7ff63c106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:01.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.234+0000 7ff641553700 1 --2- 172.21.15.121:0/2414356915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff63c101640 0x7ff63c103a30 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:01.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.234+0000 7ff641553700 1 -- 172.21.15.121:0/2414356915 >> 172.21.15.121:0/2414356915 conn(0x7ff63c0fb030 msgr2=0x7ff63c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:01.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.234+0000 7ff641553700 1 -- 172.21.15.121:0/2414356915 shutdown_connections 2024-04-05T21:26:01.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.234+0000 7ff641553700 1 -- 172.21.15.121:0/2414356915 wait complete. 2024-04-05T21:26:01.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.235+0000 7ff641553700 1 Processor -- start 2024-04-05T21:26:01.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.235+0000 7ff641553700 1 -- start start 2024-04-05T21:26:01.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.235+0000 7ff641553700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff63c101640 0x7ff63c100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:01.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.235+0000 7ff641553700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff63c103f70 0x7ff63c0ff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:01.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.235+0000 7ff641553700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c199f70 con 0x7ff63c103f70 2024-04-05T21:26:01.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.235+0000 7ff641553700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c19a0b0 con 0x7ff63c101640 2024-04-05T21:26:01.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.235+0000 7ff63a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff63c103f70 0x7ff63c0ff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:01.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.236+0000 7ff63a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff63c103f70 0x7ff63c0ff5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40898/0 (socket says 172.21.15.121:40898) 2024-04-05T21:26:01.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.236+0000 7ff63a7fc700 1 -- 172.21.15.121:0/827995581 learned_addr learned my addr 172.21.15.121:0/827995581 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:26:01.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.236+0000 7ff63affd700 1 --2- 172.21.15.121:0/827995581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff63c101640 0x7ff63c100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:01.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.236+0000 7ff63a7fc700 1 -- 172.21.15.121:0/827995581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff63c101640 msgr2=0x7ff63c100fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:01.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.236+0000 7ff63a7fc700 1 --2- 172.21.15.121:0/827995581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff63c101640 0x7ff63c100fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:01.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.236+0000 7ff63a7fc700 1 -- 172.21.15.121:0/827995581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62c0095f0 con 0x7ff63c103f70 2024-04-05T21:26:01.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.236+0000 7ff63a7fc700 1 --2- 172.21.15.121:0/827995581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff63c103f70 0x7ff63c0ff5f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff63000b4b0 tx=0x7ff63000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:01.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.236+0000 7ff623fff700 1 -- 172.21.15.121:0/827995581 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff630011800 con 0x7ff63c103f70 2024-04-05T21:26:01.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.236+0000 7ff641553700 1 -- 172.21.15.121:0/827995581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63c0ffb30 con 0x7ff63c103f70 2024-04-05T21:26:01.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.236+0000 7ff623fff700 1 -- 172.21.15.121:0/827995581 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7ff630011df0 con 0x7ff63c103f70 2024-04-05T21:26:01.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.236+0000 7ff623fff700 1 -- 172.21.15.121:0/827995581 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff630010e50 con 0x7ff63c103f70 2024-04-05T21:26:01.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.236+0000 7ff641553700 1 -- 172.21.15.121:0/827995581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63c1000d0 con 0x7ff63c103f70 2024-04-05T21:26:01.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.238+0000 7ff641553700 1 -- 172.21.15.121:0/827995581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63c190460 con 0x7ff63c103f70 2024-04-05T21:26:01.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.238+0000 7ff623fff700 1 -- 172.21.15.121:0/827995581 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 90314+0+0 (secure 0 0 0) 0x7ff630010380 con 0x7ff63c103f70 2024-04-05T21:26:01.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.239+0000 7ff623fff700 1 --2- 172.21.15.121:0/827995581 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff62406c530 0x7ff62406e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:01.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.239+0000 7ff623fff700 1 -- 172.21.15.121:0/827995581 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff63008b370 con 0x7ff63c103f70 2024-04-05T21:26:01.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.239+0000 7ff63affd700 1 --2- 172.21.15.121:0/827995581 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff62406c530 0x7ff62406e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:01.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.240+0000 7ff63affd700 1 --2- 172.21.15.121:0/827995581 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff62406c530 0x7ff62406e9f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00b5c0 tx=0x7ff62c005fb0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:01.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.245+0000 7ff623fff700 1 -- 172.21.15.121:0/827995581 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180422 (secure 0 0 0) 0x7ff630059580 con 0x7ff63c103f70 2024-04-05T21:26:01.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.530+0000 7ff641553700 1 -- 172.21.15.121:0/827995581 --> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff63c0611d0 con 0x7ff62406c530 2024-04-05T21:26:01.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.536+0000 7ff623fff700 1 -- 172.21.15.121:0/827995581 <== mgr.14211 v2:172.21.15.121:6800/2735531747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7ff63c0611d0 con 0x7ff62406c530 2024-04-05T21:26:01.537 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:26:01.537 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:26:01.537 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:26:01.537 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T21:26:01.537 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [], 2024-04-05T21:26:01.538 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "1/2 daemons upgraded", 2024-04-05T21:26:01.538 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mgr daemons", 2024-04-05T21:26:01.538 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:26:01.538 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:26:01.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.539+0000 7ff641553700 1 -- 172.21.15.121:0/827995581 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff62406c530 msgr2=0x7ff62406e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:01.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.539+0000 7ff641553700 1 --2- 172.21.15.121:0/827995581 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff62406c530 0x7ff62406e9f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff62c00b5c0 tx=0x7ff62c005fb0 comp rx=0 tx=0).stop 2024-04-05T21:26:01.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.539+0000 7ff641553700 1 -- 172.21.15.121:0/827995581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff63c103f70 msgr2=0x7ff63c0ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:01.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.539+0000 7ff641553700 1 --2- 172.21.15.121:0/827995581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff63c103f70 0x7ff63c0ff5f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff63000b4b0 tx=0x7ff63000b7c0 comp rx=0 tx=0).stop 2024-04-05T21:26:01.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.541+0000 7ff641553700 1 -- 172.21.15.121:0/827995581 shutdown_connections 2024-04-05T21:26:01.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.541+0000 7ff641553700 1 --2- 172.21.15.121:0/827995581 >> [v2:172.21.15.121:6800/2735531747,v1:172.21.15.121:6801/2735531747] conn(0x7ff62406c530 0x7ff62406e9f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:01.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.541+0000 7ff641553700 1 --2- 172.21.15.121:0/827995581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff63c101640 0x7ff63c100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:01.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.541+0000 7ff641553700 1 --2- 172.21.15.121:0/827995581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff63c103f70 0x7ff63c0ff5f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:01.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.541+0000 7ff641553700 1 -- 172.21.15.121:0/827995581 >> 172.21.15.121:0/827995581 conn(0x7ff63c0fb030 msgr2=0x7ff63c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:01.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.541+0000 7ff641553700 1 -- 172.21.15.121:0/827995581 shutdown_connections 2024-04-05T21:26:01.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:01.541+0000 7ff641553700 1 -- 172.21.15.121:0/827995581 wait complete. 2024-04-05T21:26:01.627 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:01 smithi188 ceph-mon[49715]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:01.627 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:01 smithi188 ceph-mon[49715]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:01.627 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:01 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:01.627 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:01 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:01.627 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:01 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/3235561439' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:26:01.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:01 smithi121 ceph-mon[39723]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:01.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:01 smithi121 ceph-mon[39723]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:01.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:01 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:01.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:01 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:01.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:01 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3235561439' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:26:02.486 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:02 smithi188 ceph-mon[49715]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:02.486 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:02 smithi188 ceph-mon[49715]: pgmap v262: 65 pgs: 65 active+clean; 151 MiB data, 739 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 379 op/s 2024-04-05T21:26:02.486 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:02 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:02.486 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:02 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:02.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:02 smithi121 ceph-mon[39723]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:02.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:02 smithi121 ceph-mon[39723]: pgmap v262: 65 pgs: 65 active+clean; 151 MiB data, 739 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 379 op/s 2024-04-05T21:26:02.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:02 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:02.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:02 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:03.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:03 smithi188 ceph-mon[49715]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:03.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:03 smithi121 ceph-mon[39723]: from='client.14594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:04.689 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:04 smithi188 ceph-mon[49715]: pgmap v263: 65 pgs: 65 active+clean; 156 MiB data, 763 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.8 MiB/s wr, 383 op/s 2024-04-05T21:26:04.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:04 smithi121 ceph-mon[39723]: pgmap v263: 65 pgs: 65 active+clean; 156 MiB data, 763 MiB used, 536 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.8 MiB/s wr, 383 op/s 2024-04-05T21:26:05.764 INFO:tasks.workunit.client.1.smithi188.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-05T21:26:05.773 INFO:tasks.workunit.client.1.smithi188.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-05T21:26:05.773 INFO:tasks.workunit.client.1.smithi188.stderr:+ make 2024-04-05T21:26:05.924 INFO:tasks.workunit.client.1.smithi188.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-05T21:26:06.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:06 smithi188 ceph-mon[49715]: pgmap v264: 65 pgs: 65 active+clean; 174 MiB data, 830 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 439 op/s 2024-04-05T21:26:06.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:06 smithi121 ceph-mon[39723]: pgmap v264: 65 pgs: 65 active+clean; 174 MiB data, 830 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 439 op/s 2024-04-05T21:26:07.004 INFO:tasks.workunit.client.1.smithi188.stderr:++ readlink -f fsstress 2024-04-05T21:26:07.005 INFO:tasks.workunit.client.1.smithi188.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-05T21:26:07.005 INFO:tasks.workunit.client.1.smithi188.stderr:+ popd 2024-04-05T21:26:07.006 INFO:tasks.workunit.client.1.smithi188.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-04-05T21:26:07.006 INFO:tasks.workunit.client.1.smithi188.stderr:+ popd 2024-04-05T21:26:07.007 INFO:tasks.workunit.client.1.smithi188.stdout:~/cephtest/mnt.1/client.1/tmp 2024-04-05T21:26:07.007 INFO:tasks.workunit.client.1.smithi188.stderr:++ mktemp -d -p . 2024-04-05T21:26:07.010 INFO:tasks.workunit.client.1.smithi188.stderr:+ T=./tmp.2FUf7WJ9nl 2024-04-05T21:26:07.010 INFO:tasks.workunit.client.1.smithi188.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.2FUf7WJ9nl -l 1 -n 1000 -p 10 -v 2024-04-05T21:26:07.013 INFO:tasks.workunit.client.1.smithi188.stdout:seed = 1712347750 2024-04-05T21:26:07.016 INFO:tasks.workunit.client.1.smithi188.stdout:0/0: chown . 2492 1 2024-04-05T21:26:07.016 INFO:tasks.workunit.client.1.smithi188.stdout:0/1: dwrite - no filename 2024-04-05T21:26:07.016 INFO:tasks.workunit.client.1.smithi188.stdout:0/2: dread - no filename 2024-04-05T21:26:07.017 INFO:tasks.workunit.client.1.smithi188.stdout:1/0: write - no filename 2024-04-05T21:26:07.017 INFO:tasks.workunit.client.1.smithi188.stdout:1/1: stat - no entries 2024-04-05T21:26:07.018 INFO:tasks.workunit.client.1.smithi188.stdout:0/3: symlink l0 0 2024-04-05T21:26:07.019 INFO:tasks.workunit.client.1.smithi188.stdout:0/4: creat f1 x:0 0 0 2024-04-05T21:26:07.020 INFO:tasks.workunit.client.1.smithi188.stdout:1/2: creat f0 x:0 0 0 2024-04-05T21:26:07.021 INFO:tasks.workunit.client.1.smithi188.stdout:1/3: write f0 [217369,39925] 0 2024-04-05T21:26:07.022 INFO:tasks.workunit.client.1.smithi188.stdout:0/5: mknod c2 0 2024-04-05T21:26:07.022 INFO:tasks.workunit.client.1.smithi188.stdout:2/0: getdents . 0 2024-04-05T21:26:07.024 INFO:tasks.workunit.client.1.smithi188.stdout:3/0: mknod c0 0 2024-04-05T21:26:07.025 INFO:tasks.workunit.client.1.smithi188.stdout:1/4: dread f0 [0,4194304] 0 2024-04-05T21:26:07.029 INFO:tasks.workunit.client.1.smithi188.stdout:3/1: creat f1 x:0 0 0 2024-04-05T21:26:07.029 INFO:tasks.workunit.client.1.smithi188.stdout:2/1: mknod c0 0 2024-04-05T21:26:07.030 INFO:tasks.workunit.client.1.smithi188.stdout:4/0: mkdir d0 0 2024-04-05T21:26:07.031 INFO:tasks.workunit.client.1.smithi188.stdout:1/5: unlink f0 0 2024-04-05T21:26:07.037 INFO:tasks.workunit.client.1.smithi188.stdout:3/2: unlink c0 0 2024-04-05T21:26:07.038 INFO:tasks.workunit.client.1.smithi188.stdout:2/2: rename c0 to c1 0 2024-04-05T21:26:07.039 INFO:tasks.workunit.client.1.smithi188.stdout:7/0: unlink - no file 2024-04-05T21:26:07.039 INFO:tasks.workunit.client.1.smithi188.stdout:7/1: chown . 222475 1 2024-04-05T21:26:07.039 INFO:tasks.workunit.client.1.smithi188.stdout:7/2: dwrite - no filename 2024-04-05T21:26:07.039 INFO:tasks.workunit.client.1.smithi188.stdout:7/3: readlink - no filename 2024-04-05T21:26:07.039 INFO:tasks.workunit.client.1.smithi188.stdout:7/4: chown . 44482 1 2024-04-05T21:26:07.039 INFO:tasks.workunit.client.1.smithi188.stdout:7/5: write - no filename 2024-04-05T21:26:07.039 INFO:tasks.workunit.client.1.smithi188.stdout:7/6: dread - no filename 2024-04-05T21:26:07.039 INFO:tasks.workunit.client.1.smithi188.stdout:7/7: truncate - no filename 2024-04-05T21:26:07.039 INFO:tasks.workunit.client.1.smithi188.stdout:1/6: mkdir d1 0 2024-04-05T21:26:07.040 INFO:tasks.workunit.client.1.smithi188.stdout:5/0: mknod c0 0 2024-04-05T21:26:07.040 INFO:tasks.workunit.client.1.smithi188.stdout:5/1: truncate - no filename 2024-04-05T21:26:07.040 INFO:tasks.workunit.client.1.smithi188.stdout:2/3: mknod c2 0 2024-04-05T21:26:07.040 INFO:tasks.workunit.client.1.smithi188.stdout:2/4: readlink - no filename 2024-04-05T21:26:07.041 INFO:tasks.workunit.client.1.smithi188.stdout:6/0: symlink l0 0 2024-04-05T21:26:07.041 INFO:tasks.workunit.client.1.smithi188.stdout:5/2: chown c0 13847 1 2024-04-05T21:26:07.041 INFO:tasks.workunit.client.1.smithi188.stdout:5/3: rmdir - no directory 2024-04-05T21:26:07.041 INFO:tasks.workunit.client.1.smithi188.stdout:5/4: dwrite - no filename 2024-04-05T21:26:07.041 INFO:tasks.workunit.client.1.smithi188.stdout:5/5: read - no filename 2024-04-05T21:26:07.041 INFO:tasks.workunit.client.1.smithi188.stdout:5/6: dwrite - no filename 2024-04-05T21:26:07.041 INFO:tasks.workunit.client.1.smithi188.stdout:5/7: write - no filename 2024-04-05T21:26:07.042 INFO:tasks.workunit.client.1.smithi188.stdout:4/1: symlink d0/l1 0 2024-04-05T21:26:07.044 INFO:tasks.workunit.client.1.smithi188.stdout:3/3: dwrite f1 [0,4194304] 0 2024-04-05T21:26:07.044 INFO:tasks.workunit.client.1.smithi188.stdout:8/0: rename - no filename 2024-04-05T21:26:07.045 INFO:tasks.workunit.client.1.smithi188.stdout:8/1: chown . 5 1 2024-04-05T21:26:07.047 INFO:tasks.workunit.client.1.smithi188.stdout:5/8: mknod c1 0 2024-04-05T21:26:07.047 INFO:tasks.workunit.client.1.smithi188.stdout:5/9: write - no filename 2024-04-05T21:26:07.048 INFO:tasks.workunit.client.1.smithi188.stdout:7/8: mknod c0 0 2024-04-05T21:26:07.048 INFO:tasks.workunit.client.1.smithi188.stdout:7/9: rmdir - no directory 2024-04-05T21:26:07.051 INFO:tasks.workunit.client.1.smithi188.stdout:1/7: mknod d1/c2 0 2024-04-05T21:26:07.051 INFO:tasks.workunit.client.1.smithi188.stdout:1/8: dread - no filename 2024-04-05T21:26:07.051 INFO:tasks.workunit.client.1.smithi188.stdout:9/0: stat - no entries 2024-04-05T21:26:07.051 INFO:tasks.workunit.client.1.smithi188.stdout:9/1: fdatasync - no filename 2024-04-05T21:26:07.052 INFO:tasks.workunit.client.1.smithi188.stdout:9/2: write - no filename 2024-04-05T21:26:07.052 INFO:tasks.workunit.client.1.smithi188.stdout:9/3: rmdir - no directory 2024-04-05T21:26:07.052 INFO:tasks.workunit.client.1.smithi188.stdout:9/4: dwrite - no filename 2024-04-05T21:26:07.052 INFO:tasks.workunit.client.1.smithi188.stdout:9/5: unlink - no file 2024-04-05T21:26:07.052 INFO:tasks.workunit.client.1.smithi188.stdout:9/6: rename - no filename 2024-04-05T21:26:07.052 INFO:tasks.workunit.client.1.smithi188.stdout:3/4: symlink l2 0 2024-04-05T21:26:07.054 INFO:tasks.workunit.client.1.smithi188.stdout:5/10: creat f2 x:0 0 0 2024-04-05T21:26:07.055 INFO:tasks.workunit.client.1.smithi188.stdout:5/11: dread - f2 zero size 2024-04-05T21:26:07.055 INFO:tasks.workunit.client.1.smithi188.stdout:7/10: mknod c1 0 2024-04-05T21:26:07.055 INFO:tasks.workunit.client.1.smithi188.stdout:7/11: write - no filename 2024-04-05T21:26:07.055 INFO:tasks.workunit.client.1.smithi188.stdout:7/12: readlink - no filename 2024-04-05T21:26:07.056 INFO:tasks.workunit.client.1.smithi188.stdout:5/12: truncate f2 558729 0 2024-04-05T21:26:07.056 INFO:tasks.workunit.client.1.smithi188.stdout:8/2: symlink l0 0 2024-04-05T21:26:07.057 INFO:tasks.workunit.client.1.smithi188.stdout:8/3: truncate - no filename 2024-04-05T21:26:07.058 INFO:tasks.workunit.client.1.smithi188.stdout:1/9: creat d1/f3 x:0 0 0 2024-04-05T21:26:07.060 INFO:tasks.workunit.client.1.smithi188.stdout:3/5: creat f3 x:0 0 0 2024-04-05T21:26:07.060 INFO:tasks.workunit.client.1.smithi188.stdout:6/1: link l0 l1 0 2024-04-05T21:26:07.060 INFO:tasks.workunit.client.1.smithi188.stdout:6/2: write - no filename 2024-04-05T21:26:07.061 INFO:tasks.workunit.client.1.smithi188.stdout:7/13: symlink l2 0 2024-04-05T21:26:07.062 INFO:tasks.workunit.client.1.smithi188.stdout:3/6: write f3 [397449,60584] 0 2024-04-05T21:26:07.063 INFO:tasks.workunit.client.1.smithi188.stdout:8/4: rename l0 to l1 0 2024-04-05T21:26:07.064 INFO:tasks.workunit.client.1.smithi188.stdout:8/5: dread - no filename 2024-04-05T21:26:07.064 INFO:tasks.workunit.client.1.smithi188.stdout:9/7: mknod c0 0 2024-04-05T21:26:07.066 INFO:tasks.workunit.client.1.smithi188.stdout:1/10: mkdir d1/d4 0 2024-04-05T21:26:07.066 INFO:tasks.workunit.client.1.smithi188.stdout:3/7: read f3 [365301,30573] 0 2024-04-05T21:26:07.069 INFO:tasks.workunit.client.1.smithi188.stdout:7/14: mkdir d3 0 2024-04-05T21:26:07.069 INFO:tasks.workunit.client.1.smithi188.stdout:5/13: dwrite f2 [0,4194304] 0 2024-04-05T21:26:07.069 INFO:tasks.workunit.client.1.smithi188.stdout:1/11: truncate d1/f3 265914 0 2024-04-05T21:26:07.071 INFO:tasks.workunit.client.1.smithi188.stdout:7/15: chown l2 0 1 2024-04-05T21:26:07.071 INFO:tasks.workunit.client.1.smithi188.stdout:7/16: write - no filename 2024-04-05T21:26:07.071 INFO:tasks.workunit.client.1.smithi188.stdout:5/14: write f2 [3387439,109361] 0 2024-04-05T21:26:07.072 INFO:tasks.workunit.client.1.smithi188.stdout:5/15: chown f2 7 1 2024-04-05T21:26:07.076 INFO:tasks.workunit.client.1.smithi188.stdout:6/3: unlink l0 0 2024-04-05T21:26:07.077 INFO:tasks.workunit.client.1.smithi188.stdout:6/4: write - no filename 2024-04-05T21:26:07.077 INFO:tasks.workunit.client.1.smithi188.stdout:6/5: read - no filename 2024-04-05T21:26:07.077 INFO:tasks.workunit.client.1.smithi188.stdout:6/6: read - no filename 2024-04-05T21:26:07.081 INFO:tasks.workunit.client.1.smithi188.stdout:5/16: symlink l3 0 2024-04-05T21:26:07.082 INFO:tasks.workunit.client.1.smithi188.stdout:5/17: rmdir - no directory 2024-04-05T21:26:07.083 INFO:tasks.workunit.client.1.smithi188.stdout:5/18: write f2 [2650532,113903] 0 2024-04-05T21:26:07.084 INFO:tasks.workunit.client.1.smithi188.stdout:6/7: mknod c2 0 2024-04-05T21:26:07.085 INFO:tasks.workunit.client.1.smithi188.stdout:7/17: mkdir d3/d4 0 2024-04-05T21:26:07.086 INFO:tasks.workunit.client.1.smithi188.stdout:7/18: chown c0 24393707 1 2024-04-05T21:26:07.087 INFO:tasks.workunit.client.1.smithi188.stdout:1/12: mkdir d1/d4/d5 0 2024-04-05T21:26:07.088 INFO:tasks.workunit.client.1.smithi188.stdout:1/13: chown d1/d4 54392 1 2024-04-05T21:26:07.089 INFO:tasks.workunit.client.1.smithi188.stdout:5/19: rename c0 to c4 0 2024-04-05T21:26:07.090 INFO:tasks.workunit.client.1.smithi188.stdout:1/14: dread d1/f3 [0,4194304] 0 2024-04-05T21:26:07.090 INFO:tasks.workunit.client.1.smithi188.stdout:6/8: mknod c3 0 2024-04-05T21:26:07.095 INFO:tasks.workunit.client.1.smithi188.stdout:5/20: dread f2 [0,4194304] 0 2024-04-05T21:26:07.095 INFO:tasks.workunit.client.1.smithi188.stdout:6/9: symlink l4 0 2024-04-05T21:26:07.096 INFO:tasks.workunit.client.1.smithi188.stdout:1/15: mknod d1/c6 0 2024-04-05T21:26:07.096 INFO:tasks.workunit.client.1.smithi188.stdout:7/19: mknod d3/d4/c5 0 2024-04-05T21:26:07.097 INFO:tasks.workunit.client.1.smithi188.stdout:7/20: truncate - no filename 2024-04-05T21:26:07.098 INFO:tasks.workunit.client.1.smithi188.stdout:6/10: creat f5 x:0 0 0 2024-04-05T21:26:07.098 INFO:tasks.workunit.client.1.smithi188.stdout:6/11: rmdir - no directory 2024-04-05T21:26:07.099 INFO:tasks.workunit.client.1.smithi188.stdout:6/12: chown f5 166509 1 2024-04-05T21:26:07.099 INFO:tasks.workunit.client.1.smithi188.stdout:6/13: fdatasync f5 0 2024-04-05T21:26:07.099 INFO:tasks.workunit.client.1.smithi188.stdout:6/14: dread - f5 zero size 2024-04-05T21:26:07.101 INFO:tasks.workunit.client.1.smithi188.stdout:7/21: creat d3/f6 x:0 0 0 2024-04-05T21:26:07.102 INFO:tasks.workunit.client.1.smithi188.stdout:6/15: mkdir d6 0 2024-04-05T21:26:07.103 INFO:tasks.workunit.client.1.smithi188.stdout:7/22: write d3/f6 [262283,62420] 0 2024-04-05T21:26:07.105 INFO:tasks.workunit.client.1.smithi188.stdout:6/16: mkdir d6/d7 0 2024-04-05T21:26:07.106 INFO:tasks.workunit.client.1.smithi188.stdout:6/17: read - f5 zero size 2024-04-05T21:26:07.109 INFO:tasks.workunit.client.1.smithi188.stdout:7/23: rename d3/f6 to d3/d4/f7 0 2024-04-05T21:26:07.110 INFO:tasks.workunit.client.1.smithi188.stdout:1/16: dwrite d1/f3 [0,4194304] 0 2024-04-05T21:26:07.112 INFO:tasks.workunit.client.1.smithi188.stdout:6/18: creat d6/f8 x:0 0 0 2024-04-05T21:26:07.113 INFO:tasks.workunit.client.1.smithi188.stdout:1/17: rmdir d1/d4 39 2024-04-05T21:26:07.118 INFO:tasks.workunit.client.1.smithi188.stdout:6/19: link l1 d6/l9 0 2024-04-05T21:26:07.118 INFO:tasks.workunit.client.1.smithi188.stdout:6/20: chown c3 221069588 1 2024-04-05T21:26:07.119 INFO:tasks.workunit.client.1.smithi188.stdout:1/18: dread d1/f3 [0,4194304] 0 2024-04-05T21:26:07.119 INFO:tasks.workunit.client.1.smithi188.stdout:6/21: read - d6/f8 zero size 2024-04-05T21:26:07.120 INFO:tasks.workunit.client.1.smithi188.stdout:1/19: mkdir d1/d7 0 2024-04-05T21:26:07.124 INFO:tasks.workunit.client.1.smithi188.stdout:1/20: dread d1/f3 [0,4194304] 0 2024-04-05T21:26:07.125 INFO:tasks.workunit.client.1.smithi188.stdout:1/21: chown d1 100455 1 2024-04-05T21:26:07.125 INFO:tasks.workunit.client.1.smithi188.stdout:1/22: read d1/f3 [2744360,8822] 0 2024-04-05T21:26:07.126 INFO:tasks.workunit.client.1.smithi188.stdout:1/23: chown d1/f3 0 1 2024-04-05T21:26:07.139 INFO:tasks.workunit.client.1.smithi188.stdout:7/24: fdatasync d3/d4/f7 0 2024-04-05T21:26:07.143 INFO:tasks.workunit.client.1.smithi188.stdout:7/25: creat d3/d4/f8 x:0 0 0 2024-04-05T21:26:07.145 INFO:tasks.workunit.client.1.smithi188.stdout:7/26: creat d3/f9 x:0 0 0 2024-04-05T21:26:07.146 INFO:tasks.workunit.client.1.smithi188.stdout:7/27: dread - d3/d4/f8 zero size 2024-04-05T21:26:07.146 INFO:tasks.workunit.client.1.smithi188.stdout:7/28: chown d3 1167576 1 2024-04-05T21:26:07.158 INFO:tasks.workunit.client.1.smithi188.stdout:0/6: getdents . 0 2024-04-05T21:26:07.160 INFO:tasks.workunit.client.1.smithi188.stdout:0/7: creat f3 x:0 0 0 2024-04-05T21:26:07.163 INFO:tasks.workunit.client.1.smithi188.stdout:0/8: unlink l0 0 2024-04-05T21:26:07.164 INFO:tasks.workunit.client.1.smithi188.stdout:0/9: mkdir d4 0 2024-04-05T21:26:07.170 INFO:tasks.workunit.client.1.smithi188.stdout:2/5: unlink c1 0 2024-04-05T21:26:07.173 INFO:tasks.workunit.client.1.smithi188.stdout:2/6: unlink c2 0 2024-04-05T21:26:07.173 INFO:tasks.workunit.client.1.smithi188.stdout:2/7: rmdir - no directory 2024-04-05T21:26:07.174 INFO:tasks.workunit.client.1.smithi188.stdout:0/10: dwrite f3 [0,4194304] 0 2024-04-05T21:26:07.175 INFO:tasks.workunit.client.1.smithi188.stdout:0/11: dread - f1 zero size 2024-04-05T21:26:07.177 INFO:tasks.workunit.client.1.smithi188.stdout:2/8: creat f3 x:0 0 0 2024-04-05T21:26:07.178 INFO:tasks.workunit.client.1.smithi188.stdout:2/9: stat f3 0 2024-04-05T21:26:07.181 INFO:tasks.workunit.client.1.smithi188.stdout:2/10: truncate f3 1002604 0 2024-04-05T21:26:07.184 INFO:tasks.workunit.client.1.smithi188.stdout:1/24: fsync d1/f3 0 2024-04-05T21:26:07.186 INFO:tasks.workunit.client.1.smithi188.stdout:8/6: getdents . 0 2024-04-05T21:26:07.186 INFO:tasks.workunit.client.1.smithi188.stdout:8/7: dwrite - no filename 2024-04-05T21:26:07.186 INFO:tasks.workunit.client.1.smithi188.stdout:8/8: write - no filename 2024-04-05T21:26:07.186 INFO:tasks.workunit.client.1.smithi188.stdout:8/9: dwrite - no filename 2024-04-05T21:26:07.187 INFO:tasks.workunit.client.1.smithi188.stdout:2/11: dread f3 [0,4194304] 0 2024-04-05T21:26:07.190 INFO:tasks.workunit.client.1.smithi188.stdout:0/12: dwrite f3 [0,4194304] 0 2024-04-05T21:26:07.190 INFO:tasks.workunit.client.1.smithi188.stdout:8/10: chown l1 61950 1 2024-04-05T21:26:07.191 INFO:tasks.workunit.client.1.smithi188.stdout:8/11: rmdir - no directory 2024-04-05T21:26:07.191 INFO:tasks.workunit.client.1.smithi188.stdout:8/12: fdatasync - no filename 2024-04-05T21:26:07.191 INFO:tasks.workunit.client.1.smithi188.stdout:0/13: write f3 [481409,53404] 0 2024-04-05T21:26:07.192 INFO:tasks.workunit.client.1.smithi188.stdout:1/25: unlink d1/c2 0 2024-04-05T21:26:07.193 INFO:tasks.workunit.client.1.smithi188.stdout:3/8: truncate f1 2808212 0 2024-04-05T21:26:07.193 INFO:tasks.workunit.client.1.smithi188.stdout:1/26: truncate d1/f3 4210589 0 2024-04-05T21:26:07.197 INFO:tasks.workunit.client.1.smithi188.stdout:2/12: creat f4 x:0 0 0 2024-04-05T21:26:07.197 INFO:tasks.workunit.client.1.smithi188.stdout:2/13: rmdir - no directory 2024-04-05T21:26:07.198 INFO:tasks.workunit.client.1.smithi188.stdout:2/14: dread - f4 zero size 2024-04-05T21:26:07.200 INFO:tasks.workunit.client.1.smithi188.stdout:1/27: dread d1/f3 [0,4194304] 0 2024-04-05T21:26:07.200 INFO:tasks.workunit.client.1.smithi188.stdout:8/13: creat f2 x:0 0 0 2024-04-05T21:26:07.201 INFO:tasks.workunit.client.1.smithi188.stdout:8/14: chown l1 691 1 2024-04-05T21:26:07.205 INFO:tasks.workunit.client.1.smithi188.stdout:2/15: write f3 [730130,74104] 0 2024-04-05T21:26:07.206 INFO:tasks.workunit.client.1.smithi188.stdout:0/14: getdents d4 0 2024-04-05T21:26:07.206 INFO:tasks.workunit.client.1.smithi188.stdout:0/15: fdatasync f1 0 2024-04-05T21:26:07.209 INFO:tasks.workunit.client.1.smithi188.stdout:2/16: mkdir d5 0 2024-04-05T21:26:07.213 INFO:tasks.workunit.client.1.smithi188.stdout:1/28: symlink d1/d4/d5/l8 0 2024-04-05T21:26:07.214 INFO:tasks.workunit.client.1.smithi188.stdout:0/16: mknod d4/c5 0 2024-04-05T21:26:07.218 INFO:tasks.workunit.client.1.smithi188.stdout:0/17: mkdir d4/d6 0 2024-04-05T21:26:07.219 INFO:tasks.workunit.client.1.smithi188.stdout:1/29: mknod d1/d7/c9 0 2024-04-05T21:26:07.219 INFO:tasks.workunit.client.1.smithi188.stdout:2/17: dwrite f4 [0,4194304] 0 2024-04-05T21:26:07.223 INFO:tasks.workunit.client.1.smithi188.stdout:0/18: mknod d4/c7 0 2024-04-05T21:26:07.224 INFO:tasks.workunit.client.1.smithi188.stdout:7/29: fsync d3/d4/f8 0 2024-04-05T21:26:07.224 INFO:tasks.workunit.client.1.smithi188.stdout:0/19: read f3 [2852563,103963] 0 2024-04-05T21:26:07.225 INFO:tasks.workunit.client.1.smithi188.stdout:0/20: stat d4/c5 0 2024-04-05T21:26:07.226 INFO:tasks.workunit.client.1.smithi188.stdout:7/30: write d3/d4/f8 [229789,127083] 0 2024-04-05T21:26:07.230 INFO:tasks.workunit.client.1.smithi188.stdout:0/21: mkdir d4/d8 0 2024-04-05T21:26:07.234 INFO:tasks.workunit.client.1.smithi188.stdout:7/31: rename c0 to d3/d4/ca 0 2024-04-05T21:26:07.234 INFO:tasks.workunit.client.1.smithi188.stdout:2/18: fsync f3 0 2024-04-05T21:26:07.235 INFO:tasks.workunit.client.1.smithi188.stdout:0/22: fdatasync f3 0 2024-04-05T21:26:07.236 INFO:tasks.workunit.client.1.smithi188.stdout:7/32: mkdir d3/d4/db 0 2024-04-05T21:26:07.238 INFO:tasks.workunit.client.1.smithi188.stdout:7/33: truncate d3/d4/f7 1014785 0 2024-04-05T21:26:07.239 INFO:tasks.workunit.client.1.smithi188.stdout:2/19: rename f4 to d5/f6 0 2024-04-05T21:26:07.239 INFO:tasks.workunit.client.1.smithi188.stdout:2/20: readlink - no filename 2024-04-05T21:26:07.240 INFO:tasks.workunit.client.1.smithi188.stdout:7/34: dread - d3/f9 zero size 2024-04-05T21:26:07.241 INFO:tasks.workunit.client.1.smithi188.stdout:3/9: rename f1 to f4 0 2024-04-05T21:26:07.243 INFO:tasks.workunit.client.1.smithi188.stdout:7/35: write d3/d4/f7 [880689,110527] 0 2024-04-05T21:26:07.243 INFO:tasks.workunit.client.1.smithi188.stdout:8/15: fsync f2 0 2024-04-05T21:26:07.244 INFO:tasks.workunit.client.1.smithi188.stdout:8/16: fdatasync f2 0 2024-04-05T21:26:07.244 INFO:tasks.workunit.client.1.smithi188.stdout:2/21: rmdir d5 39 2024-04-05T21:26:07.246 INFO:tasks.workunit.client.1.smithi188.stdout:0/23: creat d4/d6/f9 x:0 0 0 2024-04-05T21:26:07.248 INFO:tasks.workunit.client.1.smithi188.stdout:0/24: fdatasync f1 0 2024-04-05T21:26:07.248 INFO:tasks.workunit.client.1.smithi188.stdout:3/10: mkdir d5 0 2024-04-05T21:26:07.258 INFO:tasks.workunit.client.1.smithi188.stdout:7/36: dwrite d3/d4/f8 [0,4194304] 0 2024-04-05T21:26:07.259 INFO:tasks.workunit.client.1.smithi188.stdout:8/17: dwrite f2 [0,4194304] 0 2024-04-05T21:26:07.261 INFO:tasks.workunit.client.1.smithi188.stdout:7/37: truncate d3/f9 183405 0 2024-04-05T21:26:07.263 INFO:tasks.workunit.client.1.smithi188.stdout:8/18: unlink f2 0 2024-04-05T21:26:07.264 INFO:tasks.workunit.client.1.smithi188.stdout:1/30: truncate d1/f3 3517505 0 2024-04-05T21:26:07.268 INFO:tasks.workunit.client.1.smithi188.stdout:7/38: symlink d3/lc 0 2024-04-05T21:26:07.273 INFO:tasks.workunit.client.1.smithi188.stdout:2/22: rename d5/f6 to d5/f7 0 2024-04-05T21:26:07.278 INFO:tasks.workunit.client.1.smithi188.stdout:2/23: dread f3 [0,4194304] 0 2024-04-05T21:26:07.280 INFO:tasks.workunit.client.1.smithi188.stdout:1/31: fdatasync d1/f3 0 2024-04-05T21:26:07.281 INFO:tasks.workunit.client.1.smithi188.stdout:1/32: chown d1 855 1 2024-04-05T21:26:07.285 INFO:tasks.workunit.client.1.smithi188.stdout:8/19: getdents . 0 2024-04-05T21:26:07.285 INFO:tasks.workunit.client.1.smithi188.stdout:7/39: dwrite d3/d4/f7 [0,4194304] 0 2024-04-05T21:26:07.285 INFO:tasks.workunit.client.1.smithi188.stdout:2/24: write d5/f7 [1676271,12359] 0 2024-04-05T21:26:07.286 INFO:tasks.workunit.client.1.smithi188.stdout:2/25: chown d5 39131312 1 2024-04-05T21:26:07.287 INFO:tasks.workunit.client.1.smithi188.stdout:1/33: rename d1/f3 to d1/d7/fa 0 2024-04-05T21:26:07.288 INFO:tasks.workunit.client.1.smithi188.stdout:2/26: write d5/f7 [4405835,49290] 0 2024-04-05T21:26:07.288 INFO:tasks.workunit.client.1.smithi188.stdout:8/20: creat f3 x:0 0 0 2024-04-05T21:26:07.291 INFO:tasks.workunit.client.1.smithi188.stdout:8/21: stat l1 0 2024-04-05T21:26:07.293 INFO:tasks.workunit.client.1.smithi188.stdout:2/27: chown d5/f7 48 1 2024-04-05T21:26:07.295 INFO:tasks.workunit.client.1.smithi188.stdout:8/22: creat f4 x:0 0 0 2024-04-05T21:26:07.297 INFO:tasks.workunit.client.1.smithi188.stdout:2/28: mknod d5/c8 0 2024-04-05T21:26:07.297 INFO:tasks.workunit.client.1.smithi188.stdout:7/40: dwrite d3/d4/f7 [0,4194304] 0 2024-04-05T21:26:07.298 INFO:tasks.workunit.client.1.smithi188.stdout:8/23: mkdir d5 0 2024-04-05T21:26:07.302 INFO:tasks.workunit.client.1.smithi188.stdout:7/41: unlink d3/lc 0 2024-04-05T21:26:07.303 INFO:tasks.workunit.client.1.smithi188.stdout:7/42: stat d3/f9 0 2024-04-05T21:26:07.306 INFO:tasks.workunit.client.1.smithi188.stdout:2/29: unlink d5/f7 0 2024-04-05T21:26:07.308 INFO:tasks.workunit.client.1.smithi188.stdout:2/30: mknod d5/c9 0 2024-04-05T21:26:07.396 INFO:tasks.workunit.client.1.smithi188.stdout:2/31: creat d5/fa x:0 0 0 2024-04-05T21:26:07.402 INFO:tasks.workunit.client.1.smithi188.stdout:7/43: dwrite d3/d4/f8 [4194304,4194304] 0 2024-04-05T21:26:07.411 INFO:tasks.workunit.client.1.smithi188.stdout:7/44: dwrite d3/d4/f8 [4194304,4194304] 0 2024-04-05T21:26:07.509 INFO:tasks.workunit.client.1.smithi188.stdout:8/24: link l1 d5/l6 0 2024-04-05T21:26:07.510 INFO:tasks.workunit.client.1.smithi188.stdout:2/32: mkdir d5/db 0 2024-04-05T21:26:07.512 INFO:tasks.workunit.client.1.smithi188.stdout:7/45: creat d3/fd x:0 0 0 2024-04-05T21:26:07.514 INFO:tasks.workunit.client.1.smithi188.stdout:8/25: mknod d5/c7 0 2024-04-05T21:26:07.515 INFO:tasks.workunit.client.1.smithi188.stdout:7/46: truncate d3/f9 249875 0 2024-04-05T21:26:07.515 INFO:tasks.workunit.client.1.smithi188.stdout:8/26: dread - f4 zero size 2024-04-05T21:26:07.518 INFO:tasks.workunit.client.1.smithi188.stdout:8/27: rmdir d5 39 2024-04-05T21:26:07.519 INFO:tasks.workunit.client.1.smithi188.stdout:2/33: dwrite d5/fa [0,4194304] 0 2024-04-05T21:26:07.521 INFO:tasks.workunit.client.1.smithi188.stdout:7/47: rename d3/f9 to d3/d4/db/fe 0 2024-04-05T21:26:07.525 INFO:tasks.workunit.client.1.smithi188.stdout:2/34: creat d5/fc x:0 0 0 2024-04-05T21:26:07.526 INFO:tasks.workunit.client.1.smithi188.stdout:7/48: mkdir d3/df 0 2024-04-05T21:26:07.528 INFO:tasks.workunit.client.1.smithi188.stdout:8/28: creat d5/f8 x:0 0 0 2024-04-05T21:26:07.531 INFO:tasks.workunit.client.1.smithi188.stdout:7/49: mkdir d3/d4/db/d10 0 2024-04-05T21:26:07.532 INFO:tasks.workunit.client.1.smithi188.stdout:8/29: creat d5/f9 x:0 0 0 2024-04-05T21:26:07.533 INFO:tasks.workunit.client.1.smithi188.stdout:7/50: mknod d3/df/c11 0 2024-04-05T21:26:07.534 INFO:tasks.workunit.client.1.smithi188.stdout:7/51: fdatasync d3/d4/f7 0 2024-04-05T21:26:07.535 INFO:tasks.workunit.client.1.smithi188.stdout:8/30: symlink d5/la 0 2024-04-05T21:26:07.535 INFO:tasks.workunit.client.1.smithi188.stdout:8/31: dread - d5/f9 zero size 2024-04-05T21:26:07.538 INFO:tasks.workunit.client.1.smithi188.stdout:2/35: fsync d5/fa 0 2024-04-05T21:26:07.540 INFO:tasks.workunit.client.1.smithi188.stdout:2/36: mkdir d5/db/dd 0 2024-04-05T21:26:07.543 INFO:tasks.workunit.client.1.smithi188.stdout:2/37: symlink d5/db/dd/le 0 2024-04-05T21:26:07.543 INFO:tasks.workunit.client.1.smithi188.stdout:2/38: fsync f3 0 2024-04-05T21:26:07.544 INFO:tasks.workunit.client.1.smithi188.stdout:2/39: write f3 [52217,37955] 0 2024-04-05T21:26:07.549 INFO:tasks.workunit.client.1.smithi188.stdout:2/40: dread d5/fa [0,4194304] 0 2024-04-05T21:26:07.551 INFO:tasks.workunit.client.1.smithi188.stdout:2/41: mknod d5/cf 0 2024-04-05T21:26:07.559 INFO:tasks.workunit.client.1.smithi188.stdout:2/42: dwrite d5/fc [0,4194304] 0 2024-04-05T21:26:07.563 INFO:tasks.workunit.client.1.smithi188.stdout:2/43: link d5/fc d5/f10 0 2024-04-05T21:26:07.571 INFO:tasks.workunit.client.1.smithi188.stdout:2/44: dwrite d5/fa [0,4194304] 0 2024-04-05T21:26:07.575 INFO:tasks.workunit.client.1.smithi188.stdout:2/45: creat d5/db/dd/f11 x:0 0 0 2024-04-05T21:26:07.576 INFO:tasks.workunit.client.1.smithi188.stdout:2/46: write f3 [616391,126104] 0 2024-04-05T21:26:07.579 INFO:tasks.workunit.client.1.smithi188.stdout:2/47: dread f3 [0,4194304] 0 2024-04-05T21:26:07.581 INFO:tasks.workunit.client.1.smithi188.stdout:2/48: unlink d5/f10 0 2024-04-05T21:26:07.608 INFO:tasks.workunit.client.1.smithi188.stdout:7/52: truncate d3/d4/db/fe 131659 0 2024-04-05T21:26:07.611 INFO:tasks.workunit.client.1.smithi188.stdout:7/53: creat d3/d4/db/d10/f12 x:0 0 0 2024-04-05T21:26:07.611 INFO:tasks.workunit.client.1.smithi188.stdout:7/54: fdatasync d3/d4/f7 0 2024-04-05T21:26:07.612 INFO:tasks.workunit.client.1.smithi188.stdout:7/55: dread - d3/d4/db/d10/f12 zero size 2024-04-05T21:26:07.614 INFO:tasks.workunit.client.1.smithi188.stdout:7/56: truncate d3/d4/db/d10/f12 132053 0 2024-04-05T21:26:07.615 INFO:tasks.workunit.client.1.smithi188.stdout:7/57: creat d3/d4/db/d10/f13 x:0 0 0 2024-04-05T21:26:07.616 INFO:tasks.workunit.client.1.smithi188.stdout:7/58: read d3/d4/db/d10/f12 [112868,53299] 0 2024-04-05T21:26:07.618 INFO:tasks.workunit.client.1.smithi188.stdout:7/59: creat d3/d4/db/d10/f14 x:0 0 0 2024-04-05T21:26:07.619 INFO:tasks.workunit.client.1.smithi188.stdout:7/60: unlink d3/fd 0 2024-04-05T21:26:07.621 INFO:tasks.workunit.client.1.smithi188.stdout:7/61: write d3/d4/db/d10/f12 [597814,9372] 0 2024-04-05T21:26:07.622 INFO:tasks.workunit.client.1.smithi188.stdout:7/62: symlink d3/d4/l15 0 2024-04-05T21:26:07.624 INFO:tasks.workunit.client.1.smithi188.stdout:7/63: dread d3/d4/db/d10/f12 [0,4194304] 0 2024-04-05T21:26:07.678 INFO:tasks.workunit.client.1.smithi188.stdout:7/64: mkdir d3/d4/d16 0 2024-04-05T21:26:07.681 INFO:tasks.workunit.client.1.smithi188.stdout:7/65: symlink d3/d4/l17 0 2024-04-05T21:26:07.682 INFO:tasks.workunit.client.1.smithi188.stdout:7/66: symlink d3/df/l18 0 2024-04-05T21:26:07.684 INFO:tasks.workunit.client.1.smithi188.stdout:7/67: creat d3/d4/db/f19 x:0 0 0 2024-04-05T21:26:07.685 INFO:tasks.workunit.client.1.smithi188.stdout:7/68: symlink d3/df/l1a 0 2024-04-05T21:26:07.691 INFO:tasks.workunit.client.1.smithi188.stdout:7/69: fsync d3/d4/db/d10/f12 0 2024-04-05T21:26:07.697 INFO:tasks.workunit.client.1.smithi188.stdout:7/70: read d3/d4/db/fe [24717,55200] 0 2024-04-05T21:26:07.698 INFO:tasks.workunit.client.1.smithi188.stdout:7/71: fsync d3/d4/f8 0 2024-04-05T21:26:07.699 INFO:tasks.workunit.client.1.smithi188.stdout:7/72: write d3/d4/db/d10/f13 [805417,59945] 0 2024-04-05T21:26:07.706 INFO:tasks.workunit.client.1.smithi188.stdout:7/73: dwrite d3/d4/db/d10/f14 [0,4194304] 0 2024-04-05T21:26:07.709 INFO:tasks.workunit.client.1.smithi188.stdout:7/74: unlink d3/d4/db/f19 0 2024-04-05T21:26:07.710 INFO:tasks.workunit.client.1.smithi188.stdout:7/75: stat d3/d4/l15 0 2024-04-05T21:26:07.712 INFO:tasks.workunit.client.1.smithi188.stdout:7/76: unlink d3/d4/l17 0 2024-04-05T21:26:07.714 INFO:tasks.workunit.client.1.smithi188.stdout:7/77: creat d3/d4/f1b x:0 0 0 2024-04-05T21:26:07.721 INFO:tasks.workunit.client.1.smithi188.stdout:7/78: dwrite d3/d4/f8 [4194304,4194304] 0 2024-04-05T21:26:07.724 INFO:tasks.workunit.client.1.smithi188.stdout:7/79: write d3/d4/db/fe [405075,53578] 0 2024-04-05T21:26:07.727 INFO:tasks.workunit.client.1.smithi188.stdout:7/80: fsync d3/d4/db/d10/f14 0 2024-04-05T21:26:07.728 INFO:tasks.workunit.client.1.smithi188.stdout:7/81: write d3/d4/f1b [746560,73725] 0 2024-04-05T21:26:07.730 INFO:tasks.workunit.client.1.smithi188.stdout:7/82: read d3/d4/db/d10/f14 [750150,55286] 0 2024-04-05T21:26:07.732 INFO:tasks.workunit.client.1.smithi188.stdout:7/83: rmdir d3/d4/d16 0 2024-04-05T21:26:07.733 INFO:tasks.workunit.client.1.smithi188.stdout:7/84: stat d3/d4/f8 0 2024-04-05T21:26:07.735 INFO:tasks.workunit.client.1.smithi188.stdout:7/85: unlink d3/d4/c5 0 2024-04-05T21:26:07.737 INFO:tasks.workunit.client.1.smithi188.stdout:7/86: creat d3/d4/db/f1c x:0 0 0 2024-04-05T21:26:07.737 INFO:tasks.workunit.client.1.smithi188.stdout:7/87: chown l2 160782668 1 2024-04-05T21:26:07.738 INFO:tasks.workunit.client.1.smithi188.stdout:7/88: mkdir d3/d4/d1d 0 2024-04-05T21:26:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:07 smithi121 ceph-mon[39723]: Standby manager daemon smithi188.tgcuir restarted 2024-04-05T21:26:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:07 smithi121 ceph-mon[39723]: Standby manager daemon smithi188.tgcuir started 2024-04-05T21:26:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:07 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/crt"}]: dispatch 2024-04-05T21:26:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:07 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T21:26:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:07 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/key"}]: dispatch 2024-04-05T21:26:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:07 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T21:26:07.739 INFO:tasks.workunit.client.1.smithi188.stdout:7/89: creat d3/f1e x:0 0 0 2024-04-05T21:26:07.742 INFO:tasks.workunit.client.1.smithi188.stdout:7/90: link d3/d4/db/d10/f14 d3/df/f1f 0 2024-04-05T21:26:07.743 INFO:tasks.workunit.client.1.smithi188.stdout:7/91: readlink d3/df/l18 0 2024-04-05T21:26:07.743 INFO:tasks.workunit.client.1.smithi188.stdout:7/92: read - d3/f1e zero size 2024-04-05T21:26:07.744 INFO:tasks.workunit.client.1.smithi188.stdout:7/93: chown d3/df 4 1 2024-04-05T21:26:07.744 INFO:tasks.workunit.client.1.smithi188.stdout:7/94: write d3/d4/f7 [1162164,46666] 0 2024-04-05T21:26:07.745 INFO:tasks.workunit.client.1.smithi188.stdout:7/95: fdatasync d3/d4/db/f1c 0 2024-04-05T21:26:07.746 INFO:tasks.workunit.client.1.smithi188.stdout:7/96: truncate d3/f1e 848393 0 2024-04-05T21:26:07.747 INFO:tasks.workunit.client.1.smithi188.stdout:7/97: write d3/d4/f1b [1277251,66631] 0 2024-04-05T21:26:07.751 INFO:tasks.workunit.client.1.smithi188.stdout:7/98: dread d3/d4/f1b [0,4194304] 0 2024-04-05T21:26:07.755 INFO:tasks.workunit.client.1.smithi188.stdout:7/99: dread d3/d4/f8 [0,4194304] 0 2024-04-05T21:26:07.757 INFO:tasks.workunit.client.1.smithi188.stdout:7/100: creat d3/df/f20 x:0 0 0 2024-04-05T21:26:07.757 INFO:tasks.workunit.client.1.smithi188.stdout:7/101: readlink d3/d4/l15 0 2024-04-05T21:26:07.758 INFO:tasks.workunit.client.1.smithi188.stdout:7/102: fdatasync d3/d4/f7 0 2024-04-05T21:26:07.759 INFO:tasks.workunit.client.1.smithi188.stdout:7/103: fsync d3/d4/db/fe 0 2024-04-05T21:26:07.761 INFO:tasks.workunit.client.1.smithi188.stdout:7/104: write d3/df/f1f [959671,98011] 0 2024-04-05T21:26:07.763 INFO:tasks.workunit.client.1.smithi188.stdout:7/105: mknod d3/d4/d1d/c21 0 2024-04-05T21:26:07.773 INFO:tasks.workunit.client.1.smithi188.stdout:7/106: dwrite d3/df/f1f [0,4194304] 0 2024-04-05T21:26:07.776 INFO:tasks.workunit.client.1.smithi188.stdout:7/107: rename d3/df/l18 to d3/d4/l22 0 2024-04-05T21:26:07.776 INFO:tasks.workunit.client.1.smithi188.stdout:7/108: chown d3/df/c11 25 1 2024-04-05T21:26:07.778 INFO:tasks.workunit.client.1.smithi188.stdout:7/109: symlink d3/d4/d1d/l23 0 2024-04-05T21:26:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:07 smithi188 ceph-mon[49715]: Standby manager daemon smithi188.tgcuir restarted 2024-04-05T21:26:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:07 smithi188 ceph-mon[49715]: Standby manager daemon smithi188.tgcuir started 2024-04-05T21:26:07.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:07 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/crt"}]: dispatch 2024-04-05T21:26:07.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:07 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T21:26:07.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:07 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/key"}]: dispatch 2024-04-05T21:26:07.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:07 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T21:26:07.780 INFO:tasks.workunit.client.1.smithi188.stdout:7/110: write d3/df/f1f [4153865,101975] 0 2024-04-05T21:26:07.782 INFO:tasks.workunit.client.1.smithi188.stdout:7/111: unlink d3/d4/db/d10/f13 0 2024-04-05T21:26:07.786 INFO:tasks.workunit.client.1.smithi188.stdout:7/112: dread d3/d4/f7 [0,4194304] 0 2024-04-05T21:26:07.793 INFO:tasks.workunit.client.1.smithi188.stdout:7/113: dwrite d3/df/f20 [0,4194304] 0 2024-04-05T21:26:07.794 INFO:tasks.workunit.client.1.smithi188.stdout:7/114: chown d3/d4/d1d/c21 6 1 2024-04-05T21:26:07.796 INFO:tasks.workunit.client.1.smithi188.stdout:7/115: truncate d3/d4/db/d10/f12 1281938 0 2024-04-05T21:26:07.806 INFO:tasks.workunit.client.1.smithi188.stdout:7/116: truncate d3/d4/db/d10/f14 1369809 0 2024-04-05T21:26:07.807 INFO:tasks.workunit.client.1.smithi188.stdout:7/117: mknod d3/d4/c24 0 2024-04-05T21:26:07.813 INFO:tasks.workunit.client.1.smithi188.stdout:7/118: dread d3/df/f20 [0,4194304] 0 2024-04-05T21:26:07.820 INFO:tasks.workunit.client.1.smithi188.stdout:7/119: dwrite d3/d4/db/f1c [0,4194304] 0 2024-04-05T21:26:07.824 INFO:tasks.workunit.client.1.smithi188.stdout:7/120: unlink d3/d4/db/d10/f12 0 2024-04-05T21:26:07.824 INFO:tasks.workunit.client.1.smithi188.stdout:7/121: readlink d3/d4/l15 0 2024-04-05T21:26:07.825 INFO:tasks.workunit.client.1.smithi188.stdout:7/122: fdatasync d3/f1e 0 2024-04-05T21:26:07.828 INFO:tasks.workunit.client.1.smithi188.stdout:7/123: rmdir d3/d4/d1d 39 2024-04-05T21:26:07.829 INFO:tasks.workunit.client.1.smithi188.stdout:7/124: fdatasync d3/d4/db/fe 0 2024-04-05T21:26:07.834 INFO:tasks.workunit.client.1.smithi188.stdout:7/125: truncate d3/d4/f1b 36677 0 2024-04-05T21:26:07.837 INFO:tasks.workunit.client.1.smithi188.stdout:7/126: creat d3/d4/f25 x:0 0 0 2024-04-05T21:26:07.840 INFO:tasks.workunit.client.1.smithi188.stdout:7/127: dread d3/df/f20 [0,4194304] 0 2024-04-05T21:26:07.843 INFO:tasks.workunit.client.1.smithi188.stdout:7/128: rename d3/df/l1a to d3/l26 0 2024-04-05T21:26:07.843 INFO:tasks.workunit.client.1.smithi188.stdout:7/129: truncate d3/d4/f25 407822 0 2024-04-05T21:26:07.845 INFO:tasks.workunit.client.1.smithi188.stdout:7/130: creat d3/df/f27 x:0 0 0 2024-04-05T21:26:07.846 INFO:tasks.workunit.client.1.smithi188.stdout:7/131: read d3/df/f20 [3823279,120940] 0 2024-04-05T21:26:07.846 INFO:tasks.workunit.client.1.smithi188.stdout:7/132: chown d3/df/c11 5945087 1 2024-04-05T21:26:07.848 INFO:tasks.workunit.client.1.smithi188.stdout:7/133: dread d3/d4/db/fe [0,4194304] 0 2024-04-05T21:26:07.855 INFO:tasks.workunit.client.1.smithi188.stdout:7/134: dwrite d3/f1e [0,4194304] 0 2024-04-05T21:26:07.860 INFO:tasks.workunit.client.1.smithi188.stdout:7/135: rename d3/d4/c24 to d3/c28 0 2024-04-05T21:26:07.862 INFO:tasks.workunit.client.1.smithi188.stdout:7/136: chown d3/d4/d1d/l23 466 1 2024-04-05T21:26:07.863 INFO:tasks.workunit.client.1.smithi188.stdout:7/137: write d3/d4/f25 [642273,110213] 0 2024-04-05T21:26:07.864 INFO:tasks.workunit.client.1.smithi188.stdout:7/138: mknod d3/df/c29 0 2024-04-05T21:26:07.867 INFO:tasks.workunit.client.1.smithi188.stdout:7/139: truncate d3/df/f1f 1042328 0 2024-04-05T21:26:07.883 INFO:tasks.workunit.client.1.smithi188.stdout:7/140: dwrite d3/df/f1f [0,4194304] 0 2024-04-05T21:26:07.884 INFO:tasks.workunit.client.1.smithi188.stdout:7/141: read d3/df/f1f [1717116,105875] 0 2024-04-05T21:26:07.892 INFO:tasks.workunit.client.1.smithi188.stdout:7/142: dwrite d3/d4/f8 [4194304,4194304] 0 2024-04-05T21:26:07.893 INFO:tasks.workunit.client.1.smithi188.stdout:7/143: write d3/d4/f8 [6379031,31327] 0 2024-04-05T21:26:07.895 INFO:tasks.workunit.client.1.smithi188.stdout:7/144: creat d3/d4/db/f2a x:0 0 0 2024-04-05T21:26:07.897 INFO:tasks.workunit.client.1.smithi188.stdout:7/145: chown d3/l26 14100 1 2024-04-05T21:26:07.900 INFO:tasks.workunit.client.1.smithi188.stdout:7/146: creat d3/d4/d1d/f2b x:0 0 0 2024-04-05T21:26:07.901 INFO:tasks.workunit.client.1.smithi188.stdout:7/147: chown d3 77593876 1 2024-04-05T21:26:07.902 INFO:tasks.workunit.client.1.smithi188.stdout:7/148: mkdir d3/d2c 0 2024-04-05T21:26:08.585 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:08 smithi188 ceph-mon[49715]: pgmap v265: 65 pgs: 65 active+clean; 184 MiB data, 867 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.3 MiB/s wr, 392 op/s 2024-04-05T21:26:08.585 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:08 smithi188 ceph-mon[49715]: mgrmap e20: smithi121.etbquh(active, since 7m), standbys: smithi188.tgcuir 2024-04-05T21:26:08.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:08 smithi121 ceph-mon[39723]: pgmap v265: 65 pgs: 65 active+clean; 184 MiB data, 867 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.3 MiB/s wr, 392 op/s 2024-04-05T21:26:08.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:08 smithi121 ceph-mon[39723]: mgrmap e20: smithi121.etbquh(active, since 7m), standbys: smithi188.tgcuir 2024-04-05T21:26:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:26:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:26:09.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:09.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:26:09.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr fail", "who": "smithi121.etbquh"}]: dispatch 2024-04-05T21:26:09.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: osdmap e47: 6 total, 6 up, 6 in 2024-04-05T21:26:09.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "mgr fail", "who": "smithi121.etbquh"}]': finished 2024-04-05T21:26:09.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: mgrmap e21: smithi188.tgcuir(active, starting, since 0.00619568s) 2024-04-05T21:26:09.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:26:09.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:26:09.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T21:26:09.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:26:09.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T21:26:09.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.latnft"}]: dispatch 2024-04-05T21:26:09.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mgr metadata", "who": "smithi188.tgcuir", "id": "smithi188.tgcuir"}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: Manager daemon smithi188.tgcuir is now available 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi188.tgcuir/mirror_snapshot_schedule"}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi188.tgcuir/mirror_snapshot_schedule"}]: dispatch 2024-04-05T21:26:09.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi188.tgcuir/trash_purge_schedule"}]: dispatch 2024-04-05T21:26:09.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi188.tgcuir/trash_purge_schedule"}]: dispatch 2024-04-05T21:26:09.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: Deploying cephadm binary to smithi188 2024-04-05T21:26:09.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:09 smithi121 ceph-mon[39723]: Deploying cephadm binary to smithi121 2024-04-05T21:26:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:26:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:26:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' 2024-04-05T21:26:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:26:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr fail", "who": "smithi121.etbquh"}]: dispatch 2024-04-05T21:26:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: osdmap e47: 6 total, 6 up, 6 in 2024-04-05T21:26:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.14211 172.21.15.121:0/3430607796' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "mgr fail", "who": "smithi121.etbquh"}]': finished 2024-04-05T21:26:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: mgrmap e21: smithi188.tgcuir(active, starting, since 0.00619568s) 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.latnft"}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mgr metadata", "who": "smithi188.tgcuir", "id": "smithi188.tgcuir"}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T21:26:09.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T21:26:09.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T21:26:09.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: Manager daemon smithi188.tgcuir is now available 2024-04-05T21:26:09.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:26:09.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:26:09.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi188.tgcuir/mirror_snapshot_schedule"}]: dispatch 2024-04-05T21:26:09.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi188.tgcuir/mirror_snapshot_schedule"}]: dispatch 2024-04-05T21:26:09.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi188.tgcuir/trash_purge_schedule"}]: dispatch 2024-04-05T21:26:09.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi188.tgcuir/trash_purge_schedule"}]: dispatch 2024-04-05T21:26:09.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: Deploying cephadm binary to smithi188 2024-04-05T21:26:09.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:09 smithi188 ceph-mon[49715]: Deploying cephadm binary to smithi121 2024-04-05T21:26:10.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:10 smithi121 ceph-mon[39723]: mgrmap e22: smithi188.tgcuir(active, since 1.00889s) 2024-04-05T21:26:10.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:10 smithi121 ceph-mon[39723]: pgmap v3: 65 pgs: 65 active+clean; 261 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:26:11.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:10 smithi188 ceph-mon[49715]: mgrmap e22: smithi188.tgcuir(active, since 1.00889s) 2024-04-05T21:26:11.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:10 smithi188 ceph-mon[49715]: pgmap v3: 65 pgs: 65 active+clean; 261 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:26:11.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:11 smithi121 ceph-mon[39723]: pgmap v4: 65 pgs: 65 active+clean; 261 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:26:11.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:11 smithi121 ceph-mon[39723]: [05/Apr/2024:21:26:10] ENGINE Bus STARTING 2024-04-05T21:26:11.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:11 smithi121 ceph-mon[39723]: [05/Apr/2024:21:26:10] ENGINE Serving on https://172.21.15.188:7150 2024-04-05T21:26:11.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:11 smithi121 ceph-mon[39723]: [05/Apr/2024:21:26:11] ENGINE Serving on http://172.21.15.188:8765 2024-04-05T21:26:11.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:11 smithi121 ceph-mon[39723]: [05/Apr/2024:21:26:11] ENGINE Bus STARTED 2024-04-05T21:26:11.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:11 smithi121 ceph-mon[39723]: mgrmap e23: smithi188.tgcuir(active, since 2s) 2024-04-05T21:26:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:11 smithi188 ceph-mon[49715]: pgmap v4: 65 pgs: 65 active+clean; 261 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:26:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:11 smithi188 ceph-mon[49715]: [05/Apr/2024:21:26:10] ENGINE Bus STARTING 2024-04-05T21:26:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:11 smithi188 ceph-mon[49715]: [05/Apr/2024:21:26:10] ENGINE Serving on https://172.21.15.188:7150 2024-04-05T21:26:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:11 smithi188 ceph-mon[49715]: [05/Apr/2024:21:26:11] ENGINE Serving on http://172.21.15.188:8765 2024-04-05T21:26:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:11 smithi188 ceph-mon[49715]: [05/Apr/2024:21:26:11] ENGINE Bus STARTED 2024-04-05T21:26:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:11 smithi188 ceph-mon[49715]: mgrmap e23: smithi188.tgcuir(active, since 2s) 2024-04-05T21:26:13.912 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:13 smithi188 ceph-mon[49715]: pgmap v5: 65 pgs: 65 active+clean; 261 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:26:13.912 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:13 smithi188 ceph-mon[49715]: mgrmap e24: smithi188.tgcuir(active, since 4s) 2024-04-05T21:26:13.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:13 smithi121 ceph-mon[39723]: pgmap v5: 65 pgs: 65 active+clean; 261 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:26:13.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:13 smithi121 ceph-mon[39723]: mgrmap e24: smithi188.tgcuir(active, since 4s) 2024-04-05T21:26:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:15 smithi121 ceph-mon[39723]: pgmap v6: 65 pgs: 65 active+clean; 267 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-04-05T21:26:16.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:15 smithi188 ceph-mon[49715]: pgmap v6: 65 pgs: 65 active+clean; 267 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-04-05T21:26:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:17 smithi121 ceph-mon[39723]: pgmap v7: 65 pgs: 65 active+clean; 268 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1009 KiB/s rd, 1.9 MiB/s wr, 193 op/s 2024-04-05T21:26:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:17 smithi121 ceph-mon[39723]: Standby manager daemon smithi121.etbquh started 2024-04-05T21:26:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:17 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.121:0/1548972204' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi121.etbquh/crt"}]: dispatch 2024-04-05T21:26:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:17 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.121:0/1548972204' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T21:26:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:17 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.121:0/1548972204' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi121.etbquh/key"}]: dispatch 2024-04-05T21:26:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:17 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.121:0/1548972204' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T21:26:18.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:17 smithi188 ceph-mon[49715]: pgmap v7: 65 pgs: 65 active+clean; 268 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1009 KiB/s rd, 1.9 MiB/s wr, 193 op/s 2024-04-05T21:26:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:17 smithi188 ceph-mon[49715]: Standby manager daemon smithi121.etbquh started 2024-04-05T21:26:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:17 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.121:0/1548972204' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi121.etbquh/crt"}]: dispatch 2024-04-05T21:26:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:17 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.121:0/1548972204' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T21:26:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:17 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.121:0/1548972204' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi121.etbquh/key"}]: dispatch 2024-04-05T21:26:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:17 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.121:0/1548972204' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T21:26:18.971 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:18 smithi188 ceph-mon[49715]: mgrmap e25: smithi188.tgcuir(active, since 9s), standbys: smithi121.etbquh 2024-04-05T21:26:18.971 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:18 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mgr metadata", "who": "smithi121.etbquh", "id": "smithi121.etbquh"}]: dispatch 2024-04-05T21:26:18.971 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:18 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:18.971 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:18 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:18 smithi121 ceph-mon[39723]: mgrmap e25: smithi188.tgcuir(active, since 9s), standbys: smithi121.etbquh 2024-04-05T21:26:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:18 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mgr metadata", "who": "smithi121.etbquh", "id": "smithi121.etbquh"}]: dispatch 2024-04-05T21:26:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:18 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:18 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:19.855 INFO:tasks.workunit.client.0.smithi121.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-04-05T21:26:19.859 INFO:tasks.workunit.client.0.smithi121.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-05T21:26:19.859 INFO:tasks.workunit.client.0.smithi121.stderr:+ make 2024-04-05T21:26:19.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:19 smithi121 ceph-mon[39723]: pgmap v8: 65 pgs: 65 active+clean; 268 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 785 KiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-04-05T21:26:19.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:19 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:19.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:19 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:19.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:19 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:19.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:19 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:19.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:19 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:19.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:19 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:19.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:19 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:19.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:19 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:19.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:19 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:20.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:19 smithi188 ceph-mon[49715]: pgmap v8: 65 pgs: 65 active+clean; 268 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 785 KiB/s rd, 1.5 MiB/s wr, 150 op/s 2024-04-05T21:26:20.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:19 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:20.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:19 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:20.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:19 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:20.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:19 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:20.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:19 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:20.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:19 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:20.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:19 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:20.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:19 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:20.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:19 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:20.231 INFO:tasks.workunit.client.0.smithi121.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-04-05T21:26:20.578 INFO:tasks.workunit.client.0.smithi121.stderr:++ readlink -f fsstress 2024-04-05T21:26:20.579 INFO:tasks.workunit.client.0.smithi121.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-04-05T21:26:20.579 INFO:tasks.workunit.client.0.smithi121.stderr:+ popd 2024-04-05T21:26:20.581 INFO:tasks.workunit.client.0.smithi121.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-04-05T21:26:20.581 INFO:tasks.workunit.client.0.smithi121.stderr:+ popd 2024-04-05T21:26:20.581 INFO:tasks.workunit.client.0.smithi121.stdout:~/cephtest/mnt.0/client.0/tmp 2024-04-05T21:26:20.581 INFO:tasks.workunit.client.0.smithi121.stderr:++ mktemp -d -p . 2024-04-05T21:26:20.584 INFO:tasks.workunit.client.0.smithi121.stderr:+ T=./tmp.Yv7Bd6H50c 2024-04-05T21:26:20.584 INFO:tasks.workunit.client.0.smithi121.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.Yv7Bd6H50c -l 1 -n 1000 -p 10 -v 2024-04-05T21:26:20.588 INFO:tasks.workunit.client.0.smithi121.stdout:seed = 1712887547 2024-04-05T21:26:20.591 INFO:tasks.workunit.client.0.smithi121.stdout:0/0: rename - no filename 2024-04-05T21:26:20.591 INFO:tasks.workunit.client.0.smithi121.stdout:0/1: dread - no filename 2024-04-05T21:26:20.592 INFO:tasks.workunit.client.0.smithi121.stdout:0/2: dwrite - no filename 2024-04-05T21:26:20.592 INFO:tasks.workunit.client.0.smithi121.stdout:0/3: dread - no filename 2024-04-05T21:26:20.593 INFO:tasks.workunit.client.0.smithi121.stdout:1/0: fdatasync - no filename 2024-04-05T21:26:20.593 INFO:tasks.workunit.client.0.smithi121.stdout:1/1: dwrite - no filename 2024-04-05T21:26:20.593 INFO:tasks.workunit.client.0.smithi121.stdout:1/2: rename - no filename 2024-04-05T21:26:20.593 INFO:tasks.workunit.client.0.smithi121.stdout:0/4: mknod c0 0 2024-04-05T21:26:20.593 INFO:tasks.workunit.client.0.smithi121.stdout:0/5: dwrite - no filename 2024-04-05T21:26:20.593 INFO:tasks.workunit.client.0.smithi121.stdout:0/6: truncate - no filename 2024-04-05T21:26:20.594 INFO:tasks.workunit.client.0.smithi121.stdout:2/0: dwrite - no filename 2024-04-05T21:26:20.594 INFO:tasks.workunit.client.0.smithi121.stdout:2/1: write - no filename 2024-04-05T21:26:20.596 INFO:tasks.workunit.client.0.smithi121.stdout:0/7: creat f1 x:0 0 0 2024-04-05T21:26:20.596 INFO:tasks.workunit.client.0.smithi121.stdout:1/3: symlink l0 0 2024-04-05T21:26:20.596 INFO:tasks.workunit.client.0.smithi121.stdout:1/4: fsync - no filename 2024-04-05T21:26:20.596 INFO:tasks.workunit.client.0.smithi121.stdout:1/5: write - no filename 2024-04-05T21:26:20.597 INFO:tasks.workunit.client.0.smithi121.stdout:1/6: chown l0 0 1 2024-04-05T21:26:20.597 INFO:tasks.workunit.client.0.smithi121.stdout:1/7: write - no filename 2024-04-05T21:26:20.597 INFO:tasks.workunit.client.0.smithi121.stdout:1/8: read - no filename 2024-04-05T21:26:20.598 INFO:tasks.workunit.client.0.smithi121.stdout:3/0: dwrite - no filename 2024-04-05T21:26:20.599 INFO:tasks.workunit.client.0.smithi121.stdout:0/8: rename f1 to f2 0 2024-04-05T21:26:20.599 INFO:tasks.workunit.client.0.smithi121.stdout:0/9: rmdir - no directory 2024-04-05T21:26:20.599 INFO:tasks.workunit.client.0.smithi121.stdout:0/10: truncate f2 875897 0 2024-04-05T21:26:20.600 INFO:tasks.workunit.client.0.smithi121.stdout:1/9: rename l0 to l1 0 2024-04-05T21:26:20.600 INFO:tasks.workunit.client.0.smithi121.stdout:2/2: mkdir d0 0 2024-04-05T21:26:20.600 INFO:tasks.workunit.client.0.smithi121.stdout:2/3: dread - no filename 2024-04-05T21:26:20.600 INFO:tasks.workunit.client.0.smithi121.stdout:2/4: rename d0 to d0/d1 22 2024-04-05T21:26:20.601 INFO:tasks.workunit.client.0.smithi121.stdout:2/5: fdatasync - no filename 2024-04-05T21:26:20.601 INFO:tasks.workunit.client.0.smithi121.stdout:2/6: fsync - no filename 2024-04-05T21:26:20.601 INFO:tasks.workunit.client.0.smithi121.stdout:2/7: stat d0 0 2024-04-05T21:26:20.601 INFO:tasks.workunit.client.0.smithi121.stdout:2/8: unlink - no file 2024-04-05T21:26:20.601 INFO:tasks.workunit.client.0.smithi121.stdout:2/9: dread - no filename 2024-04-05T21:26:20.601 INFO:tasks.workunit.client.0.smithi121.stdout:2/10: dread - no filename 2024-04-05T21:26:20.602 INFO:tasks.workunit.client.0.smithi121.stdout:4/0: read - no filename 2024-04-05T21:26:20.603 INFO:tasks.workunit.client.0.smithi121.stdout:1/10: creat f2 x:0 0 0 2024-04-05T21:26:20.604 INFO:tasks.workunit.client.0.smithi121.stdout:3/1: mkdir d0 0 2024-04-05T21:26:20.605 INFO:tasks.workunit.client.0.smithi121.stdout:1/11: fdatasync f2 0 2024-04-05T21:26:20.607 INFO:tasks.workunit.client.0.smithi121.stdout:5/0: fsync - no filename 2024-04-05T21:26:20.607 INFO:tasks.workunit.client.0.smithi121.stdout:5/1: truncate - no filename 2024-04-05T21:26:20.607 INFO:tasks.workunit.client.0.smithi121.stdout:5/2: link - no file 2024-04-05T21:26:20.609 INFO:tasks.workunit.client.0.smithi121.stdout:6/0: write - no filename 2024-04-05T21:26:20.609 INFO:tasks.workunit.client.0.smithi121.stdout:6/1: dwrite - no filename 2024-04-05T21:26:20.609 INFO:tasks.workunit.client.0.smithi121.stdout:4/1: symlink l0 0 2024-04-05T21:26:20.610 INFO:tasks.workunit.client.0.smithi121.stdout:2/11: rmdir d0 0 2024-04-05T21:26:20.611 INFO:tasks.workunit.client.0.smithi121.stdout:3/2: mknod d0/c1 0 2024-04-05T21:26:20.611 INFO:tasks.workunit.client.0.smithi121.stdout:3/3: dread - no filename 2024-04-05T21:26:20.611 INFO:tasks.workunit.client.0.smithi121.stdout:3/4: fsync - no filename 2024-04-05T21:26:20.611 INFO:tasks.workunit.client.0.smithi121.stdout:3/5: dwrite - no filename 2024-04-05T21:26:20.613 INFO:tasks.workunit.client.0.smithi121.stdout:7/0: rename - no filename 2024-04-05T21:26:20.613 INFO:tasks.workunit.client.0.smithi121.stdout:7/1: link - no file 2024-04-05T21:26:20.614 INFO:tasks.workunit.client.0.smithi121.stdout:7/2: fdatasync - no filename 2024-04-05T21:26:20.615 INFO:tasks.workunit.client.0.smithi121.stdout:0/11: dwrite f2 [0,4194304] 0 2024-04-05T21:26:20.617 INFO:tasks.workunit.client.0.smithi121.stdout:5/3: creat f0 x:0 0 0 2024-04-05T21:26:20.618 INFO:tasks.workunit.client.0.smithi121.stdout:2/12: mkdir d2 0 2024-04-05T21:26:20.622 INFO:tasks.workunit.client.0.smithi121.stdout:8/0: dread - no filename 2024-04-05T21:26:20.622 INFO:tasks.workunit.client.0.smithi121.stdout:8/1: unlink - no file 2024-04-05T21:26:20.623 INFO:tasks.workunit.client.0.smithi121.stdout:6/2: creat f0 x:0 0 0 2024-04-05T21:26:20.625 INFO:tasks.workunit.client.0.smithi121.stdout:6/3: fdatasync f0 0 2024-04-05T21:26:20.625 INFO:tasks.workunit.client.0.smithi121.stdout:3/6: creat d0/f2 x:0 0 0 2024-04-05T21:26:20.626 INFO:tasks.workunit.client.0.smithi121.stdout:0/12: rename c0 to c3 0 2024-04-05T21:26:20.629 INFO:tasks.workunit.client.0.smithi121.stdout:0/13: write f2 [3383458,97516] 0 2024-04-05T21:26:20.629 INFO:tasks.workunit.client.0.smithi121.stdout:0/14: readlink - no filename 2024-04-05T21:26:20.629 INFO:tasks.workunit.client.0.smithi121.stdout:5/4: dwrite f0 [0,4194304] 0 2024-04-05T21:26:20.631 INFO:tasks.workunit.client.0.smithi121.stdout:9/0: dread - no filename 2024-04-05T21:26:20.631 INFO:tasks.workunit.client.0.smithi121.stdout:9/1: chown . 49 1 2024-04-05T21:26:20.632 INFO:tasks.workunit.client.0.smithi121.stdout:9/2: dwrite - no filename 2024-04-05T21:26:20.633 INFO:tasks.workunit.client.0.smithi121.stdout:7/3: mkdir d0 0 2024-04-05T21:26:20.634 INFO:tasks.workunit.client.0.smithi121.stdout:7/4: truncate - no filename 2024-04-05T21:26:20.641 INFO:tasks.workunit.client.0.smithi121.stdout:4/2: link l0 l1 0 2024-04-05T21:26:20.641 INFO:tasks.workunit.client.0.smithi121.stdout:4/3: dread - no filename 2024-04-05T21:26:20.641 INFO:tasks.workunit.client.0.smithi121.stdout:0/15: dwrite f2 [0,4194304] 0 2024-04-05T21:26:20.642 INFO:tasks.workunit.client.0.smithi121.stdout:6/4: mkdir d1 0 2024-04-05T21:26:20.646 INFO:tasks.workunit.client.0.smithi121.stdout:3/7: mkdir d0/d3 0 2024-04-05T21:26:20.649 INFO:tasks.workunit.client.0.smithi121.stdout:2/13: symlink d2/l3 0 2024-04-05T21:26:20.649 INFO:tasks.workunit.client.0.smithi121.stdout:2/14: dwrite - no filename 2024-04-05T21:26:20.650 INFO:tasks.workunit.client.0.smithi121.stdout:1/12: stat l1 0 2024-04-05T21:26:20.651 INFO:tasks.workunit.client.0.smithi121.stdout:0/16: dwrite f2 [0,4194304] 0 2024-04-05T21:26:20.652 INFO:tasks.workunit.client.0.smithi121.stdout:5/5: creat f1 x:0 0 0 2024-04-05T21:26:20.652 INFO:tasks.workunit.client.0.smithi121.stdout:5/6: rmdir - no directory 2024-04-05T21:26:20.653 INFO:tasks.workunit.client.0.smithi121.stdout:5/7: rmdir - no directory 2024-04-05T21:26:20.653 INFO:tasks.workunit.client.0.smithi121.stdout:8/2: symlink l0 0 2024-04-05T21:26:20.653 INFO:tasks.workunit.client.0.smithi121.stdout:8/3: dread - no filename 2024-04-05T21:26:20.653 INFO:tasks.workunit.client.0.smithi121.stdout:8/4: stat l0 0 2024-04-05T21:26:20.653 INFO:tasks.workunit.client.0.smithi121.stdout:8/5: dwrite - no filename 2024-04-05T21:26:20.654 INFO:tasks.workunit.client.0.smithi121.stdout:8/6: readlink l0 0 2024-04-05T21:26:20.654 INFO:tasks.workunit.client.0.smithi121.stdout:8/7: stat l0 0 2024-04-05T21:26:20.654 INFO:tasks.workunit.client.0.smithi121.stdout:5/8: write f0 [2618363,86362] 0 2024-04-05T21:26:20.656 INFO:tasks.workunit.client.0.smithi121.stdout:5/9: truncate f1 263513 0 2024-04-05T21:26:20.660 INFO:tasks.workunit.client.0.smithi121.stdout:1/13: rename l1 to l3 0 2024-04-05T21:26:20.661 INFO:tasks.workunit.client.0.smithi121.stdout:1/14: write f2 [869321,3614] 0 2024-04-05T21:26:20.662 INFO:tasks.workunit.client.0.smithi121.stdout:2/15: rename d2/l3 to d2/l4 0 2024-04-05T21:26:20.663 INFO:tasks.workunit.client.0.smithi121.stdout:0/17: mknod c4 0 2024-04-05T21:26:20.665 INFO:tasks.workunit.client.0.smithi121.stdout:9/3: mknod c0 0 2024-04-05T21:26:20.666 INFO:tasks.workunit.client.0.smithi121.stdout:9/4: dread - no filename 2024-04-05T21:26:20.666 INFO:tasks.workunit.client.0.smithi121.stdout:9/5: write - no filename 2024-04-05T21:26:20.666 INFO:tasks.workunit.client.0.smithi121.stdout:7/5: mknod d0/c1 0 2024-04-05T21:26:20.667 INFO:tasks.workunit.client.0.smithi121.stdout:4/4: chown l1 2542688 1 2024-04-05T21:26:20.667 INFO:tasks.workunit.client.0.smithi121.stdout:4/5: rmdir - no directory 2024-04-05T21:26:20.668 INFO:tasks.workunit.client.0.smithi121.stdout:4/6: chown l0 23164654 1 2024-04-05T21:26:20.668 INFO:tasks.workunit.client.0.smithi121.stdout:4/7: dread - no filename 2024-04-05T21:26:20.668 INFO:tasks.workunit.client.0.smithi121.stdout:5/10: mknod c2 0 2024-04-05T21:26:20.669 INFO:tasks.workunit.client.0.smithi121.stdout:6/5: mkdir d1/d2 0 2024-04-05T21:26:20.670 INFO:tasks.workunit.client.0.smithi121.stdout:1/15: dwrite f2 [0,4194304] 0 2024-04-05T21:26:20.670 INFO:tasks.workunit.client.0.smithi121.stdout:3/8: getdents d0/d3 0 2024-04-05T21:26:20.670 INFO:tasks.workunit.client.0.smithi121.stdout:4/8: readlink l0 0 2024-04-05T21:26:20.670 INFO:tasks.workunit.client.0.smithi121.stdout:4/9: write - no filename 2024-04-05T21:26:20.670 INFO:tasks.workunit.client.0.smithi121.stdout:4/10: dread - no filename 2024-04-05T21:26:20.671 INFO:tasks.workunit.client.0.smithi121.stdout:6/6: dread - f0 zero size 2024-04-05T21:26:20.671 INFO:tasks.workunit.client.0.smithi121.stdout:6/7: readlink - no filename 2024-04-05T21:26:20.672 INFO:tasks.workunit.client.0.smithi121.stdout:2/16: creat d2/f5 x:0 0 0 2024-04-05T21:26:20.673 INFO:tasks.workunit.client.0.smithi121.stdout:6/8: write f0 [930974,85821] 0 2024-04-05T21:26:20.675 INFO:tasks.workunit.client.0.smithi121.stdout:0/18: unlink c4 0 2024-04-05T21:26:20.675 INFO:tasks.workunit.client.0.smithi121.stdout:9/6: creat f1 x:0 0 0 2024-04-05T21:26:20.676 INFO:tasks.workunit.client.0.smithi121.stdout:2/17: dread - d2/f5 zero size 2024-04-05T21:26:20.677 INFO:tasks.workunit.client.0.smithi121.stdout:7/6: creat d0/f2 x:0 0 0 2024-04-05T21:26:20.678 INFO:tasks.workunit.client.0.smithi121.stdout:0/19: write f2 [213393,97911] 0 2024-04-05T21:26:20.679 INFO:tasks.workunit.client.0.smithi121.stdout:4/11: creat f2 x:0 0 0 2024-04-05T21:26:20.683 INFO:tasks.workunit.client.0.smithi121.stdout:5/11: dwrite f0 [0,4194304] 0 2024-04-05T21:26:20.684 INFO:tasks.workunit.client.0.smithi121.stdout:6/9: unlink f0 0 2024-04-05T21:26:20.684 INFO:tasks.workunit.client.0.smithi121.stdout:6/10: unlink - no file 2024-04-05T21:26:20.686 INFO:tasks.workunit.client.0.smithi121.stdout:9/7: creat f2 x:0 0 0 2024-04-05T21:26:20.686 INFO:tasks.workunit.client.0.smithi121.stdout:9/8: chown f1 1581272 1 2024-04-05T21:26:20.686 INFO:tasks.workunit.client.0.smithi121.stdout:9/9: fdatasync f1 0 2024-04-05T21:26:20.688 INFO:tasks.workunit.client.0.smithi121.stdout:2/18: symlink d2/l6 0 2024-04-05T21:26:20.688 INFO:tasks.workunit.client.0.smithi121.stdout:2/19: dread - d2/f5 zero size 2024-04-05T21:26:20.690 INFO:tasks.workunit.client.0.smithi121.stdout:7/7: creat d0/f3 x:0 0 0 2024-04-05T21:26:20.692 INFO:tasks.workunit.client.0.smithi121.stdout:0/20: symlink l5 0 2024-04-05T21:26:20.692 INFO:tasks.workunit.client.0.smithi121.stdout:3/9: creat d0/d3/f4 x:0 0 0 2024-04-05T21:26:20.694 INFO:tasks.workunit.client.0.smithi121.stdout:4/12: rename f2 to f3 0 2024-04-05T21:26:20.698 INFO:tasks.workunit.client.0.smithi121.stdout:2/20: creat d2/f7 x:0 0 0 2024-04-05T21:26:20.698 INFO:tasks.workunit.client.0.smithi121.stdout:2/21: stat d2 0 2024-04-05T21:26:20.699 INFO:tasks.workunit.client.0.smithi121.stdout:7/8: mknod d0/c4 0 2024-04-05T21:26:20.700 INFO:tasks.workunit.client.0.smithi121.stdout:1/16: write f2 [4880277,73466] 0 2024-04-05T21:26:20.700 INFO:tasks.workunit.client.0.smithi121.stdout:1/17: rmdir - no directory 2024-04-05T21:26:20.700 INFO:tasks.workunit.client.0.smithi121.stdout:5/12: fdatasync f0 0 2024-04-05T21:26:20.701 INFO:tasks.workunit.client.0.smithi121.stdout:3/10: creat d0/f5 x:0 0 0 2024-04-05T21:26:20.701 INFO:tasks.workunit.client.0.smithi121.stdout:1/18: write f2 [2483092,26057] 0 2024-04-05T21:26:20.702 INFO:tasks.workunit.client.0.smithi121.stdout:4/13: creat f4 x:0 0 0 2024-04-05T21:26:20.702 INFO:tasks.workunit.client.0.smithi121.stdout:6/11: mkdir d1/d2/d3 0 2024-04-05T21:26:20.702 INFO:tasks.workunit.client.0.smithi121.stdout:6/12: readlink - no filename 2024-04-05T21:26:20.703 INFO:tasks.workunit.client.0.smithi121.stdout:6/13: write - no filename 2024-04-05T21:26:20.703 INFO:tasks.workunit.client.0.smithi121.stdout:6/14: truncate - no filename 2024-04-05T21:26:20.703 INFO:tasks.workunit.client.0.smithi121.stdout:4/14: dread - f4 zero size 2024-04-05T21:26:20.703 INFO:tasks.workunit.client.0.smithi121.stdout:3/11: dread - d0/d3/f4 zero size 2024-04-05T21:26:20.704 INFO:tasks.workunit.client.0.smithi121.stdout:2/22: creat d2/f8 x:0 0 0 2024-04-05T21:26:20.708 INFO:tasks.workunit.client.0.smithi121.stdout:1/19: dread f2 [0,4194304] 0 2024-04-05T21:26:20.709 INFO:tasks.workunit.client.0.smithi121.stdout:7/9: link d0/f2 d0/f5 0 2024-04-05T21:26:20.709 INFO:tasks.workunit.client.0.smithi121.stdout:7/10: readlink - no filename 2024-04-05T21:26:20.710 INFO:tasks.workunit.client.0.smithi121.stdout:4/15: mkdir d5 0 2024-04-05T21:26:20.711 INFO:tasks.workunit.client.0.smithi121.stdout:4/16: read - f3 zero size 2024-04-05T21:26:20.711 INFO:tasks.workunit.client.0.smithi121.stdout:4/17: chown d5 6886115 1 2024-04-05T21:26:20.711 INFO:tasks.workunit.client.0.smithi121.stdout:3/12: creat d0/f6 x:0 0 0 2024-04-05T21:26:20.712 INFO:tasks.workunit.client.0.smithi121.stdout:4/18: dread - f3 zero size 2024-04-05T21:26:20.712 INFO:tasks.workunit.client.0.smithi121.stdout:2/23: mkdir d2/d9 0 2024-04-05T21:26:20.715 INFO:tasks.workunit.client.0.smithi121.stdout:9/10: getdents . 0 2024-04-05T21:26:20.715 INFO:tasks.workunit.client.0.smithi121.stdout:9/11: rmdir - no directory 2024-04-05T21:26:20.716 INFO:tasks.workunit.client.0.smithi121.stdout:0/21: getdents . 0 2024-04-05T21:26:20.717 INFO:tasks.workunit.client.0.smithi121.stdout:1/20: stat l3 0 2024-04-05T21:26:20.720 INFO:tasks.workunit.client.0.smithi121.stdout:3/13: creat d0/f7 x:0 0 0 2024-04-05T21:26:20.721 INFO:tasks.workunit.client.0.smithi121.stdout:4/19: rename f3 to d5/f6 0 2024-04-05T21:26:20.723 INFO:tasks.workunit.client.0.smithi121.stdout:2/24: unlink d2/f8 0 2024-04-05T21:26:20.723 INFO:tasks.workunit.client.0.smithi121.stdout:0/22: dwrite f2 [4194304,4194304] 0 2024-04-05T21:26:20.724 INFO:tasks.workunit.client.0.smithi121.stdout:2/25: dread - d2/f5 zero size 2024-04-05T21:26:20.724 INFO:tasks.workunit.client.0.smithi121.stdout:0/23: chown l5 1884979504 1 2024-04-05T21:26:20.726 INFO:tasks.workunit.client.0.smithi121.stdout:6/15: mkdir d1/d2/d3/d4 0 2024-04-05T21:26:20.726 INFO:tasks.workunit.client.0.smithi121.stdout:0/24: write f2 [5181294,10361] 0 2024-04-05T21:26:20.726 INFO:tasks.workunit.client.0.smithi121.stdout:0/25: stat f2 0 2024-04-05T21:26:20.726 INFO:tasks.workunit.client.0.smithi121.stdout:6/16: chown d1/d2/d3 549 1 2024-04-05T21:26:20.726 INFO:tasks.workunit.client.0.smithi121.stdout:6/17: dwrite - no filename 2024-04-05T21:26:20.727 INFO:tasks.workunit.client.0.smithi121.stdout:6/18: write - no filename 2024-04-05T21:26:20.727 INFO:tasks.workunit.client.0.smithi121.stdout:6/19: link - no file 2024-04-05T21:26:20.727 INFO:tasks.workunit.client.0.smithi121.stdout:6/20: unlink - no file 2024-04-05T21:26:20.727 INFO:tasks.workunit.client.0.smithi121.stdout:6/21: write - no filename 2024-04-05T21:26:20.727 INFO:tasks.workunit.client.0.smithi121.stdout:9/12: symlink l3 0 2024-04-05T21:26:20.727 INFO:tasks.workunit.client.0.smithi121.stdout:9/13: stat l3 0 2024-04-05T21:26:20.728 INFO:tasks.workunit.client.0.smithi121.stdout:1/21: fdatasync f2 0 2024-04-05T21:26:20.730 INFO:tasks.workunit.client.0.smithi121.stdout:7/11: link d0/f3 d0/f6 0 2024-04-05T21:26:20.731 INFO:tasks.workunit.client.0.smithi121.stdout:7/12: stat d0/c1 0 2024-04-05T21:26:20.733 INFO:tasks.workunit.client.0.smithi121.stdout:4/20: creat d5/f7 x:0 0 0 2024-04-05T21:26:20.733 INFO:tasks.workunit.client.0.smithi121.stdout:4/21: chown d5 12628734 1 2024-04-05T21:26:20.737 INFO:tasks.workunit.client.0.smithi121.stdout:5/13: getdents . 0 2024-04-05T21:26:20.738 INFO:tasks.workunit.client.0.smithi121.stdout:5/14: write f0 [2339571,112296] 0 2024-04-05T21:26:20.739 INFO:tasks.workunit.client.0.smithi121.stdout:0/26: mknod c6 0 2024-04-05T21:26:20.740 INFO:tasks.workunit.client.0.smithi121.stdout:6/22: mkdir d1/d2/d3/d5 0 2024-04-05T21:26:20.741 INFO:tasks.workunit.client.0.smithi121.stdout:6/23: stat d1 0 2024-04-05T21:26:20.742 INFO:tasks.workunit.client.0.smithi121.stdout:9/14: rename f1 to f4 0 2024-04-05T21:26:20.743 INFO:tasks.workunit.client.0.smithi121.stdout:9/15: dread - f4 zero size 2024-04-05T21:26:20.743 INFO:tasks.workunit.client.0.smithi121.stdout:9/16: chown f4 1 1 2024-04-05T21:26:20.744 INFO:tasks.workunit.client.0.smithi121.stdout:0/27: dread f2 [0,4194304] 0 2024-04-05T21:26:20.746 INFO:tasks.workunit.client.0.smithi121.stdout:4/22: creat d5/f8 x:0 0 0 2024-04-05T21:26:20.748 INFO:tasks.workunit.client.0.smithi121.stdout:3/14: rename d0/f6 to d0/d3/f8 0 2024-04-05T21:26:20.749 INFO:tasks.workunit.client.0.smithi121.stdout:2/26: mknod d2/d9/ca 0 2024-04-05T21:26:20.749 INFO:tasks.workunit.client.0.smithi121.stdout:3/15: write d0/f7 [501301,100412] 0 2024-04-05T21:26:20.750 INFO:tasks.workunit.client.0.smithi121.stdout:2/27: dread - d2/f7 zero size 2024-04-05T21:26:20.750 INFO:tasks.workunit.client.0.smithi121.stdout:5/15: symlink l3 0 2024-04-05T21:26:20.750 INFO:tasks.workunit.client.0.smithi121.stdout:2/28: dread - d2/f5 zero size 2024-04-05T21:26:20.752 INFO:tasks.workunit.client.0.smithi121.stdout:2/29: dread - d2/f7 zero size 2024-04-05T21:26:20.755 INFO:tasks.workunit.client.0.smithi121.stdout:1/22: truncate f2 2133670 0 2024-04-05T21:26:20.757 INFO:tasks.workunit.client.0.smithi121.stdout:0/28: mkdir d7 0 2024-04-05T21:26:20.757 INFO:tasks.workunit.client.0.smithi121.stdout:2/30: dread - d2/f7 zero size 2024-04-05T21:26:20.760 INFO:tasks.workunit.client.0.smithi121.stdout:4/23: creat d5/f9 x:0 0 0 2024-04-05T21:26:20.760 INFO:tasks.workunit.client.0.smithi121.stdout:2/31: stat d2/f7 0 2024-04-05T21:26:20.762 INFO:tasks.workunit.client.0.smithi121.stdout:6/24: symlink d1/d2/d3/d5/l6 0 2024-04-05T21:26:20.762 INFO:tasks.workunit.client.0.smithi121.stdout:6/25: write - no filename 2024-04-05T21:26:20.762 INFO:tasks.workunit.client.0.smithi121.stdout:6/26: write - no filename 2024-04-05T21:26:20.762 INFO:tasks.workunit.client.0.smithi121.stdout:6/27: write - no filename 2024-04-05T21:26:20.762 INFO:tasks.workunit.client.0.smithi121.stdout:6/28: fsync - no filename 2024-04-05T21:26:20.767 INFO:tasks.workunit.client.0.smithi121.stdout:2/32: chown d2/f5 0 1 2024-04-05T21:26:20.767 INFO:tasks.workunit.client.0.smithi121.stdout:3/16: dwrite d0/d3/f4 [0,4194304] 0 2024-04-05T21:26:20.768 INFO:tasks.workunit.client.0.smithi121.stdout:5/16: dread f0 [0,4194304] 0 2024-04-05T21:26:20.768 INFO:tasks.workunit.client.0.smithi121.stdout:4/24: creat d5/fa x:0 0 0 2024-04-05T21:26:20.772 INFO:tasks.workunit.client.0.smithi121.stdout:7/13: dwrite d0/f5 [0,4194304] 0 2024-04-05T21:26:20.776 INFO:tasks.workunit.client.0.smithi121.stdout:2/33: write d2/f5 [422884,33995] 0 2024-04-05T21:26:20.776 INFO:tasks.workunit.client.0.smithi121.stdout:3/17: chown d0/d3/f4 15847 1 2024-04-05T21:26:20.777 INFO:tasks.workunit.client.0.smithi121.stdout:4/25: creat d5/fb x:0 0 0 2024-04-05T21:26:20.777 INFO:tasks.workunit.client.0.smithi121.stdout:6/29: mknod d1/d2/c7 0 2024-04-05T21:26:20.779 INFO:tasks.workunit.client.0.smithi121.stdout:9/17: dwrite f4 [0,4194304] 0 2024-04-05T21:26:20.779 INFO:tasks.workunit.client.0.smithi121.stdout:5/17: creat f4 x:0 0 0 2024-04-05T21:26:20.781 INFO:tasks.workunit.client.0.smithi121.stdout:4/26: unlink l1 0 2024-04-05T21:26:20.783 INFO:tasks.workunit.client.0.smithi121.stdout:3/18: write d0/d3/f4 [1270890,87937] 0 2024-04-05T21:26:20.785 INFO:tasks.workunit.client.0.smithi121.stdout:9/18: write f2 [964453,9495] 0 2024-04-05T21:26:20.785 INFO:tasks.workunit.client.0.smithi121.stdout:5/18: write f0 [1176092,41839] 0 2024-04-05T21:26:20.787 INFO:tasks.workunit.client.0.smithi121.stdout:4/27: chown d5/f9 105 1 2024-04-05T21:26:20.789 INFO:tasks.workunit.client.0.smithi121.stdout:0/29: dwrite f2 [4194304,4194304] 0 2024-04-05T21:26:20.790 INFO:tasks.workunit.client.0.smithi121.stdout:3/19: fdatasync d0/d3/f8 0 2024-04-05T21:26:20.792 INFO:tasks.workunit.client.0.smithi121.stdout:6/30: rename d1/d2/d3/d5/l6 to d1/d2/d3/d5/l8 0 2024-04-05T21:26:20.793 INFO:tasks.workunit.client.0.smithi121.stdout:4/28: mkdir d5/dc 0 2024-04-05T21:26:20.796 INFO:tasks.workunit.client.0.smithi121.stdout:3/20: symlink d0/l9 0 2024-04-05T21:26:20.796 INFO:tasks.workunit.client.0.smithi121.stdout:6/31: chown d1/d2/d3 31858499 1 2024-04-05T21:26:20.798 INFO:tasks.workunit.client.0.smithi121.stdout:0/30: creat d7/f8 x:0 0 0 2024-04-05T21:26:20.798 INFO:tasks.workunit.client.0.smithi121.stdout:7/14: fsync d0/f2 0 2024-04-05T21:26:20.800 INFO:tasks.workunit.client.0.smithi121.stdout:4/29: creat d5/fd x:0 0 0 2024-04-05T21:26:20.805 INFO:tasks.workunit.client.0.smithi121.stdout:7/15: symlink d0/l7 0 2024-04-05T21:26:20.805 INFO:tasks.workunit.client.0.smithi121.stdout:5/19: dwrite f4 [0,4194304] 0 2024-04-05T21:26:20.805 INFO:tasks.workunit.client.0.smithi121.stdout:2/34: dwrite d2/f7 [0,4194304] 0 2024-04-05T21:26:20.807 INFO:tasks.workunit.client.0.smithi121.stdout:0/31: creat d7/f9 x:0 0 0 2024-04-05T21:26:20.808 INFO:tasks.workunit.client.0.smithi121.stdout:9/19: truncate f4 2002295 0 2024-04-05T21:26:20.808 INFO:tasks.workunit.client.0.smithi121.stdout:6/32: rename d1/d2/d3/d5 to d1/d2/d3/d5/d9 22 2024-04-05T21:26:20.808 INFO:tasks.workunit.client.0.smithi121.stdout:6/33: dread - no filename 2024-04-05T21:26:20.808 INFO:tasks.workunit.client.0.smithi121.stdout:6/34: write - no filename 2024-04-05T21:26:20.808 INFO:tasks.workunit.client.0.smithi121.stdout:6/35: dwrite - no filename 2024-04-05T21:26:20.810 INFO:tasks.workunit.client.0.smithi121.stdout:0/32: truncate d7/f8 133220 0 2024-04-05T21:26:20.811 INFO:tasks.workunit.client.0.smithi121.stdout:5/20: unlink f1 0 2024-04-05T21:26:20.814 INFO:tasks.workunit.client.0.smithi121.stdout:2/35: dread d2/f7 [0,4194304] 0 2024-04-05T21:26:20.814 INFO:tasks.workunit.client.0.smithi121.stdout:4/30: link d5/f7 d5/dc/fe 0 2024-04-05T21:26:20.815 INFO:tasks.workunit.client.0.smithi121.stdout:3/21: write d0/d3/f8 [616806,61296] 0 2024-04-05T21:26:20.816 INFO:tasks.workunit.client.0.smithi121.stdout:4/31: dread - f4 zero size 2024-04-05T21:26:20.818 INFO:tasks.workunit.client.0.smithi121.stdout:5/21: dread f4 [0,4194304] 0 2024-04-05T21:26:20.819 INFO:tasks.workunit.client.0.smithi121.stdout:0/33: creat d7/fa x:0 0 0 2024-04-05T21:26:20.819 INFO:tasks.workunit.client.0.smithi121.stdout:3/22: write d0/d3/f8 [223430,56891] 0 2024-04-05T21:26:20.821 INFO:tasks.workunit.client.0.smithi121.stdout:3/23: write d0/f7 [1362690,94933] 0 2024-04-05T21:26:20.821 INFO:tasks.workunit.client.0.smithi121.stdout:0/34: dread - d7/f9 zero size 2024-04-05T21:26:20.823 INFO:tasks.workunit.client.0.smithi121.stdout:2/36: creat d2/d9/fb x:0 0 0 2024-04-05T21:26:20.824 INFO:tasks.workunit.client.0.smithi121.stdout:4/32: creat d5/ff x:0 0 0 2024-04-05T21:26:20.826 INFO:tasks.workunit.client.0.smithi121.stdout:5/22: unlink f0 0 2024-04-05T21:26:20.827 INFO:tasks.workunit.client.0.smithi121.stdout:5/23: readlink l3 0 2024-04-05T21:26:20.829 INFO:tasks.workunit.client.0.smithi121.stdout:0/35: mknod d7/cb 0 2024-04-05T21:26:20.830 INFO:tasks.workunit.client.0.smithi121.stdout:0/36: chown d7 8 1 2024-04-05T21:26:20.830 INFO:tasks.workunit.client.0.smithi121.stdout:3/24: creat d0/d3/fa x:0 0 0 2024-04-05T21:26:20.832 INFO:tasks.workunit.client.0.smithi121.stdout:4/33: readlink l0 0 2024-04-05T21:26:20.832 INFO:tasks.workunit.client.0.smithi121.stdout:7/16: dwrite d0/f3 [0,4194304] 0 2024-04-05T21:26:20.833 INFO:tasks.workunit.client.0.smithi121.stdout:3/25: dread - d0/f2 zero size 2024-04-05T21:26:20.834 INFO:tasks.workunit.client.0.smithi121.stdout:0/37: write d7/f8 [923898,96962] 0 2024-04-05T21:26:20.834 INFO:tasks.workunit.client.0.smithi121.stdout:4/34: fdatasync d5/fa 0 2024-04-05T21:26:20.835 INFO:tasks.workunit.client.0.smithi121.stdout:3/26: rename d0 to d0/db 22 2024-04-05T21:26:20.836 INFO:tasks.workunit.client.0.smithi121.stdout:3/27: chown d0/c1 1047179008 1 2024-04-05T21:26:20.836 INFO:tasks.workunit.client.0.smithi121.stdout:2/37: creat d2/d9/fc x:0 0 0 2024-04-05T21:26:20.837 INFO:tasks.workunit.client.0.smithi121.stdout:0/38: chown d7/f8 6191506 1 2024-04-05T21:26:20.837 INFO:tasks.workunit.client.0.smithi121.stdout:6/36: link d1/d2/c7 d1/d2/d3/d4/ca 0 2024-04-05T21:26:20.838 INFO:tasks.workunit.client.0.smithi121.stdout:3/28: fsync d0/d3/f4 0 2024-04-05T21:26:20.838 INFO:tasks.workunit.client.0.smithi121.stdout:2/38: read - d2/d9/fb zero size 2024-04-05T21:26:20.840 INFO:tasks.workunit.client.0.smithi121.stdout:7/17: creat d0/f8 x:0 0 0 2024-04-05T21:26:20.840 INFO:tasks.workunit.client.0.smithi121.stdout:0/39: dread - d7/f9 zero size 2024-04-05T21:26:20.840 INFO:tasks.workunit.client.0.smithi121.stdout:4/35: symlink d5/l10 0 2024-04-05T21:26:20.840 INFO:tasks.workunit.client.0.smithi121.stdout:5/24: dread f4 [0,4194304] 0 2024-04-05T21:26:20.840 INFO:tasks.workunit.client.0.smithi121.stdout:4/36: fsync f4 0 2024-04-05T21:26:20.842 INFO:tasks.workunit.client.0.smithi121.stdout:7/18: write d0/f2 [2429755,59558] 0 2024-04-05T21:26:20.842 INFO:tasks.workunit.client.0.smithi121.stdout:0/40: mkdir d7/dc 0 2024-04-05T21:26:20.843 INFO:tasks.workunit.client.0.smithi121.stdout:4/37: write d5/fb [532830,121530] 0 2024-04-05T21:26:20.844 INFO:tasks.workunit.client.0.smithi121.stdout:2/39: creat d2/d9/fd x:0 0 0 2024-04-05T21:26:20.845 INFO:tasks.workunit.client.0.smithi121.stdout:5/25: creat f5 x:0 0 0 2024-04-05T21:26:20.846 INFO:tasks.workunit.client.0.smithi121.stdout:7/19: mkdir d0/d9 0 2024-04-05T21:26:20.846 INFO:tasks.workunit.client.0.smithi121.stdout:5/26: read - f5 zero size 2024-04-05T21:26:20.847 INFO:tasks.workunit.client.0.smithi121.stdout:0/41: mknod d7/dc/cd 0 2024-04-05T21:26:20.848 INFO:tasks.workunit.client.0.smithi121.stdout:4/38: mkdir d5/d11 0 2024-04-05T21:26:20.848 INFO:tasks.workunit.client.0.smithi121.stdout:5/27: readlink l3 0 2024-04-05T21:26:20.850 INFO:tasks.workunit.client.0.smithi121.stdout:5/28: read - f5 zero size 2024-04-05T21:26:20.850 INFO:tasks.workunit.client.0.smithi121.stdout:4/39: dread - d5/f7 zero size 2024-04-05T21:26:20.852 INFO:tasks.workunit.client.0.smithi121.stdout:5/29: write f5 [6016,17387] 0 2024-04-05T21:26:20.854 INFO:tasks.workunit.client.0.smithi121.stdout:3/29: rmdir d0/d3 39 2024-04-05T21:26:20.855 INFO:tasks.workunit.client.0.smithi121.stdout:5/30: truncate f5 783551 0 2024-04-05T21:26:20.858 INFO:tasks.workunit.client.0.smithi121.stdout:2/40: dwrite d2/d9/fc [0,4194304] 0 2024-04-05T21:26:20.858 INFO:tasks.workunit.client.0.smithi121.stdout:4/40: rename d5/fb to d5/d11/f12 0 2024-04-05T21:26:20.859 INFO:tasks.workunit.client.0.smithi121.stdout:4/41: chown d5/dc/fe 1 1 2024-04-05T21:26:20.861 INFO:tasks.workunit.client.0.smithi121.stdout:3/30: read d0/d3/f8 [414166,13885] 0 2024-04-05T21:26:20.861 INFO:tasks.workunit.client.0.smithi121.stdout:5/31: creat f6 x:0 0 0 2024-04-05T21:26:20.861 INFO:tasks.workunit.client.0.smithi121.stdout:0/42: link l5 d7/dc/le 0 2024-04-05T21:26:20.861 INFO:tasks.workunit.client.0.smithi121.stdout:5/32: stat f5 0 2024-04-05T21:26:20.864 INFO:tasks.workunit.client.0.smithi121.stdout:2/41: creat d2/d9/fe x:0 0 0 2024-04-05T21:26:20.867 INFO:tasks.workunit.client.0.smithi121.stdout:7/20: getdents d0 0 2024-04-05T21:26:20.868 INFO:tasks.workunit.client.0.smithi121.stdout:5/33: creat f7 x:0 0 0 2024-04-05T21:26:20.868 INFO:tasks.workunit.client.0.smithi121.stdout:5/34: fsync f5 0 2024-04-05T21:26:20.869 INFO:tasks.workunit.client.0.smithi121.stdout:5/35: chown c2 3934 1 2024-04-05T21:26:20.869 INFO:tasks.workunit.client.0.smithi121.stdout:3/31: mkdir d0/d3/dc 0 2024-04-05T21:26:20.871 INFO:tasks.workunit.client.0.smithi121.stdout:0/43: creat d7/dc/ff x:0 0 0 2024-04-05T21:26:20.873 INFO:tasks.workunit.client.0.smithi121.stdout:2/42: rename d2/f7 to d2/d9/ff 0 2024-04-05T21:26:20.874 INFO:tasks.workunit.client.0.smithi121.stdout:4/42: creat d5/dc/f13 x:0 0 0 2024-04-05T21:26:20.875 INFO:tasks.workunit.client.0.smithi121.stdout:7/21: mkdir d0/da 0 2024-04-05T21:26:20.877 INFO:tasks.workunit.client.0.smithi121.stdout:3/32: mknod d0/d3/cd 0 2024-04-05T21:26:20.882 INFO:tasks.workunit.client.0.smithi121.stdout:2/43: dwrite d2/f5 [0,4194304] 0 2024-04-05T21:26:20.883 INFO:tasks.workunit.client.0.smithi121.stdout:3/33: mkdir d0/d3/de 0 2024-04-05T21:26:20.885 INFO:tasks.workunit.client.0.smithi121.stdout:4/43: link d5/dc/f13 d5/d11/f14 0 2024-04-05T21:26:20.886 INFO:tasks.workunit.client.0.smithi121.stdout:4/44: truncate d5/f8 359480 0 2024-04-05T21:26:20.887 INFO:tasks.workunit.client.0.smithi121.stdout:2/44: creat d2/f10 x:0 0 0 2024-04-05T21:26:20.888 INFO:tasks.workunit.client.0.smithi121.stdout:7/22: rmdir d0/da 0 2024-04-05T21:26:20.889 INFO:tasks.workunit.client.0.smithi121.stdout:3/34: fdatasync d0/d3/f8 0 2024-04-05T21:26:20.892 INFO:tasks.workunit.client.0.smithi121.stdout:7/23: symlink d0/lb 0 2024-04-05T21:26:20.896 INFO:tasks.workunit.client.0.smithi121.stdout:4/45: dwrite f4 [0,4194304] 0 2024-04-05T21:26:20.897 INFO:tasks.workunit.client.0.smithi121.stdout:4/46: rename d5 to d5/d15 22 2024-04-05T21:26:20.898 INFO:tasks.workunit.client.0.smithi121.stdout:5/36: truncate f4 1885734 0 2024-04-05T21:26:20.899 INFO:tasks.workunit.client.0.smithi121.stdout:5/37: truncate f7 604282 0 2024-04-05T21:26:20.900 INFO:tasks.workunit.client.0.smithi121.stdout:2/45: fsync d2/f5 0 2024-04-05T21:26:20.901 INFO:tasks.workunit.client.0.smithi121.stdout:0/44: truncate d7/f8 264919 0 2024-04-05T21:26:20.902 INFO:tasks.workunit.client.0.smithi121.stdout:5/38: symlink l8 0 2024-04-05T21:26:20.903 INFO:tasks.workunit.client.0.smithi121.stdout:2/46: chown d2/d9/fb 0 1 2024-04-05T21:26:20.903 INFO:tasks.workunit.client.0.smithi121.stdout:5/39: truncate f7 824739 0 2024-04-05T21:26:20.904 INFO:tasks.workunit.client.0.smithi121.stdout:0/45: mkdir d7/dc/d10 0 2024-04-05T21:26:20.907 INFO:tasks.workunit.client.0.smithi121.stdout:5/40: dread f7 [0,4194304] 0 2024-04-05T21:26:20.908 INFO:tasks.workunit.client.0.smithi121.stdout:4/47: dwrite f4 [0,4194304] 0 2024-04-05T21:26:20.909 INFO:tasks.workunit.client.0.smithi121.stdout:2/47: unlink d2/d9/fc 0 2024-04-05T21:26:20.911 INFO:tasks.workunit.client.0.smithi121.stdout:2/48: readlink d2/l6 0 2024-04-05T21:26:20.911 INFO:tasks.workunit.client.0.smithi121.stdout:7/24: getdents d0 0 2024-04-05T21:26:20.914 INFO:tasks.workunit.client.0.smithi121.stdout:4/48: creat d5/f16 x:0 0 0 2024-04-05T21:26:20.918 INFO:tasks.workunit.client.0.smithi121.stdout:0/46: dwrite d7/fa [0,4194304] 0 2024-04-05T21:26:20.919 INFO:tasks.workunit.client.0.smithi121.stdout:0/47: chown f2 2589812 1 2024-04-05T21:26:20.920 INFO:tasks.workunit.client.0.smithi121.stdout:7/25: rename d0/f5 to d0/fc 0 2024-04-05T21:26:20.921 INFO:tasks.workunit.client.0.smithi121.stdout:5/41: link l3 l9 0 2024-04-05T21:26:20.922 INFO:tasks.workunit.client.0.smithi121.stdout:4/49: fdatasync d5/d11/f14 0 2024-04-05T21:26:20.923 INFO:tasks.workunit.client.0.smithi121.stdout:5/42: read f5 [46902,57284] 0 2024-04-05T21:26:20.924 INFO:tasks.workunit.client.0.smithi121.stdout:4/50: stat d5/l10 0 2024-04-05T21:26:20.926 INFO:tasks.workunit.client.0.smithi121.stdout:0/48: unlink d7/cb 0 2024-04-05T21:26:20.933 INFO:tasks.workunit.client.0.smithi121.stdout:4/51: rmdir d5 39 2024-04-05T21:26:20.935 INFO:tasks.workunit.client.0.smithi121.stdout:7/26: creat d0/d9/fd x:0 0 0 2024-04-05T21:26:20.935 INFO:tasks.workunit.client.0.smithi121.stdout:2/49: getdents d2 0 2024-04-05T21:26:20.939 INFO:tasks.workunit.client.0.smithi121.stdout:0/49: fdatasync d7/f8 0 2024-04-05T21:26:20.942 INFO:tasks.workunit.client.0.smithi121.stdout:2/50: rename d2/f5 to d2/f11 0 2024-04-05T21:26:20.943 INFO:tasks.workunit.client.0.smithi121.stdout:2/51: readlink d2/l6 0 2024-04-05T21:26:20.943 INFO:tasks.workunit.client.0.smithi121.stdout:2/52: readlink d2/l6 0 2024-04-05T21:26:20.944 INFO:tasks.workunit.client.0.smithi121.stdout:4/52: rename d5/f7 to d5/d11/f17 0 2024-04-05T21:26:20.945 INFO:tasks.workunit.client.0.smithi121.stdout:2/53: read d2/f11 [4130078,103480] 0 2024-04-05T21:26:20.947 INFO:tasks.workunit.client.0.smithi121.stdout:4/53: truncate d5/f9 416790 0 2024-04-05T21:26:20.947 INFO:tasks.workunit.client.0.smithi121.stdout:5/43: read f4 [606227,64121] 0 2024-04-05T21:26:20.948 INFO:tasks.workunit.client.0.smithi121.stdout:4/54: rename d5 to d5/d18 22 2024-04-05T21:26:20.948 INFO:tasks.workunit.client.0.smithi121.stdout:7/27: dwrite d0/f2 [0,4194304] 0 2024-04-05T21:26:20.948 INFO:tasks.workunit.client.0.smithi121.stdout:0/50: creat d7/dc/d10/f11 x:0 0 0 2024-04-05T21:26:20.949 INFO:tasks.workunit.client.0.smithi121.stdout:5/44: truncate f5 1093207 0 2024-04-05T21:26:20.949 INFO:tasks.workunit.client.0.smithi121.stdout:0/51: dread - d7/f9 zero size 2024-04-05T21:26:20.950 INFO:tasks.workunit.client.0.smithi121.stdout:4/55: symlink d5/l19 0 2024-04-05T21:26:20.951 INFO:tasks.workunit.client.0.smithi121.stdout:7/28: creat d0/d9/fe x:0 0 0 2024-04-05T21:26:20.952 INFO:tasks.workunit.client.0.smithi121.stdout:0/52: read - d7/dc/d10/f11 zero size 2024-04-05T21:26:20.952 INFO:tasks.workunit.client.0.smithi121.stdout:5/45: mknod ca 0 2024-04-05T21:26:20.954 INFO:tasks.workunit.client.0.smithi121.stdout:7/29: mknod d0/cf 0 2024-04-05T21:26:20.954 INFO:tasks.workunit.client.0.smithi121.stdout:7/30: chown d0 749 1 2024-04-05T21:26:20.954 INFO:tasks.workunit.client.0.smithi121.stdout:5/46: symlink lb 0 2024-04-05T21:26:20.956 INFO:tasks.workunit.client.0.smithi121.stdout:2/54: dwrite d2/d9/ff [0,4194304] 0 2024-04-05T21:26:20.957 INFO:tasks.workunit.client.0.smithi121.stdout:5/47: mknod cc 0 2024-04-05T21:26:20.961 INFO:tasks.workunit.client.0.smithi121.stdout:7/31: rename d0/cf to d0/d9/c10 0 2024-04-05T21:26:20.962 INFO:tasks.workunit.client.0.smithi121.stdout:7/32: write d0/f8 [388381,77252] 0 2024-04-05T21:26:20.962 INFO:tasks.workunit.client.0.smithi121.stdout:5/48: mkdir dd 0 2024-04-05T21:26:20.962 INFO:tasks.workunit.client.0.smithi121.stdout:7/33: chown d0/d9/fd 2144104528 1 2024-04-05T21:26:20.962 INFO:tasks.workunit.client.0.smithi121.stdout:5/49: chown l9 6721925 1 2024-04-05T21:26:20.963 INFO:tasks.workunit.client.0.smithi121.stdout:2/55: getdents d2 0 2024-04-05T21:26:20.963 INFO:tasks.workunit.client.0.smithi121.stdout:7/34: rmdir d0 39 2024-04-05T21:26:20.965 INFO:tasks.workunit.client.0.smithi121.stdout:5/50: mkdir dd/de 0 2024-04-05T21:26:20.965 INFO:tasks.workunit.client.0.smithi121.stdout:7/35: chown d0/c1 4701346 1 2024-04-05T21:26:20.967 INFO:tasks.workunit.client.0.smithi121.stdout:7/36: rmdir d0 39 2024-04-05T21:26:20.968 INFO:tasks.workunit.client.0.smithi121.stdout:5/51: mkdir dd/de/df 0 2024-04-05T21:26:20.971 INFO:tasks.workunit.client.0.smithi121.stdout:5/52: mknod dd/c10 0 2024-04-05T21:26:20.972 INFO:tasks.workunit.client.0.smithi121.stdout:2/56: dwrite d2/d9/fe [0,4194304] 0 2024-04-05T21:26:20.973 INFO:tasks.workunit.client.0.smithi121.stdout:2/57: chown d2/d9/ca 17699 1 2024-04-05T21:26:20.974 INFO:tasks.workunit.client.0.smithi121.stdout:7/37: rename d0/f2 to d0/f11 0 2024-04-05T21:26:20.976 INFO:tasks.workunit.client.0.smithi121.stdout:5/53: rename f5 to dd/de/f11 0 2024-04-05T21:26:20.977 INFO:tasks.workunit.client.0.smithi121.stdout:2/58: mkdir d2/d9/d12 0 2024-04-05T21:26:20.984 INFO:tasks.workunit.client.0.smithi121.stdout:2/59: dwrite d2/d9/ff [0,4194304] 0 2024-04-05T21:26:20.988 INFO:tasks.workunit.client.0.smithi121.stdout:5/54: write f4 [637167,122197] 0 2024-04-05T21:26:20.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:20 smithi121 ceph-mon[39723]: Adjusting osd_memory_target on smithi121 to 1227M 2024-04-05T21:26:20.992 INFO:tasks.workunit.client.0.smithi121.stdout:5/55: dread dd/de/f11 [0,4194304] 0 2024-04-05T21:26:20.992 INFO:tasks.workunit.client.0.smithi121.stdout:5/56: dread - f6 zero size 2024-04-05T21:26:20.994 INFO:tasks.workunit.client.0.smithi121.stdout:5/57: mknod dd/de/c12 0 2024-04-05T21:26:20.995 INFO:tasks.workunit.client.0.smithi121.stdout:5/58: mkdir dd/de/df/d13 0 2024-04-05T21:26:20.996 INFO:tasks.workunit.client.0.smithi121.stdout:5/59: chown f7 10 1 2024-04-05T21:26:21.001 INFO:tasks.workunit.client.0.smithi121.stdout:5/60: write f7 [1770823,94429] 0 2024-04-05T21:26:21.003 INFO:tasks.workunit.client.0.smithi121.stdout:5/61: rename cc to dd/de/c14 0 2024-04-05T21:26:21.009 INFO:tasks.workunit.client.0.smithi121.stdout:5/62: dwrite f6 [0,4194304] 0 2024-04-05T21:26:21.024 INFO:tasks.workunit.client.0.smithi121.stdout:5/63: fsync f6 0 2024-04-05T21:26:21.025 INFO:tasks.workunit.client.0.smithi121.stdout:5/64: mknod dd/c15 0 2024-04-05T21:26:21.028 INFO:tasks.workunit.client.0.smithi121.stdout:5/65: rename f7 to dd/de/df/d13/f16 0 2024-04-05T21:26:21.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:20 smithi188 ceph-mon[49715]: Adjusting osd_memory_target on smithi121 to 1227M 2024-04-05T21:26:21.030 INFO:tasks.workunit.client.0.smithi121.stdout:5/66: creat dd/f17 x:0 0 0 2024-04-05T21:26:21.031 INFO:tasks.workunit.client.0.smithi121.stdout:5/67: getdents dd/de 0 2024-04-05T21:26:21.032 INFO:tasks.workunit.client.0.smithi121.stdout:5/68: read dd/de/f11 [1076985,67372] 0 2024-04-05T21:26:21.034 INFO:tasks.workunit.client.0.smithi121.stdout:5/69: getdents dd/de 0 2024-04-05T21:26:21.034 INFO:tasks.workunit.client.0.smithi121.stdout:5/70: chown dd/c15 443575388 1 2024-04-05T21:26:21.035 INFO:tasks.workunit.client.0.smithi121.stdout:5/71: unlink c2 0 2024-04-05T21:26:21.037 INFO:tasks.workunit.client.0.smithi121.stdout:5/72: rename ca to dd/c18 0 2024-04-05T21:26:21.037 INFO:tasks.workunit.client.0.smithi121.stdout:5/73: truncate f4 2493960 0 2024-04-05T21:26:21.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:21 smithi121 ceph-mon[39723]: pgmap v9: 65 pgs: 65 active+clean; 280 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 925 KiB/s rd, 2.3 MiB/s wr, 218 op/s 2024-04-05T21:26:22.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:21 smithi188 ceph-mon[49715]: pgmap v9: 65 pgs: 65 active+clean; 280 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 925 KiB/s rd, 2.3 MiB/s wr, 218 op/s 2024-04-05T21:26:23.583 INFO:tasks.workunit.client.1.smithi188.stdout:5/21: sync 2024-04-05T21:26:23.583 INFO:tasks.workunit.client.1.smithi188.stdout:2/49: sync 2024-04-05T21:26:23.584 INFO:tasks.workunit.client.1.smithi188.stdout:9/8: sync 2024-04-05T21:26:23.584 INFO:tasks.workunit.client.1.smithi188.stdout:0/25: sync 2024-04-05T21:26:23.584 INFO:tasks.workunit.client.1.smithi188.stdout:3/11: sync 2024-04-05T21:26:23.584 INFO:tasks.workunit.client.1.smithi188.stdout:8/32: sync 2024-04-05T21:26:23.584 INFO:tasks.workunit.client.1.smithi188.stdout:1/34: sync 2024-04-05T21:26:23.584 INFO:tasks.workunit.client.1.smithi188.stdout:6/22: sync 2024-04-05T21:26:23.584 INFO:tasks.workunit.client.1.smithi188.stdout:4/2: sync 2024-04-05T21:26:23.584 INFO:tasks.workunit.client.1.smithi188.stdout:7/149: sync 2024-04-05T21:26:23.585 INFO:tasks.workunit.client.1.smithi188.stdout:7/150: chown d3/d4/db 1525612 1 2024-04-05T21:26:23.586 INFO:tasks.workunit.client.1.smithi188.stdout:4/3: mkdir d0/d2 0 2024-04-05T21:26:23.588 INFO:tasks.workunit.client.1.smithi188.stdout:0/26: unlink c2 0 2024-04-05T21:26:23.590 INFO:tasks.workunit.client.1.smithi188.stdout:9/9: creat f1 x:0 0 0 2024-04-05T21:26:23.591 INFO:tasks.workunit.client.1.smithi188.stdout:5/22: write f2 [334304,65769] 0 2024-04-05T21:26:23.591 INFO:tasks.workunit.client.1.smithi188.stdout:9/10: read - f1 zero size 2024-04-05T21:26:23.594 INFO:tasks.workunit.client.1.smithi188.stdout:8/33: rename f3 to d5/fb 0 2024-04-05T21:26:23.595 INFO:tasks.workunit.client.1.smithi188.stdout:2/50: mknod d5/db/dd/c12 0 2024-04-05T21:26:23.596 INFO:tasks.workunit.client.1.smithi188.stdout:4/4: symlink d0/l3 0 2024-04-05T21:26:23.598 INFO:tasks.workunit.client.1.smithi188.stdout:3/12: creat d5/f6 x:0 0 0 2024-04-05T21:26:23.599 INFO:tasks.workunit.client.1.smithi188.stdout:0/27: fdatasync f1 0 2024-04-05T21:26:23.599 INFO:tasks.workunit.client.1.smithi188.stdout:9/11: fsync f1 0 2024-04-05T21:26:23.601 INFO:tasks.workunit.client.1.smithi188.stdout:9/12: chown c0 230 1 2024-04-05T21:26:23.601 INFO:tasks.workunit.client.1.smithi188.stdout:6/23: write f5 [625344,53758] 0 2024-04-05T21:26:23.602 INFO:tasks.workunit.client.1.smithi188.stdout:5/23: mknod c5 0 2024-04-05T21:26:23.603 INFO:tasks.workunit.client.1.smithi188.stdout:5/24: chown f2 1303 1 2024-04-05T21:26:23.605 INFO:tasks.workunit.client.1.smithi188.stdout:8/34: mkdir d5/dc 0 2024-04-05T21:26:23.606 INFO:tasks.workunit.client.1.smithi188.stdout:2/51: mknod d5/db/dd/c13 0 2024-04-05T21:26:23.607 INFO:tasks.workunit.client.1.smithi188.stdout:3/13: mkdir d5/d7 0 2024-04-05T21:26:23.609 INFO:tasks.workunit.client.1.smithi188.stdout:1/35: creat d1/fb x:0 0 0 2024-04-05T21:26:23.610 INFO:tasks.workunit.client.1.smithi188.stdout:9/13: dwrite f1 [0,4194304] 0 2024-04-05T21:26:23.611 INFO:tasks.workunit.client.1.smithi188.stdout:5/25: mkdir d6 0 2024-04-05T21:26:23.612 INFO:tasks.workunit.client.1.smithi188.stdout:0/28: write f1 [136755,67170] 0 2024-04-05T21:26:23.613 INFO:tasks.workunit.client.1.smithi188.stdout:0/29: chown f3 9 1 2024-04-05T21:26:23.616 INFO:tasks.workunit.client.1.smithi188.stdout:7/151: creat d3/d4/f2d x:0 0 0 2024-04-05T21:26:23.617 INFO:tasks.workunit.client.1.smithi188.stdout:4/5: getdents d0/d2 0 2024-04-05T21:26:23.618 INFO:tasks.workunit.client.1.smithi188.stdout:4/6: chown d0/d2 3112917 1 2024-04-05T21:26:23.618 INFO:tasks.workunit.client.1.smithi188.stdout:4/7: truncate - no filename 2024-04-05T21:26:23.618 INFO:tasks.workunit.client.1.smithi188.stdout:4/8: dwrite - no filename 2024-04-05T21:26:23.618 INFO:tasks.workunit.client.1.smithi188.stdout:4/9: write - no filename 2024-04-05T21:26:23.618 INFO:tasks.workunit.client.1.smithi188.stdout:4/10: dwrite - no filename 2024-04-05T21:26:23.619 INFO:tasks.workunit.client.1.smithi188.stdout:8/35: rename d5/la to d5/dc/ld 0 2024-04-05T21:26:23.620 INFO:tasks.workunit.client.1.smithi188.stdout:3/14: rename d5 to d5/d8 22 2024-04-05T21:26:23.623 INFO:tasks.workunit.client.1.smithi188.stdout:9/14: dwrite f1 [0,4194304] 0 2024-04-05T21:26:23.623 INFO:tasks.workunit.client.1.smithi188.stdout:3/15: chown l2 2084285 1 2024-04-05T21:26:23.624 INFO:tasks.workunit.client.1.smithi188.stdout:2/52: write d5/fc [3078315,63660] 0 2024-04-05T21:26:23.624 INFO:tasks.workunit.client.1.smithi188.stdout:1/36: mkdir d1/dc 0 2024-04-05T21:26:23.625 INFO:tasks.workunit.client.1.smithi188.stdout:5/26: symlink d6/l7 0 2024-04-05T21:26:23.626 INFO:tasks.workunit.client.1.smithi188.stdout:2/53: write d5/fc [5045710,72447] 0 2024-04-05T21:26:23.630 INFO:tasks.workunit.client.1.smithi188.stdout:0/30: rename d4/d6/f9 to d4/d8/fa 0 2024-04-05T21:26:23.633 INFO:tasks.workunit.client.1.smithi188.stdout:4/11: symlink d0/d2/l4 0 2024-04-05T21:26:23.638 INFO:tasks.workunit.client.1.smithi188.stdout:5/27: symlink d6/l8 0 2024-04-05T21:26:23.639 INFO:tasks.workunit.client.1.smithi188.stdout:0/31: fdatasync f3 0 2024-04-05T21:26:23.642 INFO:tasks.workunit.client.1.smithi188.stdout:8/36: fsync d5/fb 0 2024-04-05T21:26:23.644 INFO:tasks.workunit.client.1.smithi188.stdout:1/37: symlink d1/ld 0 2024-04-05T21:26:23.645 INFO:tasks.workunit.client.1.smithi188.stdout:3/16: mknod d5/d7/c9 0 2024-04-05T21:26:23.647 INFO:tasks.workunit.client.1.smithi188.stdout:1/38: read - d1/fb zero size 2024-04-05T21:26:23.649 INFO:tasks.workunit.client.1.smithi188.stdout:7/152: getdents d3 0 2024-04-05T21:26:23.649 INFO:tasks.workunit.client.1.smithi188.stdout:2/54: dwrite f3 [0,4194304] 0 2024-04-05T21:26:23.651 INFO:tasks.workunit.client.1.smithi188.stdout:9/15: link c0 c2 0 2024-04-05T21:26:23.653 INFO:tasks.workunit.client.1.smithi188.stdout:5/28: dread f2 [0,4194304] 0 2024-04-05T21:26:23.653 INFO:tasks.workunit.client.1.smithi188.stdout:8/37: unlink d5/c7 0 2024-04-05T21:26:23.653 INFO:tasks.workunit.client.1.smithi188.stdout:1/39: readlink d1/d4/d5/l8 0 2024-04-05T21:26:23.656 INFO:tasks.workunit.client.1.smithi188.stdout:7/153: chown d3/d4/f7 43193 1 2024-04-05T21:26:23.659 INFO:tasks.workunit.client.1.smithi188.stdout:2/55: creat d5/f14 x:0 0 0 2024-04-05T21:26:23.661 INFO:tasks.workunit.client.1.smithi188.stdout:6/24: dwrite d6/f8 [0,4194304] 0 2024-04-05T21:26:23.661 INFO:tasks.workunit.client.1.smithi188.stdout:9/16: chown c2 45 1 2024-04-05T21:26:23.661 INFO:tasks.workunit.client.1.smithi188.stdout:9/17: rmdir - no directory 2024-04-05T21:26:23.662 INFO:tasks.workunit.client.1.smithi188.stdout:8/38: dread - d5/f9 zero size 2024-04-05T21:26:23.663 INFO:tasks.workunit.client.1.smithi188.stdout:9/18: chown f1 10955 1 2024-04-05T21:26:23.669 INFO:tasks.workunit.client.1.smithi188.stdout:5/29: creat d6/f9 x:0 0 0 2024-04-05T21:26:23.669 INFO:tasks.workunit.client.1.smithi188.stdout:0/32: getdents d4/d8 0 2024-04-05T21:26:23.670 INFO:tasks.workunit.client.1.smithi188.stdout:3/17: dwrite d5/f6 [0,4194304] 0 2024-04-05T21:26:23.671 INFO:tasks.workunit.client.1.smithi188.stdout:0/33: chown d4/d8/fa 34808 1 2024-04-05T21:26:23.681 INFO:tasks.workunit.client.1.smithi188.stdout:2/56: creat d5/db/dd/f15 x:0 0 0 2024-04-05T21:26:23.681 INFO:tasks.workunit.client.1.smithi188.stdout:7/154: mkdir d3/d4/db/d2e 0 2024-04-05T21:26:23.681 INFO:tasks.workunit.client.1.smithi188.stdout:1/40: dwrite d1/fb [0,4194304] 0 2024-04-05T21:26:23.681 INFO:tasks.workunit.client.1.smithi188.stdout:9/19: symlink l3 0 2024-04-05T21:26:23.682 INFO:tasks.workunit.client.1.smithi188.stdout:2/57: read - d5/f14 zero size 2024-04-05T21:26:23.683 INFO:tasks.workunit.client.1.smithi188.stdout:2/58: chown d5 1560166121 1 2024-04-05T21:26:23.684 INFO:tasks.workunit.client.1.smithi188.stdout:0/34: mkdir d4/d8/db 0 2024-04-05T21:26:23.684 INFO:tasks.workunit.client.1.smithi188.stdout:9/20: stat c2 0 2024-04-05T21:26:23.685 INFO:tasks.workunit.client.1.smithi188.stdout:1/41: write d1/fb [2385639,81859] 0 2024-04-05T21:26:23.686 INFO:tasks.workunit.client.1.smithi188.stdout:5/30: stat c4 0 2024-04-05T21:26:23.687 INFO:tasks.workunit.client.1.smithi188.stdout:9/21: creat f4 x:0 0 0 2024-04-05T21:26:23.687 INFO:tasks.workunit.client.1.smithi188.stdout:1/42: readlink d1/ld 0 2024-04-05T21:26:23.688 INFO:tasks.workunit.client.1.smithi188.stdout:0/35: mkdir d4/d8/dc 0 2024-04-05T21:26:23.688 INFO:tasks.workunit.client.1.smithi188.stdout:8/39: creat d5/fe x:0 0 0 2024-04-05T21:26:23.689 INFO:tasks.workunit.client.1.smithi188.stdout:5/31: mknod d6/ca 0 2024-04-05T21:26:23.690 INFO:tasks.workunit.client.1.smithi188.stdout:8/40: write d5/fe [452868,86955] 0 2024-04-05T21:26:23.690 INFO:tasks.workunit.client.1.smithi188.stdout:3/18: creat d5/fa x:0 0 0 2024-04-05T21:26:23.693 INFO:tasks.workunit.client.1.smithi188.stdout:7/155: mkdir d3/d4/db/d10/d2f 0 2024-04-05T21:26:23.694 INFO:tasks.workunit.client.1.smithi188.stdout:1/43: creat d1/dc/fe x:0 0 0 2024-04-05T21:26:23.695 INFO:tasks.workunit.client.1.smithi188.stdout:8/41: creat d5/ff x:0 0 0 2024-04-05T21:26:23.696 INFO:tasks.workunit.client.1.smithi188.stdout:2/59: getdents d5/db 0 2024-04-05T21:26:23.700 INFO:tasks.workunit.client.1.smithi188.stdout:7/156: fdatasync d3/d4/db/f1c 0 2024-04-05T21:26:23.706 INFO:tasks.workunit.client.1.smithi188.stdout:1/44: creat d1/d4/ff x:0 0 0 2024-04-05T21:26:23.710 INFO:tasks.workunit.client.1.smithi188.stdout:5/32: dwrite d6/f9 [0,4194304] 0 2024-04-05T21:26:23.710 INFO:tasks.workunit.client.1.smithi188.stdout:9/22: getdents . 0 2024-04-05T21:26:23.712 INFO:tasks.workunit.client.1.smithi188.stdout:8/42: link d5/ff d5/dc/f10 0 2024-04-05T21:26:23.713 INFO:tasks.workunit.client.1.smithi188.stdout:3/19: dwrite d5/fa [0,4194304] 0 2024-04-05T21:26:23.715 INFO:tasks.workunit.client.1.smithi188.stdout:9/23: chown c2 5675 1 2024-04-05T21:26:23.720 INFO:tasks.workunit.client.1.smithi188.stdout:5/33: write d6/f9 [4173873,102799] 0 2024-04-05T21:26:23.721 INFO:tasks.workunit.client.1.smithi188.stdout:8/43: creat d5/dc/f11 x:0 0 0 2024-04-05T21:26:23.722 INFO:tasks.workunit.client.1.smithi188.stdout:7/157: write d3/df/f20 [463973,66147] 0 2024-04-05T21:26:23.723 INFO:tasks.workunit.client.1.smithi188.stdout:9/24: creat f5 x:0 0 0 2024-04-05T21:26:23.725 INFO:tasks.workunit.client.1.smithi188.stdout:2/60: dwrite d5/db/dd/f15 [0,4194304] 0 2024-04-05T21:26:23.726 INFO:tasks.workunit.client.1.smithi188.stdout:8/44: unlink f4 0 2024-04-05T21:26:23.727 INFO:tasks.workunit.client.1.smithi188.stdout:5/34: mknod d6/cb 0 2024-04-05T21:26:23.731 INFO:tasks.workunit.client.1.smithi188.stdout:7/158: creat d3/df/f30 x:0 0 0 2024-04-05T21:26:23.731 INFO:tasks.workunit.client.1.smithi188.stdout:3/20: dwrite d5/f6 [0,4194304] 0 2024-04-05T21:26:23.731 INFO:tasks.workunit.client.1.smithi188.stdout:2/61: stat d5/db/dd/c13 0 2024-04-05T21:26:23.736 INFO:tasks.workunit.client.1.smithi188.stdout:5/35: rename l3 to d6/lc 0 2024-04-05T21:26:23.736 INFO:tasks.workunit.client.1.smithi188.stdout:8/45: read - d5/ff zero size 2024-04-05T21:26:23.741 INFO:tasks.workunit.client.1.smithi188.stdout:3/21: rmdir d5/d7 39 2024-04-05T21:26:23.742 INFO:tasks.workunit.client.1.smithi188.stdout:2/62: creat d5/db/f16 x:0 0 0 2024-04-05T21:26:23.743 INFO:tasks.workunit.client.1.smithi188.stdout:8/46: unlink d5/fe 0 2024-04-05T21:26:23.744 INFO:tasks.workunit.client.1.smithi188.stdout:3/22: chown l2 0 1 2024-04-05T21:26:23.746 INFO:tasks.workunit.client.1.smithi188.stdout:2/63: chown d5/db/dd/le 1550829 1 2024-04-05T21:26:23.746 INFO:tasks.workunit.client.1.smithi188.stdout:9/25: dwrite f1 [4194304,4194304] 0 2024-04-05T21:26:23.746 INFO:tasks.workunit.client.1.smithi188.stdout:8/47: chown d5/ff 33 1 2024-04-05T21:26:23.748 INFO:tasks.workunit.client.1.smithi188.stdout:9/26: write f1 [8853246,79473] 0 2024-04-05T21:26:23.748 INFO:tasks.workunit.client.1.smithi188.stdout:2/64: write d5/fc [159748,104493] 0 2024-04-05T21:26:23.750 INFO:tasks.workunit.client.1.smithi188.stdout:8/48: creat d5/f12 x:0 0 0 2024-04-05T21:26:23.751 INFO:tasks.workunit.client.1.smithi188.stdout:3/23: creat d5/fb x:0 0 0 2024-04-05T21:26:23.753 INFO:tasks.workunit.client.1.smithi188.stdout:5/36: dread d6/f9 [0,4194304] 0 2024-04-05T21:26:23.754 INFO:tasks.workunit.client.1.smithi188.stdout:3/24: stat d5/fb 0 2024-04-05T21:26:23.754 INFO:tasks.workunit.client.1.smithi188.stdout:2/65: write d5/db/dd/f15 [2595326,83552] 0 2024-04-05T21:26:23.756 INFO:tasks.workunit.client.1.smithi188.stdout:3/25: chown l2 17828 1 2024-04-05T21:26:23.756 INFO:tasks.workunit.client.1.smithi188.stdout:8/49: mknod d5/dc/c13 0 2024-04-05T21:26:23.762 INFO:tasks.workunit.client.1.smithi188.stdout:5/37: write f2 [4088274,123597] 0 2024-04-05T21:26:23.763 INFO:tasks.workunit.client.1.smithi188.stdout:5/38: stat c4 0 2024-04-05T21:26:23.765 INFO:tasks.workunit.client.1.smithi188.stdout:9/27: dwrite f4 [0,4194304] 0 2024-04-05T21:26:23.766 INFO:tasks.workunit.client.1.smithi188.stdout:8/50: unlink l1 0 2024-04-05T21:26:23.767 INFO:tasks.workunit.client.1.smithi188.stdout:5/39: creat d6/fd x:0 0 0 2024-04-05T21:26:23.768 INFO:tasks.workunit.client.1.smithi188.stdout:9/28: chown f5 14508 1 2024-04-05T21:26:23.769 INFO:tasks.workunit.client.1.smithi188.stdout:3/26: rmdir d5 39 2024-04-05T21:26:23.771 INFO:tasks.workunit.client.1.smithi188.stdout:9/29: chown f1 17 1 2024-04-05T21:26:23.772 INFO:tasks.workunit.client.1.smithi188.stdout:2/66: dwrite d5/db/f16 [0,4194304] 0 2024-04-05T21:26:23.772 INFO:tasks.workunit.client.1.smithi188.stdout:3/27: chown f3 17707 1 2024-04-05T21:26:23.772 INFO:tasks.workunit.client.1.smithi188.stdout:5/40: creat d6/fe x:0 0 0 2024-04-05T21:26:23.773 INFO:tasks.workunit.client.1.smithi188.stdout:5/41: stat f2 0 2024-04-05T21:26:23.773 INFO:tasks.workunit.client.1.smithi188.stdout:9/30: fdatasync f1 0 2024-04-05T21:26:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:23 smithi188 ceph-mon[49715]: pgmap v10: 65 pgs: 65 active+clean; 280 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 848 KiB/s rd, 2.1 MiB/s wr, 200 op/s 2024-04-05T21:26:23.784 INFO:tasks.workunit.client.1.smithi188.stdout:2/67: getdents d5/db 0 2024-04-05T21:26:23.786 INFO:tasks.workunit.client.1.smithi188.stdout:5/42: dwrite f2 [0,4194304] 0 2024-04-05T21:26:23.788 INFO:tasks.workunit.client.1.smithi188.stdout:2/68: mkdir d5/db/d17 0 2024-04-05T21:26:23.788 INFO:tasks.workunit.client.1.smithi188.stdout:9/31: dwrite f1 [0,4194304] 0 2024-04-05T21:26:23.788 INFO:tasks.workunit.client.1.smithi188.stdout:5/43: write d6/fe [418296,122044] 0 2024-04-05T21:26:23.792 INFO:tasks.workunit.client.1.smithi188.stdout:5/44: chown f2 5178 1 2024-04-05T21:26:23.802 INFO:tasks.workunit.client.1.smithi188.stdout:2/69: rmdir d5/db/d17 0 2024-04-05T21:26:23.802 INFO:tasks.workunit.client.1.smithi188.stdout:8/51: getdents d5 0 2024-04-05T21:26:23.802 INFO:tasks.workunit.client.1.smithi188.stdout:9/32: link c0 c6 0 2024-04-05T21:26:23.805 INFO:tasks.workunit.client.1.smithi188.stdout:9/33: mknod c7 0 2024-04-05T21:26:23.805 INFO:tasks.workunit.client.1.smithi188.stdout:2/70: write d5/db/f16 [4431990,124106] 0 2024-04-05T21:26:23.806 INFO:tasks.workunit.client.1.smithi188.stdout:5/45: dwrite d6/fe [0,4194304] 0 2024-04-05T21:26:23.809 INFO:tasks.workunit.client.1.smithi188.stdout:9/34: symlink l8 0 2024-04-05T21:26:23.811 INFO:tasks.workunit.client.1.smithi188.stdout:3/28: dwrite f4 [0,4194304] 0 2024-04-05T21:26:23.811 INFO:tasks.workunit.client.1.smithi188.stdout:2/71: rename d5/cf to d5/c18 0 2024-04-05T21:26:23.813 INFO:tasks.workunit.client.1.smithi188.stdout:2/72: write d5/f14 [654462,129567] 0 2024-04-05T21:26:23.815 INFO:tasks.workunit.client.1.smithi188.stdout:3/29: read - d5/fb zero size 2024-04-05T21:26:23.817 INFO:tasks.workunit.client.1.smithi188.stdout:2/73: mknod d5/db/c19 0 2024-04-05T21:26:23.818 INFO:tasks.workunit.client.1.smithi188.stdout:9/35: link c6 c9 0 2024-04-05T21:26:23.818 INFO:tasks.workunit.client.1.smithi188.stdout:2/74: write d5/db/f16 [2899701,22642] 0 2024-04-05T21:26:23.820 INFO:tasks.workunit.client.1.smithi188.stdout:2/75: write d5/fc [1918374,84977] 0 2024-04-05T21:26:23.820 INFO:tasks.workunit.client.1.smithi188.stdout:9/36: mknod ca 0 2024-04-05T21:26:23.823 INFO:tasks.workunit.client.1.smithi188.stdout:2/76: creat d5/db/dd/f1a x:0 0 0 2024-04-05T21:26:23.823 INFO:tasks.workunit.client.1.smithi188.stdout:3/30: mkdir d5/dc 0 2024-04-05T21:26:23.826 INFO:tasks.workunit.client.1.smithi188.stdout:3/31: symlink d5/dc/ld 0 2024-04-05T21:26:23.827 INFO:tasks.workunit.client.1.smithi188.stdout:9/37: dwrite f1 [4194304,4194304] 0 2024-04-05T21:26:23.829 INFO:tasks.workunit.client.1.smithi188.stdout:9/38: write f1 [2207352,95053] 0 2024-04-05T21:26:23.831 INFO:tasks.workunit.client.1.smithi188.stdout:9/39: creat fb x:0 0 0 2024-04-05T21:26:23.831 INFO:tasks.workunit.client.1.smithi188.stdout:9/40: readlink l3 0 2024-04-05T21:26:23.831 INFO:tasks.workunit.client.1.smithi188.stdout:9/41: rmdir - no directory 2024-04-05T21:26:23.832 INFO:tasks.workunit.client.1.smithi188.stdout:9/42: read - f5 zero size 2024-04-05T21:26:23.833 INFO:tasks.workunit.client.1.smithi188.stdout:9/43: creat fc x:0 0 0 2024-04-05T21:26:23.839 INFO:tasks.workunit.client.1.smithi188.stdout:9/44: dwrite f4 [0,4194304] 0 2024-04-05T21:26:23.841 INFO:tasks.workunit.client.1.smithi188.stdout:9/45: write f5 [491047,53362] 0 2024-04-05T21:26:23.841 INFO:tasks.workunit.client.1.smithi188.stdout:9/46: write f1 [801248,53923] 0 2024-04-05T21:26:23.876 INFO:tasks.workunit.client.1.smithi188.stdout:9/47: fsync f5 0 2024-04-05T21:26:23.878 INFO:tasks.workunit.client.1.smithi188.stdout:2/77: fdatasync d5/fc 0 2024-04-05T21:26:23.878 INFO:tasks.workunit.client.1.smithi188.stdout:9/48: unlink c7 0 2024-04-05T21:26:23.879 INFO:tasks.workunit.client.1.smithi188.stdout:9/49: mkdir dd 0 2024-04-05T21:26:23.880 INFO:tasks.workunit.client.1.smithi188.stdout:2/78: symlink d5/db/l1b 0 2024-04-05T21:26:23.913 INFO:tasks.workunit.client.1.smithi188.stdout:8/52: dwrite d5/f8 [0,4194304] 0 2024-04-05T21:26:23.917 INFO:tasks.workunit.client.1.smithi188.stdout:8/53: dread - d5/f9 zero size 2024-04-05T21:26:23.960 INFO:tasks.workunit.client.1.smithi188.stdout:9/50: fdatasync f4 0 2024-04-05T21:26:23.961 INFO:tasks.workunit.client.1.smithi188.stdout:2/79: getdents d5/db 0 2024-04-05T21:26:23.962 INFO:tasks.workunit.client.1.smithi188.stdout:8/54: stat d5/dc 0 2024-04-05T21:26:23.963 INFO:tasks.workunit.client.1.smithi188.stdout:3/32: dwrite f3 [0,4194304] 0 2024-04-05T21:26:23.963 INFO:tasks.workunit.client.1.smithi188.stdout:9/51: symlink dd/le 0 2024-04-05T21:26:23.964 INFO:tasks.workunit.client.1.smithi188.stdout:2/80: chown d5/db/c19 1219 1 2024-04-05T21:26:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:23 smithi121 ceph-mon[39723]: pgmap v10: 65 pgs: 65 active+clean; 280 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 848 KiB/s rd, 2.1 MiB/s wr, 200 op/s 2024-04-05T21:26:24.058 INFO:tasks.workunit.client.1.smithi188.stdout:2/81: unlink d5/db/dd/f1a 0 2024-04-05T21:26:24.058 INFO:tasks.workunit.client.1.smithi188.stdout:9/52: stat c0 0 2024-04-05T21:26:24.058 INFO:tasks.workunit.client.1.smithi188.stdout:8/55: fdatasync d5/ff 0 2024-04-05T21:26:24.060 INFO:tasks.workunit.client.1.smithi188.stdout:9/53: mkdir dd/df 0 2024-04-05T21:26:24.060 INFO:tasks.workunit.client.1.smithi188.stdout:3/33: dread d5/f6 [0,4194304] 0 2024-04-05T21:26:24.061 INFO:tasks.workunit.client.1.smithi188.stdout:9/54: write fc [192990,113294] 0 2024-04-05T21:26:24.062 INFO:tasks.workunit.client.1.smithi188.stdout:3/34: mknod d5/dc/ce 0 2024-04-05T21:26:24.063 INFO:tasks.workunit.client.1.smithi188.stdout:3/35: write d5/fb [826162,48304] 0 2024-04-05T21:26:24.063 INFO:tasks.workunit.client.1.smithi188.stdout:9/55: symlink dd/df/l10 0 2024-04-05T21:26:24.066 INFO:tasks.workunit.client.1.smithi188.stdout:9/56: mkdir dd/d11 0 2024-04-05T21:26:24.067 INFO:tasks.workunit.client.1.smithi188.stdout:3/36: mkdir d5/df 0 2024-04-05T21:26:24.069 INFO:tasks.workunit.client.1.smithi188.stdout:9/57: link f5 dd/f12 0 2024-04-05T21:26:24.070 INFO:tasks.workunit.client.1.smithi188.stdout:3/37: mknod d5/d7/c10 0 2024-04-05T21:26:24.070 INFO:tasks.workunit.client.1.smithi188.stdout:9/58: symlink dd/df/l13 0 2024-04-05T21:26:24.073 INFO:tasks.workunit.client.1.smithi188.stdout:9/59: dread dd/f12 [0,4194304] 0 2024-04-05T21:26:24.080 INFO:tasks.workunit.client.1.smithi188.stdout:9/60: dwrite f4 [0,4194304] 0 2024-04-05T21:26:24.083 INFO:tasks.workunit.client.1.smithi188.stdout:9/61: link l3 dd/d11/l14 0 2024-04-05T21:26:24.085 INFO:tasks.workunit.client.1.smithi188.stdout:9/62: rename dd/df/l13 to dd/d11/l15 0 2024-04-05T21:26:24.087 INFO:tasks.workunit.client.1.smithi188.stdout:9/63: creat dd/f16 x:0 0 0 2024-04-05T21:26:24.093 INFO:tasks.workunit.client.1.smithi188.stdout:9/64: dwrite f4 [0,4194304] 0 2024-04-05T21:26:24.095 INFO:tasks.workunit.client.1.smithi188.stdout:9/65: mkdir dd/d17 0 2024-04-05T21:26:24.096 INFO:tasks.workunit.client.1.smithi188.stdout:9/66: dread - dd/f16 zero size 2024-04-05T21:26:24.097 INFO:tasks.workunit.client.1.smithi188.stdout:9/67: mknod dd/d11/c18 0 2024-04-05T21:26:24.099 INFO:tasks.workunit.client.1.smithi188.stdout:9/68: write dd/f12 [1090795,86655] 0 2024-04-05T21:26:24.101 INFO:tasks.workunit.client.1.smithi188.stdout:9/69: creat dd/df/f19 x:0 0 0 2024-04-05T21:26:24.113 INFO:tasks.workunit.client.1.smithi188.stdout:9/70: rmdir dd 39 2024-04-05T21:26:24.115 INFO:tasks.workunit.client.1.smithi188.stdout:9/71: symlink dd/d17/l1a 0 2024-04-05T21:26:24.118 INFO:tasks.workunit.client.1.smithi188.stdout:9/72: dread f4 [0,4194304] 0 2024-04-05T21:26:24.119 INFO:tasks.workunit.client.1.smithi188.stdout:9/73: symlink dd/l1b 0 2024-04-05T21:26:24.201 INFO:tasks.workunit.client.1.smithi188.stdout:4/12: sync 2024-04-05T21:26:24.202 INFO:tasks.workunit.client.1.smithi188.stdout:4/13: creat d0/f5 x:0 0 0 2024-04-05T21:26:24.203 INFO:tasks.workunit.client.1.smithi188.stdout:4/14: chown d0/f5 9 1 2024-04-05T21:26:24.203 INFO:tasks.workunit.client.1.smithi188.stdout:4/15: write d0/f5 [233927,33261] 0 2024-04-05T21:26:24.205 INFO:tasks.workunit.client.1.smithi188.stdout:4/16: creat d0/d2/f6 x:0 0 0 2024-04-05T21:26:24.206 INFO:tasks.workunit.client.1.smithi188.stdout:4/17: truncate d0/f5 689996 0 2024-04-05T21:26:24.211 INFO:tasks.workunit.client.1.smithi188.stdout:4/18: dwrite d0/f5 [0,4194304] 0 2024-04-05T21:26:24.214 INFO:tasks.workunit.client.1.smithi188.stdout:4/19: creat d0/d2/f7 x:0 0 0 2024-04-05T21:26:24.219 INFO:tasks.workunit.client.1.smithi188.stdout:4/20: fsync d0/d2/f7 0 2024-04-05T21:26:24.219 INFO:tasks.workunit.client.1.smithi188.stdout:4/21: rename d0/d2 to d0/d2/d8 22 2024-04-05T21:26:24.220 INFO:tasks.workunit.client.1.smithi188.stdout:4/22: dread - d0/d2/f7 zero size 2024-04-05T21:26:24.226 INFO:tasks.workunit.client.1.smithi188.stdout:4/23: dwrite d0/f5 [0,4194304] 0 2024-04-05T21:26:24.228 INFO:tasks.workunit.client.1.smithi188.stdout:4/24: mkdir d0/d2/d9 0 2024-04-05T21:26:24.229 INFO:tasks.workunit.client.1.smithi188.stdout:4/25: write d0/d2/f7 [702783,87958] 0 2024-04-05T21:26:24.231 INFO:tasks.workunit.client.1.smithi188.stdout:4/26: write d0/d2/f7 [1095856,129323] 0 2024-04-05T21:26:24.234 INFO:tasks.workunit.client.1.smithi188.stdout:4/27: getdents d0 0 2024-04-05T21:26:24.236 INFO:tasks.workunit.client.1.smithi188.stdout:4/28: symlink d0/d2/la 0 2024-04-05T21:26:24.242 INFO:tasks.workunit.client.1.smithi188.stdout:4/29: truncate d0/f5 1545526 0 2024-04-05T21:26:24.242 INFO:tasks.workunit.client.1.smithi188.stdout:4/30: chown d0/l1 444817 1 2024-04-05T21:26:24.243 INFO:tasks.workunit.client.1.smithi188.stdout:4/31: unlink d0/d2/f7 0 2024-04-05T21:26:24.245 INFO:tasks.workunit.client.1.smithi188.stdout:4/32: creat d0/d2/d9/fb x:0 0 0 2024-04-05T21:26:24.245 INFO:tasks.workunit.client.1.smithi188.stdout:4/33: rename d0/d2 to d0/d2/dc 22 2024-04-05T21:26:24.247 INFO:tasks.workunit.client.1.smithi188.stdout:4/34: symlink d0/d2/ld 0 2024-04-05T21:26:24.254 INFO:tasks.workunit.client.1.smithi188.stdout:4/35: truncate d0/f5 1174887 0 2024-04-05T21:26:24.257 INFO:tasks.workunit.client.1.smithi188.stdout:4/36: link d0/d2/la d0/d2/d9/le 0 2024-04-05T21:26:24.258 INFO:tasks.workunit.client.1.smithi188.stdout:4/37: creat d0/d2/d9/ff x:0 0 0 2024-04-05T21:26:24.260 INFO:tasks.workunit.client.1.smithi188.stdout:4/38: creat d0/d2/f10 x:0 0 0 2024-04-05T21:26:24.263 INFO:tasks.workunit.client.1.smithi188.stdout:4/39: link d0/d2/l4 d0/d2/d9/l11 0 2024-04-05T21:26:24.264 INFO:tasks.workunit.client.1.smithi188.stdout:4/40: truncate d0/d2/d9/ff 568806 0 2024-04-05T21:26:24.265 INFO:tasks.workunit.client.1.smithi188.stdout:4/41: mknod d0/d2/c12 0 2024-04-05T21:26:24.266 INFO:tasks.workunit.client.1.smithi188.stdout:4/42: mknod d0/d2/d9/c13 0 2024-04-05T21:26:24.267 INFO:tasks.workunit.client.1.smithi188.stdout:4/43: mkdir d0/d14 0 2024-04-05T21:26:24.274 INFO:tasks.workunit.client.1.smithi188.stdout:1/45: sync 2024-04-05T21:26:24.275 INFO:tasks.workunit.client.1.smithi188.stdout:6/25: sync 2024-04-05T21:26:24.275 INFO:tasks.workunit.client.1.smithi188.stdout:0/36: sync 2024-04-05T21:26:24.275 INFO:tasks.workunit.client.1.smithi188.stdout:4/44: rename d0/d2/d9 to d0/d14/d15 0 2024-04-05T21:26:24.275 INFO:tasks.workunit.client.1.smithi188.stdout:1/46: read d1/fb [2097014,63862] 0 2024-04-05T21:26:24.280 INFO:tasks.workunit.client.1.smithi188.stdout:0/37: rename d4/d8/dc to d4/d8/db/dd 0 2024-04-05T21:26:24.281 INFO:tasks.workunit.client.1.smithi188.stdout:0/38: chown d4/d8/db 958 1 2024-04-05T21:26:24.281 INFO:tasks.workunit.client.1.smithi188.stdout:4/45: rmdir d0/d14/d15 39 2024-04-05T21:26:24.282 INFO:tasks.workunit.client.1.smithi188.stdout:6/26: dread d6/f8 [0,4194304] 0 2024-04-05T21:26:24.285 INFO:tasks.workunit.client.1.smithi188.stdout:1/47: truncate d1/fb 2246391 0 2024-04-05T21:26:24.285 INFO:tasks.workunit.client.1.smithi188.stdout:0/39: fdatasync d4/d8/fa 0 2024-04-05T21:26:24.287 INFO:tasks.workunit.client.1.smithi188.stdout:6/27: rmdir d6 39 2024-04-05T21:26:24.289 INFO:tasks.workunit.client.1.smithi188.stdout:1/48: rmdir d1/dc 39 2024-04-05T21:26:24.291 INFO:tasks.workunit.client.1.smithi188.stdout:0/40: creat d4/fe x:0 0 0 2024-04-05T21:26:24.295 INFO:tasks.workunit.client.1.smithi188.stdout:4/46: creat d0/d14/f16 x:0 0 0 2024-04-05T21:26:24.296 INFO:tasks.workunit.client.1.smithi188.stdout:4/47: dread - d0/d2/f6 zero size 2024-04-05T21:26:24.297 INFO:tasks.workunit.client.1.smithi188.stdout:1/49: creat d1/f10 x:0 0 0 2024-04-05T21:26:24.298 INFO:tasks.workunit.client.1.smithi188.stdout:6/28: creat d6/fa x:0 0 0 2024-04-05T21:26:24.299 INFO:tasks.workunit.client.1.smithi188.stdout:0/41: dwrite f1 [0,4194304] 0 2024-04-05T21:26:24.303 INFO:tasks.workunit.client.1.smithi188.stdout:6/29: creat d6/fb x:0 0 0 2024-04-05T21:26:24.305 INFO:tasks.workunit.client.1.smithi188.stdout:1/50: getdents d1/d7 0 2024-04-05T21:26:24.307 INFO:tasks.workunit.client.1.smithi188.stdout:7/159: sync 2024-04-05T21:26:24.307 INFO:tasks.workunit.client.1.smithi188.stdout:2/82: sync 2024-04-05T21:26:24.307 INFO:tasks.workunit.client.1.smithi188.stdout:6/30: write d6/fb [250309,7904] 0 2024-04-05T21:26:24.307 INFO:tasks.workunit.client.1.smithi188.stdout:9/74: sync 2024-04-05T21:26:24.307 INFO:tasks.workunit.client.1.smithi188.stdout:3/38: sync 2024-04-05T21:26:24.307 INFO:tasks.workunit.client.1.smithi188.stdout:5/46: sync 2024-04-05T21:26:24.308 INFO:tasks.workunit.client.1.smithi188.stdout:8/56: sync 2024-04-05T21:26:24.308 INFO:tasks.workunit.client.1.smithi188.stdout:0/42: link d4/c5 d4/d8/db/dd/cf 0 2024-04-05T21:26:24.311 INFO:tasks.workunit.client.1.smithi188.stdout:8/57: chown d5/f9 0 1 2024-04-05T21:26:24.312 INFO:tasks.workunit.client.1.smithi188.stdout:4/48: dwrite d0/d14/d15/ff [0,4194304] 0 2024-04-05T21:26:24.312 INFO:tasks.workunit.client.1.smithi188.stdout:5/47: rename c5 to d6/cf 0 2024-04-05T21:26:24.314 INFO:tasks.workunit.client.1.smithi188.stdout:2/83: unlink d5/db/dd/c12 0 2024-04-05T21:26:24.314 INFO:tasks.workunit.client.1.smithi188.stdout:7/160: chown d3/d4/f8 130433557 1 2024-04-05T21:26:24.315 INFO:tasks.workunit.client.1.smithi188.stdout:6/31: write d6/fa [860794,102615] 0 2024-04-05T21:26:24.316 INFO:tasks.workunit.client.1.smithi188.stdout:2/84: chown d5/db 50900 1 2024-04-05T21:26:24.317 INFO:tasks.workunit.client.1.smithi188.stdout:5/48: mknod d6/c10 0 2024-04-05T21:26:24.318 INFO:tasks.workunit.client.1.smithi188.stdout:4/49: dread - d0/d2/f6 zero size 2024-04-05T21:26:24.318 INFO:tasks.workunit.client.1.smithi188.stdout:2/85: chown d5/c9 332866 1 2024-04-05T21:26:24.319 INFO:tasks.workunit.client.1.smithi188.stdout:9/75: rmdir dd 39 2024-04-05T21:26:24.320 INFO:tasks.workunit.client.1.smithi188.stdout:7/161: write d3/df/f30 [489659,90430] 0 2024-04-05T21:26:24.322 INFO:tasks.workunit.client.1.smithi188.stdout:1/51: link d1/d4/ff d1/dc/f11 0 2024-04-05T21:26:24.322 INFO:tasks.workunit.client.1.smithi188.stdout:7/162: readlink d3/d4/l15 0 2024-04-05T21:26:24.323 INFO:tasks.workunit.client.1.smithi188.stdout:8/58: dwrite d5/f8 [0,4194304] 0 2024-04-05T21:26:24.324 INFO:tasks.workunit.client.1.smithi188.stdout:7/163: stat d3/d4/db 0 2024-04-05T21:26:24.325 INFO:tasks.workunit.client.1.smithi188.stdout:6/32: chown d6/l9 211 1 2024-04-05T21:26:24.331 INFO:tasks.workunit.client.1.smithi188.stdout:5/49: creat d6/f11 x:0 0 0 2024-04-05T21:26:24.331 INFO:tasks.workunit.client.1.smithi188.stdout:5/50: truncate d6/fd 306058 0 2024-04-05T21:26:24.331 INFO:tasks.workunit.client.1.smithi188.stdout:7/164: chown d3/d4/db/d10/d2f 3 1 2024-04-05T21:26:24.341 INFO:tasks.workunit.client.1.smithi188.stdout:2/86: unlink f3 0 2024-04-05T21:26:24.341 INFO:tasks.workunit.client.1.smithi188.stdout:9/76: fsync dd/f12 0 2024-04-05T21:26:24.341 INFO:tasks.workunit.client.1.smithi188.stdout:6/33: dwrite d6/fa [0,4194304] 0 2024-04-05T21:26:24.341 INFO:tasks.workunit.client.1.smithi188.stdout:0/43: rename d4/d8/db/dd to d4/d8/d10 0 2024-04-05T21:26:24.341 INFO:tasks.workunit.client.1.smithi188.stdout:4/50: mkdir d0/d2/d17 0 2024-04-05T21:26:24.341 INFO:tasks.workunit.client.1.smithi188.stdout:6/34: truncate d6/fa 4961965 0 2024-04-05T21:26:24.342 INFO:tasks.workunit.client.1.smithi188.stdout:1/52: creat d1/d4/f12 x:0 0 0 2024-04-05T21:26:24.346 INFO:tasks.workunit.client.1.smithi188.stdout:5/51: creat d6/f12 x:0 0 0 2024-04-05T21:26:24.349 INFO:tasks.workunit.client.1.smithi188.stdout:9/77: dwrite f5 [0,4194304] 0 2024-04-05T21:26:24.349 INFO:tasks.workunit.client.1.smithi188.stdout:3/39: truncate d5/f6 1040276 0 2024-04-05T21:26:24.351 INFO:tasks.workunit.client.1.smithi188.stdout:9/78: dread - fb zero size 2024-04-05T21:26:24.355 INFO:tasks.workunit.client.1.smithi188.stdout:7/165: dwrite d3/d4/db/d10/f14 [0,4194304] 0 2024-04-05T21:26:24.356 INFO:tasks.workunit.client.1.smithi188.stdout:0/44: dwrite f1 [0,4194304] 0 2024-04-05T21:26:24.389 INFO:tasks.workunit.client.1.smithi188.stdout:5/52: mknod d6/c13 0 2024-04-05T21:26:24.389 INFO:tasks.workunit.client.1.smithi188.stdout:8/59: write d5/ff [76748,53499] 0 2024-04-05T21:26:24.390 INFO:tasks.workunit.client.1.smithi188.stdout:9/79: fsync f4 0 2024-04-05T21:26:24.390 INFO:tasks.workunit.client.1.smithi188.stdout:7/166: mknod d3/d4/db/d10/c31 0 2024-04-05T21:26:24.391 INFO:tasks.workunit.client.1.smithi188.stdout:0/45: fdatasync d4/d8/fa 0 2024-04-05T21:26:24.394 INFO:tasks.workunit.client.1.smithi188.stdout:2/87: write d5/fa [1241463,44500] 0 2024-04-05T21:26:24.395 INFO:tasks.workunit.client.1.smithi188.stdout:8/60: chown d5/fb 46492735 1 2024-04-05T21:26:24.395 INFO:tasks.workunit.client.1.smithi188.stdout:2/88: stat d5/fc 0 2024-04-05T21:26:24.396 INFO:tasks.workunit.client.1.smithi188.stdout:2/89: readlink d5/db/dd/le 0 2024-04-05T21:26:24.469 INFO:tasks.workunit.client.1.smithi188.stdout:7/167: mknod d3/c32 0 2024-04-05T21:26:24.469 INFO:tasks.workunit.client.1.smithi188.stdout:0/46: creat d4/d8/db/f11 x:0 0 0 2024-04-05T21:26:24.472 INFO:tasks.workunit.client.1.smithi188.stdout:2/90: write d5/fc [1717536,70278] 0 2024-04-05T21:26:24.472 INFO:tasks.workunit.client.1.smithi188.stdout:4/51: write d0/f5 [1818355,32163] 0 2024-04-05T21:26:24.474 INFO:tasks.workunit.client.1.smithi188.stdout:5/53: link d6/l7 d6/l14 0 2024-04-05T21:26:24.475 INFO:tasks.workunit.client.1.smithi188.stdout:2/91: stat d5/db/dd/f11 0 2024-04-05T21:26:24.475 INFO:tasks.workunit.client.1.smithi188.stdout:0/47: truncate f3 397286 0 2024-04-05T21:26:24.477 INFO:tasks.workunit.client.1.smithi188.stdout:4/52: creat d0/d2/d17/f18 x:0 0 0 2024-04-05T21:26:24.483 INFO:tasks.workunit.client.1.smithi188.stdout:5/54: getdents d6 0 2024-04-05T21:26:24.483 INFO:tasks.workunit.client.1.smithi188.stdout:4/53: symlink d0/d14/d15/l19 0 2024-04-05T21:26:24.486 INFO:tasks.workunit.client.1.smithi188.stdout:8/61: dwrite d5/f9 [0,4194304] 0 2024-04-05T21:26:24.486 INFO:tasks.workunit.client.1.smithi188.stdout:4/54: read - d0/d14/f16 zero size 2024-04-05T21:26:24.486 INFO:tasks.workunit.client.1.smithi188.stdout:2/92: getdents d5 0 2024-04-05T21:26:24.487 INFO:tasks.workunit.client.1.smithi188.stdout:7/168: dread d3/d4/f7 [0,4194304] 0 2024-04-05T21:26:24.489 INFO:tasks.workunit.client.1.smithi188.stdout:8/62: truncate d5/ff 769328 0 2024-04-05T21:26:24.489 INFO:tasks.workunit.client.1.smithi188.stdout:5/55: write d6/f9 [385205,36565] 0 2024-04-05T21:26:24.490 INFO:tasks.workunit.client.1.smithi188.stdout:7/169: unlink d3/d4/f8 0 2024-04-05T21:26:24.491 INFO:tasks.workunit.client.1.smithi188.stdout:4/55: creat d0/d14/d15/f1a x:0 0 0 2024-04-05T21:26:24.494 INFO:tasks.workunit.client.1.smithi188.stdout:8/63: mknod d5/dc/c14 0 2024-04-05T21:26:24.495 INFO:tasks.workunit.client.1.smithi188.stdout:4/56: truncate d0/d2/f6 16810 0 2024-04-05T21:26:24.498 INFO:tasks.workunit.client.1.smithi188.stdout:7/170: creat d3/d4/db/d10/f33 x:0 0 0 2024-04-05T21:26:24.498 INFO:tasks.workunit.client.1.smithi188.stdout:8/64: stat d5/dc/f10 0 2024-04-05T21:26:24.500 INFO:tasks.workunit.client.1.smithi188.stdout:8/65: stat d5/ff 0 2024-04-05T21:26:24.502 INFO:tasks.workunit.client.1.smithi188.stdout:4/57: fsync d0/d14/d15/fb 0 2024-04-05T21:26:24.504 INFO:tasks.workunit.client.1.smithi188.stdout:0/48: dwrite f3 [0,4194304] 0 2024-04-05T21:26:24.505 INFO:tasks.workunit.client.1.smithi188.stdout:8/66: chown d5/f9 289 1 2024-04-05T21:26:24.508 INFO:tasks.workunit.client.1.smithi188.stdout:7/171: truncate d3/f1e 2855861 0 2024-04-05T21:26:24.508 INFO:tasks.workunit.client.1.smithi188.stdout:4/58: mkdir d0/d14/d15/d1b 0 2024-04-05T21:26:24.509 INFO:tasks.workunit.client.1.smithi188.stdout:5/56: dwrite d6/fd [0,4194304] 0 2024-04-05T21:26:24.510 INFO:tasks.workunit.client.1.smithi188.stdout:5/57: stat d6 0 2024-04-05T21:26:24.510 INFO:tasks.workunit.client.1.smithi188.stdout:7/172: truncate d3/d4/f2d 1035615 0 2024-04-05T21:26:24.511 INFO:tasks.workunit.client.1.smithi188.stdout:7/173: readlink d3/d4/l15 0 2024-04-05T21:26:24.512 INFO:tasks.workunit.client.1.smithi188.stdout:0/49: creat d4/d6/f12 x:0 0 0 2024-04-05T21:26:24.512 INFO:tasks.workunit.client.1.smithi188.stdout:5/58: mkdir d6/d15 0 2024-04-05T21:26:24.513 INFO:tasks.workunit.client.1.smithi188.stdout:0/50: chown d4/d8/db 4376861 1 2024-04-05T21:26:24.515 INFO:tasks.workunit.client.1.smithi188.stdout:4/59: creat d0/d14/f1c x:0 0 0 2024-04-05T21:26:24.516 INFO:tasks.workunit.client.1.smithi188.stdout:5/59: dread - d6/f11 zero size 2024-04-05T21:26:24.518 INFO:tasks.workunit.client.1.smithi188.stdout:0/51: unlink d4/c7 0 2024-04-05T21:26:24.520 INFO:tasks.workunit.client.1.smithi188.stdout:2/93: dwrite d5/db/dd/f11 [0,4194304] 0 2024-04-05T21:26:24.521 INFO:tasks.workunit.client.1.smithi188.stdout:7/174: getdents d3/d4/d1d 0 2024-04-05T21:26:24.521 INFO:tasks.workunit.client.1.smithi188.stdout:0/52: chown f3 15 1 2024-04-05T21:26:24.524 INFO:tasks.workunit.client.1.smithi188.stdout:0/53: dread - d4/d8/db/f11 zero size 2024-04-05T21:26:24.524 INFO:tasks.workunit.client.1.smithi188.stdout:5/60: creat d6/d15/f16 x:0 0 0 2024-04-05T21:26:24.524 INFO:tasks.workunit.client.1.smithi188.stdout:4/60: creat d0/d14/d15/d1b/f1d x:0 0 0 2024-04-05T21:26:24.525 INFO:tasks.workunit.client.1.smithi188.stdout:2/94: write d5/db/dd/f15 [652339,32453] 0 2024-04-05T21:26:24.525 INFO:tasks.workunit.client.1.smithi188.stdout:7/175: creat d3/d4/d1d/f34 x:0 0 0 2024-04-05T21:26:24.526 INFO:tasks.workunit.client.1.smithi188.stdout:5/61: mkdir d6/d17 0 2024-04-05T21:26:24.529 INFO:tasks.workunit.client.1.smithi188.stdout:0/54: mkdir d4/d6/d13 0 2024-04-05T21:26:24.530 INFO:tasks.workunit.client.1.smithi188.stdout:2/95: dread d5/fa [0,4194304] 0 2024-04-05T21:26:24.537 INFO:tasks.workunit.client.1.smithi188.stdout:5/62: mkdir d6/d17/d18 0 2024-04-05T21:26:24.538 INFO:tasks.workunit.client.1.smithi188.stdout:2/96: dwrite d5/db/dd/f11 [0,4194304] 0 2024-04-05T21:26:24.538 INFO:tasks.workunit.client.1.smithi188.stdout:7/176: creat d3/d4/db/d2e/f35 x:0 0 0 2024-04-05T21:26:24.538 INFO:tasks.workunit.client.1.smithi188.stdout:5/63: dread - d6/f11 zero size 2024-04-05T21:26:24.539 INFO:tasks.workunit.client.1.smithi188.stdout:5/64: readlink d6/lc 0 2024-04-05T21:26:24.542 INFO:tasks.workunit.client.1.smithi188.stdout:2/97: rmdir d5/db/dd 39 2024-04-05T21:26:24.544 INFO:tasks.workunit.client.1.smithi188.stdout:5/65: dread d6/fd [0,4194304] 0 2024-04-05T21:26:24.545 INFO:tasks.workunit.client.1.smithi188.stdout:7/177: dread d3/d4/db/f1c [0,4194304] 0 2024-04-05T21:26:24.547 INFO:tasks.workunit.client.1.smithi188.stdout:4/61: rmdir d0/d14/d15 39 2024-04-05T21:26:24.551 INFO:tasks.workunit.client.1.smithi188.stdout:5/66: creat d6/f19 x:0 0 0 2024-04-05T21:26:24.552 INFO:tasks.workunit.client.1.smithi188.stdout:7/178: mknod d3/d4/d1d/c36 0 2024-04-05T21:26:24.552 INFO:tasks.workunit.client.1.smithi188.stdout:7/179: chown d3/d2c 297149202 1 2024-04-05T21:26:24.553 INFO:tasks.workunit.client.1.smithi188.stdout:7/180: chown d3/df/c11 7053 1 2024-04-05T21:26:24.553 INFO:tasks.workunit.client.1.smithi188.stdout:7/181: dread - d3/d4/d1d/f34 zero size 2024-04-05T21:26:24.556 INFO:tasks.workunit.client.1.smithi188.stdout:5/67: rename d6/cb to d6/c1a 0 2024-04-05T21:26:24.558 INFO:tasks.workunit.client.1.smithi188.stdout:4/62: creat d0/d14/d15/d1b/f1e x:0 0 0 2024-04-05T21:26:24.559 INFO:tasks.workunit.client.1.smithi188.stdout:7/182: mkdir d3/d4/db/d2e/d37 0 2024-04-05T21:26:24.561 INFO:tasks.workunit.client.1.smithi188.stdout:0/55: dwrite d4/d8/fa [0,4194304] 0 2024-04-05T21:26:24.561 INFO:tasks.workunit.client.1.smithi188.stdout:2/98: link d5/db/dd/f11 d5/db/dd/f1c 0 2024-04-05T21:26:24.561 INFO:tasks.workunit.client.1.smithi188.stdout:0/56: readlink - no filename 2024-04-05T21:26:24.562 INFO:tasks.workunit.client.1.smithi188.stdout:5/68: rmdir d6/d15 39 2024-04-05T21:26:24.562 INFO:tasks.workunit.client.1.smithi188.stdout:0/57: truncate d4/fe 735096 0 2024-04-05T21:26:24.565 INFO:tasks.workunit.client.1.smithi188.stdout:7/183: creat d3/d4/db/f38 x:0 0 0 2024-04-05T21:26:24.571 INFO:tasks.workunit.client.1.smithi188.stdout:7/184: creat d3/df/f39 x:0 0 0 2024-04-05T21:26:24.573 INFO:tasks.workunit.client.1.smithi188.stdout:5/69: mkdir d6/d15/d1b 0 2024-04-05T21:26:24.576 INFO:tasks.workunit.client.1.smithi188.stdout:6/35: sync 2024-04-05T21:26:24.577 INFO:tasks.workunit.client.1.smithi188.stdout:0/58: link f1 d4/d6/d13/f14 0 2024-04-05T21:26:24.578 INFO:tasks.workunit.client.1.smithi188.stdout:7/185: truncate d3/d4/f7 2264499 0 2024-04-05T21:26:24.578 INFO:tasks.workunit.client.1.smithi188.stdout:6/36: truncate d6/fb 522533 0 2024-04-05T21:26:24.578 INFO:tasks.workunit.client.1.smithi188.stdout:2/99: creat d5/db/f1d x:0 0 0 2024-04-05T21:26:24.579 INFO:tasks.workunit.client.1.smithi188.stdout:4/63: truncate d0/f5 967207 0 2024-04-05T21:26:24.579 INFO:tasks.workunit.client.1.smithi188.stdout:5/70: mknod d6/d17/c1c 0 2024-04-05T21:26:24.579 INFO:tasks.workunit.client.1.smithi188.stdout:6/37: write d6/fb [89130,17870] 0 2024-04-05T21:26:24.581 INFO:tasks.workunit.client.1.smithi188.stdout:6/38: write f5 [715909,107575] 0 2024-04-05T21:26:24.581 INFO:tasks.workunit.client.1.smithi188.stdout:0/59: creat d4/d6/f15 x:0 0 0 2024-04-05T21:26:24.582 INFO:tasks.workunit.client.1.smithi188.stdout:2/100: rmdir d5/db/dd 39 2024-04-05T21:26:24.582 INFO:tasks.workunit.client.1.smithi188.stdout:6/39: readlink d6/l9 0 2024-04-05T21:26:24.583 INFO:tasks.workunit.client.1.smithi188.stdout:4/64: fdatasync d0/d14/d15/d1b/f1e 0 2024-04-05T21:26:24.584 INFO:tasks.workunit.client.1.smithi188.stdout:7/186: symlink d3/d2c/l3a 0 2024-04-05T21:26:24.584 INFO:tasks.workunit.client.1.smithi188.stdout:5/71: symlink d6/d17/l1d 0 2024-04-05T21:26:24.586 INFO:tasks.workunit.client.1.smithi188.stdout:0/60: mknod d4/d8/db/c16 0 2024-04-05T21:26:24.588 INFO:tasks.workunit.client.1.smithi188.stdout:4/65: write d0/d14/f1c [751802,102657] 0 2024-04-05T21:26:24.588 INFO:tasks.workunit.client.1.smithi188.stdout:5/72: chown d6/d17/d18 5309 1 2024-04-05T21:26:24.588 INFO:tasks.workunit.client.1.smithi188.stdout:7/187: mkdir d3/d4/db/d2e/d3b 0 2024-04-05T21:26:24.590 INFO:tasks.workunit.client.1.smithi188.stdout:0/61: chown d4/d8/db 21856 1 2024-04-05T21:26:24.590 INFO:tasks.workunit.client.1.smithi188.stdout:5/73: dread - d6/f19 zero size 2024-04-05T21:26:24.591 INFO:tasks.workunit.client.1.smithi188.stdout:6/40: creat d6/d7/fc x:0 0 0 2024-04-05T21:26:24.594 INFO:tasks.workunit.client.1.smithi188.stdout:0/62: chown f1 2022 1 2024-04-05T21:26:24.594 INFO:tasks.workunit.client.1.smithi188.stdout:5/74: dread - d6/f12 zero size 2024-04-05T21:26:24.594 INFO:tasks.workunit.client.1.smithi188.stdout:4/66: symlink d0/d14/l1f 0 2024-04-05T21:26:24.596 INFO:tasks.workunit.client.1.smithi188.stdout:5/75: chown d6/f19 0 1 2024-04-05T21:26:24.596 INFO:tasks.workunit.client.1.smithi188.stdout:6/41: creat d6/d7/fd x:0 0 0 2024-04-05T21:26:24.598 INFO:tasks.workunit.client.1.smithi188.stdout:2/101: dwrite d5/db/f1d [0,4194304] 0 2024-04-05T21:26:24.599 INFO:tasks.workunit.client.1.smithi188.stdout:5/76: dread - d6/f19 zero size 2024-04-05T21:26:24.599 INFO:tasks.workunit.client.1.smithi188.stdout:0/63: truncate d4/d8/db/f11 913296 0 2024-04-05T21:26:24.600 INFO:tasks.workunit.client.1.smithi188.stdout:7/188: mkdir d3/d4/db/d10/d3c 0 2024-04-05T21:26:24.605 INFO:tasks.workunit.client.1.smithi188.stdout:5/77: chown d6/f12 46582030 1 2024-04-05T21:26:24.609 INFO:tasks.workunit.client.1.smithi188.stdout:4/67: symlink d0/d14/d15/d1b/l20 0 2024-04-05T21:26:24.611 INFO:tasks.workunit.client.1.smithi188.stdout:4/68: fsync d0/d14/d15/d1b/f1e 0 2024-04-05T21:26:24.612 INFO:tasks.workunit.client.1.smithi188.stdout:7/189: rename d3/d4/db/d10/c31 to d3/d4/db/d2e/d3b/c3d 0 2024-04-05T21:26:24.613 INFO:tasks.workunit.client.1.smithi188.stdout:6/42: mknod d6/ce 0 2024-04-05T21:26:24.616 INFO:tasks.workunit.client.1.smithi188.stdout:2/102: dread d5/fc [4194304,4194304] 0 2024-04-05T21:26:24.617 INFO:tasks.workunit.client.1.smithi188.stdout:4/69: write d0/d14/d15/ff [3768406,52764] 0 2024-04-05T21:26:24.620 INFO:tasks.workunit.client.1.smithi188.stdout:4/70: write d0/d14/d15/d1b/f1d [776537,28216] 0 2024-04-05T21:26:24.622 INFO:tasks.workunit.client.1.smithi188.stdout:5/78: dread d6/fd [0,4194304] 0 2024-04-05T21:26:24.622 INFO:tasks.workunit.client.1.smithi188.stdout:4/71: write d0/d14/d15/fb [819014,89410] 0 2024-04-05T21:26:24.624 INFO:tasks.workunit.client.1.smithi188.stdout:6/43: symlink d6/lf 0 2024-04-05T21:26:24.627 INFO:tasks.workunit.client.1.smithi188.stdout:4/72: rmdir d0/d14 39 2024-04-05T21:26:24.629 INFO:tasks.workunit.client.1.smithi188.stdout:2/103: link d5/c9 d5/c1e 0 2024-04-05T21:26:24.630 INFO:tasks.workunit.client.1.smithi188.stdout:7/190: getdents d3/d2c 0 2024-04-05T21:26:24.633 INFO:tasks.workunit.client.1.smithi188.stdout:6/44: dwrite d6/fa [0,4194304] 0 2024-04-05T21:26:24.633 INFO:tasks.workunit.client.1.smithi188.stdout:4/73: dread - d0/d14/d15/f1a zero size 2024-04-05T21:26:24.634 INFO:tasks.workunit.client.1.smithi188.stdout:7/191: creat d3/f3e x:0 0 0 2024-04-05T21:26:24.637 INFO:tasks.workunit.client.1.smithi188.stdout:5/79: rmdir d6/d17/d18 0 2024-04-05T21:26:24.637 INFO:tasks.workunit.client.1.smithi188.stdout:7/192: write d3/df/f1f [408741,22718] 0 2024-04-05T21:26:24.638 INFO:tasks.workunit.client.1.smithi188.stdout:6/45: chown f5 977917862 1 2024-04-05T21:26:24.638 INFO:tasks.workunit.client.1.smithi188.stdout:6/46: fdatasync f5 0 2024-04-05T21:26:24.644 INFO:tasks.workunit.client.1.smithi188.stdout:2/104: getdents d5/db 0 2024-04-05T21:26:24.644 INFO:tasks.workunit.client.1.smithi188.stdout:6/47: rename d6/ce to d6/d7/c10 0 2024-04-05T21:26:24.644 INFO:tasks.workunit.client.1.smithi188.stdout:4/74: link d0/d14/d15/c13 d0/d14/d15/c21 0 2024-04-05T21:26:24.646 INFO:tasks.workunit.client.1.smithi188.stdout:6/48: stat d6/f8 0 2024-04-05T21:26:24.647 INFO:tasks.workunit.client.1.smithi188.stdout:2/105: symlink d5/l1f 0 2024-04-05T21:26:24.648 INFO:tasks.workunit.client.1.smithi188.stdout:5/80: dwrite f2 [0,4194304] 0 2024-04-05T21:26:24.651 INFO:tasks.workunit.client.1.smithi188.stdout:5/81: chown d6/d15/f16 28 1 2024-04-05T21:26:24.651 INFO:tasks.workunit.client.1.smithi188.stdout:2/106: fdatasync d5/db/dd/f15 0 2024-04-05T21:26:24.653 INFO:tasks.workunit.client.1.smithi188.stdout:4/75: dread d0/f5 [0,4194304] 0 2024-04-05T21:26:24.654 INFO:tasks.workunit.client.1.smithi188.stdout:2/107: stat d5/db/f16 0 2024-04-05T21:26:24.654 INFO:tasks.workunit.client.1.smithi188.stdout:7/193: dwrite d3/d4/db/d10/f14 [0,4194304] 0 2024-04-05T21:26:24.657 INFO:tasks.workunit.client.1.smithi188.stdout:2/108: stat d5/db/dd/le 0 2024-04-05T21:26:24.658 INFO:tasks.workunit.client.1.smithi188.stdout:4/76: truncate d0/d2/f6 505839 0 2024-04-05T21:26:24.659 INFO:tasks.workunit.client.1.smithi188.stdout:2/109: chown d5/db/dd/le 22758111 1 2024-04-05T21:26:24.660 INFO:tasks.workunit.client.1.smithi188.stdout:6/49: dwrite d6/f8 [0,4194304] 0 2024-04-05T21:26:24.660 INFO:tasks.workunit.client.1.smithi188.stdout:7/194: dread d3/d4/db/fe [0,4194304] 0 2024-04-05T21:26:24.662 INFO:tasks.workunit.client.1.smithi188.stdout:4/77: write d0/d14/d15/fb [647502,41398] 0 2024-04-05T21:26:24.662 INFO:tasks.workunit.client.1.smithi188.stdout:6/50: write d6/d7/fd [992557,115563] 0 2024-04-05T21:26:24.666 INFO:tasks.workunit.client.1.smithi188.stdout:2/110: fdatasync d5/db/dd/f15 0 2024-04-05T21:26:24.669 INFO:tasks.workunit.client.1.smithi188.stdout:7/195: rmdir d3 39 2024-04-05T21:26:24.669 INFO:tasks.workunit.client.1.smithi188.stdout:4/78: dread d0/d2/f6 [0,4194304] 0 2024-04-05T21:26:24.672 INFO:tasks.workunit.client.1.smithi188.stdout:2/111: write d5/fc [708434,48485] 0 2024-04-05T21:26:24.673 INFO:tasks.workunit.client.1.smithi188.stdout:4/79: symlink d0/d14/d15/l22 0 2024-04-05T21:26:24.676 INFO:tasks.workunit.client.1.smithi188.stdout:6/51: dwrite d6/d7/fd [0,4194304] 0 2024-04-05T21:26:24.676 INFO:tasks.workunit.client.1.smithi188.stdout:4/80: mknod d0/d2/c23 0 2024-04-05T21:26:24.678 INFO:tasks.workunit.client.1.smithi188.stdout:2/112: rename d5/db/f1d to d5/db/f20 0 2024-04-05T21:26:24.681 INFO:tasks.workunit.client.1.smithi188.stdout:4/81: truncate d0/d2/f10 599822 0 2024-04-05T21:26:24.681 INFO:tasks.workunit.client.1.smithi188.stdout:2/113: symlink d5/l21 0 2024-04-05T21:26:24.682 INFO:tasks.workunit.client.1.smithi188.stdout:6/52: creat d6/f11 x:0 0 0 2024-04-05T21:26:24.683 INFO:tasks.workunit.client.1.smithi188.stdout:7/196: dwrite d3/df/f20 [0,4194304] 0 2024-04-05T21:26:24.686 INFO:tasks.workunit.client.1.smithi188.stdout:7/197: chown d3/d4/db/f2a 4 1 2024-04-05T21:26:24.686 INFO:tasks.workunit.client.1.smithi188.stdout:6/53: symlink d6/l12 0 2024-04-05T21:26:24.687 INFO:tasks.workunit.client.1.smithi188.stdout:2/114: write d5/db/dd/f11 [1532728,45769] 0 2024-04-05T21:26:24.694 INFO:tasks.workunit.client.1.smithi188.stdout:4/82: rename d0/d2/c23 to d0/c24 0 2024-04-05T21:26:24.696 INFO:tasks.workunit.client.1.smithi188.stdout:2/115: chown d5/c9 1 1 2024-04-05T21:26:24.697 INFO:tasks.workunit.client.1.smithi188.stdout:6/54: mkdir d6/d13 0 2024-04-05T21:26:24.699 INFO:tasks.workunit.client.1.smithi188.stdout:4/83: mknod d0/d14/c25 0 2024-04-05T21:26:24.712 INFO:tasks.workunit.client.1.smithi188.stdout:2/116: dwrite d5/db/f20 [0,4194304] 0 2024-04-05T21:26:24.713 INFO:tasks.workunit.client.1.smithi188.stdout:6/55: dwrite d6/fb [0,4194304] 0 2024-04-05T21:26:24.719 INFO:tasks.workunit.client.1.smithi188.stdout:2/117: chown d5/db/dd/f1c 2954 1 2024-04-05T21:26:24.721 INFO:tasks.workunit.client.1.smithi188.stdout:7/198: dwrite d3/df/f27 [0,4194304] 0 2024-04-05T21:26:24.722 INFO:tasks.workunit.client.1.smithi188.stdout:4/84: dwrite d0/d2/f10 [0,4194304] 0 2024-04-05T21:26:24.724 INFO:tasks.workunit.client.1.smithi188.stdout:7/199: stat d3/df/f39 0 2024-04-05T21:26:24.726 INFO:tasks.workunit.client.1.smithi188.stdout:2/118: fsync d5/db/f20 0 2024-04-05T21:26:24.727 INFO:tasks.workunit.client.1.smithi188.stdout:4/85: fdatasync d0/d2/f6 0 2024-04-05T21:26:24.727 INFO:tasks.workunit.client.1.smithi188.stdout:7/200: creat d3/d2c/f3f x:0 0 0 2024-04-05T21:26:24.728 INFO:tasks.workunit.client.1.smithi188.stdout:2/119: rmdir d5/db 39 2024-04-05T21:26:24.730 INFO:tasks.workunit.client.1.smithi188.stdout:4/86: write d0/d14/d15/ff [3003702,4989] 0 2024-04-05T21:26:24.732 INFO:tasks.workunit.client.1.smithi188.stdout:7/201: mknod d3/d4/c40 0 2024-04-05T21:26:24.740 INFO:tasks.workunit.client.1.smithi188.stdout:4/87: dread d0/d14/d15/fb [0,4194304] 0 2024-04-05T21:26:24.742 INFO:tasks.workunit.client.1.smithi188.stdout:4/88: write d0/d14/d15/f1a [793259,67184] 0 2024-04-05T21:26:24.744 INFO:tasks.workunit.client.1.smithi188.stdout:2/120: dwrite d5/db/f16 [4194304,4194304] 0 2024-04-05T21:26:24.744 INFO:tasks.workunit.client.1.smithi188.stdout:4/89: creat d0/d2/f26 x:0 0 0 2024-04-05T21:26:24.749 INFO:tasks.workunit.client.1.smithi188.stdout:4/90: write d0/d2/d17/f18 [573182,76883] 0 2024-04-05T21:26:24.750 INFO:tasks.workunit.client.1.smithi188.stdout:7/202: write d3/f1e [2491882,16270] 0 2024-04-05T21:26:24.756 INFO:tasks.workunit.client.1.smithi188.stdout:2/121: getdents d5/db/dd 0 2024-04-05T21:26:24.756 INFO:tasks.workunit.client.1.smithi188.stdout:7/203: creat d3/d4/db/d10/d2f/f41 x:0 0 0 2024-04-05T21:26:24.756 INFO:tasks.workunit.client.1.smithi188.stdout:7/204: readlink d3/d2c/l3a 0 2024-04-05T21:26:24.758 INFO:tasks.workunit.client.1.smithi188.stdout:7/205: write d3/d4/db/d10/d2f/f41 [168975,39656] 0 2024-04-05T21:26:24.759 INFO:tasks.workunit.client.1.smithi188.stdout:2/122: dread d5/f14 [0,4194304] 0 2024-04-05T21:26:24.761 INFO:tasks.workunit.client.1.smithi188.stdout:7/206: write d3/df/f39 [631493,79175] 0 2024-04-05T21:26:24.767 INFO:tasks.workunit.client.1.smithi188.stdout:4/91: dwrite d0/f5 [0,4194304] 0 2024-04-05T21:26:24.770 INFO:tasks.workunit.client.1.smithi188.stdout:7/207: getdents d3/d4/db/d10/d2f 0 2024-04-05T21:26:24.770 INFO:tasks.workunit.client.1.smithi188.stdout:7/208: readlink l2 0 2024-04-05T21:26:24.773 INFO:tasks.workunit.client.1.smithi188.stdout:2/123: dwrite d5/db/dd/f11 [0,4194304] 0 2024-04-05T21:26:24.775 INFO:tasks.workunit.client.1.smithi188.stdout:4/92: dread d0/d2/f10 [0,4194304] 0 2024-04-05T21:26:24.776 INFO:tasks.workunit.client.1.smithi188.stdout:7/209: truncate d3/d4/db/f2a 647129 0 2024-04-05T21:26:24.778 INFO:tasks.workunit.client.1.smithi188.stdout:2/124: unlink d5/db/dd/le 0 2024-04-05T21:26:24.779 INFO:tasks.workunit.client.1.smithi188.stdout:4/93: mknod d0/d2/c27 0 2024-04-05T21:26:24.781 INFO:tasks.workunit.client.1.smithi188.stdout:4/94: symlink d0/d14/l28 0 2024-04-05T21:26:24.782 INFO:tasks.workunit.client.1.smithi188.stdout:7/210: link c1 d3/d4/db/d2e/d37/c42 0 2024-04-05T21:26:24.783 INFO:tasks.workunit.client.1.smithi188.stdout:4/95: creat d0/d14/d15/f29 x:0 0 0 2024-04-05T21:26:24.786 INFO:tasks.workunit.client.1.smithi188.stdout:4/96: dread - d0/d14/d15/d1b/f1e zero size 2024-04-05T21:26:24.789 INFO:tasks.workunit.client.1.smithi188.stdout:7/211: dwrite d3/d4/db/d2e/f35 [0,4194304] 0 2024-04-05T21:26:24.791 INFO:tasks.workunit.client.1.smithi188.stdout:7/212: creat d3/d4/db/d2e/d3b/f43 x:0 0 0 2024-04-05T21:26:24.792 INFO:tasks.workunit.client.1.smithi188.stdout:7/213: truncate d3/f1e 3612169 0 2024-04-05T21:26:24.793 INFO:tasks.workunit.client.1.smithi188.stdout:7/214: symlink d3/d4/d1d/l44 0 2024-04-05T21:26:24.794 INFO:tasks.workunit.client.1.smithi188.stdout:7/215: readlink d3/d4/d1d/l44 0 2024-04-05T21:26:24.795 INFO:tasks.workunit.client.1.smithi188.stdout:7/216: fdatasync d3/d4/db/d2e/d3b/f43 0 2024-04-05T21:26:24.808 INFO:tasks.workunit.client.1.smithi188.stdout:7/217: write d3/d4/db/f2a [1048711,40754] 0 2024-04-05T21:26:24.809 INFO:tasks.workunit.client.1.smithi188.stdout:7/218: dread - d3/d4/d1d/f2b zero size 2024-04-05T21:26:24.811 INFO:tasks.workunit.client.1.smithi188.stdout:7/219: read d3/d4/f25 [335423,55977] 0 2024-04-05T21:26:24.812 INFO:tasks.workunit.client.1.smithi188.stdout:7/220: symlink d3/d4/db/d10/d2f/l45 0 2024-04-05T21:26:24.813 INFO:tasks.workunit.client.1.smithi188.stdout:7/221: write d3/d4/d1d/f34 [53344,70921] 0 2024-04-05T21:26:24.814 INFO:tasks.workunit.client.1.smithi188.stdout:7/222: mknod d3/c46 0 2024-04-05T21:26:24.814 INFO:tasks.workunit.client.1.smithi188.stdout:7/223: chown d3/d2c/f3f 97450 1 2024-04-05T21:26:24.815 INFO:tasks.workunit.client.1.smithi188.stdout:7/224: write d3/d4/db/f2a [746049,13815] 0 2024-04-05T21:26:24.834 INFO:tasks.workunit.client.1.smithi188.stdout:1/53: sync 2024-04-05T21:26:24.836 INFO:tasks.workunit.client.1.smithi188.stdout:1/54: fdatasync d1/d7/fa 0 2024-04-05T21:26:24.842 INFO:tasks.workunit.client.1.smithi188.stdout:1/55: dwrite d1/d7/fa [0,4194304] 0 2024-04-05T21:26:24.843 INFO:tasks.workunit.client.1.smithi188.stdout:1/56: dread - d1/dc/f11 zero size 2024-04-05T21:26:24.847 INFO:tasks.workunit.client.1.smithi188.stdout:1/57: rename d1/d4/d5 to d1/d4/d13 0 2024-04-05T21:26:24.848 INFO:tasks.workunit.client.1.smithi188.stdout:1/58: chown d1/d4/ff 11 1 2024-04-05T21:26:24.850 INFO:tasks.workunit.client.1.smithi188.stdout:1/59: write d1/dc/f11 [744364,76800] 0 2024-04-05T21:26:24.851 INFO:tasks.workunit.client.1.smithi188.stdout:1/60: write d1/dc/f11 [45873,95538] 0 2024-04-05T21:26:24.858 INFO:tasks.workunit.client.1.smithi188.stdout:1/61: dwrite d1/d4/f12 [0,4194304] 0 2024-04-05T21:26:24.859 INFO:tasks.workunit.client.1.smithi188.stdout:1/62: chown d1/ld 45066940 1 2024-04-05T21:26:24.860 INFO:tasks.workunit.client.1.smithi188.stdout:1/63: rmdir d1 39 2024-04-05T21:26:24.865 INFO:tasks.workunit.client.1.smithi188.stdout:1/64: rename d1/dc/fe to d1/f14 0 2024-04-05T21:26:24.872 INFO:tasks.workunit.client.1.smithi188.stdout:1/65: dwrite d1/d4/f12 [0,4194304] 0 2024-04-05T21:26:24.873 INFO:tasks.workunit.client.1.smithi188.stdout:1/66: chown d1/dc 9721 1 2024-04-05T21:26:24.875 INFO:tasks.workunit.client.1.smithi188.stdout:1/67: rename d1/d7/fa to d1/dc/f15 0 2024-04-05T21:26:24.880 INFO:tasks.workunit.client.1.smithi188.stdout:1/68: dread d1/d4/ff [0,4194304] 0 2024-04-05T21:26:24.881 INFO:tasks.workunit.client.1.smithi188.stdout:1/69: rmdir d1 39 2024-04-05T21:26:24.883 INFO:tasks.workunit.client.1.smithi188.stdout:1/70: fdatasync d1/dc/f11 0 2024-04-05T21:26:24.892 INFO:tasks.workunit.client.1.smithi188.stdout:1/71: write d1/fb [1142211,92318] 0 2024-04-05T21:26:24.894 INFO:tasks.workunit.client.1.smithi188.stdout:1/72: mkdir d1/dc/d16 0 2024-04-05T21:26:24.901 INFO:tasks.workunit.client.1.smithi188.stdout:1/73: dwrite d1/f14 [0,4194304] 0 2024-04-05T21:26:24.902 INFO:tasks.workunit.client.1.smithi188.stdout:1/74: readlink d1/d4/d13/l8 0 2024-04-05T21:26:24.903 INFO:tasks.workunit.client.1.smithi188.stdout:1/75: write d1/f14 [2795530,53256] 0 2024-04-05T21:26:24.904 INFO:tasks.workunit.client.1.smithi188.stdout:1/76: write d1/f14 [1453166,119388] 0 2024-04-05T21:26:24.907 INFO:tasks.workunit.client.1.smithi188.stdout:1/77: unlink d1/d4/ff 0 2024-04-05T21:26:24.917 INFO:tasks.workunit.client.1.smithi188.stdout:1/78: rmdir d1/d4 39 2024-04-05T21:26:24.920 INFO:tasks.workunit.client.1.smithi188.stdout:1/79: creat d1/d4/f17 x:0 0 0 2024-04-05T21:26:24.922 INFO:tasks.workunit.client.1.smithi188.stdout:1/80: creat d1/d4/d13/f18 x:0 0 0 2024-04-05T21:26:24.924 INFO:tasks.workunit.client.1.smithi188.stdout:1/81: write d1/d4/f17 [694460,831] 0 2024-04-05T21:26:24.924 INFO:tasks.workunit.client.1.smithi188.stdout:1/82: chown d1/d7/c9 2756 1 2024-04-05T21:26:24.925 INFO:tasks.workunit.client.1.smithi188.stdout:1/83: dread - d1/f10 zero size 2024-04-05T21:26:24.930 INFO:tasks.workunit.client.1.smithi188.stdout:1/84: dwrite d1/d4/f12 [0,4194304] 0 2024-04-05T21:26:24.934 INFO:tasks.workunit.client.1.smithi188.stdout:1/85: dread d1/dc/f11 [0,4194304] 0 2024-04-05T21:26:24.938 INFO:tasks.workunit.client.1.smithi188.stdout:1/86: rename d1/d4/f17 to d1/dc/f19 0 2024-04-05T21:26:24.938 INFO:tasks.workunit.client.1.smithi188.stdout:1/87: read - d1/f10 zero size 2024-04-05T21:26:24.940 INFO:tasks.workunit.client.1.smithi188.stdout:1/88: mknod d1/d7/c1a 0 2024-04-05T21:26:24.941 INFO:tasks.workunit.client.1.smithi188.stdout:1/89: read d1/d4/f12 [3858094,31017] 0 2024-04-05T21:26:24.944 INFO:tasks.workunit.client.1.smithi188.stdout:1/90: creat d1/dc/f1b x:0 0 0 2024-04-05T21:26:24.949 INFO:tasks.workunit.client.1.smithi188.stdout:1/91: rmdir d1/dc 39 2024-04-05T21:26:24.952 INFO:tasks.workunit.client.1.smithi188.stdout:1/92: symlink d1/dc/d16/l1c 0 2024-04-05T21:26:24.952 INFO:tasks.workunit.client.1.smithi188.stdout:1/93: write d1/dc/f1b [825166,166] 0 2024-04-05T21:26:24.953 INFO:tasks.workunit.client.1.smithi188.stdout:1/94: dread - d1/d4/d13/f18 zero size 2024-04-05T21:26:24.954 INFO:tasks.workunit.client.1.smithi188.stdout:1/95: creat d1/d4/f1d x:0 0 0 2024-04-05T21:26:24.960 INFO:tasks.workunit.client.1.smithi188.stdout:1/96: dwrite d1/f14 [4194304,4194304] 0 2024-04-05T21:26:24.962 INFO:tasks.workunit.client.1.smithi188.stdout:1/97: write d1/dc/f1b [1829472,41430] 0 2024-04-05T21:26:24.964 INFO:tasks.workunit.client.1.smithi188.stdout:1/98: symlink d1/dc/l1e 0 2024-04-05T21:26:24.965 INFO:tasks.workunit.client.1.smithi188.stdout:1/99: chown d1/fb 422 1 2024-04-05T21:26:24.968 INFO:tasks.workunit.client.1.smithi188.stdout:1/100: creat d1/dc/f1f x:0 0 0 2024-04-05T21:26:24.970 INFO:tasks.workunit.client.1.smithi188.stdout:1/101: symlink d1/d4/d13/l20 0 2024-04-05T21:26:24.976 INFO:tasks.workunit.client.1.smithi188.stdout:1/102: rmdir d1/dc 39 2024-04-05T21:26:24.978 INFO:tasks.workunit.client.1.smithi188.stdout:1/103: mkdir d1/dc/d16/d21 0 2024-04-05T21:26:24.978 INFO:tasks.workunit.client.1.smithi188.stdout:3/40: sync 2024-04-05T21:26:24.978 INFO:tasks.workunit.client.1.smithi188.stdout:9/80: sync 2024-04-05T21:26:24.979 INFO:tasks.workunit.client.1.smithi188.stdout:8/67: sync 2024-04-05T21:26:24.981 INFO:tasks.workunit.client.1.smithi188.stdout:8/68: read d5/ff [4781,15878] 0 2024-04-05T21:26:24.981 INFO:tasks.workunit.client.1.smithi188.stdout:3/41: mknod d5/d7/c11 0 2024-04-05T21:26:24.983 INFO:tasks.workunit.client.1.smithi188.stdout:9/81: unlink l3 0 2024-04-05T21:26:24.985 INFO:tasks.workunit.client.1.smithi188.stdout:3/42: creat d5/f12 x:0 0 0 2024-04-05T21:26:24.986 INFO:tasks.workunit.client.1.smithi188.stdout:9/82: symlink dd/df/l1c 0 2024-04-05T21:26:24.987 INFO:tasks.workunit.client.1.smithi188.stdout:3/43: write d5/fb [869031,91695] 0 2024-04-05T21:26:24.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:24 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:26:24.991 INFO:tasks.workunit.client.1.smithi188.stdout:3/44: fdatasync d5/f12 0 2024-04-05T21:26:24.991 INFO:tasks.workunit.client.1.smithi188.stdout:1/104: dwrite d1/d4/f1d [0,4194304] 0 2024-04-05T21:26:24.998 INFO:tasks.workunit.client.1.smithi188.stdout:1/105: mknod d1/dc/c22 0 2024-04-05T21:26:25.000 INFO:tasks.workunit.client.1.smithi188.stdout:9/83: dwrite fb [0,4194304] 0 2024-04-05T21:26:25.005 INFO:tasks.workunit.client.1.smithi188.stdout:1/106: creat d1/dc/f23 x:0 0 0 2024-04-05T21:26:25.006 INFO:tasks.workunit.client.1.smithi188.stdout:3/45: dwrite f4 [0,4194304] 0 2024-04-05T21:26:25.009 INFO:tasks.workunit.client.1.smithi188.stdout:1/107: dread d1/dc/f11 [0,4194304] 0 2024-04-05T21:26:25.010 INFO:tasks.workunit.client.1.smithi188.stdout:3/46: mknod d5/dc/c13 0 2024-04-05T21:26:25.011 INFO:tasks.workunit.client.1.smithi188.stdout:3/47: dread - d5/f12 zero size 2024-04-05T21:26:25.013 INFO:tasks.workunit.client.1.smithi188.stdout:3/48: symlink d5/dc/l14 0 2024-04-05T21:26:25.015 INFO:tasks.workunit.client.1.smithi188.stdout:1/108: write d1/dc/f11 [778525,51282] 0 2024-04-05T21:26:25.022 INFO:tasks.workunit.client.1.smithi188.stdout:1/109: dread d1/dc/f19 [0,4194304] 0 2024-04-05T21:26:25.023 INFO:tasks.workunit.client.1.smithi188.stdout:1/110: readlink d1/dc/d16/l1c 0 2024-04-05T21:26:25.023 INFO:tasks.workunit.client.1.smithi188.stdout:3/49: dwrite f4 [4194304,4194304] 0 2024-04-05T21:26:25.028 INFO:tasks.workunit.client.1.smithi188.stdout:3/50: creat d5/df/f15 x:0 0 0 2024-04-05T21:26:25.029 INFO:tasks.workunit.client.1.smithi188.stdout:1/111: creat d1/f24 x:0 0 0 2024-04-05T21:26:25.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:24 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:26:25.038 INFO:tasks.workunit.client.1.smithi188.stdout:3/51: dwrite d5/fb [0,4194304] 0 2024-04-05T21:26:25.038 INFO:tasks.workunit.client.1.smithi188.stdout:1/112: dwrite d1/f14 [0,4194304] 0 2024-04-05T21:26:25.040 INFO:tasks.workunit.client.1.smithi188.stdout:1/113: write d1/f10 [743102,120445] 0 2024-04-05T21:26:25.044 INFO:tasks.workunit.client.1.smithi188.stdout:1/114: dread - d1/dc/f1f zero size 2024-04-05T21:26:25.045 INFO:tasks.workunit.client.1.smithi188.stdout:3/52: link d5/d7/c10 d5/dc/c16 0 2024-04-05T21:26:25.046 INFO:tasks.workunit.client.1.smithi188.stdout:3/53: chown d5/dc/ce 2036 1 2024-04-05T21:26:25.047 INFO:tasks.workunit.client.1.smithi188.stdout:1/115: symlink d1/l25 0 2024-04-05T21:26:25.050 INFO:tasks.workunit.client.1.smithi188.stdout:1/116: creat d1/dc/d16/d21/f26 x:0 0 0 2024-04-05T21:26:25.056 INFO:tasks.workunit.client.1.smithi188.stdout:1/117: dread d1/dc/f15 [0,4194304] 0 2024-04-05T21:26:25.059 INFO:tasks.workunit.client.1.smithi188.stdout:3/54: dwrite d5/f6 [0,4194304] 0 2024-04-05T21:26:25.060 INFO:tasks.workunit.client.1.smithi188.stdout:3/55: readlink d5/dc/l14 0 2024-04-05T21:26:25.064 INFO:tasks.workunit.client.1.smithi188.stdout:1/118: getdents d1/d4/d13 0 2024-04-05T21:26:25.064 INFO:tasks.workunit.client.1.smithi188.stdout:3/56: mknod d5/c17 0 2024-04-05T21:26:25.065 INFO:tasks.workunit.client.1.smithi188.stdout:1/119: chown d1/d4/d13/l8 325888 1 2024-04-05T21:26:25.074 INFO:tasks.workunit.client.1.smithi188.stdout:1/120: dwrite d1/dc/f23 [0,4194304] 0 2024-04-05T21:26:25.078 INFO:tasks.workunit.client.1.smithi188.stdout:3/57: truncate f3 3079984 0 2024-04-05T21:26:25.082 INFO:tasks.workunit.client.1.smithi188.stdout:3/58: rename d5/c17 to d5/df/c18 0 2024-04-05T21:26:25.082 INFO:tasks.workunit.client.1.smithi188.stdout:1/121: dwrite d1/d4/f1d [0,4194304] 0 2024-04-05T21:26:25.087 INFO:tasks.workunit.client.1.smithi188.stdout:1/122: rmdir d1 39 2024-04-05T21:26:25.088 INFO:tasks.workunit.client.1.smithi188.stdout:3/59: dread f4 [0,4194304] 0 2024-04-05T21:26:25.089 INFO:tasks.workunit.client.1.smithi188.stdout:3/60: stat f4 0 2024-04-05T21:26:25.090 INFO:tasks.workunit.client.1.smithi188.stdout:1/123: rmdir d1/dc/d16/d21 39 2024-04-05T21:26:25.093 INFO:tasks.workunit.client.1.smithi188.stdout:3/61: creat d5/dc/f19 x:0 0 0 2024-04-05T21:26:25.096 INFO:tasks.workunit.client.1.smithi188.stdout:1/124: mkdir d1/dc/d16/d21/d27 0 2024-04-05T21:26:25.102 INFO:tasks.workunit.client.1.smithi188.stdout:3/62: truncate d5/f6 1137753 0 2024-04-05T21:26:25.103 INFO:tasks.workunit.client.1.smithi188.stdout:1/125: truncate d1/f14 132665 0 2024-04-05T21:26:25.106 INFO:tasks.workunit.client.1.smithi188.stdout:1/126: creat d1/dc/f28 x:0 0 0 2024-04-05T21:26:25.109 INFO:tasks.workunit.client.1.smithi188.stdout:3/63: fsync d5/f6 0 2024-04-05T21:26:25.111 INFO:tasks.workunit.client.1.smithi188.stdout:3/64: dread - d5/df/f15 zero size 2024-04-05T21:26:25.114 INFO:tasks.workunit.client.1.smithi188.stdout:1/127: dwrite d1/fb [0,4194304] 0 2024-04-05T21:26:25.114 INFO:tasks.workunit.client.1.smithi188.stdout:3/65: rmdir d5/dc 39 2024-04-05T21:26:25.115 INFO:tasks.workunit.client.1.smithi188.stdout:3/66: truncate d5/f12 876242 0 2024-04-05T21:26:25.116 INFO:tasks.workunit.client.1.smithi188.stdout:1/128: fdatasync d1/dc/f19 0 2024-04-05T21:26:25.116 INFO:tasks.workunit.client.1.smithi188.stdout:1/129: fdatasync d1/f10 0 2024-04-05T21:26:25.119 INFO:tasks.workunit.client.1.smithi188.stdout:3/67: rmdir d5 39 2024-04-05T21:26:25.121 INFO:tasks.workunit.client.1.smithi188.stdout:1/130: rmdir d1/d7 39 2024-04-05T21:26:25.123 INFO:tasks.workunit.client.1.smithi188.stdout:1/131: rmdir d1/dc 39 2024-04-05T21:26:25.124 INFO:tasks.workunit.client.1.smithi188.stdout:3/68: creat d5/d7/f1a x:0 0 0 2024-04-05T21:26:25.125 INFO:tasks.workunit.client.1.smithi188.stdout:3/69: rename d5 to d5/d7/d1b 22 2024-04-05T21:26:25.125 INFO:tasks.workunit.client.1.smithi188.stdout:3/70: readlink l2 0 2024-04-05T21:26:25.127 INFO:tasks.workunit.client.1.smithi188.stdout:1/132: link d1/fb d1/dc/d16/d21/d27/f29 0 2024-04-05T21:26:25.129 INFO:tasks.workunit.client.1.smithi188.stdout:3/71: getdents d5/df 0 2024-04-05T21:26:25.130 INFO:tasks.workunit.client.1.smithi188.stdout:3/72: truncate d5/d7/f1a 517150 0 2024-04-05T21:26:25.131 INFO:tasks.workunit.client.1.smithi188.stdout:1/133: getdents d1/d7 0 2024-04-05T21:26:25.131 INFO:tasks.workunit.client.1.smithi188.stdout:3/73: read d5/fb [240357,77389] 0 2024-04-05T21:26:25.135 INFO:tasks.workunit.client.1.smithi188.stdout:3/74: symlink d5/l1c 0 2024-04-05T21:26:25.136 INFO:tasks.workunit.client.1.smithi188.stdout:1/134: symlink d1/dc/d16/l2a 0 2024-04-05T21:26:25.138 INFO:tasks.workunit.client.1.smithi188.stdout:1/135: mkdir d1/d2b 0 2024-04-05T21:26:25.140 INFO:tasks.workunit.client.1.smithi188.stdout:3/75: rename d5/dc/ce to d5/c1d 0 2024-04-05T21:26:25.140 INFO:tasks.workunit.client.1.smithi188.stdout:1/136: mknod d1/dc/d16/d21/c2c 0 2024-04-05T21:26:25.144 INFO:tasks.workunit.client.1.smithi188.stdout:3/76: dread f3 [0,4194304] 0 2024-04-05T21:26:25.146 INFO:tasks.workunit.client.1.smithi188.stdout:1/137: dwrite d1/d4/f1d [0,4194304] 0 2024-04-05T21:26:25.147 INFO:tasks.workunit.client.1.smithi188.stdout:3/77: creat d5/f1e x:0 0 0 2024-04-05T21:26:25.149 INFO:tasks.workunit.client.1.smithi188.stdout:1/138: mkdir d1/dc/d16/d2d 0 2024-04-05T21:26:25.149 INFO:tasks.workunit.client.1.smithi188.stdout:3/78: read d5/d7/f1a [374356,122594] 0 2024-04-05T21:26:25.150 INFO:tasks.workunit.client.1.smithi188.stdout:1/139: dread - d1/dc/f1f zero size 2024-04-05T21:26:25.152 INFO:tasks.workunit.client.1.smithi188.stdout:0/64: sync 2024-04-05T21:26:25.153 INFO:tasks.workunit.client.1.smithi188.stdout:3/79: rename f3 to d5/dc/f1f 0 2024-04-05T21:26:25.155 INFO:tasks.workunit.client.1.smithi188.stdout:1/140: mknod d1/dc/d16/c2e 0 2024-04-05T21:26:25.157 INFO:tasks.workunit.client.1.smithi188.stdout:0/65: mknod d4/d6/d13/c17 0 2024-04-05T21:26:25.157 INFO:tasks.workunit.client.1.smithi188.stdout:3/80: dread d5/fa [0,4194304] 0 2024-04-05T21:26:25.158 INFO:tasks.workunit.client.1.smithi188.stdout:0/66: stat d4/d8/db 0 2024-04-05T21:26:25.158 INFO:tasks.workunit.client.1.smithi188.stdout:3/81: chown d5/f1e 5 1 2024-04-05T21:26:25.159 INFO:tasks.workunit.client.1.smithi188.stdout:3/82: dread - d5/dc/f19 zero size 2024-04-05T21:26:25.162 INFO:tasks.workunit.client.1.smithi188.stdout:3/83: write f4 [1042353,101782] 0 2024-04-05T21:26:25.167 INFO:tasks.workunit.client.1.smithi188.stdout:1/141: dwrite d1/d4/f12 [0,4194304] 0 2024-04-05T21:26:25.169 INFO:tasks.workunit.client.1.smithi188.stdout:0/67: dwrite d4/fe [0,4194304] 0 2024-04-05T21:26:25.169 INFO:tasks.workunit.client.1.smithi188.stdout:0/68: readlink - no filename 2024-04-05T21:26:25.171 INFO:tasks.workunit.client.1.smithi188.stdout:1/142: mknod d1/dc/c2f 0 2024-04-05T21:26:25.172 INFO:tasks.workunit.client.1.smithi188.stdout:3/84: dread d5/fb [0,4194304] 0 2024-04-05T21:26:25.172 INFO:tasks.workunit.client.1.smithi188.stdout:1/143: stat d1/dc/c2f 0 2024-04-05T21:26:25.173 INFO:tasks.workunit.client.1.smithi188.stdout:0/69: mknod d4/d8/db/c18 0 2024-04-05T21:26:25.174 INFO:tasks.workunit.client.1.smithi188.stdout:3/85: truncate d5/f1e 658538 0 2024-04-05T21:26:25.174 INFO:tasks.workunit.client.1.smithi188.stdout:0/70: chown d4/d8/fa 6084 1 2024-04-05T21:26:25.175 INFO:tasks.workunit.client.1.smithi188.stdout:0/71: dread - d4/d6/f15 zero size 2024-04-05T21:26:25.175 INFO:tasks.workunit.client.1.smithi188.stdout:3/86: read d5/f1e [397898,128046] 0 2024-04-05T21:26:25.176 INFO:tasks.workunit.client.1.smithi188.stdout:0/72: stat d4/d8/d10 0 2024-04-05T21:26:25.178 INFO:tasks.workunit.client.1.smithi188.stdout:0/73: creat d4/f19 x:0 0 0 2024-04-05T21:26:25.180 INFO:tasks.workunit.client.1.smithi188.stdout:0/74: write d4/d8/db/f11 [1056125,92349] 0 2024-04-05T21:26:25.192 INFO:tasks.workunit.client.1.smithi188.stdout:0/75: write d4/d6/d13/f14 [3083792,25614] 0 2024-04-05T21:26:25.196 INFO:tasks.workunit.client.1.smithi188.stdout:3/87: dwrite d5/dc/f1f [0,4194304] 0 2024-04-05T21:26:25.196 INFO:tasks.workunit.client.1.smithi188.stdout:3/88: stat d5/dc/l14 0 2024-04-05T21:26:25.197 INFO:tasks.workunit.client.1.smithi188.stdout:3/89: chown d5/f1e 58380612 1 2024-04-05T21:26:25.201 INFO:tasks.workunit.client.1.smithi188.stdout:3/90: dread d5/fa [0,4194304] 0 2024-04-05T21:26:25.203 INFO:tasks.workunit.client.1.smithi188.stdout:3/91: creat d5/d7/f20 x:0 0 0 2024-04-05T21:26:25.206 INFO:tasks.workunit.client.1.smithi188.stdout:3/92: dread d5/fa [0,4194304] 0 2024-04-05T21:26:25.210 INFO:tasks.workunit.client.1.smithi188.stdout:3/93: dread d5/dc/f1f [0,4194304] 0 2024-04-05T21:26:25.212 INFO:tasks.workunit.client.1.smithi188.stdout:3/94: creat d5/f21 x:0 0 0 2024-04-05T21:26:25.213 INFO:tasks.workunit.client.1.smithi188.stdout:3/95: creat d5/f22 x:0 0 0 2024-04-05T21:26:25.214 INFO:tasks.workunit.client.1.smithi188.stdout:3/96: write d5/d7/f20 [10308,48566] 0 2024-04-05T21:26:25.216 INFO:tasks.workunit.client.1.smithi188.stdout:3/97: truncate d5/f6 1228730 0 2024-04-05T21:26:25.291 INFO:tasks.workunit.client.1.smithi188.stdout:3/98: dwrite d5/dc/f19 [0,4194304] 0 2024-04-05T21:26:25.294 INFO:tasks.workunit.client.1.smithi188.stdout:3/99: rename d5/dc/l14 to d5/dc/l23 0 2024-04-05T21:26:25.298 INFO:tasks.workunit.client.1.smithi188.stdout:3/100: truncate d5/f6 1961489 0 2024-04-05T21:26:25.298 INFO:tasks.workunit.client.1.smithi188.stdout:3/101: rename d5 to d5/df/d24 22 2024-04-05T21:26:25.299 INFO:tasks.workunit.client.1.smithi188.stdout:3/102: chown d5/dc/c16 0 1 2024-04-05T21:26:25.300 INFO:tasks.workunit.client.1.smithi188.stdout:3/103: readlink d5/l1c 0 2024-04-05T21:26:25.301 INFO:tasks.workunit.client.1.smithi188.stdout:3/104: creat d5/d7/f25 x:0 0 0 2024-04-05T21:26:25.302 INFO:tasks.workunit.client.1.smithi188.stdout:3/105: symlink d5/d7/l26 0 2024-04-05T21:26:25.304 INFO:tasks.workunit.client.1.smithi188.stdout:3/106: symlink d5/dc/l27 0 2024-04-05T21:26:25.305 INFO:tasks.workunit.client.1.smithi188.stdout:3/107: rmdir d5/df 39 2024-04-05T21:26:25.307 INFO:tasks.workunit.client.1.smithi188.stdout:3/108: rename d5/d7/c11 to d5/df/c28 0 2024-04-05T21:26:25.308 INFO:tasks.workunit.client.1.smithi188.stdout:3/109: dread - d5/d7/f25 zero size 2024-04-05T21:26:25.309 INFO:tasks.workunit.client.1.smithi188.stdout:3/110: fdatasync d5/fa 0 2024-04-05T21:26:25.309 INFO:tasks.workunit.client.1.smithi188.stdout:3/111: fsync d5/f21 0 2024-04-05T21:26:25.310 INFO:tasks.workunit.client.1.smithi188.stdout:3/112: chown d5/d7/f1a 216 1 2024-04-05T21:26:25.310 INFO:tasks.workunit.client.1.smithi188.stdout:3/113: stat d5/df/c18 0 2024-04-05T21:26:25.314 INFO:tasks.workunit.client.1.smithi188.stdout:3/114: fsync d5/f6 0 2024-04-05T21:26:25.316 INFO:tasks.workunit.client.1.smithi188.stdout:3/115: mknod d5/dc/c29 0 2024-04-05T21:26:25.320 INFO:tasks.workunit.client.1.smithi188.stdout:3/116: dread d5/dc/f1f [0,4194304] 0 2024-04-05T21:26:25.322 INFO:tasks.workunit.client.1.smithi188.stdout:3/117: unlink d5/f22 0 2024-04-05T21:26:25.322 INFO:tasks.workunit.client.1.smithi188.stdout:3/118: stat d5/f1e 0 2024-04-05T21:26:25.324 INFO:tasks.workunit.client.1.smithi188.stdout:3/119: creat d5/dc/f2a x:0 0 0 2024-04-05T21:26:25.326 INFO:tasks.workunit.client.1.smithi188.stdout:3/120: unlink d5/dc/l27 0 2024-04-05T21:26:25.327 INFO:tasks.workunit.client.1.smithi188.stdout:3/121: fdatasync d5/d7/f1a 0 2024-04-05T21:26:25.329 INFO:tasks.workunit.client.1.smithi188.stdout:3/122: mkdir d5/d2b 0 2024-04-05T21:26:25.330 INFO:tasks.workunit.client.1.smithi188.stdout:3/123: stat d5/f12 0 2024-04-05T21:26:25.330 INFO:tasks.workunit.client.1.smithi188.stdout:3/124: write d5/d7/f25 [140294,49006] 0 2024-04-05T21:26:25.333 INFO:tasks.workunit.client.1.smithi188.stdout:3/125: mknod d5/c2c 0 2024-04-05T21:26:25.333 INFO:tasks.workunit.client.1.smithi188.stdout:3/126: chown d5/d7/c9 29640 1 2024-04-05T21:26:25.334 INFO:tasks.workunit.client.1.smithi188.stdout:3/127: chown d5/d7/f20 38728428 1 2024-04-05T21:26:25.335 INFO:tasks.workunit.client.1.smithi188.stdout:3/128: creat d5/dc/f2d x:0 0 0 2024-04-05T21:26:25.336 INFO:tasks.workunit.client.1.smithi188.stdout:3/129: fsync d5/fb 0 2024-04-05T21:26:25.339 INFO:tasks.workunit.client.1.smithi188.stdout:3/130: link d5/f12 d5/d2b/f2e 0 2024-04-05T21:26:25.340 INFO:tasks.workunit.client.1.smithi188.stdout:3/131: read d5/f6 [1373734,1695] 0 2024-04-05T21:26:25.348 INFO:tasks.workunit.client.1.smithi188.stdout:6/56: sync 2024-04-05T21:26:25.348 INFO:tasks.workunit.client.1.smithi188.stdout:5/82: sync 2024-04-05T21:26:25.350 INFO:tasks.workunit.client.1.smithi188.stdout:6/57: rmdir d6 39 2024-04-05T21:26:25.352 INFO:tasks.workunit.client.1.smithi188.stdout:5/83: creat d6/d15/d1b/f1e x:0 0 0 2024-04-05T21:26:25.354 INFO:tasks.workunit.client.1.smithi188.stdout:6/58: creat d6/d7/f14 x:0 0 0 2024-04-05T21:26:25.355 INFO:tasks.workunit.client.1.smithi188.stdout:6/59: creat d6/d7/f15 x:0 0 0 2024-04-05T21:26:25.356 INFO:tasks.workunit.client.1.smithi188.stdout:6/60: chown d6/d13 14 1 2024-04-05T21:26:25.356 INFO:tasks.workunit.client.1.smithi188.stdout:6/61: dread - d6/d7/f15 zero size 2024-04-05T21:26:25.357 INFO:tasks.workunit.client.1.smithi188.stdout:6/62: dread - d6/d7/f15 zero size 2024-04-05T21:26:25.357 INFO:tasks.workunit.client.1.smithi188.stdout:5/84: link d6/fe d6/d17/f1f 0 2024-04-05T21:26:25.358 INFO:tasks.workunit.client.1.smithi188.stdout:6/63: chown l1 8 1 2024-04-05T21:26:25.359 INFO:tasks.workunit.client.1.smithi188.stdout:6/64: write d6/d7/f15 [304944,72303] 0 2024-04-05T21:26:25.360 INFO:tasks.workunit.client.1.smithi188.stdout:5/85: creat d6/d17/f20 x:0 0 0 2024-04-05T21:26:25.360 INFO:tasks.workunit.client.1.smithi188.stdout:6/65: creat d6/d7/f16 x:0 0 0 2024-04-05T21:26:25.361 INFO:tasks.workunit.client.1.smithi188.stdout:5/86: write d6/f11 [25226,99179] 0 2024-04-05T21:26:25.363 INFO:tasks.workunit.client.1.smithi188.stdout:6/66: creat d6/d13/f17 x:0 0 0 2024-04-05T21:26:25.364 INFO:tasks.workunit.client.1.smithi188.stdout:5/87: mkdir d6/d15/d21 0 2024-04-05T21:26:25.365 INFO:tasks.workunit.client.1.smithi188.stdout:6/67: mkdir d6/d13/d18 0 2024-04-05T21:26:25.365 INFO:tasks.workunit.client.1.smithi188.stdout:6/68: chown d6/d13/d18 387761715 1 2024-04-05T21:26:25.367 INFO:tasks.workunit.client.1.smithi188.stdout:5/88: creat d6/f22 x:0 0 0 2024-04-05T21:26:25.368 INFO:tasks.workunit.client.1.smithi188.stdout:5/89: dread - d6/d17/f20 zero size 2024-04-05T21:26:25.371 INFO:tasks.workunit.client.1.smithi188.stdout:5/90: rename d6/d15/d21 to d6/d17/d23 0 2024-04-05T21:26:25.372 INFO:tasks.workunit.client.1.smithi188.stdout:5/91: readlink d6/lc 0 2024-04-05T21:26:25.373 INFO:tasks.workunit.client.1.smithi188.stdout:6/69: dwrite d6/d7/f15 [0,4194304] 0 2024-04-05T21:26:25.373 INFO:tasks.workunit.client.1.smithi188.stdout:6/70: chown d6/f11 4873926 1 2024-04-05T21:26:25.379 INFO:tasks.workunit.client.1.smithi188.stdout:5/92: truncate d6/d17/f1f 1350869 0 2024-04-05T21:26:25.381 INFO:tasks.workunit.client.1.smithi188.stdout:5/93: write d6/d15/d1b/f1e [702606,127375] 0 2024-04-05T21:26:25.382 INFO:tasks.workunit.client.1.smithi188.stdout:6/71: getdents d6/d7 0 2024-04-05T21:26:25.383 INFO:tasks.workunit.client.1.smithi188.stdout:5/94: mkdir d6/d17/d24 0 2024-04-05T21:26:25.383 INFO:tasks.workunit.client.1.smithi188.stdout:6/72: write d6/d7/f14 [215608,68669] 0 2024-04-05T21:26:25.386 INFO:tasks.workunit.client.1.smithi188.stdout:5/95: symlink d6/d15/d1b/l25 0 2024-04-05T21:26:25.386 INFO:tasks.workunit.client.1.smithi188.stdout:5/96: fsync d6/f12 0 2024-04-05T21:26:25.388 INFO:tasks.workunit.client.1.smithi188.stdout:6/73: mkdir d6/d13/d18/d19 0 2024-04-05T21:26:25.390 INFO:tasks.workunit.client.1.smithi188.stdout:6/74: write d6/d7/fd [3897468,7289] 0 2024-04-05T21:26:25.392 INFO:tasks.workunit.client.1.smithi188.stdout:6/75: write d6/fa [3631699,43773] 0 2024-04-05T21:26:25.396 INFO:tasks.workunit.client.1.smithi188.stdout:5/97: dwrite d6/f22 [0,4194304] 0 2024-04-05T21:26:25.397 INFO:tasks.workunit.client.1.smithi188.stdout:6/76: mknod d6/d13/d18/c1a 0 2024-04-05T21:26:25.397 INFO:tasks.workunit.client.1.smithi188.stdout:5/98: write d6/f19 [34252,12006] 0 2024-04-05T21:26:25.407 INFO:tasks.workunit.client.1.smithi188.stdout:5/99: dwrite d6/f12 [0,4194304] 0 2024-04-05T21:26:25.409 INFO:tasks.workunit.client.1.smithi188.stdout:5/100: dread d6/f11 [0,4194304] 0 2024-04-05T21:26:25.411 INFO:tasks.workunit.client.1.smithi188.stdout:6/77: getdents d6/d13/d18 0 2024-04-05T21:26:25.415 INFO:tasks.workunit.client.1.smithi188.stdout:5/101: creat d6/d17/d24/f26 x:0 0 0 2024-04-05T21:26:25.415 INFO:tasks.workunit.client.1.smithi188.stdout:6/78: dread d6/d7/fd [0,4194304] 0 2024-04-05T21:26:25.416 INFO:tasks.workunit.client.1.smithi188.stdout:6/79: write d6/f11 [1025352,58116] 0 2024-04-05T21:26:25.418 INFO:tasks.workunit.client.1.smithi188.stdout:5/102: rmdir d6/d15/d1b 39 2024-04-05T21:26:25.419 INFO:tasks.workunit.client.1.smithi188.stdout:6/80: symlink d6/d13/d18/l1b 0 2024-04-05T21:26:25.419 INFO:tasks.workunit.client.1.smithi188.stdout:6/81: dread - d6/d7/fc zero size 2024-04-05T21:26:25.422 INFO:tasks.workunit.client.1.smithi188.stdout:5/103: creat d6/d15/d1b/f27 x:0 0 0 2024-04-05T21:26:25.422 INFO:tasks.workunit.client.1.smithi188.stdout:5/104: dread - d6/d17/d24/f26 zero size 2024-04-05T21:26:25.423 INFO:tasks.workunit.client.1.smithi188.stdout:5/105: readlink d6/l8 0 2024-04-05T21:26:25.423 INFO:tasks.workunit.client.1.smithi188.stdout:6/82: link d6/d13/d18/c1a d6/d13/c1c 0 2024-04-05T21:26:25.424 INFO:tasks.workunit.client.1.smithi188.stdout:5/106: write d6/d17/d24/f26 [869178,18251] 0 2024-04-05T21:26:25.425 INFO:tasks.workunit.client.1.smithi188.stdout:6/83: write d6/fb [3137255,98122] 0 2024-04-05T21:26:25.429 INFO:tasks.workunit.client.1.smithi188.stdout:5/107: symlink d6/l28 0 2024-04-05T21:26:25.429 INFO:tasks.workunit.client.1.smithi188.stdout:5/108: fsync f2 0 2024-04-05T21:26:25.431 INFO:tasks.workunit.client.1.smithi188.stdout:5/109: rmdir d6/d15 39 2024-04-05T21:26:25.432 INFO:tasks.workunit.client.1.smithi188.stdout:6/84: getdents d6/d13/d18 0 2024-04-05T21:26:25.433 INFO:tasks.workunit.client.1.smithi188.stdout:6/85: write d6/fb [3083807,77016] 0 2024-04-05T21:26:25.436 INFO:tasks.workunit.client.1.smithi188.stdout:5/110: write d6/fe [2072850,41066] 0 2024-04-05T21:26:25.438 INFO:tasks.workunit.client.1.smithi188.stdout:6/86: dread d6/d7/fd [0,4194304] 0 2024-04-05T21:26:25.440 INFO:tasks.workunit.client.1.smithi188.stdout:6/87: creat d6/d13/f1d x:0 0 0 2024-04-05T21:26:25.441 INFO:tasks.workunit.client.1.smithi188.stdout:6/88: creat d6/d7/f1e x:0 0 0 2024-04-05T21:26:25.441 INFO:tasks.workunit.client.1.smithi188.stdout:5/111: link d6/c10 d6/c29 0 2024-04-05T21:26:25.443 INFO:tasks.workunit.client.1.smithi188.stdout:6/89: mknod d6/c1f 0 2024-04-05T21:26:25.444 INFO:tasks.workunit.client.1.smithi188.stdout:6/90: write d6/d13/f1d [144867,89806] 0 2024-04-05T21:26:25.447 INFO:tasks.workunit.client.1.smithi188.stdout:6/91: dread d6/d7/fd [0,4194304] 0 2024-04-05T21:26:25.449 INFO:tasks.workunit.client.1.smithi188.stdout:6/92: symlink d6/d13/l20 0 2024-04-05T21:26:25.449 INFO:tasks.workunit.client.1.smithi188.stdout:6/93: write d6/fa [1083124,115159] 0 2024-04-05T21:26:25.450 INFO:tasks.workunit.client.1.smithi188.stdout:6/94: fdatasync d6/d7/f16 0 2024-04-05T21:26:25.453 INFO:tasks.workunit.client.1.smithi188.stdout:6/95: unlink d6/d13/c1c 0 2024-04-05T21:26:25.459 INFO:tasks.workunit.client.1.smithi188.stdout:6/96: dwrite d6/d13/f17 [0,4194304] 0 2024-04-05T21:26:25.462 INFO:tasks.workunit.client.1.smithi188.stdout:6/97: rename d6/lf to d6/d13/d18/l21 0 2024-04-05T21:26:25.464 INFO:tasks.workunit.client.1.smithi188.stdout:6/98: dread d6/d13/f1d [0,4194304] 0 2024-04-05T21:26:25.466 INFO:tasks.workunit.client.1.smithi188.stdout:6/99: rmdir d6/d13/d18 39 2024-04-05T21:26:25.467 INFO:tasks.workunit.client.1.smithi188.stdout:6/100: rmdir d6 39 2024-04-05T21:26:25.473 INFO:tasks.workunit.client.1.smithi188.stdout:2/125: sync 2024-04-05T21:26:25.475 INFO:tasks.workunit.client.1.smithi188.stdout:6/101: truncate d6/f11 601846 0 2024-04-05T21:26:25.477 INFO:tasks.workunit.client.1.smithi188.stdout:2/126: dread d5/db/dd/f1c [0,4194304] 0 2024-04-05T21:26:25.478 INFO:tasks.workunit.client.1.smithi188.stdout:6/102: mknod d6/d13/d18/d19/c22 0 2024-04-05T21:26:25.478 INFO:tasks.workunit.client.1.smithi188.stdout:2/127: fdatasync d5/db/dd/f15 0 2024-04-05T21:26:25.478 INFO:tasks.workunit.client.1.smithi188.stdout:6/103: readlink d6/l9 0 2024-04-05T21:26:25.478 INFO:tasks.workunit.client.1.smithi188.stdout:6/104: chown f5 3622 1 2024-04-05T21:26:25.485 INFO:tasks.workunit.client.1.smithi188.stdout:6/105: dread d6/d13/f17 [0,4194304] 0 2024-04-05T21:26:25.485 INFO:tasks.workunit.client.1.smithi188.stdout:2/128: dwrite d5/db/dd/f15 [0,4194304] 0 2024-04-05T21:26:25.487 INFO:tasks.workunit.client.1.smithi188.stdout:6/106: write d6/d7/f14 [1088439,125774] 0 2024-04-05T21:26:25.487 INFO:tasks.workunit.client.1.smithi188.stdout:2/129: rmdir d5/db 39 2024-04-05T21:26:25.489 INFO:tasks.workunit.client.1.smithi188.stdout:6/107: fsync d6/d7/f1e 0 2024-04-05T21:26:25.491 INFO:tasks.workunit.client.1.smithi188.stdout:2/130: creat d5/f22 x:0 0 0 2024-04-05T21:26:25.492 INFO:tasks.workunit.client.1.smithi188.stdout:6/108: stat l4 0 2024-04-05T21:26:25.494 INFO:tasks.workunit.client.1.smithi188.stdout:6/109: mknod d6/d7/c23 0 2024-04-05T21:26:25.495 INFO:tasks.workunit.client.1.smithi188.stdout:2/131: dread d5/fa [0,4194304] 0 2024-04-05T21:26:25.501 INFO:tasks.workunit.client.1.smithi188.stdout:2/132: link d5/l1f d5/db/l23 0 2024-04-05T21:26:25.501 INFO:tasks.workunit.client.1.smithi188.stdout:6/110: dwrite d6/d7/f16 [0,4194304] 0 2024-04-05T21:26:25.504 INFO:tasks.workunit.client.1.smithi188.stdout:2/133: write d5/f14 [298288,45396] 0 2024-04-05T21:26:25.507 INFO:tasks.workunit.client.1.smithi188.stdout:6/111: creat d6/f24 x:0 0 0 2024-04-05T21:26:25.508 INFO:tasks.workunit.client.1.smithi188.stdout:2/134: mkdir d5/db/dd/d24 0 2024-04-05T21:26:25.510 INFO:tasks.workunit.client.1.smithi188.stdout:6/112: rename c3 to d6/d13/d18/c25 0 2024-04-05T21:26:25.511 INFO:tasks.workunit.client.1.smithi188.stdout:6/113: read d6/fb [1470960,81198] 0 2024-04-05T21:26:25.512 INFO:tasks.workunit.client.1.smithi188.stdout:6/114: rename d6 to d6/d13/d26 22 2024-04-05T21:26:25.514 INFO:tasks.workunit.client.1.smithi188.stdout:6/115: creat d6/d13/d18/d19/f27 x:0 0 0 2024-04-05T21:26:25.516 INFO:tasks.workunit.client.1.smithi188.stdout:6/116: mkdir d6/d28 0 2024-04-05T21:26:25.518 INFO:tasks.workunit.client.1.smithi188.stdout:6/117: chown d6/d13/d18/c25 179 1 2024-04-05T21:26:25.519 INFO:tasks.workunit.client.1.smithi188.stdout:2/135: truncate d5/db/dd/f1c 1801663 0 2024-04-05T21:26:25.519 INFO:tasks.workunit.client.1.smithi188.stdout:6/118: dread - d6/d7/fc zero size 2024-04-05T21:26:25.519 INFO:tasks.workunit.client.1.smithi188.stdout:2/136: chown d5/fa 1021 1 2024-04-05T21:26:25.525 INFO:tasks.workunit.client.1.smithi188.stdout:6/119: dwrite d6/d7/f1e [0,4194304] 0 2024-04-05T21:26:25.533 INFO:tasks.workunit.client.1.smithi188.stdout:6/120: creat d6/d28/f29 x:0 0 0 2024-04-05T21:26:25.535 INFO:tasks.workunit.client.1.smithi188.stdout:6/121: truncate d6/fb 4024280 0 2024-04-05T21:26:25.535 INFO:tasks.workunit.client.1.smithi188.stdout:2/137: link d5/db/dd/f1c d5/db/f25 0 2024-04-05T21:26:25.541 INFO:tasks.workunit.client.1.smithi188.stdout:2/138: link d5/db/l23 d5/db/dd/l26 0 2024-04-05T21:26:25.542 INFO:tasks.workunit.client.1.smithi188.stdout:6/122: dwrite d6/d7/f14 [0,4194304] 0 2024-04-05T21:26:25.542 INFO:tasks.workunit.client.1.smithi188.stdout:2/139: readlink d5/db/l1b 0 2024-04-05T21:26:25.546 INFO:tasks.workunit.client.1.smithi188.stdout:6/123: symlink d6/d13/d18/d19/l2a 0 2024-04-05T21:26:25.547 INFO:tasks.workunit.client.1.smithi188.stdout:6/124: stat d6/d7/fc 0 2024-04-05T21:26:25.551 INFO:tasks.workunit.client.1.smithi188.stdout:2/140: dwrite d5/db/f16 [0,4194304] 0 2024-04-05T21:26:25.552 INFO:tasks.workunit.client.1.smithi188.stdout:2/141: readlink d5/db/l1b 0 2024-04-05T21:26:25.557 INFO:tasks.workunit.client.1.smithi188.stdout:6/125: write d6/d7/f15 [4699592,113325] 0 2024-04-05T21:26:25.562 INFO:tasks.workunit.client.1.smithi188.stdout:2/142: dwrite d5/f22 [0,4194304] 0 2024-04-05T21:26:25.563 INFO:tasks.workunit.client.1.smithi188.stdout:6/126: link d6/f24 d6/f2b 0 2024-04-05T21:26:25.563 INFO:tasks.workunit.client.1.smithi188.stdout:2/143: chown d5/l21 25450074 1 2024-04-05T21:26:25.567 INFO:tasks.workunit.client.1.smithi188.stdout:2/144: readlink d5/db/l23 0 2024-04-05T21:26:25.567 INFO:tasks.workunit.client.1.smithi188.stdout:2/145: stat d5 0 2024-04-05T21:26:25.569 INFO:tasks.workunit.client.1.smithi188.stdout:6/127: rmdir d6 39 2024-04-05T21:26:25.571 INFO:tasks.workunit.client.1.smithi188.stdout:2/146: dread d5/f22 [0,4194304] 0 2024-04-05T21:26:25.572 INFO:tasks.workunit.client.1.smithi188.stdout:6/128: mkdir d6/d13/d18/d19/d2c 0 2024-04-05T21:26:25.574 INFO:tasks.workunit.client.1.smithi188.stdout:2/147: mknod d5/db/dd/c27 0 2024-04-05T21:26:25.578 INFO:tasks.workunit.client.1.smithi188.stdout:2/148: rename d5/db/f25 to d5/db/dd/d24/f28 0 2024-04-05T21:26:25.579 INFO:tasks.workunit.client.1.smithi188.stdout:2/149: chown d5/l21 29001 1 2024-04-05T21:26:25.579 INFO:tasks.workunit.client.1.smithi188.stdout:6/129: truncate d6/d7/f15 4490695 0 2024-04-05T21:26:25.581 INFO:tasks.workunit.client.1.smithi188.stdout:2/150: dread d5/fc [4194304,4194304] 0 2024-04-05T21:26:25.583 INFO:tasks.workunit.client.1.smithi188.stdout:6/130: dread f5 [0,4194304] 0 2024-04-05T21:26:25.587 INFO:tasks.workunit.client.1.smithi188.stdout:2/151: dwrite d5/fc [0,4194304] 0 2024-04-05T21:26:25.590 INFO:tasks.workunit.client.1.smithi188.stdout:2/152: write d5/fa [2738541,54818] 0 2024-04-05T21:26:25.596 INFO:tasks.workunit.client.1.smithi188.stdout:6/131: dwrite d6/f2b [0,4194304] 0 2024-04-05T21:26:25.599 INFO:tasks.workunit.client.1.smithi188.stdout:6/132: mknod d6/d7/c2d 0 2024-04-05T21:26:25.599 INFO:tasks.workunit.client.1.smithi188.stdout:6/133: fdatasync d6/d28/f29 0 2024-04-05T21:26:25.602 INFO:tasks.workunit.client.1.smithi188.stdout:6/134: rename d6/d28/f29 to d6/f2e 0 2024-04-05T21:26:25.609 INFO:tasks.workunit.client.1.smithi188.stdout:6/135: dwrite d6/f2b [0,4194304] 0 2024-04-05T21:26:25.615 INFO:tasks.workunit.client.1.smithi188.stdout:6/136: truncate d6/f8 1806691 0 2024-04-05T21:26:25.618 INFO:tasks.workunit.client.1.smithi188.stdout:6/137: creat d6/f2f x:0 0 0 2024-04-05T21:26:25.618 INFO:tasks.workunit.client.1.smithi188.stdout:6/138: dread - d6/f2f zero size 2024-04-05T21:26:25.625 INFO:tasks.workunit.client.1.smithi188.stdout:6/139: dwrite f5 [0,4194304] 0 2024-04-05T21:26:25.628 INFO:tasks.workunit.client.1.smithi188.stdout:6/140: creat d6/d13/d18/d19/f30 x:0 0 0 2024-04-05T21:26:25.632 INFO:tasks.workunit.client.1.smithi188.stdout:6/141: creat d6/d28/f31 x:0 0 0 2024-04-05T21:26:25.641 INFO:tasks.workunit.client.1.smithi188.stdout:6/142: rename d6/f8 to d6/d13/d18/d19/d2c/f32 0 2024-04-05T21:26:25.645 INFO:tasks.workunit.client.1.smithi188.stdout:6/143: write d6/d7/fd [2576273,10996] 0 2024-04-05T21:26:25.647 INFO:tasks.workunit.client.1.smithi188.stdout:6/144: mknod d6/d13/d18/d19/d2c/c33 0 2024-04-05T21:26:25.648 INFO:tasks.workunit.client.1.smithi188.stdout:6/145: write d6/f2b [1193322,97453] 0 2024-04-05T21:26:25.649 INFO:tasks.workunit.client.1.smithi188.stdout:6/146: chown d6/d13/d18 3599 1 2024-04-05T21:26:25.650 INFO:tasks.workunit.client.1.smithi188.stdout:6/147: write d6/d7/fd [1748902,125830] 0 2024-04-05T21:26:25.654 INFO:tasks.workunit.client.1.smithi188.stdout:6/148: dread d6/d7/fd [0,4194304] 0 2024-04-05T21:26:25.762 INFO:tasks.workunit.client.1.smithi188.stdout:4/97: sync 2024-04-05T21:26:25.762 INFO:tasks.workunit.client.1.smithi188.stdout:8/69: sync 2024-04-05T21:26:25.762 INFO:tasks.workunit.client.1.smithi188.stdout:7/225: sync 2024-04-05T21:26:25.763 INFO:tasks.workunit.client.1.smithi188.stdout:8/70: write d5/f9 [464628,50462] 0 2024-04-05T21:26:25.764 INFO:tasks.workunit.client.1.smithi188.stdout:4/98: mkdir d0/d2/d2a 0 2024-04-05T21:26:25.765 INFO:tasks.workunit.client.1.smithi188.stdout:8/71: readlink d5/dc/ld 0 2024-04-05T21:26:25.766 INFO:tasks.workunit.client.1.smithi188.stdout:4/99: chown d0/d14/c25 1 1 2024-04-05T21:26:25.766 INFO:tasks.workunit.client.1.smithi188.stdout:8/72: fsync d5/dc/f11 0 2024-04-05T21:26:25.767 INFO:tasks.workunit.client.1.smithi188.stdout:7/226: rename d3/df/f27 to d3/d4/db/d2e/d37/f47 0 2024-04-05T21:26:25.771 INFO:tasks.workunit.client.1.smithi188.stdout:4/100: rename d0/d2/ld to d0/d2/d2a/l2b 0 2024-04-05T21:26:25.772 INFO:tasks.workunit.client.1.smithi188.stdout:7/227: rename d3/d2c/l3a to d3/d2c/l48 0 2024-04-05T21:26:25.775 INFO:tasks.workunit.client.1.smithi188.stdout:8/73: creat d5/f15 x:0 0 0 2024-04-05T21:26:25.776 INFO:tasks.workunit.client.1.smithi188.stdout:7/228: mkdir d3/d4/db/d2e/d49 0 2024-04-05T21:26:25.778 INFO:tasks.workunit.client.1.smithi188.stdout:4/101: dwrite d0/f5 [0,4194304] 0 2024-04-05T21:26:25.780 INFO:tasks.workunit.client.1.smithi188.stdout:7/229: dread d3/d4/f1b [0,4194304] 0 2024-04-05T21:26:25.781 INFO:tasks.workunit.client.1.smithi188.stdout:8/74: creat d5/f16 x:0 0 0 2024-04-05T21:26:25.787 INFO:tasks.workunit.client.1.smithi188.stdout:7/230: rename d3/df/f30 to d3/d4/db/f4a 0 2024-04-05T21:26:25.787 INFO:tasks.workunit.client.1.smithi188.stdout:4/102: dwrite d0/d14/d15/f1a [0,4194304] 0 2024-04-05T21:26:25.789 INFO:tasks.workunit.client.1.smithi188.stdout:8/75: rmdir d5 39 2024-04-05T21:26:25.790 INFO:tasks.workunit.client.1.smithi188.stdout:4/103: symlink d0/d14/d15/l2c 0 2024-04-05T21:26:25.792 INFO:tasks.workunit.client.1.smithi188.stdout:4/104: dread - d0/d14/d15/d1b/f1e zero size 2024-04-05T21:26:25.792 INFO:tasks.workunit.client.1.smithi188.stdout:8/76: write d5/f8 [2258477,113035] 0 2024-04-05T21:26:25.793 INFO:tasks.workunit.client.1.smithi188.stdout:7/231: getdents d3/d2c 0 2024-04-05T21:26:25.793 INFO:tasks.workunit.client.1.smithi188.stdout:4/105: chown d0/d14 718637 1 2024-04-05T21:26:25.794 INFO:tasks.workunit.client.1.smithi188.stdout:8/77: write d5/f8 [3320769,20052] 0 2024-04-05T21:26:25.795 INFO:tasks.workunit.client.1.smithi188.stdout:4/106: write d0/d14/d15/f29 [895185,67738] 0 2024-04-05T21:26:25.796 INFO:tasks.workunit.client.1.smithi188.stdout:7/232: readlink d3/d4/l22 0 2024-04-05T21:26:25.797 INFO:tasks.workunit.client.1.smithi188.stdout:7/233: read - d3/d4/d1d/f2b zero size 2024-04-05T21:26:25.798 INFO:tasks.workunit.client.1.smithi188.stdout:8/78: creat d5/dc/f17 x:0 0 0 2024-04-05T21:26:25.800 INFO:tasks.workunit.client.1.smithi188.stdout:7/234: symlink d3/d4/db/d2e/d37/l4b 0 2024-04-05T21:26:25.800 INFO:tasks.workunit.client.1.smithi188.stdout:4/107: mknod d0/c2d 0 2024-04-05T21:26:25.801 INFO:tasks.workunit.client.1.smithi188.stdout:8/79: fsync d5/fb 0 2024-04-05T21:26:25.802 INFO:tasks.workunit.client.1.smithi188.stdout:4/108: dread - d0/d14/d15/d1b/f1e zero size 2024-04-05T21:26:25.802 INFO:tasks.workunit.client.1.smithi188.stdout:7/235: creat d3/d2c/f4c x:0 0 0 2024-04-05T21:26:25.803 INFO:tasks.workunit.client.1.smithi188.stdout:7/236: chown d3/d2c/f4c 1478 1 2024-04-05T21:26:25.806 INFO:tasks.workunit.client.1.smithi188.stdout:8/80: creat d5/f18 x:0 0 0 2024-04-05T21:26:25.807 INFO:tasks.workunit.client.1.smithi188.stdout:4/109: dread d0/d14/d15/d1b/f1d [0,4194304] 0 2024-04-05T21:26:25.807 INFO:tasks.workunit.client.1.smithi188.stdout:7/237: unlink d3/d4/db/d2e/d37/f47 0 2024-04-05T21:26:25.808 INFO:tasks.workunit.client.1.smithi188.stdout:8/81: creat d5/dc/f19 x:0 0 0 2024-04-05T21:26:25.809 INFO:tasks.workunit.client.1.smithi188.stdout:7/238: dread d3/d4/f1b [0,4194304] 0 2024-04-05T21:26:25.809 INFO:tasks.workunit.client.1.smithi188.stdout:4/110: rmdir d0 39 2024-04-05T21:26:25.811 INFO:tasks.workunit.client.1.smithi188.stdout:4/111: chown d0/d14/d15/f1a 224715132 1 2024-04-05T21:26:25.812 INFO:tasks.workunit.client.1.smithi188.stdout:7/239: creat d3/d4/db/d2e/d3b/f4d x:0 0 0 2024-04-05T21:26:25.813 INFO:tasks.workunit.client.1.smithi188.stdout:8/82: mknod d5/c1a 0 2024-04-05T21:26:25.813 INFO:tasks.workunit.client.1.smithi188.stdout:4/112: fsync d0/d14/d15/fb 0 2024-04-05T21:26:25.815 INFO:tasks.workunit.client.1.smithi188.stdout:7/240: rename d3/d4/f1b to d3/d4/db/d2e/d37/f4e 0 2024-04-05T21:26:25.816 INFO:tasks.workunit.client.1.smithi188.stdout:4/113: mknod d0/d2/c2e 0 2024-04-05T21:26:25.816 INFO:tasks.workunit.client.1.smithi188.stdout:7/241: write d3/d2c/f4c [565197,115736] 0 2024-04-05T21:26:25.819 INFO:tasks.workunit.client.1.smithi188.stdout:4/114: mkdir d0/d14/d2f 0 2024-04-05T21:26:25.819 INFO:tasks.workunit.client.1.smithi188.stdout:7/242: fsync d3/d2c/f4c 0 2024-04-05T21:26:25.820 INFO:tasks.workunit.client.1.smithi188.stdout:4/115: readlink d0/d2/d2a/l2b 0 2024-04-05T21:26:25.821 INFO:tasks.workunit.client.1.smithi188.stdout:7/243: rmdir d3/d4/db 39 2024-04-05T21:26:25.822 INFO:tasks.workunit.client.1.smithi188.stdout:4/116: unlink d0/d2/f10 0 2024-04-05T21:26:25.823 INFO:tasks.workunit.client.1.smithi188.stdout:4/117: dread - d0/d14/d15/d1b/f1e zero size 2024-04-05T21:26:25.826 INFO:tasks.workunit.client.1.smithi188.stdout:4/118: creat d0/d14/d15/f30 x:0 0 0 2024-04-05T21:26:25.827 INFO:tasks.workunit.client.1.smithi188.stdout:7/244: getdents d3/d4/d1d 0 2024-04-05T21:26:25.828 INFO:tasks.workunit.client.1.smithi188.stdout:4/119: chown d0/d2/la 942736 1 2024-04-05T21:26:25.829 INFO:tasks.workunit.client.1.smithi188.stdout:4/120: dread - d0/d2/f26 zero size 2024-04-05T21:26:25.831 INFO:tasks.workunit.client.1.smithi188.stdout:4/121: symlink d0/d2/d17/l31 0 2024-04-05T21:26:25.833 INFO:tasks.workunit.client.1.smithi188.stdout:4/122: dread d0/d14/d15/fb [0,4194304] 0 2024-04-05T21:26:25.834 INFO:tasks.workunit.client.1.smithi188.stdout:4/123: fsync d0/d2/d17/f18 0 2024-04-05T21:26:25.837 INFO:tasks.workunit.client.1.smithi188.stdout:7/245: dwrite d3/d4/d1d/f2b [0,4194304] 0 2024-04-05T21:26:25.838 INFO:tasks.workunit.client.1.smithi188.stdout:4/124: mknod d0/d14/c32 0 2024-04-05T21:26:25.841 INFO:tasks.workunit.client.1.smithi188.stdout:7/246: creat d3/d4/db/d2e/f4f x:0 0 0 2024-04-05T21:26:25.842 INFO:tasks.workunit.client.1.smithi188.stdout:4/125: dread d0/d14/d15/f29 [0,4194304] 0 2024-04-05T21:26:25.844 INFO:tasks.workunit.client.1.smithi188.stdout:4/126: mknod d0/d2/d17/c33 0 2024-04-05T21:26:25.845 INFO:tasks.workunit.client.1.smithi188.stdout:7/247: dread d3/d4/f7 [0,4194304] 0 2024-04-05T21:26:25.846 INFO:tasks.workunit.client.1.smithi188.stdout:4/127: mkdir d0/d2/d2a/d34 0 2024-04-05T21:26:25.846 INFO:tasks.workunit.client.1.smithi188.stdout:7/248: write d3/d4/db/f38 [691446,82733] 0 2024-04-05T21:26:25.848 INFO:tasks.workunit.client.1.smithi188.stdout:7/249: write d3/df/f20 [3102428,128691] 0 2024-04-05T21:26:25.849 INFO:tasks.workunit.client.1.smithi188.stdout:4/128: chown d0/d14/d15/d1b/f1e 1878 1 2024-04-05T21:26:25.850 INFO:tasks.workunit.client.1.smithi188.stdout:4/129: dread - d0/d14/f16 zero size 2024-04-05T21:26:25.852 INFO:tasks.workunit.client.1.smithi188.stdout:4/130: mkdir d0/d14/d2f/d35 0 2024-04-05T21:26:25.853 INFO:tasks.workunit.client.1.smithi188.stdout:4/131: chown d0/d14/c32 1069 1 2024-04-05T21:26:25.863 INFO:tasks.workunit.client.1.smithi188.stdout:4/132: truncate d0/d14/d15/d1b/f1d 348910 0 2024-04-05T21:26:25.866 INFO:tasks.workunit.client.1.smithi188.stdout:7/250: dwrite d3/d4/db/f1c [0,4194304] 0 2024-04-05T21:26:25.866 INFO:tasks.workunit.client.1.smithi188.stdout:4/133: rmdir d0/d14/d15 39 2024-04-05T21:26:25.869 INFO:tasks.workunit.client.1.smithi188.stdout:7/251: chown d3/d4/db/d10/d2f/f41 1033736 1 2024-04-05T21:26:25.874 INFO:tasks.workunit.client.1.smithi188.stdout:4/134: rename d0/d14/d15/ff to d0/d14/d2f/f36 0 2024-04-05T21:26:25.875 INFO:tasks.workunit.client.1.smithi188.stdout:4/135: dread - d0/d14/f16 zero size 2024-04-05T21:26:25.878 INFO:tasks.workunit.client.1.smithi188.stdout:7/252: link d3/d4/db/d2e/d37/f4e d3/d4/db/d2e/d37/f50 0 2024-04-05T21:26:25.878 INFO:tasks.workunit.client.1.smithi188.stdout:7/253: readlink d3/d4/db/d2e/d37/l4b 0 2024-04-05T21:26:25.879 INFO:tasks.workunit.client.1.smithi188.stdout:4/136: creat d0/d14/f37 x:0 0 0 2024-04-05T21:26:25.880 INFO:tasks.workunit.client.1.smithi188.stdout:7/254: rmdir d3/d4/db/d10/d2f 39 2024-04-05T21:26:25.882 INFO:tasks.workunit.client.1.smithi188.stdout:7/255: truncate d3/d4/db/d2e/f35 5065290 0 2024-04-05T21:26:25.885 INFO:tasks.workunit.client.1.smithi188.stdout:7/256: rename d3/d4/db/d2e/f4f to d3/d4/db/d2e/f51 0 2024-04-05T21:26:25.888 INFO:tasks.workunit.client.1.smithi188.stdout:4/137: write d0/d14/d15/f1a [5064658,79517] 0 2024-04-05T21:26:25.890 INFO:tasks.workunit.client.1.smithi188.stdout:7/257: rename d3/d4/db/d2e/d3b/c3d to d3/d4/c52 0 2024-04-05T21:26:25.890 INFO:tasks.workunit.client.1.smithi188.stdout:4/138: fdatasync d0/d14/f1c 0 2024-04-05T21:26:25.892 INFO:tasks.workunit.client.1.smithi188.stdout:7/258: mknod d3/d4/db/d2e/d49/c53 0 2024-04-05T21:26:25.896 INFO:tasks.workunit.client.1.smithi188.stdout:4/139: dwrite d0/d14/d15/f1a [0,4194304] 0 2024-04-05T21:26:25.896 INFO:tasks.workunit.client.1.smithi188.stdout:7/259: link d3/l26 d3/d4/l54 0 2024-04-05T21:26:25.897 INFO:tasks.workunit.client.1.smithi188.stdout:4/140: stat d0/d2/d17/c33 0 2024-04-05T21:26:25.898 INFO:tasks.workunit.client.1.smithi188.stdout:4/141: dread d0/d14/d15/f29 [0,4194304] 0 2024-04-05T21:26:25.899 INFO:tasks.workunit.client.1.smithi188.stdout:4/142: write d0/d14/d15/fb [642968,6216] 0 2024-04-05T21:26:25.900 INFO:tasks.workunit.client.1.smithi188.stdout:4/143: chown d0/d14/d15/d1b/f1e 14 1 2024-04-05T21:26:25.902 INFO:tasks.workunit.client.1.smithi188.stdout:4/144: fdatasync d0/d14/d15/f29 0 2024-04-05T21:26:25.914 INFO:tasks.workunit.client.1.smithi188.stdout:4/145: fdatasync d0/d14/d15/f1a 0 2024-04-05T21:26:25.914 INFO:tasks.workunit.client.1.smithi188.stdout:4/146: write d0/d14/d15/f1a [4219643,10887] 0 2024-04-05T21:26:25.917 INFO:tasks.workunit.client.1.smithi188.stdout:4/147: creat d0/d14/f38 x:0 0 0 2024-04-05T21:26:25.919 INFO:tasks.workunit.client.1.smithi188.stdout:4/148: creat d0/d14/f39 x:0 0 0 2024-04-05T21:26:25.920 INFO:tasks.workunit.client.1.smithi188.stdout:4/149: dread - d0/d14/d15/d1b/f1e zero size 2024-04-05T21:26:25.921 INFO:tasks.workunit.client.1.smithi188.stdout:4/150: mkdir d0/d2/d17/d3a 0 2024-04-05T21:26:25.928 INFO:tasks.workunit.client.1.smithi188.stdout:4/151: truncate d0/d14/d15/f1a 2107515 0 2024-04-05T21:26:25.930 INFO:tasks.workunit.client.1.smithi188.stdout:4/152: mknod d0/d14/d2f/d35/c3b 0 2024-04-05T21:26:25.930 INFO:tasks.workunit.client.1.smithi188.stdout:4/153: chown d0/d14/d2f/d35 125 1 2024-04-05T21:26:25.931 INFO:tasks.workunit.client.1.smithi188.stdout:4/154: write d0/d14/d15/fb [9008,93032] 0 2024-04-05T21:26:25.932 INFO:tasks.workunit.client.1.smithi188.stdout:4/155: readlink d0/d2/l4 0 2024-04-05T21:26:25.933 INFO:tasks.workunit.client.1.smithi188.stdout:4/156: fsync d0/d14/f39 0 2024-04-05T21:26:25.933 INFO:tasks.workunit.client.1.smithi188.stdout:4/157: fdatasync d0/d14/f39 0 2024-04-05T21:26:25.934 INFO:tasks.workunit.client.1.smithi188.stdout:4/158: dread - d0/d14/f16 zero size 2024-04-05T21:26:25.936 INFO:tasks.workunit.client.1.smithi188.stdout:4/159: symlink d0/d14/d15/d1b/l3c 0 2024-04-05T21:26:25.936 INFO:tasks.workunit.client.1.smithi188.stdout:4/160: truncate d0/d14/f38 932704 0 2024-04-05T21:26:25.937 INFO:tasks.workunit.client.1.smithi188.stdout:4/161: write d0/d2/d17/f18 [943597,5132] 0 2024-04-05T21:26:25.939 INFO:tasks.workunit.client.1.smithi188.stdout:4/162: creat d0/d14/d15/d1b/f3d x:0 0 0 2024-04-05T21:26:25.941 INFO:tasks.workunit.client.1.smithi188.stdout:4/163: creat d0/f3e x:0 0 0 2024-04-05T21:26:25.944 INFO:tasks.workunit.client.1.smithi188.stdout:4/164: dread d0/d2/d17/f18 [0,4194304] 0 2024-04-05T21:26:25.946 INFO:tasks.workunit.client.1.smithi188.stdout:4/165: unlink d0/f3e 0 2024-04-05T21:26:25.952 INFO:tasks.workunit.client.1.smithi188.stdout:4/166: dwrite d0/d14/d15/d1b/f3d [0,4194304] 0 2024-04-05T21:26:25.960 INFO:tasks.workunit.client.1.smithi188.stdout:4/167: creat d0/f3f x:0 0 0 2024-04-05T21:26:25.966 INFO:tasks.workunit.client.1.smithi188.stdout:4/168: dwrite d0/d14/f38 [0,4194304] 0 2024-04-05T21:26:25.973 INFO:tasks.workunit.client.1.smithi188.stdout:4/169: link d0/d14/d15/f1a d0/d14/d15/d1b/f40 0 2024-04-05T21:26:25.973 INFO:tasks.workunit.client.1.smithi188.stdout:4/170: fdatasync d0/d14/f37 0 2024-04-05T21:26:25.975 INFO:tasks.workunit.client.1.smithi188.stdout:4/171: creat d0/d2/d2a/d34/f41 x:0 0 0 2024-04-05T21:26:25.977 INFO:tasks.workunit.client.1.smithi188.stdout:4/172: mkdir d0/d14/d15/d42 0 2024-04-05T21:26:25.980 INFO:tasks.workunit.client.1.smithi188.stdout:4/173: rename d0/d14/d15/d1b/f1e to d0/d2/d17/f43 0 2024-04-05T21:26:25.981 INFO:tasks.workunit.client.1.smithi188.stdout:4/174: truncate d0/d14/d15/d1b/f3d 5226934 0 2024-04-05T21:26:25.986 INFO:tasks.workunit.client.1.smithi188.stdout:4/175: dwrite d0/d14/d2f/f36 [0,4194304] 0 2024-04-05T21:26:25.987 INFO:tasks.workunit.client.1.smithi188.stdout:4/176: dread - d0/d14/f16 zero size 2024-04-05T21:26:25.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:25 smithi121 ceph-mon[39723]: pgmap v11: 65 pgs: 65 active+clean; 373 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 946 KiB/s rd, 15 MiB/s wr, 217 op/s 2024-04-05T21:26:25.991 INFO:tasks.workunit.client.1.smithi188.stdout:4/177: rmdir d0/d14 39 2024-04-05T21:26:25.995 INFO:tasks.workunit.client.1.smithi188.stdout:4/178: rename d0/d14/d2f/f36 to d0/d2/d17/f44 0 2024-04-05T21:26:25.996 INFO:tasks.workunit.client.1.smithi188.stdout:4/179: write d0/d14/d15/fb [1775292,45210] 0 2024-04-05T21:26:25.998 INFO:tasks.workunit.client.1.smithi188.stdout:4/180: stat d0/c24 0 2024-04-05T21:26:26.000 INFO:tasks.workunit.client.1.smithi188.stdout:4/181: symlink d0/d14/d2f/l45 0 2024-04-05T21:26:26.002 INFO:tasks.workunit.client.1.smithi188.stdout:4/182: write d0/d2/d17/f44 [5208890,29248] 0 2024-04-05T21:26:26.004 INFO:tasks.workunit.client.1.smithi188.stdout:4/183: truncate d0/d14/d15/fb 2408264 0 2024-04-05T21:26:26.013 INFO:tasks.workunit.client.1.smithi188.stdout:4/184: dwrite d0/d14/d15/f1a [0,4194304] 0 2024-04-05T21:26:26.015 INFO:tasks.workunit.client.1.smithi188.stdout:4/185: unlink d0/d14/f37 0 2024-04-05T21:26:26.020 INFO:tasks.workunit.client.1.smithi188.stdout:4/186: dwrite d0/d2/d17/f44 [0,4194304] 0 2024-04-05T21:26:26.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:25 smithi188 ceph-mon[49715]: pgmap v11: 65 pgs: 65 active+clean; 373 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 946 KiB/s rd, 15 MiB/s wr, 217 op/s 2024-04-05T21:26:26.036 INFO:tasks.workunit.client.1.smithi188.stdout:1/144: sync 2024-04-05T21:26:26.036 INFO:tasks.workunit.client.1.smithi188.stdout:3/132: sync 2024-04-05T21:26:26.036 INFO:tasks.workunit.client.1.smithi188.stdout:9/84: sync 2024-04-05T21:26:26.036 INFO:tasks.workunit.client.1.smithi188.stdout:0/76: sync 2024-04-05T21:26:26.038 INFO:tasks.workunit.client.1.smithi188.stdout:4/187: chown d0/d14/d2f/l45 39703984 1 2024-04-05T21:26:26.039 INFO:tasks.workunit.client.1.smithi188.stdout:9/85: read f1 [8710008,113910] 0 2024-04-05T21:26:26.040 INFO:tasks.workunit.client.1.smithi188.stdout:4/188: write d0/f3f [787175,74396] 0 2024-04-05T21:26:26.040 INFO:tasks.workunit.client.1.smithi188.stdout:9/86: chown l8 348227115 1 2024-04-05T21:26:26.041 INFO:tasks.workunit.client.1.smithi188.stdout:9/87: stat f1 0 2024-04-05T21:26:26.041 INFO:tasks.workunit.client.1.smithi188.stdout:3/133: mkdir d5/dc/d2f 0 2024-04-05T21:26:26.041 INFO:tasks.workunit.client.1.smithi188.stdout:0/77: symlink d4/d8/d10/l1a 0 2024-04-05T21:26:26.045 INFO:tasks.workunit.client.1.smithi188.stdout:3/134: creat d5/d7/f30 x:0 0 0 2024-04-05T21:26:26.047 INFO:tasks.workunit.client.1.smithi188.stdout:0/78: creat d4/d8/d10/f1b x:0 0 0 2024-04-05T21:26:26.048 INFO:tasks.workunit.client.1.smithi188.stdout:4/189: dread d0/d14/d15/fb [0,4194304] 0 2024-04-05T21:26:26.048 INFO:tasks.workunit.client.1.smithi188.stdout:1/145: dread d1/dc/d16/d21/d27/f29 [0,4194304] 0 2024-04-05T21:26:26.049 INFO:tasks.workunit.client.1.smithi188.stdout:3/135: mkdir d5/d7/d31 0 2024-04-05T21:26:26.053 INFO:tasks.workunit.client.1.smithi188.stdout:1/146: mknod d1/d4/d13/c30 0 2024-04-05T21:26:26.056 INFO:tasks.workunit.client.1.smithi188.stdout:3/136: creat d5/d7/d31/f32 x:0 0 0 2024-04-05T21:26:26.056 INFO:tasks.workunit.client.1.smithi188.stdout:0/79: dread d4/fe [0,4194304] 0 2024-04-05T21:26:26.058 INFO:tasks.workunit.client.1.smithi188.stdout:4/190: dwrite d0/f5 [4194304,4194304] 0 2024-04-05T21:26:26.060 INFO:tasks.workunit.client.1.smithi188.stdout:0/80: read f3 [4078340,98484] 0 2024-04-05T21:26:26.061 INFO:tasks.workunit.client.1.smithi188.stdout:9/88: dread f5 [0,4194304] 0 2024-04-05T21:26:26.063 INFO:tasks.workunit.client.1.smithi188.stdout:3/137: write d5/dc/f1f [2665905,5376] 0 2024-04-05T21:26:26.064 INFO:tasks.workunit.client.1.smithi188.stdout:4/191: rename d0/d14/f16 to d0/d2/d2a/d34/f46 0 2024-04-05T21:26:26.065 INFO:tasks.workunit.client.1.smithi188.stdout:1/147: dread d1/dc/f1b [0,4194304] 0 2024-04-05T21:26:26.070 INFO:tasks.workunit.client.1.smithi188.stdout:1/148: read d1/f14 [77566,122231] 0 2024-04-05T21:26:26.072 INFO:tasks.workunit.client.1.smithi188.stdout:4/192: fdatasync d0/d14/d15/d1b/f1d 0 2024-04-05T21:26:26.073 INFO:tasks.workunit.client.1.smithi188.stdout:0/81: dwrite d4/d8/d10/f1b [0,4194304] 0 2024-04-05T21:26:26.074 INFO:tasks.workunit.client.1.smithi188.stdout:3/138: dread d5/fa [0,4194304] 0 2024-04-05T21:26:26.076 INFO:tasks.workunit.client.1.smithi188.stdout:4/193: symlink d0/d2/d2a/d34/l47 0 2024-04-05T21:26:26.079 INFO:tasks.workunit.client.1.smithi188.stdout:1/149: mknod d1/dc/d16/d2d/c31 0 2024-04-05T21:26:26.081 INFO:tasks.workunit.client.1.smithi188.stdout:0/82: mkdir d4/d8/db/d1c 0 2024-04-05T21:26:26.083 INFO:tasks.workunit.client.1.smithi188.stdout:4/194: mkdir d0/d2/d17/d48 0 2024-04-05T21:26:26.083 INFO:tasks.workunit.client.1.smithi188.stdout:9/89: link dd/d11/c18 dd/d11/c1d 0 2024-04-05T21:26:26.083 INFO:tasks.workunit.client.1.smithi188.stdout:3/139: dwrite d5/dc/f2a [0,4194304] 0 2024-04-05T21:26:26.085 INFO:tasks.workunit.client.1.smithi188.stdout:0/83: truncate d4/d6/f15 26708 0 2024-04-05T21:26:26.085 INFO:tasks.workunit.client.1.smithi188.stdout:9/90: chown dd/l1b 2085711640 1 2024-04-05T21:26:26.085 INFO:tasks.workunit.client.1.smithi188.stdout:4/195: write d0/d14/d15/f1a [3115952,53917] 0 2024-04-05T21:26:26.086 INFO:tasks.workunit.client.1.smithi188.stdout:1/150: symlink d1/l32 0 2024-04-05T21:26:26.092 INFO:tasks.workunit.client.1.smithi188.stdout:9/91: rename dd/df to dd/d11/d1e 0 2024-04-05T21:26:26.095 INFO:tasks.workunit.client.1.smithi188.stdout:4/196: dread d0/d2/d17/f44 [4194304,4194304] 0 2024-04-05T21:26:26.097 INFO:tasks.workunit.client.1.smithi188.stdout:3/140: truncate d5/dc/f2a 3878737 0 2024-04-05T21:26:26.099 INFO:tasks.workunit.client.1.smithi188.stdout:4/197: unlink d0/d14/d15/le 0 2024-04-05T21:26:26.101 INFO:tasks.workunit.client.1.smithi188.stdout:3/141: creat d5/df/f33 x:0 0 0 2024-04-05T21:26:26.102 INFO:tasks.workunit.client.1.smithi188.stdout:9/92: dwrite dd/d11/d1e/f19 [0,4194304] 0 2024-04-05T21:26:26.103 INFO:tasks.workunit.client.1.smithi188.stdout:9/93: fsync fc 0 2024-04-05T21:26:26.103 INFO:tasks.workunit.client.1.smithi188.stdout:3/142: fdatasync d5/d7/d31/f32 0 2024-04-05T21:26:26.109 INFO:tasks.workunit.client.1.smithi188.stdout:9/94: mkdir dd/d11/d1f 0 2024-04-05T21:26:26.109 INFO:tasks.workunit.client.1.smithi188.stdout:4/198: write d0/d14/d15/fb [2498704,12078] 0 2024-04-05T21:26:26.110 INFO:tasks.workunit.client.1.smithi188.stdout:4/199: chown d0/d2/l4 214977 1 2024-04-05T21:26:26.111 INFO:tasks.workunit.client.1.smithi188.stdout:3/143: symlink d5/l34 0 2024-04-05T21:26:26.114 INFO:tasks.workunit.client.1.smithi188.stdout:3/144: rmdir d5/df 39 2024-04-05T21:26:26.116 INFO:tasks.workunit.client.1.smithi188.stdout:4/200: rename d0/d14/d15/f30 to d0/d2/d17/d48/f49 0 2024-04-05T21:26:26.117 INFO:tasks.workunit.client.1.smithi188.stdout:3/145: dread d5/d7/f25 [0,4194304] 0 2024-04-05T21:26:26.118 INFO:tasks.workunit.client.1.smithi188.stdout:9/95: getdents dd/d11/d1e 0 2024-04-05T21:26:26.122 INFO:tasks.workunit.client.1.smithi188.stdout:9/96: creat dd/d11/d1e/f20 x:0 0 0 2024-04-05T21:26:26.122 INFO:tasks.workunit.client.1.smithi188.stdout:3/146: mkdir d5/d35 0 2024-04-05T21:26:26.123 INFO:tasks.workunit.client.1.smithi188.stdout:9/97: write fb [444342,76598] 0 2024-04-05T21:26:26.123 INFO:tasks.workunit.client.1.smithi188.stdout:4/201: symlink d0/d2/d17/d3a/l4a 0 2024-04-05T21:26:26.127 INFO:tasks.workunit.client.1.smithi188.stdout:9/98: unlink dd/d11/d1e/f20 0 2024-04-05T21:26:26.127 INFO:tasks.workunit.client.1.smithi188.stdout:9/99: chown dd/f12 3 1 2024-04-05T21:26:26.128 INFO:tasks.workunit.client.1.smithi188.stdout:3/147: dread d5/fa [0,4194304] 0 2024-04-05T21:26:26.130 INFO:tasks.workunit.client.1.smithi188.stdout:3/148: creat d5/d7/f36 x:0 0 0 2024-04-05T21:26:26.131 INFO:tasks.workunit.client.1.smithi188.stdout:3/149: read - d5/d7/f30 zero size 2024-04-05T21:26:26.132 INFO:tasks.workunit.client.1.smithi188.stdout:4/202: rename d0/d2/la to d0/d2/d17/d48/l4b 0 2024-04-05T21:26:26.132 INFO:tasks.workunit.client.1.smithi188.stdout:4/203: stat d0/d14/l28 0 2024-04-05T21:26:26.133 INFO:tasks.workunit.client.1.smithi188.stdout:4/204: read - d0/d2/f26 zero size 2024-04-05T21:26:26.134 INFO:tasks.workunit.client.1.smithi188.stdout:4/205: write d0/d14/f39 [246098,102897] 0 2024-04-05T21:26:26.134 INFO:tasks.workunit.client.1.smithi188.stdout:9/100: creat dd/d11/f21 x:0 0 0 2024-04-05T21:26:26.137 INFO:tasks.workunit.client.1.smithi188.stdout:3/150: creat d5/f37 x:0 0 0 2024-04-05T21:26:26.137 INFO:tasks.workunit.client.1.smithi188.stdout:4/206: write d0/d14/d15/d1b/f40 [1822283,19113] 0 2024-04-05T21:26:26.137 INFO:tasks.workunit.client.1.smithi188.stdout:9/101: chown dd/d11/c1d 358 1 2024-04-05T21:26:26.137 INFO:tasks.workunit.client.1.smithi188.stdout:3/151: chown d5/d2b/f2e 101 1 2024-04-05T21:26:26.138 INFO:tasks.workunit.client.1.smithi188.stdout:4/207: readlink d0/d14/d15/d1b/l20 0 2024-04-05T21:26:26.138 INFO:tasks.workunit.client.1.smithi188.stdout:3/152: chown d5/dc/f19 0 1 2024-04-05T21:26:26.140 INFO:tasks.workunit.client.1.smithi188.stdout:9/102: mkdir dd/d11/d22 0 2024-04-05T21:26:26.147 INFO:tasks.workunit.client.1.smithi188.stdout:4/208: dwrite d0/d2/d17/f18 [0,4194304] 0 2024-04-05T21:26:26.151 INFO:tasks.workunit.client.1.smithi188.stdout:3/153: dwrite d5/dc/f2a [0,4194304] 0 2024-04-05T21:26:26.152 INFO:tasks.workunit.client.1.smithi188.stdout:9/103: dwrite dd/f16 [0,4194304] 0 2024-04-05T21:26:26.153 INFO:tasks.workunit.client.1.smithi188.stdout:9/104: stat dd/f12 0 2024-04-05T21:26:26.154 INFO:tasks.workunit.client.1.smithi188.stdout:4/209: write d0/d2/d17/f43 [758068,115093] 0 2024-04-05T21:26:26.154 INFO:tasks.workunit.client.1.smithi188.stdout:3/154: write d5/d7/f20 [722583,104931] 0 2024-04-05T21:26:26.155 INFO:tasks.workunit.client.1.smithi188.stdout:3/155: read - d5/f21 zero size 2024-04-05T21:26:26.159 INFO:tasks.workunit.client.1.smithi188.stdout:3/156: creat d5/dc/d2f/f38 x:0 0 0 2024-04-05T21:26:26.159 INFO:tasks.workunit.client.1.smithi188.stdout:3/157: readlink d5/l1c 0 2024-04-05T21:26:26.165 INFO:tasks.workunit.client.1.smithi188.stdout:4/210: dwrite d0/d2/d17/f44 [0,4194304] 0 2024-04-05T21:26:26.166 INFO:tasks.workunit.client.1.smithi188.stdout:4/211: readlink d0/d14/d15/l22 0 2024-04-05T21:26:26.166 INFO:tasks.workunit.client.1.smithi188.stdout:3/158: dread d5/d7/f20 [0,4194304] 0 2024-04-05T21:26:26.167 INFO:tasks.workunit.client.1.smithi188.stdout:4/212: stat d0/d14/d15/l22 0 2024-04-05T21:26:26.171 INFO:tasks.workunit.client.1.smithi188.stdout:3/159: mkdir d5/d39 0 2024-04-05T21:26:26.172 INFO:tasks.workunit.client.1.smithi188.stdout:4/213: creat d0/d14/d15/d42/f4c x:0 0 0 2024-04-05T21:26:26.176 INFO:tasks.workunit.client.1.smithi188.stdout:3/160: rename d5/dc/c13 to d5/c3a 0 2024-04-05T21:26:26.178 INFO:tasks.workunit.client.1.smithi188.stdout:3/161: mknod d5/d2b/c3b 0 2024-04-05T21:26:26.183 INFO:tasks.workunit.client.1.smithi188.stdout:3/162: fsync d5/dc/d2f/f38 0 2024-04-05T21:26:26.185 INFO:tasks.workunit.client.1.smithi188.stdout:3/163: mknod d5/d2b/c3c 0 2024-04-05T21:26:26.191 INFO:tasks.workunit.client.1.smithi188.stdout:4/214: dwrite d0/d2/f6 [0,4194304] 0 2024-04-05T21:26:26.205 INFO:tasks.workunit.client.1.smithi188.stdout:3/164: dwrite d5/f1e [0,4194304] 0 2024-04-05T21:26:26.210 INFO:tasks.workunit.client.1.smithi188.stdout:3/165: unlink d5/l34 0 2024-04-05T21:26:26.210 INFO:tasks.workunit.client.1.smithi188.stdout:4/215: dread d0/d14/f1c [0,4194304] 0 2024-04-05T21:26:26.210 INFO:tasks.workunit.client.1.smithi188.stdout:3/166: stat d5/d2b/c3b 0 2024-04-05T21:26:26.211 INFO:tasks.workunit.client.1.smithi188.stdout:3/167: chown d5/d7/c9 2266 1 2024-04-05T21:26:26.214 INFO:tasks.workunit.client.1.smithi188.stdout:4/216: creat d0/d14/d15/d1b/f4d x:0 0 0 2024-04-05T21:26:26.215 INFO:tasks.workunit.client.1.smithi188.stdout:4/217: chown d0/d2 3151 1 2024-04-05T21:26:26.216 INFO:tasks.workunit.client.1.smithi188.stdout:3/168: creat d5/dc/f3d x:0 0 0 2024-04-05T21:26:26.218 INFO:tasks.workunit.client.1.smithi188.stdout:4/218: unlink d0/l1 0 2024-04-05T21:26:26.221 INFO:tasks.workunit.client.1.smithi188.stdout:3/169: dread d5/fb [0,4194304] 0 2024-04-05T21:26:26.305 INFO:tasks.workunit.client.1.smithi188.stdout:6/149: sync 2024-04-05T21:26:26.305 INFO:tasks.workunit.client.1.smithi188.stdout:5/112: sync 2024-04-05T21:26:26.305 INFO:tasks.workunit.client.1.smithi188.stdout:2/153: sync 2024-04-05T21:26:26.313 INFO:tasks.workunit.client.1.smithi188.stdout:2/154: fdatasync d5/db/dd/f15 0 2024-04-05T21:26:26.314 INFO:tasks.workunit.client.1.smithi188.stdout:3/170: unlink d5/dc/c29 0 2024-04-05T21:26:26.315 INFO:tasks.workunit.client.1.smithi188.stdout:2/155: truncate d5/f14 1306286 0 2024-04-05T21:26:26.316 INFO:tasks.workunit.client.1.smithi188.stdout:5/113: rename d6/fd to d6/d15/d1b/f2a 0 2024-04-05T21:26:26.317 INFO:tasks.workunit.client.1.smithi188.stdout:4/219: dwrite d0/d2/d17/f18 [0,4194304] 0 2024-04-05T21:26:26.318 INFO:tasks.workunit.client.1.smithi188.stdout:4/220: fdatasync d0/d14/f1c 0 2024-04-05T21:26:26.320 INFO:tasks.workunit.client.1.smithi188.stdout:3/171: unlink d5/d7/d31/f32 0 2024-04-05T21:26:26.321 INFO:tasks.workunit.client.1.smithi188.stdout:3/172: dread - d5/d7/f36 zero size 2024-04-05T21:26:26.322 INFO:tasks.workunit.client.1.smithi188.stdout:2/156: symlink d5/l29 0 2024-04-05T21:26:26.322 INFO:tasks.workunit.client.1.smithi188.stdout:6/150: write d6/f11 [632485,100440] 0 2024-04-05T21:26:26.325 INFO:tasks.workunit.client.1.smithi188.stdout:4/221: rename d0/d2/f6 to d0/d2/d17/f4e 0 2024-04-05T21:26:26.326 INFO:tasks.workunit.client.1.smithi188.stdout:3/173: mknod d5/dc/c3e 0 2024-04-05T21:26:26.327 INFO:tasks.workunit.client.1.smithi188.stdout:5/114: link d6/d17/d24/f26 d6/d17/d24/f2b 0 2024-04-05T21:26:26.328 INFO:tasks.workunit.client.1.smithi188.stdout:6/151: rmdir d6/d13 39 2024-04-05T21:26:26.329 INFO:tasks.workunit.client.1.smithi188.stdout:5/115: dread d6/f11 [0,4194304] 0 2024-04-05T21:26:26.330 INFO:tasks.workunit.client.1.smithi188.stdout:4/222: mkdir d0/d2/d4f 0 2024-04-05T21:26:26.334 INFO:tasks.workunit.client.1.smithi188.stdout:5/116: write d6/d17/d24/f26 [1622401,2649] 0 2024-04-05T21:26:26.334 INFO:tasks.workunit.client.1.smithi188.stdout:3/174: getdents d5/d35 0 2024-04-05T21:26:26.335 INFO:tasks.workunit.client.1.smithi188.stdout:4/223: creat d0/d2/d17/f50 x:0 0 0 2024-04-05T21:26:26.337 INFO:tasks.workunit.client.1.smithi188.stdout:6/152: rmdir d6/d13/d18/d19 39 2024-04-05T21:26:26.340 INFO:tasks.workunit.client.1.smithi188.stdout:3/175: symlink d5/dc/l3f 0 2024-04-05T21:26:26.341 INFO:tasks.workunit.client.1.smithi188.stdout:5/117: dwrite d6/f12 [0,4194304] 0 2024-04-05T21:26:26.343 INFO:tasks.workunit.client.1.smithi188.stdout:4/224: mkdir d0/d2/d2a/d51 0 2024-04-05T21:26:26.348 INFO:tasks.workunit.client.1.smithi188.stdout:4/225: rmdir d0/d14/d15 39 2024-04-05T21:26:26.349 INFO:tasks.workunit.client.1.smithi188.stdout:6/153: mkdir d6/d13/d18/d19/d2c/d34 0 2024-04-05T21:26:26.349 INFO:tasks.workunit.client.1.smithi188.stdout:2/157: write d5/db/dd/f11 [822253,113170] 0 2024-04-05T21:26:26.350 INFO:tasks.workunit.client.1.smithi188.stdout:3/176: creat d5/d35/f40 x:0 0 0 2024-04-05T21:26:26.352 INFO:tasks.workunit.client.1.smithi188.stdout:4/226: mkdir d0/d2/d52 0 2024-04-05T21:26:26.353 INFO:tasks.workunit.client.1.smithi188.stdout:6/154: write d6/d13/f1d [1218952,52150] 0 2024-04-05T21:26:26.355 INFO:tasks.workunit.client.1.smithi188.stdout:3/177: chown d5/c3a 398592305 1 2024-04-05T21:26:26.356 INFO:tasks.workunit.client.1.smithi188.stdout:2/158: symlink d5/l2a 0 2024-04-05T21:26:26.358 INFO:tasks.workunit.client.1.smithi188.stdout:5/118: rename d6/f12 to d6/f2c 0 2024-04-05T21:26:26.361 INFO:tasks.workunit.client.1.smithi188.stdout:2/159: rename d5/db/dd to d5/db/dd/d24/d2b 22 2024-04-05T21:26:26.366 INFO:tasks.workunit.client.1.smithi188.stdout:4/227: dwrite d0/d2/d17/f43 [0,4194304] 0 2024-04-05T21:26:26.368 INFO:tasks.workunit.client.1.smithi188.stdout:6/155: dwrite f5 [0,4194304] 0 2024-04-05T21:26:26.370 INFO:tasks.workunit.client.1.smithi188.stdout:5/119: rename d6/d17/d24/f26 to d6/f2d 0 2024-04-05T21:26:26.371 INFO:tasks.workunit.client.1.smithi188.stdout:3/178: dwrite d5/d35/f40 [0,4194304] 0 2024-04-05T21:26:26.375 INFO:tasks.workunit.client.1.smithi188.stdout:5/120: read d6/d15/d1b/f1e [42152,56918] 0 2024-04-05T21:26:26.376 INFO:tasks.workunit.client.1.smithi188.stdout:4/228: mkdir d0/d14/d15/d53 0 2024-04-05T21:26:26.377 INFO:tasks.workunit.client.1.smithi188.stdout:2/160: dwrite d5/fc [0,4194304] 0 2024-04-05T21:26:26.381 INFO:tasks.workunit.client.1.smithi188.stdout:5/121: mkdir d6/d17/d2e 0 2024-04-05T21:26:26.381 INFO:tasks.workunit.client.1.smithi188.stdout:5/122: readlink d6/lc 0 2024-04-05T21:26:26.382 INFO:tasks.workunit.client.1.smithi188.stdout:3/179: creat d5/f41 x:0 0 0 2024-04-05T21:26:26.382 INFO:tasks.workunit.client.1.smithi188.stdout:5/123: chown d6/d17/l1d 156536243 1 2024-04-05T21:26:26.383 INFO:tasks.workunit.client.1.smithi188.stdout:4/229: fdatasync d0/d2/d17/d48/f49 0 2024-04-05T21:26:26.383 INFO:tasks.workunit.client.1.smithi188.stdout:3/180: read d5/dc/f2a [1157518,56471] 0 2024-04-05T21:26:26.383 INFO:tasks.workunit.client.1.smithi188.stdout:2/161: symlink d5/db/l2c 0 2024-04-05T21:26:26.386 INFO:tasks.workunit.client.1.smithi188.stdout:2/162: read d5/f14 [37010,36984] 0 2024-04-05T21:26:26.386 INFO:tasks.workunit.client.1.smithi188.stdout:2/163: readlink d5/l21 0 2024-04-05T21:26:26.387 INFO:tasks.workunit.client.1.smithi188.stdout:3/181: truncate d5/df/f15 814308 0 2024-04-05T21:26:26.388 INFO:tasks.workunit.client.1.smithi188.stdout:4/230: creat d0/d2/d17/d3a/f54 x:0 0 0 2024-04-05T21:26:26.391 INFO:tasks.workunit.client.1.smithi188.stdout:4/231: read d0/d14/d15/fb [1825512,53450] 0 2024-04-05T21:26:26.391 INFO:tasks.workunit.client.1.smithi188.stdout:3/182: dread d5/d2b/f2e [0,4194304] 0 2024-04-05T21:26:26.391 INFO:tasks.workunit.client.1.smithi188.stdout:4/232: truncate d0/f5 8523395 0 2024-04-05T21:26:26.394 INFO:tasks.workunit.client.1.smithi188.stdout:3/183: mknod d5/d2b/c42 0 2024-04-05T21:26:26.396 INFO:tasks.workunit.client.1.smithi188.stdout:4/233: unlink d0/d14/d15/l11 0 2024-04-05T21:26:26.397 INFO:tasks.workunit.client.1.smithi188.stdout:6/156: write d6/fb [412622,102225] 0 2024-04-05T21:26:26.398 INFO:tasks.workunit.client.1.smithi188.stdout:2/164: dwrite d5/f22 [0,4194304] 0 2024-04-05T21:26:26.400 INFO:tasks.workunit.client.1.smithi188.stdout:4/234: mknod d0/d14/d15/c55 0 2024-04-05T21:26:26.405 INFO:tasks.workunit.client.1.smithi188.stdout:6/157: truncate d6/d7/f15 1647599 0 2024-04-05T21:26:26.408 INFO:tasks.workunit.client.1.smithi188.stdout:3/184: dwrite d5/f41 [0,4194304] 0 2024-04-05T21:26:26.408 INFO:tasks.workunit.client.1.smithi188.stdout:2/165: dread d5/db/dd/f11 [0,4194304] 0 2024-04-05T21:26:26.412 INFO:tasks.workunit.client.1.smithi188.stdout:6/158: creat d6/d13/d18/f35 x:0 0 0 2024-04-05T21:26:26.412 INFO:tasks.workunit.client.1.smithi188.stdout:2/166: write d5/fc [1295893,7961] 0 2024-04-05T21:26:26.415 INFO:tasks.workunit.client.1.smithi188.stdout:2/167: readlink d5/l21 0 2024-04-05T21:26:26.416 INFO:tasks.workunit.client.1.smithi188.stdout:4/235: dread d0/d14/f38 [0,4194304] 0 2024-04-05T21:26:26.417 INFO:tasks.workunit.client.1.smithi188.stdout:2/168: chown d5/db/dd/d24 40034722 1 2024-04-05T21:26:26.419 INFO:tasks.workunit.client.1.smithi188.stdout:3/185: mkdir d5/d2b/d43 0 2024-04-05T21:26:26.420 INFO:tasks.workunit.client.1.smithi188.stdout:4/236: dread d0/d14/f1c [0,4194304] 0 2024-04-05T21:26:26.424 INFO:tasks.workunit.client.1.smithi188.stdout:2/169: rename d5/l29 to d5/db/dd/d24/l2d 0 2024-04-05T21:26:26.424 INFO:tasks.workunit.client.1.smithi188.stdout:6/159: dwrite d6/d28/f31 [0,4194304] 0 2024-04-05T21:26:26.431 INFO:tasks.workunit.client.1.smithi188.stdout:3/186: rename d5/dc/f19 to d5/d2b/d43/f44 0 2024-04-05T21:26:26.432 INFO:tasks.workunit.client.1.smithi188.stdout:6/160: rmdir d6/d13/d18/d19 39 2024-04-05T21:26:26.434 INFO:tasks.workunit.client.1.smithi188.stdout:6/161: write d6/d7/f1e [3811990,60859] 0 2024-04-05T21:26:26.436 INFO:tasks.workunit.client.1.smithi188.stdout:3/187: rename d5/f12 to d5/d7/d31/f45 0 2024-04-05T21:26:26.438 INFO:tasks.workunit.client.1.smithi188.stdout:4/237: rename d0/d2/d2a/l2b to d0/d2/d17/d3a/l56 0 2024-04-05T21:26:26.440 INFO:tasks.workunit.client.1.smithi188.stdout:4/238: write d0/d2/d17/d3a/f54 [85045,58345] 0 2024-04-05T21:26:26.445 INFO:tasks.workunit.client.1.smithi188.stdout:6/162: truncate d6/fa 1867777 0 2024-04-05T21:26:26.448 INFO:tasks.workunit.client.1.smithi188.stdout:6/163: read - d6/d13/d18/d19/f27 zero size 2024-04-05T21:26:26.448 INFO:tasks.workunit.client.1.smithi188.stdout:4/239: dwrite d0/d2/d17/f43 [0,4194304] 0 2024-04-05T21:26:26.449 INFO:tasks.workunit.client.1.smithi188.stdout:6/164: read d6/fb [1776673,19519] 0 2024-04-05T21:26:26.449 INFO:tasks.workunit.client.1.smithi188.stdout:2/170: dread d5/db/f20 [0,4194304] 0 2024-04-05T21:26:26.454 INFO:tasks.workunit.client.1.smithi188.stdout:4/240: link d0/d2/d17/f18 d0/d2/d17/d48/f57 0 2024-04-05T21:26:26.455 INFO:tasks.workunit.client.1.smithi188.stdout:4/241: readlink d0/d2/d17/l31 0 2024-04-05T21:26:26.457 INFO:tasks.workunit.client.1.smithi188.stdout:6/165: getdents d6/d13 0 2024-04-05T21:26:26.462 INFO:tasks.workunit.client.1.smithi188.stdout:4/242: rename d0/d2/d17/d48/f49 to d0/d2/f58 0 2024-04-05T21:26:26.462 INFO:tasks.workunit.client.1.smithi188.stdout:2/171: dwrite d5/f14 [0,4194304] 0 2024-04-05T21:26:26.464 INFO:tasks.workunit.client.1.smithi188.stdout:6/166: creat d6/d13/d18/f36 x:0 0 0 2024-04-05T21:26:26.466 INFO:tasks.workunit.client.1.smithi188.stdout:4/243: unlink d0/d14/d15/l22 0 2024-04-05T21:26:26.468 INFO:tasks.workunit.client.1.smithi188.stdout:2/172: dwrite d5/db/f20 [0,4194304] 0 2024-04-05T21:26:26.519 INFO:tasks.workunit.client.1.smithi188.stdout:8/83: sync 2024-04-05T21:26:26.525 INFO:tasks.workunit.client.1.smithi188.stdout:6/167: mknod d6/c37 0 2024-04-05T21:26:26.526 INFO:tasks.workunit.client.1.smithi188.stdout:4/244: mknod d0/d2/d2a/d34/c59 0 2024-04-05T21:26:26.529 INFO:tasks.workunit.client.1.smithi188.stdout:8/84: truncate d5/f16 993274 0 2024-04-05T21:26:26.531 INFO:tasks.workunit.client.1.smithi188.stdout:6/168: mkdir d6/d13/d38 0 2024-04-05T21:26:26.537 INFO:tasks.workunit.client.1.smithi188.stdout:2/173: dwrite d5/fc [0,4194304] 0 2024-04-05T21:26:26.540 INFO:tasks.workunit.client.1.smithi188.stdout:4/245: dread d0/d14/d15/d1b/f3d [0,4194304] 0 2024-04-05T21:26:26.543 INFO:tasks.workunit.client.1.smithi188.stdout:2/174: rmdir d5 39 2024-04-05T21:26:26.545 INFO:tasks.workunit.client.1.smithi188.stdout:4/246: write d0/d2/d2a/d34/f46 [988542,58474] 0 2024-04-05T21:26:26.546 INFO:tasks.workunit.client.1.smithi188.stdout:4/247: chown d0 525217950 1 2024-04-05T21:26:26.547 INFO:tasks.workunit.client.1.smithi188.stdout:6/169: symlink d6/d13/d38/l39 0 2024-04-05T21:26:26.548 INFO:tasks.workunit.client.1.smithi188.stdout:6/170: readlink d6/l9 0 2024-04-05T21:26:26.548 INFO:tasks.workunit.client.1.smithi188.stdout:2/175: creat d5/db/dd/d24/f2e x:0 0 0 2024-04-05T21:26:26.550 INFO:tasks.workunit.client.1.smithi188.stdout:4/248: mknod d0/d2/d2a/c5a 0 2024-04-05T21:26:26.551 INFO:tasks.workunit.client.1.smithi188.stdout:6/171: readlink d6/d13/d18/l21 0 2024-04-05T21:26:26.557 INFO:tasks.workunit.client.1.smithi188.stdout:2/176: dwrite d5/fc [4194304,4194304] 0 2024-04-05T21:26:26.560 INFO:tasks.workunit.client.1.smithi188.stdout:6/172: write d6/d7/f15 [1522028,7517] 0 2024-04-05T21:26:26.561 INFO:tasks.workunit.client.1.smithi188.stdout:4/249: truncate d0/d14/d15/f1a 763731 0 2024-04-05T21:26:26.562 INFO:tasks.workunit.client.1.smithi188.stdout:2/177: dread d5/db/dd/f1c [0,4194304] 0 2024-04-05T21:26:26.562 INFO:tasks.workunit.client.1.smithi188.stdout:6/173: write d6/f2f [1009671,115553] 0 2024-04-05T21:26:26.564 INFO:tasks.workunit.client.1.smithi188.stdout:4/250: rmdir d0/d2/d2a 39 2024-04-05T21:26:26.564 INFO:tasks.workunit.client.1.smithi188.stdout:8/85: dread d5/f8 [0,4194304] 0 2024-04-05T21:26:26.565 INFO:tasks.workunit.client.1.smithi188.stdout:2/178: write d5/f14 [3050884,25998] 0 2024-04-05T21:26:26.565 INFO:tasks.workunit.client.1.smithi188.stdout:8/86: stat d5/ff 0 2024-04-05T21:26:26.565 INFO:tasks.workunit.client.1.smithi188.stdout:6/174: symlink d6/d13/d38/l3a 0 2024-04-05T21:26:26.566 INFO:tasks.workunit.client.1.smithi188.stdout:4/251: readlink d0/d2/l4 0 2024-04-05T21:26:26.566 INFO:tasks.workunit.client.1.smithi188.stdout:8/87: dread - d5/f15 zero size 2024-04-05T21:26:26.567 INFO:tasks.workunit.client.1.smithi188.stdout:6/175: chown d6/d13/d18/d19/d2c/c33 33062 1 2024-04-05T21:26:26.568 INFO:tasks.workunit.client.1.smithi188.stdout:8/88: write d5/f15 [437908,817] 0 2024-04-05T21:26:26.569 INFO:tasks.workunit.client.1.smithi188.stdout:4/252: creat d0/d14/d15/f5b x:0 0 0 2024-04-05T21:26:26.573 INFO:tasks.workunit.client.1.smithi188.stdout:4/253: unlink d0/f3f 0 2024-04-05T21:26:26.574 INFO:tasks.workunit.client.1.smithi188.stdout:6/176: creat d6/f3b x:0 0 0 2024-04-05T21:26:26.576 INFO:tasks.workunit.client.1.smithi188.stdout:6/177: readlink d6/d13/d18/l21 0 2024-04-05T21:26:26.577 INFO:tasks.workunit.client.1.smithi188.stdout:2/179: dwrite d5/db/dd/f1c [0,4194304] 0 2024-04-05T21:26:26.580 INFO:tasks.workunit.client.1.smithi188.stdout:6/178: chown d6/d13/d18/d19/d2c/d34 60474 1 2024-04-05T21:26:26.580 INFO:tasks.workunit.client.1.smithi188.stdout:4/254: link d0/d14/d15/d42/f4c d0/d2/d4f/f5c 0 2024-04-05T21:26:26.583 INFO:tasks.workunit.client.1.smithi188.stdout:4/255: stat d0/d2/d17/d3a/l4a 0 2024-04-05T21:26:26.585 INFO:tasks.workunit.client.1.smithi188.stdout:2/180: link d5/db/dd/f15 d5/f2f 0 2024-04-05T21:26:26.586 INFO:tasks.workunit.client.1.smithi188.stdout:6/179: dwrite d6/d28/f31 [0,4194304] 0 2024-04-05T21:26:26.587 INFO:tasks.workunit.client.1.smithi188.stdout:2/181: chown d5/db/l23 2130352 1 2024-04-05T21:26:26.588 INFO:tasks.workunit.client.1.smithi188.stdout:6/180: dread - d6/f3b zero size 2024-04-05T21:26:26.588 INFO:tasks.workunit.client.1.smithi188.stdout:4/256: symlink d0/d2/l5d 0 2024-04-05T21:26:26.589 INFO:tasks.workunit.client.1.smithi188.stdout:2/182: read d5/f22 [976032,70190] 0 2024-04-05T21:26:26.591 INFO:tasks.workunit.client.1.smithi188.stdout:4/257: rename d0/d2/d4f to d0/d2/d4f/d5e 22 2024-04-05T21:26:26.592 INFO:tasks.workunit.client.1.smithi188.stdout:6/181: mkdir d6/d13/d18/d3c 0 2024-04-05T21:26:26.593 INFO:tasks.workunit.client.1.smithi188.stdout:6/182: readlink d6/d13/d18/l1b 0 2024-04-05T21:26:26.593 INFO:tasks.workunit.client.1.smithi188.stdout:6/183: dread - d6/d13/d18/f36 zero size 2024-04-05T21:26:26.596 INFO:tasks.workunit.client.1.smithi188.stdout:6/184: creat d6/d13/d18/d3c/f3d x:0 0 0 2024-04-05T21:26:26.598 INFO:tasks.workunit.client.1.smithi188.stdout:6/185: creat d6/d13/d18/d19/d2c/f3e x:0 0 0 2024-04-05T21:26:26.599 INFO:tasks.workunit.client.1.smithi188.stdout:6/186: mkdir d6/d13/d18/d19/d2c/d3f 0 2024-04-05T21:26:26.604 INFO:tasks.workunit.client.1.smithi188.stdout:6/187: dwrite d6/f24 [0,4194304] 0 2024-04-05T21:26:26.606 INFO:tasks.workunit.client.1.smithi188.stdout:6/188: write d6/d13/f1d [2142357,57087] 0 2024-04-05T21:26:26.608 INFO:tasks.workunit.client.1.smithi188.stdout:6/189: write d6/f11 [1744038,805] 0 2024-04-05T21:26:26.770 INFO:tasks.workunit.client.1.smithi188.stdout:7/260: sync 2024-04-05T21:26:26.770 INFO:tasks.workunit.client.1.smithi188.stdout:0/84: sync 2024-04-05T21:26:26.771 INFO:tasks.workunit.client.1.smithi188.stdout:7/261: stat d3/d4/db/d2e 0 2024-04-05T21:26:26.775 INFO:tasks.workunit.client.1.smithi188.stdout:7/262: mkdir d3/d4/db/d55 0 2024-04-05T21:26:26.775 INFO:tasks.workunit.client.1.smithi188.stdout:7/263: stat d3/d4/d1d/f2b 0 2024-04-05T21:26:26.776 INFO:tasks.workunit.client.1.smithi188.stdout:0/85: link d4/d6/d13/f14 d4/d8/db/f1d 0 2024-04-05T21:26:26.779 INFO:tasks.workunit.client.1.smithi188.stdout:0/86: write d4/d6/d13/f14 [1282315,106096] 0 2024-04-05T21:26:26.781 INFO:tasks.workunit.client.1.smithi188.stdout:7/264: rename l2 to d3/d4/db/d10/d2f/l56 0 2024-04-05T21:26:26.781 INFO:tasks.workunit.client.1.smithi188.stdout:7/265: dread - d3/d2c/f3f zero size 2024-04-05T21:26:26.782 INFO:tasks.workunit.client.1.smithi188.stdout:7/266: chown d3/d2c/f4c 48658562 1 2024-04-05T21:26:26.783 INFO:tasks.workunit.client.1.smithi188.stdout:0/87: link d4/f19 d4/d8/d10/f1e 0 2024-04-05T21:26:26.785 INFO:tasks.workunit.client.1.smithi188.stdout:7/267: symlink d3/d4/db/d10/d2f/l57 0 2024-04-05T21:26:26.785 INFO:tasks.workunit.client.1.smithi188.stdout:0/88: creat d4/d6/f1f x:0 0 0 2024-04-05T21:26:26.786 INFO:tasks.workunit.client.1.smithi188.stdout:0/89: chown f3 0 1 2024-04-05T21:26:26.812 INFO:tasks.workunit.client.1.smithi188.stdout:0/90: creat d4/d8/d10/f20 x:0 0 0 2024-04-05T21:26:26.813 INFO:tasks.workunit.client.1.smithi188.stdout:0/91: chown d4/d6/f1f 14769031 1 2024-04-05T21:26:26.816 INFO:tasks.workunit.client.1.smithi188.stdout:0/92: rmdir d4/d8 39 2024-04-05T21:26:26.817 INFO:tasks.workunit.client.1.smithi188.stdout:6/190: truncate d6/f2b 283936 0 2024-04-05T21:26:26.818 INFO:tasks.workunit.client.1.smithi188.stdout:6/191: dread - d6/d13/d18/d19/f30 zero size 2024-04-05T21:26:26.947 INFO:tasks.workunit.client.1.smithi188.stdout:5/124: sync 2024-04-05T21:26:26.947 INFO:tasks.workunit.client.1.smithi188.stdout:9/105: sync 2024-04-05T21:26:26.947 INFO:tasks.workunit.client.1.smithi188.stdout:1/151: sync 2024-04-05T21:26:26.947 INFO:tasks.workunit.client.1.smithi188.stdout:3/188: sync 2024-04-05T21:26:26.948 INFO:tasks.workunit.client.1.smithi188.stdout:5/125: chown d6/d17/d24 67733220 1 2024-04-05T21:26:26.949 INFO:tasks.workunit.client.1.smithi188.stdout:3/189: truncate d5/d7/f20 1777666 0 2024-04-05T21:26:26.950 INFO:tasks.workunit.client.1.smithi188.stdout:3/190: write d5/dc/d2f/f38 [133589,2650] 0 2024-04-05T21:26:27.067 INFO:tasks.workunit.client.1.smithi188.stdout:9/106: write f1 [5359242,40312] 0 2024-04-05T21:26:27.068 INFO:tasks.workunit.client.1.smithi188.stdout:9/107: stat dd/d11/c1d 0 2024-04-05T21:26:27.068 INFO:tasks.workunit.client.1.smithi188.stdout:5/126: creat d6/d17/f2f x:0 0 0 2024-04-05T21:26:27.070 INFO:tasks.workunit.client.1.smithi188.stdout:3/191: rename d5/l1c to d5/d39/l46 0 2024-04-05T21:26:27.071 INFO:tasks.workunit.client.1.smithi188.stdout:8/89: sync 2024-04-05T21:26:27.072 INFO:tasks.workunit.client.1.smithi188.stdout:3/192: write d5/f37 [1021828,4338] 0 2024-04-05T21:26:27.073 INFO:tasks.workunit.client.1.smithi188.stdout:0/93: mkdir d4/d8/db/d21 0 2024-04-05T21:26:27.073 INFO:tasks.workunit.client.1.smithi188.stdout:9/108: dread f5 [0,4194304] 0 2024-04-05T21:26:27.075 INFO:tasks.workunit.client.1.smithi188.stdout:6/192: mknod d6/d13/d18/c40 0 2024-04-05T21:26:27.076 INFO:tasks.workunit.client.1.smithi188.stdout:3/193: truncate d5/d7/f36 978079 0 2024-04-05T21:26:27.076 INFO:tasks.workunit.client.1.smithi188.stdout:9/109: dread - dd/d11/f21 zero size 2024-04-05T21:26:27.079 INFO:tasks.workunit.client.1.smithi188.stdout:7/268: getdents d3/d4/db/d10/d2f 0 2024-04-05T21:26:27.081 INFO:tasks.workunit.client.1.smithi188.stdout:5/127: dread d6/d17/f1f [0,4194304] 0 2024-04-05T21:26:27.085 INFO:tasks.workunit.client.1.smithi188.stdout:7/269: truncate d3/d4/db/d2e/d3b/f4d 302279 0 2024-04-05T21:26:27.086 INFO:tasks.workunit.client.1.smithi188.stdout:7/270: fsync d3/f3e 0 2024-04-05T21:26:27.087 INFO:tasks.workunit.client.1.smithi188.stdout:3/194: dwrite d5/d35/f40 [0,4194304] 0 2024-04-05T21:26:27.088 INFO:tasks.workunit.client.1.smithi188.stdout:7/271: stat d3/d4/db/d10/f14 0 2024-04-05T21:26:27.089 INFO:tasks.workunit.client.1.smithi188.stdout:3/195: write f4 [7739924,61476] 0 2024-04-05T21:26:27.090 INFO:tasks.workunit.client.1.smithi188.stdout:3/196: fdatasync d5/dc/f2d 0 2024-04-05T21:26:27.091 INFO:tasks.workunit.client.1.smithi188.stdout:6/193: symlink d6/l41 0 2024-04-05T21:26:27.092 INFO:tasks.workunit.client.1.smithi188.stdout:5/128: mknod d6/d17/c30 0 2024-04-05T21:26:27.094 INFO:tasks.workunit.client.1.smithi188.stdout:1/152: getdents d1/dc/d16 0 2024-04-05T21:26:27.094 INFO:tasks.workunit.client.1.smithi188.stdout:0/94: dread d4/d8/db/f1d [0,4194304] 0 2024-04-05T21:26:27.095 INFO:tasks.workunit.client.1.smithi188.stdout:1/153: readlink d1/l32 0 2024-04-05T21:26:27.095 INFO:tasks.workunit.client.1.smithi188.stdout:9/110: link dd/f16 dd/d17/f23 0 2024-04-05T21:26:27.097 INFO:tasks.workunit.client.1.smithi188.stdout:9/111: chown f1 3675 1 2024-04-05T21:26:27.097 INFO:tasks.workunit.client.1.smithi188.stdout:8/90: link d5/dc/c13 d5/dc/c1b 0 2024-04-05T21:26:27.098 INFO:tasks.workunit.client.1.smithi188.stdout:7/272: write d3/d4/db/f4a [33617,36854] 0 2024-04-05T21:26:27.098 INFO:tasks.workunit.client.1.smithi188.stdout:9/112: chown fb 9471 1 2024-04-05T21:26:27.100 INFO:tasks.workunit.client.1.smithi188.stdout:3/197: dread d5/f6 [0,4194304] 0 2024-04-05T21:26:27.100 INFO:tasks.workunit.client.1.smithi188.stdout:9/113: fdatasync dd/d11/d1e/f19 0 2024-04-05T21:26:27.101 INFO:tasks.workunit.client.1.smithi188.stdout:1/154: creat d1/dc/d16/f33 x:0 0 0 2024-04-05T21:26:27.101 INFO:tasks.workunit.client.1.smithi188.stdout:7/273: write d3/d4/db/d2e/f35 [212377,41099] 0 2024-04-05T21:26:27.102 INFO:tasks.workunit.client.1.smithi188.stdout:7/274: truncate d3/f3e 591909 0 2024-04-05T21:26:27.106 INFO:tasks.workunit.client.1.smithi188.stdout:0/95: rename d4/f19 to d4/d8/d10/f22 0 2024-04-05T21:26:27.106 INFO:tasks.workunit.client.1.smithi188.stdout:0/96: chown d4/d8/db/f1d 198198719 1 2024-04-05T21:26:27.108 INFO:tasks.workunit.client.1.smithi188.stdout:1/155: dread d1/dc/f19 [0,4194304] 0 2024-04-05T21:26:27.114 INFO:tasks.workunit.client.1.smithi188.stdout:2/183: sync 2024-04-05T21:26:27.114 INFO:tasks.workunit.client.1.smithi188.stdout:4/258: sync 2024-04-05T21:26:27.118 INFO:tasks.workunit.client.1.smithi188.stdout:4/259: dread d0/d2/d17/f44 [0,4194304] 0 2024-04-05T21:26:27.150 INFO:tasks.workunit.client.1.smithi188.stdout:9/114: fsync f5 0 2024-04-05T21:26:27.180 INFO:tasks.workunit.client.1.smithi188.stdout:3/198: mknod d5/d2b/d43/c47 0 2024-04-05T21:26:27.181 INFO:tasks.workunit.client.1.smithi188.stdout:3/199: truncate d5/d7/f30 877055 0 2024-04-05T21:26:27.183 INFO:tasks.workunit.client.1.smithi188.stdout:0/97: mkdir d4/d6/d13/d23 0 2024-04-05T21:26:27.186 INFO:tasks.workunit.client.1.smithi188.stdout:6/194: truncate d6/f11 1397195 0 2024-04-05T21:26:27.188 INFO:tasks.workunit.client.1.smithi188.stdout:5/129: truncate d6/d17/f1f 1813529 0 2024-04-05T21:26:27.189 INFO:tasks.workunit.client.1.smithi188.stdout:3/200: dwrite d5/df/f15 [0,4194304] 0 2024-04-05T21:26:27.190 INFO:tasks.workunit.client.1.smithi188.stdout:9/115: symlink dd/d17/l24 0 2024-04-05T21:26:27.191 INFO:tasks.workunit.client.1.smithi188.stdout:7/275: mknod d3/d4/c58 0 2024-04-05T21:26:27.191 INFO:tasks.workunit.client.1.smithi188.stdout:9/116: fdatasync dd/d11/d1e/f19 0 2024-04-05T21:26:27.193 INFO:tasks.workunit.client.1.smithi188.stdout:8/91: rename d5/f18 to d5/f1c 0 2024-04-05T21:26:27.193 INFO:tasks.workunit.client.1.smithi188.stdout:9/117: readlink dd/d11/d1e/l10 0 2024-04-05T21:26:27.195 INFO:tasks.workunit.client.1.smithi188.stdout:0/98: fsync d4/d8/d10/f1e 0 2024-04-05T21:26:27.196 INFO:tasks.workunit.client.1.smithi188.stdout:4/260: dwrite d0/d14/f38 [0,4194304] 0 2024-04-05T21:26:27.197 INFO:tasks.workunit.client.1.smithi188.stdout:2/184: mknod d5/db/c30 0 2024-04-05T21:26:27.199 INFO:tasks.workunit.client.1.smithi188.stdout:4/261: fsync d0/d2/f26 0 2024-04-05T21:26:27.204 INFO:tasks.workunit.client.1.smithi188.stdout:6/195: dread d6/fb [0,4194304] 0 2024-04-05T21:26:27.205 INFO:tasks.workunit.client.1.smithi188.stdout:6/196: chown d6/d13/d18/c40 82532606 1 2024-04-05T21:26:27.209 INFO:tasks.workunit.client.1.smithi188.stdout:3/201: fsync d5/fb 0 2024-04-05T21:26:27.209 INFO:tasks.workunit.client.1.smithi188.stdout:8/92: creat d5/dc/f1d x:0 0 0 2024-04-05T21:26:27.211 INFO:tasks.workunit.client.1.smithi188.stdout:3/202: fdatasync d5/fa 0 2024-04-05T21:26:27.211 INFO:tasks.workunit.client.1.smithi188.stdout:1/156: creat d1/dc/d16/d2d/f34 x:0 0 0 2024-04-05T21:26:27.213 INFO:tasks.workunit.client.1.smithi188.stdout:3/203: dread - d5/df/f33 zero size 2024-04-05T21:26:27.213 INFO:tasks.workunit.client.1.smithi188.stdout:1/157: write d1/f24 [515199,34743] 0 2024-04-05T21:26:27.216 INFO:tasks.workunit.client.1.smithi188.stdout:4/262: unlink d0/d14/d15/d42/f4c 0 2024-04-05T21:26:27.216 INFO:tasks.workunit.client.1.smithi188.stdout:1/158: dread d1/dc/f19 [0,4194304] 0 2024-04-05T21:26:27.217 INFO:tasks.workunit.client.1.smithi188.stdout:6/197: symlink d6/d13/d18/d19/l42 0 2024-04-05T21:26:27.217 INFO:tasks.workunit.client.1.smithi188.stdout:4/263: chown d0/d14/c32 141847284 1 2024-04-05T21:26:27.218 INFO:tasks.workunit.client.1.smithi188.stdout:6/198: write d6/d7/f15 [605670,57468] 0 2024-04-05T21:26:27.219 INFO:tasks.workunit.client.1.smithi188.stdout:0/99: mkdir d4/d6/d13/d23/d24 0 2024-04-05T21:26:27.220 INFO:tasks.workunit.client.1.smithi188.stdout:6/199: write d6/f2f [739682,79406] 0 2024-04-05T21:26:27.221 INFO:tasks.workunit.client.1.smithi188.stdout:0/100: dread - d4/d8/d10/f20 zero size 2024-04-05T21:26:27.222 INFO:tasks.workunit.client.1.smithi188.stdout:2/185: mknod d5/c31 0 2024-04-05T21:26:27.222 INFO:tasks.workunit.client.1.smithi188.stdout:0/101: readlink d4/d8/d10/l1a 0 2024-04-05T21:26:27.223 INFO:tasks.workunit.client.1.smithi188.stdout:5/130: rmdir d6/d17/d2e 0 2024-04-05T21:26:27.226 INFO:tasks.workunit.client.1.smithi188.stdout:1/159: rename d1/c6 to d1/d4/c35 0 2024-04-05T21:26:27.227 INFO:tasks.workunit.client.1.smithi188.stdout:2/186: dread d5/f14 [0,4194304] 0 2024-04-05T21:26:27.228 INFO:tasks.workunit.client.1.smithi188.stdout:4/264: unlink d0/d2/c12 0 2024-04-05T21:26:27.230 INFO:tasks.workunit.client.1.smithi188.stdout:9/118: dwrite dd/f16 [0,4194304] 0 2024-04-05T21:26:27.230 INFO:tasks.workunit.client.1.smithi188.stdout:4/265: readlink d0/d14/d15/l2c 0 2024-04-05T21:26:27.232 INFO:tasks.workunit.client.1.smithi188.stdout:3/204: mkdir d5/d48 0 2024-04-05T21:26:27.233 INFO:tasks.workunit.client.1.smithi188.stdout:5/131: write d6/d17/d24/f2b [217791,63919] 0 2024-04-05T21:26:27.235 INFO:tasks.workunit.client.1.smithi188.stdout:1/160: creat d1/d4/f36 x:0 0 0 2024-04-05T21:26:27.235 INFO:tasks.workunit.client.1.smithi188.stdout:1/161: dread - d1/d4/d13/f18 zero size 2024-04-05T21:26:27.236 INFO:tasks.workunit.client.1.smithi188.stdout:1/162: readlink d1/d4/d13/l8 0 2024-04-05T21:26:27.237 INFO:tasks.workunit.client.1.smithi188.stdout:8/93: link d5/f8 d5/f1e 0 2024-04-05T21:26:27.237 INFO:tasks.workunit.client.1.smithi188.stdout:1/163: chown d1/d4/f36 7960072 1 2024-04-05T21:26:27.239 INFO:tasks.workunit.client.1.smithi188.stdout:4/266: symlink d0/d14/d2f/l5f 0 2024-04-05T21:26:27.240 INFO:tasks.workunit.client.1.smithi188.stdout:0/102: mkdir d4/d25 0 2024-04-05T21:26:27.242 INFO:tasks.workunit.client.1.smithi188.stdout:3/205: rename d5/dc/d2f/f38 to d5/dc/f49 0 2024-04-05T21:26:27.245 INFO:tasks.workunit.client.1.smithi188.stdout:1/164: fdatasync d1/dc/f1b 0 2024-04-05T21:26:27.247 INFO:tasks.workunit.client.1.smithi188.stdout:4/267: creat d0/d14/d15/d1b/f60 x:0 0 0 2024-04-05T21:26:27.250 INFO:tasks.workunit.client.1.smithi188.stdout:5/132: truncate d6/d15/d1b/f2a 2715556 0 2024-04-05T21:26:27.250 INFO:tasks.workunit.client.1.smithi188.stdout:5/133: chown d6/d17/d23 75 1 2024-04-05T21:26:27.251 INFO:tasks.workunit.client.1.smithi188.stdout:5/134: chown d6/f2d 8420 1 2024-04-05T21:26:27.251 INFO:tasks.workunit.client.1.smithi188.stdout:2/187: link d5/db/dd/c13 d5/db/dd/d24/c32 0 2024-04-05T21:26:27.253 INFO:tasks.workunit.client.1.smithi188.stdout:1/165: mknod d1/d4/d13/c37 0 2024-04-05T21:26:27.253 INFO:tasks.workunit.client.1.smithi188.stdout:3/206: dread d5/dc/f2a [0,4194304] 0 2024-04-05T21:26:27.255 INFO:tasks.workunit.client.1.smithi188.stdout:4/268: unlink d0/d2/d17/f18 0 2024-04-05T21:26:27.263 INFO:tasks.workunit.client.1.smithi188.stdout:4/269: dwrite d0/d14/f1c [0,4194304] 0 2024-04-05T21:26:27.264 INFO:tasks.workunit.client.1.smithi188.stdout:2/188: mknod d5/c33 0 2024-04-05T21:26:27.265 INFO:tasks.workunit.client.1.smithi188.stdout:2/189: fsync d5/db/dd/d24/f2e 0 2024-04-05T21:26:27.267 INFO:tasks.workunit.client.1.smithi188.stdout:6/200: truncate f5 4111996 0 2024-04-05T21:26:27.267 INFO:tasks.workunit.client.1.smithi188.stdout:2/190: truncate d5/f2f 4865502 0 2024-04-05T21:26:27.268 INFO:tasks.workunit.client.1.smithi188.stdout:0/103: getdents d4/d8/d10 0 2024-04-05T21:26:27.270 INFO:tasks.workunit.client.1.smithi188.stdout:4/270: mknod d0/d2/d17/d3a/c61 0 2024-04-05T21:26:27.271 INFO:tasks.workunit.client.1.smithi188.stdout:5/135: rename d6/lc to d6/l31 0 2024-04-05T21:26:27.282 INFO:tasks.workunit.client.1.smithi188.stdout:8/94: dwrite d5/dc/f10 [0,4194304] 0 2024-04-05T21:26:27.282 INFO:tasks.workunit.client.1.smithi188.stdout:6/201: fsync d6/d13/d18/d19/d2c/f32 0 2024-04-05T21:26:27.287 INFO:tasks.workunit.client.1.smithi188.stdout:3/207: write d5/d7/f25 [657951,20937] 0 2024-04-05T21:26:27.289 INFO:tasks.workunit.client.1.smithi188.stdout:0/104: dwrite d4/d8/db/f1d [4194304,4194304] 0 2024-04-05T21:26:27.299 INFO:tasks.workunit.client.1.smithi188.stdout:1/166: getdents d1/dc/d16/d21 0 2024-04-05T21:26:27.302 INFO:tasks.workunit.client.1.smithi188.stdout:3/208: symlink d5/d2b/l4a 0 2024-04-05T21:26:27.305 INFO:tasks.workunit.client.1.smithi188.stdout:0/105: creat d4/d8/f26 x:0 0 0 2024-04-05T21:26:27.306 INFO:tasks.workunit.client.1.smithi188.stdout:6/202: link d6/d7/fc d6/d28/f43 0 2024-04-05T21:26:27.306 INFO:tasks.workunit.client.1.smithi188.stdout:5/136: getdents d6/d15 0 2024-04-05T21:26:27.307 INFO:tasks.workunit.client.1.smithi188.stdout:1/167: rmdir d1 39 2024-04-05T21:26:27.308 INFO:tasks.workunit.client.1.smithi188.stdout:5/137: readlink d6/l28 0 2024-04-05T21:26:27.308 INFO:tasks.workunit.client.1.smithi188.stdout:2/191: dwrite d5/f22 [4194304,4194304] 0 2024-04-05T21:26:27.308 INFO:tasks.workunit.client.1.smithi188.stdout:0/106: readlink d4/d8/d10/l1a 0 2024-04-05T21:26:27.311 INFO:tasks.workunit.client.1.smithi188.stdout:4/271: rename d0/d2/d2a/d34/f46 to d0/f62 0 2024-04-05T21:26:27.313 INFO:tasks.workunit.client.1.smithi188.stdout:5/138: write d6/f19 [511279,80513] 0 2024-04-05T21:26:27.315 INFO:tasks.workunit.client.1.smithi188.stdout:0/107: dread - d4/d8/d10/f20 zero size 2024-04-05T21:26:27.317 INFO:tasks.workunit.client.1.smithi188.stdout:1/168: chown d1/d4/d13/c37 123 1 2024-04-05T21:26:27.320 INFO:tasks.workunit.client.1.smithi188.stdout:1/169: write d1/dc/f19 [772046,103055] 0 2024-04-05T21:26:27.323 INFO:tasks.workunit.client.1.smithi188.stdout:6/203: link d6/d7/f15 d6/d13/d18/d3c/f44 0 2024-04-05T21:26:27.328 INFO:tasks.workunit.client.1.smithi188.stdout:1/170: write d1/dc/f19 [1700804,98834] 0 2024-04-05T21:26:27.329 INFO:tasks.workunit.client.1.smithi188.stdout:5/139: dwrite d6/d17/d24/f2b [0,4194304] 0 2024-04-05T21:26:27.329 INFO:tasks.workunit.client.1.smithi188.stdout:4/272: dread d0/f5 [0,4194304] 0 2024-04-05T21:26:27.330 INFO:tasks.workunit.client.1.smithi188.stdout:8/95: rename d5/dc/f10 to d5/f1f 0 2024-04-05T21:26:27.333 INFO:tasks.workunit.client.1.smithi188.stdout:6/204: truncate d6/d7/f16 4508872 0 2024-04-05T21:26:27.338 INFO:tasks.workunit.client.1.smithi188.stdout:2/192: dread d5/db/f20 [0,4194304] 0 2024-04-05T21:26:27.339 INFO:tasks.workunit.client.1.smithi188.stdout:3/209: dread f4 [0,4194304] 0 2024-04-05T21:26:27.341 INFO:tasks.workunit.client.1.smithi188.stdout:5/140: write d6/d17/f2f [677927,44713] 0 2024-04-05T21:26:27.342 INFO:tasks.workunit.client.1.smithi188.stdout:4/273: creat d0/d14/d15/d42/f63 x:0 0 0 2024-04-05T21:26:27.344 INFO:tasks.workunit.client.1.smithi188.stdout:5/141: chown d6/cf 20 1 2024-04-05T21:26:27.352 INFO:tasks.workunit.client.1.smithi188.stdout:6/205: dwrite d6/d13/d18/f36 [0,4194304] 0 2024-04-05T21:26:27.353 INFO:tasks.workunit.client.1.smithi188.stdout:1/171: creat d1/dc/d16/d21/f38 x:0 0 0 2024-04-05T21:26:27.357 INFO:tasks.workunit.client.1.smithi188.stdout:1/172: dread - d1/dc/d16/d2d/f34 zero size 2024-04-05T21:26:27.358 INFO:tasks.workunit.client.1.smithi188.stdout:1/173: fdatasync d1/dc/f19 0 2024-04-05T21:26:27.358 INFO:tasks.workunit.client.1.smithi188.stdout:0/108: rename d4/d8/d10/f22 to d4/f27 0 2024-04-05T21:26:27.359 INFO:tasks.workunit.client.1.smithi188.stdout:0/109: fdatasync d4/d6/f12 0 2024-04-05T21:26:27.362 INFO:tasks.workunit.client.1.smithi188.stdout:5/142: mknod d6/d15/d1b/c32 0 2024-04-05T21:26:27.362 INFO:tasks.workunit.client.1.smithi188.stdout:3/210: dwrite d5/d7/f20 [0,4194304] 0 2024-04-05T21:26:27.370 INFO:tasks.workunit.client.1.smithi188.stdout:8/96: creat d5/f20 x:0 0 0 2024-04-05T21:26:27.370 INFO:tasks.workunit.client.1.smithi188.stdout:0/110: chown d4/d6/f12 20 1 2024-04-05T21:26:27.370 INFO:tasks.workunit.client.1.smithi188.stdout:6/206: fdatasync d6/f24 0 2024-04-05T21:26:27.370 INFO:tasks.workunit.client.1.smithi188.stdout:2/193: dwrite d5/f2f [0,4194304] 0 2024-04-05T21:26:27.375 INFO:tasks.workunit.client.1.smithi188.stdout:2/194: dread d5/db/dd/f1c [0,4194304] 0 2024-04-05T21:26:27.377 INFO:tasks.workunit.client.1.smithi188.stdout:1/174: stat d1/dc/d16/l2a 0 2024-04-05T21:26:27.378 INFO:tasks.workunit.client.1.smithi188.stdout:0/111: write f3 [619835,81604] 0 2024-04-05T21:26:27.380 INFO:tasks.workunit.client.1.smithi188.stdout:6/207: mkdir d6/d28/d45 0 2024-04-05T21:26:27.384 INFO:tasks.workunit.client.1.smithi188.stdout:3/211: mknod d5/df/c4b 0 2024-04-05T21:26:27.385 INFO:tasks.workunit.client.1.smithi188.stdout:3/212: readlink d5/d7/l26 0 2024-04-05T21:26:27.442 INFO:tasks.workunit.client.1.smithi188.stdout:4/274: link d0/d14/d15/fb d0/f64 0 2024-04-05T21:26:27.443 INFO:tasks.workunit.client.1.smithi188.stdout:1/175: creat d1/d7/f39 x:0 0 0 2024-04-05T21:26:27.445 INFO:tasks.workunit.client.1.smithi188.stdout:4/275: chown d0/d2/d2a/d34 118 1 2024-04-05T21:26:27.445 INFO:tasks.workunit.client.1.smithi188.stdout:6/208: dwrite d6/f3b [0,4194304] 0 2024-04-05T21:26:27.446 INFO:tasks.workunit.client.1.smithi188.stdout:8/97: mkdir d5/d21 0 2024-04-05T21:26:27.448 INFO:tasks.workunit.client.1.smithi188.stdout:3/213: creat d5/dc/d2f/f4c x:0 0 0 2024-04-05T21:26:27.448 INFO:tasks.workunit.client.1.smithi188.stdout:1/176: mknod d1/d4/c3a 0 2024-04-05T21:26:27.449 INFO:tasks.workunit.client.1.smithi188.stdout:0/112: rmdir d4/d25 0 2024-04-05T21:26:27.453 INFO:tasks.workunit.client.1.smithi188.stdout:5/143: dread d6/d15/d1b/f2a [0,4194304] 0 2024-04-05T21:26:27.456 INFO:tasks.workunit.client.1.smithi188.stdout:1/177: rmdir d1/d2b 0 2024-04-05T21:26:27.457 INFO:tasks.workunit.client.1.smithi188.stdout:2/195: dwrite d5/f14 [0,4194304] 0 2024-04-05T21:26:27.458 INFO:tasks.workunit.client.1.smithi188.stdout:6/209: getdents d6/d13/d18/d19/d2c/d34 0 2024-04-05T21:26:27.459 INFO:tasks.workunit.client.1.smithi188.stdout:4/276: truncate d0/d14/d15/d1b/f1d 709184 0 2024-04-05T21:26:27.460 INFO:tasks.workunit.client.1.smithi188.stdout:6/210: fdatasync d6/f2f 0 2024-04-05T21:26:27.460 INFO:tasks.workunit.client.1.smithi188.stdout:5/144: symlink d6/d15/l33 0 2024-04-05T21:26:27.461 INFO:tasks.workunit.client.1.smithi188.stdout:8/98: rename d5/dc/c14 to d5/c22 0 2024-04-05T21:26:27.462 INFO:tasks.workunit.client.1.smithi188.stdout:0/113: dread d4/d6/d13/f14 [4194304,4194304] 0 2024-04-05T21:26:27.464 INFO:tasks.workunit.client.1.smithi188.stdout:6/211: write d6/d7/fd [4366706,52028] 0 2024-04-05T21:26:27.465 INFO:tasks.workunit.client.1.smithi188.stdout:2/196: mkdir d5/db/d34 0 2024-04-05T21:26:27.466 INFO:tasks.workunit.client.1.smithi188.stdout:6/212: fsync d6/d13/f1d 0 2024-04-05T21:26:27.467 INFO:tasks.workunit.client.1.smithi188.stdout:3/214: getdents d5/d2b/d43 0 2024-04-05T21:26:27.469 INFO:tasks.workunit.client.1.smithi188.stdout:0/114: creat d4/d8/db/f28 x:0 0 0 2024-04-05T21:26:27.474 INFO:tasks.workunit.client.1.smithi188.stdout:2/197: readlink d5/db/dd/d24/l2d 0 2024-04-05T21:26:27.474 INFO:tasks.workunit.client.1.smithi188.stdout:4/277: symlink d0/d14/d15/d53/l65 0 2024-04-05T21:26:27.475 INFO:tasks.workunit.client.1.smithi188.stdout:3/215: dread d5/f6 [0,4194304] 0 2024-04-05T21:26:27.476 INFO:tasks.workunit.client.1.smithi188.stdout:2/198: write d5/f14 [2151507,91531] 0 2024-04-05T21:26:27.482 INFO:tasks.workunit.client.1.smithi188.stdout:1/178: rename d1/dc/d16/d21/f38 to d1/dc/d16/f3b 0 2024-04-05T21:26:27.482 INFO:tasks.workunit.client.1.smithi188.stdout:0/115: dwrite d4/d8/f26 [0,4194304] 0 2024-04-05T21:26:27.482 INFO:tasks.workunit.client.1.smithi188.stdout:4/278: chown d0/d14/d15/d1b/l3c 74 1 2024-04-05T21:26:27.486 INFO:tasks.workunit.client.1.smithi188.stdout:6/213: mknod d6/d13/d18/d19/d2c/d3f/c46 0 2024-04-05T21:26:27.488 INFO:tasks.workunit.client.1.smithi188.stdout:2/199: fsync d5/db/dd/d24/f28 0 2024-04-05T21:26:27.490 INFO:tasks.workunit.client.1.smithi188.stdout:5/145: creat d6/d15/d1b/f34 x:0 0 0 2024-04-05T21:26:27.491 INFO:tasks.workunit.client.1.smithi188.stdout:6/214: write d6/d13/d18/d19/f30 [845629,36675] 0 2024-04-05T21:26:27.492 INFO:tasks.workunit.client.1.smithi188.stdout:5/146: chown d6/d15/d1b 2 1 2024-04-05T21:26:27.492 INFO:tasks.workunit.client.1.smithi188.stdout:2/200: chown d5/l21 31 1 2024-04-05T21:26:27.492 INFO:tasks.workunit.client.1.smithi188.stdout:3/216: dwrite d5/df/f33 [0,4194304] 0 2024-04-05T21:26:27.495 INFO:tasks.workunit.client.1.smithi188.stdout:5/147: fdatasync d6/d15/d1b/f34 0 2024-04-05T21:26:27.496 INFO:tasks.workunit.client.1.smithi188.stdout:0/116: link d4/d6/d13/c17 d4/d6/d13/d23/d24/c29 0 2024-04-05T21:26:27.496 INFO:tasks.workunit.client.1.smithi188.stdout:6/215: creat d6/d13/d38/f47 x:0 0 0 2024-04-05T21:26:27.497 INFO:tasks.workunit.client.1.smithi188.stdout:3/217: dread - d5/dc/f3d zero size 2024-04-05T21:26:27.497 INFO:tasks.workunit.client.1.smithi188.stdout:2/201: write d5/f2f [5820384,35486] 0 2024-04-05T21:26:27.502 INFO:tasks.workunit.client.1.smithi188.stdout:3/218: mknod d5/d39/c4d 0 2024-04-05T21:26:27.503 INFO:tasks.workunit.client.1.smithi188.stdout:2/202: mkdir d5/db/d35 0 2024-04-05T21:26:27.503 INFO:tasks.workunit.client.1.smithi188.stdout:0/117: truncate d4/d8/d10/f20 456020 0 2024-04-05T21:26:27.503 INFO:tasks.workunit.client.1.smithi188.stdout:1/179: dwrite d1/dc/f15 [0,4194304] 0 2024-04-05T21:26:27.507 INFO:tasks.workunit.client.1.smithi188.stdout:4/279: dread d0/d2/d17/f44 [0,4194304] 0 2024-04-05T21:26:27.507 INFO:tasks.workunit.client.1.smithi188.stdout:5/148: write d6/f2d [4263264,1393] 0 2024-04-05T21:26:27.508 INFO:tasks.workunit.client.1.smithi188.stdout:3/219: mkdir d5/d35/d4e 0 2024-04-05T21:26:27.514 INFO:tasks.workunit.client.1.smithi188.stdout:3/220: read d5/d7/f30 [840597,69095] 0 2024-04-05T21:26:27.515 INFO:tasks.workunit.client.1.smithi188.stdout:1/180: creat d1/dc/d16/f3c x:0 0 0 2024-04-05T21:26:27.519 INFO:tasks.workunit.client.1.smithi188.stdout:2/203: truncate d5/db/f20 878795 0 2024-04-05T21:26:27.520 INFO:tasks.workunit.client.1.smithi188.stdout:3/221: symlink d5/l4f 0 2024-04-05T21:26:27.524 INFO:tasks.workunit.client.1.smithi188.stdout:5/149: dwrite d6/f19 [0,4194304] 0 2024-04-05T21:26:27.525 INFO:tasks.workunit.client.1.smithi188.stdout:6/216: dread d6/fa [0,4194304] 0 2024-04-05T21:26:27.525 INFO:tasks.workunit.client.1.smithi188.stdout:2/204: mkdir d5/db/dd/d36 0 2024-04-05T21:26:27.529 INFO:tasks.workunit.client.1.smithi188.stdout:0/118: dwrite d4/d8/db/f11 [0,4194304] 0 2024-04-05T21:26:27.530 INFO:tasks.workunit.client.1.smithi188.stdout:1/181: fsync d1/dc/d16/d21/d27/f29 0 2024-04-05T21:26:27.531 INFO:tasks.workunit.client.1.smithi188.stdout:4/280: dread d0/d2/d17/f4e [0,4194304] 0 2024-04-05T21:26:27.533 INFO:tasks.workunit.client.1.smithi188.stdout:3/222: rename d5/c2c to d5/d48/c50 0 2024-04-05T21:26:27.534 INFO:tasks.workunit.client.1.smithi188.stdout:5/150: mkdir d6/d17/d23/d35 0 2024-04-05T21:26:27.536 INFO:tasks.workunit.client.1.smithi188.stdout:1/182: rmdir d1/d4/d13 39 2024-04-05T21:26:27.537 INFO:tasks.workunit.client.1.smithi188.stdout:4/281: creat d0/d14/d15/d53/f66 x:0 0 0 2024-04-05T21:26:27.538 INFO:tasks.workunit.client.1.smithi188.stdout:6/217: mkdir d6/d13/d48 0 2024-04-05T21:26:27.539 INFO:tasks.workunit.client.1.smithi188.stdout:2/205: getdents d5/db/dd/d36 0 2024-04-05T21:26:27.540 INFO:tasks.workunit.client.1.smithi188.stdout:2/206: dread - d5/db/dd/d24/f2e zero size 2024-04-05T21:26:27.540 INFO:tasks.workunit.client.1.smithi188.stdout:6/218: truncate d6/d7/fc 642339 0 2024-04-05T21:26:27.541 INFO:tasks.workunit.client.1.smithi188.stdout:0/119: link d4/d8/fa d4/d6/d13/f2a 0 2024-04-05T21:26:27.543 INFO:tasks.workunit.client.1.smithi188.stdout:3/223: mkdir d5/d51 0 2024-04-05T21:26:27.548 INFO:tasks.workunit.client.1.smithi188.stdout:2/207: rename d5/db/dd/c27 to d5/db/dd/d24/c37 0 2024-04-05T21:26:27.550 INFO:tasks.workunit.client.1.smithi188.stdout:6/219: fsync d6/d13/d18/d3c/f44 0 2024-04-05T21:26:27.553 INFO:tasks.workunit.client.1.smithi188.stdout:3/224: link d5/d7/f25 d5/d35/f52 0 2024-04-05T21:26:27.554 INFO:tasks.workunit.client.1.smithi188.stdout:3/225: dread - d5/dc/f3d zero size 2024-04-05T21:26:27.555 INFO:tasks.workunit.client.1.smithi188.stdout:2/208: mknod d5/db/dd/c38 0 2024-04-05T21:26:27.555 INFO:tasks.workunit.client.1.smithi188.stdout:2/209: dread - d5/db/dd/d24/f2e zero size 2024-04-05T21:26:27.556 INFO:tasks.workunit.client.1.smithi188.stdout:1/183: getdents d1/dc/d16/d21/d27 0 2024-04-05T21:26:27.558 INFO:tasks.workunit.client.1.smithi188.stdout:1/184: truncate d1/dc/d16/d21/f26 1047880 0 2024-04-05T21:26:27.559 INFO:tasks.workunit.client.1.smithi188.stdout:4/282: creat d0/d2/d4f/f67 x:0 0 0 2024-04-05T21:26:27.560 INFO:tasks.workunit.client.1.smithi188.stdout:2/210: dread d5/db/dd/f11 [0,4194304] 0 2024-04-05T21:26:27.561 INFO:tasks.workunit.client.1.smithi188.stdout:3/226: fsync d5/f6 0 2024-04-05T21:26:27.563 INFO:tasks.workunit.client.1.smithi188.stdout:6/220: mkdir d6/d13/d48/d49 0 2024-04-05T21:26:27.565 INFO:tasks.workunit.client.1.smithi188.stdout:4/283: symlink d0/d14/d2f/l68 0 2024-04-05T21:26:27.566 INFO:tasks.workunit.client.1.smithi188.stdout:5/151: truncate d6/f2d 1270378 0 2024-04-05T21:26:27.567 INFO:tasks.workunit.client.1.smithi188.stdout:1/185: dwrite d1/dc/f11 [0,4194304] 0 2024-04-05T21:26:27.567 INFO:tasks.workunit.client.1.smithi188.stdout:6/221: write d6/d13/d18/d19/f27 [489107,82043] 0 2024-04-05T21:26:27.568 INFO:tasks.workunit.client.1.smithi188.stdout:2/211: mknod d5/db/dd/c39 0 2024-04-05T21:26:27.573 INFO:tasks.workunit.client.1.smithi188.stdout:4/284: truncate d0/d2/d17/d3a/f54 727445 0 2024-04-05T21:26:27.575 INFO:tasks.workunit.client.1.smithi188.stdout:2/212: write d5/fc [501578,109040] 0 2024-04-05T21:26:27.580 INFO:tasks.workunit.client.1.smithi188.stdout:5/152: symlink d6/l36 0 2024-04-05T21:26:27.581 INFO:tasks.workunit.client.1.smithi188.stdout:0/120: dwrite d4/d8/d10/f1e [0,4194304] 0 2024-04-05T21:26:27.581 INFO:tasks.workunit.client.1.smithi188.stdout:3/227: link d5/dc/ld d5/d7/d31/l53 0 2024-04-05T21:26:27.583 INFO:tasks.workunit.client.1.smithi188.stdout:6/222: dwrite d6/f3b [0,4194304] 0 2024-04-05T21:26:27.585 INFO:tasks.workunit.client.1.smithi188.stdout:1/186: truncate d1/fb 2877294 0 2024-04-05T21:26:27.585 INFO:tasks.workunit.client.1.smithi188.stdout:0/121: chown d4/d6/d13/d23 1506902 1 2024-04-05T21:26:27.585 INFO:tasks.workunit.client.1.smithi188.stdout:3/228: write d5/df/f33 [440513,427] 0 2024-04-05T21:26:27.586 INFO:tasks.workunit.client.1.smithi188.stdout:4/285: symlink d0/d14/l69 0 2024-04-05T21:26:27.586 INFO:tasks.workunit.client.1.smithi188.stdout:6/223: dread - d6/f2e zero size 2024-04-05T21:26:27.590 INFO:tasks.workunit.client.1.smithi188.stdout:3/229: write d5/df/f15 [4386270,64601] 0 2024-04-05T21:26:27.593 INFO:tasks.workunit.client.1.smithi188.stdout:5/153: symlink d6/d15/l37 0 2024-04-05T21:26:27.594 INFO:tasks.workunit.client.1.smithi188.stdout:3/230: chown d5/dc/f49 2 1 2024-04-05T21:26:27.595 INFO:tasks.workunit.client.1.smithi188.stdout:4/286: mkdir d0/d14/d2f/d35/d6a 0 2024-04-05T21:26:27.596 INFO:tasks.workunit.client.1.smithi188.stdout:3/231: fsync d5/df/f33 0 2024-04-05T21:26:27.598 INFO:tasks.workunit.client.1.smithi188.stdout:6/224: symlink d6/d13/d18/d3c/l4a 0 2024-04-05T21:26:27.600 INFO:tasks.workunit.client.1.smithi188.stdout:1/187: dwrite d1/dc/f15 [0,4194304] 0 2024-04-05T21:26:27.601 INFO:tasks.workunit.client.1.smithi188.stdout:1/188: write d1/f24 [565730,13653] 0 2024-04-05T21:26:27.601 INFO:tasks.workunit.client.1.smithi188.stdout:5/154: unlink d6/d15/l37 0 2024-04-05T21:26:27.604 INFO:tasks.workunit.client.1.smithi188.stdout:3/232: dread d5/f6 [0,4194304] 0 2024-04-05T21:26:27.605 INFO:tasks.workunit.client.1.smithi188.stdout:0/122: creat d4/d8/db/d21/f2b x:0 0 0 2024-04-05T21:26:27.608 INFO:tasks.workunit.client.1.smithi188.stdout:6/225: mknod d6/d13/d18/d19/d2c/c4b 0 2024-04-05T21:26:27.609 INFO:tasks.workunit.client.1.smithi188.stdout:5/155: read - d6/d15/d1b/f27 zero size 2024-04-05T21:26:27.610 INFO:tasks.workunit.client.1.smithi188.stdout:3/233: rename d5/fb to d5/d2b/d43/f54 0 2024-04-05T21:26:27.611 INFO:tasks.workunit.client.1.smithi188.stdout:3/234: readlink d5/d2b/l4a 0 2024-04-05T21:26:27.611 INFO:tasks.workunit.client.1.smithi188.stdout:0/123: mknod d4/d6/c2c 0 2024-04-05T21:26:27.615 INFO:tasks.workunit.client.1.smithi188.stdout:5/156: fdatasync d6/f11 0 2024-04-05T21:26:27.619 INFO:tasks.workunit.client.1.smithi188.stdout:0/124: symlink d4/d8/d10/l2d 0 2024-04-05T21:26:27.620 INFO:tasks.workunit.client.1.smithi188.stdout:5/157: chown d6/d17/f1f 6882948 1 2024-04-05T21:26:27.622 INFO:tasks.workunit.client.1.smithi188.stdout:6/226: link d6/d13/f17 d6/d13/d38/f4c 0 2024-04-05T21:26:27.623 INFO:tasks.workunit.client.1.smithi188.stdout:0/125: dread d4/d6/d13/f14 [4194304,4194304] 0 2024-04-05T21:26:27.625 INFO:tasks.workunit.client.1.smithi188.stdout:6/227: unlink d6/d13/d18/d3c/f44 0 2024-04-05T21:26:27.629 INFO:tasks.workunit.client.1.smithi188.stdout:5/158: rename d6/d15/f16 to d6/f38 0 2024-04-05T21:26:27.630 INFO:tasks.workunit.client.1.smithi188.stdout:0/126: mkdir d4/d2e 0 2024-04-05T21:26:27.630 INFO:tasks.workunit.client.1.smithi188.stdout:3/235: rmdir d5 39 2024-04-05T21:26:27.632 INFO:tasks.workunit.client.1.smithi188.stdout:4/287: write d0/d14/d15/fb [808659,60354] 0 2024-04-05T21:26:27.634 INFO:tasks.workunit.client.1.smithi188.stdout:6/228: dwrite d6/d13/d38/f47 [0,4194304] 0 2024-04-05T21:26:27.634 INFO:tasks.workunit.client.1.smithi188.stdout:5/159: mkdir d6/d17/d39 0 2024-04-05T21:26:27.638 INFO:tasks.workunit.client.1.smithi188.stdout:4/288: write d0/d2/d17/d3a/f54 [604987,60418] 0 2024-04-05T21:26:27.640 INFO:tasks.workunit.client.1.smithi188.stdout:6/229: mknod d6/d13/c4d 0 2024-04-05T21:26:27.642 INFO:tasks.workunit.client.1.smithi188.stdout:3/236: fdatasync d5/d7/f25 0 2024-04-05T21:26:27.642 INFO:tasks.workunit.client.1.smithi188.stdout:4/289: dread - d0/d2/f58 zero size 2024-04-05T21:26:27.645 INFO:tasks.workunit.client.1.smithi188.stdout:0/127: dwrite f3 [4194304,4194304] 0 2024-04-05T21:26:27.647 INFO:tasks.workunit.client.1.smithi188.stdout:6/230: creat d6/d13/d38/f4e x:0 0 0 2024-04-05T21:26:27.649 INFO:tasks.workunit.client.1.smithi188.stdout:4/290: rename d0/d2/d2a/d51 to d0/d2/d2a/d34/d6b 0 2024-04-05T21:26:27.650 INFO:tasks.workunit.client.1.smithi188.stdout:6/231: write d6/d13/d18/d19/f30 [1500005,114391] 0 2024-04-05T21:26:27.651 INFO:tasks.workunit.client.1.smithi188.stdout:5/160: dwrite d6/f11 [0,4194304] 0 2024-04-05T21:26:27.653 INFO:tasks.workunit.client.1.smithi188.stdout:5/161: readlink d6/l7 0 2024-04-05T21:26:27.654 INFO:tasks.workunit.client.1.smithi188.stdout:5/162: dread - d6/d17/f20 zero size 2024-04-05T21:26:27.655 INFO:tasks.workunit.client.1.smithi188.stdout:4/291: mkdir d0/d14/d6c 0 2024-04-05T21:26:27.657 INFO:tasks.workunit.client.1.smithi188.stdout:6/232: mknod d6/d13/d18/d19/d2c/d34/c4f 0 2024-04-05T21:26:27.659 INFO:tasks.workunit.client.1.smithi188.stdout:3/237: truncate d5/dc/f2a 4813436 0 2024-04-05T21:26:27.661 INFO:tasks.workunit.client.1.smithi188.stdout:5/163: symlink d6/d15/l3a 0 2024-04-05T21:26:27.662 INFO:tasks.workunit.client.1.smithi188.stdout:4/292: mknod d0/d2/d52/c6d 0 2024-04-05T21:26:27.662 INFO:tasks.workunit.client.1.smithi188.stdout:3/238: rmdir d5/d48 39 2024-04-05T21:26:27.663 INFO:tasks.workunit.client.1.smithi188.stdout:3/239: chown d5/d7/d31 607 1 2024-04-05T21:26:27.665 INFO:tasks.workunit.client.1.smithi188.stdout:6/233: truncate d6/d13/d38/f4c 1013451 0 2024-04-05T21:26:27.665 INFO:tasks.workunit.client.1.smithi188.stdout:4/293: rmdir d0/d14/d2f/d35 39 2024-04-05T21:26:27.666 INFO:tasks.workunit.client.1.smithi188.stdout:5/164: fdatasync d6/f11 0 2024-04-05T21:26:27.668 INFO:tasks.workunit.client.1.smithi188.stdout:3/240: dread d5/df/f15 [0,4194304] 0 2024-04-05T21:26:27.672 INFO:tasks.workunit.client.1.smithi188.stdout:3/241: creat d5/d39/f55 x:0 0 0 2024-04-05T21:26:27.672 INFO:tasks.workunit.client.1.smithi188.stdout:5/165: rmdir d6/d15 39 2024-04-05T21:26:27.674 INFO:tasks.workunit.client.1.smithi188.stdout:4/294: creat d0/d14/f6e x:0 0 0 2024-04-05T21:26:27.677 INFO:tasks.workunit.client.1.smithi188.stdout:3/242: mknod d5/d2b/d43/c56 0 2024-04-05T21:26:27.677 INFO:tasks.workunit.client.1.smithi188.stdout:3/243: fsync d5/dc/f3d 0 2024-04-05T21:26:27.677 INFO:tasks.workunit.client.1.smithi188.stdout:6/234: creat d6/d13/d18/f50 x:0 0 0 2024-04-05T21:26:27.678 INFO:tasks.workunit.client.1.smithi188.stdout:6/235: write d6/d7/f16 [2645967,52664] 0 2024-04-05T21:26:27.681 INFO:tasks.workunit.client.1.smithi188.stdout:3/244: readlink d5/d39/l46 0 2024-04-05T21:26:27.682 INFO:tasks.workunit.client.1.smithi188.stdout:5/166: creat d6/d17/d23/d35/f3b x:0 0 0 2024-04-05T21:26:27.684 INFO:tasks.workunit.client.1.smithi188.stdout:6/236: stat d6/f11 0 2024-04-05T21:26:27.685 INFO:tasks.workunit.client.1.smithi188.stdout:6/237: fsync d6/d28/f31 0 2024-04-05T21:26:27.686 INFO:tasks.workunit.client.1.smithi188.stdout:5/167: creat d6/d17/d24/f3c x:0 0 0 2024-04-05T21:26:27.687 INFO:tasks.workunit.client.1.smithi188.stdout:4/295: symlink d0/d2/d4f/l6f 0 2024-04-05T21:26:27.688 INFO:tasks.workunit.client.1.smithi188.stdout:6/238: creat d6/d13/d18/d19/d2c/f51 x:0 0 0 2024-04-05T21:26:27.689 INFO:tasks.workunit.client.1.smithi188.stdout:5/168: creat d6/d17/d24/f3d x:0 0 0 2024-04-05T21:26:27.690 INFO:tasks.workunit.client.1.smithi188.stdout:6/239: mknod d6/d28/c52 0 2024-04-05T21:26:27.691 INFO:tasks.workunit.client.1.smithi188.stdout:4/296: mkdir d0/d2/d17/d48/d70 0 2024-04-05T21:26:27.692 INFO:tasks.workunit.client.1.smithi188.stdout:4/297: stat d0/d2/d17/f44 0 2024-04-05T21:26:27.693 INFO:tasks.workunit.client.1.smithi188.stdout:6/240: getdents d6/d7 0 2024-04-05T21:26:27.694 INFO:tasks.workunit.client.1.smithi188.stdout:4/298: dread - d0/d2/d4f/f5c zero size 2024-04-05T21:26:27.696 INFO:tasks.workunit.client.1.smithi188.stdout:4/299: write d0/d14/d15/d1b/f4d [186380,98872] 0 2024-04-05T21:26:27.699 INFO:tasks.workunit.client.1.smithi188.stdout:4/300: mkdir d0/d2/d2a/d71 0 2024-04-05T21:26:27.700 INFO:tasks.workunit.client.1.smithi188.stdout:4/301: chown d0/d14/d15/d1b 101205 1 2024-04-05T21:26:27.701 INFO:tasks.workunit.client.1.smithi188.stdout:6/241: dwrite d6/d28/f31 [0,4194304] 0 2024-04-05T21:26:27.702 INFO:tasks.workunit.client.1.smithi188.stdout:6/242: write d6/d13/d18/d19/d2c/f51 [47077,58312] 0 2024-04-05T21:26:27.703 INFO:tasks.workunit.client.1.smithi188.stdout:4/302: mknod d0/d14/d15/d1b/c72 0 2024-04-05T21:26:27.707 INFO:tasks.workunit.client.1.smithi188.stdout:4/303: unlink d0/d14/d15/f29 0 2024-04-05T21:26:27.710 INFO:tasks.workunit.client.1.smithi188.stdout:4/304: symlink d0/d14/l73 0 2024-04-05T21:26:27.711 INFO:tasks.workunit.client.1.smithi188.stdout:4/305: creat d0/d14/d15/d53/f74 x:0 0 0 2024-04-05T21:26:27.712 INFO:tasks.workunit.client.1.smithi188.stdout:4/306: fdatasync d0/d14/f39 0 2024-04-05T21:26:27.721 INFO:tasks.workunit.client.1.smithi188.stdout:6/243: dread d6/d7/f15 [0,4194304] 0 2024-04-05T21:26:27.722 INFO:tasks.workunit.client.1.smithi188.stdout:6/244: write d6/d13/d38/f47 [4594530,13881] 0 2024-04-05T21:26:27.759 INFO:tasks.workunit.client.1.smithi188.stdout:6/245: creat d6/d13/d18/d19/d2c/d3f/f53 x:0 0 0 2024-04-05T21:26:27.761 INFO:tasks.workunit.client.1.smithi188.stdout:4/307: link d0/d14/c32 d0/d2/d2a/d34/d6b/c75 0 2024-04-05T21:26:27.765 INFO:tasks.workunit.client.1.smithi188.stdout:4/308: rename d0/d14/d15/l2c to d0/d2/d2a/l76 0 2024-04-05T21:26:27.767 INFO:tasks.workunit.client.1.smithi188.stdout:4/309: creat d0/d2/d2a/d34/f77 x:0 0 0 2024-04-05T21:26:27.772 INFO:tasks.workunit.client.1.smithi188.stdout:4/310: mkdir d0/d14/d15/d78 0 2024-04-05T21:26:27.777 INFO:tasks.workunit.client.1.smithi188.stdout:6/246: dwrite f5 [0,4194304] 0 2024-04-05T21:26:27.779 INFO:tasks.workunit.client.1.smithi188.stdout:4/311: creat d0/d2/d52/f79 x:0 0 0 2024-04-05T21:26:27.781 INFO:tasks.workunit.client.1.smithi188.stdout:4/312: symlink d0/d2/d2a/l7a 0 2024-04-05T21:26:27.787 INFO:tasks.workunit.client.1.smithi188.stdout:6/247: dwrite d6/d13/d18/f35 [0,4194304] 0 2024-04-05T21:26:27.792 INFO:tasks.workunit.client.1.smithi188.stdout:4/313: rmdir d0 39 2024-04-05T21:26:27.794 INFO:tasks.workunit.client.1.smithi188.stdout:4/314: chown d0/d2/d17/d3a 1991 1 2024-04-05T21:26:27.797 INFO:tasks.workunit.client.1.smithi188.stdout:4/315: creat d0/d14/d2f/d35/f7b x:0 0 0 2024-04-05T21:26:27.799 INFO:tasks.workunit.client.1.smithi188.stdout:4/316: creat d0/d14/d2f/d35/f7c x:0 0 0 2024-04-05T21:26:27.800 INFO:tasks.workunit.client.1.smithi188.stdout:4/317: stat d0/d14/c32 0 2024-04-05T21:26:27.800 INFO:tasks.workunit.client.1.smithi188.stdout:4/318: stat d0/d2/d17/d3a/f54 0 2024-04-05T21:26:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:27 smithi121 ceph-mon[39723]: pgmap v12: 65 pgs: 65 active+clean; 429 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 718 KiB/s rd, 23 MiB/s wr, 176 op/s 2024-04-05T21:26:28.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:27 smithi188 ceph-mon[49715]: pgmap v12: 65 pgs: 65 active+clean; 429 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 718 KiB/s rd, 23 MiB/s wr, 176 op/s 2024-04-05T21:26:28.101 INFO:tasks.workunit.client.1.smithi188.stdout:9/119: sync 2024-04-05T21:26:28.101 INFO:tasks.workunit.client.1.smithi188.stdout:7/276: sync 2024-04-05T21:26:28.101 INFO:tasks.workunit.client.1.smithi188.stdout:9/120: write fc [1083737,52556] 0 2024-04-05T21:26:28.104 INFO:tasks.workunit.client.1.smithi188.stdout:9/121: mkdir dd/d11/d1f/d25 0 2024-04-05T21:26:28.105 INFO:tasks.workunit.client.1.smithi188.stdout:9/122: fdatasync dd/f12 0 2024-04-05T21:26:28.109 INFO:tasks.workunit.client.1.smithi188.stdout:7/277: dwrite d3/d4/db/d2e/d3b/f43 [0,4194304] 0 2024-04-05T21:26:28.110 INFO:tasks.workunit.client.1.smithi188.stdout:9/123: rename dd/d11/d1e to dd/d11/d1f/d26 0 2024-04-05T21:26:28.111 INFO:tasks.workunit.client.1.smithi188.stdout:7/278: write d3/d4/d1d/f34 [88678,92502] 0 2024-04-05T21:26:28.117 INFO:tasks.workunit.client.1.smithi188.stdout:7/279: getdents d3/d4/db/d2e 0 2024-04-05T21:26:28.117 INFO:tasks.workunit.client.1.smithi188.stdout:8/99: sync 2024-04-05T21:26:28.119 INFO:tasks.workunit.client.1.smithi188.stdout:8/100: chown d5/f20 4228 1 2024-04-05T21:26:28.120 INFO:tasks.workunit.client.1.smithi188.stdout:9/124: dwrite dd/f12 [0,4194304] 0 2024-04-05T21:26:28.125 INFO:tasks.workunit.client.1.smithi188.stdout:8/101: dread d5/f16 [0,4194304] 0 2024-04-05T21:26:28.125 INFO:tasks.workunit.client.1.smithi188.stdout:7/280: link d3/d4/db/d10/f14 d3/d4/db/d10/d2f/f59 0 2024-04-05T21:26:28.126 INFO:tasks.workunit.client.1.smithi188.stdout:8/102: chown d5/dc/c13 484 1 2024-04-05T21:26:28.126 INFO:tasks.workunit.client.1.smithi188.stdout:7/281: write d3/d4/db/d10/f33 [805519,57497] 0 2024-04-05T21:26:28.136 INFO:tasks.workunit.client.1.smithi188.stdout:7/282: dwrite d3/d4/db/d10/d2f/f59 [4194304,4194304] 0 2024-04-05T21:26:28.139 INFO:tasks.workunit.client.1.smithi188.stdout:7/283: dread d3/d4/db/d2e/d3b/f4d [0,4194304] 0 2024-04-05T21:26:28.140 INFO:tasks.workunit.client.1.smithi188.stdout:7/284: mknod d3/d4/db/c5a 0 2024-04-05T21:26:28.142 INFO:tasks.workunit.client.1.smithi188.stdout:7/285: symlink d3/d4/db/d55/l5b 0 2024-04-05T21:26:28.144 INFO:tasks.workunit.client.1.smithi188.stdout:7/286: creat d3/d4/db/d10/d2f/f5c x:0 0 0 2024-04-05T21:26:28.145 INFO:tasks.workunit.client.1.smithi188.stdout:7/287: creat d3/d4/db/d2e/d49/f5d x:0 0 0 2024-04-05T21:26:28.147 INFO:tasks.workunit.client.1.smithi188.stdout:7/288: mkdir d3/d2c/d5e 0 2024-04-05T21:26:28.148 INFO:tasks.workunit.client.1.smithi188.stdout:9/125: dread f4 [0,4194304] 0 2024-04-05T21:26:28.161 INFO:tasks.workunit.client.1.smithi188.stdout:9/126: truncate dd/f12 2046031 0 2024-04-05T21:26:28.164 INFO:tasks.workunit.client.1.smithi188.stdout:9/127: rename c6 to dd/d11/c27 0 2024-04-05T21:26:28.167 INFO:tasks.workunit.client.1.smithi188.stdout:9/128: dread dd/f16 [0,4194304] 0 2024-04-05T21:26:28.169 INFO:tasks.workunit.client.1.smithi188.stdout:9/129: mkdir dd/d11/d22/d28 0 2024-04-05T21:26:28.170 INFO:tasks.workunit.client.1.smithi188.stdout:9/130: creat dd/d11/d1f/f29 x:0 0 0 2024-04-05T21:26:28.177 INFO:tasks.workunit.client.1.smithi188.stdout:9/131: rmdir dd/d11/d1f 39 2024-04-05T21:26:28.177 INFO:tasks.workunit.client.1.smithi188.stdout:9/132: chown dd 0 1 2024-04-05T21:26:28.178 INFO:tasks.workunit.client.1.smithi188.stdout:9/133: symlink dd/l2a 0 2024-04-05T21:26:28.179 INFO:tasks.workunit.client.1.smithi188.stdout:9/134: dread - dd/d11/d1f/f29 zero size 2024-04-05T21:26:28.180 INFO:tasks.workunit.client.1.smithi188.stdout:9/135: chown c2 17995 1 2024-04-05T21:26:28.190 INFO:tasks.workunit.client.1.smithi188.stdout:9/136: dwrite f5 [0,4194304] 0 2024-04-05T21:26:28.192 INFO:tasks.workunit.client.1.smithi188.stdout:9/137: symlink dd/d11/d1f/d25/l2b 0 2024-04-05T21:26:28.193 INFO:tasks.workunit.client.1.smithi188.stdout:9/138: write dd/f16 [3598770,78023] 0 2024-04-05T21:26:28.198 INFO:tasks.workunit.client.1.smithi188.stdout:9/139: fsync dd/d17/f23 0 2024-04-05T21:26:28.199 INFO:tasks.workunit.client.1.smithi188.stdout:9/140: write f5 [4453956,119943] 0 2024-04-05T21:26:28.201 INFO:tasks.workunit.client.1.smithi188.stdout:9/141: write dd/d17/f23 [3467686,60907] 0 2024-04-05T21:26:28.206 INFO:tasks.workunit.client.1.smithi188.stdout:9/142: dwrite dd/d11/d1f/d26/f19 [0,4194304] 0 2024-04-05T21:26:28.207 INFO:tasks.workunit.client.1.smithi188.stdout:5/169: sync 2024-04-05T21:26:28.207 INFO:tasks.workunit.client.1.smithi188.stdout:4/319: sync 2024-04-05T21:26:28.207 INFO:tasks.workunit.client.1.smithi188.stdout:1/189: sync 2024-04-05T21:26:28.207 INFO:tasks.workunit.client.1.smithi188.stdout:2/213: sync 2024-04-05T21:26:28.207 INFO:tasks.workunit.client.1.smithi188.stdout:3/245: sync 2024-04-05T21:26:28.207 INFO:tasks.workunit.client.1.smithi188.stdout:0/128: sync 2024-04-05T21:26:28.207 INFO:tasks.workunit.client.1.smithi188.stdout:6/248: sync 2024-04-05T21:26:28.208 INFO:tasks.workunit.client.1.smithi188.stdout:4/320: readlink d0/d14/l1f 0 2024-04-05T21:26:28.213 INFO:tasks.workunit.client.1.smithi188.stdout:6/249: mknod d6/d13/d18/d19/d2c/d3f/c54 0 2024-04-05T21:26:28.215 INFO:tasks.workunit.client.1.smithi188.stdout:0/129: rename d4/d8/db/c18 to d4/d8/db/c2f 0 2024-04-05T21:26:28.216 INFO:tasks.workunit.client.1.smithi188.stdout:6/250: read d6/f2b [266676,56279] 0 2024-04-05T21:26:28.217 INFO:tasks.workunit.client.1.smithi188.stdout:1/190: truncate d1/dc/d16/d2d/f34 753650 0 2024-04-05T21:26:28.217 INFO:tasks.workunit.client.1.smithi188.stdout:9/143: dread dd/f12 [0,4194304] 0 2024-04-05T21:26:28.223 INFO:tasks.workunit.client.1.smithi188.stdout:2/214: dwrite d5/f22 [0,4194304] 0 2024-04-05T21:26:28.224 INFO:tasks.workunit.client.1.smithi188.stdout:3/246: getdents d5/dc 0 2024-04-05T21:26:28.227 INFO:tasks.workunit.client.1.smithi188.stdout:0/130: rename d4/d6/d13/f2a to d4/d2e/f30 0 2024-04-05T21:26:28.230 INFO:tasks.workunit.client.1.smithi188.stdout:4/321: write d0/d14/d15/d1b/f1d [779776,125480] 0 2024-04-05T21:26:28.230 INFO:tasks.workunit.client.1.smithi188.stdout:2/215: readlink d5/db/dd/d24/l2d 0 2024-04-05T21:26:28.231 INFO:tasks.workunit.client.1.smithi188.stdout:3/247: read d5/d7/d31/f45 [29,76130] 0 2024-04-05T21:26:28.232 INFO:tasks.workunit.client.1.smithi188.stdout:0/131: fsync d4/d6/f12 0 2024-04-05T21:26:28.233 INFO:tasks.workunit.client.1.smithi188.stdout:6/251: dread d6/f11 [0,4194304] 0 2024-04-05T21:26:28.235 INFO:tasks.workunit.client.1.smithi188.stdout:2/216: unlink d5/c33 0 2024-04-05T21:26:28.236 INFO:tasks.workunit.client.1.smithi188.stdout:3/248: mknod d5/d7/d31/c57 0 2024-04-05T21:26:28.237 INFO:tasks.workunit.client.1.smithi188.stdout:9/144: dwrite fc [0,4194304] 0 2024-04-05T21:26:28.238 INFO:tasks.workunit.client.1.smithi188.stdout:1/191: dwrite d1/d4/f12 [0,4194304] 0 2024-04-05T21:26:28.238 INFO:tasks.workunit.client.1.smithi188.stdout:6/252: creat d6/d13/d38/f55 x:0 0 0 2024-04-05T21:26:28.244 INFO:tasks.workunit.client.1.smithi188.stdout:2/217: symlink d5/db/dd/d36/l3a 0 2024-04-05T21:26:28.244 INFO:tasks.workunit.client.1.smithi188.stdout:9/145: creat dd/d11/d1f/d26/f2c x:0 0 0 2024-04-05T21:26:28.247 INFO:tasks.workunit.client.1.smithi188.stdout:0/132: dwrite d4/d6/d13/f14 [0,4194304] 0 2024-04-05T21:26:28.248 INFO:tasks.workunit.client.1.smithi188.stdout:6/253: truncate d6/d7/f15 1609152 0 2024-04-05T21:26:28.248 INFO:tasks.workunit.client.1.smithi188.stdout:3/249: rename d5/d7/c9 to d5/df/c58 0 2024-04-05T21:26:28.252 INFO:tasks.workunit.client.1.smithi188.stdout:3/250: fsync d5/dc/f2d 0 2024-04-05T21:26:28.252 INFO:tasks.workunit.client.1.smithi188.stdout:0/133: stat d4/d8/db/c16 0 2024-04-05T21:26:28.253 INFO:tasks.workunit.client.1.smithi188.stdout:2/218: mkdir d5/db/d35/d3b 0 2024-04-05T21:26:28.253 INFO:tasks.workunit.client.1.smithi188.stdout:9/146: creat dd/d11/d22/d28/f2d x:0 0 0 2024-04-05T21:26:28.255 INFO:tasks.workunit.client.1.smithi188.stdout:0/134: write d4/d8/d10/f20 [695242,3686] 0 2024-04-05T21:26:28.255 INFO:tasks.workunit.client.1.smithi188.stdout:0/135: stat d4/d6/f15 0 2024-04-05T21:26:28.258 INFO:tasks.workunit.client.1.smithi188.stdout:3/251: mknod d5/dc/c59 0 2024-04-05T21:26:28.260 INFO:tasks.workunit.client.1.smithi188.stdout:0/136: mknod d4/d8/db/c31 0 2024-04-05T21:26:28.260 INFO:tasks.workunit.client.1.smithi188.stdout:6/254: dwrite d6/d13/d18/d19/d2c/f3e [0,4194304] 0 2024-04-05T21:26:28.265 INFO:tasks.workunit.client.1.smithi188.stdout:3/252: creat d5/dc/f5a x:0 0 0 2024-04-05T21:26:28.265 INFO:tasks.workunit.client.1.smithi188.stdout:6/255: dread d6/d13/d18/d19/d2c/f51 [0,4194304] 0 2024-04-05T21:26:28.268 INFO:tasks.workunit.client.1.smithi188.stdout:6/256: chown d6/d13/d18/d19/f30 957667 1 2024-04-05T21:26:28.270 INFO:tasks.workunit.client.1.smithi188.stdout:6/257: write d6/d7/fd [2512415,39463] 0 2024-04-05T21:26:28.272 INFO:tasks.workunit.client.1.smithi188.stdout:3/253: getdents d5/d35 0 2024-04-05T21:26:28.273 INFO:tasks.workunit.client.1.smithi188.stdout:9/147: rmdir dd/d11/d1f/d26 39 2024-04-05T21:26:28.273 INFO:tasks.workunit.client.1.smithi188.stdout:0/137: dwrite d4/fe [0,4194304] 0 2024-04-05T21:26:28.275 INFO:tasks.workunit.client.1.smithi188.stdout:0/138: read - d4/d8/db/d21/f2b zero size 2024-04-05T21:26:28.278 INFO:tasks.workunit.client.1.smithi188.stdout:6/258: rename d6/f24 to d6/d13/d18/d19/d2c/d3f/f56 0 2024-04-05T21:26:28.283 INFO:tasks.workunit.client.1.smithi188.stdout:6/259: symlink d6/d28/l57 0 2024-04-05T21:26:28.284 INFO:tasks.workunit.client.1.smithi188.stdout:6/260: readlink d6/d28/l57 0 2024-04-05T21:26:28.284 INFO:tasks.workunit.client.1.smithi188.stdout:3/254: creat d5/d35/d4e/f5b x:0 0 0 2024-04-05T21:26:28.286 INFO:tasks.workunit.client.1.smithi188.stdout:0/139: unlink d4/d6/d13/c17 0 2024-04-05T21:26:28.287 INFO:tasks.workunit.client.1.smithi188.stdout:9/148: dwrite dd/d11/d1f/d26/f19 [0,4194304] 0 2024-04-05T21:26:28.289 INFO:tasks.workunit.client.1.smithi188.stdout:0/140: write d4/fe [3978221,97632] 0 2024-04-05T21:26:28.292 INFO:tasks.workunit.client.1.smithi188.stdout:9/149: write dd/d11/d1f/d26/f19 [2358105,59475] 0 2024-04-05T21:26:28.292 INFO:tasks.workunit.client.1.smithi188.stdout:0/141: readlink d4/d8/d10/l1a 0 2024-04-05T21:26:28.294 INFO:tasks.workunit.client.1.smithi188.stdout:6/261: dread d6/f2b [0,4194304] 0 2024-04-05T21:26:28.294 INFO:tasks.workunit.client.1.smithi188.stdout:9/150: dread - dd/d11/f21 zero size 2024-04-05T21:26:28.295 INFO:tasks.workunit.client.1.smithi188.stdout:0/142: write d4/d6/f12 [941570,91354] 0 2024-04-05T21:26:28.295 INFO:tasks.workunit.client.1.smithi188.stdout:2/219: dwrite d5/db/f20 [0,4194304] 0 2024-04-05T21:26:28.296 INFO:tasks.workunit.client.1.smithi188.stdout:0/143: write f1 [6563762,105429] 0 2024-04-05T21:26:28.299 INFO:tasks.workunit.client.1.smithi188.stdout:0/144: chown d4/d6/f15 0 1 2024-04-05T21:26:28.302 INFO:tasks.workunit.client.1.smithi188.stdout:0/145: chown d4/d8/f26 268400488 1 2024-04-05T21:26:28.307 INFO:tasks.workunit.client.1.smithi188.stdout:9/151: dread f4 [0,4194304] 0 2024-04-05T21:26:28.308 INFO:tasks.workunit.client.1.smithi188.stdout:0/146: creat d4/d8/db/d21/f32 x:0 0 0 2024-04-05T21:26:28.309 INFO:tasks.workunit.client.1.smithi188.stdout:9/152: chown fc 6932 1 2024-04-05T21:26:28.311 INFO:tasks.workunit.client.1.smithi188.stdout:2/220: dwrite d5/fc [0,4194304] 0 2024-04-05T21:26:28.313 INFO:tasks.workunit.client.1.smithi188.stdout:6/262: dwrite d6/d13/d18/d19/d2c/f51 [0,4194304] 0 2024-04-05T21:26:28.314 INFO:tasks.workunit.client.1.smithi188.stdout:0/147: dread d4/d6/f15 [0,4194304] 0 2024-04-05T21:26:28.319 INFO:tasks.workunit.client.1.smithi188.stdout:9/153: rename dd/d11/d1f/d26/l1c to dd/d11/d22/l2e 0 2024-04-05T21:26:28.321 INFO:tasks.workunit.client.1.smithi188.stdout:2/221: mkdir d5/db/dd/d36/d3c 0 2024-04-05T21:26:28.323 INFO:tasks.workunit.client.1.smithi188.stdout:0/148: mknod d4/c33 0 2024-04-05T21:26:28.325 INFO:tasks.workunit.client.1.smithi188.stdout:9/154: dread dd/f12 [0,4194304] 0 2024-04-05T21:26:28.326 INFO:tasks.workunit.client.1.smithi188.stdout:9/155: stat f1 0 2024-04-05T21:26:28.329 INFO:tasks.workunit.client.1.smithi188.stdout:0/149: creat d4/d8/db/d21/f34 x:0 0 0 2024-04-05T21:26:28.330 INFO:tasks.workunit.client.1.smithi188.stdout:9/156: mknod dd/d11/d22/d28/c2f 0 2024-04-05T21:26:28.331 INFO:tasks.workunit.client.1.smithi188.stdout:2/222: dwrite d5/f2f [4194304,4194304] 0 2024-04-05T21:26:28.332 INFO:tasks.workunit.client.1.smithi188.stdout:0/150: readlink d4/d8/d10/l2d 0 2024-04-05T21:26:28.333 INFO:tasks.workunit.client.1.smithi188.stdout:0/151: read d4/f27 [1275585,68460] 0 2024-04-05T21:26:28.336 INFO:tasks.workunit.client.1.smithi188.stdout:2/223: chown d5/db/dd/d24/c37 1 1 2024-04-05T21:26:28.338 INFO:tasks.workunit.client.1.smithi188.stdout:6/263: getdents d6 0 2024-04-05T21:26:28.341 INFO:tasks.workunit.client.1.smithi188.stdout:6/264: chown d6/d13/d18/f50 11 1 2024-04-05T21:26:28.343 INFO:tasks.workunit.client.1.smithi188.stdout:0/152: symlink d4/d6/d13/d23/d24/l35 0 2024-04-05T21:26:28.344 INFO:tasks.workunit.client.1.smithi188.stdout:9/157: dwrite dd/d11/d22/d28/f2d [0,4194304] 0 2024-04-05T21:26:28.345 INFO:tasks.workunit.client.1.smithi188.stdout:9/158: write f4 [947738,60186] 0 2024-04-05T21:26:28.349 INFO:tasks.workunit.client.1.smithi188.stdout:2/224: mkdir d5/db/dd/d36/d3d 0 2024-04-05T21:26:28.349 INFO:tasks.workunit.client.1.smithi188.stdout:6/265: creat d6/d28/f58 x:0 0 0 2024-04-05T21:26:28.352 INFO:tasks.workunit.client.1.smithi188.stdout:0/153: mknod d4/d8/db/d1c/c36 0 2024-04-05T21:26:28.357 INFO:tasks.workunit.client.1.smithi188.stdout:9/159: dwrite dd/d11/f21 [0,4194304] 0 2024-04-05T21:26:28.364 INFO:tasks.workunit.client.1.smithi188.stdout:8/103: sync 2024-04-05T21:26:28.365 INFO:tasks.workunit.client.1.smithi188.stdout:9/160: dwrite dd/f16 [4194304,4194304] 0 2024-04-05T21:26:28.367 INFO:tasks.workunit.client.1.smithi188.stdout:9/161: chown dd/d11/c18 3238616 1 2024-04-05T21:26:28.369 INFO:tasks.workunit.client.1.smithi188.stdout:8/104: rename d5/dc/c1b to d5/d21/c23 0 2024-04-05T21:26:28.372 INFO:tasks.workunit.client.1.smithi188.stdout:2/225: write d5/db/dd/f11 [1836900,84329] 0 2024-04-05T21:26:28.372 INFO:tasks.workunit.client.1.smithi188.stdout:9/162: dread dd/f16 [4194304,4194304] 0 2024-04-05T21:26:28.374 INFO:tasks.workunit.client.1.smithi188.stdout:6/266: dread d6/d13/d38/f4c [0,4194304] 0 2024-04-05T21:26:28.374 INFO:tasks.workunit.client.1.smithi188.stdout:2/226: write d5/fc [7195758,71114] 0 2024-04-05T21:26:28.382 INFO:tasks.workunit.client.1.smithi188.stdout:9/163: dwrite dd/d11/f21 [4194304,4194304] 0 2024-04-05T21:26:28.385 INFO:tasks.workunit.client.1.smithi188.stdout:0/154: symlink d4/d6/d13/d23/d24/l37 0 2024-04-05T21:26:28.385 INFO:tasks.workunit.client.1.smithi188.stdout:8/105: link d5/f16 d5/f24 0 2024-04-05T21:26:28.387 INFO:tasks.workunit.client.1.smithi188.stdout:2/227: creat d5/db/f3e x:0 0 0 2024-04-05T21:26:28.389 INFO:tasks.workunit.client.1.smithi188.stdout:9/164: rename dd/d11/d1f/d25/l2b to dd/l30 0 2024-04-05T21:26:28.389 INFO:tasks.workunit.client.1.smithi188.stdout:6/267: creat d6/d13/d18/f59 x:0 0 0 2024-04-05T21:26:28.392 INFO:tasks.workunit.client.1.smithi188.stdout:9/165: symlink dd/d11/d22/l31 0 2024-04-05T21:26:28.393 INFO:tasks.workunit.client.1.smithi188.stdout:2/228: symlink d5/db/dd/d36/d3d/l3f 0 2024-04-05T21:26:28.396 INFO:tasks.workunit.client.1.smithi188.stdout:6/268: rename d6/f2b to d6/d13/d18/d3c/f5a 0 2024-04-05T21:26:28.400 INFO:tasks.workunit.client.1.smithi188.stdout:0/155: chown d4/d6/d13/d23/d24/l35 14537830 1 2024-04-05T21:26:28.401 INFO:tasks.workunit.client.1.smithi188.stdout:2/229: dwrite d5/db/dd/f15 [4194304,4194304] 0 2024-04-05T21:26:28.401 INFO:tasks.workunit.client.1.smithi188.stdout:9/166: rename dd/d11/c1d to dd/d11/d1f/d26/c32 0 2024-04-05T21:26:28.402 INFO:tasks.workunit.client.1.smithi188.stdout:0/156: mkdir d4/d2e/d38 0 2024-04-05T21:26:28.402 INFO:tasks.workunit.client.1.smithi188.stdout:9/167: chown dd/l2a 23 1 2024-04-05T21:26:28.405 INFO:tasks.workunit.client.1.smithi188.stdout:0/157: read d4/d2e/f30 [2047955,69214] 0 2024-04-05T21:26:28.407 INFO:tasks.workunit.client.1.smithi188.stdout:0/158: readlink d4/d8/d10/l2d 0 2024-04-05T21:26:28.407 INFO:tasks.workunit.client.1.smithi188.stdout:9/168: readlink dd/d11/l15 0 2024-04-05T21:26:28.407 INFO:tasks.workunit.client.1.smithi188.stdout:8/106: write d5/f24 [1637460,36538] 0 2024-04-05T21:26:28.410 INFO:tasks.workunit.client.1.smithi188.stdout:9/169: symlink dd/d17/l33 0 2024-04-05T21:26:28.412 INFO:tasks.workunit.client.1.smithi188.stdout:2/230: dwrite d5/db/dd/f15 [4194304,4194304] 0 2024-04-05T21:26:28.413 INFO:tasks.workunit.client.1.smithi188.stdout:0/159: rename d4/d8/db/d1c to d4/d8/d39 0 2024-04-05T21:26:28.414 INFO:tasks.workunit.client.1.smithi188.stdout:2/231: fsync d5/db/dd/f1c 0 2024-04-05T21:26:28.414 INFO:tasks.workunit.client.1.smithi188.stdout:8/107: symlink d5/dc/l25 0 2024-04-05T21:26:28.423 INFO:tasks.workunit.client.1.smithi188.stdout:8/108: mknod d5/dc/c26 0 2024-04-05T21:26:28.423 INFO:tasks.workunit.client.1.smithi188.stdout:2/232: dwrite d5/f22 [4194304,4194304] 0 2024-04-05T21:26:28.426 INFO:tasks.workunit.client.1.smithi188.stdout:8/109: fsync d5/f1c 0 2024-04-05T21:26:28.426 INFO:tasks.workunit.client.1.smithi188.stdout:6/269: chown d6/d13/d18/d19/d2c/c33 378256854 1 2024-04-05T21:26:28.430 INFO:tasks.workunit.client.1.smithi188.stdout:9/170: rename dd/d17 to dd/d34 0 2024-04-05T21:26:28.431 INFO:tasks.workunit.client.1.smithi188.stdout:8/110: dread d5/f16 [0,4194304] 0 2024-04-05T21:26:28.434 INFO:tasks.workunit.client.1.smithi188.stdout:2/233: getdents d5/db 0 2024-04-05T21:26:28.438 INFO:tasks.workunit.client.1.smithi188.stdout:6/270: fsync d6/d13/d18/d19/d2c/d3f/f56 0 2024-04-05T21:26:28.439 INFO:tasks.workunit.client.1.smithi188.stdout:0/160: dwrite d4/d2e/f30 [0,4194304] 0 2024-04-05T21:26:28.439 INFO:tasks.workunit.client.1.smithi188.stdout:8/111: mknod d5/c27 0 2024-04-05T21:26:28.441 INFO:tasks.workunit.client.1.smithi188.stdout:0/161: truncate d4/d8/db/d21/f2b 285348 0 2024-04-05T21:26:28.443 INFO:tasks.workunit.client.1.smithi188.stdout:6/271: chown d6/d7/fd 147 1 2024-04-05T21:26:28.445 INFO:tasks.workunit.client.1.smithi188.stdout:2/234: truncate d5/db/dd/f11 1275828 0 2024-04-05T21:26:28.446 INFO:tasks.workunit.client.1.smithi188.stdout:2/235: read d5/db/dd/f15 [6617808,4972] 0 2024-04-05T21:26:28.448 INFO:tasks.workunit.client.1.smithi188.stdout:2/236: chown d5/db/dd/d24/l2d 631277379 1 2024-04-05T21:26:28.448 INFO:tasks.workunit.client.1.smithi188.stdout:8/112: mknod d5/c28 0 2024-04-05T21:26:28.450 INFO:tasks.workunit.client.1.smithi188.stdout:9/171: truncate dd/f12 3642267 0 2024-04-05T21:26:28.450 INFO:tasks.workunit.client.1.smithi188.stdout:8/113: write d5/f20 [798363,51562] 0 2024-04-05T21:26:28.453 INFO:tasks.workunit.client.1.smithi188.stdout:0/162: creat d4/d6/d13/d23/d24/f3a x:0 0 0 2024-04-05T21:26:28.454 INFO:tasks.workunit.client.1.smithi188.stdout:0/163: fdatasync d4/d6/f12 0 2024-04-05T21:26:28.455 INFO:tasks.workunit.client.1.smithi188.stdout:0/164: chown d4/d6/c2c 248289968 1 2024-04-05T21:26:28.457 INFO:tasks.workunit.client.1.smithi188.stdout:6/272: mknod d6/d28/d45/c5b 0 2024-04-05T21:26:28.457 INFO:tasks.workunit.client.1.smithi188.stdout:9/172: dread f4 [0,4194304] 0 2024-04-05T21:26:28.458 INFO:tasks.workunit.client.1.smithi188.stdout:6/273: dread - d6/d28/f58 zero size 2024-04-05T21:26:28.458 INFO:tasks.workunit.client.1.smithi188.stdout:9/173: chown dd/d11/f21 53 1 2024-04-05T21:26:28.459 INFO:tasks.workunit.client.1.smithi188.stdout:0/165: creat d4/d8/d10/f3b x:0 0 0 2024-04-05T21:26:28.460 INFO:tasks.workunit.client.1.smithi188.stdout:0/166: fsync d4/d8/d10/f20 0 2024-04-05T21:26:28.461 INFO:tasks.workunit.client.1.smithi188.stdout:8/114: creat d5/f29 x:0 0 0 2024-04-05T21:26:28.462 INFO:tasks.workunit.client.1.smithi188.stdout:6/274: mknod d6/d13/d18/d19/d2c/c5c 0 2024-04-05T21:26:28.463 INFO:tasks.workunit.client.1.smithi188.stdout:2/237: getdents d5/db/dd/d24 0 2024-04-05T21:26:28.464 INFO:tasks.workunit.client.1.smithi188.stdout:9/174: chown f5 83649897 1 2024-04-05T21:26:28.464 INFO:tasks.workunit.client.1.smithi188.stdout:2/238: read d5/fc [2366586,101922] 0 2024-04-05T21:26:28.466 INFO:tasks.workunit.client.1.smithi188.stdout:9/175: write dd/d11/f21 [5788945,100303] 0 2024-04-05T21:26:28.467 INFO:tasks.workunit.client.1.smithi188.stdout:8/115: truncate d5/fb 802166 0 2024-04-05T21:26:28.472 INFO:tasks.workunit.client.1.smithi188.stdout:2/239: mknod d5/db/dd/c40 0 2024-04-05T21:26:28.473 INFO:tasks.workunit.client.1.smithi188.stdout:6/275: truncate d6/d13/d18/d19/d2c/f32 2050742 0 2024-04-05T21:26:28.474 INFO:tasks.workunit.client.1.smithi188.stdout:6/276: fsync f5 0 2024-04-05T21:26:28.474 INFO:tasks.workunit.client.1.smithi188.stdout:9/176: mkdir dd/d35 0 2024-04-05T21:26:28.476 INFO:tasks.workunit.client.1.smithi188.stdout:8/116: mkdir d5/d21/d2a 0 2024-04-05T21:26:28.483 INFO:tasks.workunit.client.1.smithi188.stdout:6/277: dwrite d6/d28/f31 [4194304,4194304] 0 2024-04-05T21:26:28.484 INFO:tasks.workunit.client.1.smithi188.stdout:9/177: write f4 [946467,46078] 0 2024-04-05T21:26:28.484 INFO:tasks.workunit.client.1.smithi188.stdout:2/240: mknod d5/c41 0 2024-04-05T21:26:28.486 INFO:tasks.workunit.client.1.smithi188.stdout:6/278: chown d6/d13/d18/f36 76693241 1 2024-04-05T21:26:28.486 INFO:tasks.workunit.client.1.smithi188.stdout:0/167: truncate d4/d8/f26 1097031 0 2024-04-05T21:26:28.489 INFO:tasks.workunit.client.1.smithi188.stdout:8/117: dread d5/f8 [0,4194304] 0 2024-04-05T21:26:28.493 INFO:tasks.workunit.client.1.smithi188.stdout:0/168: mkdir d4/d8/db/d21/d3c 0 2024-04-05T21:26:28.493 INFO:tasks.workunit.client.1.smithi188.stdout:0/169: write d4/d8/d10/f1b [1469070,118160] 0 2024-04-05T21:26:28.495 INFO:tasks.workunit.client.1.smithi188.stdout:8/118: truncate d5/f24 711731 0 2024-04-05T21:26:28.496 INFO:tasks.workunit.client.1.smithi188.stdout:0/170: truncate d4/d8/db/f28 57306 0 2024-04-05T21:26:28.498 INFO:tasks.workunit.client.1.smithi188.stdout:0/171: write d4/d8/db/d21/f34 [10285,101800] 0 2024-04-05T21:26:28.499 INFO:tasks.workunit.client.1.smithi188.stdout:8/119: chown d5/f16 1489178827 1 2024-04-05T21:26:28.503 INFO:tasks.workunit.client.1.smithi188.stdout:0/172: creat d4/d8/f3d x:0 0 0 2024-04-05T21:26:28.504 INFO:tasks.workunit.client.1.smithi188.stdout:8/120: dread d5/f20 [0,4194304] 0 2024-04-05T21:26:28.507 INFO:tasks.workunit.client.1.smithi188.stdout:0/173: symlink d4/d8/db/d21/l3e 0 2024-04-05T21:26:28.508 INFO:tasks.workunit.client.1.smithi188.stdout:8/121: readlink d5/l6 0 2024-04-05T21:26:28.510 INFO:tasks.workunit.client.1.smithi188.stdout:9/178: dread dd/f12 [0,4194304] 0 2024-04-05T21:26:28.512 INFO:tasks.workunit.client.1.smithi188.stdout:9/179: chown dd/d11/d22/d28/f2d 14757 1 2024-04-05T21:26:28.513 INFO:tasks.workunit.client.1.smithi188.stdout:6/279: dwrite d6/d13/d18/d19/d2c/d3f/f56 [0,4194304] 0 2024-04-05T21:26:28.514 INFO:tasks.workunit.client.1.smithi188.stdout:8/122: chown d5/f16 5938 1 2024-04-05T21:26:28.515 INFO:tasks.workunit.client.1.smithi188.stdout:0/174: dread d4/f27 [0,4194304] 0 2024-04-05T21:26:28.516 INFO:tasks.workunit.client.1.smithi188.stdout:9/180: mknod dd/d11/d1f/d25/c36 0 2024-04-05T21:26:28.517 INFO:tasks.workunit.client.1.smithi188.stdout:8/123: symlink d5/d21/l2b 0 2024-04-05T21:26:28.519 INFO:tasks.workunit.client.1.smithi188.stdout:9/181: creat dd/d11/d22/f37 x:0 0 0 2024-04-05T21:26:28.519 INFO:tasks.workunit.client.1.smithi188.stdout:9/182: chown dd 6489 1 2024-04-05T21:26:28.522 INFO:tasks.workunit.client.1.smithi188.stdout:7/289: sync 2024-04-05T21:26:28.523 INFO:tasks.workunit.client.1.smithi188.stdout:0/175: dread d4/d6/d13/f14 [4194304,4194304] 0 2024-04-05T21:26:28.523 INFO:tasks.workunit.client.1.smithi188.stdout:6/280: dwrite d6/d7/f1e [4194304,4194304] 0 2024-04-05T21:26:28.526 INFO:tasks.workunit.client.1.smithi188.stdout:7/290: mkdir d3/d4/db/d10/d5f 0 2024-04-05T21:26:28.527 INFO:tasks.workunit.client.1.smithi188.stdout:6/281: read - d6/d13/d18/f50 zero size 2024-04-05T21:26:28.533 INFO:tasks.workunit.client.1.smithi188.stdout:0/176: write d4/d8/db/f1d [2898312,6629] 0 2024-04-05T21:26:28.536 INFO:tasks.workunit.client.1.smithi188.stdout:9/183: dwrite dd/f16 [0,4194304] 0 2024-04-05T21:26:28.540 INFO:tasks.workunit.client.1.smithi188.stdout:9/184: readlink dd/d11/l14 0 2024-04-05T21:26:28.543 INFO:tasks.workunit.client.1.smithi188.stdout:9/185: chown dd/d34/f23 0 1 2024-04-05T21:26:28.543 INFO:tasks.workunit.client.1.smithi188.stdout:7/291: dwrite d3/df/f20 [4194304,4194304] 0 2024-04-05T21:26:28.545 INFO:tasks.workunit.client.1.smithi188.stdout:6/282: dwrite d6/d13/d18/d19/f27 [0,4194304] 0 2024-04-05T21:26:28.546 INFO:tasks.workunit.client.1.smithi188.stdout:6/283: stat d6/d13 0 2024-04-05T21:26:28.554 INFO:tasks.workunit.client.1.smithi188.stdout:9/186: dwrite fb [0,4194304] 0 2024-04-05T21:26:28.558 INFO:tasks.workunit.client.1.smithi188.stdout:6/284: creat d6/d13/d18/d3c/f5d x:0 0 0 2024-04-05T21:26:28.558 INFO:tasks.workunit.client.1.smithi188.stdout:6/285: stat d6/d13/d18/f35 0 2024-04-05T21:26:28.559 INFO:tasks.workunit.client.1.smithi188.stdout:9/187: chown dd/d11/d1f/d26/c32 8611 1 2024-04-05T21:26:28.561 INFO:tasks.workunit.client.1.smithi188.stdout:9/188: mkdir dd/d11/d1f/d25/d38 0 2024-04-05T21:26:28.562 INFO:tasks.workunit.client.1.smithi188.stdout:6/286: symlink d6/d13/d18/d19/d2c/d3f/l5e 0 2024-04-05T21:26:28.564 INFO:tasks.workunit.client.1.smithi188.stdout:9/189: truncate dd/d11/d22/f37 826312 0 2024-04-05T21:26:28.569 INFO:tasks.workunit.client.1.smithi188.stdout:6/287: link d6/d13/d18/d19/d2c/d3f/c54 d6/d13/d18/d19/d2c/c5f 0 2024-04-05T21:26:28.573 INFO:tasks.workunit.client.1.smithi188.stdout:6/288: truncate d6/d13/d18/f50 899314 0 2024-04-05T21:26:28.573 INFO:tasks.workunit.client.1.smithi188.stdout:9/190: dwrite dd/d11/d1f/d26/f19 [0,4194304] 0 2024-04-05T21:26:28.574 INFO:tasks.workunit.client.1.smithi188.stdout:7/292: dwrite d3/d4/db/fe [0,4194304] 0 2024-04-05T21:26:28.576 INFO:tasks.workunit.client.1.smithi188.stdout:9/191: write f4 [2473640,63432] 0 2024-04-05T21:26:28.577 INFO:tasks.workunit.client.1.smithi188.stdout:6/289: write d6/f2f [727987,17200] 0 2024-04-05T21:26:28.581 INFO:tasks.workunit.client.1.smithi188.stdout:7/293: rename d3/d4/db/f2a to d3/d4/db/f60 0 2024-04-05T21:26:28.585 INFO:tasks.workunit.client.1.smithi188.stdout:7/294: symlink d3/l61 0 2024-04-05T21:26:28.586 INFO:tasks.workunit.client.1.smithi188.stdout:9/192: creat dd/d11/d1f/d25/d38/f39 x:0 0 0 2024-04-05T21:26:28.589 INFO:tasks.workunit.client.1.smithi188.stdout:6/290: dwrite d6/d13/d18/f59 [0,4194304] 0 2024-04-05T21:26:28.589 INFO:tasks.workunit.client.1.smithi188.stdout:7/295: chown d3/d4/db/d10/d2f/l57 14 1 2024-04-05T21:26:28.590 INFO:tasks.workunit.client.1.smithi188.stdout:9/193: truncate dd/f12 3968055 0 2024-04-05T21:26:28.594 INFO:tasks.workunit.client.1.smithi188.stdout:7/296: mknod d3/d4/db/d10/c62 0 2024-04-05T21:26:28.608 INFO:tasks.workunit.client.1.smithi188.stdout:7/297: fdatasync d3/d4/db/fe 0 2024-04-05T21:26:28.609 INFO:tasks.workunit.client.1.smithi188.stdout:7/298: readlink d3/d4/db/d2e/d37/l4b 0 2024-04-05T21:26:28.611 INFO:tasks.workunit.client.1.smithi188.stdout:9/194: unlink dd/f12 0 2024-04-05T21:26:28.612 INFO:tasks.workunit.client.1.smithi188.stdout:9/195: fsync f1 0 2024-04-05T21:26:28.612 INFO:tasks.workunit.client.1.smithi188.stdout:7/299: rmdir d3/d4/db/d2e 39 2024-04-05T21:26:28.614 INFO:tasks.workunit.client.1.smithi188.stdout:7/300: chown d3/d4/d1d/c21 72 1 2024-04-05T21:26:28.614 INFO:tasks.workunit.client.1.smithi188.stdout:6/291: dread d6/d7/f14 [0,4194304] 0 2024-04-05T21:26:28.615 INFO:tasks.workunit.client.1.smithi188.stdout:6/292: fsync d6/d13/d18/d3c/f5a 0 2024-04-05T21:26:28.618 INFO:tasks.workunit.client.1.smithi188.stdout:9/196: dread fc [0,4194304] 0 2024-04-05T21:26:28.621 INFO:tasks.workunit.client.1.smithi188.stdout:9/197: rename dd/le to dd/d35/l3a 0 2024-04-05T21:26:28.622 INFO:tasks.workunit.client.1.smithi188.stdout:9/198: fsync fc 0 2024-04-05T21:26:28.623 INFO:tasks.workunit.client.1.smithi188.stdout:6/293: dread d6/f2f [0,4194304] 0 2024-04-05T21:26:28.625 INFO:tasks.workunit.client.1.smithi188.stdout:9/199: creat dd/d11/d1f/d26/f3b x:0 0 0 2024-04-05T21:26:28.628 INFO:tasks.workunit.client.1.smithi188.stdout:9/200: write f1 [4952858,58937] 0 2024-04-05T21:26:28.628 INFO:tasks.workunit.client.1.smithi188.stdout:7/301: dwrite d3/d4/f25 [0,4194304] 0 2024-04-05T21:26:28.630 INFO:tasks.workunit.client.1.smithi188.stdout:9/201: write dd/d11/f21 [2937722,63994] 0 2024-04-05T21:26:28.631 INFO:tasks.workunit.client.1.smithi188.stdout:9/202: dread - dd/d11/d1f/d26/f3b zero size 2024-04-05T21:26:28.632 INFO:tasks.workunit.client.1.smithi188.stdout:9/203: write fc [211594,49551] 0 2024-04-05T21:26:28.638 INFO:tasks.workunit.client.1.smithi188.stdout:9/204: creat dd/d11/f3c x:0 0 0 2024-04-05T21:26:28.638 INFO:tasks.workunit.client.1.smithi188.stdout:7/302: truncate d3/f1e 655198 0 2024-04-05T21:26:28.641 INFO:tasks.workunit.client.1.smithi188.stdout:6/294: dwrite d6/d13/d38/f4c [0,4194304] 0 2024-04-05T21:26:28.641 INFO:tasks.workunit.client.1.smithi188.stdout:9/205: mkdir dd/d3d 0 2024-04-05T21:26:28.644 INFO:tasks.workunit.client.1.smithi188.stdout:9/206: symlink dd/d3d/l3e 0 2024-04-05T21:26:28.645 INFO:tasks.workunit.client.1.smithi188.stdout:6/295: mkdir d6/d13/d48/d49/d60 0 2024-04-05T21:26:28.646 INFO:tasks.workunit.client.1.smithi188.stdout:9/207: fsync dd/d11/d1f/d25/d38/f39 0 2024-04-05T21:26:28.648 INFO:tasks.workunit.client.1.smithi188.stdout:7/303: dwrite d3/d4/db/fe [0,4194304] 0 2024-04-05T21:26:28.649 INFO:tasks.workunit.client.1.smithi188.stdout:9/208: chown dd/d11/d1f/d26/f19 63688 1 2024-04-05T21:26:28.650 INFO:tasks.workunit.client.1.smithi188.stdout:7/304: chown d3/d2c/f4c 109733654 1 2024-04-05T21:26:28.650 INFO:tasks.workunit.client.1.smithi188.stdout:9/209: write fc [3026216,104663] 0 2024-04-05T21:26:28.654 INFO:tasks.workunit.client.1.smithi188.stdout:7/305: dread - d3/d4/db/d2e/f51 zero size 2024-04-05T21:26:28.655 INFO:tasks.workunit.client.1.smithi188.stdout:6/296: dread d6/d13/d18/d3c/f5a [0,4194304] 0 2024-04-05T21:26:28.657 INFO:tasks.workunit.client.1.smithi188.stdout:6/297: truncate d6/d7/f14 4684819 0 2024-04-05T21:26:28.657 INFO:tasks.workunit.client.1.smithi188.stdout:7/306: rmdir d3/d4/db/d2e/d37 39 2024-04-05T21:26:28.660 INFO:tasks.workunit.client.1.smithi188.stdout:6/298: mkdir d6/d13/d18/d19/d61 0 2024-04-05T21:26:28.661 INFO:tasks.workunit.client.1.smithi188.stdout:9/210: dwrite dd/d11/d1f/d26/f2c [0,4194304] 0 2024-04-05T21:26:28.665 INFO:tasks.workunit.client.1.smithi188.stdout:9/211: truncate dd/d11/d1f/d26/f19 4722440 0 2024-04-05T21:26:28.668 INFO:tasks.workunit.client.1.smithi188.stdout:9/212: unlink f4 0 2024-04-05T21:26:28.668 INFO:tasks.workunit.client.1.smithi188.stdout:6/299: dread d6/d13/d18/f50 [0,4194304] 0 2024-04-05T21:26:28.669 INFO:tasks.workunit.client.1.smithi188.stdout:9/213: stat dd/d11/l15 0 2024-04-05T21:26:28.670 INFO:tasks.workunit.client.1.smithi188.stdout:6/300: chown d6/d13/d18/d3c/f5d 33377 1 2024-04-05T21:26:28.671 INFO:tasks.workunit.client.1.smithi188.stdout:7/307: truncate d3/d4/d1d/f2b 2896896 0 2024-04-05T21:26:28.672 INFO:tasks.workunit.client.1.smithi188.stdout:7/308: write d3/d4/db/d2e/d3b/f43 [2236961,36678] 0 2024-04-05T21:26:28.672 INFO:tasks.workunit.client.1.smithi188.stdout:9/214: creat dd/d11/d1f/d25/f3f x:0 0 0 2024-04-05T21:26:28.673 INFO:tasks.workunit.client.1.smithi188.stdout:6/301: fdatasync d6/d28/f43 0 2024-04-05T21:26:28.676 INFO:tasks.workunit.client.1.smithi188.stdout:7/309: mkdir d3/d2c/d63 0 2024-04-05T21:26:28.677 INFO:tasks.workunit.client.1.smithi188.stdout:7/310: chown d3/d2c/f3f 15024 1 2024-04-05T21:26:28.680 INFO:tasks.workunit.client.1.smithi188.stdout:6/302: creat d6/d13/d18/d19/d61/f62 x:0 0 0 2024-04-05T21:26:28.680 INFO:tasks.workunit.client.1.smithi188.stdout:9/215: dwrite fb [0,4194304] 0 2024-04-05T21:26:28.683 INFO:tasks.workunit.client.1.smithi188.stdout:6/303: write d6/d13/d18/d3c/f5d [902600,91976] 0 2024-04-05T21:26:28.683 INFO:tasks.workunit.client.1.smithi188.stdout:9/216: chown dd/d11/d1f/d26/c32 11296516 1 2024-04-05T21:26:28.689 INFO:tasks.workunit.client.1.smithi188.stdout:9/217: mkdir dd/d11/d22/d28/d40 0 2024-04-05T21:26:28.692 INFO:tasks.workunit.client.1.smithi188.stdout:6/304: dread d6/d13/d38/f47 [0,4194304] 0 2024-04-05T21:26:28.694 INFO:tasks.workunit.client.1.smithi188.stdout:9/218: dread dd/f16 [4194304,4194304] 0 2024-04-05T21:26:28.694 INFO:tasks.workunit.client.1.smithi188.stdout:9/219: write fb [1110784,50474] 0 2024-04-05T21:26:28.695 INFO:tasks.workunit.client.1.smithi188.stdout:6/305: creat d6/d13/d38/f63 x:0 0 0 2024-04-05T21:26:28.696 INFO:tasks.workunit.client.1.smithi188.stdout:7/311: write d3/d4/db/d2e/d37/f50 [905120,14227] 0 2024-04-05T21:26:28.698 INFO:tasks.workunit.client.1.smithi188.stdout:6/306: write d6/d13/d38/f47 [1019655,21619] 0 2024-04-05T21:26:28.698 INFO:tasks.workunit.client.1.smithi188.stdout:7/312: chown d3/d4/db/d2e/d37/f50 118448 1 2024-04-05T21:26:28.699 INFO:tasks.workunit.client.1.smithi188.stdout:9/220: rename dd/l2a to dd/d11/d22/l41 0 2024-04-05T21:26:28.701 INFO:tasks.workunit.client.1.smithi188.stdout:9/221: truncate dd/f16 8847658 0 2024-04-05T21:26:28.706 INFO:tasks.workunit.client.1.smithi188.stdout:6/307: dwrite f5 [0,4194304] 0 2024-04-05T21:26:28.707 INFO:tasks.workunit.client.1.smithi188.stdout:7/313: dwrite d3/f3e [0,4194304] 0 2024-04-05T21:26:28.712 INFO:tasks.workunit.client.1.smithi188.stdout:7/314: rename d3/d4/db/d55/l5b to d3/d4/d1d/l64 0 2024-04-05T21:26:28.713 INFO:tasks.workunit.client.1.smithi188.stdout:7/315: fdatasync d3/d4/db/d2e/f35 0 2024-04-05T21:26:28.714 INFO:tasks.workunit.client.1.smithi188.stdout:9/222: truncate f1 3851425 0 2024-04-05T21:26:28.717 INFO:tasks.workunit.client.1.smithi188.stdout:7/316: rename d3/df to d3/d4/db/d2e/d37/d65 0 2024-04-05T21:26:28.719 INFO:tasks.workunit.client.1.smithi188.stdout:6/308: write d6/d13/d18/d19/f27 [4705227,47089] 0 2024-04-05T21:26:28.719 INFO:tasks.workunit.client.1.smithi188.stdout:9/223: getdents dd/d11 0 2024-04-05T21:26:28.720 INFO:tasks.workunit.client.1.smithi188.stdout:7/317: creat d3/d2c/d5e/f66 x:0 0 0 2024-04-05T21:26:28.722 INFO:tasks.workunit.client.1.smithi188.stdout:7/318: fsync d3/d4/db/d2e/d37/d65/f39 0 2024-04-05T21:26:28.723 INFO:tasks.workunit.client.1.smithi188.stdout:9/224: rename f5 to dd/f42 0 2024-04-05T21:26:28.724 INFO:tasks.workunit.client.1.smithi188.stdout:9/225: fsync fc 0 2024-04-05T21:26:28.726 INFO:tasks.workunit.client.1.smithi188.stdout:9/226: read dd/d34/f23 [4398936,74705] 0 2024-04-05T21:26:28.728 INFO:tasks.workunit.client.1.smithi188.stdout:6/309: dwrite d6/f3b [0,4194304] 0 2024-04-05T21:26:28.728 INFO:tasks.workunit.client.1.smithi188.stdout:9/227: rmdir dd/d35 39 2024-04-05T21:26:28.731 INFO:tasks.workunit.client.1.smithi188.stdout:7/319: dwrite d3/d4/f25 [0,4194304] 0 2024-04-05T21:26:28.732 INFO:tasks.workunit.client.1.smithi188.stdout:9/228: creat dd/d11/d22/d28/f43 x:0 0 0 2024-04-05T21:26:28.734 INFO:tasks.workunit.client.1.smithi188.stdout:6/310: dread d6/f11 [0,4194304] 0 2024-04-05T21:26:28.734 INFO:tasks.workunit.client.1.smithi188.stdout:9/229: creat dd/d11/d1f/d25/f44 x:0 0 0 2024-04-05T21:26:28.741 INFO:tasks.workunit.client.1.smithi188.stdout:7/320: dwrite d3/d4/db/f60 [0,4194304] 0 2024-04-05T21:26:28.744 INFO:tasks.workunit.client.1.smithi188.stdout:6/311: dwrite d6/d13/d38/f63 [0,4194304] 0 2024-04-05T21:26:28.746 INFO:tasks.workunit.client.1.smithi188.stdout:7/321: creat d3/d4/db/d2e/d49/f67 x:0 0 0 2024-04-05T21:26:28.747 INFO:tasks.workunit.client.1.smithi188.stdout:6/312: truncate d6/f2e 512848 0 2024-04-05T21:26:28.748 INFO:tasks.workunit.client.1.smithi188.stdout:6/313: stat d6/d28/f58 0 2024-04-05T21:26:28.750 INFO:tasks.workunit.client.1.smithi188.stdout:5/170: sync 2024-04-05T21:26:28.750 INFO:tasks.workunit.client.1.smithi188.stdout:4/322: sync 2024-04-05T21:26:28.753 INFO:tasks.workunit.client.1.smithi188.stdout:5/171: chown d6/d15/d1b/c32 366642 1 2024-04-05T21:26:28.755 INFO:tasks.workunit.client.1.smithi188.stdout:6/314: link d6/f2f d6/f64 0 2024-04-05T21:26:28.756 INFO:tasks.workunit.client.1.smithi188.stdout:4/323: unlink d0/d14/d15/f1a 0 2024-04-05T21:26:28.758 INFO:tasks.workunit.client.1.smithi188.stdout:6/315: fsync d6/d13/d18/f35 0 2024-04-05T21:26:28.759 INFO:tasks.workunit.client.1.smithi188.stdout:5/172: mknod d6/d15/c3e 0 2024-04-05T21:26:28.760 INFO:tasks.workunit.client.1.smithi188.stdout:4/324: mknod d0/d14/d15/d42/c7d 0 2024-04-05T21:26:28.764 INFO:tasks.workunit.client.1.smithi188.stdout:4/325: creat d0/d2/d2a/d34/d6b/f7e x:0 0 0 2024-04-05T21:26:28.766 INFO:tasks.workunit.client.1.smithi188.stdout:5/173: getdents d6/d15/d1b 0 2024-04-05T21:26:28.766 INFO:tasks.workunit.client.1.smithi188.stdout:4/326: chown d0/d2/d17/f50 634920058 1 2024-04-05T21:26:28.768 INFO:tasks.workunit.client.1.smithi188.stdout:5/174: mknod d6/d17/c3f 0 2024-04-05T21:26:28.771 INFO:tasks.workunit.client.1.smithi188.stdout:4/327: creat d0/d2/d2a/f7f x:0 0 0 2024-04-05T21:26:28.771 INFO:tasks.workunit.client.1.smithi188.stdout:6/316: dwrite d6/f2e [0,4194304] 0 2024-04-05T21:26:28.779 INFO:tasks.workunit.client.1.smithi188.stdout:5/175: dwrite d6/f19 [0,4194304] 0 2024-04-05T21:26:28.783 INFO:tasks.workunit.client.1.smithi188.stdout:6/317: creat d6/d13/d18/d19/d2c/f65 x:0 0 0 2024-04-05T21:26:28.783 INFO:tasks.workunit.client.1.smithi188.stdout:7/322: rmdir d3/d2c/d5e 39 2024-04-05T21:26:28.784 INFO:tasks.workunit.client.1.smithi188.stdout:5/176: truncate d6/fe 1144432 0 2024-04-05T21:26:28.786 INFO:tasks.workunit.client.1.smithi188.stdout:6/318: rmdir d6 39 2024-04-05T21:26:28.787 INFO:tasks.workunit.client.1.smithi188.stdout:7/323: creat d3/f68 x:0 0 0 2024-04-05T21:26:28.792 INFO:tasks.workunit.client.1.smithi188.stdout:7/324: unlink d3/d2c/l48 0 2024-04-05T21:26:28.793 INFO:tasks.workunit.client.1.smithi188.stdout:9/230: rmdir dd/d11/d22/d28 39 2024-04-05T21:26:28.795 INFO:tasks.workunit.client.1.smithi188.stdout:6/319: symlink d6/d13/d18/d19/l66 0 2024-04-05T21:26:28.796 INFO:tasks.workunit.client.1.smithi188.stdout:7/325: rmdir d3/d4/db/d2e/d37/d65 39 2024-04-05T21:26:28.800 INFO:tasks.workunit.client.1.smithi188.stdout:9/231: mknod dd/d35/c45 0 2024-04-05T21:26:28.801 INFO:tasks.workunit.client.1.smithi188.stdout:6/320: dread d6/d7/fd [4194304,4194304] 0 2024-04-05T21:26:28.804 INFO:tasks.workunit.client.1.smithi188.stdout:7/326: dwrite d3/d4/db/d2e/d37/f50 [0,4194304] 0 2024-04-05T21:26:28.805 INFO:tasks.workunit.client.1.smithi188.stdout:9/232: dread dd/d11/d22/f37 [0,4194304] 0 2024-04-05T21:26:28.810 INFO:tasks.workunit.client.1.smithi188.stdout:6/321: symlink d6/d28/l67 0 2024-04-05T21:26:28.816 INFO:tasks.workunit.client.1.smithi188.stdout:9/233: symlink dd/d35/l46 0 2024-04-05T21:26:28.816 INFO:tasks.workunit.client.1.smithi188.stdout:6/322: mknod d6/d13/d18/d19/d2c/d3f/c68 0 2024-04-05T21:26:28.822 INFO:tasks.workunit.client.1.smithi188.stdout:4/328: dread d0/f5 [4194304,4194304] 0 2024-04-05T21:26:28.822 INFO:tasks.workunit.client.1.smithi188.stdout:4/329: fdatasync d0/d2/d2a/d34/f77 0 2024-04-05T21:26:28.822 INFO:tasks.workunit.client.1.smithi188.stdout:7/327: dwrite d3/d4/db/d2e/d37/d65/f20 [4194304,4194304] 0 2024-04-05T21:26:28.823 INFO:tasks.workunit.client.1.smithi188.stdout:9/234: rmdir dd/d11/d1f 39 2024-04-05T21:26:28.824 INFO:tasks.workunit.client.1.smithi188.stdout:7/328: read d3/d4/db/f4a [42682,12326] 0 2024-04-05T21:26:28.824 INFO:tasks.workunit.client.1.smithi188.stdout:5/177: dread d6/fe [0,4194304] 0 2024-04-05T21:26:28.824 INFO:tasks.workunit.client.1.smithi188.stdout:4/330: chown d0/d2/d4f/f5c 250645 1 2024-04-05T21:26:28.825 INFO:tasks.workunit.client.1.smithi188.stdout:7/329: stat d3/d4/d1d/f34 0 2024-04-05T21:26:28.826 INFO:tasks.workunit.client.1.smithi188.stdout:5/178: read d6/f11 [3077235,45143] 0 2024-04-05T21:26:28.827 INFO:tasks.workunit.client.1.smithi188.stdout:7/330: truncate d3/d4/db/d10/d2f/f5c 986684 0 2024-04-05T21:26:28.942 INFO:tasks.workunit.client.1.smithi188.stdout:5/179: mkdir d6/d17/d23/d35/d40 0 2024-04-05T21:26:28.943 INFO:tasks.workunit.client.1.smithi188.stdout:7/331: chown d3/d4/db/d2e/d37/c42 1 1 2024-04-05T21:26:28.947 INFO:tasks.workunit.client.1.smithi188.stdout:7/332: rmdir d3/d4/db/d2e/d37/d65 39 2024-04-05T21:26:28.948 INFO:tasks.workunit.client.1.smithi188.stdout:5/180: dwrite d6/f22 [0,4194304] 0 2024-04-05T21:26:28.948 INFO:tasks.workunit.client.1.smithi188.stdout:9/235: fdatasync dd/d11/d22/d28/f43 0 2024-04-05T21:26:28.949 INFO:tasks.workunit.client.1.smithi188.stdout:9/236: readlink l8 0 2024-04-05T21:26:28.950 INFO:tasks.workunit.client.1.smithi188.stdout:4/331: creat d0/d14/f80 x:0 0 0 2024-04-05T21:26:28.950 INFO:tasks.workunit.client.1.smithi188.stdout:5/181: chown c1 44 1 2024-04-05T21:26:28.955 INFO:tasks.workunit.client.1.smithi188.stdout:5/182: unlink d6/d17/d24/f3d 0 2024-04-05T21:26:28.959 INFO:tasks.workunit.client.1.smithi188.stdout:7/333: dwrite d3/d4/db/f1c [0,4194304] 0 2024-04-05T21:26:28.960 INFO:tasks.workunit.client.1.smithi188.stdout:5/183: mknod d6/d17/d23/d35/c41 0 2024-04-05T21:26:28.961 INFO:tasks.workunit.client.1.smithi188.stdout:9/237: link c9 dd/d11/d22/d28/c47 0 2024-04-05T21:26:28.962 INFO:tasks.workunit.client.1.smithi188.stdout:5/184: readlink d6/l14 0 2024-04-05T21:26:28.962 INFO:tasks.workunit.client.1.smithi188.stdout:4/332: dwrite d0/d2/d4f/f67 [0,4194304] 0 2024-04-05T21:26:28.963 INFO:tasks.workunit.client.1.smithi188.stdout:7/334: chown d3/d4/db/d2e/d49/f67 279456 1 2024-04-05T21:26:28.966 INFO:tasks.workunit.client.1.smithi188.stdout:9/238: creat dd/d11/d1f/d26/f48 x:0 0 0 2024-04-05T21:26:28.969 INFO:tasks.workunit.client.1.smithi188.stdout:9/239: read - dd/d11/d1f/f29 zero size 2024-04-05T21:26:28.970 INFO:tasks.workunit.client.1.smithi188.stdout:9/240: stat dd/d11/d1f/d25/f44 0 2024-04-05T21:26:28.973 INFO:tasks.workunit.client.1.smithi188.stdout:9/241: mkdir dd/d49 0 2024-04-05T21:26:28.973 INFO:tasks.workunit.client.1.smithi188.stdout:4/333: dread d0/f62 [0,4194304] 0 2024-04-05T21:26:28.976 INFO:tasks.workunit.client.1.smithi188.stdout:4/334: mkdir d0/d81 0 2024-04-05T21:26:28.977 INFO:tasks.workunit.client.1.smithi188.stdout:5/185: dread d6/f2d [0,4194304] 0 2024-04-05T21:26:28.978 INFO:tasks.workunit.client.1.smithi188.stdout:4/335: write d0/d14/d15/d53/f74 [861065,38633] 0 2024-04-05T21:26:28.980 INFO:tasks.workunit.client.1.smithi188.stdout:5/186: chown d6/d15/d1b/f34 40588 1 2024-04-05T21:26:28.980 INFO:tasks.workunit.client.1.smithi188.stdout:9/242: dread fc [0,4194304] 0 2024-04-05T21:26:28.983 INFO:tasks.workunit.client.1.smithi188.stdout:9/243: symlink dd/d11/d22/d28/l4a 0 2024-04-05T21:26:28.985 INFO:tasks.workunit.client.1.smithi188.stdout:9/244: chown dd/d11/d1f/d25/d38 484981454 1 2024-04-05T21:26:28.986 INFO:tasks.workunit.client.1.smithi188.stdout:4/336: dwrite d0/d2/d2a/d34/f77 [0,4194304] 0 2024-04-05T21:26:28.987 INFO:tasks.workunit.client.1.smithi188.stdout:9/245: symlink dd/d49/l4b 0 2024-04-05T21:26:28.988 INFO:tasks.workunit.client.1.smithi188.stdout:4/337: write d0/f64 [679631,24308] 0 2024-04-05T21:26:28.992 INFO:tasks.workunit.client.1.smithi188.stdout:4/338: mkdir d0/d14/d15/d53/d82 0 2024-04-05T21:26:28.995 INFO:tasks.workunit.client.1.smithi188.stdout:5/187: rmdir d6/d17/d23/d35 39 2024-04-05T21:26:28.996 INFO:tasks.workunit.client.1.smithi188.stdout:4/339: creat d0/d14/d2f/d35/f83 x:0 0 0 2024-04-05T21:26:28.997 INFO:tasks.workunit.client.1.smithi188.stdout:7/335: dread d3/d4/db/d2e/d37/d65/f20 [0,4194304] 0 2024-04-05T21:26:28.997 INFO:tasks.workunit.client.1.smithi188.stdout:9/246: dwrite dd/d11/d1f/d25/f44 [0,4194304] 0 2024-04-05T21:26:29.003 INFO:tasks.workunit.client.1.smithi188.stdout:3/255: sync 2024-04-05T21:26:29.003 INFO:tasks.workunit.client.1.smithi188.stdout:1/192: sync 2024-04-05T21:26:29.005 INFO:tasks.workunit.client.1.smithi188.stdout:1/193: fdatasync d1/dc/d16/d21/f26 0 2024-04-05T21:26:29.006 INFO:tasks.workunit.client.1.smithi188.stdout:1/194: stat d1/d7/c1a 0 2024-04-05T21:26:29.007 INFO:tasks.workunit.client.1.smithi188.stdout:7/336: creat d3/f69 x:0 0 0 2024-04-05T21:26:29.007 INFO:tasks.workunit.client.1.smithi188.stdout:9/247: stat dd/d35/l3a 0 2024-04-05T21:26:29.008 INFO:tasks.workunit.client.1.smithi188.stdout:9/248: readlink l8 0 2024-04-05T21:26:29.009 INFO:tasks.workunit.client.1.smithi188.stdout:9/249: dread - dd/d11/d1f/d26/f48 zero size 2024-04-05T21:26:29.011 INFO:tasks.workunit.client.1.smithi188.stdout:5/188: mkdir d6/d17/d23/d35/d40/d42 0 2024-04-05T21:26:29.012 INFO:tasks.workunit.client.1.smithi188.stdout:7/337: mknod d3/c6a 0 2024-04-05T21:26:29.013 INFO:tasks.workunit.client.1.smithi188.stdout:7/338: stat d3/l61 0 2024-04-05T21:26:29.014 INFO:tasks.workunit.client.1.smithi188.stdout:9/250: dread dd/f16 [0,4194304] 0 2024-04-05T21:26:29.016 INFO:tasks.workunit.client.1.smithi188.stdout:4/340: getdents d0/d2 0 2024-04-05T21:26:29.016 INFO:tasks.workunit.client.1.smithi188.stdout:1/195: creat d1/d4/f3d x:0 0 0 2024-04-05T21:26:29.019 INFO:tasks.workunit.client.1.smithi188.stdout:4/341: truncate d0/d2/d17/d3a/f54 1401626 0 2024-04-05T21:26:29.020 INFO:tasks.workunit.client.1.smithi188.stdout:7/339: creat d3/d4/db/d2e/f6b x:0 0 0 2024-04-05T21:26:29.020 INFO:tasks.workunit.client.1.smithi188.stdout:1/196: symlink d1/dc/d16/d21/d27/l3e 0 2024-04-05T21:26:29.022 INFO:tasks.workunit.client.1.smithi188.stdout:4/342: fsync d0/d2/d2a/d34/f77 0 2024-04-05T21:26:29.023 INFO:tasks.workunit.client.1.smithi188.stdout:7/340: dread - d3/d4/db/d2e/f51 zero size 2024-04-05T21:26:29.024 INFO:tasks.workunit.client.1.smithi188.stdout:3/256: write d5/d2b/f2e [1861284,69306] 0 2024-04-05T21:26:29.027 INFO:tasks.workunit.client.1.smithi188.stdout:4/343: mknod d0/d14/d15/d53/c84 0 2024-04-05T21:26:29.028 INFO:tasks.workunit.client.1.smithi188.stdout:9/251: dwrite dd/d11/d1f/d26/f19 [0,4194304] 0 2024-04-05T21:26:29.030 INFO:tasks.workunit.client.1.smithi188.stdout:3/257: creat d5/d7/d31/f5c x:0 0 0 2024-04-05T21:26:29.032 INFO:tasks.workunit.client.1.smithi188.stdout:7/341: mknod d3/d4/c6c 0 2024-04-05T21:26:29.033 INFO:tasks.workunit.client.1.smithi188.stdout:9/252: unlink dd/d11/d22/l31 0 2024-04-05T21:26:29.036 INFO:tasks.workunit.client.1.smithi188.stdout:4/344: creat d0/d81/f85 x:0 0 0 2024-04-05T21:26:29.036 INFO:tasks.workunit.client.1.smithi188.stdout:5/189: dwrite d6/d15/d1b/f2a [0,4194304] 0 2024-04-05T21:26:29.038 INFO:tasks.workunit.client.1.smithi188.stdout:1/197: dwrite d1/dc/d16/d21/f26 [0,4194304] 0 2024-04-05T21:26:29.039 INFO:tasks.workunit.client.1.smithi188.stdout:9/253: creat dd/d3d/f4c x:0 0 0 2024-04-05T21:26:29.042 INFO:tasks.workunit.client.1.smithi188.stdout:1/198: write d1/f24 [937552,4723] 0 2024-04-05T21:26:29.042 INFO:tasks.workunit.client.1.smithi188.stdout:9/254: readlink dd/d11/d22/d28/l4a 0 2024-04-05T21:26:29.044 INFO:tasks.workunit.client.1.smithi188.stdout:5/190: symlink d6/d17/d23/d35/d40/l43 0 2024-04-05T21:26:29.046 INFO:tasks.workunit.client.1.smithi188.stdout:3/258: link d5/d2b/d43/c47 d5/d7/d31/c5d 0 2024-04-05T21:26:29.049 INFO:tasks.workunit.client.1.smithi188.stdout:7/342: dwrite d3/d4/db/f38 [0,4194304] 0 2024-04-05T21:26:29.050 INFO:tasks.workunit.client.1.smithi188.stdout:4/345: creat d0/d14/f86 x:0 0 0 2024-04-05T21:26:29.051 INFO:tasks.workunit.client.1.smithi188.stdout:5/191: mkdir d6/d17/d24/d44 0 2024-04-05T21:26:29.052 INFO:tasks.workunit.client.1.smithi188.stdout:1/199: mkdir d1/d4/d3f 0 2024-04-05T21:26:29.055 INFO:tasks.workunit.client.1.smithi188.stdout:3/259: write d5/d7/f20 [4011860,52610] 0 2024-04-05T21:26:29.059 INFO:tasks.workunit.client.1.smithi188.stdout:9/255: creat dd/d11/d22/f4d x:0 0 0 2024-04-05T21:26:29.060 INFO:tasks.workunit.client.1.smithi188.stdout:4/346: write d0/d2/d17/f4e [98826,104764] 0 2024-04-05T21:26:29.062 INFO:tasks.workunit.client.1.smithi188.stdout:7/343: rename d3/d4/c58 to d3/d4/db/d2e/c6d 0 2024-04-05T21:26:29.068 INFO:tasks.workunit.client.1.smithi188.stdout:9/256: creat dd/d11/d22/d28/d40/f4e x:0 0 0 2024-04-05T21:26:29.072 INFO:tasks.workunit.client.1.smithi188.stdout:5/192: dwrite d6/d15/d1b/f34 [0,4194304] 0 2024-04-05T21:26:29.072 INFO:tasks.workunit.client.1.smithi188.stdout:1/200: dwrite d1/dc/f1b [0,4194304] 0 2024-04-05T21:26:29.072 INFO:tasks.workunit.client.1.smithi188.stdout:7/344: dread - d3/d4/db/d2e/d49/f5d zero size 2024-04-05T21:26:29.078 INFO:tasks.workunit.client.1.smithi188.stdout:3/260: dwrite d5/d7/f20 [0,4194304] 0 2024-04-05T21:26:29.080 INFO:tasks.workunit.client.1.smithi188.stdout:5/193: creat d6/d15/d1b/f45 x:0 0 0 2024-04-05T21:26:29.081 INFO:tasks.workunit.client.1.smithi188.stdout:9/257: fsync dd/f42 0 2024-04-05T21:26:29.082 INFO:tasks.workunit.client.1.smithi188.stdout:5/194: fdatasync d6/d17/d24/f3c 0 2024-04-05T21:26:29.082 INFO:tasks.workunit.client.1.smithi188.stdout:3/261: truncate d5/dc/f5a 1015652 0 2024-04-05T21:26:29.086 INFO:tasks.workunit.client.1.smithi188.stdout:1/201: fsync d1/dc/d16/d21/d27/f29 0 2024-04-05T21:26:29.088 INFO:tasks.workunit.client.1.smithi188.stdout:1/202: fdatasync d1/d4/f3d 0 2024-04-05T21:26:29.089 INFO:tasks.workunit.client.1.smithi188.stdout:5/195: creat d6/d17/f46 x:0 0 0 2024-04-05T21:26:29.091 INFO:tasks.workunit.client.1.smithi188.stdout:3/262: creat d5/d7/f5e x:0 0 0 2024-04-05T21:26:29.092 INFO:tasks.workunit.client.1.smithi188.stdout:9/258: mknod dd/d11/c4f 0 2024-04-05T21:26:29.093 INFO:tasks.workunit.client.1.smithi188.stdout:7/345: creat d3/d4/db/f6e x:0 0 0 2024-04-05T21:26:29.094 INFO:tasks.workunit.client.1.smithi188.stdout:3/263: mkdir d5/d2b/d43/d5f 0 2024-04-05T21:26:29.095 INFO:tasks.workunit.client.1.smithi188.stdout:7/346: read - d3/d4/db/f6e zero size 2024-04-05T21:26:29.095 INFO:tasks.workunit.client.1.smithi188.stdout:3/264: chown d5/d35/f40 1 1 2024-04-05T21:26:29.096 INFO:tasks.workunit.client.1.smithi188.stdout:7/347: chown d3/d4/db/d2e 43 1 2024-04-05T21:26:29.096 INFO:tasks.workunit.client.1.smithi188.stdout:5/196: rename d6/f19 to d6/d17/d23/d35/f47 0 2024-04-05T21:26:29.097 INFO:tasks.workunit.client.1.smithi188.stdout:1/203: creat d1/d4/d3f/f40 x:0 0 0 2024-04-05T21:26:29.103 INFO:tasks.workunit.client.1.smithi188.stdout:3/265: dread d5/d7/f30 [0,4194304] 0 2024-04-05T21:26:29.103 INFO:tasks.workunit.client.1.smithi188.stdout:4/347: dread d0/f5 [0,4194304] 0 2024-04-05T21:26:29.105 INFO:tasks.workunit.client.1.smithi188.stdout:7/348: rename d3/d4/db/d2e/d37/f4e to d3/f6f 0 2024-04-05T21:26:29.105 INFO:tasks.workunit.client.1.smithi188.stdout:4/348: fdatasync d0/d14/f86 0 2024-04-05T21:26:29.105 INFO:tasks.workunit.client.1.smithi188.stdout:5/197: symlink d6/d15/l48 0 2024-04-05T21:26:29.106 INFO:tasks.workunit.client.1.smithi188.stdout:7/349: write d3/f69 [782471,45650] 0 2024-04-05T21:26:29.107 INFO:tasks.workunit.client.1.smithi188.stdout:4/349: write d0/d14/f6e [1041459,70580] 0 2024-04-05T21:26:29.111 INFO:tasks.workunit.client.1.smithi188.stdout:7/350: write d3/d4/db/d2e/d37/d65/f1f [8869313,115052] 0 2024-04-05T21:26:29.112 INFO:tasks.workunit.client.1.smithi188.stdout:9/259: getdents dd/d49 0 2024-04-05T21:26:29.117 INFO:tasks.workunit.client.1.smithi188.stdout:4/350: getdents d0/d2/d2a/d71 0 2024-04-05T21:26:29.117 INFO:tasks.workunit.client.1.smithi188.stdout:1/204: dread d1/dc/f23 [0,4194304] 0 2024-04-05T21:26:29.118 INFO:tasks.workunit.client.1.smithi188.stdout:9/260: write dd/d11/d1f/d26/f2c [3632655,97795] 0 2024-04-05T21:26:29.120 INFO:tasks.workunit.client.1.smithi188.stdout:5/198: link d6/d15/l33 d6/d15/d1b/l49 0 2024-04-05T21:26:29.121 INFO:tasks.workunit.client.1.smithi188.stdout:4/351: write d0/d14/d15/d53/f74 [558611,105903] 0 2024-04-05T21:26:29.125 INFO:tasks.workunit.client.1.smithi188.stdout:4/352: read - d0/d2/d2a/f7f zero size 2024-04-05T21:26:29.132 INFO:tasks.workunit.client.1.smithi188.stdout:7/351: dwrite d3/d4/db/d10/d2f/f5c [0,4194304] 0 2024-04-05T21:26:29.132 INFO:tasks.workunit.client.1.smithi188.stdout:3/266: dwrite d5/d2b/d43/f54 [0,4194304] 0 2024-04-05T21:26:29.135 INFO:tasks.workunit.client.1.smithi188.stdout:1/205: write d1/f14 [586428,23700] 0 2024-04-05T21:26:29.141 INFO:tasks.workunit.client.1.smithi188.stdout:9/261: link f1 dd/d11/d1f/d25/f50 0 2024-04-05T21:26:29.142 INFO:tasks.workunit.client.1.smithi188.stdout:1/206: write d1/dc/d16/f3c [78428,95472] 0 2024-04-05T21:26:29.145 INFO:tasks.workunit.client.1.smithi188.stdout:7/352: mknod d3/d2c/c70 0 2024-04-05T21:26:29.145 INFO:tasks.workunit.client.1.smithi188.stdout:4/353: dread d0/d14/f6e [0,4194304] 0 2024-04-05T21:26:29.145 INFO:tasks.workunit.client.1.smithi188.stdout:3/267: dwrite d5/f1e [0,4194304] 0 2024-04-05T21:26:29.149 INFO:tasks.workunit.client.1.smithi188.stdout:9/262: dread - dd/d11/d1f/d26/f3b zero size 2024-04-05T21:26:29.149 INFO:tasks.workunit.client.1.smithi188.stdout:5/199: dwrite d6/d17/d24/f2b [0,4194304] 0 2024-04-05T21:26:29.154 INFO:tasks.workunit.client.1.smithi188.stdout:3/268: dread - d5/dc/f2d zero size 2024-04-05T21:26:29.154 INFO:tasks.workunit.client.1.smithi188.stdout:9/263: fsync dd/d11/d1f/d26/f19 0 2024-04-05T21:26:29.154 INFO:tasks.workunit.client.1.smithi188.stdout:7/353: chown d3/d4/db/d2e/d37/c42 27905 1 2024-04-05T21:26:29.154 INFO:tasks.workunit.client.1.smithi188.stdout:8/124: sync 2024-04-05T21:26:29.154 INFO:tasks.workunit.client.1.smithi188.stdout:2/241: sync 2024-04-05T21:26:29.154 INFO:tasks.workunit.client.1.smithi188.stdout:9/264: dread - dd/d11/d22/d28/d40/f4e zero size 2024-04-05T21:26:29.157 INFO:tasks.workunit.client.1.smithi188.stdout:3/269: symlink d5/df/l60 0 2024-04-05T21:26:29.158 INFO:tasks.workunit.client.1.smithi188.stdout:8/125: creat d5/dc/f2c x:0 0 0 2024-04-05T21:26:29.159 INFO:tasks.workunit.client.1.smithi188.stdout:8/126: chown d5/f9 69 1 2024-04-05T21:26:29.160 INFO:tasks.workunit.client.1.smithi188.stdout:9/265: fsync dd/d11/d1f/d26/f19 0 2024-04-05T21:26:29.161 INFO:tasks.workunit.client.1.smithi188.stdout:1/207: rename d1/ld to d1/dc/d16/d21/d27/l41 0 2024-04-05T21:26:29.161 INFO:tasks.workunit.client.1.smithi188.stdout:5/200: stat d6/c1a 0 2024-04-05T21:26:29.164 INFO:tasks.workunit.client.1.smithi188.stdout:7/354: dread d3/d4/db/f60 [0,4194304] 0 2024-04-05T21:26:29.165 INFO:tasks.workunit.client.1.smithi188.stdout:3/270: mkdir d5/d2b/d43/d61 0 2024-04-05T21:26:29.167 INFO:tasks.workunit.client.1.smithi188.stdout:1/208: write d1/dc/d16/d21/f26 [2480352,101405] 0 2024-04-05T21:26:29.167 INFO:tasks.workunit.client.1.smithi188.stdout:8/127: getdents d5/d21/d2a 0 2024-04-05T21:26:29.168 INFO:tasks.workunit.client.1.smithi188.stdout:9/266: truncate f1 4312495 0 2024-04-05T21:26:29.170 INFO:tasks.workunit.client.1.smithi188.stdout:7/355: creat d3/f71 x:0 0 0 2024-04-05T21:26:29.171 INFO:tasks.workunit.client.1.smithi188.stdout:9/267: chown dd/d11/d1f/d26/f3b 5 1 2024-04-05T21:26:29.174 INFO:tasks.workunit.client.1.smithi188.stdout:5/201: dwrite d6/f9 [0,4194304] 0 2024-04-05T21:26:29.176 INFO:tasks.workunit.client.1.smithi188.stdout:5/202: chown d6/d17/c1c 6244 1 2024-04-05T21:26:29.176 INFO:tasks.workunit.client.1.smithi188.stdout:4/354: rename d0/d14/d15/d1b/f1d to d0/d2/d2a/d71/f87 0 2024-04-05T21:26:29.179 INFO:tasks.workunit.client.1.smithi188.stdout:4/355: write d0/d2/d2a/f7f [575170,120713] 0 2024-04-05T21:26:29.181 INFO:tasks.workunit.client.1.smithi188.stdout:9/268: mkdir dd/d11/d1f/d25/d51 0 2024-04-05T21:26:29.182 INFO:tasks.workunit.client.1.smithi188.stdout:7/356: dread d3/d4/db/fe [0,4194304] 0 2024-04-05T21:26:29.183 INFO:tasks.workunit.client.1.smithi188.stdout:1/209: rename d1/f10 to d1/dc/d16/d2d/f42 0 2024-04-05T21:26:29.185 INFO:tasks.workunit.client.1.smithi188.stdout:4/356: rmdir d0/d2/d2a/d34/d6b 39 2024-04-05T21:26:29.187 INFO:tasks.workunit.client.1.smithi188.stdout:4/357: write d0/d2/d4f/f67 [3347663,17521] 0 2024-04-05T21:26:29.190 INFO:tasks.workunit.client.1.smithi188.stdout:9/269: rename dd/d11/d1f/d25/f3f to dd/d49/f52 0 2024-04-05T21:26:29.190 INFO:tasks.workunit.client.1.smithi188.stdout:7/357: dread d3/d4/db/f4a [0,4194304] 0 2024-04-05T21:26:29.191 INFO:tasks.workunit.client.1.smithi188.stdout:5/203: link d6/d17/d24/f2b d6/d17/d39/f4a 0 2024-04-05T21:26:29.192 INFO:tasks.workunit.client.1.smithi188.stdout:1/210: unlink d1/dc/f19 0 2024-04-05T21:26:29.194 INFO:tasks.workunit.client.1.smithi188.stdout:8/128: link d5/f24 d5/f2d 0 2024-04-05T21:26:29.198 INFO:tasks.workunit.client.1.smithi188.stdout:9/270: truncate dd/f42 2117913 0 2024-04-05T21:26:29.204 INFO:tasks.workunit.client.1.smithi188.stdout:9/271: chown dd/d11/l15 888 1 2024-04-05T21:26:29.204 INFO:tasks.workunit.client.1.smithi188.stdout:9/272: chown l8 6513126 1 2024-04-05T21:26:29.204 INFO:tasks.workunit.client.1.smithi188.stdout:5/204: mknod d6/d17/d24/d44/c4b 0 2024-04-05T21:26:29.205 INFO:tasks.workunit.client.1.smithi188.stdout:1/211: symlink d1/d4/l43 0 2024-04-05T21:26:29.206 INFO:tasks.workunit.client.1.smithi188.stdout:5/205: chown d6/d17/d24 962568 1 2024-04-05T21:26:29.206 INFO:tasks.workunit.client.1.smithi188.stdout:4/358: rmdir d0/d14/d6c 0 2024-04-05T21:26:29.207 INFO:tasks.workunit.client.1.smithi188.stdout:7/358: creat d3/d2c/f72 x:0 0 0 2024-04-05T21:26:29.210 INFO:tasks.workunit.client.1.smithi188.stdout:1/212: rmdir d1/dc/d16/d2d 39 2024-04-05T21:26:29.210 INFO:tasks.workunit.client.1.smithi188.stdout:9/273: creat dd/d49/f53 x:0 0 0 2024-04-05T21:26:29.211 INFO:tasks.workunit.client.1.smithi188.stdout:4/359: creat d0/d14/d15/f88 x:0 0 0 2024-04-05T21:26:29.212 INFO:tasks.workunit.client.1.smithi188.stdout:5/206: rename d6/d17/d23/d35/d40/l43 to d6/d17/d39/l4c 0 2024-04-05T21:26:29.214 INFO:tasks.workunit.client.1.smithi188.stdout:4/360: dread - d0/d14/f80 zero size 2024-04-05T21:26:29.217 INFO:tasks.workunit.client.1.smithi188.stdout:8/129: dread d5/f1f [0,4194304] 0 2024-04-05T21:26:29.218 INFO:tasks.workunit.client.1.smithi188.stdout:9/274: rename dd/d11/d22/d28/l4a to dd/d11/d22/d28/l54 0 2024-04-05T21:26:29.220 INFO:tasks.workunit.client.1.smithi188.stdout:7/359: dwrite d3/d2c/f4c [0,4194304] 0 2024-04-05T21:26:29.222 INFO:tasks.workunit.client.1.smithi188.stdout:4/361: creat d0/d14/d15/f89 x:0 0 0 2024-04-05T21:26:29.223 INFO:tasks.workunit.client.1.smithi188.stdout:1/213: link d1/d4/d3f/f40 d1/d4/d3f/f44 0 2024-04-05T21:26:29.227 INFO:tasks.workunit.client.1.smithi188.stdout:9/275: mkdir dd/d11/d1f/d26/d55 0 2024-04-05T21:26:29.228 INFO:tasks.workunit.client.1.smithi188.stdout:7/360: mkdir d3/d4/db/d10/d2f/d73 0 2024-04-05T21:26:29.229 INFO:tasks.workunit.client.1.smithi188.stdout:1/214: creat d1/d4/d3f/f45 x:0 0 0 2024-04-05T21:26:29.230 INFO:tasks.workunit.client.1.smithi188.stdout:1/215: chown d1/d4/f36 120 1 2024-04-05T21:26:29.231 INFO:tasks.workunit.client.1.smithi188.stdout:1/216: chown d1/dc/d16/d21 12048740 1 2024-04-05T21:26:29.232 INFO:tasks.workunit.client.1.smithi188.stdout:9/276: write dd/d11/f21 [3627791,117596] 0 2024-04-05T21:26:29.232 INFO:tasks.workunit.client.1.smithi188.stdout:1/217: chown d1/d7/f39 26 1 2024-04-05T21:26:29.233 INFO:tasks.workunit.client.1.smithi188.stdout:9/277: fdatasync dd/d11/d22/f4d 0 2024-04-05T21:26:29.237 INFO:tasks.workunit.client.1.smithi188.stdout:8/130: getdents d5/dc 0 2024-04-05T21:26:29.240 INFO:tasks.workunit.client.1.smithi188.stdout:1/218: mkdir d1/dc/d16/d46 0 2024-04-05T21:26:29.241 INFO:tasks.workunit.client.1.smithi188.stdout:7/361: rename d3/d4/db/d10/d2f to d3/d4/d1d/d74 0 2024-04-05T21:26:29.242 INFO:tasks.workunit.client.1.smithi188.stdout:5/207: truncate d6/d17/d23/d35/f47 3119721 0 2024-04-05T21:26:29.244 INFO:tasks.workunit.client.1.smithi188.stdout:9/278: dwrite dd/d11/d22/d28/f2d [4194304,4194304] 0 2024-04-05T21:26:29.245 INFO:tasks.workunit.client.1.smithi188.stdout:4/362: truncate d0/d2/d17/d3a/f54 750796 0 2024-04-05T21:26:29.249 INFO:tasks.workunit.client.1.smithi188.stdout:9/279: write dd/d3d/f4c [780340,46249] 0 2024-04-05T21:26:29.251 INFO:tasks.workunit.client.1.smithi188.stdout:8/131: dwrite d5/f9 [0,4194304] 0 2024-04-05T21:26:29.254 INFO:tasks.workunit.client.1.smithi188.stdout:9/280: rmdir dd/d11/d22 39 2024-04-05T21:26:29.257 INFO:tasks.workunit.client.1.smithi188.stdout:4/363: unlink d0/d2/d17/d48/l4b 0 2024-04-05T21:26:29.257 INFO:tasks.workunit.client.1.smithi188.stdout:1/219: dwrite d1/dc/d16/d2d/f42 [0,4194304] 0 2024-04-05T21:26:29.258 INFO:tasks.workunit.client.1.smithi188.stdout:8/132: creat d5/d21/f2e x:0 0 0 2024-04-05T21:26:29.261 INFO:tasks.workunit.client.1.smithi188.stdout:1/220: write d1/dc/f1b [4031659,60208] 0 2024-04-05T21:26:29.266 INFO:tasks.workunit.client.1.smithi188.stdout:8/133: dread d5/f1f [0,4194304] 0 2024-04-05T21:26:29.269 INFO:tasks.workunit.client.1.smithi188.stdout:7/362: truncate d3/d4/d1d/d74/f41 192069 0 2024-04-05T21:26:29.271 INFO:tasks.workunit.client.1.smithi188.stdout:4/364: creat d0/d14/d15/d53/d82/f8a x:0 0 0 2024-04-05T21:26:29.273 INFO:tasks.workunit.client.1.smithi188.stdout:1/221: creat d1/dc/d16/f47 x:0 0 0 2024-04-05T21:26:29.278 INFO:tasks.workunit.client.1.smithi188.stdout:8/134: creat d5/f2f x:0 0 0 2024-04-05T21:26:29.279 INFO:tasks.workunit.client.1.smithi188.stdout:4/365: mknod d0/d2/d2a/d34/d6b/c8b 0 2024-04-05T21:26:29.280 INFO:tasks.workunit.client.1.smithi188.stdout:9/281: link dd/d11/f3c dd/d11/d22/f56 0 2024-04-05T21:26:29.282 INFO:tasks.workunit.client.1.smithi188.stdout:7/363: rename d3/d4/db/d2e/d3b/f43 to d3/d4/db/d2e/f75 0 2024-04-05T21:26:29.283 INFO:tasks.workunit.client.1.smithi188.stdout:8/135: symlink d5/dc/l30 0 2024-04-05T21:26:29.284 INFO:tasks.workunit.client.1.smithi188.stdout:8/136: chown d5/dc/ld 8264 1 2024-04-05T21:26:29.286 INFO:tasks.workunit.client.1.smithi188.stdout:8/137: write d5/f9 [3700630,19088] 0 2024-04-05T21:26:29.286 INFO:tasks.workunit.client.1.smithi188.stdout:9/282: mknod dd/d35/c57 0 2024-04-05T21:26:29.292 INFO:tasks.workunit.client.1.smithi188.stdout:9/283: mknod dd/d11/d1f/d25/c58 0 2024-04-05T21:26:29.294 INFO:tasks.workunit.client.1.smithi188.stdout:8/138: creat d5/d21/d2a/f31 x:0 0 0 2024-04-05T21:26:29.294 INFO:tasks.workunit.client.1.smithi188.stdout:8/139: dread - d5/f29 zero size 2024-04-05T21:26:29.295 INFO:tasks.workunit.client.1.smithi188.stdout:8/140: write d5/f12 [120881,47043] 0 2024-04-05T21:26:29.296 INFO:tasks.workunit.client.1.smithi188.stdout:9/284: mkdir dd/d11/d22/d28/d40/d59 0 2024-04-05T21:26:29.297 INFO:tasks.workunit.client.1.smithi188.stdout:1/222: truncate d1/dc/f23 1238701 0 2024-04-05T21:26:29.297 INFO:tasks.workunit.client.1.smithi188.stdout:8/141: write d5/f29 [449055,19614] 0 2024-04-05T21:26:29.298 INFO:tasks.workunit.client.1.smithi188.stdout:8/142: dread - d5/f2f zero size 2024-04-05T21:26:29.299 INFO:tasks.workunit.client.1.smithi188.stdout:8/143: read - d5/dc/f19 zero size 2024-04-05T21:26:29.299 INFO:tasks.workunit.client.1.smithi188.stdout:9/285: creat dd/d11/d22/f5a x:0 0 0 2024-04-05T21:26:29.302 INFO:tasks.workunit.client.1.smithi188.stdout:1/223: creat d1/f48 x:0 0 0 2024-04-05T21:26:29.302 INFO:tasks.workunit.client.1.smithi188.stdout:1/224: chown d1/dc/f15 673 1 2024-04-05T21:26:29.303 INFO:tasks.workunit.client.1.smithi188.stdout:9/286: unlink dd/d11/d1f/d26/c32 0 2024-04-05T21:26:29.305 INFO:tasks.workunit.client.1.smithi188.stdout:8/144: rename d5/f16 to d5/dc/f32 0 2024-04-05T21:26:29.306 INFO:tasks.workunit.client.1.smithi188.stdout:9/287: readlink dd/l30 0 2024-04-05T21:26:29.310 INFO:tasks.workunit.client.1.smithi188.stdout:1/225: dwrite d1/dc/d16/f33 [0,4194304] 0 2024-04-05T21:26:29.311 INFO:tasks.workunit.client.1.smithi188.stdout:1/226: chown d1/d4/d3f/f45 466047260 1 2024-04-05T21:26:29.311 INFO:tasks.workunit.client.1.smithi188.stdout:1/227: fdatasync d1/dc/f15 0 2024-04-05T21:26:29.312 INFO:tasks.workunit.client.1.smithi188.stdout:9/288: rename dd/d11/d22/d28/c47 to dd/d11/d1f/d25/c5b 0 2024-04-05T21:26:29.312 INFO:tasks.workunit.client.1.smithi188.stdout:1/228: stat d1/d4/d3f/f45 0 2024-04-05T21:26:29.314 INFO:tasks.workunit.client.1.smithi188.stdout:9/289: creat dd/d49/f5c x:0 0 0 2024-04-05T21:26:29.315 INFO:tasks.workunit.client.1.smithi188.stdout:9/290: readlink dd/d34/l24 0 2024-04-05T21:26:29.315 INFO:tasks.workunit.client.1.smithi188.stdout:1/229: mkdir d1/dc/d16/d21/d27/d49 0 2024-04-05T21:26:29.319 INFO:tasks.workunit.client.1.smithi188.stdout:1/230: getdents d1/dc/d16/d21 0 2024-04-05T21:26:29.320 INFO:tasks.workunit.client.1.smithi188.stdout:1/231: fsync d1/f14 0 2024-04-05T21:26:29.323 INFO:tasks.workunit.client.1.smithi188.stdout:9/291: dwrite dd/d34/f23 [8388608,4194304] 0 2024-04-05T21:26:29.340 INFO:tasks.workunit.client.1.smithi188.stdout:6/323: sync 2024-04-05T21:26:29.340 INFO:tasks.workunit.client.1.smithi188.stdout:0/177: sync 2024-04-05T21:26:29.344 INFO:tasks.workunit.client.1.smithi188.stdout:0/178: dread d4/d8/db/f28 [0,4194304] 0 2024-04-05T21:26:29.347 INFO:tasks.workunit.client.1.smithi188.stdout:0/179: rmdir d4/d6/d13/d23 39 2024-04-05T21:26:29.349 INFO:tasks.workunit.client.1.smithi188.stdout:6/324: rename d6/f11 to d6/d13/d48/d49/f69 0 2024-04-05T21:26:29.349 INFO:tasks.workunit.client.1.smithi188.stdout:6/325: fdatasync d6/d13/d18/d19/d2c/d3f/f56 0 2024-04-05T21:26:29.351 INFO:tasks.workunit.client.1.smithi188.stdout:6/326: chown d6/d13/d18/d19/l66 601371 1 2024-04-05T21:26:29.352 INFO:tasks.workunit.client.1.smithi188.stdout:0/180: getdents d4/d2e 0 2024-04-05T21:26:29.354 INFO:tasks.workunit.client.1.smithi188.stdout:6/327: symlink d6/d7/l6a 0 2024-04-05T21:26:29.355 INFO:tasks.workunit.client.1.smithi188.stdout:1/232: read d1/dc/f23 [188249,38289] 0 2024-04-05T21:26:29.355 INFO:tasks.workunit.client.1.smithi188.stdout:0/181: chown d4/d8/d10/l2d 12872 1 2024-04-05T21:26:29.356 INFO:tasks.workunit.client.1.smithi188.stdout:6/328: read d6/d7/f14 [1930309,118934] 0 2024-04-05T21:26:29.357 INFO:tasks.workunit.client.1.smithi188.stdout:4/366: dwrite d0/d2/d17/d48/f57 [0,4194304] 0 2024-04-05T21:26:29.359 INFO:tasks.workunit.client.1.smithi188.stdout:1/233: read - d1/dc/d16/f47 zero size 2024-04-05T21:26:29.360 INFO:tasks.workunit.client.1.smithi188.stdout:6/329: fdatasync d6/f64 0 2024-04-05T21:26:29.363 INFO:tasks.workunit.client.1.smithi188.stdout:0/182: creat d4/d2e/f3f x:0 0 0 2024-04-05T21:26:29.367 INFO:tasks.workunit.client.1.smithi188.stdout:4/367: dread d0/d2/d2a/d71/f87 [0,4194304] 0 2024-04-05T21:26:29.367 INFO:tasks.workunit.client.1.smithi188.stdout:6/330: creat d6/d13/d18/d19/d2c/d34/f6b x:0 0 0 2024-04-05T21:26:29.370 INFO:tasks.workunit.client.1.smithi188.stdout:8/145: dwrite d5/f24 [0,4194304] 0 2024-04-05T21:26:29.371 INFO:tasks.workunit.client.1.smithi188.stdout:0/183: link d4/d8/d10/f3b d4/d8/d10/f40 0 2024-04-05T21:26:29.375 INFO:tasks.workunit.client.1.smithi188.stdout:6/331: fdatasync d6/d7/fc 0 2024-04-05T21:26:29.377 INFO:tasks.workunit.client.1.smithi188.stdout:8/146: creat d5/dc/f33 x:0 0 0 2024-04-05T21:26:29.379 INFO:tasks.workunit.client.1.smithi188.stdout:4/368: truncate d0/d2/d4f/f5c 27092 0 2024-04-05T21:26:29.385 INFO:tasks.workunit.client.1.smithi188.stdout:0/184: getdents d4/d8/db/d21 0 2024-04-05T21:26:29.387 INFO:tasks.workunit.client.1.smithi188.stdout:8/147: dwrite d5/d21/d2a/f31 [0,4194304] 0 2024-04-05T21:26:29.389 INFO:tasks.workunit.client.1.smithi188.stdout:6/332: write d6/fa [2194048,60435] 0 2024-04-05T21:26:29.396 INFO:tasks.workunit.client.1.smithi188.stdout:4/369: dwrite d0/d81/f85 [0,4194304] 0 2024-04-05T21:26:29.402 INFO:tasks.workunit.client.1.smithi188.stdout:0/185: link d4/d8/d10/l2d d4/d6/d13/d23/d24/l41 0 2024-04-05T21:26:29.404 INFO:tasks.workunit.client.1.smithi188.stdout:8/148: dwrite d5/f1e [0,4194304] 0 2024-04-05T21:26:29.404 INFO:tasks.workunit.client.1.smithi188.stdout:6/333: write d6/d13/d18/d19/d2c/f32 [2567261,124963] 0 2024-04-05T21:26:29.408 INFO:tasks.workunit.client.1.smithi188.stdout:6/334: mknod d6/d13/d38/c6c 0 2024-04-05T21:26:29.409 INFO:tasks.workunit.client.1.smithi188.stdout:6/335: fsync d6/d13/d18/d19/d61/f62 0 2024-04-05T21:26:29.410 INFO:tasks.workunit.client.1.smithi188.stdout:0/186: rename d4/d2e to d4/d8/db/d42 0 2024-04-05T21:26:29.413 INFO:tasks.workunit.client.1.smithi188.stdout:8/149: creat d5/f34 x:0 0 0 2024-04-05T21:26:29.419 INFO:tasks.workunit.client.1.smithi188.stdout:8/150: dwrite d5/dc/f32 [4194304,4194304] 0 2024-04-05T21:26:29.421 INFO:tasks.workunit.client.1.smithi188.stdout:4/370: truncate d0/d14/f6e 1082735 0 2024-04-05T21:26:29.427 INFO:tasks.workunit.client.1.smithi188.stdout:8/151: creat d5/f35 x:0 0 0 2024-04-05T21:26:29.430 INFO:tasks.workunit.client.1.smithi188.stdout:8/152: fsync d5/ff 0 2024-04-05T21:26:29.432 INFO:tasks.workunit.client.1.smithi188.stdout:6/336: dwrite d6/d28/f43 [0,4194304] 0 2024-04-05T21:26:29.433 INFO:tasks.workunit.client.1.smithi188.stdout:4/371: dwrite d0/d2/d17/d48/f57 [0,4194304] 0 2024-04-05T21:26:29.435 INFO:tasks.workunit.client.1.smithi188.stdout:8/153: link d5/f1f d5/dc/f36 0 2024-04-05T21:26:29.441 INFO:tasks.workunit.client.1.smithi188.stdout:4/372: write d0/d2/d17/f43 [4625146,40900] 0 2024-04-05T21:26:29.443 INFO:tasks.workunit.client.1.smithi188.stdout:4/373: symlink d0/d2/d17/l8c 0 2024-04-05T21:26:29.447 INFO:tasks.workunit.client.1.smithi188.stdout:4/374: rename d0/d14/d2f/d35/c3b to d0/d2/d2a/d71/c8d 0 2024-04-05T21:26:29.449 INFO:tasks.workunit.client.1.smithi188.stdout:6/337: dwrite d6/d7/fd [0,4194304] 0 2024-04-05T21:26:29.452 INFO:tasks.workunit.client.1.smithi188.stdout:6/338: symlink d6/d7/l6d 0 2024-04-05T21:26:29.454 INFO:tasks.workunit.client.1.smithi188.stdout:4/375: link d0/d14/c32 d0/d14/d15/d78/c8e 0 2024-04-05T21:26:29.455 INFO:tasks.workunit.client.1.smithi188.stdout:6/339: write d6/d13/d18/d3c/f5a [3214808,120414] 0 2024-04-05T21:26:29.456 INFO:tasks.workunit.client.1.smithi188.stdout:4/376: mkdir d0/d14/d15/d53/d82/d8f 0 2024-04-05T21:26:29.457 INFO:tasks.workunit.client.1.smithi188.stdout:4/377: write d0/d14/f80 [507922,100022] 0 2024-04-05T21:26:29.457 INFO:tasks.workunit.client.1.smithi188.stdout:6/340: creat d6/d13/d18/d19/d2c/f6e x:0 0 0 2024-04-05T21:26:29.459 INFO:tasks.workunit.client.1.smithi188.stdout:4/378: read - d0/d2/d2a/d34/f41 zero size 2024-04-05T21:26:29.459 INFO:tasks.workunit.client.1.smithi188.stdout:6/341: creat d6/d13/d38/f6f x:0 0 0 2024-04-05T21:26:29.461 INFO:tasks.workunit.client.1.smithi188.stdout:4/379: creat d0/d14/d15/f90 x:0 0 0 2024-04-05T21:26:29.462 INFO:tasks.workunit.client.1.smithi188.stdout:4/380: dread - d0/d14/d15/f89 zero size 2024-04-05T21:26:29.463 INFO:tasks.workunit.client.1.smithi188.stdout:4/381: write d0/d81/f85 [1498804,41212] 0 2024-04-05T21:26:29.466 INFO:tasks.workunit.client.1.smithi188.stdout:4/382: rename d0/d2/l4 to d0/d14/d15/d78/l91 0 2024-04-05T21:26:29.469 INFO:tasks.workunit.client.1.smithi188.stdout:4/383: symlink d0/d14/l92 0 2024-04-05T21:26:29.471 INFO:tasks.workunit.client.1.smithi188.stdout:6/342: rmdir d6/d13/d18 39 2024-04-05T21:26:29.475 INFO:tasks.workunit.client.1.smithi188.stdout:4/384: readlink d0/d2/l5d 0 2024-04-05T21:26:29.476 INFO:tasks.workunit.client.1.smithi188.stdout:4/385: dread - d0/d14/d2f/d35/f7b zero size 2024-04-05T21:26:29.477 INFO:tasks.workunit.client.1.smithi188.stdout:6/343: link d6/d13/d18/d19/d2c/d3f/c54 d6/d28/c70 0 2024-04-05T21:26:29.478 INFO:tasks.workunit.client.1.smithi188.stdout:6/344: stat d6/d13/d38/f4c 0 2024-04-05T21:26:29.479 INFO:tasks.workunit.client.1.smithi188.stdout:6/345: write d6/d28/f43 [2497605,51720] 0 2024-04-05T21:26:29.484 INFO:tasks.workunit.client.1.smithi188.stdout:4/386: dwrite d0/d2/d17/f4e [0,4194304] 0 2024-04-05T21:26:29.489 INFO:tasks.workunit.client.1.smithi188.stdout:6/346: dwrite d6/d13/d18/d19/d61/f62 [0,4194304] 0 2024-04-05T21:26:29.491 INFO:tasks.workunit.client.1.smithi188.stdout:6/347: chown d6/d13/d38/f6f 453 1 2024-04-05T21:26:29.497 INFO:tasks.workunit.client.1.smithi188.stdout:6/348: getdents d6/d13/d18/d19 0 2024-04-05T21:26:29.498 INFO:tasks.workunit.client.1.smithi188.stdout:4/387: dwrite d0/d81/f85 [0,4194304] 0 2024-04-05T21:26:29.499 INFO:tasks.workunit.client.1.smithi188.stdout:6/349: write d6/f3b [3215216,99407] 0 2024-04-05T21:26:29.504 INFO:tasks.workunit.client.1.smithi188.stdout:4/388: creat d0/d14/d15/d53/d82/d8f/f93 x:0 0 0 2024-04-05T21:26:29.509 INFO:tasks.workunit.client.1.smithi188.stdout:6/350: dwrite d6/d13/d18/d19/d2c/f51 [0,4194304] 0 2024-04-05T21:26:29.510 INFO:tasks.workunit.client.1.smithi188.stdout:6/351: chown d6/d28/c52 1 1 2024-04-05T21:26:29.518 INFO:tasks.workunit.client.1.smithi188.stdout:6/352: truncate d6/d7/f16 2118096 0 2024-04-05T21:26:29.521 INFO:tasks.workunit.client.1.smithi188.stdout:6/353: chown d6/d13/d48/d49/f69 115 1 2024-04-05T21:26:29.523 INFO:tasks.workunit.client.1.smithi188.stdout:4/389: dwrite d0/d14/d15/d1b/f40 [0,4194304] 0 2024-04-05T21:26:29.524 INFO:tasks.workunit.client.1.smithi188.stdout:6/354: mkdir d6/d71 0 2024-04-05T21:26:29.528 INFO:tasks.workunit.client.1.smithi188.stdout:6/355: dread d6/f2e [0,4194304] 0 2024-04-05T21:26:29.528 INFO:tasks.workunit.client.1.smithi188.stdout:6/356: chown d6/d28/f58 6 1 2024-04-05T21:26:29.585 INFO:tasks.workunit.client.1.smithi188.stdout:6/357: dwrite d6/d13/d38/f6f [0,4194304] 0 2024-04-05T21:26:29.594 INFO:tasks.workunit.client.1.smithi188.stdout:6/358: dwrite d6/d7/fc [4194304,4194304] 0 2024-04-05T21:26:29.595 INFO:tasks.workunit.client.1.smithi188.stdout:6/359: write d6/d28/f31 [1420543,125782] 0 2024-04-05T21:26:29.600 INFO:tasks.workunit.client.1.smithi188.stdout:6/360: rename d6/d13/f17 to d6/d13/d18/d19/f72 0 2024-04-05T21:26:29.601 INFO:tasks.workunit.client.1.smithi188.stdout:6/361: write d6/d13/d18/d19/d2c/f51 [628686,78962] 0 2024-04-05T21:26:29.608 INFO:tasks.workunit.client.1.smithi188.stdout:6/362: truncate f5 3418553 0 2024-04-05T21:26:29.609 INFO:tasks.workunit.client.1.smithi188.stdout:6/363: stat d6/d13/d18/c40 0 2024-04-05T21:26:29.611 INFO:tasks.workunit.client.1.smithi188.stdout:6/364: fdatasync d6/fb 0 2024-04-05T21:26:29.702 INFO:tasks.workunit.client.1.smithi188.stdout:2/242: sync 2024-04-05T21:26:29.714 INFO:tasks.workunit.client.1.smithi188.stdout:2/243: dread d5/f2f [4194304,4194304] 0 2024-04-05T21:26:29.716 INFO:tasks.workunit.client.1.smithi188.stdout:2/244: symlink d5/db/dd/d24/l42 0 2024-04-05T21:26:29.731 INFO:tasks.workunit.client.1.smithi188.stdout:2/245: dread d5/f22 [4194304,4194304] 0 2024-04-05T21:26:29.733 INFO:tasks.workunit.client.1.smithi188.stdout:2/246: creat d5/db/d35/f43 x:0 0 0 2024-04-05T21:26:29.738 INFO:tasks.workunit.client.1.smithi188.stdout:2/247: write d5/fc [1042405,24494] 0 2024-04-05T21:26:29.742 INFO:tasks.workunit.client.1.smithi188.stdout:2/248: truncate d5/db/f20 1685566 0 2024-04-05T21:26:29.744 INFO:tasks.workunit.client.1.smithi188.stdout:2/249: unlink d5/db/c19 0 2024-04-05T21:26:29.750 INFO:tasks.workunit.client.1.smithi188.stdout:2/250: mknod d5/db/d34/c44 0 2024-04-05T21:26:29.753 INFO:tasks.workunit.client.1.smithi188.stdout:2/251: mkdir d5/db/dd/d36/d3d/d45 0 2024-04-05T21:26:29.755 INFO:tasks.workunit.client.1.smithi188.stdout:2/252: rename d5/db/dd/d36/d3d/d45 to d5/db/dd/d36/d3d/d46 0 2024-04-05T21:26:29.756 INFO:tasks.workunit.client.1.smithi188.stdout:2/253: dread - d5/db/d35/f43 zero size 2024-04-05T21:26:29.758 INFO:tasks.workunit.client.1.smithi188.stdout:2/254: symlink d5/db/dd/d36/d3d/l47 0 2024-04-05T21:26:29.760 INFO:tasks.workunit.client.1.smithi188.stdout:2/255: mkdir d5/db/dd/d36/d3c/d48 0 2024-04-05T21:26:29.761 INFO:tasks.workunit.client.1.smithi188.stdout:2/256: mknod d5/db/d35/c49 0 2024-04-05T21:26:29.773 INFO:tasks.workunit.client.1.smithi188.stdout:2/257: dwrite d5/db/dd/f11 [0,4194304] 0 2024-04-05T21:26:29.774 INFO:tasks.workunit.client.1.smithi188.stdout:2/258: write d5/fa [312469,36818] 0 2024-04-05T21:26:29.776 INFO:tasks.workunit.client.1.smithi188.stdout:2/259: fdatasync d5/db/f3e 0 2024-04-05T21:26:29.777 INFO:tasks.workunit.client.1.smithi188.stdout:2/260: fdatasync d5/db/d35/f43 0 2024-04-05T21:26:29.780 INFO:tasks.workunit.client.1.smithi188.stdout:2/261: unlink d5/db/dd/d36/d3d/l3f 0 2024-04-05T21:26:29.783 INFO:tasks.workunit.client.1.smithi188.stdout:2/262: rename d5/f14 to d5/db/dd/d36/d3c/f4a 0 2024-04-05T21:26:29.783 INFO:tasks.workunit.client.1.smithi188.stdout:2/263: chown d5/db/dd/d24/f2e 1517685 1 2024-04-05T21:26:29.790 INFO:tasks.workunit.client.1.smithi188.stdout:2/264: dread d5/db/f20 [0,4194304] 0 2024-04-05T21:26:29.792 INFO:tasks.workunit.client.1.smithi188.stdout:2/265: symlink d5/db/d35/l4b 0 2024-04-05T21:26:29.793 INFO:tasks.workunit.client.1.smithi188.stdout:2/266: truncate d5/f2f 7631633 0 2024-04-05T21:26:29.795 INFO:tasks.workunit.client.1.smithi188.stdout:2/267: unlink d5/l21 0 2024-04-05T21:26:29.798 INFO:tasks.workunit.client.1.smithi188.stdout:2/268: dread d5/db/dd/d24/f28 [0,4194304] 0 2024-04-05T21:26:29.799 INFO:tasks.workunit.client.1.smithi188.stdout:2/269: dread - d5/db/d35/f43 zero size 2024-04-05T21:26:29.801 INFO:tasks.workunit.client.1.smithi188.stdout:2/270: symlink d5/db/dd/d24/l4c 0 2024-04-05T21:26:29.805 INFO:tasks.workunit.client.1.smithi188.stdout:2/271: creat d5/db/d35/d3b/f4d x:0 0 0 2024-04-05T21:26:29.808 INFO:tasks.workunit.client.1.smithi188.stdout:2/272: rename d5/c31 to d5/db/c4e 0 2024-04-05T21:26:29.810 INFO:tasks.workunit.client.1.smithi188.stdout:5/208: sync 2024-04-05T21:26:29.810 INFO:tasks.workunit.client.1.smithi188.stdout:3/271: sync 2024-04-05T21:26:29.810 INFO:tasks.workunit.client.1.smithi188.stdout:7/364: sync 2024-04-05T21:26:29.810 INFO:tasks.workunit.client.1.smithi188.stdout:5/209: chown c1 265495 1 2024-04-05T21:26:29.811 INFO:tasks.workunit.client.1.smithi188.stdout:3/272: rename d5/d7 to d5/d7/d62 22 2024-04-05T21:26:29.811 INFO:tasks.workunit.client.1.smithi188.stdout:2/273: symlink d5/db/dd/d36/d3d/l4f 0 2024-04-05T21:26:29.811 INFO:tasks.workunit.client.1.smithi188.stdout:3/273: fsync d5/d2b/f2e 0 2024-04-05T21:26:29.812 INFO:tasks.workunit.client.1.smithi188.stdout:7/365: creat d3/d4/d1d/f76 x:0 0 0 2024-04-05T21:26:29.815 INFO:tasks.workunit.client.1.smithi188.stdout:2/274: chown d5/db/l2c 138 1 2024-04-05T21:26:29.815 INFO:tasks.workunit.client.1.smithi188.stdout:7/366: write d3/d4/db/d2e/d37/d65/f1f [5483848,77609] 0 2024-04-05T21:26:29.817 INFO:tasks.workunit.client.1.smithi188.stdout:7/367: chown d3/d4/c6c 246850048 1 2024-04-05T21:26:29.817 INFO:tasks.workunit.client.1.smithi188.stdout:5/210: dread d6/d15/d1b/f34 [0,4194304] 0 2024-04-05T21:26:29.817 INFO:tasks.workunit.client.1.smithi188.stdout:2/275: read d5/db/dd/d24/f28 [2549047,109913] 0 2024-04-05T21:26:29.818 INFO:tasks.workunit.client.1.smithi188.stdout:3/274: getdents d5/d39 0 2024-04-05T21:26:29.827 INFO:tasks.workunit.client.1.smithi188.stdout:3/275: dwrite d5/d7/d31/f5c [0,4194304] 0 2024-04-05T21:26:29.829 INFO:tasks.workunit.client.1.smithi188.stdout:7/368: link d3/d4/d1d/l64 d3/d4/d1d/l77 0 2024-04-05T21:26:29.838 INFO:tasks.workunit.client.1.smithi188.stdout:5/211: dwrite d6/d17/d39/f4a [0,4194304] 0 2024-04-05T21:26:29.838 INFO:tasks.workunit.client.1.smithi188.stdout:2/276: dwrite d5/f2f [4194304,4194304] 0 2024-04-05T21:26:29.842 INFO:tasks.workunit.client.1.smithi188.stdout:3/276: dwrite f4 [8388608,4194304] 0 2024-04-05T21:26:29.848 INFO:tasks.workunit.client.1.smithi188.stdout:3/277: write d5/d7/f36 [1456674,2612] 0 2024-04-05T21:26:29.848 INFO:tasks.workunit.client.1.smithi188.stdout:5/212: creat d6/d17/d24/d44/f4d x:0 0 0 2024-04-05T21:26:29.850 INFO:tasks.workunit.client.1.smithi188.stdout:7/369: dwrite d3/d4/db/d2e/f75 [0,4194304] 0 2024-04-05T21:26:29.851 INFO:tasks.workunit.client.1.smithi188.stdout:5/213: unlink d6/d15/d1b/f45 0 2024-04-05T21:26:29.858 INFO:tasks.workunit.client.1.smithi188.stdout:5/214: mkdir d6/d17/d23/d35/d40/d42/d4e 0 2024-04-05T21:26:29.860 INFO:tasks.workunit.client.1.smithi188.stdout:3/278: dwrite d5/d2b/f2e [0,4194304] 0 2024-04-05T21:26:29.860 INFO:tasks.workunit.client.1.smithi188.stdout:5/215: chown d6/d17/f20 255199203 1 2024-04-05T21:26:29.864 INFO:tasks.workunit.client.1.smithi188.stdout:5/216: getdents d6/d17/d23/d35/d40/d42/d4e 0 2024-04-05T21:26:29.872 INFO:tasks.workunit.client.1.smithi188.stdout:3/279: dwrite d5/dc/f5a [0,4194304] 0 2024-04-05T21:26:29.872 INFO:tasks.workunit.client.1.smithi188.stdout:0/187: sync 2024-04-05T21:26:29.872 INFO:tasks.workunit.client.1.smithi188.stdout:9/292: sync 2024-04-05T21:26:29.872 INFO:tasks.workunit.client.1.smithi188.stdout:1/234: sync 2024-04-05T21:26:29.872 INFO:tasks.workunit.client.1.smithi188.stdout:4/390: sync 2024-04-05T21:26:29.872 INFO:tasks.workunit.client.1.smithi188.stdout:6/365: sync 2024-04-05T21:26:29.872 INFO:tasks.workunit.client.1.smithi188.stdout:8/154: sync 2024-04-05T21:26:29.872 INFO:tasks.workunit.client.1.smithi188.stdout:5/217: mkdir d6/d15/d1b/d4f 0 2024-04-05T21:26:29.877 INFO:tasks.workunit.client.1.smithi188.stdout:3/280: dread - d5/dc/f3d zero size 2024-04-05T21:26:29.877 INFO:tasks.workunit.client.1.smithi188.stdout:5/218: read d6/d15/d1b/f34 [3963859,23732] 0 2024-04-05T21:26:29.880 INFO:tasks.workunit.client.1.smithi188.stdout:0/188: symlink d4/d8/d39/l43 0 2024-04-05T21:26:29.892 INFO:tasks.workunit.client.1.smithi188.stdout:4/391: symlink d0/d14/l94 0 2024-04-05T21:26:29.892 INFO:tasks.workunit.client.1.smithi188.stdout:8/155: dwrite d5/f1c [0,4194304] 0 2024-04-05T21:26:29.892 INFO:tasks.workunit.client.1.smithi188.stdout:1/235: dwrite d1/dc/d16/d2d/f34 [0,4194304] 0 2024-04-05T21:26:29.893 INFO:tasks.workunit.client.1.smithi188.stdout:5/219: creat d6/d17/d23/d35/f50 x:0 0 0 2024-04-05T21:26:29.893 INFO:tasks.workunit.client.1.smithi188.stdout:4/392: mkdir d0/d14/d15/d53/d82/d8f/d95 0 2024-04-05T21:26:29.893 INFO:tasks.workunit.client.1.smithi188.stdout:0/189: rmdir d4/d8/db/d21 39 2024-04-05T21:26:29.894 INFO:tasks.workunit.client.1.smithi188.stdout:3/281: write d5/d7/d31/f45 [4217052,2390] 0 2024-04-05T21:26:29.896 INFO:tasks.workunit.client.1.smithi188.stdout:9/293: rename dd/d35/l3a to dd/l5d 0 2024-04-05T21:26:29.896 INFO:tasks.workunit.client.1.smithi188.stdout:5/220: write d6/f22 [3619751,123577] 0 2024-04-05T21:26:29.911 INFO:tasks.workunit.client.1.smithi188.stdout:0/190: fsync d4/f27 0 2024-04-05T21:26:29.911 INFO:tasks.workunit.client.1.smithi188.stdout:5/221: mkdir d6/d17/d24/d44/d51 0 2024-04-05T21:26:29.912 INFO:tasks.workunit.client.1.smithi188.stdout:4/393: symlink d0/d14/l96 0 2024-04-05T21:26:29.913 INFO:tasks.workunit.client.1.smithi188.stdout:3/282: creat d5/d48/f63 x:0 0 0 2024-04-05T21:26:29.914 INFO:tasks.workunit.client.1.smithi188.stdout:8/156: dread d5/d21/d2a/f31 [0,4194304] 0 2024-04-05T21:26:29.919 INFO:tasks.workunit.client.1.smithi188.stdout:5/222: dread - d6/d17/d23/d35/f50 zero size 2024-04-05T21:26:29.920 INFO:tasks.workunit.client.1.smithi188.stdout:4/394: creat d0/d14/d15/d78/f97 x:0 0 0 2024-04-05T21:26:29.920 INFO:tasks.workunit.client.1.smithi188.stdout:9/294: dwrite dd/d11/d22/d28/f43 [0,4194304] 0 2024-04-05T21:26:29.924 INFO:tasks.workunit.client.1.smithi188.stdout:7/370: dwrite d3/f1e [0,4194304] 0 2024-04-05T21:26:29.925 INFO:tasks.workunit.client.1.smithi188.stdout:6/366: dwrite d6/d7/f16 [0,4194304] 0 2024-04-05T21:26:29.925 INFO:tasks.workunit.client.1.smithi188.stdout:5/223: write f2 [2677712,100968] 0 2024-04-05T21:26:29.926 INFO:tasks.workunit.client.1.smithi188.stdout:0/191: rename d4/d8/d10/f3b to d4/d8/db/d21/f44 0 2024-04-05T21:26:29.931 INFO:tasks.workunit.client.1.smithi188.stdout:1/236: write d1/fb [3603470,106543] 0 2024-04-05T21:26:29.932 INFO:tasks.workunit.client.1.smithi188.stdout:9/295: truncate dd/d11/d22/f4d 284242 0 2024-04-05T21:26:29.958 INFO:tasks.workunit.client.1.smithi188.stdout:4/395: write d0/d14/d2f/d35/f83 [232345,66826] 0 2024-04-05T21:26:29.958 INFO:tasks.workunit.client.1.smithi188.stdout:6/367: chown d6/d7/f15 159 1 2024-04-05T21:26:29.959 INFO:tasks.workunit.client.1.smithi188.stdout:5/224: creat d6/d17/d24/f52 x:0 0 0 2024-04-05T21:26:29.961 INFO:tasks.workunit.client.1.smithi188.stdout:9/296: symlink dd/d11/d1f/d26/l5e 0 2024-04-05T21:26:29.962 INFO:tasks.workunit.client.1.smithi188.stdout:3/283: dwrite d5/dc/f5a [0,4194304] 0 2024-04-05T21:26:29.963 INFO:tasks.workunit.client.1.smithi188.stdout:0/192: creat d4/d8/db/f45 x:0 0 0 2024-04-05T21:26:29.966 INFO:tasks.workunit.client.1.smithi188.stdout:6/368: write d6/d13/d18/d19/d2c/f6e [742221,58028] 0 2024-04-05T21:26:29.974 INFO:tasks.workunit.client.1.smithi188.stdout:1/237: dread d1/dc/d16/f33 [0,4194304] 0 2024-04-05T21:26:29.976 INFO:tasks.workunit.client.1.smithi188.stdout:5/225: dread d6/f9 [0,4194304] 0 2024-04-05T21:26:29.977 INFO:tasks.workunit.client.1.smithi188.stdout:4/396: dwrite d0/d14/d15/f5b [0,4194304] 0 2024-04-05T21:26:29.977 INFO:tasks.workunit.client.1.smithi188.stdout:5/226: stat d6/d15/l3a 0 2024-04-05T21:26:29.979 INFO:tasks.workunit.client.1.smithi188.stdout:9/297: creat dd/f5f x:0 0 0 2024-04-05T21:26:29.979 INFO:tasks.workunit.client.1.smithi188.stdout:5/227: chown d6/d17/d24/d44/d51 553089 1 2024-04-05T21:26:29.979 INFO:tasks.workunit.client.1.smithi188.stdout:6/369: rename d6/d13/d38/c6c to d6/d13/d48/c73 0 2024-04-05T21:26:29.982 INFO:tasks.workunit.client.1.smithi188.stdout:9/298: creat dd/d49/f60 x:0 0 0 2024-04-05T21:26:29.983 INFO:tasks.workunit.client.1.smithi188.stdout:9/299: read - dd/d49/f60 zero size 2024-04-05T21:26:29.984 INFO:tasks.workunit.client.1.smithi188.stdout:6/370: read d6/d13/d18/d3c/f5a [200332,94271] 0 2024-04-05T21:26:29.987 INFO:tasks.workunit.client.1.smithi188.stdout:4/397: creat d0/d14/d2f/f98 x:0 0 0 2024-04-05T21:26:29.988 INFO:tasks.workunit.client.1.smithi188.stdout:5/228: creat d6/f53 x:0 0 0 2024-04-05T21:26:29.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:29 smithi121 ceph-mon[39723]: pgmap v13: 65 pgs: 65 active+clean; 429 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 700 KiB/s rd, 22 MiB/s wr, 127 op/s 2024-04-05T21:26:29.989 INFO:tasks.workunit.client.1.smithi188.stdout:9/300: creat dd/d3d/f61 x:0 0 0 2024-04-05T21:26:29.990 INFO:tasks.workunit.client.1.smithi188.stdout:0/193: write d4/d8/f26 [1122577,31733] 0 2024-04-05T21:26:29.991 INFO:tasks.workunit.client.1.smithi188.stdout:5/229: write d6/d17/d23/d35/f3b [271825,47318] 0 2024-04-05T21:26:29.994 INFO:tasks.workunit.client.1.smithi188.stdout:4/398: mknod d0/d14/d2f/d35/d6a/c99 0 2024-04-05T21:26:30.005 INFO:tasks.workunit.client.1.smithi188.stdout:6/371: link d6/d7/f1e d6/f74 0 2024-04-05T21:26:30.005 INFO:tasks.workunit.client.1.smithi188.stdout:1/238: write d1/dc/f11 [5055019,98326] 0 2024-04-05T21:26:30.005 INFO:tasks.workunit.client.1.smithi188.stdout:0/194: mkdir d4/d6/d46 0 2024-04-05T21:26:30.005 INFO:tasks.workunit.client.1.smithi188.stdout:5/230: link d6/d15/d1b/c32 d6/d17/d23/d35/d40/d42/d4e/c54 0 2024-04-05T21:26:30.005 INFO:tasks.workunit.client.1.smithi188.stdout:3/284: dread d5/df/f33 [0,4194304] 0 2024-04-05T21:26:30.005 INFO:tasks.workunit.client.1.smithi188.stdout:4/399: getdents d0/d14/d15/d42 0 2024-04-05T21:26:30.006 INFO:tasks.workunit.client.1.smithi188.stdout:0/195: creat d4/d6/d46/f47 x:0 0 0 2024-04-05T21:26:30.006 INFO:tasks.workunit.client.1.smithi188.stdout:1/239: dread d1/f24 [0,4194304] 0 2024-04-05T21:26:30.008 INFO:tasks.workunit.client.1.smithi188.stdout:3/285: chown d5/d2b/c42 307248799 1 2024-04-05T21:26:30.011 INFO:tasks.workunit.client.1.smithi188.stdout:1/240: mkdir d1/dc/d16/d21/d27/d4a 0 2024-04-05T21:26:30.011 INFO:tasks.workunit.client.1.smithi188.stdout:3/286: fdatasync d5/d7/d31/f45 0 2024-04-05T21:26:30.011 INFO:tasks.workunit.client.1.smithi188.stdout:9/301: dread fc [0,4194304] 0 2024-04-05T21:26:30.011 INFO:tasks.workunit.client.1.smithi188.stdout:5/231: creat d6/d17/d23/f55 x:0 0 0 2024-04-05T21:26:30.017 INFO:tasks.workunit.client.1.smithi188.stdout:1/241: creat d1/d4/d3f/f4b x:0 0 0 2024-04-05T21:26:30.017 INFO:tasks.workunit.client.1.smithi188.stdout:5/232: fsync d6/fe 0 2024-04-05T21:26:30.018 INFO:tasks.workunit.client.1.smithi188.stdout:4/400: rename d0/d2/d2a/d34/l47 to d0/d2/d52/l9a 0 2024-04-05T21:26:30.018 INFO:tasks.workunit.client.1.smithi188.stdout:6/372: dread d6/f74 [4194304,4194304] 0 2024-04-05T21:26:30.021 INFO:tasks.workunit.client.1.smithi188.stdout:1/242: dread d1/dc/d16/f3c [0,4194304] 0 2024-04-05T21:26:30.023 INFO:tasks.workunit.client.1.smithi188.stdout:5/233: creat d6/d17/d23/d35/d40/d42/f56 x:0 0 0 2024-04-05T21:26:30.025 INFO:tasks.workunit.client.1.smithi188.stdout:3/287: creat d5/d2b/d43/d5f/f64 x:0 0 0 2024-04-05T21:26:30.026 INFO:tasks.workunit.client.1.smithi188.stdout:0/196: dread d4/d8/db/d42/f30 [0,4194304] 0 2024-04-05T21:26:30.026 INFO:tasks.workunit.client.1.smithi188.stdout:6/373: mkdir d6/d13/d48/d49/d75 0 2024-04-05T21:26:30.027 INFO:tasks.workunit.client.1.smithi188.stdout:3/288: write d5/df/f33 [5041302,82202] 0 2024-04-05T21:26:30.027 INFO:tasks.workunit.client.1.smithi188.stdout:3/289: fdatasync d5/f1e 0 2024-04-05T21:26:30.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:29 smithi188 ceph-mon[49715]: pgmap v13: 65 pgs: 65 active+clean; 429 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 700 KiB/s rd, 22 MiB/s wr, 127 op/s 2024-04-05T21:26:30.032 INFO:tasks.workunit.client.1.smithi188.stdout:5/234: rename d6/d15/d1b/f27 to d6/d17/d24/d44/f57 0 2024-04-05T21:26:30.033 INFO:tasks.workunit.client.1.smithi188.stdout:6/374: mkdir d6/d28/d76 0 2024-04-05T21:26:30.037 INFO:tasks.workunit.client.1.smithi188.stdout:6/375: dread d6/d7/f16 [0,4194304] 0 2024-04-05T21:26:30.038 INFO:tasks.workunit.client.1.smithi188.stdout:6/376: chown l1 21 1 2024-04-05T21:26:30.042 INFO:tasks.workunit.client.1.smithi188.stdout:4/401: link d0/l3 d0/d2/d2a/d34/d6b/l9b 0 2024-04-05T21:26:30.044 INFO:tasks.workunit.client.1.smithi188.stdout:3/290: dread d5/d35/f40 [0,4194304] 0 2024-04-05T21:26:30.044 INFO:tasks.workunit.client.1.smithi188.stdout:6/377: write d6/d13/d18/d19/d2c/d3f/f56 [4522189,91578] 0 2024-04-05T21:26:30.047 INFO:tasks.workunit.client.1.smithi188.stdout:1/243: getdents d1/dc/d16/d2d 0 2024-04-05T21:26:30.048 INFO:tasks.workunit.client.1.smithi188.stdout:4/402: chown d0/d14/d2f/l68 168 1 2024-04-05T21:26:30.048 INFO:tasks.workunit.client.1.smithi188.stdout:6/378: write d6/d7/fd [991816,16668] 0 2024-04-05T21:26:30.049 INFO:tasks.workunit.client.1.smithi188.stdout:5/235: link d6/l31 d6/d17/d23/d35/d40/d42/d4e/l58 0 2024-04-05T21:26:30.051 INFO:tasks.workunit.client.1.smithi188.stdout:1/244: write d1/dc/f1f [798494,59277] 0 2024-04-05T21:26:30.051 INFO:tasks.workunit.client.1.smithi188.stdout:9/302: dwrite dd/d11/d22/f56 [0,4194304] 0 2024-04-05T21:26:30.053 INFO:tasks.workunit.client.1.smithi188.stdout:3/291: rename d5/d2b/d43 to d5/d7/d65 0 2024-04-05T21:26:30.054 INFO:tasks.workunit.client.1.smithi188.stdout:9/303: chown dd/d11/d22/d28 8092462 1 2024-04-05T21:26:30.056 INFO:tasks.workunit.client.1.smithi188.stdout:4/403: chown d0/d2/f26 249137953 1 2024-04-05T21:26:30.056 INFO:tasks.workunit.client.1.smithi188.stdout:1/245: chown d1/dc/f1f 318226867 1 2024-04-05T21:26:30.057 INFO:tasks.workunit.client.1.smithi188.stdout:6/379: rename d6/d28/d76 to d6/d13/d18/d19/d2c/d3f/d77 0 2024-04-05T21:26:30.058 INFO:tasks.workunit.client.1.smithi188.stdout:4/404: fsync d0/d14/f39 0 2024-04-05T21:26:30.060 INFO:tasks.workunit.client.1.smithi188.stdout:4/405: chown d0/d2 20533157 1 2024-04-05T21:26:30.060 INFO:tasks.workunit.client.1.smithi188.stdout:1/246: fsync d1/dc/d16/d21/d27/f29 0 2024-04-05T21:26:30.061 INFO:tasks.workunit.client.1.smithi188.stdout:6/380: mknod d6/d13/d38/c78 0 2024-04-05T21:26:30.065 INFO:tasks.workunit.client.1.smithi188.stdout:9/304: dwrite dd/d11/d22/d28/d40/f4e [0,4194304] 0 2024-04-05T21:26:30.067 INFO:tasks.workunit.client.1.smithi188.stdout:3/292: rmdir d5/d51 0 2024-04-05T21:26:30.067 INFO:tasks.workunit.client.1.smithi188.stdout:9/305: read dd/d11/d22/f4d [161525,31707] 0 2024-04-05T21:26:30.069 INFO:tasks.workunit.client.1.smithi188.stdout:9/306: chown ca 278 1 2024-04-05T21:26:30.070 INFO:tasks.workunit.client.1.smithi188.stdout:9/307: chown l8 244 1 2024-04-05T21:26:30.070 INFO:tasks.workunit.client.1.smithi188.stdout:1/247: rename d1/dc/l1e to d1/d7/l4c 0 2024-04-05T21:26:30.071 INFO:tasks.workunit.client.1.smithi188.stdout:5/236: link d6/d17/f1f d6/d15/d1b/d4f/f59 0 2024-04-05T21:26:30.073 INFO:tasks.workunit.client.1.smithi188.stdout:5/237: write d6/d17/f2f [745261,99353] 0 2024-04-05T21:26:30.074 INFO:tasks.workunit.client.1.smithi188.stdout:9/308: symlink dd/d11/d1f/d26/l62 0 2024-04-05T21:26:30.075 INFO:tasks.workunit.client.1.smithi188.stdout:4/406: dread d0/d14/f6e [0,4194304] 0 2024-04-05T21:26:30.076 INFO:tasks.workunit.client.1.smithi188.stdout:3/293: mkdir d5/d7/d65/d61/d66 0 2024-04-05T21:26:30.079 INFO:tasks.workunit.client.1.smithi188.stdout:6/381: getdents d6/d13/d18/d19 0 2024-04-05T21:26:30.079 INFO:tasks.workunit.client.1.smithi188.stdout:4/407: dread - d0/d14/d15/d53/f66 zero size 2024-04-05T21:26:30.080 INFO:tasks.workunit.client.1.smithi188.stdout:5/238: symlink d6/d15/d1b/d4f/l5a 0 2024-04-05T21:26:30.081 INFO:tasks.workunit.client.1.smithi188.stdout:1/248: dwrite d1/dc/d16/f47 [0,4194304] 0 2024-04-05T21:26:30.082 INFO:tasks.workunit.client.1.smithi188.stdout:4/408: fdatasync d0/d14/d2f/d35/f7b 0 2024-04-05T21:26:30.082 INFO:tasks.workunit.client.1.smithi188.stdout:6/382: chown d6/d13/d18/d19/d2c/d3f/c68 28 1 2024-04-05T21:26:30.082 INFO:tasks.workunit.client.1.smithi188.stdout:1/249: read - d1/d4/d3f/f44 zero size 2024-04-05T21:26:30.085 INFO:tasks.workunit.client.1.smithi188.stdout:1/250: fsync d1/dc/d16/d21/d27/f29 0 2024-04-05T21:26:30.085 INFO:tasks.workunit.client.1.smithi188.stdout:1/251: readlink d1/l32 0 2024-04-05T21:26:30.087 INFO:tasks.workunit.client.1.smithi188.stdout:9/309: link dd/d11/d1f/d25/f44 dd/d11/d1f/d26/d55/f63 0 2024-04-05T21:26:30.089 INFO:tasks.workunit.client.1.smithi188.stdout:1/252: chown d1/d4/d13/f18 2380014 1 2024-04-05T21:26:30.090 INFO:tasks.workunit.client.1.smithi188.stdout:5/239: link d6/d17/d39/f4a d6/f5b 0 2024-04-05T21:26:30.091 INFO:tasks.workunit.client.1.smithi188.stdout:5/240: stat d6/d17/d23/d35 0 2024-04-05T21:26:30.093 INFO:tasks.workunit.client.1.smithi188.stdout:6/383: rename d6/d13/l20 to d6/d28/l79 0 2024-04-05T21:26:30.094 INFO:tasks.workunit.client.1.smithi188.stdout:4/409: creat d0/f9c x:0 0 0 2024-04-05T21:26:30.096 INFO:tasks.workunit.client.1.smithi188.stdout:6/384: stat d6/l12 0 2024-04-05T21:26:30.098 INFO:tasks.workunit.client.1.smithi188.stdout:9/310: symlink dd/d11/d1f/d26/d55/l64 0 2024-04-05T21:26:30.098 INFO:tasks.workunit.client.1.smithi188.stdout:4/410: mknod d0/d14/d15/d53/d82/c9d 0 2024-04-05T21:26:30.101 INFO:tasks.workunit.client.1.smithi188.stdout:6/385: creat d6/d13/d18/d19/d2c/d3f/f7a x:0 0 0 2024-04-05T21:26:30.102 INFO:tasks.workunit.client.1.smithi188.stdout:4/411: creat d0/d14/d15/d53/d82/f9e x:0 0 0 2024-04-05T21:26:30.103 INFO:tasks.workunit.client.1.smithi188.stdout:1/253: link d1/d4/d13/l8 d1/dc/d16/d46/l4d 0 2024-04-05T21:26:30.105 INFO:tasks.workunit.client.1.smithi188.stdout:1/254: mkdir d1/d7/d4e 0 2024-04-05T21:26:30.110 INFO:tasks.workunit.client.1.smithi188.stdout:4/412: dwrite d0/d14/f86 [0,4194304] 0 2024-04-05T21:26:30.113 INFO:tasks.workunit.client.1.smithi188.stdout:4/413: link d0/d14/f80 d0/d14/d15/d53/f9f 0 2024-04-05T21:26:30.136 INFO:tasks.workunit.client.1.smithi188.stdout:1/255: dread d1/dc/f1b [0,4194304] 0 2024-04-05T21:26:30.136 INFO:tasks.workunit.client.1.smithi188.stdout:1/256: dread - d1/d4/d3f/f4b zero size 2024-04-05T21:26:30.149 INFO:tasks.workunit.client.1.smithi188.stdout:4/414: creat d0/d14/d15/d42/fa0 x:0 0 0 2024-04-05T21:26:30.150 INFO:tasks.workunit.client.1.smithi188.stdout:1/257: unlink d1/d4/f36 0 2024-04-05T21:26:30.152 INFO:tasks.workunit.client.1.smithi188.stdout:4/415: unlink d0/c2d 0 2024-04-05T21:26:30.153 INFO:tasks.workunit.client.1.smithi188.stdout:6/386: truncate d6/d13/d38/f6f 2959346 0 2024-04-05T21:26:30.154 INFO:tasks.workunit.client.1.smithi188.stdout:6/387: write d6/d7/fd [3495642,56767] 0 2024-04-05T21:26:30.157 INFO:tasks.workunit.client.1.smithi188.stdout:6/388: write d6/d13/d18/d19/d61/f62 [473784,77979] 0 2024-04-05T21:26:30.158 INFO:tasks.workunit.client.1.smithi188.stdout:6/389: readlink l1 0 2024-04-05T21:26:30.159 INFO:tasks.workunit.client.1.smithi188.stdout:4/416: creat d0/d2/d2a/d34/fa1 x:0 0 0 2024-04-05T21:26:30.160 INFO:tasks.workunit.client.1.smithi188.stdout:4/417: chown d0/d2/d17/f4e 1848687173 1 2024-04-05T21:26:30.161 INFO:tasks.workunit.client.1.smithi188.stdout:4/418: fdatasync d0/d2/d17/f43 0 2024-04-05T21:26:30.161 INFO:tasks.workunit.client.1.smithi188.stdout:6/390: symlink d6/d71/l7b 0 2024-04-05T21:26:30.167 INFO:tasks.workunit.client.1.smithi188.stdout:4/419: mknod d0/d14/ca2 0 2024-04-05T21:26:30.167 INFO:tasks.workunit.client.1.smithi188.stdout:1/258: write d1/d4/d3f/f40 [530646,113769] 0 2024-04-05T21:26:30.167 INFO:tasks.workunit.client.1.smithi188.stdout:4/420: creat d0/d14/d15/d42/fa3 x:0 0 0 2024-04-05T21:26:30.172 INFO:tasks.workunit.client.1.smithi188.stdout:1/259: creat d1/d4/d13/f4f x:0 0 0 2024-04-05T21:26:30.172 INFO:tasks.workunit.client.1.smithi188.stdout:4/421: dread d0/d14/d15/f5b [0,4194304] 0 2024-04-05T21:26:30.179 INFO:tasks.workunit.client.1.smithi188.stdout:1/260: dwrite d1/d4/d3f/f44 [0,4194304] 0 2024-04-05T21:26:30.182 INFO:tasks.workunit.client.1.smithi188.stdout:4/422: dwrite d0/d14/d15/d1b/f60 [0,4194304] 0 2024-04-05T21:26:30.189 INFO:tasks.workunit.client.1.smithi188.stdout:1/261: dwrite d1/dc/d16/f47 [0,4194304] 0 2024-04-05T21:26:30.195 INFO:tasks.workunit.client.1.smithi188.stdout:1/262: rename d1/f24 to d1/d4/d13/f50 0 2024-04-05T21:26:30.195 INFO:tasks.workunit.client.1.smithi188.stdout:4/423: dread d0/f64 [0,4194304] 0 2024-04-05T21:26:30.201 INFO:tasks.workunit.client.1.smithi188.stdout:1/263: creat d1/dc/d16/d46/f51 x:0 0 0 2024-04-05T21:26:30.202 INFO:tasks.workunit.client.1.smithi188.stdout:4/424: creat d0/d14/d15/d53/d82/d8f/d95/fa4 x:0 0 0 2024-04-05T21:26:30.202 INFO:tasks.workunit.client.1.smithi188.stdout:1/264: fdatasync d1/fb 0 2024-04-05T21:26:30.205 INFO:tasks.workunit.client.1.smithi188.stdout:1/265: rmdir d1/d7 39 2024-04-05T21:26:30.206 INFO:tasks.workunit.client.1.smithi188.stdout:4/425: chown d0/d2/d4f/f5c 893 1 2024-04-05T21:26:30.208 INFO:tasks.workunit.client.1.smithi188.stdout:1/266: symlink d1/d7/l52 0 2024-04-05T21:26:30.210 INFO:tasks.workunit.client.1.smithi188.stdout:4/426: dread d0/d2/d17/f43 [4194304,4194304] 0 2024-04-05T21:26:30.211 INFO:tasks.workunit.client.1.smithi188.stdout:1/267: rename d1/d4/c3a to d1/d7/d4e/c53 0 2024-04-05T21:26:30.213 INFO:tasks.workunit.client.1.smithi188.stdout:4/427: creat d0/d14/d15/d42/fa5 x:0 0 0 2024-04-05T21:26:30.214 INFO:tasks.workunit.client.1.smithi188.stdout:4/428: chown d0/d14/d2f/l68 12456 1 2024-04-05T21:26:30.217 INFO:tasks.workunit.client.1.smithi188.stdout:1/268: dwrite d1/d4/f1d [0,4194304] 0 2024-04-05T21:26:30.226 INFO:tasks.workunit.client.1.smithi188.stdout:1/269: dwrite d1/dc/f1f [0,4194304] 0 2024-04-05T21:26:30.227 INFO:tasks.workunit.client.1.smithi188.stdout:1/270: write d1/d4/d3f/f4b [518568,127192] 0 2024-04-05T21:26:30.240 INFO:tasks.workunit.client.1.smithi188.stdout:4/429: dread d0/d14/f38 [0,4194304] 0 2024-04-05T21:26:30.242 INFO:tasks.workunit.client.1.smithi188.stdout:4/430: symlink d0/d14/d15/d53/d82/d8f/la6 0 2024-04-05T21:26:30.243 INFO:tasks.workunit.client.1.smithi188.stdout:4/431: dread - d0/d14/d15/f88 zero size 2024-04-05T21:26:30.244 INFO:tasks.workunit.client.1.smithi188.stdout:4/432: creat d0/d2/d17/fa7 x:0 0 0 2024-04-05T21:26:30.245 INFO:tasks.workunit.client.1.smithi188.stdout:4/433: readlink d0/d14/l73 0 2024-04-05T21:26:30.245 INFO:tasks.workunit.client.1.smithi188.stdout:4/434: dread - d0/d14/d15/d53/d82/f9e zero size 2024-04-05T21:26:30.248 INFO:tasks.workunit.client.1.smithi188.stdout:4/435: creat d0/fa8 x:0 0 0 2024-04-05T21:26:30.248 INFO:tasks.workunit.client.1.smithi188.stdout:4/436: stat d0/d14/d15/d53/f9f 0 2024-04-05T21:26:30.251 INFO:tasks.workunit.client.1.smithi188.stdout:4/437: rename d0/d14/f39 to d0/d2/d2a/d34/d6b/fa9 0 2024-04-05T21:26:30.254 INFO:tasks.workunit.client.1.smithi188.stdout:1/271: dread d1/d4/f12 [0,4194304] 0 2024-04-05T21:26:30.256 INFO:tasks.workunit.client.1.smithi188.stdout:1/272: mknod d1/d4/d3f/c54 0 2024-04-05T21:26:30.257 INFO:tasks.workunit.client.1.smithi188.stdout:4/438: dwrite d0/d14/d15/d53/f74 [0,4194304] 0 2024-04-05T21:26:30.261 INFO:tasks.workunit.client.1.smithi188.stdout:4/439: rmdir d0/d2/d17/d48 39 2024-04-05T21:26:30.264 INFO:tasks.workunit.client.1.smithi188.stdout:4/440: mkdir d0/d14/d2f/d35/daa 0 2024-04-05T21:26:30.264 INFO:tasks.workunit.client.1.smithi188.stdout:1/273: dread d1/dc/f1f [0,4194304] 0 2024-04-05T21:26:30.267 INFO:tasks.workunit.client.1.smithi188.stdout:4/441: symlink d0/lab 0 2024-04-05T21:26:30.269 INFO:tasks.workunit.client.1.smithi188.stdout:4/442: creat d0/d14/d15/d53/fac x:0 0 0 2024-04-05T21:26:30.271 INFO:tasks.workunit.client.1.smithi188.stdout:1/274: rename d1/dc/c2f to d1/dc/d16/c55 0 2024-04-05T21:26:30.272 INFO:tasks.workunit.client.1.smithi188.stdout:4/443: creat d0/d14/d15/d53/d82/d8f/fad x:0 0 0 2024-04-05T21:26:30.274 INFO:tasks.workunit.client.1.smithi188.stdout:4/444: rmdir d0/d2/d17/d3a 39 2024-04-05T21:26:30.275 INFO:tasks.workunit.client.1.smithi188.stdout:1/275: dread d1/d4/d3f/f44 [0,4194304] 0 2024-04-05T21:26:30.276 INFO:tasks.workunit.client.1.smithi188.stdout:1/276: fsync d1/dc/f28 0 2024-04-05T21:26:30.277 INFO:tasks.workunit.client.1.smithi188.stdout:4/445: creat d0/d14/d15/d78/fae x:0 0 0 2024-04-05T21:26:30.279 INFO:tasks.workunit.client.1.smithi188.stdout:4/446: dread d0/d2/d4f/f5c [0,4194304] 0 2024-04-05T21:26:30.280 INFO:tasks.workunit.client.1.smithi188.stdout:1/277: creat d1/dc/f56 x:0 0 0 2024-04-05T21:26:30.284 INFO:tasks.workunit.client.1.smithi188.stdout:1/278: rename d1/d4/l43 to d1/dc/d16/d21/d27/l57 0 2024-04-05T21:26:30.287 INFO:tasks.workunit.client.1.smithi188.stdout:4/447: dwrite d0/d14/d15/d53/d82/d8f/f93 [0,4194304] 0 2024-04-05T21:26:30.292 INFO:tasks.workunit.client.1.smithi188.stdout:1/279: getdents d1/dc/d16 0 2024-04-05T21:26:30.297 INFO:tasks.workunit.client.1.smithi188.stdout:4/448: dwrite d0/d14/d2f/d35/f7c [0,4194304] 0 2024-04-05T21:26:30.299 INFO:tasks.workunit.client.1.smithi188.stdout:1/280: truncate d1/dc/d16/d21/d27/f29 1424033 0 2024-04-05T21:26:30.301 INFO:tasks.workunit.client.1.smithi188.stdout:1/281: fdatasync d1/dc/f1b 0 2024-04-05T21:26:30.308 INFO:tasks.workunit.client.1.smithi188.stdout:1/282: write d1/fb [1911274,96123] 0 2024-04-05T21:26:30.309 INFO:tasks.workunit.client.1.smithi188.stdout:4/449: dwrite d0/d14/d2f/d35/f7c [4194304,4194304] 0 2024-04-05T21:26:30.313 INFO:tasks.workunit.client.1.smithi188.stdout:1/283: unlink d1/d4/f3d 0 2024-04-05T21:26:30.319 INFO:tasks.workunit.client.1.smithi188.stdout:4/450: dwrite d0/d14/d15/d42/fa3 [0,4194304] 0 2024-04-05T21:26:30.320 INFO:tasks.workunit.client.1.smithi188.stdout:1/284: link d1/d7/d4e/c53 d1/dc/d16/d21/d27/c58 0 2024-04-05T21:26:30.322 INFO:tasks.workunit.client.1.smithi188.stdout:4/451: unlink d0/d14/d15/f88 0 2024-04-05T21:26:30.326 INFO:tasks.workunit.client.1.smithi188.stdout:4/452: getdents d0/d14/d15/d1b 0 2024-04-05T21:26:30.329 INFO:tasks.workunit.client.1.smithi188.stdout:1/285: dwrite d1/d4/f12 [4194304,4194304] 0 2024-04-05T21:26:30.333 INFO:tasks.workunit.client.1.smithi188.stdout:1/286: creat d1/dc/d16/d21/d27/d49/f59 x:0 0 0 2024-04-05T21:26:30.334 INFO:tasks.workunit.client.1.smithi188.stdout:4/453: dwrite d0/d2/d17/f4e [0,4194304] 0 2024-04-05T21:26:30.335 INFO:tasks.workunit.client.1.smithi188.stdout:1/287: mkdir d1/d7/d4e/d5a 0 2024-04-05T21:26:30.338 INFO:tasks.workunit.client.1.smithi188.stdout:1/288: chown d1/dc/d16/d21/d27/l57 536 1 2024-04-05T21:26:30.344 INFO:tasks.workunit.client.1.smithi188.stdout:4/454: dwrite d0/d14/d15/d42/fa5 [0,4194304] 0 2024-04-05T21:26:30.347 INFO:tasks.workunit.client.1.smithi188.stdout:4/455: write d0/d14/d2f/d35/f83 [825507,29802] 0 2024-04-05T21:26:30.349 INFO:tasks.workunit.client.1.smithi188.stdout:1/289: dwrite d1/fb [0,4194304] 0 2024-04-05T21:26:30.350 INFO:tasks.workunit.client.1.smithi188.stdout:4/456: truncate d0/d14/d15/d53/f9f 1553723 0 2024-04-05T21:26:30.350 INFO:tasks.workunit.client.1.smithi188.stdout:1/290: chown d1/l32 10774 1 2024-04-05T21:26:30.351 INFO:tasks.workunit.client.1.smithi188.stdout:1/291: chown d1/d7/c9 371 1 2024-04-05T21:26:30.357 INFO:tasks.workunit.client.1.smithi188.stdout:4/457: rename d0/d2/d2a/d34/c59 to d0/d2/caf 0 2024-04-05T21:26:30.360 INFO:tasks.workunit.client.1.smithi188.stdout:1/292: link d1/dc/d16/d2d/c31 d1/d7/d4e/c5b 0 2024-04-05T21:26:30.360 INFO:tasks.workunit.client.1.smithi188.stdout:4/458: symlink d0/d14/d2f/lb0 0 2024-04-05T21:26:30.363 INFO:tasks.workunit.client.1.smithi188.stdout:4/459: write d0/d2/d2a/d34/f41 [1038844,74766] 0 2024-04-05T21:26:30.363 INFO:tasks.workunit.client.1.smithi188.stdout:1/293: getdents d1/d4/d3f 0 2024-04-05T21:26:30.365 INFO:tasks.workunit.client.1.smithi188.stdout:1/294: creat d1/d7/f5c x:0 0 0 2024-04-05T21:26:30.369 INFO:tasks.workunit.client.1.smithi188.stdout:1/295: symlink d1/dc/d16/d46/l5d 0 2024-04-05T21:26:30.370 INFO:tasks.workunit.client.1.smithi188.stdout:1/296: readlink d1/l32 0 2024-04-05T21:26:30.370 INFO:tasks.workunit.client.1.smithi188.stdout:1/297: rename d1/d4/d13/c37 to d1/dc/d16/d21/d27/d4a/c5e 0 2024-04-05T21:26:30.372 INFO:tasks.workunit.client.1.smithi188.stdout:1/298: mkdir d1/d7/d4e/d5f 0 2024-04-05T21:26:30.372 INFO:tasks.workunit.client.1.smithi188.stdout:1/299: chown d1/f14 5876 1 2024-04-05T21:26:30.373 INFO:tasks.workunit.client.1.smithi188.stdout:1/300: mkdir d1/d4/d13/d60 0 2024-04-05T21:26:30.376 INFO:tasks.workunit.client.1.smithi188.stdout:1/301: symlink d1/dc/d16/d2d/l61 0 2024-04-05T21:26:30.376 INFO:tasks.workunit.client.1.smithi188.stdout:2/277: sync 2024-04-05T21:26:30.380 INFO:tasks.workunit.client.1.smithi188.stdout:1/302: link d1/f14 d1/d7/d4e/f62 0 2024-04-05T21:26:30.387 INFO:tasks.workunit.client.1.smithi188.stdout:2/278: dwrite d5/db/dd/f1c [0,4194304] 0 2024-04-05T21:26:30.394 INFO:tasks.workunit.client.1.smithi188.stdout:1/303: dwrite d1/dc/f15 [0,4194304] 0 2024-04-05T21:26:30.397 INFO:tasks.workunit.client.1.smithi188.stdout:1/304: fdatasync d1/dc/f1b 0 2024-04-05T21:26:30.401 INFO:tasks.workunit.client.1.smithi188.stdout:1/305: chown d1/dc/d16/c55 664145799 1 2024-04-05T21:26:30.401 INFO:tasks.workunit.client.1.smithi188.stdout:2/279: dread d5/f2f [4194304,4194304] 0 2024-04-05T21:26:30.401 INFO:tasks.workunit.client.1.smithi188.stdout:4/460: truncate d0/d14/d2f/d35/f83 326752 0 2024-04-05T21:26:30.403 INFO:tasks.workunit.client.1.smithi188.stdout:1/306: creat d1/dc/d16/d21/d27/d4a/f63 x:0 0 0 2024-04-05T21:26:30.403 INFO:tasks.workunit.client.1.smithi188.stdout:4/461: chown d0/d14/f80 1374702 1 2024-04-05T21:26:30.405 INFO:tasks.workunit.client.1.smithi188.stdout:4/462: chown d0/d2/d2a/d34/f41 213114 1 2024-04-05T21:26:30.406 INFO:tasks.workunit.client.1.smithi188.stdout:1/307: mknod d1/dc/c64 0 2024-04-05T21:26:30.409 INFO:tasks.workunit.client.1.smithi188.stdout:2/280: dwrite d5/db/dd/d24/f2e [0,4194304] 0 2024-04-05T21:26:30.413 INFO:tasks.workunit.client.1.smithi188.stdout:1/308: rmdir d1/d7/d4e/d5a 0 2024-04-05T21:26:30.417 INFO:tasks.workunit.client.1.smithi188.stdout:1/309: symlink d1/l65 0 2024-04-05T21:26:30.417 INFO:tasks.workunit.client.1.smithi188.stdout:2/281: getdents d5/db/dd/d36/d3d 0 2024-04-05T21:26:30.419 INFO:tasks.workunit.client.1.smithi188.stdout:4/463: dwrite d0/d14/d15/d53/d82/d8f/f93 [0,4194304] 0 2024-04-05T21:26:30.424 INFO:tasks.workunit.client.1.smithi188.stdout:2/282: write d5/db/dd/d36/d3c/f4a [4557576,13698] 0 2024-04-05T21:26:30.429 INFO:tasks.workunit.client.1.smithi188.stdout:1/310: dread d1/f14 [0,4194304] 0 2024-04-05T21:26:30.432 INFO:tasks.workunit.client.1.smithi188.stdout:1/311: creat d1/d7/f66 x:0 0 0 2024-04-05T21:26:30.434 INFO:tasks.workunit.client.1.smithi188.stdout:4/464: dwrite d0/d2/d17/fa7 [0,4194304] 0 2024-04-05T21:26:30.435 INFO:tasks.workunit.client.1.smithi188.stdout:2/283: dwrite d5/db/dd/f1c [0,4194304] 0 2024-04-05T21:26:30.438 INFO:tasks.workunit.client.1.smithi188.stdout:1/312: symlink d1/dc/d16/d21/d27/d49/l67 0 2024-04-05T21:26:30.441 INFO:tasks.workunit.client.1.smithi188.stdout:2/284: chown d5/db/d35/c49 1536056 1 2024-04-05T21:26:30.441 INFO:tasks.workunit.client.1.smithi188.stdout:1/313: dread - d1/dc/d16/f3b zero size 2024-04-05T21:26:30.442 INFO:tasks.workunit.client.1.smithi188.stdout:4/465: rename d0/d14/d2f/d35 to d0/d14/d15/d78/db1 0 2024-04-05T21:26:30.444 INFO:tasks.workunit.client.1.smithi188.stdout:2/285: unlink d5/db/dd/f1c 0 2024-04-05T21:26:30.446 INFO:tasks.workunit.client.1.smithi188.stdout:1/314: rename d1/dc/d16/d2d/f34 to d1/dc/d16/d21/f68 0 2024-04-05T21:26:30.449 INFO:tasks.workunit.client.1.smithi188.stdout:1/315: write d1/dc/d16/f3b [640859,31810] 0 2024-04-05T21:26:30.452 INFO:tasks.workunit.client.1.smithi188.stdout:1/316: write d1/dc/d16/d21/f68 [3051945,71620] 0 2024-04-05T21:26:30.454 INFO:tasks.workunit.client.1.smithi188.stdout:1/317: fsync d1/dc/d16/d21/f26 0 2024-04-05T21:26:30.456 INFO:tasks.workunit.client.1.smithi188.stdout:1/318: fsync d1/dc/f11 0 2024-04-05T21:26:30.456 INFO:tasks.workunit.client.1.smithi188.stdout:2/286: dwrite d5/db/dd/f11 [0,4194304] 0 2024-04-05T21:26:30.460 INFO:tasks.workunit.client.1.smithi188.stdout:1/319: creat d1/d4/d3f/f69 x:0 0 0 2024-04-05T21:26:30.462 INFO:tasks.workunit.client.1.smithi188.stdout:1/320: creat d1/d4/d13/f6a x:0 0 0 2024-04-05T21:26:30.462 INFO:tasks.workunit.client.1.smithi188.stdout:2/287: link d5/db/dd/d36/d3d/l4f d5/db/dd/d36/d3c/d48/l50 0 2024-04-05T21:26:30.463 INFO:tasks.workunit.client.1.smithi188.stdout:1/321: read d1/dc/f15 [1115839,4596] 0 2024-04-05T21:26:30.466 INFO:tasks.workunit.client.1.smithi188.stdout:2/288: mknod d5/db/dd/d36/d3d/d46/c51 0 2024-04-05T21:26:30.466 INFO:tasks.workunit.client.1.smithi188.stdout:1/322: mknod d1/dc/d16/d46/c6b 0 2024-04-05T21:26:30.466 INFO:tasks.workunit.client.1.smithi188.stdout:2/289: write d5/db/d35/d3b/f4d [309511,89729] 0 2024-04-05T21:26:30.467 INFO:tasks.workunit.client.1.smithi188.stdout:2/290: chown d5/db/dd/d36/d3d 5500127 1 2024-04-05T21:26:30.469 INFO:tasks.workunit.client.1.smithi188.stdout:1/323: creat d1/dc/d16/f6c x:0 0 0 2024-04-05T21:26:30.470 INFO:tasks.workunit.client.1.smithi188.stdout:2/291: rename d5/fc to d5/db/dd/d24/f52 0 2024-04-05T21:26:30.472 INFO:tasks.workunit.client.1.smithi188.stdout:2/292: mknod d5/c53 0 2024-04-05T21:26:30.482 INFO:tasks.workunit.client.1.smithi188.stdout:2/293: dread d5/db/dd/d24/f52 [4194304,4194304] 0 2024-04-05T21:26:30.484 INFO:tasks.workunit.client.1.smithi188.stdout:2/294: creat d5/db/dd/f54 x:0 0 0 2024-04-05T21:26:30.492 INFO:tasks.workunit.client.1.smithi188.stdout:2/295: truncate d5/f22 7573798 0 2024-04-05T21:26:30.494 INFO:tasks.workunit.client.1.smithi188.stdout:2/296: mknod d5/db/dd/d36/d3c/d48/c55 0 2024-04-05T21:26:30.497 INFO:tasks.workunit.client.1.smithi188.stdout:2/297: rename d5/db/f3e to d5/db/dd/d36/d3c/d48/f56 0 2024-04-05T21:26:30.500 INFO:tasks.workunit.client.1.smithi188.stdout:2/298: link d5/db/d35/f43 d5/db/dd/d36/d3d/d46/f57 0 2024-04-05T21:26:30.507 INFO:tasks.workunit.client.1.smithi188.stdout:2/299: truncate d5/db/dd/d24/f52 7015936 0 2024-04-05T21:26:30.512 INFO:tasks.workunit.client.1.smithi188.stdout:2/300: dwrite d5/db/dd/f15 [4194304,4194304] 0 2024-04-05T21:26:30.531 INFO:tasks.workunit.client.1.smithi188.stdout:2/301: dwrite d5/db/dd/d24/f52 [4194304,4194304] 0 2024-04-05T21:26:30.534 INFO:tasks.workunit.client.1.smithi188.stdout:2/302: mknod d5/db/dd/d36/d3c/d48/c58 0 2024-04-05T21:26:30.536 INFO:tasks.workunit.client.1.smithi188.stdout:2/303: mknod d5/db/c59 0 2024-04-05T21:26:30.538 INFO:tasks.workunit.client.1.smithi188.stdout:2/304: creat d5/db/d35/d3b/f5a x:0 0 0 2024-04-05T21:26:30.539 INFO:tasks.workunit.client.1.smithi188.stdout:2/305: chown d5/db/dd/d36/d3c/f4a 0 1 2024-04-05T21:26:30.546 INFO:tasks.workunit.client.1.smithi188.stdout:2/306: dwrite d5/db/f16 [8388608,4194304] 0 2024-04-05T21:26:30.546 INFO:tasks.workunit.client.1.smithi188.stdout:2/307: chown d5/l2a 3 1 2024-04-05T21:26:30.550 INFO:tasks.workunit.client.1.smithi188.stdout:2/308: creat d5/db/dd/d36/d3c/d48/f5b x:0 0 0 2024-04-05T21:26:30.551 INFO:tasks.workunit.client.1.smithi188.stdout:2/309: dread - d5/db/dd/d36/d3c/d48/f5b zero size 2024-04-05T21:26:30.553 INFO:tasks.workunit.client.1.smithi188.stdout:2/310: rmdir d5/db/dd/d36/d3d 39 2024-04-05T21:26:30.555 INFO:tasks.workunit.client.1.smithi188.stdout:2/311: creat d5/f5c x:0 0 0 2024-04-05T21:26:30.560 INFO:tasks.workunit.client.1.smithi188.stdout:2/312: dwrite d5/f2f [4194304,4194304] 0 2024-04-05T21:26:30.573 INFO:tasks.workunit.client.1.smithi188.stdout:2/313: dwrite d5/db/d35/f43 [0,4194304] 0 2024-04-05T21:26:30.576 INFO:tasks.workunit.client.1.smithi188.stdout:2/314: mkdir d5/db/dd/d36/d3d/d46/d5d 0 2024-04-05T21:26:30.580 INFO:tasks.workunit.client.1.smithi188.stdout:2/315: rename d5/db/dd/d36/d3d/d46/f57 to d5/db/dd/d36/d3d/d46/d5d/f5e 0 2024-04-05T21:26:30.598 INFO:tasks.workunit.client.1.smithi188.stdout:2/316: dread d5/db/f16 [0,4194304] 0 2024-04-05T21:26:30.600 INFO:tasks.workunit.client.1.smithi188.stdout:2/317: mkdir d5/d5f 0 2024-04-05T21:26:30.603 INFO:tasks.workunit.client.1.smithi188.stdout:2/318: link d5/db/dd/d36/l3a d5/d5f/l60 0 2024-04-05T21:26:30.604 INFO:tasks.workunit.client.1.smithi188.stdout:2/319: fdatasync d5/db/dd/d24/f2e 0 2024-04-05T21:26:30.608 INFO:tasks.workunit.client.1.smithi188.stdout:2/320: getdents d5/db/dd/d36 0 2024-04-05T21:26:30.609 INFO:tasks.workunit.client.1.smithi188.stdout:2/321: chown d5/db/d35/l4b 1059282 1 2024-04-05T21:26:30.615 INFO:tasks.workunit.client.1.smithi188.stdout:2/322: dwrite d5/db/dd/d36/d3c/d48/f5b [0,4194304] 0 2024-04-05T21:26:30.621 INFO:tasks.workunit.client.1.smithi188.stdout:2/323: dwrite d5/db/d35/d3b/f4d [0,4194304] 0 2024-04-05T21:26:30.626 INFO:tasks.workunit.client.1.smithi188.stdout:2/324: getdents d5/db/dd/d36/d3c/d48 0 2024-04-05T21:26:30.629 INFO:tasks.workunit.client.1.smithi188.stdout:2/325: symlink d5/db/dd/d36/d3d/d46/l61 0 2024-04-05T21:26:30.629 INFO:tasks.workunit.client.1.smithi188.stdout:2/326: readlink d5/db/dd/l26 0 2024-04-05T21:26:30.630 INFO:tasks.workunit.client.1.smithi188.stdout:2/327: symlink d5/db/dd/d36/l62 0 2024-04-05T21:26:30.639 INFO:tasks.workunit.client.1.smithi188.stdout:2/328: write d5/f22 [6195211,76066] 0 2024-04-05T21:26:30.644 INFO:tasks.workunit.client.1.smithi188.stdout:2/329: dwrite d5/db/d35/f43 [0,4194304] 0 2024-04-05T21:26:30.646 INFO:tasks.workunit.client.1.smithi188.stdout:2/330: write d5/db/dd/f15 [4301363,110768] 0 2024-04-05T21:26:30.649 INFO:tasks.workunit.client.1.smithi188.stdout:2/331: mkdir d5/db/dd/d36/d3c/d63 0 2024-04-05T21:26:30.651 INFO:tasks.workunit.client.1.smithi188.stdout:2/332: symlink d5/db/dd/d36/d3c/d48/l64 0 2024-04-05T21:26:30.671 INFO:tasks.workunit.client.1.smithi188.stdout:2/333: dread d5/fa [0,4194304] 0 2024-04-05T21:26:30.673 INFO:tasks.workunit.client.1.smithi188.stdout:2/334: creat d5/db/dd/d36/d3d/d46/f65 x:0 0 0 2024-04-05T21:26:30.676 INFO:tasks.workunit.client.1.smithi188.stdout:2/335: rename d5/db/d34 to d5/db/dd/d36/d3d/d46/d66 0 2024-04-05T21:26:30.677 INFO:tasks.workunit.client.1.smithi188.stdout:2/336: mkdir d5/db/dd/d67 0 2024-04-05T21:26:30.697 INFO:tasks.workunit.client.1.smithi188.stdout:8/157: sync 2024-04-05T21:26:30.697 INFO:tasks.workunit.client.1.smithi188.stdout:7/371: sync 2024-04-05T21:26:30.699 INFO:tasks.workunit.client.1.smithi188.stdout:2/337: dwrite d5/f2f [4194304,4194304] 0 2024-04-05T21:26:30.701 INFO:tasks.workunit.client.1.smithi188.stdout:7/372: rmdir d3/d2c 39 2024-04-05T21:26:30.702 INFO:tasks.workunit.client.1.smithi188.stdout:8/158: symlink d5/d21/d2a/l37 0 2024-04-05T21:26:30.705 INFO:tasks.workunit.client.1.smithi188.stdout:7/373: dread - d3/d2c/f3f zero size 2024-04-05T21:26:30.710 INFO:tasks.workunit.client.1.smithi188.stdout:8/159: mknod d5/c38 0 2024-04-05T21:26:30.711 INFO:tasks.workunit.client.1.smithi188.stdout:8/160: truncate d5/f12 754540 0 2024-04-05T21:26:30.713 INFO:tasks.workunit.client.1.smithi188.stdout:7/374: creat d3/d2c/d5e/f78 x:0 0 0 2024-04-05T21:26:30.714 INFO:tasks.workunit.client.1.smithi188.stdout:2/338: link d5/db/dd/d36/d3c/d48/l50 d5/db/dd/d36/d3c/d63/l68 0 2024-04-05T21:26:30.717 INFO:tasks.workunit.client.1.smithi188.stdout:7/375: mkdir d3/d4/db/d2e/d49/d79 0 2024-04-05T21:26:30.718 INFO:tasks.workunit.client.1.smithi188.stdout:2/339: rmdir d5/db/dd/d24 39 2024-04-05T21:26:30.718 INFO:tasks.workunit.client.1.smithi188.stdout:8/161: dwrite d5/f1c [0,4194304] 0 2024-04-05T21:26:30.719 INFO:tasks.workunit.client.1.smithi188.stdout:8/162: stat d5/f1f 0 2024-04-05T21:26:30.720 INFO:tasks.workunit.client.1.smithi188.stdout:2/340: write d5/db/d35/d3b/f5a [575624,91392] 0 2024-04-05T21:26:30.723 INFO:tasks.workunit.client.1.smithi188.stdout:8/163: rmdir d5 39 2024-04-05T21:26:30.725 INFO:tasks.workunit.client.1.smithi188.stdout:7/376: creat d3/d4/d1d/d74/d73/f7a x:0 0 0 2024-04-05T21:26:30.726 INFO:tasks.workunit.client.1.smithi188.stdout:7/377: stat d3/d4/d1d/d74/l57 0 2024-04-05T21:26:30.727 INFO:tasks.workunit.client.1.smithi188.stdout:2/341: mknod d5/db/dd/c69 0 2024-04-05T21:26:30.728 INFO:tasks.workunit.client.1.smithi188.stdout:2/342: write d5/f5c [541139,35169] 0 2024-04-05T21:26:30.728 INFO:tasks.workunit.client.1.smithi188.stdout:7/378: fdatasync d3/d4/f2d 0 2024-04-05T21:26:30.730 INFO:tasks.workunit.client.1.smithi188.stdout:8/164: truncate d5/f1f 2376349 0 2024-04-05T21:26:30.730 INFO:tasks.workunit.client.1.smithi188.stdout:7/379: dread - d3/d4/d1d/d74/d73/f7a zero size 2024-04-05T21:26:30.731 INFO:tasks.workunit.client.1.smithi188.stdout:2/343: write d5/db/dd/d36/d3c/d48/f5b [4651039,93149] 0 2024-04-05T21:26:30.732 INFO:tasks.workunit.client.1.smithi188.stdout:7/380: read - d3/f71 zero size 2024-04-05T21:26:30.733 INFO:tasks.workunit.client.1.smithi188.stdout:8/165: creat d5/d21/d2a/f39 x:0 0 0 2024-04-05T21:26:30.734 INFO:tasks.workunit.client.1.smithi188.stdout:8/166: readlink d5/d21/l2b 0 2024-04-05T21:26:30.734 INFO:tasks.workunit.client.1.smithi188.stdout:7/381: write d3/f1e [4948008,103927] 0 2024-04-05T21:26:30.735 INFO:tasks.workunit.client.1.smithi188.stdout:2/344: symlink d5/db/d35/l6a 0 2024-04-05T21:26:30.736 INFO:tasks.workunit.client.1.smithi188.stdout:8/167: truncate d5/dc/f2c 46751 0 2024-04-05T21:26:30.745 INFO:tasks.workunit.client.1.smithi188.stdout:8/168: link d5/dc/f11 d5/d21/f3a 0 2024-04-05T21:26:30.745 INFO:tasks.workunit.client.1.smithi188.stdout:7/382: dwrite d3/d4/db/f6e [0,4194304] 0 2024-04-05T21:26:30.746 INFO:tasks.workunit.client.1.smithi188.stdout:2/345: dwrite d5/f2f [4194304,4194304] 0 2024-04-05T21:26:30.748 INFO:tasks.workunit.client.1.smithi188.stdout:8/169: chown d5/c27 422387810 1 2024-04-05T21:26:30.750 INFO:tasks.workunit.client.1.smithi188.stdout:2/346: mkdir d5/db/d35/d3b/d6b 0 2024-04-05T21:26:30.754 INFO:tasks.workunit.client.1.smithi188.stdout:8/170: write d5/f1e [3585234,46818] 0 2024-04-05T21:26:30.754 INFO:tasks.workunit.client.1.smithi188.stdout:2/347: symlink d5/db/d35/d3b/d6b/l6c 0 2024-04-05T21:26:30.755 INFO:tasks.workunit.client.1.smithi188.stdout:2/348: chown d5/db/d35/d3b 670 1 2024-04-05T21:26:30.758 INFO:tasks.workunit.client.1.smithi188.stdout:8/171: rename d5/d21/f2e to d5/dc/f3b 0 2024-04-05T21:26:30.758 INFO:tasks.workunit.client.1.smithi188.stdout:2/349: chown d5/d5f/l60 12362885 1 2024-04-05T21:26:30.762 INFO:tasks.workunit.client.1.smithi188.stdout:8/172: creat d5/f3c x:0 0 0 2024-04-05T21:26:30.768 INFO:tasks.workunit.client.1.smithi188.stdout:8/173: dwrite d5/f1c [0,4194304] 0 2024-04-05T21:26:30.777 INFO:tasks.workunit.client.1.smithi188.stdout:7/383: dread d3/d4/db/f38 [0,4194304] 0 2024-04-05T21:26:30.779 INFO:tasks.workunit.client.1.smithi188.stdout:8/174: write d5/dc/f36 [2323828,69477] 0 2024-04-05T21:26:30.781 INFO:tasks.workunit.client.1.smithi188.stdout:7/384: unlink d3/d4/l54 0 2024-04-05T21:26:30.786 INFO:tasks.workunit.client.1.smithi188.stdout:8/175: creat d5/f3d x:0 0 0 2024-04-05T21:26:30.786 INFO:tasks.workunit.client.1.smithi188.stdout:7/385: rename d3/d4/db/d2e/d37/d65/c29 to d3/d4/db/d55/c7b 0 2024-04-05T21:26:30.786 INFO:tasks.workunit.client.1.smithi188.stdout:8/176: rmdir d5/d21/d2a 39 2024-04-05T21:26:30.787 INFO:tasks.workunit.client.1.smithi188.stdout:7/386: mkdir d3/d4/d7c 0 2024-04-05T21:26:30.788 INFO:tasks.workunit.client.1.smithi188.stdout:8/177: fsync d5/d21/f3a 0 2024-04-05T21:26:30.789 INFO:tasks.workunit.client.1.smithi188.stdout:8/178: readlink d5/dc/ld 0 2024-04-05T21:26:30.790 INFO:tasks.workunit.client.1.smithi188.stdout:8/179: readlink d5/dc/l25 0 2024-04-05T21:26:30.790 INFO:tasks.workunit.client.1.smithi188.stdout:7/387: creat d3/d4/db/d2e/d37/d65/f7d x:0 0 0 2024-04-05T21:26:30.793 INFO:tasks.workunit.client.1.smithi188.stdout:7/388: read d3/d4/d1d/d74/f5c [2281207,70475] 0 2024-04-05T21:26:30.794 INFO:tasks.workunit.client.1.smithi188.stdout:8/180: symlink d5/l3e 0 2024-04-05T21:26:30.794 INFO:tasks.workunit.client.1.smithi188.stdout:7/389: fsync d3/d2c/f3f 0 2024-04-05T21:26:30.795 INFO:tasks.workunit.client.1.smithi188.stdout:8/181: truncate d5/f12 1323035 0 2024-04-05T21:26:30.796 INFO:tasks.workunit.client.1.smithi188.stdout:7/390: chown d3/d4/d1d/l64 16298 1 2024-04-05T21:26:30.797 INFO:tasks.workunit.client.1.smithi188.stdout:8/182: write d5/dc/f32 [5786626,119571] 0 2024-04-05T21:26:30.800 INFO:tasks.workunit.client.1.smithi188.stdout:7/391: creat d3/d2c/f7e x:0 0 0 2024-04-05T21:26:30.803 INFO:tasks.workunit.client.1.smithi188.stdout:7/392: symlink d3/d4/db/d2e/d3b/l7f 0 2024-04-05T21:26:30.804 INFO:tasks.workunit.client.1.smithi188.stdout:8/183: dwrite d5/f1f [0,4194304] 0 2024-04-05T21:26:30.805 INFO:tasks.workunit.client.1.smithi188.stdout:8/184: fdatasync d5/dc/f2c 0 2024-04-05T21:26:30.807 INFO:tasks.workunit.client.1.smithi188.stdout:7/393: creat d3/d4/d1d/d74/f80 x:0 0 0 2024-04-05T21:26:30.809 INFO:tasks.workunit.client.1.smithi188.stdout:8/185: symlink d5/dc/l3f 0 2024-04-05T21:26:30.810 INFO:tasks.workunit.client.1.smithi188.stdout:8/186: rmdir d5/d21 39 2024-04-05T21:26:30.813 INFO:tasks.workunit.client.1.smithi188.stdout:8/187: write d5/d21/d2a/f31 [17963,100097] 0 2024-04-05T21:26:30.814 INFO:tasks.workunit.client.1.smithi188.stdout:8/188: chown d5/f2f 0 1 2024-04-05T21:26:30.821 INFO:tasks.workunit.client.1.smithi188.stdout:8/189: dwrite d5/f2d [4194304,4194304] 0 2024-04-05T21:26:30.825 INFO:tasks.workunit.client.1.smithi188.stdout:8/190: dread d5/f1c [0,4194304] 0 2024-04-05T21:26:30.827 INFO:tasks.workunit.client.1.smithi188.stdout:8/191: mkdir d5/d21/d40 0 2024-04-05T21:26:30.829 INFO:tasks.workunit.client.1.smithi188.stdout:8/192: getdents d5/d21/d40 0 2024-04-05T21:26:30.835 INFO:tasks.workunit.client.1.smithi188.stdout:7/394: dread d3/d4/db/d10/f14 [4194304,4194304] 0 2024-04-05T21:26:30.836 INFO:tasks.workunit.client.1.smithi188.stdout:8/193: dwrite d5/f2d [0,4194304] 0 2024-04-05T21:26:30.838 INFO:tasks.workunit.client.1.smithi188.stdout:7/395: creat d3/d4/db/d2e/d37/d65/f81 x:0 0 0 2024-04-05T21:26:30.845 INFO:tasks.workunit.client.1.smithi188.stdout:8/194: truncate d5/f20 203288 0 2024-04-05T21:26:30.851 INFO:tasks.workunit.client.1.smithi188.stdout:8/195: dwrite d5/dc/f32 [0,4194304] 0 2024-04-05T21:26:30.854 INFO:tasks.workunit.client.1.smithi188.stdout:8/196: mknod d5/d21/d2a/c41 0 2024-04-05T21:26:30.857 INFO:tasks.workunit.client.1.smithi188.stdout:8/197: fdatasync d5/fb 0 2024-04-05T21:26:30.860 INFO:tasks.workunit.client.1.smithi188.stdout:8/198: creat d5/f42 x:0 0 0 2024-04-05T21:26:30.862 INFO:tasks.workunit.client.1.smithi188.stdout:8/199: mkdir d5/d21/d40/d43 0 2024-04-05T21:26:30.869 INFO:tasks.workunit.client.1.smithi188.stdout:8/200: truncate d5/f1c 2227064 0 2024-04-05T21:26:30.871 INFO:tasks.workunit.client.1.smithi188.stdout:8/201: mkdir d5/d21/d40/d43/d44 0 2024-04-05T21:26:30.872 INFO:tasks.workunit.client.1.smithi188.stdout:8/202: dread - d5/d21/f3a zero size 2024-04-05T21:26:30.874 INFO:tasks.workunit.client.1.smithi188.stdout:8/203: unlink d5/dc/ld 0 2024-04-05T21:26:30.884 INFO:tasks.workunit.client.1.smithi188.stdout:8/204: dread d5/d21/d2a/f31 [0,4194304] 0 2024-04-05T21:26:30.884 INFO:tasks.workunit.client.1.smithi188.stdout:8/205: fsync d5/dc/f33 0 2024-04-05T21:26:30.885 INFO:tasks.workunit.client.1.smithi188.stdout:3/294: sync 2024-04-05T21:26:30.886 INFO:tasks.workunit.client.1.smithi188.stdout:9/311: sync 2024-04-05T21:26:30.886 INFO:tasks.workunit.client.1.smithi188.stdout:4/466: sync 2024-04-05T21:26:30.886 INFO:tasks.workunit.client.1.smithi188.stdout:6/391: sync 2024-04-05T21:26:30.886 INFO:tasks.workunit.client.1.smithi188.stdout:0/197: sync 2024-04-05T21:26:30.886 INFO:tasks.workunit.client.1.smithi188.stdout:5/241: sync 2024-04-05T21:26:30.887 INFO:tasks.workunit.client.1.smithi188.stdout:9/312: chown dd/d11/l15 122727719 1 2024-04-05T21:26:30.889 INFO:tasks.workunit.client.1.smithi188.stdout:5/242: write d6/f2c [2437010,81479] 0 2024-04-05T21:26:30.890 INFO:tasks.workunit.client.1.smithi188.stdout:0/198: write d4/d8/fa [4444888,83917] 0 2024-04-05T21:26:30.891 INFO:tasks.workunit.client.1.smithi188.stdout:3/295: truncate d5/dc/f2a 1383815 0 2024-04-05T21:26:30.893 INFO:tasks.workunit.client.1.smithi188.stdout:9/313: truncate dd/f5f 228527 0 2024-04-05T21:26:30.895 INFO:tasks.workunit.client.1.smithi188.stdout:5/243: mkdir d6/d15/d5c 0 2024-04-05T21:26:30.896 INFO:tasks.workunit.client.1.smithi188.stdout:8/206: dwrite d5/f1e [0,4194304] 0 2024-04-05T21:26:30.897 INFO:tasks.workunit.client.1.smithi188.stdout:3/296: fsync d5/d35/f40 0 2024-04-05T21:26:30.898 INFO:tasks.workunit.client.1.smithi188.stdout:9/314: rename dd/d11 to dd/d11/d22/d28/d40/d59/d65 22 2024-04-05T21:26:30.902 INFO:tasks.workunit.client.1.smithi188.stdout:3/297: mkdir d5/d7/d67 0 2024-04-05T21:26:30.904 INFO:tasks.workunit.client.1.smithi188.stdout:5/244: creat d6/d17/d24/d44/f5d x:0 0 0 2024-04-05T21:26:30.905 INFO:tasks.workunit.client.1.smithi188.stdout:6/392: dwrite d6/d13/f1d [0,4194304] 0 2024-04-05T21:26:30.908 INFO:tasks.workunit.client.1.smithi188.stdout:8/207: symlink d5/d21/d40/d43/d44/l45 0 2024-04-05T21:26:30.909 INFO:tasks.workunit.client.1.smithi188.stdout:5/245: unlink d6/d17/c1c 0 2024-04-05T21:26:30.909 INFO:tasks.workunit.client.1.smithi188.stdout:3/298: chown d5/d7/d65/d5f/f64 30632571 1 2024-04-05T21:26:30.912 INFO:tasks.workunit.client.1.smithi188.stdout:6/393: creat d6/d13/d48/d49/d60/f7c x:0 0 0 2024-04-05T21:26:30.913 INFO:tasks.workunit.client.1.smithi188.stdout:5/246: read d6/f2c [294332,31369] 0 2024-04-05T21:26:30.914 INFO:tasks.workunit.client.1.smithi188.stdout:8/208: creat d5/d21/d40/d43/f46 x:0 0 0 2024-04-05T21:26:30.914 INFO:tasks.workunit.client.1.smithi188.stdout:4/467: dread d0/f5 [4194304,4194304] 0 2024-04-05T21:26:30.916 INFO:tasks.workunit.client.1.smithi188.stdout:6/394: rmdir d6/d13/d48/d49 39 2024-04-05T21:26:30.918 INFO:tasks.workunit.client.1.smithi188.stdout:8/209: mkdir d5/d21/d2a/d47 0 2024-04-05T21:26:30.920 INFO:tasks.workunit.client.1.smithi188.stdout:5/247: fdatasync d6/d17/f1f 0 2024-04-05T21:26:30.924 INFO:tasks.workunit.client.1.smithi188.stdout:3/299: dwrite d5/d7/d31/f5c [0,4194304] 0 2024-04-05T21:26:30.926 INFO:tasks.workunit.client.1.smithi188.stdout:4/468: dread d0/d14/d15/d1b/f4d [0,4194304] 0 2024-04-05T21:26:30.926 INFO:tasks.workunit.client.1.smithi188.stdout:3/300: dread - d5/d35/d4e/f5b zero size 2024-04-05T21:26:30.928 INFO:tasks.workunit.client.1.smithi188.stdout:3/301: stat f4 0 2024-04-05T21:26:30.928 INFO:tasks.workunit.client.1.smithi188.stdout:0/199: dread d4/d8/db/f1d [4194304,4194304] 0 2024-04-05T21:26:30.928 INFO:tasks.workunit.client.1.smithi188.stdout:8/210: mkdir d5/d21/d40/d43/d48 0 2024-04-05T21:26:30.931 INFO:tasks.workunit.client.1.smithi188.stdout:4/469: chown d0/d14/d15/d1b/f4d 69 1 2024-04-05T21:26:30.932 INFO:tasks.workunit.client.1.smithi188.stdout:3/302: read d5/d7/f25 [410356,78170] 0 2024-04-05T21:26:30.932 INFO:tasks.workunit.client.1.smithi188.stdout:4/470: write d0/d14/d15/d42/fa5 [2757809,19048] 0 2024-04-05T21:26:30.936 INFO:tasks.workunit.client.1.smithi188.stdout:8/211: rename d5/ff to d5/d21/d2a/d47/f49 0 2024-04-05T21:26:30.944 INFO:tasks.workunit.client.1.smithi188.stdout:9/315: dwrite dd/d11/d1f/d25/f50 [0,4194304] 0 2024-04-05T21:26:30.944 INFO:tasks.workunit.client.1.smithi188.stdout:8/212: symlink d5/d21/d2a/l4a 0 2024-04-05T21:26:30.945 INFO:tasks.workunit.client.1.smithi188.stdout:9/316: chown dd/d35/c57 67599270 1 2024-04-05T21:26:30.946 INFO:tasks.workunit.client.1.smithi188.stdout:8/213: chown d5/d21/d2a/f39 1541899880 1 2024-04-05T21:26:30.946 INFO:tasks.workunit.client.1.smithi188.stdout:4/471: read d0/d2/d17/d48/f57 [1984638,87286] 0 2024-04-05T21:26:30.947 INFO:tasks.workunit.client.1.smithi188.stdout:4/472: stat d0/d2/d17/c33 0 2024-04-05T21:26:30.947 INFO:tasks.workunit.client.1.smithi188.stdout:8/214: read - d5/d21/d40/d43/f46 zero size 2024-04-05T21:26:30.948 INFO:tasks.workunit.client.1.smithi188.stdout:4/473: chown d0/d14/l73 188098143 1 2024-04-05T21:26:30.950 INFO:tasks.workunit.client.1.smithi188.stdout:5/248: link d6/d17/d23/d35/f47 d6/d17/d23/d35/d40/d42/f5e 0 2024-04-05T21:26:30.954 INFO:tasks.workunit.client.1.smithi188.stdout:9/317: creat dd/d49/f66 x:0 0 0 2024-04-05T21:26:30.956 INFO:tasks.workunit.client.1.smithi188.stdout:6/395: dread d6/d13/d38/f63 [0,4194304] 0 2024-04-05T21:26:30.961 INFO:tasks.workunit.client.1.smithi188.stdout:8/215: dwrite d5/dc/f19 [0,4194304] 0 2024-04-05T21:26:30.963 INFO:tasks.workunit.client.1.smithi188.stdout:4/474: dwrite d0/d2/d2a/d34/f77 [0,4194304] 0 2024-04-05T21:26:30.966 INFO:tasks.workunit.client.1.smithi188.stdout:9/318: rename dd/d11/d1f/d26/f3b to dd/d35/f67 0 2024-04-05T21:26:30.974 INFO:tasks.workunit.client.1.smithi188.stdout:9/319: symlink dd/d11/d1f/d25/l68 0 2024-04-05T21:26:30.974 INFO:tasks.workunit.client.1.smithi188.stdout:3/303: dwrite d5/d7/f30 [0,4194304] 0 2024-04-05T21:26:30.975 INFO:tasks.workunit.client.1.smithi188.stdout:9/320: stat dd/d11/d1f/d26 0 2024-04-05T21:26:30.978 INFO:tasks.workunit.client.1.smithi188.stdout:4/475: creat d0/d2/d17/d3a/fb2 x:0 0 0 2024-04-05T21:26:30.984 INFO:tasks.workunit.client.1.smithi188.stdout:5/249: write d6/d17/d23/d35/d40/d42/f5e [3850088,35108] 0 2024-04-05T21:26:30.985 INFO:tasks.workunit.client.1.smithi188.stdout:9/321: link dd/l1b dd/d11/d1f/d25/d51/l69 0 2024-04-05T21:26:30.990 INFO:tasks.workunit.client.1.smithi188.stdout:6/396: truncate d6/d13/d38/f47 2800208 0 2024-04-05T21:26:30.990 INFO:tasks.workunit.client.1.smithi188.stdout:6/397: dread - d6/d13/d38/f55 zero size 2024-04-05T21:26:30.990 INFO:tasks.workunit.client.1.smithi188.stdout:5/250: link c4 d6/d17/d39/c5f 0 2024-04-05T21:26:30.993 INFO:tasks.workunit.client.1.smithi188.stdout:6/398: link d6/d28/l79 d6/d13/d18/d19/d2c/l7d 0 2024-04-05T21:26:30.997 INFO:tasks.workunit.client.1.smithi188.stdout:5/251: dwrite d6/d17/d24/f3c [0,4194304] 0 2024-04-05T21:26:30.997 INFO:tasks.workunit.client.1.smithi188.stdout:6/399: fdatasync d6/d13/d38/f6f 0 2024-04-05T21:26:30.999 INFO:tasks.workunit.client.1.smithi188.stdout:5/252: read - d6/d17/f20 zero size 2024-04-05T21:26:31.000 INFO:tasks.workunit.client.1.smithi188.stdout:5/253: chown d6/d15/d1b/l25 19 1 2024-04-05T21:26:31.003 INFO:tasks.workunit.client.1.smithi188.stdout:6/400: creat d6/d13/d18/d19/f7e x:0 0 0 2024-04-05T21:26:31.007 INFO:tasks.workunit.client.1.smithi188.stdout:5/254: truncate d6/d17/d24/d44/f57 521704 0 2024-04-05T21:26:31.008 INFO:tasks.workunit.client.1.smithi188.stdout:8/216: dwrite d5/f1c [0,4194304] 0 2024-04-05T21:26:31.011 INFO:tasks.workunit.client.1.smithi188.stdout:5/255: stat d6/d15/d1b/c32 0 2024-04-05T21:26:31.011 INFO:tasks.workunit.client.1.smithi188.stdout:8/217: mkdir d5/d4b 0 2024-04-05T21:26:31.015 INFO:tasks.workunit.client.1.smithi188.stdout:8/218: creat d5/d21/d40/d43/f4c x:0 0 0 2024-04-05T21:26:31.015 INFO:tasks.workunit.client.1.smithi188.stdout:5/256: symlink d6/d17/d24/d44/d51/l60 0 2024-04-05T21:26:31.018 INFO:tasks.workunit.client.1.smithi188.stdout:6/401: dwrite d6/d13/d18/d19/d2c/f65 [0,4194304] 0 2024-04-05T21:26:31.020 INFO:tasks.workunit.client.1.smithi188.stdout:8/219: symlink d5/d21/d2a/l4d 0 2024-04-05T21:26:31.020 INFO:tasks.workunit.client.1.smithi188.stdout:3/304: dread d5/dc/f2a [0,4194304] 0 2024-04-05T21:26:31.021 INFO:tasks.workunit.client.1.smithi188.stdout:6/402: chown d6/d13/d18/f35 685727 1 2024-04-05T21:26:31.022 INFO:tasks.workunit.client.1.smithi188.stdout:5/257: write d6/d15/d1b/f34 [1484964,64678] 0 2024-04-05T21:26:31.022 INFO:tasks.workunit.client.1.smithi188.stdout:8/220: write d5/f12 [803991,67573] 0 2024-04-05T21:26:31.023 INFO:tasks.workunit.client.1.smithi188.stdout:8/221: read - d5/f2f zero size 2024-04-05T21:26:31.026 INFO:tasks.workunit.client.1.smithi188.stdout:8/222: fsync d5/dc/f17 0 2024-04-05T21:26:31.027 INFO:tasks.workunit.client.1.smithi188.stdout:9/322: dread dd/d11/d1f/d26/f19 [0,4194304] 0 2024-04-05T21:26:31.031 INFO:tasks.workunit.client.1.smithi188.stdout:8/223: truncate d5/dc/f1d 535355 0 2024-04-05T21:26:31.032 INFO:tasks.workunit.client.1.smithi188.stdout:8/224: read d5/f15 [255513,7686] 0 2024-04-05T21:26:31.033 INFO:tasks.workunit.client.1.smithi188.stdout:6/403: rename d6/d13/d18/l21 to d6/d7/l7f 0 2024-04-05T21:26:31.035 INFO:tasks.workunit.client.1.smithi188.stdout:3/305: creat d5/f68 x:0 0 0 2024-04-05T21:26:31.035 INFO:tasks.workunit.client.1.smithi188.stdout:6/404: readlink d6/d28/l79 0 2024-04-05T21:26:31.037 INFO:tasks.workunit.client.1.smithi188.stdout:9/323: link dd/d11/d1f/d25/d38/f39 dd/d11/d1f/f6a 0 2024-04-05T21:26:31.040 INFO:tasks.workunit.client.1.smithi188.stdout:3/306: unlink d5/dc/f49 0 2024-04-05T21:26:31.044 INFO:tasks.workunit.client.1.smithi188.stdout:6/405: creat d6/d13/d48/d49/f80 x:0 0 0 2024-04-05T21:26:31.046 INFO:tasks.workunit.client.1.smithi188.stdout:8/225: rename d5/d21/d2a/l37 to d5/d21/l4e 0 2024-04-05T21:26:31.050 INFO:tasks.workunit.client.1.smithi188.stdout:3/307: truncate d5/f1e 548787 0 2024-04-05T21:26:31.053 INFO:tasks.workunit.client.1.smithi188.stdout:9/324: rename dd/d11/d22/d28 to dd/d49/d6b 0 2024-04-05T21:26:31.053 INFO:tasks.workunit.client.1.smithi188.stdout:5/258: dread d6/f2d [0,4194304] 0 2024-04-05T21:26:31.054 INFO:tasks.workunit.client.1.smithi188.stdout:6/406: dwrite d6/d7/fd [0,4194304] 0 2024-04-05T21:26:31.054 INFO:tasks.workunit.client.1.smithi188.stdout:5/259: readlink d6/l8 0 2024-04-05T21:26:31.054 INFO:tasks.workunit.client.1.smithi188.stdout:8/226: creat d5/d21/d40/f4f x:0 0 0 2024-04-05T21:26:31.057 INFO:tasks.workunit.client.1.smithi188.stdout:3/308: creat d5/dc/f69 x:0 0 0 2024-04-05T21:26:31.058 INFO:tasks.workunit.client.1.smithi188.stdout:9/325: write dd/d49/f52 [594795,21454] 0 2024-04-05T21:26:31.064 INFO:tasks.workunit.client.1.smithi188.stdout:5/260: dread d6/f5b [0,4194304] 0 2024-04-05T21:26:31.066 INFO:tasks.workunit.client.1.smithi188.stdout:5/261: stat d6/fe 0 2024-04-05T21:26:31.069 INFO:tasks.workunit.client.1.smithi188.stdout:3/309: dwrite d5/dc/f5a [0,4194304] 0 2024-04-05T21:26:31.069 INFO:tasks.workunit.client.1.smithi188.stdout:5/262: creat d6/d15/f61 x:0 0 0 2024-04-05T21:26:31.072 INFO:tasks.workunit.client.1.smithi188.stdout:5/263: chown d6/l31 76088 1 2024-04-05T21:26:31.074 INFO:tasks.workunit.client.1.smithi188.stdout:3/310: stat d5/df/c28 0 2024-04-05T21:26:31.077 INFO:tasks.workunit.client.1.smithi188.stdout:3/311: write d5/d7/f20 [763510,111431] 0 2024-04-05T21:26:31.079 INFO:tasks.workunit.client.1.smithi188.stdout:6/407: write d6/f64 [1869095,43035] 0 2024-04-05T21:26:31.083 INFO:tasks.workunit.client.1.smithi188.stdout:3/312: mknod d5/d7/d65/d61/c6a 0 2024-04-05T21:26:31.084 INFO:tasks.workunit.client.1.smithi188.stdout:5/264: dwrite d6/d17/d23/d35/f3b [0,4194304] 0 2024-04-05T21:26:31.085 INFO:tasks.workunit.client.1.smithi188.stdout:9/326: write dd/d11/d1f/d26/d55/f63 [2923905,106041] 0 2024-04-05T21:26:31.093 INFO:tasks.workunit.client.1.smithi188.stdout:5/265: creat d6/d15/d1b/f62 x:0 0 0 2024-04-05T21:26:31.093 INFO:tasks.workunit.client.1.smithi188.stdout:3/313: dread d5/d7/f25 [0,4194304] 0 2024-04-05T21:26:31.094 INFO:tasks.workunit.client.1.smithi188.stdout:6/408: dread d6/d13/d18/d19/d2c/f65 [0,4194304] 0 2024-04-05T21:26:31.095 INFO:tasks.workunit.client.1.smithi188.stdout:6/409: chown d6 0 1 2024-04-05T21:26:31.096 INFO:tasks.workunit.client.1.smithi188.stdout:5/266: symlink d6/d15/d1b/d4f/l63 0 2024-04-05T21:26:31.098 INFO:tasks.workunit.client.1.smithi188.stdout:3/314: creat d5/f6b x:0 0 0 2024-04-05T21:26:31.098 INFO:tasks.workunit.client.1.smithi188.stdout:5/267: write d6/d17/d24/d44/f4d [979037,29773] 0 2024-04-05T21:26:31.103 INFO:tasks.workunit.client.1.smithi188.stdout:3/315: creat d5/df/f6c x:0 0 0 2024-04-05T21:26:31.105 INFO:tasks.workunit.client.1.smithi188.stdout:5/268: unlink d6/d15/l3a 0 2024-04-05T21:26:31.107 INFO:tasks.workunit.client.1.smithi188.stdout:3/316: rename d5/d7/d67 to d5/d7/d6d 0 2024-04-05T21:26:31.109 INFO:tasks.workunit.client.1.smithi188.stdout:6/410: truncate d6/d7/fc 7944180 0 2024-04-05T21:26:31.112 INFO:tasks.workunit.client.1.smithi188.stdout:9/327: dread dd/d49/d6b/f43 [0,4194304] 0 2024-04-05T21:26:31.113 INFO:tasks.workunit.client.1.smithi188.stdout:3/317: getdents d5/d35/d4e 0 2024-04-05T21:26:31.113 INFO:tasks.workunit.client.1.smithi188.stdout:6/411: write d6/d7/f14 [5560701,110879] 0 2024-04-05T21:26:31.114 INFO:tasks.workunit.client.1.smithi188.stdout:9/328: creat dd/d3d/f6c x:0 0 0 2024-04-05T21:26:31.115 INFO:tasks.workunit.client.1.smithi188.stdout:3/318: fsync d5/dc/f69 0 2024-04-05T21:26:31.116 INFO:tasks.workunit.client.1.smithi188.stdout:6/412: dread - d6/d13/d18/d3c/f3d zero size 2024-04-05T21:26:31.117 INFO:tasks.workunit.client.1.smithi188.stdout:6/413: readlink d6/d13/d18/d19/l2a 0 2024-04-05T21:26:31.119 INFO:tasks.workunit.client.1.smithi188.stdout:5/269: dread d6/d15/d1b/f34 [0,4194304] 0 2024-04-05T21:26:31.120 INFO:tasks.workunit.client.1.smithi188.stdout:5/270: readlink d6/d17/l1d 0 2024-04-05T21:26:31.123 INFO:tasks.workunit.client.1.smithi188.stdout:9/329: truncate dd/d11/d1f/d26/f2c 2149216 0 2024-04-05T21:26:31.125 INFO:tasks.workunit.client.1.smithi188.stdout:6/414: truncate d6/d13/d18/f36 1647905 0 2024-04-05T21:26:31.126 INFO:tasks.workunit.client.1.smithi188.stdout:9/330: read dd/d11/d22/f37 [151387,19623] 0 2024-04-05T21:26:31.127 INFO:tasks.workunit.client.1.smithi188.stdout:6/415: mknod d6/d13/d18/c81 0 2024-04-05T21:26:31.133 INFO:tasks.workunit.client.1.smithi188.stdout:9/331: dwrite dd/f5f [0,4194304] 0 2024-04-05T21:26:31.137 INFO:tasks.workunit.client.1.smithi188.stdout:9/332: symlink dd/d11/d1f/d26/l6d 0 2024-04-05T21:26:31.138 INFO:tasks.workunit.client.1.smithi188.stdout:6/416: truncate d6/d7/f14 1097925 0 2024-04-05T21:26:31.141 INFO:tasks.workunit.client.1.smithi188.stdout:9/333: creat dd/d11/d1f/f6e x:0 0 0 2024-04-05T21:26:31.142 INFO:tasks.workunit.client.1.smithi188.stdout:9/334: chown dd/d11/d1f/f6a 126056648 1 2024-04-05T21:26:31.144 INFO:tasks.workunit.client.1.smithi188.stdout:6/417: creat d6/d13/d18/f82 x:0 0 0 2024-04-05T21:26:31.148 INFO:tasks.workunit.client.1.smithi188.stdout:6/418: dread d6/d7/fd [0,4194304] 0 2024-04-05T21:26:31.149 INFO:tasks.workunit.client.1.smithi188.stdout:6/419: write d6/d13/d18/d19/d61/f62 [750875,92593] 0 2024-04-05T21:26:31.153 INFO:tasks.workunit.client.1.smithi188.stdout:6/420: mknod d6/d13/d48/d49/c83 0 2024-04-05T21:26:31.155 INFO:tasks.workunit.client.1.smithi188.stdout:6/421: unlink d6/d28/c70 0 2024-04-05T21:26:31.161 INFO:tasks.workunit.client.1.smithi188.stdout:6/422: rmdir d6/d28 39 2024-04-05T21:26:31.161 INFO:tasks.workunit.client.1.smithi188.stdout:9/335: dwrite dd/d11/d1f/f6a [0,4194304] 0 2024-04-05T21:26:31.163 INFO:tasks.workunit.client.1.smithi188.stdout:6/423: write d6/d13/d18/d3c/f3d [460016,10551] 0 2024-04-05T21:26:31.165 INFO:tasks.workunit.client.1.smithi188.stdout:9/336: creat dd/d11/d1f/d25/d51/f6f x:0 0 0 2024-04-05T21:26:31.166 INFO:tasks.workunit.client.1.smithi188.stdout:6/424: unlink d6/d13/d18/d19/d61/f62 0 2024-04-05T21:26:31.168 INFO:tasks.workunit.client.1.smithi188.stdout:9/337: creat dd/d3d/f70 x:0 0 0 2024-04-05T21:26:31.171 INFO:tasks.workunit.client.1.smithi188.stdout:9/338: mknod dd/d49/d6b/d40/d59/c71 0 2024-04-05T21:26:31.175 INFO:tasks.workunit.client.1.smithi188.stdout:9/339: rename dd/d11/d1f/d25/d51/f6f to dd/f72 0 2024-04-05T21:26:31.177 INFO:tasks.workunit.client.1.smithi188.stdout:9/340: mkdir dd/d11/d73 0 2024-04-05T21:26:31.177 INFO:tasks.workunit.client.1.smithi188.stdout:9/341: dread - dd/d3d/f61 zero size 2024-04-05T21:26:31.178 INFO:tasks.workunit.client.1.smithi188.stdout:9/342: chown dd/d49/d6b 23309745 1 2024-04-05T21:26:31.180 INFO:tasks.workunit.client.1.smithi188.stdout:9/343: mknod dd/d49/c74 0 2024-04-05T21:26:31.184 INFO:tasks.workunit.client.1.smithi188.stdout:6/425: dread d6/d7/f16 [0,4194304] 0 2024-04-05T21:26:31.186 INFO:tasks.workunit.client.1.smithi188.stdout:9/344: dwrite dd/d11/d22/f56 [0,4194304] 0 2024-04-05T21:26:31.188 INFO:tasks.workunit.client.1.smithi188.stdout:6/426: read d6/d13/d18/d19/d2c/f32 [1692431,84535] 0 2024-04-05T21:26:31.193 INFO:tasks.workunit.client.1.smithi188.stdout:6/427: truncate d6/d7/f14 1064699 0 2024-04-05T21:26:31.193 INFO:tasks.workunit.client.1.smithi188.stdout:9/345: truncate dd/d11/f21 5514254 0 2024-04-05T21:26:31.196 INFO:tasks.workunit.client.1.smithi188.stdout:9/346: write dd/f5f [439263,76578] 0 2024-04-05T21:26:31.226 INFO:tasks.workunit.client.1.smithi188.stdout:6/428: unlink d6/d28/c52 0 2024-04-05T21:26:31.242 INFO:tasks.workunit.client.1.smithi188.stdout:6/429: dread d6/fb [0,4194304] 0 2024-04-05T21:26:31.342 INFO:tasks.workunit.client.1.smithi188.stdout:2/350: sync 2024-04-05T21:26:31.342 INFO:tasks.workunit.client.1.smithi188.stdout:7/396: sync 2024-04-05T21:26:31.342 INFO:tasks.workunit.client.1.smithi188.stdout:1/324: sync 2024-04-05T21:26:31.346 INFO:tasks.workunit.client.1.smithi188.stdout:2/351: dread d5/db/d35/d3b/f5a [0,4194304] 0 2024-04-05T21:26:31.346 INFO:tasks.workunit.client.1.smithi188.stdout:6/430: creat d6/d28/f84 x:0 0 0 2024-04-05T21:26:31.347 INFO:tasks.workunit.client.1.smithi188.stdout:2/352: chown d5/c9 34674 1 2024-04-05T21:26:31.349 INFO:tasks.workunit.client.1.smithi188.stdout:2/353: write d5/db/d35/d3b/f4d [625011,114279] 0 2024-04-05T21:26:31.353 INFO:tasks.workunit.client.1.smithi188.stdout:6/431: mknod d6/d7/c85 0 2024-04-05T21:26:31.355 INFO:tasks.workunit.client.1.smithi188.stdout:6/432: dread - d6/d13/d18/d19/d2c/d3f/f53 zero size 2024-04-05T21:26:31.357 INFO:tasks.workunit.client.1.smithi188.stdout:2/354: dwrite d5/db/dd/d36/d3c/f4a [0,4194304] 0 2024-04-05T21:26:31.359 INFO:tasks.workunit.client.1.smithi188.stdout:6/433: chown d6/d13/d18/d19/d2c/c33 1289 1 2024-04-05T21:26:31.362 INFO:tasks.workunit.client.1.smithi188.stdout:7/397: rename d3/f6f to d3/d4/db/d10/f82 0 2024-04-05T21:26:31.363 INFO:tasks.workunit.client.1.smithi188.stdout:7/398: readlink d3/d4/db/d2e/d37/l4b 0 2024-04-05T21:26:31.364 INFO:tasks.workunit.client.1.smithi188.stdout:6/434: symlink d6/d13/d18/d19/d2c/d3f/l86 0 2024-04-05T21:26:31.368 INFO:tasks.workunit.client.1.smithi188.stdout:7/399: creat d3/d4/d1d/d74/d73/f83 x:0 0 0 2024-04-05T21:26:31.370 INFO:tasks.workunit.client.1.smithi188.stdout:7/400: write d3/d4/db/f6e [4888927,88750] 0 2024-04-05T21:26:31.372 INFO:tasks.workunit.client.1.smithi188.stdout:7/401: write d3/d4/db/d2e/f6b [846472,64800] 0 2024-04-05T21:26:31.374 INFO:tasks.workunit.client.1.smithi188.stdout:7/402: write d3/d4/db/d2e/f6b [1083698,5192] 0 2024-04-05T21:26:31.374 INFO:tasks.workunit.client.1.smithi188.stdout:7/403: chown d3/d2c 505208433 1 2024-04-05T21:26:31.395 INFO:tasks.workunit.client.1.smithi188.stdout:7/404: creat d3/d4/db/d2e/f84 x:0 0 0 2024-04-05T21:26:31.396 INFO:tasks.workunit.client.1.smithi188.stdout:1/325: rename d1/dc/d16/f47 to d1/d7/d4e/f6d 0 2024-04-05T21:26:31.397 INFO:tasks.workunit.client.1.smithi188.stdout:7/405: write d3/d2c/f7e [868219,36806] 0 2024-04-05T21:26:31.399 INFO:tasks.workunit.client.1.smithi188.stdout:6/435: dwrite d6/d13/d38/f6f [0,4194304] 0 2024-04-05T21:26:31.402 INFO:tasks.workunit.client.1.smithi188.stdout:6/436: stat d6/d13/d48/d49/f80 0 2024-04-05T21:26:31.402 INFO:tasks.workunit.client.1.smithi188.stdout:2/355: rename d5/db/dd/d36/d3d/l47 to d5/db/dd/d36/d3c/d48/l6d 0 2024-04-05T21:26:31.406 INFO:tasks.workunit.client.1.smithi188.stdout:6/437: creat d6/d13/d18/d19/d61/f87 x:0 0 0 2024-04-05T21:26:31.408 INFO:tasks.workunit.client.1.smithi188.stdout:2/356: dread d5/db/dd/f15 [4194304,4194304] 0 2024-04-05T21:26:31.409 INFO:tasks.workunit.client.1.smithi188.stdout:6/438: rename d6/d13/d18/d19/l42 to d6/d13/d18/d19/l88 0 2024-04-05T21:26:31.410 INFO:tasks.workunit.client.1.smithi188.stdout:2/357: getdents d5/db/dd/d67 0 2024-04-05T21:26:31.411 INFO:tasks.workunit.client.1.smithi188.stdout:6/439: write d6/fa [1879798,4412] 0 2024-04-05T21:26:31.418 INFO:tasks.workunit.client.1.smithi188.stdout:7/406: dread d3/d4/db/d2e/d37/d65/f1f [0,4194304] 0 2024-04-05T21:26:31.419 INFO:tasks.workunit.client.1.smithi188.stdout:6/440: dwrite d6/d13/d18/d19/d2c/d3f/f56 [0,4194304] 0 2024-04-05T21:26:31.422 INFO:tasks.workunit.client.1.smithi188.stdout:7/407: symlink d3/l85 0 2024-04-05T21:26:31.423 INFO:tasks.workunit.client.1.smithi188.stdout:1/326: dread d1/dc/d16/d21/f68 [0,4194304] 0 2024-04-05T21:26:31.424 INFO:tasks.workunit.client.1.smithi188.stdout:6/441: creat d6/d13/f89 x:0 0 0 2024-04-05T21:26:31.425 INFO:tasks.workunit.client.1.smithi188.stdout:7/408: creat d3/d4/d1d/d74/f86 x:0 0 0 2024-04-05T21:26:31.425 INFO:tasks.workunit.client.1.smithi188.stdout:6/442: chown d6/d13/d38/c78 371 1 2024-04-05T21:26:31.428 INFO:tasks.workunit.client.1.smithi188.stdout:1/327: link d1/dc/f28 d1/dc/d16/d21/f6e 0 2024-04-05T21:26:31.429 INFO:tasks.workunit.client.1.smithi188.stdout:6/443: symlink d6/d13/d18/d19/d2c/d3f/l8a 0 2024-04-05T21:26:31.429 INFO:tasks.workunit.client.1.smithi188.stdout:7/409: mknod d3/d4/db/d2e/c87 0 2024-04-05T21:26:31.433 INFO:tasks.workunit.client.1.smithi188.stdout:7/410: mkdir d3/d4/d1d/d74/d88 0 2024-04-05T21:26:31.435 INFO:tasks.workunit.client.1.smithi188.stdout:7/411: mkdir d3/d89 0 2024-04-05T21:26:31.436 INFO:tasks.workunit.client.1.smithi188.stdout:1/328: dwrite d1/dc/d16/f3b [0,4194304] 0 2024-04-05T21:26:31.446 INFO:tasks.workunit.client.1.smithi188.stdout:1/329: dwrite d1/dc/f1f [4194304,4194304] 0 2024-04-05T21:26:31.452 INFO:tasks.workunit.client.1.smithi188.stdout:1/330: creat d1/dc/d16/f6f x:0 0 0 2024-04-05T21:26:31.452 INFO:tasks.workunit.client.1.smithi188.stdout:7/412: truncate d3/d4/f25 917919 0 2024-04-05T21:26:31.457 INFO:tasks.workunit.client.1.smithi188.stdout:1/331: symlink d1/dc/d16/d21/d27/l70 0 2024-04-05T21:26:31.458 INFO:tasks.workunit.client.1.smithi188.stdout:7/413: creat d3/d4/db/d2e/d49/d79/f8a x:0 0 0 2024-04-05T21:26:31.459 INFO:tasks.workunit.client.1.smithi188.stdout:7/414: chown d3/c28 2 1 2024-04-05T21:26:31.461 INFO:tasks.workunit.client.1.smithi188.stdout:1/332: rename d1/dc/d16/d46/c6b to d1/d4/d3f/c71 0 2024-04-05T21:26:31.462 INFO:tasks.workunit.client.1.smithi188.stdout:7/415: mknod d3/d89/c8b 0 2024-04-05T21:26:31.464 INFO:tasks.workunit.client.1.smithi188.stdout:7/416: mkdir d3/d4/db/d2e/d8c 0 2024-04-05T21:26:31.466 INFO:tasks.workunit.client.1.smithi188.stdout:7/417: truncate d3/d4/db/d2e/d37/d65/f7d 739233 0 2024-04-05T21:26:31.469 INFO:tasks.workunit.client.1.smithi188.stdout:7/418: link c1 d3/d4/d7c/c8d 0 2024-04-05T21:26:31.471 INFO:tasks.workunit.client.1.smithi188.stdout:7/419: mkdir d3/d2c/d5e/d8e 0 2024-04-05T21:26:31.473 INFO:tasks.workunit.client.1.smithi188.stdout:1/333: dread d1/d7/d4e/f6d [0,4194304] 0 2024-04-05T21:26:31.474 INFO:tasks.workunit.client.1.smithi188.stdout:7/420: rename d3/l61 to d3/d2c/d5e/l8f 0 2024-04-05T21:26:31.475 INFO:tasks.workunit.client.1.smithi188.stdout:1/334: rmdir d1/d7 39 2024-04-05T21:26:31.479 INFO:tasks.workunit.client.1.smithi188.stdout:1/335: symlink d1/dc/d16/d21/d27/d49/l72 0 2024-04-05T21:26:31.480 INFO:tasks.workunit.client.1.smithi188.stdout:1/336: dread - d1/dc/d16/d21/f6e zero size 2024-04-05T21:26:31.486 INFO:tasks.workunit.client.1.smithi188.stdout:7/421: truncate d3/f1e 2557236 0 2024-04-05T21:26:31.487 INFO:tasks.workunit.client.1.smithi188.stdout:7/422: chown d3/d4/db/d2e/d49/d79/f8a 11 1 2024-04-05T21:26:31.488 INFO:tasks.workunit.client.1.smithi188.stdout:7/423: write d3/d4/db/f6e [4141008,72576] 0 2024-04-05T21:26:31.489 INFO:tasks.workunit.client.1.smithi188.stdout:7/424: fdatasync d3/d4/db/f38 0 2024-04-05T21:26:31.491 INFO:tasks.workunit.client.1.smithi188.stdout:7/425: symlink d3/d4/db/d2e/d37/l90 0 2024-04-05T21:26:31.493 INFO:tasks.workunit.client.1.smithi188.stdout:7/426: rename d3/d2c/d63 to d3/d2c/d5e/d91 0 2024-04-05T21:26:31.495 INFO:tasks.workunit.client.1.smithi188.stdout:7/427: creat d3/d4/db/d2e/d3b/f92 x:0 0 0 2024-04-05T21:26:31.501 INFO:tasks.workunit.client.1.smithi188.stdout:7/428: rmdir d3 39 2024-04-05T21:26:31.508 INFO:tasks.workunit.client.1.smithi188.stdout:7/429: dwrite d3/f71 [0,4194304] 0 2024-04-05T21:26:31.510 INFO:tasks.workunit.client.1.smithi188.stdout:7/430: truncate d3/d4/db/d10/f33 1836148 0 2024-04-05T21:26:31.512 INFO:tasks.workunit.client.1.smithi188.stdout:7/431: readlink d3/d2c/d5e/l8f 0 2024-04-05T21:26:31.512 INFO:tasks.workunit.client.1.smithi188.stdout:7/432: chown d3/d4/db/d2e/d3b/f4d 115047 1 2024-04-05T21:26:31.516 INFO:tasks.workunit.client.1.smithi188.stdout:7/433: rename d3/d4/db/d2e/f84 to d3/d4/db/d2e/f93 0 2024-04-05T21:26:31.517 INFO:tasks.workunit.client.1.smithi188.stdout:7/434: write d3/d4/d1d/f34 [887394,123119] 0 2024-04-05T21:26:31.518 INFO:tasks.workunit.client.1.smithi188.stdout:7/435: chown d3/d4/db/d2e/d37/l90 1379535 1 2024-04-05T21:26:31.531 INFO:tasks.workunit.client.1.smithi188.stdout:7/436: dread d3/f1e [0,4194304] 0 2024-04-05T21:26:31.531 INFO:tasks.workunit.client.1.smithi188.stdout:7/437: write d3/d2c/f72 [596585,54219] 0 2024-04-05T21:26:31.538 INFO:tasks.workunit.client.1.smithi188.stdout:7/438: dwrite d3/d4/d1d/f34 [0,4194304] 0 2024-04-05T21:26:31.539 INFO:tasks.workunit.client.1.smithi188.stdout:7/439: chown d3/l26 1 1 2024-04-05T21:26:31.540 INFO:tasks.workunit.client.1.smithi188.stdout:7/440: mkdir d3/d4/db/d2e/d3b/d94 0 2024-04-05T21:26:31.576 INFO:tasks.workunit.client.1.smithi188.stdout:7/441: chown d3/d4/db/d2e/f93 110079 1 2024-04-05T21:26:31.579 INFO:tasks.workunit.client.1.smithi188.stdout:7/442: creat d3/d4/db/d10/f95 x:0 0 0 2024-04-05T21:26:31.580 INFO:tasks.workunit.client.1.smithi188.stdout:7/443: chown d3/d4/db/d2e/d37/d65 29 1 2024-04-05T21:26:31.581 INFO:tasks.workunit.client.1.smithi188.stdout:7/444: creat d3/d2c/f96 x:0 0 0 2024-04-05T21:26:31.583 INFO:tasks.workunit.client.1.smithi188.stdout:7/445: creat d3/d4/db/d2e/d49/d79/f97 x:0 0 0 2024-04-05T21:26:31.584 INFO:tasks.workunit.client.1.smithi188.stdout:7/446: dread - d3/d4/d1d/d74/f80 zero size 2024-04-05T21:26:31.585 INFO:tasks.workunit.client.1.smithi188.stdout:7/447: write d3/d2c/f96 [65808,101946] 0 2024-04-05T21:26:31.585 INFO:tasks.workunit.client.1.smithi188.stdout:7/448: stat d3/d2c/d5e/f66 0 2024-04-05T21:26:31.592 INFO:tasks.workunit.client.1.smithi188.stdout:7/449: rename d3/d4/db/d2e/d49/d79/f8a to d3/d2c/d5e/f98 0 2024-04-05T21:26:31.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.650+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3552675455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d781016e0 msgr2=0x7f6d78103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:31.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.650+0000 7f6d7ee12700 1 --2- 172.21.15.121:0/3552675455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d781016e0 0x7f6d78103ad0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c009910 tx=0x7f6d6c009c20 comp rx=0 tx=0).stop 2024-04-05T21:26:31.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.650+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3552675455 shutdown_connections 2024-04-05T21:26:31.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.650+0000 7f6d7ee12700 1 --2- 172.21.15.121:0/3552675455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d78104010 0x7f6d78106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:31.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.650+0000 7f6d7ee12700 1 --2- 172.21.15.121:0/3552675455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d781016e0 0x7f6d78103ad0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:31.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.650+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3552675455 >> 172.21.15.121:0/3552675455 conn(0x7f6d780fb050 msgr2=0x7f6d780fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:31.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.650+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3552675455 shutdown_connections 2024-04-05T21:26:31.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.650+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3552675455 wait complete. 2024-04-05T21:26:31.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7ee12700 1 Processor -- start 2024-04-05T21:26:31.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7ee12700 1 -- start start 2024-04-05T21:26:31.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7ee12700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d781016e0 0x7f6d78196260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:31.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7ee12700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d78104010 0x7f6d781967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:31.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7ee12700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d78196dc0 con 0x7f6d781016e0 2024-04-05T21:26:31.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7ee12700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d78196f00 con 0x7f6d78104010 2024-04-05T21:26:31.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7cbae700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d781016e0 0x7f6d78196260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:31.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7cbae700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d781016e0 0x7f6d78196260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37426/0 (socket says 172.21.15.121:37426) 2024-04-05T21:26:31.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7cbae700 1 -- 172.21.15.121:0/3211736680 learned_addr learned my addr 172.21.15.121:0/3211736680 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:26:31.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d77fff700 1 --2- 172.21.15.121:0/3211736680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d78104010 0x7f6d781967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:31.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7cbae700 1 -- 172.21.15.121:0/3211736680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d78104010 msgr2=0x7f6d781967a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:31.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7cbae700 1 --2- 172.21.15.121:0/3211736680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d78104010 0x7f6d781967a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:31.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7cbae700 1 -- 172.21.15.121:0/3211736680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c0095f0 con 0x7f6d781016e0 2024-04-05T21:26:31.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7cbae700 1 --2- 172.21.15.121:0/3211736680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d781016e0 0x7f6d78196260 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c00bde0 tx=0x7f6d6c00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:31.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d75ffb700 1 -- 172.21.15.121:0/3211736680 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d6c01d070 con 0x7f6d781016e0 2024-04-05T21:26:31.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d75ffb700 1 -- 172.21.15.121:0/3211736680 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f6d6c005dc0 con 0x7f6d781016e0 2024-04-05T21:26:31.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3211736680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7819b950 con 0x7f6d781016e0 2024-04-05T21:26:31.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.651+0000 7f6d75ffb700 1 -- 172.21.15.121:0/3211736680 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d6c003b10 con 0x7f6d781016e0 2024-04-05T21:26:31.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.652+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3211736680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d7819be60 con 0x7f6d781016e0 2024-04-05T21:26:31.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.653+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3211736680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d781903f0 con 0x7f6d781016e0 2024-04-05T21:26:31.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.653+0000 7f6d75ffb700 1 -- 172.21.15.121:0/3211736680 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 25) v1 ==== 90681+0+0 (secure 0 0 0) 0x7f6d6c0048e0 con 0x7f6d781016e0 2024-04-05T21:26:31.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.653+0000 7f6d75ffb700 1 --2- 172.21.15.121:0/3211736680 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f6d6806ca20 0x7f6d6806eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:31.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.653+0000 7f6d75ffb700 1 -- 172.21.15.121:0/3211736680 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f6d6c08d880 con 0x7f6d781016e0 2024-04-05T21:26:31.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.654+0000 7f6d77fff700 1 --2- 172.21.15.121:0/3211736680 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f6d6806ca20 0x7f6d6806eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:31.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.654+0000 7f6d77fff700 1 --2- 172.21.15.121:0/3211736680 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f6d6806ca20 0x7f6d6806eee0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6d64009c60 tx=0x7f6d64009260 comp rx=0 tx=0).ready entity=mgr.24327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:31.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.658+0000 7f6d75ffb700 1 -- 172.21.15.121:0/3211736680 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f6d6c05b360 con 0x7f6d781016e0 2024-04-05T21:26:31.700 INFO:tasks.workunit.client.1.smithi188.stdout:8/227: sync 2024-04-05T21:26:31.700 INFO:tasks.workunit.client.1.smithi188.stdout:0/200: sync 2024-04-05T21:26:31.700 INFO:tasks.workunit.client.1.smithi188.stdout:4/476: sync 2024-04-05T21:26:31.702 INFO:tasks.workunit.client.1.smithi188.stdout:8/228: dread d5/dc/f1d [0,4194304] 0 2024-04-05T21:26:31.703 INFO:tasks.workunit.client.1.smithi188.stdout:0/201: creat d4/d8/d39/f48 x:0 0 0 2024-04-05T21:26:31.706 INFO:tasks.workunit.client.1.smithi188.stdout:8/229: mkdir d5/d21/d50 0 2024-04-05T21:26:31.706 INFO:tasks.workunit.client.1.smithi188.stdout:8/230: readlink d5/l3e 0 2024-04-05T21:26:31.707 INFO:tasks.workunit.client.1.smithi188.stdout:0/202: creat d4/d8/db/d42/d38/f49 x:0 0 0 2024-04-05T21:26:31.709 INFO:tasks.workunit.client.1.smithi188.stdout:0/203: write d4/d8/db/d42/d38/f49 [640310,58534] 0 2024-04-05T21:26:31.709 INFO:tasks.workunit.client.1.smithi188.stdout:8/231: creat d5/d21/d40/d43/f51 x:0 0 0 2024-04-05T21:26:31.713 INFO:tasks.workunit.client.1.smithi188.stdout:8/232: rename d5/c28 to d5/d21/c52 0 2024-04-05T21:26:31.716 INFO:tasks.workunit.client.1.smithi188.stdout:4/477: truncate d0/f64 2175116 0 2024-04-05T21:26:31.717 INFO:tasks.workunit.client.1.smithi188.stdout:0/204: dwrite d4/d8/d10/f20 [0,4194304] 0 2024-04-05T21:26:31.719 INFO:tasks.workunit.client.1.smithi188.stdout:0/205: write d4/d8/db/f11 [4684180,20036] 0 2024-04-05T21:26:31.720 INFO:tasks.workunit.client.1.smithi188.stdout:4/478: unlink d0/d14/d15/f90 0 2024-04-05T21:26:31.725 INFO:tasks.workunit.client.1.smithi188.stdout:0/206: mkdir d4/d8/db/d21/d4a 0 2024-04-05T21:26:31.728 INFO:tasks.workunit.client.1.smithi188.stdout:8/233: write d5/f15 [1076993,128823] 0 2024-04-05T21:26:31.729 INFO:tasks.workunit.client.1.smithi188.stdout:4/479: creat d0/d2/d17/d48/fb3 x:0 0 0 2024-04-05T21:26:31.729 INFO:tasks.workunit.client.1.smithi188.stdout:4/480: fsync d0/d2/d17/fa7 0 2024-04-05T21:26:31.730 INFO:tasks.workunit.client.1.smithi188.stdout:0/207: mknod d4/d6/d13/c4b 0 2024-04-05T21:26:31.732 INFO:tasks.workunit.client.1.smithi188.stdout:0/208: chown d4/d8/d10/l1a 1 1 2024-04-05T21:26:31.733 INFO:tasks.workunit.client.1.smithi188.stdout:8/234: symlink d5/d21/l53 0 2024-04-05T21:26:31.734 INFO:tasks.workunit.client.1.smithi188.stdout:8/235: chown d5/f8 13730 1 2024-04-05T21:26:31.735 INFO:tasks.workunit.client.1.smithi188.stdout:0/209: creat d4/d8/db/d42/f4c x:0 0 0 2024-04-05T21:26:31.737 INFO:tasks.workunit.client.1.smithi188.stdout:4/481: dwrite d0/d2/d17/fa7 [0,4194304] 0 2024-04-05T21:26:31.738 INFO:tasks.workunit.client.1.smithi188.stdout:8/236: creat d5/d21/d2a/d47/f54 x:0 0 0 2024-04-05T21:26:31.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:31 smithi121 ceph-mon[39723]: pgmap v14: 65 pgs: 65 active+clean; 834 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 8.6 MiB/s rd, 91 MiB/s wr, 235 op/s 2024-04-05T21:26:31.742 INFO:tasks.workunit.client.1.smithi188.stdout:0/210: rename d4/d6/d13/d23/d24/f3a to d4/d8/f4d 0 2024-04-05T21:26:31.744 INFO:tasks.workunit.client.1.smithi188.stdout:8/237: creat d5/d21/d40/f55 x:0 0 0 2024-04-05T21:26:31.748 INFO:tasks.workunit.client.1.smithi188.stdout:0/211: dwrite d4/fe [0,4194304] 0 2024-04-05T21:26:31.749 INFO:tasks.workunit.client.1.smithi188.stdout:0/212: chown d4/d6/d13 1012368 1 2024-04-05T21:26:31.750 INFO:tasks.workunit.client.1.smithi188.stdout:0/213: symlink d4/d8/db/d21/l4e 0 2024-04-05T21:26:31.751 INFO:tasks.workunit.client.1.smithi188.stdout:0/214: chown d4/d8/f4d 26486395 1 2024-04-05T21:26:31.753 INFO:tasks.workunit.client.1.smithi188.stdout:0/215: symlink d4/d6/l4f 0 2024-04-05T21:26:31.761 INFO:tasks.workunit.client.1.smithi188.stdout:4/482: dread d0/d2/d4f/f67 [0,4194304] 0 2024-04-05T21:26:31.768 INFO:tasks.workunit.client.1.smithi188.stdout:4/483: dwrite d0/d81/f85 [4194304,4194304] 0 2024-04-05T21:26:31.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:31 smithi188 ceph-mon[49715]: pgmap v14: 65 pgs: 65 active+clean; 834 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 8.6 MiB/s rd, 91 MiB/s wr, 235 op/s 2024-04-05T21:26:31.785 INFO:tasks.workunit.client.1.smithi188.stdout:4/484: dread d0/f64 [0,4194304] 0 2024-04-05T21:26:31.785 INFO:tasks.workunit.client.1.smithi188.stdout:4/485: stat d0/d14/l28 0 2024-04-05T21:26:31.787 INFO:tasks.workunit.client.1.smithi188.stdout:4/486: unlink d0/d2/d2a/d34/f77 0 2024-04-05T21:26:31.789 INFO:tasks.workunit.client.1.smithi188.stdout:4/487: unlink d0/d14/d15/l19 0 2024-04-05T21:26:31.790 INFO:tasks.workunit.client.1.smithi188.stdout:4/488: symlink d0/d2/d2a/d34/d6b/lb4 0 2024-04-05T21:26:31.791 INFO:tasks.workunit.client.1.smithi188.stdout:4/489: write d0/d14/f86 [4341136,51960] 0 2024-04-05T21:26:31.793 INFO:tasks.workunit.client.1.smithi188.stdout:4/490: truncate d0/d14/d15/d78/fae 42411 0 2024-04-05T21:26:31.795 INFO:tasks.workunit.client.1.smithi188.stdout:4/491: creat d0/d14/d15/d78/db1/fb5 x:0 0 0 2024-04-05T21:26:31.796 INFO:tasks.workunit.client.1.smithi188.stdout:4/492: stat d0/d14/d15/d53/d82/d8f/d95 0 2024-04-05T21:26:31.798 INFO:tasks.workunit.client.1.smithi188.stdout:4/493: creat d0/d2/d2a/d71/fb6 x:0 0 0 2024-04-05T21:26:31.799 INFO:tasks.workunit.client.1.smithi188.stdout:4/494: truncate d0/d2/d2a/d34/fa1 917411 0 2024-04-05T21:26:31.801 INFO:tasks.workunit.client.1.smithi188.stdout:4/495: symlink d0/d14/d15/d53/d82/d8f/d95/lb7 0 2024-04-05T21:26:31.802 INFO:tasks.workunit.client.1.smithi188.stdout:4/496: dread d0/d14/d15/d1b/f4d [0,4194304] 0 2024-04-05T21:26:31.806 INFO:tasks.workunit.client.1.smithi188.stdout:4/497: write d0/d2/d4f/f5c [703626,84159] 0 2024-04-05T21:26:31.813 INFO:tasks.workunit.client.1.smithi188.stdout:4/498: dwrite d0/d2/f26 [0,4194304] 0 2024-04-05T21:26:31.815 INFO:tasks.workunit.client.1.smithi188.stdout:4/499: getdents d0/d14/d15/d78/db1/daa 0 2024-04-05T21:26:31.817 INFO:tasks.workunit.client.1.smithi188.stdout:4/500: rmdir d0/d2/d17/d3a 39 2024-04-05T21:26:31.823 INFO:tasks.workunit.client.1.smithi188.stdout:4/501: dwrite d0/d14/d15/d42/f63 [0,4194304] 0 2024-04-05T21:26:31.831 INFO:tasks.workunit.client.1.smithi188.stdout:4/502: dwrite d0/d14/d15/d53/d82/d8f/fad [0,4194304] 0 2024-04-05T21:26:31.833 INFO:tasks.workunit.client.1.smithi188.stdout:4/503: fsync d0/d14/d15/d53/d82/d8f/d95/fa4 0 2024-04-05T21:26:31.846 INFO:tasks.workunit.client.1.smithi188.stdout:4/504: dread d0/d14/d15/d53/f74 [0,4194304] 0 2024-04-05T21:26:31.850 INFO:tasks.workunit.client.1.smithi188.stdout:6/444: sync 2024-04-05T21:26:31.850 INFO:tasks.workunit.client.1.smithi188.stdout:2/358: sync 2024-04-05T21:26:31.850 INFO:tasks.workunit.client.1.smithi188.stdout:7/450: sync 2024-04-05T21:26:31.850 INFO:tasks.workunit.client.1.smithi188.stdout:5/271: sync 2024-04-05T21:26:31.850 INFO:tasks.workunit.client.1.smithi188.stdout:1/337: sync 2024-04-05T21:26:31.850 INFO:tasks.workunit.client.1.smithi188.stdout:3/319: sync 2024-04-05T21:26:31.850 INFO:tasks.workunit.client.1.smithi188.stdout:9/347: sync 2024-04-05T21:26:31.851 INFO:tasks.workunit.client.1.smithi188.stdout:5/272: fdatasync d6/f53 0 2024-04-05T21:26:31.851 INFO:tasks.workunit.client.1.smithi188.stdout:9/348: fdatasync dd/d3d/f61 0 2024-04-05T21:26:31.852 INFO:tasks.workunit.client.1.smithi188.stdout:2/359: dread - d5/db/dd/f54 zero size 2024-04-05T21:26:31.855 INFO:tasks.workunit.client.1.smithi188.stdout:2/360: stat d5/db/dd/c38 0 2024-04-05T21:26:31.855 INFO:tasks.workunit.client.1.smithi188.stdout:1/338: creat d1/dc/d16/d21/d27/d49/f73 x:0 0 0 2024-04-05T21:26:31.856 INFO:tasks.workunit.client.1.smithi188.stdout:5/273: write d6/d15/d1b/f2a [3479215,71461] 0 2024-04-05T21:26:31.858 INFO:tasks.workunit.client.1.smithi188.stdout:6/445: truncate d6/d7/fc 4647925 0 2024-04-05T21:26:31.865 INFO:tasks.workunit.client.1.smithi188.stdout:5/274: symlink d6/d15/d5c/l64 0 2024-04-05T21:26:31.866 INFO:tasks.workunit.client.1.smithi188.stdout:3/320: dwrite d5/dc/f69 [0,4194304] 0 2024-04-05T21:26:31.867 INFO:tasks.workunit.client.1.smithi188.stdout:2/361: chown d5/db/dd/f54 0 1 2024-04-05T21:26:31.868 INFO:tasks.workunit.client.1.smithi188.stdout:7/451: getdents d3/d4/db/d2e/d37/d65 0 2024-04-05T21:26:31.869 INFO:tasks.workunit.client.1.smithi188.stdout:4/505: dread d0/d2/d17/d48/f57 [0,4194304] 0 2024-04-05T21:26:31.869 INFO:tasks.workunit.client.1.smithi188.stdout:6/446: creat d6/d13/d18/d19/d61/f8b x:0 0 0 2024-04-05T21:26:31.870 INFO:tasks.workunit.client.1.smithi188.stdout:5/275: readlink d6/d15/l33 0 2024-04-05T21:26:31.872 INFO:tasks.workunit.client.1.smithi188.stdout:2/362: dread - d5/db/dd/d36/d3c/d48/f56 zero size 2024-04-05T21:26:31.873 INFO:tasks.workunit.client.1.smithi188.stdout:9/349: dwrite dd/f72 [0,4194304] 0 2024-04-05T21:26:31.878 INFO:tasks.workunit.client.1.smithi188.stdout:6/447: fdatasync d6/f74 0 2024-04-05T21:26:31.880 INFO:tasks.workunit.client.1.smithi188.stdout:1/339: truncate d1/d4/d3f/f44 399014 0 2024-04-05T21:26:31.880 INFO:tasks.workunit.client.1.smithi188.stdout:6/448: chown d6/d28 827271 1 2024-04-05T21:26:31.880 INFO:tasks.workunit.client.1.smithi188.stdout:7/452: write d3/f1e [3589997,93174] 0 2024-04-05T21:26:31.881 INFO:tasks.workunit.client.1.smithi188.stdout:3/321: mkdir d5/d7/d65/d61/d66/d6e 0 2024-04-05T21:26:31.883 INFO:tasks.workunit.client.1.smithi188.stdout:9/350: read dd/d11/d1f/d25/f50 [1699497,115831] 0 2024-04-05T21:26:31.885 INFO:tasks.workunit.client.1.smithi188.stdout:5/276: dread d6/d17/d24/f3c [0,4194304] 0 2024-04-05T21:26:31.885 INFO:tasks.workunit.client.1.smithi188.stdout:3/322: write d5/fa [5100753,94238] 0 2024-04-05T21:26:31.886 INFO:tasks.workunit.client.1.smithi188.stdout:9/351: chown dd/d11/d1f/f29 686 1 2024-04-05T21:26:31.887 INFO:tasks.workunit.client.1.smithi188.stdout:2/363: rename d5/db/dd/d36/d3d/d46/l61 to d5/l6e 0 2024-04-05T21:26:31.888 INFO:tasks.workunit.client.1.smithi188.stdout:3/323: chown d5/d7/d65/d5f 30678235 1 2024-04-05T21:26:31.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.891+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3211736680 --> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d780611d0 con 0x7f6d6806ca20 2024-04-05T21:26:31.892 INFO:tasks.workunit.client.1.smithi188.stdout:5/277: write d6/d15/d1b/f34 [2139269,103058] 0 2024-04-05T21:26:31.892 INFO:tasks.workunit.client.1.smithi188.stdout:3/324: rmdir d5/dc/d2f 39 2024-04-05T21:26:31.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.893+0000 7f6d75ffb700 1 -- 172.21.15.121:0/3211736680 <== mgr.24327 v2:172.21.15.188:6828/2212394428 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f6d780611d0 con 0x7f6d6806ca20 2024-04-05T21:26:31.894 INFO:tasks.workunit.client.1.smithi188.stdout:9/352: creat dd/d11/d73/f75 x:0 0 0 2024-04-05T21:26:31.894 INFO:tasks.workunit.client.1.smithi188.stdout:6/449: rename d6/d28/f31 to d6/d13/d18/f8c 0 2024-04-05T21:26:31.896 INFO:tasks.workunit.client.1.smithi188.stdout:1/340: dwrite d1/dc/d16/f33 [0,4194304] 0 2024-04-05T21:26:31.897 INFO:tasks.workunit.client.1.smithi188.stdout:2/364: creat d5/db/d35/d3b/f6f x:0 0 0 2024-04-05T21:26:31.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.895+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3211736680 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f6d6806ca20 msgr2=0x7f6d6806eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:31.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.895+0000 7f6d7ee12700 1 --2- 172.21.15.121:0/3211736680 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f6d6806ca20 0x7f6d6806eee0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6d64009c60 tx=0x7f6d64009260 comp rx=0 tx=0).stop 2024-04-05T21:26:31.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.895+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3211736680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d781016e0 msgr2=0x7f6d78196260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:31.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.895+0000 7f6d7ee12700 1 --2- 172.21.15.121:0/3211736680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d781016e0 0x7f6d78196260 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c00bde0 tx=0x7f6d6c00be10 comp rx=0 tx=0).stop 2024-04-05T21:26:31.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.895+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3211736680 shutdown_connections 2024-04-05T21:26:31.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.895+0000 7f6d7ee12700 1 --2- 172.21.15.121:0/3211736680 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f6d6806ca20 0x7f6d6806eee0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:31.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.895+0000 7f6d7ee12700 1 --2- 172.21.15.121:0/3211736680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d781016e0 0x7f6d78196260 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:31.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.895+0000 7f6d7ee12700 1 --2- 172.21.15.121:0/3211736680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6d78104010 0x7f6d781967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:31.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.895+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3211736680 >> 172.21.15.121:0/3211736680 conn(0x7f6d780fb050 msgr2=0x7f6d780fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:31.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.896+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3211736680 shutdown_connections 2024-04-05T21:26:31.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:31.896+0000 7f6d7ee12700 1 -- 172.21.15.121:0/3211736680 wait complete. 2024-04-05T21:26:31.898 INFO:tasks.workunit.client.1.smithi188.stdout:9/353: write dd/d35/f67 [956014,75266] 0 2024-04-05T21:26:31.899 INFO:tasks.workunit.client.1.smithi188.stdout:3/325: rename d5/dc/f69 to d5/d7/d31/f6f 0 2024-04-05T21:26:31.899 INFO:tasks.workunit.client.1.smithi188.stdout:6/450: write d6/d13/d38/f6f [34423,22573] 0 2024-04-05T21:26:31.904 INFO:tasks.workunit.client.1.smithi188.stdout:5/278: link d6/d17/d24/f3c d6/d17/d23/d35/d40/f65 0 2024-04-05T21:26:31.906 INFO:tasks.workunit.client.1.smithi188.stdout:9/354: chown dd/d11/d1f/d26/l10 1077 1 2024-04-05T21:26:31.906 INFO:tasks.workunit.client.1.smithi188.stdout:5/279: chown d6/f9 56047431 1 2024-04-05T21:26:31.907 INFO:tasks.workunit.client.1.smithi188.stdout:6/451: mkdir d6/d13/d18/d19/d2c/d3f/d8d 0 2024-04-05T21:26:31.910 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:26:31.912 INFO:tasks.workunit.client.1.smithi188.stdout:5/280: mknod d6/d17/c66 0 2024-04-05T21:26:31.915 INFO:tasks.workunit.client.1.smithi188.stdout:6/452: fsync d6/d7/f15 0 2024-04-05T21:26:31.917 INFO:tasks.workunit.client.1.smithi188.stdout:2/365: getdents d5/db/dd/d36/d3c/d48 0 2024-04-05T21:26:31.917 INFO:tasks.workunit.client.1.smithi188.stdout:6/453: read - d6/d13/d48/d49/f80 zero size 2024-04-05T21:26:31.918 INFO:tasks.workunit.client.1.smithi188.stdout:7/453: dread d3/d4/db/f6e [0,4194304] 0 2024-04-05T21:26:31.921 INFO:tasks.workunit.client.1.smithi188.stdout:5/281: creat d6/d17/d23/d35/d40/d42/f67 x:0 0 0 2024-04-05T21:26:31.924 INFO:tasks.workunit.client.1.smithi188.stdout:7/454: symlink d3/d4/db/d2e/d49/l99 0 2024-04-05T21:26:31.930 INFO:tasks.workunit.client.1.smithi188.stdout:5/282: rename d6/d15/d1b/d4f/l63 to d6/d17/d24/d44/d51/l68 0 2024-04-05T21:26:31.931 INFO:tasks.workunit.client.1.smithi188.stdout:7/455: mknod d3/d4/d1d/c9a 0 2024-04-05T21:26:31.931 INFO:tasks.workunit.client.1.smithi188.stdout:9/355: write dd/d11/d22/f4d [30095,21094] 0 2024-04-05T21:26:31.932 INFO:tasks.workunit.client.1.smithi188.stdout:7/456: dread - d3/d2c/d5e/f98 zero size 2024-04-05T21:26:31.933 INFO:tasks.workunit.client.1.smithi188.stdout:3/326: dwrite d5/f41 [0,4194304] 0 2024-04-05T21:26:31.934 INFO:tasks.workunit.client.1.smithi188.stdout:5/283: rename d6/d17/c66 to d6/d17/d24/c69 0 2024-04-05T21:26:31.935 INFO:tasks.workunit.client.1.smithi188.stdout:1/341: truncate d1/dc/d16/d2d/f42 4025882 0 2024-04-05T21:26:31.935 INFO:tasks.workunit.client.1.smithi188.stdout:7/457: chown d3/d4/db/d2e/d3b 959 1 2024-04-05T21:26:31.937 INFO:tasks.workunit.client.1.smithi188.stdout:6/454: write d6/d13/d18/f36 [1037894,39055] 0 2024-04-05T21:26:31.942 INFO:tasks.workunit.client.1.smithi188.stdout:2/366: dread d5/db/dd/d36/d3d/d46/d5d/f5e [0,4194304] 0 2024-04-05T21:26:31.942 INFO:tasks.workunit.client.1.smithi188.stdout:5/284: stat d6/d15/d1b/f34 0 2024-04-05T21:26:31.942 INFO:tasks.workunit.client.1.smithi188.stdout:6/455: chown d6/d13 1723323 1 2024-04-05T21:26:31.943 INFO:tasks.workunit.client.1.smithi188.stdout:1/342: mknod d1/dc/d16/d21/d27/d49/c74 0 2024-04-05T21:26:31.944 INFO:tasks.workunit.client.1.smithi188.stdout:3/327: link d5/d7/d31/c57 d5/df/c70 0 2024-04-05T21:26:31.944 INFO:tasks.workunit.client.1.smithi188.stdout:6/456: mknod d6/d71/c8e 0 2024-04-05T21:26:31.949 INFO:tasks.workunit.client.1.smithi188.stdout:5/285: write d6/d17/d24/d44/f57 [1419633,73577] 0 2024-04-05T21:26:31.950 INFO:tasks.workunit.client.1.smithi188.stdout:2/367: fsync d5/db/dd/d36/d3c/d48/f56 0 2024-04-05T21:26:31.950 INFO:tasks.workunit.client.1.smithi188.stdout:9/356: dwrite dd/d11/d22/f5a [0,4194304] 0 2024-04-05T21:26:31.950 INFO:tasks.workunit.client.1.smithi188.stdout:6/457: rmdir d6/d13/d48/d49/d60 39 2024-04-05T21:26:31.955 INFO:tasks.workunit.client.1.smithi188.stdout:9/357: readlink dd/d11/d1f/d26/l6d 0 2024-04-05T21:26:31.956 INFO:tasks.workunit.client.1.smithi188.stdout:6/458: chown d6/d13/d18/f35 2011997 1 2024-04-05T21:26:31.957 INFO:tasks.workunit.client.1.smithi188.stdout:7/458: dwrite d3/d2c/d5e/f98 [0,4194304] 0 2024-04-05T21:26:31.958 INFO:tasks.workunit.client.1.smithi188.stdout:1/343: fsync d1/d4/d3f/f44 0 2024-04-05T21:26:31.960 INFO:tasks.workunit.client.1.smithi188.stdout:2/368: mknod d5/db/dd/d36/d3d/d46/d66/c70 0 2024-04-05T21:26:31.960 INFO:tasks.workunit.client.1.smithi188.stdout:7/459: write d3/f68 [381212,32639] 0 2024-04-05T21:26:31.964 INFO:tasks.workunit.client.1.smithi188.stdout:1/344: dread - d1/d7/f5c zero size 2024-04-05T21:26:31.964 INFO:tasks.workunit.client.1.smithi188.stdout:5/286: symlink d6/d17/d23/d35/d40/d42/d4e/l6a 0 2024-04-05T21:26:31.965 INFO:tasks.workunit.client.1.smithi188.stdout:1/345: truncate d1/dc/f56 1014341 0 2024-04-05T21:26:31.967 INFO:tasks.workunit.client.1.smithi188.stdout:2/369: symlink d5/db/dd/d36/d3d/l71 0 2024-04-05T21:26:31.968 INFO:tasks.workunit.client.1.smithi188.stdout:7/460: truncate d3/d4/f7 3245777 0 2024-04-05T21:26:31.969 INFO:tasks.workunit.client.1.smithi188.stdout:7/461: readlink d3/l26 0 2024-04-05T21:26:31.970 INFO:tasks.workunit.client.1.smithi188.stdout:6/459: link d6/d13/d18/d19/d2c/d34/f6b d6/d28/f8f 0 2024-04-05T21:26:31.973 INFO:tasks.workunit.client.1.smithi188.stdout:5/287: rename d6/d17/d24/d44/d51/l60 to d6/d17/d23/d35/l6b 0 2024-04-05T21:26:31.975 INFO:tasks.workunit.client.1.smithi188.stdout:1/346: link d1/d4/d3f/f40 d1/dc/d16/d21/d27/f75 0 2024-04-05T21:26:31.976 INFO:tasks.workunit.client.1.smithi188.stdout:7/462: truncate d3/d2c/d5e/f98 4530532 0 2024-04-05T21:26:31.978 INFO:tasks.workunit.client.1.smithi188.stdout:6/460: creat d6/d13/d18/d19/d2c/d3f/d77/f90 x:0 0 0 2024-04-05T21:26:31.978 INFO:tasks.workunit.client.1.smithi188.stdout:5/288: creat d6/d15/d1b/f6c x:0 0 0 2024-04-05T21:26:31.980 INFO:tasks.workunit.client.1.smithi188.stdout:2/370: dwrite d5/db/dd/f54 [0,4194304] 0 2024-04-05T21:26:31.981 INFO:tasks.workunit.client.1.smithi188.stdout:1/347: creat d1/d7/d4e/d5f/f76 x:0 0 0 2024-04-05T21:26:31.982 INFO:tasks.workunit.client.1.smithi188.stdout:7/463: symlink d3/d2c/d5e/d91/l9b 0 2024-04-05T21:26:31.983 INFO:tasks.workunit.client.1.smithi188.stdout:1/348: read - d1/f48 zero size 2024-04-05T21:26:31.994 INFO:tasks.workunit.client.1.smithi188.stdout:3/328: dread d5/d2b/f2e [0,4194304] 0 2024-04-05T21:26:31.995 INFO:tasks.workunit.client.1.smithi188.stdout:5/289: fsync d6/d17/d23/d35/d40/f65 0 2024-04-05T21:26:31.995 INFO:tasks.workunit.client.1.smithi188.stdout:1/349: unlink d1/d4/d3f/f44 0 2024-04-05T21:26:31.995 INFO:tasks.workunit.client.1.smithi188.stdout:3/329: creat d5/df/f71 x:0 0 0 2024-04-05T21:26:31.998 INFO:tasks.workunit.client.1.smithi188.stdout:5/290: rename d6/d15/d1b/d4f/l5a to d6/d17/d39/l6d 0 2024-04-05T21:26:32.000 INFO:tasks.workunit.client.1.smithi188.stdout:1/350: chown d1/dc/d16/d21/d27/l41 4497959 1 2024-04-05T21:26:32.002 INFO:tasks.workunit.client.1.smithi188.stdout:2/371: getdents d5/db/dd/d36/d3d/d46/d5d 0 2024-04-05T21:26:32.003 INFO:tasks.workunit.client.1.smithi188.stdout:3/330: dwrite d5/f41 [0,4194304] 0 2024-04-05T21:26:32.009 INFO:tasks.workunit.client.1.smithi188.stdout:5/291: getdents d6/d17/d23/d35 0 2024-04-05T21:26:32.009 INFO:tasks.workunit.client.1.smithi188.stdout:6/461: dread d6/d13/d18/f59 [0,4194304] 0 2024-04-05T21:26:32.012 INFO:tasks.workunit.client.1.smithi188.stdout:5/292: fsync d6/d17/d23/d35/d40/f65 0 2024-04-05T21:26:32.014 INFO:tasks.workunit.client.1.smithi188.stdout:1/351: dread d1/dc/f23 [0,4194304] 0 2024-04-05T21:26:32.020 INFO:tasks.workunit.client.1.smithi188.stdout:7/464: dwrite d3/d4/db/f4a [0,4194304] 0 2024-04-05T21:26:32.020 INFO:tasks.workunit.client.1.smithi188.stdout:5/293: creat d6/d17/d39/f6e x:0 0 0 2024-04-05T21:26:32.021 INFO:tasks.workunit.client.1.smithi188.stdout:6/462: dread d6/d13/d48/d49/f69 [0,4194304] 0 2024-04-05T21:26:32.024 INFO:tasks.workunit.client.1.smithi188.stdout:3/331: dwrite d5/dc/d2f/f4c [0,4194304] 0 2024-04-05T21:26:32.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.024+0000 7fefa285d700 1 -- 172.21.15.121:0/3364375222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef9c100bb0 msgr2=0x7fef9c100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:32.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.024+0000 7fefa285d700 1 --2- 172.21.15.121:0/3364375222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef9c100bb0 0x7fef9c100fd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fef90009910 tx=0x7fef90009c20 comp rx=0 tx=0).stop 2024-04-05T21:26:32.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.024+0000 7fefa285d700 1 -- 172.21.15.121:0/3364375222 shutdown_connections 2024-04-05T21:26:32.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.024+0000 7fefa285d700 1 --2- 172.21.15.121:0/3364375222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef9c101db0 0x7fef9c102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.024+0000 7fefa285d700 1 --2- 172.21.15.121:0/3364375222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef9c100bb0 0x7fef9c100fd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.024+0000 7fefa285d700 1 -- 172.21.15.121:0/3364375222 >> 172.21.15.121:0/3364375222 conn(0x7fef9c0fc110 msgr2=0x7fef9c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:32.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.024+0000 7fefa285d700 1 -- 172.21.15.121:0/3364375222 shutdown_connections 2024-04-05T21:26:32.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.024+0000 7fefa285d700 1 -- 172.21.15.121:0/3364375222 wait complete. 2024-04-05T21:26:32.026 INFO:tasks.workunit.client.1.smithi188.stdout:7/465: readlink d3/d4/d1d/d74/l57 0 2024-04-05T21:26:32.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.025+0000 7fefa285d700 1 Processor -- start 2024-04-05T21:26:32.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.025+0000 7fefa285d700 1 -- start start 2024-04-05T21:26:32.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.025+0000 7fefa285d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef9c100bb0 0x7fef9c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:32.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.025+0000 7fefa285d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef9c101db0 0x7fef9c073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.025+0000 7fefa285d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0736c0 con 0x7fef9c100bb0 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.025+0000 7fefa285d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c073800 con 0x7fef9c101db0 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.025+0000 7fef9bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef9c100bb0 0x7fef9c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.025+0000 7fef9bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef9c100bb0 0x7fef9c074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37454/0 (socket says 172.21.15.121:37454) 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.025+0000 7fef9bfff700 1 -- 172.21.15.121:0/55452187 learned_addr learned my addr 172.21.15.121:0/55452187 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.026+0000 7fef9bfff700 1 -- 172.21.15.121:0/55452187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef9c101db0 msgr2=0x7fef9c073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.026+0000 7fef9bfff700 1 --2- 172.21.15.121:0/55452187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef9c101db0 0x7fef9c073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.026+0000 7fef9bfff700 1 -- 172.21.15.121:0/55452187 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef900095f0 con 0x7fef9c100bb0 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.026+0000 7fef9bfff700 1 --2- 172.21.15.121:0/55452187 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef9c100bb0 0x7fef9c074b30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fef9000bda0 tx=0x7fef9000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.026+0000 7fef997fa700 1 -- 172.21.15.121:0/55452187 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9001d070 con 0x7fef9c100bb0 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.026+0000 7fef997fa700 1 -- 172.21.15.121:0/55452187 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fef9000f460 con 0x7fef9c100bb0 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.026+0000 7fefa285d700 1 -- 172.21.15.121:0/55452187 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c073a80 con 0x7fef9c100bb0 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.026+0000 7fef997fa700 1 -- 172.21.15.121:0/55452187 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef90021640 con 0x7fef9c100bb0 2024-04-05T21:26:32.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.026+0000 7fefa285d700 1 -- 172.21.15.121:0/55452187 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c073f70 con 0x7fef9c100bb0 2024-04-05T21:26:32.029 INFO:tasks.workunit.client.1.smithi188.stdout:6/463: rmdir d6/d13/d18/d19/d2c 39 2024-04-05T21:26:32.030 INFO:tasks.workunit.client.1.smithi188.stdout:5/294: link d6/f53 d6/f6f 0 2024-04-05T21:26:32.030 INFO:tasks.workunit.client.1.smithi188.stdout:7/466: stat d3/d4/db/d2e/c87 0 2024-04-05T21:26:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.027+0000 7fefa285d700 1 -- 172.21.15.121:0/55452187 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c1095d0 con 0x7fef9c100bb0 2024-04-05T21:26:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.028+0000 7fef997fa700 1 -- 172.21.15.121:0/55452187 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 25) v1 ==== 90681+0+0 (secure 0 0 0) 0x7fef9000f9b0 con 0x7fef9c100bb0 2024-04-05T21:26:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.028+0000 7fef997fa700 1 --2- 172.21.15.121:0/55452187 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fef8406ca20 0x7fef8406eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.028+0000 7fef997fa700 1 -- 172.21.15.121:0/55452187 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fef9008d300 con 0x7fef9c100bb0 2024-04-05T21:26:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.029+0000 7fef9b7fe700 1 --2- 172.21.15.121:0/55452187 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fef8406ca20 0x7fef8406eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.030+0000 7fef9b7fe700 1 --2- 172.21.15.121:0/55452187 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fef8406ca20 0x7fef8406eee0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fef8c005950 tx=0x7fef8c0058e0 comp rx=0 tx=0).ready entity=mgr.24327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:32.032 INFO:tasks.workunit.client.1.smithi188.stdout:3/332: unlink d5/dc/f5a 0 2024-04-05T21:26:32.032 INFO:tasks.workunit.client.1.smithi188.stdout:1/352: dwrite d1/f14 [0,4194304] 0 2024-04-05T21:26:32.032 INFO:tasks.workunit.client.1.smithi188.stdout:6/464: truncate d6/d13/d18/f35 4823650 0 2024-04-05T21:26:32.034 INFO:tasks.workunit.client.1.smithi188.stdout:5/295: write d6/d15/f61 [1011646,112454] 0 2024-04-05T21:26:32.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.034+0000 7fef997fa700 1 -- 172.21.15.121:0/55452187 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7fef9005ae60 con 0x7fef9c100bb0 2024-04-05T21:26:32.035 INFO:tasks.workunit.client.1.smithi188.stdout:3/333: chown d5/d7/d65/f54 142 1 2024-04-05T21:26:32.036 INFO:tasks.workunit.client.1.smithi188.stdout:7/467: mknod d3/d4/db/d55/c9c 0 2024-04-05T21:26:32.037 INFO:tasks.workunit.client.1.smithi188.stdout:1/353: chown d1/dc/d16/c2e 154 1 2024-04-05T21:26:32.038 INFO:tasks.workunit.client.1.smithi188.stdout:6/465: mknod d6/d28/d45/c91 0 2024-04-05T21:26:32.038 INFO:tasks.workunit.client.1.smithi188.stdout:7/468: read - d3/d4/db/d2e/d3b/f92 zero size 2024-04-05T21:26:32.039 INFO:tasks.workunit.client.1.smithi188.stdout:5/296: creat d6/d17/d24/d44/f70 x:0 0 0 2024-04-05T21:26:32.039 INFO:tasks.workunit.client.1.smithi188.stdout:3/334: read f4 [10648365,22770] 0 2024-04-05T21:26:32.040 INFO:tasks.workunit.client.1.smithi188.stdout:1/354: rmdir d1 39 2024-04-05T21:26:32.040 INFO:tasks.workunit.client.1.smithi188.stdout:5/297: chown c1 1526860 1 2024-04-05T21:26:32.040 INFO:tasks.workunit.client.1.smithi188.stdout:3/335: read - d5/df/f71 zero size 2024-04-05T21:26:32.041 INFO:tasks.workunit.client.1.smithi188.stdout:5/298: truncate d6/f6f 392185 0 2024-04-05T21:26:32.042 INFO:tasks.workunit.client.1.smithi188.stdout:7/469: mkdir d3/d4/db/d55/d9d 0 2024-04-05T21:26:32.043 INFO:tasks.workunit.client.1.smithi188.stdout:1/355: dread - d1/dc/d16/d46/f51 zero size 2024-04-05T21:26:32.046 INFO:tasks.workunit.client.1.smithi188.stdout:1/356: dread - d1/d7/d4e/d5f/f76 zero size 2024-04-05T21:26:32.047 INFO:tasks.workunit.client.1.smithi188.stdout:7/470: fsync d3/d4/db/f6e 0 2024-04-05T21:26:32.048 INFO:tasks.workunit.client.1.smithi188.stdout:5/299: symlink d6/d17/d23/d35/d40/d42/l71 0 2024-04-05T21:26:32.051 INFO:tasks.workunit.client.1.smithi188.stdout:7/471: mknod d3/d4/db/d2e/d49/c9e 0 2024-04-05T21:26:32.055 INFO:tasks.workunit.client.1.smithi188.stdout:3/336: link d5/d7/d31/l53 d5/d2b/l72 0 2024-04-05T21:26:32.058 INFO:tasks.workunit.client.1.smithi188.stdout:6/466: truncate d6/d13/d38/f4c 837151 0 2024-04-05T21:26:32.060 INFO:tasks.workunit.client.1.smithi188.stdout:7/472: dwrite d3/d2c/f3f [0,4194304] 0 2024-04-05T21:26:32.061 INFO:tasks.workunit.client.1.smithi188.stdout:7/473: fdatasync d3/d4/db/d2e/d49/f5d 0 2024-04-05T21:26:32.063 INFO:tasks.workunit.client.1.smithi188.stdout:3/337: symlink d5/d39/l73 0 2024-04-05T21:26:32.064 INFO:tasks.workunit.client.1.smithi188.stdout:7/474: chown d3/d4/d1d/d74/f80 18977 1 2024-04-05T21:26:32.064 INFO:tasks.workunit.client.1.smithi188.stdout:5/300: link d6/d17/d39/c5f d6/d15/d1b/c72 0 2024-04-05T21:26:32.065 INFO:tasks.workunit.client.1.smithi188.stdout:3/338: write d5/f68 [872628,54526] 0 2024-04-05T21:26:32.069 INFO:tasks.workunit.client.1.smithi188.stdout:5/301: fsync d6/d17/d23/d35/f50 0 2024-04-05T21:26:32.073 INFO:tasks.workunit.client.1.smithi188.stdout:7/475: getdents d3/d4/d1d 0 2024-04-05T21:26:32.076 INFO:tasks.workunit.client.1.smithi188.stdout:1/357: dwrite d1/dc/d16/d21/d27/f75 [0,4194304] 0 2024-04-05T21:26:32.078 INFO:tasks.workunit.client.1.smithi188.stdout:5/302: creat d6/f73 x:0 0 0 2024-04-05T21:26:32.080 INFO:tasks.workunit.client.1.smithi188.stdout:7/476: dread - d3/d4/db/d2e/d3b/f92 zero size 2024-04-05T21:26:32.081 INFO:tasks.workunit.client.1.smithi188.stdout:6/467: dread d6/d13/d38/f63 [0,4194304] 0 2024-04-05T21:26:32.083 INFO:tasks.workunit.client.1.smithi188.stdout:7/477: dread - d3/d4/db/d2e/d49/f67 zero size 2024-04-05T21:26:32.087 INFO:tasks.workunit.client.1.smithi188.stdout:1/358: dwrite d1/d4/d3f/f69 [0,4194304] 0 2024-04-05T21:26:32.094 INFO:tasks.workunit.client.1.smithi188.stdout:1/359: mknod d1/d7/d4e/d5f/c77 0 2024-04-05T21:26:32.094 INFO:tasks.workunit.client.1.smithi188.stdout:6/468: rmdir d6/d13/d18/d19/d2c/d3f/d8d 0 2024-04-05T21:26:32.095 INFO:tasks.workunit.client.1.smithi188.stdout:5/303: write d6/d17/d23/d35/d40/f65 [2815311,103426] 0 2024-04-05T21:26:32.099 INFO:tasks.workunit.client.1.smithi188.stdout:7/478: getdents d3/d4/d1d/d74 0 2024-04-05T21:26:32.099 INFO:tasks.workunit.client.1.smithi188.stdout:1/360: mknod d1/dc/d16/d21/c78 0 2024-04-05T21:26:32.101 INFO:tasks.workunit.client.1.smithi188.stdout:3/339: dread d5/df/f15 [0,4194304] 0 2024-04-05T21:26:32.102 INFO:tasks.workunit.client.1.smithi188.stdout:3/340: chown d5/d48 486923 1 2024-04-05T21:26:32.104 INFO:tasks.workunit.client.1.smithi188.stdout:3/341: symlink d5/df/l74 0 2024-04-05T21:26:32.105 INFO:tasks.workunit.client.1.smithi188.stdout:7/479: creat d3/d2c/d5e/d8e/f9f x:0 0 0 2024-04-05T21:26:32.106 INFO:tasks.workunit.client.1.smithi188.stdout:3/342: mkdir d5/d7/d6d/d75 0 2024-04-05T21:26:32.107 INFO:tasks.workunit.client.1.smithi188.stdout:1/361: creat d1/d4/f79 x:0 0 0 2024-04-05T21:26:32.109 INFO:tasks.workunit.client.1.smithi188.stdout:3/343: creat d5/d7/d31/f76 x:0 0 0 2024-04-05T21:26:32.111 INFO:tasks.workunit.client.1.smithi188.stdout:3/344: symlink d5/d7/d31/l77 0 2024-04-05T21:26:32.116 INFO:tasks.workunit.client.1.smithi188.stdout:5/304: truncate d6/f22 3752851 0 2024-04-05T21:26:32.117 INFO:tasks.workunit.client.1.smithi188.stdout:6/469: write d6/d7/fc [2706732,54233] 0 2024-04-05T21:26:32.118 INFO:tasks.workunit.client.1.smithi188.stdout:7/480: rmdir d3/d2c/d5e/d8e 39 2024-04-05T21:26:32.120 INFO:tasks.workunit.client.1.smithi188.stdout:3/345: dwrite d5/d7/f36 [0,4194304] 0 2024-04-05T21:26:32.124 INFO:tasks.workunit.client.1.smithi188.stdout:7/481: rename d3/l26 to d3/d4/db/d2e/d3b/la0 0 2024-04-05T21:26:32.125 INFO:tasks.workunit.client.1.smithi188.stdout:7/482: stat d3/d4/d7c 0 2024-04-05T21:26:32.130 INFO:tasks.workunit.client.1.smithi188.stdout:7/483: mknod d3/d4/db/d55/ca1 0 2024-04-05T21:26:32.130 INFO:tasks.workunit.client.1.smithi188.stdout:7/484: chown d3/c28 451833 1 2024-04-05T21:26:32.131 INFO:tasks.workunit.client.1.smithi188.stdout:3/346: getdents d5/d7/d65/d61 0 2024-04-05T21:26:32.132 INFO:tasks.workunit.client.1.smithi188.stdout:3/347: dread - d5/dc/f2d zero size 2024-04-05T21:26:32.134 INFO:tasks.workunit.client.1.smithi188.stdout:5/305: dread d6/d15/d1b/f1e [0,4194304] 0 2024-04-05T21:26:32.135 INFO:tasks.workunit.client.1.smithi188.stdout:7/485: mknod d3/d4/db/d2e/d37/ca2 0 2024-04-05T21:26:32.138 INFO:tasks.workunit.client.1.smithi188.stdout:5/306: write d6/d17/d24/d44/f57 [1074839,85895] 0 2024-04-05T21:26:32.138 INFO:tasks.workunit.client.1.smithi188.stdout:7/486: write d3/d4/db/fe [2657832,99626] 0 2024-04-05T21:26:32.142 INFO:tasks.workunit.client.1.smithi188.stdout:5/307: creat d6/d15/d5c/f74 x:0 0 0 2024-04-05T21:26:32.146 INFO:tasks.workunit.client.1.smithi188.stdout:3/348: rmdir d5/d7/d31 39 2024-04-05T21:26:32.146 INFO:tasks.workunit.client.1.smithi188.stdout:7/487: write d3/d4/d1d/d74/f5c [2490551,101584] 0 2024-04-05T21:26:32.146 INFO:tasks.workunit.client.1.smithi188.stdout:5/308: fsync d6/d17/d23/d35/f50 0 2024-04-05T21:26:32.147 INFO:tasks.workunit.client.1.smithi188.stdout:6/470: dread d6/f2e [0,4194304] 0 2024-04-05T21:26:32.150 INFO:tasks.workunit.client.1.smithi188.stdout:7/488: rename d3/d89 to d3/d4/d1d/da3 0 2024-04-05T21:26:32.151 INFO:tasks.workunit.client.1.smithi188.stdout:6/471: truncate d6/d13/d48/d49/f80 542278 0 2024-04-05T21:26:32.153 INFO:tasks.workunit.client.1.smithi188.stdout:5/309: dwrite d6/d17/d24/f3c [0,4194304] 0 2024-04-05T21:26:32.153 INFO:tasks.workunit.client.1.smithi188.stdout:7/489: truncate d3/d4/db/d2e/d49/f5d 852221 0 2024-04-05T21:26:32.154 INFO:tasks.workunit.client.1.smithi188.stdout:6/472: creat d6/d13/d18/d19/d2c/f92 x:0 0 0 2024-04-05T21:26:32.156 INFO:tasks.workunit.client.1.smithi188.stdout:5/310: mkdir d6/d17/d24/d44/d51/d75 0 2024-04-05T21:26:32.158 INFO:tasks.workunit.client.1.smithi188.stdout:6/473: write d6/d13/d18/f59 [1170281,32426] 0 2024-04-05T21:26:32.159 INFO:tasks.workunit.client.1.smithi188.stdout:5/311: mknod d6/d17/d23/d35/d40/d42/c76 0 2024-04-05T21:26:32.161 INFO:tasks.workunit.client.1.smithi188.stdout:6/474: write d6/d13/d18/f8c [5525107,5936] 0 2024-04-05T21:26:32.163 INFO:tasks.workunit.client.1.smithi188.stdout:5/312: write d6/d17/d39/f4a [2396474,31181] 0 2024-04-05T21:26:32.163 INFO:tasks.workunit.client.1.smithi188.stdout:7/490: write d3/d4/f25 [1942413,124571] 0 2024-04-05T21:26:32.163 INFO:tasks.workunit.client.1.smithi188.stdout:6/475: fdatasync d6/f64 0 2024-04-05T21:26:32.164 INFO:tasks.workunit.client.1.smithi188.stdout:6/476: chown d6/d28/f58 1167659692 1 2024-04-05T21:26:32.165 INFO:tasks.workunit.client.1.smithi188.stdout:5/313: dread - d6/d17/d23/d35/d40/d42/f56 zero size 2024-04-05T21:26:32.167 INFO:tasks.workunit.client.1.smithi188.stdout:5/314: write d6/d17/d24/f52 [570503,88852] 0 2024-04-05T21:26:32.167 INFO:tasks.workunit.client.1.smithi188.stdout:6/477: readlink d6/d13/d18/d19/d2c/l7d 0 2024-04-05T21:26:32.169 INFO:tasks.workunit.client.1.smithi188.stdout:5/315: write d6/d17/d23/d35/d40/d42/f56 [352409,89609] 0 2024-04-05T21:26:32.169 INFO:tasks.workunit.client.1.smithi188.stdout:6/478: rmdir d6/d13/d18 39 2024-04-05T21:26:32.171 INFO:tasks.workunit.client.1.smithi188.stdout:5/316: creat d6/d17/d23/d35/d40/d42/f77 x:0 0 0 2024-04-05T21:26:32.174 INFO:tasks.workunit.client.1.smithi188.stdout:5/317: mknod d6/d17/d24/d44/d51/c78 0 2024-04-05T21:26:32.177 INFO:tasks.workunit.client.1.smithi188.stdout:6/479: truncate d6/d13/d18/d3c/f5d 740277 0 2024-04-05T21:26:32.178 INFO:tasks.workunit.client.1.smithi188.stdout:6/480: fsync d6/d7/fc 0 2024-04-05T21:26:32.181 INFO:tasks.workunit.client.1.smithi188.stdout:6/481: symlink d6/d13/d18/d19/d2c/d34/l93 0 2024-04-05T21:26:32.182 INFO:tasks.workunit.client.1.smithi188.stdout:6/482: readlink d6/d71/l7b 0 2024-04-05T21:26:32.182 INFO:tasks.workunit.client.1.smithi188.stdout:5/318: dread d6/d15/d1b/f34 [0,4194304] 0 2024-04-05T21:26:32.183 INFO:tasks.workunit.client.1.smithi188.stdout:6/483: fsync d6/d7/fd 0 2024-04-05T21:26:32.185 INFO:tasks.workunit.client.1.smithi188.stdout:7/491: dread d3/d4/db/fe [0,4194304] 0 2024-04-05T21:26:32.185 INFO:tasks.workunit.client.1.smithi188.stdout:5/319: creat d6/d15/f79 x:0 0 0 2024-04-05T21:26:32.186 INFO:tasks.workunit.client.1.smithi188.stdout:5/320: chown d6/d17/d23/d35/d40 10 1 2024-04-05T21:26:32.186 INFO:tasks.workunit.client.1.smithi188.stdout:5/321: fdatasync d6/d17/f46 0 2024-04-05T21:26:32.187 INFO:tasks.workunit.client.1.smithi188.stdout:7/492: mknod d3/d4/db/d2e/d8c/ca4 0 2024-04-05T21:26:32.188 INFO:tasks.workunit.client.1.smithi188.stdout:7/493: chown c1 0 1 2024-04-05T21:26:32.190 INFO:tasks.workunit.client.1.smithi188.stdout:7/494: mkdir d3/da5 0 2024-04-05T21:26:32.192 INFO:tasks.workunit.client.1.smithi188.stdout:5/322: rename d6/d17/d24 to d6/d15/d5c/d7a 0 2024-04-05T21:26:32.193 INFO:tasks.workunit.client.1.smithi188.stdout:7/495: write d3/d4/d1d/d74/f59 [5241426,61944] 0 2024-04-05T21:26:32.194 INFO:tasks.workunit.client.1.smithi188.stdout:7/496: dread - d3/d2c/d5e/f66 zero size 2024-04-05T21:26:32.197 INFO:tasks.workunit.client.1.smithi188.stdout:5/323: link d6/d15/f61 d6/d17/d23/d35/d40/d42/d4e/f7b 0 2024-04-05T21:26:32.203 INFO:tasks.workunit.client.1.smithi188.stdout:5/324: dwrite d6/d17/f20 [0,4194304] 0 2024-04-05T21:26:32.204 INFO:tasks.workunit.client.1.smithi188.stdout:5/325: chown d6/d15/d5c/d7a/d44/f4d 559617418 1 2024-04-05T21:26:32.205 INFO:tasks.workunit.client.1.smithi188.stdout:5/326: chown d6/l28 12 1 2024-04-05T21:26:32.207 INFO:tasks.workunit.client.1.smithi188.stdout:5/327: symlink d6/d15/l7c 0 2024-04-05T21:26:32.208 INFO:tasks.workunit.client.1.smithi188.stdout:5/328: mkdir d6/d17/d23/d7d 0 2024-04-05T21:26:32.209 INFO:tasks.workunit.client.1.smithi188.stdout:5/329: write d6/d15/d5c/d7a/f52 [560019,28585] 0 2024-04-05T21:26:32.211 INFO:tasks.workunit.client.1.smithi188.stdout:5/330: write d6/d17/d23/d35/d40/d42/f67 [109949,35861] 0 2024-04-05T21:26:32.214 INFO:tasks.workunit.client.1.smithi188.stdout:5/331: creat d6/d17/d23/f7e x:0 0 0 2024-04-05T21:26:32.217 INFO:tasks.workunit.client.1.smithi188.stdout:5/332: creat d6/d15/f7f x:0 0 0 2024-04-05T21:26:32.218 INFO:tasks.workunit.client.1.smithi188.stdout:5/333: write d6/d15/f7f [467492,69665] 0 2024-04-05T21:26:32.219 INFO:tasks.workunit.client.1.smithi188.stdout:5/334: stat d6/d17/d39/l4c 0 2024-04-05T21:26:32.221 INFO:tasks.workunit.client.1.smithi188.stdout:5/335: creat d6/d17/f80 x:0 0 0 2024-04-05T21:26:32.223 INFO:tasks.workunit.client.1.smithi188.stdout:5/336: fsync d6/f22 0 2024-04-05T21:26:32.228 INFO:tasks.workunit.client.1.smithi188.stdout:5/337: getdents d6 0 2024-04-05T21:26:32.230 INFO:tasks.workunit.client.1.smithi188.stdout:5/338: mkdir d6/d17/d23/d35/d40/d42/d81 0 2024-04-05T21:26:32.231 INFO:tasks.workunit.client.1.smithi188.stdout:5/339: fdatasync d6/d15/d1b/f62 0 2024-04-05T21:26:32.232 INFO:tasks.workunit.client.1.smithi188.stdout:5/340: chown d6/d15/d1b/d4f/f59 515435645 1 2024-04-05T21:26:32.236 INFO:tasks.workunit.client.1.smithi188.stdout:5/341: rename d6/d17/d23/d35 to d6/d82 0 2024-04-05T21:26:32.244 INFO:tasks.workunit.client.1.smithi188.stdout:5/342: dwrite d6/d17/d39/f4a [0,4194304] 0 2024-04-05T21:26:32.247 INFO:tasks.workunit.client.1.smithi188.stdout:5/343: creat d6/d17/d23/f83 x:0 0 0 2024-04-05T21:26:32.254 INFO:tasks.workunit.client.1.smithi188.stdout:5/344: dwrite d6/f11 [0,4194304] 0 2024-04-05T21:26:32.261 INFO:tasks.workunit.client.1.smithi188.stdout:5/345: mknod d6/d15/c84 0 2024-04-05T21:26:32.261 INFO:tasks.workunit.client.1.smithi188.stdout:5/346: truncate d6/d82/d40/d42/d4e/f7b 1467352 0 2024-04-05T21:26:32.264 INFO:tasks.workunit.client.1.smithi188.stdout:5/347: mkdir d6/d17/d23/d85 0 2024-04-05T21:26:32.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.281+0000 7fefa285d700 1 -- 172.21.15.121:0/55452187 --> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fef9c106700 con 0x7fef8406ca20 2024-04-05T21:26:32.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.283+0000 7fef997fa700 1 -- 172.21.15.121:0/55452187 <== mgr.24327 v2:172.21.15.188:6828/2212394428 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fef9c106700 con 0x7fef8406ca20 2024-04-05T21:26:32.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.285+0000 7fefa285d700 1 -- 172.21.15.121:0/55452187 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fef8406ca20 msgr2=0x7fef8406eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:32.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.285+0000 7fefa285d700 1 --2- 172.21.15.121:0/55452187 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fef8406ca20 0x7fef8406eee0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fef8c005950 tx=0x7fef8c0058e0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.285+0000 7fefa285d700 1 -- 172.21.15.121:0/55452187 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef9c100bb0 msgr2=0x7fef9c074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:32.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.285+0000 7fefa285d700 1 --2- 172.21.15.121:0/55452187 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef9c100bb0 0x7fef9c074b30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fef9000bda0 tx=0x7fef9000be80 comp rx=0 tx=0).stop 2024-04-05T21:26:32.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.285+0000 7fefa285d700 1 -- 172.21.15.121:0/55452187 shutdown_connections 2024-04-05T21:26:32.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.285+0000 7fefa285d700 1 --2- 172.21.15.121:0/55452187 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fef8406ca20 0x7fef8406eee0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.285+0000 7fefa285d700 1 --2- 172.21.15.121:0/55452187 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef9c100bb0 0x7fef9c074b30 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.285+0000 7fefa285d700 1 --2- 172.21.15.121:0/55452187 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef9c101db0 0x7fef9c073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.285+0000 7fefa285d700 1 -- 172.21.15.121:0/55452187 >> 172.21.15.121:0/55452187 conn(0x7fef9c0fc110 msgr2=0x7fef9c104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:32.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.285+0000 7fefa285d700 1 -- 172.21.15.121:0/55452187 shutdown_connections 2024-04-05T21:26:32.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.285+0000 7fefa285d700 1 -- 172.21.15.121:0/55452187 wait complete. 2024-04-05T21:26:32.296 INFO:tasks.workunit.client.1.smithi188.stdout:8/238: sync 2024-04-05T21:26:32.296 INFO:tasks.workunit.client.1.smithi188.stdout:9/358: sync 2024-04-05T21:26:32.296 INFO:tasks.workunit.client.1.smithi188.stdout:0/216: sync 2024-04-05T21:26:32.296 INFO:tasks.workunit.client.1.smithi188.stdout:4/506: sync 2024-04-05T21:26:32.298 INFO:tasks.workunit.client.1.smithi188.stdout:4/507: dread - d0/d14/d15/d42/fa0 zero size 2024-04-05T21:26:32.299 INFO:tasks.workunit.client.1.smithi188.stdout:8/239: symlink d5/l56 0 2024-04-05T21:26:32.300 INFO:tasks.workunit.client.1.smithi188.stdout:8/240: write d5/dc/f17 [999635,38095] 0 2024-04-05T21:26:32.301 INFO:tasks.workunit.client.1.smithi188.stdout:4/508: write d0/d2/d4f/f67 [3676260,39593] 0 2024-04-05T21:26:32.303 INFO:tasks.workunit.client.1.smithi188.stdout:9/359: creat dd/d11/f76 x:0 0 0 2024-04-05T21:26:32.306 INFO:tasks.workunit.client.1.smithi188.stdout:0/217: getdents d4/d8/d10 0 2024-04-05T21:26:32.306 INFO:tasks.workunit.client.1.smithi188.stdout:8/241: dread d5/dc/f1d [0,4194304] 0 2024-04-05T21:26:32.311 INFO:tasks.workunit.client.1.smithi188.stdout:9/360: truncate dd/d11/d1f/d26/d55/f63 3837678 0 2024-04-05T21:26:32.314 INFO:tasks.workunit.client.1.smithi188.stdout:8/242: link d5/dc/f11 d5/d21/d40/f57 0 2024-04-05T21:26:32.315 INFO:tasks.workunit.client.1.smithi188.stdout:5/348: dwrite d6/d15/d1b/f1e [0,4194304] 0 2024-04-05T21:26:32.317 INFO:tasks.workunit.client.1.smithi188.stdout:9/361: readlink dd/d11/d1f/d26/l62 0 2024-04-05T21:26:32.321 INFO:tasks.workunit.client.1.smithi188.stdout:5/349: fdatasync d6/d15/d5c/d7a/d44/f5d 0 2024-04-05T21:26:32.323 INFO:tasks.workunit.client.1.smithi188.stdout:0/218: dwrite d4/d8/db/d21/f34 [0,4194304] 0 2024-04-05T21:26:32.325 INFO:tasks.workunit.client.1.smithi188.stdout:8/243: write d5/d21/d40/f4f [290596,49827] 0 2024-04-05T21:26:32.325 INFO:tasks.workunit.client.1.smithi188.stdout:9/362: write dd/d11/d1f/f29 [102921,70464] 0 2024-04-05T21:26:32.326 INFO:tasks.workunit.client.1.smithi188.stdout:4/509: dread d0/d14/d15/f5b [0,4194304] 0 2024-04-05T21:26:32.329 INFO:tasks.workunit.client.1.smithi188.stdout:9/363: readlink dd/d11/d1f/d26/l6d 0 2024-04-05T21:26:32.329 INFO:tasks.workunit.client.1.smithi188.stdout:5/350: creat d6/d17/d39/f86 x:0 0 0 2024-04-05T21:26:32.331 INFO:tasks.workunit.client.1.smithi188.stdout:8/244: mknod d5/d21/d40/d43/d48/c58 0 2024-04-05T21:26:32.333 INFO:tasks.workunit.client.1.smithi188.stdout:9/364: creat dd/d11/d1f/d25/d38/f77 x:0 0 0 2024-04-05T21:26:32.334 INFO:tasks.workunit.client.1.smithi188.stdout:8/245: dread - d5/d21/d40/d43/f51 zero size 2024-04-05T21:26:32.334 INFO:tasks.workunit.client.1.smithi188.stdout:4/510: creat d0/d14/d2f/fb8 x:0 0 0 2024-04-05T21:26:32.336 INFO:tasks.workunit.client.1.smithi188.stdout:5/351: mknod d6/d82/d40/d42/d81/c87 0 2024-04-05T21:26:32.336 INFO:tasks.workunit.client.1.smithi188.stdout:9/365: mknod dd/d11/c78 0 2024-04-05T21:26:32.338 INFO:tasks.workunit.client.1.smithi188.stdout:0/219: dwrite d4/d8/db/f11 [0,4194304] 0 2024-04-05T21:26:32.338 INFO:tasks.workunit.client.1.smithi188.stdout:9/366: truncate fc 4806202 0 2024-04-05T21:26:32.340 INFO:tasks.workunit.client.1.smithi188.stdout:5/352: chown d6/fe 126 1 2024-04-05T21:26:32.340 INFO:tasks.workunit.client.1.smithi188.stdout:9/367: truncate fc 5052916 0 2024-04-05T21:26:32.342 INFO:tasks.workunit.client.1.smithi188.stdout:8/246: write d5/d21/d2a/d47/f49 [4008228,39383] 0 2024-04-05T21:26:32.343 INFO:tasks.workunit.client.1.smithi188.stdout:5/353: fdatasync d6/f5b 0 2024-04-05T21:26:32.347 INFO:tasks.workunit.client.1.smithi188.stdout:8/247: dread d5/dc/f2c [0,4194304] 0 2024-04-05T21:26:32.349 INFO:tasks.workunit.client.1.smithi188.stdout:4/511: dwrite d0/d81/f85 [4194304,4194304] 0 2024-04-05T21:26:32.349 INFO:tasks.workunit.client.1.smithi188.stdout:0/220: rename d4/d8/d10/l2d to d4/d8/db/d42/d38/l50 0 2024-04-05T21:26:32.350 INFO:tasks.workunit.client.1.smithi188.stdout:0/221: fdatasync f3 0 2024-04-05T21:26:32.355 INFO:tasks.workunit.client.1.smithi188.stdout:5/354: creat d6/d15/d5c/d7a/d44/d51/f88 x:0 0 0 2024-04-05T21:26:32.359 INFO:tasks.workunit.client.1.smithi188.stdout:4/512: truncate d0/d14/d15/d78/fae 406262 0 2024-04-05T21:26:32.359 INFO:tasks.workunit.client.1.smithi188.stdout:8/248: mkdir d5/d21/d40/d43/d44/d59 0 2024-04-05T21:26:32.359 INFO:tasks.workunit.client.1.smithi188.stdout:0/222: dread d4/d8/db/f11 [0,4194304] 0 2024-04-05T21:26:32.359 INFO:tasks.workunit.client.1.smithi188.stdout:8/249: dread - d5/f2f zero size 2024-04-05T21:26:32.361 INFO:tasks.workunit.client.1.smithi188.stdout:4/513: write d0/d14/d15/d53/d82/d8f/d95/fa4 [759633,80030] 0 2024-04-05T21:26:32.364 INFO:tasks.workunit.client.1.smithi188.stdout:4/514: symlink d0/d14/d15/d78/db1/lb9 0 2024-04-05T21:26:32.365 INFO:tasks.workunit.client.1.smithi188.stdout:8/250: creat d5/d21/d40/d43/d44/d59/f5a x:0 0 0 2024-04-05T21:26:32.367 INFO:tasks.workunit.client.1.smithi188.stdout:9/368: dread dd/d11/d22/f56 [0,4194304] 0 2024-04-05T21:26:32.374 INFO:tasks.workunit.client.1.smithi188.stdout:5/355: getdents d6/d15/d5c/d7a/d44/d51 0 2024-04-05T21:26:32.375 INFO:tasks.workunit.client.1.smithi188.stdout:9/369: dwrite dd/d11/d1f/f29 [0,4194304] 0 2024-04-05T21:26:32.377 INFO:tasks.workunit.client.1.smithi188.stdout:4/515: rmdir d0/d14/d15/d78/db1 39 2024-04-05T21:26:32.377 INFO:tasks.workunit.client.1.smithi188.stdout:8/251: truncate d5/f1f 3506079 0 2024-04-05T21:26:32.378 INFO:tasks.workunit.client.1.smithi188.stdout:8/252: chown d5/f1e 191 1 2024-04-05T21:26:32.379 INFO:tasks.workunit.client.1.smithi188.stdout:5/356: rmdir d6/d17 39 2024-04-05T21:26:32.380 INFO:tasks.workunit.client.1.smithi188.stdout:9/370: symlink dd/d11/d1f/d25/d51/l79 0 2024-04-05T21:26:32.381 INFO:tasks.workunit.client.1.smithi188.stdout:9/371: chown dd/d11/f76 56196 1 2024-04-05T21:26:32.381 INFO:tasks.workunit.client.1.smithi188.stdout:9/372: stat ca 0 2024-04-05T21:26:32.381 INFO:tasks.workunit.client.1.smithi188.stdout:4/516: unlink d0/lab 0 2024-04-05T21:26:32.382 INFO:tasks.workunit.client.1.smithi188.stdout:9/373: write fc [1738461,114458] 0 2024-04-05T21:26:32.384 INFO:tasks.workunit.client.1.smithi188.stdout:8/253: dread - d5/dc/f11 zero size 2024-04-05T21:26:32.386 INFO:tasks.workunit.client.1.smithi188.stdout:8/254: write d5/d21/d40/d43/f4c [159119,62573] 0 2024-04-05T21:26:32.386 INFO:tasks.workunit.client.1.smithi188.stdout:9/374: fsync dd/d11/f3c 0 2024-04-05T21:26:32.388 INFO:tasks.workunit.client.1.smithi188.stdout:4/517: mknod d0/d14/d2f/cba 0 2024-04-05T21:26:32.389 INFO:tasks.workunit.client.1.smithi188.stdout:8/255: mkdir d5/d21/d40/d43/d44/d5b 0 2024-04-05T21:26:32.393 INFO:tasks.workunit.client.1.smithi188.stdout:8/256: read d5/f8 [87297,35627] 0 2024-04-05T21:26:32.398 INFO:tasks.workunit.client.1.smithi188.stdout:4/518: rename d0/d14/d15/d1b/f4d to d0/d2/fbb 0 2024-04-05T21:26:32.400 INFO:tasks.workunit.client.1.smithi188.stdout:8/257: mknod d5/d21/d40/d43/c5c 0 2024-04-05T21:26:32.401 INFO:tasks.workunit.client.1.smithi188.stdout:5/357: truncate d6/d82/d40/d42/f5e 1760666 0 2024-04-05T21:26:32.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.403+0000 7f20dc51e700 1 -- 172.21.15.121:0/2589150115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d4100bb0 msgr2=0x7f20d4100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:32.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.403+0000 7f20dc51e700 1 --2- 172.21.15.121:0/2589150115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d4100bb0 0x7f20d4100fd0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f20d0009910 tx=0x7f20d0009c20 comp rx=0 tx=0).stop 2024-04-05T21:26:32.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.404+0000 7f20dc51e700 1 -- 172.21.15.121:0/2589150115 shutdown_connections 2024-04-05T21:26:32.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.404+0000 7f20dc51e700 1 --2- 172.21.15.121:0/2589150115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20d4101db0 0x7f20d4102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.404+0000 7f20dc51e700 1 --2- 172.21.15.121:0/2589150115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d4100bb0 0x7f20d4100fd0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.404+0000 7f20dc51e700 1 -- 172.21.15.121:0/2589150115 >> 172.21.15.121:0/2589150115 conn(0x7f20d40fc110 msgr2=0x7f20d40fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.404+0000 7f20dc51e700 1 -- 172.21.15.121:0/2589150115 shutdown_connections 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.404+0000 7f20dc51e700 1 -- 172.21.15.121:0/2589150115 wait complete. 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.404+0000 7f20dc51e700 1 Processor -- start 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.404+0000 7f20dc51e700 1 -- start start 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.404+0000 7f20dc51e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d4100bb0 0x7f20d4196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.404+0000 7f20dc51e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20d4101db0 0x7f20d41967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.404+0000 7f20dc51e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d4196dd0 con 0x7f20d4100bb0 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.404+0000 7f20dc51e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d4196f10 con 0x7f20d4101db0 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20da2ba700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d4100bb0 0x7f20d4196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20da2ba700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d4100bb0 0x7f20d4196270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37476/0 (socket says 172.21.15.121:37476) 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20da2ba700 1 -- 172.21.15.121:0/1120562345 learned_addr learned my addr 172.21.15.121:0/1120562345 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:26:32.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20d9ab9700 1 --2- 172.21.15.121:0/1120562345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20d4101db0 0x7f20d41967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:32.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20da2ba700 1 -- 172.21.15.121:0/1120562345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20d4101db0 msgr2=0x7f20d41967b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:32.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20da2ba700 1 --2- 172.21.15.121:0/1120562345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20d4101db0 0x7f20d41967b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20da2ba700 1 -- 172.21.15.121:0/1120562345 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d00095f0 con 0x7f20d4100bb0 2024-04-05T21:26:32.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20d9ab9700 1 --2- 172.21.15.121:0/1120562345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20d4101db0 0x7f20d41967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:26:32.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20da2ba700 1 --2- 172.21.15.121:0/1120562345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d4100bb0 0x7f20d4196270 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f20d000bda0 tx=0x7f20d000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:32.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20cb7fe700 1 -- 172.21.15.121:0/1120562345 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20d001d070 con 0x7f20d4100bb0 2024-04-05T21:26:32.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20cb7fe700 1 -- 172.21.15.121:0/1120562345 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f20d000f460 con 0x7f20d4100bb0 2024-04-05T21:26:32.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20dc51e700 1 -- 172.21.15.121:0/1120562345 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d419b960 con 0x7f20d4100bb0 2024-04-05T21:26:32.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20cb7fe700 1 -- 172.21.15.121:0/1120562345 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20d0021640 con 0x7f20d4100bb0 2024-04-05T21:26:32.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.405+0000 7f20dc51e700 1 -- 172.21.15.121:0/1120562345 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d419be50 con 0x7f20d4100bb0 2024-04-05T21:26:32.409 INFO:tasks.workunit.client.1.smithi188.stdout:9/375: dwrite dd/d11/d1f/d26/f2c [0,4194304] 0 2024-04-05T21:26:32.409 INFO:tasks.workunit.client.1.smithi188.stdout:9/376: write dd/f72 [2622844,124655] 0 2024-04-05T21:26:32.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.406+0000 7f20dc51e700 1 -- 172.21.15.121:0/1120562345 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d41095d0 con 0x7f20d4100bb0 2024-04-05T21:26:32.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.407+0000 7f20cb7fe700 1 -- 172.21.15.121:0/1120562345 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 25) v1 ==== 90681+0+0 (secure 0 0 0) 0x7f20d000f9b0 con 0x7f20d4100bb0 2024-04-05T21:26:32.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.407+0000 7f20cb7fe700 1 --2- 172.21.15.121:0/1120562345 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f20c006ca20 0x7f20c006eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:32.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.408+0000 7f20cb7fe700 1 -- 172.21.15.121:0/1120562345 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f20d008d340 con 0x7f20d4100bb0 2024-04-05T21:26:32.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.408+0000 7f20d9ab9700 1 --2- 172.21.15.121:0/1120562345 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f20c006ca20 0x7f20c006eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:32.411 INFO:tasks.workunit.client.1.smithi188.stdout:8/258: dwrite d5/d21/d2a/d47/f49 [0,4194304] 0 2024-04-05T21:26:32.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.409+0000 7f20d9ab9700 1 --2- 172.21.15.121:0/1120562345 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f20c006ca20 0x7f20c006eee0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f20c4005fd0 tx=0x7f20c4005f60 comp rx=0 tx=0).ready entity=mgr.24327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:32.412 INFO:tasks.workunit.client.1.smithi188.stdout:8/259: dread d5/dc/f2c [0,4194304] 0 2024-04-05T21:26:32.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.413+0000 7f20cb7fe700 1 -- 172.21.15.121:0/1120562345 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f20d005aea0 con 0x7f20d4100bb0 2024-04-05T21:26:32.481 INFO:tasks.workunit.client.1.smithi188.stdout:1/362: sync 2024-04-05T21:26:32.481 INFO:tasks.workunit.client.1.smithi188.stdout:2/372: sync 2024-04-05T21:26:32.482 INFO:tasks.workunit.client.1.smithi188.stdout:9/377: creat dd/d3d/f7a x:0 0 0 2024-04-05T21:26:32.488 INFO:tasks.workunit.client.1.smithi188.stdout:8/260: symlink d5/d21/d40/d43/d44/d5b/l5d 0 2024-04-05T21:26:32.491 INFO:tasks.workunit.client.1.smithi188.stdout:5/358: rmdir d6/d15/d5c/d7a/d44/d51/d75 0 2024-04-05T21:26:32.491 INFO:tasks.workunit.client.1.smithi188.stdout:2/373: dwrite d5/db/dd/d36/d3c/f4a [0,4194304] 0 2024-04-05T21:26:32.492 INFO:tasks.workunit.client.1.smithi188.stdout:4/519: link d0/d14/d15/d1b/f60 d0/d14/d15/d78/fbc 0 2024-04-05T21:26:32.492 INFO:tasks.workunit.client.1.smithi188.stdout:8/261: rmdir d5 39 2024-04-05T21:26:32.494 INFO:tasks.workunit.client.1.smithi188.stdout:1/363: creat d1/dc/f7a x:0 0 0 2024-04-05T21:26:32.494 INFO:tasks.workunit.client.1.smithi188.stdout:4/520: read d0/d14/d15/d42/f63 [114923,42944] 0 2024-04-05T21:26:32.495 INFO:tasks.workunit.client.1.smithi188.stdout:5/359: creat d6/d82/f89 x:0 0 0 2024-04-05T21:26:32.496 INFO:tasks.workunit.client.1.smithi188.stdout:9/378: creat dd/d11/d1f/f7b x:0 0 0 2024-04-05T21:26:32.496 INFO:tasks.workunit.client.1.smithi188.stdout:4/521: read d0/d2/d17/f44 [3196760,8619] 0 2024-04-05T21:26:32.497 INFO:tasks.workunit.client.1.smithi188.stdout:5/360: dread - d6/d82/f89 zero size 2024-04-05T21:26:32.498 INFO:tasks.workunit.client.1.smithi188.stdout:2/374: mknod d5/c72 0 2024-04-05T21:26:32.500 INFO:tasks.workunit.client.1.smithi188.stdout:8/262: dread - d5/f3c zero size 2024-04-05T21:26:32.502 INFO:tasks.workunit.client.1.smithi188.stdout:9/379: write dd/d49/f53 [89033,117711] 0 2024-04-05T21:26:32.506 INFO:tasks.workunit.client.1.smithi188.stdout:1/364: dwrite d1/d7/f66 [0,4194304] 0 2024-04-05T21:26:32.508 INFO:tasks.workunit.client.1.smithi188.stdout:1/365: chown d1/d4/d3f/f69 3239 1 2024-04-05T21:26:32.514 INFO:tasks.workunit.client.1.smithi188.stdout:2/375: dwrite d5/db/d35/d3b/f4d [0,4194304] 0 2024-04-05T21:26:32.516 INFO:tasks.workunit.client.1.smithi188.stdout:5/361: write d6/fe [446794,92944] 0 2024-04-05T21:26:32.518 INFO:tasks.workunit.client.1.smithi188.stdout:2/376: write d5/db/d35/f43 [169691,1299] 0 2024-04-05T21:26:32.518 INFO:tasks.workunit.client.1.smithi188.stdout:5/362: chown d6/d82/d40/d42/l71 1614074631 1 2024-04-05T21:26:32.520 INFO:tasks.workunit.client.1.smithi188.stdout:5/363: write d6/fe [505104,22110] 0 2024-04-05T21:26:32.522 INFO:tasks.workunit.client.1.smithi188.stdout:5/364: chown d6/d15/d5c/d7a 649465386 1 2024-04-05T21:26:32.525 INFO:tasks.workunit.client.1.smithi188.stdout:5/365: rmdir d6/d15/d1b 39 2024-04-05T21:26:32.528 INFO:tasks.workunit.client.1.smithi188.stdout:5/366: stat d6/d15/d1b/d4f 0 2024-04-05T21:26:32.528 INFO:tasks.workunit.client.1.smithi188.stdout:8/263: write d5/dc/f11 [837570,43614] 0 2024-04-05T21:26:32.531 INFO:tasks.workunit.client.1.smithi188.stdout:4/522: dwrite d0/d14/d15/d78/fbc [0,4194304] 0 2024-04-05T21:26:32.531 INFO:tasks.workunit.client.1.smithi188.stdout:2/377: dwrite d5/f2f [4194304,4194304] 0 2024-04-05T21:26:32.531 INFO:tasks.workunit.client.1.smithi188.stdout:5/367: rmdir d6/d82 39 2024-04-05T21:26:32.541 INFO:tasks.workunit.client.1.smithi188.stdout:5/368: creat d6/d15/d1b/f8a x:0 0 0 2024-04-05T21:26:32.542 INFO:tasks.workunit.client.1.smithi188.stdout:8/264: mknod d5/d21/c5e 0 2024-04-05T21:26:32.545 INFO:tasks.workunit.client.1.smithi188.stdout:4/523: dwrite d0/d14/d15/d53/d82/d8f/d95/fa4 [0,4194304] 0 2024-04-05T21:26:32.545 INFO:tasks.workunit.client.1.smithi188.stdout:5/369: fsync d6/d15/d1b/f6c 0 2024-04-05T21:26:32.545 INFO:tasks.workunit.client.1.smithi188.stdout:2/378: dread d5/f2f [4194304,4194304] 0 2024-04-05T21:26:32.545 INFO:tasks.workunit.client.1.smithi188.stdout:1/366: dwrite d1/dc/f15 [0,4194304] 0 2024-04-05T21:26:32.550 INFO:tasks.workunit.client.1.smithi188.stdout:8/265: creat d5/d21/d50/f5f x:0 0 0 2024-04-05T21:26:32.552 INFO:tasks.workunit.client.1.smithi188.stdout:2/379: fsync d5/db/dd/d24/f28 0 2024-04-05T21:26:32.553 INFO:tasks.workunit.client.1.smithi188.stdout:4/524: stat d0/d2/d52/l9a 0 2024-04-05T21:26:32.556 INFO:tasks.workunit.client.1.smithi188.stdout:2/380: write d5/db/dd/d36/d3c/f4a [1416551,31331] 0 2024-04-05T21:26:32.556 INFO:tasks.workunit.client.1.smithi188.stdout:1/367: rename d1/dc/d16/d21/d27/d49/l72 to d1/l7b 0 2024-04-05T21:26:32.558 INFO:tasks.workunit.client.1.smithi188.stdout:4/525: mkdir d0/d14/d15/d53/d82/d8f/dbd 0 2024-04-05T21:26:32.559 INFO:tasks.workunit.client.1.smithi188.stdout:4/526: fdatasync d0/d14/f1c 0 2024-04-05T21:26:32.559 INFO:tasks.workunit.client.1.smithi188.stdout:5/370: rename d6/d15/d5c/d7a/d44/f70 to d6/d82/d40/d42/d4e/f8b 0 2024-04-05T21:26:32.560 INFO:tasks.workunit.client.1.smithi188.stdout:8/266: dread d5/dc/f36 [0,4194304] 0 2024-04-05T21:26:32.564 INFO:tasks.workunit.client.1.smithi188.stdout:1/368: rename d1/dc/d16/d21/d27/l41 to d1/d4/d13/d60/l7c 0 2024-04-05T21:26:32.566 INFO:tasks.workunit.client.1.smithi188.stdout:4/527: stat d0/d14/d15/d78/l91 0 2024-04-05T21:26:32.567 INFO:tasks.workunit.client.1.smithi188.stdout:2/381: dwrite d5/db/dd/d24/f2e [0,4194304] 0 2024-04-05T21:26:32.567 INFO:tasks.workunit.client.1.smithi188.stdout:8/267: symlink d5/d21/d40/d43/d44/d5b/l60 0 2024-04-05T21:26:32.569 INFO:tasks.workunit.client.1.smithi188.stdout:4/528: read d0/d2/d17/d48/f57 [518811,81585] 0 2024-04-05T21:26:32.570 INFO:tasks.workunit.client.1.smithi188.stdout:1/369: creat d1/dc/d16/d21/f7d x:0 0 0 2024-04-05T21:26:32.574 INFO:tasks.workunit.client.1.smithi188.stdout:2/382: chown d5/db/f20 12655628 1 2024-04-05T21:26:32.575 INFO:tasks.workunit.client.1.smithi188.stdout:8/268: symlink d5/d21/d2a/d47/l61 0 2024-04-05T21:26:32.578 INFO:tasks.workunit.client.1.smithi188.stdout:8/269: write d5/f34 [1009256,76796] 0 2024-04-05T21:26:32.581 INFO:tasks.workunit.client.1.smithi188.stdout:1/370: dwrite d1/d4/d3f/f45 [0,4194304] 0 2024-04-05T21:26:32.581 INFO:tasks.workunit.client.1.smithi188.stdout:5/371: rename d6/d82/d40/d42/d4e to d6/d17/d23/d85/d8c 0 2024-04-05T21:26:32.583 INFO:tasks.workunit.client.1.smithi188.stdout:5/372: stat d6/d17/l1d 0 2024-04-05T21:26:32.583 INFO:tasks.workunit.client.1.smithi188.stdout:2/383: mkdir d5/db/dd/d73 0 2024-04-05T21:26:32.585 INFO:tasks.workunit.client.1.smithi188.stdout:5/373: fdatasync d6/d15/d1b/f6c 0 2024-04-05T21:26:32.585 INFO:tasks.workunit.client.1.smithi188.stdout:1/371: creat d1/dc/d16/d2d/f7e x:0 0 0 2024-04-05T21:26:32.595 INFO:tasks.workunit.client.1.smithi188.stdout:4/529: write d0/d2/d17/f44 [6112286,63517] 0 2024-04-05T21:26:32.595 INFO:tasks.workunit.client.1.smithi188.stdout:5/374: creat d6/d17/d23/d85/d8c/f8d x:0 0 0 2024-04-05T21:26:32.596 INFO:tasks.workunit.client.1.smithi188.stdout:5/375: dread - d6/d17/d39/f86 zero size 2024-04-05T21:26:32.597 INFO:tasks.workunit.client.1.smithi188.stdout:1/372: rmdir d1/dc/d16/d2d 39 2024-04-05T21:26:32.599 INFO:tasks.workunit.client.1.smithi188.stdout:8/270: truncate d5/f24 4052448 0 2024-04-05T21:26:32.600 INFO:tasks.workunit.client.1.smithi188.stdout:5/376: mkdir d6/d17/d39/d8e 0 2024-04-05T21:26:32.600 INFO:tasks.workunit.client.1.smithi188.stdout:8/271: write d5/d21/d40/d43/f46 [567162,75485] 0 2024-04-05T21:26:32.601 INFO:tasks.workunit.client.1.smithi188.stdout:4/530: mkdir d0/dbe 0 2024-04-05T21:26:32.602 INFO:tasks.workunit.client.1.smithi188.stdout:1/373: creat d1/dc/d16/d21/d27/d4a/f7f x:0 0 0 2024-04-05T21:26:32.604 INFO:tasks.workunit.client.1.smithi188.stdout:2/384: link d5/db/c4e d5/db/dd/d36/c74 0 2024-04-05T21:26:32.606 INFO:tasks.workunit.client.1.smithi188.stdout:5/377: creat d6/d17/d39/d8e/f8f x:0 0 0 2024-04-05T21:26:32.606 INFO:tasks.workunit.client.1.smithi188.stdout:5/378: fsync d6/d82/f89 0 2024-04-05T21:26:32.607 INFO:tasks.workunit.client.1.smithi188.stdout:1/374: creat d1/d7/f80 x:0 0 0 2024-04-05T21:26:32.609 INFO:tasks.workunit.client.1.smithi188.stdout:1/375: chown d1/dc/d16/d21/d27/f29 9620885 1 2024-04-05T21:26:32.610 INFO:tasks.workunit.client.1.smithi188.stdout:5/379: creat d6/d82/d40/f90 x:0 0 0 2024-04-05T21:26:32.611 INFO:tasks.workunit.client.1.smithi188.stdout:5/380: truncate d6/d17/f46 551898 0 2024-04-05T21:26:32.612 INFO:tasks.workunit.client.1.smithi188.stdout:4/531: rename d0/f9c to d0/fbf 0 2024-04-05T21:26:32.613 INFO:tasks.workunit.client.1.smithi188.stdout:2/385: mkdir d5/db/dd/d36/d3c/d63/d75 0 2024-04-05T21:26:32.616 INFO:tasks.workunit.client.1.smithi188.stdout:5/381: symlink d6/d15/d5c/d7a/l91 0 2024-04-05T21:26:32.622 INFO:tasks.workunit.client.1.smithi188.stdout:5/382: dread d6/d15/d5c/d7a/d44/f4d [0,4194304] 0 2024-04-05T21:26:32.625 INFO:tasks.workunit.client.1.smithi188.stdout:4/532: dwrite d0/d2/fbb [0,4194304] 0 2024-04-05T21:26:32.629 INFO:tasks.workunit.client.1.smithi188.stdout:2/386: creat d5/db/f76 x:0 0 0 2024-04-05T21:26:32.630 INFO:tasks.workunit.client.1.smithi188.stdout:1/376: truncate d1/d4/f1d 438465 0 2024-04-05T21:26:32.632 INFO:tasks.workunit.client.1.smithi188.stdout:8/272: dwrite d5/f24 [0,4194304] 0 2024-04-05T21:26:32.632 INFO:tasks.workunit.client.1.smithi188.stdout:1/377: chown d1/dc/d16/d2d 0 1 2024-04-05T21:26:32.634 INFO:tasks.workunit.client.1.smithi188.stdout:5/383: rename d6/l36 to d6/d15/d5c/d7a/d44/l92 0 2024-04-05T21:26:32.634 INFO:tasks.workunit.client.1.smithi188.stdout:1/378: stat d1/dc/d16/c2e 0 2024-04-05T21:26:32.636 INFO:tasks.workunit.client.1.smithi188.stdout:8/273: rmdir d5/dc 39 2024-04-05T21:26:32.639 INFO:tasks.workunit.client.1.smithi188.stdout:5/384: creat d6/d15/d5c/d7a/f93 x:0 0 0 2024-04-05T21:26:32.640 INFO:tasks.workunit.client.1.smithi188.stdout:5/385: chown d6/d17/c3f 366903623 1 2024-04-05T21:26:32.640 INFO:tasks.workunit.client.1.smithi188.stdout:2/387: dwrite d5/db/dd/d36/d3c/f4a [0,4194304] 0 2024-04-05T21:26:32.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.641+0000 7f20dc51e700 1 -- 172.21.15.121:0/1120562345 --> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f20d4106700 con 0x7f20c006ca20 2024-04-05T21:26:32.643 INFO:tasks.workunit.client.1.smithi188.stdout:1/379: rmdir d1/dc/d16/d21 39 2024-04-05T21:26:32.643 INFO:tasks.workunit.client.1.smithi188.stdout:8/274: truncate d5/f3c 953785 0 2024-04-05T21:26:32.645 INFO:tasks.workunit.client.1.smithi188.stdout:4/533: getdents d0/d2/d2a/d34/d6b 0 2024-04-05T21:26:32.645 INFO:tasks.workunit.client.1.smithi188.stdout:5/386: readlink d6/l8 0 2024-04-05T21:26:32.646 INFO:tasks.workunit.client.1.smithi188.stdout:1/380: write d1/d4/d3f/f4b [1276232,118617] 0 2024-04-05T21:26:32.646 INFO:tasks.workunit.client.1.smithi188.stdout:4/534: stat d0/d14/l1f 0 2024-04-05T21:26:32.650 INFO:tasks.workunit.client.1.smithi188.stdout:8/275: creat d5/d21/d40/d43/f62 x:0 0 0 2024-04-05T21:26:32.650 INFO:tasks.workunit.client.1.smithi188.stdout:2/388: link d5/db/dd/d24/l42 d5/db/dd/d36/d3c/d48/l77 0 2024-04-05T21:26:32.651 INFO:tasks.workunit.client.1.smithi188.stdout:5/387: read d6/d15/d1b/f2a [2432338,101415] 0 2024-04-05T21:26:32.656 INFO:tasks.workunit.client.1.smithi188.stdout:2/389: fsync d5/db/dd/d36/d3d/d46/d5d/f5e 0 2024-04-05T21:26:32.656 INFO:tasks.workunit.client.1.smithi188.stdout:2/390: stat d5/db/d35/d3b/f5a 0 2024-04-05T21:26:32.656 INFO:tasks.workunit.client.1.smithi188.stdout:2/391: readlink d5/db/dd/l26 0 2024-04-05T21:26:32.657 INFO:tasks.workunit.client.1.smithi188.stdout:7/497: sync 2024-04-05T21:26:32.658 INFO:tasks.workunit.client.1.smithi188.stdout:8/276: mkdir d5/d21/d50/d63 0 2024-04-05T21:26:32.658 INFO:tasks.workunit.client.1.smithi188.stdout:1/381: dread d1/f14 [0,4194304] 0 2024-04-05T21:26:32.659 INFO:tasks.workunit.client.1.smithi188.stdout:6/484: sync 2024-04-05T21:26:32.659 INFO:tasks.workunit.client.1.smithi188.stdout:3/349: sync 2024-04-05T21:26:32.660 INFO:tasks.workunit.client.1.smithi188.stdout:8/277: write d5/f1c [3804906,120744] 0 2024-04-05T21:26:32.660 INFO:tasks.workunit.client.1.smithi188.stdout:4/535: read d0/d14/d15/d1b/f3d [3376084,125995] 0 2024-04-05T21:26:32.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.660+0000 7f20cb7fe700 1 -- 172.21.15.121:0/1120562345 <== mgr.24327 v2:172.21.15.188:6828/2212394428 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f20d4106700 con 0x7f20c006ca20 2024-04-05T21:26:32.661 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:26:32.661 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (7m) 14s ago 8m 26.2M - 0.25.0 c8568f914cd2 6594a12b2bc8 2024-04-05T21:26:32.661 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (9m) 14s ago 9m 7726k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:26:32.661 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (7m) 31s ago 7m 7612k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (9m) 14s ago 9m 6651k - 18.2.1 5be31c24972a 5bf35fe2ace4 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (7m) 31s ago 7m 6651k - 18.2.1 5be31c24972a dcdfa6ae17bf 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (7m) 14s ago 8m 91.4M - 9.4.7 954c08fa6188 ee38366c66e3 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (2m) 14s ago 2m 66.2M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (2m) 14s ago 2m 135M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (2m) 31s ago 2m 78.4M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (2m) 31s ago 2m 35.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:9283,8765,8443 running (10m) 14s ago 10m 168M - 18.2.1 5be31c24972a d8cb1548d844 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (36s) 31s ago 7m 58.5M - 18.2.2-1173-g8919605b 067bdb249792 2d759c011461 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (10m) 14s ago 10m 49.7M 2048M 18.2.1 5be31c24972a 3c30ce99c006 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (7m) 31s ago 7m 35.9M 2048M 18.2.1 5be31c24972a 8ecc9636670e 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (8m) 14s ago 8m 18.2M - 1.5.0 0da6a335fe13 5ac35283f3e9 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (7m) 31s ago 7m 16.5M - 1.5.0 0da6a335fe13 bc2ad104140f 2024-04-05T21:26:32.662 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (6m) 14s ago 6m 117M 1227M 18.2.1 5be31c24972a f6f3a45429a2 2024-04-05T21:26:32.663 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (6m) 14s ago 6m 138M 1227M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:26:32.663 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (5m) 14s ago 5m 106M 1227M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:26:32.663 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (5m) 31s ago 5m 116M 4982M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:26:32.663 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (4m) 31s ago 4m 107M 4982M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:26:32.663 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (4m) 31s ago 4m 102M 4982M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:26:32.663 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (6m) 14s ago 8m 60.8M - 2.43.0 a07b618ecd1d 9395525b1c3e 2024-04-05T21:26:32.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.663+0000 7f20dc51e700 1 -- 172.21.15.121:0/1120562345 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f20c006ca20 msgr2=0x7f20c006eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:32.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.663+0000 7f20dc51e700 1 --2- 172.21.15.121:0/1120562345 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f20c006ca20 0x7f20c006eee0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f20c4005fd0 tx=0x7f20c4005f60 comp rx=0 tx=0).stop 2024-04-05T21:26:32.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.663+0000 7f20dc51e700 1 -- 172.21.15.121:0/1120562345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d4100bb0 msgr2=0x7f20d4196270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:32.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.663+0000 7f20dc51e700 1 --2- 172.21.15.121:0/1120562345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d4100bb0 0x7f20d4196270 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f20d000bda0 tx=0x7f20d000be80 comp rx=0 tx=0).stop 2024-04-05T21:26:32.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.663+0000 7f20dc51e700 1 -- 172.21.15.121:0/1120562345 shutdown_connections 2024-04-05T21:26:32.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.663+0000 7f20dc51e700 1 --2- 172.21.15.121:0/1120562345 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f20c006ca20 0x7f20c006eee0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.663+0000 7f20dc51e700 1 --2- 172.21.15.121:0/1120562345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d4100bb0 0x7f20d4196270 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.663+0000 7f20dc51e700 1 --2- 172.21.15.121:0/1120562345 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20d4101db0 0x7f20d41967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.663+0000 7f20dc51e700 1 -- 172.21.15.121:0/1120562345 >> 172.21.15.121:0/1120562345 conn(0x7f20d40fc110 msgr2=0x7f20d4104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:32.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.663+0000 7f20dc51e700 1 -- 172.21.15.121:0/1120562345 shutdown_connections 2024-04-05T21:26:32.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.663+0000 7f20dc51e700 1 -- 172.21.15.121:0/1120562345 wait complete. 2024-04-05T21:26:32.665 INFO:tasks.workunit.client.1.smithi188.stdout:2/392: chown d5/db/dd/d36/d3d/d46/d66/c44 948 1 2024-04-05T21:26:32.668 INFO:tasks.workunit.client.1.smithi188.stdout:3/350: creat d5/d7/d31/f78 x:0 0 0 2024-04-05T21:26:32.669 INFO:tasks.workunit.client.1.smithi188.stdout:7/498: rename d3/d4/c6c to d3/d4/db/d55/ca6 0 2024-04-05T21:26:32.672 INFO:tasks.workunit.client.1.smithi188.stdout:1/382: creat d1/dc/d16/d21/f81 x:0 0 0 2024-04-05T21:26:32.672 INFO:tasks.workunit.client.1.smithi188.stdout:5/388: dwrite d6/f2c [0,4194304] 0 2024-04-05T21:26:32.675 INFO:tasks.workunit.client.1.smithi188.stdout:4/536: unlink d0/d14/d15/c13 0 2024-04-05T21:26:32.678 INFO:tasks.workunit.client.1.smithi188.stdout:3/351: unlink d5/d39/f55 0 2024-04-05T21:26:32.678 INFO:tasks.workunit.client.1.smithi188.stdout:5/389: fsync d6/d17/d23/f55 0 2024-04-05T21:26:32.680 INFO:tasks.workunit.client.1.smithi188.stdout:7/499: write d3/d4/db/f1c [434574,111241] 0 2024-04-05T21:26:32.680 INFO:tasks.workunit.client.1.smithi188.stdout:6/485: dwrite d6/d13/d18/d19/d2c/f51 [0,4194304] 0 2024-04-05T21:26:32.682 INFO:tasks.workunit.client.1.smithi188.stdout:2/393: dwrite d5/f22 [4194304,4194304] 0 2024-04-05T21:26:32.685 INFO:tasks.workunit.client.1.smithi188.stdout:3/352: write d5/d7/d31/f5c [2116600,66026] 0 2024-04-05T21:26:32.686 INFO:tasks.workunit.client.1.smithi188.stdout:8/278: rename d5/f29 to d5/d21/f64 0 2024-04-05T21:26:32.687 INFO:tasks.workunit.client.1.smithi188.stdout:4/537: fdatasync d0/d14/d15/d53/d82/f8a 0 2024-04-05T21:26:32.688 INFO:tasks.workunit.client.1.smithi188.stdout:5/390: write d6/d15/d1b/f1e [559387,20105] 0 2024-04-05T21:26:32.688 INFO:tasks.workunit.client.1.smithi188.stdout:6/486: write d6/d13/f89 [799554,39723] 0 2024-04-05T21:26:32.692 INFO:tasks.workunit.client.1.smithi188.stdout:6/487: chown d6/l12 76584 1 2024-04-05T21:26:32.693 INFO:tasks.workunit.client.1.smithi188.stdout:4/538: readlink d0/d2/d52/l9a 0 2024-04-05T21:26:32.693 INFO:tasks.workunit.client.1.smithi188.stdout:8/279: stat d5/d21/c23 0 2024-04-05T21:26:32.695 INFO:tasks.workunit.client.1.smithi188.stdout:2/394: mkdir d5/db/dd/d36/d3c/d48/d78 0 2024-04-05T21:26:32.697 INFO:tasks.workunit.client.1.smithi188.stdout:6/488: truncate d6/d28/f84 846892 0 2024-04-05T21:26:32.698 INFO:tasks.workunit.client.1.smithi188.stdout:7/500: write d3/d4/db/d2e/d37/d65/f20 [2440499,33789] 0 2024-04-05T21:26:32.698 INFO:tasks.workunit.client.1.smithi188.stdout:3/353: fdatasync d5/df/f15 0 2024-04-05T21:26:32.699 INFO:tasks.workunit.client.1.smithi188.stdout:1/383: dread d1/d4/f1d [0,4194304] 0 2024-04-05T21:26:32.701 INFO:tasks.workunit.client.1.smithi188.stdout:8/280: stat d5/d21/c52 0 2024-04-05T21:26:32.706 INFO:tasks.workunit.client.1.smithi188.stdout:4/539: fdatasync d0/d14/d15/f5b 0 2024-04-05T21:26:32.706 INFO:tasks.workunit.client.1.smithi188.stdout:2/395: write d5/db/dd/d36/d3d/d46/d5d/f5e [2813303,34447] 0 2024-04-05T21:26:32.707 INFO:tasks.workunit.client.1.smithi188.stdout:7/501: fsync d3/d4/db/d2e/f93 0 2024-04-05T21:26:32.708 INFO:tasks.workunit.client.1.smithi188.stdout:8/281: mknod d5/d21/d50/c65 0 2024-04-05T21:26:32.708 INFO:tasks.workunit.client.1.smithi188.stdout:1/384: creat d1/d4/d13/f82 x:0 0 0 2024-04-05T21:26:32.709 INFO:tasks.workunit.client.1.smithi188.stdout:2/396: stat d5/db/dd 0 2024-04-05T21:26:32.710 INFO:tasks.workunit.client.1.smithi188.stdout:5/391: getdents d6/d17/d39 0 2024-04-05T21:26:32.710 INFO:tasks.workunit.client.1.smithi188.stdout:8/282: dread - d5/f3d zero size 2024-04-05T21:26:32.711 INFO:tasks.workunit.client.1.smithi188.stdout:1/385: mknod d1/dc/d16/d2d/c83 0 2024-04-05T21:26:32.715 INFO:tasks.workunit.client.1.smithi188.stdout:4/540: creat d0/d2/d17/d48/d70/fc0 x:0 0 0 2024-04-05T21:26:32.715 INFO:tasks.workunit.client.1.smithi188.stdout:5/392: creat d6/d15/d5c/f94 x:0 0 0 2024-04-05T21:26:32.721 INFO:tasks.workunit.client.1.smithi188.stdout:4/541: mkdir d0/d2/d52/dc1 0 2024-04-05T21:26:32.724 INFO:tasks.workunit.client.1.smithi188.stdout:3/354: dwrite d5/d7/d65/f44 [0,4194304] 0 2024-04-05T21:26:32.726 INFO:tasks.workunit.client.1.smithi188.stdout:5/393: write d6/d15/d5c/f94 [325994,26710] 0 2024-04-05T21:26:32.733 INFO:tasks.workunit.client.1.smithi188.stdout:2/397: dread d5/db/d35/f43 [0,4194304] 0 2024-04-05T21:26:32.733 INFO:tasks.workunit.client.1.smithi188.stdout:3/355: chown d5/d35/f40 186 1 2024-04-05T21:26:32.733 INFO:tasks.workunit.client.1.smithi188.stdout:7/502: dwrite d3/d4/d1d/f34 [0,4194304] 0 2024-04-05T21:26:32.733 INFO:tasks.workunit.client.1.smithi188.stdout:8/283: dwrite d5/f9 [4194304,4194304] 0 2024-04-05T21:26:32.735 INFO:tasks.workunit.client.1.smithi188.stdout:5/394: fdatasync d6/f2c 0 2024-04-05T21:26:32.737 INFO:tasks.workunit.client.1.smithi188.stdout:8/284: write d5/f35 [890166,123739] 0 2024-04-05T21:26:32.737 INFO:tasks.workunit.client.1.smithi188.stdout:4/542: creat d0/fc2 x:0 0 0 2024-04-05T21:26:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:32 smithi121 ceph-mon[39723]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:32.739 INFO:tasks.workunit.client.1.smithi188.stdout:5/395: fdatasync d6/d17/d23/f55 0 2024-04-05T21:26:32.741 INFO:tasks.workunit.client.1.smithi188.stdout:3/356: creat d5/d2b/f79 x:0 0 0 2024-04-05T21:26:32.744 INFO:tasks.workunit.client.1.smithi188.stdout:2/398: symlink d5/db/dd/d36/d3c/d63/d75/l79 0 2024-04-05T21:26:32.745 INFO:tasks.workunit.client.1.smithi188.stdout:1/386: dwrite d1/dc/d16/d21/f6e [0,4194304] 0 2024-04-05T21:26:32.745 INFO:tasks.workunit.client.1.smithi188.stdout:4/543: chown d0/d2/d2a/d34/d6b/c8b 19174 1 2024-04-05T21:26:32.748 INFO:tasks.workunit.client.1.smithi188.stdout:3/357: dread - d5/dc/f2d zero size 2024-04-05T21:26:32.749 INFO:tasks.workunit.client.1.smithi188.stdout:5/396: unlink d6/d17/d39/f86 0 2024-04-05T21:26:32.752 INFO:tasks.workunit.client.1.smithi188.stdout:3/358: chown d5/d7/f25 71 1 2024-04-05T21:26:32.753 INFO:tasks.workunit.client.1.smithi188.stdout:1/387: mknod d1/c84 0 2024-04-05T21:26:32.755 INFO:tasks.workunit.client.1.smithi188.stdout:5/397: mkdir d6/d82/d40/d42/d81/d95 0 2024-04-05T21:26:32.756 INFO:tasks.workunit.client.1.smithi188.stdout:3/359: write d5/df/f71 [896631,94332] 0 2024-04-05T21:26:32.758 INFO:tasks.workunit.client.1.smithi188.stdout:8/285: dwrite d5/f8 [0,4194304] 0 2024-04-05T21:26:32.759 INFO:tasks.workunit.client.1.smithi188.stdout:1/388: write d1/dc/f11 [366929,24593] 0 2024-04-05T21:26:32.761 INFO:tasks.workunit.client.1.smithi188.stdout:3/360: chown d5/d2b/c42 760572 1 2024-04-05T21:26:32.762 INFO:tasks.workunit.client.1.smithi188.stdout:7/503: dwrite d3/d4/db/d2e/f75 [0,4194304] 0 2024-04-05T21:26:32.763 INFO:tasks.workunit.client.1.smithi188.stdout:5/398: truncate d6/d82/d40/f65 5089414 0 2024-04-05T21:26:32.771 INFO:tasks.workunit.client.1.smithi188.stdout:2/399: dwrite d5/db/dd/d24/f2e [4194304,4194304] 0 2024-04-05T21:26:32.775 INFO:tasks.workunit.client.1.smithi188.stdout:1/389: chown d1/dc/d16/d21/d27/f75 104 1 2024-04-05T21:26:32.775 INFO:tasks.workunit.client.1.smithi188.stdout:1/390: stat d1/dc/d16/d21/d27/f29 0 2024-04-05T21:26:32.778 INFO:tasks.workunit.client.1.smithi188.stdout:7/504: dread d3/f69 [0,4194304] 0 2024-04-05T21:26:32.779 INFO:tasks.workunit.client.1.smithi188.stdout:1/391: write d1/dc/f28 [4727876,65094] 0 2024-04-05T21:26:32.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:32 smithi188 ceph-mon[49715]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:32.780 INFO:tasks.workunit.client.1.smithi188.stdout:2/400: mkdir d5/db/dd/d36/d3c/d48/d7a 0 2024-04-05T21:26:32.783 INFO:tasks.workunit.client.1.smithi188.stdout:2/401: fdatasync d5/db/dd/d24/f52 0 2024-04-05T21:26:32.783 INFO:tasks.workunit.client.1.smithi188.stdout:3/361: dwrite d5/d7/d31/f5c [4194304,4194304] 0 2024-04-05T21:26:32.784 INFO:tasks.workunit.client.1.smithi188.stdout:1/392: mknod d1/d4/d13/d60/c85 0 2024-04-05T21:26:32.785 INFO:tasks.workunit.client.1.smithi188.stdout:2/402: fsync d5/db/dd/d36/d3c/f4a 0 2024-04-05T21:26:32.788 INFO:tasks.workunit.client.1.smithi188.stdout:2/403: stat d5/db/l2c 0 2024-04-05T21:26:32.788 INFO:tasks.workunit.client.1.smithi188.stdout:7/505: truncate d3/d4/db/d10/f14 9105995 0 2024-04-05T21:26:32.788 INFO:tasks.workunit.client.1.smithi188.stdout:3/362: creat d5/d7/d6d/f7a x:0 0 0 2024-04-05T21:26:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.790+0000 7f7b94a22700 1 -- 172.21.15.121:0/385662451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b90103f70 msgr2=0x7f7b90106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.790+0000 7f7b94a22700 1 --2- 172.21.15.121:0/385662451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b90103f70 0x7f7b90106360 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009910 tx=0x7f7b84009c20 comp rx=0 tx=0).stop 2024-04-05T21:26:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.790+0000 7f7b94a22700 1 -- 172.21.15.121:0/385662451 shutdown_connections 2024-04-05T21:26:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.790+0000 7f7b94a22700 1 --2- 172.21.15.121:0/385662451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b90103f70 0x7f7b90106360 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.790+0000 7f7b94a22700 1 --2- 172.21.15.121:0/385662451 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b90101640 0x7f7b90103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.790+0000 7f7b94a22700 1 -- 172.21.15.121:0/385662451 >> 172.21.15.121:0/385662451 conn(0x7f7b900fb050 msgr2=0x7f7b900fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.790+0000 7f7b94a22700 1 -- 172.21.15.121:0/385662451 shutdown_connections 2024-04-05T21:26:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.790+0000 7f7b94a22700 1 -- 172.21.15.121:0/385662451 wait complete. 2024-04-05T21:26:32.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.791+0000 7f7b94a22700 1 Processor -- start 2024-04-05T21:26:32.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.791+0000 7f7b94a22700 1 -- start start 2024-04-05T21:26:32.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.791+0000 7f7b94a22700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b90101640 0x7f7b90196350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.791+0000 7f7b94a22700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b90103f70 0x7f7b90196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.791+0000 7f7b94a22700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b90196eb0 con 0x7f7b90103f70 2024-04-05T21:26:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.791+0000 7f7b94a22700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b90196ff0 con 0x7f7b90101640 2024-04-05T21:26:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.791+0000 7f7b8dd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b90103f70 0x7f7b90196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.791+0000 7f7b8dd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b90103f70 0x7f7b90196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37500/0 (socket says 172.21.15.121:37500) 2024-04-05T21:26:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.791+0000 7f7b8dd9b700 1 -- 172.21.15.121:0/4226047468 learned_addr learned my addr 172.21.15.121:0/4226047468 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:26:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.791+0000 7f7b8e59c700 1 --2- 172.21.15.121:0/4226047468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b90101640 0x7f7b90196350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.791+0000 7f7b8dd9b700 1 -- 172.21.15.121:0/4226047468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b90101640 msgr2=0x7f7b90196350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.791+0000 7f7b8dd9b700 1 --2- 172.21.15.121:0/4226047468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b90101640 0x7f7b90196350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.791+0000 7f7b8dd9b700 1 -- 172.21.15.121:0/4226047468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b840095f0 con 0x7f7b90103f70 2024-04-05T21:26:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.792+0000 7f7b8dd9b700 1 --2- 172.21.15.121:0/4226047468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b90103f70 0x7f7b90196890 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7b84006010 tx=0x7f7b8400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:32.794 INFO:tasks.workunit.client.1.smithi188.stdout:7/506: chown d3/d4/db/d2e/d3b/f92 1011289 1 2024-04-05T21:26:32.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.792+0000 7f7b7f7fe700 1 -- 172.21.15.121:0/4226047468 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b8401d070 con 0x7f7b90103f70 2024-04-05T21:26:32.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.792+0000 7f7b7f7fe700 1 -- 172.21.15.121:0/4226047468 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f7b84022470 con 0x7f7b90103f70 2024-04-05T21:26:32.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.792+0000 7f7b94a22700 1 -- 172.21.15.121:0/4226047468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b9019ba40 con 0x7f7b90103f70 2024-04-05T21:26:32.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.792+0000 7f7b7f7fe700 1 -- 172.21.15.121:0/4226047468 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b8400f690 con 0x7f7b90103f70 2024-04-05T21:26:32.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.792+0000 7f7b94a22700 1 -- 172.21.15.121:0/4226047468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b9019bfb0 con 0x7f7b90103f70 2024-04-05T21:26:32.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.794+0000 7f7b7f7fe700 1 -- 172.21.15.121:0/4226047468 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 25) v1 ==== 90681+0+0 (secure 0 0 0) 0x7f7b84004af0 con 0x7f7b90103f70 2024-04-05T21:26:32.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.794+0000 7f7b94a22700 1 -- 172.21.15.121:0/4226047468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b90190460 con 0x7f7b90103f70 2024-04-05T21:26:32.797 INFO:tasks.workunit.client.1.smithi188.stdout:3/363: creat d5/d7/d65/d61/f7b x:0 0 0 2024-04-05T21:26:32.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.794+0000 7f7b7f7fe700 1 --2- 172.21.15.121:0/4226047468 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f7b7806ca50 0x7f7b7806ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:32.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.794+0000 7f7b7f7fe700 1 -- 172.21.15.121:0/4226047468 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f7b8408d750 con 0x7f7b90103f70 2024-04-05T21:26:32.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.795+0000 7f7b8e59c700 1 --2- 172.21.15.121:0/4226047468 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f7b7806ca50 0x7f7b7806ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:32.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.795+0000 7f7b8e59c700 1 --2- 172.21.15.121:0/4226047468 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f7b7806ca50 0x7f7b7806ef10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7b80006fd0 tx=0x7f7b80008040 comp rx=0 tx=0).ready entity=mgr.24327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:32.799 INFO:tasks.workunit.client.1.smithi188.stdout:4/544: dread d0/d2/d17/f4e [0,4194304] 0 2024-04-05T21:26:32.800 INFO:tasks.workunit.client.1.smithi188.stdout:3/364: readlink d5/dc/l23 0 2024-04-05T21:26:32.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:32.799+0000 7f7b7f7fe700 1 -- 172.21.15.121:0/4226047468 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f7b8405b470 con 0x7f7b90103f70 2024-04-05T21:26:32.801 INFO:tasks.workunit.client.1.smithi188.stdout:1/393: dwrite d1/dc/d16/d21/f6e [4194304,4194304] 0 2024-04-05T21:26:32.803 INFO:tasks.workunit.client.1.smithi188.stdout:1/394: chown d1/fb 483238 1 2024-04-05T21:26:32.805 INFO:tasks.workunit.client.1.smithi188.stdout:3/365: symlink d5/d7/d6d/l7c 0 2024-04-05T21:26:32.809 INFO:tasks.workunit.client.1.smithi188.stdout:2/404: dwrite d5/db/f76 [0,4194304] 0 2024-04-05T21:26:32.811 INFO:tasks.workunit.client.1.smithi188.stdout:3/366: read d5/d7/d31/f6f [680264,75620] 0 2024-04-05T21:26:32.813 INFO:tasks.workunit.client.1.smithi188.stdout:8/286: dread d5/f1c [0,4194304] 0 2024-04-05T21:26:32.817 INFO:tasks.workunit.client.1.smithi188.stdout:0/223: sync 2024-04-05T21:26:32.817 INFO:tasks.workunit.client.1.smithi188.stdout:9/380: sync 2024-04-05T21:26:32.817 INFO:tasks.workunit.client.1.smithi188.stdout:3/367: write d5/f41 [1982400,101847] 0 2024-04-05T21:26:32.821 INFO:tasks.workunit.client.1.smithi188.stdout:2/405: write d5/fa [872009,61785] 0 2024-04-05T21:26:32.825 INFO:tasks.workunit.client.1.smithi188.stdout:4/545: dwrite d0/d14/d2f/fb8 [0,4194304] 0 2024-04-05T21:26:32.825 INFO:tasks.workunit.client.1.smithi188.stdout:0/224: creat d4/d6/d13/d23/f51 x:0 0 0 2024-04-05T21:26:32.825 INFO:tasks.workunit.client.1.smithi188.stdout:8/287: creat d5/d21/d40/d43/f66 x:0 0 0 2024-04-05T21:26:32.830 INFO:tasks.workunit.client.1.smithi188.stdout:3/368: creat d5/f7d x:0 0 0 2024-04-05T21:26:32.831 INFO:tasks.workunit.client.1.smithi188.stdout:1/395: dwrite d1/dc/d16/d21/d27/f29 [4194304,4194304] 0 2024-04-05T21:26:32.837 INFO:tasks.workunit.client.1.smithi188.stdout:2/406: unlink d5/db/dd/d36/d3d/d46/d66/c70 0 2024-04-05T21:26:32.837 INFO:tasks.workunit.client.1.smithi188.stdout:8/288: rmdir d5/d21/d40/d43/d44/d5b 39 2024-04-05T21:26:32.838 INFO:tasks.workunit.client.1.smithi188.stdout:7/507: dread d3/d4/db/d2e/f35 [0,4194304] 0 2024-04-05T21:26:32.839 INFO:tasks.workunit.client.1.smithi188.stdout:4/546: chown d0/d14/d15/d53/l65 0 1 2024-04-05T21:26:32.841 INFO:tasks.workunit.client.1.smithi188.stdout:8/289: read d5/f8 [725039,82215] 0 2024-04-05T21:26:32.841 INFO:tasks.workunit.client.1.smithi188.stdout:3/369: truncate d5/f21 563534 0 2024-04-05T21:26:32.842 INFO:tasks.workunit.client.1.smithi188.stdout:3/370: dread - d5/d7/d65/d5f/f64 zero size 2024-04-05T21:26:32.844 INFO:tasks.workunit.client.1.smithi188.stdout:9/381: getdents dd/d3d 0 2024-04-05T21:26:32.844 INFO:tasks.workunit.client.1.smithi188.stdout:1/396: mkdir d1/d4/d86 0 2024-04-05T21:26:32.845 INFO:tasks.workunit.client.1.smithi188.stdout:2/407: creat d5/db/dd/d36/d3d/d46/f7b x:0 0 0 2024-04-05T21:26:32.846 INFO:tasks.workunit.client.1.smithi188.stdout:0/225: dwrite d4/d8/db/d21/f32 [0,4194304] 0 2024-04-05T21:26:32.849 INFO:tasks.workunit.client.1.smithi188.stdout:4/547: unlink d0/d14/d15/f89 0 2024-04-05T21:26:32.851 INFO:tasks.workunit.client.1.smithi188.stdout:3/371: mkdir d5/d7e 0 2024-04-05T21:26:32.853 INFO:tasks.workunit.client.1.smithi188.stdout:1/397: write d1/f14 [4773124,116721] 0 2024-04-05T21:26:32.853 INFO:tasks.workunit.client.1.smithi188.stdout:0/226: chown d4/d8/d10/f1b 1124 1 2024-04-05T21:26:32.857 INFO:tasks.workunit.client.1.smithi188.stdout:4/548: write d0/d14/f86 [4036696,37476] 0 2024-04-05T21:26:32.857 INFO:tasks.workunit.client.1.smithi188.stdout:9/382: write dd/d11/d1f/d25/f50 [3434814,46648] 0 2024-04-05T21:26:32.857 INFO:tasks.workunit.client.1.smithi188.stdout:2/408: rmdir d5/d5f 39 2024-04-05T21:26:32.861 INFO:tasks.workunit.client.1.smithi188.stdout:8/290: creat d5/d21/f67 x:0 0 0 2024-04-05T21:26:32.861 INFO:tasks.workunit.client.1.smithi188.stdout:3/372: mknod d5/d7/c7f 0 2024-04-05T21:26:32.863 INFO:tasks.workunit.client.1.smithi188.stdout:0/227: read - d4/d8/f3d zero size 2024-04-05T21:26:32.865 INFO:tasks.workunit.client.1.smithi188.stdout:2/409: write d5/db/f16 [6615847,22743] 0 2024-04-05T21:26:32.865 INFO:tasks.workunit.client.1.smithi188.stdout:9/383: write dd/d35/f67 [1035068,36426] 0 2024-04-05T21:26:32.865 INFO:tasks.workunit.client.1.smithi188.stdout:7/508: dwrite d3/d4/db/d2e/f51 [0,4194304] 0 2024-04-05T21:26:32.869 INFO:tasks.workunit.client.1.smithi188.stdout:3/373: dread - d5/dc/f2d zero size 2024-04-05T21:26:32.873 INFO:tasks.workunit.client.1.smithi188.stdout:1/398: dwrite d1/dc/f28 [0,4194304] 0 2024-04-05T21:26:32.873 INFO:tasks.workunit.client.1.smithi188.stdout:4/549: mkdir d0/d2/d52/dc1/dc3 0 2024-04-05T21:26:32.876 INFO:tasks.workunit.client.1.smithi188.stdout:4/550: chown d0/d14/d2f/l5f 124278 1 2024-04-05T21:26:32.877 INFO:tasks.workunit.client.1.smithi188.stdout:0/228: creat d4/d6/d13/d23/f52 x:0 0 0 2024-04-05T21:26:32.878 INFO:tasks.workunit.client.1.smithi188.stdout:3/374: mkdir d5/df/d80 0 2024-04-05T21:26:32.881 INFO:tasks.workunit.client.1.smithi188.stdout:4/551: chown d0/d2/d2a/d71/fb6 97420445 1 2024-04-05T21:26:32.881 INFO:tasks.workunit.client.1.smithi188.stdout:9/384: dread dd/d49/f52 [0,4194304] 0 2024-04-05T21:26:32.884 INFO:tasks.workunit.client.1.smithi188.stdout:8/291: link d5/fb d5/d21/d40/d43/d44/f68 0 2024-04-05T21:26:32.886 INFO:tasks.workunit.client.1.smithi188.stdout:3/375: truncate d5/d7/d31/f76 384887 0 2024-04-05T21:26:32.887 INFO:tasks.workunit.client.1.smithi188.stdout:2/410: dwrite d5/db/dd/d24/f2e [4194304,4194304] 0 2024-04-05T21:26:32.890 INFO:tasks.workunit.client.1.smithi188.stdout:0/229: rename d4/fe to d4/f53 0 2024-04-05T21:26:32.892 INFO:tasks.workunit.client.1.smithi188.stdout:4/552: chown d0/d14/d15/d42/fa3 217932 1 2024-04-05T21:26:32.998 INFO:tasks.workunit.client.1.smithi188.stdout:3/376: mknod d5/d7/d65/d61/d66/d6e/c81 0 2024-04-05T21:26:33.000 INFO:tasks.workunit.client.1.smithi188.stdout:2/411: mkdir d5/db/dd/d36/d3c/d48/d7a/d7c 0 2024-04-05T21:26:33.000 INFO:tasks.workunit.client.1.smithi188.stdout:9/385: dwrite dd/d49/d6b/d40/f4e [4194304,4194304] 0 2024-04-05T21:26:33.001 INFO:tasks.workunit.client.1.smithi188.stdout:4/553: mknod d0/d14/d15/d53/d82/cc4 0 2024-04-05T21:26:33.003 INFO:tasks.workunit.client.1.smithi188.stdout:3/377: rmdir d5/d7/d65 39 2024-04-05T21:26:33.007 INFO:tasks.workunit.client.1.smithi188.stdout:2/412: rename d5/db/dd/d36/d3c/d48 to d5/db/dd/d36/d3c/d48/d78/d7d 22 2024-04-05T21:26:33.008 INFO:tasks.workunit.client.1.smithi188.stdout:0/230: dread d4/d8/db/d21/f34 [0,4194304] 0 2024-04-05T21:26:33.008 INFO:tasks.workunit.client.1.smithi188.stdout:9/386: getdents dd/d3d 0 2024-04-05T21:26:33.009 INFO:tasks.workunit.client.1.smithi188.stdout:7/509: dread d3/d2c/d5e/f98 [0,4194304] 0 2024-04-05T21:26:33.010 INFO:tasks.workunit.client.1.smithi188.stdout:8/292: dwrite d5/dc/f36 [0,4194304] 0 2024-04-05T21:26:33.012 INFO:tasks.workunit.client.1.smithi188.stdout:4/554: fsync d0/d14/d15/fb 0 2024-04-05T21:26:33.014 INFO:tasks.workunit.client.1.smithi188.stdout:9/387: read dd/d11/d22/f37 [498324,35100] 0 2024-04-05T21:26:33.020 INFO:tasks.workunit.client.1.smithi188.stdout:0/231: truncate d4/d6/d13/d23/f52 466427 0 2024-04-05T21:26:33.020 INFO:tasks.workunit.client.1.smithi188.stdout:4/555: readlink d0/d14/d15/d78/l91 0 2024-04-05T21:26:33.020 INFO:tasks.workunit.client.1.smithi188.stdout:2/413: rename d5/db/d35/d3b/f5a to d5/db/d35/d3b/d6b/f7e 0 2024-04-05T21:26:33.022 INFO:tasks.workunit.client.1.smithi188.stdout:9/388: mknod dd/d35/c7c 0 2024-04-05T21:26:33.022 INFO:tasks.workunit.client.1.smithi188.stdout:8/293: dread d5/d21/d40/f57 [0,4194304] 0 2024-04-05T21:26:33.022 INFO:tasks.workunit.client.1.smithi188.stdout:3/378: dwrite d5/fa [0,4194304] 0 2024-04-05T21:26:33.028 INFO:tasks.workunit.client.1.smithi188.stdout:2/414: write d5/db/dd/d24/f2e [3328173,56132] 0 2024-04-05T21:26:33.030 INFO:tasks.workunit.client.1.smithi188.stdout:7/510: dwrite d3/d4/db/fe [4194304,4194304] 0 2024-04-05T21:26:33.031 INFO:tasks.workunit.client.1.smithi188.stdout:9/389: fsync dd/d11/d1f/d25/f44 0 2024-04-05T21:26:33.033 INFO:tasks.workunit.client.1.smithi188.stdout:8/294: mkdir d5/d21/d50/d63/d69 0 2024-04-05T21:26:33.036 INFO:tasks.workunit.client.1.smithi188.stdout:3/379: mkdir d5/d7/d6d/d75/d82 0 2024-04-05T21:26:33.037 INFO:tasks.workunit.client.1.smithi188.stdout:9/390: stat dd/d49/d6b/d40/d59/c71 0 2024-04-05T21:26:33.042 INFO:tasks.workunit.client.1.smithi188.stdout:8/295: creat d5/d21/f6a x:0 0 0 2024-04-05T21:26:33.042 INFO:tasks.workunit.client.1.smithi188.stdout:3/380: mkdir d5/d7/d6d/d75/d83 0 2024-04-05T21:26:33.046 INFO:tasks.workunit.client.1.smithi188.stdout:4/556: dread d0/d2/d2a/d34/f41 [0,4194304] 0 2024-04-05T21:26:33.046 INFO:tasks.workunit.client.1.smithi188.stdout:0/232: dwrite d4/d8/db/f1d [4194304,4194304] 0 2024-04-05T21:26:33.046 INFO:tasks.workunit.client.1.smithi188.stdout:7/511: getdents d3/d4/d1d/d74/d73 0 2024-04-05T21:26:33.048 INFO:tasks.workunit.client.1.smithi188.stdout:3/381: mkdir d5/d39/d84 0 2024-04-05T21:26:33.050 INFO:tasks.workunit.client.1.smithi188.stdout:8/296: creat d5/d21/d50/f6b x:0 0 0 2024-04-05T21:26:33.054 INFO:tasks.workunit.client.1.smithi188.stdout:9/391: dwrite fc [4194304,4194304] 0 2024-04-05T21:26:33.054 INFO:tasks.workunit.client.1.smithi188.stdout:0/233: write d4/d8/f26 [42689,4315] 0 2024-04-05T21:26:33.056 INFO:tasks.workunit.client.1.smithi188.stdout:7/512: mknod d3/d2c/ca7 0 2024-04-05T21:26:33.056 INFO:tasks.workunit.client.1.smithi188.stdout:8/297: read - d5/d21/d50/f6b zero size 2024-04-05T21:26:33.057 INFO:tasks.workunit.client.1.smithi188.stdout:4/557: mkdir d0/d2/d52/dc1/dc3/dc5 0 2024-04-05T21:26:33.060 INFO:tasks.workunit.client.1.smithi188.stdout:0/234: dread - d4/d8/f3d zero size 2024-04-05T21:26:33.061 INFO:tasks.workunit.client.1.smithi188.stdout:3/382: unlink d5/d7/d31/f6f 0 2024-04-05T21:26:33.061 INFO:tasks.workunit.client.1.smithi188.stdout:2/415: dwrite d5/db/d35/d3b/f4d [0,4194304] 0 2024-04-05T21:26:33.062 INFO:tasks.workunit.client.1.smithi188.stdout:8/298: dread d5/dc/f1d [0,4194304] 0 2024-04-05T21:26:33.063 INFO:tasks.workunit.client.1.smithi188.stdout:7/513: fsync d3/d4/db/d2e/d37/f50 0 2024-04-05T21:26:33.063 INFO:tasks.workunit.client.1.smithi188.stdout:0/235: stat d4/d6/l4f 0 2024-04-05T21:26:33.064 INFO:tasks.workunit.client.1.smithi188.stdout:9/392: truncate dd/d11/f76 806727 0 2024-04-05T21:26:33.069 INFO:tasks.workunit.client.1.smithi188.stdout:4/558: symlink d0/d14/d15/d78/db1/lc6 0 2024-04-05T21:26:33.074 INFO:tasks.workunit.client.1.smithi188.stdout:0/236: rename d4/d8/db/f1d to d4/d8/db/d42/d38/f54 0 2024-04-05T21:26:33.075 INFO:tasks.workunit.client.1.smithi188.stdout:7/514: dread - d3/d4/d1d/d74/d73/f7a zero size 2024-04-05T21:26:33.079 INFO:tasks.workunit.client.1.smithi188.stdout:8/299: getdents d5/d21/d40/d43/d44/d5b 0 2024-04-05T21:26:33.082 INFO:tasks.workunit.client.1.smithi188.stdout:9/393: dwrite dd/d11/d22/f5a [0,4194304] 0 2024-04-05T21:26:33.084 INFO:tasks.workunit.client.1.smithi188.stdout:4/559: dread d0/d14/f80 [0,4194304] 0 2024-04-05T21:26:33.085 INFO:tasks.workunit.client.1.smithi188.stdout:7/515: unlink d3/d2c/d5e/f66 0 2024-04-05T21:26:33.087 INFO:tasks.workunit.client.1.smithi188.stdout:2/416: dwrite d5/db/dd/d36/d3d/d46/f65 [0,4194304] 0 2024-04-05T21:26:33.087 INFO:tasks.workunit.client.1.smithi188.stdout:7/516: chown d3/d4/d7c 29262 1 2024-04-05T21:26:33.088 INFO:tasks.workunit.client.1.smithi188.stdout:8/300: dread - d5/d21/d40/d43/f62 zero size 2024-04-05T21:26:33.088 INFO:tasks.workunit.client.1.smithi188.stdout:3/383: dwrite d5/d7/d65/f54 [0,4194304] 0 2024-04-05T21:26:33.092 INFO:tasks.workunit.client.1.smithi188.stdout:0/237: link d4/d8/db/d42/d38/f49 d4/d6/d13/f55 0 2024-04-05T21:26:33.097 INFO:tasks.workunit.client.1.smithi188.stdout:7/517: unlink d3/c6a 0 2024-04-05T21:26:33.100 INFO:tasks.workunit.client.1.smithi188.stdout:4/560: mkdir d0/d2/dc7 0 2024-04-05T21:26:33.101 INFO:tasks.workunit.client.1.smithi188.stdout:4/561: chown d0/d2/d52/f79 26932 1 2024-04-05T21:26:33.104 INFO:tasks.workunit.client.1.smithi188.stdout:3/384: symlink d5/d7/d65/d61/d66/d6e/l85 0 2024-04-05T21:26:33.107 INFO:tasks.workunit.client.1.smithi188.stdout:7/518: rmdir d3/d4/db/d2e/d49 39 2024-04-05T21:26:33.108 INFO:tasks.workunit.client.1.smithi188.stdout:7/519: fsync d3/d4/db/d2e/d37/d65/f20 0 2024-04-05T21:26:33.110 INFO:tasks.workunit.client.1.smithi188.stdout:4/562: fsync d0/d14/d15/d53/f9f 0 2024-04-05T21:26:33.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.113+0000 7f7b94a22700 1 -- 172.21.15.121:0/4226047468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7b9002ce30 con 0x7f7b90103f70 2024-04-05T21:26:33.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.114+0000 7f7b7f7fe700 1 -- 172.21.15.121:0/4226047468 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+770 (secure 0 0 0) 0x7f7b8405b000 con 0x7f7b90103f70 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 1 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:26:33.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:26:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:26:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 13, 2024-04-05T21:26:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 1 2024-04-05T21:26:33.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:26:33.116 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:26:33.116 INFO:tasks.workunit.client.1.smithi188.stdout:4/563: readlink d0/d2/d2a/d34/d6b/l9b 0 2024-04-05T21:26:33.117 INFO:tasks.workunit.client.1.smithi188.stdout:2/417: link d5/d5f/l60 d5/db/dd/d36/d3d/d46/l7f 0 2024-04-05T21:26:33.117 INFO:tasks.workunit.client.1.smithi188.stdout:7/520: dread d3/d4/db/d2e/d49/f5d [0,4194304] 0 2024-04-05T21:26:33.117 INFO:tasks.workunit.client.1.smithi188.stdout:8/301: getdents d5/d21/d50/d63 0 2024-04-05T21:26:33.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.116+0000 7f7b94a22700 1 -- 172.21.15.121:0/4226047468 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f7b7806ca50 msgr2=0x7f7b7806ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:33.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.116+0000 7f7b94a22700 1 --2- 172.21.15.121:0/4226047468 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f7b7806ca50 0x7f7b7806ef10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7b80006fd0 tx=0x7f7b80008040 comp rx=0 tx=0).stop 2024-04-05T21:26:33.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.116+0000 7f7b94a22700 1 -- 172.21.15.121:0/4226047468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b90103f70 msgr2=0x7f7b90196890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:33.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.116+0000 7f7b94a22700 1 --2- 172.21.15.121:0/4226047468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b90103f70 0x7f7b90196890 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7b84006010 tx=0x7f7b8400be20 comp rx=0 tx=0).stop 2024-04-05T21:26:33.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.116+0000 7f7b94a22700 1 -- 172.21.15.121:0/4226047468 shutdown_connections 2024-04-05T21:26:33.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.116+0000 7f7b94a22700 1 --2- 172.21.15.121:0/4226047468 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f7b7806ca50 0x7f7b7806ef10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:33.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.116+0000 7f7b94a22700 1 --2- 172.21.15.121:0/4226047468 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b90101640 0x7f7b90196350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:33.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.116+0000 7f7b94a22700 1 --2- 172.21.15.121:0/4226047468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b90103f70 0x7f7b90196890 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:33.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.116+0000 7f7b94a22700 1 -- 172.21.15.121:0/4226047468 >> 172.21.15.121:0/4226047468 conn(0x7f7b900fb050 msgr2=0x7f7b900fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:33.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.116+0000 7f7b94a22700 1 -- 172.21.15.121:0/4226047468 shutdown_connections 2024-04-05T21:26:33.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.116+0000 7f7b94a22700 1 -- 172.21.15.121:0/4226047468 wait complete. 2024-04-05T21:26:33.120 INFO:tasks.workunit.client.1.smithi188.stdout:0/238: dread d4/d8/f26 [0,4194304] 0 2024-04-05T21:26:33.123 INFO:tasks.workunit.client.1.smithi188.stdout:8/302: write d5/f24 [310193,25890] 0 2024-04-05T21:26:33.128 INFO:tasks.workunit.client.1.smithi188.stdout:7/521: write d3/d2c/d5e/d8e/f9f [19312,35823] 0 2024-04-05T21:26:33.128 INFO:tasks.workunit.client.1.smithi188.stdout:0/239: creat d4/d8/db/d21/f56 x:0 0 0 2024-04-05T21:26:33.131 INFO:tasks.workunit.client.1.smithi188.stdout:4/564: dwrite d0/d14/d15/d53/d82/f8a [0,4194304] 0 2024-04-05T21:26:33.131 INFO:tasks.workunit.client.1.smithi188.stdout:9/394: dread dd/d11/d1f/d25/f50 [0,4194304] 0 2024-04-05T21:26:33.133 INFO:tasks.workunit.client.1.smithi188.stdout:0/240: unlink f1 0 2024-04-05T21:26:33.135 INFO:tasks.workunit.client.1.smithi188.stdout:9/395: symlink dd/d49/d6b/d40/d59/l7d 0 2024-04-05T21:26:33.137 INFO:tasks.workunit.client.1.smithi188.stdout:3/385: dread d5/d7/d31/f5c [0,4194304] 0 2024-04-05T21:26:33.138 INFO:tasks.workunit.client.1.smithi188.stdout:0/241: mkdir d4/d6/d13/d23/d24/d57 0 2024-04-05T21:26:33.138 INFO:tasks.workunit.client.1.smithi188.stdout:9/396: mkdir dd/d11/d22/d7e 0 2024-04-05T21:26:33.140 INFO:tasks.workunit.client.1.smithi188.stdout:9/397: stat dd/d11/d1f/d25/f50 0 2024-04-05T21:26:33.142 INFO:tasks.workunit.client.1.smithi188.stdout:4/565: dwrite d0/d14/f38 [0,4194304] 0 2024-04-05T21:26:33.143 INFO:tasks.workunit.client.1.smithi188.stdout:0/242: creat d4/d8/db/d21/d4a/f58 x:0 0 0 2024-04-05T21:26:33.147 INFO:tasks.workunit.client.1.smithi188.stdout:9/398: creat dd/d49/d6b/d40/d59/f7f x:0 0 0 2024-04-05T21:26:33.147 INFO:tasks.workunit.client.1.smithi188.stdout:0/243: creat d4/d6/d46/f59 x:0 0 0 2024-04-05T21:26:33.151 INFO:tasks.workunit.client.1.smithi188.stdout:4/566: creat d0/d2/fc8 x:0 0 0 2024-04-05T21:26:33.153 INFO:tasks.workunit.client.1.smithi188.stdout:4/567: chown d0/d2/fbb 1130422 1 2024-04-05T21:26:33.154 INFO:tasks.workunit.client.1.smithi188.stdout:0/244: rename d4/d8/d10/l1a to d4/d8/db/d21/d3c/l5a 0 2024-04-05T21:26:33.155 INFO:tasks.workunit.client.1.smithi188.stdout:4/568: truncate d0/f62 1321823 0 2024-04-05T21:26:33.156 INFO:tasks.workunit.client.1.smithi188.stdout:3/386: dwrite d5/d7/d65/f54 [0,4194304] 0 2024-04-05T21:26:33.160 INFO:tasks.workunit.client.1.smithi188.stdout:9/399: dwrite dd/d11/d1f/f29 [0,4194304] 0 2024-04-05T21:26:33.164 INFO:tasks.workunit.client.1.smithi188.stdout:0/245: mknod d4/d8/db/d21/d4a/c5b 0 2024-04-05T21:26:33.172 INFO:tasks.workunit.client.1.smithi188.stdout:0/246: fdatasync d4/d8/f4d 0 2024-04-05T21:26:33.172 INFO:tasks.workunit.client.1.smithi188.stdout:9/400: fdatasync dd/d11/d1f/d25/f44 0 2024-04-05T21:26:33.174 INFO:tasks.workunit.client.1.smithi188.stdout:0/247: stat d4/d8/db/f28 0 2024-04-05T21:26:33.174 INFO:tasks.workunit.client.1.smithi188.stdout:4/569: dwrite d0/d2/d52/f79 [0,4194304] 0 2024-04-05T21:26:33.182 INFO:tasks.workunit.client.1.smithi188.stdout:9/401: read dd/d11/d1f/d26/f2c [2915931,59375] 0 2024-04-05T21:26:33.185 INFO:tasks.workunit.client.1.smithi188.stdout:4/570: mknod d0/d2/d2a/d71/cc9 0 2024-04-05T21:26:33.185 INFO:tasks.workunit.client.1.smithi188.stdout:3/387: getdents d5/d7/d65 0 2024-04-05T21:26:33.186 INFO:tasks.workunit.client.1.smithi188.stdout:4/571: readlink d0/d14/d15/d78/l91 0 2024-04-05T21:26:33.186 INFO:tasks.workunit.client.1.smithi188.stdout:4/572: read d0/d2/fbb [1981597,36412] 0 2024-04-05T21:26:33.187 INFO:tasks.workunit.client.1.smithi188.stdout:3/388: mknod d5/d7/d6d/c86 0 2024-04-05T21:26:33.189 INFO:tasks.workunit.client.1.smithi188.stdout:9/402: creat dd/d11/d22/f80 x:0 0 0 2024-04-05T21:26:33.190 INFO:tasks.workunit.client.1.smithi188.stdout:3/389: symlink d5/d7/d65/d5f/l87 0 2024-04-05T21:26:33.192 INFO:tasks.workunit.client.1.smithi188.stdout:9/403: mkdir dd/d3d/d81 0 2024-04-05T21:26:33.194 INFO:tasks.workunit.client.1.smithi188.stdout:4/573: mknod d0/d14/d15/d1b/cca 0 2024-04-05T21:26:33.197 INFO:tasks.workunit.client.1.smithi188.stdout:9/404: creat dd/f82 x:0 0 0 2024-04-05T21:26:33.198 INFO:tasks.workunit.client.1.smithi188.stdout:0/248: write d4/d6/f1f [476755,76211] 0 2024-04-05T21:26:33.200 INFO:tasks.workunit.client.1.smithi188.stdout:4/574: creat d0/d2/d17/d48/d70/fcb x:0 0 0 2024-04-05T21:26:33.201 INFO:tasks.workunit.client.1.smithi188.stdout:3/390: dwrite d5/f7d [0,4194304] 0 2024-04-05T21:26:33.204 INFO:tasks.workunit.client.1.smithi188.stdout:3/391: rmdir d5/d7/d6d 39 2024-04-05T21:26:33.210 INFO:tasks.workunit.client.1.smithi188.stdout:4/575: mkdir d0/d2/d52/dcc 0 2024-04-05T21:26:33.210 INFO:tasks.workunit.client.1.smithi188.stdout:3/392: creat d5/d7/d65/f88 x:0 0 0 2024-04-05T21:26:33.210 INFO:tasks.workunit.client.1.smithi188.stdout:0/249: creat d4/f5c x:0 0 0 2024-04-05T21:26:33.213 INFO:tasks.workunit.client.1.smithi188.stdout:9/405: rename dd/d11/d1f/d26/l62 to dd/d11/d1f/l83 0 2024-04-05T21:26:33.214 INFO:tasks.workunit.client.1.smithi188.stdout:9/406: readlink dd/d49/l4b 0 2024-04-05T21:26:33.215 INFO:tasks.workunit.client.1.smithi188.stdout:4/576: mknod d0/d14/ccd 0 2024-04-05T21:26:33.217 INFO:tasks.workunit.client.1.smithi188.stdout:9/407: creat dd/d49/f84 x:0 0 0 2024-04-05T21:26:33.220 INFO:tasks.workunit.client.1.smithi188.stdout:4/577: rename d0/d14/d15/d78/db1/lc6 to d0/d2/d52/lce 0 2024-04-05T21:26:33.220 INFO:tasks.workunit.client.1.smithi188.stdout:3/393: write d5/d35/f52 [1540474,65572] 0 2024-04-05T21:26:33.220 INFO:tasks.workunit.client.1.smithi188.stdout:0/250: symlink d4/d6/d13/d23/d24/l5d 0 2024-04-05T21:26:33.222 INFO:tasks.workunit.client.1.smithi188.stdout:0/251: chown d4/d8/db/d21/d4a/c5b 748557 1 2024-04-05T21:26:33.222 INFO:tasks.workunit.client.1.smithi188.stdout:4/578: mknod d0/d2/d2a/ccf 0 2024-04-05T21:26:33.224 INFO:tasks.workunit.client.1.smithi188.stdout:4/579: chown d0/d14/d15/d53/f74 76020679 1 2024-04-05T21:26:33.226 INFO:tasks.workunit.client.1.smithi188.stdout:4/580: write d0/d14/d15/d42/fa0 [910599,45811] 0 2024-04-05T21:26:33.227 INFO:tasks.workunit.client.1.smithi188.stdout:3/394: link d5/f7d d5/dc/d2f/f89 0 2024-04-05T21:26:33.229 INFO:tasks.workunit.client.1.smithi188.stdout:4/581: creat d0/d2/d52/fd0 x:0 0 0 2024-04-05T21:26:33.230 INFO:tasks.workunit.client.0.smithi121.stdout:1/23: sync 2024-04-05T21:26:33.230 INFO:tasks.workunit.client.0.smithi121.stdout:2/60: sync 2024-04-05T21:26:33.230 INFO:tasks.workunit.client.0.smithi121.stdout:4/56: sync 2024-04-05T21:26:33.230 INFO:tasks.workunit.client.0.smithi121.stdout:5/74: sync 2024-04-05T21:26:33.230 INFO:tasks.workunit.client.0.smithi121.stdout:0/53: sync 2024-04-05T21:26:33.230 INFO:tasks.workunit.client.0.smithi121.stdout:3/35: sync 2024-04-05T21:26:33.230 INFO:tasks.workunit.client.0.smithi121.stdout:8/8: sync 2024-04-05T21:26:33.230 INFO:tasks.workunit.client.0.smithi121.stdout:6/37: sync 2024-04-05T21:26:33.231 INFO:tasks.workunit.client.0.smithi121.stdout:9/20: sync 2024-04-05T21:26:33.231 INFO:tasks.workunit.client.0.smithi121.stdout:7/38: sync 2024-04-05T21:26:33.231 INFO:tasks.workunit.client.1.smithi188.stdout:3/395: mkdir d5/dc/d2f/d8a 0 2024-04-05T21:26:33.232 INFO:tasks.workunit.client.1.smithi188.stdout:0/252: write d4/f27 [1428154,13844] 0 2024-04-05T21:26:33.232 INFO:tasks.workunit.client.0.smithi121.stdout:5/75: chown dd/de/c14 789 1 2024-04-05T21:26:33.233 INFO:tasks.workunit.client.1.smithi188.stdout:9/408: dread dd/d11/d1f/d25/d38/f39 [0,4194304] 0 2024-04-05T21:26:33.233 INFO:tasks.workunit.client.1.smithi188.stdout:3/396: mknod d5/d7/d31/c8b 0 2024-04-05T21:26:33.234 INFO:tasks.workunit.client.1.smithi188.stdout:0/253: write d4/d8/f4d [383921,89121] 0 2024-04-05T21:26:33.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.232+0000 7f06d0a99700 1 -- 172.21.15.121:0/2071769846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc0ffad0 msgr2=0x7f06cc0ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:33.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.232+0000 7f06d0a99700 1 --2- 172.21.15.121:0/2071769846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc0ffad0 0x7f06cc0ffef0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f06c400b0a0 tx=0x7f06c400b3b0 comp rx=0 tx=0).stop 2024-04-05T21:26:33.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.232+0000 7f06d0a99700 1 -- 172.21.15.121:0/2071769846 shutdown_connections 2024-04-05T21:26:33.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.232+0000 7f06d0a99700 1 --2- 172.21.15.121:0/2071769846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06cc100cd0 0x7f06cc101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:33.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.232+0000 7f06d0a99700 1 --2- 172.21.15.121:0/2071769846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc0ffad0 0x7f06cc0ffef0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:33.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.232+0000 7f06d0a99700 1 -- 172.21.15.121:0/2071769846 >> 172.21.15.121:0/2071769846 conn(0x7f06cc0fb050 msgr2=0x7f06cc0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:33.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.232+0000 7f06d0a99700 1 -- 172.21.15.121:0/2071769846 shutdown_connections 2024-04-05T21:26:33.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.232+0000 7f06d0a99700 1 -- 172.21.15.121:0/2071769846 wait complete. 2024-04-05T21:26:33.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.233+0000 7f06d0a99700 1 Processor -- start 2024-04-05T21:26:33.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.233+0000 7f06d0a99700 1 -- start start 2024-04-05T21:26:33.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.233+0000 7f06d0a99700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc0ffad0 0x7f06cc196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:33.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.233+0000 7f06d0a99700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06cc100cd0 0x7f06cc196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:33.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.233+0000 7f06d0a99700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc196e60 con 0x7f06cc0ffad0 2024-04-05T21:26:33.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.233+0000 7f06d0a99700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc196fa0 con 0x7f06cc100cd0 2024-04-05T21:26:33.236 INFO:tasks.workunit.client.1.smithi188.stdout:4/582: creat d0/d2/d2a/d34/fd1 x:0 0 0 2024-04-05T21:26:33.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.234+0000 7f06ca59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc0ffad0 0x7f06cc196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:33.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.234+0000 7f06ca59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc0ffad0 0x7f06cc196300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37520/0 (socket says 172.21.15.121:37520) 2024-04-05T21:26:33.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.234+0000 7f06ca59c700 1 -- 172.21.15.121:0/1576185152 learned_addr learned my addr 172.21.15.121:0/1576185152 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:26:33.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.234+0000 7f06ca59c700 1 -- 172.21.15.121:0/1576185152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06cc100cd0 msgr2=0x7f06cc196840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:33.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.234+0000 7f06ca59c700 1 --2- 172.21.15.121:0/1576185152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06cc100cd0 0x7f06cc196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:33.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.234+0000 7f06ca59c700 1 -- 172.21.15.121:0/1576185152 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c4009d00 con 0x7f06cc0ffad0 2024-04-05T21:26:33.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.234+0000 7f06ca59c700 1 --2- 172.21.15.121:0/1576185152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc0ffad0 0x7f06cc196300 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f06c400b070 tx=0x7f06c40092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:33.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.235+0000 7f06c37fe700 1 -- 172.21.15.121:0/1576185152 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06c400e040 con 0x7f06cc0ffad0 2024-04-05T21:26:33.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.235+0000 7f06d0a99700 1 -- 172.21.15.121:0/1576185152 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06cc105f10 con 0x7f06cc0ffad0 2024-04-05T21:26:33.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.235+0000 7f06d0a99700 1 -- 172.21.15.121:0/1576185152 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06cc1064e0 con 0x7f06cc0ffad0 2024-04-05T21:26:33.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.235+0000 7f06c37fe700 1 -- 172.21.15.121:0/1576185152 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f06c400be90 con 0x7f06cc0ffad0 2024-04-05T21:26:33.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.235+0000 7f06c37fe700 1 -- 172.21.15.121:0/1576185152 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06c40049f0 con 0x7f06cc0ffad0 2024-04-05T21:26:33.239 INFO:tasks.workunit.client.0.smithi121.stdout:1/24: unlink l3 0 2024-04-05T21:26:33.239 INFO:tasks.workunit.client.0.smithi121.stdout:0/54: mknod d7/c12 0 2024-04-05T21:26:33.239 INFO:tasks.workunit.client.0.smithi121.stdout:7/39: rmdir d0 39 2024-04-05T21:26:33.239 INFO:tasks.workunit.client.0.smithi121.stdout:3/36: fsync d0/d3/f4 0 2024-04-05T21:26:33.239 INFO:tasks.workunit.client.0.smithi121.stdout:9/21: read f2 [817541,105440] 0 2024-04-05T21:26:33.239 INFO:tasks.workunit.client.0.smithi121.stdout:9/22: rmdir - no directory 2024-04-05T21:26:33.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.237+0000 7f06c37fe700 1 -- 172.21.15.121:0/1576185152 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 25) v1 ==== 90681+0+0 (secure 0 0 0) 0x7f06c4019070 con 0x7f06cc0ffad0 2024-04-05T21:26:33.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.237+0000 7f06c37fe700 1 --2- 172.21.15.121:0/1576185152 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f06b806ca10 0x7f06b806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:26:33.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.237+0000 7f06c37fe700 1 -- 172.21.15.121:0/1576185152 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f06c40959b0 con 0x7f06cc0ffad0 2024-04-05T21:26:33.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.237+0000 7f06c9d9b700 1 --2- 172.21.15.121:0/1576185152 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f06b806ca10 0x7f06b806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:26:33.240 INFO:tasks.workunit.client.0.smithi121.stdout:6/38: mkdir d1/d2/d3/d5/db 0 2024-04-05T21:26:33.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.238+0000 7f06c9d9b700 1 --2- 172.21.15.121:0/1576185152 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f06b806ca10 0x7f06b806eed0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f06b4005950 tx=0x7f06b40058e0 comp rx=0 tx=0).ready entity=mgr.24327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:26:33.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.239+0000 7f06d0a99700 1 -- 172.21.15.121:0/1576185152 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ac005320 con 0x7f06cc0ffad0 2024-04-05T21:26:33.241 INFO:tasks.workunit.client.0.smithi121.stdout:5/76: rename dd/de/df to dd/de/d19 0 2024-04-05T21:26:33.242 INFO:tasks.workunit.client.1.smithi188.stdout:0/254: chown d4/d8/f4d 48031 1 2024-04-05T21:26:33.242 INFO:tasks.workunit.client.1.smithi188.stdout:0/255: readlink d4/d8/db/d21/l4e 0 2024-04-05T21:26:33.242 INFO:tasks.workunit.client.1.smithi188.stdout:4/583: chown d0/d14/d15/d53/fac 393 1 2024-04-05T21:26:33.242 INFO:tasks.workunit.client.1.smithi188.stdout:9/409: rmdir dd 39 2024-04-05T21:26:33.243 INFO:tasks.workunit.client.1.smithi188.stdout:0/256: mknod d4/d8/db/d42/d38/c5e 0 2024-04-05T21:26:33.245 INFO:tasks.workunit.client.1.smithi188.stdout:9/410: mkdir dd/d11/d73/d85 0 2024-04-05T21:26:33.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.245+0000 7f06c37fe700 1 -- 172.21.15.121:0/1576185152 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f06c40633f0 con 0x7f06cc0ffad0 2024-04-05T21:26:33.247 INFO:tasks.workunit.client.0.smithi121.stdout:9/23: unlink c0 0 2024-04-05T21:26:33.247 INFO:tasks.workunit.client.0.smithi121.stdout:9/24: stat f4 0 2024-04-05T21:26:33.248 INFO:tasks.workunit.client.1.smithi188.stdout:9/411: chown dd/d11/d1f/d25/c58 9601 1 2024-04-05T21:26:33.248 INFO:tasks.workunit.client.0.smithi121.stdout:3/37: read d0/f7 [51394,119648] 0 2024-04-05T21:26:33.249 INFO:tasks.workunit.client.1.smithi188.stdout:4/584: unlink d0/d14/d15/d1b/f60 0 2024-04-05T21:26:33.249 INFO:tasks.workunit.client.0.smithi121.stdout:6/39: rename d1/d2/d3/d5/db to d1/d2/dc 0 2024-04-05T21:26:33.250 INFO:tasks.workunit.client.1.smithi188.stdout:0/257: creat d4/f5f x:0 0 0 2024-04-05T21:26:33.251 INFO:tasks.workunit.client.0.smithi121.stdout:5/77: creat dd/de/f1a x:0 0 0 2024-04-05T21:26:33.251 INFO:tasks.workunit.client.1.smithi188.stdout:9/412: creat dd/d11/d73/f86 x:0 0 0 2024-04-05T21:26:33.252 INFO:tasks.workunit.client.1.smithi188.stdout:3/397: dwrite d5/dc/f1f [4194304,4194304] 0 2024-04-05T21:26:33.253 INFO:tasks.workunit.client.1.smithi188.stdout:4/585: unlink d0/d14/f1c 0 2024-04-05T21:26:33.255 INFO:tasks.workunit.client.0.smithi121.stdout:0/55: write f2 [7299081,65858] 0 2024-04-05T21:26:33.256 INFO:tasks.workunit.client.1.smithi188.stdout:9/413: mknod dd/d11/c87 0 2024-04-05T21:26:33.257 INFO:tasks.workunit.client.1.smithi188.stdout:4/586: symlink d0/d14/d15/d78/db1/ld2 0 2024-04-05T21:26:33.258 INFO:tasks.workunit.client.0.smithi121.stdout:4/57: mknod d5/d11/c1a 0 2024-04-05T21:26:33.261 INFO:tasks.workunit.client.0.smithi121.stdout:2/61: truncate d2/d9/fb 108382 0 2024-04-05T21:26:33.263 INFO:tasks.workunit.client.0.smithi121.stdout:9/25: creat f5 x:0 0 0 2024-04-05T21:26:33.263 INFO:tasks.workunit.client.0.smithi121.stdout:9/26: readlink l3 0 2024-04-05T21:26:33.264 INFO:tasks.workunit.client.1.smithi188.stdout:3/398: dwrite d5/d2b/f2e [0,4194304] 0 2024-04-05T21:26:33.264 INFO:tasks.workunit.client.0.smithi121.stdout:1/25: write f2 [1038574,72089] 0 2024-04-05T21:26:33.265 INFO:tasks.workunit.client.0.smithi121.stdout:6/40: stat d1/d2/c7 0 2024-04-05T21:26:33.265 INFO:tasks.workunit.client.0.smithi121.stdout:1/26: chown f2 91306019 1 2024-04-05T21:26:33.265 INFO:tasks.workunit.client.0.smithi121.stdout:8/9: getdents . 0 2024-04-05T21:26:33.266 INFO:tasks.workunit.client.0.smithi121.stdout:8/10: write - no filename 2024-04-05T21:26:33.266 INFO:tasks.workunit.client.0.smithi121.stdout:8/11: dwrite - no filename 2024-04-05T21:26:33.266 INFO:tasks.workunit.client.0.smithi121.stdout:0/56: fsync d7/f9 0 2024-04-05T21:26:33.267 INFO:tasks.workunit.client.0.smithi121.stdout:1/27: write f2 [1615592,108840] 0 2024-04-05T21:26:33.269 INFO:tasks.workunit.client.1.smithi188.stdout:6/489: sync 2024-04-05T21:26:33.269 INFO:tasks.workunit.client.0.smithi121.stdout:4/58: creat d5/dc/f1b x:0 0 0 2024-04-05T21:26:33.270 INFO:tasks.workunit.client.0.smithi121.stdout:2/62: dread - d2/d9/fd zero size 2024-04-05T21:26:33.271 INFO:tasks.workunit.client.0.smithi121.stdout:9/27: mkdir d6 0 2024-04-05T21:26:33.272 INFO:tasks.workunit.client.0.smithi121.stdout:3/38: mknod d0/d3/dc/cf 0 2024-04-05T21:26:33.272 INFO:tasks.workunit.client.1.smithi188.stdout:9/414: dwrite dd/d11/d1f/f6e [0,4194304] 0 2024-04-05T21:26:33.272 INFO:tasks.workunit.client.1.smithi188.stdout:6/490: write d6/d13/d18/f82 [283781,94888] 0 2024-04-05T21:26:33.272 INFO:tasks.workunit.client.1.smithi188.stdout:3/399: write d5/d2b/f2e [456621,74539] 0 2024-04-05T21:26:33.273 INFO:tasks.workunit.client.0.smithi121.stdout:6/41: rmdir d1 39 2024-04-05T21:26:33.278 INFO:tasks.workunit.client.1.smithi188.stdout:6/491: mkdir d6/d13/d18/d94 0 2024-04-05T21:26:33.278 INFO:tasks.workunit.client.1.smithi188.stdout:9/415: mknod dd/d11/d1f/c88 0 2024-04-05T21:26:33.280 INFO:tasks.workunit.client.0.smithi121.stdout:0/57: creat d7/f13 x:0 0 0 2024-04-05T21:26:33.281 INFO:tasks.workunit.client.0.smithi121.stdout:7/40: link d0/f8 d0/d9/f12 0 2024-04-05T21:26:33.281 INFO:tasks.workunit.client.1.smithi188.stdout:4/587: dwrite d0/d14/d15/d1b/f3d [4194304,4194304] 0 2024-04-05T21:26:33.281 INFO:tasks.workunit.client.1.smithi188.stdout:6/492: rmdir d6/d13/d18/d19 39 2024-04-05T21:26:33.282 INFO:tasks.workunit.client.1.smithi188.stdout:9/416: mkdir dd/d3d/d89 0 2024-04-05T21:26:33.283 INFO:tasks.workunit.client.0.smithi121.stdout:1/28: creat f4 x:0 0 0 2024-04-05T21:26:33.284 INFO:tasks.workunit.client.0.smithi121.stdout:4/59: fdatasync d5/d11/f14 0 2024-04-05T21:26:33.286 INFO:tasks.workunit.client.0.smithi121.stdout:1/29: write f4 [1037820,124092] 0 2024-04-05T21:26:33.287 INFO:tasks.workunit.client.1.smithi188.stdout:4/588: dread d0/d2/fbb [0,4194304] 0 2024-04-05T21:26:33.290 INFO:tasks.workunit.client.1.smithi188.stdout:4/589: symlink d0/d2/d2a/d71/ld3 0 2024-04-05T21:26:33.293 INFO:tasks.workunit.client.0.smithi121.stdout:9/28: dwrite f5 [0,4194304] 0 2024-04-05T21:26:33.295 INFO:tasks.workunit.client.0.smithi121.stdout:6/42: chown d1/d2/dc 0 1 2024-04-05T21:26:33.297 INFO:tasks.workunit.client.1.smithi188.stdout:6/493: getdents d6/d13 0 2024-04-05T21:26:33.298 INFO:tasks.workunit.client.1.smithi188.stdout:6/494: chown d6/d13/c4d 3708720 1 2024-04-05T21:26:33.299 INFO:tasks.workunit.client.0.smithi121.stdout:0/58: creat d7/dc/f14 x:0 0 0 2024-04-05T21:26:33.300 INFO:tasks.workunit.client.0.smithi121.stdout:3/39: dread d0/d3/f8 [0,4194304] 0 2024-04-05T21:26:33.304 INFO:tasks.workunit.client.0.smithi121.stdout:4/60: creat d5/d11/f1c x:0 0 0 2024-04-05T21:26:33.304 INFO:tasks.workunit.client.0.smithi121.stdout:8/12: link l0 l1 0 2024-04-05T21:26:33.304 INFO:tasks.workunit.client.0.smithi121.stdout:8/13: write - no filename 2024-04-05T21:26:33.304 INFO:tasks.workunit.client.0.smithi121.stdout:8/14: write - no filename 2024-04-05T21:26:33.305 INFO:tasks.workunit.client.1.smithi188.stdout:4/590: dwrite d0/d14/d15/d78/db1/fb5 [0,4194304] 0 2024-04-05T21:26:33.305 INFO:tasks.workunit.client.0.smithi121.stdout:6/43: creat d1/d2/d3/d5/fd x:0 0 0 2024-04-05T21:26:33.305 INFO:tasks.workunit.client.0.smithi121.stdout:1/30: dwrite f2 [0,4194304] 0 2024-04-05T21:26:33.309 INFO:tasks.workunit.client.1.smithi188.stdout:6/495: mkdir d6/d13/d18/d19/d2c/d3f/d95 0 2024-04-05T21:26:33.309 INFO:tasks.workunit.client.1.smithi188.stdout:6/496: read d6/d13/f89 [827202,53275] 0 2024-04-05T21:26:33.310 INFO:tasks.workunit.client.0.smithi121.stdout:4/61: creat d5/f1d x:0 0 0 2024-04-05T21:26:33.313 INFO:tasks.workunit.client.0.smithi121.stdout:5/78: truncate dd/de/d19/d13/f16 875403 0 2024-04-05T21:26:33.313 INFO:tasks.workunit.client.0.smithi121.stdout:1/31: creat f5 x:0 0 0 2024-04-05T21:26:33.314 INFO:tasks.workunit.client.1.smithi188.stdout:4/591: truncate d0/d2/f58 537752 0 2024-04-05T21:26:33.316 INFO:tasks.workunit.client.1.smithi188.stdout:6/497: creat d6/d28/d45/f96 x:0 0 0 2024-04-05T21:26:33.316 INFO:tasks.workunit.client.0.smithi121.stdout:6/44: chown d1/d2/d3/d5/l8 788833 1 2024-04-05T21:26:33.317 INFO:tasks.workunit.client.0.smithi121.stdout:8/15: stat l1 0 2024-04-05T21:26:33.317 INFO:tasks.workunit.client.0.smithi121.stdout:8/16: dwrite - no filename 2024-04-05T21:26:33.318 INFO:tasks.workunit.client.1.smithi188.stdout:4/592: write d0/d14/d15/d53/d82/d8f/f93 [898760,97177] 0 2024-04-05T21:26:33.318 INFO:tasks.workunit.client.0.smithi121.stdout:6/45: truncate d1/d2/d3/d5/fd 707197 0 2024-04-05T21:26:33.320 INFO:tasks.workunit.client.0.smithi121.stdout:5/79: unlink lb 0 2024-04-05T21:26:33.322 INFO:tasks.workunit.client.0.smithi121.stdout:1/32: mknod c6 0 2024-04-05T21:26:33.324 INFO:tasks.workunit.client.1.smithi188.stdout:9/417: dwrite dd/d11/d1f/d25/f44 [0,4194304] 0 2024-04-05T21:26:33.324 INFO:tasks.workunit.client.0.smithi121.stdout:5/80: truncate dd/de/f1a 855069 0 2024-04-05T21:26:33.325 INFO:tasks.workunit.client.0.smithi121.stdout:2/63: write d2/d9/fb [856086,51325] 0 2024-04-05T21:26:33.325 INFO:tasks.workunit.client.1.smithi188.stdout:6/498: creat d6/d13/d18/d19/d2c/d34/f97 x:0 0 0 2024-04-05T21:26:33.326 INFO:tasks.workunit.client.0.smithi121.stdout:8/17: chown l0 206 1 2024-04-05T21:26:33.326 INFO:tasks.workunit.client.0.smithi121.stdout:1/33: mknod c7 0 2024-04-05T21:26:33.327 INFO:tasks.workunit.client.1.smithi188.stdout:9/418: rename dd/d11/d1f/d25/d38 to dd/d11/d1f/d25/d8a 0 2024-04-05T21:26:33.328 INFO:tasks.workunit.client.0.smithi121.stdout:6/46: mkdir d1/d2/d3/d4/de 0 2024-04-05T21:26:33.329 INFO:tasks.workunit.client.1.smithi188.stdout:4/593: mkdir d0/d14/d15/dd4 0 2024-04-05T21:26:33.329 INFO:tasks.workunit.client.1.smithi188.stdout:9/419: fdatasync dd/d11/d73/f75 0 2024-04-05T21:26:33.330 INFO:tasks.workunit.client.1.smithi188.stdout:9/420: dread - dd/d3d/f61 zero size 2024-04-05T21:26:33.330 INFO:tasks.workunit.client.1.smithi188.stdout:4/594: chown d0/d2/d2a/d71/cc9 34 1 2024-04-05T21:26:33.331 INFO:tasks.workunit.client.1.smithi188.stdout:6/499: link d6/d13/d38/c78 d6/d13/d18/d19/d2c/d34/c98 0 2024-04-05T21:26:33.331 INFO:tasks.workunit.client.0.smithi121.stdout:8/18: unlink l0 0 2024-04-05T21:26:33.331 INFO:tasks.workunit.client.0.smithi121.stdout:8/19: dread - no filename 2024-04-05T21:26:33.333 INFO:tasks.workunit.client.1.smithi188.stdout:4/595: mkdir d0/d14/d15/d53/d82/dd5 0 2024-04-05T21:26:33.335 INFO:tasks.workunit.client.0.smithi121.stdout:5/81: creat dd/de/f1b x:0 0 0 2024-04-05T21:26:33.337 INFO:tasks.workunit.client.1.smithi188.stdout:4/596: rename d0/fc2 to d0/d14/d15/d53/d82/dd5/fd6 0 2024-04-05T21:26:33.337 INFO:tasks.workunit.client.0.smithi121.stdout:0/59: write d7/f8 [429157,115975] 0 2024-04-05T21:26:33.340 INFO:tasks.workunit.client.1.smithi188.stdout:4/597: mknod d0/d14/d15/d42/cd7 0 2024-04-05T21:26:33.342 INFO:tasks.workunit.client.1.smithi188.stdout:6/500: dread d6/d7/f14 [0,4194304] 0 2024-04-05T21:26:33.342 INFO:tasks.workunit.client.0.smithi121.stdout:8/20: creat f2 x:0 0 0 2024-04-05T21:26:33.345 INFO:tasks.workunit.client.1.smithi188.stdout:4/598: rename d0/d2/d52/dc1/dc3 to d0/d14/dd8 0 2024-04-05T21:26:33.345 INFO:tasks.workunit.client.0.smithi121.stdout:4/62: write d5/fa [708374,130842] 0 2024-04-05T21:26:33.349 INFO:tasks.workunit.client.0.smithi121.stdout:9/29: dwrite f5 [4194304,4194304] 0 2024-04-05T21:26:33.350 INFO:tasks.workunit.client.1.smithi188.stdout:4/599: creat d0/d14/d15/d78/db1/d6a/fd9 x:0 0 0 2024-04-05T21:26:33.352 INFO:tasks.workunit.client.0.smithi121.stdout:0/60: creat d7/dc/f15 x:0 0 0 2024-04-05T21:26:33.352 INFO:tasks.workunit.client.1.smithi188.stdout:6/501: dwrite d6/d13/d18/d19/f7e [0,4194304] 0 2024-04-05T21:26:33.354 INFO:tasks.workunit.client.0.smithi121.stdout:7/41: dwrite d0/d9/fe [0,4194304] 0 2024-04-05T21:26:33.354 INFO:tasks.workunit.client.0.smithi121.stdout:9/30: read f4 [599300,123059] 0 2024-04-05T21:26:33.355 INFO:tasks.workunit.client.0.smithi121.stdout:2/64: write d2/d9/fd [826404,72213] 0 2024-04-05T21:26:33.358 INFO:tasks.workunit.client.1.smithi188.stdout:4/600: write d0/d14/d15/d1b/f3d [6033723,120692] 0 2024-04-05T21:26:33.360 INFO:tasks.workunit.client.1.smithi188.stdout:6/502: write d6/d13/f89 [1206871,89120] 0 2024-04-05T21:26:33.361 INFO:tasks.workunit.client.0.smithi121.stdout:3/40: dwrite d0/d3/f8 [0,4194304] 0 2024-04-05T21:26:33.364 INFO:tasks.workunit.client.0.smithi121.stdout:1/34: dwrite f5 [0,4194304] 0 2024-04-05T21:26:33.365 INFO:tasks.workunit.client.0.smithi121.stdout:9/31: write f4 [2747731,103433] 0 2024-04-05T21:26:33.365 INFO:tasks.workunit.client.0.smithi121.stdout:3/41: stat d0/f2 0 2024-04-05T21:26:33.367 INFO:tasks.workunit.client.1.smithi188.stdout:6/503: symlink d6/d7/l99 0 2024-04-05T21:26:33.370 INFO:tasks.workunit.client.0.smithi121.stdout:1/35: chown c6 254633679 1 2024-04-05T21:26:33.371 INFO:tasks.workunit.client.1.smithi188.stdout:4/601: dwrite d0/d2/d4f/f5c [0,4194304] 0 2024-04-05T21:26:33.373 INFO:tasks.workunit.client.0.smithi121.stdout:0/61: write d7/dc/f14 [360878,9451] 0 2024-04-05T21:26:33.373 INFO:tasks.workunit.client.0.smithi121.stdout:9/32: truncate f4 3652892 0 2024-04-05T21:26:33.373 INFO:tasks.workunit.client.1.smithi188.stdout:6/504: symlink d6/d7/l9a 0 2024-04-05T21:26:33.374 INFO:tasks.workunit.client.0.smithi121.stdout:1/36: chown f2 166 1 2024-04-05T21:26:33.374 INFO:tasks.workunit.client.0.smithi121.stdout:9/33: readlink l3 0 2024-04-05T21:26:33.374 INFO:tasks.workunit.client.0.smithi121.stdout:2/65: mknod d2/d9/c13 0 2024-04-05T21:26:33.376 INFO:tasks.workunit.client.0.smithi121.stdout:5/82: dwrite f6 [0,4194304] 0 2024-04-05T21:26:33.384 INFO:tasks.workunit.client.0.smithi121.stdout:7/42: dread d0/d9/fe [0,4194304] 0 2024-04-05T21:26:33.384 INFO:tasks.workunit.client.0.smithi121.stdout:9/34: creat d6/f7 x:0 0 0 2024-04-05T21:26:33.386 INFO:tasks.workunit.client.1.smithi188.stdout:6/505: dread d6/d7/f1e [4194304,4194304] 0 2024-04-05T21:26:33.387 INFO:tasks.workunit.client.1.smithi188.stdout:6/506: readlink d6/d28/l57 0 2024-04-05T21:26:33.387 INFO:tasks.workunit.client.0.smithi121.stdout:4/63: dwrite d5/fa [0,4194304] 0 2024-04-05T21:26:33.388 INFO:tasks.workunit.client.0.smithi121.stdout:9/35: write f4 [3404663,121415] 0 2024-04-05T21:26:33.390 INFO:tasks.workunit.client.0.smithi121.stdout:9/36: chown l3 163062 1 2024-04-05T21:26:33.390 INFO:tasks.workunit.client.1.smithi188.stdout:6/507: mknod d6/d13/d48/d49/d75/c9b 0 2024-04-05T21:26:33.391 INFO:tasks.workunit.client.0.smithi121.stdout:8/21: unlink l1 0 2024-04-05T21:26:33.392 INFO:tasks.workunit.client.1.smithi188.stdout:6/508: truncate d6/d13/d38/f55 906855 0 2024-04-05T21:26:33.394 INFO:tasks.workunit.client.0.smithi121.stdout:4/64: creat d5/d11/f1e x:0 0 0 2024-04-05T21:26:33.396 INFO:tasks.workunit.client.0.smithi121.stdout:1/37: dread f5 [0,4194304] 0 2024-04-05T21:26:33.398 INFO:tasks.workunit.client.0.smithi121.stdout:3/42: dread d0/d3/f8 [0,4194304] 0 2024-04-05T21:26:33.398 INFO:tasks.workunit.client.0.smithi121.stdout:0/62: mknod d7/dc/d10/c16 0 2024-04-05T21:26:33.399 INFO:tasks.workunit.client.1.smithi188.stdout:6/509: rename d6/f64 to d6/d13/d48/d49/f9c 0 2024-04-05T21:26:33.400 INFO:tasks.workunit.client.0.smithi121.stdout:5/83: mkdir dd/de/d19/d1c 0 2024-04-05T21:26:33.400 INFO:tasks.workunit.client.1.smithi188.stdout:6/510: write d6/f2e [3067823,124385] 0 2024-04-05T21:26:33.407 INFO:tasks.workunit.client.0.smithi121.stdout:7/43: getdents d0/d9 0 2024-04-05T21:26:33.407 INFO:tasks.workunit.client.1.smithi188.stdout:6/511: dread d6/d7/f16 [0,4194304] 0 2024-04-05T21:26:33.409 INFO:tasks.workunit.client.1.smithi188.stdout:6/512: symlink d6/d13/d18/d19/d61/l9d 0 2024-04-05T21:26:33.410 INFO:tasks.workunit.client.1.smithi188.stdout:6/513: readlink d6/l9 0 2024-04-05T21:26:33.412 INFO:tasks.workunit.client.0.smithi121.stdout:1/38: dread f4 [0,4194304] 0 2024-04-05T21:26:33.412 INFO:tasks.workunit.client.0.smithi121.stdout:7/44: mknod d0/d9/c13 0 2024-04-05T21:26:33.415 INFO:tasks.workunit.client.0.smithi121.stdout:2/66: dwrite d2/f10 [0,4194304] 0 2024-04-05T21:26:33.415 INFO:tasks.workunit.client.0.smithi121.stdout:9/37: getdents d6 0 2024-04-05T21:26:33.417 INFO:tasks.workunit.client.0.smithi121.stdout:2/67: readlink d2/l6 0 2024-04-05T21:26:33.417 INFO:tasks.workunit.client.0.smithi121.stdout:5/84: dread dd/de/d19/d13/f16 [0,4194304] 0 2024-04-05T21:26:33.421 INFO:tasks.workunit.client.0.smithi121.stdout:1/39: mkdir d8 0 2024-04-05T21:26:33.421 INFO:tasks.workunit.client.0.smithi121.stdout:4/65: write d5/fa [4194658,22653] 0 2024-04-05T21:26:33.422 INFO:tasks.workunit.client.0.smithi121.stdout:0/63: truncate f2 953871 0 2024-04-05T21:26:33.427 INFO:tasks.workunit.client.0.smithi121.stdout:4/66: fsync d5/fa 0 2024-04-05T21:26:33.428 INFO:tasks.workunit.client.0.smithi121.stdout:8/22: dwrite f2 [0,4194304] 0 2024-04-05T21:26:33.428 INFO:tasks.workunit.client.0.smithi121.stdout:0/64: mknod d7/dc/c17 0 2024-04-05T21:26:33.431 INFO:tasks.workunit.client.0.smithi121.stdout:1/40: write f5 [611194,87236] 0 2024-04-05T21:26:33.436 INFO:tasks.workunit.client.0.smithi121.stdout:4/67: symlink d5/d11/l1f 0 2024-04-05T21:26:33.441 INFO:tasks.workunit.client.0.smithi121.stdout:1/41: mknod d8/c9 0 2024-04-05T21:26:33.443 INFO:tasks.workunit.client.0.smithi121.stdout:5/85: dread f6 [0,4194304] 0 2024-04-05T21:26:33.444 INFO:tasks.workunit.client.0.smithi121.stdout:2/68: dwrite d2/d9/fd [0,4194304] 0 2024-04-05T21:26:33.447 INFO:tasks.workunit.client.0.smithi121.stdout:0/65: truncate d7/f9 819183 0 2024-04-05T21:26:33.449 INFO:tasks.workunit.client.0.smithi121.stdout:9/38: dwrite f5 [4194304,4194304] 0 2024-04-05T21:26:33.450 INFO:tasks.workunit.client.0.smithi121.stdout:1/42: write f2 [61902,100451] 0 2024-04-05T21:26:33.454 INFO:tasks.workunit.client.0.smithi121.stdout:7/45: dwrite d0/d9/fd [0,4194304] 0 2024-04-05T21:26:33.454 INFO:tasks.workunit.client.0.smithi121.stdout:0/66: fdatasync d7/fa 0 2024-04-05T21:26:33.456 INFO:tasks.workunit.client.0.smithi121.stdout:1/43: creat d8/fa x:0 0 0 2024-04-05T21:26:33.456 INFO:tasks.workunit.client.0.smithi121.stdout:5/86: truncate dd/de/f1a 1289971 0 2024-04-05T21:26:33.456 INFO:tasks.workunit.client.0.smithi121.stdout:2/69: truncate d2/f11 1227567 0 2024-04-05T21:26:33.457 INFO:tasks.workunit.client.0.smithi121.stdout:9/39: symlink d6/l8 0 2024-04-05T21:26:33.461 INFO:tasks.workunit.client.0.smithi121.stdout:8/23: dread f2 [0,4194304] 0 2024-04-05T21:26:33.468 INFO:tasks.workunit.client.0.smithi121.stdout:7/46: rename d0/d9/c13 to d0/c14 0 2024-04-05T21:26:33.469 INFO:tasks.workunit.client.0.smithi121.stdout:4/68: dwrite d5/f6 [0,4194304] 0 2024-04-05T21:26:33.471 INFO:tasks.workunit.client.0.smithi121.stdout:5/87: dread - dd/f17 zero size 2024-04-05T21:26:33.475 INFO:tasks.workunit.client.0.smithi121.stdout:4/69: unlink d5/ff 0 2024-04-05T21:26:33.476 INFO:tasks.workunit.client.0.smithi121.stdout:0/67: dwrite d7/fa [0,4194304] 0 2024-04-05T21:26:33.476 INFO:tasks.workunit.client.0.smithi121.stdout:7/47: rename d0/f11 to d0/f15 0 2024-04-05T21:26:33.477 INFO:tasks.workunit.client.0.smithi121.stdout:9/40: dread f4 [0,4194304] 0 2024-04-05T21:26:33.477 INFO:tasks.workunit.client.0.smithi121.stdout:4/70: chown d5/f16 1412 1 2024-04-05T21:26:33.478 INFO:tasks.workunit.client.0.smithi121.stdout:6/47: sync 2024-04-05T21:26:33.479 INFO:tasks.workunit.client.0.smithi121.stdout:4/71: stat d5/f1d 0 2024-04-05T21:26:33.481 INFO:tasks.workunit.client.0.smithi121.stdout:5/88: dread dd/de/d19/d13/f16 [0,4194304] 0 2024-04-05T21:26:33.481 INFO:tasks.workunit.client.0.smithi121.stdout:1/44: rmdir d8 39 2024-04-05T21:26:33.482 INFO:tasks.workunit.client.0.smithi121.stdout:6/48: chown d1/d2/c7 7232 1 2024-04-05T21:26:33.482 INFO:tasks.workunit.client.0.smithi121.stdout:8/24: getdents . 0 2024-04-05T21:26:33.484 INFO:tasks.workunit.client.0.smithi121.stdout:7/48: symlink d0/l16 0 2024-04-05T21:26:33.484 INFO:tasks.workunit.client.0.smithi121.stdout:0/68: readlink l5 0 2024-04-05T21:26:33.487 INFO:tasks.workunit.client.0.smithi121.stdout:2/70: dread d2/f11 [0,4194304] 0 2024-04-05T21:26:33.489 INFO:tasks.workunit.client.0.smithi121.stdout:2/71: readlink d2/l6 0 2024-04-05T21:26:33.490 INFO:tasks.workunit.client.0.smithi121.stdout:0/69: creat d7/f18 x:0 0 0 2024-04-05T21:26:33.490 INFO:tasks.workunit.client.0.smithi121.stdout:6/49: symlink d1/d2/d3/lf 0 2024-04-05T21:26:33.492 INFO:tasks.workunit.client.0.smithi121.stdout:8/25: link f2 f3 0 2024-04-05T21:26:33.493 INFO:tasks.workunit.client.0.smithi121.stdout:2/72: creat d2/f14 x:0 0 0 2024-04-05T21:26:33.494 INFO:tasks.workunit.client.0.smithi121.stdout:7/49: dread d0/d9/f12 [0,4194304] 0 2024-04-05T21:26:33.494 INFO:tasks.workunit.client.0.smithi121.stdout:2/73: chown d2/d9/fb 1901 1 2024-04-05T21:26:33.496 INFO:tasks.workunit.client.0.smithi121.stdout:6/50: creat d1/f10 x:0 0 0 2024-04-05T21:26:33.496 INFO:tasks.workunit.client.0.smithi121.stdout:1/45: dread f2 [0,4194304] 0 2024-04-05T21:26:33.497 INFO:tasks.workunit.client.0.smithi121.stdout:9/41: dread f4 [0,4194304] 0 2024-04-05T21:26:33.499 INFO:tasks.workunit.client.0.smithi121.stdout:7/50: truncate d0/f15 5116118 0 2024-04-05T21:26:33.499 INFO:tasks.workunit.client.0.smithi121.stdout:2/74: truncate d2/d9/ff 2877828 0 2024-04-05T21:26:33.500 INFO:tasks.workunit.client.0.smithi121.stdout:6/51: read d1/d2/d3/d5/fd [342850,65511] 0 2024-04-05T21:26:33.501 INFO:tasks.workunit.client.0.smithi121.stdout:0/70: getdents d7/dc/d10 0 2024-04-05T21:26:33.502 INFO:tasks.workunit.client.0.smithi121.stdout:2/75: stat d2/d9/d12 0 2024-04-05T21:26:33.503 INFO:tasks.workunit.client.0.smithi121.stdout:1/46: fsync f4 0 2024-04-05T21:26:33.504 INFO:tasks.workunit.client.0.smithi121.stdout:5/89: dwrite dd/de/f1a [0,4194304] 0 2024-04-05T21:26:33.505 INFO:tasks.workunit.client.0.smithi121.stdout:1/47: fdatasync f4 0 2024-04-05T21:26:33.506 INFO:tasks.workunit.client.0.smithi121.stdout:0/71: fdatasync d7/dc/ff 0 2024-04-05T21:26:33.507 INFO:tasks.workunit.client.0.smithi121.stdout:9/42: write f4 [1974018,32149] 0 2024-04-05T21:26:33.507 INFO:tasks.workunit.client.0.smithi121.stdout:8/26: write f3 [4361901,63106] 0 2024-04-05T21:26:33.510 INFO:tasks.workunit.client.0.smithi121.stdout:0/72: dread - d7/f18 zero size 2024-04-05T21:26:33.510 INFO:tasks.workunit.client.0.smithi121.stdout:6/52: read d1/d2/d3/d5/fd [462697,123739] 0 2024-04-05T21:26:33.512 INFO:tasks.workunit.client.0.smithi121.stdout:1/48: write f2 [3846127,4151] 0 2024-04-05T21:26:33.512 INFO:tasks.workunit.client.0.smithi121.stdout:9/43: chown f5 8 1 2024-04-05T21:26:33.514 INFO:tasks.workunit.client.0.smithi121.stdout:0/73: write d7/dc/f15 [705309,32285] 0 2024-04-05T21:26:33.515 INFO:tasks.workunit.client.0.smithi121.stdout:9/44: mknod d6/c9 0 2024-04-05T21:26:33.516 INFO:tasks.workunit.client.0.smithi121.stdout:1/49: creat d8/fb x:0 0 0 2024-04-05T21:26:33.519 INFO:tasks.workunit.client.0.smithi121.stdout:5/90: dwrite dd/de/f1a [0,4194304] 0 2024-04-05T21:26:33.520 INFO:tasks.workunit.client.0.smithi121.stdout:1/50: creat d8/fc x:0 0 0 2024-04-05T21:26:33.521 INFO:tasks.workunit.client.0.smithi121.stdout:0/74: rename d7/c12 to d7/c19 0 2024-04-05T21:26:33.522 INFO:tasks.workunit.client.0.smithi121.stdout:1/51: dread - d8/fc zero size 2024-04-05T21:26:33.523 INFO:tasks.workunit.client.0.smithi121.stdout:5/91: creat dd/f1d x:0 0 0 2024-04-05T21:26:33.525 INFO:tasks.workunit.client.0.smithi121.stdout:6/53: dwrite d1/f10 [0,4194304] 0 2024-04-05T21:26:33.525 INFO:tasks.workunit.client.0.smithi121.stdout:1/52: mkdir d8/dd 0 2024-04-05T21:26:33.530 INFO:tasks.workunit.client.0.smithi121.stdout:1/53: rename d8/c9 to d8/ce 0 2024-04-05T21:26:33.530 INFO:tasks.workunit.client.0.smithi121.stdout:9/45: dread f5 [0,4194304] 0 2024-04-05T21:26:33.536 INFO:tasks.workunit.client.0.smithi121.stdout:5/92: dwrite dd/f1d [0,4194304] 0 2024-04-05T21:26:33.536 INFO:tasks.workunit.client.0.smithi121.stdout:5/93: stat f6 0 2024-04-05T21:26:33.537 INFO:tasks.workunit.client.0.smithi121.stdout:5/94: stat l8 0 2024-04-05T21:26:33.537 INFO:tasks.workunit.client.0.smithi121.stdout:9/46: dread f4 [0,4194304] 0 2024-04-05T21:26:33.539 INFO:tasks.workunit.client.1.smithi188.stdout:1/399: sync 2024-04-05T21:26:33.540 INFO:tasks.workunit.client.0.smithi121.stdout:5/95: symlink dd/de/d19/d1c/l1e 0 2024-04-05T21:26:33.541 INFO:tasks.workunit.client.0.smithi121.stdout:9/47: link d6/c9 d6/ca 0 2024-04-05T21:26:33.543 INFO:tasks.workunit.client.0.smithi121.stdout:5/96: rename dd/de/f1b to dd/f1f 0 2024-04-05T21:26:33.543 INFO:tasks.workunit.client.1.smithi188.stdout:5/399: sync 2024-04-05T21:26:33.544 INFO:tasks.workunit.client.0.smithi121.stdout:9/48: write f4 [1350744,65686] 0 2024-04-05T21:26:33.544 INFO:tasks.workunit.client.1.smithi188.stdout:1/400: mknod d1/dc/d16/d21/d27/d49/c87 0 2024-04-05T21:26:33.545 INFO:tasks.workunit.client.0.smithi121.stdout:9/49: write f4 [1357729,125706] 0 2024-04-05T21:26:33.546 INFO:tasks.workunit.client.1.smithi188.stdout:5/400: stat d6/ca 0 2024-04-05T21:26:33.555 INFO:tasks.workunit.client.0.smithi121.stdout:4/72: dwrite d5/fd [0,4194304] 0 2024-04-05T21:26:33.556 INFO:tasks.workunit.client.1.smithi188.stdout:1/401: dwrite d1/dc/f7a [0,4194304] 0 2024-04-05T21:26:33.556 INFO:tasks.workunit.client.0.smithi121.stdout:9/50: dwrite f5 [0,4194304] 0 2024-04-05T21:26:33.558 INFO:tasks.workunit.client.0.smithi121.stdout:4/73: write d5/f6 [2510354,50559] 0 2024-04-05T21:26:33.559 INFO:tasks.workunit.client.1.smithi188.stdout:5/401: rename d6/c1a to d6/d15/d5c/d7a/d44/c96 0 2024-04-05T21:26:33.562 INFO:tasks.workunit.client.0.smithi121.stdout:9/51: mknod d6/cb 0 2024-04-05T21:26:33.562 INFO:tasks.workunit.client.1.smithi188.stdout:5/402: creat d6/d17/f97 x:0 0 0 2024-04-05T21:26:33.564 INFO:tasks.workunit.client.0.smithi121.stdout:9/52: creat d6/fc x:0 0 0 2024-04-05T21:26:33.565 INFO:tasks.workunit.client.0.smithi121.stdout:4/74: dread d5/fa [0,4194304] 0 2024-04-05T21:26:33.565 INFO:tasks.workunit.client.0.smithi121.stdout:9/53: dread - d6/fc zero size 2024-04-05T21:26:33.566 INFO:tasks.workunit.client.1.smithi188.stdout:1/402: rename d1/d4 to d1/dc/d16/d46/d88 0 2024-04-05T21:26:33.567 INFO:tasks.workunit.client.0.smithi121.stdout:4/75: write d5/d11/f1c [378618,27863] 0 2024-04-05T21:26:33.568 INFO:tasks.workunit.client.0.smithi121.stdout:9/54: symlink d6/ld 0 2024-04-05T21:26:33.571 INFO:tasks.workunit.client.0.smithi121.stdout:5/97: dread f4 [0,4194304] 0 2024-04-05T21:26:33.573 INFO:tasks.workunit.client.0.smithi121.stdout:8/27: symlink l4 0 2024-04-05T21:26:33.577 INFO:tasks.workunit.client.1.smithi188.stdout:1/403: dwrite d1/dc/d16/d46/d88/d13/f6a [0,4194304] 0 2024-04-05T21:26:33.579 INFO:tasks.workunit.client.0.smithi121.stdout:6/54: getdents d1 0 2024-04-05T21:26:33.579 INFO:tasks.workunit.client.1.smithi188.stdout:1/404: truncate d1/dc/f7a 4364194 0 2024-04-05T21:26:33.581 INFO:tasks.workunit.client.0.smithi121.stdout:5/98: rename l9 to dd/l20 0 2024-04-05T21:26:33.585 INFO:tasks.workunit.client.0.smithi121.stdout:6/55: mknod d1/d2/dc/c11 0 2024-04-05T21:26:33.588 INFO:tasks.workunit.client.1.smithi188.stdout:5/403: rmdir d6/d17 39 2024-04-05T21:26:33.590 INFO:tasks.workunit.client.1.smithi188.stdout:1/405: creat d1/dc/f89 x:0 0 0 2024-04-05T21:26:33.590 INFO:tasks.workunit.client.1.smithi188.stdout:5/404: chown d6/d82/d40/d42/d81 15175503 1 2024-04-05T21:26:33.591 INFO:tasks.workunit.client.0.smithi121.stdout:6/56: chown d1/d2/d3 849157 1 2024-04-05T21:26:33.591 INFO:tasks.workunit.client.0.smithi121.stdout:2/76: write d2/d9/fe [173249,124350] 0 2024-04-05T21:26:33.591 INFO:tasks.workunit.client.0.smithi121.stdout:5/99: creat dd/de/f21 x:0 0 0 2024-04-05T21:26:33.591 INFO:tasks.workunit.client.0.smithi121.stdout:6/57: readlink d1/d2/d3/lf 0 2024-04-05T21:26:33.591 INFO:tasks.workunit.client.0.smithi121.stdout:7/51: write d0/d9/f12 [221896,66697] 0 2024-04-05T21:26:33.592 INFO:tasks.workunit.client.0.smithi121.stdout:7/52: chown d0/lb 72774 1 2024-04-05T21:26:33.593 INFO:tasks.workunit.client.0.smithi121.stdout:5/100: creat dd/de/f22 x:0 0 0 2024-04-05T21:26:33.593 INFO:tasks.workunit.client.0.smithi121.stdout:6/58: truncate d1/f10 4514426 0 2024-04-05T21:26:33.594 INFO:tasks.workunit.client.0.smithi121.stdout:7/53: stat d0/lb 0 2024-04-05T21:26:33.595 INFO:tasks.workunit.client.0.smithi121.stdout:0/75: write d7/dc/d10/f11 [592844,39786] 0 2024-04-05T21:26:33.596 INFO:tasks.workunit.client.0.smithi121.stdout:9/55: dwrite d6/fc [0,4194304] 0 2024-04-05T21:26:33.598 INFO:tasks.workunit.client.0.smithi121.stdout:7/54: symlink d0/l17 0 2024-04-05T21:26:33.599 INFO:tasks.workunit.client.0.smithi121.stdout:0/76: chown d7/fa 303 1 2024-04-05T21:26:33.599 INFO:tasks.workunit.client.0.smithi121.stdout:7/55: readlink d0/l7 0 2024-04-05T21:26:33.599 INFO:tasks.workunit.client.1.smithi188.stdout:1/406: dwrite d1/fb [0,4194304] 0 2024-04-05T21:26:33.601 INFO:tasks.workunit.client.0.smithi121.stdout:2/77: getdents d2 0 2024-04-05T21:26:33.603 INFO:tasks.workunit.client.0.smithi121.stdout:8/28: dwrite f3 [0,4194304] 0 2024-04-05T21:26:33.603 INFO:tasks.workunit.client.0.smithi121.stdout:9/56: fsync d6/f7 0 2024-04-05T21:26:33.604 INFO:tasks.workunit.client.0.smithi121.stdout:3/43: sync 2024-04-05T21:26:33.604 INFO:tasks.workunit.client.0.smithi121.stdout:5/101: link f6 dd/de/d19/f23 0 2024-04-05T21:26:33.606 INFO:tasks.workunit.client.1.smithi188.stdout:1/407: mknod d1/dc/d16/c8a 0 2024-04-05T21:26:33.609 INFO:tasks.workunit.client.1.smithi188.stdout:1/408: creat d1/d7/d4e/f8b x:0 0 0 2024-04-05T21:26:33.612 INFO:tasks.workunit.client.0.smithi121.stdout:7/56: write d0/d9/f12 [118389,18339] 0 2024-04-05T21:26:33.612 INFO:tasks.workunit.client.0.smithi121.stdout:5/102: chown dd/de/d19/d13 1613 1 2024-04-05T21:26:33.612 INFO:tasks.workunit.client.0.smithi121.stdout:2/78: write d2/d9/fe [50557,106464] 0 2024-04-05T21:26:33.612 INFO:tasks.workunit.client.0.smithi121.stdout:7/57: fdatasync d0/d9/fd 0 2024-04-05T21:26:33.612 INFO:tasks.workunit.client.0.smithi121.stdout:6/59: dread d1/d2/d3/d5/fd [0,4194304] 0 2024-04-05T21:26:33.614 INFO:tasks.workunit.client.0.smithi121.stdout:1/54: getdents d8 0 2024-04-05T21:26:33.614 INFO:tasks.workunit.client.0.smithi121.stdout:0/77: unlink d7/dc/c17 0 2024-04-05T21:26:33.616 INFO:tasks.workunit.client.1.smithi188.stdout:1/409: creat d1/dc/d16/d21/d27/d4a/f8c x:0 0 0 2024-04-05T21:26:33.617 INFO:tasks.workunit.client.1.smithi188.stdout:1/410: fsync d1/dc/d16/d46/d88/d3f/f40 0 2024-04-05T21:26:33.619 INFO:tasks.workunit.client.1.smithi188.stdout:1/411: write d1/dc/d16/d21/d27/d4a/f63 [1041375,113015] 0 2024-04-05T21:26:33.619 INFO:tasks.workunit.client.0.smithi121.stdout:9/57: chown d6/c9 0 1 2024-04-05T21:26:33.621 INFO:tasks.workunit.client.0.smithi121.stdout:0/78: dread d7/f8 [0,4194304] 0 2024-04-05T21:26:33.621 INFO:tasks.workunit.client.0.smithi121.stdout:7/58: fdatasync d0/f6 0 2024-04-05T21:26:33.624 INFO:tasks.workunit.client.0.smithi121.stdout:1/55: unlink c6 0 2024-04-05T21:26:33.625 INFO:tasks.workunit.client.0.smithi121.stdout:9/58: dread f5 [0,4194304] 0 2024-04-05T21:26:33.626 INFO:tasks.workunit.client.0.smithi121.stdout:8/29: symlink l5 0 2024-04-05T21:26:33.626 INFO:tasks.workunit.client.1.smithi188.stdout:1/412: dwrite d1/dc/d16/f6c [0,4194304] 0 2024-04-05T21:26:33.629 INFO:tasks.workunit.client.0.smithi121.stdout:6/60: mknod d1/c12 0 2024-04-05T21:26:33.629 INFO:tasks.workunit.client.0.smithi121.stdout:7/59: dread d0/f15 [0,4194304] 0 2024-04-05T21:26:33.630 INFO:tasks.workunit.client.0.smithi121.stdout:3/44: symlink d0/d3/de/l10 0 2024-04-05T21:26:33.631 INFO:tasks.workunit.client.1.smithi188.stdout:1/413: creat d1/dc/d16/d46/d88/d13/f8d x:0 0 0 2024-04-05T21:26:33.631 INFO:tasks.workunit.client.1.smithi188.stdout:1/414: chown d1/dc/d16/c8a 76303 1 2024-04-05T21:26:33.632 INFO:tasks.workunit.client.1.smithi188.stdout:1/415: chown d1/dc/f1f 591 1 2024-04-05T21:26:33.633 INFO:tasks.workunit.client.1.smithi188.stdout:1/416: truncate d1/dc/d16/d46/d88/d13/f50 1047710 0 2024-04-05T21:26:33.634 INFO:tasks.workunit.client.1.smithi188.stdout:1/417: chown d1/d7/d4e/f8b 1458 1 2024-04-05T21:26:33.634 INFO:tasks.workunit.client.0.smithi121.stdout:2/79: creat d2/d9/d12/f15 x:0 0 0 2024-04-05T21:26:33.637 INFO:tasks.workunit.client.0.smithi121.stdout:0/79: creat d7/f1a x:0 0 0 2024-04-05T21:26:33.637 INFO:tasks.workunit.client.1.smithi188.stdout:1/418: link d1/dc/d16/d46/d88/d3f/c71 d1/d7/d4e/c8e 0 2024-04-05T21:26:33.639 INFO:tasks.workunit.client.0.smithi121.stdout:6/61: dread d1/d2/d3/d5/fd [0,4194304] 0 2024-04-05T21:26:33.641 INFO:tasks.workunit.client.0.smithi121.stdout:1/56: creat d8/ff x:0 0 0 2024-04-05T21:26:33.642 INFO:tasks.workunit.client.0.smithi121.stdout:4/76: dwrite d5/dc/fe [0,4194304] 0 2024-04-05T21:26:33.642 INFO:tasks.workunit.client.1.smithi188.stdout:1/419: getdents d1/dc/d16 0 2024-04-05T21:26:33.644 INFO:tasks.workunit.client.0.smithi121.stdout:4/77: fdatasync d5/f1d 0 2024-04-05T21:26:33.644 INFO:tasks.workunit.client.0.smithi121.stdout:1/57: dread - d8/fb zero size 2024-04-05T21:26:33.644 INFO:tasks.workunit.client.0.smithi121.stdout:1/58: readlink - no filename 2024-04-05T21:26:33.644 INFO:tasks.workunit.client.0.smithi121.stdout:7/60: dread d0/f15 [0,4194304] 0 2024-04-05T21:26:33.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.644+0000 7f06d0a99700 1 -- 172.21.15.121:0/1576185152 --> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f06ac000bf0 con 0x7f06b806ca10 2024-04-05T21:26:33.645 INFO:tasks.workunit.client.1.smithi188.stdout:1/420: creat d1/dc/d16/d46/d88/d86/f8f x:0 0 0 2024-04-05T21:26:33.645 INFO:tasks.workunit.client.0.smithi121.stdout:3/45: unlink d0/d3/de/l10 0 2024-04-05T21:26:33.645 INFO:tasks.workunit.client.0.smithi121.stdout:0/80: truncate d7/f13 737256 0 2024-04-05T21:26:33.646 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:26:33.646 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:26:33.646 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:26:33.646 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T21:26:33.646 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [], 2024-04-05T21:26:33.646 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "1/2 daemons upgraded", 2024-04-05T21:26:33.646 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "", 2024-04-05T21:26:33.646 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:26:33.646 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:26:33.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.645+0000 7f06c37fe700 1 -- 172.21.15.121:0/1576185152 <== mgr.24327 v2:172.21.15.188:6828/2212394428 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f06ac000bf0 con 0x7f06b806ca10 2024-04-05T21:26:33.647 INFO:tasks.workunit.client.1.smithi188.stdout:1/421: mkdir d1/dc/d16/d21/d27/d90 0 2024-04-05T21:26:33.647 INFO:tasks.workunit.client.0.smithi121.stdout:1/59: write d8/fc [795811,105880] 0 2024-04-05T21:26:33.648 INFO:tasks.workunit.client.0.smithi121.stdout:0/81: dread - d7/dc/ff zero size 2024-04-05T21:26:33.648 INFO:tasks.workunit.client.1.smithi188.stdout:1/422: write d1/dc/f1b [4755752,60965] 0 2024-04-05T21:26:33.653 INFO:tasks.workunit.client.0.smithi121.stdout:2/80: mknod d2/c16 0 2024-04-05T21:26:33.654 INFO:tasks.workunit.client.0.smithi121.stdout:6/62: creat d1/d2/f13 x:0 0 0 2024-04-05T21:26:33.654 INFO:tasks.workunit.client.0.smithi121.stdout:1/60: write f5 [2108682,47118] 0 2024-04-05T21:26:33.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.654+0000 7f06d0a99700 1 -- 172.21.15.121:0/1576185152 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f06b806ca10 msgr2=0x7f06b806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:33.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.654+0000 7f06d0a99700 1 --2- 172.21.15.121:0/1576185152 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f06b806ca10 0x7f06b806eed0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f06b4005950 tx=0x7f06b40058e0 comp rx=0 tx=0).stop 2024-04-05T21:26:33.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.654+0000 7f06d0a99700 1 -- 172.21.15.121:0/1576185152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc0ffad0 msgr2=0x7f06cc196300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:26:33.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.654+0000 7f06d0a99700 1 --2- 172.21.15.121:0/1576185152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc0ffad0 0x7f06cc196300 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f06c400b070 tx=0x7f06c40092e0 comp rx=0 tx=0).stop 2024-04-05T21:26:33.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.654+0000 7f06d0a99700 1 -- 172.21.15.121:0/1576185152 shutdown_connections 2024-04-05T21:26:33.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.654+0000 7f06d0a99700 1 --2- 172.21.15.121:0/1576185152 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f06b806ca10 0x7f06b806eed0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:33.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.654+0000 7f06d0a99700 1 --2- 172.21.15.121:0/1576185152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06cc0ffad0 0x7f06cc196300 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:33.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.654+0000 7f06d0a99700 1 --2- 172.21.15.121:0/1576185152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06cc100cd0 0x7f06cc196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:26:33.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.654+0000 7f06d0a99700 1 -- 172.21.15.121:0/1576185152 >> 172.21.15.121:0/1576185152 conn(0x7f06cc0fb050 msgr2=0x7f06cc103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:26:33.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.654+0000 7f06d0a99700 1 -- 172.21.15.121:0/1576185152 shutdown_connections 2024-04-05T21:26:33.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:26:33.654+0000 7f06d0a99700 1 -- 172.21.15.121:0/1576185152 wait complete. 2024-04-05T21:26:33.656 INFO:tasks.workunit.client.1.smithi188.stdout:1/423: dwrite d1/dc/d16/d21/f81 [0,4194304] 0 2024-04-05T21:26:33.657 INFO:tasks.workunit.client.0.smithi121.stdout:6/63: readlink d1/d2/d3/d5/l8 0 2024-04-05T21:26:33.657 INFO:tasks.workunit.client.0.smithi121.stdout:4/78: dread d5/d11/f1c [0,4194304] 0 2024-04-05T21:26:33.658 INFO:tasks.workunit.client.0.smithi121.stdout:7/61: write d0/f15 [2704646,32309] 0 2024-04-05T21:26:33.659 INFO:tasks.workunit.client.0.smithi121.stdout:1/61: write f4 [2139012,89771] 0 2024-04-05T21:26:33.659 INFO:tasks.workunit.client.0.smithi121.stdout:8/30: creat f6 x:0 0 0 2024-04-05T21:26:33.662 INFO:tasks.workunit.client.0.smithi121.stdout:2/81: mkdir d2/d9/d17 0 2024-04-05T21:26:33.664 INFO:tasks.workunit.client.0.smithi121.stdout:3/46: symlink d0/l11 0 2024-04-05T21:26:33.666 INFO:tasks.workunit.client.1.smithi188.stdout:1/424: dwrite d1/dc/d16/d46/d88/d13/f50 [0,4194304] 0 2024-04-05T21:26:33.668 INFO:tasks.workunit.client.0.smithi121.stdout:0/82: dwrite d7/fa [4194304,4194304] 0 2024-04-05T21:26:33.668 INFO:tasks.workunit.client.0.smithi121.stdout:0/83: truncate d7/f1a 347371 0 2024-04-05T21:26:33.670 INFO:tasks.workunit.client.1.smithi188.stdout:1/425: mknod d1/dc/d16/d21/c91 0 2024-04-05T21:26:33.671 INFO:tasks.workunit.client.0.smithi121.stdout:7/62: mkdir d0/d9/d18 0 2024-04-05T21:26:33.671 INFO:tasks.workunit.client.0.smithi121.stdout:6/64: mknod d1/d2/d3/d5/c14 0 2024-04-05T21:26:33.672 INFO:tasks.workunit.client.0.smithi121.stdout:0/84: write d7/f13 [1698066,98515] 0 2024-04-05T21:26:33.673 INFO:tasks.workunit.client.0.smithi121.stdout:3/47: symlink d0/d3/de/l12 0 2024-04-05T21:26:33.676 INFO:tasks.workunit.client.0.smithi121.stdout:8/31: symlink l7 0 2024-04-05T21:26:33.678 INFO:tasks.workunit.client.1.smithi188.stdout:1/426: chown d1/dc/d16/d46/f51 40450 1 2024-04-05T21:26:33.678 INFO:tasks.workunit.client.0.smithi121.stdout:4/79: dread d5/f8 [0,4194304] 0 2024-04-05T21:26:33.682 INFO:tasks.workunit.client.0.smithi121.stdout:8/32: chown l7 58 1 2024-04-05T21:26:33.682 INFO:tasks.workunit.client.1.smithi188.stdout:1/427: getdents d1/dc/d16/d46/d88/d13 0 2024-04-05T21:26:33.683 INFO:tasks.workunit.client.1.smithi188.stdout:1/428: chown d1/dc/d16/d21/d27/f29 3 1 2024-04-05T21:26:33.683 INFO:tasks.workunit.client.0.smithi121.stdout:1/62: dwrite d8/fa [0,4194304] 0 2024-04-05T21:26:33.684 INFO:tasks.workunit.client.1.smithi188.stdout:1/429: readlink d1/l7b 0 2024-04-05T21:26:33.684 INFO:tasks.workunit.client.0.smithi121.stdout:3/48: link d0/f7 d0/d3/f13 0 2024-04-05T21:26:33.685 INFO:tasks.workunit.client.1.smithi188.stdout:1/430: rmdir d1/dc/d16 39 2024-04-05T21:26:33.687 INFO:tasks.workunit.client.1.smithi188.stdout:1/431: read d1/dc/d16/d46/d88/d3f/f4b [679572,29209] 0 2024-04-05T21:26:33.691 INFO:tasks.workunit.client.1.smithi188.stdout:1/432: creat d1/dc/d16/f92 x:0 0 0 2024-04-05T21:26:33.695 INFO:tasks.workunit.client.0.smithi121.stdout:9/59: dwrite f2 [0,4194304] 0 2024-04-05T21:26:33.698 INFO:tasks.workunit.client.1.smithi188.stdout:1/433: chown d1/dc/d16/d46/d88/d13/d60/c85 1356 1 2024-04-05T21:26:33.699 INFO:tasks.workunit.client.0.smithi121.stdout:8/33: creat f8 x:0 0 0 2024-04-05T21:26:33.699 INFO:tasks.workunit.client.0.smithi121.stdout:6/65: dwrite d1/f10 [0,4194304] 0 2024-04-05T21:26:33.700 INFO:tasks.workunit.client.1.smithi188.stdout:1/434: write d1/dc/d16/d21/d27/d49/f59 [816759,18925] 0 2024-04-05T21:26:33.701 INFO:tasks.workunit.client.1.smithi188.stdout:1/435: fsync d1/dc/f11 0 2024-04-05T21:26:33.704 INFO:tasks.workunit.client.1.smithi188.stdout:1/436: mknod d1/dc/d16/d21/d27/d90/c93 0 2024-04-05T21:26:33.705 INFO:tasks.workunit.client.1.smithi188.stdout:1/437: fsync d1/dc/d16/d21/f68 0 2024-04-05T21:26:33.706 INFO:tasks.workunit.client.1.smithi188.stdout:1/438: stat d1/f48 0 2024-04-05T21:26:33.706 INFO:tasks.workunit.client.0.smithi121.stdout:8/34: write f8 [579273,51459] 0 2024-04-05T21:26:33.707 INFO:tasks.workunit.client.0.smithi121.stdout:7/63: dread d0/f3 [0,4194304] 0 2024-04-05T21:26:33.708 INFO:tasks.workunit.client.0.smithi121.stdout:4/80: dwrite d5/d11/f17 [0,4194304] 0 2024-04-05T21:26:33.708 INFO:tasks.workunit.client.0.smithi121.stdout:3/49: creat d0/f14 x:0 0 0 2024-04-05T21:26:33.710 INFO:tasks.workunit.client.0.smithi121.stdout:0/85: write d7/f9 [1803555,8864] 0 2024-04-05T21:26:33.710 INFO:tasks.workunit.client.0.smithi121.stdout:2/82: dread d2/d9/ff [0,4194304] 0 2024-04-05T21:26:33.714 INFO:tasks.workunit.client.0.smithi121.stdout:9/60: rmdir d6 39 2024-04-05T21:26:33.714 INFO:tasks.workunit.client.0.smithi121.stdout:1/63: dwrite f4 [0,4194304] 0 2024-04-05T21:26:33.714 INFO:tasks.workunit.client.0.smithi121.stdout:8/35: chown f8 2971567 1 2024-04-05T21:26:33.715 INFO:tasks.workunit.client.0.smithi121.stdout:7/64: write d0/f15 [5199819,31994] 0 2024-04-05T21:26:33.715 INFO:tasks.workunit.client.0.smithi121.stdout:3/50: dread - d0/f2 zero size 2024-04-05T21:26:33.716 INFO:tasks.workunit.client.0.smithi121.stdout:3/51: readlink d0/d3/de/l12 0 2024-04-05T21:26:33.718 INFO:tasks.workunit.client.0.smithi121.stdout:4/81: mkdir d5/d11/d20 0 2024-04-05T21:26:33.719 INFO:tasks.workunit.client.0.smithi121.stdout:1/64: mkdir d8/d10 0 2024-04-05T21:26:33.719 INFO:tasks.workunit.client.0.smithi121.stdout:2/83: dread - d2/f14 zero size 2024-04-05T21:26:33.723 INFO:tasks.workunit.client.0.smithi121.stdout:4/82: creat d5/d11/f21 x:0 0 0 2024-04-05T21:26:33.723 INFO:tasks.workunit.client.0.smithi121.stdout:0/86: fsync f2 0 2024-04-05T21:26:33.724 INFO:tasks.workunit.client.1.smithi188.stdout:2/418: sync 2024-04-05T21:26:33.725 INFO:tasks.workunit.client.0.smithi121.stdout:3/52: creat d0/f15 x:0 0 0 2024-04-05T21:26:33.727 INFO:tasks.workunit.client.0.smithi121.stdout:0/87: dread - d7/f18 zero size 2024-04-05T21:26:33.730 INFO:tasks.workunit.client.0.smithi121.stdout:2/84: link d2/d9/fe d2/d9/d17/f18 0 2024-04-05T21:26:33.732 INFO:tasks.workunit.client.0.smithi121.stdout:7/65: dwrite d0/d9/f12 [0,4194304] 0 2024-04-05T21:26:33.732 INFO:tasks.workunit.client.1.smithi188.stdout:2/419: rename d5/db/dd/d36/l3a to d5/db/dd/d36/d3c/l80 0 2024-04-05T21:26:33.734 INFO:tasks.workunit.client.0.smithi121.stdout:4/83: link d5/f6 d5/d11/f22 0 2024-04-05T21:26:33.735 INFO:tasks.workunit.client.0.smithi121.stdout:8/36: dread f8 [0,4194304] 0 2024-04-05T21:26:33.736 INFO:tasks.workunit.client.1.smithi188.stdout:2/420: creat d5/db/dd/d36/d3d/d46/f81 x:0 0 0 2024-04-05T21:26:33.736 INFO:tasks.workunit.client.1.smithi188.stdout:1/439: dread d1/d7/d4e/f6d [0,4194304] 0 2024-04-05T21:26:33.738 INFO:tasks.workunit.client.0.smithi121.stdout:9/61: dwrite f2 [0,4194304] 0 2024-04-05T21:26:33.738 INFO:tasks.workunit.client.1.smithi188.stdout:2/421: mknod d5/db/d35/d3b/d6b/c82 0 2024-04-05T21:26:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:33 smithi121 ceph-mon[39723]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:33 smithi121 ceph-mon[39723]: pgmap v15: 65 pgs: 65 active+clean; 834 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 8.4 MiB/s rd, 90 MiB/s wr, 148 op/s 2024-04-05T21:26:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:33 smithi121 ceph-mon[39723]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:33 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/4226047468' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:26:33.740 INFO:tasks.workunit.client.0.smithi121.stdout:4/84: rename d5 to d5/d11/d23 22 2024-04-05T21:26:33.740 INFO:tasks.workunit.client.0.smithi121.stdout:2/85: write d2/d9/fb [239984,52296] 0 2024-04-05T21:26:33.740 INFO:tasks.workunit.client.1.smithi188.stdout:1/440: symlink d1/dc/d16/d21/l94 0 2024-04-05T21:26:33.740 INFO:tasks.workunit.client.1.smithi188.stdout:1/441: write d1/d7/d4e/f8b [1028593,20765] 0 2024-04-05T21:26:33.741 INFO:tasks.workunit.client.1.smithi188.stdout:1/442: write d1/d7/d4e/d5f/f76 [1002279,92970] 0 2024-04-05T21:26:33.744 INFO:tasks.workunit.client.0.smithi121.stdout:1/65: dwrite f2 [0,4194304] 0 2024-04-05T21:26:33.745 INFO:tasks.workunit.client.1.smithi188.stdout:1/443: dread d1/dc/d16/d21/d27/d49/f59 [0,4194304] 0 2024-04-05T21:26:33.749 INFO:tasks.workunit.client.0.smithi121.stdout:2/86: dread - d2/d9/d12/f15 zero size 2024-04-05T21:26:33.750 INFO:tasks.workunit.client.0.smithi121.stdout:8/37: unlink f2 0 2024-04-05T21:26:33.751 INFO:tasks.workunit.client.0.smithi121.stdout:4/85: symlink d5/dc/l24 0 2024-04-05T21:26:33.753 INFO:tasks.workunit.client.1.smithi188.stdout:1/444: getdents d1/dc/d16/d46/d88/d86 0 2024-04-05T21:26:33.753 INFO:tasks.workunit.client.0.smithi121.stdout:6/66: truncate d1/f10 3183060 0 2024-04-05T21:26:33.753 INFO:tasks.workunit.client.0.smithi121.stdout:2/87: dread - d2/f14 zero size 2024-04-05T21:26:33.754 INFO:tasks.workunit.client.1.smithi188.stdout:1/445: write d1/dc/d16/d46/d88/d13/f4f [61689,62134] 0 2024-04-05T21:26:33.755 INFO:tasks.workunit.client.0.smithi121.stdout:0/88: dwrite d7/dc/f15 [0,4194304] 0 2024-04-05T21:26:33.755 INFO:tasks.workunit.client.0.smithi121.stdout:4/86: chown d5/fa 504948 1 2024-04-05T21:26:33.756 INFO:tasks.workunit.client.0.smithi121.stdout:9/62: rename d6/ca to d6/ce 0 2024-04-05T21:26:33.757 INFO:tasks.workunit.client.1.smithi188.stdout:1/446: write d1/dc/d16/d46/d88/d13/f6a [1045267,22542] 0 2024-04-05T21:26:33.757 INFO:tasks.workunit.client.0.smithi121.stdout:3/53: write d0/d3/f4 [1325779,51865] 0 2024-04-05T21:26:33.758 INFO:tasks.workunit.client.0.smithi121.stdout:8/38: symlink l9 0 2024-04-05T21:26:33.759 INFO:tasks.workunit.client.1.smithi188.stdout:1/447: mkdir d1/dc/d16/d46/d95 0 2024-04-05T21:26:33.760 INFO:tasks.workunit.client.0.smithi121.stdout:4/87: stat d5/f8 0 2024-04-05T21:26:33.761 INFO:tasks.workunit.client.0.smithi121.stdout:9/63: write f2 [320984,28180] 0 2024-04-05T21:26:33.761 INFO:tasks.workunit.client.0.smithi121.stdout:1/66: rmdir d8/dd 0 2024-04-05T21:26:33.762 INFO:tasks.workunit.client.0.smithi121.stdout:7/66: dwrite d0/d9/fe [4194304,4194304] 0 2024-04-05T21:26:33.764 INFO:tasks.workunit.client.0.smithi121.stdout:7/67: chown d0/d9/d18 33250 1 2024-04-05T21:26:33.764 INFO:tasks.workunit.client.0.smithi121.stdout:9/64: chown l3 272587 1 2024-04-05T21:26:33.764 INFO:tasks.workunit.client.0.smithi121.stdout:1/67: dread - d8/ff zero size 2024-04-05T21:26:33.765 INFO:tasks.workunit.client.0.smithi121.stdout:6/67: rename d1/d2/d3/lf to d1/l15 0 2024-04-05T21:26:33.767 INFO:tasks.workunit.client.0.smithi121.stdout:3/54: mknod d0/d3/c16 0 2024-04-05T21:26:33.769 INFO:tasks.workunit.client.0.smithi121.stdout:0/89: rename d7/f8 to d7/dc/d10/f1b 0 2024-04-05T21:26:33.771 INFO:tasks.workunit.client.0.smithi121.stdout:0/90: chown d7/dc 193778 1 2024-04-05T21:26:33.772 INFO:tasks.workunit.client.0.smithi121.stdout:2/88: creat d2/d9/d17/f19 x:0 0 0 2024-04-05T21:26:33.778 INFO:tasks.workunit.client.0.smithi121.stdout:7/68: dwrite d0/f8 [0,4194304] 0 2024-04-05T21:26:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:33 smithi188 ceph-mon[49715]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:33 smithi188 ceph-mon[49715]: pgmap v15: 65 pgs: 65 active+clean; 834 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 8.4 MiB/s rd, 90 MiB/s wr, 148 op/s 2024-04-05T21:26:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:33 smithi188 ceph-mon[49715]: from='client.14626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:33 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/4226047468' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:26:33.780 INFO:tasks.workunit.client.0.smithi121.stdout:1/68: creat d8/f11 x:0 0 0 2024-04-05T21:26:33.780 INFO:tasks.workunit.client.0.smithi121.stdout:9/65: mkdir d6/df 0 2024-04-05T21:26:33.781 INFO:tasks.workunit.client.0.smithi121.stdout:1/69: write d8/fb [349490,41715] 0 2024-04-05T21:26:33.782 INFO:tasks.workunit.client.0.smithi121.stdout:3/55: creat d0/d3/de/f17 x:0 0 0 2024-04-05T21:26:33.783 INFO:tasks.workunit.client.0.smithi121.stdout:1/70: dread - d8/ff zero size 2024-04-05T21:26:33.784 INFO:tasks.workunit.client.0.smithi121.stdout:3/56: dread - d0/f2 zero size 2024-04-05T21:26:33.786 INFO:tasks.workunit.client.0.smithi121.stdout:0/91: mkdir d7/d1c 0 2024-04-05T21:26:33.788 INFO:tasks.workunit.client.0.smithi121.stdout:2/89: fdatasync d2/d9/ff 0 2024-04-05T21:26:33.790 INFO:tasks.workunit.client.0.smithi121.stdout:7/69: symlink d0/d9/l19 0 2024-04-05T21:26:33.793 INFO:tasks.workunit.client.0.smithi121.stdout:1/71: mknod d8/c12 0 2024-04-05T21:26:33.795 INFO:tasks.workunit.client.0.smithi121.stdout:3/57: symlink d0/d3/de/l18 0 2024-04-05T21:26:33.796 INFO:tasks.workunit.client.0.smithi121.stdout:6/68: dwrite d1/f10 [4194304,4194304] 0 2024-04-05T21:26:33.798 INFO:tasks.workunit.client.0.smithi121.stdout:2/90: creat d2/f1a x:0 0 0 2024-04-05T21:26:33.800 INFO:tasks.workunit.client.0.smithi121.stdout:5/103: sync 2024-04-05T21:26:33.804 INFO:tasks.workunit.client.0.smithi121.stdout:9/66: getdents d6/df 0 2024-04-05T21:26:33.805 INFO:tasks.workunit.client.0.smithi121.stdout:7/70: dwrite d0/f15 [0,4194304] 0 2024-04-05T21:26:33.806 INFO:tasks.workunit.client.1.smithi188.stdout:1/448: dread d1/dc/d16/d21/f26 [0,4194304] 0 2024-04-05T21:26:33.809 INFO:tasks.workunit.client.1.smithi188.stdout:1/449: mknod d1/d7/d4e/c96 0 2024-04-05T21:26:33.810 INFO:tasks.workunit.client.0.smithi121.stdout:4/88: dwrite d5/f16 [0,4194304] 0 2024-04-05T21:26:33.810 INFO:tasks.workunit.client.0.smithi121.stdout:1/72: unlink d8/fa 0 2024-04-05T21:26:33.811 INFO:tasks.workunit.client.1.smithi188.stdout:1/450: fdatasync d1/dc/d16/d46/d88/d13/f6a 0 2024-04-05T21:26:33.812 INFO:tasks.workunit.client.0.smithi121.stdout:1/73: read d8/fc [847414,88500] 0 2024-04-05T21:26:33.813 INFO:tasks.workunit.client.0.smithi121.stdout:3/58: mknod d0/d3/de/c19 0 2024-04-05T21:26:33.813 INFO:tasks.workunit.client.0.smithi121.stdout:1/74: dread - d8/ff zero size 2024-04-05T21:26:33.815 INFO:tasks.workunit.client.0.smithi121.stdout:2/91: symlink d2/d9/l1b 0 2024-04-05T21:26:33.816 INFO:tasks.workunit.client.1.smithi188.stdout:1/451: creat d1/dc/d16/d2d/f97 x:0 0 0 2024-04-05T21:26:33.819 INFO:tasks.workunit.client.1.smithi188.stdout:1/452: symlink d1/dc/d16/d46/d88/d13/d60/l98 0 2024-04-05T21:26:33.820 INFO:tasks.workunit.client.0.smithi121.stdout:0/92: mkdir d7/d1c/d1d 0 2024-04-05T21:26:33.820 INFO:tasks.workunit.client.0.smithi121.stdout:5/104: stat dd/c18 0 2024-04-05T21:26:33.821 INFO:tasks.workunit.client.0.smithi121.stdout:0/93: stat d7/f1a 0 2024-04-05T21:26:33.822 INFO:tasks.workunit.client.1.smithi188.stdout:1/453: mkdir d1/dc/d16/d46/d95/d99 0 2024-04-05T21:26:33.823 INFO:tasks.workunit.client.1.smithi188.stdout:1/454: truncate d1/dc/d16/d21/f7d 737842 0 2024-04-05T21:26:33.824 INFO:tasks.workunit.client.0.smithi121.stdout:0/94: write d7/dc/f14 [795704,69666] 0 2024-04-05T21:26:33.827 INFO:tasks.workunit.client.0.smithi121.stdout:1/75: dwrite d8/fb [0,4194304] 0 2024-04-05T21:26:33.828 INFO:tasks.workunit.client.1.smithi188.stdout:1/455: truncate d1/dc/d16/d21/f68 2265642 0 2024-04-05T21:26:33.830 INFO:tasks.workunit.client.1.smithi188.stdout:1/456: dread d1/dc/f56 [0,4194304] 0 2024-04-05T21:26:33.831 INFO:tasks.workunit.client.0.smithi121.stdout:8/39: getdents . 0 2024-04-05T21:26:33.832 INFO:tasks.workunit.client.0.smithi121.stdout:2/92: mkdir d2/d9/d1c 0 2024-04-05T21:26:33.832 INFO:tasks.workunit.client.0.smithi121.stdout:6/69: getdents d1/d2/d3/d4/de 0 2024-04-05T21:26:33.834 INFO:tasks.workunit.client.0.smithi121.stdout:9/67: symlink d6/df/l10 0 2024-04-05T21:26:33.834 INFO:tasks.workunit.client.0.smithi121.stdout:0/95: symlink d7/l1e 0 2024-04-05T21:26:33.839 INFO:tasks.workunit.client.0.smithi121.stdout:2/93: write d2/f14 [248641,1918] 0 2024-04-05T21:26:33.840 INFO:tasks.workunit.client.0.smithi121.stdout:7/71: write d0/f3 [3425530,10414] 0 2024-04-05T21:26:33.840 INFO:tasks.workunit.client.0.smithi121.stdout:6/70: write d1/f10 [4582658,74988] 0 2024-04-05T21:26:33.841 INFO:tasks.workunit.client.0.smithi121.stdout:9/68: symlink d6/l11 0 2024-04-05T21:26:33.845 INFO:tasks.workunit.client.0.smithi121.stdout:0/96: truncate d7/f1a 762519 0 2024-04-05T21:26:33.850 INFO:tasks.workunit.client.0.smithi121.stdout:2/94: symlink d2/d9/l1d 0 2024-04-05T21:26:33.850 INFO:tasks.workunit.client.0.smithi121.stdout:7/72: write d0/fc [1195888,8597] 0 2024-04-05T21:26:33.852 INFO:tasks.workunit.client.0.smithi121.stdout:9/69: mknod d6/df/c12 0 2024-04-05T21:26:33.852 INFO:tasks.workunit.client.0.smithi121.stdout:1/76: dwrite f5 [4194304,4194304] 0 2024-04-05T21:26:33.853 INFO:tasks.workunit.client.0.smithi121.stdout:0/97: write d7/f13 [804417,1262] 0 2024-04-05T21:26:33.857 INFO:tasks.workunit.client.0.smithi121.stdout:4/89: dwrite d5/d11/f12 [0,4194304] 0 2024-04-05T21:26:33.858 INFO:tasks.workunit.client.1.smithi188.stdout:1/457: dread d1/d7/f66 [0,4194304] 0 2024-04-05T21:26:33.858 INFO:tasks.workunit.client.1.smithi188.stdout:1/458: dread - d1/dc/d16/d2d/f7e zero size 2024-04-05T21:26:33.858 INFO:tasks.workunit.client.0.smithi121.stdout:0/98: stat d7/l1e 0 2024-04-05T21:26:33.860 INFO:tasks.workunit.client.0.smithi121.stdout:2/95: write d2/d9/d12/f15 [1042208,74090] 0 2024-04-05T21:26:33.860 INFO:tasks.workunit.client.0.smithi121.stdout:4/90: unlink l0 0 2024-04-05T21:26:33.862 INFO:tasks.workunit.client.0.smithi121.stdout:1/77: chown d8/c12 4682261 1 2024-04-05T21:26:33.863 INFO:tasks.workunit.client.1.smithi188.stdout:1/459: link d1/dc/f7a d1/dc/d16/d2d/f9a 0 2024-04-05T21:26:33.865 INFO:tasks.workunit.client.0.smithi121.stdout:8/40: dwrite f6 [0,4194304] 0 2024-04-05T21:26:33.866 INFO:tasks.workunit.client.0.smithi121.stdout:0/99: creat d7/d1c/f1f x:0 0 0 2024-04-05T21:26:33.866 INFO:tasks.workunit.client.1.smithi188.stdout:1/460: creat d1/dc/d16/d21/d27/d49/f9b x:0 0 0 2024-04-05T21:26:33.868 INFO:tasks.workunit.client.0.smithi121.stdout:0/100: dread - d7/f18 zero size 2024-04-05T21:26:33.869 INFO:tasks.workunit.client.0.smithi121.stdout:2/96: fdatasync d2/d9/fe 0 2024-04-05T21:26:33.869 INFO:tasks.workunit.client.1.smithi188.stdout:1/461: mknod d1/dc/d16/d21/d27/d4a/c9c 0 2024-04-05T21:26:33.869 INFO:tasks.workunit.client.0.smithi121.stdout:1/78: unlink d8/c12 0 2024-04-05T21:26:33.870 INFO:tasks.workunit.client.0.smithi121.stdout:6/71: dread d1/f10 [4194304,4194304] 0 2024-04-05T21:26:33.873 INFO:tasks.workunit.client.1.smithi188.stdout:1/462: dread d1/dc/d16/d21/f81 [0,4194304] 0 2024-04-05T21:26:33.875 INFO:tasks.workunit.client.0.smithi121.stdout:7/73: dwrite d0/f8 [0,4194304] 0 2024-04-05T21:26:33.876 INFO:tasks.workunit.client.1.smithi188.stdout:1/463: symlink d1/dc/d16/d21/d27/d90/l9d 0 2024-04-05T21:26:33.877 INFO:tasks.workunit.client.0.smithi121.stdout:2/97: fdatasync d2/f1a 0 2024-04-05T21:26:33.877 INFO:tasks.workunit.client.0.smithi121.stdout:9/70: write f5 [5632147,24175] 0 2024-04-05T21:26:33.878 INFO:tasks.workunit.client.0.smithi121.stdout:6/72: fsync d1/d2/d3/d5/fd 0 2024-04-05T21:26:33.878 INFO:tasks.workunit.client.1.smithi188.stdout:1/464: mknod d1/dc/c9e 0 2024-04-05T21:26:33.879 INFO:tasks.workunit.client.0.smithi121.stdout:7/74: fdatasync d0/f15 0 2024-04-05T21:26:33.880 INFO:tasks.workunit.client.0.smithi121.stdout:1/79: dread - d8/ff zero size 2024-04-05T21:26:33.880 INFO:tasks.workunit.client.0.smithi121.stdout:1/80: readlink - no filename 2024-04-05T21:26:33.881 INFO:tasks.workunit.client.0.smithi121.stdout:4/91: link d5/dc/l24 d5/dc/l25 0 2024-04-05T21:26:33.882 INFO:tasks.workunit.client.0.smithi121.stdout:6/73: read - d1/d2/f13 zero size 2024-04-05T21:26:33.882 INFO:tasks.workunit.client.0.smithi121.stdout:0/101: creat d7/dc/d10/f20 x:0 0 0 2024-04-05T21:26:33.884 INFO:tasks.workunit.client.0.smithi121.stdout:9/71: creat d6/df/f13 x:0 0 0 2024-04-05T21:26:33.884 INFO:tasks.workunit.client.0.smithi121.stdout:4/92: chown d5/f9 87 1 2024-04-05T21:26:33.885 INFO:tasks.workunit.client.0.smithi121.stdout:0/102: dread - d7/dc/d10/f20 zero size 2024-04-05T21:26:33.886 INFO:tasks.workunit.client.0.smithi121.stdout:7/75: mknod d0/d9/c1a 0 2024-04-05T21:26:33.886 INFO:tasks.workunit.client.1.smithi188.stdout:1/465: dwrite d1/dc/d16/d46/d88/d13/f50 [0,4194304] 0 2024-04-05T21:26:33.886 INFO:tasks.workunit.client.0.smithi121.stdout:1/81: symlink d8/d10/l13 0 2024-04-05T21:26:33.888 INFO:tasks.workunit.client.0.smithi121.stdout:0/103: truncate d7/dc/d10/f20 344717 0 2024-04-05T21:26:33.888 INFO:tasks.workunit.client.1.smithi188.stdout:1/466: write d1/fb [5135626,31871] 0 2024-04-05T21:26:33.889 INFO:tasks.workunit.client.0.smithi121.stdout:1/82: truncate d8/f11 4271 0 2024-04-05T21:26:33.892 INFO:tasks.workunit.client.1.smithi188.stdout:1/467: mknod d1/dc/d16/d21/d27/d90/c9f 0 2024-04-05T21:26:33.893 INFO:tasks.workunit.client.1.smithi188.stdout:1/468: chown d1/dc/d16/d21/d27/d4a/f63 83594643 1 2024-04-05T21:26:33.894 INFO:tasks.workunit.client.1.smithi188.stdout:1/469: dread - d1/f48 zero size 2024-04-05T21:26:33.895 INFO:tasks.workunit.client.0.smithi121.stdout:1/83: write f5 [7625237,95252] 0 2024-04-05T21:26:33.895 INFO:tasks.workunit.client.0.smithi121.stdout:7/76: rename d0/c1 to d0/d9/c1b 0 2024-04-05T21:26:33.898 INFO:tasks.workunit.client.0.smithi121.stdout:2/98: truncate d2/f11 992387 0 2024-04-05T21:26:33.900 INFO:tasks.workunit.client.0.smithi121.stdout:0/104: rename c6 to d7/dc/d10/c21 0 2024-04-05T21:26:33.901 INFO:tasks.workunit.client.0.smithi121.stdout:1/84: mkdir d8/d10/d14 0 2024-04-05T21:26:33.902 INFO:tasks.workunit.client.0.smithi121.stdout:9/72: getdents d6 0 2024-04-05T21:26:33.903 INFO:tasks.workunit.client.0.smithi121.stdout:2/99: mkdir d2/d1e 0 2024-04-05T21:26:33.904 INFO:tasks.workunit.client.0.smithi121.stdout:0/105: symlink d7/dc/d10/l22 0 2024-04-05T21:26:33.905 INFO:tasks.workunit.client.0.smithi121.stdout:2/100: readlink d2/l4 0 2024-04-05T21:26:33.905 INFO:tasks.workunit.client.0.smithi121.stdout:1/85: symlink d8/l15 0 2024-04-05T21:26:33.909 INFO:tasks.workunit.client.0.smithi121.stdout:8/41: dwrite f3 [0,4194304] 0 2024-04-05T21:26:33.910 INFO:tasks.workunit.client.0.smithi121.stdout:2/101: chown d2/d9/fb 97297 1 2024-04-05T21:26:33.912 INFO:tasks.workunit.client.0.smithi121.stdout:1/86: truncate d8/f11 718164 0 2024-04-05T21:26:33.912 INFO:tasks.workunit.client.0.smithi121.stdout:4/93: dwrite d5/d11/f14 [0,4194304] 0 2024-04-05T21:26:33.915 INFO:tasks.workunit.client.0.smithi121.stdout:7/77: dread d0/f3 [0,4194304] 0 2024-04-05T21:26:33.916 INFO:tasks.workunit.client.0.smithi121.stdout:9/73: link d6/ld d6/df/l14 0 2024-04-05T21:26:33.919 INFO:tasks.workunit.client.0.smithi121.stdout:2/102: readlink d2/d9/l1b 0 2024-04-05T21:26:33.924 INFO:tasks.workunit.client.0.smithi121.stdout:2/103: chown d2/d9/d12 147 1 2024-04-05T21:26:33.924 INFO:tasks.workunit.client.0.smithi121.stdout:4/94: mkdir d5/d11/d26 0 2024-04-05T21:26:33.925 INFO:tasks.workunit.client.0.smithi121.stdout:9/74: creat d6/f15 x:0 0 0 2024-04-05T21:26:33.927 INFO:tasks.workunit.client.0.smithi121.stdout:4/95: chown d5/d11/d26 714093 1 2024-04-05T21:26:33.929 INFO:tasks.workunit.client.0.smithi121.stdout:0/106: dwrite d7/f1a [0,4194304] 0 2024-04-05T21:26:33.930 INFO:tasks.workunit.client.0.smithi121.stdout:7/78: link d0/l7 d0/l1c 0 2024-04-05T21:26:33.932 INFO:tasks.workunit.client.0.smithi121.stdout:2/104: creat d2/d9/f1f x:0 0 0 2024-04-05T21:26:33.933 INFO:tasks.workunit.client.0.smithi121.stdout:9/75: rename d6/df/f13 to d6/f16 0 2024-04-05T21:26:33.934 INFO:tasks.workunit.client.0.smithi121.stdout:4/96: write d5/dc/fe [5095699,69695] 0 2024-04-05T21:26:33.934 INFO:tasks.workunit.client.0.smithi121.stdout:8/42: dwrite f6 [0,4194304] 0 2024-04-05T21:26:33.935 INFO:tasks.workunit.client.0.smithi121.stdout:0/107: creat d7/d1c/f23 x:0 0 0 2024-04-05T21:26:33.937 INFO:tasks.workunit.client.0.smithi121.stdout:1/87: dwrite f2 [0,4194304] 0 2024-04-05T21:26:33.942 INFO:tasks.workunit.client.0.smithi121.stdout:0/108: write d7/f1a [3127548,56896] 0 2024-04-05T21:26:33.943 INFO:tasks.workunit.client.0.smithi121.stdout:7/79: creat d0/d9/f1d x:0 0 0 2024-04-05T21:26:33.944 INFO:tasks.workunit.client.0.smithi121.stdout:2/105: unlink d2/f14 0 2024-04-05T21:26:33.946 INFO:tasks.workunit.client.0.smithi121.stdout:1/88: creat d8/d10/f16 x:0 0 0 2024-04-05T21:26:33.946 INFO:tasks.workunit.client.0.smithi121.stdout:9/76: write d6/f16 [172966,84889] 0 2024-04-05T21:26:33.949 INFO:tasks.workunit.client.0.smithi121.stdout:0/109: mknod d7/c24 0 2024-04-05T21:26:33.949 INFO:tasks.workunit.client.0.smithi121.stdout:7/80: mknod d0/d9/c1e 0 2024-04-05T21:26:33.950 INFO:tasks.workunit.client.0.smithi121.stdout:9/77: chown d6/l8 85 1 2024-04-05T21:26:33.950 INFO:tasks.workunit.client.0.smithi121.stdout:8/43: creat fa x:0 0 0 2024-04-05T21:26:33.951 INFO:tasks.workunit.client.0.smithi121.stdout:4/97: dwrite d5/d11/f12 [4194304,4194304] 0 2024-04-05T21:26:33.953 INFO:tasks.workunit.client.0.smithi121.stdout:0/110: creat d7/d1c/f25 x:0 0 0 2024-04-05T21:26:33.955 INFO:tasks.workunit.client.0.smithi121.stdout:4/98: read d5/d11/f17 [1544469,89774] 0 2024-04-05T21:26:33.955 INFO:tasks.workunit.client.0.smithi121.stdout:7/81: creat d0/f1f x:0 0 0 2024-04-05T21:26:33.956 INFO:tasks.workunit.client.0.smithi121.stdout:0/111: dread - d7/f18 zero size 2024-04-05T21:26:33.960 INFO:tasks.workunit.client.0.smithi121.stdout:7/82: write d0/d9/f12 [2680694,106321] 0 2024-04-05T21:26:33.965 INFO:tasks.workunit.client.0.smithi121.stdout:4/99: dread d5/f16 [0,4194304] 0 2024-04-05T21:26:33.966 INFO:tasks.workunit.client.0.smithi121.stdout:4/100: dread - d5/d11/f1e zero size 2024-04-05T21:26:33.969 INFO:tasks.workunit.client.0.smithi121.stdout:0/112: chown d7/dc/d10/f1b 74021 1 2024-04-05T21:26:33.972 INFO:tasks.workunit.client.0.smithi121.stdout:7/83: rename d0/d9/fd to d0/d9/f20 0 2024-04-05T21:26:33.976 INFO:tasks.workunit.client.0.smithi121.stdout:1/89: truncate d8/fb 3201289 0 2024-04-05T21:26:33.977 INFO:tasks.workunit.client.0.smithi121.stdout:2/106: write d2/f11 [1724999,64577] 0 2024-04-05T21:26:33.978 INFO:tasks.workunit.client.0.smithi121.stdout:9/78: getdents d6/df 0 2024-04-05T21:26:33.978 INFO:tasks.workunit.client.0.smithi121.stdout:7/84: creat d0/d9/f21 x:0 0 0 2024-04-05T21:26:33.980 INFO:tasks.workunit.client.0.smithi121.stdout:0/113: creat d7/d1c/d1d/f26 x:0 0 0 2024-04-05T21:26:33.981 INFO:tasks.workunit.client.0.smithi121.stdout:8/44: truncate f3 183582 0 2024-04-05T21:26:33.981 INFO:tasks.workunit.client.0.smithi121.stdout:9/79: write f4 [156686,17938] 0 2024-04-05T21:26:33.982 INFO:tasks.workunit.client.0.smithi121.stdout:1/90: rename f5 to d8/d10/f17 0 2024-04-05T21:26:33.983 INFO:tasks.workunit.client.0.smithi121.stdout:0/114: read - d7/d1c/d1d/f26 zero size 2024-04-05T21:26:33.983 INFO:tasks.workunit.client.0.smithi121.stdout:2/107: rmdir d2/d9/d17 39 2024-04-05T21:26:33.984 INFO:tasks.workunit.client.0.smithi121.stdout:4/101: dread d5/d11/f17 [0,4194304] 0 2024-04-05T21:26:33.987 INFO:tasks.workunit.client.0.smithi121.stdout:0/115: write d7/dc/d10/f11 [834771,116033] 0 2024-04-05T21:26:33.988 INFO:tasks.workunit.client.0.smithi121.stdout:1/91: unlink d8/l15 0 2024-04-05T21:26:33.989 INFO:tasks.workunit.client.0.smithi121.stdout:7/85: dread d0/f3 [0,4194304] 0 2024-04-05T21:26:33.989 INFO:tasks.workunit.client.0.smithi121.stdout:9/80: getdents d6/df 0 2024-04-05T21:26:33.993 INFO:tasks.workunit.client.0.smithi121.stdout:1/92: truncate d8/ff 785097 0 2024-04-05T21:26:33.994 INFO:tasks.workunit.client.0.smithi121.stdout:1/93: stat d8 0 2024-04-05T21:26:33.996 INFO:tasks.workunit.client.0.smithi121.stdout:1/94: chown d8/d10 549841 1 2024-04-05T21:26:33.997 INFO:tasks.workunit.client.0.smithi121.stdout:0/116: dread d7/dc/f14 [0,4194304] 0 2024-04-05T21:26:33.997 INFO:tasks.workunit.client.0.smithi121.stdout:2/108: dwrite d2/d9/d12/f15 [0,4194304] 0 2024-04-05T21:26:33.999 INFO:tasks.workunit.client.0.smithi121.stdout:0/117: read - d7/f18 zero size 2024-04-05T21:26:34.000 INFO:tasks.workunit.client.0.smithi121.stdout:7/86: rename d0/d9/d18 to d0/d9/d22 0 2024-04-05T21:26:34.000 INFO:tasks.workunit.client.0.smithi121.stdout:4/102: mkdir d5/d11/d26/d27 0 2024-04-05T21:26:34.001 INFO:tasks.workunit.client.0.smithi121.stdout:7/87: readlink d0/lb 0 2024-04-05T21:26:34.002 INFO:tasks.workunit.client.0.smithi121.stdout:0/118: chown d7/dc/d10/f1b 174886032 1 2024-04-05T21:26:34.004 INFO:tasks.workunit.client.0.smithi121.stdout:9/81: dwrite d6/fc [0,4194304] 0 2024-04-05T21:26:34.004 INFO:tasks.workunit.client.0.smithi121.stdout:2/109: mknod d2/c20 0 2024-04-05T21:26:34.007 INFO:tasks.workunit.client.0.smithi121.stdout:9/82: chown f5 0 1 2024-04-05T21:26:34.008 INFO:tasks.workunit.client.0.smithi121.stdout:0/119: symlink d7/d1c/d1d/l27 0 2024-04-05T21:26:34.011 INFO:tasks.workunit.client.0.smithi121.stdout:4/103: truncate d5/fa 41504 0 2024-04-05T21:26:34.011 INFO:tasks.workunit.client.0.smithi121.stdout:7/88: dread d0/f6 [0,4194304] 0 2024-04-05T21:26:34.013 INFO:tasks.workunit.client.0.smithi121.stdout:1/95: creat d8/d10/d14/f18 x:0 0 0 2024-04-05T21:26:34.013 INFO:tasks.workunit.client.0.smithi121.stdout:0/120: mknod d7/dc/d10/c28 0 2024-04-05T21:26:34.014 INFO:tasks.workunit.client.0.smithi121.stdout:0/121: chown d7/dc 106 1 2024-04-05T21:26:34.014 INFO:tasks.workunit.client.0.smithi121.stdout:1/96: readlink d8/d10/l13 0 2024-04-05T21:26:34.015 INFO:tasks.workunit.client.0.smithi121.stdout:9/83: link f4 d6/f17 0 2024-04-05T21:26:34.016 INFO:tasks.workunit.client.0.smithi121.stdout:1/97: write d8/d10/d14/f18 [933829,13867] 0 2024-04-05T21:26:34.017 INFO:tasks.workunit.client.0.smithi121.stdout:2/110: getdents d2/d9/d17 0 2024-04-05T21:26:34.018 INFO:tasks.workunit.client.0.smithi121.stdout:1/98: read d8/f11 [9914,5842] 0 2024-04-05T21:26:34.019 INFO:tasks.workunit.client.0.smithi121.stdout:7/89: dread d0/d9/f20 [0,4194304] 0 2024-04-05T21:26:34.019 INFO:tasks.workunit.client.1.smithi188.stdout:8/303: sync 2024-04-05T21:26:34.020 INFO:tasks.workunit.client.1.smithi188.stdout:8/304: symlink d5/d21/d2a/l6c 0 2024-04-05T21:26:34.022 INFO:tasks.workunit.client.0.smithi121.stdout:7/90: mkdir d0/d23 0 2024-04-05T21:26:34.022 INFO:tasks.workunit.client.1.smithi188.stdout:8/305: mknod d5/d21/d2a/d47/c6d 0 2024-04-05T21:26:34.022 INFO:tasks.workunit.client.0.smithi121.stdout:1/99: creat d8/d10/d14/f19 x:0 0 0 2024-04-05T21:26:34.024 INFO:tasks.workunit.client.1.smithi188.stdout:8/306: write d5/d21/d40/d43/d44/d59/f5a [1017916,47837] 0 2024-04-05T21:26:34.024 INFO:tasks.workunit.client.1.smithi188.stdout:8/307: readlink d5/d21/l53 0 2024-04-05T21:26:34.024 INFO:tasks.workunit.client.0.smithi121.stdout:2/111: creat d2/d9/f21 x:0 0 0 2024-04-05T21:26:34.026 INFO:tasks.workunit.client.0.smithi121.stdout:2/112: chown d2/d9/d17/f19 1215760 1 2024-04-05T21:26:34.026 INFO:tasks.workunit.client.0.smithi121.stdout:1/100: mkdir d8/d10/d14/d1a 0 2024-04-05T21:26:34.026 INFO:tasks.workunit.client.0.smithi121.stdout:2/113: chown d2/d9 61763976 1 2024-04-05T21:26:34.027 INFO:tasks.workunit.client.0.smithi121.stdout:7/91: dread d0/d9/f20 [0,4194304] 0 2024-04-05T21:26:34.027 INFO:tasks.workunit.client.0.smithi121.stdout:2/114: readlink d2/d9/l1d 0 2024-04-05T21:26:34.027 INFO:tasks.workunit.client.1.smithi188.stdout:8/308: dread d5/f8 [0,4194304] 0 2024-04-05T21:26:34.029 INFO:tasks.workunit.client.1.smithi188.stdout:8/309: mkdir d5/d6e 0 2024-04-05T21:26:34.032 INFO:tasks.workunit.client.1.smithi188.stdout:8/310: symlink d5/d21/d40/d43/d44/l6f 0 2024-04-05T21:26:34.033 INFO:tasks.workunit.client.0.smithi121.stdout:0/122: getdents d7/dc/d10 0 2024-04-05T21:26:34.033 INFO:tasks.workunit.client.0.smithi121.stdout:2/115: dread d2/f11 [0,4194304] 0 2024-04-05T21:26:34.034 INFO:tasks.workunit.client.1.smithi188.stdout:8/311: dread - d5/d21/d50/f6b zero size 2024-04-05T21:26:34.034 INFO:tasks.workunit.client.0.smithi121.stdout:9/84: truncate f4 2071230 0 2024-04-05T21:26:34.034 INFO:tasks.workunit.client.0.smithi121.stdout:0/123: dread - d7/f18 zero size 2024-04-05T21:26:34.034 INFO:tasks.workunit.client.1.smithi188.stdout:8/312: chown d5/d21/d40/d43/d44/l45 290815 1 2024-04-05T21:26:34.035 INFO:tasks.workunit.client.0.smithi121.stdout:9/85: write d6/f15 [871598,110715] 0 2024-04-05T21:26:34.037 INFO:tasks.workunit.client.1.smithi188.stdout:8/313: mkdir d5/d21/d50/d63/d69/d70 0 2024-04-05T21:26:34.037 INFO:tasks.workunit.client.1.smithi188.stdout:8/314: chown d5 209399447 1 2024-04-05T21:26:34.038 INFO:tasks.workunit.client.1.smithi188.stdout:8/315: mkdir d5/d21/d2a/d71 0 2024-04-05T21:26:34.040 INFO:tasks.workunit.client.0.smithi121.stdout:7/92: getdents d0/d9 0 2024-04-05T21:26:34.040 INFO:tasks.workunit.client.0.smithi121.stdout:7/93: stat d0/d9/f21 0 2024-04-05T21:26:34.040 INFO:tasks.workunit.client.0.smithi121.stdout:0/124: dread d7/dc/d10/f11 [0,4194304] 0 2024-04-05T21:26:34.041 INFO:tasks.workunit.client.1.smithi188.stdout:8/316: dread d5/f20 [0,4194304] 0 2024-04-05T21:26:34.042 INFO:tasks.workunit.client.0.smithi121.stdout:9/86: mknod d6/c18 0 2024-04-05T21:26:34.042 INFO:tasks.workunit.client.0.smithi121.stdout:2/116: creat d2/d9/d1c/f22 x:0 0 0 2024-04-05T21:26:34.043 INFO:tasks.workunit.client.0.smithi121.stdout:7/94: chown d0/l17 0 1 2024-04-05T21:26:34.043 INFO:tasks.workunit.client.1.smithi188.stdout:8/317: mknod d5/d21/d40/d43/d44/d59/c72 0 2024-04-05T21:26:34.043 INFO:tasks.workunit.client.0.smithi121.stdout:9/87: write d6/f16 [375563,106593] 0 2024-04-05T21:26:34.044 INFO:tasks.workunit.client.1.smithi188.stdout:8/318: read d5/dc/f2c [39793,51266] 0 2024-04-05T21:26:34.045 INFO:tasks.workunit.client.0.smithi121.stdout:2/117: stat d2/d9/l1d 0 2024-04-05T21:26:34.045 INFO:tasks.workunit.client.0.smithi121.stdout:0/125: mkdir d7/d1c/d1d/d29 0 2024-04-05T21:26:34.046 INFO:tasks.workunit.client.1.smithi188.stdout:8/319: creat d5/d21/d40/f73 x:0 0 0 2024-04-05T21:26:34.047 INFO:tasks.workunit.client.1.smithi188.stdout:8/320: dread - d5/d21/d40/f55 zero size 2024-04-05T21:26:34.048 INFO:tasks.workunit.client.1.smithi188.stdout:8/321: truncate d5/dc/f17 1213725 0 2024-04-05T21:26:34.049 INFO:tasks.workunit.client.0.smithi121.stdout:2/118: fsync d2/d9/fe 0 2024-04-05T21:26:34.050 INFO:tasks.workunit.client.1.smithi188.stdout:8/322: creat d5/d21/d40/d43/d44/d59/f74 x:0 0 0 2024-04-05T21:26:34.050 INFO:tasks.workunit.client.0.smithi121.stdout:0/126: write d7/d1c/f1f [65062,28605] 0 2024-04-05T21:26:34.050 INFO:tasks.workunit.client.1.smithi188.stdout:8/323: readlink d5/l6 0 2024-04-05T21:26:34.051 INFO:tasks.workunit.client.1.smithi188.stdout:8/324: dread - d5/f3d zero size 2024-04-05T21:26:34.052 INFO:tasks.workunit.client.1.smithi188.stdout:8/325: mknod d5/d6e/c75 0 2024-04-05T21:26:34.053 INFO:tasks.workunit.client.0.smithi121.stdout:2/119: write d2/d9/fb [1050158,104513] 0 2024-04-05T21:26:34.054 INFO:tasks.workunit.client.1.smithi188.stdout:8/326: mknod d5/d21/d40/d43/d44/d5b/c76 0 2024-04-05T21:26:34.055 INFO:tasks.workunit.client.0.smithi121.stdout:1/101: dwrite d8/fb [4194304,4194304] 0 2024-04-05T21:26:34.058 INFO:tasks.workunit.client.1.smithi188.stdout:8/327: rename d5/d21/d40/d43/d44/f68 to d5/d21/d40/d43/d44/d59/f77 0 2024-04-05T21:26:34.059 INFO:tasks.workunit.client.1.smithi188.stdout:8/328: readlink d5/d21/l53 0 2024-04-05T21:26:34.060 INFO:tasks.workunit.client.1.smithi188.stdout:8/329: write d5/d21/d40/d43/f66 [850409,46125] 0 2024-04-05T21:26:34.062 INFO:tasks.workunit.client.1.smithi188.stdout:8/330: creat d5/dc/f78 x:0 0 0 2024-04-05T21:26:34.062 INFO:tasks.workunit.client.0.smithi121.stdout:7/95: dwrite d0/d9/f21 [0,4194304] 0 2024-04-05T21:26:34.064 INFO:tasks.workunit.client.1.smithi188.stdout:8/331: mkdir d5/d21/d40/d43/d44/d59/d79 0 2024-04-05T21:26:34.064 INFO:tasks.workunit.client.0.smithi121.stdout:1/102: mkdir d8/d10/d14/d1a/d1b 0 2024-04-05T21:26:34.065 INFO:tasks.workunit.client.1.smithi188.stdout:8/332: readlink d5/d21/l4e 0 2024-04-05T21:26:34.067 INFO:tasks.workunit.client.0.smithi121.stdout:9/88: dwrite f2 [4194304,4194304] 0 2024-04-05T21:26:34.067 INFO:tasks.workunit.client.0.smithi121.stdout:2/120: dread d2/d9/fb [0,4194304] 0 2024-04-05T21:26:34.067 INFO:tasks.workunit.client.0.smithi121.stdout:0/127: truncate d7/d1c/d1d/f26 45240 0 2024-04-05T21:26:34.067 INFO:tasks.workunit.client.0.smithi121.stdout:1/103: fsync d8/d10/d14/f18 0 2024-04-05T21:26:34.067 INFO:tasks.workunit.client.0.smithi121.stdout:0/128: fdatasync d7/dc/f15 0 2024-04-05T21:26:34.070 INFO:tasks.workunit.client.0.smithi121.stdout:7/96: chown d0/l7 17989 1 2024-04-05T21:26:34.071 INFO:tasks.workunit.client.0.smithi121.stdout:9/89: creat d6/df/f19 x:0 0 0 2024-04-05T21:26:34.072 INFO:tasks.workunit.client.0.smithi121.stdout:1/104: creat d8/f1c x:0 0 0 2024-04-05T21:26:34.075 INFO:tasks.workunit.client.0.smithi121.stdout:7/97: creat d0/d9/f24 x:0 0 0 2024-04-05T21:26:34.075 INFO:tasks.workunit.client.0.smithi121.stdout:1/105: write d8/ff [1465266,40932] 0 2024-04-05T21:26:34.076 INFO:tasks.workunit.client.0.smithi121.stdout:0/129: mkdir d7/d1c/d1d/d29/d2a 0 2024-04-05T21:26:34.078 INFO:tasks.workunit.client.0.smithi121.stdout:0/130: readlink d7/dc/le 0 2024-04-05T21:26:34.078 INFO:tasks.workunit.client.0.smithi121.stdout:9/90: mknod d6/c1a 0 2024-04-05T21:26:34.080 INFO:tasks.workunit.client.0.smithi121.stdout:0/131: truncate d7/f18 42681 0 2024-04-05T21:26:34.080 INFO:tasks.workunit.client.0.smithi121.stdout:9/91: readlink d6/l8 0 2024-04-05T21:26:34.081 INFO:tasks.workunit.client.0.smithi121.stdout:1/106: creat d8/f1d x:0 0 0 2024-04-05T21:26:34.082 INFO:tasks.workunit.client.0.smithi121.stdout:2/121: dwrite d2/d9/d12/f15 [0,4194304] 0 2024-04-05T21:26:34.083 INFO:tasks.workunit.client.0.smithi121.stdout:9/92: chown d6/l8 2 1 2024-04-05T21:26:34.087 INFO:tasks.workunit.client.0.smithi121.stdout:2/122: rmdir d2/d9 39 2024-04-05T21:26:34.089 INFO:tasks.workunit.client.0.smithi121.stdout:7/98: dwrite d0/d9/f24 [0,4194304] 0 2024-04-05T21:26:34.090 INFO:tasks.workunit.client.0.smithi121.stdout:9/93: rename d6/df/l10 to d6/df/l1b 0 2024-04-05T21:26:34.095 INFO:tasks.workunit.client.0.smithi121.stdout:1/107: link d8/ff d8/d10/d14/f1e 0 2024-04-05T21:26:34.096 INFO:tasks.workunit.client.0.smithi121.stdout:7/99: mknod d0/c25 0 2024-04-05T21:26:34.097 INFO:tasks.workunit.client.0.smithi121.stdout:1/108: stat d8/fc 0 2024-04-05T21:26:34.099 INFO:tasks.workunit.client.0.smithi121.stdout:2/123: creat d2/f23 x:0 0 0 2024-04-05T21:26:34.100 INFO:tasks.workunit.client.0.smithi121.stdout:0/132: truncate d7/f9 856416 0 2024-04-05T21:26:34.101 INFO:tasks.workunit.client.0.smithi121.stdout:1/109: creat d8/d10/f1f x:0 0 0 2024-04-05T21:26:34.102 INFO:tasks.workunit.client.0.smithi121.stdout:7/100: dread d0/f15 [0,4194304] 0 2024-04-05T21:26:34.102 INFO:tasks.workunit.client.0.smithi121.stdout:0/133: mknod d7/dc/c2b 0 2024-04-05T21:26:34.105 INFO:tasks.workunit.client.0.smithi121.stdout:1/110: creat d8/d10/d14/d1a/f20 x:0 0 0 2024-04-05T21:26:34.105 INFO:tasks.workunit.client.0.smithi121.stdout:3/59: sync 2024-04-05T21:26:34.106 INFO:tasks.workunit.client.0.smithi121.stdout:2/124: link d2/d9/d17/f19 d2/d1e/f24 0 2024-04-05T21:26:34.108 INFO:tasks.workunit.client.0.smithi121.stdout:3/60: rmdir d0/d3 39 2024-04-05T21:26:34.108 INFO:tasks.workunit.client.0.smithi121.stdout:1/111: dread d8/f11 [0,4194304] 0 2024-04-05T21:26:34.109 INFO:tasks.workunit.client.0.smithi121.stdout:7/101: link d0/d9/f21 d0/d9/f26 0 2024-04-05T21:26:34.109 INFO:tasks.workunit.client.0.smithi121.stdout:1/112: readlink d8/d10/l13 0 2024-04-05T21:26:34.110 INFO:tasks.workunit.client.0.smithi121.stdout:0/134: mkdir d7/d1c/d1d/d29/d2a/d2c 0 2024-04-05T21:26:34.111 INFO:tasks.workunit.client.0.smithi121.stdout:1/113: dread - d8/d10/d14/d1a/f20 zero size 2024-04-05T21:26:34.111 INFO:tasks.workunit.client.0.smithi121.stdout:0/135: readlink d7/dc/d10/l22 0 2024-04-05T21:26:34.115 INFO:tasks.workunit.client.0.smithi121.stdout:7/102: mkdir d0/d23/d27 0 2024-04-05T21:26:34.118 INFO:tasks.workunit.client.0.smithi121.stdout:3/61: mkdir d0/d3/dc/d1a 0 2024-04-05T21:26:34.123 INFO:tasks.workunit.client.0.smithi121.stdout:2/125: write d2/d9/fe [1975076,98834] 0 2024-04-05T21:26:34.124 INFO:tasks.workunit.client.0.smithi121.stdout:0/136: write d7/dc/ff [320489,52709] 0 2024-04-05T21:26:34.125 INFO:tasks.workunit.client.0.smithi121.stdout:1/114: truncate f2 992100 0 2024-04-05T21:26:34.127 INFO:tasks.workunit.client.0.smithi121.stdout:0/137: unlink d7/fa 0 2024-04-05T21:26:34.128 INFO:tasks.workunit.client.0.smithi121.stdout:2/126: mkdir d2/d9/d17/d25 0 2024-04-05T21:26:34.130 INFO:tasks.workunit.client.0.smithi121.stdout:1/115: mknod d8/d10/c21 0 2024-04-05T21:26:34.130 INFO:tasks.workunit.client.0.smithi121.stdout:0/138: dread d7/dc/ff [0,4194304] 0 2024-04-05T21:26:34.132 INFO:tasks.workunit.client.0.smithi121.stdout:2/127: rename d2/c16 to d2/d9/d17/d25/c26 0 2024-04-05T21:26:34.133 INFO:tasks.workunit.client.0.smithi121.stdout:2/128: fsync d2/d9/d12/f15 0 2024-04-05T21:26:34.133 INFO:tasks.workunit.client.0.smithi121.stdout:1/116: unlink d8/d10/f16 0 2024-04-05T21:26:34.137 INFO:tasks.workunit.client.0.smithi121.stdout:0/139: dwrite d7/f18 [0,4194304] 0 2024-04-05T21:26:34.140 INFO:tasks.workunit.client.0.smithi121.stdout:0/140: creat d7/dc/f2d x:0 0 0 2024-04-05T21:26:34.141 INFO:tasks.workunit.client.0.smithi121.stdout:1/117: dread d8/ff [0,4194304] 0 2024-04-05T21:26:34.143 INFO:tasks.workunit.client.0.smithi121.stdout:0/141: mknod d7/d1c/c2e 0 2024-04-05T21:26:34.145 INFO:tasks.workunit.client.0.smithi121.stdout:1/118: mknod d8/d10/d14/c22 0 2024-04-05T21:26:34.147 INFO:tasks.workunit.client.0.smithi121.stdout:0/142: unlink d7/f1a 0 2024-04-05T21:26:34.149 INFO:tasks.workunit.client.0.smithi121.stdout:1/119: rename d8/d10/f1f to d8/d10/d14/f23 0 2024-04-05T21:26:34.152 INFO:tasks.workunit.client.0.smithi121.stdout:0/143: mknod d7/d1c/d1d/d29/d2a/d2c/c2f 0 2024-04-05T21:26:34.153 INFO:tasks.workunit.client.0.smithi121.stdout:0/144: write d7/dc/f2d [948229,91006] 0 2024-04-05T21:26:34.155 INFO:tasks.workunit.client.0.smithi121.stdout:0/145: truncate f2 1354615 0 2024-04-05T21:26:34.163 INFO:tasks.workunit.client.0.smithi121.stdout:0/146: dwrite d7/d1c/f25 [0,4194304] 0 2024-04-05T21:26:34.174 INFO:tasks.workunit.client.0.smithi121.stdout:0/147: dwrite d7/dc/d10/f1b [0,4194304] 0 2024-04-05T21:26:34.175 INFO:tasks.workunit.client.0.smithi121.stdout:0/148: write d7/d1c/f1f [510477,4520] 0 2024-04-05T21:26:34.177 INFO:tasks.workunit.client.1.smithi188.stdout:7/522: sync 2024-04-05T21:26:34.179 INFO:tasks.workunit.client.0.smithi121.stdout:0/149: truncate d7/dc/f15 1594557 0 2024-04-05T21:26:34.181 INFO:tasks.workunit.client.0.smithi121.stdout:0/150: symlink d7/dc/d10/l30 0 2024-04-05T21:26:34.183 INFO:tasks.workunit.client.0.smithi121.stdout:0/151: rmdir d7/d1c/d1d 39 2024-04-05T21:26:34.185 INFO:tasks.workunit.client.0.smithi121.stdout:0/152: mknod d7/d1c/d1d/d29/d2a/d2c/c31 0 2024-04-05T21:26:34.191 INFO:tasks.workunit.client.0.smithi121.stdout:0/153: dread d7/dc/d10/f1b [0,4194304] 0 2024-04-05T21:26:34.193 INFO:tasks.workunit.client.0.smithi121.stdout:0/154: creat d7/dc/f32 x:0 0 0 2024-04-05T21:26:34.194 INFO:tasks.workunit.client.0.smithi121.stdout:0/155: rmdir d7/d1c/d1d 39 2024-04-05T21:26:34.194 INFO:tasks.workunit.client.1.smithi188.stdout:7/523: dwrite d3/d4/f7 [0,4194304] 0 2024-04-05T21:26:34.195 INFO:tasks.workunit.client.0.smithi121.stdout:0/156: write d7/f18 [3963460,62732] 0 2024-04-05T21:26:34.195 INFO:tasks.workunit.client.0.smithi121.stdout:0/157: chown d7/dc/d10/f1b 7864 1 2024-04-05T21:26:34.196 INFO:tasks.workunit.client.0.smithi121.stdout:0/158: dread - d7/dc/f32 zero size 2024-04-05T21:26:34.198 INFO:tasks.workunit.client.0.smithi121.stdout:0/159: creat d7/dc/f33 x:0 0 0 2024-04-05T21:26:34.203 INFO:tasks.workunit.client.1.smithi188.stdout:7/524: dwrite d3/f71 [0,4194304] 0 2024-04-05T21:26:34.207 INFO:tasks.workunit.client.1.smithi188.stdout:7/525: getdents d3/d4/db/d2e/d37 0 2024-04-05T21:26:34.210 INFO:tasks.workunit.client.0.smithi121.stdout:0/160: dwrite d7/dc/f15 [0,4194304] 0 2024-04-05T21:26:34.216 INFO:tasks.workunit.client.1.smithi188.stdout:7/526: dwrite d3/d4/db/d2e/d37/d65/f7d [0,4194304] 0 2024-04-05T21:26:34.216 INFO:tasks.workunit.client.0.smithi121.stdout:0/161: getdents d7 0 2024-04-05T21:26:34.216 INFO:tasks.workunit.client.0.smithi121.stdout:0/162: stat d7 0 2024-04-05T21:26:34.220 INFO:tasks.workunit.client.0.smithi121.stdout:0/163: link d7/dc/f32 d7/d1c/d1d/d29/f34 0 2024-04-05T21:26:34.220 INFO:tasks.workunit.client.0.smithi121.stdout:0/164: stat d7/dc/d10 0 2024-04-05T21:26:34.223 INFO:tasks.workunit.client.0.smithi121.stdout:0/165: creat d7/d1c/d1d/d29/d2a/f35 x:0 0 0 2024-04-05T21:26:34.223 INFO:tasks.workunit.client.0.smithi121.stdout:0/166: fsync d7/d1c/f1f 0 2024-04-05T21:26:34.225 INFO:tasks.workunit.client.0.smithi121.stdout:0/167: mknod d7/d1c/d1d/d29/c36 0 2024-04-05T21:26:34.225 INFO:tasks.workunit.client.1.smithi188.stdout:7/527: creat d3/d4/d1d/d74/d88/fa8 x:0 0 0 2024-04-05T21:26:34.226 INFO:tasks.workunit.client.1.smithi188.stdout:7/528: chown d3/d4/db/d2e/d8c 1057586728 1 2024-04-05T21:26:34.227 INFO:tasks.workunit.client.0.smithi121.stdout:0/168: dread d7/dc/ff [0,4194304] 0 2024-04-05T21:26:34.228 INFO:tasks.workunit.client.0.smithi121.stdout:0/169: read d7/dc/d10/f11 [659245,66622] 0 2024-04-05T21:26:34.229 INFO:tasks.workunit.client.0.smithi121.stdout:0/170: dread d7/d1c/d1d/f26 [0,4194304] 0 2024-04-05T21:26:34.232 INFO:tasks.workunit.client.0.smithi121.stdout:0/171: dread d7/f9 [0,4194304] 0 2024-04-05T21:26:34.234 INFO:tasks.workunit.client.0.smithi121.stdout:0/172: creat d7/d1c/d1d/d29/f37 x:0 0 0 2024-04-05T21:26:34.236 INFO:tasks.workunit.client.0.smithi121.stdout:0/173: rename d7/d1c/c2e to d7/c38 0 2024-04-05T21:26:34.238 INFO:tasks.workunit.client.0.smithi121.stdout:0/174: truncate d7/dc/ff 586723 0 2024-04-05T21:26:34.238 INFO:tasks.workunit.client.0.smithi121.stdout:0/175: stat d7/d1c/d1d/d29/d2a/d2c/c2f 0 2024-04-05T21:26:34.315 INFO:tasks.workunit.client.0.smithi121.stdout:5/105: sync 2024-04-05T21:26:34.325 INFO:tasks.workunit.client.0.smithi121.stdout:5/106: dwrite f4 [0,4194304] 0 2024-04-05T21:26:34.327 INFO:tasks.workunit.client.0.smithi121.stdout:5/107: truncate dd/f17 309136 0 2024-04-05T21:26:34.329 INFO:tasks.workunit.client.0.smithi121.stdout:5/108: truncate dd/de/d19/f23 410677 0 2024-04-05T21:26:34.330 INFO:tasks.workunit.client.0.smithi121.stdout:5/109: write dd/de/f21 [926191,35233] 0 2024-04-05T21:26:34.334 INFO:tasks.workunit.client.0.smithi121.stdout:5/110: dread dd/de/f1a [0,4194304] 0 2024-04-05T21:26:34.337 INFO:tasks.workunit.client.0.smithi121.stdout:5/111: getdents dd/de/d19/d13 0 2024-04-05T21:26:34.338 INFO:tasks.workunit.client.0.smithi121.stdout:5/112: rmdir dd/de/d19/d13 39 2024-04-05T21:26:34.339 INFO:tasks.workunit.client.1.smithi188.stdout:3/400: sync 2024-04-05T21:26:34.339 INFO:tasks.workunit.client.1.smithi188.stdout:9/421: sync 2024-04-05T21:26:34.339 INFO:tasks.workunit.client.1.smithi188.stdout:0/258: sync 2024-04-05T21:26:34.340 INFO:tasks.workunit.client.0.smithi121.stdout:5/113: fdatasync dd/de/d19/d13/f16 0 2024-04-05T21:26:34.342 INFO:tasks.workunit.client.1.smithi188.stdout:3/401: creat d5/d7/d65/f8c x:0 0 0 2024-04-05T21:26:34.343 INFO:tasks.workunit.client.0.smithi121.stdout:5/114: symlink dd/de/d19/d13/l24 0 2024-04-05T21:26:34.343 INFO:tasks.workunit.client.1.smithi188.stdout:3/402: write d5/d7/d65/f88 [732845,106198] 0 2024-04-05T21:26:34.344 INFO:tasks.workunit.client.1.smithi188.stdout:9/422: mkdir dd/d11/d22/d8b 0 2024-04-05T21:26:34.345 INFO:tasks.workunit.client.1.smithi188.stdout:0/259: mknod d4/d6/d13/d23/d24/c60 0 2024-04-05T21:26:34.347 INFO:tasks.workunit.client.0.smithi121.stdout:5/115: creat dd/de/d19/d13/f25 x:0 0 0 2024-04-05T21:26:34.347 INFO:tasks.workunit.client.1.smithi188.stdout:3/403: symlink d5/d2b/l8d 0 2024-04-05T21:26:34.347 INFO:tasks.workunit.client.1.smithi188.stdout:0/260: fsync d4/d6/f1f 0 2024-04-05T21:26:34.350 INFO:tasks.workunit.client.1.smithi188.stdout:0/261: chown d4/d6/d13/d23/d24/c29 54431 1 2024-04-05T21:26:34.351 INFO:tasks.workunit.client.1.smithi188.stdout:0/262: chown d4/f27 10850467 1 2024-04-05T21:26:34.351 INFO:tasks.workunit.client.0.smithi121.stdout:5/116: read f6 [374433,66620] 0 2024-04-05T21:26:34.352 INFO:tasks.workunit.client.1.smithi188.stdout:3/404: dread d5/dc/d2f/f89 [0,4194304] 0 2024-04-05T21:26:34.356 INFO:tasks.workunit.client.1.smithi188.stdout:3/405: creat d5/d7/d6d/d75/d83/f8e x:0 0 0 2024-04-05T21:26:34.358 INFO:tasks.workunit.client.1.smithi188.stdout:0/263: getdents d4/d8/db/d42/d38 0 2024-04-05T21:26:34.361 INFO:tasks.workunit.client.1.smithi188.stdout:3/406: rename d5/dc/c59 to d5/dc/d2f/c8f 0 2024-04-05T21:26:34.362 INFO:tasks.workunit.client.1.smithi188.stdout:0/264: rmdir d4/d6/d13/d23/d24 39 2024-04-05T21:26:34.363 INFO:tasks.workunit.client.1.smithi188.stdout:9/423: dread dd/d11/f21 [0,4194304] 0 2024-04-05T21:26:34.363 INFO:tasks.workunit.client.1.smithi188.stdout:3/407: truncate d5/df/f71 1154771 0 2024-04-05T21:26:34.365 INFO:tasks.workunit.client.1.smithi188.stdout:0/265: mkdir d4/d6/d61 0 2024-04-05T21:26:34.365 INFO:tasks.workunit.client.1.smithi188.stdout:3/408: truncate d5/d35/d4e/f5b 285452 0 2024-04-05T21:26:34.366 INFO:tasks.workunit.client.1.smithi188.stdout:9/424: mkdir dd/d11/d73/d8c 0 2024-04-05T21:26:34.366 INFO:tasks.workunit.client.1.smithi188.stdout:3/409: write d5/fa [2067082,75076] 0 2024-04-05T21:26:34.370 INFO:tasks.workunit.client.1.smithi188.stdout:3/410: mknod d5/df/d80/c90 0 2024-04-05T21:26:34.371 INFO:tasks.workunit.client.1.smithi188.stdout:0/266: creat d4/d6/d13/d23/d24/f62 x:0 0 0 2024-04-05T21:26:34.372 INFO:tasks.workunit.client.1.smithi188.stdout:3/411: truncate d5/df/f6c 172935 0 2024-04-05T21:26:34.373 INFO:tasks.workunit.client.1.smithi188.stdout:3/412: truncate d5/df/f6c 270052 0 2024-04-05T21:26:34.374 INFO:tasks.workunit.client.1.smithi188.stdout:0/267: truncate d4/d8/db/d21/d4a/f58 638533 0 2024-04-05T21:26:34.374 INFO:tasks.workunit.client.1.smithi188.stdout:3/413: chown d5/d7/f25 29750065 1 2024-04-05T21:26:34.375 INFO:tasks.workunit.client.1.smithi188.stdout:3/414: fdatasync d5/d7/d65/f44 0 2024-04-05T21:26:34.375 INFO:tasks.workunit.client.1.smithi188.stdout:0/268: fdatasync d4/d8/db/d21/f56 0 2024-04-05T21:26:34.376 INFO:tasks.workunit.client.1.smithi188.stdout:3/415: write d5/f6b [631881,109224] 0 2024-04-05T21:26:34.379 INFO:tasks.workunit.client.1.smithi188.stdout:3/416: creat d5/d7/d31/f91 x:0 0 0 2024-04-05T21:26:34.379 INFO:tasks.workunit.client.1.smithi188.stdout:9/425: dread fb [0,4194304] 0 2024-04-05T21:26:34.379 INFO:tasks.workunit.client.1.smithi188.stdout:0/269: mkdir d4/d6/d13/d23/d24/d57/d63 0 2024-04-05T21:26:34.380 INFO:tasks.workunit.client.1.smithi188.stdout:0/270: fsync d4/d8/f4d 0 2024-04-05T21:26:34.380 INFO:tasks.workunit.client.1.smithi188.stdout:9/426: chown dd/d11/d1f/f7b 10 1 2024-04-05T21:26:34.384 INFO:tasks.workunit.client.1.smithi188.stdout:0/271: creat d4/d6/d13/f64 x:0 0 0 2024-04-05T21:26:34.385 INFO:tasks.workunit.client.1.smithi188.stdout:9/427: unlink dd/d11/d1f/d25/d51/l69 0 2024-04-05T21:26:34.387 INFO:tasks.workunit.client.1.smithi188.stdout:0/272: symlink d4/d8/db/d21/l65 0 2024-04-05T21:26:34.388 INFO:tasks.workunit.client.1.smithi188.stdout:3/417: dwrite d5/d7/d6d/f7a [0,4194304] 0 2024-04-05T21:26:34.396 INFO:tasks.workunit.client.0.smithi121.stdout:5/117: creat dd/de/d19/d13/f26 x:0 0 0 2024-04-05T21:26:34.397 INFO:tasks.workunit.client.0.smithi121.stdout:5/118: write dd/de/d19/d13/f26 [950805,40982] 0 2024-04-05T21:26:34.398 INFO:tasks.workunit.client.0.smithi121.stdout:5/119: dread - dd/de/d19/d13/f25 zero size 2024-04-05T21:26:34.399 INFO:tasks.workunit.client.0.smithi121.stdout:5/120: stat dd/de/d19/d13/f25 0 2024-04-05T21:26:34.400 INFO:tasks.workunit.client.0.smithi121.stdout:5/121: write dd/de/d19/d13/f25 [815025,46145] 0 2024-04-05T21:26:34.411 INFO:tasks.workunit.client.1.smithi188.stdout:9/428: dread dd/f72 [0,4194304] 0 2024-04-05T21:26:34.413 INFO:tasks.workunit.client.1.smithi188.stdout:3/418: dread f4 [8388608,4194304] 0 2024-04-05T21:26:34.419 INFO:tasks.workunit.client.1.smithi188.stdout:3/419: dread d5/df/f33 [4194304,4194304] 0 2024-04-05T21:26:34.420 INFO:tasks.workunit.client.1.smithi188.stdout:3/420: dread - d5/d7/d65/d5f/f64 zero size 2024-04-05T21:26:34.423 INFO:tasks.workunit.client.1.smithi188.stdout:9/429: dread dd/d49/d6b/d40/f4e [0,4194304] 0 2024-04-05T21:26:34.424 INFO:tasks.workunit.client.1.smithi188.stdout:3/421: rmdir d5/dc/d2f/d8a 0 2024-04-05T21:26:34.425 INFO:tasks.workunit.client.1.smithi188.stdout:9/430: mkdir dd/d11/d73/d8d 0 2024-04-05T21:26:34.427 INFO:tasks.workunit.client.1.smithi188.stdout:3/422: truncate d5/f6 2692057 0 2024-04-05T21:26:34.429 INFO:tasks.workunit.client.1.smithi188.stdout:9/431: mknod dd/d49/d6b/d40/d59/c8e 0 2024-04-05T21:26:34.430 INFO:tasks.workunit.client.1.smithi188.stdout:9/432: fdatasync dd/d49/f53 0 2024-04-05T21:26:34.430 INFO:tasks.workunit.client.1.smithi188.stdout:9/433: stat dd/f16 0 2024-04-05T21:26:34.431 INFO:tasks.workunit.client.1.smithi188.stdout:9/434: dread - dd/d11/d73/f86 zero size 2024-04-05T21:26:34.433 INFO:tasks.workunit.client.1.smithi188.stdout:9/435: symlink dd/d11/d73/l8f 0 2024-04-05T21:26:34.435 INFO:tasks.workunit.client.1.smithi188.stdout:9/436: mknod dd/d11/d1f/d25/d8a/c90 0 2024-04-05T21:26:34.437 INFO:tasks.workunit.client.1.smithi188.stdout:3/423: truncate d5/d35/f52 884625 0 2024-04-05T21:26:34.437 INFO:tasks.workunit.client.1.smithi188.stdout:9/437: truncate dd/d11/f21 5777285 0 2024-04-05T21:26:34.437 INFO:tasks.workunit.client.1.smithi188.stdout:3/424: fdatasync d5/d7/f36 0 2024-04-05T21:26:34.438 INFO:tasks.workunit.client.1.smithi188.stdout:3/425: chown d5/d7e 58673 1 2024-04-05T21:26:34.440 INFO:tasks.workunit.client.1.smithi188.stdout:3/426: truncate d5/d7/f20 4389975 0 2024-04-05T21:26:34.440 INFO:tasks.workunit.client.1.smithi188.stdout:9/438: creat dd/d11/d22/d7e/f91 x:0 0 0 2024-04-05T21:26:34.445 INFO:tasks.workunit.client.1.smithi188.stdout:9/439: creat dd/d11/d73/f92 x:0 0 0 2024-04-05T21:26:34.448 INFO:tasks.workunit.client.1.smithi188.stdout:9/440: creat dd/d11/d73/d85/f93 x:0 0 0 2024-04-05T21:26:34.448 INFO:tasks.workunit.client.1.smithi188.stdout:9/441: chown dd/d11/d1f/f6e 487 1 2024-04-05T21:26:34.449 INFO:tasks.workunit.client.1.smithi188.stdout:9/442: stat dd/d11/d1f/d26 0 2024-04-05T21:26:34.452 INFO:tasks.workunit.client.1.smithi188.stdout:9/443: truncate dd/d11/d22/f56 1154559 0 2024-04-05T21:26:34.453 INFO:tasks.workunit.client.1.smithi188.stdout:9/444: chown dd/d11/d1f/d25/f44 3377 1 2024-04-05T21:26:34.455 INFO:tasks.workunit.client.1.smithi188.stdout:3/427: dread d5/dc/d2f/f4c [0,4194304] 0 2024-04-05T21:26:34.456 INFO:tasks.workunit.client.1.smithi188.stdout:9/445: dread f1 [0,4194304] 0 2024-04-05T21:26:34.458 INFO:tasks.workunit.client.1.smithi188.stdout:9/446: readlink dd/d11/d22/l2e 0 2024-04-05T21:26:34.458 INFO:tasks.workunit.client.1.smithi188.stdout:3/428: link d5/d7/f1a d5/d7/d65/d61/f92 0 2024-04-05T21:26:34.459 INFO:tasks.workunit.client.1.smithi188.stdout:9/447: write dd/d11/f76 [471567,120003] 0 2024-04-05T21:26:34.461 INFO:tasks.workunit.client.1.smithi188.stdout:5/405: sync 2024-04-05T21:26:34.461 INFO:tasks.workunit.client.1.smithi188.stdout:1/470: sync 2024-04-05T21:26:34.461 INFO:tasks.workunit.client.1.smithi188.stdout:6/514: sync 2024-04-05T21:26:34.461 INFO:tasks.workunit.client.1.smithi188.stdout:8/333: sync 2024-04-05T21:26:34.461 INFO:tasks.workunit.client.1.smithi188.stdout:4/602: sync 2024-04-05T21:26:34.461 INFO:tasks.workunit.client.1.smithi188.stdout:2/422: sync 2024-04-05T21:26:34.463 INFO:tasks.workunit.client.1.smithi188.stdout:3/429: creat d5/d39/f93 x:0 0 0 2024-04-05T21:26:34.464 INFO:tasks.workunit.client.1.smithi188.stdout:1/471: fsync d1/dc/d16/f33 0 2024-04-05T21:26:34.470 INFO:tasks.workunit.client.1.smithi188.stdout:1/472: dread - d1/dc/d16/d2d/f97 zero size 2024-04-05T21:26:34.470 INFO:tasks.workunit.client.1.smithi188.stdout:6/515: symlink d6/d13/d48/d49/d75/l9e 0 2024-04-05T21:26:34.471 INFO:tasks.workunit.client.1.smithi188.stdout:2/423: write d5/db/dd/d36/d3c/d48/f5b [990510,31426] 0 2024-04-05T21:26:34.473 INFO:tasks.workunit.client.1.smithi188.stdout:5/406: mkdir d6/d15/d5c/d7a/d44/d98 0 2024-04-05T21:26:34.474 INFO:tasks.workunit.client.1.smithi188.stdout:9/448: dwrite dd/d11/d1f/f6e [0,4194304] 0 2024-04-05T21:26:34.476 INFO:tasks.workunit.client.1.smithi188.stdout:1/473: fdatasync d1/dc/d16/d21/d27/f75 0 2024-04-05T21:26:34.476 INFO:tasks.workunit.client.1.smithi188.stdout:3/430: creat d5/d7/d65/d61/f94 x:0 0 0 2024-04-05T21:26:34.476 INFO:tasks.workunit.client.1.smithi188.stdout:6/516: fdatasync d6/d13/d18/d19/d2c/d34/f97 0 2024-04-05T21:26:34.478 INFO:tasks.workunit.client.1.smithi188.stdout:8/334: rename d5/d21/d40/d43/f46 to d5/d21/d50/d63/d69/d70/f7a 0 2024-04-05T21:26:34.479 INFO:tasks.workunit.client.1.smithi188.stdout:2/424: write d5/db/dd/d36/d3c/f4a [664793,102926] 0 2024-04-05T21:26:34.479 INFO:tasks.workunit.client.1.smithi188.stdout:9/449: dread - dd/d11/d1f/f7b zero size 2024-04-05T21:26:34.479 INFO:tasks.workunit.client.1.smithi188.stdout:4/603: dwrite d0/d14/d2f/fb8 [0,4194304] 0 2024-04-05T21:26:34.481 INFO:tasks.workunit.client.1.smithi188.stdout:3/431: creat d5/df/f95 x:0 0 0 2024-04-05T21:26:34.483 INFO:tasks.workunit.client.1.smithi188.stdout:2/425: fdatasync d5/f22 0 2024-04-05T21:26:34.483 INFO:tasks.workunit.client.1.smithi188.stdout:4/604: chown d0/d14/d15/d53/d82/d8f 42960015 1 2024-04-05T21:26:34.483 INFO:tasks.workunit.client.1.smithi188.stdout:6/517: creat d6/d13/d48/d49/d75/f9f x:0 0 0 2024-04-05T21:26:34.488 INFO:tasks.workunit.client.0.smithi121.stdout:8/45: sync 2024-04-05T21:26:34.488 INFO:tasks.workunit.client.0.smithi121.stdout:6/74: sync 2024-04-05T21:26:34.488 INFO:tasks.workunit.client.1.smithi188.stdout:5/407: creat d6/f99 x:0 0 0 2024-04-05T21:26:34.488 INFO:tasks.workunit.client.1.smithi188.stdout:4/605: write d0/d14/f86 [2324089,4757] 0 2024-04-05T21:26:34.489 INFO:tasks.workunit.client.1.smithi188.stdout:3/432: creat d5/d35/d4e/f96 x:0 0 0 2024-04-05T21:26:34.489 INFO:tasks.workunit.client.1.smithi188.stdout:3/433: truncate d5/f6b 1303072 0 2024-04-05T21:26:34.490 INFO:tasks.workunit.client.0.smithi121.stdout:6/75: rmdir d1/d2/d3 39 2024-04-05T21:26:34.490 INFO:tasks.workunit.client.0.smithi121.stdout:8/46: fsync f8 0 2024-04-05T21:26:34.491 INFO:tasks.workunit.client.0.smithi121.stdout:8/47: truncate f8 678163 0 2024-04-05T21:26:34.491 INFO:tasks.workunit.client.1.smithi188.stdout:1/474: rename d1/dc/d16/d21/d27/f75 to d1/dc/d16/d21/fa0 0 2024-04-05T21:26:34.491 INFO:tasks.workunit.client.1.smithi188.stdout:3/434: read d5/df/f15 [3056493,55439] 0 2024-04-05T21:26:34.492 INFO:tasks.workunit.client.0.smithi121.stdout:6/76: symlink d1/d2/dc/l16 0 2024-04-05T21:26:34.492 INFO:tasks.workunit.client.1.smithi188.stdout:3/435: chown d5/dc/f3d 1475250 1 2024-04-05T21:26:34.493 INFO:tasks.workunit.client.1.smithi188.stdout:3/436: write d5/d35/f40 [3388651,125554] 0 2024-04-05T21:26:34.495 INFO:tasks.workunit.client.1.smithi188.stdout:3/437: dread - d5/d7/d65/d61/f7b zero size 2024-04-05T21:26:34.495 INFO:tasks.workunit.client.0.smithi121.stdout:6/77: creat d1/d2/f17 x:0 0 0 2024-04-05T21:26:34.496 INFO:tasks.workunit.client.1.smithi188.stdout:3/438: dread - d5/df/f95 zero size 2024-04-05T21:26:34.497 INFO:tasks.workunit.client.0.smithi121.stdout:6/78: truncate d1/d2/f17 106724 0 2024-04-05T21:26:34.498 INFO:tasks.workunit.client.0.smithi121.stdout:8/48: dwrite fa [0,4194304] 0 2024-04-05T21:26:34.498 INFO:tasks.workunit.client.1.smithi188.stdout:4/606: unlink d0/d2/d2a/d71/f87 0 2024-04-05T21:26:34.499 INFO:tasks.workunit.client.1.smithi188.stdout:4/607: readlink d0/d14/d15/d78/db1/lb9 0 2024-04-05T21:26:34.501 INFO:tasks.workunit.client.1.smithi188.stdout:2/426: creat d5/db/dd/d36/d3c/d48/d78/f83 x:0 0 0 2024-04-05T21:26:34.502 INFO:tasks.workunit.client.0.smithi121.stdout:8/49: unlink f8 0 2024-04-05T21:26:34.503 INFO:tasks.workunit.client.0.smithi121.stdout:8/50: write fa [1330819,122063] 0 2024-04-05T21:26:34.503 INFO:tasks.workunit.client.0.smithi121.stdout:6/79: symlink d1/d2/d3/d5/l18 0 2024-04-05T21:26:34.503 INFO:tasks.workunit.client.1.smithi188.stdout:6/518: creat d6/d13/d18/d94/fa0 x:0 0 0 2024-04-05T21:26:34.504 INFO:tasks.workunit.client.0.smithi121.stdout:6/80: write d1/d2/f13 [261907,66053] 0 2024-04-05T21:26:34.505 INFO:tasks.workunit.client.0.smithi121.stdout:6/81: stat d1/d2/d3/d5/l18 0 2024-04-05T21:26:34.506 INFO:tasks.workunit.client.0.smithi121.stdout:6/82: chown d1/c12 67 1 2024-04-05T21:26:34.507 INFO:tasks.workunit.client.0.smithi121.stdout:6/83: write d1/d2/f17 [753475,71696] 0 2024-04-05T21:26:34.508 INFO:tasks.workunit.client.1.smithi188.stdout:4/608: creat d0/d2/d17/d48/d70/fda x:0 0 0 2024-04-05T21:26:34.509 INFO:tasks.workunit.client.0.smithi121.stdout:6/84: dread d1/d2/d3/d5/fd [0,4194304] 0 2024-04-05T21:26:34.511 INFO:tasks.workunit.client.0.smithi121.stdout:6/85: dread d1/d2/d3/d5/fd [0,4194304] 0 2024-04-05T21:26:34.511 INFO:tasks.workunit.client.0.smithi121.stdout:8/51: creat fb x:0 0 0 2024-04-05T21:26:34.511 INFO:tasks.workunit.client.1.smithi188.stdout:2/427: creat d5/db/d35/d3b/f84 x:0 0 0 2024-04-05T21:26:34.513 INFO:tasks.workunit.client.1.smithi188.stdout:9/450: getdents dd/d11/d73 0 2024-04-05T21:26:34.514 INFO:tasks.workunit.client.1.smithi188.stdout:9/451: stat dd/d11/d1f/d26/l5e 0 2024-04-05T21:26:34.516 INFO:tasks.workunit.client.1.smithi188.stdout:3/439: link d5/df/c28 d5/dc/d2f/c97 0 2024-04-05T21:26:34.517 INFO:tasks.workunit.client.0.smithi121.stdout:8/52: dwrite f6 [0,4194304] 0 2024-04-05T21:26:34.517 INFO:tasks.workunit.client.0.smithi121.stdout:6/86: dwrite d1/d2/f17 [0,4194304] 0 2024-04-05T21:26:34.518 INFO:tasks.workunit.client.0.smithi121.stdout:8/53: write fa [3969364,122224] 0 2024-04-05T21:26:34.518 INFO:tasks.workunit.client.1.smithi188.stdout:9/452: creat dd/d49/d6b/f94 x:0 0 0 2024-04-05T21:26:34.520 INFO:tasks.workunit.client.0.smithi121.stdout:8/54: readlink l7 0 2024-04-05T21:26:34.520 INFO:tasks.workunit.client.1.smithi188.stdout:2/428: truncate d5/f22 4634675 0 2024-04-05T21:26:34.521 INFO:tasks.workunit.client.0.smithi121.stdout:8/55: truncate fb 221683 0 2024-04-05T21:26:34.522 INFO:tasks.workunit.client.1.smithi188.stdout:9/453: read dd/d11/d1f/d25/f50 [2761463,39008] 0 2024-04-05T21:26:34.523 INFO:tasks.workunit.client.1.smithi188.stdout:9/454: dread - dd/d49/f66 zero size 2024-04-05T21:26:34.526 INFO:tasks.workunit.client.1.smithi188.stdout:3/440: dread d5/f68 [0,4194304] 0 2024-04-05T21:26:34.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:34 smithi188 ceph-mon[49715]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:34.530 INFO:tasks.workunit.client.1.smithi188.stdout:4/609: getdents d0/d2/d17/d3a 0 2024-04-05T21:26:34.532 INFO:tasks.workunit.client.1.smithi188.stdout:3/441: fdatasync d5/d7/d31/f76 0 2024-04-05T21:26:34.533 INFO:tasks.workunit.client.1.smithi188.stdout:9/455: truncate dd/d11/d1f/f7b 677308 0 2024-04-05T21:26:34.535 INFO:tasks.workunit.client.1.smithi188.stdout:6/519: dread d6/d13/d38/f6f [0,4194304] 0 2024-04-05T21:26:34.535 INFO:tasks.workunit.client.1.smithi188.stdout:2/429: dwrite d5/db/f76 [4194304,4194304] 0 2024-04-05T21:26:34.536 INFO:tasks.workunit.client.1.smithi188.stdout:3/442: mknod d5/d7/d31/c98 0 2024-04-05T21:26:34.537 INFO:tasks.workunit.client.1.smithi188.stdout:9/456: write dd/d11/f76 [953321,45207] 0 2024-04-05T21:26:34.543 INFO:tasks.workunit.client.1.smithi188.stdout:2/430: dread - d5/db/dd/d36/d3c/d48/f56 zero size 2024-04-05T21:26:34.544 INFO:tasks.workunit.client.1.smithi188.stdout:9/457: fsync dd/d11/d1f/d26/f19 0 2024-04-05T21:26:34.545 INFO:tasks.workunit.client.1.smithi188.stdout:9/458: readlink dd/d34/l33 0 2024-04-05T21:26:34.545 INFO:tasks.workunit.client.1.smithi188.stdout:4/610: link d0/d14/d15/d1b/c72 d0/d14/d15/d53/d82/d8f/dbd/cdb 0 2024-04-05T21:26:34.546 INFO:tasks.workunit.client.1.smithi188.stdout:9/459: chown fb 8418 1 2024-04-05T21:26:34.549 INFO:tasks.workunit.client.1.smithi188.stdout:3/443: dwrite d5/d7/d31/f45 [0,4194304] 0 2024-04-05T21:26:34.550 INFO:tasks.workunit.client.1.smithi188.stdout:9/460: mkdir dd/d3d/d95 0 2024-04-05T21:26:34.552 INFO:tasks.workunit.client.1.smithi188.stdout:9/461: read - dd/d11/d73/f92 zero size 2024-04-05T21:26:34.554 INFO:tasks.workunit.client.1.smithi188.stdout:3/444: rename d5/d7/d31/c8b to d5/d7e/c99 0 2024-04-05T21:26:34.558 INFO:tasks.workunit.client.1.smithi188.stdout:2/431: dwrite d5/db/dd/d36/d3c/d48/f56 [0,4194304] 0 2024-04-05T21:26:34.559 INFO:tasks.workunit.client.1.smithi188.stdout:3/445: creat d5/d7/d65/d61/f9a x:0 0 0 2024-04-05T21:26:34.559 INFO:tasks.workunit.client.0.smithi121.stdout:8/56: write f3 [731807,95363] 0 2024-04-05T21:26:34.562 INFO:tasks.workunit.client.1.smithi188.stdout:3/446: dread - d5/d7/d65/d61/f9a zero size 2024-04-05T21:26:34.563 INFO:tasks.workunit.client.0.smithi121.stdout:8/57: creat fc x:0 0 0 2024-04-05T21:26:34.563 INFO:tasks.workunit.client.0.smithi121.stdout:8/58: chown f6 0 1 2024-04-05T21:26:34.564 INFO:tasks.workunit.client.1.smithi188.stdout:2/432: read d5/db/d35/d3b/f4d [3254481,14915] 0 2024-04-05T21:26:34.566 INFO:tasks.workunit.client.1.smithi188.stdout:4/611: dwrite d0/f64 [0,4194304] 0 2024-04-05T21:26:34.566 INFO:tasks.workunit.client.1.smithi188.stdout:3/447: chown d5/d7/d6d/d75 104 1 2024-04-05T21:26:34.566 INFO:tasks.workunit.client.0.smithi121.stdout:8/59: mknod cd 0 2024-04-05T21:26:34.567 INFO:tasks.workunit.client.1.smithi188.stdout:4/612: chown d0/d2/d2a/d71/ld3 430 1 2024-04-05T21:26:34.569 INFO:tasks.workunit.client.0.smithi121.stdout:8/60: mknod ce 0 2024-04-05T21:26:34.573 INFO:tasks.workunit.client.1.smithi188.stdout:2/433: mkdir d5/db/dd/d36/d3c/d48/d85 0 2024-04-05T21:26:34.574 INFO:tasks.workunit.client.1.smithi188.stdout:4/613: mknod d0/d2/d2a/d34/cdc 0 2024-04-05T21:26:34.574 INFO:tasks.workunit.client.1.smithi188.stdout:3/448: getdents d5/dc 0 2024-04-05T21:26:34.578 INFO:tasks.workunit.client.1.smithi188.stdout:3/449: write d5/df/f33 [2491606,112742] 0 2024-04-05T21:26:34.579 INFO:tasks.workunit.client.1.smithi188.stdout:4/614: truncate d0/d14/d15/d78/db1/f7c 9202486 0 2024-04-05T21:26:34.579 INFO:tasks.workunit.client.1.smithi188.stdout:2/434: creat d5/db/dd/d36/d3c/d48/d7a/d7c/f86 x:0 0 0 2024-04-05T21:26:34.581 INFO:tasks.workunit.client.1.smithi188.stdout:1/475: dread d1/dc/f1f [0,4194304] 0 2024-04-05T21:26:34.583 INFO:tasks.workunit.client.1.smithi188.stdout:4/615: mkdir d0/d14/d2f/ddd 0 2024-04-05T21:26:34.585 INFO:tasks.workunit.client.1.smithi188.stdout:1/476: creat d1/dc/d16/d21/d27/fa1 x:0 0 0 2024-04-05T21:26:34.587 INFO:tasks.workunit.client.1.smithi188.stdout:5/408: dread d6/d17/f20 [0,4194304] 0 2024-04-05T21:26:34.588 INFO:tasks.workunit.client.1.smithi188.stdout:4/616: write d0/d14/d15/d53/f74 [468095,129073] 0 2024-04-05T21:26:34.588 INFO:tasks.workunit.client.1.smithi188.stdout:1/477: creat d1/d7/d4e/fa2 x:0 0 0 2024-04-05T21:26:34.591 INFO:tasks.workunit.client.1.smithi188.stdout:2/435: dwrite d5/db/dd/f15 [0,4194304] 0 2024-04-05T21:26:34.591 INFO:tasks.workunit.client.1.smithi188.stdout:5/409: chown d6/d15/d5c/d7a/d44/l92 11239697 1 2024-04-05T21:26:34.593 INFO:tasks.workunit.client.1.smithi188.stdout:5/410: chown d6/d17/f97 194640814 1 2024-04-05T21:26:34.594 INFO:tasks.workunit.client.1.smithi188.stdout:1/478: mknod d1/dc/d16/d46/d88/d13/ca3 0 2024-04-05T21:26:34.597 INFO:tasks.workunit.client.1.smithi188.stdout:1/479: write d1/dc/d16/d2d/f7e [669997,6009] 0 2024-04-05T21:26:34.600 INFO:tasks.workunit.client.0.smithi121.stdout:4/104: sync 2024-04-05T21:26:34.600 INFO:tasks.workunit.client.0.smithi121.stdout:9/94: sync 2024-04-05T21:26:34.600 INFO:tasks.workunit.client.0.smithi121.stdout:7/103: sync 2024-04-05T21:26:34.600 INFO:tasks.workunit.client.0.smithi121.stdout:9/95: chown d6 15016 1 2024-04-05T21:26:34.603 INFO:tasks.workunit.client.0.smithi121.stdout:9/96: chown d6/ce 2068 1 2024-04-05T21:26:34.604 INFO:tasks.workunit.client.0.smithi121.stdout:9/97: fsync d6/f16 0 2024-04-05T21:26:34.605 INFO:tasks.workunit.client.0.smithi121.stdout:7/104: mkdir d0/d9/d22/d28 0 2024-04-05T21:26:34.605 INFO:tasks.workunit.client.0.smithi121.stdout:4/105: dread d5/dc/f13 [0,4194304] 0 2024-04-05T21:26:34.605 INFO:tasks.workunit.client.0.smithi121.stdout:7/105: chown d0 128822 1 2024-04-05T21:26:34.605 INFO:tasks.workunit.client.1.smithi188.stdout:4/617: getdents d0/d14/d15/d1b 0 2024-04-05T21:26:34.606 INFO:tasks.workunit.client.0.smithi121.stdout:9/98: write d6/df/f19 [837542,22040] 0 2024-04-05T21:26:34.606 INFO:tasks.workunit.client.0.smithi121.stdout:7/106: creat d0/d9/f29 x:0 0 0 2024-04-05T21:26:34.607 INFO:tasks.workunit.client.0.smithi121.stdout:9/99: chown d6/f15 66 1 2024-04-05T21:26:34.607 INFO:tasks.workunit.client.0.smithi121.stdout:4/106: fdatasync d5/d11/f17 0 2024-04-05T21:26:34.609 INFO:tasks.workunit.client.1.smithi188.stdout:5/411: dwrite d6/d17/d23/d85/d8c/f8b [0,4194304] 0 2024-04-05T21:26:34.610 INFO:tasks.workunit.client.0.smithi121.stdout:4/107: readlink d5/dc/l25 0 2024-04-05T21:26:34.611 INFO:tasks.workunit.client.0.smithi121.stdout:4/108: readlink d5/d11/l1f 0 2024-04-05T21:26:34.613 INFO:tasks.workunit.client.0.smithi121.stdout:7/107: dread d0/f6 [0,4194304] 0 2024-04-05T21:26:34.615 INFO:tasks.workunit.client.1.smithi188.stdout:2/436: fdatasync d5/db/dd/f15 0 2024-04-05T21:26:34.616 INFO:tasks.workunit.client.1.smithi188.stdout:4/618: dwrite d0/d14/d15/d53/fac [0,4194304] 0 2024-04-05T21:26:34.617 INFO:tasks.workunit.client.0.smithi121.stdout:9/100: dwrite d6/df/f19 [0,4194304] 0 2024-04-05T21:26:34.617 INFO:tasks.workunit.client.1.smithi188.stdout:5/412: creat d6/f9a x:0 0 0 2024-04-05T21:26:34.617 INFO:tasks.workunit.client.0.smithi121.stdout:8/61: truncate fa 3593458 0 2024-04-05T21:26:34.619 INFO:tasks.workunit.client.0.smithi121.stdout:9/101: stat d6/l8 0 2024-04-05T21:26:34.619 INFO:tasks.workunit.client.1.smithi188.stdout:3/450: dread d5/d7/f20 [0,4194304] 0 2024-04-05T21:26:34.621 INFO:tasks.workunit.client.0.smithi121.stdout:9/102: write f5 [5518011,47476] 0 2024-04-05T21:26:34.622 INFO:tasks.workunit.client.0.smithi121.stdout:9/103: readlink d6/l11 0 2024-04-05T21:26:34.624 INFO:tasks.workunit.client.1.smithi188.stdout:5/413: stat d6/d82/d40/d42/d81 0 2024-04-05T21:26:34.625 INFO:tasks.workunit.client.0.smithi121.stdout:7/108: dread d0/f15 [0,4194304] 0 2024-04-05T21:26:34.626 INFO:tasks.workunit.client.0.smithi121.stdout:4/109: dwrite d5/f1d [0,4194304] 0 2024-04-05T21:26:34.626 INFO:tasks.workunit.client.1.smithi188.stdout:4/619: stat d0/d14/d15/d53/d82/d8f 0 2024-04-05T21:26:34.627 INFO:tasks.workunit.client.0.smithi121.stdout:9/104: creat d6/f1c x:0 0 0 2024-04-05T21:26:34.628 INFO:tasks.workunit.client.1.smithi188.stdout:2/437: mkdir d5/db/dd/d36/d3c/d63/d75/d87 0 2024-04-05T21:26:34.629 INFO:tasks.workunit.client.0.smithi121.stdout:9/105: write d6/df/f19 [477048,21986] 0 2024-04-05T21:26:34.630 INFO:tasks.workunit.client.0.smithi121.stdout:9/106: truncate d6/fc 5181767 0 2024-04-05T21:26:34.630 INFO:tasks.workunit.client.0.smithi121.stdout:7/109: symlink d0/d9/d22/d28/l2a 0 2024-04-05T21:26:34.634 INFO:tasks.workunit.client.0.smithi121.stdout:7/110: mkdir d0/d9/d2b 0 2024-04-05T21:26:34.634 INFO:tasks.workunit.client.1.smithi188.stdout:5/414: unlink d6/d17/d23/d85/d8c/l6a 0 2024-04-05T21:26:34.634 INFO:tasks.workunit.client.1.smithi188.stdout:4/620: unlink d0/d2/d2a/d71/fb6 0 2024-04-05T21:26:34.635 INFO:tasks.workunit.client.1.smithi188.stdout:2/438: rename d5/db/d35/d3b/d6b/l6c to d5/db/dd/d36/d3c/d63/d75/l88 0 2024-04-05T21:26:34.639 INFO:tasks.workunit.client.1.smithi188.stdout:5/415: truncate d6/d17/f97 814265 0 2024-04-05T21:26:34.639 INFO:tasks.workunit.client.1.smithi188.stdout:4/621: symlink d0/d81/lde 0 2024-04-05T21:26:34.640 INFO:tasks.workunit.client.0.smithi121.stdout:4/110: dwrite d5/d11/f1e [0,4194304] 0 2024-04-05T21:26:34.641 INFO:tasks.workunit.client.0.smithi121.stdout:7/111: dread d0/d9/f26 [0,4194304] 0 2024-04-05T21:26:34.642 INFO:tasks.workunit.client.1.smithi188.stdout:4/622: truncate d0/d14/d15/d78/db1/f83 624092 0 2024-04-05T21:26:34.642 INFO:tasks.workunit.client.0.smithi121.stdout:7/112: readlink d0/d9/l19 0 2024-04-05T21:26:34.644 INFO:tasks.workunit.client.1.smithi188.stdout:5/416: mkdir d6/d15/d5c/d7a/d44/d9b 0 2024-04-05T21:26:34.647 INFO:tasks.workunit.client.0.smithi121.stdout:4/111: rename d5/dc to d5/d11/d28 0 2024-04-05T21:26:34.648 INFO:tasks.workunit.client.1.smithi188.stdout:4/623: mkdir d0/d14/d15/dd4/ddf 0 2024-04-05T21:26:34.648 INFO:tasks.workunit.client.0.smithi121.stdout:4/112: chown d5/d11/d20 11292289 1 2024-04-05T21:26:34.648 INFO:tasks.workunit.client.1.smithi188.stdout:2/439: link d5/c8 d5/db/dd/d36/d3c/d48/d7a/d7c/c89 0 2024-04-05T21:26:34.650 INFO:tasks.workunit.client.0.smithi121.stdout:7/113: rename d0/l16 to d0/d9/d2b/l2c 0 2024-04-05T21:26:34.651 INFO:tasks.workunit.client.0.smithi121.stdout:7/114: fsync d0/d9/f24 0 2024-04-05T21:26:34.651 INFO:tasks.workunit.client.0.smithi121.stdout:4/113: read d5/f9 [16590,86501] 0 2024-04-05T21:26:34.653 INFO:tasks.workunit.client.0.smithi121.stdout:7/115: mkdir d0/d2d 0 2024-04-05T21:26:34.655 INFO:tasks.workunit.client.1.smithi188.stdout:4/624: dread d0/d14/d15/d53/fac [0,4194304] 0 2024-04-05T21:26:34.655 INFO:tasks.workunit.client.1.smithi188.stdout:2/440: fsync d5/db/d35/d3b/f4d 0 2024-04-05T21:26:34.655 INFO:tasks.workunit.client.0.smithi121.stdout:7/116: creat d0/d23/f2e x:0 0 0 2024-04-05T21:26:34.655 INFO:tasks.workunit.client.0.smithi121.stdout:4/114: dread d5/d11/f22 [0,4194304] 0 2024-04-05T21:26:34.656 INFO:tasks.workunit.client.1.smithi188.stdout:5/417: getdents d6/d82/d40/d42 0 2024-04-05T21:26:34.656 INFO:tasks.workunit.client.0.smithi121.stdout:4/115: chown d5/d11/d28/l24 3 1 2024-04-05T21:26:34.657 INFO:tasks.workunit.client.0.smithi121.stdout:4/116: stat d5/d11 0 2024-04-05T21:26:34.657 INFO:tasks.workunit.client.1.smithi188.stdout:5/418: stat d6/c13 0 2024-04-05T21:26:34.658 INFO:tasks.workunit.client.1.smithi188.stdout:5/419: readlink d6/d17/d23/d85/d8c/l58 0 2024-04-05T21:26:34.658 INFO:tasks.workunit.client.1.smithi188.stdout:2/441: read d5/db/dd/d36/d3c/d48/f5b [3292662,106830] 0 2024-04-05T21:26:34.659 INFO:tasks.workunit.client.0.smithi121.stdout:4/117: mkdir d5/d11/d26/d27/d29 0 2024-04-05T21:26:34.660 INFO:tasks.workunit.client.1.smithi188.stdout:5/420: creat d6/d82/d40/f9c x:0 0 0 2024-04-05T21:26:34.662 INFO:tasks.workunit.client.0.smithi121.stdout:8/62: read fa [1991789,49978] 0 2024-04-05T21:26:34.662 INFO:tasks.workunit.client.1.smithi188.stdout:2/442: symlink d5/db/dd/d36/d3c/l8a 0 2024-04-05T21:26:34.662 INFO:tasks.workunit.client.1.smithi188.stdout:5/421: truncate d6/f9a 205874 0 2024-04-05T21:26:34.664 INFO:tasks.workunit.client.1.smithi188.stdout:5/422: write d6/f9a [708825,11197] 0 2024-04-05T21:26:34.664 INFO:tasks.workunit.client.1.smithi188.stdout:4/625: dread d0/d14/d15/d78/db1/f83 [0,4194304] 0 2024-04-05T21:26:34.666 INFO:tasks.workunit.client.0.smithi121.stdout:8/63: stat l4 0 2024-04-05T21:26:34.666 INFO:tasks.workunit.client.0.smithi121.stdout:7/117: dwrite d0/f3 [0,4194304] 0 2024-04-05T21:26:34.667 INFO:tasks.workunit.client.0.smithi121.stdout:7/118: read d0/d9/f12 [660301,57314] 0 2024-04-05T21:26:34.668 INFO:tasks.workunit.client.0.smithi121.stdout:4/118: link d5/d11/f21 d5/d11/d28/f2a 0 2024-04-05T21:26:34.669 INFO:tasks.workunit.client.0.smithi121.stdout:7/119: write d0/f3 [3624583,3548] 0 2024-04-05T21:26:34.671 INFO:tasks.workunit.client.0.smithi121.stdout:7/120: write d0/d9/f24 [2055547,117789] 0 2024-04-05T21:26:34.671 INFO:tasks.workunit.client.1.smithi188.stdout:5/423: truncate d6/f2c 659184 0 2024-04-05T21:26:34.672 INFO:tasks.workunit.client.0.smithi121.stdout:9/107: write d6/f17 [2069634,81594] 0 2024-04-05T21:26:34.672 INFO:tasks.workunit.client.1.smithi188.stdout:5/424: write d6/d82/f50 [1032249,25375] 0 2024-04-05T21:26:34.674 INFO:tasks.workunit.client.1.smithi188.stdout:4/626: mknod d0/d14/d15/d78/db1/daa/ce0 0 2024-04-05T21:26:34.677 INFO:tasks.workunit.client.0.smithi121.stdout:9/108: mkdir d6/df/d1d 0 2024-04-05T21:26:34.677 INFO:tasks.workunit.client.0.smithi121.stdout:7/121: dread d0/d9/f26 [0,4194304] 0 2024-04-05T21:26:34.680 INFO:tasks.workunit.client.1.smithi188.stdout:5/425: creat d6/d82/d40/d42/d81/d95/f9d x:0 0 0 2024-04-05T21:26:34.681 INFO:tasks.workunit.client.1.smithi188.stdout:4/627: creat d0/d14/d15/dd4/ddf/fe1 x:0 0 0 2024-04-05T21:26:34.682 INFO:tasks.workunit.client.1.smithi188.stdout:4/628: write d0/d2/d17/d48/d70/fc0 [194453,67130] 0 2024-04-05T21:26:34.682 INFO:tasks.workunit.client.1.smithi188.stdout:5/426: creat d6/d15/d5c/f9e x:0 0 0 2024-04-05T21:26:34.683 INFO:tasks.workunit.client.1.smithi188.stdout:2/443: truncate d5/fa 3997098 0 2024-04-05T21:26:34.683 INFO:tasks.workunit.client.0.smithi121.stdout:4/119: rmdir d5/d11 39 2024-04-05T21:26:34.683 INFO:tasks.workunit.client.0.smithi121.stdout:2/129: sync 2024-04-05T21:26:34.684 INFO:tasks.workunit.client.0.smithi121.stdout:1/120: sync 2024-04-05T21:26:34.684 INFO:tasks.workunit.client.0.smithi121.stdout:3/62: sync 2024-04-05T21:26:34.684 INFO:tasks.workunit.client.1.smithi188.stdout:2/444: chown d5/db/dd/d24/f2e 30202 1 2024-04-05T21:26:34.685 INFO:tasks.workunit.client.0.smithi121.stdout:1/121: chown d8/d10/c21 17362 1 2024-04-05T21:26:34.685 INFO:tasks.workunit.client.0.smithi121.stdout:2/130: read - d2/d9/d17/f19 zero size 2024-04-05T21:26:34.686 INFO:tasks.workunit.client.0.smithi121.stdout:3/63: mknod d0/c1b 0 2024-04-05T21:26:34.687 INFO:tasks.workunit.client.0.smithi121.stdout:2/131: chown d2/d9/d17/f19 9470 1 2024-04-05T21:26:34.687 INFO:tasks.workunit.client.0.smithi121.stdout:4/120: mknod d5/d11/c2b 0 2024-04-05T21:26:34.687 INFO:tasks.workunit.client.1.smithi188.stdout:5/427: unlink d6/c10 0 2024-04-05T21:26:34.688 INFO:tasks.workunit.client.0.smithi121.stdout:2/132: read - d2/d9/f1f zero size 2024-04-05T21:26:34.689 INFO:tasks.workunit.client.1.smithi188.stdout:5/428: chown d6/fe 51295 1 2024-04-05T21:26:34.689 INFO:tasks.workunit.client.0.smithi121.stdout:9/109: fdatasync f4 0 2024-04-05T21:26:34.691 INFO:tasks.workunit.client.1.smithi188.stdout:4/629: dwrite d0/d2/d52/f79 [0,4194304] 0 2024-04-05T21:26:34.692 INFO:tasks.workunit.client.1.smithi188.stdout:5/429: read d6/d15/d5c/d7a/d44/f57 [267124,108115] 0 2024-04-05T21:26:34.700 INFO:tasks.workunit.client.1.smithi188.stdout:2/445: link d5/c8 d5/db/dd/d67/c8b 0 2024-04-05T21:26:34.700 INFO:tasks.workunit.client.1.smithi188.stdout:2/446: read - d5/db/d35/d3b/f84 zero size 2024-04-05T21:26:34.700 INFO:tasks.workunit.client.1.smithi188.stdout:7/529: sync 2024-04-05T21:26:34.703 INFO:tasks.workunit.client.1.smithi188.stdout:2/447: write d5/db/dd/d24/f28 [5083506,84075] 0 2024-04-05T21:26:34.704 INFO:tasks.workunit.client.1.smithi188.stdout:5/430: creat d6/d15/d5c/d7a/d44/d98/f9f x:0 0 0 2024-04-05T21:26:34.705 INFO:tasks.workunit.client.1.smithi188.stdout:2/448: fdatasync d5/db/dd/d36/d3c/d48/f5b 0 2024-04-05T21:26:34.707 INFO:tasks.workunit.client.1.smithi188.stdout:4/630: rename d0/d14/d15/d53/f66 to d0/d14/d15/d78/fe2 0 2024-04-05T21:26:34.708 INFO:tasks.workunit.client.1.smithi188.stdout:5/431: mkdir d6/d15/d5c/d7a/d44/da0 0 2024-04-05T21:26:34.709 INFO:tasks.workunit.client.1.smithi188.stdout:5/432: chown d6/d17/d23/f55 0 1 2024-04-05T21:26:34.710 INFO:tasks.workunit.client.1.smithi188.stdout:2/449: mkdir d5/db/dd/d36/d3d/d46/d8c 0 2024-04-05T21:26:34.712 INFO:tasks.workunit.client.1.smithi188.stdout:5/433: truncate d6/d17/d23/d85/d8c/f8d 844745 0 2024-04-05T21:26:34.713 INFO:tasks.workunit.client.1.smithi188.stdout:2/450: symlink d5/db/d35/d3b/d6b/l8d 0 2024-04-05T21:26:34.715 INFO:tasks.workunit.client.1.smithi188.stdout:5/434: chown d6/l31 41986 1 2024-04-05T21:26:34.716 INFO:tasks.workunit.client.1.smithi188.stdout:2/451: write d5/db/dd/f11 [1725565,124551] 0 2024-04-05T21:26:34.716 INFO:tasks.workunit.client.1.smithi188.stdout:2/452: chown d5/db 84897345 1 2024-04-05T21:26:34.718 INFO:tasks.workunit.client.1.smithi188.stdout:2/453: dread - d5/db/dd/d36/d3d/d46/f81 zero size 2024-04-05T21:26:34.719 INFO:tasks.workunit.client.1.smithi188.stdout:2/454: chown d5/db/dd/d36/d3c/d48/d85 3451 1 2024-04-05T21:26:34.727 INFO:tasks.workunit.client.1.smithi188.stdout:2/455: truncate d5/db/dd/d36/d3c/d48/f5b 1893139 0 2024-04-05T21:26:34.727 INFO:tasks.workunit.client.1.smithi188.stdout:2/456: stat d5/db/dd/d24/l4c 0 2024-04-05T21:26:34.728 INFO:tasks.workunit.client.1.smithi188.stdout:4/631: truncate d0/d14/d15/d53/d82/d8f/fad 3173167 0 2024-04-05T21:26:34.729 INFO:tasks.workunit.client.1.smithi188.stdout:5/435: getdents d6/d82/d40 0 2024-04-05T21:26:34.733 INFO:tasks.workunit.client.0.smithi121.stdout:8/64: dwrite fa [0,4194304] 0 2024-04-05T21:26:34.734 INFO:tasks.workunit.client.1.smithi188.stdout:2/457: creat d5/db/dd/d67/f8e x:0 0 0 2024-04-05T21:26:34.734 INFO:tasks.workunit.client.0.smithi121.stdout:4/121: mkdir d5/d11/d2c 0 2024-04-05T21:26:34.735 INFO:tasks.workunit.client.0.smithi121.stdout:2/133: symlink d2/d9/d17/l27 0 2024-04-05T21:26:34.736 INFO:tasks.workunit.client.0.smithi121.stdout:2/134: fdatasync d2/d9/fe 0 2024-04-05T21:26:34.737 INFO:tasks.workunit.client.1.smithi188.stdout:4/632: symlink d0/d2/d52/dcc/le3 0 2024-04-05T21:26:34.737 INFO:tasks.workunit.client.0.smithi121.stdout:4/122: fdatasync d5/f16 0 2024-04-05T21:26:34.738 INFO:tasks.workunit.client.1.smithi188.stdout:7/530: dread d3/d4/d1d/d74/f59 [4194304,4194304] 0 2024-04-05T21:26:34.739 INFO:tasks.workunit.client.1.smithi188.stdout:2/458: mkdir d5/db/dd/d36/d3d/d46/d66/d8f 0 2024-04-05T21:26:34.740 INFO:tasks.workunit.client.0.smithi121.stdout:9/110: mknod d6/df/c1e 0 2024-04-05T21:26:34.740 INFO:tasks.workunit.client.1.smithi188.stdout:5/436: write d6/d15/d5c/d7a/d44/f4d [824131,113314] 0 2024-04-05T21:26:34.741 INFO:tasks.workunit.client.1.smithi188.stdout:4/633: unlink d0/d2/d17/d3a/l56 0 2024-04-05T21:26:34.743 INFO:tasks.workunit.client.0.smithi121.stdout:9/111: write d6/f16 [161590,96476] 0 2024-04-05T21:26:34.743 INFO:tasks.workunit.client.0.smithi121.stdout:9/112: fsync d6/f7 0 2024-04-05T21:26:34.744 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:34 smithi121 ceph-mon[39723]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:26:34.744 INFO:tasks.workunit.client.1.smithi188.stdout:4/634: chown d0/d14/d15/d78/db1/d6a/fd9 1 1 2024-04-05T21:26:34.744 INFO:tasks.workunit.client.1.smithi188.stdout:4/635: readlink d0/d2/d17/l8c 0 2024-04-05T21:26:34.745 INFO:tasks.workunit.client.0.smithi121.stdout:8/65: creat ff x:0 0 0 2024-04-05T21:26:34.745 INFO:tasks.workunit.client.0.smithi121.stdout:9/113: chown d6/c1a 1032 1 2024-04-05T21:26:34.745 INFO:tasks.workunit.client.0.smithi121.stdout:9/114: write d6/f17 [2930410,21605] 0 2024-04-05T21:26:34.745 INFO:tasks.workunit.client.0.smithi121.stdout:2/135: mkdir d2/d28 0 2024-04-05T21:26:34.745 INFO:tasks.workunit.client.1.smithi188.stdout:2/459: creat d5/d5f/f90 x:0 0 0 2024-04-05T21:26:34.746 INFO:tasks.workunit.client.0.smithi121.stdout:4/123: mkdir d5/d11/d26/d27/d29/d2d 0 2024-04-05T21:26:34.748 INFO:tasks.workunit.client.0.smithi121.stdout:8/66: symlink l10 0 2024-04-05T21:26:34.750 INFO:tasks.workunit.client.1.smithi188.stdout:2/460: chown d5/db/dd/c13 3 1 2024-04-05T21:26:34.750 INFO:tasks.workunit.client.1.smithi188.stdout:5/437: truncate d6/d17/d39/d8e/f8f 547772 0 2024-04-05T21:26:34.751 INFO:tasks.workunit.client.0.smithi121.stdout:9/115: symlink d6/df/d1d/l1f 0 2024-04-05T21:26:34.751 INFO:tasks.workunit.client.0.smithi121.stdout:4/124: mkdir d5/d11/d26/d27/d29/d2e 0 2024-04-05T21:26:34.752 INFO:tasks.workunit.client.1.smithi188.stdout:7/531: link d3/d2c/d5e/d91/l9b d3/d4/db/d2e/la9 0 2024-04-05T21:26:34.752 INFO:tasks.workunit.client.0.smithi121.stdout:8/67: mknod c11 0 2024-04-05T21:26:34.753 INFO:tasks.workunit.client.1.smithi188.stdout:2/461: truncate d5/db/f20 756735 0 2024-04-05T21:26:34.753 INFO:tasks.workunit.client.0.smithi121.stdout:1/122: dwrite d8/ff [0,4194304] 0 2024-04-05T21:26:34.753 INFO:tasks.workunit.client.0.smithi121.stdout:4/125: dread - d5/d11/d28/f1b zero size 2024-04-05T21:26:34.754 INFO:tasks.workunit.client.1.smithi188.stdout:7/532: mknod d3/d4/db/d2e/d37/caa 0 2024-04-05T21:26:34.755 INFO:tasks.workunit.client.0.smithi121.stdout:9/116: creat d6/df/f20 x:0 0 0 2024-04-05T21:26:34.755 INFO:tasks.workunit.client.0.smithi121.stdout:4/126: readlink d5/l10 0 2024-04-05T21:26:34.755 INFO:tasks.workunit.client.0.smithi121.stdout:2/136: dread d2/d9/fe [0,4194304] 0 2024-04-05T21:26:34.756 INFO:tasks.workunit.client.0.smithi121.stdout:9/117: fdatasync d6/f15 0 2024-04-05T21:26:34.756 INFO:tasks.workunit.client.0.smithi121.stdout:1/123: rmdir d8/d10/d14 39 2024-04-05T21:26:34.757 INFO:tasks.workunit.client.0.smithi121.stdout:1/124: chown d8 312 1 2024-04-05T21:26:34.757 INFO:tasks.workunit.client.1.smithi188.stdout:2/462: rename d5/db/dd/c39 to d5/db/dd/d24/c91 0 2024-04-05T21:26:34.757 INFO:tasks.workunit.client.0.smithi121.stdout:2/137: dread - d2/d9/f1f zero size 2024-04-05T21:26:34.758 INFO:tasks.workunit.client.0.smithi121.stdout:9/118: truncate d6/f1c 657627 0 2024-04-05T21:26:34.758 INFO:tasks.workunit.client.0.smithi121.stdout:2/138: chown d2/d9/d17/l27 1172129219 1 2024-04-05T21:26:34.759 INFO:tasks.workunit.client.0.smithi121.stdout:8/68: mkdir d12 0 2024-04-05T21:26:34.761 INFO:tasks.workunit.client.1.smithi188.stdout:4/636: read d0/d14/d15/d53/d82/d8f/fad [185898,89848] 0 2024-04-05T21:26:34.762 INFO:tasks.workunit.client.0.smithi121.stdout:4/127: creat d5/d11/d28/f2f x:0 0 0 2024-04-05T21:26:34.763 INFO:tasks.workunit.client.1.smithi188.stdout:2/463: rename d5/db/dd/d36/l62 to d5/db/dd/d36/d3d/d46/d8c/l92 0 2024-04-05T21:26:34.763 INFO:tasks.workunit.client.1.smithi188.stdout:2/464: stat d5/db/dd/f54 0 2024-04-05T21:26:34.764 INFO:tasks.workunit.client.1.smithi188.stdout:2/465: dread - d5/db/dd/d67/f8e zero size 2024-04-05T21:26:34.765 INFO:tasks.workunit.client.0.smithi121.stdout:2/139: creat d2/d9/d1c/f29 x:0 0 0 2024-04-05T21:26:34.765 INFO:tasks.workunit.client.1.smithi188.stdout:4/637: creat d0/d2/d52/fe4 x:0 0 0 2024-04-05T21:26:34.767 INFO:tasks.workunit.client.1.smithi188.stdout:2/466: mknod d5/db/dd/d36/d3d/d46/d66/c93 0 2024-04-05T21:26:34.767 INFO:tasks.workunit.client.0.smithi121.stdout:8/69: getdents d12 0 2024-04-05T21:26:34.768 INFO:tasks.workunit.client.0.smithi121.stdout:1/125: dwrite d8/f1d [0,4194304] 0 2024-04-05T21:26:34.768 INFO:tasks.workunit.client.0.smithi121.stdout:8/70: read - ff zero size 2024-04-05T21:26:34.770 INFO:tasks.workunit.client.1.smithi188.stdout:2/467: link d5/db/d35/d3b/f84 d5/db/dd/d36/d3c/d48/d85/f94 0 2024-04-05T21:26:34.772 INFO:tasks.workunit.client.1.smithi188.stdout:2/468: write d5/db/dd/d36/d3c/d48/f56 [90658,19112] 0 2024-04-05T21:26:34.776 INFO:tasks.workunit.client.1.smithi188.stdout:2/469: creat d5/db/dd/f95 x:0 0 0 2024-04-05T21:26:34.776 INFO:tasks.workunit.client.0.smithi121.stdout:9/119: getdents d6/df/d1d 0 2024-04-05T21:26:34.776 INFO:tasks.workunit.client.0.smithi121.stdout:4/128: getdents d5/d11/d26/d27/d29/d2e 0 2024-04-05T21:26:34.776 INFO:tasks.workunit.client.0.smithi121.stdout:8/71: dread f3 [0,4194304] 0 2024-04-05T21:26:34.776 INFO:tasks.workunit.client.1.smithi188.stdout:2/470: truncate d5/f5c 1133078 0 2024-04-05T21:26:34.779 INFO:tasks.workunit.client.1.smithi188.stdout:2/471: mknod d5/db/dd/d36/d3c/d48/d7a/d7c/c96 0 2024-04-05T21:26:34.781 INFO:tasks.workunit.client.0.smithi121.stdout:1/126: mkdir d8/d10/d14/d1a/d1b/d24 0 2024-04-05T21:26:34.782 INFO:tasks.workunit.client.0.smithi121.stdout:8/72: dread f3 [0,4194304] 0 2024-04-05T21:26:34.782 INFO:tasks.workunit.client.0.smithi121.stdout:9/120: unlink d6/f7 0 2024-04-05T21:26:34.782 INFO:tasks.workunit.client.0.smithi121.stdout:9/121: write d6/fc [5736448,821] 0 2024-04-05T21:26:34.782 INFO:tasks.workunit.client.0.smithi121.stdout:4/129: creat d5/d11/d26/d27/f30 x:0 0 0 2024-04-05T21:26:34.782 INFO:tasks.workunit.client.0.smithi121.stdout:1/127: symlink d8/d10/d14/l25 0 2024-04-05T21:26:34.785 INFO:tasks.workunit.client.0.smithi121.stdout:1/128: write d8/f1c [15051,21120] 0 2024-04-05T21:26:34.785 INFO:tasks.workunit.client.0.smithi121.stdout:9/122: mkdir d6/d21 0 2024-04-05T21:26:34.787 INFO:tasks.workunit.client.0.smithi121.stdout:9/123: fdatasync d6/df/f20 0 2024-04-05T21:26:34.789 INFO:tasks.workunit.client.0.smithi121.stdout:8/73: dwrite fa [4194304,4194304] 0 2024-04-05T21:26:34.790 INFO:tasks.workunit.client.0.smithi121.stdout:4/130: rename d5/d11/d28/f2a to d5/d11/d26/d27/d29/d2d/f31 0 2024-04-05T21:26:34.792 INFO:tasks.workunit.client.0.smithi121.stdout:9/124: symlink d6/df/d1d/l22 0 2024-04-05T21:26:34.792 INFO:tasks.workunit.client.1.smithi188.stdout:2/472: dread d5/db/dd/d24/f52 [0,4194304] 0 2024-04-05T21:26:34.793 INFO:tasks.workunit.client.0.smithi121.stdout:9/125: fdatasync d6/f15 0 2024-04-05T21:26:34.793 INFO:tasks.workunit.client.0.smithi121.stdout:8/74: mkdir d12/d13 0 2024-04-05T21:26:34.796 INFO:tasks.workunit.client.0.smithi121.stdout:1/129: rename d8/d10/d14/f18 to d8/d10/f26 0 2024-04-05T21:26:34.797 INFO:tasks.workunit.client.0.smithi121.stdout:1/130: chown d8/d10/d14/d1a/d1b 26539 1 2024-04-05T21:26:34.797 INFO:tasks.workunit.client.0.smithi121.stdout:9/126: creat d6/df/d1d/f23 x:0 0 0 2024-04-05T21:26:34.799 INFO:tasks.workunit.client.0.smithi121.stdout:1/131: mkdir d8/d10/d14/d27 0 2024-04-05T21:26:34.800 INFO:tasks.workunit.client.0.smithi121.stdout:8/75: link c11 d12/c14 0 2024-04-05T21:26:34.802 INFO:tasks.workunit.client.0.smithi121.stdout:8/76: write f6 [1028865,98303] 0 2024-04-05T21:26:34.803 INFO:tasks.workunit.client.0.smithi121.stdout:1/132: mknod d8/d10/d14/d1a/d1b/c28 0 2024-04-05T21:26:34.805 INFO:tasks.workunit.client.0.smithi121.stdout:9/127: dwrite d6/df/f19 [0,4194304] 0 2024-04-05T21:26:34.805 INFO:tasks.workunit.client.0.smithi121.stdout:8/77: mkdir d12/d13/d15 0 2024-04-05T21:26:34.808 INFO:tasks.workunit.client.0.smithi121.stdout:9/128: creat d6/d21/f24 x:0 0 0 2024-04-05T21:26:34.810 INFO:tasks.workunit.client.0.smithi121.stdout:9/129: mknod d6/df/c25 0 2024-04-05T21:26:34.876 INFO:tasks.workunit.client.0.smithi121.stdout:2/140: write d2/d1e/f24 [746616,24684] 0 2024-04-05T21:26:34.876 INFO:tasks.workunit.client.1.smithi188.stdout:7/533: readlink d3/d4/db/d2e/d37/l90 0 2024-04-05T21:26:34.878 INFO:tasks.workunit.client.0.smithi121.stdout:2/141: mknod d2/d1e/c2a 0 2024-04-05T21:26:34.884 INFO:tasks.workunit.client.1.smithi188.stdout:7/534: readlink d3/d4/d1d/d74/l57 0 2024-04-05T21:26:34.884 INFO:tasks.workunit.client.1.smithi188.stdout:7/535: fdatasync d3/f71 0 2024-04-05T21:26:34.885 INFO:tasks.workunit.client.0.smithi121.stdout:2/142: dwrite d2/d1e/f24 [0,4194304] 0 2024-04-05T21:26:34.886 INFO:tasks.workunit.client.1.smithi188.stdout:5/438: dread d6/f5b [0,4194304] 0 2024-04-05T21:26:34.886 INFO:tasks.workunit.client.0.smithi121.stdout:9/130: dread d6/f1c [0,4194304] 0 2024-04-05T21:26:34.887 INFO:tasks.workunit.client.0.smithi121.stdout:2/143: dread d2/d9/fb [0,4194304] 0 2024-04-05T21:26:34.888 INFO:tasks.workunit.client.1.smithi188.stdout:7/536: unlink d3/d4/db/d2e/d49/f5d 0 2024-04-05T21:26:34.888 INFO:tasks.workunit.client.0.smithi121.stdout:0/176: sync 2024-04-05T21:26:34.888 INFO:tasks.workunit.client.0.smithi121.stdout:5/122: sync 2024-04-05T21:26:34.888 INFO:tasks.workunit.client.0.smithi121.stdout:6/87: sync 2024-04-05T21:26:34.889 INFO:tasks.workunit.client.1.smithi188.stdout:5/439: rmdir d6/d17 39 2024-04-05T21:26:34.889 INFO:tasks.workunit.client.0.smithi121.stdout:9/131: mkdir d6/df/d1d/d26 0 2024-04-05T21:26:34.889 INFO:tasks.workunit.client.0.smithi121.stdout:0/177: write d7/f13 [1652455,30978] 0 2024-04-05T21:26:34.891 INFO:tasks.workunit.client.0.smithi121.stdout:0/178: readlink d7/dc/d10/l30 0 2024-04-05T21:26:34.891 INFO:tasks.workunit.client.0.smithi121.stdout:4/131: truncate d5/d11/d28/f13 3756688 0 2024-04-05T21:26:34.893 INFO:tasks.workunit.client.0.smithi121.stdout:6/88: creat d1/d2/dc/f19 x:0 0 0 2024-04-05T21:26:34.896 INFO:tasks.workunit.client.0.smithi121.stdout:9/132: creat d6/df/f27 x:0 0 0 2024-04-05T21:26:34.898 INFO:tasks.workunit.client.1.smithi188.stdout:4/638: dwrite d0/d14/d15/d53/d82/d8f/fad [0,4194304] 0 2024-04-05T21:26:34.902 INFO:tasks.workunit.client.0.smithi121.stdout:5/123: dwrite dd/de/f22 [0,4194304] 0 2024-04-05T21:26:34.902 INFO:tasks.workunit.client.0.smithi121.stdout:8/78: fsync fa 0 2024-04-05T21:26:34.903 INFO:tasks.workunit.client.1.smithi188.stdout:5/440: dwrite d6/d15/f79 [0,4194304] 0 2024-04-05T21:26:34.904 INFO:tasks.workunit.client.0.smithi121.stdout:1/133: write f2 [1986480,54755] 0 2024-04-05T21:26:34.905 INFO:tasks.workunit.client.1.smithi188.stdout:4/639: fdatasync d0/d14/d15/d42/f63 0 2024-04-05T21:26:34.905 INFO:tasks.workunit.client.1.smithi188.stdout:7/537: dwrite d3/d4/db/d2e/f75 [0,4194304] 0 2024-04-05T21:26:34.908 INFO:tasks.workunit.client.1.smithi188.stdout:4/640: fdatasync d0/d2/d17/fa7 0 2024-04-05T21:26:34.908 INFO:tasks.workunit.client.1.smithi188.stdout:2/473: dread d5/db/dd/f54 [0,4194304] 0 2024-04-05T21:26:34.911 INFO:tasks.workunit.client.0.smithi121.stdout:6/89: dwrite d1/d2/f13 [0,4194304] 0 2024-04-05T21:26:34.912 INFO:tasks.workunit.client.0.smithi121.stdout:0/179: dwrite d7/d1c/f1f [0,4194304] 0 2024-04-05T21:26:34.914 INFO:tasks.workunit.client.1.smithi188.stdout:2/474: write d5/d5f/f90 [39553,28401] 0 2024-04-05T21:26:34.915 INFO:tasks.workunit.client.1.smithi188.stdout:4/641: rmdir d0/d14/d15/d78/db1/daa 39 2024-04-05T21:26:34.916 INFO:tasks.workunit.client.1.smithi188.stdout:7/538: symlink d3/d4/d7c/lab 0 2024-04-05T21:26:34.916 INFO:tasks.workunit.client.1.smithi188.stdout:5/441: getdents d6/d15/d1b/d4f 0 2024-04-05T21:26:34.917 INFO:tasks.workunit.client.0.smithi121.stdout:5/124: truncate dd/de/f21 1862808 0 2024-04-05T21:26:34.918 INFO:tasks.workunit.client.1.smithi188.stdout:4/642: mknod d0/d2/d52/ce5 0 2024-04-05T21:26:34.918 INFO:tasks.workunit.client.0.smithi121.stdout:6/90: dread - d1/d2/dc/f19 zero size 2024-04-05T21:26:34.919 INFO:tasks.workunit.client.1.smithi188.stdout:5/442: readlink d6/l28 0 2024-04-05T21:26:34.921 INFO:tasks.workunit.client.1.smithi188.stdout:7/539: creat d3/d2c/d5e/d8e/fac x:0 0 0 2024-04-05T21:26:34.922 INFO:tasks.workunit.client.1.smithi188.stdout:5/443: write d6/d15/d5c/d7a/f52 [1497576,68861] 0 2024-04-05T21:26:34.923 INFO:tasks.workunit.client.0.smithi121.stdout:9/133: truncate d6/f16 102603 0 2024-04-05T21:26:34.923 INFO:tasks.workunit.client.0.smithi121.stdout:5/125: write dd/de/f22 [4767818,104295] 0 2024-04-05T21:26:34.924 INFO:tasks.workunit.client.0.smithi121.stdout:8/79: unlink fb 0 2024-04-05T21:26:34.925 INFO:tasks.workunit.client.0.smithi121.stdout:6/91: truncate d1/d2/d3/d5/fd 679412 0 2024-04-05T21:26:34.927 INFO:tasks.workunit.client.0.smithi121.stdout:1/134: creat d8/d10/d14/d27/f29 x:0 0 0 2024-04-05T21:26:34.928 INFO:tasks.workunit.client.0.smithi121.stdout:8/80: rmdir d12 39 2024-04-05T21:26:34.929 INFO:tasks.workunit.client.1.smithi188.stdout:7/540: stat d3/d4/db/d55/c7b 0 2024-04-05T21:26:34.930 INFO:tasks.workunit.client.1.smithi188.stdout:2/475: getdents d5/db/dd/d36/d3c/d48/d7a 0 2024-04-05T21:26:34.931 INFO:tasks.workunit.client.0.smithi121.stdout:5/126: rename dd/de to dd/de/d19/d1c/d27 22 2024-04-05T21:26:34.931 INFO:tasks.workunit.client.0.smithi121.stdout:9/134: unlink d6/df/d1d/l1f 0 2024-04-05T21:26:34.932 INFO:tasks.workunit.client.1.smithi188.stdout:5/444: rename d6/d17/d23/f83 to d6/d15/d5c/d7a/fa1 0 2024-04-05T21:26:34.934 INFO:tasks.workunit.client.0.smithi121.stdout:0/180: dwrite d7/f13 [0,4194304] 0 2024-04-05T21:26:34.934 INFO:tasks.workunit.client.0.smithi121.stdout:9/135: chown d6/f15 74 1 2024-04-05T21:26:34.936 INFO:tasks.workunit.client.0.smithi121.stdout:4/132: dwrite d5/f9 [0,4194304] 0 2024-04-05T21:26:34.937 INFO:tasks.workunit.client.0.smithi121.stdout:9/136: chown d6/df/d1d/l22 331743 1 2024-04-05T21:26:34.938 INFO:tasks.workunit.client.1.smithi188.stdout:4/643: dwrite d0/d2/d17/d48/d70/fc0 [0,4194304] 0 2024-04-05T21:26:34.938 INFO:tasks.workunit.client.0.smithi121.stdout:9/137: fsync d6/fc 0 2024-04-05T21:26:34.939 INFO:tasks.workunit.client.1.smithi188.stdout:2/476: symlink d5/l97 0 2024-04-05T21:26:34.939 INFO:tasks.workunit.client.0.smithi121.stdout:8/81: rename l4 to d12/d13/d15/l16 0 2024-04-05T21:26:34.941 INFO:tasks.workunit.client.0.smithi121.stdout:4/133: symlink d5/d11/l32 0 2024-04-05T21:26:34.945 INFO:tasks.workunit.client.1.smithi188.stdout:7/541: dwrite d3/d4/db/d10/f95 [0,4194304] 0 2024-04-05T21:26:34.945 INFO:tasks.workunit.client.1.smithi188.stdout:5/445: dread d6/d17/d39/d8e/f8f [0,4194304] 0 2024-04-05T21:26:34.947 INFO:tasks.workunit.client.0.smithi121.stdout:0/181: mkdir d7/d1c/d1d/d29/d2a/d39 0 2024-04-05T21:26:34.947 INFO:tasks.workunit.client.0.smithi121.stdout:5/127: creat dd/de/d19/f28 x:0 0 0 2024-04-05T21:26:34.947 INFO:tasks.workunit.client.0.smithi121.stdout:8/82: mknod d12/c17 0 2024-04-05T21:26:34.948 INFO:tasks.workunit.client.0.smithi121.stdout:4/134: truncate d5/d11/f17 6019078 0 2024-04-05T21:26:34.950 INFO:tasks.workunit.client.0.smithi121.stdout:8/83: truncate fc 782621 0 2024-04-05T21:26:34.951 INFO:tasks.workunit.client.1.smithi188.stdout:4/644: creat d0/d2/d52/dc1/fe6 x:0 0 0 2024-04-05T21:26:34.951 INFO:tasks.workunit.client.1.smithi188.stdout:7/542: write d3/d2c/f7e [1708331,10878] 0 2024-04-05T21:26:34.951 INFO:tasks.workunit.client.0.smithi121.stdout:5/128: creat dd/f29 x:0 0 0 2024-04-05T21:26:34.952 INFO:tasks.workunit.client.0.smithi121.stdout:8/84: chown d12/d13 8 1 2024-04-05T21:26:34.953 INFO:tasks.workunit.client.0.smithi121.stdout:9/138: dwrite d6/f15 [0,4194304] 0 2024-04-05T21:26:34.954 INFO:tasks.workunit.client.1.smithi188.stdout:7/543: dread - d3/d4/db/d2e/d3b/f92 zero size 2024-04-05T21:26:34.954 INFO:tasks.workunit.client.0.smithi121.stdout:5/129: read - dd/f29 zero size 2024-04-05T21:26:34.955 INFO:tasks.workunit.client.0.smithi121.stdout:4/135: rmdir d5/d11/d26/d27/d29/d2d 39 2024-04-05T21:26:34.955 INFO:tasks.workunit.client.0.smithi121.stdout:6/92: fdatasync d1/d2/f13 0 2024-04-05T21:26:34.956 INFO:tasks.workunit.client.1.smithi188.stdout:2/477: getdents d5/db/dd/d36/d3d/d46/d8c 0 2024-04-05T21:26:34.958 INFO:tasks.workunit.client.1.smithi188.stdout:7/544: creat d3/d4/db/d2e/d37/d65/fad x:0 0 0 2024-04-05T21:26:34.961 INFO:tasks.workunit.client.0.smithi121.stdout:9/139: mknod d6/d21/c28 0 2024-04-05T21:26:34.961 INFO:tasks.workunit.client.0.smithi121.stdout:8/85: creat d12/d13/d15/f18 x:0 0 0 2024-04-05T21:26:34.961 INFO:tasks.workunit.client.1.smithi188.stdout:7/545: truncate d3/d4/db/d2e/d49/d79/f97 856434 0 2024-04-05T21:26:34.963 INFO:tasks.workunit.client.1.smithi188.stdout:5/446: link d6/d15/d5c/d7a/d44/c4b d6/d15/d1b/ca2 0 2024-04-05T21:26:34.964 INFO:tasks.workunit.client.1.smithi188.stdout:5/447: stat d6/d15/c84 0 2024-04-05T21:26:34.965 INFO:tasks.workunit.client.1.smithi188.stdout:7/546: dread d3/d4/db/d2e/d3b/f4d [0,4194304] 0 2024-04-05T21:26:34.967 INFO:tasks.workunit.client.0.smithi121.stdout:6/93: rename d1/d2/d3/d5/l18 to d1/l1a 0 2024-04-05T21:26:34.967 INFO:tasks.workunit.client.0.smithi121.stdout:1/135: rmdir d8/d10 39 2024-04-05T21:26:34.968 INFO:tasks.workunit.client.1.smithi188.stdout:5/448: creat d6/d15/d5c/fa3 x:0 0 0 2024-04-05T21:26:34.969 INFO:tasks.workunit.client.0.smithi121.stdout:4/136: dwrite d5/fd [0,4194304] 0 2024-04-05T21:26:34.969 INFO:tasks.workunit.client.1.smithi188.stdout:7/547: unlink d3/d4/db/d2e/d8c/ca4 0 2024-04-05T21:26:34.969 INFO:tasks.workunit.client.1.smithi188.stdout:5/449: dread - d6/d15/d5c/d7a/d44/d98/f9f zero size 2024-04-05T21:26:34.970 INFO:tasks.workunit.client.0.smithi121.stdout:8/86: creat d12/d13/d15/f19 x:0 0 0 2024-04-05T21:26:34.971 INFO:tasks.workunit.client.1.smithi188.stdout:7/548: write d3/d4/db/d2e/d37/d65/f7d [2687223,118498] 0 2024-04-05T21:26:34.972 INFO:tasks.workunit.client.0.smithi121.stdout:8/87: chown d12/c14 3 1 2024-04-05T21:26:34.972 INFO:tasks.workunit.client.0.smithi121.stdout:9/140: dread d6/df/f19 [0,4194304] 0 2024-04-05T21:26:34.973 INFO:tasks.workunit.client.1.smithi188.stdout:5/450: mknod d6/d15/d1b/d4f/ca4 0 2024-04-05T21:26:34.974 INFO:tasks.workunit.client.0.smithi121.stdout:6/94: mkdir d1/d2/d1b 0 2024-04-05T21:26:34.975 INFO:tasks.workunit.client.0.smithi121.stdout:8/88: dread f3 [0,4194304] 0 2024-04-05T21:26:34.976 INFO:tasks.workunit.client.0.smithi121.stdout:4/137: unlink d5/d11/f1c 0 2024-04-05T21:26:34.977 INFO:tasks.workunit.client.0.smithi121.stdout:0/182: dread f2 [0,4194304] 0 2024-04-05T21:26:34.979 INFO:tasks.workunit.client.0.smithi121.stdout:6/95: truncate d1/d2/dc/f19 1024756 0 2024-04-05T21:26:34.979 INFO:tasks.workunit.client.0.smithi121.stdout:8/89: symlink d12/l1a 0 2024-04-05T21:26:34.979 INFO:tasks.workunit.client.0.smithi121.stdout:4/138: write d5/fd [456593,31018] 0 2024-04-05T21:26:34.982 INFO:tasks.workunit.client.0.smithi121.stdout:0/183: dread - d7/d1c/d1d/d29/f37 zero size 2024-04-05T21:26:34.982 INFO:tasks.workunit.client.0.smithi121.stdout:9/141: dwrite d6/f17 [0,4194304] 0 2024-04-05T21:26:34.985 INFO:tasks.workunit.client.1.smithi188.stdout:2/478: write d5/fa [1650310,72724] 0 2024-04-05T21:26:34.985 INFO:tasks.workunit.client.0.smithi121.stdout:8/90: creat d12/d13/f1b x:0 0 0 2024-04-05T21:26:34.986 INFO:tasks.workunit.client.1.smithi188.stdout:5/451: write d6/d15/d5c/d7a/d44/f57 [1449127,83032] 0 2024-04-05T21:26:34.987 INFO:tasks.workunit.client.1.smithi188.stdout:0/273: sync 2024-04-05T21:26:34.988 INFO:tasks.workunit.client.0.smithi121.stdout:4/139: unlink d5/f9 0 2024-04-05T21:26:34.988 INFO:tasks.workunit.client.0.smithi121.stdout:5/130: getdents dd 0 2024-04-05T21:26:34.991 INFO:tasks.workunit.client.1.smithi188.stdout:5/452: creat d6/d82/d40/fa5 x:0 0 0 2024-04-05T21:26:34.992 INFO:tasks.workunit.client.1.smithi188.stdout:4/645: dwrite d0/d2/d17/d48/f57 [0,4194304] 0 2024-04-05T21:26:34.993 INFO:tasks.workunit.client.0.smithi121.stdout:8/91: creat d12/d13/d15/f1c x:0 0 0 2024-04-05T21:26:34.995 INFO:tasks.workunit.client.0.smithi121.stdout:6/96: link d1/d2/f17 d1/f1c 0 2024-04-05T21:26:34.996 INFO:tasks.workunit.client.0.smithi121.stdout:6/97: chown d1/d2/d3/d5/l8 416 1 2024-04-05T21:26:34.997 INFO:tasks.workunit.client.1.smithi188.stdout:5/453: mkdir d6/d82/d40/d42/d81/da6 0 2024-04-05T21:26:34.997 INFO:tasks.workunit.client.1.smithi188.stdout:0/274: mkdir d4/d8/db/d66 0 2024-04-05T21:26:34.998 INFO:tasks.workunit.client.0.smithi121.stdout:9/142: symlink d6/df/d1d/d26/l29 0 2024-04-05T21:26:34.998 INFO:tasks.workunit.client.0.smithi121.stdout:6/98: chown d1/d2/d3/d4/ca 3287629 1 2024-04-05T21:26:35.001 INFO:tasks.workunit.client.0.smithi121.stdout:8/92: mkdir d12/d13/d15/d1d 0 2024-04-05T21:26:35.001 INFO:tasks.workunit.client.0.smithi121.stdout:1/136: truncate d8/f1d 2539182 0 2024-04-05T21:26:35.002 INFO:tasks.workunit.client.0.smithi121.stdout:8/93: fdatasync d12/d13/d15/f18 0 2024-04-05T21:26:35.003 INFO:tasks.workunit.client.0.smithi121.stdout:1/137: write d8/ff [1821260,12408] 0 2024-04-05T21:26:35.003 INFO:tasks.workunit.client.1.smithi188.stdout:4/646: dwrite d0/d14/d15/d78/db1/d6a/fd9 [0,4194304] 0 2024-04-05T21:26:35.003 INFO:tasks.workunit.client.1.smithi188.stdout:0/275: link d4/d6/d46/f47 d4/d8/d10/f67 0 2024-04-05T21:26:35.004 INFO:tasks.workunit.client.0.smithi121.stdout:9/143: creat d6/f2a x:0 0 0 2024-04-05T21:26:35.004 INFO:tasks.workunit.client.0.smithi121.stdout:6/99: fsync d1/f10 0 2024-04-05T21:26:35.006 INFO:tasks.workunit.client.1.smithi188.stdout:4/647: write d0/d2/d4f/f67 [4190247,111992] 0 2024-04-05T21:26:35.006 INFO:tasks.workunit.client.1.smithi188.stdout:0/276: mknod d4/d8/d39/c68 0 2024-04-05T21:26:35.007 INFO:tasks.workunit.client.0.smithi121.stdout:5/131: link dd/f29 dd/de/d19/f2a 0 2024-04-05T21:26:35.007 INFO:tasks.workunit.client.0.smithi121.stdout:5/132: read - dd/de/d19/f2a zero size 2024-04-05T21:26:35.008 INFO:tasks.workunit.client.1.smithi188.stdout:4/648: chown d0/d14/d15/d53/d82/cc4 8305 1 2024-04-05T21:26:35.010 INFO:tasks.workunit.client.0.smithi121.stdout:1/138: write d8/d10/d14/d27/f29 [635961,88538] 0 2024-04-05T21:26:35.011 INFO:tasks.workunit.client.0.smithi121.stdout:9/144: creat d6/df/d1d/f2b x:0 0 0 2024-04-05T21:26:35.012 INFO:tasks.workunit.client.1.smithi188.stdout:0/277: mkdir d4/d6/d13/d23/d24/d69 0 2024-04-05T21:26:35.013 INFO:tasks.workunit.client.1.smithi188.stdout:2/479: fsync d5/fa 0 2024-04-05T21:26:35.013 INFO:tasks.workunit.client.1.smithi188.stdout:4/649: symlink d0/d2/d4f/le7 0 2024-04-05T21:26:35.013 INFO:tasks.workunit.client.0.smithi121.stdout:9/145: dread - d6/df/d1d/f23 zero size 2024-04-05T21:26:35.014 INFO:tasks.workunit.client.1.smithi188.stdout:4/650: chown d0/d14/d15/d53/c84 5310 1 2024-04-05T21:26:35.015 INFO:tasks.workunit.client.1.smithi188.stdout:2/480: mkdir d5/d5f/d98 0 2024-04-05T21:26:35.016 INFO:tasks.workunit.client.1.smithi188.stdout:0/278: creat d4/d6/d61/f6a x:0 0 0 2024-04-05T21:26:35.017 INFO:tasks.workunit.client.0.smithi121.stdout:0/184: dwrite d7/f9 [0,4194304] 0 2024-04-05T21:26:35.018 INFO:tasks.workunit.client.0.smithi121.stdout:5/133: rename dd/de/d19/f2a to dd/de/d19/d1c/f2b 0 2024-04-05T21:26:35.018 INFO:tasks.workunit.client.1.smithi188.stdout:2/481: creat d5/db/dd/d36/d3c/d48/d7a/f99 x:0 0 0 2024-04-05T21:26:35.021 INFO:tasks.workunit.client.1.smithi188.stdout:7/549: dread d3/d4/db/f4a [0,4194304] 0 2024-04-05T21:26:35.022 INFO:tasks.workunit.client.1.smithi188.stdout:7/550: fsync d3/d4/db/d2e/f35 0 2024-04-05T21:26:35.023 INFO:tasks.workunit.client.1.smithi188.stdout:2/482: creat d5/db/d35/d3b/d6b/f9a x:0 0 0 2024-04-05T21:26:35.024 INFO:tasks.workunit.client.0.smithi121.stdout:4/140: link d5/d11/d28/l24 d5/d11/d26/d27/d29/d2d/l33 0 2024-04-05T21:26:35.024 INFO:tasks.workunit.client.0.smithi121.stdout:9/146: rename d6/d21 to d6/df/d1d/d2c 0 2024-04-05T21:26:35.025 INFO:tasks.workunit.client.0.smithi121.stdout:8/94: link cd d12/d13/d15/c1e 0 2024-04-05T21:26:35.026 INFO:tasks.workunit.client.0.smithi121.stdout:1/139: creat d8/d10/d14/d1a/d1b/d24/f2a x:0 0 0 2024-04-05T21:26:35.026 INFO:tasks.workunit.client.0.smithi121.stdout:1/140: chown d8/d10/d14/d1a 505978 1 2024-04-05T21:26:35.028 INFO:tasks.workunit.client.0.smithi121.stdout:9/147: creat d6/df/d1d/f2d x:0 0 0 2024-04-05T21:26:35.029 INFO:tasks.workunit.client.1.smithi188.stdout:7/551: rename d3/d4/db/d2e/d49/d79 to d3/d2c/dae 0 2024-04-05T21:26:35.030 INFO:tasks.workunit.client.0.smithi121.stdout:6/100: getdents d1/d2 0 2024-04-05T21:26:35.030 INFO:tasks.workunit.client.1.smithi188.stdout:7/552: write d3/d2c/d5e/d8e/fac [307456,23251] 0 2024-04-05T21:26:35.031 INFO:tasks.workunit.client.0.smithi121.stdout:6/101: write d1/d2/f13 [3176480,120082] 0 2024-04-05T21:26:35.031 INFO:tasks.workunit.client.0.smithi121.stdout:1/141: mkdir d8/d10/d14/d1a/d1b/d24/d2b 0 2024-04-05T21:26:35.032 INFO:tasks.workunit.client.1.smithi188.stdout:7/553: readlink d3/d4/db/d2e/d37/l4b 0 2024-04-05T21:26:35.032 INFO:tasks.workunit.client.0.smithi121.stdout:4/141: truncate d5/fa 698270 0 2024-04-05T21:26:35.034 INFO:tasks.workunit.client.0.smithi121.stdout:9/148: creat d6/df/f2e x:0 0 0 2024-04-05T21:26:35.035 INFO:tasks.workunit.client.0.smithi121.stdout:0/185: fdatasync d7/f9 0 2024-04-05T21:26:35.035 INFO:tasks.workunit.client.0.smithi121.stdout:9/149: chown f4 115786812 1 2024-04-05T21:26:35.035 INFO:tasks.workunit.client.1.smithi188.stdout:7/554: truncate d3/d4/d1d/d74/d73/f7a 590886 0 2024-04-05T21:26:35.035 INFO:tasks.workunit.client.0.smithi121.stdout:9/150: dread - d6/df/f20 zero size 2024-04-05T21:26:35.036 INFO:tasks.workunit.client.0.smithi121.stdout:8/95: link fa d12/d13/d15/f1f 0 2024-04-05T21:26:35.037 INFO:tasks.workunit.client.0.smithi121.stdout:6/102: readlink d1/l15 0 2024-04-05T21:26:35.038 INFO:tasks.workunit.client.1.smithi188.stdout:7/555: dread d3/d4/d1d/d74/d73/f7a [0,4194304] 0 2024-04-05T21:26:35.038 INFO:tasks.workunit.client.0.smithi121.stdout:1/142: fsync d8/fc 0 2024-04-05T21:26:35.039 INFO:tasks.workunit.client.0.smithi121.stdout:4/142: dread d5/fd [0,4194304] 0 2024-04-05T21:26:35.041 INFO:tasks.workunit.client.0.smithi121.stdout:0/186: creat d7/f3a x:0 0 0 2024-04-05T21:26:35.042 INFO:tasks.workunit.client.1.smithi188.stdout:7/556: rename d3/d2c/f7e to d3/d2c/d5e/d8e/faf 0 2024-04-05T21:26:35.042 INFO:tasks.workunit.client.0.smithi121.stdout:9/151: dread d6/f17 [0,4194304] 0 2024-04-05T21:26:35.042 INFO:tasks.workunit.client.1.smithi188.stdout:0/279: write d4/d8/db/d21/f2b [951286,90868] 0 2024-04-05T21:26:35.044 INFO:tasks.workunit.client.1.smithi188.stdout:4/651: dread d0/d14/d15/d1b/f40 [0,4194304] 0 2024-04-05T21:26:35.044 INFO:tasks.workunit.client.0.smithi121.stdout:0/187: dread d7/dc/f14 [0,4194304] 0 2024-04-05T21:26:35.044 INFO:tasks.workunit.client.0.smithi121.stdout:8/96: creat d12/d13/f20 x:0 0 0 2024-04-05T21:26:35.045 INFO:tasks.workunit.client.1.smithi188.stdout:0/280: symlink d4/d8/db/l6b 0 2024-04-05T21:26:35.046 INFO:tasks.workunit.client.1.smithi188.stdout:7/557: write d3/d4/db/d10/f33 [64955,47198] 0 2024-04-05T21:26:35.051 INFO:tasks.workunit.client.0.smithi121.stdout:9/152: symlink d6/df/d1d/l2f 0 2024-04-05T21:26:35.052 INFO:tasks.workunit.client.1.smithi188.stdout:2/483: dwrite d5/db/f20 [0,4194304] 0 2024-04-05T21:26:35.052 INFO:tasks.workunit.client.0.smithi121.stdout:4/143: rmdir d5/d11/d26/d27 39 2024-04-05T21:26:35.053 INFO:tasks.workunit.client.0.smithi121.stdout:0/188: creat d7/d1c/d1d/f3b x:0 0 0 2024-04-05T21:26:35.054 INFO:tasks.workunit.client.0.smithi121.stdout:8/97: creat d12/d13/d15/f21 x:0 0 0 2024-04-05T21:26:35.055 INFO:tasks.workunit.client.1.smithi188.stdout:0/281: symlink d4/d6/d13/d23/d24/d69/l6c 0 2024-04-05T21:26:35.056 INFO:tasks.workunit.client.0.smithi121.stdout:5/134: unlink dd/de/d19/d1c/f2b 0 2024-04-05T21:26:35.057 INFO:tasks.workunit.client.1.smithi188.stdout:2/484: write d5/f2f [7628967,58658] 0 2024-04-05T21:26:35.058 INFO:tasks.workunit.client.0.smithi121.stdout:9/153: symlink d6/df/d1d/d2c/l30 0 2024-04-05T21:26:35.058 INFO:tasks.workunit.client.0.smithi121.stdout:9/154: fdatasync d6/f2a 0 2024-04-05T21:26:35.060 INFO:tasks.workunit.client.0.smithi121.stdout:4/144: dread - d5/d11/d26/d27/f30 zero size 2024-04-05T21:26:35.060 INFO:tasks.workunit.client.1.smithi188.stdout:4/652: dwrite d0/d2/d4f/f67 [0,4194304] 0 2024-04-05T21:26:35.061 INFO:tasks.workunit.client.1.smithi188.stdout:7/558: dread d3/d4/f7 [0,4194304] 0 2024-04-05T21:26:35.061 INFO:tasks.workunit.client.0.smithi121.stdout:0/189: mkdir d7/dc/d10/d3c 0 2024-04-05T21:26:35.061 INFO:tasks.workunit.client.1.smithi188.stdout:0/282: creat d4/d8/d10/f6d x:0 0 0 2024-04-05T21:26:35.062 INFO:tasks.workunit.client.0.smithi121.stdout:0/190: write d7/dc/f2d [1352989,35730] 0 2024-04-05T21:26:35.063 INFO:tasks.workunit.client.1.smithi188.stdout:2/485: stat d5/db/dd/d67 0 2024-04-05T21:26:35.065 INFO:tasks.workunit.client.0.smithi121.stdout:5/135: symlink dd/l2c 0 2024-04-05T21:26:35.066 INFO:tasks.workunit.client.1.smithi188.stdout:2/486: rmdir d5/db/d35 39 2024-04-05T21:26:35.072 INFO:tasks.workunit.client.0.smithi121.stdout:7/122: sync 2024-04-05T21:26:35.075 INFO:tasks.workunit.client.1.smithi188.stdout:0/283: unlink d4/d6/d13/d23/d24/d69/l6c 0 2024-04-05T21:26:35.075 INFO:tasks.workunit.client.1.smithi188.stdout:0/284: dread - d4/d8/d10/f67 zero size 2024-04-05T21:26:35.075 INFO:tasks.workunit.client.1.smithi188.stdout:7/559: dwrite d3/d4/d1d/d74/d88/fa8 [0,4194304] 0 2024-04-05T21:26:35.077 INFO:tasks.workunit.client.1.smithi188.stdout:4/653: getdents d0/d14/d15/d53/d82/d8f/d95 0 2024-04-05T21:26:35.077 INFO:tasks.workunit.client.1.smithi188.stdout:2/487: creat d5/f9b x:0 0 0 2024-04-05T21:26:35.078 INFO:tasks.workunit.client.1.smithi188.stdout:0/285: symlink d4/d6/d13/d23/d24/l6e 0 2024-04-05T21:26:35.080 INFO:tasks.workunit.client.1.smithi188.stdout:7/560: symlink d3/d4/db/d10/lb0 0 2024-04-05T21:26:35.080 INFO:tasks.workunit.client.1.smithi188.stdout:0/286: write d4/d6/d46/f59 [201270,58616] 0 2024-04-05T21:26:35.084 INFO:tasks.workunit.client.1.smithi188.stdout:4/654: link d0/d2/d17/d48/d70/fcb d0/d2/d2a/d34/fe8 0 2024-04-05T21:26:35.085 INFO:tasks.workunit.client.1.smithi188.stdout:0/287: symlink d4/d6/d46/l6f 0 2024-04-05T21:26:35.086 INFO:tasks.workunit.client.1.smithi188.stdout:7/561: symlink d3/d2c/d5e/d8e/lb1 0 2024-04-05T21:26:35.086 INFO:tasks.workunit.client.1.smithi188.stdout:7/562: chown d3/d2c/d5e/d91 732758 1 2024-04-05T21:26:35.087 INFO:tasks.workunit.client.1.smithi188.stdout:2/488: rename d5/db/d35/f43 to d5/db/dd/d36/d3c/d63/d75/d87/f9c 0 2024-04-05T21:26:35.088 INFO:tasks.workunit.client.0.smithi121.stdout:6/103: getdents d1/d2/d3/d5 0 2024-04-05T21:26:35.089 INFO:tasks.workunit.client.1.smithi188.stdout:0/288: symlink d4/d6/d46/l70 0 2024-04-05T21:26:35.090 INFO:tasks.workunit.client.0.smithi121.stdout:0/191: rename d7/c24 to d7/d1c/d1d/c3d 0 2024-04-05T21:26:35.092 INFO:tasks.workunit.client.0.smithi121.stdout:4/145: mknod d5/d11/d2c/c34 0 2024-04-05T21:26:35.093 INFO:tasks.workunit.client.1.smithi188.stdout:4/655: dwrite d0/d14/d15/d42/fa3 [0,4194304] 0 2024-04-05T21:26:35.097 INFO:tasks.workunit.client.1.smithi188.stdout:7/563: dwrite d3/f1e [0,4194304] 0 2024-04-05T21:26:35.097 INFO:tasks.workunit.client.0.smithi121.stdout:1/143: dwrite d8/fc [0,4194304] 0 2024-04-05T21:26:35.097 INFO:tasks.workunit.client.1.smithi188.stdout:7/564: chown d3/f71 50913 1 2024-04-05T21:26:35.100 INFO:tasks.workunit.client.1.smithi188.stdout:4/656: mknod d0/d14/d15/ce9 0 2024-04-05T21:26:35.101 INFO:tasks.workunit.client.1.smithi188.stdout:4/657: chown d0/d14/d2f/l68 483 1 2024-04-05T21:26:35.102 INFO:tasks.workunit.client.1.smithi188.stdout:7/565: unlink d3/d2c/d5e/d8e/f9f 0 2024-04-05T21:26:35.102 INFO:tasks.workunit.client.1.smithi188.stdout:4/658: fsync d0/d2/d52/fe4 0 2024-04-05T21:26:35.104 INFO:tasks.workunit.client.1.smithi188.stdout:4/659: dread - d0/d2/d52/fe4 zero size 2024-04-05T21:26:35.104 INFO:tasks.workunit.client.1.smithi188.stdout:0/289: rmdir d4/d6/d13/d23/d24/d69 0 2024-04-05T21:26:35.107 INFO:tasks.workunit.client.1.smithi188.stdout:2/489: write d5/db/dd/d36/d3c/d48/f5b [1340887,92288] 0 2024-04-05T21:26:35.110 INFO:tasks.workunit.client.1.smithi188.stdout:4/660: write d0/d14/d15/d78/f97 [350505,89183] 0 2024-04-05T21:26:35.110 INFO:tasks.workunit.client.1.smithi188.stdout:0/290: unlink d4/d8/db/d42/f30 0 2024-04-05T21:26:35.112 INFO:tasks.workunit.client.1.smithi188.stdout:2/490: dread - d5/db/dd/d36/d3c/d48/d78/f83 zero size 2024-04-05T21:26:35.112 INFO:tasks.workunit.client.1.smithi188.stdout:7/566: dwrite d3/d4/d1d/f34 [0,4194304] 0 2024-04-05T21:26:35.113 INFO:tasks.workunit.client.1.smithi188.stdout:0/291: readlink d4/d6/d13/d23/d24/l6e 0 2024-04-05T21:26:35.116 INFO:tasks.workunit.client.1.smithi188.stdout:4/661: creat d0/d2/d52/dcc/fea x:0 0 0 2024-04-05T21:26:35.118 INFO:tasks.workunit.client.1.smithi188.stdout:0/292: mkdir d4/d8/db/d42/d71 0 2024-04-05T21:26:35.119 INFO:tasks.workunit.client.1.smithi188.stdout:7/567: symlink d3/d4/db/d55/d9d/lb2 0 2024-04-05T21:26:35.123 INFO:tasks.workunit.client.1.smithi188.stdout:7/568: readlink d3/d4/db/d55/d9d/lb2 0 2024-04-05T21:26:35.124 INFO:tasks.workunit.client.1.smithi188.stdout:2/491: dwrite d5/db/dd/d36/d3c/d48/f5b [0,4194304] 0 2024-04-05T21:26:35.124 INFO:tasks.workunit.client.0.smithi121.stdout:7/123: rename d0/d9/f29 to d0/d2d/f2f 0 2024-04-05T21:26:35.125 INFO:tasks.workunit.client.1.smithi188.stdout:0/293: creat d4/d8/db/d42/d38/f72 x:0 0 0 2024-04-05T21:26:35.125 INFO:tasks.workunit.client.0.smithi121.stdout:5/136: dwrite dd/de/f1a [0,4194304] 0 2024-04-05T21:26:35.127 INFO:tasks.workunit.client.0.smithi121.stdout:0/192: symlink d7/d1c/d1d/l3e 0 2024-04-05T21:26:35.128 INFO:tasks.workunit.client.0.smithi121.stdout:0/193: stat d7/d1c/d1d/d29/d2a/f35 0 2024-04-05T21:26:35.129 INFO:tasks.workunit.client.1.smithi188.stdout:2/492: chown d5/db/l1b 34 1 2024-04-05T21:26:35.129 INFO:tasks.workunit.client.1.smithi188.stdout:4/662: dwrite d0/d14/d15/fb [0,4194304] 0 2024-04-05T21:26:35.132 INFO:tasks.workunit.client.0.smithi121.stdout:1/144: creat d8/d10/d14/f2c x:0 0 0 2024-04-05T21:26:35.133 INFO:tasks.workunit.client.0.smithi121.stdout:1/145: chown d8/d10/d14/f23 8 1 2024-04-05T21:26:35.137 INFO:tasks.workunit.client.0.smithi121.stdout:6/104: truncate d1/f10 2984948 0 2024-04-05T21:26:35.137 INFO:tasks.workunit.client.0.smithi121.stdout:8/98: getdents d12 0 2024-04-05T21:26:35.138 INFO:tasks.workunit.client.1.smithi188.stdout:4/663: truncate d0/d2/d2a/d34/f41 1224197 0 2024-04-05T21:26:35.138 INFO:tasks.workunit.client.0.smithi121.stdout:8/99: dread - d12/d13/f1b zero size 2024-04-05T21:26:35.138 INFO:tasks.workunit.client.1.smithi188.stdout:0/294: unlink d4/d6/d13/d23/d24/l37 0 2024-04-05T21:26:35.140 INFO:tasks.workunit.client.1.smithi188.stdout:7/569: write d3/d4/d1d/d74/d73/f7a [1550803,103295] 0 2024-04-05T21:26:35.140 INFO:tasks.workunit.client.0.smithi121.stdout:5/137: creat dd/de/d19/d13/f2d x:0 0 0 2024-04-05T21:26:35.141 INFO:tasks.workunit.client.1.smithi188.stdout:2/493: rename d5/db/dd/d36/d3c/d48/l6d to d5/d5f/d98/l9d 0 2024-04-05T21:26:35.141 INFO:tasks.workunit.client.0.smithi121.stdout:4/146: truncate d5/d11/f17 1642656 0 2024-04-05T21:26:35.142 INFO:tasks.workunit.client.1.smithi188.stdout:4/664: creat d0/d14/d15/d53/d82/d8f/d95/feb x:0 0 0 2024-04-05T21:26:35.143 INFO:tasks.workunit.client.0.smithi121.stdout:0/194: mkdir d7/d1c/d1d/d29/d3f 0 2024-04-05T21:26:35.143 INFO:tasks.workunit.client.1.smithi188.stdout:0/295: mkdir d4/d8/db/d21/d4a/d73 0 2024-04-05T21:26:35.143 INFO:tasks.workunit.client.0.smithi121.stdout:1/146: rmdir d8/d10/d14/d1a/d1b/d24 39 2024-04-05T21:26:35.144 INFO:tasks.workunit.client.0.smithi121.stdout:1/147: fsync f4 0 2024-04-05T21:26:35.145 INFO:tasks.workunit.client.1.smithi188.stdout:4/665: mknod d0/d14/d15/d78/db1/daa/cec 0 2024-04-05T21:26:35.146 INFO:tasks.workunit.client.0.smithi121.stdout:6/105: creat d1/d2/d3/d5/f1d x:0 0 0 2024-04-05T21:26:35.146 INFO:tasks.workunit.client.1.smithi188.stdout:0/296: creat d4/d8/db/d21/d3c/f74 x:0 0 0 2024-04-05T21:26:35.147 INFO:tasks.workunit.client.1.smithi188.stdout:4/666: creat d0/d2/d52/fed x:0 0 0 2024-04-05T21:26:35.147 INFO:tasks.workunit.client.0.smithi121.stdout:1/148: dread d8/d10/d14/d27/f29 [0,4194304] 0 2024-04-05T21:26:35.148 INFO:tasks.workunit.client.1.smithi188.stdout:4/667: chown d0/d14/l28 1 1 2024-04-05T21:26:35.148 INFO:tasks.workunit.client.0.smithi121.stdout:8/100: rename ce to d12/d13/c22 0 2024-04-05T21:26:35.150 INFO:tasks.workunit.client.0.smithi121.stdout:0/195: unlink d7/c19 0 2024-04-05T21:26:35.150 INFO:tasks.workunit.client.0.smithi121.stdout:5/138: truncate dd/de/d19/d13/f16 1887561 0 2024-04-05T21:26:35.152 INFO:tasks.workunit.client.0.smithi121.stdout:6/106: readlink d1/l1a 0 2024-04-05T21:26:35.154 INFO:tasks.workunit.client.0.smithi121.stdout:1/149: mkdir d8/d10/d14/d27/d2d 0 2024-04-05T21:26:35.158 INFO:tasks.workunit.client.1.smithi188.stdout:4/668: truncate d0/d2/d4f/f67 2760536 0 2024-04-05T21:26:35.158 INFO:tasks.workunit.client.0.smithi121.stdout:5/139: creat dd/de/f2e x:0 0 0 2024-04-05T21:26:35.161 INFO:tasks.workunit.client.1.smithi188.stdout:4/669: mknod d0/d2/d17/cee 0 2024-04-05T21:26:35.162 INFO:tasks.workunit.client.1.smithi188.stdout:0/297: dwrite d4/d8/d10/f1b [0,4194304] 0 2024-04-05T21:26:35.166 INFO:tasks.workunit.client.0.smithi121.stdout:7/124: write d0/d9/f12 [2519936,85724] 0 2024-04-05T21:26:35.166 INFO:tasks.workunit.client.0.smithi121.stdout:0/196: creat d7/d1c/d1d/d29/d2a/d39/f40 x:0 0 0 2024-04-05T21:26:35.172 INFO:tasks.workunit.client.0.smithi121.stdout:7/125: write d0/d23/f2e [633567,30482] 0 2024-04-05T21:26:35.172 INFO:tasks.workunit.client.0.smithi121.stdout:0/197: write d7/f3a [645474,129040] 0 2024-04-05T21:26:35.173 INFO:tasks.workunit.client.0.smithi121.stdout:1/150: creat d8/d10/d14/d1a/d1b/d24/d2b/f2e x:0 0 0 2024-04-05T21:26:35.177 INFO:tasks.workunit.client.1.smithi188.stdout:7/570: dread d3/d4/db/f60 [0,4194304] 0 2024-04-05T21:26:35.178 INFO:tasks.workunit.client.0.smithi121.stdout:4/147: write d5/d11/d28/f13 [2237135,48236] 0 2024-04-05T21:26:35.178 INFO:tasks.workunit.client.0.smithi121.stdout:8/101: write d12/d13/d15/f1f [8219881,100576] 0 2024-04-05T21:26:35.180 INFO:tasks.workunit.client.0.smithi121.stdout:8/102: dread - d12/d13/d15/f19 zero size 2024-04-05T21:26:35.181 INFO:tasks.workunit.client.0.smithi121.stdout:8/103: chown d12/d13/d15/d1d 3 1 2024-04-05T21:26:35.181 INFO:tasks.workunit.client.0.smithi121.stdout:0/198: link d7/d1c/f23 d7/d1c/d1d/d29/d2a/d39/f41 0 2024-04-05T21:26:35.182 INFO:tasks.workunit.client.0.smithi121.stdout:4/148: readlink d5/d11/d26/d27/d29/d2d/l33 0 2024-04-05T21:26:35.184 INFO:tasks.workunit.client.1.smithi188.stdout:7/571: dread d3/d4/db/d10/f33 [0,4194304] 0 2024-04-05T21:26:35.185 INFO:tasks.workunit.client.0.smithi121.stdout:8/104: stat l5 0 2024-04-05T21:26:35.186 INFO:tasks.workunit.client.0.smithi121.stdout:8/105: read f3 [680633,90969] 0 2024-04-05T21:26:35.187 INFO:tasks.workunit.client.0.smithi121.stdout:5/140: write dd/f17 [1089416,41531] 0 2024-04-05T21:26:35.189 INFO:tasks.workunit.client.0.smithi121.stdout:7/126: truncate d0/d9/fe 6787591 0 2024-04-05T21:26:35.189 INFO:tasks.workunit.client.1.smithi188.stdout:7/572: dread d3/d4/db/d2e/f6b [0,4194304] 0 2024-04-05T21:26:35.190 INFO:tasks.workunit.client.1.smithi188.stdout:8/335: sync 2024-04-05T21:26:35.191 INFO:tasks.workunit.client.1.smithi188.stdout:8/336: fdatasync d5/d21/d40/f55 0 2024-04-05T21:26:35.191 INFO:tasks.workunit.client.0.smithi121.stdout:5/141: write f4 [4339707,58594] 0 2024-04-05T21:26:35.192 INFO:tasks.workunit.client.1.smithi188.stdout:7/573: mkdir d3/d4/db/d2e/d3b/db3 0 2024-04-05T21:26:35.198 INFO:tasks.workunit.client.1.smithi188.stdout:4/670: dread d0/d14/d15/d78/fbc [0,4194304] 0 2024-04-05T21:26:35.198 INFO:tasks.workunit.client.0.smithi121.stdout:6/107: dwrite d1/f1c [0,4194304] 0 2024-04-05T21:26:35.199 INFO:tasks.workunit.client.0.smithi121.stdout:0/199: dwrite d7/d1c/d1d/f3b [0,4194304] 0 2024-04-05T21:26:35.200 INFO:tasks.workunit.client.0.smithi121.stdout:5/142: symlink dd/de/d19/d1c/l2f 0 2024-04-05T21:26:35.202 INFO:tasks.workunit.client.1.smithi188.stdout:8/337: dwrite d5/d21/d2a/f31 [4194304,4194304] 0 2024-04-05T21:26:35.205 INFO:tasks.workunit.client.1.smithi188.stdout:4/671: creat d0/d14/d15/d78/fef x:0 0 0 2024-04-05T21:26:35.206 INFO:tasks.workunit.client.0.smithi121.stdout:6/108: mkdir d1/d2/d1b/d1e 0 2024-04-05T21:26:35.207 INFO:tasks.workunit.client.0.smithi121.stdout:0/200: chown d7/d1c/f25 32245149 1 2024-04-05T21:26:35.207 INFO:tasks.workunit.client.1.smithi188.stdout:7/574: dwrite d3/d4/db/d10/f14 [0,4194304] 0 2024-04-05T21:26:35.209 INFO:tasks.workunit.client.1.smithi188.stdout:8/338: link d5/f2f d5/d21/d40/f7b 0 2024-04-05T21:26:35.210 INFO:tasks.workunit.client.1.smithi188.stdout:8/339: fdatasync d5/d21/f6a 0 2024-04-05T21:26:35.213 INFO:tasks.workunit.client.1.smithi188.stdout:7/575: fsync d3/d2c/d5e/d8e/fac 0 2024-04-05T21:26:35.214 INFO:tasks.workunit.client.1.smithi188.stdout:4/672: creat d0/d14/d15/d53/ff0 x:0 0 0 2024-04-05T21:26:35.214 INFO:tasks.workunit.client.0.smithi121.stdout:5/143: fdatasync dd/de/d19/d13/f25 0 2024-04-05T21:26:35.214 INFO:tasks.workunit.client.0.smithi121.stdout:4/149: dwrite d5/d11/f17 [0,4194304] 0 2024-04-05T21:26:35.215 INFO:tasks.workunit.client.0.smithi121.stdout:6/109: mkdir d1/d2/d3/d5/d1f 0 2024-04-05T21:26:35.215 INFO:tasks.workunit.client.0.smithi121.stdout:0/201: read d7/dc/d10/f1b [604855,109224] 0 2024-04-05T21:26:35.215 INFO:tasks.workunit.client.1.smithi188.stdout:8/340: write d5/f15 [1240920,5477] 0 2024-04-05T21:26:35.215 INFO:tasks.workunit.client.0.smithi121.stdout:1/151: dwrite d8/f1d [0,4194304] 0 2024-04-05T21:26:35.216 INFO:tasks.workunit.client.1.smithi188.stdout:7/576: rmdir d3/d4/d1d/da3 39 2024-04-05T21:26:35.218 INFO:tasks.workunit.client.0.smithi121.stdout:0/202: fsync d7/dc/d10/f20 0 2024-04-05T21:26:35.219 INFO:tasks.workunit.client.0.smithi121.stdout:2/144: sync 2024-04-05T21:26:35.219 INFO:tasks.workunit.client.0.smithi121.stdout:3/64: sync 2024-04-05T21:26:35.220 INFO:tasks.workunit.client.0.smithi121.stdout:8/106: dwrite d12/d13/d15/f21 [0,4194304] 0 2024-04-05T21:26:35.221 INFO:tasks.workunit.client.1.smithi188.stdout:4/673: rename d0/d14/d15/d78/db1/f83 to d0/d14/d15/d53/d82/dd5/ff1 0 2024-04-05T21:26:35.221 INFO:tasks.workunit.client.0.smithi121.stdout:0/203: stat d7/dc/f2d 0 2024-04-05T21:26:35.223 INFO:tasks.workunit.client.1.smithi188.stdout:7/577: dread d3/d4/d1d/f34 [0,4194304] 0 2024-04-05T21:26:35.224 INFO:tasks.workunit.client.1.smithi188.stdout:8/341: rename d5/dc/f36 to d5/d21/d50/d63/d69/d70/f7c 0 2024-04-05T21:26:35.225 INFO:tasks.workunit.client.1.smithi188.stdout:8/342: truncate d5/dc/f3b 310235 0 2024-04-05T21:26:35.226 INFO:tasks.workunit.client.1.smithi188.stdout:4/674: creat d0/d2/d52/dcc/ff2 x:0 0 0 2024-04-05T21:26:35.227 INFO:tasks.workunit.client.1.smithi188.stdout:7/578: write d3/d4/db/d2e/f6b [92053,39079] 0 2024-04-05T21:26:35.228 INFO:tasks.workunit.client.0.smithi121.stdout:4/150: creat d5/d11/f35 x:0 0 0 2024-04-05T21:26:35.231 INFO:tasks.workunit.client.1.smithi188.stdout:7/579: fdatasync d3/d4/d1d/f34 0 2024-04-05T21:26:35.232 INFO:tasks.workunit.client.1.smithi188.stdout:8/343: getdents d5/d21/d2a 0 2024-04-05T21:26:35.236 INFO:tasks.workunit.client.1.smithi188.stdout:8/344: write d5/d21/d40/d43/d44/d59/f74 [205690,17847] 0 2024-04-05T21:26:35.236 INFO:tasks.workunit.client.0.smithi121.stdout:1/152: getdents d8/d10 0 2024-04-05T21:26:35.238 INFO:tasks.workunit.client.1.smithi188.stdout:7/580: symlink d3/d4/db/d2e/d49/lb4 0 2024-04-05T21:26:35.240 INFO:tasks.workunit.client.0.smithi121.stdout:0/204: dread d7/f3a [0,4194304] 0 2024-04-05T21:26:35.241 INFO:tasks.workunit.client.1.smithi188.stdout:7/581: write d3/d4/f25 [1053382,51764] 0 2024-04-05T21:26:35.241 INFO:tasks.workunit.client.1.smithi188.stdout:8/345: creat d5/d21/d50/d63/f7d x:0 0 0 2024-04-05T21:26:35.242 INFO:tasks.workunit.client.0.smithi121.stdout:6/110: dwrite d1/f1c [0,4194304] 0 2024-04-05T21:26:35.247 INFO:tasks.workunit.client.0.smithi121.stdout:3/65: dwrite d0/f14 [0,4194304] 0 2024-04-05T21:26:35.250 INFO:tasks.workunit.client.0.smithi121.stdout:0/205: truncate d7/d1c/f23 948856 0 2024-04-05T21:26:35.252 INFO:tasks.workunit.client.0.smithi121.stdout:3/66: fdatasync d0/f15 0 2024-04-05T21:26:35.253 INFO:tasks.workunit.client.0.smithi121.stdout:6/111: unlink d1/d2/d3/d4/ca 0 2024-04-05T21:26:35.253 INFO:tasks.workunit.client.0.smithi121.stdout:8/107: dwrite f3 [0,4194304] 0 2024-04-05T21:26:35.255 INFO:tasks.workunit.client.1.smithi188.stdout:7/582: dwrite d3/d2c/f96 [0,4194304] 0 2024-04-05T21:26:35.257 INFO:tasks.workunit.client.0.smithi121.stdout:4/151: dread d5/d11/d28/fe [0,4194304] 0 2024-04-05T21:26:35.258 INFO:tasks.workunit.client.0.smithi121.stdout:0/206: dread - d7/d1c/d1d/d29/f34 zero size 2024-04-05T21:26:35.262 INFO:tasks.workunit.client.0.smithi121.stdout:1/153: dwrite d8/ff [0,4194304] 0 2024-04-05T21:26:35.266 INFO:tasks.workunit.client.1.smithi188.stdout:7/583: getdents d3/d4/db/d10/d3c 0 2024-04-05T21:26:35.266 INFO:tasks.workunit.client.1.smithi188.stdout:4/675: dread d0/d14/f80 [0,4194304] 0 2024-04-05T21:26:35.266 INFO:tasks.workunit.client.1.smithi188.stdout:4/676: write d0/d14/d15/fb [214418,13993] 0 2024-04-05T21:26:35.268 INFO:tasks.workunit.client.0.smithi121.stdout:6/112: mkdir d1/d2/d20 0 2024-04-05T21:26:35.269 INFO:tasks.workunit.client.0.smithi121.stdout:7/127: dread d0/d9/fe [0,4194304] 0 2024-04-05T21:26:35.270 INFO:tasks.workunit.client.0.smithi121.stdout:8/108: rename d12/l1a to d12/d13/d15/l23 0 2024-04-05T21:26:35.271 INFO:tasks.workunit.client.0.smithi121.stdout:7/128: fsync d0/f15 0 2024-04-05T21:26:35.273 INFO:tasks.workunit.client.0.smithi121.stdout:2/145: write d2/d9/d17/f19 [4379194,23428] 0 2024-04-05T21:26:35.274 INFO:tasks.workunit.client.0.smithi121.stdout:0/207: creat d7/dc/d10/f42 x:0 0 0 2024-04-05T21:26:35.275 INFO:tasks.workunit.client.0.smithi121.stdout:1/154: rmdir d8/d10/d14 39 2024-04-05T21:26:35.276 INFO:tasks.workunit.client.0.smithi121.stdout:4/152: dwrite d5/d11/f1e [0,4194304] 0 2024-04-05T21:26:35.277 INFO:tasks.workunit.client.1.smithi188.stdout:8/346: dwrite d5/d21/d40/f7b [0,4194304] 0 2024-04-05T21:26:35.278 INFO:tasks.workunit.client.0.smithi121.stdout:9/155: sync 2024-04-05T21:26:35.281 INFO:tasks.workunit.client.1.smithi188.stdout:7/584: truncate d3/d4/db/f6e 4691458 0 2024-04-05T21:26:35.282 INFO:tasks.workunit.client.1.smithi188.stdout:4/677: mknod d0/d14/d15/d1b/cf3 0 2024-04-05T21:26:35.284 INFO:tasks.workunit.client.1.smithi188.stdout:8/347: mkdir d5/d21/d40/d43/d48/d7e 0 2024-04-05T21:26:35.284 INFO:tasks.workunit.client.1.smithi188.stdout:7/585: chown d3/d4/db/d55/d9d/lb2 14 1 2024-04-05T21:26:35.284 INFO:tasks.workunit.client.0.smithi121.stdout:5/144: dwrite dd/de/f11 [0,4194304] 0 2024-04-05T21:26:35.285 INFO:tasks.workunit.client.1.smithi188.stdout:4/678: truncate d0/d14/d2f/fb8 4534110 0 2024-04-05T21:26:35.286 INFO:tasks.workunit.client.0.smithi121.stdout:3/67: getdents d0/d3 0 2024-04-05T21:26:35.287 INFO:tasks.workunit.client.0.smithi121.stdout:0/208: dread - d7/d1c/d1d/d29/f37 zero size 2024-04-05T21:26:35.290 INFO:tasks.workunit.client.0.smithi121.stdout:3/68: write d0/f14 [4137122,105252] 0 2024-04-05T21:26:35.290 INFO:tasks.workunit.client.0.smithi121.stdout:4/153: chown d5/d11/f21 3 1 2024-04-05T21:26:35.291 INFO:tasks.workunit.client.0.smithi121.stdout:2/146: symlink d2/d9/d17/l2b 0 2024-04-05T21:26:35.297 INFO:tasks.workunit.client.1.smithi188.stdout:7/586: mknod d3/d4/db/d2e/d3b/d94/cb5 0 2024-04-05T21:26:35.298 INFO:tasks.workunit.client.0.smithi121.stdout:8/109: dwrite ff [0,4194304] 0 2024-04-05T21:26:35.298 INFO:tasks.workunit.client.0.smithi121.stdout:5/145: dread - dd/de/d19/d13/f2d zero size 2024-04-05T21:26:35.301 INFO:tasks.workunit.client.0.smithi121.stdout:4/154: chown d5/d11/d2c 201080 1 2024-04-05T21:26:35.301 INFO:tasks.workunit.client.0.smithi121.stdout:3/69: symlink d0/d3/dc/l1c 0 2024-04-05T21:26:35.302 INFO:tasks.workunit.client.1.smithi188.stdout:7/587: dread d3/d4/d1d/d74/d73/f7a [0,4194304] 0 2024-04-05T21:26:35.303 INFO:tasks.workunit.client.0.smithi121.stdout:7/129: dread d0/f3 [0,4194304] 0 2024-04-05T21:26:35.306 INFO:tasks.workunit.client.0.smithi121.stdout:2/147: unlink d2/d1e/c2a 0 2024-04-05T21:26:35.307 INFO:tasks.workunit.client.0.smithi121.stdout:9/156: dwrite d6/f15 [0,4194304] 0 2024-04-05T21:26:35.307 INFO:tasks.workunit.client.0.smithi121.stdout:6/113: mkdir d1/d2/d3/d4/d21 0 2024-04-05T21:26:35.308 INFO:tasks.workunit.client.0.smithi121.stdout:0/209: creat d7/d1c/d1d/d29/d2a/d2c/f43 x:0 0 0 2024-04-05T21:26:35.308 INFO:tasks.workunit.client.0.smithi121.stdout:3/70: chown d0/f15 3 1 2024-04-05T21:26:35.309 INFO:tasks.workunit.client.0.smithi121.stdout:1/155: mknod d8/d10/d14/d1a/d1b/d24/c2f 0 2024-04-05T21:26:35.309 INFO:tasks.workunit.client.1.smithi188.stdout:7/588: link d3/d4/db/d2e/c6d d3/d4/d1d/cb6 0 2024-04-05T21:26:35.310 INFO:tasks.workunit.client.1.smithi188.stdout:7/589: fsync d3/f1e 0 2024-04-05T21:26:35.316 INFO:tasks.workunit.client.0.smithi121.stdout:0/210: write d7/f18 [3149963,53504] 0 2024-04-05T21:26:35.318 INFO:tasks.workunit.client.1.smithi188.stdout:8/348: truncate d5/d21/d2a/f31 7280660 0 2024-04-05T21:26:35.318 INFO:tasks.workunit.client.1.smithi188.stdout:7/590: readlink d3/d4/db/d2e/la9 0 2024-04-05T21:26:35.319 INFO:tasks.workunit.client.1.smithi188.stdout:4/679: dread d0/d2/d4f/f67 [0,4194304] 0 2024-04-05T21:26:35.320 INFO:tasks.workunit.client.1.smithi188.stdout:4/680: dread - d0/d2/d52/fe4 zero size 2024-04-05T21:26:35.321 INFO:tasks.workunit.client.0.smithi121.stdout:3/71: chown d0/f7 341 1 2024-04-05T21:26:35.321 INFO:tasks.workunit.client.1.smithi188.stdout:8/349: write d5/d21/d40/f73 [57397,117905] 0 2024-04-05T21:26:35.322 INFO:tasks.workunit.client.0.smithi121.stdout:3/72: chown d0 4230573 1 2024-04-05T21:26:35.323 INFO:tasks.workunit.client.0.smithi121.stdout:1/156: write d8/d10/d14/d1a/f20 [333792,87929] 0 2024-04-05T21:26:35.323 INFO:tasks.workunit.client.0.smithi121.stdout:4/155: getdents d5/d11/d26/d27/d29/d2e 0 2024-04-05T21:26:35.324 INFO:tasks.workunit.client.1.smithi188.stdout:7/591: dread d3/f68 [0,4194304] 0 2024-04-05T21:26:35.326 INFO:tasks.workunit.client.0.smithi121.stdout:3/73: dread - d0/d3/de/f17 zero size 2024-04-05T21:26:35.326 INFO:tasks.workunit.client.1.smithi188.stdout:8/350: mknod d5/d21/d40/d43/d44/c7f 0 2024-04-05T21:26:35.328 INFO:tasks.workunit.client.1.smithi188.stdout:8/351: fsync d5/d21/d40/d43/f4c 0 2024-04-05T21:26:35.328 INFO:tasks.workunit.client.1.smithi188.stdout:7/592: mkdir d3/d4/db/d2e/d8c/db7 0 2024-04-05T21:26:35.330 INFO:tasks.workunit.client.1.smithi188.stdout:4/681: dwrite d0/f64 [0,4194304] 0 2024-04-05T21:26:35.330 INFO:tasks.workunit.client.0.smithi121.stdout:1/157: readlink d8/d10/l13 0 2024-04-05T21:26:35.331 INFO:tasks.workunit.client.0.smithi121.stdout:2/148: dread d2/d9/d12/f15 [0,4194304] 0 2024-04-05T21:26:35.331 INFO:tasks.workunit.client.0.smithi121.stdout:6/114: dwrite d1/d2/f13 [0,4194304] 0 2024-04-05T21:26:35.331 INFO:tasks.workunit.client.1.smithi188.stdout:7/593: chown d3/d4/c52 96098 1 2024-04-05T21:26:35.331 INFO:tasks.workunit.client.0.smithi121.stdout:0/211: rmdir d7 39 2024-04-05T21:26:35.334 INFO:tasks.workunit.client.1.smithi188.stdout:8/352: truncate d5/f20 1201614 0 2024-04-05T21:26:35.334 INFO:tasks.workunit.client.0.smithi121.stdout:4/156: creat d5/d11/d2c/f36 x:0 0 0 2024-04-05T21:26:35.335 INFO:tasks.workunit.client.1.smithi188.stdout:7/594: write d3/d4/d1d/f76 [85726,115403] 0 2024-04-05T21:26:35.339 INFO:tasks.workunit.client.0.smithi121.stdout:5/146: truncate dd/f17 709502 0 2024-04-05T21:26:35.339 INFO:tasks.workunit.client.0.smithi121.stdout:7/130: dwrite d0/d2d/f2f [0,4194304] 0 2024-04-05T21:26:35.339 INFO:tasks.workunit.client.0.smithi121.stdout:3/74: truncate d0/d3/f13 2466381 0 2024-04-05T21:26:35.339 INFO:tasks.workunit.client.0.smithi121.stdout:4/157: rmdir d5/d11/d26 39 2024-04-05T21:26:35.341 INFO:tasks.workunit.client.0.smithi121.stdout:6/115: mknod d1/d2/dc/c22 0 2024-04-05T21:26:35.343 INFO:tasks.workunit.client.0.smithi121.stdout:4/158: write d5/d11/f12 [1136974,7180] 0 2024-04-05T21:26:35.343 INFO:tasks.workunit.client.0.smithi121.stdout:7/131: symlink d0/d9/d2b/l30 0 2024-04-05T21:26:35.344 INFO:tasks.workunit.client.1.smithi188.stdout:8/353: getdents d5/dc 0 2024-04-05T21:26:35.345 INFO:tasks.workunit.client.1.smithi188.stdout:4/682: dwrite d0/d14/d15/d42/fa0 [0,4194304] 0 2024-04-05T21:26:35.347 INFO:tasks.workunit.client.1.smithi188.stdout:4/683: dread - d0/d14/d15/d53/d82/f9e zero size 2024-04-05T21:26:35.350 INFO:tasks.workunit.client.1.smithi188.stdout:4/684: write d0/d14/d15/d53/f74 [2778097,68243] 0 2024-04-05T21:26:35.351 INFO:tasks.workunit.client.0.smithi121.stdout:1/158: unlink d8/d10/d14/d1a/d1b/d24/d2b/f2e 0 2024-04-05T21:26:35.351 INFO:tasks.workunit.client.0.smithi121.stdout:2/149: dwrite d2/d9/d1c/f22 [0,4194304] 0 2024-04-05T21:26:35.351 INFO:tasks.workunit.client.0.smithi121.stdout:6/116: symlink d1/d2/d3/d4/l23 0 2024-04-05T21:26:35.352 INFO:tasks.workunit.client.0.smithi121.stdout:1/159: chown d8/d10 6350 1 2024-04-05T21:26:35.353 INFO:tasks.workunit.client.0.smithi121.stdout:0/212: truncate d7/d1c/d1d/f26 405332 0 2024-04-05T21:26:35.357 INFO:tasks.workunit.client.0.smithi121.stdout:6/117: fsync d1/d2/d3/d5/fd 0 2024-04-05T21:26:35.359 INFO:tasks.workunit.client.0.smithi121.stdout:0/213: dread - d7/d1c/d1d/d29/d2a/f35 zero size 2024-04-05T21:26:35.360 INFO:tasks.workunit.client.1.smithi188.stdout:4/685: chown d0/d14/d15/d1b/c72 333417 1 2024-04-05T21:26:35.360 INFO:tasks.workunit.client.0.smithi121.stdout:2/150: truncate d2/d9/d1c/f29 95990 0 2024-04-05T21:26:35.361 INFO:tasks.workunit.client.0.smithi121.stdout:1/160: rename d8/f11 to d8/d10/d14/d27/f30 0 2024-04-05T21:26:35.363 INFO:tasks.workunit.client.0.smithi121.stdout:0/214: mknod d7/d1c/d1d/c44 0 2024-04-05T21:26:35.363 INFO:tasks.workunit.client.0.smithi121.stdout:4/159: dread d5/d11/f12 [4194304,4194304] 0 2024-04-05T21:26:35.364 INFO:tasks.workunit.client.1.smithi188.stdout:8/354: dread d5/f12 [0,4194304] 0 2024-04-05T21:26:35.364 INFO:tasks.workunit.client.0.smithi121.stdout:6/118: rename d1/d2/dc/c11 to d1/c24 0 2024-04-05T21:26:35.365 INFO:tasks.workunit.client.0.smithi121.stdout:1/161: creat d8/d10/d14/f31 x:0 0 0 2024-04-05T21:26:35.367 INFO:tasks.workunit.client.1.smithi188.stdout:4/686: mkdir d0/d14/d15/d53/d82/dd5/df4 0 2024-04-05T21:26:35.367 INFO:tasks.workunit.client.1.smithi188.stdout:7/595: dwrite d3/d4/f2d [0,4194304] 0 2024-04-05T21:26:35.368 INFO:tasks.workunit.client.0.smithi121.stdout:3/75: read d0/d3/f13 [603685,119490] 0 2024-04-05T21:26:35.368 INFO:tasks.workunit.client.0.smithi121.stdout:6/119: mknod d1/c25 0 2024-04-05T21:26:35.369 INFO:tasks.workunit.client.1.smithi188.stdout:8/355: mkdir d5/d21/d80 0 2024-04-05T21:26:35.370 INFO:tasks.workunit.client.1.smithi188.stdout:8/356: read - d5/dc/f78 zero size 2024-04-05T21:26:35.370 INFO:tasks.workunit.client.1.smithi188.stdout:4/687: symlink d0/d2/d2a/d34/d6b/lf5 0 2024-04-05T21:26:35.371 INFO:tasks.workunit.client.0.smithi121.stdout:3/76: chown d0/l9 8022288 1 2024-04-05T21:26:35.371 INFO:tasks.workunit.client.1.smithi188.stdout:4/688: chown d0 27876 1 2024-04-05T21:26:35.372 INFO:tasks.workunit.client.1.smithi188.stdout:7/596: symlink d3/d4/d1d/lb8 0 2024-04-05T21:26:35.373 INFO:tasks.workunit.client.0.smithi121.stdout:4/160: mknod d5/d11/c37 0 2024-04-05T21:26:35.374 INFO:tasks.workunit.client.0.smithi121.stdout:1/162: dwrite d8/d10/d14/f1e [0,4194304] 0 2024-04-05T21:26:35.375 INFO:tasks.workunit.client.1.smithi188.stdout:8/357: rmdir d5/d21/d50/d63 39 2024-04-05T21:26:35.377 INFO:tasks.workunit.client.0.smithi121.stdout:3/77: truncate d0/d3/fa 582447 0 2024-04-05T21:26:35.377 INFO:tasks.workunit.client.0.smithi121.stdout:0/215: getdents d7 0 2024-04-05T21:26:35.377 INFO:tasks.workunit.client.1.smithi188.stdout:8/358: creat d5/d21/d40/d43/f81 x:0 0 0 2024-04-05T21:26:35.378 INFO:tasks.workunit.client.0.smithi121.stdout:6/120: creat d1/d2/d3/d4/de/f26 x:0 0 0 2024-04-05T21:26:35.378 INFO:tasks.workunit.client.1.smithi188.stdout:8/359: chown d5/d21/l53 150 1 2024-04-05T21:26:35.380 INFO:tasks.workunit.client.0.smithi121.stdout:1/163: chown d8/d10/d14/d1a/d1b/d24/c2f 2749060 1 2024-04-05T21:26:35.380 INFO:tasks.workunit.client.0.smithi121.stdout:1/164: fsync f4 0 2024-04-05T21:26:35.381 INFO:tasks.workunit.client.1.smithi188.stdout:8/360: mknod d5/d21/d40/c82 0 2024-04-05T21:26:35.381 INFO:tasks.workunit.client.0.smithi121.stdout:4/161: creat d5/d11/d20/f38 x:0 0 0 2024-04-05T21:26:35.382 INFO:tasks.workunit.client.1.smithi188.stdout:4/689: dread d0/d2/d17/d3a/f54 [0,4194304] 0 2024-04-05T21:26:35.382 INFO:tasks.workunit.client.1.smithi188.stdout:8/361: read d5/dc/f2c [11702,129108] 0 2024-04-05T21:26:35.386 INFO:tasks.workunit.client.0.smithi121.stdout:1/165: creat d8/d10/d14/d1a/f32 x:0 0 0 2024-04-05T21:26:35.387 INFO:tasks.workunit.client.1.smithi188.stdout:4/690: mkdir d0/d14/df6 0 2024-04-05T21:26:35.388 INFO:tasks.workunit.client.1.smithi188.stdout:4/691: fsync d0/d2/d17/d48/f57 0 2024-04-05T21:26:35.388 INFO:tasks.workunit.client.0.smithi121.stdout:4/162: fsync d5/fd 0 2024-04-05T21:26:35.388 INFO:tasks.workunit.client.0.smithi121.stdout:1/166: read d8/d10/d14/f1e [3031474,54274] 0 2024-04-05T21:26:35.388 INFO:tasks.workunit.client.1.smithi188.stdout:8/362: rename d5/d21/d40/d43/f4c to d5/d21/d40/d43/f83 0 2024-04-05T21:26:35.389 INFO:tasks.workunit.client.0.smithi121.stdout:4/163: chown d5/d11/f35 10496031 1 2024-04-05T21:26:35.390 INFO:tasks.workunit.client.1.smithi188.stdout:4/692: chown d0/d2/d52/fed 1622562314 1 2024-04-05T21:26:35.390 INFO:tasks.workunit.client.0.smithi121.stdout:5/147: write dd/f17 [854086,129287] 0 2024-04-05T21:26:35.391 INFO:tasks.workunit.client.0.smithi121.stdout:5/148: chown dd/c10 0 1 2024-04-05T21:26:35.391 INFO:tasks.workunit.client.1.smithi188.stdout:8/363: stat d5/d21/d40/d43/d44/d5b/l5d 0 2024-04-05T21:26:35.391 INFO:tasks.workunit.client.0.smithi121.stdout:6/121: creat d1/d2/d3/d5/d1f/f27 x:0 0 0 2024-04-05T21:26:35.392 INFO:tasks.workunit.client.1.smithi188.stdout:4/693: write d0/d2/fc8 [487774,75165] 0 2024-04-05T21:26:35.394 INFO:tasks.workunit.client.0.smithi121.stdout:3/78: dread d0/f7 [0,4194304] 0 2024-04-05T21:26:35.394 INFO:tasks.workunit.client.0.smithi121.stdout:6/122: stat d1/d2/d3/d5 0 2024-04-05T21:26:35.396 INFO:tasks.workunit.client.1.smithi188.stdout:8/364: creat d5/d6e/f84 x:0 0 0 2024-04-05T21:26:35.396 INFO:tasks.workunit.client.0.smithi121.stdout:1/167: read d8/d10/f17 [6435311,118526] 0 2024-04-05T21:26:35.397 INFO:tasks.workunit.client.1.smithi188.stdout:4/694: creat d0/d14/d15/dd4/ddf/ff7 x:0 0 0 2024-04-05T21:26:35.397 INFO:tasks.workunit.client.1.smithi188.stdout:8/365: chown d5/l56 27 1 2024-04-05T21:26:35.397 INFO:tasks.workunit.client.0.smithi121.stdout:6/123: read d1/d2/f13 [2391318,62126] 0 2024-04-05T21:26:35.397 INFO:tasks.workunit.client.0.smithi121.stdout:4/164: symlink d5/d11/l39 0 2024-04-05T21:26:35.398 INFO:tasks.workunit.client.1.smithi188.stdout:7/597: dread d3/f3e [0,4194304] 0 2024-04-05T21:26:35.398 INFO:tasks.workunit.client.1.smithi188.stdout:8/366: chown d5/d21/d40/d43/f51 46861315 1 2024-04-05T21:26:35.400 INFO:tasks.workunit.client.1.smithi188.stdout:7/598: fsync d3/d4/db/d2e/d3b/f4d 0 2024-04-05T21:26:35.400 INFO:tasks.workunit.client.0.smithi121.stdout:0/216: write f2 [2010725,32874] 0 2024-04-05T21:26:35.403 INFO:tasks.workunit.client.0.smithi121.stdout:2/151: dread d2/d9/ff [0,4194304] 0 2024-04-05T21:26:35.404 INFO:tasks.workunit.client.1.smithi188.stdout:7/599: read d3/d4/d1d/d74/d73/f7a [787206,96639] 0 2024-04-05T21:26:35.404 INFO:tasks.workunit.client.0.smithi121.stdout:5/149: dread dd/de/d19/d13/f16 [0,4194304] 0 2024-04-05T21:26:35.404 INFO:tasks.workunit.client.1.smithi188.stdout:7/600: stat d3/d4/db/d2e/d49/c53 0 2024-04-05T21:26:35.405 INFO:tasks.workunit.client.1.smithi188.stdout:4/695: link d0/d2/d17/l8c d0/d14/d15/d1b/lf8 0 2024-04-05T21:26:35.405 INFO:tasks.workunit.client.0.smithi121.stdout:5/150: fdatasync dd/f17 0 2024-04-05T21:26:35.406 INFO:tasks.workunit.client.0.smithi121.stdout:2/152: fsync d2/d9/d17/f19 0 2024-04-05T21:26:35.407 INFO:tasks.workunit.client.1.smithi188.stdout:4/696: truncate d0/d14/d15/d78/fef 804685 0 2024-04-05T21:26:35.408 INFO:tasks.workunit.client.0.smithi121.stdout:6/124: creat d1/d2/d1b/f28 x:0 0 0 2024-04-05T21:26:35.408 INFO:tasks.workunit.client.0.smithi121.stdout:1/168: symlink d8/d10/d14/d1a/d1b/l33 0 2024-04-05T21:26:35.409 INFO:tasks.workunit.client.1.smithi188.stdout:8/367: getdents d5/d21/d40/d43/d44/d59 0 2024-04-05T21:26:35.411 INFO:tasks.workunit.client.1.smithi188.stdout:8/368: symlink d5/d21/d40/l85 0 2024-04-05T21:26:35.411 INFO:tasks.workunit.client.0.smithi121.stdout:0/217: dwrite d7/dc/f15 [0,4194304] 0 2024-04-05T21:26:35.412 INFO:tasks.workunit.client.0.smithi121.stdout:2/153: rename d2/d9/d17/d25 to d2/d9/d17/d25/d2c 22 2024-04-05T21:26:35.412 INFO:tasks.workunit.client.0.smithi121.stdout:4/165: creat d5/f3a x:0 0 0 2024-04-05T21:26:35.412 INFO:tasks.workunit.client.0.smithi121.stdout:6/125: chown d1/d2/d3/d5/fd 1051595 1 2024-04-05T21:26:35.413 INFO:tasks.workunit.client.0.smithi121.stdout:5/151: symlink dd/de/d19/d1c/l30 0 2024-04-05T21:26:35.413 INFO:tasks.workunit.client.1.smithi188.stdout:7/601: rename d3/d4/db/d2e/d37/d65/f39 to d3/d4/fb9 0 2024-04-05T21:26:35.414 INFO:tasks.workunit.client.1.smithi188.stdout:8/369: mkdir d5/d6e/d86 0 2024-04-05T21:26:35.416 INFO:tasks.workunit.client.0.smithi121.stdout:0/218: dread d7/f3a [0,4194304] 0 2024-04-05T21:26:35.416 INFO:tasks.workunit.client.0.smithi121.stdout:5/152: chown dd/l2c 117912367 1 2024-04-05T21:26:35.416 INFO:tasks.workunit.client.1.smithi188.stdout:4/697: getdents d0/d2/d52 0 2024-04-05T21:26:35.417 INFO:tasks.workunit.client.0.smithi121.stdout:1/169: fsync d8/d10/d14/d27/f30 0 2024-04-05T21:26:35.417 INFO:tasks.workunit.client.1.smithi188.stdout:8/370: creat d5/d21/d40/d43/f87 x:0 0 0 2024-04-05T21:26:35.418 INFO:tasks.workunit.client.0.smithi121.stdout:3/79: dwrite d0/d3/f13 [0,4194304] 0 2024-04-05T21:26:35.419 INFO:tasks.workunit.client.0.smithi121.stdout:0/219: mkdir d7/d1c/d1d/d45 0 2024-04-05T21:26:35.420 INFO:tasks.workunit.client.0.smithi121.stdout:5/153: rename dd/de/d19/d13 to dd/de/d19/d13/d31 22 2024-04-05T21:26:35.420 INFO:tasks.workunit.client.0.smithi121.stdout:3/80: stat d0/d3/de/c19 0 2024-04-05T21:26:35.422 INFO:tasks.workunit.client.0.smithi121.stdout:1/170: fdatasync d8/d10/d14/d27/f29 0 2024-04-05T21:26:35.423 INFO:tasks.workunit.client.1.smithi188.stdout:7/602: link d3/d4/db/d10/f33 d3/d4/db/d2e/d37/fba 0 2024-04-05T21:26:35.424 INFO:tasks.workunit.client.0.smithi121.stdout:1/171: write d8/f1c [122770,95150] 0 2024-04-05T21:26:35.424 INFO:tasks.workunit.client.1.smithi188.stdout:4/698: dread d0/d14/f6e [0,4194304] 0 2024-04-05T21:26:35.424 INFO:tasks.workunit.client.0.smithi121.stdout:3/81: unlink d0/f5 0 2024-04-05T21:26:35.425 INFO:tasks.workunit.client.1.smithi188.stdout:7/603: chown d3/d4/db/d2e/d49/c53 6 1 2024-04-05T21:26:35.425 INFO:tasks.workunit.client.1.smithi188.stdout:4/699: stat d0/d14/d15/d78/fe2 0 2024-04-05T21:26:35.426 INFO:tasks.workunit.client.0.smithi121.stdout:0/220: mkdir d7/d1c/d1d/d45/d46 0 2024-04-05T21:26:35.427 INFO:tasks.workunit.client.0.smithi121.stdout:6/126: fsync d1/d2/d1b/f28 0 2024-04-05T21:26:35.428 INFO:tasks.workunit.client.0.smithi121.stdout:0/221: fdatasync d7/dc/d10/f42 0 2024-04-05T21:26:35.430 INFO:tasks.workunit.client.0.smithi121.stdout:1/172: write d8/ff [1294963,122362] 0 2024-04-05T21:26:35.431 INFO:tasks.workunit.client.1.smithi188.stdout:8/371: dread d5/f15 [0,4194304] 0 2024-04-05T21:26:35.431 INFO:tasks.workunit.client.0.smithi121.stdout:1/173: chown d8/fb 199 1 2024-04-05T21:26:35.432 INFO:tasks.workunit.client.0.smithi121.stdout:5/154: dwrite dd/de/d19/d13/f2d [0,4194304] 0 2024-04-05T21:26:35.432 INFO:tasks.workunit.client.1.smithi188.stdout:7/604: creat d3/fbb x:0 0 0 2024-04-05T21:26:35.433 INFO:tasks.workunit.client.1.smithi188.stdout:8/372: unlink d5/dc/f19 0 2024-04-05T21:26:35.435 INFO:tasks.workunit.client.0.smithi121.stdout:4/166: getdents d5 0 2024-04-05T21:26:35.435 INFO:tasks.workunit.client.1.smithi188.stdout:7/605: creat d3/d4/db/d2e/d8c/db7/fbc x:0 0 0 2024-04-05T21:26:35.437 INFO:tasks.workunit.client.1.smithi188.stdout:8/373: creat d5/d21/d2a/d71/f88 x:0 0 0 2024-04-05T21:26:35.437 INFO:tasks.workunit.client.1.smithi188.stdout:7/606: chown d3/d4/db/d2e/f35 26809 1 2024-04-05T21:26:35.438 INFO:tasks.workunit.client.1.smithi188.stdout:8/374: write d5/f24 [4976953,91744] 0 2024-04-05T21:26:35.439 INFO:tasks.workunit.client.0.smithi121.stdout:6/127: stat d1/d2/c7 0 2024-04-05T21:26:35.439 INFO:tasks.workunit.client.0.smithi121.stdout:0/222: write d7/f3a [974366,78005] 0 2024-04-05T21:26:35.441 INFO:tasks.workunit.client.1.smithi188.stdout:7/607: chown d3/d4/db/d2e/d37/l90 63 1 2024-04-05T21:26:35.441 INFO:tasks.workunit.client.1.smithi188.stdout:4/700: truncate d0/d14/d2f/fb8 2746235 0 2024-04-05T21:26:35.441 INFO:tasks.workunit.client.0.smithi121.stdout:2/154: truncate d2/d9/fd 844270 0 2024-04-05T21:26:35.442 INFO:tasks.workunit.client.1.smithi188.stdout:8/375: mknod d5/d21/d40/d43/d44/d59/d79/c89 0 2024-04-05T21:26:35.443 INFO:tasks.workunit.client.0.smithi121.stdout:4/167: mkdir d5/d11/d28/d3b 0 2024-04-05T21:26:35.443 INFO:tasks.workunit.client.1.smithi188.stdout:4/701: write d0/d14/d15/d78/f97 [1435441,49958] 0 2024-04-05T21:26:35.444 INFO:tasks.workunit.client.1.smithi188.stdout:8/376: dread d5/dc/f2c [0,4194304] 0 2024-04-05T21:26:35.446 INFO:tasks.workunit.client.1.smithi188.stdout:8/377: write d5/d21/d40/d43/f81 [331101,1543] 0 2024-04-05T21:26:35.447 INFO:tasks.workunit.client.0.smithi121.stdout:1/174: rename d8/d10/d14/d1a/d1b/d24/d2b to d8/d10/d14/d1a/d1b/d24/d34 0 2024-04-05T21:26:35.447 INFO:tasks.workunit.client.0.smithi121.stdout:3/82: link d0/l9 d0/d3/de/l1d 0 2024-04-05T21:26:35.448 INFO:tasks.workunit.client.1.smithi188.stdout:4/702: read d0/d2/d17/fa7 [1544903,43154] 0 2024-04-05T21:26:35.450 INFO:tasks.workunit.client.0.smithi121.stdout:5/155: dwrite dd/de/d19/f28 [0,4194304] 0 2024-04-05T21:26:35.450 INFO:tasks.workunit.client.1.smithi188.stdout:7/608: rmdir d3/d4/db/d2e/d8c 39 2024-04-05T21:26:35.451 INFO:tasks.workunit.client.0.smithi121.stdout:0/223: unlink d7/dc/d10/f1b 0 2024-04-05T21:26:35.453 INFO:tasks.workunit.client.0.smithi121.stdout:4/168: creat d5/d11/d2c/f3c x:0 0 0 2024-04-05T21:26:35.453 INFO:tasks.workunit.client.0.smithi121.stdout:0/224: chown d7/dc/le 253 1 2024-04-05T21:26:35.457 INFO:tasks.workunit.client.1.smithi188.stdout:8/378: dwrite d5/d21/d2a/f39 [0,4194304] 0 2024-04-05T21:26:35.458 INFO:tasks.workunit.client.0.smithi121.stdout:5/156: rmdir dd/de/d19/d1c 39 2024-04-05T21:26:35.459 INFO:tasks.workunit.client.0.smithi121.stdout:6/128: dwrite d1/d2/d3/d4/de/f26 [0,4194304] 0 2024-04-05T21:26:35.459 INFO:tasks.workunit.client.0.smithi121.stdout:2/155: dread d2/d9/fe [0,4194304] 0 2024-04-05T21:26:35.461 INFO:tasks.workunit.client.0.smithi121.stdout:4/169: symlink d5/d11/d20/l3d 0 2024-04-05T21:26:35.462 INFO:tasks.workunit.client.0.smithi121.stdout:0/225: chown d7/d1c/d1d/d29/d2a/d2c/c31 5704225 1 2024-04-05T21:26:35.464 INFO:tasks.workunit.client.0.smithi121.stdout:1/175: creat d8/d10/d14/d1a/d1b/f35 x:0 0 0 2024-04-05T21:26:35.466 INFO:tasks.workunit.client.1.smithi188.stdout:8/379: write d5/d21/d40/d43/d44/d59/f74 [735012,52438] 0 2024-04-05T21:26:35.466 INFO:tasks.workunit.client.1.smithi188.stdout:4/703: dwrite d0/d14/d15/d53/f9f [0,4194304] 0 2024-04-05T21:26:35.466 INFO:tasks.workunit.client.1.smithi188.stdout:7/609: rename d3/d4/db/d2e/d37/c42 to d3/d4/db/d2e/d3b/d94/cbd 0 2024-04-05T21:26:35.466 INFO:tasks.workunit.client.1.smithi188.stdout:4/704: dread - d0/d14/d15/d53/d82/f9e zero size 2024-04-05T21:26:35.468 INFO:tasks.workunit.client.0.smithi121.stdout:6/129: symlink d1/d2/d3/l29 0 2024-04-05T21:26:35.468 INFO:tasks.workunit.client.0.smithi121.stdout:5/157: write dd/f1f [1030527,46461] 0 2024-04-05T21:26:35.468 INFO:tasks.workunit.client.1.smithi188.stdout:7/610: read d3/d4/d1d/d74/d73/f7a [1490163,126539] 0 2024-04-05T21:26:35.469 INFO:tasks.workunit.client.1.smithi188.stdout:8/380: rename d5/d21/l4e to d5/d21/d2a/d71/l8a 0 2024-04-05T21:26:35.469 INFO:tasks.workunit.client.1.smithi188.stdout:7/611: readlink d3/d4/d1d/lb8 0 2024-04-05T21:26:35.470 INFO:tasks.workunit.client.0.smithi121.stdout:6/130: chown d1/f1c 251 1 2024-04-05T21:26:35.470 INFO:tasks.workunit.client.0.smithi121.stdout:3/83: dwrite d0/d3/f4 [0,4194304] 0 2024-04-05T21:26:35.470 INFO:tasks.workunit.client.0.smithi121.stdout:2/156: mkdir d2/d9/d17/d25/d2d 0 2024-04-05T21:26:35.471 INFO:tasks.workunit.client.1.smithi188.stdout:4/705: mkdir d0/d2/d52/df9 0 2024-04-05T21:26:35.473 INFO:tasks.workunit.client.0.smithi121.stdout:0/226: symlink d7/d1c/d1d/d29/l47 0 2024-04-05T21:26:35.473 INFO:tasks.workunit.client.0.smithi121.stdout:2/157: chown d2/f1a 52080226 1 2024-04-05T21:26:35.473 INFO:tasks.workunit.client.1.smithi188.stdout:4/706: write d0/d2/f26 [4510689,70448] 0 2024-04-05T21:26:35.475 INFO:tasks.workunit.client.1.smithi188.stdout:8/381: mkdir d5/d21/d40/d43/d44/d59/d8b 0 2024-04-05T21:26:35.480 INFO:tasks.workunit.client.0.smithi121.stdout:4/170: mknod d5/d11/d28/d3b/c3e 0 2024-04-05T21:26:35.480 INFO:tasks.workunit.client.1.smithi188.stdout:6/520: sync 2024-04-05T21:26:35.480 INFO:tasks.workunit.client.0.smithi121.stdout:2/158: read d2/d9/d12/f15 [1088209,93238] 0 2024-04-05T21:26:35.481 INFO:tasks.workunit.client.1.smithi188.stdout:7/612: mkdir d3/d4/db/d10/dbe 0 2024-04-05T21:26:35.481 INFO:tasks.workunit.client.1.smithi188.stdout:8/382: dread d5/d21/f64 [0,4194304] 0 2024-04-05T21:26:35.482 INFO:tasks.workunit.client.0.smithi121.stdout:1/176: rename d8/fc to d8/d10/d14/f36 0 2024-04-05T21:26:35.482 INFO:tasks.workunit.client.1.smithi188.stdout:4/707: chown d0/d14/d15/c55 522131 1 2024-04-05T21:26:35.484 INFO:tasks.workunit.client.0.smithi121.stdout:3/84: mkdir d0/d3/dc/d1a/d1e 0 2024-04-05T21:26:35.484 INFO:tasks.workunit.client.0.smithi121.stdout:6/131: creat d1/d2/d1b/d1e/f2a x:0 0 0 2024-04-05T21:26:35.486 INFO:tasks.workunit.client.1.smithi188.stdout:6/521: write d6/d13/d18/d19/d2c/d3f/d77/f90 [128670,67114] 0 2024-04-05T21:26:35.486 INFO:tasks.workunit.client.1.smithi188.stdout:7/613: write d3/d4/db/d10/f95 [1617625,42682] 0 2024-04-05T21:26:35.487 INFO:tasks.workunit.client.0.smithi121.stdout:4/171: fsync d5/d11/d26/d27/f30 0 2024-04-05T21:26:35.489 INFO:tasks.workunit.client.0.smithi121.stdout:1/177: creat d8/d10/d14/f37 x:0 0 0 2024-04-05T21:26:35.489 INFO:tasks.workunit.client.1.smithi188.stdout:8/383: dread d5/d21/d40/d43/f81 [0,4194304] 0 2024-04-05T21:26:35.489 INFO:tasks.workunit.client.1.smithi188.stdout:7/614: chown d3/d4/db 52 1 2024-04-05T21:26:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:35 smithi121 ceph-mon[39723]: pgmap v16: 65 pgs: 65 active+clean; 1.1 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 24 MiB/s rd, 145 MiB/s wr, 237 op/s 2024-04-05T21:26:35.489 INFO:tasks.workunit.client.0.smithi121.stdout:5/158: dwrite f6 [0,4194304] 0 2024-04-05T21:26:35.490 INFO:tasks.workunit.client.0.smithi121.stdout:3/85: mknod d0/d3/dc/c1f 0 2024-04-05T21:26:35.490 INFO:tasks.workunit.client.1.smithi188.stdout:6/522: unlink d6/l12 0 2024-04-05T21:26:35.491 INFO:tasks.workunit.client.0.smithi121.stdout:2/159: creat d2/d9/d17/d25/d2d/f2e x:0 0 0 2024-04-05T21:26:35.492 INFO:tasks.workunit.client.1.smithi188.stdout:4/708: rename d0/d14/d15/d53/d82/d8f/d95/feb to d0/dbe/ffa 0 2024-04-05T21:26:35.494 INFO:tasks.workunit.client.1.smithi188.stdout:7/615: creat d3/d4/db/d2e/d3b/fbf x:0 0 0 2024-04-05T21:26:35.495 INFO:tasks.workunit.client.1.smithi188.stdout:4/709: read d0/d14/d15/d78/db1/fb5 [1676424,50003] 0 2024-04-05T21:26:35.497 INFO:tasks.workunit.client.0.smithi121.stdout:6/132: mknod d1/d2/d20/c2b 0 2024-04-05T21:26:35.497 INFO:tasks.workunit.client.0.smithi121.stdout:2/160: unlink d2/f23 0 2024-04-05T21:26:35.498 INFO:tasks.workunit.client.0.smithi121.stdout:3/86: write d0/d3/f8 [1038123,55939] 0 2024-04-05T21:26:35.499 INFO:tasks.workunit.client.0.smithi121.stdout:2/161: write d2/d9/d17/f18 [3681361,5753] 0 2024-04-05T21:26:35.499 INFO:tasks.workunit.client.0.smithi121.stdout:4/172: link d5/f16 d5/f3f 0 2024-04-05T21:26:35.500 INFO:tasks.workunit.client.1.smithi188.stdout:4/710: truncate d0/d14/d15/d53/d82/dd5/fd6 789086 0 2024-04-05T21:26:35.502 INFO:tasks.workunit.client.1.smithi188.stdout:7/616: dread d3/d4/db/f4a [0,4194304] 0 2024-04-05T21:26:35.504 INFO:tasks.workunit.client.0.smithi121.stdout:1/178: dwrite d8/d10/d14/f19 [0,4194304] 0 2024-04-05T21:26:35.506 INFO:tasks.workunit.client.0.smithi121.stdout:4/173: stat d5/d11/d28/d3b 0 2024-04-05T21:26:35.508 INFO:tasks.workunit.client.1.smithi188.stdout:8/384: write d5/f20 [1253740,42662] 0 2024-04-05T21:26:35.508 INFO:tasks.workunit.client.1.smithi188.stdout:4/711: write d0/d14/f6e [1701316,92051] 0 2024-04-05T21:26:35.509 INFO:tasks.workunit.client.1.smithi188.stdout:8/385: chown d5/dc/f17 1093722 1 2024-04-05T21:26:35.511 INFO:tasks.workunit.client.1.smithi188.stdout:4/712: fdatasync d0/d2/d17/d48/f57 0 2024-04-05T21:26:35.511 INFO:tasks.workunit.client.0.smithi121.stdout:3/87: creat d0/d3/dc/d1a/d1e/f20 x:0 0 0 2024-04-05T21:26:35.511 INFO:tasks.workunit.client.0.smithi121.stdout:0/227: write d7/dc/f32 [801899,120611] 0 2024-04-05T21:26:35.511 INFO:tasks.workunit.client.0.smithi121.stdout:3/88: stat d0/f2 0 2024-04-05T21:26:35.513 INFO:tasks.workunit.client.0.smithi121.stdout:0/228: dread - d7/dc/f33 zero size 2024-04-05T21:26:35.513 INFO:tasks.workunit.client.0.smithi121.stdout:6/133: getdents d1 0 2024-04-05T21:26:35.514 INFO:tasks.workunit.client.1.smithi188.stdout:8/386: mknod d5/d21/d40/d43/d48/d7e/c8c 0 2024-04-05T21:26:35.515 INFO:tasks.workunit.client.1.smithi188.stdout:4/713: symlink d0/d2/d2a/d34/d6b/lfb 0 2024-04-05T21:26:35.516 INFO:tasks.workunit.client.0.smithi121.stdout:6/134: fsync d1/d2/d1b/f28 0 2024-04-05T21:26:35.516 INFO:tasks.workunit.client.0.smithi121.stdout:3/89: creat d0/d3/dc/f21 x:0 0 0 2024-04-05T21:26:35.517 INFO:tasks.workunit.client.0.smithi121.stdout:6/135: stat d1/d2/d3/d5/l8 0 2024-04-05T21:26:35.518 INFO:tasks.workunit.client.1.smithi188.stdout:4/714: rename d0/d14/d15/d42/c7d to d0/d14/d15/d78/db1/d6a/cfc 0 2024-04-05T21:26:35.518 INFO:tasks.workunit.client.0.smithi121.stdout:0/229: rename d7/d1c/d1d/l3e to d7/d1c/d1d/d29/d2a/d2c/l48 0 2024-04-05T21:26:35.519 INFO:tasks.workunit.client.0.smithi121.stdout:0/230: chown d7/dc/f33 9 1 2024-04-05T21:26:35.519 INFO:tasks.workunit.client.1.smithi188.stdout:8/387: link d5/d21/d2a/d71/f88 d5/d21/d2a/f8d 0 2024-04-05T21:26:35.520 INFO:tasks.workunit.client.0.smithi121.stdout:3/90: creat d0/d3/dc/d1a/f22 x:0 0 0 2024-04-05T21:26:35.521 INFO:tasks.workunit.client.1.smithi188.stdout:8/388: dread - d5/d21/d2a/d71/f88 zero size 2024-04-05T21:26:35.521 INFO:tasks.workunit.client.0.smithi121.stdout:6/136: mkdir d1/d2/d3/d5/d1f/d2c 0 2024-04-05T21:26:35.522 INFO:tasks.workunit.client.1.smithi188.stdout:8/389: truncate d5/f3d 736584 0 2024-04-05T21:26:35.522 INFO:tasks.workunit.client.1.smithi188.stdout:4/715: rename d0/d14/d15/d78/db1/lb9 to d0/d2/d17/lfd 0 2024-04-05T21:26:35.523 INFO:tasks.workunit.client.0.smithi121.stdout:0/231: write d7/dc/f14 [559558,17908] 0 2024-04-05T21:26:35.524 INFO:tasks.workunit.client.0.smithi121.stdout:6/137: rmdir d1/d2/d3/d4 39 2024-04-05T21:26:35.525 INFO:tasks.workunit.client.1.smithi188.stdout:8/390: creat d5/d21/d2a/d47/f8e x:0 0 0 2024-04-05T21:26:35.525 INFO:tasks.workunit.client.1.smithi188.stdout:4/716: fdatasync d0/d14/d15/d53/fac 0 2024-04-05T21:26:35.525 INFO:tasks.workunit.client.0.smithi121.stdout:5/159: truncate dd/de/d19/d13/f2d 4101620 0 2024-04-05T21:26:35.526 INFO:tasks.workunit.client.0.smithi121.stdout:6/138: readlink d1/d2/d3/l29 0 2024-04-05T21:26:35.526 INFO:tasks.workunit.client.0.smithi121.stdout:5/160: chown dd 0 1 2024-04-05T21:26:35.526 INFO:tasks.workunit.client.0.smithi121.stdout:5/161: chown dd/f17 6356433 1 2024-04-05T21:26:35.527 INFO:tasks.workunit.client.0.smithi121.stdout:5/162: chown f6 633 1 2024-04-05T21:26:35.527 INFO:tasks.workunit.client.0.smithi121.stdout:5/163: fsync f4 0 2024-04-05T21:26:35.528 INFO:tasks.workunit.client.0.smithi121.stdout:0/232: rename c3 to d7/d1c/d1d/c49 0 2024-04-05T21:26:35.528 INFO:tasks.workunit.client.1.smithi188.stdout:4/717: mknod d0/d14/d15/dd4/ddf/cfe 0 2024-04-05T21:26:35.530 INFO:tasks.workunit.client.1.smithi188.stdout:4/718: symlink d0/d2/d52/dc1/lff 0 2024-04-05T21:26:35.530 INFO:tasks.workunit.client.0.smithi121.stdout:6/139: write d1/d2/d1b/d1e/f2a [310358,33940] 0 2024-04-05T21:26:35.531 INFO:tasks.workunit.client.1.smithi188.stdout:4/719: write d0/d14/d15/d42/fa0 [1272062,34465] 0 2024-04-05T21:26:35.535 INFO:tasks.workunit.client.0.smithi121.stdout:1/179: dread f4 [0,4194304] 0 2024-04-05T21:26:35.536 INFO:tasks.workunit.client.0.smithi121.stdout:5/164: write dd/de/d19/d13/f26 [452292,16842] 0 2024-04-05T21:26:35.536 INFO:tasks.workunit.client.1.smithi188.stdout:4/720: link d0/d2/f26 d0/d14/d15/d78/db1/d6a/f100 0 2024-04-05T21:26:35.537 INFO:tasks.workunit.client.0.smithi121.stdout:3/91: truncate d0/d3/f13 825584 0 2024-04-05T21:26:35.538 INFO:tasks.workunit.client.1.smithi188.stdout:4/721: fsync d0/d2/d17/d48/d70/fda 0 2024-04-05T21:26:35.540 INFO:tasks.workunit.client.0.smithi121.stdout:2/162: dwrite d2/d9/ff [0,4194304] 0 2024-04-05T21:26:35.542 INFO:tasks.workunit.client.0.smithi121.stdout:5/165: write dd/f17 [1470917,94219] 0 2024-04-05T21:26:35.543 INFO:tasks.workunit.client.1.smithi188.stdout:4/722: truncate d0/d81/f85 1594564 0 2024-04-05T21:26:35.545 INFO:tasks.workunit.client.1.smithi188.stdout:4/723: mknod d0/d14/d15/d53/d82/c101 0 2024-04-05T21:26:35.545 INFO:tasks.workunit.client.0.smithi121.stdout:0/233: dwrite d7/d1c/f1f [0,4194304] 0 2024-04-05T21:26:35.545 INFO:tasks.workunit.client.0.smithi121.stdout:1/180: mkdir d8/d10/d14/d1a/d1b/d38 0 2024-04-05T21:26:35.546 INFO:tasks.workunit.client.0.smithi121.stdout:4/174: dread f4 [0,4194304] 0 2024-04-05T21:26:35.550 INFO:tasks.workunit.client.0.smithi121.stdout:3/92: getdents d0/d3/de 0 2024-04-05T21:26:35.551 INFO:tasks.workunit.client.0.smithi121.stdout:4/175: truncate d5/d11/d2c/f36 714704 0 2024-04-05T21:26:35.553 INFO:tasks.workunit.client.0.smithi121.stdout:3/93: write d0/d3/f4 [5237420,120382] 0 2024-04-05T21:26:35.554 INFO:tasks.workunit.client.0.smithi121.stdout:6/140: dwrite d1/d2/f17 [0,4194304] 0 2024-04-05T21:26:35.554 INFO:tasks.workunit.client.0.smithi121.stdout:0/234: symlink d7/dc/l4a 0 2024-04-05T21:26:35.556 INFO:tasks.workunit.client.0.smithi121.stdout:2/163: truncate d2/d9/fd 1712263 0 2024-04-05T21:26:35.557 INFO:tasks.workunit.client.0.smithi121.stdout:5/166: getdents dd 0 2024-04-05T21:26:35.557 INFO:tasks.workunit.client.0.smithi121.stdout:1/181: truncate d8/d10/d14/d1a/f20 1188361 0 2024-04-05T21:26:35.558 INFO:tasks.workunit.client.0.smithi121.stdout:0/235: dread - d7/dc/f33 zero size 2024-04-05T21:26:35.558 INFO:tasks.workunit.client.0.smithi121.stdout:4/176: creat d5/d11/d26/d27/d29/d2d/f40 x:0 0 0 2024-04-05T21:26:35.559 INFO:tasks.workunit.client.0.smithi121.stdout:3/94: write d0/f15 [990890,94001] 0 2024-04-05T21:26:35.561 INFO:tasks.workunit.client.0.smithi121.stdout:8/110: sync 2024-04-05T21:26:35.561 INFO:tasks.workunit.client.0.smithi121.stdout:0/236: stat d7/d1c/f25 0 2024-04-05T21:26:35.561 INFO:tasks.workunit.client.0.smithi121.stdout:8/111: stat l10 0 2024-04-05T21:26:35.563 INFO:tasks.workunit.client.0.smithi121.stdout:4/177: fdatasync d5/d11/f22 0 2024-04-05T21:26:35.564 INFO:tasks.workunit.client.0.smithi121.stdout:6/141: creat d1/d2/d1b/d1e/f2d x:0 0 0 2024-04-05T21:26:35.564 INFO:tasks.workunit.client.0.smithi121.stdout:8/112: write ff [1447053,114744] 0 2024-04-05T21:26:35.565 INFO:tasks.workunit.client.0.smithi121.stdout:3/95: chown d0/d3/dc/d1a/f22 35 1 2024-04-05T21:26:35.566 INFO:tasks.workunit.client.0.smithi121.stdout:2/164: creat d2/d9/f2f x:0 0 0 2024-04-05T21:26:35.569 INFO:tasks.workunit.client.0.smithi121.stdout:5/167: dread dd/de/d19/d13/f16 [0,4194304] 0 2024-04-05T21:26:35.570 INFO:tasks.workunit.client.0.smithi121.stdout:0/237: creat d7/dc/d10/d3c/f4b x:0 0 0 2024-04-05T21:26:35.570 INFO:tasks.workunit.client.0.smithi121.stdout:5/168: readlink dd/l2c 0 2024-04-05T21:26:35.571 INFO:tasks.workunit.client.0.smithi121.stdout:4/178: unlink d5/d11/d2c/c34 0 2024-04-05T21:26:35.572 INFO:tasks.workunit.client.0.smithi121.stdout:6/142: dread d1/d2/d1b/d1e/f2a [0,4194304] 0 2024-04-05T21:26:35.575 INFO:tasks.workunit.client.0.smithi121.stdout:2/165: mkdir d2/d9/d1c/d30 0 2024-04-05T21:26:35.576 INFO:tasks.workunit.client.0.smithi121.stdout:1/182: dwrite d8/d10/f26 [0,4194304] 0 2024-04-05T21:26:35.578 INFO:tasks.workunit.client.0.smithi121.stdout:1/183: write f2 [2763329,28840] 0 2024-04-05T21:26:35.579 INFO:tasks.workunit.client.0.smithi121.stdout:0/238: creat d7/d1c/d1d/d29/d2a/f4c x:0 0 0 2024-04-05T21:26:35.583 INFO:tasks.workunit.client.0.smithi121.stdout:2/166: dwrite d2/f10 [0,4194304] 0 2024-04-05T21:26:35.584 INFO:tasks.workunit.client.0.smithi121.stdout:2/167: chown d2/d9/d17/l27 417388149 1 2024-04-05T21:26:35.590 INFO:tasks.workunit.client.0.smithi121.stdout:0/239: mknod d7/d1c/d1d/d29/d2a/d2c/c4d 0 2024-04-05T21:26:35.592 INFO:tasks.workunit.client.0.smithi121.stdout:8/113: link cd d12/c24 0 2024-04-05T21:26:35.596 INFO:tasks.workunit.client.0.smithi121.stdout:6/143: dwrite d1/d2/d3/d4/de/f26 [0,4194304] 0 2024-04-05T21:26:35.597 INFO:tasks.workunit.client.0.smithi121.stdout:0/240: symlink d7/l4e 0 2024-04-05T21:26:35.600 INFO:tasks.workunit.client.0.smithi121.stdout:8/114: mknod d12/d13/d15/c25 0 2024-04-05T21:26:35.600 INFO:tasks.workunit.client.0.smithi121.stdout:2/168: mknod d2/d28/c31 0 2024-04-05T21:26:35.601 INFO:tasks.workunit.client.0.smithi121.stdout:3/96: dread d0/d3/f13 [0,4194304] 0 2024-04-05T21:26:35.601 INFO:tasks.workunit.client.0.smithi121.stdout:5/169: getdents dd/de 0 2024-04-05T21:26:35.605 INFO:tasks.workunit.client.0.smithi121.stdout:8/115: fdatasync d12/d13/d15/f19 0 2024-04-05T21:26:35.607 INFO:tasks.workunit.client.0.smithi121.stdout:2/169: unlink d2/d28/c31 0 2024-04-05T21:26:35.608 INFO:tasks.workunit.client.0.smithi121.stdout:4/179: dwrite d5/f3f [4194304,4194304] 0 2024-04-05T21:26:35.610 INFO:tasks.workunit.client.0.smithi121.stdout:3/97: unlink d0/c1 0 2024-04-05T21:26:35.611 INFO:tasks.workunit.client.0.smithi121.stdout:8/116: read - d12/d13/d15/f19 zero size 2024-04-05T21:26:35.612 INFO:tasks.workunit.client.0.smithi121.stdout:0/241: dread d7/dc/f14 [0,4194304] 0 2024-04-05T21:26:35.612 INFO:tasks.workunit.client.0.smithi121.stdout:6/144: dwrite d1/d2/f17 [0,4194304] 0 2024-04-05T21:26:35.613 INFO:tasks.workunit.client.0.smithi121.stdout:8/117: chown c11 66698533 1 2024-04-05T21:26:35.613 INFO:tasks.workunit.client.0.smithi121.stdout:2/170: mknod d2/d9/d17/d25/c32 0 2024-04-05T21:26:35.615 INFO:tasks.workunit.client.0.smithi121.stdout:4/180: symlink d5/d11/d26/l41 0 2024-04-05T21:26:35.618 INFO:tasks.workunit.client.0.smithi121.stdout:3/98: mkdir d0/d3/dc/d1a/d23 0 2024-04-05T21:26:35.619 INFO:tasks.workunit.client.0.smithi121.stdout:8/118: symlink d12/l26 0 2024-04-05T21:26:35.620 INFO:tasks.workunit.client.0.smithi121.stdout:6/145: write d1/d2/d3/d5/d1f/f27 [970486,54885] 0 2024-04-05T21:26:35.621 INFO:tasks.workunit.client.0.smithi121.stdout:8/119: readlink l7 0 2024-04-05T21:26:35.621 INFO:tasks.workunit.client.0.smithi121.stdout:0/242: read d7/dc/f2d [438073,46977] 0 2024-04-05T21:26:35.622 INFO:tasks.workunit.client.0.smithi121.stdout:3/99: fdatasync d0/f15 0 2024-04-05T21:26:35.624 INFO:tasks.workunit.client.0.smithi121.stdout:5/170: dread dd/de/d19/f28 [0,4194304] 0 2024-04-05T21:26:35.625 INFO:tasks.workunit.client.0.smithi121.stdout:4/181: write d5/d11/d26/d27/d29/d2d/f40 [692889,68391] 0 2024-04-05T21:26:35.625 INFO:tasks.workunit.client.0.smithi121.stdout:2/171: unlink d2/d9/d17/d25/d2d/f2e 0 2024-04-05T21:26:35.626 INFO:tasks.workunit.client.0.smithi121.stdout:6/146: readlink d1/d2/d3/d4/l23 0 2024-04-05T21:26:35.628 INFO:tasks.workunit.client.0.smithi121.stdout:0/243: dread d7/f3a [0,4194304] 0 2024-04-05T21:26:35.629 INFO:tasks.workunit.client.0.smithi121.stdout:6/147: write d1/d2/dc/f19 [773291,115816] 0 2024-04-05T21:26:35.629 INFO:tasks.workunit.client.0.smithi121.stdout:8/120: rename f6 to d12/d13/d15/d1d/f27 0 2024-04-05T21:26:35.630 INFO:tasks.workunit.client.0.smithi121.stdout:3/100: mkdir d0/d3/d24 0 2024-04-05T21:26:35.631 INFO:tasks.workunit.client.0.smithi121.stdout:3/101: readlink d0/d3/dc/l1c 0 2024-04-05T21:26:35.633 INFO:tasks.workunit.client.0.smithi121.stdout:2/172: mknod d2/d28/c33 0 2024-04-05T21:26:35.633 INFO:tasks.workunit.client.0.smithi121.stdout:3/102: fsync d0/d3/f4 0 2024-04-05T21:26:35.635 INFO:tasks.workunit.client.0.smithi121.stdout:4/182: mknod d5/d11/d28/d3b/c42 0 2024-04-05T21:26:35.637 INFO:tasks.workunit.client.0.smithi121.stdout:4/183: chown d5/d11/d26/l41 30 1 2024-04-05T21:26:35.639 INFO:tasks.workunit.client.0.smithi121.stdout:0/244: rename d7/d1c/d1d/d29/d2a/f35 to d7/d1c/d1d/d29/d3f/f4f 0 2024-04-05T21:26:35.640 INFO:tasks.workunit.client.0.smithi121.stdout:8/121: dwrite d12/d13/d15/f19 [0,4194304] 0 2024-04-05T21:26:35.643 INFO:tasks.workunit.client.1.smithi188.stdout:3/451: sync 2024-04-05T21:26:35.644 INFO:tasks.workunit.client.1.smithi188.stdout:1/480: sync 2024-04-05T21:26:35.644 INFO:tasks.workunit.client.1.smithi188.stdout:9/462: sync 2024-04-05T21:26:35.644 INFO:tasks.workunit.client.0.smithi121.stdout:5/171: creat dd/de/d19/f32 x:0 0 0 2024-04-05T21:26:35.644 INFO:tasks.workunit.client.0.smithi121.stdout:4/184: symlink d5/d11/d28/d3b/l43 0 2024-04-05T21:26:35.645 INFO:tasks.workunit.client.0.smithi121.stdout:8/122: symlink d12/d13/d15/l28 0 2024-04-05T21:26:35.646 INFO:tasks.workunit.client.0.smithi121.stdout:0/245: unlink d7/dc/d10/c28 0 2024-04-05T21:26:35.647 INFO:tasks.workunit.client.1.smithi188.stdout:1/481: fdatasync d1/d7/d4e/d5f/f76 0 2024-04-05T21:26:35.647 INFO:tasks.workunit.client.0.smithi121.stdout:3/103: read d0/d3/fa [536740,120163] 0 2024-04-05T21:26:35.648 INFO:tasks.workunit.client.0.smithi121.stdout:3/104: dread - d0/d3/dc/d1a/f22 zero size 2024-04-05T21:26:35.649 INFO:tasks.workunit.client.1.smithi188.stdout:9/463: read dd/d49/d6b/d40/f4e [6081239,23453] 0 2024-04-05T21:26:35.650 INFO:tasks.workunit.client.0.smithi121.stdout:4/185: unlink d5/f3a 0 2024-04-05T21:26:35.651 INFO:tasks.workunit.client.0.smithi121.stdout:0/246: rmdir d7/d1c/d1d 39 2024-04-05T21:26:35.653 INFO:tasks.workunit.client.0.smithi121.stdout:5/172: getdents dd/de 0 2024-04-05T21:26:35.654 INFO:tasks.workunit.client.0.smithi121.stdout:3/105: getdents d0/d3/d24 0 2024-04-05T21:26:35.654 INFO:tasks.workunit.client.1.smithi188.stdout:9/464: creat dd/d49/d6b/f96 x:0 0 0 2024-04-05T21:26:35.658 INFO:tasks.workunit.client.1.smithi188.stdout:3/452: creat d5/f9b x:0 0 0 2024-04-05T21:26:35.659 INFO:tasks.workunit.client.1.smithi188.stdout:1/482: dwrite d1/dc/d16/d21/d27/f29 [4194304,4194304] 0 2024-04-05T21:26:35.661 INFO:tasks.workunit.client.0.smithi121.stdout:5/173: creat dd/de/d19/d1c/f33 x:0 0 0 2024-04-05T21:26:35.661 INFO:tasks.workunit.client.1.smithi188.stdout:9/465: mknod dd/d3d/d95/c97 0 2024-04-05T21:26:35.662 INFO:tasks.workunit.client.1.smithi188.stdout:1/483: rmdir d1/dc/d16/d21/d27/d90 39 2024-04-05T21:26:35.662 INFO:tasks.workunit.client.0.smithi121.stdout:3/106: mkdir d0/d3/d24/d25 0 2024-04-05T21:26:35.666 INFO:tasks.workunit.client.0.smithi121.stdout:5/174: creat dd/de/f34 x:0 0 0 2024-04-05T21:26:35.666 INFO:tasks.workunit.client.0.smithi121.stdout:5/175: chown dd/de 68 1 2024-04-05T21:26:35.666 INFO:tasks.workunit.client.0.smithi121.stdout:5/176: chown dd/de/d19 2289 1 2024-04-05T21:26:35.667 INFO:tasks.workunit.client.1.smithi188.stdout:9/466: mkdir dd/d11/d1f/d26/d55/d98 0 2024-04-05T21:26:35.668 INFO:tasks.workunit.client.1.smithi188.stdout:3/453: dwrite d5/d7/d31/f91 [0,4194304] 0 2024-04-05T21:26:35.669 INFO:tasks.workunit.client.0.smithi121.stdout:5/177: creat dd/de/d19/d13/f35 x:0 0 0 2024-04-05T21:26:35.673 INFO:tasks.workunit.client.1.smithi188.stdout:9/467: write dd/d49/d6b/f43 [2342001,74204] 0 2024-04-05T21:26:35.674 INFO:tasks.workunit.client.0.smithi121.stdout:3/107: dwrite d0/d3/de/f17 [0,4194304] 0 2024-04-05T21:26:35.677 INFO:tasks.workunit.client.0.smithi121.stdout:3/108: stat d0/d3/d24 0 2024-04-05T21:26:35.678 INFO:tasks.workunit.client.0.smithi121.stdout:5/178: mkdir dd/de/d19/d1c/d36 0 2024-04-05T21:26:35.678 INFO:tasks.workunit.client.0.smithi121.stdout:8/123: rmdir d12/d13 39 2024-04-05T21:26:35.678 INFO:tasks.workunit.client.0.smithi121.stdout:3/109: chown d0/f2 305625196 1 2024-04-05T21:26:35.679 INFO:tasks.workunit.client.1.smithi188.stdout:3/454: mknod d5/d39/d84/c9c 0 2024-04-05T21:26:35.679 INFO:tasks.workunit.client.1.smithi188.stdout:1/484: link d1/dc/d16/d21/d27/d4a/c9c d1/dc/d16/d46/d95/d99/ca4 0 2024-04-05T21:26:35.680 INFO:tasks.workunit.client.0.smithi121.stdout:3/110: write d0/d3/dc/f21 [602338,6908] 0 2024-04-05T21:26:35.682 INFO:tasks.workunit.client.0.smithi121.stdout:0/247: truncate d7/d1c/f25 1427717 0 2024-04-05T21:26:35.684 INFO:tasks.workunit.client.0.smithi121.stdout:5/179: dread dd/de/f11 [0,4194304] 0 2024-04-05T21:26:35.685 INFO:tasks.workunit.client.0.smithi121.stdout:3/111: stat d0/d3/de/l1d 0 2024-04-05T21:26:35.686 INFO:tasks.workunit.client.1.smithi188.stdout:9/468: rmdir dd/d11/d22/d8b 0 2024-04-05T21:26:35.687 INFO:tasks.workunit.client.0.smithi121.stdout:8/124: mkdir d12/d13/d15/d1d/d29 0 2024-04-05T21:26:35.687 INFO:tasks.workunit.client.0.smithi121.stdout:4/186: dread d5/f1d [0,4194304] 0 2024-04-05T21:26:35.687 INFO:tasks.workunit.client.0.smithi121.stdout:5/180: truncate dd/de/f11 4274458 0 2024-04-05T21:26:35.688 INFO:tasks.workunit.client.0.smithi121.stdout:3/112: write d0/f15 [740626,11862] 0 2024-04-05T21:26:35.689 INFO:tasks.workunit.client.0.smithi121.stdout:5/181: read dd/f1f [904134,28534] 0 2024-04-05T21:26:35.689 INFO:tasks.workunit.client.1.smithi188.stdout:9/469: mkdir dd/d34/d99 0 2024-04-05T21:26:35.689 INFO:tasks.workunit.client.0.smithi121.stdout:8/125: chown d12/d13/d15/d1d/d29 2 1 2024-04-05T21:26:35.691 INFO:tasks.workunit.client.0.smithi121.stdout:8/126: stat d12/d13/d15/l28 0 2024-04-05T21:26:35.693 INFO:tasks.workunit.client.0.smithi121.stdout:8/127: read - d12/d13/f20 zero size 2024-04-05T21:26:35.693 INFO:tasks.workunit.client.1.smithi188.stdout:3/455: rename d5/d7e/c99 to d5/c9d 0 2024-04-05T21:26:35.695 INFO:tasks.workunit.client.0.smithi121.stdout:3/113: link d0/d3/de/c19 d0/d3/c26 0 2024-04-05T21:26:35.695 INFO:tasks.workunit.client.0.smithi121.stdout:8/128: chown d12/d13/d15/l23 22298 1 2024-04-05T21:26:35.695 INFO:tasks.workunit.client.0.smithi121.stdout:5/182: dread dd/de/d19/f23 [0,4194304] 0 2024-04-05T21:26:35.696 INFO:tasks.workunit.client.0.smithi121.stdout:8/129: rename d12 to d12/d13/d2a 22 2024-04-05T21:26:35.696 INFO:tasks.workunit.client.0.smithi121.stdout:5/183: fdatasync dd/de/f2e 0 2024-04-05T21:26:35.697 INFO:tasks.workunit.client.0.smithi121.stdout:3/114: write d0/d3/dc/d1a/d1e/f20 [903678,6737] 0 2024-04-05T21:26:35.698 INFO:tasks.workunit.client.0.smithi121.stdout:3/115: read - d0/f2 zero size 2024-04-05T21:26:35.699 INFO:tasks.workunit.client.0.smithi121.stdout:8/130: mknod d12/d13/c2b 0 2024-04-05T21:26:35.699 INFO:tasks.workunit.client.1.smithi188.stdout:1/485: dwrite d1/dc/d16/d46/d88/d13/f8d [0,4194304] 0 2024-04-05T21:26:35.699 INFO:tasks.workunit.client.0.smithi121.stdout:3/116: truncate d0/d3/f8 4271604 0 2024-04-05T21:26:35.700 INFO:tasks.workunit.client.0.smithi121.stdout:8/131: chown d12/d13/f1b 970717 1 2024-04-05T21:26:35.702 INFO:tasks.workunit.client.0.smithi121.stdout:5/184: creat dd/de/d19/f37 x:0 0 0 2024-04-05T21:26:35.703 INFO:tasks.workunit.client.0.smithi121.stdout:3/117: creat d0/f27 x:0 0 0 2024-04-05T21:26:35.705 INFO:tasks.workunit.client.1.smithi188.stdout:9/470: rename dd/d11/f21 to dd/d34/f9a 0 2024-04-05T21:26:35.706 INFO:tasks.workunit.client.0.smithi121.stdout:3/118: readlink d0/d3/de/l12 0 2024-04-05T21:26:35.706 INFO:tasks.workunit.client.0.smithi121.stdout:8/132: symlink d12/d13/d15/d1d/d29/l2c 0 2024-04-05T21:26:35.706 INFO:tasks.workunit.client.0.smithi121.stdout:3/119: dread - d0/d3/dc/d1a/f22 zero size 2024-04-05T21:26:35.706 INFO:tasks.workunit.client.0.smithi121.stdout:5/185: mkdir dd/de/d19/d38 0 2024-04-05T21:26:35.708 INFO:tasks.workunit.client.0.smithi121.stdout:5/186: chown dd/de/c12 350 1 2024-04-05T21:26:35.710 INFO:tasks.workunit.client.0.smithi121.stdout:0/248: write d7/d1c/d1d/f26 [1270212,55651] 0 2024-04-05T21:26:35.711 INFO:tasks.workunit.client.1.smithi188.stdout:9/471: symlink dd/d3d/l9b 0 2024-04-05T21:26:35.711 INFO:tasks.workunit.client.1.smithi188.stdout:1/486: dwrite d1/fb [4194304,4194304] 0 2024-04-05T21:26:35.711 INFO:tasks.workunit.client.0.smithi121.stdout:8/133: dread d12/d13/d15/f21 [0,4194304] 0 2024-04-05T21:26:35.712 INFO:tasks.workunit.client.0.smithi121.stdout:5/187: mkdir dd/de/d39 0 2024-04-05T21:26:35.713 INFO:tasks.workunit.client.0.smithi121.stdout:4/187: write d5/f8 [223499,114043] 0 2024-04-05T21:26:35.715 INFO:tasks.workunit.client.0.smithi121.stdout:0/249: creat d7/d1c/d1d/d29/d3f/f50 x:0 0 0 2024-04-05T21:26:35.717 INFO:tasks.workunit.client.0.smithi121.stdout:8/134: chown fa 5126 1 2024-04-05T21:26:35.717 INFO:tasks.workunit.client.0.smithi121.stdout:5/188: fdatasync dd/de/f1a 0 2024-04-05T21:26:35.717 INFO:tasks.workunit.client.0.smithi121.stdout:3/120: dread d0/d3/f4 [0,4194304] 0 2024-04-05T21:26:35.718 INFO:tasks.workunit.client.0.smithi121.stdout:8/135: write fc [992587,85969] 0 2024-04-05T21:26:35.719 INFO:tasks.workunit.client.0.smithi121.stdout:0/250: rmdir d7/d1c/d1d/d45 39 2024-04-05T21:26:35.720 INFO:tasks.workunit.client.1.smithi188.stdout:9/472: dwrite dd/d11/d1f/d26/f19 [0,4194304] 0 2024-04-05T21:26:35.721 INFO:tasks.workunit.client.0.smithi121.stdout:3/121: chown d0/d3/dc/d1a/d1e/f20 324860840 1 2024-04-05T21:26:35.722 INFO:tasks.workunit.client.1.smithi188.stdout:9/473: write dd/d11/d1f/f6e [1528417,17128] 0 2024-04-05T21:26:35.724 INFO:tasks.workunit.client.1.smithi188.stdout:9/474: chown dd/d11/l15 33414162 1 2024-04-05T21:26:35.726 INFO:tasks.workunit.client.0.smithi121.stdout:4/188: unlink d5/d11/c37 0 2024-04-05T21:26:35.726 INFO:tasks.workunit.client.1.smithi188.stdout:1/487: dwrite d1/dc/f11 [4194304,4194304] 0 2024-04-05T21:26:35.727 INFO:tasks.workunit.client.0.smithi121.stdout:8/136: symlink d12/d13/d15/d1d/l2d 0 2024-04-05T21:26:35.727 INFO:tasks.workunit.client.1.smithi188.stdout:1/488: dread - d1/d7/f80 zero size 2024-04-05T21:26:35.730 INFO:tasks.workunit.client.0.smithi121.stdout:3/122: rename d0/d3/cd to d0/d3/de/c28 0 2024-04-05T21:26:35.731 INFO:tasks.workunit.client.0.smithi121.stdout:3/123: chown d0 893468 1 2024-04-05T21:26:35.731 INFO:tasks.workunit.client.0.smithi121.stdout:4/189: creat d5/d11/d28/f44 x:0 0 0 2024-04-05T21:26:35.732 INFO:tasks.workunit.client.0.smithi121.stdout:0/251: mknod d7/d1c/d1d/d45/c51 0 2024-04-05T21:26:35.734 INFO:tasks.workunit.client.1.smithi188.stdout:1/489: unlink d1/dc/d16/d21/d27/d90/c9f 0 2024-04-05T21:26:35.734 INFO:tasks.workunit.client.0.smithi121.stdout:4/190: write d5/d11/d28/f2f [601847,10762] 0 2024-04-05T21:26:35.740 INFO:tasks.workunit.client.0.smithi121.stdout:8/137: mknod d12/d13/d15/d1d/d29/c2e 0 2024-04-05T21:26:35.740 INFO:tasks.workunit.client.0.smithi121.stdout:3/124: creat d0/d3/dc/d1a/f29 x:0 0 0 2024-04-05T21:26:35.741 INFO:tasks.workunit.client.0.smithi121.stdout:0/252: rename d7/d1c/d1d/d29/c36 to d7/c52 0 2024-04-05T21:26:35.741 INFO:tasks.workunit.client.1.smithi188.stdout:9/475: dread f1 [0,4194304] 0 2024-04-05T21:26:35.743 INFO:tasks.workunit.client.0.smithi121.stdout:0/253: chown d7/dc/f15 29163 1 2024-04-05T21:26:35.743 INFO:tasks.workunit.client.0.smithi121.stdout:7/132: sync 2024-04-05T21:26:35.743 INFO:tasks.workunit.client.0.smithi121.stdout:9/157: sync 2024-04-05T21:26:35.744 INFO:tasks.workunit.client.0.smithi121.stdout:4/191: dwrite d5/d11/f14 [0,4194304] 0 2024-04-05T21:26:35.745 INFO:tasks.workunit.client.0.smithi121.stdout:0/254: stat d7/dc/f15 0 2024-04-05T21:26:35.751 INFO:tasks.workunit.client.1.smithi188.stdout:3/456: dread d5/df/f15 [0,4194304] 0 2024-04-05T21:26:35.752 INFO:tasks.workunit.client.0.smithi121.stdout:0/255: chown d7/d1c/d1d/c44 22275665 1 2024-04-05T21:26:35.753 INFO:tasks.workunit.client.0.smithi121.stdout:3/125: creat d0/d3/dc/d1a/d1e/f2a x:0 0 0 2024-04-05T21:26:35.754 INFO:tasks.workunit.client.0.smithi121.stdout:9/158: symlink d6/df/d1d/d26/l31 0 2024-04-05T21:26:35.754 INFO:tasks.workunit.client.1.smithi188.stdout:3/457: creat d5/d35/f9e x:0 0 0 2024-04-05T21:26:35.755 INFO:tasks.workunit.client.0.smithi121.stdout:4/192: fdatasync d5/d11/d26/d27/d29/d2d/f31 0 2024-04-05T21:26:35.756 INFO:tasks.workunit.client.0.smithi121.stdout:3/126: unlink d0/c1b 0 2024-04-05T21:26:35.756 INFO:tasks.workunit.client.1.smithi188.stdout:3/458: write d5/d35/d4e/f96 [448771,36413] 0 2024-04-05T21:26:35.759 INFO:tasks.workunit.client.0.smithi121.stdout:9/159: mkdir d6/d32 0 2024-04-05T21:26:35.761 INFO:tasks.workunit.client.0.smithi121.stdout:8/138: dwrite d12/d13/d15/d1d/f27 [4194304,4194304] 0 2024-04-05T21:26:35.762 INFO:tasks.workunit.client.1.smithi188.stdout:3/459: dwrite d5/f6b [0,4194304] 0 2024-04-05T21:26:35.763 INFO:tasks.workunit.client.0.smithi121.stdout:9/160: fdatasync d6/df/d1d/d2c/f24 0 2024-04-05T21:26:35.768 INFO:tasks.workunit.client.1.smithi188.stdout:3/460: rename d5/fa to d5/d7/d65/d61/d66/f9f 0 2024-04-05T21:26:35.769 INFO:tasks.workunit.client.0.smithi121.stdout:0/256: dwrite d7/dc/d10/f42 [0,4194304] 0 2024-04-05T21:26:35.769 INFO:tasks.workunit.client.0.smithi121.stdout:4/193: read d5/d11/d28/f2f [587183,74863] 0 2024-04-05T21:26:35.770 INFO:tasks.workunit.client.0.smithi121.stdout:3/127: write d0/d3/f4 [2624490,10312] 0 2024-04-05T21:26:35.770 INFO:tasks.workunit.client.1.smithi188.stdout:3/461: dread - d5/dc/f3d zero size 2024-04-05T21:26:35.772 INFO:tasks.workunit.client.0.smithi121.stdout:7/133: dwrite d0/d9/f20 [0,4194304] 0 2024-04-05T21:26:35.774 INFO:tasks.workunit.client.0.smithi121.stdout:8/139: rmdir d12/d13/d15/d1d 39 2024-04-05T21:26:35.776 INFO:tasks.workunit.client.0.smithi121.stdout:5/189: truncate dd/de/f21 675503 0 2024-04-05T21:26:35.776 INFO:tasks.workunit.client.1.smithi188.stdout:3/462: write d5/d7/d31/f5c [8774574,1406] 0 2024-04-05T21:26:35.777 INFO:tasks.workunit.client.1.smithi188.stdout:3/463: stat d5/dc/l3f 0 2024-04-05T21:26:35.777 INFO:tasks.workunit.client.0.smithi121.stdout:3/128: read d0/d3/fa [258017,50822] 0 2024-04-05T21:26:35.779 INFO:tasks.workunit.client.0.smithi121.stdout:9/161: rename f5 to d6/df/d1d/f33 0 2024-04-05T21:26:35.779 INFO:tasks.workunit.client.1.smithi188.stdout:3/464: creat d5/d35/d4e/fa0 x:0 0 0 2024-04-05T21:26:35.780 INFO:tasks.workunit.client.0.smithi121.stdout:4/194: write d5/d11/f35 [927383,56998] 0 2024-04-05T21:26:35.780 INFO:tasks.workunit.client.0.smithi121.stdout:9/162: chown l3 13 1 2024-04-05T21:26:35.781 INFO:tasks.workunit.client.1.smithi188.stdout:3/465: write d5/df/f33 [3371590,104662] 0 2024-04-05T21:26:35.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:35 smithi188 ceph-mon[49715]: pgmap v16: 65 pgs: 65 active+clean; 1.1 GiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 24 MiB/s rd, 145 MiB/s wr, 237 op/s 2024-04-05T21:26:35.783 INFO:tasks.workunit.client.0.smithi121.stdout:3/129: write d0/d3/dc/d1a/f22 [103215,3543] 0 2024-04-05T21:26:35.784 INFO:tasks.workunit.client.1.smithi188.stdout:3/466: rmdir d5/d2b 39 2024-04-05T21:26:35.790 INFO:tasks.workunit.client.0.smithi121.stdout:4/195: rmdir d5/d11/d2c 39 2024-04-05T21:26:35.793 INFO:tasks.workunit.client.0.smithi121.stdout:8/140: mknod d12/d13/d15/d1d/d29/c2f 0 2024-04-05T21:26:35.794 INFO:tasks.workunit.client.0.smithi121.stdout:3/130: dwrite d0/f15 [0,4194304] 0 2024-04-05T21:26:35.796 INFO:tasks.workunit.client.0.smithi121.stdout:0/257: mkdir d7/d1c/d1d/d45/d46/d53 0 2024-04-05T21:26:35.799 INFO:tasks.workunit.client.0.smithi121.stdout:8/141: dread ff [0,4194304] 0 2024-04-05T21:26:35.804 INFO:tasks.workunit.client.0.smithi121.stdout:9/163: write d6/f1c [918237,29075] 0 2024-04-05T21:26:35.809 INFO:tasks.workunit.client.0.smithi121.stdout:4/196: mkdir d5/d45 0 2024-04-05T21:26:35.810 INFO:tasks.workunit.client.0.smithi121.stdout:5/190: write dd/f29 [449984,106311] 0 2024-04-05T21:26:35.811 INFO:tasks.workunit.client.1.smithi188.stdout:3/467: dwrite d5/d35/f52 [0,4194304] 0 2024-04-05T21:26:35.812 INFO:tasks.workunit.client.0.smithi121.stdout:0/258: link d7/d1c/d1d/d29/f34 d7/d1c/d1d/d45/f54 0 2024-04-05T21:26:35.812 INFO:tasks.workunit.client.1.smithi188.stdout:3/468: truncate d5/d39/f93 486307 0 2024-04-05T21:26:35.813 INFO:tasks.workunit.client.0.smithi121.stdout:9/164: dwrite d6/f1c [0,4194304] 0 2024-04-05T21:26:35.814 INFO:tasks.workunit.client.0.smithi121.stdout:7/134: truncate d0/d9/fe 4084906 0 2024-04-05T21:26:35.814 INFO:tasks.workunit.client.1.smithi188.stdout:3/469: creat d5/d7/d65/d5f/fa1 x:0 0 0 2024-04-05T21:26:35.814 INFO:tasks.workunit.client.0.smithi121.stdout:8/142: mkdir d12/d30 0 2024-04-05T21:26:35.821 INFO:tasks.workunit.client.0.smithi121.stdout:9/165: rmdir d6 39 2024-04-05T21:26:35.824 INFO:tasks.workunit.client.0.smithi121.stdout:0/259: fsync d7/dc/d10/f11 0 2024-04-05T21:26:35.826 INFO:tasks.workunit.client.0.smithi121.stdout:7/135: creat d0/d9/d22/d28/f31 x:0 0 0 2024-04-05T21:26:35.827 INFO:tasks.workunit.client.1.smithi188.stdout:3/470: dwrite d5/dc/f3d [0,4194304] 0 2024-04-05T21:26:35.833 INFO:tasks.workunit.client.0.smithi121.stdout:3/131: dwrite d0/f2 [0,4194304] 0 2024-04-05T21:26:35.834 INFO:tasks.workunit.client.0.smithi121.stdout:7/136: readlink d0/l7 0 2024-04-05T21:26:35.835 INFO:tasks.workunit.client.1.smithi188.stdout:3/471: dwrite d5/d7/d6d/d75/d83/f8e [0,4194304] 0 2024-04-05T21:26:35.836 INFO:tasks.workunit.client.1.smithi188.stdout:3/472: stat d5/d2b/l8d 0 2024-04-05T21:26:35.837 INFO:tasks.workunit.client.1.smithi188.stdout:3/473: chown d5/d39/l46 118 1 2024-04-05T21:26:35.837 INFO:tasks.workunit.client.0.smithi121.stdout:3/132: fsync d0/f27 0 2024-04-05T21:26:35.839 INFO:tasks.workunit.client.1.smithi188.stdout:3/474: mkdir d5/d7/d65/da2 0 2024-04-05T21:26:35.840 INFO:tasks.workunit.client.0.smithi121.stdout:5/191: dwrite dd/de/f2e [0,4194304] 0 2024-04-05T21:26:35.840 INFO:tasks.workunit.client.0.smithi121.stdout:3/133: stat d0/d3/dc/d1a/f29 0 2024-04-05T21:26:35.843 INFO:tasks.workunit.client.0.smithi121.stdout:9/166: mknod d6/d32/c34 0 2024-04-05T21:26:35.845 INFO:tasks.workunit.client.0.smithi121.stdout:8/143: dwrite d12/d13/d15/d1d/f27 [0,4194304] 0 2024-04-05T21:26:35.845 INFO:tasks.workunit.client.0.smithi121.stdout:4/197: dwrite d5/f1d [0,4194304] 0 2024-04-05T21:26:35.849 INFO:tasks.workunit.client.1.smithi188.stdout:3/475: dwrite d5/dc/d2f/f4c [0,4194304] 0 2024-04-05T21:26:35.851 INFO:tasks.workunit.client.0.smithi121.stdout:9/167: unlink f4 0 2024-04-05T21:26:35.852 INFO:tasks.workunit.client.0.smithi121.stdout:0/260: rename d7/d1c/d1d/d45/f54 to d7/d1c/f55 0 2024-04-05T21:26:35.853 INFO:tasks.workunit.client.1.smithi188.stdout:3/476: unlink d5/d39/c4d 0 2024-04-05T21:26:35.853 INFO:tasks.workunit.client.0.smithi121.stdout:3/134: mkdir d0/d3/d24/d2b 0 2024-04-05T21:26:35.854 INFO:tasks.workunit.client.0.smithi121.stdout:7/137: rmdir d0/d23/d27 0 2024-04-05T21:26:35.858 INFO:tasks.workunit.client.0.smithi121.stdout:3/135: mknod d0/d3/c2c 0 2024-04-05T21:26:35.860 INFO:tasks.workunit.client.1.smithi188.stdout:3/477: mknod d5/d7/d65/d61/d66/d6e/ca3 0 2024-04-05T21:26:35.862 INFO:tasks.workunit.client.0.smithi121.stdout:5/192: dwrite f6 [0,4194304] 0 2024-04-05T21:26:35.862 INFO:tasks.workunit.client.0.smithi121.stdout:9/168: write d6/df/f20 [703035,61990] 0 2024-04-05T21:26:35.865 INFO:tasks.workunit.client.0.smithi121.stdout:3/136: fdatasync d0/f27 0 2024-04-05T21:26:35.869 INFO:tasks.workunit.client.0.smithi121.stdout:7/138: write d0/d23/f2e [490938,101879] 0 2024-04-05T21:26:35.870 INFO:tasks.workunit.client.0.smithi121.stdout:4/198: dwrite d5/d11/f14 [0,4194304] 0 2024-04-05T21:26:35.873 INFO:tasks.workunit.client.0.smithi121.stdout:7/139: dread - d0/f1f zero size 2024-04-05T21:26:35.877 INFO:tasks.workunit.client.0.smithi121.stdout:8/144: dwrite fc [0,4194304] 0 2024-04-05T21:26:35.877 INFO:tasks.workunit.client.0.smithi121.stdout:9/169: fdatasync d6/df/d1d/f2d 0 2024-04-05T21:26:35.878 INFO:tasks.workunit.client.0.smithi121.stdout:4/199: fsync d5/f16 0 2024-04-05T21:26:35.878 INFO:tasks.workunit.client.0.smithi121.stdout:3/137: creat d0/f2d x:0 0 0 2024-04-05T21:26:35.880 INFO:tasks.workunit.client.0.smithi121.stdout:0/261: dwrite d7/f18 [4194304,4194304] 0 2024-04-05T21:26:35.885 INFO:tasks.workunit.client.0.smithi121.stdout:9/170: write d6/df/f19 [4501327,1747] 0 2024-04-05T21:26:35.885 INFO:tasks.workunit.client.0.smithi121.stdout:7/140: truncate d0/d9/f24 4257881 0 2024-04-05T21:26:35.889 INFO:tasks.workunit.client.0.smithi121.stdout:9/171: write d6/df/f27 [1010367,56269] 0 2024-04-05T21:26:35.889 INFO:tasks.workunit.client.0.smithi121.stdout:3/138: rmdir d0/d3/d24/d2b 0 2024-04-05T21:26:35.890 INFO:tasks.workunit.client.0.smithi121.stdout:7/141: mkdir d0/d32 0 2024-04-05T21:26:35.891 INFO:tasks.workunit.client.0.smithi121.stdout:4/200: dread d5/d11/d28/f2f [0,4194304] 0 2024-04-05T21:26:35.892 INFO:tasks.workunit.client.0.smithi121.stdout:7/142: write d0/d9/f1d [462923,84788] 0 2024-04-05T21:26:35.893 INFO:tasks.workunit.client.0.smithi121.stdout:5/193: dread dd/de/d19/d13/f2d [0,4194304] 0 2024-04-05T21:26:35.893 INFO:tasks.workunit.client.0.smithi121.stdout:0/262: rename d7/d1c/d1d/d29/d2a/d2c/c2f to d7/d1c/d1d/d45/d46/c56 0 2024-04-05T21:26:35.893 INFO:tasks.workunit.client.0.smithi121.stdout:9/172: mknod d6/c35 0 2024-04-05T21:26:35.894 INFO:tasks.workunit.client.1.smithi188.stdout:2/494: sync 2024-04-05T21:26:35.894 INFO:tasks.workunit.client.1.smithi188.stdout:5/454: sync 2024-04-05T21:26:35.894 INFO:tasks.workunit.client.0.smithi121.stdout:3/139: mknod d0/d3/c2e 0 2024-04-05T21:26:35.895 INFO:tasks.workunit.client.0.smithi121.stdout:4/201: write d5/d11/f35 [1527025,124712] 0 2024-04-05T21:26:35.896 INFO:tasks.workunit.client.0.smithi121.stdout:0/263: dread - d7/d1c/d1d/d29/f37 zero size 2024-04-05T21:26:35.896 INFO:tasks.workunit.client.0.smithi121.stdout:7/143: creat d0/d23/f33 x:0 0 0 2024-04-05T21:26:35.896 INFO:tasks.workunit.client.0.smithi121.stdout:4/202: fsync d5/f3f 0 2024-04-05T21:26:35.897 INFO:tasks.workunit.client.1.smithi188.stdout:2/495: symlink d5/db/d35/d3b/l9e 0 2024-04-05T21:26:35.897 INFO:tasks.workunit.client.1.smithi188.stdout:3/478: dread d5/f7d [0,4194304] 0 2024-04-05T21:26:35.897 INFO:tasks.workunit.client.1.smithi188.stdout:5/455: mknod d6/ca7 0 2024-04-05T21:26:35.898 INFO:tasks.workunit.client.0.smithi121.stdout:3/140: mknod d0/d3/dc/d1a/c2f 0 2024-04-05T21:26:35.899 INFO:tasks.workunit.client.1.smithi188.stdout:0/298: sync 2024-04-05T21:26:35.899 INFO:tasks.workunit.client.0.smithi121.stdout:7/144: write d0/d9/f1d [304863,66477] 0 2024-04-05T21:26:35.899 INFO:tasks.workunit.client.0.smithi121.stdout:4/203: write d5/f1d [118852,104695] 0 2024-04-05T21:26:35.900 INFO:tasks.workunit.client.1.smithi188.stdout:3/479: mkdir d5/d7/d65/d61/d66/da4 0 2024-04-05T21:26:35.900 INFO:tasks.workunit.client.0.smithi121.stdout:0/264: rmdir d7/d1c/d1d/d29/d2a/d2c 39 2024-04-05T21:26:35.901 INFO:tasks.workunit.client.1.smithi188.stdout:2/496: getdents d5/db/dd/d36/d3d/d46/d5d 0 2024-04-05T21:26:35.901 INFO:tasks.workunit.client.1.smithi188.stdout:0/299: fdatasync d4/d8/db/d21/f34 0 2024-04-05T21:26:35.901 INFO:tasks.workunit.client.0.smithi121.stdout:3/141: read - d0/d3/dc/d1a/f29 zero size 2024-04-05T21:26:35.903 INFO:tasks.workunit.client.0.smithi121.stdout:7/145: write d0/f1f [348841,72696] 0 2024-04-05T21:26:35.906 INFO:tasks.workunit.client.0.smithi121.stdout:9/173: dwrite d6/df/d1d/f2d [0,4194304] 0 2024-04-05T21:26:35.906 INFO:tasks.workunit.client.0.smithi121.stdout:3/142: creat d0/d3/f30 x:0 0 0 2024-04-05T21:26:35.910 INFO:tasks.workunit.client.1.smithi188.stdout:0/300: readlink d4/d8/db/d21/l4e 0 2024-04-05T21:26:35.910 INFO:tasks.workunit.client.1.smithi188.stdout:2/497: mkdir d5/db/dd/d36/d3d/d46/d9f 0 2024-04-05T21:26:35.910 INFO:tasks.workunit.client.1.smithi188.stdout:5/456: dwrite d6/d15/d5c/d7a/f93 [0,4194304] 0 2024-04-05T21:26:35.910 INFO:tasks.workunit.client.1.smithi188.stdout:5/457: fsync d6/f5b 0 2024-04-05T21:26:35.910 INFO:tasks.workunit.client.1.smithi188.stdout:0/301: creat d4/d8/f75 x:0 0 0 2024-04-05T21:26:35.911 INFO:tasks.workunit.client.0.smithi121.stdout:0/265: creat d7/d1c/d1d/d45/d46/f57 x:0 0 0 2024-04-05T21:26:35.911 INFO:tasks.workunit.client.0.smithi121.stdout:7/146: rmdir d0/d2d 39 2024-04-05T21:26:35.911 INFO:tasks.workunit.client.0.smithi121.stdout:3/143: dread d0/d3/f13 [0,4194304] 0 2024-04-05T21:26:35.911 INFO:tasks.workunit.client.0.smithi121.stdout:4/204: mknod d5/d11/d26/d27/d29/c46 0 2024-04-05T21:26:35.911 INFO:tasks.workunit.client.0.smithi121.stdout:0/266: write d7/dc/f33 [187189,99170] 0 2024-04-05T21:26:35.911 INFO:tasks.workunit.client.1.smithi188.stdout:3/480: getdents d5/d7 0 2024-04-05T21:26:35.914 INFO:tasks.workunit.client.1.smithi188.stdout:3/481: chown d5/df/c4b 522 1 2024-04-05T21:26:35.914 INFO:tasks.workunit.client.1.smithi188.stdout:3/482: stat d5/d7/d6d/d75/d83 0 2024-04-05T21:26:35.915 INFO:tasks.workunit.client.0.smithi121.stdout:7/147: symlink d0/d32/l34 0 2024-04-05T21:26:35.915 INFO:tasks.workunit.client.1.smithi188.stdout:3/483: fsync d5/d7/d31/f76 0 2024-04-05T21:26:35.915 INFO:tasks.workunit.client.0.smithi121.stdout:3/144: truncate d0/d3/dc/f21 1387960 0 2024-04-05T21:26:35.916 INFO:tasks.workunit.client.1.smithi188.stdout:3/484: fsync d5/d7/d65/d5f/fa1 0 2024-04-05T21:26:35.918 INFO:tasks.workunit.client.1.smithi188.stdout:2/498: creat d5/db/dd/d36/d3d/d46/d9f/fa0 x:0 0 0 2024-04-05T21:26:35.920 INFO:tasks.workunit.client.0.smithi121.stdout:9/174: getdents d6/df/d1d 0 2024-04-05T21:26:35.921 INFO:tasks.workunit.client.1.smithi188.stdout:5/458: creat d6/fa8 x:0 0 0 2024-04-05T21:26:35.921 INFO:tasks.workunit.client.1.smithi188.stdout:0/302: creat d4/d8/db/f76 x:0 0 0 2024-04-05T21:26:35.922 INFO:tasks.workunit.client.1.smithi188.stdout:3/485: symlink d5/d7/d6d/d75/d82/la5 0 2024-04-05T21:26:35.922 INFO:tasks.workunit.client.0.smithi121.stdout:0/267: creat d7/d1c/d1d/d29/f58 x:0 0 0 2024-04-05T21:26:35.923 INFO:tasks.workunit.client.1.smithi188.stdout:0/303: chown d4/c33 876956 1 2024-04-05T21:26:35.924 INFO:tasks.workunit.client.1.smithi188.stdout:3/486: chown d5/d7/d65/f8c 78843 1 2024-04-05T21:26:35.924 INFO:tasks.workunit.client.0.smithi121.stdout:4/205: dwrite d5/d11/d28/f13 [0,4194304] 0 2024-04-05T21:26:35.927 INFO:tasks.workunit.client.1.smithi188.stdout:3/487: creat d5/dc/fa6 x:0 0 0 2024-04-05T21:26:35.930 INFO:tasks.workunit.client.1.smithi188.stdout:3/488: write d5/d35/d4e/fa0 [386802,42674] 0 2024-04-05T21:26:35.932 INFO:tasks.workunit.client.0.smithi121.stdout:8/145: truncate fa 3749088 0 2024-04-05T21:26:35.933 INFO:tasks.workunit.client.1.smithi188.stdout:0/304: dwrite d4/d6/f1f [0,4194304] 0 2024-04-05T21:26:35.934 INFO:tasks.workunit.client.0.smithi121.stdout:7/148: link d0/d23/f2e d0/d32/f35 0 2024-04-05T21:26:35.935 INFO:tasks.workunit.client.0.smithi121.stdout:7/149: chown d0/d23/f33 7032425 1 2024-04-05T21:26:35.936 INFO:tasks.workunit.client.1.smithi188.stdout:5/459: getdents d6 0 2024-04-05T21:26:35.936 INFO:tasks.workunit.client.0.smithi121.stdout:5/194: dread f4 [0,4194304] 0 2024-04-05T21:26:35.936 INFO:tasks.workunit.client.0.smithi121.stdout:3/145: rename d0/d3/d24/d25 to d0/d31 0 2024-04-05T21:26:35.937 INFO:tasks.workunit.client.1.smithi188.stdout:3/489: chown d5/d7/d65/d61/f92 73623524 1 2024-04-05T21:26:35.938 INFO:tasks.workunit.client.0.smithi121.stdout:7/150: read - d0/d23/f33 zero size 2024-04-05T21:26:35.939 INFO:tasks.workunit.client.1.smithi188.stdout:0/305: mkdir d4/d8/db/d42/d38/d77 0 2024-04-05T21:26:35.939 INFO:tasks.workunit.client.1.smithi188.stdout:2/499: truncate d5/db/dd/d36/d3c/f4a 3300597 0 2024-04-05T21:26:35.941 INFO:tasks.workunit.client.0.smithi121.stdout:8/146: creat d12/d13/d15/d1d/d29/f31 x:0 0 0 2024-04-05T21:26:35.941 INFO:tasks.workunit.client.0.smithi121.stdout:4/206: dwrite d5/d11/d28/f2f [0,4194304] 0 2024-04-05T21:26:35.941 INFO:tasks.workunit.client.1.smithi188.stdout:5/460: creat d6/d15/d5c/d7a/d44/d98/fa9 x:0 0 0 2024-04-05T21:26:35.943 INFO:tasks.workunit.client.1.smithi188.stdout:2/500: symlink d5/db/dd/d24/la1 0 2024-04-05T21:26:35.946 INFO:tasks.workunit.client.1.smithi188.stdout:2/501: unlink d5/db/dd/d36/d3d/d46/f65 0 2024-04-05T21:26:35.949 INFO:tasks.workunit.client.1.smithi188.stdout:2/502: creat d5/db/dd/d36/d3c/fa2 x:0 0 0 2024-04-05T21:26:35.950 INFO:tasks.workunit.client.0.smithi121.stdout:8/147: stat d12/d13/d15/l16 0 2024-04-05T21:26:35.953 INFO:tasks.workunit.client.1.smithi188.stdout:0/306: write d4/d8/db/f28 [116687,99773] 0 2024-04-05T21:26:35.954 INFO:tasks.workunit.client.1.smithi188.stdout:2/503: write d5/db/dd/d36/d3d/d46/d5d/f5e [876417,64320] 0 2024-04-05T21:26:35.954 INFO:tasks.workunit.client.0.smithi121.stdout:3/146: dread d0/d3/dc/d1a/d1e/f20 [0,4194304] 0 2024-04-05T21:26:35.954 INFO:tasks.workunit.client.0.smithi121.stdout:8/148: readlink d12/d13/d15/d1d/d29/l2c 0 2024-04-05T21:26:35.954 INFO:tasks.workunit.client.0.smithi121.stdout:3/147: stat d0 0 2024-04-05T21:26:35.955 INFO:tasks.workunit.client.1.smithi188.stdout:0/307: read d4/d8/db/d42/d38/f49 [87038,82545] 0 2024-04-05T21:26:35.956 INFO:tasks.workunit.client.0.smithi121.stdout:3/148: truncate d0/d3/f8 4392149 0 2024-04-05T21:26:35.956 INFO:tasks.workunit.client.0.smithi121.stdout:0/268: link d7/d1c/f55 d7/d1c/d1d/d29/d2a/d2c/f59 0 2024-04-05T21:26:35.956 INFO:tasks.workunit.client.0.smithi121.stdout:7/151: unlink d0/d9/d2b/l2c 0 2024-04-05T21:26:35.957 INFO:tasks.workunit.client.0.smithi121.stdout:4/207: mkdir d5/d11/d26/d47 0 2024-04-05T21:26:35.957 INFO:tasks.workunit.client.0.smithi121.stdout:0/269: write d7/f18 [2821358,40169] 0 2024-04-05T21:26:35.958 INFO:tasks.workunit.client.0.smithi121.stdout:3/149: write d0/d3/dc/d1a/f29 [459792,7321] 0 2024-04-05T21:26:35.960 INFO:tasks.workunit.client.0.smithi121.stdout:8/149: symlink d12/d30/l32 0 2024-04-05T21:26:35.963 INFO:tasks.workunit.client.0.smithi121.stdout:4/208: dread - d5/d11/d26/d27/d29/d2d/f31 zero size 2024-04-05T21:26:35.964 INFO:tasks.workunit.client.0.smithi121.stdout:0/270: mkdir d7/d1c/d1d/d45/d5a 0 2024-04-05T21:26:35.965 INFO:tasks.workunit.client.0.smithi121.stdout:3/150: symlink d0/d3/dc/d1a/l32 0 2024-04-05T21:26:35.965 INFO:tasks.workunit.client.0.smithi121.stdout:8/150: fsync d12/d13/d15/d1d/d29/f31 0 2024-04-05T21:26:35.966 INFO:tasks.workunit.client.1.smithi188.stdout:3/490: dread d5/f6 [0,4194304] 0 2024-04-05T21:26:35.968 INFO:tasks.workunit.client.0.smithi121.stdout:4/209: fdatasync d5/d11/f17 0 2024-04-05T21:26:35.968 INFO:tasks.workunit.client.1.smithi188.stdout:2/504: dwrite d5/db/dd/f54 [0,4194304] 0 2024-04-05T21:26:35.969 INFO:tasks.workunit.client.0.smithi121.stdout:9/175: readlink d6/l11 0 2024-04-05T21:26:35.970 INFO:tasks.workunit.client.0.smithi121.stdout:7/152: dwrite d0/d9/f20 [0,4194304] 0 2024-04-05T21:26:35.971 INFO:tasks.workunit.client.0.smithi121.stdout:0/271: rmdir d7/d1c/d1d/d29/d3f 39 2024-04-05T21:26:35.972 INFO:tasks.workunit.client.1.smithi188.stdout:0/308: dwrite d4/d8/f26 [0,4194304] 0 2024-04-05T21:26:35.972 INFO:tasks.workunit.client.1.smithi188.stdout:2/505: write d5/f9b [967608,68883] 0 2024-04-05T21:26:35.974 INFO:tasks.workunit.client.1.smithi188.stdout:0/309: chown d4/d8/db/d21/f44 1 1 2024-04-05T21:26:35.974 INFO:tasks.workunit.client.0.smithi121.stdout:0/272: write d7/f13 [3610858,98799] 0 2024-04-05T21:26:35.978 INFO:tasks.workunit.client.0.smithi121.stdout:3/151: rename d0/d3/dc/d1a/f29 to d0/d3/dc/d1a/f33 0 2024-04-05T21:26:35.979 INFO:tasks.workunit.client.1.smithi188.stdout:0/310: write d4/d8/f75 [88162,90847] 0 2024-04-05T21:26:35.981 INFO:tasks.workunit.client.1.smithi188.stdout:0/311: dread - d4/f5f zero size 2024-04-05T21:26:35.983 INFO:tasks.workunit.client.1.smithi188.stdout:0/312: chown d4/d8/f26 401926 1 2024-04-05T21:26:35.987 INFO:tasks.workunit.client.1.smithi188.stdout:2/506: dread d5/db/dd/f54 [0,4194304] 0 2024-04-05T21:26:35.987 INFO:tasks.workunit.client.1.smithi188.stdout:3/491: dwrite d5/dc/f1f [0,4194304] 0 2024-04-05T21:26:35.988 INFO:tasks.workunit.client.1.smithi188.stdout:2/507: fdatasync d5/db/dd/d24/f2e 0 2024-04-05T21:26:35.990 INFO:tasks.workunit.client.0.smithi121.stdout:3/152: dwrite d0/d3/dc/f21 [0,4194304] 0 2024-04-05T21:26:35.991 INFO:tasks.workunit.client.0.smithi121.stdout:3/153: readlink d0/l11 0 2024-04-05T21:26:35.993 INFO:tasks.workunit.client.1.smithi188.stdout:1/490: sync 2024-04-05T21:26:35.993 INFO:tasks.workunit.client.1.smithi188.stdout:6/523: sync 2024-04-05T21:26:35.993 INFO:tasks.workunit.client.1.smithi188.stdout:7/617: sync 2024-04-05T21:26:35.993 INFO:tasks.workunit.client.1.smithi188.stdout:4/724: sync 2024-04-05T21:26:35.993 INFO:tasks.workunit.client.1.smithi188.stdout:8/391: sync 2024-04-05T21:26:35.993 INFO:tasks.workunit.client.0.smithi121.stdout:3/154: chown d0/d3 227369 1 2024-04-05T21:26:35.994 INFO:tasks.workunit.client.0.smithi121.stdout:4/210: truncate d5/f3f 4533650 0 2024-04-05T21:26:35.995 INFO:tasks.workunit.client.1.smithi188.stdout:4/725: chown d0/d14/d15/d53/d82/dd5/fd6 20264 1 2024-04-05T21:26:35.995 INFO:tasks.workunit.client.1.smithi188.stdout:2/508: creat d5/fa3 x:0 0 0 2024-04-05T21:26:35.996 INFO:tasks.workunit.client.1.smithi188.stdout:1/491: truncate d1/dc/d16/f92 56608 0 2024-04-05T21:26:35.998 INFO:tasks.workunit.client.0.smithi121.stdout:0/273: rename d7/dc/f14 to d7/d1c/f5b 0 2024-04-05T21:26:35.999 INFO:tasks.workunit.client.0.smithi121.stdout:3/155: read d0/d3/f8 [822010,107906] 0 2024-04-05T21:26:36.000 INFO:tasks.workunit.client.1.smithi188.stdout:0/313: dread d4/d8/f26 [0,4194304] 0 2024-04-05T21:26:36.000 INFO:tasks.workunit.client.0.smithi121.stdout:3/156: read - d0/d3/f30 zero size 2024-04-05T21:26:36.001 INFO:tasks.workunit.client.1.smithi188.stdout:8/392: unlink d5/f3c 0 2024-04-05T21:26:36.001 INFO:tasks.workunit.client.0.smithi121.stdout:0/274: creat d7/dc/d10/f5c x:0 0 0 2024-04-05T21:26:36.002 INFO:tasks.workunit.client.0.smithi121.stdout:4/211: creat d5/d11/f48 x:0 0 0 2024-04-05T21:26:36.003 INFO:tasks.workunit.client.0.smithi121.stdout:0/275: stat d7/d1c/d1d/d45/d46 0 2024-04-05T21:26:36.003 INFO:tasks.workunit.client.0.smithi121.stdout:7/153: dread d0/d9/f12 [0,4194304] 0 2024-04-05T21:26:36.004 INFO:tasks.workunit.client.0.smithi121.stdout:3/157: unlink d0/l9 0 2024-04-05T21:26:36.004 INFO:tasks.workunit.client.0.smithi121.stdout:7/154: readlink d0/lb 0 2024-04-05T21:26:36.005 INFO:tasks.workunit.client.1.smithi188.stdout:2/509: read d5/db/dd/d36/d3d/d46/d5d/f5e [2481919,125903] 0 2024-04-05T21:26:36.006 INFO:tasks.workunit.client.0.smithi121.stdout:0/276: unlink d7/d1c/d1d/l27 0 2024-04-05T21:26:36.007 INFO:tasks.workunit.client.0.smithi121.stdout:0/277: chown d7/d1c/d1d/d45/c51 2 1 2024-04-05T21:26:36.007 INFO:tasks.workunit.client.1.smithi188.stdout:1/492: symlink d1/dc/d16/d46/d88/d13/d60/la5 0 2024-04-05T21:26:36.007 INFO:tasks.workunit.client.1.smithi188.stdout:2/510: write d5/db/dd/d36/d3c/d48/d7a/f99 [186655,72844] 0 2024-04-05T21:26:36.007 INFO:tasks.workunit.client.1.smithi188.stdout:6/524: rmdir d6/d7 39 2024-04-05T21:26:36.008 INFO:tasks.workunit.client.0.smithi121.stdout:4/212: mkdir d5/d11/d26/d49 0 2024-04-05T21:26:36.009 INFO:tasks.workunit.client.1.smithi188.stdout:8/393: rmdir d5/d21/d2a/d71 39 2024-04-05T21:26:36.010 INFO:tasks.workunit.client.0.smithi121.stdout:7/155: mknod d0/d23/c36 0 2024-04-05T21:26:36.011 INFO:tasks.workunit.client.1.smithi188.stdout:0/314: mkdir d4/d8/d78 0 2024-04-05T21:26:36.011 INFO:tasks.workunit.client.1.smithi188.stdout:3/492: write d5/f6b [4239250,33080] 0 2024-04-05T21:26:36.011 INFO:tasks.workunit.client.0.smithi121.stdout:0/278: symlink d7/dc/d10/d3c/l5d 0 2024-04-05T21:26:36.013 INFO:tasks.workunit.client.0.smithi121.stdout:4/213: chown d5/fd 16 1 2024-04-05T21:26:36.013 INFO:tasks.workunit.client.0.smithi121.stdout:0/279: dread d7/d1c/f5b [0,4194304] 0 2024-04-05T21:26:36.013 INFO:tasks.workunit.client.0.smithi121.stdout:8/151: truncate d12/d13/d15/d1d/f27 1971755 0 2024-04-05T21:26:36.015 INFO:tasks.workunit.client.0.smithi121.stdout:7/156: symlink d0/d9/d22/l37 0 2024-04-05T21:26:36.015 INFO:tasks.workunit.client.1.smithi188.stdout:7/618: write d3/d4/fb9 [561936,92556] 0 2024-04-05T21:26:36.016 INFO:tasks.workunit.client.1.smithi188.stdout:8/394: mkdir d5/dc/d8f 0 2024-04-05T21:26:36.018 INFO:tasks.workunit.client.1.smithi188.stdout:7/619: read - d3/d4/d1d/d74/f86 zero size 2024-04-05T21:26:36.019 INFO:tasks.workunit.client.1.smithi188.stdout:4/726: getdents d0/d2/d52/dc1 0 2024-04-05T21:26:36.020 INFO:tasks.workunit.client.1.smithi188.stdout:6/525: write d6/d13/d18/d19/d2c/f65 [1910489,27382] 0 2024-04-05T21:26:36.020 INFO:tasks.workunit.client.0.smithi121.stdout:0/280: symlink d7/dc/d10/d3c/l5e 0 2024-04-05T21:26:36.021 INFO:tasks.workunit.client.0.smithi121.stdout:4/214: rename d5/d11/f22 to d5/d11/d28/d3b/f4a 0 2024-04-05T21:26:36.023 INFO:tasks.workunit.client.0.smithi121.stdout:7/157: mknod d0/d9/d22/c38 0 2024-04-05T21:26:36.023 INFO:tasks.workunit.client.1.smithi188.stdout:2/511: mkdir d5/db/dd/da4 0 2024-04-05T21:26:36.024 INFO:tasks.workunit.client.1.smithi188.stdout:8/395: symlink d5/d21/d40/d43/d44/d59/l90 0 2024-04-05T21:26:36.024 INFO:tasks.workunit.client.1.smithi188.stdout:1/493: write d1/d7/f39 [747783,17933] 0 2024-04-05T21:26:36.024 INFO:tasks.workunit.client.0.smithi121.stdout:8/152: truncate fa 1902756 0 2024-04-05T21:26:36.025 INFO:tasks.workunit.client.0.smithi121.stdout:3/158: getdents d0/d3/dc/d1a 0 2024-04-05T21:26:36.027 INFO:tasks.workunit.client.1.smithi188.stdout:2/512: write d5/db/f76 [6845742,120108] 0 2024-04-05T21:26:36.027 INFO:tasks.workunit.client.0.smithi121.stdout:9/176: dwrite d6/f16 [0,4194304] 0 2024-04-05T21:26:36.028 INFO:tasks.workunit.client.0.smithi121.stdout:0/281: dread - d7/d1c/d1d/d29/d3f/f4f zero size 2024-04-05T21:26:36.030 INFO:tasks.workunit.client.0.smithi121.stdout:4/215: rmdir d5/d11/d26/d27 39 2024-04-05T21:26:36.031 INFO:tasks.workunit.client.1.smithi188.stdout:1/494: truncate d1/d7/f80 898416 0 2024-04-05T21:26:36.031 INFO:tasks.workunit.client.1.smithi188.stdout:1/495: write d1/dc/d16/d21/f26 [1599561,61475] 0 2024-04-05T21:26:36.031 INFO:tasks.workunit.client.1.smithi188.stdout:6/526: symlink d6/d13/d48/d49/d75/la1 0 2024-04-05T21:26:36.032 INFO:tasks.workunit.client.0.smithi121.stdout:7/158: creat d0/d9/f39 x:0 0 0 2024-04-05T21:26:36.033 INFO:tasks.workunit.client.1.smithi188.stdout:0/315: truncate d4/d8/d10/f1b 2752007 0 2024-04-05T21:26:36.034 INFO:tasks.workunit.client.0.smithi121.stdout:8/153: rmdir d12/d13 39 2024-04-05T21:26:36.034 INFO:tasks.workunit.client.0.smithi121.stdout:8/154: write f3 [26523,71315] 0 2024-04-05T21:26:36.035 INFO:tasks.workunit.client.0.smithi121.stdout:9/177: fdatasync d6/df/f20 0 2024-04-05T21:26:36.037 INFO:tasks.workunit.client.0.smithi121.stdout:0/282: fdatasync d7/d1c/d1d/d29/d3f/f4f 0 2024-04-05T21:26:36.038 INFO:tasks.workunit.client.0.smithi121.stdout:7/159: creat d0/d9/f3a x:0 0 0 2024-04-05T21:26:36.039 INFO:tasks.workunit.client.1.smithi188.stdout:3/493: dwrite d5/d7/d65/d61/f92 [0,4194304] 0 2024-04-05T21:26:36.039 INFO:tasks.workunit.client.0.smithi121.stdout:7/160: chown d0/d32 315 1 2024-04-05T21:26:36.044 INFO:tasks.workunit.client.1.smithi188.stdout:7/620: rename d3/d4/db/d2e/d37/d65/f7d to d3/d4/d1d/d74/d88/fc0 0 2024-04-05T21:26:36.044 INFO:tasks.workunit.client.1.smithi188.stdout:2/513: readlink d5/db/dd/d36/d3c/d63/l68 0 2024-04-05T21:26:36.046 INFO:tasks.workunit.client.0.smithi121.stdout:1/184: sync 2024-04-05T21:26:36.046 INFO:tasks.workunit.client.1.smithi188.stdout:4/727: creat d0/f102 x:0 0 0 2024-04-05T21:26:36.047 INFO:tasks.workunit.client.0.smithi121.stdout:4/216: mknod d5/d11/d26/d27/d29/d2d/c4b 0 2024-04-05T21:26:36.049 INFO:tasks.workunit.client.1.smithi188.stdout:8/396: rename d5/d21/f67 to d5/d21/f91 0 2024-04-05T21:26:36.050 INFO:tasks.workunit.client.1.smithi188.stdout:0/316: rmdir d4/d6/d46 39 2024-04-05T21:26:36.051 INFO:tasks.workunit.client.0.smithi121.stdout:7/161: mkdir d0/d9/d22/d28/d3b 0 2024-04-05T21:26:36.054 INFO:tasks.workunit.client.1.smithi188.stdout:0/317: fdatasync d4/d8/db/f28 0 2024-04-05T21:26:36.054 INFO:tasks.workunit.client.1.smithi188.stdout:6/527: chown d6/d7/fd 30770371 1 2024-04-05T21:26:36.055 INFO:tasks.workunit.client.1.smithi188.stdout:3/494: dwrite d5/d7/d65/d61/f94 [0,4194304] 0 2024-04-05T21:26:36.058 INFO:tasks.workunit.client.0.smithi121.stdout:1/185: creat d8/d10/d14/d1a/d1b/d24/f39 x:0 0 0 2024-04-05T21:26:36.059 INFO:tasks.workunit.client.1.smithi188.stdout:2/514: rename d5/db/d35/l6a to d5/db/d35/la5 0 2024-04-05T21:26:36.059 INFO:tasks.workunit.client.0.smithi121.stdout:9/178: rename d6/df/d1d/d26 to d6/d36 0 2024-04-05T21:26:36.061 INFO:tasks.workunit.client.0.smithi121.stdout:0/283: creat d7/d1c/d1d/f5f x:0 0 0 2024-04-05T21:26:36.063 INFO:tasks.workunit.client.1.smithi188.stdout:2/515: read d5/f5c [466382,46731] 0 2024-04-05T21:26:36.064 INFO:tasks.workunit.client.1.smithi188.stdout:6/528: dread - d6/d13/d18/d94/fa0 zero size 2024-04-05T21:26:36.065 INFO:tasks.workunit.client.0.smithi121.stdout:0/284: stat d7/d1c/d1d/d45/d5a 0 2024-04-05T21:26:36.065 INFO:tasks.workunit.client.1.smithi188.stdout:1/496: rename d1/dc/d16/l2a to d1/dc/d16/d46/d88/d13/la6 0 2024-04-05T21:26:36.067 INFO:tasks.workunit.client.0.smithi121.stdout:1/186: rename d8/d10/d14/d1a/d1b/l33 to d8/d10/d14/d1a/l3a 0 2024-04-05T21:26:36.067 INFO:tasks.workunit.client.1.smithi188.stdout:6/529: read - d6/d28/f8f zero size 2024-04-05T21:26:36.068 INFO:tasks.workunit.client.1.smithi188.stdout:2/516: chown d5/db/dd/d36/d3d 1801 1 2024-04-05T21:26:36.068 INFO:tasks.workunit.client.1.smithi188.stdout:4/728: write d0/d2/d17/d48/d70/fcb [503856,19935] 0 2024-04-05T21:26:36.069 INFO:tasks.workunit.client.0.smithi121.stdout:9/179: dwrite d6/df/d1d/f2d [0,4194304] 0 2024-04-05T21:26:36.070 INFO:tasks.workunit.client.1.smithi188.stdout:8/397: rename d5/d21/d40/d43/d44/c7f to d5/d21/d40/c92 0 2024-04-05T21:26:36.071 INFO:tasks.workunit.client.0.smithi121.stdout:9/180: stat d6/df/d1d/l2f 0 2024-04-05T21:26:36.071 INFO:tasks.workunit.client.0.smithi121.stdout:7/162: dread d0/d2d/f2f [0,4194304] 0 2024-04-05T21:26:36.071 INFO:tasks.workunit.client.1.smithi188.stdout:7/621: truncate d3/d4/db/fe 4340621 0 2024-04-05T21:26:36.072 INFO:tasks.workunit.client.0.smithi121.stdout:9/181: dread - d6/df/d1d/d2c/f24 zero size 2024-04-05T21:26:36.073 INFO:tasks.workunit.client.0.smithi121.stdout:9/182: chown d6/d36 4109198 1 2024-04-05T21:26:36.074 INFO:tasks.workunit.client.0.smithi121.stdout:0/285: creat d7/d1c/d1d/d45/f60 x:0 0 0 2024-04-05T21:26:36.076 INFO:tasks.workunit.client.1.smithi188.stdout:0/318: mkdir d4/d8/db/d21/d4a/d73/d79 0 2024-04-05T21:26:36.076 INFO:tasks.workunit.client.0.smithi121.stdout:0/286: read d7/d1c/f1f [904893,106114] 0 2024-04-05T21:26:36.077 INFO:tasks.workunit.client.0.smithi121.stdout:7/163: dread d0/d9/f20 [0,4194304] 0 2024-04-05T21:26:36.079 INFO:tasks.workunit.client.0.smithi121.stdout:1/187: mkdir d8/d10/d14/d27/d2d/d3b 0 2024-04-05T21:26:36.081 INFO:tasks.workunit.client.0.smithi121.stdout:1/188: write d8/d10/d14/f2c [129393,21232] 0 2024-04-05T21:26:36.081 INFO:tasks.workunit.client.1.smithi188.stdout:4/729: mknod d0/d2/d2a/d34/d6b/c103 0 2024-04-05T21:26:36.084 INFO:tasks.workunit.client.0.smithi121.stdout:7/164: creat d0/d9/d2b/f3c x:0 0 0 2024-04-05T21:26:36.086 INFO:tasks.workunit.client.1.smithi188.stdout:0/319: dwrite d4/d6/d13/d23/f51 [0,4194304] 0 2024-04-05T21:26:36.086 INFO:tasks.workunit.client.0.smithi121.stdout:4/217: dread d5/f3f [0,4194304] 0 2024-04-05T21:26:36.086 INFO:tasks.workunit.client.0.smithi121.stdout:8/155: chown d12/d13/d15/d1d/f27 24798 1 2024-04-05T21:26:36.087 INFO:tasks.workunit.client.1.smithi188.stdout:2/517: symlink d5/db/dd/d36/la6 0 2024-04-05T21:26:36.087 INFO:tasks.workunit.client.1.smithi188.stdout:1/497: stat d1/dc/d16/d46/d88/d13/d60/l98 0 2024-04-05T21:26:36.089 INFO:tasks.workunit.client.0.smithi121.stdout:7/165: symlink d0/d9/d22/d28/l3d 0 2024-04-05T21:26:36.091 INFO:tasks.workunit.client.0.smithi121.stdout:4/218: rmdir d5 39 2024-04-05T21:26:36.093 INFO:tasks.workunit.client.0.smithi121.stdout:0/287: rename d7/d1c/d1d/d45/c51 to d7/d1c/d1d/c61 0 2024-04-05T21:26:36.093 INFO:tasks.workunit.client.0.smithi121.stdout:9/183: getdents d6/d36 0 2024-04-05T21:26:36.094 INFO:tasks.workunit.client.0.smithi121.stdout:7/166: write d0/f6 [1462191,99266] 0 2024-04-05T21:26:36.095 INFO:tasks.workunit.client.0.smithi121.stdout:0/288: read - d7/d1c/d1d/d29/d2a/d2c/f43 zero size 2024-04-05T21:26:36.096 INFO:tasks.workunit.client.1.smithi188.stdout:7/622: link d3/d4/l15 d3/d4/db/d2e/d3b/d94/lc1 0 2024-04-05T21:26:36.097 INFO:tasks.workunit.client.0.smithi121.stdout:4/219: chown d5/d11/d28/l25 684210 1 2024-04-05T21:26:36.099 INFO:tasks.workunit.client.0.smithi121.stdout:8/156: rename d12/d13/c22 to d12/d13/d15/d1d/c33 0 2024-04-05T21:26:36.099 INFO:tasks.workunit.client.1.smithi188.stdout:4/730: unlink d0/d2/d2a/d71/c8d 0 2024-04-05T21:26:36.101 INFO:tasks.workunit.client.1.smithi188.stdout:7/623: truncate d3/d4/db/d2e/d49/f67 28327 0 2024-04-05T21:26:36.102 INFO:tasks.workunit.client.1.smithi188.stdout:6/530: creat d6/d13/d48/fa2 x:0 0 0 2024-04-05T21:26:36.102 INFO:tasks.workunit.client.1.smithi188.stdout:1/498: dwrite d1/dc/d16/f92 [0,4194304] 0 2024-04-05T21:26:36.105 INFO:tasks.workunit.client.1.smithi188.stdout:8/398: getdents d5/d6e 0 2024-04-05T21:26:36.105 INFO:tasks.workunit.client.0.smithi121.stdout:0/289: rename d7/dc/d10 to d7/d62 0 2024-04-05T21:26:36.106 INFO:tasks.workunit.client.1.smithi188.stdout:0/320: symlink d4/d8/db/d21/d4a/d73/d79/l7a 0 2024-04-05T21:26:36.107 INFO:tasks.workunit.client.1.smithi188.stdout:4/731: readlink d0/d2/d17/l8c 0 2024-04-05T21:26:36.108 INFO:tasks.workunit.client.1.smithi188.stdout:3/495: dread d5/d7/d31/f5c [4194304,4194304] 0 2024-04-05T21:26:36.113 INFO:tasks.workunit.client.0.smithi121.stdout:7/167: dwrite d0/d23/f33 [0,4194304] 0 2024-04-05T21:26:36.114 INFO:tasks.workunit.client.1.smithi188.stdout:3/496: chown d5/d39 111689163 1 2024-04-05T21:26:36.114 INFO:tasks.workunit.client.1.smithi188.stdout:7/624: read d3/d4/db/d2e/d3b/f4d [4766,78030] 0 2024-04-05T21:26:36.115 INFO:tasks.workunit.client.1.smithi188.stdout:4/732: chown d0/d2/d17/d48/d70/fda 0 1 2024-04-05T21:26:36.116 INFO:tasks.workunit.client.0.smithi121.stdout:9/184: truncate d6/f15 835805 0 2024-04-05T21:26:36.117 INFO:tasks.workunit.client.0.smithi121.stdout:4/220: dwrite d5/d11/f12 [4194304,4194304] 0 2024-04-05T21:26:36.117 INFO:tasks.workunit.client.0.smithi121.stdout:7/168: write d0/f1f [80068,59009] 0 2024-04-05T21:26:36.119 INFO:tasks.workunit.client.1.smithi188.stdout:6/531: truncate d6/d13/d38/f47 2463031 0 2024-04-05T21:26:36.120 INFO:tasks.workunit.client.1.smithi188.stdout:7/625: write d3/f71 [1048594,33896] 0 2024-04-05T21:26:36.120 INFO:tasks.workunit.client.1.smithi188.stdout:4/733: chown d0/d2/d17/d3a/l4a 13 1 2024-04-05T21:26:36.120 INFO:tasks.workunit.client.0.smithi121.stdout:8/157: dwrite d12/d13/f20 [0,4194304] 0 2024-04-05T21:26:36.121 INFO:tasks.workunit.client.1.smithi188.stdout:2/518: dread d5/db/dd/d24/f52 [0,4194304] 0 2024-04-05T21:26:36.121 INFO:tasks.workunit.client.0.smithi121.stdout:0/290: truncate d7/d1c/d1d/d29/d3f/f4f 883378 0 2024-04-05T21:26:36.122 INFO:tasks.workunit.client.0.smithi121.stdout:1/189: dread d8/fb [0,4194304] 0 2024-04-05T21:26:36.123 INFO:tasks.workunit.client.1.smithi188.stdout:0/321: write d4/d8/fa [2238053,12773] 0 2024-04-05T21:26:36.124 INFO:tasks.workunit.client.0.smithi121.stdout:9/185: mkdir d6/df/d1d/d2c/d37 0 2024-04-05T21:26:36.125 INFO:tasks.workunit.client.1.smithi188.stdout:2/519: write d5/db/f76 [6254480,61500] 0 2024-04-05T21:26:36.125 INFO:tasks.workunit.client.1.smithi188.stdout:8/399: dread d5/d21/d40/d43/f83 [0,4194304] 0 2024-04-05T21:26:36.127 INFO:tasks.workunit.client.1.smithi188.stdout:4/734: fdatasync d0/d14/d2f/f98 0 2024-04-05T21:26:36.128 INFO:tasks.workunit.client.0.smithi121.stdout:8/158: mkdir d12/d34 0 2024-04-05T21:26:36.129 INFO:tasks.workunit.client.1.smithi188.stdout:6/532: mkdir d6/d13/d18/d19/da3 0 2024-04-05T21:26:36.130 INFO:tasks.workunit.client.1.smithi188.stdout:3/497: link d5/d7/d65/d5f/l87 d5/d39/d84/la7 0 2024-04-05T21:26:36.131 INFO:tasks.workunit.client.1.smithi188.stdout:8/400: write d5/d21/d40/f7b [3132780,24661] 0 2024-04-05T21:26:36.132 INFO:tasks.workunit.client.0.smithi121.stdout:7/169: dwrite d0/d9/f12 [4194304,4194304] 0 2024-04-05T21:26:36.132 INFO:tasks.workunit.client.0.smithi121.stdout:4/221: dread d5/d11/f1e [0,4194304] 0 2024-04-05T21:26:36.132 INFO:tasks.workunit.client.0.smithi121.stdout:9/186: rmdir d6/df/d1d/d2c 39 2024-04-05T21:26:36.134 INFO:tasks.workunit.client.0.smithi121.stdout:7/170: readlink d0/d9/l19 0 2024-04-05T21:26:36.134 INFO:tasks.workunit.client.0.smithi121.stdout:0/291: rename d7/d62/c16 to d7/d1c/d1d/d45/d5a/c63 0 2024-04-05T21:26:36.134 INFO:tasks.workunit.client.0.smithi121.stdout:8/159: truncate d12/d13/d15/f18 257853 0 2024-04-05T21:26:36.139 INFO:tasks.workunit.client.0.smithi121.stdout:0/292: truncate d7/d1c/d1d/d29/f37 183458 0 2024-04-05T21:26:36.141 INFO:tasks.workunit.client.0.smithi121.stdout:8/160: rename d12/d13/c2b to d12/c35 0 2024-04-05T21:26:36.145 INFO:tasks.workunit.client.1.smithi188.stdout:7/626: rename d3/d4/d1d/f76 to d3/d4/db/d2e/d37/fc2 0 2024-04-05T21:26:36.145 INFO:tasks.workunit.client.1.smithi188.stdout:2/520: creat d5/db/d35/d3b/d6b/fa7 x:0 0 0 2024-04-05T21:26:36.146 INFO:tasks.workunit.client.0.smithi121.stdout:7/171: dwrite d0/f15 [4194304,4194304] 0 2024-04-05T21:26:36.146 INFO:tasks.workunit.client.0.smithi121.stdout:0/293: mkdir d7/d1c/d1d/d29/d2a/d64 0 2024-04-05T21:26:36.146 INFO:tasks.workunit.client.0.smithi121.stdout:9/187: dwrite d6/df/f19 [0,4194304] 0 2024-04-05T21:26:36.147 INFO:tasks.workunit.client.1.smithi188.stdout:8/401: creat d5/d21/d40/d43/d48/d7e/f93 x:0 0 0 2024-04-05T21:26:36.149 INFO:tasks.workunit.client.0.smithi121.stdout:8/161: rmdir d12/d34 0 2024-04-05T21:26:36.150 INFO:tasks.workunit.client.1.smithi188.stdout:3/498: dread d5/d7/d31/f5c [4194304,4194304] 0 2024-04-05T21:26:36.150 INFO:tasks.workunit.client.0.smithi121.stdout:0/294: read d7/f3a [956369,126278] 0 2024-04-05T21:26:36.151 INFO:tasks.workunit.client.0.smithi121.stdout:0/295: write d7/f13 [3779574,23447] 0 2024-04-05T21:26:36.152 INFO:tasks.workunit.client.0.smithi121.stdout:7/172: rename d0/d9/d2b to d0/d9/d22/d3e 0 2024-04-05T21:26:36.153 INFO:tasks.workunit.client.1.smithi188.stdout:4/735: mkdir d0/d2/dc7/d104 0 2024-04-05T21:26:36.154 INFO:tasks.workunit.client.0.smithi121.stdout:8/162: creat d12/d13/d15/d1d/f36 x:0 0 0 2024-04-05T21:26:36.154 INFO:tasks.workunit.client.1.smithi188.stdout:0/322: dwrite d4/f27 [0,4194304] 0 2024-04-05T21:26:36.155 INFO:tasks.workunit.client.1.smithi188.stdout:6/533: getdents d6/d7 0 2024-04-05T21:26:36.158 INFO:tasks.workunit.client.1.smithi188.stdout:3/499: readlink d5/d7/d6d/l7c 0 2024-04-05T21:26:36.158 INFO:tasks.workunit.client.1.smithi188.stdout:7/627: read - d3/d2c/d5e/f78 zero size 2024-04-05T21:26:36.160 INFO:tasks.workunit.client.0.smithi121.stdout:1/190: write d8/d10/d14/d27/f29 [1447989,96754] 0 2024-04-05T21:26:36.160 INFO:tasks.workunit.client.1.smithi188.stdout:6/534: fsync d6/d13/d18/d19/d2c/f51 0 2024-04-05T21:26:36.162 INFO:tasks.workunit.client.0.smithi121.stdout:9/188: rename d6/c35 to d6/d32/c38 0 2024-04-05T21:26:36.163 INFO:tasks.workunit.client.0.smithi121.stdout:1/191: write d8/ff [477681,48134] 0 2024-04-05T21:26:36.165 INFO:tasks.workunit.client.0.smithi121.stdout:0/296: dwrite d7/d1c/d1d/d29/d3f/f50 [0,4194304] 0 2024-04-05T21:26:36.168 INFO:tasks.workunit.client.1.smithi188.stdout:2/521: symlink d5/db/dd/da4/la8 0 2024-04-05T21:26:36.169 INFO:tasks.workunit.client.1.smithi188.stdout:4/736: rmdir d0/d14/d15/d53/d82/d8f/dbd 39 2024-04-05T21:26:36.169 INFO:tasks.workunit.client.0.smithi121.stdout:9/189: rmdir d6/d32 39 2024-04-05T21:26:36.171 INFO:tasks.workunit.client.1.smithi188.stdout:2/522: readlink d5/db/dd/d36/d3c/d63/l68 0 2024-04-05T21:26:36.172 INFO:tasks.workunit.client.1.smithi188.stdout:6/535: rmdir d6 39 2024-04-05T21:26:36.174 INFO:tasks.workunit.client.0.smithi121.stdout:8/163: truncate d12/d13/d15/f1f 2508913 0 2024-04-05T21:26:36.174 INFO:tasks.workunit.client.0.smithi121.stdout:0/297: rmdir d7/d62/d3c 39 2024-04-05T21:26:36.175 INFO:tasks.workunit.client.1.smithi188.stdout:3/500: creat d5/d2b/fa8 x:0 0 0 2024-04-05T21:26:36.175 INFO:tasks.workunit.client.1.smithi188.stdout:0/323: write d4/d6/d13/f14 [622040,26161] 0 2024-04-05T21:26:36.176 INFO:tasks.workunit.client.0.smithi121.stdout:9/190: symlink d6/d36/l39 0 2024-04-05T21:26:36.176 INFO:tasks.workunit.client.1.smithi188.stdout:3/501: fdatasync d5/f9b 0 2024-04-05T21:26:36.176 INFO:tasks.workunit.client.1.smithi188.stdout:9/476: sync 2024-04-05T21:26:36.177 INFO:tasks.workunit.client.0.smithi121.stdout:0/298: write d7/dc/f2d [2366642,98474] 0 2024-04-05T21:26:36.177 INFO:tasks.workunit.client.0.smithi121.stdout:0/299: chown d7/d62/f11 4575874 1 2024-04-05T21:26:36.179 INFO:tasks.workunit.client.0.smithi121.stdout:1/192: rmdir d8/d10/d14/d1a/d1b/d24/d34 0 2024-04-05T21:26:36.180 INFO:tasks.workunit.client.0.smithi121.stdout:0/300: readlink d7/d1c/d1d/d29/d2a/d2c/l48 0 2024-04-05T21:26:36.180 INFO:tasks.workunit.client.1.smithi188.stdout:7/628: rmdir d3/d4/db/d2e/d8c 39 2024-04-05T21:26:36.181 INFO:tasks.workunit.client.1.smithi188.stdout:2/523: symlink d5/la9 0 2024-04-05T21:26:36.181 INFO:tasks.workunit.client.1.smithi188.stdout:2/524: write d5/f2f [6140781,60289] 0 2024-04-05T21:26:36.181 INFO:tasks.workunit.client.0.smithi121.stdout:9/191: mkdir d6/df/d1d/d2c/d37/d3a 0 2024-04-05T21:26:36.182 INFO:tasks.workunit.client.0.smithi121.stdout:1/193: mknod d8/d10/d14/d1a/d1b/d24/c3c 0 2024-04-05T21:26:36.183 INFO:tasks.workunit.client.0.smithi121.stdout:0/301: truncate d7/f3a 326266 0 2024-04-05T21:26:36.183 INFO:tasks.workunit.client.0.smithi121.stdout:1/194: chown d8/d10/d14/f1e 472 1 2024-04-05T21:26:36.184 INFO:tasks.workunit.client.1.smithi188.stdout:2/525: creat d5/db/dd/d36/d3d/d46/faa x:0 0 0 2024-04-05T21:26:36.185 INFO:tasks.workunit.client.1.smithi188.stdout:0/324: mknod d4/d6/d13/d23/c7b 0 2024-04-05T21:26:36.188 INFO:tasks.workunit.client.0.smithi121.stdout:1/195: rename d8/d10/d14/d1a/f32 to d8/d10/d14/d27/d2d/d3b/f3d 0 2024-04-05T21:26:36.189 INFO:tasks.workunit.client.1.smithi188.stdout:7/629: symlink d3/d4/db/d2e/lc3 0 2024-04-05T21:26:36.189 INFO:tasks.workunit.client.1.smithi188.stdout:3/502: creat d5/dc/d2f/fa9 x:0 0 0 2024-04-05T21:26:36.190 INFO:tasks.workunit.client.1.smithi188.stdout:2/526: creat d5/db/d35/fab x:0 0 0 2024-04-05T21:26:36.193 INFO:tasks.workunit.client.0.smithi121.stdout:1/196: dwrite d8/f1d [0,4194304] 0 2024-04-05T21:26:36.195 INFO:tasks.workunit.client.0.smithi121.stdout:7/173: truncate d0/d9/f20 670962 0 2024-04-05T21:26:36.197 INFO:tasks.workunit.client.1.smithi188.stdout:3/503: write d5/d7/d31/f91 [1259382,22203] 0 2024-04-05T21:26:36.197 INFO:tasks.workunit.client.1.smithi188.stdout:6/536: dwrite d6/d13/d18/f82 [0,4194304] 0 2024-04-05T21:26:36.198 INFO:tasks.workunit.client.0.smithi121.stdout:7/174: read d0/d23/f2e [5591,103156] 0 2024-04-05T21:26:36.203 INFO:tasks.workunit.client.1.smithi188.stdout:0/325: dread d4/d8/db/d21/d4a/f58 [0,4194304] 0 2024-04-05T21:26:36.207 INFO:tasks.workunit.client.1.smithi188.stdout:7/630: dwrite d3/d2c/f3f [0,4194304] 0 2024-04-05T21:26:36.227 INFO:tasks.workunit.client.1.smithi188.stdout:0/326: dwrite d4/d6/d13/d23/f52 [0,4194304] 0 2024-04-05T21:26:36.227 INFO:tasks.workunit.client.1.smithi188.stdout:2/527: dwrite d5/db/dd/d36/d3d/d46/f81 [0,4194304] 0 2024-04-05T21:26:36.237 INFO:tasks.workunit.client.1.smithi188.stdout:6/537: mkdir d6/d13/d48/d49/d75/da4 0 2024-04-05T21:26:36.248 INFO:tasks.workunit.client.1.smithi188.stdout:7/631: fdatasync d3/d2c/f3f 0 2024-04-05T21:26:36.248 INFO:tasks.workunit.client.1.smithi188.stdout:2/528: mknod d5/db/dd/d24/cac 0 2024-04-05T21:26:36.251 INFO:tasks.workunit.client.1.smithi188.stdout:6/538: creat d6/d13/d18/d19/d61/fa5 x:0 0 0 2024-04-05T21:26:36.258 INFO:tasks.workunit.client.1.smithi188.stdout:7/632: mkdir d3/d4/db/d2e/d8c/db7/dc4 0 2024-04-05T21:26:36.260 INFO:tasks.workunit.client.1.smithi188.stdout:2/529: fsync d5/f22 0 2024-04-05T21:26:36.261 INFO:tasks.workunit.client.1.smithi188.stdout:3/504: getdents d5/d7/d6d/d75/d82 0 2024-04-05T21:26:36.261 INFO:tasks.workunit.client.1.smithi188.stdout:7/633: dread d3/d4/db/d2e/d49/f67 [0,4194304] 0 2024-04-05T21:26:36.262 INFO:tasks.workunit.client.1.smithi188.stdout:3/505: chown d5/d7/f20 21 1 2024-04-05T21:26:36.265 INFO:tasks.workunit.client.1.smithi188.stdout:2/530: rmdir d5/db/d35 39 2024-04-05T21:26:36.268 INFO:tasks.workunit.client.1.smithi188.stdout:7/634: rmdir d3/d4/db/d2e/d3b 39 2024-04-05T21:26:36.272 INFO:tasks.workunit.client.1.smithi188.stdout:3/506: dwrite d5/d7/d65/d61/f94 [4194304,4194304] 0 2024-04-05T21:26:36.278 INFO:tasks.workunit.client.0.smithi121.stdout:6/148: sync 2024-04-05T21:26:36.279 INFO:tasks.workunit.client.0.smithi121.stdout:1/197: symlink d8/d10/l3e 0 2024-04-05T21:26:36.281 INFO:tasks.workunit.client.0.smithi121.stdout:2/173: sync 2024-04-05T21:26:36.281 INFO:tasks.workunit.client.1.smithi188.stdout:7/635: dwrite d3/d4/d1d/d74/f80 [0,4194304] 0 2024-04-05T21:26:36.282 INFO:tasks.workunit.client.1.smithi188.stdout:3/507: rename d5/df/f71 to d5/d7/d6d/d75/faa 0 2024-04-05T21:26:36.292 INFO:tasks.workunit.client.0.smithi121.stdout:0/302: truncate f2 1327089 0 2024-04-05T21:26:36.292 INFO:tasks.workunit.client.1.smithi188.stdout:3/508: fdatasync d5/d7/f5e 0 2024-04-05T21:26:36.293 INFO:tasks.workunit.client.1.smithi188.stdout:2/531: dread d5/db/f76 [4194304,4194304] 0 2024-04-05T21:26:36.296 INFO:tasks.workunit.client.0.smithi121.stdout:7/175: link d0/d9/f24 d0/d2d/f3f 0 2024-04-05T21:26:36.299 INFO:tasks.workunit.client.0.smithi121.stdout:1/198: symlink d8/d10/d14/d1a/d1b/l3f 0 2024-04-05T21:26:36.300 INFO:tasks.workunit.client.1.smithi188.stdout:7/636: rename d3/d4/db/f1c to d3/d4/db/d2e/d3b/d94/fc5 0 2024-04-05T21:26:36.300 INFO:tasks.workunit.client.0.smithi121.stdout:1/199: chown c7 3 1 2024-04-05T21:26:36.303 INFO:tasks.workunit.client.0.smithi121.stdout:7/176: mknod d0/d9/d22/d28/d3b/c40 0 2024-04-05T21:26:36.304 INFO:tasks.workunit.client.0.smithi121.stdout:1/200: mknod d8/d10/d14/d27/c40 0 2024-04-05T21:26:36.307 INFO:tasks.workunit.client.1.smithi188.stdout:3/509: creat d5/d7/d6d/d75/d83/fab x:0 0 0 2024-04-05T21:26:36.314 INFO:tasks.workunit.client.1.smithi188.stdout:7/637: write d3/d2c/d5e/d8e/faf [1711620,129305] 0 2024-04-05T21:26:36.318 INFO:tasks.workunit.client.0.smithi121.stdout:7/177: dwrite d0/d23/f33 [0,4194304] 0 2024-04-05T21:26:36.318 INFO:tasks.workunit.client.0.smithi121.stdout:0/303: truncate d7/d1c/f23 506027 0 2024-04-05T21:26:36.318 INFO:tasks.workunit.client.0.smithi121.stdout:6/149: dwrite d1/d2/f13 [0,4194304] 0 2024-04-05T21:26:36.318 INFO:tasks.workunit.client.1.smithi188.stdout:0/327: truncate d4/f53 2186347 0 2024-04-05T21:26:36.318 INFO:tasks.workunit.client.1.smithi188.stdout:3/510: mknod d5/d7/d65/d61/d66/da4/cac 0 2024-04-05T21:26:36.320 INFO:tasks.workunit.client.0.smithi121.stdout:7/178: rmdir d0/d9/d22 39 2024-04-05T21:26:36.322 INFO:tasks.workunit.client.1.smithi188.stdout:0/328: fsync d4/d8/db/d21/f44 0 2024-04-05T21:26:36.323 INFO:tasks.workunit.client.0.smithi121.stdout:6/150: rmdir d1/d2/dc 39 2024-04-05T21:26:36.326 INFO:tasks.workunit.client.0.smithi121.stdout:0/304: dwrite d7/f18 [4194304,4194304] 0 2024-04-05T21:26:36.329 INFO:tasks.workunit.client.1.smithi188.stdout:2/532: dread d5/f2f [0,4194304] 0 2024-04-05T21:26:36.330 INFO:tasks.workunit.client.0.smithi121.stdout:6/151: stat d1/d2/dc 0 2024-04-05T21:26:36.332 INFO:tasks.workunit.client.0.smithi121.stdout:6/152: unlink d1/l15 0 2024-04-05T21:26:36.333 INFO:tasks.workunit.client.1.smithi188.stdout:7/638: dwrite d3/d4/db/d2e/d37/d65/f20 [8388608,4194304] 0 2024-04-05T21:26:36.333 INFO:tasks.workunit.client.0.smithi121.stdout:0/305: stat d7/f3a 0 2024-04-05T21:26:36.336 INFO:tasks.workunit.client.1.smithi188.stdout:0/329: mknod d4/d8/db/d21/c7c 0 2024-04-05T21:26:36.337 INFO:tasks.workunit.client.0.smithi121.stdout:2/174: dread d2/d9/d12/f15 [0,4194304] 0 2024-04-05T21:26:36.337 INFO:tasks.workunit.client.0.smithi121.stdout:0/306: write d7/d62/f20 [69404,15665] 0 2024-04-05T21:26:36.338 INFO:tasks.workunit.client.0.smithi121.stdout:6/153: mkdir d1/d2/d1b/d2e 0 2024-04-05T21:26:36.338 INFO:tasks.workunit.client.0.smithi121.stdout:6/154: chown d1/d2 1966 1 2024-04-05T21:26:36.340 INFO:tasks.workunit.client.1.smithi188.stdout:2/533: truncate d5/db/dd/d36/d3d/d46/faa 56746 0 2024-04-05T21:26:36.340 INFO:tasks.workunit.client.0.smithi121.stdout:0/307: symlink d7/d1c/d1d/d45/d5a/l65 0 2024-04-05T21:26:36.341 INFO:tasks.workunit.client.0.smithi121.stdout:7/179: write d0/d2d/f2f [4236772,119143] 0 2024-04-05T21:26:36.341 INFO:tasks.workunit.client.1.smithi188.stdout:0/330: dread - d4/d8/db/d21/f44 zero size 2024-04-05T21:26:36.342 INFO:tasks.workunit.client.0.smithi121.stdout:2/175: dread d2/f10 [0,4194304] 0 2024-04-05T21:26:36.343 INFO:tasks.workunit.client.0.smithi121.stdout:6/155: unlink d1/l1a 0 2024-04-05T21:26:36.344 INFO:tasks.workunit.client.1.smithi188.stdout:2/534: creat d5/db/dd/d67/fad x:0 0 0 2024-04-05T21:26:36.348 INFO:tasks.workunit.client.0.smithi121.stdout:2/176: write d2/d9/d12/f15 [2798851,117915] 0 2024-04-05T21:26:36.349 INFO:tasks.workunit.client.0.smithi121.stdout:6/156: creat d1/d2/d3/d5/f2f x:0 0 0 2024-04-05T21:26:36.353 INFO:tasks.workunit.client.0.smithi121.stdout:7/180: dwrite d0/d9/f39 [0,4194304] 0 2024-04-05T21:26:36.354 INFO:tasks.workunit.client.0.smithi121.stdout:2/177: mknod d2/d9/d17/d25/d2d/c34 0 2024-04-05T21:26:36.354 INFO:tasks.workunit.client.0.smithi121.stdout:0/308: dwrite d7/dc/f2d [0,4194304] 0 2024-04-05T21:26:36.355 INFO:tasks.workunit.client.0.smithi121.stdout:6/157: mknod d1/d2/d3/c30 0 2024-04-05T21:26:36.356 INFO:tasks.workunit.client.1.smithi188.stdout:7/639: dwrite d3/d4/db/f38 [0,4194304] 0 2024-04-05T21:26:36.359 INFO:tasks.workunit.client.0.smithi121.stdout:2/178: creat d2/d9/d17/d25/f35 x:0 0 0 2024-04-05T21:26:36.359 INFO:tasks.workunit.client.0.smithi121.stdout:2/179: dread - d2/d9/f1f zero size 2024-04-05T21:26:36.359 INFO:tasks.workunit.client.0.smithi121.stdout:6/158: mknod d1/d2/dc/c31 0 2024-04-05T21:26:36.360 INFO:tasks.workunit.client.1.smithi188.stdout:7/640: stat d3/d4/db/d2e/c6d 0 2024-04-05T21:26:36.360 INFO:tasks.workunit.client.1.smithi188.stdout:0/331: dwrite d4/d8/db/d21/f56 [0,4194304] 0 2024-04-05T21:26:36.360 INFO:tasks.workunit.client.0.smithi121.stdout:2/180: stat d2/d9/d1c/f29 0 2024-04-05T21:26:36.361 INFO:tasks.workunit.client.0.smithi121.stdout:7/181: unlink d0/d9/fe 0 2024-04-05T21:26:36.361 INFO:tasks.workunit.client.0.smithi121.stdout:2/181: chown d2/d9/d17/d25/d2d/c34 1498254 1 2024-04-05T21:26:36.364 INFO:tasks.workunit.client.0.smithi121.stdout:6/159: rename d1/d2/d20 to d1/d32 0 2024-04-05T21:26:36.367 INFO:tasks.workunit.client.0.smithi121.stdout:2/182: mknod d2/d28/c36 0 2024-04-05T21:26:36.367 INFO:tasks.workunit.client.0.smithi121.stdout:7/182: fdatasync d0/d9/f26 0 2024-04-05T21:26:36.367 INFO:tasks.workunit.client.1.smithi188.stdout:0/332: write d4/d8/d39/f48 [926229,22496] 0 2024-04-05T21:26:36.367 INFO:tasks.workunit.client.1.smithi188.stdout:2/535: dwrite d5/fa [0,4194304] 0 2024-04-05T21:26:36.372 INFO:tasks.workunit.client.1.smithi188.stdout:7/641: mkdir d3/d4/db/d10/d5f/dc6 0 2024-04-05T21:26:36.373 INFO:tasks.workunit.client.0.smithi121.stdout:7/183: creat d0/d32/f41 x:0 0 0 2024-04-05T21:26:36.375 INFO:tasks.workunit.client.1.smithi188.stdout:2/536: mknod d5/db/dd/d36/d3d/d46/cae 0 2024-04-05T21:26:36.376 INFO:tasks.workunit.client.1.smithi188.stdout:7/642: truncate d3/d2c/f4c 3315366 0 2024-04-05T21:26:36.376 INFO:tasks.workunit.client.0.smithi121.stdout:6/160: dread d1/d2/dc/f19 [0,4194304] 0 2024-04-05T21:26:36.376 INFO:tasks.workunit.client.0.smithi121.stdout:7/184: write d0/d32/f41 [843068,85284] 0 2024-04-05T21:26:36.381 INFO:tasks.workunit.client.0.smithi121.stdout:7/185: readlink d0/d32/l34 0 2024-04-05T21:26:36.382 INFO:tasks.workunit.client.0.smithi121.stdout:2/183: dwrite d2/f11 [0,4194304] 0 2024-04-05T21:26:36.383 INFO:tasks.workunit.client.1.smithi188.stdout:7/643: dread d3/d4/db/d10/f33 [0,4194304] 0 2024-04-05T21:26:36.383 INFO:tasks.workunit.client.0.smithi121.stdout:7/186: write d0/f3 [2009921,4704] 0 2024-04-05T21:26:36.384 INFO:tasks.workunit.client.0.smithi121.stdout:6/161: chown d1/d32/c2b 119457 1 2024-04-05T21:26:36.384 INFO:tasks.workunit.client.0.smithi121.stdout:2/184: truncate d2/d9/f1f 316754 0 2024-04-05T21:26:36.385 INFO:tasks.workunit.client.0.smithi121.stdout:6/162: dread - d1/d2/d1b/d1e/f2d zero size 2024-04-05T21:26:36.385 INFO:tasks.workunit.client.0.smithi121.stdout:2/185: chown d2/f1a 869514485 1 2024-04-05T21:26:36.386 INFO:tasks.workunit.client.0.smithi121.stdout:7/187: read d0/d9/f20 [114324,25757] 0 2024-04-05T21:26:36.386 INFO:tasks.workunit.client.1.smithi188.stdout:7/644: truncate d3/d4/db/d2e/d37/d65/fad 443772 0 2024-04-05T21:26:36.387 INFO:tasks.workunit.client.0.smithi121.stdout:6/163: chown d1 13 1 2024-04-05T21:26:36.387 INFO:tasks.workunit.client.0.smithi121.stdout:2/186: chown d2/f1a 258693057 1 2024-04-05T21:26:36.389 INFO:tasks.workunit.client.0.smithi121.stdout:6/164: symlink d1/d2/d3/d4/de/l33 0 2024-04-05T21:26:36.394 INFO:tasks.workunit.client.0.smithi121.stdout:7/188: dwrite d0/d23/f33 [0,4194304] 0 2024-04-05T21:26:36.396 INFO:tasks.workunit.client.0.smithi121.stdout:2/187: dwrite d2/d9/d17/f18 [0,4194304] 0 2024-04-05T21:26:36.402 INFO:tasks.workunit.client.0.smithi121.stdout:2/188: rename d2/d9/d1c/f29 to d2/d9/d12/f37 0 2024-04-05T21:26:36.408 INFO:tasks.workunit.client.0.smithi121.stdout:6/165: getdents d1 0 2024-04-05T21:26:36.411 INFO:tasks.workunit.client.0.smithi121.stdout:6/166: symlink d1/l34 0 2024-04-05T21:26:36.412 INFO:tasks.workunit.client.0.smithi121.stdout:0/309: dwrite d7/dc/ff [0,4194304] 0 2024-04-05T21:26:36.413 INFO:tasks.workunit.client.0.smithi121.stdout:6/167: fdatasync d1/d2/d3/d5/f2f 0 2024-04-05T21:26:36.414 INFO:tasks.workunit.client.0.smithi121.stdout:7/189: truncate d0/f3 1681890 0 2024-04-05T21:26:36.415 INFO:tasks.workunit.client.0.smithi121.stdout:7/190: chown d0/d9/f3a 3451784 1 2024-04-05T21:26:36.416 INFO:tasks.workunit.client.0.smithi121.stdout:0/310: rmdir d7/d1c/d1d/d45/d46 39 2024-04-05T21:26:36.416 INFO:tasks.workunit.client.0.smithi121.stdout:6/168: fsync d1/f10 0 2024-04-05T21:26:36.418 INFO:tasks.workunit.client.0.smithi121.stdout:6/169: write d1/f1c [5218063,91083] 0 2024-04-05T21:26:36.419 INFO:tasks.workunit.client.0.smithi121.stdout:7/191: creat d0/f42 x:0 0 0 2024-04-05T21:26:36.421 INFO:tasks.workunit.client.0.smithi121.stdout:0/311: getdents d7 0 2024-04-05T21:26:36.421 INFO:tasks.workunit.client.0.smithi121.stdout:3/159: sync 2024-04-05T21:26:36.421 INFO:tasks.workunit.client.0.smithi121.stdout:5/195: sync 2024-04-05T21:26:36.422 INFO:tasks.workunit.client.0.smithi121.stdout:5/196: readlink dd/de/d19/d1c/l1e 0 2024-04-05T21:26:36.423 INFO:tasks.workunit.client.0.smithi121.stdout:3/160: dread d0/d3/dc/d1a/d1e/f20 [0,4194304] 0 2024-04-05T21:26:36.428 INFO:tasks.workunit.client.0.smithi121.stdout:3/161: dwrite d0/d3/dc/d1a/f22 [0,4194304] 0 2024-04-05T21:26:36.436 INFO:tasks.workunit.client.0.smithi121.stdout:7/192: rename d0/d9/f39 to d0/d2d/f43 0 2024-04-05T21:26:36.442 INFO:tasks.workunit.client.0.smithi121.stdout:7/193: creat d0/d9/d22/f44 x:0 0 0 2024-04-05T21:26:36.444 INFO:tasks.workunit.client.0.smithi121.stdout:5/197: dwrite dd/f1f [0,4194304] 0 2024-04-05T21:26:36.445 INFO:tasks.workunit.client.0.smithi121.stdout:0/312: link d7/d1c/d1d/c49 d7/d1c/c66 0 2024-04-05T21:26:36.450 INFO:tasks.workunit.client.0.smithi121.stdout:7/194: rename d0/d9/d22/d28/l3d to d0/l45 0 2024-04-05T21:26:36.451 INFO:tasks.workunit.client.0.smithi121.stdout:7/195: write d0/d9/f12 [164379,97640] 0 2024-04-05T21:26:36.453 INFO:tasks.workunit.client.0.smithi121.stdout:7/196: chown d0/d9/d22/d3e 3 1 2024-04-05T21:26:36.455 INFO:tasks.workunit.client.0.smithi121.stdout:0/313: dwrite d7/dc/f2d [0,4194304] 0 2024-04-05T21:26:36.456 INFO:tasks.workunit.client.0.smithi121.stdout:7/197: creat d0/d2d/f46 x:0 0 0 2024-04-05T21:26:36.458 INFO:tasks.workunit.client.0.smithi121.stdout:5/198: fsync dd/f1f 0 2024-04-05T21:26:36.458 INFO:tasks.workunit.client.0.smithi121.stdout:7/198: fsync d0/d2d/f3f 0 2024-04-05T21:26:36.462 INFO:tasks.workunit.client.0.smithi121.stdout:7/199: read d0/d23/f2e [4662,130318] 0 2024-04-05T21:26:36.465 INFO:tasks.workunit.client.0.smithi121.stdout:0/314: dwrite d7/d1c/d1d/d29/d2a/d2c/f43 [0,4194304] 0 2024-04-05T21:26:36.465 INFO:tasks.workunit.client.0.smithi121.stdout:7/200: creat d0/d23/f47 x:0 0 0 2024-04-05T21:26:36.466 INFO:tasks.workunit.client.0.smithi121.stdout:5/199: dread f6 [0,4194304] 0 2024-04-05T21:26:36.469 INFO:tasks.workunit.client.0.smithi121.stdout:7/201: unlink d0/lb 0 2024-04-05T21:26:36.469 INFO:tasks.workunit.client.0.smithi121.stdout:7/202: readlink d0/l1c 0 2024-04-05T21:26:36.469 INFO:tasks.workunit.client.1.smithi188.stdout:5/461: sync 2024-04-05T21:26:36.470 INFO:tasks.workunit.client.0.smithi121.stdout:5/200: rmdir dd/de/d19/d13 39 2024-04-05T21:26:36.472 INFO:tasks.workunit.client.0.smithi121.stdout:0/315: mknod d7/d62/d3c/c67 0 2024-04-05T21:26:36.473 INFO:tasks.workunit.client.0.smithi121.stdout:0/316: write d7/d1c/d1d/d29/f58 [709893,9923] 0 2024-04-05T21:26:36.474 INFO:tasks.workunit.client.1.smithi188.stdout:5/462: read d6/d15/d5c/f94 [1543,40685] 0 2024-04-05T21:26:36.474 INFO:tasks.workunit.client.0.smithi121.stdout:5/201: creat dd/de/d19/d1c/d36/f3a x:0 0 0 2024-04-05T21:26:36.475 INFO:tasks.workunit.client.0.smithi121.stdout:2/189: fsync d2/d9/d12/f37 0 2024-04-05T21:26:36.477 INFO:tasks.workunit.client.0.smithi121.stdout:2/190: fsync d2/f10 0 2024-04-05T21:26:36.480 INFO:tasks.workunit.client.0.smithi121.stdout:0/317: creat d7/d1c/f68 x:0 0 0 2024-04-05T21:26:36.483 INFO:tasks.workunit.client.0.smithi121.stdout:0/318: chown d7/d62/d3c/l5e 5869 1 2024-04-05T21:26:36.484 INFO:tasks.workunit.client.0.smithi121.stdout:0/319: read d7/d62/f42 [1600670,126321] 0 2024-04-05T21:26:36.485 INFO:tasks.workunit.client.0.smithi121.stdout:5/202: dwrite dd/de/f11 [4194304,4194304] 0 2024-04-05T21:26:36.486 INFO:tasks.workunit.client.1.smithi188.stdout:5/463: dwrite d6/d15/d5c/d7a/d44/d51/f88 [0,4194304] 0 2024-04-05T21:26:36.491 INFO:tasks.workunit.client.1.smithi188.stdout:5/464: creat d6/d15/d5c/d7a/d44/da0/faa x:0 0 0 2024-04-05T21:26:36.491 INFO:tasks.workunit.client.0.smithi121.stdout:5/203: mkdir dd/de/d39/d3b 0 2024-04-05T21:26:36.496 INFO:tasks.workunit.client.0.smithi121.stdout:5/204: mkdir dd/de/d19/d3c 0 2024-04-05T21:26:36.497 INFO:tasks.workunit.client.0.smithi121.stdout:3/162: dwrite d0/d3/f8 [0,4194304] 0 2024-04-05T21:26:36.498 INFO:tasks.workunit.client.0.smithi121.stdout:5/205: stat dd/c10 0 2024-04-05T21:26:36.499 INFO:tasks.workunit.client.0.smithi121.stdout:3/163: fdatasync d0/f27 0 2024-04-05T21:26:36.500 INFO:tasks.workunit.client.0.smithi121.stdout:0/320: dwrite d7/dc/f33 [0,4194304] 0 2024-04-05T21:26:36.501 INFO:tasks.workunit.client.0.smithi121.stdout:7/203: dread d0/d9/f24 [0,4194304] 0 2024-04-05T21:26:36.509 INFO:tasks.workunit.client.0.smithi121.stdout:7/204: creat d0/d9/d22/d28/f48 x:0 0 0 2024-04-05T21:26:36.510 INFO:tasks.workunit.client.0.smithi121.stdout:7/205: read d0/f1f [377226,48501] 0 2024-04-05T21:26:36.511 INFO:tasks.workunit.client.0.smithi121.stdout:7/206: chown d0/f8 8055 1 2024-04-05T21:26:36.511 INFO:tasks.workunit.client.0.smithi121.stdout:3/164: rename d0/d3/dc/d1a/d1e/f20 to d0/d3/d24/f34 0 2024-04-05T21:26:36.511 INFO:tasks.workunit.client.0.smithi121.stdout:5/206: getdents dd/de/d19/d3c 0 2024-04-05T21:26:36.513 INFO:tasks.workunit.client.0.smithi121.stdout:3/165: fsync d0/d3/dc/d1a/d1e/f2a 0 2024-04-05T21:26:36.513 INFO:tasks.workunit.client.0.smithi121.stdout:0/321: link d7/d1c/f68 d7/d1c/d1d/d45/d5a/f69 0 2024-04-05T21:26:36.515 INFO:tasks.workunit.client.0.smithi121.stdout:7/207: mknod d0/d23/c49 0 2024-04-05T21:26:36.515 INFO:tasks.workunit.client.0.smithi121.stdout:0/322: truncate d7/d62/f5c 1002337 0 2024-04-05T21:26:36.516 INFO:tasks.workunit.client.0.smithi121.stdout:5/207: unlink dd/de/f1a 0 2024-04-05T21:26:36.517 INFO:tasks.workunit.client.0.smithi121.stdout:5/208: write dd/de/f2e [3197549,105430] 0 2024-04-05T21:26:36.519 INFO:tasks.workunit.client.0.smithi121.stdout:7/208: mkdir d0/d32/d4a 0 2024-04-05T21:26:36.521 INFO:tasks.workunit.client.0.smithi121.stdout:5/209: mknod dd/de/d19/d1c/d36/c3d 0 2024-04-05T21:26:36.522 INFO:tasks.workunit.client.0.smithi121.stdout:0/323: creat d7/d1c/d1d/d45/f6a x:0 0 0 2024-04-05T21:26:36.525 INFO:tasks.workunit.client.0.smithi121.stdout:5/210: symlink dd/de/d19/d38/l3e 0 2024-04-05T21:26:36.526 INFO:tasks.workunit.client.1.smithi188.stdout:5/465: dread d6/d15/d1b/f34 [0,4194304] 0 2024-04-05T21:26:36.527 INFO:tasks.workunit.client.1.smithi188.stdout:3/511: truncate d5/f68 421431 0 2024-04-05T21:26:36.528 INFO:tasks.workunit.client.1.smithi188.stdout:5/466: mknod d6/cab 0 2024-04-05T21:26:36.529 INFO:tasks.workunit.client.0.smithi121.stdout:2/191: truncate d2/d9/ff 3745405 0 2024-04-05T21:26:36.530 INFO:tasks.workunit.client.0.smithi121.stdout:0/324: dwrite d7/dc/f33 [0,4194304] 0 2024-04-05T21:26:36.530 INFO:tasks.workunit.client.0.smithi121.stdout:2/192: fdatasync d2/d9/d17/f19 0 2024-04-05T21:26:36.530 INFO:tasks.workunit.client.0.smithi121.stdout:5/211: creat dd/de/d19/d13/f3f x:0 0 0 2024-04-05T21:26:36.531 INFO:tasks.workunit.client.1.smithi188.stdout:3/512: mknod d5/d7/d6d/d75/d83/cad 0 2024-04-05T21:26:36.532 INFO:tasks.workunit.client.1.smithi188.stdout:5/467: readlink d6/l8 0 2024-04-05T21:26:36.533 INFO:tasks.workunit.client.1.smithi188.stdout:3/513: chown d5/dc/l3f 85839535 1 2024-04-05T21:26:36.533 INFO:tasks.workunit.client.0.smithi121.stdout:5/212: write dd/de/d19/d13/f35 [600171,15955] 0 2024-04-05T21:26:36.534 INFO:tasks.workunit.client.0.smithi121.stdout:5/213: write dd/de/d19/d1c/f33 [767357,85549] 0 2024-04-05T21:26:36.535 INFO:tasks.workunit.client.0.smithi121.stdout:0/325: mkdir d7/d62/d3c/d6b 0 2024-04-05T21:26:36.537 INFO:tasks.workunit.client.0.smithi121.stdout:0/326: chown d7/d62/c21 0 1 2024-04-05T21:26:36.538 INFO:tasks.workunit.client.1.smithi188.stdout:5/468: read d6/f6f [260739,104096] 0 2024-04-05T21:26:36.540 INFO:tasks.workunit.client.0.smithi121.stdout:2/193: getdents d2/d9/d12 0 2024-04-05T21:26:36.541 INFO:tasks.workunit.client.0.smithi121.stdout:0/327: fdatasync d7/d1c/d1d/d45/d46/f57 0 2024-04-05T21:26:36.541 INFO:tasks.workunit.client.0.smithi121.stdout:3/166: write d0/d3/f13 [960236,50529] 0 2024-04-05T21:26:36.542 INFO:tasks.workunit.client.1.smithi188.stdout:7/645: dwrite d3/d4/d1d/d74/f41 [0,4194304] 0 2024-04-05T21:26:36.542 INFO:tasks.workunit.client.0.smithi121.stdout:5/214: creat dd/de/d19/d3c/f40 x:0 0 0 2024-04-05T21:26:36.543 INFO:tasks.workunit.client.1.smithi188.stdout:5/469: dread - d6/d17/f80 zero size 2024-04-05T21:26:36.544 INFO:tasks.workunit.client.0.smithi121.stdout:3/167: readlink d0/d3/dc/l1c 0 2024-04-05T21:26:36.547 INFO:tasks.workunit.client.0.smithi121.stdout:5/215: unlink dd/de/d19/d1c/l30 0 2024-04-05T21:26:36.549 INFO:tasks.workunit.client.1.smithi188.stdout:7/646: symlink d3/d4/db/d2e/d8c/lc7 0 2024-04-05T21:26:36.550 INFO:tasks.workunit.client.0.smithi121.stdout:5/216: stat dd/de/d19/f32 0 2024-04-05T21:26:36.551 INFO:tasks.workunit.client.0.smithi121.stdout:5/217: fdatasync dd/de/d19/d13/f26 0 2024-04-05T21:26:36.551 INFO:tasks.workunit.client.0.smithi121.stdout:7/209: dwrite d0/d32/f35 [0,4194304] 0 2024-04-05T21:26:36.552 INFO:tasks.workunit.client.1.smithi188.stdout:5/470: rename d6/ca to d6/d17/cac 0 2024-04-05T21:26:36.552 INFO:tasks.workunit.client.0.smithi121.stdout:2/194: rename d2/d9/d17/d25 to d2/d9/d38 0 2024-04-05T21:26:36.553 INFO:tasks.workunit.client.0.smithi121.stdout:3/168: mknod d0/d3/dc/d1a/d1e/c35 0 2024-04-05T21:26:36.554 INFO:tasks.workunit.client.0.smithi121.stdout:5/218: chown dd/de/d19/d1c/d36/c3d 4 1 2024-04-05T21:26:36.555 INFO:tasks.workunit.client.1.smithi188.stdout:7/647: mkdir d3/da5/dc8 0 2024-04-05T21:26:36.555 INFO:tasks.workunit.client.0.smithi121.stdout:3/169: write d0/f2 [4126470,124456] 0 2024-04-05T21:26:36.555 INFO:tasks.workunit.client.0.smithi121.stdout:5/219: dread - dd/de/d19/f32 zero size 2024-04-05T21:26:36.557 INFO:tasks.workunit.client.0.smithi121.stdout:2/195: rmdir d2/d1e 39 2024-04-05T21:26:36.557 INFO:tasks.workunit.client.1.smithi188.stdout:3/514: truncate d5/d2b/f2e 403253 0 2024-04-05T21:26:36.557 INFO:tasks.workunit.client.1.smithi188.stdout:7/648: write d3/f1e [3373017,37308] 0 2024-04-05T21:26:36.559 INFO:tasks.workunit.client.0.smithi121.stdout:7/210: creat d0/d2d/f4b x:0 0 0 2024-04-05T21:26:36.561 INFO:tasks.workunit.client.0.smithi121.stdout:5/220: mkdir dd/d41 0 2024-04-05T21:26:36.565 INFO:tasks.workunit.client.1.smithi188.stdout:7/649: readlink d3/d4/d1d/l23 0 2024-04-05T21:26:36.565 INFO:tasks.workunit.client.0.smithi121.stdout:7/211: mknod d0/d9/d22/d28/c4c 0 2024-04-05T21:26:36.567 INFO:tasks.workunit.client.1.smithi188.stdout:3/515: dread d5/d7/d6d/d75/d83/f8e [0,4194304] 0 2024-04-05T21:26:36.568 INFO:tasks.workunit.client.0.smithi121.stdout:7/212: write d0/d9/d22/f44 [808522,103714] 0 2024-04-05T21:26:36.568 INFO:tasks.workunit.client.1.smithi188.stdout:3/516: stat d5/d7/d6d/d75/d83 0 2024-04-05T21:26:36.570 INFO:tasks.workunit.client.1.smithi188.stdout:7/650: link d3/d4/db/d10/f95 d3/d4/db/d10/fc9 0 2024-04-05T21:26:36.571 INFO:tasks.workunit.client.1.smithi188.stdout:3/517: rename d5/dc/d2f/fa9 to d5/d7/d65/d61/fae 0 2024-04-05T21:26:36.572 INFO:tasks.workunit.client.0.smithi121.stdout:0/328: dwrite d7/d1c/d1d/d29/f34 [0,4194304] 0 2024-04-05T21:26:36.573 INFO:tasks.workunit.client.0.smithi121.stdout:5/221: dread dd/de/f21 [0,4194304] 0 2024-04-05T21:26:36.573 INFO:tasks.workunit.client.0.smithi121.stdout:3/170: link d0/d3/dc/d1a/f33 d0/d3/de/f36 0 2024-04-05T21:26:36.574 INFO:tasks.workunit.client.0.smithi121.stdout:2/196: link d2/d9/d38/d2d/c34 d2/d1e/c39 0 2024-04-05T21:26:36.575 INFO:tasks.workunit.client.0.smithi121.stdout:0/329: symlink d7/d1c/d1d/d45/d46/l6c 0 2024-04-05T21:26:36.577 INFO:tasks.workunit.client.0.smithi121.stdout:2/197: dread d2/d9/d12/f37 [0,4194304] 0 2024-04-05T21:26:36.580 INFO:tasks.workunit.client.0.smithi121.stdout:2/198: write d2/d9/d1c/f22 [68481,50419] 0 2024-04-05T21:26:36.582 INFO:tasks.workunit.client.0.smithi121.stdout:5/222: dwrite dd/de/d19/f37 [0,4194304] 0 2024-04-05T21:26:36.584 INFO:tasks.workunit.client.0.smithi121.stdout:5/223: stat dd/de/f21 0 2024-04-05T21:26:36.586 INFO:tasks.workunit.client.0.smithi121.stdout:7/213: dread d0/d9/d22/f44 [0,4194304] 0 2024-04-05T21:26:36.586 INFO:tasks.workunit.client.0.smithi121.stdout:5/224: chown dd/de/f21 19 1 2024-04-05T21:26:36.588 INFO:tasks.workunit.client.1.smithi188.stdout:5/471: dread d6/f9 [0,4194304] 0 2024-04-05T21:26:36.589 INFO:tasks.workunit.client.1.smithi188.stdout:5/472: readlink d6/d17/d39/l6d 0 2024-04-05T21:26:36.589 INFO:tasks.workunit.client.0.smithi121.stdout:2/199: dwrite d2/f1a [0,4194304] 0 2024-04-05T21:26:36.590 INFO:tasks.workunit.client.1.smithi188.stdout:3/518: dwrite d5/f7d [0,4194304] 0 2024-04-05T21:26:36.597 INFO:tasks.workunit.client.1.smithi188.stdout:3/519: dread d5/df/f15 [0,4194304] 0 2024-04-05T21:26:36.599 INFO:tasks.workunit.client.1.smithi188.stdout:3/520: mknod d5/d35/d4e/caf 0 2024-04-05T21:26:36.601 INFO:tasks.workunit.client.1.smithi188.stdout:3/521: symlink d5/d7/d31/lb0 0 2024-04-05T21:26:36.601 INFO:tasks.workunit.client.1.smithi188.stdout:3/522: chown d5/d2b 6104858 1 2024-04-05T21:26:36.601 INFO:tasks.workunit.client.1.smithi188.stdout:1/499: sync 2024-04-05T21:26:36.603 INFO:tasks.workunit.client.1.smithi188.stdout:1/500: write d1/dc/f11 [5370975,30235] 0 2024-04-05T21:26:36.604 INFO:tasks.workunit.client.1.smithi188.stdout:5/473: write d6/d17/f2f [1198205,7614] 0 2024-04-05T21:26:36.605 INFO:tasks.workunit.client.1.smithi188.stdout:1/501: stat d1/dc/d16/d46/f51 0 2024-04-05T21:26:36.605 INFO:tasks.workunit.client.1.smithi188.stdout:1/502: readlink d1/l65 0 2024-04-05T21:26:36.607 INFO:tasks.workunit.client.1.smithi188.stdout:3/523: read d5/d7/d65/f54 [157664,34793] 0 2024-04-05T21:26:36.608 INFO:tasks.workunit.client.1.smithi188.stdout:5/474: dread d6/d17/d39/d8e/f8f [0,4194304] 0 2024-04-05T21:26:36.614 INFO:tasks.workunit.client.1.smithi188.stdout:1/503: unlink d1/dc/d16/d46/d88/d13/d60/l98 0 2024-04-05T21:26:36.614 INFO:tasks.workunit.client.1.smithi188.stdout:3/524: creat d5/d7/d31/fb1 x:0 0 0 2024-04-05T21:26:36.622 INFO:tasks.workunit.client.0.smithi121.stdout:4/222: sync 2024-04-05T21:26:36.622 INFO:tasks.workunit.client.1.smithi188.stdout:1/504: chown d1/dc/d16/d21/d27/d49/f73 15779 1 2024-04-05T21:26:36.622 INFO:tasks.workunit.client.0.smithi121.stdout:4/223: read d5/d11/f1e [1155463,5773] 0 2024-04-05T21:26:36.623 INFO:tasks.workunit.client.0.smithi121.stdout:4/224: read d5/f8 [223457,42734] 0 2024-04-05T21:26:36.625 INFO:tasks.workunit.client.1.smithi188.stdout:5/475: rename d6/d17/d23/d85/d8c to d6/dad 0 2024-04-05T21:26:36.626 INFO:tasks.workunit.client.1.smithi188.stdout:3/525: getdents d5/d7 0 2024-04-05T21:26:36.630 INFO:tasks.workunit.client.1.smithi188.stdout:5/476: mknod d6/d15/cae 0 2024-04-05T21:26:36.632 INFO:tasks.workunit.client.1.smithi188.stdout:5/477: write d6/d15/d1b/f6c [872509,97968] 0 2024-04-05T21:26:36.633 INFO:tasks.workunit.client.1.smithi188.stdout:5/478: dread - d6/f38 zero size 2024-04-05T21:26:36.638 INFO:tasks.workunit.client.1.smithi188.stdout:1/505: truncate d1/dc/d16/d46/d88/d13/f8d 312855 0 2024-04-05T21:26:36.639 INFO:tasks.workunit.client.1.smithi188.stdout:5/479: mknod d6/d82/d40/d42/d81/da6/caf 0 2024-04-05T21:26:36.639 INFO:tasks.workunit.client.1.smithi188.stdout:5/480: chown d6/d82/d40/d42 243921365 1 2024-04-05T21:26:36.640 INFO:tasks.workunit.client.1.smithi188.stdout:1/506: symlink d1/dc/d16/d46/la7 0 2024-04-05T21:26:36.645 INFO:tasks.workunit.client.1.smithi188.stdout:3/526: dread d5/d7/f30 [0,4194304] 0 2024-04-05T21:26:36.647 INFO:tasks.workunit.client.1.smithi188.stdout:5/481: dwrite d6/d82/d40/d42/f77 [0,4194304] 0 2024-04-05T21:26:36.654 INFO:tasks.workunit.client.1.smithi188.stdout:1/507: dread d1/dc/d16/d46/d88/d13/f6a [0,4194304] 0 2024-04-05T21:26:36.664 INFO:tasks.workunit.client.0.smithi121.stdout:0/330: rmdir d7/d1c/d1d/d29/d2a/d2c 39 2024-04-05T21:26:36.668 INFO:tasks.workunit.client.0.smithi121.stdout:7/214: symlink d0/d9/d22/d3e/l4d 0 2024-04-05T21:26:36.669 INFO:tasks.workunit.client.0.smithi121.stdout:5/225: creat dd/de/d19/d3c/f42 x:0 0 0 2024-04-05T21:26:36.670 INFO:tasks.workunit.client.0.smithi121.stdout:2/200: mkdir d2/d9/d17/d3a 0 2024-04-05T21:26:36.675 INFO:tasks.workunit.client.0.smithi121.stdout:3/171: write d0/f15 [4555096,62299] 0 2024-04-05T21:26:36.677 INFO:tasks.workunit.client.0.smithi121.stdout:7/215: mkdir d0/d32/d4e 0 2024-04-05T21:26:36.678 INFO:tasks.workunit.client.0.smithi121.stdout:5/226: dwrite f4 [0,4194304] 0 2024-04-05T21:26:36.679 INFO:tasks.workunit.client.0.smithi121.stdout:7/216: write d0/d2d/f4b [599341,65982] 0 2024-04-05T21:26:36.679 INFO:tasks.workunit.client.0.smithi121.stdout:5/227: chown dd/de/d19/d38 6685 1 2024-04-05T21:26:36.681 INFO:tasks.workunit.client.0.smithi121.stdout:4/225: link d5/d11/d2c/f3c d5/d11/d26/d27/f4c 0 2024-04-05T21:26:36.685 INFO:tasks.workunit.client.0.smithi121.stdout:5/228: rmdir dd/de/d19/d1c 39 2024-04-05T21:26:36.685 INFO:tasks.workunit.client.0.smithi121.stdout:2/201: dread d2/d9/fd [0,4194304] 0 2024-04-05T21:26:36.686 INFO:tasks.workunit.client.0.smithi121.stdout:2/202: readlink d2/d9/l1b 0 2024-04-05T21:26:36.687 INFO:tasks.workunit.client.0.smithi121.stdout:3/172: rmdir d0/d31 0 2024-04-05T21:26:36.687 INFO:tasks.workunit.client.0.smithi121.stdout:3/173: fsync d0/f27 0 2024-04-05T21:26:36.689 INFO:tasks.workunit.client.1.smithi188.stdout:3/527: write d5/d7/d31/f5c [9457092,24608] 0 2024-04-05T21:26:36.690 INFO:tasks.workunit.client.1.smithi188.stdout:3/528: chown l2 505 1 2024-04-05T21:26:36.690 INFO:tasks.workunit.client.1.smithi188.stdout:5/482: rename d6/d15/d5c/d7a/d44/d98/fa9 to d6/d15/d5c/d7a/d44/da0/fb0 0 2024-04-05T21:26:36.690 INFO:tasks.workunit.client.0.smithi121.stdout:2/203: mknod d2/d9/d38/c3b 0 2024-04-05T21:26:36.691 INFO:tasks.workunit.client.1.smithi188.stdout:5/483: write d6/dad/f8d [521436,71362] 0 2024-04-05T21:26:36.692 INFO:tasks.workunit.client.0.smithi121.stdout:4/226: creat d5/d11/d26/d27/d29/d2e/f4d x:0 0 0 2024-04-05T21:26:36.692 INFO:tasks.workunit.client.1.smithi188.stdout:3/529: write d5/dc/f2d [83035,87203] 0 2024-04-05T21:26:36.692 INFO:tasks.workunit.client.1.smithi188.stdout:1/508: mkdir d1/dc/d16/d2d/da8 0 2024-04-05T21:26:36.693 INFO:tasks.workunit.client.0.smithi121.stdout:4/227: fsync d5/d11/f17 0 2024-04-05T21:26:36.693 INFO:tasks.workunit.client.0.smithi121.stdout:3/174: mknod d0/d3/dc/c37 0 2024-04-05T21:26:36.693 INFO:tasks.workunit.client.1.smithi188.stdout:1/509: readlink d1/dc/d16/d21/d27/l70 0 2024-04-05T21:26:36.694 INFO:tasks.workunit.client.1.smithi188.stdout:5/484: stat d6/d17/cac 0 2024-04-05T21:26:36.695 INFO:tasks.workunit.client.1.smithi188.stdout:1/510: stat d1/dc/d16/d46/d88/d13/f50 0 2024-04-05T21:26:36.696 INFO:tasks.workunit.client.0.smithi121.stdout:2/204: symlink d2/d9/d38/l3c 0 2024-04-05T21:26:36.697 INFO:tasks.workunit.client.1.smithi188.stdout:3/530: rename d5/d7/f25 to d5/d7/d31/fb2 0 2024-04-05T21:26:36.697 INFO:tasks.workunit.client.0.smithi121.stdout:2/205: write d2/d9/f1f [559296,35793] 0 2024-04-05T21:26:36.698 INFO:tasks.workunit.client.1.smithi188.stdout:3/531: write d5/d7/d65/d61/f9a [446107,7723] 0 2024-04-05T21:26:36.698 INFO:tasks.workunit.client.0.smithi121.stdout:5/229: link dd/de/d19/d13/l24 dd/de/d39/l43 0 2024-04-05T21:26:36.700 INFO:tasks.workunit.client.0.smithi121.stdout:7/217: write d0/d2d/f3f [3214038,130054] 0 2024-04-05T21:26:36.701 INFO:tasks.workunit.client.1.smithi188.stdout:1/511: mknod d1/dc/d16/d46/d95/ca9 0 2024-04-05T21:26:36.702 INFO:tasks.workunit.client.0.smithi121.stdout:4/228: symlink d5/d11/l4e 0 2024-04-05T21:26:36.703 INFO:tasks.workunit.client.1.smithi188.stdout:3/532: creat d5/d2b/fb3 x:0 0 0 2024-04-05T21:26:36.703 INFO:tasks.workunit.client.1.smithi188.stdout:3/533: read - d5/d2b/fa8 zero size 2024-04-05T21:26:36.704 INFO:tasks.workunit.client.0.smithi121.stdout:7/218: write d0/d23/f33 [835747,25899] 0 2024-04-05T21:26:36.704 INFO:tasks.workunit.client.0.smithi121.stdout:4/229: readlink d5/d11/d20/l3d 0 2024-04-05T21:26:36.704 INFO:tasks.workunit.client.0.smithi121.stdout:5/230: creat dd/de/d39/f44 x:0 0 0 2024-04-05T21:26:36.706 INFO:tasks.workunit.client.1.smithi188.stdout:1/512: chown d1/dc/d16/d46/l4d 1442252443 1 2024-04-05T21:26:36.707 INFO:tasks.workunit.client.1.smithi188.stdout:1/513: chown d1/dc/f1b 0 1 2024-04-05T21:26:36.708 INFO:tasks.workunit.client.1.smithi188.stdout:1/514: write d1/fb [8894338,71914] 0 2024-04-05T21:26:36.708 INFO:tasks.workunit.client.0.smithi121.stdout:7/219: read d0/d9/f1d [446196,74863] 0 2024-04-05T21:26:36.708 INFO:tasks.workunit.client.0.smithi121.stdout:2/206: link d2/d28/c36 d2/d1e/c3d 0 2024-04-05T21:26:36.710 INFO:tasks.workunit.client.1.smithi188.stdout:1/515: symlink d1/dc/d16/d46/d88/d3f/laa 0 2024-04-05T21:26:36.711 INFO:tasks.workunit.client.0.smithi121.stdout:7/220: write d0/f15 [893234,129771] 0 2024-04-05T21:26:36.711 INFO:tasks.workunit.client.0.smithi121.stdout:3/175: dwrite d0/d3/de/f17 [0,4194304] 0 2024-04-05T21:26:36.712 INFO:tasks.workunit.client.0.smithi121.stdout:5/231: mkdir dd/de/d19/d38/d45 0 2024-04-05T21:26:36.714 INFO:tasks.workunit.client.1.smithi188.stdout:1/516: creat d1/dc/d16/d46/d88/d86/fab x:0 0 0 2024-04-05T21:26:36.716 INFO:tasks.workunit.client.0.smithi121.stdout:7/221: dread - d0/d9/d22/d28/f48 zero size 2024-04-05T21:26:36.716 INFO:tasks.workunit.client.0.smithi121.stdout:5/232: fdatasync dd/de/d19/d1c/f33 0 2024-04-05T21:26:36.717 INFO:tasks.workunit.client.0.smithi121.stdout:4/230: dwrite d5/d11/f35 [0,4194304] 0 2024-04-05T21:26:36.717 INFO:tasks.workunit.client.0.smithi121.stdout:2/207: mknod d2/d9/c3e 0 2024-04-05T21:26:36.719 INFO:tasks.workunit.client.0.smithi121.stdout:5/233: chown dd/de/f21 0 1 2024-04-05T21:26:36.719 INFO:tasks.workunit.client.0.smithi121.stdout:4/231: dread - d5/d11/d26/d27/f30 zero size 2024-04-05T21:26:36.719 INFO:tasks.workunit.client.1.smithi188.stdout:1/517: fsync d1/dc/d16/d2d/f9a 0 2024-04-05T21:26:36.719 INFO:tasks.workunit.client.1.smithi188.stdout:5/485: dwrite d6/d82/d40/d42/f5e [0,4194304] 0 2024-04-05T21:26:36.721 INFO:tasks.workunit.client.0.smithi121.stdout:3/176: mkdir d0/d3/dc/d1a/d23/d38 0 2024-04-05T21:26:36.723 INFO:tasks.workunit.client.1.smithi188.stdout:5/486: write d6/d15/d5c/f94 [1396541,44885] 0 2024-04-05T21:26:36.725 INFO:tasks.workunit.client.0.smithi121.stdout:7/222: unlink d0/d9/d22/d28/d3b/c40 0 2024-04-05T21:26:36.726 INFO:tasks.workunit.client.0.smithi121.stdout:7/223: write d0/f8 [426894,108019] 0 2024-04-05T21:26:36.730 INFO:tasks.workunit.client.1.smithi188.stdout:1/518: write d1/dc/d16/d46/d88/d13/f8d [267230,26839] 0 2024-04-05T21:26:36.732 INFO:tasks.workunit.client.0.smithi121.stdout:3/177: symlink d0/d3/dc/d1a/d1e/l39 0 2024-04-05T21:26:36.733 INFO:tasks.workunit.client.1.smithi188.stdout:1/519: symlink d1/dc/d16/d46/d88/d3f/lac 0 2024-04-05T21:26:36.733 INFO:tasks.workunit.client.0.smithi121.stdout:2/208: dwrite d2/f10 [0,4194304] 0 2024-04-05T21:26:36.735 INFO:tasks.workunit.client.0.smithi121.stdout:3/178: chown d0/d3/dc/f21 1 1 2024-04-05T21:26:36.739 INFO:tasks.workunit.client.0.smithi121.stdout:3/179: fdatasync d0/f2d 0 2024-04-05T21:26:36.740 INFO:tasks.workunit.client.1.smithi188.stdout:1/520: dwrite d1/dc/d16/d46/d88/d86/fab [0,4194304] 0 2024-04-05T21:26:36.742 INFO:tasks.workunit.client.1.smithi188.stdout:1/521: write d1/dc/d16/d21/fa0 [864865,67086] 0 2024-04-05T21:26:36.743 INFO:tasks.workunit.client.0.smithi121.stdout:7/224: dwrite d0/d9/f24 [0,4194304] 0 2024-04-05T21:26:36.744 INFO:tasks.workunit.client.1.smithi188.stdout:1/522: write d1/dc/d16/f33 [3813796,52081] 0 2024-04-05T21:26:36.750 INFO:tasks.workunit.client.1.smithi188.stdout:1/523: getdents d1/dc/d16/d46/d88/d3f 0 2024-04-05T21:26:36.750 INFO:tasks.workunit.client.1.smithi188.stdout:1/524: stat d1/dc/d16/d2d/f7e 0 2024-04-05T21:26:36.751 INFO:tasks.workunit.client.0.smithi121.stdout:5/234: dwrite dd/de/d19/d13/f16 [0,4194304] 0 2024-04-05T21:26:36.754 INFO:tasks.workunit.client.0.smithi121.stdout:2/209: dwrite d2/d9/fb [0,4194304] 0 2024-04-05T21:26:36.755 INFO:tasks.workunit.client.0.smithi121.stdout:5/235: mkdir dd/de/d19/d1c/d36/d46 0 2024-04-05T21:26:36.758 INFO:tasks.workunit.client.0.smithi121.stdout:5/236: write dd/de/f2e [3376295,113777] 0 2024-04-05T21:26:36.758 INFO:tasks.workunit.client.1.smithi188.stdout:1/525: dwrite d1/dc/d16/f33 [4194304,4194304] 0 2024-04-05T21:26:36.760 INFO:tasks.workunit.client.0.smithi121.stdout:5/237: mknod dd/de/d39/c47 0 2024-04-05T21:26:36.763 INFO:tasks.workunit.client.0.smithi121.stdout:5/238: unlink l3 0 2024-04-05T21:26:36.764 INFO:tasks.workunit.client.0.smithi121.stdout:5/239: dread - dd/de/d19/d3c/f42 zero size 2024-04-05T21:26:36.766 INFO:tasks.workunit.client.1.smithi188.stdout:1/526: getdents d1/dc/d16/d21/d27/d4a 0 2024-04-05T21:26:36.768 INFO:tasks.workunit.client.1.smithi188.stdout:1/527: truncate d1/dc/d16/d46/f51 317866 0 2024-04-05T21:26:36.769 INFO:tasks.workunit.client.1.smithi188.stdout:1/528: fsync d1/d7/d4e/f8b 0 2024-04-05T21:26:36.771 INFO:tasks.workunit.client.1.smithi188.stdout:1/529: rmdir d1/dc/d16 39 2024-04-05T21:26:36.782 INFO:tasks.workunit.client.0.smithi121.stdout:9/192: sync 2024-04-05T21:26:36.783 INFO:tasks.workunit.client.0.smithi121.stdout:8/164: sync 2024-04-05T21:26:36.784 INFO:tasks.workunit.client.0.smithi121.stdout:3/180: truncate d0/f14 1521684 0 2024-04-05T21:26:36.785 INFO:tasks.workunit.client.1.smithi188.stdout:1/530: dwrite d1/dc/d16/f3b [0,4194304] 0 2024-04-05T21:26:36.785 INFO:tasks.workunit.client.0.smithi121.stdout:8/165: symlink d12/d13/l37 0 2024-04-05T21:26:36.788 INFO:tasks.workunit.client.1.smithi188.stdout:1/531: rmdir d1/dc/d16/d46/d95/d99 39 2024-04-05T21:26:36.789 INFO:tasks.workunit.client.0.smithi121.stdout:4/232: dread d5/d11/f17 [0,4194304] 0 2024-04-05T21:26:36.791 INFO:tasks.workunit.client.1.smithi188.stdout:1/532: chown d1/dc/d16/d21/d27/d4a 12 1 2024-04-05T21:26:36.794 INFO:tasks.workunit.client.0.smithi121.stdout:9/193: creat d6/df/d1d/d2c/d37/d3a/f3b x:0 0 0 2024-04-05T21:26:36.796 INFO:tasks.workunit.client.0.smithi121.stdout:5/240: write dd/de/d19/d13/f2d [3452500,41133] 0 2024-04-05T21:26:36.796 INFO:tasks.workunit.client.0.smithi121.stdout:8/166: chown d12/d13/d15/c1e 1 1 2024-04-05T21:26:36.797 INFO:tasks.workunit.client.1.smithi188.stdout:1/533: link d1/dc/d16/f6f d1/dc/d16/d21/d27/d4a/fad 0 2024-04-05T21:26:36.798 INFO:tasks.workunit.client.0.smithi121.stdout:4/233: symlink d5/d11/d26/d27/d29/d2e/l4f 0 2024-04-05T21:26:36.798 INFO:tasks.workunit.client.0.smithi121.stdout:7/225: dwrite d0/d9/f20 [0,4194304] 0 2024-04-05T21:26:36.805 INFO:tasks.workunit.client.0.smithi121.stdout:3/181: dread d0/f7 [0,4194304] 0 2024-04-05T21:26:36.805 INFO:tasks.workunit.client.0.smithi121.stdout:8/167: creat d12/d30/f38 x:0 0 0 2024-04-05T21:26:36.809 INFO:tasks.workunit.client.0.smithi121.stdout:9/194: dwrite d6/f16 [0,4194304] 0 2024-04-05T21:26:36.812 INFO:tasks.workunit.client.0.smithi121.stdout:4/234: write d5/d11/d28/fe [4070763,5405] 0 2024-04-05T21:26:36.815 INFO:tasks.workunit.client.0.smithi121.stdout:3/182: creat d0/d3/dc/f3a x:0 0 0 2024-04-05T21:26:36.815 INFO:tasks.workunit.client.1.smithi188.stdout:1/534: dread d1/dc/f23 [0,4194304] 0 2024-04-05T21:26:36.817 INFO:tasks.workunit.client.0.smithi121.stdout:9/195: fsync d6/df/d1d/f23 0 2024-04-05T21:26:36.817 INFO:tasks.workunit.client.0.smithi121.stdout:8/168: mknod d12/c39 0 2024-04-05T21:26:36.820 INFO:tasks.workunit.client.0.smithi121.stdout:7/226: dwrite d0/d23/f2e [4194304,4194304] 0 2024-04-05T21:26:36.821 INFO:tasks.workunit.client.1.smithi188.stdout:1/535: read d1/dc/d16/d46/d88/d13/f8d [152479,48690] 0 2024-04-05T21:26:36.823 INFO:tasks.workunit.client.1.smithi188.stdout:1/536: truncate d1/dc/d16/d2d/f7e 1119632 0 2024-04-05T21:26:36.823 INFO:tasks.workunit.client.0.smithi121.stdout:3/183: mkdir d0/d3/de/d3b 0 2024-04-05T21:26:36.824 INFO:tasks.workunit.client.1.smithi188.stdout:1/537: chown d1/dc/d16/d2d/f42 3909319 1 2024-04-05T21:26:36.826 INFO:tasks.workunit.client.0.smithi121.stdout:5/241: creat dd/de/d19/d1c/f48 x:0 0 0 2024-04-05T21:26:36.826 INFO:tasks.workunit.client.0.smithi121.stdout:8/169: truncate d12/d13/f1b 611010 0 2024-04-05T21:26:36.826 INFO:tasks.workunit.client.1.smithi188.stdout:1/538: symlink d1/dc/d16/d46/d88/d13/lae 0 2024-04-05T21:26:36.828 INFO:tasks.workunit.client.0.smithi121.stdout:9/196: mkdir d6/df/d1d/d2c/d37/d3c 0 2024-04-05T21:26:36.830 INFO:tasks.workunit.client.0.smithi121.stdout:3/184: rmdir d0/d3/dc 39 2024-04-05T21:26:36.830 INFO:tasks.workunit.client.0.smithi121.stdout:9/197: fsync d6/df/d1d/d2c/f24 0 2024-04-05T21:26:36.832 INFO:tasks.workunit.client.0.smithi121.stdout:7/227: creat d0/d9/d22/d28/d3b/f4f x:0 0 0 2024-04-05T21:26:36.833 INFO:tasks.workunit.client.0.smithi121.stdout:5/242: readlink dd/de/d19/d13/l24 0 2024-04-05T21:26:36.833 INFO:tasks.workunit.client.1.smithi188.stdout:1/539: truncate d1/dc/d16/d46/f51 120292 0 2024-04-05T21:26:36.835 INFO:tasks.workunit.client.0.smithi121.stdout:7/228: read d0/d9/f26 [1636933,78933] 0 2024-04-05T21:26:36.835 INFO:tasks.workunit.client.0.smithi121.stdout:5/243: truncate dd/de/d19/d13/f3f 113237 0 2024-04-05T21:26:36.837 INFO:tasks.workunit.client.0.smithi121.stdout:4/235: link d5/d11/d28/d3b/c3e d5/d11/d26/d27/d29/d2d/c50 0 2024-04-05T21:26:36.838 INFO:tasks.workunit.client.0.smithi121.stdout:9/198: creat d6/df/d1d/d2c/d37/d3c/f3d x:0 0 0 2024-04-05T21:26:36.839 INFO:tasks.workunit.client.0.smithi121.stdout:4/236: write d5/d11/f48 [298162,129841] 0 2024-04-05T21:26:36.840 INFO:tasks.workunit.client.1.smithi188.stdout:1/540: dread d1/dc/d16/f3c [0,4194304] 0 2024-04-05T21:26:36.843 INFO:tasks.workunit.client.1.smithi188.stdout:1/541: symlink d1/dc/d16/d46/d88/d3f/laf 0 2024-04-05T21:26:36.843 INFO:tasks.workunit.client.0.smithi121.stdout:7/229: link d0/d9/f3a d0/d9/f50 0 2024-04-05T21:26:36.844 INFO:tasks.workunit.client.0.smithi121.stdout:7/230: readlink d0/d9/l19 0 2024-04-05T21:26:36.844 INFO:tasks.workunit.client.0.smithi121.stdout:3/185: dread d0/d3/dc/d1a/f22 [0,4194304] 0 2024-04-05T21:26:36.845 INFO:tasks.workunit.client.0.smithi121.stdout:9/199: mknod d6/df/d1d/d2c/d37/c3e 0 2024-04-05T21:26:36.847 INFO:tasks.workunit.client.0.smithi121.stdout:5/244: getdents dd/de/d19/d38 0 2024-04-05T21:26:36.849 INFO:tasks.workunit.client.1.smithi188.stdout:1/542: dwrite d1/dc/d16/d21/f81 [0,4194304] 0 2024-04-05T21:26:36.849 INFO:tasks.workunit.client.0.smithi121.stdout:3/186: creat d0/d3/d24/f3c x:0 0 0 2024-04-05T21:26:36.851 INFO:tasks.workunit.client.0.smithi121.stdout:7/231: creat d0/d9/d22/d28/f51 x:0 0 0 2024-04-05T21:26:36.851 INFO:tasks.workunit.client.0.smithi121.stdout:3/187: read d0/d3/de/f17 [2682592,70261] 0 2024-04-05T21:26:36.852 INFO:tasks.workunit.client.0.smithi121.stdout:9/200: unlink d6/df/d1d/d2c/c28 0 2024-04-05T21:26:36.852 INFO:tasks.workunit.client.0.smithi121.stdout:7/232: chown d0/d2d/f43 517446 1 2024-04-05T21:26:36.854 INFO:tasks.workunit.client.1.smithi188.stdout:1/543: truncate d1/d7/d4e/f8b 385881 0 2024-04-05T21:26:36.854 INFO:tasks.workunit.client.0.smithi121.stdout:3/188: write d0/d3/f4 [5841044,77834] 0 2024-04-05T21:26:36.854 INFO:tasks.workunit.client.0.smithi121.stdout:8/170: dread ff [0,4194304] 0 2024-04-05T21:26:36.855 INFO:tasks.workunit.client.0.smithi121.stdout:7/233: fsync d0/d2d/f4b 0 2024-04-05T21:26:36.856 INFO:tasks.workunit.client.0.smithi121.stdout:5/245: rename dd/f1d to dd/de/d19/d13/f49 0 2024-04-05T21:26:36.856 INFO:tasks.workunit.client.1.smithi188.stdout:1/544: mkdir d1/dc/d16/d2d/db0 0 2024-04-05T21:26:36.856 INFO:tasks.workunit.client.0.smithi121.stdout:8/171: readlink d12/l26 0 2024-04-05T21:26:36.859 INFO:tasks.workunit.client.0.smithi121.stdout:7/234: mkdir d0/d2d/d52 0 2024-04-05T21:26:36.861 INFO:tasks.workunit.client.0.smithi121.stdout:5/246: unlink dd/de/d39/f44 0 2024-04-05T21:26:36.861 INFO:tasks.workunit.client.1.smithi188.stdout:1/545: rename d1/d7/d4e/c96 to d1/dc/d16/d46/d88/d3f/cb1 0 2024-04-05T21:26:36.862 INFO:tasks.workunit.client.1.smithi188.stdout:1/546: dread - d1/dc/d16/d21/d27/d4a/f8c zero size 2024-04-05T21:26:36.862 INFO:tasks.workunit.client.0.smithi121.stdout:3/189: rename d0/d3/dc to d0/d3/d24/d3d 0 2024-04-05T21:26:36.863 INFO:tasks.workunit.client.1.smithi188.stdout:1/547: read d1/dc/d16/d21/f81 [1313584,58486] 0 2024-04-05T21:26:36.863 INFO:tasks.workunit.client.0.smithi121.stdout:8/172: rmdir d12/d13/d15/d1d 39 2024-04-05T21:26:36.864 INFO:tasks.workunit.client.1.smithi188.stdout:1/548: chown d1/dc/d16/d46/d88/d13/f50 1632 1 2024-04-05T21:26:36.866 INFO:tasks.workunit.client.0.smithi121.stdout:7/235: write d0/d9/d22/f44 [777553,90938] 0 2024-04-05T21:26:36.867 INFO:tasks.workunit.client.0.smithi121.stdout:7/236: readlink d0/l17 0 2024-04-05T21:26:36.868 INFO:tasks.workunit.client.1.smithi188.stdout:1/549: mkdir d1/db2 0 2024-04-05T21:26:36.869 INFO:tasks.workunit.client.0.smithi121.stdout:7/237: chown d0/d2d/f43 288082911 1 2024-04-05T21:26:36.869 INFO:tasks.workunit.client.0.smithi121.stdout:4/237: dread d5/d11/f12 [0,4194304] 0 2024-04-05T21:26:36.869 INFO:tasks.workunit.client.0.smithi121.stdout:5/247: rename dd/de/d19/f32 to dd/de/d39/d3b/f4a 0 2024-04-05T21:26:36.870 INFO:tasks.workunit.client.1.smithi188.stdout:1/550: unlink d1/dc/d16/d2d/l61 0 2024-04-05T21:26:36.870 INFO:tasks.workunit.client.0.smithi121.stdout:8/173: creat d12/d30/f3a x:0 0 0 2024-04-05T21:26:36.871 INFO:tasks.workunit.client.1.smithi188.stdout:1/551: fsync d1/dc/d16/d2d/f7e 0 2024-04-05T21:26:36.875 INFO:tasks.workunit.client.0.smithi121.stdout:7/238: write d0/f6 [263412,14749] 0 2024-04-05T21:26:36.876 INFO:tasks.workunit.client.0.smithi121.stdout:4/238: chown d5/f6 223 1 2024-04-05T21:26:36.877 INFO:tasks.workunit.client.0.smithi121.stdout:4/239: dread - d5/d11/d26/d27/d29/d2d/f31 zero size 2024-04-05T21:26:36.878 INFO:tasks.workunit.client.0.smithi121.stdout:5/248: symlink dd/de/l4b 0 2024-04-05T21:26:36.879 INFO:tasks.workunit.client.0.smithi121.stdout:8/174: mknod d12/d13/c3b 0 2024-04-05T21:26:36.879 INFO:tasks.workunit.client.0.smithi121.stdout:9/201: link d6/df/d1d/d2c/l30 d6/l3f 0 2024-04-05T21:26:36.884 INFO:tasks.workunit.client.0.smithi121.stdout:5/249: dwrite dd/de/f22 [0,4194304] 0 2024-04-05T21:26:36.886 INFO:tasks.workunit.client.0.smithi121.stdout:7/239: creat d0/d32/d4a/f53 x:0 0 0 2024-04-05T21:26:36.888 INFO:tasks.workunit.client.1.smithi188.stdout:4/737: sync 2024-04-05T21:26:36.888 INFO:tasks.workunit.client.1.smithi188.stdout:9/477: sync 2024-04-05T21:26:36.888 INFO:tasks.workunit.client.1.smithi188.stdout:8/402: sync 2024-04-05T21:26:36.888 INFO:tasks.workunit.client.0.smithi121.stdout:4/240: symlink d5/d11/d26/l51 0 2024-04-05T21:26:36.889 INFO:tasks.workunit.client.0.smithi121.stdout:5/250: chown dd/l20 2389491 1 2024-04-05T21:26:36.891 INFO:tasks.workunit.client.0.smithi121.stdout:9/202: creat d6/f40 x:0 0 0 2024-04-05T21:26:36.892 INFO:tasks.workunit.client.1.smithi188.stdout:9/478: truncate dd/d3d/f4c 1203427 0 2024-04-05T21:26:36.892 INFO:tasks.workunit.client.0.smithi121.stdout:7/240: rename d0/d9/d22/d28/f51 to d0/d23/f54 0 2024-04-05T21:26:36.893 INFO:tasks.workunit.client.0.smithi121.stdout:4/241: truncate d5/f3f 78718 0 2024-04-05T21:26:36.893 INFO:tasks.workunit.client.1.smithi188.stdout:9/479: fsync dd/d11/d22/f4d 0 2024-04-05T21:26:36.894 INFO:tasks.workunit.client.1.smithi188.stdout:8/403: mknod d5/d6e/d86/c94 0 2024-04-05T21:26:36.895 INFO:tasks.workunit.client.1.smithi188.stdout:4/738: getdents d0/d14/d2f/ddd 0 2024-04-05T21:26:36.898 INFO:tasks.workunit.client.0.smithi121.stdout:9/203: unlink d6/df/f19 0 2024-04-05T21:26:36.898 INFO:tasks.workunit.client.0.smithi121.stdout:3/190: write d0/f14 [586710,81456] 0 2024-04-05T21:26:36.899 INFO:tasks.workunit.client.1.smithi188.stdout:4/739: dread d0/d14/d15/d53/d82/dd5/ff1 [0,4194304] 0 2024-04-05T21:26:36.899 INFO:tasks.workunit.client.1.smithi188.stdout:8/404: mkdir d5/d21/d2a/d95 0 2024-04-05T21:26:36.900 INFO:tasks.workunit.client.0.smithi121.stdout:4/242: mknod d5/d11/d28/c52 0 2024-04-05T21:26:36.900 INFO:tasks.workunit.client.0.smithi121.stdout:8/175: dread f3 [0,4194304] 0 2024-04-05T21:26:36.900 INFO:tasks.workunit.client.1.smithi188.stdout:9/480: creat dd/d11/d22/f9c x:0 0 0 2024-04-05T21:26:36.901 INFO:tasks.workunit.client.0.smithi121.stdout:7/241: symlink d0/d2d/d52/l55 0 2024-04-05T21:26:36.901 INFO:tasks.workunit.client.0.smithi121.stdout:5/251: dwrite dd/de/d19/d13/f2d [4194304,4194304] 0 2024-04-05T21:26:36.904 INFO:tasks.workunit.client.0.smithi121.stdout:4/243: mknod d5/d11/d2c/c53 0 2024-04-05T21:26:36.904 INFO:tasks.workunit.client.1.smithi188.stdout:8/405: creat d5/d21/d80/f96 x:0 0 0 2024-04-05T21:26:36.905 INFO:tasks.workunit.client.1.smithi188.stdout:1/552: dwrite d1/dc/d16/d46/d88/f12 [4194304,4194304] 0 2024-04-05T21:26:36.905 INFO:tasks.workunit.client.0.smithi121.stdout:7/242: fsync d0/f1f 0 2024-04-05T21:26:36.906 INFO:tasks.workunit.client.0.smithi121.stdout:4/244: read d5/d11/d28/f13 [386280,104104] 0 2024-04-05T21:26:36.907 INFO:tasks.workunit.client.1.smithi188.stdout:4/740: creat d0/d2/dc7/d104/f105 x:0 0 0 2024-04-05T21:26:36.908 INFO:tasks.workunit.client.0.smithi121.stdout:5/252: mknod dd/c4c 0 2024-04-05T21:26:36.910 INFO:tasks.workunit.client.1.smithi188.stdout:4/741: read - d0/d2/d52/dcc/ff2 zero size 2024-04-05T21:26:36.911 INFO:tasks.workunit.client.0.smithi121.stdout:7/243: mkdir d0/d9/d22/d28/d56 0 2024-04-05T21:26:36.913 INFO:tasks.workunit.client.1.smithi188.stdout:1/553: rename d1/d7/d4e/f62 to d1/dc/d16/d46/d95/d99/fb3 0 2024-04-05T21:26:36.914 INFO:tasks.workunit.client.0.smithi121.stdout:8/176: rename d12/d13/d15/c1e to d12/d13/d15/d1d/c3c 0 2024-04-05T21:26:36.914 INFO:tasks.workunit.client.1.smithi188.stdout:9/481: write dd/f42 [2762909,125523] 0 2024-04-05T21:26:36.915 INFO:tasks.workunit.client.0.smithi121.stdout:8/177: write d12/d13/d15/f19 [2683973,18882] 0 2024-04-05T21:26:36.915 INFO:tasks.workunit.client.1.smithi188.stdout:4/742: rmdir d0/d2/d52/dcc 39 2024-04-05T21:26:36.915 INFO:tasks.workunit.client.1.smithi188.stdout:1/554: chown d1/dc/d16/d21/d27/d49/f73 0 1 2024-04-05T21:26:36.917 INFO:tasks.workunit.client.0.smithi121.stdout:8/178: fsync d12/d30/f3a 0 2024-04-05T21:26:36.919 INFO:tasks.workunit.client.1.smithi188.stdout:8/406: creat d5/d21/d2a/f97 x:0 0 0 2024-04-05T21:26:36.920 INFO:tasks.workunit.client.0.smithi121.stdout:5/253: creat dd/de/d19/d38/f4d x:0 0 0 2024-04-05T21:26:36.922 INFO:tasks.workunit.client.1.smithi188.stdout:1/555: creat d1/db2/fb4 x:0 0 0 2024-04-05T21:26:36.925 INFO:tasks.workunit.client.1.smithi188.stdout:4/743: mkdir d0/d14/d15/d53/d82/dd5/df4/d106 0 2024-04-05T21:26:36.926 INFO:tasks.workunit.client.0.smithi121.stdout:7/244: mkdir d0/d57 0 2024-04-05T21:26:36.927 INFO:tasks.workunit.client.0.smithi121.stdout:8/179: fsync d12/d13/d15/f1f 0 2024-04-05T21:26:36.928 INFO:tasks.workunit.client.1.smithi188.stdout:8/407: dwrite d5/dc/f78 [0,4194304] 0 2024-04-05T21:26:36.928 INFO:tasks.workunit.client.0.smithi121.stdout:7/245: read d0/f8 [4169853,71996] 0 2024-04-05T21:26:36.929 INFO:tasks.workunit.client.1.smithi188.stdout:1/556: rename d1/d7/d4e/fa2 to d1/dc/d16/d46/d95/fb5 0 2024-04-05T21:26:36.929 INFO:tasks.workunit.client.0.smithi121.stdout:5/254: creat dd/de/d19/d38/d45/f4e x:0 0 0 2024-04-05T21:26:36.929 INFO:tasks.workunit.client.0.smithi121.stdout:7/246: chown d0/d2d 273336 1 2024-04-05T21:26:36.930 INFO:tasks.workunit.client.1.smithi188.stdout:8/408: mkdir d5/d6e/d98 0 2024-04-05T21:26:36.931 INFO:tasks.workunit.client.0.smithi121.stdout:3/191: dwrite d0/d3/de/f17 [0,4194304] 0 2024-04-05T21:26:36.931 INFO:tasks.workunit.client.1.smithi188.stdout:1/557: read - d1/dc/d16/d46/d88/d13/f82 zero size 2024-04-05T21:26:36.933 INFO:tasks.workunit.client.0.smithi121.stdout:7/247: truncate d0/d9/d22/f44 1463512 0 2024-04-05T21:26:36.937 INFO:tasks.workunit.client.0.smithi121.stdout:5/255: dread dd/de/d19/f37 [0,4194304] 0 2024-04-05T21:26:36.938 INFO:tasks.workunit.client.0.smithi121.stdout:7/248: truncate d0/f1f 43060 0 2024-04-05T21:26:36.939 INFO:tasks.workunit.client.1.smithi188.stdout:8/409: rename d5/d21/d50/d63 to d5/d21/d40/d43/d44/d5b/d99 0 2024-04-05T21:26:36.940 INFO:tasks.workunit.client.1.smithi188.stdout:8/410: chown d5/d21/l2b 2724 1 2024-04-05T21:26:36.941 INFO:tasks.workunit.client.0.smithi121.stdout:7/249: dread - d0/d9/d22/d28/f31 zero size 2024-04-05T21:26:36.942 INFO:tasks.workunit.client.0.smithi121.stdout:5/256: creat dd/de/d19/d38/f4f x:0 0 0 2024-04-05T21:26:36.943 INFO:tasks.workunit.client.0.smithi121.stdout:4/245: dwrite d5/d11/d28/d3b/f4a [0,4194304] 0 2024-04-05T21:26:36.945 INFO:tasks.workunit.client.1.smithi188.stdout:1/558: write d1/dc/f23 [1904381,66592] 0 2024-04-05T21:26:36.946 INFO:tasks.workunit.client.0.smithi121.stdout:4/246: write d5/d11/d28/fe [2667990,118364] 0 2024-04-05T21:26:36.947 INFO:tasks.workunit.client.1.smithi188.stdout:1/559: creat d1/d7/d4e/fb6 x:0 0 0 2024-04-05T21:26:36.949 INFO:tasks.workunit.client.0.smithi121.stdout:4/247: write d5/d11/f35 [2105920,54069] 0 2024-04-05T21:26:36.955 INFO:tasks.workunit.client.1.smithi188.stdout:8/411: rmdir d5 39 2024-04-05T21:26:36.957 INFO:tasks.workunit.client.0.smithi121.stdout:5/257: dwrite dd/de/f2e [0,4194304] 0 2024-04-05T21:26:36.962 INFO:tasks.workunit.client.0.smithi121.stdout:8/180: dwrite ff [0,4194304] 0 2024-04-05T21:26:36.963 INFO:tasks.workunit.client.0.smithi121.stdout:7/250: dwrite d0/f8 [0,4194304] 0 2024-04-05T21:26:36.965 INFO:tasks.workunit.client.0.smithi121.stdout:5/258: mkdir dd/de/d39/d3b/d50 0 2024-04-05T21:26:36.965 INFO:tasks.workunit.client.0.smithi121.stdout:7/251: stat d0/d2d/f2f 0 2024-04-05T21:26:36.966 INFO:tasks.workunit.client.0.smithi121.stdout:8/181: readlink d12/d13/d15/d1d/l2d 0 2024-04-05T21:26:36.969 INFO:tasks.workunit.client.0.smithi121.stdout:7/252: write d0/d32/f35 [4379797,38682] 0 2024-04-05T21:26:36.971 INFO:tasks.workunit.client.0.smithi121.stdout:7/253: read d0/d9/d22/f44 [808263,39827] 0 2024-04-05T21:26:36.972 INFO:tasks.workunit.client.1.smithi188.stdout:8/412: truncate d5/dc/f17 446410 0 2024-04-05T21:26:36.973 INFO:tasks.workunit.client.1.smithi188.stdout:8/413: stat d5/d21/d40/d43/c5c 0 2024-04-05T21:26:36.974 INFO:tasks.workunit.client.0.smithi121.stdout:5/259: dread - dd/de/d19/d38/f4f zero size 2024-04-05T21:26:36.974 INFO:tasks.workunit.client.0.smithi121.stdout:8/182: rename d12/d30/f38 to d12/f3d 0 2024-04-05T21:26:36.976 INFO:tasks.workunit.client.1.smithi188.stdout:8/414: link d5/l6 d5/d21/d40/d43/l9a 0 2024-04-05T21:26:36.977 INFO:tasks.workunit.client.1.smithi188.stdout:8/415: stat d5/dc/l3f 0 2024-04-05T21:26:36.977 INFO:tasks.workunit.client.0.smithi121.stdout:5/260: dread dd/de/d19/f37 [0,4194304] 0 2024-04-05T21:26:36.978 INFO:tasks.workunit.client.1.smithi188.stdout:8/416: read - d5/d21/d2a/d47/f54 zero size 2024-04-05T21:26:36.982 INFO:tasks.workunit.client.0.smithi121.stdout:7/254: dwrite d0/d9/f1d [0,4194304] 0 2024-04-05T21:26:36.983 INFO:tasks.workunit.client.0.smithi121.stdout:7/255: chown d0/f15 3 1 2024-04-05T21:26:36.984 INFO:tasks.workunit.client.0.smithi121.stdout:5/261: rename dd/de/d19/d3c/f42 to dd/de/d19/f51 0 2024-04-05T21:26:36.984 INFO:tasks.workunit.client.0.smithi121.stdout:7/256: dread - d0/d9/f50 zero size 2024-04-05T21:26:36.985 INFO:tasks.workunit.client.1.smithi188.stdout:8/417: truncate d5/f12 570935 0 2024-04-05T21:26:36.986 INFO:tasks.workunit.client.1.smithi188.stdout:8/418: fsync d5/d21/d40/d43/f87 0 2024-04-05T21:26:36.987 INFO:tasks.workunit.client.0.smithi121.stdout:5/262: symlink dd/de/d39/l52 0 2024-04-05T21:26:36.988 INFO:tasks.workunit.client.0.smithi121.stdout:5/263: write dd/de/f34 [39428,86712] 0 2024-04-05T21:26:36.989 INFO:tasks.workunit.client.1.smithi188.stdout:8/419: read d5/d21/d40/f73 [72659,116737] 0 2024-04-05T21:26:36.989 INFO:tasks.workunit.client.0.smithi121.stdout:7/257: creat d0/d9/d22/d28/d56/f58 x:0 0 0 2024-04-05T21:26:36.990 INFO:tasks.workunit.client.1.smithi188.stdout:8/420: dread - d5/d21/f91 zero size 2024-04-05T21:26:36.990 INFO:tasks.workunit.client.0.smithi121.stdout:7/258: fdatasync d0/d2d/f2f 0 2024-04-05T21:26:36.991 INFO:tasks.workunit.client.0.smithi121.stdout:5/264: mkdir dd/de/d19/d13/d53 0 2024-04-05T21:26:36.991 INFO:tasks.workunit.client.0.smithi121.stdout:7/259: write d0/d9/d22/d28/d56/f58 [255211,79955] 0 2024-04-05T21:26:36.993 INFO:tasks.workunit.client.0.smithi121.stdout:5/265: creat dd/de/d19/d38/d45/f54 x:0 0 0 2024-04-05T21:26:36.996 INFO:tasks.workunit.client.0.smithi121.stdout:4/248: dread d5/d11/d28/f2f [0,4194304] 0 2024-04-05T21:26:36.997 INFO:tasks.workunit.client.1.smithi188.stdout:8/421: truncate d5/f15 1020814 0 2024-04-05T21:26:36.998 INFO:tasks.workunit.client.0.smithi121.stdout:5/266: symlink dd/de/d19/d38/d45/l55 0 2024-04-05T21:26:36.998 INFO:tasks.workunit.client.1.smithi188.stdout:8/422: write d5/f34 [1158830,66472] 0 2024-04-05T21:26:36.999 INFO:tasks.workunit.client.0.smithi121.stdout:4/249: write d5/d11/d26/d27/f30 [344252,77452] 0 2024-04-05T21:26:37.000 INFO:tasks.workunit.client.0.smithi121.stdout:7/260: dwrite d0/fc [4194304,4194304] 0 2024-04-05T21:26:37.000 INFO:tasks.workunit.client.1.smithi188.stdout:8/423: rmdir d5/d21/d40/d43/d44/d59/d79 39 2024-04-05T21:26:37.001 INFO:tasks.workunit.client.0.smithi121.stdout:7/261: chown d0/d32/f41 910087613 1 2024-04-05T21:26:37.002 INFO:tasks.workunit.client.0.smithi121.stdout:7/262: write d0/d9/d22/d28/d56/f58 [761275,14491] 0 2024-04-05T21:26:37.003 INFO:tasks.workunit.client.0.smithi121.stdout:4/250: symlink d5/d11/d28/d3b/l54 0 2024-04-05T21:26:37.004 INFO:tasks.workunit.client.1.smithi188.stdout:8/424: mkdir d5/d21/d40/d43/d44/d5b/d99/d69/d9b 0 2024-04-05T21:26:37.004 INFO:tasks.workunit.client.0.smithi121.stdout:7/263: chown d0/d9/c1e 63357 1 2024-04-05T21:26:37.006 INFO:tasks.workunit.client.0.smithi121.stdout:4/251: creat d5/f55 x:0 0 0 2024-04-05T21:26:37.007 INFO:tasks.workunit.client.0.smithi121.stdout:7/264: unlink d0/fc 0 2024-04-05T21:26:37.010 INFO:tasks.workunit.client.0.smithi121.stdout:7/265: write d0/d9/d22/d28/f48 [901391,84360] 0 2024-04-05T21:26:37.010 INFO:tasks.workunit.client.0.smithi121.stdout:5/267: dwrite dd/de/d19/f51 [0,4194304] 0 2024-04-05T21:26:37.011 INFO:tasks.workunit.client.0.smithi121.stdout:4/252: symlink d5/d11/d26/d49/l56 0 2024-04-05T21:26:37.012 INFO:tasks.workunit.client.0.smithi121.stdout:7/266: dread - d0/d23/f47 zero size 2024-04-05T21:26:37.012 INFO:tasks.workunit.client.1.smithi188.stdout:8/425: dwrite d5/d21/d40/d43/d44/d59/f5a [0,4194304] 0 2024-04-05T21:26:37.013 INFO:tasks.workunit.client.0.smithi121.stdout:7/267: dread - d0/d9/f3a zero size 2024-04-05T21:26:37.014 INFO:tasks.workunit.client.1.smithi188.stdout:8/426: write d5/d21/d2a/f97 [27196,18799] 0 2024-04-05T21:26:37.014 INFO:tasks.workunit.client.0.smithi121.stdout:5/268: mkdir dd/de/d19/d38/d56 0 2024-04-05T21:26:37.016 INFO:tasks.workunit.client.0.smithi121.stdout:7/268: write d0/d9/d22/d3e/f3c [687189,69880] 0 2024-04-05T21:26:37.019 INFO:tasks.workunit.client.0.smithi121.stdout:5/269: creat dd/de/d19/f57 x:0 0 0 2024-04-05T21:26:37.019 INFO:tasks.workunit.client.1.smithi188.stdout:8/427: mknod d5/d21/d40/d43/d44/c9c 0 2024-04-05T21:26:37.021 INFO:tasks.workunit.client.0.smithi121.stdout:7/269: read d0/d2d/f4b [480259,42361] 0 2024-04-05T21:26:37.022 INFO:tasks.workunit.client.0.smithi121.stdout:4/253: dwrite d5/d11/d26/d27/d29/d2d/f40 [0,4194304] 0 2024-04-05T21:26:37.024 INFO:tasks.workunit.client.0.smithi121.stdout:5/270: chown dd/d41 233469692 1 2024-04-05T21:26:37.025 INFO:tasks.workunit.client.0.smithi121.stdout:7/270: creat d0/d2d/f59 x:0 0 0 2024-04-05T21:26:37.027 INFO:tasks.workunit.client.0.smithi121.stdout:4/254: creat d5/d11/d20/f57 x:0 0 0 2024-04-05T21:26:37.028 INFO:tasks.workunit.client.0.smithi121.stdout:5/271: truncate dd/de/d19/d13/f35 705770 0 2024-04-05T21:26:37.028 INFO:tasks.workunit.client.1.smithi188.stdout:8/428: dwrite d5/d21/d40/d43/d44/d59/f74 [0,4194304] 0 2024-04-05T21:26:37.031 INFO:tasks.workunit.client.0.smithi121.stdout:4/255: mkdir d5/d11/d26/d27/d29/d58 0 2024-04-05T21:26:37.033 INFO:tasks.workunit.client.1.smithi188.stdout:8/429: rename d5/f9 to d5/d21/d40/d43/d44/d5b/d99/d69/d70/f9d 0 2024-04-05T21:26:37.034 INFO:tasks.workunit.client.1.smithi188.stdout:8/430: dread - d5/d21/d80/f96 zero size 2024-04-05T21:26:37.038 INFO:tasks.workunit.client.1.smithi188.stdout:8/431: dread d5/d21/d40/f4f [0,4194304] 0 2024-04-05T21:26:37.038 INFO:tasks.workunit.client.0.smithi121.stdout:7/271: dwrite d0/f42 [0,4194304] 0 2024-04-05T21:26:37.039 INFO:tasks.workunit.client.1.smithi188.stdout:8/432: truncate d5/dc/f32 5990149 0 2024-04-05T21:26:37.039 INFO:tasks.workunit.client.1.smithi188.stdout:8/433: fsync d5/f2f 0 2024-04-05T21:26:37.041 INFO:tasks.workunit.client.0.smithi121.stdout:7/272: mkdir d0/d23/d5a 0 2024-04-05T21:26:37.043 INFO:tasks.workunit.client.0.smithi121.stdout:7/273: truncate d0/d9/f3a 570188 0 2024-04-05T21:26:37.049 INFO:tasks.workunit.client.1.smithi188.stdout:8/434: dwrite d5/d21/d40/d43/f81 [0,4194304] 0 2024-04-05T21:26:37.050 INFO:tasks.workunit.client.1.smithi188.stdout:8/435: stat d5/d21/d80/f96 0 2024-04-05T21:26:37.051 INFO:tasks.workunit.client.0.smithi121.stdout:5/272: truncate dd/de/d19/f37 3084674 0 2024-04-05T21:26:37.052 INFO:tasks.workunit.client.0.smithi121.stdout:7/274: dread d0/d32/f41 [0,4194304] 0 2024-04-05T21:26:37.055 INFO:tasks.workunit.client.0.smithi121.stdout:5/273: fdatasync dd/de/d19/d13/f16 0 2024-04-05T21:26:37.058 INFO:tasks.workunit.client.0.smithi121.stdout:4/256: dwrite d5/f16 [0,4194304] 0 2024-04-05T21:26:37.061 INFO:tasks.workunit.client.0.smithi121.stdout:5/274: rename dd/de/d39/l52 to dd/de/d19/d1c/l58 0 2024-04-05T21:26:37.061 INFO:tasks.workunit.client.1.smithi188.stdout:8/436: dwrite d5/d21/d40/d43/f87 [0,4194304] 0 2024-04-05T21:26:37.063 INFO:tasks.workunit.client.0.smithi121.stdout:4/257: rename d5/f6 to d5/d11/d28/d3b/f59 0 2024-04-05T21:26:37.067 INFO:tasks.workunit.client.1.smithi188.stdout:8/437: mkdir d5/d21/d40/d43/d44/d59/d9e 0 2024-04-05T21:26:37.067 INFO:tasks.workunit.client.0.smithi121.stdout:1/201: sync 2024-04-05T21:26:37.067 INFO:tasks.workunit.client.0.smithi121.stdout:6/170: sync 2024-04-05T21:26:37.067 INFO:tasks.workunit.client.0.smithi121.stdout:6/171: chown d1/d2/d1b/d1e 30736173 1 2024-04-05T21:26:37.069 INFO:tasks.workunit.client.0.smithi121.stdout:6/172: mkdir d1/d2/d1b/d35 0 2024-04-05T21:26:37.069 INFO:tasks.workunit.client.0.smithi121.stdout:4/258: dread d5/d11/f48 [0,4194304] 0 2024-04-05T21:26:37.070 INFO:tasks.workunit.client.0.smithi121.stdout:6/173: fdatasync d1/d2/d3/d4/de/f26 0 2024-04-05T21:26:37.072 INFO:tasks.workunit.client.1.smithi188.stdout:8/438: creat d5/d21/d40/d43/d44/d59/d8b/f9f x:0 0 0 2024-04-05T21:26:37.072 INFO:tasks.workunit.client.0.smithi121.stdout:4/259: mknod d5/d11/d26/d27/c5a 0 2024-04-05T21:26:37.075 INFO:tasks.workunit.client.1.smithi188.stdout:8/439: unlink d5/d21/d40/d43/d44/d59/d8b/f9f 0 2024-04-05T21:26:37.077 INFO:tasks.workunit.client.0.smithi121.stdout:6/174: dread d1/d2/d3/d5/d1f/f27 [0,4194304] 0 2024-04-05T21:26:37.078 INFO:tasks.workunit.client.1.smithi188.stdout:8/440: creat d5/d21/d50/fa0 x:0 0 0 2024-04-05T21:26:37.089 INFO:tasks.workunit.client.1.smithi188.stdout:8/441: dread d5/f1c [0,4194304] 0 2024-04-05T21:26:37.091 INFO:tasks.workunit.client.1.smithi188.stdout:8/442: mknod d5/d4b/ca1 0 2024-04-05T21:26:37.094 INFO:tasks.workunit.client.1.smithi188.stdout:8/443: dread d5/d21/d40/d43/d44/d59/f77 [0,4194304] 0 2024-04-05T21:26:37.096 INFO:tasks.workunit.client.1.smithi188.stdout:8/444: creat d5/d21/fa2 x:0 0 0 2024-04-05T21:26:37.096 INFO:tasks.workunit.client.1.smithi188.stdout:8/445: write d5/f24 [2081049,58296] 0 2024-04-05T21:26:37.099 INFO:tasks.workunit.client.1.smithi188.stdout:8/446: rmdir d5/d21/d2a/d71 39 2024-04-05T21:26:37.105 INFO:tasks.workunit.client.1.smithi188.stdout:8/447: dwrite d5/d21/d40/d43/f81 [0,4194304] 0 2024-04-05T21:26:37.108 INFO:tasks.workunit.client.1.smithi188.stdout:8/448: mkdir d5/d21/d40/d43/d44/d5b/d99/da3 0 2024-04-05T21:26:37.109 INFO:tasks.workunit.client.0.smithi121.stdout:6/175: read d1/f10 [2513375,4126] 0 2024-04-05T21:26:37.109 INFO:tasks.workunit.client.0.smithi121.stdout:4/260: symlink d5/d11/d28/d3b/l5b 0 2024-04-05T21:26:37.110 INFO:tasks.workunit.client.1.smithi188.stdout:8/449: dread - d5/d21/d40/d43/d44/d5b/d99/f7d zero size 2024-04-05T21:26:37.110 INFO:tasks.workunit.client.0.smithi121.stdout:6/176: dread - d1/d2/d1b/d1e/f2d zero size 2024-04-05T21:26:37.111 INFO:tasks.workunit.client.1.smithi188.stdout:7/651: sync 2024-04-05T21:26:37.111 INFO:tasks.workunit.client.1.smithi188.stdout:6/539: sync 2024-04-05T21:26:37.111 INFO:tasks.workunit.client.1.smithi188.stdout:2/537: sync 2024-04-05T21:26:37.111 INFO:tasks.workunit.client.1.smithi188.stdout:0/333: sync 2024-04-05T21:26:37.115 INFO:tasks.workunit.client.1.smithi188.stdout:7/652: symlink d3/d4/db/d2e/d37/d65/lca 0 2024-04-05T21:26:37.116 INFO:tasks.workunit.client.1.smithi188.stdout:6/540: symlink d6/d13/d18/d19/la6 0 2024-04-05T21:26:37.116 INFO:tasks.workunit.client.0.smithi121.stdout:6/177: dread d1/d2/d1b/d1e/f2a [0,4194304] 0 2024-04-05T21:26:37.116 INFO:tasks.workunit.client.0.smithi121.stdout:1/202: dread f4 [0,4194304] 0 2024-04-05T21:26:37.117 INFO:tasks.workunit.client.1.smithi188.stdout:8/450: dread d5/f35 [0,4194304] 0 2024-04-05T21:26:37.117 INFO:tasks.workunit.client.1.smithi188.stdout:2/538: creat d5/db/dd/d36/d3d/faf x:0 0 0 2024-04-05T21:26:37.117 INFO:tasks.workunit.client.0.smithi121.stdout:6/178: unlink d1/d2/d3/d5/f2f 0 2024-04-05T21:26:37.118 INFO:tasks.workunit.client.1.smithi188.stdout:0/334: truncate d4/d8/db/d21/f44 419577 0 2024-04-05T21:26:37.119 INFO:tasks.workunit.client.0.smithi121.stdout:1/203: creat d8/d10/d14/d27/d2d/f41 x:0 0 0 2024-04-05T21:26:37.119 INFO:tasks.workunit.client.1.smithi188.stdout:7/653: creat d3/d4/d1d/d74/d73/fcb x:0 0 0 2024-04-05T21:26:37.120 INFO:tasks.workunit.client.1.smithi188.stdout:7/654: chown d3/d4/l22 8219309 1 2024-04-05T21:26:37.121 INFO:tasks.workunit.client.1.smithi188.stdout:2/539: fsync d5/f5c 0 2024-04-05T21:26:37.122 INFO:tasks.workunit.client.1.smithi188.stdout:7/655: write d3/d4/db/d2e/d8c/db7/fbc [942904,64732] 0 2024-04-05T21:26:37.127 INFO:tasks.workunit.client.1.smithi188.stdout:6/541: creat d6/d13/d48/d49/fa7 x:0 0 0 2024-04-05T21:26:37.128 INFO:tasks.workunit.client.1.smithi188.stdout:7/656: stat d3/d4/db/fe 0 2024-04-05T21:26:37.130 INFO:tasks.workunit.client.1.smithi188.stdout:0/335: rmdir d4/d8/db/d66 0 2024-04-05T21:26:37.133 INFO:tasks.workunit.client.1.smithi188.stdout:7/657: dread - d3/d4/db/d2e/f93 zero size 2024-04-05T21:26:37.133 INFO:tasks.workunit.client.1.smithi188.stdout:2/540: dwrite d5/db/d35/d3b/d6b/fa7 [0,4194304] 0 2024-04-05T21:26:37.134 INFO:tasks.workunit.client.1.smithi188.stdout:6/542: read d6/f3b [1593600,38510] 0 2024-04-05T21:26:37.135 INFO:tasks.workunit.client.1.smithi188.stdout:0/336: symlink d4/d6/d61/l7d 0 2024-04-05T21:26:37.140 INFO:tasks.workunit.client.1.smithi188.stdout:2/541: creat d5/db/dd/d36/d3d/d46/d9f/fb0 x:0 0 0 2024-04-05T21:26:37.142 INFO:tasks.workunit.client.1.smithi188.stdout:0/337: readlink d4/d8/db/d21/d3c/l5a 0 2024-04-05T21:26:37.143 INFO:tasks.workunit.client.1.smithi188.stdout:6/543: dread d6/d13/d48/d49/f69 [0,4194304] 0 2024-04-05T21:26:37.147 INFO:tasks.workunit.client.1.smithi188.stdout:7/658: write d3/d4/d1d/d74/d73/f7a [2511189,110408] 0 2024-04-05T21:26:37.148 INFO:tasks.workunit.client.1.smithi188.stdout:8/451: dwrite d5/f15 [0,4194304] 0 2024-04-05T21:26:37.151 INFO:tasks.workunit.client.1.smithi188.stdout:0/338: rename d4/d6/d13/f55 to d4/d8/db/d21/f7e 0 2024-04-05T21:26:37.152 INFO:tasks.workunit.client.1.smithi188.stdout:7/659: dread d3/d4/db/d10/f33 [0,4194304] 0 2024-04-05T21:26:37.158 INFO:tasks.workunit.client.1.smithi188.stdout:7/660: creat d3/d4/db/d10/d5f/fcc x:0 0 0 2024-04-05T21:26:37.159 INFO:tasks.workunit.client.1.smithi188.stdout:8/452: dwrite d5/f1f [0,4194304] 0 2024-04-05T21:26:37.161 INFO:tasks.workunit.client.0.smithi121.stdout:6/179: read d1/d2/f13 [1207124,76802] 0 2024-04-05T21:26:37.161 INFO:tasks.workunit.client.1.smithi188.stdout:6/544: rename d6/d13/d18/c81 to d6/d13/d18/d19/ca8 0 2024-04-05T21:26:37.165 INFO:tasks.workunit.client.0.smithi121.stdout:6/180: mkdir d1/d2/d3/d5/d36 0 2024-04-05T21:26:37.166 INFO:tasks.workunit.client.1.smithi188.stdout:8/453: chown d5/d21/d2a/f31 238438915 1 2024-04-05T21:26:37.166 INFO:tasks.workunit.client.0.smithi121.stdout:4/261: dwrite d5/d11/f21 [0,4194304] 0 2024-04-05T21:26:37.166 INFO:tasks.workunit.client.1.smithi188.stdout:8/454: read - d5/d21/f91 zero size 2024-04-05T21:26:37.169 INFO:tasks.workunit.client.0.smithi121.stdout:6/181: chown d1/d2/d1b/d1e 801 1 2024-04-05T21:26:37.169 INFO:tasks.workunit.client.0.smithi121.stdout:4/262: dread - d5/d11/d2c/f3c zero size 2024-04-05T21:26:37.170 INFO:tasks.workunit.client.0.smithi121.stdout:1/204: dwrite d8/fb [0,4194304] 0 2024-04-05T21:26:37.171 INFO:tasks.workunit.client.1.smithi188.stdout:8/455: mkdir d5/d21/d40/d43/d44/d59/d8b/da4 0 2024-04-05T21:26:37.173 INFO:tasks.workunit.client.1.smithi188.stdout:0/339: dwrite d4/d8/db/f45 [0,4194304] 0 2024-04-05T21:26:37.173 INFO:tasks.workunit.client.0.smithi121.stdout:6/182: mknod d1/d2/d1b/d2e/c37 0 2024-04-05T21:26:37.176 INFO:tasks.workunit.client.0.smithi121.stdout:1/205: rename d8/d10/d14/f31 to d8/d10/d14/d1a/d1b/d38/f42 0 2024-04-05T21:26:37.176 INFO:tasks.workunit.client.0.smithi121.stdout:4/263: symlink d5/d45/l5c 0 2024-04-05T21:26:37.178 INFO:tasks.workunit.client.0.smithi121.stdout:4/264: chown d5/d11/d26/d27/d29/d2d/f31 377 1 2024-04-05T21:26:37.179 INFO:tasks.workunit.client.0.smithi121.stdout:1/206: symlink d8/d10/d14/d1a/d1b/d38/l43 0 2024-04-05T21:26:37.179 INFO:tasks.workunit.client.1.smithi188.stdout:8/456: dread d5/d21/d40/f4f [0,4194304] 0 2024-04-05T21:26:37.180 INFO:tasks.workunit.client.1.smithi188.stdout:0/340: truncate d4/d8/db/f76 941549 0 2024-04-05T21:26:37.181 INFO:tasks.workunit.client.0.smithi121.stdout:1/207: unlink f4 0 2024-04-05T21:26:37.182 INFO:tasks.workunit.client.0.smithi121.stdout:1/208: chown d8/ff 163993 1 2024-04-05T21:26:37.183 INFO:tasks.workunit.client.1.smithi188.stdout:6/545: dwrite d6/d13/d18/f50 [0,4194304] 0 2024-04-05T21:26:37.185 INFO:tasks.workunit.client.0.smithi121.stdout:6/183: dwrite d1/d2/f17 [4194304,4194304] 0 2024-04-05T21:26:37.187 INFO:tasks.workunit.client.0.smithi121.stdout:1/209: mkdir d8/d10/d14/d1a/d1b/d44 0 2024-04-05T21:26:37.189 INFO:tasks.workunit.client.1.smithi188.stdout:7/661: dwrite d3/d4/db/d10/f33 [0,4194304] 0 2024-04-05T21:26:37.190 INFO:tasks.workunit.client.0.smithi121.stdout:1/210: chown d8/d10/d14/d1a/d1b/d24/f2a 3750 1 2024-04-05T21:26:37.192 INFO:tasks.workunit.client.1.smithi188.stdout:2/542: dread d5/db/f76 [0,4194304] 0 2024-04-05T21:26:37.192 INFO:tasks.workunit.client.1.smithi188.stdout:6/546: chown d6/d13/d18/d19/ca8 121992 1 2024-04-05T21:26:37.192 INFO:tasks.workunit.client.0.smithi121.stdout:6/184: getdents d1 0 2024-04-05T21:26:37.193 INFO:tasks.workunit.client.0.smithi121.stdout:6/185: readlink d1/d2/d3/d4/l23 0 2024-04-05T21:26:37.196 INFO:tasks.workunit.client.0.smithi121.stdout:6/186: creat d1/f38 x:0 0 0 2024-04-05T21:26:37.197 INFO:tasks.workunit.client.1.smithi188.stdout:7/662: write d3/d2c/d5e/f78 [939763,47666] 0 2024-04-05T21:26:37.202 INFO:tasks.workunit.client.0.smithi121.stdout:1/211: dread d8/d10/d14/d27/f29 [0,4194304] 0 2024-04-05T21:26:37.204 INFO:tasks.workunit.client.0.smithi121.stdout:1/212: symlink d8/d10/d14/d1a/d1b/d24/l45 0 2024-04-05T21:26:37.205 INFO:tasks.workunit.client.0.smithi121.stdout:1/213: chown d8/d10/d14/d27/c40 49231358 1 2024-04-05T21:26:37.206 INFO:tasks.workunit.client.1.smithi188.stdout:0/341: unlink d4/d6/d13/d23/f52 0 2024-04-05T21:26:37.207 INFO:tasks.workunit.client.0.smithi121.stdout:1/214: creat d8/d10/d14/d1a/f46 x:0 0 0 2024-04-05T21:26:37.209 INFO:tasks.workunit.client.1.smithi188.stdout:2/543: symlink d5/db/dd/d36/d3c/d48/d85/lb1 0 2024-04-05T21:26:37.210 INFO:tasks.workunit.client.1.smithi188.stdout:8/457: creat d5/d21/d40/d43/d44/d5b/d99/da3/fa5 x:0 0 0 2024-04-05T21:26:37.213 INFO:tasks.workunit.client.1.smithi188.stdout:2/544: write d5/db/f76 [1095022,73927] 0 2024-04-05T21:26:37.213 INFO:tasks.workunit.client.1.smithi188.stdout:6/547: link d6/c37 d6/d13/d38/ca9 0 2024-04-05T21:26:37.214 INFO:tasks.workunit.client.1.smithi188.stdout:8/458: write d5/d21/d40/f73 [119562,34110] 0 2024-04-05T21:26:37.214 INFO:tasks.workunit.client.1.smithi188.stdout:0/342: symlink d4/d8/db/d42/d71/l7f 0 2024-04-05T21:26:37.214 INFO:tasks.workunit.client.0.smithi121.stdout:1/215: read d8/f1c [72142,107257] 0 2024-04-05T21:26:37.221 INFO:tasks.workunit.client.1.smithi188.stdout:6/548: creat d6/d13/d18/d19/d2c/d3f/d95/faa x:0 0 0 2024-04-05T21:26:37.221 INFO:tasks.workunit.client.0.smithi121.stdout:1/216: dwrite d8/d10/d14/d1a/d1b/d24/f39 [0,4194304] 0 2024-04-05T21:26:37.225 INFO:tasks.workunit.client.1.smithi188.stdout:8/459: dread d5/d21/d40/d43/f81 [0,4194304] 0 2024-04-05T21:26:37.227 INFO:tasks.workunit.client.1.smithi188.stdout:0/343: dwrite d4/d6/d13/d23/d24/f62 [0,4194304] 0 2024-04-05T21:26:37.230 INFO:tasks.workunit.client.1.smithi188.stdout:0/344: write d4/d8/d39/f48 [1139756,70737] 0 2024-04-05T21:26:37.233 INFO:tasks.workunit.client.0.smithi121.stdout:1/217: dread d8/d10/d14/f2c [0,4194304] 0 2024-04-05T21:26:37.237 INFO:tasks.workunit.client.1.smithi188.stdout:0/345: unlink d4/d8/db/d42/f3f 0 2024-04-05T21:26:37.239 INFO:tasks.workunit.client.0.smithi121.stdout:1/218: dwrite d8/d10/f26 [0,4194304] 0 2024-04-05T21:26:37.240 INFO:tasks.workunit.client.1.smithi188.stdout:0/346: readlink d4/d6/l4f 0 2024-04-05T21:26:37.240 INFO:tasks.workunit.client.1.smithi188.stdout:8/460: dwrite d5/d21/d2a/d47/f49 [0,4194304] 0 2024-04-05T21:26:37.241 INFO:tasks.workunit.client.0.smithi121.stdout:4/265: fdatasync d5/d11/d26/d27/d29/d2d/f31 0 2024-04-05T21:26:37.241 INFO:tasks.workunit.client.1.smithi188.stdout:7/663: dread d3/d2c/f96 [0,4194304] 0 2024-04-05T21:26:37.242 INFO:tasks.workunit.client.1.smithi188.stdout:0/347: stat d4/d6/d13/d23/c7b 0 2024-04-05T21:26:37.242 INFO:tasks.workunit.client.0.smithi121.stdout:1/219: symlink d8/d10/d14/d27/l47 0 2024-04-05T21:26:37.244 INFO:tasks.workunit.client.0.smithi121.stdout:4/266: getdents d5/d11/d20 0 2024-04-05T21:26:37.245 INFO:tasks.workunit.client.0.smithi121.stdout:1/220: truncate d8/d10/d14/d27/d2d/d3b/f3d 764383 0 2024-04-05T21:26:37.251 INFO:tasks.workunit.client.1.smithi188.stdout:0/348: unlink d4/d6/d46/f47 0 2024-04-05T21:26:37.251 INFO:tasks.workunit.client.1.smithi188.stdout:7/664: rename d3/d4/d1d/d74/d73/f7a to d3/d4/db/fcd 0 2024-04-05T21:26:37.253 INFO:tasks.workunit.client.0.smithi121.stdout:4/267: dwrite d5/d11/d26/d27/f30 [0,4194304] 0 2024-04-05T21:26:37.258 INFO:tasks.workunit.client.0.smithi121.stdout:4/268: getdents d5/d11/d26/d27/d29 0 2024-04-05T21:26:37.259 INFO:tasks.workunit.client.0.smithi121.stdout:2/210: sync 2024-04-05T21:26:37.259 INFO:tasks.workunit.client.0.smithi121.stdout:0/331: sync 2024-04-05T21:26:37.260 INFO:tasks.workunit.client.1.smithi188.stdout:7/665: rename d3/d4/db/d2e/d37/d65 to d3/d2c/dce 0 2024-04-05T21:26:37.261 INFO:tasks.workunit.client.0.smithi121.stdout:4/269: unlink d5/d11/d28/f2f 0 2024-04-05T21:26:37.261 INFO:tasks.workunit.client.1.smithi188.stdout:7/666: write d3/d2c/d5e/d8e/fac [284039,32867] 0 2024-04-05T21:26:37.263 INFO:tasks.workunit.client.0.smithi121.stdout:6/187: dwrite d1/f10 [0,4194304] 0 2024-04-05T21:26:37.264 INFO:tasks.workunit.client.1.smithi188.stdout:7/667: truncate d3/d2c/d5e/f78 1450319 0 2024-04-05T21:26:37.264 INFO:tasks.workunit.client.0.smithi121.stdout:2/211: dread d2/d9/fd [0,4194304] 0 2024-04-05T21:26:37.267 INFO:tasks.workunit.client.1.smithi188.stdout:7/668: rename d3/c46 to d3/d4/db/d55/ccf 0 2024-04-05T21:26:37.271 INFO:tasks.workunit.client.0.smithi121.stdout:2/212: rmdir d2/d9/d38 39 2024-04-05T21:26:37.272 INFO:tasks.workunit.client.1.smithi188.stdout:7/669: creat d3/d2c/fd0 x:0 0 0 2024-04-05T21:26:37.273 INFO:tasks.workunit.client.0.smithi121.stdout:0/332: getdents d7/d1c/d1d/d29/d3f 0 2024-04-05T21:26:37.274 INFO:tasks.workunit.client.1.smithi188.stdout:7/670: write d3/d4/fb9 [1664242,95667] 0 2024-04-05T21:26:37.274 INFO:tasks.workunit.client.0.smithi121.stdout:6/188: dread d1/d2/f17 [4194304,4194304] 0 2024-04-05T21:26:37.274 INFO:tasks.workunit.client.0.smithi121.stdout:4/270: dread d5/d11/d26/d27/d29/d2d/f40 [0,4194304] 0 2024-04-05T21:26:37.275 INFO:tasks.workunit.client.0.smithi121.stdout:2/213: mknod d2/d9/d17/c3f 0 2024-04-05T21:26:37.275 INFO:tasks.workunit.client.0.smithi121.stdout:0/333: write d7/d62/f11 [1521092,22470] 0 2024-04-05T21:26:37.276 INFO:tasks.workunit.client.1.smithi188.stdout:6/549: write d6/d7/f15 [2079779,125663] 0 2024-04-05T21:26:37.277 INFO:tasks.workunit.client.0.smithi121.stdout:2/214: chown d2/d9/d17/f18 468506149 1 2024-04-05T21:26:37.278 INFO:tasks.workunit.client.0.smithi121.stdout:4/271: dread d5/d11/d2c/f36 [0,4194304] 0 2024-04-05T21:26:37.280 INFO:tasks.workunit.client.1.smithi188.stdout:6/550: fsync d6/d13/d18/d94/fa0 0 2024-04-05T21:26:37.282 INFO:tasks.workunit.client.1.smithi188.stdout:7/671: creat d3/d4/d1d/d74/fd1 x:0 0 0 2024-04-05T21:26:37.284 INFO:tasks.workunit.client.1.smithi188.stdout:2/545: dwrite d5/db/dd/d36/d3c/d63/d75/d87/f9c [0,4194304] 0 2024-04-05T21:26:37.284 INFO:tasks.workunit.client.0.smithi121.stdout:0/334: creat d7/d62/f6d x:0 0 0 2024-04-05T21:26:37.284 INFO:tasks.workunit.client.0.smithi121.stdout:1/221: dread d8/fb [4194304,4194304] 0 2024-04-05T21:26:37.286 INFO:tasks.workunit.client.0.smithi121.stdout:4/272: link d5/d11/f35 d5/d11/d28/f5d 0 2024-04-05T21:26:37.287 INFO:tasks.workunit.client.0.smithi121.stdout:4/273: readlink d5/d11/d26/d27/d29/d2d/l33 0 2024-04-05T21:26:37.287 INFO:tasks.workunit.client.1.smithi188.stdout:6/551: creat d6/d13/d18/d19/d61/fab x:0 0 0 2024-04-05T21:26:37.288 INFO:tasks.workunit.client.0.smithi121.stdout:0/335: symlink d7/d1c/d1d/d45/d46/l6e 0 2024-04-05T21:26:37.291 INFO:tasks.workunit.client.1.smithi188.stdout:0/349: write d4/d8/db/d42/d38/f49 [350466,25926] 0 2024-04-05T21:26:37.295 INFO:tasks.workunit.client.1.smithi188.stdout:2/546: dwrite d5/db/dd/d36/d3d/faf [0,4194304] 0 2024-04-05T21:26:37.297 INFO:tasks.workunit.client.1.smithi188.stdout:8/461: dread d5/d21/d2a/f39 [0,4194304] 0 2024-04-05T21:26:37.297 INFO:tasks.workunit.client.0.smithi121.stdout:6/189: truncate d1/f1c 5818557 0 2024-04-05T21:26:37.297 INFO:tasks.workunit.client.0.smithi121.stdout:1/222: creat d8/d10/d14/d1a/d1b/d44/f48 x:0 0 0 2024-04-05T21:26:37.297 INFO:tasks.workunit.client.0.smithi121.stdout:0/336: mknod d7/d1c/c6f 0 2024-04-05T21:26:37.298 INFO:tasks.workunit.client.0.smithi121.stdout:2/215: write d2/d9/ff [3900929,87103] 0 2024-04-05T21:26:37.299 INFO:tasks.workunit.client.1.smithi188.stdout:7/672: mknod d3/d4/db/d2e/cd2 0 2024-04-05T21:26:37.299 INFO:tasks.workunit.client.1.smithi188.stdout:2/547: fsync d5/db/dd/d36/d3d/d46/f7b 0 2024-04-05T21:26:37.300 INFO:tasks.workunit.client.1.smithi188.stdout:6/552: creat d6/d13/d18/d19/d2c/d34/fac x:0 0 0 2024-04-05T21:26:37.300 INFO:tasks.workunit.client.1.smithi188.stdout:0/350: mkdir d4/d8/db/d42/d71/d80 0 2024-04-05T21:26:37.301 INFO:tasks.workunit.client.0.smithi121.stdout:6/190: mkdir d1/d2/d1b/d35/d39 0 2024-04-05T21:26:37.302 INFO:tasks.workunit.client.0.smithi121.stdout:2/216: write d2/d9/d1c/f22 [704305,113005] 0 2024-04-05T21:26:37.307 INFO:tasks.workunit.client.0.smithi121.stdout:4/274: link d5/d11/d26/d27/c5a d5/c5e 0 2024-04-05T21:26:37.307 INFO:tasks.workunit.client.0.smithi121.stdout:6/191: write d1/f38 [662233,75925] 0 2024-04-05T21:26:37.307 INFO:tasks.workunit.client.0.smithi121.stdout:4/275: read d5/d11/f48 [200084,104837] 0 2024-04-05T21:26:37.307 INFO:tasks.workunit.client.0.smithi121.stdout:1/223: rename d8/d10/d14/f36 to d8/d10/d14/d1a/d1b/d24/f49 0 2024-04-05T21:26:37.307 INFO:tasks.workunit.client.0.smithi121.stdout:0/337: dread d7/d1c/d1d/d29/f58 [0,4194304] 0 2024-04-05T21:26:37.307 INFO:tasks.workunit.client.0.smithi121.stdout:2/217: mknod d2/d28/c40 0 2024-04-05T21:26:37.308 INFO:tasks.workunit.client.0.smithi121.stdout:2/218: fdatasync d2/d9/fe 0 2024-04-05T21:26:37.309 INFO:tasks.workunit.client.0.smithi121.stdout:6/192: dread - d1/d2/d1b/f28 zero size 2024-04-05T21:26:37.309 INFO:tasks.workunit.client.1.smithi188.stdout:2/548: getdents d5/db/dd 0 2024-04-05T21:26:37.310 INFO:tasks.workunit.client.1.smithi188.stdout:0/351: dwrite d4/d6/d13/d23/d24/f62 [4194304,4194304] 0 2024-04-05T21:26:37.310 INFO:tasks.workunit.client.0.smithi121.stdout:2/219: write d2/d9/fe [4620347,130222] 0 2024-04-05T21:26:37.310 INFO:tasks.workunit.client.0.smithi121.stdout:4/276: mknod d5/d11/d26/d27/d29/d58/c5f 0 2024-04-05T21:26:37.312 INFO:tasks.workunit.client.1.smithi188.stdout:7/673: write d3/d4/db/f60 [2039472,30491] 0 2024-04-05T21:26:37.312 INFO:tasks.workunit.client.1.smithi188.stdout:8/462: write d5/d21/d40/d43/d44/d59/f77 [1627017,40528] 0 2024-04-05T21:26:37.314 INFO:tasks.workunit.client.1.smithi188.stdout:2/549: symlink d5/db/dd/d73/lb2 0 2024-04-05T21:26:37.316 INFO:tasks.workunit.client.1.smithi188.stdout:2/550: write d5/db/dd/d36/d3d/d46/faa [941091,98230] 0 2024-04-05T21:26:37.316 INFO:tasks.workunit.client.1.smithi188.stdout:8/463: unlink d5/d21/d40/d43/d44/d5b/l60 0 2024-04-05T21:26:37.317 INFO:tasks.workunit.client.0.smithi121.stdout:4/277: dread - d5/d11/d20/f57 zero size 2024-04-05T21:26:37.317 INFO:tasks.workunit.client.0.smithi121.stdout:1/224: getdents d8/d10 0 2024-04-05T21:26:37.318 INFO:tasks.workunit.client.1.smithi188.stdout:0/352: creat d4/d6/d13/d23/d24/f81 x:0 0 0 2024-04-05T21:26:37.318 INFO:tasks.workunit.client.1.smithi188.stdout:3/534: sync 2024-04-05T21:26:37.318 INFO:tasks.workunit.client.1.smithi188.stdout:2/551: write d5/db/dd/f11 [3914545,84815] 0 2024-04-05T21:26:37.319 INFO:tasks.workunit.client.0.smithi121.stdout:6/193: symlink d1/d2/l3a 0 2024-04-05T21:26:37.319 INFO:tasks.workunit.client.0.smithi121.stdout:4/278: fdatasync d5/d11/d26/d27/d29/d2d/f31 0 2024-04-05T21:26:37.321 INFO:tasks.workunit.client.0.smithi121.stdout:0/338: symlink d7/d1c/d1d/d29/l70 0 2024-04-05T21:26:37.321 INFO:tasks.workunit.client.0.smithi121.stdout:1/225: truncate d8/d10/d14/d1a/d1b/d24/f2a 907155 0 2024-04-05T21:26:37.322 INFO:tasks.workunit.client.1.smithi188.stdout:3/535: creat d5/d7/d65/d5f/fb4 x:0 0 0 2024-04-05T21:26:37.323 INFO:tasks.workunit.client.0.smithi121.stdout:2/220: creat d2/d9/d38/d2d/f41 x:0 0 0 2024-04-05T21:26:37.324 INFO:tasks.workunit.client.0.smithi121.stdout:2/221: dread - d2/d9/d38/f35 zero size 2024-04-05T21:26:37.324 INFO:tasks.workunit.client.0.smithi121.stdout:2/222: readlink d2/l4 0 2024-04-05T21:26:37.325 INFO:tasks.workunit.client.0.smithi121.stdout:2/223: write d2/d9/d38/f35 [97276,122809] 0 2024-04-05T21:26:37.326 INFO:tasks.workunit.client.0.smithi121.stdout:6/194: rename d1/d2/d3/d4/de/l33 to d1/l3b 0 2024-04-05T21:26:37.326 INFO:tasks.workunit.client.0.smithi121.stdout:2/224: chown d2/l4 7 1 2024-04-05T21:26:37.326 INFO:tasks.workunit.client.1.smithi188.stdout:8/464: getdents d5/d21/d40/d43/d44 0 2024-04-05T21:26:37.327 INFO:tasks.workunit.client.0.smithi121.stdout:4/279: mkdir d5/d11/d60 0 2024-04-05T21:26:37.328 INFO:tasks.workunit.client.0.smithi121.stdout:0/339: getdents d7/d1c/d1d/d29/d3f 0 2024-04-05T21:26:37.329 INFO:tasks.workunit.client.0.smithi121.stdout:4/280: mknod d5/d11/c61 0 2024-04-05T21:26:37.331 INFO:tasks.workunit.client.1.smithi188.stdout:2/552: dwrite d5/db/f20 [0,4194304] 0 2024-04-05T21:26:37.334 INFO:tasks.workunit.client.1.smithi188.stdout:3/536: dread d5/df/f15 [0,4194304] 0 2024-04-05T21:26:37.334 INFO:tasks.workunit.client.0.smithi121.stdout:6/195: link d1/d2/d3/d5/d1f/f27 d1/d2/d3/d5/f3c 0 2024-04-05T21:26:37.337 INFO:tasks.workunit.client.0.smithi121.stdout:6/196: creat d1/d2/d3/d5/d36/f3d x:0 0 0 2024-04-05T21:26:37.338 INFO:tasks.workunit.client.1.smithi188.stdout:0/353: dwrite d4/d8/d10/f1e [0,4194304] 0 2024-04-05T21:26:37.339 INFO:tasks.workunit.client.0.smithi121.stdout:6/197: fdatasync d1/d2/d3/d5/f3c 0 2024-04-05T21:26:37.342 INFO:tasks.workunit.client.0.smithi121.stdout:1/226: dread d8/d10/d14/d1a/d1b/d24/f49 [0,4194304] 0 2024-04-05T21:26:37.342 INFO:tasks.workunit.client.1.smithi188.stdout:0/354: unlink d4/d8/d39/l43 0 2024-04-05T21:26:37.343 INFO:tasks.workunit.client.1.smithi188.stdout:2/553: creat d5/db/dd/d36/d3d/d46/d66/fb3 x:0 0 0 2024-04-05T21:26:37.347 INFO:tasks.workunit.client.0.smithi121.stdout:1/227: symlink d8/d10/d14/d1a/d1b/d24/l4a 0 2024-04-05T21:26:37.350 INFO:tasks.workunit.client.0.smithi121.stdout:6/198: dwrite d1/d2/d1b/d1e/f2d [0,4194304] 0 2024-04-05T21:26:37.351 INFO:tasks.workunit.client.1.smithi188.stdout:8/465: dwrite d5/d21/d2a/f8d [0,4194304] 0 2024-04-05T21:26:37.351 INFO:tasks.workunit.client.0.smithi121.stdout:1/228: mkdir d8/d10/d14/d1a/d1b/d38/d4b 0 2024-04-05T21:26:37.354 INFO:tasks.workunit.client.1.smithi188.stdout:8/466: dread - d5/d21/d40/d43/d48/d7e/f93 zero size 2024-04-05T21:26:37.355 INFO:tasks.workunit.client.0.smithi121.stdout:6/199: unlink d1/d2/dc/c31 0 2024-04-05T21:26:37.355 INFO:tasks.workunit.client.0.smithi121.stdout:1/229: truncate d8/d10/d14/d27/d2d/d3b/f3d 1230850 0 2024-04-05T21:26:37.356 INFO:tasks.workunit.client.1.smithi188.stdout:8/467: write d5/f2f [3654935,78036] 0 2024-04-05T21:26:37.356 INFO:tasks.workunit.client.1.smithi188.stdout:0/355: dwrite d4/d6/d13/d23/d24/f81 [0,4194304] 0 2024-04-05T21:26:37.356 INFO:tasks.workunit.client.1.smithi188.stdout:2/554: link d5/db/dd/d36/d3d/d46/cae d5/db/dd/da4/cb4 0 2024-04-05T21:26:37.357 INFO:tasks.workunit.client.1.smithi188.stdout:8/468: write d5/d21/d50/f5f [938715,18044] 0 2024-04-05T21:26:37.361 INFO:tasks.workunit.client.1.smithi188.stdout:0/356: creat d4/d8/db/d42/d38/f82 x:0 0 0 2024-04-05T21:26:37.361 INFO:tasks.workunit.client.1.smithi188.stdout:3/537: dread d5/d7/d65/f88 [0,4194304] 0 2024-04-05T21:26:37.362 INFO:tasks.workunit.client.1.smithi188.stdout:0/357: write d4/f5c [409246,31827] 0 2024-04-05T21:26:37.363 INFO:tasks.workunit.client.0.smithi121.stdout:6/200: creat d1/d2/dc/f3e x:0 0 0 2024-04-05T21:26:37.366 INFO:tasks.workunit.client.1.smithi188.stdout:3/538: mkdir d5/d7/d65/d61/db5 0 2024-04-05T21:26:37.367 INFO:tasks.workunit.client.0.smithi121.stdout:0/340: rename d7/d1c/d1d to d7/d71 0 2024-04-05T21:26:37.369 INFO:tasks.workunit.client.0.smithi121.stdout:1/230: mkdir d8/d10/d14/d1a/d1b/d4c 0 2024-04-05T21:26:37.369 INFO:tasks.workunit.client.1.smithi188.stdout:3/539: chown d5/d2b/f2e 1294 1 2024-04-05T21:26:37.371 INFO:tasks.workunit.client.0.smithi121.stdout:6/201: rename d1/c12 to d1/d2/d1b/d1e/c3f 0 2024-04-05T21:26:37.373 INFO:tasks.workunit.client.1.smithi188.stdout:0/358: dwrite d4/d8/db/f76 [0,4194304] 0 2024-04-05T21:26:37.373 INFO:tasks.workunit.client.1.smithi188.stdout:3/540: mkdir d5/d35/d4e/db6 0 2024-04-05T21:26:37.376 INFO:tasks.workunit.client.1.smithi188.stdout:0/359: chown d4/d6/d13/d23/d24/d57 42898 1 2024-04-05T21:26:37.377 INFO:tasks.workunit.client.0.smithi121.stdout:0/341: dwrite d7/d71/d45/f60 [0,4194304] 0 2024-04-05T21:26:37.377 INFO:tasks.workunit.client.1.smithi188.stdout:3/541: fsync d5/d7/d65/d61/fae 0 2024-04-05T21:26:37.377 INFO:tasks.workunit.client.0.smithi121.stdout:2/225: dread d2/d9/d17/f19 [0,4194304] 0 2024-04-05T21:26:37.378 INFO:tasks.workunit.client.0.smithi121.stdout:1/231: rename d8/d10/d14/d1a/d1b/f35 to d8/d10/d14/d1a/d1b/d4c/f4d 0 2024-04-05T21:26:37.379 INFO:tasks.workunit.client.1.smithi188.stdout:0/360: creat d4/d8/d39/f83 x:0 0 0 2024-04-05T21:26:37.379 INFO:tasks.workunit.client.0.smithi121.stdout:0/342: chown d7/d1c/f55 776009 1 2024-04-05T21:26:37.381 INFO:tasks.workunit.client.0.smithi121.stdout:4/281: dread d5/f1d [0,4194304] 0 2024-04-05T21:26:37.381 INFO:tasks.workunit.client.0.smithi121.stdout:1/232: mknod d8/d10/d14/d27/c4e 0 2024-04-05T21:26:37.381 INFO:tasks.workunit.client.1.smithi188.stdout:3/542: mknod d5/dc/d2f/cb7 0 2024-04-05T21:26:37.383 INFO:tasks.workunit.client.1.smithi188.stdout:3/543: fdatasync d5/d35/f40 0 2024-04-05T21:26:37.385 INFO:tasks.workunit.client.0.smithi121.stdout:4/282: fsync d5/d11/d2c/f36 0 2024-04-05T21:26:37.386 INFO:tasks.workunit.client.1.smithi188.stdout:3/544: chown d5/d2b/c3b 18 1 2024-04-05T21:26:37.387 INFO:tasks.workunit.client.0.smithi121.stdout:1/233: read d8/d10/d14/d1a/d1b/d24/f49 [3541725,46014] 0 2024-04-05T21:26:37.387 INFO:tasks.workunit.client.1.smithi188.stdout:3/545: read - d5/f9b zero size 2024-04-05T21:26:37.388 INFO:tasks.workunit.client.0.smithi121.stdout:2/226: dwrite d2/d9/fe [4194304,4194304] 0 2024-04-05T21:26:37.389 INFO:tasks.workunit.client.1.smithi188.stdout:0/361: dwrite d4/d8/fa [0,4194304] 0 2024-04-05T21:26:37.390 INFO:tasks.workunit.client.1.smithi188.stdout:3/546: read d5/dc/f2d [141458,70064] 0 2024-04-05T21:26:37.390 INFO:tasks.workunit.client.1.smithi188.stdout:8/469: dread d5/f34 [0,4194304] 0 2024-04-05T21:26:37.390 INFO:tasks.workunit.client.0.smithi121.stdout:1/234: dread - d8/d10/d14/d1a/f46 zero size 2024-04-05T21:26:37.392 INFO:tasks.workunit.client.1.smithi188.stdout:3/547: dread - d5/d7/d65/d5f/fb4 zero size 2024-04-05T21:26:37.396 INFO:tasks.workunit.client.1.smithi188.stdout:8/470: creat d5/d21/d40/d43/d44/d5b/d99/da3/fa6 x:0 0 0 2024-04-05T21:26:37.396 INFO:tasks.workunit.client.1.smithi188.stdout:3/548: creat d5/d7/d6d/d75/fb8 x:0 0 0 2024-04-05T21:26:37.397 INFO:tasks.workunit.client.0.smithi121.stdout:1/235: write d8/d10/d14/d1a/f46 [1029200,18863] 0 2024-04-05T21:26:37.398 INFO:tasks.workunit.client.0.smithi121.stdout:0/343: dread d7/d62/f42 [0,4194304] 0 2024-04-05T21:26:37.400 INFO:tasks.workunit.client.0.smithi121.stdout:0/344: fdatasync d7/d71/d45/d46/f57 0 2024-04-05T21:26:37.401 INFO:tasks.workunit.client.1.smithi188.stdout:0/362: dwrite d4/d8/d10/f20 [0,4194304] 0 2024-04-05T21:26:37.401 INFO:tasks.workunit.client.1.smithi188.stdout:3/549: symlink d5/d7/d65/d61/d66/lb9 0 2024-04-05T21:26:37.402 INFO:tasks.workunit.client.0.smithi121.stdout:1/236: chown d8/d10/d14/f1e 1 1 2024-04-05T21:26:37.402 INFO:tasks.workunit.client.0.smithi121.stdout:4/283: rename d5/d11/d28/d3b/l5b to d5/l62 0 2024-04-05T21:26:37.403 INFO:tasks.workunit.client.1.smithi188.stdout:8/471: truncate d5/dc/f17 193639 0 2024-04-05T21:26:37.406 INFO:tasks.workunit.client.0.smithi121.stdout:9/204: sync 2024-04-05T21:26:37.407 INFO:tasks.workunit.client.1.smithi188.stdout:3/550: rename d5/df/d80/c90 to d5/d2b/cba 0 2024-04-05T21:26:37.408 INFO:tasks.workunit.client.0.smithi121.stdout:9/205: stat f2 0 2024-04-05T21:26:37.408 INFO:tasks.workunit.client.0.smithi121.stdout:1/237: creat d8/d10/d14/d1a/d1b/d4c/f4f x:0 0 0 2024-04-05T21:26:37.408 INFO:tasks.workunit.client.0.smithi121.stdout:4/284: symlink d5/d11/d26/d49/l63 0 2024-04-05T21:26:37.409 INFO:tasks.workunit.client.0.smithi121.stdout:2/227: dwrite d2/d9/d17/f18 [0,4194304] 0 2024-04-05T21:26:37.410 INFO:tasks.workunit.client.1.smithi188.stdout:0/363: truncate d4/d8/db/f28 194515 0 2024-04-05T21:26:37.411 INFO:tasks.workunit.client.0.smithi121.stdout:4/285: dread - d5/d11/d20/f57 zero size 2024-04-05T21:26:37.411 INFO:tasks.workunit.client.0.smithi121.stdout:1/238: chown d8/d10/d14/d1a/d1b/d38/d4b 760158158 1 2024-04-05T21:26:37.411 INFO:tasks.workunit.client.0.smithi121.stdout:0/345: rename d7/d71/d45/f60 to d7/d71/d29/d3f/f72 0 2024-04-05T21:26:37.415 INFO:tasks.workunit.client.0.smithi121.stdout:2/228: chown d2/d9/d1c/d30 35406059 1 2024-04-05T21:26:37.416 INFO:tasks.workunit.client.0.smithi121.stdout:6/202: chown d1/f1c 7277769 1 2024-04-05T21:26:37.416 INFO:tasks.workunit.client.1.smithi188.stdout:8/472: write d5/d21/d40/d43/d44/d59/f5a [4914475,58373] 0 2024-04-05T21:26:37.416 INFO:tasks.workunit.client.0.smithi121.stdout:1/239: chown d8/d10/d14/d1a/f46 4523 1 2024-04-05T21:26:37.417 INFO:tasks.workunit.client.0.smithi121.stdout:9/206: write d6/df/d1d/f33 [8090897,23142] 0 2024-04-05T21:26:37.417 INFO:tasks.workunit.client.1.smithi188.stdout:8/473: write d5/d21/d40/f55 [96374,63175] 0 2024-04-05T21:26:37.418 INFO:tasks.workunit.client.0.smithi121.stdout:4/286: chown d5/fa 1322 1 2024-04-05T21:26:37.420 INFO:tasks.workunit.client.0.smithi121.stdout:1/240: mknod d8/d10/d14/c50 0 2024-04-05T21:26:37.421 INFO:tasks.workunit.client.0.smithi121.stdout:0/346: getdents d7/d1c 0 2024-04-05T21:26:37.421 INFO:tasks.workunit.client.0.smithi121.stdout:9/207: dread - d6/df/d1d/d2c/d37/d3c/f3d zero size 2024-04-05T21:26:37.422 INFO:tasks.workunit.client.1.smithi188.stdout:8/474: symlink d5/d21/d2a/la7 0 2024-04-05T21:26:37.423 INFO:tasks.workunit.client.0.smithi121.stdout:6/203: mknod d1/d2/d3/d5/d1f/c40 0 2024-04-05T21:26:37.424 INFO:tasks.workunit.client.1.smithi188.stdout:8/475: chown d5/d21/d2a/d71/f88 37602323 1 2024-04-05T21:26:37.425 INFO:tasks.workunit.client.0.smithi121.stdout:2/229: rename d2/d9/fb to d2/d9/d17/f42 0 2024-04-05T21:26:37.425 INFO:tasks.workunit.client.0.smithi121.stdout:6/204: dread d1/d2/d1b/d1e/f2a [0,4194304] 0 2024-04-05T21:26:37.431 INFO:tasks.workunit.client.1.smithi188.stdout:8/476: dwrite d5/d21/d2a/d47/f49 [0,4194304] 0 2024-04-05T21:26:37.432 INFO:tasks.workunit.client.0.smithi121.stdout:9/208: rmdir d6/df/d1d/d2c/d37/d3c 39 2024-04-05T21:26:37.432 INFO:tasks.workunit.client.0.smithi121.stdout:2/230: mknod d2/c43 0 2024-04-05T21:26:37.432 INFO:tasks.workunit.client.0.smithi121.stdout:1/241: mknod d8/d10/d14/d1a/d1b/c51 0 2024-04-05T21:26:37.433 INFO:tasks.workunit.client.1.smithi188.stdout:8/477: chown d5 481866119 1 2024-04-05T21:26:37.434 INFO:tasks.workunit.client.0.smithi121.stdout:0/347: dwrite d7/d71/d45/d46/f57 [0,4194304] 0 2024-04-05T21:26:37.434 INFO:tasks.workunit.client.0.smithi121.stdout:6/205: fdatasync d1/d2/f13 0 2024-04-05T21:26:37.435 INFO:tasks.workunit.client.1.smithi188.stdout:8/478: symlink d5/d21/d2a/d47/la8 0 2024-04-05T21:26:37.435 INFO:tasks.workunit.client.1.smithi188.stdout:8/479: chown d5/f34 53716947 1 2024-04-05T21:26:37.436 INFO:tasks.workunit.client.0.smithi121.stdout:2/231: fdatasync d2/d9/d1c/f22 0 2024-04-05T21:26:37.438 INFO:tasks.workunit.client.0.smithi121.stdout:6/206: chown d1/d2/d3/d5/f3c 54 1 2024-04-05T21:26:37.438 INFO:tasks.workunit.client.1.smithi188.stdout:3/551: dread d5/d7/f1a [0,4194304] 0 2024-04-05T21:26:37.440 INFO:tasks.workunit.client.1.smithi188.stdout:8/480: getdents d5/d21 0 2024-04-05T21:26:37.440 INFO:tasks.workunit.client.0.smithi121.stdout:4/287: dread d5/f8 [0,4194304] 0 2024-04-05T21:26:37.441 INFO:tasks.workunit.client.0.smithi121.stdout:2/232: chown d2/d9/d38/f35 3916 1 2024-04-05T21:26:37.442 INFO:tasks.workunit.client.1.smithi188.stdout:3/552: rmdir d5/d35/d4e 39 2024-04-05T21:26:37.443 INFO:tasks.workunit.client.0.smithi121.stdout:1/242: mknod d8/d10/d14/d1a/d1b/d24/c52 0 2024-04-05T21:26:37.444 INFO:tasks.workunit.client.0.smithi121.stdout:4/288: rmdir d5/d11/d28 39 2024-04-05T21:26:37.444 INFO:tasks.workunit.client.0.smithi121.stdout:1/243: chown d8/d10/d14/d27/c4e 30 1 2024-04-05T21:26:37.450 INFO:tasks.workunit.client.0.smithi121.stdout:0/348: creat d7/d62/f73 x:0 0 0 2024-04-05T21:26:37.450 INFO:tasks.workunit.client.0.smithi121.stdout:2/233: write d2/f1a [1240611,72563] 0 2024-04-05T21:26:37.450 INFO:tasks.workunit.client.0.smithi121.stdout:9/209: dwrite d6/f16 [0,4194304] 0 2024-04-05T21:26:37.450 INFO:tasks.workunit.client.0.smithi121.stdout:0/349: chown d7/d62/d3c/d6b 53 1 2024-04-05T21:26:37.451 INFO:tasks.workunit.client.0.smithi121.stdout:2/234: chown d2/d9/f21 785475 1 2024-04-05T21:26:37.451 INFO:tasks.workunit.client.0.smithi121.stdout:9/210: chown d6/df/f20 149750 1 2024-04-05T21:26:37.451 INFO:tasks.workunit.client.1.smithi188.stdout:8/481: dread d5/d21/d40/f73 [0,4194304] 0 2024-04-05T21:26:37.452 INFO:tasks.workunit.client.0.smithi121.stdout:1/244: write d8/d10/d14/d1a/d1b/d4c/f4d [968704,10910] 0 2024-04-05T21:26:37.454 INFO:tasks.workunit.client.0.smithi121.stdout:1/245: dread d8/d10/d14/f2c [0,4194304] 0 2024-04-05T21:26:37.454 INFO:tasks.workunit.client.1.smithi188.stdout:8/482: mknod d5/d21/d40/d43/d44/ca9 0 2024-04-05T21:26:37.454 INFO:tasks.workunit.client.0.smithi121.stdout:4/289: rename d5/d11/d26/d27/d29/d2d to d5/d11/d26/d27/d64 0 2024-04-05T21:26:37.455 INFO:tasks.workunit.client.0.smithi121.stdout:1/246: fsync d8/d10/d14/f1e 0 2024-04-05T21:26:37.458 INFO:tasks.workunit.client.0.smithi121.stdout:0/350: dread f2 [0,4194304] 0 2024-04-05T21:26:37.458 INFO:tasks.workunit.client.0.smithi121.stdout:1/247: fdatasync d8/d10/d14/d27/f30 0 2024-04-05T21:26:37.459 INFO:tasks.workunit.client.0.smithi121.stdout:4/290: dread d5/d11/f21 [0,4194304] 0 2024-04-05T21:26:37.460 INFO:tasks.workunit.client.0.smithi121.stdout:4/291: fsync d5/f16 0 2024-04-05T21:26:37.461 INFO:tasks.workunit.client.1.smithi188.stdout:0/364: dread d4/d8/db/d21/f34 [0,4194304] 0 2024-04-05T21:26:37.461 INFO:tasks.workunit.client.1.smithi188.stdout:8/483: dwrite d5/f2f [0,4194304] 0 2024-04-05T21:26:37.462 INFO:tasks.workunit.client.0.smithi121.stdout:0/351: rename d7/d1c/c6f to d7/d71/d45/d46/c74 0 2024-04-05T21:26:37.463 INFO:tasks.workunit.client.0.smithi121.stdout:1/248: mkdir d8/d10/d14/d27/d2d/d3b/d53 0 2024-04-05T21:26:37.466 INFO:tasks.workunit.client.1.smithi188.stdout:8/484: creat d5/d21/d40/d43/faa x:0 0 0 2024-04-05T21:26:37.466 INFO:tasks.workunit.client.0.smithi121.stdout:4/292: fdatasync d5/d11/d28/f44 0 2024-04-05T21:26:37.470 INFO:tasks.workunit.client.0.smithi121.stdout:0/352: unlink d7/d71/d29/d2a/d39/f41 0 2024-04-05T21:26:37.471 INFO:tasks.workunit.client.1.smithi188.stdout:3/553: truncate d5/f7d 1694577 0 2024-04-05T21:26:37.471 INFO:tasks.workunit.client.1.smithi188.stdout:0/365: dread d4/d6/d46/f59 [0,4194304] 0 2024-04-05T21:26:37.472 INFO:tasks.workunit.client.1.smithi188.stdout:0/366: fdatasync f3 0 2024-04-05T21:26:37.473 INFO:tasks.workunit.client.0.smithi121.stdout:4/293: creat d5/d11/d2c/f65 x:0 0 0 2024-04-05T21:26:37.474 INFO:tasks.workunit.client.0.smithi121.stdout:1/249: dwrite d8/fb [0,4194304] 0 2024-04-05T21:26:37.478 INFO:tasks.workunit.client.1.smithi188.stdout:8/485: dwrite d5/d21/d2a/f8d [4194304,4194304] 0 2024-04-05T21:26:37.479 INFO:tasks.workunit.client.0.smithi121.stdout:1/250: rename d8/fb to d8/d10/d14/d1a/d1b/d38/d4b/f54 0 2024-04-05T21:26:37.479 INFO:tasks.workunit.client.1.smithi188.stdout:8/486: readlink d5/d21/l2b 0 2024-04-05T21:26:37.480 INFO:tasks.workunit.client.0.smithi121.stdout:4/294: creat d5/d11/d26/d47/f66 x:0 0 0 2024-04-05T21:26:37.482 INFO:tasks.workunit.client.0.smithi121.stdout:6/207: dread d1/d2/d3/d4/de/f26 [0,4194304] 0 2024-04-05T21:26:37.484 INFO:tasks.workunit.client.1.smithi188.stdout:8/487: unlink d5/d21/fa2 0 2024-04-05T21:26:37.486 INFO:tasks.workunit.client.0.smithi121.stdout:1/251: symlink d8/d10/d14/d1a/d1b/l55 0 2024-04-05T21:26:37.487 INFO:tasks.workunit.client.0.smithi121.stdout:6/208: getdents d1/d2/d1b/d35/d39 0 2024-04-05T21:26:37.488 INFO:tasks.workunit.client.0.smithi121.stdout:0/353: link d7/l4e d7/d1c/l75 0 2024-04-05T21:26:37.489 INFO:tasks.workunit.client.0.smithi121.stdout:1/252: mknod d8/d10/d14/d27/d2d/c56 0 2024-04-05T21:26:37.490 INFO:tasks.workunit.client.0.smithi121.stdout:6/209: write d1/d2/d3/d5/d1f/f27 [2049401,76708] 0 2024-04-05T21:26:37.491 INFO:tasks.workunit.client.0.smithi121.stdout:0/354: mknod d7/d62/c76 0 2024-04-05T21:26:37.495 INFO:tasks.workunit.client.0.smithi121.stdout:1/253: mkdir d8/d10/d14/d27/d57 0 2024-04-05T21:26:37.496 INFO:tasks.workunit.client.0.smithi121.stdout:0/355: mknod d7/c77 0 2024-04-05T21:26:37.497 INFO:tasks.workunit.client.1.smithi188.stdout:0/367: dwrite d4/d8/d10/f67 [0,4194304] 0 2024-04-05T21:26:37.497 INFO:tasks.workunit.client.0.smithi121.stdout:4/295: write d5/d11/d28/f13 [1392530,17475] 0 2024-04-05T21:26:37.499 INFO:tasks.workunit.client.0.smithi121.stdout:4/296: fdatasync d5/d11/f1e 0 2024-04-05T21:26:37.500 INFO:tasks.workunit.client.1.smithi188.stdout:0/368: mknod d4/d6/d61/c84 0 2024-04-05T21:26:37.502 INFO:tasks.workunit.client.0.smithi121.stdout:1/254: truncate d8/d10/d14/f1e 3424929 0 2024-04-05T21:26:37.503 INFO:tasks.workunit.client.1.smithi188.stdout:0/369: write d4/d8/d10/f6d [598911,84501] 0 2024-04-05T21:26:37.503 INFO:tasks.workunit.client.0.smithi121.stdout:4/297: mkdir d5/d45/d67 0 2024-04-05T21:26:37.506 INFO:tasks.workunit.client.0.smithi121.stdout:6/210: write d1/d2/f17 [705728,13469] 0 2024-04-05T21:26:37.507 INFO:tasks.workunit.client.0.smithi121.stdout:6/211: chown d1/c24 0 1 2024-04-05T21:26:37.507 INFO:tasks.workunit.client.1.smithi188.stdout:0/370: rmdir d4/d8 39 2024-04-05T21:26:37.509 INFO:tasks.workunit.client.0.smithi121.stdout:6/212: mkdir d1/d2/d3/d5/d1f/d2c/d41 0 2024-04-05T21:26:37.512 INFO:tasks.workunit.client.0.smithi121.stdout:6/213: unlink d1/d2/f17 0 2024-04-05T21:26:37.512 INFO:tasks.workunit.client.1.smithi188.stdout:3/554: dread d5/dc/f1f [4194304,4194304] 0 2024-04-05T21:26:37.512 INFO:tasks.workunit.client.0.smithi121.stdout:6/214: read - d1/d2/d1b/f28 zero size 2024-04-05T21:26:37.514 INFO:tasks.workunit.client.1.smithi188.stdout:0/371: symlink d4/d8/db/d21/d3c/l85 0 2024-04-05T21:26:37.515 INFO:tasks.workunit.client.0.smithi121.stdout:1/255: truncate d8/d10/d14/f19 1035593 0 2024-04-05T21:26:37.516 INFO:tasks.workunit.client.1.smithi188.stdout:3/555: mkdir d5/d7/d65/d61/db5/dbb 0 2024-04-05T21:26:37.517 INFO:tasks.workunit.client.0.smithi121.stdout:1/256: fsync d8/f1d 0 2024-04-05T21:26:37.520 INFO:tasks.workunit.client.0.smithi121.stdout:1/257: mknod d8/d10/d14/c58 0 2024-04-05T21:26:37.522 INFO:tasks.workunit.client.0.smithi121.stdout:6/215: dwrite d1/d2/dc/f19 [0,4194304] 0 2024-04-05T21:26:37.523 INFO:tasks.workunit.client.0.smithi121.stdout:4/298: dwrite d5/d11/d2c/f3c [0,4194304] 0 2024-04-05T21:26:37.523 INFO:tasks.workunit.client.1.smithi188.stdout:0/372: link d4/d8/db/d21/f34 d4/d8/db/d42/d38/d77/f86 0 2024-04-05T21:26:37.523 INFO:tasks.workunit.client.1.smithi188.stdout:8/488: dwrite d5/d21/d40/d43/faa [0,4194304] 0 2024-04-05T21:26:37.526 INFO:tasks.workunit.client.0.smithi121.stdout:1/258: symlink d8/d10/d14/d1a/d1b/d44/l59 0 2024-04-05T21:26:37.532 INFO:tasks.workunit.client.0.smithi121.stdout:4/299: creat d5/d11/d26/d47/f68 x:0 0 0 2024-04-05T21:26:37.532 INFO:tasks.workunit.client.1.smithi188.stdout:0/373: creat d4/d8/db/d42/d71/f87 x:0 0 0 2024-04-05T21:26:37.533 INFO:tasks.workunit.client.0.smithi121.stdout:1/259: rmdir d8/d10/d14/d1a/d1b/d38/d4b 39 2024-04-05T21:26:37.534 INFO:tasks.workunit.client.0.smithi121.stdout:4/300: write d5/d11/d26/d47/f66 [1047141,83675] 0 2024-04-05T21:26:37.534 INFO:tasks.workunit.client.0.smithi121.stdout:1/260: fsync d8/d10/d14/d1a/f20 0 2024-04-05T21:26:37.535 INFO:tasks.workunit.client.0.smithi121.stdout:4/301: fsync d5/d11/f17 0 2024-04-05T21:26:37.535 INFO:tasks.workunit.client.1.smithi188.stdout:0/374: readlink d4/d6/d13/d23/d24/l35 0 2024-04-05T21:26:37.536 INFO:tasks.workunit.client.0.smithi121.stdout:4/302: read d5/d11/d26/d27/d64/f31 [763310,16860] 0 2024-04-05T21:26:37.537 INFO:tasks.workunit.client.1.smithi188.stdout:0/375: chown d4/d8/db/d21/c7c 1809 1 2024-04-05T21:26:37.537 INFO:tasks.workunit.client.0.smithi121.stdout:6/216: dread d1/f38 [0,4194304] 0 2024-04-05T21:26:37.538 INFO:tasks.workunit.client.0.smithi121.stdout:1/261: fdatasync d8/d10/d14/d27/f30 0 2024-04-05T21:26:37.539 INFO:tasks.workunit.client.1.smithi188.stdout:3/556: dwrite d5/d7/f30 [0,4194304] 0 2024-04-05T21:26:37.539 INFO:tasks.workunit.client.0.smithi121.stdout:4/303: symlink d5/d11/d28/l69 0 2024-04-05T21:26:37.542 INFO:tasks.workunit.client.1.smithi188.stdout:3/557: fsync d5/d2b/f79 0 2024-04-05T21:26:37.545 INFO:tasks.workunit.client.1.smithi188.stdout:3/558: mknod d5/d7/d65/d5f/cbc 0 2024-04-05T21:26:37.546 INFO:tasks.workunit.client.0.smithi121.stdout:4/304: stat d5/d11/d26/d27/d29/d58 0 2024-04-05T21:26:37.547 INFO:tasks.workunit.client.0.smithi121.stdout:1/262: link d8/d10/d14/d1a/d1b/d24/f2a d8/d10/d14/d1a/f5a 0 2024-04-05T21:26:37.548 INFO:tasks.workunit.client.0.smithi121.stdout:4/305: mknod d5/d11/d20/c6a 0 2024-04-05T21:26:37.549 INFO:tasks.workunit.client.0.smithi121.stdout:4/306: chown d5/d11/d20/f38 51338059 1 2024-04-05T21:26:37.550 INFO:tasks.workunit.client.1.smithi188.stdout:5/487: sync 2024-04-05T21:26:37.550 INFO:tasks.workunit.client.1.smithi188.stdout:1/560: sync 2024-04-05T21:26:37.550 INFO:tasks.workunit.client.1.smithi188.stdout:4/744: sync 2024-04-05T21:26:37.550 INFO:tasks.workunit.client.1.smithi188.stdout:9/482: sync 2024-04-05T21:26:37.551 INFO:tasks.workunit.client.0.smithi121.stdout:1/263: write d8/d10/d14/f2c [999227,81870] 0 2024-04-05T21:26:37.551 INFO:tasks.workunit.client.0.smithi121.stdout:4/307: chown d5/f3f 1 1 2024-04-05T21:26:37.552 INFO:tasks.workunit.client.1.smithi188.stdout:8/489: dwrite d5/d21/d40/d43/d44/d5b/d99/d69/d70/f9d [8388608,4194304] 0 2024-04-05T21:26:37.552 INFO:tasks.workunit.client.1.smithi188.stdout:3/559: creat d5/d7/d65/d5f/fbd x:0 0 0 2024-04-05T21:26:37.557 INFO:tasks.workunit.client.0.smithi121.stdout:4/308: dwrite d5/d11/f14 [0,4194304] 0 2024-04-05T21:26:37.557 INFO:tasks.workunit.client.1.smithi188.stdout:5/488: read d6/d17/d39/f4a [2608672,45926] 0 2024-04-05T21:26:37.558 INFO:tasks.workunit.client.1.smithi188.stdout:0/376: write d4/d6/f15 [910011,65659] 0 2024-04-05T21:26:37.558 INFO:tasks.workunit.client.1.smithi188.stdout:4/745: rmdir d0/d2/dc7 39 2024-04-05T21:26:37.560 INFO:tasks.workunit.client.0.smithi121.stdout:6/217: mkdir d1/d2/d3/d5/d1f/d2c/d41/d42 0 2024-04-05T21:26:37.563 INFO:tasks.workunit.client.1.smithi188.stdout:8/490: dread d5/f1c [0,4194304] 0 2024-04-05T21:26:37.565 INFO:tasks.workunit.client.1.smithi188.stdout:1/561: fsync d1/dc/f56 0 2024-04-05T21:26:37.566 INFO:tasks.workunit.client.1.smithi188.stdout:9/483: truncate dd/d34/f23 3607640 0 2024-04-05T21:26:37.567 INFO:tasks.workunit.client.0.smithi121.stdout:4/309: creat d5/d11/d26/f6b x:0 0 0 2024-04-05T21:26:37.568 INFO:tasks.workunit.client.1.smithi188.stdout:8/491: fsync d5/d21/d2a/f8d 0 2024-04-05T21:26:37.569 INFO:tasks.workunit.client.1.smithi188.stdout:0/377: dread d4/d8/d10/f20 [0,4194304] 0 2024-04-05T21:26:37.570 INFO:tasks.workunit.client.0.smithi121.stdout:4/310: symlink d5/d45/d67/l6c 0 2024-04-05T21:26:37.571 INFO:tasks.workunit.client.0.smithi121.stdout:4/311: fdatasync d5/f1d 0 2024-04-05T21:26:37.572 INFO:tasks.workunit.client.0.smithi121.stdout:6/218: dwrite d1/f10 [0,4194304] 0 2024-04-05T21:26:37.573 INFO:tasks.workunit.client.0.smithi121.stdout:4/312: truncate d5/d11/d20/f57 272451 0 2024-04-05T21:26:37.573 INFO:tasks.workunit.client.1.smithi188.stdout:9/484: symlink dd/d34/l9d 0 2024-04-05T21:26:37.575 INFO:tasks.workunit.client.1.smithi188.stdout:9/485: stat dd/d49 0 2024-04-05T21:26:37.575 INFO:tasks.workunit.client.1.smithi188.stdout:5/489: dwrite d6/d17/f20 [0,4194304] 0 2024-04-05T21:26:37.577 INFO:tasks.workunit.client.0.smithi121.stdout:4/313: read d5/d11/d26/d47/f66 [965513,65792] 0 2024-04-05T21:26:37.579 INFO:tasks.workunit.client.1.smithi188.stdout:3/560: truncate d5/d7/d31/f91 3710250 0 2024-04-05T21:26:37.579 INFO:tasks.workunit.client.1.smithi188.stdout:1/562: dread d1/d7/f39 [0,4194304] 0 2024-04-05T21:26:37.580 INFO:tasks.workunit.client.1.smithi188.stdout:5/490: truncate d6/d15/d5c/d7a/d44/da0/fb0 952505 0 2024-04-05T21:26:37.580 INFO:tasks.workunit.client.0.smithi121.stdout:6/219: dread d1/d2/d3/d4/de/f26 [0,4194304] 0 2024-04-05T21:26:37.580 INFO:tasks.workunit.client.1.smithi188.stdout:8/492: symlink d5/d21/d2a/d71/lab 0 2024-04-05T21:26:37.582 INFO:tasks.workunit.client.1.smithi188.stdout:3/561: readlink d5/d2b/l4a 0 2024-04-05T21:26:37.584 INFO:tasks.workunit.client.1.smithi188.stdout:8/493: unlink d5/f42 0 2024-04-05T21:26:37.585 INFO:tasks.workunit.client.1.smithi188.stdout:5/491: truncate d6/d15/d1b/d4f/f59 2079015 0 2024-04-05T21:26:37.585 INFO:tasks.workunit.client.0.smithi121.stdout:1/264: dread d8/d10/d14/f19 [0,4194304] 0 2024-04-05T21:26:37.586 INFO:tasks.workunit.client.0.smithi121.stdout:4/314: creat d5/d11/d26/d27/d29/d58/f6d x:0 0 0 2024-04-05T21:26:37.586 INFO:tasks.workunit.client.1.smithi188.stdout:0/378: creat d4/d8/db/f88 x:0 0 0 2024-04-05T21:26:37.587 INFO:tasks.workunit.client.0.smithi121.stdout:6/220: mknod d1/d2/d3/d5/d1f/d2c/d41/d42/c43 0 2024-04-05T21:26:37.588 INFO:tasks.workunit.client.1.smithi188.stdout:0/379: stat d4/d8/db/d42/d71/l7f 0 2024-04-05T21:26:37.588 INFO:tasks.workunit.client.0.smithi121.stdout:4/315: creat d5/d11/d26/d27/d29/d58/f6e x:0 0 0 2024-04-05T21:26:37.589 INFO:tasks.workunit.client.0.smithi121.stdout:1/265: rmdir d8/d10/d14/d1a/d1b/d38/d4b 39 2024-04-05T21:26:37.590 INFO:tasks.workunit.client.0.smithi121.stdout:4/316: chown d5/d11/d26/d27/f30 14891428 1 2024-04-05T21:26:37.591 INFO:tasks.workunit.client.0.smithi121.stdout:4/317: read d5/d11/d28/d3b/f4a [1556248,110783] 0 2024-04-05T21:26:37.592 INFO:tasks.workunit.client.0.smithi121.stdout:6/221: dread d1/d2/d3/d5/d1f/f27 [0,4194304] 0 2024-04-05T21:26:37.592 INFO:tasks.workunit.client.0.smithi121.stdout:4/318: read d5/d11/f21 [1115109,11675] 0 2024-04-05T21:26:37.593 INFO:tasks.workunit.client.1.smithi188.stdout:5/492: unlink d6/d15/d1b/c72 0 2024-04-05T21:26:37.595 INFO:tasks.workunit.client.1.smithi188.stdout:0/380: symlink d4/d8/db/d21/d4a/l89 0 2024-04-05T21:26:37.596 INFO:tasks.workunit.client.0.smithi121.stdout:1/266: dwrite f2 [0,4194304] 0 2024-04-05T21:26:37.597 INFO:tasks.workunit.client.1.smithi188.stdout:1/563: truncate d1/dc/d16/d46/d88/d86/fab 1837308 0 2024-04-05T21:26:37.597 INFO:tasks.workunit.client.0.smithi121.stdout:6/222: read d1/f38 [419630,61624] 0 2024-04-05T21:26:37.599 INFO:tasks.workunit.client.1.smithi188.stdout:0/381: mknod d4/d6/d61/c8a 0 2024-04-05T21:26:37.600 INFO:tasks.workunit.client.1.smithi188.stdout:9/486: dwrite dd/d49/f60 [0,4194304] 0 2024-04-05T21:26:37.600 INFO:tasks.workunit.client.0.smithi121.stdout:4/319: dread d5/d11/f21 [0,4194304] 0 2024-04-05T21:26:37.601 INFO:tasks.workunit.client.0.smithi121.stdout:1/267: mkdir d8/d10/d14/d27/d2d/d3b/d53/d5b 0 2024-04-05T21:26:37.601 INFO:tasks.workunit.client.1.smithi188.stdout:3/562: dread d5/d7/d31/f45 [0,4194304] 0 2024-04-05T21:26:37.603 INFO:tasks.workunit.client.0.smithi121.stdout:4/320: write d5/d11/d26/d27/d64/f40 [3460445,83409] 0 2024-04-05T21:26:37.603 INFO:tasks.workunit.client.0.smithi121.stdout:6/223: getdents d1/d2/d1b/d1e 0 2024-04-05T21:26:37.605 INFO:tasks.workunit.client.0.smithi121.stdout:1/268: mkdir d8/d10/d14/d27/d5c 0 2024-04-05T21:26:37.610 INFO:tasks.workunit.client.0.smithi121.stdout:4/321: link d5/d11/d26/d27/d29/c46 d5/d11/d26/d27/c6f 0 2024-04-05T21:26:37.611 INFO:tasks.workunit.client.1.smithi188.stdout:3/563: creat d5/d7/d65/fbe x:0 0 0 2024-04-05T21:26:37.611 INFO:tasks.workunit.client.1.smithi188.stdout:9/487: creat dd/d3d/d95/f9e x:0 0 0 2024-04-05T21:26:37.614 INFO:tasks.workunit.client.1.smithi188.stdout:9/488: chown dd/d35/c7c 138379906 1 2024-04-05T21:26:37.614 INFO:tasks.workunit.client.0.smithi121.stdout:1/269: dwrite d8/f1c [0,4194304] 0 2024-04-05T21:26:37.617 INFO:tasks.workunit.client.0.smithi121.stdout:1/270: chown d8/d10/d14/d1a/d1b/d24/f2a 4 1 2024-04-05T21:26:37.619 INFO:tasks.workunit.client.0.smithi121.stdout:4/322: dwrite d5/d11/d20/f38 [0,4194304] 0 2024-04-05T21:26:37.623 INFO:tasks.workunit.client.0.smithi121.stdout:3/192: sync 2024-04-05T21:26:37.623 INFO:tasks.workunit.client.0.smithi121.stdout:4/323: truncate d5/d11/d28/fe 1201290 0 2024-04-05T21:26:37.623 INFO:tasks.workunit.client.0.smithi121.stdout:8/183: sync 2024-04-05T21:26:37.624 INFO:tasks.workunit.client.1.smithi188.stdout:3/564: creat d5/d7/d65/d61/db5/fbf x:0 0 0 2024-04-05T21:26:37.624 INFO:tasks.workunit.client.0.smithi121.stdout:4/324: fdatasync d5/f3f 0 2024-04-05T21:26:37.625 INFO:tasks.workunit.client.0.smithi121.stdout:8/184: readlink d12/d13/d15/d1d/l2d 0 2024-04-05T21:26:37.625 INFO:tasks.workunit.client.1.smithi188.stdout:9/489: fsync dd/d11/d73/d85/f93 0 2024-04-05T21:26:37.626 INFO:tasks.workunit.client.0.smithi121.stdout:4/325: write d5/f55 [527560,20026] 0 2024-04-05T21:26:37.627 INFO:tasks.workunit.client.1.smithi188.stdout:9/490: readlink dd/l30 0 2024-04-05T21:26:37.629 INFO:tasks.workunit.client.1.smithi188.stdout:0/382: dwrite d4/d8/db/f76 [0,4194304] 0 2024-04-05T21:26:37.629 INFO:tasks.workunit.client.0.smithi121.stdout:8/185: mkdir d12/d13/d15/d1d/d29/d3e 0 2024-04-05T21:26:37.631 INFO:tasks.workunit.client.0.smithi121.stdout:4/326: mknod d5/d45/d67/c70 0 2024-04-05T21:26:37.633 INFO:tasks.workunit.client.0.smithi121.stdout:3/193: rmdir d0/d3/de/d3b 0 2024-04-05T21:26:37.634 INFO:tasks.workunit.client.0.smithi121.stdout:8/186: symlink d12/d13/d15/d1d/l3f 0 2024-04-05T21:26:37.635 INFO:tasks.workunit.client.0.smithi121.stdout:3/194: dread - d0/d3/f30 zero size 2024-04-05T21:26:37.637 INFO:tasks.workunit.client.0.smithi121.stdout:6/224: dwrite d1/d2/d3/d5/fd [0,4194304] 0 2024-04-05T21:26:37.640 INFO:tasks.workunit.client.0.smithi121.stdout:4/327: rename d5/d11/l4e to d5/l71 0 2024-04-05T21:26:37.640 INFO:tasks.workunit.client.0.smithi121.stdout:6/225: chown d1/d2/d1b/d1e/f2d 887854 1 2024-04-05T21:26:37.645 INFO:tasks.workunit.client.0.smithi121.stdout:8/187: creat d12/d13/d15/d1d/d29/f40 x:0 0 0 2024-04-05T21:26:37.645 INFO:tasks.workunit.client.0.smithi121.stdout:6/226: fsync d1/d2/d3/d5/d1f/f27 0 2024-04-05T21:26:37.645 INFO:tasks.workunit.client.1.smithi188.stdout:3/565: rename d5/d7/d65/d61/f94 to d5/d2b/fc0 0 2024-04-05T21:26:37.649 INFO:tasks.workunit.client.1.smithi188.stdout:9/491: unlink dd/d3d/f4c 0 2024-04-05T21:26:37.650 INFO:tasks.workunit.client.0.smithi121.stdout:6/227: unlink d1/d2/d3/d5/f1d 0 2024-04-05T21:26:37.651 INFO:tasks.workunit.client.1.smithi188.stdout:3/566: unlink d5/d2b/f79 0 2024-04-05T21:26:37.652 INFO:tasks.workunit.client.0.smithi121.stdout:3/195: dwrite d0/d3/de/f17 [0,4194304] 0 2024-04-05T21:26:37.653 INFO:tasks.workunit.client.0.smithi121.stdout:6/228: write d1/d2/dc/f3e [111304,116495] 0 2024-04-05T21:26:37.654 INFO:tasks.workunit.client.0.smithi121.stdout:8/188: dwrite fc [0,4194304] 0 2024-04-05T21:26:37.655 INFO:tasks.workunit.client.1.smithi188.stdout:9/492: symlink dd/d49/d6b/d40/l9f 0 2024-04-05T21:26:37.655 INFO:tasks.workunit.client.0.smithi121.stdout:3/196: chown d0/d3/d24/d3d/d1a/d1e 1653844857 1 2024-04-05T21:26:37.656 INFO:tasks.workunit.client.1.smithi188.stdout:3/567: creat d5/d7/d6d/d75/fc1 x:0 0 0 2024-04-05T21:26:37.657 INFO:tasks.workunit.client.0.smithi121.stdout:6/229: mknod d1/d2/d3/c44 0 2024-04-05T21:26:37.658 INFO:tasks.workunit.client.0.smithi121.stdout:8/189: symlink d12/d13/d15/l41 0 2024-04-05T21:26:37.659 INFO:tasks.workunit.client.1.smithi188.stdout:9/493: mknod dd/ca0 0 2024-04-05T21:26:37.659 INFO:tasks.workunit.client.0.smithi121.stdout:8/190: write fc [2439232,118443] 0 2024-04-05T21:26:37.661 INFO:tasks.workunit.client.1.smithi188.stdout:3/568: getdents d5/d7/d65/d61/db5/dbb 0 2024-04-05T21:26:37.666 INFO:tasks.workunit.client.0.smithi121.stdout:1/271: dread d8/f1d [0,4194304] 0 2024-04-05T21:26:37.667 INFO:tasks.workunit.client.0.smithi121.stdout:6/230: dwrite d1/d2/d3/d4/de/f26 [0,4194304] 0 2024-04-05T21:26:37.668 INFO:tasks.workunit.client.0.smithi121.stdout:1/272: chown d8/d10/d14/d1a 2226 1 2024-04-05T21:26:37.668 INFO:tasks.workunit.client.1.smithi188.stdout:3/569: creat d5/d7/d65/d61/d66/fc2 x:0 0 0 2024-04-05T21:26:37.670 INFO:tasks.workunit.client.0.smithi121.stdout:8/191: dwrite d12/d13/d15/f19 [0,4194304] 0 2024-04-05T21:26:37.677 INFO:tasks.workunit.client.1.smithi188.stdout:3/570: link d5/df/c28 d5/cc3 0 2024-04-05T21:26:37.679 INFO:tasks.workunit.client.0.smithi121.stdout:6/231: symlink d1/d2/d1b/l45 0 2024-04-05T21:26:37.680 INFO:tasks.workunit.client.1.smithi188.stdout:6/553: sync 2024-04-05T21:26:37.680 INFO:tasks.workunit.client.1.smithi188.stdout:2/555: sync 2024-04-05T21:26:37.680 INFO:tasks.workunit.client.1.smithi188.stdout:7/674: sync 2024-04-05T21:26:37.681 INFO:tasks.workunit.client.0.smithi121.stdout:8/192: creat d12/d13/d15/d1d/f42 x:0 0 0 2024-04-05T21:26:37.682 INFO:tasks.workunit.client.0.smithi121.stdout:6/232: creat d1/d2/d3/d5/d1f/f46 x:0 0 0 2024-04-05T21:26:37.683 INFO:tasks.workunit.client.0.smithi121.stdout:1/273: creat d8/d10/d14/d1a/d1b/d38/f5d x:0 0 0 2024-04-05T21:26:37.684 INFO:tasks.workunit.client.0.smithi121.stdout:6/233: dread - d1/d2/d3/d5/d36/f3d zero size 2024-04-05T21:26:37.685 INFO:tasks.workunit.client.0.smithi121.stdout:1/274: fsync d8/d10/d14/d1a/d1b/d24/f39 0 2024-04-05T21:26:37.686 INFO:tasks.workunit.client.0.smithi121.stdout:6/234: write d1/d2/d3/d5/d36/f3d [71733,94375] 0 2024-04-05T21:26:37.687 INFO:tasks.workunit.client.1.smithi188.stdout:7/675: creat d3/da5/fd3 x:0 0 0 2024-04-05T21:26:37.689 INFO:tasks.workunit.client.0.smithi121.stdout:8/193: unlink d12/c24 0 2024-04-05T21:26:37.689 INFO:tasks.workunit.client.1.smithi188.stdout:7/676: chown d3 256 1 2024-04-05T21:26:37.691 INFO:tasks.workunit.client.0.smithi121.stdout:1/275: dread d8/d10/f26 [0,4194304] 0 2024-04-05T21:26:37.692 INFO:tasks.workunit.client.0.smithi121.stdout:8/194: mknod d12/d13/c43 0 2024-04-05T21:26:37.693 INFO:tasks.workunit.client.0.smithi121.stdout:3/197: truncate d0/d3/d24/d3d/d1a/f22 1549718 0 2024-04-05T21:26:37.695 INFO:tasks.workunit.client.1.smithi188.stdout:3/571: dwrite d5/d7/d65/d61/f7b [0,4194304] 0 2024-04-05T21:26:37.695 INFO:tasks.workunit.client.0.smithi121.stdout:1/276: dread - d8/d10/d14/d27/d2d/f41 zero size 2024-04-05T21:26:37.696 INFO:tasks.workunit.client.1.smithi188.stdout:0/383: dwrite d4/d8/db/d21/f44 [0,4194304] 0 2024-04-05T21:26:37.698 INFO:tasks.workunit.client.1.smithi188.stdout:6/554: link d6/d13/d18/c25 d6/d7/cad 0 2024-04-05T21:26:37.699 INFO:tasks.workunit.client.0.smithi121.stdout:6/235: dwrite d1/d2/d3/d5/d1f/f27 [0,4194304] 0 2024-04-05T21:26:37.701 INFO:tasks.workunit.client.1.smithi188.stdout:2/556: dwrite d5/db/f16 [4194304,4194304] 0 2024-04-05T21:26:37.703 INFO:tasks.workunit.client.0.smithi121.stdout:8/195: read d12/d13/d15/d1d/f27 [680673,124619] 0 2024-04-05T21:26:37.708 INFO:tasks.workunit.client.1.smithi188.stdout:3/572: write d5/d7/d6d/d75/faa [1583586,118883] 0 2024-04-05T21:26:37.709 INFO:tasks.workunit.client.0.smithi121.stdout:1/277: symlink d8/d10/d14/d27/d2d/d3b/d53/l5e 0 2024-04-05T21:26:37.710 INFO:tasks.workunit.client.1.smithi188.stdout:3/573: creat d5/df/d80/fc4 x:0 0 0 2024-04-05T21:26:37.711 INFO:tasks.workunit.client.0.smithi121.stdout:6/236: unlink d1/d2/d3/d5/d36/f3d 0 2024-04-05T21:26:37.711 INFO:tasks.workunit.client.1.smithi188.stdout:3/574: chown d5/d7/d65/d5f/fb4 2415 1 2024-04-05T21:26:37.713 INFO:tasks.workunit.client.0.smithi121.stdout:6/237: write d1/d2/d3/d5/d1f/f46 [461278,121883] 0 2024-04-05T21:26:37.713 INFO:tasks.workunit.client.1.smithi188.stdout:6/555: creat d6/d13/d18/d19/d2c/d3f/fae x:0 0 0 2024-04-05T21:26:37.714 INFO:tasks.workunit.client.1.smithi188.stdout:2/557: mkdir d5/db/dd/d36/d3c/d48/d7a/d7c/db5 0 2024-04-05T21:26:37.715 INFO:tasks.workunit.client.1.smithi188.stdout:2/558: readlink d5/db/d35/l4b 0 2024-04-05T21:26:37.717 INFO:tasks.workunit.client.1.smithi188.stdout:6/556: mkdir d6/d13/d18/d3c/daf 0 2024-04-05T21:26:37.717 INFO:tasks.workunit.client.1.smithi188.stdout:3/575: getdents d5/d7/d65/da2 0 2024-04-05T21:26:37.718 INFO:tasks.workunit.client.0.smithi121.stdout:6/238: creat d1/d2/d1b/d35/f47 x:0 0 0 2024-04-05T21:26:37.718 INFO:tasks.workunit.client.1.smithi188.stdout:2/559: mknod d5/db/dd/d36/d3c/d48/d78/cb6 0 2024-04-05T21:26:37.719 INFO:tasks.workunit.client.0.smithi121.stdout:1/278: rmdir d8/d10/d14/d1a/d1b/d38/d4b 39 2024-04-05T21:26:37.719 INFO:tasks.workunit.client.1.smithi188.stdout:6/557: chown d6/d13/d18/d19/d61/fab 14037229 1 2024-04-05T21:26:37.719 INFO:tasks.workunit.client.1.smithi188.stdout:2/560: chown d5/db/dd/c38 262287541 1 2024-04-05T21:26:37.720 INFO:tasks.workunit.client.1.smithi188.stdout:3/576: readlink d5/d39/d84/la7 0 2024-04-05T21:26:37.723 INFO:tasks.workunit.client.0.smithi121.stdout:8/196: dwrite f3 [0,4194304] 0 2024-04-05T21:26:37.723 INFO:tasks.workunit.client.0.smithi121.stdout:6/239: write d1/d2/d3/d5/fd [1775085,62956] 0 2024-04-05T21:26:37.725 INFO:tasks.workunit.client.1.smithi188.stdout:2/561: creat d5/fb7 x:0 0 0 2024-04-05T21:26:37.726 INFO:tasks.workunit.client.0.smithi121.stdout:1/279: creat d8/d10/d14/d1a/d1b/d24/f5f x:0 0 0 2024-04-05T21:26:37.727 INFO:tasks.workunit.client.0.smithi121.stdout:6/240: mkdir d1/d2/d3/d5/d1f/d2c/d41/d42/d48 0 2024-04-05T21:26:37.727 INFO:tasks.workunit.client.1.smithi188.stdout:2/562: dread - d5/db/d35/fab zero size 2024-04-05T21:26:37.728 INFO:tasks.workunit.client.0.smithi121.stdout:8/197: mknod d12/d13/d15/c44 0 2024-04-05T21:26:37.729 INFO:tasks.workunit.client.0.smithi121.stdout:8/198: write d12/d13/d15/f1c [1032037,2697] 0 2024-04-05T21:26:37.730 INFO:tasks.workunit.client.1.smithi188.stdout:0/384: truncate d4/d8/db/f45 1150927 0 2024-04-05T21:26:37.731 INFO:tasks.workunit.client.1.smithi188.stdout:6/558: dwrite d6/d13/d18/d19/d2c/f65 [0,4194304] 0 2024-04-05T21:26:37.733 INFO:tasks.workunit.client.0.smithi121.stdout:1/280: rename d8/d10/d14/d27/f29 to d8/d10/d14/d27/d57/f60 0 2024-04-05T21:26:37.733 INFO:tasks.workunit.client.0.smithi121.stdout:1/281: dread - d8/d10/d14/f23 zero size 2024-04-05T21:26:37.733 INFO:tasks.workunit.client.0.smithi121.stdout:6/241: creat d1/d2/d3/d5/d1f/d2c/d41/f49 x:0 0 0 2024-04-05T21:26:37.735 INFO:tasks.workunit.client.1.smithi188.stdout:2/563: mknod d5/d5f/d98/cb8 0 2024-04-05T21:26:37.735 INFO:tasks.workunit.client.0.smithi121.stdout:6/242: stat d1/d2/d1b/d2e 0 2024-04-05T21:26:37.735 INFO:tasks.workunit.client.1.smithi188.stdout:0/385: creat d4/d6/d46/f8b x:0 0 0 2024-04-05T21:26:37.737 INFO:tasks.workunit.client.1.smithi188.stdout:6/559: readlink d6/d13/d48/d49/d75/l9e 0 2024-04-05T21:26:37.738 INFO:tasks.workunit.client.0.smithi121.stdout:1/282: rmdir d8/d10 39 2024-04-05T21:26:37.739 INFO:tasks.workunit.client.0.smithi121.stdout:6/243: fsync d1/f38 0 2024-04-05T21:26:37.739 INFO:tasks.workunit.client.1.smithi188.stdout:0/386: creat d4/d8/db/d42/d71/f8c x:0 0 0 2024-04-05T21:26:37.740 INFO:tasks.workunit.client.0.smithi121.stdout:8/199: creat d12/d13/d15/d1d/d29/d3e/f45 x:0 0 0 2024-04-05T21:26:37.740 INFO:tasks.workunit.client.0.smithi121.stdout:6/244: truncate d1/f10 4746002 0 2024-04-05T21:26:37.743 INFO:tasks.workunit.client.0.smithi121.stdout:6/245: write d1/d2/dc/f19 [1669602,118465] 0 2024-04-05T21:26:37.745 INFO:tasks.workunit.client.0.smithi121.stdout:6/246: dread - d1/d2/d3/d5/d1f/d2c/d41/f49 zero size 2024-04-05T21:26:37.746 INFO:tasks.workunit.client.0.smithi121.stdout:1/283: rename d8/d10/l3e to d8/d10/d14/d27/d2d/l61 0 2024-04-05T21:26:37.746 INFO:tasks.workunit.client.0.smithi121.stdout:6/247: mknod d1/d2/d3/d5/c4a 0 2024-04-05T21:26:37.747 INFO:tasks.workunit.client.1.smithi188.stdout:3/577: dwrite d5/dc/f2a [0,4194304] 0 2024-04-05T21:26:37.748 INFO:tasks.workunit.client.0.smithi121.stdout:6/248: mkdir d1/d4b 0 2024-04-05T21:26:37.748 INFO:tasks.workunit.client.1.smithi188.stdout:0/387: mknod d4/d6/d46/c8d 0 2024-04-05T21:26:37.749 INFO:tasks.workunit.client.0.smithi121.stdout:6/249: creat d1/d2/dc/f4c x:0 0 0 2024-04-05T21:26:37.751 INFO:tasks.workunit.client.1.smithi188.stdout:3/578: getdents d5/df/d80 0 2024-04-05T21:26:37.751 INFO:tasks.workunit.client.0.smithi121.stdout:6/250: truncate d1/d2/d3/d5/d1f/d2c/d41/f49 591681 0 2024-04-05T21:26:37.752 INFO:tasks.workunit.client.0.smithi121.stdout:6/251: chown d1/d2/d3/d5/d1f/f27 590 1 2024-04-05T21:26:37.754 INFO:tasks.workunit.client.0.smithi121.stdout:6/252: symlink d1/d2/d1b/d2e/l4d 0 2024-04-05T21:26:37.756 INFO:tasks.workunit.client.1.smithi188.stdout:7/677: dread d3/f1e [0,4194304] 0 2024-04-05T21:26:37.756 INFO:tasks.workunit.client.1.smithi188.stdout:0/388: mknod d4/d8/db/d42/d38/d77/c8e 0 2024-04-05T21:26:37.758 INFO:tasks.workunit.client.0.smithi121.stdout:3/198: dread d0/d3/d24/d3d/f21 [0,4194304] 0 2024-04-05T21:26:37.762 INFO:tasks.workunit.client.1.smithi188.stdout:0/389: creat d4/d6/d61/f8f x:0 0 0 2024-04-05T21:26:37.762 INFO:tasks.workunit.client.0.smithi121.stdout:3/199: mknod d0/d3/d24/d3d/c3e 0 2024-04-05T21:26:37.763 INFO:tasks.workunit.client.0.smithi121.stdout:1/284: dread - d8/d10/d14/f37 zero size 2024-04-05T21:26:37.764 INFO:tasks.workunit.client.0.smithi121.stdout:3/200: dread - d0/d3/d24/d3d/d1a/d1e/f2a zero size 2024-04-05T21:26:37.765 INFO:tasks.workunit.client.0.smithi121.stdout:3/201: truncate d0/d3/f30 288685 0 2024-04-05T21:26:37.765 INFO:tasks.workunit.client.1.smithi188.stdout:0/390: rename d4/d8/db/d42/f4c to d4/d8/db/d21/f90 0 2024-04-05T21:26:37.767 INFO:tasks.workunit.client.1.smithi188.stdout:3/579: write d5/dc/d2f/f89 [825955,119247] 0 2024-04-05T21:26:37.767 INFO:tasks.workunit.client.0.smithi121.stdout:3/202: mkdir d0/d3/d24/d3f 0 2024-04-05T21:26:37.768 INFO:tasks.workunit.client.1.smithi188.stdout:7/678: write d3/d4/db/d10/fc9 [1290902,82943] 0 2024-04-05T21:26:37.770 INFO:tasks.workunit.client.0.smithi121.stdout:1/285: dwrite d8/d10/d14/d1a/f20 [0,4194304] 0 2024-04-05T21:26:37.772 INFO:tasks.workunit.client.0.smithi121.stdout:3/203: mknod d0/d3/d24/d3d/d1a/d1e/c40 0 2024-04-05T21:26:37.772 INFO:tasks.workunit.client.1.smithi188.stdout:6/560: dwrite d6/d13/f1d [0,4194304] 0 2024-04-05T21:26:37.772 INFO:tasks.workunit.client.1.smithi188.stdout:0/391: mkdir d4/d8/db/d21/d4a/d73/d79/d91 0 2024-04-05T21:26:37.772 INFO:tasks.workunit.client.1.smithi188.stdout:3/580: truncate d5/df/d80/fc4 381414 0 2024-04-05T21:26:37.774 INFO:tasks.workunit.client.0.smithi121.stdout:1/286: creat d8/d10/d14/d27/d2d/d3b/f62 x:0 0 0 2024-04-05T21:26:37.775 INFO:tasks.workunit.client.1.smithi188.stdout:7/679: mkdir d3/d4/db/d10/d5f/dd4 0 2024-04-05T21:26:37.776 INFO:tasks.workunit.client.0.smithi121.stdout:1/287: stat d8/d10/d14/d1a/d1b/d24/c2f 0 2024-04-05T21:26:37.776 INFO:tasks.workunit.client.1.smithi188.stdout:3/581: creat d5/d2b/fc5 x:0 0 0 2024-04-05T21:26:37.777 INFO:tasks.workunit.client.1.smithi188.stdout:3/582: readlink d5/dc/l3f 0 2024-04-05T21:26:37.778 INFO:tasks.workunit.client.1.smithi188.stdout:0/392: creat d4/d6/d13/d23/d24/d57/f92 x:0 0 0 2024-04-05T21:26:37.778 INFO:tasks.workunit.client.0.smithi121.stdout:3/204: rename d0/d3/d24/f34 to d0/f41 0 2024-04-05T21:26:37.779 INFO:tasks.workunit.client.0.smithi121.stdout:1/288: creat d8/d10/d14/d1a/d1b/d4c/f63 x:0 0 0 2024-04-05T21:26:37.780 INFO:tasks.workunit.client.1.smithi188.stdout:7/680: creat d3/d4/db/d2e/d3b/fd5 x:0 0 0 2024-04-05T21:26:37.780 INFO:tasks.workunit.client.0.smithi121.stdout:3/205: creat d0/d3/d24/f42 x:0 0 0 2024-04-05T21:26:37.782 INFO:tasks.workunit.client.0.smithi121.stdout:1/289: truncate d8/d10/d14/d27/f30 50896 0 2024-04-05T21:26:37.783 INFO:tasks.workunit.client.0.smithi121.stdout:3/206: mknod d0/d3/d24/d3d/d1a/d23/d38/c43 0 2024-04-05T21:26:37.784 INFO:tasks.workunit.client.0.smithi121.stdout:3/207: fdatasync d0/f15 0 2024-04-05T21:26:37.785 INFO:tasks.workunit.client.1.smithi188.stdout:3/583: creat d5/d7/d65/d5f/fc6 x:0 0 0 2024-04-05T21:26:37.786 INFO:tasks.workunit.client.1.smithi188.stdout:7/681: mkdir d3/d4/db/d2e/d8c/db7/dd6 0 2024-04-05T21:26:37.786 INFO:tasks.workunit.client.0.smithi121.stdout:3/208: read d0/d3/de/f17 [866722,99356] 0 2024-04-05T21:26:37.789 INFO:tasks.workunit.client.1.smithi188.stdout:0/393: dwrite d4/d8/db/f76 [0,4194304] 0 2024-04-05T21:26:37.789 INFO:tasks.workunit.client.1.smithi188.stdout:3/584: rename d5/df/f95 to d5/d7e/fc7 0 2024-04-05T21:26:37.790 INFO:tasks.workunit.client.0.smithi121.stdout:8/200: dread d12/d13/f20 [0,4194304] 0 2024-04-05T21:26:37.791 INFO:tasks.workunit.client.0.smithi121.stdout:3/209: symlink d0/d3/l44 0 2024-04-05T21:26:37.792 INFO:tasks.workunit.client.0.smithi121.stdout:1/290: dwrite d8/d10/d14/d1a/d1b/d38/f42 [0,4194304] 0 2024-04-05T21:26:37.793 INFO:tasks.workunit.client.0.smithi121.stdout:8/201: chown d12/d13/d15/d1d/d29/c2f 172 1 2024-04-05T21:26:37.794 INFO:tasks.workunit.client.0.smithi121.stdout:8/202: truncate d12/d13/d15/d1d/f42 789401 0 2024-04-05T21:26:37.796 INFO:tasks.workunit.client.1.smithi188.stdout:0/394: creat d4/d6/d61/f93 x:0 0 0 2024-04-05T21:26:37.797 INFO:tasks.workunit.client.0.smithi121.stdout:3/210: unlink d0/d3/f13 0 2024-04-05T21:26:37.797 INFO:tasks.workunit.client.0.smithi121.stdout:3/211: chown d0/d3/d24/d3f 86 1 2024-04-05T21:26:37.798 INFO:tasks.workunit.client.0.smithi121.stdout:1/291: mknod d8/d10/d14/d1a/c64 0 2024-04-05T21:26:37.799 INFO:tasks.workunit.client.0.smithi121.stdout:1/292: fsync d8/d10/d14/d1a/d1b/d44/f48 0 2024-04-05T21:26:37.799 INFO:tasks.workunit.client.0.smithi121.stdout:8/203: rename d12/d13/d15/d1d/l2d to d12/d13/d15/d1d/l46 0 2024-04-05T21:26:37.801 INFO:tasks.workunit.client.1.smithi188.stdout:2/564: dread d5/db/dd/f11 [0,4194304] 0 2024-04-05T21:26:37.801 INFO:tasks.workunit.client.0.smithi121.stdout:8/204: mknod d12/d30/c47 0 2024-04-05T21:26:37.802 INFO:tasks.workunit.client.1.smithi188.stdout:3/585: getdents d5/d7/d65/d61/d66/da4 0 2024-04-05T21:26:37.804 INFO:tasks.workunit.client.0.smithi121.stdout:8/205: mknod d12/d13/d15/d1d/c48 0 2024-04-05T21:26:37.805 INFO:tasks.workunit.client.0.smithi121.stdout:8/206: write d12/d13/d15/f18 [423812,89522] 0 2024-04-05T21:26:37.806 INFO:tasks.workunit.client.0.smithi121.stdout:1/293: rename d8/d10/d14/d27/d2d/d3b/f62 to d8/d10/d14/d1a/d1b/f65 0 2024-04-05T21:26:37.806 INFO:tasks.workunit.client.0.smithi121.stdout:8/207: chown d12/d30/c47 47389974 1 2024-04-05T21:26:37.808 INFO:tasks.workunit.client.0.smithi121.stdout:8/208: dread - d12/d13/d15/d1d/f36 zero size 2024-04-05T21:26:37.808 INFO:tasks.workunit.client.0.smithi121.stdout:1/294: write d8/d10/d14/f2c [1258254,63217] 0 2024-04-05T21:26:37.809 INFO:tasks.workunit.client.1.smithi188.stdout:0/395: dwrite d4/d6/d13/d23/d24/f81 [0,4194304] 0 2024-04-05T21:26:37.809 INFO:tasks.workunit.client.0.smithi121.stdout:8/209: chown fa 14386 1 2024-04-05T21:26:37.811 INFO:tasks.workunit.client.0.smithi121.stdout:1/295: chown d8/d10/d14/d27/d2d/l61 131519 1 2024-04-05T21:26:37.813 INFO:tasks.workunit.client.0.smithi121.stdout:8/210: symlink d12/d13/d15/d1d/d29/d3e/l49 0 2024-04-05T21:26:37.815 INFO:tasks.workunit.client.1.smithi188.stdout:0/396: truncate d4/d8/f3d 86225 0 2024-04-05T21:26:37.818 INFO:tasks.workunit.client.0.smithi121.stdout:1/296: getdents d8/d10/d14/d1a/d1b/d38/d4b 0 2024-04-05T21:26:37.818 INFO:tasks.workunit.client.1.smithi188.stdout:2/565: dwrite d5/db/dd/d67/f8e [0,4194304] 0 2024-04-05T21:26:37.819 INFO:tasks.workunit.client.0.smithi121.stdout:1/297: fsync d8/d10/d14/d27/d2d/f41 0 2024-04-05T21:26:37.819 INFO:tasks.workunit.client.0.smithi121.stdout:1/298: fsync d8/d10/d14/d1a/d1b/d44/f48 0 2024-04-05T21:26:37.820 INFO:tasks.workunit.client.0.smithi121.stdout:1/299: dread - d8/d10/d14/d1a/d1b/d24/f5f zero size 2024-04-05T21:26:37.824 INFO:tasks.workunit.client.1.smithi188.stdout:7/682: dwrite d3/d4/db/d2e/d3b/f4d [0,4194304] 0 2024-04-05T21:26:37.824 INFO:tasks.workunit.client.1.smithi188.stdout:0/397: rename d4/f5c to d4/d8/db/d21/d4a/d73/f94 0 2024-04-05T21:26:37.825 INFO:tasks.workunit.client.1.smithi188.stdout:2/566: symlink d5/db/dd/d36/d3c/d63/lb9 0 2024-04-05T21:26:37.827 INFO:tasks.workunit.client.0.smithi121.stdout:1/300: dwrite d8/d10/d14/d1a/d1b/d38/d4b/f54 [0,4194304] 0 2024-04-05T21:26:37.828 INFO:tasks.workunit.client.1.smithi188.stdout:0/398: chown d4/d8/db/d21/f56 4 1 2024-04-05T21:26:37.829 INFO:tasks.workunit.client.1.smithi188.stdout:7/683: mknod d3/d4/db/d2e/d8c/db7/dc4/cd7 0 2024-04-05T21:26:37.830 INFO:tasks.workunit.client.1.smithi188.stdout:6/561: dread d6/d13/d18/f35 [0,4194304] 0 2024-04-05T21:26:37.830 INFO:tasks.workunit.client.0.smithi121.stdout:1/301: creat d8/d10/d14/d1a/f66 x:0 0 0 2024-04-05T21:26:37.831 INFO:tasks.workunit.client.0.smithi121.stdout:1/302: stat d8/d10/d14/f2c 0 2024-04-05T21:26:37.831 INFO:tasks.workunit.client.1.smithi188.stdout:2/567: rename d5/c53 to d5/db/dd/d36/d3c/d48/d7a/d7c/cba 0 2024-04-05T21:26:37.833 INFO:tasks.workunit.client.1.smithi188.stdout:0/399: fdatasync d4/d8/db/d42/d38/d77/f86 0 2024-04-05T21:26:37.838 INFO:tasks.workunit.client.1.smithi188.stdout:6/562: mkdir d6/d13/d18/d19/d2c/d3f/d77/db0 0 2024-04-05T21:26:37.839 INFO:tasks.workunit.client.1.smithi188.stdout:0/400: unlink d4/d8/db/c31 0 2024-04-05T21:26:37.840 INFO:tasks.workunit.client.1.smithi188.stdout:7/684: mkdir d3/d4/db/d2e/dd8 0 2024-04-05T21:26:37.844 INFO:tasks.workunit.client.1.smithi188.stdout:0/401: creat d4/d8/db/d42/d38/d77/f95 x:0 0 0 2024-04-05T21:26:37.845 INFO:tasks.workunit.client.0.smithi121.stdout:7/275: sync 2024-04-05T21:26:37.845 INFO:tasks.workunit.client.0.smithi121.stdout:5/275: sync 2024-04-05T21:26:37.845 INFO:tasks.workunit.client.1.smithi188.stdout:2/568: dwrite d5/db/d35/fab [0,4194304] 0 2024-04-05T21:26:37.848 INFO:tasks.workunit.client.1.smithi188.stdout:0/402: chown d4/d8/db/d42/d71/l7f 6435 1 2024-04-05T21:26:37.849 INFO:tasks.workunit.client.1.smithi188.stdout:6/563: truncate d6/d13/d18/d19/d2c/d3f/d95/faa 157155 0 2024-04-05T21:26:37.849 INFO:tasks.workunit.client.1.smithi188.stdout:7/685: rename d3/d4/db/d2e/d37/f50 to d3/d4/d7c/fd9 0 2024-04-05T21:26:37.850 INFO:tasks.workunit.client.0.smithi121.stdout:7/276: rename d0/d2d to d0/d23/d5b 0 2024-04-05T21:26:37.852 INFO:tasks.workunit.client.0.smithi121.stdout:5/276: dwrite dd/de/d19/d13/f16 [0,4194304] 0 2024-04-05T21:26:37.853 INFO:tasks.workunit.client.0.smithi121.stdout:7/277: unlink d0/d23/d5b/f59 0 2024-04-05T21:26:37.857 INFO:tasks.workunit.client.0.smithi121.stdout:7/278: truncate d0/d9/d22/d28/f31 1006879 0 2024-04-05T21:26:37.861 INFO:tasks.workunit.client.0.smithi121.stdout:5/277: creat dd/de/d19/d1c/f59 x:0 0 0 2024-04-05T21:26:37.861 INFO:tasks.workunit.client.1.smithi188.stdout:6/564: symlink d6/d13/d18/d3c/daf/lb1 0 2024-04-05T21:26:37.863 INFO:tasks.workunit.client.0.smithi121.stdout:5/278: truncate dd/de/d19/d38/d45/f4e 341031 0 2024-04-05T21:26:37.864 INFO:tasks.workunit.client.0.smithi121.stdout:7/279: unlink d0/d9/d22/d28/l2a 0 2024-04-05T21:26:37.866 INFO:tasks.workunit.client.0.smithi121.stdout:8/211: truncate ff 3235628 0 2024-04-05T21:26:37.866 INFO:tasks.workunit.client.0.smithi121.stdout:8/212: chown l7 3 1 2024-04-05T21:26:37.869 INFO:tasks.workunit.client.1.smithi188.stdout:6/565: dwrite d6/d13/f89 [0,4194304] 0 2024-04-05T21:26:37.869 INFO:tasks.workunit.client.0.smithi121.stdout:1/303: getdents d8/d10/d14/d1a 0 2024-04-05T21:26:37.870 INFO:tasks.workunit.client.0.smithi121.stdout:7/280: mknod d0/d57/c5c 0 2024-04-05T21:26:37.871 INFO:tasks.workunit.client.0.smithi121.stdout:1/304: chown d8/d10/d14/d1a/d1b/d4c 216941 1 2024-04-05T21:26:37.871 INFO:tasks.workunit.client.0.smithi121.stdout:8/213: symlink d12/d30/l4a 0 2024-04-05T21:26:37.873 INFO:tasks.workunit.client.0.smithi121.stdout:8/214: chown d12/d13/d15/l16 131331 1 2024-04-05T21:26:37.874 INFO:tasks.workunit.client.1.smithi188.stdout:6/566: mkdir d6/d13/d18/d19/d2c/db2 0 2024-04-05T21:26:37.875 INFO:tasks.workunit.client.0.smithi121.stdout:8/215: write d12/d13/d15/f19 [2115080,40112] 0 2024-04-05T21:26:37.876 INFO:tasks.workunit.client.0.smithi121.stdout:8/216: dread - d12/d13/d15/d1d/d29/f40 zero size 2024-04-05T21:26:37.877 INFO:tasks.workunit.client.0.smithi121.stdout:7/281: link d0/d23/c36 d0/d32/d4a/c5d 0 2024-04-05T21:26:37.877 INFO:tasks.workunit.client.1.smithi188.stdout:6/567: creat d6/d7/fb3 x:0 0 0 2024-04-05T21:26:37.877 INFO:tasks.workunit.client.1.smithi188.stdout:7/686: truncate d3/d4/db/f60 3874303 0 2024-04-05T21:26:37.881 INFO:tasks.workunit.client.1.smithi188.stdout:6/568: creat d6/d13/d18/d19/d2c/d3f/d77/fb4 x:0 0 0 2024-04-05T21:26:37.883 INFO:tasks.workunit.client.0.smithi121.stdout:1/305: link d8/d10/d14/d1a/d1b/l55 d8/d10/d14/d1a/l67 0 2024-04-05T21:26:37.884 INFO:tasks.workunit.client.0.smithi121.stdout:7/282: creat d0/d9/f5e x:0 0 0 2024-04-05T21:26:37.884 INFO:tasks.workunit.client.1.smithi188.stdout:0/403: dwrite d4/d6/f12 [0,4194304] 0 2024-04-05T21:26:37.886 INFO:tasks.workunit.client.0.smithi121.stdout:8/217: creat d12/d13/d15/f4b x:0 0 0 2024-04-05T21:26:37.887 INFO:tasks.workunit.client.1.smithi188.stdout:0/404: chown d4/d6/d13/d23/d24/d57/d63 81 1 2024-04-05T21:26:37.887 INFO:tasks.workunit.client.1.smithi188.stdout:6/569: symlink d6/d13/d18/d19/lb5 0 2024-04-05T21:26:37.889 INFO:tasks.workunit.client.1.smithi188.stdout:0/405: chown d4/d6/d13/c4b 150 1 2024-04-05T21:26:37.890 INFO:tasks.workunit.client.0.smithi121.stdout:8/218: truncate d12/d13/d15/d1d/d29/d3e/f45 318814 0 2024-04-05T21:26:37.890 INFO:tasks.workunit.client.0.smithi121.stdout:8/219: chown d12/d13 34 1 2024-04-05T21:26:37.890 INFO:tasks.workunit.client.1.smithi188.stdout:6/570: mknod d6/d13/d48/d49/d75/cb6 0 2024-04-05T21:26:37.891 INFO:tasks.workunit.client.0.smithi121.stdout:7/283: symlink d0/d23/d5a/l5f 0 2024-04-05T21:26:37.893 INFO:tasks.workunit.client.0.smithi121.stdout:8/220: chown d12/d13/d15/d1d/c3c 102346770 1 2024-04-05T21:26:37.894 INFO:tasks.workunit.client.1.smithi188.stdout:6/571: mkdir d6/d13/d18/d19/db7 0 2024-04-05T21:26:37.895 INFO:tasks.workunit.client.0.smithi121.stdout:8/221: readlink d12/d13/d15/l41 0 2024-04-05T21:26:37.895 INFO:tasks.workunit.client.1.smithi188.stdout:7/687: dwrite d3/f3e [0,4194304] 0 2024-04-05T21:26:37.897 INFO:tasks.workunit.client.0.smithi121.stdout:5/279: truncate dd/de/d19/f51 3668138 0 2024-04-05T21:26:37.897 INFO:tasks.workunit.client.1.smithi188.stdout:6/572: stat d6/d13/d18/d19/d2c/l7d 0 2024-04-05T21:26:37.898 INFO:tasks.workunit.client.0.smithi121.stdout:7/284: unlink d0/d32/d4a/c5d 0 2024-04-05T21:26:37.898 INFO:tasks.workunit.client.0.smithi121.stdout:5/280: chown dd/de/d19/d13/f35 0 1 2024-04-05T21:26:37.899 INFO:tasks.workunit.client.1.smithi188.stdout:3/586: dread d5/dc/f3d [0,4194304] 0 2024-04-05T21:26:37.899 INFO:tasks.workunit.client.1.smithi188.stdout:0/406: rename d4/d8/db/f45 to d4/d6/d13/d23/d24/d57/d63/f96 0 2024-04-05T21:26:37.899 INFO:tasks.workunit.client.1.smithi188.stdout:7/688: truncate d3/d4/db/d2e/d37/fc2 403816 0 2024-04-05T21:26:37.900 INFO:tasks.workunit.client.0.smithi121.stdout:8/222: write d12/d13/d15/f21 [886744,80173] 0 2024-04-05T21:26:37.903 INFO:tasks.workunit.client.0.smithi121.stdout:5/281: write dd/f17 [2549718,66727] 0 2024-04-05T21:26:37.904 INFO:tasks.workunit.client.1.smithi188.stdout:7/689: dread d3/d2c/dae/f97 [0,4194304] 0 2024-04-05T21:26:37.907 INFO:tasks.workunit.client.1.smithi188.stdout:6/573: write d6/d13/d18/d19/d2c/d3f/f56 [5155307,27380] 0 2024-04-05T21:26:37.911 INFO:tasks.workunit.client.0.smithi121.stdout:1/306: truncate d8/d10/d14/f2c 793778 0 2024-04-05T21:26:37.911 INFO:tasks.workunit.client.0.smithi121.stdout:5/282: unlink dd/de/f21 0 2024-04-05T21:26:37.911 INFO:tasks.workunit.client.1.smithi188.stdout:3/587: dwrite d5/dc/f1f [0,4194304] 0 2024-04-05T21:26:37.911 INFO:tasks.workunit.client.1.smithi188.stdout:0/407: dwrite d4/d8/db/f11 [0,4194304] 0 2024-04-05T21:26:37.912 INFO:tasks.workunit.client.0.smithi121.stdout:8/223: creat d12/d13/d15/d1d/f4c x:0 0 0 2024-04-05T21:26:37.916 INFO:tasks.workunit.client.1.smithi188.stdout:6/574: truncate d6/d13/d38/f4e 947871 0 2024-04-05T21:26:37.916 INFO:tasks.workunit.client.0.smithi121.stdout:7/285: getdents d0/d9/d22/d3e 0 2024-04-05T21:26:37.916 INFO:tasks.workunit.client.0.smithi121.stdout:8/224: readlink d12/d30/l4a 0 2024-04-05T21:26:37.919 INFO:tasks.workunit.client.1.smithi188.stdout:7/690: dwrite d3/d4/db/d2e/d8c/db7/fbc [0,4194304] 0 2024-04-05T21:26:37.921 INFO:tasks.workunit.client.1.smithi188.stdout:0/408: symlink d4/d6/d13/d23/d24/d57/l97 0 2024-04-05T21:26:37.922 INFO:tasks.workunit.client.1.smithi188.stdout:6/575: unlink d6/d13/d18/d19/c22 0 2024-04-05T21:26:37.924 INFO:tasks.workunit.client.1.smithi188.stdout:7/691: write d3/d4/db/d2e/f6b [1098686,1723] 0 2024-04-05T21:26:37.924 INFO:tasks.workunit.client.0.smithi121.stdout:7/286: write d0/d23/d5b/f3f [4723002,44639] 0 2024-04-05T21:26:37.924 INFO:tasks.workunit.client.0.smithi121.stdout:1/307: dwrite f2 [0,4194304] 0 2024-04-05T21:26:37.924 INFO:tasks.workunit.client.0.smithi121.stdout:1/308: fdatasync d8/d10/d14/d1a/f66 0 2024-04-05T21:26:37.924 INFO:tasks.workunit.client.0.smithi121.stdout:7/287: mknod d0/d32/d4e/c60 0 2024-04-05T21:26:37.926 INFO:tasks.workunit.client.0.smithi121.stdout:1/309: write d8/f1d [65068,8176] 0 2024-04-05T21:26:37.928 INFO:tasks.workunit.client.0.smithi121.stdout:5/283: dwrite dd/de/d19/f57 [0,4194304] 0 2024-04-05T21:26:37.929 INFO:tasks.workunit.client.1.smithi188.stdout:3/588: dwrite d5/d7/d6d/d75/faa [0,4194304] 0 2024-04-05T21:26:37.930 INFO:tasks.workunit.client.0.smithi121.stdout:1/310: chown d8/d10/d14/d1a/d1b/d24/f49 467433912 1 2024-04-05T21:26:37.932 INFO:tasks.workunit.client.1.smithi188.stdout:6/576: rename d6/d13/d18/d94/fa0 to d6/d13/d48/d49/d75/da4/fb8 0 2024-04-05T21:26:37.937 INFO:tasks.workunit.client.1.smithi188.stdout:7/692: creat d3/d2c/dae/fda x:0 0 0 2024-04-05T21:26:37.938 INFO:tasks.workunit.client.0.smithi121.stdout:5/284: symlink dd/de/d39/d3b/l5a 0 2024-04-05T21:26:37.939 INFO:tasks.workunit.client.0.smithi121.stdout:8/225: rmdir d12/d13/d15 39 2024-04-05T21:26:37.939 INFO:tasks.workunit.client.0.smithi121.stdout:1/311: rmdir d8/d10/d14/d1a/d1b/d4c 39 2024-04-05T21:26:37.941 INFO:tasks.workunit.client.1.smithi188.stdout:0/409: rmdir d4/d8/db/d21/d4a/d73/d79/d91 0 2024-04-05T21:26:37.943 INFO:tasks.workunit.client.0.smithi121.stdout:5/285: creat dd/de/d39/d3b/d50/f5b x:0 0 0 2024-04-05T21:26:37.944 INFO:tasks.workunit.client.0.smithi121.stdout:1/312: mkdir d8/d10/d14/d27/d2d/d3b/d53/d68 0 2024-04-05T21:26:37.944 INFO:tasks.workunit.client.1.smithi188.stdout:7/693: chown d3/d4/d1d/d74 2 1 2024-04-05T21:26:37.945 INFO:tasks.workunit.client.1.smithi188.stdout:0/410: dread d4/d8/f3d [0,4194304] 0 2024-04-05T21:26:37.946 INFO:tasks.workunit.client.1.smithi188.stdout:0/411: stat d4 0 2024-04-05T21:26:37.949 INFO:tasks.workunit.client.0.smithi121.stdout:8/226: symlink d12/d13/d15/d1d/l4d 0 2024-04-05T21:26:37.950 INFO:tasks.workunit.client.0.smithi121.stdout:5/286: creat dd/de/d19/d38/d45/f5c x:0 0 0 2024-04-05T21:26:37.950 INFO:tasks.workunit.client.1.smithi188.stdout:3/589: rename d5/d7/d31/lb0 to d5/lc8 0 2024-04-05T21:26:37.950 INFO:tasks.workunit.client.1.smithi188.stdout:7/694: symlink d3/d4/db/d2e/d37/ldb 0 2024-04-05T21:26:37.952 INFO:tasks.workunit.client.1.smithi188.stdout:3/590: creat d5/d35/d4e/fc9 x:0 0 0 2024-04-05T21:26:37.952 INFO:tasks.workunit.client.0.smithi121.stdout:5/287: rename f4 to dd/de/d19/d13/f5d 0 2024-04-05T21:26:37.953 INFO:tasks.workunit.client.1.smithi188.stdout:0/412: rename d4/d8/f4d to d4/d8/db/d42/d71/f98 0 2024-04-05T21:26:37.954 INFO:tasks.workunit.client.1.smithi188.stdout:3/591: creat d5/d7/d65/d61/fca x:0 0 0 2024-04-05T21:26:37.955 INFO:tasks.workunit.client.0.smithi121.stdout:1/313: dwrite d8/d10/d14/d1a/f66 [0,4194304] 0 2024-04-05T21:26:37.955 INFO:tasks.workunit.client.1.smithi188.stdout:3/592: readlink d5/dc/l3f 0 2024-04-05T21:26:37.956 INFO:tasks.workunit.client.0.smithi121.stdout:8/227: mknod d12/d13/d15/d1d/c4e 0 2024-04-05T21:26:37.956 INFO:tasks.workunit.client.1.smithi188.stdout:7/695: link d3/d4/c52 d3/d4/db/d2e/d49/cdc 0 2024-04-05T21:26:37.957 INFO:tasks.workunit.client.1.smithi188.stdout:3/593: write d5/d7/d65/d61/f7b [832554,124282] 0 2024-04-05T21:26:37.958 INFO:tasks.workunit.client.1.smithi188.stdout:0/413: rename d4/d8/db/d42/d71/f87 to d4/d8/db/d21/d4a/d73/f99 0 2024-04-05T21:26:37.959 INFO:tasks.workunit.client.1.smithi188.stdout:3/594: write d5/df/f33 [2663779,71025] 0 2024-04-05T21:26:37.962 INFO:tasks.workunit.client.0.smithi121.stdout:8/228: getdents d12/d30 0 2024-04-05T21:26:37.964 INFO:tasks.workunit.client.0.smithi121.stdout:5/288: dwrite dd/de/d19/d13/f2d [0,4194304] 0 2024-04-05T21:26:37.964 INFO:tasks.workunit.client.0.smithi121.stdout:8/229: rmdir d12/d13/d15/d1d 39 2024-04-05T21:26:37.966 INFO:tasks.workunit.client.0.smithi121.stdout:5/289: chown dd/de/d19/d38/f4d 25 1 2024-04-05T21:26:37.968 INFO:tasks.workunit.client.0.smithi121.stdout:5/290: dread - dd/de/d19/d38/d45/f5c zero size 2024-04-05T21:26:37.968 INFO:tasks.workunit.client.1.smithi188.stdout:7/696: dwrite d3/d4/db/d10/f95 [0,4194304] 0 2024-04-05T21:26:37.969 INFO:tasks.workunit.client.0.smithi121.stdout:5/291: chown dd/de/d19/f23 36 1 2024-04-05T21:26:37.969 INFO:tasks.workunit.client.0.smithi121.stdout:8/230: write d12/d13/d15/d1d/f4c [821528,22798] 0 2024-04-05T21:26:37.974 INFO:tasks.workunit.client.0.smithi121.stdout:5/292: rename dd/de/f2e to dd/de/d39/d3b/f5e 0 2024-04-05T21:26:37.975 INFO:tasks.workunit.client.1.smithi188.stdout:3/595: dwrite d5/d7/d65/f88 [0,4194304] 0 2024-04-05T21:26:37.977 INFO:tasks.workunit.client.1.smithi188.stdout:0/414: dwrite d4/d8/db/d42/d38/f82 [0,4194304] 0 2024-04-05T21:26:37.977 INFO:tasks.workunit.client.0.smithi121.stdout:8/231: creat d12/d13/d15/d1d/f4f x:0 0 0 2024-04-05T21:26:37.978 INFO:tasks.workunit.client.1.smithi188.stdout:3/596: write d5/d7/d31/f78 [7808,89855] 0 2024-04-05T21:26:37.979 INFO:tasks.workunit.client.1.smithi188.stdout:7/697: symlink d3/ldd 0 2024-04-05T21:26:37.980 INFO:tasks.workunit.client.0.smithi121.stdout:5/293: creat dd/de/d19/d3c/f5f x:0 0 0 2024-04-05T21:26:37.983 INFO:tasks.workunit.client.1.smithi188.stdout:7/698: stat d3/d4/db/d2e/d3b/d94/cb5 0 2024-04-05T21:26:37.983 INFO:tasks.workunit.client.1.smithi188.stdout:7/699: readlink d3/d4/db/d55/d9d/lb2 0 2024-04-05T21:26:37.986 INFO:tasks.workunit.client.0.smithi121.stdout:5/294: mknod dd/de/d39/c60 0 2024-04-05T21:26:37.987 INFO:tasks.workunit.client.0.smithi121.stdout:1/314: write d8/d10/d14/d27/f30 [29182,52094] 0 2024-04-05T21:26:37.987 INFO:tasks.workunit.client.0.smithi121.stdout:5/295: write dd/de/d19/d38/d45/f54 [123615,72327] 0 2024-04-05T21:26:37.988 INFO:tasks.workunit.client.1.smithi188.stdout:3/597: symlink d5/d7/d65/d5f/lcb 0 2024-04-05T21:26:37.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:37 smithi121 ceph-mon[39723]: pgmap v17: 65 pgs: 65 active+clean; 1.3 GiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 34 MiB/s rd, 163 MiB/s wr, 285 op/s 2024-04-05T21:26:37.990 INFO:tasks.workunit.client.0.smithi121.stdout:1/315: chown d8/d10/d14/d1a/d1b/d24/l45 8 1 2024-04-05T21:26:37.990 INFO:tasks.workunit.client.0.smithi121.stdout:5/296: truncate dd/de/d19/d38/f4f 721931 0 2024-04-05T21:26:37.993 INFO:tasks.workunit.client.0.smithi121.stdout:1/316: chown d8/d10/d14/d1a/d1b/d4c/f63 33854195 1 2024-04-05T21:26:37.994 INFO:tasks.workunit.client.1.smithi188.stdout:7/700: creat d3/d4/db/d2e/d8c/db7/dd6/fde x:0 0 0 2024-04-05T21:26:37.994 INFO:tasks.workunit.client.0.smithi121.stdout:5/297: mknod dd/de/d39/d3b/d50/c61 0 2024-04-05T21:26:37.995 INFO:tasks.workunit.client.1.smithi188.stdout:3/598: creat d5/d35/d4e/db6/fcc x:0 0 0 2024-04-05T21:26:37.996 INFO:tasks.workunit.client.0.smithi121.stdout:5/298: symlink dd/de/d19/d1c/d36/l62 0 2024-04-05T21:26:37.997 INFO:tasks.workunit.client.1.smithi188.stdout:6/577: dread d6/d13/d18/d19/f72 [0,4194304] 0 2024-04-05T21:26:37.998 INFO:tasks.workunit.client.0.smithi121.stdout:1/317: creat d8/d10/d14/d1a/d1b/f69 x:0 0 0 2024-04-05T21:26:37.999 INFO:tasks.workunit.client.1.smithi188.stdout:7/701: mknod d3/d4/d1d/d74/d88/cdf 0 2024-04-05T21:26:38.001 INFO:tasks.workunit.client.0.smithi121.stdout:5/299: link dd/c10 dd/de/d19/d38/d45/c63 0 2024-04-05T21:26:38.002 INFO:tasks.workunit.client.0.smithi121.stdout:5/300: chown dd/de/c12 119432 1 2024-04-05T21:26:38.002 INFO:tasks.workunit.client.0.smithi121.stdout:8/232: truncate d12/d13/d15/f21 3699833 0 2024-04-05T21:26:38.004 INFO:tasks.workunit.client.1.smithi188.stdout:0/415: dwrite d4/d8/db/d21/d4a/d73/f99 [0,4194304] 0 2024-04-05T21:26:38.004 INFO:tasks.workunit.client.0.smithi121.stdout:8/233: mkdir d12/d50 0 2024-04-05T21:26:38.005 INFO:tasks.workunit.client.1.smithi188.stdout:3/599: getdents d5/d7/d65/d61 0 2024-04-05T21:26:38.005 INFO:tasks.workunit.client.0.smithi121.stdout:5/301: creat dd/de/d19/d1c/d36/d46/f64 x:0 0 0 2024-04-05T21:26:38.006 INFO:tasks.workunit.client.1.smithi188.stdout:0/416: dread - d4/d6/d61/f8f zero size 2024-04-05T21:26:38.007 INFO:tasks.workunit.client.1.smithi188.stdout:0/417: chown d4/d6/d13/d23/d24 103031022 1 2024-04-05T21:26:38.008 INFO:tasks.workunit.client.0.smithi121.stdout:8/234: creat d12/f51 x:0 0 0 2024-04-05T21:26:38.008 INFO:tasks.workunit.client.1.smithi188.stdout:0/418: fsync d4/d8/db/d21/f32 0 2024-04-05T21:26:38.009 INFO:tasks.workunit.client.1.smithi188.stdout:7/702: dread d3/d2c/d5e/f78 [0,4194304] 0 2024-04-05T21:26:38.010 INFO:tasks.workunit.client.1.smithi188.stdout:0/419: fdatasync d4/d6/d13/d23/d24/f81 0 2024-04-05T21:26:38.012 INFO:tasks.workunit.client.0.smithi121.stdout:5/302: rename dd/de/f11 to dd/f65 0 2024-04-05T21:26:38.012 INFO:tasks.workunit.client.0.smithi121.stdout:5/303: chown dd/de/d19 4969 1 2024-04-05T21:26:38.013 INFO:tasks.workunit.client.1.smithi188.stdout:0/420: creat d4/d8/d39/f9a x:0 0 0 2024-04-05T21:26:38.016 INFO:tasks.workunit.client.1.smithi188.stdout:0/421: dread - d4/d8/db/d42/d38/d77/f95 zero size 2024-04-05T21:26:38.020 INFO:tasks.workunit.client.0.smithi121.stdout:1/318: truncate d8/d10/d14/d27/d57/f60 1032825 0 2024-04-05T21:26:38.022 INFO:tasks.workunit.client.0.smithi121.stdout:8/235: getdents d12 0 2024-04-05T21:26:38.023 INFO:tasks.workunit.client.1.smithi188.stdout:3/600: truncate d5/d2b/fc0 1239914 0 2024-04-05T21:26:38.023 INFO:tasks.workunit.client.1.smithi188.stdout:7/703: write d3/d2c/f4c [575184,34968] 0 2024-04-05T21:26:38.025 INFO:tasks.workunit.client.1.smithi188.stdout:3/601: write d5/d7/d65/d61/d66/fc2 [70925,79046] 0 2024-04-05T21:26:38.025 INFO:tasks.workunit.client.0.smithi121.stdout:8/236: symlink d12/l52 0 2024-04-05T21:26:38.025 INFO:tasks.workunit.client.0.smithi121.stdout:1/319: dread d8/d10/d14/d1a/f46 [0,4194304] 0 2024-04-05T21:26:38.028 INFO:tasks.workunit.client.0.smithi121.stdout:1/320: unlink d8/d10/d14/d27/l47 0 2024-04-05T21:26:38.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:37 smithi188 ceph-mon[49715]: pgmap v17: 65 pgs: 65 active+clean; 1.3 GiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 34 MiB/s rd, 163 MiB/s wr, 285 op/s 2024-04-05T21:26:38.030 INFO:tasks.workunit.client.0.smithi121.stdout:2/235: sync 2024-04-05T21:26:38.030 INFO:tasks.workunit.client.1.smithi188.stdout:6/578: dread d6/d7/f1e [0,4194304] 0 2024-04-05T21:26:38.031 INFO:tasks.workunit.client.1.smithi188.stdout:6/579: write d6/d13/d38/f6f [2215535,98854] 0 2024-04-05T21:26:38.033 INFO:tasks.workunit.client.1.smithi188.stdout:6/580: rmdir d6/d13/d18/d19/d2c 39 2024-04-05T21:26:38.034 INFO:tasks.workunit.client.1.smithi188.stdout:0/422: dread d4/d6/d13/d23/d24/d57/d63/f96 [0,4194304] 0 2024-04-05T21:26:38.036 INFO:tasks.workunit.client.1.smithi188.stdout:6/581: chown d6/d13/d18/c1a 2531 1 2024-04-05T21:26:38.036 INFO:tasks.workunit.client.0.smithi121.stdout:2/236: dwrite d2/d9/d17/f18 [4194304,4194304] 0 2024-04-05T21:26:38.038 INFO:tasks.workunit.client.1.smithi188.stdout:6/582: write d6/d13/d18/d19/d2c/f51 [282902,103327] 0 2024-04-05T21:26:38.039 INFO:tasks.workunit.client.1.smithi188.stdout:0/423: dread d4/d8/d10/f20 [0,4194304] 0 2024-04-05T21:26:38.040 INFO:tasks.workunit.client.0.smithi121.stdout:5/304: dread dd/f65 [0,4194304] 0 2024-04-05T21:26:38.040 INFO:tasks.workunit.client.1.smithi188.stdout:6/583: fdatasync d6/fa 0 2024-04-05T21:26:38.043 INFO:tasks.workunit.client.0.smithi121.stdout:2/237: rmdir d2/d9 39 2024-04-05T21:26:38.045 INFO:tasks.workunit.client.1.smithi188.stdout:6/584: truncate d6/d7/fd 4428600 0 2024-04-05T21:26:38.045 INFO:tasks.workunit.client.1.smithi188.stdout:6/585: chown d6/d7/c10 47 1 2024-04-05T21:26:38.048 INFO:tasks.workunit.client.0.smithi121.stdout:1/321: write d8/d10/d14/d1a/d1b/d24/f2a [165851,100032] 0 2024-04-05T21:26:38.050 INFO:tasks.workunit.client.0.smithi121.stdout:1/322: chown d8/d10/d14/d1a/d1b/d4c/f63 5545 1 2024-04-05T21:26:38.057 INFO:tasks.workunit.client.1.smithi188.stdout:0/424: truncate d4/d8/db/f76 398644 0 2024-04-05T21:26:38.058 INFO:tasks.workunit.client.0.smithi121.stdout:5/305: truncate dd/de/d19/d13/f2d 4496861 0 2024-04-05T21:26:38.061 INFO:tasks.workunit.client.1.smithi188.stdout:0/425: creat d4/d8/db/d42/d71/d80/f9b x:0 0 0 2024-04-05T21:26:38.062 INFO:tasks.workunit.client.0.smithi121.stdout:5/306: rename dd/de/d19/d13/l24 to dd/de/d19/d38/d45/l66 0 2024-04-05T21:26:38.062 INFO:tasks.workunit.client.1.smithi188.stdout:0/426: chown d4/d8/db/f11 1 1 2024-04-05T21:26:38.063 INFO:tasks.workunit.client.0.smithi121.stdout:5/307: fsync dd/f17 0 2024-04-05T21:26:38.063 INFO:tasks.workunit.client.1.smithi188.stdout:6/586: dwrite d6/d7/fc [0,4194304] 0 2024-04-05T21:26:38.063 INFO:tasks.workunit.client.0.smithi121.stdout:5/308: chown dd/de/d19/f23 180 1 2024-04-05T21:26:38.065 INFO:tasks.workunit.client.0.smithi121.stdout:5/309: readlink dd/l2c 0 2024-04-05T21:26:38.066 INFO:tasks.workunit.client.0.smithi121.stdout:5/310: write dd/de/d19/d13/f35 [1444415,44004] 0 2024-04-05T21:26:38.067 INFO:tasks.workunit.client.1.smithi188.stdout:6/587: unlink d6/d13/d38/f47 0 2024-04-05T21:26:38.070 INFO:tasks.workunit.client.0.smithi121.stdout:5/311: creat dd/de/d39/d3b/d50/f67 x:0 0 0 2024-04-05T21:26:38.071 INFO:tasks.workunit.client.0.smithi121.stdout:1/323: dread d8/d10/d14/f1e [0,4194304] 0 2024-04-05T21:26:38.072 INFO:tasks.workunit.client.0.smithi121.stdout:1/324: chown d8/d10/d14/l25 836899 1 2024-04-05T21:26:38.073 INFO:tasks.workunit.client.0.smithi121.stdout:1/325: rename d8/d10 to d8/d10/d6a 22 2024-04-05T21:26:38.076 INFO:tasks.workunit.client.0.smithi121.stdout:1/326: write d8/d10/d14/d1a/d1b/d24/f2a [1930152,108262] 0 2024-04-05T21:26:38.077 INFO:tasks.workunit.client.0.smithi121.stdout:8/237: dread d12/d13/d15/f21 [0,4194304] 0 2024-04-05T21:26:38.080 INFO:tasks.workunit.client.0.smithi121.stdout:8/238: creat d12/d13/d15/d1d/f53 x:0 0 0 2024-04-05T21:26:38.080 INFO:tasks.workunit.client.0.smithi121.stdout:1/327: dread d8/d10/d14/d27/d2d/d3b/f3d [0,4194304] 0 2024-04-05T21:26:38.083 INFO:tasks.workunit.client.0.smithi121.stdout:1/328: rename d8/ff to d8/d10/d14/d1a/d1b/d38/d4b/f6b 0 2024-04-05T21:26:38.086 INFO:tasks.workunit.client.0.smithi121.stdout:1/329: read d8/d10/d14/d27/f30 [48257,12250] 0 2024-04-05T21:26:38.087 INFO:tasks.workunit.client.0.smithi121.stdout:8/239: dwrite d12/d13/d15/d1d/f53 [0,4194304] 0 2024-04-05T21:26:38.088 INFO:tasks.workunit.client.0.smithi121.stdout:5/312: truncate dd/de/d19/d38/d45/f4e 178360 0 2024-04-05T21:26:38.091 INFO:tasks.workunit.client.0.smithi121.stdout:8/240: mknod d12/d13/c54 0 2024-04-05T21:26:38.094 INFO:tasks.workunit.client.1.smithi188.stdout:6/588: dread d6/f2e [0,4194304] 0 2024-04-05T21:26:38.095 INFO:tasks.workunit.client.0.smithi121.stdout:1/330: rmdir d8/d10/d14/d27/d2d/d3b/d53/d68 0 2024-04-05T21:26:38.096 INFO:tasks.workunit.client.0.smithi121.stdout:5/313: dwrite dd/de/d19/d1c/f33 [0,4194304] 0 2024-04-05T21:26:38.097 INFO:tasks.workunit.client.1.smithi188.stdout:6/589: truncate d6/d7/fd 5119169 0 2024-04-05T21:26:38.098 INFO:tasks.workunit.client.0.smithi121.stdout:8/241: rename l5 to d12/d13/d15/l55 0 2024-04-05T21:26:38.098 INFO:tasks.workunit.client.0.smithi121.stdout:5/314: dread - dd/de/d19/d1c/d36/f3a zero size 2024-04-05T21:26:38.099 INFO:tasks.workunit.client.1.smithi188.stdout:6/590: mkdir d6/d13/db9 0 2024-04-05T21:26:38.100 INFO:tasks.workunit.client.0.smithi121.stdout:1/331: fdatasync d8/d10/d14/d1a/d1b/d24/f49 0 2024-04-05T21:26:38.100 INFO:tasks.workunit.client.0.smithi121.stdout:5/315: write dd/de/f34 [576325,54075] 0 2024-04-05T21:26:38.102 INFO:tasks.workunit.client.0.smithi121.stdout:8/242: mkdir d12/d13/d56 0 2024-04-05T21:26:38.103 INFO:tasks.workunit.client.0.smithi121.stdout:5/316: rmdir dd/de/d39 39 2024-04-05T21:26:38.105 INFO:tasks.workunit.client.0.smithi121.stdout:1/332: creat d8/d10/d14/d1a/d1b/d38/d4b/f6c x:0 0 0 2024-04-05T21:26:38.106 INFO:tasks.workunit.client.0.smithi121.stdout:1/333: stat d8/d10/d14/d27/d57 0 2024-04-05T21:26:38.108 INFO:tasks.workunit.client.0.smithi121.stdout:5/317: mkdir dd/de/d19/d38/d45/d68 0 2024-04-05T21:26:38.110 INFO:tasks.workunit.client.0.smithi121.stdout:8/243: mkdir d12/d13/d56/d57 0 2024-04-05T21:26:38.113 INFO:tasks.workunit.client.0.smithi121.stdout:8/244: creat d12/d13/d15/d1d/f58 x:0 0 0 2024-04-05T21:26:38.116 INFO:tasks.workunit.client.1.smithi188.stdout:6/591: dwrite d6/d13/d18/d19/d2c/d3f/d77/fb4 [0,4194304] 0 2024-04-05T21:26:38.117 INFO:tasks.workunit.client.0.smithi121.stdout:8/245: chown d12/d30/l4a 43 1 2024-04-05T21:26:38.117 INFO:tasks.workunit.client.1.smithi188.stdout:6/592: chown d6/d13/d38/f55 376065 1 2024-04-05T21:26:38.118 INFO:tasks.workunit.client.0.smithi121.stdout:1/334: link d8/d10/d14/d27/f30 d8/d10/d14/d1a/d1b/d38/d4b/f6d 0 2024-04-05T21:26:38.120 INFO:tasks.workunit.client.1.smithi188.stdout:6/593: rmdir d6/d13/d48 39 2024-04-05T21:26:38.121 INFO:tasks.workunit.client.0.smithi121.stdout:5/318: write dd/de/d19/d13/f2d [557159,83312] 0 2024-04-05T21:26:38.121 INFO:tasks.workunit.client.0.smithi121.stdout:8/246: dread d12/d13/f20 [0,4194304] 0 2024-04-05T21:26:38.126 INFO:tasks.workunit.client.0.smithi121.stdout:5/319: symlink dd/de/d19/d1c/d36/d46/l69 0 2024-04-05T21:26:38.128 INFO:tasks.workunit.client.0.smithi121.stdout:8/247: rename d12/d13/d15/d1d/d29/d3e to d12/d50/d59 0 2024-04-05T21:26:38.129 INFO:tasks.workunit.client.0.smithi121.stdout:8/248: truncate f3 4261071 0 2024-04-05T21:26:38.130 INFO:tasks.workunit.client.0.smithi121.stdout:1/335: truncate d8/d10/d14/d1a/d1b/d38/f42 167611 0 2024-04-05T21:26:38.132 INFO:tasks.workunit.client.0.smithi121.stdout:8/249: unlink d12/d13/d15/l41 0 2024-04-05T21:26:38.132 INFO:tasks.workunit.client.0.smithi121.stdout:5/320: mknod dd/de/d19/d13/d53/c6a 0 2024-04-05T21:26:38.133 INFO:tasks.workunit.client.0.smithi121.stdout:1/336: chown d8/d10/d14/d27/d57/f60 5537 1 2024-04-05T21:26:38.136 INFO:tasks.workunit.client.0.smithi121.stdout:1/337: write d8/d10/f17 [6824347,23140] 0 2024-04-05T21:26:38.136 INFO:tasks.workunit.client.0.smithi121.stdout:5/321: creat dd/de/d19/f6b x:0 0 0 2024-04-05T21:26:38.137 INFO:tasks.workunit.client.0.smithi121.stdout:5/322: chown dd 4516330 1 2024-04-05T21:26:38.137 INFO:tasks.workunit.client.1.smithi188.stdout:8/494: sync 2024-04-05T21:26:38.137 INFO:tasks.workunit.client.1.smithi188.stdout:4/746: sync 2024-04-05T21:26:38.140 INFO:tasks.workunit.client.1.smithi188.stdout:8/495: chown d5/d21/c23 20 1 2024-04-05T21:26:38.140 INFO:tasks.workunit.client.0.smithi121.stdout:5/323: truncate dd/de/d19/d13/f25 1584043 0 2024-04-05T21:26:38.140 INFO:tasks.workunit.client.1.smithi188.stdout:4/747: symlink d0/d2/d17/d48/d70/l107 0 2024-04-05T21:26:38.142 INFO:tasks.workunit.client.1.smithi188.stdout:4/748: fdatasync d0/f5 0 2024-04-05T21:26:38.144 INFO:tasks.workunit.client.0.smithi121.stdout:1/338: rename d8/d10/d14/d1a/l67 to d8/d10/d14/d1a/l6e 0 2024-04-05T21:26:38.144 INFO:tasks.workunit.client.0.smithi121.stdout:8/250: write d12/d13/d15/f1f [2933796,82272] 0 2024-04-05T21:26:38.145 INFO:tasks.workunit.client.1.smithi188.stdout:6/594: dread d6/d13/d18/d19/d2c/f51 [0,4194304] 0 2024-04-05T21:26:38.147 INFO:tasks.workunit.client.1.smithi188.stdout:4/749: creat d0/d2/d52/dc1/f108 x:0 0 0 2024-04-05T21:26:38.147 INFO:tasks.workunit.client.0.smithi121.stdout:1/339: dread - d8/d10/d14/d1a/d1b/d38/d4b/f6c zero size 2024-04-05T21:26:38.148 INFO:tasks.workunit.client.0.smithi121.stdout:5/324: dwrite dd/de/d19/f57 [0,4194304] 0 2024-04-05T21:26:38.148 INFO:tasks.workunit.client.0.smithi121.stdout:8/251: creat d12/d13/d56/d57/f5a x:0 0 0 2024-04-05T21:26:38.149 INFO:tasks.workunit.client.1.smithi188.stdout:6/595: fsync d6/f2f 0 2024-04-05T21:26:38.150 INFO:tasks.workunit.client.1.smithi188.stdout:8/496: dwrite d5/d21/d40/d43/d44/d5b/d99/da3/fa6 [0,4194304] 0 2024-04-05T21:26:38.154 INFO:tasks.workunit.client.0.smithi121.stdout:8/252: rmdir d12/d13/d15/d1d 39 2024-04-05T21:26:38.154 INFO:tasks.workunit.client.0.smithi121.stdout:1/340: chown d8/d10/d14/d1a/d1b/c28 124044569 1 2024-04-05T21:26:38.154 INFO:tasks.workunit.client.0.smithi121.stdout:5/325: rmdir dd/de/d19/d13 39 2024-04-05T21:26:38.154 INFO:tasks.workunit.client.1.smithi188.stdout:6/596: write d6/d13/d18/d19/d2c/f65 [1239318,10990] 0 2024-04-05T21:26:38.155 INFO:tasks.workunit.client.0.smithi121.stdout:8/253: chown d12/d13/d15/d1d/f58 498555 1 2024-04-05T21:26:38.156 INFO:tasks.workunit.client.0.smithi121.stdout:5/326: dread - dd/de/d19/d1c/d36/d46/f64 zero size 2024-04-05T21:26:38.156 INFO:tasks.workunit.client.0.smithi121.stdout:8/254: write d12/d30/f3a [53558,26167] 0 2024-04-05T21:26:38.158 INFO:tasks.workunit.client.0.smithi121.stdout:8/255: write d12/d13/d15/f4b [677733,106638] 0 2024-04-05T21:26:38.159 INFO:tasks.workunit.client.1.smithi188.stdout:8/497: unlink d5/dc/f78 0 2024-04-05T21:26:38.159 INFO:tasks.workunit.client.0.smithi121.stdout:8/256: dread - d12/d13/d15/d1d/f58 zero size 2024-04-05T21:26:38.159 INFO:tasks.workunit.client.1.smithi188.stdout:4/750: dwrite d0/d14/d15/d78/db1/f7c [4194304,4194304] 0 2024-04-05T21:26:38.160 INFO:tasks.workunit.client.0.smithi121.stdout:8/257: chown d12/c39 86 1 2024-04-05T21:26:38.161 INFO:tasks.workunit.client.1.smithi188.stdout:6/597: fsync d6/d13/d18/d19/d2c/d3f/f53 0 2024-04-05T21:26:38.162 INFO:tasks.workunit.client.1.smithi188.stdout:8/498: dread - d5/d21/d40/d43/f51 zero size 2024-04-05T21:26:38.163 INFO:tasks.workunit.client.0.smithi121.stdout:5/327: rename dd/de/d19/d13/f25 to dd/f6c 0 2024-04-05T21:26:38.164 INFO:tasks.workunit.client.0.smithi121.stdout:8/258: symlink d12/l5b 0 2024-04-05T21:26:38.166 INFO:tasks.workunit.client.1.smithi188.stdout:6/598: dread d6/d28/f84 [0,4194304] 0 2024-04-05T21:26:38.166 INFO:tasks.workunit.client.0.smithi121.stdout:5/328: mkdir dd/de/d6d 0 2024-04-05T21:26:38.167 INFO:tasks.workunit.client.0.smithi121.stdout:8/259: mknod d12/d13/d15/d1d/c5c 0 2024-04-05T21:26:38.168 INFO:tasks.workunit.client.0.smithi121.stdout:5/329: chown dd/de/d39 140135569 1 2024-04-05T21:26:38.169 INFO:tasks.workunit.client.0.smithi121.stdout:8/260: fsync d12/d30/f3a 0 2024-04-05T21:26:38.170 INFO:tasks.workunit.client.1.smithi188.stdout:6/599: rename d6/d28/d45/c91 to d6/d13/d18/d19/d2c/d3f/d95/cba 0 2024-04-05T21:26:38.172 INFO:tasks.workunit.client.0.smithi121.stdout:5/330: read f6 [1920247,53453] 0 2024-04-05T21:26:38.172 INFO:tasks.workunit.client.1.smithi188.stdout:6/600: truncate d6/d13/d18/d19/d2c/d34/fac 166831 0 2024-04-05T21:26:38.172 INFO:tasks.workunit.client.1.smithi188.stdout:6/601: chown d6/d13/d18/d19/f72 708643 1 2024-04-05T21:26:38.174 INFO:tasks.workunit.client.0.smithi121.stdout:5/331: creat dd/de/d6d/f6e x:0 0 0 2024-04-05T21:26:38.174 INFO:tasks.workunit.client.0.smithi121.stdout:5/332: readlink dd/de/l4b 0 2024-04-05T21:26:38.176 INFO:tasks.workunit.client.0.smithi121.stdout:5/333: read dd/de/d19/d13/f3f [94281,129932] 0 2024-04-05T21:26:38.177 INFO:tasks.workunit.client.0.smithi121.stdout:1/341: truncate d8/d10/f17 4117541 0 2024-04-05T21:26:38.177 INFO:tasks.workunit.client.0.smithi121.stdout:5/334: write dd/de/d19/d38/d45/f5c [216517,37429] 0 2024-04-05T21:26:38.178 INFO:tasks.workunit.client.1.smithi188.stdout:6/602: read d6/d13/d18/d19/f27 [2593698,22295] 0 2024-04-05T21:26:38.178 INFO:tasks.workunit.client.0.smithi121.stdout:1/342: chown d8/d10/d14/d1a/d1b 127 1 2024-04-05T21:26:38.178 INFO:tasks.workunit.client.1.smithi188.stdout:6/603: readlink d6/d13/d18/d19/d61/l9d 0 2024-04-05T21:26:38.179 INFO:tasks.workunit.client.1.smithi188.stdout:8/499: write d5/d21/d40/d43/f51 [636570,53375] 0 2024-04-05T21:26:38.180 INFO:tasks.workunit.client.0.smithi121.stdout:5/335: write dd/de/d39/d3b/f4a [322892,8532] 0 2024-04-05T21:26:38.183 INFO:tasks.workunit.client.1.smithi188.stdout:8/500: creat d5/d21/d40/d43/d44/d5b/d99/fac x:0 0 0 2024-04-05T21:26:38.183 INFO:tasks.workunit.client.0.smithi121.stdout:1/343: getdents d8/d10/d14/d1a/d1b/d38/d4b 0 2024-04-05T21:26:38.183 INFO:tasks.workunit.client.1.smithi188.stdout:6/604: truncate d6/d13/d18/d19/d61/f8b 824804 0 2024-04-05T21:26:38.184 INFO:tasks.workunit.client.1.smithi188.stdout:8/501: fsync d5/d21/d40/d43/f51 0 2024-04-05T21:26:38.185 INFO:tasks.workunit.client.0.smithi121.stdout:1/344: mknod d8/d10/d14/d27/d2d/c6f 0 2024-04-05T21:26:38.186 INFO:tasks.workunit.client.1.smithi188.stdout:4/751: dread d0/d2/d4f/f67 [0,4194304] 0 2024-04-05T21:26:38.186 INFO:tasks.workunit.client.0.smithi121.stdout:1/345: stat d8/d10/d14/d1a/d1b/d24/f49 0 2024-04-05T21:26:38.187 INFO:tasks.workunit.client.1.smithi188.stdout:6/605: rename d6/d7/l99 to d6/d13/d18/d3c/lbb 0 2024-04-05T21:26:38.188 INFO:tasks.workunit.client.0.smithi121.stdout:1/346: mknod d8/d10/d14/d1a/c70 0 2024-04-05T21:26:38.190 INFO:tasks.workunit.client.1.smithi188.stdout:4/752: creat d0/d2/dc7/f109 x:0 0 0 2024-04-05T21:26:38.190 INFO:tasks.workunit.client.0.smithi121.stdout:1/347: dread d8/d10/d14/d27/d2d/d3b/f3d [0,4194304] 0 2024-04-05T21:26:38.192 INFO:tasks.workunit.client.0.smithi121.stdout:5/336: write dd/de/d19/f51 [1263107,98816] 0 2024-04-05T21:26:38.194 INFO:tasks.workunit.client.1.smithi188.stdout:8/502: dwrite d5/d21/d40/d43/f62 [0,4194304] 0 2024-04-05T21:26:38.195 INFO:tasks.workunit.client.1.smithi188.stdout:4/753: getdents d0/d14/d15/d53 0 2024-04-05T21:26:38.196 INFO:tasks.workunit.client.1.smithi188.stdout:4/754: readlink d0/d14/d15/d1b/lf8 0 2024-04-05T21:26:38.198 INFO:tasks.workunit.client.0.smithi121.stdout:1/348: dwrite d8/d10/d14/d1a/d1b/f69 [0,4194304] 0 2024-04-05T21:26:38.200 INFO:tasks.workunit.client.0.smithi121.stdout:5/337: dread dd/de/d19/d13/f2d [0,4194304] 0 2024-04-05T21:26:38.201 INFO:tasks.workunit.client.1.smithi188.stdout:6/606: write d6/d7/f14 [243323,81925] 0 2024-04-05T21:26:38.203 INFO:tasks.workunit.client.0.smithi121.stdout:5/338: write dd/de/d19/d13/f26 [1356839,15981] 0 2024-04-05T21:26:38.203 INFO:tasks.workunit.client.1.smithi188.stdout:6/607: rmdir d6 39 2024-04-05T21:26:38.204 INFO:tasks.workunit.client.0.smithi121.stdout:1/349: creat d8/d10/d14/d1a/f71 x:0 0 0 2024-04-05T21:26:38.206 INFO:tasks.workunit.client.0.smithi121.stdout:5/339: symlink dd/l6f 0 2024-04-05T21:26:38.209 INFO:tasks.workunit.client.0.smithi121.stdout:1/350: getdents d8/d10/d14/d1a/d1b 0 2024-04-05T21:26:38.211 INFO:tasks.workunit.client.0.smithi121.stdout:1/351: creat d8/d10/d14/d1a/f72 x:0 0 0 2024-04-05T21:26:38.211 INFO:tasks.workunit.client.1.smithi188.stdout:6/608: getdents d6/d13/d18/d19/db7 0 2024-04-05T21:26:38.212 INFO:tasks.workunit.client.1.smithi188.stdout:8/503: dwrite d5/d21/d2a/f31 [4194304,4194304] 0 2024-04-05T21:26:38.213 INFO:tasks.workunit.client.0.smithi121.stdout:1/352: symlink d8/d10/d14/d1a/d1b/l73 0 2024-04-05T21:26:38.214 INFO:tasks.workunit.client.0.smithi121.stdout:5/340: dread dd/f6c [0,4194304] 0 2024-04-05T21:26:38.215 INFO:tasks.workunit.client.0.smithi121.stdout:5/341: write dd/de/d39/d3b/d50/f67 [789253,8675] 0 2024-04-05T21:26:38.216 INFO:tasks.workunit.client.1.smithi188.stdout:8/504: stat d5/d21/d80 0 2024-04-05T21:26:38.218 INFO:tasks.workunit.client.0.smithi121.stdout:5/342: getdents dd/de/d19/d38/d56 0 2024-04-05T21:26:38.220 INFO:tasks.workunit.client.0.smithi121.stdout:5/343: chown dd/de/d39/l43 319 1 2024-04-05T21:26:38.221 INFO:tasks.workunit.client.1.smithi188.stdout:4/755: dread d0/d14/d15/d53/d82/d8f/f93 [0,4194304] 0 2024-04-05T21:26:38.221 INFO:tasks.workunit.client.1.smithi188.stdout:8/505: unlink d5/d21/d40/d43/d44/l45 0 2024-04-05T21:26:38.221 INFO:tasks.workunit.client.0.smithi121.stdout:5/344: creat dd/de/d6d/f70 x:0 0 0 2024-04-05T21:26:38.222 INFO:tasks.workunit.client.1.smithi188.stdout:8/506: mkdir d5/d6e/dad 0 2024-04-05T21:26:38.224 INFO:tasks.workunit.client.1.smithi188.stdout:4/756: mknod d0/d2/d4f/c10a 0 2024-04-05T21:26:38.224 INFO:tasks.workunit.client.0.smithi121.stdout:5/345: creat dd/de/d19/d1c/d36/f71 x:0 0 0 2024-04-05T21:26:38.226 INFO:tasks.workunit.client.1.smithi188.stdout:8/507: getdents d5/d21 0 2024-04-05T21:26:38.226 INFO:tasks.workunit.client.0.smithi121.stdout:5/346: symlink dd/de/d19/d1c/l72 0 2024-04-05T21:26:38.227 INFO:tasks.workunit.client.0.smithi121.stdout:5/347: stat dd/de/d19/d1c/d36/d46 0 2024-04-05T21:26:38.228 INFO:tasks.workunit.client.0.smithi121.stdout:5/348: truncate dd/de/d6d/f70 563425 0 2024-04-05T21:26:38.228 INFO:tasks.workunit.client.1.smithi188.stdout:8/508: mkdir d5/d21/d40/d43/d44/d5b/d99/d69/d70/dae 0 2024-04-05T21:26:38.230 INFO:tasks.workunit.client.1.smithi188.stdout:8/509: symlink d5/laf 0 2024-04-05T21:26:38.231 INFO:tasks.workunit.client.1.smithi188.stdout:8/510: rmdir d5/d21/d2a/d71 39 2024-04-05T21:26:38.232 INFO:tasks.workunit.client.1.smithi188.stdout:8/511: stat d5/dc/f11 0 2024-04-05T21:26:38.232 INFO:tasks.workunit.client.0.smithi121.stdout:5/349: link dd/de/d19/d1c/d36/d46/f64 dd/de/d19/d38/d56/f73 0 2024-04-05T21:26:38.233 INFO:tasks.workunit.client.0.smithi121.stdout:5/350: mkdir dd/de/d19/d1c/d74 0 2024-04-05T21:26:38.234 INFO:tasks.workunit.client.0.smithi121.stdout:5/351: dread - dd/de/d19/d1c/d36/f3a zero size 2024-04-05T21:26:38.236 INFO:tasks.workunit.client.1.smithi188.stdout:6/609: dread f5 [0,4194304] 0 2024-04-05T21:26:38.237 INFO:tasks.workunit.client.0.smithi121.stdout:4/328: sync 2024-04-05T21:26:38.237 INFO:tasks.workunit.client.0.smithi121.stdout:9/211: sync 2024-04-05T21:26:38.237 INFO:tasks.workunit.client.0.smithi121.stdout:0/356: sync 2024-04-05T21:26:38.237 INFO:tasks.workunit.client.0.smithi121.stdout:6/253: sync 2024-04-05T21:26:38.237 INFO:tasks.workunit.client.0.smithi121.stdout:7/288: sync 2024-04-05T21:26:38.237 INFO:tasks.workunit.client.0.smithi121.stdout:3/212: sync 2024-04-05T21:26:38.238 INFO:tasks.workunit.client.1.smithi188.stdout:6/610: write d6/d7/f14 [1525212,17169] 0 2024-04-05T21:26:38.239 INFO:tasks.workunit.client.0.smithi121.stdout:7/289: rmdir d0/d57 39 2024-04-05T21:26:38.239 INFO:tasks.workunit.client.1.smithi188.stdout:8/512: rmdir d5/d21/d40/d43/d44/d5b/d99 39 2024-04-05T21:26:38.242 INFO:tasks.workunit.client.0.smithi121.stdout:4/329: dread d5/f55 [0,4194304] 0 2024-04-05T21:26:38.243 INFO:tasks.workunit.client.1.smithi188.stdout:6/611: dread d6/d13/d48/d49/f80 [0,4194304] 0 2024-04-05T21:26:38.243 INFO:tasks.workunit.client.1.smithi188.stdout:8/513: chown d5/d21/d40/d43/d44/d5b/d99/da3/fa5 181072 1 2024-04-05T21:26:38.245 INFO:tasks.workunit.client.1.smithi188.stdout:4/757: dwrite d0/d14/d15/d53/d82/d8f/d95/fa4 [4194304,4194304] 0 2024-04-05T21:26:38.247 INFO:tasks.workunit.client.1.smithi188.stdout:4/758: chown d0/d14/d2f/f98 3 1 2024-04-05T21:26:38.248 INFO:tasks.workunit.client.1.smithi188.stdout:6/612: creat d6/d13/d48/d49/d60/fbc x:0 0 0 2024-04-05T21:26:38.248 INFO:tasks.workunit.client.1.smithi188.stdout:4/759: chown d0/d14/d15/d53/d82/d8f/d95 219080 1 2024-04-05T21:26:38.250 INFO:tasks.workunit.client.1.smithi188.stdout:8/514: dread d5/f35 [0,4194304] 0 2024-04-05T21:26:38.251 INFO:tasks.workunit.client.1.smithi188.stdout:6/613: creat d6/fbd x:0 0 0 2024-04-05T21:26:38.258 INFO:tasks.workunit.client.1.smithi188.stdout:8/515: dwrite d5/d21/d40/f73 [0,4194304] 0 2024-04-05T21:26:38.262 INFO:tasks.workunit.client.1.smithi188.stdout:8/516: rename d5/dc/f17 to d5/d6e/d86/fb0 0 2024-04-05T21:26:38.268 INFO:tasks.workunit.client.1.smithi188.stdout:8/517: dwrite d5/d21/d40/d43/d44/d5b/d99/f7d [0,4194304] 0 2024-04-05T21:26:38.284 INFO:tasks.workunit.client.0.smithi121.stdout:6/254: truncate d1/d2/d3/d5/fd 4730421 0 2024-04-05T21:26:38.286 INFO:tasks.workunit.client.0.smithi121.stdout:5/352: dwrite dd/de/d19/d3c/f40 [0,4194304] 0 2024-04-05T21:26:38.288 INFO:tasks.workunit.client.0.smithi121.stdout:0/357: rename d7/d71/d45/d5a/l65 to d7/d71/d45/d5a/l78 0 2024-04-05T21:26:38.288 INFO:tasks.workunit.client.0.smithi121.stdout:6/255: stat d1/d2/d3/d5/d1f/f27 0 2024-04-05T21:26:38.293 INFO:tasks.workunit.client.0.smithi121.stdout:3/213: dwrite d0/d3/f30 [0,4194304] 0 2024-04-05T21:26:38.294 INFO:tasks.workunit.client.0.smithi121.stdout:9/212: dwrite d6/df/f2e [0,4194304] 0 2024-04-05T21:26:38.297 INFO:tasks.workunit.client.0.smithi121.stdout:4/330: getdents d5/d11 0 2024-04-05T21:26:38.300 INFO:tasks.workunit.client.0.smithi121.stdout:6/256: creat d1/d2/d3/f4e x:0 0 0 2024-04-05T21:26:38.303 INFO:tasks.workunit.client.0.smithi121.stdout:4/331: creat d5/d11/d26/d49/f72 x:0 0 0 2024-04-05T21:26:38.303 INFO:tasks.workunit.client.0.smithi121.stdout:0/358: dwrite d7/dc/ff [0,4194304] 0 2024-04-05T21:26:38.304 INFO:tasks.workunit.client.0.smithi121.stdout:9/213: symlink d6/d32/l41 0 2024-04-05T21:26:38.306 INFO:tasks.workunit.client.0.smithi121.stdout:3/214: creat d0/f45 x:0 0 0 2024-04-05T21:26:38.307 INFO:tasks.workunit.client.0.smithi121.stdout:4/332: stat d5/d11/c1a 0 2024-04-05T21:26:38.308 INFO:tasks.workunit.client.0.smithi121.stdout:6/257: dread d1/d2/d3/d5/d1f/d2c/d41/f49 [0,4194304] 0 2024-04-05T21:26:38.310 INFO:tasks.workunit.client.0.smithi121.stdout:9/214: mknod d6/d32/c42 0 2024-04-05T21:26:38.311 INFO:tasks.workunit.client.1.smithi188.stdout:1/564: sync 2024-04-05T21:26:38.311 INFO:tasks.workunit.client.1.smithi188.stdout:2/569: sync 2024-04-05T21:26:38.311 INFO:tasks.workunit.client.1.smithi188.stdout:5/493: sync 2024-04-05T21:26:38.311 INFO:tasks.workunit.client.1.smithi188.stdout:9/494: sync 2024-04-05T21:26:38.311 INFO:tasks.workunit.client.0.smithi121.stdout:6/258: dread d1/d2/d1b/d1e/f2a [0,4194304] 0 2024-04-05T21:26:38.313 INFO:tasks.workunit.client.0.smithi121.stdout:0/359: read d7/f3a [137940,20733] 0 2024-04-05T21:26:38.322 INFO:tasks.workunit.client.0.smithi121.stdout:6/259: mkdir d1/d2/d3/d5/d1f/d2c/d4f 0 2024-04-05T21:26:38.324 INFO:tasks.workunit.client.1.smithi188.stdout:2/570: fsync d5/db/dd/f95 0 2024-04-05T21:26:38.325 INFO:tasks.workunit.client.0.smithi121.stdout:0/360: creat d7/d71/d29/f79 x:0 0 0 2024-04-05T21:26:38.326 INFO:tasks.workunit.client.0.smithi121.stdout:3/215: dwrite d0/f14 [0,4194304] 0 2024-04-05T21:26:38.326 INFO:tasks.workunit.client.0.smithi121.stdout:4/333: dwrite d5/d11/d28/f1b [0,4194304] 0 2024-04-05T21:26:38.329 INFO:tasks.workunit.client.0.smithi121.stdout:6/260: mknod d1/d2/d1b/d2e/c50 0 2024-04-05T21:26:38.331 INFO:tasks.workunit.client.0.smithi121.stdout:0/361: truncate d7/d62/d3c/f4b 287714 0 2024-04-05T21:26:38.331 INFO:tasks.workunit.client.0.smithi121.stdout:3/216: dread - d0/d3/d24/f3c zero size 2024-04-05T21:26:38.332 INFO:tasks.workunit.client.1.smithi188.stdout:2/571: truncate d5/fb7 471781 0 2024-04-05T21:26:38.332 INFO:tasks.workunit.client.0.smithi121.stdout:5/353: dread dd/de/d19/f51 [0,4194304] 0 2024-04-05T21:26:38.332 INFO:tasks.workunit.client.0.smithi121.stdout:9/215: dwrite d6/df/f2e [0,4194304] 0 2024-04-05T21:26:38.333 INFO:tasks.workunit.client.1.smithi188.stdout:5/494: rmdir d6/d15/d5c/d7a 39 2024-04-05T21:26:38.333 INFO:tasks.workunit.client.0.smithi121.stdout:0/362: dread - d7/d71/d29/f79 zero size 2024-04-05T21:26:38.335 INFO:tasks.workunit.client.0.smithi121.stdout:3/217: write d0/f15 [5092452,112855] 0 2024-04-05T21:26:38.336 INFO:tasks.workunit.client.1.smithi188.stdout:8/518: dwrite d5/d21/f91 [0,4194304] 0 2024-04-05T21:26:38.337 INFO:tasks.workunit.client.1.smithi188.stdout:1/565: truncate d1/d7/d4e/f8b 1042024 0 2024-04-05T21:26:38.338 INFO:tasks.workunit.client.0.smithi121.stdout:6/261: mkdir d1/d2/d1b/d35/d51 0 2024-04-05T21:26:38.338 INFO:tasks.workunit.client.0.smithi121.stdout:9/216: symlink d6/d32/l43 0 2024-04-05T21:26:38.340 INFO:tasks.workunit.client.0.smithi121.stdout:9/217: dread - d6/df/d1d/d2c/f24 zero size 2024-04-05T21:26:38.340 INFO:tasks.workunit.client.0.smithi121.stdout:5/354: fdatasync dd/de/d19/f37 0 2024-04-05T21:26:38.342 INFO:tasks.workunit.client.0.smithi121.stdout:4/334: truncate d5/f16 4033161 0 2024-04-05T21:26:38.343 INFO:tasks.workunit.client.1.smithi188.stdout:1/566: write d1/dc/d16/d46/d88/d13/f4f [732845,48936] 0 2024-04-05T21:26:38.344 INFO:tasks.workunit.client.0.smithi121.stdout:0/363: creat d7/d62/d3c/d6b/f7a x:0 0 0 2024-04-05T21:26:38.345 INFO:tasks.workunit.client.0.smithi121.stdout:9/218: mkdir d6/df/d44 0 2024-04-05T21:26:38.347 INFO:tasks.workunit.client.0.smithi121.stdout:0/364: read d7/f9 [790266,44348] 0 2024-04-05T21:26:38.347 INFO:tasks.workunit.client.0.smithi121.stdout:6/262: rename d1/d2/d3/l29 to d1/d2/d3/d5/d1f/d2c/d41/l52 0 2024-04-05T21:26:38.349 INFO:tasks.workunit.client.1.smithi188.stdout:4/760: dwrite d0/d2/d17/d3a/fb2 [0,4194304] 0 2024-04-05T21:26:38.351 INFO:tasks.workunit.client.0.smithi121.stdout:9/219: fsync d6/df/d1d/d2c/d37/d3a/f3b 0 2024-04-05T21:26:38.351 INFO:tasks.workunit.client.1.smithi188.stdout:1/567: dread - d1/dc/d16/d21/d27/d4a/f7f zero size 2024-04-05T21:26:38.352 INFO:tasks.workunit.client.0.smithi121.stdout:5/355: dwrite dd/de/d6d/f6e [0,4194304] 0 2024-04-05T21:26:38.354 INFO:tasks.workunit.client.1.smithi188.stdout:8/519: mkdir d5/d21/d40/d43/d44/d59/d8b/da4/db1 0 2024-04-05T21:26:38.354 INFO:tasks.workunit.client.0.smithi121.stdout:6/263: mkdir d1/d2/d1b/d2e/d53 0 2024-04-05T21:26:38.354 INFO:tasks.workunit.client.0.smithi121.stdout:9/220: write d6/f16 [3114850,9474] 0 2024-04-05T21:26:38.355 INFO:tasks.workunit.client.1.smithi188.stdout:2/572: dread d5/f5c [0,4194304] 0 2024-04-05T21:26:38.359 INFO:tasks.workunit.client.1.smithi188.stdout:9/495: dwrite dd/d11/d22/f37 [0,4194304] 0 2024-04-05T21:26:38.359 INFO:tasks.workunit.client.1.smithi188.stdout:2/573: stat d5/fa 0 2024-04-05T21:26:38.359 INFO:tasks.workunit.client.0.smithi121.stdout:4/335: dwrite d5/d11/d26/d49/f72 [0,4194304] 0 2024-04-05T21:26:38.359 INFO:tasks.workunit.client.0.smithi121.stdout:6/264: truncate d1/d2/d3/f4e 107261 0 2024-04-05T21:26:38.360 INFO:tasks.workunit.client.0.smithi121.stdout:5/356: rmdir dd/de/d19/d13 39 2024-04-05T21:26:38.363 INFO:tasks.workunit.client.0.smithi121.stdout:0/365: link d7/d62/f73 d7/d71/d29/d2a/d2c/f7b 0 2024-04-05T21:26:38.364 INFO:tasks.workunit.client.0.smithi121.stdout:9/221: rmdir d6/df/d1d/d2c 39 2024-04-05T21:26:38.365 INFO:tasks.workunit.client.1.smithi188.stdout:5/495: dwrite d6/d15/d5c/d7a/f93 [0,4194304] 0 2024-04-05T21:26:38.366 INFO:tasks.workunit.client.1.smithi188.stdout:8/520: write d5/d21/d40/d43/f83 [45225,121586] 0 2024-04-05T21:26:38.370 INFO:tasks.workunit.client.1.smithi188.stdout:2/574: mknod d5/db/dd/d36/d3d/d46/d5d/cbb 0 2024-04-05T21:26:38.373 INFO:tasks.workunit.client.1.smithi188.stdout:5/496: truncate f2 3153242 0 2024-04-05T21:26:38.373 INFO:tasks.workunit.client.1.smithi188.stdout:9/496: getdents dd/d11/d73/d85 0 2024-04-05T21:26:38.374 INFO:tasks.workunit.client.1.smithi188.stdout:8/521: unlink d5/d21/d40/d43/d44/d5b/d99/d69/d70/f7c 0 2024-04-05T21:26:38.376 INFO:tasks.workunit.client.0.smithi121.stdout:4/336: chown d5/l62 599020 1 2024-04-05T21:26:38.376 INFO:tasks.workunit.client.0.smithi121.stdout:4/337: readlink d5/d45/d67/l6c 0 2024-04-05T21:26:38.377 INFO:tasks.workunit.client.1.smithi188.stdout:2/575: unlink d5/db/dd/d36/d3c/d48/d7a/d7c/c89 0 2024-04-05T21:26:38.379 INFO:tasks.workunit.client.1.smithi188.stdout:8/522: creat d5/d21/d80/fb2 x:0 0 0 2024-04-05T21:26:38.380 INFO:tasks.workunit.client.0.smithi121.stdout:0/366: creat d7/d71/d29/d2a/d2c/f7c x:0 0 0 2024-04-05T21:26:38.380 INFO:tasks.workunit.client.0.smithi121.stdout:5/357: dread dd/de/d19/d38/d45/f4e [0,4194304] 0 2024-04-05T21:26:38.381 INFO:tasks.workunit.client.1.smithi188.stdout:5/497: rename d6/d82/c41 to d6/d15/d5c/d7a/d44/d9b/cb1 0 2024-04-05T21:26:38.383 INFO:tasks.workunit.client.1.smithi188.stdout:4/761: write d0/d14/d15/d42/f63 [77609,98950] 0 2024-04-05T21:26:38.386 INFO:tasks.workunit.client.1.smithi188.stdout:5/498: write d6/f9 [3519114,20058] 0 2024-04-05T21:26:38.386 INFO:tasks.workunit.client.0.smithi121.stdout:6/265: getdents d1/d2/d3/d4/d21 0 2024-04-05T21:26:38.388 INFO:tasks.workunit.client.1.smithi188.stdout:2/576: getdents d5/db/dd/d36/d3c/d48 0 2024-04-05T21:26:38.390 INFO:tasks.workunit.client.0.smithi121.stdout:3/218: write d0/d3/fa [1351243,26704] 0 2024-04-05T21:26:38.391 INFO:tasks.workunit.client.1.smithi188.stdout:5/499: chown d6/d15/d5c/d7a/fa1 21379 1 2024-04-05T21:26:38.395 INFO:tasks.workunit.client.0.smithi121.stdout:9/222: rename d6/c18 to d6/d32/c45 0 2024-04-05T21:26:38.395 INFO:tasks.workunit.client.1.smithi188.stdout:4/762: creat d0/d14/d15/d78/f10b x:0 0 0 2024-04-05T21:26:38.396 INFO:tasks.workunit.client.1.smithi188.stdout:1/568: dwrite d1/dc/d16/d21/d27/d4a/fad [0,4194304] 0 2024-04-05T21:26:38.397 INFO:tasks.workunit.client.1.smithi188.stdout:5/500: fsync d6/d15/d5c/d7a/d44/da0/faa 0 2024-04-05T21:26:38.397 INFO:tasks.workunit.client.1.smithi188.stdout:2/577: symlink d5/lbc 0 2024-04-05T21:26:38.398 INFO:tasks.workunit.client.0.smithi121.stdout:5/358: unlink dd/de/d19/d38/d56/f73 0 2024-04-05T21:26:38.399 INFO:tasks.workunit.client.1.smithi188.stdout:4/763: chown d0/d2/d17/l8c 5438972 1 2024-04-05T21:26:38.401 INFO:tasks.workunit.client.0.smithi121.stdout:4/338: dread d5/f1d [0,4194304] 0 2024-04-05T21:26:38.401 INFO:tasks.workunit.client.1.smithi188.stdout:4/764: fsync d0/d14/d15/d53/ff0 0 2024-04-05T21:26:38.401 INFO:tasks.workunit.client.1.smithi188.stdout:5/501: fdatasync d6/d15/d1b/f6c 0 2024-04-05T21:26:38.402 INFO:tasks.workunit.client.0.smithi121.stdout:0/367: getdents d7 0 2024-04-05T21:26:38.402 INFO:tasks.workunit.client.0.smithi121.stdout:6/266: dwrite d1/d2/d3/d5/d1f/f27 [4194304,4194304] 0 2024-04-05T21:26:38.403 INFO:tasks.workunit.client.0.smithi121.stdout:3/219: rename d0/d3/c2e to d0/d3/de/c46 0 2024-04-05T21:26:38.404 INFO:tasks.workunit.client.0.smithi121.stdout:5/359: creat dd/de/d19/d1c/f75 x:0 0 0 2024-04-05T21:26:38.404 INFO:tasks.workunit.client.0.smithi121.stdout:0/368: write d7/dc/f33 [547879,24554] 0 2024-04-05T21:26:38.405 INFO:tasks.workunit.client.1.smithi188.stdout:8/523: dwrite d5/d21/d2a/d47/f8e [0,4194304] 0 2024-04-05T21:26:38.406 INFO:tasks.workunit.client.1.smithi188.stdout:9/497: dread f1 [0,4194304] 0 2024-04-05T21:26:38.406 INFO:tasks.workunit.client.0.smithi121.stdout:4/339: unlink d5/d11/d2c/f3c 0 2024-04-05T21:26:38.407 INFO:tasks.workunit.client.0.smithi121.stdout:5/360: truncate dd/de/d19/d38/f4d 690515 0 2024-04-05T21:26:38.408 INFO:tasks.workunit.client.0.smithi121.stdout:6/267: mknod d1/d2/d3/d5/c54 0 2024-04-05T21:26:38.409 INFO:tasks.workunit.client.0.smithi121.stdout:6/268: chown d1/d2/d3/d5/d1f/c40 53 1 2024-04-05T21:26:38.410 INFO:tasks.workunit.client.0.smithi121.stdout:6/269: dread - d1/d2/d1b/d35/f47 zero size 2024-04-05T21:26:38.411 INFO:tasks.workunit.client.0.smithi121.stdout:0/369: mkdir d7/d7d 0 2024-04-05T21:26:38.412 INFO:tasks.workunit.client.0.smithi121.stdout:4/340: mkdir d5/d11/d28/d73 0 2024-04-05T21:26:38.412 INFO:tasks.workunit.client.0.smithi121.stdout:4/341: stat d5/d11/d2c 0 2024-04-05T21:26:38.414 INFO:tasks.workunit.client.1.smithi188.stdout:1/569: link d1/dc/c9e d1/cb7 0 2024-04-05T21:26:38.414 INFO:tasks.workunit.client.0.smithi121.stdout:6/270: rmdir d1/d2/d1b 39 2024-04-05T21:26:38.416 INFO:tasks.workunit.client.0.smithi121.stdout:6/271: read d1/d2/d3/d5/d1f/f27 [1451405,97977] 0 2024-04-05T21:26:38.416 INFO:tasks.workunit.client.0.smithi121.stdout:0/370: mkdir d7/d7e 0 2024-04-05T21:26:38.417 INFO:tasks.workunit.client.1.smithi188.stdout:8/524: write d5/d21/d40/d43/faa [1213240,84411] 0 2024-04-05T21:26:38.419 INFO:tasks.workunit.client.1.smithi188.stdout:9/498: rename dd/d11/d1f/d25/c36 to dd/d11/d22/d7e/ca1 0 2024-04-05T21:26:38.420 INFO:tasks.workunit.client.0.smithi121.stdout:5/361: read dd/de/d19/d13/f49 [1020265,34466] 0 2024-04-05T21:26:38.420 INFO:tasks.workunit.client.1.smithi188.stdout:5/502: getdents d6/d15/d5c 0 2024-04-05T21:26:38.421 INFO:tasks.workunit.client.1.smithi188.stdout:1/570: fdatasync d1/dc/d16/d21/d27/d49/f73 0 2024-04-05T21:26:38.421 INFO:tasks.workunit.client.0.smithi121.stdout:2/238: sync 2024-04-05T21:26:38.421 INFO:tasks.workunit.client.0.smithi121.stdout:4/342: rename d5/d11/d28/d3b/f4a to d5/d45/f74 0 2024-04-05T21:26:38.422 INFO:tasks.workunit.client.0.smithi121.stdout:3/220: getdents d0/d3/d24 0 2024-04-05T21:26:38.423 INFO:tasks.workunit.client.1.smithi188.stdout:2/578: dwrite d5/db/dd/d36/d3d/d46/d9f/fb0 [0,4194304] 0 2024-04-05T21:26:38.424 INFO:tasks.workunit.client.0.smithi121.stdout:6/272: write d1/d2/d1b/f28 [30378,96235] 0 2024-04-05T21:26:38.424 INFO:tasks.workunit.client.0.smithi121.stdout:0/371: symlink d7/d62/d3c/d6b/l7f 0 2024-04-05T21:26:38.425 INFO:tasks.workunit.client.0.smithi121.stdout:5/362: truncate dd/de/d19/d1c/d36/f3a 643437 0 2024-04-05T21:26:38.426 INFO:tasks.workunit.client.1.smithi188.stdout:9/499: rename dd/d11/d22/f37 to dd/d11/d73/d8d/fa2 0 2024-04-05T21:26:38.427 INFO:tasks.workunit.client.0.smithi121.stdout:5/363: write dd/de/f22 [4862085,72370] 0 2024-04-05T21:26:38.427 INFO:tasks.workunit.client.0.smithi121.stdout:4/343: chown d5/l71 1756 1 2024-04-05T21:26:38.428 INFO:tasks.workunit.client.1.smithi188.stdout:2/579: write d5/db/dd/d36/d3c/d48/d7a/d7c/f86 [633425,18425] 0 2024-04-05T21:26:38.429 INFO:tasks.workunit.client.0.smithi121.stdout:3/221: symlink d0/d3/d24/d3d/l47 0 2024-04-05T21:26:38.431 INFO:tasks.workunit.client.1.smithi188.stdout:8/525: creat d5/d21/d40/d43/d44/d5b/d99/d69/fb3 x:0 0 0 2024-04-05T21:26:38.431 INFO:tasks.workunit.client.1.smithi188.stdout:5/503: mkdir d6/d82/d40/d42/d81/da6/db2 0 2024-04-05T21:26:38.431 INFO:tasks.workunit.client.1.smithi188.stdout:8/526: chown d5/d21/d2a/c41 45960 1 2024-04-05T21:26:38.434 INFO:tasks.workunit.client.1.smithi188.stdout:9/500: symlink dd/d11/d73/la3 0 2024-04-05T21:26:38.435 INFO:tasks.workunit.client.0.smithi121.stdout:2/239: rename d2/d9/d12/f15 to d2/d28/f44 0 2024-04-05T21:26:38.437 INFO:tasks.workunit.client.1.smithi188.stdout:2/580: creat d5/db/dd/d36/d3d/d46/fbd x:0 0 0 2024-04-05T21:26:38.437 INFO:tasks.workunit.client.1.smithi188.stdout:5/504: truncate d6/d15/d5c/d7a/d44/da0/fb0 1297093 0 2024-04-05T21:26:38.438 INFO:tasks.workunit.client.0.smithi121.stdout:5/364: symlink dd/de/d39/d3b/d50/l76 0 2024-04-05T21:26:38.440 INFO:tasks.workunit.client.1.smithi188.stdout:9/501: rmdir dd/d3d/d95 39 2024-04-05T21:26:38.441 INFO:tasks.workunit.client.0.smithi121.stdout:6/273: creat d1/d2/f55 x:0 0 0 2024-04-05T21:26:38.441 INFO:tasks.workunit.client.1.smithi188.stdout:7/704: sync 2024-04-05T21:26:38.443 INFO:tasks.workunit.client.0.smithi121.stdout:9/223: dread d6/f15 [0,4194304] 0 2024-04-05T21:26:38.443 INFO:tasks.workunit.client.0.smithi121.stdout:3/222: rename d0/d3/d24/d3d/c1f to d0/d3/d24/c48 0 2024-04-05T21:26:38.444 INFO:tasks.workunit.client.0.smithi121.stdout:5/365: chown dd/de/d39/d3b/d50/l76 344494072 1 2024-04-05T21:26:38.444 INFO:tasks.workunit.client.1.smithi188.stdout:8/527: dwrite d5/d21/d40/d43/f51 [0,4194304] 0 2024-04-05T21:26:38.444 INFO:tasks.workunit.client.1.smithi188.stdout:5/505: creat d6/d17/d39/fb3 x:0 0 0 2024-04-05T21:26:38.445 INFO:tasks.workunit.client.0.smithi121.stdout:4/344: dwrite d5/d11/d20/f57 [0,4194304] 0 2024-04-05T21:26:38.446 INFO:tasks.workunit.client.0.smithi121.stdout:4/345: dread d5/f8 [0,4194304] 0 2024-04-05T21:26:38.447 INFO:tasks.workunit.client.1.smithi188.stdout:7/705: unlink d3/d4/d1d/d74/f5c 0 2024-04-05T21:26:38.447 INFO:tasks.workunit.client.1.smithi188.stdout:5/506: read - d6/d82/d40/f90 zero size 2024-04-05T21:26:38.448 INFO:tasks.workunit.client.1.smithi188.stdout:8/528: stat d5/d21/d40/d43/d48/d7e/c8c 0 2024-04-05T21:26:38.450 INFO:tasks.workunit.client.1.smithi188.stdout:4/765: truncate d0/d2/d17/f4e 70551 0 2024-04-05T21:26:38.453 INFO:tasks.workunit.client.1.smithi188.stdout:7/706: creat d3/d4/db/d55/fe0 x:0 0 0 2024-04-05T21:26:38.455 INFO:tasks.workunit.client.0.smithi121.stdout:6/274: fsync d1/f1c 0 2024-04-05T21:26:38.456 INFO:tasks.workunit.client.1.smithi188.stdout:4/766: truncate d0/d2/d17/d3a/f54 397577 0 2024-04-05T21:26:38.456 INFO:tasks.workunit.client.0.smithi121.stdout:4/346: rename d5/d11/d26/d27/d64/f40 to d5/d11/d26/d27/d64/f75 0 2024-04-05T21:26:38.458 INFO:tasks.workunit.client.0.smithi121.stdout:4/347: read d5/d11/f21 [3884252,72630] 0 2024-04-05T21:26:38.458 INFO:tasks.workunit.client.1.smithi188.stdout:7/707: write d3/d4/d1d/d74/d88/fc0 [4428467,64879] 0 2024-04-05T21:26:38.459 INFO:tasks.workunit.client.0.smithi121.stdout:9/224: mkdir d6/d46 0 2024-04-05T21:26:38.459 INFO:tasks.workunit.client.1.smithi188.stdout:4/767: rmdir d0/d2/d2a 39 2024-04-05T21:26:38.459 INFO:tasks.workunit.client.1.smithi188.stdout:7/708: stat d3/d4/db/d10/f14 0 2024-04-05T21:26:38.459 INFO:tasks.workunit.client.1.smithi188.stdout:8/529: dread d5/d21/d40/d43/d44/d5b/d99/d69/d70/f7a [0,4194304] 0 2024-04-05T21:26:38.460 INFO:tasks.workunit.client.1.smithi188.stdout:7/709: write d3/d2c/fd0 [405526,103560] 0 2024-04-05T21:26:38.464 INFO:tasks.workunit.client.1.smithi188.stdout:1/571: truncate d1/dc/d16/d46/d95/d99/fb3 4550692 0 2024-04-05T21:26:38.464 INFO:tasks.workunit.client.0.smithi121.stdout:5/366: getdents dd/de/d39/d3b 0 2024-04-05T21:26:38.465 INFO:tasks.workunit.client.0.smithi121.stdout:5/367: chown dd/l6f 506882 1 2024-04-05T21:26:38.466 INFO:tasks.workunit.client.1.smithi188.stdout:1/572: write d1/dc/d16/d21/d27/d4a/fad [4448618,37425] 0 2024-04-05T21:26:38.467 INFO:tasks.workunit.client.1.smithi188.stdout:8/530: creat d5/dc/d8f/fb4 x:0 0 0 2024-04-05T21:26:38.468 INFO:tasks.workunit.client.0.smithi121.stdout:0/372: dwrite d7/d62/f42 [0,4194304] 0 2024-04-05T21:26:38.468 INFO:tasks.workunit.client.1.smithi188.stdout:4/768: dread d0/d2/d2a/d34/f41 [0,4194304] 0 2024-04-05T21:26:38.469 INFO:tasks.workunit.client.0.smithi121.stdout:0/373: chown d7/d1c 3876 1 2024-04-05T21:26:38.471 INFO:tasks.workunit.client.1.smithi188.stdout:3/602: sync 2024-04-05T21:26:38.472 INFO:tasks.workunit.client.1.smithi188.stdout:8/531: creat d5/d21/d40/d43/fb5 x:0 0 0 2024-04-05T21:26:38.474 INFO:tasks.workunit.client.1.smithi188.stdout:3/603: mknod d5/d48/ccd 0 2024-04-05T21:26:38.474 INFO:tasks.workunit.client.1.smithi188.stdout:2/581: dread d5/db/d35/d3b/f4d [0,4194304] 0 2024-04-05T21:26:38.475 INFO:tasks.workunit.client.0.smithi121.stdout:0/374: creat d7/d71/d45/d5a/f80 x:0 0 0 2024-04-05T21:26:38.476 INFO:tasks.workunit.client.0.smithi121.stdout:5/368: getdents dd/de/d19/d38 0 2024-04-05T21:26:38.476 INFO:tasks.workunit.client.1.smithi188.stdout:4/769: rename d0/d2/d17/d48/d70/fc0 to d0/dbe/f10c 0 2024-04-05T21:26:38.477 INFO:tasks.workunit.client.0.smithi121.stdout:0/375: dread - d7/d71/d29/d2a/f4c zero size 2024-04-05T21:26:38.477 INFO:tasks.workunit.client.0.smithi121.stdout:2/240: dread d2/f10 [0,4194304] 0 2024-04-05T21:26:38.478 INFO:tasks.workunit.client.0.smithi121.stdout:0/376: readlink d7/d62/l30 0 2024-04-05T21:26:38.478 INFO:tasks.workunit.client.0.smithi121.stdout:2/241: write d2/f11 [5172929,43344] 0 2024-04-05T21:26:38.479 INFO:tasks.workunit.client.0.smithi121.stdout:5/369: symlink dd/l77 0 2024-04-05T21:26:38.482 INFO:tasks.workunit.client.0.smithi121.stdout:0/377: fsync d7/d71/d29/d3f/f72 0 2024-04-05T21:26:38.482 INFO:tasks.workunit.client.1.smithi188.stdout:3/604: fdatasync d5/d7/d65/d61/f92 0 2024-04-05T21:26:38.485 INFO:tasks.workunit.client.0.smithi121.stdout:2/242: mknod d2/c45 0 2024-04-05T21:26:38.486 INFO:tasks.workunit.client.1.smithi188.stdout:8/532: creat d5/d21/d40/d43/d44/d59/d8b/fb6 x:0 0 0 2024-04-05T21:26:38.486 INFO:tasks.workunit.client.0.smithi121.stdout:5/370: dread dd/de/d19/d38/d45/f54 [0,4194304] 0 2024-04-05T21:26:38.488 INFO:tasks.workunit.client.0.smithi121.stdout:0/378: fsync f2 0 2024-04-05T21:26:38.488 INFO:tasks.workunit.client.0.smithi121.stdout:3/223: dwrite d0/d3/de/f17 [0,4194304] 0 2024-04-05T21:26:38.489 INFO:tasks.workunit.client.0.smithi121.stdout:5/371: stat dd/de/d39/d3b/f5e 0 2024-04-05T21:26:38.490 INFO:tasks.workunit.client.1.smithi188.stdout:4/770: dwrite d0/d2/d2a/d34/fd1 [0,4194304] 0 2024-04-05T21:26:38.490 INFO:tasks.workunit.client.1.smithi188.stdout:7/710: dread d3/d4/f25 [0,4194304] 0 2024-04-05T21:26:38.492 INFO:tasks.workunit.client.0.smithi121.stdout:0/379: creat d7/d7d/f81 x:0 0 0 2024-04-05T21:26:38.492 INFO:tasks.workunit.client.1.smithi188.stdout:5/507: dread d6/d15/d1b/f2a [0,4194304] 0 2024-04-05T21:26:38.492 INFO:tasks.workunit.client.0.smithi121.stdout:3/224: symlink d0/d3/d24/d3f/l49 0 2024-04-05T21:26:38.493 INFO:tasks.workunit.client.1.smithi188.stdout:7/711: chown d3/d4/db/d2e/d8c/db7/dd6 996323524 1 2024-04-05T21:26:38.493 INFO:tasks.workunit.client.1.smithi188.stdout:9/502: dread dd/d34/f23 [0,4194304] 0 2024-04-05T21:26:38.493 INFO:tasks.workunit.client.0.smithi121.stdout:5/372: symlink dd/d41/l78 0 2024-04-05T21:26:38.494 INFO:tasks.workunit.client.0.smithi121.stdout:3/225: write d0/f27 [122940,53781] 0 2024-04-05T21:26:38.494 INFO:tasks.workunit.client.0.smithi121.stdout:5/373: fdatasync dd/de/d19/f57 0 2024-04-05T21:26:38.495 INFO:tasks.workunit.client.0.smithi121.stdout:5/374: read dd/de/d19/d3c/f40 [2891526,39750] 0 2024-04-05T21:26:38.495 INFO:tasks.workunit.client.0.smithi121.stdout:0/380: mknod d7/d71/d29/c82 0 2024-04-05T21:26:38.496 INFO:tasks.workunit.client.0.smithi121.stdout:0/381: dread - d7/d62/f73 zero size 2024-04-05T21:26:38.497 INFO:tasks.workunit.client.0.smithi121.stdout:5/375: chown dd/de/c14 0 1 2024-04-05T21:26:38.501 INFO:tasks.workunit.client.0.smithi121.stdout:0/382: creat d7/d71/d29/d2a/d64/f83 x:0 0 0 2024-04-05T21:26:38.502 INFO:tasks.workunit.client.0.smithi121.stdout:5/376: creat dd/de/d19/d38/d45/d68/f79 x:0 0 0 2024-04-05T21:26:38.504 INFO:tasks.workunit.client.0.smithi121.stdout:5/377: symlink dd/de/d19/d1c/d36/d46/l7a 0 2024-04-05T21:26:38.506 INFO:tasks.workunit.client.1.smithi188.stdout:3/605: mknod d5/d39/d84/cce 0 2024-04-05T21:26:38.506 INFO:tasks.workunit.client.0.smithi121.stdout:5/378: dread - dd/de/d19/d1c/f48 zero size 2024-04-05T21:26:38.506 INFO:tasks.workunit.client.0.smithi121.stdout:0/383: link d7/d71/d29/d2a/d64/f83 d7/f84 0 2024-04-05T21:26:38.507 INFO:tasks.workunit.client.0.smithi121.stdout:0/384: chown d7/d62/c21 1345960 1 2024-04-05T21:26:38.507 INFO:tasks.workunit.client.1.smithi188.stdout:5/508: symlink d6/d15/d1b/lb4 0 2024-04-05T21:26:38.507 INFO:tasks.workunit.client.1.smithi188.stdout:8/533: chown d5/d21/d40/d43/d44/l6f 25627213 1 2024-04-05T21:26:38.510 INFO:tasks.workunit.client.1.smithi188.stdout:3/606: fsync d5/d7/d65/d61/d66/fc2 0 2024-04-05T21:26:38.510 INFO:tasks.workunit.client.1.smithi188.stdout:7/712: creat d3/d4/db/d10/d5f/fe1 x:0 0 0 2024-04-05T21:26:38.510 INFO:tasks.workunit.client.0.smithi121.stdout:5/379: dread - dd/de/d19/d1c/d36/d46/f64 zero size 2024-04-05T21:26:38.510 INFO:tasks.workunit.client.0.smithi121.stdout:0/385: truncate d7/d71/d29/d2a/d64/f83 588319 0 2024-04-05T21:26:38.510 INFO:tasks.workunit.client.0.smithi121.stdout:5/380: chown dd/l77 3178 1 2024-04-05T21:26:38.511 INFO:tasks.workunit.client.1.smithi188.stdout:5/509: write d6/d17/d39/fb3 [904455,44563] 0 2024-04-05T21:26:38.512 INFO:tasks.workunit.client.0.smithi121.stdout:0/386: mknod d7/d71/d29/d2a/d2c/c85 0 2024-04-05T21:26:38.513 INFO:tasks.workunit.client.0.smithi121.stdout:0/387: chown d7/d62/c76 2037444 1 2024-04-05T21:26:38.513 INFO:tasks.workunit.client.0.smithi121.stdout:5/381: mkdir dd/de/d19/d1c/d7b 0 2024-04-05T21:26:38.514 INFO:tasks.workunit.client.1.smithi188.stdout:4/771: rename d0/d2/caf to d0/dbe/c10d 0 2024-04-05T21:26:38.516 INFO:tasks.workunit.client.0.smithi121.stdout:5/382: write dd/de/d19/d13/f2d [4369614,75498] 0 2024-04-05T21:26:38.518 INFO:tasks.workunit.client.0.smithi121.stdout:5/383: dread dd/de/d19/d1c/d36/f3a [0,4194304] 0 2024-04-05T21:26:38.521 INFO:tasks.workunit.client.0.smithi121.stdout:5/384: unlink dd/de/d19/d1c/f59 0 2024-04-05T21:26:38.521 INFO:tasks.workunit.client.1.smithi188.stdout:7/713: dwrite d3/d4/db/f38 [4194304,4194304] 0 2024-04-05T21:26:38.524 INFO:tasks.workunit.client.0.smithi121.stdout:5/385: dread dd/de/d6d/f70 [0,4194304] 0 2024-04-05T21:26:38.527 INFO:tasks.workunit.client.0.smithi121.stdout:5/386: creat dd/de/d19/d38/f7c x:0 0 0 2024-04-05T21:26:38.527 INFO:tasks.workunit.client.1.smithi188.stdout:8/534: creat d5/d6e/fb7 x:0 0 0 2024-04-05T21:26:38.527 INFO:tasks.workunit.client.1.smithi188.stdout:1/573: dwrite d1/dc/d16/d46/f51 [0,4194304] 0 2024-04-05T21:26:38.527 INFO:tasks.workunit.client.1.smithi188.stdout:3/607: write d5/d7/d65/f54 [4875467,56260] 0 2024-04-05T21:26:38.527 INFO:tasks.workunit.client.1.smithi188.stdout:5/510: dread - d6/d15/d5c/d7a/d44/f5d zero size 2024-04-05T21:26:38.527 INFO:tasks.workunit.client.1.smithi188.stdout:4/772: dread - d0/d2/d2a/d34/d6b/f7e zero size 2024-04-05T21:26:38.528 INFO:tasks.workunit.client.1.smithi188.stdout:9/503: rmdir dd/d34/d99 0 2024-04-05T21:26:38.531 INFO:tasks.workunit.client.1.smithi188.stdout:2/582: truncate d5/db/f76 5177020 0 2024-04-05T21:26:38.533 INFO:tasks.workunit.client.0.smithi121.stdout:5/387: symlink dd/de/d39/d3b/l7d 0 2024-04-05T21:26:38.533 INFO:tasks.workunit.client.0.smithi121.stdout:2/243: dread d2/d9/d17/f18 [0,4194304] 0 2024-04-05T21:26:38.534 INFO:tasks.workunit.client.1.smithi188.stdout:1/574: creat d1/dc/d16/d21/d27/d49/fb8 x:0 0 0 2024-04-05T21:26:38.537 INFO:tasks.workunit.client.1.smithi188.stdout:4/773: mknod d0/d2/d2a/d71/c10e 0 2024-04-05T21:26:38.538 INFO:tasks.workunit.client.1.smithi188.stdout:8/535: dwrite d5/d21/d2a/d47/f54 [0,4194304] 0 2024-04-05T21:26:38.543 INFO:tasks.workunit.client.1.smithi188.stdout:2/583: write d5/db/dd/d36/d3d/d46/d66/fb3 [107566,125880] 0 2024-04-05T21:26:38.543 INFO:tasks.workunit.client.1.smithi188.stdout:8/536: stat d5/d21/d40/d43/d48 0 2024-04-05T21:26:38.545 INFO:tasks.workunit.client.0.smithi121.stdout:5/388: dwrite dd/de/d19/d1c/d36/f3a [0,4194304] 0 2024-04-05T21:26:38.546 INFO:tasks.workunit.client.0.smithi121.stdout:5/389: readlink dd/de/d19/d1c/l2f 0 2024-04-05T21:26:38.547 INFO:tasks.workunit.client.0.smithi121.stdout:0/388: dread d7/dc/f32 [0,4194304] 0 2024-04-05T21:26:38.549 INFO:tasks.workunit.client.0.smithi121.stdout:0/389: chown d7/d71/d29/f58 20014 1 2024-04-05T21:26:38.550 INFO:tasks.workunit.client.0.smithi121.stdout:2/244: dwrite d2/d9/fd [0,4194304] 0 2024-04-05T21:26:38.552 INFO:tasks.workunit.client.0.smithi121.stdout:2/245: write d2/d9/d17/f19 [5327218,33632] 0 2024-04-05T21:26:38.553 INFO:tasks.workunit.client.1.smithi188.stdout:3/608: dread d5/f1e [0,4194304] 0 2024-04-05T21:26:38.554 INFO:tasks.workunit.client.0.smithi121.stdout:5/390: rename dd/de/d19/d38/d45/f54 to dd/de/d19/d13/d53/f7e 0 2024-04-05T21:26:38.557 INFO:tasks.workunit.client.0.smithi121.stdout:5/391: creat dd/de/d19/d13/d53/f7f x:0 0 0 2024-04-05T21:26:38.557 INFO:tasks.workunit.client.0.smithi121.stdout:5/392: stat dd/l20 0 2024-04-05T21:26:38.558 INFO:tasks.workunit.client.0.smithi121.stdout:2/246: link d2/d9/d38/c26 d2/d1e/c46 0 2024-04-05T21:26:38.559 INFO:tasks.workunit.client.0.smithi121.stdout:2/247: dread - d2/d9/f2f zero size 2024-04-05T21:26:38.560 INFO:tasks.workunit.client.1.smithi188.stdout:3/609: symlink d5/d7/d6d/lcf 0 2024-04-05T21:26:38.561 INFO:tasks.workunit.client.0.smithi121.stdout:5/393: truncate dd/f17 1787626 0 2024-04-05T21:26:38.565 INFO:tasks.workunit.client.0.smithi121.stdout:2/248: dwrite d2/d9/ff [0,4194304] 0 2024-04-05T21:26:38.567 INFO:tasks.workunit.client.0.smithi121.stdout:2/249: write d2/d9/ff [4603696,55256] 0 2024-04-05T21:26:38.569 INFO:tasks.workunit.client.0.smithi121.stdout:2/250: truncate d2/d28/f44 5111440 0 2024-04-05T21:26:38.570 INFO:tasks.workunit.client.1.smithi188.stdout:4/774: rename d0/d2/d52/lce to d0/d14/d2f/l10f 0 2024-04-05T21:26:38.570 INFO:tasks.workunit.client.1.smithi188.stdout:9/504: dwrite fb [4194304,4194304] 0 2024-04-05T21:26:38.570 INFO:tasks.workunit.client.1.smithi188.stdout:3/610: symlink d5/d35/d4e/ld0 0 2024-04-05T21:26:38.570 INFO:tasks.workunit.client.1.smithi188.stdout:4/775: read - d0/fa8 zero size 2024-04-05T21:26:38.571 INFO:tasks.workunit.client.0.smithi121.stdout:2/251: fdatasync d2/d9/d38/d2d/f41 0 2024-04-05T21:26:38.573 INFO:tasks.workunit.client.0.smithi121.stdout:2/252: mkdir d2/d9/d38/d2d/d47 0 2024-04-05T21:26:38.574 INFO:tasks.workunit.client.1.smithi188.stdout:8/537: rename d5/d21/d40/d43/d44/d59/d9e to d5/dc/d8f/db8 0 2024-04-05T21:26:38.576 INFO:tasks.workunit.client.0.smithi121.stdout:2/253: mknod d2/d9/d38/c48 0 2024-04-05T21:26:38.576 INFO:tasks.workunit.client.1.smithi188.stdout:0/427: sync 2024-04-05T21:26:38.577 INFO:tasks.workunit.client.1.smithi188.stdout:6/614: sync 2024-04-05T21:26:38.579 INFO:tasks.workunit.client.1.smithi188.stdout:6/615: chown d6/d7/c10 257775465 1 2024-04-05T21:26:38.583 INFO:tasks.workunit.client.1.smithi188.stdout:9/505: dwrite dd/d11/d22/f4d [0,4194304] 0 2024-04-05T21:26:38.587 INFO:tasks.workunit.client.1.smithi188.stdout:0/428: unlink d4/d8/d10/f20 0 2024-04-05T21:26:38.588 INFO:tasks.workunit.client.0.smithi121.stdout:2/254: fsync d2/d1e/f24 0 2024-04-05T21:26:38.589 INFO:tasks.workunit.client.1.smithi188.stdout:1/575: dread d1/dc/d16/f6c [0,4194304] 0 2024-04-05T21:26:38.589 INFO:tasks.workunit.client.0.smithi121.stdout:2/255: chown d2/d9/d38/d2d/c34 758 1 2024-04-05T21:26:38.590 INFO:tasks.workunit.client.0.smithi121.stdout:2/256: readlink d2/l6 0 2024-04-05T21:26:38.590 INFO:tasks.workunit.client.1.smithi188.stdout:8/538: write d5/d21/d40/f7b [3819541,56307] 0 2024-04-05T21:26:38.591 INFO:tasks.workunit.client.1.smithi188.stdout:4/776: dwrite d0/d14/d15/d53/d82/d8f/fad [0,4194304] 0 2024-04-05T21:26:38.595 INFO:tasks.workunit.client.0.smithi121.stdout:2/257: dwrite d2/f1a [0,4194304] 0 2024-04-05T21:26:38.597 INFO:tasks.workunit.client.1.smithi188.stdout:9/506: chown dd/l5d 103532 1 2024-04-05T21:26:38.599 INFO:tasks.workunit.client.1.smithi188.stdout:9/507: stat dd/d11/d1f/d25/d8a/c90 0 2024-04-05T21:26:38.600 INFO:tasks.workunit.client.0.smithi121.stdout:2/258: link d2/d28/c33 d2/d9/d12/c49 0 2024-04-05T21:26:38.603 INFO:tasks.workunit.client.1.smithi188.stdout:5/511: dwrite f2 [0,4194304] 0 2024-04-05T21:26:38.604 INFO:tasks.workunit.client.1.smithi188.stdout:5/512: readlink d6/d15/d1b/l25 0 2024-04-05T21:26:38.611 INFO:tasks.workunit.client.0.smithi121.stdout:5/394: fsync dd/de/d19/d13/d53/f7f 0 2024-04-05T21:26:38.615 INFO:tasks.workunit.client.1.smithi188.stdout:1/576: creat d1/dc/d16/d21/d27/d4a/fb9 x:0 0 0 2024-04-05T21:26:38.616 INFO:tasks.workunit.client.1.smithi188.stdout:0/429: mknod d4/d8/db/d21/d3c/c9c 0 2024-04-05T21:26:38.619 INFO:tasks.workunit.client.1.smithi188.stdout:3/611: rename d5/d2b/c3c to d5/dc/cd1 0 2024-04-05T21:26:38.622 INFO:tasks.workunit.client.1.smithi188.stdout:3/612: fdatasync d5/d39/f93 0 2024-04-05T21:26:38.622 INFO:tasks.workunit.client.1.smithi188.stdout:8/539: rmdir d5 39 2024-04-05T21:26:38.635 INFO:tasks.workunit.client.1.smithi188.stdout:5/513: mknod d6/d15/d1b/cb5 0 2024-04-05T21:26:38.636 INFO:tasks.workunit.client.1.smithi188.stdout:1/577: read d1/dc/d16/d46/d88/d13/f8d [282537,44315] 0 2024-04-05T21:26:38.636 INFO:tasks.workunit.client.1.smithi188.stdout:4/777: truncate d0/d14/d15/d78/db1/d6a/f100 743810 0 2024-04-05T21:26:38.637 INFO:tasks.workunit.client.1.smithi188.stdout:2/584: dwrite d5/db/d35/d3b/f84 [0,4194304] 0 2024-04-05T21:26:38.638 INFO:tasks.workunit.client.1.smithi188.stdout:9/508: symlink dd/d11/la4 0 2024-04-05T21:26:38.639 INFO:tasks.workunit.client.1.smithi188.stdout:5/514: chown d6/d82/d40 934840002 1 2024-04-05T21:26:38.641 INFO:tasks.workunit.client.1.smithi188.stdout:5/515: chown d6/cab 948 1 2024-04-05T21:26:38.664 INFO:tasks.workunit.client.1.smithi188.stdout:6/616: getdents d6/d13/d48/d49/d75 0 2024-04-05T21:26:38.665 INFO:tasks.workunit.client.1.smithi188.stdout:0/430: mkdir d4/d8/d78/d9d 0 2024-04-05T21:26:38.667 INFO:tasks.workunit.client.1.smithi188.stdout:8/540: readlink d5/d21/d40/d43/d44/d59/l90 0 2024-04-05T21:26:38.667 INFO:tasks.workunit.client.1.smithi188.stdout:3/613: mknod d5/d7e/cd2 0 2024-04-05T21:26:38.674 INFO:tasks.workunit.client.1.smithi188.stdout:6/617: unlink d6/d28/f84 0 2024-04-05T21:26:38.675 INFO:tasks.workunit.client.1.smithi188.stdout:4/778: mknod d0/d14/d2f/ddd/c110 0 2024-04-05T21:26:38.676 INFO:tasks.workunit.client.1.smithi188.stdout:3/614: fdatasync d5/d2b/f2e 0 2024-04-05T21:26:38.677 INFO:tasks.workunit.client.1.smithi188.stdout:2/585: mkdir d5/db/dd/d36/dbe 0 2024-04-05T21:26:38.677 INFO:tasks.workunit.client.1.smithi188.stdout:1/578: symlink d1/dc/d16/d2d/db0/lba 0 2024-04-05T21:26:38.678 INFO:tasks.workunit.client.1.smithi188.stdout:3/615: readlink d5/d7/l26 0 2024-04-05T21:26:38.678 INFO:tasks.workunit.client.1.smithi188.stdout:9/509: mkdir dd/d11/d1f/d26/d55/d98/da5 0 2024-04-05T21:26:38.679 INFO:tasks.workunit.client.1.smithi188.stdout:4/779: chown d0/d14/d15/d53/d82/dd5/ff1 2035797 1 2024-04-05T21:26:38.681 INFO:tasks.workunit.client.1.smithi188.stdout:1/579: readlink d1/dc/d16/d46/d88/d3f/laa 0 2024-04-05T21:26:38.682 INFO:tasks.workunit.client.1.smithi188.stdout:4/780: dread - d0/d14/d15/d78/f10b zero size 2024-04-05T21:26:38.682 INFO:tasks.workunit.client.1.smithi188.stdout:0/431: link d4/d6/d46/f8b d4/d8/db/d21/d3c/f9e 0 2024-04-05T21:26:38.682 INFO:tasks.workunit.client.1.smithi188.stdout:9/510: dread - dd/d11/d73/f92 zero size 2024-04-05T21:26:38.685 INFO:tasks.workunit.client.0.smithi121.stdout:8/261: sync 2024-04-05T21:26:38.685 INFO:tasks.workunit.client.1.smithi188.stdout:8/541: creat d5/d6e/dad/fb9 x:0 0 0 2024-04-05T21:26:38.685 INFO:tasks.workunit.client.0.smithi121.stdout:8/262: fdatasync d12/d13/d15/f19 0 2024-04-05T21:26:38.686 INFO:tasks.workunit.client.1.smithi188.stdout:0/432: fsync d4/d8/d39/f48 0 2024-04-05T21:26:38.686 INFO:tasks.workunit.client.1.smithi188.stdout:3/616: creat d5/d35/d4e/fd3 x:0 0 0 2024-04-05T21:26:38.687 INFO:tasks.workunit.client.1.smithi188.stdout:6/618: symlink d6/d13/d48/lbe 0 2024-04-05T21:26:38.688 INFO:tasks.workunit.client.1.smithi188.stdout:5/516: dread d6/f22 [0,4194304] 0 2024-04-05T21:26:38.689 INFO:tasks.workunit.client.1.smithi188.stdout:6/619: write d6/d28/f43 [3292728,121902] 0 2024-04-05T21:26:38.689 INFO:tasks.workunit.client.1.smithi188.stdout:4/781: unlink d0/d2/d17/f43 0 2024-04-05T21:26:38.689 INFO:tasks.workunit.client.0.smithi121.stdout:8/263: rmdir d12/d30 39 2024-04-05T21:26:38.690 INFO:tasks.workunit.client.1.smithi188.stdout:5/517: chown d6/d15/d1b/f2a 3202 1 2024-04-05T21:26:38.691 INFO:tasks.workunit.client.0.smithi121.stdout:5/395: creat dd/de/d19/f80 x:0 0 0 2024-04-05T21:26:38.691 INFO:tasks.workunit.client.1.smithi188.stdout:1/580: mkdir d1/dc/d16/d21/dbb 0 2024-04-05T21:26:38.692 INFO:tasks.workunit.client.1.smithi188.stdout:2/586: symlink d5/db/dd/d36/lbf 0 2024-04-05T21:26:38.692 INFO:tasks.workunit.client.0.smithi121.stdout:2/259: getdents d2/d9/d12 0 2024-04-05T21:26:38.694 INFO:tasks.workunit.client.1.smithi188.stdout:9/511: readlink dd/d49/d6b/l54 0 2024-04-05T21:26:38.695 INFO:tasks.workunit.client.0.smithi121.stdout:2/260: unlink d2/d28/f44 0 2024-04-05T21:26:38.695 INFO:tasks.workunit.client.1.smithi188.stdout:0/433: creat d4/d8/f9f x:0 0 0 2024-04-05T21:26:38.695 INFO:tasks.workunit.client.1.smithi188.stdout:9/512: fsync dd/d11/d22/f4d 0 2024-04-05T21:26:38.695 INFO:tasks.workunit.client.0.smithi121.stdout:5/396: fsync dd/f17 0 2024-04-05T21:26:38.696 INFO:tasks.workunit.client.1.smithi188.stdout:0/434: chown d4/d8/db/d21/l3e 663535226 1 2024-04-05T21:26:38.696 INFO:tasks.workunit.client.0.smithi121.stdout:5/397: fsync dd/de/d19/d1c/d36/f71 0 2024-04-05T21:26:38.697 INFO:tasks.workunit.client.1.smithi188.stdout:3/617: fdatasync d5/d7/d6d/d75/d83/f8e 0 2024-04-05T21:26:38.697 INFO:tasks.workunit.client.0.smithi121.stdout:8/264: rename d12/d13/d15/f1f to d12/f5d 0 2024-04-05T21:26:38.698 INFO:tasks.workunit.client.0.smithi121.stdout:2/261: rmdir d2/d9/d12 39 2024-04-05T21:26:38.698 INFO:tasks.workunit.client.0.smithi121.stdout:8/265: fdatasync fc 0 2024-04-05T21:26:38.702 INFO:tasks.workunit.client.1.smithi188.stdout:6/620: creat d6/d13/d48/d49/d75/fbf x:0 0 0 2024-04-05T21:26:38.704 INFO:tasks.workunit.client.1.smithi188.stdout:4/782: mkdir d0/d14/d15/dd4/d111 0 2024-04-05T21:26:38.706 INFO:tasks.workunit.client.1.smithi188.stdout:3/618: dread d5/df/f15 [4194304,4194304] 0 2024-04-05T21:26:38.708 INFO:tasks.workunit.client.1.smithi188.stdout:2/587: rename d5/db/f16 to d5/db/dd/d36/d3d/d46/fc0 0 2024-04-05T21:26:38.709 INFO:tasks.workunit.client.1.smithi188.stdout:3/619: write d5/d7/d65/d61/d66/fc2 [130288,93946] 0 2024-04-05T21:26:38.710 INFO:tasks.workunit.client.1.smithi188.stdout:2/588: chown d5/db/dd/d36/d3c/d48/d78 134410576 1 2024-04-05T21:26:38.711 INFO:tasks.workunit.client.1.smithi188.stdout:9/513: dwrite dd/f42 [0,4194304] 0 2024-04-05T21:26:38.712 INFO:tasks.workunit.client.1.smithi188.stdout:8/542: link d5/d21/d40/f73 d5/d21/d40/d43/d44/d5b/d99/da3/fba 0 2024-04-05T21:26:38.713 INFO:tasks.workunit.client.1.smithi188.stdout:3/620: chown d5/d7/d6d/lcf 8279522 1 2024-04-05T21:26:38.723 INFO:tasks.workunit.client.1.smithi188.stdout:9/514: symlink dd/d11/d1f/d25/d8a/la6 0 2024-04-05T21:26:38.725 INFO:tasks.workunit.client.1.smithi188.stdout:8/543: mkdir d5/d21/d40/d43/dbb 0 2024-04-05T21:26:38.730 INFO:tasks.workunit.client.1.smithi188.stdout:2/589: creat d5/db/dd/d36/d3d/d46/d66/d8f/fc1 x:0 0 0 2024-04-05T21:26:38.731 INFO:tasks.workunit.client.1.smithi188.stdout:9/515: mkdir dd/d34/da7 0 2024-04-05T21:26:38.735 INFO:tasks.workunit.client.1.smithi188.stdout:0/435: rename d4/d6/d61/f6a to d4/fa0 0 2024-04-05T21:26:38.737 INFO:tasks.workunit.client.1.smithi188.stdout:2/590: creat d5/db/dd/da4/fc2 x:0 0 0 2024-04-05T21:26:38.738 INFO:tasks.workunit.client.1.smithi188.stdout:2/591: chown d5/db/d35 29585997 1 2024-04-05T21:26:38.739 INFO:tasks.workunit.client.1.smithi188.stdout:8/544: creat d5/d21/d2a/d71/fbc x:0 0 0 2024-04-05T21:26:38.742 INFO:tasks.workunit.client.1.smithi188.stdout:0/436: dread d4/d8/f75 [0,4194304] 0 2024-04-05T21:26:38.746 INFO:tasks.workunit.client.0.smithi121.stdout:5/398: dread dd/f6c [0,4194304] 0 2024-04-05T21:26:38.746 INFO:tasks.workunit.client.1.smithi188.stdout:2/592: dread d5/db/dd/d24/f52 [4194304,4194304] 0 2024-04-05T21:26:38.747 INFO:tasks.workunit.client.0.smithi121.stdout:5/399: chown dd/de/d19/d1c/d36/d46/f64 1633424 1 2024-04-05T21:26:38.748 INFO:tasks.workunit.client.1.smithi188.stdout:2/593: write d5/db/dd/d36/d3c/d48/d7a/f99 [95778,123891] 0 2024-04-05T21:26:38.749 INFO:tasks.workunit.client.0.smithi121.stdout:2/262: rename d2/d9/fe to d2/d9/d17/d3a/f4a 0 2024-04-05T21:26:38.749 INFO:tasks.workunit.client.1.smithi188.stdout:2/594: chown d5/db/dd/c69 517842 1 2024-04-05T21:26:38.753 INFO:tasks.workunit.client.1.smithi188.stdout:0/437: getdents d4/d8/db/d21 0 2024-04-05T21:26:38.753 INFO:tasks.workunit.client.1.smithi188.stdout:5/518: truncate d6/d17/f1f 1171682 0 2024-04-05T21:26:38.753 INFO:tasks.workunit.client.0.smithi121.stdout:8/266: dwrite d12/d30/f3a [0,4194304] 0 2024-04-05T21:26:38.753 INFO:tasks.workunit.client.0.smithi121.stdout:5/400: read dd/f1f [3765519,15924] 0 2024-04-05T21:26:38.754 INFO:tasks.workunit.client.0.smithi121.stdout:8/267: chown d12/d13/d15/d1d/f27 510302 1 2024-04-05T21:26:38.756 INFO:tasks.workunit.client.0.smithi121.stdout:8/268: fdatasync d12/d13/d15/f18 0 2024-04-05T21:26:38.756 INFO:tasks.workunit.client.1.smithi188.stdout:1/581: truncate d1/dc/d16/f3b 491331 0 2024-04-05T21:26:38.756 INFO:tasks.workunit.client.1.smithi188.stdout:2/595: mknod d5/db/dd/d36/dbe/cc3 0 2024-04-05T21:26:38.757 INFO:tasks.workunit.client.1.smithi188.stdout:4/783: write d0/d14/d15/d78/db1/f7b [493910,82116] 0 2024-04-05T21:26:38.757 INFO:tasks.workunit.client.0.smithi121.stdout:8/269: mkdir d12/d13/d56/d57/d5e 0 2024-04-05T21:26:38.760 INFO:tasks.workunit.client.1.smithi188.stdout:6/621: dwrite d6/d13/d18/d19/d2c/f3e [0,4194304] 0 2024-04-05T21:26:38.761 INFO:tasks.workunit.client.0.smithi121.stdout:8/270: dread d12/d13/f20 [0,4194304] 0 2024-04-05T21:26:38.762 INFO:tasks.workunit.client.1.smithi188.stdout:5/519: write d6/d82/d40/d42/d81/d95/f9d [882626,12054] 0 2024-04-05T21:26:38.766 INFO:tasks.workunit.client.0.smithi121.stdout:8/271: symlink d12/d13/l5f 0 2024-04-05T21:26:38.766 INFO:tasks.workunit.client.0.smithi121.stdout:8/272: fdatasync fa 0 2024-04-05T21:26:38.766 INFO:tasks.workunit.client.1.smithi188.stdout:0/438: write d4/d8/db/d42/d38/d77/f95 [54520,65384] 0 2024-04-05T21:26:38.766 INFO:tasks.workunit.client.1.smithi188.stdout:9/516: truncate dd/d11/d73/d8d/fa2 434337 0 2024-04-05T21:26:38.766 INFO:tasks.workunit.client.1.smithi188.stdout:4/784: mkdir d0/d14/d15/d78/d112 0 2024-04-05T21:26:38.771 INFO:tasks.workunit.client.1.smithi188.stdout:4/785: readlink d0/d2/d2a/d34/d6b/lf5 0 2024-04-05T21:26:38.772 INFO:tasks.workunit.client.0.smithi121.stdout:8/273: dwrite d12/d13/d15/f18 [0,4194304] 0 2024-04-05T21:26:38.772 INFO:tasks.workunit.client.1.smithi188.stdout:0/439: write d4/d8/d10/f6d [106486,116506] 0 2024-04-05T21:26:38.773 INFO:tasks.workunit.client.1.smithi188.stdout:2/596: dread d5/db/dd/d36/d3c/d48/d7a/f99 [0,4194304] 0 2024-04-05T21:26:38.774 INFO:tasks.workunit.client.1.smithi188.stdout:9/517: truncate dd/d11/d1f/f29 296174 0 2024-04-05T21:26:38.779 INFO:tasks.workunit.client.1.smithi188.stdout:9/518: symlink dd/d49/d6b/la8 0 2024-04-05T21:26:38.779 INFO:tasks.workunit.client.1.smithi188.stdout:3/621: dwrite d5/dc/f2d [0,4194304] 0 2024-04-05T21:26:38.779 INFO:tasks.workunit.client.1.smithi188.stdout:3/622: dread - d5/f9b zero size 2024-04-05T21:26:38.779 INFO:tasks.workunit.client.1.smithi188.stdout:4/786: symlink d0/d14/d2f/l113 0 2024-04-05T21:26:38.780 INFO:tasks.workunit.client.1.smithi188.stdout:0/440: mkdir d4/d6/d13/da1 0 2024-04-05T21:26:38.781 INFO:tasks.workunit.client.1.smithi188.stdout:2/597: mknod d5/db/dd/cc4 0 2024-04-05T21:26:38.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:38 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:26:38.788 INFO:tasks.workunit.client.1.smithi188.stdout:3/623: link d5/dc/l3f d5/d39/ld4 0 2024-04-05T21:26:38.789 INFO:tasks.workunit.client.1.smithi188.stdout:0/441: dwrite d4/d8/db/d21/f32 [4194304,4194304] 0 2024-04-05T21:26:38.796 INFO:tasks.workunit.client.1.smithi188.stdout:0/442: dwrite d4/d8/db/d21/f56 [0,4194304] 0 2024-04-05T21:26:38.797 INFO:tasks.workunit.client.1.smithi188.stdout:0/443: write d4/d8/db/d42/d38/d77/f95 [599489,48894] 0 2024-04-05T21:26:38.800 INFO:tasks.workunit.client.1.smithi188.stdout:0/444: mkdir d4/d8/db/d42/da2 0 2024-04-05T21:26:38.801 INFO:tasks.workunit.client.1.smithi188.stdout:0/445: creat d4/d6/d13/da1/fa3 x:0 0 0 2024-04-05T21:26:38.803 INFO:tasks.workunit.client.0.smithi121.stdout:1/353: sync 2024-04-05T21:26:38.803 INFO:tasks.workunit.client.0.smithi121.stdout:6/275: sync 2024-04-05T21:26:38.803 INFO:tasks.workunit.client.0.smithi121.stdout:7/290: sync 2024-04-05T21:26:38.804 INFO:tasks.workunit.client.0.smithi121.stdout:1/354: dread - d8/d10/d14/d1a/f72 zero size 2024-04-05T21:26:38.811 INFO:tasks.workunit.client.0.smithi121.stdout:1/355: symlink d8/d10/d14/d1a/d1b/d4c/l74 0 2024-04-05T21:26:38.811 INFO:tasks.workunit.client.0.smithi121.stdout:7/291: dwrite d0/d23/f2e [0,4194304] 0 2024-04-05T21:26:38.813 INFO:tasks.workunit.client.0.smithi121.stdout:6/276: dwrite d1/d2/d3/d5/f3c [4194304,4194304] 0 2024-04-05T21:26:38.816 INFO:tasks.workunit.client.0.smithi121.stdout:7/292: rmdir d0/d9/d22 39 2024-04-05T21:26:38.818 INFO:tasks.workunit.client.0.smithi121.stdout:6/277: symlink d1/d2/d3/d5/d36/l56 0 2024-04-05T21:26:38.820 INFO:tasks.workunit.client.0.smithi121.stdout:1/356: link d8/d10/d14/f1e d8/d10/d14/d27/f75 0 2024-04-05T21:26:38.825 INFO:tasks.workunit.client.0.smithi121.stdout:1/357: chown d8/d10/d14/d1a/d1b/d38/f42 931978 1 2024-04-05T21:26:38.825 INFO:tasks.workunit.client.0.smithi121.stdout:6/278: dwrite d1/d2/d3/f4e [0,4194304] 0 2024-04-05T21:26:38.828 INFO:tasks.workunit.client.1.smithi188.stdout:7/714: sync 2024-04-05T21:26:38.828 INFO:tasks.workunit.client.0.smithi121.stdout:7/293: dread d0/d32/f41 [0,4194304] 0 2024-04-05T21:26:38.838 INFO:tasks.workunit.client.1.smithi188.stdout:7/715: symlink d3/le2 0 2024-04-05T21:26:38.848 INFO:tasks.workunit.client.1.smithi188.stdout:7/716: dwrite d3/d4/db/d2e/d37/fba [0,4194304] 0 2024-04-05T21:26:38.850 INFO:tasks.workunit.client.1.smithi188.stdout:5/520: rename d6/fe to d6/d82/d40/fb6 0 2024-04-05T21:26:38.856 INFO:tasks.workunit.client.1.smithi188.stdout:9/519: rename dd/d11/d22/f9c to dd/d11/d1f/d26/d55/d98/fa9 0 2024-04-05T21:26:38.857 INFO:tasks.workunit.client.1.smithi188.stdout:5/521: dread d6/d17/d39/fb3 [0,4194304] 0 2024-04-05T21:26:38.858 INFO:tasks.workunit.client.1.smithi188.stdout:7/717: mknod d3/d4/db/d55/ce3 0 2024-04-05T21:26:38.859 INFO:tasks.workunit.client.1.smithi188.stdout:9/520: chown dd/d3d/f7a 9315 1 2024-04-05T21:26:38.866 INFO:tasks.workunit.client.1.smithi188.stdout:5/522: truncate d6/dad/f7b 561574 0 2024-04-05T21:26:38.867 INFO:tasks.workunit.client.1.smithi188.stdout:9/521: symlink dd/d49/d6b/d40/laa 0 2024-04-05T21:26:38.867 INFO:tasks.workunit.client.1.smithi188.stdout:0/446: dread d4/d8/db/d42/d38/d77/f86 [0,4194304] 0 2024-04-05T21:26:38.868 INFO:tasks.workunit.client.1.smithi188.stdout:7/718: unlink d3/d2c/d5e/d8e/faf 0 2024-04-05T21:26:38.869 INFO:tasks.workunit.client.1.smithi188.stdout:1/582: dwrite d1/dc/d16/d46/d88/d13/f18 [0,4194304] 0 2024-04-05T21:26:38.871 INFO:tasks.workunit.client.0.smithi121.stdout:5/401: dread dd/de/d19/d13/f5d [0,4194304] 0 2024-04-05T21:26:38.871 INFO:tasks.workunit.client.1.smithi188.stdout:6/622: dwrite d6/fa [0,4194304] 0 2024-04-05T21:26:38.872 INFO:tasks.workunit.client.0.smithi121.stdout:5/402: write dd/de/d39/d3b/d50/f67 [1744918,4025] 0 2024-04-05T21:26:38.873 INFO:tasks.workunit.client.0.smithi121.stdout:5/403: write dd/de/d19/d13/f35 [1884160,112918] 0 2024-04-05T21:26:38.874 INFO:tasks.workunit.client.1.smithi188.stdout:0/447: write d4/d8/fa [894439,63454] 0 2024-04-05T21:26:38.877 INFO:tasks.workunit.client.1.smithi188.stdout:9/522: symlink dd/d11/d1f/d26/lab 0 2024-04-05T21:26:38.881 INFO:tasks.workunit.client.1.smithi188.stdout:5/523: creat d6/fb7 x:0 0 0 2024-04-05T21:26:38.883 INFO:tasks.workunit.client.1.smithi188.stdout:2/598: creat d5/db/dd/d36/d3d/d46/fc5 x:0 0 0 2024-04-05T21:26:38.885 INFO:tasks.workunit.client.1.smithi188.stdout:2/599: dread - d5/db/dd/d36/d3d/d46/d9f/fa0 zero size 2024-04-05T21:26:38.885 INFO:tasks.workunit.client.1.smithi188.stdout:9/523: creat dd/d11/d73/d8d/fac x:0 0 0 2024-04-05T21:26:38.885 INFO:tasks.workunit.client.1.smithi188.stdout:4/787: write d0/d14/d15/d1b/f40 [2648399,53960] 0 2024-04-05T21:26:38.886 INFO:tasks.workunit.client.1.smithi188.stdout:9/524: truncate f1 4906678 0 2024-04-05T21:26:38.887 INFO:tasks.workunit.client.1.smithi188.stdout:4/788: chown d0/fbf 1607 1 2024-04-05T21:26:38.887 INFO:tasks.workunit.client.1.smithi188.stdout:6/623: creat d6/d13/d18/d19/d2c/d3f/d77/fc0 x:0 0 0 2024-04-05T21:26:38.889 INFO:tasks.workunit.client.1.smithi188.stdout:1/583: getdents d1/dc/d16/d2d 0 2024-04-05T21:26:38.891 INFO:tasks.workunit.client.1.smithi188.stdout:0/448: creat d4/d8/db/d42/da2/fa4 x:0 0 0 2024-04-05T21:26:38.892 INFO:tasks.workunit.client.1.smithi188.stdout:2/600: rmdir d5/db/dd/d36/d3c/d48 39 2024-04-05T21:26:38.892 INFO:tasks.workunit.client.1.smithi188.stdout:0/449: fdatasync d4/d6/d13/f14 0 2024-04-05T21:26:38.895 INFO:tasks.workunit.client.1.smithi188.stdout:9/525: dread - dd/d3d/d95/f9e zero size 2024-04-05T21:26:38.895 INFO:tasks.workunit.client.1.smithi188.stdout:6/624: dread - d6/d28/f58 zero size 2024-04-05T21:26:38.895 INFO:tasks.workunit.client.1.smithi188.stdout:0/450: write d4/d8/db/d21/d3c/f74 [242062,84220] 0 2024-04-05T21:26:38.898 INFO:tasks.workunit.client.0.smithi121.stdout:1/358: rename d8/d10/d14/d27/d2d/d3b to d8/d10/d14/d1a/d1b/d4c/d76 0 2024-04-05T21:26:38.898 INFO:tasks.workunit.client.1.smithi188.stdout:0/451: truncate d4/d8/db/d42/d38/f72 605010 0 2024-04-05T21:26:38.902 INFO:tasks.workunit.client.1.smithi188.stdout:0/452: read d4/d8/db/d42/d38/f54 [6456232,124181] 0 2024-04-05T21:26:38.902 INFO:tasks.workunit.client.0.smithi121.stdout:5/404: creat dd/de/d19/d1c/d74/f81 x:0 0 0 2024-04-05T21:26:38.903 INFO:tasks.workunit.client.1.smithi188.stdout:9/526: creat dd/d11/d22/d7e/fad x:0 0 0 2024-04-05T21:26:38.903 INFO:tasks.workunit.client.1.smithi188.stdout:4/789: dread d0/d14/d15/d78/db1/d6a/f100 [0,4194304] 0 2024-04-05T21:26:38.904 INFO:tasks.workunit.client.1.smithi188.stdout:0/453: fdatasync d4/d6/d61/f8f 0 2024-04-05T21:26:38.905 INFO:tasks.workunit.client.1.smithi188.stdout:5/524: link d6/d17/d39/d8e/f8f d6/d17/fb8 0 2024-04-05T21:26:38.906 INFO:tasks.workunit.client.1.smithi188.stdout:5/525: fsync d6/d15/d5c/f9e 0 2024-04-05T21:26:38.907 INFO:tasks.workunit.client.0.smithi121.stdout:5/405: mkdir dd/de/d39/d3b/d50/d82 0 2024-04-05T21:26:38.907 INFO:tasks.workunit.client.1.smithi188.stdout:0/454: write d4/d8/d10/f1e [1704601,58872] 0 2024-04-05T21:26:38.907 INFO:tasks.workunit.client.0.smithi121.stdout:5/406: dread - dd/de/d19/d1c/f48 zero size 2024-04-05T21:26:38.909 INFO:tasks.workunit.client.1.smithi188.stdout:0/455: write d4/d8/d10/f67 [3717035,85087] 0 2024-04-05T21:26:38.910 INFO:tasks.workunit.client.0.smithi121.stdout:1/359: write d8/d10/d14/d1a/d1b/d38/d4b/f6b [2912672,30281] 0 2024-04-05T21:26:38.913 INFO:tasks.workunit.client.0.smithi121.stdout:1/360: dread - d8/d10/d14/f37 zero size 2024-04-05T21:26:38.914 INFO:tasks.workunit.client.0.smithi121.stdout:5/407: link f6 dd/de/d19/d3c/f83 0 2024-04-05T21:26:38.914 INFO:tasks.workunit.client.0.smithi121.stdout:1/361: mkdir d8/d10/d14/d27/d57/d77 0 2024-04-05T21:26:38.914 INFO:tasks.workunit.client.0.smithi121.stdout:5/408: read - dd/de/d39/d3b/d50/f5b zero size 2024-04-05T21:26:38.915 INFO:tasks.workunit.client.0.smithi121.stdout:5/409: stat dd/de/d19/d38/d56 0 2024-04-05T21:26:38.915 INFO:tasks.workunit.client.0.smithi121.stdout:1/362: chown d8/d10/d14/d1a/d1b/d4c/f4d 4 1 2024-04-05T21:26:38.915 INFO:tasks.workunit.client.1.smithi188.stdout:6/625: fdatasync d6/d13/d18/d3c/f5d 0 2024-04-05T21:26:38.915 INFO:tasks.workunit.client.1.smithi188.stdout:6/626: readlink d6/d13/d18/d19/l2a 0 2024-04-05T21:26:38.915 INFO:tasks.workunit.client.1.smithi188.stdout:9/527: mknod dd/d49/d6b/d40/d59/cae 0 2024-04-05T21:26:38.915 INFO:tasks.workunit.client.1.smithi188.stdout:4/790: unlink d0/d14/l94 0 2024-04-05T21:26:38.917 INFO:tasks.workunit.client.0.smithi121.stdout:1/363: write d8/d10/f26 [5054419,31305] 0 2024-04-05T21:26:38.918 INFO:tasks.workunit.client.0.smithi121.stdout:5/410: link dd/de/f34 dd/de/d19/d38/d45/f84 0 2024-04-05T21:26:38.918 INFO:tasks.workunit.client.1.smithi188.stdout:1/584: link d1/dc/d16/d46/d88/d3f/f40 d1/fbc 0 2024-04-05T21:26:38.921 INFO:tasks.workunit.client.0.smithi121.stdout:5/411: symlink dd/de/d19/d38/d45/l85 0 2024-04-05T21:26:38.922 INFO:tasks.workunit.client.1.smithi188.stdout:5/526: dread d6/f53 [0,4194304] 0 2024-04-05T21:26:38.922 INFO:tasks.workunit.client.1.smithi188.stdout:2/601: creat d5/db/dd/fc6 x:0 0 0 2024-04-05T21:26:38.923 INFO:tasks.workunit.client.0.smithi121.stdout:5/412: stat dd/de/d39/d3b/d50/c61 0 2024-04-05T21:26:38.927 INFO:tasks.workunit.client.1.smithi188.stdout:9/528: rename dd/d11/d1f/d26/f19 to dd/d11/d1f/d25/d51/faf 0 2024-04-05T21:26:38.928 INFO:tasks.workunit.client.1.smithi188.stdout:1/585: dwrite d1/dc/d16/d21/d27/f29 [8388608,4194304] 0 2024-04-05T21:26:38.929 INFO:tasks.workunit.client.0.smithi121.stdout:1/364: dwrite d8/d10/d14/d27/f30 [0,4194304] 0 2024-04-05T21:26:38.930 INFO:tasks.workunit.client.1.smithi188.stdout:0/456: link d4/f5f d4/d8/db/d21/d3c/fa5 0 2024-04-05T21:26:38.932 INFO:tasks.workunit.client.0.smithi121.stdout:1/365: mknod d8/d10/d14/d27/d5c/c78 0 2024-04-05T21:26:38.933 INFO:tasks.workunit.client.1.smithi188.stdout:5/527: mknod d6/d15/d5c/d7a/d44/d98/cb9 0 2024-04-05T21:26:38.937 INFO:tasks.workunit.client.1.smithi188.stdout:9/529: fsync dd/d3d/f61 0 2024-04-05T21:26:38.941 INFO:tasks.workunit.client.0.smithi121.stdout:1/366: dwrite d8/d10/d14/d1a/d1b/d4c/d76/f3d [0,4194304] 0 2024-04-05T21:26:38.943 INFO:tasks.workunit.client.1.smithi188.stdout:4/791: rename d0/d14/d15/d1b/l3c to d0/d14/df6/l114 0 2024-04-05T21:26:38.945 INFO:tasks.workunit.client.1.smithi188.stdout:1/586: creat d1/dc/d16/d2d/da8/fbd x:0 0 0 2024-04-05T21:26:38.948 INFO:tasks.workunit.client.1.smithi188.stdout:1/587: chown d1/dc/d16/d21/d27/d49/c87 7822 1 2024-04-05T21:26:38.948 INFO:tasks.workunit.client.1.smithi188.stdout:2/602: dread d5/db/dd/f15 [0,4194304] 0 2024-04-05T21:26:38.950 INFO:tasks.workunit.client.1.smithi188.stdout:1/588: chown d1/dc/d16/f33 97 1 2024-04-05T21:26:38.951 INFO:tasks.workunit.client.1.smithi188.stdout:9/530: rename dd/d11/c87 to dd/d11/d1f/d25/cb0 0 2024-04-05T21:26:38.954 INFO:tasks.workunit.client.0.smithi121.stdout:1/367: dwrite d8/d10/d14/d1a/d1b/d44/f48 [0,4194304] 0 2024-04-05T21:26:38.954 INFO:tasks.workunit.client.1.smithi188.stdout:9/531: read dd/d11/d1f/d25/f50 [3050411,28517] 0 2024-04-05T21:26:38.956 INFO:tasks.workunit.client.1.smithi188.stdout:4/792: dwrite d0/d2/d17/d48/f57 [4194304,4194304] 0 2024-04-05T21:26:38.957 INFO:tasks.workunit.client.0.smithi121.stdout:1/368: truncate d8/d10/d14/d1a/d1b/d4c/f4d 1369034 0 2024-04-05T21:26:38.958 INFO:tasks.workunit.client.0.smithi121.stdout:1/369: write d8/d10/d14/d1a/d1b/d4c/f63 [923216,27627] 0 2024-04-05T21:26:38.959 INFO:tasks.workunit.client.1.smithi188.stdout:1/589: rmdir d1/dc/d16 39 2024-04-05T21:26:38.959 INFO:tasks.workunit.client.1.smithi188.stdout:9/532: readlink dd/d11/d1f/d26/d55/l64 0 2024-04-05T21:26:38.962 INFO:tasks.workunit.client.0.smithi121.stdout:1/370: mkdir d8/d10/d14/d1a/d1b/d24/d79 0 2024-04-05T21:26:38.964 INFO:tasks.workunit.client.1.smithi188.stdout:2/603: dwrite d5/f2f [4194304,4194304] 0 2024-04-05T21:26:38.965 INFO:tasks.workunit.client.1.smithi188.stdout:6/627: write d6/d13/d18/d19/f27 [3509319,101196] 0 2024-04-05T21:26:38.968 INFO:tasks.workunit.client.1.smithi188.stdout:9/533: dread - dd/d3d/f70 zero size 2024-04-05T21:26:38.971 INFO:tasks.workunit.client.1.smithi188.stdout:6/628: dread - d6/d13/d18/d19/d61/f87 zero size 2024-04-05T21:26:38.973 INFO:tasks.workunit.client.1.smithi188.stdout:2/604: write d5/db/dd/d36/d3d/d46/d5d/f5e [72108,119711] 0 2024-04-05T21:26:38.974 INFO:tasks.workunit.client.1.smithi188.stdout:6/629: chown d6/d13/d18/d19/d2c/d3f/l8a 12 1 2024-04-05T21:26:38.976 INFO:tasks.workunit.client.1.smithi188.stdout:0/457: truncate d4/d8/f26 853790 0 2024-04-05T21:26:38.977 INFO:tasks.workunit.client.1.smithi188.stdout:4/793: dwrite d0/d14/d15/d53/fac [0,4194304] 0 2024-04-05T21:26:38.982 INFO:tasks.workunit.client.1.smithi188.stdout:2/605: dread d5/db/dd/f15 [0,4194304] 0 2024-04-05T21:26:38.983 INFO:tasks.workunit.client.1.smithi188.stdout:1/590: truncate d1/dc/d16/d21/fa0 2584902 0 2024-04-05T21:26:38.986 INFO:tasks.workunit.client.1.smithi188.stdout:1/591: truncate d1/dc/d16/d46/d95/fb5 360325 0 2024-04-05T21:26:38.988 INFO:tasks.workunit.client.1.smithi188.stdout:4/794: dread d0/d14/d15/d78/fae [0,4194304] 0 2024-04-05T21:26:38.988 INFO:tasks.workunit.client.1.smithi188.stdout:0/458: creat d4/d6/d13/d23/fa6 x:0 0 0 2024-04-05T21:26:38.988 INFO:tasks.workunit.client.1.smithi188.stdout:2/606: symlink d5/db/dd/d36/d3c/d48/d78/lc7 0 2024-04-05T21:26:38.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:38 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:26:38.990 INFO:tasks.workunit.client.1.smithi188.stdout:6/630: getdents d6/d13/d18/d3c 0 2024-04-05T21:26:38.993 INFO:tasks.workunit.client.1.smithi188.stdout:2/607: write d5/db/dd/d36/d3d/d46/d5d/f5e [4137380,26032] 0 2024-04-05T21:26:38.994 INFO:tasks.workunit.client.1.smithi188.stdout:6/631: fsync d6/d7/f16 0 2024-04-05T21:26:38.996 INFO:tasks.workunit.client.1.smithi188.stdout:1/592: truncate d1/dc/d16/d46/d88/d86/fab 2824908 0 2024-04-05T21:26:38.998 INFO:tasks.workunit.client.1.smithi188.stdout:2/608: stat d5/db/dd/d36/d3d/d46/d66 0 2024-04-05T21:26:39.001 INFO:tasks.workunit.client.1.smithi188.stdout:1/593: creat d1/db2/fbe x:0 0 0 2024-04-05T21:26:39.002 INFO:tasks.workunit.client.1.smithi188.stdout:6/632: symlink d6/d13/d48/d49/lc1 0 2024-04-05T21:26:39.003 INFO:tasks.workunit.client.1.smithi188.stdout:4/795: dwrite d0/d2/d52/dcc/fea [0,4194304] 0 2024-04-05T21:26:39.004 INFO:tasks.workunit.client.1.smithi188.stdout:0/459: truncate d4/d6/f12 3521452 0 2024-04-05T21:26:39.013 INFO:tasks.workunit.client.1.smithi188.stdout:9/534: dread dd/d11/d1f/d25/d8a/f39 [0,4194304] 0 2024-04-05T21:26:39.014 INFO:tasks.workunit.client.1.smithi188.stdout:4/796: dread d0/d2/d2a/d34/fa1 [0,4194304] 0 2024-04-05T21:26:39.017 INFO:tasks.workunit.client.1.smithi188.stdout:1/594: dwrite d1/dc/d16/d46/d88/d13/f18 [4194304,4194304] 0 2024-04-05T21:26:39.018 INFO:tasks.workunit.client.1.smithi188.stdout:9/535: dread - dd/d11/d73/d85/f93 zero size 2024-04-05T21:26:39.018 INFO:tasks.workunit.client.1.smithi188.stdout:2/609: dwrite d5/db/dd/da4/fc2 [0,4194304] 0 2024-04-05T21:26:39.070 INFO:tasks.workunit.client.1.smithi188.stdout:4/797: dread d0/d2/d17/f4e [0,4194304] 0 2024-04-05T21:26:39.071 INFO:tasks.workunit.client.1.smithi188.stdout:0/460: link d4/d8/db/d21/d3c/f74 d4/d8/db/d42/da2/fa7 0 2024-04-05T21:26:39.071 INFO:tasks.workunit.client.1.smithi188.stdout:9/536: mknod dd/cb1 0 2024-04-05T21:26:39.072 INFO:tasks.workunit.client.1.smithi188.stdout:2/610: mkdir d5/db/dd/d24/dc8 0 2024-04-05T21:26:39.077 INFO:tasks.workunit.client.1.smithi188.stdout:0/461: mknod d4/d8/d10/ca8 0 2024-04-05T21:26:39.077 INFO:tasks.workunit.client.1.smithi188.stdout:9/537: mkdir dd/d11/d1f/db2 0 2024-04-05T21:26:39.079 INFO:tasks.workunit.client.1.smithi188.stdout:0/462: creat d4/d8/db/d21/d3c/fa9 x:0 0 0 2024-04-05T21:26:39.080 INFO:tasks.workunit.client.1.smithi188.stdout:0/463: stat d4/d8/db/d42/d71 0 2024-04-05T21:26:39.081 INFO:tasks.workunit.client.1.smithi188.stdout:4/798: getdents d0/d2/d2a/d71 0 2024-04-05T21:26:39.081 INFO:tasks.workunit.client.1.smithi188.stdout:0/464: chown d4/d6/d46/f59 0 1 2024-04-05T21:26:39.082 INFO:tasks.workunit.client.1.smithi188.stdout:4/799: chown d0/d2/d52/dcc/le3 0 1 2024-04-05T21:26:39.082 INFO:tasks.workunit.client.1.smithi188.stdout:0/465: read d4/d6/d46/f59 [59370,85644] 0 2024-04-05T21:26:39.083 INFO:tasks.workunit.client.1.smithi188.stdout:9/538: creat dd/d11/d1f/d25/d8a/fb3 x:0 0 0 2024-04-05T21:26:39.087 INFO:tasks.workunit.client.1.smithi188.stdout:9/539: getdents dd/d3d/d89 0 2024-04-05T21:26:39.087 INFO:tasks.workunit.client.1.smithi188.stdout:0/466: dread d4/d6/f15 [0,4194304] 0 2024-04-05T21:26:39.088 INFO:tasks.workunit.client.1.smithi188.stdout:2/611: truncate d5/db/dd/d36/d3d/d46/fc0 6134258 0 2024-04-05T21:26:39.090 INFO:tasks.workunit.client.1.smithi188.stdout:0/467: write d4/d6/d61/f93 [973857,56612] 0 2024-04-05T21:26:39.093 INFO:tasks.workunit.client.0.smithi121.stdout:4/348: sync 2024-04-05T21:26:39.093 INFO:tasks.workunit.client.0.smithi121.stdout:9/225: sync 2024-04-05T21:26:39.094 INFO:tasks.workunit.client.1.smithi188.stdout:2/612: creat d5/db/dd/d36/d3d/d46/d66/fc9 x:0 0 0 2024-04-05T21:26:39.094 INFO:tasks.workunit.client.1.smithi188.stdout:9/540: write dd/d49/f5c [548207,65507] 0 2024-04-05T21:26:39.096 INFO:tasks.workunit.client.1.smithi188.stdout:4/800: dwrite d0/f5 [8388608,4194304] 0 2024-04-05T21:26:39.097 INFO:tasks.workunit.client.0.smithi121.stdout:9/226: rmdir d6/df/d44 0 2024-04-05T21:26:39.100 INFO:tasks.workunit.client.0.smithi121.stdout:9/227: rename d6/c9 to d6/df/c47 0 2024-04-05T21:26:39.104 INFO:tasks.workunit.client.0.smithi121.stdout:4/349: dwrite d5/d11/d2c/f36 [0,4194304] 0 2024-04-05T21:26:39.107 INFO:tasks.workunit.client.1.smithi188.stdout:0/468: dwrite d4/d6/d13/da1/fa3 [0,4194304] 0 2024-04-05T21:26:39.110 INFO:tasks.workunit.client.1.smithi188.stdout:2/613: dread d5/db/dd/d36/d3c/d48/d7a/d7c/f86 [0,4194304] 0 2024-04-05T21:26:39.112 INFO:tasks.workunit.client.0.smithi121.stdout:4/350: dwrite d5/d11/d2c/f65 [0,4194304] 0 2024-04-05T21:26:39.112 INFO:tasks.workunit.client.1.smithi188.stdout:0/469: truncate d4/d6/f1f 4389602 0 2024-04-05T21:26:39.114 INFO:tasks.workunit.client.0.smithi121.stdout:4/351: stat d5/d11/d28/d3b/c3e 0 2024-04-05T21:26:39.115 INFO:tasks.workunit.client.0.smithi121.stdout:4/352: dread - d5/d11/d28/f44 zero size 2024-04-05T21:26:39.116 INFO:tasks.workunit.client.1.smithi188.stdout:2/614: chown d5/db/dd/d24/f52 15469 1 2024-04-05T21:26:39.117 INFO:tasks.workunit.client.1.smithi188.stdout:9/541: dwrite dd/d11/d1f/f7b [0,4194304] 0 2024-04-05T21:26:39.118 INFO:tasks.workunit.client.1.smithi188.stdout:4/801: dwrite d0/d2/d2a/d34/d6b/fa9 [0,4194304] 0 2024-04-05T21:26:39.119 INFO:tasks.workunit.client.0.smithi121.stdout:4/353: creat d5/d11/d28/d73/f76 x:0 0 0 2024-04-05T21:26:39.120 INFO:tasks.workunit.client.1.smithi188.stdout:2/615: stat d5/db/dd/d36/d3d/d46/f7b 0 2024-04-05T21:26:39.122 INFO:tasks.workunit.client.0.smithi121.stdout:4/354: fdatasync d5/d11/f17 0 2024-04-05T21:26:39.124 INFO:tasks.workunit.client.0.smithi121.stdout:4/355: creat d5/d11/d28/d3b/f77 x:0 0 0 2024-04-05T21:26:39.125 INFO:tasks.workunit.client.1.smithi188.stdout:0/470: rename d4/d8/db/d21/f44 to d4/d6/faa 0 2024-04-05T21:26:39.130 INFO:tasks.workunit.client.1.smithi188.stdout:9/542: dwrite f1 [0,4194304] 0 2024-04-05T21:26:39.131 INFO:tasks.workunit.client.1.smithi188.stdout:0/471: fdatasync d4/d8/db/f88 0 2024-04-05T21:26:39.139 INFO:tasks.workunit.client.1.smithi188.stdout:2/616: link d5/db/dd/d36/d3c/d48/d85/f94 d5/db/dd/d73/fca 0 2024-04-05T21:26:39.142 INFO:tasks.workunit.client.1.smithi188.stdout:0/472: dwrite d4/d8/db/d42/d38/f72 [0,4194304] 0 2024-04-05T21:26:39.146 INFO:tasks.workunit.client.1.smithi188.stdout:2/617: getdents d5 0 2024-04-05T21:26:39.146 INFO:tasks.workunit.client.1.smithi188.stdout:0/473: fsync d4/d8/db/d21/d4a/d73/f94 0 2024-04-05T21:26:39.146 INFO:tasks.workunit.client.1.smithi188.stdout:2/618: chown d5/f9b 0 1 2024-04-05T21:26:39.147 INFO:tasks.workunit.client.1.smithi188.stdout:0/474: readlink d4/d6/l4f 0 2024-04-05T21:26:39.150 INFO:tasks.workunit.client.1.smithi188.stdout:2/619: mkdir d5/db/dd/d36/d3d/d46/d66/d8f/dcb 0 2024-04-05T21:26:39.153 INFO:tasks.workunit.client.1.smithi188.stdout:2/620: rename d5/c8 to d5/db/dd/d36/d3c/d63/d75/ccc 0 2024-04-05T21:26:39.154 INFO:tasks.workunit.client.1.smithi188.stdout:2/621: dread - d5/db/dd/d36/d3c/d48/d78/f83 zero size 2024-04-05T21:26:39.166 INFO:tasks.workunit.client.0.smithi121.stdout:4/356: truncate d5/d11/d2c/f36 1771366 0 2024-04-05T21:26:39.167 INFO:tasks.workunit.client.1.smithi188.stdout:9/543: write dd/d11/d1f/d25/d8a/f77 [686455,73866] 0 2024-04-05T21:26:39.170 INFO:tasks.workunit.client.0.smithi121.stdout:4/357: creat d5/d11/d26/d27/d29/d58/f78 x:0 0 0 2024-04-05T21:26:39.174 INFO:tasks.workunit.client.0.smithi121.stdout:4/358: write d5/d11/d26/d49/f72 [4939522,112207] 0 2024-04-05T21:26:39.174 INFO:tasks.workunit.client.0.smithi121.stdout:4/359: chown d5/d11/d26/l51 161747 1 2024-04-05T21:26:39.175 INFO:tasks.workunit.client.0.smithi121.stdout:4/360: rename d5/d11/d28/d3b/l54 to d5/d11/d20/l79 0 2024-04-05T21:26:39.175 INFO:tasks.workunit.client.1.smithi188.stdout:4/802: dwrite d0/d14/d15/d78/fae [0,4194304] 0 2024-04-05T21:26:39.178 INFO:tasks.workunit.client.1.smithi188.stdout:0/475: dwrite d4/d8/db/f76 [0,4194304] 0 2024-04-05T21:26:39.179 INFO:tasks.workunit.client.1.smithi188.stdout:2/622: dread d5/db/dd/f54 [0,4194304] 0 2024-04-05T21:26:39.180 INFO:tasks.workunit.client.1.smithi188.stdout:9/544: link dd/d3d/l3e dd/d11/d1f/d26/lb4 0 2024-04-05T21:26:39.181 INFO:tasks.workunit.client.1.smithi188.stdout:0/476: truncate d4/d6/d46/f59 1006330 0 2024-04-05T21:26:39.182 INFO:tasks.workunit.client.1.smithi188.stdout:4/803: chown d0/d14/d15/d78/db1/ld2 14791450 1 2024-04-05T21:26:39.183 INFO:tasks.workunit.client.0.smithi121.stdout:4/361: write d5/f3f [1720516,18399] 0 2024-04-05T21:26:39.184 INFO:tasks.workunit.client.0.smithi121.stdout:4/362: read - d5/d11/d26/d27/d29/d58/f6e zero size 2024-04-05T21:26:39.185 INFO:tasks.workunit.client.1.smithi188.stdout:9/545: symlink dd/d49/lb5 0 2024-04-05T21:26:39.188 INFO:tasks.workunit.client.1.smithi188.stdout:2/623: write d5/f22 [170171,69494] 0 2024-04-05T21:26:39.192 INFO:tasks.workunit.client.1.smithi188.stdout:2/624: rmdir d5/db/dd/d36/d3d/d46/d9f 39 2024-04-05T21:26:39.198 INFO:tasks.workunit.client.1.smithi188.stdout:0/477: dwrite d4/d8/db/d21/f90 [0,4194304] 0 2024-04-05T21:26:39.199 INFO:tasks.workunit.client.0.smithi121.stdout:3/226: sync 2024-04-05T21:26:39.199 INFO:tasks.workunit.client.0.smithi121.stdout:0/390: sync 2024-04-05T21:26:39.199 INFO:tasks.workunit.client.0.smithi121.stdout:6/279: sync 2024-04-05T21:26:39.199 INFO:tasks.workunit.client.0.smithi121.stdout:8/274: sync 2024-04-05T21:26:39.200 INFO:tasks.workunit.client.0.smithi121.stdout:2/263: sync 2024-04-05T21:26:39.200 INFO:tasks.workunit.client.0.smithi121.stdout:7/294: sync 2024-04-05T21:26:39.200 INFO:tasks.workunit.client.0.smithi121.stdout:5/413: sync 2024-04-05T21:26:39.200 INFO:tasks.workunit.client.0.smithi121.stdout:4/363: dread d5/d11/d26/d47/f66 [0,4194304] 0 2024-04-05T21:26:39.202 INFO:tasks.workunit.client.0.smithi121.stdout:2/264: chown d2/d9/d17/f42 746628 1 2024-04-05T21:26:39.202 INFO:tasks.workunit.client.0.smithi121.stdout:0/391: chown d7/f3a 0 1 2024-04-05T21:26:39.203 INFO:tasks.workunit.client.0.smithi121.stdout:8/275: creat d12/d50/d59/f60 x:0 0 0 2024-04-05T21:26:39.204 INFO:tasks.workunit.client.1.smithi188.stdout:2/625: write d5/db/dd/d36/d3c/f4a [3971838,1403] 0 2024-04-05T21:26:39.206 INFO:tasks.workunit.client.0.smithi121.stdout:2/265: chown d2/d9/d17/f42 31322945 1 2024-04-05T21:26:39.207 INFO:tasks.workunit.client.0.smithi121.stdout:8/276: fdatasync fa 0 2024-04-05T21:26:39.207 INFO:tasks.workunit.client.1.smithi188.stdout:0/478: dwrite d4/d8/db/d42/d38/f49 [0,4194304] 0 2024-04-05T21:26:39.212 INFO:tasks.workunit.client.0.smithi121.stdout:2/266: mkdir d2/d9/d1c/d4b 0 2024-04-05T21:26:39.212 INFO:tasks.workunit.client.1.smithi188.stdout:2/626: rename d5/db/dd/d36/d3d to d5/db/dd/d67/dcd 0 2024-04-05T21:26:39.213 INFO:tasks.workunit.client.0.smithi121.stdout:0/392: getdents d7/d71/d45/d46/d53 0 2024-04-05T21:26:39.213 INFO:tasks.workunit.client.0.smithi121.stdout:8/277: rmdir d12/d13 39 2024-04-05T21:26:39.214 INFO:tasks.workunit.client.1.smithi188.stdout:0/479: mknod d4/d6/d61/cab 0 2024-04-05T21:26:39.214 INFO:tasks.workunit.client.1.smithi188.stdout:9/546: dread fc [0,4194304] 0 2024-04-05T21:26:39.214 INFO:tasks.workunit.client.0.smithi121.stdout:7/295: dwrite d0/d23/f33 [0,4194304] 0 2024-04-05T21:26:39.215 INFO:tasks.workunit.client.0.smithi121.stdout:6/280: getdents d1/d2/d3/d4 0 2024-04-05T21:26:39.218 INFO:tasks.workunit.client.0.smithi121.stdout:2/267: mkdir d2/d9/d38/d2d/d4c 0 2024-04-05T21:26:39.219 INFO:tasks.workunit.client.0.smithi121.stdout:3/227: truncate d0/d3/d24/d3d/f21 3512354 0 2024-04-05T21:26:39.219 INFO:tasks.workunit.client.0.smithi121.stdout:2/268: stat d2/l4 0 2024-04-05T21:26:39.220 INFO:tasks.workunit.client.1.smithi188.stdout:0/480: mkdir d4/d8/db/dac 0 2024-04-05T21:26:39.220 INFO:tasks.workunit.client.1.smithi188.stdout:9/547: truncate dd/d3d/f7a 165956 0 2024-04-05T21:26:39.221 INFO:tasks.workunit.client.1.smithi188.stdout:9/548: readlink dd/l5d 0 2024-04-05T21:26:39.221 INFO:tasks.workunit.client.1.smithi188.stdout:0/481: write d4/d6/f1f [4248078,23564] 0 2024-04-05T21:26:39.222 INFO:tasks.workunit.client.0.smithi121.stdout:4/364: dwrite d5/d11/d28/d3b/f59 [0,4194304] 0 2024-04-05T21:26:39.222 INFO:tasks.workunit.client.0.smithi121.stdout:7/296: rmdir d0/d9/d22/d28/d56 39 2024-04-05T21:26:39.223 INFO:tasks.workunit.client.1.smithi188.stdout:2/627: dwrite d5/db/dd/d67/dcd/d46/d66/fb3 [0,4194304] 0 2024-04-05T21:26:39.225 INFO:tasks.workunit.client.0.smithi121.stdout:8/278: rename d12/d30 to d12/d50/d61 0 2024-04-05T21:26:39.226 INFO:tasks.workunit.client.0.smithi121.stdout:0/393: dread d7/f3a [0,4194304] 0 2024-04-05T21:26:39.231 INFO:tasks.workunit.client.0.smithi121.stdout:5/414: dwrite dd/de/d19/d3c/f83 [0,4194304] 0 2024-04-05T21:26:39.232 INFO:tasks.workunit.client.1.smithi188.stdout:9/549: rename dd/d11/d1f/d26/l5e to dd/d35/lb6 0 2024-04-05T21:26:39.232 INFO:tasks.workunit.client.0.smithi121.stdout:3/228: rename d0/d3/d24/d3d/c37 to d0/c4a 0 2024-04-05T21:26:39.234 INFO:tasks.workunit.client.0.smithi121.stdout:0/394: mknod d7/dc/c86 0 2024-04-05T21:26:39.240 INFO:tasks.workunit.client.1.smithi188.stdout:2/628: dwrite d5/db/dd/d67/dcd/d46/d9f/fa0 [0,4194304] 0 2024-04-05T21:26:39.243 INFO:tasks.workunit.client.0.smithi121.stdout:4/365: rename d5/d11/c61 to d5/d11/d26/d27/c7a 0 2024-04-05T21:26:39.243 INFO:tasks.workunit.client.0.smithi121.stdout:7/297: dwrite d0/d9/d22/d28/d3b/f4f [0,4194304] 0 2024-04-05T21:26:39.243 INFO:tasks.workunit.client.0.smithi121.stdout:5/415: symlink dd/de/d19/d3c/l86 0 2024-04-05T21:26:39.244 INFO:tasks.workunit.client.1.smithi188.stdout:9/550: read dd/d11/f3c [1008948,97493] 0 2024-04-05T21:26:39.245 INFO:tasks.workunit.client.0.smithi121.stdout:0/395: chown d7/d1c/f23 63559526 1 2024-04-05T21:26:39.246 INFO:tasks.workunit.client.1.smithi188.stdout:2/629: unlink d5/db/dd/cc4 0 2024-04-05T21:26:39.247 INFO:tasks.workunit.client.0.smithi121.stdout:8/279: dwrite d12/d13/d15/d1d/f4f [0,4194304] 0 2024-04-05T21:26:39.248 INFO:tasks.workunit.client.1.smithi188.stdout:9/551: mknod dd/d35/cb7 0 2024-04-05T21:26:39.249 INFO:tasks.workunit.client.1.smithi188.stdout:2/630: fsync d5/db/dd/d36/d3c/d63/d75/d87/f9c 0 2024-04-05T21:26:39.250 INFO:tasks.workunit.client.0.smithi121.stdout:0/396: fdatasync d7/d71/d45/f6a 0 2024-04-05T21:26:39.250 INFO:tasks.workunit.client.1.smithi188.stdout:2/631: write d5/fb7 [438172,40629] 0 2024-04-05T21:26:39.254 INFO:tasks.workunit.client.1.smithi188.stdout:2/632: mkdir d5/db/dd/d67/dcd/d46/d66/d8f/dcb/dce 0 2024-04-05T21:26:39.256 INFO:tasks.workunit.client.0.smithi121.stdout:5/416: unlink dd/de/d19/d1c/l72 0 2024-04-05T21:26:39.257 INFO:tasks.workunit.client.0.smithi121.stdout:0/397: creat d7/d71/d29/d2a/d2c/f87 x:0 0 0 2024-04-05T21:26:39.258 INFO:tasks.workunit.client.1.smithi188.stdout:2/633: fsync d5/db/dd/d67/dcd/d46/d66/fb3 0 2024-04-05T21:26:39.259 INFO:tasks.workunit.client.1.smithi188.stdout:2/634: chown d5/db/d35/d3b/d6b/l8d 227796931 1 2024-04-05T21:26:39.261 INFO:tasks.workunit.client.0.smithi121.stdout:5/417: creat dd/de/d19/d38/d56/f87 x:0 0 0 2024-04-05T21:26:39.261 INFO:tasks.workunit.client.0.smithi121.stdout:8/280: creat d12/d13/d56/d57/d5e/f62 x:0 0 0 2024-04-05T21:26:39.262 INFO:tasks.workunit.client.0.smithi121.stdout:5/418: write dd/de/d19/d38/d45/d68/f79 [923461,83458] 0 2024-04-05T21:26:39.264 INFO:tasks.workunit.client.0.smithi121.stdout:8/281: mkdir d12/d50/d59/d63 0 2024-04-05T21:26:39.267 INFO:tasks.workunit.client.0.smithi121.stdout:5/419: mkdir dd/de/d19/d1c/d7b/d88 0 2024-04-05T21:26:39.268 INFO:tasks.workunit.client.0.smithi121.stdout:6/281: dread d1/d2/f13 [0,4194304] 0 2024-04-05T21:26:39.269 INFO:tasks.workunit.client.0.smithi121.stdout:5/420: read - dd/de/d19/d1c/d36/d46/f64 zero size 2024-04-05T21:26:39.272 INFO:tasks.workunit.client.1.smithi188.stdout:2/635: dwrite d5/db/dd/d36/d3c/f4a [0,4194304] 0 2024-04-05T21:26:39.277 INFO:tasks.workunit.client.0.smithi121.stdout:6/282: dwrite d1/d2/dc/f4c [0,4194304] 0 2024-04-05T21:26:39.339 INFO:tasks.workunit.client.0.smithi121.stdout:9/228: sync 2024-04-05T21:26:39.339 INFO:tasks.workunit.client.0.smithi121.stdout:1/371: sync 2024-04-05T21:26:39.340 INFO:tasks.workunit.client.0.smithi121.stdout:5/421: dwrite dd/de/d19/f6b [0,4194304] 0 2024-04-05T21:26:39.345 INFO:tasks.workunit.client.0.smithi121.stdout:1/372: stat d8/d10/d14/d1a/d1b/d24 0 2024-04-05T21:26:39.346 INFO:tasks.workunit.client.0.smithi121.stdout:6/283: write d1/d2/d3/d5/f3c [6717473,24205] 0 2024-04-05T21:26:39.350 INFO:tasks.workunit.client.0.smithi121.stdout:6/284: fsync d1/d2/dc/f4c 0 2024-04-05T21:26:39.351 INFO:tasks.workunit.client.0.smithi121.stdout:1/373: write d8/d10/d14/d27/d2d/f41 [797038,62632] 0 2024-04-05T21:26:39.352 INFO:tasks.workunit.client.0.smithi121.stdout:3/229: dwrite d0/d3/d24/d3d/d1a/f33 [0,4194304] 0 2024-04-05T21:26:39.355 INFO:tasks.workunit.client.0.smithi121.stdout:7/298: dread d0/d23/d5b/f43 [0,4194304] 0 2024-04-05T21:26:39.359 INFO:tasks.workunit.client.0.smithi121.stdout:5/422: dwrite dd/de/d19/d13/f26 [0,4194304] 0 2024-04-05T21:26:39.359 INFO:tasks.workunit.client.0.smithi121.stdout:4/366: dread d5/fd [0,4194304] 0 2024-04-05T21:26:39.360 INFO:tasks.workunit.client.0.smithi121.stdout:1/374: write d8/d10/d14/d1a/d1b/d44/f48 [1365597,11192] 0 2024-04-05T21:26:39.363 INFO:tasks.workunit.client.0.smithi121.stdout:3/230: creat d0/d3/d24/d3f/f4b x:0 0 0 2024-04-05T21:26:39.367 INFO:tasks.workunit.client.0.smithi121.stdout:7/299: chown d0/d9/f50 878 1 2024-04-05T21:26:39.368 INFO:tasks.workunit.client.0.smithi121.stdout:6/285: dwrite d1/d2/f13 [4194304,4194304] 0 2024-04-05T21:26:39.369 INFO:tasks.workunit.client.0.smithi121.stdout:6/286: fsync d1/d2/d3/d5/d1f/f46 0 2024-04-05T21:26:39.369 INFO:tasks.workunit.client.0.smithi121.stdout:1/375: symlink d8/d10/d14/d1a/d1b/d24/l7a 0 2024-04-05T21:26:39.370 INFO:tasks.workunit.client.0.smithi121.stdout:3/231: readlink d0/d3/d24/d3d/l1c 0 2024-04-05T21:26:39.370 INFO:tasks.workunit.client.0.smithi121.stdout:5/423: truncate dd/de/d19/f37 1496542 0 2024-04-05T21:26:39.372 INFO:tasks.workunit.client.0.smithi121.stdout:4/367: fsync d5/d11/d26/d27/d64/f75 0 2024-04-05T21:26:39.373 INFO:tasks.workunit.client.0.smithi121.stdout:7/300: fsync d0/d9/d22/d3e/f3c 0 2024-04-05T21:26:39.373 INFO:tasks.workunit.client.0.smithi121.stdout:3/232: mknod d0/c4c 0 2024-04-05T21:26:39.374 INFO:tasks.workunit.client.0.smithi121.stdout:1/376: mknod d8/d10/d14/d1a/d1b/c7b 0 2024-04-05T21:26:39.375 INFO:tasks.workunit.client.0.smithi121.stdout:3/233: stat d0/d3/d24/f42 0 2024-04-05T21:26:39.375 INFO:tasks.workunit.client.0.smithi121.stdout:6/287: truncate d1/d2/d3/f4e 4717751 0 2024-04-05T21:26:39.379 INFO:tasks.workunit.client.0.smithi121.stdout:1/377: stat d8/d10/d14/d1a/c64 0 2024-04-05T21:26:39.379 INFO:tasks.workunit.client.0.smithi121.stdout:1/378: fsync d8/f1c 0 2024-04-05T21:26:39.379 INFO:tasks.workunit.client.0.smithi121.stdout:7/301: mkdir d0/d23/d5b/d52/d61 0 2024-04-05T21:26:39.379 INFO:tasks.workunit.client.0.smithi121.stdout:4/368: creat d5/d11/d26/d27/d29/d2e/f7b x:0 0 0 2024-04-05T21:26:39.381 INFO:tasks.workunit.client.0.smithi121.stdout:7/302: write d0/f8 [5832323,125800] 0 2024-04-05T21:26:39.382 INFO:tasks.workunit.client.0.smithi121.stdout:7/303: readlink d0/l17 0 2024-04-05T21:26:39.383 INFO:tasks.workunit.client.0.smithi121.stdout:7/304: chown d0/d9/c10 401 1 2024-04-05T21:26:39.385 INFO:tasks.workunit.client.0.smithi121.stdout:7/305: dread - d0/d9/f5e zero size 2024-04-05T21:26:39.387 INFO:tasks.workunit.client.0.smithi121.stdout:6/288: rename d1/d2/d3/d5/d1f/d2c/d41 to d1/d2/d3/d4/d21/d57 0 2024-04-05T21:26:39.389 INFO:tasks.workunit.client.0.smithi121.stdout:7/306: write d0/d9/d22/d28/f31 [1123626,36285] 0 2024-04-05T21:26:39.390 INFO:tasks.workunit.client.0.smithi121.stdout:1/379: link d8/d10/d14/d1a/d1b/d24/f49 d8/d10/d14/d1a/d1b/d4c/d76/f7c 0 2024-04-05T21:26:39.390 INFO:tasks.workunit.client.0.smithi121.stdout:5/424: dread dd/de/d19/d13/f35 [0,4194304] 0 2024-04-05T21:26:39.391 INFO:tasks.workunit.client.0.smithi121.stdout:4/369: dwrite d5/d11/d26/d49/f72 [4194304,4194304] 0 2024-04-05T21:26:39.391 INFO:tasks.workunit.client.0.smithi121.stdout:7/307: chown d0/d9 13 1 2024-04-05T21:26:39.392 INFO:tasks.workunit.client.0.smithi121.stdout:6/289: write d1/d2/dc/f4c [4652081,9399] 0 2024-04-05T21:26:39.394 INFO:tasks.workunit.client.0.smithi121.stdout:1/380: write d8/f1c [3345028,115452] 0 2024-04-05T21:26:39.397 INFO:tasks.workunit.client.0.smithi121.stdout:4/370: stat d5/d11/d26/d27/d29/d2e/f4d 0 2024-04-05T21:26:39.397 INFO:tasks.workunit.client.0.smithi121.stdout:1/381: stat d8/d10/d14/d1a/d1b/d38/d4b/f6d 0 2024-04-05T21:26:39.398 INFO:tasks.workunit.client.1.smithi188.stdout:8/545: sync 2024-04-05T21:26:39.398 INFO:tasks.workunit.client.1.smithi188.stdout:3/624: sync 2024-04-05T21:26:39.399 INFO:tasks.workunit.client.0.smithi121.stdout:6/290: creat d1/d2/d3/d5/f58 x:0 0 0 2024-04-05T21:26:39.400 INFO:tasks.workunit.client.0.smithi121.stdout:1/382: dread - d8/d10/d14/d1a/f71 zero size 2024-04-05T21:26:39.401 INFO:tasks.workunit.client.0.smithi121.stdout:4/371: chown d5/d11/d28/f13 40469289 1 2024-04-05T21:26:39.401 INFO:tasks.workunit.client.1.smithi188.stdout:3/625: creat d5/d7/d65/da2/fd5 x:0 0 0 2024-04-05T21:26:39.402 INFO:tasks.workunit.client.0.smithi121.stdout:5/425: unlink dd/de/d19/d13/d53/c6a 0 2024-04-05T21:26:39.403 INFO:tasks.workunit.client.0.smithi121.stdout:7/308: dwrite d0/d9/f20 [0,4194304] 0 2024-04-05T21:26:39.403 INFO:tasks.workunit.client.0.smithi121.stdout:1/383: readlink d8/d10/d14/d27/d2d/l61 0 2024-04-05T21:26:39.406 INFO:tasks.workunit.client.0.smithi121.stdout:1/384: dread - d8/d10/d14/f23 zero size 2024-04-05T21:26:39.408 INFO:tasks.workunit.client.1.smithi188.stdout:3/626: dwrite d5/df/d80/fc4 [0,4194304] 0 2024-04-05T21:26:39.409 INFO:tasks.workunit.client.1.smithi188.stdout:8/546: read d5/d21/d2a/d47/f49 [2242759,85599] 0 2024-04-05T21:26:39.411 INFO:tasks.workunit.client.0.smithi121.stdout:6/291: dwrite d1/d2/f13 [0,4194304] 0 2024-04-05T21:26:39.411 INFO:tasks.workunit.client.0.smithi121.stdout:7/309: read d0/d23/d5b/f3f [4434950,122640] 0 2024-04-05T21:26:39.411 INFO:tasks.workunit.client.1.smithi188.stdout:3/627: fsync d5/f21 0 2024-04-05T21:26:39.412 INFO:tasks.workunit.client.1.smithi188.stdout:8/547: symlink d5/d21/d40/lbd 0 2024-04-05T21:26:39.413 INFO:tasks.workunit.client.0.smithi121.stdout:4/372: dread d5/f1d [0,4194304] 0 2024-04-05T21:26:39.414 INFO:tasks.workunit.client.0.smithi121.stdout:5/426: rename dd/de/d19/d1c/l58 to dd/de/d19/d1c/d7b/l89 0 2024-04-05T21:26:39.414 INFO:tasks.workunit.client.1.smithi188.stdout:3/628: write d5/d35/f52 [2855748,55290] 0 2024-04-05T21:26:39.415 INFO:tasks.workunit.client.1.smithi188.stdout:8/548: fdatasync d5/d21/d2a/d47/f54 0 2024-04-05T21:26:39.415 INFO:tasks.workunit.client.1.smithi188.stdout:3/629: write d5/df/f33 [4524831,92503] 0 2024-04-05T21:26:39.417 INFO:tasks.workunit.client.1.smithi188.stdout:8/549: read d5/d21/d40/d43/f51 [2846348,10378] 0 2024-04-05T21:26:39.422 INFO:tasks.workunit.client.0.smithi121.stdout:7/310: write d0/d23/d5b/f3f [4816816,92647] 0 2024-04-05T21:26:39.424 INFO:tasks.workunit.client.1.smithi188.stdout:3/630: rename d5/dc/c16 to d5/d7/d31/cd6 0 2024-04-05T21:26:39.424 INFO:tasks.workunit.client.1.smithi188.stdout:8/550: rmdir d5/d21/d80 39 2024-04-05T21:26:39.425 INFO:tasks.workunit.client.0.smithi121.stdout:4/373: dread d5/d11/d26/d49/f72 [4194304,4194304] 0 2024-04-05T21:26:39.429 INFO:tasks.workunit.client.0.smithi121.stdout:7/311: link d0/d23/f54 d0/d23/d5a/f62 0 2024-04-05T21:26:39.430 INFO:tasks.workunit.client.0.smithi121.stdout:1/385: dwrite d8/d10/d14/d1a/f5a [0,4194304] 0 2024-04-05T21:26:39.430 INFO:tasks.workunit.client.1.smithi188.stdout:8/551: write d5/d21/d2a/d47/f8e [861660,2952] 0 2024-04-05T21:26:39.432 INFO:tasks.workunit.client.1.smithi188.stdout:3/631: dwrite d5/d7/d65/d5f/fb4 [0,4194304] 0 2024-04-05T21:26:39.433 INFO:tasks.workunit.client.0.smithi121.stdout:7/312: dread d0/f1f [0,4194304] 0 2024-04-05T21:26:39.433 INFO:tasks.workunit.client.0.smithi121.stdout:1/386: symlink d8/d10/d14/d27/d57/d77/l7d 0 2024-04-05T21:26:39.436 INFO:tasks.workunit.client.1.smithi188.stdout:8/552: mkdir d5/d21/d40/d43/d44/d5b/d99/d69/dbe 0 2024-04-05T21:26:39.439 INFO:tasks.workunit.client.1.smithi188.stdout:3/632: write d5/d7/d31/f45 [167091,1292] 0 2024-04-05T21:26:39.440 INFO:tasks.workunit.client.0.smithi121.stdout:7/313: dread d0/d9/d22/d28/f31 [0,4194304] 0 2024-04-05T21:26:39.443 INFO:tasks.workunit.client.1.smithi188.stdout:3/633: symlink d5/df/ld7 0 2024-04-05T21:26:39.448 INFO:tasks.workunit.client.1.smithi188.stdout:8/553: dwrite d5/d21/f6a [0,4194304] 0 2024-04-05T21:26:39.449 INFO:tasks.workunit.client.1.smithi188.stdout:3/634: dread d5/d35/d4e/f5b [0,4194304] 0 2024-04-05T21:26:39.449 INFO:tasks.workunit.client.1.smithi188.stdout:8/554: write d5/d21/d2a/f31 [9020153,34988] 0 2024-04-05T21:26:39.452 INFO:tasks.workunit.client.1.smithi188.stdout:3/635: rename d5/dc to d5/d7/d6d/dd8 0 2024-04-05T21:26:39.455 INFO:tasks.workunit.client.1.smithi188.stdout:3/636: mknod d5/d7/d65/da2/cd9 0 2024-04-05T21:26:39.457 INFO:tasks.workunit.client.1.smithi188.stdout:3/637: write d5/f7d [2035683,38914] 0 2024-04-05T21:26:39.465 INFO:tasks.workunit.client.1.smithi188.stdout:3/638: dwrite d5/d7/d65/f8c [0,4194304] 0 2024-04-05T21:26:39.469 INFO:tasks.workunit.client.1.smithi188.stdout:8/555: dwrite d5/d21/d40/d43/d44/d5b/d99/da3/fba [0,4194304] 0 2024-04-05T21:26:39.476 INFO:tasks.workunit.client.1.smithi188.stdout:8/556: symlink d5/d21/d40/d43/d44/d5b/d99/d69/d9b/lbf 0 2024-04-05T21:26:39.476 INFO:tasks.workunit.client.1.smithi188.stdout:3/639: dwrite d5/d7/d65/d5f/fa1 [0,4194304] 0 2024-04-05T21:26:39.477 INFO:tasks.workunit.client.0.smithi121.stdout:2/269: sync 2024-04-05T21:26:39.499 INFO:tasks.workunit.client.1.smithi188.stdout:7/719: sync 2024-04-05T21:26:39.500 INFO:tasks.workunit.client.1.smithi188.stdout:7/720: truncate d3/d4/d1d/d74/d88/fc0 5146764 0 2024-04-05T21:26:39.501 INFO:tasks.workunit.client.1.smithi188.stdout:7/721: fdatasync d3/d4/db/d2e/f6b 0 2024-04-05T21:26:39.503 INFO:tasks.workunit.client.0.smithi121.stdout:7/314: fdatasync d0/d9/d22/d28/d56/f58 0 2024-04-05T21:26:39.504 INFO:tasks.workunit.client.0.smithi121.stdout:7/315: write d0/d9/f5e [954880,77959] 0 2024-04-05T21:26:39.505 INFO:tasks.workunit.client.0.smithi121.stdout:2/270: creat d2/d9/d17/d3a/f4d x:0 0 0 2024-04-05T21:26:39.507 INFO:tasks.workunit.client.1.smithi188.stdout:7/722: dwrite d3/d4/db/d2e/d3b/fd5 [0,4194304] 0 2024-04-05T21:26:39.508 INFO:tasks.workunit.client.0.smithi121.stdout:1/387: getdents d8/d10 0 2024-04-05T21:26:39.512 INFO:tasks.workunit.client.0.smithi121.stdout:5/427: truncate dd/de/d6d/f6e 1056315 0 2024-04-05T21:26:39.515 INFO:tasks.workunit.client.0.smithi121.stdout:4/374: write f4 [2942021,36992] 0 2024-04-05T21:26:39.518 INFO:tasks.workunit.client.0.smithi121.stdout:2/271: dwrite d2/d9/f1f [0,4194304] 0 2024-04-05T21:26:39.518 INFO:tasks.workunit.client.0.smithi121.stdout:5/428: rmdir dd/de/d19/d38/d56 39 2024-04-05T21:26:39.520 INFO:tasks.workunit.client.0.smithi121.stdout:1/388: dread d8/d10/d14/d1a/d1b/d24/f2a [0,4194304] 0 2024-04-05T21:26:39.520 INFO:tasks.workunit.client.0.smithi121.stdout:4/375: rmdir d5/d11/d28 39 2024-04-05T21:26:39.522 INFO:tasks.workunit.client.0.smithi121.stdout:4/376: write d5/d11/d26/d27/d29/d58/f6d [305699,12856] 0 2024-04-05T21:26:39.524 INFO:tasks.workunit.client.0.smithi121.stdout:2/272: unlink d2/d9/d38/c32 0 2024-04-05T21:26:39.525 INFO:tasks.workunit.client.0.smithi121.stdout:4/377: mknod d5/d11/d20/c7c 0 2024-04-05T21:26:39.527 INFO:tasks.workunit.client.0.smithi121.stdout:2/273: chown d2/d1e/c46 27131 1 2024-04-05T21:26:39.527 INFO:tasks.workunit.client.0.smithi121.stdout:2/274: chown d2 277560 1 2024-04-05T21:26:39.528 INFO:tasks.workunit.client.0.smithi121.stdout:7/316: dread d0/f42 [0,4194304] 0 2024-04-05T21:26:39.528 INFO:tasks.workunit.client.0.smithi121.stdout:2/275: chown d2/d28/c40 5 1 2024-04-05T21:26:39.529 INFO:tasks.workunit.client.0.smithi121.stdout:7/317: write d0/d9/f5e [1862881,30175] 0 2024-04-05T21:26:39.531 INFO:tasks.workunit.client.0.smithi121.stdout:2/276: dread - d2/d9/f21 zero size 2024-04-05T21:26:39.534 INFO:tasks.workunit.client.0.smithi121.stdout:7/318: creat d0/d23/d5a/f63 x:0 0 0 2024-04-05T21:26:39.534 INFO:tasks.workunit.client.0.smithi121.stdout:5/429: write dd/de/d19/d38/d45/f4e [690236,77425] 0 2024-04-05T21:26:39.535 INFO:tasks.workunit.client.0.smithi121.stdout:5/430: readlink dd/l6f 0 2024-04-05T21:26:39.535 INFO:tasks.workunit.client.0.smithi121.stdout:7/319: chown d0/f1f 19820 1 2024-04-05T21:26:39.536 INFO:tasks.workunit.client.0.smithi121.stdout:7/320: chown d0/d9/f21 4144 1 2024-04-05T21:26:39.538 INFO:tasks.workunit.client.0.smithi121.stdout:2/277: dwrite d2/d9/d1c/f22 [0,4194304] 0 2024-04-05T21:26:39.539 INFO:tasks.workunit.client.0.smithi121.stdout:5/431: truncate dd/de/d19/d38/d45/f5c 627795 0 2024-04-05T21:26:39.539 INFO:tasks.workunit.client.0.smithi121.stdout:5/432: stat dd/f17 0 2024-04-05T21:26:39.540 INFO:tasks.workunit.client.0.smithi121.stdout:2/278: chown d2/d9/d1c/f22 11379064 1 2024-04-05T21:26:39.542 INFO:tasks.workunit.client.0.smithi121.stdout:2/279: readlink d2/d9/d38/l3c 0 2024-04-05T21:26:39.546 INFO:tasks.workunit.client.0.smithi121.stdout:2/280: unlink d2/d9/d17/c3f 0 2024-04-05T21:26:39.547 INFO:tasks.workunit.client.0.smithi121.stdout:7/321: dwrite d0/f15 [4194304,4194304] 0 2024-04-05T21:26:39.547 INFO:tasks.workunit.client.0.smithi121.stdout:4/378: dwrite d5/d11/d28/f5d [0,4194304] 0 2024-04-05T21:26:39.548 INFO:tasks.workunit.client.0.smithi121.stdout:4/379: fdatasync d5/d11/f1e 0 2024-04-05T21:26:39.549 INFO:tasks.workunit.client.0.smithi121.stdout:4/380: fsync d5/f1d 0 2024-04-05T21:26:39.551 INFO:tasks.workunit.client.0.smithi121.stdout:4/381: fdatasync d5/d11/d20/f57 0 2024-04-05T21:26:39.552 INFO:tasks.workunit.client.0.smithi121.stdout:2/281: getdents d2/d9/d12 0 2024-04-05T21:26:39.553 INFO:tasks.workunit.client.0.smithi121.stdout:2/282: write d2/d1e/f24 [4366523,19767] 0 2024-04-05T21:26:39.558 INFO:tasks.workunit.client.0.smithi121.stdout:2/283: mknod d2/d9/d17/c4e 0 2024-04-05T21:26:39.559 INFO:tasks.workunit.client.0.smithi121.stdout:2/284: fdatasync d2/d9/d17/d3a/f4d 0 2024-04-05T21:26:39.559 INFO:tasks.workunit.client.0.smithi121.stdout:2/285: chown d2/d1e/c46 206229 1 2024-04-05T21:26:39.561 INFO:tasks.workunit.client.0.smithi121.stdout:4/382: write d5/fa [1293652,26312] 0 2024-04-05T21:26:39.562 INFO:tasks.workunit.client.0.smithi121.stdout:2/286: write d2/f10 [2956675,21726] 0 2024-04-05T21:26:39.565 INFO:tasks.workunit.client.0.smithi121.stdout:4/383: dread - d5/d11/d26/d27/d29/d2e/f7b zero size 2024-04-05T21:26:39.566 INFO:tasks.workunit.client.0.smithi121.stdout:7/322: dwrite d0/d9/d22/f44 [0,4194304] 0 2024-04-05T21:26:39.568 INFO:tasks.workunit.client.0.smithi121.stdout:7/323: stat d0/f8 0 2024-04-05T21:26:39.570 INFO:tasks.workunit.client.0.smithi121.stdout:7/324: chown d0/d9/d22/d28/d3b/f4f 90296202 1 2024-04-05T21:26:39.571 INFO:tasks.workunit.client.0.smithi121.stdout:2/287: dwrite d2/d9/f2f [0,4194304] 0 2024-04-05T21:26:39.573 INFO:tasks.workunit.client.0.smithi121.stdout:4/384: unlink d5/d45/d67/l6c 0 2024-04-05T21:26:39.577 INFO:tasks.workunit.client.0.smithi121.stdout:7/325: creat d0/d9/d22/d28/d56/f64 x:0 0 0 2024-04-05T21:26:39.577 INFO:tasks.workunit.client.0.smithi121.stdout:2/288: rmdir d2/d9/d38/d2d 39 2024-04-05T21:26:39.578 INFO:tasks.workunit.client.0.smithi121.stdout:4/385: mknod d5/d11/d26/d27/d64/c7d 0 2024-04-05T21:26:39.579 INFO:tasks.workunit.client.0.smithi121.stdout:2/289: write d2/d9/ff [2772452,2514] 0 2024-04-05T21:26:39.580 INFO:tasks.workunit.client.0.smithi121.stdout:7/326: creat d0/d32/f65 x:0 0 0 2024-04-05T21:26:39.580 INFO:tasks.workunit.client.0.smithi121.stdout:4/386: fsync d5/d11/d2c/f65 0 2024-04-05T21:26:39.582 INFO:tasks.workunit.client.0.smithi121.stdout:2/290: rmdir d2/d28 39 2024-04-05T21:26:39.583 INFO:tasks.workunit.client.0.smithi121.stdout:7/327: truncate d0/d9/d22/d28/d56/f58 1013472 0 2024-04-05T21:26:39.586 INFO:tasks.workunit.client.0.smithi121.stdout:7/328: symlink d0/d32/l66 0 2024-04-05T21:26:39.587 INFO:tasks.workunit.client.0.smithi121.stdout:2/291: truncate d2/d9/d38/d2d/f41 14914 0 2024-04-05T21:26:39.588 INFO:tasks.workunit.client.0.smithi121.stdout:4/387: dwrite d5/f3f [4194304,4194304] 0 2024-04-05T21:26:39.588 INFO:tasks.workunit.client.1.smithi188.stdout:7/723: rename d3/d4/db/d2e/d3b/d94/fc5 to d3/d4/db/d2e/d8c/db7/dd6/fe4 0 2024-04-05T21:26:39.590 INFO:tasks.workunit.client.0.smithi121.stdout:2/292: write d2/d1e/f24 [5920244,130433] 0 2024-04-05T21:26:39.590 INFO:tasks.workunit.client.0.smithi121.stdout:4/388: chown d5/d11/f14 779759121 1 2024-04-05T21:26:39.591 INFO:tasks.workunit.client.1.smithi188.stdout:7/724: mknod d3/d4/d7c/ce5 0 2024-04-05T21:26:39.591 INFO:tasks.workunit.client.1.smithi188.stdout:7/725: fsync d3/f71 0 2024-04-05T21:26:39.593 INFO:tasks.workunit.client.0.smithi121.stdout:7/329: mknod d0/d9/d22/d28/d3b/c67 0 2024-04-05T21:26:39.594 INFO:tasks.workunit.client.1.smithi188.stdout:7/726: write d3/d4/db/fcd [1983908,109529] 0 2024-04-05T21:26:39.595 INFO:tasks.workunit.client.0.smithi121.stdout:4/389: fdatasync d5/f8 0 2024-04-05T21:26:39.598 INFO:tasks.workunit.client.0.smithi121.stdout:7/330: creat d0/d23/d5a/f68 x:0 0 0 2024-04-05T21:26:39.598 INFO:tasks.workunit.client.0.smithi121.stdout:4/390: dread d5/f8 [0,4194304] 0 2024-04-05T21:26:39.600 INFO:tasks.workunit.client.0.smithi121.stdout:4/391: rmdir d5/d11/d28/d73 39 2024-04-05T21:26:39.601 INFO:tasks.workunit.client.0.smithi121.stdout:4/392: fdatasync d5/d11/d28/d3b/f59 0 2024-04-05T21:26:39.602 INFO:tasks.workunit.client.0.smithi121.stdout:4/393: fdatasync d5/fd 0 2024-04-05T21:26:39.602 INFO:tasks.workunit.client.0.smithi121.stdout:7/331: creat d0/d9/f69 x:0 0 0 2024-04-05T21:26:39.603 INFO:tasks.workunit.client.1.smithi188.stdout:7/727: dwrite d3/d4/db/d2e/d37/fc2 [0,4194304] 0 2024-04-05T21:26:39.603 INFO:tasks.workunit.client.0.smithi121.stdout:4/394: chown d5/d11/f17 152333 1 2024-04-05T21:26:39.603 INFO:tasks.workunit.client.1.smithi188.stdout:7/728: stat d3/d4/d1d/l23 0 2024-04-05T21:26:39.604 INFO:tasks.workunit.client.1.smithi188.stdout:7/729: readlink d3/d4/db/d2e/d49/l99 0 2024-04-05T21:26:39.610 INFO:tasks.workunit.client.0.smithi121.stdout:4/395: dwrite d5/d11/f1e [0,4194304] 0 2024-04-05T21:26:39.612 INFO:tasks.workunit.client.0.smithi121.stdout:4/396: chown d5/d11/d28/d3b/l43 30 1 2024-04-05T21:26:39.613 INFO:tasks.workunit.client.0.smithi121.stdout:7/332: dread d0/d9/f5e [0,4194304] 0 2024-04-05T21:26:39.615 INFO:tasks.workunit.client.0.smithi121.stdout:4/397: read d5/d45/f74 [511938,76305] 0 2024-04-05T21:26:39.617 INFO:tasks.workunit.client.0.smithi121.stdout:7/333: rename d0/d9/d22/d28/d3b/c67 to d0/d9/d22/c6a 0 2024-04-05T21:26:39.617 INFO:tasks.workunit.client.0.smithi121.stdout:4/398: mknod d5/d11/d26/d27/c7e 0 2024-04-05T21:26:39.618 INFO:tasks.workunit.client.0.smithi121.stdout:7/334: creat d0/d32/f6b x:0 0 0 2024-04-05T21:26:39.619 INFO:tasks.workunit.client.0.smithi121.stdout:7/335: read - d0/d32/f6b zero size 2024-04-05T21:26:39.620 INFO:tasks.workunit.client.0.smithi121.stdout:4/399: fsync d5/d11/d28/fe 0 2024-04-05T21:26:39.621 INFO:tasks.workunit.client.0.smithi121.stdout:4/400: rename d5 to d5/d11/d28/d3b/d7f 22 2024-04-05T21:26:39.621 INFO:tasks.workunit.client.1.smithi188.stdout:6/633: sync 2024-04-05T21:26:39.622 INFO:tasks.workunit.client.1.smithi188.stdout:9/552: sync 2024-04-05T21:26:39.622 INFO:tasks.workunit.client.1.smithi188.stdout:5/528: sync 2024-04-05T21:26:39.622 INFO:tasks.workunit.client.1.smithi188.stdout:1/595: sync 2024-04-05T21:26:39.622 INFO:tasks.workunit.client.1.smithi188.stdout:2/636: sync 2024-04-05T21:26:39.622 INFO:tasks.workunit.client.1.smithi188.stdout:4/804: sync 2024-04-05T21:26:39.622 INFO:tasks.workunit.client.1.smithi188.stdout:0/482: sync 2024-04-05T21:26:39.623 INFO:tasks.workunit.client.0.smithi121.stdout:7/336: creat d0/d9/d22/f6c x:0 0 0 2024-04-05T21:26:39.623 INFO:tasks.workunit.client.0.smithi121.stdout:7/337: chown d0/d23/f54 0 1 2024-04-05T21:26:39.623 INFO:tasks.workunit.client.0.smithi121.stdout:2/293: dread d2/f11 [0,4194304] 0 2024-04-05T21:26:39.625 INFO:tasks.workunit.client.0.smithi121.stdout:7/338: unlink d0/l17 0 2024-04-05T21:26:39.626 INFO:tasks.workunit.client.1.smithi188.stdout:7/730: dwrite d3/d4/db/f60 [0,4194304] 0 2024-04-05T21:26:39.627 INFO:tasks.workunit.client.1.smithi188.stdout:4/805: mknod d0/d14/d15/c115 0 2024-04-05T21:26:39.627 INFO:tasks.workunit.client.0.smithi121.stdout:4/401: dwrite d5/d11/d20/f57 [0,4194304] 0 2024-04-05T21:26:39.629 INFO:tasks.workunit.client.1.smithi188.stdout:6/634: write d6/d13/d48/d49/d75/fbf [629910,71375] 0 2024-04-05T21:26:39.629 INFO:tasks.workunit.client.0.smithi121.stdout:2/294: creat d2/d28/f4f x:0 0 0 2024-04-05T21:26:39.630 INFO:tasks.workunit.client.1.smithi188.stdout:1/596: unlink d1/dc/d16/d21/f6e 0 2024-04-05T21:26:39.632 INFO:tasks.workunit.client.0.smithi121.stdout:2/295: chown d2/d9/f2f 7 1 2024-04-05T21:26:39.636 INFO:tasks.workunit.client.0.smithi121.stdout:4/402: chown d5/d11/d28/d3b/c3e 31910 1 2024-04-05T21:26:39.638 INFO:tasks.workunit.client.0.smithi121.stdout:7/339: dwrite d0/f3 [0,4194304] 0 2024-04-05T21:26:39.641 INFO:tasks.workunit.client.0.smithi121.stdout:7/340: symlink d0/d9/d22/d3e/l6d 0 2024-04-05T21:26:39.642 INFO:tasks.workunit.client.1.smithi188.stdout:9/553: dread fc [0,4194304] 0 2024-04-05T21:26:39.643 INFO:tasks.workunit.client.0.smithi121.stdout:4/403: mkdir d5/d11/d80 0 2024-04-05T21:26:39.644 INFO:tasks.workunit.client.0.smithi121.stdout:4/404: write d5/d11/d28/d3b/f59 [3088425,30823] 0 2024-04-05T21:26:39.645 INFO:tasks.workunit.client.1.smithi188.stdout:5/529: dwrite d6/d15/d5c/d7a/f93 [0,4194304] 0 2024-04-05T21:26:39.647 INFO:tasks.workunit.client.0.smithi121.stdout:4/405: creat d5/f81 x:0 0 0 2024-04-05T21:26:39.648 INFO:tasks.workunit.client.1.smithi188.stdout:5/530: fsync d6/d82/d40/d42/f77 0 2024-04-05T21:26:39.649 INFO:tasks.workunit.client.1.smithi188.stdout:4/806: dwrite d0/d14/d15/d78/db1/d6a/fd9 [0,4194304] 0 2024-04-05T21:26:39.650 INFO:tasks.workunit.client.1.smithi188.stdout:4/807: stat d0/d14/d2f/l45 0 2024-04-05T21:26:39.651 INFO:tasks.workunit.client.0.smithi121.stdout:4/406: rename d5/d11/f21 to d5/d11/d20/f82 0 2024-04-05T21:26:39.655 INFO:tasks.workunit.client.0.smithi121.stdout:4/407: symlink d5/d11/d28/l83 0 2024-04-05T21:26:39.657 INFO:tasks.workunit.client.1.smithi188.stdout:2/637: getdents d5/db/dd/d36/d3c/d48/d7a 0 2024-04-05T21:26:39.658 INFO:tasks.workunit.client.1.smithi188.stdout:7/731: mknod d3/d4/d7c/ce6 0 2024-04-05T21:26:39.659 INFO:tasks.workunit.client.0.smithi121.stdout:0/398: sync 2024-04-05T21:26:39.660 INFO:tasks.workunit.client.1.smithi188.stdout:1/597: chown d1/d7/d4e/c53 179235841 1 2024-04-05T21:26:39.661 INFO:tasks.workunit.client.1.smithi188.stdout:9/554: dread - dd/d49/d6b/d40/d59/f7f zero size 2024-04-05T21:26:39.662 INFO:tasks.workunit.client.0.smithi121.stdout:4/408: dwrite d5/d11/f48 [0,4194304] 0 2024-04-05T21:26:39.663 INFO:tasks.workunit.client.1.smithi188.stdout:4/808: mknod d0/d14/d15/d53/d82/c116 0 2024-04-05T21:26:39.664 INFO:tasks.workunit.client.1.smithi188.stdout:2/638: rmdir d5/db/dd/d36/d3c/d48/d85 39 2024-04-05T21:26:39.668 INFO:tasks.workunit.client.1.smithi188.stdout:1/598: mkdir d1/d7/dbf 0 2024-04-05T21:26:39.669 INFO:tasks.workunit.client.1.smithi188.stdout:5/531: getdents d6/d82/d40/d42/d81/da6/db2 0 2024-04-05T21:26:39.671 INFO:tasks.workunit.client.0.smithi121.stdout:4/409: dwrite d5/d11/d28/f5d [4194304,4194304] 0 2024-04-05T21:26:39.672 INFO:tasks.workunit.client.1.smithi188.stdout:4/809: write d0/dbe/f10c [2671633,29937] 0 2024-04-05T21:26:39.672 INFO:tasks.workunit.client.1.smithi188.stdout:6/635: link d6/d13/d18/d19/l66 d6/d28/d45/lc2 0 2024-04-05T21:26:39.673 INFO:tasks.workunit.client.0.smithi121.stdout:4/410: chown d5/d11/f17 490 1 2024-04-05T21:26:39.673 INFO:tasks.workunit.client.1.smithi188.stdout:4/810: dread - d0/d2/dc7/d104/f105 zero size 2024-04-05T21:26:39.676 INFO:tasks.workunit.client.1.smithi188.stdout:9/555: rename dd/d11/d1f/d26/d55/d98/fa9 to dd/d11/fb8 0 2024-04-05T21:26:39.677 INFO:tasks.workunit.client.0.smithi121.stdout:4/411: unlink d5/d11/f35 0 2024-04-05T21:26:39.677 INFO:tasks.workunit.client.1.smithi188.stdout:6/636: truncate d6/d13/d38/f55 1431489 0 2024-04-05T21:26:39.678 INFO:tasks.workunit.client.1.smithi188.stdout:4/811: dread - d0/d2/d17/d48/fb3 zero size 2024-04-05T21:26:39.679 INFO:tasks.workunit.client.0.smithi121.stdout:4/412: creat d5/d45/d67/f84 x:0 0 0 2024-04-05T21:26:39.680 INFO:tasks.workunit.client.1.smithi188.stdout:9/556: unlink dd/d49/d6b/l54 0 2024-04-05T21:26:39.681 INFO:tasks.workunit.client.0.smithi121.stdout:4/413: write d5/d11/d28/f5d [4863567,11259] 0 2024-04-05T21:26:39.682 INFO:tasks.workunit.client.0.smithi121.stdout:4/414: stat d5/d11/d28/l25 0 2024-04-05T21:26:39.683 INFO:tasks.workunit.client.1.smithi188.stdout:1/599: rename d1/dc/c64 to d1/dc/d16/d46/d95/cc0 0 2024-04-05T21:26:39.684 INFO:tasks.workunit.client.1.smithi188.stdout:6/637: symlink d6/d13/d18/d19/d2c/db2/lc3 0 2024-04-05T21:26:39.685 INFO:tasks.workunit.client.1.smithi188.stdout:6/638: stat d6/d13/d18/d19/f72 0 2024-04-05T21:26:39.685 INFO:tasks.workunit.client.0.smithi121.stdout:4/415: rename d5/d11/d26/d27 to d5/d11/d26/d49/d85 0 2024-04-05T21:26:39.687 INFO:tasks.workunit.client.1.smithi188.stdout:6/639: stat d6/d13/d18/d19/d2c/d34 0 2024-04-05T21:26:39.687 INFO:tasks.workunit.client.1.smithi188.stdout:6/640: chown d6/d13/d38 162960808 1 2024-04-05T21:26:39.688 INFO:tasks.workunit.client.1.smithi188.stdout:4/812: rename d0/d14/d15/d78/db1/daa to d0/d14/d15/d53/d82/d8f/d117 0 2024-04-05T21:26:39.689 INFO:tasks.workunit.client.0.smithi121.stdout:4/416: creat d5/f86 x:0 0 0 2024-04-05T21:26:39.692 INFO:tasks.workunit.client.1.smithi188.stdout:2/639: dread d5/db/dd/d36/d3c/d48/f5b [0,4194304] 0 2024-04-05T21:26:39.693 INFO:tasks.workunit.client.0.smithi121.stdout:4/417: unlink d5/d11/d28/d3b/c3e 0 2024-04-05T21:26:39.694 INFO:tasks.workunit.client.1.smithi188.stdout:4/813: dread d0/d2/f26 [0,4194304] 0 2024-04-05T21:26:39.698 INFO:tasks.workunit.client.1.smithi188.stdout:2/640: chown d5/db/dd/d67/dcd/d46/d9f/fa0 6083342 1 2024-04-05T21:26:39.699 INFO:tasks.workunit.client.1.smithi188.stdout:9/557: rename dd/d11/l14 to dd/d49/d6b/d40/d59/lb9 0 2024-04-05T21:26:39.701 INFO:tasks.workunit.client.1.smithi188.stdout:4/814: mknod d0/d14/df6/c118 0 2024-04-05T21:26:39.703 INFO:tasks.workunit.client.0.smithi121.stdout:2/296: truncate d2/f10 1632140 0 2024-04-05T21:26:39.704 INFO:tasks.workunit.client.1.smithi188.stdout:1/600: dread d1/dc/d16/d21/d27/d4a/f63 [0,4194304] 0 2024-04-05T21:26:39.704 INFO:tasks.workunit.client.1.smithi188.stdout:7/732: dread d3/d4/d1d/d74/d88/fc0 [0,4194304] 0 2024-04-05T21:26:39.704 INFO:tasks.workunit.client.1.smithi188.stdout:2/641: mkdir d5/db/dd/d36/d3c/d48/dcf 0 2024-04-05T21:26:39.707 INFO:tasks.workunit.client.1.smithi188.stdout:4/815: creat d0/d2/d17/d48/d70/f119 x:0 0 0 2024-04-05T21:26:39.708 INFO:tasks.workunit.client.1.smithi188.stdout:1/601: stat d1/dc/d16/d46/d88/d13/ca3 0 2024-04-05T21:26:39.710 INFO:tasks.workunit.client.1.smithi188.stdout:5/532: dread d6/f2d [0,4194304] 0 2024-04-05T21:26:39.710 INFO:tasks.workunit.client.1.smithi188.stdout:2/642: write d5/db/d35/d3b/f4d [469880,93689] 0 2024-04-05T21:26:39.712 INFO:tasks.workunit.client.0.smithi121.stdout:7/341: dwrite d0/d9/d22/d28/f31 [0,4194304] 0 2024-04-05T21:26:39.714 INFO:tasks.workunit.client.0.smithi121.stdout:2/297: dread d2/d9/d1c/f22 [0,4194304] 0 2024-04-05T21:26:39.714 INFO:tasks.workunit.client.1.smithi188.stdout:4/816: chown d0/d14/d2f/l10f 15412 1 2024-04-05T21:26:39.715 INFO:tasks.workunit.client.1.smithi188.stdout:1/602: fsync d1/dc/d16/f6f 0 2024-04-05T21:26:39.715 INFO:tasks.workunit.client.1.smithi188.stdout:1/603: chown d1/dc/d16/d2d/f97 1 1 2024-04-05T21:26:39.715 INFO:tasks.workunit.client.1.smithi188.stdout:7/733: symlink d3/d4/db/d2e/d8c/db7/dd6/le7 0 2024-04-05T21:26:39.718 INFO:tasks.workunit.client.1.smithi188.stdout:5/533: write d6/f5b [2464506,101073] 0 2024-04-05T21:26:39.718 INFO:tasks.workunit.client.1.smithi188.stdout:2/643: write d5/db/dd/d36/d3c/d48/f5b [276311,117273] 0 2024-04-05T21:26:39.720 INFO:tasks.workunit.client.0.smithi121.stdout:0/399: dwrite f2 [0,4194304] 0 2024-04-05T21:26:39.720 INFO:tasks.workunit.client.0.smithi121.stdout:7/342: creat d0/d23/d5b/d52/f6e x:0 0 0 2024-04-05T21:26:39.720 INFO:tasks.workunit.client.0.smithi121.stdout:2/298: fdatasync d2/d9/d17/d3a/f4a 0 2024-04-05T21:26:39.721 INFO:tasks.workunit.client.1.smithi188.stdout:1/604: mkdir d1/dc/d16/d21/d27/d4a/dc1 0 2024-04-05T21:26:39.722 INFO:tasks.workunit.client.1.smithi188.stdout:2/644: read - d5/db/dd/d67/fad zero size 2024-04-05T21:26:39.722 INFO:tasks.workunit.client.1.smithi188.stdout:5/534: chown d6/d17/c3f 0 1 2024-04-05T21:26:39.722 INFO:tasks.workunit.client.0.smithi121.stdout:7/343: dread d0/f1f [0,4194304] 0 2024-04-05T21:26:39.724 INFO:tasks.workunit.client.1.smithi188.stdout:2/645: stat d5/db/dd/d36/d3c/d48 0 2024-04-05T21:26:39.725 INFO:tasks.workunit.client.0.smithi121.stdout:4/418: write d5/d11/d26/d49/d85/f4c [3016834,22788] 0 2024-04-05T21:26:39.726 INFO:tasks.workunit.client.1.smithi188.stdout:1/605: mknod d1/dc/d16/d46/d88/d13/cc2 0 2024-04-05T21:26:39.727 INFO:tasks.workunit.client.0.smithi121.stdout:7/344: creat d0/d9/d22/d28/d3b/f6f x:0 0 0 2024-04-05T21:26:39.729 INFO:tasks.workunit.client.1.smithi188.stdout:5/535: mknod d6/d15/d5c/d7a/d44/d51/cba 0 2024-04-05T21:26:39.729 INFO:tasks.workunit.client.1.smithi188.stdout:5/536: chown d6/d15/l33 3 1 2024-04-05T21:26:39.729 INFO:tasks.workunit.client.1.smithi188.stdout:7/734: unlink d3/d4/ca 0 2024-04-05T21:26:39.730 INFO:tasks.workunit.client.0.smithi121.stdout:2/299: mknod d2/d9/c50 0 2024-04-05T21:26:39.731 INFO:tasks.workunit.client.0.smithi121.stdout:4/419: rmdir d5/d11/d26/d49/d85/d29 39 2024-04-05T21:26:39.734 INFO:tasks.workunit.client.1.smithi188.stdout:1/606: rmdir d1/dc/d16/d46/d88/d13/d60 39 2024-04-05T21:26:39.736 INFO:tasks.workunit.client.1.smithi188.stdout:1/607: truncate d1/d7/d4e/d5f/f76 1110696 0 2024-04-05T21:26:39.737 INFO:tasks.workunit.client.0.smithi121.stdout:2/300: dwrite d2/d9/d17/f19 [0,4194304] 0 2024-04-05T21:26:39.740 INFO:tasks.workunit.client.0.smithi121.stdout:0/400: truncate d7/d71/d29/d3f/f50 704006 0 2024-04-05T21:26:39.741 INFO:tasks.workunit.client.0.smithi121.stdout:7/345: mknod d0/d57/c70 0 2024-04-05T21:26:39.745 INFO:tasks.workunit.client.0.smithi121.stdout:2/301: dwrite d2/d9/d17/d3a/f4a [0,4194304] 0 2024-04-05T21:26:39.746 INFO:tasks.workunit.client.1.smithi188.stdout:2/646: rename d5/db/dd/d24/c91 to d5/db/dd/d36/cd0 0 2024-04-05T21:26:39.747 INFO:tasks.workunit.client.0.smithi121.stdout:2/302: readlink d2/d9/d38/l3c 0 2024-04-05T21:26:39.747 INFO:tasks.workunit.client.0.smithi121.stdout:7/346: symlink d0/d9/d22/l71 0 2024-04-05T21:26:39.748 INFO:tasks.workunit.client.1.smithi188.stdout:7/735: creat d3/d2c/d5e/fe8 x:0 0 0 2024-04-05T21:26:39.750 INFO:tasks.workunit.client.0.smithi121.stdout:7/347: mknod d0/d9/d22/c72 0 2024-04-05T21:26:39.751 INFO:tasks.workunit.client.1.smithi188.stdout:9/558: truncate dd/d49/d6b/d40/f4e 5379203 0 2024-04-05T21:26:39.751 INFO:tasks.workunit.client.0.smithi121.stdout:2/303: mkdir d2/d9/d38/d2d/d47/d51 0 2024-04-05T21:26:39.752 INFO:tasks.workunit.client.1.smithi188.stdout:2/647: dread - d5/db/dd/d67/fad zero size 2024-04-05T21:26:39.752 INFO:tasks.workunit.client.0.smithi121.stdout:7/348: write d0/d9/f24 [5382977,30705] 0 2024-04-05T21:26:39.752 INFO:tasks.workunit.client.1.smithi188.stdout:4/817: write d0/d14/d15/d1b/f3d [4310232,58900] 0 2024-04-05T21:26:39.753 INFO:tasks.workunit.client.0.smithi121.stdout:2/304: write d2/d9/f2f [4773478,92956] 0 2024-04-05T21:26:39.753 INFO:tasks.workunit.client.1.smithi188.stdout:5/537: rename d6/d15/d1b/f2a to d6/d17/fbb 0 2024-04-05T21:26:39.754 INFO:tasks.workunit.client.0.smithi121.stdout:7/349: mkdir d0/d32/d4e/d73 0 2024-04-05T21:26:39.755 INFO:tasks.workunit.client.0.smithi121.stdout:2/305: creat d2/d1e/f52 x:0 0 0 2024-04-05T21:26:39.755 INFO:tasks.workunit.client.1.smithi188.stdout:6/641: dwrite d6/d13/d18/d19/f72 [0,4194304] 0 2024-04-05T21:26:39.756 INFO:tasks.workunit.client.1.smithi188.stdout:2/648: chown d5/db/d35/d3b/d6b/l8d 1519 1 2024-04-05T21:26:39.757 INFO:tasks.workunit.client.1.smithi188.stdout:9/559: creat dd/d3d/d95/fba x:0 0 0 2024-04-05T21:26:39.759 INFO:tasks.workunit.client.1.smithi188.stdout:5/538: readlink d6/d17/d39/l6d 0 2024-04-05T21:26:39.759 INFO:tasks.workunit.client.0.smithi121.stdout:0/401: dread d7/d71/f26 [0,4194304] 0 2024-04-05T21:26:39.760 INFO:tasks.workunit.client.1.smithi188.stdout:1/608: write d1/dc/f56 [1262300,106618] 0 2024-04-05T21:26:39.760 INFO:tasks.workunit.client.1.smithi188.stdout:7/736: symlink d3/d4/d1d/le9 0 2024-04-05T21:26:39.761 INFO:tasks.workunit.client.1.smithi188.stdout:9/560: chown dd/d11/d1f/f6a 94673109 1 2024-04-05T21:26:39.765 INFO:tasks.workunit.client.1.smithi188.stdout:9/561: unlink dd/d49/d6b/f43 0 2024-04-05T21:26:39.766 INFO:tasks.workunit.client.1.smithi188.stdout:5/539: readlink d6/d82/d40/d42/l71 0 2024-04-05T21:26:39.767 INFO:tasks.workunit.client.1.smithi188.stdout:4/818: rmdir d0/d2/d52/df9 0 2024-04-05T21:26:39.770 INFO:tasks.workunit.client.0.smithi121.stdout:4/420: dwrite d5/d11/d28/f5d [0,4194304] 0 2024-04-05T21:26:39.777 INFO:tasks.workunit.client.0.smithi121.stdout:2/306: rmdir d2/d9/d38 39 2024-04-05T21:26:39.778 INFO:tasks.workunit.client.0.smithi121.stdout:2/307: fsync d2/d9/d17/f18 0 2024-04-05T21:26:39.779 INFO:tasks.workunit.client.1.smithi188.stdout:3/640: sync 2024-04-05T21:26:39.779 INFO:tasks.workunit.client.1.smithi188.stdout:8/557: sync 2024-04-05T21:26:39.783 INFO:tasks.workunit.client.1.smithi188.stdout:3/641: truncate d5/d7/d31/f78 482229 0 2024-04-05T21:26:39.784 INFO:tasks.workunit.client.1.smithi188.stdout:6/642: getdents d6/d13/d48/d49 0 2024-04-05T21:26:39.785 INFO:tasks.workunit.client.0.smithi121.stdout:7/350: dwrite d0/d9/f21 [0,4194304] 0 2024-04-05T21:26:39.785 INFO:tasks.workunit.client.1.smithi188.stdout:2/649: link d5/db/dd/d36/lbf d5/db/dd/d36/d3c/d48/ld1 0 2024-04-05T21:26:39.786 INFO:tasks.workunit.client.0.smithi121.stdout:2/308: dwrite d2/d9/d17/f19 [0,4194304] 0 2024-04-05T21:26:39.787 INFO:tasks.workunit.client.1.smithi188.stdout:7/737: creat d3/fea x:0 0 0 2024-04-05T21:26:39.787 INFO:tasks.workunit.client.0.smithi121.stdout:2/309: truncate d2/d28/f4f 580955 0 2024-04-05T21:26:39.788 INFO:tasks.workunit.client.1.smithi188.stdout:8/558: readlink d5/d21/d2a/d71/l8a 0 2024-04-05T21:26:39.788 INFO:tasks.workunit.client.0.smithi121.stdout:7/351: unlink d0/d9/f3a 0 2024-04-05T21:26:39.790 INFO:tasks.workunit.client.1.smithi188.stdout:7/738: write d3/d2c/dce/f1f [193200,91433] 0 2024-04-05T21:26:39.791 INFO:tasks.workunit.client.0.smithi121.stdout:7/352: rename d0/d23/f47 to d0/d23/f74 0 2024-04-05T21:26:39.792 INFO:tasks.workunit.client.1.smithi188.stdout:2/650: creat d5/d5f/d98/fd2 x:0 0 0 2024-04-05T21:26:39.792 INFO:tasks.workunit.client.0.smithi121.stdout:0/402: dread d7/d71/d29/d3f/f50 [0,4194304] 0 2024-04-05T21:26:39.792 INFO:tasks.workunit.client.1.smithi188.stdout:1/609: creat d1/dc/d16/d46/d88/d13/fc3 x:0 0 0 2024-04-05T21:26:39.793 INFO:tasks.workunit.client.1.smithi188.stdout:7/739: stat d3/d4/db/d2e/f93 0 2024-04-05T21:26:39.793 INFO:tasks.workunit.client.0.smithi121.stdout:7/353: write d0/d23/d5a/f68 [505252,106854] 0 2024-04-05T21:26:39.794 INFO:tasks.workunit.client.1.smithi188.stdout:4/819: rename d0/d2/d17/d48/d70/l107 to d0/d14/d15/d78/l11a 0 2024-04-05T21:26:39.794 INFO:tasks.workunit.client.1.smithi188.stdout:7/740: fsync d3/d4/d1d/d74/f86 0 2024-04-05T21:26:39.795 INFO:tasks.workunit.client.0.smithi121.stdout:0/403: mkdir d7/d88 0 2024-04-05T21:26:39.796 INFO:tasks.workunit.client.0.smithi121.stdout:7/354: write d0/d9/d22/d28/d56/f58 [1838627,50003] 0 2024-04-05T21:26:39.797 INFO:tasks.workunit.client.0.smithi121.stdout:7/355: stat d0/d9/d22/d3e/l6d 0 2024-04-05T21:26:39.798 INFO:tasks.workunit.client.1.smithi188.stdout:4/820: readlink d0/d2/d52/l9a 0 2024-04-05T21:26:39.799 INFO:tasks.workunit.client.1.smithi188.stdout:7/741: dread - d3/fea zero size 2024-04-05T21:26:39.799 INFO:tasks.workunit.client.0.smithi121.stdout:4/421: dread d5/d11/d28/f13 [0,4194304] 0 2024-04-05T21:26:39.801 INFO:tasks.workunit.client.0.smithi121.stdout:7/356: unlink d0/d23/c49 0 2024-04-05T21:26:39.802 INFO:tasks.workunit.client.0.smithi121.stdout:4/422: mkdir d5/d11/d28/d87 0 2024-04-05T21:26:39.803 INFO:tasks.workunit.client.1.smithi188.stdout:1/610: write d1/dc/d16/d46/d88/d13/f6a [4552220,105776] 0 2024-04-05T21:26:39.803 INFO:tasks.workunit.client.1.smithi188.stdout:7/742: write d3/d4/f2d [75133,117877] 0 2024-04-05T21:26:39.803 INFO:tasks.workunit.client.0.smithi121.stdout:0/404: dwrite d7/d62/f42 [0,4194304] 0 2024-04-05T21:26:39.804 INFO:tasks.workunit.client.1.smithi188.stdout:1/611: read - d1/db2/fb4 zero size 2024-04-05T21:26:39.804 INFO:tasks.workunit.client.0.smithi121.stdout:7/357: write d0/d9/f1d [3142380,8479] 0 2024-04-05T21:26:39.805 INFO:tasks.workunit.client.1.smithi188.stdout:1/612: chown d1 209981 1 2024-04-05T21:26:39.806 INFO:tasks.workunit.client.1.smithi188.stdout:5/540: write d6/d82/d40/fb6 [1659516,5812] 0 2024-04-05T21:26:39.806 INFO:tasks.workunit.client.1.smithi188.stdout:6/643: creat d6/d13/d18/d19/d2c/d34/fc4 x:0 0 0 2024-04-05T21:26:39.809 INFO:tasks.workunit.client.0.smithi121.stdout:4/423: symlink d5/d11/d28/l88 0 2024-04-05T21:26:39.809 INFO:tasks.workunit.client.0.smithi121.stdout:7/358: write d0/d9/f12 [4595272,50494] 0 2024-04-05T21:26:39.811 INFO:tasks.workunit.client.1.smithi188.stdout:6/644: readlink d6/l9 0 2024-04-05T21:26:39.812 INFO:tasks.workunit.client.1.smithi188.stdout:8/559: mkdir d5/d21/d2a/d95/dc0 0 2024-04-05T21:26:39.812 INFO:tasks.workunit.client.1.smithi188.stdout:2/651: mknod d5/db/dd/d36/d3c/d48/d7a/d7c/db5/cd3 0 2024-04-05T21:26:39.814 INFO:tasks.workunit.client.0.smithi121.stdout:4/424: mkdir d5/d89 0 2024-04-05T21:26:39.814 INFO:tasks.workunit.client.0.smithi121.stdout:7/359: dread d0/d9/d22/d28/d3b/f4f [0,4194304] 0 2024-04-05T21:26:39.815 INFO:tasks.workunit.client.0.smithi121.stdout:7/360: stat d0/d9/d22/c38 0 2024-04-05T21:26:39.815 INFO:tasks.workunit.client.1.smithi188.stdout:2/652: dread - d5/db/dd/d67/dcd/d46/fc5 zero size 2024-04-05T21:26:39.815 INFO:tasks.workunit.client.1.smithi188.stdout:9/562: dread dd/d11/d22/f4d [0,4194304] 0 2024-04-05T21:26:39.815 INFO:tasks.workunit.client.1.smithi188.stdout:9/563: stat dd/d3d/f6c 0 2024-04-05T21:26:39.816 INFO:tasks.workunit.client.0.smithi121.stdout:7/361: symlink d0/d9/d22/l75 0 2024-04-05T21:26:39.817 INFO:tasks.workunit.client.1.smithi188.stdout:4/821: symlink d0/d2/d17/d48/l11b 0 2024-04-05T21:26:39.818 INFO:tasks.workunit.client.0.smithi121.stdout:4/425: creat d5/d11/f8a x:0 0 0 2024-04-05T21:26:39.821 INFO:tasks.workunit.client.0.smithi121.stdout:7/362: dread d0/d9/d22/f44 [0,4194304] 0 2024-04-05T21:26:39.822 INFO:tasks.workunit.client.1.smithi188.stdout:9/564: unlink dd/d49/f60 0 2024-04-05T21:26:39.822 INFO:tasks.workunit.client.0.smithi121.stdout:4/426: truncate d5/d11/d26/d49/d85/d64/f75 4725577 0 2024-04-05T21:26:39.823 INFO:tasks.workunit.client.1.smithi188.stdout:8/560: fsync d5/d21/d80/fb2 0 2024-04-05T21:26:39.824 INFO:tasks.workunit.client.0.smithi121.stdout:4/427: mkdir d5/d45/d8b 0 2024-04-05T21:26:39.826 INFO:tasks.workunit.client.1.smithi188.stdout:2/653: dwrite d5/db/d35/d3b/f4d [0,4194304] 0 2024-04-05T21:26:39.826 INFO:tasks.workunit.client.1.smithi188.stdout:7/743: creat d3/d4/d1d/feb x:0 0 0 2024-04-05T21:26:39.826 INFO:tasks.workunit.client.0.smithi121.stdout:4/428: mkdir d5/d11/d2c/d8c 0 2024-04-05T21:26:39.826 INFO:tasks.workunit.client.0.smithi121.stdout:2/310: fsync d2/d1e/f24 0 2024-04-05T21:26:39.827 INFO:tasks.workunit.client.1.smithi188.stdout:7/744: dread - d3/d2c/dae/fda zero size 2024-04-05T21:26:39.827 INFO:tasks.workunit.client.0.smithi121.stdout:4/429: write d5/d11/f48 [5168548,106679] 0 2024-04-05T21:26:39.829 INFO:tasks.workunit.client.1.smithi188.stdout:3/642: link d5/d7/d6d/dd8/d2f/c97 d5/d39/cda 0 2024-04-05T21:26:39.830 INFO:tasks.workunit.client.0.smithi121.stdout:2/311: rename d2/c20 to d2/d9/d1c/d4b/c53 0 2024-04-05T21:26:39.831 INFO:tasks.workunit.client.0.smithi121.stdout:2/312: write d2/d9/fd [1748306,17773] 0 2024-04-05T21:26:39.833 INFO:tasks.workunit.client.1.smithi188.stdout:2/654: read d5/db/dd/d67/dcd/faf [2277182,105176] 0 2024-04-05T21:26:39.835 INFO:tasks.workunit.client.1.smithi188.stdout:6/645: creat d6/d13/d48/fc5 x:0 0 0 2024-04-05T21:26:39.836 INFO:tasks.workunit.client.1.smithi188.stdout:2/655: stat d5/db/dd/d36/d3c/d48/d78/lc7 0 2024-04-05T21:26:39.836 INFO:tasks.workunit.client.1.smithi188.stdout:9/565: mkdir dd/d11/d73/dbb 0 2024-04-05T21:26:39.838 INFO:tasks.workunit.client.1.smithi188.stdout:2/656: write d5/db/dd/d67/dcd/d46/f7b [845993,69716] 0 2024-04-05T21:26:39.838 INFO:tasks.workunit.client.1.smithi188.stdout:1/613: getdents d1/dc/d16/d2d 0 2024-04-05T21:26:39.838 INFO:tasks.workunit.client.1.smithi188.stdout:8/561: write d5/f34 [208919,107523] 0 2024-04-05T21:26:39.841 INFO:tasks.workunit.client.1.smithi188.stdout:4/822: creat d0/d14/dd8/dc5/f11c x:0 0 0 2024-04-05T21:26:39.841 INFO:tasks.workunit.client.1.smithi188.stdout:2/657: write d5/db/dd/d36/d3c/f4a [3243495,17534] 0 2024-04-05T21:26:39.846 INFO:tasks.workunit.client.1.smithi188.stdout:4/823: dread d0/d2/fc8 [0,4194304] 0 2024-04-05T21:26:39.847 INFO:tasks.workunit.client.1.smithi188.stdout:9/566: mknod dd/d11/d73/dbb/cbc 0 2024-04-05T21:26:39.848 INFO:tasks.workunit.client.1.smithi188.stdout:4/824: dread - d0/d14/dd8/dc5/f11c zero size 2024-04-05T21:26:39.848 INFO:tasks.workunit.client.1.smithi188.stdout:8/562: dread d5/d21/d40/f57 [0,4194304] 0 2024-04-05T21:26:39.852 INFO:tasks.workunit.client.1.smithi188.stdout:4/825: write d0/d14/d15/fb [292863,95499] 0 2024-04-05T21:26:39.852 INFO:tasks.workunit.client.1.smithi188.stdout:7/745: creat d3/d2c/d5e/fec x:0 0 0 2024-04-05T21:26:39.854 INFO:tasks.workunit.client.1.smithi188.stdout:5/541: truncate d6/d17/f97 790614 0 2024-04-05T21:26:39.857 INFO:tasks.workunit.client.1.smithi188.stdout:2/658: creat d5/db/dd/fd4 x:0 0 0 2024-04-05T21:26:39.858 INFO:tasks.workunit.client.0.smithi121.stdout:2/313: dread d2/d9/ff [0,4194304] 0 2024-04-05T21:26:39.858 INFO:tasks.workunit.client.1.smithi188.stdout:4/826: symlink d0/d14/dd8/dc5/l11d 0 2024-04-05T21:26:39.859 INFO:tasks.workunit.client.1.smithi188.stdout:5/542: write d6/d17/f80 [423446,64647] 0 2024-04-05T21:26:39.860 INFO:tasks.workunit.client.1.smithi188.stdout:7/746: mkdir d3/d4/db/d10/dbe/ded 0 2024-04-05T21:26:39.861 INFO:tasks.workunit.client.1.smithi188.stdout:9/567: dwrite dd/d11/d73/f92 [0,4194304] 0 2024-04-05T21:26:39.864 INFO:tasks.workunit.client.1.smithi188.stdout:8/563: dwrite d5/d21/d40/d43/f62 [0,4194304] 0 2024-04-05T21:26:39.865 INFO:tasks.workunit.client.1.smithi188.stdout:4/827: write d0/d14/d15/d53/d82/d8f/f93 [4062144,116805] 0 2024-04-05T21:26:39.865 INFO:tasks.workunit.client.1.smithi188.stdout:7/747: fsync d3/d2c/dae/f97 0 2024-04-05T21:26:39.867 INFO:tasks.workunit.client.1.smithi188.stdout:8/564: write d5/d21/d40/d43/fb5 [231391,94241] 0 2024-04-05T21:26:39.868 INFO:tasks.workunit.client.0.smithi121.stdout:8/282: sync 2024-04-05T21:26:39.868 INFO:tasks.workunit.client.0.smithi121.stdout:9/229: sync 2024-04-05T21:26:39.868 INFO:tasks.workunit.client.1.smithi188.stdout:9/568: mknod dd/d35/cbd 0 2024-04-05T21:26:39.869 INFO:tasks.workunit.client.0.smithi121.stdout:8/283: write d12/d13/d15/f18 [1121069,34748] 0 2024-04-05T21:26:39.871 INFO:tasks.workunit.client.1.smithi188.stdout:2/659: rename d5/db/dd/d24/c32 to d5/db/dd/d36/cd5 0 2024-04-05T21:26:39.872 INFO:tasks.workunit.client.1.smithi188.stdout:4/828: symlink d0/d2/d52/dc1/l11e 0 2024-04-05T21:26:39.872 INFO:tasks.workunit.client.1.smithi188.stdout:8/565: dread d5/dc/f1d [0,4194304] 0 2024-04-05T21:26:39.873 INFO:tasks.workunit.client.1.smithi188.stdout:7/748: creat d3/d4/d1d/d74/d73/fee x:0 0 0 2024-04-05T21:26:39.875 INFO:tasks.workunit.client.1.smithi188.stdout:5/543: rename d6/d82/d40/d42/c76 to d6/d82/d40/cbc 0 2024-04-05T21:26:39.877 INFO:tasks.workunit.client.0.smithi121.stdout:4/430: dread - d5/d11/d26/d49/d85/d29/d2e/f4d zero size 2024-04-05T21:26:39.879 INFO:tasks.workunit.client.0.smithi121.stdout:0/405: write d7/d71/d45/d5a/f69 [82972,78041] 0 2024-04-05T21:26:39.880 INFO:tasks.workunit.client.1.smithi188.stdout:7/749: write d3/d4/db/d55/fe0 [487950,113565] 0 2024-04-05T21:26:39.880 INFO:tasks.workunit.client.1.smithi188.stdout:9/569: creat dd/d3d/fbe x:0 0 0 2024-04-05T21:26:39.881 INFO:tasks.workunit.client.0.smithi121.stdout:2/314: fdatasync d2/d9/d38/f35 0 2024-04-05T21:26:39.883 INFO:tasks.workunit.client.0.smithi121.stdout:9/230: mkdir d6/df/d48 0 2024-04-05T21:26:39.884 INFO:tasks.workunit.client.1.smithi188.stdout:9/570: write dd/d11/d73/d8d/fac [134705,106171] 0 2024-04-05T21:26:39.885 INFO:tasks.workunit.client.1.smithi188.stdout:5/544: rename d6/d15/d5c/d7a/d44/d51/f88 to d6/dad/fbd 0 2024-04-05T21:26:39.886 INFO:tasks.workunit.client.1.smithi188.stdout:9/571: fdatasync dd/d11/d22/f4d 0 2024-04-05T21:26:39.889 INFO:tasks.workunit.client.1.smithi188.stdout:7/750: rename d3/d4/d1d/le9 to d3/d4/db/d2e/d8c/db7/dc4/lef 0 2024-04-05T21:26:39.889 INFO:tasks.workunit.client.1.smithi188.stdout:5/545: fdatasync d6/d17/fbb 0 2024-04-05T21:26:39.890 INFO:tasks.workunit.client.0.smithi121.stdout:4/431: rename d5/d11/d26/d49/d85 to d5/d11/d2c/d8d 0 2024-04-05T21:26:39.891 INFO:tasks.workunit.client.0.smithi121.stdout:8/284: creat d12/d50/d59/d63/f64 x:0 0 0 2024-04-05T21:26:39.892 INFO:tasks.workunit.client.1.smithi188.stdout:7/751: creat d3/d4/d7c/ff0 x:0 0 0 2024-04-05T21:26:39.895 INFO:tasks.workunit.client.0.smithi121.stdout:4/432: dread d5/d11/d28/f5d [4194304,4194304] 0 2024-04-05T21:26:39.897 INFO:tasks.workunit.client.1.smithi188.stdout:5/546: rename d6/c13 to d6/d15/d5c/d7a/d44/d98/cbe 0 2024-04-05T21:26:39.898 INFO:tasks.workunit.client.0.smithi121.stdout:2/315: creat d2/d9/f54 x:0 0 0 2024-04-05T21:26:39.899 INFO:tasks.workunit.client.0.smithi121.stdout:2/316: stat d2/d9/d12 0 2024-04-05T21:26:39.899 INFO:tasks.workunit.client.0.smithi121.stdout:2/317: dread - d2/d9/f54 zero size 2024-04-05T21:26:39.901 INFO:tasks.workunit.client.0.smithi121.stdout:4/433: mknod d5/d11/d2c/d8c/c8e 0 2024-04-05T21:26:39.902 INFO:tasks.workunit.client.0.smithi121.stdout:2/318: rmdir d2/d9/d38 39 2024-04-05T21:26:39.903 INFO:tasks.workunit.client.1.smithi188.stdout:7/752: creat d3/d2c/d5e/ff1 x:0 0 0 2024-04-05T21:26:39.903 INFO:tasks.workunit.client.0.smithi121.stdout:2/319: write d2/d28/f4f [660248,62118] 0 2024-04-05T21:26:39.906 INFO:tasks.workunit.client.1.smithi188.stdout:1/614: dwrite d1/dc/f28 [0,4194304] 0 2024-04-05T21:26:39.907 INFO:tasks.workunit.client.0.smithi121.stdout:0/406: truncate d7/d1c/f55 3270550 0 2024-04-05T21:26:39.911 INFO:tasks.workunit.client.1.smithi188.stdout:5/547: dwrite d6/f9 [0,4194304] 0 2024-04-05T21:26:39.911 INFO:tasks.workunit.client.1.smithi188.stdout:4/829: write d0/d2/f58 [390322,91747] 0 2024-04-05T21:26:39.912 INFO:tasks.workunit.client.0.smithi121.stdout:0/407: link d7/d71/d29/f37 d7/d7e/f89 0 2024-04-05T21:26:39.912 INFO:tasks.workunit.client.0.smithi121.stdout:8/285: dread fc [0,4194304] 0 2024-04-05T21:26:39.914 INFO:tasks.workunit.client.0.smithi121.stdout:8/286: dread - d12/d13/d15/d1d/f58 zero size 2024-04-05T21:26:39.916 INFO:tasks.workunit.client.1.smithi188.stdout:2/660: dread d5/db/d35/d3b/d6b/fa7 [0,4194304] 0 2024-04-05T21:26:39.916 INFO:tasks.workunit.client.1.smithi188.stdout:7/753: write d3/d4/db/d10/f14 [7418105,112877] 0 2024-04-05T21:26:39.917 INFO:tasks.workunit.client.0.smithi121.stdout:2/320: truncate d2/d9/f2f 3508410 0 2024-04-05T21:26:39.917 INFO:tasks.workunit.client.0.smithi121.stdout:8/287: write d12/d13/d15/d1d/f4f [4615690,103625] 0 2024-04-05T21:26:39.917 INFO:tasks.workunit.client.0.smithi121.stdout:0/408: symlink d7/d71/d29/d2a/d2c/l8a 0 2024-04-05T21:26:39.918 INFO:tasks.workunit.client.1.smithi188.stdout:5/548: readlink d6/l14 0 2024-04-05T21:26:39.918 INFO:tasks.workunit.client.0.smithi121.stdout:2/321: dread - d2/d1e/f52 zero size 2024-04-05T21:26:39.920 INFO:tasks.workunit.client.0.smithi121.stdout:0/409: fsync d7/d62/f20 0 2024-04-05T21:26:39.921 INFO:tasks.workunit.client.0.smithi121.stdout:8/288: truncate d12/f3d 636394 0 2024-04-05T21:26:39.922 INFO:tasks.workunit.client.1.smithi188.stdout:2/661: stat d5/db/dd/d67/dcd/d46/d66/fb3 0 2024-04-05T21:26:39.922 INFO:tasks.workunit.client.1.smithi188.stdout:4/830: symlink d0/d14/d15/dd4/l11f 0 2024-04-05T21:26:39.922 INFO:tasks.workunit.client.0.smithi121.stdout:2/322: creat d2/d9/d17/d3a/f55 x:0 0 0 2024-04-05T21:26:39.923 INFO:tasks.workunit.client.0.smithi121.stdout:9/231: dread d6/f17 [0,4194304] 0 2024-04-05T21:26:39.924 INFO:tasks.workunit.client.0.smithi121.stdout:2/323: truncate d2/d1e/f52 878604 0 2024-04-05T21:26:39.925 INFO:tasks.workunit.client.0.smithi121.stdout:9/232: chown d6/df/d1d/l22 122769 1 2024-04-05T21:26:39.926 INFO:tasks.workunit.client.0.smithi121.stdout:8/289: creat d12/d50/d59/f65 x:0 0 0 2024-04-05T21:26:39.927 INFO:tasks.workunit.client.0.smithi121.stdout:8/290: write d12/d13/d15/d1d/d29/f40 [649542,129740] 0 2024-04-05T21:26:39.928 INFO:tasks.workunit.client.0.smithi121.stdout:0/410: dread d7/d71/d45/d5a/f69 [0,4194304] 0 2024-04-05T21:26:39.928 INFO:tasks.workunit.client.0.smithi121.stdout:9/233: truncate d6/df/d1d/f23 393783 0 2024-04-05T21:26:39.930 INFO:tasks.workunit.client.0.smithi121.stdout:2/324: mkdir d2/d9/d56 0 2024-04-05T21:26:39.931 INFO:tasks.workunit.client.0.smithi121.stdout:4/434: dread d5/f16 [0,4194304] 0 2024-04-05T21:26:39.934 INFO:tasks.workunit.client.0.smithi121.stdout:0/411: dread d7/d62/f5c [0,4194304] 0 2024-04-05T21:26:39.934 INFO:tasks.workunit.client.0.smithi121.stdout:8/291: dread d12/d13/f1b [0,4194304] 0 2024-04-05T21:26:39.936 INFO:tasks.workunit.client.0.smithi121.stdout:4/435: rmdir d5/d11/d28 39 2024-04-05T21:26:39.940 INFO:tasks.workunit.client.0.smithi121.stdout:9/234: mkdir d6/df/d1d/d2c/d37/d3a/d49 0 2024-04-05T21:26:39.942 INFO:tasks.workunit.client.0.smithi121.stdout:4/436: mkdir d5/d11/d26/d49/d8f 0 2024-04-05T21:26:39.944 INFO:tasks.workunit.client.0.smithi121.stdout:8/292: dwrite d12/d13/d15/d1d/f36 [0,4194304] 0 2024-04-05T21:26:39.944 INFO:tasks.workunit.client.0.smithi121.stdout:0/412: mkdir d7/d62/d8b 0 2024-04-05T21:26:39.946 INFO:tasks.workunit.client.0.smithi121.stdout:8/293: dread - d12/d50/d59/f60 zero size 2024-04-05T21:26:39.947 INFO:tasks.workunit.client.0.smithi121.stdout:9/235: mkdir d6/df/d1d/d2c/d37/d3a/d4a 0 2024-04-05T21:26:39.950 INFO:tasks.workunit.client.0.smithi121.stdout:4/437: mknod d5/d11/d26/d47/c90 0 2024-04-05T21:26:39.953 INFO:tasks.workunit.client.0.smithi121.stdout:2/325: dread d2/f1a [0,4194304] 0 2024-04-05T21:26:39.954 INFO:tasks.workunit.client.0.smithi121.stdout:2/326: fsync d2/d1e/f52 0 2024-04-05T21:26:39.955 INFO:tasks.workunit.client.0.smithi121.stdout:8/294: dwrite d12/d13/d15/d1d/f4f [0,4194304] 0 2024-04-05T21:26:39.959 INFO:tasks.workunit.client.0.smithi121.stdout:9/236: mknod d6/c4b 0 2024-04-05T21:26:39.961 INFO:tasks.workunit.client.0.smithi121.stdout:0/413: rmdir d7/d88 0 2024-04-05T21:26:39.962 INFO:tasks.workunit.client.0.smithi121.stdout:0/414: fdatasync d7/f9 0 2024-04-05T21:26:39.962 INFO:tasks.workunit.client.0.smithi121.stdout:8/295: readlink d12/d13/d15/d1d/l46 0 2024-04-05T21:26:39.964 INFO:tasks.workunit.client.0.smithi121.stdout:4/438: unlink d5/d11/d2c/d8d/d64/c7d 0 2024-04-05T21:26:39.964 INFO:tasks.workunit.client.0.smithi121.stdout:2/327: dwrite d2/d9/d17/d3a/f4a [0,4194304] 0 2024-04-05T21:26:39.966 INFO:tasks.workunit.client.0.smithi121.stdout:0/415: creat d7/d71/d29/d2a/d2c/f8c x:0 0 0 2024-04-05T21:26:39.967 INFO:tasks.workunit.client.0.smithi121.stdout:9/237: creat d6/df/d48/f4c x:0 0 0 2024-04-05T21:26:39.967 INFO:tasks.workunit.client.0.smithi121.stdout:2/328: symlink d2/d1e/l57 0 2024-04-05T21:26:39.968 INFO:tasks.workunit.client.0.smithi121.stdout:4/439: symlink d5/d11/l91 0 2024-04-05T21:26:39.969 INFO:tasks.workunit.client.0.smithi121.stdout:9/238: truncate d6/f40 911176 0 2024-04-05T21:26:39.970 INFO:tasks.workunit.client.0.smithi121.stdout:0/416: creat d7/d71/d29/d2a/d2c/f8d x:0 0 0 2024-04-05T21:26:39.970 INFO:tasks.workunit.client.0.smithi121.stdout:4/440: dread - d5/d11/f8a zero size 2024-04-05T21:26:39.971 INFO:tasks.workunit.client.0.smithi121.stdout:2/329: write d2/d9/d17/f18 [4829439,112523] 0 2024-04-05T21:26:39.972 INFO:tasks.workunit.client.0.smithi121.stdout:9/239: read d6/f15 [217451,44642] 0 2024-04-05T21:26:39.973 INFO:tasks.workunit.client.0.smithi121.stdout:8/296: dwrite d12/d50/d59/d63/f64 [0,4194304] 0 2024-04-05T21:26:39.974 INFO:tasks.workunit.client.0.smithi121.stdout:0/417: write d7/dc/f2d [1823826,58085] 0 2024-04-05T21:26:39.977 INFO:tasks.workunit.client.0.smithi121.stdout:0/418: write f2 [4795112,2786] 0 2024-04-05T21:26:39.977 INFO:tasks.workunit.client.0.smithi121.stdout:4/441: symlink d5/d11/d26/d47/l92 0 2024-04-05T21:26:39.980 INFO:tasks.workunit.client.0.smithi121.stdout:9/240: unlink d6/f2a 0 2024-04-05T21:26:39.980 INFO:tasks.workunit.client.0.smithi121.stdout:8/297: mknod d12/d13/d15/d1d/d29/c66 0 2024-04-05T21:26:39.982 INFO:tasks.workunit.client.0.smithi121.stdout:2/330: dwrite d2/d1e/f24 [0,4194304] 0 2024-04-05T21:26:39.983 INFO:tasks.workunit.client.0.smithi121.stdout:9/241: dread d6/f40 [0,4194304] 0 2024-04-05T21:26:39.985 INFO:tasks.workunit.client.1.smithi188.stdout:9/572: dwrite dd/d34/f23 [0,4194304] 0 2024-04-05T21:26:39.985 INFO:tasks.workunit.client.1.smithi188.stdout:2/662: chown d5/d5f/f90 9990227 1 2024-04-05T21:26:39.985 INFO:tasks.workunit.client.1.smithi188.stdout:0/483: sync 2024-04-05T21:26:39.985 INFO:tasks.workunit.client.1.smithi188.stdout:5/549: chown d6/d15/d5c/d7a/d44/l92 330069 1 2024-04-05T21:26:39.986 INFO:tasks.workunit.client.1.smithi188.stdout:1/615: truncate d1/dc/d16/f92 3714436 0 2024-04-05T21:26:39.987 INFO:tasks.workunit.client.0.smithi121.stdout:9/242: write d6/df/d48/f4c [596040,27728] 0 2024-04-05T21:26:39.987 INFO:tasks.workunit.client.0.smithi121.stdout:0/419: mkdir d7/d71/d45/d46/d53/d8e 0 2024-04-05T21:26:39.989 INFO:tasks.workunit.client.0.smithi121.stdout:8/298: unlink d12/d13/d15/d1d/f53 0 2024-04-05T21:26:39.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:39 smithi121 ceph-mon[39723]: pgmap v18: 65 pgs: 65 active+clean; 1.3 GiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 34 MiB/s rd, 154 MiB/s wr, 262 op/s 2024-04-05T21:26:39.990 INFO:tasks.workunit.client.0.smithi121.stdout:2/331: write d2/f11 [1631135,69838] 0 2024-04-05T21:26:39.990 INFO:tasks.workunit.client.0.smithi121.stdout:4/442: creat d5/d11/d28/d3b/f93 x:0 0 0 2024-04-05T21:26:39.991 INFO:tasks.workunit.client.0.smithi121.stdout:9/243: symlink d6/d36/l4d 0 2024-04-05T21:26:39.995 INFO:tasks.workunit.client.1.smithi188.stdout:7/754: creat d3/d4/db/d2e/d3b/d94/ff2 x:0 0 0 2024-04-05T21:26:40.001 INFO:tasks.workunit.client.0.smithi121.stdout:0/420: creat d7/d71/f8f x:0 0 0 2024-04-05T21:26:40.002 INFO:tasks.workunit.client.0.smithi121.stdout:2/332: dwrite d2/d9/d17/f18 [0,4194304] 0 2024-04-05T21:26:40.002 INFO:tasks.workunit.client.0.smithi121.stdout:9/244: mknod d6/c4e 0 2024-04-05T21:26:40.002 INFO:tasks.workunit.client.0.smithi121.stdout:2/333: chown d2/d9/d1c 0 1 2024-04-05T21:26:40.002 INFO:tasks.workunit.client.0.smithi121.stdout:0/421: creat d7/d62/d3c/d6b/f90 x:0 0 0 2024-04-05T21:26:40.002 INFO:tasks.workunit.client.0.smithi121.stdout:9/245: mknod d6/d36/c4f 0 2024-04-05T21:26:40.002 INFO:tasks.workunit.client.1.smithi188.stdout:5/550: unlink d6/d15/d1b/l25 0 2024-04-05T21:26:40.003 INFO:tasks.workunit.client.1.smithi188.stdout:4/831: symlink d0/d14/d15/d53/d82/dd5/df4/l120 0 2024-04-05T21:26:40.003 INFO:tasks.workunit.client.0.smithi121.stdout:9/246: chown d6/df/d1d/d2c 6 1 2024-04-05T21:26:40.004 INFO:tasks.workunit.client.1.smithi188.stdout:2/663: symlink d5/db/dd/d36/d3c/d48/d7a/d7c/db5/ld6 0 2024-04-05T21:26:40.005 INFO:tasks.workunit.client.0.smithi121.stdout:9/247: dread d6/f40 [0,4194304] 0 2024-04-05T21:26:40.005 INFO:tasks.workunit.client.1.smithi188.stdout:1/616: getdents d1/db2 0 2024-04-05T21:26:40.009 INFO:tasks.workunit.client.0.smithi121.stdout:2/334: creat d2/d9/d38/d2d/f58 x:0 0 0 2024-04-05T21:26:40.010 INFO:tasks.workunit.client.1.smithi188.stdout:4/832: mknod d0/d2/d2a/d71/c121 0 2024-04-05T21:26:40.012 INFO:tasks.workunit.client.0.smithi121.stdout:9/248: getdents d6/df/d1d/d2c 0 2024-04-05T21:26:40.012 INFO:tasks.workunit.client.1.smithi188.stdout:2/664: mkdir d5/db/dd/d36/d3c/d48/dd7 0 2024-04-05T21:26:40.013 INFO:tasks.workunit.client.1.smithi188.stdout:1/617: creat d1/d7/dbf/fc4 x:0 0 0 2024-04-05T21:26:40.014 INFO:tasks.workunit.client.0.smithi121.stdout:9/249: mkdir d6/df/d50 0 2024-04-05T21:26:40.015 INFO:tasks.workunit.client.1.smithi188.stdout:5/551: getdents d6/d15/d1b/d4f 0 2024-04-05T21:26:40.015 INFO:tasks.workunit.client.1.smithi188.stdout:5/552: fdatasync d6/d82/d40/fb6 0 2024-04-05T21:26:40.017 INFO:tasks.workunit.client.1.smithi188.stdout:2/665: creat d5/db/dd/d67/dcd/d46/d5d/fd8 x:0 0 0 2024-04-05T21:26:40.019 INFO:tasks.workunit.client.1.smithi188.stdout:2/666: stat d5/db/dd/d36/d3c/d63/d75/d87/f9c 0 2024-04-05T21:26:40.021 INFO:tasks.workunit.client.1.smithi188.stdout:2/667: write d5/d5f/d98/fd2 [972732,32007] 0 2024-04-05T21:26:40.022 INFO:tasks.workunit.client.0.smithi121.stdout:9/250: dwrite d6/df/d1d/f2b [0,4194304] 0 2024-04-05T21:26:40.024 INFO:tasks.workunit.client.0.smithi121.stdout:9/251: chown d6/d36/l31 15248 1 2024-04-05T21:26:40.024 INFO:tasks.workunit.client.0.smithi121.stdout:9/252: stat d6/f15 0 2024-04-05T21:26:40.025 INFO:tasks.workunit.client.1.smithi188.stdout:5/553: dwrite d6/f38 [0,4194304] 0 2024-04-05T21:26:40.026 INFO:tasks.workunit.client.0.smithi121.stdout:9/253: chown d6/ld 102 1 2024-04-05T21:26:40.027 INFO:tasks.workunit.client.0.smithi121.stdout:9/254: write d6/df/f2e [1193142,49199] 0 2024-04-05T21:26:40.029 INFO:tasks.workunit.client.1.smithi188.stdout:5/554: mkdir d6/d15/d5c/d7a/d44/d98/dbf 0 2024-04-05T21:26:40.031 INFO:tasks.workunit.client.1.smithi188.stdout:2/668: dwrite d5/db/dd/d24/f52 [0,4194304] 0 2024-04-05T21:26:40.037 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:39 smithi188 ceph-mon[49715]: pgmap v18: 65 pgs: 65 active+clean; 1.3 GiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 34 MiB/s rd, 154 MiB/s wr, 262 op/s 2024-04-05T21:26:40.037 INFO:tasks.workunit.client.1.smithi188.stdout:9/573: write dd/d11/d73/f75 [402396,127160] 0 2024-04-05T21:26:40.038 INFO:tasks.workunit.client.1.smithi188.stdout:1/618: fsync d1/dc/d16/f92 0 2024-04-05T21:26:40.042 INFO:tasks.workunit.client.1.smithi188.stdout:9/574: creat dd/d49/fbf x:0 0 0 2024-04-05T21:26:40.043 INFO:tasks.workunit.client.0.smithi121.stdout:6/292: sync 2024-04-05T21:26:40.043 INFO:tasks.workunit.client.0.smithi121.stdout:3/234: sync 2024-04-05T21:26:40.045 INFO:tasks.workunit.client.0.smithi121.stdout:3/235: mknod d0/d3/c4d 0 2024-04-05T21:26:40.049 INFO:tasks.workunit.client.0.smithi121.stdout:6/293: rename d1/d2/d3/d5/d36 to d1/d2/d3/d59 0 2024-04-05T21:26:40.050 INFO:tasks.workunit.client.1.smithi188.stdout:9/575: unlink dd/l30 0 2024-04-05T21:26:40.051 INFO:tasks.workunit.client.0.smithi121.stdout:3/236: creat d0/d3/d24/d3d/d1a/f4e x:0 0 0 2024-04-05T21:26:40.051 INFO:tasks.workunit.client.1.smithi188.stdout:1/619: symlink d1/d7/dbf/lc5 0 2024-04-05T21:26:40.051 INFO:tasks.workunit.client.0.smithi121.stdout:3/237: chown d0/d3/d24/d3d/d1a/d1e 26 1 2024-04-05T21:26:40.052 INFO:tasks.workunit.client.1.smithi188.stdout:2/669: rename d5/db/dd/d67/dcd/d46/fc0 to d5/db/dd/d36/d3c/d48/dcf/fd9 0 2024-04-05T21:26:40.053 INFO:tasks.workunit.client.1.smithi188.stdout:4/833: dread d0/d14/f80 [0,4194304] 0 2024-04-05T21:26:40.054 INFO:tasks.workunit.client.0.smithi121.stdout:6/294: rename d1/d2/dc/l16 to d1/d2/d1b/d2e/l5a 0 2024-04-05T21:26:40.054 INFO:tasks.workunit.client.1.smithi188.stdout:7/755: dwrite d3/d4/d1d/f2b [0,4194304] 0 2024-04-05T21:26:40.056 INFO:tasks.workunit.client.1.smithi188.stdout:9/576: chown dd/d49/f66 0 1 2024-04-05T21:26:40.058 INFO:tasks.workunit.client.1.smithi188.stdout:5/555: dwrite d6/d82/d40/d42/d81/d95/f9d [0,4194304] 0 2024-04-05T21:26:40.059 INFO:tasks.workunit.client.1.smithi188.stdout:0/484: dwrite d4/f5f [0,4194304] 0 2024-04-05T21:26:40.059 INFO:tasks.workunit.client.0.smithi121.stdout:6/295: creat d1/d2/dc/f5b x:0 0 0 2024-04-05T21:26:40.060 INFO:tasks.workunit.client.0.smithi121.stdout:3/238: rename d0/d3/de/c19 to d0/d3/d24/d3d/d1a/d1e/c4f 0 2024-04-05T21:26:40.062 INFO:tasks.workunit.client.0.smithi121.stdout:6/296: truncate d1/d2/d1b/d1e/f2a 816738 0 2024-04-05T21:26:40.062 INFO:tasks.workunit.client.1.smithi188.stdout:1/620: fdatasync d1/dc/d16/d46/d88/f1d 0 2024-04-05T21:26:40.065 INFO:tasks.workunit.client.0.smithi121.stdout:3/239: symlink d0/d3/d24/d3d/l50 0 2024-04-05T21:26:40.066 INFO:tasks.workunit.client.1.smithi188.stdout:4/834: mknod d0/d14/d15/d53/d82/d8f/d95/c122 0 2024-04-05T21:26:40.068 INFO:tasks.workunit.client.1.smithi188.stdout:1/621: dread - d1/d7/f5c zero size 2024-04-05T21:26:40.069 INFO:tasks.workunit.client.1.smithi188.stdout:2/670: mknod d5/db/dd/d36/d3c/d48/dd7/cda 0 2024-04-05T21:26:40.070 INFO:tasks.workunit.client.1.smithi188.stdout:9/577: fsync dd/d3d/f7a 0 2024-04-05T21:26:40.072 INFO:tasks.workunit.client.1.smithi188.stdout:0/485: dread d4/d6/d46/f59 [0,4194304] 0 2024-04-05T21:26:40.072 INFO:tasks.workunit.client.1.smithi188.stdout:5/556: dread d6/d17/fb8 [0,4194304] 0 2024-04-05T21:26:40.072 INFO:tasks.workunit.client.1.smithi188.stdout:4/835: creat d0/d14/d2f/ddd/f123 x:0 0 0 2024-04-05T21:26:40.074 INFO:tasks.workunit.client.0.smithi121.stdout:6/297: mknod d1/d2/d3/d5/d1f/d2c/d4f/c5c 0 2024-04-05T21:26:40.074 INFO:tasks.workunit.client.0.smithi121.stdout:3/240: mknod d0/d3/d24/d3d/d1a/c51 0 2024-04-05T21:26:40.075 INFO:tasks.workunit.client.0.smithi121.stdout:8/299: write d12/d13/d15/d1d/f27 [1311453,82222] 0 2024-04-05T21:26:40.075 INFO:tasks.workunit.client.0.smithi121.stdout:0/422: write d7/f3a [573270,92079] 0 2024-04-05T21:26:40.075 INFO:tasks.workunit.client.0.smithi121.stdout:9/255: truncate d6/df/d1d/f33 5211361 0 2024-04-05T21:26:40.076 INFO:tasks.workunit.client.1.smithi188.stdout:2/671: mknod d5/db/dd/d67/dcd/d46/d66/d8f/cdb 0 2024-04-05T21:26:40.076 INFO:tasks.workunit.client.0.smithi121.stdout:3/241: dread - d0/d3/d24/d3f/f4b zero size 2024-04-05T21:26:40.077 INFO:tasks.workunit.client.0.smithi121.stdout:6/298: write d1/d2/d3/d5/fd [3335635,122173] 0 2024-04-05T21:26:40.077 INFO:tasks.workunit.client.1.smithi188.stdout:0/486: chown d4/d6/f12 84 1 2024-04-05T21:26:40.077 INFO:tasks.workunit.client.1.smithi188.stdout:1/622: read - d1/f48 zero size 2024-04-05T21:26:40.078 INFO:tasks.workunit.client.0.smithi121.stdout:0/423: dread - d7/d71/d29/d2a/d2c/f7b zero size 2024-04-05T21:26:40.079 INFO:tasks.workunit.client.1.smithi188.stdout:4/836: truncate d0/d2/d2a/d34/d6b/f7e 244371 0 2024-04-05T21:26:40.080 INFO:tasks.workunit.client.0.smithi121.stdout:8/300: rename d12/d50/d59 to d12/d50/d61/d67 0 2024-04-05T21:26:40.081 INFO:tasks.workunit.client.0.smithi121.stdout:9/256: symlink d6/l51 0 2024-04-05T21:26:40.081 INFO:tasks.workunit.client.0.smithi121.stdout:8/301: chown fa 2 1 2024-04-05T21:26:40.081 INFO:tasks.workunit.client.1.smithi188.stdout:2/672: mknod d5/db/dd/d36/d3c/d63/d75/cdc 0 2024-04-05T21:26:40.083 INFO:tasks.workunit.client.0.smithi121.stdout:8/302: readlink d12/d13/d15/d1d/l3f 0 2024-04-05T21:26:40.084 INFO:tasks.workunit.client.1.smithi188.stdout:1/623: mknod d1/d7/dbf/cc6 0 2024-04-05T21:26:40.085 INFO:tasks.workunit.client.1.smithi188.stdout:9/578: rename dd/d49 to dd/d11/d1f/d26/dc0 0 2024-04-05T21:26:40.086 INFO:tasks.workunit.client.1.smithi188.stdout:0/487: write d4/d6/d13/d23/d24/d57/f92 [985220,84011] 0 2024-04-05T21:26:40.086 INFO:tasks.workunit.client.0.smithi121.stdout:3/242: dwrite d0/f2d [0,4194304] 0 2024-04-05T21:26:40.086 INFO:tasks.workunit.client.1.smithi188.stdout:5/557: dwrite d6/d17/f80 [0,4194304] 0 2024-04-05T21:26:40.087 INFO:tasks.workunit.client.0.smithi121.stdout:9/257: dread - d6/df/d1d/d2c/d37/d3c/f3d zero size 2024-04-05T21:26:40.087 INFO:tasks.workunit.client.1.smithi188.stdout:2/673: dread - d5/db/dd/d36/d3c/d48/d78/f83 zero size 2024-04-05T21:26:40.088 INFO:tasks.workunit.client.0.smithi121.stdout:3/243: stat d0/d3/d24/d3d/d1a/c51 0 2024-04-05T21:26:40.088 INFO:tasks.workunit.client.0.smithi121.stdout:8/303: mknod d12/d13/d15/c68 0 2024-04-05T21:26:40.090 INFO:tasks.workunit.client.1.smithi188.stdout:2/674: chown d5/db/dd/d67/dcd/d46/d66/d8f 1289216997 1 2024-04-05T21:26:40.090 INFO:tasks.workunit.client.0.smithi121.stdout:3/244: unlink d0/d3/c4d 0 2024-04-05T21:26:40.092 INFO:tasks.workunit.client.0.smithi121.stdout:8/304: symlink d12/d13/d15/d1d/l69 0 2024-04-05T21:26:40.093 INFO:tasks.workunit.client.1.smithi188.stdout:0/488: unlink d4/d6/f12 0 2024-04-05T21:26:40.095 INFO:tasks.workunit.client.0.smithi121.stdout:3/245: read d0/f14 [3932210,97107] 0 2024-04-05T21:26:40.098 INFO:tasks.workunit.client.1.smithi188.stdout:5/558: rename d6/d15/f79 to d6/d82/d40/d42/d81/da6/db2/fc0 0 2024-04-05T21:26:40.099 INFO:tasks.workunit.client.0.smithi121.stdout:3/246: symlink d0/d3/d24/l52 0 2024-04-05T21:26:40.099 INFO:tasks.workunit.client.1.smithi188.stdout:0/489: write d4/d8/db/d21/d4a/d73/f94 [1250607,3909] 0 2024-04-05T21:26:40.101 INFO:tasks.workunit.client.1.smithi188.stdout:9/579: getdents dd/d11/d1f/d25 0 2024-04-05T21:26:40.102 INFO:tasks.workunit.client.1.smithi188.stdout:5/559: mkdir d6/d82/d40/dc1 0 2024-04-05T21:26:40.103 INFO:tasks.workunit.client.0.smithi121.stdout:3/247: symlink d0/d3/d24/d3d/l53 0 2024-04-05T21:26:40.104 INFO:tasks.workunit.client.1.smithi188.stdout:9/580: dread - dd/d11/d1f/d25/d8a/fb3 zero size 2024-04-05T21:26:40.105 INFO:tasks.workunit.client.1.smithi188.stdout:0/490: truncate d4/d8/db/d21/f90 5087037 0 2024-04-05T21:26:40.106 INFO:tasks.workunit.client.1.smithi188.stdout:2/675: dread d5/db/dd/da4/fc2 [0,4194304] 0 2024-04-05T21:26:40.106 INFO:tasks.workunit.client.0.smithi121.stdout:8/305: rename d12/d50/d61/d67 to d12/d50/d6a 0 2024-04-05T21:26:40.108 INFO:tasks.workunit.client.1.smithi188.stdout:5/560: symlink d6/d82/d40/d42/d81/d95/lc2 0 2024-04-05T21:26:40.109 INFO:tasks.workunit.client.0.smithi121.stdout:8/306: symlink d12/d50/d6a/d63/l6b 0 2024-04-05T21:26:40.109 INFO:tasks.workunit.client.1.smithi188.stdout:5/561: chown d6/d17/c3f 256562 1 2024-04-05T21:26:40.110 INFO:tasks.workunit.client.0.smithi121.stdout:3/248: dwrite d0/f15 [4194304,4194304] 0 2024-04-05T21:26:40.110 INFO:tasks.workunit.client.0.smithi121.stdout:8/307: stat d12/c35 0 2024-04-05T21:26:40.110 INFO:tasks.workunit.client.1.smithi188.stdout:2/676: mknod d5/db/dd/d36/d3c/d48/d7a/d7c/db5/cdd 0 2024-04-05T21:26:40.111 INFO:tasks.workunit.client.0.smithi121.stdout:3/249: fsync d0/d3/d24/d3d/d1a/f33 0 2024-04-05T21:26:40.113 INFO:tasks.workunit.client.1.smithi188.stdout:5/562: rename d6/d15/d5c/d7a/d44/d51/c78 to d6/d82/cc3 0 2024-04-05T21:26:40.114 INFO:tasks.workunit.client.0.smithi121.stdout:3/250: creat d0/d3/f54 x:0 0 0 2024-04-05T21:26:40.117 INFO:tasks.workunit.client.0.smithi121.stdout:3/251: read d0/d3/f30 [1517184,82342] 0 2024-04-05T21:26:40.117 INFO:tasks.workunit.client.1.smithi188.stdout:5/563: mkdir d6/d17/d23/d7d/dc4 0 2024-04-05T21:26:40.120 INFO:tasks.workunit.client.1.smithi188.stdout:5/564: creat d6/d17/d23/d85/fc5 x:0 0 0 2024-04-05T21:26:40.121 INFO:tasks.workunit.client.1.smithi188.stdout:9/581: rmdir dd 39 2024-04-05T21:26:40.123 INFO:tasks.workunit.client.1.smithi188.stdout:7/756: dread d3/f1e [0,4194304] 0 2024-04-05T21:26:40.124 INFO:tasks.workunit.client.1.smithi188.stdout:7/757: readlink d3/d4/db/d55/d9d/lb2 0 2024-04-05T21:26:40.125 INFO:tasks.workunit.client.1.smithi188.stdout:7/758: stat d3/d4/db/d2e/d49/c9e 0 2024-04-05T21:26:40.127 INFO:tasks.workunit.client.1.smithi188.stdout:5/565: rename d6/d15/d5c/d7a/d44/da0 to d6/d15/d5c/d7a/d44/d51/dc6 0 2024-04-05T21:26:40.127 INFO:tasks.workunit.client.1.smithi188.stdout:4/837: dread d0/d2/d4f/f5c [0,4194304] 0 2024-04-05T21:26:40.128 INFO:tasks.workunit.client.1.smithi188.stdout:5/566: read - d6/d82/d40/f90 zero size 2024-04-05T21:26:40.131 INFO:tasks.workunit.client.1.smithi188.stdout:4/838: symlink d0/d14/dd8/l124 0 2024-04-05T21:26:40.132 INFO:tasks.workunit.client.1.smithi188.stdout:4/839: truncate d0/dbe/ffa 231794 0 2024-04-05T21:26:40.132 INFO:tasks.workunit.client.1.smithi188.stdout:7/759: mkdir d3/d4/db/d10/d3c/df3 0 2024-04-05T21:26:40.132 INFO:tasks.workunit.client.1.smithi188.stdout:4/840: stat d0/d14/d2f 0 2024-04-05T21:26:40.133 INFO:tasks.workunit.client.1.smithi188.stdout:0/491: write d4/d8/db/d21/f34 [5223247,10151] 0 2024-04-05T21:26:40.135 INFO:tasks.workunit.client.0.smithi121.stdout:8/308: dread d12/d13/d15/f19 [0,4194304] 0 2024-04-05T21:26:40.136 INFO:tasks.workunit.client.1.smithi188.stdout:0/492: write d4/d6/d61/f8f [405192,104322] 0 2024-04-05T21:26:40.136 INFO:tasks.workunit.client.1.smithi188.stdout:7/760: creat d3/d2c/dce/ff4 x:0 0 0 2024-04-05T21:26:40.141 INFO:tasks.workunit.client.0.smithi121.stdout:8/309: dwrite d12/d13/d15/d1d/f42 [0,4194304] 0 2024-04-05T21:26:40.142 INFO:tasks.workunit.client.1.smithi188.stdout:7/761: creat d3/d4/db/d10/d3c/ff5 x:0 0 0 2024-04-05T21:26:40.142 INFO:tasks.workunit.client.0.smithi121.stdout:8/310: write d12/d13/d15/d1d/f4f [4783201,14085] 0 2024-04-05T21:26:40.143 INFO:tasks.workunit.client.1.smithi188.stdout:0/493: rename d4/d8/db/d21/d4a/d73 to d4/d8/db/d21/d4a/d73/d79/dad 22 2024-04-05T21:26:40.144 INFO:tasks.workunit.client.0.smithi121.stdout:8/311: stat d12/f3d 0 2024-04-05T21:26:40.144 INFO:tasks.workunit.client.0.smithi121.stdout:8/312: read - d12/d50/d6a/f60 zero size 2024-04-05T21:26:40.147 INFO:tasks.workunit.client.1.smithi188.stdout:4/841: dwrite d0/d14/d15/dd4/ddf/ff7 [0,4194304] 0 2024-04-05T21:26:40.152 INFO:tasks.workunit.client.1.smithi188.stdout:9/582: dwrite dd/d11/d22/f56 [0,4194304] 0 2024-04-05T21:26:40.154 INFO:tasks.workunit.client.1.smithi188.stdout:4/842: creat d0/d14/df6/f125 x:0 0 0 2024-04-05T21:26:40.155 INFO:tasks.workunit.client.1.smithi188.stdout:5/567: dwrite d6/f11 [4194304,4194304] 0 2024-04-05T21:26:40.158 INFO:tasks.workunit.client.1.smithi188.stdout:7/762: rename d3/d2c/dce/f1f to d3/d4/db/d2e/d8c/db7/dd6/ff6 0 2024-04-05T21:26:40.160 INFO:tasks.workunit.client.1.smithi188.stdout:0/494: link d4/d8/db/d42/d38/f82 d4/d6/d61/fae 0 2024-04-05T21:26:40.165 INFO:tasks.workunit.client.1.smithi188.stdout:9/583: unlink dd/d11/d1f/d26/dc0/d6b/la8 0 2024-04-05T21:26:40.166 INFO:tasks.workunit.client.1.smithi188.stdout:5/568: creat d6/d15/d5c/d7a/fc7 x:0 0 0 2024-04-05T21:26:40.167 INFO:tasks.workunit.client.1.smithi188.stdout:7/763: truncate d3/d2c/f72 1647190 0 2024-04-05T21:26:40.169 INFO:tasks.workunit.client.1.smithi188.stdout:4/843: mkdir d0/d2/d52/d126 0 2024-04-05T21:26:40.170 INFO:tasks.workunit.client.1.smithi188.stdout:7/764: fdatasync d3/d2c/d5e/d8e/fac 0 2024-04-05T21:26:40.171 INFO:tasks.workunit.client.1.smithi188.stdout:0/495: dwrite d4/d6/d13/d23/d24/d57/f92 [0,4194304] 0 2024-04-05T21:26:40.173 INFO:tasks.workunit.client.1.smithi188.stdout:4/844: write d0/d2/d2a/d34/fe8 [49092,87470] 0 2024-04-05T21:26:40.174 INFO:tasks.workunit.client.1.smithi188.stdout:0/496: write d4/d8/f9f [583704,81283] 0 2024-04-05T21:26:40.175 INFO:tasks.workunit.client.1.smithi188.stdout:5/569: write d6/d17/d39/fb3 [40546,74147] 0 2024-04-05T21:26:40.177 INFO:tasks.workunit.client.1.smithi188.stdout:0/497: chown d4/d8/db/d21/d4a/d73 42886 1 2024-04-05T21:26:40.177 INFO:tasks.workunit.client.1.smithi188.stdout:4/845: chown d0/d14/d2f/l45 776802369 1 2024-04-05T21:26:40.178 INFO:tasks.workunit.client.1.smithi188.stdout:7/765: mknod d3/d4/d1d/da3/cf7 0 2024-04-05T21:26:40.180 INFO:tasks.workunit.client.1.smithi188.stdout:9/584: dread dd/d11/d1f/d26/dc0/f52 [0,4194304] 0 2024-04-05T21:26:40.181 INFO:tasks.workunit.client.1.smithi188.stdout:4/846: fsync d0/d14/d15/d1b/f3d 0 2024-04-05T21:26:40.181 INFO:tasks.workunit.client.1.smithi188.stdout:5/570: symlink d6/d15/d5c/lc8 0 2024-04-05T21:26:40.181 INFO:tasks.workunit.client.1.smithi188.stdout:0/498: truncate d4/d8/f3d 509244 0 2024-04-05T21:26:40.183 INFO:tasks.workunit.client.1.smithi188.stdout:4/847: read d0/d14/d15/d78/fef [631641,79165] 0 2024-04-05T21:26:40.187 INFO:tasks.workunit.client.1.smithi188.stdout:0/499: link d4/d6/d13/da1/fa3 d4/d8/db/d42/d71/faf 0 2024-04-05T21:26:40.188 INFO:tasks.workunit.client.1.smithi188.stdout:7/766: dwrite d3/d4/d1d/d74/f59 [4194304,4194304] 0 2024-04-05T21:26:40.189 INFO:tasks.workunit.client.1.smithi188.stdout:4/848: unlink d0/d14/d15/dd4/ddf/ff7 0 2024-04-05T21:26:40.191 INFO:tasks.workunit.client.1.smithi188.stdout:4/849: chown d0/d2/d2a/d34/d6b/l9b 14499 1 2024-04-05T21:26:40.192 INFO:tasks.workunit.client.1.smithi188.stdout:7/767: creat d3/d2c/dce/ff8 x:0 0 0 2024-04-05T21:26:40.193 INFO:tasks.workunit.client.1.smithi188.stdout:4/850: chown d0/d14/d15/d53/d82/d8f/d95 1589 1 2024-04-05T21:26:40.193 INFO:tasks.workunit.client.1.smithi188.stdout:0/500: dread d4/d8/db/d21/f90 [0,4194304] 0 2024-04-05T21:26:40.195 INFO:tasks.workunit.client.1.smithi188.stdout:4/851: fdatasync d0/d14/d15/d78/db1/f7b 0 2024-04-05T21:26:40.198 INFO:tasks.workunit.client.1.smithi188.stdout:7/768: creat d3/da5/dc8/ff9 x:0 0 0 2024-04-05T21:26:40.199 INFO:tasks.workunit.client.1.smithi188.stdout:0/501: mknod d4/d8/db/d42/d71/d80/cb0 0 2024-04-05T21:26:40.199 INFO:tasks.workunit.client.1.smithi188.stdout:7/769: rmdir d3/d4/db/d2e/d37 39 2024-04-05T21:26:40.200 INFO:tasks.workunit.client.1.smithi188.stdout:0/502: unlink d4/d6/d46/f8b 0 2024-04-05T21:26:40.200 INFO:tasks.workunit.client.1.smithi188.stdout:7/770: fdatasync d3/d4/db/d2e/d8c/db7/fbc 0 2024-04-05T21:26:40.204 INFO:tasks.workunit.client.1.smithi188.stdout:4/852: dwrite d0/d14/d15/d1b/f40 [4194304,4194304] 0 2024-04-05T21:26:40.207 INFO:tasks.workunit.client.1.smithi188.stdout:0/503: creat d4/d6/d13/da1/fb1 x:0 0 0 2024-04-05T21:26:40.210 INFO:tasks.workunit.client.1.smithi188.stdout:7/771: rename d3/d4/db/c5a to d3/d4/db/d10/d5f/dd4/cfa 0 2024-04-05T21:26:40.210 INFO:tasks.workunit.client.1.smithi188.stdout:4/853: fdatasync d0/d2/d17/f44 0 2024-04-05T21:26:40.213 INFO:tasks.workunit.client.1.smithi188.stdout:7/772: mkdir d3/d2c/d5e/d91/dfb 0 2024-04-05T21:26:40.214 INFO:tasks.workunit.client.1.smithi188.stdout:4/854: rmdir d0/d81 39 2024-04-05T21:26:40.216 INFO:tasks.workunit.client.1.smithi188.stdout:7/773: rmdir d3/d4/db/d55 39 2024-04-05T21:26:40.218 INFO:tasks.workunit.client.1.smithi188.stdout:4/855: write d0/d14/d15/d78/fe2 [783098,87945] 0 2024-04-05T21:26:40.219 INFO:tasks.workunit.client.1.smithi188.stdout:0/504: creat d4/d8/d10/fb2 x:0 0 0 2024-04-05T21:26:40.220 INFO:tasks.workunit.client.1.smithi188.stdout:4/856: chown d0/d2/d17/c33 1477985281 1 2024-04-05T21:26:40.221 INFO:tasks.workunit.client.1.smithi188.stdout:7/774: creat d3/d4/db/d2e/d3b/d94/ffc x:0 0 0 2024-04-05T21:26:40.222 INFO:tasks.workunit.client.1.smithi188.stdout:4/857: dread - d0/d14/d15/d53/ff0 zero size 2024-04-05T21:26:40.224 INFO:tasks.workunit.client.1.smithi188.stdout:0/505: creat d4/d8/db/d42/fb3 x:0 0 0 2024-04-05T21:26:40.225 INFO:tasks.workunit.client.1.smithi188.stdout:7/775: write d3/d4/db/d10/d5f/fcc [217276,73272] 0 2024-04-05T21:26:40.227 INFO:tasks.workunit.client.1.smithi188.stdout:4/858: symlink d0/d2/d2a/d34/d6b/l127 0 2024-04-05T21:26:40.228 INFO:tasks.workunit.client.1.smithi188.stdout:9/585: dwrite dd/d11/d1f/d26/dc0/d6b/f94 [0,4194304] 0 2024-04-05T21:26:40.232 INFO:tasks.workunit.client.1.smithi188.stdout:9/586: write dd/d11/d1f/f6e [4581278,106570] 0 2024-04-05T21:26:40.232 INFO:tasks.workunit.client.1.smithi188.stdout:0/506: creat d4/d8/fb4 x:0 0 0 2024-04-05T21:26:40.235 INFO:tasks.workunit.client.1.smithi188.stdout:9/587: truncate dd/d3d/fbe 87163 0 2024-04-05T21:26:40.236 INFO:tasks.workunit.client.1.smithi188.stdout:4/859: unlink d0/d2/d2a/d34/fe8 0 2024-04-05T21:26:40.240 INFO:tasks.workunit.client.1.smithi188.stdout:0/507: dwrite d4/d8/fa [0,4194304] 0 2024-04-05T21:26:40.245 INFO:tasks.workunit.client.1.smithi188.stdout:0/508: dread d4/d8/fa [0,4194304] 0 2024-04-05T21:26:40.246 INFO:tasks.workunit.client.1.smithi188.stdout:7/776: rename d3/d4/db to d3/d2c/dae/dfd 0 2024-04-05T21:26:40.247 INFO:tasks.workunit.client.1.smithi188.stdout:0/509: write d4/d8/db/d21/f56 [2036524,38669] 0 2024-04-05T21:26:40.248 INFO:tasks.workunit.client.1.smithi188.stdout:0/510: stat d4/d6/d13/d23/d24/d57/d63/f96 0 2024-04-05T21:26:40.249 INFO:tasks.workunit.client.1.smithi188.stdout:9/588: symlink dd/d11/d22/lc1 0 2024-04-05T21:26:40.252 INFO:tasks.workunit.client.1.smithi188.stdout:7/777: mknod d3/d2c/dae/dfd/d2e/d8c/cfe 0 2024-04-05T21:26:40.253 INFO:tasks.workunit.client.1.smithi188.stdout:7/778: rename d3/d2c/dae/dfd to d3/d2c/dae/dfd/d2e/d49/dff 22 2024-04-05T21:26:40.254 INFO:tasks.workunit.client.1.smithi188.stdout:0/511: creat d4/d8/db/d21/d4a/d73/d79/fb5 x:0 0 0 2024-04-05T21:26:40.260 INFO:tasks.workunit.client.0.smithi121.stdout:5/433: sync 2024-04-05T21:26:40.260 INFO:tasks.workunit.client.0.smithi121.stdout:7/363: sync 2024-04-05T21:26:40.260 INFO:tasks.workunit.client.0.smithi121.stdout:1/389: sync 2024-04-05T21:26:40.263 INFO:tasks.workunit.client.0.smithi121.stdout:5/434: creat dd/de/d19/d1c/d36/d46/f8a x:0 0 0 2024-04-05T21:26:40.266 INFO:tasks.workunit.client.0.smithi121.stdout:7/364: mkdir d0/d9/d22/d3e/d76 0 2024-04-05T21:26:40.268 INFO:tasks.workunit.client.0.smithi121.stdout:7/365: stat d0/d9/f50 0 2024-04-05T21:26:40.269 INFO:tasks.workunit.client.0.smithi121.stdout:7/366: fsync d0/d23/d5b/d52/f6e 0 2024-04-05T21:26:40.270 INFO:tasks.workunit.client.1.smithi188.stdout:7/779: dread d3/d2c/dae/dfd/d10/f95 [0,4194304] 0 2024-04-05T21:26:40.271 INFO:tasks.workunit.client.0.smithi121.stdout:7/367: read d0/d9/f20 [219900,23096] 0 2024-04-05T21:26:40.275 INFO:tasks.workunit.client.0.smithi121.stdout:1/390: dread d8/f1d [0,4194304] 0 2024-04-05T21:26:40.275 INFO:tasks.workunit.client.0.smithi121.stdout:7/368: creat d0/d23/d5b/d52/f77 x:0 0 0 2024-04-05T21:26:40.276 INFO:tasks.workunit.client.0.smithi121.stdout:7/369: mkdir d0/d23/d5b/d78 0 2024-04-05T21:26:40.277 INFO:tasks.workunit.client.0.smithi121.stdout:1/391: creat d8/d10/d14/d1a/d1b/d4c/f7e x:0 0 0 2024-04-05T21:26:40.278 INFO:tasks.workunit.client.0.smithi121.stdout:7/370: rmdir d0/d32/d4a 39 2024-04-05T21:26:40.278 INFO:tasks.workunit.client.0.smithi121.stdout:5/435: dread dd/de/d19/f23 [0,4194304] 0 2024-04-05T21:26:40.281 INFO:tasks.workunit.client.0.smithi121.stdout:5/436: mknod dd/de/d6d/c8b 0 2024-04-05T21:26:40.284 INFO:tasks.workunit.client.0.smithi121.stdout:5/437: chown dd/de/d19/d38/d56/f87 546040 1 2024-04-05T21:26:40.284 INFO:tasks.workunit.client.0.smithi121.stdout:1/392: dwrite d8/d10/d14/d27/f30 [0,4194304] 0 2024-04-05T21:26:40.285 INFO:tasks.workunit.client.0.smithi121.stdout:7/371: dread d0/d9/f21 [0,4194304] 0 2024-04-05T21:26:40.287 INFO:tasks.workunit.client.0.smithi121.stdout:7/372: write d0/d9/d22/d28/f48 [909340,11743] 0 2024-04-05T21:26:40.287 INFO:tasks.workunit.client.0.smithi121.stdout:1/393: creat d8/d10/d14/d27/d57/f7f x:0 0 0 2024-04-05T21:26:40.295 INFO:tasks.workunit.client.1.smithi188.stdout:9/589: link fb dd/d11/d1f/d25/d51/fc2 0 2024-04-05T21:26:40.296 INFO:tasks.workunit.client.1.smithi188.stdout:0/512: mknod d4/d8/db/d42/cb6 0 2024-04-05T21:26:40.297 INFO:tasks.workunit.client.0.smithi121.stdout:7/373: getdents d0/d9/d22/d28 0 2024-04-05T21:26:40.300 INFO:tasks.workunit.client.0.smithi121.stdout:7/374: chown d0/l45 943724 1 2024-04-05T21:26:40.301 INFO:tasks.workunit.client.0.smithi121.stdout:7/375: chown d0/d9/d22/f44 25 1 2024-04-05T21:26:40.301 INFO:tasks.workunit.client.1.smithi188.stdout:7/780: write d3/d2c/f96 [3681591,86083] 0 2024-04-05T21:26:40.302 INFO:tasks.workunit.client.0.smithi121.stdout:7/376: stat d0/d9/d22/d3e/d76 0 2024-04-05T21:26:40.302 INFO:tasks.workunit.client.1.smithi188.stdout:0/513: stat d4/d6/d13/d23/d24/l41 0 2024-04-05T21:26:40.304 INFO:tasks.workunit.client.1.smithi188.stdout:7/781: read d3/d4/d7c/fd9 [1907313,41819] 0 2024-04-05T21:26:40.304 INFO:tasks.workunit.client.1.smithi188.stdout:9/590: link dd/d11/d1f/f6e dd/d3d/d89/fc3 0 2024-04-05T21:26:40.305 INFO:tasks.workunit.client.1.smithi188.stdout:0/514: creat d4/d6/d46/fb7 x:0 0 0 2024-04-05T21:26:40.305 INFO:tasks.workunit.client.0.smithi121.stdout:5/438: dread dd/de/d19/d38/d45/f5c [0,4194304] 0 2024-04-05T21:26:40.306 INFO:tasks.workunit.client.1.smithi188.stdout:0/515: chown d4/d8/f9f 15342309 1 2024-04-05T21:26:40.307 INFO:tasks.workunit.client.0.smithi121.stdout:7/377: dread d0/d23/d5b/f43 [0,4194304] 0 2024-04-05T21:26:40.309 INFO:tasks.workunit.client.0.smithi121.stdout:7/378: creat d0/d9/d22/f79 x:0 0 0 2024-04-05T21:26:40.309 INFO:tasks.workunit.client.1.smithi188.stdout:9/591: rename dd/d11/d1f/d25/c58 to dd/d11/d1f/d26/dc0/d6b/cc4 0 2024-04-05T21:26:40.310 INFO:tasks.workunit.client.0.smithi121.stdout:7/379: readlink d0/l7 0 2024-04-05T21:26:40.311 INFO:tasks.workunit.client.1.smithi188.stdout:0/516: mknod d4/d8/db/cb8 0 2024-04-05T21:26:40.311 INFO:tasks.workunit.client.1.smithi188.stdout:7/782: rmdir d3/d2c 39 2024-04-05T21:26:40.313 INFO:tasks.workunit.client.1.smithi188.stdout:9/592: truncate dd/d11/d1f/d25/d8a/f77 1406543 0 2024-04-05T21:26:40.313 INFO:tasks.workunit.client.0.smithi121.stdout:5/439: dwrite dd/de/d19/d13/f16 [0,4194304] 0 2024-04-05T21:26:40.317 INFO:tasks.workunit.client.0.smithi121.stdout:1/394: dread f2 [0,4194304] 0 2024-04-05T21:26:40.318 INFO:tasks.workunit.client.0.smithi121.stdout:5/440: mkdir dd/de/d19/d1c/d74/d8c 0 2024-04-05T21:26:40.319 INFO:tasks.workunit.client.0.smithi121.stdout:7/380: dwrite d0/f42 [0,4194304] 0 2024-04-05T21:26:40.319 INFO:tasks.workunit.client.1.smithi188.stdout:9/593: creat dd/fc5 x:0 0 0 2024-04-05T21:26:40.321 INFO:tasks.workunit.client.1.smithi188.stdout:7/783: dwrite d3/d2c/dae/dfd/d10/d5f/fe1 [0,4194304] 0 2024-04-05T21:26:40.322 INFO:tasks.workunit.client.1.smithi188.stdout:7/784: write d3/d2c/d5e/fec [427433,34903] 0 2024-04-05T21:26:40.323 INFO:tasks.workunit.client.1.smithi188.stdout:0/517: write d4/d6/d13/d23/d24/d57/d63/f96 [537798,90678] 0 2024-04-05T21:26:40.324 INFO:tasks.workunit.client.0.smithi121.stdout:5/441: creat dd/de/d39/f8d x:0 0 0 2024-04-05T21:26:40.325 INFO:tasks.workunit.client.0.smithi121.stdout:7/381: mkdir d0/d23/d5b/d78/d7a 0 2024-04-05T21:26:40.327 INFO:tasks.workunit.client.0.smithi121.stdout:1/395: dwrite d8/d10/d14/d27/f75 [0,4194304] 0 2024-04-05T21:26:40.329 INFO:tasks.workunit.client.0.smithi121.stdout:5/442: mkdir dd/de/d19/d3c/d8e 0 2024-04-05T21:26:40.329 INFO:tasks.workunit.client.0.smithi121.stdout:1/396: chown d8/d10/d14/d1a/d1b/d44/f48 28 1 2024-04-05T21:26:40.330 INFO:tasks.workunit.client.1.smithi188.stdout:9/594: dread - dd/d11/d1f/d26/dc0/f84 zero size 2024-04-05T21:26:40.331 INFO:tasks.workunit.client.0.smithi121.stdout:7/382: unlink d0/d32/l34 0 2024-04-05T21:26:40.332 INFO:tasks.workunit.client.0.smithi121.stdout:7/383: chown d0/c4 3989230 1 2024-04-05T21:26:40.332 INFO:tasks.workunit.client.1.smithi188.stdout:7/785: rename d3/d2c/d5e/d91/l9b to d3/d2c/dae/dfd/d2e/d3b/d94/l100 0 2024-04-05T21:26:40.334 INFO:tasks.workunit.client.0.smithi121.stdout:7/384: readlink d0/d9/d22/d3e/l4d 0 2024-04-05T21:26:40.334 INFO:tasks.workunit.client.0.smithi121.stdout:5/443: creat dd/de/d19/d38/d45/d68/f8f x:0 0 0 2024-04-05T21:26:40.335 INFO:tasks.workunit.client.1.smithi188.stdout:0/518: write d4/d8/db/d21/d3c/fa9 [85381,48069] 0 2024-04-05T21:26:40.336 INFO:tasks.workunit.client.1.smithi188.stdout:7/786: write d3/d2c/dae/dfd/d10/d3c/ff5 [1007776,85500] 0 2024-04-05T21:26:40.337 INFO:tasks.workunit.client.0.smithi121.stdout:5/444: stat dd/de/d39/d3b/d50/l76 0 2024-04-05T21:26:40.339 INFO:tasks.workunit.client.0.smithi121.stdout:5/445: fdatasync dd/de/d19/d13/f35 0 2024-04-05T21:26:40.340 INFO:tasks.workunit.client.1.smithi188.stdout:7/787: fdatasync d3/d4/d1d/d74/fd1 0 2024-04-05T21:26:40.343 INFO:tasks.workunit.client.1.smithi188.stdout:9/595: dwrite dd/d11/d22/f56 [0,4194304] 0 2024-04-05T21:26:40.343 INFO:tasks.workunit.client.1.smithi188.stdout:7/788: write d3/d2c/dae/dfd/d10/d5f/fcc [522205,30500] 0 2024-04-05T21:26:40.357 INFO:tasks.workunit.client.1.smithi188.stdout:9/596: dread dd/d35/f67 [0,4194304] 0 2024-04-05T21:26:40.362 INFO:tasks.workunit.client.1.smithi188.stdout:0/519: dread d4/f53 [0,4194304] 0 2024-04-05T21:26:40.363 INFO:tasks.workunit.client.1.smithi188.stdout:9/597: rename dd/d11/d1f/d26/dc0/d6b/d40/d59/cae to dd/d11/cc6 0 2024-04-05T21:26:40.366 INFO:tasks.workunit.client.0.smithi121.stdout:5/446: dwrite dd/de/d19/d13/f3f [0,4194304] 0 2024-04-05T21:26:40.367 INFO:tasks.workunit.client.1.smithi188.stdout:9/598: truncate dd/d3d/f61 507296 0 2024-04-05T21:26:40.368 INFO:tasks.workunit.client.1.smithi188.stdout:7/789: dwrite d3/d2c/dce/f81 [0,4194304] 0 2024-04-05T21:26:40.369 INFO:tasks.workunit.client.0.smithi121.stdout:5/447: symlink dd/de/d6d/l90 0 2024-04-05T21:26:40.371 INFO:tasks.workunit.client.0.smithi121.stdout:7/385: dread d0/d32/f35 [0,4194304] 0 2024-04-05T21:26:40.371 INFO:tasks.workunit.client.0.smithi121.stdout:1/397: dread d8/d10/f26 [0,4194304] 0 2024-04-05T21:26:40.372 INFO:tasks.workunit.client.1.smithi188.stdout:0/520: rename d4/d6/d13/d23/d24/f62 to d4/d8/d78/d9d/fb9 0 2024-04-05T21:26:40.373 INFO:tasks.workunit.client.0.smithi121.stdout:5/448: creat dd/de/d19/d38/d45/d68/f91 x:0 0 0 2024-04-05T21:26:40.377 INFO:tasks.workunit.client.1.smithi188.stdout:9/599: unlink dd/d11/d73/f92 0 2024-04-05T21:26:40.377 INFO:tasks.workunit.client.1.smithi188.stdout:0/521: mkdir d4/d6/d46/dba 0 2024-04-05T21:26:40.378 INFO:tasks.workunit.client.0.smithi121.stdout:1/398: rename d8/d10/d14/d1a/d1b/c7b to d8/d10/d14/d1a/d1b/d4c/d76/c80 0 2024-04-05T21:26:40.378 INFO:tasks.workunit.client.0.smithi121.stdout:7/386: link d0/d23/f54 d0/d23/d5a/f7b 0 2024-04-05T21:26:40.381 INFO:tasks.workunit.client.1.smithi188.stdout:7/790: dwrite d3/d4/d1d/feb [0,4194304] 0 2024-04-05T21:26:40.382 INFO:tasks.workunit.client.0.smithi121.stdout:1/399: mknod d8/d10/c81 0 2024-04-05T21:26:40.383 INFO:tasks.workunit.client.0.smithi121.stdout:5/449: dwrite dd/de/d19/f51 [0,4194304] 0 2024-04-05T21:26:40.385 INFO:tasks.workunit.client.1.smithi188.stdout:7/791: chown d3/d2c/dae/dfd/d55/d9d/lb2 941578981 1 2024-04-05T21:26:40.386 INFO:tasks.workunit.client.0.smithi121.stdout:7/387: dread d0/d9/f26 [0,4194304] 0 2024-04-05T21:26:40.386 INFO:tasks.workunit.client.0.smithi121.stdout:7/388: chown d0/d32/f41 5277702 1 2024-04-05T21:26:40.387 INFO:tasks.workunit.client.0.smithi121.stdout:5/450: getdents dd/de/d19/d13 0 2024-04-05T21:26:40.388 INFO:tasks.workunit.client.0.smithi121.stdout:7/389: chown d0/d23/d5a/f63 1794449 1 2024-04-05T21:26:40.388 INFO:tasks.workunit.client.1.smithi188.stdout:0/522: rename d4/d6/l4f to d4/d6/d46/dba/lbb 0 2024-04-05T21:26:40.392 INFO:tasks.workunit.client.1.smithi188.stdout:6/646: sync 2024-04-05T21:26:40.393 INFO:tasks.workunit.client.1.smithi188.stdout:3/643: sync 2024-04-05T21:26:40.393 INFO:tasks.workunit.client.1.smithi188.stdout:0/523: symlink d4/d8/d78/d9d/lbc 0 2024-04-05T21:26:40.393 INFO:tasks.workunit.client.1.smithi188.stdout:3/644: chown d5/d7/f36 1585 1 2024-04-05T21:26:40.398 INFO:tasks.workunit.client.0.smithi121.stdout:5/451: dwrite dd/de/d19/d38/f4d [0,4194304] 0 2024-04-05T21:26:40.398 INFO:tasks.workunit.client.0.smithi121.stdout:7/390: dwrite d0/d9/d22/f6c [0,4194304] 0 2024-04-05T21:26:40.399 INFO:tasks.workunit.client.1.smithi188.stdout:3/645: mknod d5/d7/cdb 0 2024-04-05T21:26:40.399 INFO:tasks.workunit.client.0.smithi121.stdout:7/391: readlink d0/l1c 0 2024-04-05T21:26:40.400 INFO:tasks.workunit.client.1.smithi188.stdout:0/524: creat d4/d6/d13/d23/d24/d57/d63/fbd x:0 0 0 2024-04-05T21:26:40.401 INFO:tasks.workunit.client.0.smithi121.stdout:1/400: fdatasync d8/d10/d14/d27/d57/f7f 0 2024-04-05T21:26:40.401 INFO:tasks.workunit.client.0.smithi121.stdout:7/392: truncate d0/d9/d22/f79 815357 0 2024-04-05T21:26:40.402 INFO:tasks.workunit.client.1.smithi188.stdout:7/792: getdents d3/d2c/dae/dfd/d2e/d8c/db7 0 2024-04-05T21:26:40.403 INFO:tasks.workunit.client.0.smithi121.stdout:5/452: creat dd/de/d19/d38/d56/f92 x:0 0 0 2024-04-05T21:26:40.403 INFO:tasks.workunit.client.0.smithi121.stdout:7/393: chown d0/d32/d4a/f53 0 1 2024-04-05T21:26:40.404 INFO:tasks.workunit.client.0.smithi121.stdout:7/394: chown d0/f42 174 1 2024-04-05T21:26:40.405 INFO:tasks.workunit.client.0.smithi121.stdout:5/453: creat dd/de/d19/d1c/d36/f93 x:0 0 0 2024-04-05T21:26:40.405 INFO:tasks.workunit.client.1.smithi188.stdout:3/646: mknod d5/d7/d65/d61/d66/da4/cdc 0 2024-04-05T21:26:40.406 INFO:tasks.workunit.client.1.smithi188.stdout:9/600: dwrite dd/d34/f9a [4194304,4194304] 0 2024-04-05T21:26:40.407 INFO:tasks.workunit.client.0.smithi121.stdout:7/395: mkdir d0/d32/d7c 0 2024-04-05T21:26:40.408 INFO:tasks.workunit.client.0.smithi121.stdout:7/396: chown d0/d9/d22/f44 505458 1 2024-04-05T21:26:40.413 INFO:tasks.workunit.client.0.smithi121.stdout:5/454: dwrite dd/de/d19/d13/f35 [0,4194304] 0 2024-04-05T21:26:40.415 INFO:tasks.workunit.client.0.smithi121.stdout:7/397: dread d0/d9/d22/d28/f31 [0,4194304] 0 2024-04-05T21:26:40.416 INFO:tasks.workunit.client.1.smithi188.stdout:0/525: creat d4/d6/d13/d23/d24/fbe x:0 0 0 2024-04-05T21:26:40.417 INFO:tasks.workunit.client.0.smithi121.stdout:7/398: write d0/d9/d22/d28/d56/f58 [2290493,113513] 0 2024-04-05T21:26:40.422 INFO:tasks.workunit.client.0.smithi121.stdout:7/399: getdents d0/d57 0 2024-04-05T21:26:40.422 INFO:tasks.workunit.client.0.smithi121.stdout:7/400: readlink d0/l7 0 2024-04-05T21:26:40.424 INFO:tasks.workunit.client.0.smithi121.stdout:5/455: dwrite dd/de/d19/f80 [0,4194304] 0 2024-04-05T21:26:40.424 INFO:tasks.workunit.client.0.smithi121.stdout:5/456: chown dd/l6f 24817 1 2024-04-05T21:26:40.425 INFO:tasks.workunit.client.1.smithi188.stdout:9/601: link dd/d34/f9a dd/d11/d1f/d25/d51/fc7 0 2024-04-05T21:26:40.425 INFO:tasks.workunit.client.1.smithi188.stdout:0/526: mknod d4/d6/d13/d23/d24/d57/d63/cbf 0 2024-04-05T21:26:40.425 INFO:tasks.workunit.client.1.smithi188.stdout:6/647: dwrite d6/d13/d18/d19/d2c/d34/f97 [0,4194304] 0 2024-04-05T21:26:40.426 INFO:tasks.workunit.client.1.smithi188.stdout:9/602: mknod dd/d3d/d95/cc8 0 2024-04-05T21:26:40.426 INFO:tasks.workunit.client.0.smithi121.stdout:7/401: mknod d0/d32/c7d 0 2024-04-05T21:26:40.428 INFO:tasks.workunit.client.0.smithi121.stdout:5/457: creat dd/de/d39/d3b/f94 x:0 0 0 2024-04-05T21:26:40.428 INFO:tasks.workunit.client.1.smithi188.stdout:3/647: dread d5/d7/d31/f78 [0,4194304] 0 2024-04-05T21:26:40.429 INFO:tasks.workunit.client.0.smithi121.stdout:5/458: readlink dd/de/d19/d1c/l2f 0 2024-04-05T21:26:40.432 INFO:tasks.workunit.client.0.smithi121.stdout:5/459: rename dd/l77 to dd/de/d19/d13/d53/l95 0 2024-04-05T21:26:40.433 INFO:tasks.workunit.client.1.smithi188.stdout:6/648: creat d6/d13/d18/d19/d61/fc6 x:0 0 0 2024-04-05T21:26:40.434 INFO:tasks.workunit.client.1.smithi188.stdout:9/603: creat dd/d3d/d81/fc9 x:0 0 0 2024-04-05T21:26:40.435 INFO:tasks.workunit.client.1.smithi188.stdout:9/604: readlink dd/d34/l9d 0 2024-04-05T21:26:40.435 INFO:tasks.workunit.client.1.smithi188.stdout:6/649: read d6/d13/d18/d19/d2c/d3f/f56 [1777851,38911] 0 2024-04-05T21:26:40.437 INFO:tasks.workunit.client.1.smithi188.stdout:3/648: rename d5/d35/d4e/db6 to d5/ddd 0 2024-04-05T21:26:40.438 INFO:tasks.workunit.client.0.smithi121.stdout:5/460: rename dd/de/d19/d13/f2d to dd/f96 0 2024-04-05T21:26:40.438 INFO:tasks.workunit.client.1.smithi188.stdout:3/649: write d5/d39/f93 [1263603,115300] 0 2024-04-05T21:26:40.440 INFO:tasks.workunit.client.1.smithi188.stdout:9/605: mknod dd/cca 0 2024-04-05T21:26:40.441 INFO:tasks.workunit.client.1.smithi188.stdout:6/650: symlink d6/d13/d18/d19/lc7 0 2024-04-05T21:26:40.442 INFO:tasks.workunit.client.0.smithi121.stdout:5/461: getdents dd/de/d19/d38 0 2024-04-05T21:26:40.443 INFO:tasks.workunit.client.1.smithi188.stdout:9/606: write dd/d11/d1f/d26/dc0/d6b/f96 [387769,40167] 0 2024-04-05T21:26:40.443 INFO:tasks.workunit.client.1.smithi188.stdout:3/650: creat d5/d39/d84/fde x:0 0 0 2024-04-05T21:26:40.444 INFO:tasks.workunit.client.0.smithi121.stdout:5/462: creat dd/de/d19/d38/f97 x:0 0 0 2024-04-05T21:26:40.445 INFO:tasks.workunit.client.1.smithi188.stdout:3/651: write d5/d7/d31/fb2 [1051949,117337] 0 2024-04-05T21:26:40.447 INFO:tasks.workunit.client.1.smithi188.stdout:3/652: mknod d5/d7/d65/cdf 0 2024-04-05T21:26:40.448 INFO:tasks.workunit.client.1.smithi188.stdout:6/651: dread d6/d13/d18/d3c/f3d [0,4194304] 0 2024-04-05T21:26:40.449 INFO:tasks.workunit.client.1.smithi188.stdout:9/607: rmdir dd/d11/d73/d8c 0 2024-04-05T21:26:40.449 INFO:tasks.workunit.client.1.smithi188.stdout:6/652: stat d6/d13/d18/c1a 0 2024-04-05T21:26:40.450 INFO:tasks.workunit.client.1.smithi188.stdout:6/653: stat d6/d13/d18/d19/d2c/c4b 0 2024-04-05T21:26:40.451 INFO:tasks.workunit.client.1.smithi188.stdout:3/653: mkdir d5/d7/d65/d61/d66/da4/de0 0 2024-04-05T21:26:40.452 INFO:tasks.workunit.client.1.smithi188.stdout:6/654: write d6/d13/d18/d19/d61/fc6 [926401,109746] 0 2024-04-05T21:26:40.453 INFO:tasks.workunit.client.1.smithi188.stdout:9/608: creat dd/d11/d73/d85/fcb x:0 0 0 2024-04-05T21:26:40.455 INFO:tasks.workunit.client.1.smithi188.stdout:3/654: unlink d5/d39/d84/la7 0 2024-04-05T21:26:40.456 INFO:tasks.workunit.client.0.smithi121.stdout:5/463: creat dd/de/d19/d38/d45/d68/f98 x:0 0 0 2024-04-05T21:26:40.458 INFO:tasks.workunit.client.1.smithi188.stdout:6/655: fdatasync d6/d13/d18/d19/f7e 0 2024-04-05T21:26:40.460 INFO:tasks.workunit.client.1.smithi188.stdout:3/655: rename d5/d7/d65/da2/fd5 to d5/d7e/fe1 0 2024-04-05T21:26:40.460 INFO:tasks.workunit.client.1.smithi188.stdout:0/527: rmdir d4 39 2024-04-05T21:26:40.462 INFO:tasks.workunit.client.1.smithi188.stdout:7/793: truncate d3/d2c/dae/dfd/d2e/d37/fba 1172894 0 2024-04-05T21:26:40.464 INFO:tasks.workunit.client.1.smithi188.stdout:7/794: write d3/d2c/fd0 [945718,125705] 0 2024-04-05T21:26:40.469 INFO:tasks.workunit.client.1.smithi188.stdout:0/528: symlink d4/d8/db/d42/d71/d80/lc0 0 2024-04-05T21:26:40.470 INFO:tasks.workunit.client.1.smithi188.stdout:9/609: link c0 dd/d11/d73/d8d/ccc 0 2024-04-05T21:26:40.473 INFO:tasks.workunit.client.1.smithi188.stdout:6/656: rename d6/d13/d48/d49/d75 to d6/d13/d18/d19/dc8 0 2024-04-05T21:26:40.474 INFO:tasks.workunit.client.1.smithi188.stdout:7/795: dwrite d3/d2c/d5e/fec [0,4194304] 0 2024-04-05T21:26:40.475 INFO:tasks.workunit.client.1.smithi188.stdout:9/610: mkdir dd/d11/d1f/d26/d55/dcd 0 2024-04-05T21:26:40.475 INFO:tasks.workunit.client.1.smithi188.stdout:0/529: mkdir d4/d6/d13/dc1 0 2024-04-05T21:26:40.477 INFO:tasks.workunit.client.1.smithi188.stdout:6/657: chown d6/d13/d18/d19/d2c/d3f/fae 337 1 2024-04-05T21:26:40.477 INFO:tasks.workunit.client.0.smithi121.stdout:4/443: sync 2024-04-05T21:26:40.478 INFO:tasks.workunit.client.1.smithi188.stdout:7/796: mkdir d3/d4/d101 0 2024-04-05T21:26:40.479 INFO:tasks.workunit.client.1.smithi188.stdout:0/530: rmdir d4/d8/db/d21/d4a/d73 39 2024-04-05T21:26:40.480 INFO:tasks.workunit.client.0.smithi121.stdout:5/464: truncate dd/de/d19/d13/f5d 3375908 0 2024-04-05T21:26:40.481 INFO:tasks.workunit.client.1.smithi188.stdout:0/531: chown d4/d8/db/d21/l65 8602756 1 2024-04-05T21:26:40.481 INFO:tasks.workunit.client.1.smithi188.stdout:3/656: dread d5/d7/d65/d5f/fb4 [0,4194304] 0 2024-04-05T21:26:40.482 INFO:tasks.workunit.client.1.smithi188.stdout:7/797: symlink d3/da5/l102 0 2024-04-05T21:26:40.484 INFO:tasks.workunit.client.0.smithi121.stdout:4/444: dwrite f4 [0,4194304] 0 2024-04-05T21:26:40.486 INFO:tasks.workunit.client.1.smithi188.stdout:7/798: mknod d3/d2c/dae/dfd/d2e/d8c/db7/c103 0 2024-04-05T21:26:40.488 INFO:tasks.workunit.client.1.smithi188.stdout:7/799: truncate d3/fbb 115859 0 2024-04-05T21:26:40.489 INFO:tasks.workunit.client.1.smithi188.stdout:6/658: rename d6/d13/d18/d19/d2c/c5f to d6/d13/d18/d19/d2c/d3f/d77/cc9 0 2024-04-05T21:26:40.494 INFO:tasks.workunit.client.1.smithi188.stdout:3/657: dwrite d5/d7/d6d/dd8/f2a [0,4194304] 0 2024-04-05T21:26:40.496 INFO:tasks.workunit.client.0.smithi121.stdout:4/445: dread d5/d11/d2c/f36 [0,4194304] 0 2024-04-05T21:26:40.498 INFO:tasks.workunit.client.1.smithi188.stdout:6/659: rmdir d6/d13/d18/d19/d61 39 2024-04-05T21:26:40.500 INFO:tasks.workunit.client.1.smithi188.stdout:3/658: write d5/d7/d6d/dd8/f1f [9192242,13773] 0 2024-04-05T21:26:40.501 INFO:tasks.workunit.client.0.smithi121.stdout:4/446: link d5/d11/d26/d49/f72 d5/d11/d2c/d8c/f94 0 2024-04-05T21:26:40.501 INFO:tasks.workunit.client.1.smithi188.stdout:7/800: mkdir d3/d2c/dae/dfd/d10/d5f/dc6/d104 0 2024-04-05T21:26:40.502 INFO:tasks.workunit.client.0.smithi121.stdout:4/447: write d5/fa [179076,83392] 0 2024-04-05T21:26:40.503 INFO:tasks.workunit.client.0.smithi121.stdout:4/448: chown d5/d11/f14 1 1 2024-04-05T21:26:40.505 INFO:tasks.workunit.client.1.smithi188.stdout:6/660: read d6/d13/d18/d19/f27 [4643193,113751] 0 2024-04-05T21:26:40.506 INFO:tasks.workunit.client.1.smithi188.stdout:3/659: fdatasync d5/d35/d4e/f5b 0 2024-04-05T21:26:40.507 INFO:tasks.workunit.client.1.smithi188.stdout:9/611: dread f1 [0,4194304] 0 2024-04-05T21:26:40.507 INFO:tasks.workunit.client.1.smithi188.stdout:3/660: stat d5/df/c4b 0 2024-04-05T21:26:40.511 INFO:tasks.workunit.client.1.smithi188.stdout:6/661: unlink d6/d7/l9a 0 2024-04-05T21:26:40.513 INFO:tasks.workunit.client.1.smithi188.stdout:7/801: link d3/d2c/dae/dfd/d2e/d8c/db7/dd6/fde d3/d2c/d5e/d91/dfb/f105 0 2024-04-05T21:26:40.514 INFO:tasks.workunit.client.1.smithi188.stdout:3/661: creat d5/d7/d65/d5f/fe2 x:0 0 0 2024-04-05T21:26:40.516 INFO:tasks.workunit.client.1.smithi188.stdout:6/662: truncate d6/d13/d18/d19/d2c/d3f/d77/f90 644678 0 2024-04-05T21:26:40.516 INFO:tasks.workunit.client.1.smithi188.stdout:9/612: creat dd/d11/d1f/d25/fce x:0 0 0 2024-04-05T21:26:40.516 INFO:tasks.workunit.client.1.smithi188.stdout:3/662: readlink d5/d7/d65/d61/d66/d6e/l85 0 2024-04-05T21:26:40.517 INFO:tasks.workunit.client.1.smithi188.stdout:7/802: write d3/d4/f25 [535083,79849] 0 2024-04-05T21:26:40.518 INFO:tasks.workunit.client.1.smithi188.stdout:9/613: readlink dd/d35/lb6 0 2024-04-05T21:26:40.521 INFO:tasks.workunit.client.1.smithi188.stdout:7/803: mknod d3/d2c/d5e/d91/c106 0 2024-04-05T21:26:40.522 INFO:tasks.workunit.client.1.smithi188.stdout:3/663: mkdir d5/d7/d65/d61/d66/da4/de3 0 2024-04-05T21:26:40.523 INFO:tasks.workunit.client.1.smithi188.stdout:9/614: mkdir dd/d11/d1f/d26/d55/dcf 0 2024-04-05T21:26:40.524 INFO:tasks.workunit.client.1.smithi188.stdout:6/663: truncate d6/d13/d38/f4e 1030800 0 2024-04-05T21:26:40.526 INFO:tasks.workunit.client.1.smithi188.stdout:3/664: rmdir d5/d7/d65/da2 39 2024-04-05T21:26:40.534 INFO:tasks.workunit.client.1.smithi188.stdout:7/804: dread d3/d2c/dae/dfd/d2e/f51 [0,4194304] 0 2024-04-05T21:26:40.548 INFO:tasks.workunit.client.0.smithi121.stdout:4/449: symlink d5/d11/d60/l95 0 2024-04-05T21:26:40.549 INFO:tasks.workunit.client.0.smithi121.stdout:4/450: write d5/d11/d2c/d8d/d29/d2e/f7b [444942,113966] 0 2024-04-05T21:26:40.551 INFO:tasks.workunit.client.0.smithi121.stdout:4/451: creat d5/d11/d28/d87/f96 x:0 0 0 2024-04-05T21:26:40.552 INFO:tasks.workunit.client.0.smithi121.stdout:4/452: write d5/d11/d2c/d8d/d29/d58/f6d [90035,130904] 0 2024-04-05T21:26:40.555 INFO:tasks.workunit.client.0.smithi121.stdout:4/453: symlink d5/d11/d2c/d8d/d29/d58/l97 0 2024-04-05T21:26:40.556 INFO:tasks.workunit.client.0.smithi121.stdout:4/454: write d5/d11/f12 [3784128,102975] 0 2024-04-05T21:26:40.562 INFO:tasks.workunit.client.0.smithi121.stdout:4/455: dwrite d5/d11/d2c/d8d/d29/d58/f78 [0,4194304] 0 2024-04-05T21:26:40.574 INFO:tasks.workunit.client.1.smithi188.stdout:8/566: sync 2024-04-05T21:26:40.575 INFO:tasks.workunit.client.1.smithi188.stdout:8/567: chown d5/f24 607621509 1 2024-04-05T21:26:40.575 INFO:tasks.workunit.client.1.smithi188.stdout:8/568: chown d5/d21/d2a/f8d 5 1 2024-04-05T21:26:40.584 INFO:tasks.workunit.client.0.smithi121.stdout:4/456: dread d5/d11/d2c/d8c/f94 [4194304,4194304] 0 2024-04-05T21:26:40.585 INFO:tasks.workunit.client.1.smithi188.stdout:3/665: symlink d5/d7/d6d/le4 0 2024-04-05T21:26:40.587 INFO:tasks.workunit.client.1.smithi188.stdout:7/805: mknod d3/d2c/dae/dfd/d2e/d8c/c107 0 2024-04-05T21:26:40.588 INFO:tasks.workunit.client.0.smithi121.stdout:4/457: rename d5/d11/d26/l51 to d5/d11/d26/l98 0 2024-04-05T21:26:40.590 INFO:tasks.workunit.client.1.smithi188.stdout:3/666: mkdir d5/d2b/de5 0 2024-04-05T21:26:40.590 INFO:tasks.workunit.client.0.smithi121.stdout:4/458: symlink d5/d11/d2c/d8d/d29/d58/l99 0 2024-04-05T21:26:40.592 INFO:tasks.workunit.client.0.smithi121.stdout:4/459: mknod d5/d11/d2c/d8d/d29/d2e/c9a 0 2024-04-05T21:26:40.594 INFO:tasks.workunit.client.1.smithi188.stdout:3/667: mknod d5/d39/d84/ce6 0 2024-04-05T21:26:40.594 INFO:tasks.workunit.client.0.smithi121.stdout:4/460: unlink d5/d11/d20/c7c 0 2024-04-05T21:26:40.595 INFO:tasks.workunit.client.1.smithi188.stdout:8/569: dread d5/d21/d40/d43/d44/d5b/d99/f7d [0,4194304] 0 2024-04-05T21:26:40.596 INFO:tasks.workunit.client.0.smithi121.stdout:4/461: creat d5/d11/d2c/d8d/d29/f9b x:0 0 0 2024-04-05T21:26:40.598 INFO:tasks.workunit.client.0.smithi121.stdout:4/462: write d5/d11/d28/d87/f96 [9926,12405] 0 2024-04-05T21:26:40.598 INFO:tasks.workunit.client.1.smithi188.stdout:6/664: link d6/d13/d18/d19/d61/f8b d6/d13/d18/d19/da3/fca 0 2024-04-05T21:26:40.603 INFO:tasks.workunit.client.1.smithi188.stdout:8/570: creat d5/d21/d40/d43/dbb/fc1 x:0 0 0 2024-04-05T21:26:40.603 INFO:tasks.workunit.client.0.smithi121.stdout:4/463: write d5/f1d [3714153,83311] 0 2024-04-05T21:26:40.603 INFO:tasks.workunit.client.0.smithi121.stdout:4/464: mknod d5/d11/d28/d87/c9c 0 2024-04-05T21:26:40.603 INFO:tasks.workunit.client.0.smithi121.stdout:4/465: rename d5/d11/d2c/d8d/d64 to d5/d11/d2c/d9d 0 2024-04-05T21:26:40.604 INFO:tasks.workunit.client.0.smithi121.stdout:4/466: write d5/d11/d28/d87/f96 [43851,98647] 0 2024-04-05T21:26:40.606 INFO:tasks.workunit.client.0.smithi121.stdout:9/258: sync 2024-04-05T21:26:40.606 INFO:tasks.workunit.client.0.smithi121.stdout:2/335: sync 2024-04-05T21:26:40.606 INFO:tasks.workunit.client.0.smithi121.stdout:6/299: sync 2024-04-05T21:26:40.606 INFO:tasks.workunit.client.0.smithi121.stdout:0/424: sync 2024-04-05T21:26:40.607 INFO:tasks.workunit.client.0.smithi121.stdout:3/252: sync 2024-04-05T21:26:40.607 INFO:tasks.workunit.client.0.smithi121.stdout:8/313: sync 2024-04-05T21:26:40.607 INFO:tasks.workunit.client.1.smithi188.stdout:7/806: truncate d3/d2c/dae/dfd/d10/f95 295788 0 2024-04-05T21:26:40.610 INFO:tasks.workunit.client.0.smithi121.stdout:8/314: fdatasync d12/d13/d15/f19 0 2024-04-05T21:26:40.610 INFO:tasks.workunit.client.0.smithi121.stdout:9/259: write d6/df/d1d/f2d [2665307,19665] 0 2024-04-05T21:26:40.610 INFO:tasks.workunit.client.1.smithi188.stdout:8/571: truncate d5/d21/d40/d43/f51 4575892 0 2024-04-05T21:26:40.610 INFO:tasks.workunit.client.1.smithi188.stdout:6/665: dwrite d6/d13/d18/d19/dc8/fbf [0,4194304] 0 2024-04-05T21:26:40.611 INFO:tasks.workunit.client.0.smithi121.stdout:0/425: dread - d7/d71/d29/d2a/d2c/f87 zero size 2024-04-05T21:26:40.612 INFO:tasks.workunit.client.0.smithi121.stdout:9/260: chown d6/df/d48 97 1 2024-04-05T21:26:40.612 INFO:tasks.workunit.client.0.smithi121.stdout:2/336: mkdir d2/d9/d1c/d30/d59 0 2024-04-05T21:26:40.613 INFO:tasks.workunit.client.0.smithi121.stdout:4/467: dread d5/d11/d28/f13 [0,4194304] 0 2024-04-05T21:26:40.614 INFO:tasks.workunit.client.1.smithi188.stdout:9/615: dwrite dd/d11/d1f/d26/f48 [0,4194304] 0 2024-04-05T21:26:40.616 INFO:tasks.workunit.client.0.smithi121.stdout:8/315: symlink d12/d50/l6c 0 2024-04-05T21:26:40.616 INFO:tasks.workunit.client.1.smithi188.stdout:8/572: stat d5/d21/d40/d43/d44/d5b/d99/da3 0 2024-04-05T21:26:40.616 INFO:tasks.workunit.client.1.smithi188.stdout:9/616: chown dd/d11/d1f/d26/d55/l64 1747 1 2024-04-05T21:26:40.618 INFO:tasks.workunit.client.0.smithi121.stdout:9/261: write d6/df/d1d/d2c/d37/d3c/f3d [504634,34600] 0 2024-04-05T21:26:40.619 INFO:tasks.workunit.client.1.smithi188.stdout:9/617: write dd/f42 [2801826,7312] 0 2024-04-05T21:26:40.621 INFO:tasks.workunit.client.1.smithi188.stdout:3/668: dread d5/d7/d6d/dd8/f3d [0,4194304] 0 2024-04-05T21:26:40.623 INFO:tasks.workunit.client.1.smithi188.stdout:9/618: dread dd/d35/f67 [0,4194304] 0 2024-04-05T21:26:40.624 INFO:tasks.workunit.client.1.smithi188.stdout:6/666: rename d6/d13/d18/d19/d61/fa5 to d6/d13/d18/d19/fcb 0 2024-04-05T21:26:40.625 INFO:tasks.workunit.client.1.smithi188.stdout:7/807: link d3/d4/d1d/da3/c8b d3/d2c/dae/dfd/d10/d5f/c108 0 2024-04-05T21:26:40.625 INFO:tasks.workunit.client.1.smithi188.stdout:3/669: read d5/d35/d4e/f5b [93260,44556] 0 2024-04-05T21:26:40.627 INFO:tasks.workunit.client.1.smithi188.stdout:9/619: rmdir dd/d11/d73/d8d 39 2024-04-05T21:26:40.627 INFO:tasks.workunit.client.0.smithi121.stdout:3/253: dread d0/f7 [0,4194304] 0 2024-04-05T21:26:40.627 INFO:tasks.workunit.client.0.smithi121.stdout:0/426: rename d7/d62/c21 to d7/d71/d29/d2a/d39/c91 0 2024-04-05T21:26:40.627 INFO:tasks.workunit.client.0.smithi121.stdout:8/316: mkdir d12/d13/d56/d6d 0 2024-04-05T21:26:40.628 INFO:tasks.workunit.client.0.smithi121.stdout:9/262: mknod d6/df/d1d/d2c/d37/d3c/c52 0 2024-04-05T21:26:40.630 INFO:tasks.workunit.client.0.smithi121.stdout:4/468: write d5/f8 [941588,77729] 0 2024-04-05T21:26:40.631 INFO:tasks.workunit.client.0.smithi121.stdout:8/317: truncate d12/d50/d6a/f45 1219600 0 2024-04-05T21:26:40.632 INFO:tasks.workunit.client.1.smithi188.stdout:8/573: link d5/d21/d2a/d47/f54 d5/d21/d40/d43/d44/d5b/d99/d69/d70/dae/fc2 0 2024-04-05T21:26:40.632 INFO:tasks.workunit.client.0.smithi121.stdout:9/263: chown d6/df/d1d/d2c/f24 330873759 1 2024-04-05T21:26:40.632 INFO:tasks.workunit.client.0.smithi121.stdout:3/254: mkdir d0/d3/d24/d3d/d1a/d23/d38/d55 0 2024-04-05T21:26:40.633 INFO:tasks.workunit.client.0.smithi121.stdout:4/469: readlink d5/d11/d20/l3d 0 2024-04-05T21:26:40.633 INFO:tasks.workunit.client.1.smithi188.stdout:6/667: dwrite d6/d7/fc [4194304,4194304] 0 2024-04-05T21:26:40.634 INFO:tasks.workunit.client.0.smithi121.stdout:2/337: creat d2/d9/d38/f5a x:0 0 0 2024-04-05T21:26:40.635 INFO:tasks.workunit.client.0.smithi121.stdout:8/318: creat d12/d50/d6a/f6e x:0 0 0 2024-04-05T21:26:40.635 INFO:tasks.workunit.client.1.smithi188.stdout:3/670: rename d5/d7/d65/d61/f92 to d5/d2b/fe7 0 2024-04-05T21:26:40.636 INFO:tasks.workunit.client.1.smithi188.stdout:3/671: readlink d5/df/ld7 0 2024-04-05T21:26:40.637 INFO:tasks.workunit.client.0.smithi121.stdout:9/264: mkdir d6/df/d1d/d2c/d53 0 2024-04-05T21:26:40.637 INFO:tasks.workunit.client.1.smithi188.stdout:9/620: symlink dd/d11/d1f/db2/ld0 0 2024-04-05T21:26:40.641 INFO:tasks.workunit.client.0.smithi121.stdout:0/427: dwrite d7/d71/d29/d3f/f4f [0,4194304] 0 2024-04-05T21:26:40.643 INFO:tasks.workunit.client.0.smithi121.stdout:8/319: mkdir d12/d13/d56/d6d/d6f 0 2024-04-05T21:26:40.646 INFO:tasks.workunit.client.0.smithi121.stdout:9/265: rename d6/l51 to d6/df/d1d/d2c/d53/l54 0 2024-04-05T21:26:40.647 INFO:tasks.workunit.client.0.smithi121.stdout:6/300: dread d1/f10 [0,4194304] 0 2024-04-05T21:26:40.648 INFO:tasks.workunit.client.1.smithi188.stdout:8/574: symlink d5/d21/lc3 0 2024-04-05T21:26:40.648 INFO:tasks.workunit.client.1.smithi188.stdout:6/668: read d6/d13/d18/d19/f30 [164023,123726] 0 2024-04-05T21:26:40.648 INFO:tasks.workunit.client.1.smithi188.stdout:3/672: symlink d5/ddd/le8 0 2024-04-05T21:26:40.650 INFO:tasks.workunit.client.0.smithi121.stdout:8/320: write d12/d13/d15/f4b [235535,34383] 0 2024-04-05T21:26:40.650 INFO:tasks.workunit.client.0.smithi121.stdout:2/338: dread d2/f1a [0,4194304] 0 2024-04-05T21:26:40.651 INFO:tasks.workunit.client.0.smithi121.stdout:1/401: sync 2024-04-05T21:26:40.652 INFO:tasks.workunit.client.0.smithi121.stdout:3/255: link d0/f15 d0/d3/d24/d3d/d1a/d23/f56 0 2024-04-05T21:26:40.652 INFO:tasks.workunit.client.0.smithi121.stdout:8/321: stat d12/d50/d61/l4a 0 2024-04-05T21:26:40.653 INFO:tasks.workunit.client.0.smithi121.stdout:4/470: dwrite d5/d11/d2c/d8d/f30 [0,4194304] 0 2024-04-05T21:26:40.654 INFO:tasks.workunit.client.0.smithi121.stdout:0/428: chown d7/d1c/f25 15 1 2024-04-05T21:26:40.660 INFO:tasks.workunit.client.0.smithi121.stdout:9/266: creat d6/d46/f55 x:0 0 0 2024-04-05T21:26:40.660 INFO:tasks.workunit.client.1.smithi188.stdout:9/621: dwrite dd/d11/d1f/f6e [0,4194304] 0 2024-04-05T21:26:40.664 INFO:tasks.workunit.client.0.smithi121.stdout:3/256: creat d0/d3/d24/d3d/f57 x:0 0 0 2024-04-05T21:26:40.664 INFO:tasks.workunit.client.0.smithi121.stdout:1/402: truncate d8/d10/d14/d1a/d1b/d38/f42 1143611 0 2024-04-05T21:26:40.671 INFO:tasks.workunit.client.0.smithi121.stdout:2/339: dread d2/f11 [4194304,4194304] 0 2024-04-05T21:26:40.671 INFO:tasks.workunit.client.1.smithi188.stdout:1/624: sync 2024-04-05T21:26:40.672 INFO:tasks.workunit.client.0.smithi121.stdout:9/267: creat d6/df/d1d/d2c/d37/d3c/f56 x:0 0 0 2024-04-05T21:26:40.675 INFO:tasks.workunit.client.0.smithi121.stdout:0/429: dwrite d7/d62/f20 [0,4194304] 0 2024-04-05T21:26:40.676 INFO:tasks.workunit.client.0.smithi121.stdout:9/268: chown d6/df/f27 761 1 2024-04-05T21:26:40.676 INFO:tasks.workunit.client.0.smithi121.stdout:4/471: dwrite d5/d11/d28/d3b/f77 [0,4194304] 0 2024-04-05T21:26:40.676 INFO:tasks.workunit.client.0.smithi121.stdout:8/322: dread d12/d13/d15/f19 [0,4194304] 0 2024-04-05T21:26:40.678 INFO:tasks.workunit.client.1.smithi188.stdout:8/575: mknod d5/d21/d40/cc4 0 2024-04-05T21:26:40.678 INFO:tasks.workunit.client.0.smithi121.stdout:2/340: fdatasync d2/d9/d17/d3a/f55 0 2024-04-05T21:26:40.680 INFO:tasks.workunit.client.1.smithi188.stdout:7/808: dread d3/d2c/dae/dfd/f60 [0,4194304] 0 2024-04-05T21:26:40.681 INFO:tasks.workunit.client.1.smithi188.stdout:3/673: creat d5/d7/d6d/dd8/d2f/fe9 x:0 0 0 2024-04-05T21:26:40.684 INFO:tasks.workunit.client.0.smithi121.stdout:3/257: mknod d0/d3/d24/d3d/d1a/d23/d38/d55/c58 0 2024-04-05T21:26:40.684 INFO:tasks.workunit.client.0.smithi121.stdout:0/430: mknod d7/c92 0 2024-04-05T21:26:40.685 INFO:tasks.workunit.client.0.smithi121.stdout:4/472: dread d5/f16 [0,4194304] 0 2024-04-05T21:26:40.686 INFO:tasks.workunit.client.0.smithi121.stdout:9/269: chown d6/d32/c45 1132091 1 2024-04-05T21:26:40.687 INFO:tasks.workunit.client.1.smithi188.stdout:1/625: creat d1/d7/dbf/fc7 x:0 0 0 2024-04-05T21:26:40.687 INFO:tasks.workunit.client.1.smithi188.stdout:6/669: creat d6/d13/db9/fcc x:0 0 0 2024-04-05T21:26:40.688 INFO:tasks.workunit.client.0.smithi121.stdout:4/473: chown d5/d11/d2c/d8d/d29/d58/l99 396014437 1 2024-04-05T21:26:40.688 INFO:tasks.workunit.client.1.smithi188.stdout:9/622: mkdir dd/d3d/d89/dd1 0 2024-04-05T21:26:40.690 INFO:tasks.workunit.client.0.smithi121.stdout:4/474: write d5/d11/d2c/d8d/d29/f9b [589744,129195] 0 2024-04-05T21:26:40.690 INFO:tasks.workunit.client.1.smithi188.stdout:7/809: read d3/d2c/dae/dfd/d10/fc9 [280125,98153] 0 2024-04-05T21:26:40.691 INFO:tasks.workunit.client.0.smithi121.stdout:1/403: link d8/d10/d14/d27/d57/d77/l7d d8/d10/d14/d1a/l82 0 2024-04-05T21:26:40.691 INFO:tasks.workunit.client.1.smithi188.stdout:1/626: mkdir d1/dc/d16/d21/dc8 0 2024-04-05T21:26:40.693 INFO:tasks.workunit.client.1.smithi188.stdout:6/670: creat d6/d13/d38/fcd x:0 0 0 2024-04-05T21:26:40.694 INFO:tasks.workunit.client.0.smithi121.stdout:0/431: getdents d7/d62/d8b 0 2024-04-05T21:26:40.695 INFO:tasks.workunit.client.0.smithi121.stdout:4/475: fsync d5/d11/d20/f82 0 2024-04-05T21:26:40.698 INFO:tasks.workunit.client.1.smithi188.stdout:9/623: creat dd/d11/fd2 x:0 0 0 2024-04-05T21:26:40.698 INFO:tasks.workunit.client.0.smithi121.stdout:4/476: symlink d5/d11/d2c/d8d/d29/d58/l9e 0 2024-04-05T21:26:40.699 INFO:tasks.workunit.client.1.smithi188.stdout:3/674: dwrite d5/f7d [0,4194304] 0 2024-04-05T21:26:40.699 INFO:tasks.workunit.client.1.smithi188.stdout:7/810: symlink d3/d2c/dae/dfd/d2e/d8c/db7/dd6/l109 0 2024-04-05T21:26:40.701 INFO:tasks.workunit.client.0.smithi121.stdout:1/404: dread d8/d10/d14/d1a/d1b/d38/f42 [0,4194304] 0 2024-04-05T21:26:40.701 INFO:tasks.workunit.client.0.smithi121.stdout:0/432: creat d7/d71/d29/f93 x:0 0 0 2024-04-05T21:26:40.703 INFO:tasks.workunit.client.1.smithi188.stdout:8/576: dwrite d5/d21/d2a/f39 [0,4194304] 0 2024-04-05T21:26:40.704 INFO:tasks.workunit.client.0.smithi121.stdout:2/341: truncate d2/f11 136386 0 2024-04-05T21:26:40.704 INFO:tasks.workunit.client.1.smithi188.stdout:6/671: read d6/d13/f89 [1665128,36894] 0 2024-04-05T21:26:40.705 INFO:tasks.workunit.client.0.smithi121.stdout:1/405: symlink d8/d10/d14/l83 0 2024-04-05T21:26:40.706 INFO:tasks.workunit.client.0.smithi121.stdout:3/258: write d0/d3/d24/d3d/f21 [2153304,110288] 0 2024-04-05T21:26:40.707 INFO:tasks.workunit.client.1.smithi188.stdout:1/627: rename d1/db2/fbe to d1/dc/d16/d21/dbb/fc9 0 2024-04-05T21:26:40.707 INFO:tasks.workunit.client.0.smithi121.stdout:9/270: write d6/fc [6783853,48862] 0 2024-04-05T21:26:40.708 INFO:tasks.workunit.client.1.smithi188.stdout:4/860: sync 2024-04-05T21:26:40.708 INFO:tasks.workunit.client.1.smithi188.stdout:0/532: sync 2024-04-05T21:26:40.709 INFO:tasks.workunit.client.0.smithi121.stdout:0/433: mkdir d7/d71/d94 0 2024-04-05T21:26:40.710 INFO:tasks.workunit.client.1.smithi188.stdout:5/571: sync 2024-04-05T21:26:40.710 INFO:tasks.workunit.client.1.smithi188.stdout:2/677: sync 2024-04-05T21:26:40.711 INFO:tasks.workunit.client.0.smithi121.stdout:2/342: mknod d2/c5b 0 2024-04-05T21:26:40.711 INFO:tasks.workunit.client.1.smithi188.stdout:6/672: symlink d6/d13/d18/d19/d61/lce 0 2024-04-05T21:26:40.711 INFO:tasks.workunit.client.1.smithi188.stdout:9/624: creat dd/d11/d73/fd3 x:0 0 0 2024-04-05T21:26:40.714 INFO:tasks.workunit.client.1.smithi188.stdout:1/628: creat d1/d7/d4e/fca x:0 0 0 2024-04-05T21:26:40.714 INFO:tasks.workunit.client.0.smithi121.stdout:1/406: chown d8/d10/d14/d27/f75 115637023 1 2024-04-05T21:26:40.715 INFO:tasks.workunit.client.0.smithi121.stdout:4/477: dwrite d5/d11/d2c/d8d/d29/f9b [0,4194304] 0 2024-04-05T21:26:40.715 INFO:tasks.workunit.client.1.smithi188.stdout:4/861: rename d0/d2/d17/d48/l11b to d0/d14/d2f/ddd/l128 0 2024-04-05T21:26:40.719 INFO:tasks.workunit.client.1.smithi188.stdout:8/577: truncate d5/d21/d40/d43/d44/d5b/d99/da3/fa5 740638 0 2024-04-05T21:26:40.719 INFO:tasks.workunit.client.0.smithi121.stdout:1/407: rmdir d8/d10/d14/d1a 39 2024-04-05T21:26:40.719 INFO:tasks.workunit.client.1.smithi188.stdout:7/811: link d3/d2c/dae/dfd/d10/d5f/fe1 d3/d2c/f10a 0 2024-04-05T21:26:40.721 INFO:tasks.workunit.client.0.smithi121.stdout:8/323: dread d12/d13/d15/f21 [0,4194304] 0 2024-04-05T21:26:40.721 INFO:tasks.workunit.client.0.smithi121.stdout:2/343: truncate d2/d1e/f52 240589 0 2024-04-05T21:26:40.722 INFO:tasks.workunit.client.1.smithi188.stdout:2/678: mknod d5/db/dd/da4/cde 0 2024-04-05T21:26:40.723 INFO:tasks.workunit.client.0.smithi121.stdout:2/344: readlink d2/d1e/l57 0 2024-04-05T21:26:40.723 INFO:tasks.workunit.client.0.smithi121.stdout:4/478: creat d5/d11/d60/f9f x:0 0 0 2024-04-05T21:26:40.725 INFO:tasks.workunit.client.0.smithi121.stdout:1/408: mkdir d8/d10/d14/d27/d2d/d84 0 2024-04-05T21:26:40.728 INFO:tasks.workunit.client.0.smithi121.stdout:8/324: symlink d12/d13/l70 0 2024-04-05T21:26:40.728 INFO:tasks.workunit.client.0.smithi121.stdout:2/345: write d2/d9/f54 [609749,10939] 0 2024-04-05T21:26:40.728 INFO:tasks.workunit.client.0.smithi121.stdout:0/434: dwrite d7/d71/d29/f93 [0,4194304] 0 2024-04-05T21:26:40.729 INFO:tasks.workunit.client.1.smithi188.stdout:2/679: read - d5/db/dd/d67/fad zero size 2024-04-05T21:26:40.729 INFO:tasks.workunit.client.1.smithi188.stdout:9/625: rename dd/l5d to dd/d35/ld4 0 2024-04-05T21:26:40.731 INFO:tasks.workunit.client.0.smithi121.stdout:0/435: write d7/d71/d29/d3f/f4f [427870,53056] 0 2024-04-05T21:26:40.734 INFO:tasks.workunit.client.0.smithi121.stdout:2/346: dread - d2/d9/d38/f5a zero size 2024-04-05T21:26:40.735 INFO:tasks.workunit.client.1.smithi188.stdout:3/675: dwrite d5/d7/d65/d5f/fb4 [0,4194304] 0 2024-04-05T21:26:40.735 INFO:tasks.workunit.client.1.smithi188.stdout:5/572: creat d6/d17/d23/d7d/dc4/fc9 x:0 0 0 2024-04-05T21:26:40.735 INFO:tasks.workunit.client.1.smithi188.stdout:8/578: creat d5/d21/d2a/d47/fc5 x:0 0 0 2024-04-05T21:26:40.740 INFO:tasks.workunit.client.1.smithi188.stdout:2/680: chown d5/db/dd/d36/d3c/d48/f5b 68483 1 2024-04-05T21:26:40.741 INFO:tasks.workunit.client.0.smithi121.stdout:3/259: truncate d0/f41 879219 0 2024-04-05T21:26:40.741 INFO:tasks.workunit.client.1.smithi188.stdout:5/573: fsync d6/d82/d40/d42/f5e 0 2024-04-05T21:26:40.742 INFO:tasks.workunit.client.1.smithi188.stdout:7/812: dread d3/d2c/f10a [0,4194304] 0 2024-04-05T21:26:40.743 INFO:tasks.workunit.client.1.smithi188.stdout:9/626: dread dd/d3d/fbe [0,4194304] 0 2024-04-05T21:26:40.743 INFO:tasks.workunit.client.1.smithi188.stdout:4/862: dwrite d0/d2/dc7/d104/f105 [0,4194304] 0 2024-04-05T21:26:40.745 INFO:tasks.workunit.client.1.smithi188.stdout:2/681: write d5/db/f20 [3551450,105132] 0 2024-04-05T21:26:40.748 INFO:tasks.workunit.client.0.smithi121.stdout:1/409: dwrite d8/d10/d14/d1a/d1b/d4c/d76/f3d [0,4194304] 0 2024-04-05T21:26:40.748 INFO:tasks.workunit.client.0.smithi121.stdout:2/347: rename d2/d9/d1c/d4b to d2/d9/d56/d5c 0 2024-04-05T21:26:40.748 INFO:tasks.workunit.client.0.smithi121.stdout:7/402: sync 2024-04-05T21:26:40.748 INFO:tasks.workunit.client.1.smithi188.stdout:3/676: unlink d5/d7/d65/d5f/lcb 0 2024-04-05T21:26:40.750 INFO:tasks.workunit.client.0.smithi121.stdout:3/260: rmdir d0/d3/d24/d3d 39 2024-04-05T21:26:40.753 INFO:tasks.workunit.client.0.smithi121.stdout:5/465: sync 2024-04-05T21:26:40.754 INFO:tasks.workunit.client.0.smithi121.stdout:9/271: dwrite d6/f17 [0,4194304] 0 2024-04-05T21:26:40.755 INFO:tasks.workunit.client.1.smithi188.stdout:1/629: getdents d1/dc/d16 0 2024-04-05T21:26:40.757 INFO:tasks.workunit.client.1.smithi188.stdout:3/677: chown d5/d7/d6d/dd8/d2f/f89 168677 1 2024-04-05T21:26:40.757 INFO:tasks.workunit.client.1.smithi188.stdout:5/574: fdatasync d6/d82/d40/f65 0 2024-04-05T21:26:40.758 INFO:tasks.workunit.client.1.smithi188.stdout:8/579: dread d5/d21/f3a [0,4194304] 0 2024-04-05T21:26:40.758 INFO:tasks.workunit.client.1.smithi188.stdout:9/627: dread dd/d11/d1f/d26/dc0/f52 [0,4194304] 0 2024-04-05T21:26:40.759 INFO:tasks.workunit.client.1.smithi188.stdout:7/813: read - d3/d2c/dae/dfd/d2e/d8c/db7/dd6/fde zero size 2024-04-05T21:26:40.760 INFO:tasks.workunit.client.1.smithi188.stdout:1/630: chown d1/l65 41426891 1 2024-04-05T21:26:40.761 INFO:tasks.workunit.client.0.smithi121.stdout:8/325: write d12/d13/d15/f21 [439595,48168] 0 2024-04-05T21:26:40.761 INFO:tasks.workunit.client.1.smithi188.stdout:3/678: chown d5/d35/d4e/fc9 1900225523 1 2024-04-05T21:26:40.762 INFO:tasks.workunit.client.1.smithi188.stdout:2/682: symlink d5/db/dd/d36/d3c/d48/d78/ldf 0 2024-04-05T21:26:40.763 INFO:tasks.workunit.client.1.smithi188.stdout:6/673: write d6/d13/d18/d19/dc8/f9f [674624,111437] 0 2024-04-05T21:26:40.764 INFO:tasks.workunit.client.1.smithi188.stdout:3/679: fsync d5/d2b/fc5 0 2024-04-05T21:26:40.767 INFO:tasks.workunit.client.0.smithi121.stdout:5/466: creat dd/de/d19/d1c/d7b/f99 x:0 0 0 2024-04-05T21:26:40.767 INFO:tasks.workunit.client.1.smithi188.stdout:2/683: write d5/db/dd/d67/dcd/d46/d9f/fb0 [179925,68885] 0 2024-04-05T21:26:40.767 INFO:tasks.workunit.client.0.smithi121.stdout:1/410: dwrite d8/d10/d14/d27/f75 [0,4194304] 0 2024-04-05T21:26:40.768 INFO:tasks.workunit.client.1.smithi188.stdout:6/674: chown d6/d13/d18/d19/d2c/d34/f6b 1245153444 1 2024-04-05T21:26:40.772 INFO:tasks.workunit.client.0.smithi121.stdout:5/467: fdatasync dd/de/d39/d3b/d50/f5b 0 2024-04-05T21:26:40.773 INFO:tasks.workunit.client.0.smithi121.stdout:1/411: rmdir d8/d10/d14 39 2024-04-05T21:26:40.773 INFO:tasks.workunit.client.1.smithi188.stdout:7/814: creat d3/d2c/dae/dfd/d55/f10b x:0 0 0 2024-04-05T21:26:40.773 INFO:tasks.workunit.client.0.smithi121.stdout:3/261: dwrite d0/d3/d24/d3d/f57 [0,4194304] 0 2024-04-05T21:26:40.776 INFO:tasks.workunit.client.1.smithi188.stdout:1/631: unlink d1/d7/d4e/c5b 0 2024-04-05T21:26:40.776 INFO:tasks.workunit.client.1.smithi188.stdout:3/680: stat d5/d7/d6d/dd8/cd1 0 2024-04-05T21:26:40.778 INFO:tasks.workunit.client.0.smithi121.stdout:4/479: dread d5/d11/d28/f1b [0,4194304] 0 2024-04-05T21:26:40.778 INFO:tasks.workunit.client.0.smithi121.stdout:8/326: dwrite d12/d13/d15/d1d/f58 [0,4194304] 0 2024-04-05T21:26:40.778 INFO:tasks.workunit.client.0.smithi121.stdout:0/436: truncate d7/d62/f42 787491 0 2024-04-05T21:26:40.782 INFO:tasks.workunit.client.0.smithi121.stdout:5/468: creat dd/de/d19/d1c/d36/d46/f9a x:0 0 0 2024-04-05T21:26:40.784 INFO:tasks.workunit.client.1.smithi188.stdout:9/628: mkdir dd/d3d/dd5 0 2024-04-05T21:26:40.788 INFO:tasks.workunit.client.0.smithi121.stdout:5/469: dread - dd/de/d19/d1c/d74/f81 zero size 2024-04-05T21:26:40.789 INFO:tasks.workunit.client.1.smithi188.stdout:8/580: dwrite d5/d21/d40/d43/d44/d5b/d99/f7d [0,4194304] 0 2024-04-05T21:26:40.789 INFO:tasks.workunit.client.0.smithi121.stdout:3/262: rename d0/d3/l44 to d0/d3/d24/d3d/d1a/l59 0 2024-04-05T21:26:40.792 INFO:tasks.workunit.client.1.smithi188.stdout:3/681: creat d5/d7/d65/d61/d66/da4/de3/fea x:0 0 0 2024-04-05T21:26:40.792 INFO:tasks.workunit.client.0.smithi121.stdout:0/437: mkdir d7/dc/d95 0 2024-04-05T21:26:40.794 INFO:tasks.workunit.client.0.smithi121.stdout:4/480: dwrite d5/d11/d28/d87/f96 [0,4194304] 0 2024-04-05T21:26:40.794 INFO:tasks.workunit.client.1.smithi188.stdout:3/682: fsync d5/d35/d4e/fc9 0 2024-04-05T21:26:40.795 INFO:tasks.workunit.client.0.smithi121.stdout:5/470: chown dd/de/d19/d38/d45/l66 2672 1 2024-04-05T21:26:40.796 INFO:tasks.workunit.client.0.smithi121.stdout:3/263: rmdir d0/d3/d24/d3d/d1a/d1e 39 2024-04-05T21:26:40.796 INFO:tasks.workunit.client.1.smithi188.stdout:9/629: unlink dd/d11/d1f/d26/dc0/f52 0 2024-04-05T21:26:40.797 INFO:tasks.workunit.client.0.smithi121.stdout:4/481: write d5/d11/f12 [6168319,77540] 0 2024-04-05T21:26:40.798 INFO:tasks.workunit.client.0.smithi121.stdout:3/264: truncate d0/d3/f54 540150 0 2024-04-05T21:26:40.800 INFO:tasks.workunit.client.0.smithi121.stdout:0/438: creat d7/dc/d95/f96 x:0 0 0 2024-04-05T21:26:40.802 INFO:tasks.workunit.client.0.smithi121.stdout:5/471: mkdir dd/de/d19/d38/d9b 0 2024-04-05T21:26:40.803 INFO:tasks.workunit.client.1.smithi188.stdout:2/684: link d5/db/dd/c13 d5/db/d35/ce0 0 2024-04-05T21:26:40.803 INFO:tasks.workunit.client.1.smithi188.stdout:1/632: creat d1/dc/d16/d46/d88/d13/fcb x:0 0 0 2024-04-05T21:26:40.804 INFO:tasks.workunit.client.0.smithi121.stdout:0/439: mkdir d7/d7e/d97 0 2024-04-05T21:26:40.805 INFO:tasks.workunit.client.1.smithi188.stdout:8/581: dread d5/f35 [0,4194304] 0 2024-04-05T21:26:40.806 INFO:tasks.workunit.client.0.smithi121.stdout:0/440: dread - d7/d71/d29/d2a/d2c/f8d zero size 2024-04-05T21:26:40.806 INFO:tasks.workunit.client.0.smithi121.stdout:5/472: mkdir dd/de/d19/d1c/d7b/d88/d9c 0 2024-04-05T21:26:40.806 INFO:tasks.workunit.client.1.smithi188.stdout:9/630: rename dd/d11/d1f/d26/dc0/l4b to dd/d34/da7/ld6 0 2024-04-05T21:26:40.806 INFO:tasks.workunit.client.1.smithi188.stdout:2/685: mkdir d5/db/dd/d24/de1 0 2024-04-05T21:26:40.809 INFO:tasks.workunit.client.1.smithi188.stdout:1/633: symlink d1/dc/d16/d21/d27/d4a/lcc 0 2024-04-05T21:26:40.815 INFO:tasks.workunit.client.0.smithi121.stdout:8/327: dwrite d12/d13/d15/d1d/f58 [4194304,4194304] 0 2024-04-05T21:26:40.815 INFO:tasks.workunit.client.1.smithi188.stdout:4/863: dwrite d0/d81/f85 [0,4194304] 0 2024-04-05T21:26:40.816 INFO:tasks.workunit.client.1.smithi188.stdout:2/686: mknod d5/db/dd/d67/dcd/d46/d8c/ce2 0 2024-04-05T21:26:40.817 INFO:tasks.workunit.client.0.smithi121.stdout:5/473: chown dd/de/d19/d38/d45/d68/f8f 0 1 2024-04-05T21:26:40.818 INFO:tasks.workunit.client.1.smithi188.stdout:8/582: write d5/d21/d40/d43/d44/d5b/d99/d69/fb3 [841032,124722] 0 2024-04-05T21:26:40.821 INFO:tasks.workunit.client.0.smithi121.stdout:8/328: write d12/d50/d6a/f65 [682762,30842] 0 2024-04-05T21:26:40.825 INFO:tasks.workunit.client.0.smithi121.stdout:0/441: dwrite d7/d71/f26 [0,4194304] 0 2024-04-05T21:26:40.826 INFO:tasks.workunit.client.1.smithi188.stdout:5/575: dwrite d6/d82/f3b [0,4194304] 0 2024-04-05T21:26:40.826 INFO:tasks.workunit.client.1.smithi188.stdout:4/864: chown d0/d14/d15/d78/fae 1583600 1 2024-04-05T21:26:40.828 INFO:tasks.workunit.client.0.smithi121.stdout:2/348: dwrite d2/f11 [0,4194304] 0 2024-04-05T21:26:40.832 INFO:tasks.workunit.client.0.smithi121.stdout:7/403: dread d0/d9/f1d [0,4194304] 0 2024-04-05T21:26:40.833 INFO:tasks.workunit.client.1.smithi188.stdout:3/683: link d5/d2b/cba d5/d7/d6d/d75/d83/ceb 0 2024-04-05T21:26:40.834 INFO:tasks.workunit.client.0.smithi121.stdout:0/442: unlink d7/d62/d3c/d6b/l7f 0 2024-04-05T21:26:40.837 INFO:tasks.workunit.client.0.smithi121.stdout:2/349: mknod d2/d9/d1c/c5d 0 2024-04-05T21:26:40.837 INFO:tasks.workunit.client.0.smithi121.stdout:5/474: rename dd/f29 to dd/de/d19/d1c/d36/f9d 0 2024-04-05T21:26:40.839 INFO:tasks.workunit.client.1.smithi188.stdout:2/687: dwrite d5/db/dd/fd4 [0,4194304] 0 2024-04-05T21:26:40.840 INFO:tasks.workunit.client.1.smithi188.stdout:6/675: write d6/d13/d38/f63 [2004505,96217] 0 2024-04-05T21:26:40.841 INFO:tasks.workunit.client.0.smithi121.stdout:7/404: write d0/d9/d22/f44 [2100565,2] 0 2024-04-05T21:26:40.841 INFO:tasks.workunit.client.1.smithi188.stdout:7/815: truncate d3/d4/d1d/d74/f41 1291998 0 2024-04-05T21:26:40.843 INFO:tasks.workunit.client.1.smithi188.stdout:6/676: dread - d6/d28/f8f zero size 2024-04-05T21:26:40.843 INFO:tasks.workunit.client.1.smithi188.stdout:8/583: unlink d5/d21/d40/d43/fb5 0 2024-04-05T21:26:40.845 INFO:tasks.workunit.client.0.smithi121.stdout:2/350: creat d2/d9/d1c/d30/f5e x:0 0 0 2024-04-05T21:26:40.846 INFO:tasks.workunit.client.1.smithi188.stdout:7/816: dread - d3/d4/d1d/d74/d73/fee zero size 2024-04-05T21:26:40.847 INFO:tasks.workunit.client.1.smithi188.stdout:8/584: write d5/d21/d40/f73 [2821412,77054] 0 2024-04-05T21:26:40.850 INFO:tasks.workunit.client.0.smithi121.stdout:8/329: dwrite d12/d13/d15/f1c [0,4194304] 0 2024-04-05T21:26:40.850 INFO:tasks.workunit.client.0.smithi121.stdout:4/482: dwrite d5/d11/f17 [0,4194304] 0 2024-04-05T21:26:40.853 INFO:tasks.workunit.client.0.smithi121.stdout:1/412: dwrite d8/d10/d14/d1a/d1b/d24/f49 [0,4194304] 0 2024-04-05T21:26:40.855 INFO:tasks.workunit.client.0.smithi121.stdout:7/405: chown d0/d9/f24 570341911 1 2024-04-05T21:26:40.856 INFO:tasks.workunit.client.1.smithi188.stdout:5/576: dread d6/d17/d39/fb3 [0,4194304] 0 2024-04-05T21:26:40.859 INFO:tasks.workunit.client.0.smithi121.stdout:2/351: symlink d2/d9/d1c/l5f 0 2024-04-05T21:26:40.861 INFO:tasks.workunit.client.1.smithi188.stdout:9/631: write dd/d11/d22/d7e/f91 [866637,42691] 0 2024-04-05T21:26:40.861 INFO:tasks.workunit.client.0.smithi121.stdout:7/406: fdatasync d0/d9/d22/d28/d3b/f4f 0 2024-04-05T21:26:40.862 INFO:tasks.workunit.client.1.smithi188.stdout:2/688: write d5/db/dd/d24/f28 [2234594,21227] 0 2024-04-05T21:26:40.865 INFO:tasks.workunit.client.1.smithi188.stdout:9/632: readlink dd/d11/d1f/d26/dc0/d6b/d40/d59/l7d 0 2024-04-05T21:26:40.865 INFO:tasks.workunit.client.1.smithi188.stdout:2/689: write d5/db/dd/d67/dcd/d46/faa [209971,10515] 0 2024-04-05T21:26:40.867 INFO:tasks.workunit.client.1.smithi188.stdout:6/677: rename d6/d13/d38/fcd to d6/d13/d18/d3c/fcf 0 2024-04-05T21:26:40.868 INFO:tasks.workunit.client.1.smithi188.stdout:2/690: stat d5/db/dd/d73/fca 0 2024-04-05T21:26:40.869 INFO:tasks.workunit.client.0.smithi121.stdout:8/330: dwrite d12/d13/d15/d1d/f36 [0,4194304] 0 2024-04-05T21:26:40.872 INFO:tasks.workunit.client.0.smithi121.stdout:5/475: link dd/de/d19/d13/d53/f7f dd/de/d19/d1c/d36/d46/f9e 0 2024-04-05T21:26:40.872 INFO:tasks.workunit.client.0.smithi121.stdout:1/413: mkdir d8/d10/d14/d1a/d1b/d85 0 2024-04-05T21:26:40.876 INFO:tasks.workunit.client.0.smithi121.stdout:5/476: read - dd/de/d39/f8d zero size 2024-04-05T21:26:40.878 INFO:tasks.workunit.client.1.smithi188.stdout:2/691: write d5/db/dd/d67/dcd/d46/f81 [1296156,4173] 0 2024-04-05T21:26:40.878 INFO:tasks.workunit.client.1.smithi188.stdout:9/633: write dd/d11/d1f/d25/fce [97859,76525] 0 2024-04-05T21:26:40.878 INFO:tasks.workunit.client.0.smithi121.stdout:8/331: symlink d12/d13/d15/d1d/l71 0 2024-04-05T21:26:40.880 INFO:tasks.workunit.client.1.smithi188.stdout:3/684: link d5/d7/d65/d61/c6a d5/d7/d65/d5f/cec 0 2024-04-05T21:26:40.880 INFO:tasks.workunit.client.0.smithi121.stdout:1/414: mknod d8/d10/d14/d27/d5c/c86 0 2024-04-05T21:26:40.883 INFO:tasks.workunit.client.0.smithi121.stdout:5/477: dread - dd/de/d19/d1c/d36/d46/f64 zero size 2024-04-05T21:26:40.883 INFO:tasks.workunit.client.0.smithi121.stdout:2/352: dwrite d2/d9/d17/d3a/f4a [0,4194304] 0 2024-04-05T21:26:40.884 INFO:tasks.workunit.client.0.smithi121.stdout:3/265: dread d0/f41 [0,4194304] 0 2024-04-05T21:26:40.885 INFO:tasks.workunit.client.0.smithi121.stdout:7/407: dwrite d0/d23/d5b/f3f [0,4194304] 0 2024-04-05T21:26:40.886 INFO:tasks.workunit.client.1.smithi188.stdout:5/577: symlink d6/d15/d5c/d7a/d44/d98/dbf/lca 0 2024-04-05T21:26:40.886 INFO:tasks.workunit.client.0.smithi121.stdout:5/478: chown dd/de/d39/c60 236333996 1 2024-04-05T21:26:40.887 INFO:tasks.workunit.client.1.smithi188.stdout:3/685: mknod d5/df/ced 0 2024-04-05T21:26:40.888 INFO:tasks.workunit.client.0.smithi121.stdout:3/266: fsync d0/d3/d24/d3f/f4b 0 2024-04-05T21:26:40.888 INFO:tasks.workunit.client.0.smithi121.stdout:1/415: unlink d8/d10/d14/d27/d5c/c78 0 2024-04-05T21:26:40.888 INFO:tasks.workunit.client.0.smithi121.stdout:3/267: chown d0/f14 6499 1 2024-04-05T21:26:40.890 INFO:tasks.workunit.client.1.smithi188.stdout:9/634: mkdir dd/d11/d1f/d26/dc0/d6b/d40/dd7 0 2024-04-05T21:26:40.891 INFO:tasks.workunit.client.0.smithi121.stdout:7/408: fsync d0/d9/f26 0 2024-04-05T21:26:40.891 INFO:tasks.workunit.client.0.smithi121.stdout:3/268: write d0/d3/d24/f3c [146696,50345] 0 2024-04-05T21:26:40.892 INFO:tasks.workunit.client.1.smithi188.stdout:2/692: truncate d5/db/dd/f95 322368 0 2024-04-05T21:26:40.893 INFO:tasks.workunit.client.1.smithi188.stdout:9/635: dread - dd/f82 zero size 2024-04-05T21:26:40.897 INFO:tasks.workunit.client.1.smithi188.stdout:6/678: truncate d6/d13/d48/d49/f69 2000563 0 2024-04-05T21:26:40.897 INFO:tasks.workunit.client.0.smithi121.stdout:3/269: read d0/d3/d24/d3d/f57 [3388437,44668] 0 2024-04-05T21:26:40.897 INFO:tasks.workunit.client.0.smithi121.stdout:3/270: write d0/d3/fa [1818499,96343] 0 2024-04-05T21:26:40.898 INFO:tasks.workunit.client.1.smithi188.stdout:8/585: dwrite d5/d21/d40/f4f [0,4194304] 0 2024-04-05T21:26:40.898 INFO:tasks.workunit.client.1.smithi188.stdout:5/578: mknod d6/d82/d40/d42/d81/da6/db2/ccb 0 2024-04-05T21:26:40.899 INFO:tasks.workunit.client.1.smithi188.stdout:4/865: dwrite d0/d2/d17/d48/d70/fda [0,4194304] 0 2024-04-05T21:26:40.899 INFO:tasks.workunit.client.1.smithi188.stdout:2/693: truncate d5/f5c 301315 0 2024-04-05T21:26:40.900 INFO:tasks.workunit.client.0.smithi121.stdout:6/301: sync 2024-04-05T21:26:40.901 INFO:tasks.workunit.client.0.smithi121.stdout:7/409: mknod d0/d9/d22/d28/c7e 0 2024-04-05T21:26:40.903 INFO:tasks.workunit.client.0.smithi121.stdout:1/416: chown d8/d10/d14/d1a/d1b/f69 204 1 2024-04-05T21:26:40.904 INFO:tasks.workunit.client.0.smithi121.stdout:2/353: link d2/c43 d2/d9/d38/c60 0 2024-04-05T21:26:40.905 INFO:tasks.workunit.client.0.smithi121.stdout:7/410: creat d0/d23/f7f x:0 0 0 2024-04-05T21:26:40.908 INFO:tasks.workunit.client.0.smithi121.stdout:0/443: dread d7/d1c/f1f [0,4194304] 0 2024-04-05T21:26:40.909 INFO:tasks.workunit.client.1.smithi188.stdout:9/636: creat dd/d11/d1f/db2/fd8 x:0 0 0 2024-04-05T21:26:40.911 INFO:tasks.workunit.client.1.smithi188.stdout:4/866: write d0/d14/d15/d53/ff0 [568979,105712] 0 2024-04-05T21:26:40.912 INFO:tasks.workunit.client.0.smithi121.stdout:1/417: dread d8/d10/d14/d27/f75 [0,4194304] 0 2024-04-05T21:26:40.913 INFO:tasks.workunit.client.0.smithi121.stdout:6/302: link d1/d2/d3/d5/l8 d1/d2/d1b/d35/d51/l5d 0 2024-04-05T21:26:40.914 INFO:tasks.workunit.client.1.smithi188.stdout:6/679: link d6/f2e d6/d13/d38/fd0 0 2024-04-05T21:26:40.914 INFO:tasks.workunit.client.1.smithi188.stdout:9/637: fsync dd/d11/d1f/d26/dc0/d6b/f94 0 2024-04-05T21:26:40.914 INFO:tasks.workunit.client.0.smithi121.stdout:7/411: truncate d0/d9/f26 4289792 0 2024-04-05T21:26:40.916 INFO:tasks.workunit.client.0.smithi121.stdout:3/271: symlink d0/l5a 0 2024-04-05T21:26:40.919 INFO:tasks.workunit.client.1.smithi188.stdout:7/817: dread d3/d2c/f3f [0,4194304] 0 2024-04-05T21:26:40.920 INFO:tasks.workunit.client.0.smithi121.stdout:0/444: mknod d7/d71/d94/c98 0 2024-04-05T21:26:40.920 INFO:tasks.workunit.client.0.smithi121.stdout:2/354: link d2/d9/fd d2/d9/d1c/d30/f61 0 2024-04-05T21:26:40.920 INFO:tasks.workunit.client.0.smithi121.stdout:6/303: truncate d1/f38 1428240 0 2024-04-05T21:26:40.920 INFO:tasks.workunit.client.0.smithi121.stdout:2/355: readlink d2/d9/d17/l2b 0 2024-04-05T21:26:40.922 INFO:tasks.workunit.client.0.smithi121.stdout:1/418: write d8/d10/d14/d1a/d1b/d38/d4b/f6c [77849,2556] 0 2024-04-05T21:26:40.922 INFO:tasks.workunit.client.1.smithi188.stdout:4/867: dwrite d0/d14/dd8/dc5/f11c [0,4194304] 0 2024-04-05T21:26:40.923 INFO:tasks.workunit.client.1.smithi188.stdout:5/579: link d6/d17/c30 d6/d15/d5c/ccc 0 2024-04-05T21:26:40.925 INFO:tasks.workunit.client.1.smithi188.stdout:9/638: read fc [6080306,87089] 0 2024-04-05T21:26:40.925 INFO:tasks.workunit.client.0.smithi121.stdout:6/304: fsync d1/d2/d1b/f28 0 2024-04-05T21:26:40.926 INFO:tasks.workunit.client.0.smithi121.stdout:9/272: sync 2024-04-05T21:26:40.930 INFO:tasks.workunit.client.1.smithi188.stdout:7/818: creat d3/d2c/dae/f10c x:0 0 0 2024-04-05T21:26:40.931 INFO:tasks.workunit.client.0.smithi121.stdout:4/483: dread d5/d11/d2c/d8c/f94 [0,4194304] 0 2024-04-05T21:26:40.935 INFO:tasks.workunit.client.1.smithi188.stdout:9/639: write dd/d11/d73/d8d/fac [696520,32062] 0 2024-04-05T21:26:40.935 INFO:tasks.workunit.client.0.smithi121.stdout:1/419: dwrite d8/d10/d14/d1a/d1b/d38/d4b/f6d [0,4194304] 0 2024-04-05T21:26:40.935 INFO:tasks.workunit.client.0.smithi121.stdout:6/305: readlink d1/d2/d3/d4/d21/d57/l52 0 2024-04-05T21:26:40.936 INFO:tasks.workunit.client.0.smithi121.stdout:3/272: creat d0/d3/f5b x:0 0 0 2024-04-05T21:26:40.936 INFO:tasks.workunit.client.1.smithi188.stdout:5/580: rename d6/d82/d40/d42/f77 to d6/d15/d5c/d7a/d44/d98/fcd 0 2024-04-05T21:26:40.938 INFO:tasks.workunit.client.0.smithi121.stdout:4/484: mknod d5/d11/d2c/d8c/ca0 0 2024-04-05T21:26:40.938 INFO:tasks.workunit.client.0.smithi121.stdout:9/273: unlink d6/f40 0 2024-04-05T21:26:40.938 INFO:tasks.workunit.client.0.smithi121.stdout:2/356: fdatasync d2/d9/d17/f18 0 2024-04-05T21:26:40.938 INFO:tasks.workunit.client.0.smithi121.stdout:3/273: stat d0/d3/d24/d3d/d1a/d23/d38 0 2024-04-05T21:26:40.939 INFO:tasks.workunit.client.1.smithi188.stdout:6/680: link d6/d13/d18/d19/d61/f8b d6/fd1 0 2024-04-05T21:26:40.939 INFO:tasks.workunit.client.1.smithi188.stdout:7/819: dread d3/d2c/dae/dfd/d10/f95 [0,4194304] 0 2024-04-05T21:26:40.940 INFO:tasks.workunit.client.0.smithi121.stdout:4/485: stat d5/f55 0 2024-04-05T21:26:40.940 INFO:tasks.workunit.client.0.smithi121.stdout:1/420: write d8/d10/d14/d27/d2d/f41 [44769,10719] 0 2024-04-05T21:26:40.942 INFO:tasks.workunit.client.1.smithi188.stdout:4/868: dwrite d0/d14/d2f/ddd/f123 [0,4194304] 0 2024-04-05T21:26:40.942 INFO:tasks.workunit.client.0.smithi121.stdout:9/274: chown d6/df/l14 3595 1 2024-04-05T21:26:40.943 INFO:tasks.workunit.client.0.smithi121.stdout:6/306: unlink d1/d2/d3/d4/d21/d57/l52 0 2024-04-05T21:26:40.944 INFO:tasks.workunit.client.1.smithi188.stdout:4/869: readlink d0/d14/d15/d53/d82/dd5/df4/l120 0 2024-04-05T21:26:40.946 INFO:tasks.workunit.client.0.smithi121.stdout:8/332: dread d12/d13/f20 [0,4194304] 0 2024-04-05T21:26:40.946 INFO:tasks.workunit.client.0.smithi121.stdout:9/275: write d6/d46/f55 [586212,76066] 0 2024-04-05T21:26:40.947 INFO:tasks.workunit.client.0.smithi121.stdout:5/479: truncate dd/de/d19/f80 1867730 0 2024-04-05T21:26:40.947 INFO:tasks.workunit.client.0.smithi121.stdout:9/276: stat d6/df/d48/f4c 0 2024-04-05T21:26:40.951 INFO:tasks.workunit.client.0.smithi121.stdout:6/307: mkdir d1/d2/d1b/d5e 0 2024-04-05T21:26:40.952 INFO:tasks.workunit.client.0.smithi121.stdout:3/274: mknod d0/d3/d24/d3d/d1a/c5c 0 2024-04-05T21:26:40.953 INFO:tasks.workunit.client.1.smithi188.stdout:3/686: dread d5/df/f33 [0,4194304] 0 2024-04-05T21:26:40.953 INFO:tasks.workunit.client.0.smithi121.stdout:2/357: mkdir d2/d9/d56/d62 0 2024-04-05T21:26:40.954 INFO:tasks.workunit.client.1.smithi188.stdout:7/820: read d3/d4/f2d [4013975,127574] 0 2024-04-05T21:26:40.955 INFO:tasks.workunit.client.0.smithi121.stdout:8/333: write d12/f51 [966711,70123] 0 2024-04-05T21:26:40.957 INFO:tasks.workunit.client.0.smithi121.stdout:5/480: creat dd/de/d6d/f9f x:0 0 0 2024-04-05T21:26:40.961 INFO:tasks.workunit.client.1.smithi188.stdout:5/581: rmdir d6/d15 39 2024-04-05T21:26:40.962 INFO:tasks.workunit.client.1.smithi188.stdout:7/821: dread - d3/d2c/d5e/d91/dfb/f105 zero size 2024-04-05T21:26:40.963 INFO:tasks.workunit.client.1.smithi188.stdout:6/681: write d6/d13/d18/f36 [2515902,48947] 0 2024-04-05T21:26:40.963 INFO:tasks.workunit.client.0.smithi121.stdout:7/412: dread d0/f3 [0,4194304] 0 2024-04-05T21:26:40.963 INFO:tasks.workunit.client.0.smithi121.stdout:4/486: truncate d5/d11/d20/f82 315337 0 2024-04-05T21:26:40.965 INFO:tasks.workunit.client.1.smithi188.stdout:8/586: dwrite d5/dc/f33 [0,4194304] 0 2024-04-05T21:26:40.965 INFO:tasks.workunit.client.0.smithi121.stdout:7/413: dread - d0/d23/d5b/d52/f77 zero size 2024-04-05T21:26:40.966 INFO:tasks.workunit.client.1.smithi188.stdout:5/582: chown d6/d17/l1d 4802231 1 2024-04-05T21:26:40.967 INFO:tasks.workunit.client.1.smithi188.stdout:2/694: dread d5/db/dd/d67/dcd/d46/d5d/f5e [0,4194304] 0 2024-04-05T21:26:40.967 INFO:tasks.workunit.client.1.smithi188.stdout:3/687: chown d5/d48/c50 26432 1 2024-04-05T21:26:40.967 INFO:tasks.workunit.client.0.smithi121.stdout:0/445: truncate f2 475130 0 2024-04-05T21:26:40.968 INFO:tasks.workunit.client.0.smithi121.stdout:0/446: readlink d7/dc/le 0 2024-04-05T21:26:40.968 INFO:tasks.workunit.client.0.smithi121.stdout:9/277: mknod d6/df/d50/c57 0 2024-04-05T21:26:40.969 INFO:tasks.workunit.client.1.smithi188.stdout:4/870: dread d0/d14/d15/d53/d82/dd5/ff1 [0,4194304] 0 2024-04-05T21:26:40.971 INFO:tasks.workunit.client.1.smithi188.stdout:3/688: dread - d5/d2b/fb3 zero size 2024-04-05T21:26:40.972 INFO:tasks.workunit.client.0.smithi121.stdout:3/275: symlink d0/d3/d24/l5d 0 2024-04-05T21:26:40.972 INFO:tasks.workunit.client.1.smithi188.stdout:8/587: chown d5/d21/d40/d43/d48/d7e 421342189 1 2024-04-05T21:26:40.973 INFO:tasks.workunit.client.1.smithi188.stdout:5/583: creat d6/d15/d1b/d4f/fce x:0 0 0 2024-04-05T21:26:40.974 INFO:tasks.workunit.client.0.smithi121.stdout:4/487: creat d5/d11/d28/d87/fa1 x:0 0 0 2024-04-05T21:26:40.975 INFO:tasks.workunit.client.1.smithi188.stdout:3/689: dread - d5/d7/d65/d61/d66/da4/de3/fea zero size 2024-04-05T21:26:40.976 INFO:tasks.workunit.client.0.smithi121.stdout:1/421: write d8/d10/d14/f19 [1111527,48660] 0 2024-04-05T21:26:40.977 INFO:tasks.workunit.client.1.smithi188.stdout:6/682: getdents d6/d13/d48 0 2024-04-05T21:26:40.977 INFO:tasks.workunit.client.0.smithi121.stdout:2/358: truncate d2/d9/d17/f18 2703793 0 2024-04-05T21:26:40.978 INFO:tasks.workunit.client.0.smithi121.stdout:1/422: dread - d8/d10/d14/d1a/d1b/d38/f5d zero size 2024-04-05T21:26:40.979 INFO:tasks.workunit.client.1.smithi188.stdout:5/584: mkdir d6/d17/d23/d7d/dcf 0 2024-04-05T21:26:40.979 INFO:tasks.workunit.client.0.smithi121.stdout:5/481: symlink dd/de/d19/d38/la0 0 2024-04-05T21:26:40.981 INFO:tasks.workunit.client.0.smithi121.stdout:5/482: chown dd/de/d19/d38/d45/l66 69 1 2024-04-05T21:26:40.982 INFO:tasks.workunit.client.0.smithi121.stdout:1/423: dread d8/d10/d14/d1a/d1b/d38/f42 [0,4194304] 0 2024-04-05T21:26:40.984 INFO:tasks.workunit.client.1.smithi188.stdout:2/695: truncate d5/db/dd/d67/dcd/d46/d9f/fb0 478871 0 2024-04-05T21:26:40.986 INFO:tasks.workunit.client.1.smithi188.stdout:9/640: dread dd/f72 [0,4194304] 0 2024-04-05T21:26:40.987 INFO:tasks.workunit.client.1.smithi188.stdout:5/585: rename d6/dad/f8b to d6/d15/d5c/d7a/fd0 0 2024-04-05T21:26:40.990 INFO:tasks.workunit.client.1.smithi188.stdout:8/588: dwrite d5/d21/d2a/d47/fc5 [0,4194304] 0 2024-04-05T21:26:40.994 INFO:tasks.workunit.client.1.smithi188.stdout:6/683: dread d6/d13/d38/f55 [0,4194304] 0 2024-04-05T21:26:41.001 INFO:tasks.workunit.client.1.smithi188.stdout:4/871: dread d0/d14/d15/d53/fac [0,4194304] 0 2024-04-05T21:26:41.003 INFO:tasks.workunit.client.1.smithi188.stdout:9/641: mknod dd/cd9 0 2024-04-05T21:26:41.006 INFO:tasks.workunit.client.1.smithi188.stdout:5/586: dwrite d6/d15/d5c/d7a/f93 [4194304,4194304] 0 2024-04-05T21:26:41.006 INFO:tasks.workunit.client.1.smithi188.stdout:6/684: mkdir d6/d13/d18/d3c/dd2 0 2024-04-05T21:26:41.006 INFO:tasks.workunit.client.1.smithi188.stdout:9/642: chown dd/d11/d1f/d25/d8a/c90 3 1 2024-04-05T21:26:41.008 INFO:tasks.workunit.client.1.smithi188.stdout:7/822: dread d3/d2c/dae/dfd/d2e/f75 [0,4194304] 0 2024-04-05T21:26:41.009 INFO:tasks.workunit.client.1.smithi188.stdout:5/587: readlink d6/l7 0 2024-04-05T21:26:41.009 INFO:tasks.workunit.client.1.smithi188.stdout:9/643: stat dd/d3d/fbe 0 2024-04-05T21:26:41.013 INFO:tasks.workunit.client.1.smithi188.stdout:6/685: symlink d6/d71/ld3 0 2024-04-05T21:26:41.016 INFO:tasks.workunit.client.1.smithi188.stdout:4/872: creat d0/d2/d52/d126/f129 x:0 0 0 2024-04-05T21:26:41.016 INFO:tasks.workunit.client.1.smithi188.stdout:4/873: write d0/d2/dc7/f109 [635504,85008] 0 2024-04-05T21:26:41.018 INFO:tasks.workunit.client.1.smithi188.stdout:9/644: creat dd/d11/d73/d85/fda x:0 0 0 2024-04-05T21:26:41.022 INFO:tasks.workunit.client.1.smithi188.stdout:5/588: creat d6/d82/d40/dc1/fd1 x:0 0 0 2024-04-05T21:26:41.025 INFO:tasks.workunit.client.1.smithi188.stdout:9/645: dread - dd/d11/d1f/db2/fd8 zero size 2024-04-05T21:26:41.027 INFO:tasks.workunit.client.1.smithi188.stdout:4/874: dread d0/d2/dc7/f109 [0,4194304] 0 2024-04-05T21:26:41.027 INFO:tasks.workunit.client.1.smithi188.stdout:3/690: dread d5/d7/d65/f44 [0,4194304] 0 2024-04-05T21:26:41.029 INFO:tasks.workunit.client.1.smithi188.stdout:8/589: dwrite d5/d21/d2a/d47/f54 [4194304,4194304] 0 2024-04-05T21:26:41.032 INFO:tasks.workunit.client.1.smithi188.stdout:2/696: dread d5/db/dd/d36/d3c/d48/d85/f94 [0,4194304] 0 2024-04-05T21:26:41.035 INFO:tasks.workunit.client.1.smithi188.stdout:7/823: dwrite d3/d2c/dae/dfd/d10/f33 [0,4194304] 0 2024-04-05T21:26:41.036 INFO:tasks.workunit.client.1.smithi188.stdout:3/691: chown d5/d7/d6d/f7a 14284709 1 2024-04-05T21:26:41.036 INFO:tasks.workunit.client.1.smithi188.stdout:4/875: write d0/d14/d15/d53/d82/d8f/fad [4063407,104249] 0 2024-04-05T21:26:41.040 INFO:tasks.workunit.client.1.smithi188.stdout:5/589: dread d6/d82/d40/fb6 [0,4194304] 0 2024-04-05T21:26:41.042 INFO:tasks.workunit.client.1.smithi188.stdout:7/824: write d3/d2c/dce/f20 [4746107,118809] 0 2024-04-05T21:26:41.042 INFO:tasks.workunit.client.1.smithi188.stdout:5/590: fdatasync d6/d82/f47 0 2024-04-05T21:26:41.043 INFO:tasks.workunit.client.1.smithi188.stdout:3/692: creat d5/df/d80/fee x:0 0 0 2024-04-05T21:26:41.046 INFO:tasks.workunit.client.1.smithi188.stdout:9/646: dwrite dd/d11/d73/d85/fda [0,4194304] 0 2024-04-05T21:26:41.047 INFO:tasks.workunit.client.1.smithi188.stdout:3/693: fsync d5/d7e/fc7 0 2024-04-05T21:26:41.049 INFO:tasks.workunit.client.1.smithi188.stdout:4/876: dread - d0/d14/d15/dd4/ddf/fe1 zero size 2024-04-05T21:26:41.055 INFO:tasks.workunit.client.0.smithi121.stdout:0/447: unlink d7/d71/d29/d2a/d2c/f43 0 2024-04-05T21:26:41.055 INFO:tasks.workunit.client.1.smithi188.stdout:2/697: truncate d5/db/f20 3169712 0 2024-04-05T21:26:41.056 INFO:tasks.workunit.client.0.smithi121.stdout:9/278: symlink d6/df/l58 0 2024-04-05T21:26:41.058 INFO:tasks.workunit.client.0.smithi121.stdout:8/334: dread d12/d13/d15/f18 [0,4194304] 0 2024-04-05T21:26:41.059 INFO:tasks.workunit.client.1.smithi188.stdout:5/591: truncate d6/d82/f47 3015297 0 2024-04-05T21:26:41.059 INFO:tasks.workunit.client.0.smithi121.stdout:5/483: rmdir dd/de 39 2024-04-05T21:26:41.060 INFO:tasks.workunit.client.0.smithi121.stdout:1/424: rmdir d8/d10/d14/d27 39 2024-04-05T21:26:41.060 INFO:tasks.workunit.client.1.smithi188.stdout:4/877: creat d0/d14/d15/d53/d82/dd5/df4/d106/f12a x:0 0 0 2024-04-05T21:26:41.061 INFO:tasks.workunit.client.0.smithi121.stdout:6/308: link d1/d2/d1b/d2e/l5a d1/d2/d1b/d5e/l5f 0 2024-04-05T21:26:41.061 INFO:tasks.workunit.client.0.smithi121.stdout:4/488: symlink d5/d11/la2 0 2024-04-05T21:26:41.061 INFO:tasks.workunit.client.1.smithi188.stdout:2/698: symlink d5/db/dd/d24/le3 0 2024-04-05T21:26:41.062 INFO:tasks.workunit.client.1.smithi188.stdout:9/647: dwrite dd/d11/d1f/db2/fd8 [0,4194304] 0 2024-04-05T21:26:41.062 INFO:tasks.workunit.client.0.smithi121.stdout:2/359: creat d2/d9/d1c/d30/d59/f63 x:0 0 0 2024-04-05T21:26:41.062 INFO:tasks.workunit.client.0.smithi121.stdout:9/279: mknod d6/df/d50/c59 0 2024-04-05T21:26:41.062 INFO:tasks.workunit.client.0.smithi121.stdout:4/489: fsync d5/d11/f8a 0 2024-04-05T21:26:41.063 INFO:tasks.workunit.client.0.smithi121.stdout:7/414: write d0/d9/d22/d28/f31 [2962608,107260] 0 2024-04-05T21:26:41.064 INFO:tasks.workunit.client.0.smithi121.stdout:0/448: mknod d7/d71/c99 0 2024-04-05T21:26:41.067 INFO:tasks.workunit.client.0.smithi121.stdout:5/484: read - dd/de/d19/d1c/d36/d46/f9e zero size 2024-04-05T21:26:41.067 INFO:tasks.workunit.client.0.smithi121.stdout:9/280: write d6/df/f2e [3120608,24371] 0 2024-04-05T21:26:41.070 INFO:tasks.workunit.client.1.smithi188.stdout:5/592: stat d6/d15/d5c/d7a/d44/d51/dc6 0 2024-04-05T21:26:41.074 INFO:tasks.workunit.client.1.smithi188.stdout:7/825: dwrite d3/d4/d1d/d74/d73/f83 [0,4194304] 0 2024-04-05T21:26:41.075 INFO:tasks.workunit.client.0.smithi121.stdout:6/309: dread d1/d2/dc/f4c [4194304,4194304] 0 2024-04-05T21:26:41.076 INFO:tasks.workunit.client.1.smithi188.stdout:9/648: symlink dd/d11/d73/d8d/ldb 0 2024-04-05T21:26:41.076 INFO:tasks.workunit.client.0.smithi121.stdout:5/485: stat dd/de/d39/l43 0 2024-04-05T21:26:41.077 INFO:tasks.workunit.client.0.smithi121.stdout:7/415: creat d0/d23/d5b/f80 x:0 0 0 2024-04-05T21:26:41.077 INFO:tasks.workunit.client.1.smithi188.stdout:4/878: write d0/d2/d4f/f5c [4684291,118073] 0 2024-04-05T21:26:41.079 INFO:tasks.workunit.client.1.smithi188.stdout:2/699: dwrite d5/db/dd/da4/fc2 [0,4194304] 0 2024-04-05T21:26:41.081 INFO:tasks.workunit.client.0.smithi121.stdout:9/281: rename l3 to d6/df/d1d/d2c/d37/d3c/l5a 0 2024-04-05T21:26:41.081 INFO:tasks.workunit.client.1.smithi188.stdout:4/879: dread - d0/d14/df6/f125 zero size 2024-04-05T21:26:41.082 INFO:tasks.workunit.client.0.smithi121.stdout:0/449: dread d7/f3a [0,4194304] 0 2024-04-05T21:26:41.083 INFO:tasks.workunit.client.1.smithi188.stdout:8/590: dread d5/d21/d2a/f8d [0,4194304] 0 2024-04-05T21:26:41.083 INFO:tasks.workunit.client.1.smithi188.stdout:9/649: creat dd/d11/d1f/d26/dc0/d6b/d40/d59/fdc x:0 0 0 2024-04-05T21:26:41.084 INFO:tasks.workunit.client.0.smithi121.stdout:2/360: getdents d2/d9/d56 0 2024-04-05T21:26:41.085 INFO:tasks.workunit.client.1.smithi188.stdout:3/694: dread d5/d7/d31/f5c [0,4194304] 0 2024-04-05T21:26:41.087 INFO:tasks.workunit.client.1.smithi188.stdout:8/591: unlink d5/fb 0 2024-04-05T21:26:41.088 INFO:tasks.workunit.client.0.smithi121.stdout:5/486: creat dd/de/d39/d3b/fa1 x:0 0 0 2024-04-05T21:26:41.089 INFO:tasks.workunit.client.0.smithi121.stdout:8/335: dwrite d12/d13/d15/f18 [0,4194304] 0 2024-04-05T21:26:41.089 INFO:tasks.workunit.client.0.smithi121.stdout:6/310: chown d1/d2/d3/f4e 234809215 1 2024-04-05T21:26:41.096 INFO:tasks.workunit.client.1.smithi188.stdout:2/700: rename d5/db/dd/d36/d3c/d48/l50 to d5/db/le4 0 2024-04-05T21:26:41.097 INFO:tasks.workunit.client.0.smithi121.stdout:7/416: write d0/d9/f69 [965657,2926] 0 2024-04-05T21:26:41.099 INFO:tasks.workunit.client.1.smithi188.stdout:8/592: mkdir d5/d6e/d98/dc6 0 2024-04-05T21:26:41.100 INFO:tasks.workunit.client.1.smithi188.stdout:8/593: dread - d5/d21/d2a/d71/fbc zero size 2024-04-05T21:26:41.103 INFO:tasks.workunit.client.1.smithi188.stdout:9/650: getdents dd/d11/d1f 0 2024-04-05T21:26:41.103 INFO:tasks.workunit.client.1.smithi188.stdout:5/593: write d6/d15/f7f [883367,123988] 0 2024-04-05T21:26:41.103 INFO:tasks.workunit.client.1.smithi188.stdout:7/826: write d3/d2c/f10a [2383202,56961] 0 2024-04-05T21:26:41.104 INFO:tasks.workunit.client.0.smithi121.stdout:5/487: creat dd/de/d19/d38/d45/d68/fa2 x:0 0 0 2024-04-05T21:26:41.104 INFO:tasks.workunit.client.0.smithi121.stdout:2/361: chown d2/d9/d1c/d30/f5e 14092942 1 2024-04-05T21:26:41.104 INFO:tasks.workunit.client.0.smithi121.stdout:1/425: dread d8/d10/d14/d27/f30 [0,4194304] 0 2024-04-05T21:26:41.104 INFO:tasks.workunit.client.0.smithi121.stdout:4/490: dwrite d5/d11/d28/f44 [0,4194304] 0 2024-04-05T21:26:41.104 INFO:tasks.workunit.client.0.smithi121.stdout:6/311: rename d1/d2/l3a to d1/d2/d3/d5/d1f/l60 0 2024-04-05T21:26:41.104 INFO:tasks.workunit.client.0.smithi121.stdout:4/491: stat d5/d11/d26/d47/l92 0 2024-04-05T21:26:41.104 INFO:tasks.workunit.client.0.smithi121.stdout:7/417: mkdir d0/d9/d22/d81 0 2024-04-05T21:26:41.105 INFO:tasks.workunit.client.1.smithi188.stdout:4/880: write d0/fa8 [300079,48487] 0 2024-04-05T21:26:41.105 INFO:tasks.workunit.client.0.smithi121.stdout:2/362: unlink d2/d9/d1c/f22 0 2024-04-05T21:26:41.106 INFO:tasks.workunit.client.1.smithi188.stdout:8/594: dread - d5/d6e/f84 zero size 2024-04-05T21:26:41.107 INFO:tasks.workunit.client.0.smithi121.stdout:4/492: fdatasync d5/d11/d28/d3b/f59 0 2024-04-05T21:26:41.108 INFO:tasks.workunit.client.1.smithi188.stdout:7/827: chown d3/d4/d1d/d74/d73 7703 1 2024-04-05T21:26:41.108 INFO:tasks.workunit.client.1.smithi188.stdout:9/651: creat dd/d11/d1f/fdd x:0 0 0 2024-04-05T21:26:41.109 INFO:tasks.workunit.client.1.smithi188.stdout:4/881: chown d0/d14/d15/d42/fa5 198379 1 2024-04-05T21:26:41.109 INFO:tasks.workunit.client.1.smithi188.stdout:8/595: chown d5/f34 925 1 2024-04-05T21:26:41.110 INFO:tasks.workunit.client.0.smithi121.stdout:9/282: rename d6/c4b to d6/df/d1d/d2c/d53/c5b 0 2024-04-05T21:26:41.110 INFO:tasks.workunit.client.0.smithi121.stdout:5/488: mkdir dd/de/d19/d38/da3 0 2024-04-05T21:26:41.110 INFO:tasks.workunit.client.1.smithi188.stdout:5/594: symlink d6/d17/d23/d7d/dcf/ld2 0 2024-04-05T21:26:41.111 INFO:tasks.workunit.client.0.smithi121.stdout:6/312: symlink d1/d2/d3/d4/de/l61 0 2024-04-05T21:26:41.112 INFO:tasks.workunit.client.0.smithi121.stdout:1/426: mknod d8/d10/d14/d1a/d1b/d4c/d76/c87 0 2024-04-05T21:26:41.112 INFO:tasks.workunit.client.0.smithi121.stdout:6/313: stat d1/d2/d1b/d2e 0 2024-04-05T21:26:41.112 INFO:tasks.workunit.client.1.smithi188.stdout:9/652: unlink dd/d3d/d95/fba 0 2024-04-05T21:26:41.113 INFO:tasks.workunit.client.0.smithi121.stdout:7/418: read d0/d23/d5b/f2f [2017526,99663] 0 2024-04-05T21:26:41.115 INFO:tasks.workunit.client.1.smithi188.stdout:5/595: creat d6/d15/d5c/d7a/fd3 x:0 0 0 2024-04-05T21:26:41.115 INFO:tasks.workunit.client.0.smithi121.stdout:0/450: rename d7/d71/d29/d2a/d2c/l8a to d7/d71/d45/d46/d53/l9a 0 2024-04-05T21:26:41.116 INFO:tasks.workunit.client.1.smithi188.stdout:9/653: creat dd/d34/fde x:0 0 0 2024-04-05T21:26:41.118 INFO:tasks.workunit.client.1.smithi188.stdout:8/596: dwrite d5/d21/d2a/f31 [0,4194304] 0 2024-04-05T21:26:41.119 INFO:tasks.workunit.client.0.smithi121.stdout:6/314: chown d1/d2/d3/d4/d21/d57/d42/d48 1616 1 2024-04-05T21:26:41.120 INFO:tasks.workunit.client.0.smithi121.stdout:5/489: dwrite dd/de/d19/f51 [0,4194304] 0 2024-04-05T21:26:41.121 INFO:tasks.workunit.client.1.smithi188.stdout:8/597: write d5/d21/d2a/f8d [2242283,74772] 0 2024-04-05T21:26:41.121 INFO:tasks.workunit.client.0.smithi121.stdout:7/419: unlink d0/d23/d5b/f3f 0 2024-04-05T21:26:41.123 INFO:tasks.workunit.client.1.smithi188.stdout:8/598: symlink d5/d21/d2a/d71/lc7 0 2024-04-05T21:26:41.124 INFO:tasks.workunit.client.0.smithi121.stdout:5/490: stat dd/de/d39/d3b/d50/f5b 0 2024-04-05T21:26:41.125 INFO:tasks.workunit.client.1.smithi188.stdout:5/596: dwrite d6/d17/d39/f4a [0,4194304] 0 2024-04-05T21:26:41.125 INFO:tasks.workunit.client.0.smithi121.stdout:4/493: rename d5/d11/d26/d49/l56 to d5/d89/la3 0 2024-04-05T21:26:41.126 INFO:tasks.workunit.client.1.smithi188.stdout:7/828: dread d3/d2c/dae/dfd/f4a [0,4194304] 0 2024-04-05T21:26:41.126 INFO:tasks.workunit.client.0.smithi121.stdout:2/363: creat d2/f64 x:0 0 0 2024-04-05T21:26:41.127 INFO:tasks.workunit.client.0.smithi121.stdout:6/315: readlink d1/l34 0 2024-04-05T21:26:41.127 INFO:tasks.workunit.client.1.smithi188.stdout:8/599: unlink d5/dc/l3f 0 2024-04-05T21:26:41.127 INFO:tasks.workunit.client.1.smithi188.stdout:8/600: stat d5/dc/d8f/db8 0 2024-04-05T21:26:41.129 INFO:tasks.workunit.client.1.smithi188.stdout:7/829: mkdir d3/d2c/dae/dfd/d10/d5f/dc6/d104/d10d 0 2024-04-05T21:26:41.130 INFO:tasks.workunit.client.0.smithi121.stdout:9/283: dread d6/f15 [0,4194304] 0 2024-04-05T21:26:41.130 INFO:tasks.workunit.client.0.smithi121.stdout:5/491: write dd/de/d19/d1c/d36/d46/f8a [840047,72421] 0 2024-04-05T21:26:41.130 INFO:tasks.workunit.client.0.smithi121.stdout:4/494: fdatasync d5/d11/d20/f82 0 2024-04-05T21:26:41.131 INFO:tasks.workunit.client.1.smithi188.stdout:7/830: write d3/f1e [1416606,83291] 0 2024-04-05T21:26:41.131 INFO:tasks.workunit.client.1.smithi188.stdout:8/601: dread d5/d21/d2a/f39 [0,4194304] 0 2024-04-05T21:26:41.133 INFO:tasks.workunit.client.0.smithi121.stdout:2/364: mknod d2/d9/d56/d5c/c65 0 2024-04-05T21:26:41.133 INFO:tasks.workunit.client.0.smithi121.stdout:0/451: link d7/d1c/f68 d7/d71/d29/f9b 0 2024-04-05T21:26:41.134 INFO:tasks.workunit.client.0.smithi121.stdout:1/427: dwrite d8/d10/d14/d27/f75 [4194304,4194304] 0 2024-04-05T21:26:41.135 INFO:tasks.workunit.client.0.smithi121.stdout:5/492: dread - dd/de/d19/d38/f97 zero size 2024-04-05T21:26:41.135 INFO:tasks.workunit.client.1.smithi188.stdout:8/602: rename d5/d21/d40/d43/d44/ca9 to d5/d21/d40/d43/d44/d5b/d99/cc8 0 2024-04-05T21:26:41.137 INFO:tasks.workunit.client.1.smithi188.stdout:3/695: dread d5/d7/f30 [0,4194304] 0 2024-04-05T21:26:41.138 INFO:tasks.workunit.client.1.smithi188.stdout:8/603: read d5/f2d [2654191,125134] 0 2024-04-05T21:26:41.139 INFO:tasks.workunit.client.1.smithi188.stdout:7/831: dwrite d3/da5/fd3 [0,4194304] 0 2024-04-05T21:26:41.139 INFO:tasks.workunit.client.1.smithi188.stdout:8/604: readlink d5/dc/l25 0 2024-04-05T21:26:41.140 INFO:tasks.workunit.client.0.smithi121.stdout:2/365: dread d2/d9/d12/f37 [0,4194304] 0 2024-04-05T21:26:41.140 INFO:tasks.workunit.client.1.smithi188.stdout:3/696: write d5/d7/d65/d61/db5/fbf [519440,105684] 0 2024-04-05T21:26:41.142 INFO:tasks.workunit.client.0.smithi121.stdout:5/493: creat dd/de/d19/d1c/d74/fa4 x:0 0 0 2024-04-05T21:26:41.143 INFO:tasks.workunit.client.1.smithi188.stdout:7/832: mkdir d3/d4/d1d/d74/d10e 0 2024-04-05T21:26:41.143 INFO:tasks.workunit.client.0.smithi121.stdout:2/366: rmdir d2/d9/d38/d2d 39 2024-04-05T21:26:41.145 INFO:tasks.workunit.client.0.smithi121.stdout:1/428: symlink d8/d10/d14/d1a/d1b/l88 0 2024-04-05T21:26:41.145 INFO:tasks.workunit.client.0.smithi121.stdout:2/367: truncate d2/d9/d1c/d30/f5e 309375 0 2024-04-05T21:26:41.146 INFO:tasks.workunit.client.1.smithi188.stdout:3/697: getdents d5/d7/d65/d61/d66/da4 0 2024-04-05T21:26:41.147 INFO:tasks.workunit.client.1.smithi188.stdout:3/698: fdatasync d5/d7/d65/d61/d66/da4/de3/fea 0 2024-04-05T21:26:41.147 INFO:tasks.workunit.client.0.smithi121.stdout:1/429: stat d8/d10/d14/d1a/d1b/d24/f39 0 2024-04-05T21:26:41.148 INFO:tasks.workunit.client.0.smithi121.stdout:5/494: dread - dd/de/d19/d38/f7c zero size 2024-04-05T21:26:41.148 INFO:tasks.workunit.client.0.smithi121.stdout:1/430: chown f2 2615177 1 2024-04-05T21:26:41.148 INFO:tasks.workunit.client.1.smithi188.stdout:3/699: mknod d5/ddd/cef 0 2024-04-05T21:26:41.149 INFO:tasks.workunit.client.1.smithi188.stdout:3/700: fsync d5/df/d80/fee 0 2024-04-05T21:26:41.149 INFO:tasks.workunit.client.0.smithi121.stdout:2/368: symlink d2/d9/d56/d5c/l66 0 2024-04-05T21:26:41.150 INFO:tasks.workunit.client.0.smithi121.stdout:5/495: mknod dd/de/d19/d13/d53/ca5 0 2024-04-05T21:26:41.152 INFO:tasks.workunit.client.0.smithi121.stdout:1/431: unlink d8/d10/d14/d27/d2d/l61 0 2024-04-05T21:26:41.152 INFO:tasks.workunit.client.0.smithi121.stdout:5/496: dread - dd/de/d19/d1c/d36/f71 zero size 2024-04-05T21:26:41.157 INFO:tasks.workunit.client.0.smithi121.stdout:5/497: mkdir dd/de/d39/d3b/d50/d82/da6 0 2024-04-05T21:26:41.159 INFO:tasks.workunit.client.0.smithi121.stdout:7/420: dread d0/d9/f20 [0,4194304] 0 2024-04-05T21:26:41.160 INFO:tasks.workunit.client.0.smithi121.stdout:2/369: dwrite d2/d9/d38/f35 [0,4194304] 0 2024-04-05T21:26:41.162 INFO:tasks.workunit.client.0.smithi121.stdout:1/432: dwrite d8/d10/d14/d1a/f66 [0,4194304] 0 2024-04-05T21:26:41.168 INFO:tasks.workunit.client.0.smithi121.stdout:7/421: read d0/d9/d22/f44 [58137,78274] 0 2024-04-05T21:26:41.168 INFO:tasks.workunit.client.1.smithi188.stdout:8/605: dread d5/d21/d40/d43/f62 [0,4194304] 0 2024-04-05T21:26:41.172 INFO:tasks.workunit.client.0.smithi121.stdout:5/498: dwrite dd/de/d19/d38/d45/d68/f91 [0,4194304] 0 2024-04-05T21:26:41.173 INFO:tasks.workunit.client.0.smithi121.stdout:7/422: mknod d0/d9/d22/d3e/c82 0 2024-04-05T21:26:41.174 INFO:tasks.workunit.client.0.smithi121.stdout:5/499: chown dd/f1f 5 1 2024-04-05T21:26:41.174 INFO:tasks.workunit.client.0.smithi121.stdout:7/423: read d0/d9/f21 [879223,18367] 0 2024-04-05T21:26:41.174 INFO:tasks.workunit.client.1.smithi188.stdout:8/606: dwrite d5/d21/d2a/f8d [4194304,4194304] 0 2024-04-05T21:26:41.177 INFO:tasks.workunit.client.0.smithi121.stdout:2/370: dread d2/d9/f54 [0,4194304] 0 2024-04-05T21:26:41.178 INFO:tasks.workunit.client.1.smithi188.stdout:8/607: link d5/d21/d40/f57 d5/d21/d2a/d95/dc0/fc9 0 2024-04-05T21:26:41.178 INFO:tasks.workunit.client.0.smithi121.stdout:2/371: write d2/d9/f1f [1013289,93604] 0 2024-04-05T21:26:41.178 INFO:tasks.workunit.client.0.smithi121.stdout:5/500: mknod dd/de/d19/ca7 0 2024-04-05T21:26:41.179 INFO:tasks.workunit.client.1.smithi188.stdout:8/608: fdatasync d5/f34 0 2024-04-05T21:26:41.180 INFO:tasks.workunit.client.0.smithi121.stdout:7/424: creat d0/d32/f83 x:0 0 0 2024-04-05T21:26:41.181 INFO:tasks.workunit.client.0.smithi121.stdout:5/501: chown dd/de/d39/d3b/fa1 13735 1 2024-04-05T21:26:41.181 INFO:tasks.workunit.client.0.smithi121.stdout:7/425: stat d0/d32/d4a 0 2024-04-05T21:26:41.182 INFO:tasks.workunit.client.0.smithi121.stdout:2/372: mkdir d2/d9/d56/d62/d67 0 2024-04-05T21:26:41.182 INFO:tasks.workunit.client.0.smithi121.stdout:5/502: chown dd/de/d19/d1c/d74/d8c 26414781 1 2024-04-05T21:26:41.184 INFO:tasks.workunit.client.0.smithi121.stdout:7/426: write d0/d9/f12 [6925523,127865] 0 2024-04-05T21:26:41.184 INFO:tasks.workunit.client.0.smithi121.stdout:2/373: chown d2/d9/d38/d2d 109 1 2024-04-05T21:26:41.185 INFO:tasks.workunit.client.1.smithi188.stdout:0/533: sync 2024-04-05T21:26:41.186 INFO:tasks.workunit.client.0.smithi121.stdout:5/503: fsync dd/de/d19/d38/f4f 0 2024-04-05T21:26:41.187 INFO:tasks.workunit.client.1.smithi188.stdout:8/609: rename d5/d21/d2a/la7 to d5/d21/d40/d43/d44/d5b/lca 0 2024-04-05T21:26:41.192 INFO:tasks.workunit.client.0.smithi121.stdout:7/427: creat d0/d23/d5b/d52/f84 x:0 0 0 2024-04-05T21:26:41.192 INFO:tasks.workunit.client.0.smithi121.stdout:5/504: symlink dd/de/d19/d1c/d36/d46/la8 0 2024-04-05T21:26:41.192 INFO:tasks.workunit.client.0.smithi121.stdout:7/428: write d0/d32/f83 [166185,20096] 0 2024-04-05T21:26:41.192 INFO:tasks.workunit.client.0.smithi121.stdout:7/429: chown d0/d32/f6b 12542 1 2024-04-05T21:26:41.192 INFO:tasks.workunit.client.0.smithi121.stdout:2/374: link d2/d1e/f52 d2/d9/d1c/d30/f68 0 2024-04-05T21:26:41.193 INFO:tasks.workunit.client.0.smithi121.stdout:2/375: chown d2/d9/d17/f42 1699735 1 2024-04-05T21:26:41.194 INFO:tasks.workunit.client.0.smithi121.stdout:5/505: mknod dd/de/d19/d38/da3/ca9 0 2024-04-05T21:26:41.196 INFO:tasks.workunit.client.0.smithi121.stdout:7/430: mknod d0/d23/c85 0 2024-04-05T21:26:41.197 INFO:tasks.workunit.client.1.smithi188.stdout:8/610: dread - d5/d6e/f84 zero size 2024-04-05T21:26:41.199 INFO:tasks.workunit.client.0.smithi121.stdout:5/506: symlink dd/de/d19/d1c/d36/laa 0 2024-04-05T21:26:41.199 INFO:tasks.workunit.client.0.smithi121.stdout:7/431: write d0/d23/d5b/d52/f77 [51635,70172] 0 2024-04-05T21:26:41.213 INFO:tasks.workunit.client.1.smithi188.stdout:0/534: dread d4/d8/db/d21/d4a/f58 [0,4194304] 0 2024-04-05T21:26:41.218 INFO:tasks.workunit.client.0.smithi121.stdout:7/432: read d0/d9/f24 [2254863,112048] 0 2024-04-05T21:26:41.221 INFO:tasks.workunit.client.0.smithi121.stdout:7/433: rmdir d0/d23/d5b/d78/d7a 0 2024-04-05T21:26:41.222 INFO:tasks.workunit.client.0.smithi121.stdout:9/284: write d6/f1c [4524030,42956] 0 2024-04-05T21:26:41.223 INFO:tasks.workunit.client.0.smithi121.stdout:9/285: fdatasync d6/df/d1d/f2b 0 2024-04-05T21:26:41.224 INFO:tasks.workunit.client.0.smithi121.stdout:4/495: truncate d5/d11/d28/d87/f96 1763399 0 2024-04-05T21:26:41.225 INFO:tasks.workunit.client.0.smithi121.stdout:3/276: sync 2024-04-05T21:26:41.227 INFO:tasks.workunit.client.0.smithi121.stdout:1/433: rmdir d8 39 2024-04-05T21:26:41.227 INFO:tasks.workunit.client.0.smithi121.stdout:9/286: dread d6/df/d48/f4c [0,4194304] 0 2024-04-05T21:26:41.228 INFO:tasks.workunit.client.0.smithi121.stdout:4/496: write d5/d11/d26/d47/f66 [1897077,86910] 0 2024-04-05T21:26:41.232 INFO:tasks.workunit.client.0.smithi121.stdout:1/434: write d8/d10/d14/f37 [287609,21921] 0 2024-04-05T21:26:41.233 INFO:tasks.workunit.client.0.smithi121.stdout:4/497: mkdir d5/d11/d2c/da4 0 2024-04-05T21:26:41.234 INFO:tasks.workunit.client.0.smithi121.stdout:4/498: read - d5/d45/d67/f84 zero size 2024-04-05T21:26:41.235 INFO:tasks.workunit.client.0.smithi121.stdout:7/434: rename d0/d23/d5b/d52/f84 to d0/d9/f86 0 2024-04-05T21:26:41.236 INFO:tasks.workunit.client.0.smithi121.stdout:3/277: creat d0/d3/f5e x:0 0 0 2024-04-05T21:26:41.237 INFO:tasks.workunit.client.0.smithi121.stdout:0/452: dread d7/d1c/f5b [0,4194304] 0 2024-04-05T21:26:41.237 INFO:tasks.workunit.client.0.smithi121.stdout:6/316: dread d1/d2/d3/d4/de/f26 [0,4194304] 0 2024-04-05T21:26:41.238 INFO:tasks.workunit.client.0.smithi121.stdout:1/435: mkdir d8/d89 0 2024-04-05T21:26:41.239 INFO:tasks.workunit.client.0.smithi121.stdout:3/278: chown d0/d3/d24/d3d 1800358 1 2024-04-05T21:26:41.239 INFO:tasks.workunit.client.0.smithi121.stdout:8/336: dread fc [0,4194304] 0 2024-04-05T21:26:41.242 INFO:tasks.workunit.client.0.smithi121.stdout:6/317: rename d1/f10 to d1/d2/d1b/d35/f62 0 2024-04-05T21:26:41.245 INFO:tasks.workunit.client.0.smithi121.stdout:1/436: truncate d8/d10/d14/d1a/f5a 5072649 0 2024-04-05T21:26:41.246 INFO:tasks.workunit.client.0.smithi121.stdout:7/435: dread d0/d9/d22/d28/f48 [0,4194304] 0 2024-04-05T21:26:41.250 INFO:tasks.workunit.client.0.smithi121.stdout:0/453: dwrite d7/d71/d29/f79 [0,4194304] 0 2024-04-05T21:26:41.250 INFO:tasks.workunit.client.0.smithi121.stdout:7/436: chown d0/d9/d22/f6c 1639 1 2024-04-05T21:26:41.251 INFO:tasks.workunit.client.0.smithi121.stdout:6/318: mknod d1/d2/d3/d4/d21/d57/d42/c63 0 2024-04-05T21:26:41.254 INFO:tasks.workunit.client.0.smithi121.stdout:6/319: write d1/d2/d3/d5/fd [1894819,56660] 0 2024-04-05T21:26:41.254 INFO:tasks.workunit.client.0.smithi121.stdout:8/337: getdents d12/d13/d15/d1d 0 2024-04-05T21:26:41.256 INFO:tasks.workunit.client.0.smithi121.stdout:1/437: creat d8/d10/d14/d27/d2d/f8a x:0 0 0 2024-04-05T21:26:41.256 INFO:tasks.workunit.client.1.smithi188.stdout:7/833: dread d3/d2c/dae/dfd/d10/f82 [0,4194304] 0 2024-04-05T21:26:41.258 INFO:tasks.workunit.client.0.smithi121.stdout:3/279: link d0/f27 d0/d3/d24/d3d/d1a/f5f 0 2024-04-05T21:26:41.259 INFO:tasks.workunit.client.1.smithi188.stdout:3/701: dread d5/d7/d6d/dd8/d2f/f4c [0,4194304] 0 2024-04-05T21:26:41.260 INFO:tasks.workunit.client.0.smithi121.stdout:7/437: truncate d0/d9/f21 1955671 0 2024-04-05T21:26:41.266 INFO:tasks.workunit.client.0.smithi121.stdout:8/338: creat d12/d13/d56/d6d/f72 x:0 0 0 2024-04-05T21:26:41.270 INFO:tasks.workunit.client.0.smithi121.stdout:1/438: creat d8/d10/d14/d1a/d1b/d24/f8b x:0 0 0 2024-04-05T21:26:41.273 INFO:tasks.workunit.client.0.smithi121.stdout:7/438: symlink d0/d9/d22/d3e/l87 0 2024-04-05T21:26:41.274 INFO:tasks.workunit.client.0.smithi121.stdout:5/507: dwrite dd/de/d6d/f70 [0,4194304] 0 2024-04-05T21:26:41.280 INFO:tasks.workunit.client.1.smithi188.stdout:1/634: sync 2024-04-05T21:26:41.281 INFO:tasks.workunit.client.0.smithi121.stdout:0/454: rename d7/c38 to d7/d71/d29/d2a/c9c 0 2024-04-05T21:26:41.281 INFO:tasks.workunit.client.0.smithi121.stdout:4/499: truncate d5/d11/d2c/d8d/f4c 1268965 0 2024-04-05T21:26:41.285 INFO:tasks.workunit.client.0.smithi121.stdout:0/455: chown d7/f3a 93068 1 2024-04-05T21:26:41.289 INFO:tasks.workunit.client.0.smithi121.stdout:1/439: dwrite d8/d10/d14/d27/d57/f7f [0,4194304] 0 2024-04-05T21:26:41.290 INFO:tasks.workunit.client.0.smithi121.stdout:6/320: dread d1/d2/d3/f4e [0,4194304] 0 2024-04-05T21:26:41.291 INFO:tasks.workunit.client.0.smithi121.stdout:1/440: chown d8/d10/d14/d27/d5c/c86 120504 1 2024-04-05T21:26:41.291 INFO:tasks.workunit.client.1.smithi188.stdout:1/635: dwrite d1/dc/d16/d46/d88/d13/f82 [0,4194304] 0 2024-04-05T21:26:41.295 INFO:tasks.workunit.client.0.smithi121.stdout:8/339: truncate d12/f5d 632742 0 2024-04-05T21:26:41.297 INFO:tasks.workunit.client.0.smithi121.stdout:5/508: dread dd/de/d19/d1c/d36/d46/f8a [0,4194304] 0 2024-04-05T21:26:41.299 INFO:tasks.workunit.client.0.smithi121.stdout:8/340: write d12/d13/d56/d57/d5e/f62 [771389,78404] 0 2024-04-05T21:26:41.300 INFO:tasks.workunit.client.0.smithi121.stdout:5/509: fdatasync dd/de/d19/d1c/d74/fa4 0 2024-04-05T21:26:41.305 INFO:tasks.workunit.client.1.smithi188.stdout:2/701: write d5/db/f76 [2797599,119426] 0 2024-04-05T21:26:41.305 INFO:tasks.workunit.client.0.smithi121.stdout:1/441: dwrite d8/d10/d14/d1a/d1b/d38/f5d [0,4194304] 0 2024-04-05T21:26:41.307 INFO:tasks.workunit.client.1.smithi188.stdout:0/535: mkdir d4/d8/d78/dc2 0 2024-04-05T21:26:41.308 INFO:tasks.workunit.client.1.smithi188.stdout:1/636: truncate d1/d7/d4e/fb6 823218 0 2024-04-05T21:26:41.310 INFO:tasks.workunit.client.1.smithi188.stdout:3/702: fdatasync d5/d7/d65/d61/d66/f9f 0 2024-04-05T21:26:41.313 INFO:tasks.workunit.client.1.smithi188.stdout:3/703: write d5/ddd/fcc [939323,51130] 0 2024-04-05T21:26:41.313 INFO:tasks.workunit.client.0.smithi121.stdout:1/442: readlink d8/d10/d14/d1a/d1b/d4c/d76/d53/l5e 0 2024-04-05T21:26:41.314 INFO:tasks.workunit.client.0.smithi121.stdout:1/443: dread - d8/d10/d14/f23 zero size 2024-04-05T21:26:41.324 INFO:tasks.workunit.client.1.smithi188.stdout:0/536: stat d4/d8/d10/f1b 0 2024-04-05T21:26:41.328 INFO:tasks.workunit.client.0.smithi121.stdout:6/321: creat d1/d2/d3/d5/d1f/f64 x:0 0 0 2024-04-05T21:26:41.328 INFO:tasks.workunit.client.1.smithi188.stdout:3/704: unlink d5/d7/d31/fb2 0 2024-04-05T21:26:41.332 INFO:tasks.workunit.client.0.smithi121.stdout:5/510: mknod dd/de/d19/d1c/d7b/d88/cab 0 2024-04-05T21:26:41.333 INFO:tasks.workunit.client.1.smithi188.stdout:9/654: mknod dd/d11/d1f/d25/cdf 0 2024-04-05T21:26:41.335 INFO:tasks.workunit.client.1.smithi188.stdout:0/537: dwrite d4/d6/d61/f8f [0,4194304] 0 2024-04-05T21:26:41.337 INFO:tasks.workunit.client.1.smithi188.stdout:3/705: creat d5/d7/d65/d61/db5/ff0 x:0 0 0 2024-04-05T21:26:41.343 INFO:tasks.workunit.client.1.smithi188.stdout:0/538: mknod d4/d8/db/d42/d71/cc3 0 2024-04-05T21:26:41.349 INFO:tasks.workunit.client.1.smithi188.stdout:7/834: dread d3/d2c/dae/dfd/f6e [0,4194304] 0 2024-04-05T21:26:41.351 INFO:tasks.workunit.client.1.smithi188.stdout:7/835: write d3/d2c/f3f [2750106,2120] 0 2024-04-05T21:26:41.352 INFO:tasks.workunit.client.1.smithi188.stdout:3/706: dwrite d5/d7/d6d/d75/fb8 [0,4194304] 0 2024-04-05T21:26:41.354 INFO:tasks.workunit.client.1.smithi188.stdout:3/707: dread - d5/d7/f5e zero size 2024-04-05T21:26:41.356 INFO:tasks.workunit.client.1.smithi188.stdout:3/708: fdatasync d5/d7/d65/d61/d66/da4/de3/fea 0 2024-04-05T21:26:41.358 INFO:tasks.workunit.client.1.smithi188.stdout:7/836: write d3/d4/d1d/d74/f59 [5169639,26528] 0 2024-04-05T21:26:41.359 INFO:tasks.workunit.client.1.smithi188.stdout:0/539: truncate d4/d8/db/d21/d3c/f9e 758205 0 2024-04-05T21:26:41.366 INFO:tasks.workunit.client.1.smithi188.stdout:0/540: dread d4/f53 [0,4194304] 0 2024-04-05T21:26:41.366 INFO:tasks.workunit.client.1.smithi188.stdout:7/837: creat d3/d2c/dae/dfd/d10/dbe/ded/f10f x:0 0 0 2024-04-05T21:26:41.367 INFO:tasks.workunit.client.0.smithi121.stdout:6/322: creat d1/d2/d1b/d35/d39/f65 x:0 0 0 2024-04-05T21:26:41.368 INFO:tasks.workunit.client.0.smithi121.stdout:6/323: stat d1/d2/d1b/d35/f47 0 2024-04-05T21:26:41.369 INFO:tasks.workunit.client.0.smithi121.stdout:6/324: write d1/d2/dc/f19 [3028604,116463] 0 2024-04-05T21:26:41.370 INFO:tasks.workunit.client.1.smithi188.stdout:7/838: creat d3/d2c/dae/dfd/d10/dbe/ded/f110 x:0 0 0 2024-04-05T21:26:41.374 INFO:tasks.workunit.client.1.smithi188.stdout:7/839: creat d3/d4/d7c/f111 x:0 0 0 2024-04-05T21:26:41.376 INFO:tasks.workunit.client.0.smithi121.stdout:6/325: truncate d1/f1c 5633955 0 2024-04-05T21:26:41.382 INFO:tasks.workunit.client.1.smithi188.stdout:7/840: link d3/d2c/dae/dfd/d2e/d8c/db7/dc4/cd7 d3/d2c/dae/dfd/d10/c112 0 2024-04-05T21:26:41.382 INFO:tasks.workunit.client.0.smithi121.stdout:0/456: mkdir d7/d62/d9d 0 2024-04-05T21:26:41.386 INFO:tasks.workunit.client.0.smithi121.stdout:0/457: symlink d7/d71/d45/d46/d53/d8e/l9e 0 2024-04-05T21:26:41.387 INFO:tasks.workunit.client.0.smithi121.stdout:0/458: dread - d7/dc/d95/f96 zero size 2024-04-05T21:26:41.387 INFO:tasks.workunit.client.0.smithi121.stdout:0/459: chown d7/d62/d3c/f4b 5 1 2024-04-05T21:26:41.390 INFO:tasks.workunit.client.0.smithi121.stdout:0/460: creat d7/d71/d29/d2a/d64/f9f x:0 0 0 2024-04-05T21:26:41.396 INFO:tasks.workunit.client.0.smithi121.stdout:0/461: truncate d7/d71/d29/f34 2087692 0 2024-04-05T21:26:41.399 INFO:tasks.workunit.client.0.smithi121.stdout:0/462: dread - d7/d71/d29/d2a/d64/f9f zero size 2024-04-05T21:26:41.401 INFO:tasks.workunit.client.1.smithi188.stdout:8/611: truncate d5/d21/d40/d43/d44/d5b/d99/da3/fba 1604369 0 2024-04-05T21:26:41.401 INFO:tasks.workunit.client.0.smithi121.stdout:4/500: dwrite d5/d11/d20/f82 [0,4194304] 0 2024-04-05T21:26:41.401 INFO:tasks.workunit.client.0.smithi121.stdout:0/463: mkdir d7/d71/d94/da0 0 2024-04-05T21:26:41.403 INFO:tasks.workunit.client.1.smithi188.stdout:8/612: write d5/d21/d40/d43/d44/d5b/d99/fac [476247,119063] 0 2024-04-05T21:26:41.405 INFO:tasks.workunit.client.1.smithi188.stdout:2/702: write d5/f5c [399418,18339] 0 2024-04-05T21:26:41.405 INFO:tasks.workunit.client.0.smithi121.stdout:9/287: sync 2024-04-05T21:26:41.406 INFO:tasks.workunit.client.0.smithi121.stdout:2/376: sync 2024-04-05T21:26:41.408 INFO:tasks.workunit.client.0.smithi121.stdout:5/511: symlink dd/lac 0 2024-04-05T21:26:41.409 INFO:tasks.workunit.client.0.smithi121.stdout:0/464: truncate d7/d71/d29/d2a/d64/f9f 166919 0 2024-04-05T21:26:41.409 INFO:tasks.workunit.client.0.smithi121.stdout:9/288: mkdir d6/d36/d5c 0 2024-04-05T21:26:41.411 INFO:tasks.workunit.client.0.smithi121.stdout:1/444: truncate d8/d10/d14/f1e 6489369 0 2024-04-05T21:26:41.412 INFO:tasks.workunit.client.1.smithi188.stdout:8/613: rename d5/d21/d50 to d5/d6e/d86/dcb 0 2024-04-05T21:26:41.413 INFO:tasks.workunit.client.0.smithi121.stdout:2/377: creat d2/d9/d1c/f69 x:0 0 0 2024-04-05T21:26:41.413 INFO:tasks.workunit.client.0.smithi121.stdout:0/465: write d7/d62/d3c/f4b [869910,110998] 0 2024-04-05T21:26:41.416 INFO:tasks.workunit.client.0.smithi121.stdout:5/512: mknod dd/de/d19/d13/cad 0 2024-04-05T21:26:41.416 INFO:tasks.workunit.client.0.smithi121.stdout:4/501: dwrite d5/d11/d2c/f65 [0,4194304] 0 2024-04-05T21:26:41.416 INFO:tasks.workunit.client.0.smithi121.stdout:4/502: dread - d5/f81 zero size 2024-04-05T21:26:41.417 INFO:tasks.workunit.client.1.smithi188.stdout:2/703: dwrite d5/db/dd/d67/f8e [0,4194304] 0 2024-04-05T21:26:41.418 INFO:tasks.workunit.client.1.smithi188.stdout:2/704: readlink d5/lbc 0 2024-04-05T21:26:41.421 INFO:tasks.workunit.client.1.smithi188.stdout:8/614: creat d5/d21/d40/d43/d44/d5b/d99/d69/dbe/fcc x:0 0 0 2024-04-05T21:26:41.421 INFO:tasks.workunit.client.1.smithi188.stdout:9/655: truncate dd/d11/d1f/d26/dc0/f5c 511172 0 2024-04-05T21:26:41.422 INFO:tasks.workunit.client.1.smithi188.stdout:8/615: write d5/d21/d2a/d71/f88 [9267629,16157] 0 2024-04-05T21:26:41.423 INFO:tasks.workunit.client.0.smithi121.stdout:8/341: creat d12/f73 x:0 0 0 2024-04-05T21:26:41.423 INFO:tasks.workunit.client.0.smithi121.stdout:8/342: stat d12/d50/l6c 0 2024-04-05T21:26:41.423 INFO:tasks.workunit.client.0.smithi121.stdout:2/378: readlink d2/d9/d17/l2b 0 2024-04-05T21:26:41.425 INFO:tasks.workunit.client.1.smithi188.stdout:0/541: creat d4/fc4 x:0 0 0 2024-04-05T21:26:41.426 INFO:tasks.workunit.client.1.smithi188.stdout:2/705: creat d5/db/dd/d36/d3c/d48/d7a/d7c/fe5 x:0 0 0 2024-04-05T21:26:41.428 INFO:tasks.workunit.client.0.smithi121.stdout:8/343: write d12/d13/d15/f19 [2957584,28618] 0 2024-04-05T21:26:41.429 INFO:tasks.workunit.client.1.smithi188.stdout:9/656: creat dd/d3d/d95/fe0 x:0 0 0 2024-04-05T21:26:41.431 INFO:tasks.workunit.client.1.smithi188.stdout:0/542: mkdir d4/d8/db/d42/d71/dc5 0 2024-04-05T21:26:41.433 INFO:tasks.workunit.client.0.smithi121.stdout:4/503: dread d5/d11/d2c/d8d/d29/d2e/f7b [0,4194304] 0 2024-04-05T21:26:41.433 INFO:tasks.workunit.client.1.smithi188.stdout:1/637: rmdir d1/dc/d16/d2d 39 2024-04-05T21:26:41.434 INFO:tasks.workunit.client.1.smithi188.stdout:9/657: symlink dd/d11/d73/dbb/le1 0 2024-04-05T21:26:41.435 INFO:tasks.workunit.client.0.smithi121.stdout:4/504: chown d5/d11/f14 1769277 1 2024-04-05T21:26:41.436 INFO:tasks.workunit.client.0.smithi121.stdout:8/344: write d12/d13/f20 [1805471,101946] 0 2024-04-05T21:26:41.437 INFO:tasks.workunit.client.1.smithi188.stdout:2/706: truncate d5/db/dd/d67/dcd/d46/d66/fb3 668086 0 2024-04-05T21:26:41.438 INFO:tasks.workunit.client.0.smithi121.stdout:2/379: dwrite d2/f11 [0,4194304] 0 2024-04-05T21:26:41.439 INFO:tasks.workunit.client.0.smithi121.stdout:3/280: rename d0/d3/de/c28 to d0/c60 0 2024-04-05T21:26:41.439 INFO:tasks.workunit.client.1.smithi188.stdout:2/707: write d5/db/dd/d36/d3c/f4a [4417216,107080] 0 2024-04-05T21:26:41.440 INFO:tasks.workunit.client.1.smithi188.stdout:9/658: symlink dd/d3d/d95/le2 0 2024-04-05T21:26:41.440 INFO:tasks.workunit.client.0.smithi121.stdout:3/281: readlink d0/d3/d24/d3d/l1c 0 2024-04-05T21:26:41.441 INFO:tasks.workunit.client.0.smithi121.stdout:4/505: mkdir d5/d11/d20/da5 0 2024-04-05T21:26:41.441 INFO:tasks.workunit.client.1.smithi188.stdout:9/659: chown dd/d11/d1f/d26/dc0/d6b/f96 226 1 2024-04-05T21:26:41.442 INFO:tasks.workunit.client.0.smithi121.stdout:8/345: creat d12/d13/d56/d57/f74 x:0 0 0 2024-04-05T21:26:41.442 INFO:tasks.workunit.client.1.smithi188.stdout:0/543: mknod d4/d6/d13/d23/cc6 0 2024-04-05T21:26:41.443 INFO:tasks.workunit.client.0.smithi121.stdout:4/506: fsync d5/d11/d2c/d8d/f30 0 2024-04-05T21:26:41.444 INFO:tasks.workunit.client.0.smithi121.stdout:6/326: creat d1/d2/d3/d5/f66 x:0 0 0 2024-04-05T21:26:41.447 INFO:tasks.workunit.client.0.smithi121.stdout:3/282: symlink d0/d3/d24/d3d/d1a/d23/l61 0 2024-04-05T21:26:41.447 INFO:tasks.workunit.client.0.smithi121.stdout:6/327: chown d1/d2/d1b/l45 721 1 2024-04-05T21:26:41.451 INFO:tasks.workunit.client.1.smithi188.stdout:9/660: mkdir dd/d11/d22/d7e/de3 0 2024-04-05T21:26:41.451 INFO:tasks.workunit.client.0.smithi121.stdout:8/346: mknod d12/d13/c75 0 2024-04-05T21:26:41.452 INFO:tasks.workunit.client.0.smithi121.stdout:7/439: rename d0/c14 to d0/d9/d22/d28/c88 0 2024-04-05T21:26:41.453 INFO:tasks.workunit.client.1.smithi188.stdout:3/709: symlink d5/d7/d65/d61/lf1 0 2024-04-05T21:26:41.454 INFO:tasks.workunit.client.0.smithi121.stdout:3/283: mknod d0/d3/d24/d3d/d1a/d23/c62 0 2024-04-05T21:26:41.455 INFO:tasks.workunit.client.0.smithi121.stdout:6/328: creat d1/d2/d3/d4/d21/d57/d42/f67 x:0 0 0 2024-04-05T21:26:41.456 INFO:tasks.workunit.client.1.smithi188.stdout:9/661: creat dd/d3d/d95/fe4 x:0 0 0 2024-04-05T21:26:41.456 INFO:tasks.workunit.client.1.smithi188.stdout:9/662: stat dd/d11/d73/d8d/ldb 0 2024-04-05T21:26:41.458 INFO:tasks.workunit.client.1.smithi188.stdout:9/663: write dd/d11/d22/f4d [3575732,107024] 0 2024-04-05T21:26:41.459 INFO:tasks.workunit.client.0.smithi121.stdout:8/347: rename d12/d13/d56/d6d/d6f to d12/d50/d76 0 2024-04-05T21:26:41.460 INFO:tasks.workunit.client.0.smithi121.stdout:8/348: read - d12/f73 zero size 2024-04-05T21:26:41.461 INFO:tasks.workunit.client.1.smithi188.stdout:7/841: write d3/f68 [325911,103277] 0 2024-04-05T21:26:41.461 INFO:tasks.workunit.client.1.smithi188.stdout:9/664: fdatasync dd/d11/fb8 0 2024-04-05T21:26:41.463 INFO:tasks.workunit.client.0.smithi121.stdout:3/284: dread - d0/d3/d24/d3d/d1a/d1e/f2a zero size 2024-04-05T21:26:41.463 INFO:tasks.workunit.client.0.smithi121.stdout:6/329: mkdir d1/d4b/d68 0 2024-04-05T21:26:41.464 INFO:tasks.workunit.client.0.smithi121.stdout:8/349: symlink d12/d50/l77 0 2024-04-05T21:26:41.464 INFO:tasks.workunit.client.1.smithi188.stdout:7/842: mkdir d3/d113 0 2024-04-05T21:26:41.465 INFO:tasks.workunit.client.1.smithi188.stdout:9/665: symlink dd/d3d/d89/dd1/le5 0 2024-04-05T21:26:41.465 INFO:tasks.workunit.client.0.smithi121.stdout:6/330: dread - d1/d2/d1b/d35/d39/f65 zero size 2024-04-05T21:26:41.465 INFO:tasks.workunit.client.0.smithi121.stdout:2/380: dread d2/f1a [0,4194304] 0 2024-04-05T21:26:41.466 INFO:tasks.workunit.client.0.smithi121.stdout:3/285: mknod d0/d3/d24/d3d/d1a/d23/d38/d55/c63 0 2024-04-05T21:26:41.466 INFO:tasks.workunit.client.1.smithi188.stdout:7/843: readlink d3/d2c/dae/dfd/d2e/d8c/db7/dd6/le7 0 2024-04-05T21:26:41.469 INFO:tasks.workunit.client.0.smithi121.stdout:6/331: symlink d1/d2/d3/d59/l69 0 2024-04-05T21:26:41.470 INFO:tasks.workunit.client.0.smithi121.stdout:5/513: getdents dd/de/d19/d13 0 2024-04-05T21:26:41.471 INFO:tasks.workunit.client.1.smithi188.stdout:2/708: creat d5/db/fe6 x:0 0 0 2024-04-05T21:26:41.472 INFO:tasks.workunit.client.0.smithi121.stdout:2/381: mknod d2/d9/d17/d3a/c6a 0 2024-04-05T21:26:41.473 INFO:tasks.workunit.client.1.smithi188.stdout:9/666: rename dd/d3d/f61 to dd/d11/d1f/d26/d55/d98/da5/fe6 0 2024-04-05T21:26:41.473 INFO:tasks.workunit.client.0.smithi121.stdout:1/445: truncate d8/d10/d14/d1a/f66 2484543 0 2024-04-05T21:26:41.475 INFO:tasks.workunit.client.0.smithi121.stdout:8/350: link d12/d50/d6a/f60 d12/d13/d56/d57/f78 0 2024-04-05T21:26:41.476 INFO:tasks.workunit.client.1.smithi188.stdout:1/638: link d1/dc/d16/d21/fa0 d1/dc/d16/d46/d88/d13/fcd 0 2024-04-05T21:26:41.476 INFO:tasks.workunit.client.0.smithi121.stdout:0/466: write d7/d71/d29/d2a/d39/f40 [737496,39815] 0 2024-04-05T21:26:41.477 INFO:tasks.workunit.client.0.smithi121.stdout:0/467: stat d7/d62/f20 0 2024-04-05T21:26:41.477 INFO:tasks.workunit.client.0.smithi121.stdout:5/514: symlink dd/de/d19/d13/d53/lae 0 2024-04-05T21:26:41.478 INFO:tasks.workunit.client.1.smithi188.stdout:1/639: read d1/d7/f80 [628093,96241] 0 2024-04-05T21:26:41.479 INFO:tasks.workunit.client.0.smithi121.stdout:2/382: mknod d2/d9/d38/d2d/d47/c6b 0 2024-04-05T21:26:41.479 INFO:tasks.workunit.client.1.smithi188.stdout:9/667: symlink dd/d11/d1f/d26/dc0/d6b/le7 0 2024-04-05T21:26:41.480 INFO:tasks.workunit.client.1.smithi188.stdout:7/844: dread d3/d2c/d5e/fec [0,4194304] 0 2024-04-05T21:26:41.481 INFO:tasks.workunit.client.0.smithi121.stdout:7/440: dread d0/d32/f35 [4194304,4194304] 0 2024-04-05T21:26:41.481 INFO:tasks.workunit.client.0.smithi121.stdout:3/286: rename d0/d3/c16 to d0/d3/c64 0 2024-04-05T21:26:41.484 INFO:tasks.workunit.client.0.smithi121.stdout:1/446: mkdir d8/d8c 0 2024-04-05T21:26:41.485 INFO:tasks.workunit.client.0.smithi121.stdout:2/383: symlink d2/d28/l6c 0 2024-04-05T21:26:41.487 INFO:tasks.workunit.client.1.smithi188.stdout:9/668: dread dd/d11/d1f/d25/d8a/f77 [0,4194304] 0 2024-04-05T21:26:41.487 INFO:tasks.workunit.client.1.smithi188.stdout:3/710: write d5/d35/d4e/fa0 [261372,68142] 0 2024-04-05T21:26:41.490 INFO:tasks.workunit.client.0.smithi121.stdout:2/384: dread d2/d28/f4f [0,4194304] 0 2024-04-05T21:26:41.490 INFO:tasks.workunit.client.1.smithi188.stdout:7/845: dwrite d3/d2c/dae/dfd/d10/f33 [0,4194304] 0 2024-04-05T21:26:41.493 INFO:tasks.workunit.client.0.smithi121.stdout:4/507: truncate d5/d11/d2c/f65 958479 0 2024-04-05T21:26:41.496 INFO:tasks.workunit.client.0.smithi121.stdout:7/441: creat d0/d23/d5b/d52/d61/f89 x:0 0 0 2024-04-05T21:26:41.497 INFO:tasks.workunit.client.1.smithi188.stdout:7/846: truncate d3/d4/d1d/f34 3404604 0 2024-04-05T21:26:41.497 INFO:tasks.workunit.client.0.smithi121.stdout:7/442: write d0/d9/f12 [3729016,123888] 0 2024-04-05T21:26:41.499 INFO:tasks.workunit.client.0.smithi121.stdout:2/385: fdatasync d2/f1a 0 2024-04-05T21:26:41.501 INFO:tasks.workunit.client.0.smithi121.stdout:1/447: dread d8/f1d [0,4194304] 0 2024-04-05T21:26:41.501 INFO:tasks.workunit.client.1.smithi188.stdout:7/847: fsync d3/d2c/d5e/fec 0 2024-04-05T21:26:41.502 INFO:tasks.workunit.client.1.smithi188.stdout:1/640: dread d1/dc/f28 [0,4194304] 0 2024-04-05T21:26:41.503 INFO:tasks.workunit.client.1.smithi188.stdout:9/669: link dd/d11/d1f/d26/dc0/c74 dd/d11/d1f/d26/d55/d98/ce8 0 2024-04-05T21:26:41.504 INFO:tasks.workunit.client.1.smithi188.stdout:7/848: stat d3/d2c/dae/dfd/d2e/d8c/db7/fbc 0 2024-04-05T21:26:41.508 INFO:tasks.workunit.client.0.smithi121.stdout:2/386: unlink d2/d1e/c39 0 2024-04-05T21:26:41.508 INFO:tasks.workunit.client.1.smithi188.stdout:1/641: chown d1/dc/d16/d2d/f7e 332 1 2024-04-05T21:26:41.508 INFO:tasks.workunit.client.1.smithi188.stdout:9/670: creat dd/d11/d1f/d26/dc0/d6b/d40/d59/fe9 x:0 0 0 2024-04-05T21:26:41.509 INFO:tasks.workunit.client.1.smithi188.stdout:7/849: fdatasync d3/d2c/dae/dfd/d2e/f75 0 2024-04-05T21:26:41.511 INFO:tasks.workunit.client.0.smithi121.stdout:1/448: mkdir d8/d10/d14/d1a/d1b/d4c/d8d 0 2024-04-05T21:26:41.513 INFO:tasks.workunit.client.1.smithi188.stdout:9/671: mkdir dd/d11/d1f/d26/d55/d98/da5/dea 0 2024-04-05T21:26:41.514 INFO:tasks.workunit.client.0.smithi121.stdout:3/287: getdents d0/d3/d24/d3d/d1a/d23 0 2024-04-05T21:26:41.515 INFO:tasks.workunit.client.1.smithi188.stdout:7/850: creat d3/d2c/d5e/d91/f114 x:0 0 0 2024-04-05T21:26:41.515 INFO:tasks.workunit.client.1.smithi188.stdout:1/642: truncate d1/dc/d16/d21/d27/d49/f9b 892992 0 2024-04-05T21:26:41.517 INFO:tasks.workunit.client.0.smithi121.stdout:4/508: link d5/d11/d28/d3b/f59 d5/d11/d2c/d8d/fa6 0 2024-04-05T21:26:41.517 INFO:tasks.workunit.client.1.smithi188.stdout:7/851: read d3/f1e [2693857,98250] 0 2024-04-05T21:26:41.518 INFO:tasks.workunit.client.1.smithi188.stdout:0/544: dwrite d4/d8/db/f28 [0,4194304] 0 2024-04-05T21:26:41.519 INFO:tasks.workunit.client.0.smithi121.stdout:5/515: truncate dd/de/d19/d38/d45/f4e 651148 0 2024-04-05T21:26:41.519 INFO:tasks.workunit.client.0.smithi121.stdout:1/449: dwrite d8/d10/d14/d1a/d1b/d4c/d76/f7c [0,4194304] 0 2024-04-05T21:26:41.521 INFO:tasks.workunit.client.1.smithi188.stdout:7/852: write d3/d4/d1d/d74/d73/f83 [1738292,10711] 0 2024-04-05T21:26:41.523 INFO:tasks.workunit.client.1.smithi188.stdout:7/853: write d3/d2c/d5e/fe8 [850098,127620] 0 2024-04-05T21:26:41.527 INFO:tasks.workunit.client.1.smithi188.stdout:1/643: rename d1/dc/d16/d21/d27/fa1 to d1/dc/d16/d21/d27/d4a/dc1/fce 0 2024-04-05T21:26:41.528 INFO:tasks.workunit.client.0.smithi121.stdout:0/468: link d7/d71/d29/f34 d7/d71/d29/d3f/fa1 0 2024-04-05T21:26:41.528 INFO:tasks.workunit.client.0.smithi121.stdout:7/443: creat d0/d32/f8a x:0 0 0 2024-04-05T21:26:41.528 INFO:tasks.workunit.client.0.smithi121.stdout:2/387: mkdir d2/d9/d38/d6d 0 2024-04-05T21:26:41.532 INFO:tasks.workunit.client.0.smithi121.stdout:4/509: readlink d5/d89/la3 0 2024-04-05T21:26:41.532 INFO:tasks.workunit.client.0.smithi121.stdout:1/450: fdatasync d8/d10/d14/d1a/f46 0 2024-04-05T21:26:41.533 INFO:tasks.workunit.client.0.smithi121.stdout:5/516: mkdir dd/de/d39/d3b/daf 0 2024-04-05T21:26:41.533 INFO:tasks.workunit.client.0.smithi121.stdout:3/288: dread d0/d3/fa [0,4194304] 0 2024-04-05T21:26:41.534 INFO:tasks.workunit.client.1.smithi188.stdout:0/545: creat d4/d6/fc7 x:0 0 0 2024-04-05T21:26:41.535 INFO:tasks.workunit.client.0.smithi121.stdout:7/444: rmdir d0/d23/d5b/d52/d61 39 2024-04-05T21:26:41.535 INFO:tasks.workunit.client.0.smithi121.stdout:5/517: dread - dd/de/d19/d1c/f75 zero size 2024-04-05T21:26:41.535 INFO:tasks.workunit.client.0.smithi121.stdout:1/451: readlink d8/d10/d14/d1a/d1b/d4c/d76/d53/l5e 0 2024-04-05T21:26:41.535 INFO:tasks.workunit.client.0.smithi121.stdout:3/289: fdatasync d0/d3/d24/f42 0 2024-04-05T21:26:41.535 INFO:tasks.workunit.client.1.smithi188.stdout:1/644: creat d1/dc/d16/d46/d88/d3f/fcf x:0 0 0 2024-04-05T21:26:41.536 INFO:tasks.workunit.client.0.smithi121.stdout:0/469: symlink d7/dc/la2 0 2024-04-05T21:26:41.537 INFO:tasks.workunit.client.1.smithi188.stdout:2/709: dwrite d5/d5f/f90 [0,4194304] 0 2024-04-05T21:26:41.537 INFO:tasks.workunit.client.0.smithi121.stdout:0/470: dread - d7/d62/f73 zero size 2024-04-05T21:26:41.540 INFO:tasks.workunit.client.0.smithi121.stdout:7/445: creat d0/d23/f8b x:0 0 0 2024-04-05T21:26:41.540 INFO:tasks.workunit.client.1.smithi188.stdout:1/645: readlink d1/dc/d16/d46/l4d 0 2024-04-05T21:26:41.541 INFO:tasks.workunit.client.1.smithi188.stdout:3/711: write d5/d7/d6d/f7a [4902411,43995] 0 2024-04-05T21:26:41.543 INFO:tasks.workunit.client.1.smithi188.stdout:1/646: rmdir d1/dc/d16/d46 39 2024-04-05T21:26:41.545 INFO:tasks.workunit.client.0.smithi121.stdout:4/510: dread d5/d11/d28/d87/f96 [0,4194304] 0 2024-04-05T21:26:41.546 INFO:tasks.workunit.client.1.smithi188.stdout:3/712: unlink d5/d48/ccd 0 2024-04-05T21:26:41.548 INFO:tasks.workunit.client.0.smithi121.stdout:1/452: unlink d8/d10/d14/d1a/d1b/d38/f42 0 2024-04-05T21:26:41.549 INFO:tasks.workunit.client.0.smithi121.stdout:2/388: symlink d2/d9/d38/d6d/l6e 0 2024-04-05T21:26:41.550 INFO:tasks.workunit.client.0.smithi121.stdout:3/290: creat d0/d3/d24/d3d/d1a/d23/d38/f65 x:0 0 0 2024-04-05T21:26:41.550 INFO:tasks.workunit.client.0.smithi121.stdout:2/389: chown d2/d9/d38/l3c 16217367 1 2024-04-05T21:26:41.551 INFO:tasks.workunit.client.1.smithi188.stdout:9/672: write dd/d11/d1f/f29 [1176149,43192] 0 2024-04-05T21:26:41.551 INFO:tasks.workunit.client.0.smithi121.stdout:1/453: chown d8/d10/d14/d1a/d1b/d4c/d76/d53/d5b 16 1 2024-04-05T21:26:41.551 INFO:tasks.workunit.client.0.smithi121.stdout:2/390: fdatasync d2/d9/d12/f37 0 2024-04-05T21:26:41.552 INFO:tasks.workunit.client.1.smithi188.stdout:9/673: stat dd/d11/d1f/fdd 0 2024-04-05T21:26:41.552 INFO:tasks.workunit.client.1.smithi188.stdout:7/854: write d3/d2c/dae/dfd/d2e/d49/f67 [967868,60879] 0 2024-04-05T21:26:41.554 INFO:tasks.workunit.client.0.smithi121.stdout:7/446: unlink d0/l1c 0 2024-04-05T21:26:41.554 INFO:tasks.workunit.client.1.smithi188.stdout:0/546: write d4/d8/db/d42/d71/faf [164527,72397] 0 2024-04-05T21:26:41.556 INFO:tasks.workunit.client.0.smithi121.stdout:0/471: truncate d7/d71/d29/d2a/d2c/f7b 354351 0 2024-04-05T21:26:41.556 INFO:tasks.workunit.client.1.smithi188.stdout:3/713: dread d5/f37 [0,4194304] 0 2024-04-05T21:26:41.556 INFO:tasks.workunit.client.1.smithi188.stdout:9/674: symlink dd/d11/d1f/d26/d55/leb 0 2024-04-05T21:26:41.557 INFO:tasks.workunit.client.1.smithi188.stdout:0/547: write d4/d6/d13/d23/fa6 [1013294,119806] 0 2024-04-05T21:26:41.558 INFO:tasks.workunit.client.1.smithi188.stdout:2/710: truncate d5/db/dd/d36/d3c/f4a 3869105 0 2024-04-05T21:26:41.558 INFO:tasks.workunit.client.0.smithi121.stdout:4/511: rename d5/d11/f14 to d5/d89/fa7 0 2024-04-05T21:26:41.561 INFO:tasks.workunit.client.1.smithi188.stdout:3/714: rename d5/d7/d6d/le4 to d5/d7/d65/d61/d66/lf2 0 2024-04-05T21:26:41.562 INFO:tasks.workunit.client.1.smithi188.stdout:3/715: chown d5/df/c4b 100 1 2024-04-05T21:26:41.562 INFO:tasks.workunit.client.1.smithi188.stdout:0/548: creat d4/d6/d13/d23/d24/d57/fc8 x:0 0 0 2024-04-05T21:26:41.563 INFO:tasks.workunit.client.0.smithi121.stdout:0/472: mkdir d7/dc/da3 0 2024-04-05T21:26:41.563 INFO:tasks.workunit.client.1.smithi188.stdout:3/716: chown d5/d39/d84 116389 1 2024-04-05T21:26:41.564 INFO:tasks.workunit.client.0.smithi121.stdout:5/518: link dd/de/d19/d38/d56/f92 dd/de/d19/d38/d9b/fb0 0 2024-04-05T21:26:41.567 INFO:tasks.workunit.client.0.smithi121.stdout:2/391: rename d2/d9/f1f to d2/d9/d17/d3a/f6f 0 2024-04-05T21:26:41.567 INFO:tasks.workunit.client.1.smithi188.stdout:0/549: creat d4/d6/d13/dc1/fc9 x:0 0 0 2024-04-05T21:26:41.568 INFO:tasks.workunit.client.0.smithi121.stdout:1/454: creat d8/d8c/f8e x:0 0 0 2024-04-05T21:26:41.568 INFO:tasks.workunit.client.0.smithi121.stdout:5/519: fdatasync dd/de/d19/d1c/d36/f93 0 2024-04-05T21:26:41.569 INFO:tasks.workunit.client.1.smithi188.stdout:3/717: unlink d5/d7/d65/c47 0 2024-04-05T21:26:41.571 INFO:tasks.workunit.client.0.smithi121.stdout:7/447: dwrite d0/d9/d22/d28/d56/f58 [0,4194304] 0 2024-04-05T21:26:41.571 INFO:tasks.workunit.client.1.smithi188.stdout:3/718: dread - d5/d7/d6d/d75/d83/fab zero size 2024-04-05T21:26:41.571 INFO:tasks.workunit.client.1.smithi188.stdout:1/647: dread d1/dc/d16/d21/fa0 [0,4194304] 0 2024-04-05T21:26:41.571 INFO:tasks.workunit.client.1.smithi188.stdout:1/648: chown d1/dc 7801 1 2024-04-05T21:26:41.572 INFO:tasks.workunit.client.0.smithi121.stdout:0/473: mknod d7/d71/d94/ca4 0 2024-04-05T21:26:41.572 INFO:tasks.workunit.client.0.smithi121.stdout:3/291: mknod d0/c66 0 2024-04-05T21:26:41.573 INFO:tasks.workunit.client.0.smithi121.stdout:2/392: rmdir d2/d9/d56 39 2024-04-05T21:26:41.573 INFO:tasks.workunit.client.1.smithi188.stdout:3/719: unlink d5/d7/d65/f88 0 2024-04-05T21:26:41.574 INFO:tasks.workunit.client.0.smithi121.stdout:1/455: readlink d8/d10/d14/d1a/l82 0 2024-04-05T21:26:41.575 INFO:tasks.workunit.client.1.smithi188.stdout:3/720: symlink d5/d7/d31/lf3 0 2024-04-05T21:26:41.576 INFO:tasks.workunit.client.0.smithi121.stdout:4/512: symlink d5/d11/d2c/d8d/d29/la8 0 2024-04-05T21:26:41.576 INFO:tasks.workunit.client.1.smithi188.stdout:3/721: chown d5/d39/d84 7352 1 2024-04-05T21:26:41.576 INFO:tasks.workunit.client.0.smithi121.stdout:7/448: fsync d0/d32/f41 0 2024-04-05T21:26:41.577 INFO:tasks.workunit.client.1.smithi188.stdout:9/675: dread dd/f16 [0,4194304] 0 2024-04-05T21:26:41.577 INFO:tasks.workunit.client.0.smithi121.stdout:0/474: creat d7/d71/d29/d2a/d64/fa5 x:0 0 0 2024-04-05T21:26:41.577 INFO:tasks.workunit.client.0.smithi121.stdout:7/449: fsync d0/d32/d4a/f53 0 2024-04-05T21:26:41.580 INFO:tasks.workunit.client.1.smithi188.stdout:9/676: chown dd/d35/cbd 0 1 2024-04-05T21:26:41.581 INFO:tasks.workunit.client.0.smithi121.stdout:4/513: fsync d5/d11/d2c/d9d/f31 0 2024-04-05T21:26:41.582 INFO:tasks.workunit.client.0.smithi121.stdout:4/514: dread - d5/d11/d2c/d8d/d29/d58/f6e zero size 2024-04-05T21:26:41.585 INFO:tasks.workunit.client.1.smithi188.stdout:9/677: mknod dd/d11/d1f/d26/d55/d98/da5/cec 0 2024-04-05T21:26:41.586 INFO:tasks.workunit.client.1.smithi188.stdout:3/722: dwrite d5/d7/d65/d5f/fb4 [0,4194304] 0 2024-04-05T21:26:41.589 INFO:tasks.workunit.client.0.smithi121.stdout:4/515: mknod d5/d45/d67/ca9 0 2024-04-05T21:26:41.589 INFO:tasks.workunit.client.1.smithi188.stdout:9/678: creat dd/d3d/d89/fed x:0 0 0 2024-04-05T21:26:41.589 INFO:tasks.workunit.client.0.smithi121.stdout:4/516: chown d5/d45 21329 1 2024-04-05T21:26:41.590 INFO:tasks.workunit.client.0.smithi121.stdout:4/517: stat d5/d11/l91 0 2024-04-05T21:26:41.591 INFO:tasks.workunit.client.1.smithi188.stdout:3/723: rmdir d5/d7/d65/d61 39 2024-04-05T21:26:41.593 INFO:tasks.workunit.client.1.smithi188.stdout:9/679: creat dd/d11/d73/d8d/fee x:0 0 0 2024-04-05T21:26:41.593 INFO:tasks.workunit.client.1.smithi188.stdout:3/724: chown d5/d7/d6d/dd8/f3d 0 1 2024-04-05T21:26:41.595 INFO:tasks.workunit.client.0.smithi121.stdout:0/475: mkdir d7/d62/d9d/da6 0 2024-04-05T21:26:41.595 INFO:tasks.workunit.client.0.smithi121.stdout:0/476: fdatasync d7/d62/f11 0 2024-04-05T21:26:41.596 INFO:tasks.workunit.client.1.smithi188.stdout:1/649: dwrite d1/dc/d16/d21/d27/d4a/f7f [0,4194304] 0 2024-04-05T21:26:41.598 INFO:tasks.workunit.client.0.smithi121.stdout:1/456: creat d8/d10/d14/f8f x:0 0 0 2024-04-05T21:26:41.601 INFO:tasks.workunit.client.0.smithi121.stdout:4/518: dwrite d5/d11/d2c/d8d/d29/d58/f6e [0,4194304] 0 2024-04-05T21:26:41.601 INFO:tasks.workunit.client.0.smithi121.stdout:7/450: dread d0/d23/f33 [0,4194304] 0 2024-04-05T21:26:41.603 INFO:tasks.workunit.client.0.smithi121.stdout:3/292: truncate d0/f2d 606962 0 2024-04-05T21:26:41.603 INFO:tasks.workunit.client.0.smithi121.stdout:5/520: truncate dd/de/d19/d13/d53/f7e 186216 0 2024-04-05T21:26:41.604 INFO:tasks.workunit.client.0.smithi121.stdout:1/457: symlink d8/d10/d14/d1a/d1b/d24/l90 0 2024-04-05T21:26:41.605 INFO:tasks.workunit.client.0.smithi121.stdout:7/451: rmdir d0/d57 39 2024-04-05T21:26:41.608 INFO:tasks.workunit.client.0.smithi121.stdout:1/458: chown d8/d10/d14/d1a/l6e 5932000 1 2024-04-05T21:26:41.609 INFO:tasks.workunit.client.0.smithi121.stdout:3/293: unlink d0/d3/d24/d3d/d1a/f5f 0 2024-04-05T21:26:41.610 INFO:tasks.workunit.client.0.smithi121.stdout:5/521: mkdir dd/de/d19/d1c/d7b/d88/d9c/db1 0 2024-04-05T21:26:41.611 INFO:tasks.workunit.client.0.smithi121.stdout:1/459: chown d8/d10/d14/d27/d2d/d84 426 1 2024-04-05T21:26:41.612 INFO:tasks.workunit.client.0.smithi121.stdout:3/294: dread - d0/d3/f5b zero size 2024-04-05T21:26:41.614 INFO:tasks.workunit.client.0.smithi121.stdout:7/452: getdents d0/d23 0 2024-04-05T21:26:41.615 INFO:tasks.workunit.client.0.smithi121.stdout:4/519: dwrite d5/f3f [4194304,4194304] 0 2024-04-05T21:26:41.616 INFO:tasks.workunit.client.0.smithi121.stdout:0/477: dread d7/d62/d3c/f4b [0,4194304] 0 2024-04-05T21:26:41.617 INFO:tasks.workunit.client.0.smithi121.stdout:1/460: dread - d8/d10/d14/d1a/d1b/d4c/f4f zero size 2024-04-05T21:26:41.617 INFO:tasks.workunit.client.0.smithi121.stdout:5/522: mknod dd/de/d19/d38/da3/cb2 0 2024-04-05T21:26:41.617 INFO:tasks.workunit.client.1.smithi188.stdout:1/650: stat d1/dc/d16/d21/l94 0 2024-04-05T21:26:41.618 INFO:tasks.workunit.client.0.smithi121.stdout:4/520: chown d5/d11/d28/d3b/f77 28062 1 2024-04-05T21:26:41.619 INFO:tasks.workunit.client.0.smithi121.stdout:7/453: creat d0/d23/f8c x:0 0 0 2024-04-05T21:26:41.620 INFO:tasks.workunit.client.0.smithi121.stdout:0/478: symlink d7/d71/d45/d5a/la7 0 2024-04-05T21:26:41.621 INFO:tasks.workunit.client.0.smithi121.stdout:1/461: mknod d8/d10/d14/d27/d2d/c91 0 2024-04-05T21:26:41.626 INFO:tasks.workunit.client.0.smithi121.stdout:4/521: write d5/d11/d28/d3b/f77 [4846397,29735] 0 2024-04-05T21:26:41.626 INFO:tasks.workunit.client.0.smithi121.stdout:7/454: dread - d0/d23/f74 zero size 2024-04-05T21:26:41.629 INFO:tasks.workunit.client.0.smithi121.stdout:7/455: chown d0/d9/d22/d28/f48 7 1 2024-04-05T21:26:41.631 INFO:tasks.workunit.client.0.smithi121.stdout:0/479: creat d7/d62/fa8 x:0 0 0 2024-04-05T21:26:41.632 INFO:tasks.workunit.client.0.smithi121.stdout:3/295: dwrite d0/d3/d24/d3d/d1a/d23/d38/f65 [0,4194304] 0 2024-04-05T21:26:41.633 INFO:tasks.workunit.client.0.smithi121.stdout:0/480: dread - d7/d71/d29/d2a/d2c/f8d zero size 2024-04-05T21:26:41.634 INFO:tasks.workunit.client.0.smithi121.stdout:1/462: link d8/d10/d14/f37 d8/d10/d14/d27/d2d/f92 0 2024-04-05T21:26:41.635 INFO:tasks.workunit.client.0.smithi121.stdout:7/456: chown d0/d9/d22/d28/c4c 0 1 2024-04-05T21:26:41.642 INFO:tasks.workunit.client.0.smithi121.stdout:0/481: creat d7/d71/d45/d5a/fa9 x:0 0 0 2024-04-05T21:26:41.644 INFO:tasks.workunit.client.0.smithi121.stdout:3/296: symlink d0/d3/de/l67 0 2024-04-05T21:26:41.645 INFO:tasks.workunit.client.0.smithi121.stdout:4/522: dwrite d5/d11/d28/d87/fa1 [0,4194304] 0 2024-04-05T21:26:41.645 INFO:tasks.workunit.client.0.smithi121.stdout:1/463: truncate d8/d10/d14/d27/f75 130234 0 2024-04-05T21:26:41.647 INFO:tasks.workunit.client.0.smithi121.stdout:7/457: dwrite d0/f42 [0,4194304] 0 2024-04-05T21:26:41.648 INFO:tasks.workunit.client.0.smithi121.stdout:3/297: read - d0/d3/d24/d3d/f3a zero size 2024-04-05T21:26:41.649 INFO:tasks.workunit.client.0.smithi121.stdout:4/523: dread - d5/d11/d2c/d8d/d29/d2e/f4d zero size 2024-04-05T21:26:41.649 INFO:tasks.workunit.client.0.smithi121.stdout:3/298: fsync d0/d3/f8 0 2024-04-05T21:26:41.650 INFO:tasks.workunit.client.0.smithi121.stdout:5/523: write dd/de/d19/d38/d45/f5c [1321949,76505] 0 2024-04-05T21:26:41.653 INFO:tasks.workunit.client.0.smithi121.stdout:1/464: dread - d8/d10/d14/d1a/d1b/f65 zero size 2024-04-05T21:26:41.656 INFO:tasks.workunit.client.0.smithi121.stdout:7/458: rename d0/d9/d22/d28/c4c to d0/d23/d5b/d78/c8d 0 2024-04-05T21:26:41.660 INFO:tasks.workunit.client.0.smithi121.stdout:5/524: creat dd/de/d19/d1c/d7b/d88/fb3 x:0 0 0 2024-04-05T21:26:41.661 INFO:tasks.workunit.client.0.smithi121.stdout:3/299: mknod d0/d3/d24/d3d/d1a/d23/d38/c68 0 2024-04-05T21:26:41.663 INFO:tasks.workunit.client.0.smithi121.stdout:1/465: creat d8/d10/d14/d1a/d1b/d44/f93 x:0 0 0 2024-04-05T21:26:41.666 INFO:tasks.workunit.client.0.smithi121.stdout:7/459: symlink d0/d9/d22/d28/d56/l8e 0 2024-04-05T21:26:41.670 INFO:tasks.workunit.client.0.smithi121.stdout:5/525: symlink dd/de/lb4 0 2024-04-05T21:26:41.671 INFO:tasks.workunit.client.0.smithi121.stdout:1/466: mknod d8/d10/d14/d1a/d1b/d24/c94 0 2024-04-05T21:26:41.671 INFO:tasks.workunit.client.0.smithi121.stdout:4/524: dread d5/d11/d28/d3b/f59 [0,4194304] 0 2024-04-05T21:26:41.673 INFO:tasks.workunit.client.0.smithi121.stdout:4/525: chown d5/d11/d2c/d8c/c8e 2 1 2024-04-05T21:26:41.675 INFO:tasks.workunit.client.0.smithi121.stdout:3/300: getdents d0/d3/d24/d3f 0 2024-04-05T21:26:41.676 INFO:tasks.workunit.client.0.smithi121.stdout:0/482: dread d7/d1c/f25 [0,4194304] 0 2024-04-05T21:26:41.676 INFO:tasks.workunit.client.0.smithi121.stdout:1/467: mknod d8/d10/d14/d1a/d1b/d85/c95 0 2024-04-05T21:26:41.679 INFO:tasks.workunit.client.0.smithi121.stdout:1/468: symlink d8/d8c/l96 0 2024-04-05T21:26:41.680 INFO:tasks.workunit.client.0.smithi121.stdout:3/301: write d0/d3/d24/d3d/d1a/d1e/f2a [321695,120974] 0 2024-04-05T21:26:41.680 INFO:tasks.workunit.client.0.smithi121.stdout:4/526: creat d5/d11/d2c/d8d/faa x:0 0 0 2024-04-05T21:26:41.681 INFO:tasks.workunit.client.0.smithi121.stdout:5/526: dwrite dd/de/d19/d13/f49 [0,4194304] 0 2024-04-05T21:26:41.683 INFO:tasks.workunit.client.0.smithi121.stdout:4/527: read - d5/d11/d28/d3b/f93 zero size 2024-04-05T21:26:41.684 INFO:tasks.workunit.client.0.smithi121.stdout:0/483: creat d7/d71/d94/da0/faa x:0 0 0 2024-04-05T21:26:41.685 INFO:tasks.workunit.client.0.smithi121.stdout:1/469: mknod d8/d10/d14/d27/c97 0 2024-04-05T21:26:41.687 INFO:tasks.workunit.client.0.smithi121.stdout:4/528: mkdir d5/d45/d67/dab 0 2024-04-05T21:26:41.688 INFO:tasks.workunit.client.0.smithi121.stdout:5/527: symlink dd/de/d19/d3c/d8e/lb5 0 2024-04-05T21:26:41.691 INFO:tasks.workunit.client.0.smithi121.stdout:5/528: dread - dd/de/d19/d1c/d7b/d88/fb3 zero size 2024-04-05T21:26:41.694 INFO:tasks.workunit.client.0.smithi121.stdout:1/470: unlink d8/d10/f26 0 2024-04-05T21:26:41.696 INFO:tasks.workunit.client.1.smithi188.stdout:9/680: creat dd/d11/d22/d7e/de3/fef x:0 0 0 2024-04-05T21:26:41.697 INFO:tasks.workunit.client.1.smithi188.stdout:3/725: symlink d5/d35/lf4 0 2024-04-05T21:26:41.698 INFO:tasks.workunit.client.0.smithi121.stdout:0/484: rename d7/d71/d29/f34 to d7/dc/fab 0 2024-04-05T21:26:41.698 INFO:tasks.workunit.client.1.smithi188.stdout:1/651: write d1/f48 [322993,106891] 0 2024-04-05T21:26:41.700 INFO:tasks.workunit.client.0.smithi121.stdout:4/529: mknod d5/d45/d8b/cac 0 2024-04-05T21:26:41.702 INFO:tasks.workunit.client.0.smithi121.stdout:0/485: rename d7/d1c/f25 to d7/d62/d8b/fac 0 2024-04-05T21:26:41.703 INFO:tasks.workunit.client.0.smithi121.stdout:4/530: read d5/d11/d2c/d8d/d29/d58/f6e [2866033,93011] 0 2024-04-05T21:26:41.704 INFO:tasks.workunit.client.0.smithi121.stdout:1/471: creat d8/d89/f98 x:0 0 0 2024-04-05T21:26:41.706 INFO:tasks.workunit.client.0.smithi121.stdout:3/302: dwrite d0/f15 [4194304,4194304] 0 2024-04-05T21:26:41.708 INFO:tasks.workunit.client.0.smithi121.stdout:0/486: write d7/d71/d29/d2a/f4c [58045,84414] 0 2024-04-05T21:26:41.710 INFO:tasks.workunit.client.0.smithi121.stdout:1/472: fsync f2 0 2024-04-05T21:26:41.711 INFO:tasks.workunit.client.0.smithi121.stdout:1/473: chown d8/d10/d14/d27/d2d/c6f 21908814 1 2024-04-05T21:26:41.713 INFO:tasks.workunit.client.0.smithi121.stdout:1/474: readlink d8/d10/d14/d1a/d1b/d24/l90 0 2024-04-05T21:26:41.714 INFO:tasks.workunit.client.0.smithi121.stdout:1/475: write d8/d10/d14/d1a/f20 [488343,125674] 0 2024-04-05T21:26:41.715 INFO:tasks.workunit.client.1.smithi188.stdout:1/652: fdatasync d1/dc/d16/d21/f68 0 2024-04-05T21:26:41.716 INFO:tasks.workunit.client.0.smithi121.stdout:0/487: mknod d7/d7e/cad 0 2024-04-05T21:26:41.717 INFO:tasks.workunit.client.1.smithi188.stdout:9/681: rename dd/d11/d1f/db2/ld0 to dd/d11/d1f/d26/d55/dcd/lf0 0 2024-04-05T21:26:41.717 INFO:tasks.workunit.client.1.smithi188.stdout:3/726: link d5/df/l74 d5/d7/d6d/d75/d83/lf5 0 2024-04-05T21:26:41.718 INFO:tasks.workunit.client.0.smithi121.stdout:1/476: truncate d8/d8c/f8e 397894 0 2024-04-05T21:26:41.720 INFO:tasks.workunit.client.1.smithi188.stdout:9/682: mknod dd/d11/d1f/d25/d8a/cf1 0 2024-04-05T21:26:41.720 INFO:tasks.workunit.client.0.smithi121.stdout:3/303: rename d0/d3/f8 to d0/d3/f69 0 2024-04-05T21:26:41.721 INFO:tasks.workunit.client.0.smithi121.stdout:0/488: stat d7/d62/d8b/fac 0 2024-04-05T21:26:41.723 INFO:tasks.workunit.client.0.smithi121.stdout:4/531: getdents d5/d11/d2c/d8d 0 2024-04-05T21:26:41.723 INFO:tasks.workunit.client.0.smithi121.stdout:5/529: dread dd/de/d19/d1c/d36/f3a [0,4194304] 0 2024-04-05T21:26:41.725 INFO:tasks.workunit.client.0.smithi121.stdout:1/477: unlink d8/d10/d14/d27/d2d/f92 0 2024-04-05T21:26:41.725 INFO:tasks.workunit.client.0.smithi121.stdout:4/532: read - d5/d11/d2c/d8d/faa zero size 2024-04-05T21:26:41.725 INFO:tasks.workunit.client.1.smithi188.stdout:3/727: mkdir d5/d7/d65/d61/d66/da4/de3/df6 0 2024-04-05T21:26:41.726 INFO:tasks.workunit.client.0.smithi121.stdout:3/304: unlink d0/f14 0 2024-04-05T21:26:41.727 INFO:tasks.workunit.client.1.smithi188.stdout:1/653: creat d1/dc/d16/d46/d88/d13/d60/fd0 x:0 0 0 2024-04-05T21:26:41.727 INFO:tasks.workunit.client.1.smithi188.stdout:9/683: dread dd/d11/d73/f75 [0,4194304] 0 2024-04-05T21:26:41.727 INFO:tasks.workunit.client.0.smithi121.stdout:0/489: dread d7/d71/d29/d2a/d39/f40 [0,4194304] 0 2024-04-05T21:26:41.728 INFO:tasks.workunit.client.0.smithi121.stdout:5/530: mknod dd/de/d19/d3c/cb6 0 2024-04-05T21:26:41.729 INFO:tasks.workunit.client.0.smithi121.stdout:1/478: creat d8/d10/d14/d27/d57/f99 x:0 0 0 2024-04-05T21:26:41.729 INFO:tasks.workunit.client.1.smithi188.stdout:1/654: chown d1/dc/d16/d46/d88/d13/f6a 15 1 2024-04-05T21:26:41.729 INFO:tasks.workunit.client.1.smithi188.stdout:3/728: dread - d5/d35/d4e/fc9 zero size 2024-04-05T21:26:41.730 INFO:tasks.workunit.client.1.smithi188.stdout:9/684: mknod dd/d35/cf2 0 2024-04-05T21:26:41.731 INFO:tasks.workunit.client.0.smithi121.stdout:4/533: link d5/d11/d28/d3b/f77 d5/d11/d20/fad 0 2024-04-05T21:26:41.734 INFO:tasks.workunit.client.0.smithi121.stdout:4/534: fsync d5/d11/d2c/d9d/f75 0 2024-04-05T21:26:41.734 INFO:tasks.workunit.client.0.smithi121.stdout:1/479: dread d8/d10/d14/d1a/d1b/d38/d4b/f6b [0,4194304] 0 2024-04-05T21:26:41.734 INFO:tasks.workunit.client.0.smithi121.stdout:0/490: getdents d7/d62/d9d/da6 0 2024-04-05T21:26:41.735 INFO:tasks.workunit.client.1.smithi188.stdout:1/655: rename d1/dc/f7a to d1/dc/d16/d2d/fd1 0 2024-04-05T21:26:41.735 INFO:tasks.workunit.client.1.smithi188.stdout:3/729: fsync d5/d35/f52 0 2024-04-05T21:26:41.735 INFO:tasks.workunit.client.0.smithi121.stdout:3/305: creat d0/d3/de/f6a x:0 0 0 2024-04-05T21:26:41.735 INFO:tasks.workunit.client.1.smithi188.stdout:3/730: dread - d5/d7/d65/fbe zero size 2024-04-05T21:26:41.736 INFO:tasks.workunit.client.1.smithi188.stdout:1/656: mknod d1/d7/dbf/cd2 0 2024-04-05T21:26:41.737 INFO:tasks.workunit.client.0.smithi121.stdout:1/480: read d8/d10/d14/d1a/f46 [958022,128619] 0 2024-04-05T21:26:41.739 INFO:tasks.workunit.client.1.smithi188.stdout:3/731: rename d5/d7/d6d/dd8/cd1 to d5/d7/d65/d61/db5/cf7 0 2024-04-05T21:26:41.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:41 smithi121 ceph-mon[39723]: pgmap v19: 65 pgs: 65 active+clean; 2.1 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 71 MiB/s rd, 291 MiB/s wr, 504 op/s 2024-04-05T21:26:41.740 INFO:tasks.workunit.client.0.smithi121.stdout:1/481: write d8/d10/d14/d1a/d1b/d24/f8b [1013447,52472] 0 2024-04-05T21:26:41.741 INFO:tasks.workunit.client.0.smithi121.stdout:3/306: creat d0/d3/d24/d3d/d1a/d23/f6b x:0 0 0 2024-04-05T21:26:41.742 INFO:tasks.workunit.client.1.smithi188.stdout:1/657: dwrite d1/dc/f56 [0,4194304] 0 2024-04-05T21:26:41.743 INFO:tasks.workunit.client.0.smithi121.stdout:3/307: write d0/d3/d24/d3d/d1a/d1e/f2a [1485466,29670] 0 2024-04-05T21:26:41.744 INFO:tasks.workunit.client.1.smithi188.stdout:3/732: creat d5/d35/ff8 x:0 0 0 2024-04-05T21:26:41.745 INFO:tasks.workunit.client.0.smithi121.stdout:4/535: rename d5/d11/d2c/d8d to d5/d11/d2c/d9d/dae 0 2024-04-05T21:26:41.745 INFO:tasks.workunit.client.0.smithi121.stdout:5/531: write dd/de/d19/f80 [2159095,75003] 0 2024-04-05T21:26:41.746 INFO:tasks.workunit.client.0.smithi121.stdout:1/482: rmdir d8/d10/d14/d27/d5c 39 2024-04-05T21:26:41.747 INFO:tasks.workunit.client.0.smithi121.stdout:5/532: chown dd/de/d19/d1c/d36/laa 609467 1 2024-04-05T21:26:41.747 INFO:tasks.workunit.client.1.smithi188.stdout:1/658: creat d1/d7/fd3 x:0 0 0 2024-04-05T21:26:41.748 INFO:tasks.workunit.client.0.smithi121.stdout:1/483: write d8/d10/d14/d1a/d1b/d4c/d76/f3d [488502,3147] 0 2024-04-05T21:26:41.748 INFO:tasks.workunit.client.0.smithi121.stdout:3/308: mknod d0/d3/d24/d3d/d1a/d23/d38/c6c 0 2024-04-05T21:26:41.751 INFO:tasks.workunit.client.1.smithi188.stdout:3/733: mknod d5/d7/d65/d61/d66/da4/cf9 0 2024-04-05T21:26:41.753 INFO:tasks.workunit.client.0.smithi121.stdout:1/484: chown d8/d10/d14/d1a/f46 347641 1 2024-04-05T21:26:41.753 INFO:tasks.workunit.client.1.smithi188.stdout:3/734: chown d5/d7/d65/d5f/fb4 801 1 2024-04-05T21:26:41.754 INFO:tasks.workunit.client.0.smithi121.stdout:3/309: chown d0/d3/de/l12 52 1 2024-04-05T21:26:41.756 INFO:tasks.workunit.client.1.smithi188.stdout:1/659: rename d1/dc/d16/d46/d95/d99/fb3 to d1/dc/d16/d46/d88/d13/d60/fd4 0 2024-04-05T21:26:41.756 INFO:tasks.workunit.client.1.smithi188.stdout:9/685: dwrite dd/d11/f76 [0,4194304] 0 2024-04-05T21:26:41.757 INFO:tasks.workunit.client.1.smithi188.stdout:9/686: fdatasync dd/f42 0 2024-04-05T21:26:41.760 INFO:tasks.workunit.client.1.smithi188.stdout:9/687: fdatasync dd/d11/d1f/d26/dc0/d6b/f96 0 2024-04-05T21:26:41.760 INFO:tasks.workunit.client.0.smithi121.stdout:3/310: mkdir d0/d3/d24/d6d 0 2024-04-05T21:26:41.761 INFO:tasks.workunit.client.1.smithi188.stdout:9/688: dread - dd/d3d/f70 zero size 2024-04-05T21:26:41.762 INFO:tasks.workunit.client.0.smithi121.stdout:5/533: dwrite dd/de/d19/d1c/f75 [0,4194304] 0 2024-04-05T21:26:41.762 INFO:tasks.workunit.client.1.smithi188.stdout:1/660: rename d1/dc/d16/d21/d27/d90 to d1/dc/d16/d2d/dd5 0 2024-04-05T21:26:41.762 INFO:tasks.workunit.client.0.smithi121.stdout:0/491: dwrite d7/d71/d45/d5a/f69 [0,4194304] 0 2024-04-05T21:26:41.763 INFO:tasks.workunit.client.1.smithi188.stdout:9/689: creat dd/d11/d1f/d26/ff3 x:0 0 0 2024-04-05T21:26:41.763 INFO:tasks.workunit.client.0.smithi121.stdout:1/485: creat d8/d10/d14/d1a/d1b/d24/d79/f9a x:0 0 0 2024-04-05T21:26:41.765 INFO:tasks.workunit.client.0.smithi121.stdout:1/486: chown d8/d10/d14/d1a/d1b/f65 1593 1 2024-04-05T21:26:41.768 INFO:tasks.workunit.client.0.smithi121.stdout:3/311: symlink d0/d3/d24/d3d/d1a/d1e/l6e 0 2024-04-05T21:26:41.770 INFO:tasks.workunit.client.0.smithi121.stdout:3/312: fsync d0/d3/de/f17 0 2024-04-05T21:26:41.770 INFO:tasks.workunit.client.0.smithi121.stdout:5/534: truncate dd/de/d19/d38/d45/f4e 1162208 0 2024-04-05T21:26:41.772 INFO:tasks.workunit.client.0.smithi121.stdout:3/313: fsync d0/d3/f4 0 2024-04-05T21:26:41.773 INFO:tasks.workunit.client.0.smithi121.stdout:5/535: dread - dd/de/d39/d3b/f94 zero size 2024-04-05T21:26:41.773 INFO:tasks.workunit.client.0.smithi121.stdout:4/536: dwrite d5/d11/d2c/f65 [0,4194304] 0 2024-04-05T21:26:41.774 INFO:tasks.workunit.client.0.smithi121.stdout:3/314: fdatasync d0/f15 0 2024-04-05T21:26:41.776 INFO:tasks.workunit.client.0.smithi121.stdout:3/315: chown d0/d3/d24/d3d/l50 12808 1 2024-04-05T21:26:41.777 INFO:tasks.workunit.client.1.smithi188.stdout:8/616: sync 2024-04-05T21:26:41.777 INFO:tasks.workunit.client.1.smithi188.stdout:5/597: sync 2024-04-05T21:26:41.777 INFO:tasks.workunit.client.1.smithi188.stdout:4/882: sync 2024-04-05T21:26:41.777 INFO:tasks.workunit.client.1.smithi188.stdout:6/686: sync 2024-04-05T21:26:41.778 INFO:tasks.workunit.client.0.smithi121.stdout:5/536: creat dd/de/d19/d3c/d8e/fb7 x:0 0 0 2024-04-05T21:26:41.779 INFO:tasks.workunit.client.0.smithi121.stdout:4/537: dread d5/d11/f17 [0,4194304] 0 2024-04-05T21:26:41.780 INFO:tasks.workunit.client.0.smithi121.stdout:5/537: fsync dd/de/d19/d38/f4d 0 2024-04-05T21:26:41.781 INFO:tasks.workunit.client.0.smithi121.stdout:4/538: readlink d5/l19 0 2024-04-05T21:26:41.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:41 smithi188 ceph-mon[49715]: pgmap v19: 65 pgs: 65 active+clean; 2.1 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 71 MiB/s rd, 291 MiB/s wr, 504 op/s 2024-04-05T21:26:41.782 INFO:tasks.workunit.client.1.smithi188.stdout:8/617: chown d5/d21/d2a/d95/dc0 51746 1 2024-04-05T21:26:41.782 INFO:tasks.workunit.client.0.smithi121.stdout:5/538: read dd/f17 [658856,8406] 0 2024-04-05T21:26:41.783 INFO:tasks.workunit.client.1.smithi188.stdout:5/598: write d6/d15/d5c/d7a/fd3 [932069,129162] 0 2024-04-05T21:26:41.785 INFO:tasks.workunit.client.1.smithi188.stdout:9/690: write f1 [1103274,66989] 0 2024-04-05T21:26:41.788 INFO:tasks.workunit.client.1.smithi188.stdout:4/883: mknod d0/d2/d2a/c12b 0 2024-04-05T21:26:41.788 INFO:tasks.workunit.client.0.smithi121.stdout:0/492: dread d7/dc/fab [0,4194304] 0 2024-04-05T21:26:41.788 INFO:tasks.workunit.client.0.smithi121.stdout:5/539: symlink dd/de/d19/d1c/d36/lb8 0 2024-04-05T21:26:41.790 INFO:tasks.workunit.client.1.smithi188.stdout:5/599: rmdir d6 39 2024-04-05T21:26:41.791 INFO:tasks.workunit.client.0.smithi121.stdout:4/539: dwrite d5/d11/d2c/d9d/f31 [0,4194304] 0 2024-04-05T21:26:41.792 INFO:tasks.workunit.client.1.smithi188.stdout:3/735: truncate d5/d7/d6d/dd8/d2f/f4c 2918023 0 2024-04-05T21:26:41.793 INFO:tasks.workunit.client.1.smithi188.stdout:9/691: creat dd/d11/d1f/d26/dc0/d6b/ff4 x:0 0 0 2024-04-05T21:26:41.794 INFO:tasks.workunit.client.0.smithi121.stdout:5/540: mkdir dd/de/d19/d1c/d7b/d88/d9c/db1/db9 0 2024-04-05T21:26:41.795 INFO:tasks.workunit.client.1.smithi188.stdout:8/618: dwrite d5/f1c [0,4194304] 0 2024-04-05T21:26:41.796 INFO:tasks.workunit.client.0.smithi121.stdout:5/541: mkdir dd/de/d19/d13/d53/dba 0 2024-04-05T21:26:41.797 INFO:tasks.workunit.client.1.smithi188.stdout:3/736: rmdir d5/ddd 39 2024-04-05T21:26:41.797 INFO:tasks.workunit.client.0.smithi121.stdout:5/542: write dd/de/d19/d38/d45/d68/f79 [1103854,53462] 0 2024-04-05T21:26:41.797 INFO:tasks.workunit.client.1.smithi188.stdout:9/692: creat dd/d11/d1f/db2/ff5 x:0 0 0 2024-04-05T21:26:41.798 INFO:tasks.workunit.client.0.smithi121.stdout:5/543: fdatasync dd/de/d39/d3b/d50/f5b 0 2024-04-05T21:26:41.799 INFO:tasks.workunit.client.0.smithi121.stdout:4/540: dread d5/d11/d2c/d9d/dae/f4c [0,4194304] 0 2024-04-05T21:26:41.802 INFO:tasks.workunit.client.1.smithi188.stdout:8/619: chown d5/d21/d2a/d95 750494521 1 2024-04-05T21:26:41.802 INFO:tasks.workunit.client.0.smithi121.stdout:5/544: creat dd/de/d19/d13/d53/fbb x:0 0 0 2024-04-05T21:26:41.803 INFO:tasks.workunit.client.0.smithi121.stdout:4/541: mknod d5/d11/d2c/d9d/dae/d29/d2e/caf 0 2024-04-05T21:26:41.804 INFO:tasks.workunit.client.0.smithi121.stdout:4/542: fsync d5/d11/d2c/d9d/dae/f30 0 2024-04-05T21:26:41.805 INFO:tasks.workunit.client.0.smithi121.stdout:5/545: symlink dd/de/d39/d3b/d50/d82/da6/lbc 0 2024-04-05T21:26:41.805 INFO:tasks.workunit.client.1.smithi188.stdout:3/737: dwrite d5/d7/d6d/dd8/f2a [0,4194304] 0 2024-04-05T21:26:41.808 INFO:tasks.workunit.client.0.smithi121.stdout:5/546: creat dd/de/d19/d3c/fbd x:0 0 0 2024-04-05T21:26:41.808 INFO:tasks.workunit.client.1.smithi188.stdout:8/620: symlink d5/d4b/lcd 0 2024-04-05T21:26:41.809 INFO:tasks.workunit.client.1.smithi188.stdout:9/693: dread dd/d11/d1f/d26/d55/d98/da5/fe6 [0,4194304] 0 2024-04-05T21:26:41.809 INFO:tasks.workunit.client.1.smithi188.stdout:3/738: stat d5/d7/d6d/d75/d83/ceb 0 2024-04-05T21:26:41.809 INFO:tasks.workunit.client.0.smithi121.stdout:0/493: write d7/f13 [4708248,100025] 0 2024-04-05T21:26:41.810 INFO:tasks.workunit.client.1.smithi188.stdout:5/600: rename d6/d15/d1b/cb5 to d6/d15/d5c/d7a/d44/cd4 0 2024-04-05T21:26:41.813 INFO:tasks.workunit.client.0.smithi121.stdout:0/494: creat d7/d71/fae x:0 0 0 2024-04-05T21:26:41.814 INFO:tasks.workunit.client.1.smithi188.stdout:3/739: getdents d5/d39/d84 0 2024-04-05T21:26:41.818 INFO:tasks.workunit.client.0.smithi121.stdout:5/547: dwrite dd/de/d19/f23 [0,4194304] 0 2024-04-05T21:26:41.819 INFO:tasks.workunit.client.1.smithi188.stdout:5/601: dwrite d6/d15/d5c/d7a/d44/d51/dc6/faa [0,4194304] 0 2024-04-05T21:26:41.820 INFO:tasks.workunit.client.0.smithi121.stdout:0/495: write d7/d71/d29/d2a/d39/f40 [485915,106219] 0 2024-04-05T21:26:41.821 INFO:tasks.workunit.client.0.smithi121.stdout:0/496: chown d7/d1c 223706 1 2024-04-05T21:26:41.822 INFO:tasks.workunit.client.1.smithi188.stdout:5/602: mknod d6/d82/d40/d42/d81/da6/db2/cd5 0 2024-04-05T21:26:41.830 INFO:tasks.workunit.client.0.smithi121.stdout:4/543: truncate d5/d45/f74 2005012 0 2024-04-05T21:26:41.830 INFO:tasks.workunit.client.0.smithi121.stdout:5/548: creat dd/de/d39/d3b/d50/d82/fbe x:0 0 0 2024-04-05T21:26:41.833 INFO:tasks.workunit.client.0.smithi121.stdout:3/316: dread d0/d3/d24/d3d/d1a/f33 [0,4194304] 0 2024-04-05T21:26:41.834 INFO:tasks.workunit.client.0.smithi121.stdout:5/549: symlink dd/de/d19/d1c/d7b/lbf 0 2024-04-05T21:26:41.834 INFO:tasks.workunit.client.0.smithi121.stdout:3/317: write d0/d3/f5b [742507,130762] 0 2024-04-05T21:26:41.835 INFO:tasks.workunit.client.0.smithi121.stdout:4/544: getdents d5/d45/d8b 0 2024-04-05T21:26:41.836 INFO:tasks.workunit.client.0.smithi121.stdout:3/318: fsync d0/d3/de/f36 0 2024-04-05T21:26:41.838 INFO:tasks.workunit.client.0.smithi121.stdout:4/545: creat d5/d45/fb0 x:0 0 0 2024-04-05T21:26:41.839 INFO:tasks.workunit.client.0.smithi121.stdout:3/319: chown d0/d3/de/c46 6789241 1 2024-04-05T21:26:41.840 INFO:tasks.workunit.client.0.smithi121.stdout:4/546: creat d5/d45/fb1 x:0 0 0 2024-04-05T21:26:41.840 INFO:tasks.workunit.client.0.smithi121.stdout:5/550: dread dd/de/d19/f23 [0,4194304] 0 2024-04-05T21:26:41.841 INFO:tasks.workunit.client.0.smithi121.stdout:3/320: stat d0/d3/d24/d3d/d1a/f22 0 2024-04-05T21:26:41.841 INFO:tasks.workunit.client.0.smithi121.stdout:4/547: chown d5/d11/d28/d87/f96 4997 1 2024-04-05T21:26:41.842 INFO:tasks.workunit.client.0.smithi121.stdout:9/289: sync 2024-04-05T21:26:41.844 INFO:tasks.workunit.client.0.smithi121.stdout:4/548: mkdir d5/d11/d28/d3b/db2 0 2024-04-05T21:26:41.845 INFO:tasks.workunit.client.0.smithi121.stdout:3/321: symlink d0/d3/d24/d3d/d1a/l6f 0 2024-04-05T21:26:41.847 INFO:tasks.workunit.client.0.smithi121.stdout:9/290: unlink d6/df/d1d/l22 0 2024-04-05T21:26:41.848 INFO:tasks.workunit.client.0.smithi121.stdout:4/549: symlink d5/d11/d28/d3b/lb3 0 2024-04-05T21:26:41.849 INFO:tasks.workunit.client.0.smithi121.stdout:9/291: fsync d6/f1c 0 2024-04-05T21:26:41.850 INFO:tasks.workunit.client.0.smithi121.stdout:3/322: creat d0/d3/d24/d3d/d1a/d23/f70 x:0 0 0 2024-04-05T21:26:41.851 INFO:tasks.workunit.client.0.smithi121.stdout:5/551: dwrite dd/de/d19/d38/f4f [0,4194304] 0 2024-04-05T21:26:41.852 INFO:tasks.workunit.client.1.smithi188.stdout:6/687: dwrite d6/d13/d18/d3c/f5a [0,4194304] 0 2024-04-05T21:26:41.852 INFO:tasks.workunit.client.1.smithi188.stdout:1/661: dwrite d1/dc/d16/d2d/fd1 [0,4194304] 0 2024-04-05T21:26:41.853 INFO:tasks.workunit.client.1.smithi188.stdout:6/688: readlink d6/d13/d18/d19/d2c/d34/l93 0 2024-04-05T21:26:41.855 INFO:tasks.workunit.client.1.smithi188.stdout:1/662: mknod d1/dc/d16/d21/d27/d4a/cd6 0 2024-04-05T21:26:41.855 INFO:tasks.workunit.client.0.smithi121.stdout:5/552: readlink dd/de/d19/d1c/d36/l62 0 2024-04-05T21:26:41.858 INFO:tasks.workunit.client.0.smithi121.stdout:9/292: mkdir d6/df/d1d/d2c/d37/d3a/d5d 0 2024-04-05T21:26:41.860 INFO:tasks.workunit.client.0.smithi121.stdout:9/293: chown d6/c4e 126361146 1 2024-04-05T21:26:41.860 INFO:tasks.workunit.client.1.smithi188.stdout:4/884: write d0/d2/d2a/d34/fa1 [1920955,37023] 0 2024-04-05T21:26:41.862 INFO:tasks.workunit.client.1.smithi188.stdout:1/663: symlink d1/dc/d16/d46/d88/ld7 0 2024-04-05T21:26:41.863 INFO:tasks.workunit.client.0.smithi121.stdout:3/323: creat d0/d3/d24/d3d/d1a/d1e/f71 x:0 0 0 2024-04-05T21:26:41.863 INFO:tasks.workunit.client.0.smithi121.stdout:4/550: dwrite d5/f16 [0,4194304] 0 2024-04-05T21:26:41.864 INFO:tasks.workunit.client.1.smithi188.stdout:1/664: fsync d1/dc/d16/f33 0 2024-04-05T21:26:41.865 INFO:tasks.workunit.client.1.smithi188.stdout:1/665: dread - d1/dc/d16/d21/d27/d49/f73 zero size 2024-04-05T21:26:41.867 INFO:tasks.workunit.client.1.smithi188.stdout:1/666: dread - d1/d7/d4e/fca zero size 2024-04-05T21:26:41.871 INFO:tasks.workunit.client.0.smithi121.stdout:4/551: getdents d5/d45/d8b 0 2024-04-05T21:26:41.872 INFO:tasks.workunit.client.0.smithi121.stdout:9/294: creat d6/f5e x:0 0 0 2024-04-05T21:26:41.877 INFO:tasks.workunit.client.1.smithi188.stdout:5/603: write d6/d17/f97 [1300562,74383] 0 2024-04-05T21:26:41.877 INFO:tasks.workunit.client.0.smithi121.stdout:9/295: creat d6/df/f5f x:0 0 0 2024-04-05T21:26:41.878 INFO:tasks.workunit.client.1.smithi188.stdout:6/689: write d6/d28/d45/f96 [94054,86338] 0 2024-04-05T21:26:41.879 INFO:tasks.workunit.client.1.smithi188.stdout:5/604: chown d6/fb7 97 1 2024-04-05T21:26:41.879 INFO:tasks.workunit.client.1.smithi188.stdout:4/885: getdents d0/d14/d15/d1b 0 2024-04-05T21:26:41.880 INFO:tasks.workunit.client.1.smithi188.stdout:5/605: readlink d6/d15/l48 0 2024-04-05T21:26:41.882 INFO:tasks.workunit.client.0.smithi121.stdout:3/324: dread d0/f41 [0,4194304] 0 2024-04-05T21:26:41.883 INFO:tasks.workunit.client.1.smithi188.stdout:5/606: read d6/d15/d5c/d7a/f52 [772792,94592] 0 2024-04-05T21:26:41.883 INFO:tasks.workunit.client.1.smithi188.stdout:6/690: link d6/d71/ld3 d6/d13/d18/d19/d2c/d34/ld4 0 2024-04-05T21:26:41.884 INFO:tasks.workunit.client.0.smithi121.stdout:0/497: dread d7/dc/f15 [0,4194304] 0 2024-04-05T21:26:41.884 INFO:tasks.workunit.client.0.smithi121.stdout:4/552: dwrite d5/d11/d28/f1b [0,4194304] 0 2024-04-05T21:26:41.885 INFO:tasks.workunit.client.1.smithi188.stdout:5/607: chown d6/d82/d40/d42/d81/c87 27144 1 2024-04-05T21:26:41.886 INFO:tasks.workunit.client.0.smithi121.stdout:0/498: chown d7/d1c/f1f 94390377 1 2024-04-05T21:26:41.888 INFO:tasks.workunit.client.0.smithi121.stdout:0/499: write d7/d71/d29/f79 [3015267,44688] 0 2024-04-05T21:26:41.888 INFO:tasks.workunit.client.1.smithi188.stdout:4/886: dwrite d0/d2/d52/dcc/ff2 [0,4194304] 0 2024-04-05T21:26:41.890 INFO:tasks.workunit.client.0.smithi121.stdout:4/553: mkdir d5/d11/d2c/d9d/dae/d29/db4 0 2024-04-05T21:26:41.891 INFO:tasks.workunit.client.1.smithi188.stdout:6/691: rename d6/d7/c23 to d6/cd5 0 2024-04-05T21:26:41.894 INFO:tasks.workunit.client.0.smithi121.stdout:4/554: unlink d5/d11/d2c/d9d/dae/c7e 0 2024-04-05T21:26:41.895 INFO:tasks.workunit.client.0.smithi121.stdout:4/555: write d5/d11/d20/f82 [370247,32401] 0 2024-04-05T21:26:41.895 INFO:tasks.workunit.client.0.smithi121.stdout:5/553: dread dd/de/d19/f37 [0,4194304] 0 2024-04-05T21:26:41.896 INFO:tasks.workunit.client.1.smithi188.stdout:4/887: creat d0/d14/f12c x:0 0 0 2024-04-05T21:26:41.897 INFO:tasks.workunit.client.1.smithi188.stdout:6/692: mkdir d6/d13/d18/d3c/dd2/dd6 0 2024-04-05T21:26:41.898 INFO:tasks.workunit.client.1.smithi188.stdout:6/693: rmdir d6 39 2024-04-05T21:26:41.899 INFO:tasks.workunit.client.0.smithi121.stdout:4/556: write d5/d11/d20/f82 [3755137,101122] 0 2024-04-05T21:26:41.899 INFO:tasks.workunit.client.0.smithi121.stdout:4/557: chown d5/d11/d60 0 1 2024-04-05T21:26:41.899 INFO:tasks.workunit.client.0.smithi121.stdout:4/558: write f4 [2476087,50756] 0 2024-04-05T21:26:41.901 INFO:tasks.workunit.client.0.smithi121.stdout:5/554: readlink dd/de/d19/d1c/d7b/l89 0 2024-04-05T21:26:41.903 INFO:tasks.workunit.client.0.smithi121.stdout:5/555: creat dd/de/d39/d3b/fc0 x:0 0 0 2024-04-05T21:26:41.905 INFO:tasks.workunit.client.1.smithi188.stdout:4/888: link d0/d2/d2a/d34/fd1 d0/d14/d15/d78/d112/f12d 0 2024-04-05T21:26:41.906 INFO:tasks.workunit.client.0.smithi121.stdout:5/556: mkdir dd/de/dc1 0 2024-04-05T21:26:41.906 INFO:tasks.workunit.client.1.smithi188.stdout:6/694: dread d6/d13/d38/f4e [0,4194304] 0 2024-04-05T21:26:41.907 INFO:tasks.workunit.client.0.smithi121.stdout:5/557: write dd/de/d19/d1c/d74/f81 [905896,11841] 0 2024-04-05T21:26:41.908 INFO:tasks.workunit.client.1.smithi188.stdout:6/695: read - d6/d13/d18/d3c/fcf zero size 2024-04-05T21:26:41.911 INFO:tasks.workunit.client.1.smithi188.stdout:6/696: stat d6/d13/d48/d49/d60/f7c 0 2024-04-05T21:26:41.911 INFO:tasks.workunit.client.0.smithi121.stdout:9/296: write d6/df/f27 [1119118,20729] 0 2024-04-05T21:26:41.912 INFO:tasks.workunit.client.1.smithi188.stdout:4/889: dread d0/d2/d2a/d34/fa1 [0,4194304] 0 2024-04-05T21:26:41.916 INFO:tasks.workunit.client.1.smithi188.stdout:4/890: rename d0/d2/d17/d48/d70 to d0/d14/d15/d53/d12e 0 2024-04-05T21:26:41.916 INFO:tasks.workunit.client.0.smithi121.stdout:9/297: fdatasync d6/df/d1d/d2c/f24 0 2024-04-05T21:26:41.916 INFO:tasks.workunit.client.0.smithi121.stdout:9/298: fdatasync d6/f5e 0 2024-04-05T21:26:41.917 INFO:tasks.workunit.client.0.smithi121.stdout:3/325: dread d0/d3/de/f17 [0,4194304] 0 2024-04-05T21:26:41.917 INFO:tasks.workunit.client.1.smithi188.stdout:4/891: mkdir d0/d14/d15/d53/d12e/d12f 0 2024-04-05T21:26:41.920 INFO:tasks.workunit.client.0.smithi121.stdout:3/326: write d0/d3/de/f17 [3877566,111631] 0 2024-04-05T21:26:41.922 INFO:tasks.workunit.client.0.smithi121.stdout:4/559: write d5/d45/f74 [1349463,50089] 0 2024-04-05T21:26:41.923 INFO:tasks.workunit.client.0.smithi121.stdout:5/558: dwrite dd/de/d39/d3b/f94 [0,4194304] 0 2024-04-05T21:26:41.927 INFO:tasks.workunit.client.1.smithi188.stdout:5/608: write d6/f73 [417352,101230] 0 2024-04-05T21:26:41.928 INFO:tasks.workunit.client.0.smithi121.stdout:9/299: dwrite d6/f16 [0,4194304] 0 2024-04-05T21:26:41.930 INFO:tasks.workunit.client.0.smithi121.stdout:4/560: truncate d5/d11/d28/d73/f76 806681 0 2024-04-05T21:26:41.930 INFO:tasks.workunit.client.1.smithi188.stdout:6/697: write d6/d7/f1e [4948540,121461] 0 2024-04-05T21:26:41.930 INFO:tasks.workunit.client.1.smithi188.stdout:1/667: dwrite d1/dc/d16/d21/d27/d4a/f63 [0,4194304] 0 2024-04-05T21:26:41.933 INFO:tasks.workunit.client.0.smithi121.stdout:4/561: write d5/d11/d60/f9f [132062,47590] 0 2024-04-05T21:26:41.934 INFO:tasks.workunit.client.0.smithi121.stdout:3/327: rename d0/d3/d24/d3d/d1a/f33 to d0/d3/de/f72 0 2024-04-05T21:26:41.936 INFO:tasks.workunit.client.0.smithi121.stdout:4/562: chown d5/d11/d28/f1b 79990 1 2024-04-05T21:26:41.938 INFO:tasks.workunit.client.0.smithi121.stdout:9/300: creat d6/df/d1d/d2c/f60 x:0 0 0 2024-04-05T21:26:41.938 INFO:tasks.workunit.client.0.smithi121.stdout:5/559: dwrite dd/de/d19/d1c/f75 [0,4194304] 0 2024-04-05T21:26:41.943 INFO:tasks.workunit.client.0.smithi121.stdout:4/563: chown d5/d11/d2c/d9d/dae/d29/la8 156471091 1 2024-04-05T21:26:41.943 INFO:tasks.workunit.client.0.smithi121.stdout:0/500: dread d7/d71/f3b [0,4194304] 0 2024-04-05T21:26:41.944 INFO:tasks.workunit.client.0.smithi121.stdout:5/560: creat dd/de/d19/d1c/fc2 x:0 0 0 2024-04-05T21:26:41.945 INFO:tasks.workunit.client.1.smithi188.stdout:5/609: dread - d6/d17/d23/f55 zero size 2024-04-05T21:26:41.947 INFO:tasks.workunit.client.0.smithi121.stdout:0/501: creat d7/dc/faf x:0 0 0 2024-04-05T21:26:41.949 INFO:tasks.workunit.client.1.smithi188.stdout:6/698: truncate d6/d13/d18/d19/d2c/f32 1463079 0 2024-04-05T21:26:41.949 INFO:tasks.workunit.client.0.smithi121.stdout:4/564: getdents d5/d11/d20 0 2024-04-05T21:26:41.950 INFO:tasks.workunit.client.1.smithi188.stdout:6/699: chown f5 233 1 2024-04-05T21:26:41.953 INFO:tasks.workunit.client.0.smithi121.stdout:0/502: read d7/d71/d29/d2a/d2c/f7b [9342,123514] 0 2024-04-05T21:26:41.953 INFO:tasks.workunit.client.1.smithi188.stdout:6/700: chown d6/d13/d18/d19/dc8/cb6 247726 1 2024-04-05T21:26:41.954 INFO:tasks.workunit.client.1.smithi188.stdout:6/701: write d6/d13/d38/f4c [34861,20672] 0 2024-04-05T21:26:41.955 INFO:tasks.workunit.client.0.smithi121.stdout:4/565: creat d5/d11/d2c/d9d/fb5 x:0 0 0 2024-04-05T21:26:41.955 INFO:tasks.workunit.client.0.smithi121.stdout:0/503: rmdir d7/d7e 39 2024-04-05T21:26:41.956 INFO:tasks.workunit.client.0.smithi121.stdout:5/561: dwrite dd/de/d19/d38/f97 [0,4194304] 0 2024-04-05T21:26:41.958 INFO:tasks.workunit.client.1.smithi188.stdout:5/610: rename d6/f99 to d6/d17/d23/d7d/dcf/fd6 0 2024-04-05T21:26:41.960 INFO:tasks.workunit.client.1.smithi188.stdout:5/611: dread - d6/d15/d5c/d7a/fa1 zero size 2024-04-05T21:26:41.960 INFO:tasks.workunit.client.1.smithi188.stdout:4/892: dread d0/d2/d2a/d34/fd1 [0,4194304] 0 2024-04-05T21:26:41.961 INFO:tasks.workunit.client.1.smithi188.stdout:6/702: getdents d6/d13/d18/d19/da3 0 2024-04-05T21:26:41.962 INFO:tasks.workunit.client.1.smithi188.stdout:6/703: chown d6/d13/d18/d3c/dd2 14345 1 2024-04-05T21:26:41.963 INFO:tasks.workunit.client.0.smithi121.stdout:4/566: dread d5/f16 [4194304,4194304] 0 2024-04-05T21:26:41.964 INFO:tasks.workunit.client.0.smithi121.stdout:0/504: write d7/d71/f26 [4516839,92553] 0 2024-04-05T21:26:41.965 INFO:tasks.workunit.client.0.smithi121.stdout:3/328: getdents d0/d3/de 0 2024-04-05T21:26:41.966 INFO:tasks.workunit.client.1.smithi188.stdout:4/893: creat d0/d14/d15/d53/d82/dd5/df4/d106/f130 x:0 0 0 2024-04-05T21:26:41.967 INFO:tasks.workunit.client.0.smithi121.stdout:5/562: link dd/de/d39/d3b/d50/d82/fbe dd/de/fc3 0 2024-04-05T21:26:41.968 INFO:tasks.workunit.client.1.smithi188.stdout:4/894: write d0/d2/d4f/f5c [2931259,126374] 0 2024-04-05T21:26:41.969 INFO:tasks.workunit.client.1.smithi188.stdout:4/895: readlink d0/d81/lde 0 2024-04-05T21:26:41.972 INFO:tasks.workunit.client.1.smithi188.stdout:6/704: dwrite d6/f2f [0,4194304] 0 2024-04-05T21:26:41.973 INFO:tasks.workunit.client.1.smithi188.stdout:4/896: creat d0/dbe/f131 x:0 0 0 2024-04-05T21:26:41.974 INFO:tasks.workunit.client.1.smithi188.stdout:6/705: dread - d6/d13/d48/d49/d60/f7c zero size 2024-04-05T21:26:41.978 INFO:tasks.workunit.client.0.smithi121.stdout:5/563: creat dd/de/d19/d38/d45/d68/fc4 x:0 0 0 2024-04-05T21:26:41.978 INFO:tasks.workunit.client.0.smithi121.stdout:9/301: dwrite d6/f15 [0,4194304] 0 2024-04-05T21:26:41.979 INFO:tasks.workunit.client.0.smithi121.stdout:4/567: link d5/d11/d20/l79 d5/d11/d2c/d9d/lb6 0 2024-04-05T21:26:41.981 INFO:tasks.workunit.client.1.smithi188.stdout:1/668: dread d1/d7/f66 [0,4194304] 0 2024-04-05T21:26:41.981 INFO:tasks.workunit.client.1.smithi188.stdout:1/669: chown d1/dc/d16/d46/d88/ld7 0 1 2024-04-05T21:26:41.982 INFO:tasks.workunit.client.1.smithi188.stdout:6/706: rename d6/d13/d18/d3c/f3d to d6/d13/d38/fd7 0 2024-04-05T21:26:41.982 INFO:tasks.workunit.client.1.smithi188.stdout:1/670: dread - d1/db2/fb4 zero size 2024-04-05T21:26:41.982 INFO:tasks.workunit.client.0.smithi121.stdout:9/302: write d6/f5e [822598,45064] 0 2024-04-05T21:26:41.982 INFO:tasks.workunit.client.0.smithi121.stdout:0/505: dwrite d7/d71/f8f [0,4194304] 0 2024-04-05T21:26:41.984 INFO:tasks.workunit.client.0.smithi121.stdout:3/329: write d0/f27 [285733,70584] 0 2024-04-05T21:26:41.989 INFO:tasks.workunit.client.1.smithi188.stdout:4/897: write d0/d2/d52/f79 [5129719,73592] 0 2024-04-05T21:26:41.990 INFO:tasks.workunit.client.1.smithi188.stdout:6/707: link d6/d13/d18/d19/f7e d6/d71/fd8 0 2024-04-05T21:26:41.991 INFO:tasks.workunit.client.0.smithi121.stdout:5/564: mkdir dd/de/d19/d1c/d74/d8c/dc5 0 2024-04-05T21:26:41.991 INFO:tasks.workunit.client.0.smithi121.stdout:9/303: mknod d6/df/c61 0 2024-04-05T21:26:41.993 INFO:tasks.workunit.client.0.smithi121.stdout:0/506: symlink d7/d71/d45/d5a/lb0 0 2024-04-05T21:26:41.994 INFO:tasks.workunit.client.1.smithi188.stdout:4/898: mkdir d0/d14/d15/d1b/d132 0 2024-04-05T21:26:41.995 INFO:tasks.workunit.client.1.smithi188.stdout:5/612: dread d6/d82/d40/d42/d81/d95/f9d [0,4194304] 0 2024-04-05T21:26:41.997 INFO:tasks.workunit.client.0.smithi121.stdout:4/568: link d5/d11/d2c/d8c/c8e d5/d11/d2c/d8c/cb7 0 2024-04-05T21:26:41.998 INFO:tasks.workunit.client.1.smithi188.stdout:5/613: dread d6/d17/d39/d8e/f8f [0,4194304] 0 2024-04-05T21:26:41.999 INFO:tasks.workunit.client.0.smithi121.stdout:4/569: rmdir d5/d11/d60 39 2024-04-05T21:26:42.000 INFO:tasks.workunit.client.1.smithi188.stdout:5/614: rmdir d6/d15/d5c/d7a/d44/d9b 39 2024-04-05T21:26:42.013 INFO:tasks.workunit.client.1.smithi188.stdout:1/671: write d1/dc/d16/d21/d27/d49/f9b [302821,100617] 0 2024-04-05T21:26:42.014 INFO:tasks.workunit.client.0.smithi121.stdout:9/304: write d6/df/f20 [1613563,101773] 0 2024-04-05T21:26:42.015 INFO:tasks.workunit.client.1.smithi188.stdout:4/899: write d0/f102 [918399,121362] 0 2024-04-05T21:26:42.019 INFO:tasks.workunit.client.0.smithi121.stdout:5/565: dwrite dd/de/d19/d13/f5d [0,4194304] 0 2024-04-05T21:26:42.020 INFO:tasks.workunit.client.1.smithi188.stdout:1/672: write d1/dc/d16/d46/d88/d13/d60/fd4 [143580,121997] 0 2024-04-05T21:26:42.024 INFO:tasks.workunit.client.1.smithi188.stdout:6/708: dwrite d6/d13/d18/d19/f30 [0,4194304] 0 2024-04-05T21:26:42.025 INFO:tasks.workunit.client.0.smithi121.stdout:5/566: truncate dd/de/d19/d13/f49 3064289 0 2024-04-05T21:26:42.028 INFO:tasks.workunit.client.1.smithi188.stdout:1/673: symlink d1/dc/d16/d2d/db0/ld8 0 2024-04-05T21:26:42.029 INFO:tasks.workunit.client.1.smithi188.stdout:5/615: dwrite d6/d17/d23/f55 [0,4194304] 0 2024-04-05T21:26:42.030 INFO:tasks.workunit.client.1.smithi188.stdout:6/709: mknod d6/d13/db9/cd9 0 2024-04-05T21:26:42.031 INFO:tasks.workunit.client.1.smithi188.stdout:5/616: stat d6/d82/d40/d42 0 2024-04-05T21:26:42.032 INFO:tasks.workunit.client.1.smithi188.stdout:1/674: write d1/dc/d16/d21/d27/f29 [6913643,72066] 0 2024-04-05T21:26:42.032 INFO:tasks.workunit.client.1.smithi188.stdout:5/617: write d6/d82/d40/dc1/fd1 [964649,36267] 0 2024-04-05T21:26:42.035 INFO:tasks.workunit.client.1.smithi188.stdout:6/710: read d6/d13/d18/f50 [115710,60666] 0 2024-04-05T21:26:42.038 INFO:tasks.workunit.client.1.smithi188.stdout:4/900: dread d0/d14/d2f/fb8 [0,4194304] 0 2024-04-05T21:26:42.040 INFO:tasks.workunit.client.1.smithi188.stdout:4/901: readlink d0/d2/d2a/d34/d6b/lf5 0 2024-04-05T21:26:42.041 INFO:tasks.workunit.client.1.smithi188.stdout:5/618: link d6/f5b d6/d17/d23/d7d/fd7 0 2024-04-05T21:26:42.042 INFO:tasks.workunit.client.1.smithi188.stdout:6/711: truncate d6/d13/d48/d49/d60/f7c 101799 0 2024-04-05T21:26:42.042 INFO:tasks.workunit.client.0.smithi121.stdout:5/567: dread dd/de/d19/f28 [0,4194304] 0 2024-04-05T21:26:42.044 INFO:tasks.workunit.client.0.smithi121.stdout:5/568: mknod dd/de/d19/d38/da3/cc6 0 2024-04-05T21:26:42.046 INFO:tasks.workunit.client.1.smithi188.stdout:6/712: creat d6/d13/d18/d19/d2c/d34/fda x:0 0 0 2024-04-05T21:26:42.047 INFO:tasks.workunit.client.1.smithi188.stdout:4/902: creat d0/d2/d2a/d34/f133 x:0 0 0 2024-04-05T21:26:42.048 INFO:tasks.workunit.client.1.smithi188.stdout:5/619: dwrite d6/f11 [8388608,4194304] 0 2024-04-05T21:26:42.049 INFO:tasks.workunit.client.0.smithi121.stdout:5/569: dwrite dd/de/d39/d3b/f4a [0,4194304] 0 2024-04-05T21:26:42.050 INFO:tasks.workunit.client.1.smithi188.stdout:6/713: creat d6/d13/d38/fdb x:0 0 0 2024-04-05T21:26:42.053 INFO:tasks.workunit.client.0.smithi121.stdout:5/570: unlink dd/de/d19/d1c/d36/d46/l7a 0 2024-04-05T21:26:42.057 INFO:tasks.workunit.client.1.smithi188.stdout:4/903: dwrite d0/d14/df6/f125 [0,4194304] 0 2024-04-05T21:26:42.059 INFO:tasks.workunit.client.1.smithi188.stdout:5/620: getdents d6/d17/d39/d8e 0 2024-04-05T21:26:42.060 INFO:tasks.workunit.client.1.smithi188.stdout:6/714: dread d6/d13/d18/d19/da3/fca [0,4194304] 0 2024-04-05T21:26:42.062 INFO:tasks.workunit.client.1.smithi188.stdout:6/715: stat d6/d13/d38/fdb 0 2024-04-05T21:26:42.065 INFO:tasks.workunit.client.1.smithi188.stdout:5/621: unlink d6/d82/d40/d42/f67 0 2024-04-05T21:26:42.066 INFO:tasks.workunit.client.1.smithi188.stdout:4/904: creat d0/d14/f134 x:0 0 0 2024-04-05T21:26:42.067 INFO:tasks.workunit.client.1.smithi188.stdout:5/622: mknod d6/d17/cd8 0 2024-04-05T21:26:42.069 INFO:tasks.workunit.client.1.smithi188.stdout:4/905: creat d0/d14/d15/d78/db1/f135 x:0 0 0 2024-04-05T21:26:42.073 INFO:tasks.workunit.client.1.smithi188.stdout:5/623: link d6/d15/d5c/d7a/d44/d98/dbf/lca d6/d15/d5c/d7a/ld9 0 2024-04-05T21:26:42.077 INFO:tasks.workunit.client.1.smithi188.stdout:1/675: dwrite d1/dc/d16/d46/d88/d3f/f45 [0,4194304] 0 2024-04-05T21:26:42.078 INFO:tasks.workunit.client.1.smithi188.stdout:4/906: dwrite d0/d14/d15/d78/fae [0,4194304] 0 2024-04-05T21:26:42.081 INFO:tasks.workunit.client.1.smithi188.stdout:5/624: creat d6/d15/d5c/d7a/d44/fda x:0 0 0 2024-04-05T21:26:42.082 INFO:tasks.workunit.client.1.smithi188.stdout:1/676: creat d1/d7/dbf/fd9 x:0 0 0 2024-04-05T21:26:42.086 INFO:tasks.workunit.client.1.smithi188.stdout:5/625: read d6/dad/fbd [89636,94730] 0 2024-04-05T21:26:42.087 INFO:tasks.workunit.client.1.smithi188.stdout:4/907: read d0/d14/f86 [2978403,115237] 0 2024-04-05T21:26:42.088 INFO:tasks.workunit.client.1.smithi188.stdout:5/626: read d6/d17/f97 [142584,120371] 0 2024-04-05T21:26:42.092 INFO:tasks.workunit.client.1.smithi188.stdout:5/627: creat d6/d17/d39/d8e/fdb x:0 0 0 2024-04-05T21:26:42.097 INFO:tasks.workunit.client.1.smithi188.stdout:1/677: write d1/fbc [2478437,27980] 0 2024-04-05T21:26:42.098 INFO:tasks.workunit.client.1.smithi188.stdout:4/908: getdents d0/d14/d15/d53/d82/d8f/dbd 0 2024-04-05T21:26:42.101 INFO:tasks.workunit.client.1.smithi188.stdout:6/716: dwrite d6/d13/d18/d19/d2c/f92 [0,4194304] 0 2024-04-05T21:26:42.102 INFO:tasks.workunit.client.1.smithi188.stdout:6/717: fsync d6/d7/fc 0 2024-04-05T21:26:42.107 INFO:tasks.workunit.client.1.smithi188.stdout:1/678: dwrite d1/d7/d4e/d5f/f76 [0,4194304] 0 2024-04-05T21:26:42.110 INFO:tasks.workunit.client.1.smithi188.stdout:4/909: link d0/d14/f6e d0/d14/d15/d78/d112/f136 0 2024-04-05T21:26:42.111 INFO:tasks.workunit.client.1.smithi188.stdout:1/679: stat d1/dc/d16/d46/d88/d13/d60 0 2024-04-05T21:26:42.113 INFO:tasks.workunit.client.1.smithi188.stdout:4/910: mkdir d0/dbe/d137 0 2024-04-05T21:26:42.116 INFO:tasks.workunit.client.1.smithi188.stdout:1/680: getdents d1/d7/dbf 0 2024-04-05T21:26:42.116 INFO:tasks.workunit.client.1.smithi188.stdout:4/911: truncate d0/d2/d52/fe4 32507 0 2024-04-05T21:26:42.119 INFO:tasks.workunit.client.1.smithi188.stdout:4/912: mknod d0/d2/d2a/d34/d6b/c138 0 2024-04-05T21:26:42.126 INFO:tasks.workunit.client.1.smithi188.stdout:1/681: write d1/dc/d16/d21/d27/d4a/dc1/fce [179820,43857] 0 2024-04-05T21:26:42.127 INFO:tasks.workunit.client.1.smithi188.stdout:6/718: dwrite d6/d13/d48/d49/d60/f7c [0,4194304] 0 2024-04-05T21:26:42.129 INFO:tasks.workunit.client.1.smithi188.stdout:6/719: fsync d6/d13/d18/f50 0 2024-04-05T21:26:42.130 INFO:tasks.workunit.client.1.smithi188.stdout:6/720: chown d6/d28/l57 191537 1 2024-04-05T21:26:42.130 INFO:tasks.workunit.client.1.smithi188.stdout:6/721: readlink d6/d28/l67 0 2024-04-05T21:26:42.132 INFO:tasks.workunit.client.1.smithi188.stdout:1/682: rename d1/dc/d16/d46/d88/d3f/laa to d1/d7/dbf/lda 0 2024-04-05T21:26:42.136 INFO:tasks.workunit.client.1.smithi188.stdout:6/722: rename d6/d13/d48/d49 to d6/d13/d18/d3c/ddc 0 2024-04-05T21:26:42.136 INFO:tasks.workunit.client.1.smithi188.stdout:1/683: truncate d1/dc/d16/d21/f68 1572172 0 2024-04-05T21:26:42.141 INFO:tasks.workunit.client.1.smithi188.stdout:1/684: dread d1/dc/d16/d46/d88/d13/f82 [0,4194304] 0 2024-04-05T21:26:42.142 INFO:tasks.workunit.client.1.smithi188.stdout:1/685: truncate d1/d7/dbf/fc4 878381 0 2024-04-05T21:26:42.145 INFO:tasks.workunit.client.1.smithi188.stdout:1/686: fsync d1/dc/d16/d21/d27/d49/fb8 0 2024-04-05T21:26:42.152 INFO:tasks.workunit.client.1.smithi188.stdout:6/723: dread d6/d28/f43 [0,4194304] 0 2024-04-05T21:26:42.152 INFO:tasks.workunit.client.1.smithi188.stdout:1/687: truncate d1/dc/d16/d46/d88/d13/f18 8132385 0 2024-04-05T21:26:42.154 INFO:tasks.workunit.client.1.smithi188.stdout:6/724: creat d6/d7/fdd x:0 0 0 2024-04-05T21:26:42.154 INFO:tasks.workunit.client.1.smithi188.stdout:1/688: read d1/dc/d16/f92 [3556584,55262] 0 2024-04-05T21:26:42.155 INFO:tasks.workunit.client.1.smithi188.stdout:6/725: stat d6/d13/d18/d19/f30 0 2024-04-05T21:26:42.156 INFO:tasks.workunit.client.1.smithi188.stdout:1/689: chown d1/dc/d16/d46/d88/d86/f8f 235081 1 2024-04-05T21:26:42.157 INFO:tasks.workunit.client.0.smithi121.stdout:8/351: sync 2024-04-05T21:26:42.157 INFO:tasks.workunit.client.0.smithi121.stdout:7/460: sync 2024-04-05T21:26:42.157 INFO:tasks.workunit.client.0.smithi121.stdout:1/487: sync 2024-04-05T21:26:42.157 INFO:tasks.workunit.client.0.smithi121.stdout:6/332: sync 2024-04-05T21:26:42.157 INFO:tasks.workunit.client.0.smithi121.stdout:2/393: sync 2024-04-05T21:26:42.159 INFO:tasks.workunit.client.1.smithi188.stdout:6/726: getdents d6/d13/d18/d19/d2c/db2 0 2024-04-05T21:26:42.159 INFO:tasks.workunit.client.1.smithi188.stdout:1/690: dread d1/dc/d16/d21/d27/d4a/dc1/fce [0,4194304] 0 2024-04-05T21:26:42.159 INFO:tasks.workunit.client.0.smithi121.stdout:8/352: dread - d12/d13/d56/d57/f5a zero size 2024-04-05T21:26:42.160 INFO:tasks.workunit.client.1.smithi188.stdout:6/727: stat d6/d13/d18/d3c/ddc/d60 0 2024-04-05T21:26:42.161 INFO:tasks.workunit.client.0.smithi121.stdout:6/333: creat d1/d2/d3/d5/d1f/f6a x:0 0 0 2024-04-05T21:26:42.161 INFO:tasks.workunit.client.1.smithi188.stdout:6/728: creat d6/d13/d18/d3c/daf/fde x:0 0 0 2024-04-05T21:26:42.162 INFO:tasks.workunit.client.0.smithi121.stdout:2/394: creat d2/d9/d38/d2d/d47/f70 x:0 0 0 2024-04-05T21:26:42.163 INFO:tasks.workunit.client.0.smithi121.stdout:2/395: truncate d2/d9/d38/f35 5071293 0 2024-04-05T21:26:42.164 INFO:tasks.workunit.client.0.smithi121.stdout:1/488: creat d8/d10/d14/d1a/d1b/d4c/d76/d53/f9b x:0 0 0 2024-04-05T21:26:42.164 INFO:tasks.workunit.client.1.smithi188.stdout:6/729: symlink d6/d28/d45/ldf 0 2024-04-05T21:26:42.165 INFO:tasks.workunit.client.1.smithi188.stdout:1/691: dwrite d1/dc/d16/d2d/da8/fbd [0,4194304] 0 2024-04-05T21:26:42.166 INFO:tasks.workunit.client.0.smithi121.stdout:7/461: getdents d0 0 2024-04-05T21:26:42.168 INFO:tasks.workunit.client.0.smithi121.stdout:6/334: rename d1/d2/d3/c30 to d1/d2/d1b/d35/c6b 0 2024-04-05T21:26:42.171 INFO:tasks.workunit.client.0.smithi121.stdout:2/396: unlink d2/d9/d17/d3a/f4d 0 2024-04-05T21:26:42.173 INFO:tasks.workunit.client.0.smithi121.stdout:8/353: getdents d12/d13/d56/d57 0 2024-04-05T21:26:42.179 INFO:tasks.workunit.client.0.smithi121.stdout:7/462: dwrite d0/d23/d5b/d52/f6e [0,4194304] 0 2024-04-05T21:26:42.182 INFO:tasks.workunit.client.0.smithi121.stdout:8/354: truncate d12/d13/d56/d57/f74 54898 0 2024-04-05T21:26:42.183 INFO:tasks.workunit.client.0.smithi121.stdout:1/489: dwrite d8/d10/d14/d1a/d1b/d44/f93 [0,4194304] 0 2024-04-05T21:26:42.185 INFO:tasks.workunit.client.0.smithi121.stdout:2/397: mknod d2/d9/d38/d2d/d4c/c71 0 2024-04-05T21:26:42.185 INFO:tasks.workunit.client.0.smithi121.stdout:8/355: chown d12/c39 1372 1 2024-04-05T21:26:42.186 INFO:tasks.workunit.client.0.smithi121.stdout:6/335: mknod d1/d2/d3/d4/d21/d57/d42/d48/c6c 0 2024-04-05T21:26:42.186 INFO:tasks.workunit.client.0.smithi121.stdout:2/398: stat d2/d9/c13 0 2024-04-05T21:26:42.189 INFO:tasks.workunit.client.0.smithi121.stdout:2/399: creat d2/d9/d38/d2d/f72 x:0 0 0 2024-04-05T21:26:42.190 INFO:tasks.workunit.client.0.smithi121.stdout:1/490: mkdir d8/d10/d14/d1a/d1b/d85/d9c 0 2024-04-05T21:26:42.190 INFO:tasks.workunit.client.0.smithi121.stdout:8/356: getdents d12/d50/d76 0 2024-04-05T21:26:42.195 INFO:tasks.workunit.client.0.smithi121.stdout:2/400: creat d2/d9/f73 x:0 0 0 2024-04-05T21:26:42.198 INFO:tasks.workunit.client.0.smithi121.stdout:6/336: dwrite d1/d2/d3/d5/d1f/f46 [0,4194304] 0 2024-04-05T21:26:42.200 INFO:tasks.workunit.client.0.smithi121.stdout:2/401: symlink d2/d9/d38/d6d/l74 0 2024-04-05T21:26:42.202 INFO:tasks.workunit.client.0.smithi121.stdout:7/463: write d0/d9/f50 [131853,3558] 0 2024-04-05T21:26:42.204 INFO:tasks.workunit.client.0.smithi121.stdout:8/357: dwrite d12/d13/d15/f18 [0,4194304] 0 2024-04-05T21:26:42.209 INFO:tasks.workunit.client.0.smithi121.stdout:8/358: creat d12/d50/f79 x:0 0 0 2024-04-05T21:26:42.212 INFO:tasks.workunit.client.0.smithi121.stdout:7/464: creat d0/d32/f8f x:0 0 0 2024-04-05T21:26:42.213 INFO:tasks.workunit.client.0.smithi121.stdout:2/402: rmdir d2/d9/d56/d62/d67 0 2024-04-05T21:26:42.220 INFO:tasks.workunit.client.0.smithi121.stdout:1/491: dwrite d8/d10/d14/d27/f75 [0,4194304] 0 2024-04-05T21:26:42.221 INFO:tasks.workunit.client.0.smithi121.stdout:6/337: dwrite d1/d2/d1b/d35/f62 [0,4194304] 0 2024-04-05T21:26:42.221 INFO:tasks.workunit.client.0.smithi121.stdout:2/403: dread d2/d1e/f52 [0,4194304] 0 2024-04-05T21:26:42.225 INFO:tasks.workunit.client.0.smithi121.stdout:8/359: dwrite d12/d13/d15/d1d/f36 [0,4194304] 0 2024-04-05T21:26:42.228 INFO:tasks.workunit.client.1.smithi188.stdout:7/855: sync 2024-04-05T21:26:42.229 INFO:tasks.workunit.client.0.smithi121.stdout:6/338: mkdir d1/d2/d3/d5/d1f/d6d 0 2024-04-05T21:26:42.230 INFO:tasks.workunit.client.0.smithi121.stdout:6/339: chown d1/d2/d3/c44 196 1 2024-04-05T21:26:42.230 INFO:tasks.workunit.client.1.smithi188.stdout:7/856: fdatasync d3/d2c/dae/dfd/fcd 0 2024-04-05T21:26:42.231 INFO:tasks.workunit.client.0.smithi121.stdout:8/360: rename d12/d13/d15/d1d/c48 to d12/d50/c7a 0 2024-04-05T21:26:42.232 INFO:tasks.workunit.client.0.smithi121.stdout:6/340: mknod d1/d2/d3/d5/d1f/d2c/c6e 0 2024-04-05T21:26:42.232 INFO:tasks.workunit.client.0.smithi121.stdout:6/341: stat d1/d2/d1b/d1e/c3f 0 2024-04-05T21:26:42.233 INFO:tasks.workunit.client.1.smithi188.stdout:7/857: mknod d3/d4/d101/c115 0 2024-04-05T21:26:42.234 INFO:tasks.workunit.client.0.smithi121.stdout:8/361: getdents d12/d13/d56/d6d 0 2024-04-05T21:26:42.235 INFO:tasks.workunit.client.0.smithi121.stdout:8/362: stat d12/d13/d15/d1d/l71 0 2024-04-05T21:26:42.237 INFO:tasks.workunit.client.0.smithi121.stdout:8/363: readlink d12/d13/d15/d1d/d29/l2c 0 2024-04-05T21:26:42.237 INFO:tasks.workunit.client.0.smithi121.stdout:6/342: unlink d1/d2/d3/d5/d1f/c40 0 2024-04-05T21:26:42.238 INFO:tasks.workunit.client.0.smithi121.stdout:6/343: truncate d1/d2/d3/d5/d1f/f6a 605592 0 2024-04-05T21:26:42.239 INFO:tasks.workunit.client.1.smithi188.stdout:7/858: link d3/fea d3/d4/f116 0 2024-04-05T21:26:42.241 INFO:tasks.workunit.client.0.smithi121.stdout:8/364: creat d12/d13/d56/d57/f7b x:0 0 0 2024-04-05T21:26:42.243 INFO:tasks.workunit.client.1.smithi188.stdout:7/859: mknod d3/d2c/dce/c117 0 2024-04-05T21:26:42.243 INFO:tasks.workunit.client.0.smithi121.stdout:6/344: rename d1/d4b/d68 to d1/d2/d3/d5/d1f/d2c/d4f/d6f 0 2024-04-05T21:26:42.244 INFO:tasks.workunit.client.1.smithi188.stdout:7/860: chown d3/d2c/dae/dfd/d10/dbe/ded/f110 2251 1 2024-04-05T21:26:42.245 INFO:tasks.workunit.client.1.smithi188.stdout:7/861: rmdir d3/d4/d1d/d74 39 2024-04-05T21:26:42.247 INFO:tasks.workunit.client.1.smithi188.stdout:7/862: mknod d3/d2c/dae/dfd/d10/d5f/dc6/c118 0 2024-04-05T21:26:42.248 INFO:tasks.workunit.client.0.smithi121.stdout:7/465: dread d0/d23/d5b/f4b [0,4194304] 0 2024-04-05T21:26:42.248 INFO:tasks.workunit.client.1.smithi188.stdout:7/863: readlink d3/d4/d1d/d74/l45 0 2024-04-05T21:26:42.249 INFO:tasks.workunit.client.0.smithi121.stdout:6/345: write d1/d2/d3/d4/d21/d57/d42/f67 [633326,114519] 0 2024-04-05T21:26:42.249 INFO:tasks.workunit.client.0.smithi121.stdout:8/365: getdents d12/d50/d6a/d63 0 2024-04-05T21:26:42.250 INFO:tasks.workunit.client.1.smithi188.stdout:7/864: symlink d3/d4/d1d/l119 0 2024-04-05T21:26:42.251 INFO:tasks.workunit.client.0.smithi121.stdout:7/466: chown d0/d23/d5b/f80 18883 1 2024-04-05T21:26:42.253 INFO:tasks.workunit.client.1.smithi188.stdout:7/865: creat d3/d2c/dae/dfd/d10/d5f/dc6/d104/f11a x:0 0 0 2024-04-05T21:26:42.253 INFO:tasks.workunit.client.0.smithi121.stdout:2/404: dwrite d2/d9/d1c/d30/f68 [0,4194304] 0 2024-04-05T21:26:42.255 INFO:tasks.workunit.client.1.smithi188.stdout:7/866: unlink d3/d2c/dce/c117 0 2024-04-05T21:26:42.255 INFO:tasks.workunit.client.0.smithi121.stdout:7/467: fdatasync d0/d23/d5b/d52/d61/f89 0 2024-04-05T21:26:42.255 INFO:tasks.workunit.client.0.smithi121.stdout:2/405: chown d2/d9/d1c/d30/f68 122 1 2024-04-05T21:26:42.258 INFO:tasks.workunit.client.0.smithi121.stdout:2/406: write d2/d9/d38/f35 [4860692,42970] 0 2024-04-05T21:26:42.260 INFO:tasks.workunit.client.0.smithi121.stdout:1/492: dread d8/d10/d14/d1a/d1b/d4c/d76/f3d [0,4194304] 0 2024-04-05T21:26:42.261 INFO:tasks.workunit.client.0.smithi121.stdout:6/346: link d1/c25 d1/d32/c70 0 2024-04-05T21:26:42.263 INFO:tasks.workunit.client.0.smithi121.stdout:1/493: symlink d8/d10/d14/d27/d2d/l9d 0 2024-04-05T21:26:42.264 INFO:tasks.workunit.client.0.smithi121.stdout:6/347: rmdir d1/d32 39 2024-04-05T21:26:42.264 INFO:tasks.workunit.client.0.smithi121.stdout:1/494: chown d8/d10/d14/d1a/c64 21823 1 2024-04-05T21:26:42.265 INFO:tasks.workunit.client.0.smithi121.stdout:2/407: creat d2/d9/f75 x:0 0 0 2024-04-05T21:26:42.266 INFO:tasks.workunit.client.0.smithi121.stdout:7/468: getdents d0/d23/d5b/d78 0 2024-04-05T21:26:42.266 INFO:tasks.workunit.client.0.smithi121.stdout:2/408: readlink d2/l4 0 2024-04-05T21:26:42.268 INFO:tasks.workunit.client.0.smithi121.stdout:3/330: sync 2024-04-05T21:26:42.268 INFO:tasks.workunit.client.0.smithi121.stdout:2/409: chown d2/d9/d38/d6d/l74 78268726 1 2024-04-05T21:26:42.270 INFO:tasks.workunit.client.0.smithi121.stdout:2/410: dread d2/d9/d1c/d30/f5e [0,4194304] 0 2024-04-05T21:26:42.271 INFO:tasks.workunit.client.0.smithi121.stdout:3/331: symlink d0/d3/d24/d3d/d1a/d1e/l73 0 2024-04-05T21:26:42.272 INFO:tasks.workunit.client.0.smithi121.stdout:3/332: chown d0/d3/f4 352 1 2024-04-05T21:26:42.273 INFO:tasks.workunit.client.0.smithi121.stdout:7/469: link d0/d23/d5a/f7b d0/d32/f90 0 2024-04-05T21:26:42.274 INFO:tasks.workunit.client.0.smithi121.stdout:2/411: getdents d2/d9/d12 0 2024-04-05T21:26:42.276 INFO:tasks.workunit.client.0.smithi121.stdout:7/470: chown d0/d9/c1a 0 1 2024-04-05T21:26:42.278 INFO:tasks.workunit.client.1.smithi188.stdout:7/867: truncate d3/da5/fd3 1563703 0 2024-04-05T21:26:42.278 INFO:tasks.workunit.client.0.smithi121.stdout:8/366: write ff [3749527,65805] 0 2024-04-05T21:26:42.280 INFO:tasks.workunit.client.0.smithi121.stdout:7/471: rmdir d0/d23/d5b 39 2024-04-05T21:26:42.280 INFO:tasks.workunit.client.1.smithi188.stdout:7/868: truncate d3/d2c/dae/dfd/d2e/d3b/f92 657663 0 2024-04-05T21:26:42.282 INFO:tasks.workunit.client.0.smithi121.stdout:3/333: dwrite d0/d3/f5e [0,4194304] 0 2024-04-05T21:26:42.282 INFO:tasks.workunit.client.1.smithi188.stdout:7/869: creat d3/d4/d1d/d74/d88/f11b x:0 0 0 2024-04-05T21:26:42.283 INFO:tasks.workunit.client.0.smithi121.stdout:8/367: chown d12/d13/d15/d1d/d29/f31 880 1 2024-04-05T21:26:42.283 INFO:tasks.workunit.client.0.smithi121.stdout:1/495: rename d8/d10/d14/d27 to d8/d89/d9e 0 2024-04-05T21:26:42.284 INFO:tasks.workunit.client.1.smithi188.stdout:7/870: symlink d3/d2c/dae/dfd/d2e/l11c 0 2024-04-05T21:26:42.285 INFO:tasks.workunit.client.1.smithi188.stdout:7/871: truncate d3/d4/d1d/d74/d73/fcb 77400 0 2024-04-05T21:26:42.285 INFO:tasks.workunit.client.0.smithi121.stdout:8/368: write f3 [1656648,89012] 0 2024-04-05T21:26:42.286 INFO:tasks.workunit.client.1.smithi188.stdout:7/872: chown d3/d4/d1d/d74/f80 1 1 2024-04-05T21:26:42.288 INFO:tasks.workunit.client.1.smithi188.stdout:7/873: fsync d3/d2c/dce/fad 0 2024-04-05T21:26:42.288 INFO:tasks.workunit.client.0.smithi121.stdout:2/412: dwrite d2/d9/d17/f42 [0,4194304] 0 2024-04-05T21:26:42.289 INFO:tasks.workunit.client.1.smithi188.stdout:7/874: dread - d3/d2c/dae/dfd/d55/f10b zero size 2024-04-05T21:26:42.290 INFO:tasks.workunit.client.0.smithi121.stdout:6/348: rename d1/d2/d3/d59/l56 to d1/d2/d1b/d35/d51/l71 0 2024-04-05T21:26:42.290 INFO:tasks.workunit.client.1.smithi188.stdout:7/875: write d3/d4/d7c/ff0 [807445,89069] 0 2024-04-05T21:26:42.291 INFO:tasks.workunit.client.1.smithi188.stdout:7/876: chown d3/d2c/dce 32 1 2024-04-05T21:26:42.293 INFO:tasks.workunit.client.1.smithi188.stdout:7/877: mkdir d3/d4/d1d/d11d 0 2024-04-05T21:26:42.297 INFO:tasks.workunit.client.0.smithi121.stdout:6/349: write d1/d2/d1b/d1e/f2d [3916096,92352] 0 2024-04-05T21:26:42.297 INFO:tasks.workunit.client.0.smithi121.stdout:7/472: mknod d0/d23/d5b/d52/c91 0 2024-04-05T21:26:42.297 INFO:tasks.workunit.client.0.smithi121.stdout:6/350: read d1/f38 [1195562,46653] 0 2024-04-05T21:26:42.300 INFO:tasks.workunit.client.0.smithi121.stdout:6/351: chown d1/d2/d3/d4/d21/d57/d42/c63 42 1 2024-04-05T21:26:42.301 INFO:tasks.workunit.client.1.smithi188.stdout:7/878: dwrite d3/d2c/dce/ff8 [0,4194304] 0 2024-04-05T21:26:42.301 INFO:tasks.workunit.client.0.smithi121.stdout:3/334: dwrite d0/d3/f30 [0,4194304] 0 2024-04-05T21:26:42.301 INFO:tasks.workunit.client.0.smithi121.stdout:1/496: getdents d8/d10 0 2024-04-05T21:26:42.302 INFO:tasks.workunit.client.0.smithi121.stdout:7/473: write d0/d9/d22/d28/d3b/f6f [168999,128075] 0 2024-04-05T21:26:42.303 INFO:tasks.workunit.client.0.smithi121.stdout:3/335: stat d0/d3/de 0 2024-04-05T21:26:42.306 INFO:tasks.workunit.client.0.smithi121.stdout:1/497: truncate d8/d10/d14/d1a/d1b/f69 4917279 0 2024-04-05T21:26:42.307 INFO:tasks.workunit.client.0.smithi121.stdout:3/336: mknod d0/d3/d24/d3f/c74 0 2024-04-05T21:26:42.313 INFO:tasks.workunit.client.0.smithi121.stdout:3/337: mknod d0/d3/de/c75 0 2024-04-05T21:26:42.314 INFO:tasks.workunit.client.0.smithi121.stdout:6/352: read d1/d2/d1b/d1e/f2d [789469,9502] 0 2024-04-05T21:26:42.315 INFO:tasks.workunit.client.0.smithi121.stdout:8/369: truncate d12/d13/d15/d1d/f42 3486742 0 2024-04-05T21:26:42.315 INFO:tasks.workunit.client.0.smithi121.stdout:3/338: chown d0/d3/d24/d3d/d1a/d23/d38/c6c 1728746363 1 2024-04-05T21:26:42.316 INFO:tasks.workunit.client.0.smithi121.stdout:8/370: fdatasync d12/f73 0 2024-04-05T21:26:42.316 INFO:tasks.workunit.client.1.smithi188.stdout:7/879: write d3/d2c/dae/f97 [350857,57860] 0 2024-04-05T21:26:42.317 INFO:tasks.workunit.client.0.smithi121.stdout:1/498: rename d8/d10/d14/d1a/d1b/l3f to d8/d10/d14/l9f 0 2024-04-05T21:26:42.317 INFO:tasks.workunit.client.1.smithi188.stdout:7/880: stat d3/d2c/dae/dfd/d2e/d37/fc2 0 2024-04-05T21:26:42.318 INFO:tasks.workunit.client.0.smithi121.stdout:6/353: creat d1/f72 x:0 0 0 2024-04-05T21:26:42.320 INFO:tasks.workunit.client.0.smithi121.stdout:8/371: rename d12/d13/l37 to d12/d13/d56/d6d/l7c 0 2024-04-05T21:26:42.320 INFO:tasks.workunit.client.0.smithi121.stdout:3/339: rename d0/d3/d24/d3d to d0/d3/d24/d3d/d1a/d23/d38/d55/d76 22 2024-04-05T21:26:42.321 INFO:tasks.workunit.client.0.smithi121.stdout:1/499: creat d8/d10/d14/d1a/d1b/d85/fa0 x:0 0 0 2024-04-05T21:26:42.322 INFO:tasks.workunit.client.0.smithi121.stdout:8/372: fdatasync d12/d13/d15/d1d/d29/f40 0 2024-04-05T21:26:42.323 INFO:tasks.workunit.client.0.smithi121.stdout:3/340: truncate d0/d3/d24/d3d/f3a 305239 0 2024-04-05T21:26:42.325 INFO:tasks.workunit.client.1.smithi188.stdout:7/881: write d3/d2c/d5e/f98 [4209742,93348] 0 2024-04-05T21:26:42.325 INFO:tasks.workunit.client.0.smithi121.stdout:1/500: unlink d8/d10/d14/d1a/d1b/d44/f93 0 2024-04-05T21:26:42.327 INFO:tasks.workunit.client.1.smithi188.stdout:7/882: creat d3/d2c/dae/dfd/d10/dbe/f11e x:0 0 0 2024-04-05T21:26:42.328 INFO:tasks.workunit.client.1.smithi188.stdout:7/883: fdatasync d3/f68 0 2024-04-05T21:26:42.329 INFO:tasks.workunit.client.1.smithi188.stdout:7/884: dread - d3/d2c/d5e/ff1 zero size 2024-04-05T21:26:42.331 INFO:tasks.workunit.client.0.smithi121.stdout:1/501: write d8/d10/d14/d1a/d1b/d4c/d76/f3d [2881584,27350] 0 2024-04-05T21:26:42.332 INFO:tasks.workunit.client.1.smithi188.stdout:7/885: unlink d3/d4/d1d/da3/cf7 0 2024-04-05T21:26:42.332 INFO:tasks.workunit.client.1.smithi188.stdout:7/886: write d3/f3e [614980,37553] 0 2024-04-05T21:26:42.332 INFO:tasks.workunit.client.0.smithi121.stdout:3/341: dread d0/d3/de/f17 [0,4194304] 0 2024-04-05T21:26:42.333 INFO:tasks.workunit.client.0.smithi121.stdout:1/502: fdatasync d8/d10/d14/f19 0 2024-04-05T21:26:42.336 INFO:tasks.workunit.client.0.smithi121.stdout:6/354: fsync d1/d2/d1b/d1e/f2a 0 2024-04-05T21:26:42.341 INFO:tasks.workunit.client.0.smithi121.stdout:3/342: getdents d0/d3/d24/d3d/d1a/d1e 0 2024-04-05T21:26:42.342 INFO:tasks.workunit.client.0.smithi121.stdout:6/355: mknod d1/d2/d1b/d35/d39/c73 0 2024-04-05T21:26:42.343 INFO:tasks.workunit.client.0.smithi121.stdout:1/503: dwrite d8/d10/d14/d1a/d1b/d85/fa0 [0,4194304] 0 2024-04-05T21:26:42.345 INFO:tasks.workunit.client.0.smithi121.stdout:6/356: truncate d1/d2/d3/d5/f58 586938 0 2024-04-05T21:26:42.347 INFO:tasks.workunit.client.0.smithi121.stdout:8/373: dread d12/d13/d15/d1d/f42 [0,4194304] 0 2024-04-05T21:26:42.348 INFO:tasks.workunit.client.0.smithi121.stdout:1/504: mkdir d8/d10/d14/d1a/d1b/d4c/da1 0 2024-04-05T21:26:42.348 INFO:tasks.workunit.client.0.smithi121.stdout:8/374: chown d12/d13/d15/d1d/f42 442 1 2024-04-05T21:26:42.349 INFO:tasks.workunit.client.0.smithi121.stdout:3/343: truncate d0/f15 1468966 0 2024-04-05T21:26:42.350 INFO:tasks.workunit.client.0.smithi121.stdout:6/357: mkdir d1/d2/d1b/d35/d74 0 2024-04-05T21:26:42.355 INFO:tasks.workunit.client.0.smithi121.stdout:1/505: rename d8/d89/d9e/d5c to d8/d8c/da2 0 2024-04-05T21:26:42.357 INFO:tasks.workunit.client.0.smithi121.stdout:3/344: getdents d0/d3/d24/d3f 0 2024-04-05T21:26:42.358 INFO:tasks.workunit.client.0.smithi121.stdout:6/358: truncate d1/d2/d3/f4e 2525539 0 2024-04-05T21:26:42.362 INFO:tasks.workunit.client.0.smithi121.stdout:8/375: dwrite d12/d13/d56/d6d/f72 [0,4194304] 0 2024-04-05T21:26:42.365 INFO:tasks.workunit.client.0.smithi121.stdout:1/506: rename d8/d10/d14/d1a/d1b/d4c/d8d to d8/d89/d9e/da3 0 2024-04-05T21:26:42.368 INFO:tasks.workunit.client.0.smithi121.stdout:3/345: dread d0/d3/de/f72 [0,4194304] 0 2024-04-05T21:26:42.370 INFO:tasks.workunit.client.0.smithi121.stdout:6/359: rename d1/d2/d3/d4/de/l61 to d1/d4b/l75 0 2024-04-05T21:26:42.373 INFO:tasks.workunit.client.0.smithi121.stdout:1/507: mkdir d8/d10/d14/d1a/d1b/d4c/d76/da4 0 2024-04-05T21:26:42.374 INFO:tasks.workunit.client.0.smithi121.stdout:1/508: chown d8/d89/d9e/f75 3416 1 2024-04-05T21:26:42.375 INFO:tasks.workunit.client.0.smithi121.stdout:3/346: unlink d0/f2 0 2024-04-05T21:26:42.377 INFO:tasks.workunit.client.0.smithi121.stdout:8/376: creat d12/f7d x:0 0 0 2024-04-05T21:26:42.377 INFO:tasks.workunit.client.0.smithi121.stdout:6/360: mknod d1/d4b/c76 0 2024-04-05T21:26:42.379 INFO:tasks.workunit.client.0.smithi121.stdout:1/509: creat d8/d89/fa5 x:0 0 0 2024-04-05T21:26:42.380 INFO:tasks.workunit.client.0.smithi121.stdout:7/474: dread d0/d32/f83 [0,4194304] 0 2024-04-05T21:26:42.382 INFO:tasks.workunit.client.0.smithi121.stdout:8/377: dread d12/d50/d6a/f45 [0,4194304] 0 2024-04-05T21:26:42.383 INFO:tasks.workunit.client.0.smithi121.stdout:6/361: creat d1/d4b/f77 x:0 0 0 2024-04-05T21:26:42.385 INFO:tasks.workunit.client.0.smithi121.stdout:3/347: dwrite d0/d3/d24/d3f/f4b [0,4194304] 0 2024-04-05T21:26:42.385 INFO:tasks.workunit.client.0.smithi121.stdout:6/362: stat d1/d2/d1b/d1e/c3f 0 2024-04-05T21:26:42.386 INFO:tasks.workunit.client.0.smithi121.stdout:9/305: sync 2024-04-05T21:26:42.386 INFO:tasks.workunit.client.0.smithi121.stdout:0/507: sync 2024-04-05T21:26:42.388 INFO:tasks.workunit.client.0.smithi121.stdout:8/378: read d12/d13/d15/d1d/f58 [794411,117770] 0 2024-04-05T21:26:42.388 INFO:tasks.workunit.client.0.smithi121.stdout:4/570: sync 2024-04-05T21:26:42.388 INFO:tasks.workunit.client.0.smithi121.stdout:7/475: mknod d0/d32/d4e/c92 0 2024-04-05T21:26:42.390 INFO:tasks.workunit.client.0.smithi121.stdout:0/508: write d7/d71/f26 [2857266,98474] 0 2024-04-05T21:26:42.390 INFO:tasks.workunit.client.0.smithi121.stdout:9/306: chown d6/df/d1d/d2c/d37/d3c 82092730 1 2024-04-05T21:26:42.390 INFO:tasks.workunit.client.0.smithi121.stdout:4/571: fsync d5/d11/d2c/f65 0 2024-04-05T21:26:42.391 INFO:tasks.workunit.client.0.smithi121.stdout:3/348: creat d0/d3/d24/d3d/d1a/d23/d38/d55/f77 x:0 0 0 2024-04-05T21:26:42.395 INFO:tasks.workunit.client.0.smithi121.stdout:7/476: creat d0/d32/d7c/f93 x:0 0 0 2024-04-05T21:26:42.396 INFO:tasks.workunit.client.0.smithi121.stdout:9/307: creat d6/df/d48/f62 x:0 0 0 2024-04-05T21:26:42.397 INFO:tasks.workunit.client.0.smithi121.stdout:0/509: rmdir d7/d71/d45/d46/d53/d8e 39 2024-04-05T21:26:42.398 INFO:tasks.workunit.client.0.smithi121.stdout:3/349: readlink d0/d3/d24/d3d/d1a/d1e/l6e 0 2024-04-05T21:26:42.404 INFO:tasks.workunit.client.0.smithi121.stdout:7/477: symlink d0/d9/l94 0 2024-04-05T21:26:42.405 INFO:tasks.workunit.client.0.smithi121.stdout:6/363: dwrite d1/d2/d1b/d1e/f2a [0,4194304] 0 2024-04-05T21:26:42.411 INFO:tasks.workunit.client.0.smithi121.stdout:4/572: dread d5/d11/d2c/d9d/f31 [0,4194304] 0 2024-04-05T21:26:42.414 INFO:tasks.workunit.client.1.smithi188.stdout:2/711: sync 2024-04-05T21:26:42.414 INFO:tasks.workunit.client.1.smithi188.stdout:0/550: sync 2024-04-05T21:26:42.415 INFO:tasks.workunit.client.0.smithi121.stdout:3/350: getdents d0/d3/d24/d3d/d1a/d23/d38 0 2024-04-05T21:26:42.416 INFO:tasks.workunit.client.0.smithi121.stdout:9/308: link d6/df/l1b d6/l63 0 2024-04-05T21:26:42.416 INFO:tasks.workunit.client.0.smithi121.stdout:4/573: readlink d5/d11/d2c/d9d/dae/d29/la8 0 2024-04-05T21:26:42.419 INFO:tasks.workunit.client.0.smithi121.stdout:6/364: chown d1/d2/d3/d5/d1f/d2c/d4f/c5c 1 1 2024-04-05T21:26:42.419 INFO:tasks.workunit.client.0.smithi121.stdout:9/309: chown d6/df/d1d/f2d 47285 1 2024-04-05T21:26:42.420 INFO:tasks.workunit.client.0.smithi121.stdout:7/478: dwrite d0/d23/f8b [0,4194304] 0 2024-04-05T21:26:42.421 INFO:tasks.workunit.client.1.smithi188.stdout:0/551: dwrite f3 [4194304,4194304] 0 2024-04-05T21:26:42.424 INFO:tasks.workunit.client.1.smithi188.stdout:0/552: mkdir d4/d6/d13/dca 0 2024-04-05T21:26:42.424 INFO:tasks.workunit.client.0.smithi121.stdout:8/379: dread d12/d13/d15/f19 [0,4194304] 0 2024-04-05T21:26:42.425 INFO:tasks.workunit.client.0.smithi121.stdout:3/351: write d0/d3/d24/f3c [299459,43565] 0 2024-04-05T21:26:42.428 INFO:tasks.workunit.client.1.smithi188.stdout:2/712: write d5/db/dd/d36/d3c/f4a [3047065,100575] 0 2024-04-05T21:26:42.429 INFO:tasks.workunit.client.0.smithi121.stdout:4/574: dread d5/d11/d60/f9f [0,4194304] 0 2024-04-05T21:26:42.429 INFO:tasks.workunit.client.0.smithi121.stdout:1/510: dread d8/d10/d14/d1a/d1b/f69 [0,4194304] 0 2024-04-05T21:26:42.433 INFO:tasks.workunit.client.1.smithi188.stdout:0/553: dwrite d4/d8/d10/f6d [0,4194304] 0 2024-04-05T21:26:42.433 INFO:tasks.workunit.client.1.smithi188.stdout:2/713: truncate d5/db/dd/d67/dcd/faf 677313 0 2024-04-05T21:26:42.433 INFO:tasks.workunit.client.0.smithi121.stdout:3/352: stat d0/d3/d24/d3d/d1a/d23/d38/d55/c63 0 2024-04-05T21:26:42.434 INFO:tasks.workunit.client.0.smithi121.stdout:9/310: chown d6/df/d1d/d2c/l30 4539542 1 2024-04-05T21:26:42.434 INFO:tasks.workunit.client.0.smithi121.stdout:7/479: dread d0/f42 [0,4194304] 0 2024-04-05T21:26:42.436 INFO:tasks.workunit.client.0.smithi121.stdout:6/365: symlink d1/d32/l78 0 2024-04-05T21:26:42.436 INFO:tasks.workunit.client.0.smithi121.stdout:7/480: truncate d0/d32/f65 414130 0 2024-04-05T21:26:42.442 INFO:tasks.workunit.client.1.smithi188.stdout:2/714: dwrite d5/d5f/d98/fd2 [0,4194304] 0 2024-04-05T21:26:42.444 INFO:tasks.workunit.client.0.smithi121.stdout:8/380: dread d12/d50/d6a/f65 [0,4194304] 0 2024-04-05T21:26:42.445 INFO:tasks.workunit.client.0.smithi121.stdout:6/366: mknod d1/d2/d1b/c79 0 2024-04-05T21:26:42.445 INFO:tasks.workunit.client.1.smithi188.stdout:2/715: creat d5/fe7 x:0 0 0 2024-04-05T21:26:42.445 INFO:tasks.workunit.client.1.smithi188.stdout:2/716: readlink d5/db/dd/d36/d3c/d48/l64 0 2024-04-05T21:26:42.446 INFO:tasks.workunit.client.0.smithi121.stdout:7/481: write d0/f3 [2069272,105862] 0 2024-04-05T21:26:42.450 INFO:tasks.workunit.client.0.smithi121.stdout:7/482: chown d0/f15 168 1 2024-04-05T21:26:42.452 INFO:tasks.workunit.client.1.smithi188.stdout:0/554: dwrite d4/d8/f75 [0,4194304] 0 2024-04-05T21:26:42.456 INFO:tasks.workunit.client.0.smithi121.stdout:3/353: dwrite d0/d3/f5b [0,4194304] 0 2024-04-05T21:26:42.456 INFO:tasks.workunit.client.1.smithi188.stdout:0/555: unlink d4/d8/fa 0 2024-04-05T21:26:42.457 INFO:tasks.workunit.client.0.smithi121.stdout:0/510: dwrite d7/d7e/f89 [0,4194304] 0 2024-04-05T21:26:42.460 INFO:tasks.workunit.client.1.smithi188.stdout:0/556: mkdir d4/d8/db/dcb 0 2024-04-05T21:26:42.462 INFO:tasks.workunit.client.0.smithi121.stdout:6/367: rename d1/d2/d3/d4/l23 to d1/d2/d1b/d2e/l7a 0 2024-04-05T21:26:42.462 INFO:tasks.workunit.client.0.smithi121.stdout:7/483: read d0/d23/d5b/f4b [366065,1619] 0 2024-04-05T21:26:42.462 INFO:tasks.workunit.client.0.smithi121.stdout:1/511: dwrite d8/d10/d14/d1a/d1b/d4c/f4d [0,4194304] 0 2024-04-05T21:26:42.464 INFO:tasks.workunit.client.1.smithi188.stdout:0/557: truncate d4/d8/db/d21/d4a/d73/d79/fb5 608515 0 2024-04-05T21:26:42.465 INFO:tasks.workunit.client.0.smithi121.stdout:8/381: chown d12/d13/d15/d1d/c4e 0 1 2024-04-05T21:26:42.468 INFO:tasks.workunit.client.0.smithi121.stdout:1/512: chown d8/d10/d14/d1a/d1b/d4c/d76/f3d 28 1 2024-04-05T21:26:42.469 INFO:tasks.workunit.client.0.smithi121.stdout:0/511: rename d7/d71/d29/d2a/d64/fa5 to d7/dc/da3/fb1 0 2024-04-05T21:26:42.469 INFO:tasks.workunit.client.0.smithi121.stdout:3/354: symlink d0/d3/d24/d3d/d1a/l78 0 2024-04-05T21:26:42.475 INFO:tasks.workunit.client.0.smithi121.stdout:1/513: write d8/d8c/f8e [1285733,99532] 0 2024-04-05T21:26:42.477 INFO:tasks.workunit.client.0.smithi121.stdout:8/382: truncate d12/d50/d6a/f60 288303 0 2024-04-05T21:26:42.478 INFO:tasks.workunit.client.1.smithi188.stdout:0/558: dread d4/d6/d46/f59 [0,4194304] 0 2024-04-05T21:26:42.478 INFO:tasks.workunit.client.1.smithi188.stdout:0/559: chown d4/d6/c2c 29104549 1 2024-04-05T21:26:42.481 INFO:tasks.workunit.client.0.smithi121.stdout:6/368: rename d1/d2/d3/c44 to d1/d2/d3/d5/d1f/c7b 0 2024-04-05T21:26:42.482 INFO:tasks.workunit.client.0.smithi121.stdout:6/369: chown d1/d2/d3/d4/d21 24855 1 2024-04-05T21:26:42.483 INFO:tasks.workunit.client.0.smithi121.stdout:7/484: dwrite d0/d23/d5a/f63 [0,4194304] 0 2024-04-05T21:26:42.484 INFO:tasks.workunit.client.0.smithi121.stdout:1/514: mknod d8/d10/d14/d1a/d1b/d44/ca6 0 2024-04-05T21:26:42.486 INFO:tasks.workunit.client.0.smithi121.stdout:4/575: dread f4 [0,4194304] 0 2024-04-05T21:26:42.487 INFO:tasks.workunit.client.0.smithi121.stdout:7/485: stat d0/d9/d22/d3e/l4d 0 2024-04-05T21:26:42.490 INFO:tasks.workunit.client.0.smithi121.stdout:6/370: chown d1/d2/d1b/d35/d39/c73 1396 1 2024-04-05T21:26:42.490 INFO:tasks.workunit.client.0.smithi121.stdout:3/355: dwrite d0/d3/f5b [0,4194304] 0 2024-04-05T21:26:42.492 INFO:tasks.workunit.client.0.smithi121.stdout:9/311: dread d6/fc [0,4194304] 0 2024-04-05T21:26:42.499 INFO:tasks.workunit.client.1.smithi188.stdout:0/560: dread d4/d8/db/d42/d38/f54 [4194304,4194304] 0 2024-04-05T21:26:42.500 INFO:tasks.workunit.client.0.smithi121.stdout:6/371: mkdir d1/d2/d3/d4/d21/d57/d42/d48/d7c 0 2024-04-05T21:26:42.502 INFO:tasks.workunit.client.1.smithi188.stdout:0/561: mkdir d4/d6/d13/d23/dcc 0 2024-04-05T21:26:42.504 INFO:tasks.workunit.client.0.smithi121.stdout:1/515: dwrite d8/d10/d14/d1a/f5a [0,4194304] 0 2024-04-05T21:26:42.504 INFO:tasks.workunit.client.1.smithi188.stdout:0/562: symlink d4/d8/db/d21/d4a/d73/lcd 0 2024-04-05T21:26:42.505 INFO:tasks.workunit.client.0.smithi121.stdout:7/486: creat d0/d9/f95 x:0 0 0 2024-04-05T21:26:42.505 INFO:tasks.workunit.client.1.smithi188.stdout:0/563: read d4/d6/d13/f14 [8020287,38143] 0 2024-04-05T21:26:42.507 INFO:tasks.workunit.client.1.smithi188.stdout:0/564: mknod d4/d6/d13/d23/d24/d57/cce 0 2024-04-05T21:26:42.509 INFO:tasks.workunit.client.1.smithi188.stdout:0/565: read d4/d8/d10/f40 [3780773,125832] 0 2024-04-05T21:26:42.509 INFO:tasks.workunit.client.1.smithi188.stdout:0/566: chown d4 21273157 1 2024-04-05T21:26:42.511 INFO:tasks.workunit.client.1.smithi188.stdout:0/567: write d4/d6/d13/d23/d24/f81 [1216928,70606] 0 2024-04-05T21:26:42.512 INFO:tasks.workunit.client.1.smithi188.stdout:0/568: fsync d4/d8/db/d42/d38/d77/f95 0 2024-04-05T21:26:42.513 INFO:tasks.workunit.client.0.smithi121.stdout:1/516: symlink d8/d89/la7 0 2024-04-05T21:26:42.513 INFO:tasks.workunit.client.0.smithi121.stdout:3/356: dwrite d0/d3/d24/d3d/f3a [0,4194304] 0 2024-04-05T21:26:42.514 INFO:tasks.workunit.client.0.smithi121.stdout:8/383: dread fa [0,4194304] 0 2024-04-05T21:26:42.515 INFO:tasks.workunit.client.0.smithi121.stdout:6/372: rename d1/d2/dc/f3e to d1/d2/d1b/d35/d74/f7d 0 2024-04-05T21:26:42.517 INFO:tasks.workunit.client.0.smithi121.stdout:9/312: dwrite d6/f15 [0,4194304] 0 2024-04-05T21:26:42.525 INFO:tasks.workunit.client.0.smithi121.stdout:3/357: symlink d0/l79 0 2024-04-05T21:26:42.525 INFO:tasks.workunit.client.0.smithi121.stdout:1/517: unlink d8/d10/d14/d1a/d1b/c28 0 2024-04-05T21:26:42.526 INFO:tasks.workunit.client.0.smithi121.stdout:6/373: symlink d1/d2/d3/d4/d21/d57/d42/d48/l7e 0 2024-04-05T21:26:42.527 INFO:tasks.workunit.client.0.smithi121.stdout:9/313: write d6/df/d1d/f2d [1835058,11082] 0 2024-04-05T21:26:42.530 INFO:tasks.workunit.client.1.smithi188.stdout:0/569: dwrite d4/d8/d10/f1b [0,4194304] 0 2024-04-05T21:26:42.532 INFO:tasks.workunit.client.0.smithi121.stdout:0/512: dwrite d7/d71/d29/d2a/d64/f83 [0,4194304] 0 2024-04-05T21:26:42.533 INFO:tasks.workunit.client.0.smithi121.stdout:1/518: chown d8/d10/d14/d1a/l82 30085035 1 2024-04-05T21:26:42.533 INFO:tasks.workunit.client.0.smithi121.stdout:4/576: dwrite d5/d11/d2c/d9d/dae/f30 [0,4194304] 0 2024-04-05T21:26:42.533 INFO:tasks.workunit.client.0.smithi121.stdout:9/314: fsync f2 0 2024-04-05T21:26:42.535 INFO:tasks.workunit.client.1.smithi188.stdout:0/570: link d4/d8/db/d21/d4a/f58 d4/d6/d46/fcf 0 2024-04-05T21:26:42.536 INFO:tasks.workunit.client.0.smithi121.stdout:4/577: chown d5/d11/d28 30126 1 2024-04-05T21:26:42.537 INFO:tasks.workunit.client.1.smithi188.stdout:4/913: sync 2024-04-05T21:26:42.537 INFO:tasks.workunit.client.1.smithi188.stdout:9/694: sync 2024-04-05T21:26:42.537 INFO:tasks.workunit.client.1.smithi188.stdout:8/621: sync 2024-04-05T21:26:42.538 INFO:tasks.workunit.client.0.smithi121.stdout:0/513: write d7/dc/faf [562099,463] 0 2024-04-05T21:26:42.538 INFO:tasks.workunit.client.0.smithi121.stdout:8/384: dwrite fc [4194304,4194304] 0 2024-04-05T21:26:42.538 INFO:tasks.workunit.client.1.smithi188.stdout:0/571: creat d4/d8/d39/fd0 x:0 0 0 2024-04-05T21:26:42.539 INFO:tasks.workunit.client.0.smithi121.stdout:1/519: write d8/d10/d14/f8f [981079,124801] 0 2024-04-05T21:26:42.540 INFO:tasks.workunit.client.1.smithi188.stdout:4/914: stat d0/d14/d15/d53/d12e/d12f 0 2024-04-05T21:26:42.540 INFO:tasks.workunit.client.1.smithi188.stdout:8/622: write d5/d6e/d86/dcb/f5f [654482,201] 0 2024-04-05T21:26:42.542 INFO:tasks.workunit.client.1.smithi188.stdout:4/915: write d0/d14/d15/d78/db1/f7b [163387,123654] 0 2024-04-05T21:26:42.545 INFO:tasks.workunit.client.1.smithi188.stdout:0/572: unlink d4/d8/db/d42/d71/faf 0 2024-04-05T21:26:42.545 INFO:tasks.workunit.client.1.smithi188.stdout:9/695: mknod dd/d11/d1f/d26/d55/cf6 0 2024-04-05T21:26:42.547 INFO:tasks.workunit.client.1.smithi188.stdout:9/696: rmdir dd/d11/d73 39 2024-04-05T21:26:42.548 INFO:tasks.workunit.client.0.smithi121.stdout:9/315: mknod d6/df/d1d/d2c/d37/d3c/c64 0 2024-04-05T21:26:42.548 INFO:tasks.workunit.client.0.smithi121.stdout:9/316: chown d6/df 2 1 2024-04-05T21:26:42.549 INFO:tasks.workunit.client.0.smithi121.stdout:1/520: readlink d8/d10/d14/d1a/d1b/l55 0 2024-04-05T21:26:42.549 INFO:tasks.workunit.client.1.smithi188.stdout:9/697: truncate dd/d3d/d89/fed 84999 0 2024-04-05T21:26:42.550 INFO:tasks.workunit.client.0.smithi121.stdout:4/578: chown d5/d11/d2c/d9d/dae/c7a 7169615 1 2024-04-05T21:26:42.551 INFO:tasks.workunit.client.0.smithi121.stdout:0/514: mknod d7/d1c/cb2 0 2024-04-05T21:26:42.551 INFO:tasks.workunit.client.0.smithi121.stdout:1/521: write d8/d10/d14/d1a/f5a [1859307,36723] 0 2024-04-05T21:26:42.553 INFO:tasks.workunit.client.0.smithi121.stdout:8/385: read d12/d13/f20 [270898,31129] 0 2024-04-05T21:26:42.555 INFO:tasks.workunit.client.0.smithi121.stdout:9/317: symlink d6/df/d1d/l65 0 2024-04-05T21:26:42.556 INFO:tasks.workunit.client.0.smithi121.stdout:1/522: read d8/d89/d9e/d2d/f41 [284239,27529] 0 2024-04-05T21:26:42.556 INFO:tasks.workunit.client.0.smithi121.stdout:6/374: link d1/c25 d1/d2/d3/d4/d21/d57/d42/d48/d7c/c7f 0 2024-04-05T21:26:42.558 INFO:tasks.workunit.client.0.smithi121.stdout:4/579: symlink d5/d45/d67/dab/lb8 0 2024-04-05T21:26:42.558 INFO:tasks.workunit.client.1.smithi188.stdout:9/698: unlink dd/d11/d1f/d26/d55/l64 0 2024-04-05T21:26:42.559 INFO:tasks.workunit.client.0.smithi121.stdout:0/515: mknod d7/d1c/cb3 0 2024-04-05T21:26:42.559 INFO:tasks.workunit.client.0.smithi121.stdout:4/580: stat d5/d45/d67/c70 0 2024-04-05T21:26:42.560 INFO:tasks.workunit.client.0.smithi121.stdout:8/386: symlink d12/d13/d15/l7e 0 2024-04-05T21:26:42.560 INFO:tasks.workunit.client.0.smithi121.stdout:8/387: chown d12/d13/d56/d6d/f72 115 1 2024-04-05T21:26:42.562 INFO:tasks.workunit.client.1.smithi188.stdout:9/699: symlink dd/d11/d1f/d26/d55/d98/da5/dea/lf7 0 2024-04-05T21:26:42.562 INFO:tasks.workunit.client.0.smithi121.stdout:1/523: rmdir d8/d10/d14/d1a/d1b/d4c 39 2024-04-05T21:26:42.563 INFO:tasks.workunit.client.0.smithi121.stdout:6/375: readlink d1/d2/d3/d5/d1f/l60 0 2024-04-05T21:26:42.564 INFO:tasks.workunit.client.0.smithi121.stdout:0/516: write d7/f3a [800132,56050] 0 2024-04-05T21:26:42.565 INFO:tasks.workunit.client.1.smithi188.stdout:9/700: mknod dd/d3d/d81/cf8 0 2024-04-05T21:26:42.566 INFO:tasks.workunit.client.1.smithi188.stdout:4/916: dwrite d0/d2/d2a/d34/d6b/f7e [0,4194304] 0 2024-04-05T21:26:42.566 INFO:tasks.workunit.client.0.smithi121.stdout:9/318: mkdir d6/df/d1d/d2c/d37/d3a/d49/d66 0 2024-04-05T21:26:42.567 INFO:tasks.workunit.client.0.smithi121.stdout:4/581: creat d5/d11/d26/d49/fb9 x:0 0 0 2024-04-05T21:26:42.568 INFO:tasks.workunit.client.0.smithi121.stdout:1/524: chown d8/d10/d14/f37 5276 1 2024-04-05T21:26:42.569 INFO:tasks.workunit.client.1.smithi188.stdout:9/701: fsync dd/d11/d1f/d25/d8a/f77 0 2024-04-05T21:26:42.570 INFO:tasks.workunit.client.0.smithi121.stdout:1/525: chown d8/d10/d14/d1a/l6e 90629 1 2024-04-05T21:26:42.570 INFO:tasks.workunit.client.1.smithi188.stdout:9/702: write f1 [212294,103906] 0 2024-04-05T21:26:42.570 INFO:tasks.workunit.client.0.smithi121.stdout:0/517: rename d7/d1c/c66 to d7/d71/d94/cb4 0 2024-04-05T21:26:42.571 INFO:tasks.workunit.client.0.smithi121.stdout:4/582: mknod d5/d11/d26/d47/cba 0 2024-04-05T21:26:42.572 INFO:tasks.workunit.client.0.smithi121.stdout:1/526: rmdir d8/d89/d9e/d2d 39 2024-04-05T21:26:42.574 INFO:tasks.workunit.client.1.smithi188.stdout:4/917: mkdir d0/d14/d15/d139 0 2024-04-05T21:26:42.574 INFO:tasks.workunit.client.1.smithi188.stdout:9/703: symlink dd/d11/d1f/d25/d8a/lf9 0 2024-04-05T21:26:42.574 INFO:tasks.workunit.client.0.smithi121.stdout:0/518: mknod d7/d71/d45/d5a/cb5 0 2024-04-05T21:26:42.575 INFO:tasks.workunit.client.0.smithi121.stdout:9/319: link d6/df/d1d/l2f d6/d36/d5c/l67 0 2024-04-05T21:26:42.576 INFO:tasks.workunit.client.1.smithi188.stdout:9/704: write dd/d11/d1f/d25/f50 [4326141,110649] 0 2024-04-05T21:26:42.577 INFO:tasks.workunit.client.1.smithi188.stdout:9/705: read - dd/d11/d22/d7e/de3/fef zero size 2024-04-05T21:26:42.577 INFO:tasks.workunit.client.1.smithi188.stdout:9/706: stat ca 0 2024-04-05T21:26:42.578 INFO:tasks.workunit.client.1.smithi188.stdout:4/918: rename d0/d2/d17/d48/f57 to d0/dbe/d137/f13a 0 2024-04-05T21:26:42.578 INFO:tasks.workunit.client.0.smithi121.stdout:4/583: symlink d5/d11/d2c/d9d/dae/d29/db4/lbb 0 2024-04-05T21:26:42.578 INFO:tasks.workunit.client.0.smithi121.stdout:1/527: mkdir d8/d89/d9e/d2d/d84/da8 0 2024-04-05T21:26:42.579 INFO:tasks.workunit.client.1.smithi188.stdout:9/707: fsync dd/d11/d1f/d26/dc0/d6b/f96 0 2024-04-05T21:26:42.579 INFO:tasks.workunit.client.1.smithi188.stdout:4/919: chown d0/dbe/f10c 537255 1 2024-04-05T21:26:42.580 INFO:tasks.workunit.client.0.smithi121.stdout:1/528: readlink d8/d8c/l96 0 2024-04-05T21:26:42.581 INFO:tasks.workunit.client.0.smithi121.stdout:0/519: creat d7/d1c/fb6 x:0 0 0 2024-04-05T21:26:42.581 INFO:tasks.workunit.client.0.smithi121.stdout:4/584: mkdir d5/d11/d26/dbc 0 2024-04-05T21:26:42.584 INFO:tasks.workunit.client.0.smithi121.stdout:0/520: rmdir d7/d71/d45/d46 39 2024-04-05T21:26:42.586 INFO:tasks.workunit.client.0.smithi121.stdout:9/320: dwrite d6/df/d1d/d2c/d37/d3c/f56 [0,4194304] 0 2024-04-05T21:26:42.588 INFO:tasks.workunit.client.0.smithi121.stdout:4/585: write d5/d11/d2c/d9d/dae/d29/d2e/f7b [265925,35565] 0 2024-04-05T21:26:42.589 INFO:tasks.workunit.client.1.smithi188.stdout:0/573: dwrite d4/d6/d13/da1/fa3 [0,4194304] 0 2024-04-05T21:26:42.590 INFO:tasks.workunit.client.1.smithi188.stdout:4/920: dwrite d0/d2/d52/f79 [4194304,4194304] 0 2024-04-05T21:26:42.591 INFO:tasks.workunit.client.1.smithi188.stdout:9/708: truncate dd/d11/d1f/d25/d51/faf 4039740 0 2024-04-05T21:26:42.595 INFO:tasks.workunit.client.0.smithi121.stdout:0/521: symlink d7/d7e/d97/lb7 0 2024-04-05T21:26:42.595 INFO:tasks.workunit.client.0.smithi121.stdout:4/586: creat d5/d45/d8b/fbd x:0 0 0 2024-04-05T21:26:42.595 INFO:tasks.workunit.client.0.smithi121.stdout:9/321: dread d6/f15 [0,4194304] 0 2024-04-05T21:26:42.596 INFO:tasks.workunit.client.1.smithi188.stdout:9/709: rename dd/d11/d1f/d26/dc0/d6b/le7 to dd/d11/d1f/d26/d55/d98/lfa 0 2024-04-05T21:26:42.597 INFO:tasks.workunit.client.1.smithi188.stdout:4/921: mkdir d0/d2/d13b 0 2024-04-05T21:26:42.598 INFO:tasks.workunit.client.0.smithi121.stdout:4/587: unlink d5/d11/d2c/c53 0 2024-04-05T21:26:42.598 INFO:tasks.workunit.client.0.smithi121.stdout:8/388: dread d12/d50/d61/f3a [0,4194304] 0 2024-04-05T21:26:42.598 INFO:tasks.workunit.client.1.smithi188.stdout:0/574: mkdir d4/d6/d13/d23/dd1 0 2024-04-05T21:26:42.599 INFO:tasks.workunit.client.0.smithi121.stdout:4/588: chown d5/d11/d2c/d9d/fb5 474 1 2024-04-05T21:26:42.599 INFO:tasks.workunit.client.1.smithi188.stdout:0/575: chown d4/d6/d13/d23/d24/l5d 1554465 1 2024-04-05T21:26:42.600 INFO:tasks.workunit.client.1.smithi188.stdout:9/710: truncate dd/d11/d73/d8d/fac 1042973 0 2024-04-05T21:26:42.601 INFO:tasks.workunit.client.0.smithi121.stdout:4/589: write d5/d45/fb0 [776091,76924] 0 2024-04-05T21:26:42.602 INFO:tasks.workunit.client.0.smithi121.stdout:9/322: creat d6/df/d50/f68 x:0 0 0 2024-04-05T21:26:42.603 INFO:tasks.workunit.client.1.smithi188.stdout:9/711: rmdir dd/d11/d73/d8d 39 2024-04-05T21:26:42.607 INFO:tasks.workunit.client.0.smithi121.stdout:8/389: mkdir d12/d50/d61/d7f 0 2024-04-05T21:26:42.608 INFO:tasks.workunit.client.0.smithi121.stdout:1/529: rmdir d8/d89 39 2024-04-05T21:26:42.609 INFO:tasks.workunit.client.1.smithi188.stdout:0/576: dwrite d4/d8/db/d42/fb3 [0,4194304] 0 2024-04-05T21:26:42.609 INFO:tasks.workunit.client.0.smithi121.stdout:9/323: getdents d6/df/d1d/d2c/d37/d3a/d49/d66 0 2024-04-05T21:26:42.610 INFO:tasks.workunit.client.0.smithi121.stdout:1/530: fsync d8/d10/f17 0 2024-04-05T21:26:42.611 INFO:tasks.workunit.client.0.smithi121.stdout:9/324: mknod d6/df/d1d/d2c/d37/d3a/c69 0 2024-04-05T21:26:42.613 INFO:tasks.workunit.client.0.smithi121.stdout:8/390: mknod d12/d50/d61/c80 0 2024-04-05T21:26:42.613 INFO:tasks.workunit.client.1.smithi188.stdout:0/577: dread d4/d8/f3d [0,4194304] 0 2024-04-05T21:26:42.615 INFO:tasks.workunit.client.0.smithi121.stdout:9/325: creat d6/df/d50/f6a x:0 0 0 2024-04-05T21:26:42.618 INFO:tasks.workunit.client.0.smithi121.stdout:5/571: sync 2024-04-05T21:26:42.618 INFO:tasks.workunit.client.0.smithi121.stdout:1/531: creat d8/d10/d14/d1a/d1b/d4c/d76/d53/d5b/fa9 x:0 0 0 2024-04-05T21:26:42.618 INFO:tasks.workunit.client.0.smithi121.stdout:9/326: mkdir d6/df/d1d/d2c/d37/d3a/d6b 0 2024-04-05T21:26:42.620 INFO:tasks.workunit.client.0.smithi121.stdout:8/391: read d12/d13/d15/f1c [1861238,40659] 0 2024-04-05T21:26:42.621 INFO:tasks.workunit.client.0.smithi121.stdout:1/532: fdatasync d8/d10/d14/d1a/d1b/d4c/f4d 0 2024-04-05T21:26:42.622 INFO:tasks.workunit.client.0.smithi121.stdout:5/572: dread - dd/de/d19/d1c/d36/d46/f9e zero size 2024-04-05T21:26:42.622 INFO:tasks.workunit.client.0.smithi121.stdout:9/327: unlink d6/d46/f55 0 2024-04-05T21:26:42.623 INFO:tasks.workunit.client.1.smithi188.stdout:4/922: truncate d0/d2/d4f/f5c 3696532 0 2024-04-05T21:26:42.626 INFO:tasks.workunit.client.0.smithi121.stdout:5/573: creat dd/de/d6d/fc7 x:0 0 0 2024-04-05T21:26:42.627 INFO:tasks.workunit.client.1.smithi188.stdout:0/578: truncate d4/d8/d10/f1b 635597 0 2024-04-05T21:26:42.628 INFO:tasks.workunit.client.1.smithi188.stdout:4/923: truncate d0/d2/d52/dc1/fe6 460948 0 2024-04-05T21:26:42.629 INFO:tasks.workunit.client.0.smithi121.stdout:8/392: dread d12/d13/d56/d57/d5e/f62 [0,4194304] 0 2024-04-05T21:26:42.629 INFO:tasks.workunit.client.0.smithi121.stdout:1/533: fdatasync d8/d89/d9e/d57/f60 0 2024-04-05T21:26:42.629 INFO:tasks.workunit.client.0.smithi121.stdout:8/393: readlink l10 0 2024-04-05T21:26:42.630 INFO:tasks.workunit.client.1.smithi188.stdout:9/712: dwrite dd/f82 [0,4194304] 0 2024-04-05T21:26:42.630 INFO:tasks.workunit.client.1.smithi188.stdout:0/579: mkdir d4/d8/db/d42/da2/dd2 0 2024-04-05T21:26:42.632 INFO:tasks.workunit.client.1.smithi188.stdout:4/924: write d0/d14/f12c [223923,109418] 0 2024-04-05T21:26:42.633 INFO:tasks.workunit.client.1.smithi188.stdout:9/713: read dd/d11/d1f/d26/dc0/d6b/f94 [1537625,62547] 0 2024-04-05T21:26:42.637 INFO:tasks.workunit.client.0.smithi121.stdout:8/394: unlink d12/d13/d15/d1d/c5c 0 2024-04-05T21:26:42.637 INFO:tasks.workunit.client.1.smithi188.stdout:0/580: creat d4/d8/db/d42/da2/dd2/fd3 x:0 0 0 2024-04-05T21:26:42.637 INFO:tasks.workunit.client.0.smithi121.stdout:4/590: dread d5/d11/d2c/f65 [0,4194304] 0 2024-04-05T21:26:42.639 INFO:tasks.workunit.client.1.smithi188.stdout:4/925: rename d0/d14/d2f/f98 to d0/d14/d2f/f13c 0 2024-04-05T21:26:42.641 INFO:tasks.workunit.client.0.smithi121.stdout:5/574: dwrite dd/de/d19/d3c/f5f [0,4194304] 0 2024-04-05T21:26:42.643 INFO:tasks.workunit.client.1.smithi188.stdout:0/581: write d4/d8/db/d21/d3c/fa5 [2672256,92758] 0 2024-04-05T21:26:42.643 INFO:tasks.workunit.client.1.smithi188.stdout:4/926: rmdir d0/d14/d15/d53/d82/dd5 39 2024-04-05T21:26:42.645 INFO:tasks.workunit.client.0.smithi121.stdout:1/534: link d8/d10/d14/d1a/l82 d8/d10/d14/d1a/d1b/d4c/d76/d53/d5b/laa 0 2024-04-05T21:26:42.646 INFO:tasks.workunit.client.0.smithi121.stdout:5/575: write dd/de/d19/d38/f97 [1105228,101924] 0 2024-04-05T21:26:42.647 INFO:tasks.workunit.client.0.smithi121.stdout:4/591: mknod d5/d11/d2c/d9d/dae/d29/d2e/cbe 0 2024-04-05T21:26:42.648 INFO:tasks.workunit.client.1.smithi188.stdout:0/582: unlink d4/d8/db/d42/fb3 0 2024-04-05T21:26:42.650 INFO:tasks.workunit.client.0.smithi121.stdout:1/535: read d8/d10/d14/d1a/d1b/d24/f49 [2978445,107473] 0 2024-04-05T21:26:42.651 INFO:tasks.workunit.client.0.smithi121.stdout:9/328: truncate d6/df/f2e 2752558 0 2024-04-05T21:26:42.651 INFO:tasks.workunit.client.1.smithi188.stdout:4/927: creat d0/d14/dd8/f13d x:0 0 0 2024-04-05T21:26:42.651 INFO:tasks.workunit.client.1.smithi188.stdout:0/583: mknod d4/d8/d39/cd4 0 2024-04-05T21:26:42.652 INFO:tasks.workunit.client.0.smithi121.stdout:8/395: dwrite d12/d50/d6a/d63/f64 [0,4194304] 0 2024-04-05T21:26:42.652 INFO:tasks.workunit.client.1.smithi188.stdout:0/584: readlink d4/d6/d61/l7d 0 2024-04-05T21:26:42.653 INFO:tasks.workunit.client.0.smithi121.stdout:4/592: unlink d5/d11/l91 0 2024-04-05T21:26:42.653 INFO:tasks.workunit.client.0.smithi121.stdout:9/329: write d6/df/d50/f68 [752567,46825] 0 2024-04-05T21:26:42.654 INFO:tasks.workunit.client.1.smithi188.stdout:4/928: symlink d0/d14/d15/d139/l13e 0 2024-04-05T21:26:42.656 INFO:tasks.workunit.client.0.smithi121.stdout:5/576: symlink dd/de/d19/d13/lc8 0 2024-04-05T21:26:42.657 INFO:tasks.workunit.client.0.smithi121.stdout:1/536: mknod d8/d89/d9e/d57/d77/cab 0 2024-04-05T21:26:42.660 INFO:tasks.workunit.client.0.smithi121.stdout:8/396: truncate d12/d50/d6a/f45 1550027 0 2024-04-05T21:26:42.661 INFO:tasks.workunit.client.1.smithi188.stdout:0/585: creat d4/d6/d13/d23/dd1/fd5 x:0 0 0 2024-04-05T21:26:42.663 INFO:tasks.workunit.client.0.smithi121.stdout:1/537: mknod d8/d10/d14/d1a/d1b/d24/cac 0 2024-04-05T21:26:42.666 INFO:tasks.workunit.client.0.smithi121.stdout:4/593: dwrite d5/d11/d28/d87/fa1 [4194304,4194304] 0 2024-04-05T21:26:42.667 INFO:tasks.workunit.client.0.smithi121.stdout:5/577: link dd/de/d19/d38/d45/d68/f79 dd/de/d39/d3b/fc9 0 2024-04-05T21:26:42.672 INFO:tasks.workunit.client.1.smithi188.stdout:3/740: sync 2024-04-05T21:26:42.672 INFO:tasks.workunit.client.1.smithi188.stdout:5/628: sync 2024-04-05T21:26:42.674 INFO:tasks.workunit.client.0.smithi121.stdout:1/538: mkdir d8/d89/d9e/d2d/d84/da8/dad 0 2024-04-05T21:26:42.675 INFO:tasks.workunit.client.0.smithi121.stdout:5/578: symlink dd/de/d19/d38/d9b/lca 0 2024-04-05T21:26:42.676 INFO:tasks.workunit.client.1.smithi188.stdout:4/929: symlink d0/d2/d2a/l13f 0 2024-04-05T21:26:42.676 INFO:tasks.workunit.client.0.smithi121.stdout:4/594: symlink d5/d11/d28/lbf 0 2024-04-05T21:26:42.677 INFO:tasks.workunit.client.1.smithi188.stdout:0/586: creat d4/d8/db/d21/fd6 x:0 0 0 2024-04-05T21:26:42.679 INFO:tasks.workunit.client.0.smithi121.stdout:1/539: write d8/d10/d14/d1a/d1b/d4c/f4f [203363,97343] 0 2024-04-05T21:26:42.680 INFO:tasks.workunit.client.0.smithi121.stdout:5/579: creat dd/de/d19/fcb x:0 0 0 2024-04-05T21:26:42.680 INFO:tasks.workunit.client.1.smithi188.stdout:0/587: symlink d4/d6/d13/dca/ld7 0 2024-04-05T21:26:42.681 INFO:tasks.workunit.client.0.smithi121.stdout:9/330: dwrite f2 [8388608,4194304] 0 2024-04-05T21:26:42.682 INFO:tasks.workunit.client.1.smithi188.stdout:0/588: symlink d4/d6/d46/ld8 0 2024-04-05T21:26:42.683 INFO:tasks.workunit.client.1.smithi188.stdout:0/589: fdatasync d4/d8/db/d21/d3c/fa9 0 2024-04-05T21:26:42.684 INFO:tasks.workunit.client.0.smithi121.stdout:9/331: unlink d6/f16 0 2024-04-05T21:26:42.685 INFO:tasks.workunit.client.0.smithi121.stdout:9/332: write f2 [2297642,61250] 0 2024-04-05T21:26:42.687 INFO:tasks.workunit.client.0.smithi121.stdout:4/595: rename d5/d11/d26/d49/d8f to d5/d11/d28/dc0 0 2024-04-05T21:26:42.687 INFO:tasks.workunit.client.1.smithi188.stdout:0/590: unlink d4/d8/d39/f48 0 2024-04-05T21:26:42.687 INFO:tasks.workunit.client.1.smithi188.stdout:5/629: dread d6/d15/d5c/d7a/d44/d98/fcd [0,4194304] 0 2024-04-05T21:26:42.690 INFO:tasks.workunit.client.0.smithi121.stdout:9/333: dread d6/fc [4194304,4194304] 0 2024-04-05T21:26:42.691 INFO:tasks.workunit.client.0.smithi121.stdout:5/580: dread dd/de/d19/d38/d45/f5c [0,4194304] 0 2024-04-05T21:26:42.692 INFO:tasks.workunit.client.0.smithi121.stdout:1/540: link d8/d10/d14/d1a/c70 d8/d10/d14/d1a/d1b/d4c/d76/da4/cae 0 2024-04-05T21:26:42.692 INFO:tasks.workunit.client.1.smithi188.stdout:0/591: rename d4/c5 to d4/d6/d46/cd9 0 2024-04-05T21:26:42.694 INFO:tasks.workunit.client.1.smithi188.stdout:0/592: chown d4/d8/db/d21/d3c/f74 544 1 2024-04-05T21:26:42.695 INFO:tasks.workunit.client.0.smithi121.stdout:9/334: rename d6/l8 to d6/df/d1d/d2c/l6c 0 2024-04-05T21:26:42.698 INFO:tasks.workunit.client.1.smithi188.stdout:0/593: rename d4/d8/db/d21/d4a/d73/d79/l7a to d4/d8/db/d21/d3c/lda 0 2024-04-05T21:26:42.699 INFO:tasks.workunit.client.1.smithi188.stdout:0/594: rmdir d4/d6/d13/d23/d24 39 2024-04-05T21:26:42.701 INFO:tasks.workunit.client.1.smithi188.stdout:0/595: creat d4/d6/d46/dba/fdb x:0 0 0 2024-04-05T21:26:42.703 INFO:tasks.workunit.client.0.smithi121.stdout:5/581: dread dd/de/d19/d1c/d36/f3a [0,4194304] 0 2024-04-05T21:26:42.703 INFO:tasks.workunit.client.0.smithi121.stdout:1/541: dread d8/d10/d14/f2c [0,4194304] 0 2024-04-05T21:26:42.704 INFO:tasks.workunit.client.0.smithi121.stdout:1/542: chown d8/d10/d14/d1a/d1b/d24/l7a 99468 1 2024-04-05T21:26:42.705 INFO:tasks.workunit.client.0.smithi121.stdout:4/596: dread d5/d11/d28/d87/f96 [0,4194304] 0 2024-04-05T21:26:42.707 INFO:tasks.workunit.client.0.smithi121.stdout:5/582: rename dd/de/d19/d1c/d36/f93 to dd/de/d19/d1c/d7b/d88/fcc 0 2024-04-05T21:26:42.707 INFO:tasks.workunit.client.1.smithi188.stdout:0/596: dwrite d4/d8/db/d21/d4a/d73/f94 [0,4194304] 0 2024-04-05T21:26:42.708 INFO:tasks.workunit.client.0.smithi121.stdout:4/597: mknod d5/d11/d20/da5/cc1 0 2024-04-05T21:26:42.709 INFO:tasks.workunit.client.0.smithi121.stdout:5/583: chown dd/de/d19/d1c/d74/f81 0 1 2024-04-05T21:26:42.709 INFO:tasks.workunit.client.1.smithi188.stdout:8/623: sync 2024-04-05T21:26:42.709 INFO:tasks.workunit.client.1.smithi188.stdout:6/730: sync 2024-04-05T21:26:42.709 INFO:tasks.workunit.client.1.smithi188.stdout:2/717: sync 2024-04-05T21:26:42.709 INFO:tasks.workunit.client.1.smithi188.stdout:7/887: sync 2024-04-05T21:26:42.709 INFO:tasks.workunit.client.1.smithi188.stdout:1/692: sync 2024-04-05T21:26:42.710 INFO:tasks.workunit.client.0.smithi121.stdout:1/543: rename d8/d10/d14/d1a/d1b/d4c/d76/d53/d5b/fa9 to d8/d89/d9e/d57/faf 0 2024-04-05T21:26:42.711 INFO:tasks.workunit.client.0.smithi121.stdout:1/544: write d8/d89/f98 [998935,29417] 0 2024-04-05T21:26:42.711 INFO:tasks.workunit.client.0.smithi121.stdout:5/584: mkdir dd/de/d39/dcd 0 2024-04-05T21:26:42.712 INFO:tasks.workunit.client.0.smithi121.stdout:4/598: creat d5/d11/d80/fc2 x:0 0 0 2024-04-05T21:26:42.713 INFO:tasks.workunit.client.0.smithi121.stdout:1/545: chown d8/d10/d14/d1a/d1b/d38/d4b/f54 2 1 2024-04-05T21:26:42.713 INFO:tasks.workunit.client.1.smithi188.stdout:8/624: fsync d5/d21/d40/d43/f81 0 2024-04-05T21:26:42.714 INFO:tasks.workunit.client.0.smithi121.stdout:4/599: fsync d5/d11/d20/f82 0 2024-04-05T21:26:42.716 INFO:tasks.workunit.client.0.smithi121.stdout:5/585: rename dd/de/d39/d3b/d50/d82 to dd/d41/dce 0 2024-04-05T21:26:42.716 INFO:tasks.workunit.client.1.smithi188.stdout:1/693: dread d1/dc/f56 [0,4194304] 0 2024-04-05T21:26:42.717 INFO:tasks.workunit.client.1.smithi188.stdout:7/888: mkdir d3/d2c/dae/dfd/d2e/d8c/db7/d11f 0 2024-04-05T21:26:42.717 INFO:tasks.workunit.client.0.smithi121.stdout:4/600: mknod d5/d11/cc3 0 2024-04-05T21:26:42.718 INFO:tasks.workunit.client.0.smithi121.stdout:4/601: fdatasync d5/d11/d2c/d9d/dae/d29/d2e/f7b 0 2024-04-05T21:26:42.719 INFO:tasks.workunit.client.0.smithi121.stdout:1/546: rename d8/d10/d14/d1a/d1b/d24/f5f to d8/d89/d9e/da3/fb0 0 2024-04-05T21:26:42.721 INFO:tasks.workunit.client.1.smithi188.stdout:2/718: getdents d5/db/dd/d36/d3c/d48/d7a/d7c 0 2024-04-05T21:26:42.722 INFO:tasks.workunit.client.0.smithi121.stdout:5/586: truncate dd/f1f 1241457 0 2024-04-05T21:26:42.722 INFO:tasks.workunit.client.0.smithi121.stdout:4/602: read d5/d11/d2c/d9d/dae/d29/f9b [3290448,33110] 0 2024-04-05T21:26:42.722 INFO:tasks.workunit.client.1.smithi188.stdout:7/889: dread d3/d2c/dae/dfd/d10/f14 [8388608,4194304] 0 2024-04-05T21:26:42.723 INFO:tasks.workunit.client.0.smithi121.stdout:1/547: mknod d8/d10/cb1 0 2024-04-05T21:26:42.724 INFO:tasks.workunit.client.1.smithi188.stdout:2/719: creat d5/db/dd/d67/dcd/d46/d9f/fe8 x:0 0 0 2024-04-05T21:26:42.726 INFO:tasks.workunit.client.1.smithi188.stdout:2/720: chown d5/db/dd/fc6 0 1 2024-04-05T21:26:42.727 INFO:tasks.workunit.client.1.smithi188.stdout:7/890: creat d3/d4/d1d/f120 x:0 0 0 2024-04-05T21:26:42.730 INFO:tasks.workunit.client.1.smithi188.stdout:7/891: mknod d3/d2c/dae/dfd/d10/d5f/dc6/d104/c121 0 2024-04-05T21:26:42.730 INFO:tasks.workunit.client.1.smithi188.stdout:1/694: dread d1/dc/d16/d46/d88/d86/fab [0,4194304] 0 2024-04-05T21:26:42.732 INFO:tasks.workunit.client.1.smithi188.stdout:7/892: read - d3/d2c/dae/dfd/d2e/d3b/fbf zero size 2024-04-05T21:26:42.734 INFO:tasks.workunit.client.1.smithi188.stdout:7/893: symlink d3/d2c/d5e/l122 0 2024-04-05T21:26:42.735 INFO:tasks.workunit.client.1.smithi188.stdout:1/695: getdents d1/d7/dbf 0 2024-04-05T21:26:42.739 INFO:tasks.workunit.client.1.smithi188.stdout:2/721: dread d5/db/dd/d24/f52 [4194304,4194304] 0 2024-04-05T21:26:42.740 INFO:tasks.workunit.client.1.smithi188.stdout:7/894: rename d3/d2c/dae/dfd/d10/d3c/ff5 to d3/d4/d1d/f123 0 2024-04-05T21:26:42.742 INFO:tasks.workunit.client.1.smithi188.stdout:1/696: rename d1/dc/d16/d21/d27/d49/f73 to d1/dc/d16/d46/d88/d3f/fdb 0 2024-04-05T21:26:42.744 INFO:tasks.workunit.client.1.smithi188.stdout:7/895: creat d3/d4/d101/f124 x:0 0 0 2024-04-05T21:26:42.745 INFO:tasks.workunit.client.1.smithi188.stdout:2/722: creat d5/db/dd/d36/d3c/d48/d7a/d7c/fe9 x:0 0 0 2024-04-05T21:26:42.748 INFO:tasks.workunit.client.1.smithi188.stdout:2/723: mkdir d5/db/dd/d67/dcd/d46/d5d/dea 0 2024-04-05T21:26:42.753 INFO:tasks.workunit.client.1.smithi188.stdout:1/697: dwrite d1/dc/d16/d21/dbb/fc9 [0,4194304] 0 2024-04-05T21:26:42.754 INFO:tasks.workunit.client.1.smithi188.stdout:1/698: chown d1/d7/d4e/d5f/f76 711940728 1 2024-04-05T21:26:42.755 INFO:tasks.workunit.client.1.smithi188.stdout:6/731: dread d6/d13/d18/d3c/ddc/f80 [0,4194304] 0 2024-04-05T21:26:42.759 INFO:tasks.workunit.client.0.smithi121.stdout:8/397: truncate ff 1246934 0 2024-04-05T21:26:42.763 INFO:tasks.workunit.client.1.smithi188.stdout:6/732: rename d6/d71/c8e to d6/d7/ce0 0 2024-04-05T21:26:42.763 INFO:tasks.workunit.client.1.smithi188.stdout:2/724: link d5/db/dd/d67/dcd/d46/d9f/fb0 d5/db/dd/d36/d3c/d63/feb 0 2024-04-05T21:26:42.763 INFO:tasks.workunit.client.1.smithi188.stdout:0/597: read d4/d8/db/d21/d4a/d73/f99 [985996,122561] 0 2024-04-05T21:26:42.764 INFO:tasks.workunit.client.1.smithi188.stdout:9/714: dread dd/d11/d1f/d25/d51/faf [0,4194304] 0 2024-04-05T21:26:42.764 INFO:tasks.workunit.client.1.smithi188.stdout:6/733: chown d6/d13/d18/c1a 229044988 1 2024-04-05T21:26:42.767 INFO:tasks.workunit.client.1.smithi188.stdout:6/734: read d6/d13/d18/d19/f27 [4258975,2959] 0 2024-04-05T21:26:42.770 INFO:tasks.workunit.client.1.smithi188.stdout:2/725: truncate d5/db/dd/d36/d3c/d63/d75/d87/f9c 2172292 0 2024-04-05T21:26:42.770 INFO:tasks.workunit.client.0.smithi121.stdout:4/603: read d5/d11/d2c/d8c/f94 [2781070,85620] 0 2024-04-05T21:26:42.770 INFO:tasks.workunit.client.0.smithi121.stdout:8/398: dwrite d12/d50/d6a/f6e [0,4194304] 0 2024-04-05T21:26:42.772 INFO:tasks.workunit.client.1.smithi188.stdout:6/735: unlink d6/d13/d18/d19/lc7 0 2024-04-05T21:26:42.775 INFO:tasks.workunit.client.0.smithi121.stdout:4/604: symlink d5/d11/d28/lc4 0 2024-04-05T21:26:42.776 INFO:tasks.workunit.client.0.smithi121.stdout:4/605: fsync d5/d11/d28/f1b 0 2024-04-05T21:26:42.777 INFO:tasks.workunit.client.1.smithi188.stdout:2/726: dread d5/db/dd/d67/dcd/d46/f7b [0,4194304] 0 2024-04-05T21:26:42.782 INFO:tasks.workunit.client.1.smithi188.stdout:2/727: readlink d5/d5f/d98/l9d 0 2024-04-05T21:26:42.788 INFO:tasks.workunit.client.0.smithi121.stdout:5/587: write dd/de/d19/d1c/d36/d46/f9e [406779,45044] 0 2024-04-05T21:26:42.796 INFO:tasks.workunit.client.0.smithi121.stdout:1/548: symlink d8/d10/d14/d1a/d1b/d4c/da1/lb2 0 2024-04-05T21:26:42.804 INFO:tasks.workunit.client.0.smithi121.stdout:1/549: rename d8/d89/la7 to d8/d8c/lb3 0 2024-04-05T21:26:42.804 INFO:tasks.workunit.client.0.smithi121.stdout:5/588: dwrite dd/de/d19/d1c/d36/d46/f9e [0,4194304] 0 2024-04-05T21:26:42.805 INFO:tasks.workunit.client.0.smithi121.stdout:8/399: dwrite d12/d13/d56/d57/f78 [0,4194304] 0 2024-04-05T21:26:42.814 INFO:tasks.workunit.client.0.smithi121.stdout:9/335: dread f2 [4194304,4194304] 0 2024-04-05T21:26:42.814 INFO:tasks.workunit.client.0.smithi121.stdout:1/550: rmdir d8/d89/d9e/d2d/d84/da8 39 2024-04-05T21:26:42.816 INFO:tasks.workunit.client.0.smithi121.stdout:1/551: chown d8/d10/d14/c22 35 1 2024-04-05T21:26:42.817 INFO:tasks.workunit.client.0.smithi121.stdout:8/400: dwrite f3 [4194304,4194304] 0 2024-04-05T21:26:42.819 INFO:tasks.workunit.client.0.smithi121.stdout:8/401: write d12/f51 [827957,977] 0 2024-04-05T21:26:42.822 INFO:tasks.workunit.client.0.smithi121.stdout:9/336: rmdir d6/df/d1d/d2c/d37/d3a/d4a 0 2024-04-05T21:26:42.831 INFO:tasks.workunit.client.0.smithi121.stdout:1/552: dwrite d8/d10/d14/d1a/d1b/d24/f49 [4194304,4194304] 0 2024-04-05T21:26:42.832 INFO:tasks.workunit.client.0.smithi121.stdout:9/337: dwrite d6/fc [0,4194304] 0 2024-04-05T21:26:42.834 INFO:tasks.workunit.client.0.smithi121.stdout:1/553: read d8/d10/d14/d1a/d1b/d4c/d76/f3d [2215021,62046] 0 2024-04-05T21:26:42.841 INFO:tasks.workunit.client.0.smithi121.stdout:1/554: mknod d8/d89/d9e/d57/d77/cb4 0 2024-04-05T21:26:42.845 INFO:tasks.workunit.client.1.smithi188.stdout:3/741: truncate d5/d7/d6d/dd8/f3d 3714244 0 2024-04-05T21:26:42.846 INFO:tasks.workunit.client.0.smithi121.stdout:8/402: dwrite d12/d50/d6a/f45 [0,4194304] 0 2024-04-05T21:26:42.849 INFO:tasks.workunit.client.0.smithi121.stdout:8/403: chown d12/d13/c3b 5 1 2024-04-05T21:26:42.850 INFO:tasks.workunit.client.1.smithi188.stdout:8/625: write d5/dc/f2c [650305,109884] 0 2024-04-05T21:26:42.850 INFO:tasks.workunit.client.0.smithi121.stdout:1/555: dwrite d8/d10/d14/d1a/d1b/d4c/f4f [0,4194304] 0 2024-04-05T21:26:42.852 INFO:tasks.workunit.client.0.smithi121.stdout:2/413: sync 2024-04-05T21:26:42.853 INFO:tasks.workunit.client.1.smithi188.stdout:0/598: dread d4/f5f [0,4194304] 0 2024-04-05T21:26:42.859 INFO:tasks.workunit.client.1.smithi188.stdout:4/930: dwrite d0/d2/d17/f4e [0,4194304] 0 2024-04-05T21:26:42.860 INFO:tasks.workunit.client.0.smithi121.stdout:2/414: creat d2/d9/d56/f76 x:0 0 0 2024-04-05T21:26:42.860 INFO:tasks.workunit.client.0.smithi121.stdout:8/404: rename d12/d13/d15/l28 to d12/d13/d15/l81 0 2024-04-05T21:26:42.860 INFO:tasks.workunit.client.1.smithi188.stdout:4/931: dread - d0/d2/d52/d126/f129 zero size 2024-04-05T21:26:42.861 INFO:tasks.workunit.client.0.smithi121.stdout:2/415: mkdir d2/d9/d77 0 2024-04-05T21:26:42.861 INFO:tasks.workunit.client.0.smithi121.stdout:1/556: dwrite d8/d10/d14/d1a/f71 [0,4194304] 0 2024-04-05T21:26:42.863 INFO:tasks.workunit.client.1.smithi188.stdout:1/699: write d1/dc/f1b [2925035,98796] 0 2024-04-05T21:26:42.863 INFO:tasks.workunit.client.0.smithi121.stdout:2/416: write d2/d9/d38/d2d/d47/f70 [61393,1573] 0 2024-04-05T21:26:42.864 INFO:tasks.workunit.client.0.smithi121.stdout:8/405: symlink d12/d13/d15/l82 0 2024-04-05T21:26:42.864 INFO:tasks.workunit.client.0.smithi121.stdout:1/557: creat d8/d89/d9e/d2d/fb5 x:0 0 0 2024-04-05T21:26:42.866 INFO:tasks.workunit.client.0.smithi121.stdout:8/406: dread - d12/d13/d56/d57/f7b zero size 2024-04-05T21:26:42.868 INFO:tasks.workunit.client.0.smithi121.stdout:9/338: truncate d6/df/d1d/d2c/d37/d3c/f56 3961427 0 2024-04-05T21:26:42.868 INFO:tasks.workunit.client.0.smithi121.stdout:1/558: write d8/d89/d9e/f75 [4350719,123021] 0 2024-04-05T21:26:42.868 INFO:tasks.workunit.client.1.smithi188.stdout:5/630: dwrite d6/d15/d1b/f34 [0,4194304] 0 2024-04-05T21:26:42.870 INFO:tasks.workunit.client.0.smithi121.stdout:2/417: creat d2/d9/d38/d2d/d4c/f78 x:0 0 0 2024-04-05T21:26:42.872 INFO:tasks.workunit.client.0.smithi121.stdout:1/559: fdatasync d8/d10/d14/d1a/f72 0 2024-04-05T21:26:42.872 INFO:tasks.workunit.client.0.smithi121.stdout:8/407: write d12/d13/d15/f19 [2329920,63203] 0 2024-04-05T21:26:42.874 INFO:tasks.workunit.client.0.smithi121.stdout:9/339: creat d6/df/f6d x:0 0 0 2024-04-05T21:26:42.875 INFO:tasks.workunit.client.0.smithi121.stdout:2/418: mkdir d2/d9/d38/d2d/d4c/d79 0 2024-04-05T21:26:42.877 INFO:tasks.workunit.client.1.smithi188.stdout:8/626: fdatasync d5/d21/d40/f55 0 2024-04-05T21:26:42.880 INFO:tasks.workunit.client.0.smithi121.stdout:1/560: rename d8/d10/d14 to d8/d8c/db6 0 2024-04-05T21:26:42.883 INFO:tasks.workunit.client.0.smithi121.stdout:2/419: creat d2/d9/d56/d62/f7a x:0 0 0 2024-04-05T21:26:42.884 INFO:tasks.workunit.client.0.smithi121.stdout:1/561: creat d8/d8c/db6/d1a/d1b/d24/d79/fb7 x:0 0 0 2024-04-05T21:26:42.885 INFO:tasks.workunit.client.0.smithi121.stdout:2/420: truncate d2/d9/d38/d2d/f58 69408 0 2024-04-05T21:26:42.885 INFO:tasks.workunit.client.1.smithi188.stdout:2/728: dread d5/db/dd/f15 [0,4194304] 0 2024-04-05T21:26:42.885 INFO:tasks.workunit.client.1.smithi188.stdout:7/896: dwrite d3/da5/fd3 [0,4194304] 0 2024-04-05T21:26:42.885 INFO:tasks.workunit.client.0.smithi121.stdout:8/408: creat d12/d13/d15/f83 x:0 0 0 2024-04-05T21:26:42.886 INFO:tasks.workunit.client.0.smithi121.stdout:2/421: write d2/f11 [3060226,24550] 0 2024-04-05T21:26:42.886 INFO:tasks.workunit.client.0.smithi121.stdout:1/562: write d8/d89/d9e/d2d/fb5 [676435,48910] 0 2024-04-05T21:26:42.890 INFO:tasks.workunit.client.0.smithi121.stdout:1/563: creat d8/d8c/db6/d1a/d1b/d38/d4b/fb8 x:0 0 0 2024-04-05T21:26:42.891 INFO:tasks.workunit.client.1.smithi188.stdout:3/742: link d5/d39/d84/fde d5/d7/d65/d61/d66/da4/ffa 0 2024-04-05T21:26:42.896 INFO:tasks.workunit.client.1.smithi188.stdout:8/627: creat d5/d4b/fce x:0 0 0 2024-04-05T21:26:42.897 INFO:tasks.workunit.client.0.smithi121.stdout:1/564: dread d8/d89/f98 [0,4194304] 0 2024-04-05T21:26:42.900 INFO:tasks.workunit.client.0.smithi121.stdout:8/409: write d12/d13/f1b [1432704,90480] 0 2024-04-05T21:26:42.902 INFO:tasks.workunit.client.1.smithi188.stdout:7/897: read - d3/d2c/dae/dfd/d10/dbe/ded/f110 zero size 2024-04-05T21:26:42.902 INFO:tasks.workunit.client.1.smithi188.stdout:0/599: rmdir d4/d6/d13/d23/dcc 0 2024-04-05T21:26:42.902 INFO:tasks.workunit.client.1.smithi188.stdout:6/736: dwrite d6/f2e [0,4194304] 0 2024-04-05T21:26:42.903 INFO:tasks.workunit.client.0.smithi121.stdout:2/422: write d2/f10 [2326726,9238] 0 2024-04-05T21:26:42.906 INFO:tasks.workunit.client.1.smithi188.stdout:4/932: dwrite d0/d14/d2f/ddd/f123 [0,4194304] 0 2024-04-05T21:26:42.906 INFO:tasks.workunit.client.1.smithi188.stdout:8/628: mkdir d5/d21/d40/d43/d48/d7e/dcf 0 2024-04-05T21:26:42.906 INFO:tasks.workunit.client.1.smithi188.stdout:1/700: write d1/dc/d16/d21/d27/d49/f59 [661879,16919] 0 2024-04-05T21:26:42.906 INFO:tasks.workunit.client.0.smithi121.stdout:1/565: dwrite d8/d8c/db6/d1a/d1b/d38/d4b/f6c [0,4194304] 0 2024-04-05T21:26:42.907 INFO:tasks.workunit.client.1.smithi188.stdout:7/898: write d3/d2c/dae/f10c [392040,116824] 0 2024-04-05T21:26:42.908 INFO:tasks.workunit.client.1.smithi188.stdout:5/631: getdents d6/d17/d23/d7d 0 2024-04-05T21:26:42.910 INFO:tasks.workunit.client.0.smithi121.stdout:2/423: rename d2/d9/fd to d2/f7b 0 2024-04-05T21:26:42.910 INFO:tasks.workunit.client.0.smithi121.stdout:1/566: creat d8/d89/fb9 x:0 0 0 2024-04-05T21:26:42.911 INFO:tasks.workunit.client.1.smithi188.stdout:0/600: dread d4/d6/d46/f59 [0,4194304] 0 2024-04-05T21:26:42.913 INFO:tasks.workunit.client.1.smithi188.stdout:5/632: mkdir d6/d15/d1b/ddc 0 2024-04-05T21:26:42.918 INFO:tasks.workunit.client.0.smithi121.stdout:8/410: dread d12/d13/f1b [0,4194304] 0 2024-04-05T21:26:42.921 INFO:tasks.workunit.client.1.smithi188.stdout:6/737: creat d6/d13/d18/d94/fe1 x:0 0 0 2024-04-05T21:26:42.922 INFO:tasks.workunit.client.1.smithi188.stdout:3/743: dwrite d5/d7/d65/f8c [0,4194304] 0 2024-04-05T21:26:42.923 INFO:tasks.workunit.client.1.smithi188.stdout:2/729: symlink d5/db/dd/d67/dcd/d46/d66/d8f/dcb/dce/lec 0 2024-04-05T21:26:42.924 INFO:tasks.workunit.client.0.smithi121.stdout:8/411: dwrite d12/d13/f20 [0,4194304] 0 2024-04-05T21:26:42.925 INFO:tasks.workunit.client.1.smithi188.stdout:6/738: dread - d6/d13/d18/d19/d2c/d3f/d77/fc0 zero size 2024-04-05T21:26:42.927 INFO:tasks.workunit.client.1.smithi188.stdout:3/744: chown d5/f41 313 1 2024-04-05T21:26:42.927 INFO:tasks.workunit.client.1.smithi188.stdout:5/633: mknod d6/d82/d40/d42/cdd 0 2024-04-05T21:26:42.928 INFO:tasks.workunit.client.1.smithi188.stdout:4/933: unlink d0/d14/l96 0 2024-04-05T21:26:42.929 INFO:tasks.workunit.client.1.smithi188.stdout:7/899: dwrite d3/f68 [0,4194304] 0 2024-04-05T21:26:42.929 INFO:tasks.workunit.client.0.smithi121.stdout:8/412: rename d12/d13/d15/f19 to d12/d13/f84 0 2024-04-05T21:26:42.931 INFO:tasks.workunit.client.0.smithi121.stdout:8/413: unlink d12/d13/d15/f1c 0 2024-04-05T21:26:42.934 INFO:tasks.workunit.client.1.smithi188.stdout:2/730: symlink d5/d5f/d98/led 0 2024-04-05T21:26:42.936 INFO:tasks.workunit.client.0.smithi121.stdout:8/414: rename d12/d13/d15/l7e to d12/l85 0 2024-04-05T21:26:42.941 INFO:tasks.workunit.client.0.smithi121.stdout:8/415: unlink d12/d50/d6a/l49 0 2024-04-05T21:26:42.943 INFO:tasks.workunit.client.0.smithi121.stdout:8/416: symlink d12/d13/d15/d1d/l86 0 2024-04-05T21:26:42.943 INFO:tasks.workunit.client.1.smithi188.stdout:0/601: mkdir d4/d6/d13/d23/d24/ddc 0 2024-04-05T21:26:42.944 INFO:tasks.workunit.client.0.smithi121.stdout:1/567: truncate d8/d89/f98 314639 0 2024-04-05T21:26:42.946 INFO:tasks.workunit.client.0.smithi121.stdout:8/417: mkdir d12/d50/d6a/d87 0 2024-04-05T21:26:42.947 INFO:tasks.workunit.client.1.smithi188.stdout:7/900: mknod d3/d2c/dae/dfd/d55/c125 0 2024-04-05T21:26:42.950 INFO:tasks.workunit.client.0.smithi121.stdout:8/418: creat d12/d13/d56/d57/f88 x:0 0 0 2024-04-05T21:26:42.950 INFO:tasks.workunit.client.0.smithi121.stdout:2/424: dwrite d2/d28/f4f [0,4194304] 0 2024-04-05T21:26:42.951 INFO:tasks.workunit.client.1.smithi188.stdout:5/634: rename d6/d17/d39/l6d to d6/d15/d1b/lde 0 2024-04-05T21:26:42.951 INFO:tasks.workunit.client.1.smithi188.stdout:2/731: symlink d5/db/dd/d67/dcd/d46/d66/d8f/dcb/lee 0 2024-04-05T21:26:42.951 INFO:tasks.workunit.client.1.smithi188.stdout:6/739: unlink d6/d13/d18/d3c/ddc/lc1 0 2024-04-05T21:26:42.952 INFO:tasks.workunit.client.0.smithi121.stdout:2/425: stat d2/d9/d17/d3a 0 2024-04-05T21:26:42.953 INFO:tasks.workunit.client.1.smithi188.stdout:1/701: getdents d1/dc/d16/d46/d95/d99 0 2024-04-05T21:26:42.953 INFO:tasks.workunit.client.1.smithi188.stdout:2/732: rmdir d5/db/d35/d3b/d6b 39 2024-04-05T21:26:42.953 INFO:tasks.workunit.client.1.smithi188.stdout:8/629: getdents d5/d6e/d86/dcb 0 2024-04-05T21:26:42.954 INFO:tasks.workunit.client.1.smithi188.stdout:4/934: dwrite d0/d14/d15/d53/f9f [0,4194304] 0 2024-04-05T21:26:42.955 INFO:tasks.workunit.client.0.smithi121.stdout:1/568: dwrite d8/d8c/db6/d1a/f20 [0,4194304] 0 2024-04-05T21:26:42.956 INFO:tasks.workunit.client.1.smithi188.stdout:5/635: mknod d6/d82/d40/d42/d81/da6/cdf 0 2024-04-05T21:26:42.957 INFO:tasks.workunit.client.0.smithi121.stdout:1/569: write d8/d8c/db6/d1a/d1b/d85/fa0 [1577548,53543] 0 2024-04-05T21:26:42.960 INFO:tasks.workunit.client.0.smithi121.stdout:8/419: dwrite d12/d13/d15/d1d/f36 [0,4194304] 0 2024-04-05T21:26:42.960 INFO:tasks.workunit.client.0.smithi121.stdout:1/570: write d8/d8c/db6/d1a/f46 [1565463,51362] 0 2024-04-05T21:26:42.960 INFO:tasks.workunit.client.1.smithi188.stdout:2/733: dread - d5/db/dd/d67/dcd/d46/fbd zero size 2024-04-05T21:26:42.960 INFO:tasks.workunit.client.1.smithi188.stdout:6/740: dread d6/d13/d18/d3c/ddc/f80 [0,4194304] 0 2024-04-05T21:26:42.964 INFO:tasks.workunit.client.0.smithi121.stdout:8/420: unlink d12/d50/d61/l32 0 2024-04-05T21:26:42.968 INFO:tasks.workunit.client.0.smithi121.stdout:8/421: creat d12/f89 x:0 0 0 2024-04-05T21:26:42.968 INFO:tasks.workunit.client.0.smithi121.stdout:8/422: chown fc 906845 1 2024-04-05T21:26:42.969 INFO:tasks.workunit.client.1.smithi188.stdout:8/630: read d5/d21/d40/d43/d44/d5b/d99/da3/fa5 [679097,30384] 0 2024-04-05T21:26:42.970 INFO:tasks.workunit.client.1.smithi188.stdout:4/935: rename d0/d2/d52/dc1/lff to d0/d14/d15/d42/l140 0 2024-04-05T21:26:42.971 INFO:tasks.workunit.client.1.smithi188.stdout:7/901: fsync d3/f68 0 2024-04-05T21:26:42.972 INFO:tasks.workunit.client.1.smithi188.stdout:8/631: stat d5/dc/d8f 0 2024-04-05T21:26:42.972 INFO:tasks.workunit.client.0.smithi121.stdout:8/423: rmdir d12/d50/d76 0 2024-04-05T21:26:42.973 INFO:tasks.workunit.client.0.smithi121.stdout:8/424: fdatasync d12/d13/f1b 0 2024-04-05T21:26:42.976 INFO:tasks.workunit.client.0.smithi121.stdout:8/425: creat d12/d13/d56/d6d/f8a x:0 0 0 2024-04-05T21:26:42.980 INFO:tasks.workunit.client.1.smithi188.stdout:2/734: truncate d5/db/dd/d36/d3c/d48/d78/f83 32147 0 2024-04-05T21:26:42.981 INFO:tasks.workunit.client.1.smithi188.stdout:7/902: fdatasync d3/d4/d1d/d74/d73/fee 0 2024-04-05T21:26:42.981 INFO:tasks.workunit.client.1.smithi188.stdout:4/936: fsync d0/d14/d15/d42/fa5 0 2024-04-05T21:26:42.986 INFO:tasks.workunit.client.1.smithi188.stdout:4/937: truncate d0/d14/d15/d78/f10b 488274 0 2024-04-05T21:26:42.987 INFO:tasks.workunit.client.1.smithi188.stdout:7/903: fdatasync d3/d2c/dae/fda 0 2024-04-05T21:26:42.987 INFO:tasks.workunit.client.1.smithi188.stdout:5/636: dread d6/d82/f50 [0,4194304] 0 2024-04-05T21:26:42.987 INFO:tasks.workunit.client.1.smithi188.stdout:8/632: mknod d5/d21/d40/d43/d44/d5b/d99/d69/cd0 0 2024-04-05T21:26:42.991 INFO:tasks.workunit.client.1.smithi188.stdout:5/637: chown d6/d15/d5c/d7a/d44/d98/dbf/lca 37441639 1 2024-04-05T21:26:42.993 INFO:tasks.workunit.client.1.smithi188.stdout:2/735: creat d5/db/dd/d67/dcd/d46/d66/fef x:0 0 0 2024-04-05T21:26:42.998 INFO:tasks.workunit.client.1.smithi188.stdout:0/602: dwrite d4/d8/db/d42/d38/f49 [4194304,4194304] 0 2024-04-05T21:26:42.998 INFO:tasks.workunit.client.1.smithi188.stdout:4/938: rename d0/d2/d17/d3a/f54 to d0/d14/f141 0 2024-04-05T21:26:43.001 INFO:tasks.workunit.client.1.smithi188.stdout:7/904: creat d3/d2c/f126 x:0 0 0 2024-04-05T21:26:43.002 INFO:tasks.workunit.client.1.smithi188.stdout:4/939: write d0/d14/d15/d53/d12e/fda [2574196,98108] 0 2024-04-05T21:26:43.003 INFO:tasks.workunit.client.0.smithi121.stdout:8/426: truncate d12/d13/d15/d1d/f4c 410016 0 2024-04-05T21:26:43.004 INFO:tasks.workunit.client.1.smithi188.stdout:5/638: creat d6/d15/d5c/d7a/d44/d98/dbf/fe0 x:0 0 0 2024-04-05T21:26:43.004 INFO:tasks.workunit.client.0.smithi121.stdout:8/427: mkdir d12/d50/d6a/d63/d8b 0 2024-04-05T21:26:43.006 INFO:tasks.workunit.client.1.smithi188.stdout:0/603: chown d4/d6/d13/d23/dd1 22 1 2024-04-05T21:26:43.006 INFO:tasks.workunit.client.0.smithi121.stdout:8/428: write d12/f89 [649599,86597] 0 2024-04-05T21:26:43.007 INFO:tasks.workunit.client.0.smithi121.stdout:8/429: dread - d12/d13/d56/d6d/f8a zero size 2024-04-05T21:26:43.008 INFO:tasks.workunit.client.1.smithi188.stdout:2/736: write d5/db/dd/d67/dcd/d46/d66/fc9 [379244,70130] 0 2024-04-05T21:26:43.011 INFO:tasks.workunit.client.0.smithi121.stdout:8/430: symlink d12/d13/d15/d1d/d29/l8c 0 2024-04-05T21:26:43.011 INFO:tasks.workunit.client.1.smithi188.stdout:0/604: creat d4/d6/d13/d23/dd1/fdd x:0 0 0 2024-04-05T21:26:43.012 INFO:tasks.workunit.client.0.smithi121.stdout:8/431: truncate d12/d13/d56/d57/f74 523960 0 2024-04-05T21:26:43.012 INFO:tasks.workunit.client.1.smithi188.stdout:4/940: write d0/fbf [281472,56251] 0 2024-04-05T21:26:43.015 INFO:tasks.workunit.client.0.smithi121.stdout:8/432: link d12/d13/d56/d57/f74 d12/d50/d6a/d63/d8b/f8d 0 2024-04-05T21:26:43.017 INFO:tasks.workunit.client.1.smithi188.stdout:7/905: dread - d3/d2c/dae/dfd/d2e/d8c/db7/dd6/fde zero size 2024-04-05T21:26:43.020 INFO:tasks.workunit.client.1.smithi188.stdout:2/737: mknod d5/db/dd/d67/dcd/cf0 0 2024-04-05T21:26:43.021 INFO:tasks.workunit.client.1.smithi188.stdout:4/941: fsync d0/d2/d17/f50 0 2024-04-05T21:26:43.021 INFO:tasks.workunit.client.0.smithi121.stdout:8/433: dwrite d12/d13/d56/d57/f88 [0,4194304] 0 2024-04-05T21:26:43.023 INFO:tasks.workunit.client.0.smithi121.stdout:8/434: write d12/d13/d15/d1d/f36 [1907526,57748] 0 2024-04-05T21:26:43.025 INFO:tasks.workunit.client.1.smithi188.stdout:1/702: dread d1/dc/d16/d46/d88/d13/f50 [0,4194304] 0 2024-04-05T21:26:43.026 INFO:tasks.workunit.client.1.smithi188.stdout:6/741: dread d6/d7/f15 [0,4194304] 0 2024-04-05T21:26:43.026 INFO:tasks.workunit.client.0.smithi121.stdout:8/435: mkdir d12/d50/d6a/d87/d8e 0 2024-04-05T21:26:43.027 INFO:tasks.workunit.client.1.smithi188.stdout:8/633: dread d5/d21/d2a/d47/f49 [0,4194304] 0 2024-04-05T21:26:43.029 INFO:tasks.workunit.client.1.smithi188.stdout:6/742: chown d6/d13/d18/d19/f27 8064611 1 2024-04-05T21:26:43.029 INFO:tasks.workunit.client.1.smithi188.stdout:8/634: dread - d5/d21/d80/fb2 zero size 2024-04-05T21:26:43.033 INFO:tasks.workunit.client.0.smithi121.stdout:8/436: dwrite d12/d13/d15/f4b [0,4194304] 0 2024-04-05T21:26:43.034 INFO:tasks.workunit.client.1.smithi188.stdout:9/715: sync 2024-04-05T21:26:43.035 INFO:tasks.workunit.client.1.smithi188.stdout:2/738: rename d5/db/dd/d36/d3c/d48/d7a/d7c/cba to d5/db/dd/d67/dcd/d46/d5d/cf1 0 2024-04-05T21:26:43.036 INFO:tasks.workunit.client.0.smithi121.stdout:8/437: write d12/d13/d56/d57/f7b [672377,61305] 0 2024-04-05T21:26:43.037 INFO:tasks.workunit.client.1.smithi188.stdout:2/739: stat d5/db/dd/d36/d3c/d48/dcf 0 2024-04-05T21:26:43.050 INFO:tasks.workunit.client.1.smithi188.stdout:8/635: unlink d5/dc/l25 0 2024-04-05T21:26:43.050 INFO:tasks.workunit.client.1.smithi188.stdout:0/605: getdents d4/d6/d13/dc1 0 2024-04-05T21:26:43.050 INFO:tasks.workunit.client.1.smithi188.stdout:9/716: creat dd/d3d/d89/dd1/ffb x:0 0 0 2024-04-05T21:26:43.050 INFO:tasks.workunit.client.1.smithi188.stdout:5/639: write d6/d17/f46 [459874,9947] 0 2024-04-05T21:26:43.053 INFO:tasks.workunit.client.1.smithi188.stdout:4/942: mknod d0/d14/d15/d1b/d132/c142 0 2024-04-05T21:26:43.053 INFO:tasks.workunit.client.1.smithi188.stdout:9/717: truncate dd/d11/d1f/fdd 1022999 0 2024-04-05T21:26:43.054 INFO:tasks.workunit.client.1.smithi188.stdout:0/606: creat d4/d8/db/d42/d71/d80/fde x:0 0 0 2024-04-05T21:26:43.056 INFO:tasks.workunit.client.1.smithi188.stdout:7/906: dread d3/d4/d1d/d74/f80 [0,4194304] 0 2024-04-05T21:26:43.058 INFO:tasks.workunit.client.1.smithi188.stdout:8/636: rename d5/d21/d40/d43/d44/d5b/d99/d69/dbe to d5/dc/d8f/dd1 0 2024-04-05T21:26:43.059 INFO:tasks.workunit.client.1.smithi188.stdout:1/703: write d1/dc/d16/d2d/f7e [2062843,63446] 0 2024-04-05T21:26:43.061 INFO:tasks.workunit.client.1.smithi188.stdout:6/743: write d6/d13/d18/d3c/ddc/f80 [792487,39035] 0 2024-04-05T21:26:43.063 INFO:tasks.workunit.client.1.smithi188.stdout:4/943: fdatasync d0/d2/d17/f50 0 2024-04-05T21:26:43.066 INFO:tasks.workunit.client.1.smithi188.stdout:9/718: fsync dd/d11/d1f/d26/dc0/d6b/d40/d59/f7f 0 2024-04-05T21:26:43.066 INFO:tasks.workunit.client.1.smithi188.stdout:4/944: write d0/d14/d15/d78/db1/f7b [1212022,6526] 0 2024-04-05T21:26:43.068 INFO:tasks.workunit.client.1.smithi188.stdout:9/719: chown dd/d11/d1f/d26/dc0/fbf 508688 1 2024-04-05T21:26:43.068 INFO:tasks.workunit.client.1.smithi188.stdout:7/907: creat d3/d4/d1d/da3/f127 x:0 0 0 2024-04-05T21:26:43.070 INFO:tasks.workunit.client.1.smithi188.stdout:8/637: unlink d5/d4b/lcd 0 2024-04-05T21:26:43.074 INFO:tasks.workunit.client.1.smithi188.stdout:5/640: truncate d6/f2c 206711 0 2024-04-05T21:26:43.075 INFO:tasks.workunit.client.1.smithi188.stdout:6/744: chown d6/d13/d18/d19/d2c/d3f/c54 8 1 2024-04-05T21:26:43.077 INFO:tasks.workunit.client.1.smithi188.stdout:0/607: creat d4/d8/db/d42/d71/dc5/fdf x:0 0 0 2024-04-05T21:26:43.079 INFO:tasks.workunit.client.1.smithi188.stdout:0/608: read d4/d8/f75 [1722716,99510] 0 2024-04-05T21:26:43.079 INFO:tasks.workunit.client.1.smithi188.stdout:4/945: mknod d0/d14/d15/d53/d82/d8f/dbd/c143 0 2024-04-05T21:26:43.083 INFO:tasks.workunit.client.1.smithi188.stdout:7/908: dread d3/f68 [0,4194304] 0 2024-04-05T21:26:43.084 INFO:tasks.workunit.client.1.smithi188.stdout:2/740: getdents d5/db/dd/d67/dcd/d46/d66/d8f/dcb/dce 0 2024-04-05T21:26:43.084 INFO:tasks.workunit.client.1.smithi188.stdout:5/641: mkdir d6/d17/d39/de1 0 2024-04-05T21:26:43.086 INFO:tasks.workunit.client.0.smithi121.stdout:7/487: sync 2024-04-05T21:26:43.087 INFO:tasks.workunit.client.0.smithi121.stdout:7/488: write d0/d23/f8b [2804645,24681] 0 2024-04-05T21:26:43.087 INFO:tasks.workunit.client.1.smithi188.stdout:6/745: rmdir d6/d71 39 2024-04-05T21:26:43.087 INFO:tasks.workunit.client.1.smithi188.stdout:2/741: chown d5/db/dd/d36/d3c/d48/d78/ldf 6984271 1 2024-04-05T21:26:43.089 INFO:tasks.workunit.client.1.smithi188.stdout:1/704: dwrite d1/d7/f39 [0,4194304] 0 2024-04-05T21:26:43.089 INFO:tasks.workunit.client.0.smithi121.stdout:7/489: mknod d0/d9/d22/d28/d56/c96 0 2024-04-05T21:26:43.093 INFO:tasks.workunit.client.1.smithi188.stdout:9/720: creat dd/d11/d1f/d26/dc0/d6b/d40/dd7/ffc x:0 0 0 2024-04-05T21:26:43.093 INFO:tasks.workunit.client.1.smithi188.stdout:0/609: rmdir d4/d6/d13/d23/d24/d57 39 2024-04-05T21:26:43.094 INFO:tasks.workunit.client.0.smithi121.stdout:7/490: dwrite d0/f8 [0,4194304] 0 2024-04-05T21:26:43.095 INFO:tasks.workunit.client.0.smithi121.stdout:7/491: chown d0/d32/f6b 0 1 2024-04-05T21:26:43.097 INFO:tasks.workunit.client.1.smithi188.stdout:7/909: truncate d3/d4/fb9 2643064 0 2024-04-05T21:26:43.099 INFO:tasks.workunit.client.1.smithi188.stdout:0/610: write d4/d6/d61/f93 [757562,61143] 0 2024-04-05T21:26:43.103 INFO:tasks.workunit.client.0.smithi121.stdout:7/492: rename d0/d23/d5a/l5f to d0/d23/l97 0 2024-04-05T21:26:43.104 INFO:tasks.workunit.client.0.smithi121.stdout:7/493: getdents d0/d23/d5b/d78 0 2024-04-05T21:26:43.106 INFO:tasks.workunit.client.0.smithi121.stdout:7/494: symlink d0/d23/d5b/d52/d61/l98 0 2024-04-05T21:26:43.107 INFO:tasks.workunit.client.1.smithi188.stdout:9/721: dwrite dd/d11/d22/f4d [0,4194304] 0 2024-04-05T21:26:43.108 INFO:tasks.workunit.client.1.smithi188.stdout:1/705: dread d1/dc/d16/d21/dbb/fc9 [0,4194304] 0 2024-04-05T21:26:43.110 INFO:tasks.workunit.client.0.smithi121.stdout:7/495: link d0/d9/c10 d0/d9/d22/d28/c99 0 2024-04-05T21:26:43.111 INFO:tasks.workunit.client.1.smithi188.stdout:2/742: rename d5/c1e to d5/cf2 0 2024-04-05T21:26:43.116 INFO:tasks.workunit.client.1.smithi188.stdout:7/910: dread d3/d4/d7c/ff0 [0,4194304] 0 2024-04-05T21:26:43.119 INFO:tasks.workunit.client.1.smithi188.stdout:4/946: creat d0/d2/d13b/f144 x:0 0 0 2024-04-05T21:26:43.119 INFO:tasks.workunit.client.1.smithi188.stdout:1/706: readlink d1/dc/d16/d2d/dd5/l9d 0 2024-04-05T21:26:43.121 INFO:tasks.workunit.client.1.smithi188.stdout:0/611: truncate d4/f53 1487438 0 2024-04-05T21:26:43.123 INFO:tasks.workunit.client.1.smithi188.stdout:8/638: dread d5/d21/d40/d43/f87 [0,4194304] 0 2024-04-05T21:26:43.124 INFO:tasks.workunit.client.1.smithi188.stdout:4/947: write d0/d14/d15/d53/f9f [5103431,25901] 0 2024-04-05T21:26:43.133 INFO:tasks.workunit.client.1.smithi188.stdout:5/642: creat d6/d15/d5c/d7a/d44/fe2 x:0 0 0 2024-04-05T21:26:43.133 INFO:tasks.workunit.client.1.smithi188.stdout:7/911: creat d3/d2c/dae/dfd/d55/d9d/f128 x:0 0 0 2024-04-05T21:26:43.133 INFO:tasks.workunit.client.1.smithi188.stdout:9/722: dwrite dd/d11/fd2 [0,4194304] 0 2024-04-05T21:26:43.133 INFO:tasks.workunit.client.1.smithi188.stdout:1/707: rename d1/dc/d16/d21/dc8 to d1/d7/d4e/d5f/ddc 0 2024-04-05T21:26:43.134 INFO:tasks.workunit.client.1.smithi188.stdout:0/612: truncate d4/d6/d13/dc1/fc9 92344 0 2024-04-05T21:26:43.135 INFO:tasks.workunit.client.1.smithi188.stdout:8/639: creat d5/d6e/d86/dcb/fd2 x:0 0 0 2024-04-05T21:26:43.135 INFO:tasks.workunit.client.1.smithi188.stdout:4/948: fdatasync d0/d2/d2a/d34/fa1 0 2024-04-05T21:26:43.138 INFO:tasks.workunit.client.1.smithi188.stdout:7/912: write d3/d2c/f4c [1174334,14865] 0 2024-04-05T21:26:43.139 INFO:tasks.workunit.client.1.smithi188.stdout:2/743: creat d5/db/dd/ff3 x:0 0 0 2024-04-05T21:26:43.142 INFO:tasks.workunit.client.0.smithi121.stdout:7/496: dread d0/f6 [0,4194304] 0 2024-04-05T21:26:43.143 INFO:tasks.workunit.client.1.smithi188.stdout:4/949: creat d0/d14/d15/dd4/f145 x:0 0 0 2024-04-05T21:26:43.145 INFO:tasks.workunit.client.0.smithi121.stdout:7/497: creat d0/d9/d22/d81/f9a x:0 0 0 2024-04-05T21:26:43.145 INFO:tasks.workunit.client.1.smithi188.stdout:9/723: creat dd/d11/d1f/d26/d55/dcd/ffd x:0 0 0 2024-04-05T21:26:43.146 INFO:tasks.workunit.client.0.smithi121.stdout:7/498: readlink d0/d9/d22/d3e/l30 0 2024-04-05T21:26:43.146 INFO:tasks.workunit.client.1.smithi188.stdout:8/640: mknod d5/d21/d2a/d71/cd3 0 2024-04-05T21:26:43.147 INFO:tasks.workunit.client.0.smithi121.stdout:7/499: mknod d0/d9/d22/d28/d3b/c9b 0 2024-04-05T21:26:43.148 INFO:tasks.workunit.client.1.smithi188.stdout:0/613: dwrite d4/d8/d10/f67 [0,4194304] 0 2024-04-05T21:26:43.148 INFO:tasks.workunit.client.0.smithi121.stdout:7/500: write d0/d9/d22/d28/d3b/f6f [1337072,62279] 0 2024-04-05T21:26:43.149 INFO:tasks.workunit.client.1.smithi188.stdout:4/950: stat d0/d14/d15/d42/l140 0 2024-04-05T21:26:43.150 INFO:tasks.workunit.client.0.smithi121.stdout:7/501: fdatasync d0/d23/f74 0 2024-04-05T21:26:43.151 INFO:tasks.workunit.client.0.smithi121.stdout:7/502: mkdir d0/d9/d22/d3e/d9c 0 2024-04-05T21:26:43.152 INFO:tasks.workunit.client.0.smithi121.stdout:7/503: truncate d0/d23/f74 890392 0 2024-04-05T21:26:43.152 INFO:tasks.workunit.client.0.smithi121.stdout:7/504: readlink d0/d9/d22/l71 0 2024-04-05T21:26:43.153 INFO:tasks.workunit.client.0.smithi121.stdout:7/505: read - d0/d32/d4a/f53 zero size 2024-04-05T21:26:43.155 INFO:tasks.workunit.client.1.smithi188.stdout:9/724: mkdir dd/d11/d1f/d26/d55/d98/dfe 0 2024-04-05T21:26:43.155 INFO:tasks.workunit.client.1.smithi188.stdout:7/913: mkdir d3/d2c/dae/dfd/d2e/d3b/db3/d129 0 2024-04-05T21:26:43.157 INFO:tasks.workunit.client.0.smithi121.stdout:7/506: link d0/d9/c10 d0/d32/d4a/c9d 0 2024-04-05T21:26:43.157 INFO:tasks.workunit.client.1.smithi188.stdout:5/643: getdents d6/d17/d23/d85 0 2024-04-05T21:26:43.158 INFO:tasks.workunit.client.1.smithi188.stdout:7/914: write d3/d4/d1d/d74/f86 [705993,35047] 0 2024-04-05T21:26:43.158 INFO:tasks.workunit.client.1.smithi188.stdout:1/708: creat d1/dc/d16/d46/d88/fdd x:0 0 0 2024-04-05T21:26:43.158 INFO:tasks.workunit.client.0.smithi121.stdout:7/507: symlink d0/l9e 0 2024-04-05T21:26:43.161 INFO:tasks.workunit.client.1.smithi188.stdout:9/725: write dd/d11/d22/d7e/de3/fef [646551,90276] 0 2024-04-05T21:26:43.162 INFO:tasks.workunit.client.1.smithi188.stdout:0/614: link d4/d6/d13/da1/fa3 d4/d8/db/fe0 0 2024-04-05T21:26:43.164 INFO:tasks.workunit.client.1.smithi188.stdout:1/709: creat d1/dc/d16/d46/d88/d86/fde x:0 0 0 2024-04-05T21:26:43.166 INFO:tasks.workunit.client.1.smithi188.stdout:1/710: readlink d1/dc/d16/d21/l94 0 2024-04-05T21:26:43.168 INFO:tasks.workunit.client.1.smithi188.stdout:0/615: creat d4/d8/d78/fe1 x:0 0 0 2024-04-05T21:26:43.169 INFO:tasks.workunit.client.1.smithi188.stdout:0/616: read d4/d8/db/fe0 [2893801,19891] 0 2024-04-05T21:26:43.172 INFO:tasks.workunit.client.0.smithi121.stdout:7/508: write d0/d9/d22/f79 [354594,20736] 0 2024-04-05T21:26:43.175 INFO:tasks.workunit.client.1.smithi188.stdout:0/617: creat d4/d8/db/d21/fe2 x:0 0 0 2024-04-05T21:26:43.176 INFO:tasks.workunit.client.1.smithi188.stdout:1/711: dwrite d1/dc/f1b [4194304,4194304] 0 2024-04-05T21:26:43.176 INFO:tasks.workunit.client.1.smithi188.stdout:5/644: mkdir d6/d17/d39/de1/de3 0 2024-04-05T21:26:43.178 INFO:tasks.workunit.client.0.smithi121.stdout:7/509: link d0/c4 d0/d9/d22/d3e/d76/c9f 0 2024-04-05T21:26:43.180 INFO:tasks.workunit.client.1.smithi188.stdout:7/915: getdents d3/d2c/dae/dfd/d10 0 2024-04-05T21:26:43.181 INFO:tasks.workunit.client.1.smithi188.stdout:7/916: chown d3/d2c/f126 435792 1 2024-04-05T21:26:43.181 INFO:tasks.workunit.client.1.smithi188.stdout:2/744: write d5/db/dd/d36/d3c/d48/f56 [4327362,103868] 0 2024-04-05T21:26:43.181 INFO:tasks.workunit.client.0.smithi121.stdout:7/510: creat d0/d32/fa0 x:0 0 0 2024-04-05T21:26:43.184 INFO:tasks.workunit.client.0.smithi121.stdout:7/511: symlink d0/d9/d22/d3e/la1 0 2024-04-05T21:26:43.187 INFO:tasks.workunit.client.0.smithi121.stdout:7/512: write d0/f3 [779103,73996] 0 2024-04-05T21:26:43.188 INFO:tasks.workunit.client.1.smithi188.stdout:8/641: dwrite d5/d6e/f84 [0,4194304] 0 2024-04-05T21:26:43.188 INFO:tasks.workunit.client.1.smithi188.stdout:2/745: write d5/db/dd/d36/d3c/f4a [1774149,2892] 0 2024-04-05T21:26:43.190 INFO:tasks.workunit.client.0.smithi121.stdout:7/513: unlink d0/d9/d22/d28/c7e 0 2024-04-05T21:26:43.191 INFO:tasks.workunit.client.0.smithi121.stdout:7/514: readlink d0/d9/d22/d3e/l87 0 2024-04-05T21:26:43.193 INFO:tasks.workunit.client.0.smithi121.stdout:7/515: write d0/d23/d5b/f43 [1186564,15758] 0 2024-04-05T21:26:43.198 INFO:tasks.workunit.client.0.smithi121.stdout:7/516: mkdir d0/d9/d22/d81/da2 0 2024-04-05T21:26:43.201 INFO:tasks.workunit.client.1.smithi188.stdout:9/726: dread fb [0,4194304] 0 2024-04-05T21:26:43.201 INFO:tasks.workunit.client.1.smithi188.stdout:2/746: creat d5/db/dd/d36/d3c/d63/ff4 x:0 0 0 2024-04-05T21:26:43.204 INFO:tasks.workunit.client.1.smithi188.stdout:5/645: rename d6/l28 to d6/d15/d1b/ddc/le4 0 2024-04-05T21:26:43.206 INFO:tasks.workunit.client.1.smithi188.stdout:2/747: unlink d5/db/d35/fab 0 2024-04-05T21:26:43.209 INFO:tasks.workunit.client.1.smithi188.stdout:7/917: getdents d3/d2c/dae/dfd/d2e/d37 0 2024-04-05T21:26:43.209 INFO:tasks.workunit.client.1.smithi188.stdout:0/618: getdents d4/d8/db 0 2024-04-05T21:26:43.211 INFO:tasks.workunit.client.1.smithi188.stdout:9/727: dwrite dd/d11/d1f/d26/dc0/d6b/d40/d59/fdc [0,4194304] 0 2024-04-05T21:26:43.211 INFO:tasks.workunit.client.1.smithi188.stdout:8/642: read d5/d21/d40/d43/d44/d5b/d99/da3/fa5 [721908,27870] 0 2024-04-05T21:26:43.212 INFO:tasks.workunit.client.0.smithi121.stdout:7/517: dread d0/d9/d22/d28/d3b/f6f [0,4194304] 0 2024-04-05T21:26:43.212 INFO:tasks.workunit.client.1.smithi188.stdout:5/646: mknod d6/d82/d40/ce5 0 2024-04-05T21:26:43.213 INFO:tasks.workunit.client.0.smithi121.stdout:7/518: creat d0/d9/d22/fa3 x:0 0 0 2024-04-05T21:26:43.214 INFO:tasks.workunit.client.0.smithi121.stdout:7/519: chown d0/d23/c36 14498 1 2024-04-05T21:26:43.216 INFO:tasks.workunit.client.0.smithi121.stdout:6/376: sync 2024-04-05T21:26:43.216 INFO:tasks.workunit.client.0.smithi121.stdout:3/358: sync 2024-04-05T21:26:43.216 INFO:tasks.workunit.client.0.smithi121.stdout:0/522: sync 2024-04-05T21:26:43.217 INFO:tasks.workunit.client.0.smithi121.stdout:7/520: mkdir d0/d23/da4 0 2024-04-05T21:26:43.217 INFO:tasks.workunit.client.1.smithi188.stdout:0/619: chown d4/d6/f15 232 1 2024-04-05T21:26:43.218 INFO:tasks.workunit.client.0.smithi121.stdout:3/359: chown d0/d3/d24/l52 4930 1 2024-04-05T21:26:43.218 INFO:tasks.workunit.client.0.smithi121.stdout:6/377: write d1/d2/d3/d5/fd [1672404,33328] 0 2024-04-05T21:26:43.218 INFO:tasks.workunit.client.0.smithi121.stdout:6/378: stat d1/d2 0 2024-04-05T21:26:43.219 INFO:tasks.workunit.client.1.smithi188.stdout:2/748: truncate d5/db/dd/d67/dcd/d46/faa 1887936 0 2024-04-05T21:26:43.221 INFO:tasks.workunit.client.1.smithi188.stdout:1/712: dwrite d1/dc/d16/d46/d88/f79 [0,4194304] 0 2024-04-05T21:26:43.224 INFO:tasks.workunit.client.0.smithi121.stdout:3/360: truncate d0/f2d 1210509 0 2024-04-05T21:26:43.224 INFO:tasks.workunit.client.0.smithi121.stdout:7/521: link d0/d23/f54 d0/d9/d22/d81/da2/fa5 0 2024-04-05T21:26:43.226 INFO:tasks.workunit.client.0.smithi121.stdout:7/522: chown d0/d32/f41 61 1 2024-04-05T21:26:43.228 INFO:tasks.workunit.client.0.smithi121.stdout:6/379: dwrite d1/d2/d3/d5/d1f/f27 [4194304,4194304] 0 2024-04-05T21:26:43.228 INFO:tasks.workunit.client.1.smithi188.stdout:7/918: truncate d3/d2c/d5e/f78 2206198 0 2024-04-05T21:26:43.228 INFO:tasks.workunit.client.1.smithi188.stdout:5/647: dread d6/d17/fb8 [0,4194304] 0 2024-04-05T21:26:43.228 INFO:tasks.workunit.client.1.smithi188.stdout:3/745: sync 2024-04-05T21:26:43.233 INFO:tasks.workunit.client.1.smithi188.stdout:3/746: rmdir d5/d35 39 2024-04-05T21:26:43.233 INFO:tasks.workunit.client.1.smithi188.stdout:2/749: write d5/db/dd/d36/d3c/d48/d7a/d7c/f86 [1549372,126162] 0 2024-04-05T21:26:43.234 INFO:tasks.workunit.client.0.smithi121.stdout:3/361: link d0/d3/de/l67 d0/d3/de/l7a 0 2024-04-05T21:26:43.235 INFO:tasks.workunit.client.1.smithi188.stdout:7/919: mkdir d3/d2c/dae/dfd/d2e/dd8/d12a 0 2024-04-05T21:26:43.237 INFO:tasks.workunit.client.1.smithi188.stdout:5/648: truncate d6/d15/d1b/f1e 2336324 0 2024-04-05T21:26:43.238 INFO:tasks.workunit.client.1.smithi188.stdout:3/747: readlink d5/d7/d65/d61/d66/lb9 0 2024-04-05T21:26:43.241 INFO:tasks.workunit.client.0.smithi121.stdout:0/523: dwrite d7/dc/fab [0,4194304] 0 2024-04-05T21:26:43.242 INFO:tasks.workunit.client.0.smithi121.stdout:0/524: chown d7/d71/c99 58211 1 2024-04-05T21:26:43.243 INFO:tasks.workunit.client.0.smithi121.stdout:0/525: write d7/d71/d29/d3f/fa1 [2902718,64833] 0 2024-04-05T21:26:43.244 INFO:tasks.workunit.client.1.smithi188.stdout:2/750: mknod d5/db/dd/d24/dc8/cf5 0 2024-04-05T21:26:43.245 INFO:tasks.workunit.client.1.smithi188.stdout:5/649: fdatasync d6/f73 0 2024-04-05T21:26:43.246 INFO:tasks.workunit.client.0.smithi121.stdout:0/526: symlink d7/d62/lb8 0 2024-04-05T21:26:43.248 INFO:tasks.workunit.client.1.smithi188.stdout:0/620: dwrite d4/d8/db/d42/d38/d77/f86 [0,4194304] 0 2024-04-05T21:26:43.248 INFO:tasks.workunit.client.0.smithi121.stdout:0/527: creat d7/d62/d9d/da6/fb9 x:0 0 0 2024-04-05T21:26:43.250 INFO:tasks.workunit.client.0.smithi121.stdout:0/528: creat d7/d71/d29/d3f/fba x:0 0 0 2024-04-05T21:26:43.253 INFO:tasks.workunit.client.0.smithi121.stdout:0/529: readlink d7/d71/d45/d46/d53/l9a 0 2024-04-05T21:26:43.256 INFO:tasks.workunit.client.0.smithi121.stdout:0/530: creat d7/d62/d9d/fbb x:0 0 0 2024-04-05T21:26:43.256 INFO:tasks.workunit.client.1.smithi188.stdout:1/713: dwrite d1/dc/d16/d2d/fd1 [4194304,4194304] 0 2024-04-05T21:26:43.256 INFO:tasks.workunit.client.1.smithi188.stdout:9/728: truncate dd/d11/d73/d85/fda 1385458 0 2024-04-05T21:26:43.258 INFO:tasks.workunit.client.0.smithi121.stdout:6/380: write d1/f38 [264027,100866] 0 2024-04-05T21:26:43.259 INFO:tasks.workunit.client.1.smithi188.stdout:1/714: stat d1/dc/d16/d46/d88/d13/f6a 0 2024-04-05T21:26:43.261 INFO:tasks.workunit.client.1.smithi188.stdout:0/621: creat d4/d8/db/d42/fe3 x:0 0 0 2024-04-05T21:26:43.261 INFO:tasks.workunit.client.0.smithi121.stdout:3/362: truncate d0/d3/d24/d3d/d1a/d1e/f2a 178837 0 2024-04-05T21:26:43.262 INFO:tasks.workunit.client.0.smithi121.stdout:6/381: write d1/d2/d1b/d1e/f2d [2267446,4236] 0 2024-04-05T21:26:43.263 INFO:tasks.workunit.client.0.smithi121.stdout:0/531: truncate d7/d71/d29/d3f/f50 1697874 0 2024-04-05T21:26:43.264 INFO:tasks.workunit.client.0.smithi121.stdout:0/532: fdatasync d7/d71/d45/d5a/fa9 0 2024-04-05T21:26:43.266 INFO:tasks.workunit.client.0.smithi121.stdout:3/363: mkdir d0/d3/d24/d6d/d7b 0 2024-04-05T21:26:43.266 INFO:tasks.workunit.client.1.smithi188.stdout:0/622: chown d4/d8/db/d42/d38/d77/c8e 30950152 1 2024-04-05T21:26:43.266 INFO:tasks.workunit.client.0.smithi121.stdout:6/382: creat d1/d4b/f80 x:0 0 0 2024-04-05T21:26:43.268 INFO:tasks.workunit.client.0.smithi121.stdout:0/533: chown d7/d71/d45/d5a/cb5 842304094 1 2024-04-05T21:26:43.268 INFO:tasks.workunit.client.0.smithi121.stdout:3/364: fsync d0/d3/de/f36 0 2024-04-05T21:26:43.268 INFO:tasks.workunit.client.1.smithi188.stdout:7/920: dwrite d3/f68 [0,4194304] 0 2024-04-05T21:26:43.269 INFO:tasks.workunit.client.0.smithi121.stdout:3/365: chown d0/d3/f5e 7 1 2024-04-05T21:26:43.270 INFO:tasks.workunit.client.0.smithi121.stdout:6/383: symlink d1/d2/dc/l81 0 2024-04-05T21:26:43.271 INFO:tasks.workunit.client.1.smithi188.stdout:5/650: link d6/d17/c30 d6/d17/d39/de1/de3/ce6 0 2024-04-05T21:26:43.271 INFO:tasks.workunit.client.1.smithi188.stdout:9/729: creat dd/d11/d1f/d26/d55/dcf/fff x:0 0 0 2024-04-05T21:26:43.272 INFO:tasks.workunit.client.0.smithi121.stdout:0/534: symlink d7/dc/lbc 0 2024-04-05T21:26:43.272 INFO:tasks.workunit.client.1.smithi188.stdout:1/715: read d1/d7/d4e/f8b [835443,68481] 0 2024-04-05T21:26:43.273 INFO:tasks.workunit.client.0.smithi121.stdout:3/366: write d0/d3/d24/d3d/d1a/d23/f6b [170659,42466] 0 2024-04-05T21:26:43.273 INFO:tasks.workunit.client.1.smithi188.stdout:0/623: rmdir d4/d8/db/d42/d71/d80 39 2024-04-05T21:26:43.273 INFO:tasks.workunit.client.1.smithi188.stdout:8/643: dwrite d5/d21/d40/f57 [0,4194304] 0 2024-04-05T21:26:43.275 INFO:tasks.workunit.client.0.smithi121.stdout:6/384: creat d1/d2/d3/d4/d21/f82 x:0 0 0 2024-04-05T21:26:43.275 INFO:tasks.workunit.client.0.smithi121.stdout:3/367: chown d0/d3/d24/f42 116526068 1 2024-04-05T21:26:43.276 INFO:tasks.workunit.client.0.smithi121.stdout:0/535: fsync d7/d71/d29/f58 0 2024-04-05T21:26:43.277 INFO:tasks.workunit.client.1.smithi188.stdout:2/751: dwrite d5/db/dd/d67/dcd/d46/d9f/fa0 [0,4194304] 0 2024-04-05T21:26:43.277 INFO:tasks.workunit.client.0.smithi121.stdout:6/385: fdatasync d1/d2/d3/d5/d1f/f64 0 2024-04-05T21:26:43.277 INFO:tasks.workunit.client.0.smithi121.stdout:3/368: stat d0/d3/d24/d3d/d1a/d23/l61 0 2024-04-05T21:26:43.278 INFO:tasks.workunit.client.0.smithi121.stdout:3/369: chown d0/f27 56485 1 2024-04-05T21:26:43.279 INFO:tasks.workunit.client.1.smithi188.stdout:7/921: rmdir d3/d2c/dae/dfd/d2e/d3b/db3 39 2024-04-05T21:26:43.281 INFO:tasks.workunit.client.1.smithi188.stdout:7/922: readlink d3/da5/l102 0 2024-04-05T21:26:43.281 INFO:tasks.workunit.client.1.smithi188.stdout:5/651: chown d6/d82/d40/f9c 160152055 1 2024-04-05T21:26:43.283 INFO:tasks.workunit.client.0.smithi121.stdout:0/536: rename d7/d62/d3c/f4b to d7/d71/d29/d2a/fbd 0 2024-04-05T21:26:43.284 INFO:tasks.workunit.client.1.smithi188.stdout:2/752: write d5/fb7 [646019,104219] 0 2024-04-05T21:26:43.285 INFO:tasks.workunit.client.0.smithi121.stdout:0/537: write d7/d71/d45/d5a/fa9 [272855,127565] 0 2024-04-05T21:26:43.288 INFO:tasks.workunit.client.1.smithi188.stdout:5/652: dread d6/d17/d39/d8e/f8f [0,4194304] 0 2024-04-05T21:26:43.288 INFO:tasks.workunit.client.1.smithi188.stdout:9/730: rmdir dd/d3d/d81 39 2024-04-05T21:26:43.292 INFO:tasks.workunit.client.0.smithi121.stdout:0/538: mknod d7/d62/cbe 0 2024-04-05T21:26:43.293 INFO:tasks.workunit.client.0.smithi121.stdout:0/539: read - d7/d71/d29/d2a/d2c/f7c zero size 2024-04-05T21:26:43.294 INFO:tasks.workunit.client.1.smithi188.stdout:0/624: mknod d4/d8/db/d42/da2/ce4 0 2024-04-05T21:26:43.294 INFO:tasks.workunit.client.1.smithi188.stdout:8/644: creat d5/d21/d40/d43/d44/d5b/d99/d69/d9b/fd4 x:0 0 0 2024-04-05T21:26:43.296 INFO:tasks.workunit.client.1.smithi188.stdout:9/731: mknod dd/d11/d73/dbb/c100 0 2024-04-05T21:26:43.296 INFO:tasks.workunit.client.0.smithi121.stdout:0/540: creat d7/d71/d45/d46/d53/d8e/fbf x:0 0 0 2024-04-05T21:26:43.297 INFO:tasks.workunit.client.1.smithi188.stdout:8/645: chown d5/d6e/d98/dc6 75317 1 2024-04-05T21:26:43.298 INFO:tasks.workunit.client.0.smithi121.stdout:0/541: creat d7/d62/d9d/fc0 x:0 0 0 2024-04-05T21:26:43.300 INFO:tasks.workunit.client.1.smithi188.stdout:0/625: fdatasync d4/d8/db/d21/f90 0 2024-04-05T21:26:43.300 INFO:tasks.workunit.client.0.smithi121.stdout:0/542: creat d7/dc/da3/fc1 x:0 0 0 2024-04-05T21:26:43.301 INFO:tasks.workunit.client.1.smithi188.stdout:5/653: creat d6/d17/d39/de1/de3/fe7 x:0 0 0 2024-04-05T21:26:43.302 INFO:tasks.workunit.client.0.smithi121.stdout:0/543: creat d7/d62/fc2 x:0 0 0 2024-04-05T21:26:43.302 INFO:tasks.workunit.client.1.smithi188.stdout:7/923: dread d3/d2c/dae/dfd/d55/fe0 [0,4194304] 0 2024-04-05T21:26:43.303 INFO:tasks.workunit.client.1.smithi188.stdout:1/716: link d1/dc/d16/d46/d88/d13/f6a d1/dc/d16/d46/d95/fdf 0 2024-04-05T21:26:43.303 INFO:tasks.workunit.client.1.smithi188.stdout:3/748: dread d5/d7/d6d/d75/faa [0,4194304] 0 2024-04-05T21:26:43.304 INFO:tasks.workunit.client.0.smithi121.stdout:0/544: creat d7/d71/d29/d2a/d2c/fc3 x:0 0 0 2024-04-05T21:26:43.307 INFO:tasks.workunit.client.0.smithi121.stdout:0/545: mkdir d7/d71/d45/d46/d53/d8e/dc4 0 2024-04-05T21:26:43.307 INFO:tasks.workunit.client.1.smithi188.stdout:9/732: rename dd/d11/d1f/d26/dc0/d6b/d40/dd7 to dd/d11/d1f/d25/d8a/d101 0 2024-04-05T21:26:43.307 INFO:tasks.workunit.client.1.smithi188.stdout:2/753: dwrite d5/db/dd/d36/d3c/d48/d78/f83 [0,4194304] 0 2024-04-05T21:26:43.311 INFO:tasks.workunit.client.0.smithi121.stdout:0/546: unlink d7/d71/d29/d2a/d2c/c4d 0 2024-04-05T21:26:43.314 INFO:tasks.workunit.client.0.smithi121.stdout:6/386: dread d1/d2/d3/f4e [0,4194304] 0 2024-04-05T21:26:43.314 INFO:tasks.workunit.client.1.smithi188.stdout:0/626: creat d4/d8/db/fe5 x:0 0 0 2024-04-05T21:26:43.316 INFO:tasks.workunit.client.0.smithi121.stdout:3/370: truncate d0/d3/f5b 1853114 0 2024-04-05T21:26:43.316 INFO:tasks.workunit.client.0.smithi121.stdout:6/387: dread - d1/d2/dc/f5b zero size 2024-04-05T21:26:43.317 INFO:tasks.workunit.client.1.smithi188.stdout:8/646: chown d5/d21/d40/d43/d44/d59/f77 235278716 1 2024-04-05T21:26:43.318 INFO:tasks.workunit.client.0.smithi121.stdout:3/371: fdatasync d0/d3/de/f72 0 2024-04-05T21:26:43.319 INFO:tasks.workunit.client.0.smithi121.stdout:5/589: sync 2024-04-05T21:26:43.320 INFO:tasks.workunit.client.0.smithi121.stdout:9/340: sync 2024-04-05T21:26:43.320 INFO:tasks.workunit.client.0.smithi121.stdout:1/571: sync 2024-04-05T21:26:43.320 INFO:tasks.workunit.client.0.smithi121.stdout:4/606: sync 2024-04-05T21:26:43.320 INFO:tasks.workunit.client.0.smithi121.stdout:8/438: sync 2024-04-05T21:26:43.320 INFO:tasks.workunit.client.0.smithi121.stdout:2/426: sync 2024-04-05T21:26:43.320 INFO:tasks.workunit.client.1.smithi188.stdout:3/749: mknod d5/d7/d65/d61/db5/cfb 0 2024-04-05T21:26:43.321 INFO:tasks.workunit.client.0.smithi121.stdout:0/547: dwrite d7/d71/d29/f37 [4194304,4194304] 0 2024-04-05T21:26:43.322 INFO:tasks.workunit.client.1.smithi188.stdout:1/717: read d1/dc/f15 [655929,124079] 0 2024-04-05T21:26:43.325 INFO:tasks.workunit.client.0.smithi121.stdout:2/427: fdatasync d2/d1e/f52 0 2024-04-05T21:26:43.325 INFO:tasks.workunit.client.0.smithi121.stdout:6/388: creat d1/d2/d3/d4/d21/d57/d42/f83 x:0 0 0 2024-04-05T21:26:43.326 INFO:tasks.workunit.client.1.smithi188.stdout:0/627: symlink d4/d6/d13/d23/dd1/le6 0 2024-04-05T21:26:43.328 INFO:tasks.workunit.client.1.smithi188.stdout:3/750: symlink d5/df/lfc 0 2024-04-05T21:26:43.329 INFO:tasks.workunit.client.0.smithi121.stdout:0/548: chown d7/d71/c99 1724438116 1 2024-04-05T21:26:43.329 INFO:tasks.workunit.client.1.smithi188.stdout:1/718: mknod d1/db2/ce0 0 2024-04-05T21:26:43.330 INFO:tasks.workunit.client.0.smithi121.stdout:8/439: write d12/d50/d6a/d63/f64 [2975353,50007] 0 2024-04-05T21:26:43.330 INFO:tasks.workunit.client.0.smithi121.stdout:2/428: chown d2/d9/d38/d2d/d47/d51 467019490 1 2024-04-05T21:26:43.331 INFO:tasks.workunit.client.1.smithi188.stdout:1/719: fsync d1/dc/d16/d46/d88/fdd 0 2024-04-05T21:26:43.332 INFO:tasks.workunit.client.0.smithi121.stdout:3/372: symlink d0/d3/d24/d3d/d1a/l7c 0 2024-04-05T21:26:43.334 INFO:tasks.workunit.client.1.smithi188.stdout:3/751: mkdir d5/d7/d65/d61/d66/da4/dfd 0 2024-04-05T21:26:43.334 INFO:tasks.workunit.client.0.smithi121.stdout:5/590: mkdir dd/de/d19/d38/d45/d68/dcf 0 2024-04-05T21:26:43.335 INFO:tasks.workunit.client.0.smithi121.stdout:0/549: mknod d7/d62/d8b/cc5 0 2024-04-05T21:26:43.336 INFO:tasks.workunit.client.0.smithi121.stdout:1/572: link d8/d89/d9e/c4e d8/d8c/db6/d1a/d1b/d24/cba 0 2024-04-05T21:26:43.338 INFO:tasks.workunit.client.1.smithi188.stdout:7/924: write d3/d2c/dae/dfd/d10/f95 [110964,67264] 0 2024-04-05T21:26:43.338 INFO:tasks.workunit.client.1.smithi188.stdout:0/628: truncate d4/d8/db/fe5 227500 0 2024-04-05T21:26:43.339 INFO:tasks.workunit.client.0.smithi121.stdout:6/389: dwrite d1/d2/d3/d5/d1f/f46 [0,4194304] 0 2024-04-05T21:26:43.339 INFO:tasks.workunit.client.0.smithi121.stdout:9/341: rename d6/d32/c34 to d6/df/d1d/d2c/d37/d3a/c6e 0 2024-04-05T21:26:43.344 INFO:tasks.workunit.client.0.smithi121.stdout:5/591: write dd/de/d19/d3c/f83 [3590942,125358] 0 2024-04-05T21:26:43.344 INFO:tasks.workunit.client.1.smithi188.stdout:5/654: dwrite d6/d17/d39/f4a [0,4194304] 0 2024-04-05T21:26:43.346 INFO:tasks.workunit.client.1.smithi188.stdout:9/733: chown dd/d11/d73/d85/fda 390 1 2024-04-05T21:26:43.350 INFO:tasks.workunit.client.0.smithi121.stdout:5/592: stat dd/de/d19/d1c/d7b/d88/fb3 0 2024-04-05T21:26:43.351 INFO:tasks.workunit.client.0.smithi121.stdout:3/373: truncate d0/d3/f4 227462 0 2024-04-05T21:26:43.352 INFO:tasks.workunit.client.1.smithi188.stdout:2/754: dread d5/db/dd/fd4 [0,4194304] 0 2024-04-05T21:26:43.354 INFO:tasks.workunit.client.0.smithi121.stdout:9/342: mknod d6/df/d1d/d2c/d37/d3a/d49/c6f 0 2024-04-05T21:26:43.354 INFO:tasks.workunit.client.0.smithi121.stdout:3/374: dread - d0/d3/d24/d3d/d1a/f4e zero size 2024-04-05T21:26:43.355 INFO:tasks.workunit.client.1.smithi188.stdout:2/755: write d5/db/dd/d67/fad [995828,72553] 0 2024-04-05T21:26:43.357 INFO:tasks.workunit.client.1.smithi188.stdout:9/734: mkdir dd/d11/d1f/d26/dc0/d102 0 2024-04-05T21:26:43.360 INFO:tasks.workunit.client.1.smithi188.stdout:5/655: truncate d6/d15/d5c/d7a/d44/fda 98697 0 2024-04-05T21:26:43.361 INFO:tasks.workunit.client.0.smithi121.stdout:6/390: dwrite d1/d2/d3/d5/d1f/f27 [8388608,4194304] 0 2024-04-05T21:26:43.362 INFO:tasks.workunit.client.0.smithi121.stdout:1/573: truncate d8/d8c/db6/d1a/d1b/d4c/d76/f3d 4156045 0 2024-04-05T21:26:43.362 INFO:tasks.workunit.client.1.smithi188.stdout:7/925: creat d3/d4/d1d/d74/d10e/f12b x:0 0 0 2024-04-05T21:26:43.364 INFO:tasks.workunit.client.0.smithi121.stdout:9/343: write d6/df/d1d/f2d [1573765,17137] 0 2024-04-05T21:26:43.366 INFO:tasks.workunit.client.0.smithi121.stdout:4/607: truncate d5/d11/d28/d3b/f59 1518147 0 2024-04-05T21:26:43.366 INFO:tasks.workunit.client.1.smithi188.stdout:9/735: symlink dd/d11/d1f/l103 0 2024-04-05T21:26:43.368 INFO:tasks.workunit.client.0.smithi121.stdout:1/574: chown d8/d8c/db6/d1a/l6e 74202 1 2024-04-05T21:26:43.368 INFO:tasks.workunit.client.0.smithi121.stdout:2/429: dread d2/d9/d17/d3a/f6f [0,4194304] 0 2024-04-05T21:26:43.368 INFO:tasks.workunit.client.0.smithi121.stdout:8/440: truncate f3 8058323 0 2024-04-05T21:26:43.370 INFO:tasks.workunit.client.1.smithi188.stdout:9/736: stat dd/d35/f67 0 2024-04-05T21:26:43.372 INFO:tasks.workunit.client.0.smithi121.stdout:0/550: link d7/d71/d45/d46/d53/l9a d7/dc/da3/lc6 0 2024-04-05T21:26:43.372 INFO:tasks.workunit.client.0.smithi121.stdout:2/430: dread d2/d9/d12/f37 [0,4194304] 0 2024-04-05T21:26:43.374 INFO:tasks.workunit.client.0.smithi121.stdout:2/431: readlink d2/d9/l1b 0 2024-04-05T21:26:43.374 INFO:tasks.workunit.client.0.smithi121.stdout:0/551: write d7/d71/d45/d5a/f80 [274405,29108] 0 2024-04-05T21:26:43.375 INFO:tasks.workunit.client.0.smithi121.stdout:3/375: unlink d0/d3/d24/d3d/d1a/d23/d38/d55/c58 0 2024-04-05T21:26:43.375 INFO:tasks.workunit.client.0.smithi121.stdout:2/432: write d2/f1a [218622,112238] 0 2024-04-05T21:26:43.376 INFO:tasks.workunit.client.0.smithi121.stdout:6/391: write d1/d2/d3/d4/de/f26 [4784960,52807] 0 2024-04-05T21:26:43.381 INFO:tasks.workunit.client.0.smithi121.stdout:5/593: creat dd/de/d19/d1c/d7b/d88/d9c/db1/db9/fd0 x:0 0 0 2024-04-05T21:26:43.381 INFO:tasks.workunit.client.1.smithi188.stdout:7/926: creat d3/d2c/dae/dfd/d2e/d3b/d94/f12c x:0 0 0 2024-04-05T21:26:43.383 INFO:tasks.workunit.client.0.smithi121.stdout:9/344: symlink d6/df/d1d/d2c/d37/d3a/d49/d66/l70 0 2024-04-05T21:26:43.383 INFO:tasks.workunit.client.0.smithi121.stdout:4/608: unlink d5/d11/d2c/d9d/dae/faa 0 2024-04-05T21:26:43.384 INFO:tasks.workunit.client.1.smithi188.stdout:2/756: dwrite d5/db/dd/d67/f8e [4194304,4194304] 0 2024-04-05T21:26:43.384 INFO:tasks.workunit.client.1.smithi188.stdout:9/737: mknod dd/d3d/d89/c104 0 2024-04-05T21:26:43.385 INFO:tasks.workunit.client.0.smithi121.stdout:1/575: mkdir d8/d8c/db6/d1a/d1b/dbb 0 2024-04-05T21:26:43.385 INFO:tasks.workunit.client.1.smithi188.stdout:5/656: getdents d6/d15/d5c/d7a/d44/d9b 0 2024-04-05T21:26:43.387 INFO:tasks.workunit.client.0.smithi121.stdout:8/441: symlink d12/d13/d15/l8f 0 2024-04-05T21:26:43.388 INFO:tasks.workunit.client.1.smithi188.stdout:1/720: dwrite d1/dc/d16/d46/d88/f1d [0,4194304] 0 2024-04-05T21:26:43.389 INFO:tasks.workunit.client.0.smithi121.stdout:6/392: creat d1/d2/d3/d4/d21/f84 x:0 0 0 2024-04-05T21:26:43.390 INFO:tasks.workunit.client.1.smithi188.stdout:3/752: dwrite d5/d35/f9e [0,4194304] 0 2024-04-05T21:26:43.390 INFO:tasks.workunit.client.1.smithi188.stdout:0/629: dwrite d4/d8/db/d42/d38/d77/f95 [0,4194304] 0 2024-04-05T21:26:43.392 INFO:tasks.workunit.client.0.smithi121.stdout:9/345: rename d6/df/d50/f68 to d6/df/f71 0 2024-04-05T21:26:43.392 INFO:tasks.workunit.client.0.smithi121.stdout:6/393: chown d1/d2/d3/d5/f66 40104125 1 2024-04-05T21:26:43.393 INFO:tasks.workunit.client.0.smithi121.stdout:3/376: dwrite d0/d3/d24/d3f/f4b [0,4194304] 0 2024-04-05T21:26:43.393 INFO:tasks.workunit.client.0.smithi121.stdout:0/552: symlink d7/d71/d45/d46/d53/d8e/dc4/lc7 0 2024-04-05T21:26:43.394 INFO:tasks.workunit.client.1.smithi188.stdout:9/738: fdatasync dd/d11/d22/f4d 0 2024-04-05T21:26:43.395 INFO:tasks.workunit.client.0.smithi121.stdout:6/394: fdatasync d1/d2/d1b/d1e/f2d 0 2024-04-05T21:26:43.400 INFO:tasks.workunit.client.1.smithi188.stdout:0/630: read d4/d6/d13/da1/fa3 [3546860,23180] 0 2024-04-05T21:26:43.401 INFO:tasks.workunit.client.1.smithi188.stdout:2/757: creat d5/db/dd/d36/d3c/d63/d75/d87/ff6 x:0 0 0 2024-04-05T21:26:43.401 INFO:tasks.workunit.client.1.smithi188.stdout:0/631: dread - d4/d8/fb4 zero size 2024-04-05T21:26:43.401 INFO:tasks.workunit.client.1.smithi188.stdout:1/721: write d1/d7/d4e/fb6 [1335551,96145] 0 2024-04-05T21:26:43.404 INFO:tasks.workunit.client.1.smithi188.stdout:5/657: rmdir d6/d17/d23 39 2024-04-05T21:26:43.405 INFO:tasks.workunit.client.0.smithi121.stdout:6/395: dread d1/d2/d3/d5/f58 [0,4194304] 0 2024-04-05T21:26:43.407 INFO:tasks.workunit.client.1.smithi188.stdout:3/753: mkdir d5/d7/d65/d61/d66/d6e/dfe 0 2024-04-05T21:26:43.408 INFO:tasks.workunit.client.0.smithi121.stdout:4/609: rmdir d5/d11/d28/d3b/db2 0 2024-04-05T21:26:43.410 INFO:tasks.workunit.client.0.smithi121.stdout:9/346: dwrite d6/f5e [0,4194304] 0 2024-04-05T21:26:43.410 INFO:tasks.workunit.client.1.smithi188.stdout:0/632: link d4/f5f d4/d6/d46/fe7 0 2024-04-05T21:26:43.411 INFO:tasks.workunit.client.0.smithi121.stdout:4/610: write d5/d11/d28/f1b [4614271,85609] 0 2024-04-05T21:26:43.413 INFO:tasks.workunit.client.0.smithi121.stdout:3/377: rename d0/d3/d24/d3d/d1a/l7c to d0/d3/d24/d3d/d1a/d23/d38/d55/l7d 0 2024-04-05T21:26:43.414 INFO:tasks.workunit.client.1.smithi188.stdout:7/927: rename d3/d2c/dae/dfd/d2e/d3b/db3/d129 to d3/da5/dc8/d12d 0 2024-04-05T21:26:43.415 INFO:tasks.workunit.client.0.smithi121.stdout:1/576: dread d8/d8c/db6/d1a/f66 [0,4194304] 0 2024-04-05T21:26:43.416 INFO:tasks.workunit.client.1.smithi188.stdout:7/928: write d3/f68 [1280775,128737] 0 2024-04-05T21:26:43.417 INFO:tasks.workunit.client.0.smithi121.stdout:8/442: getdents d12/d13/d15/d1d/d29 0 2024-04-05T21:26:43.418 INFO:tasks.workunit.client.1.smithi188.stdout:2/758: dwrite d5/db/dd/d24/f52 [0,4194304] 0 2024-04-05T21:26:43.421 INFO:tasks.workunit.client.0.smithi121.stdout:6/396: mknod d1/d2/d1b/d5e/c85 0 2024-04-05T21:26:43.422 INFO:tasks.workunit.client.1.smithi188.stdout:0/633: mkdir d4/d6/d13/d23/de8 0 2024-04-05T21:26:43.422 INFO:tasks.workunit.client.1.smithi188.stdout:3/754: rename d5/df/f15 to d5/d7/d6d/d75/d82/fff 0 2024-04-05T21:26:43.423 INFO:tasks.workunit.client.0.smithi121.stdout:1/577: fsync d8/d8c/db6/d1a/f66 0 2024-04-05T21:26:43.425 INFO:tasks.workunit.client.0.smithi121.stdout:8/443: creat d12/d50/d6a/f90 x:0 0 0 2024-04-05T21:26:43.425 INFO:tasks.workunit.client.0.smithi121.stdout:6/397: dread - d1/d2/d3/d5/d1f/f64 zero size 2024-04-05T21:26:43.426 INFO:tasks.workunit.client.1.smithi188.stdout:2/759: creat d5/db/dd/d36/d3c/d48/d7a/ff7 x:0 0 0 2024-04-05T21:26:43.426 INFO:tasks.workunit.client.0.smithi121.stdout:9/347: mkdir d6/df/d1d/d72 0 2024-04-05T21:26:43.427 INFO:tasks.workunit.client.1.smithi188.stdout:5/658: dread d6/d82/d40/d42/f56 [0,4194304] 0 2024-04-05T21:26:43.428 INFO:tasks.workunit.client.0.smithi121.stdout:8/444: chown d12/d13/d15/f4b 52257 1 2024-04-05T21:26:43.428 INFO:tasks.workunit.client.0.smithi121.stdout:0/553: link d7/d71/c49 d7/d71/d29/d2a/d39/cc8 0 2024-04-05T21:26:43.428 INFO:tasks.workunit.client.0.smithi121.stdout:3/378: dwrite d0/d3/d24/d3d/d1a/d23/d38/d55/f77 [0,4194304] 0 2024-04-05T21:26:43.429 INFO:tasks.workunit.client.0.smithi121.stdout:4/611: mkdir d5/d11/d26/dbc/dc5 0 2024-04-05T21:26:43.430 INFO:tasks.workunit.client.1.smithi188.stdout:5/659: write f2 [20931,26078] 0 2024-04-05T21:26:43.430 INFO:tasks.workunit.client.1.smithi188.stdout:2/760: write d5/db/dd/d36/d3c/f4a [3041018,125905] 0 2024-04-05T21:26:43.430 INFO:tasks.workunit.client.1.smithi188.stdout:7/929: rename d3/d2c/dae/dfd/d10/dbe/ded/f110 to d3/d2c/dae/dfd/d2e/dd8/d12a/f12e 0 2024-04-05T21:26:43.430 INFO:tasks.workunit.client.1.smithi188.stdout:0/634: creat d4/d6/d61/fe9 x:0 0 0 2024-04-05T21:26:43.431 INFO:tasks.workunit.client.0.smithi121.stdout:9/348: readlink d6/df/d1d/l65 0 2024-04-05T21:26:43.434 INFO:tasks.workunit.client.0.smithi121.stdout:3/379: chown d0/d3/d24/d3d/f21 1983 1 2024-04-05T21:26:43.438 INFO:tasks.workunit.client.1.smithi188.stdout:7/930: creat d3/da5/dc8/f12f x:0 0 0 2024-04-05T21:26:43.439 INFO:tasks.workunit.client.1.smithi188.stdout:2/761: chown d5/db/dd/d36/d3c/d48/d78/ldf 6610666 1 2024-04-05T21:26:43.440 INFO:tasks.workunit.client.0.smithi121.stdout:0/554: mknod d7/d62/d3c/cc9 0 2024-04-05T21:26:43.442 INFO:tasks.workunit.client.1.smithi188.stdout:0/635: mkdir d4/d6/d13/dea 0 2024-04-05T21:26:43.442 INFO:tasks.workunit.client.1.smithi188.stdout:3/755: write d5/d7/d6d/d75/d82/fff [1096391,80186] 0 2024-04-05T21:26:43.442 INFO:tasks.workunit.client.1.smithi188.stdout:2/762: mkdir d5/db/d35/df8 0 2024-04-05T21:26:43.443 INFO:tasks.workunit.client.1.smithi188.stdout:7/931: write d3/d2c/dae/dfd/d2e/d3b/f4d [5035952,123649] 0 2024-04-05T21:26:43.444 INFO:tasks.workunit.client.1.smithi188.stdout:1/722: write d1/dc/d16/d46/d88/f12 [3294360,21124] 0 2024-04-05T21:26:43.449 INFO:tasks.workunit.client.1.smithi188.stdout:3/756: truncate d5/d7/d65/d61/d66/da4/ffa 315245 0 2024-04-05T21:26:43.450 INFO:tasks.workunit.client.1.smithi188.stdout:0/636: truncate d4/d8/db/d42/d71/dc5/fdf 1029980 0 2024-04-05T21:26:43.450 INFO:tasks.workunit.client.1.smithi188.stdout:7/932: symlink d3/da5/dc8/d12d/l130 0 2024-04-05T21:26:43.451 INFO:tasks.workunit.client.0.smithi121.stdout:8/445: rename d12/d13/c3b to d12/c91 0 2024-04-05T21:26:43.454 INFO:tasks.workunit.client.1.smithi188.stdout:2/763: creat d5/db/dd/d36/d3c/d48/d78/ff9 x:0 0 0 2024-04-05T21:26:43.455 INFO:tasks.workunit.client.0.smithi121.stdout:1/578: getdents d8/d8c/db6 0 2024-04-05T21:26:43.455 INFO:tasks.workunit.client.0.smithi121.stdout:9/349: dwrite d6/df/f5f [0,4194304] 0 2024-04-05T21:26:43.456 INFO:tasks.workunit.client.1.smithi188.stdout:7/933: chown d3/d2c/dae/dfd/d2e/dd8/d12a 89142682 1 2024-04-05T21:26:43.458 INFO:tasks.workunit.client.0.smithi121.stdout:6/398: rename d1/d2/d3/d5/d1f/d2c to d1/d2/d3/d4/d21/d57/d86 0 2024-04-05T21:26:43.458 INFO:tasks.workunit.client.0.smithi121.stdout:3/380: dwrite d0/d3/f5e [4194304,4194304] 0 2024-04-05T21:26:43.461 INFO:tasks.workunit.client.0.smithi121.stdout:2/433: dwrite d2/d9/ff [0,4194304] 0 2024-04-05T21:26:43.462 INFO:tasks.workunit.client.1.smithi188.stdout:2/764: mknod d5/db/dd/d73/cfa 0 2024-04-05T21:26:43.463 INFO:tasks.workunit.client.0.smithi121.stdout:0/555: symlink d7/d71/d45/lca 0 2024-04-05T21:26:43.464 INFO:tasks.workunit.client.0.smithi121.stdout:4/612: dwrite d5/d11/d2c/f36 [0,4194304] 0 2024-04-05T21:26:43.465 INFO:tasks.workunit.client.0.smithi121.stdout:6/399: truncate d1/d4b/f80 232641 0 2024-04-05T21:26:43.466 INFO:tasks.workunit.client.1.smithi188.stdout:6/746: sync 2024-04-05T21:26:43.469 INFO:tasks.workunit.client.1.smithi188.stdout:3/757: link d5/df/c18 d5/df/d80/c100 0 2024-04-05T21:26:43.470 INFO:tasks.workunit.client.0.smithi121.stdout:8/446: rmdir d12/d50/d6a/d63 39 2024-04-05T21:26:43.474 INFO:tasks.workunit.client.0.smithi121.stdout:1/579: symlink d8/d89/d9e/d2d/lbc 0 2024-04-05T21:26:43.477 INFO:tasks.workunit.client.0.smithi121.stdout:4/613: stat d5/d11/d2c/d9d/dae/d29/c46 0 2024-04-05T21:26:43.478 INFO:tasks.workunit.client.1.smithi188.stdout:6/747: read d6/d13/d18/d19/d2c/d34/f97 [3946106,56381] 0 2024-04-05T21:26:43.478 INFO:tasks.workunit.client.1.smithi188.stdout:3/758: truncate d5/d35/d4e/f96 436157 0 2024-04-05T21:26:43.478 INFO:tasks.workunit.client.1.smithi188.stdout:1/723: write d1/dc/d16/d46/d88/d86/fab [3623551,105349] 0 2024-04-05T21:26:43.478 INFO:tasks.workunit.client.0.smithi121.stdout:0/556: dwrite d7/d71/d29/d2a/d39/f40 [0,4194304] 0 2024-04-05T21:26:43.480 INFO:tasks.workunit.client.0.smithi121.stdout:2/434: rename d2/d9/d38/d6d to d2/d9/d38/d2d/d4c/d7c 0 2024-04-05T21:26:43.481 INFO:tasks.workunit.client.0.smithi121.stdout:6/400: dwrite d1/d2/d3/d5/f3c [8388608,4194304] 0 2024-04-05T21:26:43.482 INFO:tasks.workunit.client.1.smithi188.stdout:3/759: rename d5/d2b to d5/d7e/d101 0 2024-04-05T21:26:43.482 INFO:tasks.workunit.client.1.smithi188.stdout:1/724: dread - d1/dc/d16/d2d/f97 zero size 2024-04-05T21:26:43.483 INFO:tasks.workunit.client.1.smithi188.stdout:2/765: rmdir d5/db/dd 39 2024-04-05T21:26:43.484 INFO:tasks.workunit.client.0.smithi121.stdout:2/435: chown d2/d9 1822 1 2024-04-05T21:26:43.484 INFO:tasks.workunit.client.0.smithi121.stdout:0/557: chown d7/d62/d9d/da6 1054134745 1 2024-04-05T21:26:43.487 INFO:tasks.workunit.client.0.smithi121.stdout:6/401: write d1/d2/d3/d5/fd [1398961,93979] 0 2024-04-05T21:26:43.488 INFO:tasks.workunit.client.0.smithi121.stdout:1/580: write d8/d89/d9e/f30 [1934826,85183] 0 2024-04-05T21:26:43.488 INFO:tasks.workunit.client.0.smithi121.stdout:9/350: rename d6/df/d1d/d2c/d37/d3c/f3d to d6/df/d1d/d2c/d37/d3a/f73 0 2024-04-05T21:26:43.488 INFO:tasks.workunit.client.0.smithi121.stdout:0/558: fdatasync d7/d71/d45/f6a 0 2024-04-05T21:26:43.490 INFO:tasks.workunit.client.1.smithi188.stdout:2/766: stat d5/db/dd/d36/d3c/d48/d7a/f99 0 2024-04-05T21:26:43.491 INFO:tasks.workunit.client.1.smithi188.stdout:3/760: unlink d5/d7/d6d/dd8/d2f/c97 0 2024-04-05T21:26:43.492 INFO:tasks.workunit.client.0.smithi121.stdout:9/351: chown d6/df 1 1 2024-04-05T21:26:43.493 INFO:tasks.workunit.client.0.smithi121.stdout:2/436: mknod d2/d9/d56/d5c/c7d 0 2024-04-05T21:26:43.494 INFO:tasks.workunit.client.0.smithi121.stdout:4/614: dwrite d5/d11/f8a [0,4194304] 0 2024-04-05T21:26:43.494 INFO:tasks.workunit.client.0.smithi121.stdout:3/381: getdents d0/d3/d24/d6d 0 2024-04-05T21:26:43.495 INFO:tasks.workunit.client.0.smithi121.stdout:1/581: symlink d8/d8c/db6/d1a/d1b/d4c/da1/lbd 0 2024-04-05T21:26:43.498 INFO:tasks.workunit.client.1.smithi188.stdout:3/761: fsync d5/d7/d6d/dd8/f3d 0 2024-04-05T21:26:43.499 INFO:tasks.workunit.client.0.smithi121.stdout:4/615: stat d5/d11/d2c/d9d/c50 0 2024-04-05T21:26:43.501 INFO:tasks.workunit.client.1.smithi188.stdout:3/762: mkdir d5/d7/d65/d5f/d102 0 2024-04-05T21:26:43.502 INFO:tasks.workunit.client.1.smithi188.stdout:7/934: dread d3/d4/d1d/d74/d88/fa8 [0,4194304] 0 2024-04-05T21:26:43.502 INFO:tasks.workunit.client.0.smithi121.stdout:3/382: creat d0/d3/d24/d3d/f7e x:0 0 0 2024-04-05T21:26:43.503 INFO:tasks.workunit.client.0.smithi121.stdout:2/437: dread d2/d9/ff [0,4194304] 0 2024-04-05T21:26:43.509 INFO:tasks.workunit.client.1.smithi188.stdout:6/748: dread d6/f74 [4194304,4194304] 0 2024-04-05T21:26:43.509 INFO:tasks.workunit.client.1.smithi188.stdout:6/749: readlink l4 0 2024-04-05T21:26:43.511 INFO:tasks.workunit.client.1.smithi188.stdout:3/763: creat d5/d7/d65/d61/d66/da4/de3/df6/f103 x:0 0 0 2024-04-05T21:26:43.511 INFO:tasks.workunit.client.0.smithi121.stdout:0/559: symlink d7/d71/d45/d46/d53/d8e/lcb 0 2024-04-05T21:26:43.512 INFO:tasks.workunit.client.1.smithi188.stdout:7/935: read d3/d2c/dae/dfd/d2e/f35 [253214,88309] 0 2024-04-05T21:26:43.513 INFO:tasks.workunit.client.0.smithi121.stdout:0/560: fsync d7/d71/d29/d3f/f72 0 2024-04-05T21:26:43.513 INFO:tasks.workunit.client.0.smithi121.stdout:2/438: mkdir d2/d9/d38/d2d/d4c/d7c/d7e 0 2024-04-05T21:26:43.514 INFO:tasks.workunit.client.1.smithi188.stdout:6/750: creat d6/d13/d18/d19/d2c/d3f/d95/fe2 x:0 0 0 2024-04-05T21:26:43.516 INFO:tasks.workunit.client.0.smithi121.stdout:1/582: mknod d8/d8c/db6/d1a/d1b/dbb/cbe 0 2024-04-05T21:26:43.517 INFO:tasks.workunit.client.1.smithi188.stdout:3/764: mkdir d5/d7/d65/d61/d66/d6e/d104 0 2024-04-05T21:26:43.517 INFO:tasks.workunit.client.0.smithi121.stdout:8/447: truncate f3 7270191 0 2024-04-05T21:26:43.518 INFO:tasks.workunit.client.0.smithi121.stdout:1/583: truncate d8/d8c/db6/f23 728472 0 2024-04-05T21:26:43.519 INFO:tasks.workunit.client.1.smithi188.stdout:2/767: dread d5/db/dd/d67/dcd/d46/d5d/f5e [0,4194304] 0 2024-04-05T21:26:43.519 INFO:tasks.workunit.client.0.smithi121.stdout:2/439: rename d2/d9/d38/c26 to d2/d1e/c7f 0 2024-04-05T21:26:43.520 INFO:tasks.workunit.client.1.smithi188.stdout:6/751: unlink d6/d13/d18/d19/d2c/c33 0 2024-04-05T21:26:43.521 INFO:tasks.workunit.client.1.smithi188.stdout:3/765: symlink d5/d48/l105 0 2024-04-05T21:26:43.522 INFO:tasks.workunit.client.0.smithi121.stdout:9/352: write d6/df/d1d/d2c/d37/d3c/f56 [2576568,23149] 0 2024-04-05T21:26:43.522 INFO:tasks.workunit.client.0.smithi121.stdout:2/440: chown d2/d9/d1c/d30/f5e 3738341 1 2024-04-05T21:26:43.523 INFO:tasks.workunit.client.0.smithi121.stdout:1/584: creat d8/d89/d9e/da3/fbf x:0 0 0 2024-04-05T21:26:43.524 INFO:tasks.workunit.client.0.smithi121.stdout:7/523: sync 2024-04-05T21:26:43.526 INFO:tasks.workunit.client.1.smithi188.stdout:1/725: dwrite d1/dc/f23 [0,4194304] 0 2024-04-05T21:26:43.527 INFO:tasks.workunit.client.0.smithi121.stdout:3/383: getdents d0/d3/de 0 2024-04-05T21:26:43.528 INFO:tasks.workunit.client.0.smithi121.stdout:6/402: truncate d1/d2/d3/d5/fd 4132904 0 2024-04-05T21:26:43.529 INFO:tasks.workunit.client.0.smithi121.stdout:1/585: dread - d8/d89/d9e/da3/fb0 zero size 2024-04-05T21:26:43.531 INFO:tasks.workunit.client.0.smithi121.stdout:0/561: rename d7/l4e to d7/d71/d29/d2a/d64/lcc 0 2024-04-05T21:26:43.534 INFO:tasks.workunit.client.1.smithi188.stdout:2/768: dread d5/db/dd/d36/d3c/d63/feb [0,4194304] 0 2024-04-05T21:26:43.534 INFO:tasks.workunit.client.1.smithi188.stdout:6/752: dwrite d6/fbd [0,4194304] 0 2024-04-05T21:26:43.535 INFO:tasks.workunit.client.0.smithi121.stdout:2/441: dwrite d2/d28/f4f [0,4194304] 0 2024-04-05T21:26:43.535 INFO:tasks.workunit.client.1.smithi188.stdout:1/726: write d1/dc/d16/d46/d88/d13/d60/fd0 [488462,90141] 0 2024-04-05T21:26:43.537 INFO:tasks.workunit.client.0.smithi121.stdout:8/448: link d12/d50/d6a/d63/f64 d12/d13/d56/d6d/f92 0 2024-04-05T21:26:43.537 INFO:tasks.workunit.client.0.smithi121.stdout:9/353: symlink d6/df/d1d/d72/l74 0 2024-04-05T21:26:43.538 INFO:tasks.workunit.client.1.smithi188.stdout:6/753: readlink d6/d13/d18/d3c/daf/lb1 0 2024-04-05T21:26:43.539 INFO:tasks.workunit.client.0.smithi121.stdout:8/449: fdatasync d12/d13/d56/d57/f88 0 2024-04-05T21:26:43.540 INFO:tasks.workunit.client.0.smithi121.stdout:1/586: write d8/d89/d9e/d57/faf [267750,115139] 0 2024-04-05T21:26:43.542 INFO:tasks.workunit.client.0.smithi121.stdout:2/442: fsync d2/d9/d1c/d30/f5e 0 2024-04-05T21:26:43.543 INFO:tasks.workunit.client.0.smithi121.stdout:4/616: dread d5/d11/d20/f38 [0,4194304] 0 2024-04-05T21:26:43.545 INFO:tasks.workunit.client.0.smithi121.stdout:6/403: link d1/f38 d1/d2/d3/d4/d21/d57/f87 0 2024-04-05T21:26:43.546 INFO:tasks.workunit.client.1.smithi188.stdout:2/769: creat d5/db/dd/d73/ffb x:0 0 0 2024-04-05T21:26:43.549 INFO:tasks.workunit.client.1.smithi188.stdout:1/727: creat d1/d7/d4e/d5f/fe1 x:0 0 0 2024-04-05T21:26:43.549 INFO:tasks.workunit.client.0.smithi121.stdout:4/617: readlink d5/d11/d20/l79 0 2024-04-05T21:26:43.549 INFO:tasks.workunit.client.0.smithi121.stdout:2/443: mknod d2/d9/d56/c80 0 2024-04-05T21:26:43.550 INFO:tasks.workunit.client.0.smithi121.stdout:6/404: readlink d1/d2/d1b/d2e/l4d 0 2024-04-05T21:26:43.551 INFO:tasks.workunit.client.0.smithi121.stdout:1/587: rmdir d8/d8c/db6/d1a/d1b/d4c/d76/da4 39 2024-04-05T21:26:43.552 INFO:tasks.workunit.client.1.smithi188.stdout:1/728: mkdir d1/d7/d4e/de2 0 2024-04-05T21:26:43.552 INFO:tasks.workunit.client.1.smithi188.stdout:2/770: readlink d5/db/dd/d36/d3c/d48/l77 0 2024-04-05T21:26:43.552 INFO:tasks.workunit.client.1.smithi188.stdout:6/754: creat d6/d13/d18/d19/da3/fe3 x:0 0 0 2024-04-05T21:26:43.553 INFO:tasks.workunit.client.1.smithi188.stdout:2/771: read - d5/db/fe6 zero size 2024-04-05T21:26:43.554 INFO:tasks.workunit.client.1.smithi188.stdout:7/936: dread d3/d2c/dae/dfd/d10/d5f/fe1 [0,4194304] 0 2024-04-05T21:26:43.555 INFO:tasks.workunit.client.0.smithi121.stdout:4/618: dread - d5/d45/d8b/fbd zero size 2024-04-05T21:26:43.555 INFO:tasks.workunit.client.1.smithi188.stdout:6/755: mkdir d6/d13/d18/d19/d2c/db2/de4 0 2024-04-05T21:26:43.556 INFO:tasks.workunit.client.0.smithi121.stdout:7/524: dread d0/d32/f41 [0,4194304] 0 2024-04-05T21:26:43.557 INFO:tasks.workunit.client.1.smithi188.stdout:6/756: write d6/d13/f89 [2914471,121180] 0 2024-04-05T21:26:43.557 INFO:tasks.workunit.client.1.smithi188.stdout:2/772: mknod d5/db/dd/d67/dcd/d46/d8c/cfc 0 2024-04-05T21:26:43.558 INFO:tasks.workunit.client.0.smithi121.stdout:3/384: write d0/d3/f5b [1110193,83290] 0 2024-04-05T21:26:43.558 INFO:tasks.workunit.client.0.smithi121.stdout:6/405: creat d1/d2/d3/d4/f88 x:0 0 0 2024-04-05T21:26:43.559 INFO:tasks.workunit.client.1.smithi188.stdout:7/937: symlink d3/d2c/dae/dfd/d55/l131 0 2024-04-05T21:26:43.560 INFO:tasks.workunit.client.0.smithi121.stdout:2/444: mknod d2/d9/d17/d3a/c81 0 2024-04-05T21:26:43.561 INFO:tasks.workunit.client.1.smithi188.stdout:2/773: write d5/f2f [9007352,37375] 0 2024-04-05T21:26:43.561 INFO:tasks.workunit.client.0.smithi121.stdout:8/450: dwrite d12/d13/f1b [0,4194304] 0 2024-04-05T21:26:43.562 INFO:tasks.workunit.client.1.smithi188.stdout:7/938: mkdir d3/d2c/dae/dfd/d2e/d8c/db7/dc4/d132 0 2024-04-05T21:26:43.563 INFO:tasks.workunit.client.1.smithi188.stdout:7/939: stat d3/d2c 0 2024-04-05T21:26:43.563 INFO:tasks.workunit.client.0.smithi121.stdout:1/588: write d8/d8c/db6/d1a/d1b/d24/f8b [1760099,26309] 0 2024-04-05T21:26:43.566 INFO:tasks.workunit.client.1.smithi188.stdout:2/774: unlink d5/db/dd/d36/d3c/d48/d78/ff9 0 2024-04-05T21:26:43.566 INFO:tasks.workunit.client.0.smithi121.stdout:9/354: dwrite f2 [4194304,4194304] 0 2024-04-05T21:26:43.567 INFO:tasks.workunit.client.1.smithi188.stdout:2/775: chown d5/db/dd/d36/d3c/d63 223126704 1 2024-04-05T21:26:43.567 INFO:tasks.workunit.client.0.smithi121.stdout:7/525: fsync d0/d23/d5b/f2f 0 2024-04-05T21:26:43.567 INFO:tasks.workunit.client.0.smithi121.stdout:4/619: write d5/d11/d28/d73/f76 [62874,16578] 0 2024-04-05T21:26:43.571 INFO:tasks.workunit.client.0.smithi121.stdout:6/406: readlink d1/d2/d1b/d35/d51/l71 0 2024-04-05T21:26:43.572 INFO:tasks.workunit.client.1.smithi188.stdout:1/729: rmdir d1/d7 39 2024-04-05T21:26:43.573 INFO:tasks.workunit.client.0.smithi121.stdout:7/526: readlink d0/d9/d22/d3e/l6d 0 2024-04-05T21:26:43.574 INFO:tasks.workunit.client.0.smithi121.stdout:8/451: fsync d12/d13/d15/d1d/d29/f40 0 2024-04-05T21:26:43.575 INFO:tasks.workunit.client.0.smithi121.stdout:7/527: chown d0/d23/d5a 3574989 1 2024-04-05T21:26:43.576 INFO:tasks.workunit.client.0.smithi121.stdout:8/452: chown d12/d50/d6a/d87 27105245 1 2024-04-05T21:26:43.576 INFO:tasks.workunit.client.1.smithi188.stdout:2/776: symlink d5/db/dd/d36/d3c/d48/dcf/lfd 0 2024-04-05T21:26:43.576 INFO:tasks.workunit.client.1.smithi188.stdout:7/940: dwrite d3/d2c/dae/dfd/d2e/d3b/d94/ffc [0,4194304] 0 2024-04-05T21:26:43.576 INFO:tasks.workunit.client.0.smithi121.stdout:4/620: read d5/d11/d60/f9f [25298,76246] 0 2024-04-05T21:26:43.577 INFO:tasks.workunit.client.0.smithi121.stdout:7/528: stat d0/d23/f7f 0 2024-04-05T21:26:43.579 INFO:tasks.workunit.client.0.smithi121.stdout:4/621: read d5/d11/f12 [1313178,78013] 0 2024-04-05T21:26:43.583 INFO:tasks.workunit.client.1.smithi188.stdout:1/730: mknod d1/d7/ce3 0 2024-04-05T21:26:43.583 INFO:tasks.workunit.client.0.smithi121.stdout:0/562: dread d7/f9 [0,4194304] 0 2024-04-05T21:26:43.583 INFO:tasks.workunit.client.0.smithi121.stdout:0/563: stat d7/d71/d45/d5a/cb5 0 2024-04-05T21:26:43.585 INFO:tasks.workunit.client.1.smithi188.stdout:1/731: readlink d1/dc/d16/d21/d27/d49/l67 0 2024-04-05T21:26:43.587 INFO:tasks.workunit.client.1.smithi188.stdout:7/941: getdents d3/da5 0 2024-04-05T21:26:43.590 INFO:tasks.workunit.client.1.smithi188.stdout:3/766: dread d5/d7/f30 [0,4194304] 0 2024-04-05T21:26:43.594 INFO:tasks.workunit.client.1.smithi188.stdout:2/777: dwrite d5/db/dd/f15 [0,4194304] 0 2024-04-05T21:26:43.594 INFO:tasks.workunit.client.1.smithi188.stdout:3/767: fdatasync d5/d7/d6d/f7a 0 2024-04-05T21:26:43.596 INFO:tasks.workunit.client.0.smithi121.stdout:8/453: dread fc [0,4194304] 0 2024-04-05T21:26:43.597 INFO:tasks.workunit.client.0.smithi121.stdout:8/454: read d12/d50/d6a/f65 [115637,106420] 0 2024-04-05T21:26:43.599 INFO:tasks.workunit.client.1.smithi188.stdout:3/768: rmdir d5/d7 39 2024-04-05T21:26:43.601 INFO:tasks.workunit.client.1.smithi188.stdout:1/732: dwrite d1/dc/d16/d46/d88/d13/fcd [0,4194304] 0 2024-04-05T21:26:43.603 INFO:tasks.workunit.client.1.smithi188.stdout:3/769: fsync d5/d7/d65/d5f/fe2 0 2024-04-05T21:26:43.606 INFO:tasks.workunit.client.1.smithi188.stdout:3/770: rmdir d5/d35/d4e 39 2024-04-05T21:26:43.611 INFO:tasks.workunit.client.1.smithi188.stdout:1/733: creat d1/dc/d16/d46/d88/d86/fe4 x:0 0 0 2024-04-05T21:26:43.612 INFO:tasks.workunit.client.1.smithi188.stdout:2/778: dwrite d5/db/dd/d67/dcd/d46/fc5 [0,4194304] 0 2024-04-05T21:26:43.617 INFO:tasks.workunit.client.1.smithi188.stdout:6/757: dread d6/d13/d18/d19/f27 [0,4194304] 0 2024-04-05T21:26:43.621 INFO:tasks.workunit.client.1.smithi188.stdout:6/758: mknod d6/d13/d18/d19/da3/ce5 0 2024-04-05T21:26:43.622 INFO:tasks.workunit.client.1.smithi188.stdout:1/734: mkdir d1/dc/d16/d46/de5 0 2024-04-05T21:26:43.622 INFO:tasks.workunit.client.1.smithi188.stdout:2/779: read d5/db/dd/d67/dcd/d46/f81 [2906515,75677] 0 2024-04-05T21:26:43.627 INFO:tasks.workunit.client.1.smithi188.stdout:1/735: rmdir d1/dc/d16/d46 39 2024-04-05T21:26:43.627 INFO:tasks.workunit.client.1.smithi188.stdout:3/771: dread d5/d7/d6d/d75/faa [0,4194304] 0 2024-04-05T21:26:43.632 INFO:tasks.workunit.client.1.smithi188.stdout:1/736: mkdir d1/dc/d16/d2d/de6 0 2024-04-05T21:26:43.632 INFO:tasks.workunit.client.1.smithi188.stdout:3/772: truncate d5/d7/f36 5099330 0 2024-04-05T21:26:43.633 INFO:tasks.workunit.client.1.smithi188.stdout:2/780: link d5/db/dd/d67/dcd/d46/d66/d8f/cdb d5/db/dd/d36/dbe/cfe 0 2024-04-05T21:26:43.638 INFO:tasks.workunit.client.1.smithi188.stdout:1/737: dwrite d1/dc/d16/d21/d27/d4a/f63 [0,4194304] 0 2024-04-05T21:26:43.641 INFO:tasks.workunit.client.1.smithi188.stdout:2/781: rename d5/db/dd/d36/d3c/d48/d7a/d7c/db5/cd3 to d5/db/dd/d67/dcd/d46/d5d/dea/cff 0 2024-04-05T21:26:43.646 INFO:tasks.workunit.client.1.smithi188.stdout:2/782: creat d5/db/d35/f100 x:0 0 0 2024-04-05T21:26:43.647 INFO:tasks.workunit.client.1.smithi188.stdout:3/773: dread d5/d7/d65/d61/f9a [0,4194304] 0 2024-04-05T21:26:43.647 INFO:tasks.workunit.client.1.smithi188.stdout:3/774: stat d5/f41 0 2024-04-05T21:26:43.648 INFO:tasks.workunit.client.1.smithi188.stdout:2/783: readlink d5/db/dd/d36/la6 0 2024-04-05T21:26:43.650 INFO:tasks.workunit.client.1.smithi188.stdout:3/775: symlink d5/d48/l106 0 2024-04-05T21:26:43.651 INFO:tasks.workunit.client.1.smithi188.stdout:2/784: creat d5/db/d35/df8/f101 x:0 0 0 2024-04-05T21:26:43.654 INFO:tasks.workunit.client.1.smithi188.stdout:7/942: write d3/d4/d1d/d74/f41 [1526977,109486] 0 2024-04-05T21:26:43.656 INFO:tasks.workunit.client.1.smithi188.stdout:7/943: stat d3/d2c/dae/dfd/d10/dbe/ded 0 2024-04-05T21:26:43.658 INFO:tasks.workunit.client.1.smithi188.stdout:6/759: truncate d6/f2e 3167585 0 2024-04-05T21:26:43.659 INFO:tasks.workunit.client.0.smithi121.stdout:3/385: creat d0/d3/f7f x:0 0 0 2024-04-05T21:26:43.660 INFO:tasks.workunit.client.1.smithi188.stdout:2/785: unlink d5/db/dd/d67/dcd/d46/l7f 0 2024-04-05T21:26:43.661 INFO:tasks.workunit.client.0.smithi121.stdout:5/594: sync 2024-04-05T21:26:43.662 INFO:tasks.workunit.client.0.smithi121.stdout:5/595: write dd/de/d19/d3c/f5f [1064650,81555] 0 2024-04-05T21:26:43.666 INFO:tasks.workunit.client.0.smithi121.stdout:6/407: symlink d1/d2/d3/d4/d21/d57/d42/d48/l89 0 2024-04-05T21:26:43.667 INFO:tasks.workunit.client.0.smithi121.stdout:0/564: read d7/dc/ff [2562633,116672] 0 2024-04-05T21:26:43.667 INFO:tasks.workunit.client.0.smithi121.stdout:8/455: write d12/d50/d6a/f65 [257302,73282] 0 2024-04-05T21:26:43.669 INFO:tasks.workunit.client.0.smithi121.stdout:9/355: write d6/df/d1d/d2c/d37/d3a/f3b [665992,63834] 0 2024-04-05T21:26:43.669 INFO:tasks.workunit.client.0.smithi121.stdout:8/456: read d12/d50/d61/f3a [2188975,3448] 0 2024-04-05T21:26:43.671 INFO:tasks.workunit.client.0.smithi121.stdout:8/457: chown d12/d13/d15/d1d 2 1 2024-04-05T21:26:43.672 INFO:tasks.workunit.client.0.smithi121.stdout:7/529: truncate d0/d32/f90 290101 0 2024-04-05T21:26:43.673 INFO:tasks.workunit.client.1.smithi188.stdout:6/760: fsync d6/d13/d18/d3c/fcf 0 2024-04-05T21:26:43.673 INFO:tasks.workunit.client.1.smithi188.stdout:1/738: write d1/dc/d16/d21/f68 [1600293,104454] 0 2024-04-05T21:26:43.674 INFO:tasks.workunit.client.0.smithi121.stdout:8/458: dread d12/f3d [0,4194304] 0 2024-04-05T21:26:43.675 INFO:tasks.workunit.client.0.smithi121.stdout:7/530: dread - d0/d32/fa0 zero size 2024-04-05T21:26:43.676 INFO:tasks.workunit.client.0.smithi121.stdout:3/386: link d0/d3/d24/d3d/d1a/d23/f70 d0/d3/d24/d3f/f80 0 2024-04-05T21:26:43.676 INFO:tasks.workunit.client.1.smithi188.stdout:1/739: fdatasync d1/dc/d16/d21/d27/d4a/f63 0 2024-04-05T21:26:43.677 INFO:tasks.workunit.client.0.smithi121.stdout:9/356: symlink d6/df/d1d/d72/l75 0 2024-04-05T21:26:43.677 INFO:tasks.workunit.client.1.smithi188.stdout:6/761: dread d6/d7/f15 [0,4194304] 0 2024-04-05T21:26:43.679 INFO:tasks.workunit.client.1.smithi188.stdout:3/776: dread d5/d7/d6d/dd8/d2f/f89 [0,4194304] 0 2024-04-05T21:26:43.680 INFO:tasks.workunit.client.0.smithi121.stdout:1/589: write d8/d8c/db6/d1a/d1b/d4c/d76/f3d [1861371,41980] 0 2024-04-05T21:26:43.680 INFO:tasks.workunit.client.1.smithi188.stdout:1/740: chown d1/d7/dbf/fc4 1 1 2024-04-05T21:26:43.682 INFO:tasks.workunit.client.0.smithi121.stdout:9/357: chown d6/d36/l39 21744 1 2024-04-05T21:26:43.683 INFO:tasks.workunit.client.0.smithi121.stdout:4/622: getdents d5/d11/d2c 0 2024-04-05T21:26:43.684 INFO:tasks.workunit.client.0.smithi121.stdout:6/408: dwrite d1/d2/d3/d5/fd [0,4194304] 0 2024-04-05T21:26:43.686 INFO:tasks.workunit.client.0.smithi121.stdout:5/596: dwrite dd/de/d19/d38/d45/f5c [0,4194304] 0 2024-04-05T21:26:43.688 INFO:tasks.workunit.client.0.smithi121.stdout:4/623: truncate d5/d11/d80/fc2 354665 0 2024-04-05T21:26:43.691 INFO:tasks.workunit.client.1.smithi188.stdout:4/951: sync 2024-04-05T21:26:43.691 INFO:tasks.workunit.client.1.smithi188.stdout:8/647: sync 2024-04-05T21:26:43.692 INFO:tasks.workunit.client.1.smithi188.stdout:4/952: chown d0/d14/d15/d78/db1 7782 1 2024-04-05T21:26:43.693 INFO:tasks.workunit.client.1.smithi188.stdout:4/953: chown d0/d14/d15/d42 4384726 1 2024-04-05T21:26:43.694 INFO:tasks.workunit.client.0.smithi121.stdout:7/531: creat d0/d9/d22/d3e/d76/fa6 x:0 0 0 2024-04-05T21:26:43.694 INFO:tasks.workunit.client.1.smithi188.stdout:4/954: readlink d0/d2/d4f/l6f 0 2024-04-05T21:26:43.697 INFO:tasks.workunit.client.1.smithi188.stdout:2/786: mknod d5/db/dd/d36/d3c/d48/d85/c102 0 2024-04-05T21:26:43.697 INFO:tasks.workunit.client.1.smithi188.stdout:3/777: symlink d5/d7/d6d/d75/l107 0 2024-04-05T21:26:43.697 INFO:tasks.workunit.client.1.smithi188.stdout:1/741: mknod d1/dc/d16/d2d/da8/ce7 0 2024-04-05T21:26:43.698 INFO:tasks.workunit.client.0.smithi121.stdout:6/409: chown d1/f1c 229 1 2024-04-05T21:26:43.698 INFO:tasks.workunit.client.0.smithi121.stdout:8/459: rmdir d12/d50/d6a/d87/d8e 0 2024-04-05T21:26:43.699 INFO:tasks.workunit.client.0.smithi121.stdout:5/597: mknod dd/de/d19/d1c/d7b/d88/cd1 0 2024-04-05T21:26:43.700 INFO:tasks.workunit.client.1.smithi188.stdout:7/944: rename d3/f3e to d3/d2c/dae/dfd/d10/d5f/dc6/d104/d10d/f133 0 2024-04-05T21:26:43.700 INFO:tasks.workunit.client.0.smithi121.stdout:4/624: creat d5/d11/d20/da5/fc6 x:0 0 0 2024-04-05T21:26:43.701 INFO:tasks.workunit.client.1.smithi188.stdout:3/778: creat d5/d7/d65/d61/d66/f108 x:0 0 0 2024-04-05T21:26:43.701 INFO:tasks.workunit.client.0.smithi121.stdout:3/387: link d0/d3/d24/d3d/d1a/f4e d0/d3/d24/d6d/f81 0 2024-04-05T21:26:43.702 INFO:tasks.workunit.client.1.smithi188.stdout:2/787: write d5/db/dd/d36/d3c/d48/d7a/d7c/fe5 [164191,112112] 0 2024-04-05T21:26:43.702 INFO:tasks.workunit.client.0.smithi121.stdout:0/565: dwrite d7/d62/f42 [0,4194304] 0 2024-04-05T21:26:43.703 INFO:tasks.workunit.client.0.smithi121.stdout:1/590: truncate d8/d10/f17 4088900 0 2024-04-05T21:26:43.704 INFO:tasks.workunit.client.0.smithi121.stdout:1/591: fsync d8/d8c/db6/f19 0 2024-04-05T21:26:43.707 INFO:tasks.workunit.client.0.smithi121.stdout:8/460: chown d12/d50/c7a 118727269 1 2024-04-05T21:26:43.708 INFO:tasks.workunit.client.0.smithi121.stdout:6/410: rename d1/d32/c2b to d1/d2/d3/d59/c8a 0 2024-04-05T21:26:43.708 INFO:tasks.workunit.client.1.smithi188.stdout:1/742: rmdir d1/dc/d16/d21/d27/d4a 39 2024-04-05T21:26:43.709 INFO:tasks.workunit.client.0.smithi121.stdout:9/358: creat d6/df/d1d/f76 x:0 0 0 2024-04-05T21:26:43.709 INFO:tasks.workunit.client.1.smithi188.stdout:7/945: dread d3/fbb [0,4194304] 0 2024-04-05T21:26:43.710 INFO:tasks.workunit.client.1.smithi188.stdout:1/743: stat d1/dc/d16/d2d/fd1 0 2024-04-05T21:26:43.710 INFO:tasks.workunit.client.0.smithi121.stdout:4/625: fdatasync d5/f3f 0 2024-04-05T21:26:43.711 INFO:tasks.workunit.client.0.smithi121.stdout:1/592: dread d8/d8c/db6/f2c [0,4194304] 0 2024-04-05T21:26:43.712 INFO:tasks.workunit.client.1.smithi188.stdout:7/946: truncate d3/d4/d1d/f123 1970014 0 2024-04-05T21:26:43.713 INFO:tasks.workunit.client.0.smithi121.stdout:5/598: mkdir dd/de/d19/d38/d45/dd2 0 2024-04-05T21:26:43.714 INFO:tasks.workunit.client.1.smithi188.stdout:3/779: rename d5/d35/d4e to d5/d7/d6d/d75/d82/d109 0 2024-04-05T21:26:43.716 INFO:tasks.workunit.client.0.smithi121.stdout:6/411: mknod d1/d2/d1b/d35/d51/c8b 0 2024-04-05T21:26:43.717 INFO:tasks.workunit.client.0.smithi121.stdout:5/599: fdatasync dd/de/d19/f51 0 2024-04-05T21:26:43.717 INFO:tasks.workunit.client.0.smithi121.stdout:1/593: dread - d8/d8c/db6/d1a/d1b/f65 zero size 2024-04-05T21:26:43.717 INFO:tasks.workunit.client.1.smithi188.stdout:7/947: symlink d3/d4/d1d/d74/l134 0 2024-04-05T21:26:43.718 INFO:tasks.workunit.client.0.smithi121.stdout:7/532: getdents d0/d23/d5b/d52 0 2024-04-05T21:26:43.719 INFO:tasks.workunit.client.0.smithi121.stdout:8/461: dwrite d12/d13/d56/d57/f7b [0,4194304] 0 2024-04-05T21:26:43.721 INFO:tasks.workunit.client.1.smithi188.stdout:1/744: unlink d1/dc/d16/d46/la7 0 2024-04-05T21:26:43.721 INFO:tasks.workunit.client.0.smithi121.stdout:0/566: rename d7/d71/d29/d3f/f50 to d7/d62/d8b/fcd 0 2024-04-05T21:26:43.727 INFO:tasks.workunit.client.1.smithi188.stdout:7/948: link d3/d4/d7c/f111 d3/d2c/dae/dfd/d2e/d8c/db7/dc4/f135 0 2024-04-05T21:26:43.730 INFO:tasks.workunit.client.1.smithi188.stdout:7/949: fdatasync d3/d2c/dae/dfd/d10/d5f/fcc 0 2024-04-05T21:26:43.730 INFO:tasks.workunit.client.0.smithi121.stdout:4/626: truncate d5/d11/d2c/d9d/dae/fa6 1819085 0 2024-04-05T21:26:43.731 INFO:tasks.workunit.client.0.smithi121.stdout:6/412: dwrite d1/d2/d3/d4/d21/f84 [0,4194304] 0 2024-04-05T21:26:43.733 INFO:tasks.workunit.client.0.smithi121.stdout:8/462: mkdir d12/d50/d93 0 2024-04-05T21:26:43.733 INFO:tasks.workunit.client.1.smithi188.stdout:2/788: rename d5/db/dd/d67/c8b to d5/db/dd/c103 0 2024-04-05T21:26:43.733 INFO:tasks.workunit.client.1.smithi188.stdout:1/745: rmdir d1/dc/d16/d21/d27/d4a 39 2024-04-05T21:26:43.734 INFO:tasks.workunit.client.1.smithi188.stdout:6/762: truncate d6/d13/d18/d19/d2c/f51 4885042 0 2024-04-05T21:26:43.735 INFO:tasks.workunit.client.0.smithi121.stdout:8/463: readlink d12/d13/d15/d1d/l86 0 2024-04-05T21:26:43.736 INFO:tasks.workunit.client.0.smithi121.stdout:0/567: dwrite d7/d62/f42 [0,4194304] 0 2024-04-05T21:26:43.737 INFO:tasks.workunit.client.0.smithi121.stdout:5/600: rename dd/de/d19/d1c/d7b/d88/d9c to dd/d41/dce/da6/dd3 0 2024-04-05T21:26:43.737 INFO:tasks.workunit.client.0.smithi121.stdout:8/464: write d12/d13/d15/d1d/f36 [2229163,104576] 0 2024-04-05T21:26:43.738 INFO:tasks.workunit.client.1.smithi188.stdout:1/746: symlink d1/dc/d16/d46/le8 0 2024-04-05T21:26:43.742 INFO:tasks.workunit.client.0.smithi121.stdout:3/388: link d0/f15 d0/d3/de/f82 0 2024-04-05T21:26:43.744 INFO:tasks.workunit.client.0.smithi121.stdout:7/533: getdents d0/d23/da4 0 2024-04-05T21:26:43.746 INFO:tasks.workunit.client.0.smithi121.stdout:0/568: mkdir d7/d62/d9d/da6/dce 0 2024-04-05T21:26:43.747 INFO:tasks.workunit.client.0.smithi121.stdout:5/601: mkdir dd/de/d19/d13/dd4 0 2024-04-05T21:26:43.747 INFO:tasks.workunit.client.1.smithi188.stdout:3/780: dread d5/d7/d6d/d75/d82/fff [0,4194304] 0 2024-04-05T21:26:43.749 INFO:tasks.workunit.client.0.smithi121.stdout:8/465: rename d12/d50/d6a/f60 to d12/d13/d15/d1d/f94 0 2024-04-05T21:26:43.750 INFO:tasks.workunit.client.0.smithi121.stdout:1/594: creat d8/d8c/db6/d1a/d1b/fc0 x:0 0 0 2024-04-05T21:26:43.752 INFO:tasks.workunit.client.0.smithi121.stdout:1/595: chown d8/d8c/db6/d1a/d1b/d38/d4b/f6d 7 1 2024-04-05T21:26:43.752 INFO:tasks.workunit.client.0.smithi121.stdout:6/413: mknod d1/c8c 0 2024-04-05T21:26:43.753 INFO:tasks.workunit.client.1.smithi188.stdout:2/789: dwrite d5/fe7 [0,4194304] 0 2024-04-05T21:26:43.753 INFO:tasks.workunit.client.1.smithi188.stdout:1/747: chown d1/dc/d16/d46/d88/d3f/fcf 227467534 1 2024-04-05T21:26:43.756 INFO:tasks.workunit.client.1.smithi188.stdout:7/950: truncate d3/d2c/dae/dfd/d2e/f6b 661227 0 2024-04-05T21:26:43.757 INFO:tasks.workunit.client.1.smithi188.stdout:7/951: chown d3 495692173 1 2024-04-05T21:26:43.757 INFO:tasks.workunit.client.1.smithi188.stdout:1/748: stat d1/dc/d16/d21/f26 0 2024-04-05T21:26:43.758 INFO:tasks.workunit.client.1.smithi188.stdout:4/955: truncate d0/d2/d2a/d34/d6b/fa9 2910578 0 2024-04-05T21:26:43.761 INFO:tasks.workunit.client.1.smithi188.stdout:6/763: creat d6/d13/d18/d19/d2c/db2/fe6 x:0 0 0 2024-04-05T21:26:43.762 INFO:tasks.workunit.client.0.smithi121.stdout:7/534: rename d0/d23/d5b/d78 to d0/d9/d22/d3e/d9c/da7 0 2024-04-05T21:26:43.763 INFO:tasks.workunit.client.0.smithi121.stdout:1/596: dwrite d8/d8c/db6/d1a/d1b/d24/f2a [0,4194304] 0 2024-04-05T21:26:43.765 INFO:tasks.workunit.client.0.smithi121.stdout:6/414: unlink d1/d2/d3/d5/f3c 0 2024-04-05T21:26:43.767 INFO:tasks.workunit.client.0.smithi121.stdout:0/569: dwrite d7/d71/d29/d2a/fbd [0,4194304] 0 2024-04-05T21:26:43.767 INFO:tasks.workunit.client.1.smithi188.stdout:6/764: readlink d6/d13/d18/d19/dc8/la1 0 2024-04-05T21:26:43.768 INFO:tasks.workunit.client.0.smithi121.stdout:1/597: readlink d8/d8c/db6/l9f 0 2024-04-05T21:26:43.768 INFO:tasks.workunit.client.1.smithi188.stdout:9/739: sync 2024-04-05T21:26:43.769 INFO:tasks.workunit.client.0.smithi121.stdout:5/602: creat dd/de/d19/d1c/d74/d8c/dc5/fd5 x:0 0 0 2024-04-05T21:26:43.770 INFO:tasks.workunit.client.1.smithi188.stdout:8/648: dwrite d5/dc/f1d [0,4194304] 0 2024-04-05T21:26:43.773 INFO:tasks.workunit.client.1.smithi188.stdout:2/790: mknod d5/db/d35/d3b/d6b/c104 0 2024-04-05T21:26:43.774 INFO:tasks.workunit.client.0.smithi121.stdout:5/603: fsync dd/de/d39/d3b/d50/f67 0 2024-04-05T21:26:43.775 INFO:tasks.workunit.client.0.smithi121.stdout:0/570: unlink d7/dc/la2 0 2024-04-05T21:26:43.779 INFO:tasks.workunit.client.0.smithi121.stdout:7/535: dwrite d0/d32/f65 [0,4194304] 0 2024-04-05T21:26:43.782 INFO:tasks.workunit.client.0.smithi121.stdout:3/389: mkdir d0/d3/de/d83 0 2024-04-05T21:26:43.784 INFO:tasks.workunit.client.1.smithi188.stdout:3/781: rename d5/cc3 to d5/d7/d65/d61/d66/da4/dfd/c10a 0 2024-04-05T21:26:43.785 INFO:tasks.workunit.client.1.smithi188.stdout:6/765: chown d6/d13/d18/d19/d2c/d3f/d95/faa 22 1 2024-04-05T21:26:43.785 INFO:tasks.workunit.client.1.smithi188.stdout:3/782: readlink d5/l4f 0 2024-04-05T21:26:43.785 INFO:tasks.workunit.client.1.smithi188.stdout:4/956: symlink d0/d14/d15/d53/d82/d8f/d95/l146 0 2024-04-05T21:26:43.785 INFO:tasks.workunit.client.1.smithi188.stdout:2/791: truncate d5/db/dd/d36/d3c/d48/d7a/d7c/fe5 288311 0 2024-04-05T21:26:43.785 INFO:tasks.workunit.client.1.smithi188.stdout:2/792: chown d5/db/dd/d24/dc8 59480992 1 2024-04-05T21:26:43.786 INFO:tasks.workunit.client.1.smithi188.stdout:7/952: dwrite d3/d2c/dae/dfd/d2e/d3b/d94/f12c [0,4194304] 0 2024-04-05T21:26:43.788 INFO:tasks.workunit.client.0.smithi121.stdout:6/415: dwrite d1/d2/d3/d4/d21/d57/d42/f83 [0,4194304] 0 2024-04-05T21:26:43.791 INFO:tasks.workunit.client.0.smithi121.stdout:0/571: creat d7/d62/d9d/da6/dce/fcf x:0 0 0 2024-04-05T21:26:43.795 INFO:tasks.workunit.client.0.smithi121.stdout:7/536: chown d0/d23/d5b/f46 1770 1 2024-04-05T21:26:43.797 INFO:tasks.workunit.client.0.smithi121.stdout:7/537: dread - d0/d9/f95 zero size 2024-04-05T21:26:43.797 INFO:tasks.workunit.client.1.smithi188.stdout:2/793: mkdir d5/db/d35/d105 0 2024-04-05T21:26:43.797 INFO:tasks.workunit.client.1.smithi188.stdout:4/957: unlink d0/d14/d15/dd4/ddf/cfe 0 2024-04-05T21:26:43.797 INFO:tasks.workunit.client.1.smithi188.stdout:8/649: rmdir d5/d21/d2a 39 2024-04-05T21:26:43.798 INFO:tasks.workunit.client.0.smithi121.stdout:3/390: symlink d0/d3/d24/d3d/d1a/d23/d38/d55/l84 0 2024-04-05T21:26:43.798 INFO:tasks.workunit.client.1.smithi188.stdout:8/650: rmdir d5/d21 39 2024-04-05T21:26:43.799 INFO:tasks.workunit.client.1.smithi188.stdout:4/958: mknod d0/d2/d52/c147 0 2024-04-05T21:26:43.800 INFO:tasks.workunit.client.0.smithi121.stdout:6/416: mknod d1/d4b/c8d 0 2024-04-05T21:26:43.800 INFO:tasks.workunit.client.1.smithi188.stdout:9/740: dwrite fb [0,4194304] 0 2024-04-05T21:26:43.800 INFO:tasks.workunit.client.1.smithi188.stdout:2/794: symlink d5/db/dd/d36/d3c/d48/d85/l106 0 2024-04-05T21:26:43.803 INFO:tasks.workunit.client.0.smithi121.stdout:1/598: dread d8/d89/d9e/d57/f60 [0,4194304] 0 2024-04-05T21:26:43.803 INFO:tasks.workunit.client.0.smithi121.stdout:8/466: dwrite d12/d13/d15/d1d/d29/f31 [0,4194304] 0 2024-04-05T21:26:43.804 INFO:tasks.workunit.client.1.smithi188.stdout:8/651: rmdir d5/d6e/dad 39 2024-04-05T21:26:43.805 INFO:tasks.workunit.client.0.smithi121.stdout:7/538: mknod d0/d9/d22/d28/d3b/ca8 0 2024-04-05T21:26:43.806 INFO:tasks.workunit.client.0.smithi121.stdout:1/599: stat d8/d8c/db6/d1a/c64 0 2024-04-05T21:26:43.808 INFO:tasks.workunit.client.0.smithi121.stdout:6/417: symlink d1/d2/d3/d5/d1f/l8e 0 2024-04-05T21:26:43.808 INFO:tasks.workunit.client.1.smithi188.stdout:9/741: write dd/d11/d1f/d25/d8a/d101/ffc [1026426,122041] 0 2024-04-05T21:26:43.809 INFO:tasks.workunit.client.0.smithi121.stdout:3/391: link d0/d3/de/f36 d0/d3/d24/d6d/d7b/f85 0 2024-04-05T21:26:43.810 INFO:tasks.workunit.client.1.smithi188.stdout:2/795: fdatasync d5/db/d35/d3b/f6f 0 2024-04-05T21:26:43.810 INFO:tasks.workunit.client.1.smithi188.stdout:4/959: write d0/d14/f12c [1167682,73542] 0 2024-04-05T21:26:43.811 INFO:tasks.workunit.client.0.smithi121.stdout:1/600: chown d8/d89/d9e/d57/d77/l7d 16626 1 2024-04-05T21:26:43.812 INFO:tasks.workunit.client.0.smithi121.stdout:6/418: mknod d1/d2/d1b/d1e/c8f 0 2024-04-05T21:26:43.814 INFO:tasks.workunit.client.0.smithi121.stdout:6/419: read d1/d2/d3/d5/d1f/f46 [833878,2867] 0 2024-04-05T21:26:43.816 INFO:tasks.workunit.client.1.smithi188.stdout:2/796: chown d5/db/dd/d36/d3c/d63/d75/l79 5032 1 2024-04-05T21:26:43.816 INFO:tasks.workunit.client.0.smithi121.stdout:4/627: write d5/d11/d28/d3b/f59 [1827416,46730] 0 2024-04-05T21:26:43.816 INFO:tasks.workunit.client.0.smithi121.stdout:8/467: creat d12/d50/d93/f95 x:0 0 0 2024-04-05T21:26:43.817 INFO:tasks.workunit.client.0.smithi121.stdout:1/601: creat d8/d89/d9e/d2d/fc1 x:0 0 0 2024-04-05T21:26:43.817 INFO:tasks.workunit.client.0.smithi121.stdout:4/628: fdatasync d5/f16 0 2024-04-05T21:26:43.818 INFO:tasks.workunit.client.1.smithi188.stdout:8/652: creat d5/d21/d40/d43/dbb/fd5 x:0 0 0 2024-04-05T21:26:43.818 INFO:tasks.workunit.client.0.smithi121.stdout:3/392: fdatasync d0/f15 0 2024-04-05T21:26:43.819 INFO:tasks.workunit.client.0.smithi121.stdout:7/539: creat d0/d9/fa9 x:0 0 0 2024-04-05T21:26:43.821 INFO:tasks.workunit.client.1.smithi188.stdout:1/749: dwrite d1/dc/d16/d46/d88/d3f/f69 [0,4194304] 0 2024-04-05T21:26:43.821 INFO:tasks.workunit.client.0.smithi121.stdout:7/540: truncate d0/d32/f83 450327 0 2024-04-05T21:26:43.821 INFO:tasks.workunit.client.0.smithi121.stdout:6/420: unlink d1/d2/d1b/f28 0 2024-04-05T21:26:43.822 INFO:tasks.workunit.client.0.smithi121.stdout:7/541: dread - d0/d9/f86 zero size 2024-04-05T21:26:43.823 INFO:tasks.workunit.client.1.smithi188.stdout:8/653: mknod d5/dc/d8f/cd6 0 2024-04-05T21:26:43.823 INFO:tasks.workunit.client.0.smithi121.stdout:7/542: chown d0/d9/d22/d3e/la1 4 1 2024-04-05T21:26:43.824 INFO:tasks.workunit.client.1.smithi188.stdout:9/742: dwrite dd/d11/f76 [0,4194304] 0 2024-04-05T21:26:43.824 INFO:tasks.workunit.client.0.smithi121.stdout:7/543: truncate d0/d23/f33 5185185 0 2024-04-05T21:26:43.825 INFO:tasks.workunit.client.0.smithi121.stdout:8/468: rename d12/d13/d56/d6d to d12/d50/d93/d96 0 2024-04-05T21:26:43.825 INFO:tasks.workunit.client.0.smithi121.stdout:1/602: mkdir d8/d8c/db6/d1a/d1b/d24/dc2 0 2024-04-05T21:26:43.827 INFO:tasks.workunit.client.1.smithi188.stdout:4/960: creat d0/d14/d15/d1b/f148 x:0 0 0 2024-04-05T21:26:43.828 INFO:tasks.workunit.client.0.smithi121.stdout:3/393: mkdir d0/d3/d24/d3d/d1a/d1e/d86 0 2024-04-05T21:26:43.828 INFO:tasks.workunit.client.0.smithi121.stdout:6/421: symlink d1/d2/d1b/d35/d74/l90 0 2024-04-05T21:26:43.830 INFO:tasks.workunit.client.1.smithi188.stdout:1/750: fdatasync d1/dc/d16/d46/d88/d13/fcb 0 2024-04-05T21:26:43.831 INFO:tasks.workunit.client.1.smithi188.stdout:2/797: creat d5/db/dd/d36/d3c/d63/d75/d87/f107 x:0 0 0 2024-04-05T21:26:43.831 INFO:tasks.workunit.client.1.smithi188.stdout:2/798: stat d5/db/dd/d24/cac 0 2024-04-05T21:26:43.832 INFO:tasks.workunit.client.0.smithi121.stdout:7/544: mknod d0/caa 0 2024-04-05T21:26:43.834 INFO:tasks.workunit.client.0.smithi121.stdout:8/469: fsync d12/d13/d15/d1d/f4c 0 2024-04-05T21:26:43.838 INFO:tasks.workunit.client.0.smithi121.stdout:2/445: sync 2024-04-05T21:26:43.840 INFO:tasks.workunit.client.0.smithi121.stdout:3/394: getdents d0/d3/d24/d3d 0 2024-04-05T21:26:43.840 INFO:tasks.workunit.client.1.smithi188.stdout:8/654: creat d5/dc/d8f/fd7 x:0 0 0 2024-04-05T21:26:43.841 INFO:tasks.workunit.client.1.smithi188.stdout:9/743: chown dd/d11/d1f/d26/d55/d98/dfe 7790 1 2024-04-05T21:26:43.841 INFO:tasks.workunit.client.0.smithi121.stdout:7/545: mknod d0/d9/d22/d3e/d9c/cab 0 2024-04-05T21:26:43.841 INFO:tasks.workunit.client.0.smithi121.stdout:8/470: dread d12/d13/f1b [0,4194304] 0 2024-04-05T21:26:43.842 INFO:tasks.workunit.client.0.smithi121.stdout:7/546: readlink d0/l9e 0 2024-04-05T21:26:43.843 INFO:tasks.workunit.client.1.smithi188.stdout:6/766: rename d6/d13/d38/fd0 to d6/d13/d18/d3c/ddc/d60/fe7 0 2024-04-05T21:26:43.844 INFO:tasks.workunit.client.0.smithi121.stdout:2/446: unlink d2/f10 0 2024-04-05T21:26:43.844 INFO:tasks.workunit.client.0.smithi121.stdout:2/447: stat d2/d9/d17/d3a/f55 0 2024-04-05T21:26:43.847 INFO:tasks.workunit.client.0.smithi121.stdout:5/604: truncate dd/de/d19/d13/f5d 2739232 0 2024-04-05T21:26:43.850 INFO:tasks.workunit.client.0.smithi121.stdout:2/448: write d2/d9/f54 [1646614,32416] 0 2024-04-05T21:26:43.850 INFO:tasks.workunit.client.1.smithi188.stdout:8/655: creat d5/d21/d40/d43/d44/d5b/d99/da3/fd8 x:0 0 0 2024-04-05T21:26:43.851 INFO:tasks.workunit.client.1.smithi188.stdout:9/744: dread dd/d11/d1f/d26/dc0/f5c [0,4194304] 0 2024-04-05T21:26:43.852 INFO:tasks.workunit.client.0.smithi121.stdout:8/471: creat d12/d50/d61/d7f/f97 x:0 0 0 2024-04-05T21:26:43.852 INFO:tasks.workunit.client.1.smithi188.stdout:8/656: write d5/d21/d40/d43/d44/d5b/d99/f7d [3396942,31502] 0 2024-04-05T21:26:43.853 INFO:tasks.workunit.client.0.smithi121.stdout:8/472: chown d12/d50/f79 90 1 2024-04-05T21:26:43.853 INFO:tasks.workunit.client.0.smithi121.stdout:2/449: write d2/d1e/f24 [2758937,28518] 0 2024-04-05T21:26:43.855 INFO:tasks.workunit.client.0.smithi121.stdout:5/605: mknod dd/de/d19/d1c/d74/cd6 0 2024-04-05T21:26:43.856 INFO:tasks.workunit.client.0.smithi121.stdout:2/450: write d2/f11 [5230194,97662] 0 2024-04-05T21:26:43.856 INFO:tasks.workunit.client.0.smithi121.stdout:8/473: write d12/d13/d15/d1d/d29/f40 [565116,59745] 0 2024-04-05T21:26:43.857 INFO:tasks.workunit.client.0.smithi121.stdout:7/547: dread d0/f3 [0,4194304] 0 2024-04-05T21:26:43.858 INFO:tasks.workunit.client.1.smithi188.stdout:7/953: rename d3/d2c/dae/dfd/d10/fc9 to d3/d2c/dae/dfd/f136 0 2024-04-05T21:26:43.860 INFO:tasks.workunit.client.1.smithi188.stdout:3/783: dread d5/d35/f40 [0,4194304] 0 2024-04-05T21:26:43.861 INFO:tasks.workunit.client.1.smithi188.stdout:8/657: dread d5/f3d [0,4194304] 0 2024-04-05T21:26:43.862 INFO:tasks.workunit.client.0.smithi121.stdout:8/474: rmdir d12/d13 39 2024-04-05T21:26:43.862 INFO:tasks.workunit.client.1.smithi188.stdout:4/961: rename d0/d14/l28 to d0/d14/d15/d42/l149 0 2024-04-05T21:26:43.863 INFO:tasks.workunit.client.1.smithi188.stdout:7/954: truncate d3/d2c/dae/fda 991445 0 2024-04-05T21:26:43.864 INFO:tasks.workunit.client.0.smithi121.stdout:5/606: rename dd/de/d19/d38/d9b/lca to dd/de/d39/d3b/d50/ld7 0 2024-04-05T21:26:43.866 INFO:tasks.workunit.client.1.smithi188.stdout:8/658: fdatasync d5/f3d 0 2024-04-05T21:26:43.867 INFO:tasks.workunit.client.1.smithi188.stdout:8/659: chown d5/d21/d40/f7b 1013242 1 2024-04-05T21:26:43.869 INFO:tasks.workunit.client.1.smithi188.stdout:3/784: creat d5/d35/f10b x:0 0 0 2024-04-05T21:26:43.870 INFO:tasks.workunit.client.0.smithi121.stdout:7/548: rename d0/d9/f24 to d0/d32/d7c/fac 0 2024-04-05T21:26:43.870 INFO:tasks.workunit.client.0.smithi121.stdout:4/629: write d5/d11/d26/f6b [424080,77475] 0 2024-04-05T21:26:43.870 INFO:tasks.workunit.client.0.smithi121.stdout:2/451: dwrite d2/d9/d56/f76 [0,4194304] 0 2024-04-05T21:26:43.871 INFO:tasks.workunit.client.1.smithi188.stdout:1/751: truncate d1/dc/d16/d21/d27/f29 3862835 0 2024-04-05T21:26:43.872 INFO:tasks.workunit.client.1.smithi188.stdout:1/752: stat d1/d7/c1a 0 2024-04-05T21:26:43.873 INFO:tasks.workunit.client.0.smithi121.stdout:5/607: symlink dd/de/d39/ld8 0 2024-04-05T21:26:43.873 INFO:tasks.workunit.client.1.smithi188.stdout:5/660: sync 2024-04-05T21:26:43.873 INFO:tasks.workunit.client.1.smithi188.stdout:8/660: unlink d5/d21/d40/d43/d44/d5b/d99/d69/d70/f7a 0 2024-04-05T21:26:43.874 INFO:tasks.workunit.client.0.smithi121.stdout:8/475: rename d12/d13/d15/d1d/l4d to d12/d13/d56/l98 0 2024-04-05T21:26:43.874 INFO:tasks.workunit.client.1.smithi188.stdout:5/661: chown d6/dad/c54 3 1 2024-04-05T21:26:43.874 INFO:tasks.workunit.client.1.smithi188.stdout:1/753: read d1/dc/f23 [3113723,109778] 0 2024-04-05T21:26:43.875 INFO:tasks.workunit.client.0.smithi121.stdout:4/630: mkdir d5/d89/dc7 0 2024-04-05T21:26:43.875 INFO:tasks.workunit.client.1.smithi188.stdout:0/637: sync 2024-04-05T21:26:43.876 INFO:tasks.workunit.client.0.smithi121.stdout:4/631: fdatasync d5/d11/d2c/f36 0 2024-04-05T21:26:43.876 INFO:tasks.workunit.client.1.smithi188.stdout:3/785: rename d5/d39/d84/fde to d5/d39/f10c 0 2024-04-05T21:26:43.878 INFO:tasks.workunit.client.1.smithi188.stdout:6/767: write d6/d13/d18/d19/d2c/f6e [28413,47753] 0 2024-04-05T21:26:43.879 INFO:tasks.workunit.client.0.smithi121.stdout:5/608: mkdir dd/de/d6d/dd9 0 2024-04-05T21:26:43.879 INFO:tasks.workunit.client.1.smithi188.stdout:1/754: fdatasync d1/dc/d16/d46/d88/d13/fc3 0 2024-04-05T21:26:43.880 INFO:tasks.workunit.client.0.smithi121.stdout:8/476: symlink d12/d13/d15/d1d/d29/l99 0 2024-04-05T21:26:43.880 INFO:tasks.workunit.client.0.smithi121.stdout:1/603: truncate d8/d8c/db6/d1a/d1b/d4c/d76/f7c 7626667 0 2024-04-05T21:26:43.882 INFO:tasks.workunit.client.0.smithi121.stdout:4/632: fdatasync f4 0 2024-04-05T21:26:43.882 INFO:tasks.workunit.client.0.smithi121.stdout:8/477: read d12/d50/d93/d96/f72 [3390194,75971] 0 2024-04-05T21:26:43.892 INFO:tasks.workunit.client.0.smithi121.stdout:3/395: dwrite d0/d3/d24/d3d/d1a/d23/f56 [0,4194304] 0 2024-04-05T21:26:43.892 INFO:tasks.workunit.client.0.smithi121.stdout:7/549: link d0/d9/d22/l71 d0/d9/d22/d3e/d9c/lad 0 2024-04-05T21:26:43.893 INFO:tasks.workunit.client.1.smithi188.stdout:1/755: stat d1/dc/d16/d2d/da8/ce7 0 2024-04-05T21:26:43.893 INFO:tasks.workunit.client.1.smithi188.stdout:3/786: rename d5/d7/d65/d61/d66/da4/cf9 to d5/d7/d6d/dd8/d2f/c10d 0 2024-04-05T21:26:43.894 INFO:tasks.workunit.client.1.smithi188.stdout:9/745: write dd/d11/d1f/d26/d55/d98/da5/fe6 [65200,65648] 0 2024-04-05T21:26:43.894 INFO:tasks.workunit.client.1.smithi188.stdout:0/638: dread d4/d6/d46/f59 [0,4194304] 0 2024-04-05T21:26:43.894 INFO:tasks.workunit.client.1.smithi188.stdout:8/661: mknod d5/d21/d2a/cd9 0 2024-04-05T21:26:43.897 INFO:tasks.workunit.client.0.smithi121.stdout:7/550: chown d0/d57/c5c 11 1 2024-04-05T21:26:43.897 INFO:tasks.workunit.client.0.smithi121.stdout:3/396: rmdir d0/d3/d24/d3d 39 2024-04-05T21:26:43.900 INFO:tasks.workunit.client.0.smithi121.stdout:7/551: stat d0/d9/d22/d3e/d9c/da7/c8d 0 2024-04-05T21:26:43.901 INFO:tasks.workunit.client.1.smithi188.stdout:9/746: chown dd/d11/d1f/d26/dc0/f66 0 1 2024-04-05T21:26:43.902 INFO:tasks.workunit.client.0.smithi121.stdout:3/397: chown d0/d3/d24/d3d/d1a 912 1 2024-04-05T21:26:43.902 INFO:tasks.workunit.client.0.smithi121.stdout:4/633: dwrite d5/d11/f17 [0,4194304] 0 2024-04-05T21:26:43.903 INFO:tasks.workunit.client.0.smithi121.stdout:1/604: link d8/d8c/db6/d1a/d1b/d4c/d76/c80 d8/d8c/db6/d1a/d1b/d85/d9c/cc3 0 2024-04-05T21:26:43.911 INFO:tasks.workunit.client.1.smithi188.stdout:6/768: dwrite d6/f74 [4194304,4194304] 0 2024-04-05T21:26:43.912 INFO:tasks.workunit.client.1.smithi188.stdout:3/787: creat d5/d7/d65/d5f/f10e x:0 0 0 2024-04-05T21:26:43.912 INFO:tasks.workunit.client.1.smithi188.stdout:8/662: chown d5/d21/c23 11 1 2024-04-05T21:26:43.912 INFO:tasks.workunit.client.1.smithi188.stdout:1/756: creat d1/fe9 x:0 0 0 2024-04-05T21:26:43.914 INFO:tasks.workunit.client.0.smithi121.stdout:1/605: dwrite d8/d8c/db6/d1a/d1b/d38/d4b/f6c [0,4194304] 0 2024-04-05T21:26:43.918 INFO:tasks.workunit.client.1.smithi188.stdout:7/955: fsync d3/d2c/dae/dfd/d10/d5f/dc6/d104/d10d/f133 0 2024-04-05T21:26:43.919 INFO:tasks.workunit.client.0.smithi121.stdout:7/552: link d0/d9/d22/l71 d0/d9/d22/d81/lae 0 2024-04-05T21:26:43.921 INFO:tasks.workunit.client.0.smithi121.stdout:1/606: mknod d8/d8c/db6/d1a/d1b/d4c/da1/cc4 0 2024-04-05T21:26:43.923 INFO:tasks.workunit.client.0.smithi121.stdout:7/553: write d0/d9/f86 [875087,98387] 0 2024-04-05T21:26:43.924 INFO:tasks.workunit.client.1.smithi188.stdout:1/757: mkdir d1/dc/d16/d46/d88/dea 0 2024-04-05T21:26:43.925 INFO:tasks.workunit.client.0.smithi121.stdout:1/607: creat d8/d89/d9e/da3/fc5 x:0 0 0 2024-04-05T21:26:43.926 INFO:tasks.workunit.client.1.smithi188.stdout:3/788: readlink d5/d7/d31/l53 0 2024-04-05T21:26:43.926 INFO:tasks.workunit.client.1.smithi188.stdout:8/663: dread - d5/dc/d8f/dd1/fcc zero size 2024-04-05T21:26:43.927 INFO:tasks.workunit.client.0.smithi121.stdout:1/608: readlink d8/d8c/db6/l9f 0 2024-04-05T21:26:43.928 INFO:tasks.workunit.client.0.smithi121.stdout:2/452: dwrite d2/d9/d17/f18 [0,4194304] 0 2024-04-05T21:26:43.928 INFO:tasks.workunit.client.0.smithi121.stdout:7/554: write d0/d23/d5b/d52/d61/f89 [217172,10620] 0 2024-04-05T21:26:43.929 INFO:tasks.workunit.client.1.smithi188.stdout:8/664: stat d5/f20 0 2024-04-05T21:26:43.931 INFO:tasks.workunit.client.1.smithi188.stdout:5/662: dwrite d6/d17/f97 [0,4194304] 0 2024-04-05T21:26:43.931 INFO:tasks.workunit.client.1.smithi188.stdout:1/758: chown d1/dc/d16/d46/d88/d86/fe4 614 1 2024-04-05T21:26:43.934 INFO:tasks.workunit.client.0.smithi121.stdout:1/609: rmdir d8/d89/d9e/d2d/d84/da8 39 2024-04-05T21:26:43.935 INFO:tasks.workunit.client.0.smithi121.stdout:5/609: dread dd/de/d19/d13/f49 [0,4194304] 0 2024-04-05T21:26:43.936 INFO:tasks.workunit.client.1.smithi188.stdout:4/962: dwrite d0/d2/d4f/f5c [0,4194304] 0 2024-04-05T21:26:43.940 INFO:tasks.workunit.client.1.smithi188.stdout:8/665: dread - d5/d21/d2a/d71/fbc zero size 2024-04-05T21:26:43.943 INFO:tasks.workunit.client.1.smithi188.stdout:4/963: chown d0/d2/f26 30654862 1 2024-04-05T21:26:43.943 INFO:tasks.workunit.client.0.smithi121.stdout:7/555: dwrite d0/d9/d22/f79 [0,4194304] 0 2024-04-05T21:26:43.945 INFO:tasks.workunit.client.1.smithi188.stdout:6/769: dwrite d6/d13/d18/d19/f27 [0,4194304] 0 2024-04-05T21:26:43.945 INFO:tasks.workunit.client.0.smithi121.stdout:5/610: getdents dd/d41/dce 0 2024-04-05T21:26:43.948 INFO:tasks.workunit.client.1.smithi188.stdout:1/759: creat d1/dc/d16/d46/feb x:0 0 0 2024-04-05T21:26:43.953 INFO:tasks.workunit.client.1.smithi188.stdout:4/964: rmdir d0 39 2024-04-05T21:26:43.955 INFO:tasks.workunit.client.0.smithi121.stdout:3/398: dwrite d0/d3/d24/d3d/d1a/f22 [0,4194304] 0 2024-04-05T21:26:43.955 INFO:tasks.workunit.client.0.smithi121.stdout:1/610: link d8/d8c/db6/d1a/d1b/d24/d79/f9a d8/d8c/db6/d1a/d1b/d24/fc6 0 2024-04-05T21:26:43.956 INFO:tasks.workunit.client.1.smithi188.stdout:0/639: dread d4/d6/f1f [0,4194304] 0 2024-04-05T21:26:43.956 INFO:tasks.workunit.client.1.smithi188.stdout:9/747: write dd/d11/d1f/d26/dc0/f53 [198344,92899] 0 2024-04-05T21:26:43.958 INFO:tasks.workunit.client.0.smithi121.stdout:4/634: dread d5/d11/d28/f5d [0,4194304] 0 2024-04-05T21:26:43.959 INFO:tasks.workunit.client.0.smithi121.stdout:4/635: chown d5/d45/d8b 545726298 1 2024-04-05T21:26:43.959 INFO:tasks.workunit.client.1.smithi188.stdout:0/640: chown d4/d8/db/d42/d38/f82 2090548 1 2024-04-05T21:26:43.963 INFO:tasks.workunit.client.1.smithi188.stdout:9/748: rmdir dd/d11 39 2024-04-05T21:26:43.963 INFO:tasks.workunit.client.0.smithi121.stdout:3/399: fdatasync d0/d3/d24/d3f/f80 0 2024-04-05T21:26:43.963 INFO:tasks.workunit.client.1.smithi188.stdout:3/789: getdents d5/d7/d6d/d75 0 2024-04-05T21:26:43.963 INFO:tasks.workunit.client.0.smithi121.stdout:4/636: mkdir d5/d45/d67/dab/dc8 0 2024-04-05T21:26:43.965 INFO:tasks.workunit.client.1.smithi188.stdout:4/965: read d0/f62 [256136,130428] 0 2024-04-05T21:26:43.966 INFO:tasks.workunit.client.1.smithi188.stdout:6/770: rename d6/d13/d48/c73 to d6/d13/d18/d94/ce8 0 2024-04-05T21:26:43.966 INFO:tasks.workunit.client.0.smithi121.stdout:3/400: mknod d0/d3/d24/d3d/d1a/c87 0 2024-04-05T21:26:43.967 INFO:tasks.workunit.client.0.smithi121.stdout:5/611: dread dd/de/d19/d1c/d74/f81 [0,4194304] 0 2024-04-05T21:26:43.968 INFO:tasks.workunit.client.1.smithi188.stdout:1/760: getdents d1/dc/d16/d2d/dd5 0 2024-04-05T21:26:43.969 INFO:tasks.workunit.client.0.smithi121.stdout:5/612: truncate dd/de/d19/d1c/d7b/d88/fcc 947571 0 2024-04-05T21:26:43.969 INFO:tasks.workunit.client.1.smithi188.stdout:0/641: symlink d4/d8/db/d42/d71/d80/leb 0 2024-04-05T21:26:43.970 INFO:tasks.workunit.client.0.smithi121.stdout:5/613: rename dd to dd/dda 22 2024-04-05T21:26:43.971 INFO:tasks.workunit.client.1.smithi188.stdout:9/749: rmdir dd/d11/d1f/d25 39 2024-04-05T21:26:43.971 INFO:tasks.workunit.client.0.smithi121.stdout:7/556: dread d0/d9/d22/d3e/f3c [0,4194304] 0 2024-04-05T21:26:43.971 INFO:tasks.workunit.client.0.smithi121.stdout:3/401: dread d0/d3/f54 [0,4194304] 0 2024-04-05T21:26:43.973 INFO:tasks.workunit.client.1.smithi188.stdout:9/750: rmdir dd/d11/d73/d85 39 2024-04-05T21:26:43.974 INFO:tasks.workunit.client.1.smithi188.stdout:0/642: dread - d4/d6/d13/d23/d24/d57/d63/fbd zero size 2024-04-05T21:26:43.974 INFO:tasks.workunit.client.0.smithi121.stdout:5/614: creat dd/fdb x:0 0 0 2024-04-05T21:26:43.975 INFO:tasks.workunit.client.0.smithi121.stdout:3/402: symlink d0/d3/d24/d3d/d1a/d23/l88 0 2024-04-05T21:26:43.976 INFO:tasks.workunit.client.0.smithi121.stdout:7/557: fdatasync d0/f6 0 2024-04-05T21:26:43.977 INFO:tasks.workunit.client.1.smithi188.stdout:3/790: link d5/d7/d65/d61/d66/d6e/ca3 d5/d35/c10f 0 2024-04-05T21:26:43.977 INFO:tasks.workunit.client.0.smithi121.stdout:5/615: mknod dd/de/dc1/cdc 0 2024-04-05T21:26:43.978 INFO:tasks.workunit.client.1.smithi188.stdout:0/643: creat d4/d6/d13/d23/d24/d57/fec x:0 0 0 2024-04-05T21:26:43.980 INFO:tasks.workunit.client.0.smithi121.stdout:3/403: write d0/d3/de/f6a [227432,121670] 0 2024-04-05T21:26:43.980 INFO:tasks.workunit.client.0.smithi121.stdout:7/558: creat d0/d9/d22/faf x:0 0 0 2024-04-05T21:26:43.981 INFO:tasks.workunit.client.1.smithi188.stdout:5/663: write d6/d82/d40/d42/f56 [66895,103361] 0 2024-04-05T21:26:43.981 INFO:tasks.workunit.client.0.smithi121.stdout:5/616: rmdir dd/de/d19/d38/da3 39 2024-04-05T21:26:43.982 INFO:tasks.workunit.client.1.smithi188.stdout:3/791: rename d5/d7e/d101/c3b to d5/d7e/d101/de5/c110 0 2024-04-05T21:26:43.985 INFO:tasks.workunit.client.0.smithi121.stdout:5/617: symlink dd/de/d39/ldd 0 2024-04-05T21:26:43.985 INFO:tasks.workunit.client.1.smithi188.stdout:5/664: chown d6/d17/d39/c5f 713589 1 2024-04-05T21:26:43.986 INFO:tasks.workunit.client.1.smithi188.stdout:3/792: fdatasync d5/d7/d65/d5f/fc6 0 2024-04-05T21:26:43.986 INFO:tasks.workunit.client.1.smithi188.stdout:8/666: dread d5/d21/d40/d43/d44/d59/f5a [0,4194304] 0 2024-04-05T21:26:43.988 INFO:tasks.workunit.client.1.smithi188.stdout:0/644: symlink d4/d6/d13/d23/d24/ddc/led 0 2024-04-05T21:26:43.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:43 smithi121 ceph-mon[39723]: pgmap v20: 65 pgs: 65 active+clean; 2.1 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 63 MiB/s rd, 223 MiB/s wr, 396 op/s 2024-04-05T21:26:43.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:43 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:43.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:43 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:43.991 INFO:tasks.workunit.client.1.smithi188.stdout:3/793: truncate d5/d7/d6d/dd8/f2a 4538586 0 2024-04-05T21:26:43.991 INFO:tasks.workunit.client.1.smithi188.stdout:8/667: mknod d5/d21/d2a/d47/cda 0 2024-04-05T21:26:43.992 INFO:tasks.workunit.client.0.smithi121.stdout:7/559: dwrite d0/d32/f8f [0,4194304] 0 2024-04-05T21:26:43.992 INFO:tasks.workunit.client.0.smithi121.stdout:5/618: symlink dd/de/d19/d38/da3/lde 0 2024-04-05T21:26:43.993 INFO:tasks.workunit.client.1.smithi188.stdout:5/665: truncate d6/d82/d40/f90 545182 0 2024-04-05T21:26:43.993 INFO:tasks.workunit.client.1.smithi188.stdout:0/645: rmdir d4/d8/db/d42/d71 39 2024-04-05T21:26:43.995 INFO:tasks.workunit.client.1.smithi188.stdout:3/794: dread - d5/d7e/d101/fa8 zero size 2024-04-05T21:26:43.996 INFO:tasks.workunit.client.1.smithi188.stdout:5/666: chown d6/d15/d5c/d7a/d44/d98/dbf 26 1 2024-04-05T21:26:43.998 INFO:tasks.workunit.client.0.smithi121.stdout:5/619: stat dd/de/d19/d3c/d8e/fb7 0 2024-04-05T21:26:43.998 INFO:tasks.workunit.client.0.smithi121.stdout:4/637: dread d5/d11/d2c/f65 [0,4194304] 0 2024-04-05T21:26:43.998 INFO:tasks.workunit.client.0.smithi121.stdout:7/560: rmdir d0/d57 39 2024-04-05T21:26:43.999 INFO:tasks.workunit.client.1.smithi188.stdout:8/668: creat d5/d21/d40/d43/d48/d7e/fdb x:0 0 0 2024-04-05T21:26:43.999 INFO:tasks.workunit.client.1.smithi188.stdout:4/966: dread d0/d14/df6/f125 [0,4194304] 0 2024-04-05T21:26:43.999 INFO:tasks.workunit.client.0.smithi121.stdout:5/620: creat dd/de/d19/d3c/fdf x:0 0 0 2024-04-05T21:26:44.000 INFO:tasks.workunit.client.1.smithi188.stdout:9/751: dwrite dd/d11/d1f/f6a [4194304,4194304] 0 2024-04-05T21:26:44.001 INFO:tasks.workunit.client.0.smithi121.stdout:1/611: dread d8/d89/d9e/f30 [0,4194304] 0 2024-04-05T21:26:44.001 INFO:tasks.workunit.client.1.smithi188.stdout:5/667: dread - d6/d15/d5c/d7a/d44/f5d zero size 2024-04-05T21:26:44.001 INFO:tasks.workunit.client.0.smithi121.stdout:4/638: creat d5/d89/fc9 x:0 0 0 2024-04-05T21:26:44.001 INFO:tasks.workunit.client.0.smithi121.stdout:7/561: rmdir d0/d9/d22/d3e/d9c/da7 39 2024-04-05T21:26:44.005 INFO:tasks.workunit.client.1.smithi188.stdout:9/752: dread dd/d3d/d89/fed [0,4194304] 0 2024-04-05T21:26:44.006 INFO:tasks.workunit.client.0.smithi121.stdout:1/612: unlink d8/d8c/db6/d1a/l6e 0 2024-04-05T21:26:44.007 INFO:tasks.workunit.client.1.smithi188.stdout:0/646: dwrite d4/d8/db/d42/da2/dd2/fd3 [0,4194304] 0 2024-04-05T21:26:44.009 INFO:tasks.workunit.client.0.smithi121.stdout:4/639: mkdir d5/d11/d26/dbc/dca 0 2024-04-05T21:26:44.009 INFO:tasks.workunit.client.1.smithi188.stdout:4/967: read d0/d14/d15/d53/f9f [3264928,112405] 0 2024-04-05T21:26:44.010 INFO:tasks.workunit.client.0.smithi121.stdout:3/404: dwrite d0/d3/de/f17 [4194304,4194304] 0 2024-04-05T21:26:44.011 INFO:tasks.workunit.client.1.smithi188.stdout:4/968: chown d0/d14/d15/d78/db1/d6a/c99 3 1 2024-04-05T21:26:44.011 INFO:tasks.workunit.client.1.smithi188.stdout:9/753: write dd/d11/d1f/d25/f50 [634823,62828] 0 2024-04-05T21:26:44.013 INFO:tasks.workunit.client.0.smithi121.stdout:1/613: fsync d8/d8c/db6/d1a/d1b/d24/f39 0 2024-04-05T21:26:44.014 INFO:tasks.workunit.client.1.smithi188.stdout:0/647: chown d4/d8/db/d21/fe2 5 1 2024-04-05T21:26:44.014 INFO:tasks.workunit.client.1.smithi188.stdout:9/754: write dd/d11/fb8 [368455,13978] 0 2024-04-05T21:26:44.015 INFO:tasks.workunit.client.0.smithi121.stdout:3/405: truncate d0/d3/d24/d3d/d1a/f4e 444410 0 2024-04-05T21:26:44.018 INFO:tasks.workunit.client.0.smithi121.stdout:1/614: write d8/d8c/db6/d1a/d1b/d4c/f4d [4318024,123918] 0 2024-04-05T21:26:44.018 INFO:tasks.workunit.client.0.smithi121.stdout:5/621: dread dd/de/f34 [0,4194304] 0 2024-04-05T21:26:44.019 INFO:tasks.workunit.client.1.smithi188.stdout:9/755: symlink dd/d11/d1f/d26/d55/l105 0 2024-04-05T21:26:44.019 INFO:tasks.workunit.client.1.smithi188.stdout:0/648: mkdir d4/d8/db/d21/d3c/dee 0 2024-04-05T21:26:44.021 INFO:tasks.workunit.client.1.smithi188.stdout:9/756: creat dd/d11/d1f/d26/dc0/d6b/f106 x:0 0 0 2024-04-05T21:26:44.022 INFO:tasks.workunit.client.0.smithi121.stdout:5/622: creat dd/de/d19/d38/d56/fe0 x:0 0 0 2024-04-05T21:26:44.023 INFO:tasks.workunit.client.0.smithi121.stdout:1/615: rename d8/d89/d9e/d2d/lbc to d8/d89/lc7 0 2024-04-05T21:26:44.023 INFO:tasks.workunit.client.1.smithi188.stdout:0/649: truncate d4/d8/db/fe0 4018265 0 2024-04-05T21:26:44.023 INFO:tasks.workunit.client.1.smithi188.stdout:9/757: write dd/d34/fde [885690,24637] 0 2024-04-05T21:26:44.023 INFO:tasks.workunit.client.0.smithi121.stdout:1/616: chown d8/d8c/da2 240171649 1 2024-04-05T21:26:44.024 INFO:tasks.workunit.client.1.smithi188.stdout:4/969: dread d0/f62 [0,4194304] 0 2024-04-05T21:26:44.024 INFO:tasks.workunit.client.0.smithi121.stdout:5/623: mkdir dd/d41/dce/da6/dd3/de1 0 2024-04-05T21:26:44.025 INFO:tasks.workunit.client.1.smithi188.stdout:0/650: write d4/d6/d13/f14 [3918035,85324] 0 2024-04-05T21:26:44.025 INFO:tasks.workunit.client.0.smithi121.stdout:5/624: fdatasync dd/de/d19/d13/f16 0 2024-04-05T21:26:44.026 INFO:tasks.workunit.client.0.smithi121.stdout:5/625: write dd/fdb [755818,50055] 0 2024-04-05T21:26:44.027 INFO:tasks.workunit.client.1.smithi188.stdout:0/651: chown d4/d6/d61/l7d 13203619 1 2024-04-05T21:26:44.028 INFO:tasks.workunit.client.0.smithi121.stdout:4/640: dread d5/d11/d2c/d9d/dae/d29/d2e/f7b [0,4194304] 0 2024-04-05T21:26:44.029 INFO:tasks.workunit.client.1.smithi188.stdout:9/758: creat dd/d3d/d81/f107 x:0 0 0 2024-04-05T21:26:44.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:43 smithi188 ceph-mon[49715]: pgmap v20: 65 pgs: 65 active+clean; 2.1 GiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 63 MiB/s rd, 223 MiB/s wr, 396 op/s 2024-04-05T21:26:44.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:43 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:44.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:43 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:44.032 INFO:tasks.workunit.client.0.smithi121.stdout:1/617: read d8/d89/d9e/d57/faf [157681,125803] 0 2024-04-05T21:26:44.032 INFO:tasks.workunit.client.1.smithi188.stdout:0/652: creat d4/d8/d78/d9d/fef x:0 0 0 2024-04-05T21:26:44.032 INFO:tasks.workunit.client.1.smithi188.stdout:9/759: write dd/d11/d22/f56 [4731639,122851] 0 2024-04-05T21:26:44.033 INFO:tasks.workunit.client.1.smithi188.stdout:0/653: dread - d4/d8/db/f88 zero size 2024-04-05T21:26:44.034 INFO:tasks.workunit.client.0.smithi121.stdout:4/641: creat d5/d89/dc7/fcb x:0 0 0 2024-04-05T21:26:44.034 INFO:tasks.workunit.client.0.smithi121.stdout:1/618: mkdir d8/d89/d9e/dc8 0 2024-04-05T21:26:44.034 INFO:tasks.workunit.client.0.smithi121.stdout:4/642: fsync d5/d11/d28/f1b 0 2024-04-05T21:26:44.035 INFO:tasks.workunit.client.0.smithi121.stdout:3/406: dread d0/f7 [0,4194304] 0 2024-04-05T21:26:44.035 INFO:tasks.workunit.client.0.smithi121.stdout:4/643: readlink d5/l71 0 2024-04-05T21:26:44.035 INFO:tasks.workunit.client.1.smithi188.stdout:9/760: readlink dd/d11/d1f/l83 0 2024-04-05T21:26:44.037 INFO:tasks.workunit.client.0.smithi121.stdout:3/407: truncate d0/d3/f4 1184570 0 2024-04-05T21:26:44.038 INFO:tasks.workunit.client.0.smithi121.stdout:3/408: fdatasync d0/d3/d24/d3d/d1a/d23/f56 0 2024-04-05T21:26:44.040 INFO:tasks.workunit.client.0.smithi121.stdout:4/644: link d5/d11/d28/d3b/f77 d5/d11/d2c/d9d/fcc 0 2024-04-05T21:26:44.041 INFO:tasks.workunit.client.1.smithi188.stdout:0/654: creat d4/d8/db/d42/ff0 x:0 0 0 2024-04-05T21:26:44.042 INFO:tasks.workunit.client.0.smithi121.stdout:4/645: mkdir d5/d11/d2c/d9d/dae/d29/d2e/dcd 0 2024-04-05T21:26:44.043 INFO:tasks.workunit.client.0.smithi121.stdout:4/646: mkdir d5/d11/d26/d47/dce 0 2024-04-05T21:26:44.045 INFO:tasks.workunit.client.0.smithi121.stdout:4/647: mknod d5/d11/d2c/da4/ccf 0 2024-04-05T21:26:44.048 INFO:tasks.workunit.client.0.smithi121.stdout:4/648: getdents d5/d11/d2c/d9d/dae 0 2024-04-05T21:26:44.048 INFO:tasks.workunit.client.1.smithi188.stdout:0/655: dread d4/d8/db/d42/d38/d77/f95 [0,4194304] 0 2024-04-05T21:26:44.049 INFO:tasks.workunit.client.0.smithi121.stdout:4/649: creat d5/d11/d20/da5/fd0 x:0 0 0 2024-04-05T21:26:44.051 INFO:tasks.workunit.client.1.smithi188.stdout:9/761: rename dd/d11/d1f/d26/d55/dcd to dd/d3d/d108 0 2024-04-05T21:26:44.052 INFO:tasks.workunit.client.0.smithi121.stdout:4/650: link d5/d11/d20/da5/fc6 d5/d89/fd1 0 2024-04-05T21:26:44.053 INFO:tasks.workunit.client.0.smithi121.stdout:4/651: mknod d5/d11/d28/d3b/cd2 0 2024-04-05T21:26:44.059 INFO:tasks.workunit.client.0.smithi121.stdout:7/562: truncate d0/f15 6308671 0 2024-04-05T21:26:44.061 INFO:tasks.workunit.client.0.smithi121.stdout:4/652: dwrite d5/d11/d2c/d9d/dae/fa6 [0,4194304] 0 2024-04-05T21:26:44.061 INFO:tasks.workunit.client.0.smithi121.stdout:1/619: readlink d8/d89/lc7 0 2024-04-05T21:26:44.061 INFO:tasks.workunit.client.0.smithi121.stdout:9/359: sync 2024-04-05T21:26:44.062 INFO:tasks.workunit.client.1.smithi188.stdout:9/762: dwrite dd/d3d/d81/f107 [0,4194304] 0 2024-04-05T21:26:44.062 INFO:tasks.workunit.client.0.smithi121.stdout:7/563: symlink d0/d23/d5b/lb0 0 2024-04-05T21:26:44.063 INFO:tasks.workunit.client.1.smithi188.stdout:4/970: dread d0/d14/d15/d78/db1/f7c [4194304,4194304] 0 2024-04-05T21:26:44.064 INFO:tasks.workunit.client.0.smithi121.stdout:4/653: fdatasync d5/d11/f17 0 2024-04-05T21:26:44.065 INFO:tasks.workunit.client.1.smithi188.stdout:9/763: write dd/d11/d1f/d26/dc0/d6b/d40/d59/fe9 [533491,101652] 0 2024-04-05T21:26:44.067 INFO:tasks.workunit.client.0.smithi121.stdout:9/360: symlink d6/l77 0 2024-04-05T21:26:44.067 INFO:tasks.workunit.client.0.smithi121.stdout:7/564: chown d0/d32/c7d 588462977 1 2024-04-05T21:26:44.068 INFO:tasks.workunit.client.0.smithi121.stdout:7/565: fsync d0/d9/d22/fa3 0 2024-04-05T21:26:44.069 INFO:tasks.workunit.client.0.smithi121.stdout:9/361: stat d6/df/f27 0 2024-04-05T21:26:44.071 INFO:tasks.workunit.client.0.smithi121.stdout:5/626: dread dd/de/d19/d3c/f5f [0,4194304] 0 2024-04-05T21:26:44.072 INFO:tasks.workunit.client.0.smithi121.stdout:4/654: chown d5/d11/d2c/d9d/c4b 98 1 2024-04-05T21:26:44.075 INFO:tasks.workunit.client.1.smithi188.stdout:9/764: symlink dd/d11/d1f/l109 0 2024-04-05T21:26:44.076 INFO:tasks.workunit.client.0.smithi121.stdout:1/620: mkdir d8/d89/d9e/d2d/d84/dc9 0 2024-04-05T21:26:44.077 INFO:tasks.workunit.client.0.smithi121.stdout:7/566: unlink d0/d32/f90 0 2024-04-05T21:26:44.078 INFO:tasks.workunit.client.0.smithi121.stdout:4/655: dread d5/d11/d2c/f36 [0,4194304] 0 2024-04-05T21:26:44.078 INFO:tasks.workunit.client.1.smithi188.stdout:4/971: getdents d0/d2/d2a 0 2024-04-05T21:26:44.078 INFO:tasks.workunit.client.0.smithi121.stdout:9/362: creat d6/df/d1d/d2c/d37/d3a/d6b/f78 x:0 0 0 2024-04-05T21:26:44.079 INFO:tasks.workunit.client.1.smithi188.stdout:4/972: chown d0/d14/d2f/fb8 661 1 2024-04-05T21:26:44.080 INFO:tasks.workunit.client.0.smithi121.stdout:1/621: creat d8/d8c/db6/d1a/d1b/d85/fca x:0 0 0 2024-04-05T21:26:44.080 INFO:tasks.workunit.client.0.smithi121.stdout:7/567: symlink d0/d23/d5b/lb1 0 2024-04-05T21:26:44.081 INFO:tasks.workunit.client.1.smithi188.stdout:9/765: write dd/d11/d73/d8d/fac [130966,38939] 0 2024-04-05T21:26:44.082 INFO:tasks.workunit.client.0.smithi121.stdout:1/622: stat d8/d89/d9e/d2d/f41 0 2024-04-05T21:26:44.085 INFO:tasks.workunit.client.0.smithi121.stdout:7/568: creat d0/d9/d22/d3e/d9c/fb2 x:0 0 0 2024-04-05T21:26:44.087 INFO:tasks.workunit.client.0.smithi121.stdout:9/363: dread d6/df/d1d/d2c/d37/d3a/f3b [0,4194304] 0 2024-04-05T21:26:44.087 INFO:tasks.workunit.client.1.smithi188.stdout:0/656: dwrite d4/d8/db/d42/da2/fa7 [0,4194304] 0 2024-04-05T21:26:44.088 INFO:tasks.workunit.client.1.smithi188.stdout:9/766: read dd/d11/d1f/f6a [4646886,116681] 0 2024-04-05T21:26:44.090 INFO:tasks.workunit.client.0.smithi121.stdout:9/364: mkdir d6/df/d1d/d2c/d37/d3a/d5d/d79 0 2024-04-05T21:26:44.092 INFO:tasks.workunit.client.0.smithi121.stdout:5/627: dread dd/de/d19/d13/d53/f7f [0,4194304] 0 2024-04-05T21:26:44.095 INFO:tasks.workunit.client.1.smithi188.stdout:0/657: mknod d4/d8/db/d42/cf1 0 2024-04-05T21:26:44.096 INFO:tasks.workunit.client.0.smithi121.stdout:4/656: rename d5/d11/d20/da5 to d5/d11/d2c/d9d/dae/d29/d2e/dd3 0 2024-04-05T21:26:44.097 INFO:tasks.workunit.client.1.smithi188.stdout:0/658: rename d4 to d4/d8/d10/df2 22 2024-04-05T21:26:44.098 INFO:tasks.workunit.client.1.smithi188.stdout:4/973: dwrite d0/d14/d15/d53/ff0 [0,4194304] 0 2024-04-05T21:26:44.099 INFO:tasks.workunit.client.0.smithi121.stdout:4/657: read d5/d11/d26/f6b [337307,13267] 0 2024-04-05T21:26:44.100 INFO:tasks.workunit.client.1.smithi188.stdout:0/659: chown d4/d8/db/d21/d3c 1174590171 1 2024-04-05T21:26:44.101 INFO:tasks.workunit.client.0.smithi121.stdout:9/365: dwrite d6/f17 [0,4194304] 0 2024-04-05T21:26:44.102 INFO:tasks.workunit.client.0.smithi121.stdout:1/623: rename d8/d89/d9e/d57/d77/l7d to d8/d8c/da2/lcb 0 2024-04-05T21:26:44.105 INFO:tasks.workunit.client.0.smithi121.stdout:5/628: creat dd/de/d19/d38/fe2 x:0 0 0 2024-04-05T21:26:44.106 INFO:tasks.workunit.client.0.smithi121.stdout:4/658: rmdir d5/d11/d2c/d9d/dae/d29/d2e 39 2024-04-05T21:26:44.108 INFO:tasks.workunit.client.0.smithi121.stdout:7/569: dread d0/d9/d22/d28/f31 [0,4194304] 0 2024-04-05T21:26:44.112 INFO:tasks.workunit.client.0.smithi121.stdout:5/629: unlink dd/de/d6d/l90 0 2024-04-05T21:26:44.113 INFO:tasks.workunit.client.1.smithi188.stdout:0/660: symlink d4/d6/d13/dea/lf3 0 2024-04-05T21:26:44.113 INFO:tasks.workunit.client.0.smithi121.stdout:4/659: unlink d5/d11/d2c/d9d/dae/d29/f9b 0 2024-04-05T21:26:44.114 INFO:tasks.workunit.client.1.smithi188.stdout:4/974: dread d0/f102 [0,4194304] 0 2024-04-05T21:26:44.116 INFO:tasks.workunit.client.0.smithi121.stdout:1/624: creat d8/d8c/db6/d1a/d1b/d24/dc2/fcc x:0 0 0 2024-04-05T21:26:44.118 INFO:tasks.workunit.client.1.smithi188.stdout:0/661: mkdir d4/d6/df4 0 2024-04-05T21:26:44.118 INFO:tasks.workunit.client.0.smithi121.stdout:7/570: mknod d0/d23/cb3 0 2024-04-05T21:26:44.121 INFO:tasks.workunit.client.0.smithi121.stdout:4/660: rename d5/d89/dc7 to d5/d11/d2c/d9d/dae/d29/db4/dd4 0 2024-04-05T21:26:44.121 INFO:tasks.workunit.client.1.smithi188.stdout:9/767: dwrite fc [8388608,4194304] 0 2024-04-05T21:26:44.122 INFO:tasks.workunit.client.0.smithi121.stdout:9/366: link d6/df/d1d/d2c/d37/d3a/d49/d66/l70 d6/df/d1d/d2c/l7a 0 2024-04-05T21:26:44.122 INFO:tasks.workunit.client.0.smithi121.stdout:4/661: stat d5/d11/d2c/d9d/f31 0 2024-04-05T21:26:44.125 INFO:tasks.workunit.client.0.smithi121.stdout:4/662: unlink d5/l62 0 2024-04-05T21:26:44.126 INFO:tasks.workunit.client.0.smithi121.stdout:9/367: rmdir d6/df/d50 39 2024-04-05T21:26:44.127 INFO:tasks.workunit.client.0.smithi121.stdout:9/368: dread - d6/df/d1d/f76 zero size 2024-04-05T21:26:44.129 INFO:tasks.workunit.client.1.smithi188.stdout:0/662: truncate d4/d8/d10/f1b 1003890 0 2024-04-05T21:26:44.129 INFO:tasks.workunit.client.0.smithi121.stdout:5/630: getdents dd 0 2024-04-05T21:26:44.132 INFO:tasks.workunit.client.0.smithi121.stdout:1/625: rename d8/d8c/db6/l83 to d8/d8c/db6/d1a/d1b/d38/lcd 0 2024-04-05T21:26:44.132 INFO:tasks.workunit.client.1.smithi188.stdout:4/975: dwrite d0/d2/fc8 [0,4194304] 0 2024-04-05T21:26:44.133 INFO:tasks.workunit.client.0.smithi121.stdout:4/663: mkdir d5/d11/d26/dbc/dc5/dd5 0 2024-04-05T21:26:44.134 INFO:tasks.workunit.client.1.smithi188.stdout:0/663: creat d4/d8/db/d42/da2/ff5 x:0 0 0 2024-04-05T21:26:44.135 INFO:tasks.workunit.client.0.smithi121.stdout:7/571: getdents d0/d23/d5b 0 2024-04-05T21:26:44.137 INFO:tasks.workunit.client.1.smithi188.stdout:0/664: readlink d4/d8/db/d21/d3c/l5a 0 2024-04-05T21:26:44.139 INFO:tasks.workunit.client.1.smithi188.stdout:4/976: getdents d0/d14/d15 0 2024-04-05T21:26:44.139 INFO:tasks.workunit.client.0.smithi121.stdout:4/664: write d5/d11/d26/d49/fb9 [1004432,72369] 0 2024-04-05T21:26:44.140 INFO:tasks.workunit.client.0.smithi121.stdout:7/572: write d0/d23/d5b/f80 [480357,8167] 0 2024-04-05T21:26:44.140 INFO:tasks.workunit.client.0.smithi121.stdout:5/631: dwrite dd/de/d19/d1c/d74/d8c/dc5/fd5 [0,4194304] 0 2024-04-05T21:26:44.142 INFO:tasks.workunit.client.1.smithi188.stdout:0/665: symlink d4/d8/db/d21/d4a/d73/d79/lf6 0 2024-04-05T21:26:44.143 INFO:tasks.workunit.client.1.smithi188.stdout:4/977: chown d0/d2/c27 129 1 2024-04-05T21:26:44.143 INFO:tasks.workunit.client.0.smithi121.stdout:7/573: fdatasync d0/d9/f95 0 2024-04-05T21:26:44.145 INFO:tasks.workunit.client.0.smithi121.stdout:7/574: chown d0/d9/d22/d3e/c82 81 1 2024-04-05T21:26:44.146 INFO:tasks.workunit.client.1.smithi188.stdout:0/666: write d4/d8/d10/f40 [2437892,127752] 0 2024-04-05T21:26:44.146 INFO:tasks.workunit.client.0.smithi121.stdout:7/575: chown d0/d32/f83 8271 1 2024-04-05T21:26:44.147 INFO:tasks.workunit.client.0.smithi121.stdout:9/369: rename d6/df/d1d/d2c/d37/d3a to d6/d36/d7b 0 2024-04-05T21:26:44.149 INFO:tasks.workunit.client.0.smithi121.stdout:4/665: mknod d5/d11/d26/d47/dce/cd6 0 2024-04-05T21:26:44.151 INFO:tasks.workunit.client.0.smithi121.stdout:4/666: write d5/d11/d28/d3b/f93 [116027,51650] 0 2024-04-05T21:26:44.151 INFO:tasks.workunit.client.1.smithi188.stdout:9/768: dread dd/d11/f3c [0,4194304] 0 2024-04-05T21:26:44.153 INFO:tasks.workunit.client.0.smithi121.stdout:9/370: creat d6/df/d1d/d2c/d37/d3c/f7c x:0 0 0 2024-04-05T21:26:44.154 INFO:tasks.workunit.client.1.smithi188.stdout:4/978: dread d0/d2/d2a/d34/fa1 [0,4194304] 0 2024-04-05T21:26:44.154 INFO:tasks.workunit.client.1.smithi188.stdout:9/769: write dd/d11/d73/d85/f93 [996100,117885] 0 2024-04-05T21:26:44.155 INFO:tasks.workunit.client.0.smithi121.stdout:9/371: readlink d6/l11 0 2024-04-05T21:26:44.158 INFO:tasks.workunit.client.0.smithi121.stdout:5/632: getdents dd/de/d19/d3c/d8e 0 2024-04-05T21:26:44.159 INFO:tasks.workunit.client.1.smithi188.stdout:9/770: creat dd/d11/d1f/f10a x:0 0 0 2024-04-05T21:26:44.160 INFO:tasks.workunit.client.0.smithi121.stdout:9/372: write d6/d36/d7b/f73 [945575,107434] 0 2024-04-05T21:26:44.161 INFO:tasks.workunit.client.0.smithi121.stdout:4/667: dwrite d5/f86 [0,4194304] 0 2024-04-05T21:26:44.164 INFO:tasks.workunit.client.0.smithi121.stdout:9/373: dread - d6/df/d1d/d2c/f60 zero size 2024-04-05T21:26:44.164 INFO:tasks.workunit.client.1.smithi188.stdout:9/771: unlink dd/d11/d1f/d25/d8a/fb3 0 2024-04-05T21:26:44.166 INFO:tasks.workunit.client.1.smithi188.stdout:0/667: write d4/d8/db/d42/d71/d80/f9b [416860,3364] 0 2024-04-05T21:26:44.166 INFO:tasks.workunit.client.0.smithi121.stdout:9/374: read d6/df/d1d/f2d [721869,71637] 0 2024-04-05T21:26:44.170 INFO:tasks.workunit.client.0.smithi121.stdout:7/576: write d0/d32/d7c/fac [5191843,96025] 0 2024-04-05T21:26:44.170 INFO:tasks.workunit.client.1.smithi188.stdout:9/772: write dd/d11/d1f/d26/f48 [4397586,13123] 0 2024-04-05T21:26:44.170 INFO:tasks.workunit.client.1.smithi188.stdout:0/668: dread - d4/d8/db/d21/fd6 zero size 2024-04-05T21:26:44.171 INFO:tasks.workunit.client.1.smithi188.stdout:9/773: fsync dd/d11/d1f/fdd 0 2024-04-05T21:26:44.171 INFO:tasks.workunit.client.0.smithi121.stdout:5/633: dwrite dd/de/d39/d3b/fa1 [0,4194304] 0 2024-04-05T21:26:44.172 INFO:tasks.workunit.client.1.smithi188.stdout:0/669: creat d4/d6/d13/dea/ff7 x:0 0 0 2024-04-05T21:26:44.173 INFO:tasks.workunit.client.0.smithi121.stdout:1/626: getdents d8/d8c/db6/d1a/d1b 0 2024-04-05T21:26:44.174 INFO:tasks.workunit.client.1.smithi188.stdout:9/774: write dd/d11/d1f/f6e [3210844,38901] 0 2024-04-05T21:26:44.176 INFO:tasks.workunit.client.0.smithi121.stdout:1/627: write d8/f1d [1493969,34593] 0 2024-04-05T21:26:44.177 INFO:tasks.workunit.client.1.smithi188.stdout:0/670: creat d4/d8/db/d42/d38/ff8 x:0 0 0 2024-04-05T21:26:44.178 INFO:tasks.workunit.client.1.smithi188.stdout:4/979: dwrite d0/d14/d15/d78/fbc [0,4194304] 0 2024-04-05T21:26:44.178 INFO:tasks.workunit.client.0.smithi121.stdout:4/668: truncate d5/d11/d2c/d9d/dae/d29/d2e/f7b 445651 0 2024-04-05T21:26:44.179 INFO:tasks.workunit.client.0.smithi121.stdout:1/628: write d8/d89/d9e/d57/f99 [689722,53081] 0 2024-04-05T21:26:44.179 INFO:tasks.workunit.client.1.smithi188.stdout:0/671: dread - d4/d8/db/d42/d38/ff8 zero size 2024-04-05T21:26:44.181 INFO:tasks.workunit.client.1.smithi188.stdout:9/775: getdents dd/d3d/d81 0 2024-04-05T21:26:44.183 INFO:tasks.workunit.client.0.smithi121.stdout:7/577: unlink d0/d9/d22/d3e/d76/fa6 0 2024-04-05T21:26:44.183 INFO:tasks.workunit.client.0.smithi121.stdout:5/634: mkdir dd/de/d19/d38/de3 0 2024-04-05T21:26:44.184 INFO:tasks.workunit.client.1.smithi188.stdout:0/672: fsync d4/d8/db/d21/d3c/f74 0 2024-04-05T21:26:44.184 INFO:tasks.workunit.client.1.smithi188.stdout:9/776: symlink dd/d34/l10b 0 2024-04-05T21:26:44.184 INFO:tasks.workunit.client.0.smithi121.stdout:5/635: chown dd/de/d19/d3c/d8e 63 1 2024-04-05T21:26:44.184 INFO:tasks.workunit.client.0.smithi121.stdout:9/375: truncate d6/df/d1d/f2b 1941206 0 2024-04-05T21:26:44.185 INFO:tasks.workunit.client.1.smithi188.stdout:9/777: write fb [4252701,101382] 0 2024-04-05T21:26:44.185 INFO:tasks.workunit.client.0.smithi121.stdout:4/669: creat d5/d11/d26/d47/fd7 x:0 0 0 2024-04-05T21:26:44.186 INFO:tasks.workunit.client.1.smithi188.stdout:4/980: creat d0/d14/d15/d53/d82/d8f/f14a x:0 0 0 2024-04-05T21:26:44.188 INFO:tasks.workunit.client.1.smithi188.stdout:0/673: creat d4/d6/d13/d23/dd1/ff9 x:0 0 0 2024-04-05T21:26:44.188 INFO:tasks.workunit.client.1.smithi188.stdout:9/778: dread - dd/d11/d1f/d26/dc0/d6b/f106 zero size 2024-04-05T21:26:44.189 INFO:tasks.workunit.client.0.smithi121.stdout:5/636: chown dd/de/d19/d3c/f5f 1843 1 2024-04-05T21:26:44.190 INFO:tasks.workunit.client.0.smithi121.stdout:9/376: mkdir d6/d36/d7b/d5d/d7d 0 2024-04-05T21:26:44.192 INFO:tasks.workunit.client.0.smithi121.stdout:5/637: creat dd/de/dc1/fe4 x:0 0 0 2024-04-05T21:26:44.193 INFO:tasks.workunit.client.1.smithi188.stdout:0/674: link d4/d6/d13/dc1/fc9 d4/d6/d13/d23/d24/ddc/ffa 0 2024-04-05T21:26:44.193 INFO:tasks.workunit.client.0.smithi121.stdout:1/629: creat d8/d8c/db6/d1a/d1b/d4c/d76/fce x:0 0 0 2024-04-05T21:26:44.195 INFO:tasks.workunit.client.0.smithi121.stdout:1/630: chown d8/d89/d9e 896 1 2024-04-05T21:26:44.195 INFO:tasks.workunit.client.1.smithi188.stdout:0/675: creat d4/d8/d78/d9d/ffb x:0 0 0 2024-04-05T21:26:44.196 INFO:tasks.workunit.client.1.smithi188.stdout:9/779: link dd/d11/d1f/d26/dc0/d6b/d40/d59/c8e dd/d11/d73/dbb/c10c 0 2024-04-05T21:26:44.197 INFO:tasks.workunit.client.0.smithi121.stdout:5/638: symlink dd/de/d19/d1c/d36/d46/le5 0 2024-04-05T21:26:44.198 INFO:tasks.workunit.client.0.smithi121.stdout:4/670: dwrite d5/f81 [0,4194304] 0 2024-04-05T21:26:44.203 INFO:tasks.workunit.client.0.smithi121.stdout:4/671: symlink d5/d89/ld8 0 2024-04-05T21:26:44.204 INFO:tasks.workunit.client.1.smithi188.stdout:9/780: dwrite dd/d11/d1f/d26/dc0/d6b/f94 [0,4194304] 0 2024-04-05T21:26:44.207 INFO:tasks.workunit.client.1.smithi188.stdout:9/781: write dd/d11/d1f/f6a [1952287,24438] 0 2024-04-05T21:26:44.208 INFO:tasks.workunit.client.0.smithi121.stdout:5/639: rename dd/de/d39/d3b/d50/c61 to dd/d41/dce/da6/dd3/de1/ce6 0 2024-04-05T21:26:44.209 INFO:tasks.workunit.client.0.smithi121.stdout:9/377: dwrite d6/df/d1d/d2c/d37/d3c/f7c [0,4194304] 0 2024-04-05T21:26:44.212 INFO:tasks.workunit.client.1.smithi188.stdout:9/782: chown dd/d35/f67 0 1 2024-04-05T21:26:44.213 INFO:tasks.workunit.client.0.smithi121.stdout:4/672: symlink d5/d45/ld9 0 2024-04-05T21:26:44.214 INFO:tasks.workunit.client.0.smithi121.stdout:1/631: dwrite d8/d8c/db6/d1a/d1b/d4c/f4d [0,4194304] 0 2024-04-05T21:26:44.214 INFO:tasks.workunit.client.1.smithi188.stdout:2/799: sync 2024-04-05T21:26:44.217 INFO:tasks.workunit.client.1.smithi188.stdout:2/800: write d5/db/dd/d36/d3c/d63/ff4 [757067,98104] 0 2024-04-05T21:26:44.218 INFO:tasks.workunit.client.0.smithi121.stdout:1/632: stat d8/d8c/db6/d1a/d1b/d44/ca6 0 2024-04-05T21:26:44.220 INFO:tasks.workunit.client.1.smithi188.stdout:9/783: truncate dd/d11/d1f/d26/dc0/d6b/f106 186139 0 2024-04-05T21:26:44.221 INFO:tasks.workunit.client.1.smithi188.stdout:2/801: symlink d5/db/dd/d67/dcd/d46/l108 0 2024-04-05T21:26:44.222 INFO:tasks.workunit.client.1.smithi188.stdout:0/676: write d4/d8/d39/f9a [971094,84370] 0 2024-04-05T21:26:44.222 INFO:tasks.workunit.client.1.smithi188.stdout:9/784: symlink dd/d11/d22/d7e/l10d 0 2024-04-05T21:26:44.225 INFO:tasks.workunit.client.0.smithi121.stdout:9/378: link d6/d36/d7b/d6b/f78 d6/df/d48/f7e 0 2024-04-05T21:26:44.225 INFO:tasks.workunit.client.0.smithi121.stdout:5/640: link dd/de/d19/d3c/cb6 dd/de/d19/d13/d53/ce7 0 2024-04-05T21:26:44.227 INFO:tasks.workunit.client.0.smithi121.stdout:7/578: dread d0/d9/d22/d28/d56/f58 [0,4194304] 0 2024-04-05T21:26:44.228 INFO:tasks.workunit.client.0.smithi121.stdout:1/633: write d8/d8c/db6/f23 [987166,14540] 0 2024-04-05T21:26:44.231 INFO:tasks.workunit.client.0.smithi121.stdout:5/641: mkdir dd/de/d39/d3b/de8 0 2024-04-05T21:26:44.233 INFO:tasks.workunit.client.0.smithi121.stdout:5/642: chown dd/de/d19/d1c/d36/laa 987 1 2024-04-05T21:26:44.233 INFO:tasks.workunit.client.0.smithi121.stdout:4/673: rename d5/d11/d2c/d9d/fcc to d5/d11/d2c/d9d/dae/d29/fda 0 2024-04-05T21:26:44.234 INFO:tasks.workunit.client.0.smithi121.stdout:7/579: mknod d0/d57/cb4 0 2024-04-05T21:26:44.236 INFO:tasks.workunit.client.0.smithi121.stdout:5/643: dread - dd/de/d39/d3b/fc0 zero size 2024-04-05T21:26:44.237 INFO:tasks.workunit.client.0.smithi121.stdout:9/379: dwrite d6/df/d1d/d2c/d37/d3c/f56 [0,4194304] 0 2024-04-05T21:26:44.237 INFO:tasks.workunit.client.0.smithi121.stdout:4/674: write d5/d11/d20/f82 [459908,39055] 0 2024-04-05T21:26:44.240 INFO:tasks.workunit.client.0.smithi121.stdout:7/580: unlink d0/d23/d5b/d52/f77 0 2024-04-05T21:26:44.242 INFO:tasks.workunit.client.0.smithi121.stdout:7/581: write d0/d9/f50 [1351819,71952] 0 2024-04-05T21:26:44.248 INFO:tasks.workunit.client.0.smithi121.stdout:7/582: rename d0/d9/d22/c72 to d0/d23/cb5 0 2024-04-05T21:26:44.251 INFO:tasks.workunit.client.0.smithi121.stdout:7/583: getdents d0/d23/d5b 0 2024-04-05T21:26:44.252 INFO:tasks.workunit.client.0.smithi121.stdout:9/380: dread d6/df/f27 [0,4194304] 0 2024-04-05T21:26:44.254 INFO:tasks.workunit.client.0.smithi121.stdout:9/381: creat d6/df/d1d/d72/f7f x:0 0 0 2024-04-05T21:26:44.267 INFO:tasks.workunit.client.0.smithi121.stdout:1/634: dread d8/d8c/db6/d1a/d1b/d24/f49 [0,4194304] 0 2024-04-05T21:26:44.267 INFO:tasks.workunit.client.0.smithi121.stdout:7/584: dread d0/d32/d7c/fac [0,4194304] 0 2024-04-05T21:26:44.272 INFO:tasks.workunit.client.0.smithi121.stdout:7/585: truncate d0/d9/d22/d28/d56/f58 4763933 0 2024-04-05T21:26:44.274 INFO:tasks.workunit.client.0.smithi121.stdout:7/586: creat d0/d32/d4e/fb6 x:0 0 0 2024-04-05T21:26:44.275 INFO:tasks.workunit.client.0.smithi121.stdout:7/587: write d0/d9/f86 [1458376,118212] 0 2024-04-05T21:26:44.278 INFO:tasks.workunit.client.0.smithi121.stdout:1/635: dread d8/d8c/db6/f8f [0,4194304] 0 2024-04-05T21:26:44.279 INFO:tasks.workunit.client.0.smithi121.stdout:7/588: write d0/d9/d22/d28/f48 [1309253,87034] 0 2024-04-05T21:26:44.286 INFO:tasks.workunit.client.0.smithi121.stdout:1/636: dwrite d8/d8c/db6/d1a/d1b/d24/f2a [0,4194304] 0 2024-04-05T21:26:44.287 INFO:tasks.workunit.client.0.smithi121.stdout:7/589: dwrite d0/d32/f41 [0,4194304] 0 2024-04-05T21:26:44.288 INFO:tasks.workunit.client.0.smithi121.stdout:7/590: dread - d0/d32/d7c/f93 zero size 2024-04-05T21:26:44.289 INFO:tasks.workunit.client.0.smithi121.stdout:1/637: rmdir d8/d8c/db6/d1a 39 2024-04-05T21:26:44.290 INFO:tasks.workunit.client.0.smithi121.stdout:7/591: symlink d0/d32/d4a/lb7 0 2024-04-05T21:26:44.292 INFO:tasks.workunit.client.0.smithi121.stdout:7/592: stat d0/d9/d22/d3e/la1 0 2024-04-05T21:26:44.294 INFO:tasks.workunit.client.0.smithi121.stdout:7/593: chown d0 9 1 2024-04-05T21:26:44.294 INFO:tasks.workunit.client.0.smithi121.stdout:1/638: dread - d8/d8c/db6/d1a/d1b/d24/d79/fb7 zero size 2024-04-05T21:26:44.295 INFO:tasks.workunit.client.0.smithi121.stdout:1/639: chown d8/d8c/db6/d1a/d1b/f69 12 1 2024-04-05T21:26:44.296 INFO:tasks.workunit.client.0.smithi121.stdout:1/640: dread - d8/d8c/db6/d1a/d1b/d4c/f7e zero size 2024-04-05T21:26:44.296 INFO:tasks.workunit.client.0.smithi121.stdout:7/594: rename d0/d23/d5b/lb1 to d0/d9/d22/lb8 0 2024-04-05T21:26:44.297 INFO:tasks.workunit.client.0.smithi121.stdout:1/641: stat d8/d8c/db6/d1a/d1b/dbb/cbe 0 2024-04-05T21:26:44.299 INFO:tasks.workunit.client.0.smithi121.stdout:7/595: mkdir d0/d9/d22/d28/d56/db9 0 2024-04-05T21:26:44.299 INFO:tasks.workunit.client.0.smithi121.stdout:1/642: mknod d8/d8c/db6/d1a/d1b/d38/d4b/ccf 0 2024-04-05T21:26:44.303 INFO:tasks.workunit.client.0.smithi121.stdout:1/643: link d8/d89/d9e/d2d/c6f d8/d8c/db6/d1a/d1b/d38/d4b/cd0 0 2024-04-05T21:26:44.304 INFO:tasks.workunit.client.0.smithi121.stdout:1/644: write d8/d8c/db6/f19 [1616069,97886] 0 2024-04-05T21:26:44.305 INFO:tasks.workunit.client.0.smithi121.stdout:1/645: creat d8/d89/d9e/d2d/fd1 x:0 0 0 2024-04-05T21:26:44.309 INFO:tasks.workunit.client.0.smithi121.stdout:1/646: rename d8/d8c/db6/d1a/d1b/d38/d4b/f6c to d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b/fd2 0 2024-04-05T21:26:44.310 INFO:tasks.workunit.client.0.smithi121.stdout:1/647: write d8/d8c/db6/d1a/d1b/d4c/f7e [776344,125776] 0 2024-04-05T21:26:44.312 INFO:tasks.workunit.client.0.smithi121.stdout:1/648: write d8/d8c/db6/d1a/d1b/d24/f2a [601142,71928] 0 2024-04-05T21:26:44.357 INFO:tasks.workunit.client.1.smithi188.stdout:0/677: dread d4/d8/db/fe5 [0,4194304] 0 2024-04-05T21:26:44.357 INFO:tasks.workunit.client.1.smithi188.stdout:9/785: symlink dd/d11/d73/l10e 0 2024-04-05T21:26:44.362 INFO:tasks.workunit.client.1.smithi188.stdout:0/678: mknod d4/d6/d13/da1/cfc 0 2024-04-05T21:26:44.363 INFO:tasks.workunit.client.1.smithi188.stdout:9/786: rmdir dd/d3d/dd5 0 2024-04-05T21:26:44.366 INFO:tasks.workunit.client.0.smithi121.stdout:5/644: dread dd/de/d19/d1c/d36/f3a [0,4194304] 0 2024-04-05T21:26:44.393 INFO:tasks.workunit.client.0.smithi121.stdout:0/572: sync 2024-04-05T21:26:44.401 INFO:tasks.workunit.client.0.smithi121.stdout:0/573: mkdir d7/d71/dd0 0 2024-04-05T21:26:44.405 INFO:tasks.workunit.client.0.smithi121.stdout:7/596: dread d0/d9/d22/f44 [0,4194304] 0 2024-04-05T21:26:44.405 INFO:tasks.workunit.client.0.smithi121.stdout:4/675: truncate d5/d89/fa7 3117145 0 2024-04-05T21:26:44.406 INFO:tasks.workunit.client.0.smithi121.stdout:4/676: readlink d5/d11/d28/lbf 0 2024-04-05T21:26:44.408 INFO:tasks.workunit.client.0.smithi121.stdout:7/597: dread d0/d9/d22/d81/da2/fa5 [0,4194304] 0 2024-04-05T21:26:44.410 INFO:tasks.workunit.client.0.smithi121.stdout:9/382: write d6/df/d1d/f33 [4188809,86382] 0 2024-04-05T21:26:44.411 INFO:tasks.workunit.client.0.smithi121.stdout:7/598: readlink d0/d9/d22/d3e/d9c/lad 0 2024-04-05T21:26:44.412 INFO:tasks.workunit.client.0.smithi121.stdout:9/383: chown d6/df/f2e 179 1 2024-04-05T21:26:44.413 INFO:tasks.workunit.client.0.smithi121.stdout:9/384: fsync d6/df/d1d/d72/f7f 0 2024-04-05T21:26:44.415 INFO:tasks.workunit.client.0.smithi121.stdout:9/385: dread - d6/df/f6d zero size 2024-04-05T21:26:44.416 INFO:tasks.workunit.client.0.smithi121.stdout:7/599: rename d0/d9/d22/d28/c88 to d0/d23/d5a/cba 0 2024-04-05T21:26:44.419 INFO:tasks.workunit.client.0.smithi121.stdout:7/600: mknod d0/d32/d4a/cbb 0 2024-04-05T21:26:44.421 INFO:tasks.workunit.client.0.smithi121.stdout:7/601: mkdir d0/d32/d4e/d73/dbc 0 2024-04-05T21:26:44.424 INFO:tasks.workunit.client.0.smithi121.stdout:7/602: unlink d0/d9/d22/c6a 0 2024-04-05T21:26:44.425 INFO:tasks.workunit.client.0.smithi121.stdout:7/603: fsync d0/d23/d5a/f7b 0 2024-04-05T21:26:44.432 INFO:tasks.workunit.client.0.smithi121.stdout:4/677: truncate d5/f81 2613350 0 2024-04-05T21:26:44.433 INFO:tasks.workunit.client.0.smithi121.stdout:7/604: getdents d0/d9/d22/d28 0 2024-04-05T21:26:44.433 INFO:tasks.workunit.client.0.smithi121.stdout:9/386: write d6/df/f2e [840239,3093] 0 2024-04-05T21:26:44.436 INFO:tasks.workunit.client.0.smithi121.stdout:5/645: dwrite dd/de/d19/f37 [0,4194304] 0 2024-04-05T21:26:44.437 INFO:tasks.workunit.client.0.smithi121.stdout:4/678: creat d5/d11/d28/d3b/fdb x:0 0 0 2024-04-05T21:26:44.439 INFO:tasks.workunit.client.0.smithi121.stdout:9/387: symlink d6/d32/l80 0 2024-04-05T21:26:44.442 INFO:tasks.workunit.client.0.smithi121.stdout:5/646: symlink dd/de/d19/d38/d45/d68/le9 0 2024-04-05T21:26:44.446 INFO:tasks.workunit.client.0.smithi121.stdout:9/388: chown d6/d36/d7b/d49/d66/l70 0 1 2024-04-05T21:26:44.446 INFO:tasks.workunit.client.0.smithi121.stdout:5/647: symlink dd/de/d19/d1c/lea 0 2024-04-05T21:26:44.446 INFO:tasks.workunit.client.0.smithi121.stdout:7/605: dwrite d0/d32/d4a/f53 [0,4194304] 0 2024-04-05T21:26:44.449 INFO:tasks.workunit.client.0.smithi121.stdout:5/648: chown dd/de/d19/d3c 2210509 1 2024-04-05T21:26:44.449 INFO:tasks.workunit.client.0.smithi121.stdout:7/606: readlink d0/d23/d5b/d52/l55 0 2024-04-05T21:26:44.449 INFO:tasks.workunit.client.0.smithi121.stdout:9/389: mkdir d6/d81 0 2024-04-05T21:26:44.456 INFO:tasks.workunit.client.0.smithi121.stdout:9/390: symlink d6/df/d1d/d2c/d37/l82 0 2024-04-05T21:26:44.456 INFO:tasks.workunit.client.0.smithi121.stdout:4/679: dwrite d5/d11/d2c/d9d/dae/d29/d2e/dd3/fd0 [0,4194304] 0 2024-04-05T21:26:44.457 INFO:tasks.workunit.client.0.smithi121.stdout:5/649: write dd/de/d19/d3c/fdf [821585,97053] 0 2024-04-05T21:26:44.459 INFO:tasks.workunit.client.0.smithi121.stdout:4/680: fdatasync d5/d11/d26/f6b 0 2024-04-05T21:26:44.460 INFO:tasks.workunit.client.0.smithi121.stdout:9/391: creat d6/df/d1d/d2c/f83 x:0 0 0 2024-04-05T21:26:44.461 INFO:tasks.workunit.client.1.smithi188.stdout:0/679: unlink d4/d8/d78/d9d/fb9 0 2024-04-05T21:26:44.462 INFO:tasks.workunit.client.0.smithi121.stdout:5/650: creat dd/de/d19/d1c/d74/feb x:0 0 0 2024-04-05T21:26:44.462 INFO:tasks.workunit.client.0.smithi121.stdout:9/392: fdatasync d6/df/d1d/f33 0 2024-04-05T21:26:44.463 INFO:tasks.workunit.client.0.smithi121.stdout:4/681: creat d5/d11/d26/d47/fdc x:0 0 0 2024-04-05T21:26:44.463 INFO:tasks.workunit.client.1.smithi188.stdout:2/802: rename d5/db/dd/d67/dcd/d46 to d5/db/dd/d73/d109 0 2024-04-05T21:26:44.464 INFO:tasks.workunit.client.0.smithi121.stdout:5/651: write dd/de/d39/f8d [312062,80129] 0 2024-04-05T21:26:44.466 INFO:tasks.workunit.client.0.smithi121.stdout:5/652: chown dd/de/d19/d13/d53 0 1 2024-04-05T21:26:44.467 INFO:tasks.workunit.client.0.smithi121.stdout:9/393: unlink d6/l63 0 2024-04-05T21:26:44.467 INFO:tasks.workunit.client.1.smithi188.stdout:2/803: mknod d5/db/dd/d73/d109/d9f/c10a 0 2024-04-05T21:26:44.468 INFO:tasks.workunit.client.0.smithi121.stdout:9/394: write f2 [11671278,108583] 0 2024-04-05T21:26:44.468 INFO:tasks.workunit.client.1.smithi188.stdout:9/787: link dd/d11/d1f/d26/dc0/f66 dd/d11/d73/dbb/f10f 0 2024-04-05T21:26:44.468 INFO:tasks.workunit.client.1.smithi188.stdout:2/804: readlink d5/d5f/d98/led 0 2024-04-05T21:26:44.469 INFO:tasks.workunit.client.1.smithi188.stdout:9/788: write dd/d11/d1f/d25/fce [1220743,39737] 0 2024-04-05T21:26:44.470 INFO:tasks.workunit.client.1.smithi188.stdout:2/805: write d5/db/dd/d36/d3c/d48/f5b [1821011,58080] 0 2024-04-05T21:26:44.471 INFO:tasks.workunit.client.0.smithi121.stdout:5/653: link dd/de/d19/d38/d45/d68/f98 dd/de/d6d/fec 0 2024-04-05T21:26:44.473 INFO:tasks.workunit.client.1.smithi188.stdout:9/789: chown dd/d11/d1f/d26/dc0/f66 22 1 2024-04-05T21:26:44.475 INFO:tasks.workunit.client.1.smithi188.stdout:2/806: fsync d5/db/dd/d73/d109/d66/fb3 0 2024-04-05T21:26:44.475 INFO:tasks.workunit.client.0.smithi121.stdout:5/654: mknod dd/de/ced 0 2024-04-05T21:26:44.476 INFO:tasks.workunit.client.1.smithi188.stdout:9/790: creat dd/d11/d1f/d25/d51/f110 x:0 0 0 2024-04-05T21:26:44.476 INFO:tasks.workunit.client.0.smithi121.stdout:9/395: dwrite d6/df/d1d/f33 [0,4194304] 0 2024-04-05T21:26:44.478 INFO:tasks.workunit.client.0.smithi121.stdout:5/655: truncate dd/de/d19/d3c/fdf 1897162 0 2024-04-05T21:26:44.478 INFO:tasks.workunit.client.0.smithi121.stdout:7/607: dread d0/d23/f2e [0,4194304] 0 2024-04-05T21:26:44.479 INFO:tasks.workunit.client.1.smithi188.stdout:9/791: creat dd/d11/d1f/d26/d55/d98/f111 x:0 0 0 2024-04-05T21:26:44.479 INFO:tasks.workunit.client.0.smithi121.stdout:7/608: stat d0/d57/c5c 0 2024-04-05T21:26:44.480 INFO:tasks.workunit.client.0.smithi121.stdout:5/656: chown dd/de/d19/d1c/d7b/d88/fcc 157 1 2024-04-05T21:26:44.481 INFO:tasks.workunit.client.1.smithi188.stdout:9/792: fdatasync dd/d11/d1f/db2/ff5 0 2024-04-05T21:26:44.482 INFO:tasks.workunit.client.0.smithi121.stdout:9/396: creat d6/df/d50/f84 x:0 0 0 2024-04-05T21:26:44.486 INFO:tasks.workunit.client.1.smithi188.stdout:9/793: fdatasync dd/d11/f3c 0 2024-04-05T21:26:44.486 INFO:tasks.workunit.client.0.smithi121.stdout:7/609: link d0/d23/d5b/f80 d0/d9/d22/d81/da2/fbd 0 2024-04-05T21:26:44.489 INFO:tasks.workunit.client.0.smithi121.stdout:5/657: dwrite dd/de/d19/d1c/d74/d8c/dc5/fd5 [0,4194304] 0 2024-04-05T21:26:44.512 INFO:tasks.workunit.client.1.smithi188.stdout:9/794: dread dd/d11/d1f/f6e [0,4194304] 0 2024-04-05T21:26:44.518 INFO:tasks.workunit.client.1.smithi188.stdout:9/795: dwrite dd/d11/d1f/d26/dc0/fbf [0,4194304] 0 2024-04-05T21:26:44.519 INFO:tasks.workunit.client.1.smithi188.stdout:9/796: write fc [5673751,1985] 0 2024-04-05T21:26:44.522 INFO:tasks.workunit.client.1.smithi188.stdout:9/797: symlink dd/d11/d1f/d26/dc0/d6b/l112 0 2024-04-05T21:26:44.524 INFO:tasks.workunit.client.1.smithi188.stdout:9/798: creat dd/d3d/d95/f113 x:0 0 0 2024-04-05T21:26:44.525 INFO:tasks.workunit.client.1.smithi188.stdout:9/799: fsync dd/d3d/f7a 0 2024-04-05T21:26:44.526 INFO:tasks.workunit.client.1.smithi188.stdout:9/800: dread - dd/d11/d1f/d26/ff3 zero size 2024-04-05T21:26:44.528 INFO:tasks.workunit.client.1.smithi188.stdout:9/801: creat dd/d3d/d89/dd1/f114 x:0 0 0 2024-04-05T21:26:44.530 INFO:tasks.workunit.client.1.smithi188.stdout:7/956: sync 2024-04-05T21:26:44.535 INFO:tasks.workunit.client.1.smithi188.stdout:7/957: symlink d3/d2c/dae/l137 0 2024-04-05T21:26:44.541 INFO:tasks.workunit.client.1.smithi188.stdout:9/802: rmdir dd/d11/d1f 39 2024-04-05T21:26:44.542 INFO:tasks.workunit.client.1.smithi188.stdout:7/958: dwrite d3/f71 [4194304,4194304] 0 2024-04-05T21:26:44.543 INFO:tasks.workunit.client.0.smithi121.stdout:9/397: fsync d6/df/d50/f84 0 2024-04-05T21:26:44.545 INFO:tasks.workunit.client.0.smithi121.stdout:9/398: rmdir d6/df/d1d/d2c/d37 39 2024-04-05T21:26:44.547 INFO:tasks.workunit.client.1.smithi188.stdout:7/959: unlink d3/d2c/d5e/f98 0 2024-04-05T21:26:44.549 INFO:tasks.workunit.client.1.smithi188.stdout:7/960: symlink d3/d113/l138 0 2024-04-05T21:26:44.550 INFO:tasks.workunit.client.0.smithi121.stdout:5/658: dwrite dd/de/d19/d38/d9b/fb0 [0,4194304] 0 2024-04-05T21:26:44.560 INFO:tasks.workunit.client.1.smithi188.stdout:9/803: dread dd/d11/d22/d7e/de3/fef [0,4194304] 0 2024-04-05T21:26:44.560 INFO:tasks.workunit.client.1.smithi188.stdout:9/804: chown dd/d3d/d95/f9e 12741 1 2024-04-05T21:26:44.570 INFO:tasks.workunit.client.0.smithi121.stdout:8/478: sync 2024-04-05T21:26:44.570 INFO:tasks.workunit.client.0.smithi121.stdout:6/422: sync 2024-04-05T21:26:44.571 INFO:tasks.workunit.client.0.smithi121.stdout:6/423: chown d1/d4b/c76 15 1 2024-04-05T21:26:44.572 INFO:tasks.workunit.client.1.smithi188.stdout:9/805: read dd/d11/d1f/d25/d51/faf [1359368,95390] 0 2024-04-05T21:26:44.573 INFO:tasks.workunit.client.1.smithi188.stdout:9/806: chown dd/d11/d73/d8d/ldb 20 1 2024-04-05T21:26:44.573 INFO:tasks.workunit.client.1.smithi188.stdout:9/807: stat dd/d11/d73/l10e 0 2024-04-05T21:26:44.578 INFO:tasks.workunit.client.1.smithi188.stdout:9/808: getdents dd/d11/d1f/d25/d8a/d101 0 2024-04-05T21:26:44.578 INFO:tasks.workunit.client.1.smithi188.stdout:7/961: dwrite d3/f1e [0,4194304] 0 2024-04-05T21:26:44.581 INFO:tasks.workunit.client.1.smithi188.stdout:9/809: creat dd/d11/d1f/d26/d55/d98/da5/f115 x:0 0 0 2024-04-05T21:26:44.582 INFO:tasks.workunit.client.1.smithi188.stdout:7/962: truncate d3/d2c/dae/dfd/f4a 3368491 0 2024-04-05T21:26:44.586 INFO:tasks.workunit.client.1.smithi188.stdout:9/810: dread fc [8388608,4194304] 0 2024-04-05T21:26:44.594 INFO:tasks.workunit.client.1.smithi188.stdout:7/963: dwrite d3/d4/f116 [0,4194304] 0 2024-04-05T21:26:44.595 INFO:tasks.workunit.client.0.smithi121.stdout:9/399: rename d6/d36/d5c to d6/d36/d7b/d5d/d85 0 2024-04-05T21:26:44.596 INFO:tasks.workunit.client.0.smithi121.stdout:5/659: fsync dd/de/d6d/fec 0 2024-04-05T21:26:44.596 INFO:tasks.workunit.client.0.smithi121.stdout:9/400: fsync d6/df/d1d/f33 0 2024-04-05T21:26:44.597 INFO:tasks.workunit.client.0.smithi121.stdout:8/479: readlink d12/l85 0 2024-04-05T21:26:44.598 INFO:tasks.workunit.client.0.smithi121.stdout:8/480: truncate d12/f73 797570 0 2024-04-05T21:26:44.599 INFO:tasks.workunit.client.0.smithi121.stdout:6/424: mkdir d1/d2/d1b/d91 0 2024-04-05T21:26:44.602 INFO:tasks.workunit.client.0.smithi121.stdout:8/481: chown ff 241581349 1 2024-04-05T21:26:44.602 INFO:tasks.workunit.client.0.smithi121.stdout:8/482: stat d12/l5b 0 2024-04-05T21:26:44.605 INFO:tasks.workunit.client.0.smithi121.stdout:5/660: rmdir dd/de/d39/dcd 0 2024-04-05T21:26:44.606 INFO:tasks.workunit.client.0.smithi121.stdout:8/483: mkdir d12/d50/d61/d7f/d9a 0 2024-04-05T21:26:44.607 INFO:tasks.workunit.client.0.smithi121.stdout:5/661: symlink dd/de/d19/d38/d45/d68/lee 0 2024-04-05T21:26:44.608 INFO:tasks.workunit.client.0.smithi121.stdout:8/484: creat d12/d50/d6a/d63/d8b/f9b x:0 0 0 2024-04-05T21:26:44.609 INFO:tasks.workunit.client.0.smithi121.stdout:5/662: mknod dd/d41/dce/da6/dd3/db1/cef 0 2024-04-05T21:26:44.610 INFO:tasks.workunit.client.0.smithi121.stdout:8/485: mkdir d12/d50/d6a/d63/d9c 0 2024-04-05T21:26:44.612 INFO:tasks.workunit.client.0.smithi121.stdout:8/486: mkdir d12/d13/d15/d1d/d29/d9d 0 2024-04-05T21:26:44.616 INFO:tasks.workunit.client.0.smithi121.stdout:8/487: rename d12/d13/d15/d1d/l3f to d12/d50/d93/d96/l9e 0 2024-04-05T21:26:44.616 INFO:tasks.workunit.client.0.smithi121.stdout:5/663: read dd/de/f22 [1288580,94674] 0 2024-04-05T21:26:44.616 INFO:tasks.workunit.client.0.smithi121.stdout:6/425: dread d1/d2/d3/d4/d21/d57/f87 [0,4194304] 0 2024-04-05T21:26:44.628 INFO:tasks.workunit.client.1.smithi188.stdout:7/964: dread d3/d4/d1d/d74/f59 [0,4194304] 0 2024-04-05T21:26:44.630 INFO:tasks.workunit.client.1.smithi188.stdout:7/965: chown d3/d2c/dae/dfd/d2e/d3b/la0 1 1 2024-04-05T21:26:44.638 INFO:tasks.workunit.client.1.smithi188.stdout:7/966: dwrite d3/da5/dc8/ff9 [0,4194304] 0 2024-04-05T21:26:44.640 INFO:tasks.workunit.client.1.smithi188.stdout:7/967: fdatasync d3/d4/d1d/d74/d10e/f12b 0 2024-04-05T21:26:44.643 INFO:tasks.workunit.client.1.smithi188.stdout:7/968: mknod d3/d2c/dae/dfd/d2e/dd8/d12a/c139 0 2024-04-05T21:26:44.645 INFO:tasks.workunit.client.1.smithi188.stdout:7/969: chown d3/d4/l15 69 1 2024-04-05T21:26:44.646 INFO:tasks.workunit.client.1.smithi188.stdout:7/970: chown d3/d2c/dae/dfd/d2e/d37/ldb 208783 1 2024-04-05T21:26:44.650 INFO:tasks.workunit.client.1.smithi188.stdout:7/971: link d3/c28 d3/d2c/dae/dfd/d55/d9d/c13a 0 2024-04-05T21:26:44.652 INFO:tasks.workunit.client.1.smithi188.stdout:7/972: creat d3/d2c/dae/dfd/d10/dbe/f13b x:0 0 0 2024-04-05T21:26:44.654 INFO:tasks.workunit.client.1.smithi188.stdout:7/973: mknod d3/d4/c13c 0 2024-04-05T21:26:44.656 INFO:tasks.workunit.client.1.smithi188.stdout:7/974: creat d3/d2c/dae/dfd/d55/f13d x:0 0 0 2024-04-05T21:26:44.660 INFO:tasks.workunit.client.1.smithi188.stdout:7/975: dread d3/f71 [4194304,4194304] 0 2024-04-05T21:26:44.682 INFO:tasks.workunit.client.1.smithi188.stdout:7/976: dread d3/d2c/dae/dfd/d10/d5f/dc6/d104/d10d/f133 [0,4194304] 0 2024-04-05T21:26:44.683 INFO:tasks.workunit.client.1.smithi188.stdout:7/977: write d3/d2c/dae/f97 [574862,48236] 0 2024-04-05T21:26:44.687 INFO:tasks.workunit.client.1.smithi188.stdout:3/795: sync 2024-04-05T21:26:44.687 INFO:tasks.workunit.client.1.smithi188.stdout:8/669: sync 2024-04-05T21:26:44.687 INFO:tasks.workunit.client.1.smithi188.stdout:1/761: sync 2024-04-05T21:26:44.687 INFO:tasks.workunit.client.1.smithi188.stdout:6/771: sync 2024-04-05T21:26:44.689 INFO:tasks.workunit.client.1.smithi188.stdout:6/772: chown d6/d13/d18/d19/d2c/c4b 124 1 2024-04-05T21:26:44.693 INFO:tasks.workunit.client.1.smithi188.stdout:7/978: rename d3/d2c/f3f to d3/d2c/dae/dfd/d2e/d37/f13e 0 2024-04-05T21:26:44.694 INFO:tasks.workunit.client.1.smithi188.stdout:8/670: mkdir d5/d21/d80/ddc 0 2024-04-05T21:26:44.695 INFO:tasks.workunit.client.1.smithi188.stdout:3/796: mknod d5/d7/d65/d61/d66/c111 0 2024-04-05T21:26:44.704 INFO:tasks.workunit.client.1.smithi188.stdout:7/979: rename d3/d2c/dce/ff8 to d3/d2c/dae/dfd/d10/d3c/f13f 0 2024-04-05T21:26:44.707 INFO:tasks.workunit.client.1.smithi188.stdout:1/762: dwrite d1/dc/f28 [8388608,4194304] 0 2024-04-05T21:26:44.708 INFO:tasks.workunit.client.1.smithi188.stdout:1/763: chown d1/l65 1750284 1 2024-04-05T21:26:44.712 INFO:tasks.workunit.client.0.smithi121.stdout:8/488: rename d12/d13/d15/d1d/c3c to d12/d13/d56/d57/c9f 0 2024-04-05T21:26:44.714 INFO:tasks.workunit.client.0.smithi121.stdout:8/489: chown d12/d13/d56/d57/f88 649 1 2024-04-05T21:26:44.714 INFO:tasks.workunit.client.0.smithi121.stdout:5/664: link dd/de/c12 dd/de/dc1/cf0 0 2024-04-05T21:26:44.719 INFO:tasks.workunit.client.0.smithi121.stdout:8/490: getdents d12/d50/d6a/d87 0 2024-04-05T21:26:44.720 INFO:tasks.workunit.client.0.smithi121.stdout:6/426: dwrite d1/d2/d3/d4/d21/d57/f87 [0,4194304] 0 2024-04-05T21:26:44.721 INFO:tasks.workunit.client.0.smithi121.stdout:5/665: link dd/de/d19/d1c/d36/l62 dd/de/d19/d13/lf1 0 2024-04-05T21:26:44.727 INFO:tasks.workunit.client.0.smithi121.stdout:8/491: mknod d12/ca0 0 2024-04-05T21:26:44.730 INFO:tasks.workunit.client.0.smithi121.stdout:6/427: dwrite d1/d2/d3/d4/d21/f82 [0,4194304] 0 2024-04-05T21:26:44.732 INFO:tasks.workunit.client.0.smithi121.stdout:8/492: unlink d12/f5d 0 2024-04-05T21:26:44.735 INFO:tasks.workunit.client.0.smithi121.stdout:8/493: rename d12/d13/d15/d1d/c33 to d12/d13/d56/d57/ca1 0 2024-04-05T21:26:44.737 INFO:tasks.workunit.client.0.smithi121.stdout:5/666: write dd/f6c [28273,117055] 0 2024-04-05T21:26:44.738 INFO:tasks.workunit.client.0.smithi121.stdout:1/649: sync 2024-04-05T21:26:44.738 INFO:tasks.workunit.client.0.smithi121.stdout:2/453: sync 2024-04-05T21:26:44.739 INFO:tasks.workunit.client.0.smithi121.stdout:3/409: sync 2024-04-05T21:26:44.740 INFO:tasks.workunit.client.0.smithi121.stdout:3/410: stat d0/f45 0 2024-04-05T21:26:44.743 INFO:tasks.workunit.client.0.smithi121.stdout:2/454: rename d2/d9/c50 to d2/d9/d56/c82 0 2024-04-05T21:26:44.745 INFO:tasks.workunit.client.0.smithi121.stdout:2/455: unlink d2/d9/c3e 0 2024-04-05T21:26:44.747 INFO:tasks.workunit.client.0.smithi121.stdout:3/411: rename d0/f45 to d0/d3/d24/f89 0 2024-04-05T21:26:44.748 INFO:tasks.workunit.client.0.smithi121.stdout:2/456: mknod d2/d9/d17/d3a/c83 0 2024-04-05T21:26:44.749 INFO:tasks.workunit.client.0.smithi121.stdout:2/457: chown d2/d9/d56/f76 6 1 2024-04-05T21:26:44.753 INFO:tasks.workunit.client.0.smithi121.stdout:2/458: mkdir d2/d9/d12/d84 0 2024-04-05T21:26:44.754 INFO:tasks.workunit.client.0.smithi121.stdout:3/412: getdents d0/d3/d24/d3d/d1a/d23/d38/d55 0 2024-04-05T21:26:44.756 INFO:tasks.workunit.client.0.smithi121.stdout:2/459: read d2/d9/d17/d3a/f6f [3002323,29337] 0 2024-04-05T21:26:44.756 INFO:tasks.workunit.client.0.smithi121.stdout:5/667: dwrite dd/de/d19/d13/f5d [0,4194304] 0 2024-04-05T21:26:44.757 INFO:tasks.workunit.client.0.smithi121.stdout:3/413: write d0/d3/de/f17 [9395736,79905] 0 2024-04-05T21:26:44.759 INFO:tasks.workunit.client.0.smithi121.stdout:8/494: dread d12/d13/d15/d1d/f36 [0,4194304] 0 2024-04-05T21:26:44.760 INFO:tasks.workunit.client.0.smithi121.stdout:2/460: rmdir d2/d9/d56/d62 39 2024-04-05T21:26:44.761 INFO:tasks.workunit.client.0.smithi121.stdout:2/461: chown d2/d9/d38/f35 1 1 2024-04-05T21:26:44.764 INFO:tasks.workunit.client.0.smithi121.stdout:3/414: creat d0/d3/de/f8a x:0 0 0 2024-04-05T21:26:44.764 INFO:tasks.workunit.client.0.smithi121.stdout:5/668: dread dd/de/d19/d1c/d36/f9d [0,4194304] 0 2024-04-05T21:26:44.767 INFO:tasks.workunit.client.0.smithi121.stdout:5/669: chown dd/de/d39/c60 8290247 1 2024-04-05T21:26:44.769 INFO:tasks.workunit.client.1.smithi188.stdout:3/797: truncate d5/f21 763674 0 2024-04-05T21:26:44.769 INFO:tasks.workunit.client.0.smithi121.stdout:8/495: dwrite d12/d13/d56/d57/f88 [0,4194304] 0 2024-04-05T21:26:44.771 INFO:tasks.workunit.client.1.smithi188.stdout:1/764: mknod d1/d7/d4e/cec 0 2024-04-05T21:26:44.771 INFO:tasks.workunit.client.1.smithi188.stdout:1/765: chown d1/d7 4054602 1 2024-04-05T21:26:44.772 INFO:tasks.workunit.client.1.smithi188.stdout:7/980: creat d3/d4/d1d/d11d/f140 x:0 0 0 2024-04-05T21:26:44.773 INFO:tasks.workunit.client.0.smithi121.stdout:5/670: mknod dd/de/d39/d3b/d50/cf2 0 2024-04-05T21:26:44.773 INFO:tasks.workunit.client.0.smithi121.stdout:3/415: fsync d0/d3/de/f8a 0 2024-04-05T21:26:44.774 INFO:tasks.workunit.client.1.smithi188.stdout:1/766: chown d1/dc/d16/d46/de5 111 1 2024-04-05T21:26:44.775 INFO:tasks.workunit.client.1.smithi188.stdout:7/981: chown d3/d2c/d5e/fe8 536069752 1 2024-04-05T21:26:44.778 INFO:tasks.workunit.client.0.smithi121.stdout:2/462: dwrite d2/d9/f75 [0,4194304] 0 2024-04-05T21:26:44.778 INFO:tasks.workunit.client.1.smithi188.stdout:3/798: dwrite d5/d7/d65/d5f/fe2 [0,4194304] 0 2024-04-05T21:26:44.779 INFO:tasks.workunit.client.0.smithi121.stdout:8/496: unlink d12/f7d 0 2024-04-05T21:26:44.780 INFO:tasks.workunit.client.1.smithi188.stdout:7/982: symlink d3/d2c/d5e/d91/dfb/l141 0 2024-04-05T21:26:44.781 INFO:tasks.workunit.client.0.smithi121.stdout:2/463: chown d2/d9/d38/d2d/d4c/c71 2 1 2024-04-05T21:26:44.782 INFO:tasks.workunit.client.1.smithi188.stdout:3/799: unlink d5/d7/d6d/dd8/d2f/fe9 0 2024-04-05T21:26:44.783 INFO:tasks.workunit.client.0.smithi121.stdout:2/464: mkdir d2/d9/d1c/d85 0 2024-04-05T21:26:44.784 INFO:tasks.workunit.client.0.smithi121.stdout:2/465: chown d2/d9/f21 362158 1 2024-04-05T21:26:44.784 INFO:tasks.workunit.client.1.smithi188.stdout:3/800: chown d5/df/d80/c100 12 1 2024-04-05T21:26:44.787 INFO:tasks.workunit.client.1.smithi188.stdout:3/801: creat d5/d7/d65/d61/db5/f112 x:0 0 0 2024-04-05T21:26:44.788 INFO:tasks.workunit.client.0.smithi121.stdout:2/466: read d2/d9/d38/d2d/d47/f70 [24348,15070] 0 2024-04-05T21:26:44.788 INFO:tasks.workunit.client.1.smithi188.stdout:3/802: chown d5/d7/d65/d61/d66/c111 120156 1 2024-04-05T21:26:44.789 INFO:tasks.workunit.client.1.smithi188.stdout:3/803: readlink d5/d7/d31/l53 0 2024-04-05T21:26:44.797 INFO:tasks.workunit.client.1.smithi188.stdout:6/773: write d6/d71/fd8 [4289144,77709] 0 2024-04-05T21:26:44.802 INFO:tasks.workunit.client.1.smithi188.stdout:6/774: dwrite d6/fbd [0,4194304] 0 2024-04-05T21:26:44.806 INFO:tasks.workunit.client.1.smithi188.stdout:6/775: symlink d6/d13/d18/d19/db7/le9 0 2024-04-05T21:26:44.808 INFO:tasks.workunit.client.0.smithi121.stdout:2/467: dread d2/d9/ff [0,4194304] 0 2024-04-05T21:26:44.811 INFO:tasks.workunit.client.0.smithi121.stdout:2/468: symlink d2/d9/d12/d84/l86 0 2024-04-05T21:26:44.813 INFO:tasks.workunit.client.0.smithi121.stdout:2/469: symlink d2/d9/d1c/d85/l87 0 2024-04-05T21:26:44.818 INFO:tasks.workunit.client.1.smithi188.stdout:4/981: sync 2024-04-05T21:26:44.818 INFO:tasks.workunit.client.1.smithi188.stdout:0/680: sync 2024-04-05T21:26:44.818 INFO:tasks.workunit.client.1.smithi188.stdout:2/807: sync 2024-04-05T21:26:44.818 INFO:tasks.workunit.client.1.smithi188.stdout:5/668: sync 2024-04-05T21:26:44.820 INFO:tasks.workunit.client.1.smithi188.stdout:5/669: symlink d6/d15/d1b/ddc/le8 0 2024-04-05T21:26:44.821 INFO:tasks.workunit.client.1.smithi188.stdout:7/983: dread d3/d2c/dae/dfd/f6e [0,4194304] 0 2024-04-05T21:26:44.822 INFO:tasks.workunit.client.1.smithi188.stdout:2/808: mkdir d5/db/dd/d36/d3c/d48/dd7/d10b 0 2024-04-05T21:26:44.822 INFO:tasks.workunit.client.1.smithi188.stdout:5/670: read d6/d15/d5c/d7a/d44/fda [18151,98997] 0 2024-04-05T21:26:44.823 INFO:tasks.workunit.client.1.smithi188.stdout:2/809: chown d5/db/f76 201042 1 2024-04-05T21:26:44.825 INFO:tasks.workunit.client.1.smithi188.stdout:2/810: stat d5/db/dd/d36/d3c/d48/l77 0 2024-04-05T21:26:44.827 INFO:tasks.workunit.client.1.smithi188.stdout:2/811: chown d5/db/dd/d36/d3c/d48 44 1 2024-04-05T21:26:44.828 INFO:tasks.workunit.client.1.smithi188.stdout:5/671: unlink d6/d15/c3e 0 2024-04-05T21:26:44.830 INFO:tasks.workunit.client.1.smithi188.stdout:5/672: readlink d6/d15/l7c 0 2024-04-05T21:26:44.831 INFO:tasks.workunit.client.0.smithi121.stdout:5/671: write dd/de/d19/d1c/f48 [122903,129289] 0 2024-04-05T21:26:44.831 INFO:tasks.workunit.client.0.smithi121.stdout:3/416: write d0/d3/d24/d6d/f81 [798737,106734] 0 2024-04-05T21:26:44.831 INFO:tasks.workunit.client.0.smithi121.stdout:8/497: truncate d12/d13/d15/d1d/f58 405286 0 2024-04-05T21:26:44.832 INFO:tasks.workunit.client.1.smithi188.stdout:0/681: link d4/d6/d13/d23/cc6 d4/d6/d13/d23/dd1/cfd 0 2024-04-05T21:26:44.833 INFO:tasks.workunit.client.0.smithi121.stdout:5/672: symlink dd/de/d19/d38/de3/lf3 0 2024-04-05T21:26:44.833 INFO:tasks.workunit.client.0.smithi121.stdout:8/498: dread - d12/d50/d61/d7f/f97 zero size 2024-04-05T21:26:44.835 INFO:tasks.workunit.client.1.smithi188.stdout:0/682: creat d4/d8/d78/d9d/ffe x:0 0 0 2024-04-05T21:26:44.836 INFO:tasks.workunit.client.1.smithi188.stdout:0/683: chown d4/d6/d13/c4b 53096717 1 2024-04-05T21:26:44.837 INFO:tasks.workunit.client.0.smithi121.stdout:5/673: write dd/de/d19/d38/d56/f92 [3291663,8355] 0 2024-04-05T21:26:44.838 INFO:tasks.workunit.client.1.smithi188.stdout:2/812: truncate d5/db/d35/d3b/d6b/f7e 725814 0 2024-04-05T21:26:44.839 INFO:tasks.workunit.client.1.smithi188.stdout:5/673: read - d6/d17/d23/d7d/dcf/fd6 zero size 2024-04-05T21:26:44.839 INFO:tasks.workunit.client.0.smithi121.stdout:3/417: dwrite d0/d3/de/f17 [0,4194304] 0 2024-04-05T21:26:44.841 INFO:tasks.workunit.client.0.smithi121.stdout:8/499: dread d12/f89 [0,4194304] 0 2024-04-05T21:26:44.842 INFO:tasks.workunit.client.0.smithi121.stdout:8/500: dread - d12/d13/d56/d57/f5a zero size 2024-04-05T21:26:44.843 INFO:tasks.workunit.client.0.smithi121.stdout:5/674: mknod dd/de/d19/d3c/d8e/cf4 0 2024-04-05T21:26:44.844 INFO:tasks.workunit.client.1.smithi188.stdout:0/684: symlink d4/d8/db/dac/lff 0 2024-04-05T21:26:44.845 INFO:tasks.workunit.client.1.smithi188.stdout:2/813: rename d5/db/dd/d67 to d5/db/dd/d73/d109/d9f/d10c 0 2024-04-05T21:26:44.846 INFO:tasks.workunit.client.0.smithi121.stdout:5/675: write dd/de/d39/d3b/fc0 [467955,85042] 0 2024-04-05T21:26:44.851 INFO:tasks.workunit.client.0.smithi121.stdout:3/418: dwrite d0/d3/d24/d3f/f4b [0,4194304] 0 2024-04-05T21:26:44.852 INFO:tasks.workunit.client.0.smithi121.stdout:3/419: chown d0/d3/de 6569 1 2024-04-05T21:26:44.854 INFO:tasks.workunit.client.1.smithi188.stdout:5/674: dwrite d6/d15/d5c/d7a/d44/d98/fcd [0,4194304] 0 2024-04-05T21:26:44.854 INFO:tasks.workunit.client.1.smithi188.stdout:6/776: dread d6/d13/d18/d19/d2c/d3f/d77/fb4 [0,4194304] 0 2024-04-05T21:26:44.856 INFO:tasks.workunit.client.1.smithi188.stdout:0/685: read d4/d8/db/d42/d38/f82 [2711351,87266] 0 2024-04-05T21:26:44.857 INFO:tasks.workunit.client.1.smithi188.stdout:2/814: mkdir d5/db/dd/d73/d109/d66/d8f/d10d 0 2024-04-05T21:26:44.863 INFO:tasks.workunit.client.0.smithi121.stdout:8/501: truncate d12/d13/d15/d1d/f58 1279706 0 2024-04-05T21:26:44.868 INFO:tasks.workunit.client.1.smithi188.stdout:6/777: mknod d6/d13/d18/d3c/ddc/d60/cea 0 2024-04-05T21:26:44.869 INFO:tasks.workunit.client.1.smithi188.stdout:6/778: chown d6/d13/d18/d19/dc8/c9b 1819 1 2024-04-05T21:26:44.869 INFO:tasks.workunit.client.1.smithi188.stdout:2/815: mkdir d5/db/dd/d73/d109/d9f/d10c/dcd/d10e 0 2024-04-05T21:26:44.872 INFO:tasks.workunit.client.1.smithi188.stdout:4/982: truncate d0/dbe/f10c 3000419 0 2024-04-05T21:26:44.873 INFO:tasks.workunit.client.0.smithi121.stdout:8/502: dread d12/d13/d56/d57/d5e/f62 [0,4194304] 0 2024-04-05T21:26:44.874 INFO:tasks.workunit.client.0.smithi121.stdout:5/676: dwrite dd/de/d6d/f6e [0,4194304] 0 2024-04-05T21:26:44.874 INFO:tasks.workunit.client.0.smithi121.stdout:3/420: dwrite d0/f2d [0,4194304] 0 2024-04-05T21:26:44.874 INFO:tasks.workunit.client.0.smithi121.stdout:8/503: stat d12/f89 0 2024-04-05T21:26:44.875 INFO:tasks.workunit.client.1.smithi188.stdout:6/779: fdatasync d6/d13/d18/d19/d2c/d34/f6b 0 2024-04-05T21:26:44.876 INFO:tasks.workunit.client.0.smithi121.stdout:0/574: sync 2024-04-05T21:26:44.877 INFO:tasks.workunit.client.0.smithi121.stdout:5/677: creat dd/d41/dce/ff5 x:0 0 0 2024-04-05T21:26:44.878 INFO:tasks.workunit.client.1.smithi188.stdout:4/983: mkdir d0/d2/d17/d3a/d14b 0 2024-04-05T21:26:44.879 INFO:tasks.workunit.client.0.smithi121.stdout:0/575: mkdir d7/d71/d94/dd1 0 2024-04-05T21:26:44.881 INFO:tasks.workunit.client.0.smithi121.stdout:3/421: unlink d0/d3/de/l12 0 2024-04-05T21:26:44.881 INFO:tasks.workunit.client.1.smithi188.stdout:2/816: fdatasync d5/db/dd/d36/d3c/d48/d7a/f99 0 2024-04-05T21:26:44.882 INFO:tasks.workunit.client.1.smithi188.stdout:6/780: stat d6/d13/d18/d19/d2c/f65 0 2024-04-05T21:26:44.883 INFO:tasks.workunit.client.0.smithi121.stdout:5/678: rename dd/de/d39/ld8 to dd/lf6 0 2024-04-05T21:26:44.884 INFO:tasks.workunit.client.0.smithi121.stdout:3/422: write d0/d3/de/f8a [278390,95367] 0 2024-04-05T21:26:44.889 INFO:tasks.workunit.client.1.smithi188.stdout:3/804: dwrite d5/d7e/d101/fa8 [0,4194304] 0 2024-04-05T21:26:44.891 INFO:tasks.workunit.client.0.smithi121.stdout:0/576: dread d7/d71/d45/d5a/f80 [0,4194304] 0 2024-04-05T21:26:44.893 INFO:tasks.workunit.client.0.smithi121.stdout:0/577: creat d7/d71/d29/d3f/fd2 x:0 0 0 2024-04-05T21:26:44.896 INFO:tasks.workunit.client.1.smithi188.stdout:0/686: dwrite d4/d6/d13/d23/dd1/fdd [0,4194304] 0 2024-04-05T21:26:44.902 INFO:tasks.workunit.client.1.smithi188.stdout:7/984: dwrite d3/d2c/dae/dfd/d2e/d8c/db7/dd6/fde [0,4194304] 0 2024-04-05T21:26:44.903 INFO:tasks.workunit.client.1.smithi188.stdout:4/984: fsync d0/d14/d15/d53/d82/dd5/ff1 0 2024-04-05T21:26:44.908 INFO:tasks.workunit.client.1.smithi188.stdout:5/675: dwrite d6/d15/d5c/d7a/d44/d98/f9f [0,4194304] 0 2024-04-05T21:26:44.911 INFO:tasks.workunit.client.1.smithi188.stdout:0/687: unlink d4/d6/d13/d23/d24/d57/f92 0 2024-04-05T21:26:44.912 INFO:tasks.workunit.client.1.smithi188.stdout:5/676: readlink d6/dad/l58 0 2024-04-05T21:26:44.913 INFO:tasks.workunit.client.0.smithi121.stdout:3/423: rmdir d0 39 2024-04-05T21:26:44.914 INFO:tasks.workunit.client.1.smithi188.stdout:0/688: write d4/d8/db/d42/d38/f49 [5966126,115280] 0 2024-04-05T21:26:44.915 INFO:tasks.workunit.client.1.smithi188.stdout:2/817: rename d5/db/f20 to d5/db/dd/d24/f10f 0 2024-04-05T21:26:44.919 INFO:tasks.workunit.client.0.smithi121.stdout:5/679: mknod dd/de/d19/d13/cf7 0 2024-04-05T21:26:44.921 INFO:tasks.workunit.client.1.smithi188.stdout:0/689: readlink d4/d8/db/d21/l65 0 2024-04-05T21:26:44.921 INFO:tasks.workunit.client.1.smithi188.stdout:0/690: write d4/d8/db/d42/d38/f54 [226436,82978] 0 2024-04-05T21:26:44.924 INFO:tasks.workunit.client.1.smithi188.stdout:6/781: rmdir d6/d13/d18/d19/d2c/d3f/d77/db0 0 2024-04-05T21:26:44.925 INFO:tasks.workunit.client.1.smithi188.stdout:5/677: dwrite d6/f9 [0,4194304] 0 2024-04-05T21:26:44.926 INFO:tasks.workunit.client.0.smithi121.stdout:5/680: dwrite dd/de/d39/d3b/d50/f67 [0,4194304] 0 2024-04-05T21:26:44.926 INFO:tasks.workunit.client.1.smithi188.stdout:3/805: rename d5/d7/d31/c5d to d5/d7/d6d/c113 0 2024-04-05T21:26:44.927 INFO:tasks.workunit.client.0.smithi121.stdout:5/681: stat dd/de/d19 0 2024-04-05T21:26:44.927 INFO:tasks.workunit.client.0.smithi121.stdout:5/682: readlink dd/de/lb4 0 2024-04-05T21:26:44.929 INFO:tasks.workunit.client.1.smithi188.stdout:5/678: write f2 [758967,25806] 0 2024-04-05T21:26:44.929 INFO:tasks.workunit.client.1.smithi188.stdout:7/985: creat d3/d2c/dae/dfd/d10/f142 x:0 0 0 2024-04-05T21:26:44.931 INFO:tasks.workunit.client.1.smithi188.stdout:6/782: chown d6/d13/d18/d19/d2c/f32 34 1 2024-04-05T21:26:44.931 INFO:tasks.workunit.client.0.smithi121.stdout:5/683: mkdir dd/d41/df8 0 2024-04-05T21:26:44.932 INFO:tasks.workunit.client.0.smithi121.stdout:5/684: fsync dd/de/d19/d1c/d74/fa4 0 2024-04-05T21:26:44.933 INFO:tasks.workunit.client.1.smithi188.stdout:0/691: symlink d4/d8/db/d42/d71/dc5/l100 0 2024-04-05T21:26:44.935 INFO:tasks.workunit.client.1.smithi188.stdout:3/806: mkdir d5/d7/d65/d61/d66/da4/d114 0 2024-04-05T21:26:44.935 INFO:tasks.workunit.client.0.smithi121.stdout:5/685: unlink dd/de/d39/d3b/l5a 0 2024-04-05T21:26:44.938 INFO:tasks.workunit.client.1.smithi188.stdout:5/679: unlink d6/d17/d23/d85/fc5 0 2024-04-05T21:26:44.938 INFO:tasks.workunit.client.0.smithi121.stdout:5/686: mkdir dd/de/df9 0 2024-04-05T21:26:44.940 INFO:tasks.workunit.client.0.smithi121.stdout:0/578: dread d7/dc/fab [0,4194304] 0 2024-04-05T21:26:44.941 INFO:tasks.workunit.client.0.smithi121.stdout:8/504: dread d12/d13/d15/f4b [0,4194304] 0 2024-04-05T21:26:44.942 INFO:tasks.workunit.client.0.smithi121.stdout:0/579: truncate d7/d62/d9d/fc0 287731 0 2024-04-05T21:26:44.944 INFO:tasks.workunit.client.0.smithi121.stdout:5/687: rename dd/de/d39/d3b/f5e to dd/de/d19/d13/d53/dba/ffa 0 2024-04-05T21:26:44.944 INFO:tasks.workunit.client.1.smithi188.stdout:6/783: mkdir d6/d13/deb 0 2024-04-05T21:26:44.946 INFO:tasks.workunit.client.1.smithi188.stdout:5/680: truncate d6/d17/d23/f7e 208849 0 2024-04-05T21:26:44.947 INFO:tasks.workunit.client.0.smithi121.stdout:3/424: write d0/d3/f54 [449677,42112] 0 2024-04-05T21:26:44.947 INFO:tasks.workunit.client.1.smithi188.stdout:6/784: chown d6/d13/d18/d3c/ddc/d60/f7c 87857925 1 2024-04-05T21:26:44.948 INFO:tasks.workunit.client.0.smithi121.stdout:0/580: creat d7/dc/d95/fd3 x:0 0 0 2024-04-05T21:26:44.949 INFO:tasks.workunit.client.0.smithi121.stdout:8/505: dread fa [0,4194304] 0 2024-04-05T21:26:44.949 INFO:tasks.workunit.client.1.smithi188.stdout:7/986: link d3/d2c/dae/dfd/d2e/f6b d3/d2c/d5e/d91/f143 0 2024-04-05T21:26:44.949 INFO:tasks.workunit.client.0.smithi121.stdout:3/425: chown d0/c4a 1 1 2024-04-05T21:26:44.951 INFO:tasks.workunit.client.1.smithi188.stdout:7/987: read d3/d2c/dce/fad [347353,68210] 0 2024-04-05T21:26:44.951 INFO:tasks.workunit.client.0.smithi121.stdout:8/506: chown d12/d13/d15/d1d/d29 102 1 2024-04-05T21:26:44.951 INFO:tasks.workunit.client.1.smithi188.stdout:6/785: symlink d6/d13/d18/d19/dc8/lec 0 2024-04-05T21:26:44.952 INFO:tasks.workunit.client.0.smithi121.stdout:5/688: link dd/de/d19/d1c/d36/c3d dd/de/d19/d1c/d7b/cfb 0 2024-04-05T21:26:44.952 INFO:tasks.workunit.client.0.smithi121.stdout:3/426: write d0/d3/d24/d3d/f3a [2217703,16990] 0 2024-04-05T21:26:44.952 INFO:tasks.workunit.client.0.smithi121.stdout:0/581: mkdir d7/d71/d94/dd1/dd4 0 2024-04-05T21:26:44.953 INFO:tasks.workunit.client.1.smithi188.stdout:7/988: creat d3/da5/dc8/f144 x:0 0 0 2024-04-05T21:26:44.953 INFO:tasks.workunit.client.0.smithi121.stdout:5/689: dread - dd/de/d19/d1c/d74/fa4 zero size 2024-04-05T21:26:44.954 INFO:tasks.workunit.client.0.smithi121.stdout:0/582: dread d7/d71/d45/d5a/f80 [0,4194304] 0 2024-04-05T21:26:44.954 INFO:tasks.workunit.client.0.smithi121.stdout:8/507: creat d12/d13/d56/fa2 x:0 0 0 2024-04-05T21:26:44.955 INFO:tasks.workunit.client.0.smithi121.stdout:3/427: chown d0/d3/c26 1696300 1 2024-04-05T21:26:44.956 INFO:tasks.workunit.client.1.smithi188.stdout:7/989: chown d3/d2c/d5e/l8f 3 1 2024-04-05T21:26:44.957 INFO:tasks.workunit.client.0.smithi121.stdout:5/690: write dd/de/d19/d3c/f5f [4015535,56482] 0 2024-04-05T21:26:44.959 INFO:tasks.workunit.client.1.smithi188.stdout:4/985: dwrite d0/d2/d52/dc1/fe6 [0,4194304] 0 2024-04-05T21:26:44.959 INFO:tasks.workunit.client.1.smithi188.stdout:7/990: mknod d3/d4/d101/c145 0 2024-04-05T21:26:44.961 INFO:tasks.workunit.client.0.smithi121.stdout:0/583: creat d7/d71/d94/dd1/fd5 x:0 0 0 2024-04-05T21:26:44.962 INFO:tasks.workunit.client.0.smithi121.stdout:3/428: creat d0/d3/d24/d3d/d1a/d1e/d86/f8b x:0 0 0 2024-04-05T21:26:44.962 INFO:tasks.workunit.client.0.smithi121.stdout:0/584: write d7/d1c/f5b [1818410,17591] 0 2024-04-05T21:26:44.963 INFO:tasks.workunit.client.0.smithi121.stdout:0/585: chown d7/d71/d29/d3f/fd2 0 1 2024-04-05T21:26:44.964 INFO:tasks.workunit.client.1.smithi188.stdout:7/991: write d3/d4/d7c/fd9 [2930925,25855] 0 2024-04-05T21:26:44.964 INFO:tasks.workunit.client.1.smithi188.stdout:4/986: symlink d0/d2/dc7/l14c 0 2024-04-05T21:26:44.967 INFO:tasks.workunit.client.0.smithi121.stdout:3/429: link d0/d3/d24/d3d/l50 d0/d3/de/l8c 0 2024-04-05T21:26:44.969 INFO:tasks.workunit.client.1.smithi188.stdout:5/681: write d6/d15/d1b/f62 [303857,16232] 0 2024-04-05T21:26:44.970 INFO:tasks.workunit.client.1.smithi188.stdout:6/786: write d6/d13/d18/f59 [2098229,27344] 0 2024-04-05T21:26:44.971 INFO:tasks.workunit.client.0.smithi121.stdout:3/430: rmdir d0/d3/d24/d3d/d1a/d23 39 2024-04-05T21:26:44.971 INFO:tasks.workunit.client.1.smithi188.stdout:4/987: rmdir d0/d14/d15/d53/d12e 39 2024-04-05T21:26:44.971 INFO:tasks.workunit.client.0.smithi121.stdout:0/586: dwrite d7/f84 [0,4194304] 0 2024-04-05T21:26:44.973 INFO:tasks.workunit.client.0.smithi121.stdout:3/431: write d0/f27 [1032974,84500] 0 2024-04-05T21:26:44.974 INFO:tasks.workunit.client.1.smithi188.stdout:0/692: dwrite d4/d8/db/d42/d38/f82 [0,4194304] 0 2024-04-05T21:26:44.975 INFO:tasks.workunit.client.1.smithi188.stdout:3/807: dwrite d5/d7/d6d/dd8/f2d [0,4194304] 0 2024-04-05T21:26:44.977 INFO:tasks.workunit.client.1.smithi188.stdout:7/992: chown d3/d2c/dae/dfd/d10/f82 0 1 2024-04-05T21:26:44.977 INFO:tasks.workunit.client.1.smithi188.stdout:6/787: rmdir d6/d13 39 2024-04-05T21:26:44.977 INFO:tasks.workunit.client.0.smithi121.stdout:0/587: unlink d7/d62/f42 0 2024-04-05T21:26:44.978 INFO:tasks.workunit.client.1.smithi188.stdout:6/788: fdatasync d6/fbd 0 2024-04-05T21:26:44.980 INFO:tasks.workunit.client.1.smithi188.stdout:5/682: chown d6/d15/d5c/d7a/d44/c4b 741011 1 2024-04-05T21:26:44.980 INFO:tasks.workunit.client.1.smithi188.stdout:4/988: getdents d0/d2/d52/d126 0 2024-04-05T21:26:44.981 INFO:tasks.workunit.client.1.smithi188.stdout:0/693: mkdir d4/d6/d13/d23/d24/d101 0 2024-04-05T21:26:44.981 INFO:tasks.workunit.client.1.smithi188.stdout:3/808: write d5/d7/d6d/dd8/f2d [1925296,41335] 0 2024-04-05T21:26:44.982 INFO:tasks.workunit.client.0.smithi121.stdout:0/588: dread d7/d71/d29/d2a/d2c/f7b [0,4194304] 0 2024-04-05T21:26:44.983 INFO:tasks.workunit.client.0.smithi121.stdout:3/432: dwrite d0/d3/d24/d3d/f21 [0,4194304] 0 2024-04-05T21:26:44.984 INFO:tasks.workunit.client.0.smithi121.stdout:0/589: write d7/d71/f26 [610149,110026] 0 2024-04-05T21:26:44.986 INFO:tasks.workunit.client.1.smithi188.stdout:4/989: symlink d0/d14/l14d 0 2024-04-05T21:26:44.987 INFO:tasks.workunit.client.1.smithi188.stdout:6/789: readlink d6/d13/d18/d19/l66 0 2024-04-05T21:26:44.987 INFO:tasks.workunit.client.0.smithi121.stdout:3/433: dread - d0/d3/d24/d3d/d1a/d1e/f71 zero size 2024-04-05T21:26:44.988 INFO:tasks.workunit.client.0.smithi121.stdout:3/434: truncate d0/f2d 4302759 0 2024-04-05T21:26:44.989 INFO:tasks.workunit.client.1.smithi188.stdout:3/809: creat d5/d7e/d101/de5/f115 x:0 0 0 2024-04-05T21:26:44.990 INFO:tasks.workunit.client.1.smithi188.stdout:4/990: creat d0/d81/f14e x:0 0 0 2024-04-05T21:26:44.991 INFO:tasks.workunit.client.0.smithi121.stdout:0/590: creat d7/d62/d9d/fd6 x:0 0 0 2024-04-05T21:26:44.997 INFO:tasks.workunit.client.1.smithi188.stdout:4/991: chown d0/d2/d2a/d34/f41 31456699 1 2024-04-05T21:26:44.997 INFO:tasks.workunit.client.1.smithi188.stdout:0/694: creat d4/d8/db/d42/da2/dd2/f102 x:0 0 0 2024-04-05T21:26:44.998 INFO:tasks.workunit.client.0.smithi121.stdout:0/591: dwrite d7/dc/d95/fd3 [0,4194304] 0 2024-04-05T21:26:44.998 INFO:tasks.workunit.client.1.smithi188.stdout:3/810: chown d5/d48/l105 16329 1 2024-04-05T21:26:45.000 INFO:tasks.workunit.client.0.smithi121.stdout:0/592: write d7/d62/d9d/fd6 [1010656,81266] 0 2024-04-05T21:26:45.002 INFO:tasks.workunit.client.1.smithi188.stdout:7/993: dwrite d3/d4/d1d/d11d/f140 [0,4194304] 0 2024-04-05T21:26:45.003 INFO:tasks.workunit.client.0.smithi121.stdout:0/593: unlink d7/d71/f8f 0 2024-04-05T21:26:45.004 INFO:tasks.workunit.client.0.smithi121.stdout:0/594: chown d7/d71/d29/d2a/d2c/f7c 786762 1 2024-04-05T21:26:45.005 INFO:tasks.workunit.client.0.smithi121.stdout:0/595: stat d7/dc/faf 0 2024-04-05T21:26:45.008 INFO:tasks.workunit.client.0.smithi121.stdout:0/596: rename d7/d71/d29/d2a/d64/f9f to d7/d71/d94/fd7 0 2024-04-05T21:26:45.011 INFO:tasks.workunit.client.1.smithi188.stdout:5/683: dwrite d6/d15/d5c/d7a/d44/d98/f9f [4194304,4194304] 0 2024-04-05T21:26:45.014 INFO:tasks.workunit.client.0.smithi121.stdout:0/597: symlink d7/d1c/ld8 0 2024-04-05T21:26:45.018 INFO:tasks.workunit.client.1.smithi188.stdout:0/695: rename d4/d8/db/d21/d4a/d73/d79 to d4/d6/d13/d103 0 2024-04-05T21:26:45.020 INFO:tasks.workunit.client.1.smithi188.stdout:0/696: write d4/d8/db/d42/da2/fa7 [3287085,87416] 0 2024-04-05T21:26:45.030 INFO:tasks.workunit.client.1.smithi188.stdout:6/790: dread d6/d13/d38/f63 [0,4194304] 0 2024-04-05T21:26:45.031 INFO:tasks.workunit.client.1.smithi188.stdout:6/791: readlink d6/d13/d38/l3a 0 2024-04-05T21:26:45.038 INFO:tasks.workunit.client.0.smithi121.stdout:3/435: dread d0/d3/de/f36 [0,4194304] 0 2024-04-05T21:26:45.041 INFO:tasks.workunit.client.0.smithi121.stdout:0/598: getdents d7/d71/d29/d2a/d64 0 2024-04-05T21:26:45.042 INFO:tasks.workunit.client.0.smithi121.stdout:0/599: write d7/d71/d29/d3f/f72 [2003271,25040] 0 2024-04-05T21:26:45.042 INFO:tasks.workunit.client.1.smithi188.stdout:0/697: truncate d4/d6/faa 2906204 0 2024-04-05T21:26:45.043 INFO:tasks.workunit.client.1.smithi188.stdout:6/792: write d6/d13/d18/d3c/ddc/d60/fe7 [1813146,33807] 0 2024-04-05T21:26:45.045 INFO:tasks.workunit.client.0.smithi121.stdout:0/600: getdents d7/dc/d95 0 2024-04-05T21:26:45.047 INFO:tasks.workunit.client.0.smithi121.stdout:3/436: rename d0/d3/d24/d3d/d1a/d23/f70 to d0/d3/d24/d3d/d1a/d23/d38/d55/f8d 0 2024-04-05T21:26:45.048 INFO:tasks.workunit.client.0.smithi121.stdout:0/601: symlink d7/d71/d29/d2a/d39/ld9 0 2024-04-05T21:26:45.049 INFO:tasks.workunit.client.0.smithi121.stdout:3/437: mkdir d0/d3/d24/d3d/d1a/d1e/d86/d8e 0 2024-04-05T21:26:45.050 INFO:tasks.workunit.client.0.smithi121.stdout:0/602: rmdir d7/d71/d45/d46/d53/d8e/dc4 39 2024-04-05T21:26:45.050 INFO:tasks.workunit.client.0.smithi121.stdout:3/438: read d0/d3/de/f17 [2327640,79897] 0 2024-04-05T21:26:45.052 INFO:tasks.workunit.client.1.smithi188.stdout:6/793: readlink d6/d13/d18/d19/d2c/d3f/l8a 0 2024-04-05T21:26:45.053 INFO:tasks.workunit.client.0.smithi121.stdout:3/439: write d0/d3/d24/d3d/f21 [1343086,85039] 0 2024-04-05T21:26:45.054 INFO:tasks.workunit.client.1.smithi188.stdout:7/994: dwrite d3/d2c/dae/dfd/f38 [0,4194304] 0 2024-04-05T21:26:45.055 INFO:tasks.workunit.client.1.smithi188.stdout:3/811: dwrite d5/d48/f63 [0,4194304] 0 2024-04-05T21:26:45.055 INFO:tasks.workunit.client.1.smithi188.stdout:5/684: dread d6/d15/d5c/d7a/f3c [0,4194304] 0 2024-04-05T21:26:45.058 INFO:tasks.workunit.client.1.smithi188.stdout:7/995: readlink d3/d2c/dce/lca 0 2024-04-05T21:26:45.058 INFO:tasks.workunit.client.1.smithi188.stdout:0/698: write d4/d6/d46/fe7 [2596802,51412] 0 2024-04-05T21:26:45.058 INFO:tasks.workunit.client.0.smithi121.stdout:3/440: rename d0/d3/d24/d3d/d1a/c87 to d0/d3/d24/d3d/d1a/d23/d38/c8f 0 2024-04-05T21:26:45.059 INFO:tasks.workunit.client.1.smithi188.stdout:4/992: dwrite d0/d2/d2a/f7f [0,4194304] 0 2024-04-05T21:26:45.060 INFO:tasks.workunit.client.0.smithi121.stdout:0/603: link d7/d62/d8b/fac d7/fda 0 2024-04-05T21:26:45.061 INFO:tasks.workunit.client.1.smithi188.stdout:3/812: chown d5/d7/d65/d5f/fe2 176072 1 2024-04-05T21:26:45.062 INFO:tasks.workunit.client.0.smithi121.stdout:0/604: chown d7/d71/d29/d2a/d64/f83 30425670 1 2024-04-05T21:26:45.063 INFO:tasks.workunit.client.1.smithi188.stdout:5/685: chown d6/d15/d5c/d7a/d44/c96 2051649711 1 2024-04-05T21:26:45.064 INFO:tasks.workunit.client.1.smithi188.stdout:6/794: chown d6/d13/d18/d3c/ddc/f9c 2268009 1 2024-04-05T21:26:45.064 INFO:tasks.workunit.client.1.smithi188.stdout:3/813: chown d5/d7/d65/f54 4782446 1 2024-04-05T21:26:45.068 INFO:tasks.workunit.client.1.smithi188.stdout:4/993: rmdir d0/d14/d15/d1b/d132 39 2024-04-05T21:26:45.070 INFO:tasks.workunit.client.0.smithi121.stdout:0/605: dwrite d7/d62/d9d/fbb [0,4194304] 0 2024-04-05T21:26:45.071 INFO:tasks.workunit.client.1.smithi188.stdout:6/795: rmdir d6/d13/d18/d19/d61 39 2024-04-05T21:26:45.072 INFO:tasks.workunit.client.1.smithi188.stdout:7/996: creat d3/d2c/dae/dfd/d10/d5f/f146 x:0 0 0 2024-04-05T21:26:45.072 INFO:tasks.workunit.client.1.smithi188.stdout:3/814: truncate d5/d7/d65/d5f/fc6 131479 0 2024-04-05T21:26:45.074 INFO:tasks.workunit.client.0.smithi121.stdout:0/606: mkdir d7/d62/ddb 0 2024-04-05T21:26:45.075 INFO:tasks.workunit.client.0.smithi121.stdout:0/607: readlink d7/l1e 0 2024-04-05T21:26:45.075 INFO:tasks.workunit.client.0.smithi121.stdout:0/608: stat d7/d62/c76 0 2024-04-05T21:26:45.076 INFO:tasks.workunit.client.0.smithi121.stdout:0/609: fdatasync d7/d71/d29/f58 0 2024-04-05T21:26:45.078 INFO:tasks.workunit.client.0.smithi121.stdout:0/610: getdents d7/d71/d94/dd1/dd4 0 2024-04-05T21:26:45.078 INFO:tasks.workunit.client.1.smithi188.stdout:5/686: dwrite d6/f73 [0,4194304] 0 2024-04-05T21:26:45.080 INFO:tasks.workunit.client.0.smithi121.stdout:0/611: creat d7/d71/fdc x:0 0 0 2024-04-05T21:26:45.082 INFO:tasks.workunit.client.0.smithi121.stdout:4/682: sync 2024-04-05T21:26:45.082 INFO:tasks.workunit.client.0.smithi121.stdout:9/401: sync 2024-04-05T21:26:45.082 INFO:tasks.workunit.client.0.smithi121.stdout:7/610: sync 2024-04-05T21:26:45.082 INFO:tasks.workunit.client.0.smithi121.stdout:0/612: write d7/d71/d29/f79 [3712155,108259] 0 2024-04-05T21:26:45.083 INFO:tasks.workunit.client.1.smithi188.stdout:6/796: symlink d6/d13/d18/d3c/daf/led 0 2024-04-05T21:26:45.083 INFO:tasks.workunit.client.1.smithi188.stdout:7/997: creat d3/d2c/dae/dfd/d2e/d3b/d94/f147 x:0 0 0 2024-04-05T21:26:45.084 INFO:tasks.workunit.client.1.smithi188.stdout:7/998: stat d3/d4/d7c/ce6 0 2024-04-05T21:26:45.086 INFO:tasks.workunit.client.1.smithi188.stdout:3/815: creat d5/d7/d65/d5f/d102/f116 x:0 0 0 2024-04-05T21:26:45.086 INFO:tasks.workunit.client.1.smithi188.stdout:6/797: readlink d6/d28/d45/ldf 0 2024-04-05T21:26:45.086 INFO:tasks.workunit.client.1.smithi188.stdout:5/687: rename d6/d15/d5c/ccc to d6/d17/d23/d7d/ce9 0 2024-04-05T21:26:45.086 INFO:tasks.workunit.client.0.smithi121.stdout:7/611: rename d0/d23/d5b/f2f to d0/d23/d5b/fbe 0 2024-04-05T21:26:45.089 INFO:tasks.workunit.client.0.smithi121.stdout:4/683: mknod d5/d11/d26/dbc/dc5/cdd 0 2024-04-05T21:26:45.092 INFO:tasks.workunit.client.0.smithi121.stdout:0/613: dwrite d7/d62/fc2 [0,4194304] 0 2024-04-05T21:26:45.099 INFO:tasks.workunit.client.0.smithi121.stdout:9/402: link d6/f15 d6/df/d1d/d2c/d37/f86 0 2024-04-05T21:26:45.099 INFO:tasks.workunit.client.0.smithi121.stdout:9/403: readlink d6/df/d1d/d72/l75 0 2024-04-05T21:26:45.099 INFO:tasks.workunit.client.0.smithi121.stdout:7/612: rename d0/d23/f8c to d0/d9/d22/d3e/d9c/da7/fbf 0 2024-04-05T21:26:45.099 INFO:tasks.workunit.client.0.smithi121.stdout:0/614: rmdir d7/d71/d29/d2a 39 2024-04-05T21:26:45.099 INFO:tasks.workunit.client.0.smithi121.stdout:0/615: stat d7/l1e 0 2024-04-05T21:26:45.100 INFO:tasks.workunit.client.0.smithi121.stdout:9/404: rmdir d6/d36/d7b/d6b 39 2024-04-05T21:26:45.101 INFO:tasks.workunit.client.0.smithi121.stdout:7/613: mknod d0/d23/cc0 0 2024-04-05T21:26:45.103 INFO:tasks.workunit.client.0.smithi121.stdout:9/405: creat d6/d36/d7b/d5d/f87 x:0 0 0 2024-04-05T21:26:45.104 INFO:tasks.workunit.client.0.smithi121.stdout:9/406: dread - d6/df/d48/f62 zero size 2024-04-05T21:26:45.108 INFO:tasks.workunit.client.0.smithi121.stdout:0/616: dwrite d7/d71/d29/d2a/d64/f83 [0,4194304] 0 2024-04-05T21:26:45.108 INFO:tasks.workunit.client.0.smithi121.stdout:9/407: read d6/df/d1d/f23 [334737,68594] 0 2024-04-05T21:26:45.109 INFO:tasks.workunit.client.0.smithi121.stdout:9/408: stat d6/df/f2e 0 2024-04-05T21:26:45.111 INFO:tasks.workunit.client.0.smithi121.stdout:7/614: dwrite d0/d9/f1d [0,4194304] 0 2024-04-05T21:26:45.114 INFO:tasks.workunit.client.0.smithi121.stdout:7/615: stat d0/d23/d5b/d52/d61/l98 0 2024-04-05T21:26:45.119 INFO:tasks.workunit.client.0.smithi121.stdout:9/409: dwrite d6/df/d50/f6a [0,4194304] 0 2024-04-05T21:26:45.121 INFO:tasks.workunit.client.0.smithi121.stdout:9/410: write d6/df/d1d/d2c/d37/d3c/f7c [3537721,96101] 0 2024-04-05T21:26:45.123 INFO:tasks.workunit.client.0.smithi121.stdout:9/411: dread d6/df/d1d/f23 [0,4194304] 0 2024-04-05T21:26:45.125 INFO:tasks.workunit.client.0.smithi121.stdout:4/684: dread d5/d11/d28/f13 [0,4194304] 0 2024-04-05T21:26:45.126 INFO:tasks.workunit.client.0.smithi121.stdout:9/412: creat d6/df/d1d/d2c/d53/f88 x:0 0 0 2024-04-05T21:26:45.128 INFO:tasks.workunit.client.0.smithi121.stdout:9/413: mknod d6/d36/c89 0 2024-04-05T21:26:45.130 INFO:tasks.workunit.client.0.smithi121.stdout:9/414: mkdir d6/d46/d8a 0 2024-04-05T21:26:45.131 INFO:tasks.workunit.client.1.smithi188.stdout:9/811: sync 2024-04-05T21:26:45.132 INFO:tasks.workunit.client.1.smithi188.stdout:7/999: rmdir d3/d2c/dae/dfd/d10/d5f 39 2024-04-05T21:26:45.134 INFO:tasks.workunit.client.0.smithi121.stdout:9/415: dread d6/df/d1d/d2c/d37/d3c/f7c [0,4194304] 0 2024-04-05T21:26:45.138 INFO:tasks.workunit.client.1.smithi188.stdout:4/994: write d0/d14/d2f/f13c [847641,59564] 0 2024-04-05T21:26:45.139 INFO:tasks.workunit.client.0.smithi121.stdout:9/416: rename d6/df/d50/f6a to d6/d36/d7b/d49/f8b 0 2024-04-05T21:26:45.141 INFO:tasks.workunit.client.0.smithi121.stdout:9/417: creat d6/d36/d7b/d6b/f8c x:0 0 0 2024-04-05T21:26:45.141 INFO:tasks.workunit.client.1.smithi188.stdout:6/798: symlink d6/d13/lee 0 2024-04-05T21:26:45.141 INFO:tasks.workunit.client.0.smithi121.stdout:9/418: read d6/d36/d7b/d49/f8b [3418943,102106] 0 2024-04-05T21:26:45.142 INFO:tasks.workunit.client.0.smithi121.stdout:9/419: fdatasync d6/df/d1d/f23 0 2024-04-05T21:26:45.143 INFO:tasks.workunit.client.0.smithi121.stdout:9/420: write d6/d36/d7b/d5d/f87 [509913,49958] 0 2024-04-05T21:26:45.144 INFO:tasks.workunit.client.1.smithi188.stdout:4/995: chown d0/d14/d15/d53/d12e 2391 1 2024-04-05T21:26:45.146 INFO:tasks.workunit.client.0.smithi121.stdout:9/421: write d6/df/d48/f4c [470846,78168] 0 2024-04-05T21:26:45.146 INFO:tasks.workunit.client.1.smithi188.stdout:6/799: dread - d6/d13/d48/fc5 zero size 2024-04-05T21:26:45.147 INFO:tasks.workunit.client.0.smithi121.stdout:3/441: dread d0/d3/de/f82 [0,4194304] 0 2024-04-05T21:26:45.147 INFO:tasks.workunit.client.0.smithi121.stdout:9/422: chown d6/d36/d7b/f3b 1930719 1 2024-04-05T21:26:45.148 INFO:tasks.workunit.client.1.smithi188.stdout:9/812: dwrite dd/d3d/d81/f107 [0,4194304] 0 2024-04-05T21:26:45.148 INFO:tasks.workunit.client.1.smithi188.stdout:4/996: readlink d0/d2/d2a/d34/d6b/lf5 0 2024-04-05T21:26:45.150 INFO:tasks.workunit.client.1.smithi188.stdout:6/800: mknod d6/d28/d45/cef 0 2024-04-05T21:26:45.151 INFO:tasks.workunit.client.1.smithi188.stdout:4/997: mkdir d0/d2/d2a/d34/d6b/d14f 0 2024-04-05T21:26:45.152 INFO:tasks.workunit.client.1.smithi188.stdout:9/813: mknod dd/d11/d1f/d26/dc0/d6b/d40/c116 0 2024-04-05T21:26:45.153 INFO:tasks.workunit.client.0.smithi121.stdout:9/423: link d6/df/d1d/d2c/d53/c5b d6/d36/d7b/d49/d66/c8d 0 2024-04-05T21:26:45.153 INFO:tasks.workunit.client.1.smithi188.stdout:6/801: creat d6/d13/d18/d19/da3/ff0 x:0 0 0 2024-04-05T21:26:45.153 INFO:tasks.workunit.client.1.smithi188.stdout:4/998: dread - d0/d14/d15/dd4/f145 zero size 2024-04-05T21:26:45.157 INFO:tasks.workunit.client.1.smithi188.stdout:9/814: mknod dd/d11/d1f/d26/dc0/d102/c117 0 2024-04-05T21:26:45.161 INFO:tasks.workunit.client.1.smithi188.stdout:9/815: write dd/d11/d1f/db2/ff5 [661291,101566] 0 2024-04-05T21:26:45.162 INFO:tasks.workunit.client.0.smithi121.stdout:3/442: dread d0/d3/fa [0,4194304] 0 2024-04-05T21:26:45.164 INFO:tasks.workunit.client.0.smithi121.stdout:0/617: truncate d7/f3a 139045 0 2024-04-05T21:26:45.165 INFO:tasks.workunit.client.0.smithi121.stdout:3/443: mkdir d0/d3/d24/d3d/d1a/d23/d90 0 2024-04-05T21:26:45.166 INFO:tasks.workunit.client.1.smithi188.stdout:9/816: mknod dd/d11/d1f/d26/dc0/d6b/d40/c118 0 2024-04-05T21:26:45.167 INFO:tasks.workunit.client.1.smithi188.stdout:3/816: dwrite d5/d7/d65/d61/fae [0,4194304] 0 2024-04-05T21:26:45.169 INFO:tasks.workunit.client.1.smithi188.stdout:9/817: read f1 [1979357,90110] 0 2024-04-05T21:26:45.170 INFO:tasks.workunit.client.0.smithi121.stdout:9/424: rmdir d6/df/d1d 39 2024-04-05T21:26:45.171 INFO:tasks.workunit.client.1.smithi188.stdout:3/817: fdatasync d5/d7/d65/f44 0 2024-04-05T21:26:45.172 INFO:tasks.workunit.client.0.smithi121.stdout:0/618: dwrite d7/d71/fdc [0,4194304] 0 2024-04-05T21:26:45.175 INFO:tasks.workunit.client.0.smithi121.stdout:9/425: symlink d6/d32/l8e 0 2024-04-05T21:26:45.179 INFO:tasks.workunit.client.1.smithi188.stdout:3/818: symlink d5/d7/d6d/d75/l117 0 2024-04-05T21:26:45.180 INFO:tasks.workunit.client.0.smithi121.stdout:4/685: dread d5/d11/d2c/d9d/f31 [0,4194304] 0 2024-04-05T21:26:45.181 INFO:tasks.workunit.client.0.smithi121.stdout:0/619: write d7/d71/d45/d46/d53/d8e/fbf [801719,44315] 0 2024-04-05T21:26:45.182 INFO:tasks.workunit.client.0.smithi121.stdout:9/426: unlink d6/df/d1d/d2c/d53/l54 0 2024-04-05T21:26:45.183 INFO:tasks.workunit.client.0.smithi121.stdout:4/686: mkdir d5/d11/d2c/d9d/dae/d29/d2e/dde 0 2024-04-05T21:26:45.185 INFO:tasks.workunit.client.0.smithi121.stdout:3/444: truncate d0/d3/de/f72 2206755 0 2024-04-05T21:26:45.186 INFO:tasks.workunit.client.0.smithi121.stdout:9/427: creat d6/d36/d7b/d6b/f8f x:0 0 0 2024-04-05T21:26:45.187 INFO:tasks.workunit.client.1.smithi188.stdout:5/688: dwrite d6/d15/d5c/f74 [0,4194304] 0 2024-04-05T21:26:45.187 INFO:tasks.workunit.client.0.smithi121.stdout:9/428: chown d6/df/l58 144 1 2024-04-05T21:26:45.188 INFO:tasks.workunit.client.0.smithi121.stdout:3/445: fdatasync d0/d3/d24/d3d/d1a/f22 0 2024-04-05T21:26:45.190 INFO:tasks.workunit.client.0.smithi121.stdout:4/687: dread d5/d11/d28/d73/f76 [0,4194304] 0 2024-04-05T21:26:45.191 INFO:tasks.workunit.client.1.smithi188.stdout:4/999: dread d0/d2/d2a/d34/fd1 [0,4194304] 0 2024-04-05T21:26:45.191 INFO:tasks.workunit.client.1.smithi188.stdout:3/819: rename d5/d7/d31/lf3 to d5/d48/l118 0 2024-04-05T21:26:45.191 INFO:tasks.workunit.client.0.smithi121.stdout:9/429: symlink d6/df/d1d/d2c/l90 0 2024-04-05T21:26:45.192 INFO:tasks.workunit.client.0.smithi121.stdout:3/446: dread d0/d3/d24/d3d/d1a/d1e/f2a [0,4194304] 0 2024-04-05T21:26:45.192 INFO:tasks.workunit.client.0.smithi121.stdout:9/430: readlink d6/d36/l29 0 2024-04-05T21:26:45.194 INFO:tasks.workunit.client.0.smithi121.stdout:3/447: truncate d0/d3/de/f8a 444676 0 2024-04-05T21:26:45.194 INFO:tasks.workunit.client.0.smithi121.stdout:9/431: rmdir d6/df/d1d/d2c 39 2024-04-05T21:26:45.194 INFO:tasks.workunit.client.0.smithi121.stdout:4/688: dread d5/d45/fb0 [0,4194304] 0 2024-04-05T21:26:45.195 INFO:tasks.workunit.client.0.smithi121.stdout:9/432: fsync d6/df/f20 0 2024-04-05T21:26:45.195 INFO:tasks.workunit.client.1.smithi188.stdout:9/818: dwrite dd/d11/d1f/d26/d55/d98/da5/fe6 [0,4194304] 0 2024-04-05T21:26:45.197 INFO:tasks.workunit.client.0.smithi121.stdout:9/433: chown d6/d36/d7b/d5d/f87 284682498 1 2024-04-05T21:26:45.197 INFO:tasks.workunit.client.1.smithi188.stdout:9/819: write dd/d11/d22/d7e/f91 [479754,44591] 0 2024-04-05T21:26:45.199 INFO:tasks.workunit.client.0.smithi121.stdout:3/448: mknod d0/d3/d24/d6d/d7b/c91 0 2024-04-05T21:26:45.201 INFO:tasks.workunit.client.0.smithi121.stdout:9/434: creat d6/d36/d7b/d49/d66/f91 x:0 0 0 2024-04-05T21:26:45.201 INFO:tasks.workunit.client.1.smithi188.stdout:5/689: chown d6/d82/d40/d42/f5e 201237 1 2024-04-05T21:26:45.204 INFO:tasks.workunit.client.0.smithi121.stdout:4/689: dwrite d5/d11/d28/f1b [4194304,4194304] 0 2024-04-05T21:26:45.204 INFO:tasks.workunit.client.1.smithi188.stdout:9/820: truncate dd/f72 2972170 0 2024-04-05T21:26:45.205 INFO:tasks.workunit.client.0.smithi121.stdout:4/690: truncate d5/d45/f74 4918357 0 2024-04-05T21:26:45.206 INFO:tasks.workunit.client.1.smithi188.stdout:6/802: dwrite d6/d7/f16 [4194304,4194304] 0 2024-04-05T21:26:45.206 INFO:tasks.workunit.client.1.smithi188.stdout:5/690: rmdir d6/d17/d39/d8e 39 2024-04-05T21:26:45.207 INFO:tasks.workunit.client.0.smithi121.stdout:4/691: write d5/fd [970149,7632] 0 2024-04-05T21:26:45.208 INFO:tasks.workunit.client.1.smithi188.stdout:9/821: write dd/d11/fd2 [3699429,13326] 0 2024-04-05T21:26:45.209 INFO:tasks.workunit.client.1.smithi188.stdout:3/820: rename d5/d7e/d101/cba to d5/d7/d6d/d75/d83/c119 0 2024-04-05T21:26:45.209 INFO:tasks.workunit.client.0.smithi121.stdout:4/692: readlink d5/d11/d26/d47/l92 0 2024-04-05T21:26:45.210 INFO:tasks.workunit.client.0.smithi121.stdout:9/435: link d6/df/d1d/d2c/f60 d6/d36/d7b/d6b/f92 0 2024-04-05T21:26:45.214 INFO:tasks.workunit.client.1.smithi188.stdout:3/821: chown d5/d7/d6d/d75/fb8 726922068 1 2024-04-05T21:26:45.217 INFO:tasks.workunit.client.1.smithi188.stdout:6/803: creat d6/d13/d18/d19/d2c/db2/ff1 x:0 0 0 2024-04-05T21:26:45.217 INFO:tasks.workunit.client.1.smithi188.stdout:9/822: mknod dd/d11/d73/dbb/c119 0 2024-04-05T21:26:45.217 INFO:tasks.workunit.client.0.smithi121.stdout:0/620: dwrite d7/d62/f5c [0,4194304] 0 2024-04-05T21:26:45.218 INFO:tasks.workunit.client.1.smithi188.stdout:6/804: fsync d6/d28/f43 0 2024-04-05T21:26:45.218 INFO:tasks.workunit.client.0.smithi121.stdout:9/436: creat d6/d46/d8a/f93 x:0 0 0 2024-04-05T21:26:45.219 INFO:tasks.workunit.client.0.smithi121.stdout:0/621: readlink d7/dc/lbc 0 2024-04-05T21:26:45.221 INFO:tasks.workunit.client.0.smithi121.stdout:4/693: dwrite d5/d11/d2c/d9d/dae/f30 [0,4194304] 0 2024-04-05T21:26:45.221 INFO:tasks.workunit.client.1.smithi188.stdout:3/822: write d5/d7/d31/f5c [5253941,67000] 0 2024-04-05T21:26:45.223 INFO:tasks.workunit.client.0.smithi121.stdout:9/437: mkdir d6/d36/d7b/d5d/d79/d94 0 2024-04-05T21:26:45.223 INFO:tasks.workunit.client.1.smithi188.stdout:6/805: creat d6/d13/d18/d19/d2c/d3f/d95/ff2 x:0 0 0 2024-04-05T21:26:45.224 INFO:tasks.workunit.client.0.smithi121.stdout:0/622: rmdir d7/d71/d94/dd1 39 2024-04-05T21:26:45.224 INFO:tasks.workunit.client.1.smithi188.stdout:9/823: mkdir dd/d11/d1f/d26/dc0/d102/d11a 0 2024-04-05T21:26:45.224 INFO:tasks.workunit.client.0.smithi121.stdout:9/438: readlink d6/df/l14 0 2024-04-05T21:26:45.225 INFO:tasks.workunit.client.1.smithi188.stdout:3/823: fsync d5/d7/d65/d5f/f64 0 2024-04-05T21:26:45.226 INFO:tasks.workunit.client.0.smithi121.stdout:0/623: stat d7/dc/l4a 0 2024-04-05T21:26:45.227 INFO:tasks.workunit.client.0.smithi121.stdout:4/694: creat d5/d11/d2c/d9d/dae/fdf x:0 0 0 2024-04-05T21:26:45.228 INFO:tasks.workunit.client.0.smithi121.stdout:3/449: dread d0/f27 [0,4194304] 0 2024-04-05T21:26:45.231 INFO:tasks.workunit.client.1.smithi188.stdout:9/824: symlink dd/d34/l11b 0 2024-04-05T21:26:45.233 INFO:tasks.workunit.client.0.smithi121.stdout:0/624: creat d7/fdd x:0 0 0 2024-04-05T21:26:45.235 INFO:tasks.workunit.client.0.smithi121.stdout:9/439: link d6/d36/d7b/f73 d6/d36/d7b/d5d/d79/f95 0 2024-04-05T21:26:45.236 INFO:tasks.workunit.client.1.smithi188.stdout:3/824: chown d5/d7/d6d/dd8/l3f 22 1 2024-04-05T21:26:45.237 INFO:tasks.workunit.client.0.smithi121.stdout:0/625: creat d7/d71/d29/d3f/fde x:0 0 0 2024-04-05T21:26:45.237 INFO:tasks.workunit.client.1.smithi188.stdout:5/691: write d6/f53 [1299008,7925] 0 2024-04-05T21:26:45.238 INFO:tasks.workunit.client.0.smithi121.stdout:0/626: write d7/d62/d9d/da6/fb9 [353531,89487] 0 2024-04-05T21:26:45.238 INFO:tasks.workunit.client.0.smithi121.stdout:9/440: mkdir d6/df/d48/d96 0 2024-04-05T21:26:45.238 INFO:tasks.workunit.client.1.smithi188.stdout:3/825: stat d5/d7/d65/d61 0 2024-04-05T21:26:45.239 INFO:tasks.workunit.client.1.smithi188.stdout:5/692: chown d6/cf 345 1 2024-04-05T21:26:45.240 INFO:tasks.workunit.client.1.smithi188.stdout:9/825: fdatasync dd/d11/d73/d8d/fa2 0 2024-04-05T21:26:45.241 INFO:tasks.workunit.client.0.smithi121.stdout:0/627: creat d7/d71/d94/da0/fdf x:0 0 0 2024-04-05T21:26:45.241 INFO:tasks.workunit.client.1.smithi188.stdout:3/826: symlink d5/d7/d6d/d75/d83/l11a 0 2024-04-05T21:26:45.242 INFO:tasks.workunit.client.0.smithi121.stdout:3/450: link d0/d3/d24/d3d/d1a/l32 d0/d3/l92 0 2024-04-05T21:26:45.243 INFO:tasks.workunit.client.1.smithi188.stdout:5/693: symlink d6/d82/d40/d42/lea 0 2024-04-05T21:26:45.245 INFO:tasks.workunit.client.1.smithi188.stdout:9/826: mknod dd/d11/d1f/d26/dc0/d102/c11c 0 2024-04-05T21:26:45.247 INFO:tasks.workunit.client.1.smithi188.stdout:3/827: unlink d5/d7/d6d/d75/d82/la5 0 2024-04-05T21:26:45.247 INFO:tasks.workunit.client.0.smithi121.stdout:4/695: mkdir d5/d11/d2c/d9d/dae/d29/db4/de0 0 2024-04-05T21:26:45.248 INFO:tasks.workunit.client.1.smithi188.stdout:9/827: stat dd/d11/d73/dbb/cbc 0 2024-04-05T21:26:45.248 INFO:tasks.workunit.client.1.smithi188.stdout:6/806: dwrite d6/d7/f15 [0,4194304] 0 2024-04-05T21:26:45.248 INFO:tasks.workunit.client.0.smithi121.stdout:3/451: dwrite d0/d3/d24/f3c [0,4194304] 0 2024-04-05T21:26:45.250 INFO:tasks.workunit.client.1.smithi188.stdout:5/694: symlink d6/d82/d40/d42/d81/leb 0 2024-04-05T21:26:45.251 INFO:tasks.workunit.client.0.smithi121.stdout:4/696: creat d5/d11/d26/dbc/dc5/dd5/fe1 x:0 0 0 2024-04-05T21:26:45.251 INFO:tasks.workunit.client.0.smithi121.stdout:3/452: symlink d0/d3/de/d83/l93 0 2024-04-05T21:26:45.252 INFO:tasks.workunit.client.1.smithi188.stdout:6/807: fsync d6/d13/d18/d19/d2c/f92 0 2024-04-05T21:26:45.253 INFO:tasks.workunit.client.0.smithi121.stdout:4/697: read d5/d11/d28/d3b/f59 [963874,35420] 0 2024-04-05T21:26:45.255 INFO:tasks.workunit.client.0.smithi121.stdout:3/453: creat d0/d3/d24/d3d/d1a/d23/d90/f94 x:0 0 0 2024-04-05T21:26:45.257 INFO:tasks.workunit.client.1.smithi188.stdout:3/828: truncate d5/d7/d31/f78 1368230 0 2024-04-05T21:26:45.257 INFO:tasks.workunit.client.0.smithi121.stdout:3/454: write d0/d3/de/f6a [1211082,114547] 0 2024-04-05T21:26:45.257 INFO:tasks.workunit.client.1.smithi188.stdout:3/829: fsync d5/df/d80/fee 0 2024-04-05T21:26:45.258 INFO:tasks.workunit.client.1.smithi188.stdout:6/808: creat d6/d13/d18/d19/da3/ff3 x:0 0 0 2024-04-05T21:26:45.259 INFO:tasks.workunit.client.0.smithi121.stdout:3/455: readlink d0/d3/d24/d3d/d1a/d23/l61 0 2024-04-05T21:26:45.260 INFO:tasks.workunit.client.0.smithi121.stdout:4/698: dwrite d5/d11/d2c/d9d/f75 [4194304,4194304] 0 2024-04-05T21:26:45.261 INFO:tasks.workunit.client.1.smithi188.stdout:3/830: read - d5/d7/d6d/dd8/fa6 zero size 2024-04-05T21:26:45.261 INFO:tasks.workunit.client.1.smithi188.stdout:5/695: getdents d6/d17/d23/d85 0 2024-04-05T21:26:45.262 INFO:tasks.workunit.client.0.smithi121.stdout:4/699: write d5/d11/d28/d3b/fdb [299748,71218] 0 2024-04-05T21:26:45.264 INFO:tasks.workunit.client.1.smithi188.stdout:6/809: dread d6/d13/d18/d19/d2c/d34/fac [0,4194304] 0 2024-04-05T21:26:45.265 INFO:tasks.workunit.client.1.smithi188.stdout:5/696: symlink d6/d15/d5c/d7a/lec 0 2024-04-05T21:26:45.266 INFO:tasks.workunit.client.1.smithi188.stdout:3/831: getdents d5/d7/d65/d61/d66/d6e/d104 0 2024-04-05T21:26:45.267 INFO:tasks.workunit.client.1.smithi188.stdout:6/810: fsync d6/d13/d18/d19/d2c/d3f/d77/fc0 0 2024-04-05T21:26:45.269 INFO:tasks.workunit.client.0.smithi121.stdout:3/456: dwrite d0/d3/d24/d3d/d1a/d23/d90/f94 [0,4194304] 0 2024-04-05T21:26:45.269 INFO:tasks.workunit.client.1.smithi188.stdout:5/697: creat d6/d82/d40/d42/fed x:0 0 0 2024-04-05T21:26:45.269 INFO:tasks.workunit.client.1.smithi188.stdout:5/698: dread - d6/d15/d5c/d7a/fa1 zero size 2024-04-05T21:26:45.270 INFO:tasks.workunit.client.0.smithi121.stdout:4/700: getdents d5/d11/d26/d49 0 2024-04-05T21:26:45.273 INFO:tasks.workunit.client.1.smithi188.stdout:6/811: creat d6/d71/ff4 x:0 0 0 2024-04-05T21:26:45.275 INFO:tasks.workunit.client.0.smithi121.stdout:3/457: unlink d0/d3/d24/d3d/d1a/d1e/c40 0 2024-04-05T21:26:45.276 INFO:tasks.workunit.client.1.smithi188.stdout:5/699: rename d6/d15/d5c/d7a/l91 to d6/d15/d5c/d7a/d44/d98/lee 0 2024-04-05T21:26:45.277 INFO:tasks.workunit.client.1.smithi188.stdout:3/832: dwrite d5/d7/d6d/d75/d82/d109/fc9 [0,4194304] 0 2024-04-05T21:26:45.278 INFO:tasks.workunit.client.0.smithi121.stdout:3/458: mknod d0/d3/d24/d3d/d1a/d23/d90/c95 0 2024-04-05T21:26:45.280 INFO:tasks.workunit.client.0.smithi121.stdout:4/701: dread d5/d11/d2c/d9d/dae/fa6 [4194304,4194304] 0 2024-04-05T21:26:45.286 INFO:tasks.workunit.client.0.smithi121.stdout:4/702: rename d5/l71 to d5/d11/d28/d73/le2 0 2024-04-05T21:26:45.288 INFO:tasks.workunit.client.0.smithi121.stdout:4/703: mkdir d5/d89/de3 0 2024-04-05T21:26:45.289 INFO:tasks.workunit.client.1.smithi188.stdout:9/828: dwrite dd/d11/d1f/d26/d55/f63 [4194304,4194304] 0 2024-04-05T21:26:45.291 INFO:tasks.workunit.client.1.smithi188.stdout:9/829: dread dd/d11/d22/d7e/de3/fef [0,4194304] 0 2024-04-05T21:26:45.294 INFO:tasks.workunit.client.1.smithi188.stdout:9/830: chown dd/d11/d1f/d26/dc0/d6b/f94 47724140 1 2024-04-05T21:26:45.294 INFO:tasks.workunit.client.1.smithi188.stdout:9/831: dread - dd/d3d/d95/f113 zero size 2024-04-05T21:26:45.296 INFO:tasks.workunit.client.1.smithi188.stdout:6/812: write d6/d13/d38/f6f [2216756,75014] 0 2024-04-05T21:26:45.296 INFO:tasks.workunit.client.1.smithi188.stdout:5/700: write d6/d17/d23/d7d/dcf/fd6 [882291,57956] 0 2024-04-05T21:26:45.297 INFO:tasks.workunit.client.1.smithi188.stdout:3/833: write d5/d7/d65/d61/d66/fc2 [430646,95427] 0 2024-04-05T21:26:45.302 INFO:tasks.workunit.client.0.smithi121.stdout:4/704: truncate d5/d11/d2c/d9d/dae/d29/d58/f78 361646 0 2024-04-05T21:26:45.303 INFO:tasks.workunit.client.0.smithi121.stdout:4/705: write d5/f8 [1434930,25207] 0 2024-04-05T21:26:45.303 INFO:tasks.workunit.client.0.smithi121.stdout:1/650: sync 2024-04-05T21:26:45.303 INFO:tasks.workunit.client.0.smithi121.stdout:2/470: sync 2024-04-05T21:26:45.303 INFO:tasks.workunit.client.0.smithi121.stdout:6/428: sync 2024-04-05T21:26:45.303 INFO:tasks.workunit.client.1.smithi188.stdout:5/701: fdatasync d6/d82/d40/d42/fed 0 2024-04-05T21:26:45.303 INFO:tasks.workunit.client.1.smithi188.stdout:6/813: dread - d6/d13/d18/d19/d2c/d3f/f53 zero size 2024-04-05T21:26:45.303 INFO:tasks.workunit.client.1.smithi188.stdout:9/832: creat dd/d11/d1f/d26/dc0/d102/f11d x:0 0 0 2024-04-05T21:26:45.303 INFO:tasks.workunit.client.1.smithi188.stdout:9/833: chown dd/d11/d1f/db2/ff5 9254 1 2024-04-05T21:26:45.304 INFO:tasks.workunit.client.1.smithi188.stdout:3/834: creat d5/d7/d65/da2/f11b x:0 0 0 2024-04-05T21:26:45.305 INFO:tasks.workunit.client.1.smithi188.stdout:5/702: getdents d6/d15/d5c/d7a/d44/d9b 0 2024-04-05T21:26:45.305 INFO:tasks.workunit.client.0.smithi121.stdout:2/471: mknod d2/d9/d1c/c88 0 2024-04-05T21:26:45.305 INFO:tasks.workunit.client.0.smithi121.stdout:1/651: write d8/d8c/db6/f1e [5184430,96610] 0 2024-04-05T21:26:45.306 INFO:tasks.workunit.client.1.smithi188.stdout:9/834: mkdir dd/d11/d1f/d26/dc0/d6b/d11e 0 2024-04-05T21:26:45.310 INFO:tasks.workunit.client.1.smithi188.stdout:8/671: sync 2024-04-05T21:26:45.310 INFO:tasks.workunit.client.1.smithi188.stdout:5/703: mkdir d6/d17/d39/d8e/def 0 2024-04-05T21:26:45.310 INFO:tasks.workunit.client.1.smithi188.stdout:6/814: dwrite d6/d13/d18/f36 [0,4194304] 0 2024-04-05T21:26:45.310 INFO:tasks.workunit.client.0.smithi121.stdout:6/429: rename d1/d2/d3/d4/de to d1/d2/d1b/d35/d39/d92 0 2024-04-05T21:26:45.312 INFO:tasks.workunit.client.1.smithi188.stdout:5/704: readlink d6/d17/d23/d7d/dcf/ld2 0 2024-04-05T21:26:45.313 INFO:tasks.workunit.client.0.smithi121.stdout:2/472: creat d2/d9/d1c/d30/d59/f89 x:0 0 0 2024-04-05T21:26:45.313 INFO:tasks.workunit.client.1.smithi188.stdout:6/815: chown d6/d13/d18/d19/d2c/d34/f6b 31823 1 2024-04-05T21:26:45.313 INFO:tasks.workunit.client.1.smithi188.stdout:5/705: readlink d6/d15/d1b/ddc/le8 0 2024-04-05T21:26:45.314 INFO:tasks.workunit.client.0.smithi121.stdout:1/652: unlink d8/d89/d9e/da3/fc5 0 2024-04-05T21:26:45.316 INFO:tasks.workunit.client.1.smithi188.stdout:9/835: link dd/d35/cbd dd/d3d/c11f 0 2024-04-05T21:26:45.318 INFO:tasks.workunit.client.1.smithi188.stdout:3/835: dread d5/d7/d65/d61/db5/fbf [0,4194304] 0 2024-04-05T21:26:45.318 INFO:tasks.workunit.client.1.smithi188.stdout:9/836: chown dd/d11/d1f/d26/dc0/d6b/d40 336089796 1 2024-04-05T21:26:45.319 INFO:tasks.workunit.client.1.smithi188.stdout:8/672: rmdir d5/d21/d80/ddc 0 2024-04-05T21:26:45.319 INFO:tasks.workunit.client.1.smithi188.stdout:5/706: symlink d6/lf0 0 2024-04-05T21:26:45.320 INFO:tasks.workunit.client.0.smithi121.stdout:6/430: creat d1/d2/d3/d4/d21/f93 x:0 0 0 2024-04-05T21:26:45.321 INFO:tasks.workunit.client.1.smithi188.stdout:3/836: read d5/d7/d65/f44 [3329433,128313] 0 2024-04-05T21:26:45.321 INFO:tasks.workunit.client.0.smithi121.stdout:2/473: symlink d2/d9/d1c/d85/l8a 0 2024-04-05T21:26:45.323 INFO:tasks.workunit.client.1.smithi188.stdout:5/707: truncate d6/f22 1944328 0 2024-04-05T21:26:45.323 INFO:tasks.workunit.client.0.smithi121.stdout:6/431: write d1/d2/dc/f4c [5594235,37589] 0 2024-04-05T21:26:45.328 INFO:tasks.workunit.client.1.smithi188.stdout:8/673: creat d5/d21/d40/d43/d44/d59/d8b/da4/db1/fdd x:0 0 0 2024-04-05T21:26:45.328 INFO:tasks.workunit.client.1.smithi188.stdout:5/708: dread d6/f73 [0,4194304] 0 2024-04-05T21:26:45.330 INFO:tasks.workunit.client.0.smithi121.stdout:6/432: write d1/d2/d3/d4/d21/d57/f87 [2914531,93070] 0 2024-04-05T21:26:45.332 INFO:tasks.workunit.client.1.smithi188.stdout:8/674: dread d5/d21/d40/d43/d44/d5b/d99/da3/fa5 [0,4194304] 0 2024-04-05T21:26:45.333 INFO:tasks.workunit.client.0.smithi121.stdout:4/706: dwrite d5/d89/fa7 [0,4194304] 0 2024-04-05T21:26:45.335 INFO:tasks.workunit.client.1.smithi188.stdout:9/837: write dd/d35/f67 [1191487,107160] 0 2024-04-05T21:26:45.336 INFO:tasks.workunit.client.0.smithi121.stdout:6/433: rename d1/d4b/c8d to d1/d2/d1b/d1e/c94 0 2024-04-05T21:26:45.336 INFO:tasks.workunit.client.0.smithi121.stdout:2/474: dwrite d2/d9/d38/d2d/f72 [0,4194304] 0 2024-04-05T21:26:45.339 INFO:tasks.workunit.client.1.smithi188.stdout:9/838: unlink dd/d35/cf2 0 2024-04-05T21:26:45.339 INFO:tasks.workunit.client.0.smithi121.stdout:4/707: chown d5/d11/d2c/d9d/dae/d29/fda 2915 1 2024-04-05T21:26:45.340 INFO:tasks.workunit.client.1.smithi188.stdout:5/709: dwrite d6/d17/d39/f4a [4194304,4194304] 0 2024-04-05T21:26:45.343 INFO:tasks.workunit.client.0.smithi121.stdout:6/434: read d1/d2/d3/d4/d21/d57/d42/f67 [451993,112688] 0 2024-04-05T21:26:45.343 INFO:tasks.workunit.client.0.smithi121.stdout:2/475: creat d2/d9/d38/d2d/d4c/d7c/f8b x:0 0 0 2024-04-05T21:26:45.343 INFO:tasks.workunit.client.1.smithi188.stdout:8/675: creat d5/d21/d40/d43/d48/fde x:0 0 0 2024-04-05T21:26:45.344 INFO:tasks.workunit.client.1.smithi188.stdout:5/710: mkdir d6/d15/d5c/d7a/d44/d51/df1 0 2024-04-05T21:26:45.345 INFO:tasks.workunit.client.0.smithi121.stdout:4/708: mkdir d5/d11/d28/dc0/de4 0 2024-04-05T21:26:45.346 INFO:tasks.workunit.client.0.smithi121.stdout:4/709: chown d5/d11/f48 62500 1 2024-04-05T21:26:45.346 INFO:tasks.workunit.client.0.smithi121.stdout:6/435: mkdir d1/d2/d1b/d1e/d95 0 2024-04-05T21:26:45.347 INFO:tasks.workunit.client.1.smithi188.stdout:3/837: dwrite d5/d7/d65/d61/fca [0,4194304] 0 2024-04-05T21:26:45.347 INFO:tasks.workunit.client.1.smithi188.stdout:9/839: creat dd/d11/d1f/d26/d55/d98/f120 x:0 0 0 2024-04-05T21:26:45.349 INFO:tasks.workunit.client.1.smithi188.stdout:5/711: symlink d6/d15/d5c/d7a/d44/d9b/lf2 0 2024-04-05T21:26:45.351 INFO:tasks.workunit.client.0.smithi121.stdout:6/436: write d1/d2/d3/d5/f66 [967506,120206] 0 2024-04-05T21:26:45.351 INFO:tasks.workunit.client.1.smithi188.stdout:3/838: chown d5/d7/d6d/d75/d82/d109/fd3 50001 1 2024-04-05T21:26:45.353 INFO:tasks.workunit.client.0.smithi121.stdout:4/710: rename d5/d11/d2c/d9d/dae/d29/la8 to d5/d89/de3/le5 0 2024-04-05T21:26:45.354 INFO:tasks.workunit.client.0.smithi121.stdout:4/711: truncate d5/d11/f48 5575285 0 2024-04-05T21:26:45.354 INFO:tasks.workunit.client.1.smithi188.stdout:3/839: truncate d5/d7/d65/d61/fca 5111858 0 2024-04-05T21:26:45.356 INFO:tasks.workunit.client.0.smithi121.stdout:6/437: getdents d1/d2/d3/d4 0 2024-04-05T21:26:45.356 INFO:tasks.workunit.client.1.smithi188.stdout:9/840: link dd/d3d/d89/dd1/f114 dd/d11/d1f/f121 0 2024-04-05T21:26:45.358 INFO:tasks.workunit.client.0.smithi121.stdout:4/712: creat d5/d11/d2c/d9d/dae/d29/db4/fe6 x:0 0 0 2024-04-05T21:26:45.360 INFO:tasks.workunit.client.0.smithi121.stdout:4/713: symlink d5/d11/d2c/d8c/le7 0 2024-04-05T21:26:45.365 INFO:tasks.workunit.client.1.smithi188.stdout:9/841: dwrite dd/d11/d1f/d25/d8a/d101/ffc [0,4194304] 0 2024-04-05T21:26:45.375 INFO:tasks.workunit.client.1.smithi188.stdout:3/840: dread d5/d7e/d101/fe7 [0,4194304] 0 2024-04-05T21:26:45.375 INFO:tasks.workunit.client.0.smithi121.stdout:6/438: stat d1/d2/d1b/d1e/c94 0 2024-04-05T21:26:45.376 INFO:tasks.workunit.client.1.smithi188.stdout:5/712: write d6/d15/d1b/f6c [196112,53756] 0 2024-04-05T21:26:45.378 INFO:tasks.workunit.client.0.smithi121.stdout:6/439: symlink d1/d2/d1b/d1e/l96 0 2024-04-05T21:26:45.378 INFO:tasks.workunit.client.1.smithi188.stdout:3/841: unlink d5/d7/d65/d61/f7b 0 2024-04-05T21:26:45.380 INFO:tasks.workunit.client.1.smithi188.stdout:5/713: creat d6/d82/d40/dc1/ff3 x:0 0 0 2024-04-05T21:26:45.380 INFO:tasks.workunit.client.0.smithi121.stdout:2/476: dread d2/d9/d17/f18 [0,4194304] 0 2024-04-05T21:26:45.381 INFO:tasks.workunit.client.1.smithi188.stdout:5/714: stat d6/dad/l58 0 2024-04-05T21:26:45.385 INFO:tasks.workunit.client.0.smithi121.stdout:2/477: truncate d2/d9/d56/d62/f7a 318266 0 2024-04-05T21:26:45.387 INFO:tasks.workunit.client.1.smithi188.stdout:9/842: dwrite dd/d3d/f70 [0,4194304] 0 2024-04-05T21:26:45.387 INFO:tasks.workunit.client.0.smithi121.stdout:2/478: creat d2/d1e/f8c x:0 0 0 2024-04-05T21:26:45.389 INFO:tasks.workunit.client.1.smithi188.stdout:3/842: dread d5/d7/d65/d61/d66/fc2 [0,4194304] 0 2024-04-05T21:26:45.390 INFO:tasks.workunit.client.0.smithi121.stdout:2/479: mkdir d2/d9/d56/d8d 0 2024-04-05T21:26:45.392 INFO:tasks.workunit.client.0.smithi121.stdout:4/714: dread d5/fd [0,4194304] 0 2024-04-05T21:26:45.392 INFO:tasks.workunit.client.1.smithi188.stdout:9/843: symlink dd/d11/d22/d7e/l122 0 2024-04-05T21:26:45.393 INFO:tasks.workunit.client.0.smithi121.stdout:6/440: getdents d1/d2/d1b/d1e 0 2024-04-05T21:26:45.393 INFO:tasks.workunit.client.0.smithi121.stdout:2/480: mknod d2/d9/d56/d62/c8e 0 2024-04-05T21:26:45.395 INFO:tasks.workunit.client.1.smithi188.stdout:3/843: creat d5/d7/d65/d61/d66/da4/de3/df6/f11c x:0 0 0 2024-04-05T21:26:45.395 INFO:tasks.workunit.client.1.smithi188.stdout:5/715: write d6/d15/d1b/f8a [516787,22291] 0 2024-04-05T21:26:45.396 INFO:tasks.workunit.client.1.smithi188.stdout:3/844: stat d5/d7e/cd2 0 2024-04-05T21:26:45.397 INFO:tasks.workunit.client.0.smithi121.stdout:2/481: symlink d2/d9/d38/d2d/d4c/d7c/d7e/l8f 0 2024-04-05T21:26:45.398 INFO:tasks.workunit.client.1.smithi188.stdout:9/844: rmdir dd/d11/d1f/d26/dc0/d102 39 2024-04-05T21:26:45.400 INFO:tasks.workunit.client.0.smithi121.stdout:2/482: unlink d2/d9/d38/d2d/d4c/d7c/d7e/l8f 0 2024-04-05T21:26:45.400 INFO:tasks.workunit.client.0.smithi121.stdout:6/441: link d1/d2/c7 d1/d2/d3/d4/d21/d57/d86/c97 0 2024-04-05T21:26:45.401 INFO:tasks.workunit.client.1.smithi188.stdout:5/716: rename d6/dad/l58 to d6/d15/d5c/d7a/d44/d98/lf4 0 2024-04-05T21:26:45.402 INFO:tasks.workunit.client.1.smithi188.stdout:5/717: write d6/d15/d5c/d7a/fd3 [432180,109139] 0 2024-04-05T21:26:45.403 INFO:tasks.workunit.client.0.smithi121.stdout:6/442: mknod d1/d2/d3/d59/c98 0 2024-04-05T21:26:45.403 INFO:tasks.workunit.client.1.smithi188.stdout:3/845: write d5/f37 [559046,18272] 0 2024-04-05T21:26:45.404 INFO:tasks.workunit.client.0.smithi121.stdout:4/715: write d5/d11/f12 [8429570,54066] 0 2024-04-05T21:26:45.405 INFO:tasks.workunit.client.1.smithi188.stdout:9/845: unlink dd/f72 0 2024-04-05T21:26:45.406 INFO:tasks.workunit.client.1.smithi188.stdout:3/846: rmdir d5/d7/d6d/d75/d82/d109 39 2024-04-05T21:26:45.407 INFO:tasks.workunit.client.1.smithi188.stdout:9/846: write dd/d11/d1f/d26/dc0/d6b/f94 [776593,10260] 0 2024-04-05T21:26:45.409 INFO:tasks.workunit.client.1.smithi188.stdout:3/847: mkdir d5/d7/d65/d5f/d11d 0 2024-04-05T21:26:45.409 INFO:tasks.workunit.client.0.smithi121.stdout:6/443: symlink d1/d2/d3/d4/d21/d57/d42/d48/d7c/l99 0 2024-04-05T21:26:45.410 INFO:tasks.workunit.client.0.smithi121.stdout:2/483: dwrite d2/d9/d38/d2d/d47/f70 [0,4194304] 0 2024-04-05T21:26:45.414 INFO:tasks.workunit.client.0.smithi121.stdout:6/444: truncate d1/d2/d3/d5/f58 673309 0 2024-04-05T21:26:45.415 INFO:tasks.workunit.client.0.smithi121.stdout:4/716: link d5/d11/l1f d5/d11/d2c/da4/le8 0 2024-04-05T21:26:45.418 INFO:tasks.workunit.client.0.smithi121.stdout:2/484: dwrite d2/d1e/f52 [0,4194304] 0 2024-04-05T21:26:45.419 INFO:tasks.workunit.client.0.smithi121.stdout:4/717: write d5/d11/d2c/d9d/f75 [2589797,45704] 0 2024-04-05T21:26:45.419 INFO:tasks.workunit.client.0.smithi121.stdout:6/445: write d1/d2/d1b/d35/d74/f7d [26524,6691] 0 2024-04-05T21:26:45.419 INFO:tasks.workunit.client.1.smithi188.stdout:5/718: write d6/d15/d5c/d7a/d44/fda [1023186,51458] 0 2024-04-05T21:26:45.419 INFO:tasks.workunit.client.1.smithi188.stdout:5/719: mkdir d6/d82/d40/dc1/df5 0 2024-04-05T21:26:45.424 INFO:tasks.workunit.client.1.smithi188.stdout:3/848: dwrite d5/f9b [0,4194304] 0 2024-04-05T21:26:45.425 INFO:tasks.workunit.client.0.smithi121.stdout:6/446: truncate d1/d2/d3/f4e 1763165 0 2024-04-05T21:26:45.426 INFO:tasks.workunit.client.0.smithi121.stdout:4/718: mkdir d5/d11/d26/dbc/dca/de9 0 2024-04-05T21:26:45.426 INFO:tasks.workunit.client.0.smithi121.stdout:6/447: mkdir d1/d2/dc/d9a 0 2024-04-05T21:26:45.428 INFO:tasks.workunit.client.0.smithi121.stdout:5/691: sync 2024-04-05T21:26:45.428 INFO:tasks.workunit.client.0.smithi121.stdout:8/508: sync 2024-04-05T21:26:45.428 INFO:tasks.workunit.client.0.smithi121.stdout:7/616: sync 2024-04-05T21:26:45.430 INFO:tasks.workunit.client.0.smithi121.stdout:6/448: fsync d1/d2/d3/d5/d1f/f46 0 2024-04-05T21:26:45.434 INFO:tasks.workunit.client.1.smithi188.stdout:5/720: dwrite d6/d82/d40/dc1/ff3 [0,4194304] 0 2024-04-05T21:26:45.438 INFO:tasks.workunit.client.0.smithi121.stdout:6/449: link d1/l34 d1/d2/d1b/l9b 0 2024-04-05T21:26:45.439 INFO:tasks.workunit.client.1.smithi188.stdout:5/721: symlink d6/d15/lf6 0 2024-04-05T21:26:45.440 INFO:tasks.workunit.client.1.smithi188.stdout:9/847: dread dd/d11/d1f/d25/d8a/f39 [0,4194304] 0 2024-04-05T21:26:45.441 INFO:tasks.workunit.client.1.smithi188.stdout:9/848: stat dd/d3d/d89/dd1 0 2024-04-05T21:26:45.444 INFO:tasks.workunit.client.0.smithi121.stdout:5/692: dwrite dd/de/d39/d3b/d50/f5b [0,4194304] 0 2024-04-05T21:26:45.445 INFO:tasks.workunit.client.0.smithi121.stdout:8/509: dwrite d12/d13/d15/f83 [0,4194304] 0 2024-04-05T21:26:45.447 INFO:tasks.workunit.client.0.smithi121.stdout:8/510: chown d12/d13/d56/d57 921 1 2024-04-05T21:26:45.454 INFO:tasks.workunit.client.0.smithi121.stdout:8/511: rename d12/d13/d15/d1d/f58 to d12/d13/d15/d1d/fa3 0 2024-04-05T21:26:45.460 INFO:tasks.workunit.client.0.smithi121.stdout:5/693: dwrite dd/de/d19/d38/f97 [0,4194304] 0 2024-04-05T21:26:45.460 INFO:tasks.workunit.client.1.smithi188.stdout:3/849: truncate d5/d7/d6d/d75/fb8 3138218 0 2024-04-05T21:26:45.461 INFO:tasks.workunit.client.0.smithi121.stdout:8/512: rename d12/d13/d15/d1d/d29 to d12/d13/d56/d57/da4 0 2024-04-05T21:26:45.465 INFO:tasks.workunit.client.0.smithi121.stdout:8/513: chown d12/d13/d15/d1d/f42 9597 1 2024-04-05T21:26:45.466 INFO:tasks.workunit.client.1.smithi188.stdout:5/722: dread d6/d15/d5c/d7a/d44/d51/dc6/faa [0,4194304] 0 2024-04-05T21:26:45.467 INFO:tasks.workunit.client.1.smithi188.stdout:3/850: link d5/df/lfc d5/d7/d65/d61/d66/da4/d114/l11e 0 2024-04-05T21:26:45.469 INFO:tasks.workunit.client.0.smithi121.stdout:5/694: getdents dd/de/d19/d1c/d36 0 2024-04-05T21:26:45.470 INFO:tasks.workunit.client.0.smithi121.stdout:8/514: read d12/d13/d56/d57/f78 [47212,61575] 0 2024-04-05T21:26:45.471 INFO:tasks.workunit.client.0.smithi121.stdout:4/719: dread d5/f3f [0,4194304] 0 2024-04-05T21:26:45.471 INFO:tasks.workunit.client.0.smithi121.stdout:7/617: dwrite d0/d32/f6b [0,4194304] 0 2024-04-05T21:26:45.471 INFO:tasks.workunit.client.1.smithi188.stdout:3/851: mknod d5/d48/c11f 0 2024-04-05T21:26:45.472 INFO:tasks.workunit.client.0.smithi121.stdout:8/515: chown d12/f51 2 1 2024-04-05T21:26:45.472 INFO:tasks.workunit.client.1.smithi188.stdout:5/723: fsync d6/d82/d40/d42/f5e 0 2024-04-05T21:26:45.477 INFO:tasks.workunit.client.0.smithi121.stdout:5/695: creat dd/de/d19/d1c/d36/ffc x:0 0 0 2024-04-05T21:26:45.477 INFO:tasks.workunit.client.0.smithi121.stdout:7/618: mknod d0/d23/d5b/cc1 0 2024-04-05T21:26:45.477 INFO:tasks.workunit.client.0.smithi121.stdout:4/720: creat d5/d11/d26/d47/fea x:0 0 0 2024-04-05T21:26:45.481 INFO:tasks.workunit.client.1.smithi188.stdout:5/724: dwrite d6/f6f [0,4194304] 0 2024-04-05T21:26:45.482 INFO:tasks.workunit.client.0.smithi121.stdout:8/516: link d12/l52 d12/d50/d93/d96/la5 0 2024-04-05T21:26:45.482 INFO:tasks.workunit.client.0.smithi121.stdout:6/450: dread d1/d2/dc/f4c [0,4194304] 0 2024-04-05T21:26:45.484 INFO:tasks.workunit.client.1.smithi188.stdout:5/725: mknod d6/d15/d1b/ddc/cf7 0 2024-04-05T21:26:45.485 INFO:tasks.workunit.client.0.smithi121.stdout:5/696: link dd/de/d19/d38/la0 dd/de/d39/lfd 0 2024-04-05T21:26:45.487 INFO:tasks.workunit.client.0.smithi121.stdout:8/517: truncate ff 1459052 0 2024-04-05T21:26:45.489 INFO:tasks.workunit.client.1.smithi188.stdout:5/726: creat d6/d15/ff8 x:0 0 0 2024-04-05T21:26:45.490 INFO:tasks.workunit.client.1.smithi188.stdout:9/849: dread dd/f42 [0,4194304] 0 2024-04-05T21:26:45.492 INFO:tasks.workunit.client.0.smithi121.stdout:7/619: rename d0/c4 to d0/d9/d22/d81/cc2 0 2024-04-05T21:26:45.493 INFO:tasks.workunit.client.0.smithi121.stdout:5/697: symlink dd/de/d19/d3c/lfe 0 2024-04-05T21:26:45.496 INFO:tasks.workunit.client.0.smithi121.stdout:6/451: dread d1/d2/d3/d4/d21/d57/d42/f67 [0,4194304] 0 2024-04-05T21:26:45.496 INFO:tasks.workunit.client.0.smithi121.stdout:5/698: fdatasync dd/de/d39/d3b/fa1 0 2024-04-05T21:26:45.497 INFO:tasks.workunit.client.0.smithi121.stdout:7/620: mkdir d0/d57/dc3 0 2024-04-05T21:26:45.498 INFO:tasks.workunit.client.1.smithi188.stdout:9/850: dread dd/d3d/d81/f107 [0,4194304] 0 2024-04-05T21:26:45.499 INFO:tasks.workunit.client.0.smithi121.stdout:8/518: dwrite d12/d50/d61/d7f/f97 [0,4194304] 0 2024-04-05T21:26:45.499 INFO:tasks.workunit.client.1.smithi188.stdout:9/851: chown dd/d11/d73/la3 488 1 2024-04-05T21:26:45.500 INFO:tasks.workunit.client.0.smithi121.stdout:7/621: write d0/f3 [521987,111811] 0 2024-04-05T21:26:45.500 INFO:tasks.workunit.client.1.smithi188.stdout:5/727: dwrite d6/d15/d5c/d7a/d44/d51/dc6/faa [0,4194304] 0 2024-04-05T21:26:45.501 INFO:tasks.workunit.client.0.smithi121.stdout:8/519: readlink d12/d50/l77 0 2024-04-05T21:26:45.502 INFO:tasks.workunit.client.0.smithi121.stdout:8/520: write d12/d13/d15/f18 [3126597,44304] 0 2024-04-05T21:26:45.505 INFO:tasks.workunit.client.0.smithi121.stdout:5/699: mknod dd/de/d19/d38/de3/cff 0 2024-04-05T21:26:45.506 INFO:tasks.workunit.client.0.smithi121.stdout:4/721: write d5/d11/d20/f38 [2817609,33906] 0 2024-04-05T21:26:45.506 INFO:tasks.workunit.client.0.smithi121.stdout:6/452: rmdir d1/d2/d3/d4/d21/d57/d86/d4f 39 2024-04-05T21:26:45.506 INFO:tasks.workunit.client.0.smithi121.stdout:4/722: chown d5/d11/d26/d47 166019040 1 2024-04-05T21:26:45.507 INFO:tasks.workunit.client.0.smithi121.stdout:5/700: write dd/d41/dce/da6/dd3/db1/db9/fd0 [315463,83999] 0 2024-04-05T21:26:45.507 INFO:tasks.workunit.client.1.smithi188.stdout:5/728: mknod d6/d17/d23/d85/cf9 0 2024-04-05T21:26:45.507 INFO:tasks.workunit.client.1.smithi188.stdout:9/852: fsync dd/d11/d22/d7e/f91 0 2024-04-05T21:26:45.508 INFO:tasks.workunit.client.1.smithi188.stdout:5/729: chown d6/cf 1133 1 2024-04-05T21:26:45.508 INFO:tasks.workunit.client.1.smithi188.stdout:3/852: dread d5/f6b [0,4194304] 0 2024-04-05T21:26:45.508 INFO:tasks.workunit.client.1.smithi188.stdout:9/853: mknod dd/d11/d1f/d25/d8a/c123 0 2024-04-05T21:26:45.508 INFO:tasks.workunit.client.0.smithi121.stdout:4/723: unlink d5/d11/d28/d3b/f59 0 2024-04-05T21:26:45.509 INFO:tasks.workunit.client.1.smithi188.stdout:5/730: fdatasync d6/d15/d5c/d7a/f93 0 2024-04-05T21:26:45.511 INFO:tasks.workunit.client.1.smithi188.stdout:2/818: sync 2024-04-05T21:26:45.511 INFO:tasks.workunit.client.1.smithi188.stdout:9/854: mknod dd/d11/d1f/d25/c124 0 2024-04-05T21:26:45.512 INFO:tasks.workunit.client.0.smithi121.stdout:8/521: truncate d12/d50/d93/d96/f72 345982 0 2024-04-05T21:26:45.513 INFO:tasks.workunit.client.1.smithi188.stdout:0/699: sync 2024-04-05T21:26:45.513 INFO:tasks.workunit.client.1.smithi188.stdout:1/767: sync 2024-04-05T21:26:45.514 INFO:tasks.workunit.client.1.smithi188.stdout:3/853: dread d5/f21 [0,4194304] 0 2024-04-05T21:26:45.516 INFO:tasks.workunit.client.1.smithi188.stdout:1/768: read d1/dc/f28 [7081962,16128] 0 2024-04-05T21:26:45.517 INFO:tasks.workunit.client.1.smithi188.stdout:2/819: mknod d5/db/d35/d3b/d6b/c110 0 2024-04-05T21:26:45.517 INFO:tasks.workunit.client.0.smithi121.stdout:5/701: dwrite dd/de/d19/d1c/d36/d46/f9a [0,4194304] 0 2024-04-05T21:26:45.518 INFO:tasks.workunit.client.0.smithi121.stdout:8/522: creat d12/d13/d56/d57/d5e/fa6 x:0 0 0 2024-04-05T21:26:45.521 INFO:tasks.workunit.client.0.smithi121.stdout:7/622: getdents d0/d23/d5b 0 2024-04-05T21:26:45.522 INFO:tasks.workunit.client.0.smithi121.stdout:5/702: write dd/de/d19/d38/d45/d68/fa2 [484517,116303] 0 2024-04-05T21:26:45.522 INFO:tasks.workunit.client.0.smithi121.stdout:8/523: fdatasync d12/d50/d61/f3a 0 2024-04-05T21:26:45.524 INFO:tasks.workunit.client.0.smithi121.stdout:7/623: fsync d0/d9/d22/d28/f48 0 2024-04-05T21:26:45.525 INFO:tasks.workunit.client.0.smithi121.stdout:5/703: chown dd/de/lb4 7137 1 2024-04-05T21:26:45.525 INFO:tasks.workunit.client.1.smithi188.stdout:3/854: rename d5/d7/d31/f91 to d5/d7/d65/d5f/f120 0 2024-04-05T21:26:45.526 INFO:tasks.workunit.client.1.smithi188.stdout:1/769: chown d1/dc/d16/f3b 911 1 2024-04-05T21:26:45.528 INFO:tasks.workunit.client.1.smithi188.stdout:9/855: dwrite dd/d11/d1f/d26/ff3 [0,4194304] 0 2024-04-05T21:26:45.528 INFO:tasks.workunit.client.0.smithi121.stdout:5/704: mkdir dd/de/d19/d38/de3/d100 0 2024-04-05T21:26:45.531 INFO:tasks.workunit.client.0.smithi121.stdout:7/624: truncate d0/d9/f21 1006030 0 2024-04-05T21:26:45.535 INFO:tasks.workunit.client.0.smithi121.stdout:6/453: truncate d1/d2/d3/d4/d21/d57/f87 285919 0 2024-04-05T21:26:45.536 INFO:tasks.workunit.client.1.smithi188.stdout:0/700: link d4/d8/db/d21/f90 d4/d8/db/d42/d71/f104 0 2024-04-05T21:26:45.537 INFO:tasks.workunit.client.1.smithi188.stdout:1/770: dread d1/d7/d4e/f8b [0,4194304] 0 2024-04-05T21:26:45.540 INFO:tasks.workunit.client.1.smithi188.stdout:5/731: truncate d6/d17/d39/f4a 3613186 0 2024-04-05T21:26:45.540 INFO:tasks.workunit.client.1.smithi188.stdout:2/820: dwrite d5/db/dd/d73/d109/f7b [0,4194304] 0 2024-04-05T21:26:45.540 INFO:tasks.workunit.client.0.smithi121.stdout:8/524: write d12/d13/d15/d1d/fa3 [1616701,105118] 0 2024-04-05T21:26:45.542 INFO:tasks.workunit.client.0.smithi121.stdout:4/724: dwrite d5/d11/d28/d3b/f77 [4194304,4194304] 0 2024-04-05T21:26:45.544 INFO:tasks.workunit.client.0.smithi121.stdout:7/625: rmdir d0/d57/dc3 0 2024-04-05T21:26:45.546 INFO:tasks.workunit.client.1.smithi188.stdout:1/771: write d1/dc/d16/d21/fa0 [2303501,39469] 0 2024-04-05T21:26:45.549 INFO:tasks.workunit.client.0.smithi121.stdout:6/454: dread d1/d2/d3/d4/d21/f82 [0,4194304] 0 2024-04-05T21:26:45.550 INFO:tasks.workunit.client.1.smithi188.stdout:5/732: unlink d6/d17/l1d 0 2024-04-05T21:26:45.551 INFO:tasks.workunit.client.1.smithi188.stdout:2/821: symlink d5/db/dd/d36/d3c/d48/d85/l111 0 2024-04-05T21:26:45.552 INFO:tasks.workunit.client.0.smithi121.stdout:8/525: mkdir d12/d13/d15/d1d/da7 0 2024-04-05T21:26:45.554 INFO:tasks.workunit.client.1.smithi188.stdout:9/856: dwrite dd/d3d/d95/fe0 [0,4194304] 0 2024-04-05T21:26:45.555 INFO:tasks.workunit.client.0.smithi121.stdout:4/725: mknod d5/d11/d26/d49/ceb 0 2024-04-05T21:26:45.556 INFO:tasks.workunit.client.1.smithi188.stdout:1/772: fdatasync d1/dc/d16/d21/d27/d4a/f8c 0 2024-04-05T21:26:45.556 INFO:tasks.workunit.client.0.smithi121.stdout:5/705: link dd/de/d19/d13/lf1 dd/de/d39/d3b/daf/l101 0 2024-04-05T21:26:45.556 INFO:tasks.workunit.client.0.smithi121.stdout:6/455: truncate d1/f1c 6175177 0 2024-04-05T21:26:45.557 INFO:tasks.workunit.client.0.smithi121.stdout:5/706: dread - dd/de/fc3 zero size 2024-04-05T21:26:45.557 INFO:tasks.workunit.client.1.smithi188.stdout:2/822: truncate d5/db/dd/d36/d3c/d63/d75/d87/f107 479224 0 2024-04-05T21:26:45.561 INFO:tasks.workunit.client.1.smithi188.stdout:1/773: mknod d1/dc/d16/d21/ced 0 2024-04-05T21:26:45.562 INFO:tasks.workunit.client.0.smithi121.stdout:4/726: symlink d5/d11/d26/d47/lec 0 2024-04-05T21:26:45.565 INFO:tasks.workunit.client.0.smithi121.stdout:7/626: link d0/d9/d22/d28/d3b/ca8 d0/d32/d4e/d73/dbc/cc4 0 2024-04-05T21:26:45.565 INFO:tasks.workunit.client.1.smithi188.stdout:5/733: link d6/d82/d40/d42/d81/da6/caf d6/d15/d1b/ddc/cfa 0 2024-04-05T21:26:45.566 INFO:tasks.workunit.client.0.smithi121.stdout:7/627: write d0/d9/f69 [1258608,130792] 0 2024-04-05T21:26:45.567 INFO:tasks.workunit.client.0.smithi121.stdout:6/456: symlink d1/d2/d1b/d35/d74/l9c 0 2024-04-05T21:26:45.570 INFO:tasks.workunit.client.1.smithi188.stdout:1/774: mknod d1/dc/d16/d2d/de6/cee 0 2024-04-05T21:26:45.570 INFO:tasks.workunit.client.1.smithi188.stdout:3/855: dwrite d5/d7/d31/fb1 [0,4194304] 0 2024-04-05T21:26:45.571 INFO:tasks.workunit.client.0.smithi121.stdout:7/628: rmdir d0/d9/d22/d28 39 2024-04-05T21:26:45.572 INFO:tasks.workunit.client.0.smithi121.stdout:6/457: dread - d1/d2/d1b/d35/f47 zero size 2024-04-05T21:26:45.573 INFO:tasks.workunit.client.1.smithi188.stdout:1/775: stat d1/d7/fd3 0 2024-04-05T21:26:45.574 INFO:tasks.workunit.client.0.smithi121.stdout:6/458: chown d1/d2/d3/d4/d21/d57/d42/f83 4864 1 2024-04-05T21:26:45.575 INFO:tasks.workunit.client.0.smithi121.stdout:5/707: creat dd/f102 x:0 0 0 2024-04-05T21:26:45.576 INFO:tasks.workunit.client.0.smithi121.stdout:8/526: link d12/c17 d12/d50/d93/d96/ca8 0 2024-04-05T21:26:45.577 INFO:tasks.workunit.client.0.smithi121.stdout:8/527: write d12/d13/d15/f18 [655624,7445] 0 2024-04-05T21:26:45.578 INFO:tasks.workunit.client.0.smithi121.stdout:7/629: creat d0/d23/d5b/fc5 x:0 0 0 2024-04-05T21:26:45.579 INFO:tasks.workunit.client.1.smithi188.stdout:2/823: dwrite d5/db/dd/d73/d109/f7b [4194304,4194304] 0 2024-04-05T21:26:45.579 INFO:tasks.workunit.client.1.smithi188.stdout:5/734: truncate d6/d15/d1b/f8a 1489052 0 2024-04-05T21:26:45.579 INFO:tasks.workunit.client.1.smithi188.stdout:1/776: stat d1/dc/d16/d21/d27/d49/f59 0 2024-04-05T21:26:45.579 INFO:tasks.workunit.client.0.smithi121.stdout:4/727: creat d5/d11/d2c/d9d/dae/d29/db4/dd4/fed x:0 0 0 2024-04-05T21:26:45.580 INFO:tasks.workunit.client.1.smithi188.stdout:2/824: chown d5/db/dd/d24/cac 76100122 1 2024-04-05T21:26:45.581 INFO:tasks.workunit.client.1.smithi188.stdout:5/735: mkdir d6/d17/d23/d7d/dfb 0 2024-04-05T21:26:45.583 INFO:tasks.workunit.client.0.smithi121.stdout:8/528: fdatasync d12/d13/d15/d1d/f4f 0 2024-04-05T21:26:45.587 INFO:tasks.workunit.client.0.smithi121.stdout:7/630: dwrite d0/d32/fa0 [0,4194304] 0 2024-04-05T21:26:45.589 INFO:tasks.workunit.client.0.smithi121.stdout:5/708: mkdir dd/de/d39/d3b/de8/d103 0 2024-04-05T21:26:45.590 INFO:tasks.workunit.client.0.smithi121.stdout:8/529: creat d12/d13/d15/d1d/fa9 x:0 0 0 2024-04-05T21:26:45.592 INFO:tasks.workunit.client.1.smithi188.stdout:2/825: dwrite d5/db/dd/d73/fca [0,4194304] 0 2024-04-05T21:26:45.593 INFO:tasks.workunit.client.0.smithi121.stdout:4/728: creat d5/d11/d28/fee x:0 0 0 2024-04-05T21:26:45.593 INFO:tasks.workunit.client.0.smithi121.stdout:6/459: link d1/c24 d1/d2/d1b/d35/d74/c9d 0 2024-04-05T21:26:45.594 INFO:tasks.workunit.client.0.smithi121.stdout:5/709: dread - dd/de/fc3 zero size 2024-04-05T21:26:45.595 INFO:tasks.workunit.client.1.smithi188.stdout:5/736: dwrite d6/d82/d40/d42/d81/d95/f9d [4194304,4194304] 0 2024-04-05T21:26:45.597 INFO:tasks.workunit.client.0.smithi121.stdout:4/729: write d5/d89/fc9 [154907,60200] 0 2024-04-05T21:26:45.597 INFO:tasks.workunit.client.0.smithi121.stdout:6/460: creat d1/d2/d1b/f9e x:0 0 0 2024-04-05T21:26:45.598 INFO:tasks.workunit.client.0.smithi121.stdout:5/710: dread - dd/de/dc1/fe4 zero size 2024-04-05T21:26:45.598 INFO:tasks.workunit.client.0.smithi121.stdout:8/530: dread d12/d13/d15/d1d/f4c [0,4194304] 0 2024-04-05T21:26:45.600 INFO:tasks.workunit.client.1.smithi188.stdout:1/777: dwrite d1/dc/d16/d21/d27/d4a/f7f [0,4194304] 0 2024-04-05T21:26:45.600 INFO:tasks.workunit.client.1.smithi188.stdout:3/856: write d5/d7/d65/f44 [1539275,59820] 0 2024-04-05T21:26:45.601 INFO:tasks.workunit.client.0.smithi121.stdout:7/631: dwrite d0/d23/d5b/d52/f6e [0,4194304] 0 2024-04-05T21:26:45.603 INFO:tasks.workunit.client.1.smithi188.stdout:5/737: fdatasync d6/d82/d40/d42/d81/da6/db2/fc0 0 2024-04-05T21:26:45.603 INFO:tasks.workunit.client.1.smithi188.stdout:1/778: rmdir d1/dc/d16/d46/d88/d3f 39 2024-04-05T21:26:45.604 INFO:tasks.workunit.client.0.smithi121.stdout:7/632: chown d0/d23/c85 1002381 1 2024-04-05T21:26:45.606 INFO:tasks.workunit.client.1.smithi188.stdout:5/738: truncate d6/d15/d1b/d4f/f59 2351504 0 2024-04-05T21:26:45.607 INFO:tasks.workunit.client.0.smithi121.stdout:6/461: write d1/d2/d3/d5/fd [4364627,62066] 0 2024-04-05T21:26:45.607 INFO:tasks.workunit.client.1.smithi188.stdout:3/857: write d5/d7/d6d/d75/d82/d109/fc9 [3360590,82525] 0 2024-04-05T21:26:45.607 INFO:tasks.workunit.client.1.smithi188.stdout:5/739: chown d6/d17/d39/d8e 333867 1 2024-04-05T21:26:45.614 INFO:tasks.workunit.client.1.smithi188.stdout:1/779: creat d1/dc/d16/d46/d88/d13/fef x:0 0 0 2024-04-05T21:26:45.614 INFO:tasks.workunit.client.1.smithi188.stdout:3/858: read d5/d7/d31/f5c [7439304,32688] 0 2024-04-05T21:26:45.615 INFO:tasks.workunit.client.1.smithi188.stdout:1/780: chown d1/dc/d16/d46/d88/d13/fcb 106 1 2024-04-05T21:26:45.617 INFO:tasks.workunit.client.0.smithi121.stdout:4/730: dwrite d5/d11/f8a [0,4194304] 0 2024-04-05T21:26:45.617 INFO:tasks.workunit.client.0.smithi121.stdout:6/462: creat d1/d2/d1b/d91/f9f x:0 0 0 2024-04-05T21:26:45.617 INFO:tasks.workunit.client.0.smithi121.stdout:4/731: chown d5/d11 40582 1 2024-04-05T21:26:45.618 INFO:tasks.workunit.client.0.smithi121.stdout:6/463: rename d1 to d1/d2/dc/da0 22 2024-04-05T21:26:45.619 INFO:tasks.workunit.client.1.smithi188.stdout:5/740: rename d6/d15/d1b/ddc/cfa to d6/d15/d5c/d7a/d44/d51/cfc 0 2024-04-05T21:26:45.619 INFO:tasks.workunit.client.1.smithi188.stdout:3/859: creat d5/d7/d65/d61/db5/f121 x:0 0 0 2024-04-05T21:26:45.620 INFO:tasks.workunit.client.1.smithi188.stdout:3/860: stat d5/d7/d6d/l7c 0 2024-04-05T21:26:45.620 INFO:tasks.workunit.client.1.smithi188.stdout:1/781: mknod d1/d7/cf0 0 2024-04-05T21:26:45.623 INFO:tasks.workunit.client.1.smithi188.stdout:3/861: chown d5/ddd/fcc 1852952 1 2024-04-05T21:26:45.623 INFO:tasks.workunit.client.1.smithi188.stdout:1/782: symlink d1/dc/d16/d21/lf1 0 2024-04-05T21:26:45.625 INFO:tasks.workunit.client.1.smithi188.stdout:3/862: chown d5/d7/d65/d61/d66/d6e/l85 28 1 2024-04-05T21:26:45.625 INFO:tasks.workunit.client.0.smithi121.stdout:6/464: dwrite d1/d2/dc/f5b [0,4194304] 0 2024-04-05T21:26:45.626 INFO:tasks.workunit.client.1.smithi188.stdout:5/741: link d6/d82/f3b d6/d82/d40/d42/ffd 0 2024-04-05T21:26:45.628 INFO:tasks.workunit.client.1.smithi188.stdout:3/863: write d5/d7e/fe1 [903469,110099] 0 2024-04-05T21:26:45.628 INFO:tasks.workunit.client.0.smithi121.stdout:6/465: mkdir d1/d2/d1b/da1 0 2024-04-05T21:26:45.629 INFO:tasks.workunit.client.1.smithi188.stdout:1/783: mknod d1/dc/d16/d21/d27/d4a/dc1/cf2 0 2024-04-05T21:26:45.631 INFO:tasks.workunit.client.1.smithi188.stdout:3/864: dread - d5/d7/d65/d61/db5/f121 zero size 2024-04-05T21:26:45.632 INFO:tasks.workunit.client.0.smithi121.stdout:6/466: getdents d1/d2/d1b/d2e/d53 0 2024-04-05T21:26:45.634 INFO:tasks.workunit.client.1.smithi188.stdout:5/742: creat d6/d15/d5c/ffe x:0 0 0 2024-04-05T21:26:45.640 INFO:tasks.workunit.client.1.smithi188.stdout:5/743: dread d6/f9a [0,4194304] 0 2024-04-05T21:26:45.640 INFO:tasks.workunit.client.1.smithi188.stdout:3/865: rename d5/d7/d6d/d75/fb8 to d5/d7/d65/da2/f122 0 2024-04-05T21:26:45.642 INFO:tasks.workunit.client.0.smithi121.stdout:8/531: write ff [843122,112041] 0 2024-04-05T21:26:45.643 INFO:tasks.workunit.client.1.smithi188.stdout:5/744: symlink d6/d17/d39/d8e/def/lff 0 2024-04-05T21:26:45.643 INFO:tasks.workunit.client.1.smithi188.stdout:3/866: truncate d5/d7/d31/f78 1012872 0 2024-04-05T21:26:45.644 INFO:tasks.workunit.client.0.smithi121.stdout:8/532: creat d12/d50/d61/faa x:0 0 0 2024-04-05T21:26:45.646 INFO:tasks.workunit.client.1.smithi188.stdout:5/745: creat d6/d15/d5c/d7a/d44/d51/f100 x:0 0 0 2024-04-05T21:26:45.647 INFO:tasks.workunit.client.1.smithi188.stdout:5/746: rmdir d6 39 2024-04-05T21:26:45.649 INFO:tasks.workunit.client.0.smithi121.stdout:8/533: dwrite d12/d13/d15/f83 [0,4194304] 0 2024-04-05T21:26:45.651 INFO:tasks.workunit.client.1.smithi188.stdout:5/747: fdatasync d6/d15/f61 0 2024-04-05T21:26:45.651 INFO:tasks.workunit.client.0.smithi121.stdout:7/633: write d0/d9/f26 [25240,96468] 0 2024-04-05T21:26:45.653 INFO:tasks.workunit.client.0.smithi121.stdout:8/534: creat d12/d50/d93/d96/fab x:0 0 0 2024-04-05T21:26:45.656 INFO:tasks.workunit.client.1.smithi188.stdout:5/748: getdents d6/d15/d5c/d7a/d44/d51/dc6 0 2024-04-05T21:26:45.656 INFO:tasks.workunit.client.0.smithi121.stdout:5/711: dread dd/de/d39/d3b/f94 [0,4194304] 0 2024-04-05T21:26:45.657 INFO:tasks.workunit.client.1.smithi188.stdout:5/749: symlink d6/d17/d23/d7d/dc4/l101 0 2024-04-05T21:26:45.659 INFO:tasks.workunit.client.0.smithi121.stdout:5/712: chown dd/de/d19/d13/d53/dba/ffa 13815 1 2024-04-05T21:26:45.660 INFO:tasks.workunit.client.1.smithi188.stdout:5/750: symlink d6/d17/d39/d8e/l102 0 2024-04-05T21:26:45.660 INFO:tasks.workunit.client.0.smithi121.stdout:8/535: getdents d12/d50/d61 0 2024-04-05T21:26:45.661 INFO:tasks.workunit.client.0.smithi121.stdout:6/467: dwrite d1/d2/d3/f4e [0,4194304] 0 2024-04-05T21:26:45.665 INFO:tasks.workunit.client.1.smithi188.stdout:5/751: unlink d6/l31 0 2024-04-05T21:26:45.666 INFO:tasks.workunit.client.0.smithi121.stdout:8/536: truncate d12/d13/d56/d57/f74 248891 0 2024-04-05T21:26:45.666 INFO:tasks.workunit.client.0.smithi121.stdout:6/468: write d1/d2/dc/f4c [2107225,12794] 0 2024-04-05T21:26:45.667 INFO:tasks.workunit.client.1.smithi188.stdout:5/752: truncate d6/d15/d5c/d7a/d44/d51/dc6/faa 4515102 0 2024-04-05T21:26:45.668 INFO:tasks.workunit.client.0.smithi121.stdout:6/469: stat d1/l3b 0 2024-04-05T21:26:45.669 INFO:tasks.workunit.client.0.smithi121.stdout:8/537: creat d12/d50/d6a/d63/d9c/fac x:0 0 0 2024-04-05T21:26:45.672 INFO:tasks.workunit.client.1.smithi188.stdout:1/784: write d1/fb [4719056,127182] 0 2024-04-05T21:26:45.674 INFO:tasks.workunit.client.1.smithi188.stdout:3/867: dread d5/d7/d6d/d75/d83/f8e [0,4194304] 0 2024-04-05T21:26:45.676 INFO:tasks.workunit.client.1.smithi188.stdout:5/753: link d6/d82/f3b d6/d17/d23/d7d/dcf/f103 0 2024-04-05T21:26:45.677 INFO:tasks.workunit.client.0.smithi121.stdout:8/538: dwrite d12/d50/d6a/d63/d9c/fac [0,4194304] 0 2024-04-05T21:26:45.679 INFO:tasks.workunit.client.0.smithi121.stdout:5/713: write dd/de/d19/d3c/f40 [1077728,90175] 0 2024-04-05T21:26:45.679 INFO:tasks.workunit.client.0.smithi121.stdout:8/539: dread d12/f73 [0,4194304] 0 2024-04-05T21:26:45.682 INFO:tasks.workunit.client.0.smithi121.stdout:5/714: rmdir dd/de/d39/d3b/daf 39 2024-04-05T21:26:45.684 INFO:tasks.workunit.client.0.smithi121.stdout:5/715: write dd/de/d19/d13/d53/dba/ffa [1611959,128043] 0 2024-04-05T21:26:45.687 INFO:tasks.workunit.client.0.smithi121.stdout:8/540: link d12/d13/d15/d1d/l86 d12/d50/d6a/d87/lad 0 2024-04-05T21:26:45.688 INFO:tasks.workunit.client.0.smithi121.stdout:5/716: chown dd/de/d19/d38/d45/f4e 0 1 2024-04-05T21:26:45.689 INFO:tasks.workunit.client.1.smithi188.stdout:3/868: mkdir d5/d7e/d123 0 2024-04-05T21:26:45.689 INFO:tasks.workunit.client.0.smithi121.stdout:5/717: fsync dd/de/d39/d3b/fa1 0 2024-04-05T21:26:45.691 INFO:tasks.workunit.client.0.smithi121.stdout:5/718: mkdir dd/de/dc1/d104 0 2024-04-05T21:26:45.696 INFO:tasks.workunit.client.0.smithi121.stdout:5/719: rename dd/de/d19/d38/de3/d100 to dd/de/d105 0 2024-04-05T21:26:45.702 INFO:tasks.workunit.client.0.smithi121.stdout:5/720: read dd/de/d19/d13/d53/f7f [2821180,92265] 0 2024-04-05T21:26:45.705 INFO:tasks.workunit.client.0.smithi121.stdout:5/721: creat dd/d41/dce/da6/dd3/db1/f106 x:0 0 0 2024-04-05T21:26:45.706 INFO:tasks.workunit.client.0.smithi121.stdout:5/722: chown dd/de/d39/d3b/d50/cf2 122064 1 2024-04-05T21:26:45.707 INFO:tasks.workunit.client.0.smithi121.stdout:5/723: stat dd/de/d19/d3c/lfe 0 2024-04-05T21:26:45.709 INFO:tasks.workunit.client.1.smithi188.stdout:1/785: creat d1/dc/d16/d21/d27/ff3 x:0 0 0 2024-04-05T21:26:45.710 INFO:tasks.workunit.client.1.smithi188.stdout:1/786: fsync d1/d7/dbf/fd9 0 2024-04-05T21:26:45.716 INFO:tasks.workunit.client.1.smithi188.stdout:5/754: rmdir d6/d82/d40/dc1/df5 0 2024-04-05T21:26:45.718 INFO:tasks.workunit.client.1.smithi188.stdout:5/755: unlink d6/d15/l48 0 2024-04-05T21:26:45.718 INFO:tasks.workunit.client.1.smithi188.stdout:1/787: dwrite d1/dc/d16/d21/d27/d4a/f63 [0,4194304] 0 2024-04-05T21:26:45.719 INFO:tasks.workunit.client.1.smithi188.stdout:1/788: write d1/fbc [5032865,54226] 0 2024-04-05T21:26:45.720 INFO:tasks.workunit.client.1.smithi188.stdout:5/756: write d6/d17/d23/d7d/dcf/fd6 [396984,28908] 0 2024-04-05T21:26:45.726 INFO:tasks.workunit.client.1.smithi188.stdout:5/757: mkdir d6/d15/d5c/d7a/d44/d51/df1/d104 0 2024-04-05T21:26:45.728 INFO:tasks.workunit.client.1.smithi188.stdout:1/789: rename d1/dc/d16/d46/d95/ca9 to d1/dc/d16/d46/d88/cf4 0 2024-04-05T21:26:45.729 INFO:tasks.workunit.client.1.smithi188.stdout:5/758: mknod d6/d82/d40/d42/d81/da6/c105 0 2024-04-05T21:26:45.731 INFO:tasks.workunit.client.1.smithi188.stdout:1/790: truncate d1/dc/d16/d46/d88/d13/d60/fd0 883192 0 2024-04-05T21:26:45.731 INFO:tasks.workunit.client.1.smithi188.stdout:5/759: truncate d6/d15/d5c/d7a/fa1 857742 0 2024-04-05T21:26:45.732 INFO:tasks.workunit.client.1.smithi188.stdout:1/791: write d1/dc/d16/f33 [8819538,125673] 0 2024-04-05T21:26:45.735 INFO:tasks.workunit.client.1.smithi188.stdout:1/792: symlink d1/dc/d16/d2d/lf5 0 2024-04-05T21:26:45.738 INFO:tasks.workunit.client.1.smithi188.stdout:5/760: rename d6/d17/d39 to d6/d106 0 2024-04-05T21:26:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:45 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:45 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:45 smithi121 ceph-mon[39723]: pgmap v21: 65 pgs: 65 active+clean; 2.8 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 109 MiB/s rd, 326 MiB/s wr, 640 op/s 2024-04-05T21:26:45.740 INFO:tasks.workunit.client.1.smithi188.stdout:1/793: rename d1/dc/d16/d2d/de6/cee to d1/d7/d4e/d5f/cf6 0 2024-04-05T21:26:45.741 INFO:tasks.workunit.client.1.smithi188.stdout:5/761: unlink d6/d15/d1b/lb4 0 2024-04-05T21:26:45.748 INFO:tasks.workunit.client.1.smithi188.stdout:3/869: write d5/d7e/d101/fc0 [1303248,18865] 0 2024-04-05T21:26:45.766 INFO:tasks.workunit.client.0.smithi121.stdout:9/441: sync 2024-04-05T21:26:45.768 INFO:tasks.workunit.client.0.smithi121.stdout:9/442: getdents d6/df/d48/d96 0 2024-04-05T21:26:45.769 INFO:tasks.workunit.client.0.smithi121.stdout:9/443: mknod d6/d46/c97 0 2024-04-05T21:26:45.770 INFO:tasks.workunit.client.0.smithi121.stdout:9/444: getdents d6/df/d48/d96 0 2024-04-05T21:26:45.771 INFO:tasks.workunit.client.0.smithi121.stdout:9/445: chown d6/d36/d7b/d49/d66/f91 835 1 2024-04-05T21:26:45.771 INFO:tasks.workunit.client.0.smithi121.stdout:9/446: dread - d6/d46/d8a/f93 zero size 2024-04-05T21:26:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:45 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:45 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:45 smithi188 ceph-mon[49715]: pgmap v21: 65 pgs: 65 active+clean; 2.8 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 109 MiB/s rd, 326 MiB/s wr, 640 op/s 2024-04-05T21:26:45.783 INFO:tasks.workunit.client.1.smithi188.stdout:6/816: sync 2024-04-05T21:26:45.785 INFO:tasks.workunit.client.1.smithi188.stdout:3/870: mknod d5/d7/d65/d61/d66/da4/de3/df6/c124 0 2024-04-05T21:26:45.786 INFO:tasks.workunit.client.1.smithi188.stdout:6/817: creat d6/d13/d18/d19/dc8/ff5 x:0 0 0 2024-04-05T21:26:45.789 INFO:tasks.workunit.client.1.smithi188.stdout:6/818: creat d6/d13/d18/d3c/dd2/ff6 x:0 0 0 2024-04-05T21:26:45.789 INFO:tasks.workunit.client.1.smithi188.stdout:6/819: stat d6/d13/d38/l39 0 2024-04-05T21:26:45.813 INFO:tasks.workunit.client.1.smithi188.stdout:6/820: mkdir d6/d7/df7 0 2024-04-05T21:26:45.818 INFO:tasks.workunit.client.1.smithi188.stdout:5/762: write d6/d82/d40/fa5 [32560,8042] 0 2024-04-05T21:26:45.827 INFO:tasks.workunit.client.1.smithi188.stdout:5/763: unlink d6/d15/d5c/d7a/f93 0 2024-04-05T21:26:45.827 INFO:tasks.workunit.client.1.smithi188.stdout:6/821: dwrite d6/d13/d18/d3c/f5a [0,4194304] 0 2024-04-05T21:26:45.831 INFO:tasks.workunit.client.1.smithi188.stdout:1/794: dwrite d1/d7/d4e/f8b [0,4194304] 0 2024-04-05T21:26:45.832 INFO:tasks.workunit.client.1.smithi188.stdout:3/871: dwrite d5/d7/d65/d61/f9a [0,4194304] 0 2024-04-05T21:26:45.835 INFO:tasks.workunit.client.1.smithi188.stdout:3/872: read d5/ddd/fcc [566404,87878] 0 2024-04-05T21:26:45.835 INFO:tasks.workunit.client.1.smithi188.stdout:5/764: mkdir d6/d82/d40/d42/d81/da6/db2/d107 0 2024-04-05T21:26:45.836 INFO:tasks.workunit.client.1.smithi188.stdout:6/822: dread - d6/d13/d18/d19/d61/fab zero size 2024-04-05T21:26:45.838 INFO:tasks.workunit.client.1.smithi188.stdout:3/873: write d5/d7/d65/fbe [207765,113776] 0 2024-04-05T21:26:45.841 INFO:tasks.workunit.client.1.smithi188.stdout:1/795: rename d1/d7/d4e to d1/dc/d16/d46/df7 0 2024-04-05T21:26:45.844 INFO:tasks.workunit.client.1.smithi188.stdout:3/874: dread d5/d7/d65/d61/fca [0,4194304] 0 2024-04-05T21:26:45.846 INFO:tasks.workunit.client.1.smithi188.stdout:5/765: creat d6/d82/d40/d42/d81/da6/f108 x:0 0 0 2024-04-05T21:26:45.847 INFO:tasks.workunit.client.1.smithi188.stdout:6/823: creat d6/d13/d18/d94/ff8 x:0 0 0 2024-04-05T21:26:45.850 INFO:tasks.workunit.client.1.smithi188.stdout:3/875: fdatasync d5/ddd/fcc 0 2024-04-05T21:26:45.850 INFO:tasks.workunit.client.1.smithi188.stdout:5/766: truncate d6/f9a 1530795 0 2024-04-05T21:26:45.854 INFO:tasks.workunit.client.1.smithi188.stdout:3/876: unlink d5/d39/l46 0 2024-04-05T21:26:45.856 INFO:tasks.workunit.client.1.smithi188.stdout:5/767: symlink d6/d82/d40/d42/d81/da6/db2/l109 0 2024-04-05T21:26:45.856 INFO:tasks.workunit.client.1.smithi188.stdout:5/768: chown d6/cab 0 1 2024-04-05T21:26:45.857 INFO:tasks.workunit.client.1.smithi188.stdout:5/769: dread - d6/d15/d5c/d7a/d44/fe2 zero size 2024-04-05T21:26:45.858 INFO:tasks.workunit.client.1.smithi188.stdout:3/877: unlink d5/d7/c7f 0 2024-04-05T21:26:45.861 INFO:tasks.workunit.client.1.smithi188.stdout:3/878: creat d5/d7/d65/d61/db5/f125 x:0 0 0 2024-04-05T21:26:45.861 INFO:tasks.workunit.client.1.smithi188.stdout:5/770: creat d6/d15/d5c/d7a/d44/f10a x:0 0 0 2024-04-05T21:26:45.863 INFO:tasks.workunit.client.1.smithi188.stdout:6/824: write d6/d13/d18/d19/d2c/d3f/f7a [648948,57032] 0 2024-04-05T21:26:45.863 INFO:tasks.workunit.client.1.smithi188.stdout:5/771: chown d6/d15/d5c/d7a/f3c 7298211 1 2024-04-05T21:26:45.870 INFO:tasks.workunit.client.1.smithi188.stdout:6/825: dread d6/d13/d18/d19/da3/fca [0,4194304] 0 2024-04-05T21:26:45.871 INFO:tasks.workunit.client.1.smithi188.stdout:1/796: dwrite d1/dc/d16/d46/d88/d3f/f4b [0,4194304] 0 2024-04-05T21:26:45.872 INFO:tasks.workunit.client.1.smithi188.stdout:3/879: dwrite d5/d7/d65/d61/db5/f112 [0,4194304] 0 2024-04-05T21:26:45.874 INFO:tasks.workunit.client.1.smithi188.stdout:3/880: chown d5/d35/f9e 847899053 1 2024-04-05T21:26:45.875 INFO:tasks.workunit.client.1.smithi188.stdout:3/881: readlink d5/d48/l105 0 2024-04-05T21:26:45.876 INFO:tasks.workunit.client.1.smithi188.stdout:1/797: fsync d1/dc/d16/d46/d88/d13/d60/fd4 0 2024-04-05T21:26:45.877 INFO:tasks.workunit.client.1.smithi188.stdout:5/772: truncate d6/f6f 276954 0 2024-04-05T21:26:45.880 INFO:tasks.workunit.client.1.smithi188.stdout:5/773: write d6/d15/f7f [1478530,78459] 0 2024-04-05T21:26:45.882 INFO:tasks.workunit.client.1.smithi188.stdout:1/798: chown d1/dc/d16/d2d/f9a 48145 1 2024-04-05T21:26:45.886 INFO:tasks.workunit.client.1.smithi188.stdout:5/774: creat d6/f10b x:0 0 0 2024-04-05T21:26:45.886 INFO:tasks.workunit.client.1.smithi188.stdout:6/826: dwrite d6/d13/d18/d19/dc8/f9f [0,4194304] 0 2024-04-05T21:26:45.887 INFO:tasks.workunit.client.1.smithi188.stdout:5/775: stat d6/d106/d8e/def 0 2024-04-05T21:26:45.888 INFO:tasks.workunit.client.1.smithi188.stdout:6/827: creat d6/d13/db9/ff9 x:0 0 0 2024-04-05T21:26:45.889 INFO:tasks.workunit.client.1.smithi188.stdout:5/776: stat d6/d106/d8e/f8f 0 2024-04-05T21:26:45.890 INFO:tasks.workunit.client.1.smithi188.stdout:1/799: rmdir d1/dc/d16/d2d/de6 0 2024-04-05T21:26:45.893 INFO:tasks.workunit.client.1.smithi188.stdout:5/777: read - d6/d82/f89 zero size 2024-04-05T21:26:45.899 INFO:tasks.workunit.client.1.smithi188.stdout:5/778: chown d6/d15/d5c/d7a/d44/d98/lee 11779 1 2024-04-05T21:26:45.901 INFO:tasks.workunit.client.1.smithi188.stdout:1/800: dwrite d1/dc/d16/d46/d88/d13/f82 [0,4194304] 0 2024-04-05T21:26:45.906 INFO:tasks.workunit.client.1.smithi188.stdout:5/779: fsync d6/d15/d5c/d7a/f2b 0 2024-04-05T21:26:45.907 INFO:tasks.workunit.client.1.smithi188.stdout:1/801: unlink d1/dc/d16/d46/d95/fb5 0 2024-04-05T21:26:45.910 INFO:tasks.workunit.client.1.smithi188.stdout:6/828: dwrite d6/d13/d18/d3c/ddc/d60/fbc [0,4194304] 0 2024-04-05T21:26:45.912 INFO:tasks.workunit.client.0.smithi121.stdout:1/653: sync 2024-04-05T21:26:45.912 INFO:tasks.workunit.client.0.smithi121.stdout:0/628: sync 2024-04-05T21:26:45.912 INFO:tasks.workunit.client.0.smithi121.stdout:3/459: sync 2024-04-05T21:26:45.913 INFO:tasks.workunit.client.1.smithi188.stdout:3/882: dread d5/df/d80/fc4 [0,4194304] 0 2024-04-05T21:26:45.913 INFO:tasks.workunit.client.1.smithi188.stdout:6/829: write d6/d13/db9/ff9 [506559,40575] 0 2024-04-05T21:26:45.913 INFO:tasks.workunit.client.1.smithi188.stdout:1/802: rmdir d1/dc/d16/d2d/dd5 39 2024-04-05T21:26:45.918 INFO:tasks.workunit.client.0.smithi121.stdout:3/460: creat d0/f96 x:0 0 0 2024-04-05T21:26:45.918 INFO:tasks.workunit.client.0.smithi121.stdout:1/654: rename d8/d8c/db6/d1a/d1b/d38/d4b/ccf to d8/d8c/db6/d1a/d1b/d4c/d76/d53/cd3 0 2024-04-05T21:26:45.918 INFO:tasks.workunit.client.0.smithi121.stdout:3/461: truncate d0/d3/d24/d3d/f21 4342509 0 2024-04-05T21:26:45.918 INFO:tasks.workunit.client.0.smithi121.stdout:0/629: link d7/d71/c99 d7/d71/d29/d2a/ce0 0 2024-04-05T21:26:45.919 INFO:tasks.workunit.client.1.smithi188.stdout:6/830: mkdir d6/d13/d18/d19/dc8/da4/dfa 0 2024-04-05T21:26:45.919 INFO:tasks.workunit.client.0.smithi121.stdout:1/655: truncate d8/d89/d9e/d2d/fd1 84987 0 2024-04-05T21:26:45.921 INFO:tasks.workunit.client.0.smithi121.stdout:0/630: dread d7/d1c/f23 [0,4194304] 0 2024-04-05T21:26:45.922 INFO:tasks.workunit.client.1.smithi188.stdout:1/803: link d1/d7/f80 d1/dc/d16/d46/d88/dea/ff8 0 2024-04-05T21:26:45.922 INFO:tasks.workunit.client.0.smithi121.stdout:1/656: symlink d8/d8c/db6/d1a/d1b/d4c/ld4 0 2024-04-05T21:26:45.923 INFO:tasks.workunit.client.0.smithi121.stdout:0/631: write d7/dc/fab [4671704,15511] 0 2024-04-05T21:26:45.924 INFO:tasks.workunit.client.1.smithi188.stdout:1/804: truncate d1/dc/d16/d21/d27/d4a/dc1/fce 295879 0 2024-04-05T21:26:45.926 INFO:tasks.workunit.client.0.smithi121.stdout:1/657: write d8/d8c/db6/d1a/d1b/f69 [4730535,37161] 0 2024-04-05T21:26:45.929 INFO:tasks.workunit.client.1.smithi188.stdout:1/805: link d1/dc/d16/d21/lf1 d1/dc/d16/d46/df7/de2/lf9 0 2024-04-05T21:26:45.933 INFO:tasks.workunit.client.0.smithi121.stdout:1/658: dwrite d8/d8c/db6/d1a/d1b/d4c/d76/fce [0,4194304] 0 2024-04-05T21:26:45.934 INFO:tasks.workunit.client.1.smithi188.stdout:5/780: dread d6/f2d [0,4194304] 0 2024-04-05T21:26:45.935 INFO:tasks.workunit.client.1.smithi188.stdout:1/806: dread d1/dc/d16/d21/d27/d4a/f63 [0,4194304] 0 2024-04-05T21:26:45.935 INFO:tasks.workunit.client.0.smithi121.stdout:1/659: write d8/d8c/db6/d1a/d1b/d4c/f7e [342870,67339] 0 2024-04-05T21:26:45.936 INFO:tasks.workunit.client.1.smithi188.stdout:5/781: truncate d6/d15/d5c/ffe 608194 0 2024-04-05T21:26:45.938 INFO:tasks.workunit.client.1.smithi188.stdout:1/807: truncate d1/dc/d16/d21/d27/d49/f9b 1115634 0 2024-04-05T21:26:45.939 INFO:tasks.workunit.client.1.smithi188.stdout:5/782: mkdir d6/d82/d40/d42/d81/d10c 0 2024-04-05T21:26:45.940 INFO:tasks.workunit.client.1.smithi188.stdout:1/808: read d1/d7/f66 [630137,12797] 0 2024-04-05T21:26:45.940 INFO:tasks.workunit.client.1.smithi188.stdout:3/883: dread d5/d7/d65/d61/d66/f9f [0,4194304] 0 2024-04-05T21:26:45.941 INFO:tasks.workunit.client.1.smithi188.stdout:5/783: stat d6/d17/d23/d7d/dc4 0 2024-04-05T21:26:45.942 INFO:tasks.workunit.client.0.smithi121.stdout:1/660: rename d8/d8c/db6/d1a/d1b/d38/d4b/fb8 to d8/d8c/db6/d1a/d1b/d44/fd5 0 2024-04-05T21:26:45.942 INFO:tasks.workunit.client.0.smithi121.stdout:3/462: write d0/d3/de/f36 [1518136,121599] 0 2024-04-05T21:26:45.943 INFO:tasks.workunit.client.0.smithi121.stdout:0/632: write d7/d7d/f81 [452795,101860] 0 2024-04-05T21:26:45.944 INFO:tasks.workunit.client.1.smithi188.stdout:6/831: dread d6/d13/d38/f4c [0,4194304] 0 2024-04-05T21:26:45.946 INFO:tasks.workunit.client.1.smithi188.stdout:5/784: chown d6/d15/d5c/d7a/fc7 3299 1 2024-04-05T21:26:45.946 INFO:tasks.workunit.client.1.smithi188.stdout:1/809: fdatasync d1/d7/f80 0 2024-04-05T21:26:45.948 INFO:tasks.workunit.client.1.smithi188.stdout:3/884: creat d5/d7/d65/d5f/d102/f126 x:0 0 0 2024-04-05T21:26:45.949 INFO:tasks.workunit.client.1.smithi188.stdout:1/810: creat d1/dc/d16/d21/ffa x:0 0 0 2024-04-05T21:26:45.950 INFO:tasks.workunit.client.1.smithi188.stdout:5/785: mkdir d6/d82/d40/d42/d81/d10c/d10d 0 2024-04-05T21:26:45.951 INFO:tasks.workunit.client.1.smithi188.stdout:3/885: write d5/d7/d65/d61/db5/ff0 [740352,12004] 0 2024-04-05T21:26:45.952 INFO:tasks.workunit.client.1.smithi188.stdout:3/886: readlink d5/d7/d65/d61/d66/lb9 0 2024-04-05T21:26:45.953 INFO:tasks.workunit.client.1.smithi188.stdout:1/811: mknod d1/dc/d16/d21/d27/d49/cfb 0 2024-04-05T21:26:45.956 INFO:tasks.workunit.client.1.smithi188.stdout:1/812: mkdir d1/dc/dfc 0 2024-04-05T21:26:45.956 INFO:tasks.workunit.client.1.smithi188.stdout:5/786: dread d6/d17/d23/d7d/fd7 [0,4194304] 0 2024-04-05T21:26:45.957 INFO:tasks.workunit.client.1.smithi188.stdout:5/787: chown d6/d106/de1 739578 1 2024-04-05T21:26:45.958 INFO:tasks.workunit.client.0.smithi121.stdout:0/633: write d7/d62/f73 [135678,122383] 0 2024-04-05T21:26:45.959 INFO:tasks.workunit.client.1.smithi188.stdout:1/813: mknod d1/d7/cfd 0 2024-04-05T21:26:45.964 INFO:tasks.workunit.client.0.smithi121.stdout:1/661: dwrite d8/d8c/db6/d1a/d1b/d4c/f63 [0,4194304] 0 2024-04-05T21:26:45.964 INFO:tasks.workunit.client.1.smithi188.stdout:5/788: symlink d6/d15/d5c/d7a/d44/d98/dbf/l10e 0 2024-04-05T21:26:45.965 INFO:tasks.workunit.client.0.smithi121.stdout:1/662: stat d8/d8c/db6/d1a/d1b/d85 0 2024-04-05T21:26:45.966 INFO:tasks.workunit.client.0.smithi121.stdout:0/634: read d7/d71/f26 [990322,18003] 0 2024-04-05T21:26:45.966 INFO:tasks.workunit.client.1.smithi188.stdout:5/789: symlink d6/d106/d8e/l10f 0 2024-04-05T21:26:45.968 INFO:tasks.workunit.client.0.smithi121.stdout:1/663: unlink d8/d8c/db6/d1a/d1b/d4c/ld4 0 2024-04-05T21:26:45.969 INFO:tasks.workunit.client.0.smithi121.stdout:1/664: stat d8/d89/d9e/d2d 0 2024-04-05T21:26:45.970 INFO:tasks.workunit.client.1.smithi188.stdout:3/887: dread d5/d7/d65/da2/f122 [0,4194304] 0 2024-04-05T21:26:45.972 INFO:tasks.workunit.client.0.smithi121.stdout:3/463: dread d0/d3/d24/d3d/f3a [0,4194304] 0 2024-04-05T21:26:45.973 INFO:tasks.workunit.client.1.smithi188.stdout:6/832: dwrite d6/fa [0,4194304] 0 2024-04-05T21:26:45.973 INFO:tasks.workunit.client.0.smithi121.stdout:1/665: mkdir d8/d89/d9e/d57/d77/dd6 0 2024-04-05T21:26:45.974 INFO:tasks.workunit.client.0.smithi121.stdout:3/464: write d0/d3/f5b [2281152,22644] 0 2024-04-05T21:26:45.975 INFO:tasks.workunit.client.1.smithi188.stdout:5/790: dread d6/d15/d1b/f6c [0,4194304] 0 2024-04-05T21:26:45.977 INFO:tasks.workunit.client.0.smithi121.stdout:3/465: truncate d0/d3/f4 1012338 0 2024-04-05T21:26:45.978 INFO:tasks.workunit.client.1.smithi188.stdout:1/814: dwrite d1/dc/d16/f3b [0,4194304] 0 2024-04-05T21:26:45.980 INFO:tasks.workunit.client.0.smithi121.stdout:1/666: read d8/d8c/db6/d1a/f5a [3055286,94556] 0 2024-04-05T21:26:45.981 INFO:tasks.workunit.client.1.smithi188.stdout:3/888: creat d5/d7/d6d/d75/d82/f127 x:0 0 0 2024-04-05T21:26:45.982 INFO:tasks.workunit.client.0.smithi121.stdout:1/667: chown d8/d8c/db6/d1a/d1b/dbb 28253051 1 2024-04-05T21:26:45.984 INFO:tasks.workunit.client.0.smithi121.stdout:3/466: dwrite d0/d3/f5e [4194304,4194304] 0 2024-04-05T21:26:45.984 INFO:tasks.workunit.client.1.smithi188.stdout:6/833: rename d6/d13/d18/f50 to d6/d13/d18/d3c/ffb 0 2024-04-05T21:26:45.985 INFO:tasks.workunit.client.0.smithi121.stdout:1/668: symlink d8/d89/d9e/da3/ld7 0 2024-04-05T21:26:45.985 INFO:tasks.workunit.client.1.smithi188.stdout:5/791: fsync d6/d82/d40/f90 0 2024-04-05T21:26:45.987 INFO:tasks.workunit.client.1.smithi188.stdout:1/815: creat d1/dc/d16/d46/d88/d3f/ffe x:0 0 0 2024-04-05T21:26:45.989 INFO:tasks.workunit.client.1.smithi188.stdout:3/889: mkdir d5/d35/d128 0 2024-04-05T21:26:45.995 INFO:tasks.workunit.client.0.smithi121.stdout:3/467: dwrite d0/d3/d24/d3d/d1a/d23/f56 [0,4194304] 0 2024-04-05T21:26:45.996 INFO:tasks.workunit.client.0.smithi121.stdout:1/669: dwrite d8/d89/fa5 [0,4194304] 0 2024-04-05T21:26:45.998 INFO:tasks.workunit.client.1.smithi188.stdout:6/834: dwrite d6/fa [0,4194304] 0 2024-04-05T21:26:45.998 INFO:tasks.workunit.client.1.smithi188.stdout:1/816: getdents d1/dc/d16/d46/d88/d13/d60 0 2024-04-05T21:26:45.998 INFO:tasks.workunit.client.0.smithi121.stdout:3/468: write d0/d3/d24/d3d/d1a/d23/d38/f65 [2301290,38136] 0 2024-04-05T21:26:46.001 INFO:tasks.workunit.client.1.smithi188.stdout:6/835: mknod d6/d7/cfc 0 2024-04-05T21:26:46.003 INFO:tasks.workunit.client.0.smithi121.stdout:3/469: creat d0/f97 x:0 0 0 2024-04-05T21:26:46.004 INFO:tasks.workunit.client.0.smithi121.stdout:1/670: creat d8/d89/d9e/d57/d77/dd6/fd8 x:0 0 0 2024-04-05T21:26:46.004 INFO:tasks.workunit.client.1.smithi188.stdout:8/676: sync 2024-04-05T21:26:46.005 INFO:tasks.workunit.client.0.smithi121.stdout:3/470: stat d0/d3/d24/d3d/d1a/d23/d90/f94 0 2024-04-05T21:26:46.008 INFO:tasks.workunit.client.0.smithi121.stdout:3/471: write d0/d3/de/f36 [2879524,30848] 0 2024-04-05T21:26:46.011 INFO:tasks.workunit.client.1.smithi188.stdout:6/836: unlink d6/d13/d18/d3c/l4a 0 2024-04-05T21:26:46.013 INFO:tasks.workunit.client.1.smithi188.stdout:6/837: read d6/d7/f16 [6453254,104787] 0 2024-04-05T21:26:46.013 INFO:tasks.workunit.client.1.smithi188.stdout:1/817: link d1/l65 d1/dc/d16/d2d/db0/lff 0 2024-04-05T21:26:46.013 INFO:tasks.workunit.client.0.smithi121.stdout:3/472: write d0/f27 [618454,110086] 0 2024-04-05T21:26:46.013 INFO:tasks.workunit.client.0.smithi121.stdout:1/671: getdents d8/d8c/db6/d1a/d1b/d4c/d76/da4 0 2024-04-05T21:26:46.014 INFO:tasks.workunit.client.1.smithi188.stdout:8/677: creat d5/d21/d2a/d95/fdf x:0 0 0 2024-04-05T21:26:46.014 INFO:tasks.workunit.client.0.smithi121.stdout:1/672: symlink d8/d8c/db6/d1a/d1b/d24/ld9 0 2024-04-05T21:26:46.015 INFO:tasks.workunit.client.0.smithi121.stdout:3/473: symlink d0/d3/d24/d6d/l98 0 2024-04-05T21:26:46.016 INFO:tasks.workunit.client.1.smithi188.stdout:5/792: truncate d6/d82/d40/f90 452128 0 2024-04-05T21:26:46.017 INFO:tasks.workunit.client.0.smithi121.stdout:1/673: write d8/d8c/db6/f8f [262847,86570] 0 2024-04-05T21:26:46.019 INFO:tasks.workunit.client.0.smithi121.stdout:3/474: creat d0/d3/d24/f99 x:0 0 0 2024-04-05T21:26:46.021 INFO:tasks.workunit.client.1.smithi188.stdout:6/838: truncate d6/d7/fc 791872 0 2024-04-05T21:26:46.021 INFO:tasks.workunit.client.0.smithi121.stdout:3/475: write d0/d3/d24/f42 [565799,82549] 0 2024-04-05T21:26:46.021 INFO:tasks.workunit.client.0.smithi121.stdout:1/674: unlink d8/d89/d9e/d57/f99 0 2024-04-05T21:26:46.021 INFO:tasks.workunit.client.1.smithi188.stdout:5/793: mkdir d6/dad/d110 0 2024-04-05T21:26:46.024 INFO:tasks.workunit.client.1.smithi188.stdout:6/839: mknod d6/d28/cfd 0 2024-04-05T21:26:46.024 INFO:tasks.workunit.client.0.smithi121.stdout:1/675: creat d8/d8c/db6/d1a/d1b/d4c/da1/fda x:0 0 0 2024-04-05T21:26:46.025 INFO:tasks.workunit.client.1.smithi188.stdout:8/678: dwrite d5/d21/d40/f4f [0,4194304] 0 2024-04-05T21:26:46.026 INFO:tasks.workunit.client.0.smithi121.stdout:3/476: creat d0/d3/d24/d3d/d1a/d1e/f9a x:0 0 0 2024-04-05T21:26:46.027 INFO:tasks.workunit.client.1.smithi188.stdout:1/818: dwrite d1/d7/fd3 [0,4194304] 0 2024-04-05T21:26:46.029 INFO:tasks.workunit.client.1.smithi188.stdout:5/794: fsync d6/d106/f4a 0 2024-04-05T21:26:46.032 INFO:tasks.workunit.client.0.smithi121.stdout:1/676: dwrite d8/d8c/db6/d1a/d1b/d4c/f63 [0,4194304] 0 2024-04-05T21:26:46.034 INFO:tasks.workunit.client.1.smithi188.stdout:9/857: sync 2024-04-05T21:26:46.034 INFO:tasks.workunit.client.1.smithi188.stdout:0/701: sync 2024-04-05T21:26:46.034 INFO:tasks.workunit.client.1.smithi188.stdout:2/826: sync 2024-04-05T21:26:46.036 INFO:tasks.workunit.client.0.smithi121.stdout:1/677: mkdir d8/d8c/db6/d1a/d1b/d38/d4b/ddb 0 2024-04-05T21:26:46.038 INFO:tasks.workunit.client.0.smithi121.stdout:1/678: mknod d8/d89/cdc 0 2024-04-05T21:26:46.039 INFO:tasks.workunit.client.0.smithi121.stdout:1/679: chown d8/d89/d9e/c97 1830884 1 2024-04-05T21:26:46.041 INFO:tasks.workunit.client.1.smithi188.stdout:9/858: unlink dd/d11/d1f/f6e 0 2024-04-05T21:26:46.042 INFO:tasks.workunit.client.0.smithi121.stdout:1/680: write d8/d8c/db6/d1a/f20 [3911092,2846] 0 2024-04-05T21:26:46.042 INFO:tasks.workunit.client.0.smithi121.stdout:3/477: dread d0/d3/de/f36 [0,4194304] 0 2024-04-05T21:26:46.043 INFO:tasks.workunit.client.0.smithi121.stdout:3/478: write d0/d3/de/f82 [2226889,57932] 0 2024-04-05T21:26:46.044 INFO:tasks.workunit.client.1.smithi188.stdout:3/890: dread d5/d35/f52 [0,4194304] 0 2024-04-05T21:26:46.044 INFO:tasks.workunit.client.1.smithi188.stdout:0/702: dread - d4/d8/db/d42/d71/f8c zero size 2024-04-05T21:26:46.046 INFO:tasks.workunit.client.0.smithi121.stdout:1/681: mknod d8/d8c/db6/cdd 0 2024-04-05T21:26:46.048 INFO:tasks.workunit.client.0.smithi121.stdout:1/682: write d8/d89/fb9 [557048,43362] 0 2024-04-05T21:26:46.051 INFO:tasks.workunit.client.1.smithi188.stdout:6/840: dread d6/d13/d18/d3c/ddc/f69 [0,4194304] 0 2024-04-05T21:26:46.051 INFO:tasks.workunit.client.1.smithi188.stdout:2/827: link d5/db/dd/d73/d109/d9f/fb0 d5/db/dd/d36/d3c/d48/dcf/f112 0 2024-04-05T21:26:46.052 INFO:tasks.workunit.client.0.smithi121.stdout:4/732: sync 2024-04-05T21:26:46.053 INFO:tasks.workunit.client.0.smithi121.stdout:2/485: sync 2024-04-05T21:26:46.054 INFO:tasks.workunit.client.0.smithi121.stdout:4/733: write d5/d45/d67/f84 [1903,33122] 0 2024-04-05T21:26:46.055 INFO:tasks.workunit.client.1.smithi188.stdout:0/703: mknod d4/d8/db/d21/d3c/dee/c105 0 2024-04-05T21:26:46.056 INFO:tasks.workunit.client.0.smithi121.stdout:2/486: dread d2/d9/d38/d2d/f58 [0,4194304] 0 2024-04-05T21:26:46.056 INFO:tasks.workunit.client.1.smithi188.stdout:6/841: dread d6/d13/d38/f55 [0,4194304] 0 2024-04-05T21:26:46.057 INFO:tasks.workunit.client.0.smithi121.stdout:3/479: dread d0/d3/d24/d3d/d1a/d23/f6b [0,4194304] 0 2024-04-05T21:26:46.060 INFO:tasks.workunit.client.1.smithi188.stdout:0/704: write d4/d6/d46/fe7 [3843011,59402] 0 2024-04-05T21:26:46.060 INFO:tasks.workunit.client.1.smithi188.stdout:3/891: mknod d5/d7e/d101/c129 0 2024-04-05T21:26:46.060 INFO:tasks.workunit.client.1.smithi188.stdout:5/795: write d6/d82/f89 [273367,67734] 0 2024-04-05T21:26:46.060 INFO:tasks.workunit.client.0.smithi121.stdout:3/480: fsync d0/d3/de/f6a 0 2024-04-05T21:26:46.060 INFO:tasks.workunit.client.0.smithi121.stdout:3/481: write d0/d3/d24/d3d/d1a/d23/f56 [3432754,107782] 0 2024-04-05T21:26:46.060 INFO:tasks.workunit.client.0.smithi121.stdout:2/487: symlink d2/d9/d1c/d30/l90 0 2024-04-05T21:26:46.061 INFO:tasks.workunit.client.0.smithi121.stdout:2/488: write d2/f11 [6080387,68047] 0 2024-04-05T21:26:46.062 INFO:tasks.workunit.client.1.smithi188.stdout:2/828: mkdir d5/d113 0 2024-04-05T21:26:46.062 INFO:tasks.workunit.client.0.smithi121.stdout:1/683: rename d8/d8c/db6/d1a/d1b/d38/d4b to d8/d89/d9e/d2d/d84/da8/dad/dde 0 2024-04-05T21:26:46.064 INFO:tasks.workunit.client.1.smithi188.stdout:1/819: link d1/d7/l4c d1/dc/d16/d46/d95/d99/l100 0 2024-04-05T21:26:46.066 INFO:tasks.workunit.client.1.smithi188.stdout:3/892: readlink d5/ddd/le8 0 2024-04-05T21:26:46.067 INFO:tasks.workunit.client.0.smithi121.stdout:3/482: mknod d0/d3/d24/d3d/d1a/d23/d90/c9b 0 2024-04-05T21:26:46.067 INFO:tasks.workunit.client.0.smithi121.stdout:3/483: fsync d0/f96 0 2024-04-05T21:26:46.068 INFO:tasks.workunit.client.1.smithi188.stdout:9/859: getdents dd/d11/d73/d85 0 2024-04-05T21:26:46.068 INFO:tasks.workunit.client.1.smithi188.stdout:5/796: fdatasync d6/d17/f80 0 2024-04-05T21:26:46.069 INFO:tasks.workunit.client.0.smithi121.stdout:2/489: mkdir d2/d9/d12/d84/d91 0 2024-04-05T21:26:46.070 INFO:tasks.workunit.client.0.smithi121.stdout:1/684: creat d8/d8c/db6/d1a/d1b/d4c/d76/da4/fdf x:0 0 0 2024-04-05T21:26:46.071 INFO:tasks.workunit.client.1.smithi188.stdout:3/893: write d5/d7/d65/fbe [183769,116105] 0 2024-04-05T21:26:46.072 INFO:tasks.workunit.client.0.smithi121.stdout:4/734: rmdir d5/d11/d2c/d9d/dae/d29/d2e/dde 0 2024-04-05T21:26:46.072 INFO:tasks.workunit.client.1.smithi188.stdout:2/829: chown d5/db/dd/d36/d3c/d63/feb 1 1 2024-04-05T21:26:46.073 INFO:tasks.workunit.client.1.smithi188.stdout:6/842: dread d6/d13/d18/d19/dc8/f9f [0,4194304] 0 2024-04-05T21:26:46.073 INFO:tasks.workunit.client.0.smithi121.stdout:2/490: unlink d2/d9/d1c/l5f 0 2024-04-05T21:26:46.073 INFO:tasks.workunit.client.0.smithi121.stdout:4/735: read - d5/d89/fd1 zero size 2024-04-05T21:26:46.073 INFO:tasks.workunit.client.1.smithi188.stdout:9/860: chown dd/d3d/d95/fe0 1304169 1 2024-04-05T21:26:46.077 INFO:tasks.workunit.client.0.smithi121.stdout:4/736: rmdir d5/d45/d8b 39 2024-04-05T21:26:46.079 INFO:tasks.workunit.client.0.smithi121.stdout:3/484: dwrite d0/d3/de/f8a [0,4194304] 0 2024-04-05T21:26:46.079 INFO:tasks.workunit.client.0.smithi121.stdout:2/491: write d2/d9/d1c/d30/f68 [2068303,66684] 0 2024-04-05T21:26:46.080 INFO:tasks.workunit.client.1.smithi188.stdout:8/679: dread d5/f1f [0,4194304] 0 2024-04-05T21:26:46.081 INFO:tasks.workunit.client.1.smithi188.stdout:9/861: write dd/d11/d73/d8d/fac [1854005,54737] 0 2024-04-05T21:26:46.085 INFO:tasks.workunit.client.0.smithi121.stdout:1/685: dwrite d8/d8c/db6/d1a/d1b/d4c/d76/da4/fdf [0,4194304] 0 2024-04-05T21:26:46.086 INFO:tasks.workunit.client.1.smithi188.stdout:3/894: unlink d5/d7/d6d/dd8/d2f/f4c 0 2024-04-05T21:26:46.087 INFO:tasks.workunit.client.1.smithi188.stdout:6/843: truncate d6/d28/d45/f96 1185283 0 2024-04-05T21:26:46.090 INFO:tasks.workunit.client.1.smithi188.stdout:1/820: dwrite d1/dc/d16/d46/d88/d3f/f69 [0,4194304] 0 2024-04-05T21:26:46.090 INFO:tasks.workunit.client.1.smithi188.stdout:0/705: write d4/d8/db/d42/da2/fa4 [1021956,124778] 0 2024-04-05T21:26:46.090 INFO:tasks.workunit.client.0.smithi121.stdout:1/686: write d8/d8c/db6/d1a/d1b/d4c/f4d [562077,61726] 0 2024-04-05T21:26:46.090 INFO:tasks.workunit.client.0.smithi121.stdout:3/485: creat d0/d3/de/f9c x:0 0 0 2024-04-05T21:26:46.092 INFO:tasks.workunit.client.0.smithi121.stdout:2/492: dwrite d2/f11 [0,4194304] 0 2024-04-05T21:26:46.092 INFO:tasks.workunit.client.0.smithi121.stdout:4/737: dread d5/d11/d2c/d9d/f31 [0,4194304] 0 2024-04-05T21:26:46.097 INFO:tasks.workunit.client.0.smithi121.stdout:4/738: creat d5/d11/d28/d3b/fef x:0 0 0 2024-04-05T21:26:46.099 INFO:tasks.workunit.client.1.smithi188.stdout:3/895: write d5/d7/d6d/f7a [3384976,16908] 0 2024-04-05T21:26:46.205 INFO:tasks.workunit.client.0.smithi121.stdout:3/486: mknod d0/d3/d24/d3d/d1a/d1e/d86/d8e/c9d 0 2024-04-05T21:26:46.214 INFO:tasks.workunit.client.1.smithi188.stdout:6/844: symlink d6/d13/db9/lfe 0 2024-04-05T21:26:46.214 INFO:tasks.workunit.client.1.smithi188.stdout:1/821: dread - d1/dc/d16/d46/feb zero size 2024-04-05T21:26:46.214 INFO:tasks.workunit.client.1.smithi188.stdout:8/680: truncate d5/d21/d40/d43/d48/d7e/fdb 141273 0 2024-04-05T21:26:46.214 INFO:tasks.workunit.client.1.smithi188.stdout:5/797: getdents d6 0 2024-04-05T21:26:46.214 INFO:tasks.workunit.client.1.smithi188.stdout:0/706: readlink d4/d6/d13/d23/d24/l6e 0 2024-04-05T21:26:46.214 INFO:tasks.workunit.client.1.smithi188.stdout:6/845: mkdir d6/d28/dff 0 2024-04-05T21:26:46.214 INFO:tasks.workunit.client.1.smithi188.stdout:1/822: creat d1/dc/d16/d46/df7/f101 x:0 0 0 2024-04-05T21:26:46.214 INFO:tasks.workunit.client.1.smithi188.stdout:5/798: mknod d6/dad/c111 0 2024-04-05T21:26:46.214 INFO:tasks.workunit.client.1.smithi188.stdout:9/862: dread dd/d11/d1f/f6a [0,4194304] 0 2024-04-05T21:26:46.214 INFO:tasks.workunit.client.1.smithi188.stdout:2/830: dread d5/db/dd/d36/d3c/d48/dcf/fd9 [0,4194304] 0 2024-04-05T21:26:46.214 INFO:tasks.workunit.client.1.smithi188.stdout:9/863: chown dd/d34/da7 15938 1 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:3/896: dwrite d5/f37 [0,4194304] 0 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:6/846: creat d6/d28/dff/f100 x:0 0 0 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:2/831: fsync d5/db/dd/fc6 0 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:3/897: unlink d5/d39/f10c 0 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:9/864: dread dd/d3d/f7a [0,4194304] 0 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:5/799: dread d6/d82/d40/f90 [0,4194304] 0 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:6/847: truncate d6/d13/d38/f55 1151827 0 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:6/848: fdatasync d6/d13/db9/ff9 0 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:2/832: creat d5/db/dd/d36/d3c/d48/dd7/d10b/f114 x:0 0 0 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:3/898: link d5/d7e/cd2 d5/d7/d65/d61/c12a 0 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:5/800: unlink d6/d15/d5c/d7a/d44/cd4 0 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:9/865: creat dd/d3d/f125 x:0 0 0 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:6/849: rmdir d6/d13 39 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:2/833: rename d5/db/dd/d73/d109/d66/fb3 to d5/d5f/f115 0 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:6/850: dread - d6/d13/d18/d3c/daf/fde zero size 2024-04-05T21:26:46.215 INFO:tasks.workunit.client.1.smithi188.stdout:5/801: mknod d6/d106/c112 0 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:5/802: dread - d6/d106/d8e/fdb zero size 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:6/851: mkdir d6/d7/df7/d101 0 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:2/834: truncate d5/db/d35/d3b/d6b/f7e 1570708 0 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:6/852: write d6/d13/d18/d19/d2c/d3f/f56 [481434,80744] 0 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:9/866: dwrite dd/d3d/d89/dd1/ffb [0,4194304] 0 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:8/681: dwrite d5/dc/f3b [0,4194304] 0 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:6/853: rmdir d6/d13/d18/d3c/ddc 39 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:2/835: rename d5/db/dd/d24/de1 to d5/db/dd/d24/d116 0 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:2/836: fsync d5/db/dd/d36/d3c/d48/d7a/d7c/f86 0 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:9/867: mknod dd/c126 0 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:6/854: rmdir d6/d13/d18/d94 39 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:6/855: chown d6/d13/d18/d19/d2c/d3f/d77 223871248 1 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:1/823: dwrite d1/dc/d16/d46/d88/dea/ff8 [0,4194304] 0 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:6/856: fdatasync d6/d13/d18/d19/f72 0 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:9/868: dwrite dd/d11/d1f/d26/dc0/fbf [4194304,4194304] 0 2024-04-05T21:26:46.216 INFO:tasks.workunit.client.1.smithi188.stdout:6/857: chown d6/d71 3074604 1 2024-04-05T21:26:46.217 INFO:tasks.workunit.client.1.smithi188.stdout:1/824: read - d1/dc/f89 zero size 2024-04-05T21:26:46.219 INFO:tasks.workunit.client.1.smithi188.stdout:3/899: dwrite d5/df/d80/fc4 [0,4194304] 0 2024-04-05T21:26:46.219 INFO:tasks.workunit.client.0.smithi121.stdout:9/447: sync 2024-04-05T21:26:46.219 INFO:tasks.workunit.client.0.smithi121.stdout:7/634: sync 2024-04-05T21:26:46.219 INFO:tasks.workunit.client.0.smithi121.stdout:6/470: sync 2024-04-05T21:26:46.219 INFO:tasks.workunit.client.0.smithi121.stdout:8/541: sync 2024-04-05T21:26:46.219 INFO:tasks.workunit.client.0.smithi121.stdout:2/493: dwrite d2/d9/d56/d62/f7a [0,4194304] 0 2024-04-05T21:26:46.219 INFO:tasks.workunit.client.0.smithi121.stdout:5/724: sync 2024-04-05T21:26:46.219 INFO:tasks.workunit.client.0.smithi121.stdout:3/487: getdents d0 0 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:6/471: chown d1/d2/dc/f5b 8 1 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:9/448: fdatasync d6/df/d1d/f33 0 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:8/542: chown d12/d13/d15/d1d/f4c 5977363 1 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:7/635: chown d0/d9/d22/d28/f31 25608 1 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:2/494: readlink d2/d9/d38/d2d/d4c/d7c/l6e 0 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:4/739: dwrite d5/d11/d28/f13 [0,4194304] 0 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:1/687: dwrite d8/d8c/db6/d1a/d1b/d4c/d76/fce [0,4194304] 0 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:2/495: chown d2/d9/d17/d3a 1929 1 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:3/488: truncate d0/d3/d24/f99 955584 0 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:9/449: symlink d6/d36/d7b/d5d/d79/d94/l98 0 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:5/725: link dd/de/d19/d38/d56/fe0 dd/de/d19/d38/d45/f107 0 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:9/450: mknod d6/d36/d7b/d49/d66/c99 0 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:4/740: symlink d5/d11/d2c/d9d/dae/d29/db4/dd4/lf0 0 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:4/741: chown f4 51907976 1 2024-04-05T21:26:46.220 INFO:tasks.workunit.client.0.smithi121.stdout:7/636: dwrite d0/d9/f86 [0,4194304] 0 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:8/543: dwrite d12/d13/d15/d1d/f36 [0,4194304] 0 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:2/496: rmdir d2/d9/d38/d2d/d47/d51 0 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:5/726: creat dd/de/d19/d38/d45/d68/dcf/f108 x:0 0 0 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:1/688: dwrite d8/d8c/db6/d1a/f71 [0,4194304] 0 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:8/544: write d12/d13/d15/d1d/fa9 [624057,80432] 0 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:7/637: dread - d0/d9/d22/d28/d56/f64 zero size 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:5/727: dread - dd/de/d19/d38/d45/d68/fc4 zero size 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:2/497: write d2/d9/d38/f35 [3768026,70424] 0 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:7/638: fdatasync d0/d9/d22/d3e/d9c/fb2 0 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:9/451: link d6/df/l58 d6/df/d48/l9a 0 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:1/689: symlink d8/d8c/db6/d1a/d1b/d44/le0 0 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:6/472: dread d1/f38 [0,4194304] 0 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:5/728: creat dd/de/d19/d3c/f109 x:0 0 0 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:6/473: truncate d1/d4b/f77 103205 0 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:5/729: dread - dd/d41/dce/da6/dd3/db1/f106 zero size 2024-04-05T21:26:46.221 INFO:tasks.workunit.client.0.smithi121.stdout:7/639: creat d0/d23/fc6 x:0 0 0 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:4/742: dwrite d5/d11/d28/d3b/fef [0,4194304] 0 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:2/498: dwrite d2/d9/d38/d2d/d4c/f78 [0,4194304] 0 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:4/743: symlink d5/d11/d2c/d9d/dae/d29/db4/dd4/lf1 0 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:5/730: symlink dd/de/d6d/l10a 0 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:2/499: dwrite d2/d28/f4f [0,4194304] 0 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:7/640: creat d0/d9/d22/d81/da2/fc7 x:0 0 0 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:5/731: mkdir dd/de/d19/d1c/d74/d10b 0 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:7/641: chown d0/d9/f95 400818 1 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:6/474: rename d1/d2/d3/d4/d21/d57/d86/d4f/c5c to d1/d2/d3/d4/d21/d57/d42/d48/d7c/ca2 0 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:5/732: fdatasync dd/de/d6d/fec 0 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:4/744: chown d5/d11/d28/f13 205 1 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:2/500: creat d2/f92 x:0 0 0 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:6/475: mkdir d1/d2/d1b/d91/da3 0 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:4/745: dread d5/d11/d28/d3b/fef [0,4194304] 0 2024-04-05T21:26:46.222 INFO:tasks.workunit.client.0.smithi121.stdout:9/452: write d6/df/d1d/f2d [4767600,5690] 0 2024-04-05T21:26:46.223 INFO:tasks.workunit.client.0.smithi121.stdout:1/690: write d8/d8c/db6/d1a/d1b/d24/f49 [4650237,36229] 0 2024-04-05T21:26:46.223 INFO:tasks.workunit.client.0.smithi121.stdout:1/691: chown d8/d8c 149056 1 2024-04-05T21:26:46.223 INFO:tasks.workunit.client.0.smithi121.stdout:9/453: write d6/df/f5f [2594964,104392] 0 2024-04-05T21:26:46.223 INFO:tasks.workunit.client.0.smithi121.stdout:7/642: getdents d0/d23/d5b 0 2024-04-05T21:26:46.223 INFO:tasks.workunit.client.0.smithi121.stdout:4/746: fsync d5/d11/d28/f5d 0 2024-04-05T21:26:46.223 INFO:tasks.workunit.client.0.smithi121.stdout:5/733: link dd/de/d19/d13/d53/lae dd/de/d19/d3c/d8e/l10c 0 2024-04-05T21:26:46.223 INFO:tasks.workunit.client.0.smithi121.stdout:2/501: creat d2/d9/d56/f93 x:0 0 0 2024-04-05T21:26:46.223 INFO:tasks.workunit.client.0.smithi121.stdout:6/476: dwrite d1/d2/d1b/f9e [0,4194304] 0 2024-04-05T21:26:46.223 INFO:tasks.workunit.client.0.smithi121.stdout:1/692: mknod d8/d89/d9e/d2d/d84/ce1 0 2024-04-05T21:26:46.223 INFO:tasks.workunit.client.0.smithi121.stdout:9/454: read d6/f17 [2041570,56863] 0 2024-04-05T21:26:46.223 INFO:tasks.workunit.client.0.smithi121.stdout:4/747: write d5/d11/d2c/d9d/dae/d29/db4/dd4/fcb [82860,44246] 0 2024-04-05T21:26:46.223 INFO:tasks.workunit.client.0.smithi121.stdout:7/643: rename d0/d9/d22/l37 to d0/d9/d22/d3e/lc8 0 2024-04-05T21:26:46.223 INFO:tasks.workunit.client.0.smithi121.stdout:6/477: symlink d1/d2/d1b/d35/la4 0 2024-04-05T21:26:46.224 INFO:tasks.workunit.client.0.smithi121.stdout:1/693: mknod d8/d8c/da2/ce2 0 2024-04-05T21:26:46.224 INFO:tasks.workunit.client.1.smithi188.stdout:5/803: dread d6/d82/f47 [0,4194304] 0 2024-04-05T21:26:46.224 INFO:tasks.workunit.client.1.smithi188.stdout:6/858: symlink d6/d13/d18/d19/da3/l102 0 2024-04-05T21:26:46.224 INFO:tasks.workunit.client.1.smithi188.stdout:5/804: symlink d6/d15/d1b/d4f/l113 0 2024-04-05T21:26:46.226 INFO:tasks.workunit.client.1.smithi188.stdout:6/859: write d6/d13/d38/f4c [3425958,57619] 0 2024-04-05T21:26:46.226 INFO:tasks.workunit.client.0.smithi121.stdout:2/502: rename d2/d9/d12 to d2/d9/d38/d2d/d47/d94 0 2024-04-05T21:26:46.226 INFO:tasks.workunit.client.1.smithi188.stdout:9/869: dwrite dd/d11/d1f/d26/d55/f63 [0,4194304] 0 2024-04-05T21:26:46.228 INFO:tasks.workunit.client.1.smithi188.stdout:9/870: write dd/d11/d1f/db2/ff5 [341751,27759] 0 2024-04-05T21:26:46.232 INFO:tasks.workunit.client.0.smithi121.stdout:5/734: dwrite dd/de/d19/d38/d45/f5c [0,4194304] 0 2024-04-05T21:26:46.235 INFO:tasks.workunit.client.1.smithi188.stdout:3/900: unlink d5/d7/d31/c98 0 2024-04-05T21:26:46.235 INFO:tasks.workunit.client.1.smithi188.stdout:5/805: symlink d6/d106/de1/l114 0 2024-04-05T21:26:46.236 INFO:tasks.workunit.client.1.smithi188.stdout:9/871: symlink dd/d11/d1f/d26/dc0/d6b/d40/d59/l127 0 2024-04-05T21:26:46.240 INFO:tasks.workunit.client.1.smithi188.stdout:3/901: chown d5/d7/f36 130 1 2024-04-05T21:26:46.241 INFO:tasks.workunit.client.1.smithi188.stdout:3/902: fsync d5/d7e/d101/fb3 0 2024-04-05T21:26:46.242 INFO:tasks.workunit.client.1.smithi188.stdout:9/872: stat dd/d11/d1f/d25/cb0 0 2024-04-05T21:26:46.242 INFO:tasks.workunit.client.0.smithi121.stdout:6/478: rename d1/d2/d1b/d35/d74/l90 to d1/d2/d3/d4/d21/d57/d42/la5 0 2024-04-05T21:26:46.242 INFO:tasks.workunit.client.0.smithi121.stdout:9/455: dwrite d6/df/d48/f4c [0,4194304] 0 2024-04-05T21:26:46.248 INFO:tasks.workunit.client.1.smithi188.stdout:6/860: creat d6/f103 x:0 0 0 2024-04-05T21:26:46.249 INFO:tasks.workunit.client.1.smithi188.stdout:5/806: rename d6/fb7 to d6/d106/de1/f115 0 2024-04-05T21:26:46.251 INFO:tasks.workunit.client.1.smithi188.stdout:1/825: dwrite d1/dc/d16/d21/d27/d4a/dc1/fce [0,4194304] 0 2024-04-05T21:26:46.251 INFO:tasks.workunit.client.1.smithi188.stdout:3/903: fdatasync d5/d7/d65/d61/d66/fc2 0 2024-04-05T21:26:46.251 INFO:tasks.workunit.client.0.smithi121.stdout:9/456: write d6/df/d1d/f2d [4839190,116178] 0 2024-04-05T21:26:46.251 INFO:tasks.workunit.client.0.smithi121.stdout:7/644: dwrite d0/d9/d22/d81/f9a [0,4194304] 0 2024-04-05T21:26:46.251 INFO:tasks.workunit.client.0.smithi121.stdout:2/503: symlink d2/d9/d38/d2d/d4c/d7c/d7e/l95 0 2024-04-05T21:26:46.252 INFO:tasks.workunit.client.0.smithi121.stdout:5/735: creat dd/de/d19/d13/f10d x:0 0 0 2024-04-05T21:26:46.254 INFO:tasks.workunit.client.0.smithi121.stdout:6/479: creat d1/d2/d1b/da1/fa6 x:0 0 0 2024-04-05T21:26:46.254 INFO:tasks.workunit.client.0.smithi121.stdout:1/694: dwrite d8/d89/d9e/da3/fbf [0,4194304] 0 2024-04-05T21:26:46.255 INFO:tasks.workunit.client.0.smithi121.stdout:0/635: sync 2024-04-05T21:26:46.256 INFO:tasks.workunit.client.1.smithi188.stdout:6/861: readlink d6/d13/d38/l39 0 2024-04-05T21:26:46.256 INFO:tasks.workunit.client.1.smithi188.stdout:6/862: stat d6/d13/d18/d19/d2c/d3f 0 2024-04-05T21:26:46.256 INFO:tasks.workunit.client.0.smithi121.stdout:2/504: symlink d2/d9/d17/l96 0 2024-04-05T21:26:46.262 INFO:tasks.workunit.client.0.smithi121.stdout:6/480: write d1/d2/d3/d5/d1f/f64 [767185,95710] 0 2024-04-05T21:26:46.263 INFO:tasks.workunit.client.1.smithi188.stdout:8/682: dwrite d5/d6e/d86/fb0 [0,4194304] 0 2024-04-05T21:26:46.263 INFO:tasks.workunit.client.1.smithi188.stdout:8/683: fdatasync d5/dc/f2c 0 2024-04-05T21:26:46.263 INFO:tasks.workunit.client.1.smithi188.stdout:8/684: readlink d5/d21/d40/d43/d44/d5b/d99/d69/d9b/lbf 0 2024-04-05T21:26:46.264 INFO:tasks.workunit.client.0.smithi121.stdout:6/481: chown d1/d2/d3/d5/d1f/f6a 5734707 1 2024-04-05T21:26:46.264 INFO:tasks.workunit.client.1.smithi188.stdout:9/873: mknod dd/d11/d1f/d26/d55/d98/c128 0 2024-04-05T21:26:46.264 INFO:tasks.workunit.client.1.smithi188.stdout:3/904: mkdir d5/d7/d65/d61/d12b 0 2024-04-05T21:26:46.265 INFO:tasks.workunit.client.1.smithi188.stdout:6/863: creat d6/d13/d18/d19/d2c/d3f/d95/f104 x:0 0 0 2024-04-05T21:26:46.265 INFO:tasks.workunit.client.1.smithi188.stdout:1/826: creat d1/dc/d16/d46/d95/d99/f102 x:0 0 0 2024-04-05T21:26:46.265 INFO:tasks.workunit.client.0.smithi121.stdout:7/645: mknod d0/d32/d7c/cc9 0 2024-04-05T21:26:46.266 INFO:tasks.workunit.client.0.smithi121.stdout:2/505: mknod d2/d9/d1c/c97 0 2024-04-05T21:26:46.266 INFO:tasks.workunit.client.0.smithi121.stdout:1/695: chown d8/d8c/db6/l9f 3009 1 2024-04-05T21:26:46.266 INFO:tasks.workunit.client.0.smithi121.stdout:0/636: mkdir d7/d71/d29/de1 0 2024-04-05T21:26:46.267 INFO:tasks.workunit.client.1.smithi188.stdout:8/685: write d5/d21/d80/fb2 [461857,100550] 0 2024-04-05T21:26:46.268 INFO:tasks.workunit.client.0.smithi121.stdout:7/646: creat d0/d32/d4e/d73/fca x:0 0 0 2024-04-05T21:26:46.274 INFO:tasks.workunit.client.1.smithi188.stdout:8/686: dread - d5/d21/d2a/d95/fdf zero size 2024-04-05T21:26:46.275 INFO:tasks.workunit.client.1.smithi188.stdout:2/837: dread d5/db/dd/da4/fc2 [0,4194304] 0 2024-04-05T21:26:46.296 INFO:tasks.workunit.client.0.smithi121.stdout:5/736: read dd/de/d19/d13/f35 [1458966,14506] 0 2024-04-05T21:26:46.297 INFO:tasks.workunit.client.0.smithi121.stdout:2/506: symlink d2/d9/d38/d2d/l98 0 2024-04-05T21:26:46.325 INFO:tasks.workunit.client.0.smithi121.stdout:7/647: mkdir d0/d23/d5b/d52/d61/dcb 0 2024-04-05T21:26:46.326 INFO:tasks.workunit.client.0.smithi121.stdout:5/737: creat dd/de/d19/d1c/d74/f10e x:0 0 0 2024-04-05T21:26:46.329 INFO:tasks.workunit.client.0.smithi121.stdout:6/482: dread d1/d2/f13 [0,4194304] 0 2024-04-05T21:26:46.330 INFO:tasks.workunit.client.1.smithi188.stdout:1/827: mknod d1/db2/c103 0 2024-04-05T21:26:46.330 INFO:tasks.workunit.client.0.smithi121.stdout:6/483: write d1/f72 [378940,17932] 0 2024-04-05T21:26:46.332 INFO:tasks.workunit.client.0.smithi121.stdout:6/484: write d1/d2/d1b/f9e [4510364,60164] 0 2024-04-05T21:26:46.332 INFO:tasks.workunit.client.0.smithi121.stdout:0/637: creat d7/d62/fe2 x:0 0 0 2024-04-05T21:26:46.334 INFO:tasks.workunit.client.0.smithi121.stdout:9/457: truncate d6/df/d48/f4c 3629659 0 2024-04-05T21:26:46.348 INFO:tasks.workunit.client.0.smithi121.stdout:9/458: creat d6/f9b x:0 0 0 2024-04-05T21:26:46.350 INFO:tasks.workunit.client.0.smithi121.stdout:9/459: write d6/d36/d7b/d49/f8b [533530,65622] 0 2024-04-05T21:26:46.352 INFO:tasks.workunit.client.0.smithi121.stdout:0/638: creat d7/d62/d3c/fe3 x:0 0 0 2024-04-05T21:26:46.355 INFO:tasks.workunit.client.0.smithi121.stdout:9/460: mkdir d6/d36/d7b/d5d/d79/d9c 0 2024-04-05T21:26:46.356 INFO:tasks.workunit.client.0.smithi121.stdout:0/639: readlink d7/d1c/l75 0 2024-04-05T21:26:46.358 INFO:tasks.workunit.client.1.smithi188.stdout:1/828: symlink d1/dc/dfc/l104 0 2024-04-05T21:26:46.358 INFO:tasks.workunit.client.0.smithi121.stdout:0/640: dread d7/d71/d94/fd7 [0,4194304] 0 2024-04-05T21:26:46.359 INFO:tasks.workunit.client.1.smithi188.stdout:1/829: write d1/dc/d16/d2d/fd1 [3591219,55405] 0 2024-04-05T21:26:46.360 INFO:tasks.workunit.client.0.smithi121.stdout:0/641: write d7/d71/d29/d2a/d2c/f87 [529886,47945] 0 2024-04-05T21:26:46.360 INFO:tasks.workunit.client.1.smithi188.stdout:1/830: dread - d1/dc/d16/d2d/f97 zero size 2024-04-05T21:26:46.361 INFO:tasks.workunit.client.1.smithi188.stdout:5/807: chown d6/d15/d5c/d7a/c69 3 1 2024-04-05T21:26:46.365 INFO:tasks.workunit.client.1.smithi188.stdout:0/707: sync 2024-04-05T21:26:46.369 INFO:tasks.workunit.client.0.smithi121.stdout:1/696: rename f2 to d8/d89/d9e/d57/fe3 0 2024-04-05T21:26:46.375 INFO:tasks.workunit.client.1.smithi188.stdout:5/808: dwrite d6/d15/d1b/f34 [0,4194304] 0 2024-04-05T21:26:46.378 INFO:tasks.workunit.client.1.smithi188.stdout:1/831: rename d1/dc/d16/d21/c78 to d1/dc/d16/d2d/db0/c105 0 2024-04-05T21:26:46.379 INFO:tasks.workunit.client.0.smithi121.stdout:7/648: write d0/f1f [705193,2294] 0 2024-04-05T21:26:46.379 INFO:tasks.workunit.client.0.smithi121.stdout:9/461: unlink d6/df/d48/f4c 0 2024-04-05T21:26:46.380 INFO:tasks.workunit.client.1.smithi188.stdout:1/832: stat d1/dc/d16/d46/d88/d13/f82 0 2024-04-05T21:26:46.389 INFO:tasks.workunit.client.1.smithi188.stdout:1/833: rmdir d1/d7/dbf 39 2024-04-05T21:26:46.395 INFO:tasks.workunit.client.0.smithi121.stdout:6/485: dwrite d1/d2/d3/d5/d1f/f27 [0,4194304] 0 2024-04-05T21:26:46.397 INFO:tasks.workunit.client.0.smithi121.stdout:7/649: dread d0/d32/d7c/fac [0,4194304] 0 2024-04-05T21:26:46.400 INFO:tasks.workunit.client.0.smithi121.stdout:0/642: getdents d7/d62/d3c 0 2024-04-05T21:26:46.404 INFO:tasks.workunit.client.1.smithi188.stdout:0/708: dread d4/d8/d10/f1e [0,4194304] 0 2024-04-05T21:26:46.405 INFO:tasks.workunit.client.1.smithi188.stdout:0/709: write d4/d6/d46/dba/fdb [458068,61518] 0 2024-04-05T21:26:46.407 INFO:tasks.workunit.client.0.smithi121.stdout:0/643: chown d7/d71/d45/d5a/f69 319505333 1 2024-04-05T21:26:46.410 INFO:tasks.workunit.client.1.smithi188.stdout:6/864: write d6/d13/d18/d19/d61/fab [996459,104254] 0 2024-04-05T21:26:46.411 INFO:tasks.workunit.client.1.smithi188.stdout:9/874: truncate fb 5608897 0 2024-04-05T21:26:46.411 INFO:tasks.workunit.client.0.smithi121.stdout:1/697: creat d8/d8c/fe4 x:0 0 0 2024-04-05T21:26:46.412 INFO:tasks.workunit.client.1.smithi188.stdout:0/710: rename d4/d8/db/d21/d4a/l89 to d4/d6/d13/dea/l106 0 2024-04-05T21:26:46.412 INFO:tasks.workunit.client.0.smithi121.stdout:2/507: symlink d2/d9/d1c/l99 0 2024-04-05T21:26:46.413 INFO:tasks.workunit.client.0.smithi121.stdout:1/698: chown d8/d89/d9e/d57/f60 19093241 1 2024-04-05T21:26:46.414 INFO:tasks.workunit.client.0.smithi121.stdout:9/462: rmdir d6/d36/d7b/d5d/d79 39 2024-04-05T21:26:46.415 INFO:tasks.workunit.client.1.smithi188.stdout:6/865: mkdir d6/d105 0 2024-04-05T21:26:46.415 INFO:tasks.workunit.client.1.smithi188.stdout:9/875: truncate dd/d3d/f6c 179179 0 2024-04-05T21:26:46.416 INFO:tasks.workunit.client.0.smithi121.stdout:6/486: mkdir d1/d4b/da7 0 2024-04-05T21:26:46.417 INFO:tasks.workunit.client.1.smithi188.stdout:3/905: dwrite d5/d7/d6d/dd8/f1f [4194304,4194304] 0 2024-04-05T21:26:46.418 INFO:tasks.workunit.client.1.smithi188.stdout:9/876: chown dd/d11/d1f/f10a 96121 1 2024-04-05T21:26:46.419 INFO:tasks.workunit.client.0.smithi121.stdout:7/650: rename d0/d9/d22/d3e/la1 to d0/d9/d22/d28/lcc 0 2024-04-05T21:26:46.419 INFO:tasks.workunit.client.1.smithi188.stdout:3/906: chown d5/d48/c50 88 1 2024-04-05T21:26:46.420 INFO:tasks.workunit.client.1.smithi188.stdout:8/687: creat d5/d21/fe0 x:0 0 0 2024-04-05T21:26:46.420 INFO:tasks.workunit.client.1.smithi188.stdout:0/711: truncate d4/d8/db/fe5 362030 0 2024-04-05T21:26:46.422 INFO:tasks.workunit.client.1.smithi188.stdout:6/866: dread - d6/d13/d18/d3c/ddc/fa7 zero size 2024-04-05T21:26:46.422 INFO:tasks.workunit.client.1.smithi188.stdout:9/877: rmdir dd/d11/d73/d85 39 2024-04-05T21:26:46.423 INFO:tasks.workunit.client.0.smithi121.stdout:9/463: creat d6/df/d1d/d2c/d37/d3c/f9d x:0 0 0 2024-04-05T21:26:46.424 INFO:tasks.workunit.client.1.smithi188.stdout:8/688: symlink d5/d21/d80/le1 0 2024-04-05T21:26:46.425 INFO:tasks.workunit.client.1.smithi188.stdout:8/689: fsync d5/d21/d2a/d47/f54 0 2024-04-05T21:26:46.428 INFO:tasks.workunit.client.1.smithi188.stdout:6/867: creat d6/d28/d45/f106 x:0 0 0 2024-04-05T21:26:46.429 INFO:tasks.workunit.client.0.smithi121.stdout:6/487: rename d1/d2/d1b/d1e/l96 to d1/d2/d3/d4/d21/d57/d42/d48/d7c/la8 0 2024-04-05T21:26:46.431 INFO:tasks.workunit.client.0.smithi121.stdout:2/508: truncate d2/d9/d1c/d30/f61 3119520 0 2024-04-05T21:26:46.432 INFO:tasks.workunit.client.0.smithi121.stdout:1/699: mknod d8/d8c/db6/d1a/d1b/d38/ce5 0 2024-04-05T21:26:46.434 INFO:tasks.workunit.client.0.smithi121.stdout:6/488: creat d1/d2/d3/d4/d21/d57/d42/d48/d7c/fa9 x:0 0 0 2024-04-05T21:26:46.435 INFO:tasks.workunit.client.0.smithi121.stdout:0/644: link d7/d62/lb8 d7/d71/d29/d3f/le4 0 2024-04-05T21:26:46.436 INFO:tasks.workunit.client.1.smithi188.stdout:3/907: truncate d5/d7/d65/d61/d66/f9f 4053018 0 2024-04-05T21:26:46.436 INFO:tasks.workunit.client.1.smithi188.stdout:1/834: dread d1/f14 [0,4194304] 0 2024-04-05T21:26:46.436 INFO:tasks.workunit.client.1.smithi188.stdout:8/690: creat d5/dc/d8f/dd1/fe2 x:0 0 0 2024-04-05T21:26:46.437 INFO:tasks.workunit.client.0.smithi121.stdout:9/464: mknod d6/df/d1d/c9e 0 2024-04-05T21:26:46.438 INFO:tasks.workunit.client.0.smithi121.stdout:1/700: creat d8/d8c/db6/d1a/d1b/d4c/da1/fe6 x:0 0 0 2024-04-05T21:26:46.439 INFO:tasks.workunit.client.1.smithi188.stdout:0/712: dread d4/d8/d10/f1b [0,4194304] 0 2024-04-05T21:26:46.440 INFO:tasks.workunit.client.1.smithi188.stdout:3/908: chown d5/d7/f36 1539347 1 2024-04-05T21:26:46.441 INFO:tasks.workunit.client.0.smithi121.stdout:9/465: mknod d6/df/d1d/d2c/d53/c9f 0 2024-04-05T21:26:46.442 INFO:tasks.workunit.client.1.smithi188.stdout:3/909: chown d5/ddd/fcc 2032844 1 2024-04-05T21:26:46.443 INFO:tasks.workunit.client.0.smithi121.stdout:0/645: creat d7/d71/d29/de1/fe5 x:0 0 0 2024-04-05T21:26:46.443 INFO:tasks.workunit.client.1.smithi188.stdout:8/691: creat d5/d21/d40/d43/d44/d5b/d99/d69/fe3 x:0 0 0 2024-04-05T21:26:46.444 INFO:tasks.workunit.client.0.smithi121.stdout:0/646: write d7/d7e/f89 [3893047,126847] 0 2024-04-05T21:26:46.444 INFO:tasks.workunit.client.0.smithi121.stdout:1/701: write d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b/fd2 [3728928,60587] 0 2024-04-05T21:26:46.447 INFO:tasks.workunit.client.1.smithi188.stdout:1/835: dwrite d1/dc/d16/d46/df7/d5f/f76 [0,4194304] 0 2024-04-05T21:26:46.447 INFO:tasks.workunit.client.1.smithi188.stdout:0/713: unlink d4/d6/d46/fe7 0 2024-04-05T21:26:46.449 INFO:tasks.workunit.client.0.smithi121.stdout:0/647: mkdir d7/d71/d45/d5a/de6 0 2024-04-05T21:26:46.449 INFO:tasks.workunit.client.1.smithi188.stdout:6/868: creat d6/d13/d18/d19/f107 x:0 0 0 2024-04-05T21:26:46.450 INFO:tasks.workunit.client.0.smithi121.stdout:9/466: rename d6/d36/d7b/d49/d66/c8d to d6/d81/ca0 0 2024-04-05T21:26:46.451 INFO:tasks.workunit.client.1.smithi188.stdout:6/869: chown d6/d13/d18/d19/d2c/db2 1434 1 2024-04-05T21:26:46.452 INFO:tasks.workunit.client.0.smithi121.stdout:1/702: mknod d8/d8c/db6/d1a/d1b/d4c/ce7 0 2024-04-05T21:26:46.452 INFO:tasks.workunit.client.1.smithi188.stdout:1/836: mknod d1/dc/d16/d46/c106 0 2024-04-05T21:26:46.455 INFO:tasks.workunit.client.1.smithi188.stdout:8/692: creat d5/d21/d40/d43/d48/d7e/fe4 x:0 0 0 2024-04-05T21:26:46.455 INFO:tasks.workunit.client.0.smithi121.stdout:0/648: rmdir d7/d71/d45/d46 39 2024-04-05T21:26:46.460 INFO:tasks.workunit.client.1.smithi188.stdout:0/714: dwrite d4/d8/d10/f67 [0,4194304] 0 2024-04-05T21:26:46.461 INFO:tasks.workunit.client.1.smithi188.stdout:5/809: symlink d6/d15/d5c/d7a/d44/l116 0 2024-04-05T21:26:46.461 INFO:tasks.workunit.client.0.smithi121.stdout:9/467: mknod d6/df/d1d/d72/ca1 0 2024-04-05T21:26:46.461 INFO:tasks.workunit.client.0.smithi121.stdout:9/468: chown d6/df/d1d/d72/l74 337832643 1 2024-04-05T21:26:46.463 INFO:tasks.workunit.client.0.smithi121.stdout:9/469: write d6/df/d48/f62 [748755,109907] 0 2024-04-05T21:26:46.464 INFO:tasks.workunit.client.0.smithi121.stdout:0/649: rmdir d7/dc 39 2024-04-05T21:26:46.465 INFO:tasks.workunit.client.1.smithi188.stdout:6/870: truncate d6/d13/d18/d19/d61/f8b 62940 0 2024-04-05T21:26:46.465 INFO:tasks.workunit.client.1.smithi188.stdout:3/910: creat d5/d7/d65/d61/d66/da4/dfd/f12c x:0 0 0 2024-04-05T21:26:46.465 INFO:tasks.workunit.client.1.smithi188.stdout:2/838: creat d5/db/d35/f117 x:0 0 0 2024-04-05T21:26:46.466 INFO:tasks.workunit.client.0.smithi121.stdout:1/703: symlink d8/d89/d9e/dc8/le8 0 2024-04-05T21:26:46.468 INFO:tasks.workunit.client.0.smithi121.stdout:2/509: dread d2/d1e/f24 [0,4194304] 0 2024-04-05T21:26:46.470 INFO:tasks.workunit.client.0.smithi121.stdout:1/704: write d8/d8c/db6/d1a/d1b/d4c/f4f [3880652,116964] 0 2024-04-05T21:26:46.471 INFO:tasks.workunit.client.0.smithi121.stdout:9/470: rename d6/df/d1d/c9e to d6/d46/ca2 0 2024-04-05T21:26:46.473 INFO:tasks.workunit.client.0.smithi121.stdout:7/651: link d0/d9/d22/d28/d56/c96 d0/d23/d5b/d52/d61/dcb/ccd 0 2024-04-05T21:26:46.473 INFO:tasks.workunit.client.1.smithi188.stdout:6/871: dread - d6/d13/d18/d94/ff8 zero size 2024-04-05T21:26:46.473 INFO:tasks.workunit.client.0.smithi121.stdout:6/489: dread d1/d2/dc/f19 [0,4194304] 0 2024-04-05T21:26:46.475 INFO:tasks.workunit.client.0.smithi121.stdout:0/650: unlink d7/d71/d45/d46/l6c 0 2024-04-05T21:26:46.475 INFO:tasks.workunit.client.0.smithi121.stdout:6/490: write d1/d2/dc/f4c [5256992,64717] 0 2024-04-05T21:26:46.476 INFO:tasks.workunit.client.0.smithi121.stdout:2/510: creat d2/d9/d17/d3a/f9a x:0 0 0 2024-04-05T21:26:46.476 INFO:tasks.workunit.client.1.smithi188.stdout:6/872: readlink l4 0 2024-04-05T21:26:46.477 INFO:tasks.workunit.client.0.smithi121.stdout:1/705: creat d8/d8c/db6/d1a/d1b/d24/fe9 x:0 0 0 2024-04-05T21:26:46.478 INFO:tasks.workunit.client.0.smithi121.stdout:2/511: write d2/d9/f75 [580743,94329] 0 2024-04-05T21:26:46.478 INFO:tasks.workunit.client.1.smithi188.stdout:0/715: dwrite d4/d8/db/d21/fe2 [0,4194304] 0 2024-04-05T21:26:46.479 INFO:tasks.workunit.client.0.smithi121.stdout:7/652: unlink d0/d23/d5b/fc5 0 2024-04-05T21:26:46.479 INFO:tasks.workunit.client.0.smithi121.stdout:6/491: chown d1/d2/d3/d5/f66 1288 1 2024-04-05T21:26:46.480 INFO:tasks.workunit.client.1.smithi188.stdout:8/693: link d5/d21/d40/d43/d48/d7e/f93 d5/d21/d40/d43/d44/d59/d8b/fe5 0 2024-04-05T21:26:46.481 INFO:tasks.workunit.client.1.smithi188.stdout:1/837: link d1/dc/d16/d46/d88/d3f/fdb d1/d7/f107 0 2024-04-05T21:26:46.482 INFO:tasks.workunit.client.0.smithi121.stdout:6/492: chown d1/d2/d3/d4/d21/d57/d42/d48/l7e 2 1 2024-04-05T21:26:46.483 INFO:tasks.workunit.client.1.smithi188.stdout:8/694: chown d5/d21/lc3 24 1 2024-04-05T21:26:46.487 INFO:tasks.workunit.client.1.smithi188.stdout:1/838: chown d1/dc/d16/d21/fa0 283559 1 2024-04-05T21:26:46.490 INFO:tasks.workunit.client.1.smithi188.stdout:8/695: chown d5/d21/d2a/d47/c6d 16455714 1 2024-04-05T21:26:46.490 INFO:tasks.workunit.client.1.smithi188.stdout:1/839: fdatasync d1/db2/fb4 0 2024-04-05T21:26:46.491 INFO:tasks.workunit.client.0.smithi121.stdout:0/651: link d7/d62/d9d/fd6 d7/d62/d9d/da6/fe7 0 2024-04-05T21:26:46.491 INFO:tasks.workunit.client.0.smithi121.stdout:0/652: write d7/f84 [1394324,68635] 0 2024-04-05T21:26:46.491 INFO:tasks.workunit.client.0.smithi121.stdout:7/653: getdents d0/d9/d22/d28/d56/db9 0 2024-04-05T21:26:46.491 INFO:tasks.workunit.client.0.smithi121.stdout:3/489: sync 2024-04-05T21:26:46.492 INFO:tasks.workunit.client.1.smithi188.stdout:2/839: dwrite d5/db/d35/d3b/f84 [0,4194304] 0 2024-04-05T21:26:46.492 INFO:tasks.workunit.client.1.smithi188.stdout:3/911: dwrite d5/d7/d65/d5f/d102/f116 [0,4194304] 0 2024-04-05T21:26:46.493 INFO:tasks.workunit.client.0.smithi121.stdout:1/706: creat d8/fea x:0 0 0 2024-04-05T21:26:46.494 INFO:tasks.workunit.client.0.smithi121.stdout:7/654: read d0/d9/d22/d81/f9a [3716678,115161] 0 2024-04-05T21:26:46.496 INFO:tasks.workunit.client.0.smithi121.stdout:9/471: dread d6/df/d48/f62 [0,4194304] 0 2024-04-05T21:26:46.496 INFO:tasks.workunit.client.1.smithi188.stdout:3/912: dread - d5/d7/d65/d5f/f10e zero size 2024-04-05T21:26:46.496 INFO:tasks.workunit.client.0.smithi121.stdout:3/490: dread - d0/d3/d24/f89 zero size 2024-04-05T21:26:46.501 INFO:tasks.workunit.client.0.smithi121.stdout:7/655: write d0/d23/f33 [864741,12878] 0 2024-04-05T21:26:46.501 INFO:tasks.workunit.client.0.smithi121.stdout:3/491: dread - d0/d3/de/f9c zero size 2024-04-05T21:26:46.503 INFO:tasks.workunit.client.1.smithi188.stdout:5/810: write d6/d82/d40/f65 [239953,79188] 0 2024-04-05T21:26:46.504 INFO:tasks.workunit.client.1.smithi188.stdout:6/873: link d6/d13/d18/d19/d2c/d3f/d95/fe2 d6/d13/d18/d19/d2c/d3f/d95/f108 0 2024-04-05T21:26:46.506 INFO:tasks.workunit.client.1.smithi188.stdout:1/840: symlink d1/dc/d16/d21/d27/d4a/l108 0 2024-04-05T21:26:46.507 INFO:tasks.workunit.client.0.smithi121.stdout:0/653: dwrite d7/d71/d29/d2a/d2c/f87 [0,4194304] 0 2024-04-05T21:26:46.508 INFO:tasks.workunit.client.1.smithi188.stdout:8/696: symlink d5/d21/d40/d43/d44/d59/d8b/da4/db1/le6 0 2024-04-05T21:26:46.509 INFO:tasks.workunit.client.0.smithi121.stdout:3/492: truncate d0/d3/d24/d6d/f81 1387345 0 2024-04-05T21:26:46.509 INFO:tasks.workunit.client.0.smithi121.stdout:7/656: dread - d0/d23/f7f zero size 2024-04-05T21:26:46.510 INFO:tasks.workunit.client.1.smithi188.stdout:2/840: unlink d5/d5f/f115 0 2024-04-05T21:26:46.512 INFO:tasks.workunit.client.1.smithi188.stdout:8/697: truncate d5/d4b/fce 735622 0 2024-04-05T21:26:46.512 INFO:tasks.workunit.client.1.smithi188.stdout:5/811: creat d6/d82/d40/d42/d81/d10c/f117 x:0 0 0 2024-04-05T21:26:46.514 INFO:tasks.workunit.client.1.smithi188.stdout:1/841: fsync d1/dc/d16/d2d/f97 0 2024-04-05T21:26:46.514 INFO:tasks.workunit.client.0.smithi121.stdout:7/657: dread - d0/d9/f95 zero size 2024-04-05T21:26:46.514 INFO:tasks.workunit.client.0.smithi121.stdout:6/493: dread d1/d2/d1b/d1e/f2d [0,4194304] 0 2024-04-05T21:26:46.515 INFO:tasks.workunit.client.0.smithi121.stdout:9/472: creat d6/df/d1d/fa3 x:0 0 0 2024-04-05T21:26:46.518 INFO:tasks.workunit.client.0.smithi121.stdout:2/512: truncate d2/d9/d1c/d30/f68 3648015 0 2024-04-05T21:26:46.519 INFO:tasks.workunit.client.0.smithi121.stdout:1/707: dwrite d8/d8c/db6/d1a/d1b/f65 [0,4194304] 0 2024-04-05T21:26:46.520 INFO:tasks.workunit.client.0.smithi121.stdout:3/493: chown d0/d3/de/l18 82 1 2024-04-05T21:26:46.520 INFO:tasks.workunit.client.1.smithi188.stdout:6/874: rename d6/d7/fdd to d6/d13/d18/d3c/ddc/d60/f109 0 2024-04-05T21:26:46.520 INFO:tasks.workunit.client.1.smithi188.stdout:8/698: fsync d5/f24 0 2024-04-05T21:26:46.522 INFO:tasks.workunit.client.0.smithi121.stdout:2/513: truncate d2/d9/f54 2428732 0 2024-04-05T21:26:46.525 INFO:tasks.workunit.client.1.smithi188.stdout:8/699: write d5/d21/d2a/d95/dc0/fc9 [813930,102056] 0 2024-04-05T21:26:46.525 INFO:tasks.workunit.client.1.smithi188.stdout:9/878: dread dd/d11/d1f/d25/d51/fc2 [0,4194304] 0 2024-04-05T21:26:46.525 INFO:tasks.workunit.client.1.smithi188.stdout:1/842: stat d1/dc/d16/d46/d88/c35 0 2024-04-05T21:26:46.525 INFO:tasks.workunit.client.1.smithi188.stdout:6/875: read d6/d13/d18/d19/d61/fab [130426,51084] 0 2024-04-05T21:26:46.526 INFO:tasks.workunit.client.0.smithi121.stdout:0/654: rmdir d7/d71/d45/d5a/de6 0 2024-04-05T21:26:46.530 INFO:tasks.workunit.client.0.smithi121.stdout:2/514: rename d2/d9/d1c/d85/l87 to d2/d9/d1c/d85/l9b 0 2024-04-05T21:26:46.531 INFO:tasks.workunit.client.1.smithi188.stdout:1/843: unlink d1/dc/f28 0 2024-04-05T21:26:46.531 INFO:tasks.workunit.client.0.smithi121.stdout:6/494: write d1/d2/d3/d5/d1f/f46 [2800993,73373] 0 2024-04-05T21:26:46.532 INFO:tasks.workunit.client.0.smithi121.stdout:9/473: mknod d6/d36/d7b/d5d/d79/d94/ca4 0 2024-04-05T21:26:46.533 INFO:tasks.workunit.client.0.smithi121.stdout:1/708: dwrite d8/d8c/db6/d1a/f71 [0,4194304] 0 2024-04-05T21:26:46.534 INFO:tasks.workunit.client.0.smithi121.stdout:2/515: rmdir d2/d9/d1c/d85 39 2024-04-05T21:26:46.535 INFO:tasks.workunit.client.0.smithi121.stdout:0/655: write d7/dc/d95/fd3 [4709630,129814] 0 2024-04-05T21:26:46.539 INFO:tasks.workunit.client.0.smithi121.stdout:2/516: write d2/f92 [357928,43597] 0 2024-04-05T21:26:46.540 INFO:tasks.workunit.client.0.smithi121.stdout:3/494: dwrite d0/d3/f5b [0,4194304] 0 2024-04-05T21:26:46.542 INFO:tasks.workunit.client.1.smithi188.stdout:6/876: rename d6/d13/d18/d19/d2c/d3f/l8a to d6/d13/d18/d19/d2c/db2/l10a 0 2024-04-05T21:26:46.542 INFO:tasks.workunit.client.1.smithi188.stdout:8/700: dwrite d5/d21/d2a/d47/fc5 [0,4194304] 0 2024-04-05T21:26:46.544 INFO:tasks.workunit.client.0.smithi121.stdout:9/474: write d6/df/d1d/f23 [568295,49752] 0 2024-04-05T21:26:46.545 INFO:tasks.workunit.client.1.smithi188.stdout:9/879: truncate dd/d11/d1f/d26/dc0/d6b/f96 124577 0 2024-04-05T21:26:46.545 INFO:tasks.workunit.client.1.smithi188.stdout:8/701: chown d5/d21/d2a/d95/dc0 3988 1 2024-04-05T21:26:46.548 INFO:tasks.workunit.client.0.smithi121.stdout:3/495: read d0/f27 [424192,113260] 0 2024-04-05T21:26:46.549 INFO:tasks.workunit.client.0.smithi121.stdout:1/709: symlink d8/leb 0 2024-04-05T21:26:46.551 INFO:tasks.workunit.client.0.smithi121.stdout:6/495: dwrite d1/d2/d3/d5/d1f/f6a [0,4194304] 0 2024-04-05T21:26:46.552 INFO:tasks.workunit.client.0.smithi121.stdout:2/517: creat d2/d9/d38/d2d/d4c/d79/f9c x:0 0 0 2024-04-05T21:26:46.553 INFO:tasks.workunit.client.0.smithi121.stdout:1/710: readlink d8/d8c/db6/d1a/d1b/d24/l4a 0 2024-04-05T21:26:46.555 INFO:tasks.workunit.client.0.smithi121.stdout:9/475: rmdir d6/d36/d7b/d5d/d7d 0 2024-04-05T21:26:46.556 INFO:tasks.workunit.client.1.smithi188.stdout:3/913: dwrite d5/d7/d65/d5f/f120 [0,4194304] 0 2024-04-05T21:26:46.557 INFO:tasks.workunit.client.0.smithi121.stdout:1/711: chown d8/d89/d9e/d2d/d84/da8/dad 968829021 1 2024-04-05T21:26:46.558 INFO:tasks.workunit.client.0.smithi121.stdout:6/496: readlink d1/d2/d1b/d2e/l7a 0 2024-04-05T21:26:46.559 INFO:tasks.workunit.client.1.smithi188.stdout:0/716: dread d4/d8/f75 [0,4194304] 0 2024-04-05T21:26:46.560 INFO:tasks.workunit.client.1.smithi188.stdout:2/841: dwrite d5/db/dd/d73/d109/d66/d8f/fc1 [0,4194304] 0 2024-04-05T21:26:46.561 INFO:tasks.workunit.client.1.smithi188.stdout:6/877: rename d6/d13/d18/d19/f30 to d6/d71/f10b 0 2024-04-05T21:26:46.562 INFO:tasks.workunit.client.1.smithi188.stdout:8/702: stat d5/d21/c5e 0 2024-04-05T21:26:46.563 INFO:tasks.workunit.client.1.smithi188.stdout:3/914: unlink d5/d7/d6d/d75/d82/fff 0 2024-04-05T21:26:46.566 INFO:tasks.workunit.client.0.smithi121.stdout:3/496: dwrite d0/d3/f7f [0,4194304] 0 2024-04-05T21:26:46.572 INFO:tasks.workunit.client.0.smithi121.stdout:1/712: mknod d8/d89/d9e/d2d/d84/da8/dad/cec 0 2024-04-05T21:26:46.573 INFO:tasks.workunit.client.0.smithi121.stdout:2/518: dwrite d2/d9/d38/d2d/d47/d94/f37 [0,4194304] 0 2024-04-05T21:26:46.574 INFO:tasks.workunit.client.0.smithi121.stdout:9/476: rename d6/df/d1d/d2c/f60 to d6/d36/d7b/d5d/d85/fa5 0 2024-04-05T21:26:46.575 INFO:tasks.workunit.client.1.smithi188.stdout:5/812: dwrite d6/d17/d23/f7e [0,4194304] 0 2024-04-05T21:26:46.576 INFO:tasks.workunit.client.0.smithi121.stdout:7/658: dread d0/d9/d22/d28/f31 [0,4194304] 0 2024-04-05T21:26:46.576 INFO:tasks.workunit.client.0.smithi121.stdout:2/519: write d2/d9/d38/d2d/d47/f70 [1380852,103343] 0 2024-04-05T21:26:46.577 INFO:tasks.workunit.client.0.smithi121.stdout:3/497: rmdir d0/d3/de 39 2024-04-05T21:26:46.578 INFO:tasks.workunit.client.1.smithi188.stdout:9/880: truncate dd/d11/d1f/d25/d8a/d101/ffc 5166566 0 2024-04-05T21:26:46.579 INFO:tasks.workunit.client.0.smithi121.stdout:3/498: readlink d0/d3/d24/d3d/l47 0 2024-04-05T21:26:46.581 INFO:tasks.workunit.client.1.smithi188.stdout:0/717: dread - d4/d6/d13/d23/d24/d57/d63/fbd zero size 2024-04-05T21:26:46.584 INFO:tasks.workunit.client.0.smithi121.stdout:7/659: creat d0/d23/d5b/d52/d61/fce x:0 0 0 2024-04-05T21:26:46.584 INFO:tasks.workunit.client.1.smithi188.stdout:0/718: chown d4/d8/d39/cd4 450267 1 2024-04-05T21:26:46.585 INFO:tasks.workunit.client.0.smithi121.stdout:7/660: write d0/d32/f6b [4954503,96702] 0 2024-04-05T21:26:46.586 INFO:tasks.workunit.client.0.smithi121.stdout:3/499: mkdir d0/d3/d24/d3d/d1a/d23/d90/d9e 0 2024-04-05T21:26:46.587 INFO:tasks.workunit.client.0.smithi121.stdout:1/713: rename d8/d8c/db6/d1a/d1b/d85/d9c/cc3 to d8/d89/d9e/d57/d77/ced 0 2024-04-05T21:26:46.590 INFO:tasks.workunit.client.1.smithi188.stdout:2/842: dread - d5/db/dd/d36/d3c/fa2 zero size 2024-04-05T21:26:46.595 INFO:tasks.workunit.client.0.smithi121.stdout:2/520: rename d2/d9/d17/d3a/c6a to d2/d9/d56/c9d 0 2024-04-05T21:26:46.596 INFO:tasks.workunit.client.1.smithi188.stdout:3/915: unlink d5/d39/ld4 0 2024-04-05T21:26:46.597 INFO:tasks.workunit.client.0.smithi121.stdout:2/521: readlink d2/d9/d1c/d85/l8a 0 2024-04-05T21:26:46.598 INFO:tasks.workunit.client.0.smithi121.stdout:3/500: link d0/d3/d24/d3d/d1a/l59 d0/d3/de/d83/l9f 0 2024-04-05T21:26:46.600 INFO:tasks.workunit.client.0.smithi121.stdout:2/522: write d2/d9/d38/d2d/d4c/d7c/f8b [549337,23698] 0 2024-04-05T21:26:46.601 INFO:tasks.workunit.client.0.smithi121.stdout:3/501: fsync d0/d3/d24/d3d/d1a/d23/f6b 0 2024-04-05T21:26:46.604 INFO:tasks.workunit.client.0.smithi121.stdout:7/661: dread d0/d32/f35 [0,4194304] 0 2024-04-05T21:26:46.605 INFO:tasks.workunit.client.0.smithi121.stdout:2/523: symlink d2/d9/d38/d2d/d47/d94/d84/l9e 0 2024-04-05T21:26:46.605 INFO:tasks.workunit.client.1.smithi188.stdout:9/881: symlink dd/d11/d1f/d26/dc0/d6b/d40/d59/l129 0 2024-04-05T21:26:46.605 INFO:tasks.workunit.client.1.smithi188.stdout:9/882: stat dd/d11/d22/f4d 0 2024-04-05T21:26:46.606 INFO:tasks.workunit.client.1.smithi188.stdout:5/813: creat d6/d15/d5c/d7a/d44/d51/df1/d104/f118 x:0 0 0 2024-04-05T21:26:46.606 INFO:tasks.workunit.client.1.smithi188.stdout:8/703: creat d5/fe7 x:0 0 0 2024-04-05T21:26:46.607 INFO:tasks.workunit.client.0.smithi121.stdout:3/502: creat d0/d3/d24/d3d/d1a/d23/d90/d9e/fa0 x:0 0 0 2024-04-05T21:26:46.608 INFO:tasks.workunit.client.0.smithi121.stdout:7/662: creat d0/d23/d5b/fcf x:0 0 0 2024-04-05T21:26:46.608 INFO:tasks.workunit.client.0.smithi121.stdout:1/714: dread d8/d8c/db6/d1a/d1b/d24/f49 [4194304,4194304] 0 2024-04-05T21:26:46.609 INFO:tasks.workunit.client.1.smithi188.stdout:0/719: creat d4/d6/d13/d23/de8/f107 x:0 0 0 2024-04-05T21:26:46.609 INFO:tasks.workunit.client.1.smithi188.stdout:6/878: link d6/d13/d38/l39 d6/d13/d18/d19/d2c/d3f/d95/l10c 0 2024-04-05T21:26:46.610 INFO:tasks.workunit.client.0.smithi121.stdout:1/715: write d8/d8c/db6/d1a/d1b/f65 [552995,15746] 0 2024-04-05T21:26:46.612 INFO:tasks.workunit.client.0.smithi121.stdout:7/663: creat d0/d32/d4e/fd0 x:0 0 0 2024-04-05T21:26:46.613 INFO:tasks.workunit.client.0.smithi121.stdout:1/716: creat d8/d8c/db6/d1a/d1b/d44/fee x:0 0 0 2024-04-05T21:26:46.614 INFO:tasks.workunit.client.0.smithi121.stdout:7/664: stat d0/d9/d22/d3e/d9c/da7/fbf 0 2024-04-05T21:26:46.615 INFO:tasks.workunit.client.1.smithi188.stdout:1/844: dread d1/dc/d16/d21/dbb/fc9 [0,4194304] 0 2024-04-05T21:26:46.615 INFO:tasks.workunit.client.1.smithi188.stdout:8/704: stat d5/d21/d2a/d95/dc0 0 2024-04-05T21:26:46.615 INFO:tasks.workunit.client.1.smithi188.stdout:3/916: dwrite d5/d7/d31/fb1 [4194304,4194304] 0 2024-04-05T21:26:46.615 INFO:tasks.workunit.client.1.smithi188.stdout:9/883: truncate dd/d11/d1f/d26/d55/dcf/fff 329654 0 2024-04-05T21:26:46.616 INFO:tasks.workunit.client.0.smithi121.stdout:9/477: rmdir d6/df 39 2024-04-05T21:26:46.617 INFO:tasks.workunit.client.1.smithi188.stdout:5/814: mkdir d6/d15/d5c/d119 0 2024-04-05T21:26:46.617 INFO:tasks.workunit.client.1.smithi188.stdout:1/845: read d1/dc/d16/d46/d88/d3f/f4b [2696400,103149] 0 2024-04-05T21:26:46.620 INFO:tasks.workunit.client.1.smithi188.stdout:9/884: write dd/d11/d1f/d26/ff3 [3249807,52382] 0 2024-04-05T21:26:46.621 INFO:tasks.workunit.client.0.smithi121.stdout:9/478: dread - d6/d36/d7b/d6b/f8c zero size 2024-04-05T21:26:46.625 INFO:tasks.workunit.client.0.smithi121.stdout:6/497: dwrite d1/f1c [4194304,4194304] 0 2024-04-05T21:26:46.627 INFO:tasks.workunit.client.1.smithi188.stdout:5/815: chown d6/c29 1535842764 1 2024-04-05T21:26:46.629 INFO:tasks.workunit.client.0.smithi121.stdout:9/479: write d6/d46/d8a/f93 [217840,34045] 0 2024-04-05T21:26:46.629 INFO:tasks.workunit.client.1.smithi188.stdout:8/705: mknod d5/d21/d40/d43/d44/d59/ce8 0 2024-04-05T21:26:46.631 INFO:tasks.workunit.client.1.smithi188.stdout:1/846: symlink d1/dc/d16/d46/d95/l109 0 2024-04-05T21:26:46.633 INFO:tasks.workunit.client.0.smithi121.stdout:7/665: dwrite d0/d9/d22/fa3 [0,4194304] 0 2024-04-05T21:26:46.634 INFO:tasks.workunit.client.1.smithi188.stdout:9/885: mkdir dd/d3d/d89/d12a 0 2024-04-05T21:26:46.634 INFO:tasks.workunit.client.0.smithi121.stdout:1/717: dwrite d8/d89/d9e/da3/fb0 [0,4194304] 0 2024-04-05T21:26:46.642 INFO:tasks.workunit.client.1.smithi188.stdout:8/706: dwrite d5/d21/f3a [0,4194304] 0 2024-04-05T21:26:46.643 INFO:tasks.workunit.client.0.smithi121.stdout:1/718: mknod d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b/cef 0 2024-04-05T21:26:46.644 INFO:tasks.workunit.client.1.smithi188.stdout:5/816: fsync d6/d17/f2f 0 2024-04-05T21:26:46.644 INFO:tasks.workunit.client.1.smithi188.stdout:6/879: dwrite d6/d13/d18/d19/d2c/d34/fac [0,4194304] 0 2024-04-05T21:26:46.645 INFO:tasks.workunit.client.0.smithi121.stdout:7/666: mknod d0/d9/d22/d3e/d9c/cd1 0 2024-04-05T21:26:46.647 INFO:tasks.workunit.client.0.smithi121.stdout:9/480: rename d6/l3f to d6/d36/d7b/d5d/la6 0 2024-04-05T21:26:46.648 INFO:tasks.workunit.client.0.smithi121.stdout:3/503: dread d0/f2d [0,4194304] 0 2024-04-05T21:26:46.651 INFO:tasks.workunit.client.0.smithi121.stdout:3/504: write d0/d3/d24/d3d/d1a/d23/d38/d55/f77 [3131780,2997] 0 2024-04-05T21:26:46.651 INFO:tasks.workunit.client.1.smithi188.stdout:3/917: link d5/d48/l118 d5/d35/l12d 0 2024-04-05T21:26:46.652 INFO:tasks.workunit.client.0.smithi121.stdout:7/667: rename d0/d32/d4a/c9d to d0/d32/cd2 0 2024-04-05T21:26:46.654 INFO:tasks.workunit.client.1.smithi188.stdout:8/707: symlink d5/d21/d40/d43/d48/d7e/le9 0 2024-04-05T21:26:46.655 INFO:tasks.workunit.client.0.smithi121.stdout:3/505: mkdir d0/d3/de/d83/da1 0 2024-04-05T21:26:46.655 INFO:tasks.workunit.client.0.smithi121.stdout:7/668: fsync d0/d9/d22/fa3 0 2024-04-05T21:26:46.657 INFO:tasks.workunit.client.0.smithi121.stdout:1/719: rename d8/d8c/db6/d1a/d1b/d4c/da1/lb2 to d8/d8c/db6/d1a/d1b/d4c/d76/d53/lf0 0 2024-04-05T21:26:46.657 INFO:tasks.workunit.client.0.smithi121.stdout:7/669: chown d0/d9/d22/d3e/d9c/da7 183 1 2024-04-05T21:26:46.657 INFO:tasks.workunit.client.0.smithi121.stdout:9/481: truncate d6/d36/d7b/d5d/d79/f95 136236 0 2024-04-05T21:26:46.658 INFO:tasks.workunit.client.1.smithi188.stdout:9/886: rename dd/d35/c45 to dd/d11/d22/c12b 0 2024-04-05T21:26:46.659 INFO:tasks.workunit.client.1.smithi188.stdout:5/817: creat d6/d15/f11a x:0 0 0 2024-04-05T21:26:46.659 INFO:tasks.workunit.client.0.smithi121.stdout:3/506: creat d0/d3/d24/d3d/d1a/d1e/d86/fa2 x:0 0 0 2024-04-05T21:26:46.659 INFO:tasks.workunit.client.0.smithi121.stdout:1/720: rmdir d8/d10 39 2024-04-05T21:26:46.661 INFO:tasks.workunit.client.0.smithi121.stdout:2/524: dread d2/d9/d38/d2d/d47/f70 [0,4194304] 0 2024-04-05T21:26:46.661 INFO:tasks.workunit.client.0.smithi121.stdout:3/507: stat d0/d3/d24/d6d/d7b 0 2024-04-05T21:26:46.663 INFO:tasks.workunit.client.0.smithi121.stdout:7/670: truncate d0/d9/d22/d28/f48 2239380 0 2024-04-05T21:26:46.663 INFO:tasks.workunit.client.1.smithi188.stdout:3/918: mknod d5/d7/d65/d61/d66/da4/de3/c12e 0 2024-04-05T21:26:46.664 INFO:tasks.workunit.client.1.smithi188.stdout:5/818: mkdir d6/d15/d1b/d4f/d11b 0 2024-04-05T21:26:46.664 INFO:tasks.workunit.client.0.smithi121.stdout:2/525: chown d2/d9/d17/d3a/f55 2 1 2024-04-05T21:26:46.664 INFO:tasks.workunit.client.0.smithi121.stdout:1/721: mknod d8/d89/d9e/cf1 0 2024-04-05T21:26:46.664 INFO:tasks.workunit.client.0.smithi121.stdout:7/671: read - d0/d32/d4e/fb6 zero size 2024-04-05T21:26:46.665 INFO:tasks.workunit.client.1.smithi188.stdout:5/819: chown d6/d82/d40/d42/d81 194409 1 2024-04-05T21:26:46.665 INFO:tasks.workunit.client.0.smithi121.stdout:1/722: write d8/d8c/f8e [2099659,7012] 0 2024-04-05T21:26:46.666 INFO:tasks.workunit.client.1.smithi188.stdout:9/887: truncate dd/d11/d22/f4d 682116 0 2024-04-05T21:26:46.666 INFO:tasks.workunit.client.1.smithi188.stdout:5/820: stat d6/d82/d40/f9c 0 2024-04-05T21:26:46.667 INFO:tasks.workunit.client.1.smithi188.stdout:1/847: dread d1/dc/d16/d46/d88/d86/fab [0,4194304] 0 2024-04-05T21:26:46.667 INFO:tasks.workunit.client.1.smithi188.stdout:6/880: rename d6/d13/d18/d19/l66 to d6/d13/d18/l10d 0 2024-04-05T21:26:46.669 INFO:tasks.workunit.client.0.smithi121.stdout:3/508: getdents d0/d3/d24/d3d/d1a 0 2024-04-05T21:26:46.670 INFO:tasks.workunit.client.0.smithi121.stdout:9/482: dread d6/d36/d7b/d5d/f87 [0,4194304] 0 2024-04-05T21:26:46.670 INFO:tasks.workunit.client.0.smithi121.stdout:7/672: read d0/d23/d5b/f43 [542278,11421] 0 2024-04-05T21:26:46.671 INFO:tasks.workunit.client.0.smithi121.stdout:1/723: unlink d8/d8c/db6/f2c 0 2024-04-05T21:26:46.671 INFO:tasks.workunit.client.1.smithi188.stdout:5/821: chown d6/d82/d40/d42/d81/da6/db2/cd5 91207800 1 2024-04-05T21:26:46.672 INFO:tasks.workunit.client.1.smithi188.stdout:6/881: symlink d6/d13/d18/d19/l10e 0 2024-04-05T21:26:46.674 INFO:tasks.workunit.client.1.smithi188.stdout:1/848: mknod d1/dc/d16/d46/df7/c10a 0 2024-04-05T21:26:46.675 INFO:tasks.workunit.client.0.smithi121.stdout:3/509: getdents d0 0 2024-04-05T21:26:46.676 INFO:tasks.workunit.client.0.smithi121.stdout:1/724: mkdir d8/d89/d9e/da3/df2 0 2024-04-05T21:26:46.677 INFO:tasks.workunit.client.0.smithi121.stdout:7/673: mknod d0/d9/d22/d3e/d9c/cd3 0 2024-04-05T21:26:46.679 INFO:tasks.workunit.client.1.smithi188.stdout:6/882: write d6/d13/d18/d19/d2c/d34/f97 [4148663,67892] 0 2024-04-05T21:26:46.679 INFO:tasks.workunit.client.1.smithi188.stdout:9/888: dwrite dd/d11/d1f/d25/fce [0,4194304] 0 2024-04-05T21:26:46.679 INFO:tasks.workunit.client.1.smithi188.stdout:1/849: write d1/dc/d16/d46/d88/d13/fc3 [457733,10174] 0 2024-04-05T21:26:46.679 INFO:tasks.workunit.client.0.smithi121.stdout:2/526: dread d2/d1e/f52 [0,4194304] 0 2024-04-05T21:26:46.681 INFO:tasks.workunit.client.1.smithi188.stdout:6/883: chown d6/d7/c10 14721 1 2024-04-05T21:26:46.681 INFO:tasks.workunit.client.1.smithi188.stdout:5/822: getdents d6/d15/d5c/d7a/d44/d51/df1/d104 0 2024-04-05T21:26:46.681 INFO:tasks.workunit.client.0.smithi121.stdout:9/483: rename d6/df/f20 to d6/d36/d7b/d5d/d79/fa7 0 2024-04-05T21:26:46.682 INFO:tasks.workunit.client.0.smithi121.stdout:2/527: readlink d2/d9/d38/d2d/d4c/d7c/d7e/l95 0 2024-04-05T21:26:46.684 INFO:tasks.workunit.client.0.smithi121.stdout:3/510: write d0/d3/f69 [1904655,109383] 0 2024-04-05T21:26:46.685 INFO:tasks.workunit.client.1.smithi188.stdout:5/823: write d6/d82/d40/d42/d81/d10c/f117 [639405,45329] 0 2024-04-05T21:26:46.687 INFO:tasks.workunit.client.1.smithi188.stdout:1/850: creat d1/dc/d16/d46/de5/f10b x:0 0 0 2024-04-05T21:26:46.690 INFO:tasks.workunit.client.0.smithi121.stdout:7/674: symlink d0/d32/d7c/ld4 0 2024-04-05T21:26:46.690 INFO:tasks.workunit.client.0.smithi121.stdout:3/511: fdatasync d0/d3/d24/d3d/d1a/d23/d90/d9e/fa0 0 2024-04-05T21:26:46.690 INFO:tasks.workunit.client.0.smithi121.stdout:7/675: chown d0/d57 0 1 2024-04-05T21:26:46.690 INFO:tasks.workunit.client.0.smithi121.stdout:2/528: rmdir d2/d9 39 2024-04-05T21:26:46.692 INFO:tasks.workunit.client.0.smithi121.stdout:9/484: write d6/d36/d7b/d5d/d79/fa7 [415903,107464] 0 2024-04-05T21:26:46.693 INFO:tasks.workunit.client.0.smithi121.stdout:7/676: unlink d0/d9/d22/f79 0 2024-04-05T21:26:46.694 INFO:tasks.workunit.client.0.smithi121.stdout:3/512: mkdir d0/d3/d24/d6d/d7b/da3 0 2024-04-05T21:26:46.697 INFO:tasks.workunit.client.1.smithi188.stdout:6/884: creat d6/d13/d18/d19/d2c/d3f/d95/f10f x:0 0 0 2024-04-05T21:26:46.699 INFO:tasks.workunit.client.0.smithi121.stdout:9/485: write d6/df/d1d/d2c/d37/d3c/f7c [3423131,6705] 0 2024-04-05T21:26:46.701 INFO:tasks.workunit.client.0.smithi121.stdout:7/677: write d0/d9/d22/d28/d56/f58 [1835610,55632] 0 2024-04-05T21:26:46.701 INFO:tasks.workunit.client.0.smithi121.stdout:9/486: stat d6/d36/d7b/d5d/d79/d94/l98 0 2024-04-05T21:26:46.702 INFO:tasks.workunit.client.1.smithi188.stdout:9/889: dwrite dd/d11/d1f/d26/d55/d98/da5/fe6 [0,4194304] 0 2024-04-05T21:26:46.702 INFO:tasks.workunit.client.0.smithi121.stdout:3/513: mkdir d0/d3/d24/d6d/d7b/da4 0 2024-04-05T21:26:46.704 INFO:tasks.workunit.client.0.smithi121.stdout:2/529: fdatasync d2/d9/d38/d2d/f41 0 2024-04-05T21:26:46.705 INFO:tasks.workunit.client.0.smithi121.stdout:7/678: write d0/d9/d22/faf [311761,33400] 0 2024-04-05T21:26:46.706 INFO:tasks.workunit.client.0.smithi121.stdout:9/487: symlink d6/df/d50/la8 0 2024-04-05T21:26:46.707 INFO:tasks.workunit.client.0.smithi121.stdout:1/725: dread d8/d8c/db6/d1a/d1b/d4c/d76/f7c [0,4194304] 0 2024-04-05T21:26:46.707 INFO:tasks.workunit.client.0.smithi121.stdout:2/530: chown d2/d28/c40 90139 1 2024-04-05T21:26:46.707 INFO:tasks.workunit.client.0.smithi121.stdout:3/514: read d0/d3/d24/d3d/f21 [1987587,60780] 0 2024-04-05T21:26:46.709 INFO:tasks.workunit.client.1.smithi188.stdout:9/890: dwrite dd/d11/d1f/d26/ff3 [0,4194304] 0 2024-04-05T21:26:46.710 INFO:tasks.workunit.client.0.smithi121.stdout:9/488: write d6/df/d1d/fa3 [446459,85635] 0 2024-04-05T21:26:46.716 INFO:tasks.workunit.client.0.smithi121.stdout:7/679: dread d0/d9/f86 [0,4194304] 0 2024-04-05T21:26:46.718 INFO:tasks.workunit.client.0.smithi121.stdout:1/726: rename d8/d8c/db6/d1a/d1b/f69 to d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b/ff3 0 2024-04-05T21:26:46.719 INFO:tasks.workunit.client.1.smithi188.stdout:6/885: dread d6/d13/d18/d19/d2c/f6e [0,4194304] 0 2024-04-05T21:26:46.719 INFO:tasks.workunit.client.0.smithi121.stdout:9/489: fsync d6/df/d1d/d2c/d37/f86 0 2024-04-05T21:26:46.722 INFO:tasks.workunit.client.0.smithi121.stdout:1/727: write d8/d8c/db6/f1e [1228699,18505] 0 2024-04-05T21:26:46.725 INFO:tasks.workunit.client.0.smithi121.stdout:7/680: dwrite d0/d32/f8f [0,4194304] 0 2024-04-05T21:26:46.729 INFO:tasks.workunit.client.0.smithi121.stdout:1/728: mkdir d8/d89/d9e/dc8/df4 0 2024-04-05T21:26:46.730 INFO:tasks.workunit.client.1.smithi188.stdout:1/851: symlink d1/dc/d16/d21/l10c 0 2024-04-05T21:26:46.735 INFO:tasks.workunit.client.0.smithi121.stdout:7/681: rename d0/d23/d5a/f7b to d0/d9/d22/d81/da2/fd5 0 2024-04-05T21:26:46.735 INFO:tasks.workunit.client.1.smithi188.stdout:5/824: rename d6/d15/c84 to d6/d15/c11c 0 2024-04-05T21:26:46.736 INFO:tasks.workunit.client.1.smithi188.stdout:6/886: fdatasync d6/d13/d18/d19/d2c/f51 0 2024-04-05T21:26:46.737 INFO:tasks.workunit.client.0.smithi121.stdout:7/682: symlink d0/d9/d22/d3e/ld6 0 2024-04-05T21:26:46.737 INFO:tasks.workunit.client.1.smithi188.stdout:5/825: stat d6/f9 0 2024-04-05T21:26:46.738 INFO:tasks.workunit.client.1.smithi188.stdout:6/887: chown d6/d13/d18/d19/d2c/d3f/d77 27 1 2024-04-05T21:26:46.738 INFO:tasks.workunit.client.1.smithi188.stdout:5/826: readlink d6/d15/d5c/l64 0 2024-04-05T21:26:46.739 INFO:tasks.workunit.client.0.smithi121.stdout:7/683: creat d0/d23/fd7 x:0 0 0 2024-04-05T21:26:46.741 INFO:tasks.workunit.client.1.smithi188.stdout:6/888: unlink d6/d13/d18/d19/d2c/f32 0 2024-04-05T21:26:46.741 INFO:tasks.workunit.client.1.smithi188.stdout:5/827: mknod d6/dad/d110/c11d 0 2024-04-05T21:26:46.742 INFO:tasks.workunit.client.1.smithi188.stdout:6/889: symlink d6/d13/d38/l110 0 2024-04-05T21:26:46.748 INFO:tasks.workunit.client.1.smithi188.stdout:5/828: rename d6/d106/d8e/l102 to d6/d15/d1b/d4f/d11b/l11e 0 2024-04-05T21:26:46.750 INFO:tasks.workunit.client.1.smithi188.stdout:6/890: dwrite d6/d13/d18/d3c/fcf [0,4194304] 0 2024-04-05T21:26:46.751 INFO:tasks.workunit.client.0.smithi121.stdout:1/729: write d8/d8c/db6/d1a/d1b/d4c/da1/fe6 [955128,59044] 0 2024-04-05T21:26:46.755 INFO:tasks.workunit.client.0.smithi121.stdout:2/531: dwrite d2/d9/f21 [0,4194304] 0 2024-04-05T21:26:46.756 INFO:tasks.workunit.client.0.smithi121.stdout:7/684: dwrite d0/d23/f8b [4194304,4194304] 0 2024-04-05T21:26:46.759 INFO:tasks.workunit.client.1.smithi188.stdout:5/829: creat d6/d15/d5c/d7a/d44/d98/f11f x:0 0 0 2024-04-05T21:26:46.764 INFO:tasks.workunit.client.0.smithi121.stdout:9/490: dwrite d6/df/d1d/d2c/d37/f86 [4194304,4194304] 0 2024-04-05T21:26:46.764 INFO:tasks.workunit.client.0.smithi121.stdout:3/515: dwrite d0/d3/f4 [0,4194304] 0 2024-04-05T21:26:46.765 INFO:tasks.workunit.client.0.smithi121.stdout:2/532: mknod d2/d9/c9f 0 2024-04-05T21:26:46.771 INFO:tasks.workunit.client.1.smithi188.stdout:9/891: write dd/d11/d1f/d26/dc0/f5c [735753,103465] 0 2024-04-05T21:26:46.772 INFO:tasks.workunit.client.0.smithi121.stdout:7/685: creat d0/d23/d5b/d52/fd8 x:0 0 0 2024-04-05T21:26:46.772 INFO:tasks.workunit.client.0.smithi121.stdout:9/491: write d6/df/d1d/fa3 [961838,6049] 0 2024-04-05T21:26:46.772 INFO:tasks.workunit.client.1.smithi188.stdout:1/852: write d1/dc/d16/f92 [1742743,56951] 0 2024-04-05T21:26:46.772 INFO:tasks.workunit.client.0.smithi121.stdout:1/730: truncate d8/d8c/db6/d1a/d1b/d24/fe9 826046 0 2024-04-05T21:26:46.773 INFO:tasks.workunit.client.0.smithi121.stdout:9/492: truncate d6/df/d48/f7e 143652 0 2024-04-05T21:26:46.773 INFO:tasks.workunit.client.0.smithi121.stdout:2/533: rename d2/d9/d38/d2d/d4c/d7c/l6e to d2/d9/d56/d8d/la0 0 2024-04-05T21:26:46.775 INFO:tasks.workunit.client.1.smithi188.stdout:9/892: write dd/d11/d1f/d26/d55/d98/f120 [678707,61139] 0 2024-04-05T21:26:46.776 INFO:tasks.workunit.client.0.smithi121.stdout:1/731: mknod d8/d89/d9e/d57/cf5 0 2024-04-05T21:26:46.777 INFO:tasks.workunit.client.1.smithi188.stdout:9/893: chown dd/d11/d1f/d26/ff3 3649 1 2024-04-05T21:26:46.778 INFO:tasks.workunit.client.0.smithi121.stdout:1/732: write d8/d8c/db6/f1e [5107626,130212] 0 2024-04-05T21:26:46.779 INFO:tasks.workunit.client.1.smithi188.stdout:6/891: dwrite d6/d13/d18/d19/d2c/f51 [4194304,4194304] 0 2024-04-05T21:26:46.780 INFO:tasks.workunit.client.1.smithi188.stdout:9/894: fsync dd/d11/d1f/d26/dc0/d6b/ff4 0 2024-04-05T21:26:46.780 INFO:tasks.workunit.client.0.smithi121.stdout:9/493: rmdir d6/d36/d7b/d5d/d79 39 2024-04-05T21:26:46.781 INFO:tasks.workunit.client.0.smithi121.stdout:3/516: dread d0/d3/d24/d6d/d7b/f85 [0,4194304] 0 2024-04-05T21:26:46.783 INFO:tasks.workunit.client.1.smithi188.stdout:6/892: getdents d6/d13/d18/d3c/dd2/dd6 0 2024-04-05T21:26:46.784 INFO:tasks.workunit.client.1.smithi188.stdout:9/895: rename dd/d11/d1f/d26/dc0/lb5 to dd/d11/d1f/d26/dc0/d6b/d40/l12c 0 2024-04-05T21:26:46.787 INFO:tasks.workunit.client.0.smithi121.stdout:7/686: getdents d0/d23/d5b/d52 0 2024-04-05T21:26:46.788 INFO:tasks.workunit.client.0.smithi121.stdout:7/687: fdatasync d0/d23/d5b/d52/f6e 0 2024-04-05T21:26:46.790 INFO:tasks.workunit.client.0.smithi121.stdout:9/494: getdents d6/df/d1d 0 2024-04-05T21:26:46.790 INFO:tasks.workunit.client.0.smithi121.stdout:8/545: sync 2024-04-05T21:26:46.791 INFO:tasks.workunit.client.0.smithi121.stdout:1/733: creat d8/d10/ff6 x:0 0 0 2024-04-05T21:26:46.792 INFO:tasks.workunit.client.1.smithi188.stdout:9/896: dwrite dd/d3d/d89/fc3 [4194304,4194304] 0 2024-04-05T21:26:46.792 INFO:tasks.workunit.client.0.smithi121.stdout:2/534: truncate d2/d9/d17/d3a/f4a 2721315 0 2024-04-05T21:26:46.793 INFO:tasks.workunit.client.0.smithi121.stdout:9/495: dread d6/df/d1d/f23 [0,4194304] 0 2024-04-05T21:26:46.795 INFO:tasks.workunit.client.1.smithi188.stdout:9/897: mkdir dd/d3d/d89/dd1/d12d 0 2024-04-05T21:26:46.797 INFO:tasks.workunit.client.1.smithi188.stdout:9/898: write dd/d3d/d81/f107 [2797832,130853] 0 2024-04-05T21:26:46.800 INFO:tasks.workunit.client.1.smithi188.stdout:9/899: mkdir dd/d11/d1f/d26/dc0/d6b/d12e 0 2024-04-05T21:26:46.800 INFO:tasks.workunit.client.1.smithi188.stdout:9/900: fdatasync dd/d11/d1f/d25/f44 0 2024-04-05T21:26:46.801 INFO:tasks.workunit.client.1.smithi188.stdout:9/901: stat dd/d11/d73/d8d/ldb 0 2024-04-05T21:26:46.811 INFO:tasks.workunit.client.1.smithi188.stdout:5/830: write d6/d82/f47 [2111312,46116] 0 2024-04-05T21:26:46.812 INFO:tasks.workunit.client.1.smithi188.stdout:5/831: symlink d6/dad/d110/l120 0 2024-04-05T21:26:46.815 INFO:tasks.workunit.client.1.smithi188.stdout:5/832: dread d6/d17/fb8 [0,4194304] 0 2024-04-05T21:26:46.816 INFO:tasks.workunit.client.1.smithi188.stdout:6/893: write d6/d13/d18/d19/d2c/d3f/f53 [898566,69825] 0 2024-04-05T21:26:46.821 INFO:tasks.workunit.client.1.smithi188.stdout:5/833: mknod d6/d82/d40/d42/d81/d10c/c121 0 2024-04-05T21:26:46.825 INFO:tasks.workunit.client.1.smithi188.stdout:1/853: dwrite d1/d7/f66 [0,4194304] 0 2024-04-05T21:26:46.827 INFO:tasks.workunit.client.1.smithi188.stdout:6/894: dread d6/d13/d18/d19/d2c/f6e [0,4194304] 0 2024-04-05T21:26:46.829 INFO:tasks.workunit.client.1.smithi188.stdout:9/902: dwrite dd/d11/d1f/d26/dc0/d6b/f96 [0,4194304] 0 2024-04-05T21:26:46.836 INFO:tasks.workunit.client.1.smithi188.stdout:9/903: symlink dd/d3d/d95/l12f 0 2024-04-05T21:26:46.836 INFO:tasks.workunit.client.1.smithi188.stdout:9/904: write dd/d11/d1f/f10a [970169,5236] 0 2024-04-05T21:26:46.836 INFO:tasks.workunit.client.1.smithi188.stdout:6/895: mknod d6/d13/d18/d3c/ddc/c111 0 2024-04-05T21:26:46.837 INFO:tasks.workunit.client.1.smithi188.stdout:5/834: write d6/d17/d23/d7d/dc4/fc9 [994502,90393] 0 2024-04-05T21:26:46.838 INFO:tasks.workunit.client.1.smithi188.stdout:1/854: fdatasync d1/dc/d16/d21/d27/d4a/f8c 0 2024-04-05T21:26:46.839 INFO:tasks.workunit.client.1.smithi188.stdout:6/896: symlink d6/d28/l112 0 2024-04-05T21:26:46.840 INFO:tasks.workunit.client.0.smithi121.stdout:3/517: write d0/d3/d24/d3d/d1a/d1e/f2a [700008,52094] 0 2024-04-05T21:26:46.840 INFO:tasks.workunit.client.1.smithi188.stdout:9/905: mknod dd/d11/c130 0 2024-04-05T21:26:46.841 INFO:tasks.workunit.client.0.smithi121.stdout:8/546: symlink d12/d50/d61/lae 0 2024-04-05T21:26:46.841 INFO:tasks.workunit.client.1.smithi188.stdout:5/835: mkdir d6/d17/d23/d7d/dc4/d122 0 2024-04-05T21:26:46.842 INFO:tasks.workunit.client.0.smithi121.stdout:9/496: write d6/f9b [6113,18459] 0 2024-04-05T21:26:46.842 INFO:tasks.workunit.client.1.smithi188.stdout:6/897: mknod d6/d7/df7/c113 0 2024-04-05T21:26:46.843 INFO:tasks.workunit.client.1.smithi188.stdout:9/906: rmdir dd/d11/d1f/d26/d55 39 2024-04-05T21:26:46.843 INFO:tasks.workunit.client.1.smithi188.stdout:9/907: readlink dd/d35/l46 0 2024-04-05T21:26:46.845 INFO:tasks.workunit.client.1.smithi188.stdout:1/855: mknod d1/dc/d16/d46/d88/c10d 0 2024-04-05T21:26:46.845 INFO:tasks.workunit.client.0.smithi121.stdout:2/535: unlink d2/d9/d56/c9d 0 2024-04-05T21:26:46.845 INFO:tasks.workunit.client.0.smithi121.stdout:8/547: fsync d12/d13/f84 0 2024-04-05T21:26:46.846 INFO:tasks.workunit.client.1.smithi188.stdout:6/898: write d6/d13/d18/d19/da3/ff3 [393687,29551] 0 2024-04-05T21:26:46.847 INFO:tasks.workunit.client.0.smithi121.stdout:9/497: symlink d6/d36/d7b/d6b/la9 0 2024-04-05T21:26:46.847 INFO:tasks.workunit.client.0.smithi121.stdout:2/536: chown d2/d9/d38/d2d/d47/d94/f37 36 1 2024-04-05T21:26:46.850 INFO:tasks.workunit.client.1.smithi188.stdout:9/908: creat dd/d11/d1f/d26/dc0/d102/f131 x:0 0 0 2024-04-05T21:26:46.852 INFO:tasks.workunit.client.1.smithi188.stdout:5/836: dwrite d6/d15/ff8 [0,4194304] 0 2024-04-05T21:26:46.857 INFO:tasks.workunit.client.1.smithi188.stdout:9/909: fsync dd/d11/d1f/d26/dc0/f66 0 2024-04-05T21:26:46.857 INFO:tasks.workunit.client.1.smithi188.stdout:5/837: mkdir d6/d15/d5c/d7a/d44/d98/dbf/d123 0 2024-04-05T21:26:46.862 INFO:tasks.workunit.client.1.smithi188.stdout:5/838: unlink d6/d106/d8e/fdb 0 2024-04-05T21:26:46.863 INFO:tasks.workunit.client.1.smithi188.stdout:5/839: chown d6/d15/d1b/f6c 8 1 2024-04-05T21:26:46.870 INFO:tasks.workunit.client.1.smithi188.stdout:1/856: dwrite d1/dc/d16/d2d/f97 [0,4194304] 0 2024-04-05T21:26:46.873 INFO:tasks.workunit.client.1.smithi188.stdout:1/857: rmdir d1/dc/d16/d46/d95/d99 39 2024-04-05T21:26:46.876 INFO:tasks.workunit.client.1.smithi188.stdout:6/899: dread d6/d7/f16 [0,4194304] 0 2024-04-05T21:26:46.877 INFO:tasks.workunit.client.1.smithi188.stdout:6/900: write d6/f74 [2434680,19376] 0 2024-04-05T21:26:46.878 INFO:tasks.workunit.client.1.smithi188.stdout:1/858: read d1/dc/d16/d21/fa0 [111732,103204] 0 2024-04-05T21:26:46.884 INFO:tasks.workunit.client.1.smithi188.stdout:1/859: link d1/c84 d1/dc/d16/d21/d27/d49/c10e 0 2024-04-05T21:26:46.887 INFO:tasks.workunit.client.1.smithi188.stdout:5/840: dwrite d6/d15/d5c/d7a/d44/f5d [0,4194304] 0 2024-04-05T21:26:46.889 INFO:tasks.workunit.client.1.smithi188.stdout:6/901: dwrite d6/d13/d18/d19/d2c/d3f/d77/fc0 [0,4194304] 0 2024-04-05T21:26:46.899 INFO:tasks.workunit.client.1.smithi188.stdout:5/841: mkdir d6/d15/d5c/d7a/d44/d51/df1/d104/d124 0 2024-04-05T21:26:46.915 INFO:tasks.workunit.client.1.smithi188.stdout:1/860: dread d1/dc/f56 [0,4194304] 0 2024-04-05T21:26:46.919 INFO:tasks.workunit.client.1.smithi188.stdout:5/842: dwrite d6/d17/d23/d7d/fd7 [0,4194304] 0 2024-04-05T21:26:46.919 INFO:tasks.workunit.client.1.smithi188.stdout:6/902: dwrite d6/d13/d48/fa2 [0,4194304] 0 2024-04-05T21:26:46.924 INFO:tasks.workunit.client.0.smithi121.stdout:3/518: mknod d0/d3/de/ca5 0 2024-04-05T21:26:46.925 INFO:tasks.workunit.client.0.smithi121.stdout:7/688: read d0/d9/d22/d28/d3b/f4f [3781151,99504] 0 2024-04-05T21:26:46.925 INFO:tasks.workunit.client.0.smithi121.stdout:9/498: creat d6/df/d48/faa x:0 0 0 2024-04-05T21:26:46.926 INFO:tasks.workunit.client.1.smithi188.stdout:5/843: write d6/f5b [4907577,126726] 0 2024-04-05T21:26:46.927 INFO:tasks.workunit.client.0.smithi121.stdout:7/689: fdatasync d0/d9/d22/d3e/d9c/fb2 0 2024-04-05T21:26:46.929 INFO:tasks.workunit.client.0.smithi121.stdout:2/537: mknod d2/d9/d56/ca1 0 2024-04-05T21:26:46.931 INFO:tasks.workunit.client.1.smithi188.stdout:6/903: symlink d6/d13/d48/l114 0 2024-04-05T21:26:46.932 INFO:tasks.workunit.client.0.smithi121.stdout:3/519: rename d0/d3/de/d83 to d0/d3/de/da6 0 2024-04-05T21:26:46.932 INFO:tasks.workunit.client.1.smithi188.stdout:5/844: creat d6/d17/f125 x:0 0 0 2024-04-05T21:26:46.938 INFO:tasks.workunit.client.0.smithi121.stdout:8/548: dwrite d12/f73 [0,4194304] 0 2024-04-05T21:26:46.938 INFO:tasks.workunit.client.0.smithi121.stdout:3/520: symlink d0/d3/d24/d3d/d1a/d23/la7 0 2024-04-05T21:26:46.938 INFO:tasks.workunit.client.1.smithi188.stdout:1/861: dwrite d1/dc/d16/d46/d88/d13/f4f [0,4194304] 0 2024-04-05T21:26:46.938 INFO:tasks.workunit.client.1.smithi188.stdout:5/845: truncate d6/dad/f8d 1372122 0 2024-04-05T21:26:46.940 INFO:tasks.workunit.client.0.smithi121.stdout:2/538: link d2/d9/d38/d2d/d4c/f78 d2/d9/d38/d2d/d4c/d79/fa2 0 2024-04-05T21:26:46.941 INFO:tasks.workunit.client.1.smithi188.stdout:1/862: unlink d1/d7/dbf/cd2 0 2024-04-05T21:26:46.942 INFO:tasks.workunit.client.0.smithi121.stdout:3/521: creat d0/d3/d24/d3d/fa8 x:0 0 0 2024-04-05T21:26:46.942 INFO:tasks.workunit.client.0.smithi121.stdout:2/539: truncate d2/d9/f75 5044610 0 2024-04-05T21:26:46.942 INFO:tasks.workunit.client.1.smithi188.stdout:6/904: dread d6/d13/d38/fd7 [0,4194304] 0 2024-04-05T21:26:46.944 INFO:tasks.workunit.client.1.smithi188.stdout:1/863: read d1/d7/dbf/fc4 [831765,116670] 0 2024-04-05T21:26:46.945 INFO:tasks.workunit.client.1.smithi188.stdout:6/905: chown d6/d13/d18/d3c/ddc/c111 4274 1 2024-04-05T21:26:46.947 INFO:tasks.workunit.client.1.smithi188.stdout:5/846: link d6/d15/d5c/d7a/f52 d6/d82/d40/d42/d81/da6/db2/f126 0 2024-04-05T21:26:46.947 INFO:tasks.workunit.client.0.smithi121.stdout:2/540: rename d2/d9/d38/d2d/l98 to d2/d9/d38/d2d/d4c/d7c/la3 0 2024-04-05T21:26:46.948 INFO:tasks.workunit.client.1.smithi188.stdout:1/864: mkdir d1/dc/d16/d21/d27/d10f 0 2024-04-05T21:26:46.951 INFO:tasks.workunit.client.1.smithi188.stdout:6/906: mknod d6/d13/d18/d3c/ddc/c115 0 2024-04-05T21:26:46.954 INFO:tasks.workunit.client.0.smithi121.stdout:2/541: dwrite d2/d1e/f8c [0,4194304] 0 2024-04-05T21:26:46.954 INFO:tasks.workunit.client.1.smithi188.stdout:1/865: unlink d1/dc/d16/f3c 0 2024-04-05T21:26:46.956 INFO:tasks.workunit.client.0.smithi121.stdout:2/542: mknod d2/d9/d38/d2d/d4c/d7c/ca4 0 2024-04-05T21:26:46.961 INFO:tasks.workunit.client.1.smithi188.stdout:1/866: creat d1/dc/d16/d46/d88/d13/f110 x:0 0 0 2024-04-05T21:26:46.961 INFO:tasks.workunit.client.0.smithi121.stdout:8/549: dread d12/d13/d15/d1d/f27 [0,4194304] 0 2024-04-05T21:26:46.963 INFO:tasks.workunit.client.1.smithi188.stdout:6/907: dwrite d6/d13/d18/d19/d2c/d3f/f7a [0,4194304] 0 2024-04-05T21:26:46.964 INFO:tasks.workunit.client.0.smithi121.stdout:8/550: truncate d12/d13/d56/d57/f88 5145968 0 2024-04-05T21:26:46.965 INFO:tasks.workunit.client.0.smithi121.stdout:2/543: dwrite d2/d9/d38/d2d/d47/f70 [0,4194304] 0 2024-04-05T21:26:46.966 INFO:tasks.workunit.client.1.smithi188.stdout:1/867: mkdir d1/dc/dfc/d111 0 2024-04-05T21:26:46.966 INFO:tasks.workunit.client.0.smithi121.stdout:8/551: write d12/d50/d6a/f45 [881190,112130] 0 2024-04-05T21:26:46.969 INFO:tasks.workunit.client.1.smithi188.stdout:6/908: mknod d6/d71/c116 0 2024-04-05T21:26:46.969 INFO:tasks.workunit.client.0.smithi121.stdout:8/552: write d12/d13/d56/d57/f88 [1939189,49515] 0 2024-04-05T21:26:46.972 INFO:tasks.workunit.client.1.smithi188.stdout:6/909: read d6/d71/f10b [755836,110229] 0 2024-04-05T21:26:46.973 INFO:tasks.workunit.client.0.smithi121.stdout:2/544: dread d2/d9/d38/d2d/d4c/d79/fa2 [0,4194304] 0 2024-04-05T21:26:46.973 INFO:tasks.workunit.client.0.smithi121.stdout:8/553: symlink d12/d50/d61/d7f/laf 0 2024-04-05T21:26:46.974 INFO:tasks.workunit.client.0.smithi121.stdout:2/545: read - d2/d9/d1c/f69 zero size 2024-04-05T21:26:46.977 INFO:tasks.workunit.client.0.smithi121.stdout:7/690: dread d0/f15 [0,4194304] 0 2024-04-05T21:26:46.977 INFO:tasks.workunit.client.1.smithi188.stdout:1/868: creat d1/d7/f112 x:0 0 0 2024-04-05T21:26:46.979 INFO:tasks.workunit.client.1.smithi188.stdout:5/847: dwrite d6/d15/d5c/d7a/fc7 [0,4194304] 0 2024-04-05T21:26:46.979 INFO:tasks.workunit.client.1.smithi188.stdout:6/910: mkdir d6/d13/d18/d19/da3/d117 0 2024-04-05T21:26:46.979 INFO:tasks.workunit.client.0.smithi121.stdout:2/546: creat d2/d9/d38/d2d/fa5 x:0 0 0 2024-04-05T21:26:46.979 INFO:tasks.workunit.client.0.smithi121.stdout:4/748: sync 2024-04-05T21:26:46.982 INFO:tasks.workunit.client.1.smithi188.stdout:1/869: mknod d1/dc/d16/d21/d27/d10f/c113 0 2024-04-05T21:26:46.983 INFO:tasks.workunit.client.0.smithi121.stdout:2/547: creat d2/d9/d38/d2d/d4c/fa6 x:0 0 0 2024-04-05T21:26:46.983 INFO:tasks.workunit.client.1.smithi188.stdout:6/911: mkdir d6/d13/deb/d118 0 2024-04-05T21:26:46.985 INFO:tasks.workunit.client.0.smithi121.stdout:4/749: chown d5/d11/d2c/d9d/dae/d29/d2e/l4f 1569095 1 2024-04-05T21:26:46.987 INFO:tasks.workunit.client.1.smithi188.stdout:5/848: getdents d6/d17 0 2024-04-05T21:26:46.987 INFO:tasks.workunit.client.0.smithi121.stdout:8/554: dwrite d12/d13/d15/d1d/f42 [0,4194304] 0 2024-04-05T21:26:46.988 INFO:tasks.workunit.client.1.smithi188.stdout:6/912: rename d6/d13/d18/d19/l10e to d6/d7/df7/l119 0 2024-04-05T21:26:46.990 INFO:tasks.workunit.client.0.smithi121.stdout:7/691: dwrite d0/d32/d7c/fac [0,4194304] 0 2024-04-05T21:26:46.991 INFO:tasks.workunit.client.0.smithi121.stdout:3/522: dread d0/f15 [0,4194304] 0 2024-04-05T21:26:46.992 INFO:tasks.workunit.client.1.smithi188.stdout:5/849: unlink d6/dad/c54 0 2024-04-05T21:26:46.992 INFO:tasks.workunit.client.0.smithi121.stdout:8/555: unlink d12/d13/c75 0 2024-04-05T21:26:46.993 INFO:tasks.workunit.client.1.smithi188.stdout:6/913: unlink d6/d13/d18/d19/d2c/d3f/fae 0 2024-04-05T21:26:46.995 INFO:tasks.workunit.client.1.smithi188.stdout:5/850: truncate d6/d15/d5c/d7a/d44/f10a 97012 0 2024-04-05T21:26:46.997 INFO:tasks.workunit.client.0.smithi121.stdout:3/523: fsync d0/d3/de/f9c 0 2024-04-05T21:26:46.998 INFO:tasks.workunit.client.0.smithi121.stdout:7/692: fsync d0/d23/d5b/f80 0 2024-04-05T21:26:46.998 INFO:tasks.workunit.client.1.smithi188.stdout:5/851: chown d6/d15/ff8 310580987 1 2024-04-05T21:26:46.999 INFO:tasks.workunit.client.0.smithi121.stdout:8/556: creat d12/d50/d61/d7f/d9a/fb0 x:0 0 0 2024-04-05T21:26:47.001 INFO:tasks.workunit.client.0.smithi121.stdout:2/548: dwrite d2/d9/d56/d62/f7a [0,4194304] 0 2024-04-05T21:26:47.002 INFO:tasks.workunit.client.1.smithi188.stdout:1/870: rmdir d1/dc/d16/d46/d88/d13 39 2024-04-05T21:26:47.003 INFO:tasks.workunit.client.0.smithi121.stdout:2/549: write d2/d9/d38/d2d/d4c/fa6 [467498,36024] 0 2024-04-05T21:26:47.004 INFO:tasks.workunit.client.0.smithi121.stdout:8/557: rmdir d12/d50/d6a 39 2024-04-05T21:26:47.004 INFO:tasks.workunit.client.1.smithi188.stdout:5/852: fsync d6/d15/d1b/f8a 0 2024-04-05T21:26:47.005 INFO:tasks.workunit.client.0.smithi121.stdout:8/558: write d12/f51 [108525,44659] 0 2024-04-05T21:26:47.006 INFO:tasks.workunit.client.0.smithi121.stdout:7/693: mkdir d0/d9/d22/d3e/d9c/da7/dd9 0 2024-04-05T21:26:47.007 INFO:tasks.workunit.client.1.smithi188.stdout:5/853: chown d6/d15/d1b/l49 6 1 2024-04-05T21:26:47.008 INFO:tasks.workunit.client.0.smithi121.stdout:3/524: creat d0/d3/d24/d3d/d1a/d23/d38/d55/fa9 x:0 0 0 2024-04-05T21:26:47.009 INFO:tasks.workunit.client.1.smithi188.stdout:5/854: fsync d6/d15/d5c/d7a/d44/f10a 0 2024-04-05T21:26:47.011 INFO:tasks.workunit.client.1.smithi188.stdout:6/914: dwrite d6/d13/d18/d3c/ddc/d60/f109 [0,4194304] 0 2024-04-05T21:26:47.012 INFO:tasks.workunit.client.1.smithi188.stdout:1/871: symlink d1/dc/d16/d46/d88/dea/l114 0 2024-04-05T21:26:47.012 INFO:tasks.workunit.client.0.smithi121.stdout:7/694: mknod d0/d9/d22/d28/cda 0 2024-04-05T21:26:47.014 INFO:tasks.workunit.client.0.smithi121.stdout:2/550: dwrite d2/d9/d38/d2d/d4c/d7c/f8b [0,4194304] 0 2024-04-05T21:26:47.015 INFO:tasks.workunit.client.1.smithi188.stdout:1/872: read d1/dc/d16/d46/df7/d5f/f76 [3252216,120195] 0 2024-04-05T21:26:47.017 INFO:tasks.workunit.client.0.smithi121.stdout:8/559: truncate d12/d50/d93/d96/f72 753778 0 2024-04-05T21:26:47.018 INFO:tasks.workunit.client.0.smithi121.stdout:2/551: creat d2/d9/d17/fa7 x:0 0 0 2024-04-05T21:26:47.021 INFO:tasks.workunit.client.1.smithi188.stdout:1/873: dwrite d1/d7/fd3 [0,4194304] 0 2024-04-05T21:26:47.023 INFO:tasks.workunit.client.0.smithi121.stdout:3/525: getdents d0/d3/d24/d6d/d7b/da3 0 2024-04-05T21:26:47.025 INFO:tasks.workunit.client.0.smithi121.stdout:7/695: fsync d0/d9/d22/d28/d3b/f6f 0 2024-04-05T21:26:47.025 INFO:tasks.workunit.client.0.smithi121.stdout:3/526: chown d0/d3/d24/d3d/d1a/d23/l88 5 1 2024-04-05T21:26:47.026 INFO:tasks.workunit.client.0.smithi121.stdout:2/552: mknod d2/d9/d38/d2d/d4c/d7c/ca8 0 2024-04-05T21:26:47.028 INFO:tasks.workunit.client.0.smithi121.stdout:2/553: stat d2/f64 0 2024-04-05T21:26:47.028 INFO:tasks.workunit.client.0.smithi121.stdout:4/750: dwrite d5/d11/d2c/d9d/dae/d29/d2e/dd3/fc6 [0,4194304] 0 2024-04-05T21:26:47.028 INFO:tasks.workunit.client.1.smithi188.stdout:1/874: mknod d1/dc/d16/d46/d88/d3f/c115 0 2024-04-05T21:26:47.031 INFO:tasks.workunit.client.1.smithi188.stdout:2/843: sync 2024-04-05T21:26:47.032 INFO:tasks.workunit.client.0.smithi121.stdout:2/554: rename d2/f1a to d2/d9/d17/fa9 0 2024-04-05T21:26:47.033 INFO:tasks.workunit.client.0.smithi121.stdout:3/527: mknod d0/d3/d24/d6d/d7b/da3/caa 0 2024-04-05T21:26:47.035 INFO:tasks.workunit.client.1.smithi188.stdout:1/875: symlink d1/dc/d16/d46/d88/l116 0 2024-04-05T21:26:47.036 INFO:tasks.workunit.client.0.smithi121.stdout:3/528: fdatasync d0/d3/d24/d3f/f80 0 2024-04-05T21:26:47.037 INFO:tasks.workunit.client.0.smithi121.stdout:3/529: fdatasync d0/d3/f69 0 2024-04-05T21:26:47.046 INFO:tasks.workunit.client.0.smithi121.stdout:8/560: write fa [667551,96437] 0 2024-04-05T21:26:47.048 INFO:tasks.workunit.client.0.smithi121.stdout:8/561: fsync d12/d13/d15/d1d/fa9 0 2024-04-05T21:26:47.050 INFO:tasks.workunit.client.0.smithi121.stdout:2/555: write d2/d9/d38/d2d/d4c/d79/fa2 [369753,40729] 0 2024-04-05T21:26:47.053 INFO:tasks.workunit.client.0.smithi121.stdout:8/562: rmdir d12/d13/d56/d57/da4/d9d 0 2024-04-05T21:26:47.056 INFO:tasks.workunit.client.0.smithi121.stdout:8/563: stat d12/d50/d93/d96/f72 0 2024-04-05T21:26:47.057 INFO:tasks.workunit.client.0.smithi121.stdout:2/556: dwrite d2/d9/f75 [0,4194304] 0 2024-04-05T21:26:47.058 INFO:tasks.workunit.client.1.smithi188.stdout:1/876: dread d1/dc/d16/d46/df7/f6d [0,4194304] 0 2024-04-05T21:26:47.059 INFO:tasks.workunit.client.1.smithi188.stdout:1/877: dread - d1/dc/d16/d21/d27/d49/fb8 zero size 2024-04-05T21:26:47.060 INFO:tasks.workunit.client.1.smithi188.stdout:6/915: write f5 [608199,98569] 0 2024-04-05T21:26:47.060 INFO:tasks.workunit.client.0.smithi121.stdout:8/564: dread fa [0,4194304] 0 2024-04-05T21:26:47.060 INFO:tasks.workunit.client.0.smithi121.stdout:7/696: dread d0/f42 [0,4194304] 0 2024-04-05T21:26:47.061 INFO:tasks.workunit.client.0.smithi121.stdout:2/557: mkdir d2/daa 0 2024-04-05T21:26:47.061 INFO:tasks.workunit.client.0.smithi121.stdout:7/697: chown d0/d32/d7c 2658565 1 2024-04-05T21:26:47.061 INFO:tasks.workunit.client.1.smithi188.stdout:5/855: truncate d6/d82/d40/d42/f5e 1777014 0 2024-04-05T21:26:47.062 INFO:tasks.workunit.client.0.smithi121.stdout:7/698: chown d0/d9/f50 92799231 1 2024-04-05T21:26:47.063 INFO:tasks.workunit.client.0.smithi121.stdout:8/565: creat d12/d50/d6a/d63/d9c/fb1 x:0 0 0 2024-04-05T21:26:47.064 INFO:tasks.workunit.client.0.smithi121.stdout:8/566: read fa [208785,55487] 0 2024-04-05T21:26:47.064 INFO:tasks.workunit.client.0.smithi121.stdout:7/699: chown d0/d9/c10 2 1 2024-04-05T21:26:47.065 INFO:tasks.workunit.client.1.smithi188.stdout:2/844: write d5/db/dd/d73/d109/d5d/f5e [2357604,27404] 0 2024-04-05T21:26:47.065 INFO:tasks.workunit.client.1.smithi188.stdout:6/916: fsync d6/d13/d18/d19/f7e 0 2024-04-05T21:26:47.065 INFO:tasks.workunit.client.0.smithi121.stdout:7/700: stat d0/d23/d5a/f63 0 2024-04-05T21:26:47.067 INFO:tasks.workunit.client.0.smithi121.stdout:2/558: rename d2/d9/d17/d3a/f6f to d2/d9/d1c/d30/fab 0 2024-04-05T21:26:47.068 INFO:tasks.workunit.client.0.smithi121.stdout:8/567: truncate d12/d13/d56/d57/d5e/fa6 32747 0 2024-04-05T21:26:47.068 INFO:tasks.workunit.client.0.smithi121.stdout:2/559: stat d2/d9/d17/d3a/c83 0 2024-04-05T21:26:47.070 INFO:tasks.workunit.client.0.smithi121.stdout:8/568: rmdir d12/d50/d6a/d63 39 2024-04-05T21:26:47.070 INFO:tasks.workunit.client.1.smithi188.stdout:6/917: symlink d6/d13/d18/d19/l11a 0 2024-04-05T21:26:47.071 INFO:tasks.workunit.client.0.smithi121.stdout:2/560: symlink d2/daa/lac 0 2024-04-05T21:26:47.072 INFO:tasks.workunit.client.1.smithi188.stdout:6/918: chown d6/d13/d18/d3c/ddc/c115 195585204 1 2024-04-05T21:26:47.072 INFO:tasks.workunit.client.1.smithi188.stdout:2/845: rename d5/db/dd/d73/d109/f81 to d5/db/dd/d73/d109/d66/d8f/dcb/f118 0 2024-04-05T21:26:47.073 INFO:tasks.workunit.client.1.smithi188.stdout:6/919: chown d6/f103 293077201 1 2024-04-05T21:26:47.074 INFO:tasks.workunit.client.1.smithi188.stdout:5/856: truncate d6/d82/d40/d42/d81/d10c/f117 306983 0 2024-04-05T21:26:47.077 INFO:tasks.workunit.client.1.smithi188.stdout:2/846: mknod d5/db/dd/d73/d109/d66/d8f/dcb/dce/c119 0 2024-04-05T21:26:47.078 INFO:tasks.workunit.client.1.smithi188.stdout:6/920: fdatasync d6/d13/f1d 0 2024-04-05T21:26:47.079 INFO:tasks.workunit.client.0.smithi121.stdout:2/561: dwrite d2/d9/d17/f42 [0,4194304] 0 2024-04-05T21:26:47.080 INFO:tasks.workunit.client.1.smithi188.stdout:1/878: dwrite d1/dc/d16/d21/f26 [0,4194304] 0 2024-04-05T21:26:47.083 INFO:tasks.workunit.client.0.smithi121.stdout:2/562: creat d2/d9/d38/d2d/d4c/d79/fad x:0 0 0 2024-04-05T21:26:47.085 INFO:tasks.workunit.client.0.smithi121.stdout:7/701: dread d0/d32/d4a/f53 [0,4194304] 0 2024-04-05T21:26:47.086 INFO:tasks.workunit.client.1.smithi188.stdout:5/857: dread d6/f9a [0,4194304] 0 2024-04-05T21:26:47.088 INFO:tasks.workunit.client.1.smithi188.stdout:6/921: rename d6/d13/d18/d19/d2c/l7d to d6/d13/d18/d19/db7/l11b 0 2024-04-05T21:26:47.089 INFO:tasks.workunit.client.1.smithi188.stdout:1/879: mknod d1/dc/d16/d46/df7/c117 0 2024-04-05T21:26:47.092 INFO:tasks.workunit.client.0.smithi121.stdout:7/702: creat d0/d23/d5b/fdb x:0 0 0 2024-04-05T21:26:47.092 INFO:tasks.workunit.client.0.smithi121.stdout:7/703: stat d0/d9/d22/l71 0 2024-04-05T21:26:47.094 INFO:tasks.workunit.client.0.smithi121.stdout:2/563: creat d2/d9/d1c/fae x:0 0 0 2024-04-05T21:26:47.094 INFO:tasks.workunit.client.1.smithi188.stdout:1/880: readlink d1/dc/d16/d21/d27/l3e 0 2024-04-05T21:26:47.095 INFO:tasks.workunit.client.1.smithi188.stdout:6/922: creat d6/d13/d18/d3c/dd2/dd6/f11c x:0 0 0 2024-04-05T21:26:47.095 INFO:tasks.workunit.client.1.smithi188.stdout:2/847: dwrite d5/db/dd/d36/d3c/d63/d75/d87/ff6 [0,4194304] 0 2024-04-05T21:26:47.097 INFO:tasks.workunit.client.0.smithi121.stdout:8/569: dread d12/d13/f20 [0,4194304] 0 2024-04-05T21:26:47.097 INFO:tasks.workunit.client.0.smithi121.stdout:7/704: dread d0/f15 [0,4194304] 0 2024-04-05T21:26:47.097 INFO:tasks.workunit.client.1.smithi188.stdout:1/881: stat d1/dc/d16/d21/d27/d49/cfb 0 2024-04-05T21:26:47.100 INFO:tasks.workunit.client.0.smithi121.stdout:8/570: creat d12/d50/d93/d96/fb2 x:0 0 0 2024-04-05T21:26:47.100 INFO:tasks.workunit.client.1.smithi188.stdout:6/923: mkdir d6/d13/d18/d19/d2c/d11d 0 2024-04-05T21:26:47.102 INFO:tasks.workunit.client.0.smithi121.stdout:8/571: dread - d12/d50/d6a/d63/d8b/f9b zero size 2024-04-05T21:26:47.103 INFO:tasks.workunit.client.0.smithi121.stdout:7/705: truncate d0/d9/d22/d28/d56/f64 787141 0 2024-04-05T21:26:47.103 INFO:tasks.workunit.client.0.smithi121.stdout:8/572: readlink d12/d13/d15/l16 0 2024-04-05T21:26:47.104 INFO:tasks.workunit.client.1.smithi188.stdout:1/882: link d1/dc/d16/d2d/f9a d1/dc/d16/d21/d27/d4a/f118 0 2024-04-05T21:26:47.105 INFO:tasks.workunit.client.0.smithi121.stdout:2/564: write d2/d9/d1c/d30/d59/f63 [772570,13457] 0 2024-04-05T21:26:47.107 INFO:tasks.workunit.client.0.smithi121.stdout:7/706: creat d0/d23/d5b/d52/d61/dcb/fdc x:0 0 0 2024-04-05T21:26:47.107 INFO:tasks.workunit.client.0.smithi121.stdout:2/565: readlink d2/d9/d1c/d30/l90 0 2024-04-05T21:26:47.107 INFO:tasks.workunit.client.0.smithi121.stdout:8/573: creat d12/d50/fb3 x:0 0 0 2024-04-05T21:26:47.108 INFO:tasks.workunit.client.0.smithi121.stdout:2/566: chown d2/d9/f21 6023720 1 2024-04-05T21:26:47.108 INFO:tasks.workunit.client.0.smithi121.stdout:7/707: write d0/d32/d7c/fac [3659378,61807] 0 2024-04-05T21:26:47.110 INFO:tasks.workunit.client.1.smithi188.stdout:5/858: write d6/d82/f50 [1802435,106899] 0 2024-04-05T21:26:47.113 INFO:tasks.workunit.client.1.smithi188.stdout:5/859: dread - d6/d15/d5c/d7a/d44/d98/dbf/fe0 zero size 2024-04-05T21:26:47.115 INFO:tasks.workunit.client.0.smithi121.stdout:7/708: rename d0/d9/d22/d3e/d9c/da7/fbf to d0/d32/d7c/fdd 0 2024-04-05T21:26:47.115 INFO:tasks.workunit.client.0.smithi121.stdout:8/574: dwrite d12/d50/f79 [0,4194304] 0 2024-04-05T21:26:47.117 INFO:tasks.workunit.client.1.smithi188.stdout:2/848: dread d5/db/dd/d36/d3c/d48/d7a/d7c/f86 [0,4194304] 0 2024-04-05T21:26:47.118 INFO:tasks.workunit.client.1.smithi188.stdout:6/924: write d6/d13/d18/d19/d61/f8b [446545,52107] 0 2024-04-05T21:26:47.120 INFO:tasks.workunit.client.1.smithi188.stdout:1/883: truncate d1/dc/d16/d21/fa0 225181 0 2024-04-05T21:26:47.124 INFO:tasks.workunit.client.0.smithi121.stdout:8/575: link d12/ca0 d12/d13/d56/d57/d5e/cb4 0 2024-04-05T21:26:47.127 INFO:tasks.workunit.client.0.smithi121.stdout:8/576: mkdir d12/d50/d93/db5 0 2024-04-05T21:26:47.128 INFO:tasks.workunit.client.1.smithi188.stdout:5/860: dwrite d6/d17/d23/d7d/dcf/f103 [4194304,4194304] 0 2024-04-05T21:26:47.128 INFO:tasks.workunit.client.1.smithi188.stdout:6/925: link d6/d13/d18/d94/ff8 d6/d13/d18/d94/f11e 0 2024-04-05T21:26:47.130 INFO:tasks.workunit.client.1.smithi188.stdout:1/884: mkdir d1/dc/d16/d46/d88/d3f/d119 0 2024-04-05T21:26:47.130 INFO:tasks.workunit.client.0.smithi121.stdout:8/577: mknod d12/d13/cb6 0 2024-04-05T21:26:47.131 INFO:tasks.workunit.client.0.smithi121.stdout:8/578: chown d12/c35 737982 1 2024-04-05T21:26:47.133 INFO:tasks.workunit.client.0.smithi121.stdout:0/656: sync 2024-04-05T21:26:47.133 INFO:tasks.workunit.client.0.smithi121.stdout:5/738: sync 2024-04-05T21:26:47.134 INFO:tasks.workunit.client.0.smithi121.stdout:7/709: dwrite d0/d9/d22/d81/da2/fa5 [0,4194304] 0 2024-04-05T21:26:47.135 INFO:tasks.workunit.client.0.smithi121.stdout:0/657: stat d7/d62/d9d/fc0 0 2024-04-05T21:26:47.136 INFO:tasks.workunit.client.1.smithi188.stdout:2/849: write d5/db/dd/d36/d3c/d48/dcf/fd9 [1375228,122730] 0 2024-04-05T21:26:47.136 INFO:tasks.workunit.client.1.smithi188.stdout:5/861: dread - d6/d15/d5c/f9e zero size 2024-04-05T21:26:47.137 INFO:tasks.workunit.client.1.smithi188.stdout:2/850: chown d5/db/dd/d36/d3c/d48/d85/f94 278 1 2024-04-05T21:26:47.137 INFO:tasks.workunit.client.1.smithi188.stdout:1/885: mknod d1/dc/dfc/c11a 0 2024-04-05T21:26:47.142 INFO:tasks.workunit.client.1.smithi188.stdout:5/862: mknod d6/d15/d5c/d7a/d44/d9b/c127 0 2024-04-05T21:26:47.142 INFO:tasks.workunit.client.1.smithi188.stdout:5/863: readlink d6/d15/d5c/lc8 0 2024-04-05T21:26:47.142 INFO:tasks.workunit.client.0.smithi121.stdout:0/658: dread d7/f3a [0,4194304] 0 2024-04-05T21:26:47.143 INFO:tasks.workunit.client.0.smithi121.stdout:2/567: dread d2/d9/d56/f76 [0,4194304] 0 2024-04-05T21:26:47.143 INFO:tasks.workunit.client.0.smithi121.stdout:7/710: creat d0/d32/d4e/d73/dbc/fde x:0 0 0 2024-04-05T21:26:47.143 INFO:tasks.workunit.client.0.smithi121.stdout:8/579: getdents d12/d50/d61/d7f 0 2024-04-05T21:26:47.143 INFO:tasks.workunit.client.0.smithi121.stdout:5/739: getdents dd/de/d19 0 2024-04-05T21:26:47.145 INFO:tasks.workunit.client.1.smithi188.stdout:2/851: symlink d5/db/dd/da4/l11a 0 2024-04-05T21:26:47.146 INFO:tasks.workunit.client.1.smithi188.stdout:6/926: getdents d6/d13/d18/d3c/daf 0 2024-04-05T21:26:47.149 INFO:tasks.workunit.client.0.smithi121.stdout:0/659: mkdir d7/d71/d29/de8 0 2024-04-05T21:26:47.149 INFO:tasks.workunit.client.1.smithi188.stdout:5/864: rename d6/d17/d23/d7d/dc4 to d6/d17/d23/d7d/d128 0 2024-04-05T21:26:47.149 INFO:tasks.workunit.client.0.smithi121.stdout:2/568: dwrite d2/d9/d17/f19 [0,4194304] 0 2024-04-05T21:26:47.151 INFO:tasks.workunit.client.0.smithi121.stdout:7/711: mkdir d0/d9/d22/d28/d56/db9/ddf 0 2024-04-05T21:26:47.151 INFO:tasks.workunit.client.0.smithi121.stdout:5/740: dread - dd/de/d19/d1c/d36/f71 zero size 2024-04-05T21:26:47.151 INFO:tasks.workunit.client.1.smithi188.stdout:1/886: rename d1/dc/d16/d21/d27/d4a/c5e to d1/dc/d16/d46/d95/d99/c11b 0 2024-04-05T21:26:47.153 INFO:tasks.workunit.client.0.smithi121.stdout:7/712: write d0/d32/d4e/fb6 [904295,18373] 0 2024-04-05T21:26:47.153 INFO:tasks.workunit.client.0.smithi121.stdout:5/741: chown dd/de/d39/lfd 42 1 2024-04-05T21:26:47.155 INFO:tasks.workunit.client.0.smithi121.stdout:8/580: link d12/c35 d12/d50/d93/db5/cb7 0 2024-04-05T21:26:47.156 INFO:tasks.workunit.client.0.smithi121.stdout:0/660: mknod d7/d62/ddb/ce9 0 2024-04-05T21:26:47.159 INFO:tasks.workunit.client.0.smithi121.stdout:5/742: dread dd/d41/dce/da6/dd3/db1/db9/fd0 [0,4194304] 0 2024-04-05T21:26:47.162 INFO:tasks.workunit.client.0.smithi121.stdout:5/743: read dd/de/d39/d3b/fa1 [4070230,88825] 0 2024-04-05T21:26:47.163 INFO:tasks.workunit.client.0.smithi121.stdout:8/581: dwrite d12/f73 [0,4194304] 0 2024-04-05T21:26:47.165 INFO:tasks.workunit.client.1.smithi188.stdout:1/887: fsync d1/dc/f11 0 2024-04-05T21:26:47.166 INFO:tasks.workunit.client.1.smithi188.stdout:2/852: dwrite d5/db/dd/d73/d109/f7b [4194304,4194304] 0 2024-04-05T21:26:47.166 INFO:tasks.workunit.client.1.smithi188.stdout:5/865: rename d6/d106/f6e to d6/d15/d1b/ddc/f129 0 2024-04-05T21:26:47.166 INFO:tasks.workunit.client.1.smithi188.stdout:5/866: chown d6/d17/d23/d7d/d128/l101 43426 1 2024-04-05T21:26:47.166 INFO:tasks.workunit.client.1.smithi188.stdout:2/853: chown d5/db/dd/d36/d3c/d48/d7a/d7c/db5/cdd 122180 1 2024-04-05T21:26:47.166 INFO:tasks.workunit.client.1.smithi188.stdout:5/867: write d6/d82/f89 [1013563,130766] 0 2024-04-05T21:26:47.168 INFO:tasks.workunit.client.0.smithi121.stdout:0/661: dwrite d7/d7e/f89 [0,4194304] 0 2024-04-05T21:26:47.168 INFO:tasks.workunit.client.0.smithi121.stdout:5/744: mkdir dd/d41/dce/da6/d10f 0 2024-04-05T21:26:47.168 INFO:tasks.workunit.client.0.smithi121.stdout:7/713: dread d0/d32/f6b [4194304,4194304] 0 2024-04-05T21:26:47.169 INFO:tasks.workunit.client.1.smithi188.stdout:1/888: write d1/dc/d16/d46/de5/f10b [261646,20781] 0 2024-04-05T21:26:47.171 INFO:tasks.workunit.client.1.smithi188.stdout:5/868: symlink d6/d15/d5c/d7a/d44/d98/dbf/l12a 0 2024-04-05T21:26:47.171 INFO:tasks.workunit.client.0.smithi121.stdout:7/714: fsync d0/d9/d22/fa3 0 2024-04-05T21:26:47.172 INFO:tasks.workunit.client.1.smithi188.stdout:5/869: stat d6/d15/lf6 0 2024-04-05T21:26:47.174 INFO:tasks.workunit.client.1.smithi188.stdout:2/854: rename d5/db/d35/f100 to d5/db/dd/d73/d109/d8c/f11b 0 2024-04-05T21:26:47.174 INFO:tasks.workunit.client.0.smithi121.stdout:7/715: creat d0/d23/d5b/fe0 x:0 0 0 2024-04-05T21:26:47.175 INFO:tasks.workunit.client.0.smithi121.stdout:8/582: dwrite d12/d13/d15/f83 [0,4194304] 0 2024-04-05T21:26:47.178 INFO:tasks.workunit.client.1.smithi188.stdout:1/889: chown d1/dc/d16/d46/d88/d3f/c71 12100009 1 2024-04-05T21:26:47.180 INFO:tasks.workunit.client.1.smithi188.stdout:6/927: truncate d6/d13/d18/d19/f27 2234296 0 2024-04-05T21:26:47.180 INFO:tasks.workunit.client.1.smithi188.stdout:5/870: symlink d6/d17/d23/d7d/l12b 0 2024-04-05T21:26:47.180 INFO:tasks.workunit.client.1.smithi188.stdout:2/855: truncate d5/db/dd/fc6 443127 0 2024-04-05T21:26:47.182 INFO:tasks.workunit.client.0.smithi121.stdout:5/745: read dd/de/d19/d38/f97 [907459,17789] 0 2024-04-05T21:26:47.186 INFO:tasks.workunit.client.0.smithi121.stdout:8/583: dwrite d12/d13/d15/d1d/fa9 [0,4194304] 0 2024-04-05T21:26:47.187 INFO:tasks.workunit.client.0.smithi121.stdout:7/716: dwrite d0/d32/d4e/d73/fca [0,4194304] 0 2024-04-05T21:26:47.188 INFO:tasks.workunit.client.0.smithi121.stdout:5/746: unlink dd/de/l4b 0 2024-04-05T21:26:47.189 INFO:tasks.workunit.client.1.smithi188.stdout:6/928: write d6/d13/d18/d3c/ddc/f9c [4024604,48228] 0 2024-04-05T21:26:47.189 INFO:tasks.workunit.client.0.smithi121.stdout:5/747: chown dd/de/ced 32459562 1 2024-04-05T21:26:47.191 INFO:tasks.workunit.client.0.smithi121.stdout:7/717: mkdir d0/d9/d22/d3e/d76/de1 0 2024-04-05T21:26:47.193 INFO:tasks.workunit.client.0.smithi121.stdout:7/718: readlink d0/d9/l94 0 2024-04-05T21:26:47.193 INFO:tasks.workunit.client.0.smithi121.stdout:5/748: rmdir dd/de/d19/d1c/d7b 39 2024-04-05T21:26:47.194 INFO:tasks.workunit.client.1.smithi188.stdout:2/856: dwrite d5/db/dd/d36/d3c/d48/d85/f94 [0,4194304] 0 2024-04-05T21:26:47.195 INFO:tasks.workunit.client.0.smithi121.stdout:8/584: creat d12/d50/d6a/fb8 x:0 0 0 2024-04-05T21:26:47.199 INFO:tasks.workunit.client.0.smithi121.stdout:2/569: dwrite d2/d9/d1c/d30/f68 [0,4194304] 0 2024-04-05T21:26:47.202 INFO:tasks.workunit.client.0.smithi121.stdout:7/719: rename d0/d23/d5b/d52/fd8 to d0/d23/d5b/d52/d61/dcb/fe2 0 2024-04-05T21:26:47.204 INFO:tasks.workunit.client.0.smithi121.stdout:2/570: creat d2/d1e/faf x:0 0 0 2024-04-05T21:26:47.205 INFO:tasks.workunit.client.1.smithi188.stdout:5/871: dread d6/dad/f8d [0,4194304] 0 2024-04-05T21:26:47.206 INFO:tasks.workunit.client.0.smithi121.stdout:7/720: unlink d0/d23/c85 0 2024-04-05T21:26:47.210 INFO:tasks.workunit.client.0.smithi121.stdout:2/571: chown d2/d9/d1c/d30/f61 61499 1 2024-04-05T21:26:47.210 INFO:tasks.workunit.client.0.smithi121.stdout:7/721: chown d0/d23/d5b/f43 18 1 2024-04-05T21:26:47.211 INFO:tasks.workunit.client.1.smithi188.stdout:2/857: truncate d5/db/d35/d3b/d6b/f7e 1859429 0 2024-04-05T21:26:47.211 INFO:tasks.workunit.client.0.smithi121.stdout:2/572: readlink d2/d9/d38/d2d/d47/d94/d84/l9e 0 2024-04-05T21:26:47.212 INFO:tasks.workunit.client.0.smithi121.stdout:5/749: dwrite dd/f6c [0,4194304] 0 2024-04-05T21:26:47.216 INFO:tasks.workunit.client.0.smithi121.stdout:5/750: unlink dd/de/d19/d3c/fdf 0 2024-04-05T21:26:47.217 INFO:tasks.workunit.client.0.smithi121.stdout:2/573: creat d2/fb0 x:0 0 0 2024-04-05T21:26:47.221 INFO:tasks.workunit.client.0.smithi121.stdout:5/751: creat dd/de/d19/d1c/d7b/f110 x:0 0 0 2024-04-05T21:26:47.222 INFO:tasks.workunit.client.0.smithi121.stdout:5/752: readlink dd/de/d39/ldd 0 2024-04-05T21:26:47.225 INFO:tasks.workunit.client.0.smithi121.stdout:2/574: dwrite d2/d1e/faf [0,4194304] 0 2024-04-05T21:26:47.225 INFO:tasks.workunit.client.0.smithi121.stdout:5/753: write dd/de/d19/d38/d56/f92 [635562,111670] 0 2024-04-05T21:26:47.229 INFO:tasks.workunit.client.0.smithi121.stdout:5/754: mknod dd/d41/dce/da6/dd3/de1/c111 0 2024-04-05T21:26:47.230 INFO:tasks.workunit.client.1.smithi188.stdout:1/890: dread d1/dc/d16/d46/d88/d13/f6a [0,4194304] 0 2024-04-05T21:26:47.230 INFO:tasks.workunit.client.0.smithi121.stdout:2/575: truncate d2/d9/d1c/d30/f5e 215918 0 2024-04-05T21:26:47.231 INFO:tasks.workunit.client.1.smithi188.stdout:1/891: readlink d1/dc/d16/d46/d88/d13/lae 0 2024-04-05T21:26:47.232 INFO:tasks.workunit.client.0.smithi121.stdout:2/576: creat d2/d9/d1c/d30/fb1 x:0 0 0 2024-04-05T21:26:47.233 INFO:tasks.workunit.client.0.smithi121.stdout:5/755: write dd/d41/dce/da6/dd3/db1/db9/fd0 [180703,126448] 0 2024-04-05T21:26:47.234 INFO:tasks.workunit.client.0.smithi121.stdout:5/756: fsync dd/de/d19/d3c/f5f 0 2024-04-05T21:26:47.235 INFO:tasks.workunit.client.0.smithi121.stdout:2/577: symlink d2/d9/d1c/d30/lb2 0 2024-04-05T21:26:47.235 INFO:tasks.workunit.client.1.smithi188.stdout:1/892: rename d1/dc/d16/d46/df7/de2 to d1/dc/d16/d46/d88/d86/d11c 0 2024-04-05T21:26:47.237 INFO:tasks.workunit.client.0.smithi121.stdout:8/585: dread d12/d13/d15/d1d/fa3 [0,4194304] 0 2024-04-05T21:26:47.240 INFO:tasks.workunit.client.1.smithi188.stdout:1/893: getdents d1/dc/d16/d2d/dd5 0 2024-04-05T21:26:47.241 INFO:tasks.workunit.client.1.smithi188.stdout:6/929: dread d6/f3b [0,4194304] 0 2024-04-05T21:26:47.247 INFO:tasks.workunit.client.1.smithi188.stdout:1/894: fdatasync d1/dc/d16/f6f 0 2024-04-05T21:26:47.247 INFO:tasks.workunit.client.0.smithi121.stdout:8/586: getdents d12/d50/d61 0 2024-04-05T21:26:47.247 INFO:tasks.workunit.client.1.smithi188.stdout:6/930: mknod d6/d13/c11f 0 2024-04-05T21:26:47.248 INFO:tasks.workunit.client.0.smithi121.stdout:8/587: fsync d12/d50/f79 0 2024-04-05T21:26:47.250 INFO:tasks.workunit.client.1.smithi188.stdout:1/895: mknod d1/db2/c11d 0 2024-04-05T21:26:47.251 INFO:tasks.workunit.client.0.smithi121.stdout:0/662: dwrite d7/dc/f15 [0,4194304] 0 2024-04-05T21:26:47.253 INFO:tasks.workunit.client.0.smithi121.stdout:5/757: dwrite dd/d41/dce/ff5 [0,4194304] 0 2024-04-05T21:26:47.255 INFO:tasks.workunit.client.0.smithi121.stdout:2/578: dwrite d2/d9/d38/d2d/d4c/fa6 [0,4194304] 0 2024-04-05T21:26:47.257 INFO:tasks.workunit.client.0.smithi121.stdout:2/579: write d2/d9/d56/d62/f7a [2674804,74644] 0 2024-04-05T21:26:47.257 INFO:tasks.workunit.client.0.smithi121.stdout:0/663: read - d7/d71/d29/d3f/fde zero size 2024-04-05T21:26:47.259 INFO:tasks.workunit.client.0.smithi121.stdout:8/588: fsync d12/d13/d15/d1d/f4f 0 2024-04-05T21:26:47.262 INFO:tasks.workunit.client.0.smithi121.stdout:5/758: readlink dd/l2c 0 2024-04-05T21:26:47.264 INFO:tasks.workunit.client.1.smithi188.stdout:5/872: dwrite d6/d15/d5c/d7a/f52 [0,4194304] 0 2024-04-05T21:26:47.264 INFO:tasks.workunit.client.1.smithi188.stdout:6/931: dwrite d6/d13/d18/d3c/ddc/d60/fbc [0,4194304] 0 2024-04-05T21:26:47.265 INFO:tasks.workunit.client.0.smithi121.stdout:2/580: unlink d2/d9/f54 0 2024-04-05T21:26:47.265 INFO:tasks.workunit.client.0.smithi121.stdout:5/759: stat dd/de/d19/d1c/d7b/f110 0 2024-04-05T21:26:47.266 INFO:tasks.workunit.client.0.smithi121.stdout:7/722: dwrite d0/d9/d22/d28/d3b/f4f [0,4194304] 0 2024-04-05T21:26:47.267 INFO:tasks.workunit.client.1.smithi188.stdout:2/858: dread d5/db/dd/da4/fc2 [0,4194304] 0 2024-04-05T21:26:47.273 INFO:tasks.workunit.client.0.smithi121.stdout:2/581: rename d2/d9/d17/d3a/c81 to d2/d9/d56/d5c/cb3 0 2024-04-05T21:26:47.274 INFO:tasks.workunit.client.1.smithi188.stdout:6/932: fdatasync d6/d7/f14 0 2024-04-05T21:26:47.278 INFO:tasks.workunit.client.0.smithi121.stdout:2/582: symlink d2/d9/d38/lb4 0 2024-04-05T21:26:47.280 INFO:tasks.workunit.client.0.smithi121.stdout:2/583: creat d2/daa/fb5 x:0 0 0 2024-04-05T21:26:47.282 INFO:tasks.workunit.client.1.smithi188.stdout:6/933: write d6/d13/d18/d19/d2c/db2/ff1 [61562,22468] 0 2024-04-05T21:26:47.282 INFO:tasks.workunit.client.1.smithi188.stdout:6/934: dread - d6/d13/d18/d19/d2c/d34/f6b zero size 2024-04-05T21:26:47.282 INFO:tasks.workunit.client.1.smithi188.stdout:1/896: dread d1/dc/d16/d46/d88/d13/d60/fd0 [0,4194304] 0 2024-04-05T21:26:47.282 INFO:tasks.workunit.client.1.smithi188.stdout:2/859: mknod d5/db/dd/d24/dc8/c11c 0 2024-04-05T21:26:47.287 INFO:tasks.workunit.client.1.smithi188.stdout:1/897: symlink d1/dc/d16/d46/l11e 0 2024-04-05T21:26:47.287 INFO:tasks.workunit.client.1.smithi188.stdout:2/860: truncate d5/db/f76 1792014 0 2024-04-05T21:26:47.289 INFO:tasks.workunit.client.1.smithi188.stdout:5/873: getdents d6/d17 0 2024-04-05T21:26:47.290 INFO:tasks.workunit.client.1.smithi188.stdout:1/898: creat d1/dc/d16/d46/d95/d99/f11f x:0 0 0 2024-04-05T21:26:47.291 INFO:tasks.workunit.client.1.smithi188.stdout:2/861: truncate d5/db/dd/d36/d3c/d48/d7a/d7c/fe5 825356 0 2024-04-05T21:26:47.291 INFO:tasks.workunit.client.0.smithi121.stdout:6/498: sync 2024-04-05T21:26:47.294 INFO:tasks.workunit.client.0.smithi121.stdout:8/589: dread d12/d13/d15/f21 [0,4194304] 0 2024-04-05T21:26:47.294 INFO:tasks.workunit.client.1.smithi188.stdout:2/862: mkdir d5/db/dd/d73/d109/d8c/d11d 0 2024-04-05T21:26:47.295 INFO:tasks.workunit.client.1.smithi188.stdout:1/899: creat d1/dc/d16/d46/d95/f120 x:0 0 0 2024-04-05T21:26:47.297 INFO:tasks.workunit.client.1.smithi188.stdout:2/863: symlink d5/db/dd/d36/d3c/d48/dd7/l11e 0 2024-04-05T21:26:47.298 INFO:tasks.workunit.client.1.smithi188.stdout:1/900: symlink d1/dc/d16/d46/d95/l121 0 2024-04-05T21:26:47.300 INFO:tasks.workunit.client.1.smithi188.stdout:2/864: unlink d5/f22 0 2024-04-05T21:26:47.302 INFO:tasks.workunit.client.0.smithi121.stdout:8/590: dread d12/d13/d56/d57/da4/f40 [0,4194304] 0 2024-04-05T21:26:47.303 INFO:tasks.workunit.client.1.smithi188.stdout:2/865: stat d5/db/dd/d36/d3c/d48/ld1 0 2024-04-05T21:26:47.303 INFO:tasks.workunit.client.1.smithi188.stdout:6/935: dread d6/d13/d18/d19/d2c/d3f/f56 [0,4194304] 0 2024-04-05T21:26:47.304 INFO:tasks.workunit.client.0.smithi121.stdout:8/591: mkdir d12/d13/d56/d57/d5e/db9 0 2024-04-05T21:26:47.304 INFO:tasks.workunit.client.0.smithi121.stdout:8/592: readlink d12/d13/d15/l82 0 2024-04-05T21:26:47.305 INFO:tasks.workunit.client.1.smithi188.stdout:6/936: fsync d6/d13/d18/d19/da3/fca 0 2024-04-05T21:26:47.307 INFO:tasks.workunit.client.0.smithi121.stdout:8/593: link d12/d13/d15/d1d/c4e d12/d50/d93/d96/cba 0 2024-04-05T21:26:47.308 INFO:tasks.workunit.client.1.smithi188.stdout:6/937: mkdir d6/d13/d18/d19/dc8/da4/dfa/d120 0 2024-04-05T21:26:47.312 INFO:tasks.workunit.client.0.smithi121.stdout:2/584: getdents d2/d9 0 2024-04-05T21:26:47.313 INFO:tasks.workunit.client.0.smithi121.stdout:0/664: truncate d7/d71/fdc 186951 0 2024-04-05T21:26:47.316 INFO:tasks.workunit.client.0.smithi121.stdout:8/594: getdents d12 0 2024-04-05T21:26:47.316 INFO:tasks.workunit.client.0.smithi121.stdout:2/585: mkdir d2/db6 0 2024-04-05T21:26:47.317 INFO:tasks.workunit.client.1.smithi188.stdout:1/901: write d1/dc/d16/d2d/fd1 [932351,119600] 0 2024-04-05T21:26:47.318 INFO:tasks.workunit.client.0.smithi121.stdout:5/760: dwrite dd/de/d19/d13/f3f [0,4194304] 0 2024-04-05T21:26:47.318 INFO:tasks.workunit.client.0.smithi121.stdout:8/595: fdatasync d12/d50/d6a/f90 0 2024-04-05T21:26:47.319 INFO:tasks.workunit.client.0.smithi121.stdout:0/665: symlink d7/d71/d29/d2a/lea 0 2024-04-05T21:26:47.320 INFO:tasks.workunit.client.1.smithi188.stdout:5/874: dwrite d6/d15/d5c/d7a/fd0 [4194304,4194304] 0 2024-04-05T21:26:47.323 INFO:tasks.workunit.client.1.smithi188.stdout:6/938: rename d6/d13/d18/d19/d2c/d3f/c68 to d6/d13/d18/d19/c121 0 2024-04-05T21:26:47.323 INFO:tasks.workunit.client.0.smithi121.stdout:2/586: rename d2/daa/lac to d2/d9/d77/lb7 0 2024-04-05T21:26:47.324 INFO:tasks.workunit.client.0.smithi121.stdout:0/666: write d7/dc/da3/fb1 [131941,110835] 0 2024-04-05T21:26:47.325 INFO:tasks.workunit.client.0.smithi121.stdout:8/596: truncate d12/d13/d15/d1d/f4c 721065 0 2024-04-05T21:26:47.326 INFO:tasks.workunit.client.1.smithi188.stdout:1/902: mknod d1/dc/d16/d21/d27/d4a/dc1/c122 0 2024-04-05T21:26:47.327 INFO:tasks.workunit.client.1.smithi188.stdout:6/939: mknod d6/d13/d18/d19/d2c/c122 0 2024-04-05T21:26:47.328 INFO:tasks.workunit.client.1.smithi188.stdout:5/875: mkdir d6/d82/d40/d12c 0 2024-04-05T21:26:47.330 INFO:tasks.workunit.client.0.smithi121.stdout:0/667: write d7/d71/d45/d5a/f80 [12536,81249] 0 2024-04-05T21:26:47.330 INFO:tasks.workunit.client.1.smithi188.stdout:6/940: write d6/d13/d18/d19/d2c/db2/ff1 [713234,117948] 0 2024-04-05T21:26:47.333 INFO:tasks.workunit.client.1.smithi188.stdout:2/866: dread d5/db/dd/d24/f52 [0,4194304] 0 2024-04-05T21:26:47.334 INFO:tasks.workunit.client.0.smithi121.stdout:2/587: dread d2/f92 [0,4194304] 0 2024-04-05T21:26:47.335 INFO:tasks.workunit.client.1.smithi188.stdout:2/867: write d5/db/dd/d73/d109/f7b [3878470,115146] 0 2024-04-05T21:26:47.336 INFO:tasks.workunit.client.0.smithi121.stdout:8/597: dwrite d12/d13/d15/f18 [0,4194304] 0 2024-04-05T21:26:47.338 INFO:tasks.workunit.client.0.smithi121.stdout:5/761: truncate dd/de/d19/d1c/d36/d46/f9e 3693553 0 2024-04-05T21:26:47.338 INFO:tasks.workunit.client.0.smithi121.stdout:8/598: fdatasync d12/d13/d56/fa2 0 2024-04-05T21:26:47.341 INFO:tasks.workunit.client.0.smithi121.stdout:2/588: dread d2/d1e/faf [0,4194304] 0 2024-04-05T21:26:47.341 INFO:tasks.workunit.client.0.smithi121.stdout:5/762: creat dd/de/d19/d38/d9b/f112 x:0 0 0 2024-04-05T21:26:47.342 INFO:tasks.workunit.client.0.smithi121.stdout:8/599: mkdir d12/d50/dbb 0 2024-04-05T21:26:47.344 INFO:tasks.workunit.client.0.smithi121.stdout:2/589: unlink d2/d9/d17/l27 0 2024-04-05T21:26:47.345 INFO:tasks.workunit.client.0.smithi121.stdout:8/600: rmdir d12/d13/d56/d57/d5e 39 2024-04-05T21:26:47.350 INFO:tasks.workunit.client.1.smithi188.stdout:6/941: creat d6/d13/d18/d19/d2c/db2/de4/f123 x:0 0 0 2024-04-05T21:26:47.350 INFO:tasks.workunit.client.0.smithi121.stdout:5/763: rename dd/de/d19/d1c/d36/c3d to dd/de/d19/c113 0 2024-04-05T21:26:47.350 INFO:tasks.workunit.client.0.smithi121.stdout:2/590: creat d2/d9/d38/d2d/d4c/d79/fb8 x:0 0 0 2024-04-05T21:26:47.350 INFO:tasks.workunit.client.0.smithi121.stdout:2/591: chown d2/daa 1106905977 1 2024-04-05T21:26:47.351 INFO:tasks.workunit.client.1.smithi188.stdout:6/942: creat d6/d7/df7/f124 x:0 0 0 2024-04-05T21:26:47.352 INFO:tasks.workunit.client.0.smithi121.stdout:5/764: chown dd/lac 249692630 1 2024-04-05T21:26:47.353 INFO:tasks.workunit.client.0.smithi121.stdout:2/592: write d2/d9/d1c/fae [366338,47481] 0 2024-04-05T21:26:47.355 INFO:tasks.workunit.client.0.smithi121.stdout:8/601: dwrite d12/d50/d93/d96/fb2 [0,4194304] 0 2024-04-05T21:26:47.355 INFO:tasks.workunit.client.1.smithi188.stdout:2/868: dread d5/db/dd/d73/d109/d9f/d10c/dcd/faf [0,4194304] 0 2024-04-05T21:26:47.357 INFO:tasks.workunit.client.0.smithi121.stdout:2/593: mkdir d2/d9/d38/d2d/d4c/d7c/d7e/db9 0 2024-04-05T21:26:47.359 INFO:tasks.workunit.client.1.smithi188.stdout:6/943: dwrite d6/d13/d18/d3c/ddc/d60/fbc [0,4194304] 0 2024-04-05T21:26:47.360 INFO:tasks.workunit.client.1.smithi188.stdout:2/869: chown d5/db/dd/d73/d109/d9f/d10c/dcd/cf0 5698133 1 2024-04-05T21:26:47.360 INFO:tasks.workunit.client.0.smithi121.stdout:2/594: readlink d2/d9/d1c/d85/l8a 0 2024-04-05T21:26:47.365 INFO:tasks.workunit.client.1.smithi188.stdout:6/944: rename d6/d28/cfd to d6/d13/d18/d19/d2c/db2/de4/c125 0 2024-04-05T21:26:47.366 INFO:tasks.workunit.client.0.smithi121.stdout:0/668: dwrite d7/d71/d29/d2a/d2c/f8c [0,4194304] 0 2024-04-05T21:26:47.366 INFO:tasks.workunit.client.0.smithi121.stdout:2/595: rename d2/daa to d2/d9/d1c/d30/dba 0 2024-04-05T21:26:47.367 INFO:tasks.workunit.client.0.smithi121.stdout:0/669: fdatasync d7/d71/d29/f37 0 2024-04-05T21:26:47.368 INFO:tasks.workunit.client.0.smithi121.stdout:8/602: link d12/d13/d15/d1d/l86 d12/d50/d93/d96/lbc 0 2024-04-05T21:26:47.371 INFO:tasks.workunit.client.0.smithi121.stdout:0/670: rmdir d7/d71/d45/d5a 39 2024-04-05T21:26:47.376 INFO:tasks.workunit.client.0.smithi121.stdout:0/671: write d7/d71/f26 [2538055,3431] 0 2024-04-05T21:26:47.376 INFO:tasks.workunit.client.0.smithi121.stdout:2/596: mkdir d2/db6/dbb 0 2024-04-05T21:26:47.377 INFO:tasks.workunit.client.1.smithi188.stdout:2/870: dwrite d5/db/dd/d36/d3c/d48/f5b [0,4194304] 0 2024-04-05T21:26:47.379 INFO:tasks.workunit.client.1.smithi188.stdout:1/903: write d1/dc/d16/f6f [2121747,15419] 0 2024-04-05T21:26:47.383 INFO:tasks.workunit.client.1.smithi188.stdout:1/904: dread - d1/dc/d16/d46/d95/d99/f11f zero size 2024-04-05T21:26:47.384 INFO:tasks.workunit.client.1.smithi188.stdout:5/876: dwrite d6/d15/d5c/f94 [0,4194304] 0 2024-04-05T21:26:47.392 INFO:tasks.workunit.client.0.smithi121.stdout:8/603: dwrite d12/d13/d15/d1d/fa3 [0,4194304] 0 2024-04-05T21:26:47.401 INFO:tasks.workunit.client.1.smithi188.stdout:5/877: read d6/d82/d40/d42/d81/d95/f9d [5542723,70900] 0 2024-04-05T21:26:47.403 INFO:tasks.workunit.client.1.smithi188.stdout:2/871: dread d5/db/dd/d24/f28 [0,4194304] 0 2024-04-05T21:26:47.405 INFO:tasks.workunit.client.1.smithi188.stdout:5/878: creat d6/d82/d40/d42/d81/d10c/f12d x:0 0 0 2024-04-05T21:26:47.408 INFO:tasks.workunit.client.0.smithi121.stdout:2/597: chown d2/d9/d56/d5c/c53 1077097 1 2024-04-05T21:26:47.408 INFO:tasks.workunit.client.1.smithi188.stdout:2/872: symlink d5/db/dd/d36/d3c/d48/d7a/d7c/l11f 0 2024-04-05T21:26:47.409 INFO:tasks.workunit.client.0.smithi121.stdout:2/598: read d2/d9/f75 [967499,126089] 0 2024-04-05T21:26:47.410 INFO:tasks.workunit.client.0.smithi121.stdout:2/599: chown d2/d9/d38/c48 0 1 2024-04-05T21:26:47.411 INFO:tasks.workunit.client.0.smithi121.stdout:5/765: creat dd/de/d19/d13/dd4/f114 x:0 0 0 2024-04-05T21:26:47.414 INFO:tasks.workunit.client.0.smithi121.stdout:2/600: mkdir d2/d9/d38/d2d/d4c/d7c/dbc 0 2024-04-05T21:26:47.415 INFO:tasks.workunit.client.0.smithi121.stdout:5/766: chown dd/de/d19/d38/d9b/f112 25 1 2024-04-05T21:26:47.418 INFO:tasks.workunit.client.0.smithi121.stdout:8/604: link d12/d13/f84 d12/d13/d15/fbd 0 2024-04-05T21:26:47.418 INFO:tasks.workunit.client.0.smithi121.stdout:2/601: fsync d2/d9/d17/d3a/f55 0 2024-04-05T21:26:47.421 INFO:tasks.workunit.client.0.smithi121.stdout:2/602: unlink d2/d9/ca 0 2024-04-05T21:26:47.422 INFO:tasks.workunit.client.0.smithi121.stdout:0/672: dread d7/d71/d29/d2a/d2c/f87 [0,4194304] 0 2024-04-05T21:26:47.423 INFO:tasks.workunit.client.0.smithi121.stdout:2/603: write d2/d1e/f24 [707601,12277] 0 2024-04-05T21:26:47.423 INFO:tasks.workunit.client.0.smithi121.stdout:0/673: chown d7/d7e/f89 185997869 1 2024-04-05T21:26:47.429 INFO:tasks.workunit.client.0.smithi121.stdout:5/767: creat dd/de/d19/d38/f115 x:0 0 0 2024-04-05T21:26:47.432 INFO:tasks.workunit.client.0.smithi121.stdout:2/604: rmdir d2/d9/d56 39 2024-04-05T21:26:47.438 INFO:tasks.workunit.client.0.smithi121.stdout:2/605: write d2/d9/d38/d2d/d4c/d79/fa2 [835331,39305] 0 2024-04-05T21:26:47.439 INFO:tasks.workunit.client.0.smithi121.stdout:0/674: link d7/d71/d29/d3f/f4f d7/d71/d29/de1/feb 0 2024-04-05T21:26:47.443 INFO:tasks.workunit.client.0.smithi121.stdout:2/606: rename d2/d28/c40 to d2/d9/d38/d2d/d4c/cbd 0 2024-04-05T21:26:47.447 INFO:tasks.workunit.client.0.smithi121.stdout:2/607: creat d2/d9/d1c/d30/d59/fbe x:0 0 0 2024-04-05T21:26:47.448 INFO:tasks.workunit.client.1.smithi188.stdout:6/945: write d6/d13/d18/d19/d61/fab [1981584,116312] 0 2024-04-05T21:26:47.450 INFO:tasks.workunit.client.1.smithi188.stdout:2/873: mkdir d5/db/dd/d73/d120 0 2024-04-05T21:26:47.451 INFO:tasks.workunit.client.1.smithi188.stdout:2/874: readlink d5/la9 0 2024-04-05T21:26:47.451 INFO:tasks.workunit.client.1.smithi188.stdout:5/879: dwrite d6/d82/f50 [0,4194304] 0 2024-04-05T21:26:47.454 INFO:tasks.workunit.client.1.smithi188.stdout:2/875: chown d5/d5f 2477508 1 2024-04-05T21:26:47.454 INFO:tasks.workunit.client.1.smithi188.stdout:6/946: fsync d6/d13/d18/f35 0 2024-04-05T21:26:47.454 INFO:tasks.workunit.client.1.smithi188.stdout:5/880: chown d6/d82/d40/d42/d81/da6/db2/cd5 1333367306 1 2024-04-05T21:26:47.455 INFO:tasks.workunit.client.1.smithi188.stdout:2/876: write d5/db/dd/d73/d109/fbd [708307,8759] 0 2024-04-05T21:26:47.456 INFO:tasks.workunit.client.0.smithi121.stdout:2/608: dwrite d2/d1e/f8c [0,4194304] 0 2024-04-05T21:26:47.457 INFO:tasks.workunit.client.1.smithi188.stdout:5/881: stat d6/d15/d5c/d7a/d44/c4b 0 2024-04-05T21:26:47.459 INFO:tasks.workunit.client.0.smithi121.stdout:9/499: sync 2024-04-05T21:26:47.459 INFO:tasks.workunit.client.0.smithi121.stdout:1/734: sync 2024-04-05T21:26:47.461 INFO:tasks.workunit.client.1.smithi188.stdout:2/877: rmdir d5/db/dd/d73 39 2024-04-05T21:26:47.462 INFO:tasks.workunit.client.1.smithi188.stdout:6/947: creat d6/d13/f126 x:0 0 0 2024-04-05T21:26:47.465 INFO:tasks.workunit.client.0.smithi121.stdout:1/735: write d8/d8c/db6/d1a/f72 [461256,53413] 0 2024-04-05T21:26:47.467 INFO:tasks.workunit.client.1.smithi188.stdout:2/878: read d5/db/dd/d73/d109/faa [1676310,34462] 0 2024-04-05T21:26:47.470 INFO:tasks.workunit.client.1.smithi188.stdout:6/948: chown d6/d13/d18/d19/d2c/d3f/d95/l10c 5724830 1 2024-04-05T21:26:47.471 INFO:tasks.workunit.client.0.smithi121.stdout:8/605: dwrite d12/d13/d15/d1d/f4c [0,4194304] 0 2024-04-05T21:26:47.472 INFO:tasks.workunit.client.1.smithi188.stdout:5/882: dwrite d6/d82/d40/d42/d81/d95/f9d [0,4194304] 0 2024-04-05T21:26:47.476 INFO:tasks.workunit.client.1.smithi188.stdout:2/879: creat d5/db/dd/d36/d3c/d48/f121 x:0 0 0 2024-04-05T21:26:47.477 INFO:tasks.workunit.client.1.smithi188.stdout:2/880: dread - d5/db/d35/d3b/f6f zero size 2024-04-05T21:26:47.479 INFO:tasks.workunit.client.0.smithi121.stdout:5/768: dwrite dd/f17 [0,4194304] 0 2024-04-05T21:26:47.479 INFO:tasks.workunit.client.0.smithi121.stdout:1/736: stat d8/d8c/db6/d1a/f20 0 2024-04-05T21:26:47.479 INFO:tasks.workunit.client.0.smithi121.stdout:2/609: link d2/d9/d38/d2d/d47/d94/c49 d2/d9/d1c/d85/cbf 0 2024-04-05T21:26:47.479 INFO:tasks.workunit.client.0.smithi121.stdout:0/675: dwrite d7/d1c/f1f [0,4194304] 0 2024-04-05T21:26:47.482 INFO:tasks.workunit.client.0.smithi121.stdout:5/769: chown l8 762 1 2024-04-05T21:26:47.483 INFO:tasks.workunit.client.0.smithi121.stdout:0/676: stat d7/d71/d29/d2a/d2c/f7b 0 2024-04-05T21:26:47.483 INFO:tasks.workunit.client.0.smithi121.stdout:0/677: readlink d7/d7e/d97/lb7 0 2024-04-05T21:26:47.484 INFO:tasks.workunit.client.0.smithi121.stdout:5/770: rmdir dd/de/d19/d38/d9b 39 2024-04-05T21:26:47.484 INFO:tasks.workunit.client.0.smithi121.stdout:9/500: truncate d6/f17 2330387 0 2024-04-05T21:26:47.486 INFO:tasks.workunit.client.0.smithi121.stdout:2/610: write d2/d9/d38/d2d/d47/f70 [2799992,75799] 0 2024-04-05T21:26:47.486 INFO:tasks.workunit.client.0.smithi121.stdout:5/771: chown dd/de/d6d/c8b 137 1 2024-04-05T21:26:47.486 INFO:tasks.workunit.client.0.smithi121.stdout:8/606: rmdir d12 39 2024-04-05T21:26:47.488 INFO:tasks.workunit.client.0.smithi121.stdout:0/678: creat d7/d62/d9d/fec x:0 0 0 2024-04-05T21:26:47.491 INFO:tasks.workunit.client.0.smithi121.stdout:5/772: dread - dd/de/d19/d13/d53/fbb zero size 2024-04-05T21:26:47.493 INFO:tasks.workunit.client.1.smithi188.stdout:0/720: sync 2024-04-05T21:26:47.493 INFO:tasks.workunit.client.1.smithi188.stdout:3/919: sync 2024-04-05T21:26:47.493 INFO:tasks.workunit.client.1.smithi188.stdout:8/708: sync 2024-04-05T21:26:47.493 INFO:tasks.workunit.client.1.smithi188.stdout:9/910: sync 2024-04-05T21:26:47.494 INFO:tasks.workunit.client.0.smithi121.stdout:0/679: creat d7/d62/d8b/fed x:0 0 0 2024-04-05T21:26:47.495 INFO:tasks.workunit.client.0.smithi121.stdout:9/501: mkdir d6/df/d1d/d2c/d37/dab 0 2024-04-05T21:26:47.496 INFO:tasks.workunit.client.0.smithi121.stdout:5/773: fsync dd/de/d19/d38/d9b/fb0 0 2024-04-05T21:26:47.497 INFO:tasks.workunit.client.0.smithi121.stdout:9/502: readlink d6/d32/l41 0 2024-04-05T21:26:47.497 INFO:tasks.workunit.client.0.smithi121.stdout:0/680: fdatasync d7/d71/f5f 0 2024-04-05T21:26:47.498 INFO:tasks.workunit.client.0.smithi121.stdout:2/611: getdents d2/d9 0 2024-04-05T21:26:47.501 INFO:tasks.workunit.client.0.smithi121.stdout:5/774: fsync dd/de/d6d/f9f 0 2024-04-05T21:26:47.501 INFO:tasks.workunit.client.0.smithi121.stdout:8/607: dwrite d12/d50/d93/d96/fab [0,4194304] 0 2024-04-05T21:26:47.503 INFO:tasks.workunit.client.0.smithi121.stdout:0/681: symlink d7/d62/d9d/lee 0 2024-04-05T21:26:47.507 INFO:tasks.workunit.client.0.smithi121.stdout:5/775: write dd/d41/dce/ff5 [4542642,111974] 0 2024-04-05T21:26:47.508 INFO:tasks.workunit.client.0.smithi121.stdout:9/503: rename d6/df/f2e to d6/d46/fac 0 2024-04-05T21:26:47.509 INFO:tasks.workunit.client.0.smithi121.stdout:0/682: write d7/d62/d9d/fc0 [494445,2044] 0 2024-04-05T21:26:47.509 INFO:tasks.workunit.client.0.smithi121.stdout:5/776: dread - dd/de/d6d/fec zero size 2024-04-05T21:26:47.512 INFO:tasks.workunit.client.0.smithi121.stdout:9/504: creat d6/d46/fad x:0 0 0 2024-04-05T21:26:47.513 INFO:tasks.workunit.client.0.smithi121.stdout:2/612: dwrite d2/d28/f4f [0,4194304] 0 2024-04-05T21:26:47.515 INFO:tasks.workunit.client.0.smithi121.stdout:5/777: write dd/de/d19/d1c/f48 [111153,25091] 0 2024-04-05T21:26:47.519 INFO:tasks.workunit.client.0.smithi121.stdout:2/613: dread - d2/d9/d17/fa7 zero size 2024-04-05T21:26:47.521 INFO:tasks.workunit.client.0.smithi121.stdout:8/608: dwrite d12/d13/d15/d1d/fa9 [4194304,4194304] 0 2024-04-05T21:26:47.524 INFO:tasks.workunit.client.1.smithi188.stdout:8/709: read - d5/d21/d40/d43/d44/d59/d8b/fb6 zero size 2024-04-05T21:26:47.526 INFO:tasks.workunit.client.0.smithi121.stdout:0/683: dwrite d7/d1c/fb6 [0,4194304] 0 2024-04-05T21:26:47.529 INFO:tasks.workunit.client.1.smithi188.stdout:0/721: unlink d4/d8/db/f11 0 2024-04-05T21:26:47.530 INFO:tasks.workunit.client.1.smithi188.stdout:8/710: symlink d5/d21/d40/d43/lea 0 2024-04-05T21:26:47.531 INFO:tasks.workunit.client.1.smithi188.stdout:5/883: write d6/d15/d5c/fa3 [489308,51935] 0 2024-04-05T21:26:47.531 INFO:tasks.workunit.client.1.smithi188.stdout:6/949: write d6/d13/d18/d19/d2c/d3f/d95/fe2 [782222,10583] 0 2024-04-05T21:26:47.532 INFO:tasks.workunit.client.0.smithi121.stdout:2/614: unlink d2/d9/l1d 0 2024-04-05T21:26:47.532 INFO:tasks.workunit.client.0.smithi121.stdout:8/609: mknod d12/d50/d6a/d63/cbe 0 2024-04-05T21:26:47.535 INFO:tasks.workunit.client.0.smithi121.stdout:9/505: dwrite d6/df/d1d/d2c/d37/d3c/f7c [0,4194304] 0 2024-04-05T21:26:47.537 INFO:tasks.workunit.client.0.smithi121.stdout:8/610: creat d12/d50/d61/fbf x:0 0 0 2024-04-05T21:26:47.538 INFO:tasks.workunit.client.1.smithi188.stdout:5/884: mkdir d6/d82/d40/d12e 0 2024-04-05T21:26:47.539 INFO:tasks.workunit.client.0.smithi121.stdout:2/615: getdents d2/d1e 0 2024-04-05T21:26:47.540 INFO:tasks.workunit.client.0.smithi121.stdout:9/506: symlink d6/d36/d7b/d5d/lae 0 2024-04-05T21:26:47.541 INFO:tasks.workunit.client.1.smithi188.stdout:6/950: chown d6/d13/d18/d19/l88 13951 1 2024-04-05T21:26:47.542 INFO:tasks.workunit.client.1.smithi188.stdout:5/885: creat d6/d106/d8e/def/f12f x:0 0 0 2024-04-05T21:26:47.543 INFO:tasks.workunit.client.1.smithi188.stdout:6/951: chown d6/d13/d18/d19/d2c/d34 1728 1 2024-04-05T21:26:47.543 INFO:tasks.workunit.client.0.smithi121.stdout:8/611: getdents d12/d13/d15 0 2024-04-05T21:26:47.544 INFO:tasks.workunit.client.0.smithi121.stdout:8/612: chown d12/d13/cb6 128592 1 2024-04-05T21:26:47.545 INFO:tasks.workunit.client.0.smithi121.stdout:8/613: write d12/d13/d15/d1d/f36 [2596185,40989] 0 2024-04-05T21:26:47.546 INFO:tasks.workunit.client.1.smithi188.stdout:5/886: write d6/d15/d5c/d7a/d44/f10a [978823,27394] 0 2024-04-05T21:26:47.548 INFO:tasks.workunit.client.1.smithi188.stdout:0/722: dwrite d4/d8/db/d42/da2/dd2/fd3 [0,4194304] 0 2024-04-05T21:26:47.550 INFO:tasks.workunit.client.0.smithi121.stdout:8/614: rename d12/d13/d15/fbd to d12/d13/d56/d57/d5e/db9/fc0 0 2024-04-05T21:26:47.553 INFO:tasks.workunit.client.1.smithi188.stdout:5/887: mknod d6/d15/d1b/c130 0 2024-04-05T21:26:47.554 INFO:tasks.workunit.client.1.smithi188.stdout:6/952: creat d6/d13/d18/d19/da3/d117/f127 x:0 0 0 2024-04-05T21:26:47.557 INFO:tasks.workunit.client.1.smithi188.stdout:5/888: mkdir d6/d131 0 2024-04-05T21:26:47.558 INFO:tasks.workunit.client.1.smithi188.stdout:0/723: truncate d4/d8/db/d21/d3c/fa5 1626484 0 2024-04-05T21:26:47.560 INFO:tasks.workunit.client.1.smithi188.stdout:6/953: rename d6/d13/f1d to d6/d13/d18/d19/d2c/d3f/d95/f128 0 2024-04-05T21:26:47.566 INFO:tasks.workunit.client.1.smithi188.stdout:5/889: symlink d6/d15/d5c/d7a/d44/d51/df1/d104/d124/l132 0 2024-04-05T21:26:47.567 INFO:tasks.workunit.client.1.smithi188.stdout:6/954: dread d6/d13/d48/fa2 [0,4194304] 0 2024-04-05T21:26:47.571 INFO:tasks.workunit.client.1.smithi188.stdout:5/890: dread d6/d15/d5c/d7a/f2b [0,4194304] 0 2024-04-05T21:26:47.572 INFO:tasks.workunit.client.1.smithi188.stdout:6/955: link d6/d13/d38/fdb d6/d13/d18/d19/da3/f129 0 2024-04-05T21:26:47.573 INFO:tasks.workunit.client.1.smithi188.stdout:3/920: dread d5/d7/d65/f44 [0,4194304] 0 2024-04-05T21:26:47.573 INFO:tasks.workunit.client.1.smithi188.stdout:6/956: fsync d6/d13/f126 0 2024-04-05T21:26:47.576 INFO:tasks.workunit.client.1.smithi188.stdout:3/921: truncate f4 5147798 0 2024-04-05T21:26:47.576 INFO:tasks.workunit.client.1.smithi188.stdout:3/922: fdatasync d5/d48/f63 0 2024-04-05T21:26:47.578 INFO:tasks.workunit.client.1.smithi188.stdout:5/891: symlink d6/d17/d23/d7d/dcf/l133 0 2024-04-05T21:26:47.579 INFO:tasks.workunit.client.1.smithi188.stdout:6/957: fdatasync d6/d13/d18/d19/dc8/da4/fb8 0 2024-04-05T21:26:47.580 INFO:tasks.workunit.client.0.smithi121.stdout:0/684: dread d7/d71/d29/de1/feb [0,4194304] 0 2024-04-05T21:26:47.580 INFO:tasks.workunit.client.1.smithi188.stdout:9/911: truncate dd/d11/d1f/d26/dc0/d6b/f106 131220 0 2024-04-05T21:26:47.582 INFO:tasks.workunit.client.0.smithi121.stdout:0/685: rename d7 to d7/d62/d3c/d6b/def 22 2024-04-05T21:26:47.585 INFO:tasks.workunit.client.1.smithi188.stdout:6/958: fsync d6/d13/d18/d94/f11e 0 2024-04-05T21:26:47.585 INFO:tasks.workunit.client.0.smithi121.stdout:9/507: read d6/f17 [337418,54162] 0 2024-04-05T21:26:47.587 INFO:tasks.workunit.client.1.smithi188.stdout:9/912: symlink dd/d3d/d89/dd1/l132 0 2024-04-05T21:26:47.588 INFO:tasks.workunit.client.1.smithi188.stdout:5/892: creat d6/d82/d40/d42/d81/d10c/f134 x:0 0 0 2024-04-05T21:26:47.588 INFO:tasks.workunit.client.0.smithi121.stdout:0/686: dwrite d7/d62/d9d/fbb [0,4194304] 0 2024-04-05T21:26:47.589 INFO:tasks.workunit.client.1.smithi188.stdout:3/923: dwrite d5/d7/d65/d61/db5/ff0 [0,4194304] 0 2024-04-05T21:26:47.594 INFO:tasks.workunit.client.1.smithi188.stdout:6/959: mkdir d6/d13/d18/d19/d2c/d3f/d95/d12a 0 2024-04-05T21:26:47.595 INFO:tasks.workunit.client.0.smithi121.stdout:0/687: creat d7/d71/dd0/ff0 x:0 0 0 2024-04-05T21:26:47.597 INFO:tasks.workunit.client.0.smithi121.stdout:5/778: dwrite dd/de/d19/f28 [0,4194304] 0 2024-04-05T21:26:47.603 INFO:tasks.workunit.client.1.smithi188.stdout:6/960: rename d6/d13/d18/d3c/ddc to d6/d13/d18/d3c/daf/d12b 0 2024-04-05T21:26:47.605 INFO:tasks.workunit.client.1.smithi188.stdout:5/893: dread d6/d17/f46 [0,4194304] 0 2024-04-05T21:26:47.605 INFO:tasks.workunit.client.0.smithi121.stdout:8/615: write d12/d13/d15/f83 [1867431,43709] 0 2024-04-05T21:26:47.608 INFO:tasks.workunit.client.1.smithi188.stdout:0/724: dread d4/d8/d10/f40 [0,4194304] 0 2024-04-05T21:26:47.609 INFO:tasks.workunit.client.0.smithi121.stdout:9/508: truncate d6/d36/d7b/d5d/d79/fa7 1643011 0 2024-04-05T21:26:47.609 INFO:tasks.workunit.client.1.smithi188.stdout:9/913: rename dd/d11/d1f/db2 to dd/d11/d1f/d26/d55/d98/d133 0 2024-04-05T21:26:47.609 INFO:tasks.workunit.client.0.smithi121.stdout:0/688: write d7/dc/f2d [3226872,60327] 0 2024-04-05T21:26:47.610 INFO:tasks.workunit.client.0.smithi121.stdout:2/616: dwrite d2/d9/d1c/d30/dba/fb5 [0,4194304] 0 2024-04-05T21:26:47.612 INFO:tasks.workunit.client.0.smithi121.stdout:8/616: chown d12/d13/d15/d1d/l69 162 1 2024-04-05T21:26:47.612 INFO:tasks.workunit.client.0.smithi121.stdout:9/509: rmdir d6/d36/d7b/d5d/d85 39 2024-04-05T21:26:47.613 INFO:tasks.workunit.client.0.smithi121.stdout:5/779: link dd/de/d19/d38/de3/lf3 dd/de/d39/d3b/de8/d103/l116 0 2024-04-05T21:26:47.614 INFO:tasks.workunit.client.0.smithi121.stdout:5/780: stat dd/de/d6d 0 2024-04-05T21:26:47.615 INFO:tasks.workunit.client.1.smithi188.stdout:6/961: creat d6/d13/d48/f12c x:0 0 0 2024-04-05T21:26:47.615 INFO:tasks.workunit.client.0.smithi121.stdout:0/689: chown d7/d71/d45/d46/d53/d8e/l9e 21705614 1 2024-04-05T21:26:47.617 INFO:tasks.workunit.client.1.smithi188.stdout:9/914: mknod dd/d3d/d89/dd1/c134 0 2024-04-05T21:26:47.617 INFO:tasks.workunit.client.0.smithi121.stdout:0/690: chown d7/d71/d29/d3f/fba 966 1 2024-04-05T21:26:47.617 INFO:tasks.workunit.client.0.smithi121.stdout:9/510: creat d6/d36/d7b/d5d/d79/faf x:0 0 0 2024-04-05T21:26:47.619 INFO:tasks.workunit.client.1.smithi188.stdout:0/725: dwrite d4/d8/db/d21/d4a/d73/f94 [0,4194304] 0 2024-04-05T21:26:47.619 INFO:tasks.workunit.client.1.smithi188.stdout:6/962: readlink d6/d13/d18/d19/d2c/db2/lc3 0 2024-04-05T21:26:47.620 INFO:tasks.workunit.client.0.smithi121.stdout:8/617: getdents d12/d50/d6a/d63/d8b 0 2024-04-05T21:26:47.620 INFO:tasks.workunit.client.0.smithi121.stdout:8/618: stat d12/f73 0 2024-04-05T21:26:47.622 INFO:tasks.workunit.client.0.smithi121.stdout:9/511: rename d6/ld to d6/d32/lb0 0 2024-04-05T21:26:47.624 INFO:tasks.workunit.client.0.smithi121.stdout:0/691: creat d7/d71/d29/ff1 x:0 0 0 2024-04-05T21:26:47.626 INFO:tasks.workunit.client.0.smithi121.stdout:9/512: chown d6/df/l58 809646 1 2024-04-05T21:26:47.627 INFO:tasks.workunit.client.1.smithi188.stdout:5/894: creat d6/d82/d40/d42/d81/da6/f135 x:0 0 0 2024-04-05T21:26:47.628 INFO:tasks.workunit.client.0.smithi121.stdout:8/619: dwrite d12/d13/d56/d57/f7b [0,4194304] 0 2024-04-05T21:26:47.630 INFO:tasks.workunit.client.1.smithi188.stdout:9/915: dwrite dd/d11/d1f/fdd [0,4194304] 0 2024-04-05T21:26:47.632 INFO:tasks.workunit.client.1.smithi188.stdout:0/726: stat d4/d6/d13/d23/d24/d57/d63/fbd 0 2024-04-05T21:26:47.632 INFO:tasks.workunit.client.0.smithi121.stdout:0/692: link d7/d71/f26 d7/d62/ff2 0 2024-04-05T21:26:47.636 INFO:tasks.workunit.client.0.smithi121.stdout:9/513: link d6/df/d1d/d2c/d37/d3c/f9d d6/df/d50/fb1 0 2024-04-05T21:26:47.637 INFO:tasks.workunit.client.1.smithi188.stdout:9/916: creat dd/d11/d1f/d26/d55/d98/da5/f135 x:0 0 0 2024-04-05T21:26:47.637 INFO:tasks.workunit.client.1.smithi188.stdout:0/727: symlink d4/d8/db/d42/da2/dd2/l108 0 2024-04-05T21:26:47.637 INFO:tasks.workunit.client.1.smithi188.stdout:6/963: creat d6/d13/d18/d19/d2c/d3f/d95/f12d x:0 0 0 2024-04-05T21:26:47.638 INFO:tasks.workunit.client.0.smithi121.stdout:0/693: symlink d7/d71/d45/lf3 0 2024-04-05T21:26:47.640 INFO:tasks.workunit.client.1.smithi188.stdout:5/895: write d6/d17/d23/d7d/dcf/fd6 [1848532,64173] 0 2024-04-05T21:26:47.640 INFO:tasks.workunit.client.0.smithi121.stdout:9/514: mkdir d6/df/d48/d96/db2 0 2024-04-05T21:26:47.642 INFO:tasks.workunit.client.0.smithi121.stdout:5/781: write dd/de/d39/d3b/f4a [3406392,55077] 0 2024-04-05T21:26:47.646 INFO:tasks.workunit.client.0.smithi121.stdout:2/617: dwrite d2/d9/f75 [0,4194304] 0 2024-04-05T21:26:47.646 INFO:tasks.workunit.client.1.smithi188.stdout:6/964: write d6/d13/d18/d3c/daf/d12b/d60/f109 [3982383,114057] 0 2024-04-05T21:26:47.646 INFO:tasks.workunit.client.1.smithi188.stdout:3/924: dwrite f4 [0,4194304] 0 2024-04-05T21:26:47.647 INFO:tasks.workunit.client.0.smithi121.stdout:0/694: creat d7/d71/d45/d46/d53/d8e/dc4/ff4 x:0 0 0 2024-04-05T21:26:47.648 INFO:tasks.workunit.client.0.smithi121.stdout:2/618: write d2/d28/f4f [3944231,50672] 0 2024-04-05T21:26:47.648 INFO:tasks.workunit.client.0.smithi121.stdout:9/515: creat d6/d36/d7b/fb3 x:0 0 0 2024-04-05T21:26:47.651 INFO:tasks.workunit.client.1.smithi188.stdout:3/925: truncate d5/d7e/d101/fb3 1010022 0 2024-04-05T21:26:47.651 INFO:tasks.workunit.client.1.smithi188.stdout:5/896: creat d6/d106/de1/f136 x:0 0 0 2024-04-05T21:26:47.651 INFO:tasks.workunit.client.0.smithi121.stdout:0/695: chown d7/d71/d45/d5a/f69 612725070 1 2024-04-05T21:26:47.652 INFO:tasks.workunit.client.1.smithi188.stdout:3/926: stat d5/d7e/d101 0 2024-04-05T21:26:47.652 INFO:tasks.workunit.client.1.smithi188.stdout:6/965: rmdir d6/d13/d18/d3c/daf/d12b/d60 39 2024-04-05T21:26:47.652 INFO:tasks.workunit.client.0.smithi121.stdout:5/782: link dd/de/d19/d38/f4d dd/de/d105/f117 0 2024-04-05T21:26:47.653 INFO:tasks.workunit.client.0.smithi121.stdout:0/696: readlink d7/d71/d45/d46/d53/d8e/lcb 0 2024-04-05T21:26:47.653 INFO:tasks.workunit.client.0.smithi121.stdout:9/516: rmdir d6/df/d1d/d72 39 2024-04-05T21:26:47.655 INFO:tasks.workunit.client.0.smithi121.stdout:9/517: chown d6/d36/d7b/d49/d66/f91 2435 1 2024-04-05T21:26:47.655 INFO:tasks.workunit.client.0.smithi121.stdout:0/697: write d7/d71/d29/d2a/d2c/f59 [4688979,81961] 0 2024-04-05T21:26:47.662 INFO:tasks.workunit.client.1.smithi188.stdout:6/966: readlink d6/d13/d18/d19/d2c/d34/ld4 0 2024-04-05T21:26:47.663 INFO:tasks.workunit.client.1.smithi188.stdout:5/897: mkdir d6/d15/d5c/d7a/d44/d51/df1/d137 0 2024-04-05T21:26:47.664 INFO:tasks.workunit.client.1.smithi188.stdout:3/927: mknod d5/d7/d65/d61/d66/da4/de3/df6/c12f 0 2024-04-05T21:26:47.666 INFO:tasks.workunit.client.0.smithi121.stdout:8/620: dread d12/d50/d93/d96/f92 [0,4194304] 0 2024-04-05T21:26:47.667 INFO:tasks.workunit.client.0.smithi121.stdout:0/698: dwrite d7/d62/d9d/da6/fb9 [0,4194304] 0 2024-04-05T21:26:47.668 INFO:tasks.workunit.client.0.smithi121.stdout:9/518: dwrite d6/df/d1d/f2d [0,4194304] 0 2024-04-05T21:26:47.669 INFO:tasks.workunit.client.1.smithi188.stdout:3/928: creat d5/ddd/f130 x:0 0 0 2024-04-05T21:26:47.669 INFO:tasks.workunit.client.1.smithi188.stdout:6/967: write d6/d13/d18/d3c/daf/d12b/f80 [1151520,43095] 0 2024-04-05T21:26:47.673 INFO:tasks.workunit.client.0.smithi121.stdout:8/621: unlink d12/d13/d56/d57/da4/l8c 0 2024-04-05T21:26:47.674 INFO:tasks.workunit.client.1.smithi188.stdout:0/728: dwrite d4/d6/d13/d23/d24/fbe [0,4194304] 0 2024-04-05T21:26:47.674 INFO:tasks.workunit.client.1.smithi188.stdout:9/917: dwrite dd/d3d/d89/fed [0,4194304] 0 2024-04-05T21:26:47.674 INFO:tasks.workunit.client.0.smithi121.stdout:0/699: mkdir d7/dc/da3/df5 0 2024-04-05T21:26:47.676 INFO:tasks.workunit.client.0.smithi121.stdout:8/622: chown d12/ca0 21 1 2024-04-05T21:26:47.677 INFO:tasks.workunit.client.0.smithi121.stdout:0/700: symlink d7/d62/d9d/da6/lf6 0 2024-04-05T21:26:47.677 INFO:tasks.workunit.client.0.smithi121.stdout:8/623: chown d12/d50/d93/d96/f8a 13 1 2024-04-05T21:26:47.678 INFO:tasks.workunit.client.0.smithi121.stdout:9/519: symlink d6/df/d1d/d2c/d53/lb4 0 2024-04-05T21:26:47.679 INFO:tasks.workunit.client.0.smithi121.stdout:4/751: sync 2024-04-05T21:26:47.679 INFO:tasks.workunit.client.1.smithi188.stdout:5/898: write d6/d15/d5c/d7a/d44/d98/f9f [1899355,37163] 0 2024-04-05T21:26:47.680 INFO:tasks.workunit.client.0.smithi121.stdout:9/520: readlink d6/df/d1d/d2c/l7a 0 2024-04-05T21:26:47.680 INFO:tasks.workunit.client.0.smithi121.stdout:0/701: write d7/d62/d9d/da6/dce/fcf [984635,86051] 0 2024-04-05T21:26:47.680 INFO:tasks.workunit.client.0.smithi121.stdout:4/752: chown d5/d11/c2b 115434 1 2024-04-05T21:26:47.681 INFO:tasks.workunit.client.0.smithi121.stdout:0/702: chown d7/d62/d3c/c67 22 1 2024-04-05T21:26:47.682 INFO:tasks.workunit.client.0.smithi121.stdout:8/624: unlink d12/d13/d56/d57/da4/f40 0 2024-04-05T21:26:47.685 INFO:tasks.workunit.client.1.smithi188.stdout:9/918: creat dd/d11/d1f/d26/d55/d98/f136 x:0 0 0 2024-04-05T21:26:47.685 INFO:tasks.workunit.client.1.smithi188.stdout:9/919: dread - dd/d3d/d108/ffd zero size 2024-04-05T21:26:47.686 INFO:tasks.workunit.client.0.smithi121.stdout:8/625: truncate d12/d13/d15/d1d/f27 650429 0 2024-04-05T21:26:47.686 INFO:tasks.workunit.client.0.smithi121.stdout:0/703: link d7/l1e d7/dc/da3/df5/lf7 0 2024-04-05T21:26:47.686 INFO:tasks.workunit.client.0.smithi121.stdout:0/704: stat d7/dc/le 0 2024-04-05T21:26:47.690 INFO:tasks.workunit.client.1.smithi188.stdout:0/729: creat d4/d8/d39/f109 x:0 0 0 2024-04-05T21:26:47.694 INFO:tasks.workunit.client.1.smithi188.stdout:5/899: truncate d6/d106/de1/de3/fe7 915778 0 2024-04-05T21:26:47.702 INFO:tasks.workunit.client.1.smithi188.stdout:6/968: dwrite d6/d13/d18/d19/d2c/d3f/d77/fc0 [4194304,4194304] 0 2024-04-05T21:26:47.702 INFO:tasks.workunit.client.1.smithi188.stdout:3/929: link d5/d7/d31/l77 d5/d35/d128/l131 0 2024-04-05T21:26:47.703 INFO:tasks.workunit.client.1.smithi188.stdout:5/900: creat d6/d17/d23/d7d/f138 x:0 0 0 2024-04-05T21:26:47.706 INFO:tasks.workunit.client.1.smithi188.stdout:5/901: read - d6/d15/d5c/d7a/d44/fe2 zero size 2024-04-05T21:26:47.723 INFO:tasks.workunit.client.1.smithi188.stdout:3/930: rmdir d5/d35 39 2024-04-05T21:26:47.723 INFO:tasks.workunit.client.1.smithi188.stdout:6/969: dread - d6/d13/d18/d19/d2c/d34/fc4 zero size 2024-04-05T21:26:47.723 INFO:tasks.workunit.client.0.smithi121.stdout:8/626: symlink d12/d50/d61/d7f/d9a/lc1 0 2024-04-05T21:26:47.724 INFO:tasks.workunit.client.0.smithi121.stdout:0/705: write d7/d71/d45/d5a/f80 [1292930,92700] 0 2024-04-05T21:26:47.726 INFO:tasks.workunit.client.1.smithi188.stdout:5/902: symlink d6/d15/d5c/d7a/d44/l139 0 2024-04-05T21:26:47.726 INFO:tasks.workunit.client.1.smithi188.stdout:6/970: truncate d6/d28/f58 70076 0 2024-04-05T21:26:47.727 INFO:tasks.workunit.client.0.smithi121.stdout:0/706: creat d7/d62/ddb/ff8 x:0 0 0 2024-04-05T21:26:47.730 INFO:tasks.workunit.client.1.smithi188.stdout:0/730: write d4/d8/db/d21/d4a/f58 [309348,231] 0 2024-04-05T21:26:47.731 INFO:tasks.workunit.client.0.smithi121.stdout:8/627: link d12/f3d d12/d50/d6a/d87/fc2 0 2024-04-05T21:26:47.731 INFO:tasks.workunit.client.1.smithi188.stdout:5/903: dread d6/d106/f4a [4194304,4194304] 0 2024-04-05T21:26:47.732 INFO:tasks.workunit.client.1.smithi188.stdout:3/931: dwrite d5/d7/d65/f8c [4194304,4194304] 0 2024-04-05T21:26:47.733 INFO:tasks.workunit.client.0.smithi121.stdout:2/619: dread d2/d9/d38/f35 [0,4194304] 0 2024-04-05T21:26:47.734 INFO:tasks.workunit.client.1.smithi188.stdout:9/920: dread dd/d11/d1f/d26/dc0/d6b/f94 [0,4194304] 0 2024-04-05T21:26:47.735 INFO:tasks.workunit.client.0.smithi121.stdout:8/628: unlink d12/d13/d56/d57/d5e/fa6 0 2024-04-05T21:26:47.735 INFO:tasks.workunit.client.1.smithi188.stdout:5/904: mkdir d6/d106/d8e/def/d13a 0 2024-04-05T21:26:47.736 INFO:tasks.workunit.client.0.smithi121.stdout:2/620: readlink d2/d9/d1c/d85/l9b 0 2024-04-05T21:26:47.737 INFO:tasks.workunit.client.1.smithi188.stdout:3/932: write d5/d7/d65/d61/d66/da4/de3/df6/f11c [829050,120269] 0 2024-04-05T21:26:47.741 INFO:tasks.workunit.client.1.smithi188.stdout:9/921: creat dd/d11/d1f/d25/d51/f137 x:0 0 0 2024-04-05T21:26:47.742 INFO:tasks.workunit.client.1.smithi188.stdout:5/905: write d6/d17/d23/d7d/fd7 [5887772,74198] 0 2024-04-05T21:26:47.743 INFO:tasks.workunit.client.0.smithi121.stdout:4/753: write d5/d11/d26/f6b [456414,92913] 0 2024-04-05T21:26:47.744 INFO:tasks.workunit.client.1.smithi188.stdout:9/922: creat dd/d3d/d108/f138 x:0 0 0 2024-04-05T21:26:47.745 INFO:tasks.workunit.client.0.smithi121.stdout:2/621: mknod d2/d9/d56/d62/cc0 0 2024-04-05T21:26:47.746 INFO:tasks.workunit.client.1.smithi188.stdout:3/933: getdents d5/d39/d84 0 2024-04-05T21:26:47.747 INFO:tasks.workunit.client.1.smithi188.stdout:9/923: chown dd/d11/d1f/fdd 13 1 2024-04-05T21:26:47.747 INFO:tasks.workunit.client.0.smithi121.stdout:8/629: dwrite d12/d13/d56/fa2 [0,4194304] 0 2024-04-05T21:26:47.749 INFO:tasks.workunit.client.0.smithi121.stdout:2/622: fdatasync d2/d9/d38/d2d/d47/f70 0 2024-04-05T21:26:47.750 INFO:tasks.workunit.client.1.smithi188.stdout:9/924: chown dd/d3d/d81/f107 240571 1 2024-04-05T21:26:47.750 INFO:tasks.workunit.client.1.smithi188.stdout:5/906: rename d6/d82/d40/d42/d81/da6/f108 to d6/d17/f13b 0 2024-04-05T21:26:47.751 INFO:tasks.workunit.client.1.smithi188.stdout:9/925: chown dd/d11/d1f/d25/f50 1130 1 2024-04-05T21:26:47.751 INFO:tasks.workunit.client.0.smithi121.stdout:2/623: dread - d2/d9/d1c/d30/d59/fbe zero size 2024-04-05T21:26:47.752 INFO:tasks.workunit.client.1.smithi188.stdout:9/926: stat dd/d11/d73/f86 0 2024-04-05T21:26:47.752 INFO:tasks.workunit.client.0.smithi121.stdout:2/624: fsync d2/d9/d38/d2d/d4c/d79/fa2 0 2024-04-05T21:26:47.753 INFO:tasks.workunit.client.0.smithi121.stdout:4/754: unlink d5/d11/d2c/d9d/dae/d29/d2e/f7b 0 2024-04-05T21:26:47.755 INFO:tasks.workunit.client.0.smithi121.stdout:8/630: creat d12/d50/d93/d96/fc3 x:0 0 0 2024-04-05T21:26:47.755 INFO:tasks.workunit.client.0.smithi121.stdout:4/755: chown d5/d11/d26/d49/ceb 855444060 1 2024-04-05T21:26:47.755 INFO:tasks.workunit.client.1.smithi188.stdout:5/907: fdatasync d6/d15/d1b/f1e 0 2024-04-05T21:26:47.756 INFO:tasks.workunit.client.0.smithi121.stdout:0/707: truncate d7/dc/faf 70372 0 2024-04-05T21:26:47.757 INFO:tasks.workunit.client.1.smithi188.stdout:5/908: chown d6/d17/d23/d7d/d128/d122 300 1 2024-04-05T21:26:47.762 INFO:tasks.workunit.client.1.smithi188.stdout:5/909: mknod d6/d82/d40/dc1/c13c 0 2024-04-05T21:26:47.763 INFO:tasks.workunit.client.1.smithi188.stdout:9/927: getdents dd/d3d 0 2024-04-05T21:26:47.763 INFO:tasks.workunit.client.1.smithi188.stdout:5/910: symlink d6/d82/d40/d42/d81/d10c/d10d/l13d 0 2024-04-05T21:26:47.763 INFO:tasks.workunit.client.0.smithi121.stdout:0/708: unlink d7/d71/d45/d46/d53/d8e/l9e 0 2024-04-05T21:26:47.763 INFO:tasks.workunit.client.0.smithi121.stdout:0/709: write d7/d71/dd0/ff0 [641934,4996] 0 2024-04-05T21:26:47.764 INFO:tasks.workunit.client.0.smithi121.stdout:8/631: mkdir d12/d50/d6a/dc4 0 2024-04-05T21:26:47.765 INFO:tasks.workunit.client.1.smithi188.stdout:9/928: symlink dd/d11/d73/d8d/l139 0 2024-04-05T21:26:47.768 INFO:tasks.workunit.client.0.smithi121.stdout:4/756: link d5/d11/d26/d47/cba d5/d11/d28/d87/cf2 0 2024-04-05T21:26:47.768 INFO:tasks.workunit.client.1.smithi188.stdout:9/929: read dd/d11/d1f/f10a [588705,70218] 0 2024-04-05T21:26:47.770 INFO:tasks.workunit.client.1.smithi188.stdout:5/911: dwrite d6/d106/f4a [0,4194304] 0 2024-04-05T21:26:47.771 INFO:tasks.workunit.client.1.smithi188.stdout:9/930: write dd/d11/d1f/d26/dc0/f5c [1080196,42729] 0 2024-04-05T21:26:47.771 INFO:tasks.workunit.client.0.smithi121.stdout:8/632: mknod d12/d50/dbb/cc5 0 2024-04-05T21:26:47.774 INFO:tasks.workunit.client.0.smithi121.stdout:2/625: truncate d2/d9/d56/d62/f7a 2870604 0 2024-04-05T21:26:47.775 INFO:tasks.workunit.client.0.smithi121.stdout:8/633: symlink d12/d50/d6a/d63/d8b/lc6 0 2024-04-05T21:26:47.775 INFO:tasks.workunit.client.1.smithi188.stdout:5/912: creat d6/d15/d5c/d7a/f13e x:0 0 0 2024-04-05T21:26:47.776 INFO:tasks.workunit.client.1.smithi188.stdout:9/931: rmdir dd/d11/d1f/d26/dc0 39 2024-04-05T21:26:47.778 INFO:tasks.workunit.client.0.smithi121.stdout:8/634: mkdir d12/d50/d93/db5/dc7 0 2024-04-05T21:26:47.779 INFO:tasks.workunit.client.1.smithi188.stdout:9/932: fsync dd/d11/d1f/d26/dc0/d6b/d40/d59/fe9 0 2024-04-05T21:26:47.779 INFO:tasks.workunit.client.1.smithi188.stdout:0/731: dread d4/d8/db/d42/d38/f49 [4194304,4194304] 0 2024-04-05T21:26:47.780 INFO:tasks.workunit.client.0.smithi121.stdout:8/635: fsync d12/d13/d15/d1d/fa9 0 2024-04-05T21:26:47.781 INFO:tasks.workunit.client.0.smithi121.stdout:2/626: dwrite d2/f11 [4194304,4194304] 0 2024-04-05T21:26:47.784 INFO:tasks.workunit.client.1.smithi188.stdout:5/913: unlink d6/d82/d40/d42/d81/da6/cdf 0 2024-04-05T21:26:47.785 INFO:tasks.workunit.client.0.smithi121.stdout:0/710: truncate d7/d62/d9d/fc0 353686 0 2024-04-05T21:26:47.786 INFO:tasks.workunit.client.1.smithi188.stdout:9/933: mknod dd/d11/d73/d8d/c13a 0 2024-04-05T21:26:47.787 INFO:tasks.workunit.client.0.smithi121.stdout:2/627: mknod d2/cc1 0 2024-04-05T21:26:47.788 INFO:tasks.workunit.client.0.smithi121.stdout:8/636: dwrite d12/d13/d15/d1d/f36 [0,4194304] 0 2024-04-05T21:26:47.789 INFO:tasks.workunit.client.1.smithi188.stdout:5/914: rename d6/d106/de1/de3 to d6/d82/d40/d42/d81/d13f 0 2024-04-05T21:26:47.790 INFO:tasks.workunit.client.0.smithi121.stdout:2/628: write d2/d9/d38/d2d/d4c/d79/fa2 [3533692,95841] 0 2024-04-05T21:26:47.790 INFO:tasks.workunit.client.0.smithi121.stdout:0/711: readlink d7/d71/d29/d3f/le4 0 2024-04-05T21:26:47.792 INFO:tasks.workunit.client.0.smithi121.stdout:8/637: mknod d12/d50/d61/d7f/d9a/cc8 0 2024-04-05T21:26:47.793 INFO:tasks.workunit.client.1.smithi188.stdout:0/732: symlink d4/d8/db/d21/d3c/l10a 0 2024-04-05T21:26:47.795 INFO:tasks.workunit.client.0.smithi121.stdout:4/757: write d5/f81 [3344636,68231] 0 2024-04-05T21:26:47.796 INFO:tasks.workunit.client.1.smithi188.stdout:9/934: write dd/d11/d1f/d25/d51/fc7 [6466949,87884] 0 2024-04-05T21:26:47.796 INFO:tasks.workunit.client.0.smithi121.stdout:2/629: mkdir d2/d9/d38/d2d/d4c/d7c/dc2 0 2024-04-05T21:26:47.798 INFO:tasks.workunit.client.0.smithi121.stdout:8/638: mkdir d12/d13/d56/d57/da4/dc9 0 2024-04-05T21:26:47.799 INFO:tasks.workunit.client.1.smithi188.stdout:0/733: mknod d4/d6/d13/d23/de8/c10b 0 2024-04-05T21:26:47.801 INFO:tasks.workunit.client.0.smithi121.stdout:2/630: mknod d2/d9/d56/cc3 0 2024-04-05T21:26:47.801 INFO:tasks.workunit.client.1.smithi188.stdout:9/935: chown c0 1 1 2024-04-05T21:26:47.804 INFO:tasks.workunit.client.0.smithi121.stdout:2/631: symlink d2/d9/d56/d62/lc4 0 2024-04-05T21:26:47.804 INFO:tasks.workunit.client.1.smithi188.stdout:5/915: dwrite d6/d17/d23/d7d/fd7 [0,4194304] 0 2024-04-05T21:26:47.806 INFO:tasks.workunit.client.0.smithi121.stdout:8/639: dwrite d12/d50/d61/d7f/d9a/fb0 [0,4194304] 0 2024-04-05T21:26:47.810 INFO:tasks.workunit.client.1.smithi188.stdout:0/734: rename d4/d8/db/d42/d38/d77/f95 to d4/d6/d13/d23/dd1/f10c 0 2024-04-05T21:26:47.813 INFO:tasks.workunit.client.0.smithi121.stdout:2/632: dwrite d2/d9/d38/d2d/d4c/d7c/f8b [0,4194304] 0 2024-04-05T21:26:47.815 INFO:tasks.workunit.client.0.smithi121.stdout:2/633: write d2/d9/d38/d2d/d4c/d79/fb8 [1020405,110859] 0 2024-04-05T21:26:47.821 INFO:tasks.workunit.client.0.smithi121.stdout:3/530: sync 2024-04-05T21:26:47.821 INFO:tasks.workunit.client.0.smithi121.stdout:1/737: sync 2024-04-05T21:26:47.821 INFO:tasks.workunit.client.0.smithi121.stdout:6/499: sync 2024-04-05T21:26:47.821 INFO:tasks.workunit.client.0.smithi121.stdout:7/723: sync 2024-04-05T21:26:47.821 INFO:tasks.workunit.client.1.smithi188.stdout:3/934: dwrite d5/d7/d65/d5f/fbd [0,4194304] 0 2024-04-05T21:26:47.821 INFO:tasks.workunit.client.1.smithi188.stdout:0/735: creat d4/d6/d13/d23/dd1/f10d x:0 0 0 2024-04-05T21:26:47.824 INFO:tasks.workunit.client.0.smithi121.stdout:3/531: fdatasync d0/f27 0 2024-04-05T21:26:47.826 INFO:tasks.workunit.client.0.smithi121.stdout:4/758: truncate d5/d11/d26/d49/fb9 191920 0 2024-04-05T21:26:47.826 INFO:tasks.workunit.client.0.smithi121.stdout:0/712: dwrite d7/d62/d9d/da6/fe7 [0,4194304] 0 2024-04-05T21:26:47.827 INFO:tasks.workunit.client.1.smithi188.stdout:0/736: readlink d4/d6/d46/dba/lbb 0 2024-04-05T21:26:47.828 INFO:tasks.workunit.client.1.smithi188.stdout:0/737: fsync d4/d8/db/d21/fe2 0 2024-04-05T21:26:47.832 INFO:tasks.workunit.client.0.smithi121.stdout:3/532: dread - d0/d3/d24/d3d/d1a/d1e/d86/f8b zero size 2024-04-05T21:26:47.832 INFO:tasks.workunit.client.0.smithi121.stdout:3/533: chown d0/d3/d24/d3f 264196 1 2024-04-05T21:26:47.832 INFO:tasks.workunit.client.0.smithi121.stdout:1/738: unlink d8/d8c/db6/d1a/d1b/d4c/ce7 0 2024-04-05T21:26:47.833 INFO:tasks.workunit.client.0.smithi121.stdout:0/713: write d7/d62/d3c/fe3 [808585,20308] 0 2024-04-05T21:26:47.833 INFO:tasks.workunit.client.1.smithi188.stdout:3/935: getdents d5 0 2024-04-05T21:26:47.833 INFO:tasks.workunit.client.1.smithi188.stdout:0/738: dread - d4/d6/d13/d23/d24/d57/fec zero size 2024-04-05T21:26:47.834 INFO:tasks.workunit.client.0.smithi121.stdout:6/500: creat d1/d2/d1b/d2e/d53/faa x:0 0 0 2024-04-05T21:26:47.837 INFO:tasks.workunit.client.0.smithi121.stdout:7/724: rename d0/d9/d22/d3e/d9c/cd3 to d0/d23/d5b/d52/ce3 0 2024-04-05T21:26:47.838 INFO:tasks.workunit.client.0.smithi121.stdout:7/725: chown d0/d32/c7d 8401 1 2024-04-05T21:26:47.840 INFO:tasks.workunit.client.0.smithi121.stdout:4/759: truncate d5/d11/f1e 1599694 0 2024-04-05T21:26:47.840 INFO:tasks.workunit.client.0.smithi121.stdout:6/501: mknod d1/d2/d1b/d91/cab 0 2024-04-05T21:26:47.841 INFO:tasks.workunit.client.0.smithi121.stdout:0/714: creat d7/dc/da3/df5/ff9 x:0 0 0 2024-04-05T21:26:47.843 INFO:tasks.workunit.client.0.smithi121.stdout:3/534: dwrite d0/d3/f54 [0,4194304] 0 2024-04-05T21:26:47.843 INFO:tasks.workunit.client.0.smithi121.stdout:1/739: symlink d8/d8c/db6/lf7 0 2024-04-05T21:26:47.844 INFO:tasks.workunit.client.0.smithi121.stdout:6/502: read - d1/d2/d1b/d35/f47 zero size 2024-04-05T21:26:47.844 INFO:tasks.workunit.client.1.smithi188.stdout:5/916: write d6/d82/d40/d42/f5e [1890298,85849] 0 2024-04-05T21:26:47.845 INFO:tasks.workunit.client.1.smithi188.stdout:9/936: dread dd/f82 [0,4194304] 0 2024-04-05T21:26:47.846 INFO:tasks.workunit.client.1.smithi188.stdout:5/917: truncate d6/d82/f47 2110743 0 2024-04-05T21:26:47.846 INFO:tasks.workunit.client.0.smithi121.stdout:3/535: write d0/d3/f5e [4297681,111864] 0 2024-04-05T21:26:47.848 INFO:tasks.workunit.client.1.smithi188.stdout:5/918: creat d6/d17/f140 x:0 0 0 2024-04-05T21:26:47.851 INFO:tasks.workunit.client.0.smithi121.stdout:6/503: creat d1/d2/d3/d5/fac x:0 0 0 2024-04-05T21:26:47.853 INFO:tasks.workunit.client.0.smithi121.stdout:4/760: mknod d5/d11/d2c/d9d/dae/d29/d2e/cf3 0 2024-04-05T21:26:47.853 INFO:tasks.workunit.client.0.smithi121.stdout:7/726: dread d0/d9/f69 [0,4194304] 0 2024-04-05T21:26:47.854 INFO:tasks.workunit.client.1.smithi188.stdout:9/937: dwrite dd/d11/d22/f56 [0,4194304] 0 2024-04-05T21:26:47.855 INFO:tasks.workunit.client.1.smithi188.stdout:0/739: write d4/d8/db/d42/d71/f104 [4252833,50534] 0 2024-04-05T21:26:47.855 INFO:tasks.workunit.client.0.smithi121.stdout:3/536: mknod d0/d3/d24/d3d/d1a/d23/cab 0 2024-04-05T21:26:47.856 INFO:tasks.workunit.client.0.smithi121.stdout:6/504: chown d1/d2/d3/d4/d21/d57/d42/d48/d7c/l99 2478 1 2024-04-05T21:26:47.858 INFO:tasks.workunit.client.0.smithi121.stdout:1/740: creat d8/d8c/db6/d1a/d1b/d85/ff8 x:0 0 0 2024-04-05T21:26:47.859 INFO:tasks.workunit.client.0.smithi121.stdout:4/761: mkdir d5/d11/d60/df4 0 2024-04-05T21:26:47.861 INFO:tasks.workunit.client.1.smithi188.stdout:5/919: rename d6/d15/d5c/d7a/d44/d98/f9f to d6/d15/f141 0 2024-04-05T21:26:47.861 INFO:tasks.workunit.client.1.smithi188.stdout:0/740: fdatasync d4/d8/f3d 0 2024-04-05T21:26:47.862 INFO:tasks.workunit.client.0.smithi121.stdout:7/727: rename d0/d23/fc6 to d0/d32/d4e/d73/fe4 0 2024-04-05T21:26:47.863 INFO:tasks.workunit.client.1.smithi188.stdout:5/920: chown d6/d15/d1b/lde 562 1 2024-04-05T21:26:47.863 INFO:tasks.workunit.client.1.smithi188.stdout:0/741: chown d4/d8/d10/f40 11980 1 2024-04-05T21:26:47.865 INFO:tasks.workunit.client.1.smithi188.stdout:5/921: fdatasync d6/d15/d5c/d7a/fd0 0 2024-04-05T21:26:47.868 INFO:tasks.workunit.client.0.smithi121.stdout:3/537: fdatasync d0/d3/de/f82 0 2024-04-05T21:26:47.870 INFO:tasks.workunit.client.1.smithi188.stdout:0/742: creat d4/d8/d10/f10e x:0 0 0 2024-04-05T21:26:47.871 INFO:tasks.workunit.client.0.smithi121.stdout:8/640: dwrite d12/d50/d93/d96/f72 [0,4194304] 0 2024-04-05T21:26:47.871 INFO:tasks.workunit.client.0.smithi121.stdout:6/505: write d1/d2/d3/d5/d1f/f6a [4235878,87275] 0 2024-04-05T21:26:47.875 INFO:tasks.workunit.client.1.smithi188.stdout:0/743: dread d4/d6/d13/d23/d24/ddc/ffa [0,4194304] 0 2024-04-05T21:26:47.876 INFO:tasks.workunit.client.0.smithi121.stdout:2/634: dwrite d2/d9/d1c/d30/f61 [0,4194304] 0 2024-04-05T21:26:47.877 INFO:tasks.workunit.client.0.smithi121.stdout:3/538: dread - d0/d3/d24/d3f/f80 zero size 2024-04-05T21:26:47.877 INFO:tasks.workunit.client.1.smithi188.stdout:0/744: mknod d4/c10f 0 2024-04-05T21:26:47.880 INFO:tasks.workunit.client.0.smithi121.stdout:4/762: truncate d5/f3f 7084341 0 2024-04-05T21:26:47.881 INFO:tasks.workunit.client.0.smithi121.stdout:0/715: write d7/d71/d45/d5a/f69 [4506550,127291] 0 2024-04-05T21:26:47.881 INFO:tasks.workunit.client.1.smithi188.stdout:0/745: creat d4/d8/db/d21/d4a/f110 x:0 0 0 2024-04-05T21:26:47.883 INFO:tasks.workunit.client.0.smithi121.stdout:6/506: fdatasync d1/d2/d1b/d35/d39/d92/f26 0 2024-04-05T21:26:47.890 INFO:tasks.workunit.client.0.smithi121.stdout:3/539: creat d0/d3/d24/d3d/d1a/d23/d38/d55/fac x:0 0 0 2024-04-05T21:26:47.891 INFO:tasks.workunit.client.0.smithi121.stdout:8/641: creat d12/d50/fca x:0 0 0 2024-04-05T21:26:47.891 INFO:tasks.workunit.client.0.smithi121.stdout:6/507: dread d1/d2/d3/d5/fd [4194304,4194304] 0 2024-04-05T21:26:47.892 INFO:tasks.workunit.client.1.smithi188.stdout:3/936: dread d5/d7/d65/d5f/fe2 [0,4194304] 0 2024-04-05T21:26:47.893 INFO:tasks.workunit.client.1.smithi188.stdout:9/938: truncate dd/d11/d1f/d26/d55/d98/da5/fe6 166690 0 2024-04-05T21:26:47.893 INFO:tasks.workunit.client.1.smithi188.stdout:0/746: dwrite d4/d8/db/d21/d4a/d73/f99 [4194304,4194304] 0 2024-04-05T21:26:47.893 INFO:tasks.workunit.client.1.smithi188.stdout:5/922: write d6/f53 [163564,75622] 0 2024-04-05T21:26:47.893 INFO:tasks.workunit.client.0.smithi121.stdout:7/728: dwrite d0/d9/d22/d28/f48 [0,4194304] 0 2024-04-05T21:26:47.893 INFO:tasks.workunit.client.0.smithi121.stdout:3/540: write d0/d3/f54 [2739910,89776] 0 2024-04-05T21:26:47.895 INFO:tasks.workunit.client.0.smithi121.stdout:1/741: truncate d8/d8c/db6/d1a/d1b/d4c/da1/fe6 988169 0 2024-04-05T21:26:47.896 INFO:tasks.workunit.client.1.smithi188.stdout:0/747: write d4/d8/d78/fe1 [18313,3213] 0 2024-04-05T21:26:47.897 INFO:tasks.workunit.client.1.smithi188.stdout:9/939: unlink dd/d11/f3c 0 2024-04-05T21:26:47.898 INFO:tasks.workunit.client.0.smithi121.stdout:4/763: creat d5/d11/d26/d49/ff5 x:0 0 0 2024-04-05T21:26:47.898 INFO:tasks.workunit.client.1.smithi188.stdout:0/748: write d4/d6/d13/d23/de8/f107 [145116,32819] 0 2024-04-05T21:26:47.902 INFO:tasks.workunit.client.1.smithi188.stdout:5/923: truncate d6/dad/f8d 466949 0 2024-04-05T21:26:47.904 INFO:tasks.workunit.client.1.smithi188.stdout:3/937: mkdir d5/d7/d65/d61/d66/da4/de0/d132 0 2024-04-05T21:26:47.905 INFO:tasks.workunit.client.1.smithi188.stdout:3/938: readlink d5/d7/d6d/lcf 0 2024-04-05T21:26:47.905 INFO:tasks.workunit.client.1.smithi188.stdout:9/940: creat dd/d3d/d89/f13b x:0 0 0 2024-04-05T21:26:47.906 INFO:tasks.workunit.client.1.smithi188.stdout:5/924: read d6/d15/d5c/d7a/d44/d98/fcd [1097139,74910] 0 2024-04-05T21:26:47.907 INFO:tasks.workunit.client.1.smithi188.stdout:3/939: rmdir d5/d7/d6d 39 2024-04-05T21:26:47.907 INFO:tasks.workunit.client.0.smithi121.stdout:2/635: mknod d2/d9/d38/d2d/d47/d94/d84/d91/cc5 0 2024-04-05T21:26:47.907 INFO:tasks.workunit.client.0.smithi121.stdout:0/716: symlink d7/d71/d94/dd1/lfa 0 2024-04-05T21:26:47.909 INFO:tasks.workunit.client.1.smithi188.stdout:5/925: write d6/d17/d23/d7d/fd7 [2560937,24977] 0 2024-04-05T21:26:47.910 INFO:tasks.workunit.client.1.smithi188.stdout:3/940: dread - d5/d7/d65/da2/f11b zero size 2024-04-05T21:26:47.910 INFO:tasks.workunit.client.1.smithi188.stdout:5/926: read - d6/d15/f11a zero size 2024-04-05T21:26:47.912 INFO:tasks.workunit.client.0.smithi121.stdout:1/742: fdatasync d8/d8c/db6/d1a/d1b/d24/f39 0 2024-04-05T21:26:47.912 INFO:tasks.workunit.client.0.smithi121.stdout:6/508: symlink d1/lad 0 2024-04-05T21:26:47.912 INFO:tasks.workunit.client.0.smithi121.stdout:8/642: creat d12/d13/d56/d57/da4/dc9/fcb x:0 0 0 2024-04-05T21:26:47.913 INFO:tasks.workunit.client.0.smithi121.stdout:1/743: stat d8/d8c/db6/d1a/d1b/d38 0 2024-04-05T21:26:47.915 INFO:tasks.workunit.client.0.smithi121.stdout:2/636: unlink d2/d9/d38/d2d/d4c/cbd 0 2024-04-05T21:26:47.918 INFO:tasks.workunit.client.0.smithi121.stdout:0/717: getdents d7/d7e 0 2024-04-05T21:26:47.919 INFO:tasks.workunit.client.0.smithi121.stdout:6/509: dwrite d1/d2/d3/d5/d1f/f46 [0,4194304] 0 2024-04-05T21:26:47.920 INFO:tasks.workunit.client.0.smithi121.stdout:0/718: stat d7/dc/le 0 2024-04-05T21:26:47.920 INFO:tasks.workunit.client.0.smithi121.stdout:1/744: creat d8/d10/ff9 x:0 0 0 2024-04-05T21:26:47.921 INFO:tasks.workunit.client.0.smithi121.stdout:8/643: getdents d12/d13/d56 0 2024-04-05T21:26:47.922 INFO:tasks.workunit.client.1.smithi188.stdout:3/941: symlink d5/d7/d65/d61/d66/d6e/dfe/l133 0 2024-04-05T21:26:47.923 INFO:tasks.workunit.client.1.smithi188.stdout:9/941: dwrite dd/d11/d73/d85/f93 [0,4194304] 0 2024-04-05T21:26:47.925 INFO:tasks.workunit.client.1.smithi188.stdout:5/927: getdents d6/d17/d23/d7d 0 2024-04-05T21:26:47.925 INFO:tasks.workunit.client.1.smithi188.stdout:9/942: dread - dd/d3d/d89/dd1/f114 zero size 2024-04-05T21:26:47.927 INFO:tasks.workunit.client.1.smithi188.stdout:5/928: chown d6/d15/d5c 0 1 2024-04-05T21:26:47.928 INFO:tasks.workunit.client.1.smithi188.stdout:5/929: chown d6/d17/d23 859 1 2024-04-05T21:26:47.929 INFO:tasks.workunit.client.0.smithi121.stdout:0/719: creat d7/d71/d94/dd1/dd4/ffb x:0 0 0 2024-04-05T21:26:47.929 INFO:tasks.workunit.client.0.smithi121.stdout:1/745: symlink d8/d8c/db6/d1a/d1b/lfa 0 2024-04-05T21:26:47.929 INFO:tasks.workunit.client.1.smithi188.stdout:9/943: creat dd/d11/d1f/d26/d55/d98/d133/f13c x:0 0 0 2024-04-05T21:26:47.930 INFO:tasks.workunit.client.0.smithi121.stdout:0/720: chown d7/d71/f3b 45041 1 2024-04-05T21:26:47.931 INFO:tasks.workunit.client.0.smithi121.stdout:6/510: getdents d1/d2/d3/d5/d1f 0 2024-04-05T21:26:47.932 INFO:tasks.workunit.client.0.smithi121.stdout:8/644: dwrite d12/d50/fca [0,4194304] 0 2024-04-05T21:26:47.932 INFO:tasks.workunit.client.1.smithi188.stdout:3/942: rename d5/d7/d6d/dd8/d2f/f89 to d5/d7/d65/da2/f134 0 2024-04-05T21:26:47.935 INFO:tasks.workunit.client.0.smithi121.stdout:8/645: readlink d12/d13/d56/d57/da4/l2c 0 2024-04-05T21:26:47.935 INFO:tasks.workunit.client.1.smithi188.stdout:5/930: getdents d6/d15/d5c/d7a/d44/d98/dbf/d123 0 2024-04-05T21:26:47.936 INFO:tasks.workunit.client.1.smithi188.stdout:5/931: chown d6 333926 1 2024-04-05T21:26:47.936 INFO:tasks.workunit.client.0.smithi121.stdout:6/511: creat d1/d32/fae x:0 0 0 2024-04-05T21:26:47.937 INFO:tasks.workunit.client.0.smithi121.stdout:0/721: write d7/d71/d45/d5a/fa9 [657498,118518] 0 2024-04-05T21:26:47.937 INFO:tasks.workunit.client.1.smithi188.stdout:5/932: creat d6/d15/d5c/d7a/d44/d51/dc6/f142 x:0 0 0 2024-04-05T21:26:47.939 INFO:tasks.workunit.client.0.smithi121.stdout:0/722: chown d7/dc/l4a 2 1 2024-04-05T21:26:47.940 INFO:tasks.workunit.client.0.smithi121.stdout:8/646: mkdir d12/d50/d6a/d63/d8b/dcc 0 2024-04-05T21:26:47.940 INFO:tasks.workunit.client.1.smithi188.stdout:5/933: rename d6/d15/ff8 to d6/d82/d40/d42/d81/d95/f143 0 2024-04-05T21:26:47.942 INFO:tasks.workunit.client.0.smithi121.stdout:1/746: mknod d8/d8c/db6/d1a/d1b/d85/cfb 0 2024-04-05T21:26:47.942 INFO:tasks.workunit.client.1.smithi188.stdout:5/934: readlink d6/d15/d5c/d7a/d44/d98/lf4 0 2024-04-05T21:26:47.945 INFO:tasks.workunit.client.0.smithi121.stdout:0/723: link d7/d71/d29/d2a/lea d7/d71/d29/d3f/lfc 0 2024-04-05T21:26:47.945 INFO:tasks.workunit.client.0.smithi121.stdout:1/747: fdatasync d8/d8c/db6/d1a/d1b/d24/fe9 0 2024-04-05T21:26:47.953 INFO:tasks.workunit.client.0.smithi121.stdout:3/541: dread d0/d3/d24/f3c [0,4194304] 0 2024-04-05T21:26:47.954 INFO:tasks.workunit.client.0.smithi121.stdout:1/748: rename d8/d8c/l96 to d8/d89/d9e/d2d/d84/da8/dad/lfc 0 2024-04-05T21:26:47.955 INFO:tasks.workunit.client.0.smithi121.stdout:6/512: dread d1/d2/d3/d5/f66 [0,4194304] 0 2024-04-05T21:26:47.956 INFO:tasks.workunit.client.0.smithi121.stdout:1/749: fdatasync d8/d89/d9e/d2d/fd1 0 2024-04-05T21:26:47.956 INFO:tasks.workunit.client.1.smithi188.stdout:0/749: write d4/d6/d46/fb7 [344466,83897] 0 2024-04-05T21:26:47.956 INFO:tasks.workunit.client.1.smithi188.stdout:5/935: dwrite d6/d15/d5c/d7a/d44/fda [0,4194304] 0 2024-04-05T21:26:47.960 INFO:tasks.workunit.client.0.smithi121.stdout:1/750: fsync d8/d8c/db6/f23 0 2024-04-05T21:26:47.962 INFO:tasks.workunit.client.0.smithi121.stdout:3/542: link d0/d3/de/f8a d0/d3/d24/d3d/d1a/d23/d90/d9e/fad 0 2024-04-05T21:26:47.966 INFO:tasks.workunit.client.0.smithi121.stdout:4/764: dwrite d5/f55 [0,4194304] 0 2024-04-05T21:26:47.966 INFO:tasks.workunit.client.1.smithi188.stdout:3/943: write d5/d7/d65/f54 [4226996,3727] 0 2024-04-05T21:26:47.966 INFO:tasks.workunit.client.1.smithi188.stdout:9/944: write dd/d11/d1f/d26/dc0/f84 [81164,121558] 0 2024-04-05T21:26:47.966 INFO:tasks.workunit.client.1.smithi188.stdout:5/936: creat d6/d106/de1/f144 x:0 0 0 2024-04-05T21:26:47.969 INFO:tasks.workunit.client.1.smithi188.stdout:9/945: chown dd/d11/d1f/d26/dc0/d6b/d40/d59/f7f 4130284 1 2024-04-05T21:26:47.969 INFO:tasks.workunit.client.0.smithi121.stdout:6/513: rename d1/d2/d3/d4/d21/d57/d86/d4f to d1/d2/daf 0 2024-04-05T21:26:47.971 INFO:tasks.workunit.client.0.smithi121.stdout:1/751: mknod d8/d8c/db6/d1a/d1b/d85/cfd 0 2024-04-05T21:26:47.971 INFO:tasks.workunit.client.1.smithi188.stdout:3/944: creat d5/d7/d6d/dd8/d2f/f135 x:0 0 0 2024-04-05T21:26:47.971 INFO:tasks.workunit.client.0.smithi121.stdout:3/543: mkdir d0/d3/d24/d3d/d1a/d23/d38/dae 0 2024-04-05T21:26:47.972 INFO:tasks.workunit.client.1.smithi188.stdout:5/937: creat d6/d106/de1/f145 x:0 0 0 2024-04-05T21:26:47.974 INFO:tasks.workunit.client.1.smithi188.stdout:9/946: unlink dd/d11/d1f/d26/d55/dcf/fff 0 2024-04-05T21:26:47.974 INFO:tasks.workunit.client.1.smithi188.stdout:3/945: mkdir d5/ddd/d136 0 2024-04-05T21:26:47.975 INFO:tasks.workunit.client.0.smithi121.stdout:4/765: mkdir d5/d11/d26/dbc/dca/de9/df6 0 2024-04-05T21:26:47.976 INFO:tasks.workunit.client.0.smithi121.stdout:1/752: write d8/d8c/db6/d1a/d1b/d24/dc2/fcc [840238,18929] 0 2024-04-05T21:26:47.979 INFO:tasks.workunit.client.0.smithi121.stdout:4/766: write d5/f55 [948542,19348] 0 2024-04-05T21:26:47.982 INFO:tasks.workunit.client.0.smithi121.stdout:8/647: dread d12/d13/d15/f4b [0,4194304] 0 2024-04-05T21:26:47.983 INFO:tasks.workunit.client.0.smithi121.stdout:3/544: link d0/d3/f7f d0/d3/d24/d3d/faf 0 2024-04-05T21:26:47.983 INFO:tasks.workunit.client.0.smithi121.stdout:6/514: dwrite d1/d32/fae [0,4194304] 0 2024-04-05T21:26:47.985 INFO:tasks.workunit.client.0.smithi121.stdout:1/753: chown d8/d8c/db6/d1a/d1b/d24/l90 465444 1 2024-04-05T21:26:47.986 INFO:tasks.workunit.client.0.smithi121.stdout:8/648: fdatasync d12/d50/d61/faa 0 2024-04-05T21:26:47.987 INFO:tasks.workunit.client.0.smithi121.stdout:4/767: read d5/d11/d28/f1b [1865488,104194] 0 2024-04-05T21:26:47.988 INFO:tasks.workunit.client.0.smithi121.stdout:3/545: fdatasync d0/f7 0 2024-04-05T21:26:47.988 INFO:tasks.workunit.client.1.smithi188.stdout:3/946: dwrite d5/f37 [0,4194304] 0 2024-04-05T21:26:47.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:47 smithi121 ceph-mon[39723]: pgmap v22: 65 pgs: 65 active+clean; 2.9 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 104 MiB/s rd, 292 MiB/s wr, 584 op/s 2024-04-05T21:26:47.989 INFO:tasks.workunit.client.1.smithi188.stdout:5/938: dwrite d6/d17/f13b [0,4194304] 0 2024-04-05T21:26:47.996 INFO:tasks.workunit.client.0.smithi121.stdout:0/724: dwrite f2 [0,4194304] 0 2024-04-05T21:26:47.996 INFO:tasks.workunit.client.0.smithi121.stdout:8/649: creat d12/d50/d6a/d63/d8b/fcd x:0 0 0 2024-04-05T21:26:47.997 INFO:tasks.workunit.client.1.smithi188.stdout:5/939: mkdir d6/d82/d40/d42/d81/d13f/d146 0 2024-04-05T21:26:47.998 INFO:tasks.workunit.client.1.smithi188.stdout:0/750: dwrite d4/d8/db/d21/d3c/fa9 [0,4194304] 0 2024-04-05T21:26:48.000 INFO:tasks.workunit.client.0.smithi121.stdout:1/754: rmdir d8/d8c/db6/d1a/d1b/d44 39 2024-04-05T21:26:48.001 INFO:tasks.workunit.client.0.smithi121.stdout:1/755: read - d8/fea zero size 2024-04-05T21:26:48.006 INFO:tasks.workunit.client.0.smithi121.stdout:6/515: rename d1/d2/d3/d4/d21/d57/d86/c6e to d1/d2/d3/d4/d21/cb0 0 2024-04-05T21:26:48.008 INFO:tasks.workunit.client.0.smithi121.stdout:0/725: stat d7/d62/d9d/fd6 0 2024-04-05T21:26:48.009 INFO:tasks.workunit.client.1.smithi188.stdout:3/947: dwrite d5/d7/d65/da2/f11b [0,4194304] 0 2024-04-05T21:26:48.009 INFO:tasks.workunit.client.0.smithi121.stdout:8/650: symlink d12/d50/d61/lce 0 2024-04-05T21:26:48.012 INFO:tasks.workunit.client.0.smithi121.stdout:4/768: rmdir d5/d45/d67/dab/dc8 0 2024-04-05T21:26:48.013 INFO:tasks.workunit.client.0.smithi121.stdout:4/769: write f4 [1016602,112761] 0 2024-04-05T21:26:48.015 INFO:tasks.workunit.client.1.smithi188.stdout:3/948: read d5/df/f6c [120225,41445] 0 2024-04-05T21:26:48.016 INFO:tasks.workunit.client.1.smithi188.stdout:5/940: dwrite d6/d15/f61 [0,4194304] 0 2024-04-05T21:26:48.017 INFO:tasks.workunit.client.1.smithi188.stdout:3/949: read - d5/d35/f10b zero size 2024-04-05T21:26:48.019 INFO:tasks.workunit.client.0.smithi121.stdout:8/651: symlink d12/d50/d61/d7f/d9a/lcf 0 2024-04-05T21:26:48.022 INFO:tasks.workunit.client.0.smithi121.stdout:0/726: creat d7/d71/d29/d2a/ffd x:0 0 0 2024-04-05T21:26:48.024 INFO:tasks.workunit.client.1.smithi188.stdout:5/941: rmdir d6/d17/d23/d7d 39 2024-04-05T21:26:48.024 INFO:tasks.workunit.client.0.smithi121.stdout:6/516: getdents d1/d2/d1b/d35/d51 0 2024-04-05T21:26:48.024 INFO:tasks.workunit.client.0.smithi121.stdout:3/546: dwrite d0/d3/d24/f89 [0,4194304] 0 2024-04-05T21:26:48.024 INFO:tasks.workunit.client.0.smithi121.stdout:8/652: readlink d12/d13/d15/d1d/l69 0 2024-04-05T21:26:48.026 INFO:tasks.workunit.client.0.smithi121.stdout:4/770: rename d5/d11/d2c/d9d/dae/d29/db4/dd4 to d5/d11/d26/d47/df7 0 2024-04-05T21:26:48.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:47 smithi188 ceph-mon[49715]: pgmap v22: 65 pgs: 65 active+clean; 2.9 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 104 MiB/s rd, 292 MiB/s wr, 584 op/s 2024-04-05T21:26:48.031 INFO:tasks.workunit.client.0.smithi121.stdout:6/517: dread - d1/d2/d3/d4/f88 zero size 2024-04-05T21:26:48.031 INFO:tasks.workunit.client.0.smithi121.stdout:0/727: chown d7/d71/d29/d3f/fba 4429103 1 2024-04-05T21:26:48.031 INFO:tasks.workunit.client.0.smithi121.stdout:3/547: chown d0/d3/f7f 0 1 2024-04-05T21:26:48.032 INFO:tasks.workunit.client.0.smithi121.stdout:8/653: truncate d12/d50/d61/fbf 1043554 0 2024-04-05T21:26:48.032 INFO:tasks.workunit.client.1.smithi188.stdout:3/950: dwrite d5/d7/d65/d5f/f10e [0,4194304] 0 2024-04-05T21:26:48.032 INFO:tasks.workunit.client.0.smithi121.stdout:6/518: mknod d1/d2/d3/d5/cb1 0 2024-04-05T21:26:48.034 INFO:tasks.workunit.client.1.smithi188.stdout:9/947: dread dd/d11/d1f/d26/dc0/d6b/f96 [0,4194304] 0 2024-04-05T21:26:48.034 INFO:tasks.workunit.client.0.smithi121.stdout:6/519: write d1/d2/d3/d4/d21/d57/d42/d48/d7c/fa9 [277441,110097] 0 2024-04-05T21:26:48.035 INFO:tasks.workunit.client.1.smithi188.stdout:5/942: rename d6/d82/f3b to d6/d15/d5c/d7a/d44/d51/df1/d104/d124/f147 0 2024-04-05T21:26:48.035 INFO:tasks.workunit.client.0.smithi121.stdout:8/654: rmdir d12/d13 39 2024-04-05T21:26:48.036 INFO:tasks.workunit.client.1.smithi188.stdout:9/948: chown dd/d11/d73/d8d/fac 125 1 2024-04-05T21:26:48.036 INFO:tasks.workunit.client.0.smithi121.stdout:6/520: dread - d1/d2/d1b/da1/fa6 zero size 2024-04-05T21:26:48.036 INFO:tasks.workunit.client.1.smithi188.stdout:5/943: chown d6/d15/d5c/d7a/d44 22159 1 2024-04-05T21:26:48.037 INFO:tasks.workunit.client.0.smithi121.stdout:0/728: creat d7/d71/d29/de8/ffe x:0 0 0 2024-04-05T21:26:48.037 INFO:tasks.workunit.client.1.smithi188.stdout:9/949: stat dd/d3d/d95/le2 0 2024-04-05T21:26:48.038 INFO:tasks.workunit.client.0.smithi121.stdout:8/655: stat d12/d13/d15/d1d/fa9 0 2024-04-05T21:26:48.038 INFO:tasks.workunit.client.0.smithi121.stdout:6/521: symlink d1/d2/dc/d9a/lb2 0 2024-04-05T21:26:48.041 INFO:tasks.workunit.client.0.smithi121.stdout:0/729: mknod d7/d71/d94/cff 0 2024-04-05T21:26:48.042 INFO:tasks.workunit.client.1.smithi188.stdout:5/944: creat d6/d82/d40/d42/d81/da6/db2/d107/f148 x:0 0 0 2024-04-05T21:26:48.044 INFO:tasks.workunit.client.1.smithi188.stdout:3/951: link d5/d7/d65/d61/d66/da4/d114/l11e d5/d7/d65/d61/d66/da4/d114/l137 0 2024-04-05T21:26:48.045 INFO:tasks.workunit.client.1.smithi188.stdout:5/945: rmdir d6/d17/d23/d7d/d128 39 2024-04-05T21:26:48.045 INFO:tasks.workunit.client.1.smithi188.stdout:3/952: readlink d5/d7/d6d/d75/l117 0 2024-04-05T21:26:48.046 INFO:tasks.workunit.client.1.smithi188.stdout:0/751: dread d4/d8/d39/f9a [0,4194304] 0 2024-04-05T21:26:48.048 INFO:tasks.workunit.client.1.smithi188.stdout:3/953: creat d5/d7/d6d/f138 x:0 0 0 2024-04-05T21:26:48.049 INFO:tasks.workunit.client.1.smithi188.stdout:3/954: write d5/d7/d31/fb1 [2684074,85652] 0 2024-04-05T21:26:48.050 INFO:tasks.workunit.client.1.smithi188.stdout:5/946: creat d6/d106/d8e/def/d13a/f149 x:0 0 0 2024-04-05T21:26:48.052 INFO:tasks.workunit.client.1.smithi188.stdout:5/947: mknod d6/d82/d40/d42/d81/da6/c14a 0 2024-04-05T21:26:48.053 INFO:tasks.workunit.client.1.smithi188.stdout:0/752: creat d4/d6/d13/f111 x:0 0 0 2024-04-05T21:26:48.056 INFO:tasks.workunit.client.0.smithi121.stdout:5/783: sync 2024-04-05T21:26:48.057 INFO:tasks.workunit.client.1.smithi188.stdout:5/948: creat d6/d17/d23/d7d/dcf/f14b x:0 0 0 2024-04-05T21:26:48.057 INFO:tasks.workunit.client.1.smithi188.stdout:1/905: sync 2024-04-05T21:26:48.057 INFO:tasks.workunit.client.1.smithi188.stdout:2/881: sync 2024-04-05T21:26:48.058 INFO:tasks.workunit.client.1.smithi188.stdout:5/949: dread - d6/d106/de1/f144 zero size 2024-04-05T21:26:48.058 INFO:tasks.workunit.client.0.smithi121.stdout:6/522: rename d1/d2/d3/d4/d21/d57/d86 to d1/d2/d3/d4/d21/d57/d42/d48/d7c/db3 0 2024-04-05T21:26:48.058 INFO:tasks.workunit.client.1.smithi188.stdout:0/753: mkdir d4/d8/db/dac/d112 0 2024-04-05T21:26:48.062 INFO:tasks.workunit.client.1.smithi188.stdout:9/950: write dd/d11/d1f/d26/dc0/d6b/f2d [6921112,82801] 0 2024-04-05T21:26:48.063 INFO:tasks.workunit.client.0.smithi121.stdout:0/730: rename d7/d71/d29/d2a/d39/c91 to d7/d71/d94/da0/c100 0 2024-04-05T21:26:48.068 INFO:tasks.workunit.client.0.smithi121.stdout:6/523: dwrite d1/d2/d3/d5/d1f/f46 [4194304,4194304] 0 2024-04-05T21:26:48.069 INFO:tasks.workunit.client.1.smithi188.stdout:1/906: dwrite d1/dc/d16/d21/d27/ff3 [0,4194304] 0 2024-04-05T21:26:48.070 INFO:tasks.workunit.client.1.smithi188.stdout:9/951: stat dd/d11/d22/d7e 0 2024-04-05T21:26:48.070 INFO:tasks.workunit.client.0.smithi121.stdout:4/771: dread d5/d11/d28/d3b/fef [0,4194304] 0 2024-04-05T21:26:48.070 INFO:tasks.workunit.client.0.smithi121.stdout:0/731: unlink d7/dc/d95/f96 0 2024-04-05T21:26:48.070 INFO:tasks.workunit.client.0.smithi121.stdout:5/784: link dd/de/d19/f57 dd/de/d39/d3b/d50/f118 0 2024-04-05T21:26:48.071 INFO:tasks.workunit.client.1.smithi188.stdout:3/955: write d5/d7/d65/d61/d66/f9f [3346923,79835] 0 2024-04-05T21:26:48.071 INFO:tasks.workunit.client.1.smithi188.stdout:2/882: write d5/db/dd/d73/d109/faa [1381433,103007] 0 2024-04-05T21:26:48.072 INFO:tasks.workunit.client.0.smithi121.stdout:3/548: truncate d0/f2d 2171912 0 2024-04-05T21:26:48.073 INFO:tasks.workunit.client.1.smithi188.stdout:5/950: dwrite d6/d15/d5c/d7a/f2b [0,4194304] 0 2024-04-05T21:26:48.073 INFO:tasks.workunit.client.1.smithi188.stdout:3/956: readlink d5/d48/l105 0 2024-04-05T21:26:48.073 INFO:tasks.workunit.client.1.smithi188.stdout:9/952: write dd/d3d/d89/fc3 [7656692,84275] 0 2024-04-05T21:26:48.074 INFO:tasks.workunit.client.0.smithi121.stdout:3/549: readlink d0/d3/d24/d6d/l98 0 2024-04-05T21:26:48.076 INFO:tasks.workunit.client.0.smithi121.stdout:4/772: fdatasync d5/f16 0 2024-04-05T21:26:48.076 INFO:tasks.workunit.client.0.smithi121.stdout:3/550: write d0/d3/d24/d3d/d1a/d23/d38/d55/fac [24846,18085] 0 2024-04-05T21:26:48.079 INFO:tasks.workunit.client.0.smithi121.stdout:0/732: link d7/d1c/fb6 d7/d71/d29/d2a/d39/f101 0 2024-04-05T21:26:48.080 INFO:tasks.workunit.client.0.smithi121.stdout:8/656: write d12/d50/d6a/d63/d8b/f8d [123510,129845] 0 2024-04-05T21:26:48.082 INFO:tasks.workunit.client.1.smithi188.stdout:2/883: write d5/db/dd/d73/d109/faa [2061066,90030] 0 2024-04-05T21:26:48.084 INFO:tasks.workunit.client.0.smithi121.stdout:8/657: write d12/d50/d6a/d63/d9c/fb1 [82219,99822] 0 2024-04-05T21:26:48.086 INFO:tasks.workunit.client.0.smithi121.stdout:8/658: write d12/d50/d6a/d63/d8b/fcd [606617,109860] 0 2024-04-05T21:26:48.088 INFO:tasks.workunit.client.1.smithi188.stdout:3/957: mkdir d5/d7/d65/d139 0 2024-04-05T21:26:48.089 INFO:tasks.workunit.client.0.smithi121.stdout:8/659: write d12/d50/d6a/d63/d8b/fcd [1764751,193] 0 2024-04-05T21:26:48.089 INFO:tasks.workunit.client.0.smithi121.stdout:3/551: creat d0/d3/fb0 x:0 0 0 2024-04-05T21:26:48.089 INFO:tasks.workunit.client.0.smithi121.stdout:5/785: link dd/de/d19/d38/da3/ca9 dd/de/d39/d3b/c119 0 2024-04-05T21:26:48.090 INFO:tasks.workunit.client.1.smithi188.stdout:5/951: mkdir d6/d17/d23/d7d/dfb/d14c 0 2024-04-05T21:26:48.090 INFO:tasks.workunit.client.1.smithi188.stdout:1/907: dwrite d1/dc/d16/d21/d27/d4a/fad [4194304,4194304] 0 2024-04-05T21:26:48.092 INFO:tasks.workunit.client.0.smithi121.stdout:3/552: mkdir d0/d3/d24/d3d/d1a/d1e/db1 0 2024-04-05T21:26:48.093 INFO:tasks.workunit.client.0.smithi121.stdout:5/786: creat dd/de/d19/d13/d53/f11a x:0 0 0 2024-04-05T21:26:48.093 INFO:tasks.workunit.client.1.smithi188.stdout:1/908: chown d1/dc/d16/d2d/db0 204457681 1 2024-04-05T21:26:48.094 INFO:tasks.workunit.client.0.smithi121.stdout:3/553: chown d0/d3/d24/d3d/l50 34 1 2024-04-05T21:26:48.095 INFO:tasks.workunit.client.1.smithi188.stdout:2/884: mkdir d5/db/dd/d73/d109/d5d/d122 0 2024-04-05T21:26:48.096 INFO:tasks.workunit.client.0.smithi121.stdout:5/787: chown dd/de/dc1/cf0 464 1 2024-04-05T21:26:48.096 INFO:tasks.workunit.client.1.smithi188.stdout:1/909: fdatasync d1/dc/d16/d21/d27/d49/fb8 0 2024-04-05T21:26:48.097 INFO:tasks.workunit.client.1.smithi188.stdout:5/952: symlink d6/d15/d5c/d7a/d44/d9b/l14d 0 2024-04-05T21:26:48.097 INFO:tasks.workunit.client.0.smithi121.stdout:3/554: creat d0/d3/de/da6/fb2 x:0 0 0 2024-04-05T21:26:48.101 INFO:tasks.workunit.client.1.smithi188.stdout:9/953: dread dd/d11/d1f/f29 [0,4194304] 0 2024-04-05T21:26:48.101 INFO:tasks.workunit.client.1.smithi188.stdout:1/910: read - d1/d7/dbf/fd9 zero size 2024-04-05T21:26:48.101 INFO:tasks.workunit.client.1.smithi188.stdout:3/958: getdents d5 0 2024-04-05T21:26:48.101 INFO:tasks.workunit.client.1.smithi188.stdout:8/711: sync 2024-04-05T21:26:48.101 INFO:tasks.workunit.client.1.smithi188.stdout:6/971: sync 2024-04-05T21:26:48.103 INFO:tasks.workunit.client.1.smithi188.stdout:9/954: chown dd/d35/cb7 122 1 2024-04-05T21:26:48.103 INFO:tasks.workunit.client.1.smithi188.stdout:0/754: dread d4/d6/d61/fae [0,4194304] 0 2024-04-05T21:26:48.105 INFO:tasks.workunit.client.1.smithi188.stdout:5/953: creat d6/d17/d23/d7d/dcf/f14e x:0 0 0 2024-04-05T21:26:48.106 INFO:tasks.workunit.client.1.smithi188.stdout:1/911: dread - d1/dc/d16/d21/ffa zero size 2024-04-05T21:26:48.108 INFO:tasks.workunit.client.1.smithi188.stdout:6/972: unlink d6/d13/d18/d19/d2c/d3f/d77/fc0 0 2024-04-05T21:26:48.109 INFO:tasks.workunit.client.1.smithi188.stdout:8/712: write d5/d6e/f84 [434589,841] 0 2024-04-05T21:26:48.109 INFO:tasks.workunit.client.1.smithi188.stdout:5/954: chown d6/d15/d5c/fa3 478 1 2024-04-05T21:26:48.111 INFO:tasks.workunit.client.1.smithi188.stdout:3/959: symlink d5/d7/d6d/d75/l13a 0 2024-04-05T21:26:48.118 INFO:tasks.workunit.client.1.smithi188.stdout:1/912: read d1/dc/d16/d46/d88/d13/f50 [14681,109156] 0 2024-04-05T21:26:48.118 INFO:tasks.workunit.client.0.smithi121.stdout:4/773: truncate d5/d11/d2c/d9d/f31 2914093 0 2024-04-05T21:26:48.118 INFO:tasks.workunit.client.0.smithi121.stdout:0/733: dread d7/d71/d45/d5a/f69 [0,4194304] 0 2024-04-05T21:26:48.119 INFO:tasks.workunit.client.0.smithi121.stdout:0/734: readlink d7/d62/l22 0 2024-04-05T21:26:48.120 INFO:tasks.workunit.client.1.smithi188.stdout:2/885: truncate d5/db/dd/f11 4459672 0 2024-04-05T21:26:48.121 INFO:tasks.workunit.client.1.smithi188.stdout:3/960: symlink d5/d7/d65/da2/l13b 0 2024-04-05T21:26:48.121 INFO:tasks.workunit.client.0.smithi121.stdout:4/774: symlink d5/d11/d26/dbc/dc5/dd5/lf8 0 2024-04-05T21:26:48.122 INFO:tasks.workunit.client.0.smithi121.stdout:4/775: chown d5/fa 88 1 2024-04-05T21:26:48.122 INFO:tasks.workunit.client.1.smithi188.stdout:5/955: fdatasync d6/d15/d5c/d7a/d44/d98/fcd 0 2024-04-05T21:26:48.123 INFO:tasks.workunit.client.0.smithi121.stdout:0/735: unlink d7/d71/d45/d5a/fa9 0 2024-04-05T21:26:48.123 INFO:tasks.workunit.client.1.smithi188.stdout:9/955: link dd/d11/d73/dbb/cbc dd/d11/d1f/c13d 0 2024-04-05T21:26:48.124 INFO:tasks.workunit.client.1.smithi188.stdout:3/961: creat d5/d7/d65/d61/d66/da4/de3/f13c x:0 0 0 2024-04-05T21:26:48.125 INFO:tasks.workunit.client.1.smithi188.stdout:9/956: dread - dd/d11/d1f/f121 zero size 2024-04-05T21:26:48.126 INFO:tasks.workunit.client.0.smithi121.stdout:3/555: truncate d0/d3/f5b 1132268 0 2024-04-05T21:26:48.126 INFO:tasks.workunit.client.0.smithi121.stdout:4/776: write d5/d11/d28/fe [1779998,97344] 0 2024-04-05T21:26:48.127 INFO:tasks.workunit.client.1.smithi188.stdout:2/886: mknod d5/db/dd/d36/d3c/d48/d7a/d7c/c123 0 2024-04-05T21:26:48.127 INFO:tasks.workunit.client.1.smithi188.stdout:9/957: stat dd/d11/d73/l8f 0 2024-04-05T21:26:48.128 INFO:tasks.workunit.client.0.smithi121.stdout:0/736: rename d7/d71/d29/d2a/d39/f40 to d7/d62/ddb/f102 0 2024-04-05T21:26:48.129 INFO:tasks.workunit.client.0.smithi121.stdout:3/556: chown d0/d3/d24/d6d/d7b/da3 267360 1 2024-04-05T21:26:48.130 INFO:tasks.workunit.client.0.smithi121.stdout:0/737: dread - d7/d71/d29/d3f/fba zero size 2024-04-05T21:26:48.131 INFO:tasks.workunit.client.0.smithi121.stdout:5/788: dwrite dd/de/d19/d38/f7c [0,4194304] 0 2024-04-05T21:26:48.132 INFO:tasks.workunit.client.1.smithi188.stdout:6/973: write d6/d13/d18/d19/d2c/f65 [570908,62298] 0 2024-04-05T21:26:48.135 INFO:tasks.workunit.client.0.smithi121.stdout:4/777: truncate d5/d11/d28/d87/f96 89070 0 2024-04-05T21:26:48.136 INFO:tasks.workunit.client.0.smithi121.stdout:0/738: symlink d7/dc/da3/df5/l103 0 2024-04-05T21:26:48.136 INFO:tasks.workunit.client.0.smithi121.stdout:0/739: chown d7/d62/d3c 9 1 2024-04-05T21:26:48.138 INFO:tasks.workunit.client.1.smithi188.stdout:5/956: getdents d6/d106 0 2024-04-05T21:26:48.138 INFO:tasks.workunit.client.0.smithi121.stdout:4/778: rmdir d5/d11/d2c/d9d/dae/d29/d58 39 2024-04-05T21:26:48.139 INFO:tasks.workunit.client.0.smithi121.stdout:4/779: write d5/f86 [1067806,70233] 0 2024-04-05T21:26:48.141 INFO:tasks.workunit.client.0.smithi121.stdout:0/740: creat d7/d62/d3c/f104 x:0 0 0 2024-04-05T21:26:48.143 INFO:tasks.workunit.client.0.smithi121.stdout:4/780: creat d5/d11/d28/d3b/ff9 x:0 0 0 2024-04-05T21:26:48.143 INFO:tasks.workunit.client.1.smithi188.stdout:6/974: rename d6/d13/d18/d19 to d6/d13/d18/d19/dc8/da4/dfa/d12e 22 2024-04-05T21:26:48.144 INFO:tasks.workunit.client.0.smithi121.stdout:4/781: write d5/d89/fd1 [1627353,93097] 0 2024-04-05T21:26:48.145 INFO:tasks.workunit.client.0.smithi121.stdout:4/782: readlink d5/d11/d28/lc4 0 2024-04-05T21:26:48.145 INFO:tasks.workunit.client.1.smithi188.stdout:0/755: dread d4/d8/db/d42/d38/f72 [0,4194304] 0 2024-04-05T21:26:48.151 INFO:tasks.workunit.client.0.smithi121.stdout:0/741: link d7/d71/d29/d2a/f4c d7/d71/d29/de8/f105 0 2024-04-05T21:26:48.152 INFO:tasks.workunit.client.1.smithi188.stdout:5/957: mkdir d6/d17/d23/d7d/d128/d14f 0 2024-04-05T21:26:48.152 INFO:tasks.workunit.client.0.smithi121.stdout:0/742: chown d7/d71/d29/de1/feb 500 1 2024-04-05T21:26:48.152 INFO:tasks.workunit.client.0.smithi121.stdout:3/557: write d0/d3/d24/d3d/f21 [1919789,90933] 0 2024-04-05T21:26:48.155 INFO:tasks.workunit.client.1.smithi188.stdout:8/713: dread d5/dc/f32 [0,4194304] 0 2024-04-05T21:26:48.155 INFO:tasks.workunit.client.0.smithi121.stdout:0/743: mkdir d7/d71/d45/d46/d106 0 2024-04-05T21:26:48.159 INFO:tasks.workunit.client.1.smithi188.stdout:2/887: dwrite d5/db/dd/d36/d3c/d48/d7a/d7c/f86 [0,4194304] 0 2024-04-05T21:26:48.159 INFO:tasks.workunit.client.1.smithi188.stdout:9/958: dwrite dd/d11/d1f/d25/f50 [0,4194304] 0 2024-04-05T21:26:48.160 INFO:tasks.workunit.client.0.smithi121.stdout:0/744: link d7/d71/d29/d2a/d2c/f59 d7/d62/d3c/f107 0 2024-04-05T21:26:48.161 INFO:tasks.workunit.client.1.smithi188.stdout:0/756: symlink d4/d8/db/dcb/l113 0 2024-04-05T21:26:48.161 INFO:tasks.workunit.client.0.smithi121.stdout:0/745: chown d7/d71/d29/d2a/d2c/l48 120 1 2024-04-05T21:26:48.164 INFO:tasks.workunit.client.0.smithi121.stdout:0/746: mknod d7/d71/d29/d2a/d2c/c108 0 2024-04-05T21:26:48.170 INFO:tasks.workunit.client.1.smithi188.stdout:8/714: creat d5/d21/d40/d43/d44/d59/d8b/feb x:0 0 0 2024-04-05T21:26:48.170 INFO:tasks.workunit.client.1.smithi188.stdout:6/975: rename d6/d13/d18/d19/d2c/d3f/f7a to d6/d13/d18/d19/d2c/db2/f12f 0 2024-04-05T21:26:48.170 INFO:tasks.workunit.client.0.smithi121.stdout:4/783: dwrite d5/d11/d2c/d9d/dae/d29/d2e/f4d [0,4194304] 0 2024-04-05T21:26:48.172 INFO:tasks.workunit.client.1.smithi188.stdout:5/958: dwrite d6/d17/fb8 [0,4194304] 0 2024-04-05T21:26:48.173 INFO:tasks.workunit.client.0.smithi121.stdout:0/747: write d7/d1c/f55 [2874743,98024] 0 2024-04-05T21:26:48.174 INFO:tasks.workunit.client.1.smithi188.stdout:8/715: readlink d5/d21/d40/d43/lea 0 2024-04-05T21:26:48.176 INFO:tasks.workunit.client.1.smithi188.stdout:3/962: dread d5/d7/d65/d5f/f120 [0,4194304] 0 2024-04-05T21:26:48.176 INFO:tasks.workunit.client.1.smithi188.stdout:0/757: creat d4/d6/d13/d23/d24/d101/f114 x:0 0 0 2024-04-05T21:26:48.176 INFO:tasks.workunit.client.0.smithi121.stdout:3/558: dread d0/d3/f4 [0,4194304] 0 2024-04-05T21:26:48.176 INFO:tasks.workunit.client.0.smithi121.stdout:0/748: truncate d7/d62/fc2 4965778 0 2024-04-05T21:26:48.178 INFO:tasks.workunit.client.1.smithi188.stdout:2/888: rename d5/db/dd/d73/d109/f7b to d5/db/dd/d24/d116/f124 0 2024-04-05T21:26:48.180 INFO:tasks.workunit.client.0.smithi121.stdout:0/749: fdatasync d7/d71/d29/d2a/ffd 0 2024-04-05T21:26:48.182 INFO:tasks.workunit.client.0.smithi121.stdout:3/559: mkdir d0/d3/d24/db3 0 2024-04-05T21:26:48.183 INFO:tasks.workunit.client.1.smithi188.stdout:5/959: write d6/d15/d1b/f62 [1313911,120506] 0 2024-04-05T21:26:48.183 INFO:tasks.workunit.client.1.smithi188.stdout:0/758: chown d4/d6/d13/da1/fb1 10807 1 2024-04-05T21:26:48.183 INFO:tasks.workunit.client.1.smithi188.stdout:8/716: write d5/dc/d8f/fd7 [717212,101486] 0 2024-04-05T21:26:48.184 INFO:tasks.workunit.client.0.smithi121.stdout:0/750: creat d7/d71/d29/de8/f109 x:0 0 0 2024-04-05T21:26:48.185 INFO:tasks.workunit.client.0.smithi121.stdout:3/560: fdatasync d0/d3/de/f8a 0 2024-04-05T21:26:48.186 INFO:tasks.workunit.client.0.smithi121.stdout:4/784: dwrite d5/d11/d28/d3b/ff9 [0,4194304] 0 2024-04-05T21:26:48.187 INFO:tasks.workunit.client.1.smithi188.stdout:5/960: dread - d6/d17/f140 zero size 2024-04-05T21:26:48.187 INFO:tasks.workunit.client.0.smithi121.stdout:3/561: stat d0/d3/d24/d3d/d1a/d23/d38/c68 0 2024-04-05T21:26:48.192 INFO:tasks.workunit.client.1.smithi188.stdout:0/759: symlink d4/d8/db/d21/d3c/dee/l115 0 2024-04-05T21:26:48.193 INFO:tasks.workunit.client.1.smithi188.stdout:9/959: dwrite dd/d11/d1f/d26/dc0/d6b/f2d [8388608,4194304] 0 2024-04-05T21:26:48.195 INFO:tasks.workunit.client.0.smithi121.stdout:3/562: truncate d0/d3/d24/d3d/d1a/d23/d38/d55/fac 874735 0 2024-04-05T21:26:48.195 INFO:tasks.workunit.client.0.smithi121.stdout:0/751: dwrite d7/d71/d29/de8/ffe [0,4194304] 0 2024-04-05T21:26:48.195 INFO:tasks.workunit.client.1.smithi188.stdout:5/961: fdatasync d6/d15/d5c/d7a/d44/d51/dc6/faa 0 2024-04-05T21:26:48.195 INFO:tasks.workunit.client.1.smithi188.stdout:8/717: mknod d5/d6e/d86/cec 0 2024-04-05T21:26:48.196 INFO:tasks.workunit.client.1.smithi188.stdout:9/960: read - dd/d3d/d95/f113 zero size 2024-04-05T21:26:48.196 INFO:tasks.workunit.client.0.smithi121.stdout:0/752: dread - d7/d71/d94/dd1/dd4/ffb zero size 2024-04-05T21:26:48.197 INFO:tasks.workunit.client.0.smithi121.stdout:4/785: rename d5/d11/d2c/d9d/dae/c7a to d5/cfa 0 2024-04-05T21:26:48.198 INFO:tasks.workunit.client.1.smithi188.stdout:6/976: dwrite d6/f2e [0,4194304] 0 2024-04-05T21:26:48.205 INFO:tasks.workunit.client.0.smithi121.stdout:4/786: mknod d5/d11/d2c/d9d/dae/d29/db4/cfb 0 2024-04-05T21:26:48.206 INFO:tasks.workunit.client.0.smithi121.stdout:3/563: dwrite d0/f27 [0,4194304] 0 2024-04-05T21:26:48.209 INFO:tasks.workunit.client.0.smithi121.stdout:3/564: creat d0/d3/de/da6/fb4 x:0 0 0 2024-04-05T21:26:48.211 INFO:tasks.workunit.client.0.smithi121.stdout:4/787: chown d5/d45/d8b 755330203 1 2024-04-05T21:26:48.212 INFO:tasks.workunit.client.0.smithi121.stdout:3/565: mknod d0/d3/d24/d3d/d1a/d23/d38/d55/cb5 0 2024-04-05T21:26:48.212 INFO:tasks.workunit.client.1.smithi188.stdout:8/718: creat d5/d21/d2a/d71/fed x:0 0 0 2024-04-05T21:26:48.212 INFO:tasks.workunit.client.0.smithi121.stdout:4/788: readlink d5/d45/l5c 0 2024-04-05T21:26:48.213 INFO:tasks.workunit.client.1.smithi188.stdout:9/961: getdents dd/d11/d1f/d25/d8a/d101 0 2024-04-05T21:26:48.214 INFO:tasks.workunit.client.0.smithi121.stdout:9/521: sync 2024-04-05T21:26:48.215 INFO:tasks.workunit.client.0.smithi121.stdout:9/522: stat d6/d36/d7b/d49/f8b 0 2024-04-05T21:26:48.215 INFO:tasks.workunit.client.0.smithi121.stdout:4/789: getdents d5/d11/d28/dc0/de4 0 2024-04-05T21:26:48.217 INFO:tasks.workunit.client.1.smithi188.stdout:5/962: dwrite d6/d15/d5c/d7a/d44/f10a [0,4194304] 0 2024-04-05T21:26:48.218 INFO:tasks.workunit.client.0.smithi121.stdout:9/523: chown d6/df/c47 373936 1 2024-04-05T21:26:48.218 INFO:tasks.workunit.client.0.smithi121.stdout:3/566: getdents d0/d3/d24/d3d/d1a/d1e/d86/d8e 0 2024-04-05T21:26:48.224 INFO:tasks.workunit.client.1.smithi188.stdout:9/962: truncate dd/d3d/d81/fc9 199134 0 2024-04-05T21:26:48.225 INFO:tasks.workunit.client.1.smithi188.stdout:0/760: rename d4/d6/d13/d23 to d4/d8/db/d21/d4a/d73/d116 0 2024-04-05T21:26:48.225 INFO:tasks.workunit.client.1.smithi188.stdout:5/963: chown d6/l7 56872490 1 2024-04-05T21:26:48.226 INFO:tasks.workunit.client.1.smithi188.stdout:3/963: write d5/d7/d65/d5f/f64 [218119,17865] 0 2024-04-05T21:26:48.228 INFO:tasks.workunit.client.1.smithi188.stdout:5/964: write d6/d15/d5c/fa3 [317463,114548] 0 2024-04-05T21:26:48.229 INFO:tasks.workunit.client.1.smithi188.stdout:0/761: creat d4/d8/d78/d9d/f117 x:0 0 0 2024-04-05T21:26:48.229 INFO:tasks.workunit.client.1.smithi188.stdout:9/963: unlink dd/d11/fb8 0 2024-04-05T21:26:48.231 INFO:tasks.workunit.client.1.smithi188.stdout:0/762: stat d4/d8/db/d42/d71/dc5/fdf 0 2024-04-05T21:26:48.232 INFO:tasks.workunit.client.1.smithi188.stdout:3/964: creat d5/d7e/d101/de5/f13d x:0 0 0 2024-04-05T21:26:48.232 INFO:tasks.workunit.client.1.smithi188.stdout:5/965: truncate d6/d17/f2f 496120 0 2024-04-05T21:26:48.233 INFO:tasks.workunit.client.1.smithi188.stdout:0/763: truncate d4/d8/fb4 1045228 0 2024-04-05T21:26:48.233 INFO:tasks.workunit.client.1.smithi188.stdout:9/964: write dd/d11/d1f/d25/d51/fc2 [1242533,83674] 0 2024-04-05T21:26:48.234 INFO:tasks.workunit.client.1.smithi188.stdout:3/965: mkdir d5/d7/d65/da2/d13e 0 2024-04-05T21:26:48.236 INFO:tasks.workunit.client.1.smithi188.stdout:5/966: rmdir d6/d15/d1b/ddc 39 2024-04-05T21:26:48.237 INFO:tasks.workunit.client.0.smithi121.stdout:4/790: dwrite d5/d11/d28/d3b/f93 [0,4194304] 0 2024-04-05T21:26:48.238 INFO:tasks.workunit.client.1.smithi188.stdout:9/965: mkdir dd/d11/d1f/d26/dc0/d6b/d13e 0 2024-04-05T21:26:48.239 INFO:tasks.workunit.client.0.smithi121.stdout:9/524: link d6/df/l14 d6/d36/d7b/lb5 0 2024-04-05T21:26:48.242 INFO:tasks.workunit.client.0.smithi121.stdout:3/567: rename d0/d3/d24/d3d/d1a/d1e/d86/fa2 to d0/d3/d24/d3d/d1a/d1e/d86/d8e/fb6 0 2024-04-05T21:26:48.242 INFO:tasks.workunit.client.0.smithi121.stdout:9/525: write d6/df/d1d/d2c/d37/d3c/f56 [3759863,15118] 0 2024-04-05T21:26:48.242 INFO:tasks.workunit.client.1.smithi188.stdout:5/967: symlink d6/d15/d1b/l150 0 2024-04-05T21:26:48.244 INFO:tasks.workunit.client.1.smithi188.stdout:9/966: readlink dd/d34/da7/ld6 0 2024-04-05T21:26:48.244 INFO:tasks.workunit.client.1.smithi188.stdout:2/889: truncate d5/db/dd/fd4 2076259 0 2024-04-05T21:26:48.244 INFO:tasks.workunit.client.0.smithi121.stdout:4/791: unlink d5/d45/fb0 0 2024-04-05T21:26:48.245 INFO:tasks.workunit.client.0.smithi121.stdout:0/753: truncate d7/dc/f15 81514 0 2024-04-05T21:26:48.248 INFO:tasks.workunit.client.0.smithi121.stdout:3/568: creat d0/d3/d24/d3d/d1a/d23/d38/fb7 x:0 0 0 2024-04-05T21:26:48.253 INFO:tasks.workunit.client.1.smithi188.stdout:3/966: dwrite d5/d7/d65/d61/d66/fc2 [0,4194304] 0 2024-04-05T21:26:48.253 INFO:tasks.workunit.client.0.smithi121.stdout:4/792: dwrite d5/d11/d2c/d9d/dae/d29/db4/fe6 [0,4194304] 0 2024-04-05T21:26:48.254 INFO:tasks.workunit.client.1.smithi188.stdout:0/764: dwrite d4/d8/db/d42/da2/dd2/f102 [0,4194304] 0 2024-04-05T21:26:48.255 INFO:tasks.workunit.client.1.smithi188.stdout:9/967: write dd/d11/fd2 [5046966,47870] 0 2024-04-05T21:26:48.256 INFO:tasks.workunit.client.0.smithi121.stdout:3/569: creat d0/d3/de/da6/da1/fb8 x:0 0 0 2024-04-05T21:26:48.256 INFO:tasks.workunit.client.0.smithi121.stdout:0/754: rename d7/d62/d8b/fcd to d7/d71/d29/f10a 0 2024-04-05T21:26:48.262 INFO:tasks.workunit.client.1.smithi188.stdout:2/890: creat d5/db/dd/d36/d3c/d48/d7a/d7c/db5/f125 x:0 0 0 2024-04-05T21:26:48.265 INFO:tasks.workunit.client.0.smithi121.stdout:3/570: chown d0/d3/l92 3325 1 2024-04-05T21:26:48.266 INFO:tasks.workunit.client.1.smithi188.stdout:3/967: symlink d5/d7e/d101/l13f 0 2024-04-05T21:26:48.266 INFO:tasks.workunit.client.1.smithi188.stdout:5/968: write d6/d15/d5c/d7a/d44/f4d [1694700,113386] 0 2024-04-05T21:26:48.266 INFO:tasks.workunit.client.1.smithi188.stdout:0/765: fdatasync d4/d8/db/d42/d71/f8c 0 2024-04-05T21:26:48.267 INFO:tasks.workunit.client.1.smithi188.stdout:9/968: mknod dd/d11/d1f/d25/d51/c13f 0 2024-04-05T21:26:48.269 INFO:tasks.workunit.client.0.smithi121.stdout:0/755: readlink d7/d71/d29/d2a/lea 0 2024-04-05T21:26:48.270 INFO:tasks.workunit.client.0.smithi121.stdout:4/793: getdents d5/d45 0 2024-04-05T21:26:48.270 INFO:tasks.workunit.client.1.smithi188.stdout:6/977: dwrite d6/d13/d18/d19/d61/fc6 [0,4194304] 0 2024-04-05T21:26:48.271 INFO:tasks.workunit.client.0.smithi121.stdout:4/794: chown d5/d11/d26/d47/fd7 22 1 2024-04-05T21:26:48.272 INFO:tasks.workunit.client.0.smithi121.stdout:0/756: readlink d7/dc/da3/df5/lf7 0 2024-04-05T21:26:48.272 INFO:tasks.workunit.client.1.smithi188.stdout:2/891: write d5/db/dd/d36/d3c/d63/d75/d87/ff6 [966810,91504] 0 2024-04-05T21:26:48.275 INFO:tasks.workunit.client.1.smithi188.stdout:8/719: dwrite d5/d21/d40/d43/d44/d59/d8b/fb6 [0,4194304] 0 2024-04-05T21:26:48.275 INFO:tasks.workunit.client.1.smithi188.stdout:5/969: chown d6/d106/f4a 115734981 1 2024-04-05T21:26:48.278 INFO:tasks.workunit.client.0.smithi121.stdout:3/571: creat d0/d3/d24/d6d/d7b/da4/fb9 x:0 0 0 2024-04-05T21:26:48.278 INFO:tasks.workunit.client.0.smithi121.stdout:4/795: truncate d5/d11/d2c/d9d/fb5 105262 0 2024-04-05T21:26:48.281 INFO:tasks.workunit.client.0.smithi121.stdout:0/757: mkdir d7/d71/d29/d2a/d10b 0 2024-04-05T21:26:48.282 INFO:tasks.workunit.client.1.smithi188.stdout:2/892: fdatasync d5/d5f/f90 0 2024-04-05T21:26:48.282 INFO:tasks.workunit.client.0.smithi121.stdout:3/572: mknod d0/d3/d24/d3d/d1a/cba 0 2024-04-05T21:26:48.283 INFO:tasks.workunit.client.1.smithi188.stdout:8/720: rename d5/d21/d2a/f39 to d5/d21/d40/d43/d44/d5b/d99/d69/d70/fee 0 2024-04-05T21:26:48.283 INFO:tasks.workunit.client.0.smithi121.stdout:4/796: dread d5/d45/d67/f84 [0,4194304] 0 2024-04-05T21:26:48.285 INFO:tasks.workunit.client.0.smithi121.stdout:3/573: fsync d0/d3/d24/d3d/faf 0 2024-04-05T21:26:48.285 INFO:tasks.workunit.client.0.smithi121.stdout:0/758: symlink d7/d71/d45/d46/d106/l10c 0 2024-04-05T21:26:48.286 INFO:tasks.workunit.client.0.smithi121.stdout:0/759: readlink d7/d71/d94/dd1/lfa 0 2024-04-05T21:26:48.287 INFO:tasks.workunit.client.0.smithi121.stdout:4/797: unlink d5/d11/d28/d3b/fef 0 2024-04-05T21:26:48.287 INFO:tasks.workunit.client.1.smithi188.stdout:9/969: rename dd/d11/d1f/f121 to dd/d11/d1f/d26/d55/d98/da5/dea/f140 0 2024-04-05T21:26:48.288 INFO:tasks.workunit.client.0.smithi121.stdout:9/526: dread d6/d46/fac [0,4194304] 0 2024-04-05T21:26:48.289 INFO:tasks.workunit.client.1.smithi188.stdout:3/968: read d5/d7/f30 [394450,79787] 0 2024-04-05T21:26:48.289 INFO:tasks.workunit.client.1.smithi188.stdout:2/893: mknod d5/db/dd/d73/d109/d9f/d10c/dcd/c126 0 2024-04-05T21:26:48.292 INFO:tasks.workunit.client.0.smithi121.stdout:0/760: creat d7/d62/d9d/da6/f10d x:0 0 0 2024-04-05T21:26:48.292 INFO:tasks.workunit.client.1.smithi188.stdout:6/978: dread d6/d13/d18/d19/d61/f8b [0,4194304] 0 2024-04-05T21:26:48.293 INFO:tasks.workunit.client.1.smithi188.stdout:0/766: dread d4/d8/db/d21/f34 [4194304,4194304] 0 2024-04-05T21:26:48.295 INFO:tasks.workunit.client.0.smithi121.stdout:0/761: write d7/dc/da3/fc1 [133033,40079] 0 2024-04-05T21:26:48.295 INFO:tasks.workunit.client.1.smithi188.stdout:6/979: readlink d6/d13/d18/d3c/lbb 0 2024-04-05T21:26:48.296 INFO:tasks.workunit.client.0.smithi121.stdout:4/798: dwrite d5/f8 [0,4194304] 0 2024-04-05T21:26:48.297 INFO:tasks.workunit.client.0.smithi121.stdout:9/527: dread d6/df/d1d/f2d [0,4194304] 0 2024-04-05T21:26:48.300 INFO:tasks.workunit.client.1.smithi188.stdout:8/721: dwrite d5/d21/d40/d43/d48/fde [0,4194304] 0 2024-04-05T21:26:48.300 INFO:tasks.workunit.client.1.smithi188.stdout:6/980: chown d6/d13/db9/ff9 3 1 2024-04-05T21:26:48.301 INFO:tasks.workunit.client.1.smithi188.stdout:0/767: symlink d4/d8/d10/l118 0 2024-04-05T21:26:48.303 INFO:tasks.workunit.client.0.smithi121.stdout:9/528: creat d6/d36/d7b/d49/fb6 x:0 0 0 2024-04-05T21:26:48.304 INFO:tasks.workunit.client.0.smithi121.stdout:0/762: dread d7/d71/d45/d46/d53/d8e/fbf [0,4194304] 0 2024-04-05T21:26:48.304 INFO:tasks.workunit.client.1.smithi188.stdout:3/969: dwrite d5/d7/d65/d61/db5/ff0 [0,4194304] 0 2024-04-05T21:26:48.304 INFO:tasks.workunit.client.1.smithi188.stdout:0/768: read d4/d8/fb4 [405181,75010] 0 2024-04-05T21:26:48.305 INFO:tasks.workunit.client.0.smithi121.stdout:9/529: chown d6/df/f27 37972195 1 2024-04-05T21:26:48.306 INFO:tasks.workunit.client.0.smithi121.stdout:0/763: write d7/d7d/f81 [963223,111332] 0 2024-04-05T21:26:48.306 INFO:tasks.workunit.client.1.smithi188.stdout:9/970: link dd/d11/d1f/d26/dc0/f53 dd/d11/d1f/d26/dc0/d6b/d11e/f141 0 2024-04-05T21:26:48.307 INFO:tasks.workunit.client.1.smithi188.stdout:9/971: write dd/d35/f67 [1176824,101853] 0 2024-04-05T21:26:48.312 INFO:tasks.workunit.client.1.smithi188.stdout:3/970: dread - d5/d7e/d101/fc5 zero size 2024-04-05T21:26:48.315 INFO:tasks.workunit.client.0.smithi121.stdout:0/764: stat d7/d71/d45/d46/d53/l9a 0 2024-04-05T21:26:48.317 INFO:tasks.workunit.client.1.smithi188.stdout:9/972: creat dd/d11/d73/dbb/f142 x:0 0 0 2024-04-05T21:26:48.317 INFO:tasks.workunit.client.0.smithi121.stdout:4/799: link d5/d45/fb1 d5/d11/d26/ffc 0 2024-04-05T21:26:48.317 INFO:tasks.workunit.client.0.smithi121.stdout:0/765: stat d7/dc/da3 0 2024-04-05T21:26:48.317 INFO:tasks.workunit.client.1.smithi188.stdout:9/973: stat dd/d11/d1f/d26/dc0/d6b/ff4 0 2024-04-05T21:26:48.322 INFO:tasks.workunit.client.0.smithi121.stdout:4/800: fdatasync d5/d11/d2c/d9d/f31 0 2024-04-05T21:26:48.322 INFO:tasks.workunit.client.0.smithi121.stdout:0/766: mkdir d7/d71/d45/d46/d53/d10e 0 2024-04-05T21:26:48.322 INFO:tasks.workunit.client.0.smithi121.stdout:4/801: stat d5/d11/d60/f9f 0 2024-04-05T21:26:48.324 INFO:tasks.workunit.client.0.smithi121.stdout:3/574: dwrite d0/d3/d24/d3f/f80 [0,4194304] 0 2024-04-05T21:26:48.324 INFO:tasks.workunit.client.0.smithi121.stdout:0/767: dread - d7/d71/d94/da0/fdf zero size 2024-04-05T21:26:48.326 INFO:tasks.workunit.client.1.smithi188.stdout:3/971: mkdir d5/d7/d65/d61/d66/da4/de0/d140 0 2024-04-05T21:26:48.327 INFO:tasks.workunit.client.0.smithi121.stdout:0/768: mkdir d7/d10f 0 2024-04-05T21:26:48.328 INFO:tasks.workunit.client.1.smithi188.stdout:0/769: dread d4/d6/d13/f14 [4194304,4194304] 0 2024-04-05T21:26:48.328 INFO:tasks.workunit.client.0.smithi121.stdout:4/802: read d5/d11/d2c/d9d/dae/d29/d2e/dd3/fd0 [3036677,32148] 0 2024-04-05T21:26:48.329 INFO:tasks.workunit.client.1.smithi188.stdout:9/974: creat dd/d11/d1f/d25/f143 x:0 0 0 2024-04-05T21:26:48.329 INFO:tasks.workunit.client.1.smithi188.stdout:8/722: creat d5/d21/d40/d43/d44/d5b/fef x:0 0 0 2024-04-05T21:26:48.329 INFO:tasks.workunit.client.1.smithi188.stdout:6/981: link d6/d13/db9/cd9 d6/d13/c130 0 2024-04-05T21:26:48.331 INFO:tasks.workunit.client.0.smithi121.stdout:0/769: rename d7/dc/f2d to d7/d62/d9d/da6/dce/f110 0 2024-04-05T21:26:48.332 INFO:tasks.workunit.client.1.smithi188.stdout:9/975: readlink dd/d11/d1f/d26/dc0/d6b/l112 0 2024-04-05T21:26:48.332 INFO:tasks.workunit.client.0.smithi121.stdout:4/803: mkdir d5/d89/dfd 0 2024-04-05T21:26:48.333 INFO:tasks.workunit.client.1.smithi188.stdout:3/972: mknod d5/d7/d65/d61/d66/da4/dfd/c141 0 2024-04-05T21:26:48.336 INFO:tasks.workunit.client.0.smithi121.stdout:4/804: mknod d5/d11/d60/df4/cfe 0 2024-04-05T21:26:48.339 INFO:tasks.workunit.client.0.smithi121.stdout:4/805: symlink d5/d45/d67/dab/lff 0 2024-04-05T21:26:48.339 INFO:tasks.workunit.client.0.smithi121.stdout:9/530: dread d6/df/d1d/d2c/d37/d3c/f56 [0,4194304] 0 2024-04-05T21:26:48.339 INFO:tasks.workunit.client.0.smithi121.stdout:0/770: dread d7/d71/d29/d2a/d2c/f7b [0,4194304] 0 2024-04-05T21:26:48.340 INFO:tasks.workunit.client.1.smithi188.stdout:8/723: truncate d5/d6e/dad/fb9 529686 0 2024-04-05T21:26:48.341 INFO:tasks.workunit.client.0.smithi121.stdout:0/771: fsync d7/d71/d45/d46/d53/d8e/dc4/ff4 0 2024-04-05T21:26:48.341 INFO:tasks.workunit.client.0.smithi121.stdout:4/806: symlink d5/d11/d80/l100 0 2024-04-05T21:26:48.341 INFO:tasks.workunit.client.0.smithi121.stdout:0/772: readlink d7/dc/l4a 0 2024-04-05T21:26:48.343 INFO:tasks.workunit.client.0.smithi121.stdout:9/531: unlink d6/d36/d7b/c69 0 2024-04-05T21:26:48.343 INFO:tasks.workunit.client.0.smithi121.stdout:4/807: truncate d5/d11/d26/d47/fd7 768322 0 2024-04-05T21:26:48.344 INFO:tasks.workunit.client.0.smithi121.stdout:0/773: creat d7/d62/d3c/d6b/f111 x:0 0 0 2024-04-05T21:26:48.346 INFO:tasks.workunit.client.1.smithi188.stdout:0/770: dwrite d4/d8/d39/f109 [0,4194304] 0 2024-04-05T21:26:48.347 INFO:tasks.workunit.client.0.smithi121.stdout:4/808: creat d5/d11/d28/d3b/f101 x:0 0 0 2024-04-05T21:26:48.347 INFO:tasks.workunit.client.1.smithi188.stdout:6/982: dwrite d6/d13/d18/d19/d2c/d3f/d95/f10f [0,4194304] 0 2024-04-05T21:26:48.350 INFO:tasks.workunit.client.1.smithi188.stdout:9/976: dwrite dd/d11/d1f/d25/d51/fc7 [4194304,4194304] 0 2024-04-05T21:26:48.351 INFO:tasks.workunit.client.0.smithi121.stdout:4/809: mknod d5/c102 0 2024-04-05T21:26:48.352 INFO:tasks.workunit.client.0.smithi121.stdout:9/532: dwrite d6/d36/d7b/d5d/d79/faf [0,4194304] 0 2024-04-05T21:26:48.353 INFO:tasks.workunit.client.0.smithi121.stdout:0/774: dread d7/d71/d29/de8/ffe [0,4194304] 0 2024-04-05T21:26:48.354 INFO:tasks.workunit.client.1.smithi188.stdout:8/724: creat d5/d21/d2a/d95/dc0/ff0 x:0 0 0 2024-04-05T21:26:48.359 INFO:tasks.workunit.client.1.smithi188.stdout:8/725: dread - d5/dc/d8f/dd1/fe2 zero size 2024-04-05T21:26:48.360 INFO:tasks.workunit.client.0.smithi121.stdout:0/775: rename d7/dc/c2b to d7/dc/d95/c112 0 2024-04-05T21:26:48.360 INFO:tasks.workunit.client.0.smithi121.stdout:9/533: link d6/d32/l80 d6/df/d1d/d2c/d53/lb7 0 2024-04-05T21:26:48.362 INFO:tasks.workunit.client.1.smithi188.stdout:0/771: truncate d4/d6/f15 387751 0 2024-04-05T21:26:48.362 INFO:tasks.workunit.client.1.smithi188.stdout:6/983: chown d6/d13/d18/d19/d2c/d3f/d95/f108 36691085 1 2024-04-05T21:26:48.362 INFO:tasks.workunit.client.1.smithi188.stdout:6/984: stat d6/d13/d18/c40 0 2024-04-05T21:26:48.366 INFO:tasks.workunit.client.1.smithi188.stdout:9/977: write dd/d11/d1f/d26/dc0/d6b/f96 [3004495,24100] 0 2024-04-05T21:26:48.366 INFO:tasks.workunit.client.1.smithi188.stdout:3/973: dwrite d5/f21 [0,4194304] 0 2024-04-05T21:26:48.366 INFO:tasks.workunit.client.0.smithi121.stdout:4/810: dwrite d5/d11/d26/dbc/dc5/dd5/fe1 [0,4194304] 0 2024-04-05T21:26:48.366 INFO:tasks.workunit.client.0.smithi121.stdout:9/534: getdents d6/d46 0 2024-04-05T21:26:48.367 INFO:tasks.workunit.client.1.smithi188.stdout:8/726: chown d5/d21/d40/f57 141 1 2024-04-05T21:26:48.368 INFO:tasks.workunit.client.1.smithi188.stdout:3/974: chown d5/d7/d65/f8c 5467 1 2024-04-05T21:26:48.368 INFO:tasks.workunit.client.1.smithi188.stdout:0/772: chown d4/d8/d10/ca8 2 1 2024-04-05T21:26:48.371 INFO:tasks.workunit.client.1.smithi188.stdout:8/727: write d5/dc/f3b [3303834,126763] 0 2024-04-05T21:26:48.371 INFO:tasks.workunit.client.1.smithi188.stdout:9/978: unlink dd/d11/d1f/d25/cb0 0 2024-04-05T21:26:48.371 INFO:tasks.workunit.client.0.smithi121.stdout:4/811: symlink d5/d89/dfd/l103 0 2024-04-05T21:26:48.374 INFO:tasks.workunit.client.1.smithi188.stdout:0/773: rmdir d4/d6/d13/dea 39 2024-04-05T21:26:48.375 INFO:tasks.workunit.client.0.smithi121.stdout:4/812: fdatasync d5/d11/d2c/d9d/f31 0 2024-04-05T21:26:48.375 INFO:tasks.workunit.client.0.smithi121.stdout:0/776: dwrite d7/d71/d29/d2a/d2c/fc3 [0,4194304] 0 2024-04-05T21:26:48.377 INFO:tasks.workunit.client.1.smithi188.stdout:3/975: mkdir d5/d7/d65/da2/d13e/d142 0 2024-04-05T21:26:48.378 INFO:tasks.workunit.client.1.smithi188.stdout:9/979: creat dd/d3d/d81/f144 x:0 0 0 2024-04-05T21:26:48.379 INFO:tasks.workunit.client.1.smithi188.stdout:6/985: link d6/d13/d18/d19/d61/f8b d6/d13/d18/d19/f131 0 2024-04-05T21:26:48.381 INFO:tasks.workunit.client.1.smithi188.stdout:8/728: mknod d5/d21/d40/d43/d44/d59/d79/cf1 0 2024-04-05T21:26:48.383 INFO:tasks.workunit.client.1.smithi188.stdout:9/980: mkdir dd/d11/d1f/d25/d145 0 2024-04-05T21:26:48.385 INFO:tasks.workunit.client.1.smithi188.stdout:8/729: write d5/d6e/d86/dcb/fd2 [459184,15185] 0 2024-04-05T21:26:48.386 INFO:tasks.workunit.client.1.smithi188.stdout:0/774: getdents d4/d8/db/d21/d4a/d73/d116 0 2024-04-05T21:26:48.386 INFO:tasks.workunit.client.1.smithi188.stdout:8/730: readlink d5/laf 0 2024-04-05T21:26:48.387 INFO:tasks.workunit.client.1.smithi188.stdout:9/981: fsync dd/d11/d73/d85/fcb 0 2024-04-05T21:26:48.387 INFO:tasks.workunit.client.0.smithi121.stdout:0/777: rename d7/d62/d3c/cc9 to d7/d71/d45/d46/d106/c113 0 2024-04-05T21:26:48.388 INFO:tasks.workunit.client.1.smithi188.stdout:6/986: creat d6/d13/d18/d19/dc8/da4/dfa/d120/f132 x:0 0 0 2024-04-05T21:26:48.389 INFO:tasks.workunit.client.1.smithi188.stdout:6/987: dread - d6/d13/d18/d19/da3/ff0 zero size 2024-04-05T21:26:48.390 INFO:tasks.workunit.client.1.smithi188.stdout:0/775: mknod d4/d6/d46/dba/c119 0 2024-04-05T21:26:48.392 INFO:tasks.workunit.client.1.smithi188.stdout:9/982: mkdir dd/d11/d1f/d26/dc0/d102/d11a/d146 0 2024-04-05T21:26:48.393 INFO:tasks.workunit.client.1.smithi188.stdout:6/988: symlink d6/d13/d18/d19/d61/l133 0 2024-04-05T21:26:48.396 INFO:tasks.workunit.client.0.smithi121.stdout:0/778: rmdir d7/d71/d94 39 2024-04-05T21:26:48.397 INFO:tasks.workunit.client.1.smithi188.stdout:6/989: rmdir d6/d13/d48 39 2024-04-05T21:26:48.397 INFO:tasks.workunit.client.1.smithi188.stdout:6/990: chown d6/d28/l112 2145 1 2024-04-05T21:26:48.398 INFO:tasks.workunit.client.1.smithi188.stdout:9/983: mknod dd/d11/d1f/d26/dc0/d6b/d11e/c147 0 2024-04-05T21:26:48.399 INFO:tasks.workunit.client.1.smithi188.stdout:9/984: chown dd/d11/d1f/d26/dc0/d6b/d11e 5985 1 2024-04-05T21:26:48.400 INFO:tasks.workunit.client.0.smithi121.stdout:9/535: truncate d6/d36/d7b/d5d/d79/faf 1184437 0 2024-04-05T21:26:48.401 INFO:tasks.workunit.client.1.smithi188.stdout:6/991: symlink d6/d13/d18/d19/dc8/da4/l134 0 2024-04-05T21:26:48.402 INFO:tasks.workunit.client.0.smithi121.stdout:4/813: write d5/d11/d2c/d9d/dae/d29/d58/f6d [719537,30741] 0 2024-04-05T21:26:48.404 INFO:tasks.workunit.client.1.smithi188.stdout:9/985: creat dd/d11/d1f/d26/dc0/d6b/d11e/f148 x:0 0 0 2024-04-05T21:26:48.405 INFO:tasks.workunit.client.0.smithi121.stdout:0/779: mknod d7/d62/d8b/c114 0 2024-04-05T21:26:48.406 INFO:tasks.workunit.client.0.smithi121.stdout:9/536: mknod d6/df/d1d/d2c/d53/cb8 0 2024-04-05T21:26:48.407 INFO:tasks.workunit.client.0.smithi121.stdout:4/814: mknod d5/d11/d26/dbc/dca/de9/c104 0 2024-04-05T21:26:48.407 INFO:tasks.workunit.client.0.smithi121.stdout:0/780: dread d7/f3a [0,4194304] 0 2024-04-05T21:26:48.408 INFO:tasks.workunit.client.1.smithi188.stdout:3/976: write d5/d7/f20 [408237,94071] 0 2024-04-05T21:26:48.409 INFO:tasks.workunit.client.0.smithi121.stdout:4/815: creat d5/d89/f105 x:0 0 0 2024-04-05T21:26:48.409 INFO:tasks.workunit.client.0.smithi121.stdout:4/816: stat d5/d11/d26/dbc/dc5 0 2024-04-05T21:26:48.410 INFO:tasks.workunit.client.0.smithi121.stdout:0/781: read d7/d62/f5c [49174,37812] 0 2024-04-05T21:26:48.410 INFO:tasks.workunit.client.1.smithi188.stdout:0/776: write d4/d8/fb4 [383586,75479] 0 2024-04-05T21:26:48.411 INFO:tasks.workunit.client.0.smithi121.stdout:4/817: truncate d5/d89/fc9 1033291 0 2024-04-05T21:26:48.411 INFO:tasks.workunit.client.0.smithi121.stdout:0/782: chown d7/d62/ddb/ce9 31 1 2024-04-05T21:26:48.413 INFO:tasks.workunit.client.0.smithi121.stdout:4/818: mknod d5/d11/d26/dbc/dca/c106 0 2024-04-05T21:26:48.414 INFO:tasks.workunit.client.0.smithi121.stdout:0/783: mknod d7/d71/d94/dd1/c115 0 2024-04-05T21:26:48.415 INFO:tasks.workunit.client.0.smithi121.stdout:0/784: chown d7/d62/d9d/lee 2206 1 2024-04-05T21:26:48.415 INFO:tasks.workunit.client.1.smithi188.stdout:3/977: rename d5/d7/d65/d61/d66/da4/dfd/f12c to d5/d35/d128/f143 0 2024-04-05T21:26:48.416 INFO:tasks.workunit.client.1.smithi188.stdout:9/986: dwrite dd/d11/d1f/d26/d55/d98/d133/f13c [0,4194304] 0 2024-04-05T21:26:48.419 INFO:tasks.workunit.client.1.smithi188.stdout:0/777: rename d4/d8/db/d42/d71 to d4/d8/d78/dc2/d11a 0 2024-04-05T21:26:48.421 INFO:tasks.workunit.client.0.smithi121.stdout:0/785: dwrite d7/d71/d29/d3f/fa1 [0,4194304] 0 2024-04-05T21:26:48.424 INFO:tasks.workunit.client.1.smithi188.stdout:3/978: rename d5/d7/d6d/d75/d83/l11a to d5/d7/d65/d5f/d102/l144 0 2024-04-05T21:26:48.426 INFO:tasks.workunit.client.0.smithi121.stdout:9/537: rename d6/df/d1d to d6/df/d48/d96/db9 0 2024-04-05T21:26:48.427 INFO:tasks.workunit.client.0.smithi121.stdout:6/524: sync 2024-04-05T21:26:48.427 INFO:tasks.workunit.client.0.smithi121.stdout:1/756: sync 2024-04-05T21:26:48.427 INFO:tasks.workunit.client.0.smithi121.stdout:8/660: sync 2024-04-05T21:26:48.428 INFO:tasks.workunit.client.0.smithi121.stdout:7/729: sync 2024-04-05T21:26:48.428 INFO:tasks.workunit.client.0.smithi121.stdout:2/637: sync 2024-04-05T21:26:48.428 INFO:tasks.workunit.client.0.smithi121.stdout:9/538: stat d6/d36/d7b/d5d/d79 0 2024-04-05T21:26:48.429 INFO:tasks.workunit.client.0.smithi121.stdout:0/786: creat d7/d62/d8b/f116 x:0 0 0 2024-04-05T21:26:48.430 INFO:tasks.workunit.client.1.smithi188.stdout:3/979: mkdir d5/d7/d65/d139/d145 0 2024-04-05T21:26:48.431 INFO:tasks.workunit.client.0.smithi121.stdout:7/730: fdatasync d0/d23/f33 0 2024-04-05T21:26:48.434 INFO:tasks.workunit.client.0.smithi121.stdout:4/819: truncate d5/d11/d28/f1b 6228390 0 2024-04-05T21:26:48.434 INFO:tasks.workunit.client.0.smithi121.stdout:9/539: fsync d6/df/d48/d96/db9/d2c/d37/f86 0 2024-04-05T21:26:48.434 INFO:tasks.workunit.client.1.smithi188.stdout:0/778: rename d4/d8/d78/dc2/d11a/dc5/fdf to d4/d6/f11b 0 2024-04-05T21:26:48.436 INFO:tasks.workunit.client.0.smithi121.stdout:0/787: write d7/d71/d94/dd1/dd4/ffb [987949,117002] 0 2024-04-05T21:26:48.436 INFO:tasks.workunit.client.0.smithi121.stdout:1/757: stat d8/d8c/db6/d1a/d1b/dbb/cbe 0 2024-04-05T21:26:48.436 INFO:tasks.workunit.client.0.smithi121.stdout:6/525: chown d1/d2/d1b/d2e/d53/faa 1 1 2024-04-05T21:26:48.436 INFO:tasks.workunit.client.0.smithi121.stdout:7/731: creat d0/d23/d5b/d52/fe5 x:0 0 0 2024-04-05T21:26:48.437 INFO:tasks.workunit.client.0.smithi121.stdout:4/820: chown d5/d11/d2c/f36 127550396 1 2024-04-05T21:26:48.438 INFO:tasks.workunit.client.1.smithi188.stdout:0/779: rename d4/d8/db/d21/l4e to d4/d8/d78/l11c 0 2024-04-05T21:26:48.440 INFO:tasks.workunit.client.0.smithi121.stdout:2/638: creat d2/d9/d56/d62/fc6 x:0 0 0 2024-04-05T21:26:48.442 INFO:tasks.workunit.client.0.smithi121.stdout:1/758: read d8/d89/fa5 [2567350,11119] 0 2024-04-05T21:26:48.443 INFO:tasks.workunit.client.1.smithi188.stdout:0/780: mknod d4/d6/d46/dba/c11d 0 2024-04-05T21:26:48.446 INFO:tasks.workunit.client.0.smithi121.stdout:4/821: write d5/d11/d2c/d9d/dae/d29/db4/fe6 [972356,47300] 0 2024-04-05T21:26:48.448 INFO:tasks.workunit.client.0.smithi121.stdout:8/661: dread d12/d50/d6a/d63/f64 [0,4194304] 0 2024-04-05T21:26:48.448 INFO:tasks.workunit.client.1.smithi188.stdout:0/781: unlink d4/d8/db/d21/f34 0 2024-04-05T21:26:48.449 INFO:tasks.workunit.client.0.smithi121.stdout:6/526: rename d1/d2/d1b/d35/d39/c73 to d1/d2/d1b/d35/d39/cb4 0 2024-04-05T21:26:48.451 INFO:tasks.workunit.client.1.smithi188.stdout:9/987: dwrite dd/d3d/d89/dd1/f114 [0,4194304] 0 2024-04-05T21:26:48.452 INFO:tasks.workunit.client.0.smithi121.stdout:2/639: readlink d2/d9/d56/d8d/la0 0 2024-04-05T21:26:48.452 INFO:tasks.workunit.client.1.smithi188.stdout:0/782: read - d4/d6/fc7 zero size 2024-04-05T21:26:48.455 INFO:tasks.workunit.client.1.smithi188.stdout:9/988: chown dd/d3d/c11f 48634069 1 2024-04-05T21:26:48.456 INFO:tasks.workunit.client.0.smithi121.stdout:1/759: read d8/d8c/db6/f37 [292616,25583] 0 2024-04-05T21:26:48.456 INFO:tasks.workunit.client.1.smithi188.stdout:0/783: creat d4/d8/db/d21/d4a/d73/f11e x:0 0 0 2024-04-05T21:26:48.459 INFO:tasks.workunit.client.0.smithi121.stdout:6/527: write d1/d2/d3/d4/d21/f84 [2370307,68834] 0 2024-04-05T21:26:48.460 INFO:tasks.workunit.client.1.smithi188.stdout:0/784: symlink d4/d8/db/d21/d3c/dee/l11f 0 2024-04-05T21:26:48.462 INFO:tasks.workunit.client.0.smithi121.stdout:8/662: unlink fa 0 2024-04-05T21:26:48.462 INFO:tasks.workunit.client.0.smithi121.stdout:0/788: dwrite d7/d71/d94/dd1/dd4/ffb [0,4194304] 0 2024-04-05T21:26:48.462 INFO:tasks.workunit.client.0.smithi121.stdout:4/822: fdatasync d5/d11/d26/ffc 0 2024-04-05T21:26:48.462 INFO:tasks.workunit.client.0.smithi121.stdout:6/528: write d1/d2/d3/f4e [2021440,56434] 0 2024-04-05T21:26:48.463 INFO:tasks.workunit.client.1.smithi188.stdout:0/785: creat d4/d8/db/d42/da2/dd2/f120 x:0 0 0 2024-04-05T21:26:48.463 INFO:tasks.workunit.client.0.smithi121.stdout:9/540: write d6/d36/d7b/d5d/d79/f95 [392700,65872] 0 2024-04-05T21:26:48.466 INFO:tasks.workunit.client.0.smithi121.stdout:7/732: dread d0/d9/d22/d28/d56/f58 [0,4194304] 0 2024-04-05T21:26:48.469 INFO:tasks.workunit.client.0.smithi121.stdout:9/541: stat d6/df/d50 0 2024-04-05T21:26:48.469 INFO:tasks.workunit.client.0.smithi121.stdout:1/760: mkdir d8/d8c/db6/d1a/d1b/d85/dfe 0 2024-04-05T21:26:48.470 INFO:tasks.workunit.client.0.smithi121.stdout:0/789: truncate d7/f18 7932041 0 2024-04-05T21:26:48.472 INFO:tasks.workunit.client.0.smithi121.stdout:8/663: creat d12/d13/d56/d57/d5e/db9/fd0 x:0 0 0 2024-04-05T21:26:48.473 INFO:tasks.workunit.client.1.smithi188.stdout:0/786: dwrite d4/d8/db/d21/d4a/d73/d116/de8/f107 [0,4194304] 0 2024-04-05T21:26:48.474 INFO:tasks.workunit.client.0.smithi121.stdout:4/823: creat d5/d45/f107 x:0 0 0 2024-04-05T21:26:48.477 INFO:tasks.workunit.client.0.smithi121.stdout:6/529: mknod d1/d2/d3/cb5 0 2024-04-05T21:26:48.478 INFO:tasks.workunit.client.1.smithi188.stdout:0/787: creat d4/d8/db/d21/d4a/d73/d116/d24/ddc/f121 x:0 0 0 2024-04-05T21:26:48.479 INFO:tasks.workunit.client.0.smithi121.stdout:1/761: dread - d8/d8c/db6/d1a/d1b/d44/fd5 zero size 2024-04-05T21:26:48.481 INFO:tasks.workunit.client.1.smithi188.stdout:0/788: symlink d4/d8/d78/dc2/d11a/dc5/l122 0 2024-04-05T21:26:48.481 INFO:tasks.workunit.client.0.smithi121.stdout:8/664: mkdir d12/d50/d6a/dd1 0 2024-04-05T21:26:48.483 INFO:tasks.workunit.client.0.smithi121.stdout:4/824: mknod d5/d11/d28/d3b/c108 0 2024-04-05T21:26:48.484 INFO:tasks.workunit.client.1.smithi188.stdout:0/789: unlink d4/d6/d46/ld8 0 2024-04-05T21:26:48.485 INFO:tasks.workunit.client.1.smithi188.stdout:0/790: symlink d4/d6/d46/dba/l123 0 2024-04-05T21:26:48.486 INFO:tasks.workunit.client.0.smithi121.stdout:9/542: dwrite d6/f9b [0,4194304] 0 2024-04-05T21:26:48.486 INFO:tasks.workunit.client.0.smithi121.stdout:6/530: unlink d1/d32/c70 0 2024-04-05T21:26:48.488 INFO:tasks.workunit.client.0.smithi121.stdout:0/790: dwrite d7/dc/f32 [0,4194304] 0 2024-04-05T21:26:48.488 INFO:tasks.workunit.client.0.smithi121.stdout:7/733: link d0/d23/cb5 d0/d9/d22/d3e/d9c/da7/dd9/ce6 0 2024-04-05T21:26:48.488 INFO:tasks.workunit.client.1.smithi188.stdout:0/791: getdents d4/d8/d78/dc2/d11a 0 2024-04-05T21:26:48.488 INFO:tasks.workunit.client.0.smithi121.stdout:4/825: write d5/d11/f12 [8327067,88206] 0 2024-04-05T21:26:48.490 INFO:tasks.workunit.client.0.smithi121.stdout:8/665: read d12/d50/d93/d96/fb2 [2281754,30220] 0 2024-04-05T21:26:48.490 INFO:tasks.workunit.client.0.smithi121.stdout:0/791: chown d7/d71/d29/f79 128345973 1 2024-04-05T21:26:48.491 INFO:tasks.workunit.client.1.smithi188.stdout:0/792: creat d4/d6/d13/da1/f124 x:0 0 0 2024-04-05T21:26:48.492 INFO:tasks.workunit.client.0.smithi121.stdout:9/543: chown d6/d36/l31 46276951 1 2024-04-05T21:26:48.492 INFO:tasks.workunit.client.0.smithi121.stdout:6/531: creat d1/d2/dc/d9a/fb6 x:0 0 0 2024-04-05T21:26:48.493 INFO:tasks.workunit.client.0.smithi121.stdout:7/734: mkdir d0/d57/de7 0 2024-04-05T21:26:48.494 INFO:tasks.workunit.client.1.smithi188.stdout:0/793: mkdir d4/d6/d13/dca/d125 0 2024-04-05T21:26:48.494 INFO:tasks.workunit.client.0.smithi121.stdout:0/792: mkdir d7/d62/ddb/d117 0 2024-04-05T21:26:48.496 INFO:tasks.workunit.client.0.smithi121.stdout:4/826: chown d5/d11/d2c/d9d/dae/d29/d58/f6e 0 1 2024-04-05T21:26:48.496 INFO:tasks.workunit.client.0.smithi121.stdout:9/544: mkdir d6/d36/dba 0 2024-04-05T21:26:48.497 INFO:tasks.workunit.client.0.smithi121.stdout:6/532: write d1/d32/fae [2971074,9232] 0 2024-04-05T21:26:48.497 INFO:tasks.workunit.client.1.smithi188.stdout:0/794: mkdir d4/d8/db/d21/d4a/d73/d116/dd1/d126 0 2024-04-05T21:26:48.500 INFO:tasks.workunit.client.0.smithi121.stdout:6/533: read - d1/d2/d1b/da1/fa6 zero size 2024-04-05T21:26:48.500 INFO:tasks.workunit.client.0.smithi121.stdout:8/666: write d12/d13/d56/d57/da4/dc9/fcb [236487,10150] 0 2024-04-05T21:26:48.503 INFO:tasks.workunit.client.1.smithi188.stdout:0/795: rename d4/d8/db/d42/d38/f72 to d4/d8/db/d42/d38/f127 0 2024-04-05T21:26:48.504 INFO:tasks.workunit.client.0.smithi121.stdout:6/534: fsync d1/d2/d3/f4e 0 2024-04-05T21:26:48.504 INFO:tasks.workunit.client.0.smithi121.stdout:8/667: write d12/d13/d56/d57/da4/f31 [229591,75200] 0 2024-04-05T21:26:48.504 INFO:tasks.workunit.client.1.smithi188.stdout:0/796: creat d4/d6/d13/da1/f128 x:0 0 0 2024-04-05T21:26:48.505 INFO:tasks.workunit.client.0.smithi121.stdout:9/545: mkdir d6/df/d48/d96/db9/d2c/d53/dbb 0 2024-04-05T21:26:48.505 INFO:tasks.workunit.client.1.smithi188.stdout:0/797: chown d4/d8/db/d42/da2/ce4 483306144 1 2024-04-05T21:26:48.506 INFO:tasks.workunit.client.0.smithi121.stdout:8/668: chown d12/d50/d93/db5/dc7 9512575 1 2024-04-05T21:26:48.506 INFO:tasks.workunit.client.1.smithi188.stdout:0/798: stat d4/d6/d46/dba/fdb 0 2024-04-05T21:26:48.506 INFO:tasks.workunit.client.1.smithi188.stdout:0/799: read - d4/d8/d10/f10e zero size 2024-04-05T21:26:48.508 INFO:tasks.workunit.client.0.smithi121.stdout:7/735: read d0/d23/f2e [2256022,15415] 0 2024-04-05T21:26:48.511 INFO:tasks.workunit.client.0.smithi121.stdout:9/546: chown d6/df/d48/d96/db9/f2b 462706847 1 2024-04-05T21:26:48.511 INFO:tasks.workunit.client.0.smithi121.stdout:6/535: creat d1/d2/d3/d5/fb7 x:0 0 0 2024-04-05T21:26:48.513 INFO:tasks.workunit.client.0.smithi121.stdout:4/827: chown d5/d11/d2c/d9d/dae/d29/d2e/cf3 26 1 2024-04-05T21:26:48.514 INFO:tasks.workunit.client.0.smithi121.stdout:0/793: getdents d7/dc/da3 0 2024-04-05T21:26:48.515 INFO:tasks.workunit.client.0.smithi121.stdout:9/547: stat d6/d32/l41 0 2024-04-05T21:26:48.517 INFO:tasks.workunit.client.0.smithi121.stdout:8/669: dwrite ff [0,4194304] 0 2024-04-05T21:26:48.518 INFO:tasks.workunit.client.0.smithi121.stdout:7/736: dread d0/d23/d5b/d52/d61/f89 [0,4194304] 0 2024-04-05T21:26:48.520 INFO:tasks.workunit.client.0.smithi121.stdout:0/794: creat d7/d71/d45/d46/d106/f118 x:0 0 0 2024-04-05T21:26:48.521 INFO:tasks.workunit.client.0.smithi121.stdout:4/828: rmdir d5/d11/d28 39 2024-04-05T21:26:48.522 INFO:tasks.workunit.client.1.smithi188.stdout:0/800: dread d4/d8/db/d21/f7e [0,4194304] 0 2024-04-05T21:26:48.523 INFO:tasks.workunit.client.0.smithi121.stdout:8/670: truncate d12/d13/d56/d57/f74 488795 0 2024-04-05T21:26:48.524 INFO:tasks.workunit.client.0.smithi121.stdout:9/548: rmdir d6/d46/d8a 39 2024-04-05T21:26:48.524 INFO:tasks.workunit.client.1.smithi188.stdout:0/801: creat d4/d6/d13/dca/d125/f129 x:0 0 0 2024-04-05T21:26:48.526 INFO:tasks.workunit.client.0.smithi121.stdout:6/536: dwrite d1/d2/d3/d4/d21/f93 [0,4194304] 0 2024-04-05T21:26:48.528 INFO:tasks.workunit.client.1.smithi188.stdout:0/802: read d4/d8/db/fe0 [3416177,5705] 0 2024-04-05T21:26:48.529 INFO:tasks.workunit.client.1.smithi188.stdout:0/803: dread - d4/d8/db/d21/d4a/f110 zero size 2024-04-05T21:26:48.529 INFO:tasks.workunit.client.0.smithi121.stdout:7/737: rmdir d0/d23 39 2024-04-05T21:26:48.530 INFO:tasks.workunit.client.1.smithi188.stdout:0/804: chown d4/d8/d10/f1b 10357 1 2024-04-05T21:26:48.534 INFO:tasks.workunit.client.0.smithi121.stdout:0/795: dwrite d7/d62/d3c/d6b/f111 [0,4194304] 0 2024-04-05T21:26:48.536 INFO:tasks.workunit.client.0.smithi121.stdout:8/671: write d12/d50/d93/d96/f92 [2653087,99851] 0 2024-04-05T21:26:48.541 INFO:tasks.workunit.client.0.smithi121.stdout:9/549: dwrite d6/f15 [4194304,4194304] 0 2024-04-05T21:26:48.541 INFO:tasks.workunit.client.1.smithi188.stdout:0/805: dwrite d4/d8/db/d21/d4a/d73/d116/d24/d57/fec [0,4194304] 0 2024-04-05T21:26:48.543 INFO:tasks.workunit.client.1.smithi188.stdout:0/806: stat d4/d8/db/d21/d4a/d73/d116/d24/d57 0 2024-04-05T21:26:48.544 INFO:tasks.workunit.client.0.smithi121.stdout:8/672: readlink d12/d50/d93/d96/l9e 0 2024-04-05T21:26:48.546 INFO:tasks.workunit.client.1.smithi188.stdout:0/807: write d4/d8/db/d42/d38/f54 [3234374,98215] 0 2024-04-05T21:26:48.547 INFO:tasks.workunit.client.1.smithi188.stdout:0/808: dread - d4/d8/db/d21/d4a/d73/f11e zero size 2024-04-05T21:26:48.551 INFO:tasks.workunit.client.1.smithi188.stdout:0/809: fsync d4/d8/db/f28 0 2024-04-05T21:26:48.553 INFO:tasks.workunit.client.1.smithi188.stdout:0/810: mkdir d4/d6/d13/dca/d125/d12a 0 2024-04-05T21:26:48.554 INFO:tasks.workunit.client.0.smithi121.stdout:6/537: dwrite d1/d2/f13 [0,4194304] 0 2024-04-05T21:26:48.558 INFO:tasks.workunit.client.0.smithi121.stdout:4/829: rename d5/d11/d28/d3b/c108 to d5/c109 0 2024-04-05T21:26:48.560 INFO:tasks.workunit.client.0.smithi121.stdout:8/673: truncate d12/d13/d15/d1d/f4f 5725353 0 2024-04-05T21:26:48.563 INFO:tasks.workunit.client.0.smithi121.stdout:9/550: link d6/d36/d7b/d6b/la9 d6/df/d48/d96/db9/d2c/d37/d3c/lbc 0 2024-04-05T21:26:48.563 INFO:tasks.workunit.client.0.smithi121.stdout:0/796: read d7/d71/d29/f93 [3113752,36436] 0 2024-04-05T21:26:48.564 INFO:tasks.workunit.client.0.smithi121.stdout:7/738: link d0/d9/d22/f6c d0/d23/fe8 0 2024-04-05T21:26:48.566 INFO:tasks.workunit.client.0.smithi121.stdout:7/739: write d0/d9/d22/d81/da2/fd5 [2698790,23798] 0 2024-04-05T21:26:48.566 INFO:tasks.workunit.client.0.smithi121.stdout:8/674: creat d12/d13/fd2 x:0 0 0 2024-04-05T21:26:48.569 INFO:tasks.workunit.client.0.smithi121.stdout:7/740: readlink d0/d9/d22/d3e/lc8 0 2024-04-05T21:26:48.570 INFO:tasks.workunit.client.0.smithi121.stdout:7/741: fdatasync d0/d9/d22/faf 0 2024-04-05T21:26:48.571 INFO:tasks.workunit.client.0.smithi121.stdout:6/538: write d1/d2/d1b/d35/f62 [5071349,65419] 0 2024-04-05T21:26:48.573 INFO:tasks.workunit.client.0.smithi121.stdout:9/551: dread d6/df/f71 [0,4194304] 0 2024-04-05T21:26:48.574 INFO:tasks.workunit.client.0.smithi121.stdout:6/539: write d1/d2/d3/d4/d21/f93 [1074105,114943] 0 2024-04-05T21:26:48.577 INFO:tasks.workunit.client.0.smithi121.stdout:0/797: getdents d7/d62/d9d/da6 0 2024-04-05T21:26:48.578 INFO:tasks.workunit.client.0.smithi121.stdout:9/552: mknod d6/d36/d7b/d49/cbd 0 2024-04-05T21:26:48.579 INFO:tasks.workunit.client.0.smithi121.stdout:8/675: getdents d12/d50/d93/db5 0 2024-04-05T21:26:48.581 INFO:tasks.workunit.client.0.smithi121.stdout:8/676: write d12/d13/d56/d57/f74 [496721,59207] 0 2024-04-05T21:26:48.582 INFO:tasks.workunit.client.0.smithi121.stdout:0/798: rename d7/d62/cbe to d7/d62/d3c/d6b/c119 0 2024-04-05T21:26:48.582 INFO:tasks.workunit.client.0.smithi121.stdout:8/677: read d12/d50/d93/d96/f92 [3975042,88730] 0 2024-04-05T21:26:48.583 INFO:tasks.workunit.client.0.smithi121.stdout:7/742: truncate d0/d23/f54 340969 0 2024-04-05T21:26:48.587 INFO:tasks.workunit.client.0.smithi121.stdout:0/799: creat d7/d71/d29/de8/f11a x:0 0 0 2024-04-05T21:26:48.587 INFO:tasks.workunit.client.0.smithi121.stdout:8/678: truncate d12/d13/d56/d57/d5e/f62 1082119 0 2024-04-05T21:26:48.588 INFO:tasks.workunit.client.0.smithi121.stdout:6/540: creat d1/d2/d3/fb8 x:0 0 0 2024-04-05T21:26:48.589 INFO:tasks.workunit.client.1.smithi188.stdout:0/811: write d4/d6/faa [3085410,114771] 0 2024-04-05T21:26:48.590 INFO:tasks.workunit.client.0.smithi121.stdout:8/679: unlink l7 0 2024-04-05T21:26:48.594 INFO:tasks.workunit.client.0.smithi121.stdout:0/800: unlink d7/d71/c49 0 2024-04-05T21:26:48.595 INFO:tasks.workunit.client.1.smithi188.stdout:0/812: dwrite d4/d6/d13/da1/f128 [0,4194304] 0 2024-04-05T21:26:48.596 INFO:tasks.workunit.client.0.smithi121.stdout:6/541: read d1/d2/d3/d5/d1f/f64 [161146,108101] 0 2024-04-05T21:26:48.597 INFO:tasks.workunit.client.0.smithi121.stdout:9/553: dwrite d6/df/d48/d96/db9/d2c/d37/d3c/f56 [4194304,4194304] 0 2024-04-05T21:26:48.597 INFO:tasks.workunit.client.0.smithi121.stdout:8/680: write d12/d50/d93/d96/fab [1890932,70913] 0 2024-04-05T21:26:48.598 INFO:tasks.workunit.client.1.smithi188.stdout:0/813: creat d4/d6/df4/f12b x:0 0 0 2024-04-05T21:26:48.601 INFO:tasks.workunit.client.1.smithi188.stdout:0/814: creat d4/d8/db/dac/d112/f12c x:0 0 0 2024-04-05T21:26:48.601 INFO:tasks.workunit.client.0.smithi121.stdout:7/743: dwrite d0/d23/d5b/d52/d61/dcb/fdc [0,4194304] 0 2024-04-05T21:26:48.603 INFO:tasks.workunit.client.0.smithi121.stdout:6/542: stat d1/d2/d3/d5/cb1 0 2024-04-05T21:26:48.603 INFO:tasks.workunit.client.0.smithi121.stdout:0/801: chown d7/d71/d45/d46/c74 25 1 2024-04-05T21:26:48.609 INFO:tasks.workunit.client.0.smithi121.stdout:8/681: chown d12/d50/d6a/d63/d8b/f9b 4374 1 2024-04-05T21:26:48.610 INFO:tasks.workunit.client.0.smithi121.stdout:7/744: write d0/d9/f50 [2354618,96456] 0 2024-04-05T21:26:48.614 INFO:tasks.workunit.client.0.smithi121.stdout:4/830: dread d5/d89/fa7 [0,4194304] 0 2024-04-05T21:26:48.617 INFO:tasks.workunit.client.0.smithi121.stdout:6/543: dwrite d1/d2/dc/f5b [0,4194304] 0 2024-04-05T21:26:48.618 INFO:tasks.workunit.client.0.smithi121.stdout:8/682: mkdir d12/d13/dd3 0 2024-04-05T21:26:48.620 INFO:tasks.workunit.client.0.smithi121.stdout:7/745: mkdir d0/d9/d22/de9 0 2024-04-05T21:26:48.626 INFO:tasks.workunit.client.0.smithi121.stdout:8/683: link d12/d13/fd2 d12/d13/fd4 0 2024-04-05T21:26:48.626 INFO:tasks.workunit.client.0.smithi121.stdout:0/802: dwrite d7/d71/d29/f9b [0,4194304] 0 2024-04-05T21:26:48.628 INFO:tasks.workunit.client.0.smithi121.stdout:6/544: unlink d1/d2/d3/d4/d21/d57/d42/d48/d7c/c7f 0 2024-04-05T21:26:48.628 INFO:tasks.workunit.client.0.smithi121.stdout:8/684: chown d12/ca0 5822850 1 2024-04-05T21:26:48.630 INFO:tasks.workunit.client.0.smithi121.stdout:0/803: creat d7/d7e/d97/f11b x:0 0 0 2024-04-05T21:26:48.630 INFO:tasks.workunit.client.1.smithi188.stdout:0/815: dread d4/d6/d61/fae [0,4194304] 0 2024-04-05T21:26:48.631 INFO:tasks.workunit.client.0.smithi121.stdout:8/685: mknod d12/d13/d56/d57/da4/cd5 0 2024-04-05T21:26:48.632 INFO:tasks.workunit.client.0.smithi121.stdout:0/804: creat d7/d62/d3c/d6b/f11c x:0 0 0 2024-04-05T21:26:48.633 INFO:tasks.workunit.client.0.smithi121.stdout:8/686: write d12/d13/d56/d57/f5a [718832,25662] 0 2024-04-05T21:26:48.635 INFO:tasks.workunit.client.0.smithi121.stdout:6/545: getdents d1/d2/d1b/d35/d39/d92 0 2024-04-05T21:26:48.638 INFO:tasks.workunit.client.0.smithi121.stdout:0/805: mknod d7/d71/d45/c11d 0 2024-04-05T21:26:48.638 INFO:tasks.workunit.client.0.smithi121.stdout:6/546: readlink d1/d2/d3/d4/d21/d57/d42/d48/l89 0 2024-04-05T21:26:48.639 INFO:tasks.workunit.client.0.smithi121.stdout:8/687: mknod d12/d50/d93/db5/cd6 0 2024-04-05T21:26:48.639 INFO:tasks.workunit.client.0.smithi121.stdout:6/547: readlink d1/lad 0 2024-04-05T21:26:48.641 INFO:tasks.workunit.client.0.smithi121.stdout:7/746: dread d0/d32/f35 [0,4194304] 0 2024-04-05T21:26:48.642 INFO:tasks.workunit.client.0.smithi121.stdout:0/806: mkdir d7/d71/d45/d46/d106/d11e 0 2024-04-05T21:26:48.643 INFO:tasks.workunit.client.0.smithi121.stdout:0/807: chown d7/d71/d94/dd1/c115 793941971 1 2024-04-05T21:26:48.644 INFO:tasks.workunit.client.0.smithi121.stdout:6/548: symlink d1/d2/d1b/d2e/lb9 0 2024-04-05T21:26:48.645 INFO:tasks.workunit.client.0.smithi121.stdout:4/831: write d5/d11/d2c/d9d/dae/f4c [725213,98488] 0 2024-04-05T21:26:48.647 INFO:tasks.workunit.client.0.smithi121.stdout:6/549: chown d1/d2/d1b/c79 2 1 2024-04-05T21:26:48.647 INFO:tasks.workunit.client.0.smithi121.stdout:7/747: rmdir d0/d23/d5b/d52 39 2024-04-05T21:26:48.648 INFO:tasks.workunit.client.0.smithi121.stdout:7/748: read - d0/d23/f7f zero size 2024-04-05T21:26:48.650 INFO:tasks.workunit.client.0.smithi121.stdout:7/749: write d0/d23/f8b [4341819,127594] 0 2024-04-05T21:26:48.650 INFO:tasks.workunit.client.0.smithi121.stdout:4/832: creat d5/d45/d8b/f10a x:0 0 0 2024-04-05T21:26:48.656 INFO:tasks.workunit.client.0.smithi121.stdout:6/550: link d1/d2/d3/d5/fac d1/d2/d1b/d5e/fba 0 2024-04-05T21:26:48.658 INFO:tasks.workunit.client.1.smithi188.stdout:0/816: dread d4/d8/db/d42/da2/dd2/fd3 [0,4194304] 0 2024-04-05T21:26:48.658 INFO:tasks.workunit.client.0.smithi121.stdout:6/551: stat d1/d2/d3/d4/d21/f93 0 2024-04-05T21:26:48.658 INFO:tasks.workunit.client.0.smithi121.stdout:4/833: mknod d5/d11/d2c/da4/c10b 0 2024-04-05T21:26:48.659 INFO:tasks.workunit.client.1.smithi188.stdout:0/817: symlink d4/d6/df4/l12d 0 2024-04-05T21:26:48.661 INFO:tasks.workunit.client.0.smithi121.stdout:7/750: rename d0/d32/d7c/fac to d0/d9/d22/d28/d56/db9/fea 0 2024-04-05T21:26:48.662 INFO:tasks.workunit.client.1.smithi188.stdout:0/818: unlink d4/d8/db/d42/d38/f127 0 2024-04-05T21:26:48.662 INFO:tasks.workunit.client.0.smithi121.stdout:7/751: dread - d0/d9/fa9 zero size 2024-04-05T21:26:48.664 INFO:tasks.workunit.client.1.smithi188.stdout:0/819: creat d4/d8/db/d21/d4a/d73/d116/de8/f12e x:0 0 0 2024-04-05T21:26:48.664 INFO:tasks.workunit.client.0.smithi121.stdout:6/552: dwrite d1/d2/dc/f4c [0,4194304] 0 2024-04-05T21:26:48.665 INFO:tasks.workunit.client.0.smithi121.stdout:4/834: rename d5/d11/d26/dbc/dca/de9/c104 to d5/d11/d28/d3b/c10c 0 2024-04-05T21:26:48.665 INFO:tasks.workunit.client.1.smithi188.stdout:0/820: write d4/d6/f1f [546353,73211] 0 2024-04-05T21:26:48.666 INFO:tasks.workunit.client.0.smithi121.stdout:7/752: write d0/d32/f83 [653931,66934] 0 2024-04-05T21:26:48.667 INFO:tasks.workunit.client.1.smithi188.stdout:0/821: stat d4/d8/db/cb8 0 2024-04-05T21:26:48.667 INFO:tasks.workunit.client.0.smithi121.stdout:8/688: dread d12/d50/d93/d96/fab [0,4194304] 0 2024-04-05T21:26:48.668 INFO:tasks.workunit.client.0.smithi121.stdout:6/553: symlink d1/d2/d3/d5/d1f/d6d/lbb 0 2024-04-05T21:26:48.669 INFO:tasks.workunit.client.0.smithi121.stdout:0/808: write d7/d1c/fb6 [4819420,29278] 0 2024-04-05T21:26:48.669 INFO:tasks.workunit.client.0.smithi121.stdout:8/689: read - d12/d50/fb3 zero size 2024-04-05T21:26:48.670 INFO:tasks.workunit.client.0.smithi121.stdout:4/835: creat d5/d11/d26/dbc/dca/de9/f10d x:0 0 0 2024-04-05T21:26:48.672 INFO:tasks.workunit.client.0.smithi121.stdout:7/753: creat d0/d32/d4a/feb x:0 0 0 2024-04-05T21:26:48.673 INFO:tasks.workunit.client.0.smithi121.stdout:8/690: chown d12/d13/d56/d57/d5e/cb4 164626 1 2024-04-05T21:26:48.673 INFO:tasks.workunit.client.1.smithi188.stdout:0/822: rename d4/d8/db/dac/d112 to d4/d6/d13/dca/d12f 0 2024-04-05T21:26:48.676 INFO:tasks.workunit.client.0.smithi121.stdout:4/836: fsync d5/d45/d67/f84 0 2024-04-05T21:26:48.676 INFO:tasks.workunit.client.1.smithi188.stdout:0/823: stat d4/d8/db/d42/d38/d77/c8e 0 2024-04-05T21:26:48.677 INFO:tasks.workunit.client.1.smithi188.stdout:0/824: chown d4/d8/d39/f109 989816102 1 2024-04-05T21:26:48.677 INFO:tasks.workunit.client.0.smithi121.stdout:7/754: mknod d0/d23/d5b/d52/d61/cec 0 2024-04-05T21:26:48.678 INFO:tasks.workunit.client.0.smithi121.stdout:8/691: mkdir d12/d50/d6a/d63/d8b/dd7 0 2024-04-05T21:26:48.679 INFO:tasks.workunit.client.1.smithi188.stdout:0/825: read d4/d8/db/d21/d3c/fa5 [979865,26126] 0 2024-04-05T21:26:48.680 INFO:tasks.workunit.client.0.smithi121.stdout:8/692: dread - d12/d13/d56/d57/d5e/db9/fd0 zero size 2024-04-05T21:26:48.680 INFO:tasks.workunit.client.0.smithi121.stdout:4/837: mkdir d5/d11/d26/d49/d10e 0 2024-04-05T21:26:48.683 INFO:tasks.workunit.client.0.smithi121.stdout:8/693: chown d12/d50/d93/d96/l7c 3831 1 2024-04-05T21:26:48.683 INFO:tasks.workunit.client.0.smithi121.stdout:4/838: read d5/d11/d28/f1b [3997497,41468] 0 2024-04-05T21:26:48.689 INFO:tasks.workunit.client.0.smithi121.stdout:4/839: getdents d5/d11/d26/dbc/dca/de9 0 2024-04-05T21:26:48.690 INFO:tasks.workunit.client.1.smithi188.stdout:0/826: dwrite d4/d8/db/f28 [0,4194304] 0 2024-04-05T21:26:48.692 INFO:tasks.workunit.client.1.smithi188.stdout:0/827: mkdir d4/d6/d13/da1/d130 0 2024-04-05T21:26:48.692 INFO:tasks.workunit.client.0.smithi121.stdout:5/789: sync 2024-04-05T21:26:48.693 INFO:tasks.workunit.client.1.smithi188.stdout:0/828: stat d4/d8/db/d21/d4a/d73/f94 0 2024-04-05T21:26:48.694 INFO:tasks.workunit.client.0.smithi121.stdout:8/694: dwrite d12/d50/d6a/d63/d8b/fcd [0,4194304] 0 2024-04-05T21:26:48.695 INFO:tasks.workunit.client.1.smithi188.stdout:0/829: mknod d4/d8/db/d21/d4a/d73/d116/dd1/d126/c131 0 2024-04-05T21:26:48.697 INFO:tasks.workunit.client.1.smithi188.stdout:0/830: read d4/d8/db/d21/d3c/f9e [63466,130234] 0 2024-04-05T21:26:48.699 INFO:tasks.workunit.client.0.smithi121.stdout:6/554: dwrite d1/d2/d3/d5/fd [4194304,4194304] 0 2024-04-05T21:26:48.700 INFO:tasks.workunit.client.0.smithi121.stdout:4/840: mknod d5/d11/d2c/d9d/c10f 0 2024-04-05T21:26:48.700 INFO:tasks.workunit.client.1.smithi188.stdout:0/831: symlink d4/d8/db/d21/d4a/d73/d116/de8/l132 0 2024-04-05T21:26:48.700 INFO:tasks.workunit.client.0.smithi121.stdout:7/755: dwrite d0/d23/d5b/f46 [0,4194304] 0 2024-04-05T21:26:48.712 INFO:tasks.workunit.client.1.smithi188.stdout:0/832: rename d4/d6/d46/dba/lbb to d4/d8/d78/dc2/d11a/d80/l133 0 2024-04-05T21:26:48.743 INFO:tasks.workunit.client.0.smithi121.stdout:8/695: creat d12/d50/d93/db5/fd8 x:0 0 0 2024-04-05T21:26:48.744 INFO:tasks.workunit.client.0.smithi121.stdout:4/841: dread - d5/d11/d26/ffc zero size 2024-04-05T21:26:48.745 INFO:tasks.workunit.client.0.smithi121.stdout:7/756: write d0/d23/d5a/f62 [1308984,99837] 0 2024-04-05T21:26:48.746 INFO:tasks.workunit.client.1.smithi188.stdout:0/833: rmdir d4/d6/d13/dea 39 2024-04-05T21:26:48.748 INFO:tasks.workunit.client.0.smithi121.stdout:7/757: mkdir d0/d57/ded 0 2024-04-05T21:26:48.749 INFO:tasks.workunit.client.0.smithi121.stdout:7/758: write d0/d32/fa0 [5105609,93273] 0 2024-04-05T21:26:48.750 INFO:tasks.workunit.client.1.smithi188.stdout:0/834: mkdir d4/d8/db/d21/d4a/d73/d116/d24/d57/d134 0 2024-04-05T21:26:48.751 INFO:tasks.workunit.client.0.smithi121.stdout:4/842: dread d5/d11/d60/f9f [0,4194304] 0 2024-04-05T21:26:48.756 INFO:tasks.workunit.client.0.smithi121.stdout:7/759: rename d0/d23/d5b/d52/d61/dcb/ccd to d0/d9/d22/de9/cee 0 2024-04-05T21:26:48.757 INFO:tasks.workunit.client.0.smithi121.stdout:6/555: dwrite d1/d2/d3/d5/f58 [0,4194304] 0 2024-04-05T21:26:48.757 INFO:tasks.workunit.client.0.smithi121.stdout:8/696: write d12/d13/d15/d1d/f4f [2190977,75059] 0 2024-04-05T21:26:48.764 INFO:tasks.workunit.client.0.smithi121.stdout:7/760: rename d0/d23/d5b to d0/d9/d22/d28/d56/db9/ddf/def 0 2024-04-05T21:26:48.765 INFO:tasks.workunit.client.0.smithi121.stdout:4/843: truncate d5/d11/f17 1645048 0 2024-04-05T21:26:48.766 INFO:tasks.workunit.client.0.smithi121.stdout:8/697: write fc [160192,8760] 0 2024-04-05T21:26:48.768 INFO:tasks.workunit.client.0.smithi121.stdout:6/556: link d1/d2/d1b/d1e/c94 d1/d2/d3/d4/d21/d57/d42/d48/cbc 0 2024-04-05T21:26:48.768 INFO:tasks.workunit.client.0.smithi121.stdout:4/844: write d5/d11/d2c/d9d/dae/d29/db4/fe6 [5194785,13049] 0 2024-04-05T21:26:48.770 INFO:tasks.workunit.client.0.smithi121.stdout:8/698: creat d12/d50/d61/d7f/d9a/fd9 x:0 0 0 2024-04-05T21:26:48.771 INFO:tasks.workunit.client.0.smithi121.stdout:7/761: write d0/d9/d22/d28/f31 [3454833,31118] 0 2024-04-05T21:26:48.774 INFO:tasks.workunit.client.0.smithi121.stdout:8/699: chown d12/d50/d6a/d63/d8b/fcd 1670 1 2024-04-05T21:26:48.774 INFO:tasks.workunit.client.0.smithi121.stdout:6/557: rename d1/d2/d1b/d2e/d53 to d1/d2/d1b/d5e/dbd 0 2024-04-05T21:26:48.781 INFO:tasks.workunit.client.0.smithi121.stdout:7/762: rename d0/d9/d22/d3e/d9c/da7 to d0/d32/d4e/d73/dbc/df0 0 2024-04-05T21:26:48.782 INFO:tasks.workunit.client.1.smithi188.stdout:1/913: sync 2024-04-05T21:26:48.782 INFO:tasks.workunit.client.1.smithi188.stdout:5/970: sync 2024-04-05T21:26:48.783 INFO:tasks.workunit.client.0.smithi121.stdout:7/763: symlink d0/d9/d22/d28/d56/db9/ddf/def/d52/lf1 0 2024-04-05T21:26:48.784 INFO:tasks.workunit.client.1.smithi188.stdout:5/971: unlink d6/d17/d23/f55 0 2024-04-05T21:26:48.786 INFO:tasks.workunit.client.1.smithi188.stdout:5/972: truncate d6/d82/f89 1260062 0 2024-04-05T21:26:48.786 INFO:tasks.workunit.client.0.smithi121.stdout:7/764: creat d0/d9/d22/d3e/d76/ff2 x:0 0 0 2024-04-05T21:26:48.786 INFO:tasks.workunit.client.1.smithi188.stdout:1/914: creat d1/dc/dfc/d111/f123 x:0 0 0 2024-04-05T21:26:48.787 INFO:tasks.workunit.client.0.smithi121.stdout:8/700: link d12/d13/d15/c44 d12/cda 0 2024-04-05T21:26:48.787 INFO:tasks.workunit.client.0.smithi121.stdout:7/765: stat d0/d23/f8b 0 2024-04-05T21:26:48.789 INFO:tasks.workunit.client.1.smithi188.stdout:1/915: mknod d1/dc/d16/d46/d88/d3f/d119/c124 0 2024-04-05T21:26:48.792 INFO:tasks.workunit.client.1.smithi188.stdout:5/973: getdents d6/d106/d8e/def 0 2024-04-05T21:26:48.792 INFO:tasks.workunit.client.0.smithi121.stdout:7/766: truncate d0/f8 7609444 0 2024-04-05T21:26:48.794 INFO:tasks.workunit.client.1.smithi188.stdout:5/974: readlink d6/d15/d1b/ddc/le8 0 2024-04-05T21:26:48.796 INFO:tasks.workunit.client.1.smithi188.stdout:5/975: chown d6/d106/c112 1277684828 1 2024-04-05T21:26:48.798 INFO:tasks.workunit.client.0.smithi121.stdout:6/558: dwrite d1/d2/f55 [0,4194304] 0 2024-04-05T21:26:48.798 INFO:tasks.workunit.client.0.smithi121.stdout:4/845: dread d5/d89/fd1 [0,4194304] 0 2024-04-05T21:26:48.802 INFO:tasks.workunit.client.0.smithi121.stdout:8/701: dwrite d12/d13/d15/d1d/f4c [0,4194304] 0 2024-04-05T21:26:48.802 INFO:tasks.workunit.client.1.smithi188.stdout:1/916: truncate d1/dc/d16/d21/d27/d4a/f7f 3027634 0 2024-04-05T21:26:48.804 INFO:tasks.workunit.client.1.smithi188.stdout:5/976: read d6/f11 [6310072,9086] 0 2024-04-05T21:26:48.804 INFO:tasks.workunit.client.0.smithi121.stdout:8/702: stat d12/c91 0 2024-04-05T21:26:48.804 INFO:tasks.workunit.client.0.smithi121.stdout:6/559: symlink d1/d2/daf/d6f/lbe 0 2024-04-05T21:26:48.806 INFO:tasks.workunit.client.0.smithi121.stdout:7/767: mknod d0/d9/d22/d3e/cf3 0 2024-04-05T21:26:48.809 INFO:tasks.workunit.client.0.smithi121.stdout:8/703: chown d12/d13/d15/d1d/f27 1936224 1 2024-04-05T21:26:48.809 INFO:tasks.workunit.client.1.smithi188.stdout:0/835: dwrite d4/d6/d13/da1/fb1 [0,4194304] 0 2024-04-05T21:26:48.810 INFO:tasks.workunit.client.0.smithi121.stdout:7/768: chown d0/d9/d22 1254 1 2024-04-05T21:26:48.810 INFO:tasks.workunit.client.1.smithi188.stdout:5/977: symlink d6/d82/d40/d42/d81/da6/l151 0 2024-04-05T21:26:48.811 INFO:tasks.workunit.client.0.smithi121.stdout:8/704: chown d12/d50/d93/db5 7 1 2024-04-05T21:26:48.811 INFO:tasks.workunit.client.0.smithi121.stdout:6/560: symlink d1/d2/d1b/d35/lbf 0 2024-04-05T21:26:48.811 INFO:tasks.workunit.client.1.smithi188.stdout:1/917: getdents d1/d7 0 2024-04-05T21:26:48.812 INFO:tasks.workunit.client.0.smithi121.stdout:4/846: dwrite d5/d11/d26/dbc/dc5/dd5/fe1 [0,4194304] 0 2024-04-05T21:26:48.813 INFO:tasks.workunit.client.1.smithi188.stdout:5/978: stat d6/d82/d40/f90 0 2024-04-05T21:26:48.813 INFO:tasks.workunit.client.1.smithi188.stdout:0/836: readlink d4/d8/db/d21/d4a/d73/d116/dd1/le6 0 2024-04-05T21:26:48.815 INFO:tasks.workunit.client.0.smithi121.stdout:6/561: write d1/d2/d1b/d5e/dbd/faa [125003,74284] 0 2024-04-05T21:26:48.815 INFO:tasks.workunit.client.0.smithi121.stdout:8/705: creat d12/d50/d6a/d63/fdb x:0 0 0 2024-04-05T21:26:48.819 INFO:tasks.workunit.client.0.smithi121.stdout:4/847: rmdir d5/d11/d26/dbc 39 2024-04-05T21:26:48.820 INFO:tasks.workunit.client.1.smithi188.stdout:1/918: dwrite d1/dc/d16/d46/df7/f8b [0,4194304] 0 2024-04-05T21:26:48.822 INFO:tasks.workunit.client.0.smithi121.stdout:7/769: rmdir d0/d9/d22/d28/d56/db9/ddf/def/d52/d61 39 2024-04-05T21:26:48.823 INFO:tasks.workunit.client.0.smithi121.stdout:7/770: chown d0/d32/f8f 204 1 2024-04-05T21:26:48.824 INFO:tasks.workunit.client.0.smithi121.stdout:8/706: symlink d12/d50/d6a/ldc 0 2024-04-05T21:26:48.825 INFO:tasks.workunit.client.0.smithi121.stdout:7/771: chown d0/d9/fa9 0 1 2024-04-05T21:26:48.829 INFO:tasks.workunit.client.0.smithi121.stdout:7/772: read d0/f15 [4763142,127498] 0 2024-04-05T21:26:48.831 INFO:tasks.workunit.client.0.smithi121.stdout:4/848: dwrite d5/d11/d28/d73/f76 [0,4194304] 0 2024-04-05T21:26:48.835 INFO:tasks.workunit.client.0.smithi121.stdout:4/849: rename d5/d11/d2c/d8c to d5/d11/d26/d47/df7/d110 0 2024-04-05T21:26:48.836 INFO:tasks.workunit.client.1.smithi188.stdout:2/894: sync 2024-04-05T21:26:48.836 INFO:tasks.workunit.client.1.smithi188.stdout:8/731: sync 2024-04-05T21:26:48.836 INFO:tasks.workunit.client.1.smithi188.stdout:6/992: sync 2024-04-05T21:26:48.839 INFO:tasks.workunit.client.0.smithi121.stdout:4/850: rename d5/d11/d2c/f65 to d5/d45/f111 0 2024-04-05T21:26:48.839 INFO:tasks.workunit.client.0.smithi121.stdout:7/773: read d0/d32/f65 [667707,13812] 0 2024-04-05T21:26:48.840 INFO:tasks.workunit.client.1.smithi188.stdout:2/895: mkdir d5/db/dd/d73/d109/d66/d8f/d127 0 2024-04-05T21:26:48.843 INFO:tasks.workunit.client.1.smithi188.stdout:2/896: write d5/db/dd/d36/d3c/d63/ff4 [358166,43009] 0 2024-04-05T21:26:48.843 INFO:tasks.workunit.client.0.smithi121.stdout:6/562: dread d1/f38 [0,4194304] 0 2024-04-05T21:26:48.844 INFO:tasks.workunit.client.0.smithi121.stdout:4/851: creat d5/d45/d67/dab/f112 x:0 0 0 2024-04-05T21:26:48.849 INFO:tasks.workunit.client.0.smithi121.stdout:4/852: rename d5/d11/d28/d3b/fdb to d5/d11/d26/d49/f113 0 2024-04-05T21:26:48.851 INFO:tasks.workunit.client.0.smithi121.stdout:4/853: symlink d5/d45/d8b/l114 0 2024-04-05T21:26:48.852 INFO:tasks.workunit.client.0.smithi121.stdout:4/854: rmdir d5/d89 39 2024-04-05T21:26:48.854 INFO:tasks.workunit.client.0.smithi121.stdout:7/774: write d0/d9/d22/f6c [4940847,97167] 0 2024-04-05T21:26:48.855 INFO:tasks.workunit.client.0.smithi121.stdout:4/855: dread d5/d11/d60/f9f [0,4194304] 0 2024-04-05T21:26:48.855 INFO:tasks.workunit.client.0.smithi121.stdout:6/563: write d1/d2/d1b/d5e/fba [467490,98695] 0 2024-04-05T21:26:48.857 INFO:tasks.workunit.client.1.smithi188.stdout:5/979: write d6/d106/fb3 [1868696,29902] 0 2024-04-05T21:26:48.858 INFO:tasks.workunit.client.0.smithi121.stdout:7/775: rmdir d0/d32/d4e/d73/dbc 39 2024-04-05T21:26:48.861 INFO:tasks.workunit.client.0.smithi121.stdout:6/564: rename d1/d2/d1b/d1e to d1/d2/dc/d9a/dc0 0 2024-04-05T21:26:48.862 INFO:tasks.workunit.client.0.smithi121.stdout:4/856: dread d5/d11/d28/d87/f96 [0,4194304] 0 2024-04-05T21:26:48.862 INFO:tasks.workunit.client.0.smithi121.stdout:6/565: dread - d1/d2/d1b/da1/fa6 zero size 2024-04-05T21:26:48.865 INFO:tasks.workunit.client.0.smithi121.stdout:4/857: mknod d5/d11/d28/c115 0 2024-04-05T21:26:48.866 INFO:tasks.workunit.client.1.smithi188.stdout:2/897: write d5/fb7 [1689624,127383] 0 2024-04-05T21:26:48.866 INFO:tasks.workunit.client.0.smithi121.stdout:4/858: read d5/d11/d26/f6b [218355,99292] 0 2024-04-05T21:26:48.867 INFO:tasks.workunit.client.0.smithi121.stdout:4/859: readlink d5/d11/d26/d47/lec 0 2024-04-05T21:26:48.868 INFO:tasks.workunit.client.1.smithi188.stdout:5/980: chown d6/dad/f8d 366935 1 2024-04-05T21:26:48.868 INFO:tasks.workunit.client.0.smithi121.stdout:6/566: creat d1/d2/d3/d4/d21/d57/d42/d48/d7c/db3/fc1 x:0 0 0 2024-04-05T21:26:48.873 INFO:tasks.workunit.client.0.smithi121.stdout:6/567: dwrite d1/f72 [0,4194304] 0 2024-04-05T21:26:48.874 INFO:tasks.workunit.client.1.smithi188.stdout:6/993: dwrite d6/d13/d18/d94/f11e [0,4194304] 0 2024-04-05T21:26:48.879 INFO:tasks.workunit.client.1.smithi188.stdout:2/898: chown d5/db/d35/d3b/l9e 24085037 1 2024-04-05T21:26:48.879 INFO:tasks.workunit.client.1.smithi188.stdout:1/919: dwrite d1/dc/d16/d46/d88/f79 [0,4194304] 0 2024-04-05T21:26:48.884 INFO:tasks.workunit.client.1.smithi188.stdout:0/837: dwrite d4/d8/db/d21/d4a/d73/d116/de8/f107 [4194304,4194304] 0 2024-04-05T21:26:48.884 INFO:tasks.workunit.client.1.smithi188.stdout:5/981: symlink d6/d82/d40/d42/d81/l152 0 2024-04-05T21:26:48.884 INFO:tasks.workunit.client.1.smithi188.stdout:8/732: dwrite d5/d21/d40/d43/d44/d5b/d99/d69/fb3 [0,4194304] 0 2024-04-05T21:26:48.885 INFO:tasks.workunit.client.1.smithi188.stdout:2/899: dread - d5/db/dd/d36/d3c/fa2 zero size 2024-04-05T21:26:48.888 INFO:tasks.workunit.client.0.smithi121.stdout:6/568: write d1/d2/d3/d4/d21/d57/f49 [624223,113796] 0 2024-04-05T21:26:48.888 INFO:tasks.workunit.client.0.smithi121.stdout:6/569: fsync d1/d4b/f77 0 2024-04-05T21:26:48.890 INFO:tasks.workunit.client.0.smithi121.stdout:6/570: write d1/d2/d3/d4/d21/d57/f49 [1688118,23142] 0 2024-04-05T21:26:48.893 INFO:tasks.workunit.client.0.smithi121.stdout:6/571: symlink d1/d2/d1b/d35/d39/d92/lc2 0 2024-04-05T21:26:48.894 INFO:tasks.workunit.client.0.smithi121.stdout:6/572: stat d1/c24 0 2024-04-05T21:26:48.895 INFO:tasks.workunit.client.0.smithi121.stdout:6/573: write d1/d2/d3/d5/fb7 [832914,77221] 0 2024-04-05T21:26:48.898 INFO:tasks.workunit.client.1.smithi188.stdout:2/900: rename d5/db/dd/d73/d109/d66/fef to d5/db/dd/d73/d109/d8c/d11d/f128 0 2024-04-05T21:26:48.898 INFO:tasks.workunit.client.1.smithi188.stdout:0/838: mkdir d4/d8/db/d21/d4a/d73/d116/d24/d101/d135 0 2024-04-05T21:26:48.899 INFO:tasks.workunit.client.1.smithi188.stdout:5/982: symlink d6/d15/d5c/d7a/d44/d98/dbf/d123/l153 0 2024-04-05T21:26:48.901 INFO:tasks.workunit.client.1.smithi188.stdout:3/980: sync 2024-04-05T21:26:48.902 INFO:tasks.workunit.client.1.smithi188.stdout:1/920: creat d1/dc/d16/d46/d95/f125 x:0 0 0 2024-04-05T21:26:48.904 INFO:tasks.workunit.client.1.smithi188.stdout:8/733: dwrite d5/d21/d40/d43/d44/d59/d8b/feb [0,4194304] 0 2024-04-05T21:26:48.906 INFO:tasks.workunit.client.1.smithi188.stdout:2/901: creat d5/db/dd/d73/d109/f129 x:0 0 0 2024-04-05T21:26:48.907 INFO:tasks.workunit.client.1.smithi188.stdout:3/981: fsync d5/d7/d65/d61/db5/fbf 0 2024-04-05T21:26:48.909 INFO:tasks.workunit.client.1.smithi188.stdout:5/983: creat d6/d15/d5c/d7a/d44/d51/df1/d104/d124/f154 x:0 0 0 2024-04-05T21:26:48.911 INFO:tasks.workunit.client.1.smithi188.stdout:0/839: link d4/d6/d13/da1/cfc d4/d6/d13/dc1/c136 0 2024-04-05T21:26:48.914 INFO:tasks.workunit.client.1.smithi188.stdout:1/921: getdents d1/dc/d16/d46/d88/d13/d60 0 2024-04-05T21:26:48.917 INFO:tasks.workunit.client.1.smithi188.stdout:9/989: sync 2024-04-05T21:26:48.918 INFO:tasks.workunit.client.1.smithi188.stdout:1/922: creat d1/d7/dbf/f126 x:0 0 0 2024-04-05T21:26:48.918 INFO:tasks.workunit.client.0.smithi121.stdout:3/575: sync 2024-04-05T21:26:48.919 INFO:tasks.workunit.client.1.smithi188.stdout:2/902: link d5/db/c4e d5/db/dd/d73/d120/c12a 0 2024-04-05T21:26:48.920 INFO:tasks.workunit.client.1.smithi188.stdout:5/984: truncate d6/d15/d1b/ddc/f129 431640 0 2024-04-05T21:26:48.921 INFO:tasks.workunit.client.0.smithi121.stdout:3/576: write d0/d3/d24/d3d/d1a/d23/d90/d9e/fa0 [700522,120484] 0 2024-04-05T21:26:48.922 INFO:tasks.workunit.client.1.smithi188.stdout:2/903: stat d5/db/dd/d36/d3c/d48/d85/f94 0 2024-04-05T21:26:48.923 INFO:tasks.workunit.client.0.smithi121.stdout:3/577: write d0/d3/d24/d3d/d1a/d23/f56 [1048966,48674] 0 2024-04-05T21:26:48.924 INFO:tasks.workunit.client.1.smithi188.stdout:5/985: truncate d6/d15/d5c/d7a/fd0 8879867 0 2024-04-05T21:26:48.925 INFO:tasks.workunit.client.1.smithi188.stdout:1/923: rename d1/dc/d16/d46/d88/d13/d60/l7c to d1/dc/dfc/l127 0 2024-04-05T21:26:48.926 INFO:tasks.workunit.client.0.smithi121.stdout:3/578: creat d0/d3/d24/d3d/d1a/d1e/db1/fbb x:0 0 0 2024-04-05T21:26:48.927 INFO:tasks.workunit.client.1.smithi188.stdout:0/840: link d4/d8/d39/f83 d4/d8/db/d42/f137 0 2024-04-05T21:26:48.929 INFO:tasks.workunit.client.1.smithi188.stdout:2/904: mkdir d5/db/dd/d73/d109/d5d/d12b 0 2024-04-05T21:26:48.929 INFO:tasks.workunit.client.1.smithi188.stdout:9/990: getdents dd/d11/d73 0 2024-04-05T21:26:48.929 INFO:tasks.workunit.client.0.smithi121.stdout:3/579: mknod d0/d3/d24/db3/cbc 0 2024-04-05T21:26:48.929 INFO:tasks.workunit.client.0.smithi121.stdout:6/574: write d1/d2/dc/d9a/dc0/f2a [3908865,118536] 0 2024-04-05T21:26:48.931 INFO:tasks.workunit.client.1.smithi188.stdout:2/905: chown d5/d5f 17540 1 2024-04-05T21:26:48.931 INFO:tasks.workunit.client.0.smithi121.stdout:6/575: read d1/d2/d1b/d5e/fba [427360,16850] 0 2024-04-05T21:26:48.931 INFO:tasks.workunit.client.1.smithi188.stdout:0/841: write d4/d8/fb4 [1960399,39050] 0 2024-04-05T21:26:48.935 INFO:tasks.workunit.client.0.smithi121.stdout:3/580: dwrite d0/d3/f4 [0,4194304] 0 2024-04-05T21:26:48.937 INFO:tasks.workunit.client.1.smithi188.stdout:9/991: chown dd/d11/d1f/d26/dc0/d6b/d11e/f148 14 1 2024-04-05T21:26:48.939 INFO:tasks.workunit.client.0.smithi121.stdout:3/581: symlink d0/d3/d24/d6d/d7b/da4/lbd 0 2024-04-05T21:26:48.941 INFO:tasks.workunit.client.0.smithi121.stdout:6/576: dwrite d1/d2/d3/d4/d21/d57/d42/d48/d7c/fa9 [0,4194304] 0 2024-04-05T21:26:48.942 INFO:tasks.workunit.client.0.smithi121.stdout:3/582: write d0/d3/d24/d3d/d1a/d1e/db1/fbb [357264,112624] 0 2024-04-05T21:26:48.943 INFO:tasks.workunit.client.0.smithi121.stdout:6/577: mknod d1/d4b/cc3 0 2024-04-05T21:26:48.944 INFO:tasks.workunit.client.1.smithi188.stdout:5/986: dwrite d6/d15/d5c/d7a/d44/d98/fcd [0,4194304] 0 2024-04-05T21:26:48.945 INFO:tasks.workunit.client.1.smithi188.stdout:6/994: dwrite d6/d13/d18/d19/d2c/f3e [0,4194304] 0 2024-04-05T21:26:48.945 INFO:tasks.workunit.client.0.smithi121.stdout:3/583: write d0/d3/d24/d3d/d1a/d23/d38/d55/fac [749007,64707] 0 2024-04-05T21:26:48.947 INFO:tasks.workunit.client.0.smithi121.stdout:3/584: chown d0/d3/d24/l52 372012 1 2024-04-05T21:26:48.949 INFO:tasks.workunit.client.0.smithi121.stdout:6/578: creat d1/d2/d3/d5/d1f/fc4 x:0 0 0 2024-04-05T21:26:48.952 INFO:tasks.workunit.client.0.smithi121.stdout:6/579: mknod d1/d2/d3/d4/d21/d57/d42/d48/d7c/db3/cc5 0 2024-04-05T21:26:48.953 INFO:tasks.workunit.client.0.smithi121.stdout:6/580: write d1/d32/fae [3215741,1401] 0 2024-04-05T21:26:48.954 INFO:tasks.workunit.client.0.smithi121.stdout:6/581: write d1/d2/d3/d5/d1f/f27 [9909067,108511] 0 2024-04-05T21:26:48.957 INFO:tasks.workunit.client.1.smithi188.stdout:3/982: dread d5/d7/d6d/d75/faa [0,4194304] 0 2024-04-05T21:26:48.958 INFO:tasks.workunit.client.1.smithi188.stdout:1/924: dread d1/dc/d16/d2d/f42 [0,4194304] 0 2024-04-05T21:26:48.959 INFO:tasks.workunit.client.1.smithi188.stdout:8/734: dwrite d5/d6e/d86/dcb/fa0 [0,4194304] 0 2024-04-05T21:26:48.959 INFO:tasks.workunit.client.1.smithi188.stdout:6/995: creat d6/d13/d18/d19/d2c/d3f/d95/f135 x:0 0 0 2024-04-05T21:26:48.960 INFO:tasks.workunit.client.1.smithi188.stdout:5/987: chown d6/d82/d40/dc1/fd1 2 1 2024-04-05T21:26:48.962 INFO:tasks.workunit.client.1.smithi188.stdout:3/983: mknod d5/d7/d65/d139/d145/c146 0 2024-04-05T21:26:48.963 INFO:tasks.workunit.client.1.smithi188.stdout:1/925: creat d1/dc/d16/d46/d95/d99/f128 x:0 0 0 2024-04-05T21:26:48.963 INFO:tasks.workunit.client.1.smithi188.stdout:9/992: link dd/d11/d22/f56 dd/d11/d22/d7e/de3/f149 0 2024-04-05T21:26:48.964 INFO:tasks.workunit.client.0.smithi121.stdout:6/582: dwrite d1/d2/d1b/d5e/fba [0,4194304] 0 2024-04-05T21:26:48.965 INFO:tasks.workunit.client.1.smithi188.stdout:8/735: creat d5/d21/d40/d43/d44/d5b/d99/da3/ff2 x:0 0 0 2024-04-05T21:26:48.968 INFO:tasks.workunit.client.1.smithi188.stdout:5/988: rename d6/d82/d40/d42/d81/d10c to d6/d106/d8e/def/d13a/d155 0 2024-04-05T21:26:48.968 INFO:tasks.workunit.client.1.smithi188.stdout:8/736: truncate d5/d21/fe0 124584 0 2024-04-05T21:26:48.969 INFO:tasks.workunit.client.0.smithi121.stdout:6/583: rename d1/d2/dc/l81 to d1/d2/d3/d5/d1f/lc6 0 2024-04-05T21:26:48.973 INFO:tasks.workunit.client.1.smithi188.stdout:1/926: fdatasync d1/dc/d16/d46/d88/d13/f8d 0 2024-04-05T21:26:48.975 INFO:tasks.workunit.client.1.smithi188.stdout:3/984: rename d5/d7/d65 to d5/d7/d65/d61/d66/da4/d147 22 2024-04-05T21:26:48.976 INFO:tasks.workunit.client.1.smithi188.stdout:2/906: dread d5/db/dd/d73/fca [0,4194304] 0 2024-04-05T21:26:48.978 INFO:tasks.workunit.client.1.smithi188.stdout:8/737: truncate d5/d21/d2a/d47/f54 360718 0 2024-04-05T21:26:48.978 INFO:tasks.workunit.client.1.smithi188.stdout:1/927: write d1/dc/d16/d46/d88/d3f/f69 [4988038,109860] 0 2024-04-05T21:26:48.980 INFO:tasks.workunit.client.1.smithi188.stdout:3/985: read d5/f6 [533257,11439] 0 2024-04-05T21:26:48.985 INFO:tasks.workunit.client.1.smithi188.stdout:6/996: link d6/d13/d18/d19/dc8/da4/l134 d6/d13/d18/d19/d2c/d3f/d77/l136 0 2024-04-05T21:26:48.986 INFO:tasks.workunit.client.1.smithi188.stdout:1/928: write d1/dc/d16/d46/feb [860986,88821] 0 2024-04-05T21:26:48.986 INFO:tasks.workunit.client.1.smithi188.stdout:8/738: creat d5/d21/d40/d43/d44/d59/d8b/ff3 x:0 0 0 2024-04-05T21:26:48.988 INFO:tasks.workunit.client.1.smithi188.stdout:8/739: chown d5/d21/d2a/d71/lab 4684 1 2024-04-05T21:26:48.992 INFO:tasks.workunit.client.1.smithi188.stdout:6/997: rename d6/d13/d18/d19/d2c/f3e to d6/d13/d18/d19/d2c/f137 0 2024-04-05T21:26:48.993 INFO:tasks.workunit.client.1.smithi188.stdout:8/740: symlink d5/d21/d40/d43/d44/d59/d8b/lf4 0 2024-04-05T21:26:48.994 INFO:tasks.workunit.client.0.smithi121.stdout:6/584: rename d1/d2/d3/d5 to d1/d2/daf/d6f/dc7 0 2024-04-05T21:26:48.994 INFO:tasks.workunit.client.1.smithi188.stdout:3/986: getdents d5/d7/d65/d61/d66/d6e 0 2024-04-05T21:26:48.995 INFO:tasks.workunit.client.0.smithi121.stdout:6/585: mknod d1/d2/d1b/d91/da3/cc8 0 2024-04-05T21:26:48.996 INFO:tasks.workunit.client.1.smithi188.stdout:1/929: symlink d1/dc/d16/d46/d88/l129 0 2024-04-05T21:26:48.996 INFO:tasks.workunit.client.1.smithi188.stdout:0/842: truncate d4/d8/db/d21/d4a/d73/f99 7499546 0 2024-04-05T21:26:48.997 INFO:tasks.workunit.client.0.smithi121.stdout:6/586: creat d1/d2/d1b/d35/d39/fc9 x:0 0 0 2024-04-05T21:26:48.999 INFO:tasks.workunit.client.1.smithi188.stdout:0/843: write d4/d8/fb4 [3032222,71228] 0 2024-04-05T21:26:48.999 INFO:tasks.workunit.client.0.smithi121.stdout:6/587: unlink d1/d2/daf/d6f/dc7/d1f/f27 0 2024-04-05T21:26:49.002 INFO:tasks.workunit.client.1.smithi188.stdout:9/993: dread dd/d11/d1f/d26/dc0/d6b/d40/d59/fdc [0,4194304] 0 2024-04-05T21:26:49.005 INFO:tasks.workunit.client.0.smithi121.stdout:6/588: dwrite d1/f72 [0,4194304] 0 2024-04-05T21:26:49.005 INFO:tasks.workunit.client.1.smithi188.stdout:3/987: creat d5/d7/d6d/dd8/f148 x:0 0 0 2024-04-05T21:26:49.006 INFO:tasks.workunit.client.1.smithi188.stdout:8/741: dwrite d5/dc/f3b [0,4194304] 0 2024-04-05T21:26:49.008 INFO:tasks.workunit.client.1.smithi188.stdout:0/844: mkdir d4/d8/db/d42/da2/d138 0 2024-04-05T21:26:49.008 INFO:tasks.workunit.client.1.smithi188.stdout:8/742: readlink d5/laf 0 2024-04-05T21:26:49.012 INFO:tasks.workunit.client.1.smithi188.stdout:2/907: write d5/db/dd/f15 [6820121,73826] 0 2024-04-05T21:26:49.017 INFO:tasks.workunit.client.1.smithi188.stdout:8/743: creat d5/d21/d40/d43/d44/d5b/d99/d69/ff5 x:0 0 0 2024-04-05T21:26:49.018 INFO:tasks.workunit.client.1.smithi188.stdout:8/744: write d5/d4b/fce [1188349,115548] 0 2024-04-05T21:26:49.021 INFO:tasks.workunit.client.1.smithi188.stdout:0/845: rename d4/d8/db/d21/d4a/d73/d116/dd1/f10d to d4/d6/df4/f139 0 2024-04-05T21:26:49.022 INFO:tasks.workunit.client.1.smithi188.stdout:9/994: link dd/d3d/d81/fc9 dd/d3d/d108/f14a 0 2024-04-05T21:26:49.024 INFO:tasks.workunit.client.1.smithi188.stdout:6/998: dread d6/d13/d18/d19/d2c/f92 [0,4194304] 0 2024-04-05T21:26:49.027 INFO:tasks.workunit.client.1.smithi188.stdout:2/908: truncate d5/db/dd/d36/d3c/d48/dcf/f112 781765 0 2024-04-05T21:26:49.027 INFO:tasks.workunit.client.1.smithi188.stdout:0/846: truncate d4/d8/db/d21/d4a/d73/d116/dd1/ff9 509874 0 2024-04-05T21:26:49.030 INFO:tasks.workunit.client.1.smithi188.stdout:0/847: dread - d4/d8/d78/d9d/ffb zero size 2024-04-05T21:26:49.030 INFO:tasks.workunit.client.0.smithi121.stdout:6/589: dread d1/d2/dc/d9a/dc0/f2d [0,4194304] 0 2024-04-05T21:26:49.031 INFO:tasks.workunit.client.1.smithi188.stdout:1/930: truncate d1/dc/f1f 1467482 0 2024-04-05T21:26:49.031 INFO:tasks.workunit.client.0.smithi121.stdout:6/590: truncate d1/d2/d1b/d35/f47 792575 0 2024-04-05T21:26:49.032 INFO:tasks.workunit.client.1.smithi188.stdout:9/995: link dd/d11/d1f/d26/d55/d98/f136 dd/d11/d1f/d26/d55/dcf/f14b 0 2024-04-05T21:26:49.034 INFO:tasks.workunit.client.0.smithi121.stdout:6/591: symlink d1/d2/d1b/d2e/lca 0 2024-04-05T21:26:49.035 INFO:tasks.workunit.client.1.smithi188.stdout:3/988: write d5/d7/d65/d5f/fb4 [4239078,44300] 0 2024-04-05T21:26:49.035 INFO:tasks.workunit.client.0.smithi121.stdout:6/592: mkdir d1/d4b/dcb 0 2024-04-05T21:26:49.036 INFO:tasks.workunit.client.0.smithi121.stdout:6/593: chown d1/d2/d3/d4/d21/d57/d42/d48/d7c/db3/cc5 33463292 1 2024-04-05T21:26:49.036 INFO:tasks.workunit.client.1.smithi188.stdout:0/848: mkdir d4/d6/d13/da1/d13a 0 2024-04-05T21:26:49.037 INFO:tasks.workunit.client.1.smithi188.stdout:2/909: mkdir d5/db/dd/d73/d120/d12c 0 2024-04-05T21:26:49.041 INFO:tasks.workunit.client.1.smithi188.stdout:1/931: symlink d1/l12a 0 2024-04-05T21:26:49.045 INFO:tasks.workunit.client.1.smithi188.stdout:6/999: dread f5 [0,4194304] 0 2024-04-05T21:26:49.048 INFO:tasks.workunit.client.0.smithi121.stdout:6/594: dread d1/d2/daf/d6f/dc7/d1f/f46 [0,4194304] 0 2024-04-05T21:26:49.050 INFO:tasks.workunit.client.0.smithi121.stdout:6/595: mkdir d1/d4b/dcc 0 2024-04-05T21:26:49.051 INFO:tasks.workunit.client.1.smithi188.stdout:3/989: dwrite d5/d7/d65/fbe [0,4194304] 0 2024-04-05T21:26:49.051 INFO:tasks.workunit.client.1.smithi188.stdout:2/910: write d5/db/dd/d73/d109/d9f/d10c/dcd/faf [216601,39599] 0 2024-04-05T21:26:49.056 INFO:tasks.workunit.client.1.smithi188.stdout:2/911: chown d5/db/d35/d3b/d6b 7 1 2024-04-05T21:26:49.058 INFO:tasks.workunit.client.1.smithi188.stdout:0/849: dwrite d4/d8/db/d21/f7e [4194304,4194304] 0 2024-04-05T21:26:49.059 INFO:tasks.workunit.client.1.smithi188.stdout:1/932: rename d1/dc/d16/d46/d88/d13/fc3 to d1/dc/d16/d21/d27/f12b 0 2024-04-05T21:26:49.060 INFO:tasks.workunit.client.1.smithi188.stdout:8/745: dwrite d5/f12 [0,4194304] 0 2024-04-05T21:26:49.060 INFO:tasks.workunit.client.0.smithi121.stdout:6/596: truncate d1/d2/dc/d9a/dc0/f2a 3677897 0 2024-04-05T21:26:49.061 INFO:tasks.workunit.client.1.smithi188.stdout:3/990: stat d5/d7/d65/d61/d66/da4/de3/df6/f103 0 2024-04-05T21:26:49.063 INFO:tasks.workunit.client.0.smithi121.stdout:6/597: rename d1/d2/d1b/d35/f62 to d1/d2/d1b/d35/d39/fcd 0 2024-04-05T21:26:49.064 INFO:tasks.workunit.client.1.smithi188.stdout:3/991: readlink d5/d7/d65/d61/d66/d6e/l85 0 2024-04-05T21:26:49.065 INFO:tasks.workunit.client.0.smithi121.stdout:6/598: mknod d1/d2/d1b/d91/da3/cce 0 2024-04-05T21:26:49.068 INFO:tasks.workunit.client.0.smithi121.stdout:6/599: creat d1/d2/daf/d6f/dc7/d1f/d6d/fcf x:0 0 0 2024-04-05T21:26:49.071 INFO:tasks.workunit.client.0.smithi121.stdout:6/600: creat d1/d2/daf/d6f/dc7/d1f/fd0 x:0 0 0 2024-04-05T21:26:49.072 INFO:tasks.workunit.client.1.smithi188.stdout:1/933: mkdir d1/dc/d16/d46/d95/d12c 0 2024-04-05T21:26:49.072 INFO:tasks.workunit.client.1.smithi188.stdout:3/992: fdatasync d5/d7e/fe1 0 2024-04-05T21:26:49.072 INFO:tasks.workunit.client.1.smithi188.stdout:0/850: rename d4/d8/d39/fd0 to d4/d8/db/d21/d4a/d73/d116/d24/f13b 0 2024-04-05T21:26:49.072 INFO:tasks.workunit.client.1.smithi188.stdout:8/746: dread d5/d21/d40/d43/d44/d5b/d99/d69/fb3 [0,4194304] 0 2024-04-05T21:26:49.073 INFO:tasks.workunit.client.0.smithi121.stdout:6/601: write d1/d2/d3/d4/f88 [1015748,59324] 0 2024-04-05T21:26:49.073 INFO:tasks.workunit.client.1.smithi188.stdout:1/934: creat d1/dc/d16/d21/d27/d4a/dc1/f12d x:0 0 0 2024-04-05T21:26:49.074 INFO:tasks.workunit.client.1.smithi188.stdout:0/851: truncate d4/d8/d10/f10e 210464 0 2024-04-05T21:26:49.077 INFO:tasks.workunit.client.1.smithi188.stdout:3/993: link d5/d7/d31/f5c d5/ddd/d136/f149 0 2024-04-05T21:26:49.079 INFO:tasks.workunit.client.1.smithi188.stdout:1/935: rmdir d1/dc/d16/d21/d27 39 2024-04-05T21:26:49.082 INFO:tasks.workunit.client.1.smithi188.stdout:0/852: creat d4/d8/db/d21/d4a/d73/d116/d24/d101/d135/f13c x:0 0 0 2024-04-05T21:26:49.085 INFO:tasks.workunit.client.1.smithi188.stdout:3/994: creat d5/d7e/d101/de5/f14a x:0 0 0 2024-04-05T21:26:49.085 INFO:tasks.workunit.client.0.smithi121.stdout:6/602: write d1/d2/daf/d6f/dc7/fac [4385631,97411] 0 2024-04-05T21:26:49.086 INFO:tasks.workunit.client.1.smithi188.stdout:8/747: link d5/d21/d40/d43/f51 d5/d21/d40/d43/d44/d5b/d99/d69/ff6 0 2024-04-05T21:26:49.086 INFO:tasks.workunit.client.1.smithi188.stdout:1/936: chown d1/dc/d16/d46/d95/cc0 36 1 2024-04-05T21:26:49.088 INFO:tasks.workunit.client.1.smithi188.stdout:0/853: dread - d4/d6/d13/dea/ff7 zero size 2024-04-05T21:26:49.088 INFO:tasks.workunit.client.1.smithi188.stdout:3/995: creat d5/d7/d65/d61/d66/da4/de3/df6/f14b x:0 0 0 2024-04-05T21:26:49.089 INFO:tasks.workunit.client.0.smithi121.stdout:6/603: dread d1/d2/d1b/d5e/fba [0,4194304] 0 2024-04-05T21:26:49.089 INFO:tasks.workunit.client.1.smithi188.stdout:8/748: unlink d5/dc/l30 0 2024-04-05T21:26:49.093 INFO:tasks.workunit.client.1.smithi188.stdout:3/996: symlink d5/l14c 0 2024-04-05T21:26:49.093 INFO:tasks.workunit.client.1.smithi188.stdout:0/854: creat d4/d6/d13/da1/f13d x:0 0 0 2024-04-05T21:26:49.094 INFO:tasks.workunit.client.1.smithi188.stdout:2/912: dread d5/f2f [0,4194304] 0 2024-04-05T21:26:49.095 INFO:tasks.workunit.client.1.smithi188.stdout:1/937: creat d1/dc/d16/d21/d27/d4a/f12e x:0 0 0 2024-04-05T21:26:49.099 INFO:tasks.workunit.client.0.smithi121.stdout:6/604: dwrite d1/d2/daf/d6f/dc7/d1f/f64 [0,4194304] 0 2024-04-05T21:26:49.100 INFO:tasks.workunit.client.0.smithi121.stdout:6/605: chown d1/d2/d1b/f9e 835 1 2024-04-05T21:26:49.101 INFO:tasks.workunit.client.1.smithi188.stdout:1/938: symlink d1/dc/d16/d46/l12f 0 2024-04-05T21:26:49.101 INFO:tasks.workunit.client.1.smithi188.stdout:0/855: write d4/d8/db/d21/d4a/d73/d116/d24/d101/f114 [93097,29093] 0 2024-04-05T21:26:49.102 INFO:tasks.workunit.client.1.smithi188.stdout:8/749: dwrite d5/d21/d2a/d95/dc0/fc9 [4194304,4194304] 0 2024-04-05T21:26:49.104 INFO:tasks.workunit.client.1.smithi188.stdout:3/997: creat d5/d7/d6d/d75/d82/f14d x:0 0 0 2024-04-05T21:26:49.104 INFO:tasks.workunit.client.1.smithi188.stdout:3/998: chown d5/c9d 3825246 1 2024-04-05T21:26:49.107 INFO:tasks.workunit.client.0.smithi121.stdout:6/606: dwrite d1/d4b/f80 [0,4194304] 0 2024-04-05T21:26:49.109 INFO:tasks.workunit.client.0.smithi121.stdout:6/607: rmdir d1/d2/d1b 39 2024-04-05T21:26:49.111 INFO:tasks.workunit.client.0.smithi121.stdout:6/608: mknod d1/d4b/cd1 0 2024-04-05T21:26:49.112 INFO:tasks.workunit.client.0.smithi121.stdout:6/609: truncate d1/f38 983461 0 2024-04-05T21:26:49.113 INFO:tasks.workunit.client.0.smithi121.stdout:6/610: chown d1/d2/d1b/d91/da3 223628 1 2024-04-05T21:26:49.114 INFO:tasks.workunit.client.1.smithi188.stdout:1/939: chown d1/dc/d16/d21/dbb 173043 1 2024-04-05T21:26:49.115 INFO:tasks.workunit.client.1.smithi188.stdout:3/999: mknod d5/d7/d65/d61/d66/d6e/c14e 0 2024-04-05T21:26:49.116 INFO:tasks.workunit.client.0.smithi121.stdout:2/640: sync 2024-04-05T21:26:49.116 INFO:tasks.workunit.client.0.smithi121.stdout:6/611: creat d1/d2/d1b/fd2 x:0 0 0 2024-04-05T21:26:49.117 INFO:tasks.workunit.client.0.smithi121.stdout:6/612: stat d1/d2/d1b/d35/d39/fcd 0 2024-04-05T21:26:49.118 INFO:tasks.workunit.client.1.smithi188.stdout:0/856: mkdir d4/d6/d46/dba/d13e 0 2024-04-05T21:26:49.119 INFO:tasks.workunit.client.0.smithi121.stdout:2/641: dread d2/d9/d38/d2d/f58 [0,4194304] 0 2024-04-05T21:26:49.119 INFO:tasks.workunit.client.1.smithi188.stdout:8/750: symlink d5/d6e/lf7 0 2024-04-05T21:26:49.119 INFO:tasks.workunit.client.0.smithi121.stdout:2/642: chown d2/d9/d1c/d30/d59 12 1 2024-04-05T21:26:49.122 INFO:tasks.workunit.client.0.smithi121.stdout:2/643: symlink d2/d9/d1c/d30/dba/lc7 0 2024-04-05T21:26:49.123 INFO:tasks.workunit.client.1.smithi188.stdout:0/857: symlink d4/d6/d13/da1/d13a/l13f 0 2024-04-05T21:26:49.124 INFO:tasks.workunit.client.1.smithi188.stdout:8/751: mkdir d5/d6e/d86/dcb/df8 0 2024-04-05T21:26:49.127 INFO:tasks.workunit.client.1.smithi188.stdout:8/752: rmdir d5/d21/d2a/d71 39 2024-04-05T21:26:49.128 INFO:tasks.workunit.client.1.smithi188.stdout:0/858: getdents d4/d6/d13/da1/d130 0 2024-04-05T21:26:49.130 INFO:tasks.workunit.client.1.smithi188.stdout:8/753: read - d5/d21/d80/f96 zero size 2024-04-05T21:26:49.134 INFO:tasks.workunit.client.1.smithi188.stdout:0/859: rename d4/d8/db/d21/d4a/d73/d116/d24/f13b to d4/d6/d13/dca/d125/d12a/f140 0 2024-04-05T21:26:49.136 INFO:tasks.workunit.client.0.smithi121.stdout:2/644: dread d2/d9/d17/fa9 [0,4194304] 0 2024-04-05T21:26:49.137 INFO:tasks.workunit.client.1.smithi188.stdout:2/913: dwrite d5/db/dd/fd4 [0,4194304] 0 2024-04-05T21:26:49.139 INFO:tasks.workunit.client.1.smithi188.stdout:0/860: mknod d4/d6/d13/dca/d125/c141 0 2024-04-05T21:26:49.141 INFO:tasks.workunit.client.1.smithi188.stdout:0/861: mknod d4/d8/d39/c142 0 2024-04-05T21:26:49.142 INFO:tasks.workunit.client.1.smithi188.stdout:1/940: write d1/dc/d16/d46/d88/d3f/f45 [1212914,74941] 0 2024-04-05T21:26:49.146 INFO:tasks.workunit.client.1.smithi188.stdout:2/914: creat d5/db/dd/d73/d109/d9f/f12d x:0 0 0 2024-04-05T21:26:49.146 INFO:tasks.workunit.client.1.smithi188.stdout:8/754: write d5/d21/d40/d43/d44/d5b/d99/da3/fba [2648148,50104] 0 2024-04-05T21:26:49.147 INFO:tasks.workunit.client.0.smithi121.stdout:6/613: dread d1/d32/fae [0,4194304] 0 2024-04-05T21:26:49.147 INFO:tasks.workunit.client.1.smithi188.stdout:2/915: stat d5/db/d35/d105 0 2024-04-05T21:26:49.149 INFO:tasks.workunit.client.1.smithi188.stdout:1/941: dread d1/dc/d16/d46/de5/f10b [0,4194304] 0 2024-04-05T21:26:49.151 INFO:tasks.workunit.client.1.smithi188.stdout:0/862: creat d4/d8/db/d42/d38/d77/f143 x:0 0 0 2024-04-05T21:26:49.151 INFO:tasks.workunit.client.1.smithi188.stdout:8/755: write d5/d21/d40/d43/d44/d5b/d99/d69/fe3 [214733,115938] 0 2024-04-05T21:26:49.152 INFO:tasks.workunit.client.1.smithi188.stdout:8/756: readlink d5/d6e/lf7 0 2024-04-05T21:26:49.152 INFO:tasks.workunit.client.0.smithi121.stdout:6/614: dwrite d1/f1c [0,4194304] 0 2024-04-05T21:26:49.153 INFO:tasks.workunit.client.1.smithi188.stdout:8/757: chown d5/d21/d40/d43/d44/d59/d8b/da4 251265985 1 2024-04-05T21:26:49.154 INFO:tasks.workunit.client.1.smithi188.stdout:2/916: dwrite d5/fe7 [0,4194304] 0 2024-04-05T21:26:49.156 INFO:tasks.workunit.client.0.smithi121.stdout:6/615: creat d1/d2/d1b/d35/d39/d92/fd3 x:0 0 0 2024-04-05T21:26:49.158 INFO:tasks.workunit.client.0.smithi121.stdout:5/790: sync 2024-04-05T21:26:49.158 INFO:tasks.workunit.client.0.smithi121.stdout:0/809: sync 2024-04-05T21:26:49.158 INFO:tasks.workunit.client.0.smithi121.stdout:7/776: sync 2024-04-05T21:26:49.158 INFO:tasks.workunit.client.0.smithi121.stdout:4/860: sync 2024-04-05T21:26:49.158 INFO:tasks.workunit.client.0.smithi121.stdout:9/554: sync 2024-04-05T21:26:49.158 INFO:tasks.workunit.client.0.smithi121.stdout:1/762: sync 2024-04-05T21:26:49.158 INFO:tasks.workunit.client.0.smithi121.stdout:3/585: sync 2024-04-05T21:26:49.159 INFO:tasks.workunit.client.0.smithi121.stdout:8/707: sync 2024-04-05T21:26:49.159 INFO:tasks.workunit.client.1.smithi188.stdout:2/917: fsync d5/db/d35/d3b/d6b/fa7 0 2024-04-05T21:26:49.161 INFO:tasks.workunit.client.1.smithi188.stdout:8/758: chown d5/d21/d40/d43/d44/d5b/d99/d69/d70/dae/fc2 19663 1 2024-04-05T21:26:49.161 INFO:tasks.workunit.client.1.smithi188.stdout:2/918: dread - d5/db/dd/d36/d3c/d48/d7a/d7c/db5/f125 zero size 2024-04-05T21:26:49.163 INFO:tasks.workunit.client.1.smithi188.stdout:2/919: mkdir d5/db/dd/d73/d109/d66/d8f/d12e 0 2024-04-05T21:26:49.164 INFO:tasks.workunit.client.1.smithi188.stdout:8/759: truncate d5/d21/f91 4808002 0 2024-04-05T21:26:49.164 INFO:tasks.workunit.client.1.smithi188.stdout:2/920: mkdir d5/d5f/d12f 0 2024-04-05T21:26:49.165 INFO:tasks.workunit.client.1.smithi188.stdout:8/760: readlink d5/d21/d40/d43/d44/d59/d8b/da4/db1/le6 0 2024-04-05T21:26:49.166 INFO:tasks.workunit.client.0.smithi121.stdout:6/616: unlink d1/d2/d3/d4/d21/d57/d42/d48/c6c 0 2024-04-05T21:26:49.166 INFO:tasks.workunit.client.0.smithi121.stdout:6/617: chown d1/d2/d3/d4/d21 80617 1 2024-04-05T21:26:49.166 INFO:tasks.workunit.client.0.smithi121.stdout:8/708: chown d12/d13/d15/d1d/l46 2835 1 2024-04-05T21:26:49.166 INFO:tasks.workunit.client.0.smithi121.stdout:0/810: rmdir d7/d71/d29/d2a 39 2024-04-05T21:26:49.166 INFO:tasks.workunit.client.0.smithi121.stdout:2/645: dread d2/d9/f21 [0,4194304] 0 2024-04-05T21:26:49.168 INFO:tasks.workunit.client.0.smithi121.stdout:1/763: mknod d8/d8c/da2/cff 0 2024-04-05T21:26:49.168 INFO:tasks.workunit.client.0.smithi121.stdout:6/618: truncate d1/d2/d1b/fd2 543700 0 2024-04-05T21:26:49.168 INFO:tasks.workunit.client.1.smithi188.stdout:2/921: write d5/db/dd/d36/d3c/d48/dd7/d10b/f114 [363358,40401] 0 2024-04-05T21:26:49.168 INFO:tasks.workunit.client.1.smithi188.stdout:8/761: mknod d5/d6e/d98/cf9 0 2024-04-05T21:26:49.168 INFO:tasks.workunit.client.1.smithi188.stdout:8/762: chown d5/d21 1 1 2024-04-05T21:26:49.169 INFO:tasks.workunit.client.0.smithi121.stdout:9/555: rename d6/d36 to d6/d32/dbe 0 2024-04-05T21:26:49.170 INFO:tasks.workunit.client.1.smithi188.stdout:2/922: mknod d5/db/dd/d73/d120/d12c/c130 0 2024-04-05T21:26:49.171 INFO:tasks.workunit.client.0.smithi121.stdout:3/586: mknod d0/d3/de/cbe 0 2024-04-05T21:26:49.172 INFO:tasks.workunit.client.1.smithi188.stdout:8/763: dread d5/f12 [0,4194304] 0 2024-04-05T21:26:49.172 INFO:tasks.workunit.client.0.smithi121.stdout:0/811: creat d7/d62/d9d/f11f x:0 0 0 2024-04-05T21:26:49.174 INFO:tasks.workunit.client.0.smithi121.stdout:0/812: stat d7/d62/d8b/f116 0 2024-04-05T21:26:49.175 INFO:tasks.workunit.client.0.smithi121.stdout:6/619: symlink d1/d2/daf/d6f/ld4 0 2024-04-05T21:26:49.177 INFO:tasks.workunit.client.0.smithi121.stdout:0/813: write d7/dc/da3/fc1 [29817,36987] 0 2024-04-05T21:26:49.177 INFO:tasks.workunit.client.0.smithi121.stdout:6/620: chown d1/d2/d1b/d5e/c85 110770 1 2024-04-05T21:26:49.178 INFO:tasks.workunit.client.0.smithi121.stdout:3/587: mknod d0/d3/d24/d3d/d1a/d1e/d86/cbf 0 2024-04-05T21:26:49.180 INFO:tasks.workunit.client.1.smithi188.stdout:2/923: link d5/db/dd/c13 d5/db/dd/d73/d109/d5d/c131 0 2024-04-05T21:26:49.180 INFO:tasks.workunit.client.0.smithi121.stdout:3/588: write d0/d3/f4 [2671,113363] 0 2024-04-05T21:26:49.184 INFO:tasks.workunit.client.0.smithi121.stdout:0/814: unlink d7/d1c/cb2 0 2024-04-05T21:26:49.185 INFO:tasks.workunit.client.0.smithi121.stdout:3/589: truncate d0/d3/d24/d3d/d1a/d1e/d86/d8e/fb6 394579 0 2024-04-05T21:26:49.187 INFO:tasks.workunit.client.0.smithi121.stdout:4/861: getdents d5/d11/d28/d3b 0 2024-04-05T21:26:49.190 INFO:tasks.workunit.client.0.smithi121.stdout:5/791: getdents dd/de/d19/d1c/d7b/d88 0 2024-04-05T21:26:49.190 INFO:tasks.workunit.client.0.smithi121.stdout:6/621: dwrite d1/d2/d1b/d35/d74/f7d [0,4194304] 0 2024-04-05T21:26:49.191 INFO:tasks.workunit.client.0.smithi121.stdout:3/590: chown d0/d3/d24/d3d/cf 1872 1 2024-04-05T21:26:49.191 INFO:tasks.workunit.client.1.smithi188.stdout:2/924: rename d5/db/dd/d73 to d5/db/dd/d132 0 2024-04-05T21:26:49.192 INFO:tasks.workunit.client.0.smithi121.stdout:5/792: chown dd/de/d39/d3b/de8/d103 0 1 2024-04-05T21:26:49.193 INFO:tasks.workunit.client.0.smithi121.stdout:0/815: dread d7/d71/d29/d2a/d2c/f7b [0,4194304] 0 2024-04-05T21:26:49.195 INFO:tasks.workunit.client.0.smithi121.stdout:8/709: dread d12/d50/d61/f3a [0,4194304] 0 2024-04-05T21:26:49.195 INFO:tasks.workunit.client.0.smithi121.stdout:3/591: creat d0/fc0 x:0 0 0 2024-04-05T21:26:49.196 INFO:tasks.workunit.client.0.smithi121.stdout:0/816: chown d7/d71/d45/lca 2 1 2024-04-05T21:26:49.197 INFO:tasks.workunit.client.1.smithi188.stdout:8/764: dread d5/d21/d40/d43/d44/d5b/d99/d69/d70/fee [0,4194304] 0 2024-04-05T21:26:49.197 INFO:tasks.workunit.client.0.smithi121.stdout:5/793: stat dd/de/d19/d13/dd4/f114 0 2024-04-05T21:26:49.197 INFO:tasks.workunit.client.0.smithi121.stdout:6/622: creat d1/d2/d3/d59/fd5 x:0 0 0 2024-04-05T21:26:49.198 INFO:tasks.workunit.client.0.smithi121.stdout:0/817: chown d7/d62/ddb 13014 1 2024-04-05T21:26:49.199 INFO:tasks.workunit.client.1.smithi188.stdout:2/925: dread d5/db/dd/d24/f52 [4194304,4194304] 0 2024-04-05T21:26:49.200 INFO:tasks.workunit.client.0.smithi121.stdout:3/592: mknod d0/d3/d24/d3d/cc1 0 2024-04-05T21:26:49.200 INFO:tasks.workunit.client.0.smithi121.stdout:2/646: write d2/d9/d1c/d30/f5e [885094,84326] 0 2024-04-05T21:26:49.201 INFO:tasks.workunit.client.1.smithi188.stdout:2/926: fsync d5/db/dd/f95 0 2024-04-05T21:26:49.202 INFO:tasks.workunit.client.0.smithi121.stdout:1/764: write d8/d8c/db6/d1a/d1b/d4c/d76/d53/f9b [704640,11050] 0 2024-04-05T21:26:49.203 INFO:tasks.workunit.client.1.smithi188.stdout:2/927: readlink d5/db/dd/d36/d3c/d48/dcf/lfd 0 2024-04-05T21:26:49.204 INFO:tasks.workunit.client.0.smithi121.stdout:5/794: chown dd/de/d19/d13/f3f 2331 1 2024-04-05T21:26:49.204 INFO:tasks.workunit.client.0.smithi121.stdout:0/818: write d7/dc/da3/fc1 [869609,86594] 0 2024-04-05T21:26:49.206 INFO:tasks.workunit.client.1.smithi188.stdout:2/928: creat d5/db/dd/d132/d109/d9f/d10c/f133 x:0 0 0 2024-04-05T21:26:49.207 INFO:tasks.workunit.client.0.smithi121.stdout:2/647: chown d2/d9/d56/d5c 0 1 2024-04-05T21:26:49.208 INFO:tasks.workunit.client.0.smithi121.stdout:4/862: write d5/d11/d28/fe [521672,67260] 0 2024-04-05T21:26:49.208 INFO:tasks.workunit.client.0.smithi121.stdout:6/623: rename d1/d2/d1b/d35/d39/d92/lc2 to d1/d4b/da7/ld6 0 2024-04-05T21:26:49.209 INFO:tasks.workunit.client.1.smithi188.stdout:2/929: creat d5/db/dd/d36/d3c/d48/d7a/d7c/db5/f134 x:0 0 0 2024-04-05T21:26:49.210 INFO:tasks.workunit.client.0.smithi121.stdout:7/777: dwrite d0/d9/d22/d28/d56/f64 [0,4194304] 0 2024-04-05T21:26:49.210 INFO:tasks.workunit.client.0.smithi121.stdout:3/593: rename d0/fc0 to d0/fc2 0 2024-04-05T21:26:49.212 INFO:tasks.workunit.client.1.smithi188.stdout:2/930: creat d5/db/dd/d132/d109/d5d/f135 x:0 0 0 2024-04-05T21:26:49.212 INFO:tasks.workunit.client.0.smithi121.stdout:7/778: stat d0/d57/ded 0 2024-04-05T21:26:49.213 INFO:tasks.workunit.client.0.smithi121.stdout:0/819: creat d7/d7e/d97/f120 x:0 0 0 2024-04-05T21:26:49.213 INFO:tasks.workunit.client.0.smithi121.stdout:5/795: creat dd/de/d19/d1c/d7b/f11b x:0 0 0 2024-04-05T21:26:49.215 INFO:tasks.workunit.client.0.smithi121.stdout:2/648: mknod d2/d9/d17/cc8 0 2024-04-05T21:26:49.216 INFO:tasks.workunit.client.0.smithi121.stdout:1/765: readlink d8/d8c/db6/d1a/d1b/l88 0 2024-04-05T21:26:49.216 INFO:tasks.workunit.client.0.smithi121.stdout:5/796: fdatasync dd/de/d19/d13/f16 0 2024-04-05T21:26:49.217 INFO:tasks.workunit.client.0.smithi121.stdout:6/624: creat d1/d2/daf/d6f/fd7 x:0 0 0 2024-04-05T21:26:49.218 INFO:tasks.workunit.client.1.smithi188.stdout:8/765: dread d5/d21/d40/d43/f51 [0,4194304] 0 2024-04-05T21:26:49.220 INFO:tasks.workunit.client.0.smithi121.stdout:0/820: symlink d7/d71/d94/l121 0 2024-04-05T21:26:49.221 INFO:tasks.workunit.client.1.smithi188.stdout:2/931: write d5/db/d35/d3b/d6b/f7e [1996871,112359] 0 2024-04-05T21:26:49.222 INFO:tasks.workunit.client.0.smithi121.stdout:4/863: truncate d5/d11/d26/d47/df7/d110/f94 5545965 0 2024-04-05T21:26:49.223 INFO:tasks.workunit.client.1.smithi188.stdout:2/932: stat d5/db/d35/d105 0 2024-04-05T21:26:49.225 INFO:tasks.workunit.client.0.smithi121.stdout:2/649: creat d2/d9/d38/d2d/d4c/d7c/dc2/fc9 x:0 0 0 2024-04-05T21:26:49.226 INFO:tasks.workunit.client.0.smithi121.stdout:7/779: dwrite d0/d9/d22/fa3 [0,4194304] 0 2024-04-05T21:26:49.226 INFO:tasks.workunit.client.1.smithi188.stdout:8/766: dwrite d5/d21/d40/d43/d48/d7e/fe4 [0,4194304] 0 2024-04-05T21:26:49.228 INFO:tasks.workunit.client.0.smithi121.stdout:5/797: symlink dd/d41/dce/da6/dd3/l11c 0 2024-04-05T21:26:49.229 INFO:tasks.workunit.client.0.smithi121.stdout:0/821: creat d7/dc/d95/f122 x:0 0 0 2024-04-05T21:26:49.232 INFO:tasks.workunit.client.1.smithi188.stdout:2/933: symlink d5/db/dd/d36/d3c/d48/l136 0 2024-04-05T21:26:49.232 INFO:tasks.workunit.client.1.smithi188.stdout:8/767: truncate d5/f8 4384327 0 2024-04-05T21:26:49.233 INFO:tasks.workunit.client.1.smithi188.stdout:8/768: read d5/d21/d40/d43/d44/d5b/d99/d69/d70/fee [2369734,102046] 0 2024-04-05T21:26:49.234 INFO:tasks.workunit.client.1.smithi188.stdout:8/769: chown d5/d21/d40/d43/d44/d59/d8b/lf4 1058187 1 2024-04-05T21:26:49.237 INFO:tasks.workunit.client.0.smithi121.stdout:4/864: creat d5/d11/d28/dc0/f116 x:0 0 0 2024-04-05T21:26:49.237 INFO:tasks.workunit.client.0.smithi121.stdout:0/822: fdatasync d7/fdd 0 2024-04-05T21:26:49.237 INFO:tasks.workunit.client.0.smithi121.stdout:3/594: dread d0/d3/d24/f89 [0,4194304] 0 2024-04-05T21:26:49.237 INFO:tasks.workunit.client.0.smithi121.stdout:6/625: dwrite d1/d2/d1b/d35/d39/d92/f26 [0,4194304] 0 2024-04-05T21:26:49.240 INFO:tasks.workunit.client.0.smithi121.stdout:3/595: mkdir d0/d3/d24/db3/dc3 0 2024-04-05T21:26:49.241 INFO:tasks.workunit.client.0.smithi121.stdout:0/823: creat d7/dc/da3/df5/f123 x:0 0 0 2024-04-05T21:26:49.242 INFO:tasks.workunit.client.1.smithi188.stdout:8/770: read d5/d21/d2a/f97 [16226,45088] 0 2024-04-05T21:26:49.244 INFO:tasks.workunit.client.1.smithi188.stdout:8/771: chown d5/d21/d40/d43/d44/d5b/d99/d69/fe3 4022 1 2024-04-05T21:26:49.246 INFO:tasks.workunit.client.0.smithi121.stdout:3/596: truncate d0/d3/d24/d3d/d1a/d23/d38/d55/fac 1454557 0 2024-04-05T21:26:49.247 INFO:tasks.workunit.client.1.smithi188.stdout:8/772: mkdir d5/d21/d40/d43/d44/d5b/dfa 0 2024-04-05T21:26:49.247 INFO:tasks.workunit.client.0.smithi121.stdout:7/780: rename d0/d9/f26 to d0/ff4 0 2024-04-05T21:26:49.250 INFO:tasks.workunit.client.0.smithi121.stdout:0/824: dwrite d7/d62/d3c/d6b/f111 [0,4194304] 0 2024-04-05T21:26:49.252 INFO:tasks.workunit.client.0.smithi121.stdout:4/865: getdents d5/d11/d2c/da4 0 2024-04-05T21:26:49.255 INFO:tasks.workunit.client.0.smithi121.stdout:2/650: link d2/d9/d17/d3a/f4a d2/d9/d38/d2d/d4c/d7c/d7e/fca 0 2024-04-05T21:26:49.257 INFO:tasks.workunit.client.1.smithi188.stdout:2/934: dread d5/db/dd/f54 [0,4194304] 0 2024-04-05T21:26:49.258 INFO:tasks.workunit.client.0.smithi121.stdout:3/597: creat d0/d3/d24/d3d/d1a/d1e/d86/fc4 x:0 0 0 2024-04-05T21:26:49.259 INFO:tasks.workunit.client.0.smithi121.stdout:0/825: symlink d7/d71/d94/dd1/dd4/l124 0 2024-04-05T21:26:49.260 INFO:tasks.workunit.client.0.smithi121.stdout:6/626: truncate d1/d2/d1b/d5e/fba 2516464 0 2024-04-05T21:26:49.262 INFO:tasks.workunit.client.0.smithi121.stdout:4/866: mkdir d5/d45/d67/dab/d117 0 2024-04-05T21:26:49.264 INFO:tasks.workunit.client.0.smithi121.stdout:5/798: dwrite dd/de/d19/d38/d45/d68/f79 [0,4194304] 0 2024-04-05T21:26:49.267 INFO:tasks.workunit.client.0.smithi121.stdout:7/781: symlink d0/d9/d22/lf5 0 2024-04-05T21:26:49.271 INFO:tasks.workunit.client.0.smithi121.stdout:6/627: creat d1/d2/d1b/d35/d39/d92/fd8 x:0 0 0 2024-04-05T21:26:49.273 INFO:tasks.workunit.client.0.smithi121.stdout:7/782: rmdir d0/d9/d22/d81 39 2024-04-05T21:26:49.275 INFO:tasks.workunit.client.0.smithi121.stdout:5/799: fsync dd/de/d19/d38/d45/f4e 0 2024-04-05T21:26:49.277 INFO:tasks.workunit.client.0.smithi121.stdout:2/651: dwrite d2/d9/d38/d2d/d4c/d79/fb8 [0,4194304] 0 2024-04-05T21:26:49.281 INFO:tasks.workunit.client.0.smithi121.stdout:2/652: fsync d2/d9/d1c/fae 0 2024-04-05T21:26:49.282 INFO:tasks.workunit.client.0.smithi121.stdout:0/826: dwrite d7/d7e/d97/f120 [0,4194304] 0 2024-04-05T21:26:49.282 INFO:tasks.workunit.client.0.smithi121.stdout:6/628: truncate d1/d2/d1b/d35/d39/d92/fd3 729713 0 2024-04-05T21:26:49.284 INFO:tasks.workunit.client.0.smithi121.stdout:7/783: mknod d0/d9/d22/cf6 0 2024-04-05T21:26:49.286 INFO:tasks.workunit.client.0.smithi121.stdout:4/867: getdents d5/d11/d2c/d9d/dae/d29/d2e/dd3 0 2024-04-05T21:26:49.286 INFO:tasks.workunit.client.0.smithi121.stdout:5/800: creat dd/de/d19/d38/d45/d68/dcf/f11d x:0 0 0 2024-04-05T21:26:49.289 INFO:tasks.workunit.client.0.smithi121.stdout:5/801: chown dd/de/d19/d1c/fc2 27 1 2024-04-05T21:26:49.289 INFO:tasks.workunit.client.0.smithi121.stdout:2/653: write d2/d9/f21 [930506,15136] 0 2024-04-05T21:26:49.289 INFO:tasks.workunit.client.0.smithi121.stdout:0/827: creat d7/d71/d29/d2a/d2c/f125 x:0 0 0 2024-04-05T21:26:49.293 INFO:tasks.workunit.client.0.smithi121.stdout:6/629: rename d1/d2/d1b/d91/da3 to d1/d2/d1b/d2e/dd9 0 2024-04-05T21:26:49.293 INFO:tasks.workunit.client.0.smithi121.stdout:2/654: read d2/d9/d17/f19 [355930,7146] 0 2024-04-05T21:26:49.293 INFO:tasks.workunit.client.0.smithi121.stdout:6/630: stat d1/d4b/cc3 0 2024-04-05T21:26:49.293 INFO:tasks.workunit.client.0.smithi121.stdout:7/784: readlink d0/d32/d4a/lb7 0 2024-04-05T21:26:49.296 INFO:tasks.workunit.client.1.smithi188.stdout:2/935: dread d5/d5f/f90 [0,4194304] 0 2024-04-05T21:26:49.298 INFO:tasks.workunit.client.0.smithi121.stdout:7/785: mkdir d0/d9/d22/d3e/d76/df7 0 2024-04-05T21:26:49.299 INFO:tasks.workunit.client.1.smithi188.stdout:2/936: fsync d5/db/dd/d36/d3c/d48/d85/f94 0 2024-04-05T21:26:49.300 INFO:tasks.workunit.client.0.smithi121.stdout:2/655: mkdir d2/d9/dcb 0 2024-04-05T21:26:49.300 INFO:tasks.workunit.client.0.smithi121.stdout:2/656: fdatasync d2/fb0 0 2024-04-05T21:26:49.300 INFO:tasks.workunit.client.1.smithi188.stdout:2/937: symlink d5/db/dd/d132/d109/d9f/d10c/l137 0 2024-04-05T21:26:49.301 INFO:tasks.workunit.client.0.smithi121.stdout:7/786: creat d0/d57/ded/ff8 x:0 0 0 2024-04-05T21:26:49.302 INFO:tasks.workunit.client.0.smithi121.stdout:2/657: rmdir d2/d28 39 2024-04-05T21:26:49.305 INFO:tasks.workunit.client.0.smithi121.stdout:2/658: mknod d2/db6/dbb/ccc 0 2024-04-05T21:26:49.307 INFO:tasks.workunit.client.0.smithi121.stdout:7/787: dread d0/d9/d22/d28/d56/db9/ddf/def/d52/d61/dcb/fdc [0,4194304] 0 2024-04-05T21:26:49.308 INFO:tasks.workunit.client.1.smithi188.stdout:2/938: write d5/db/dd/d132/d109/d9f/fe8 [288909,113700] 0 2024-04-05T21:26:49.310 INFO:tasks.workunit.client.1.smithi188.stdout:2/939: mknod d5/db/dd/d132/d109/d9f/d10c/c138 0 2024-04-05T21:26:49.311 INFO:tasks.workunit.client.0.smithi121.stdout:2/659: link d2/d9/d38/d2d/f72 d2/d9/d1c/d30/fcd 0 2024-04-05T21:26:49.316 INFO:tasks.workunit.client.1.smithi188.stdout:2/940: dwrite d5/db/dd/d36/d3c/d63/d75/d87/f9c [0,4194304] 0 2024-04-05T21:26:49.319 INFO:tasks.workunit.client.1.smithi188.stdout:2/941: truncate d5/db/f76 1204218 0 2024-04-05T21:26:49.320 INFO:tasks.workunit.client.1.smithi188.stdout:2/942: readlink d5/db/dd/d132/lb2 0 2024-04-05T21:26:49.321 INFO:tasks.workunit.client.1.smithi188.stdout:2/943: unlink d5/l2a 0 2024-04-05T21:26:49.322 INFO:tasks.workunit.client.1.smithi188.stdout:2/944: readlink d5/d5f/l60 0 2024-04-05T21:26:49.323 INFO:tasks.workunit.client.1.smithi188.stdout:2/945: readlink d5/l1f 0 2024-04-05T21:26:49.326 INFO:tasks.workunit.client.1.smithi188.stdout:2/946: link d5/db/dd/d132/d109/faa d5/db/dd/d132/d120/d12c/f139 0 2024-04-05T21:26:49.329 INFO:tasks.workunit.client.1.smithi188.stdout:2/947: rename d5/db/dd/d132/fca to d5/db/d35/df8/f13a 0 2024-04-05T21:26:49.332 INFO:tasks.workunit.client.1.smithi188.stdout:2/948: rename d5/db/dd/l26 to d5/db/dd/d24/d116/l13b 0 2024-04-05T21:26:49.334 INFO:tasks.workunit.client.1.smithi188.stdout:2/949: mknod d5/db/dd/d36/d3c/d48/d7a/d7c/db5/c13c 0 2024-04-05T21:26:49.336 INFO:tasks.workunit.client.1.smithi188.stdout:2/950: symlink d5/db/d35/d3b/d6b/l13d 0 2024-04-05T21:26:49.336 INFO:tasks.workunit.client.1.smithi188.stdout:2/951: chown d5/db/dd/d132/d120/d12c/f139 0 1 2024-04-05T21:26:49.345 INFO:tasks.workunit.client.1.smithi188.stdout:2/952: write d5/db/dd/d132/d109/d9f/d10c/fad [2054756,106832] 0 2024-04-05T21:26:49.350 INFO:tasks.workunit.client.1.smithi188.stdout:2/953: write d5/db/dd/d132/d109/d66/fc9 [47506,20930] 0 2024-04-05T21:26:49.351 INFO:tasks.workunit.client.1.smithi188.stdout:2/954: dread - d5/fa3 zero size 2024-04-05T21:26:49.352 INFO:tasks.workunit.client.1.smithi188.stdout:2/955: write d5/db/dd/d132/d109/fbd [1319700,63744] 0 2024-04-05T21:26:49.353 INFO:tasks.workunit.client.1.smithi188.stdout:2/956: read - d5/db/dd/d132/d109/d8c/f11b zero size 2024-04-05T21:26:49.355 INFO:tasks.workunit.client.1.smithi188.stdout:2/957: fsync d5/db/dd/d132/d109/fc5 0 2024-04-05T21:26:49.356 INFO:tasks.workunit.client.1.smithi188.stdout:2/958: chown d5/db/d35/d3b/f4d 26901189 1 2024-04-05T21:26:49.358 INFO:tasks.workunit.client.1.smithi188.stdout:2/959: symlink d5/db/dd/d36/d3c/d48/l13e 0 2024-04-05T21:26:49.360 INFO:tasks.workunit.client.1.smithi188.stdout:2/960: symlink d5/db/d35/d3b/l13f 0 2024-04-05T21:26:49.362 INFO:tasks.workunit.client.1.smithi188.stdout:2/961: fsync d5/db/dd/d24/f28 0 2024-04-05T21:26:49.364 INFO:tasks.workunit.client.1.smithi188.stdout:2/962: rmdir d5/db/dd/d132/d109/d66/d8f 39 2024-04-05T21:26:49.367 INFO:tasks.workunit.client.1.smithi188.stdout:2/963: mkdir d5/d113/d140 0 2024-04-05T21:26:49.367 INFO:tasks.workunit.client.1.smithi188.stdout:2/964: stat d5/d5f/d98 0 2024-04-05T21:26:49.369 INFO:tasks.workunit.client.1.smithi188.stdout:2/965: mkdir d5/db/dd/d36/d3c/d63/d141 0 2024-04-05T21:26:49.371 INFO:tasks.workunit.client.1.smithi188.stdout:2/966: mkdir d5/db/dd/d132/d142 0 2024-04-05T21:26:49.371 INFO:tasks.workunit.client.0.smithi121.stdout:9/556: sync 2024-04-05T21:26:49.375 INFO:tasks.workunit.client.1.smithi188.stdout:2/967: link d5/db/dd/d132/d109/d9f/d10c/f8e d5/db/d35/df8/f143 0 2024-04-05T21:26:49.377 INFO:tasks.workunit.client.0.smithi121.stdout:2/660: write d2/d9/d38/d2d/d47/f70 [626098,91387] 0 2024-04-05T21:26:49.377 INFO:tasks.workunit.client.0.smithi121.stdout:7/788: dwrite d0/d23/f54 [0,4194304] 0 2024-04-05T21:26:49.378 INFO:tasks.workunit.client.1.smithi188.stdout:2/968: rename d5/db/dd/d36/d3c/d48/d85 to d5/db/dd/d132/d109/d9f/d10c/d144 0 2024-04-05T21:26:49.383 INFO:tasks.workunit.client.0.smithi121.stdout:2/661: symlink d2/d9/d38/d2d/d4c/d7c/d7e/lce 0 2024-04-05T21:26:49.386 INFO:tasks.workunit.client.0.smithi121.stdout:9/557: dwrite d6/df/d48/d96/db9/d2c/d37/d3c/f7c [0,4194304] 0 2024-04-05T21:26:49.389 INFO:tasks.workunit.client.0.smithi121.stdout:7/789: dwrite d0/d32/f83 [0,4194304] 0 2024-04-05T21:26:49.391 INFO:tasks.workunit.client.0.smithi121.stdout:9/558: truncate d6/d32/dbe/d7b/d6b/f8f 35598 0 2024-04-05T21:26:49.392 INFO:tasks.workunit.client.1.smithi188.stdout:2/969: write d5/db/dd/d36/d3c/d48/dcf/f112 [1287353,102049] 0 2024-04-05T21:26:49.395 INFO:tasks.workunit.client.0.smithi121.stdout:2/662: dread d2/d9/d1c/d30/f5e [0,4194304] 0 2024-04-05T21:26:49.398 INFO:tasks.workunit.client.1.smithi188.stdout:2/970: write d5/db/dd/d36/d3c/d48/dcf/f112 [1599874,92647] 0 2024-04-05T21:26:49.401 INFO:tasks.workunit.client.1.smithi188.stdout:2/971: write d5/db/dd/d132/d109/d66/d8f/fc1 [162706,3517] 0 2024-04-05T21:26:49.405 INFO:tasks.workunit.client.1.smithi188.stdout:2/972: creat d5/db/dd/d36/d3c/d48/d7a/d7c/f145 x:0 0 0 2024-04-05T21:26:49.406 INFO:tasks.workunit.client.0.smithi121.stdout:5/802: write dd/de/d19/d13/d53/fbb [170111,37707] 0 2024-04-05T21:26:49.407 INFO:tasks.workunit.client.0.smithi121.stdout:4/868: truncate d5/d11/f48 948207 0 2024-04-05T21:26:49.408 INFO:tasks.workunit.client.1.smithi188.stdout:2/973: symlink d5/db/dd/d132/d109/d66/d8f/d12e/l146 0 2024-04-05T21:26:49.409 INFO:tasks.workunit.client.1.smithi188.stdout:2/974: chown d5/db/dd/d36/d3c/d48/c58 1617 1 2024-04-05T21:26:49.411 INFO:tasks.workunit.client.0.smithi121.stdout:9/559: dwrite d6/df/d48/d96/db9/d2c/d53/f88 [0,4194304] 0 2024-04-05T21:26:49.414 INFO:tasks.workunit.client.0.smithi121.stdout:0/828: dwrite d7/d62/f11 [0,4194304] 0 2024-04-05T21:26:49.416 INFO:tasks.workunit.client.0.smithi121.stdout:0/829: chown d7/d7e/f89 351481 1 2024-04-05T21:26:49.416 INFO:tasks.workunit.client.0.smithi121.stdout:4/869: read d5/d11/d2c/d9d/dae/f4c [242036,42328] 0 2024-04-05T21:26:49.417 INFO:tasks.workunit.client.1.smithi188.stdout:2/975: dwrite d5/db/dd/d36/d3c/d63/d75/d87/f9c [0,4194304] 0 2024-04-05T21:26:49.419 INFO:tasks.workunit.client.1.smithi188.stdout:2/976: write d5/db/dd/d36/d3c/d63/ff4 [123944,76603] 0 2024-04-05T21:26:49.420 INFO:tasks.workunit.client.1.smithi188.stdout:2/977: stat d5/db/dd/d132/d109/fbd 0 2024-04-05T21:26:49.422 INFO:tasks.workunit.client.0.smithi121.stdout:5/803: dwrite dd/d41/dce/da6/dd3/db1/db9/fd0 [0,4194304] 0 2024-04-05T21:26:49.431 INFO:tasks.workunit.client.1.smithi188.stdout:5/989: sync 2024-04-05T21:26:49.433 INFO:tasks.workunit.client.1.smithi188.stdout:5/990: rmdir d6/d17/d23/d7d/d128 39 2024-04-05T21:26:49.435 INFO:tasks.workunit.client.0.smithi121.stdout:7/790: dwrite d0/d32/f35 [4194304,4194304] 0 2024-04-05T21:26:49.436 INFO:tasks.workunit.client.0.smithi121.stdout:4/870: symlink d5/d11/d28/d3b/l118 0 2024-04-05T21:26:49.437 INFO:tasks.workunit.client.0.smithi121.stdout:0/830: rename d7/d71/d94/cff to d7/d7e/c126 0 2024-04-05T21:26:49.438 INFO:tasks.workunit.client.0.smithi121.stdout:5/804: fsync dd/de/d19/d38/d56/f87 0 2024-04-05T21:26:49.441 INFO:tasks.workunit.client.1.smithi188.stdout:5/991: rename d6/d15/d5c/d7a/d44/d9b/l14d to d6/d15/d5c/d7a/d44/d98/l156 0 2024-04-05T21:26:49.442 INFO:tasks.workunit.client.1.smithi188.stdout:2/978: dread d5/db/d35/df8/f143 [0,4194304] 0 2024-04-05T21:26:49.443 INFO:tasks.workunit.client.0.smithi121.stdout:7/791: mkdir d0/d23/d5a/df9 0 2024-04-05T21:26:49.444 INFO:tasks.workunit.client.1.smithi188.stdout:2/979: write d5/db/dd/d132/d109/d9f/d10c/dcd/faf [1381789,130381] 0 2024-04-05T21:26:49.446 INFO:tasks.workunit.client.1.smithi188.stdout:2/980: chown d5/db/d35/df8/f13a 0 1 2024-04-05T21:26:49.447 INFO:tasks.workunit.client.1.smithi188.stdout:5/992: link d6/d82/d40/d42/d81/da6/db2/d107/f148 d6/f157 0 2024-04-05T21:26:49.450 INFO:tasks.workunit.client.0.smithi121.stdout:9/560: creat d6/df/d48/d96/db9/d2c/fbf x:0 0 0 2024-04-05T21:26:49.451 INFO:tasks.workunit.client.1.smithi188.stdout:5/993: read d6/d17/f80 [3495508,36847] 0 2024-04-05T21:26:49.453 INFO:tasks.workunit.client.0.smithi121.stdout:2/663: rename d2/d9/d1c/d85/cbf to d2/d9/d56/d8d/ccf 0 2024-04-05T21:26:49.454 INFO:tasks.workunit.client.0.smithi121.stdout:4/871: dread d5/d11/f12 [4194304,4194304] 0 2024-04-05T21:26:49.454 INFO:tasks.workunit.client.1.smithi188.stdout:5/994: chown d6/ca7 438081361 1 2024-04-05T21:26:49.455 INFO:tasks.workunit.client.0.smithi121.stdout:0/831: symlink d7/d71/d29/l127 0 2024-04-05T21:26:49.455 INFO:tasks.workunit.client.1.smithi188.stdout:2/981: dwrite d5/db/dd/d132/d109/d5d/f135 [0,4194304] 0 2024-04-05T21:26:49.458 INFO:tasks.workunit.client.1.smithi188.stdout:5/995: creat d6/d106/d8e/def/f158 x:0 0 0 2024-04-05T21:26:49.458 INFO:tasks.workunit.client.1.smithi188.stdout:2/982: mknod d5/d113/c147 0 2024-04-05T21:26:49.460 INFO:tasks.workunit.client.1.smithi188.stdout:5/996: write d6/d15/f141 [5781761,68586] 0 2024-04-05T21:26:49.460 INFO:tasks.workunit.client.0.smithi121.stdout:7/792: link d0/d9/d22/d28/d3b/f6f d0/d23/ffa 0 2024-04-05T21:26:49.461 INFO:tasks.workunit.client.1.smithi188.stdout:2/983: fdatasync d5/f2f 0 2024-04-05T21:26:49.461 INFO:tasks.workunit.client.1.smithi188.stdout:2/984: stat d5/db/c59 0 2024-04-05T21:26:49.463 INFO:tasks.workunit.client.1.smithi188.stdout:5/997: write d6/d15/d1b/f6c [682289,13163] 0 2024-04-05T21:26:49.463 INFO:tasks.workunit.client.0.smithi121.stdout:2/664: fsync d2/d9/f73 0 2024-04-05T21:26:49.464 INFO:tasks.workunit.client.0.smithi121.stdout:2/665: chown d2/d9/d38/d2d/d4c/d79/fad 143 1 2024-04-05T21:26:49.465 INFO:tasks.workunit.client.1.smithi188.stdout:5/998: write d6/d17/f13b [1630158,5366] 0 2024-04-05T21:26:49.469 INFO:tasks.workunit.client.0.smithi121.stdout:2/666: creat d2/d9/d56/d62/fd0 x:0 0 0 2024-04-05T21:26:49.470 INFO:tasks.workunit.client.0.smithi121.stdout:5/805: getdents dd/de/d19/d3c/d8e 0 2024-04-05T21:26:49.472 INFO:tasks.workunit.client.1.smithi188.stdout:5/999: dwrite d6/d106/d8e/def/d13a/f149 [0,4194304] 0 2024-04-05T21:26:49.473 INFO:tasks.workunit.client.0.smithi121.stdout:2/667: creat d2/db6/fd1 x:0 0 0 2024-04-05T21:26:49.474 INFO:tasks.workunit.client.0.smithi121.stdout:7/793: creat d0/d32/d4e/d73/dbc/df0/dd9/ffb x:0 0 0 2024-04-05T21:26:49.479 INFO:tasks.workunit.client.0.smithi121.stdout:7/794: mkdir d0/d32/d4e/d73/dbc/df0/dd9/dfc 0 2024-04-05T21:26:49.481 INFO:tasks.workunit.client.0.smithi121.stdout:2/668: dwrite d2/d9/d38/d2d/d4c/d79/fb8 [0,4194304] 0 2024-04-05T21:26:49.492 INFO:tasks.workunit.client.0.smithi121.stdout:9/561: write d6/df/d48/d96/db9/f23 [361788,108500] 0 2024-04-05T21:26:49.493 INFO:tasks.workunit.client.0.smithi121.stdout:2/669: write d2/d1e/faf [42018,50388] 0 2024-04-05T21:26:49.494 INFO:tasks.workunit.client.0.smithi121.stdout:4/872: write d5/d45/fb1 [672482,24928] 0 2024-04-05T21:26:49.496 INFO:tasks.workunit.client.0.smithi121.stdout:9/562: mknod d6/d32/dbe/d7b/d5d/d79/cc0 0 2024-04-05T21:26:49.499 INFO:tasks.workunit.client.0.smithi121.stdout:7/795: write d0/d23/f7f [337015,60355] 0 2024-04-05T21:26:49.502 INFO:tasks.workunit.client.0.smithi121.stdout:4/873: creat d5/f119 x:0 0 0 2024-04-05T21:26:49.503 INFO:tasks.workunit.client.0.smithi121.stdout:5/806: dwrite dd/de/d19/d38/d45/f107 [0,4194304] 0 2024-04-05T21:26:49.503 INFO:tasks.workunit.client.0.smithi121.stdout:9/563: creat d6/d32/dbe/d7b/d49/d66/fc1 x:0 0 0 2024-04-05T21:26:49.506 INFO:tasks.workunit.client.0.smithi121.stdout:5/807: chown dd/de/d19/d1c/d36/f71 10 1 2024-04-05T21:26:49.506 INFO:tasks.workunit.client.0.smithi121.stdout:9/564: readlink d6/d32/dbe/l31 0 2024-04-05T21:26:49.509 INFO:tasks.workunit.client.0.smithi121.stdout:7/796: creat d0/d32/d4e/d73/ffd x:0 0 0 2024-04-05T21:26:49.510 INFO:tasks.workunit.client.0.smithi121.stdout:5/808: chown dd/de/d19/c113 536303568 1 2024-04-05T21:26:49.515 INFO:tasks.workunit.client.0.smithi121.stdout:2/670: write d2/d9/d1c/d30/fcd [4638938,101896] 0 2024-04-05T21:26:49.516 INFO:tasks.workunit.client.0.smithi121.stdout:9/565: write d6/df/d48/d96/db9/f2d [4762375,107399] 0 2024-04-05T21:26:49.516 INFO:tasks.workunit.client.0.smithi121.stdout:4/874: write d5/d11/d20/f82 [1755983,78410] 0 2024-04-05T21:26:49.519 INFO:tasks.workunit.client.0.smithi121.stdout:7/797: mknod d0/d9/d22/d28/d56/db9/ddf/def/d52/cfe 0 2024-04-05T21:26:49.519 INFO:tasks.workunit.client.0.smithi121.stdout:4/875: chown d5/d11/d60/df4/cfe 22 1 2024-04-05T21:26:49.520 INFO:tasks.workunit.client.0.smithi121.stdout:5/809: creat dd/f11e x:0 0 0 2024-04-05T21:26:49.520 INFO:tasks.workunit.client.1.smithi188.stdout:0/863: sync 2024-04-05T21:26:49.521 INFO:tasks.workunit.client.1.smithi188.stdout:9/996: sync 2024-04-05T21:26:49.521 INFO:tasks.workunit.client.1.smithi188.stdout:8/773: sync 2024-04-05T21:26:49.521 INFO:tasks.workunit.client.1.smithi188.stdout:1/942: sync 2024-04-05T21:26:49.521 INFO:tasks.workunit.client.0.smithi121.stdout:7/798: dread - d0/d32/d4e/d73/dbc/df0/dd9/ffb zero size 2024-04-05T21:26:49.523 INFO:tasks.workunit.client.0.smithi121.stdout:5/810: truncate dd/de/d19/d13/f10d 954083 0 2024-04-05T21:26:49.524 INFO:tasks.workunit.client.0.smithi121.stdout:9/566: rename d6/d32/dbe/d7b/d5d to d6/df/d48/d96/dc2 0 2024-04-05T21:26:49.525 INFO:tasks.workunit.client.1.smithi188.stdout:1/943: write d1/dc/d16/d21/d27/f12b [1476258,8918] 0 2024-04-05T21:26:49.526 INFO:tasks.workunit.client.1.smithi188.stdout:0/864: read d4/d6/f1f [4060869,55758] 0 2024-04-05T21:26:49.527 INFO:tasks.workunit.client.0.smithi121.stdout:7/799: rmdir d0/d9/d22/d81 39 2024-04-05T21:26:49.530 INFO:tasks.workunit.client.1.smithi188.stdout:8/774: rename d5/d6e/d86/dcb/fa0 to d5/d21/d40/d43/d44/d59/d8b/da4/db1/ffb 0 2024-04-05T21:26:49.531 INFO:tasks.workunit.client.1.smithi188.stdout:0/865: mkdir d4/d6/d13/dca/d125/d144 0 2024-04-05T21:26:49.532 INFO:tasks.workunit.client.0.smithi121.stdout:9/567: dwrite d6/d32/dbe/d7b/f73 [0,4194304] 0 2024-04-05T21:26:49.533 INFO:tasks.workunit.client.1.smithi188.stdout:9/997: dwrite dd/d11/d1f/d26/dc0/d6b/f2d [0,4194304] 0 2024-04-05T21:26:49.533 INFO:tasks.workunit.client.0.smithi121.stdout:2/671: write d2/d9/d17/f19 [6751637,23265] 0 2024-04-05T21:26:49.534 INFO:tasks.workunit.client.1.smithi188.stdout:8/775: chown d5/d21/d40/d43/d44/d59/d8b/ff3 15268409 1 2024-04-05T21:26:49.534 INFO:tasks.workunit.client.0.smithi121.stdout:4/876: write d5/f1d [3741049,102457] 0 2024-04-05T21:26:49.535 INFO:tasks.workunit.client.1.smithi188.stdout:1/944: rename d1/dc/d16/d46/d95/l121 to d1/dc/d16/d46/de5/l130 0 2024-04-05T21:26:49.535 INFO:tasks.workunit.client.0.smithi121.stdout:9/568: write d6/d46/fad [1026132,1555] 0 2024-04-05T21:26:49.535 INFO:tasks.workunit.client.0.smithi121.stdout:7/800: dread d0/d32/d4e/fb6 [0,4194304] 0 2024-04-05T21:26:49.536 INFO:tasks.workunit.client.0.smithi121.stdout:5/811: write dd/de/d19/f51 [2229988,97755] 0 2024-04-05T21:26:49.538 INFO:tasks.workunit.client.0.smithi121.stdout:4/877: mknod d5/d45/d67/c11a 0 2024-04-05T21:26:49.538 INFO:tasks.workunit.client.0.smithi121.stdout:9/569: readlink d6/df/d48/d96/dc2/d79/d94/l98 0 2024-04-05T21:26:49.541 INFO:tasks.workunit.client.0.smithi121.stdout:2/672: read d2/d9/d38/d2d/d4c/d7c/f8b [117757,98947] 0 2024-04-05T21:26:49.541 INFO:tasks.workunit.client.1.smithi188.stdout:9/998: fsync dd/d11/d22/d7e/de3/fef 0 2024-04-05T21:26:49.542 INFO:tasks.workunit.client.1.smithi188.stdout:0/866: getdents d4/d8/d78/dc2/d11a 0 2024-04-05T21:26:49.543 INFO:tasks.workunit.client.0.smithi121.stdout:5/812: creat dd/d41/dce/da6/d10f/f11f x:0 0 0 2024-04-05T21:26:49.543 INFO:tasks.workunit.client.0.smithi121.stdout:2/673: readlink d2/d9/d38/d2d/d4c/d7c/l74 0 2024-04-05T21:26:49.544 INFO:tasks.workunit.client.1.smithi188.stdout:1/945: write d1/dc/d16/d21/d27/d4a/f12e [696026,126028] 0 2024-04-05T21:26:49.544 INFO:tasks.workunit.client.0.smithi121.stdout:4/878: mkdir d5/d11/d26/dbc/dca/d11b 0 2024-04-05T21:26:49.544 INFO:tasks.workunit.client.1.smithi188.stdout:9/999: chown dd/d11/d1f/d26/dc0/f66 246315 1 2024-04-05T21:26:49.545 INFO:tasks.workunit.client.1.smithi188.stdout:8/776: rename d5/d21/d80/f96 to d5/d21/d40/d43/ffc 0 2024-04-05T21:26:49.547 INFO:tasks.workunit.client.0.smithi121.stdout:9/570: rmdir d6/df/d48/d96/db9/d2c 39 2024-04-05T21:26:49.547 INFO:tasks.workunit.client.0.smithi121.stdout:4/879: stat d5/f3f 0 2024-04-05T21:26:49.549 INFO:tasks.workunit.client.0.smithi121.stdout:5/813: creat dd/de/d6d/f120 x:0 0 0 2024-04-05T21:26:49.549 INFO:tasks.workunit.client.0.smithi121.stdout:2/674: write d2/d9/d38/d2d/d4c/d79/fb8 [3160368,61434] 0 2024-04-05T21:26:49.551 INFO:tasks.workunit.client.0.smithi121.stdout:9/571: write d6/df/d48/d96/dc2/f87 [1194108,110098] 0 2024-04-05T21:26:49.551 INFO:tasks.workunit.client.0.smithi121.stdout:2/675: rmdir d2/d9 39 2024-04-05T21:26:49.552 INFO:tasks.workunit.client.0.smithi121.stdout:4/880: symlink d5/d11/d26/d47/df7/d110/l11c 0 2024-04-05T21:26:49.553 INFO:tasks.workunit.client.0.smithi121.stdout:9/572: dread - d6/d32/dbe/d7b/d6b/f8c zero size 2024-04-05T21:26:49.554 INFO:tasks.workunit.client.1.smithi188.stdout:1/946: read d1/dc/d16/d2d/da8/fbd [1993395,35789] 0 2024-04-05T21:26:49.554 INFO:tasks.workunit.client.1.smithi188.stdout:8/777: rename d5/d6e/d86/dcb/c65 to d5/d21/d40/d43/d44/d5b/cfd 0 2024-04-05T21:26:49.554 INFO:tasks.workunit.client.0.smithi121.stdout:2/676: dread - d2/d9/d17/fa7 zero size 2024-04-05T21:26:49.554 INFO:tasks.workunit.client.0.smithi121.stdout:5/814: creat dd/de/d6d/f121 x:0 0 0 2024-04-05T21:26:49.556 INFO:tasks.workunit.client.1.smithi188.stdout:1/947: unlink d1/d7/l52 0 2024-04-05T21:26:49.556 INFO:tasks.workunit.client.0.smithi121.stdout:2/677: rmdir d2/d9/d17 39 2024-04-05T21:26:49.557 INFO:tasks.workunit.client.1.smithi188.stdout:1/948: mknod d1/dc/d16/d46/c131 0 2024-04-05T21:26:49.558 INFO:tasks.workunit.client.1.smithi188.stdout:2/985: dread d5/db/dd/d132/d109/faa [0,4194304] 0 2024-04-05T21:26:49.558 INFO:tasks.workunit.client.0.smithi121.stdout:9/573: creat d6/d46/d8a/fc3 x:0 0 0 2024-04-05T21:26:49.561 INFO:tasks.workunit.client.0.smithi121.stdout:2/678: creat d2/d9/d77/fd2 x:0 0 0 2024-04-05T21:26:49.562 INFO:tasks.workunit.client.0.smithi121.stdout:5/815: getdents dd/de/d19/d13/d53 0 2024-04-05T21:26:49.562 INFO:tasks.workunit.client.0.smithi121.stdout:9/574: mkdir d6/d32/dbe/d7b/dc4 0 2024-04-05T21:26:49.564 INFO:tasks.workunit.client.0.smithi121.stdout:5/816: fsync dd/de/d19/d1c/d74/f10e 0 2024-04-05T21:26:49.564 INFO:tasks.workunit.client.0.smithi121.stdout:2/679: mknod d2/d9/dcb/cd3 0 2024-04-05T21:26:49.565 INFO:tasks.workunit.client.0.smithi121.stdout:9/575: creat d6/d46/d8a/fc5 x:0 0 0 2024-04-05T21:26:49.566 INFO:tasks.workunit.client.0.smithi121.stdout:2/680: fsync d2/d9/d1c/d30/f68 0 2024-04-05T21:26:49.570 INFO:tasks.workunit.client.0.smithi121.stdout:4/881: dread d5/d11/d2c/f36 [0,4194304] 0 2024-04-05T21:26:49.572 INFO:tasks.workunit.client.0.smithi121.stdout:7/801: truncate d0/d9/d22/f44 2795893 0 2024-04-05T21:26:49.574 INFO:tasks.workunit.client.0.smithi121.stdout:7/802: mkdir d0/d9/d22/d28/d56/dff 0 2024-04-05T21:26:49.576 INFO:tasks.workunit.client.0.smithi121.stdout:7/803: creat d0/d9/d22/d3e/d76/f100 x:0 0 0 2024-04-05T21:26:49.578 INFO:tasks.workunit.client.0.smithi121.stdout:7/804: fdatasync d0/f8 0 2024-04-05T21:26:49.579 INFO:tasks.workunit.client.1.smithi188.stdout:0/867: write d4/d8/f3d [285615,63064] 0 2024-04-05T21:26:49.580 INFO:tasks.workunit.client.0.smithi121.stdout:7/805: creat d0/d32/d4e/f101 x:0 0 0 2024-04-05T21:26:49.585 INFO:tasks.workunit.client.0.smithi121.stdout:4/882: write d5/d11/d60/f9f [37646,55844] 0 2024-04-05T21:26:49.587 INFO:tasks.workunit.client.0.smithi121.stdout:1/766: sync 2024-04-05T21:26:49.587 INFO:tasks.workunit.client.0.smithi121.stdout:8/710: sync 2024-04-05T21:26:49.591 INFO:tasks.workunit.client.0.smithi121.stdout:8/711: read d12/d13/d15/d1d/f4c [4091795,64537] 0 2024-04-05T21:26:49.592 INFO:tasks.workunit.client.1.smithi188.stdout:1/949: dwrite d1/d7/dbf/f126 [0,4194304] 0 2024-04-05T21:26:49.592 INFO:tasks.workunit.client.0.smithi121.stdout:4/883: write d5/d11/d28/dc0/f116 [579880,61453] 0 2024-04-05T21:26:49.593 INFO:tasks.workunit.client.1.smithi188.stdout:0/868: dwrite d4/d6/d13/da1/f128 [0,4194304] 0 2024-04-05T21:26:49.595 INFO:tasks.workunit.client.0.smithi121.stdout:7/806: dwrite d0/d9/d22/d3e/d76/f100 [0,4194304] 0 2024-04-05T21:26:49.597 INFO:tasks.workunit.client.0.smithi121.stdout:2/681: dwrite d2/d9/d17/f18 [0,4194304] 0 2024-04-05T21:26:49.602 INFO:tasks.workunit.client.0.smithi121.stdout:7/807: readlink d0/d9/d22/d3e/lc8 0 2024-04-05T21:26:49.603 INFO:tasks.workunit.client.1.smithi188.stdout:0/869: fsync d4/d8/db/d21/d4a/d73/f99 0 2024-04-05T21:26:49.603 INFO:tasks.workunit.client.1.smithi188.stdout:0/870: chown d4/d8/d10/f40 1762117962 1 2024-04-05T21:26:49.604 INFO:tasks.workunit.client.1.smithi188.stdout:1/950: dwrite d1/dc/d16/d21/d27/d49/fb8 [0,4194304] 0 2024-04-05T21:26:49.611 INFO:tasks.workunit.client.0.smithi121.stdout:4/884: creat d5/d11/d2c/d9d/dae/d29/db4/de0/f11d x:0 0 0 2024-04-05T21:26:49.613 INFO:tasks.workunit.client.1.smithi188.stdout:0/871: write d4/d8/db/d42/ff0 [322272,81656] 0 2024-04-05T21:26:49.613 INFO:tasks.workunit.client.0.smithi121.stdout:1/767: write d8/d89/d9e/d57/faf [509842,35930] 0 2024-04-05T21:26:49.615 INFO:tasks.workunit.client.0.smithi121.stdout:2/682: rename d2/d9/d56/d62/cc0 to d2/d9/d38/d2d/cd4 0 2024-04-05T21:26:49.620 INFO:tasks.workunit.client.0.smithi121.stdout:7/808: creat d0/d9/d22/d3e/d76/de1/f102 x:0 0 0 2024-04-05T21:26:49.620 INFO:tasks.workunit.client.0.smithi121.stdout:4/885: symlink d5/d11/d26/d47/df7/d110/l11e 0 2024-04-05T21:26:49.621 INFO:tasks.workunit.client.0.smithi121.stdout:2/683: rename d2/d9/d1c/d30/fab to d2/d1e/fd5 0 2024-04-05T21:26:49.621 INFO:tasks.workunit.client.0.smithi121.stdout:8/712: link d12/d13/fd4 d12/d50/d6a/d63/d8b/fdd 0 2024-04-05T21:26:49.624 INFO:tasks.workunit.client.0.smithi121.stdout:4/886: dread d5/d45/d67/f84 [0,4194304] 0 2024-04-05T21:26:49.625 INFO:tasks.workunit.client.0.smithi121.stdout:2/684: creat d2/d9/d56/d8d/fd6 x:0 0 0 2024-04-05T21:26:49.628 INFO:tasks.workunit.client.0.smithi121.stdout:7/809: mknod d0/d9/d22/d28/d56/dff/c103 0 2024-04-05T21:26:49.631 INFO:tasks.workunit.client.0.smithi121.stdout:7/810: write d0/d23/fe8 [1986203,122220] 0 2024-04-05T21:26:49.633 INFO:tasks.workunit.client.0.smithi121.stdout:1/768: dwrite d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b/ff3 [0,4194304] 0 2024-04-05T21:26:49.634 INFO:tasks.workunit.client.0.smithi121.stdout:2/685: creat d2/d9/d38/d2d/d47/d94/d84/fd7 x:0 0 0 2024-04-05T21:26:49.638 INFO:tasks.workunit.client.0.smithi121.stdout:4/887: dwrite d5/d45/d67/f84 [0,4194304] 0 2024-04-05T21:26:49.638 INFO:tasks.workunit.client.0.smithi121.stdout:2/686: chown d2/d9/ff 3 1 2024-04-05T21:26:49.639 INFO:tasks.workunit.client.0.smithi121.stdout:2/687: fsync d2/d9/d1c/d30/dba/fb5 0 2024-04-05T21:26:49.639 INFO:tasks.workunit.client.0.smithi121.stdout:4/888: dread - d5/d11/d2c/d9d/dae/fdf zero size 2024-04-05T21:26:49.640 INFO:tasks.workunit.client.0.smithi121.stdout:7/811: creat d0/d9/d22/d3e/d76/df7/f104 x:0 0 0 2024-04-05T21:26:49.644 INFO:tasks.workunit.client.0.smithi121.stdout:2/688: mkdir d2/d9/d56/d8d/dd8 0 2024-04-05T21:26:49.645 INFO:tasks.workunit.client.0.smithi121.stdout:7/812: fdatasync d0/d9/d22/d81/da2/fa5 0 2024-04-05T21:26:49.646 INFO:tasks.workunit.client.0.smithi121.stdout:4/889: dread d5/d11/d28/dc0/f116 [0,4194304] 0 2024-04-05T21:26:49.647 INFO:tasks.workunit.client.0.smithi121.stdout:2/689: creat d2/d9/d1c/d85/fd9 x:0 0 0 2024-04-05T21:26:49.648 INFO:tasks.workunit.client.0.smithi121.stdout:7/813: symlink d0/d57/ded/l105 0 2024-04-05T21:26:49.649 INFO:tasks.workunit.client.0.smithi121.stdout:4/890: mkdir d5/d11/d26/dbc/dca/de9/d11f 0 2024-04-05T21:26:49.651 INFO:tasks.workunit.client.0.smithi121.stdout:4/891: write d5/d11/d28/d3b/f93 [1806381,107050] 0 2024-04-05T21:26:49.654 INFO:tasks.workunit.client.0.smithi121.stdout:7/814: dread d0/d23/f2e [4194304,4194304] 0 2024-04-05T21:26:49.658 INFO:tasks.workunit.client.0.smithi121.stdout:4/892: link d5/d11/d2c/d9d/dae/d29/d58/c5f d5/c120 0 2024-04-05T21:26:49.660 INFO:tasks.workunit.client.0.smithi121.stdout:4/893: write d5/d11/d28/fee [130694,36413] 0 2024-04-05T21:26:49.661 INFO:tasks.workunit.client.0.smithi121.stdout:7/815: dwrite d0/d32/d4e/fb6 [0,4194304] 0 2024-04-05T21:26:49.663 INFO:tasks.workunit.client.0.smithi121.stdout:7/816: truncate d0/d32/d4e/d73/dbc/fde 474888 0 2024-04-05T21:26:49.665 INFO:tasks.workunit.client.0.smithi121.stdout:7/817: write d0/d9/d22/d81/da2/fd5 [1202066,7670] 0 2024-04-05T21:26:49.666 INFO:tasks.workunit.client.0.smithi121.stdout:7/818: stat d0/d9/d22/d28/d56/db9/ddf 0 2024-04-05T21:26:49.669 INFO:tasks.workunit.client.0.smithi121.stdout:4/894: dwrite d5/d11/d28/dc0/f116 [0,4194304] 0 2024-04-05T21:26:49.672 INFO:tasks.workunit.client.0.smithi121.stdout:4/895: symlink d5/d11/d28/l121 0 2024-04-05T21:26:49.677 INFO:tasks.workunit.client.0.smithi121.stdout:7/819: dwrite d0/d9/d22/d3e/d76/de1/f102 [0,4194304] 0 2024-04-05T21:26:49.681 INFO:tasks.workunit.client.1.smithi188.stdout:0/872: truncate d4/d6/d46/fcf 67345 0 2024-04-05T21:26:49.685 INFO:tasks.workunit.client.1.smithi188.stdout:0/873: creat d4/d6/d61/f145 x:0 0 0 2024-04-05T21:26:49.689 INFO:tasks.workunit.client.1.smithi188.stdout:1/951: dwrite d1/dc/d16/d46/d88/d3f/f40 [0,4194304] 0 2024-04-05T21:26:49.691 INFO:tasks.workunit.client.0.smithi121.stdout:2/690: write d2/d9/d56/d62/f7a [3897258,68919] 0 2024-04-05T21:26:49.692 INFO:tasks.workunit.client.1.smithi188.stdout:1/952: stat d1/d7/l4c 0 2024-04-05T21:26:49.693 INFO:tasks.workunit.client.0.smithi121.stdout:8/713: dwrite d12/d13/fd2 [0,4194304] 0 2024-04-05T21:26:49.694 INFO:tasks.workunit.client.1.smithi188.stdout:1/953: mknod d1/dc/dfc/c132 0 2024-04-05T21:26:49.697 INFO:tasks.workunit.client.1.smithi188.stdout:1/954: creat d1/dc/d16/d2d/db0/f133 x:0 0 0 2024-04-05T21:26:49.698 INFO:tasks.workunit.client.1.smithi188.stdout:1/955: readlink d1/dc/d16/d46/d88/dea/l114 0 2024-04-05T21:26:49.701 INFO:tasks.workunit.client.1.smithi188.stdout:1/956: creat d1/dc/d16/d21/d27/d4a/f134 x:0 0 0 2024-04-05T21:26:49.703 INFO:tasks.workunit.client.1.smithi188.stdout:1/957: mkdir d1/dc/d16/d21/d135 0 2024-04-05T21:26:49.729 INFO:tasks.workunit.client.1.smithi188.stdout:1/958: write d1/dc/d16/d21/d27/d4a/f7f [4040208,25707] 0 2024-04-05T21:26:49.763 INFO:tasks.workunit.client.1.smithi188.stdout:1/959: write d1/dc/d16/d2d/f7e [1503370,8595] 0 2024-04-05T21:26:49.764 INFO:tasks.workunit.client.1.smithi188.stdout:1/960: rmdir d1/d7/dbf 39 2024-04-05T21:26:49.767 INFO:tasks.workunit.client.1.smithi188.stdout:1/961: rename d1/dc/d16/f33 to d1/dc/d16/d21/d135/f136 0 2024-04-05T21:26:49.769 INFO:tasks.workunit.client.1.smithi188.stdout:1/962: truncate d1/dc/d16/d21/d27/d4a/dc1/f12d 125406 0 2024-04-05T21:26:49.775 INFO:tasks.workunit.client.1.smithi188.stdout:1/963: dwrite d1/dc/d16/d21/d27/d4a/dc1/fce [4194304,4194304] 0 2024-04-05T21:26:49.779 INFO:tasks.workunit.client.1.smithi188.stdout:1/964: mkdir d1/dc/d16/d46/d88/d137 0 2024-04-05T21:26:49.782 INFO:tasks.workunit.client.1.smithi188.stdout:1/965: truncate d1/dc/d16/d46/df7/d5f/f76 2815675 0 2024-04-05T21:26:49.784 INFO:tasks.workunit.client.1.smithi188.stdout:1/966: creat d1/dc/d16/d2d/da8/f138 x:0 0 0 2024-04-05T21:26:49.788 INFO:tasks.workunit.client.1.smithi188.stdout:1/967: read d1/dc/d16/d21/d27/d4a/fad [5816747,20411] 0 2024-04-05T21:26:49.790 INFO:tasks.workunit.client.1.smithi188.stdout:1/968: chown d1/dc/d16/d21/d27/d4a/dc1 523688 1 2024-04-05T21:26:49.792 INFO:tasks.workunit.client.1.smithi188.stdout:1/969: fdatasync d1/dc/d16/d21/dbb/fc9 0 2024-04-05T21:26:49.793 INFO:tasks.workunit.client.1.smithi188.stdout:1/970: write d1/dc/d16/d2d/db0/f133 [529438,118299] 0 2024-04-05T21:26:49.794 INFO:tasks.workunit.client.1.smithi188.stdout:1/971: chown d1/dc/d16/d46/df7/f101 0 1 2024-04-05T21:26:49.795 INFO:tasks.workunit.client.1.smithi188.stdout:1/972: read d1/dc/d16/d2d/f97 [2902847,65468] 0 2024-04-05T21:26:49.796 INFO:tasks.workunit.client.1.smithi188.stdout:1/973: dread - d1/dc/d16/d21/d27/d4a/f8c zero size 2024-04-05T21:26:49.799 INFO:tasks.workunit.client.1.smithi188.stdout:1/974: rename d1/dc/d16/d21/d27/l3e to d1/dc/d16/d46/df7/l139 0 2024-04-05T21:26:49.804 INFO:tasks.workunit.client.1.smithi188.stdout:1/975: dread d1/dc/d16/d21/d27/f12b [0,4194304] 0 2024-04-05T21:26:49.809 INFO:tasks.workunit.client.1.smithi188.stdout:1/976: rename d1/dc/d16/d46/d88/d13/la6 to d1/dc/d16/d46/d88/dea/l13a 0 2024-04-05T21:26:49.810 INFO:tasks.workunit.client.1.smithi188.stdout:1/977: mknod d1/dc/d16/d46/d88/d3f/d119/c13b 0 2024-04-05T21:26:49.812 INFO:tasks.workunit.client.1.smithi188.stdout:1/978: truncate d1/d7/f39 4922456 0 2024-04-05T21:26:49.817 INFO:tasks.workunit.client.1.smithi188.stdout:1/979: write d1/dc/d16/d21/d27/d49/f59 [697214,25893] 0 2024-04-05T21:26:49.818 INFO:tasks.workunit.client.1.smithi188.stdout:1/980: chown d1/dc/d16/d46/d95/d99/f11f 141 1 2024-04-05T21:26:49.820 INFO:tasks.workunit.client.1.smithi188.stdout:1/981: write d1/dc/d16/d46/d88/d3f/f45 [2636288,114979] 0 2024-04-05T21:26:49.828 INFO:tasks.workunit.client.1.smithi188.stdout:1/982: truncate d1/dc/d16/d46/feb 734525 0 2024-04-05T21:26:49.835 INFO:tasks.workunit.client.0.smithi121.stdout:1/769: dwrite d8/d89/d9e/d2d/d84/da8/dad/dde/f6d [4194304,4194304] 0 2024-04-05T21:26:49.836 INFO:tasks.workunit.client.0.smithi121.stdout:2/691: chown d2/d9/d38/d2d/d4c/d7c/d7e/lce 40 1 2024-04-05T21:26:49.837 INFO:tasks.workunit.client.0.smithi121.stdout:8/714: readlink d12/d13/d15/l81 0 2024-04-05T21:26:49.841 INFO:tasks.workunit.client.0.smithi121.stdout:4/896: dwrite d5/d11/d26/d49/fb9 [0,4194304] 0 2024-04-05T21:26:49.842 INFO:tasks.workunit.client.0.smithi121.stdout:2/692: creat d2/d9/d1c/d30/dba/fda x:0 0 0 2024-04-05T21:26:49.844 INFO:tasks.workunit.client.0.smithi121.stdout:8/715: rmdir d12/d50/d61/d7f/d9a 39 2024-04-05T21:26:49.844 INFO:tasks.workunit.client.0.smithi121.stdout:7/820: dwrite d0/d9/f12 [0,4194304] 0 2024-04-05T21:26:49.846 INFO:tasks.workunit.client.0.smithi121.stdout:2/693: chown d2/d9/d77/lb7 11 1 2024-04-05T21:26:49.847 INFO:tasks.workunit.client.0.smithi121.stdout:1/770: creat d8/d8c/db6/d1a/d1b/d4c/d76/f100 x:0 0 0 2024-04-05T21:26:49.849 INFO:tasks.workunit.client.1.smithi188.stdout:1/983: dread d1/dc/d16/d46/d88/d13/d60/fd4 [0,4194304] 0 2024-04-05T21:26:49.850 INFO:tasks.workunit.client.0.smithi121.stdout:2/694: write d2/d9/d56/f93 [112723,105086] 0 2024-04-05T21:26:49.852 INFO:tasks.workunit.client.1.smithi188.stdout:1/984: chown d1/d7/dbf/fc4 386 1 2024-04-05T21:26:49.853 INFO:tasks.workunit.client.0.smithi121.stdout:1/771: fsync d8/d8c/db6/d1a/d1b/d4c/d76/f7c 0 2024-04-05T21:26:49.853 INFO:tasks.workunit.client.0.smithi121.stdout:2/695: chown d2/d9/d1c/d85/l9b 0 1 2024-04-05T21:26:49.855 INFO:tasks.workunit.client.1.smithi188.stdout:1/985: rename d1/dc/d16/d46/d95/d99/l100 to d1/db2/l13c 0 2024-04-05T21:26:49.858 INFO:tasks.workunit.client.1.smithi188.stdout:1/986: creat d1/dc/d16/d21/d27/d49/f13d x:0 0 0 2024-04-05T21:26:49.859 INFO:tasks.workunit.client.0.smithi121.stdout:4/897: rename d5/d45/d67/f84 to d5/d11/d2c/d9d/f122 0 2024-04-05T21:26:49.860 INFO:tasks.workunit.client.0.smithi121.stdout:8/716: dwrite d12/d13/d56/d57/da4/dc9/fcb [0,4194304] 0 2024-04-05T21:26:49.861 INFO:tasks.workunit.client.0.smithi121.stdout:4/898: truncate d5/d45/d8b/f10a 373442 0 2024-04-05T21:26:49.862 INFO:tasks.workunit.client.0.smithi121.stdout:2/696: read d2/d9/d17/f19 [586367,81477] 0 2024-04-05T21:26:49.863 INFO:tasks.workunit.client.0.smithi121.stdout:7/821: getdents d0/d9/d22 0 2024-04-05T21:26:49.864 INFO:tasks.workunit.client.0.smithi121.stdout:8/717: dread - d12/d50/d6a/d63/d8b/f9b zero size 2024-04-05T21:26:49.865 INFO:tasks.workunit.client.0.smithi121.stdout:7/822: stat d0/d32/f8f 0 2024-04-05T21:26:49.867 INFO:tasks.workunit.client.0.smithi121.stdout:1/772: getdents d8/d8c/db6/d1a 0 2024-04-05T21:26:49.868 INFO:tasks.workunit.client.0.smithi121.stdout:7/823: fdatasync d0/d9/f69 0 2024-04-05T21:26:49.869 INFO:tasks.workunit.client.0.smithi121.stdout:8/718: dread d12/d13/d56/d57/f88 [4194304,4194304] 0 2024-04-05T21:26:49.875 INFO:tasks.workunit.client.0.smithi121.stdout:2/697: rename d2/d9/d38/d2d/c34 to d2/d9/d38/d2d/d4c/d7c/cdb 0 2024-04-05T21:26:49.877 INFO:tasks.workunit.client.0.smithi121.stdout:7/824: dwrite d0/d57/ded/ff8 [0,4194304] 0 2024-04-05T21:26:49.879 INFO:tasks.workunit.client.0.smithi121.stdout:2/698: readlink d2/d9/d56/d8d/la0 0 2024-04-05T21:26:49.882 INFO:tasks.workunit.client.1.smithi188.stdout:1/987: truncate d1/dc/d16/d21/d27/d49/f9b 76992 0 2024-04-05T21:26:49.883 INFO:tasks.workunit.client.0.smithi121.stdout:2/699: fdatasync d2/d9/d38/d2d/f72 0 2024-04-05T21:26:49.884 INFO:tasks.workunit.client.0.smithi121.stdout:1/773: link d8/d8c/db6/d1a/d1b/d4c/d76/c80 d8/d10/c101 0 2024-04-05T21:26:49.885 INFO:tasks.workunit.client.0.smithi121.stdout:7/825: dread d0/d23/f7f [0,4194304] 0 2024-04-05T21:26:49.887 INFO:tasks.workunit.client.0.smithi121.stdout:4/899: dwrite d5/d11/d28/f44 [4194304,4194304] 0 2024-04-05T21:26:49.887 INFO:tasks.workunit.client.1.smithi188.stdout:1/988: rename d1/dc/d16/d2d/f9a to d1/dc/d16/d46/d95/f13e 0 2024-04-05T21:26:49.888 INFO:tasks.workunit.client.0.smithi121.stdout:2/700: fdatasync d2/d9/d56/d62/fd0 0 2024-04-05T21:26:49.889 INFO:tasks.workunit.client.0.smithi121.stdout:8/719: dwrite d12/d50/d93/d96/fb2 [0,4194304] 0 2024-04-05T21:26:49.891 INFO:tasks.workunit.client.1.smithi188.stdout:1/989: creat d1/d7/f13f x:0 0 0 2024-04-05T21:26:49.892 INFO:tasks.workunit.client.0.smithi121.stdout:7/826: creat d0/d23/d5a/f106 x:0 0 0 2024-04-05T21:26:49.893 INFO:tasks.workunit.client.0.smithi121.stdout:8/720: readlink d12/d13/d56/l98 0 2024-04-05T21:26:49.897 INFO:tasks.workunit.client.0.smithi121.stdout:2/701: fsync d2/d9/d38/d2d/d4c/d7c/f8b 0 2024-04-05T21:26:49.900 INFO:tasks.workunit.client.0.smithi121.stdout:1/774: dwrite d8/d8c/fe4 [0,4194304] 0 2024-04-05T21:26:49.902 INFO:tasks.workunit.client.0.smithi121.stdout:2/702: write d2/d9/d1c/d30/dba/fda [959164,18912] 0 2024-04-05T21:26:49.903 INFO:tasks.workunit.client.0.smithi121.stdout:7/827: symlink d0/d9/d22/d28/d56/db9/ddf/def/d52/d61/l107 0 2024-04-05T21:26:49.906 INFO:tasks.workunit.client.0.smithi121.stdout:2/703: truncate d2/d9/d1c/d85/fd9 322969 0 2024-04-05T21:26:49.906 INFO:tasks.workunit.client.0.smithi121.stdout:1/775: symlink d8/d89/d9e/da3/l102 0 2024-04-05T21:26:49.907 INFO:tasks.workunit.client.0.smithi121.stdout:8/721: rmdir d12/d50/d6a/dc4 0 2024-04-05T21:26:49.909 INFO:tasks.workunit.client.0.smithi121.stdout:7/828: creat d0/d9/d22/d3e/d76/f108 x:0 0 0 2024-04-05T21:26:49.912 INFO:tasks.workunit.client.0.smithi121.stdout:2/704: rename d2/d9/d38/f5a to d2/d9/d17/fdc 0 2024-04-05T21:26:49.913 INFO:tasks.workunit.client.0.smithi121.stdout:8/722: symlink d12/d13/d56/d57/da4/dc9/lde 0 2024-04-05T21:26:49.914 INFO:tasks.workunit.client.0.smithi121.stdout:1/776: symlink d8/d89/d9e/l103 0 2024-04-05T21:26:49.916 INFO:tasks.workunit.client.0.smithi121.stdout:1/777: chown d8/f1c 6 1 2024-04-05T21:26:49.917 INFO:tasks.workunit.client.0.smithi121.stdout:8/723: mkdir d12/d50/d6a/d87/ddf 0 2024-04-05T21:26:49.917 INFO:tasks.workunit.client.0.smithi121.stdout:4/900: dwrite d5/d11/d28/d87/f96 [0,4194304] 0 2024-04-05T21:26:49.919 INFO:tasks.workunit.client.0.smithi121.stdout:8/724: chown d12/d50/d93/db5 6685 1 2024-04-05T21:26:49.919 INFO:tasks.workunit.client.0.smithi121.stdout:2/705: rename d2/d9/d38/d2d/d4c/c71 to d2/d1e/cdd 0 2024-04-05T21:26:49.921 INFO:tasks.workunit.client.1.smithi188.stdout:1/990: dwrite d1/dc/d16/d46/df7/d5f/fe1 [0,4194304] 0 2024-04-05T21:26:49.922 INFO:tasks.workunit.client.1.smithi188.stdout:0/874: sync 2024-04-05T21:26:49.922 INFO:tasks.workunit.client.1.smithi188.stdout:8/778: sync 2024-04-05T21:26:49.922 INFO:tasks.workunit.client.1.smithi188.stdout:2/986: sync 2024-04-05T21:26:49.922 INFO:tasks.workunit.client.0.smithi121.stdout:2/706: chown d2/d9/d17/c4e 0 1 2024-04-05T21:26:49.922 INFO:tasks.workunit.client.0.smithi121.stdout:7/829: dread d0/d9/d22/d81/da2/fbd [0,4194304] 0 2024-04-05T21:26:49.923 INFO:tasks.workunit.client.0.smithi121.stdout:1/778: dread d8/d89/fb9 [0,4194304] 0 2024-04-05T21:26:49.923 INFO:tasks.workunit.client.0.smithi121.stdout:4/901: mknod d5/d11/d2c/d9d/dae/d29/db4/de0/c123 0 2024-04-05T21:26:49.925 INFO:tasks.workunit.client.0.smithi121.stdout:7/830: readlink d0/d9/d22/d28/d56/l8e 0 2024-04-05T21:26:49.925 INFO:tasks.workunit.client.0.smithi121.stdout:8/725: fsync d12/d50/d93/d96/f92 0 2024-04-05T21:26:49.926 INFO:tasks.workunit.client.1.smithi188.stdout:8/779: write d5/dc/d8f/fd7 [1798531,5236] 0 2024-04-05T21:26:49.927 INFO:tasks.workunit.client.0.smithi121.stdout:8/726: readlink d12/d13/d56/d57/da4/l99 0 2024-04-05T21:26:49.931 INFO:tasks.workunit.client.0.smithi121.stdout:4/902: creat d5/d11/d2c/d9d/dae/d29/d2e/dd3/f124 x:0 0 0 2024-04-05T21:26:49.932 INFO:tasks.workunit.client.1.smithi188.stdout:0/875: unlink d4/d6/d13/dca/d125/f129 0 2024-04-05T21:26:49.932 INFO:tasks.workunit.client.1.smithi188.stdout:2/987: fdatasync d5/db/d35/d3b/d6b/fa7 0 2024-04-05T21:26:49.932 INFO:tasks.workunit.client.1.smithi188.stdout:1/991: mkdir d1/dc/d16/d46/d88/d3f/d119/d140 0 2024-04-05T21:26:49.932 INFO:tasks.workunit.client.0.smithi121.stdout:1/779: fdatasync d8/d8c/db6/d1a/d1b/d24/d79/f9a 0 2024-04-05T21:26:49.933 INFO:tasks.workunit.client.1.smithi188.stdout:2/988: readlink d5/db/d35/l4b 0 2024-04-05T21:26:49.933 INFO:tasks.workunit.client.1.smithi188.stdout:0/876: readlink d4/d8/d78/l11c 0 2024-04-05T21:26:49.934 INFO:tasks.workunit.client.0.smithi121.stdout:7/831: rename d0/d9/d22/lb8 to d0/d23/d5a/l109 0 2024-04-05T21:26:49.934 INFO:tasks.workunit.client.1.smithi188.stdout:1/992: write d1/dc/d16/d46/d88/f79 [1750304,123480] 0 2024-04-05T21:26:49.935 INFO:tasks.workunit.client.0.smithi121.stdout:7/832: fdatasync d0/d32/d4e/fd0 0 2024-04-05T21:26:49.936 INFO:tasks.workunit.client.1.smithi188.stdout:0/877: creat d4/d8/d39/f146 x:0 0 0 2024-04-05T21:26:49.937 INFO:tasks.workunit.client.0.smithi121.stdout:1/780: unlink d8/d89/d9e/c40 0 2024-04-05T21:26:49.937 INFO:tasks.workunit.client.1.smithi188.stdout:2/989: chown d5/db/dd/c40 3014 1 2024-04-05T21:26:49.938 INFO:tasks.workunit.client.0.smithi121.stdout:8/727: creat d12/d50/d61/d7f/d9a/fe0 x:0 0 0 2024-04-05T21:26:49.938 INFO:tasks.workunit.client.0.smithi121.stdout:4/903: truncate d5/d11/f1e 1593784 0 2024-04-05T21:26:49.940 INFO:tasks.workunit.client.0.smithi121.stdout:1/781: mkdir d8/d8c/db6/d1a/d1b/d85/dfe/d104 0 2024-04-05T21:26:49.941 INFO:tasks.workunit.client.0.smithi121.stdout:4/904: mkdir d5/d11/d60/d125 0 2024-04-05T21:26:49.942 INFO:tasks.workunit.client.0.smithi121.stdout:8/728: rename d12/d50/d93/db5/dc7 to d12/d13/d15/d1d/da7/de1 0 2024-04-05T21:26:49.943 INFO:tasks.workunit.client.1.smithi188.stdout:1/993: rename d1/dc/d16/d46/de5/l130 to d1/dc/d16/l141 0 2024-04-05T21:26:49.943 INFO:tasks.workunit.client.0.smithi121.stdout:1/782: symlink d8/d8c/db6/d1a/d1b/d4c/da1/l105 0 2024-04-05T21:26:49.944 INFO:tasks.workunit.client.0.smithi121.stdout:8/729: rmdir d12/d13/d15 39 2024-04-05T21:26:49.945 INFO:tasks.workunit.client.1.smithi188.stdout:2/990: rename d5/db/dd/d36/d3c/d48/f56 to d5/db/dd/d132/d120/d12c/f148 0 2024-04-05T21:26:49.945 INFO:tasks.workunit.client.0.smithi121.stdout:4/905: mkdir d5/d11/d2c/d9d/dae/d29/db4/de0/d126 0 2024-04-05T21:26:49.946 INFO:tasks.workunit.client.0.smithi121.stdout:1/783: creat d8/d8c/db6/d1a/d1b/d44/f106 x:0 0 0 2024-04-05T21:26:49.947 INFO:tasks.workunit.client.1.smithi188.stdout:2/991: creat d5/d113/f149 x:0 0 0 2024-04-05T21:26:49.947 INFO:tasks.workunit.client.0.smithi121.stdout:4/906: fdatasync d5/d11/d28/dc0/f116 0 2024-04-05T21:26:49.947 INFO:tasks.workunit.client.1.smithi188.stdout:1/994: dread d1/dc/d16/d46/d88/d13/d60/fd4 [0,4194304] 0 2024-04-05T21:26:49.948 INFO:tasks.workunit.client.0.smithi121.stdout:1/784: chown d8/d89/d9e 683 1 2024-04-05T21:26:49.949 INFO:tasks.workunit.client.0.smithi121.stdout:1/785: chown d8/d89/d9e/l103 607849853 1 2024-04-05T21:26:49.950 INFO:tasks.workunit.client.1.smithi188.stdout:1/995: stat d1/dc/d16/d46/d88/l116 0 2024-04-05T21:26:49.950 INFO:tasks.workunit.client.0.smithi121.stdout:8/730: dwrite d12/d50/fca [0,4194304] 0 2024-04-05T21:26:49.951 INFO:tasks.workunit.client.0.smithi121.stdout:4/907: creat d5/d11/d2c/d9d/dae/d29/d2e/f127 x:0 0 0 2024-04-05T21:26:49.951 INFO:tasks.workunit.client.0.smithi121.stdout:1/786: chown d8/d8c/db6/d1a/d1b/d24/l4a 50 1 2024-04-05T21:26:49.952 INFO:tasks.workunit.client.1.smithi188.stdout:1/996: unlink d1/dc/d16/d2d/db0/lba 0 2024-04-05T21:26:49.955 INFO:tasks.workunit.client.1.smithi188.stdout:1/997: creat d1/dc/f142 x:0 0 0 2024-04-05T21:26:49.956 INFO:tasks.workunit.client.0.smithi121.stdout:4/908: mknod d5/d11/d28/dc0/de4/c128 0 2024-04-05T21:26:49.957 INFO:tasks.workunit.client.1.smithi188.stdout:1/998: chown d1/dc/c22 1666269 1 2024-04-05T21:26:49.958 INFO:tasks.workunit.client.0.smithi121.stdout:2/707: rmdir d2/d9/d38/d2d/d4c 39 2024-04-05T21:26:49.960 INFO:tasks.workunit.client.0.smithi121.stdout:4/909: mknod d5/d11/d26/dbc/dca/de9/d11f/c129 0 2024-04-05T21:26:49.961 INFO:tasks.workunit.client.1.smithi188.stdout:1/999: creat d1/dc/d16/d21/d27/d4a/dc1/f143 x:0 0 0 2024-04-05T21:26:49.961 INFO:tasks.workunit.client.0.smithi121.stdout:2/708: readlink d2/d9/d38/d2d/d4c/d7c/d7e/lce 0 2024-04-05T21:26:49.962 INFO:tasks.workunit.client.1.smithi188.stdout:8/780: truncate d5/d21/d40/f57 1398389 0 2024-04-05T21:26:49.963 INFO:tasks.workunit.client.0.smithi121.stdout:2/709: chown d2/d9/d38/d2d/d4c/f78 0 1 2024-04-05T21:26:49.964 INFO:tasks.workunit.client.0.smithi121.stdout:4/910: creat d5/f12a x:0 0 0 2024-04-05T21:26:49.964 INFO:tasks.workunit.client.0.smithi121.stdout:2/710: fdatasync d2/d9/d38/f35 0 2024-04-05T21:26:49.966 INFO:tasks.workunit.client.1.smithi188.stdout:8/781: mkdir d5/d21/d40/d43/d44/d5b/d99/d69/d70/dfe 0 2024-04-05T21:26:49.967 INFO:tasks.workunit.client.0.smithi121.stdout:3/598: sync 2024-04-05T21:26:49.970 INFO:tasks.workunit.client.1.smithi188.stdout:8/782: truncate d5/f35 1635077 0 2024-04-05T21:26:49.970 INFO:tasks.workunit.client.1.smithi188.stdout:2/992: dread d5/db/dd/d24/f10f [0,4194304] 0 2024-04-05T21:26:49.970 INFO:tasks.workunit.client.0.smithi121.stdout:7/833: dread - d0/d9/d22/d3e/d76/f108 zero size 2024-04-05T21:26:49.971 INFO:tasks.workunit.client.1.smithi188.stdout:0/878: dwrite d4/d6/d13/f64 [0,4194304] 0 2024-04-05T21:26:49.971 INFO:tasks.workunit.client.1.smithi188.stdout:8/783: stat d5/f1c 0 2024-04-05T21:26:49.974 INFO:tasks.workunit.client.0.smithi121.stdout:8/731: dread d12/d13/f20 [0,4194304] 0 2024-04-05T21:26:49.975 INFO:tasks.workunit.client.0.smithi121.stdout:8/732: readlink d12/l5b 0 2024-04-05T21:26:49.976 INFO:tasks.workunit.client.0.smithi121.stdout:8/733: chown d12/d50/d6a/d87/fc2 11 1 2024-04-05T21:26:49.977 INFO:tasks.workunit.client.0.smithi121.stdout:3/599: creat d0/d3/fc5 x:0 0 0 2024-04-05T21:26:49.978 INFO:tasks.workunit.client.0.smithi121.stdout:7/834: symlink d0/d9/d22/de9/l10a 0 2024-04-05T21:26:49.979 INFO:tasks.workunit.client.0.smithi121.stdout:3/600: readlink d0/d3/d24/d3d/d1a/d1e/l73 0 2024-04-05T21:26:49.980 INFO:tasks.workunit.client.0.smithi121.stdout:2/711: getdents d2/d9/d38/d2d/d47/d94 0 2024-04-05T21:26:49.980 INFO:tasks.workunit.client.0.smithi121.stdout:3/601: chown d0/d3/de/f6a 170635121 1 2024-04-05T21:26:49.981 INFO:tasks.workunit.client.0.smithi121.stdout:8/734: rmdir d12/d50/d93/db5 39 2024-04-05T21:26:49.983 INFO:tasks.workunit.client.1.smithi188.stdout:0/879: unlink d4/d8/db/d21/d4a/d73/d116/d24/l6e 0 2024-04-05T21:26:49.986 INFO:tasks.workunit.client.0.smithi121.stdout:2/712: symlink d2/d9/d56/d8d/dd8/lde 0 2024-04-05T21:26:49.987 INFO:tasks.workunit.client.1.smithi188.stdout:0/880: rename d4/d8/db/d21/d3c/dee/l11f to d4/d8/db/d21/d3c/l147 0 2024-04-05T21:26:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:49 smithi121 ceph-mon[39723]: pgmap v23: 65 pgs: 65 active+clean; 2.9 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 94 MiB/s rd, 262 MiB/s wr, 519 op/s 2024-04-05T21:26:49.989 INFO:tasks.workunit.client.0.smithi121.stdout:7/835: fsync d0/d9/d22/d81/da2/fbd 0 2024-04-05T21:26:49.991 INFO:tasks.workunit.client.0.smithi121.stdout:1/787: dwrite d8/d89/d9e/d2d/f41 [0,4194304] 0 2024-04-05T21:26:49.991 INFO:tasks.workunit.client.1.smithi188.stdout:0/881: dread d4/d6/d13/da1/fb1 [0,4194304] 0 2024-04-05T21:26:49.992 INFO:tasks.workunit.client.0.smithi121.stdout:8/735: creat d12/d13/fe2 x:0 0 0 2024-04-05T21:26:49.992 INFO:tasks.workunit.client.0.smithi121.stdout:3/602: dwrite d0/d3/d24/d3d/d1a/d23/d90/f94 [0,4194304] 0 2024-04-05T21:26:49.993 INFO:tasks.workunit.client.1.smithi188.stdout:0/882: dread - d4/d8/d78/d9d/fef zero size 2024-04-05T21:26:49.995 INFO:tasks.workunit.client.0.smithi121.stdout:1/788: chown d8/d89/d9e/d57/cf5 8624 1 2024-04-05T21:26:49.996 INFO:tasks.workunit.client.1.smithi188.stdout:2/993: write d5/db/dd/d36/d3c/d48/d7a/d7c/fe9 [775015,42594] 0 2024-04-05T21:26:49.998 INFO:tasks.workunit.client.1.smithi188.stdout:2/994: read d5/db/dd/d24/f10f [2410885,49723] 0 2024-04-05T21:26:49.999 INFO:tasks.workunit.client.0.smithi121.stdout:3/603: readlink d0/d3/d24/l5d 0 2024-04-05T21:26:50.000 INFO:tasks.workunit.client.1.smithi188.stdout:8/784: dwrite d5/d21/f6a [0,4194304] 0 2024-04-05T21:26:50.001 INFO:tasks.workunit.client.0.smithi121.stdout:2/713: getdents d2/d9/d56 0 2024-04-05T21:26:50.002 INFO:tasks.workunit.client.0.smithi121.stdout:9/576: sync 2024-04-05T21:26:50.002 INFO:tasks.workunit.client.0.smithi121.stdout:5/817: sync 2024-04-05T21:26:50.002 INFO:tasks.workunit.client.0.smithi121.stdout:6/631: sync 2024-04-05T21:26:50.002 INFO:tasks.workunit.client.0.smithi121.stdout:0/832: sync 2024-04-05T21:26:50.003 INFO:tasks.workunit.client.0.smithi121.stdout:3/604: mkdir d0/dc6 0 2024-04-05T21:26:50.004 INFO:tasks.workunit.client.0.smithi121.stdout:4/911: dwrite d5/d45/f74 [0,4194304] 0 2024-04-05T21:26:50.004 INFO:tasks.workunit.client.0.smithi121.stdout:1/789: truncate d8/d89/d9e/d2d/f8a 247999 0 2024-04-05T21:26:50.006 INFO:tasks.workunit.client.0.smithi121.stdout:0/833: fsync d7/fdd 0 2024-04-05T21:26:50.008 INFO:tasks.workunit.client.0.smithi121.stdout:2/714: truncate d2/d9/d17/fa7 718780 0 2024-04-05T21:26:50.009 INFO:tasks.workunit.client.0.smithi121.stdout:6/632: chown d1/f38 68082 1 2024-04-05T21:26:50.013 INFO:tasks.workunit.client.0.smithi121.stdout:1/790: mkdir d8/d89/d9e/dc8/df4/d107 0 2024-04-05T21:26:50.015 INFO:tasks.workunit.client.1.smithi188.stdout:2/995: dwrite d5/db/dd/d132/d109/fbd [0,4194304] 0 2024-04-05T21:26:50.015 INFO:tasks.workunit.client.0.smithi121.stdout:0/834: symlink d7/l128 0 2024-04-05T21:26:50.016 INFO:tasks.workunit.client.0.smithi121.stdout:6/633: mknod d1/d2/d3/d4/d21/d57/d42/d48/d7c/cda 0 2024-04-05T21:26:50.017 INFO:tasks.workunit.client.1.smithi188.stdout:2/996: chown d5/db/dd/d36/d3c/d48/d7a/ff7 0 1 2024-04-05T21:26:50.017 INFO:tasks.workunit.client.1.smithi188.stdout:2/997: stat d5/fb7 0 2024-04-05T21:26:50.020 INFO:tasks.workunit.client.0.smithi121.stdout:4/912: rename d5/d11/d26/dbc/dca/c106 to d5/d11/d28/c12b 0 2024-04-05T21:26:50.022 INFO:tasks.workunit.client.0.smithi121.stdout:9/577: rmdir d6/df/d48/d96/db9/d2c/d53/dbb 0 2024-04-05T21:26:50.024 INFO:tasks.workunit.client.0.smithi121.stdout:2/715: getdents d2/d9/d1c/d30 0 2024-04-05T21:26:50.025 INFO:tasks.workunit.client.0.smithi121.stdout:4/913: creat d5/d11/d26/d49/f12c x:0 0 0 2024-04-05T21:26:50.026 INFO:tasks.workunit.client.0.smithi121.stdout:4/914: readlink d5/d11/d28/lbf 0 2024-04-05T21:26:50.027 INFO:tasks.workunit.client.0.smithi121.stdout:9/578: symlink d6/df/d48/d96/dc2/d79/d94/lc6 0 2024-04-05T21:26:50.028 INFO:tasks.workunit.client.0.smithi121.stdout:2/716: creat d2/d9/d38/fdf x:0 0 0 2024-04-05T21:26:50.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:49 smithi188 ceph-mon[49715]: pgmap v23: 65 pgs: 65 active+clean; 2.9 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 94 MiB/s rd, 262 MiB/s wr, 519 op/s 2024-04-05T21:26:50.030 INFO:tasks.workunit.client.0.smithi121.stdout:2/717: readlink d2/d9/d38/d2d/d4c/d7c/d7e/lce 0 2024-04-05T21:26:50.031 INFO:tasks.workunit.client.0.smithi121.stdout:9/579: creat d6/df/d48/d96/dc2/d85/fc7 x:0 0 0 2024-04-05T21:26:50.032 INFO:tasks.workunit.client.0.smithi121.stdout:9/580: readlink d6/df/d48/d96/dc2/d79/d94/l98 0 2024-04-05T21:26:50.034 INFO:tasks.workunit.client.0.smithi121.stdout:2/718: truncate d2/d9/d38/d2d/d4c/d79/fad 539361 0 2024-04-05T21:26:50.035 INFO:tasks.workunit.client.0.smithi121.stdout:9/581: write d6/f17 [1573939,124572] 0 2024-04-05T21:26:50.036 INFO:tasks.workunit.client.0.smithi121.stdout:4/915: dwrite d5/d11/d2c/d9d/dae/d29/d2e/dd3/fd0 [0,4194304] 0 2024-04-05T21:26:50.037 INFO:tasks.workunit.client.1.smithi188.stdout:0/883: dread d4/d8/db/d21/f56 [0,4194304] 0 2024-04-05T21:26:50.039 INFO:tasks.workunit.client.0.smithi121.stdout:4/916: creat d5/d11/d20/f12d x:0 0 0 2024-04-05T21:26:50.040 INFO:tasks.workunit.client.1.smithi188.stdout:0/884: dread d4/d8/db/d21/d4a/f58 [0,4194304] 0 2024-04-05T21:26:50.040 INFO:tasks.workunit.client.0.smithi121.stdout:9/582: dread - d6/df/d48/d96/db9/d72/f7f zero size 2024-04-05T21:26:50.041 INFO:tasks.workunit.client.0.smithi121.stdout:4/917: stat d5/d11/d26/f6b 0 2024-04-05T21:26:50.044 INFO:tasks.workunit.client.0.smithi121.stdout:2/719: rename d2/d9/d38/d2d/d4c/d7c/la3 to d2/d9/d38/d2d/le0 0 2024-04-05T21:26:50.046 INFO:tasks.workunit.client.0.smithi121.stdout:4/918: rename d5/l10 to d5/d11/d26/d47/l12e 0 2024-04-05T21:26:50.050 INFO:tasks.workunit.client.0.smithi121.stdout:9/583: rename d6/df/d48/d96/db9/d2c/d53/lb4 to d6/d32/dbe/d7b/dc4/lc8 0 2024-04-05T21:26:50.051 INFO:tasks.workunit.client.0.smithi121.stdout:4/919: mknod d5/d45/c12f 0 2024-04-05T21:26:50.051 INFO:tasks.workunit.client.0.smithi121.stdout:2/720: dwrite d2/d9/d38/d2d/d4c/d79/fb8 [0,4194304] 0 2024-04-05T21:26:50.052 INFO:tasks.workunit.client.0.smithi121.stdout:9/584: write d6/df/d48/d96/db9/f23 [1432491,116588] 0 2024-04-05T21:26:50.056 INFO:tasks.workunit.client.0.smithi121.stdout:4/920: creat d5/d89/de3/f130 x:0 0 0 2024-04-05T21:26:50.058 INFO:tasks.workunit.client.0.smithi121.stdout:9/585: mkdir d6/df/d48/d96/dc2/dc9 0 2024-04-05T21:26:50.059 INFO:tasks.workunit.client.0.smithi121.stdout:2/721: rename d2/d9/d17/cc8 to d2/d9/ce1 0 2024-04-05T21:26:50.059 INFO:tasks.workunit.client.0.smithi121.stdout:4/921: symlink d5/d11/d2c/d9d/dae/d29/d58/l131 0 2024-04-05T21:26:50.060 INFO:tasks.workunit.client.0.smithi121.stdout:4/922: chown d5/d89/fd1 9028149 1 2024-04-05T21:26:50.061 INFO:tasks.workunit.client.0.smithi121.stdout:2/722: mkdir d2/d9/d17/de2 0 2024-04-05T21:26:50.062 INFO:tasks.workunit.client.0.smithi121.stdout:4/923: creat d5/d45/d67/dab/f132 x:0 0 0 2024-04-05T21:26:50.064 INFO:tasks.workunit.client.0.smithi121.stdout:4/924: rename d5/d11/d2c/d9d/dae/d29/d2e/f4d to d5/d11/d60/d125/f133 0 2024-04-05T21:26:50.065 INFO:tasks.workunit.client.0.smithi121.stdout:4/925: fdatasync d5/d11/d20/f38 0 2024-04-05T21:26:50.092 INFO:tasks.workunit.client.1.smithi188.stdout:0/885: truncate d4/d8/d78/dc2/d11a/f104 1513299 0 2024-04-05T21:26:50.094 INFO:tasks.workunit.client.1.smithi188.stdout:0/886: unlink d4/d8/db/d21/d3c/lda 0 2024-04-05T21:26:50.095 INFO:tasks.workunit.client.1.smithi188.stdout:0/887: fsync d4/d6/d46/fcf 0 2024-04-05T21:26:50.106 INFO:tasks.workunit.client.1.smithi188.stdout:0/888: rmdir d4/d8 39 2024-04-05T21:26:50.112 INFO:tasks.workunit.client.1.smithi188.stdout:0/889: rename d4/d8/db/d42/d38/f49 to d4/d6/d46/f148 0 2024-04-05T21:26:50.114 INFO:tasks.workunit.client.1.smithi188.stdout:0/890: truncate d4/d8/d78/dc2/d11a/d80/f9b 1418737 0 2024-04-05T21:26:50.114 INFO:tasks.workunit.client.1.smithi188.stdout:0/891: stat d4/c10f 0 2024-04-05T21:26:50.116 INFO:tasks.workunit.client.1.smithi188.stdout:0/892: unlink d4/d8/db/d42/da2/dd2/f102 0 2024-04-05T21:26:50.118 INFO:tasks.workunit.client.1.smithi188.stdout:0/893: read d4/d6/d61/fae [1832956,86069] 0 2024-04-05T21:26:50.120 INFO:tasks.workunit.client.1.smithi188.stdout:0/894: mkdir d4/d6/d13/dca/d149 0 2024-04-05T21:26:50.120 INFO:tasks.workunit.client.1.smithi188.stdout:0/895: chown d4/d8/d10/l118 486922292 1 2024-04-05T21:26:50.122 INFO:tasks.workunit.client.1.smithi188.stdout:0/896: unlink d4/d8/db/d42/cf1 0 2024-04-05T21:26:50.123 INFO:tasks.workunit.client.1.smithi188.stdout:0/897: readlink d4/d8/db/d21/d4a/d73/d116/de8/l132 0 2024-04-05T21:26:50.124 INFO:tasks.workunit.client.1.smithi188.stdout:0/898: mkdir d4/d8/d39/d14a 0 2024-04-05T21:26:50.125 INFO:tasks.workunit.client.1.smithi188.stdout:0/899: chown d4/d8/db/d21/d4a/d73/d116/de8/l132 119 1 2024-04-05T21:26:50.126 INFO:tasks.workunit.client.1.smithi188.stdout:0/900: write d4/d6/d13/dca/d12f/f12c [937565,123783] 0 2024-04-05T21:26:50.128 INFO:tasks.workunit.client.1.smithi188.stdout:0/901: unlink d4/d8/db/d21/fd6 0 2024-04-05T21:26:50.135 INFO:tasks.workunit.client.1.smithi188.stdout:0/902: rename d4/d8/db/d42/da2/dd2 to d4/d6/df4/d14b 0 2024-04-05T21:26:50.136 INFO:tasks.workunit.client.1.smithi188.stdout:0/903: write d4/d6/d13/dca/d12f/f12c [1465788,49837] 0 2024-04-05T21:26:50.138 INFO:tasks.workunit.client.1.smithi188.stdout:0/904: fsync d4/d8/db/d21/d4a/d73/d116/d24/ddc/f121 0 2024-04-05T21:26:50.140 INFO:tasks.workunit.client.1.smithi188.stdout:0/905: creat d4/d6/d46/f14c x:0 0 0 2024-04-05T21:26:50.142 INFO:tasks.workunit.client.1.smithi188.stdout:0/906: rmdir d4/d6/d13/d103 39 2024-04-05T21:26:50.145 INFO:tasks.workunit.client.0.smithi121.stdout:7/836: truncate d0/d23/f54 1871642 0 2024-04-05T21:26:50.146 INFO:tasks.workunit.client.0.smithi121.stdout:8/736: truncate d12/d50/d93/d96/f72 3140519 0 2024-04-05T21:26:50.147 INFO:tasks.workunit.client.0.smithi121.stdout:3/605: truncate d0/d3/d24/d6d/d7b/f85 903105 0 2024-04-05T21:26:50.150 INFO:tasks.workunit.client.0.smithi121.stdout:6/634: truncate d1/d2/daf/d6f/dc7/f58 2066617 0 2024-04-05T21:26:50.151 INFO:tasks.workunit.client.1.smithi188.stdout:0/907: dwrite d4/d6/d46/f59 [0,4194304] 0 2024-04-05T21:26:50.152 INFO:tasks.workunit.client.0.smithi121.stdout:8/737: chown d12/d50/d6a/d63/fdb 11633 1 2024-04-05T21:26:50.153 INFO:tasks.workunit.client.0.smithi121.stdout:4/926: dwrite d5/d11/d26/d47/fea [0,4194304] 0 2024-04-05T21:26:50.158 INFO:tasks.workunit.client.1.smithi188.stdout:0/908: symlink d4/d6/d13/dca/d149/l14d 0 2024-04-05T21:26:50.159 INFO:tasks.workunit.client.1.smithi188.stdout:0/909: mkdir d4/d6/d13/da1/d130/d14e 0 2024-04-05T21:26:50.160 INFO:tasks.workunit.client.1.smithi188.stdout:0/910: fsync d4/d6/d46/dba/fdb 0 2024-04-05T21:26:50.165 INFO:tasks.workunit.client.1.smithi188.stdout:0/911: rename d4/d8/db/d42/d38/f54 to d4/d6/d13/dea/f14f 0 2024-04-05T21:26:50.167 INFO:tasks.workunit.client.0.smithi121.stdout:3/606: chown d0/d3/de/f9c 5 1 2024-04-05T21:26:50.168 INFO:tasks.workunit.client.0.smithi121.stdout:5/818: dwrite dd/de/d19/d1c/d36/f3a [0,4194304] 0 2024-04-05T21:26:50.171 INFO:tasks.workunit.client.0.smithi121.stdout:0/835: dwrite d7/d71/fdc [0,4194304] 0 2024-04-05T21:26:50.175 INFO:tasks.workunit.client.0.smithi121.stdout:1/791: dwrite d8/f1c [0,4194304] 0 2024-04-05T21:26:50.175 INFO:tasks.workunit.client.0.smithi121.stdout:7/837: dwrite d0/f6 [4194304,4194304] 0 2024-04-05T21:26:50.177 INFO:tasks.workunit.client.0.smithi121.stdout:8/738: symlink d12/d50/d6a/d63/le3 0 2024-04-05T21:26:50.178 INFO:tasks.workunit.client.0.smithi121.stdout:2/723: chown d2/d9/ce1 21312 1 2024-04-05T21:26:50.180 INFO:tasks.workunit.client.0.smithi121.stdout:0/836: symlink d7/dc/l129 0 2024-04-05T21:26:50.180 INFO:tasks.workunit.client.0.smithi121.stdout:4/927: creat d5/d11/d2c/d9d/dae/d29/db4/f134 x:0 0 0 2024-04-05T21:26:50.183 INFO:tasks.workunit.client.0.smithi121.stdout:2/724: fdatasync d2/d9/d38/d2d/d4c/fa6 0 2024-04-05T21:26:50.184 INFO:tasks.workunit.client.0.smithi121.stdout:5/819: mknod dd/de/d19/d38/d45/c122 0 2024-04-05T21:26:50.184 INFO:tasks.workunit.client.0.smithi121.stdout:7/838: readlink d0/l9e 0 2024-04-05T21:26:50.185 INFO:tasks.workunit.client.0.smithi121.stdout:4/928: truncate d5/d11/d26/d49/ff5 175142 0 2024-04-05T21:26:50.186 INFO:tasks.workunit.client.0.smithi121.stdout:1/792: creat d8/d89/d9e/d2d/f108 x:0 0 0 2024-04-05T21:26:50.186 INFO:tasks.workunit.client.0.smithi121.stdout:0/837: chown d7/d62/d9d/f11f 347 1 2024-04-05T21:26:50.189 INFO:tasks.workunit.client.0.smithi121.stdout:3/607: rename d0/d3/d24/d3d/d1a/d23/d38/d55/f77 to d0/d3/d24/d6d/d7b/fc7 0 2024-04-05T21:26:50.190 INFO:tasks.workunit.client.0.smithi121.stdout:8/739: creat d12/d50/d6a/d63/d9c/fe4 x:0 0 0 2024-04-05T21:26:50.191 INFO:tasks.workunit.client.0.smithi121.stdout:4/929: unlink d5/f3f 0 2024-04-05T21:26:50.191 INFO:tasks.workunit.client.0.smithi121.stdout:8/740: read d12/d13/fd2 [2143082,82323] 0 2024-04-05T21:26:50.192 INFO:tasks.workunit.client.0.smithi121.stdout:4/930: chown d5/d45/d67/dab/lb8 113700455 1 2024-04-05T21:26:50.193 INFO:tasks.workunit.client.0.smithi121.stdout:8/741: write d12/d50/d6a/f45 [3233900,109792] 0 2024-04-05T21:26:50.195 INFO:tasks.workunit.client.0.smithi121.stdout:0/838: rmdir d7/d62 39 2024-04-05T21:26:50.200 INFO:tasks.workunit.client.0.smithi121.stdout:4/931: unlink d5/d45/d8b/fbd 0 2024-04-05T21:26:50.201 INFO:tasks.workunit.client.0.smithi121.stdout:8/742: mkdir d12/d50/dbb/de5 0 2024-04-05T21:26:50.201 INFO:tasks.workunit.client.0.smithi121.stdout:4/932: chown d5/d11/d28/d3b/l43 33060217 1 2024-04-05T21:26:50.202 INFO:tasks.workunit.client.0.smithi121.stdout:5/820: truncate dd/de/d19/d1c/d36/d46/f9e 3119726 0 2024-04-05T21:26:50.204 INFO:tasks.workunit.client.0.smithi121.stdout:0/839: mkdir d7/d71/d45/d5a/d12a 0 2024-04-05T21:26:50.206 INFO:tasks.workunit.client.1.smithi188.stdout:0/912: dread d4/d8/db/d42/da2/fa7 [0,4194304] 0 2024-04-05T21:26:50.206 INFO:tasks.workunit.client.0.smithi121.stdout:6/635: dread d1/d2/dc/f4c [0,4194304] 0 2024-04-05T21:26:50.208 INFO:tasks.workunit.client.0.smithi121.stdout:5/821: mknod dd/de/d19/d38/d45/d68/c123 0 2024-04-05T21:26:50.208 INFO:tasks.workunit.client.0.smithi121.stdout:6/636: chown d1/c8c 14 1 2024-04-05T21:26:50.208 INFO:tasks.workunit.client.0.smithi121.stdout:8/743: truncate d12/d50/d93/f95 888707 0 2024-04-05T21:26:50.209 INFO:tasks.workunit.client.0.smithi121.stdout:3/608: link d0/d3/de/da6/l93 d0/d3/d24/db3/lc8 0 2024-04-05T21:26:50.211 INFO:tasks.workunit.client.0.smithi121.stdout:4/933: creat d5/d45/d67/dab/d117/f135 x:0 0 0 2024-04-05T21:26:50.211 INFO:tasks.workunit.client.1.smithi188.stdout:0/913: dwrite d4/d8/db/d21/d4a/d73/f94 [0,4194304] 0 2024-04-05T21:26:50.212 INFO:tasks.workunit.client.0.smithi121.stdout:5/822: write dd/de/d39/d3b/f94 [3649562,64394] 0 2024-04-05T21:26:50.212 INFO:tasks.workunit.client.0.smithi121.stdout:2/725: dread d2/d9/d1c/d30/fcd [0,4194304] 0 2024-04-05T21:26:50.218 INFO:tasks.workunit.client.1.smithi188.stdout:0/914: symlink d4/d6/d13/dca/d125/d12a/l150 0 2024-04-05T21:26:50.218 INFO:tasks.workunit.client.0.smithi121.stdout:6/637: mknod d1/d2/d1b/d35/d39/d92/cdb 0 2024-04-05T21:26:50.218 INFO:tasks.workunit.client.0.smithi121.stdout:3/609: read d0/f27 [1148024,49740] 0 2024-04-05T21:26:50.218 INFO:tasks.workunit.client.1.smithi188.stdout:8/785: sync 2024-04-05T21:26:50.219 INFO:tasks.workunit.client.1.smithi188.stdout:0/915: write d4/d8/db/d21/d4a/d73/f11e [815128,114565] 0 2024-04-05T21:26:50.219 INFO:tasks.workunit.client.0.smithi121.stdout:4/934: rename d5/d11/d26/d47/df7/d110/ca0 to d5/d11/d20/c136 0 2024-04-05T21:26:50.221 INFO:tasks.workunit.client.1.smithi188.stdout:0/916: creat d4/d8/d78/d9d/f151 x:0 0 0 2024-04-05T21:26:50.221 INFO:tasks.workunit.client.0.smithi121.stdout:2/726: symlink d2/d9/d1c/le3 0 2024-04-05T21:26:50.222 INFO:tasks.workunit.client.1.smithi188.stdout:8/786: mkdir d5/d21/d40/d43/d44/d5b/d99/d69/d70/dae/dff 0 2024-04-05T21:26:50.222 INFO:tasks.workunit.client.1.smithi188.stdout:0/917: dread - d4/d6/d61/f145 zero size 2024-04-05T21:26:50.223 INFO:tasks.workunit.client.0.smithi121.stdout:2/727: read d2/d9/d17/fa9 [425787,67541] 0 2024-04-05T21:26:50.224 INFO:tasks.workunit.client.1.smithi188.stdout:0/918: write d4/d8/d78/d9d/ffe [1042213,129599] 0 2024-04-05T21:26:50.224 INFO:tasks.workunit.client.0.smithi121.stdout:4/935: symlink d5/d11/d2c/d9d/dae/d29/db4/l137 0 2024-04-05T21:26:50.225 INFO:tasks.workunit.client.1.smithi188.stdout:8/787: write d5/d21/d2a/d71/fed [149674,18248] 0 2024-04-05T21:26:50.227 INFO:tasks.workunit.client.0.smithi121.stdout:5/823: rename dd/de/d19/d3c/fbd to dd/de/d39/d3b/f124 0 2024-04-05T21:26:50.229 INFO:tasks.workunit.client.1.smithi188.stdout:0/919: dread d4/d6/d13/da1/f128 [0,4194304] 0 2024-04-05T21:26:50.231 INFO:tasks.workunit.client.1.smithi188.stdout:0/920: chown d4/d8/db/d21/d4a/d73/d116/fa6 1 1 2024-04-05T21:26:50.231 INFO:tasks.workunit.client.0.smithi121.stdout:5/824: getdents dd/de/d39/d3b/daf 0 2024-04-05T21:26:50.232 INFO:tasks.workunit.client.1.smithi188.stdout:0/921: chown d4/d8/db/d21/d4a/d73/d116/dd1/ff9 4 1 2024-04-05T21:26:50.232 INFO:tasks.workunit.client.0.smithi121.stdout:5/825: read dd/d41/dce/da6/dd3/db1/db9/fd0 [705563,79804] 0 2024-04-05T21:26:50.233 INFO:tasks.workunit.client.0.smithi121.stdout:5/826: chown dd/de/d39/l43 114319580 1 2024-04-05T21:26:50.235 INFO:tasks.workunit.client.0.smithi121.stdout:5/827: creat dd/de/d19/d38/d9b/f125 x:0 0 0 2024-04-05T21:26:50.238 INFO:tasks.workunit.client.0.smithi121.stdout:5/828: fsync dd/de/d39/d3b/d50/f118 0 2024-04-05T21:26:50.239 INFO:tasks.workunit.client.0.smithi121.stdout:5/829: fsync dd/de/d19/d13/f5d 0 2024-04-05T21:26:50.241 INFO:tasks.workunit.client.0.smithi121.stdout:5/830: truncate dd/de/d19/d38/d45/f84 219791 0 2024-04-05T21:26:50.244 INFO:tasks.workunit.client.0.smithi121.stdout:0/840: dread d7/d71/d45/d5a/f69 [0,4194304] 0 2024-04-05T21:26:50.245 INFO:tasks.workunit.client.0.smithi121.stdout:3/610: dread d0/d3/d24/d3d/f21 [0,4194304] 0 2024-04-05T21:26:50.246 INFO:tasks.workunit.client.1.smithi188.stdout:8/788: write d5/f1c [2757602,61792] 0 2024-04-05T21:26:50.246 INFO:tasks.workunit.client.0.smithi121.stdout:5/831: creat dd/d41/dce/da6/f126 x:0 0 0 2024-04-05T21:26:50.250 INFO:tasks.workunit.client.0.smithi121.stdout:5/832: creat dd/de/d19/d1c/d7b/d88/f127 x:0 0 0 2024-04-05T21:26:50.252 INFO:tasks.workunit.client.0.smithi121.stdout:3/611: creat d0/d3/d24/db3/dc3/fc9 x:0 0 0 2024-04-05T21:26:50.254 INFO:tasks.workunit.client.0.smithi121.stdout:5/833: creat dd/de/d19/d1c/d7b/d88/f128 x:0 0 0 2024-04-05T21:26:50.254 INFO:tasks.workunit.client.0.smithi121.stdout:3/612: write d0/d3/d24/d3d/d1a/d1e/db1/fbb [197119,45616] 0 2024-04-05T21:26:50.255 INFO:tasks.workunit.client.1.smithi188.stdout:8/789: dread d5/d21/d40/f57 [0,4194304] 0 2024-04-05T21:26:50.256 INFO:tasks.workunit.client.0.smithi121.stdout:3/613: fsync d0/d3/d24/d6d/f81 0 2024-04-05T21:26:50.256 INFO:tasks.workunit.client.1.smithi188.stdout:8/790: chown d5/d21/d40/d43/lea 445748 1 2024-04-05T21:26:50.258 INFO:tasks.workunit.client.1.smithi188.stdout:8/791: rmdir d5/dc 39 2024-04-05T21:26:50.260 INFO:tasks.workunit.client.0.smithi121.stdout:5/834: symlink dd/de/d19/d1c/d74/d10b/l129 0 2024-04-05T21:26:50.261 INFO:tasks.workunit.client.0.smithi121.stdout:1/793: truncate d8/d8c/f8e 819633 0 2024-04-05T21:26:50.262 INFO:tasks.workunit.client.0.smithi121.stdout:3/614: mknod d0/d3/d24/d3d/d1a/d23/d38/d55/cca 0 2024-04-05T21:26:50.263 INFO:tasks.workunit.client.0.smithi121.stdout:0/841: dread d7/d62/d9d/da6/fe7 [0,4194304] 0 2024-04-05T21:26:50.265 INFO:tasks.workunit.client.0.smithi121.stdout:1/794: mknod d8/d89/d9e/d2d/d84/dc9/c109 0 2024-04-05T21:26:50.270 INFO:tasks.workunit.client.0.smithi121.stdout:1/795: mknod d8/d8c/db6/d1a/d1b/d4c/da1/c10a 0 2024-04-05T21:26:50.272 INFO:tasks.workunit.client.0.smithi121.stdout:5/835: dread dd/de/d19/f80 [0,4194304] 0 2024-04-05T21:26:50.273 INFO:tasks.workunit.client.0.smithi121.stdout:3/615: link d0/d3/d24/d3d/f7e d0/fcb 0 2024-04-05T21:26:50.273 INFO:tasks.workunit.client.0.smithi121.stdout:5/836: chown dd/de/d19/d13/f5d 5642319 1 2024-04-05T21:26:50.274 INFO:tasks.workunit.client.0.smithi121.stdout:8/744: truncate d12/d50/d93/d96/fb2 2691093 0 2024-04-05T21:26:50.274 INFO:tasks.workunit.client.0.smithi121.stdout:1/796: mknod d8/d8c/db6/d1a/d1b/d24/d79/c10b 0 2024-04-05T21:26:50.275 INFO:tasks.workunit.client.0.smithi121.stdout:0/842: link d7/f3a d7/d62/d3c/d6b/f12b 0 2024-04-05T21:26:50.275 INFO:tasks.workunit.client.0.smithi121.stdout:5/837: chown dd/de/d19/d38/d45/d68/f98 51 1 2024-04-05T21:26:50.276 INFO:tasks.workunit.client.0.smithi121.stdout:0/843: readlink d7/l128 0 2024-04-05T21:26:50.277 INFO:tasks.workunit.client.0.smithi121.stdout:6/638: truncate d1/d32/fae 554285 0 2024-04-05T21:26:50.278 INFO:tasks.workunit.client.0.smithi121.stdout:0/844: fdatasync d7/d71/d45/d46/d106/f118 0 2024-04-05T21:26:50.280 INFO:tasks.workunit.client.0.smithi121.stdout:6/639: write d1/d2/d3/d4/d21/d57/d42/d48/d7c/db3/fc1 [704029,1911] 0 2024-04-05T21:26:50.282 INFO:tasks.workunit.client.0.smithi121.stdout:3/616: rename d0/d3/de/f36 to d0/d3/d24/d6d/fcc 0 2024-04-05T21:26:50.283 INFO:tasks.workunit.client.0.smithi121.stdout:4/936: truncate d5/d11/d26/d49/fb9 2864574 0 2024-04-05T21:26:50.285 INFO:tasks.workunit.client.0.smithi121.stdout:8/745: mkdir d12/d50/d6a/de6 0 2024-04-05T21:26:50.288 INFO:tasks.workunit.client.0.smithi121.stdout:0/845: unlink d7/d62/fc2 0 2024-04-05T21:26:50.288 INFO:tasks.workunit.client.0.smithi121.stdout:2/728: dwrite d2/d9/d38/d2d/f58 [0,4194304] 0 2024-04-05T21:26:50.292 INFO:tasks.workunit.client.0.smithi121.stdout:0/846: readlink d7/d71/d29/d2a/d2c/l48 0 2024-04-05T21:26:50.294 INFO:tasks.workunit.client.0.smithi121.stdout:8/746: dread d12/f51 [0,4194304] 0 2024-04-05T21:26:50.295 INFO:tasks.workunit.client.1.smithi188.stdout:8/792: rename d5/d21/d40/c92 to d5/d21/d40/d43/d44/d59/d79/c100 0 2024-04-05T21:26:50.296 INFO:tasks.workunit.client.1.smithi188.stdout:8/793: stat d5/f12 0 2024-04-05T21:26:50.297 INFO:tasks.workunit.client.0.smithi121.stdout:4/937: dwrite d5/d11/d26/d47/df7/fcb [0,4194304] 0 2024-04-05T21:26:50.297 INFO:tasks.workunit.client.1.smithi188.stdout:8/794: write d5/d21/d40/d43/d44/d5b/fef [443670,52096] 0 2024-04-05T21:26:50.298 INFO:tasks.workunit.client.0.smithi121.stdout:4/938: read - d5/d45/d67/dab/d117/f135 zero size 2024-04-05T21:26:50.299 INFO:tasks.workunit.client.1.smithi188.stdout:8/795: readlink d5/d21/d40/d43/d44/d59/l90 0 2024-04-05T21:26:50.304 INFO:tasks.workunit.client.0.smithi121.stdout:6/640: creat d1/fdc x:0 0 0 2024-04-05T21:26:50.307 INFO:tasks.workunit.client.0.smithi121.stdout:4/939: fsync d5/d11/d2c/d9d/dae/f4c 0 2024-04-05T21:26:50.310 INFO:tasks.workunit.client.1.smithi188.stdout:8/796: dread d5/d21/d40/d43/f66 [0,4194304] 0 2024-04-05T21:26:50.310 INFO:tasks.workunit.client.1.smithi188.stdout:8/797: chown d5/d6e/d98/dc6 1178433 1 2024-04-05T21:26:50.311 INFO:tasks.workunit.client.0.smithi121.stdout:1/797: getdents d8/d8c/db6/d1a/d1b/d4c/da1 0 2024-04-05T21:26:50.312 INFO:tasks.workunit.client.1.smithi188.stdout:8/798: creat d5/d21/d80/f101 x:0 0 0 2024-04-05T21:26:50.312 INFO:tasks.workunit.client.0.smithi121.stdout:3/617: rename d0/d3/d24/d3d/d1a/d23/c62 to d0/d3/d24/d3d/d1a/ccd 0 2024-04-05T21:26:50.313 INFO:tasks.workunit.client.0.smithi121.stdout:1/798: write d8/d8c/db6/d1a/d1b/d4c/d76/f100 [1047511,121450] 0 2024-04-05T21:26:50.314 INFO:tasks.workunit.client.0.smithi121.stdout:1/799: chown d8/d89/d9e/dc8/df4/d107 1 1 2024-04-05T21:26:50.315 INFO:tasks.workunit.client.0.smithi121.stdout:6/641: stat d1/d2/d3/d4/d21/d57/d42/la5 0 2024-04-05T21:26:50.317 INFO:tasks.workunit.client.0.smithi121.stdout:2/729: creat d2/fe4 x:0 0 0 2024-04-05T21:26:50.318 INFO:tasks.workunit.client.0.smithi121.stdout:6/642: read d1/d2/dc/f5b [3976056,60371] 0 2024-04-05T21:26:50.319 INFO:tasks.workunit.client.0.smithi121.stdout:1/800: truncate d8/d89/d9e/d2d/f108 467407 0 2024-04-05T21:26:50.320 INFO:tasks.workunit.client.0.smithi121.stdout:8/747: mkdir d12/d13/d15/de7 0 2024-04-05T21:26:50.320 INFO:tasks.workunit.client.0.smithi121.stdout:2/730: write d2/d9/d38/d2d/d47/f70 [2416535,74712] 0 2024-04-05T21:26:50.322 INFO:tasks.workunit.client.0.smithi121.stdout:3/618: write d0/f27 [1465021,54165] 0 2024-04-05T21:26:50.326 INFO:tasks.workunit.client.0.smithi121.stdout:4/940: dread d5/fd [0,4194304] 0 2024-04-05T21:26:50.327 INFO:tasks.workunit.client.0.smithi121.stdout:1/801: unlink d8/d89/d9e/d2d/f41 0 2024-04-05T21:26:50.329 INFO:tasks.workunit.client.0.smithi121.stdout:3/619: fsync d0/d3/d24/d6d/d7b/da4/fb9 0 2024-04-05T21:26:50.330 INFO:tasks.workunit.client.0.smithi121.stdout:2/731: mkdir d2/d9/d38/d2d/d4c/de5 0 2024-04-05T21:26:50.331 INFO:tasks.workunit.client.0.smithi121.stdout:5/838: dwrite dd/de/d19/d13/f49 [0,4194304] 0 2024-04-05T21:26:50.332 INFO:tasks.workunit.client.0.smithi121.stdout:6/643: creat d1/d2/d3/d4/d21/d57/d42/d48/d7c/fdd x:0 0 0 2024-04-05T21:26:50.336 INFO:tasks.workunit.client.0.smithi121.stdout:4/941: chown d5/d11/d2c/d9d/dae/d29/d58/c5f 15210053 1 2024-04-05T21:26:50.337 INFO:tasks.workunit.client.0.smithi121.stdout:3/620: mkdir d0/d3/d24/d3f/dce 0 2024-04-05T21:26:50.337 INFO:tasks.workunit.client.0.smithi121.stdout:2/732: mknod d2/d9/d38/d2d/d4c/d7c/dc2/ce6 0 2024-04-05T21:26:50.339 INFO:tasks.workunit.client.0.smithi121.stdout:5/839: symlink dd/de/d19/d38/de3/l12a 0 2024-04-05T21:26:50.339 INFO:tasks.workunit.client.0.smithi121.stdout:6/644: write d1/d2/d1b/d5e/dbd/faa [266880,33535] 0 2024-04-05T21:26:50.342 INFO:tasks.workunit.client.0.smithi121.stdout:5/840: stat dd/de/d19/d38/d45/d68 0 2024-04-05T21:26:50.344 INFO:tasks.workunit.client.0.smithi121.stdout:4/942: write d5/d11/d2c/d9d/f75 [1754390,112248] 0 2024-04-05T21:26:50.345 INFO:tasks.workunit.client.0.smithi121.stdout:3/621: mkdir d0/d3/d24/d3d/dcf 0 2024-04-05T21:26:50.345 INFO:tasks.workunit.client.0.smithi121.stdout:1/802: dwrite d8/d8c/db6/d1a/d1b/d4c/d76/fce [0,4194304] 0 2024-04-05T21:26:50.347 INFO:tasks.workunit.client.0.smithi121.stdout:2/733: creat d2/d9/d38/d2d/d4c/d7c/fe7 x:0 0 0 2024-04-05T21:26:50.348 INFO:tasks.workunit.client.0.smithi121.stdout:5/841: fdatasync dd/de/d39/d3b/d50/f5b 0 2024-04-05T21:26:50.350 INFO:tasks.workunit.client.0.smithi121.stdout:3/622: fsync d0/d3/d24/d3d/d1a/d1e/f9a 0 2024-04-05T21:26:50.351 INFO:tasks.workunit.client.0.smithi121.stdout:2/734: readlink d2/d9/d38/d2d/d4c/d7c/d7e/l95 0 2024-04-05T21:26:50.353 INFO:tasks.workunit.client.0.smithi121.stdout:1/803: write d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b/fd2 [1449176,19809] 0 2024-04-05T21:26:50.353 INFO:tasks.workunit.client.0.smithi121.stdout:4/943: write d5/d11/d2c/d9d/dae/fa6 [2530659,102058] 0 2024-04-05T21:26:50.354 INFO:tasks.workunit.client.0.smithi121.stdout:5/842: dread - dd/de/d19/d38/d45/d68/f8f zero size 2024-04-05T21:26:50.356 INFO:tasks.workunit.client.0.smithi121.stdout:6/645: getdents d1/d2/daf/d6f 0 2024-04-05T21:26:50.356 INFO:tasks.workunit.client.0.smithi121.stdout:2/735: write d2/d9/d38/d2d/d4c/d79/fb8 [430132,10628] 0 2024-04-05T21:26:50.356 INFO:tasks.workunit.client.0.smithi121.stdout:3/623: truncate d0/d3/f7f 789963 0 2024-04-05T21:26:50.357 INFO:tasks.workunit.client.0.smithi121.stdout:5/843: write dd/f102 [276132,112084] 0 2024-04-05T21:26:50.359 INFO:tasks.workunit.client.0.smithi121.stdout:3/624: dread - d0/fcb zero size 2024-04-05T21:26:50.360 INFO:tasks.workunit.client.0.smithi121.stdout:5/844: chown dd/de/lb4 17120726 1 2024-04-05T21:26:50.361 INFO:tasks.workunit.client.0.smithi121.stdout:5/845: stat dd/d41/dce/da6/dd3/db1/cef 0 2024-04-05T21:26:50.362 INFO:tasks.workunit.client.0.smithi121.stdout:2/736: creat d2/d9/d56/fe8 x:0 0 0 2024-04-05T21:26:50.363 INFO:tasks.workunit.client.0.smithi121.stdout:6/646: mknod d1/d2/dc/cde 0 2024-04-05T21:26:50.365 INFO:tasks.workunit.client.0.smithi121.stdout:3/625: chown d0/d3/de/da6/l9f 37471 1 2024-04-05T21:26:50.367 INFO:tasks.workunit.client.0.smithi121.stdout:4/944: rename d5/d11/d2c/d9d/dae/d29/d58/f78 to d5/d11/d2c/f138 0 2024-04-05T21:26:50.369 INFO:tasks.workunit.client.0.smithi121.stdout:3/626: fsync d0/d3/d24/d3d/d1a/d1e/f2a 0 2024-04-05T21:26:50.370 INFO:tasks.workunit.client.1.smithi188.stdout:8/799: mknod d5/d6e/c102 0 2024-04-05T21:26:50.372 INFO:tasks.workunit.client.0.smithi121.stdout:8/748: dread d12/d50/d61/f3a [0,4194304] 0 2024-04-05T21:26:50.372 INFO:tasks.workunit.client.0.smithi121.stdout:4/945: chown d5/d11/d28/d3b/ff9 31809 1 2024-04-05T21:26:50.372 INFO:tasks.workunit.client.0.smithi121.stdout:5/846: dwrite dd/de/d39/d3b/f94 [4194304,4194304] 0 2024-04-05T21:26:50.374 INFO:tasks.workunit.client.1.smithi188.stdout:8/800: dread d5/d21/d40/f55 [0,4194304] 0 2024-04-05T21:26:50.375 INFO:tasks.workunit.client.0.smithi121.stdout:6/647: dwrite d1/d2/daf/d6f/dc7/d1f/f64 [0,4194304] 0 2024-04-05T21:26:50.376 INFO:tasks.workunit.client.1.smithi188.stdout:2/998: sync 2024-04-05T21:26:50.376 INFO:tasks.workunit.client.1.smithi188.stdout:8/801: truncate d5/dc/f2c 1735820 0 2024-04-05T21:26:50.377 INFO:tasks.workunit.client.0.smithi121.stdout:8/749: read d12/d50/d6a/d63/f64 [3880175,10006] 0 2024-04-05T21:26:50.377 INFO:tasks.workunit.client.1.smithi188.stdout:8/802: stat d5/l3e 0 2024-04-05T21:26:50.379 INFO:tasks.workunit.client.0.smithi121.stdout:3/627: truncate d0/d3/f30 485983 0 2024-04-05T21:26:50.390 INFO:tasks.workunit.client.0.smithi121.stdout:4/946: creat d5/d11/f139 x:0 0 0 2024-04-05T21:26:50.390 INFO:tasks.workunit.client.0.smithi121.stdout:4/947: chown d5/d11/d26/d47/fd7 1760469 1 2024-04-05T21:26:50.390 INFO:tasks.workunit.client.0.smithi121.stdout:5/847: mkdir dd/d41/df8/d12b 0 2024-04-05T21:26:50.391 INFO:tasks.workunit.client.0.smithi121.stdout:3/628: fsync d0/d3/d24/d3d/f7e 0 2024-04-05T21:26:50.391 INFO:tasks.workunit.client.1.smithi188.stdout:2/999: write d5/db/dd/d132/ffb [345173,29652] 0 2024-04-05T21:26:50.392 INFO:tasks.workunit.client.0.smithi121.stdout:5/848: fsync dd/de/d19/d1c/d36/ffc 0 2024-04-05T21:26:50.392 INFO:tasks.workunit.client.0.smithi121.stdout:4/948: creat d5/d45/d67/f13a x:0 0 0 2024-04-05T21:26:50.392 INFO:tasks.workunit.client.0.smithi121.stdout:9/586: sync 2024-04-05T21:26:50.393 INFO:tasks.workunit.client.0.smithi121.stdout:6/648: dread d1/f72 [0,4194304] 0 2024-04-05T21:26:50.394 INFO:tasks.workunit.client.0.smithi121.stdout:3/629: dread - d0/d3/d24/d3d/d1a/d23/d38/fb7 zero size 2024-04-05T21:26:50.395 INFO:tasks.workunit.client.0.smithi121.stdout:4/949: chown d5/d45/d67/dab/f112 12591 1 2024-04-05T21:26:50.398 INFO:tasks.workunit.client.0.smithi121.stdout:8/750: link d12/d13/d15/f83 d12/d50/d61/fe8 0 2024-04-05T21:26:50.399 INFO:tasks.workunit.client.0.smithi121.stdout:9/587: symlink d6/df/d48/d96/dc2/d79/d94/lca 0 2024-04-05T21:26:50.399 INFO:tasks.workunit.client.0.smithi121.stdout:4/950: read d5/d89/fa7 [387315,59677] 0 2024-04-05T21:26:50.400 INFO:tasks.workunit.client.0.smithi121.stdout:6/649: symlink d1/d2/d1b/ldf 0 2024-04-05T21:26:50.400 INFO:tasks.workunit.client.0.smithi121.stdout:4/951: chown d5/d11/l39 55411 1 2024-04-05T21:26:50.402 INFO:tasks.workunit.client.0.smithi121.stdout:3/630: symlink d0/d3/d24/d3d/d1a/d1e/db1/ld0 0 2024-04-05T21:26:50.412 INFO:tasks.workunit.client.0.smithi121.stdout:6/650: mkdir d1/d2/d1b/d2e/dd9/de0 0 2024-04-05T21:26:50.413 INFO:tasks.workunit.client.0.smithi121.stdout:4/952: read d5/d11/d80/fc2 [59776,13674] 0 2024-04-05T21:26:50.418 INFO:tasks.workunit.client.0.smithi121.stdout:1/804: dwrite d8/d89/d9e/d2d/d84/da8/dad/dde/f54 [4194304,4194304] 0 2024-04-05T21:26:50.423 INFO:tasks.workunit.client.0.smithi121.stdout:1/805: rename d8/d8c/db6/d1a to d8/d8c/db6/d1a/d1b/d4c/d76/da4/d10c 22 2024-04-05T21:26:50.423 INFO:tasks.workunit.client.0.smithi121.stdout:9/588: dread d6/f15 [0,4194304] 0 2024-04-05T21:26:50.432 INFO:tasks.workunit.client.0.smithi121.stdout:2/737: dwrite d2/d9/d38/d2d/f41 [0,4194304] 0 2024-04-05T21:26:50.433 INFO:tasks.workunit.client.0.smithi121.stdout:8/751: getdents d12/d13/dd3 0 2024-04-05T21:26:50.435 INFO:tasks.workunit.client.0.smithi121.stdout:6/651: mknod d1/d2/d1b/d35/d51/ce1 0 2024-04-05T21:26:50.436 INFO:tasks.workunit.client.0.smithi121.stdout:4/953: unlink d5/d45/d67/dab/lff 0 2024-04-05T21:26:50.439 INFO:tasks.workunit.client.0.smithi121.stdout:1/806: unlink d8/d89/d9e/d57/d77/cab 0 2024-04-05T21:26:50.444 INFO:tasks.workunit.client.0.smithi121.stdout:2/738: dread d2/d9/d38/d2d/d4c/d79/fb8 [0,4194304] 0 2024-04-05T21:26:50.444 INFO:tasks.workunit.client.0.smithi121.stdout:5/849: dwrite dd/de/f34 [0,4194304] 0 2024-04-05T21:26:50.445 INFO:tasks.workunit.client.0.smithi121.stdout:5/850: fsync dd/de/d19/f51 0 2024-04-05T21:26:50.448 INFO:tasks.workunit.client.0.smithi121.stdout:5/851: truncate dd/de/d19/d13/dd4/f114 501936 0 2024-04-05T21:26:50.455 INFO:tasks.workunit.client.0.smithi121.stdout:7/839: sync 2024-04-05T21:26:50.455 INFO:tasks.workunit.client.0.smithi121.stdout:0/847: sync 2024-04-05T21:26:50.456 INFO:tasks.workunit.client.0.smithi121.stdout:0/848: fsync d7/d1c/fb6 0 2024-04-05T21:26:50.459 INFO:tasks.workunit.client.0.smithi121.stdout:9/589: rename d6/d46/d8a/fc3 to d6/df/d48/d96/dc2/fcb 0 2024-04-05T21:26:50.463 INFO:tasks.workunit.client.0.smithi121.stdout:8/752: link d12/d50/d6a/fb8 d12/d13/fe9 0 2024-04-05T21:26:50.465 INFO:tasks.workunit.client.0.smithi121.stdout:2/739: chown d2/d9/d56/d62/fd0 726 1 2024-04-05T21:26:50.467 INFO:tasks.workunit.client.0.smithi121.stdout:7/840: dread d0/d9/d22/d28/d56/db9/ddf/def/d52/d61/f89 [0,4194304] 0 2024-04-05T21:26:50.470 INFO:tasks.workunit.client.0.smithi121.stdout:5/852: rename dd/de/f22 to dd/de/d19/d1c/d36/f12c 0 2024-04-05T21:26:50.470 INFO:tasks.workunit.client.0.smithi121.stdout:9/590: fsync d6/d46/fac 0 2024-04-05T21:26:50.471 INFO:tasks.workunit.client.0.smithi121.stdout:6/652: link d1/d2/d3/d4/d21/d57/d42/f83 d1/d2/d1b/da1/fe2 0 2024-04-05T21:26:50.473 INFO:tasks.workunit.client.0.smithi121.stdout:6/653: dread - d1/d2/d1b/da1/fa6 zero size 2024-04-05T21:26:50.475 INFO:tasks.workunit.client.0.smithi121.stdout:2/740: chown d2/d9/c13 138293215 1 2024-04-05T21:26:50.475 INFO:tasks.workunit.client.0.smithi121.stdout:7/841: write d0/d32/d7c/fdd [547294,23299] 0 2024-04-05T21:26:50.478 INFO:tasks.workunit.client.0.smithi121.stdout:2/741: stat d2/d9/d38/d2d/fa5 0 2024-04-05T21:26:50.479 INFO:tasks.workunit.client.0.smithi121.stdout:5/853: dwrite dd/de/d19/d38/d9b/f112 [0,4194304] 0 2024-04-05T21:26:50.481 INFO:tasks.workunit.client.0.smithi121.stdout:9/591: fdatasync d6/df/d48/d96/dc2/d79/fa7 0 2024-04-05T21:26:50.482 INFO:tasks.workunit.client.0.smithi121.stdout:3/631: write d0/d3/de/f72 [401889,77604] 0 2024-04-05T21:26:50.484 INFO:tasks.workunit.client.0.smithi121.stdout:6/654: creat d1/d2/d1b/d35/d39/d92/fe3 x:0 0 0 2024-04-05T21:26:50.486 INFO:tasks.workunit.client.0.smithi121.stdout:0/849: link d7/d71/c61 d7/d71/d94/dd1/c12c 0 2024-04-05T21:26:50.488 INFO:tasks.workunit.client.0.smithi121.stdout:5/854: symlink dd/de/d19/d38/d56/l12d 0 2024-04-05T21:26:50.490 INFO:tasks.workunit.client.0.smithi121.stdout:2/742: rename d2/d9/d56 to d2/d9/d38/d2d/d4c/de9 0 2024-04-05T21:26:50.493 INFO:tasks.workunit.client.0.smithi121.stdout:9/592: readlink d6/d32/dbe/d7b/dc4/lc8 0 2024-04-05T21:26:50.494 INFO:tasks.workunit.client.0.smithi121.stdout:6/655: dwrite d1/d2/d1b/d35/d39/fc9 [0,4194304] 0 2024-04-05T21:26:50.496 INFO:tasks.workunit.client.0.smithi121.stdout:7/842: mknod d0/d23/d5a/df9/c10b 0 2024-04-05T21:26:50.497 INFO:tasks.workunit.client.0.smithi121.stdout:4/954: write d5/f16 [5110943,53509] 0 2024-04-05T21:26:50.499 INFO:tasks.workunit.client.0.smithi121.stdout:5/855: truncate dd/d41/dce/fbe 653340 0 2024-04-05T21:26:50.501 INFO:tasks.workunit.client.0.smithi121.stdout:6/656: write d1/d2/d1b/d91/f9f [692627,58363] 0 2024-04-05T21:26:50.502 INFO:tasks.workunit.client.0.smithi121.stdout:0/850: dwrite d7/d71/d29/ff1 [0,4194304] 0 2024-04-05T21:26:50.503 INFO:tasks.workunit.client.0.smithi121.stdout:2/743: mknod d2/d9/d38/d2d/d4c/de9/cea 0 2024-04-05T21:26:50.507 INFO:tasks.workunit.client.0.smithi121.stdout:0/851: chown d7/d71/d29/f79 0 1 2024-04-05T21:26:50.508 INFO:tasks.workunit.client.0.smithi121.stdout:1/807: dwrite d8/d8c/db6/d1a/d1b/d24/f39 [0,4194304] 0 2024-04-05T21:26:50.508 INFO:tasks.workunit.client.0.smithi121.stdout:9/593: creat d6/df/d48/d96/db9/d2c/d53/fcc x:0 0 0 2024-04-05T21:26:50.509 INFO:tasks.workunit.client.0.smithi121.stdout:4/955: read d5/d11/d60/f9f [132209,64783] 0 2024-04-05T21:26:50.512 INFO:tasks.workunit.client.0.smithi121.stdout:0/852: write d7/d62/d8b/f116 [683047,97210] 0 2024-04-05T21:26:50.512 INFO:tasks.workunit.client.0.smithi121.stdout:9/594: dread - d6/df/d48/faa zero size 2024-04-05T21:26:50.513 INFO:tasks.workunit.client.0.smithi121.stdout:7/843: creat d0/d32/d4e/d73/dbc/df0/dd9/dfc/f10c x:0 0 0 2024-04-05T21:26:50.515 INFO:tasks.workunit.client.0.smithi121.stdout:5/856: dread dd/f102 [0,4194304] 0 2024-04-05T21:26:50.516 INFO:tasks.workunit.client.0.smithi121.stdout:0/853: readlink d7/d7e/d97/lb7 0 2024-04-05T21:26:50.518 INFO:tasks.workunit.client.0.smithi121.stdout:2/744: rename d2/d9/d1c/d30/dba/fda to d2/d9/d1c/d30/feb 0 2024-04-05T21:26:50.518 INFO:tasks.workunit.client.0.smithi121.stdout:7/844: read - d0/d9/d22/d3e/d76/f108 zero size 2024-04-05T21:26:50.519 INFO:tasks.workunit.client.0.smithi121.stdout:4/956: creat d5/d11/d28/d73/f13b x:0 0 0 2024-04-05T21:26:50.519 INFO:tasks.workunit.client.0.smithi121.stdout:8/753: dwrite d12/f3d [0,4194304] 0 2024-04-05T21:26:50.523 INFO:tasks.workunit.client.0.smithi121.stdout:7/845: dread d0/d32/d4e/fb6 [0,4194304] 0 2024-04-05T21:26:50.525 INFO:tasks.workunit.client.0.smithi121.stdout:1/808: symlink d8/d8c/db6/d1a/d1b/d4c/d76/da4/l10d 0 2024-04-05T21:26:50.526 INFO:tasks.workunit.client.0.smithi121.stdout:3/632: write d0/d3/d24/d3f/f4b [4365434,36330] 0 2024-04-05T21:26:50.526 INFO:tasks.workunit.client.0.smithi121.stdout:1/809: read d8/d8c/db6/f8f [637309,125063] 0 2024-04-05T21:26:50.527 INFO:tasks.workunit.client.0.smithi121.stdout:6/657: unlink d1/d2/daf/d6f/dc7/d1f/lc6 0 2024-04-05T21:26:50.530 INFO:tasks.workunit.client.0.smithi121.stdout:0/854: rmdir d7/d7d 39 2024-04-05T21:26:50.531 INFO:tasks.workunit.client.0.smithi121.stdout:0/855: fsync d7/dc/fab 0 2024-04-05T21:26:50.532 INFO:tasks.workunit.client.1.smithi188.stdout:0/922: sync 2024-04-05T21:26:50.533 INFO:tasks.workunit.client.0.smithi121.stdout:4/957: fdatasync d5/d11/d80/fc2 0 2024-04-05T21:26:50.534 INFO:tasks.workunit.client.1.smithi188.stdout:0/923: chown d4/d8/db/d21/d4a/d73/d116/dd1/cfd 4 1 2024-04-05T21:26:50.536 INFO:tasks.workunit.client.1.smithi188.stdout:0/924: symlink d4/d8/d10/l152 0 2024-04-05T21:26:50.537 INFO:tasks.workunit.client.0.smithi121.stdout:3/633: rmdir d0/d3/d24/d3d/d1a/d23/d38/d55 39 2024-04-05T21:26:50.538 INFO:tasks.workunit.client.1.smithi188.stdout:0/925: write d4/d8/db/d21/d4a/d73/d116/de8/f107 [6711711,71116] 0 2024-04-05T21:26:50.538 INFO:tasks.workunit.client.0.smithi121.stdout:6/658: symlink d1/d4b/da7/le4 0 2024-04-05T21:26:50.540 INFO:tasks.workunit.client.0.smithi121.stdout:9/595: symlink d6/df/d48/d96/dc2/lcd 0 2024-04-05T21:26:50.540 INFO:tasks.workunit.client.1.smithi188.stdout:0/926: creat d4/d6/d46/f153 x:0 0 0 2024-04-05T21:26:50.541 INFO:tasks.workunit.client.0.smithi121.stdout:2/745: stat d2/d28/c33 0 2024-04-05T21:26:50.542 INFO:tasks.workunit.client.1.smithi188.stdout:0/927: mknod d4/d6/d13/dca/d125/d144/c154 0 2024-04-05T21:26:50.542 INFO:tasks.workunit.client.0.smithi121.stdout:7/846: dwrite d0/ff4 [0,4194304] 0 2024-04-05T21:26:50.543 INFO:tasks.workunit.client.0.smithi121.stdout:8/754: symlink d12/d50/d93/lea 0 2024-04-05T21:26:50.543 INFO:tasks.workunit.client.1.smithi188.stdout:0/928: readlink d4/d8/db/d21/d4a/d73/d116/de8/l132 0 2024-04-05T21:26:50.545 INFO:tasks.workunit.client.0.smithi121.stdout:7/847: fsync d0/d23/f33 0 2024-04-05T21:26:50.547 INFO:tasks.workunit.client.0.smithi121.stdout:6/659: creat d1/d2/d1b/d2e/fe5 x:0 0 0 2024-04-05T21:26:50.547 INFO:tasks.workunit.client.0.smithi121.stdout:5/857: dread dd/de/d19/f23 [0,4194304] 0 2024-04-05T21:26:50.547 INFO:tasks.workunit.client.1.smithi188.stdout:0/929: link d4/d6/faa d4/d6/d13/f155 0 2024-04-05T21:26:50.549 INFO:tasks.workunit.client.0.smithi121.stdout:4/958: mknod d5/d11/d2c/d9d/c13c 0 2024-04-05T21:26:50.550 INFO:tasks.workunit.client.0.smithi121.stdout:3/634: symlink d0/d3/d24/d3f/dce/ld1 0 2024-04-05T21:26:50.551 INFO:tasks.workunit.client.1.smithi188.stdout:0/930: link d4/fc4 d4/d6/d13/dca/d125/d12a/f156 0 2024-04-05T21:26:50.553 INFO:tasks.workunit.client.0.smithi121.stdout:0/856: symlink d7/d71/d29/d2a/d39/l12d 0 2024-04-05T21:26:50.553 INFO:tasks.workunit.client.1.smithi188.stdout:0/931: unlink d4/d8/d78/d9d/f151 0 2024-04-05T21:26:50.555 INFO:tasks.workunit.client.0.smithi121.stdout:9/596: link d6/df/f71 d6/df/d48/d96/fce 0 2024-04-05T21:26:50.556 INFO:tasks.workunit.client.0.smithi121.stdout:5/858: stat dd/de/d39/d3b/d50/ld7 0 2024-04-05T21:26:50.556 INFO:tasks.workunit.client.1.smithi188.stdout:0/932: creat d4/d8/db/d21/d4a/d73/d116/d24/d57/d134/f157 x:0 0 0 2024-04-05T21:26:50.558 INFO:tasks.workunit.client.0.smithi121.stdout:8/755: write d12/d13/d15/d1d/f94 [3194171,28316] 0 2024-04-05T21:26:50.559 INFO:tasks.workunit.client.0.smithi121.stdout:9/597: write d6/d32/dbe/d7b/d49/f8b [4127116,29961] 0 2024-04-05T21:26:50.561 INFO:tasks.workunit.client.0.smithi121.stdout:4/959: dwrite d5/d11/d28/d3b/f101 [0,4194304] 0 2024-04-05T21:26:50.561 INFO:tasks.workunit.client.0.smithi121.stdout:6/660: link d1/d2/dc/d9a/dc0/f2d d1/d2/d1b/d2e/dd9/de0/fe6 0 2024-04-05T21:26:50.562 INFO:tasks.workunit.client.1.smithi188.stdout:0/933: dwrite d4/d8/db/d21/d3c/fa9 [0,4194304] 0 2024-04-05T21:26:50.564 INFO:tasks.workunit.client.0.smithi121.stdout:2/746: getdents d2/d9/d1c 0 2024-04-05T21:26:50.566 INFO:tasks.workunit.client.1.smithi188.stdout:0/934: mkdir d4/d8/db/d42/d158 0 2024-04-05T21:26:50.566 INFO:tasks.workunit.client.0.smithi121.stdout:6/661: write d1/d2/d1b/d35/d39/d92/f26 [4053633,126449] 0 2024-04-05T21:26:50.566 INFO:tasks.workunit.client.0.smithi121.stdout:9/598: stat d6/c1a 0 2024-04-05T21:26:50.568 INFO:tasks.workunit.client.1.smithi188.stdout:0/935: write d4/d8/db/d21/f7e [1899795,85060] 0 2024-04-05T21:26:50.569 INFO:tasks.workunit.client.1.smithi188.stdout:0/936: chown f3 3 1 2024-04-05T21:26:50.572 INFO:tasks.workunit.client.1.smithi188.stdout:0/937: mknod d4/d8/db/d21/d4a/d73/d116/dd1/c159 0 2024-04-05T21:26:50.573 INFO:tasks.workunit.client.0.smithi121.stdout:4/960: write d5/d11/d28/d87/f96 [1671345,122861] 0 2024-04-05T21:26:50.573 INFO:tasks.workunit.client.0.smithi121.stdout:0/857: creat d7/d71/d45/d46/d53/f12e x:0 0 0 2024-04-05T21:26:50.573 INFO:tasks.workunit.client.0.smithi121.stdout:2/747: mknod d2/d9/d1c/d30/d59/cec 0 2024-04-05T21:26:50.573 INFO:tasks.workunit.client.0.smithi121.stdout:4/961: write d5/d89/fc9 [287922,128336] 0 2024-04-05T21:26:50.573 INFO:tasks.workunit.client.0.smithi121.stdout:2/748: readlink d2/d9/d1c/l99 0 2024-04-05T21:26:50.576 INFO:tasks.workunit.client.1.smithi188.stdout:0/938: creat d4/d6/d13/f15a x:0 0 0 2024-04-05T21:26:50.576 INFO:tasks.workunit.client.0.smithi121.stdout:9/599: creat d6/d32/fcf x:0 0 0 2024-04-05T21:26:50.577 INFO:tasks.workunit.client.0.smithi121.stdout:6/662: stat d1/d2/daf/d6f/dc7/d1f/f46 0 2024-04-05T21:26:50.577 INFO:tasks.workunit.client.0.smithi121.stdout:5/859: truncate dd/de/d19/d1c/fc2 298559 0 2024-04-05T21:26:50.577 INFO:tasks.workunit.client.0.smithi121.stdout:2/749: fdatasync d2/d9/d17/f18 0 2024-04-05T21:26:50.577 INFO:tasks.workunit.client.0.smithi121.stdout:4/962: stat d5/d11/d2c/d9d/dae/d29/fda 0 2024-04-05T21:26:50.578 INFO:tasks.workunit.client.0.smithi121.stdout:0/858: creat d7/d71/d45/d5a/d12a/f12f x:0 0 0 2024-04-05T21:26:50.579 INFO:tasks.workunit.client.0.smithi121.stdout:2/750: read - d2/d9/d1c/d30/d59/f89 zero size 2024-04-05T21:26:50.580 INFO:tasks.workunit.client.1.smithi188.stdout:0/939: mknod d4/d8/db/d21/d4a/d73/d116/d24/ddc/c15b 0 2024-04-05T21:26:50.580 INFO:tasks.workunit.client.0.smithi121.stdout:7/848: dread d0/d9/d22/d81/da2/fa5 [0,4194304] 0 2024-04-05T21:26:50.581 INFO:tasks.workunit.client.0.smithi121.stdout:5/860: creat dd/d41/dce/da6/dd3/db1/f12e x:0 0 0 2024-04-05T21:26:50.581 INFO:tasks.workunit.client.0.smithi121.stdout:6/663: chown d1/d32/fae 3281 1 2024-04-05T21:26:50.582 INFO:tasks.workunit.client.0.smithi121.stdout:0/859: read d7/d71/d94/fd7 [7766,99007] 0 2024-04-05T21:26:50.584 INFO:tasks.workunit.client.1.smithi188.stdout:0/940: dread d4/d6/d13/da1/f128 [0,4194304] 0 2024-04-05T21:26:50.585 INFO:tasks.workunit.client.0.smithi121.stdout:4/963: symlink d5/d11/d2c/d9d/dae/l13d 0 2024-04-05T21:26:50.585 INFO:tasks.workunit.client.0.smithi121.stdout:2/751: dread - d2/d9/d1c/d30/d59/f89 zero size 2024-04-05T21:26:50.588 INFO:tasks.workunit.client.1.smithi188.stdout:0/941: rename d4/d6/d13/dea to d4/d6/d13/dca/d125/d144/d15c 0 2024-04-05T21:26:50.588 INFO:tasks.workunit.client.0.smithi121.stdout:7/849: dread d0/d23/d5a/f68 [0,4194304] 0 2024-04-05T21:26:50.590 INFO:tasks.workunit.client.1.smithi188.stdout:0/942: symlink d4/d6/d13/da1/d130/l15d 0 2024-04-05T21:26:50.591 INFO:tasks.workunit.client.0.smithi121.stdout:5/861: symlink dd/de/d19/d38/d9b/l12f 0 2024-04-05T21:26:50.597 INFO:tasks.workunit.client.0.smithi121.stdout:6/664: rmdir d1/d2/dc/d9a/dc0/d95 0 2024-04-05T21:26:50.598 INFO:tasks.workunit.client.1.smithi188.stdout:0/943: dwrite d4/d8/db/d42/d38/f82 [0,4194304] 0 2024-04-05T21:26:50.599 INFO:tasks.workunit.client.0.smithi121.stdout:4/964: dwrite d5/d11/d28/f13 [4194304,4194304] 0 2024-04-05T21:26:50.599 INFO:tasks.workunit.client.0.smithi121.stdout:3/635: dread d0/d3/d24/d3f/f4b [0,4194304] 0 2024-04-05T21:26:50.599 INFO:tasks.workunit.client.0.smithi121.stdout:2/752: symlink d2/d9/d38/d2d/d4c/de9/d5c/led 0 2024-04-05T21:26:50.601 INFO:tasks.workunit.client.0.smithi121.stdout:6/665: fdatasync d1/d2/f55 0 2024-04-05T21:26:50.602 INFO:tasks.workunit.client.0.smithi121.stdout:5/862: truncate dd/de/d19/d1c/d36/d46/f9a 5068446 0 2024-04-05T21:26:50.603 INFO:tasks.workunit.client.0.smithi121.stdout:7/850: dwrite d0/f6 [4194304,4194304] 0 2024-04-05T21:26:50.605 INFO:tasks.workunit.client.1.smithi188.stdout:0/944: getdents d4/d8/db/d21/d4a/d73/d116/d24/d101/d135 0 2024-04-05T21:26:50.609 INFO:tasks.workunit.client.1.smithi188.stdout:0/945: symlink d4/d8/db/d21/d4a/l15e 0 2024-04-05T21:26:50.609 INFO:tasks.workunit.client.0.smithi121.stdout:2/753: mkdir d2/db6/dbb/dee 0 2024-04-05T21:26:50.610 INFO:tasks.workunit.client.0.smithi121.stdout:7/851: readlink d0/d9/d22/d28/d56/db9/ddf/def/d52/d61/l98 0 2024-04-05T21:26:50.613 INFO:tasks.workunit.client.0.smithi121.stdout:6/666: creat d1/d2/d1b/fe7 x:0 0 0 2024-04-05T21:26:50.614 INFO:tasks.workunit.client.0.smithi121.stdout:5/863: mknod dd/de/d39/d3b/daf/c130 0 2024-04-05T21:26:50.617 INFO:tasks.workunit.client.0.smithi121.stdout:3/636: dwrite d0/f7 [0,4194304] 0 2024-04-05T21:26:50.617 INFO:tasks.workunit.client.1.smithi188.stdout:8/803: sync 2024-04-05T21:26:50.618 INFO:tasks.workunit.client.0.smithi121.stdout:2/754: mkdir d2/d9/d1c/d85/def 0 2024-04-05T21:26:50.619 INFO:tasks.workunit.client.0.smithi121.stdout:1/810: dwrite d8/d89/d9e/d57/f7f [0,4194304] 0 2024-04-05T21:26:50.620 INFO:tasks.workunit.client.0.smithi121.stdout:2/755: chown d2/f11 0 1 2024-04-05T21:26:50.621 INFO:tasks.workunit.client.1.smithi188.stdout:8/804: chown d5/d21/d40/d43/d44/c9c 742572378 1 2024-04-05T21:26:50.622 INFO:tasks.workunit.client.0.smithi121.stdout:4/965: link d5/d11/d26/ffc d5/d11/d26/d49/f13e 0 2024-04-05T21:26:50.623 INFO:tasks.workunit.client.1.smithi188.stdout:8/805: chown d5/d21/d40/d43/f87 23220 1 2024-04-05T21:26:50.626 INFO:tasks.workunit.client.0.smithi121.stdout:6/667: symlink d1/d2/dc/d9a/dc0/le8 0 2024-04-05T21:26:50.626 INFO:tasks.workunit.client.0.smithi121.stdout:6/668: read - d1/d2/d3/fb8 zero size 2024-04-05T21:26:50.627 INFO:tasks.workunit.client.0.smithi121.stdout:1/811: rmdir d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b 39 2024-04-05T21:26:50.627 INFO:tasks.workunit.client.0.smithi121.stdout:8/756: write d12/d50/d93/f95 [687331,57316] 0 2024-04-05T21:26:50.629 INFO:tasks.workunit.client.0.smithi121.stdout:3/637: dread d0/d3/d24/d3d/d1a/d23/d90/d9e/fa0 [0,4194304] 0 2024-04-05T21:26:50.631 INFO:tasks.workunit.client.0.smithi121.stdout:8/757: write d12/d50/d6a/d63/fdb [461754,25963] 0 2024-04-05T21:26:50.632 INFO:tasks.workunit.client.0.smithi121.stdout:4/966: symlink d5/d11/d26/dbc/dca/de9/l13f 0 2024-04-05T21:26:50.634 INFO:tasks.workunit.client.0.smithi121.stdout:0/860: link d7/d71/d29/d2a/d39/f101 d7/d71/d45/d46/d53/d10e/f130 0 2024-04-05T21:26:50.634 INFO:tasks.workunit.client.1.smithi188.stdout:8/806: link d5/l6 d5/d21/d40/d43/d44/d59/d79/l103 0 2024-04-05T21:26:50.637 INFO:tasks.workunit.client.1.smithi188.stdout:8/807: mknod d5/d21/d2a/d95/dc0/c104 0 2024-04-05T21:26:50.637 INFO:tasks.workunit.client.0.smithi121.stdout:9/600: truncate d6/df/d48/d96/db9/d2c/d37/d3c/f7c 1457789 0 2024-04-05T21:26:50.639 INFO:tasks.workunit.client.0.smithi121.stdout:2/756: getdents d2/d9/d38/d2d/d4c/de5 0 2024-04-05T21:26:50.640 INFO:tasks.workunit.client.1.smithi188.stdout:8/808: creat d5/d21/d40/d43/d44/d59/d8b/f105 x:0 0 0 2024-04-05T21:26:50.641 INFO:tasks.workunit.client.0.smithi121.stdout:3/638: dread - d0/d3/d24/d3d/d1a/d1e/f71 zero size 2024-04-05T21:26:50.642 INFO:tasks.workunit.client.1.smithi188.stdout:8/809: creat d5/d21/d40/d43/d44/d5b/d99/d69/d70/dae/f106 x:0 0 0 2024-04-05T21:26:50.644 INFO:tasks.workunit.client.1.smithi188.stdout:8/810: mknod d5/dc/c107 0 2024-04-05T21:26:50.646 INFO:tasks.workunit.client.0.smithi121.stdout:9/601: dwrite d6/d32/dbe/d7b/d6b/f8f [0,4194304] 0 2024-04-05T21:26:50.646 INFO:tasks.workunit.client.0.smithi121.stdout:0/861: rename d7/d71/d29/d3f/fa1 to d7/d71/d29/de8/f131 0 2024-04-05T21:26:50.648 INFO:tasks.workunit.client.0.smithi121.stdout:1/812: truncate d8/d89/f98 4938 0 2024-04-05T21:26:50.649 INFO:tasks.workunit.client.0.smithi121.stdout:6/669: creat d1/d4b/dcc/fe9 x:0 0 0 2024-04-05T21:26:50.651 INFO:tasks.workunit.client.0.smithi121.stdout:3/639: dwrite d0/d3/de/f72 [0,4194304] 0 2024-04-05T21:26:50.653 INFO:tasks.workunit.client.0.smithi121.stdout:0/862: dread d7/d62/d8b/f116 [0,4194304] 0 2024-04-05T21:26:50.654 INFO:tasks.workunit.client.0.smithi121.stdout:4/967: creat d5/d11/d2c/d9d/dae/f140 x:0 0 0 2024-04-05T21:26:50.655 INFO:tasks.workunit.client.0.smithi121.stdout:6/670: mkdir d1/d2/d1b/d35/d39/d92/dea 0 2024-04-05T21:26:50.656 INFO:tasks.workunit.client.0.smithi121.stdout:6/671: dread - d1/fdc zero size 2024-04-05T21:26:50.656 INFO:tasks.workunit.client.0.smithi121.stdout:1/813: fdatasync d8/d8c/db6/d1a/d1b/d24/f8b 0 2024-04-05T21:26:50.658 INFO:tasks.workunit.client.0.smithi121.stdout:1/814: stat d8/d8c/db6/d1a/d1b/d85/ff8 0 2024-04-05T21:26:50.665 INFO:tasks.workunit.client.0.smithi121.stdout:3/640: rename d0/d3/d24/d3d/d1a/d23/d38/d55/fac to d0/d3/d24/d3d/d1a/d1e/d86/d8e/fd2 0 2024-04-05T21:26:50.666 INFO:tasks.workunit.client.0.smithi121.stdout:1/815: creat d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b/f10e x:0 0 0 2024-04-05T21:26:50.669 INFO:tasks.workunit.client.0.smithi121.stdout:1/816: creat d8/d8c/da2/f10f x:0 0 0 2024-04-05T21:26:50.669 INFO:tasks.workunit.client.0.smithi121.stdout:3/641: dread d0/d3/d24/d6d/fcc [0,4194304] 0 2024-04-05T21:26:50.673 INFO:tasks.workunit.client.0.smithi121.stdout:1/817: dread d8/f1c [0,4194304] 0 2024-04-05T21:26:50.674 INFO:tasks.workunit.client.0.smithi121.stdout:3/642: link d0/d3/d24/d3d/d1a/d23/d38/d55/cb5 d0/d3/de/da6/cd3 0 2024-04-05T21:26:50.686 INFO:tasks.workunit.client.0.smithi121.stdout:3/643: dread d0/d3/d24/f89 [0,4194304] 0 2024-04-05T21:26:50.688 INFO:tasks.workunit.client.0.smithi121.stdout:6/672: dread d1/d2/d1b/d35/d39/d92/f26 [0,4194304] 0 2024-04-05T21:26:50.688 INFO:tasks.workunit.client.0.smithi121.stdout:3/644: symlink d0/dc6/ld4 0 2024-04-05T21:26:50.690 INFO:tasks.workunit.client.0.smithi121.stdout:3/645: chown d0/d3/d24/d6d/d7b/da4/lbd 5015411 1 2024-04-05T21:26:50.690 INFO:tasks.workunit.client.0.smithi121.stdout:6/673: mknod d1/d2/ceb 0 2024-04-05T21:26:50.693 INFO:tasks.workunit.client.0.smithi121.stdout:3/646: creat d0/d3/d24/d3d/d1a/d23/d38/dae/fd5 x:0 0 0 2024-04-05T21:26:50.694 INFO:tasks.workunit.client.0.smithi121.stdout:3/647: readlink d0/d3/d24/d3d/d1a/d1e/l6e 0 2024-04-05T21:26:50.695 INFO:tasks.workunit.client.0.smithi121.stdout:3/648: dread - d0/d3/d24/d3d/fa8 zero size 2024-04-05T21:26:50.699 INFO:tasks.workunit.client.0.smithi121.stdout:6/674: read d1/d2/d1b/d5e/fba [386146,124678] 0 2024-04-05T21:26:50.700 INFO:tasks.workunit.client.0.smithi121.stdout:5/864: write dd/de/d39/f8d [1002537,108993] 0 2024-04-05T21:26:50.702 INFO:tasks.workunit.client.0.smithi121.stdout:6/675: symlink d1/d4b/da7/lec 0 2024-04-05T21:26:50.704 INFO:tasks.workunit.client.0.smithi121.stdout:7/852: truncate d0/d9/d22/d3e/d76/f100 2015808 0 2024-04-05T21:26:50.706 INFO:tasks.workunit.client.1.smithi188.stdout:8/811: dwrite d5/d6e/d86/dcb/f5f [0,4194304] 0 2024-04-05T21:26:50.707 INFO:tasks.workunit.client.0.smithi121.stdout:5/865: symlink dd/d41/dce/da6/l131 0 2024-04-05T21:26:50.709 INFO:tasks.workunit.client.0.smithi121.stdout:6/676: unlink d1/d2/d1b/d35/lbf 0 2024-04-05T21:26:50.709 INFO:tasks.workunit.client.0.smithi121.stdout:1/818: dread d8/d89/d9e/f30 [4194304,4194304] 0 2024-04-05T21:26:50.714 INFO:tasks.workunit.client.0.smithi121.stdout:5/866: creat dd/d41/dce/da6/f132 x:0 0 0 2024-04-05T21:26:50.715 INFO:tasks.workunit.client.0.smithi121.stdout:7/853: mknod d0/d32/c10d 0 2024-04-05T21:26:50.717 INFO:tasks.workunit.client.1.smithi188.stdout:8/812: dwrite d5/dc/f1d [4194304,4194304] 0 2024-04-05T21:26:50.719 INFO:tasks.workunit.client.0.smithi121.stdout:8/758: dwrite d12/d50/d6a/d63/d8b/fdd [0,4194304] 0 2024-04-05T21:26:50.719 INFO:tasks.workunit.client.0.smithi121.stdout:7/854: chown d0/l7 11 1 2024-04-05T21:26:50.722 INFO:tasks.workunit.client.0.smithi121.stdout:1/819: mknod d8/d8c/db6/d1a/d1b/d24/dc2/c110 0 2024-04-05T21:26:50.722 INFO:tasks.workunit.client.0.smithi121.stdout:4/968: truncate d5/fd 153012 0 2024-04-05T21:26:50.726 INFO:tasks.workunit.client.0.smithi121.stdout:6/677: rename d1/d2/d3/d4/d21/d57/d42/c43 to d1/d2/d3/d4/d21/d57/d42/d48/ced 0 2024-04-05T21:26:50.730 INFO:tasks.workunit.client.1.smithi188.stdout:8/813: dread d5/d21/d40/d43/d44/d5b/d99/d69/fe3 [0,4194304] 0 2024-04-05T21:26:50.731 INFO:tasks.workunit.client.0.smithi121.stdout:6/678: fsync d1/d2/d1b/d35/d74/f7d 0 2024-04-05T21:26:50.731 INFO:tasks.workunit.client.0.smithi121.stdout:0/863: dwrite d7/d62/ff2 [0,4194304] 0 2024-04-05T21:26:50.732 INFO:tasks.workunit.client.1.smithi188.stdout:8/814: write d5/d21/d40/d43/d44/d5b/d99/d69/d70/dae/f106 [141712,116745] 0 2024-04-05T21:26:50.734 INFO:tasks.workunit.client.1.smithi188.stdout:0/946: sync 2024-04-05T21:26:50.734 INFO:tasks.workunit.client.1.smithi188.stdout:8/815: dread - d5/d21/d40/d43/d44/d59/d8b/ff3 zero size 2024-04-05T21:26:50.735 INFO:tasks.workunit.client.1.smithi188.stdout:8/816: chown d5/c1a 1 1 2024-04-05T21:26:50.735 INFO:tasks.workunit.client.0.smithi121.stdout:4/969: mkdir d5/d11/d26/dbc/d141 0 2024-04-05T21:26:50.737 INFO:tasks.workunit.client.0.smithi121.stdout:0/864: chown d7/d71/d29/d2a/d39/l12d 38729556 1 2024-04-05T21:26:50.738 INFO:tasks.workunit.client.0.smithi121.stdout:8/759: dwrite d12/d50/d61/fbf [0,4194304] 0 2024-04-05T21:26:50.740 INFO:tasks.workunit.client.0.smithi121.stdout:4/970: truncate d5/d45/d67/f13a 958655 0 2024-04-05T21:26:50.741 INFO:tasks.workunit.client.0.smithi121.stdout:1/820: symlink d8/d89/d9e/d2d/d84/l111 0 2024-04-05T21:26:50.742 INFO:tasks.workunit.client.0.smithi121.stdout:0/865: dread - d7/d71/d45/d46/d53/f12e zero size 2024-04-05T21:26:50.742 INFO:tasks.workunit.client.0.smithi121.stdout:4/971: stat d5/d11/d28/d87/fa1 0 2024-04-05T21:26:50.745 INFO:tasks.workunit.client.0.smithi121.stdout:1/821: stat d8/d8c/db6/d1a/d1b/d24/d79/f9a 0 2024-04-05T21:26:50.745 INFO:tasks.workunit.client.0.smithi121.stdout:3/649: write d0/d3/d24/d3d/d1a/f22 [4097768,70694] 0 2024-04-05T21:26:50.746 INFO:tasks.workunit.client.0.smithi121.stdout:8/760: unlink d12/d50/d6a/d63/f64 0 2024-04-05T21:26:50.764 INFO:tasks.workunit.client.1.smithi188.stdout:0/947: unlink d4/d8/db/d21/f2b 0 2024-04-05T21:26:50.765 INFO:tasks.workunit.client.1.smithi188.stdout:8/817: unlink d5/d21/d2a/d95/fdf 0 2024-04-05T21:26:50.767 INFO:tasks.workunit.client.1.smithi188.stdout:0/948: symlink d4/d6/d13/dca/d125/l15f 0 2024-04-05T21:26:50.768 INFO:tasks.workunit.client.1.smithi188.stdout:0/949: fdatasync d4/d6/d46/f148 0 2024-04-05T21:26:50.769 INFO:tasks.workunit.client.1.smithi188.stdout:8/818: mkdir d5/d21/d40/d43/d44/d5b/d99/d69/d70/dfe/d108 0 2024-04-05T21:26:50.769 INFO:tasks.workunit.client.1.smithi188.stdout:0/950: write d4/d6/d13/dca/d12f/f12c [2100136,106392] 0 2024-04-05T21:26:50.776 INFO:tasks.workunit.client.1.smithi188.stdout:8/819: dwrite d5/d21/d2a/d47/fc5 [4194304,4194304] 0 2024-04-05T21:26:50.796 INFO:tasks.workunit.client.1.smithi188.stdout:0/951: truncate d4/d6/d61/fae 2312291 0 2024-04-05T21:26:50.798 INFO:tasks.workunit.client.0.smithi121.stdout:0/866: rmdir d7/d71/d29/d2a/d64 39 2024-04-05T21:26:50.799 INFO:tasks.workunit.client.1.smithi188.stdout:0/952: creat d4/d8/db/d42/d158/f160 x:0 0 0 2024-04-05T21:26:50.799 INFO:tasks.workunit.client.0.smithi121.stdout:4/972: truncate d5/d11/d2c/d9d/dae/f4c 1093887 0 2024-04-05T21:26:50.800 INFO:tasks.workunit.client.1.smithi188.stdout:8/820: dwrite d5/f1e [0,4194304] 0 2024-04-05T21:26:50.800 INFO:tasks.workunit.client.0.smithi121.stdout:7/855: link d0/d32/d4a/lb7 d0/d9/d22/d28/d56/db9/ddf/def/l10e 0 2024-04-05T21:26:50.802 INFO:tasks.workunit.client.0.smithi121.stdout:5/867: write dd/de/d19/d38/f4f [398624,14494] 0 2024-04-05T21:26:50.803 INFO:tasks.workunit.client.1.smithi188.stdout:0/953: unlink d4/d8/db/fe5 0 2024-04-05T21:26:50.804 INFO:tasks.workunit.client.0.smithi121.stdout:1/822: dwrite d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b/f10e [0,4194304] 0 2024-04-05T21:26:50.806 INFO:tasks.workunit.client.1.smithi188.stdout:8/821: symlink d5/d21/d40/d43/d48/d7e/dcf/l109 0 2024-04-05T21:26:50.807 INFO:tasks.workunit.client.0.smithi121.stdout:3/650: mknod d0/d3/d24/d6d/d7b/da4/cd6 0 2024-04-05T21:26:50.807 INFO:tasks.workunit.client.0.smithi121.stdout:6/679: truncate d1/d2/d1b/da1/fe2 4118451 0 2024-04-05T21:26:50.808 INFO:tasks.workunit.client.1.smithi188.stdout:8/822: chown d5/d21/d40/d43/d48/d7e/f93 30428241 1 2024-04-05T21:26:50.814 INFO:tasks.workunit.client.0.smithi121.stdout:6/680: truncate d1/d2/d1b/da1/fa6 1046094 0 2024-04-05T21:26:50.814 INFO:tasks.workunit.client.0.smithi121.stdout:8/761: rename d12/d50/d93/db5 to d12/d50/d6a/d87/deb 0 2024-04-05T21:26:50.814 INFO:tasks.workunit.client.0.smithi121.stdout:3/651: readlink d0/d3/d24/d3f/l49 0 2024-04-05T21:26:50.817 INFO:tasks.workunit.client.0.smithi121.stdout:6/681: unlink d1/d2/daf/d6f/dc7/c14 0 2024-04-05T21:26:50.820 INFO:tasks.workunit.client.0.smithi121.stdout:3/652: rmdir d0/d3/de/da6 39 2024-04-05T21:26:50.822 INFO:tasks.workunit.client.0.smithi121.stdout:1/823: truncate d8/d89/d9e/d57/fe3 2856066 0 2024-04-05T21:26:50.824 INFO:tasks.workunit.client.0.smithi121.stdout:7/856: getdents d0/d9/d22/d81/da2 0 2024-04-05T21:26:50.829 INFO:tasks.workunit.client.0.smithi121.stdout:6/682: dwrite d1/d2/d1b/d91/f9f [0,4194304] 0 2024-04-05T21:26:50.830 INFO:tasks.workunit.client.1.smithi188.stdout:8/823: dwrite d5/d21/d40/d43/d44/d59/f74 [4194304,4194304] 0 2024-04-05T21:26:50.830 INFO:tasks.workunit.client.0.smithi121.stdout:4/973: rename d5/d11/d2c/da4/le8 to d5/d11/d2c/d9d/dae/d29/d2e/dd3/l142 0 2024-04-05T21:26:50.832 INFO:tasks.workunit.client.1.smithi188.stdout:8/824: chown d5/d21/d40/d43/d44/d59/f77 1199 1 2024-04-05T21:26:50.836 INFO:tasks.workunit.client.1.smithi188.stdout:8/825: unlink d5/d21/d40/d43/d48/d7e/le9 0 2024-04-05T21:26:50.836 INFO:tasks.workunit.client.0.smithi121.stdout:7/857: mkdir d0/d32/d4e/d73/dbc/df0/dd9/d10f 0 2024-04-05T21:26:50.838 INFO:tasks.workunit.client.0.smithi121.stdout:6/683: creat d1/d2/dc/d9a/dc0/fee x:0 0 0 2024-04-05T21:26:50.841 INFO:tasks.workunit.client.0.smithi121.stdout:0/867: dwrite d7/d62/d8b/fac [0,4194304] 0 2024-04-05T21:26:50.844 INFO:tasks.workunit.client.0.smithi121.stdout:5/868: dread dd/de/d19/d1c/d74/d8c/dc5/fd5 [0,4194304] 0 2024-04-05T21:26:50.844 INFO:tasks.workunit.client.0.smithi121.stdout:8/762: write d12/d50/d93/d96/fab [881567,63172] 0 2024-04-05T21:26:50.845 INFO:tasks.workunit.client.0.smithi121.stdout:3/653: creat d0/d3/d24/d3d/d1a/d23/d38/fd7 x:0 0 0 2024-04-05T21:26:50.847 INFO:tasks.workunit.client.0.smithi121.stdout:6/684: rename d1/d2/d3/fb8 to d1/d2/d1b/d5e/fef 0 2024-04-05T21:26:50.847 INFO:tasks.workunit.client.0.smithi121.stdout:1/824: creat d8/f112 x:0 0 0 2024-04-05T21:26:50.848 INFO:tasks.workunit.client.0.smithi121.stdout:6/685: stat d1/d2/daf/d6f/dc7/d1f/l8e 0 2024-04-05T21:26:50.850 INFO:tasks.workunit.client.0.smithi121.stdout:5/869: creat dd/de/d39/d3b/daf/f133 x:0 0 0 2024-04-05T21:26:50.851 INFO:tasks.workunit.client.0.smithi121.stdout:3/654: symlink d0/d3/d24/d3d/d1a/d23/d38/dae/ld8 0 2024-04-05T21:26:50.853 INFO:tasks.workunit.client.0.smithi121.stdout:3/655: chown d0/d3/de/l18 3 1 2024-04-05T21:26:50.853 INFO:tasks.workunit.client.0.smithi121.stdout:7/858: link d0/d32/cd2 d0/d9/d22/d28/d56/c110 0 2024-04-05T21:26:50.853 INFO:tasks.workunit.client.0.smithi121.stdout:1/825: read d8/d8c/f8e [375701,110091] 0 2024-04-05T21:26:50.854 INFO:tasks.workunit.client.0.smithi121.stdout:6/686: mkdir d1/d2/d3/d4/df0 0 2024-04-05T21:26:50.855 INFO:tasks.workunit.client.0.smithi121.stdout:5/870: creat dd/de/d19/d1c/d74/f134 x:0 0 0 2024-04-05T21:26:50.856 INFO:tasks.workunit.client.0.smithi121.stdout:3/656: mknod d0/d3/d24/d3d/cd9 0 2024-04-05T21:26:50.858 INFO:tasks.workunit.client.0.smithi121.stdout:1/826: creat d8/d89/d9e/d2d/d84/da8/f113 x:0 0 0 2024-04-05T21:26:50.859 INFO:tasks.workunit.client.0.smithi121.stdout:7/859: creat d0/d32/d4e/d73/dbc/df0/dd9/d10f/f111 x:0 0 0 2024-04-05T21:26:50.860 INFO:tasks.workunit.client.0.smithi121.stdout:5/871: symlink dd/de/d19/d1c/d74/d10b/l135 0 2024-04-05T21:26:50.966 INFO:tasks.workunit.client.0.smithi121.stdout:3/657: creat d0/fda x:0 0 0 2024-04-05T21:26:50.967 INFO:tasks.workunit.client.0.smithi121.stdout:6/687: getdents d1/d2/d1b/d35/d39/d92/dea 0 2024-04-05T21:26:50.967 INFO:tasks.workunit.client.0.smithi121.stdout:1/827: mknod d8/d89/d9e/da3/c114 0 2024-04-05T21:26:50.967 INFO:tasks.workunit.client.0.smithi121.stdout:5/872: truncate dd/d41/dce/ff5 4981144 0 2024-04-05T21:26:50.970 INFO:tasks.workunit.client.0.smithi121.stdout:7/860: dread d0/d9/d22/d81/da2/fa5 [0,4194304] 0 2024-04-05T21:26:50.971 INFO:tasks.workunit.client.0.smithi121.stdout:3/658: symlink d0/d3/d24/d6d/d7b/da3/ldb 0 2024-04-05T21:26:50.976 INFO:tasks.workunit.client.0.smithi121.stdout:7/861: mknod d0/d9/c112 0 2024-04-05T21:26:50.977 INFO:tasks.workunit.client.0.smithi121.stdout:4/974: dwrite d5/d11/d26/d47/fdc [0,4194304] 0 2024-04-05T21:26:50.977 INFO:tasks.workunit.client.0.smithi121.stdout:8/763: dread d12/d13/d15/d1d/f4c [0,4194304] 0 2024-04-05T21:26:50.978 INFO:tasks.workunit.client.0.smithi121.stdout:7/862: fsync d0/d23/fd7 0 2024-04-05T21:26:50.981 INFO:tasks.workunit.client.0.smithi121.stdout:1/828: getdents d8/d8c/db6/d1a/d1b/d85/d9c 0 2024-04-05T21:26:50.982 INFO:tasks.workunit.client.0.smithi121.stdout:1/829: read d8/d8c/db6/d1a/f46 [1070110,69714] 0 2024-04-05T21:26:50.983 INFO:tasks.workunit.client.0.smithi121.stdout:4/975: creat d5/d11/d26/dbc/dca/de9/f143 x:0 0 0 2024-04-05T21:26:50.984 INFO:tasks.workunit.client.0.smithi121.stdout:7/863: rmdir d0/d9/d22 39 2024-04-05T21:26:50.984 INFO:tasks.workunit.client.0.smithi121.stdout:4/976: dread - d5/d11/d20/f12d zero size 2024-04-05T21:26:50.986 INFO:tasks.workunit.client.0.smithi121.stdout:4/977: write d5/d11/d20/fad [876350,69224] 0 2024-04-05T21:26:50.987 INFO:tasks.workunit.client.0.smithi121.stdout:8/764: unlink d12/d13/d15/d1d/fa3 0 2024-04-05T21:26:50.991 INFO:tasks.workunit.client.0.smithi121.stdout:3/659: creat d0/d3/d24/d3d/d1a/fdc x:0 0 0 2024-04-05T21:26:50.994 INFO:tasks.workunit.client.0.smithi121.stdout:1/830: chown d8/d8c/lb3 63111 1 2024-04-05T21:26:50.995 INFO:tasks.workunit.client.0.smithi121.stdout:4/978: dwrite d5/d11/d28/d73/f13b [0,4194304] 0 2024-04-05T21:26:50.995 INFO:tasks.workunit.client.0.smithi121.stdout:5/873: write dd/de/d19/f80 [3112203,114820] 0 2024-04-05T21:26:50.998 INFO:tasks.workunit.client.0.smithi121.stdout:8/765: mknod d12/d50/dbb/cec 0 2024-04-05T21:26:51.001 INFO:tasks.workunit.client.0.smithi121.stdout:3/660: mkdir d0/d3/d24/d3d/d1a/d1e/db1/ddd 0 2024-04-05T21:26:51.002 INFO:tasks.workunit.client.0.smithi121.stdout:6/688: truncate d1/d2/daf/d6f/dc7/d1f/f6a 446091 0 2024-04-05T21:26:51.004 INFO:tasks.workunit.client.0.smithi121.stdout:7/864: chown d0/d9/d22/d28/d3b/ca8 3 1 2024-04-05T21:26:51.011 INFO:tasks.workunit.client.0.smithi121.stdout:3/661: dwrite d0/fda [0,4194304] 0 2024-04-05T21:26:51.012 INFO:tasks.workunit.client.0.smithi121.stdout:8/766: creat d12/d13/d15/de7/fed x:0 0 0 2024-04-05T21:26:51.013 INFO:tasks.workunit.client.0.smithi121.stdout:3/662: dread - d0/fc2 zero size 2024-04-05T21:26:51.014 INFO:tasks.workunit.client.0.smithi121.stdout:8/767: chown d12/ca0 595153 1 2024-04-05T21:26:51.015 INFO:tasks.workunit.client.0.smithi121.stdout:7/865: unlink d0/d9/d22/d28/d56/db9/ddf/def/fe0 0 2024-04-05T21:26:51.020 INFO:tasks.workunit.client.0.smithi121.stdout:1/831: getdents d8/d89/d9e/d2d/d84 0 2024-04-05T21:26:51.021 INFO:tasks.workunit.client.0.smithi121.stdout:4/979: dread d5/d89/fd1 [0,4194304] 0 2024-04-05T21:26:51.021 INFO:tasks.workunit.client.0.smithi121.stdout:7/866: fsync d0/d32/d4e/d73/dbc/df0/dd9/d10f/f111 0 2024-04-05T21:26:51.022 INFO:tasks.workunit.client.0.smithi121.stdout:5/874: dwrite dd/f96 [0,4194304] 0 2024-04-05T21:26:51.025 INFO:tasks.workunit.client.0.smithi121.stdout:7/867: stat d0/d32/d4e/d73/dbc/df0 0 2024-04-05T21:26:51.028 INFO:tasks.workunit.client.0.smithi121.stdout:8/768: getdents d12/d50/dbb 0 2024-04-05T21:26:51.029 INFO:tasks.workunit.client.0.smithi121.stdout:3/663: getdents d0/d3/d24/d3f 0 2024-04-05T21:26:51.031 INFO:tasks.workunit.client.0.smithi121.stdout:5/875: rename dd/de/d19/d38/d45/d68/f91 to dd/de/d19/d3c/d8e/f136 0 2024-04-05T21:26:51.032 INFO:tasks.workunit.client.0.smithi121.stdout:8/769: fdatasync d12/d13/d15/d1d/f36 0 2024-04-05T21:26:51.035 INFO:tasks.workunit.client.0.smithi121.stdout:3/664: dread d0/d3/d24/f99 [0,4194304] 0 2024-04-05T21:26:51.037 INFO:tasks.workunit.client.0.smithi121.stdout:7/868: dwrite d0/d9/d22/d28/d56/db9/ddf/def/f46 [0,4194304] 0 2024-04-05T21:26:51.039 INFO:tasks.workunit.client.0.smithi121.stdout:5/876: mknod dd/d41/dce/c137 0 2024-04-05T21:26:51.041 INFO:tasks.workunit.client.0.smithi121.stdout:1/832: mkdir d8/d89/d9e/d2d/d84/da8/dad/dde/ddb/d115 0 2024-04-05T21:26:51.042 INFO:tasks.workunit.client.0.smithi121.stdout:3/665: creat d0/d3/d24/d3d/d1a/d1e/d86/fde x:0 0 0 2024-04-05T21:26:51.043 INFO:tasks.workunit.client.0.smithi121.stdout:5/877: dread - dd/de/d19/d3c/f109 zero size 2024-04-05T21:26:51.044 INFO:tasks.workunit.client.0.smithi121.stdout:1/833: read d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b/f10e [3722724,69228] 0 2024-04-05T21:26:51.048 INFO:tasks.workunit.client.0.smithi121.stdout:3/666: symlink d0/d3/d24/d6d/d7b/da3/ldf 0 2024-04-05T21:26:51.049 INFO:tasks.workunit.client.0.smithi121.stdout:3/667: stat d0/dc6 0 2024-04-05T21:26:51.050 INFO:tasks.workunit.client.0.smithi121.stdout:3/668: fdatasync d0/f27 0 2024-04-05T21:26:51.054 INFO:tasks.workunit.client.0.smithi121.stdout:2/757: sync 2024-04-05T21:26:51.056 INFO:tasks.workunit.client.0.smithi121.stdout:1/834: creat d8/d8c/db6/d1a/d1b/d4c/d76/da4/f116 x:0 0 0 2024-04-05T21:26:51.057 INFO:tasks.workunit.client.0.smithi121.stdout:3/669: mknod d0/d3/d24/d6d/ce0 0 2024-04-05T21:26:51.058 INFO:tasks.workunit.client.0.smithi121.stdout:5/878: creat dd/de/d19/d38/f138 x:0 0 0 2024-04-05T21:26:51.058 INFO:tasks.workunit.client.0.smithi121.stdout:2/758: mkdir d2/d9/d38/d2d/d4c/de9/d5c/df0 0 2024-04-05T21:26:51.059 INFO:tasks.workunit.client.1.smithi188.stdout:0/954: sync 2024-04-05T21:26:51.060 INFO:tasks.workunit.client.0.smithi121.stdout:8/770: dread d12/d50/d93/d96/fab [0,4194304] 0 2024-04-05T21:26:51.061 INFO:tasks.workunit.client.1.smithi188.stdout:0/955: mkdir d4/d8/db/d21/d3c/d161 0 2024-04-05T21:26:51.062 INFO:tasks.workunit.client.0.smithi121.stdout:3/670: rmdir d0/d3/d24/d6d/d7b 39 2024-04-05T21:26:51.062 INFO:tasks.workunit.client.1.smithi188.stdout:0/956: mkdir d4/d8/db/d42/d38/d162 0 2024-04-05T21:26:51.064 INFO:tasks.workunit.client.0.smithi121.stdout:1/835: write d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b/ff3 [750783,53563] 0 2024-04-05T21:26:51.064 INFO:tasks.workunit.client.0.smithi121.stdout:5/879: fdatasync dd/de/d19/d1c/d36/f12c 0 2024-04-05T21:26:51.065 INFO:tasks.workunit.client.0.smithi121.stdout:2/759: mknod d2/d9/d38/d2d/d4c/d7c/dc2/cf1 0 2024-04-05T21:26:51.066 INFO:tasks.workunit.client.0.smithi121.stdout:7/869: dwrite d0/d9/f20 [0,4194304] 0 2024-04-05T21:26:51.067 INFO:tasks.workunit.client.0.smithi121.stdout:8/771: creat d12/d50/dbb/fee x:0 0 0 2024-04-05T21:26:51.072 INFO:tasks.workunit.client.0.smithi121.stdout:3/671: write d0/d3/d24/d3d/d1a/d23/d90/d9e/fa0 [10891,105464] 0 2024-04-05T21:26:51.074 INFO:tasks.workunit.client.0.smithi121.stdout:3/672: fsync d0/d3/d24/d3d/d1a/d1e/d86/f8b 0 2024-04-05T21:26:51.075 INFO:tasks.workunit.client.0.smithi121.stdout:2/760: fsync d2/d9/d1c/d30/d59/f89 0 2024-04-05T21:26:51.075 INFO:tasks.workunit.client.0.smithi121.stdout:7/870: symlink d0/d32/d4a/l113 0 2024-04-05T21:26:51.076 INFO:tasks.workunit.client.0.smithi121.stdout:1/836: fsync d8/d89/f98 0 2024-04-05T21:26:51.077 INFO:tasks.workunit.client.0.smithi121.stdout:5/880: fdatasync dd/de/fc3 0 2024-04-05T21:26:51.079 INFO:tasks.workunit.client.0.smithi121.stdout:2/761: write d2/d9/d17/d3a/f4a [4084842,105144] 0 2024-04-05T21:26:51.079 INFO:tasks.workunit.client.0.smithi121.stdout:8/772: unlink d12/d13/d15/l16 0 2024-04-05T21:26:51.085 INFO:tasks.workunit.client.0.smithi121.stdout:3/673: rename d0/d3/d24/d3d/d1a/d1e/d86/d8e to d0/d3/d24/d3d/d1a/d1e/db1/ddd/de1 0 2024-04-05T21:26:51.085 INFO:tasks.workunit.client.0.smithi121.stdout:7/871: fsync d0/d9/d22/d28/d56/db9/ddf/def/d52/d61/dcb/fe2 0 2024-04-05T21:26:51.087 INFO:tasks.workunit.client.0.smithi121.stdout:5/881: mknod dd/de/d19/d38/d9b/c139 0 2024-04-05T21:26:51.089 INFO:tasks.workunit.client.0.smithi121.stdout:1/837: dwrite d8/d89/d9e/da3/fbf [0,4194304] 0 2024-04-05T21:26:51.089 INFO:tasks.workunit.client.0.smithi121.stdout:8/773: creat d12/d50/d6a/d63/d8b/fef x:0 0 0 2024-04-05T21:26:51.091 INFO:tasks.workunit.client.0.smithi121.stdout:7/872: creat d0/d9/d22/d81/da2/f114 x:0 0 0 2024-04-05T21:26:51.092 INFO:tasks.workunit.client.0.smithi121.stdout:8/774: write d12/d13/fe2 [367948,103995] 0 2024-04-05T21:26:51.095 INFO:tasks.workunit.client.0.smithi121.stdout:3/674: mkdir d0/d3/de/da6/de2 0 2024-04-05T21:26:51.097 INFO:tasks.workunit.client.0.smithi121.stdout:7/873: creat d0/d32/d4e/d73/dbc/f115 x:0 0 0 2024-04-05T21:26:51.098 INFO:tasks.workunit.client.0.smithi121.stdout:8/775: rename d12/d13/d15/d1d to d12/d13/dd3/df0 0 2024-04-05T21:26:51.100 INFO:tasks.workunit.client.0.smithi121.stdout:7/874: chown d0/d9/c1b 1 1 2024-04-05T21:26:51.100 INFO:tasks.workunit.client.0.smithi121.stdout:8/776: write d12/d50/d61/d7f/d9a/fb0 [2659977,121246] 0 2024-04-05T21:26:51.102 INFO:tasks.workunit.client.0.smithi121.stdout:8/777: write d12/d50/f79 [302818,74369] 0 2024-04-05T21:26:51.103 INFO:tasks.workunit.client.0.smithi121.stdout:7/875: dread - d0/d32/d4e/d73/dbc/df0/dd9/d10f/f111 zero size 2024-04-05T21:26:51.104 INFO:tasks.workunit.client.0.smithi121.stdout:3/675: mkdir d0/d3/d24/d6d/d7b/de3 0 2024-04-05T21:26:51.106 INFO:tasks.workunit.client.0.smithi121.stdout:2/762: truncate d2/d9/d38/d2d/d47/d94/f37 2978102 0 2024-04-05T21:26:51.108 INFO:tasks.workunit.client.0.smithi121.stdout:2/763: truncate d2/d9/d1c/d30/fb1 341493 0 2024-04-05T21:26:51.113 INFO:tasks.workunit.client.0.smithi121.stdout:1/838: dread d8/d8c/db6/d1a/f5a [0,4194304] 0 2024-04-05T21:26:51.117 INFO:tasks.workunit.client.0.smithi121.stdout:7/876: getdents d0/d9/d22/d3e/d76/de1 0 2024-04-05T21:26:51.118 INFO:tasks.workunit.client.0.smithi121.stdout:2/764: creat d2/ff2 x:0 0 0 2024-04-05T21:26:51.118 INFO:tasks.workunit.client.0.smithi121.stdout:5/882: dread dd/de/d19/f6b [0,4194304] 0 2024-04-05T21:26:51.120 INFO:tasks.workunit.client.0.smithi121.stdout:2/765: readlink d2/d1e/l57 0 2024-04-05T21:26:51.122 INFO:tasks.workunit.client.0.smithi121.stdout:1/839: getdents d8/d8c/db6/d1a/d1b/d24/d79 0 2024-04-05T21:26:51.123 INFO:tasks.workunit.client.0.smithi121.stdout:5/883: write dd/de/d6d/f121 [354782,68991] 0 2024-04-05T21:26:51.124 INFO:tasks.workunit.client.0.smithi121.stdout:2/766: chown d2/d9/d38/d2d/d4c/d7c/ca8 0 1 2024-04-05T21:26:51.126 INFO:tasks.workunit.client.0.smithi121.stdout:3/676: dwrite d0/d3/d24/d3d/d1a/d23/d90/d9e/fad [4194304,4194304] 0 2024-04-05T21:26:51.127 INFO:tasks.workunit.client.0.smithi121.stdout:8/778: dread d12/d13/d15/f83 [0,4194304] 0 2024-04-05T21:26:51.129 INFO:tasks.workunit.client.0.smithi121.stdout:7/877: dread d0/f15 [4194304,4194304] 0 2024-04-05T21:26:51.131 INFO:tasks.workunit.client.0.smithi121.stdout:2/767: creat d2/d9/d38/d2d/d4c/d7c/dc2/ff3 x:0 0 0 2024-04-05T21:26:51.133 INFO:tasks.workunit.client.0.smithi121.stdout:2/768: chown d2/d9/d1c/d30/d59/f63 18200356 1 2024-04-05T21:26:51.133 INFO:tasks.workunit.client.0.smithi121.stdout:7/878: fdatasync d0/d9/d22/d28/d56/db9/fea 0 2024-04-05T21:26:51.136 INFO:tasks.workunit.client.0.smithi121.stdout:5/884: rename dd/de/d19/d13/lf1 to dd/de/d19/d38/d45/d68/dcf/l13a 0 2024-04-05T21:26:51.136 INFO:tasks.workunit.client.0.smithi121.stdout:7/879: truncate d0/d32/d4a/feb 671936 0 2024-04-05T21:26:51.141 INFO:tasks.workunit.client.0.smithi121.stdout:3/677: mknod d0/d3/d24/d3d/dcf/ce4 0 2024-04-05T21:26:51.141 INFO:tasks.workunit.client.0.smithi121.stdout:8/779: creat d12/d50/d93/ff1 x:0 0 0 2024-04-05T21:26:51.141 INFO:tasks.workunit.client.0.smithi121.stdout:2/769: link d2/d9/d1c/d30/f68 d2/d9/d1c/d30/d59/ff4 0 2024-04-05T21:26:51.144 INFO:tasks.workunit.client.0.smithi121.stdout:7/880: unlink d0/d9/d22/d28/d56/db9/ddf/def/d52/d61/fce 0 2024-04-05T21:26:51.145 INFO:tasks.workunit.client.0.smithi121.stdout:1/840: dread d8/d89/fa5 [0,4194304] 0 2024-04-05T21:26:51.147 INFO:tasks.workunit.client.0.smithi121.stdout:3/678: dwrite d0/d3/d24/d3d/d1a/d1e/d86/f8b [0,4194304] 0 2024-04-05T21:26:51.148 INFO:tasks.workunit.client.0.smithi121.stdout:1/841: dread - d8/d8c/db6/d1a/d1b/d85/ff8 zero size 2024-04-05T21:26:51.149 INFO:tasks.workunit.client.0.smithi121.stdout:7/881: rmdir d0/d57 39 2024-04-05T21:26:51.151 INFO:tasks.workunit.client.0.smithi121.stdout:3/679: creat d0/d3/d24/d3d/d1a/d1e/fe5 x:0 0 0 2024-04-05T21:26:51.154 INFO:tasks.workunit.client.0.smithi121.stdout:1/842: creat d8/d8c/db6/d1a/d1b/dbb/f117 x:0 0 0 2024-04-05T21:26:51.156 INFO:tasks.workunit.client.0.smithi121.stdout:5/885: dwrite dd/de/d105/f117 [4194304,4194304] 0 2024-04-05T21:26:51.156 INFO:tasks.workunit.client.0.smithi121.stdout:7/882: creat d0/d57/ded/f116 x:0 0 0 2024-04-05T21:26:51.158 INFO:tasks.workunit.client.0.smithi121.stdout:2/770: link d2/d9/d38/d2d/d47/d94/d84/l9e d2/d9/d1c/d85/lf5 0 2024-04-05T21:26:51.162 INFO:tasks.workunit.client.0.smithi121.stdout:7/883: symlink d0/d9/d22/d28/d56/l117 0 2024-04-05T21:26:51.162 INFO:tasks.workunit.client.0.smithi121.stdout:1/843: getdents d8/d89/d9e/da3/df2 0 2024-04-05T21:26:51.163 INFO:tasks.workunit.client.0.smithi121.stdout:7/884: chown d0/d23 60068 1 2024-04-05T21:26:51.165 INFO:tasks.workunit.client.0.smithi121.stdout:2/771: creat d2/d9/d38/d2d/d4c/de5/ff6 x:0 0 0 2024-04-05T21:26:51.166 INFO:tasks.workunit.client.0.smithi121.stdout:7/885: mkdir d0/d9/d22/d81/d118 0 2024-04-05T21:26:51.167 INFO:tasks.workunit.client.0.smithi121.stdout:7/886: read d0/d23/f33 [4556387,71421] 0 2024-04-05T21:26:51.167 INFO:tasks.workunit.client.0.smithi121.stdout:1/844: unlink d8/d8c/db6/d1a/d1b/d4c/d76/fce 0 2024-04-05T21:26:51.169 INFO:tasks.workunit.client.0.smithi121.stdout:3/680: link d0/d3/d24/d3d/d1a/d23/d38/d55/cb5 d0/d3/d24/db3/ce6 0 2024-04-05T21:26:51.171 INFO:tasks.workunit.client.0.smithi121.stdout:5/886: write dd/de/d19/d1c/d74/fa4 [183276,107712] 0 2024-04-05T21:26:51.172 INFO:tasks.workunit.client.0.smithi121.stdout:7/887: rename d0/f3 to d0/d57/de7/f119 0 2024-04-05T21:26:51.173 INFO:tasks.workunit.client.0.smithi121.stdout:7/888: chown d0/d32/d4e 11 1 2024-04-05T21:26:51.174 INFO:tasks.workunit.client.0.smithi121.stdout:7/889: stat d0/d23/f2e 0 2024-04-05T21:26:51.174 INFO:tasks.workunit.client.0.smithi121.stdout:2/772: mknod d2/d9/d38/d2d/d4c/de9/d8d/cf7 0 2024-04-05T21:26:51.175 INFO:tasks.workunit.client.0.smithi121.stdout:2/773: fsync d2/d9/d1c/d30/dba/fb5 0 2024-04-05T21:26:51.179 INFO:tasks.workunit.client.0.smithi121.stdout:1/845: creat d8/d8c/db6/d1a/d1b/d4c/d76/d53/f118 x:0 0 0 2024-04-05T21:26:51.180 INFO:tasks.workunit.client.0.smithi121.stdout:3/681: creat d0/d3/fe7 x:0 0 0 2024-04-05T21:26:51.181 INFO:tasks.workunit.client.0.smithi121.stdout:2/774: symlink d2/d9/d38/d2d/d4c/de5/lf8 0 2024-04-05T21:26:51.182 INFO:tasks.workunit.client.0.smithi121.stdout:7/890: dwrite d0/d32/d4a/feb [0,4194304] 0 2024-04-05T21:26:51.184 INFO:tasks.workunit.client.0.smithi121.stdout:1/846: mkdir d8/d89/d9e/dc8/d119 0 2024-04-05T21:26:51.186 INFO:tasks.workunit.client.0.smithi121.stdout:2/775: symlink d2/db6/lf9 0 2024-04-05T21:26:51.187 INFO:tasks.workunit.client.0.smithi121.stdout:7/891: creat d0/d9/d22/d28/d56/f11a x:0 0 0 2024-04-05T21:26:51.190 INFO:tasks.workunit.client.0.smithi121.stdout:3/682: rename d0/d3/d24/d6d/d7b/da4/fb9 to d0/d3/d24/db3/fe8 0 2024-04-05T21:26:51.191 INFO:tasks.workunit.client.0.smithi121.stdout:3/683: chown d0/d3/d24/d3d/d1a/d1e/c35 52894244 1 2024-04-05T21:26:51.193 INFO:tasks.workunit.client.0.smithi121.stdout:5/887: write dd/f1f [1116432,39563] 0 2024-04-05T21:26:51.195 INFO:tasks.workunit.client.0.smithi121.stdout:2/776: symlink d2/d9/lfa 0 2024-04-05T21:26:51.195 INFO:tasks.workunit.client.0.smithi121.stdout:3/684: creat d0/d3/de/da6/de2/fe9 x:0 0 0 2024-04-05T21:26:51.196 INFO:tasks.workunit.client.0.smithi121.stdout:5/888: readlink dd/l6f 0 2024-04-05T21:26:51.197 INFO:tasks.workunit.client.0.smithi121.stdout:2/777: symlink d2/d9/d77/lfb 0 2024-04-05T21:26:51.197 INFO:tasks.workunit.client.0.smithi121.stdout:5/889: readlink dd/de/d19/d38/da3/lde 0 2024-04-05T21:26:51.199 INFO:tasks.workunit.client.0.smithi121.stdout:3/685: creat d0/d3/d24/d3d/d1a/d1e/db1/fea x:0 0 0 2024-04-05T21:26:51.199 INFO:tasks.workunit.client.0.smithi121.stdout:5/890: stat dd/de/d19/d3c/d8e/cf4 0 2024-04-05T21:26:51.200 INFO:tasks.workunit.client.0.smithi121.stdout:2/778: write d2/d9/d38/d2d/f72 [2353118,97821] 0 2024-04-05T21:26:51.202 INFO:tasks.workunit.client.0.smithi121.stdout:3/686: write d0/fda [233816,79776] 0 2024-04-05T21:26:51.204 INFO:tasks.workunit.client.0.smithi121.stdout:5/891: symlink dd/de/dc1/d104/l13b 0 2024-04-05T21:26:51.205 INFO:tasks.workunit.client.0.smithi121.stdout:3/687: stat d0/d3/d24/d6d/d7b/fc7 0 2024-04-05T21:26:51.206 INFO:tasks.workunit.client.1.smithi188.stdout:8/826: sync 2024-04-05T21:26:51.207 INFO:tasks.workunit.client.0.smithi121.stdout:2/779: creat d2/d9/ffc x:0 0 0 2024-04-05T21:26:51.207 INFO:tasks.workunit.client.0.smithi121.stdout:5/892: read dd/f6c [3093780,1041] 0 2024-04-05T21:26:51.210 INFO:tasks.workunit.client.1.smithi188.stdout:8/827: getdents d5/d21/d40/d43/d44/d59/d79 0 2024-04-05T21:26:51.211 INFO:tasks.workunit.client.0.smithi121.stdout:1/847: dwrite d8/d8c/db6/d1a/d1b/d24/f8b [0,4194304] 0 2024-04-05T21:26:51.212 INFO:tasks.workunit.client.0.smithi121.stdout:7/892: dread d0/d57/de7/f119 [0,4194304] 0 2024-04-05T21:26:51.212 INFO:tasks.workunit.client.0.smithi121.stdout:2/780: creat d2/d9/dcb/ffd x:0 0 0 2024-04-05T21:26:51.213 INFO:tasks.workunit.client.1.smithi188.stdout:8/828: unlink d5/d21/d40/d43/l9a 0 2024-04-05T21:26:51.214 INFO:tasks.workunit.client.0.smithi121.stdout:1/848: chown d8/d8c/db6/d1a/f46 31885681 1 2024-04-05T21:26:51.214 INFO:tasks.workunit.client.0.smithi121.stdout:5/893: mknod dd/d41/dce/da6/dd3/c13c 0 2024-04-05T21:26:51.218 INFO:tasks.workunit.client.0.smithi121.stdout:5/894: fsync dd/d41/dce/ff5 0 2024-04-05T21:26:51.219 INFO:tasks.workunit.client.0.smithi121.stdout:3/688: dread d0/d3/d24/d3d/d1a/d23/d90/d9e/fad [4194304,4194304] 0 2024-04-05T21:26:51.220 INFO:tasks.workunit.client.0.smithi121.stdout:3/689: dread - d0/f97 zero size 2024-04-05T21:26:51.220 INFO:tasks.workunit.client.0.smithi121.stdout:7/893: symlink d0/d9/d22/d28/d3b/l11b 0 2024-04-05T21:26:51.220 INFO:tasks.workunit.client.0.smithi121.stdout:1/849: read d8/d10/f17 [2562240,100822] 0 2024-04-05T21:26:51.221 INFO:tasks.workunit.client.1.smithi188.stdout:8/829: dwrite d5/d21/d40/d43/d44/d5b/d99/da3/fba [0,4194304] 0 2024-04-05T21:26:51.222 INFO:tasks.workunit.client.0.smithi121.stdout:2/781: truncate d2/d9/d38/d2d/d4c/de9/d8d/fd6 920469 0 2024-04-05T21:26:51.222 INFO:tasks.workunit.client.0.smithi121.stdout:5/895: creat dd/de/d19/d38/de3/f13d x:0 0 0 2024-04-05T21:26:51.223 INFO:tasks.workunit.client.1.smithi188.stdout:8/830: dread d5/d21/d2a/f97 [0,4194304] 0 2024-04-05T21:26:51.225 INFO:tasks.workunit.client.0.smithi121.stdout:5/896: fdatasync dd/de/d19/d1c/f75 0 2024-04-05T21:26:51.227 INFO:tasks.workunit.client.1.smithi188.stdout:8/831: mkdir d5/d21/d40/d43/d48/d7e/d10a 0 2024-04-05T21:26:51.230 INFO:tasks.workunit.client.1.smithi188.stdout:8/832: truncate d5/d21/d40/d43/f83 631851 0 2024-04-05T21:26:51.231 INFO:tasks.workunit.client.0.smithi121.stdout:5/897: mknod dd/de/d19/d1c/d74/d10b/c13e 0 2024-04-05T21:26:51.232 INFO:tasks.workunit.client.1.smithi188.stdout:8/833: fsync d5/d21/d40/d43/d48/fde 0 2024-04-05T21:26:51.232 INFO:tasks.workunit.client.0.smithi121.stdout:2/782: dwrite d2/d9/d1c/d30/d59/fbe [0,4194304] 0 2024-04-05T21:26:51.232 INFO:tasks.workunit.client.1.smithi188.stdout:8/834: dread - d5/d21/d2a/d95/dc0/ff0 zero size 2024-04-05T21:26:51.234 INFO:tasks.workunit.client.0.smithi121.stdout:2/783: dread - d2/d9/d77/fd2 zero size 2024-04-05T21:26:51.238 INFO:tasks.workunit.client.0.smithi121.stdout:3/690: truncate d0/d3/d24/f89 3255471 0 2024-04-05T21:26:51.238 INFO:tasks.workunit.client.0.smithi121.stdout:2/784: rmdir d2/d9/d17 39 2024-04-05T21:26:51.239 INFO:tasks.workunit.client.1.smithi188.stdout:8/835: write d5/d21/d40/d43/f66 [458230,82709] 0 2024-04-05T21:26:51.240 INFO:tasks.workunit.client.0.smithi121.stdout:1/850: truncate d8/d8c/db6/d1a/d1b/d24/f2a 412864 0 2024-04-05T21:26:51.242 INFO:tasks.workunit.client.1.smithi188.stdout:8/836: creat d5/d21/d40/f10b x:0 0 0 2024-04-05T21:26:51.243 INFO:tasks.workunit.client.0.smithi121.stdout:5/898: write dd/de/d19/d13/f26 [2590234,104772] 0 2024-04-05T21:26:51.243 INFO:tasks.workunit.client.0.smithi121.stdout:7/894: dwrite d0/d9/d22/d81/da2/fd5 [0,4194304] 0 2024-04-05T21:26:51.244 INFO:tasks.workunit.client.0.smithi121.stdout:2/785: mknod d2/d9/d38/d2d/d4c/d7c/d7e/db9/cfe 0 2024-04-05T21:26:51.245 INFO:tasks.workunit.client.0.smithi121.stdout:2/786: write d2/d9/d38/fdf [494275,379] 0 2024-04-05T21:26:51.245 INFO:tasks.workunit.client.1.smithi188.stdout:8/837: truncate d5/d21/d40/d43/d44/d5b/d99/d69/ff6 523168 0 2024-04-05T21:26:51.246 INFO:tasks.workunit.client.1.smithi188.stdout:8/838: chown d5/d21/d80/fb2 1 1 2024-04-05T21:26:51.246 INFO:tasks.workunit.client.0.smithi121.stdout:3/691: read d0/d3/f5b [801969,87790] 0 2024-04-05T21:26:51.248 INFO:tasks.workunit.client.0.smithi121.stdout:2/787: rename d2/d9/d1c/d85/def to d2/d9/d1c/d85/def/dff 22 2024-04-05T21:26:51.249 INFO:tasks.workunit.client.1.smithi188.stdout:0/957: sync 2024-04-05T21:26:51.250 INFO:tasks.workunit.client.0.smithi121.stdout:7/895: symlink d0/d9/l11c 0 2024-04-05T21:26:51.252 INFO:tasks.workunit.client.0.smithi121.stdout:5/899: dread dd/de/d39/d3b/fc0 [0,4194304] 0 2024-04-05T21:26:51.252 INFO:tasks.workunit.client.0.smithi121.stdout:7/896: stat d0/d9/d22/d3e/d76/de1 0 2024-04-05T21:26:51.255 INFO:tasks.workunit.client.1.smithi188.stdout:0/958: truncate d4/d8/d78/dc2/d11a/f104 1720200 0 2024-04-05T21:26:51.257 INFO:tasks.workunit.client.0.smithi121.stdout:2/788: creat d2/d9/d38/d2d/d4c/de9/d5c/f100 x:0 0 0 2024-04-05T21:26:51.257 INFO:tasks.workunit.client.1.smithi188.stdout:0/959: symlink d4/d8/db/d21/d4a/d73/d116/d24/d57/d63/l163 0 2024-04-05T21:26:51.258 INFO:tasks.workunit.client.1.smithi188.stdout:0/960: chown d4/d8/db/d21/d4a/d73/d116/d24/d57/fc8 206739139 1 2024-04-05T21:26:51.260 INFO:tasks.workunit.client.1.smithi188.stdout:0/961: rename d4/f5f to d4/d8/d78/f164 0 2024-04-05T21:26:51.262 INFO:tasks.workunit.client.0.smithi121.stdout:3/692: dwrite d0/d3/d24/f3c [4194304,4194304] 0 2024-04-05T21:26:51.264 INFO:tasks.workunit.client.1.smithi188.stdout:0/962: unlink d4/d8/d10/l118 0 2024-04-05T21:26:51.266 INFO:tasks.workunit.client.0.smithi121.stdout:3/693: dread - d0/d3/de/da6/de2/fe9 zero size 2024-04-05T21:26:51.266 INFO:tasks.workunit.client.1.smithi188.stdout:0/963: chown d4/d8/db/d21/d4a/d73/d116/dd1/ff9 11896 1 2024-04-05T21:26:51.267 INFO:tasks.workunit.client.0.smithi121.stdout:2/789: dread d2/d9/d38/f35 [4194304,4194304] 0 2024-04-05T21:26:51.267 INFO:tasks.workunit.client.0.smithi121.stdout:7/897: dwrite d0/d32/fa0 [0,4194304] 0 2024-04-05T21:26:51.268 INFO:tasks.workunit.client.1.smithi188.stdout:0/964: write d4/d8/db/d21/d3c/fa9 [713088,62107] 0 2024-04-05T21:26:51.270 INFO:tasks.workunit.client.0.smithi121.stdout:1/851: dread d8/d8c/db6/d1a/f66 [0,4194304] 0 2024-04-05T21:26:51.274 INFO:tasks.workunit.client.0.smithi121.stdout:2/790: readlink d2/d9/d1c/d30/dba/lc7 0 2024-04-05T21:26:51.274 INFO:tasks.workunit.client.0.smithi121.stdout:3/694: rename d0/d3/d24/l5d to d0/d3/d24/d3d/d1a/d23/leb 0 2024-04-05T21:26:51.281 INFO:tasks.workunit.client.0.smithi121.stdout:2/791: creat d2/d9/d38/d2d/d47/f101 x:0 0 0 2024-04-05T21:26:51.281 INFO:tasks.workunit.client.0.smithi121.stdout:1/852: creat d8/d89/d9e/d2d/d84/da8/dad/dde/ddb/d115/f11a x:0 0 0 2024-04-05T21:26:51.281 INFO:tasks.workunit.client.0.smithi121.stdout:5/900: dwrite dd/de/d19/d13/d53/f7f [0,4194304] 0 2024-04-05T21:26:51.282 INFO:tasks.workunit.client.0.smithi121.stdout:2/792: fdatasync d2/d9/d1c/d30/f5e 0 2024-04-05T21:26:51.286 INFO:tasks.workunit.client.0.smithi121.stdout:5/901: mkdir dd/de/dc1/d104/d13f 0 2024-04-05T21:26:51.289 INFO:tasks.workunit.client.0.smithi121.stdout:7/898: truncate d0/d9/d22/f6c 2858126 0 2024-04-05T21:26:51.290 INFO:tasks.workunit.client.0.smithi121.stdout:1/853: write d8/d8c/db6/f19 [38397,115216] 0 2024-04-05T21:26:51.294 INFO:tasks.workunit.client.0.smithi121.stdout:5/902: symlink dd/de/d19/d1c/d7b/d88/l140 0 2024-04-05T21:26:51.296 INFO:tasks.workunit.client.0.smithi121.stdout:7/899: rename d0/d9/d22/f44 to d0/d23/da4/f11d 0 2024-04-05T21:26:51.296 INFO:tasks.workunit.client.0.smithi121.stdout:2/793: dwrite d2/d9/ff [0,4194304] 0 2024-04-05T21:26:51.310 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:51 smithi121 ceph-mon[39723]: pgmap v24: 65 pgs: 65 active+clean; 3.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 143 MiB/s rd, 395 MiB/s wr, 786 op/s 2024-04-05T21:26:51.332 INFO:tasks.workunit.client.0.smithi121.stdout:9/602: sync 2024-04-05T21:26:51.334 INFO:tasks.workunit.client.0.smithi121.stdout:2/794: dread d2/d9/d38/d2d/f58 [0,4194304] 0 2024-04-05T21:26:51.347 INFO:tasks.workunit.client.1.smithi188.stdout:8/839: write d5/d21/d40/d43/f51 [725879,46508] 0 2024-04-05T21:26:51.400 INFO:tasks.workunit.client.1.smithi188.stdout:0/965: dwrite d4/d8/d10/fb2 [0,4194304] 0 2024-04-05T21:26:51.402 INFO:tasks.workunit.client.1.smithi188.stdout:0/966: chown d4/d8/db/d42/d38/d162 24 1 2024-04-05T21:26:51.407 INFO:tasks.workunit.client.1.smithi188.stdout:0/967: unlink d4/d8/db/d21/d4a/d73/d116/de8/f12e 0 2024-04-05T21:26:51.424 INFO:tasks.workunit.client.1.smithi188.stdout:0/968: dread d4/d8/db/d21/d4a/d73/d116/dd1/f10c [0,4194304] 0 2024-04-05T21:26:51.426 INFO:tasks.workunit.client.1.smithi188.stdout:0/969: mknod d4/d8/db/d21/d4a/d73/d116/d24/ddc/c165 0 2024-04-05T21:26:51.431 INFO:tasks.workunit.client.1.smithi188.stdout:0/970: dread d4/d8/d78/dc2/d11a/d80/f9b [0,4194304] 0 2024-04-05T21:26:51.435 INFO:tasks.workunit.client.1.smithi188.stdout:0/971: link d4/d8/db/d21/d3c/fa9 d4/d8/db/d21/d4a/d73/d116/d24/ddc/f166 0 2024-04-05T21:26:51.437 INFO:tasks.workunit.client.1.smithi188.stdout:0/972: fdatasync d4/d8/db/d21/d3c/fa5 0 2024-04-05T21:26:51.442 INFO:tasks.workunit.client.1.smithi188.stdout:0/973: creat d4/d6/d13/dca/d125/f167 x:0 0 0 2024-04-05T21:26:51.442 INFO:tasks.workunit.client.1.smithi188.stdout:0/974: truncate d4/d8/db/d21/d4a/d73/d116/dd1/fd5 851284 0 2024-04-05T21:26:51.443 INFO:tasks.workunit.client.1.smithi188.stdout:0/975: chown d4/d8/db/c16 453 1 2024-04-05T21:26:51.478 INFO:tasks.workunit.client.0.smithi121.stdout:7/900: rmdir d0/d32/d4e/d73/dbc 39 2024-04-05T21:26:51.479 INFO:tasks.workunit.client.0.smithi121.stdout:7/901: chown d0/d9/d22/d28/d56/db9/ddf/def/d52/d61/dcb/fdc 6713930 1 2024-04-05T21:26:51.480 INFO:tasks.workunit.client.0.smithi121.stdout:5/903: symlink dd/de/d19/d1c/d74/d8c/dc5/l141 0 2024-04-05T21:26:51.484 INFO:tasks.workunit.client.0.smithi121.stdout:9/603: creat d6/d32/dbe/d7b/dc4/fd0 x:0 0 0 2024-04-05T21:26:51.485 INFO:tasks.workunit.client.0.smithi121.stdout:9/604: chown d6/d32/c45 1 1 2024-04-05T21:26:51.526 INFO:tasks.workunit.client.0.smithi121.stdout:7/902: dwrite d0/d9/d22/fa3 [0,4194304] 0 2024-04-05T21:26:51.527 INFO:tasks.workunit.client.0.smithi121.stdout:2/795: getdents d2/d9/d17/de2 0 2024-04-05T21:26:51.529 INFO:tasks.workunit.client.0.smithi121.stdout:9/605: creat d6/df/d48/d96/db2/fd1 x:0 0 0 2024-04-05T21:26:51.529 INFO:tasks.workunit.client.0.smithi121.stdout:2/796: readlink d2/d9/d38/d2d/d4c/de9/d62/lc4 0 2024-04-05T21:26:51.530 INFO:tasks.workunit.client.0.smithi121.stdout:6/689: sync 2024-04-05T21:26:51.530 INFO:tasks.workunit.client.0.smithi121.stdout:0/868: sync 2024-04-05T21:26:51.532 INFO:tasks.workunit.client.0.smithi121.stdout:6/690: chown d1/d2/d1b/d2e/dd9/de0 19 1 2024-04-05T21:26:51.532 INFO:tasks.workunit.client.0.smithi121.stdout:7/903: truncate d0/d32/f8a 405581 0 2024-04-05T21:26:51.533 INFO:tasks.workunit.client.0.smithi121.stdout:4/980: sync 2024-04-05T21:26:51.539 INFO:tasks.workunit.client.0.smithi121.stdout:2/797: dwrite d2/d9/d38/f35 [0,4194304] 0 2024-04-05T21:26:51.539 INFO:tasks.workunit.client.0.smithi121.stdout:6/691: mknod d1/d2/dc/d9a/dc0/cf1 0 2024-04-05T21:26:51.543 INFO:tasks.workunit.client.0.smithi121.stdout:7/904: truncate d0/d9/d22/d28/d56/db9/ddf/def/f4b 392126 0 2024-04-05T21:26:51.544 INFO:tasks.workunit.client.0.smithi121.stdout:2/798: symlink d2/d9/d1c/l102 0 2024-04-05T21:26:51.548 INFO:tasks.workunit.client.0.smithi121.stdout:0/869: dwrite d7/d71/d94/dd1/dd4/ffb [0,4194304] 0 2024-04-05T21:26:51.549 INFO:tasks.workunit.client.0.smithi121.stdout:2/799: write d2/d9/d38/d2d/d4c/de5/ff6 [483891,127964] 0 2024-04-05T21:26:51.550 INFO:tasks.workunit.client.0.smithi121.stdout:0/870: readlink d7/dc/le 0 2024-04-05T21:26:51.553 INFO:tasks.workunit.client.0.smithi121.stdout:7/905: symlink d0/d9/d22/d28/d56/db9/ddf/def/d52/d61/dcb/l11e 0 2024-04-05T21:26:51.555 INFO:tasks.workunit.client.0.smithi121.stdout:0/871: chown d7/dc/l129 93234896 1 2024-04-05T21:26:51.557 INFO:tasks.workunit.client.0.smithi121.stdout:6/692: dwrite d1/d2/d1b/d35/d74/f7d [0,4194304] 0 2024-04-05T21:26:51.559 INFO:tasks.workunit.client.0.smithi121.stdout:2/800: truncate d2/d9/d17/d3a/f55 397660 0 2024-04-05T21:26:51.561 INFO:tasks.workunit.client.0.smithi121.stdout:0/872: creat d7/d7e/d97/f132 x:0 0 0 2024-04-05T21:26:51.565 INFO:tasks.workunit.client.0.smithi121.stdout:4/981: dread d5/d11/d26/d47/df7/fcb [0,4194304] 0 2024-04-05T21:26:51.569 INFO:tasks.workunit.client.0.smithi121.stdout:1/854: sync 2024-04-05T21:26:51.569 INFO:tasks.workunit.client.0.smithi121.stdout:3/695: sync 2024-04-05T21:26:51.569 INFO:tasks.workunit.client.0.smithi121.stdout:8/780: sync 2024-04-05T21:26:51.570 INFO:tasks.workunit.client.0.smithi121.stdout:3/696: stat d0/c4a 0 2024-04-05T21:26:51.571 INFO:tasks.workunit.client.0.smithi121.stdout:3/697: dread - d0/d3/d24/db3/fe8 zero size 2024-04-05T21:26:51.573 INFO:tasks.workunit.client.0.smithi121.stdout:3/698: dread - d0/d3/d24/db3/fe8 zero size 2024-04-05T21:26:51.575 INFO:tasks.workunit.client.0.smithi121.stdout:1/855: creat d8/d89/d9e/d2d/d84/da8/dad/f11b x:0 0 0 2024-04-05T21:26:51.579 INFO:tasks.workunit.client.0.smithi121.stdout:8/781: rename d12/d13/dd3/df0/f94 to d12/d50/ff2 0 2024-04-05T21:26:51.582 INFO:tasks.workunit.client.0.smithi121.stdout:6/693: dread d1/d4b/f80 [0,4194304] 0 2024-04-05T21:26:51.584 INFO:tasks.workunit.client.0.smithi121.stdout:8/782: creat d12/d13/dd3/df0/da7/ff3 x:0 0 0 2024-04-05T21:26:51.585 INFO:tasks.workunit.client.0.smithi121.stdout:8/783: fdatasync d12/d13/d56/d57/f74 0 2024-04-05T21:26:51.589 INFO:tasks.workunit.client.0.smithi121.stdout:1/856: link d8/d8c/da2/cff d8/d8c/db6/d1a/d1b/d4c/c11c 0 2024-04-05T21:26:51.592 INFO:tasks.workunit.client.0.smithi121.stdout:6/694: dwrite d1/f72 [0,4194304] 0 2024-04-05T21:26:51.593 INFO:tasks.workunit.client.0.smithi121.stdout:1/857: mkdir d8/d89/d9e/d2d/d84/da8/d11d 0 2024-04-05T21:26:51.594 INFO:tasks.workunit.client.0.smithi121.stdout:1/858: dread - d8/d10/ff6 zero size 2024-04-05T21:26:51.594 INFO:tasks.workunit.client.0.smithi121.stdout:3/699: dread d0/d3/d24/d3d/d1a/d23/d38/d55/f8d [0,4194304] 0 2024-04-05T21:26:51.596 INFO:tasks.workunit.client.0.smithi121.stdout:3/700: chown d0/d3/c2c 1354795882 1 2024-04-05T21:26:51.600 INFO:tasks.workunit.client.0.smithi121.stdout:3/701: dread d0/d3/d24/d3d/d1a/d23/d38/d55/f8d [0,4194304] 0 2024-04-05T21:26:51.602 INFO:tasks.workunit.client.0.smithi121.stdout:3/702: truncate d0/d3/fa 607074 0 2024-04-05T21:26:51.605 INFO:tasks.workunit.client.0.smithi121.stdout:3/703: mkdir d0/d3/d24/d6d/d7b/da3/dec 0 2024-04-05T21:26:51.607 INFO:tasks.workunit.client.0.smithi121.stdout:3/704: truncate d0/d3/d24/d3d/d1a/d1e/db1/fea 12175 0 2024-04-05T21:26:51.610 INFO:tasks.workunit.client.0.smithi121.stdout:3/705: dread d0/d3/d24/f89 [0,4194304] 0 2024-04-05T21:26:51.611 INFO:tasks.workunit.client.0.smithi121.stdout:1/859: symlink d8/d8c/db6/d1a/d1b/l11e 0 2024-04-05T21:26:51.618 INFO:tasks.workunit.client.0.smithi121.stdout:6/695: creat d1/d2/ff2 x:0 0 0 2024-04-05T21:26:51.620 INFO:tasks.workunit.client.0.smithi121.stdout:6/696: truncate d1/d2/daf/d6f/dc7/d1f/fd0 182878 0 2024-04-05T21:26:51.623 INFO:tasks.workunit.client.0.smithi121.stdout:5/904: write dd/de/d19/f57 [1222119,118307] 0 2024-04-05T21:26:51.625 INFO:tasks.workunit.client.0.smithi121.stdout:5/905: truncate dd/de/d19/d1c/d74/f10e 531415 0 2024-04-05T21:26:51.626 INFO:tasks.workunit.client.0.smithi121.stdout:9/606: write d6/df/d48/d96/db9/d2c/d37/d3c/f7c [806723,63342] 0 2024-04-05T21:26:51.628 INFO:tasks.workunit.client.0.smithi121.stdout:1/860: rename d8/fea to d8/d89/d9e/dc8/f11f 0 2024-04-05T21:26:51.628 INFO:tasks.workunit.client.0.smithi121.stdout:9/607: chown d6/df/d48/d96/dc2/dc9 64449653 1 2024-04-05T21:26:51.633 INFO:tasks.workunit.client.0.smithi121.stdout:9/608: dread d6/df/f71 [0,4194304] 0 2024-04-05T21:26:51.635 INFO:tasks.workunit.client.0.smithi121.stdout:2/801: fsync d2/d9/d1c/d30/d59/fbe 0 2024-04-05T21:26:51.638 INFO:tasks.workunit.client.0.smithi121.stdout:9/609: symlink d6/d81/ld2 0 2024-04-05T21:26:51.639 INFO:tasks.workunit.client.0.smithi121.stdout:3/706: unlink d0/d3/f7f 0 2024-04-05T21:26:51.647 INFO:tasks.workunit.client.0.smithi121.stdout:3/707: creat d0/d3/d24/d3d/d1a/d1e/d86/fed x:0 0 0 2024-04-05T21:26:51.647 INFO:tasks.workunit.client.0.smithi121.stdout:1/861: dread d8/d8c/db6/d1a/d1b/d4c/f63 [0,4194304] 0 2024-04-05T21:26:51.647 INFO:tasks.workunit.client.0.smithi121.stdout:9/610: rename d6/df/d48/d96/dc2/d85/l67 to d6/df/d48/d96/dc2/d85/ld3 0 2024-04-05T21:26:51.648 INFO:tasks.workunit.client.0.smithi121.stdout:9/611: write d6/d46/fac [2073645,5612] 0 2024-04-05T21:26:51.650 INFO:tasks.workunit.client.0.smithi121.stdout:2/802: dread d2/d9/d17/f18 [0,4194304] 0 2024-04-05T21:26:51.652 INFO:tasks.workunit.client.0.smithi121.stdout:3/708: symlink d0/d3/lee 0 2024-04-05T21:26:51.654 INFO:tasks.workunit.client.0.smithi121.stdout:9/612: write d6/df/d48/d96/db9/d2c/d37/f86 [7611761,121252] 0 2024-04-05T21:26:51.656 INFO:tasks.workunit.client.0.smithi121.stdout:0/873: write d7/dc/ff [2917724,76041] 0 2024-04-05T21:26:51.658 INFO:tasks.workunit.client.0.smithi121.stdout:3/709: symlink d0/d3/d24/d3d/d1a/d1e/db1/ddd/lef 0 2024-04-05T21:26:51.659 INFO:tasks.workunit.client.0.smithi121.stdout:4/982: mknod d5/d11/d2c/d9d/c144 0 2024-04-05T21:26:51.660 INFO:tasks.workunit.client.1.smithi188.stdout:8/840: sync 2024-04-05T21:26:51.660 INFO:tasks.workunit.client.0.smithi121.stdout:2/803: dwrite d2/d9/d1c/d30/fb1 [0,4194304] 0 2024-04-05T21:26:51.661 INFO:tasks.workunit.client.0.smithi121.stdout:9/613: mknod d6/d32/dbe/d7b/d49/d66/cd4 0 2024-04-05T21:26:51.664 INFO:tasks.workunit.client.0.smithi121.stdout:0/874: creat d7/d71/d94/dd1/f133 x:0 0 0 2024-04-05T21:26:51.665 INFO:tasks.workunit.client.1.smithi188.stdout:8/841: write d5/d21/d40/d43/f87 [4119925,53248] 0 2024-04-05T21:26:51.666 INFO:tasks.workunit.client.0.smithi121.stdout:2/804: dread d2/d9/d38/d2d/d4c/d7c/d7e/fca [0,4194304] 0 2024-04-05T21:26:51.668 INFO:tasks.workunit.client.0.smithi121.stdout:9/614: mkdir d6/d46/d8a/dd5 0 2024-04-05T21:26:51.671 INFO:tasks.workunit.client.0.smithi121.stdout:3/710: creat d0/d3/d24/d6d/d7b/de3/ff0 x:0 0 0 2024-04-05T21:26:51.674 INFO:tasks.workunit.client.0.smithi121.stdout:0/875: dread d7/d71/d45/d46/d53/d8e/fbf [0,4194304] 0 2024-04-05T21:26:51.675 INFO:tasks.workunit.client.0.smithi121.stdout:3/711: creat d0/d3/d24/d3d/d1a/d1e/db1/ff1 x:0 0 0 2024-04-05T21:26:51.676 INFO:tasks.workunit.client.0.smithi121.stdout:0/876: stat d7/d71/d29/d2a/d2c/c85 0 2024-04-05T21:26:51.677 INFO:tasks.workunit.client.0.smithi121.stdout:9/615: getdents d6/df/d48/d96/dc2/dc9 0 2024-04-05T21:26:51.678 INFO:tasks.workunit.client.0.smithi121.stdout:8/784: mknod d12/d50/d6a/d63/cf4 0 2024-04-05T21:26:51.678 INFO:tasks.workunit.client.0.smithi121.stdout:9/616: read d6/d46/d8a/f93 [142562,48075] 0 2024-04-05T21:26:51.679 INFO:tasks.workunit.client.0.smithi121.stdout:3/712: chown d0/d3/d24/d3d/d1a/l6f 41 1 2024-04-05T21:26:51.680 INFO:tasks.workunit.client.1.smithi188.stdout:8/842: dwrite d5/f3d [0,4194304] 0 2024-04-05T21:26:51.680 INFO:tasks.workunit.client.0.smithi121.stdout:2/805: link d2/d9/f75 d2/d9/d38/d2d/d4c/f103 0 2024-04-05T21:26:51.682 INFO:tasks.workunit.client.0.smithi121.stdout:1/862: rmdir d8/d89/d9e/d2d/d84/da8 39 2024-04-05T21:26:51.683 INFO:tasks.workunit.client.1.smithi188.stdout:8/843: mknod d5/d21/d2a/d95/c10c 0 2024-04-05T21:26:51.683 INFO:tasks.workunit.client.0.smithi121.stdout:2/806: chown d2/d9/d38/d2d/d47/f101 6 1 2024-04-05T21:26:51.684 INFO:tasks.workunit.client.0.smithi121.stdout:3/713: dread d0/d3/d24/d3d/d1a/d1e/db1/ddd/de1/fb6 [0,4194304] 0 2024-04-05T21:26:51.686 INFO:tasks.workunit.client.1.smithi188.stdout:8/844: write d5/d6e/d86/fb0 [3645522,15320] 0 2024-04-05T21:26:51.692 INFO:tasks.workunit.client.0.smithi121.stdout:3/714: write d0/d3/d24/d3d/d1a/d1e/db1/ddd/de1/fd2 [662012,17314] 0 2024-04-05T21:26:51.692 INFO:tasks.workunit.client.1.smithi188.stdout:8/845: rmdir d5/d21/d40/d43/d44/d5b/d99/d69/d70/dae 39 2024-04-05T21:26:51.693 INFO:tasks.workunit.client.0.smithi121.stdout:8/785: dread d12/d13/dd3/df0/fa9 [4194304,4194304] 0 2024-04-05T21:26:51.694 INFO:tasks.workunit.client.0.smithi121.stdout:7/906: creat d0/d9/f11f x:0 0 0 2024-04-05T21:26:51.694 INFO:tasks.workunit.client.0.smithi121.stdout:9/617: dread d6/df/d48/d96/dc2/f87 [0,4194304] 0 2024-04-05T21:26:51.695 INFO:tasks.workunit.client.0.smithi121.stdout:7/907: read - d0/d9/d22/d3e/d76/f108 zero size 2024-04-05T21:26:51.695 INFO:tasks.workunit.client.1.smithi188.stdout:8/846: unlink d5/f20 0 2024-04-05T21:26:51.696 INFO:tasks.workunit.client.1.smithi188.stdout:8/847: chown d5/d21/d2a/l4a 23954896 1 2024-04-05T21:26:51.696 INFO:tasks.workunit.client.0.smithi121.stdout:9/618: write d6/df/d48/d96/db9/d72/f7f [1024788,19612] 0 2024-04-05T21:26:51.697 INFO:tasks.workunit.client.1.smithi188.stdout:8/848: write d5/d21/d40/f4f [290902,85171] 0 2024-04-05T21:26:51.697 INFO:tasks.workunit.client.0.smithi121.stdout:3/715: mkdir d0/d3/d24/df2 0 2024-04-05T21:26:51.699 INFO:tasks.workunit.client.0.smithi121.stdout:7/908: fdatasync d0/d23/d5a/f68 0 2024-04-05T21:26:51.700 INFO:tasks.workunit.client.0.smithi121.stdout:6/697: write d1/f38 [1254767,116263] 0 2024-04-05T21:26:51.706 INFO:tasks.workunit.client.0.smithi121.stdout:9/619: dwrite d6/d46/d8a/fc5 [0,4194304] 0 2024-04-05T21:26:51.709 INFO:tasks.workunit.client.0.smithi121.stdout:3/716: link d0/d3/d24/d3d/d1a/d23/f6b d0/d3/d24/d3d/dcf/ff3 0 2024-04-05T21:26:51.709 INFO:tasks.workunit.client.0.smithi121.stdout:7/909: dread d0/d9/d22/d28/d56/db9/ddf/def/d52/d61/f89 [0,4194304] 0 2024-04-05T21:26:51.710 INFO:tasks.workunit.client.0.smithi121.stdout:5/906: dwrite dd/de/d19/f37 [0,4194304] 0 2024-04-05T21:26:51.711 INFO:tasks.workunit.client.0.smithi121.stdout:3/717: read - d0/d3/d24/d3d/f7e zero size 2024-04-05T21:26:51.711 INFO:tasks.workunit.client.1.smithi188.stdout:0/976: sync 2024-04-05T21:26:51.712 INFO:tasks.workunit.client.1.smithi188.stdout:0/977: chown d4/d8/db/d21/d4a/d73/d116/dd1/ff9 3 1 2024-04-05T21:26:51.715 INFO:tasks.workunit.client.1.smithi188.stdout:0/978: write d4/d8/db/d42/da2/fa7 [4213555,115892] 0 2024-04-05T21:26:51.720 INFO:tasks.workunit.client.1.smithi188.stdout:0/979: mkdir d4/d6/d13/da1/d130/d14e/d168 0 2024-04-05T21:26:51.722 INFO:tasks.workunit.client.0.smithi121.stdout:6/698: getdents d1/d2/d1b/d2e 0 2024-04-05T21:26:51.722 INFO:tasks.workunit.client.1.smithi188.stdout:0/980: mkdir d4/d6/d46/d169 0 2024-04-05T21:26:51.724 INFO:tasks.workunit.client.0.smithi121.stdout:7/910: link d0/d9/d22/d3e/d76/f108 d0/d9/d22/d3e/d76/f120 0 2024-04-05T21:26:51.725 INFO:tasks.workunit.client.0.smithi121.stdout:4/983: truncate d5/d11/d2c/d9d/dae/d29/d2e/dd3/fd0 3623154 0 2024-04-05T21:26:51.728 INFO:tasks.workunit.client.0.smithi121.stdout:3/718: creat d0/d3/d24/d6d/ff4 x:0 0 0 2024-04-05T21:26:51.729 INFO:tasks.workunit.client.0.smithi121.stdout:6/699: mkdir d1/d2/d1b/d35/d74/df3 0 2024-04-05T21:26:51.731 INFO:tasks.workunit.client.0.smithi121.stdout:7/911: unlink d0/d9/d22/d3e/l87 0 2024-04-05T21:26:51.732 INFO:tasks.workunit.client.0.smithi121.stdout:9/620: getdents d6/d32/dbe/d7b/d49 0 2024-04-05T21:26:51.733 INFO:tasks.workunit.client.0.smithi121.stdout:0/877: write d7/d71/d29/d3f/f72 [4177699,72540] 0 2024-04-05T21:26:51.736 INFO:tasks.workunit.client.0.smithi121.stdout:1/863: write d8/d8c/db6/d1a/f20 [3097388,90024] 0 2024-04-05T21:26:51.740 INFO:tasks.workunit.client.0.smithi121.stdout:4/984: rename d5/d45/d67/ca9 to d5/d11/d28/c145 0 2024-04-05T21:26:51.741 INFO:tasks.workunit.client.1.smithi188.stdout:0/981: dwrite d4/d8/db/f88 [0,4194304] 0 2024-04-05T21:26:51.742 INFO:tasks.workunit.client.0.smithi121.stdout:6/700: creat d1/ff4 x:0 0 0 2024-04-05T21:26:51.743 INFO:tasks.workunit.client.1.smithi188.stdout:0/982: fdatasync d4/d8/db/d42/da2/fa7 0 2024-04-05T21:26:51.744 INFO:tasks.workunit.client.0.smithi121.stdout:5/907: dread dd/d41/dce/da6/dd3/db1/db9/fd0 [0,4194304] 0 2024-04-05T21:26:51.745 INFO:tasks.workunit.client.0.smithi121.stdout:7/912: rename d0/d9/f95 to d0/d57/de7/f121 0 2024-04-05T21:26:51.747 INFO:tasks.workunit.client.0.smithi121.stdout:9/621: dwrite d6/df/d48/d96/db9/d72/f7f [0,4194304] 0 2024-04-05T21:26:51.751 INFO:tasks.workunit.client.0.smithi121.stdout:7/913: readlink d0/d32/l66 0 2024-04-05T21:26:51.754 INFO:tasks.workunit.client.0.smithi121.stdout:0/878: dwrite d7/d71/fdc [0,4194304] 0 2024-04-05T21:26:51.754 INFO:tasks.workunit.client.0.smithi121.stdout:1/864: truncate d8/d8c/db6/d1a/d1b/d24/dc2/fcc 1205050 0 2024-04-05T21:26:51.755 INFO:tasks.workunit.client.0.smithi121.stdout:9/622: chown d6/d46/d8a/dd5 386 1 2024-04-05T21:26:51.755 INFO:tasks.workunit.client.0.smithi121.stdout:6/701: creat d1/d2/d3/d4/d21/ff5 x:0 0 0 2024-04-05T21:26:51.760 INFO:tasks.workunit.client.0.smithi121.stdout:2/807: dwrite d2/d9/f2f [0,4194304] 0 2024-04-05T21:26:51.763 INFO:tasks.workunit.client.0.smithi121.stdout:9/623: chown d6/d32/dbe/d7b/d49 579143 1 2024-04-05T21:26:51.763 INFO:tasks.workunit.client.0.smithi121.stdout:3/719: dread - d0/d3/d24/d3d/d1a/d1e/d86/fc4 zero size 2024-04-05T21:26:51.766 INFO:tasks.workunit.client.0.smithi121.stdout:2/808: dread - d2/d9/d38/d2d/d4c/de9/d62/fc6 zero size 2024-04-05T21:26:51.771 INFO:tasks.workunit.client.0.smithi121.stdout:7/914: read d0/d32/d7c/fdd [317000,119337] 0 2024-04-05T21:26:51.776 INFO:tasks.workunit.client.0.smithi121.stdout:6/702: unlink d1/d2/d1b/fe7 0 2024-04-05T21:26:51.776 INFO:tasks.workunit.client.0.smithi121.stdout:4/985: dwrite d5/d11/d26/d47/fea [0,4194304] 0 2024-04-05T21:26:51.779 INFO:tasks.workunit.client.0.smithi121.stdout:1/865: fdatasync d8/d89/d9e/d2d/f8a 0 2024-04-05T21:26:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:51 smithi188 ceph-mon[49715]: pgmap v24: 65 pgs: 65 active+clean; 3.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 143 MiB/s rd, 395 MiB/s wr, 786 op/s 2024-04-05T21:26:51.780 INFO:tasks.workunit.client.0.smithi121.stdout:3/720: unlink d0/d3/fe7 0 2024-04-05T21:26:51.781 INFO:tasks.workunit.client.0.smithi121.stdout:3/721: fsync d0/d3/d24/d6d/f81 0 2024-04-05T21:26:51.786 INFO:tasks.workunit.client.0.smithi121.stdout:6/703: mknod d1/d2/daf/d6f/dc7/d1f/cf6 0 2024-04-05T21:26:51.787 INFO:tasks.workunit.client.0.smithi121.stdout:9/624: rename d6/df/d48/d96/dc2/lcd to d6/df/d48/d96/db9/d72/ld6 0 2024-04-05T21:26:51.789 INFO:tasks.workunit.client.0.smithi121.stdout:2/809: symlink d2/d9/d17/l104 0 2024-04-05T21:26:51.790 INFO:tasks.workunit.client.0.smithi121.stdout:2/810: chown d2/d9/d38/d2d/d4c/de9/cea 117 1 2024-04-05T21:26:51.793 INFO:tasks.workunit.client.0.smithi121.stdout:7/915: read d0/d9/f50 [132796,115212] 0 2024-04-05T21:26:51.793 INFO:tasks.workunit.client.0.smithi121.stdout:5/908: write dd/de/fc3 [559973,103577] 0 2024-04-05T21:26:51.794 INFO:tasks.workunit.client.0.smithi121.stdout:2/811: write d2/d9/d1c/d30/fcd [2638432,87734] 0 2024-04-05T21:26:51.797 INFO:tasks.workunit.client.0.smithi121.stdout:6/704: creat d1/d2/d1b/d2e/ff7 x:0 0 0 2024-04-05T21:26:51.797 INFO:tasks.workunit.client.0.smithi121.stdout:3/722: write d0/d3/d24/d3d/d1a/fdc [66495,43959] 0 2024-04-05T21:26:51.799 INFO:tasks.workunit.client.0.smithi121.stdout:9/625: fdatasync d6/df/d48/f62 0 2024-04-05T21:26:51.807 INFO:tasks.workunit.client.0.smithi121.stdout:2/812: write d2/d9/d38/d2d/d4c/d7c/dc2/ff3 [796634,78842] 0 2024-04-05T21:26:51.809 INFO:tasks.workunit.client.0.smithi121.stdout:3/723: mkdir d0/d3/de/da6/df5 0 2024-04-05T21:26:51.810 INFO:tasks.workunit.client.1.smithi188.stdout:0/983: write d4/d8/db/fe0 [1408831,123983] 0 2024-04-05T21:26:51.810 INFO:tasks.workunit.client.1.smithi188.stdout:0/984: chown d4/d8/db/dac/lff 19 1 2024-04-05T21:26:51.813 INFO:tasks.workunit.client.0.smithi121.stdout:9/626: symlink d6/df/d50/ld7 0 2024-04-05T21:26:51.814 INFO:tasks.workunit.client.0.smithi121.stdout:7/916: dwrite d0/d9/f12 [4194304,4194304] 0 2024-04-05T21:26:51.814 INFO:tasks.workunit.client.1.smithi188.stdout:0/985: write d4/d8/d78/f164 [2560266,46903] 0 2024-04-05T21:26:51.817 INFO:tasks.workunit.client.0.smithi121.stdout:5/909: dwrite dd/de/f34 [0,4194304] 0 2024-04-05T21:26:51.817 INFO:tasks.workunit.client.0.smithi121.stdout:6/705: creat d1/d2/d1b/d35/d39/d92/dea/ff8 x:0 0 0 2024-04-05T21:26:51.818 INFO:tasks.workunit.client.0.smithi121.stdout:1/866: write d8/d8c/db6/d1a/d1b/d4c/d76/da4/fdf [1465967,41774] 0 2024-04-05T21:26:51.819 INFO:tasks.workunit.client.1.smithi188.stdout:0/986: creat d4/d8/d10/f16a x:0 0 0 2024-04-05T21:26:51.819 INFO:tasks.workunit.client.0.smithi121.stdout:4/986: truncate d5/d11/d28/fe 1178759 0 2024-04-05T21:26:51.820 INFO:tasks.workunit.client.1.smithi188.stdout:0/987: symlink d4/d6/d46/dba/l16b 0 2024-04-05T21:26:51.820 INFO:tasks.workunit.client.1.smithi188.stdout:0/988: chown d4/d6/d13/da1/f124 7047612 1 2024-04-05T21:26:51.823 INFO:tasks.workunit.client.0.smithi121.stdout:0/879: dread d7/d71/d29/d2a/d2c/f8c [0,4194304] 0 2024-04-05T21:26:51.824 INFO:tasks.workunit.client.1.smithi188.stdout:0/989: truncate d4/d6/d46/fcf 291799 0 2024-04-05T21:26:51.828 INFO:tasks.workunit.client.1.smithi188.stdout:0/990: link d4/d8/db/d21/f56 d4/d6/d13/da1/f16c 0 2024-04-05T21:26:51.828 INFO:tasks.workunit.client.0.smithi121.stdout:3/724: rename d0/d3/d24/d3f to d0/d3/d24/d6d/d7b/da3/dec/df6 0 2024-04-05T21:26:51.832 INFO:tasks.workunit.client.0.smithi121.stdout:9/627: creat d6/fd8 x:0 0 0 2024-04-05T21:26:51.832 INFO:tasks.workunit.client.0.smithi121.stdout:5/910: dwrite dd/de/d19/d38/d45/d68/f79 [0,4194304] 0 2024-04-05T21:26:51.835 INFO:tasks.workunit.client.1.smithi188.stdout:0/991: dwrite d4/d8/db/d42/da2/fa7 [0,4194304] 0 2024-04-05T21:26:51.837 INFO:tasks.workunit.client.0.smithi121.stdout:5/911: fdatasync dd/de/d19/d1c/d74/f10e 0 2024-04-05T21:26:51.840 INFO:tasks.workunit.client.1.smithi188.stdout:0/992: rename d4/d6/d13/da1/cfc to d4/d6/df4/d14b/c16d 0 2024-04-05T21:26:51.841 INFO:tasks.workunit.client.0.smithi121.stdout:7/917: mkdir d0/d57/de7/d122 0 2024-04-05T21:26:51.842 INFO:tasks.workunit.client.0.smithi121.stdout:2/813: dwrite d2/d9/d1c/f69 [0,4194304] 0 2024-04-05T21:26:51.852 INFO:tasks.workunit.client.0.smithi121.stdout:0/880: chown d7/d71/d29/de8/f105 3385204 1 2024-04-05T21:26:51.852 INFO:tasks.workunit.client.0.smithi121.stdout:3/725: unlink d0/d3/d24/d6d/d7b/da4/cd6 0 2024-04-05T21:26:51.853 INFO:tasks.workunit.client.0.smithi121.stdout:2/814: creat d2/d9/dcb/f105 x:0 0 0 2024-04-05T21:26:51.857 INFO:tasks.workunit.client.0.smithi121.stdout:4/987: mkdir d5/d45/d146 0 2024-04-05T21:26:51.858 INFO:tasks.workunit.client.0.smithi121.stdout:5/912: mknod dd/d41/dce/da6/dd3/db1/db9/c142 0 2024-04-05T21:26:51.860 INFO:tasks.workunit.client.0.smithi121.stdout:4/988: stat d5/d11/d2c/d9d/dae 0 2024-04-05T21:26:51.860 INFO:tasks.workunit.client.0.smithi121.stdout:0/881: symlink d7/d62/d3c/d6b/l134 0 2024-04-05T21:26:51.861 INFO:tasks.workunit.client.0.smithi121.stdout:7/918: dread d0/d32/f8a [0,4194304] 0 2024-04-05T21:26:51.863 INFO:tasks.workunit.client.0.smithi121.stdout:6/706: getdents d1/d2/d1b/da1 0 2024-04-05T21:26:51.864 INFO:tasks.workunit.client.0.smithi121.stdout:3/726: creat d0/d3/d24/d6d/d7b/ff7 x:0 0 0 2024-04-05T21:26:51.867 INFO:tasks.workunit.client.0.smithi121.stdout:5/913: dread dd/d41/dce/da6/dd3/db1/db9/fd0 [0,4194304] 0 2024-04-05T21:26:51.868 INFO:tasks.workunit.client.0.smithi121.stdout:4/989: rename d5/d11/d2c/d9d/dae/f140 to d5/d45/d8b/f147 0 2024-04-05T21:26:51.870 INFO:tasks.workunit.client.0.smithi121.stdout:9/628: dread d6/df/d48/d96/db9/d2c/d37/d3c/f7c [0,4194304] 0 2024-04-05T21:26:51.870 INFO:tasks.workunit.client.0.smithi121.stdout:3/727: symlink d0/d3/de/da6/lf8 0 2024-04-05T21:26:51.871 INFO:tasks.workunit.client.0.smithi121.stdout:9/629: chown d6/df/d48/d96/dc2/d79/d94 465 1 2024-04-05T21:26:51.872 INFO:tasks.workunit.client.0.smithi121.stdout:6/707: symlink d1/d2/d1b/d35/d39/d92/dea/lf9 0 2024-04-05T21:26:51.873 INFO:tasks.workunit.client.0.smithi121.stdout:5/914: symlink dd/de/d19/d1c/d36/l143 0 2024-04-05T21:26:51.875 INFO:tasks.workunit.client.0.smithi121.stdout:4/990: mknod d5/d45/d67/dab/c148 0 2024-04-05T21:26:51.875 INFO:tasks.workunit.client.0.smithi121.stdout:0/882: read d7/d71/d29/f37 [4469671,32013] 0 2024-04-05T21:26:51.875 INFO:tasks.workunit.client.1.smithi188.stdout:0/993: write d4/d8/db/d21/d4a/f58 [25491,128912] 0 2024-04-05T21:26:51.876 INFO:tasks.workunit.client.0.smithi121.stdout:0/883: readlink d7/d71/d45/d46/l6e 0 2024-04-05T21:26:51.877 INFO:tasks.workunit.client.0.smithi121.stdout:7/919: dread d0/d9/d22/d3e/f3c [0,4194304] 0 2024-04-05T21:26:51.877 INFO:tasks.workunit.client.0.smithi121.stdout:9/630: creat d6/df/d50/fd9 x:0 0 0 2024-04-05T21:26:51.878 INFO:tasks.workunit.client.0.smithi121.stdout:2/815: write d2/d9/d17/d3a/f55 [199240,54714] 0 2024-04-05T21:26:51.880 INFO:tasks.workunit.client.0.smithi121.stdout:6/708: symlink d1/d2/d1b/da1/lfa 0 2024-04-05T21:26:51.880 INFO:tasks.workunit.client.0.smithi121.stdout:5/915: symlink dd/d41/dce/da6/dd3/de1/l144 0 2024-04-05T21:26:51.884 INFO:tasks.workunit.client.0.smithi121.stdout:5/916: chown dd/de/d19/d3c/f5f 4 1 2024-04-05T21:26:51.884 INFO:tasks.workunit.client.0.smithi121.stdout:1/867: dread d8/d8c/db6/d1a/f20 [0,4194304] 0 2024-04-05T21:26:51.885 INFO:tasks.workunit.client.1.smithi188.stdout:0/994: dwrite d4/d6/d13/da1/f16c [0,4194304] 0 2024-04-05T21:26:51.886 INFO:tasks.workunit.client.0.smithi121.stdout:1/868: write d8/d8c/db6/d1a/d1b/d4c/d76/da4/fdf [2418788,43581] 0 2024-04-05T21:26:51.888 INFO:tasks.workunit.client.0.smithi121.stdout:2/816: creat d2/d9/d38/d2d/d4c/de9/d8d/dd8/f106 x:0 0 0 2024-04-05T21:26:51.894 INFO:tasks.workunit.client.1.smithi188.stdout:8/849: sync 2024-04-05T21:26:51.894 INFO:tasks.workunit.client.0.smithi121.stdout:0/884: link d7/d71/d94/fd7 d7/d71/d45/d46/d106/d11e/f135 0 2024-04-05T21:26:51.895 INFO:tasks.workunit.client.0.smithi121.stdout:3/728: write d0/d3/d24/d3d/f7e [862489,67390] 0 2024-04-05T21:26:51.896 INFO:tasks.workunit.client.0.smithi121.stdout:0/885: chown d7/d62/d9d/fbb 1040098245 1 2024-04-05T21:26:51.898 INFO:tasks.workunit.client.0.smithi121.stdout:5/917: read dd/de/d19/d38/d45/f5c [4027530,61388] 0 2024-04-05T21:26:51.898 INFO:tasks.workunit.client.0.smithi121.stdout:4/991: truncate d5/d11/d20/fad 7690989 0 2024-04-05T21:26:51.899 INFO:tasks.workunit.client.0.smithi121.stdout:1/869: creat d8/d89/d9e/d2d/f120 x:0 0 0 2024-04-05T21:26:51.900 INFO:tasks.workunit.client.1.smithi188.stdout:8/850: dwrite d5/d6e/f84 [0,4194304] 0 2024-04-05T21:26:51.900 INFO:tasks.workunit.client.0.smithi121.stdout:3/729: chown d0/d3/d24/d6d/d7b/da3/caa 0 1 2024-04-05T21:26:51.901 INFO:tasks.workunit.client.0.smithi121.stdout:2/817: symlink d2/d9/d38/d2d/d4c/d7c/d7e/l107 0 2024-04-05T21:26:51.901 INFO:tasks.workunit.client.1.smithi188.stdout:8/851: write d5/d21/fe0 [982651,127271] 0 2024-04-05T21:26:51.903 INFO:tasks.workunit.client.0.smithi121.stdout:6/709: rename d1/d2/dc/d9a/dc0/c94 to d1/d2/d3/d4/d21/d57/cfb 0 2024-04-05T21:26:51.903 INFO:tasks.workunit.client.0.smithi121.stdout:1/870: chown d8/d89/f98 10125 1 2024-04-05T21:26:51.904 INFO:tasks.workunit.client.1.smithi188.stdout:0/995: dread d4/d8/db/d21/d4a/d73/d116/f51 [0,4194304] 0 2024-04-05T21:26:51.905 INFO:tasks.workunit.client.0.smithi121.stdout:7/920: creat d0/d9/d22/d28/d56/db9/ddf/def/d52/f123 x:0 0 0 2024-04-05T21:26:51.905 INFO:tasks.workunit.client.1.smithi188.stdout:8/852: creat d5/d21/d40/d43/d44/d59/d8b/da4/f10d x:0 0 0 2024-04-05T21:26:51.906 INFO:tasks.workunit.client.0.smithi121.stdout:9/631: dwrite d6/df/d48/d96/fce [0,4194304] 0 2024-04-05T21:26:51.906 INFO:tasks.workunit.client.0.smithi121.stdout:0/886: symlink d7/d62/d3c/l136 0 2024-04-05T21:26:51.906 INFO:tasks.workunit.client.1.smithi188.stdout:8/853: fsync d5/dc/f3b 0 2024-04-05T21:26:51.907 INFO:tasks.workunit.client.1.smithi188.stdout:0/996: mkdir d4/d6/d13/da1/d16e 0 2024-04-05T21:26:51.909 INFO:tasks.workunit.client.1.smithi188.stdout:8/854: mkdir d5/d10e 0 2024-04-05T21:26:51.911 INFO:tasks.workunit.client.1.smithi188.stdout:0/997: getdents d4/d6/d13/da1/d130 0 2024-04-05T21:26:51.915 INFO:tasks.workunit.client.0.smithi121.stdout:2/818: symlink d2/d9/d38/d2d/d4c/de9/d5c/df0/l108 0 2024-04-05T21:26:51.915 INFO:tasks.workunit.client.1.smithi188.stdout:0/998: symlink d4/d6/d13/dca/d125/l16f 0 2024-04-05T21:26:51.916 INFO:tasks.workunit.client.0.smithi121.stdout:4/992: chown d5/d11/d20/fad 181986 1 2024-04-05T21:26:51.917 INFO:tasks.workunit.client.1.smithi188.stdout:0/999: read - d4/d8/db/d42/d38/ff8 zero size 2024-04-05T21:26:51.921 INFO:tasks.workunit.client.0.smithi121.stdout:5/918: dwrite dd/de/d39/d3b/d50/f5b [0,4194304] 0 2024-04-05T21:26:51.921 INFO:tasks.workunit.client.0.smithi121.stdout:7/921: chown d0/d32/d4e/d73/dbc/df0/dd9 14 1 2024-04-05T21:26:51.922 INFO:tasks.workunit.client.0.smithi121.stdout:6/710: mknod d1/d2/d3/d4/d21/d57/d42/d48/d7c/cfc 0 2024-04-05T21:26:51.924 INFO:tasks.workunit.client.0.smithi121.stdout:3/730: dwrite d0/d3/d24/d3d/d1a/d1e/d86/fde [0,4194304] 0 2024-04-05T21:26:51.925 INFO:tasks.workunit.client.0.smithi121.stdout:0/887: mknod d7/d71/d94/c137 0 2024-04-05T21:26:51.927 INFO:tasks.workunit.client.0.smithi121.stdout:4/993: mkdir d5/d11/d2c/da4/d149 0 2024-04-05T21:26:51.927 INFO:tasks.workunit.client.0.smithi121.stdout:3/731: write d0/f97 [619868,45089] 0 2024-04-05T21:26:51.928 INFO:tasks.workunit.client.0.smithi121.stdout:9/632: dwrite d6/f9b [0,4194304] 0 2024-04-05T21:26:51.930 INFO:tasks.workunit.client.0.smithi121.stdout:1/871: symlink d8/d89/d9e/dc8/d119/l121 0 2024-04-05T21:26:51.930 INFO:tasks.workunit.client.0.smithi121.stdout:3/732: fdatasync d0/fda 0 2024-04-05T21:26:51.931 INFO:tasks.workunit.client.0.smithi121.stdout:9/633: fdatasync d6/f15 0 2024-04-05T21:26:51.932 INFO:tasks.workunit.client.0.smithi121.stdout:5/919: dread - dd/d41/dce/da6/dd3/db1/f106 zero size 2024-04-05T21:26:51.935 INFO:tasks.workunit.client.0.smithi121.stdout:6/711: chown d1/d2/d3/d4/d21/d57/cfb 0 1 2024-04-05T21:26:51.938 INFO:tasks.workunit.client.0.smithi121.stdout:5/920: truncate dd/de/d19/d38/f138 253944 0 2024-04-05T21:26:51.941 INFO:tasks.workunit.client.0.smithi121.stdout:7/922: write d0/f1f [81126,40003] 0 2024-04-05T21:26:51.942 INFO:tasks.workunit.client.0.smithi121.stdout:1/872: write d8/f1c [3897067,1710] 0 2024-04-05T21:26:51.943 INFO:tasks.workunit.client.0.smithi121.stdout:3/733: mkdir d0/d3/d24/d6d/d7b/da3/dec/df9 0 2024-04-05T21:26:51.945 INFO:tasks.workunit.client.0.smithi121.stdout:7/923: fdatasync d0/d32/d4e/d73/ffd 0 2024-04-05T21:26:51.945 INFO:tasks.workunit.client.1.smithi188.stdout:8/855: dread d5/d21/d40/d43/f87 [0,4194304] 0 2024-04-05T21:26:51.945 INFO:tasks.workunit.client.0.smithi121.stdout:4/994: symlink d5/d11/d26/dbc/dca/de9/df6/l14a 0 2024-04-05T21:26:51.948 INFO:tasks.workunit.client.0.smithi121.stdout:0/888: dwrite d7/d62/ddb/f102 [0,4194304] 0 2024-04-05T21:26:51.948 INFO:tasks.workunit.client.0.smithi121.stdout:5/921: creat dd/de/d39/d3b/de8/d103/f145 x:0 0 0 2024-04-05T21:26:51.948 INFO:tasks.workunit.client.0.smithi121.stdout:3/734: write d0/d3/d24/f3c [4888091,88231] 0 2024-04-05T21:26:51.954 INFO:tasks.workunit.client.0.smithi121.stdout:1/873: fsync d8/d89/d9e/dc8/f11f 0 2024-04-05T21:26:51.954 INFO:tasks.workunit.client.0.smithi121.stdout:7/924: creat d0/d57/f124 x:0 0 0 2024-04-05T21:26:51.954 INFO:tasks.workunit.client.0.smithi121.stdout:5/922: mknod dd/de/d19/d38/d9b/c146 0 2024-04-05T21:26:51.954 INFO:tasks.workunit.client.0.smithi121.stdout:0/889: truncate d7/dc/f33 2614777 0 2024-04-05T21:26:51.954 INFO:tasks.workunit.client.0.smithi121.stdout:3/735: fsync d0/d3/d24/d3d/fa8 0 2024-04-05T21:26:51.956 INFO:tasks.workunit.client.0.smithi121.stdout:5/923: dread - dd/d41/dce/da6/f132 zero size 2024-04-05T21:26:51.956 INFO:tasks.workunit.client.1.smithi188.stdout:8/856: write d5/f34 [1329206,35333] 0 2024-04-05T21:26:51.958 INFO:tasks.workunit.client.0.smithi121.stdout:7/925: mkdir d0/d9/d22/de9/d125 0 2024-04-05T21:26:51.958 INFO:tasks.workunit.client.1.smithi188.stdout:8/857: creat d5/d6e/dad/f10f x:0 0 0 2024-04-05T21:26:51.959 INFO:tasks.workunit.client.1.smithi188.stdout:8/858: readlink d5/d21/d40/d43/d44/d59/d8b/lf4 0 2024-04-05T21:26:51.960 INFO:tasks.workunit.client.0.smithi121.stdout:7/926: dread - d0/d9/d22/d28/d56/f11a zero size 2024-04-05T21:26:51.960 INFO:tasks.workunit.client.1.smithi188.stdout:8/859: chown d5/dc/d8f/cd6 1 1 2024-04-05T21:26:51.961 INFO:tasks.workunit.client.0.smithi121.stdout:6/712: write d1/d2/daf/d6f/dc7/fac [3308020,115078] 0 2024-04-05T21:26:51.961 INFO:tasks.workunit.client.0.smithi121.stdout:3/736: mknod d0/d3/d24/d3d/d1a/d1e/d86/cfa 0 2024-04-05T21:26:51.961 INFO:tasks.workunit.client.0.smithi121.stdout:5/924: truncate dd/de/d19/d1c/d7b/f110 882128 0 2024-04-05T21:26:51.961 INFO:tasks.workunit.client.0.smithi121.stdout:0/890: write d7/d71/d29/de8/f131 [2586997,98516] 0 2024-04-05T21:26:51.961 INFO:tasks.workunit.client.0.smithi121.stdout:9/634: truncate d6/df/d48/d96/fce 2501717 0 2024-04-05T21:26:51.963 INFO:tasks.workunit.client.1.smithi188.stdout:8/860: unlink d5/d21/d40/d43/d44/d59/c72 0 2024-04-05T21:26:51.965 INFO:tasks.workunit.client.1.smithi188.stdout:8/861: truncate d5/d21/d40/d43/d44/d59/d8b/f105 715939 0 2024-04-05T21:26:51.966 INFO:tasks.workunit.client.0.smithi121.stdout:4/995: truncate d5/f86 2846495 0 2024-04-05T21:26:51.966 INFO:tasks.workunit.client.0.smithi121.stdout:7/927: creat d0/d9/d22/d28/d56/dff/f126 x:0 0 0 2024-04-05T21:26:51.968 INFO:tasks.workunit.client.0.smithi121.stdout:0/891: truncate d7/d71/f5f 594393 0 2024-04-05T21:26:51.968 INFO:tasks.workunit.client.1.smithi188.stdout:8/862: getdents d5/d21/d40/d43/d48/d7e/dcf 0 2024-04-05T21:26:51.969 INFO:tasks.workunit.client.0.smithi121.stdout:9/635: unlink d6/df/d48/d96/db9/d2c/f83 0 2024-04-05T21:26:51.970 INFO:tasks.workunit.client.1.smithi188.stdout:8/863: mknod d5/d21/d40/d43/d48/d7e/c110 0 2024-04-05T21:26:51.970 INFO:tasks.workunit.client.0.smithi121.stdout:3/737: symlink d0/d3/d24/d3d/d1a/d23/lfb 0 2024-04-05T21:26:51.971 INFO:tasks.workunit.client.0.smithi121.stdout:3/738: write d0/d3/d24/d3d/d1a/d1e/f9a [99708,61208] 0 2024-04-05T21:26:51.974 INFO:tasks.workunit.client.0.smithi121.stdout:6/713: creat d1/d2/d3/d4/d21/d57/d42/d48/d7c/ffd x:0 0 0 2024-04-05T21:26:51.976 INFO:tasks.workunit.client.0.smithi121.stdout:4/996: creat d5/d11/d2c/d9d/dae/d29/db4/f14b x:0 0 0 2024-04-05T21:26:51.979 INFO:tasks.workunit.client.0.smithi121.stdout:9/636: symlink d6/d32/lda 0 2024-04-05T21:26:51.981 INFO:tasks.workunit.client.0.smithi121.stdout:3/739: creat d0/d3/d24/d6d/d7b/da4/ffc x:0 0 0 2024-04-05T21:26:51.982 INFO:tasks.workunit.client.0.smithi121.stdout:9/637: write d6/df/d48/d96/db9/f23 [1811258,54783] 0 2024-04-05T21:26:51.984 INFO:tasks.workunit.client.0.smithi121.stdout:6/714: symlink d1/d4b/lfe 0 2024-04-05T21:26:51.984 INFO:tasks.workunit.client.0.smithi121.stdout:5/925: write dd/de/d19/d1c/d74/d8c/dc5/fd5 [924663,35746] 0 2024-04-05T21:26:51.985 INFO:tasks.workunit.client.0.smithi121.stdout:4/997: dwrite d5/d45/d67/f13a [0,4194304] 0 2024-04-05T21:26:51.986 INFO:tasks.workunit.client.0.smithi121.stdout:6/715: read - d1/fdc zero size 2024-04-05T21:26:51.987 INFO:tasks.workunit.client.0.smithi121.stdout:7/928: creat d0/d9/d22/de9/d125/f127 x:0 0 0 2024-04-05T21:26:51.988 INFO:tasks.workunit.client.1.smithi188.stdout:8/864: dread d5/d21/d40/d43/d44/d5b/d99/f7d [0,4194304] 0 2024-04-05T21:26:51.989 INFO:tasks.workunit.client.1.smithi188.stdout:8/865: chown d5/d21/d40/d43/dbb/fc1 0 1 2024-04-05T21:26:51.991 INFO:tasks.workunit.client.0.smithi121.stdout:4/998: write d5/d11/d26/d49/f12c [774925,45513] 0 2024-04-05T21:26:51.991 INFO:tasks.workunit.client.0.smithi121.stdout:7/929: truncate d0/d32/d4e/d73/fe4 1020928 0 2024-04-05T21:26:51.993 INFO:tasks.workunit.client.0.smithi121.stdout:3/740: creat d0/d3/d24/d3d/d1a/d1e/db1/ffd x:0 0 0 2024-04-05T21:26:51.993 INFO:tasks.workunit.client.1.smithi188.stdout:8/866: rename d5/dc/c13 to d5/d21/d40/c111 0 2024-04-05T21:26:51.994 INFO:tasks.workunit.client.0.smithi121.stdout:9/638: mknod d6/df/d48/d96/db9/d2c/d37/d3c/cdb 0 2024-04-05T21:26:51.996 INFO:tasks.workunit.client.0.smithi121.stdout:6/716: chown d1/l34 1932 1 2024-04-05T21:26:51.996 INFO:tasks.workunit.client.0.smithi121.stdout:9/639: truncate d6/d32/dbe/d7b/fb3 509237 0 2024-04-05T21:26:51.997 INFO:tasks.workunit.client.0.smithi121.stdout:9/640: stat d6/df/d50 0 2024-04-05T21:26:51.997 INFO:tasks.workunit.client.0.smithi121.stdout:0/892: link d7/d62/d3c/f104 d7/d71/d45/d46/d53/f138 0 2024-04-05T21:26:51.999 INFO:tasks.workunit.client.0.smithi121.stdout:0/893: write d7/d62/d3c/d6b/f11c [763605,103267] 0 2024-04-05T21:26:52.001 INFO:tasks.workunit.client.0.smithi121.stdout:0/894: fdatasync d7/f84 0 2024-04-05T21:26:52.004 INFO:tasks.workunit.client.0.smithi121.stdout:6/717: mknod d1/d2/daf/d6f/dc7/d1f/d6d/cff 0 2024-04-05T21:26:52.007 INFO:tasks.workunit.client.1.smithi188.stdout:8/867: dwrite d5/d21/d40/d43/d44/d5b/d99/d69/fe3 [0,4194304] 0 2024-04-05T21:26:52.010 INFO:tasks.workunit.client.0.smithi121.stdout:5/926: truncate dd/de/d19/d38/d56/f92 3367247 0 2024-04-05T21:26:52.010 INFO:tasks.workunit.client.1.smithi188.stdout:8/868: mknod d5/d21/d2a/d47/c112 0 2024-04-05T21:26:52.012 INFO:tasks.workunit.client.0.smithi121.stdout:4/999: write d5/d11/d2c/d9d/dae/f30 [2343314,8083] 0 2024-04-05T21:26:52.013 INFO:tasks.workunit.client.0.smithi121.stdout:5/927: dread dd/de/d19/d38/f138 [0,4194304] 0 2024-04-05T21:26:52.015 INFO:tasks.workunit.client.1.smithi188.stdout:8/869: dwrite d5/f12 [0,4194304] 0 2024-04-05T21:26:52.018 INFO:tasks.workunit.client.0.smithi121.stdout:3/741: dwrite d0/d3/f5b [0,4194304] 0 2024-04-05T21:26:52.021 INFO:tasks.workunit.client.0.smithi121.stdout:5/928: write dd/d41/dce/da6/dd3/db1/f12e [27966,97315] 0 2024-04-05T21:26:52.024 INFO:tasks.workunit.client.0.smithi121.stdout:7/930: getdents d0/d9/d22 0 2024-04-05T21:26:52.028 INFO:tasks.workunit.client.0.smithi121.stdout:9/641: rename d6/df/d48/d96/dc2/d79/f95 to d6/df/d48/d96/db9/d2c/d37/fdc 0 2024-04-05T21:26:52.030 INFO:tasks.workunit.client.0.smithi121.stdout:6/718: write d1/d2/d1b/d35/f47 [361393,59470] 0 2024-04-05T21:26:52.031 INFO:tasks.workunit.client.0.smithi121.stdout:5/929: write dd/de/d19/f80 [4100752,88026] 0 2024-04-05T21:26:52.031 INFO:tasks.workunit.client.0.smithi121.stdout:9/642: write d6/f9b [776639,28622] 0 2024-04-05T21:26:52.033 INFO:tasks.workunit.client.0.smithi121.stdout:7/931: dread d0/d9/d22/faf [0,4194304] 0 2024-04-05T21:26:52.033 INFO:tasks.workunit.client.0.smithi121.stdout:6/719: dread - d1/fdc zero size 2024-04-05T21:26:52.034 INFO:tasks.workunit.client.0.smithi121.stdout:0/895: dwrite d7/d71/d29/d2a/d2c/f7c [0,4194304] 0 2024-04-05T21:26:52.035 INFO:tasks.workunit.client.0.smithi121.stdout:6/720: chown d1/d4b/da7 1883785 1 2024-04-05T21:26:52.036 INFO:tasks.workunit.client.0.smithi121.stdout:9/643: dread - d6/df/d48/d96/db2/fd1 zero size 2024-04-05T21:26:52.036 INFO:tasks.workunit.client.0.smithi121.stdout:3/742: dread d0/d3/d24/d3d/d1a/d23/d38/d55/f8d [0,4194304] 0 2024-04-05T21:26:52.041 INFO:tasks.workunit.client.0.smithi121.stdout:5/930: creat dd/f147 x:0 0 0 2024-04-05T21:26:52.042 INFO:tasks.workunit.client.0.smithi121.stdout:7/932: rmdir d0/d9/d22/d28/d56/db9/ddf/def/d52/d61 39 2024-04-05T21:26:52.044 INFO:tasks.workunit.client.0.smithi121.stdout:0/896: read d7/dc/f15 [41398,128057] 0 2024-04-05T21:26:52.046 INFO:tasks.workunit.client.0.smithi121.stdout:0/897: fsync d7/d7e/f89 0 2024-04-05T21:26:52.046 INFO:tasks.workunit.client.0.smithi121.stdout:9/644: write d6/d32/dbe/d7b/f73 [3382558,36026] 0 2024-04-05T21:26:52.047 INFO:tasks.workunit.client.1.smithi188.stdout:8/870: dread d5/d21/d40/d43/d44/d59/d8b/feb [0,4194304] 0 2024-04-05T21:26:52.047 INFO:tasks.workunit.client.0.smithi121.stdout:3/743: rename d0/d3/d24/d3d/d1a/d23/d90/c9b to d0/d3/d24/d6d/d7b/da4/cfe 0 2024-04-05T21:26:52.048 INFO:tasks.workunit.client.0.smithi121.stdout:5/931: creat dd/de/d6d/f148 x:0 0 0 2024-04-05T21:26:52.049 INFO:tasks.workunit.client.1.smithi188.stdout:8/871: dread d5/d21/d40/f57 [0,4194304] 0 2024-04-05T21:26:52.050 INFO:tasks.workunit.client.0.smithi121.stdout:7/933: creat d0/d9/d22/d81/f128 x:0 0 0 2024-04-05T21:26:52.054 INFO:tasks.workunit.client.0.smithi121.stdout:0/898: truncate d7/d62/f6d 499311 0 2024-04-05T21:26:52.054 INFO:tasks.workunit.client.0.smithi121.stdout:0/899: write d7/d71/d45/d46/d53/f12e [943892,86672] 0 2024-04-05T21:26:52.055 INFO:tasks.workunit.client.0.smithi121.stdout:5/932: creat dd/de/d19/d1c/d36/f149 x:0 0 0 2024-04-05T21:26:52.056 INFO:tasks.workunit.client.0.smithi121.stdout:3/744: mkdir d0/d3/d24/d6d/dff 0 2024-04-05T21:26:52.057 INFO:tasks.workunit.client.0.smithi121.stdout:3/745: stat d0/d3/d24/db3/dc3/fc9 0 2024-04-05T21:26:52.059 INFO:tasks.workunit.client.0.smithi121.stdout:6/721: truncate d1/d2/d3/d4/d21/d57/d42/d48/d7c/fa9 301569 0 2024-04-05T21:26:52.060 INFO:tasks.workunit.client.1.smithi188.stdout:8/872: dwrite d5/d21/d40/d43/f81 [4194304,4194304] 0 2024-04-05T21:26:52.060 INFO:tasks.workunit.client.0.smithi121.stdout:0/900: creat d7/d71/d94/dd1/dd4/f139 x:0 0 0 2024-04-05T21:26:52.061 INFO:tasks.workunit.client.0.smithi121.stdout:0/901: chown d7/d71/d94/dd1/c115 0 1 2024-04-05T21:26:52.061 INFO:tasks.workunit.client.0.smithi121.stdout:9/645: rmdir d6/df/d48/d96/dc2/dc9 0 2024-04-05T21:26:52.063 INFO:tasks.workunit.client.0.smithi121.stdout:3/746: chown d0/d3/d24/d3d/cc1 1736 1 2024-04-05T21:26:52.063 INFO:tasks.workunit.client.1.smithi188.stdout:8/873: rmdir d5/d21/d40/d43/d44/d5b/d99/d69 39 2024-04-05T21:26:52.064 INFO:tasks.workunit.client.0.smithi121.stdout:6/722: unlink d1/d2/d1b/d35/d39/fcd 0 2024-04-05T21:26:52.065 INFO:tasks.workunit.client.0.smithi121.stdout:7/934: getdents d0/d9/d22/d28/d56/db9/ddf 0 2024-04-05T21:26:52.066 INFO:tasks.workunit.client.0.smithi121.stdout:3/747: truncate d0/d3/d24/d3d/d1a/d23/f6b 533155 0 2024-04-05T21:26:52.070 INFO:tasks.workunit.client.0.smithi121.stdout:7/935: mknod d0/d9/d22/d3e/c129 0 2024-04-05T21:26:52.072 INFO:tasks.workunit.client.0.smithi121.stdout:5/933: getdents dd/d41/dce/da6/dd3/db1 0 2024-04-05T21:26:52.073 INFO:tasks.workunit.client.0.smithi121.stdout:6/723: dread d1/f72 [0,4194304] 0 2024-04-05T21:26:52.074 INFO:tasks.workunit.client.0.smithi121.stdout:9/646: link d6/d32/dbe/d7b/fb3 d6/df/d48/d96/dc2/d79/d9c/fdd 0 2024-04-05T21:26:52.075 INFO:tasks.workunit.client.0.smithi121.stdout:0/902: dwrite d7/d71/d29/de8/ffe [0,4194304] 0 2024-04-05T21:26:52.077 INFO:tasks.workunit.client.1.smithi188.stdout:8/874: write d5/dc/f33 [4726269,119574] 0 2024-04-05T21:26:52.078 INFO:tasks.workunit.client.0.smithi121.stdout:9/647: dread - d6/df/d50/fd9 zero size 2024-04-05T21:26:52.078 INFO:tasks.workunit.client.0.smithi121.stdout:3/748: chown d0/d3/de/da6/cd3 1779343 1 2024-04-05T21:26:52.081 INFO:tasks.workunit.client.1.smithi188.stdout:8/875: dread d5/f8 [0,4194304] 0 2024-04-05T21:26:52.083 INFO:tasks.workunit.client.0.smithi121.stdout:0/903: dread - d7/d62/fe2 zero size 2024-04-05T21:26:52.085 INFO:tasks.workunit.client.0.smithi121.stdout:3/749: fsync d0/d3/d24/d3d/d1a/d1e/d86/f8b 0 2024-04-05T21:26:52.085 INFO:tasks.workunit.client.0.smithi121.stdout:9/648: truncate d6/df/d48/d96/db9/d2c/d37/d3c/f9d 939103 0 2024-04-05T21:26:52.086 INFO:tasks.workunit.client.1.smithi188.stdout:8/876: rename d5/d21/c23 to d5/d6e/d98/dc6/c113 0 2024-04-05T21:26:52.086 INFO:tasks.workunit.client.0.smithi121.stdout:6/724: read d1/d2/d3/d4/d21/f82 [932500,63674] 0 2024-04-05T21:26:52.088 INFO:tasks.workunit.client.0.smithi121.stdout:0/904: dread d7/f3a [0,4194304] 0 2024-04-05T21:26:52.090 INFO:tasks.workunit.client.0.smithi121.stdout:9/649: creat d6/df/d48/d96/db9/d2c/d37/d3c/fde x:0 0 0 2024-04-05T21:26:52.091 INFO:tasks.workunit.client.0.smithi121.stdout:0/905: mknod d7/d62/d8b/c13a 0 2024-04-05T21:26:52.091 INFO:tasks.workunit.client.1.smithi188.stdout:8/877: rename d5/d21/d40/d43/dbb to d5/d21/d40/d43/d114 0 2024-04-05T21:26:52.093 INFO:tasks.workunit.client.0.smithi121.stdout:6/725: read d1/d32/fae [395623,113307] 0 2024-04-05T21:26:52.093 INFO:tasks.workunit.client.0.smithi121.stdout:3/750: rename d0/d3/d24/d3d/d1a/d1e/db1/ff1 to d0/d3/d24/d6d/dff/f100 0 2024-04-05T21:26:52.094 INFO:tasks.workunit.client.0.smithi121.stdout:9/650: truncate d6/d32/dbe/d7b/d6b/f78 155182 0 2024-04-05T21:26:52.096 INFO:tasks.workunit.client.0.smithi121.stdout:6/726: fdatasync d1/d2/d1b/d2e/fe5 0 2024-04-05T21:26:52.097 INFO:tasks.workunit.client.0.smithi121.stdout:3/751: read d0/d3/d24/d3d/d1a/d1e/d86/fde [2227511,95218] 0 2024-04-05T21:26:52.097 INFO:tasks.workunit.client.0.smithi121.stdout:9/651: readlink d6/df/d48/d96/db9/d2c/l90 0 2024-04-05T21:26:52.100 INFO:tasks.workunit.client.0.smithi121.stdout:5/934: dread dd/de/d19/d13/f26 [0,4194304] 0 2024-04-05T21:26:52.100 INFO:tasks.workunit.client.0.smithi121.stdout:0/906: dwrite d7/d62/d3c/d6b/f111 [0,4194304] 0 2024-04-05T21:26:52.100 INFO:tasks.workunit.client.1.smithi188.stdout:8/878: dwrite d5/d21/d40/d43/d44/d5b/d99/da3/fba [4194304,4194304] 0 2024-04-05T21:26:52.101 INFO:tasks.workunit.client.0.smithi121.stdout:9/652: creat d6/d46/d8a/fdf x:0 0 0 2024-04-05T21:26:52.102 INFO:tasks.workunit.client.0.smithi121.stdout:5/935: readlink dd/de/lb4 0 2024-04-05T21:26:52.104 INFO:tasks.workunit.client.0.smithi121.stdout:3/752: truncate d0/d3/d24/f89 1538188 0 2024-04-05T21:26:52.105 INFO:tasks.workunit.client.0.smithi121.stdout:9/653: symlink d6/df/d48/le0 0 2024-04-05T21:26:52.105 INFO:tasks.workunit.client.0.smithi121.stdout:5/936: readlink dd/de/d19/d38/d45/l66 0 2024-04-05T21:26:52.106 INFO:tasks.workunit.client.0.smithi121.stdout:0/907: symlink d7/d71/d29/d2a/d39/l13b 0 2024-04-05T21:26:52.109 INFO:tasks.workunit.client.0.smithi121.stdout:3/753: mknod d0/d3/d24/d6d/d7b/c101 0 2024-04-05T21:26:52.110 INFO:tasks.workunit.client.0.smithi121.stdout:7/936: dread d0/d9/d22/d28/d56/db9/ddf/def/f43 [0,4194304] 0 2024-04-05T21:26:52.111 INFO:tasks.workunit.client.0.smithi121.stdout:5/937: rename dd/de/d6d/fc7 to dd/de/d39/d3b/de8/f14a 0 2024-04-05T21:26:52.112 INFO:tasks.workunit.client.0.smithi121.stdout:9/654: symlink d6/d46/le1 0 2024-04-05T21:26:52.113 INFO:tasks.workunit.client.0.smithi121.stdout:3/754: mkdir d0/d3/d24/d6d/d7b/da3/dec/df6/d102 0 2024-04-05T21:26:52.114 INFO:tasks.workunit.client.0.smithi121.stdout:9/655: fsync d6/df/d48/d96/db2/fd1 0 2024-04-05T21:26:52.117 INFO:tasks.workunit.client.0.smithi121.stdout:5/938: mkdir dd/de/df9/d14b 0 2024-04-05T21:26:52.118 INFO:tasks.workunit.client.0.smithi121.stdout:0/908: getdents d7 0 2024-04-05T21:26:52.120 INFO:tasks.workunit.client.0.smithi121.stdout:9/656: rename d6/d46/ca2 to d6/d32/dbe/ce2 0 2024-04-05T21:26:52.124 INFO:tasks.workunit.client.0.smithi121.stdout:3/755: dwrite d0/d3/d24/d3d/fa8 [0,4194304] 0 2024-04-05T21:26:52.126 INFO:tasks.workunit.client.0.smithi121.stdout:6/727: truncate d1/d2/d3/d4/d21/d57/f49 1308183 0 2024-04-05T21:26:52.127 INFO:tasks.workunit.client.0.smithi121.stdout:3/756: write d0/d3/d24/d3d/d1a/d23/d38/dae/fd5 [282475,76220] 0 2024-04-05T21:26:52.130 INFO:tasks.workunit.client.0.smithi121.stdout:0/909: mknod d7/d71/d45/d46/d106/d11e/c13c 0 2024-04-05T21:26:52.132 INFO:tasks.workunit.client.0.smithi121.stdout:6/728: mknod d1/d2/daf/d6f/c100 0 2024-04-05T21:26:52.134 INFO:tasks.workunit.client.0.smithi121.stdout:3/757: mkdir d0/d3/d24/d3d/d1a/d23/d38/d55/d103 0 2024-04-05T21:26:52.135 INFO:tasks.workunit.client.0.smithi121.stdout:6/729: dread d1/d2/daf/d6f/dc7/d1f/fd0 [0,4194304] 0 2024-04-05T21:26:52.135 INFO:tasks.workunit.client.0.smithi121.stdout:7/937: truncate d0/d9/f21 811586 0 2024-04-05T21:26:52.138 INFO:tasks.workunit.client.0.smithi121.stdout:0/910: rename d7/d62/lb8 to d7/l13d 0 2024-04-05T21:26:52.143 INFO:tasks.workunit.client.0.smithi121.stdout:3/758: creat d0/d3/de/da6/da1/f104 x:0 0 0 2024-04-05T21:26:52.144 INFO:tasks.workunit.client.0.smithi121.stdout:7/938: dwrite d0/d23/f8b [8388608,4194304] 0 2024-04-05T21:26:52.144 INFO:tasks.workunit.client.0.smithi121.stdout:5/939: write dd/de/d19/d3c/d8e/f136 [3643442,130700] 0 2024-04-05T21:26:52.145 INFO:tasks.workunit.client.0.smithi121.stdout:6/730: write d1/d2/daf/d6f/dc7/d1f/f46 [3017224,127027] 0 2024-04-05T21:26:52.148 INFO:tasks.workunit.client.0.smithi121.stdout:6/731: readlink d1/d2/dc/d9a/lb2 0 2024-04-05T21:26:52.150 INFO:tasks.workunit.client.0.smithi121.stdout:3/759: creat d0/d3/d24/d3d/d1a/d1e/f105 x:0 0 0 2024-04-05T21:26:52.153 INFO:tasks.workunit.client.0.smithi121.stdout:7/939: creat d0/d9/d22/d81/da2/f12a x:0 0 0 2024-04-05T21:26:52.156 INFO:tasks.workunit.client.0.smithi121.stdout:9/657: dwrite d6/d32/dbe/d7b/d6b/f92 [0,4194304] 0 2024-04-05T21:26:52.159 INFO:tasks.workunit.client.0.smithi121.stdout:6/732: mkdir d1/d2/daf/d6f/dc7/d1f/d6d/d101 0 2024-04-05T21:26:52.162 INFO:tasks.workunit.client.0.smithi121.stdout:9/658: write d6/d32/dbe/d7b/d49/f8b [1198930,57015] 0 2024-04-05T21:26:52.162 INFO:tasks.workunit.client.0.smithi121.stdout:3/760: symlink d0/d3/d24/d3d/dcf/l106 0 2024-04-05T21:26:52.165 INFO:tasks.workunit.client.0.smithi121.stdout:7/940: dwrite d0/d23/d5a/f62 [0,4194304] 0 2024-04-05T21:26:52.168 INFO:tasks.workunit.client.0.smithi121.stdout:6/733: creat d1/d2/d1b/d91/f102 x:0 0 0 2024-04-05T21:26:52.168 INFO:tasks.workunit.client.0.smithi121.stdout:3/761: mknod d0/d3/d24/db3/c107 0 2024-04-05T21:26:52.172 INFO:tasks.workunit.client.0.smithi121.stdout:0/911: symlink d7/d71/d45/l13e 0 2024-04-05T21:26:52.173 INFO:tasks.workunit.client.0.smithi121.stdout:5/940: write dd/de/d19/d38/d9b/fb0 [1076227,92224] 0 2024-04-05T21:26:52.176 INFO:tasks.workunit.client.0.smithi121.stdout:7/941: truncate d0/d9/d22/d81/da2/fc7 707740 0 2024-04-05T21:26:52.178 INFO:tasks.workunit.client.0.smithi121.stdout:9/659: dread d6/df/d48/d96/db9/d72/f7f [0,4194304] 0 2024-04-05T21:26:52.179 INFO:tasks.workunit.client.0.smithi121.stdout:0/912: write d7/d71/d94/da0/fdf [882439,40732] 0 2024-04-05T21:26:52.180 INFO:tasks.workunit.client.0.smithi121.stdout:3/762: rename d0/d3/d24/d3d/d1a/d23/d38/fb7 to d0/d3/d24/d6d/d7b/da3/dec/df9/f108 0 2024-04-05T21:26:52.181 INFO:tasks.workunit.client.0.smithi121.stdout:6/734: dread d1/d2/d1b/d5e/dbd/faa [0,4194304] 0 2024-04-05T21:26:52.185 INFO:tasks.workunit.client.0.smithi121.stdout:7/942: dread d0/d23/f74 [0,4194304] 0 2024-04-05T21:26:52.186 INFO:tasks.workunit.client.0.smithi121.stdout:3/763: write d0/d3/d24/d3d/d1a/d1e/db1/ffd [295998,37935] 0 2024-04-05T21:26:52.187 INFO:tasks.workunit.client.0.smithi121.stdout:0/913: truncate d7/d62/fa8 68712 0 2024-04-05T21:26:52.188 INFO:tasks.workunit.client.0.smithi121.stdout:5/941: dwrite dd/de/d19/d13/f16 [0,4194304] 0 2024-04-05T21:26:52.189 INFO:tasks.workunit.client.0.smithi121.stdout:3/764: mknod d0/d3/d24/c109 0 2024-04-05T21:26:52.190 INFO:tasks.workunit.client.0.smithi121.stdout:6/735: truncate d1/d2/daf/d6f/dc7/d1f/fd0 169687 0 2024-04-05T21:26:52.192 INFO:tasks.workunit.client.0.smithi121.stdout:6/736: chown d1/d2/d3/d59/c98 72036896 1 2024-04-05T21:26:52.234 INFO:tasks.workunit.client.0.smithi121.stdout:0/914: creat d7/d71/d29/d2a/f13f x:0 0 0 2024-04-05T21:26:52.236 INFO:tasks.workunit.client.0.smithi121.stdout:3/765: mkdir d0/d3/d24/d6d/d10a 0 2024-04-05T21:26:52.236 INFO:tasks.workunit.client.0.smithi121.stdout:5/942: write dd/de/d19/d1c/d7b/d88/fcc [1495793,73731] 0 2024-04-05T21:26:52.238 INFO:tasks.workunit.client.0.smithi121.stdout:6/737: creat d1/d2/daf/d6f/dc7/d1f/d6d/d101/f103 x:0 0 0 2024-04-05T21:26:52.239 INFO:tasks.workunit.client.0.smithi121.stdout:5/943: readlink dd/lf6 0 2024-04-05T21:26:52.239 INFO:tasks.workunit.client.0.smithi121.stdout:6/738: chown d1/d2/d1b/d35/d39 1976818 1 2024-04-05T21:26:52.240 INFO:tasks.workunit.client.0.smithi121.stdout:9/660: dread d6/df/d48/d96/db9/d2c/d37/fdc [0,4194304] 0 2024-04-05T21:26:52.242 INFO:tasks.workunit.client.0.smithi121.stdout:0/915: creat d7/d71/d29/d2a/d10b/f140 x:0 0 0 2024-04-05T21:26:52.243 INFO:tasks.workunit.client.0.smithi121.stdout:5/944: creat dd/d41/f14c x:0 0 0 2024-04-05T21:26:52.244 INFO:tasks.workunit.client.0.smithi121.stdout:0/916: fdatasync d7/d71/d29/d2a/d2c/f7c 0 2024-04-05T21:26:52.245 INFO:tasks.workunit.client.0.smithi121.stdout:9/661: symlink d6/d46/d8a/le3 0 2024-04-05T21:26:52.252 INFO:tasks.workunit.client.0.smithi121.stdout:3/766: dwrite d0/d3/d24/d3d/d1a/d23/d38/f65 [0,4194304] 0 2024-04-05T21:26:52.254 INFO:tasks.workunit.client.0.smithi121.stdout:6/739: link d1/d2/d3/d4/d21/d57/d42/d48/d7c/db3/fc1 d1/d2/d3/f104 0 2024-04-05T21:26:52.255 INFO:tasks.workunit.client.0.smithi121.stdout:0/917: write d7/d71/f5f [186931,21876] 0 2024-04-05T21:26:52.257 INFO:tasks.workunit.client.0.smithi121.stdout:5/945: rmdir dd/de/d6d/dd9 0 2024-04-05T21:26:52.263 INFO:tasks.workunit.client.0.smithi121.stdout:6/740: rename d1/d2/d1b/da1 to d1/d2/d1b/d35/d51/d105 0 2024-04-05T21:26:52.263 INFO:tasks.workunit.client.0.smithi121.stdout:9/662: dread d6/df/d48/d96/db9/d2c/d37/f86 [0,4194304] 0 2024-04-05T21:26:52.264 INFO:tasks.workunit.client.0.smithi121.stdout:3/767: dwrite d0/d3/d24/d3d/d1a/d1e/db1/fbb [0,4194304] 0 2024-04-05T21:26:52.268 INFO:tasks.workunit.client.0.smithi121.stdout:5/946: symlink dd/d41/dce/da6/dd3/l14d 0 2024-04-05T21:26:52.273 INFO:tasks.workunit.client.0.smithi121.stdout:9/663: creat d6/d32/dbe/dba/fe4 x:0 0 0 2024-04-05T21:26:52.279 INFO:tasks.workunit.client.0.smithi121.stdout:3/768: truncate d0/f7 631217 0 2024-04-05T21:26:52.279 INFO:tasks.workunit.client.0.smithi121.stdout:0/918: dwrite d7/d71/d45/d46/f57 [0,4194304] 0 2024-04-05T21:26:52.281 INFO:tasks.workunit.client.0.smithi121.stdout:6/741: dwrite d1/d2/dc/f4c [0,4194304] 0 2024-04-05T21:26:52.282 INFO:tasks.workunit.client.0.smithi121.stdout:5/947: dwrite dd/de/d19/d38/d56/fe0 [4194304,4194304] 0 2024-04-05T21:26:52.291 INFO:tasks.workunit.client.0.smithi121.stdout:9/664: rename d6/f1c to d6/df/d48/d96/db9/d2c/d53/fe5 0 2024-04-05T21:26:52.291 INFO:tasks.workunit.client.0.smithi121.stdout:3/769: write d0/d3/d24/d3d/d1a/f4e [157612,115202] 0 2024-04-05T21:26:52.291 INFO:tasks.workunit.client.0.smithi121.stdout:9/665: stat d6/df/d48/f7e 0 2024-04-05T21:26:52.293 INFO:tasks.workunit.client.0.smithi121.stdout:9/666: write d6/d32/dbe/d7b/d6b/f92 [4180710,62293] 0 2024-04-05T21:26:52.296 INFO:tasks.workunit.client.0.smithi121.stdout:6/742: read d1/d2/d3/d4/d21/d57/f49 [575066,13158] 0 2024-04-05T21:26:52.296 INFO:tasks.workunit.client.0.smithi121.stdout:5/948: dwrite dd/de/d19/d1c/d7b/d88/fcc [0,4194304] 0 2024-04-05T21:26:52.299 INFO:tasks.workunit.client.0.smithi121.stdout:3/770: getdents d0/d3/de/da6 0 2024-04-05T21:26:52.299 INFO:tasks.workunit.client.0.smithi121.stdout:9/667: chown d6/d32/dbe/d7b/c6e 61 1 2024-04-05T21:26:52.302 INFO:tasks.workunit.client.0.smithi121.stdout:9/668: fdatasync d6/d32/dbe/d7b/d49/fb6 0 2024-04-05T21:26:52.302 INFO:tasks.workunit.client.0.smithi121.stdout:5/949: readlink dd/de/d19/d38/d45/l55 0 2024-04-05T21:26:52.305 INFO:tasks.workunit.client.0.smithi121.stdout:6/743: unlink d1/d2/d3/d4/d21/d57/d42/d48/d7c/cfc 0 2024-04-05T21:26:52.311 INFO:tasks.workunit.client.0.smithi121.stdout:3/771: read d0/d3/d24/d3d/f3a [2833546,106185] 0 2024-04-05T21:26:52.314 INFO:tasks.workunit.client.0.smithi121.stdout:5/950: mkdir dd/de/d39/d3b/d50/d14e 0 2024-04-05T21:26:52.314 INFO:tasks.workunit.client.0.smithi121.stdout:6/744: creat d1/d2/d1b/d35/d51/d105/f106 x:0 0 0 2024-04-05T21:26:52.317 INFO:tasks.workunit.client.0.smithi121.stdout:5/951: truncate dd/d41/f14c 803339 0 2024-04-05T21:26:52.319 INFO:tasks.workunit.client.0.smithi121.stdout:6/745: fdatasync d1/d2/d3/d4/d21/f82 0 2024-04-05T21:26:52.319 INFO:tasks.workunit.client.0.smithi121.stdout:3/772: dwrite d0/d3/d24/d3d/d1a/f22 [0,4194304] 0 2024-04-05T21:26:52.320 INFO:tasks.workunit.client.0.smithi121.stdout:5/952: mkdir dd/de/d39/d3b/de8/d14f 0 2024-04-05T21:26:52.321 INFO:tasks.workunit.client.0.smithi121.stdout:3/773: write d0/f97 [844719,87687] 0 2024-04-05T21:26:52.325 INFO:tasks.workunit.client.0.smithi121.stdout:3/774: chown d0/d3/d24/d6d/d7b/da3/dec/df6/f4b 81428144 1 2024-04-05T21:26:52.326 INFO:tasks.workunit.client.0.smithi121.stdout:5/953: unlink dd/d41/dce/da6/dd3/db1/cef 0 2024-04-05T21:26:52.327 INFO:tasks.workunit.client.0.smithi121.stdout:5/954: fdatasync dd/de/d19/d1c/d36/f3a 0 2024-04-05T21:26:52.329 INFO:tasks.workunit.client.0.smithi121.stdout:9/669: dread d6/df/d48/d96/db9/f33 [0,4194304] 0 2024-04-05T21:26:52.330 INFO:tasks.workunit.client.0.smithi121.stdout:9/670: dread - d6/fd8 zero size 2024-04-05T21:26:52.330 INFO:tasks.workunit.client.0.smithi121.stdout:6/746: link d1/f72 d1/d2/daf/d6f/f107 0 2024-04-05T21:26:52.333 INFO:tasks.workunit.client.0.smithi121.stdout:5/955: mkdir dd/de/d39/d3b/de8/d150 0 2024-04-05T21:26:52.334 INFO:tasks.workunit.client.0.smithi121.stdout:3/775: rmdir d0/d3/d24/d6d/d10a 0 2024-04-05T21:26:52.338 INFO:tasks.workunit.client.0.smithi121.stdout:9/671: dread d6/df/d48/d96/db9/f33 [0,4194304] 0 2024-04-05T21:26:52.339 INFO:tasks.workunit.client.0.smithi121.stdout:5/956: rename dd/f65 to dd/de/d19/d13/dd4/f151 0 2024-04-05T21:26:52.339 INFO:tasks.workunit.client.0.smithi121.stdout:6/747: dwrite d1/d4b/dcc/fe9 [0,4194304] 0 2024-04-05T21:26:52.342 INFO:tasks.workunit.client.0.smithi121.stdout:6/748: truncate d1/d2/d1b/d2e/fe5 557093 0 2024-04-05T21:26:52.346 INFO:tasks.workunit.client.0.smithi121.stdout:5/957: creat dd/de/d19/d1c/d74/d8c/f152 x:0 0 0 2024-04-05T21:26:52.347 INFO:tasks.workunit.client.0.smithi121.stdout:6/749: creat d1/d2/d1b/d35/d74/f108 x:0 0 0 2024-04-05T21:26:52.348 INFO:tasks.workunit.client.0.smithi121.stdout:9/672: dread d6/d32/dbe/d7b/f3b [0,4194304] 0 2024-04-05T21:26:52.348 INFO:tasks.workunit.client.0.smithi121.stdout:5/958: read dd/de/d19/d13/d53/f7e [88267,29007] 0 2024-04-05T21:26:52.351 INFO:tasks.workunit.client.0.smithi121.stdout:3/776: truncate d0/d3/f54 3689890 0 2024-04-05T21:26:52.354 INFO:tasks.workunit.client.0.smithi121.stdout:3/777: truncate d0/d3/de/f9c 730172 0 2024-04-05T21:26:52.354 INFO:tasks.workunit.client.0.smithi121.stdout:6/750: rename d1/d32 to d1/d2/d1b/d5e/dbd/d109 0 2024-04-05T21:26:52.356 INFO:tasks.workunit.client.0.smithi121.stdout:5/959: dwrite dd/de/d39/d3b/f94 [4194304,4194304] 0 2024-04-05T21:26:52.356 INFO:tasks.workunit.client.0.smithi121.stdout:9/673: creat d6/df/d48/d96/db9/fe6 x:0 0 0 2024-04-05T21:26:52.358 INFO:tasks.workunit.client.0.smithi121.stdout:3/778: write d0/d3/d24/db3/dc3/fc9 [132101,109105] 0 2024-04-05T21:26:52.358 INFO:tasks.workunit.client.0.smithi121.stdout:6/751: chown d1/d2/d1b/d35/d39/d92/fe3 3224967 1 2024-04-05T21:26:52.360 INFO:tasks.workunit.client.0.smithi121.stdout:9/674: read - d6/d32/dbe/d7b/d49/fb6 zero size 2024-04-05T21:26:52.360 INFO:tasks.workunit.client.0.smithi121.stdout:9/675: stat f2 0 2024-04-05T21:26:52.365 INFO:tasks.workunit.client.0.smithi121.stdout:5/960: getdents dd/de/d39/d3b/d50 0 2024-04-05T21:26:52.366 INFO:tasks.workunit.client.0.smithi121.stdout:6/752: dwrite d1/d2/d1b/d35/f47 [0,4194304] 0 2024-04-05T21:26:52.371 INFO:tasks.workunit.client.0.smithi121.stdout:5/961: rename dd/d41/dce/da6/d10f to dd/de/d39/d3b/de8/d150/d153 0 2024-04-05T21:26:52.372 INFO:tasks.workunit.client.0.smithi121.stdout:5/962: write dd/de/d19/f37 [2895609,36308] 0 2024-04-05T21:26:52.373 INFO:tasks.workunit.client.0.smithi121.stdout:6/753: creat d1/d2/d1b/d2e/dd9/f10a x:0 0 0 2024-04-05T21:26:52.389 INFO:tasks.workunit.client.0.smithi121.stdout:6/754: dread d1/d2/d3/d4/d21/f93 [0,4194304] 0 2024-04-05T21:26:52.389 INFO:tasks.workunit.client.0.smithi121.stdout:5/963: dread dd/de/d39/d3b/fc0 [0,4194304] 0 2024-04-05T21:26:52.391 INFO:tasks.workunit.client.0.smithi121.stdout:6/755: creat d1/d2/dc/f10b x:0 0 0 2024-04-05T21:26:52.393 INFO:tasks.workunit.client.0.smithi121.stdout:6/756: creat d1/d4b/da7/f10c x:0 0 0 2024-04-05T21:26:52.397 INFO:tasks.workunit.client.0.smithi121.stdout:6/757: getdents d1/d2/d1b/d5e/dbd 0 2024-04-05T21:26:52.398 INFO:tasks.workunit.client.0.smithi121.stdout:6/758: symlink d1/d2/d3/d4/d21/d57/d42/l10d 0 2024-04-05T21:26:52.400 INFO:tasks.workunit.client.0.smithi121.stdout:6/759: dread - d1/d2/d1b/d5e/fef zero size 2024-04-05T21:26:52.401 INFO:tasks.workunit.client.0.smithi121.stdout:6/760: readlink d1/d2/dc/d9a/dc0/le8 0 2024-04-05T21:26:52.402 INFO:tasks.workunit.client.0.smithi121.stdout:5/964: dwrite dd/de/d19/d13/d53/f7e [0,4194304] 0 2024-04-05T21:26:52.403 INFO:tasks.workunit.client.0.smithi121.stdout:6/761: chown d1/d2/daf/d6f/dc7/d1f/f64 1072773533 1 2024-04-05T21:26:52.404 INFO:tasks.workunit.client.0.smithi121.stdout:6/762: read - d1/d2/dc/d9a/fb6 zero size 2024-04-05T21:26:52.405 INFO:tasks.workunit.client.0.smithi121.stdout:5/965: mknod dd/de/dc1/c154 0 2024-04-05T21:26:52.407 INFO:tasks.workunit.client.0.smithi121.stdout:6/763: creat d1/d2/daf/d6f/dc7/f10e x:0 0 0 2024-04-05T21:26:52.410 INFO:tasks.workunit.client.0.smithi121.stdout:5/966: creat dd/d41/dce/f155 x:0 0 0 2024-04-05T21:26:52.412 INFO:tasks.workunit.client.0.smithi121.stdout:5/967: write dd/de/d19/d1c/d36/ffc [587287,64113] 0 2024-04-05T21:26:52.412 INFO:tasks.workunit.client.0.smithi121.stdout:5/968: chown dd/de/d19/f37 269717597 1 2024-04-05T21:26:52.420 INFO:tasks.workunit.client.0.smithi121.stdout:8/786: sync 2024-04-05T21:26:52.420 INFO:tasks.workunit.client.0.smithi121.stdout:2/819: sync 2024-04-05T21:26:52.420 INFO:tasks.workunit.client.0.smithi121.stdout:1/874: sync 2024-04-05T21:26:52.427 INFO:tasks.workunit.client.0.smithi121.stdout:6/764: dread d1/d2/d3/d4/f88 [0,4194304] 0 2024-04-05T21:26:52.429 INFO:tasks.workunit.client.0.smithi121.stdout:2/820: stat d2/d9/d38/d2d/d4c/d79 0 2024-04-05T21:26:52.430 INFO:tasks.workunit.client.0.smithi121.stdout:6/765: chown d1/d2/d3/d4/d21/d57/f49 2055815 1 2024-04-05T21:26:52.431 INFO:tasks.workunit.client.0.smithi121.stdout:6/766: write d1/d2/daf/d6f/dc7/d1f/f64 [4659137,31817] 0 2024-04-05T21:26:52.437 INFO:tasks.workunit.client.0.smithi121.stdout:5/969: dread dd/de/d19/d13/f3f [0,4194304] 0 2024-04-05T21:26:52.442 INFO:tasks.workunit.client.0.smithi121.stdout:1/875: unlink d8/d89/f98 0 2024-04-05T21:26:52.443 INFO:tasks.workunit.client.0.smithi121.stdout:2/821: unlink d2/c5b 0 2024-04-05T21:26:52.446 INFO:tasks.workunit.client.0.smithi121.stdout:1/876: rmdir d8/d8c/db6/d1a/d1b 39 2024-04-05T21:26:52.447 INFO:tasks.workunit.client.0.smithi121.stdout:6/767: unlink d1/d2/d1b/d35/d51/d105/fe2 0 2024-04-05T21:26:52.449 INFO:tasks.workunit.client.0.smithi121.stdout:2/822: creat d2/d9/d38/d2d/d4c/de9/d5c/df0/f109 x:0 0 0 2024-04-05T21:26:52.450 INFO:tasks.workunit.client.0.smithi121.stdout:1/877: mknod d8/d8c/c122 0 2024-04-05T21:26:52.452 INFO:tasks.workunit.client.0.smithi121.stdout:6/768: unlink d1/d2/d1b/d35/c6b 0 2024-04-05T21:26:52.454 INFO:tasks.workunit.client.0.smithi121.stdout:2/823: creat d2/d9/d1c/f10a x:0 0 0 2024-04-05T21:26:52.455 INFO:tasks.workunit.client.0.smithi121.stdout:8/787: dread d12/d13/d15/f21 [0,4194304] 0 2024-04-05T21:26:52.455 INFO:tasks.workunit.client.0.smithi121.stdout:2/824: stat d2/d9/d17/l2b 0 2024-04-05T21:26:52.455 INFO:tasks.workunit.client.0.smithi121.stdout:1/878: mknod d8/d89/d9e/dc8/df4/d107/c123 0 2024-04-05T21:26:52.456 INFO:tasks.workunit.client.0.smithi121.stdout:6/769: symlink d1/d2/d1b/d2e/dd9/l10f 0 2024-04-05T21:26:52.459 INFO:tasks.workunit.client.0.smithi121.stdout:5/970: dread dd/de/d19/d3c/f40 [0,4194304] 0 2024-04-05T21:26:52.462 INFO:tasks.workunit.client.0.smithi121.stdout:6/770: fdatasync d1/d2/d1b/d5e/fef 0 2024-04-05T21:26:52.462 INFO:tasks.workunit.client.0.smithi121.stdout:5/971: dread - dd/de/d39/d3b/de8/d150/d153/f11f zero size 2024-04-05T21:26:52.462 INFO:tasks.workunit.client.0.smithi121.stdout:5/972: stat dd/de/f34 0 2024-04-05T21:26:52.463 INFO:tasks.workunit.client.0.smithi121.stdout:5/973: truncate dd/de/f34 4432724 0 2024-04-05T21:26:52.464 INFO:tasks.workunit.client.0.smithi121.stdout:2/825: rename d2/d9/d38/d2d/d4c/de9/d8d/dd8 to d2/d9/d1c/d10b 0 2024-04-05T21:26:52.466 INFO:tasks.workunit.client.0.smithi121.stdout:5/974: mknod dd/de/d19/d3c/d8e/c156 0 2024-04-05T21:26:52.466 INFO:tasks.workunit.client.0.smithi121.stdout:2/826: read d2/d9/d38/d2d/d4c/de5/ff6 [475609,64151] 0 2024-04-05T21:26:52.469 INFO:tasks.workunit.client.0.smithi121.stdout:1/879: rename d8/d8c/db6/d1a/d1b/d4c/da1/fe6 to d8/d89/d9e/dc8/df4/d107/f124 0 2024-04-05T21:26:52.474 INFO:tasks.workunit.client.0.smithi121.stdout:6/771: link d1/d2/dc/d9a/dc0/c8f d1/d2/daf/d6f/dc7/c110 0 2024-04-05T21:26:52.477 INFO:tasks.workunit.client.0.smithi121.stdout:2/827: mknod d2/d9/d1c/d30/d59/c10c 0 2024-04-05T21:26:52.478 INFO:tasks.workunit.client.0.smithi121.stdout:5/975: dwrite dd/de/d19/d38/f97 [0,4194304] 0 2024-04-05T21:26:52.480 INFO:tasks.workunit.client.0.smithi121.stdout:1/880: link d8/f1c d8/d89/d9e/d2d/f125 0 2024-04-05T21:26:52.482 INFO:tasks.workunit.client.0.smithi121.stdout:1/881: dread - d8/d8c/db6/d1a/d1b/d24/d79/f9a zero size 2024-04-05T21:26:52.483 INFO:tasks.workunit.client.0.smithi121.stdout:1/882: stat d8/d8c/db6/d1a/d1b/d85/c95 0 2024-04-05T21:26:52.484 INFO:tasks.workunit.client.0.smithi121.stdout:5/976: rmdir dd/de/d19/d1c/d74/d10b 39 2024-04-05T21:26:52.485 INFO:tasks.workunit.client.0.smithi121.stdout:5/977: dread - dd/de/d19/d1c/d7b/f11b zero size 2024-04-05T21:26:52.487 INFO:tasks.workunit.client.0.smithi121.stdout:2/828: creat d2/d9/d17/de2/f10d x:0 0 0 2024-04-05T21:26:52.488 INFO:tasks.workunit.client.0.smithi121.stdout:5/978: write dd/de/d19/d1c/f48 [616332,81651] 0 2024-04-05T21:26:52.488 INFO:tasks.workunit.client.0.smithi121.stdout:1/883: mknod d8/d8c/db6/d1a/d1b/d85/c126 0 2024-04-05T21:26:52.489 INFO:tasks.workunit.client.0.smithi121.stdout:5/979: write dd/f11e [415776,118814] 0 2024-04-05T21:26:52.491 INFO:tasks.workunit.client.0.smithi121.stdout:5/980: chown dd/de/d19/f23 2219868 1 2024-04-05T21:26:52.492 INFO:tasks.workunit.client.0.smithi121.stdout:2/829: creat d2/d9/d38/d2d/d4c/de9/d5c/f10e x:0 0 0 2024-04-05T21:26:52.494 INFO:tasks.workunit.client.0.smithi121.stdout:5/981: creat dd/de/dc1/f157 x:0 0 0 2024-04-05T21:26:52.496 INFO:tasks.workunit.client.0.smithi121.stdout:5/982: write dd/d41/dce/da6/f132 [593790,121775] 0 2024-04-05T21:26:52.500 INFO:tasks.workunit.client.0.smithi121.stdout:2/830: dwrite d2/d9/d1c/d30/fb1 [0,4194304] 0 2024-04-05T21:26:52.504 INFO:tasks.workunit.client.0.smithi121.stdout:2/831: write d2/d9/d17/d3a/f4a [290225,82803] 0 2024-04-05T21:26:52.505 INFO:tasks.workunit.client.1.smithi188.stdout:8/879: sync 2024-04-05T21:26:52.506 INFO:tasks.workunit.client.0.smithi121.stdout:8/788: write d12/d50/d93/d96/f8a [89109,56318] 0 2024-04-05T21:26:52.508 INFO:tasks.workunit.client.1.smithi188.stdout:8/880: getdents d5/d21/d40/d43/d48/d7e/dcf 0 2024-04-05T21:26:52.510 INFO:tasks.workunit.client.0.smithi121.stdout:5/983: dwrite dd/de/d19/d13/d53/f11a [0,4194304] 0 2024-04-05T21:26:52.510 INFO:tasks.workunit.client.0.smithi121.stdout:2/832: creat d2/d9/dcb/f10f x:0 0 0 2024-04-05T21:26:52.510 INFO:tasks.workunit.client.1.smithi188.stdout:8/881: mknod d5/d6e/dad/c115 0 2024-04-05T21:26:52.513 INFO:tasks.workunit.client.0.smithi121.stdout:2/833: mknod d2/d9/d1c/d10b/c110 0 2024-04-05T21:26:52.515 INFO:tasks.workunit.client.1.smithi188.stdout:8/882: rename d5/d21/d40/d43/d44/d5b/d99/d69/d70/dfe to d5/d21/d2a/d116 0 2024-04-05T21:26:52.517 INFO:tasks.workunit.client.0.smithi121.stdout:2/834: dread - d2/d9/d17/de2/f10d zero size 2024-04-05T21:26:52.518 INFO:tasks.workunit.client.0.smithi121.stdout:8/789: link d12/d50/d6a/d63/d8b/f9b d12/d50/d6a/d87/ff5 0 2024-04-05T21:26:52.518 INFO:tasks.workunit.client.1.smithi188.stdout:8/883: unlink d5/d21/d40/d43/d44/d5b/d99/d69/ff6 0 2024-04-05T21:26:52.528 INFO:tasks.workunit.client.0.smithi121.stdout:5/984: dwrite dd/de/d19/d38/d45/f84 [0,4194304] 0 2024-04-05T21:26:52.528 INFO:tasks.workunit.client.0.smithi121.stdout:6/772: dwrite d1/d2/daf/d6f/dc7/d1f/fd0 [0,4194304] 0 2024-04-05T21:26:52.534 INFO:tasks.workunit.client.0.smithi121.stdout:1/884: dwrite d8/d89/d9e/dc8/f11f [0,4194304] 0 2024-04-05T21:26:52.536 INFO:tasks.workunit.client.0.smithi121.stdout:2/835: dwrite d2/d9/f2f [0,4194304] 0 2024-04-05T21:26:52.537 INFO:tasks.workunit.client.0.smithi121.stdout:5/985: unlink dd/de/d39/d3b/l7d 0 2024-04-05T21:26:52.539 INFO:tasks.workunit.client.0.smithi121.stdout:5/986: write dd/de/d19/f57 [4545549,61254] 0 2024-04-05T21:26:52.543 INFO:tasks.workunit.client.0.smithi121.stdout:5/987: readlink dd/de/d19/d38/de3/l12a 0 2024-04-05T21:26:52.544 INFO:tasks.workunit.client.0.smithi121.stdout:5/988: write dd/de/d19/d1c/f48 [113990,34432] 0 2024-04-05T21:26:52.544 INFO:tasks.workunit.client.0.smithi121.stdout:5/989: chown dd/f17 163050153 1 2024-04-05T21:26:52.547 INFO:tasks.workunit.client.0.smithi121.stdout:5/990: symlink dd/de/d39/d3b/daf/l158 0 2024-04-05T21:26:52.549 INFO:tasks.workunit.client.0.smithi121.stdout:2/836: truncate d2/d9/d1c/d30/d59/fbe 4055831 0 2024-04-05T21:26:52.550 INFO:tasks.workunit.client.0.smithi121.stdout:2/837: dread - d2/d9/d38/d2d/d4c/de9/d5c/df0/f109 zero size 2024-04-05T21:26:52.551 INFO:tasks.workunit.client.0.smithi121.stdout:2/838: readlink d2/d9/d1c/d10b/lde 0 2024-04-05T21:26:52.558 INFO:tasks.workunit.client.0.smithi121.stdout:2/839: dwrite d2/fb0 [0,4194304] 0 2024-04-05T21:26:52.559 INFO:tasks.workunit.client.1.smithi188.stdout:8/884: dread d5/f1f [0,4194304] 0 2024-04-05T21:26:52.560 INFO:tasks.workunit.client.1.smithi188.stdout:8/885: stat d5/d21/d40/d43/d44/d59/d8b 0 2024-04-05T21:26:52.561 INFO:tasks.workunit.client.0.smithi121.stdout:2/840: chown d2/d9/d17/fdc 98 1 2024-04-05T21:26:52.563 INFO:tasks.workunit.client.0.smithi121.stdout:2/841: chown d2/d9/d38/d2d/le0 42772633 1 2024-04-05T21:26:52.564 INFO:tasks.workunit.client.1.smithi188.stdout:8/886: symlink d5/d21/d40/d43/d44/d5b/d99/l117 0 2024-04-05T21:26:52.565 INFO:tasks.workunit.client.0.smithi121.stdout:2/842: mknod d2/db6/dbb/c111 0 2024-04-05T21:26:52.565 INFO:tasks.workunit.client.1.smithi188.stdout:8/887: chown d5/d21/d40/f4f 6035 1 2024-04-05T21:26:52.565 INFO:tasks.workunit.client.0.smithi121.stdout:2/843: chown d2/ff2 348 1 2024-04-05T21:26:52.568 INFO:tasks.workunit.client.1.smithi188.stdout:8/888: unlink d5/d21/d40/d43/d44/d59/d79/c100 0 2024-04-05T21:26:52.569 INFO:tasks.workunit.client.1.smithi188.stdout:8/889: chown d5/d21/d40/c111 899 1 2024-04-05T21:26:52.573 INFO:tasks.workunit.client.0.smithi121.stdout:8/790: dread d12/d50/d6a/d63/d9c/fac [0,4194304] 0 2024-04-05T21:26:52.574 INFO:tasks.workunit.client.0.smithi121.stdout:2/844: dwrite d2/d28/f4f [0,4194304] 0 2024-04-05T21:26:52.576 INFO:tasks.workunit.client.0.smithi121.stdout:8/791: fdatasync d12/d50/d61/d7f/d9a/fd9 0 2024-04-05T21:26:52.577 INFO:tasks.workunit.client.0.smithi121.stdout:6/773: dwrite d1/d2/daf/d6f/dc7/f58 [0,4194304] 0 2024-04-05T21:26:52.578 INFO:tasks.workunit.client.0.smithi121.stdout:8/792: fdatasync d12/d13/fe2 0 2024-04-05T21:26:52.584 INFO:tasks.workunit.client.0.smithi121.stdout:8/793: unlink d12/d50/d6a/f45 0 2024-04-05T21:26:52.584 INFO:tasks.workunit.client.0.smithi121.stdout:6/774: creat d1/d2/daf/d6f/dc7/d1f/d6d/d101/f111 x:0 0 0 2024-04-05T21:26:52.585 INFO:tasks.workunit.client.0.smithi121.stdout:2/845: link d2/d9/d1c/d30/d59/ff4 d2/d1e/f112 0 2024-04-05T21:26:52.586 INFO:tasks.workunit.client.1.smithi188.stdout:8/890: dwrite d5/d21/d40/d43/faa [4194304,4194304] 0 2024-04-05T21:26:52.586 INFO:tasks.workunit.client.0.smithi121.stdout:2/846: stat d2/d9/d38/d2d/d4c/de9/d62/fc6 0 2024-04-05T21:26:52.587 INFO:tasks.workunit.client.0.smithi121.stdout:6/775: mknod d1/d2/d3/d4/d21/c112 0 2024-04-05T21:26:52.588 INFO:tasks.workunit.client.0.smithi121.stdout:8/794: truncate d12/d13/f84 2886618 0 2024-04-05T21:26:52.590 INFO:tasks.workunit.client.0.smithi121.stdout:8/795: mknod d12/d50/d61/cf6 0 2024-04-05T21:26:52.590 INFO:tasks.workunit.client.1.smithi188.stdout:8/891: creat d5/d21/d2a/d95/dc0/f118 x:0 0 0 2024-04-05T21:26:52.595 INFO:tasks.workunit.client.1.smithi188.stdout:8/892: dread d5/d21/d40/d43/d44/d5b/d99/d69/fe3 [0,4194304] 0 2024-04-05T21:26:52.596 INFO:tasks.workunit.client.0.smithi121.stdout:8/796: creat d12/d13/d15/de7/ff7 x:0 0 0 2024-04-05T21:26:52.596 INFO:tasks.workunit.client.0.smithi121.stdout:2/847: link d2/c43 d2/d9/d38/c113 0 2024-04-05T21:26:52.597 INFO:tasks.workunit.client.1.smithi188.stdout:8/893: mknod d5/d21/d2a/d47/c119 0 2024-04-05T21:26:52.598 INFO:tasks.workunit.client.0.smithi121.stdout:8/797: mknod d12/d13/d56/d57/cf8 0 2024-04-05T21:26:52.600 INFO:tasks.workunit.client.0.smithi121.stdout:2/848: link d2/d9/d38/d2d/d4c/de9/d62/fc6 d2/d9/dcb/f114 0 2024-04-05T21:26:52.600 INFO:tasks.workunit.client.1.smithi188.stdout:8/894: rename d5/d21/d40/d43/d44/d5b/dfa to d5/d6e/d11a 0 2024-04-05T21:26:52.601 INFO:tasks.workunit.client.0.smithi121.stdout:8/798: creat d12/d13/d56/d57/d5e/db9/ff9 x:0 0 0 2024-04-05T21:26:52.606 INFO:tasks.workunit.client.0.smithi121.stdout:6/776: write d1/d2/dc/d9a/dc0/f2d [1007809,84520] 0 2024-04-05T21:26:52.612 INFO:tasks.workunit.client.0.smithi121.stdout:2/849: truncate d2/d9/d1c/d30/fb1 1752230 0 2024-04-05T21:26:52.614 INFO:tasks.workunit.client.0.smithi121.stdout:6/777: dwrite d1/d2/d3/d4/d21/f93 [0,4194304] 0 2024-04-05T21:26:52.614 INFO:tasks.workunit.client.1.smithi188.stdout:8/895: dwrite d5/d21/d2a/f31 [0,4194304] 0 2024-04-05T21:26:52.615 INFO:tasks.workunit.client.1.smithi188.stdout:8/896: stat d5/d4b/fce 0 2024-04-05T21:26:52.615 INFO:tasks.workunit.client.0.smithi121.stdout:2/850: creat d2/d9/d17/d3a/f115 x:0 0 0 2024-04-05T21:26:52.616 INFO:tasks.workunit.client.1.smithi188.stdout:8/897: stat d5/d21/d2a/d71/lc7 0 2024-04-05T21:26:52.619 INFO:tasks.workunit.client.0.smithi121.stdout:6/778: unlink d1/d2/daf/d6f/dc7/d1f/d6d/fcf 0 2024-04-05T21:26:52.621 INFO:tasks.workunit.client.0.smithi121.stdout:2/851: dread d2/d9/d1c/d30/feb [0,4194304] 0 2024-04-05T21:26:52.621 INFO:tasks.workunit.client.0.smithi121.stdout:6/779: mknod d1/d2/dc/d9a/dc0/c113 0 2024-04-05T21:26:52.622 INFO:tasks.workunit.client.1.smithi188.stdout:8/898: symlink d5/d6e/d86/l11b 0 2024-04-05T21:26:52.623 INFO:tasks.workunit.client.0.smithi121.stdout:2/852: truncate d2/d9/d1c/d30/f61 4812034 0 2024-04-05T21:26:52.624 INFO:tasks.workunit.client.1.smithi188.stdout:8/899: mknod d5/d6e/d86/dcb/c11c 0 2024-04-05T21:26:52.625 INFO:tasks.workunit.client.0.smithi121.stdout:6/780: unlink d1/d2/d1b/d35/d39/d92/fd3 0 2024-04-05T21:26:52.636 INFO:tasks.workunit.client.0.smithi121.stdout:7/943: sync 2024-04-05T21:26:52.640 INFO:tasks.workunit.client.0.smithi121.stdout:7/944: mkdir d0/d9/d22/d28/d56/db9/ddf/def/d52/d61/dcb/d12b 0 2024-04-05T21:26:52.641 INFO:tasks.workunit.client.0.smithi121.stdout:7/945: write d0/d9/f12 [2563840,40703] 0 2024-04-05T21:26:52.646 INFO:tasks.workunit.client.1.smithi188.stdout:8/900: dread d5/d21/d40/d43/d44/d5b/d99/d69/fb3 [0,4194304] 0 2024-04-05T21:26:52.646 INFO:tasks.workunit.client.1.smithi188.stdout:8/901: stat d5/d6e/d86/dcb/c11c 0 2024-04-05T21:26:52.648 INFO:tasks.workunit.client.1.smithi188.stdout:8/902: creat d5/d21/d2a/d95/f11d x:0 0 0 2024-04-05T21:26:52.651 INFO:tasks.workunit.client.1.smithi188.stdout:8/903: rename d5/fe7 to d5/d21/d40/d43/d48/d7e/d10a/f11e 0 2024-04-05T21:26:52.652 INFO:tasks.workunit.client.1.smithi188.stdout:8/904: write d5/f2d [588711,104100] 0 2024-04-05T21:26:52.659 INFO:tasks.workunit.client.1.smithi188.stdout:8/905: getdents d5 0 2024-04-05T21:26:52.660 INFO:tasks.workunit.client.1.smithi188.stdout:8/906: mkdir d5/d21/d2a/d95/dc0/d11f 0 2024-04-05T21:26:52.675 INFO:tasks.workunit.client.1.smithi188.stdout:8/907: dread d5/d21/d2a/d71/f88 [0,4194304] 0 2024-04-05T21:26:52.682 INFO:tasks.workunit.client.1.smithi188.stdout:8/908: dwrite d5/d21/d2a/d95/f11d [0,4194304] 0 2024-04-05T21:26:52.683 INFO:tasks.workunit.client.1.smithi188.stdout:8/909: chown d5/d21/d2a/d47/c6d 5427573 1 2024-04-05T21:26:52.685 INFO:tasks.workunit.client.1.smithi188.stdout:8/910: mknod d5/d21/d40/d43/d44/d5b/d99/d69/d9b/c120 0 2024-04-05T21:26:52.703 INFO:tasks.workunit.client.1.smithi188.stdout:8/911: dwrite d5/d6e/dad/fb9 [0,4194304] 0 2024-04-05T21:26:52.704 INFO:tasks.workunit.client.1.smithi188.stdout:8/912: rmdir d5/dc/d8f 39 2024-04-05T21:26:52.708 INFO:tasks.workunit.client.1.smithi188.stdout:8/913: dread d5/d21/d40/d43/f81 [4194304,4194304] 0 2024-04-05T21:26:52.709 INFO:tasks.workunit.client.1.smithi188.stdout:8/914: chown d5/d21/d40/d43/f66 1674341 1 2024-04-05T21:26:52.710 INFO:tasks.workunit.client.1.smithi188.stdout:8/915: stat d5/d21/d80/f101 0 2024-04-05T21:26:52.711 INFO:tasks.workunit.client.1.smithi188.stdout:8/916: dread - d5/d21/d2a/d71/fbc zero size 2024-04-05T21:26:52.714 INFO:tasks.workunit.client.1.smithi188.stdout:8/917: mkdir d5/d21/d40/d43/d48/d7e/d121 0 2024-04-05T21:26:52.739 INFO:tasks.workunit.client.0.smithi121.stdout:6/781: write d1/d2/d1b/d35/d39/f65 [608200,124520] 0 2024-04-05T21:26:52.742 INFO:tasks.workunit.client.0.smithi121.stdout:6/782: chown d1/d4b/da7/lec 2749725 1 2024-04-05T21:26:52.744 INFO:tasks.workunit.client.0.smithi121.stdout:2/853: dwrite d2/d9/dcb/f114 [0,4194304] 0 2024-04-05T21:26:52.745 INFO:tasks.workunit.client.0.smithi121.stdout:7/946: dread d0/d9/f5e [0,4194304] 0 2024-04-05T21:26:52.750 INFO:tasks.workunit.client.0.smithi121.stdout:2/854: mkdir d2/d9/d38/d2d/d4c/d116 0 2024-04-05T21:26:52.755 INFO:tasks.workunit.client.0.smithi121.stdout:2/855: fsync d2/d9/d38/d2d/d47/f101 0 2024-04-05T21:26:52.755 INFO:tasks.workunit.client.0.smithi121.stdout:6/783: dwrite d1/d2/daf/d6f/dc7/d1f/fd0 [4194304,4194304] 0 2024-04-05T21:26:52.760 INFO:tasks.workunit.client.0.smithi121.stdout:6/784: mkdir d1/d2/d1b/d35/d74/d114 0 2024-04-05T21:26:52.762 INFO:tasks.workunit.client.0.smithi121.stdout:7/947: dwrite d0/d9/d22/d28/d56/db9/ddf/def/f43 [0,4194304] 0 2024-04-05T21:26:52.766 INFO:tasks.workunit.client.0.smithi121.stdout:7/948: read d0/d32/fa0 [4438299,28340] 0 2024-04-05T21:26:52.766 INFO:tasks.workunit.client.0.smithi121.stdout:7/949: stat d0/d9/d22/d28/d56 0 2024-04-05T21:26:52.767 INFO:tasks.workunit.client.0.smithi121.stdout:7/950: dread - d0/d9/d22/d3e/d76/f108 zero size 2024-04-05T21:26:52.767 INFO:tasks.workunit.client.0.smithi121.stdout:6/785: getdents d1/d2/d1b/d2e/dd9/de0 0 2024-04-05T21:26:52.768 INFO:tasks.workunit.client.0.smithi121.stdout:7/951: dread - d0/d57/f124 zero size 2024-04-05T21:26:52.770 INFO:tasks.workunit.client.0.smithi121.stdout:6/786: unlink d1/d2/d1b/d35/d39/f65 0 2024-04-05T21:26:52.771 INFO:tasks.workunit.client.0.smithi121.stdout:6/787: write d1/d4b/da7/f10c [504339,116135] 0 2024-04-05T21:26:52.772 INFO:tasks.workunit.client.0.smithi121.stdout:7/952: creat d0/d9/d22/d28/d56/db9/f12c x:0 0 0 2024-04-05T21:26:52.772 INFO:tasks.workunit.client.0.smithi121.stdout:7/953: read - d0/d9/f11f zero size 2024-04-05T21:26:52.775 INFO:tasks.workunit.client.0.smithi121.stdout:7/954: chown d0/d9/d22/d28/d56/db9/ddf/def/d52/d61 161980 1 2024-04-05T21:26:52.777 INFO:tasks.workunit.client.0.smithi121.stdout:6/788: dread d1/d2/daf/d6f/dc7/f66 [0,4194304] 0 2024-04-05T21:26:52.778 INFO:tasks.workunit.client.0.smithi121.stdout:6/789: write d1/d2/ff2 [881128,44908] 0 2024-04-05T21:26:52.782 INFO:tasks.workunit.client.0.smithi121.stdout:6/790: link d1/d2/d3/d4/d21/d57/d42/la5 d1/d2/d3/d59/l115 0 2024-04-05T21:26:52.784 INFO:tasks.workunit.client.0.smithi121.stdout:6/791: creat d1/d2/d1b/d35/f116 x:0 0 0 2024-04-05T21:26:52.785 INFO:tasks.workunit.client.0.smithi121.stdout:2/856: dread d2/d9/d1c/d30/d59/fbe [0,4194304] 0 2024-04-05T21:26:52.787 INFO:tasks.workunit.client.0.smithi121.stdout:6/792: fdatasync d1/d2/dc/f19 0 2024-04-05T21:26:52.787 INFO:tasks.workunit.client.0.smithi121.stdout:2/857: read d2/d9/ff [1599938,68680] 0 2024-04-05T21:26:52.788 INFO:tasks.workunit.client.0.smithi121.stdout:6/793: stat d1/d2/d1b/d5e/l5f 0 2024-04-05T21:26:52.789 INFO:tasks.workunit.client.0.smithi121.stdout:7/955: write d0/d23/ffa [504219,91059] 0 2024-04-05T21:26:52.791 INFO:tasks.workunit.client.0.smithi121.stdout:2/858: mkdir d2/d9/d38/d2d/d47/d117 0 2024-04-05T21:26:52.793 INFO:tasks.workunit.client.0.smithi121.stdout:6/794: rename d1/d2/d1b/d2e/dd9/cce to d1/d4b/dcc/c117 0 2024-04-05T21:26:52.795 INFO:tasks.workunit.client.0.smithi121.stdout:7/956: unlink d0/d23/fe8 0 2024-04-05T21:26:52.797 INFO:tasks.workunit.client.0.smithi121.stdout:2/859: symlink d2/d9/d1c/d85/def/l118 0 2024-04-05T21:26:52.798 INFO:tasks.workunit.client.0.smithi121.stdout:6/795: creat d1/d2/f118 x:0 0 0 2024-04-05T21:26:52.800 INFO:tasks.workunit.client.0.smithi121.stdout:2/860: mknod d2/d9/d38/d2d/d4c/d79/c119 0 2024-04-05T21:26:52.800 INFO:tasks.workunit.client.0.smithi121.stdout:0/919: sync 2024-04-05T21:26:52.802 INFO:tasks.workunit.client.0.smithi121.stdout:7/957: dwrite d0/d9/d22/fa3 [0,4194304] 0 2024-04-05T21:26:52.807 INFO:tasks.workunit.client.0.smithi121.stdout:0/920: rename d7/d62/f11 to d7/d71/d45/d5a/f141 0 2024-04-05T21:26:52.812 INFO:tasks.workunit.client.0.smithi121.stdout:7/958: rename d0/d9/d22/d28/d56/db9/ddf/def/d52/c91 to d0/d9/d22/d28/d56/db9/ddf/def/d52/d61/dcb/c12d 0 2024-04-05T21:26:52.813 INFO:tasks.workunit.client.0.smithi121.stdout:0/921: symlink d7/dc/d95/l142 0 2024-04-05T21:26:52.815 INFO:tasks.workunit.client.0.smithi121.stdout:7/959: truncate d0/d9/d22/d28/d56/dff/f126 210252 0 2024-04-05T21:26:52.816 INFO:tasks.workunit.client.0.smithi121.stdout:2/861: write d2/d9/d38/d2d/d4c/d79/f9c [448068,81936] 0 2024-04-05T21:26:52.818 INFO:tasks.workunit.client.0.smithi121.stdout:2/862: fdatasync d2/f11 0 2024-04-05T21:26:52.818 INFO:tasks.workunit.client.0.smithi121.stdout:6/796: dread d1/d2/daf/d6f/dc7/fac [0,4194304] 0 2024-04-05T21:26:52.819 INFO:tasks.workunit.client.0.smithi121.stdout:0/922: mkdir d7/d71/d45/d46/d53/d143 0 2024-04-05T21:26:52.820 INFO:tasks.workunit.client.0.smithi121.stdout:2/863: stat d2/d9/d38/d2d/d4c/d7c/dc2 0 2024-04-05T21:26:52.822 INFO:tasks.workunit.client.0.smithi121.stdout:6/797: mkdir d1/d2/dc/d9a/d119 0 2024-04-05T21:26:52.825 INFO:tasks.workunit.client.0.smithi121.stdout:2/864: symlink d2/d9/d1c/d30/l11a 0 2024-04-05T21:26:52.826 INFO:tasks.workunit.client.0.smithi121.stdout:2/865: dread - d2/d9/dcb/ffd zero size 2024-04-05T21:26:52.828 INFO:tasks.workunit.client.0.smithi121.stdout:2/866: dread - d2/d9/d1c/d30/d59/f89 zero size 2024-04-05T21:26:52.828 INFO:tasks.workunit.client.0.smithi121.stdout:6/798: mkdir d1/d2/d3/d4/df0/d11a 0 2024-04-05T21:26:52.832 INFO:tasks.workunit.client.0.smithi121.stdout:6/799: symlink d1/d4b/l11b 0 2024-04-05T21:26:52.836 INFO:tasks.workunit.client.0.smithi121.stdout:7/960: dwrite d0/d9/d22/d28/d56/db9/ddf/def/f80 [0,4194304] 0 2024-04-05T21:26:52.836 INFO:tasks.workunit.client.0.smithi121.stdout:0/923: dwrite d7/d1c/f68 [0,4194304] 0 2024-04-05T21:26:52.839 INFO:tasks.workunit.client.0.smithi121.stdout:6/800: rename d1/d2/d1b/fd2 to d1/d2/daf/d6f/f11c 0 2024-04-05T21:26:52.841 INFO:tasks.workunit.client.0.smithi121.stdout:6/801: stat d1/d2/d1b/d35/d74/f7d 0 2024-04-05T21:26:52.842 INFO:tasks.workunit.client.0.smithi121.stdout:7/961: rmdir d0/d23/d5a 39 2024-04-05T21:26:52.848 INFO:tasks.workunit.client.0.smithi121.stdout:2/867: dwrite d2/d9/d17/f42 [0,4194304] 0 2024-04-05T21:26:52.849 INFO:tasks.workunit.client.0.smithi121.stdout:2/868: write d2/d9/d1c/d10b/f106 [754277,22564] 0 2024-04-05T21:26:52.903 INFO:tasks.workunit.client.0.smithi121.stdout:7/962: creat d0/d32/d7c/f12e x:0 0 0 2024-04-05T21:26:52.904 INFO:tasks.workunit.client.0.smithi121.stdout:6/802: rename d1/d2/dc/d9a/fb6 to d1/d2/d1b/d35/d74/df3/f11d 0 2024-04-05T21:26:52.908 INFO:tasks.workunit.client.0.smithi121.stdout:7/963: getdents d0/d9/d22/d81/d118 0 2024-04-05T21:26:52.910 INFO:tasks.workunit.client.0.smithi121.stdout:0/924: dwrite d7/d71/d45/d46/d106/d11e/f135 [0,4194304] 0 2024-04-05T21:26:52.912 INFO:tasks.workunit.client.0.smithi121.stdout:2/869: truncate d2/d9/f2f 1461735 0 2024-04-05T21:26:52.913 INFO:tasks.workunit.client.0.smithi121.stdout:2/870: write d2/d9/d17/de2/f10d [659158,35511] 0 2024-04-05T21:26:52.915 INFO:tasks.workunit.client.0.smithi121.stdout:6/803: truncate d1/f1c 1526080 0 2024-04-05T21:26:52.915 INFO:tasks.workunit.client.0.smithi121.stdout:7/964: rmdir d0/d23/d5a/df9 39 2024-04-05T21:26:52.918 INFO:tasks.workunit.client.0.smithi121.stdout:0/925: rename d7/d71/d29/f9b to d7/d71/f144 0 2024-04-05T21:26:52.920 INFO:tasks.workunit.client.0.smithi121.stdout:7/965: symlink d0/d9/d22/d28/d56/dff/l12f 0 2024-04-05T21:26:52.921 INFO:tasks.workunit.client.0.smithi121.stdout:2/871: rename d2/d9/d38/d2d/d4c/de9/d5c/c53 to d2/d9/d38/d2d/d4c/de9/d62/c11b 0 2024-04-05T21:26:52.922 INFO:tasks.workunit.client.0.smithi121.stdout:6/804: symlink d1/d2/d1b/d35/d74/d114/l11e 0 2024-04-05T21:26:52.922 INFO:tasks.workunit.client.0.smithi121.stdout:2/872: fdatasync d2/f11 0 2024-04-05T21:26:52.923 INFO:tasks.workunit.client.0.smithi121.stdout:2/873: readlink d2/l6 0 2024-04-05T21:26:52.923 INFO:tasks.workunit.client.0.smithi121.stdout:6/805: stat d1/d2/daf/d6f/f107 0 2024-04-05T21:26:52.925 INFO:tasks.workunit.client.0.smithi121.stdout:2/874: fdatasync d2/d1e/f24 0 2024-04-05T21:26:52.925 INFO:tasks.workunit.client.0.smithi121.stdout:2/875: readlink d2/d9/d38/d2d/le0 0 2024-04-05T21:26:52.926 INFO:tasks.workunit.client.0.smithi121.stdout:6/806: mknod d1/d2/d1b/d2e/c11f 0 2024-04-05T21:26:52.926 INFO:tasks.workunit.client.0.smithi121.stdout:7/966: link d0/l7 d0/d9/d22/d28/d56/db9/ddf/def/d52/d61/l130 0 2024-04-05T21:26:52.926 INFO:tasks.workunit.client.0.smithi121.stdout:2/876: chown d2/d9/d17/d3a 30 1 2024-04-05T21:26:52.929 INFO:tasks.workunit.client.0.smithi121.stdout:7/967: mkdir d0/d9/d22/d28/d56/dff/d131 0 2024-04-05T21:26:52.931 INFO:tasks.workunit.client.0.smithi121.stdout:2/877: rename d2/d9/d38/c48 to d2/d9/d38/d2d/d4c/d7c/d7e/db9/c11c 0 2024-04-05T21:26:52.934 INFO:tasks.workunit.client.0.smithi121.stdout:2/878: mkdir d2/d28/d11d 0 2024-04-05T21:26:52.936 INFO:tasks.workunit.client.0.smithi121.stdout:2/879: stat d2/d9/d38/d2d/fa5 0 2024-04-05T21:26:52.937 INFO:tasks.workunit.client.0.smithi121.stdout:7/968: dwrite d0/d57/ded/f116 [0,4194304] 0 2024-04-05T21:26:52.941 INFO:tasks.workunit.client.0.smithi121.stdout:0/926: dread d7/dc/d95/fd3 [0,4194304] 0 2024-04-05T21:26:52.943 INFO:tasks.workunit.client.0.smithi121.stdout:2/880: rename d2/d9/d17/d3a/f9a to d2/d9/d1c/d30/d59/f11e 0 2024-04-05T21:26:52.944 INFO:tasks.workunit.client.0.smithi121.stdout:0/927: truncate d7/d71/d29/d3f/fba 828074 0 2024-04-05T21:26:52.945 INFO:tasks.workunit.client.0.smithi121.stdout:7/969: creat d0/d9/d22/d81/d118/f132 x:0 0 0 2024-04-05T21:26:52.947 INFO:tasks.workunit.client.0.smithi121.stdout:6/807: dread d1/d2/f55 [0,4194304] 0 2024-04-05T21:26:52.947 INFO:tasks.workunit.client.0.smithi121.stdout:7/970: chown d0/d9/d22/de9/l10a 10088170 1 2024-04-05T21:26:52.948 INFO:tasks.workunit.client.0.smithi121.stdout:6/808: chown d1/d2/ceb 62163 1 2024-04-05T21:26:52.949 INFO:tasks.workunit.client.0.smithi121.stdout:2/881: mkdir d2/d9/d38/d2d/d4c/d7c/dbc/d11f 0 2024-04-05T21:26:52.951 INFO:tasks.workunit.client.0.smithi121.stdout:7/971: write d0/d9/d22/d28/d56/dff/f126 [738999,88221] 0 2024-04-05T21:26:52.952 INFO:tasks.workunit.client.0.smithi121.stdout:2/882: stat d2/d9/d38/d2d/d4c/de9/d62/fd0 0 2024-04-05T21:26:52.955 INFO:tasks.workunit.client.0.smithi121.stdout:0/928: dwrite d7/d62/d3c/d6b/f111 [0,4194304] 0 2024-04-05T21:26:52.955 INFO:tasks.workunit.client.0.smithi121.stdout:3/779: sync 2024-04-05T21:26:52.960 INFO:tasks.workunit.client.0.smithi121.stdout:0/929: unlink d7/dc/c86 0 2024-04-05T21:26:52.962 INFO:tasks.workunit.client.0.smithi121.stdout:3/780: link d0/d3/d24/d6d/dff/f100 d0/d3/d24/d6d/d7b/da3/dec/df6/d102/f10b 0 2024-04-05T21:26:52.962 INFO:tasks.workunit.client.0.smithi121.stdout:0/930: symlink d7/d62/d8b/l145 0 2024-04-05T21:26:52.964 INFO:tasks.workunit.client.0.smithi121.stdout:3/781: chown d0/d3/d24/d3d/d1a/d1e/f71 3417 1 2024-04-05T21:26:52.966 INFO:tasks.workunit.client.0.smithi121.stdout:0/931: truncate d7/dc/da3/df5/ff9 384439 0 2024-04-05T21:26:52.966 INFO:tasks.workunit.client.0.smithi121.stdout:2/883: rename d2/d9/d38/d2d/d4c/de9/ca1 to d2/d9/d1c/c120 0 2024-04-05T21:26:52.967 INFO:tasks.workunit.client.0.smithi121.stdout:3/782: fdatasync d0/d3/de/f72 0 2024-04-05T21:26:52.969 INFO:tasks.workunit.client.0.smithi121.stdout:0/932: unlink d7/d62/d3c/d6b/l134 0 2024-04-05T21:26:52.970 INFO:tasks.workunit.client.0.smithi121.stdout:6/809: dread d1/d2/dc/f5b [0,4194304] 0 2024-04-05T21:26:52.972 INFO:tasks.workunit.client.0.smithi121.stdout:7/972: rename d0/d9/d22/d28/d56/dff/f126 to d0/d9/d22/d28/d56/db9/f133 0 2024-04-05T21:26:52.972 INFO:tasks.workunit.client.0.smithi121.stdout:2/884: write d2/d9/d38/d2d/d47/d94/d84/fd7 [948416,84798] 0 2024-04-05T21:26:52.973 INFO:tasks.workunit.client.0.smithi121.stdout:6/810: fsync d1/d2/daf/d6f/dc7/d1f/fc4 0 2024-04-05T21:26:52.973 INFO:tasks.workunit.client.0.smithi121.stdout:3/783: chown d0/d3/d24/d6d/d7b/da4/cfe 3 1 2024-04-05T21:26:52.978 INFO:tasks.workunit.client.0.smithi121.stdout:3/784: symlink d0/d3/de/l10c 0 2024-04-05T21:26:52.980 INFO:tasks.workunit.client.0.smithi121.stdout:6/811: creat d1/d2/f120 x:0 0 0 2024-04-05T21:26:52.982 INFO:tasks.workunit.client.0.smithi121.stdout:6/812: write d1/d2/d1b/d35/f47 [867937,95372] 0 2024-04-05T21:26:52.982 INFO:tasks.workunit.client.0.smithi121.stdout:0/933: link d7/d71/d94/cb4 d7/d1c/c146 0 2024-04-05T21:26:52.984 INFO:tasks.workunit.client.0.smithi121.stdout:3/785: mknod d0/d3/d24/d6d/d7b/da3/dec/df6/d102/c10d 0 2024-04-05T21:26:52.985 INFO:tasks.workunit.client.0.smithi121.stdout:3/786: read d0/d3/d24/d3d/d1a/d23/d38/f65 [570768,124881] 0 2024-04-05T21:26:52.992 INFO:tasks.workunit.client.0.smithi121.stdout:0/934: creat d7/d1c/f147 x:0 0 0 2024-04-05T21:26:52.995 INFO:tasks.workunit.client.0.smithi121.stdout:6/813: getdents d1/d2/d1b/d5e 0 2024-04-05T21:26:52.997 INFO:tasks.workunit.client.0.smithi121.stdout:6/814: dread - d1/d2/d3/d4/d21/d57/d42/d48/d7c/fdd zero size 2024-04-05T21:26:52.999 INFO:tasks.workunit.client.0.smithi121.stdout:2/885: dwrite d2/d9/d38/d2d/d47/d94/f37 [0,4194304] 0 2024-04-05T21:26:53.000 INFO:tasks.workunit.client.0.smithi121.stdout:2/886: fsync d2/d9/d17/f42 0 2024-04-05T21:26:53.001 INFO:tasks.workunit.client.0.smithi121.stdout:3/787: truncate d0/d3/d24/d6d/d7b/fc7 2176126 0 2024-04-05T21:26:53.003 INFO:tasks.workunit.client.0.smithi121.stdout:2/887: write d2/d9/d17/de2/f10d [181813,35618] 0 2024-04-05T21:26:53.005 INFO:tasks.workunit.client.0.smithi121.stdout:3/788: mkdir d0/d3/d24/d6d/dff/d10e 0 2024-04-05T21:26:53.006 INFO:tasks.workunit.client.0.smithi121.stdout:6/815: write d1/d2/d1b/d5e/dbd/d109/fae [578804,14674] 0 2024-04-05T21:26:53.006 INFO:tasks.workunit.client.1.smithi188.stdout:8/918: sync 2024-04-05T21:26:53.006 INFO:tasks.workunit.client.1.smithi188.stdout:8/919: chown d5/f34 4647 1 2024-04-05T21:26:53.007 INFO:tasks.workunit.client.0.smithi121.stdout:2/888: mkdir d2/d9/d38/d2d/d4c/de9/d5c/df0/d121 0 2024-04-05T21:26:53.007 INFO:tasks.workunit.client.0.smithi121.stdout:0/935: write d7/f13 [5211517,96250] 0 2024-04-05T21:26:53.010 INFO:tasks.workunit.client.0.smithi121.stdout:6/816: mknod d1/d2/d3/d4/d21/d57/d42/c121 0 2024-04-05T21:26:53.010 INFO:tasks.workunit.client.1.smithi188.stdout:8/920: mknod d5/d21/d40/d43/d48/d7e/d121/c122 0 2024-04-05T21:26:53.011 INFO:tasks.workunit.client.0.smithi121.stdout:0/936: mkdir d7/d71/d29/de8/d148 0 2024-04-05T21:26:53.011 INFO:tasks.workunit.client.0.smithi121.stdout:6/817: readlink d1/d2/d3/d59/l69 0 2024-04-05T21:26:53.011 INFO:tasks.workunit.client.1.smithi188.stdout:8/921: chown d5/d21/l53 39 1 2024-04-05T21:26:53.013 INFO:tasks.workunit.client.0.smithi121.stdout:2/889: link d2/d9/d1c/d30/dba/fb5 d2/d9/d38/d2d/d4c/de5/f122 0 2024-04-05T21:26:53.014 INFO:tasks.workunit.client.0.smithi121.stdout:0/937: creat d7/dc/da3/df5/f149 x:0 0 0 2024-04-05T21:26:53.014 INFO:tasks.workunit.client.0.smithi121.stdout:6/818: mknod d1/d2/d1b/d91/c122 0 2024-04-05T21:26:53.015 INFO:tasks.workunit.client.0.smithi121.stdout:2/890: mknod d2/db6/dbb/c123 0 2024-04-05T21:26:53.016 INFO:tasks.workunit.client.0.smithi121.stdout:6/819: chown d1/d2/f118 2 1 2024-04-05T21:26:53.017 INFO:tasks.workunit.client.0.smithi121.stdout:0/938: unlink d7/d71/d29/d3f/lfc 0 2024-04-05T21:26:53.017 INFO:tasks.workunit.client.0.smithi121.stdout:6/820: fsync d1/d2/d1b/d35/d51/d105/f106 0 2024-04-05T21:26:53.020 INFO:tasks.workunit.client.0.smithi121.stdout:6/821: write d1/d2/daf/d6f/dc7/fd [9045929,52734] 0 2024-04-05T21:26:53.023 INFO:tasks.workunit.client.0.smithi121.stdout:0/939: dread d7/d62/d3c/d6b/f11c [0,4194304] 0 2024-04-05T21:26:53.025 INFO:tasks.workunit.client.0.smithi121.stdout:6/822: symlink d1/d2/dc/d9a/l123 0 2024-04-05T21:26:53.025 INFO:tasks.workunit.client.0.smithi121.stdout:0/940: write d7/dc/f32 [2195637,44895] 0 2024-04-05T21:26:53.027 INFO:tasks.workunit.client.0.smithi121.stdout:0/941: rmdir d7/d7e 39 2024-04-05T21:26:53.029 INFO:tasks.workunit.client.1.smithi188.stdout:8/922: dread d5/d21/d2a/d47/fc5 [0,4194304] 0 2024-04-05T21:26:53.030 INFO:tasks.workunit.client.0.smithi121.stdout:6/823: getdents d1/d2/d1b/d35/d51 0 2024-04-05T21:26:53.031 INFO:tasks.workunit.client.0.smithi121.stdout:3/789: dread d0/d3/d24/d6d/d7b/f85 [0,4194304] 0 2024-04-05T21:26:53.031 INFO:tasks.workunit.client.1.smithi188.stdout:8/923: dread - d5/d21/d40/d43/ffc zero size 2024-04-05T21:26:53.035 INFO:tasks.workunit.client.0.smithi121.stdout:3/790: symlink d0/dc6/l10f 0 2024-04-05T21:26:53.036 INFO:tasks.workunit.client.0.smithi121.stdout:0/942: dread d7/d71/d94/fd7 [0,4194304] 0 2024-04-05T21:26:53.039 INFO:tasks.workunit.client.0.smithi121.stdout:3/791: creat d0/d3/de/da6/de2/f110 x:0 0 0 2024-04-05T21:26:53.043 INFO:tasks.workunit.client.0.smithi121.stdout:3/792: write d0/d3/d24/d6d/d7b/da4/ffc [725746,114667] 0 2024-04-05T21:26:53.044 INFO:tasks.workunit.client.0.smithi121.stdout:0/943: truncate d7/d62/fa8 514888 0 2024-04-05T21:26:53.046 INFO:tasks.workunit.client.1.smithi188.stdout:8/924: dwrite d5/d6e/fb7 [0,4194304] 0 2024-04-05T21:26:53.047 INFO:tasks.workunit.client.0.smithi121.stdout:2/891: dwrite d2/d1e/f52 [4194304,4194304] 0 2024-04-05T21:26:53.051 INFO:tasks.workunit.client.0.smithi121.stdout:6/824: dread d1/d2/d1b/d91/f9f [0,4194304] 0 2024-04-05T21:26:53.053 INFO:tasks.workunit.client.0.smithi121.stdout:2/892: symlink d2/d9/d38/d2d/d47/l124 0 2024-04-05T21:26:53.054 INFO:tasks.workunit.client.1.smithi188.stdout:8/925: dwrite d5/d21/d40/d43/d44/d59/d8b/feb [4194304,4194304] 0 2024-04-05T21:26:53.056 INFO:tasks.workunit.client.0.smithi121.stdout:3/793: dwrite d0/d3/d24/f3c [4194304,4194304] 0 2024-04-05T21:26:53.057 INFO:tasks.workunit.client.0.smithi121.stdout:0/944: read d7/d62/fa8 [414885,15099] 0 2024-04-05T21:26:53.058 INFO:tasks.workunit.client.1.smithi188.stdout:8/926: fsync d5/d21/d2a/d71/fed 0 2024-04-05T21:26:53.060 INFO:tasks.workunit.client.1.smithi188.stdout:8/927: chown d5/d21/d40/d43/d44/d59/d8b/fb6 133486 1 2024-04-05T21:26:53.061 INFO:tasks.workunit.client.0.smithi121.stdout:6/825: creat d1/d2/d3/d4/d21/d57/f124 x:0 0 0 2024-04-05T21:26:53.064 INFO:tasks.workunit.client.0.smithi121.stdout:0/945: symlink d7/d71/d45/d46/d53/d8e/dc4/l14a 0 2024-04-05T21:26:53.065 INFO:tasks.workunit.client.0.smithi121.stdout:6/826: dread - d1/d2/d1b/d35/d39/d92/fd8 zero size 2024-04-05T21:26:53.066 INFO:tasks.workunit.client.0.smithi121.stdout:2/893: dwrite d2/db6/fd1 [0,4194304] 0 2024-04-05T21:26:53.066 INFO:tasks.workunit.client.1.smithi188.stdout:8/928: dread d5/d21/f64 [0,4194304] 0 2024-04-05T21:26:53.068 INFO:tasks.workunit.client.0.smithi121.stdout:2/894: chown d2/d9/d77/fd2 18 1 2024-04-05T21:26:53.069 INFO:tasks.workunit.client.1.smithi188.stdout:8/929: rmdir d5/d21/d40/d43/d44/d5b/d99/d69 39 2024-04-05T21:26:53.070 INFO:tasks.workunit.client.1.smithi188.stdout:8/930: read d5/f3d [1495420,7298] 0 2024-04-05T21:26:53.070 INFO:tasks.workunit.client.0.smithi121.stdout:2/895: write d2/d9/d38/d2d/d4c/de9/d5c/f10e [956960,87568] 0 2024-04-05T21:26:53.070 INFO:tasks.workunit.client.0.smithi121.stdout:3/794: rename d0/d3/d24/d3d/d1a/d1e/f71 to d0/d3/d24/db3/f111 0 2024-04-05T21:26:53.071 INFO:tasks.workunit.client.0.smithi121.stdout:6/827: write d1/d2/daf/d6f/dc7/fac [3603545,130557] 0 2024-04-05T21:26:53.072 INFO:tasks.workunit.client.1.smithi188.stdout:8/931: dread - d5/d21/d40/d43/d44/d59/d8b/ff3 zero size 2024-04-05T21:26:53.072 INFO:tasks.workunit.client.1.smithi188.stdout:8/932: chown d5/f1f 1973 1 2024-04-05T21:26:53.072 INFO:tasks.workunit.client.0.smithi121.stdout:6/828: stat d1/d2/d1b/d2e/l7a 0 2024-04-05T21:26:53.073 INFO:tasks.workunit.client.0.smithi121.stdout:0/946: rename d7/d71/d29/l47 to d7/d62/d8b/l14b 0 2024-04-05T21:26:53.076 INFO:tasks.workunit.client.0.smithi121.stdout:0/947: read d7/d62/d9d/fd6 [3061491,2991] 0 2024-04-05T21:26:53.076 INFO:tasks.workunit.client.0.smithi121.stdout:2/896: dread d2/d9/d38/d2d/d4c/de5/ff6 [0,4194304] 0 2024-04-05T21:26:53.077 INFO:tasks.workunit.client.0.smithi121.stdout:3/795: link d0/d3/de/da6/da1/f104 d0/d3/d24/d3d/d1a/d23/d90/f112 0 2024-04-05T21:26:53.078 INFO:tasks.workunit.client.1.smithi188.stdout:8/933: link d5/dc/f3b d5/d21/f123 0 2024-04-05T21:26:53.078 INFO:tasks.workunit.client.0.smithi121.stdout:0/948: stat d7/d71/d45/d46/d53/f138 0 2024-04-05T21:26:53.079 INFO:tasks.workunit.client.1.smithi188.stdout:8/934: chown d5/laf 15421 1 2024-04-05T21:26:53.080 INFO:tasks.workunit.client.0.smithi121.stdout:6/829: link d1/d2/daf/d6f/dc7/d1f/f64 d1/d2/d1b/d91/f125 0 2024-04-05T21:26:53.081 INFO:tasks.workunit.client.1.smithi188.stdout:8/935: rmdir d5/d21/d40/d43/d48 39 2024-04-05T21:26:53.082 INFO:tasks.workunit.client.0.smithi121.stdout:6/830: readlink d1/d2/d1b/d35/d74/d114/l11e 0 2024-04-05T21:26:53.082 INFO:tasks.workunit.client.0.smithi121.stdout:3/796: dread d0/d3/d24/d6d/fcc [0,4194304] 0 2024-04-05T21:26:53.083 INFO:tasks.workunit.client.1.smithi188.stdout:8/936: creat d5/dc/d8f/f124 x:0 0 0 2024-04-05T21:26:53.084 INFO:tasks.workunit.client.0.smithi121.stdout:6/831: mkdir d1/d2/daf/d6f/d126 0 2024-04-05T21:26:53.086 INFO:tasks.workunit.client.1.smithi188.stdout:8/937: truncate d5/d21/d40/d43/d44/d59/f5a 1416974 0 2024-04-05T21:26:53.088 INFO:tasks.workunit.client.0.smithi121.stdout:6/832: getdents d1/d2/d3 0 2024-04-05T21:26:53.090 INFO:tasks.workunit.client.0.smithi121.stdout:3/797: dwrite d0/f97 [0,4194304] 0 2024-04-05T21:26:53.090 INFO:tasks.workunit.client.0.smithi121.stdout:6/833: mknod d1/d4b/c127 0 2024-04-05T21:26:53.091 INFO:tasks.workunit.client.0.smithi121.stdout:6/834: dread - d1/d2/daf/d6f/fd7 zero size 2024-04-05T21:26:53.095 INFO:tasks.workunit.client.0.smithi121.stdout:3/798: unlink d0/d3/de/da6/fb2 0 2024-04-05T21:26:53.100 INFO:tasks.workunit.client.0.smithi121.stdout:6/835: dwrite d1/d2/daf/d6f/dc7/d1f/fd0 [0,4194304] 0 2024-04-05T21:26:53.100 INFO:tasks.workunit.client.0.smithi121.stdout:0/949: write d7/d71/d45/d46/d53/d10e/f130 [2763494,110321] 0 2024-04-05T21:26:53.104 INFO:tasks.workunit.client.0.smithi121.stdout:0/950: read - d7/dc/da3/df5/f149 zero size 2024-04-05T21:26:53.104 INFO:tasks.workunit.client.0.smithi121.stdout:6/836: rmdir d1/d2/d1b/d35/d74 39 2024-04-05T21:26:53.105 INFO:tasks.workunit.client.0.smithi121.stdout:6/837: dread d1/d4b/f77 [0,4194304] 0 2024-04-05T21:26:53.105 INFO:tasks.workunit.client.0.smithi121.stdout:3/799: link d0/d3/d24/db3/cbc d0/d3/d24/d3d/d1a/c113 0 2024-04-05T21:26:53.107 INFO:tasks.workunit.client.0.smithi121.stdout:3/800: creat d0/f114 x:0 0 0 2024-04-05T21:26:53.108 INFO:tasks.workunit.client.0.smithi121.stdout:3/801: chown d0/d3/de/da6/de2 15 1 2024-04-05T21:26:53.110 INFO:tasks.workunit.client.0.smithi121.stdout:6/838: truncate d1/d4b/f77 22817 0 2024-04-05T21:26:53.112 INFO:tasks.workunit.client.0.smithi121.stdout:3/802: rmdir d0/d3/d24/d3d/d1a/d23/d38 39 2024-04-05T21:26:53.115 INFO:tasks.workunit.client.0.smithi121.stdout:3/803: symlink d0/d3/d24/d3d/d1a/d1e/db1/ddd/de1/l115 0 2024-04-05T21:26:53.117 INFO:tasks.workunit.client.0.smithi121.stdout:6/839: mknod d1/d2/d1b/d35/d74/c128 0 2024-04-05T21:26:53.124 INFO:tasks.workunit.client.0.smithi121.stdout:6/840: dwrite d1/d2/dc/d9a/dc0/fee [0,4194304] 0 2024-04-05T21:26:53.128 INFO:tasks.workunit.client.0.smithi121.stdout:6/841: fsync d1/d2/daf/d6f/f107 0 2024-04-05T21:26:53.131 INFO:tasks.workunit.client.0.smithi121.stdout:2/897: dread d2/d9/d38/d2d/d4c/d7c/f8b [0,4194304] 0 2024-04-05T21:26:53.131 INFO:tasks.workunit.client.0.smithi121.stdout:6/842: mknod d1/d2/d3/d4/c129 0 2024-04-05T21:26:53.133 INFO:tasks.workunit.client.0.smithi121.stdout:9/676: sync 2024-04-05T21:26:53.133 INFO:tasks.workunit.client.0.smithi121.stdout:1/885: sync 2024-04-05T21:26:53.133 INFO:tasks.workunit.client.0.smithi121.stdout:3/804: dread d0/d3/d24/d3d/d1a/d23/f56 [0,4194304] 0 2024-04-05T21:26:53.135 INFO:tasks.workunit.client.0.smithi121.stdout:6/843: creat d1/d2/daf/d6f/dc7/d1f/d6d/f12a x:0 0 0 2024-04-05T21:26:53.136 INFO:tasks.workunit.client.0.smithi121.stdout:9/677: write d6/d32/dbe/d7b/f73 [2572473,40608] 0 2024-04-05T21:26:53.137 INFO:tasks.workunit.client.0.smithi121.stdout:1/886: dread - d8/d8c/db6/d1a/d1b/d4c/da1/fda zero size 2024-04-05T21:26:53.138 INFO:tasks.workunit.client.0.smithi121.stdout:6/844: readlink d1/d2/d1b/d35/d51/l5d 0 2024-04-05T21:26:53.138 INFO:tasks.workunit.client.0.smithi121.stdout:1/887: stat d8/d89/d9e/d2d/fc1 0 2024-04-05T21:26:53.138 INFO:tasks.workunit.client.0.smithi121.stdout:2/898: write d2/d9/ff [3483520,96520] 0 2024-04-05T21:26:53.140 INFO:tasks.workunit.client.0.smithi121.stdout:2/899: chown d2/d9/c13 22432 1 2024-04-05T21:26:53.140 INFO:tasks.workunit.client.0.smithi121.stdout:1/888: chown d8/d89/d9e/dc8/le8 398 1 2024-04-05T21:26:53.141 INFO:tasks.workunit.client.0.smithi121.stdout:2/900: chown d2/d9 7 1 2024-04-05T21:26:53.143 INFO:tasks.workunit.client.0.smithi121.stdout:3/805: write d0/d3/d24/d3d/f57 [2357941,25212] 0 2024-04-05T21:26:53.145 INFO:tasks.workunit.client.0.smithi121.stdout:5/991: sync 2024-04-05T21:26:53.146 INFO:tasks.workunit.client.0.smithi121.stdout:9/678: write d6/fc [3269155,24629] 0 2024-04-05T21:26:53.146 INFO:tasks.workunit.client.0.smithi121.stdout:1/889: dread d8/d8c/db6/d1a/f72 [0,4194304] 0 2024-04-05T21:26:53.148 INFO:tasks.workunit.client.0.smithi121.stdout:5/992: write dd/de/f34 [4802846,72025] 0 2024-04-05T21:26:53.150 INFO:tasks.workunit.client.0.smithi121.stdout:2/901: write d2/d9/d1c/d30/d59/f63 [842508,68896] 0 2024-04-05T21:26:53.150 INFO:tasks.workunit.client.0.smithi121.stdout:6/845: dread d1/d2/daf/d6f/dc7/fb7 [0,4194304] 0 2024-04-05T21:26:53.151 INFO:tasks.workunit.client.0.smithi121.stdout:9/679: creat d6/df/d48/d96/dc2/d85/fe7 x:0 0 0 2024-04-05T21:26:53.153 INFO:tasks.workunit.client.0.smithi121.stdout:1/890: dread d8/d89/fb9 [0,4194304] 0 2024-04-05T21:26:53.155 INFO:tasks.workunit.client.0.smithi121.stdout:6/846: stat d1/d2/d1b/d2e/l7a 0 2024-04-05T21:26:53.156 INFO:tasks.workunit.client.0.smithi121.stdout:2/902: creat d2/d9/d38/d2d/f125 x:0 0 0 2024-04-05T21:26:53.157 INFO:tasks.workunit.client.0.smithi121.stdout:9/680: creat d6/d32/dbe/d7b/d49/d66/fe8 x:0 0 0 2024-04-05T21:26:53.159 INFO:tasks.workunit.client.0.smithi121.stdout:3/806: dwrite d0/d3/d24/d3d/d1a/d1e/db1/ddd/de1/fd2 [0,4194304] 0 2024-04-05T21:26:53.160 INFO:tasks.workunit.client.0.smithi121.stdout:9/681: rmdir d6/d32/dbe/d7b/d6b 39 2024-04-05T21:26:53.162 INFO:tasks.workunit.client.0.smithi121.stdout:6/847: truncate d1/d2/d1b/d5e/dbd/d109/fae 1603140 0 2024-04-05T21:26:53.163 INFO:tasks.workunit.client.0.smithi121.stdout:1/891: truncate d8/d8c/db6/d1a/d1b/d44/fee 187224 0 2024-04-05T21:26:53.163 INFO:tasks.workunit.client.0.smithi121.stdout:3/807: creat d0/d3/d24/db3/dc3/f116 x:0 0 0 2024-04-05T21:26:53.165 INFO:tasks.workunit.client.0.smithi121.stdout:2/903: link d2/f92 d2/db6/dbb/dee/f126 0 2024-04-05T21:26:53.167 INFO:tasks.workunit.client.0.smithi121.stdout:3/808: creat d0/d3/d24/d6d/d7b/da3/dec/df9/f117 x:0 0 0 2024-04-05T21:26:53.168 INFO:tasks.workunit.client.0.smithi121.stdout:3/809: stat d0/d3/d24/d6d/d7b/da4/ffc 0 2024-04-05T21:26:53.170 INFO:tasks.workunit.client.0.smithi121.stdout:3/810: unlink d0/d3/d24/db3/c107 0 2024-04-05T21:26:53.170 INFO:tasks.workunit.client.0.smithi121.stdout:3/811: read - d0/d3/de/da6/fb4 zero size 2024-04-05T21:26:53.173 INFO:tasks.workunit.client.0.smithi121.stdout:3/812: mknod d0/d3/d24/d6d/c118 0 2024-04-05T21:26:53.175 INFO:tasks.workunit.client.0.smithi121.stdout:6/848: dread d1/f1c [0,4194304] 0 2024-04-05T21:26:53.176 INFO:tasks.workunit.client.0.smithi121.stdout:3/813: rename d0/d3/d24/d6d/c118 to d0/d3/d24/d3d/d1a/d1e/db1/ddd/de1/c119 0 2024-04-05T21:26:53.179 INFO:tasks.workunit.client.0.smithi121.stdout:3/814: dread d0/d3/d24/d3d/d1a/d1e/db1/fea [0,4194304] 0 2024-04-05T21:26:53.182 INFO:tasks.workunit.client.0.smithi121.stdout:3/815: write d0/d3/d24/d6d/d7b/da3/dec/df9/f117 [586669,42160] 0 2024-04-05T21:26:53.182 INFO:tasks.workunit.client.0.smithi121.stdout:6/849: dwrite d1/d2/d3/d4/d21/ff5 [0,4194304] 0 2024-04-05T21:26:53.186 INFO:tasks.workunit.client.0.smithi121.stdout:6/850: write d1/d2/d3/d4/d21/d57/f124 [585836,47109] 0 2024-04-05T21:26:53.187 INFO:tasks.workunit.client.0.smithi121.stdout:6/851: dread - d1/d2/d1b/d2e/ff7 zero size 2024-04-05T21:26:53.191 INFO:tasks.workunit.client.0.smithi121.stdout:6/852: stat d1/d2/d3/d4/d21/d57/d42/d48/d7c/cda 0 2024-04-05T21:26:53.193 INFO:tasks.workunit.client.0.smithi121.stdout:3/816: dwrite d0/d3/d24/d3d/d1a/d23/f56 [4194304,4194304] 0 2024-04-05T21:26:53.194 INFO:tasks.workunit.client.0.smithi121.stdout:6/853: readlink d1/d4b/da7/ld6 0 2024-04-05T21:26:53.195 INFO:tasks.workunit.client.0.smithi121.stdout:6/854: fsync d1/d2/d3/d4/d21/f84 0 2024-04-05T21:26:53.196 INFO:tasks.workunit.client.0.smithi121.stdout:6/855: write d1/d4b/dcc/fe9 [1080333,109500] 0 2024-04-05T21:26:53.198 INFO:tasks.workunit.client.0.smithi121.stdout:3/817: creat d0/dc6/f11a x:0 0 0 2024-04-05T21:26:53.201 INFO:tasks.workunit.client.0.smithi121.stdout:3/818: mknod d0/d3/d24/d6d/c11b 0 2024-04-05T21:26:53.202 INFO:tasks.workunit.client.0.smithi121.stdout:5/993: dread dd/de/d19/d38/d56/fe0 [0,4194304] 0 2024-04-05T21:26:53.204 INFO:tasks.workunit.client.0.smithi121.stdout:3/819: mkdir d0/d3/d11c 0 2024-04-05T21:26:53.207 INFO:tasks.workunit.client.0.smithi121.stdout:3/820: mkdir d0/d3/d11c/d11d 0 2024-04-05T21:26:53.208 INFO:tasks.workunit.client.0.smithi121.stdout:5/994: dwrite dd/de/d19/f57 [0,4194304] 0 2024-04-05T21:26:53.214 INFO:tasks.workunit.client.0.smithi121.stdout:3/821: link d0/d3/fc5 d0/d3/d24/d6d/d7b/de3/f11e 0 2024-04-05T21:26:53.216 INFO:tasks.workunit.client.1.smithi188.stdout:8/938: sync 2024-04-05T21:26:53.217 INFO:tasks.workunit.client.0.smithi121.stdout:3/822: stat d0/d3/d24/d3d/d1a/d23/d90/d9e/fa0 0 2024-04-05T21:26:53.219 INFO:tasks.workunit.client.0.smithi121.stdout:5/995: dwrite dd/de/d19/d1c/d36/ffc [0,4194304] 0 2024-04-05T21:26:53.221 INFO:tasks.workunit.client.1.smithi188.stdout:8/939: rename d5/d21/d40/d43/d44/d59/d8b/da4/db1/le6 to d5/d21/d2a/d95/dc0/d11f/l125 0 2024-04-05T21:26:53.221 INFO:tasks.workunit.client.0.smithi121.stdout:5/996: truncate dd/de/d19/d1c/f33 3027126 0 2024-04-05T21:26:53.222 INFO:tasks.workunit.client.1.smithi188.stdout:8/940: chown d5/d21/d40/d43/d44/d59/d8b/da4/db1/ffb 228537 1 2024-04-05T21:26:53.225 INFO:tasks.workunit.client.1.smithi188.stdout:8/941: mknod d5/d6e/d98/dc6/c126 0 2024-04-05T21:26:53.228 INFO:tasks.workunit.client.1.smithi188.stdout:8/942: symlink d5/d21/d40/d43/d44/d59/l127 0 2024-04-05T21:26:53.245 INFO:tasks.workunit.client.1.smithi188.stdout:8/943: rename d5/d21/d40/d43/d44/d59/d8b/da4/db1 to d5/d21/d2a/d116/d108/d128 0 2024-04-05T21:26:53.245 INFO:tasks.workunit.client.1.smithi188.stdout:8/944: chown d5/d21/d40/d43 0 1 2024-04-05T21:26:53.252 INFO:tasks.workunit.client.1.smithi188.stdout:8/945: dwrite d5/dc/d8f/dd1/fe2 [0,4194304] 0 2024-04-05T21:26:53.252 INFO:tasks.workunit.client.0.smithi121.stdout:6/856: dread d1/d2/daf/d6f/f107 [0,4194304] 0 2024-04-05T21:26:53.253 INFO:tasks.workunit.client.0.smithi121.stdout:6/857: rmdir d1/d2/d3/d59 39 2024-04-05T21:26:53.254 INFO:tasks.workunit.client.0.smithi121.stdout:6/858: write d1/d2/f120 [793904,50243] 0 2024-04-05T21:26:53.257 INFO:tasks.workunit.client.0.smithi121.stdout:6/859: mknod d1/d2/d1b/d35/d51/c12b 0 2024-04-05T21:26:53.259 INFO:tasks.workunit.client.0.smithi121.stdout:6/860: dread - d1/d2/daf/d6f/dc7/d1f/d6d/d101/f111 zero size 2024-04-05T21:26:53.260 INFO:tasks.workunit.client.0.smithi121.stdout:6/861: readlink d1/d2/daf/d6f/dc7/d1f/l8e 0 2024-04-05T21:26:53.262 INFO:tasks.workunit.client.0.smithi121.stdout:6/862: creat d1/d2/dc/f12c x:0 0 0 2024-04-05T21:26:53.262 INFO:tasks.workunit.client.1.smithi188.stdout:8/946: write d5/d21/d40/d43/f83 [475502,65844] 0 2024-04-05T21:26:53.264 INFO:tasks.workunit.client.0.smithi121.stdout:6/863: mkdir d1/d2/daf/d12d 0 2024-04-05T21:26:53.265 INFO:tasks.workunit.client.1.smithi188.stdout:8/947: mknod d5/d6e/d86/dcb/df8/c129 0 2024-04-05T21:26:53.265 INFO:tasks.workunit.client.1.smithi188.stdout:8/948: chown d5/f12 24965 1 2024-04-05T21:26:53.266 INFO:tasks.workunit.client.1.smithi188.stdout:8/949: chown d5/d6e/dad/f10f 48033 1 2024-04-05T21:26:53.267 INFO:tasks.workunit.client.0.smithi121.stdout:6/864: symlink d1/d2/daf/d6f/dc7/d1f/d6d/l12e 0 2024-04-05T21:26:53.268 INFO:tasks.workunit.client.0.smithi121.stdout:3/823: dread d0/d3/f5e [0,4194304] 0 2024-04-05T21:26:53.268 INFO:tasks.workunit.client.1.smithi188.stdout:8/950: mknod d5/d21/d40/d43/d44/d59/d79/c12a 0 2024-04-05T21:26:53.269 INFO:tasks.workunit.client.0.smithi121.stdout:6/865: fdatasync d1/d2/daf/d6f/dc7/fb7 0 2024-04-05T21:26:53.273 INFO:tasks.workunit.client.0.smithi121.stdout:6/866: write d1/d2/d3/d4/d21/ff5 [344226,25901] 0 2024-04-05T21:26:53.275 INFO:tasks.workunit.client.0.smithi121.stdout:3/824: dwrite d0/f114 [0,4194304] 0 2024-04-05T21:26:53.275 INFO:tasks.workunit.client.1.smithi188.stdout:8/951: truncate d5/dc/f1d 3407764 0 2024-04-05T21:26:53.276 INFO:tasks.workunit.client.0.smithi121.stdout:6/867: fsync d1/d2/daf/d6f/dc7/d1f/d6d/d101/f103 0 2024-04-05T21:26:53.276 INFO:tasks.workunit.client.0.smithi121.stdout:3/825: stat d0/d3/d24/d6d/d7b/da4/lbd 0 2024-04-05T21:26:53.277 INFO:tasks.workunit.client.1.smithi188.stdout:8/952: rmdir d5/d21/d40/d43/d48/d7e/d10a 39 2024-04-05T21:26:53.278 INFO:tasks.workunit.client.1.smithi188.stdout:8/953: chown d5/d6e/lf7 5274626 1 2024-04-05T21:26:53.279 INFO:tasks.workunit.client.0.smithi121.stdout:6/868: creat d1/d2/daf/d6f/f12f x:0 0 0 2024-04-05T21:26:53.281 INFO:tasks.workunit.client.0.smithi121.stdout:3/826: rename d0/d3/d24/db3/dc3/fc9 to d0/d3/de/da6/de2/f11f 0 2024-04-05T21:26:53.282 INFO:tasks.workunit.client.0.smithi121.stdout:6/869: dread d1/d2/daf/d6f/f11c [0,4194304] 0 2024-04-05T21:26:53.284 INFO:tasks.workunit.client.0.smithi121.stdout:6/870: write d1/d2/d3/d4/d21/d57/d42/d48/d7c/ffd [585924,1982] 0 2024-04-05T21:26:53.284 INFO:tasks.workunit.client.0.smithi121.stdout:3/827: creat d0/d3/d24/d3d/d1a/d23/d90/d9e/f120 x:0 0 0 2024-04-05T21:26:53.286 INFO:tasks.workunit.client.0.smithi121.stdout:6/871: creat d1/d2/d3/d4/df0/f130 x:0 0 0 2024-04-05T21:26:53.287 INFO:tasks.workunit.client.0.smithi121.stdout:3/828: creat d0/d3/d24/db3/dc3/f121 x:0 0 0 2024-04-05T21:26:53.288 INFO:tasks.workunit.client.0.smithi121.stdout:6/872: stat d1/d2/d3/d59/c8a 0 2024-04-05T21:26:53.290 INFO:tasks.workunit.client.0.smithi121.stdout:6/873: write d1/d2/d1b/d91/f102 [58929,127610] 0 2024-04-05T21:26:53.291 INFO:tasks.workunit.client.0.smithi121.stdout:6/874: read - d1/d2/d1b/d35/d39/d92/fe3 zero size 2024-04-05T21:26:53.300 INFO:tasks.workunit.client.0.smithi121.stdout:6/875: write d1/d2/daf/d6f/dc7/d1f/f6a [1486320,58942] 0 2024-04-05T21:26:53.303 INFO:tasks.workunit.client.0.smithi121.stdout:6/876: creat d1/d2/d3/d4/d21/d57/d42/f131 x:0 0 0 2024-04-05T21:26:53.309 INFO:tasks.workunit.client.0.smithi121.stdout:6/877: dwrite d1/d2/daf/d6f/dc7/d1f/d6d/f12a [0,4194304] 0 2024-04-05T21:26:53.310 INFO:tasks.workunit.client.0.smithi121.stdout:6/878: dread - d1/d2/daf/d6f/dc7/f10e zero size 2024-04-05T21:26:53.312 INFO:tasks.workunit.client.0.smithi121.stdout:3/829: dread d0/d3/de/f8a [4194304,4194304] 0 2024-04-05T21:26:53.314 INFO:tasks.workunit.client.1.smithi188.stdout:8/954: dread d5/d21/f123 [0,4194304] 0 2024-04-05T21:26:53.317 INFO:tasks.workunit.client.0.smithi121.stdout:3/830: symlink d0/d3/de/da6/df5/l122 0 2024-04-05T21:26:53.318 INFO:tasks.workunit.client.0.smithi121.stdout:6/879: dwrite d1/d2/d1b/d2e/fe5 [0,4194304] 0 2024-04-05T21:26:53.320 INFO:tasks.workunit.client.1.smithi188.stdout:8/955: dwrite d5/d21/d40/d43/f66 [0,4194304] 0 2024-04-05T21:26:53.325 INFO:tasks.workunit.client.0.smithi121.stdout:3/831: dwrite d0/fcb [0,4194304] 0 2024-04-05T21:26:53.326 INFO:tasks.workunit.client.0.smithi121.stdout:6/880: rename d1/d2/d1b/f9e to d1/d2/d1b/f132 0 2024-04-05T21:26:53.328 INFO:tasks.workunit.client.1.smithi188.stdout:8/956: getdents d5/d21/d40 0 2024-04-05T21:26:53.330 INFO:tasks.workunit.client.0.smithi121.stdout:3/832: chown d0/d3/d24/d3d/d1a/d23/d38/d55/f8d 4 1 2024-04-05T21:26:53.330 INFO:tasks.workunit.client.0.smithi121.stdout:6/881: creat d1/d2/daf/d6f/d126/f133 x:0 0 0 2024-04-05T21:26:53.330 INFO:tasks.workunit.client.0.smithi121.stdout:3/833: stat d0/d3/d24/d3d/d1a/d23/d90/c95 0 2024-04-05T21:26:53.332 INFO:tasks.workunit.client.0.smithi121.stdout:6/882: symlink d1/d2/d1b/d35/d74/df3/l134 0 2024-04-05T21:26:53.333 INFO:tasks.workunit.client.0.smithi121.stdout:3/834: mknod d0/d3/d11c/d11d/c123 0 2024-04-05T21:26:53.334 INFO:tasks.workunit.client.1.smithi188.stdout:8/957: rename d5/d21/d40/d43/d48/d7e/d121/c122 to d5/d21/d40/d43/c12b 0 2024-04-05T21:26:53.334 INFO:tasks.workunit.client.0.smithi121.stdout:3/835: chown d0/dc6/f11a 1597453 1 2024-04-05T21:26:53.335 INFO:tasks.workunit.client.0.smithi121.stdout:3/836: chown d0/d3/fa 2 1 2024-04-05T21:26:53.336 INFO:tasks.workunit.client.0.smithi121.stdout:6/883: truncate d1/d2/d3/d4/d21/d57/d42/f67 1610697 0 2024-04-05T21:26:53.338 INFO:tasks.workunit.client.1.smithi188.stdout:8/958: rename d5/d6e/d86/dcb/c11c to d5/d21/d40/d43/d44/d59/d79/c12c 0 2024-04-05T21:26:53.339 INFO:tasks.workunit.client.0.smithi121.stdout:3/837: mkdir d0/d3/d24/d6d/d7b/da3/dec/df6/dce/d124 0 2024-04-05T21:26:53.339 INFO:tasks.workunit.client.1.smithi188.stdout:8/959: write d5/dc/d8f/fd7 [41191,47941] 0 2024-04-05T21:26:53.339 INFO:tasks.workunit.client.0.smithi121.stdout:3/838: dread - d0/d3/d24/db3/fe8 zero size 2024-04-05T21:26:53.341 INFO:tasks.workunit.client.0.smithi121.stdout:3/839: symlink d0/d3/de/da6/df5/l125 0 2024-04-05T21:26:53.353 INFO:tasks.workunit.client.0.smithi121.stdout:6/884: write d1/d2/daf/d6f/f107 [1705542,40816] 0 2024-04-05T21:26:53.354 INFO:tasks.workunit.client.0.smithi121.stdout:6/885: chown d1/d2/daf/d6f/d126 863 1 2024-04-05T21:26:53.357 INFO:tasks.workunit.client.0.smithi121.stdout:6/886: getdents d1/d2/d1b/d35/d39/d92/dea 0 2024-04-05T21:26:53.357 INFO:tasks.workunit.client.0.smithi121.stdout:6/887: readlink d1/d4b/l11b 0 2024-04-05T21:26:53.358 INFO:tasks.workunit.client.0.smithi121.stdout:3/840: dread d0/d3/de/da6/de2/f11f [0,4194304] 0 2024-04-05T21:26:53.358 INFO:tasks.workunit.client.1.smithi188.stdout:8/960: dread d5/f1c [0,4194304] 0 2024-04-05T21:26:53.360 INFO:tasks.workunit.client.0.smithi121.stdout:6/888: creat d1/d2/daf/d6f/dc7/d1f/d6d/f135 x:0 0 0 2024-04-05T21:26:53.361 INFO:tasks.workunit.client.1.smithi188.stdout:8/961: creat d5/d21/d40/d43/d48/d7e/dcf/f12d x:0 0 0 2024-04-05T21:26:53.361 INFO:tasks.workunit.client.0.smithi121.stdout:3/841: symlink d0/d3/d24/d3d/d1a/d23/d38/dae/l126 0 2024-04-05T21:26:53.361 INFO:tasks.workunit.client.0.smithi121.stdout:6/889: dread - d1/d2/d1b/d35/d74/f108 zero size 2024-04-05T21:26:53.362 INFO:tasks.workunit.client.0.smithi121.stdout:6/890: chown d1/d2/d3/d4/d21/f93 49163631 1 2024-04-05T21:26:53.363 INFO:tasks.workunit.client.0.smithi121.stdout:3/842: truncate d0/d3/d24/d3d/d1a/d23/f6b 656906 0 2024-04-05T21:26:53.364 INFO:tasks.workunit.client.0.smithi121.stdout:6/891: unlink d1/d2/d1b/d35/d51/c8b 0 2024-04-05T21:26:53.365 INFO:tasks.workunit.client.1.smithi188.stdout:8/962: rename d5/d21/d40/d43/d44/d59/f74 to d5/d21/d40/d43/d44/d5b/d99/d69/f12e 0 2024-04-05T21:26:53.365 INFO:tasks.workunit.client.0.smithi121.stdout:3/843: mkdir d0/d3/d24/d3d/dcf/d127 0 2024-04-05T21:26:53.366 INFO:tasks.workunit.client.0.smithi121.stdout:6/892: symlink d1/d2/d3/d4/d21/l136 0 2024-04-05T21:26:53.367 INFO:tasks.workunit.client.1.smithi188.stdout:8/963: fdatasync d5/d21/d40/f7b 0 2024-04-05T21:26:53.367 INFO:tasks.workunit.client.0.smithi121.stdout:3/844: creat d0/dc6/f128 x:0 0 0 2024-04-05T21:26:53.369 INFO:tasks.workunit.client.0.smithi121.stdout:6/893: rename d1/d2/d3/d4/f88 to d1/d2/d3/d59/f137 0 2024-04-05T21:26:53.370 INFO:tasks.workunit.client.0.smithi121.stdout:3/845: write d0/d3/d24/d3d/d1a/d23/d90/d9e/fad [4377406,1481] 0 2024-04-05T21:26:53.371 INFO:tasks.workunit.client.0.smithi121.stdout:3/846: dread - d0/d3/d24/d3d/d1a/d1e/f105 zero size 2024-04-05T21:26:53.374 INFO:tasks.workunit.client.0.smithi121.stdout:6/894: link d1/d2/ceb d1/d2/d1b/d35/d74/d114/c138 0 2024-04-05T21:26:53.375 INFO:tasks.workunit.client.0.smithi121.stdout:6/895: dread d1/d2/daf/d6f/dc7/fb7 [0,4194304] 0 2024-04-05T21:26:53.376 INFO:tasks.workunit.client.0.smithi121.stdout:3/847: dread d0/d3/d24/f42 [0,4194304] 0 2024-04-05T21:26:53.376 INFO:tasks.workunit.client.0.smithi121.stdout:6/896: chown d1/d2/dc/d9a/dc0/le8 3809012 1 2024-04-05T21:26:53.378 INFO:tasks.workunit.client.0.smithi121.stdout:6/897: fsync d1/d2/daf/d6f/dc7/fb7 0 2024-04-05T21:26:53.378 INFO:tasks.workunit.client.0.smithi121.stdout:3/848: mkdir d0/d3/d24/d6d/d7b/da3/d129 0 2024-04-05T21:26:53.386 INFO:tasks.workunit.client.0.smithi121.stdout:3/849: write d0/d3/d24/d3d/d1a/d1e/db1/fea [669503,64262] 0 2024-04-05T21:26:53.389 INFO:tasks.workunit.client.0.smithi121.stdout:6/898: dread d1/d2/daf/d6f/dc7/d1f/f6a [0,4194304] 0 2024-04-05T21:26:53.390 INFO:tasks.workunit.client.0.smithi121.stdout:3/850: dread d0/d3/d24/d6d/d7b/f85 [0,4194304] 0 2024-04-05T21:26:53.391 INFO:tasks.workunit.client.0.smithi121.stdout:6/899: creat d1/d2/dc/d9a/f139 x:0 0 0 2024-04-05T21:26:53.393 INFO:tasks.workunit.client.0.smithi121.stdout:6/900: mkdir d1/d2/d1b/d91/d13a 0 2024-04-05T21:26:53.394 INFO:tasks.workunit.client.0.smithi121.stdout:3/851: write d0/d3/fa [737907,65398] 0 2024-04-05T21:26:53.395 INFO:tasks.workunit.client.0.smithi121.stdout:6/901: getdents d1/d2/d1b/d2e/dd9/de0 0 2024-04-05T21:26:53.397 INFO:tasks.workunit.client.0.smithi121.stdout:6/902: read d1/d2/d3/d4/d21/d57/d42/d48/d7c/db3/fc1 [182350,14800] 0 2024-04-05T21:26:53.397 INFO:tasks.workunit.client.0.smithi121.stdout:3/852: creat d0/d3/d24/d3d/d1a/d23/d38/d55/d103/f12a x:0 0 0 2024-04-05T21:26:53.401 INFO:tasks.workunit.client.0.smithi121.stdout:6/903: creat d1/f13b x:0 0 0 2024-04-05T21:26:53.403 INFO:tasks.workunit.client.0.smithi121.stdout:6/904: mknod d1/d2/d1b/d5e/c13c 0 2024-04-05T21:26:53.403 INFO:tasks.workunit.client.0.smithi121.stdout:7/973: sync 2024-04-05T21:26:53.403 INFO:tasks.workunit.client.0.smithi121.stdout:8/799: sync 2024-04-05T21:26:53.403 INFO:tasks.workunit.client.0.smithi121.stdout:3/853: dread d0/d3/d24/d6d/f81 [0,4194304] 0 2024-04-05T21:26:53.405 INFO:tasks.workunit.client.0.smithi121.stdout:6/905: rmdir d1/d2/d1b/d35/d74 39 2024-04-05T21:26:53.406 INFO:tasks.workunit.client.0.smithi121.stdout:8/800: creat d12/d13/d56/d57/d5e/ffa x:0 0 0 2024-04-05T21:26:53.406 INFO:tasks.workunit.client.0.smithi121.stdout:3/854: creat d0/d3/d24/df2/f12b x:0 0 0 2024-04-05T21:26:53.407 INFO:tasks.workunit.client.0.smithi121.stdout:6/906: symlink d1/d2/d3/d4/df0/d11a/l13d 0 2024-04-05T21:26:53.408 INFO:tasks.workunit.client.0.smithi121.stdout:3/855: chown d0/d3/d24/d3d/d1a/d23/l61 42 1 2024-04-05T21:26:53.409 INFO:tasks.workunit.client.0.smithi121.stdout:3/856: write d0/d3/d24/d3d/d1a/d1e/f2a [237860,45410] 0 2024-04-05T21:26:53.457 INFO:tasks.workunit.client.0.smithi121.stdout:6/907: mkdir d1/d2/d3/d4/df0/d11a/d13e 0 2024-04-05T21:26:53.458 INFO:tasks.workunit.client.0.smithi121.stdout:3/857: creat d0/d3/d24/d3d/d1a/d1e/f12c x:0 0 0 2024-04-05T21:26:53.464 INFO:tasks.workunit.client.0.smithi121.stdout:6/908: dwrite d1/d2/daf/d6f/dc7/d1f/d6d/f135 [0,4194304] 0 2024-04-05T21:26:53.466 INFO:tasks.workunit.client.0.smithi121.stdout:3/858: dwrite d0/d3/d24/d6d/d7b/ff7 [0,4194304] 0 2024-04-05T21:26:53.472 INFO:tasks.workunit.client.0.smithi121.stdout:6/909: dwrite d1/d2/d3/d4/d21/d57/d42/d48/d7c/ffd [0,4194304] 0 2024-04-05T21:26:53.498 INFO:tasks.workunit.client.0.smithi121.stdout:9/682: sync 2024-04-05T21:26:53.498 INFO:tasks.workunit.client.0.smithi121.stdout:0/951: sync 2024-04-05T21:26:53.498 INFO:tasks.workunit.client.0.smithi121.stdout:5/997: sync 2024-04-05T21:26:53.498 INFO:tasks.workunit.client.0.smithi121.stdout:2/904: sync 2024-04-05T21:26:53.498 INFO:tasks.workunit.client.0.smithi121.stdout:1/892: sync 2024-04-05T21:26:53.499 INFO:tasks.workunit.client.0.smithi121.stdout:9/683: chown d6/df/c25 3 1 2024-04-05T21:26:53.502 INFO:tasks.workunit.client.0.smithi121.stdout:8/801: truncate d12/d13/d56/d57/f88 4478596 0 2024-04-05T21:26:53.522 INFO:tasks.workunit.client.0.smithi121.stdout:6/910: mkdir d1/d2/d1b/d35/d39/d92/d13f 0 2024-04-05T21:26:53.523 INFO:tasks.workunit.client.0.smithi121.stdout:0/952: fdatasync d7/d1c/fb6 0 2024-04-05T21:26:53.523 INFO:tasks.workunit.client.0.smithi121.stdout:1/893: rmdir d8/d89/d9e/dc8 39 2024-04-05T21:26:53.525 INFO:tasks.workunit.client.0.smithi121.stdout:1/894: write d8/d8c/db6/d1a/d1b/d85/ff8 [254152,6491] 0 2024-04-05T21:26:53.529 INFO:tasks.workunit.client.0.smithi121.stdout:6/911: creat d1/d2/d1b/f140 x:0 0 0 2024-04-05T21:26:53.531 INFO:tasks.workunit.client.0.smithi121.stdout:0/953: unlink d7/d62/f5c 0 2024-04-05T21:26:53.534 INFO:tasks.workunit.client.0.smithi121.stdout:9/684: mkdir d6/df/d48/d96/db9/d2c/de9 0 2024-04-05T21:26:53.535 INFO:tasks.workunit.client.1.smithi188.stdout:8/964: sync 2024-04-05T21:26:53.538 INFO:tasks.workunit.client.1.smithi188.stdout:8/965: mkdir d5/d21/d40/d43/d44/d59/d79/d12f 0 2024-04-05T21:26:53.540 INFO:tasks.workunit.client.1.smithi188.stdout:8/966: symlink d5/dc/l130 0 2024-04-05T21:26:53.541 INFO:tasks.workunit.client.0.smithi121.stdout:0/954: write d7/f84 [2435587,39469] 0 2024-04-05T21:26:53.543 INFO:tasks.workunit.client.0.smithi121.stdout:9/685: unlink d6/d46/le1 0 2024-04-05T21:26:53.546 INFO:tasks.workunit.client.0.smithi121.stdout:5/998: getdents dd/de/d19/d13 0 2024-04-05T21:26:53.546 INFO:tasks.workunit.client.0.smithi121.stdout:0/955: dread d7/d71/d45/d46/d53/f12e [0,4194304] 0 2024-04-05T21:26:53.547 INFO:tasks.workunit.client.0.smithi121.stdout:9/686: truncate d6/df/d48/d96/db9/d2c/d37/d3c/fde 557847 0 2024-04-05T21:26:53.550 INFO:tasks.workunit.client.0.smithi121.stdout:0/956: mknod d7/d62/d8b/c14c 0 2024-04-05T21:26:53.551 INFO:tasks.workunit.client.0.smithi121.stdout:5/999: mknod dd/de/d19/d1c/d74/d8c/c159 0 2024-04-05T21:26:53.553 INFO:tasks.workunit.client.0.smithi121.stdout:1/895: creat d8/d89/d9e/d2d/d84/f127 x:0 0 0 2024-04-05T21:26:53.554 INFO:tasks.workunit.client.0.smithi121.stdout:0/957: chown d7/d71/d29/d3f/le4 2 1 2024-04-05T21:26:53.556 INFO:tasks.workunit.client.0.smithi121.stdout:9/687: creat d6/d46/fea x:0 0 0 2024-04-05T21:26:53.556 INFO:tasks.workunit.client.1.smithi188.stdout:8/967: dread d5/d21/d40/f7b [0,4194304] 0 2024-04-05T21:26:53.558 INFO:tasks.workunit.client.0.smithi121.stdout:0/958: mkdir d7/d62/d3c/d6b/d14d 0 2024-04-05T21:26:53.559 INFO:tasks.workunit.client.1.smithi188.stdout:8/968: rmdir d5/d21/d40/d43/d44/d59/d79/d12f 0 2024-04-05T21:26:53.560 INFO:tasks.workunit.client.0.smithi121.stdout:0/959: mknod d7/d71/d94/da0/c14e 0 2024-04-05T21:26:53.561 INFO:tasks.workunit.client.0.smithi121.stdout:1/896: rename d8/d8c/db6/d1a/d1b/d44/fee to d8/d89/d9e/dc8/df4/d107/f128 0 2024-04-05T21:26:53.561 INFO:tasks.workunit.client.1.smithi188.stdout:8/969: symlink d5/d6e/d86/l131 0 2024-04-05T21:26:53.566 INFO:tasks.workunit.client.1.smithi188.stdout:8/970: dread d5/d6e/d86/dcb/f5f [0,4194304] 0 2024-04-05T21:26:53.574 INFO:tasks.workunit.client.1.smithi188.stdout:8/971: dwrite d5/d21/d40/f10b [0,4194304] 0 2024-04-05T21:26:53.575 INFO:tasks.workunit.client.0.smithi121.stdout:2/905: dwrite d2/d9/d1c/d30/d59/f89 [0,4194304] 0 2024-04-05T21:26:53.575 INFO:tasks.workunit.client.1.smithi188.stdout:8/972: rmdir d5/d6e/d86/dcb 39 2024-04-05T21:26:53.577 INFO:tasks.workunit.client.1.smithi188.stdout:8/973: fsync d5/dc/d8f/dd1/fe2 0 2024-04-05T21:26:53.579 INFO:tasks.workunit.client.0.smithi121.stdout:8/802: dwrite d12/d50/d93/d96/f92 [0,4194304] 0 2024-04-05T21:26:53.580 INFO:tasks.workunit.client.0.smithi121.stdout:1/897: fdatasync d8/d89/d9e/d2d/d84/da8/dad/dde/f54 0 2024-04-05T21:26:53.582 INFO:tasks.workunit.client.0.smithi121.stdout:0/960: truncate d7/d71/d29/de8/ffe 617811 0 2024-04-05T21:26:53.583 INFO:tasks.workunit.client.0.smithi121.stdout:6/912: dwrite d1/d2/d3/d4/d21/d57/d42/d48/d7c/db3/fc1 [0,4194304] 0 2024-04-05T21:26:53.587 INFO:tasks.workunit.client.0.smithi121.stdout:6/913: dread - d1/d2/d1b/d5e/fef zero size 2024-04-05T21:26:53.588 INFO:tasks.workunit.client.0.smithi121.stdout:8/803: creat d12/d13/d15/de7/ffb x:0 0 0 2024-04-05T21:26:53.592 INFO:tasks.workunit.client.0.smithi121.stdout:0/961: symlink d7/d71/d45/d5a/l14f 0 2024-04-05T21:26:53.593 INFO:tasks.workunit.client.0.smithi121.stdout:2/906: rename d2/d9/d38/d2d/d4c/d7c/cdb to d2/d28/d11d/c127 0 2024-04-05T21:26:53.594 INFO:tasks.workunit.client.0.smithi121.stdout:1/898: readlink d8/d8c/db6/d1a/d1b/d38/lcd 0 2024-04-05T21:26:53.596 INFO:tasks.workunit.client.0.smithi121.stdout:0/962: creat d7/d71/d94/da0/f150 x:0 0 0 2024-04-05T21:26:53.599 INFO:tasks.workunit.client.0.smithi121.stdout:0/963: truncate d7/d71/d29/de8/f11a 455821 0 2024-04-05T21:26:53.600 INFO:tasks.workunit.client.0.smithi121.stdout:2/907: rename d2/d9/l1b to d2/d28/d11d/l128 0 2024-04-05T21:26:53.601 INFO:tasks.workunit.client.0.smithi121.stdout:6/914: getdents d1/d2/d3/d4/df0 0 2024-04-05T21:26:53.603 INFO:tasks.workunit.client.0.smithi121.stdout:2/908: rmdir d2/d9/d1c/d85/def 39 2024-04-05T21:26:53.603 INFO:tasks.workunit.client.1.smithi188.stdout:8/974: dwrite d5/d21/d40/d43/d114/fd5 [0,4194304] 0 2024-04-05T21:26:53.607 INFO:tasks.workunit.client.0.smithi121.stdout:2/909: fsync d2/d1e/f52 0 2024-04-05T21:26:53.607 INFO:tasks.workunit.client.0.smithi121.stdout:8/804: rmdir d12/d13/d15/de7 39 2024-04-05T21:26:53.608 INFO:tasks.workunit.client.0.smithi121.stdout:2/910: truncate d2/d9/ffc 384234 0 2024-04-05T21:26:53.611 INFO:tasks.workunit.client.0.smithi121.stdout:6/915: dwrite d1/d2/f13 [4194304,4194304] 0 2024-04-05T21:26:53.615 INFO:tasks.workunit.client.0.smithi121.stdout:2/911: mkdir d2/db6/d129 0 2024-04-05T21:26:53.615 INFO:tasks.workunit.client.0.smithi121.stdout:0/964: write d7/d71/d29/d3f/f4f [1432875,60280] 0 2024-04-05T21:26:53.617 INFO:tasks.workunit.client.0.smithi121.stdout:8/805: symlink d12/d13/dd3/df0/da7/lfc 0 2024-04-05T21:26:53.620 INFO:tasks.workunit.client.0.smithi121.stdout:2/912: dread d2/d9/d17/fa7 [0,4194304] 0 2024-04-05T21:26:53.625 INFO:tasks.workunit.client.0.smithi121.stdout:0/965: rename d7/dc/da3/df5 to d7/d71/d29/d151 0 2024-04-05T21:26:53.626 INFO:tasks.workunit.client.0.smithi121.stdout:1/899: dwrite d8/d8c/db6/d1a/d1b/d24/fe9 [0,4194304] 0 2024-04-05T21:26:53.627 INFO:tasks.workunit.client.1.smithi188.stdout:8/975: write d5/d21/d40/d43/d44/d5b/d99/d69/d70/dae/fc2 [581673,42816] 0 2024-04-05T21:26:53.627 INFO:tasks.workunit.client.1.smithi188.stdout:8/976: creat d5/d21/d2a/d47/f132 x:0 0 0 2024-04-05T21:26:53.630 INFO:tasks.workunit.client.1.smithi188.stdout:8/977: symlink d5/d21/d40/d43/d44/d5b/l133 0 2024-04-05T21:26:53.630 INFO:tasks.workunit.client.0.smithi121.stdout:0/966: write d7/d1c/f55 [1799307,34231] 0 2024-04-05T21:26:53.632 INFO:tasks.workunit.client.1.smithi188.stdout:8/978: creat d5/d21/d2a/d116/d108/d128/f134 x:0 0 0 2024-04-05T21:26:53.633 INFO:tasks.workunit.client.0.smithi121.stdout:2/913: creat d2/d9/d1c/d85/f12a x:0 0 0 2024-04-05T21:26:53.633 INFO:tasks.workunit.client.1.smithi188.stdout:8/979: chown d5/d21/d40/l85 63803 1 2024-04-05T21:26:53.634 INFO:tasks.workunit.client.0.smithi121.stdout:8/806: dwrite ff [0,4194304] 0 2024-04-05T21:26:53.634 INFO:tasks.workunit.client.0.smithi121.stdout:1/900: truncate d8/d10/f17 3528124 0 2024-04-05T21:26:53.634 INFO:tasks.workunit.client.1.smithi188.stdout:8/980: chown d5/d21/d2a/f8d 380849527 1 2024-04-05T21:26:53.635 INFO:tasks.workunit.client.0.smithi121.stdout:2/914: chown d2/d1e/faf 111075631 1 2024-04-05T21:26:53.637 INFO:tasks.workunit.client.0.smithi121.stdout:8/807: chown d12/d50/d61/cf6 0 1 2024-04-05T21:26:53.637 INFO:tasks.workunit.client.1.smithi188.stdout:8/981: getdents d5/d21/d40 0 2024-04-05T21:26:53.638 INFO:tasks.workunit.client.0.smithi121.stdout:2/915: write d2/d9/dcb/f105 [199776,53529] 0 2024-04-05T21:26:53.639 INFO:tasks.workunit.client.0.smithi121.stdout:1/901: unlink d8/d8c/db6/d1a/f46 0 2024-04-05T21:26:53.639 INFO:tasks.workunit.client.0.smithi121.stdout:6/916: dwrite d1/d2/d3/d4/d21/d57/f49 [0,4194304] 0 2024-04-05T21:26:53.641 INFO:tasks.workunit.client.0.smithi121.stdout:2/916: stat d2/d9/d38/d2d 0 2024-04-05T21:26:53.642 INFO:tasks.workunit.client.0.smithi121.stdout:6/917: read - d1/d2/dc/d9a/f139 zero size 2024-04-05T21:26:53.642 INFO:tasks.workunit.client.1.smithi188.stdout:8/982: link d5/d21/d2a/d116/d108/d128/ffb d5/d21/d2a/d116/f135 0 2024-04-05T21:26:53.643 INFO:tasks.workunit.client.0.smithi121.stdout:1/902: dread - d8/d8c/db6/d1a/d1b/d24/fc6 zero size 2024-04-05T21:26:53.646 INFO:tasks.workunit.client.0.smithi121.stdout:2/917: chown d2/d9/d17/d3a/f4a 1 1 2024-04-05T21:26:53.646 INFO:tasks.workunit.client.1.smithi188.stdout:8/983: creat d5/d6e/d86/dcb/df8/f136 x:0 0 0 2024-04-05T21:26:53.648 INFO:tasks.workunit.client.0.smithi121.stdout:1/903: fdatasync d8/d8c/db6/d1a/d1b/d4c/f4f 0 2024-04-05T21:26:53.648 INFO:tasks.workunit.client.0.smithi121.stdout:7/974: sync 2024-04-05T21:26:53.649 INFO:tasks.workunit.client.1.smithi188.stdout:8/984: symlink d5/d21/d2a/d116/d108/d128/l137 0 2024-04-05T21:26:53.650 INFO:tasks.workunit.client.0.smithi121.stdout:6/918: creat d1/d2/d3/d4/df0/d11a/d13e/f141 x:0 0 0 2024-04-05T21:26:53.655 INFO:tasks.workunit.client.0.smithi121.stdout:7/975: dwrite d0/f1f [0,4194304] 0 2024-04-05T21:26:53.656 INFO:tasks.workunit.client.0.smithi121.stdout:1/904: dread - d8/d8c/db6/d1a/d1b/fc0 zero size 2024-04-05T21:26:53.658 INFO:tasks.workunit.client.0.smithi121.stdout:7/976: chown d0/d9/d22/d3e/d9c/lad 626259 1 2024-04-05T21:26:53.660 INFO:tasks.workunit.client.0.smithi121.stdout:7/977: chown d0/l7 303 1 2024-04-05T21:26:53.664 INFO:tasks.workunit.client.0.smithi121.stdout:1/905: creat d8/d89/d9e/d2d/d84/da8/d11d/f129 x:0 0 0 2024-04-05T21:26:53.666 INFO:tasks.workunit.client.0.smithi121.stdout:6/919: getdents d1/d4b/dcc 0 2024-04-05T21:26:53.668 INFO:tasks.workunit.client.0.smithi121.stdout:0/967: dwrite d7/dc/f33 [0,4194304] 0 2024-04-05T21:26:53.668 INFO:tasks.workunit.client.0.smithi121.stdout:1/906: dread - d8/d8c/db6/d1a/d1b/d44/fd5 zero size 2024-04-05T21:26:53.670 INFO:tasks.workunit.client.0.smithi121.stdout:8/808: dread d12/d13/d15/f18 [0,4194304] 0 2024-04-05T21:26:53.671 INFO:tasks.workunit.client.0.smithi121.stdout:2/918: write d2/f92 [665654,65120] 0 2024-04-05T21:26:53.676 INFO:tasks.workunit.client.0.smithi121.stdout:7/978: fsync d0/d9/d22/d28/d56/db9/ddf/def/d52/d61/dcb/fdc 0 2024-04-05T21:26:53.676 INFO:tasks.workunit.client.0.smithi121.stdout:6/920: creat d1/d2/d3/d4/d21/d57/f142 x:0 0 0 2024-04-05T21:26:53.676 INFO:tasks.workunit.client.0.smithi121.stdout:0/968: write d7/fdd [995677,94473] 0 2024-04-05T21:26:53.677 INFO:tasks.workunit.client.0.smithi121.stdout:2/919: write d2/f92 [1411473,115542] 0 2024-04-05T21:26:53.677 INFO:tasks.workunit.client.0.smithi121.stdout:7/979: chown d0/d32/d4e/d73/dbc 6 1 2024-04-05T21:26:53.678 INFO:tasks.workunit.client.0.smithi121.stdout:7/980: write d0/f8 [5297565,38001] 0 2024-04-05T21:26:53.679 INFO:tasks.workunit.client.0.smithi121.stdout:2/920: chown d2/d9/d38/d2d/d4c/de9/d8d/cf7 1206388499 1 2024-04-05T21:26:53.680 INFO:tasks.workunit.client.0.smithi121.stdout:8/809: rename d12/d50/d61/d7f/d9a/fb0 to d12/d50/ffd 0 2024-04-05T21:26:53.680 INFO:tasks.workunit.client.0.smithi121.stdout:1/907: write d8/d8c/db6/d1a/d1b/d4c/f7e [586304,439] 0 2024-04-05T21:26:53.683 INFO:tasks.workunit.client.0.smithi121.stdout:8/810: dread - d12/d13/d56/d57/d5e/db9/ff9 zero size 2024-04-05T21:26:53.684 INFO:tasks.workunit.client.0.smithi121.stdout:2/921: creat d2/d9/d38/d2d/d4c/d79/f12b x:0 0 0 2024-04-05T21:26:53.684 INFO:tasks.workunit.client.0.smithi121.stdout:1/908: dread - d8/d8c/db6/d1a/d1b/d4c/da1/fda zero size 2024-04-05T21:26:53.685 INFO:tasks.workunit.client.0.smithi121.stdout:7/981: creat d0/d32/d7c/f134 x:0 0 0 2024-04-05T21:26:53.685 INFO:tasks.workunit.client.0.smithi121.stdout:2/922: chown d2/d9/d1c 10833 1 2024-04-05T21:26:53.686 INFO:tasks.workunit.client.0.smithi121.stdout:6/921: mkdir d1/d2/d1b/d35/d74/df3/d143 0 2024-04-05T21:26:53.689 INFO:tasks.workunit.client.0.smithi121.stdout:8/811: rename d12/d13/l5f to d12/d50/dbb/de5/lfe 0 2024-04-05T21:26:53.689 INFO:tasks.workunit.client.0.smithi121.stdout:7/982: write d0/d9/d22/d28/d56/db9/ddf/def/d52/f123 [464882,82953] 0 2024-04-05T21:26:53.690 INFO:tasks.workunit.client.0.smithi121.stdout:0/969: link d7/d71/d29/d2a/d2c/fc3 d7/d71/d45/f152 0 2024-04-05T21:26:53.693 INFO:tasks.workunit.client.0.smithi121.stdout:1/909: truncate d8/d89/d9e/d57/fe3 2235469 0 2024-04-05T21:26:53.694 INFO:tasks.workunit.client.0.smithi121.stdout:1/910: dread - d8/d89/d9e/d2d/d84/da8/f113 zero size 2024-04-05T21:26:53.694 INFO:tasks.workunit.client.0.smithi121.stdout:2/923: rename d2/d9/d1c/d30/dba to d2/d9/d38/d2d/d4c/d7c/d7e/db9/d12c 0 2024-04-05T21:26:53.697 INFO:tasks.workunit.client.0.smithi121.stdout:2/924: chown d2/d9/d38/d2d/d4c/de9/d5c/c65 1352009 1 2024-04-05T21:26:53.700 INFO:tasks.workunit.client.0.smithi121.stdout:8/812: rename d12/d50/d93/d96/f8a to d12/d13/d56/d57/d5e/fff 0 2024-04-05T21:26:53.700 INFO:tasks.workunit.client.0.smithi121.stdout:6/922: link d1/d4b/lfe d1/d2/d1b/d35/d74/df3/l144 0 2024-04-05T21:26:53.701 INFO:tasks.workunit.client.0.smithi121.stdout:1/911: dwrite d8/d89/d9e/dc8/f11f [0,4194304] 0 2024-04-05T21:26:53.710 INFO:tasks.workunit.client.0.smithi121.stdout:6/923: unlink d1/d2/daf/d6f/dc7/fd 0 2024-04-05T21:26:53.713 INFO:tasks.workunit.client.0.smithi121.stdout:8/813: mknod d12/d13/c100 0 2024-04-05T21:26:53.714 INFO:tasks.workunit.client.0.smithi121.stdout:0/970: getdents d7/d7e/d97 0 2024-04-05T21:26:53.715 INFO:tasks.workunit.client.0.smithi121.stdout:7/983: dread d0/d9/d22/d3e/d76/de1/f102 [0,4194304] 0 2024-04-05T21:26:53.715 INFO:tasks.workunit.client.0.smithi121.stdout:1/912: symlink d8/d8c/db6/d1a/l12a 0 2024-04-05T21:26:53.717 INFO:tasks.workunit.client.0.smithi121.stdout:0/971: write d7/d71/d94/dd1/dd4/ffb [387707,54578] 0 2024-04-05T21:26:53.718 INFO:tasks.workunit.client.0.smithi121.stdout:2/925: write d2/d1e/fd5 [1578614,72865] 0 2024-04-05T21:26:53.720 INFO:tasks.workunit.client.0.smithi121.stdout:2/926: write d2/d9/d38/d2d/d4c/d7c/dc2/fc9 [60011,54275] 0 2024-04-05T21:26:53.721 INFO:tasks.workunit.client.0.smithi121.stdout:2/927: fdatasync d2/d1e/f52 0 2024-04-05T21:26:53.723 INFO:tasks.workunit.client.0.smithi121.stdout:8/814: mkdir d12/d50/d6a/d63/d8b/dd7/d101 0 2024-04-05T21:26:53.724 INFO:tasks.workunit.client.0.smithi121.stdout:8/815: chown d12/d50/fca 369617 1 2024-04-05T21:26:53.725 INFO:tasks.workunit.client.0.smithi121.stdout:7/984: mkdir d0/d23/d135 0 2024-04-05T21:26:53.729 INFO:tasks.workunit.client.0.smithi121.stdout:7/985: symlink d0/d9/d22/d28/d56/db9/ddf/l136 0 2024-04-05T21:26:53.730 INFO:tasks.workunit.client.0.smithi121.stdout:8/816: getdents d12/d50/d6a/de6 0 2024-04-05T21:26:53.731 INFO:tasks.workunit.client.0.smithi121.stdout:1/913: write d8/d8c/db6/d1a/d1b/d4c/f63 [986992,53427] 0 2024-04-05T21:26:53.732 INFO:tasks.workunit.client.0.smithi121.stdout:7/986: creat d0/d9/d22/d81/f137 x:0 0 0 2024-04-05T21:26:53.733 INFO:tasks.workunit.client.0.smithi121.stdout:6/924: dread d1/d2/dc/d9a/dc0/f2d [0,4194304] 0 2024-04-05T21:26:53.735 INFO:tasks.workunit.client.0.smithi121.stdout:6/925: stat d1/d2/daf/d6f/dc7/f66 0 2024-04-05T21:26:53.736 INFO:tasks.workunit.client.0.smithi121.stdout:0/972: truncate d7/d1c/f68 1760273 0 2024-04-05T21:26:53.737 INFO:tasks.workunit.client.0.smithi121.stdout:7/987: mknod d0/d9/d22/d3e/c138 0 2024-04-05T21:26:53.739 INFO:tasks.workunit.client.0.smithi121.stdout:6/926: mknod d1/d2/dc/d9a/dc0/c145 0 2024-04-05T21:26:53.740 INFO:tasks.workunit.client.0.smithi121.stdout:0/973: dread d7/d71/f5f [0,4194304] 0 2024-04-05T21:26:53.743 INFO:tasks.workunit.client.0.smithi121.stdout:7/988: getdents d0/d9/d22/d81/da2 0 2024-04-05T21:26:53.747 INFO:tasks.workunit.client.0.smithi121.stdout:6/927: creat d1/d2/d1b/d35/d51/f146 x:0 0 0 2024-04-05T21:26:53.748 INFO:tasks.workunit.client.0.smithi121.stdout:8/817: dwrite d12/d50/d6a/d87/ff5 [0,4194304] 0 2024-04-05T21:26:53.749 INFO:tasks.workunit.client.0.smithi121.stdout:1/914: dwrite d8/d89/d9e/f75 [0,4194304] 0 2024-04-05T21:26:53.750 INFO:tasks.workunit.client.0.smithi121.stdout:0/974: rename d7/d71/d94/dd1/f133 to d7/d71/d29/de1/f153 0 2024-04-05T21:26:53.751 INFO:tasks.workunit.client.0.smithi121.stdout:0/975: readlink d7/d71/d94/dd1/lfa 0 2024-04-05T21:26:53.754 INFO:tasks.workunit.client.0.smithi121.stdout:6/928: creat d1/d2/d3/d4/d21/d57/f147 x:0 0 0 2024-04-05T21:26:53.756 INFO:tasks.workunit.client.0.smithi121.stdout:0/976: mkdir d7/dc/da3/d154 0 2024-04-05T21:26:53.756 INFO:tasks.workunit.client.0.smithi121.stdout:8/818: symlink d12/d50/d6a/de6/l102 0 2024-04-05T21:26:53.760 INFO:tasks.workunit.client.0.smithi121.stdout:0/977: symlink d7/dc/da3/l155 0 2024-04-05T21:26:53.763 INFO:tasks.workunit.client.0.smithi121.stdout:6/929: rename d1/d2/d1b/d35/d39/d92/fd8 to d1/d2/d3/f148 0 2024-04-05T21:26:53.764 INFO:tasks.workunit.client.0.smithi121.stdout:1/915: getdents d8/d89/d9e/dc8/df4 0 2024-04-05T21:26:53.769 INFO:tasks.workunit.client.0.smithi121.stdout:8/819: getdents d12/d13/d15/de7 0 2024-04-05T21:26:53.771 INFO:tasks.workunit.client.0.smithi121.stdout:8/820: mknod d12/d50/dbb/de5/c103 0 2024-04-05T21:26:53.777 INFO:tasks.workunit.client.0.smithi121.stdout:6/930: dread d1/d2/daf/d6f/dc7/f58 [0,4194304] 0 2024-04-05T21:26:53.779 INFO:tasks.workunit.client.0.smithi121.stdout:8/821: rmdir d12/d50/d6a 39 2024-04-05T21:26:53.780 INFO:tasks.workunit.client.0.smithi121.stdout:1/916: dread d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b/fd2 [0,4194304] 0 2024-04-05T21:26:53.780 INFO:tasks.workunit.client.0.smithi121.stdout:6/931: symlink d1/d2/d1b/d91/l149 0 2024-04-05T21:26:53.780 INFO:tasks.workunit.client.0.smithi121.stdout:8/822: read - d12/d50/d93/ff1 zero size 2024-04-05T21:26:53.781 INFO:tasks.workunit.client.0.smithi121.stdout:8/823: fdatasync d12/d13/d56/d57/f5a 0 2024-04-05T21:26:53.782 INFO:tasks.workunit.client.0.smithi121.stdout:1/917: fdatasync d8/d10/ff9 0 2024-04-05T21:26:53.782 INFO:tasks.workunit.client.0.smithi121.stdout:1/918: dread - d8/d10/ff6 zero size 2024-04-05T21:26:53.787 INFO:tasks.workunit.client.0.smithi121.stdout:1/919: truncate d8/d89/d9e/d2d/fd1 431353 0 2024-04-05T21:26:53.789 INFO:tasks.workunit.client.0.smithi121.stdout:8/824: symlink d12/d50/d93/l104 0 2024-04-05T21:26:53.794 INFO:tasks.workunit.client.0.smithi121.stdout:6/932: dwrite d1/d2/dc/f5b [4194304,4194304] 0 2024-04-05T21:26:53.796 INFO:tasks.workunit.client.0.smithi121.stdout:1/920: dwrite d8/d89/d9e/d57/f7f [4194304,4194304] 0 2024-04-05T21:26:53.797 INFO:tasks.workunit.client.0.smithi121.stdout:6/933: stat d1/d2/d3/d4/d21/d57/f87 0 2024-04-05T21:26:53.803 INFO:tasks.workunit.client.0.smithi121.stdout:6/934: getdents d1/d2/d1b/d35/d51 0 2024-04-05T21:26:53.803 INFO:tasks.workunit.client.0.smithi121.stdout:1/921: link d8/d8c/db6/d1a/l12a d8/d89/d9e/d2d/d84/da8/dad/dde/l12b 0 2024-04-05T21:26:53.805 INFO:tasks.workunit.client.0.smithi121.stdout:3/859: sync 2024-04-05T21:26:53.805 INFO:tasks.workunit.client.0.smithi121.stdout:8/825: dwrite d12/d50/d61/d7f/f97 [4194304,4194304] 0 2024-04-05T21:26:53.809 INFO:tasks.workunit.client.0.smithi121.stdout:3/860: write d0/d3/d24/db3/dc3/f121 [198018,60251] 0 2024-04-05T21:26:53.809 INFO:tasks.workunit.client.1.smithi188.stdout:8/985: sync 2024-04-05T21:26:53.811 INFO:tasks.workunit.client.1.smithi188.stdout:8/986: mkdir d5/d21/d2a/d116/d108/d138 0 2024-04-05T21:26:53.812 INFO:tasks.workunit.client.1.smithi188.stdout:8/987: stat d5/d21/d2a/d47/f49 0 2024-04-05T21:26:53.813 INFO:tasks.workunit.client.1.smithi188.stdout:8/988: write d5/d21/d2a/f31 [1456733,84144] 0 2024-04-05T21:26:53.817 INFO:tasks.workunit.client.1.smithi188.stdout:8/989: creat d5/d21/d2a/d116/d108/d128/f139 x:0 0 0 2024-04-05T21:26:53.818 INFO:tasks.workunit.client.0.smithi121.stdout:0/978: read d7/d62/f6d [390504,40230] 0 2024-04-05T21:26:53.821 INFO:tasks.workunit.client.0.smithi121.stdout:6/935: creat d1/d2/d3/d4/d21/d57/d42/d48/d7c/db3/f14a x:0 0 0 2024-04-05T21:26:53.823 INFO:tasks.workunit.client.0.smithi121.stdout:7/989: dread d0/d32/d4a/f53 [0,4194304] 0 2024-04-05T21:26:53.823 INFO:tasks.workunit.client.0.smithi121.stdout:8/826: rename d12/d50/d6a to d12/d13/d56/d57/d105 0 2024-04-05T21:26:53.825 INFO:tasks.workunit.client.0.smithi121.stdout:0/979: creat d7/d71/d29/d2a/d2c/f156 x:0 0 0 2024-04-05T21:26:53.826 INFO:tasks.workunit.client.0.smithi121.stdout:6/936: write d1/d2/d1b/d5e/dbd/d109/fae [1193597,63253] 0 2024-04-05T21:26:53.828 INFO:tasks.workunit.client.0.smithi121.stdout:3/861: dwrite d0/d3/d24/d3d/d1a/d23/d38/dae/fd5 [0,4194304] 0 2024-04-05T21:26:53.828 INFO:tasks.workunit.client.0.smithi121.stdout:7/990: write d0/f6 [3920353,90397] 0 2024-04-05T21:26:53.829 INFO:tasks.workunit.client.0.smithi121.stdout:0/980: truncate d7/dc/d95/f122 246305 0 2024-04-05T21:26:53.830 INFO:tasks.workunit.client.0.smithi121.stdout:8/827: creat d12/d13/d56/d57/d5e/db9/f106 x:0 0 0 2024-04-05T21:26:53.832 INFO:tasks.workunit.client.0.smithi121.stdout:0/981: stat d7/d62/d8b/fed 0 2024-04-05T21:26:53.832 INFO:tasks.workunit.client.0.smithi121.stdout:6/937: dread d1/d2/daf/d6f/dc7/fb7 [0,4194304] 0 2024-04-05T21:26:53.834 INFO:tasks.workunit.client.0.smithi121.stdout:6/938: readlink d1/d2/d1b/d2e/lca 0 2024-04-05T21:26:53.835 INFO:tasks.workunit.client.0.smithi121.stdout:6/939: fsync d1/d2/d1b/d2e/dd9/f10a 0 2024-04-05T21:26:53.835 INFO:tasks.workunit.client.0.smithi121.stdout:3/862: unlink d0/d3/d24/d3d/d1a/d23/d90/d9e/f120 0 2024-04-05T21:26:53.838 INFO:tasks.workunit.client.0.smithi121.stdout:8/828: symlink d12/l107 0 2024-04-05T21:26:53.843 INFO:tasks.workunit.client.0.smithi121.stdout:8/829: mkdir d12/d13/d56/d57/da4/d108 0 2024-04-05T21:26:53.844 INFO:tasks.workunit.client.0.smithi121.stdout:3/863: write d0/d3/d24/d3d/d1a/d23/d90/d9e/fad [4490847,25518] 0 2024-04-05T21:26:53.846 INFO:tasks.workunit.client.0.smithi121.stdout:0/982: dwrite d7/d71/d29/d3f/f72 [0,4194304] 0 2024-04-05T21:26:53.848 INFO:tasks.workunit.client.0.smithi121.stdout:6/940: truncate d1/d2/daf/d6f/dc7/d1f/d6d/f12a 4337845 0 2024-04-05T21:26:53.848 INFO:tasks.workunit.client.0.smithi121.stdout:3/864: chown d0/d3/d24/d3d/d1a 922489378 1 2024-04-05T21:26:53.849 INFO:tasks.workunit.client.0.smithi121.stdout:6/941: fsync d1/d2/dc/d9a/dc0/fee 0 2024-04-05T21:26:53.851 INFO:tasks.workunit.client.0.smithi121.stdout:8/830: symlink d12/d50/dbb/de5/l109 0 2024-04-05T21:26:53.852 INFO:tasks.workunit.client.0.smithi121.stdout:0/983: creat d7/d62/d3c/f157 x:0 0 0 2024-04-05T21:26:53.854 INFO:tasks.workunit.client.0.smithi121.stdout:8/831: dread - d12/d13/d56/d57/d105/d87/deb/fd8 zero size 2024-04-05T21:26:53.854 INFO:tasks.workunit.client.0.smithi121.stdout:3/865: readlink d0/d3/d24/d3d/d1a/d23/leb 0 2024-04-05T21:26:53.856 INFO:tasks.workunit.client.0.smithi121.stdout:6/942: rename d1/d2/d1b/d2e/l7a to d1/d2/d1b/d2e/dd9/l14b 0 2024-04-05T21:26:53.858 INFO:tasks.workunit.client.0.smithi121.stdout:8/832: write d12/d50/fca [382541,125916] 0 2024-04-05T21:26:53.859 INFO:tasks.workunit.client.0.smithi121.stdout:0/984: creat d7/d62/d3c/d6b/f158 x:0 0 0 2024-04-05T21:26:53.860 INFO:tasks.workunit.client.1.smithi188.stdout:8/990: dread d5/d21/f91 [0,4194304] 0 2024-04-05T21:26:53.863 INFO:tasks.workunit.client.1.smithi188.stdout:8/991: mkdir d5/d21/d40/d43/d44/d5b/d99/da3/d13a 0 2024-04-05T21:26:53.863 INFO:tasks.workunit.client.0.smithi121.stdout:1/922: dread d8/d8c/fe4 [0,4194304] 0 2024-04-05T21:26:53.864 INFO:tasks.workunit.client.0.smithi121.stdout:3/866: write d0/d3/d24/d3d/d1a/d1e/fe5 [985425,61625] 0 2024-04-05T21:26:53.864 INFO:tasks.workunit.client.0.smithi121.stdout:6/943: truncate d1/d2/f120 1527599 0 2024-04-05T21:26:53.864 INFO:tasks.workunit.client.0.smithi121.stdout:1/923: chown d8/d89/d9e/dc8/f11f 16018819 1 2024-04-05T21:26:53.865 INFO:tasks.workunit.client.0.smithi121.stdout:8/833: write d12/d13/d56/d57/d5e/fff [821202,49423] 0 2024-04-05T21:26:53.866 INFO:tasks.workunit.client.0.smithi121.stdout:3/867: write d0/d3/d24/db3/dc3/f121 [227788,110425] 0 2024-04-05T21:26:53.867 INFO:tasks.workunit.client.0.smithi121.stdout:6/944: chown d1/d2/daf/d6f/dc7/d1f/cf6 7514 1 2024-04-05T21:26:53.869 INFO:tasks.workunit.client.0.smithi121.stdout:7/991: dread d0/d9/f86 [0,4194304] 0 2024-04-05T21:26:53.870 INFO:tasks.workunit.client.1.smithi188.stdout:8/992: dwrite d5/d21/d2a/f31 [0,4194304] 0 2024-04-05T21:26:53.870 INFO:tasks.workunit.client.0.smithi121.stdout:0/985: getdents d7/d71/d29/d2a 0 2024-04-05T21:26:53.871 INFO:tasks.workunit.client.0.smithi121.stdout:6/945: truncate d1/f13b 280954 0 2024-04-05T21:26:53.873 INFO:tasks.workunit.client.0.smithi121.stdout:8/834: unlink d12/d13/dd3/df0/da7/ff3 0 2024-04-05T21:26:53.879 INFO:tasks.workunit.client.1.smithi188.stdout:8/993: dwrite d5/d21/d40/d43/d44/d5b/d99/d69/d70/dae/f106 [0,4194304] 0 2024-04-05T21:26:53.881 INFO:tasks.workunit.client.0.smithi121.stdout:1/924: mkdir d8/d89/d9e/da3/df2/d12c 0 2024-04-05T21:26:53.883 INFO:tasks.workunit.client.0.smithi121.stdout:6/946: creat d1/d2/d1b/d35/d74/df3/d143/f14c x:0 0 0 2024-04-05T21:26:53.884 INFO:tasks.workunit.client.0.smithi121.stdout:8/835: mkdir d12/d13/d56/d57/d105/d63/d8b/dcc/d10a 0 2024-04-05T21:26:53.886 INFO:tasks.workunit.client.0.smithi121.stdout:0/986: creat d7/d71/d29/d2a/d39/f159 x:0 0 0 2024-04-05T21:26:53.889 INFO:tasks.workunit.client.1.smithi188.stdout:8/994: dwrite d5/d21/d40/d43/d44/d5b/d99/da3/ff2 [0,4194304] 0 2024-04-05T21:26:53.890 INFO:tasks.workunit.client.0.smithi121.stdout:6/947: mknod d1/d2/d1b/d35/d51/d105/c14d 0 2024-04-05T21:26:53.891 INFO:tasks.workunit.client.0.smithi121.stdout:3/868: dwrite d0/f7 [0,4194304] 0 2024-04-05T21:26:53.892 INFO:tasks.workunit.client.0.smithi121.stdout:0/987: mkdir d7/dc/da3/d15a 0 2024-04-05T21:26:53.895 INFO:tasks.workunit.client.0.smithi121.stdout:7/992: dread d0/d9/f12 [0,4194304] 0 2024-04-05T21:26:53.898 INFO:tasks.workunit.client.0.smithi121.stdout:3/869: truncate d0/dc6/f128 436544 0 2024-04-05T21:26:53.898 INFO:tasks.workunit.client.0.smithi121.stdout:0/988: symlink d7/d71/d94/dd1/l15b 0 2024-04-05T21:26:53.898 INFO:tasks.workunit.client.0.smithi121.stdout:1/925: dwrite d8/d89/d9e/f30 [4194304,4194304] 0 2024-04-05T21:26:53.900 INFO:tasks.workunit.client.0.smithi121.stdout:3/870: fdatasync d0/d3/d24/d3d/f57 0 2024-04-05T21:26:53.903 INFO:tasks.workunit.client.0.smithi121.stdout:0/989: creat d7/d62/d9d/f15c x:0 0 0 2024-04-05T21:26:53.903 INFO:tasks.workunit.client.0.smithi121.stdout:3/871: chown d0/d3/d24/d6d 281482 1 2024-04-05T21:26:53.905 INFO:tasks.workunit.client.0.smithi121.stdout:0/990: fdatasync d7/fda 0 2024-04-05T21:26:53.905 INFO:tasks.workunit.client.0.smithi121.stdout:6/948: dwrite d1/d4b/dcc/fe9 [0,4194304] 0 2024-04-05T21:26:53.909 INFO:tasks.workunit.client.0.smithi121.stdout:0/991: unlink d7/d62/d8b/cc5 0 2024-04-05T21:26:53.909 INFO:tasks.workunit.client.0.smithi121.stdout:7/993: link d0/d9/d22/d28/d3b/ca8 d0/d23/c139 0 2024-04-05T21:26:53.910 INFO:tasks.workunit.client.0.smithi121.stdout:7/994: chown d0/d32 1589715 1 2024-04-05T21:26:53.911 INFO:tasks.workunit.client.0.smithi121.stdout:6/949: mkdir d1/d2/dc/d9a/d14e 0 2024-04-05T21:26:53.913 INFO:tasks.workunit.client.0.smithi121.stdout:3/872: mkdir d0/d3/d24/d3d/d1a/d23/d38/d55/d103/d12d 0 2024-04-05T21:26:53.916 INFO:tasks.workunit.client.0.smithi121.stdout:6/950: mkdir d1/d2/d1b/d2e/dd9/d14f 0 2024-04-05T21:26:53.927 INFO:tasks.workunit.client.0.smithi121.stdout:3/873: dwrite d0/d3/d24/d3d/d1a/d1e/db1/fea [0,4194304] 0 2024-04-05T21:26:53.929 INFO:tasks.workunit.client.0.smithi121.stdout:6/951: dwrite d1/fdc [0,4194304] 0 2024-04-05T21:26:53.934 INFO:tasks.workunit.client.0.smithi121.stdout:3/874: rename d0/d3/d24/d6d/d7b/da3/dec/df6/d102 to d0/d3/d24/d3d/d1a/d23/d38/d55/d103/d12d/d12e 0 2024-04-05T21:26:53.935 INFO:tasks.workunit.client.0.smithi121.stdout:6/952: mknod d1/d2/daf/d12d/c150 0 2024-04-05T21:26:53.937 INFO:tasks.workunit.client.0.smithi121.stdout:0/992: dread d7/d7d/f81 [0,4194304] 0 2024-04-05T21:26:53.939 INFO:tasks.workunit.client.0.smithi121.stdout:3/875: creat d0/d3/d24/d3d/d1a/d1e/f12f x:0 0 0 2024-04-05T21:26:53.941 INFO:tasks.workunit.client.0.smithi121.stdout:3/876: chown d0/l79 121 1 2024-04-05T21:26:53.941 INFO:tasks.workunit.client.0.smithi121.stdout:1/926: dread d8/d8c/db6/d1a/d1b/d4c/d76/f3d [0,4194304] 0 2024-04-05T21:26:53.944 INFO:tasks.workunit.client.0.smithi121.stdout:7/995: write d0/d32/f6b [4334094,99892] 0 2024-04-05T21:26:53.947 INFO:tasks.workunit.client.1.smithi188.stdout:8/995: dwrite d5/dc/d8f/fb4 [0,4194304] 0 2024-04-05T21:26:53.948 INFO:tasks.workunit.client.0.smithi121.stdout:1/927: dread d8/d8c/db6/d1a/d1b/d24/fe9 [0,4194304] 0 2024-04-05T21:26:53.948 INFO:tasks.workunit.client.1.smithi188.stdout:8/996: chown d5/dc/d8f/fb4 15938 1 2024-04-05T21:26:53.949 INFO:tasks.workunit.client.0.smithi121.stdout:0/993: mkdir d7/d71/d45/d46/d53/d143/d15d 0 2024-04-05T21:26:53.950 INFO:tasks.workunit.client.0.smithi121.stdout:1/928: fdatasync d8/d8c/db6/d1a/d1b/d4c/d76/da4/fdf 0 2024-04-05T21:26:53.952 INFO:tasks.workunit.client.0.smithi121.stdout:7/996: creat d0/d9/d22/d28/d56/db9/ddf/f13a x:0 0 0 2024-04-05T21:26:53.954 INFO:tasks.workunit.client.0.smithi121.stdout:1/929: symlink d8/d8c/db6/d1a/d1b/d44/l12d 0 2024-04-05T21:26:53.954 INFO:tasks.workunit.client.1.smithi188.stdout:8/997: read d5/dc/f1d [824818,2987] 0 2024-04-05T21:26:53.956 INFO:tasks.workunit.client.0.smithi121.stdout:1/930: chown d8/d8c/db6/d1a/d1b/d4c/d76/da4 1551904053 1 2024-04-05T21:26:53.956 INFO:tasks.workunit.client.0.smithi121.stdout:0/994: dread d7/dc/da3/fc1 [0,4194304] 0 2024-04-05T21:26:53.958 INFO:tasks.workunit.client.1.smithi188.stdout:8/998: creat d5/d21/d40/d43/d48/d7e/d10a/f13b x:0 0 0 2024-04-05T21:26:53.959 INFO:tasks.workunit.client.0.smithi121.stdout:1/931: creat d8/d89/d9e/d57/f12e x:0 0 0 2024-04-05T21:26:53.960 INFO:tasks.workunit.client.0.smithi121.stdout:0/995: creat d7/d71/d45/d46/d53/d8e/f15e x:0 0 0 2024-04-05T21:26:53.961 INFO:tasks.workunit.client.0.smithi121.stdout:1/932: mknod d8/d10/c12f 0 2024-04-05T21:26:53.963 INFO:tasks.workunit.client.0.smithi121.stdout:1/933: write d8/d89/d9e/d2d/d84/da8/f113 [836791,25859] 0 2024-04-05T21:26:53.969 INFO:tasks.workunit.client.0.smithi121.stdout:6/953: truncate d1/d2/d3/d4/d21/f93 894597 0 2024-04-05T21:26:53.969 INFO:tasks.workunit.client.0.smithi121.stdout:1/934: creat d8/d10/f130 x:0 0 0 2024-04-05T21:26:53.974 INFO:tasks.workunit.client.0.smithi121.stdout:6/954: mkdir d1/d2/daf/d12d/d151 0 2024-04-05T21:26:53.974 INFO:tasks.workunit.client.0.smithi121.stdout:7/997: dread d0/d9/d22/d81/da2/fbd [0,4194304] 0 2024-04-05T21:26:53.975 INFO:tasks.workunit.client.0.smithi121.stdout:1/935: symlink d8/d89/d9e/l131 0 2024-04-05T21:26:53.976 INFO:tasks.workunit.client.0.smithi121.stdout:6/955: chown d1/d2/d3/d4/d21/d57/d42/d48 7 1 2024-04-05T21:26:53.978 INFO:tasks.workunit.client.0.smithi121.stdout:7/998: mknod d0/d32/d4e/d73/dbc/c13b 0 2024-04-05T21:26:53.981 INFO:tasks.workunit.client.1.smithi188.stdout:8/999: dread d5/d21/d40/d43/d44/d59/d8b/fb6 [0,4194304] 0 2024-04-05T21:26:53.981 INFO:tasks.workunit.client.0.smithi121.stdout:3/877: dread d0/f27 [0,4194304] 0 2024-04-05T21:26:53.981 INFO:tasks.workunit.client.0.smithi121.stdout:6/956: creat d1/d2/daf/d12d/d151/f152 x:0 0 0 2024-04-05T21:26:53.982 INFO:tasks.workunit.client.0.smithi121.stdout:0/996: dwrite d7/d62/f20 [0,4194304] 0 2024-04-05T21:26:53.983 INFO:tasks.workunit.client.0.smithi121.stdout:0/997: chown d7/dc/da3/d15a 1713668 1 2024-04-05T21:26:53.985 INFO:tasks.workunit.client.0.smithi121.stdout:1/936: rename d8/d8c/db6/d1a/l12a to d8/d8c/db6/d1a/d1b/d38/l132 0 2024-04-05T21:26:53.985 INFO:tasks.workunit.client.1.smithi188.stderr:+ rm -rf -- ./tmp.2FUf7WJ9nl 2024-04-05T21:26:53.992 INFO:tasks.workunit.client.0.smithi121.stdout:3/878: dwrite d0/d3/d24/d3d/d1a/d23/d38/dae/fd5 [0,4194304] 0 2024-04-05T21:26:53.994 INFO:tasks.workunit.client.0.smithi121.stdout:1/937: unlink d8/d8c/db6/d1a/d1b/d4c/da1/cc4 0 2024-04-05T21:26:53.994 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:53 smithi121 ceph-mon[39723]: pgmap v25: 65 pgs: 65 active+clean; 3.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 106 MiB/s rd, 257 MiB/s wr, 544 op/s 2024-04-05T21:26:53.995 INFO:tasks.workunit.client.0.smithi121.stdout:1/938: truncate d8/d10/f130 946267 0 2024-04-05T21:26:53.998 INFO:tasks.workunit.client.0.smithi121.stdout:7/999: rename d0/d32/d4e to d0/d9/d22/d28/d56/dff/d13c 0 2024-04-05T21:26:53.998 INFO:tasks.workunit.client.0.smithi121.stdout:6/957: dread d1/d2/d1b/d91/f9f [0,4194304] 0 2024-04-05T21:26:54.001 INFO:tasks.workunit.client.0.smithi121.stdout:1/939: creat d8/d8c/db6/d1a/d1b/d24/f133 x:0 0 0 2024-04-05T21:26:54.006 INFO:tasks.workunit.client.0.smithi121.stdout:0/998: truncate d7/d62/f20 1317398 0 2024-04-05T21:26:54.007 INFO:tasks.workunit.client.0.smithi121.stdout:1/940: unlink d8/d8c/da2/c86 0 2024-04-05T21:26:54.008 INFO:tasks.workunit.client.0.smithi121.stdout:0/999: rmdir d7/d71/d29 39 2024-04-05T21:26:54.011 INFO:tasks.workunit.client.0.smithi121.stdout:3/879: dread d0/d3/f4 [0,4194304] 0 2024-04-05T21:26:54.013 INFO:tasks.workunit.client.0.smithi121.stdout:6/958: truncate d1/d2/daf/d6f/f107 1435118 0 2024-04-05T21:26:54.018 INFO:tasks.workunit.client.0.smithi121.stdout:3/880: dwrite d0/d3/d24/db3/dc3/f116 [0,4194304] 0 2024-04-05T21:26:54.023 INFO:tasks.workunit.client.0.smithi121.stdout:3/881: write d0/d3/d24/d3d/d1a/d1e/d86/f8b [3400689,81466] 0 2024-04-05T21:26:54.023 INFO:tasks.workunit.client.0.smithi121.stdout:6/959: dwrite d1/d2/d1b/d35/d51/f146 [0,4194304] 0 2024-04-05T21:26:54.024 INFO:tasks.workunit.client.0.smithi121.stdout:3/882: stat d0/d3/d24/d3d/d1a/d23/la7 0 2024-04-05T21:26:54.026 INFO:tasks.workunit.client.0.smithi121.stdout:6/960: creat d1/d2/d1b/d35/d39/f153 x:0 0 0 2024-04-05T21:26:54.027 INFO:tasks.workunit.client.0.smithi121.stdout:3/883: mknod d0/d3/d24/d3d/d1a/d1e/c130 0 2024-04-05T21:26:54.028 INFO:tasks.workunit.client.0.smithi121.stdout:3/884: read - d0/d3/d24/d3d/d1a/d1e/f12f zero size 2024-04-05T21:26:54.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:53 smithi188 ceph-mon[49715]: pgmap v25: 65 pgs: 65 active+clean; 3.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 106 MiB/s rd, 257 MiB/s wr, 544 op/s 2024-04-05T21:26:54.031 INFO:tasks.workunit.client.0.smithi121.stdout:6/961: rename d1/d2/d1b/d5e/l5f to d1/d2/dc/d9a/d14e/l154 0 2024-04-05T21:26:54.032 INFO:tasks.workunit.client.0.smithi121.stdout:9/688: sync 2024-04-05T21:26:54.033 INFO:tasks.workunit.client.0.smithi121.stdout:9/689: fsync d6/d32/fcf 0 2024-04-05T21:26:54.035 INFO:tasks.workunit.client.0.smithi121.stdout:3/885: dread d0/d3/d24/db3/dc3/f121 [0,4194304] 0 2024-04-05T21:26:54.035 INFO:tasks.workunit.client.0.smithi121.stdout:6/962: mkdir d1/d2/d155 0 2024-04-05T21:26:54.036 INFO:tasks.workunit.client.0.smithi121.stdout:6/963: fsync d1/ff4 0 2024-04-05T21:26:54.039 INFO:tasks.workunit.client.0.smithi121.stdout:9/690: unlink d6/d32/dbe/d7b/d49/d66/fe8 0 2024-04-05T21:26:54.043 INFO:tasks.workunit.client.0.smithi121.stdout:1/941: dread d8/d8c/db6/d1a/d1b/d85/fa0 [0,4194304] 0 2024-04-05T21:26:54.046 INFO:tasks.workunit.client.0.smithi121.stdout:6/964: dread d1/d2/f120 [0,4194304] 0 2024-04-05T21:26:54.047 INFO:tasks.workunit.client.0.smithi121.stdout:9/691: creat d6/d46/d8a/dd5/feb x:0 0 0 2024-04-05T21:26:54.049 INFO:tasks.workunit.client.0.smithi121.stdout:3/886: getdents d0/d3/d24/d6d/dff 0 2024-04-05T21:26:54.049 INFO:tasks.workunit.client.0.smithi121.stdout:3/887: chown d0/c60 7 1 2024-04-05T21:26:54.049 INFO:tasks.workunit.client.0.smithi121.stdout:9/692: creat d6/df/d48/fec x:0 0 0 2024-04-05T21:26:54.053 INFO:tasks.workunit.client.0.smithi121.stdout:3/888: creat d0/d3/d24/db3/dc3/f131 x:0 0 0 2024-04-05T21:26:54.054 INFO:tasks.workunit.client.0.smithi121.stdout:6/965: link d1/d2/daf/d12d/c150 d1/d2/daf/d6f/d126/c156 0 2024-04-05T21:26:54.055 INFO:tasks.workunit.client.0.smithi121.stdout:6/966: fdatasync d1/d2/d1b/d35/d39/d92/fe3 0 2024-04-05T21:26:54.055 INFO:tasks.workunit.client.0.smithi121.stdout:9/693: mkdir d6/df/d48/d96/dc2/ded 0 2024-04-05T21:26:54.057 INFO:tasks.workunit.client.0.smithi121.stdout:3/889: creat d0/dc6/f132 x:0 0 0 2024-04-05T21:26:54.057 INFO:tasks.workunit.client.0.smithi121.stdout:6/967: dread - d1/d2/d1b/d35/d39/f153 zero size 2024-04-05T21:26:54.058 INFO:tasks.workunit.client.0.smithi121.stdout:9/694: creat d6/d32/fee x:0 0 0 2024-04-05T21:26:54.060 INFO:tasks.workunit.client.0.smithi121.stdout:3/890: symlink d0/d3/d24/d6d/d7b/da4/l133 0 2024-04-05T21:26:54.061 INFO:tasks.workunit.client.0.smithi121.stdout:6/968: fsync d1/d2/d1b/d2e/dd9/de0/fe6 0 2024-04-05T21:26:54.062 INFO:tasks.workunit.client.0.smithi121.stdout:9/695: chown d6/df/d48/d96/dc2/d85/ld3 6243636 1 2024-04-05T21:26:54.063 INFO:tasks.workunit.client.0.smithi121.stdout:9/696: read - d6/df/d48/d96/dc2/fcb zero size 2024-04-05T21:26:54.064 INFO:tasks.workunit.client.0.smithi121.stdout:3/891: rename d0/d3/de/ca5 to d0/d3/d24/d3d/d1a/d23/d90/d9e/c134 0 2024-04-05T21:26:54.065 INFO:tasks.workunit.client.0.smithi121.stdout:1/942: write d8/d89/d9e/d2d/fd1 [593156,21828] 0 2024-04-05T21:26:54.065 INFO:tasks.workunit.client.0.smithi121.stdout:6/969: symlink d1/d2/d3/d4/df0/d11a/l157 0 2024-04-05T21:26:54.067 INFO:tasks.workunit.client.0.smithi121.stdout:1/943: chown d8/d8c/db6/d1a/d1b/d24/l90 581 1 2024-04-05T21:26:54.070 INFO:tasks.workunit.client.0.smithi121.stdout:1/944: unlink d8/d89/d9e/d2d/d84/f127 0 2024-04-05T21:26:54.071 INFO:tasks.workunit.client.0.smithi121.stdout:1/945: chown d8/d8c/db6/d1a/d1b/fc0 10 1 2024-04-05T21:26:54.074 INFO:tasks.workunit.client.0.smithi121.stdout:1/946: creat d8/d8c/db6/d1a/d1b/d4c/d76/d53/f134 x:0 0 0 2024-04-05T21:26:54.078 INFO:tasks.workunit.client.0.smithi121.stdout:1/947: link d8/d89/d9e/d57/d77/ced d8/d8c/db6/d1a/d1b/d4c/c135 0 2024-04-05T21:26:54.080 INFO:tasks.workunit.client.0.smithi121.stdout:1/948: write d8/d8c/db6/d1a/d1b/d4c/f63 [3440067,98786] 0 2024-04-05T21:26:54.085 INFO:tasks.workunit.client.0.smithi121.stdout:6/970: truncate d1/fdc 2074179 0 2024-04-05T21:26:54.090 INFO:tasks.workunit.client.0.smithi121.stdout:6/971: mknod d1/d2/d1b/d2e/c158 0 2024-04-05T21:26:54.094 INFO:tasks.workunit.client.0.smithi121.stdout:3/892: dwrite d0/d3/de/f6a [0,4194304] 0 2024-04-05T21:26:54.100 INFO:tasks.workunit.client.0.smithi121.stdout:6/972: dread - d1/d2/d1b/d35/d74/df3/f11d zero size 2024-04-05T21:26:54.100 INFO:tasks.workunit.client.0.smithi121.stdout:1/949: dwrite d8/d8c/db6/d1a/d1b/d4c/f4f [0,4194304] 0 2024-04-05T21:26:54.101 INFO:tasks.workunit.client.0.smithi121.stdout:9/697: dwrite d6/df/d48/f7e [0,4194304] 0 2024-04-05T21:26:54.105 INFO:tasks.workunit.client.0.smithi121.stdout:3/893: creat d0/d3/d24/d6d/d7b/da3/dec/f135 x:0 0 0 2024-04-05T21:26:54.106 INFO:tasks.workunit.client.0.smithi121.stdout:3/894: dread - d0/dc6/f11a zero size 2024-04-05T21:26:54.107 INFO:tasks.workunit.client.0.smithi121.stdout:9/698: mkdir d6/d32/dbe/dba/def 0 2024-04-05T21:26:54.108 INFO:tasks.workunit.client.0.smithi121.stdout:1/950: creat d8/d8c/db6/d1a/d1b/d24/dc2/f136 x:0 0 0 2024-04-05T21:26:54.111 INFO:tasks.workunit.client.0.smithi121.stdout:9/699: mknod d6/d46/d8a/cf0 0 2024-04-05T21:26:54.112 INFO:tasks.workunit.client.0.smithi121.stdout:1/951: chown d8/d89/d9e/d57/d77/dd6 2 1 2024-04-05T21:26:54.112 INFO:tasks.workunit.client.0.smithi121.stdout:6/973: dwrite d1/d2/d3/f148 [0,4194304] 0 2024-04-05T21:26:54.114 INFO:tasks.workunit.client.0.smithi121.stdout:3/895: unlink d0/d3/d24/d6d/d7b/da3/dec/df6/f4b 0 2024-04-05T21:26:54.114 INFO:tasks.workunit.client.0.smithi121.stdout:2/928: sync 2024-04-05T21:26:54.115 INFO:tasks.workunit.client.0.smithi121.stdout:9/700: truncate d6/df/f5f 3642399 0 2024-04-05T21:26:54.116 INFO:tasks.workunit.client.0.smithi121.stdout:6/974: mkdir d1/d2/d3/d4/d159 0 2024-04-05T21:26:54.117 INFO:tasks.workunit.client.0.smithi121.stdout:9/701: readlink d6/df/d48/d96/db9/d2c/d37/l82 0 2024-04-05T21:26:54.118 INFO:tasks.workunit.client.0.smithi121.stdout:1/952: mknod d8/d8c/db6/d1a/d1b/d44/c137 0 2024-04-05T21:26:54.120 INFO:tasks.workunit.client.0.smithi121.stdout:2/929: dread d2/d9/d38/d2d/d4c/de9/d8d/fd6 [0,4194304] 0 2024-04-05T21:26:54.122 INFO:tasks.workunit.client.0.smithi121.stdout:3/896: symlink d0/d3/d24/d6d/l136 0 2024-04-05T21:26:54.124 INFO:tasks.workunit.client.0.smithi121.stdout:6/975: fsync d1/d2/d3/d4/d21/d57/d42/f83 0 2024-04-05T21:26:54.127 INFO:tasks.workunit.client.0.smithi121.stdout:3/897: symlink d0/dc6/l137 0 2024-04-05T21:26:54.132 INFO:tasks.workunit.client.0.smithi121.stdout:6/976: dwrite d1/d2/f118 [0,4194304] 0 2024-04-05T21:26:54.136 INFO:tasks.workunit.client.0.smithi121.stdout:3/898: symlink d0/d3/de/da6/df5/l138 0 2024-04-05T21:26:54.137 INFO:tasks.workunit.client.0.smithi121.stdout:3/899: stat d0/d3/de/f72 0 2024-04-05T21:26:54.138 INFO:tasks.workunit.client.0.smithi121.stdout:9/702: write d6/df/f6d [74723,120293] 0 2024-04-05T21:26:54.139 INFO:tasks.workunit.client.0.smithi121.stdout:2/930: link d2/d9/d1c/c120 d2/d9/d38/d2d/d4c/d7c/c12d 0 2024-04-05T21:26:54.139 INFO:tasks.workunit.client.0.smithi121.stdout:6/977: write d1/d2/d3/f104 [2138372,104231] 0 2024-04-05T21:26:54.140 INFO:tasks.workunit.client.0.smithi121.stdout:1/953: link d8/d89/d9e/d2d/f125 d8/d89/d9e/d2d/d84/da8/dad/dde/ddb/d115/f138 0 2024-04-05T21:26:54.143 INFO:tasks.workunit.client.0.smithi121.stdout:2/931: write d2/d28/f4f [2815899,30764] 0 2024-04-05T21:26:54.144 INFO:tasks.workunit.client.0.smithi121.stdout:9/703: mkdir d6/df/d48/d96/db9/d72/df1 0 2024-04-05T21:26:54.144 INFO:tasks.workunit.client.0.smithi121.stdout:1/954: write d8/d89/d9e/d57/f7f [8106692,19710] 0 2024-04-05T21:26:54.148 INFO:tasks.workunit.client.0.smithi121.stdout:6/978: rename d1/d2/dc/d9a/dc0/le8 to d1/d2/daf/d12d/l15a 0 2024-04-05T21:26:54.150 INFO:tasks.workunit.client.0.smithi121.stdout:1/955: creat d8/d89/d9e/d2d/d84/da8/d11d/f139 x:0 0 0 2024-04-05T21:26:54.151 INFO:tasks.workunit.client.0.smithi121.stdout:9/704: rename d6/df/d48/d96/dc2/d85/ld3 to d6/df/d48/d96/db2/lf2 0 2024-04-05T21:26:54.152 INFO:tasks.workunit.client.0.smithi121.stdout:2/932: mknod d2/d9/d38/d2d/d47/d117/c12e 0 2024-04-05T21:26:54.154 INFO:tasks.workunit.client.0.smithi121.stdout:9/705: fsync d6/f17 0 2024-04-05T21:26:54.154 INFO:tasks.workunit.client.0.smithi121.stdout:6/979: truncate d1/d2/daf/d6f/dc7/d1f/d6d/d101/f111 60484 0 2024-04-05T21:26:54.155 INFO:tasks.workunit.client.0.smithi121.stdout:1/956: creat d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b/f13a x:0 0 0 2024-04-05T21:26:54.156 INFO:tasks.workunit.client.0.smithi121.stdout:9/706: stat d6/df/d48/d96/db9/d72/l74 0 2024-04-05T21:26:54.159 INFO:tasks.workunit.client.0.smithi121.stdout:1/957: fdatasync d8/d8c/db6/d1a/d1b/d85/ff8 0 2024-04-05T21:26:54.159 INFO:tasks.workunit.client.0.smithi121.stdout:6/980: symlink d1/d2/d1b/d35/d51/d105/l15b 0 2024-04-05T21:26:54.159 INFO:tasks.workunit.client.0.smithi121.stdout:9/707: dread - d6/d46/d8a/dd5/feb zero size 2024-04-05T21:26:54.159 INFO:tasks.workunit.client.0.smithi121.stdout:3/900: dread d0/d3/d24/d3d/d1a/d23/d38/f65 [0,4194304] 0 2024-04-05T21:26:54.160 INFO:tasks.workunit.client.0.smithi121.stdout:1/958: chown d8/d8c/db6/d1a/d1b/d85 21569 1 2024-04-05T21:26:54.161 INFO:tasks.workunit.client.0.smithi121.stdout:2/933: rename d2/d1e/f24 to d2/d9/d38/d2d/d4c/de9/d5c/df0/f12f 0 2024-04-05T21:26:54.161 INFO:tasks.workunit.client.0.smithi121.stdout:9/708: dread - d6/df/d48/d96/dc2/d85/fe7 zero size 2024-04-05T21:26:54.163 INFO:tasks.workunit.client.0.smithi121.stdout:6/981: write d1/d2/daf/d6f/f11c [869420,95139] 0 2024-04-05T21:26:54.163 INFO:tasks.workunit.client.0.smithi121.stdout:8/836: sync 2024-04-05T21:26:54.165 INFO:tasks.workunit.client.0.smithi121.stdout:1/959: mkdir d8/d8c/db6/d1a/d1b/d4c/da1/d13b 0 2024-04-05T21:26:54.166 INFO:tasks.workunit.client.0.smithi121.stdout:2/934: mkdir d2/d9/d38/d2d/d4c/d7c/d7e/db9/d130 0 2024-04-05T21:26:54.168 INFO:tasks.workunit.client.0.smithi121.stdout:9/709: symlink d6/df/d48/d96/db9/lf3 0 2024-04-05T21:26:54.170 INFO:tasks.workunit.client.0.smithi121.stdout:6/982: mkdir d1/d2/d3/d4/df0/d11a/d15c 0 2024-04-05T21:26:54.171 INFO:tasks.workunit.client.0.smithi121.stdout:8/837: symlink d12/d13/dd3/df0/da7/de1/l10b 0 2024-04-05T21:26:54.172 INFO:tasks.workunit.client.0.smithi121.stdout:1/960: truncate d8/d89/d9e/d57/d77/dd6/fd8 31151 0 2024-04-05T21:26:54.173 INFO:tasks.workunit.client.0.smithi121.stdout:1/961: dread - d8/f112 zero size 2024-04-05T21:26:54.174 INFO:tasks.workunit.client.0.smithi121.stdout:9/710: chown d6/d32/dbe/d7b/d6b/f92 324 1 2024-04-05T21:26:54.175 INFO:tasks.workunit.client.0.smithi121.stdout:9/711: write d6/f9b [4931529,21227] 0 2024-04-05T21:26:54.177 INFO:tasks.workunit.client.0.smithi121.stdout:8/838: mknod d12/d13/d56/d57/d105/d63/d9c/c10c 0 2024-04-05T21:26:54.178 INFO:tasks.workunit.client.0.smithi121.stdout:1/962: dread d8/d89/d9e/dc8/f11f [0,4194304] 0 2024-04-05T21:26:54.180 INFO:tasks.workunit.client.0.smithi121.stdout:6/983: mkdir d1/d2/d1b/d91/d13a/d15d 0 2024-04-05T21:26:54.181 INFO:tasks.workunit.client.0.smithi121.stdout:9/712: fdatasync d6/df/d48/d96/db9/d72/f7f 0 2024-04-05T21:26:54.182 INFO:tasks.workunit.client.0.smithi121.stdout:2/935: creat d2/d9/d38/f131 x:0 0 0 2024-04-05T21:26:54.185 INFO:tasks.workunit.client.0.smithi121.stdout:3/901: dwrite d0/d3/d24/d3d/dcf/ff3 [0,4194304] 0 2024-04-05T21:26:54.185 INFO:tasks.workunit.client.0.smithi121.stdout:6/984: readlink d1/d2/d3/d4/d21/d57/d42/la5 0 2024-04-05T21:26:54.193 INFO:tasks.workunit.client.0.smithi121.stdout:8/839: creat d12/d50/d93/d96/f10d x:0 0 0 2024-04-05T21:26:54.194 INFO:tasks.workunit.client.0.smithi121.stdout:1/963: rmdir d8/d89/d9e/da3/df2/d12c 0 2024-04-05T21:26:54.194 INFO:tasks.workunit.client.0.smithi121.stdout:2/936: dwrite d2/d9/d1c/d30/d59/f89 [4194304,4194304] 0 2024-04-05T21:26:54.196 INFO:tasks.workunit.client.0.smithi121.stdout:3/902: unlink d0/d3/d24/db3/dc3/f116 0 2024-04-05T21:26:54.199 INFO:tasks.workunit.client.0.smithi121.stdout:1/964: write d8/d8c/db6/d1a/d1b/d85/ff8 [302042,5255] 0 2024-04-05T21:26:54.203 INFO:tasks.workunit.client.0.smithi121.stdout:2/937: rename d2/d28 to d2/db6/dbb/dee/d132 0 2024-04-05T21:26:54.204 INFO:tasks.workunit.client.0.smithi121.stdout:3/903: fsync d0/d3/d24/d3d/d1a/d23/d38/d55/d103/d12d/d12e/f10b 0 2024-04-05T21:26:54.206 INFO:tasks.workunit.client.0.smithi121.stdout:1/965: creat d8/d89/d9e/d2d/d84/da8/d11d/f13c x:0 0 0 2024-04-05T21:26:54.208 INFO:tasks.workunit.client.0.smithi121.stdout:2/938: creat d2/db6/dbb/dee/d132/d11d/f133 x:0 0 0 2024-04-05T21:26:54.208 INFO:tasks.workunit.client.0.smithi121.stdout:1/966: write d8/d8c/db6/d1a/d1b/d44/f106 [832587,4426] 0 2024-04-05T21:26:54.210 INFO:tasks.workunit.client.0.smithi121.stdout:2/939: truncate d2/d9/d77/fd2 392766 0 2024-04-05T21:26:54.211 INFO:tasks.workunit.client.0.smithi121.stdout:3/904: rmdir d0/d3/d24/d6d 39 2024-04-05T21:26:54.212 INFO:tasks.workunit.client.0.smithi121.stdout:8/840: getdents d12/d50/d93 0 2024-04-05T21:26:54.213 INFO:tasks.workunit.client.0.smithi121.stdout:8/841: chown d12/d50/dbb 3052684 1 2024-04-05T21:26:54.214 INFO:tasks.workunit.client.0.smithi121.stdout:1/967: mkdir d8/d89/d9e/da3/d13d 0 2024-04-05T21:26:54.217 INFO:tasks.workunit.client.0.smithi121.stdout:2/940: creat d2/d9/d38/d2d/d47/d94/d84/f134 x:0 0 0 2024-04-05T21:26:54.217 INFO:tasks.workunit.client.0.smithi121.stdout:3/905: symlink d0/d3/d24/d3d/d1a/d23/d90/l139 0 2024-04-05T21:26:54.219 INFO:tasks.workunit.client.0.smithi121.stdout:8/842: truncate d12/d13/d56/d57/f7b 2173415 0 2024-04-05T21:26:54.221 INFO:tasks.workunit.client.0.smithi121.stdout:9/713: truncate d6/df/d48/d96/db9/d2c/d37/d3c/fde 48946 0 2024-04-05T21:26:54.224 INFO:tasks.workunit.client.0.smithi121.stdout:8/843: mknod d12/d13/d15/de7/c10e 0 2024-04-05T21:26:54.225 INFO:tasks.workunit.client.0.smithi121.stdout:1/968: mknod d8/d8c/db6/d1a/d1b/d4c/d76/c13e 0 2024-04-05T21:26:54.226 INFO:tasks.workunit.client.0.smithi121.stdout:1/969: chown d8/d8c/db6/d1a/d1b/d44/ca6 54 1 2024-04-05T21:26:54.226 INFO:tasks.workunit.client.0.smithi121.stdout:2/941: rename d2/d9/d38/d2d/d4c/de9/d62/c11b to d2/d9/d38/d2d/d4c/d7c/dbc/c135 0 2024-04-05T21:26:54.227 INFO:tasks.workunit.client.0.smithi121.stdout:2/942: dread - d2/ff2 zero size 2024-04-05T21:26:54.230 INFO:tasks.workunit.client.0.smithi121.stdout:1/970: readlink d8/d8c/db6/l25 0 2024-04-05T21:26:54.232 INFO:tasks.workunit.client.0.smithi121.stdout:3/906: write d0/d3/d24/d3d/d1a/d23/d38/d55/fa9 [624507,83411] 0 2024-04-05T21:26:54.233 INFO:tasks.workunit.client.0.smithi121.stdout:9/714: fsync d6/df/d48/d96/db9/d2c/d37/d3c/fde 0 2024-04-05T21:26:54.234 INFO:tasks.workunit.client.0.smithi121.stdout:2/943: creat d2/d9/d38/d2d/d4c/f136 x:0 0 0 2024-04-05T21:26:54.235 INFO:tasks.workunit.client.0.smithi121.stdout:2/944: readlink d2/d9/d38/d2d/le0 0 2024-04-05T21:26:54.238 INFO:tasks.workunit.client.0.smithi121.stdout:8/844: rename l9 to d12/d13/dd3/df0/da7/l10f 0 2024-04-05T21:26:54.241 INFO:tasks.workunit.client.0.smithi121.stdout:9/715: unlink d6/df/d48/d96/db9/lf3 0 2024-04-05T21:26:54.242 INFO:tasks.workunit.client.0.smithi121.stdout:9/716: chown d6/d32/fee 318025 1 2024-04-05T21:26:54.245 INFO:tasks.workunit.client.0.smithi121.stdout:3/907: creat d0/d3/d24/d6d/d7b/f13a x:0 0 0 2024-04-05T21:26:54.246 INFO:tasks.workunit.client.0.smithi121.stdout:3/908: dread - d0/d3/d24/d6d/d7b/da3/dec/f135 zero size 2024-04-05T21:26:54.247 INFO:tasks.workunit.client.0.smithi121.stdout:1/971: creat d8/d8c/db6/d1a/d1b/d4c/d76/d53/f13f x:0 0 0 2024-04-05T21:26:54.248 INFO:tasks.workunit.client.0.smithi121.stdout:3/909: truncate d0/d3/d24/d6d/d7b/f13a 710896 0 2024-04-05T21:26:54.248 INFO:tasks.workunit.client.0.smithi121.stdout:2/945: write d2/d9/f75 [358921,55622] 0 2024-04-05T21:26:54.249 INFO:tasks.workunit.client.0.smithi121.stdout:8/845: mknod d12/d13/d56/d57/d105/c110 0 2024-04-05T21:26:54.250 INFO:tasks.workunit.client.0.smithi121.stdout:2/946: stat d2/d9/d38/d2d/d4c/fa6 0 2024-04-05T21:26:54.253 INFO:tasks.workunit.client.0.smithi121.stdout:3/910: mkdir d0/d3/d24/d3d/d1a/d23/d13b 0 2024-04-05T21:26:54.254 INFO:tasks.workunit.client.0.smithi121.stdout:1/972: symlink d8/d89/d9e/d2d/d84/dc9/l140 0 2024-04-05T21:26:54.254 INFO:tasks.workunit.client.0.smithi121.stdout:9/717: link d6/df/d48/d96/db9/d72/l74 d6/df/d48/d96/db9/d72/df1/lf4 0 2024-04-05T21:26:54.255 INFO:tasks.workunit.client.0.smithi121.stdout:1/973: write d8/d8c/db6/d1a/d1b/dbb/f117 [551656,76021] 0 2024-04-05T21:26:54.259 INFO:tasks.workunit.client.0.smithi121.stdout:9/718: fsync d6/df/d48/d96/dc2/d79/faf 0 2024-04-05T21:26:54.260 INFO:tasks.workunit.client.0.smithi121.stdout:1/974: rmdir d8/d8c/da2 39 2024-04-05T21:26:54.263 INFO:tasks.workunit.client.0.smithi121.stdout:3/911: mkdir d0/d3/d24/d3d/d13c 0 2024-04-05T21:26:54.264 INFO:tasks.workunit.client.0.smithi121.stdout:2/947: write d2/d9/d1c/d30/d59/fbe [4060696,55084] 0 2024-04-05T21:26:54.267 INFO:tasks.workunit.client.0.smithi121.stdout:1/975: read d8/d89/d9e/d2d/d84/da8/dad/dde/f6d [3611910,6907] 0 2024-04-05T21:26:54.267 INFO:tasks.workunit.client.0.smithi121.stdout:8/846: dwrite d12/d13/d56/d57/d105/fb8 [0,4194304] 0 2024-04-05T21:26:54.273 INFO:tasks.workunit.client.0.smithi121.stdout:1/976: write d8/d89/d9e/d2d/d84/da8/d11d/f13c [156317,67648] 0 2024-04-05T21:26:54.277 INFO:tasks.workunit.client.0.smithi121.stdout:9/719: dwrite d6/df/d48/d96/dc2/d79/faf [0,4194304] 0 2024-04-05T21:26:54.278 INFO:tasks.workunit.client.0.smithi121.stdout:1/977: stat d8/d89/d9e/d57/d77/dd6/fd8 0 2024-04-05T21:26:54.281 INFO:tasks.workunit.client.0.smithi121.stdout:2/948: dwrite d2/d9/d1c/d30/d59/f63 [0,4194304] 0 2024-04-05T21:26:54.286 INFO:tasks.workunit.client.0.smithi121.stdout:9/720: mkdir d6/df/d48/d96/db9/d2c/d37/df5 0 2024-04-05T21:26:54.286 INFO:tasks.workunit.client.0.smithi121.stdout:1/978: creat d8/d8c/db6/d1a/d1b/d44/f141 x:0 0 0 2024-04-05T21:26:54.287 INFO:tasks.workunit.client.0.smithi121.stdout:1/979: dread - d8/f112 zero size 2024-04-05T21:26:54.292 INFO:tasks.workunit.client.0.smithi121.stdout:2/949: dread d2/d9/d1c/d30/feb [0,4194304] 0 2024-04-05T21:26:54.293 INFO:tasks.workunit.client.0.smithi121.stdout:1/980: rename d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b/fd2 to d8/d8c/db6/d1a/d1b/d24/d79/f142 0 2024-04-05T21:26:54.294 INFO:tasks.workunit.client.0.smithi121.stdout:2/950: dread - d2/d9/d1c/d85/f12a zero size 2024-04-05T21:26:54.294 INFO:tasks.workunit.client.0.smithi121.stdout:3/912: dread d0/d3/d24/d3d/f7e [0,4194304] 0 2024-04-05T21:26:54.295 INFO:tasks.workunit.client.0.smithi121.stdout:9/721: creat d6/ff6 x:0 0 0 2024-04-05T21:26:54.297 INFO:tasks.workunit.client.0.smithi121.stdout:1/981: symlink d8/d8c/l143 0 2024-04-05T21:26:54.297 INFO:tasks.workunit.client.0.smithi121.stdout:3/913: fdatasync d0/d3/fc5 0 2024-04-05T21:26:54.299 INFO:tasks.workunit.client.0.smithi121.stdout:2/951: mkdir d2/d9/d1c/d137 0 2024-04-05T21:26:54.300 INFO:tasks.workunit.client.0.smithi121.stdout:1/982: write d8/d8c/db6/d1a/d1b/d4c/d76/da4/f116 [580585,56718] 0 2024-04-05T21:26:54.301 INFO:tasks.workunit.client.0.smithi121.stdout:3/914: readlink d0/d3/d24/d3d/d1a/d1e/db1/ddd/lef 0 2024-04-05T21:26:54.305 INFO:tasks.workunit.client.0.smithi121.stdout:2/952: symlink d2/db6/d129/l138 0 2024-04-05T21:26:54.306 INFO:tasks.workunit.client.0.smithi121.stdout:3/915: chown d0/d3/d24/d3d/d1a/d1e/l39 92826567 1 2024-04-05T21:26:54.307 INFO:tasks.workunit.client.0.smithi121.stdout:1/983: mkdir d8/d8c/db6/d1a/d1b/d24/d144 0 2024-04-05T21:26:54.310 INFO:tasks.workunit.client.0.smithi121.stdout:3/916: fsync d0/d3/de/da6/da1/f104 0 2024-04-05T21:26:54.310 INFO:tasks.workunit.client.0.smithi121.stdout:9/722: dwrite d6/df/d48/d96/db9/d2c/d37/fdc [0,4194304] 0 2024-04-05T21:26:54.312 INFO:tasks.workunit.client.0.smithi121.stdout:8/847: dread d12/d13/dd3/df0/fa9 [4194304,4194304] 0 2024-04-05T21:26:54.314 INFO:tasks.workunit.client.0.smithi121.stdout:1/984: truncate d8/d89/d9e/d2d/fb5 669322 0 2024-04-05T21:26:54.315 INFO:tasks.workunit.client.0.smithi121.stdout:3/917: mknod d0/d3/de/da6/de2/c13d 0 2024-04-05T21:26:54.317 INFO:tasks.workunit.client.0.smithi121.stdout:9/723: truncate d6/f5e 541106 0 2024-04-05T21:26:54.317 INFO:tasks.workunit.client.0.smithi121.stdout:8/848: dread d12/d13/d56/d57/d105/d63/d8b/f9b [0,4194304] 0 2024-04-05T21:26:54.323 INFO:tasks.workunit.client.0.smithi121.stdout:1/985: creat d8/d8c/db6/d1a/d1b/f145 x:0 0 0 2024-04-05T21:26:54.325 INFO:tasks.workunit.client.0.smithi121.stdout:9/724: mkdir d6/d32/dbe/dba/def/df7 0 2024-04-05T21:26:54.325 INFO:tasks.workunit.client.0.smithi121.stdout:2/953: dwrite d2/d9/d38/d2d/d4c/d79/fa2 [4194304,4194304] 0 2024-04-05T21:26:54.327 INFO:tasks.workunit.client.0.smithi121.stdout:2/954: dread - d2/d9/d1c/f10a zero size 2024-04-05T21:26:54.327 INFO:tasks.workunit.client.0.smithi121.stdout:9/725: dread - d6/df/d48/d96/db9/fe6 zero size 2024-04-05T21:26:54.328 INFO:tasks.workunit.client.0.smithi121.stdout:9/726: dread - d6/df/d48/d96/db9/d2c/fbf zero size 2024-04-05T21:26:54.329 INFO:tasks.workunit.client.0.smithi121.stdout:1/986: truncate d8/d8c/db6/d1a/f71 2592136 0 2024-04-05T21:26:54.330 INFO:tasks.workunit.client.0.smithi121.stdout:8/849: getdents d12/d13/d56/d57/d105/d63/d8b/dcc 0 2024-04-05T21:26:54.331 INFO:tasks.workunit.client.0.smithi121.stdout:2/955: mknod d2/d9/d17/d3a/c139 0 2024-04-05T21:26:54.336 INFO:tasks.workunit.client.0.smithi121.stdout:8/850: creat d12/d13/d56/d57/d5e/db9/f111 x:0 0 0 2024-04-05T21:26:54.338 INFO:tasks.workunit.client.0.smithi121.stdout:8/851: dread - d12/d50/fb3 zero size 2024-04-05T21:26:54.342 INFO:tasks.workunit.client.0.smithi121.stdout:3/918: dwrite d0/d3/d24/d3d/d1a/d23/d38/d55/d103/d12d/d12e/f10b [0,4194304] 0 2024-04-05T21:26:54.344 INFO:tasks.workunit.client.0.smithi121.stdout:3/919: write d0/d3/de/da6/de2/fe9 [585573,20828] 0 2024-04-05T21:26:54.348 INFO:tasks.workunit.client.0.smithi121.stdout:9/727: fsync d6/df/d50/fb1 0 2024-04-05T21:26:54.350 INFO:tasks.workunit.client.0.smithi121.stdout:9/728: fdatasync d6/df/d48/d96/db9/fe6 0 2024-04-05T21:26:54.351 INFO:tasks.workunit.client.0.smithi121.stdout:3/920: symlink d0/d3/d24/d6d/d7b/da3/dec/df6/dce/d124/l13e 0 2024-04-05T21:26:54.354 INFO:tasks.workunit.client.0.smithi121.stdout:1/987: dwrite d8/f1d [4194304,4194304] 0 2024-04-05T21:26:54.363 INFO:tasks.workunit.client.0.smithi121.stdout:3/921: dwrite d0/fcb [0,4194304] 0 2024-04-05T21:26:54.364 INFO:tasks.workunit.client.0.smithi121.stdout:9/729: write d6/df/f71 [2667450,34488] 0 2024-04-05T21:26:54.368 INFO:tasks.workunit.client.0.smithi121.stdout:9/730: unlink d6/df/d50/c59 0 2024-04-05T21:26:54.369 INFO:tasks.workunit.client.0.smithi121.stdout:8/852: dread d12/d50/d93/d96/f92 [0,4194304] 0 2024-04-05T21:26:54.369 INFO:tasks.workunit.client.0.smithi121.stdout:1/988: dwrite d8/d89/d9e/d2d/d84/da8/d11d/f13c [0,4194304] 0 2024-04-05T21:26:54.371 INFO:tasks.workunit.client.0.smithi121.stdout:9/731: read d6/d32/dbe/d7b/d6b/f78 [924898,100360] 0 2024-04-05T21:26:54.373 INFO:tasks.workunit.client.0.smithi121.stdout:9/732: chown d6/df/c1e 4 1 2024-04-05T21:26:54.374 INFO:tasks.workunit.client.0.smithi121.stdout:1/989: symlink d8/d8c/da2/l146 0 2024-04-05T21:26:54.375 INFO:tasks.workunit.client.0.smithi121.stdout:9/733: readlink d6/df/d48/d96/db9/d2c/d53/lb7 0 2024-04-05T21:26:54.375 INFO:tasks.workunit.client.0.smithi121.stdout:1/990: chown d8/d89/d9e/d57/f12e 61191 1 2024-04-05T21:26:54.379 INFO:tasks.workunit.client.0.smithi121.stdout:1/991: unlink d8/d8c/db6/d1a/d1b/d4c/f7e 0 2024-04-05T21:26:54.381 INFO:tasks.workunit.client.0.smithi121.stdout:9/734: truncate d6/df/d48/d96/db9/d2c/d37/f86 6798124 0 2024-04-05T21:26:54.383 INFO:tasks.workunit.client.0.smithi121.stdout:8/853: read d12/d50/ff2 [1656230,86738] 0 2024-04-05T21:26:54.383 INFO:tasks.workunit.client.0.smithi121.stdout:1/992: mkdir d8/d89/d9e/d2d/d84/da8/dad/d147 0 2024-04-05T21:26:54.385 INFO:tasks.workunit.client.0.smithi121.stdout:9/735: rmdir d6/d32/dbe 39 2024-04-05T21:26:54.386 INFO:tasks.workunit.client.0.smithi121.stdout:1/993: chown d8/d8c/db6/d1a/d1b/d24/f8b 24 1 2024-04-05T21:26:54.389 INFO:tasks.workunit.client.0.smithi121.stdout:9/736: creat d6/df/d48/d96/dc2/d85/ff8 x:0 0 0 2024-04-05T21:26:54.390 INFO:tasks.workunit.client.0.smithi121.stdout:3/922: dwrite d0/f41 [0,4194304] 0 2024-04-05T21:26:54.395 INFO:tasks.workunit.client.0.smithi121.stdout:3/923: creat d0/d3/d24/d3d/d1a/d23/d13b/f13f x:0 0 0 2024-04-05T21:26:54.400 INFO:tasks.workunit.client.0.smithi121.stdout:1/994: dwrite d8/d8c/db6/d1a/d1b/d85/ff8 [0,4194304] 0 2024-04-05T21:26:54.402 INFO:tasks.workunit.client.0.smithi121.stdout:3/924: dread - d0/d3/d24/d6d/d7b/da3/dec/f135 zero size 2024-04-05T21:26:54.403 INFO:tasks.workunit.client.0.smithi121.stdout:9/737: dwrite d6/d46/d8a/fdf [0,4194304] 0 2024-04-05T21:26:54.408 INFO:tasks.workunit.client.0.smithi121.stdout:1/995: chown d8/d8c/db6/d1a/d1b/d24/f2a 4148984 1 2024-04-05T21:26:54.413 INFO:tasks.workunit.client.0.smithi121.stdout:8/854: dwrite d12/d13/d56/d57/d5e/db9/fc0 [0,4194304] 0 2024-04-05T21:26:54.413 INFO:tasks.workunit.client.0.smithi121.stdout:1/996: rename d8/d89/d9e/d57/f60 to d8/d8c/db6/d1a/d1b/d24/dc2/f148 0 2024-04-05T21:26:54.417 INFO:tasks.workunit.client.0.smithi121.stdout:8/855: mkdir d12/d13/d56/d57/d105/d63/d8b/dd7/d101/d112 0 2024-04-05T21:26:54.420 INFO:tasks.workunit.client.0.smithi121.stdout:8/856: creat d12/d13/d56/d57/d105/d87/f113 x:0 0 0 2024-04-05T21:26:54.431 INFO:tasks.workunit.client.0.smithi121.stdout:3/925: write d0/d3/de/da6/da1/fb8 [173403,51248] 0 2024-04-05T21:26:54.431 INFO:tasks.workunit.client.0.smithi121.stdout:9/738: write d6/d32/dbe/d7b/fb3 [1031804,84449] 0 2024-04-05T21:26:54.433 INFO:tasks.workunit.client.0.smithi121.stdout:1/997: truncate d8/d89/d9e/d2d/f108 316392 0 2024-04-05T21:26:54.449 INFO:tasks.workunit.client.0.smithi121.stdout:8/857: dread d12/d13/d56/d57/f78 [0,4194304] 0 2024-04-05T21:26:54.449 INFO:tasks.workunit.client.0.smithi121.stdout:3/926: mknod d0/d3/d24/d3d/d1a/d23/d38/d55/c140 0 2024-04-05T21:26:54.451 INFO:tasks.workunit.client.0.smithi121.stdout:1/998: mkdir d8/d8c/db6/d1a/d1b/d85/d9c/d149 0 2024-04-05T21:26:54.453 INFO:tasks.workunit.client.0.smithi121.stdout:1/999: chown d8/d8c/db6/d1a/d1b/d4c/d76/d53/d5b 323 1 2024-04-05T21:26:54.453 INFO:tasks.workunit.client.0.smithi121.stdout:9/739: creat d6/df/d48/d96/dc2/ff9 x:0 0 0 2024-04-05T21:26:54.453 INFO:tasks.workunit.client.0.smithi121.stdout:3/927: symlink d0/d3/d24/d3d/d1a/d1e/l141 0 2024-04-05T21:26:54.456 INFO:tasks.workunit.client.0.smithi121.stdout:3/928: chown d0/d3/d24/d3d/d1a/d23/d90/c95 1394869257 1 2024-04-05T21:26:54.459 INFO:tasks.workunit.client.0.smithi121.stdout:3/929: fsync d0/d3/d24/d6d/d7b/de3/f11e 0 2024-04-05T21:26:54.461 INFO:tasks.workunit.client.0.smithi121.stdout:9/740: dwrite d6/d32/dbe/d7b/d49/f8b [4194304,4194304] 0 2024-04-05T21:26:54.464 INFO:tasks.workunit.client.0.smithi121.stdout:9/741: chown d6/df/c61 4509 1 2024-04-05T21:26:54.464 INFO:tasks.workunit.client.0.smithi121.stdout:8/858: dread d12/d50/fca [0,4194304] 0 2024-04-05T21:26:54.465 INFO:tasks.workunit.client.0.smithi121.stdout:3/930: fdatasync d0/d3/d24/d6d/d7b/fc7 0 2024-04-05T21:26:54.472 INFO:tasks.workunit.client.0.smithi121.stdout:9/742: mkdir d6/df/d48/d96/db9/dfa 0 2024-04-05T21:26:54.473 INFO:tasks.workunit.client.0.smithi121.stdout:3/931: dwrite d0/f114 [4194304,4194304] 0 2024-04-05T21:26:54.473 INFO:tasks.workunit.client.0.smithi121.stdout:8/859: mknod d12/d50/c114 0 2024-04-05T21:26:54.476 INFO:tasks.workunit.client.0.smithi121.stdout:9/743: chown d6/df/d48/d96/db9/d72/df1/lf4 98194 1 2024-04-05T21:26:54.478 INFO:tasks.workunit.client.0.smithi121.stdout:3/932: creat d0/d3/d11c/f142 x:0 0 0 2024-04-05T21:26:54.479 INFO:tasks.workunit.client.0.smithi121.stdout:9/744: stat d6/df/d50/f84 0 2024-04-05T21:26:54.479 INFO:tasks.workunit.client.0.smithi121.stdout:8/860: chown d12/d13/d56/d57/d105/d63/d8b/dd7 3186737 1 2024-04-05T21:26:54.481 INFO:tasks.workunit.client.0.smithi121.stdout:8/861: fsync d12/d50/d93/d96/f10d 0 2024-04-05T21:26:54.486 INFO:tasks.workunit.client.0.smithi121.stdout:3/933: dwrite d0/dc6/f128 [0,4194304] 0 2024-04-05T21:26:54.487 INFO:tasks.workunit.client.0.smithi121.stdout:3/934: fdatasync d0/d3/d24/df2/f12b 0 2024-04-05T21:26:54.491 INFO:tasks.workunit.client.0.smithi121.stdout:3/935: rename d0/d3/de/da6/de2/f11f to d0/d3/d24/d3d/d1a/d23/d13b/f143 0 2024-04-05T21:26:54.493 INFO:tasks.workunit.client.0.smithi121.stdout:3/936: rmdir d0/d3/d24/d6d/d7b/da3 39 2024-04-05T21:26:54.497 INFO:tasks.workunit.client.0.smithi121.stdout:8/862: dwrite fc [8388608,4194304] 0 2024-04-05T21:26:54.499 INFO:tasks.workunit.client.0.smithi121.stdout:3/937: rename d0/d3/d24/d3d/d1a/d1e/l39 to d0/dc6/l144 0 2024-04-05T21:26:54.506 INFO:tasks.workunit.client.0.smithi121.stdout:8/863: symlink d12/d13/dd3/df0/l115 0 2024-04-05T21:26:54.506 INFO:tasks.workunit.client.0.smithi121.stdout:9/745: dread d6/f17 [0,4194304] 0 2024-04-05T21:26:54.506 INFO:tasks.workunit.client.0.smithi121.stdout:8/864: dread - d12/d50/dbb/fee zero size 2024-04-05T21:26:54.508 INFO:tasks.workunit.client.0.smithi121.stdout:9/746: mknod d6/df/d48/cfb 0 2024-04-05T21:26:54.511 INFO:tasks.workunit.client.0.smithi121.stdout:9/747: dread - d6/d32/dbe/d7b/d49/d66/f91 zero size 2024-04-05T21:26:54.513 INFO:tasks.workunit.client.0.smithi121.stdout:3/938: dwrite d0/d3/d24/d3d/f3a [0,4194304] 0 2024-04-05T21:26:54.519 INFO:tasks.workunit.client.0.smithi121.stdout:9/748: dwrite d6/df/d48/d96/dc2/fcb [0,4194304] 0 2024-04-05T21:26:54.529 INFO:tasks.workunit.client.0.smithi121.stdout:8/865: dread ff [0,4194304] 0 2024-04-05T21:26:54.532 INFO:tasks.workunit.client.0.smithi121.stdout:3/939: dread d0/d3/d24/d6d/d7b/da3/dec/df9/f117 [0,4194304] 0 2024-04-05T21:26:54.535 INFO:tasks.workunit.client.0.smithi121.stdout:8/866: dwrite d12/d13/f84 [0,4194304] 0 2024-04-05T21:26:54.536 INFO:tasks.workunit.client.0.smithi121.stdout:9/749: dread d6/df/d48/d96/db9/d2c/d37/f86 [4194304,4194304] 0 2024-04-05T21:26:54.536 INFO:tasks.workunit.client.0.smithi121.stdout:3/940: write d0/d3/d24/d3d/d1a/d23/d38/d55/f8d [3827475,120479] 0 2024-04-05T21:26:54.539 INFO:tasks.workunit.client.0.smithi121.stdout:8/867: creat d12/d13/d56/d57/da4/f116 x:0 0 0 2024-04-05T21:26:54.541 INFO:tasks.workunit.client.0.smithi121.stdout:9/750: creat d6/df/d48/d96/db9/d2c/d37/d3c/ffc x:0 0 0 2024-04-05T21:26:54.542 INFO:tasks.workunit.client.0.smithi121.stdout:3/941: rmdir d0/d3/d24/d3d/d1a/d23/d38/d55/d103/d12d 39 2024-04-05T21:26:54.542 INFO:tasks.workunit.client.0.smithi121.stdout:9/751: stat d6/d32/dbe/d7b/d49 0 2024-04-05T21:26:54.545 INFO:tasks.workunit.client.0.smithi121.stdout:8/868: link d12/d13/d56/fa2 d12/d13/dd3/df0/da7/f117 0 2024-04-05T21:26:54.547 INFO:tasks.workunit.client.0.smithi121.stdout:9/752: getdents d6/df/d48/d96/db9/dfa 0 2024-04-05T21:26:54.549 INFO:tasks.workunit.client.0.smithi121.stdout:8/869: mknod d12/d13/d56/d57/d105/d87/ddf/c118 0 2024-04-05T21:26:54.550 INFO:tasks.workunit.client.0.smithi121.stdout:8/870: readlink d12/l85 0 2024-04-05T21:26:54.551 INFO:tasks.workunit.client.0.smithi121.stdout:9/753: fdatasync d6/df/f5f 0 2024-04-05T21:26:54.553 INFO:tasks.workunit.client.0.smithi121.stdout:9/754: dread - d6/df/d48/d96/dc2/d85/fc7 zero size 2024-04-05T21:26:54.556 INFO:tasks.workunit.client.0.smithi121.stdout:8/871: write d12/f89 [556237,56232] 0 2024-04-05T21:26:54.558 INFO:tasks.workunit.client.0.smithi121.stdout:8/872: readlink d12/d13/dd3/df0/l86 0 2024-04-05T21:26:54.559 INFO:tasks.workunit.client.0.smithi121.stdout:3/942: dread d0/d3/d24/d3d/d1a/d1e/db1/fbb [0,4194304] 0 2024-04-05T21:26:54.562 INFO:tasks.workunit.client.0.smithi121.stdout:8/873: dread ff [0,4194304] 0 2024-04-05T21:26:54.563 INFO:tasks.workunit.client.0.smithi121.stdout:8/874: fsync d12/d13/d15/de7/ffb 0 2024-04-05T21:26:54.564 INFO:tasks.workunit.client.0.smithi121.stdout:8/875: chown d12/f51 2886462 1 2024-04-05T21:26:54.568 INFO:tasks.workunit.client.0.smithi121.stdout:9/755: dread d6/f9b [0,4194304] 0 2024-04-05T21:26:54.568 INFO:tasks.workunit.client.0.smithi121.stdout:3/943: dwrite d0/d3/d24/db3/dc3/f121 [0,4194304] 0 2024-04-05T21:26:54.570 INFO:tasks.workunit.client.0.smithi121.stdout:8/876: mknod d12/d13/d56/d57/d105/d63/d8b/c119 0 2024-04-05T21:26:54.572 INFO:tasks.workunit.client.0.smithi121.stdout:6/985: sync 2024-04-05T21:26:54.573 INFO:tasks.workunit.client.0.smithi121.stdout:3/944: getdents d0/d3/d11c/d11d 0 2024-04-05T21:26:54.577 INFO:tasks.workunit.client.0.smithi121.stdout:3/945: getdents d0/d3/d24/d3d/d13c 0 2024-04-05T21:26:54.578 INFO:tasks.workunit.client.0.smithi121.stdout:8/877: dwrite fc [4194304,4194304] 0 2024-04-05T21:26:54.578 INFO:tasks.workunit.client.0.smithi121.stdout:9/756: write d6/df/d48/d96/db9/d2c/d37/d3c/f56 [1223873,87970] 0 2024-04-05T21:26:54.582 INFO:tasks.workunit.client.0.smithi121.stdout:3/946: mknod d0/dc6/c145 0 2024-04-05T21:26:54.583 INFO:tasks.workunit.client.0.smithi121.stdout:9/757: write d6/fd8 [83118,75585] 0 2024-04-05T21:26:54.583 INFO:tasks.workunit.client.0.smithi121.stdout:8/878: chown d12/d50/d93/d96/fab 6 1 2024-04-05T21:26:54.584 INFO:tasks.workunit.client.0.smithi121.stdout:6/986: dwrite d1/d2/d1b/d5e/fef [0,4194304] 0 2024-04-05T21:26:54.586 INFO:tasks.workunit.client.0.smithi121.stdout:3/947: rmdir d0/d3/d24/d6d/d7b/da3 39 2024-04-05T21:26:54.589 INFO:tasks.workunit.client.0.smithi121.stdout:8/879: unlink d12/d13/d56/d57/d105/d63/d9c/fe4 0 2024-04-05T21:26:54.593 INFO:tasks.workunit.client.0.smithi121.stdout:9/758: rename d6/f9b to d6/df/d48/d96/db9/dfa/ffd 0 2024-04-05T21:26:54.595 INFO:tasks.workunit.client.0.smithi121.stdout:3/948: dwrite d0/d3/f5b [4194304,4194304] 0 2024-04-05T21:26:54.597 INFO:tasks.workunit.client.0.smithi121.stdout:8/880: dread d12/d13/d56/d57/f78 [0,4194304] 0 2024-04-05T21:26:54.598 INFO:tasks.workunit.client.0.smithi121.stdout:6/987: getdents d1/d2/d3/d4/d21/d57/d42/d48 0 2024-04-05T21:26:54.599 INFO:tasks.workunit.client.0.smithi121.stdout:9/759: symlink d6/d32/dbe/dba/def/lfe 0 2024-04-05T21:26:54.600 INFO:tasks.workunit.client.0.smithi121.stdout:6/988: dread - d1/d2/d1b/d35/d74/df3/f11d zero size 2024-04-05T21:26:54.602 INFO:tasks.workunit.client.0.smithi121.stdout:8/881: symlink d12/d13/d56/d57/d5e/db9/l11a 0 2024-04-05T21:26:54.602 INFO:tasks.workunit.client.0.smithi121.stdout:6/989: dread - d1/d2/d1b/d35/d51/d105/f106 zero size 2024-04-05T21:26:54.602 INFO:tasks.workunit.client.0.smithi121.stdout:9/760: readlink d6/d81/ld2 0 2024-04-05T21:26:54.604 INFO:tasks.workunit.client.0.smithi121.stdout:9/761: write d6/df/d48/faa [723919,58936] 0 2024-04-05T21:26:54.605 INFO:tasks.workunit.client.0.smithi121.stdout:3/949: rename d0/d3/d24/d3d/d1a/d23/d38/d55/d103/d12d to d0/d3/d11c/d11d/d146 0 2024-04-05T21:26:54.606 INFO:tasks.workunit.client.0.smithi121.stdout:8/882: truncate d12/d50/d61/d7f/d9a/fd9 478867 0 2024-04-05T21:26:54.609 INFO:tasks.workunit.client.0.smithi121.stdout:6/990: rename d1/d2/d1b/d35/d74/df3 to d1/d2/d1b/d2e/dd9/d15e 0 2024-04-05T21:26:54.610 INFO:tasks.workunit.client.0.smithi121.stdout:9/762: mknod d6/df/d48/d96/db9/d72/cff 0 2024-04-05T21:26:54.614 INFO:tasks.workunit.client.0.smithi121.stdout:9/763: readlink d6/d32/dbe/d7b/lb5 0 2024-04-05T21:26:54.615 INFO:tasks.workunit.client.0.smithi121.stdout:8/883: getdents d12/d13/dd3/df0 0 2024-04-05T21:26:54.617 INFO:tasks.workunit.client.0.smithi121.stdout:6/991: dwrite d1/d2/dc/d9a/f139 [0,4194304] 0 2024-04-05T21:26:54.617 INFO:tasks.workunit.client.0.smithi121.stdout:8/884: dread - d12/d13/d15/de7/ffb zero size 2024-04-05T21:26:54.618 INFO:tasks.workunit.client.0.smithi121.stdout:9/764: chown d6/d32/dbe/d7b/d49/d66/l70 47 1 2024-04-05T21:26:54.620 INFO:tasks.workunit.client.0.smithi121.stdout:6/992: stat d1/d2/daf/d6f/dc7/d1f/d6d/cff 0 2024-04-05T21:26:54.626 INFO:tasks.workunit.client.0.smithi121.stdout:9/765: symlink d6/df/d48/d96/db9/d2c/d37/l100 0 2024-04-05T21:26:54.628 INFO:tasks.workunit.client.0.smithi121.stdout:9/766: mkdir d6/df/d48/d96/db9/d72/df1/d101 0 2024-04-05T21:26:54.629 INFO:tasks.workunit.client.0.smithi121.stdout:9/767: dread - d6/d32/fcf zero size 2024-04-05T21:26:54.631 INFO:tasks.workunit.client.0.smithi121.stdout:9/768: symlink d6/df/d50/l102 0 2024-04-05T21:26:54.632 INFO:tasks.workunit.client.0.smithi121.stdout:6/993: truncate d1/d2/dc/f5b 1387061 0 2024-04-05T21:26:54.633 INFO:tasks.workunit.client.0.smithi121.stdout:9/769: fdatasync d6/df/d48/d96/db9/d72/f7f 0 2024-04-05T21:26:54.634 INFO:tasks.workunit.client.0.smithi121.stdout:8/885: dwrite d12/d13/d15/f18 [0,4194304] 0 2024-04-05T21:26:54.638 INFO:tasks.workunit.client.0.smithi121.stdout:8/886: mkdir d12/d13/d56/d57/d105/d87/deb/d11b 0 2024-04-05T21:26:54.639 INFO:tasks.workunit.client.0.smithi121.stdout:9/770: link d6/d32/c42 d6/df/d48/d96/db9/c103 0 2024-04-05T21:26:54.640 INFO:tasks.workunit.client.0.smithi121.stdout:9/771: stat d6/df 0 2024-04-05T21:26:54.641 INFO:tasks.workunit.client.0.smithi121.stdout:8/887: symlink d12/d13/d56/d57/d105/d87/deb/d11b/l11c 0 2024-04-05T21:26:54.643 INFO:tasks.workunit.client.0.smithi121.stdout:8/888: symlink d12/d13/d56/d57/d105/d87/deb/d11b/l11d 0 2024-04-05T21:26:54.644 INFO:tasks.workunit.client.0.smithi121.stdout:8/889: readlink d12/d50/d61/d7f/d9a/lcf 0 2024-04-05T21:26:54.646 INFO:tasks.workunit.client.0.smithi121.stdout:8/890: mkdir d12/d50/d61/d7f/d9a/d11e 0 2024-04-05T21:26:54.651 INFO:tasks.workunit.client.0.smithi121.stdout:8/891: rename d12/d13/d56/d57/da4/dc9/fcb to d12/d13/d56/d57/f11f 0 2024-04-05T21:26:54.653 INFO:tasks.workunit.client.0.smithi121.stdout:8/892: creat d12/d13/d56/d57/da4/f120 x:0 0 0 2024-04-05T21:26:54.654 INFO:tasks.workunit.client.0.smithi121.stdout:8/893: chown d12/d13/d56/d57/d105/d87/deb/cd6 2 1 2024-04-05T21:26:54.657 INFO:tasks.workunit.client.0.smithi121.stdout:6/994: dread d1/d2/daf/d6f/dc7/f58 [0,4194304] 0 2024-04-05T21:26:54.659 INFO:tasks.workunit.client.0.smithi121.stdout:8/894: fsync d12/d13/dd3/df0/f42 0 2024-04-05T21:26:54.661 INFO:tasks.workunit.client.0.smithi121.stdout:6/995: fsync d1/d2/dc/f19 0 2024-04-05T21:26:54.661 INFO:tasks.workunit.client.0.smithi121.stdout:6/996: chown d1/d2/d1b/d2e/dd9 72 1 2024-04-05T21:26:54.662 INFO:tasks.workunit.client.0.smithi121.stdout:8/895: read d12/d13/fd2 [2633471,63877] 0 2024-04-05T21:26:54.668 INFO:tasks.workunit.client.0.smithi121.stdout:9/772: dread d6/df/d48/d96/db9/f33 [0,4194304] 0 2024-04-05T21:26:54.670 INFO:tasks.workunit.client.0.smithi121.stdout:6/997: getdents d1/d2/dc/d9a/d14e 0 2024-04-05T21:26:54.670 INFO:tasks.workunit.client.0.smithi121.stdout:8/896: dwrite d12/d50/d61/f3a [4194304,4194304] 0 2024-04-05T21:26:54.672 INFO:tasks.workunit.client.0.smithi121.stdout:9/773: mkdir d6/df/d104 0 2024-04-05T21:26:54.674 INFO:tasks.workunit.client.0.smithi121.stdout:6/998: write d1/d2/dc/f19 [1551222,51988] 0 2024-04-05T21:26:54.676 INFO:tasks.workunit.client.0.smithi121.stdout:9/774: fsync d6/df/d48/d96/db9/d2c/d53/fe5 0 2024-04-05T21:26:54.684 INFO:tasks.workunit.client.0.smithi121.stdout:8/897: link d12/d50/d93/d96/l9e d12/d13/d56/d57/d105/d87/deb/d11b/l121 0 2024-04-05T21:26:54.685 INFO:tasks.workunit.client.0.smithi121.stdout:9/775: dwrite d6/d32/dbe/d7b/d49/d66/fc1 [0,4194304] 0 2024-04-05T21:26:54.686 INFO:tasks.workunit.client.0.smithi121.stdout:8/898: dread - d12/d13/d56/d57/d105/d87/f113 zero size 2024-04-05T21:26:54.688 INFO:tasks.workunit.client.0.smithi121.stdout:9/776: symlink d6/df/d48/d96/db2/l105 0 2024-04-05T21:26:54.689 INFO:tasks.workunit.client.0.smithi121.stdout:8/899: dread - d12/d13/d56/d57/da4/f120 zero size 2024-04-05T21:26:54.691 INFO:tasks.workunit.client.0.smithi121.stdout:8/900: chown d12/d50/d93/f95 782005 1 2024-04-05T21:26:54.692 INFO:tasks.workunit.client.0.smithi121.stdout:6/999: dwrite d1/d2/d3/d4/d21/d57/d42/f83 [0,4194304] 0 2024-04-05T21:26:54.694 INFO:tasks.workunit.client.0.smithi121.stdout:8/901: creat d12/d13/d56/d57/d105/d87/ddf/f122 x:0 0 0 2024-04-05T21:26:54.699 INFO:tasks.workunit.client.0.smithi121.stdout:8/902: mkdir d12/d13/d56/d57/d105/d63/d8b/dd7/d123 0 2024-04-05T21:26:54.700 INFO:tasks.workunit.client.0.smithi121.stdout:9/777: write d6/df/d48/f7e [2480293,127125] 0 2024-04-05T21:26:54.703 INFO:tasks.workunit.client.0.smithi121.stdout:9/778: symlink d6/df/d104/l106 0 2024-04-05T21:26:54.705 INFO:tasks.workunit.client.0.smithi121.stdout:9/779: creat d6/d32/dbe/d7b/d49/d66/f107 x:0 0 0 2024-04-05T21:26:54.708 INFO:tasks.workunit.client.0.smithi121.stdout:9/780: creat d6/df/d48/d96/dc2/d79/d9c/f108 x:0 0 0 2024-04-05T21:26:54.710 INFO:tasks.workunit.client.0.smithi121.stdout:8/903: dwrite d12/d13/d56/d57/d105/d63/d9c/fac [0,4194304] 0 2024-04-05T21:26:54.712 INFO:tasks.workunit.client.0.smithi121.stdout:9/781: unlink d6/cb 0 2024-04-05T21:26:54.715 INFO:tasks.workunit.client.0.smithi121.stdout:9/782: mkdir d6/d109 0 2024-04-05T21:26:54.716 INFO:tasks.workunit.client.0.smithi121.stdout:9/783: symlink d6/df/d50/l10a 0 2024-04-05T21:26:54.716 INFO:tasks.workunit.client.0.smithi121.stdout:9/784: chown d6/df/d48/d96/dc2 656827 1 2024-04-05T21:26:54.721 INFO:tasks.workunit.client.0.smithi121.stdout:9/785: truncate d6/df/d48/d96/db9/dfa/ffd 5476818 0 2024-04-05T21:26:54.722 INFO:tasks.workunit.client.0.smithi121.stdout:8/904: dwrite d12/d50/d61/faa [0,4194304] 0 2024-04-05T21:26:54.731 INFO:tasks.workunit.client.0.smithi121.stdout:8/905: dwrite d12/d50/d93/f95 [0,4194304] 0 2024-04-05T21:26:54.733 INFO:tasks.workunit.client.0.smithi121.stdout:8/906: chown d12/d13/d56/d57/d105/d63/d8b/c119 50277 1 2024-04-05T21:26:54.735 INFO:tasks.workunit.client.0.smithi121.stdout:9/786: dwrite d6/df/d48/d96/db9/f76 [0,4194304] 0 2024-04-05T21:26:54.737 INFO:tasks.workunit.client.0.smithi121.stdout:8/907: symlink d12/d13/d56/d57/d105/d63/d8b/dcc/d10a/l124 0 2024-04-05T21:26:54.738 INFO:tasks.workunit.client.0.smithi121.stdout:9/787: symlink d6/df/d48/d96/dc2/d79/d9c/l10b 0 2024-04-05T21:26:54.742 INFO:tasks.workunit.client.0.smithi121.stdout:9/788: mknod d6/df/d48/d96/dc2/d79/c10c 0 2024-04-05T21:26:54.747 INFO:tasks.workunit.client.0.smithi121.stdout:9/789: mknod d6/df/d48/d96/db9/d2c/de9/c10d 0 2024-04-05T21:26:54.747 INFO:tasks.workunit.client.0.smithi121.stdout:8/908: dwrite d12/d50/d61/faa [0,4194304] 0 2024-04-05T21:26:54.749 INFO:tasks.workunit.client.0.smithi121.stdout:9/790: write d6/df/f6d [1085846,8105] 0 2024-04-05T21:26:54.755 INFO:tasks.workunit.client.0.smithi121.stdout:8/909: mknod d12/d13/d56/d57/d105/c125 0 2024-04-05T21:26:54.757 INFO:tasks.workunit.client.0.smithi121.stdout:9/791: write d6/df/d50/f84 [542425,75739] 0 2024-04-05T21:26:54.758 INFO:tasks.workunit.client.0.smithi121.stdout:8/910: creat d12/d13/d56/d57/d105/d63/d8b/dd7/d101/f126 x:0 0 0 2024-04-05T21:26:54.764 INFO:tasks.workunit.client.0.smithi121.stdout:8/911: link d12/d50/d61/d7f/d9a/cc8 d12/d50/c127 0 2024-04-05T21:26:54.766 INFO:tasks.workunit.client.0.smithi121.stdout:9/792: dwrite d6/f17 [0,4194304] 0 2024-04-05T21:26:54.769 INFO:tasks.workunit.client.0.smithi121.stdout:8/912: truncate d12/f51 860231 0 2024-04-05T21:26:54.773 INFO:tasks.workunit.client.0.smithi121.stdout:8/913: read d12/d50/d61/d7f/d9a/fd9 [291149,113268] 0 2024-04-05T21:26:54.774 INFO:tasks.workunit.client.0.smithi121.stdout:9/793: dwrite d6/df/d50/f84 [0,4194304] 0 2024-04-05T21:26:54.775 INFO:tasks.workunit.client.0.smithi121.stdout:8/914: truncate d12/d13/dd3/df0/fa9 8696660 0 2024-04-05T21:26:54.777 INFO:tasks.workunit.client.0.smithi121.stdout:8/915: dread - d12/d13/d56/d57/d5e/db9/f111 zero size 2024-04-05T21:26:54.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:54 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:26:54.781 INFO:tasks.workunit.client.0.smithi121.stdout:9/794: dwrite d6/d32/dbe/d7b/d6b/f92 [4194304,4194304] 0 2024-04-05T21:26:54.783 INFO:tasks.workunit.client.0.smithi121.stdout:9/795: fsync d6/df/d48/d96/dc2/d85/fc7 0 2024-04-05T21:26:54.787 INFO:tasks.workunit.client.0.smithi121.stdout:9/796: unlink d6/df/d48/d96/db9/fe6 0 2024-04-05T21:26:54.788 INFO:tasks.workunit.client.0.smithi121.stdout:9/797: fdatasync d6/df/d48/d96/db9/d72/f7f 0 2024-04-05T21:26:54.796 INFO:tasks.workunit.client.0.smithi121.stdout:8/916: dwrite d12/d13/d15/f21 [0,4194304] 0 2024-04-05T21:26:54.803 INFO:tasks.workunit.client.0.smithi121.stdout:8/917: dread d12/d13/d15/f21 [0,4194304] 0 2024-04-05T21:26:54.805 INFO:tasks.workunit.client.0.smithi121.stdout:8/918: read d12/d50/ffd [739989,110753] 0 2024-04-05T21:26:54.807 INFO:tasks.workunit.client.0.smithi121.stdout:8/919: symlink d12/d50/dbb/de5/l128 0 2024-04-05T21:26:54.809 INFO:tasks.workunit.client.0.smithi121.stdout:8/920: unlink d12/d13/d56/d57/d5e/db9/l11a 0 2024-04-05T21:26:54.809 INFO:tasks.workunit.client.0.smithi121.stdout:8/921: read fc [8675902,76949] 0 2024-04-05T21:26:54.812 INFO:tasks.workunit.client.0.smithi121.stdout:8/922: mkdir d12/d13/d56/d57/d5e/db9/d129 0 2024-04-05T21:26:54.816 INFO:tasks.workunit.client.0.smithi121.stdout:8/923: link d12/d50/d93/d96/l7c d12/d50/d61/d7f/l12a 0 2024-04-05T21:26:54.817 INFO:tasks.workunit.client.0.smithi121.stdout:8/924: creat d12/d13/dd3/df0/da7/f12b x:0 0 0 2024-04-05T21:26:54.818 INFO:tasks.workunit.client.0.smithi121.stdout:8/925: read - d12/d13/d56/d57/d5e/db9/ff9 zero size 2024-04-05T21:26:54.820 INFO:tasks.workunit.client.0.smithi121.stdout:8/926: mkdir d12/d13/d56/d57/d105/d87/deb/d12c 0 2024-04-05T21:26:54.826 INFO:tasks.workunit.client.0.smithi121.stdout:8/927: dwrite d12/d50/fb3 [0,4194304] 0 2024-04-05T21:26:54.829 INFO:tasks.workunit.client.0.smithi121.stdout:8/928: fdatasync d12/d50/ff2 0 2024-04-05T21:26:54.830 INFO:tasks.workunit.client.0.smithi121.stdout:8/929: readlink d12/d13/d56/d57/d105/d87/lad 0 2024-04-05T21:26:54.841 INFO:tasks.workunit.client.0.smithi121.stdout:8/930: write d12/d50/d93/d96/fb2 [3265159,130446] 0 2024-04-05T21:26:54.844 INFO:tasks.workunit.client.0.smithi121.stdout:8/931: mknod d12/d13/d56/d57/d105/d63/d8b/dd7/d123/c12d 0 2024-04-05T21:26:54.844 INFO:tasks.workunit.client.0.smithi121.stdout:8/932: chown d12/d50/d93/d96 3 1 2024-04-05T21:26:54.848 INFO:tasks.workunit.client.0.smithi121.stdout:8/933: rmdir d12/d13/d56/d57/d105/d63/d8b/dd7/d101/d112 0 2024-04-05T21:26:54.849 INFO:tasks.workunit.client.0.smithi121.stdout:8/934: readlink d12/d13/dd3/df0/da7/lfc 0 2024-04-05T21:26:54.851 INFO:tasks.workunit.client.0.smithi121.stdout:8/935: creat d12/d50/dbb/de5/f12e x:0 0 0 2024-04-05T21:26:54.852 INFO:tasks.workunit.client.0.smithi121.stdout:8/936: write d12/d13/d56/d57/d5e/db9/ff9 [113723,1566] 0 2024-04-05T21:26:54.876 INFO:tasks.workunit.client.0.smithi121.stdout:2/956: sync 2024-04-05T21:26:54.881 INFO:tasks.workunit.client.0.smithi121.stdout:2/957: write d2/d9/d38/d2d/d4c/de9/d8d/fd6 [335395,126605] 0 2024-04-05T21:26:54.884 INFO:tasks.workunit.client.0.smithi121.stdout:2/958: creat d2/d9/d38/d2d/d47/d94/f13a x:0 0 0 2024-04-05T21:26:54.886 INFO:tasks.workunit.client.0.smithi121.stdout:2/959: creat d2/d9/d38/d2d/d4c/d7c/d7e/db9/d12c/f13b x:0 0 0 2024-04-05T21:26:54.887 INFO:tasks.workunit.client.0.smithi121.stdout:2/960: chown d2/d9/d77/lb7 873848055 1 2024-04-05T21:26:54.887 INFO:tasks.workunit.client.0.smithi121.stdout:2/961: readlink d2/d9/d38/d2d/d47/l124 0 2024-04-05T21:26:54.888 INFO:tasks.workunit.client.0.smithi121.stdout:2/962: stat d2/d9/d38/c113 0 2024-04-05T21:26:54.889 INFO:tasks.workunit.client.0.smithi121.stdout:2/963: read - d2/d9/d38/d2d/d4c/d79/f12b zero size 2024-04-05T21:26:54.890 INFO:tasks.workunit.client.0.smithi121.stdout:2/964: dread - d2/d9/d38/d2d/d4c/de9/fe8 zero size 2024-04-05T21:26:54.901 INFO:tasks.workunit.client.0.smithi121.stdout:2/965: dread d2/d1e/fd5 [0,4194304] 0 2024-04-05T21:26:54.902 INFO:tasks.workunit.client.0.smithi121.stdout:2/966: write d2/d9/d38/d2d/d47/f101 [955940,123536] 0 2024-04-05T21:26:54.905 INFO:tasks.workunit.client.0.smithi121.stdout:2/967: unlink d2/d9/d38/d2d/d47/d94/d84/l9e 0 2024-04-05T21:26:54.920 INFO:tasks.workunit.client.0.smithi121.stdout:2/968: dread d2/d9/d38/d2d/d4c/de9/d5c/df0/f12f [0,4194304] 0 2024-04-05T21:26:54.934 INFO:tasks.workunit.client.0.smithi121.stdout:2/969: dread d2/d9/d38/f35 [0,4194304] 0 2024-04-05T21:26:54.935 INFO:tasks.workunit.client.0.smithi121.stdout:2/970: write d2/d9/ffc [435738,35017] 0 2024-04-05T21:26:54.937 INFO:tasks.workunit.client.0.smithi121.stdout:2/971: mkdir d2/d9/d13c 0 2024-04-05T21:26:54.938 INFO:tasks.workunit.client.0.smithi121.stdout:2/972: chown d2/d1e/fd5 5498 1 2024-04-05T21:26:54.940 INFO:tasks.workunit.client.0.smithi121.stdout:2/973: write d2/d1e/fd5 [2237814,39176] 0 2024-04-05T21:26:54.943 INFO:tasks.workunit.client.0.smithi121.stdout:2/974: symlink d2/d9/d1c/d85/def/l13d 0 2024-04-05T21:26:54.954 INFO:tasks.workunit.client.0.smithi121.stdout:2/975: dwrite d2/d9/f73 [0,4194304] 0 2024-04-05T21:26:54.956 INFO:tasks.workunit.client.0.smithi121.stdout:2/976: creat d2/d9/d38/d2d/d4c/f13e x:0 0 0 2024-04-05T21:26:54.966 INFO:tasks.workunit.client.0.smithi121.stdout:2/977: truncate d2/d9/d38/d2d/d4c/d79/fb8 4169816 0 2024-04-05T21:26:54.971 INFO:tasks.workunit.client.0.smithi121.stdout:2/978: dwrite d2/d9/d38/d2d/d4c/d79/f12b [0,4194304] 0 2024-04-05T21:26:54.973 INFO:tasks.workunit.client.0.smithi121.stdout:2/979: write d2/d9/d17/d3a/f4a [3354065,56442] 0 2024-04-05T21:26:54.974 INFO:tasks.workunit.client.0.smithi121.stdout:2/980: truncate d2/d9/ff 5378263 0 2024-04-05T21:26:54.977 INFO:tasks.workunit.client.0.smithi121.stdout:2/981: unlink d2/db6/dbb/c111 0 2024-04-05T21:26:54.979 INFO:tasks.workunit.client.0.smithi121.stdout:2/982: fdatasync d2/d9/d38/d2d/d4c/de9/d8d/fd6 0 2024-04-05T21:26:54.981 INFO:tasks.workunit.client.0.smithi121.stdout:2/983: creat d2/d9/d38/d2d/d4c/de9/d5c/f13f x:0 0 0 2024-04-05T21:26:54.982 INFO:tasks.workunit.client.0.smithi121.stdout:2/984: stat d2/d9 0 2024-04-05T21:26:54.986 INFO:tasks.workunit.client.0.smithi121.stdout:2/985: truncate d2/d9/d17/d3a/f115 401854 0 2024-04-05T21:26:54.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:54 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:26:54.989 INFO:tasks.workunit.client.0.smithi121.stdout:2/986: mkdir d2/d9/d38/d2d/d4c/de9/d5c/df0/d140 0 2024-04-05T21:26:55.140 INFO:tasks.workunit.client.0.smithi121.stdout:3/950: sync 2024-04-05T21:26:55.140 INFO:tasks.workunit.client.0.smithi121.stdout:3/951: truncate d0/dc6/f132 910964 0 2024-04-05T21:26:55.141 INFO:tasks.workunit.client.0.smithi121.stdout:3/952: dread - d0/d3/d24/d3d/d1a/d1e/f105 zero size 2024-04-05T21:26:55.143 INFO:tasks.workunit.client.0.smithi121.stdout:3/953: chown d0/d3/d24/d3d/d1a/c2f 1384 1 2024-04-05T21:26:55.145 INFO:tasks.workunit.client.0.smithi121.stdout:3/954: write d0/d3/d24/d3d/f21 [3800531,23038] 0 2024-04-05T21:26:55.153 INFO:tasks.workunit.client.0.smithi121.stdout:3/955: dread d0/f27 [0,4194304] 0 2024-04-05T21:26:55.158 INFO:tasks.workunit.client.0.smithi121.stdout:9/798: sync 2024-04-05T21:26:55.158 INFO:tasks.workunit.client.0.smithi121.stdout:8/937: sync 2024-04-05T21:26:55.158 INFO:tasks.workunit.client.0.smithi121.stdout:2/987: sync 2024-04-05T21:26:55.160 INFO:tasks.workunit.client.0.smithi121.stdout:9/799: rmdir d6/d32/dbe/d7b/d49 39 2024-04-05T21:26:55.161 INFO:tasks.workunit.client.0.smithi121.stdout:3/956: write d0/d3/d24/d3d/d1a/d23/d90/f112 [443460,27300] 0 2024-04-05T21:26:55.163 INFO:tasks.workunit.client.0.smithi121.stdout:3/957: fdatasync d0/d3/d24/d3d/d1a/d1e/f2a 0 2024-04-05T21:26:55.164 INFO:tasks.workunit.client.0.smithi121.stdout:8/938: write d12/d13/d56/d57/d105/d63/d8b/fcd [3777451,114578] 0 2024-04-05T21:26:55.166 INFO:tasks.workunit.client.0.smithi121.stdout:9/800: creat d6/d32/dbe/d7b/f10e x:0 0 0 2024-04-05T21:26:55.167 INFO:tasks.workunit.client.0.smithi121.stdout:3/958: mknod d0/d3/d24/d3d/d1a/d1e/d86/c147 0 2024-04-05T21:26:55.169 INFO:tasks.workunit.client.0.smithi121.stdout:9/801: creat d6/df/d48/d96/db2/f10f x:0 0 0 2024-04-05T21:26:55.170 INFO:tasks.workunit.client.0.smithi121.stdout:8/939: dread d12/d13/d15/f18 [0,4194304] 0 2024-04-05T21:26:55.170 INFO:tasks.workunit.client.0.smithi121.stdout:3/959: symlink d0/d3/d24/db3/dc3/l148 0 2024-04-05T21:26:55.171 INFO:tasks.workunit.client.0.smithi121.stdout:9/802: symlink d6/d46/l110 0 2024-04-05T21:26:55.172 INFO:tasks.workunit.client.0.smithi121.stdout:8/940: readlink d12/d13/d15/l81 0 2024-04-05T21:26:55.173 INFO:tasks.workunit.client.0.smithi121.stdout:3/960: write d0/d3/d24/d3d/d1a/d1e/db1/ddd/de1/fb6 [1123595,65387] 0 2024-04-05T21:26:55.174 INFO:tasks.workunit.client.0.smithi121.stdout:9/803: unlink d6/df/d50/fd9 0 2024-04-05T21:26:55.177 INFO:tasks.workunit.client.0.smithi121.stdout:2/988: dread d2/f7b [0,4194304] 0 2024-04-05T21:26:55.178 INFO:tasks.workunit.client.0.smithi121.stdout:8/941: unlink d12/d13/d15/c68 0 2024-04-05T21:26:55.180 INFO:tasks.workunit.client.0.smithi121.stdout:3/961: creat d0/d3/d24/d6d/f149 x:0 0 0 2024-04-05T21:26:55.182 INFO:tasks.workunit.client.0.smithi121.stdout:2/989: getdents d2/db6/dbb 0 2024-04-05T21:26:55.185 INFO:tasks.workunit.client.0.smithi121.stdout:9/804: dwrite d6/df/d48/d96/dc2/f87 [0,4194304] 0 2024-04-05T21:26:55.187 INFO:tasks.workunit.client.0.smithi121.stdout:3/962: unlink d0/d3/d24/d3d/d1a/d23/d38/c43 0 2024-04-05T21:26:55.190 INFO:tasks.workunit.client.0.smithi121.stdout:3/963: read - d0/d3/fc5 zero size 2024-04-05T21:26:55.190 INFO:tasks.workunit.client.0.smithi121.stdout:8/942: dwrite d12/d13/d15/de7/fed [0,4194304] 0 2024-04-05T21:26:55.190 INFO:tasks.workunit.client.0.smithi121.stdout:2/990: unlink d2/db6/dbb/dee/d132/f4f 0 2024-04-05T21:26:55.193 INFO:tasks.workunit.client.0.smithi121.stdout:9/805: rmdir d6/d46/d8a 39 2024-04-05T21:26:55.195 INFO:tasks.workunit.client.0.smithi121.stdout:9/806: dread - d6/df/d48/d96/dc2/d85/fc7 zero size 2024-04-05T21:26:55.200 INFO:tasks.workunit.client.0.smithi121.stdout:2/991: creat d2/d9/d38/d2d/d4c/de5/f141 x:0 0 0 2024-04-05T21:26:55.201 INFO:tasks.workunit.client.0.smithi121.stdout:9/807: creat d6/df/d48/d96/f111 x:0 0 0 2024-04-05T21:26:55.207 INFO:tasks.workunit.client.0.smithi121.stdout:3/964: dwrite d0/d3/d24/d6d/d7b/da3/dec/df6/f80 [4194304,4194304] 0 2024-04-05T21:26:55.210 INFO:tasks.workunit.client.0.smithi121.stdout:8/943: write f3 [1357390,44945] 0 2024-04-05T21:26:55.211 INFO:tasks.workunit.client.0.smithi121.stdout:2/992: truncate d2/d1e/fd5 1634969 0 2024-04-05T21:26:55.211 INFO:tasks.workunit.client.0.smithi121.stdout:3/965: mknod d0/d3/d11c/d11d/d146/d12e/c14a 0 2024-04-05T21:26:55.213 INFO:tasks.workunit.client.0.smithi121.stdout:8/944: write d12/d50/d93/ff1 [271448,94925] 0 2024-04-05T21:26:55.214 INFO:tasks.workunit.client.0.smithi121.stdout:2/993: creat d2/d9/d1c/d10b/f142 x:0 0 0 2024-04-05T21:26:55.215 INFO:tasks.workunit.client.0.smithi121.stdout:3/966: write d0/d3/d24/d6d/d7b/f85 [96526,25469] 0 2024-04-05T21:26:55.220 INFO:tasks.workunit.client.0.smithi121.stdout:9/808: dwrite d6/df/f5f [0,4194304] 0 2024-04-05T21:26:55.220 INFO:tasks.workunit.client.0.smithi121.stdout:3/967: write d0/d3/f4 [4365156,6396] 0 2024-04-05T21:26:55.220 INFO:tasks.workunit.client.0.smithi121.stdout:2/994: read d2/d9/d38/d2d/d4c/d7c/dc2/fc9 [3872,42535] 0 2024-04-05T21:26:55.226 INFO:tasks.workunit.client.0.smithi121.stdout:8/945: dwrite d12/d13/dd3/df0/f4c [0,4194304] 0 2024-04-05T21:26:55.227 INFO:tasks.workunit.client.0.smithi121.stdout:3/968: chown d0/d3/de/c46 27662264 1 2024-04-05T21:26:55.233 INFO:tasks.workunit.client.0.smithi121.stdout:2/995: dread d2/d9/d17/f19 [0,4194304] 0 2024-04-05T21:26:55.233 INFO:tasks.workunit.client.0.smithi121.stdout:3/969: write d0/d3/f4 [3649680,65989] 0 2024-04-05T21:26:55.233 INFO:tasks.workunit.client.0.smithi121.stdout:2/996: chown d2/d9/d38/d2d/d4c/fa6 88 1 2024-04-05T21:26:55.236 INFO:tasks.workunit.client.0.smithi121.stdout:3/970: write d0/d3/d24/d6d/d7b/de3/ff0 [292127,40091] 0 2024-04-05T21:26:55.237 INFO:tasks.workunit.client.0.smithi121.stdout:2/997: creat d2/d9/d38/d2d/d4c/d79/f143 x:0 0 0 2024-04-05T21:26:55.238 INFO:tasks.workunit.client.0.smithi121.stdout:8/946: dwrite d12/d13/dd3/df0/f36 [0,4194304] 0 2024-04-05T21:26:55.240 INFO:tasks.workunit.client.0.smithi121.stdout:2/998: write d2/d9/d38/d2d/d47/d94/f13a [915904,4952] 0 2024-04-05T21:26:55.241 INFO:tasks.workunit.client.0.smithi121.stdout:2/999: chown d2/c45 5 1 2024-04-05T21:26:55.241 INFO:tasks.workunit.client.0.smithi121.stdout:3/971: write d0/d3/d24/d3d/d1a/d1e/d86/fde [2619898,73496] 0 2024-04-05T21:26:55.243 INFO:tasks.workunit.client.0.smithi121.stdout:8/947: creat d12/d13/d15/de7/f12f x:0 0 0 2024-04-05T21:26:55.243 INFO:tasks.workunit.client.0.smithi121.stdout:3/972: stat d0/d3/de/da6/da1 0 2024-04-05T21:26:55.247 INFO:tasks.workunit.client.0.smithi121.stdout:3/973: creat d0/d3/d24/d6d/d7b/da3/dec/f14b x:0 0 0 2024-04-05T21:26:55.248 INFO:tasks.workunit.client.0.smithi121.stdout:9/809: truncate d6/d32/dbe/d7b/d49/d66/fc1 2605389 0 2024-04-05T21:26:55.250 INFO:tasks.workunit.client.0.smithi121.stdout:9/810: stat d6/df/d48 0 2024-04-05T21:26:55.252 INFO:tasks.workunit.client.0.smithi121.stdout:3/974: mkdir d0/d3/de/da6/de2/d14c 0 2024-04-05T21:26:55.253 INFO:tasks.workunit.client.0.smithi121.stdout:9/811: symlink d6/df/d48/l112 0 2024-04-05T21:26:55.253 INFO:tasks.workunit.client.0.smithi121.stdout:8/948: dwrite d12/d50/ffd [0,4194304] 0 2024-04-05T21:26:55.254 INFO:tasks.workunit.client.0.smithi121.stdout:3/975: write d0/dc6/f132 [1417003,79] 0 2024-04-05T21:26:55.255 INFO:tasks.workunit.client.0.smithi121.stdout:9/812: write d6/d32/dbe/d7b/d6b/f8f [4612007,22492] 0 2024-04-05T21:26:55.257 INFO:tasks.workunit.client.0.smithi121.stdout:8/949: mknod d12/d13/d56/d57/d105/d63/d8b/dcc/c130 0 2024-04-05T21:26:55.258 INFO:tasks.workunit.client.0.smithi121.stdout:3/976: mknod d0/d3/d24/d3d/d1a/d23/d38/d55/c14d 0 2024-04-05T21:26:55.259 INFO:tasks.workunit.client.0.smithi121.stdout:9/813: mknod d6/df/d48/d96/db9/dfa/c113 0 2024-04-05T21:26:55.260 INFO:tasks.workunit.client.0.smithi121.stdout:3/977: mkdir d0/d3/d24/df2/d14e 0 2024-04-05T21:26:55.261 INFO:tasks.workunit.client.0.smithi121.stdout:8/950: symlink d12/d13/d56/d57/d105/d87/deb/d11b/l131 0 2024-04-05T21:26:55.266 INFO:tasks.workunit.client.0.smithi121.stdout:3/978: getdents d0/d3/de/da6/de2/d14c 0 2024-04-05T21:26:55.269 INFO:tasks.workunit.client.0.smithi121.stdout:3/979: rmdir d0/d3/d24/d6d/dff 39 2024-04-05T21:26:55.270 INFO:tasks.workunit.client.0.smithi121.stdout:9/814: dwrite d6/d46/d8a/dd5/feb [0,4194304] 0 2024-04-05T21:26:55.273 INFO:tasks.workunit.client.0.smithi121.stdout:3/980: creat d0/d3/d24/d3d/d1a/d23/d90/d9e/f14f x:0 0 0 2024-04-05T21:26:55.280 INFO:tasks.workunit.client.0.smithi121.stdout:9/815: dwrite d6/df/d48/d96/dc2/d79/d9c/f108 [0,4194304] 0 2024-04-05T21:26:55.285 INFO:tasks.workunit.client.0.smithi121.stdout:3/981: dwrite d0/d3/d24/d3d/d1a/d1e/db1/ddd/de1/fb6 [0,4194304] 0 2024-04-05T21:26:55.288 INFO:tasks.workunit.client.0.smithi121.stdout:8/951: dwrite d12/d13/d56/d57/da4/f31 [4194304,4194304] 0 2024-04-05T21:26:55.293 INFO:tasks.workunit.client.0.smithi121.stdout:3/982: creat d0/d3/d24/df2/d14e/f150 x:0 0 0 2024-04-05T21:26:55.293 INFO:tasks.workunit.client.0.smithi121.stdout:9/816: dread d6/d32/dbe/d7b/d6b/f8f [4194304,4194304] 0 2024-04-05T21:26:55.293 INFO:tasks.workunit.client.0.smithi121.stdout:3/983: fsync d0/d3/d24/d6d/d7b/f85 0 2024-04-05T21:26:55.293 INFO:tasks.workunit.client.0.smithi121.stdout:9/817: dread - d6/df/d48/d96/db9/d2c/fbf zero size 2024-04-05T21:26:55.295 INFO:tasks.workunit.client.0.smithi121.stdout:8/952: truncate d12/d50/ff2 3098673 0 2024-04-05T21:26:55.295 INFO:tasks.workunit.client.0.smithi121.stdout:9/818: write d6/d32/dbe/d7b/fb3 [789995,16102] 0 2024-04-05T21:26:55.296 INFO:tasks.workunit.client.0.smithi121.stdout:3/984: rmdir d0/d3/d24/d3d/d1a 39 2024-04-05T21:26:55.297 INFO:tasks.workunit.client.0.smithi121.stdout:9/819: write d6/df/d48/d96/dc2/fcb [1724553,47844] 0 2024-04-05T21:26:55.300 INFO:tasks.workunit.client.0.smithi121.stdout:8/953: mknod d12/d13/d56/d57/d105/c132 0 2024-04-05T21:26:55.302 INFO:tasks.workunit.client.0.smithi121.stdout:3/985: creat d0/d3/d24/d3d/d13c/f151 x:0 0 0 2024-04-05T21:26:55.302 INFO:tasks.workunit.client.0.smithi121.stdout:9/820: mkdir d6/df/d48/d96/db9/dfa/d114 0 2024-04-05T21:26:55.303 INFO:tasks.workunit.client.0.smithi121.stdout:9/821: chown d6/fd8 3240785 1 2024-04-05T21:26:55.304 INFO:tasks.workunit.client.0.smithi121.stdout:8/954: creat d12/d13/d56/d57/d105/d63/f133 x:0 0 0 2024-04-05T21:26:55.305 INFO:tasks.workunit.client.0.smithi121.stdout:3/986: mkdir d0/d3/d24/d6d/d7b/d152 0 2024-04-05T21:26:55.306 INFO:tasks.workunit.client.0.smithi121.stdout:3/987: dread - d0/d3/de/da6/fb4 zero size 2024-04-05T21:26:55.308 INFO:tasks.workunit.client.0.smithi121.stdout:8/955: rename d12/d13/d56/d57/d105/d63/d8b/dd7/d123 to d12/d13/d134 0 2024-04-05T21:26:55.311 INFO:tasks.workunit.client.0.smithi121.stdout:8/956: creat d12/d13/dd3/df0/da7/f135 x:0 0 0 2024-04-05T21:26:55.342 INFO:tasks.workunit.client.0.smithi121.stdout:8/957: mkdir d12/d13/d56/d57/d105/d63/d8b/dd7/d101/d136 0 2024-04-05T21:26:55.343 INFO:tasks.workunit.client.0.smithi121.stdout:3/988: write d0/d3/d24/d3d/faf [1420302,119717] 0 2024-04-05T21:26:55.344 INFO:tasks.workunit.client.0.smithi121.stdout:8/958: mknod d12/d13/d56/d57/da4/dc9/c137 0 2024-04-05T21:26:55.345 INFO:tasks.workunit.client.0.smithi121.stdout:3/989: write d0/d3/d24/d3d/d1a/d1e/d86/f8b [965336,7179] 0 2024-04-05T21:26:55.347 INFO:tasks.workunit.client.0.smithi121.stdout:8/959: creat d12/d13/f138 x:0 0 0 2024-04-05T21:26:55.348 INFO:tasks.workunit.client.0.smithi121.stdout:8/960: chown d12/d13/d56/d57/da4/f116 61115 1 2024-04-05T21:26:55.350 INFO:tasks.workunit.client.0.smithi121.stdout:8/961: truncate d12/d13/f138 34296 0 2024-04-05T21:26:55.352 INFO:tasks.workunit.client.0.smithi121.stdout:3/990: dwrite d0/d3/d24/d3d/d1a/d1e/d86/fde [0,4194304] 0 2024-04-05T21:26:55.376 INFO:tasks.workunit.client.0.smithi121.stdout:8/962: rename cd to d12/d13/dd3/c139 0 2024-04-05T21:26:55.377 INFO:tasks.workunit.client.0.smithi121.stdout:3/991: chown d0/d3/d24/d3d/l1c 33 1 2024-04-05T21:26:55.378 INFO:tasks.workunit.client.0.smithi121.stdout:8/963: rmdir d12/d13/d56/d57/d5e 39 2024-04-05T21:26:55.382 INFO:tasks.workunit.client.0.smithi121.stdout:3/992: link d0/l79 d0/l153 0 2024-04-05T21:26:55.383 INFO:tasks.workunit.client.0.smithi121.stdout:8/964: mkdir d12/d13/d56/d57/d105/d87/deb/d12c/d13a 0 2024-04-05T21:26:55.385 INFO:tasks.workunit.client.0.smithi121.stdout:8/965: chown d12/d13/d56/d57/d5e/db9/ff9 1349177171 1 2024-04-05T21:26:55.386 INFO:tasks.workunit.client.0.smithi121.stdout:3/993: mknod d0/d3/c154 0 2024-04-05T21:26:55.388 INFO:tasks.workunit.client.0.smithi121.stdout:8/966: unlink d12/d13/d56/d57/f5a 0 2024-04-05T21:26:55.390 INFO:tasks.workunit.client.0.smithi121.stdout:3/994: mknod d0/d3/d24/d3d/d1a/d23/d90/d9e/c155 0 2024-04-05T21:26:55.394 INFO:tasks.workunit.client.0.smithi121.stdout:8/967: dwrite d12/d13/d56/d57/d105/d87/fc2 [0,4194304] 0 2024-04-05T21:26:55.394 INFO:tasks.workunit.client.0.smithi121.stdout:3/995: fdatasync d0/d3/d24/db3/f111 0 2024-04-05T21:26:55.395 INFO:tasks.workunit.client.0.smithi121.stdout:8/968: readlink d12/d13/dd3/df0/l115 0 2024-04-05T21:26:55.399 INFO:tasks.workunit.client.0.smithi121.stdout:8/969: creat d12/d13/dd3/df0/f13b x:0 0 0 2024-04-05T21:26:55.403 INFO:tasks.workunit.client.0.smithi121.stdout:3/996: truncate d0/d3/f5e 6568386 0 2024-04-05T21:26:55.407 INFO:tasks.workunit.client.0.smithi121.stdout:3/997: rmdir d0/d3/d24/d6d/d7b/d152 0 2024-04-05T21:26:55.407 INFO:tasks.workunit.client.0.smithi121.stdout:8/970: read d12/d50/f79 [4107297,40377] 0 2024-04-05T21:26:55.411 INFO:tasks.workunit.client.0.smithi121.stdout:8/971: creat d12/d13/d56/d57/d105/d63/d9c/f13c x:0 0 0 2024-04-05T21:26:55.412 INFO:tasks.workunit.client.0.smithi121.stdout:3/998: getdents d0/d3/d24/d6d/d7b/da3/dec/df9 0 2024-04-05T21:26:55.414 INFO:tasks.workunit.client.0.smithi121.stdout:8/972: rename d12/d13/f138 to d12/d13/dd3/f13d 0 2024-04-05T21:26:55.414 INFO:tasks.workunit.client.0.smithi121.stdout:8/973: chown d12/c14 1223 1 2024-04-05T21:26:55.549 INFO:tasks.workunit.client.0.smithi121.stdout:9/822: sync 2024-04-05T21:26:55.550 INFO:tasks.workunit.client.0.smithi121.stdout:9/823: write d6/d32/dbe/d7b/d49/f8b [7589066,84050] 0 2024-04-05T21:26:55.551 INFO:tasks.workunit.client.0.smithi121.stdout:9/824: write d6/df/d50/fb1 [1678517,221] 0 2024-04-05T21:26:55.552 INFO:tasks.workunit.client.0.smithi121.stdout:9/825: readlink d6/d32/l8e 0 2024-04-05T21:26:55.553 INFO:tasks.workunit.client.0.smithi121.stdout:9/826: dread - d6/df/d48/d96/dc2/ff9 zero size 2024-04-05T21:26:55.554 INFO:tasks.workunit.client.0.smithi121.stdout:9/827: symlink d6/d81/l115 0 2024-04-05T21:26:55.557 INFO:tasks.workunit.client.0.smithi121.stdout:9/828: unlink d6/d32/dbe/d7b/f3b 0 2024-04-05T21:26:55.559 INFO:tasks.workunit.client.0.smithi121.stdout:9/829: symlink d6/df/d48/d96/db9/d72/df1/l116 0 2024-04-05T21:26:55.574 INFO:tasks.workunit.client.0.smithi121.stdout:9/830: dread d6/df/d48/d96/db9/d72/f7f [0,4194304] 0 2024-04-05T21:26:55.575 INFO:tasks.workunit.client.0.smithi121.stdout:9/831: mknod d6/df/d104/c117 0 2024-04-05T21:26:55.576 INFO:tasks.workunit.client.0.smithi121.stdout:9/832: chown d6/d46/c97 50528 1 2024-04-05T21:26:55.577 INFO:tasks.workunit.client.0.smithi121.stdout:9/833: read d6/df/d48/d96/db9/f23 [1124987,95880] 0 2024-04-05T21:26:55.579 INFO:tasks.workunit.client.0.smithi121.stdout:9/834: creat d6/df/d48/d96/db2/f118 x:0 0 0 2024-04-05T21:26:55.580 INFO:tasks.workunit.client.0.smithi121.stdout:9/835: creat d6/df/d48/d96/f119 x:0 0 0 2024-04-05T21:26:55.583 INFO:tasks.workunit.client.0.smithi121.stdout:9/836: creat d6/df/d48/d96/db9/d2c/f11a x:0 0 0 2024-04-05T21:26:55.584 INFO:tasks.workunit.client.0.smithi121.stdout:9/837: write d6/df/d48/d96/dc2/d85/fe7 [112991,111874] 0 2024-04-05T21:26:55.593 INFO:tasks.workunit.client.0.smithi121.stdout:9/838: dwrite d6/df/d48/d96/db9/d2c/d53/f88 [0,4194304] 0 2024-04-05T21:26:55.599 INFO:tasks.workunit.client.0.smithi121.stdout:9/839: dwrite d6/d32/dbe/d7b/d6b/f78 [0,4194304] 0 2024-04-05T21:26:55.689 INFO:tasks.workunit.client.0.smithi121.stdout:3/999: sync 2024-04-05T21:26:55.784 INFO:tasks.workunit.client.0.smithi121.stdout:9/840: sync 2024-04-05T21:26:55.784 INFO:tasks.workunit.client.0.smithi121.stdout:8/974: sync 2024-04-05T21:26:55.786 INFO:tasks.workunit.client.0.smithi121.stdout:8/975: stat d12/d50/dbb/de5/lfe 0 2024-04-05T21:26:55.789 INFO:tasks.workunit.client.0.smithi121.stdout:8/976: getdents d12/d13/dd3/df0/da7/de1 0 2024-04-05T21:26:55.790 INFO:tasks.workunit.client.0.smithi121.stdout:9/841: dwrite d6/d32/dbe/d7b/d49/fb6 [0,4194304] 0 2024-04-05T21:26:55.793 INFO:tasks.workunit.client.0.smithi121.stdout:8/977: creat d12/d13/d56/d57/d5e/db9/f13e x:0 0 0 2024-04-05T21:26:55.794 INFO:tasks.workunit.client.0.smithi121.stdout:9/842: mknod d6/df/d48/c11b 0 2024-04-05T21:26:55.796 INFO:tasks.workunit.client.0.smithi121.stdout:8/978: unlink d12/f51 0 2024-04-05T21:26:55.797 INFO:tasks.workunit.client.0.smithi121.stdout:8/979: readlink d12/d13/d56/d57/d105/de6/l102 0 2024-04-05T21:26:55.800 INFO:tasks.workunit.client.0.smithi121.stdout:8/980: link d12/d13/d15/de7/f12f d12/d13/d134/f13f 0 2024-04-05T21:26:55.803 INFO:tasks.workunit.client.0.smithi121.stdout:9/843: dread d6/df/d48/d96/db9/d2c/d37/d3c/f7c [0,4194304] 0 2024-04-05T21:26:55.806 INFO:tasks.workunit.client.0.smithi121.stdout:9/844: truncate d6/df/d48/d96/db2/fd1 533463 0 2024-04-05T21:26:55.807 INFO:tasks.workunit.client.0.smithi121.stdout:8/981: dwrite d12/d13/dd3/df0/da7/f135 [0,4194304] 0 2024-04-05T21:26:55.808 INFO:tasks.workunit.client.0.smithi121.stdout:9/845: fsync d6/df/d48/d96/db9/d2c/d37/d3c/fde 0 2024-04-05T21:26:55.809 INFO:tasks.workunit.client.0.smithi121.stdout:9/846: chown d6/d32/fee 18 1 2024-04-05T21:26:55.812 INFO:tasks.workunit.client.0.smithi121.stdout:8/982: rename d12/d50/d61/d7f/l12a to d12/d50/dbb/l140 0 2024-04-05T21:26:55.813 INFO:tasks.workunit.client.0.smithi121.stdout:8/983: dread - d12/d13/d56/d57/d105/d63/d8b/fef zero size 2024-04-05T21:26:55.814 INFO:tasks.workunit.client.0.smithi121.stdout:8/984: chown d12/d13/dd3/df0/l69 5630 1 2024-04-05T21:26:55.815 INFO:tasks.workunit.client.0.smithi121.stdout:9/847: rmdir d6/df/d48/d96/dc2 39 2024-04-05T21:26:55.816 INFO:tasks.workunit.client.0.smithi121.stdout:9/848: chown d6/df/d48/d96 3394722 1 2024-04-05T21:26:55.817 INFO:tasks.workunit.client.0.smithi121.stdout:8/985: truncate d12/d13/d56/d57/d5e/db9/fc0 5082224 0 2024-04-05T21:26:55.820 INFO:tasks.workunit.client.0.smithi121.stdout:9/849: symlink d6/df/d48/d96/db9/l11c 0 2024-04-05T21:26:55.822 INFO:tasks.workunit.client.0.smithi121.stdout:9/850: symlink d6/df/d48/d96/db9/dfa/d114/l11d 0 2024-04-05T21:26:55.823 INFO:tasks.workunit.client.0.smithi121.stdout:9/851: fdatasync d6/df/d48/d96/db9/d2c/d53/fcc 0 2024-04-05T21:26:55.825 INFO:tasks.workunit.client.0.smithi121.stdout:8/986: write d12/d13/fd2 [2476567,126462] 0 2024-04-05T21:26:55.827 INFO:tasks.workunit.client.0.smithi121.stdout:9/852: unlink d6/df/d48/d96/dc2/d79/d9c/l10b 0 2024-04-05T21:26:55.829 INFO:tasks.workunit.client.0.smithi121.stdout:9/853: mknod d6/c11e 0 2024-04-05T21:26:55.830 INFO:tasks.workunit.client.0.smithi121.stdout:8/987: dread d12/d13/d56/d57/f78 [0,4194304] 0 2024-04-05T21:26:55.836 INFO:tasks.workunit.client.0.smithi121.stdout:9/854: dwrite d6/d32/dbe/d7b/f10e [0,4194304] 0 2024-04-05T21:26:55.837 INFO:tasks.workunit.client.0.smithi121.stdout:8/988: dwrite d12/d13/d56/d57/d105/d63/d8b/fcd [0,4194304] 0 2024-04-05T21:26:55.839 INFO:tasks.workunit.client.0.smithi121.stdout:9/855: dread - d6/df/d48/d96/db9/d2c/d53/fcc zero size 2024-04-05T21:26:55.839 INFO:tasks.workunit.client.0.smithi121.stdout:8/989: chown d12/d13/d56/d57/d105/d87/ddf 5493252 1 2024-04-05T21:26:55.841 INFO:tasks.workunit.client.0.smithi121.stdout:8/990: write d12/d50/d61/fbf [1349444,77316] 0 2024-04-05T21:26:55.849 INFO:tasks.workunit.client.0.smithi121.stdout:9/856: dread d6/f5e [0,4194304] 0 2024-04-05T21:26:55.852 INFO:tasks.workunit.client.0.smithi121.stdout:9/857: chown d6/df/d48/d96/db9/d2c/l6c 727589938 1 2024-04-05T21:26:55.865 INFO:tasks.workunit.client.0.smithi121.stdout:9/858: write d6/df/d48/d96/db9/f2d [5919331,113669] 0 2024-04-05T21:26:55.870 INFO:tasks.workunit.client.0.smithi121.stdout:9/859: link d6/d32/fee d6/d32/dbe/d7b/f11f 0 2024-04-05T21:26:55.872 INFO:tasks.workunit.client.0.smithi121.stdout:9/860: creat d6/df/d48/d96/dc2/d79/d9c/f120 x:0 0 0 2024-04-05T21:26:55.876 INFO:tasks.workunit.client.0.smithi121.stdout:9/861: mkdir d6/df/d48/d96/db9/d2c/d121 0 2024-04-05T21:26:55.878 INFO:tasks.workunit.client.0.smithi121.stdout:9/862: write d6/df/d48/d96/fce [575579,120828] 0 2024-04-05T21:26:55.888 INFO:tasks.workunit.client.0.smithi121.stdout:9/863: dwrite d6/d32/fcf [0,4194304] 0 2024-04-05T21:26:55.893 INFO:tasks.workunit.client.0.smithi121.stdout:9/864: symlink d6/df/d48/d96/db9/dfa/d114/l122 0 2024-04-05T21:26:55.898 INFO:tasks.workunit.client.0.smithi121.stdout:9/865: link d6/df/d50/c57 d6/d32/dbe/d7b/d6b/c123 0 2024-04-05T21:26:55.905 INFO:tasks.workunit.client.0.smithi121.stdout:9/866: dwrite d6/df/d48/d96/db9/d72/f7f [0,4194304] 0 2024-04-05T21:26:55.917 INFO:tasks.workunit.client.0.smithi121.stdout:9/867: dwrite d6/df/d48/d96/dc2/d85/fc7 [0,4194304] 0 2024-04-05T21:26:55.918 INFO:tasks.workunit.client.0.smithi121.stdout:9/868: chown d6/df/d48/d96/db9/d2c/d37/d3c/cdb 61174010 1 2024-04-05T21:26:55.929 INFO:tasks.workunit.client.0.smithi121.stdout:9/869: dwrite d6/d32/dbe/d7b/d49/d66/f107 [0,4194304] 0 2024-04-05T21:26:55.930 INFO:tasks.workunit.client.0.smithi121.stdout:9/870: chown d6/df/d48/d96/db9/d2c/l7a 0 1 2024-04-05T21:26:55.936 INFO:tasks.workunit.client.0.smithi121.stdout:9/871: link d6/d32/c38 d6/d46/d8a/c124 0 2024-04-05T21:26:55.937 INFO:tasks.workunit.client.0.smithi121.stdout:9/872: write d6/df/d48/d96/dc2/d79/d9c/f108 [1124403,1332] 0 2024-04-05T21:26:55.940 INFO:tasks.workunit.client.0.smithi121.stdout:9/873: symlink d6/df/d48/d96/db9/d2c/d121/l125 0 2024-04-05T21:26:55.941 INFO:tasks.workunit.client.0.smithi121.stdout:9/874: stat d6/df/d48/d96/db9/l11c 0 2024-04-05T21:26:55.947 INFO:tasks.workunit.client.0.smithi121.stdout:9/875: rename d6/df/d48/d96/dc2/d85/fa5 to d6/df/d48/d96/db9/d2c/d37/dab/f126 0 2024-04-05T21:26:55.950 INFO:tasks.workunit.client.0.smithi121.stdout:9/876: mkdir d6/df/d48/d96/db9/d2c/d53/d127 0 2024-04-05T21:26:55.952 INFO:tasks.workunit.client.0.smithi121.stdout:9/877: symlink d6/d32/dbe/d7b/d49/d66/l128 0 2024-04-05T21:26:55.954 INFO:tasks.workunit.client.0.smithi121.stdout:9/878: chown d6/df/l14 435 1 2024-04-05T21:26:55.961 INFO:tasks.workunit.client.0.smithi121.stdout:9/879: dwrite d6/d46/d8a/fdf [0,4194304] 0 2024-04-05T21:26:55.968 INFO:tasks.workunit.client.0.smithi121.stdout:9/880: dread d6/fd8 [0,4194304] 0 2024-04-05T21:26:55.977 INFO:tasks.workunit.client.0.smithi121.stdout:8/991: sync 2024-04-05T21:26:55.978 INFO:tasks.workunit.client.0.smithi121.stdout:9/881: dwrite d6/df/d48/d96/db9/d2c/d37/d3c/f7c [0,4194304] 0 2024-04-05T21:26:55.980 INFO:tasks.workunit.client.0.smithi121.stdout:8/992: dread - d12/d13/d56/d57/d105/d63/d9c/f13c zero size 2024-04-05T21:26:55.981 INFO:tasks.workunit.client.0.smithi121.stdout:8/993: dread - d12/d13/d56/d57/da4/f116 zero size 2024-04-05T21:26:55.984 INFO:tasks.workunit.client.0.smithi121.stdout:8/994: creat d12/d50/d61/d7f/d9a/d11e/f141 x:0 0 0 2024-04-05T21:26:55.986 INFO:tasks.workunit.client.0.smithi121.stdout:8/995: creat d12/d50/dbb/f142 x:0 0 0 2024-04-05T21:26:55.987 INFO:tasks.workunit.client.0.smithi121.stdout:9/882: write d6/df/d48/d96/db9/d2c/f24 [174083,30696] 0 2024-04-05T21:26:55.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:55 smithi121 ceph-mon[39723]: pgmap v26: 65 pgs: 65 active+clean; 4.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 139 MiB/s rd, 326 MiB/s wr, 653 op/s 2024-04-05T21:26:55.989 INFO:tasks.workunit.client.0.smithi121.stdout:9/883: write d6/df/d48/d96/db9/d2c/d37/fdc [4679926,40281] 0 2024-04-05T21:26:55.990 INFO:tasks.workunit.client.0.smithi121.stdout:8/996: mkdir d12/d13/d56/d57/d105/d63/d8b/dcc/d143 0 2024-04-05T21:26:55.994 INFO:tasks.workunit.client.0.smithi121.stdout:9/884: mknod d6/df/d48/d96/db9/dfa/c129 0 2024-04-05T21:26:55.997 INFO:tasks.workunit.client.0.smithi121.stdout:9/885: rename d6/df/d48/d96/db9/d72/df1/lf4 to d6/df/d48/d96/db9/d2c/d121/l12a 0 2024-04-05T21:26:56.002 INFO:tasks.workunit.client.0.smithi121.stdout:9/886: rename f2 to d6/df/d48/d96/db9/dfa/d114/f12b 0 2024-04-05T21:26:56.005 INFO:tasks.workunit.client.0.smithi121.stdout:8/997: dwrite d12/d13/d56/d57/d105/f90 [0,4194304] 0 2024-04-05T21:26:56.008 INFO:tasks.workunit.client.0.smithi121.stdout:9/887: getdents d6/d32 0 2024-04-05T21:26:56.009 INFO:tasks.workunit.client.0.smithi121.stdout:9/888: readlink d6/df/d48/d96/db9/d2c/d53/lb7 0 2024-04-05T21:26:56.011 INFO:tasks.workunit.client.0.smithi121.stdout:9/889: chown d6/df/d48/d96/db9/d72/l74 253483 1 2024-04-05T21:26:56.018 INFO:tasks.workunit.client.0.smithi121.stdout:9/890: dwrite d6/df/d48/d96/db9/d2c/d53/fcc [0,4194304] 0 2024-04-05T21:26:56.020 INFO:tasks.workunit.client.0.smithi121.stdout:9/891: write d6/df/d48/d96/db9/f23 [179483,25842] 0 2024-04-05T21:26:56.023 INFO:tasks.workunit.client.0.smithi121.stdout:9/892: write d6/df/d48/d96/dc2/d85/ff8 [302202,52959] 0 2024-04-05T21:26:56.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:55 smithi188 ceph-mon[49715]: pgmap v26: 65 pgs: 65 active+clean; 4.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 139 MiB/s rd, 326 MiB/s wr, 653 op/s 2024-04-05T21:26:56.030 INFO:tasks.workunit.client.0.smithi121.stdout:9/893: dwrite d6/df/d48/d96/db9/d2c/d37/fdc [0,4194304] 0 2024-04-05T21:26:56.039 INFO:tasks.workunit.client.0.smithi121.stdout:9/894: dwrite d6/df/d48/faa [0,4194304] 0 2024-04-05T21:26:56.040 INFO:tasks.workunit.client.0.smithi121.stdout:8/998: dread f3 [0,4194304] 0 2024-04-05T21:26:56.045 INFO:tasks.workunit.client.0.smithi121.stdout:9/895: mknod d6/d109/c12c 0 2024-04-05T21:26:56.045 INFO:tasks.workunit.client.0.smithi121.stdout:8/999: fsync d12/d50/d93/d96/fc3 0 2024-04-05T21:26:56.051 INFO:tasks.workunit.client.0.smithi121.stdout:9/896: getdents d6/df/d48/d96/db9 0 2024-04-05T21:26:56.057 INFO:tasks.workunit.client.0.smithi121.stdout:9/897: dread d6/df/d48/d96/dc2/d85/fc7 [0,4194304] 0 2024-04-05T21:26:56.062 INFO:tasks.workunit.client.0.smithi121.stdout:9/898: truncate d6/df/f71 957750 0 2024-04-05T21:26:56.065 INFO:tasks.workunit.client.0.smithi121.stdout:9/899: rename d6/df/d48/d96/db9/d72/l74 to d6/d81/l12d 0 2024-04-05T21:26:56.071 INFO:tasks.workunit.client.0.smithi121.stdout:9/900: dwrite d6/d32/dbe/d7b/d49/f8b [0,4194304] 0 2024-04-05T21:26:56.074 INFO:tasks.workunit.client.0.smithi121.stdout:9/901: mknod d6/df/d48/d96/dc2/d85/c12e 0 2024-04-05T21:26:56.079 INFO:tasks.workunit.client.0.smithi121.stdout:9/902: link d6/d32/dbe/d7b/d49/d66/f107 d6/d32/dbe/dba/def/df7/f12f 0 2024-04-05T21:26:56.081 INFO:tasks.workunit.client.0.smithi121.stdout:9/903: symlink d6/df/l130 0 2024-04-05T21:26:56.085 INFO:tasks.workunit.client.0.smithi121.stdout:9/904: dread d6/df/d48/d96/dc2/d85/fc7 [0,4194304] 0 2024-04-05T21:26:56.252 INFO:tasks.workunit.client.0.smithi121.stdout:9/905: sync 2024-04-05T21:26:56.254 INFO:tasks.workunit.client.0.smithi121.stdout:9/906: creat d6/df/d104/f131 x:0 0 0 2024-04-05T21:26:56.257 INFO:tasks.workunit.client.0.smithi121.stdout:9/907: link d6/d46/d8a/fdf d6/df/d48/d96/db9/d2c/de9/f132 0 2024-04-05T21:26:56.261 INFO:tasks.workunit.client.0.smithi121.stdout:9/908: unlink d6/df/d48/d96/db9/dfa/d114/f12b 0 2024-04-05T21:26:56.262 INFO:tasks.workunit.client.0.smithi121.stdout:9/909: dread - d6/df/d48/d96/dc2/ff9 zero size 2024-04-05T21:26:56.262 INFO:tasks.workunit.client.0.smithi121.stdout:9/910: chown d6/d109/c12c 1095463 1 2024-04-05T21:26:56.264 INFO:tasks.workunit.client.0.smithi121.stdout:9/911: fsync d6/df/d48/d96/db9/f33 0 2024-04-05T21:26:56.268 INFO:tasks.workunit.client.0.smithi121.stdout:9/912: write d6/d46/d8a/fdf [4875670,130033] 0 2024-04-05T21:26:56.274 INFO:tasks.workunit.client.0.smithi121.stdout:9/913: dwrite d6/df/d48/d96/db9/f2d [4194304,4194304] 0 2024-04-05T21:26:56.280 INFO:tasks.workunit.client.0.smithi121.stdout:9/914: mknod d6/df/d48/d96/db9/d2c/d121/c133 0 2024-04-05T21:26:56.283 INFO:tasks.workunit.client.0.smithi121.stdout:9/915: unlink d6/df/d48/d96/db9/d2c/l6c 0 2024-04-05T21:26:56.287 INFO:tasks.workunit.client.0.smithi121.stdout:9/916: unlink d6/df/d48/d96/db2/lf2 0 2024-04-05T21:26:56.290 INFO:tasks.workunit.client.0.smithi121.stdout:9/917: rmdir d6/df/d48/d96/dc2/d79/d9c 39 2024-04-05T21:26:56.488 INFO:tasks.workunit.client.0.smithi121.stdout:9/918: sync 2024-04-05T21:26:56.489 INFO:tasks.workunit.client.0.smithi121.stdout:9/919: chown d6/d46/d8a/le3 113 1 2024-04-05T21:26:56.491 INFO:tasks.workunit.client.0.smithi121.stdout:9/920: rename d6/df/l58 to d6/df/d48/d96/db9/d2c/d121/l134 0 2024-04-05T21:26:56.492 INFO:tasks.workunit.client.0.smithi121.stdout:9/921: write d6/df/d48/d96/db9/d2c/f11a [332220,72178] 0 2024-04-05T21:26:56.493 INFO:tasks.workunit.client.0.smithi121.stdout:9/922: truncate d6/df/f5f 4287349 0 2024-04-05T21:26:56.494 INFO:tasks.workunit.client.0.smithi121.stdout:9/923: fdatasync d6/df/d48/d96/dc2/ff9 0 2024-04-05T21:26:56.513 INFO:tasks.workunit.client.0.smithi121.stdout:9/924: write d6/d32/fee [819286,524] 0 2024-04-05T21:26:56.514 INFO:tasks.workunit.client.0.smithi121.stdout:9/925: truncate d6/df/d48/d96/db9/f76 4249792 0 2024-04-05T21:26:56.516 INFO:tasks.workunit.client.0.smithi121.stdout:9/926: symlink d6/df/d48/d96/dc2/d79/d94/l135 0 2024-04-05T21:26:56.517 INFO:tasks.workunit.client.0.smithi121.stdout:9/927: chown d6/df/d50/l102 1552 1 2024-04-05T21:26:56.518 INFO:tasks.workunit.client.0.smithi121.stdout:9/928: fdatasync d6/d32/dbe/d7b/d49/d66/f107 0 2024-04-05T21:26:56.520 INFO:tasks.workunit.client.0.smithi121.stdout:9/929: unlink d6/df/d104/f131 0 2024-04-05T21:26:56.523 INFO:tasks.workunit.client.0.smithi121.stdout:9/930: rename d6/d46/fac to d6/d32/f136 0 2024-04-05T21:26:56.709 INFO:tasks.workunit.client.0.smithi121.stdout:9/931: sync 2024-04-05T21:26:56.846 INFO:tasks.workunit.client.0.smithi121.stdout:9/932: sync 2024-04-05T21:26:56.849 INFO:tasks.workunit.client.0.smithi121.stdout:9/933: creat d6/df/d48/d96/db9/d2c/d37/dab/f137 x:0 0 0 2024-04-05T21:26:56.865 INFO:tasks.workunit.client.0.smithi121.stdout:9/934: dwrite d6/d32/dbe/d7b/d49/d66/fc1 [0,4194304] 0 2024-04-05T21:26:56.870 INFO:tasks.workunit.client.0.smithi121.stdout:9/935: dread d6/d32/fcf [0,4194304] 0 2024-04-05T21:26:56.885 INFO:tasks.workunit.client.0.smithi121.stdout:9/936: truncate d6/d32/dbe/d7b/f10e 2246701 0 2024-04-05T21:26:56.892 INFO:tasks.workunit.client.0.smithi121.stdout:9/937: link d6/df/l1b d6/df/d48/d96/dc2/ded/l138 0 2024-04-05T21:26:56.895 INFO:tasks.workunit.client.0.smithi121.stdout:9/938: fsync d6/df/d48/d96/dc2/d79/d9c/f108 0 2024-04-05T21:26:56.898 INFO:tasks.workunit.client.0.smithi121.stdout:9/939: unlink d6/d32/l8e 0 2024-04-05T21:26:56.900 INFO:tasks.workunit.client.0.smithi121.stdout:9/940: mkdir d6/d46/d8a/d139 0 2024-04-05T21:26:56.915 INFO:tasks.workunit.client.0.smithi121.stdout:9/941: dread d6/f15 [4194304,4194304] 0 2024-04-05T21:26:56.916 INFO:tasks.workunit.client.0.smithi121.stdout:9/942: write d6/df/d48/d96/db9/d2c/d37/d3c/ffc [806041,98733] 0 2024-04-05T21:26:56.921 INFO:tasks.workunit.client.0.smithi121.stdout:9/943: symlink d6/d32/dbe/d7b/l13a 0 2024-04-05T21:26:56.922 INFO:tasks.workunit.client.0.smithi121.stdout:9/944: write d6/d32/dbe/d7b/d6b/f92 [2144018,67097] 0 2024-04-05T21:26:56.926 INFO:tasks.workunit.client.0.smithi121.stdout:9/945: chown d6/df/d50/c57 0 1 2024-04-05T21:26:56.928 INFO:tasks.workunit.client.0.smithi121.stdout:9/946: dread d6/df/d48/d96/dc2/d85/fe7 [0,4194304] 0 2024-04-05T21:26:56.936 INFO:tasks.workunit.client.0.smithi121.stdout:9/947: rename d6/d32/dbe/d7b/d6b/f92 to d6/df/d48/d96/db9/d2c/f13b 0 2024-04-05T21:26:56.937 INFO:tasks.workunit.client.0.smithi121.stdout:9/948: write d6/df/d48/d96/dc2/d85/fc7 [2578165,46] 0 2024-04-05T21:26:56.938 INFO:tasks.workunit.client.0.smithi121.stdout:9/949: dread - d6/df/d48/d96/f111 zero size 2024-04-05T21:26:56.941 INFO:tasks.workunit.client.0.smithi121.stdout:9/950: mknod d6/df/d48/d96/dc2/d85/c13c 0 2024-04-05T21:26:56.942 INFO:tasks.workunit.client.0.smithi121.stdout:9/951: rename d6/df/d48 to d6/df/d48/d96/db9/dfa/d13d 22 2024-04-05T21:26:56.947 INFO:tasks.workunit.client.0.smithi121.stdout:9/952: rename d6/d32/l41 to d6/d32/dbe/d7b/l13e 0 2024-04-05T21:26:56.968 INFO:tasks.workunit.client.0.smithi121.stdout:9/953: truncate d6/df/d48/d96/db9/d2c/d37/d3c/f7c 326016 0 2024-04-05T21:26:56.970 INFO:tasks.workunit.client.0.smithi121.stdout:9/954: chown d6/df/d48/d96/db9/d2c/d37/dab 43007920 1 2024-04-05T21:26:56.971 INFO:tasks.workunit.client.0.smithi121.stdout:9/955: fdatasync d6/df/d48/d96/dc2/d85/fc7 0 2024-04-05T21:26:56.979 INFO:tasks.workunit.client.0.smithi121.stdout:9/956: dwrite d6/df/d48/d96/db9/d2c/d53/fcc [4194304,4194304] 0 2024-04-05T21:26:56.981 INFO:tasks.workunit.client.0.smithi121.stdout:9/957: chown d6/df/d48/d96/db9/d2c/d37/d3c/f9d 835936 1 2024-04-05T21:26:56.984 INFO:tasks.workunit.client.0.smithi121.stdout:9/958: mknod d6/d46/d8a/dd5/c13f 0 2024-04-05T21:26:56.988 INFO:tasks.workunit.client.0.smithi121.stdout:9/959: fsync d6/df/d48/d96/fce 0 2024-04-05T21:26:56.990 INFO:tasks.workunit.client.0.smithi121.stdout:9/960: mkdir d6/df/d50/d140 0 2024-04-05T21:26:56.994 INFO:tasks.workunit.client.0.smithi121.stdout:9/961: unlink d6/df/d48/d96/db9/d72/cff 0 2024-04-05T21:26:56.999 INFO:tasks.workunit.client.0.smithi121.stdout:9/962: unlink d6/df/d48/d96/db2/f118 0 2024-04-05T21:26:57.017 INFO:tasks.workunit.client.0.smithi121.stdout:9/963: truncate d6/df/d48/d96/db9/d2c/d37/d3c/f7c 218241 0 2024-04-05T21:26:57.023 INFO:tasks.workunit.client.0.smithi121.stdout:9/964: rename d6/df/d48/d96/db9/d2c/d121/l12a to d6/d46/d8a/dd5/l141 0 2024-04-05T21:26:57.160 INFO:tasks.workunit.client.0.smithi121.stdout:9/965: sync 2024-04-05T21:26:57.163 INFO:tasks.workunit.client.0.smithi121.stdout:9/966: creat d6/df/d48/d96/f142 x:0 0 0 2024-04-05T21:26:57.164 INFO:tasks.workunit.client.0.smithi121.stdout:9/967: write d6/d32/fcf [3910188,64781] 0 2024-04-05T21:26:57.166 INFO:tasks.workunit.client.0.smithi121.stdout:9/968: mkdir d6/df/d48/d96/db9/d2c/d53/d143 0 2024-04-05T21:26:57.170 INFO:tasks.workunit.client.0.smithi121.stdout:9/969: dread d6/fd8 [0,4194304] 0 2024-04-05T21:26:57.170 INFO:tasks.workunit.client.0.smithi121.stdout:9/970: stat d6/d46/d8a/dd5 0 2024-04-05T21:26:57.170 INFO:tasks.workunit.client.0.smithi121.stdout:9/971: creat d6/df/d48/d96/db9/d2c/d53/d127/f144 x:0 0 0 2024-04-05T21:26:57.172 INFO:tasks.workunit.client.0.smithi121.stdout:9/972: mknod d6/d32/dbe/d7b/d49/d66/c145 0 2024-04-05T21:26:57.172 INFO:tasks.workunit.client.0.smithi121.stdout:9/973: chown d6/df/d48/d96/db9/d2c/d53 1393 1 2024-04-05T21:26:57.174 INFO:tasks.workunit.client.0.smithi121.stdout:9/974: mknod d6/df/d48/c146 0 2024-04-05T21:26:57.175 INFO:tasks.workunit.client.0.smithi121.stdout:9/975: rmdir d6/d32/dbe/dba/def 39 2024-04-05T21:26:57.177 INFO:tasks.workunit.client.0.smithi121.stdout:9/976: creat d6/df/d48/d96/db9/dfa/f147 x:0 0 0 2024-04-05T21:26:57.178 INFO:tasks.workunit.client.0.smithi121.stdout:9/977: fdatasync d6/df/f71 0 2024-04-05T21:26:57.180 INFO:tasks.workunit.client.0.smithi121.stdout:9/978: creat d6/df/d48/d96/dc2/d79/d94/f148 x:0 0 0 2024-04-05T21:26:57.184 INFO:tasks.workunit.client.0.smithi121.stdout:9/979: creat d6/d46/d8a/d139/f149 x:0 0 0 2024-04-05T21:26:57.188 INFO:tasks.workunit.client.0.smithi121.stdout:9/980: rmdir d6/df/d48/d96/db9/d72/df1/d101 0 2024-04-05T21:26:57.190 INFO:tasks.workunit.client.0.smithi121.stdout:9/981: creat d6/df/d48/d96/db9/d2c/d37/dab/f14a x:0 0 0 2024-04-05T21:26:57.193 INFO:tasks.workunit.client.0.smithi121.stdout:9/982: rename d6/df/d48/d96/db9/dfa/f147 to d6/d109/f14b 0 2024-04-05T21:26:57.195 INFO:tasks.workunit.client.0.smithi121.stdout:9/983: creat d6/df/d48/d96/dc2/ded/f14c x:0 0 0 2024-04-05T21:26:57.197 INFO:tasks.workunit.client.0.smithi121.stdout:9/984: getdents d6/d81 0 2024-04-05T21:26:57.198 INFO:tasks.workunit.client.0.smithi121.stdout:9/985: symlink d6/df/d48/d96/db2/l14d 0 2024-04-05T21:26:57.204 INFO:tasks.workunit.client.0.smithi121.stdout:9/986: dwrite d6/df/d48/fec [0,4194304] 0 2024-04-05T21:26:57.206 INFO:tasks.workunit.client.0.smithi121.stdout:9/987: readlink d6/df/l1b 0 2024-04-05T21:26:57.207 INFO:tasks.workunit.client.0.smithi121.stdout:9/988: stat d6/df/d48/d96/db9/f2b 0 2024-04-05T21:26:57.214 INFO:tasks.workunit.client.0.smithi121.stdout:9/989: dwrite d6/df/d48/d96/dc2/d79/d9c/f120 [0,4194304] 0 2024-04-05T21:26:57.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:57.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:57.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:57.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:57.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:57.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:57.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:57.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:57.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: Adjusting osd_memory_target on smithi188 to 2251M 2024-04-05T21:26:57.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:57.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:26:57.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:26:57.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: Updating smithi121:/etc/ceph/ceph.conf 2024-04-05T21:26:57.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:56 smithi121 ceph-mon[39723]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-05T21:26:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-05T21:26:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: Adjusting osd_memory_target on smithi188 to 2251M 2024-04-05T21:26:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:57.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:26:57.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:26:57.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: Updating smithi121:/etc/ceph/ceph.conf 2024-04-05T21:26:57.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:56 smithi188 ceph-mon[49715]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-05T21:26:57.360 INFO:tasks.workunit.client.0.smithi121.stdout:9/990: sync 2024-04-05T21:26:57.369 INFO:tasks.workunit.client.0.smithi121.stdout:9/991: dwrite d6/f5e [0,4194304] 0 2024-04-05T21:26:57.375 INFO:tasks.workunit.client.0.smithi121.stdout:9/992: dwrite d6/d32/dbe/d7b/fb3 [0,4194304] 0 2024-04-05T21:26:57.378 INFO:tasks.workunit.client.0.smithi121.stdout:9/993: write d6/df/d48/d96/dc2/d85/fe7 [545343,128739] 0 2024-04-05T21:26:57.383 INFO:tasks.workunit.client.0.smithi121.stdout:9/994: rename d6/df/d48/d96/db9/dfa/c113 to d6/d32/dbe/d7b/d49/d66/c14e 0 2024-04-05T21:26:57.384 INFO:tasks.workunit.client.0.smithi121.stdout:9/995: unlink d6/d32/dbe/d7b/f11f 0 2024-04-05T21:26:57.388 INFO:tasks.workunit.client.0.smithi121.stdout:9/996: getdents d6/df/d48/d96/db9/d2c/d53/d127 0 2024-04-05T21:26:57.405 INFO:tasks.workunit.client.0.smithi121.stdout:9/997: dwrite d6/d46/fad [0,4194304] 0 2024-04-05T21:26:57.410 INFO:tasks.workunit.client.0.smithi121.stdout:9/998: dwrite d6/d32/dbe/d7b/d49/d66/fc1 [0,4194304] 0 2024-04-05T21:26:57.412 INFO:tasks.workunit.client.0.smithi121.stdout:9/999: chown d6/df/d48/cfb 30756440 1 2024-04-05T21:26:57.415 INFO:tasks.workunit.client.0.smithi121.stderr:+ rm -rf -- ./tmp.Yv7Bd6H50c 2024-04-05T21:26:58.413 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:58 smithi121 ceph-mon[39723]: Updating smithi121:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:26:58.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:58 smithi121 ceph-mon[39723]: Updating smithi188:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:26:58.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:58 smithi121 ceph-mon[39723]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-04-05T21:26:58.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:58 smithi121 ceph-mon[39723]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-05T21:26:58.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:58 smithi121 ceph-mon[39723]: pgmap v27: 65 pgs: 65 active+clean; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 100 MiB/s rd, 240 MiB/s wr, 516 op/s 2024-04-05T21:26:58.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:58 smithi121 ceph-mon[39723]: Updating smithi121:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T21:26:58.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:58 smithi121 ceph-mon[39723]: Updating smithi188:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T21:26:58.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:58 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:58.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:58 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:58.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:58 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:58.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:58 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:58.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:58 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:58.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:58 smithi188 ceph-mon[49715]: Updating smithi121:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:26:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:58 smithi188 ceph-mon[49715]: Updating smithi188:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:26:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:58 smithi188 ceph-mon[49715]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-04-05T21:26:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:58 smithi188 ceph-mon[49715]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-05T21:26:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:58 smithi188 ceph-mon[49715]: pgmap v27: 65 pgs: 65 active+clean; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 100 MiB/s rd, 240 MiB/s wr, 516 op/s 2024-04-05T21:26:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:58 smithi188 ceph-mon[49715]: Updating smithi121:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T21:26:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:58 smithi188 ceph-mon[49715]: Updating smithi188:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T21:26:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:58 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:58 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:58 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:58 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:58.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:58 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:26:59.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:59 smithi121 ceph-mon[39723]: Reconfiguring prometheus.smithi121 (dependencies changed)... 2024-04-05T21:26:59.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:26:59 smithi121 ceph-mon[39723]: Reconfiguring daemon prometheus.smithi121 on smithi121 2024-04-05T21:26:59.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:59 smithi188 ceph-mon[49715]: Reconfiguring prometheus.smithi121 (dependencies changed)... 2024-04-05T21:26:59.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:26:59 smithi188 ceph-mon[49715]: Reconfiguring daemon prometheus.smithi121 on smithi121 2024-04-05T21:27:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:00 smithi121 ceph-mon[39723]: pgmap v28: 65 pgs: 65 active+clean; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 90 MiB/s rd, 219 MiB/s wr, 482 op/s 2024-04-05T21:27:00.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:00 smithi188 ceph-mon[49715]: pgmap v28: 65 pgs: 65 active+clean; 4.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 90 MiB/s rd, 219 MiB/s wr, 482 op/s 2024-04-05T21:27:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:01 smithi121 ceph-mon[39723]: pgmap v29: 65 pgs: 65 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 94 MiB/s rd, 240 MiB/s wr, 594 op/s 2024-04-05T21:27:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:01 smithi188 ceph-mon[49715]: pgmap v29: 65 pgs: 65 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 94 MiB/s rd, 240 MiB/s wr, 594 op/s 2024-04-05T21:27:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.803+0000 7f510d993700 1 -- 172.21.15.121:0/4006783466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5108109040 msgr2=0x7f5108109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.803+0000 7f510d993700 1 --2- 172.21.15.121:0/4006783466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5108109040 0x7f5108109460 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f510000c210 tx=0x7f510000c520 comp rx=0 tx=0).stop 2024-04-05T21:27:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.803+0000 7f510d993700 1 -- 172.21.15.121:0/4006783466 shutdown_connections 2024-04-05T21:27:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.803+0000 7f510d993700 1 --2- 172.21.15.121:0/4006783466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5108107d90 0x7f51081081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.803+0000 7f510d993700 1 --2- 172.21.15.121:0/4006783466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5108109040 0x7f5108109460 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.803+0000 7f510d993700 1 -- 172.21.15.121:0/4006783466 >> 172.21.15.121:0/4006783466 conn(0x7f510806d9f0 msgr2=0x7f510806fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.803+0000 7f510d993700 1 -- 172.21.15.121:0/4006783466 shutdown_connections 2024-04-05T21:27:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.803+0000 7f510d993700 1 -- 172.21.15.121:0/4006783466 wait complete. 2024-04-05T21:27:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.803+0000 7f510d993700 1 Processor -- start 2024-04-05T21:27:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.804+0000 7f510d993700 1 -- start start 2024-04-05T21:27:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.804+0000 7f510d993700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5108107d90 0x7f510819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.804+0000 7f510d993700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5108109040 0x7f510819cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.804+0000 7f510d993700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510819d400 con 0x7f5108107d90 2024-04-05T21:27:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.804+0000 7f510d993700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510819d540 con 0x7f5108109040 2024-04-05T21:27:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.804+0000 7f5106ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5108107d90 0x7f510819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.804+0000 7f5106ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5108107d90 0x7f510819c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43398/0 (socket says 172.21.15.121:43398) 2024-04-05T21:27:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.804+0000 7f5106ffd700 1 -- 172.21.15.121:0/3705264198 learned_addr learned my addr 172.21.15.121:0/3705264198 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:27:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.804+0000 7f51067fc700 1 --2- 172.21.15.121:0/3705264198 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5108109040 0x7f510819cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.804+0000 7f5106ffd700 1 -- 172.21.15.121:0/3705264198 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5108109040 msgr2=0x7f510819cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.804+0000 7f5106ffd700 1 --2- 172.21.15.121:0/3705264198 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5108109040 0x7f510819cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.804+0000 7f5106ffd700 1 -- 172.21.15.121:0/3705264198 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510000ae10 con 0x7f5108107d90 2024-04-05T21:27:03.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.805+0000 7f5106ffd700 1 --2- 172.21.15.121:0/3705264198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5108107d90 0x7f510819c8a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f510000c9a0 tx=0x7f510000a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:03.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.805+0000 7f510c991700 1 -- 172.21.15.121:0/3705264198 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f510000f040 con 0x7f5108107d90 2024-04-05T21:27:03.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.805+0000 7f510d993700 1 -- 172.21.15.121:0/3705264198 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51081a1f90 con 0x7f5108107d90 2024-04-05T21:27:03.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.805+0000 7f510c991700 1 -- 172.21.15.121:0/3705264198 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f51000046a0 con 0x7f5108107d90 2024-04-05T21:27:03.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.805+0000 7f510c991700 1 -- 172.21.15.121:0/3705264198 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f510001e960 con 0x7f5108107d90 2024-04-05T21:27:03.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.805+0000 7f510d993700 1 -- 172.21.15.121:0/3705264198 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51081a2480 con 0x7f5108107d90 2024-04-05T21:27:03.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.806+0000 7f510d993700 1 -- 172.21.15.121:0/3705264198 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510810fbd0 con 0x7f5108107d90 2024-04-05T21:27:03.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.807+0000 7f510c991700 1 -- 172.21.15.121:0/3705264198 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 25) v1 ==== 90681+0+0 (secure 0 0 0) 0x7f510001a070 con 0x7f5108107d90 2024-04-05T21:27:03.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.808+0000 7f510c991700 1 --2- 172.21.15.121:0/3705264198 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f50f006ca70 0x7f50f006ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:03.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.808+0000 7f510c991700 1 -- 172.21.15.121:0/3705264198 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f510008ea70 con 0x7f5108107d90 2024-04-05T21:27:03.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.808+0000 7f51067fc700 1 --2- 172.21.15.121:0/3705264198 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f50f006ca70 0x7f50f006ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:03.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.813+0000 7f51067fc700 1 --2- 172.21.15.121:0/3705264198 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f50f006ca70 0x7f50f006ef30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f50fc005950 tx=0x7f50fc00b410 comp rx=0 tx=0).ready entity=mgr.24327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:03.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:03.813+0000 7f510c991700 1 -- 172.21.15.121:0/3705264198 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f510005c4b0 con 0x7f5108107d90 2024-04-05T21:27:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:03 smithi121 ceph-mon[39723]: pgmap v30: 65 pgs: 65 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 45 MiB/s rd, 107 MiB/s wr, 327 op/s 2024-04-05T21:27:04.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:03 smithi188 ceph-mon[49715]: pgmap v30: 65 pgs: 65 active+clean; 3.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 45 MiB/s rd, 107 MiB/s wr, 327 op/s 2024-04-05T21:27:04.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.145+0000 7f510d993700 1 -- 172.21.15.121:0/3705264198 --> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f510810d6f0 con 0x7f50f006ca70 2024-04-05T21:27:04.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.147+0000 7f510c991700 1 -- 172.21.15.121:0/3705264198 <== mgr.24327 v2:172.21.15.188:6828/2212394428 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f510810d6f0 con 0x7f50f006ca70 2024-04-05T21:27:04.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.149+0000 7f510d993700 1 -- 172.21.15.121:0/3705264198 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f50f006ca70 msgr2=0x7f50f006ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:04.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.149+0000 7f510d993700 1 --2- 172.21.15.121:0/3705264198 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f50f006ca70 0x7f50f006ef30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f50fc005950 tx=0x7f50fc00b410 comp rx=0 tx=0).stop 2024-04-05T21:27:04.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.149+0000 7f510d993700 1 -- 172.21.15.121:0/3705264198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5108107d90 msgr2=0x7f510819c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:04.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.149+0000 7f510d993700 1 --2- 172.21.15.121:0/3705264198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5108107d90 0x7f510819c8a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f510000c9a0 tx=0x7f510000a300 comp rx=0 tx=0).stop 2024-04-05T21:27:04.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.149+0000 7f510d993700 1 -- 172.21.15.121:0/3705264198 shutdown_connections 2024-04-05T21:27:04.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.149+0000 7f510d993700 1 --2- 172.21.15.121:0/3705264198 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f50f006ca70 0x7f50f006ef30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:04.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.149+0000 7f510d993700 1 --2- 172.21.15.121:0/3705264198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5108107d90 0x7f510819c8a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:04.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.149+0000 7f510d993700 1 --2- 172.21.15.121:0/3705264198 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5108109040 0x7f510819cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:04.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.149+0000 7f510d993700 1 -- 172.21.15.121:0/3705264198 >> 172.21.15.121:0/3705264198 conn(0x7f510806d9f0 msgr2=0x7f510810bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:04.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.149+0000 7f510d993700 1 -- 172.21.15.121:0/3705264198 shutdown_connections 2024-04-05T21:27:04.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.149+0000 7f510d993700 1 -- 172.21.15.121:0/3705264198 wait complete. 2024-04-05T21:27:04.165 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:27:04.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.274+0000 7f6deceed700 1 -- 172.21.15.121:0/1788198572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de8107d90 msgr2=0x7f6de81081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:04.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.274+0000 7f6deceed700 1 --2- 172.21.15.121:0/1788198572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de8107d90 0x7f6de81081f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009910 tx=0x7f6ddc009c20 comp rx=0 tx=0).stop 2024-04-05T21:27:04.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.274+0000 7f6deceed700 1 -- 172.21.15.121:0/1788198572 shutdown_connections 2024-04-05T21:27:04.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.274+0000 7f6deceed700 1 --2- 172.21.15.121:0/1788198572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de8107d90 0x7f6de81081f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:04.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.274+0000 7f6deceed700 1 --2- 172.21.15.121:0/1788198572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de8109040 0x7f6de8109460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:04.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.274+0000 7f6deceed700 1 -- 172.21.15.121:0/1788198572 >> 172.21.15.121:0/1788198572 conn(0x7f6de806d9f0 msgr2=0x7f6de806fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:04.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.274+0000 7f6deceed700 1 -- 172.21.15.121:0/1788198572 shutdown_connections 2024-04-05T21:27:04.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.274+0000 7f6deceed700 1 -- 172.21.15.121:0/1788198572 wait complete. 2024-04-05T21:27:04.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.275+0000 7f6deceed700 1 Processor -- start 2024-04-05T21:27:04.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.275+0000 7f6deceed700 1 -- start start 2024-04-05T21:27:04.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.275+0000 7f6deceed700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de8107d90 0x7f6de819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:04.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.275+0000 7f6deceed700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de8109040 0x7f6de819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:04.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.275+0000 7f6deceed700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de819f610 con 0x7f6de8107d90 2024-04-05T21:27:04.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.275+0000 7f6deceed700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de819f750 con 0x7f6de8109040 2024-04-05T21:27:04.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.276+0000 7f6de659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de8107d90 0x7f6de819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:04.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.276+0000 7f6de659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de8107d90 0x7f6de819eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43426/0 (socket says 172.21.15.121:43426) 2024-04-05T21:27:04.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.276+0000 7f6de659c700 1 -- 172.21.15.121:0/3508016939 learned_addr learned my addr 172.21.15.121:0/3508016939 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:27:04.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.276+0000 7f6de5d9b700 1 --2- 172.21.15.121:0/3508016939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de8109040 0x7f6de819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:04.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.276+0000 7f6de659c700 1 -- 172.21.15.121:0/3508016939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de8109040 msgr2=0x7f6de819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:04.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.276+0000 7f6de659c700 1 --2- 172.21.15.121:0/3508016939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de8109040 0x7f6de819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:04.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.276+0000 7f6de659c700 1 -- 172.21.15.121:0/3508016939 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc0095f0 con 0x7f6de8107d90 2024-04-05T21:27:04.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.276+0000 7f6de659c700 1 --2- 172.21.15.121:0/3508016939 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de8107d90 0x7f6de819eab0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800ea00 tx=0x7f6dd800ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:04.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.276+0000 7f6dd77fe700 1 -- 172.21.15.121:0/3508016939 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd800cca0 con 0x7f6de8107d90 2024-04-05T21:27:04.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.276+0000 7f6deceed700 1 -- 172.21.15.121:0/3508016939 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de81a4200 con 0x7f6de8107d90 2024-04-05T21:27:04.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.276+0000 7f6dd77fe700 1 -- 172.21.15.121:0/3508016939 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f6dd800ce00 con 0x7f6de8107d90 2024-04-05T21:27:04.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.276+0000 7f6deceed700 1 -- 172.21.15.121:0/3508016939 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de81a47d0 con 0x7f6de8107d90 2024-04-05T21:27:04.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.276+0000 7f6dd77fe700 1 -- 172.21.15.121:0/3508016939 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd8015390 con 0x7f6de8107d90 2024-04-05T21:27:04.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.278+0000 7f6deceed700 1 -- 172.21.15.121:0/3508016939 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de810fbd0 con 0x7f6de8107d90 2024-04-05T21:27:04.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.278+0000 7f6dd77fe700 1 -- 172.21.15.121:0/3508016939 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 25) v1 ==== 90681+0+0 (secure 0 0 0) 0x7f6dd8003fc0 con 0x7f6de8107d90 2024-04-05T21:27:04.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.278+0000 7f6dd77fe700 1 --2- 172.21.15.121:0/3508016939 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f6dd006ca20 0x7f6dd006eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:04.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.279+0000 7f6dd77fe700 1 -- 172.21.15.121:0/3508016939 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f6dd808c060 con 0x7f6de8107d90 2024-04-05T21:27:04.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.279+0000 7f6de5d9b700 1 --2- 172.21.15.121:0/3508016939 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f6dd006ca20 0x7f6dd006eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:04.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.279+0000 7f6de5d9b700 1 --2- 172.21.15.121:0/3508016939 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f6dd006ca20 0x7f6dd006eee0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009fd0 tx=0x7f6ddc00b5a0 comp rx=0 tx=0).ready entity=mgr.24327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:04.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.283+0000 7f6dd77fe700 1 -- 172.21.15.121:0/3508016939 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f6dd8059bc0 con 0x7f6de8107d90 2024-04-05T21:27:04.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.589+0000 7f6deceed700 1 -- 172.21.15.121:0/3508016939 --> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6de810d6f0 con 0x7f6dd006ca20 2024-04-05T21:27:04.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.590+0000 7f6dd77fe700 1 -- 172.21.15.121:0/3508016939 <== mgr.24327 v2:172.21.15.188:6828/2212394428 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f6de810d6f0 con 0x7f6dd006ca20 2024-04-05T21:27:04.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.594+0000 7f6deceed700 1 -- 172.21.15.121:0/3508016939 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f6dd006ca20 msgr2=0x7f6dd006eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:04.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.594+0000 7f6deceed700 1 --2- 172.21.15.121:0/3508016939 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f6dd006ca20 0x7f6dd006eee0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009fd0 tx=0x7f6ddc00b5a0 comp rx=0 tx=0).stop 2024-04-05T21:27:04.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.594+0000 7f6deceed700 1 -- 172.21.15.121:0/3508016939 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de8107d90 msgr2=0x7f6de819eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:04.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.594+0000 7f6deceed700 1 --2- 172.21.15.121:0/3508016939 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de8107d90 0x7f6de819eab0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800ea00 tx=0x7f6dd800ed10 comp rx=0 tx=0).stop 2024-04-05T21:27:04.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.594+0000 7f6deceed700 1 -- 172.21.15.121:0/3508016939 shutdown_connections 2024-04-05T21:27:04.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.594+0000 7f6deceed700 1 --2- 172.21.15.121:0/3508016939 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f6dd006ca20 0x7f6dd006eee0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:04.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.594+0000 7f6deceed700 1 --2- 172.21.15.121:0/3508016939 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de8107d90 0x7f6de819eab0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:04.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.594+0000 7f6deceed700 1 --2- 172.21.15.121:0/3508016939 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de8109040 0x7f6de819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:04.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.594+0000 7f6deceed700 1 -- 172.21.15.121:0/3508016939 >> 172.21.15.121:0/3508016939 conn(0x7f6de806d9f0 msgr2=0x7f6de810bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:04.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.594+0000 7f6deceed700 1 -- 172.21.15.121:0/3508016939 shutdown_connections 2024-04-05T21:27:04.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.595+0000 7f6deceed700 1 -- 172.21.15.121:0/3508016939 wait complete. 2024-04-05T21:27:04.701 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:04 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:04.701 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:04 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:04.701 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:04 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:27:04.701 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:04 smithi121 ceph-mon[39723]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:27:04.701 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:04 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:27:04.701 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:04 smithi121 ceph-mon[39723]: Upgrade: Need to upgrade myself (mgr.smithi188.tgcuir) 2024-04-05T21:27:04.701 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:04 smithi121 ceph-mon[39723]: Upgrade: Need to upgrade myself (mgr.smithi188.tgcuir) 2024-04-05T21:27:04.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.729+0000 7fd218e25700 1 -- 172.21.15.121:0/3508424615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2140ffad0 msgr2=0x7fd2140ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.729+0000 7fd218e25700 1 --2- 172.21.15.121:0/3508424615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2140ffad0 0x7fd2140ffef0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd204009910 tx=0x7fd204009c20 comp rx=0 tx=0).stop 2024-04-05T21:27:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.729+0000 7fd218e25700 1 -- 172.21.15.121:0/3508424615 shutdown_connections 2024-04-05T21:27:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.729+0000 7fd218e25700 1 --2- 172.21.15.121:0/3508424615 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd214100cd0 0x7fd214101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.729+0000 7fd218e25700 1 --2- 172.21.15.121:0/3508424615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2140ffad0 0x7fd2140ffef0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.729+0000 7fd218e25700 1 -- 172.21.15.121:0/3508424615 >> 172.21.15.121:0/3508424615 conn(0x7fd2140fb050 msgr2=0x7fd2140fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.729+0000 7fd218e25700 1 -- 172.21.15.121:0/3508424615 shutdown_connections 2024-04-05T21:27:04.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.729+0000 7fd218e25700 1 -- 172.21.15.121:0/3508424615 wait complete. 2024-04-05T21:27:04.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.730+0000 7fd218e25700 1 Processor -- start 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.730+0000 7fd218e25700 1 -- start start 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.730+0000 7fd218e25700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2140ffad0 0x7fd2141962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.730+0000 7fd218e25700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd214100cd0 0x7fd2141967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.730+0000 7fd218e25700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd214196e00 con 0x7fd2140ffad0 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.730+0000 7fd218e25700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd214196f40 con 0x7fd214100cd0 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.731+0000 7fd21259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2140ffad0 0x7fd2141962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.731+0000 7fd21259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2140ffad0 0x7fd2141962a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50284/0 (socket says 172.21.15.121:50284) 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.731+0000 7fd21259c700 1 -- 172.21.15.121:0/1174236639 learned_addr learned my addr 172.21.15.121:0/1174236639 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.731+0000 7fd211d9b700 1 --2- 172.21.15.121:0/1174236639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd214100cd0 0x7fd2141967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.731+0000 7fd21259c700 1 -- 172.21.15.121:0/1174236639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd214100cd0 msgr2=0x7fd2141967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.731+0000 7fd21259c700 1 --2- 172.21.15.121:0/1174236639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd214100cd0 0x7fd2141967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.731+0000 7fd21259c700 1 -- 172.21.15.121:0/1174236639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2040095f0 con 0x7fd2140ffad0 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.731+0000 7fd211d9b700 1 --2- 172.21.15.121:0/1174236639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd214100cd0 0x7fd2141967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-05T21:27:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.731+0000 7fd21259c700 1 --2- 172.21.15.121:0/1174236639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2140ffad0 0x7fd2141962a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd204000c00 tx=0x7fd20400bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.731+0000 7fd2037fe700 1 -- 172.21.15.121:0/1174236639 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd20401d070 con 0x7fd2140ffad0 2024-04-05T21:27:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.731+0000 7fd218e25700 1 -- 172.21.15.121:0/1174236639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd21419b990 con 0x7fd2140ffad0 2024-04-05T21:27:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.731+0000 7fd2037fe700 1 -- 172.21.15.121:0/1174236639 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fd204005710 con 0x7fd2140ffad0 2024-04-05T21:27:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.732+0000 7fd2037fe700 1 -- 172.21.15.121:0/1174236639 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2040039c0 con 0x7fd2140ffad0 2024-04-05T21:27:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.732+0000 7fd218e25700 1 -- 172.21.15.121:0/1174236639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd21419bea0 con 0x7fd2140ffad0 2024-04-05T21:27:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.733+0000 7fd2037fe700 1 -- 172.21.15.121:0/1174236639 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 25) v1 ==== 90681+0+0 (secure 0 0 0) 0x7fd204003b40 con 0x7fd2140ffad0 2024-04-05T21:27:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.733+0000 7fd2037fe700 1 --2- 172.21.15.121:0/1174236639 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fd1fc06ca10 0x7fd1fc06eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.734+0000 7fd2037fe700 1 -- 172.21.15.121:0/1174236639 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fd20408d7e0 con 0x7fd2140ffad0 2024-04-05T21:27:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.734+0000 7fd218e25700 1 -- 172.21.15.121:0/1174236639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2141084f0 con 0x7fd2140ffad0 2024-04-05T21:27:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.734+0000 7fd211d9b700 1 --2- 172.21.15.121:0/1174236639 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fd1fc06ca10 0x7fd1fc06eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.736+0000 7fd211d9b700 1 --2- 172.21.15.121:0/1174236639 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fd1fc06ca10 0x7fd1fc06eed0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd208011490 tx=0x7fd208009260 comp rx=0 tx=0).ready entity=mgr.24327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:04.739+0000 7fd2037fe700 1 -- 172.21.15.121:0/1174236639 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7fd204026080 con 0x7fd2140ffad0 2024-04-05T21:27:05.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:04 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:05.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:04 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:05.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:04 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:27:05.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:04 smithi188 ceph-mon[49715]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:27:05.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:04 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:27:05.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:04 smithi188 ceph-mon[49715]: Upgrade: Need to upgrade myself (mgr.smithi188.tgcuir) 2024-04-05T21:27:05.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:04 smithi188 ceph-mon[49715]: Upgrade: Need to upgrade myself (mgr.smithi188.tgcuir) 2024-04-05T21:27:05.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.086+0000 7fd218e25700 1 -- 172.21.15.121:0/1174236639 --> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd21419c540 con 0x7fd1fc06ca10 2024-04-05T21:27:05.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.102+0000 7fd2037fe700 1 -- 172.21.15.121:0/1174236639 <== mgr.24327 v2:172.21.15.188:6828/2212394428 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd21419c540 con 0x7fd1fc06ca10 2024-04-05T21:27:05.103 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:27:05.103 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (7m) 46s ago 9m 26.2M - 0.25.0 c8568f914cd2 6594a12b2bc8 2024-04-05T21:27:05.103 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (9m) 46s ago 9m 7726k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:27:05.103 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (8m) 21s ago 8m 7633k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:27:05.103 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (9m) 46s ago 9m 6651k - 18.2.1 5be31c24972a 5bf35fe2ace4 2024-04-05T21:27:05.103 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (8m) 21s ago 8m 6651k - 18.2.1 5be31c24972a dcdfa6ae17bf 2024-04-05T21:27:05.103 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (7m) 46s ago 9m 91.4M - 9.4.7 954c08fa6188 ee38366c66e3 2024-04-05T21:27:05.103 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (3m) 46s ago 3m 66.2M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:27:05.103 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (3m) 46s ago 3m 135M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (3m) 21s ago 3m 101M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (3m) 21s ago 3m 72.4M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:9283,8765,8443 running (10m) 46s ago 10m 168M - 18.2.1 5be31c24972a d8cb1548d844 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (68s) 21s ago 8m 482M - 18.2.2-1173-g8919605b 067bdb249792 2d759c011461 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (11m) 46s ago 11m 49.7M 2048M 18.2.1 5be31c24972a 3c30ce99c006 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (8m) 21s ago 8m 39.6M 2048M 18.2.1 5be31c24972a 8ecc9636670e 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (9m) 46s ago 9m 18.2M - 1.5.0 0da6a335fe13 5ac35283f3e9 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (8m) 21s ago 8m 16.5M - 1.5.0 0da6a335fe13 bc2ad104140f 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (7m) 46s ago 7m 117M 1227M 18.2.1 5be31c24972a f6f3a45429a2 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (6m) 46s ago 6m 138M 1227M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (6m) 46s ago 6m 106M 1227M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (5m) 21s ago 5m 134M 2251M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (5m) 21s ago 5m 125M 2251M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (4m) 21s ago 4m 114M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:27:05.104 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 starting - - - - 2024-04-05T21:27:05.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.104+0000 7fd218e25700 1 -- 172.21.15.121:0/1174236639 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fd1fc06ca10 msgr2=0x7fd1fc06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:05.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.104+0000 7fd218e25700 1 --2- 172.21.15.121:0/1174236639 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fd1fc06ca10 0x7fd1fc06eed0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd208011490 tx=0x7fd208009260 comp rx=0 tx=0).stop 2024-04-05T21:27:05.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.104+0000 7fd218e25700 1 -- 172.21.15.121:0/1174236639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2140ffad0 msgr2=0x7fd2141962a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:05.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.104+0000 7fd218e25700 1 --2- 172.21.15.121:0/1174236639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2140ffad0 0x7fd2141962a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd204000c00 tx=0x7fd20400bde0 comp rx=0 tx=0).stop 2024-04-05T21:27:05.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.104+0000 7fd218e25700 1 -- 172.21.15.121:0/1174236639 shutdown_connections 2024-04-05T21:27:05.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.104+0000 7fd218e25700 1 --2- 172.21.15.121:0/1174236639 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fd1fc06ca10 0x7fd1fc06eed0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:05.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.104+0000 7fd218e25700 1 --2- 172.21.15.121:0/1174236639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2140ffad0 0x7fd2141962a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:05.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.104+0000 7fd218e25700 1 --2- 172.21.15.121:0/1174236639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd214100cd0 0x7fd2141967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:05.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.104+0000 7fd218e25700 1 -- 172.21.15.121:0/1174236639 >> 172.21.15.121:0/1174236639 conn(0x7fd2140fb050 msgr2=0x7fd214068d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:05.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.104+0000 7fd218e25700 1 -- 172.21.15.121:0/1174236639 shutdown_connections 2024-04-05T21:27:05.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.104+0000 7fd218e25700 1 -- 172.21.15.121:0/1174236639 wait complete. 2024-04-05T21:27:05.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.229+0000 7f1cbff97700 1 -- 172.21.15.121:0/2078711508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb8100ce0 msgr2=0x7f1cb8101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:05.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.229+0000 7f1cbff97700 1 --2- 172.21.15.121:0/2078711508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb8100ce0 0x7f1cb8101140 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8009910 tx=0x7f1ca8009c20 comp rx=0 tx=0).stop 2024-04-05T21:27:05.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.230+0000 7f1cbff97700 1 -- 172.21.15.121:0/2078711508 shutdown_connections 2024-04-05T21:27:05.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.230+0000 7f1cbff97700 1 --2- 172.21.15.121:0/2078711508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb8100ce0 0x7f1cb8101140 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:05.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.230+0000 7f1cbff97700 1 --2- 172.21.15.121:0/2078711508 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cb80ffae0 0x7f1cb80fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:05.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.230+0000 7f1cbff97700 1 -- 172.21.15.121:0/2078711508 >> 172.21.15.121:0/2078711508 conn(0x7f1cb80fb080 msgr2=0x7f1cb80fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:05.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.230+0000 7f1cbff97700 1 -- 172.21.15.121:0/2078711508 shutdown_connections 2024-04-05T21:27:05.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.230+0000 7f1cbff97700 1 -- 172.21.15.121:0/2078711508 wait complete. 2024-04-05T21:27:05.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.230+0000 7f1cbff97700 1 Processor -- start 2024-04-05T21:27:05.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbff97700 1 -- start start 2024-04-05T21:27:05.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbff97700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb80ffae0 0x7f1cb81940a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:05.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbff97700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cb8100ce0 0x7f1cb81945e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:05.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbff97700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb8194c00 con 0x7f1cb80ffae0 2024-04-05T21:27:05.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbff97700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb8194d40 con 0x7f1cb8100ce0 2024-04-05T21:27:05.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbdd33700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb80ffae0 0x7f1cb81940a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:05.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbdd33700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb80ffae0 0x7f1cb81940a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50314/0 (socket says 172.21.15.121:50314) 2024-04-05T21:27:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbdd33700 1 -- 172.21.15.121:0/978975702 learned_addr learned my addr 172.21.15.121:0/978975702 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:27:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbd532700 1 --2- 172.21.15.121:0/978975702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cb8100ce0 0x7f1cb81945e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbdd33700 1 -- 172.21.15.121:0/978975702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cb8100ce0 msgr2=0x7f1cb81945e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbdd33700 1 --2- 172.21.15.121:0/978975702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cb8100ce0 0x7f1cb81945e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbdd33700 1 -- 172.21.15.121:0/978975702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca80095f0 con 0x7f1cb80ffae0 2024-04-05T21:27:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbdd33700 1 --2- 172.21.15.121:0/978975702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb80ffae0 0x7f1cb81940a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400d900 tx=0x7f1cb400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1caeffd700 1 -- 172.21.15.121:0/978975702 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cb400b920 con 0x7f1cb80ffae0 2024-04-05T21:27:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1caeffd700 1 -- 172.21.15.121:0/978975702 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f1cb4010460 con 0x7f1cb80ffae0 2024-04-05T21:27:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1cbff97700 1 -- 172.21.15.121:0/978975702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb81997f0 con 0x7f1cb80ffae0 2024-04-05T21:27:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.231+0000 7f1caeffd700 1 -- 172.21.15.121:0/978975702 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cb400f5d0 con 0x7f1cb80ffae0 2024-04-05T21:27:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.232+0000 7f1cbff97700 1 -- 172.21.15.121:0/978975702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb8199d40 con 0x7f1cb80ffae0 2024-04-05T21:27:05.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.233+0000 7f1cbff97700 1 -- 172.21.15.121:0/978975702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb8108500 con 0x7f1cb80ffae0 2024-04-05T21:27:05.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.233+0000 7f1caeffd700 1 -- 172.21.15.121:0/978975702 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 25) v1 ==== 90681+0+0 (secure 0 0 0) 0x7f1cb40105d0 con 0x7f1cb80ffae0 2024-04-05T21:27:05.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.234+0000 7f1caeffd700 1 --2- 172.21.15.121:0/978975702 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f1ca406ca20 0x7f1ca406eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:05.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.234+0000 7f1caeffd700 1 -- 172.21.15.121:0/978975702 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7f1cb408b4f0 con 0x7f1cb80ffae0 2024-04-05T21:27:05.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.234+0000 7f1cbd532700 1 --2- 172.21.15.121:0/978975702 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f1ca406ca20 0x7f1ca406eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:05.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.235+0000 7f1cbd532700 1 --2- 172.21.15.121:0/978975702 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f1ca406ca20 0x7f1ca406eee0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8006010 tx=0x7f1ca800b540 comp rx=0 tx=0).ready entity=mgr.24327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:05.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.242+0000 7f1caeffd700 1 -- 172.21.15.121:0/978975702 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f1cb4058fc0 con 0x7f1cb80ffae0 2024-04-05T21:27:05.383 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-05T21:27:05.383 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-04-05T21:27:05.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.608+0000 7f1cbff97700 1 -- 172.21.15.121:0/978975702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1cb819a3d0 con 0x7f1cb80ffae0 2024-04-05T21:27:05.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.609+0000 7f1caeffd700 1 -- 172.21.15.121:0/978975702 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+770 (secure 0 0 0) 0x7f1cb4058b50 con 0x7f1cb80ffae0 2024-04-05T21:27:05.610 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:27:05.610 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:27:05.610 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-05T21:27:05.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:27:05.611 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:27:05.611 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:27:05.611 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 1 2024-04-05T21:27:05.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:27:05.611 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:27:05.611 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-05T21:27:05.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:27:05.611 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:27:05.611 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:27:05.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:27:05.611 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:27:05.611 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 13, 2024-04-05T21:27:05.612 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 1 2024-04-05T21:27:05.612 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:27:05.612 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:27:05.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.613+0000 7f1cbff97700 1 -- 172.21.15.121:0/978975702 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f1ca406ca20 msgr2=0x7f1ca406eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:05.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.613+0000 7f1cbff97700 1 --2- 172.21.15.121:0/978975702 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f1ca406ca20 0x7f1ca406eee0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8006010 tx=0x7f1ca800b540 comp rx=0 tx=0).stop 2024-04-05T21:27:05.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.613+0000 7f1cbff97700 1 -- 172.21.15.121:0/978975702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb80ffae0 msgr2=0x7f1cb81940a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:05.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.613+0000 7f1cbff97700 1 --2- 172.21.15.121:0/978975702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb80ffae0 0x7f1cb81940a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400d900 tx=0x7f1cb400dc10 comp rx=0 tx=0).stop 2024-04-05T21:27:05.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.614+0000 7f1cbff97700 1 -- 172.21.15.121:0/978975702 shutdown_connections 2024-04-05T21:27:05.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.614+0000 7f1cbff97700 1 --2- 172.21.15.121:0/978975702 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7f1ca406ca20 0x7f1ca406eee0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:05.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.614+0000 7f1cbff97700 1 --2- 172.21.15.121:0/978975702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb80ffae0 0x7f1cb81940a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:05.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.614+0000 7f1cbff97700 1 --2- 172.21.15.121:0/978975702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1cb8100ce0 0x7f1cb81945e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:05.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.614+0000 7f1cbff97700 1 -- 172.21.15.121:0/978975702 >> 172.21.15.121:0/978975702 conn(0x7f1cb80fb080 msgr2=0x7f1cb8068d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:05.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.614+0000 7f1cbff97700 1 -- 172.21.15.121:0/978975702 shutdown_connections 2024-04-05T21:27:05.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.614+0000 7f1cbff97700 1 -- 172.21.15.121:0/978975702 wait complete. 2024-04-05T21:27:05.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.754+0000 7fb8a6507700 1 -- 172.21.15.121:0/3623572021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8a0100cb0 msgr2=0x7fb8a0101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:05.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.754+0000 7fb8a6507700 1 --2- 172.21.15.121:0/3623572021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8a0100cb0 0x7fb8a0101110 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb894009910 tx=0x7fb894009c20 comp rx=0 tx=0).stop 2024-04-05T21:27:05.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.754+0000 7fb8a6507700 1 -- 172.21.15.121:0/3623572021 shutdown_connections 2024-04-05T21:27:05.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.754+0000 7fb8a6507700 1 --2- 172.21.15.121:0/3623572021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8a0100cb0 0x7fb8a0101110 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:05.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.754+0000 7fb8a6507700 1 --2- 172.21.15.121:0/3623572021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a00ffab0 0x7fb8a00ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:05.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.754+0000 7fb8a6507700 1 -- 172.21.15.121:0/3623572021 >> 172.21.15.121:0/3623572021 conn(0x7fb8a00fb050 msgr2=0x7fb8a00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:05.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.754+0000 7fb8a6507700 1 -- 172.21.15.121:0/3623572021 shutdown_connections 2024-04-05T21:27:05.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.754+0000 7fb8a6507700 1 -- 172.21.15.121:0/3623572021 wait complete. 2024-04-05T21:27:05.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.755+0000 7fb8a6507700 1 Processor -- start 2024-04-05T21:27:05.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.755+0000 7fb8a6507700 1 -- start start 2024-04-05T21:27:05.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.755+0000 7fb8a6507700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8a00ffab0 0x7fb8a0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:05.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.755+0000 7fb8a6507700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a0100cb0 0x7fb8a0198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:05.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.755+0000 7fb8a6507700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a01990b0 con 0x7fb8a0100cb0 2024-04-05T21:27:05.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.755+0000 7fb8a6507700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a01991f0 con 0x7fb8a00ffab0 2024-04-05T21:27:05.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.755+0000 7fb89ffff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8a00ffab0 0x7fb8a0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:05.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.755+0000 7fb89ffff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8a00ffab0 0x7fb8a0198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.121:47254/0 (socket says 172.21.15.121:47254) 2024-04-05T21:27:05.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.755+0000 7fb89ffff700 1 -- 172.21.15.121:0/2486349578 learned_addr learned my addr 172.21.15.121:0/2486349578 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:27:05.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.756+0000 7fb89f7fe700 1 --2- 172.21.15.121:0/2486349578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a0100cb0 0x7fb8a0198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:05.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.756+0000 7fb89ffff700 1 -- 172.21.15.121:0/2486349578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a0100cb0 msgr2=0x7fb8a0198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:05.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.756+0000 7fb89ffff700 1 --2- 172.21.15.121:0/2486349578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a0100cb0 0x7fb8a0198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:05.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.756+0000 7fb89ffff700 1 -- 172.21.15.121:0/2486349578 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8940095f0 con 0x7fb8a00ffab0 2024-04-05T21:27:05.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.756+0000 7fb89ffff700 1 --2- 172.21.15.121:0/2486349578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8a00ffab0 0x7fb8a0198550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb89000b480 tx=0x7fb89000b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:05.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.756+0000 7fb89d7fa700 1 -- 172.21.15.121:0/2486349578 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb890011800 con 0x7fb8a00ffab0 2024-04-05T21:27:05.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.756+0000 7fb8a6507700 1 -- 172.21.15.121:0/2486349578 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a019dca0 con 0x7fb8a00ffab0 2024-04-05T21:27:05.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.756+0000 7fb89d7fa700 1 -- 172.21.15.121:0/2486349578 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fb890011df0 con 0x7fb8a00ffab0 2024-04-05T21:27:05.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.756+0000 7fb89d7fa700 1 -- 172.21.15.121:0/2486349578 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb890010e30 con 0x7fb8a00ffab0 2024-04-05T21:27:05.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.756+0000 7fb8a6507700 1 -- 172.21.15.121:0/2486349578 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a019e220 con 0x7fb8a00ffab0 2024-04-05T21:27:05.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.757+0000 7fb8a6507700 1 -- 172.21.15.121:0/2486349578 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a0107460 con 0x7fb8a00ffab0 2024-04-05T21:27:05.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.759+0000 7fb89d7fa700 1 -- 172.21.15.121:0/2486349578 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 25) v1 ==== 90681+0+0 (secure 0 0 0) 0x7fb890011960 con 0x7fb8a00ffab0 2024-04-05T21:27:05.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.759+0000 7fb89d7fa700 1 --2- 172.21.15.121:0/2486349578 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fb88806ca10 0x7fb88806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:05.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.759+0000 7fb89d7fa700 1 -- 172.21.15.121:0/2486349578 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fb89008ba70 con 0x7fb8a00ffab0 2024-04-05T21:27:05.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.759+0000 7fb89f7fe700 1 --2- 172.21.15.121:0/2486349578 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fb88806ca10 0x7fb88806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:05.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.760+0000 7fb89f7fe700 1 --2- 172.21.15.121:0/2486349578 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fb88806ca10 0x7fb88806eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb894006010 tx=0x7fb89400b560 comp rx=0 tx=0).ready entity=mgr.24327 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:05.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:05.765+0000 7fb89d7fa700 1 -- 172.21.15.121:0/2486349578 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7fb8900596e0 con 0x7fb8a00ffab0 2024-04-05T21:27:05.992 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:05 smithi121 ceph-mon[39723]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:05.992 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:05 smithi121 ceph-mon[39723]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:05.992 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:05 smithi121 ceph-mon[39723]: pgmap v31: 65 pgs: 65 active+clean; 2.2 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 46 MiB/s rd, 108 MiB/s wr, 441 op/s 2024-04-05T21:27:05.992 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:05 smithi121 ceph-mon[39723]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:05.993 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:05 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:05.993 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:05 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi121.etbquh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T21:27:05.993 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:05 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi121.etbquh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T21:27:05.993 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:05 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T21:27:05.993 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:05 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:27:05.993 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:05 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/978975702' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:27:06.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:05 smithi188 ceph-mon[49715]: from='client.14638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:06.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:05 smithi188 ceph-mon[49715]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:06.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:05 smithi188 ceph-mon[49715]: pgmap v31: 65 pgs: 65 active+clean; 2.2 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 46 MiB/s rd, 108 MiB/s wr, 441 op/s 2024-04-05T21:27:06.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:05 smithi188 ceph-mon[49715]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:06.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:05 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:06.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:05 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi121.etbquh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T21:27:06.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:05 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi121.etbquh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T21:27:06.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:05 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T21:27:06.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:05 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:27:06.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:05 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/978975702' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:27:06.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:06.127+0000 7fb8a6507700 1 -- 172.21.15.121:0/2486349578 --> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8a0105600 con 0x7fb88806ca10 2024-04-05T21:27:06.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:06.128+0000 7fb89d7fa700 1 -- 172.21.15.121:0/2486349578 <== mgr.24327 v2:172.21.15.188:6828/2212394428 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fb8a0105600 con 0x7fb88806ca10 2024-04-05T21:27:06.129 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:27:06.129 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:27:06.129 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:27:06.129 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T21:27:06.129 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [], 2024-04-05T21:27:06.129 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "1/2 daemons upgraded", 2024-04-05T21:27:06.130 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mgr daemons", 2024-04-05T21:27:06.130 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:27:06.130 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:27:06.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:06.130+0000 7fb8a6507700 1 -- 172.21.15.121:0/2486349578 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fb88806ca10 msgr2=0x7fb88806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:06.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:06.130+0000 7fb8a6507700 1 --2- 172.21.15.121:0/2486349578 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fb88806ca10 0x7fb88806eed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb894006010 tx=0x7fb89400b560 comp rx=0 tx=0).stop 2024-04-05T21:27:06.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:06.130+0000 7fb8a6507700 1 -- 172.21.15.121:0/2486349578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8a00ffab0 msgr2=0x7fb8a0198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:06.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:06.130+0000 7fb8a6507700 1 --2- 172.21.15.121:0/2486349578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8a00ffab0 0x7fb8a0198550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb89000b480 tx=0x7fb89000b790 comp rx=0 tx=0).stop 2024-04-05T21:27:06.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:06.130+0000 7fb8a6507700 1 -- 172.21.15.121:0/2486349578 shutdown_connections 2024-04-05T21:27:06.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:06.130+0000 7fb8a6507700 1 --2- 172.21.15.121:0/2486349578 >> [v2:172.21.15.188:6828/2212394428,v1:172.21.15.188:6829/2212394428] conn(0x7fb88806ca10 0x7fb88806eed0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:06.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:06.130+0000 7fb8a6507700 1 --2- 172.21.15.121:0/2486349578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb8a00ffab0 0x7fb8a0198550 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:06.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:06.130+0000 7fb8a6507700 1 --2- 172.21.15.121:0/2486349578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a0100cb0 0x7fb8a0198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:06.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:06.130+0000 7fb8a6507700 1 -- 172.21.15.121:0/2486349578 >> 172.21.15.121:0/2486349578 conn(0x7fb8a00fb050 msgr2=0x7fb8a0103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:06.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:06.130+0000 7fb8a6507700 1 -- 172.21.15.121:0/2486349578 shutdown_connections 2024-04-05T21:27:06.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:06.130+0000 7fb8a6507700 1 -- 172.21.15.121:0/2486349578 wait complete. 2024-04-05T21:27:06.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:06 smithi121 ceph-mon[39723]: Upgrade: Updating mgr.smithi121.etbquh 2024-04-05T21:27:06.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:06 smithi121 ceph-mon[39723]: Deploying daemon mgr.smithi121.etbquh on smithi121 2024-04-05T21:27:06.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:06 smithi121 ceph-mon[39723]: from='client.24395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:07.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:06 smithi188 ceph-mon[49715]: Upgrade: Updating mgr.smithi121.etbquh 2024-04-05T21:27:07.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:06 smithi188 ceph-mon[49715]: Deploying daemon mgr.smithi121.etbquh on smithi121 2024-04-05T21:27:07.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:06 smithi188 ceph-mon[49715]: from='client.24395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:07.719 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:07 smithi121 ceph-mon[39723]: pgmap v32: 65 pgs: 65 active+clean; 1.9 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 12 MiB/s rd, 40 MiB/s wr, 343 op/s 2024-04-05T21:27:08.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:07 smithi188 ceph-mon[49715]: pgmap v32: 65 pgs: 65 active+clean; 1.9 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 12 MiB/s rd, 40 MiB/s wr, 343 op/s 2024-04-05T21:27:08.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:08 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:27:09.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:08 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:27:09.471 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-05T21:27:09.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-05T21:27:09.722 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:09 smithi121 ceph-mon[39723]: pgmap v33: 65 pgs: 65 active+clean; 1.9 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 5.8 MiB/s rd, 22 MiB/s wr, 237 op/s 2024-04-05T21:27:10.002 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:09 smithi188 ceph-mon[49715]: pgmap v33: 65 pgs: 65 active+clean; 1.9 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 5.8 MiB/s rd, 22 MiB/s wr, 237 op/s 2024-04-05T21:27:10.827 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:10 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:11.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:10 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:11.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:10 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:27:11.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:10 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:11.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:10 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:10 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:11.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:10 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:11.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:10 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:27:11.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:10 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:11.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:10 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:12.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:11 smithi121 ceph-mon[39723]: pgmap v34: 65 pgs: 65 active+clean; 655 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 6.4 MiB/s rd, 23 MiB/s wr, 370 op/s 2024-04-05T21:27:12.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:11 smithi188 ceph-mon[49715]: pgmap v34: 65 pgs: 65 active+clean; 655 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 6.4 MiB/s rd, 23 MiB/s wr, 370 op/s 2024-04-05T21:27:13.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:13 smithi121 ceph-mon[39723]: pgmap v35: 65 pgs: 65 active+clean; 655 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 257 op/s 2024-04-05T21:27:14.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:13 smithi188 ceph-mon[49715]: pgmap v35: 65 pgs: 65 active+clean; 655 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 257 op/s 2024-04-05T21:27:16.203 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:15 smithi121 ceph-mon[39723]: pgmap v36: 65 pgs: 65 active+clean; 355 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 364 op/s 2024-04-05T21:27:16.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:15 smithi188 ceph-mon[49715]: pgmap v36: 65 pgs: 65 active+clean; 355 MiB data, 1.6 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 364 op/s 2024-04-05T21:27:17.869 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:17 smithi121 ceph-mon[39723]: pgmap v37: 65 pgs: 65 active+clean; 288 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 294 op/s 2024-04-05T21:27:17.870 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:17 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:17.870 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:17 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:17.870 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:17 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:17.870 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:17 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:18.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:17 smithi188 ceph-mon[49715]: pgmap v37: 65 pgs: 65 active+clean; 288 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 294 op/s 2024-04-05T21:27:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:17 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:17 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:17 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:17 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:18 smithi121 ceph-mon[39723]: Standby manager daemon smithi121.etbquh restarted 2024-04-05T21:27:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:18 smithi121 ceph-mon[39723]: Standby manager daemon smithi121.etbquh started 2024-04-05T21:27:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:18 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi121.etbquh/crt"}]: dispatch 2024-04-05T21:27:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:18 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T21:27:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:18 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi121.etbquh/key"}]: dispatch 2024-04-05T21:27:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:18 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T21:27:19.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:18 smithi188 ceph-mon[49715]: Standby manager daemon smithi121.etbquh restarted 2024-04-05T21:27:19.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:18 smithi188 ceph-mon[49715]: Standby manager daemon smithi121.etbquh started 2024-04-05T21:27:19.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:18 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi121.etbquh/crt"}]: dispatch 2024-04-05T21:27:19.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:18 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T21:27:19.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:18 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi121.etbquh/key"}]: dispatch 2024-04-05T21:27:19.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:18 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T21:27:19.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:19 smithi121 ceph-mon[39723]: pgmap v38: 65 pgs: 65 active+clean; 288 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 283 op/s 2024-04-05T21:27:19.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:19 smithi121 ceph-mon[39723]: mgrmap e26: smithi188.tgcuir(active, since 70s), standbys: smithi121.etbquh 2024-04-05T21:27:19.995 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:19 smithi188 ceph-mon[49715]: pgmap v38: 65 pgs: 65 active+clean; 288 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 283 op/s 2024-04-05T21:27:19.996 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:19 smithi188 ceph-mon[49715]: mgrmap e26: smithi188.tgcuir(active, since 70s), standbys: smithi121.etbquh 2024-04-05T21:27:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:21 smithi121 ceph-mon[39723]: pgmap v39: 65 pgs: 65 active+clean; 292 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 451 op/s 2024-04-05T21:27:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:21 smithi188 ceph-mon[49715]: pgmap v39: 65 pgs: 65 active+clean; 292 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 451 op/s 2024-04-05T21:27:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:23 smithi121 ceph-mon[39723]: pgmap v40: 65 pgs: 65 active+clean; 292 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 319 op/s 2024-04-05T21:27:24.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:23 smithi188 ceph-mon[49715]: pgmap v40: 65 pgs: 65 active+clean; 292 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 319 op/s 2024-04-05T21:27:24.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:24 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:27:25.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:24 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:27:25.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:25 smithi121 ceph-mon[39723]: pgmap v41: 65 pgs: 65 active+clean; 292 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 436 op/s 2024-04-05T21:27:26.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:25 smithi188 ceph-mon[49715]: pgmap v41: 65 pgs: 65 active+clean; 292 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 436 op/s 2024-04-05T21:27:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:27 smithi121 ceph-mon[39723]: pgmap v42: 65 pgs: 65 active+clean; 296 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 368 op/s 2024-04-05T21:27:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:27 smithi121 ceph-mon[39723]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:27:28.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:27 smithi188 ceph-mon[49715]: pgmap v42: 65 pgs: 65 active+clean; 296 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 368 op/s 2024-04-05T21:27:28.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:27 smithi188 ceph-mon[49715]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:27:29.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:29 smithi121 ceph-mon[39723]: pgmap v43: 65 pgs: 65 active+clean; 296 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 324 op/s 2024-04-05T21:27:29.934 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:29 smithi188 ceph-mon[49715]: pgmap v43: 65 pgs: 65 active+clean; 296 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 324 op/s 2024-04-05T21:27:31.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:31.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:31.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:27:31.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:27:31.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:31.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:27:31.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mgr fail", "who": "smithi188.tgcuir"}]: dispatch 2024-04-05T21:27:31.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mgr fail", "who": "smithi188.tgcuir"}]: dispatch 2024-04-05T21:27:31.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: osdmap e48: 6 total, 6 up, 6 in 2024-04-05T21:27:31.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd='[{"prefix": "mgr fail", "who": "smithi188.tgcuir"}]': finished 2024-04-05T21:27:31.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: mgrmap e27: smithi121.etbquh(active, starting, since 0.00648169s) 2024-04-05T21:27:31.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:27:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:27:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T21:27:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:27:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T21:27:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.latnft"}]: dispatch 2024-04-05T21:27:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr metadata", "who": "smithi121.etbquh", "id": "smithi121.etbquh"}]: dispatch 2024-04-05T21:27:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:27:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:27:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:27:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:27:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:27:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:27:31.243 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T21:27:31.243 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T21:27:31.243 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T21:27:31.243 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: Manager daemon smithi121.etbquh is now available 2024-04-05T21:27:31.243 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:27:31.243 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:27:31.243 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.etbquh/mirror_snapshot_schedule"}]: dispatch 2024-04-05T21:27:31.243 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.etbquh/trash_purge_schedule"}]: dispatch 2024-04-05T21:27:31.243 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:30 smithi121 ceph-mon[39723]: [05/Apr/2024:21:27:30] ENGINE Bus STARTING 2024-04-05T21:27:31.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:31.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:31.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:27:31.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:27:31.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' 2024-04-05T21:27:31.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:27:31.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.24327 172.21.15.188:0/49319054' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mgr fail", "who": "smithi188.tgcuir"}]: dispatch 2024-04-05T21:27:31.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "mgr fail", "who": "smithi188.tgcuir"}]: dispatch 2024-04-05T21:27:31.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: osdmap e48: 6 total, 6 up, 6 in 2024-04-05T21:27:31.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.24327 ' entity='mgr.smithi188.tgcuir' cmd='[{"prefix": "mgr fail", "who": "smithi188.tgcuir"}]': finished 2024-04-05T21:27:31.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: mgrmap e27: smithi121.etbquh(active, starting, since 0.00648169s) 2024-04-05T21:27:31.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:27:31.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:27:31.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T21:27:31.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:27:31.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T21:27:31.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.latnft"}]: dispatch 2024-04-05T21:27:31.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr metadata", "who": "smithi121.etbquh", "id": "smithi121.etbquh"}]: dispatch 2024-04-05T21:27:31.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:27:31.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:27:31.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:27:31.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:27:31.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:27:31.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:27:31.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-05T21:27:31.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-05T21:27:31.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-05T21:27:31.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: Manager daemon smithi121.etbquh is now available 2024-04-05T21:27:31.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:27:31.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:27:31.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.etbquh/mirror_snapshot_schedule"}]: dispatch 2024-04-05T21:27:31.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.etbquh/trash_purge_schedule"}]: dispatch 2024-04-05T21:27:31.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:30 smithi188 ceph-mon[49715]: [05/Apr/2024:21:27:30] ENGINE Bus STARTING 2024-04-05T21:27:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:32 smithi121 ceph-mon[39723]: [05/Apr/2024:21:27:30] ENGINE Serving on https://172.21.15.121:7150 2024-04-05T21:27:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:32 smithi121 ceph-mon[39723]: [05/Apr/2024:21:27:30] ENGINE Serving on http://172.21.15.121:8765 2024-04-05T21:27:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:32 smithi121 ceph-mon[39723]: [05/Apr/2024:21:27:30] ENGINE Bus STARTED 2024-04-05T21:27:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:32 smithi121 ceph-mon[39723]: mgrmap e28: smithi121.etbquh(active, since 1.00811s) 2024-04-05T21:27:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:32 smithi121 ceph-mon[39723]: pgmap v3: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:27:32.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:32 smithi188 ceph-mon[49715]: [05/Apr/2024:21:27:30] ENGINE Serving on https://172.21.15.121:7150 2024-04-05T21:27:32.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:32 smithi188 ceph-mon[49715]: [05/Apr/2024:21:27:30] ENGINE Serving on http://172.21.15.121:8765 2024-04-05T21:27:32.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:32 smithi188 ceph-mon[49715]: [05/Apr/2024:21:27:30] ENGINE Bus STARTED 2024-04-05T21:27:32.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:32 smithi188 ceph-mon[49715]: mgrmap e28: smithi121.etbquh(active, since 1.00811s) 2024-04-05T21:27:32.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:32 smithi188 ceph-mon[49715]: pgmap v3: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:27:33.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:33 smithi121 ceph-mon[39723]: pgmap v4: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:27:33.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:33 smithi121 ceph-mon[39723]: mgrmap e29: smithi121.etbquh(active, since 2s) 2024-04-05T21:27:33.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:33 smithi188 ceph-mon[49715]: pgmap v4: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:27:33.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:33 smithi188 ceph-mon[49715]: mgrmap e29: smithi121.etbquh(active, since 2s) 2024-04-05T21:27:35.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:35 smithi188 ceph-mon[49715]: pgmap v5: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:27:35.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:35 smithi188 ceph-mon[49715]: mgrmap e30: smithi121.etbquh(active, since 4s) 2024-04-05T21:27:35.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:35 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:35.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:35 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:35.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:35 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:35.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:35 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:35 smithi121 ceph-mon[39723]: pgmap v5: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:27:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:35 smithi121 ceph-mon[39723]: mgrmap e30: smithi121.etbquh(active, since 4s) 2024-04-05T21:27:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:35 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:35 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:35 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:35 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:36.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.258+0000 7f54b3155700 1 -- 172.21.15.121:0/988614498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54ac100bb0 msgr2=0x7f54ac100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:36.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.258+0000 7f54b3155700 1 --2- 172.21.15.121:0/988614498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54ac100bb0 0x7f54ac100fd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f54a0009910 tx=0x7f54a0009c20 comp rx=0 tx=0).stop 2024-04-05T21:27:36.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.258+0000 7f54b3155700 1 -- 172.21.15.121:0/988614498 shutdown_connections 2024-04-05T21:27:36.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.258+0000 7f54b3155700 1 --2- 172.21.15.121:0/988614498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54ac101db0 0x7f54ac102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:36.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.258+0000 7f54b3155700 1 --2- 172.21.15.121:0/988614498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54ac100bb0 0x7f54ac100fd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:36.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.258+0000 7f54b3155700 1 -- 172.21.15.121:0/988614498 >> 172.21.15.121:0/988614498 conn(0x7f54ac0fc110 msgr2=0x7f54ac0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:36.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.258+0000 7f54b3155700 1 -- 172.21.15.121:0/988614498 shutdown_connections 2024-04-05T21:27:36.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.258+0000 7f54b3155700 1 -- 172.21.15.121:0/988614498 wait complete. 2024-04-05T21:27:36.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.259+0000 7f54b3155700 1 Processor -- start 2024-04-05T21:27:36.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.259+0000 7f54b3155700 1 -- start start 2024-04-05T21:27:36.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.259+0000 7f54b3155700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54ac100bb0 0x7f54ac196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:36.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.259+0000 7f54b3155700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54ac101db0 0x7f54ac1967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:36.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.259+0000 7f54b3155700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac196dd0 con 0x7f54ac101db0 2024-04-05T21:27:36.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.259+0000 7f54b3155700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ac196f10 con 0x7f54ac100bb0 2024-04-05T21:27:36.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.260+0000 7f54abfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54ac101db0 0x7f54ac1967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:36.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.260+0000 7f54b0ef1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54ac100bb0 0x7f54ac196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:36.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.260+0000 7f54abfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54ac101db0 0x7f54ac1967b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34744/0 (socket says 172.21.15.121:34744) 2024-04-05T21:27:36.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.260+0000 7f54b0ef1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54ac100bb0 0x7f54ac196270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.121:57516/0 (socket says 172.21.15.121:57516) 2024-04-05T21:27:36.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.260+0000 7f54abfff700 1 -- 172.21.15.121:0/1619930882 learned_addr learned my addr 172.21.15.121:0/1619930882 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:27:36.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.260+0000 7f54abfff700 1 -- 172.21.15.121:0/1619930882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54ac100bb0 msgr2=0x7f54ac196270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:36.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.260+0000 7f54abfff700 1 --2- 172.21.15.121:0/1619930882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54ac100bb0 0x7f54ac196270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:36.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.260+0000 7f54abfff700 1 -- 172.21.15.121:0/1619930882 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a00095f0 con 0x7f54ac101db0 2024-04-05T21:27:36.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.260+0000 7f54abfff700 1 --2- 172.21.15.121:0/1619930882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54ac101db0 0x7f54ac1967b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f549800d900 tx=0x7f549800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:36.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.260+0000 7f54a9ffb700 1 -- 172.21.15.121:0/1619930882 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54980041f0 con 0x7f54ac101db0 2024-04-05T21:27:36.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.260+0000 7f54b3155700 1 -- 172.21.15.121:0/1619930882 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54ac19b9c0 con 0x7f54ac101db0 2024-04-05T21:27:36.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.261+0000 7f54a9ffb700 1 -- 172.21.15.121:0/1619930882 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f5498004cc0 con 0x7f54ac101db0 2024-04-05T21:27:36.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.261+0000 7f54a9ffb700 1 -- 172.21.15.121:0/1619930882 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5498005660 con 0x7f54ac101db0 2024-04-05T21:27:36.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.261+0000 7f54b3155700 1 -- 172.21.15.121:0/1619930882 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54ac19bf10 con 0x7f54ac101db0 2024-04-05T21:27:36.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.262+0000 7f54b3155700 1 -- 172.21.15.121:0/1619930882 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54ac109720 con 0x7f54ac101db0 2024-04-05T21:27:36.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.262+0000 7f54a9ffb700 1 -- 172.21.15.121:0/1619930882 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 30) v1 ==== 45821+0+0 (secure 0 0 0) 0x7f5498004750 con 0x7f54ac101db0 2024-04-05T21:27:36.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.262+0000 7f54a9ffb700 1 --2- 172.21.15.121:0/1619930882 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f549c038a90 0x7f549c03af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:36.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.262+0000 7f54a9ffb700 1 -- 172.21.15.121:0/1619930882 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f549804ca20 con 0x7f54ac101db0 2024-04-05T21:27:36.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.263+0000 7f54b0ef1700 1 --2- 172.21.15.121:0/1619930882 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f549c038a90 0x7f549c03af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:36.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.263+0000 7f54b0ef1700 1 --2- 172.21.15.121:0/1619930882 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f549c038a90 0x7f549c03af50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f54a0006010 tx=0x7f54a0005ec0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:36.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.268+0000 7f54a9ffb700 1 -- 172.21.15.121:0/1619930882 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f5498022920 con 0x7f54ac101db0 2024-04-05T21:27:36.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.575+0000 7f54b3155700 1 -- 172.21.15.121:0/1619930882 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f54ac106700 con 0x7f549c038a90 2024-04-05T21:27:36.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.576+0000 7f54a9ffb700 1 -- 172.21.15.121:0/1619930882 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f54ac106700 con 0x7f549c038a90 2024-04-05T21:27:36.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.579+0000 7f54b3155700 1 -- 172.21.15.121:0/1619930882 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f549c038a90 msgr2=0x7f549c03af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:36.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.579+0000 7f54b3155700 1 --2- 172.21.15.121:0/1619930882 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f549c038a90 0x7f549c03af50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f54a0006010 tx=0x7f54a0005ec0 comp rx=0 tx=0).stop 2024-04-05T21:27:36.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.579+0000 7f54b3155700 1 -- 172.21.15.121:0/1619930882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54ac101db0 msgr2=0x7f54ac1967b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:36.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.579+0000 7f54b3155700 1 --2- 172.21.15.121:0/1619930882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54ac101db0 0x7f54ac1967b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f549800d900 tx=0x7f549800dc10 comp rx=0 tx=0).stop 2024-04-05T21:27:36.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.580+0000 7f54b3155700 1 -- 172.21.15.121:0/1619930882 shutdown_connections 2024-04-05T21:27:36.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.580+0000 7f54b3155700 1 --2- 172.21.15.121:0/1619930882 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f549c038a90 0x7f549c03af50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:36.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.580+0000 7f54b3155700 1 --2- 172.21.15.121:0/1619930882 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54ac100bb0 0x7f54ac196270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:36.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.580+0000 7f54b3155700 1 --2- 172.21.15.121:0/1619930882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54ac101db0 0x7f54ac1967b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:36.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.580+0000 7f54b3155700 1 -- 172.21.15.121:0/1619930882 >> 172.21.15.121:0/1619930882 conn(0x7f54ac0fc110 msgr2=0x7f54ac104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:36.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.580+0000 7f54b3155700 1 -- 172.21.15.121:0/1619930882 shutdown_connections 2024-04-05T21:27:36.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.580+0000 7f54b3155700 1 -- 172.21.15.121:0/1619930882 wait complete. 2024-04-05T21:27:36.596 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:27:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.742+0000 7f345b312700 1 -- 172.21.15.121:0/1376398395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3454108400 msgr2=0x7f3454108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.742+0000 7f345b312700 1 --2- 172.21.15.121:0/1376398395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3454108400 0x7f3454108860 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3450009910 tx=0x7f3450009c20 comp rx=0 tx=0).stop 2024-04-05T21:27:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.742+0000 7f345b312700 1 -- 172.21.15.121:0/1376398395 shutdown_connections 2024-04-05T21:27:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.742+0000 7f345b312700 1 --2- 172.21.15.121:0/1376398395 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3454108400 0x7f3454108860 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.742+0000 7f345b312700 1 --2- 172.21.15.121:0/1376398395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3454107200 0x7f3454107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.742+0000 7f345b312700 1 -- 172.21.15.121:0/1376398395 >> 172.21.15.121:0/1376398395 conn(0x7f34540761b0 msgr2=0x7f3454078610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.742+0000 7f345b312700 1 -- 172.21.15.121:0/1376398395 shutdown_connections 2024-04-05T21:27:36.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.743+0000 7f345b312700 1 -- 172.21.15.121:0/1376398395 wait complete. 2024-04-05T21:27:36.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.743+0000 7f345b312700 1 Processor -- start 2024-04-05T21:27:36.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.743+0000 7f345b312700 1 -- start start 2024-04-05T21:27:36.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.744+0000 7f345b312700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3454107200 0x7f345419ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:36.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.744+0000 7f345b312700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3454108400 0x7f345419cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:36.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.744+0000 7f345b312700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345419d5f0 con 0x7f3454108400 2024-04-05T21:27:36.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.744+0000 7f345b312700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345419d730 con 0x7f3454107200 2024-04-05T21:27:36.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.744+0000 7f34588ad700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3454108400 0x7f345419cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:36.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.744+0000 7f34588ad700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3454108400 0x7f345419cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34764/0 (socket says 172.21.15.121:34764) 2024-04-05T21:27:36.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.744+0000 7f34588ad700 1 -- 172.21.15.121:0/979006972 learned_addr learned my addr 172.21.15.121:0/979006972 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:27:36.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.745+0000 7f34588ad700 1 -- 172.21.15.121:0/979006972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3454107200 msgr2=0x7f345419ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:36.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.745+0000 7f34590ae700 1 --2- 172.21.15.121:0/979006972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3454107200 0x7f345419ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:36.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.745+0000 7f34588ad700 1 --2- 172.21.15.121:0/979006972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3454107200 0x7f345419ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:36.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.745+0000 7f34588ad700 1 -- 172.21.15.121:0/979006972 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34500095f0 con 0x7f3454108400 2024-04-05T21:27:36.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.745+0000 7f34590ae700 1 --2- 172.21.15.121:0/979006972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3454107200 0x7f345419ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:27:36.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.745+0000 7f34588ad700 1 --2- 172.21.15.121:0/979006972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3454108400 0x7f345419cfd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3450005b40 tx=0x7f345000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:36.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.748+0000 7f344a7fc700 1 -- 172.21.15.121:0/979006972 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f345001d070 con 0x7f3454108400 2024-04-05T21:27:36.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.748+0000 7f344a7fc700 1 -- 172.21.15.121:0/979006972 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f3450022470 con 0x7f3454108400 2024-04-05T21:27:36.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.748+0000 7f344a7fc700 1 -- 172.21.15.121:0/979006972 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f345000f690 con 0x7f3454108400 2024-04-05T21:27:36.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.748+0000 7f345b312700 1 -- 172.21.15.121:0/979006972 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34541a2180 con 0x7f3454108400 2024-04-05T21:27:36.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.748+0000 7f345b312700 1 -- 172.21.15.121:0/979006972 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34541a2670 con 0x7f3454108400 2024-04-05T21:27:36.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.749+0000 7f345b312700 1 -- 172.21.15.121:0/979006972 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345410fcc0 con 0x7f3454108400 2024-04-05T21:27:36.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.759+0000 7f344a7fc700 1 -- 172.21.15.121:0/979006972 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 30) v1 ==== 45821+0+0 (secure 0 0 0) 0x7f3450004af0 con 0x7f3454108400 2024-04-05T21:27:36.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.760+0000 7f344a7fc700 1 --2- 172.21.15.121:0/979006972 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3440038b30 0x7f344003aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:36.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.760+0000 7f344a7fc700 1 -- 172.21.15.121:0/979006972 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f345004ea90 con 0x7f3454108400 2024-04-05T21:27:36.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.760+0000 7f344a7fc700 1 -- 172.21.15.121:0/979006972 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f345007ba50 con 0x7f3454108400 2024-04-05T21:27:36.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.764+0000 7f34590ae700 1 --2- 172.21.15.121:0/979006972 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3440038b30 0x7f344003aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:36.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:36.764+0000 7f34590ae700 1 --2- 172.21.15.121:0/979006972 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3440038b30 0x7f344003aff0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3454108260 tx=0x7f3444008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:37.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.125+0000 7f345b312700 1 -- 172.21.15.121:0/979006972 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f345410cd50 con 0x7f3440038b30 2024-04-05T21:27:37.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.127+0000 7f344a7fc700 1 -- 172.21.15.121:0/979006972 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f345410cd50 con 0x7f3440038b30 2024-04-05T21:27:37.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.130+0000 7f345b312700 1 -- 172.21.15.121:0/979006972 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3440038b30 msgr2=0x7f344003aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:37.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.130+0000 7f345b312700 1 --2- 172.21.15.121:0/979006972 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3440038b30 0x7f344003aff0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3454108260 tx=0x7f3444008040 comp rx=0 tx=0).stop 2024-04-05T21:27:37.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.130+0000 7f345b312700 1 -- 172.21.15.121:0/979006972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3454108400 msgr2=0x7f345419cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:37.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.130+0000 7f345b312700 1 --2- 172.21.15.121:0/979006972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3454108400 0x7f345419cfd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3450005b40 tx=0x7f345000be20 comp rx=0 tx=0).stop 2024-04-05T21:27:37.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.130+0000 7f345b312700 1 -- 172.21.15.121:0/979006972 shutdown_connections 2024-04-05T21:27:37.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.130+0000 7f345b312700 1 --2- 172.21.15.121:0/979006972 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3440038b30 0x7f344003aff0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.130+0000 7f345b312700 1 --2- 172.21.15.121:0/979006972 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3454107200 0x7f345419ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.130+0000 7f345b312700 1 --2- 172.21.15.121:0/979006972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3454108400 0x7f345419cfd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.130+0000 7f345b312700 1 -- 172.21.15.121:0/979006972 >> 172.21.15.121:0/979006972 conn(0x7f34540761b0 msgr2=0x7f345410b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:37.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.130+0000 7f345b312700 1 -- 172.21.15.121:0/979006972 shutdown_connections 2024-04-05T21:27:37.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.130+0000 7f345b312700 1 -- 172.21.15.121:0/979006972 wait complete. 2024-04-05T21:27:37.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.279+0000 7f008a609700 1 -- 172.21.15.121:0/782082994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0084103f70 msgr2=0x7f0084106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:37.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.279+0000 7f008a609700 1 --2- 172.21.15.121:0/782082994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0084103f70 0x7f0084106360 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0078009910 tx=0x7f0078009c20 comp rx=0 tx=0).stop 2024-04-05T21:27:37.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.279+0000 7f008a609700 1 -- 172.21.15.121:0/782082994 shutdown_connections 2024-04-05T21:27:37.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.279+0000 7f008a609700 1 --2- 172.21.15.121:0/782082994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0084103f70 0x7f0084106360 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.279+0000 7f008a609700 1 --2- 172.21.15.121:0/782082994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0084101640 0x7f0084103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.279+0000 7f008a609700 1 -- 172.21.15.121:0/782082994 >> 172.21.15.121:0/782082994 conn(0x7f00840fb030 msgr2=0x7f00840fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:37.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.279+0000 7f008a609700 1 -- 172.21.15.121:0/782082994 shutdown_connections 2024-04-05T21:27:37.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.279+0000 7f008a609700 1 -- 172.21.15.121:0/782082994 wait complete. 2024-04-05T21:27:37.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.280+0000 7f008a609700 1 Processor -- start 2024-04-05T21:27:37.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.280+0000 7f008a609700 1 -- start start 2024-04-05T21:27:37.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.281+0000 7f008a609700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0084101640 0x7f0084196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:37.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.281+0000 7f008a609700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0084103f70 0x7f0084196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:37.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.281+0000 7f008a609700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0084196db0 con 0x7f0084101640 2024-04-05T21:27:37.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.281+0000 7f008a609700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0084196ef0 con 0x7f0084103f70 2024-04-05T21:27:37.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.281+0000 7f0083fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0084101640 0x7f0084196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:37.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.281+0000 7f0083fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0084101640 0x7f0084196250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34780/0 (socket says 172.21.15.121:34780) 2024-04-05T21:27:37.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.281+0000 7f0083fff700 1 -- 172.21.15.121:0/1952107558 learned_addr learned my addr 172.21.15.121:0/1952107558 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:27:37.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.281+0000 7f00837fe700 1 --2- 172.21.15.121:0/1952107558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0084103f70 0x7f0084196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:37.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.281+0000 7f0083fff700 1 -- 172.21.15.121:0/1952107558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0084103f70 msgr2=0x7f0084196790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:37.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.281+0000 7f0083fff700 1 --2- 172.21.15.121:0/1952107558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0084103f70 0x7f0084196790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.281+0000 7f0083fff700 1 -- 172.21.15.121:0/1952107558 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00780095f0 con 0x7f0084101640 2024-04-05T21:27:37.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.281+0000 7f0083fff700 1 --2- 172.21.15.121:0/1952107558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0084101640 0x7f0084196250 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f007400e920 tx=0x7f007400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:37.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.281+0000 7f00817fa700 1 -- 172.21.15.121:0/1952107558 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f007400c9d0 con 0x7f0084101640 2024-04-05T21:27:37.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.282+0000 7f00817fa700 1 -- 172.21.15.121:0/1952107558 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f0074010430 con 0x7f0084101640 2024-04-05T21:27:37.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.282+0000 7f00817fa700 1 -- 172.21.15.121:0/1952107558 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0074018630 con 0x7f0084101640 2024-04-05T21:27:37.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.282+0000 7f008a609700 1 -- 172.21.15.121:0/1952107558 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f008419b9a0 con 0x7f0084101640 2024-04-05T21:27:37.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.282+0000 7f008a609700 1 -- 172.21.15.121:0/1952107558 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008419bf20 con 0x7f0084101640 2024-04-05T21:27:37.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.283+0000 7f00817fa700 1 -- 172.21.15.121:0/1952107558 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 30) v1 ==== 45821+0+0 (secure 0 0 0) 0x7f00740188f0 con 0x7f0084101640 2024-04-05T21:27:37.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.283+0000 7f00817fa700 1 --2- 172.21.15.121:0/1952107558 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f006c038a90 0x7f006c03af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:37.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.283+0000 7f00817fa700 1 -- 172.21.15.121:0/1952107558 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f0074014070 con 0x7f0084101640 2024-04-05T21:27:37.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.283+0000 7f00837fe700 1 --2- 172.21.15.121:0/1952107558 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f006c038a90 0x7f006c03af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:37.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.283+0000 7f00837fe700 1 --2- 172.21.15.121:0/1952107558 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f006c038a90 0x7f006c03af50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0078009fd0 tx=0x7f007800b5a0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:37.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.284+0000 7f008a609700 1 -- 172.21.15.121:0/1952107558 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00841903e0 con 0x7f0084101640 2024-04-05T21:27:37.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.291+0000 7f00817fa700 1 -- 172.21.15.121:0/1952107558 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f007401f920 con 0x7f0084101640 2024-04-05T21:27:37.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.590+0000 7f008a609700 1 -- 172.21.15.121:0/1952107558 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f00840611d0 con 0x7f006c038a90 2024-04-05T21:27:37.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.603+0000 7f00817fa700 1 -- 172.21.15.121:0/1952107558 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f00840611d0 con 0x7f006c038a90 2024-04-05T21:27:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:27:37.604 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (8m) 0s ago 9m 26.2M - 0.25.0 c8568f914cd2 6594a12b2bc8 2024-04-05T21:27:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (10m) 0s ago 10m 7826k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:27:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (9m) 3s ago 9m 7746k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:27:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (10m) 0s ago 10m 6651k - 18.2.1 5be31c24972a 5bf35fe2ace4 2024-04-05T21:27:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (8m) 3s ago 8m 6651k - 18.2.1 5be31c24972a dcdfa6ae17bf 2024-04-05T21:27:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (8m) 0s ago 9m 92.0M - 9.4.7 954c08fa6188 ee38366c66e3 2024-04-05T21:27:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (3m) 0s ago 3m 101M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:27:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (3m) 0s ago 3m 171M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:27:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (3m) 3s ago 3m 116M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:27:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (3m) 3s ago 3m 92.9M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:27:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (28s) 0s ago 11m 471M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:27:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (101s) 3s ago 8m 65.0M - 18.2.2-1173-g8919605b 067bdb249792 2d759c011461 2024-04-05T21:27:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (11m) 0s ago 11m 51.0M 2048M 18.2.1 5be31c24972a 3c30ce99c006 2024-04-05T21:27:37.605 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (8m) 3s ago 8m 41.3M 2048M 18.2.1 5be31c24972a 8ecc9636670e 2024-04-05T21:27:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (10m) 0s ago 10m 18.3M - 1.5.0 0da6a335fe13 5ac35283f3e9 2024-04-05T21:27:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (8m) 3s ago 8m 16.4M - 1.5.0 0da6a335fe13 bc2ad104140f 2024-04-05T21:27:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (7m) 0s ago 7m 191M 1227M 18.2.1 5be31c24972a f6f3a45429a2 2024-04-05T21:27:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (7m) 0s ago 7m 215M 1227M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:27:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (6m) 0s ago 6m 165M 1227M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:27:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (6m) 3s ago 6m 260M 2251M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:27:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (5m) 3s ago 5m 225M 2251M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:27:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (5m) 3s ago 5m 209M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:27:37.606 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (34s) 0s ago 9m 47.0M - 2.43.0 a07b618ecd1d b905b13222c1 2024-04-05T21:27:37.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.606+0000 7f008a609700 1 -- 172.21.15.121:0/1952107558 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f006c038a90 msgr2=0x7f006c03af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:37.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.606+0000 7f008a609700 1 --2- 172.21.15.121:0/1952107558 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f006c038a90 0x7f006c03af50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0078009fd0 tx=0x7f007800b5a0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.606+0000 7f008a609700 1 -- 172.21.15.121:0/1952107558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0084101640 msgr2=0x7f0084196250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:37.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.606+0000 7f008a609700 1 --2- 172.21.15.121:0/1952107558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0084101640 0x7f0084196250 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f007400e920 tx=0x7f007400ec30 comp rx=0 tx=0).stop 2024-04-05T21:27:37.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.606+0000 7f008a609700 1 -- 172.21.15.121:0/1952107558 shutdown_connections 2024-04-05T21:27:37.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.606+0000 7f008a609700 1 --2- 172.21.15.121:0/1952107558 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f006c038a90 0x7f006c03af50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.606+0000 7f008a609700 1 --2- 172.21.15.121:0/1952107558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0084101640 0x7f0084196250 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.606+0000 7f008a609700 1 --2- 172.21.15.121:0/1952107558 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0084103f70 0x7f0084196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.606+0000 7f008a609700 1 -- 172.21.15.121:0/1952107558 >> 172.21.15.121:0/1952107558 conn(0x7f00840fb030 msgr2=0x7f00840fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:37.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.606+0000 7f008a609700 1 -- 172.21.15.121:0/1952107558 shutdown_connections 2024-04-05T21:27:37.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.606+0000 7f008a609700 1 -- 172.21.15.121:0/1952107558 wait complete. 2024-04-05T21:27:37.726 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:37 smithi188 ceph-mon[49715]: pgmap v6: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 489 KiB/s rd, 590 KiB/s wr, 78 op/s 2024-04-05T21:27:37.726 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:37 smithi188 ceph-mon[49715]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:37.726 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:37.726 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:37 smithi121 ceph-mon[39723]: pgmap v6: 65 pgs: 65 active+clean; 297 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 489 KiB/s rd, 590 KiB/s wr, 78 op/s 2024-04-05T21:27:37.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:37 smithi121 ceph-mon[39723]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:37.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:37.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:37.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.765+0000 7f2f062c5700 1 -- 172.21.15.121:0/2252636202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f000759d0 msgr2=0x7f2f00077e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:37.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.765+0000 7f2f062c5700 1 --2- 172.21.15.121:0/2252636202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f000759d0 0x7f2f00077e60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400aaa0 tx=0x7f2ef400adb0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.767+0000 7f2f062c5700 1 -- 172.21.15.121:0/2252636202 shutdown_connections 2024-04-05T21:27:37.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.767+0000 7f2f062c5700 1 --2- 172.21.15.121:0/2252636202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f000759d0 0x7f2f00077e60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.767+0000 7f2f062c5700 1 --2- 172.21.15.121:0/2252636202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f00072ea0 0x7f2f000732c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.767+0000 7f2f062c5700 1 -- 172.21.15.121:0/2252636202 >> 172.21.15.121:0/2252636202 conn(0x7f2f0006d9f0 msgr2=0x7f2f0006fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:37.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.767+0000 7f2f062c5700 1 -- 172.21.15.121:0/2252636202 shutdown_connections 2024-04-05T21:27:37.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.767+0000 7f2f062c5700 1 -- 172.21.15.121:0/2252636202 wait complete. 2024-04-05T21:27:37.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.768+0000 7f2f062c5700 1 Processor -- start 2024-04-05T21:27:37.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.768+0000 7f2f062c5700 1 -- start start 2024-04-05T21:27:37.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.768+0000 7f2f062c5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f00072ea0 0x7f2f00082de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:37.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.768+0000 7f2f062c5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f000759d0 0x7f2f00083320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:37.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.768+0000 7f2f062c5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f000838f0 con 0x7f2f00072ea0 2024-04-05T21:27:37.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.768+0000 7f2f062c5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f00083a30 con 0x7f2f000759d0 2024-04-05T21:27:37.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.768+0000 7f2efffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f00072ea0 0x7f2f00082de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.768+0000 7f2efffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f00072ea0 0x7f2f00082de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34808/0 (socket says 172.21.15.121:34808) 2024-04-05T21:27:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.768+0000 7f2efffff700 1 -- 172.21.15.121:0/1908570532 learned_addr learned my addr 172.21.15.121:0/1908570532 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:27:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.769+0000 7f2eff7fe700 1 --2- 172.21.15.121:0/1908570532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f000759d0 0x7f2f00083320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.769+0000 7f2eff7fe700 1 -- 172.21.15.121:0/1908570532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f00072ea0 msgr2=0x7f2f00082de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.769+0000 7f2eff7fe700 1 --2- 172.21.15.121:0/1908570532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f00072ea0 0x7f2f00082de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.769+0000 7f2eff7fe700 1 -- 172.21.15.121:0/1908570532 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef400a700 con 0x7f2f000759d0 2024-04-05T21:27:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.769+0000 7f2eff7fe700 1 --2- 172.21.15.121:0/1908570532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f000759d0 0x7f2f00083320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4005310 tx=0x7f2ef4005da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:37.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.770+0000 7f2efd7fa700 1 -- 172.21.15.121:0/1908570532 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ef4024070 con 0x7f2f000759d0 2024-04-05T21:27:37.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.770+0000 7f2efd7fa700 1 -- 172.21.15.121:0/1908570532 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f2ef40056b0 con 0x7f2f000759d0 2024-04-05T21:27:37.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.770+0000 7f2efd7fa700 1 -- 172.21.15.121:0/1908570532 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ef4020780 con 0x7f2f000759d0 2024-04-05T21:27:37.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.771+0000 7f2f062c5700 1 -- 172.21.15.121:0/1908570532 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f001b2f60 con 0x7f2f000759d0 2024-04-05T21:27:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.771+0000 7f2f062c5700 1 -- 172.21.15.121:0/1908570532 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f001b3450 con 0x7f2f000759d0 2024-04-05T21:27:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.772+0000 7f2f062c5700 1 -- 172.21.15.121:0/1908570532 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0007cef0 con 0x7f2f000759d0 2024-04-05T21:27:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.772+0000 7f2efd7fa700 1 -- 172.21.15.121:0/1908570532 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 30) v1 ==== 45821+0+0 (secure 0 0 0) 0x7f2ef400eb60 con 0x7f2f000759d0 2024-04-05T21:27:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.773+0000 7f2efd7fa700 1 --2- 172.21.15.121:0/1908570532 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2ee8038b30 0x7f2ee803aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.773+0000 7f2efd7fa700 1 -- 172.21.15.121:0/1908570532 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f2ef4013070 con 0x7f2f000759d0 2024-04-05T21:27:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.773+0000 7f2efffff700 1 --2- 172.21.15.121:0/1908570532 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2ee8038b30 0x7f2ee803aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:37.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.774+0000 7f2efffff700 1 --2- 172.21.15.121:0/1908570532 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2ee8038b30 0x7f2ee803aff0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0005950 tx=0x7f2ef0004080 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:37.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:37.780+0000 7f2efd7fa700 1 -- 172.21.15.121:0/1908570532 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f2ef400d3e0 con 0x7f2f000759d0 2024-04-05T21:27:38.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.219+0000 7f2f062c5700 1 -- 172.21.15.121:0/1908570532 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2f0002ce80 con 0x7f2f000759d0 2024-04-05T21:27:38.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.219+0000 7f2efd7fa700 1 -- 172.21.15.121:0/1908570532 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+679 (secure 0 0 0) 0x7f2ef401ce60 con 0x7f2f000759d0 2024-04-05T21:27:38.221 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:27:38.221 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:27:38.221 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-05T21:27:38.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:27:38.222 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:27:38.222 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 1 2024-04-05T21:27:38.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:27:38.222 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:27:38.222 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-05T21:27:38.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:27:38.222 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:27:38.222 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:27:38.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:27:38.222 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:27:38.223 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-05T21:27:38.223 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 1 2024-04-05T21:27:38.223 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:27:38.223 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:27:38.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.222+0000 7f2f062c5700 1 -- 172.21.15.121:0/1908570532 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2ee8038b30 msgr2=0x7f2ee803aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:38.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.222+0000 7f2f062c5700 1 --2- 172.21.15.121:0/1908570532 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2ee8038b30 0x7f2ee803aff0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0005950 tx=0x7f2ef0004080 comp rx=0 tx=0).stop 2024-04-05T21:27:38.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.222+0000 7f2f062c5700 1 -- 172.21.15.121:0/1908570532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f000759d0 msgr2=0x7f2f00083320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:38.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.222+0000 7f2f062c5700 1 --2- 172.21.15.121:0/1908570532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f000759d0 0x7f2f00083320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4005310 tx=0x7f2ef4005da0 comp rx=0 tx=0).stop 2024-04-05T21:27:38.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.222+0000 7f2f062c5700 1 -- 172.21.15.121:0/1908570532 shutdown_connections 2024-04-05T21:27:38.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.222+0000 7f2f062c5700 1 --2- 172.21.15.121:0/1908570532 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2ee8038b30 0x7f2ee803aff0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:38.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.222+0000 7f2f062c5700 1 --2- 172.21.15.121:0/1908570532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f00072ea0 0x7f2f00082de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:38.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.222+0000 7f2f062c5700 1 --2- 172.21.15.121:0/1908570532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f000759d0 0x7f2f00083320 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:38.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.222+0000 7f2f062c5700 1 -- 172.21.15.121:0/1908570532 >> 172.21.15.121:0/1908570532 conn(0x7f2f0006d9f0 msgr2=0x7f2f0006fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:38.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.222+0000 7f2f062c5700 1 -- 172.21.15.121:0/1908570532 shutdown_connections 2024-04-05T21:27:38.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.222+0000 7f2f062c5700 1 -- 172.21.15.121:0/1908570532 wait complete. 2024-04-05T21:27:38.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.369+0000 7fd7f0096700 1 -- 172.21.15.121:0/2313800498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e8101e60 msgr2=0x7fd7e81022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:38.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.369+0000 7fd7f0096700 1 --2- 172.21.15.121:0/2313800498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e8101e60 0x7fd7e81022c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8009910 tx=0x7fd7d8009c20 comp rx=0 tx=0).stop 2024-04-05T21:27:38.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.369+0000 7fd7f0096700 1 -- 172.21.15.121:0/2313800498 shutdown_connections 2024-04-05T21:27:38.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.369+0000 7fd7f0096700 1 --2- 172.21.15.121:0/2313800498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e8101e60 0x7fd7e81022c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:38.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.369+0000 7fd7f0096700 1 --2- 172.21.15.121:0/2313800498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7e80fee80 0x7fd7e80ff2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:38.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.369+0000 7fd7f0096700 1 -- 172.21.15.121:0/2313800498 >> 172.21.15.121:0/2313800498 conn(0x7fd7e80fa7c0 msgr2=0x7fd7e80fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:38.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.369+0000 7fd7f0096700 1 -- 172.21.15.121:0/2313800498 shutdown_connections 2024-04-05T21:27:38.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.369+0000 7fd7f0096700 1 -- 172.21.15.121:0/2313800498 wait complete. 2024-04-05T21:27:38.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.370+0000 7fd7f0096700 1 Processor -- start 2024-04-05T21:27:38.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.370+0000 7fd7f0096700 1 -- start start 2024-04-05T21:27:38.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.370+0000 7fd7f0096700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7e80fee80 0x7fd7e81962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:38.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.370+0000 7fd7f0096700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e8101e60 0x7fd7e81967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:38.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.370+0000 7fd7f0096700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e8196e10 con 0x7fd7e8101e60 2024-04-05T21:27:38.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.370+0000 7fd7f0096700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e8196f50 con 0x7fd7e80fee80 2024-04-05T21:27:38.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.370+0000 7fd7ed631700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e8101e60 0x7fd7e81967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:38.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.370+0000 7fd7ed631700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e8101e60 0x7fd7e81967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34826/0 (socket says 172.21.15.121:34826) 2024-04-05T21:27:38.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.370+0000 7fd7ed631700 1 -- 172.21.15.121:0/2690123571 learned_addr learned my addr 172.21.15.121:0/2690123571 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:27:38.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.370+0000 7fd7ede32700 1 --2- 172.21.15.121:0/2690123571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7e80fee80 0x7fd7e81962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:38.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.371+0000 7fd7ed631700 1 -- 172.21.15.121:0/2690123571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7e80fee80 msgr2=0x7fd7e81962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:38.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.371+0000 7fd7ed631700 1 --2- 172.21.15.121:0/2690123571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7e80fee80 0x7fd7e81962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:38.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.371+0000 7fd7ed631700 1 -- 172.21.15.121:0/2690123571 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d80095f0 con 0x7fd7e8101e60 2024-04-05T21:27:38.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.371+0000 7fd7ed631700 1 --2- 172.21.15.121:0/2690123571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e8101e60 0x7fd7e81967f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd7d800bde0 tx=0x7fd7d800be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:38.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.371+0000 7fd7deffd700 1 -- 172.21.15.121:0/2690123571 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7d801d070 con 0x7fd7e8101e60 2024-04-05T21:27:38.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.371+0000 7fd7f0096700 1 -- 172.21.15.121:0/2690123571 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e806a8f0 con 0x7fd7e8101e60 2024-04-05T21:27:38.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.371+0000 7fd7deffd700 1 -- 172.21.15.121:0/2690123571 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fd7d8005dc0 con 0x7fd7e8101e60 2024-04-05T21:27:38.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.371+0000 7fd7deffd700 1 -- 172.21.15.121:0/2690123571 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7d8003ab0 con 0x7fd7e8101e60 2024-04-05T21:27:38.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.371+0000 7fd7f0096700 1 -- 172.21.15.121:0/2690123571 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e806ae00 con 0x7fd7e8101e60 2024-04-05T21:27:38.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.372+0000 7fd7deffd700 1 -- 172.21.15.121:0/2690123571 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 30) v1 ==== 45821+0+0 (secure 0 0 0) 0x7fd7d8003c10 con 0x7fd7e8101e60 2024-04-05T21:27:38.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.372+0000 7fd7deffd700 1 --2- 172.21.15.121:0/2690123571 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d4038a90 0x7fd7d403af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:27:38.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.372+0000 7fd7deffd700 1 -- 172.21.15.121:0/2690123571 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fd7d804e3f0 con 0x7fd7e8101e60 2024-04-05T21:27:38.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.372+0000 7fd7ede32700 1 --2- 172.21.15.121:0/2690123571 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d4038a90 0x7fd7d403af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:27:38.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.372+0000 7fd7f0096700 1 -- 172.21.15.121:0/2690123571 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e81095e0 con 0x7fd7e8101e60 2024-04-05T21:27:38.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.373+0000 7fd7ede32700 1 --2- 172.21.15.121:0/2690123571 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d4038a90 0x7fd7d403af50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8068210 tx=0x7fd7e4009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:27:38.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.382+0000 7fd7deffd700 1 -- 172.21.15.121:0/2690123571 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7fd7d8022d10 con 0x7fd7e8101e60 2024-04-05T21:27:38.647 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:38 smithi121 ceph-mon[39723]: from='client.14684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:38.647 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:38 smithi121 ceph-mon[39723]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:38.647 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:38.647 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:38.647 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:38 smithi121 ceph-mon[39723]: Standby manager daemon smithi188.tgcuir started 2024-04-05T21:27:38.647 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:38 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/1426106737' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/crt"}]: dispatch 2024-04-05T21:27:38.647 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:38 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/1426106737' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T21:27:38.647 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:38 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/1426106737' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/key"}]: dispatch 2024-04-05T21:27:38.647 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:38 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/1426106737' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T21:27:38.647 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:38 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1908570532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:27:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.668+0000 7fd7deffd700 1 -- 172.21.15.121:0/2690123571 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 31) v1 ==== 90946+0+0 (secure 0 0 0) 0x7fd7d804e080 con 0x7fd7e8101e60 2024-04-05T21:27:38.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.682+0000 7fd7f0096700 1 -- 172.21.15.121:0/2690123571 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7e81067b0 con 0x7fd7d4038a90 2024-04-05T21:27:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.685+0000 7fd7deffd700 1 -- 172.21.15.121:0/2690123571 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fd7e81067b0 con 0x7fd7d4038a90 2024-04-05T21:27:38.688 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:27:38.688 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:27:38.688 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:27:38.688 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T21:27:38.688 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:27:38.688 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr" 2024-04-05T21:27:38.688 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:27:38.689 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "2/2 daemons upgraded", 2024-04-05T21:27:38.689 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "", 2024-04-05T21:27:38.689 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:27:38.689 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:27:38.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.689+0000 7fd7f0096700 1 -- 172.21.15.121:0/2690123571 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d4038a90 msgr2=0x7fd7d403af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:38.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.689+0000 7fd7f0096700 1 --2- 172.21.15.121:0/2690123571 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d4038a90 0x7fd7d403af50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8068210 tx=0x7fd7e4009260 comp rx=0 tx=0).stop 2024-04-05T21:27:38.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.689+0000 7fd7f0096700 1 -- 172.21.15.121:0/2690123571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e8101e60 msgr2=0x7fd7e81967f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:27:38.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.689+0000 7fd7f0096700 1 --2- 172.21.15.121:0/2690123571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e8101e60 0x7fd7e81967f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd7d800bde0 tx=0x7fd7d800be10 comp rx=0 tx=0).stop 2024-04-05T21:27:38.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.689+0000 7fd7f0096700 1 -- 172.21.15.121:0/2690123571 shutdown_connections 2024-04-05T21:27:38.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.689+0000 7fd7f0096700 1 --2- 172.21.15.121:0/2690123571 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d4038a90 0x7fd7d403af50 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8068210 tx=0x7fd7e4009260 comp rx=0 tx=0).stop 2024-04-05T21:27:38.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.689+0000 7fd7f0096700 1 --2- 172.21.15.121:0/2690123571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7e80fee80 0x7fd7e81962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:38.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.689+0000 7fd7f0096700 1 --2- 172.21.15.121:0/2690123571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e8101e60 0x7fd7e81967f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:27:38.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.689+0000 7fd7f0096700 1 -- 172.21.15.121:0/2690123571 >> 172.21.15.121:0/2690123571 conn(0x7fd7e80fa7c0 msgr2=0x7fd7e8105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:27:38.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.692+0000 7fd7f0096700 1 -- 172.21.15.121:0/2690123571 shutdown_connections 2024-04-05T21:27:38.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:27:38.692+0000 7fd7f0096700 1 -- 172.21.15.121:0/2690123571 wait complete. 2024-04-05T21:27:39.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:38 smithi188 ceph-mon[49715]: from='client.14684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:38 smithi188 ceph-mon[49715]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:38 smithi188 ceph-mon[49715]: Standby manager daemon smithi188.tgcuir started 2024-04-05T21:27:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:38 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/1426106737' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/crt"}]: dispatch 2024-04-05T21:27:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:38 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/1426106737' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T21:27:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:38 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/1426106737' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/key"}]: dispatch 2024-04-05T21:27:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:38 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/1426106737' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T21:27:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:38 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1908570532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:27:39.888 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:39 smithi121 ceph-mon[39723]: pgmap v7: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 332 op/s 2024-04-05T21:27:39.889 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:39 smithi121 ceph-mon[39723]: mgrmap e31: smithi121.etbquh(active, since 8s), standbys: smithi188.tgcuir 2024-04-05T21:27:39.889 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:39 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr metadata", "who": "smithi188.tgcuir", "id": "smithi188.tgcuir"}]: dispatch 2024-04-05T21:27:39.889 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:39 smithi121 ceph-mon[39723]: from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:39.997 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:39 smithi188 ceph-mon[49715]: pgmap v7: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 332 op/s 2024-04-05T21:27:39.997 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:39 smithi188 ceph-mon[49715]: mgrmap e31: smithi121.etbquh(active, since 8s), standbys: smithi188.tgcuir 2024-04-05T21:27:39.998 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:39 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr metadata", "who": "smithi188.tgcuir", "id": "smithi188.tgcuir"}]: dispatch 2024-04-05T21:27:39.998 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:39 smithi188 ceph-mon[49715]: from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:27:41.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:41 smithi121 ceph-mon[39723]: pgmap v8: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1002 KiB/s rd, 914 KiB/s wr, 258 op/s 2024-04-05T21:27:42.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:41 smithi188 ceph-mon[49715]: pgmap v8: 65 pgs: 65 active+clean; 292 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1002 KiB/s rd, 914 KiB/s wr, 258 op/s 2024-04-05T21:27:43.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:43 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:43.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:43 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:43.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:43 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:27:43.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:43 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:27:43.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:43 smithi121 ceph-mon[39723]: Updating smithi121:/etc/ceph/ceph.conf 2024-04-05T21:27:43.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:43 smithi121 ceph-mon[39723]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-05T21:27:43.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:43 smithi121 ceph-mon[39723]: pgmap v9: 65 pgs: 65 active+clean; 283 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 385 op/s 2024-04-05T21:27:43.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:43 smithi121 ceph-mon[39723]: Updating smithi121:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:27:43.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:43 smithi121 ceph-mon[39723]: Updating smithi188:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:27:43.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:43 smithi121 ceph-mon[39723]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-04-05T21:27:43.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:43 smithi121 ceph-mon[39723]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-05T21:27:43.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:43 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:43.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:43 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:43.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:43 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:27:43.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:43 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:27:43.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:43 smithi188 ceph-mon[49715]: Updating smithi121:/etc/ceph/ceph.conf 2024-04-05T21:27:43.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:43 smithi188 ceph-mon[49715]: Updating smithi188:/etc/ceph/ceph.conf 2024-04-05T21:27:43.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:43 smithi188 ceph-mon[49715]: pgmap v9: 65 pgs: 65 active+clean; 283 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 385 op/s 2024-04-05T21:27:43.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:43 smithi188 ceph-mon[49715]: Updating smithi121:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:27:43.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:43 smithi188 ceph-mon[49715]: Updating smithi188:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.conf 2024-04-05T21:27:43.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:43 smithi188 ceph-mon[49715]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-04-05T21:27:43.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:43 smithi188 ceph-mon[49715]: Updating smithi188:/etc/ceph/ceph.client.admin.keyring 2024-04-05T21:27:44.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:44 smithi121 ceph-mon[39723]: Updating smithi121:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T21:27:44.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:44 smithi121 ceph-mon[39723]: Updating smithi188:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T21:27:44.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:44 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:44.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:44 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:44.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:44 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:44.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:44 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:44.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:44 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:44.483 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:44 smithi121 ceph-mon[39723]: Reconfiguring prometheus.smithi121 (dependencies changed)... 2024-04-05T21:27:44.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:44 smithi188 ceph-mon[49715]: Updating smithi121:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T21:27:44.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:44 smithi188 ceph-mon[49715]: Updating smithi188:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-05T21:27:44.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:44 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:44.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:44 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:44.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:44 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:44.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:44 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:44.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:44 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:44.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:44 smithi188 ceph-mon[49715]: Reconfiguring prometheus.smithi121 (dependencies changed)... 2024-04-05T21:27:45.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:45 smithi188 ceph-mon[49715]: Reconfiguring daemon prometheus.smithi121 on smithi121 2024-04-05T21:27:45.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:45 smithi188 ceph-mon[49715]: pgmap v10: 65 pgs: 65 active+clean; 283 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 352 op/s 2024-04-05T21:27:45.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:45 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:27:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:45 smithi121 ceph-mon[39723]: Reconfiguring daemon prometheus.smithi121 on smithi121 2024-04-05T21:27:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:45 smithi121 ceph-mon[39723]: pgmap v10: 65 pgs: 65 active+clean; 283 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 352 op/s 2024-04-05T21:27:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:45 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:27:47.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:47 smithi121 ceph-mon[39723]: pgmap v11: 65 pgs: 65 active+clean; 285 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 384 op/s 2024-04-05T21:27:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:47 smithi188 ceph-mon[49715]: pgmap v11: 65 pgs: 65 active+clean; 285 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 384 op/s 2024-04-05T21:27:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:49 smithi121 ceph-mon[39723]: pgmap v12: 65 pgs: 65 active+clean; 285 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 522 op/s 2024-04-05T21:27:49.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:49 smithi188 ceph-mon[49715]: pgmap v12: 65 pgs: 65 active+clean; 285 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 522 op/s 2024-04-05T21:27:51.170 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:51 smithi188 ceph-mon[49715]: pgmap v13: 65 pgs: 65 active+clean; 285 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 361 op/s 2024-04-05T21:27:51.170 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:51 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:51.170 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:51 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:51.170 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:51 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:27:51.170 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:51 smithi188 ceph-mon[49715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:27:51.171 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:51 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:27:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:51 smithi121 ceph-mon[39723]: pgmap v13: 65 pgs: 65 active+clean; 285 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 361 op/s 2024-04-05T21:27:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:51 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:51 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:27:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:51 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:27:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:51 smithi121 ceph-mon[39723]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:27:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:51 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:27:53.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:53 smithi121 ceph-mon[39723]: pgmap v14: 65 pgs: 65 active+clean; 287 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 593 op/s 2024-04-05T21:27:53.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:53 smithi188 ceph-mon[49715]: pgmap v14: 65 pgs: 65 active+clean; 287 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 593 op/s 2024-04-05T21:27:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:55 smithi121 ceph-mon[39723]: pgmap v15: 65 pgs: 65 active+clean; 287 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 434 op/s 2024-04-05T21:27:55.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:55 smithi188 ceph-mon[49715]: pgmap v15: 65 pgs: 65 active+clean; 287 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 434 op/s 2024-04-05T21:27:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:57 smithi121 ceph-mon[39723]: pgmap v16: 65 pgs: 65 active+clean; 290 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 479 op/s 2024-04-05T21:27:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:57 smithi188 ceph-mon[49715]: pgmap v16: 65 pgs: 65 active+clean; 290 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 479 op/s 2024-04-05T21:27:58.688 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:58 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi188.tgcuir", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T21:27:58.688 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:58 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T21:27:58.688 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:58 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:27:58.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:58 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi188.tgcuir", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-05T21:27:58.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:58 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-05T21:27:58.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:58 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:27:59.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:59 smithi121 ceph-mon[39723]: Upgrade: Updating mgr.smithi188.tgcuir 2024-04-05T21:27:59.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:59 smithi121 ceph-mon[39723]: Deploying daemon mgr.smithi188.tgcuir on smithi188 2024-04-05T21:27:59.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:27:59 smithi121 ceph-mon[39723]: pgmap v17: 65 pgs: 65 active+clean; 286 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 619 op/s 2024-04-05T21:27:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:59 smithi188 ceph-mon[49715]: Upgrade: Updating mgr.smithi188.tgcuir 2024-04-05T21:27:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:59 smithi188 ceph-mon[49715]: Deploying daemon mgr.smithi188.tgcuir on smithi188 2024-04-05T21:27:59.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:27:59 smithi188 ceph-mon[49715]: pgmap v17: 65 pgs: 65 active+clean; 286 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 619 op/s 2024-04-05T21:28:00.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:00 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:28:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:00 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:28:01.630 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:01 smithi188 ceph-mon[49715]: pgmap v18: 65 pgs: 65 active+clean; 286 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 448 op/s 2024-04-05T21:28:01.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:01 smithi121 ceph-mon[39723]: pgmap v18: 65 pgs: 65 active+clean; 286 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 448 op/s 2024-04-05T21:28:03.385 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-04-05T21:28:03.385 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-04-05T21:28:03.743 DEBUG:teuthology.parallel:result is None 2024-04-05T21:28:03.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:03 smithi188 ceph-mon[49715]: pgmap v19: 65 pgs: 65 active+clean; 281 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 662 op/s 2024-04-05T21:28:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:03 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:03 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:03 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:28:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:03 smithi121 ceph-mon[39723]: pgmap v19: 65 pgs: 65 active+clean; 281 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 662 op/s 2024-04-05T21:28:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:03 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:03 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:03 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:28:05.615 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:05 smithi188 ceph-mon[49715]: pgmap v20: 65 pgs: 65 active+clean; 281 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 430 op/s 2024-04-05T21:28:05.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:05 smithi121 ceph-mon[39723]: pgmap v20: 65 pgs: 65 active+clean; 281 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 430 op/s 2024-04-05T21:28:06.933 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-04-05T21:28:06.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-05T21:28:07.232 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:07 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:07.232 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:07 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:07.232 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:07 smithi121 ceph-mon[39723]: pgmap v21: 65 pgs: 65 active+clean; 283 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 468 op/s 2024-04-05T21:28:07.232 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:07 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:07.232 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:07 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:07.277 DEBUG:teuthology.parallel:result is None 2024-04-05T21:28:07.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-05T21:28:07.320 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-05T21:28:07.320 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-04-05T21:28:07.347 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:07 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:07.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:07 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:07.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:07 smithi188 ceph-mon[49715]: pgmap v21: 65 pgs: 65 active+clean; 283 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 468 op/s 2024-04-05T21:28:07.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:07 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:07.348 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:07 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:07.358 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-04-05T21:28:07.358 DEBUG:teuthology.parallel:result is None 2024-04-05T21:28:08.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.807+0000 7f4d7f126700 1 -- 172.21.15.121:0/3577477577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d78101e00 msgr2=0x7f4d78102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:08.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.807+0000 7f4d7f126700 1 --2- 172.21.15.121:0/3577477577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d78101e00 0x7f4d78102260 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4d64009910 tx=0x7f4d64009c20 comp rx=0 tx=0).stop 2024-04-05T21:28:08.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.807+0000 7f4d7f126700 1 -- 172.21.15.121:0/3577477577 shutdown_connections 2024-04-05T21:28:08.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.807+0000 7f4d7f126700 1 --2- 172.21.15.121:0/3577477577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d78101e00 0x7f4d78102260 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:08.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.807+0000 7f4d7f126700 1 --2- 172.21.15.121:0/3577477577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d78100c00 0x7f4d78101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:08.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.807+0000 7f4d7f126700 1 -- 172.21.15.121:0/3577477577 >> 172.21.15.121:0/3577477577 conn(0x7f4d780fc180 msgr2=0x7f4d780fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:08.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.808+0000 7f4d7f126700 1 -- 172.21.15.121:0/3577477577 shutdown_connections 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.808+0000 7f4d7f126700 1 -- 172.21.15.121:0/3577477577 wait complete. 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.808+0000 7f4d7f126700 1 Processor -- start 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.808+0000 7f4d7f126700 1 -- start start 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.808+0000 7f4d7f126700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d78100c00 0x7f4d78196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.808+0000 7f4d7f126700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d78101e00 0x7f4d781967d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.808+0000 7f4d7f126700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d78196df0 con 0x7f4d78101e00 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.808+0000 7f4d7f126700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d78196f30 con 0x7f4d78100c00 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.808+0000 7f4d77fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d78101e00 0x7f4d781967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.808+0000 7f4d77fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d78101e00 0x7f4d781967d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44746/0 (socket says 172.21.15.121:44746) 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.808+0000 7f4d77fff700 1 -- 172.21.15.121:0/2512017310 learned_addr learned my addr 172.21.15.121:0/2512017310 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.808+0000 7f4d7cec2700 1 --2- 172.21.15.121:0/2512017310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d78100c00 0x7f4d78196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.809+0000 7f4d77fff700 1 -- 172.21.15.121:0/2512017310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d78100c00 msgr2=0x7f4d78196290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.809+0000 7f4d77fff700 1 --2- 172.21.15.121:0/2512017310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d78100c00 0x7f4d78196290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:08.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.809+0000 7f4d77fff700 1 -- 172.21.15.121:0/2512017310 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d640095f0 con 0x7f4d78101e00 2024-04-05T21:28:08.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.809+0000 7f4d77fff700 1 --2- 172.21.15.121:0/2512017310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d78101e00 0x7f4d781967d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4d64006010 tx=0x7f4d6400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:08.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.809+0000 7f4d75ffb700 1 -- 172.21.15.121:0/2512017310 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d6401d070 con 0x7f4d78101e00 2024-04-05T21:28:08.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.809+0000 7f4d7f126700 1 -- 172.21.15.121:0/2512017310 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d7819b980 con 0x7f4d78101e00 2024-04-05T21:28:08.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.809+0000 7f4d75ffb700 1 -- 172.21.15.121:0/2512017310 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f4d6400f460 con 0x7f4d78101e00 2024-04-05T21:28:08.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.809+0000 7f4d75ffb700 1 -- 172.21.15.121:0/2512017310 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d64021640 con 0x7f4d78101e00 2024-04-05T21:28:08.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.809+0000 7f4d7f126700 1 -- 172.21.15.121:0/2512017310 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d7819bef0 con 0x7f4d78101e00 2024-04-05T21:28:08.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.810+0000 7f4d7f126700 1 -- 172.21.15.121:0/2512017310 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d78109620 con 0x7f4d78101e00 2024-04-05T21:28:08.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.811+0000 7f4d75ffb700 1 -- 172.21.15.121:0/2512017310 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 31) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f4d6400f9b0 con 0x7f4d78101e00 2024-04-05T21:28:08.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.811+0000 7f4d75ffb700 1 --2- 172.21.15.121:0/2512017310 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4d6806ce20 0x7f4d6806f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:08.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.811+0000 7f4d75ffb700 1 -- 172.21.15.121:0/2512017310 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f4d6408d9b0 con 0x7f4d78101e00 2024-04-05T21:28:08.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.811+0000 7f4d7cec2700 1 --2- 172.21.15.121:0/2512017310 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4d6806ce20 0x7f4d6806f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:08.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.812+0000 7f4d7cec2700 1 --2- 172.21.15.121:0/2512017310 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4d6806ce20 0x7f4d6806f2e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4d78101c60 tx=0x7f4d6c00b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:08.816+0000 7f4d75ffb700 1 -- 172.21.15.121:0/2512017310 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f4d6405b340 con 0x7f4d78101e00 2024-04-05T21:28:09.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.033+0000 7f4d7f126700 1 -- 172.21.15.121:0/2512017310 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d78106750 con 0x7f4d6806ce20 2024-04-05T21:28:09.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.035+0000 7f4d75ffb700 1 -- 172.21.15.121:0/2512017310 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f4d78106750 con 0x7f4d6806ce20 2024-04-05T21:28:09.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.037+0000 7f4d7f126700 1 -- 172.21.15.121:0/2512017310 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4d6806ce20 msgr2=0x7f4d6806f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:09.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.037+0000 7f4d7f126700 1 --2- 172.21.15.121:0/2512017310 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4d6806ce20 0x7f4d6806f2e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4d78101c60 tx=0x7f4d6c00b410 comp rx=0 tx=0).stop 2024-04-05T21:28:09.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.037+0000 7f4d7f126700 1 -- 172.21.15.121:0/2512017310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d78101e00 msgr2=0x7f4d781967d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:09.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.037+0000 7f4d7f126700 1 --2- 172.21.15.121:0/2512017310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d78101e00 0x7f4d781967d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4d64006010 tx=0x7f4d6400bd70 comp rx=0 tx=0).stop 2024-04-05T21:28:09.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.037+0000 7f4d7f126700 1 -- 172.21.15.121:0/2512017310 shutdown_connections 2024-04-05T21:28:09.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.037+0000 7f4d7f126700 1 --2- 172.21.15.121:0/2512017310 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4d6806ce20 0x7f4d6806f2e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.037+0000 7f4d7f126700 1 --2- 172.21.15.121:0/2512017310 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d78100c00 0x7f4d78196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.037+0000 7f4d7f126700 1 --2- 172.21.15.121:0/2512017310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d78101e00 0x7f4d781967d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.037+0000 7f4d7f126700 1 -- 172.21.15.121:0/2512017310 >> 172.21.15.121:0/2512017310 conn(0x7f4d780fc180 msgr2=0x7f4d78105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:09.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.037+0000 7f4d7f126700 1 -- 172.21.15.121:0/2512017310 shutdown_connections 2024-04-05T21:28:09.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.037+0000 7f4d7f126700 1 -- 172.21.15.121:0/2512017310 wait complete. 2024-04-05T21:28:09.049 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:28:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.155+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4084530195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9e8103f70 msgr2=0x7fa9e8106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.155+0000 7fa9ed6ce700 1 --2- 172.21.15.121:0/4084530195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9e8103f70 0x7fa9e8106360 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009910 tx=0x7fa9dc009c20 comp rx=0 tx=0).stop 2024-04-05T21:28:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.155+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4084530195 shutdown_connections 2024-04-05T21:28:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.155+0000 7fa9ed6ce700 1 --2- 172.21.15.121:0/4084530195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9e8103f70 0x7fa9e8106360 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.155+0000 7fa9ed6ce700 1 --2- 172.21.15.121:0/4084530195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9e8101640 0x7fa9e8103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.155+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4084530195 >> 172.21.15.121:0/4084530195 conn(0x7fa9e80fb030 msgr2=0x7fa9e80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.155+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4084530195 shutdown_connections 2024-04-05T21:28:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.155+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4084530195 wait complete. 2024-04-05T21:28:09.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.155+0000 7fa9ed6ce700 1 Processor -- start 2024-04-05T21:28:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.156+0000 7fa9ed6ce700 1 -- start start 2024-04-05T21:28:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.156+0000 7fa9ed6ce700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9e8101640 0x7fa9e8100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.156+0000 7fa9ed6ce700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9e8103f70 0x7fa9e80ff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.156+0000 7fa9ed6ce700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80ffb30 con 0x7fa9e8101640 2024-04-05T21:28:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.156+0000 7fa9ed6ce700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e80ffc70 con 0x7fa9e8103f70 2024-04-05T21:28:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.156+0000 7fa9e6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9e8101640 0x7fa9e8100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:09.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.156+0000 7fa9e6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9e8101640 0x7fa9e8100fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44760/0 (socket says 172.21.15.121:44760) 2024-04-05T21:28:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.156+0000 7fa9e6ffd700 1 -- 172.21.15.121:0/4216529005 learned_addr learned my addr 172.21.15.121:0/4216529005 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:28:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.156+0000 7fa9e67fc700 1 --2- 172.21.15.121:0/4216529005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9e8103f70 0x7fa9e80ff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.156+0000 7fa9e67fc700 1 -- 172.21.15.121:0/4216529005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9e8101640 msgr2=0x7fa9e8100fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.156+0000 7fa9e67fc700 1 --2- 172.21.15.121:0/4216529005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9e8101640 0x7fa9e8100fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.156+0000 7fa9e67fc700 1 -- 172.21.15.121:0/4216529005 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc0095f0 con 0x7fa9e8103f70 2024-04-05T21:28:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.157+0000 7fa9e67fc700 1 --2- 172.21.15.121:0/4216529005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9e8103f70 0x7fa9e80ff5f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009910 tx=0x7fa9dc00be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.157+0000 7fa9cffff700 1 -- 172.21.15.121:0/4216529005 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9dc01d070 con 0x7fa9e8103f70 2024-04-05T21:28:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.157+0000 7fa9cffff700 1 -- 172.21.15.121:0/4216529005 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fa9dc00f460 con 0x7fa9e8103f70 2024-04-05T21:28:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.157+0000 7fa9cffff700 1 -- 172.21.15.121:0/4216529005 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9dc021640 con 0x7fa9e8103f70 2024-04-05T21:28:09.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.157+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4216529005 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e80ffef0 con 0x7fa9e8103f70 2024-04-05T21:28:09.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.157+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4216529005 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e81003e0 con 0x7fa9e8103f70 2024-04-05T21:28:09.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.158+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4216529005 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e81903e0 con 0x7fa9e8103f70 2024-04-05T21:28:09.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.159+0000 7fa9cffff700 1 -- 172.21.15.121:0/4216529005 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 31) v1 ==== 90946+0+0 (secure 0 0 0) 0x7fa9dc00f9b0 con 0x7fa9e8103f70 2024-04-05T21:28:09.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.159+0000 7fa9cffff700 1 --2- 172.21.15.121:0/4216529005 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa9d006ce20 0x7fa9d006f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:09.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.159+0000 7fa9cffff700 1 -- 172.21.15.121:0/4216529005 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fa9dc08d920 con 0x7fa9e8103f70 2024-04-05T21:28:09.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.159+0000 7fa9e6ffd700 1 --2- 172.21.15.121:0/4216529005 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa9d006ce20 0x7fa9d006f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:09.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.159+0000 7fa9e6ffd700 1 --2- 172.21.15.121:0/4216529005 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa9d006ce20 0x7fa9d006f2e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8005950 tx=0x7fa9d8004080 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:09.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.164+0000 7fa9cffff700 1 -- 172.21.15.121:0/4216529005 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7fa9dc05b2f0 con 0x7fa9e8103f70 2024-04-05T21:28:09.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:09 smithi188 ceph-mon[49715]: pgmap v22: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 563 op/s 2024-04-05T21:28:09.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.388+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4216529005 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9e8061ba0 con 0x7fa9d006ce20 2024-04-05T21:28:09.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.389+0000 7fa9cffff700 1 -- 172.21.15.121:0/4216529005 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fa9e8061ba0 con 0x7fa9d006ce20 2024-04-05T21:28:09.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.391+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4216529005 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa9d006ce20 msgr2=0x7fa9d006f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:09.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.391+0000 7fa9ed6ce700 1 --2- 172.21.15.121:0/4216529005 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa9d006ce20 0x7fa9d006f2e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8005950 tx=0x7fa9d8004080 comp rx=0 tx=0).stop 2024-04-05T21:28:09.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.391+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4216529005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9e8103f70 msgr2=0x7fa9e80ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:09.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.391+0000 7fa9ed6ce700 1 --2- 172.21.15.121:0/4216529005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9e8103f70 0x7fa9e80ff5f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009910 tx=0x7fa9dc00be20 comp rx=0 tx=0).stop 2024-04-05T21:28:09.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.391+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4216529005 shutdown_connections 2024-04-05T21:28:09.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.391+0000 7fa9ed6ce700 1 --2- 172.21.15.121:0/4216529005 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa9d006ce20 0x7fa9d006f2e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.391+0000 7fa9ed6ce700 1 --2- 172.21.15.121:0/4216529005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9e8101640 0x7fa9e8100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.391+0000 7fa9ed6ce700 1 --2- 172.21.15.121:0/4216529005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa9e8103f70 0x7fa9e80ff5f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.391+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4216529005 >> 172.21.15.121:0/4216529005 conn(0x7fa9e80fb030 msgr2=0x7fa9e80fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:09.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.391+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4216529005 shutdown_connections 2024-04-05T21:28:09.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.391+0000 7fa9ed6ce700 1 -- 172.21.15.121:0/4216529005 wait complete. 2024-04-05T21:28:09.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:09 smithi121 ceph-mon[39723]: pgmap v22: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 563 op/s 2024-04-05T21:28:09.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.494+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/3138051674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b140ffae0 msgr2=0x7f6b140fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:09.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.494+0000 7f6b1ba7b700 1 --2- 172.21.15.121:0/3138051674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b140ffae0 0x7f6b140fff00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6b10009910 tx=0x7f6b10009c20 comp rx=0 tx=0).stop 2024-04-05T21:28:09.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.494+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/3138051674 shutdown_connections 2024-04-05T21:28:09.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.494+0000 7f6b1ba7b700 1 --2- 172.21.15.121:0/3138051674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b14100ce0 0x7f6b14101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.494+0000 7f6b1ba7b700 1 --2- 172.21.15.121:0/3138051674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b140ffae0 0x7f6b140fff00 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.494+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/3138051674 >> 172.21.15.121:0/3138051674 conn(0x7f6b140fb080 msgr2=0x7f6b140fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:09.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.495+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/3138051674 shutdown_connections 2024-04-05T21:28:09.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.495+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/3138051674 wait complete. 2024-04-05T21:28:09.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.495+0000 7f6b1ba7b700 1 Processor -- start 2024-04-05T21:28:09.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.495+0000 7f6b1ba7b700 1 -- start start 2024-04-05T21:28:09.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.495+0000 7f6b1ba7b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b140ffae0 0x7f6b14069950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:09.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.495+0000 7f6b1ba7b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b14100ce0 0x7f6b1419afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:09.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.495+0000 7f6b1ba7b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1419b5b0 con 0x7f6b140ffae0 2024-04-05T21:28:09.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.495+0000 7f6b1ba7b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1419b720 con 0x7f6b14100ce0 2024-04-05T21:28:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.495+0000 7f6b19817700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b140ffae0 0x7f6b14069950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.496+0000 7f6b19817700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b140ffae0 0x7f6b14069950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44780/0 (socket says 172.21.15.121:44780) 2024-04-05T21:28:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.496+0000 7f6b19817700 1 -- 172.21.15.121:0/758434186 learned_addr learned my addr 172.21.15.121:0/758434186 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:28:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.496+0000 7f6b19016700 1 --2- 172.21.15.121:0/758434186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b14100ce0 0x7f6b1419afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.496+0000 7f6b19817700 1 -- 172.21.15.121:0/758434186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b14100ce0 msgr2=0x7f6b1419afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.496+0000 7f6b19817700 1 --2- 172.21.15.121:0/758434186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b14100ce0 0x7f6b1419afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.496+0000 7f6b19817700 1 -- 172.21.15.121:0/758434186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b100095f0 con 0x7f6b140ffae0 2024-04-05T21:28:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.496+0000 7f6b19817700 1 --2- 172.21.15.121:0/758434186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b140ffae0 0x7f6b14069950 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000bda0 tx=0x7f6b1000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.496+0000 7f6b0affd700 1 -- 172.21.15.121:0/758434186 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b1001d070 con 0x7f6b140ffae0 2024-04-05T21:28:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.496+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/758434186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b1419b9a0 con 0x7f6b140ffae0 2024-04-05T21:28:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.496+0000 7f6b0affd700 1 -- 172.21.15.121:0/758434186 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f6b1000f460 con 0x7f6b140ffae0 2024-04-05T21:28:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.496+0000 7f6b0affd700 1 -- 172.21.15.121:0/758434186 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b10022650 con 0x7f6b140ffae0 2024-04-05T21:28:09.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.496+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/758434186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1419be90 con 0x7f6b140ffae0 2024-04-05T21:28:09.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.497+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/758434186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b14108500 con 0x7f6b140ffae0 2024-04-05T21:28:09.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.497+0000 7f6b0affd700 1 -- 172.21.15.121:0/758434186 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 31) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f6b1000fa20 con 0x7f6b140ffae0 2024-04-05T21:28:09.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.498+0000 7f6b0affd700 1 --2- 172.21.15.121:0/758434186 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6b0006ce20 0x7f6b0006f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:09.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.498+0000 7f6b0affd700 1 -- 172.21.15.121:0/758434186 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f6b1008daf0 con 0x7f6b140ffae0 2024-04-05T21:28:09.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.498+0000 7f6b19016700 1 --2- 172.21.15.121:0/758434186 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6b0006ce20 0x7f6b0006f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:09.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.498+0000 7f6b19016700 1 --2- 172.21.15.121:0/758434186 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6b0006ce20 0x7f6b0006f2e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b14069eb0 tx=0x7f6b040073d0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:09.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.502+0000 7f6b0affd700 1 -- 172.21.15.121:0/758434186 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f6b1005b480 con 0x7f6b140ffae0 2024-04-05T21:28:09.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.723+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/758434186 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b14105630 con 0x7f6b0006ce20 2024-04-05T21:28:09.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.733+0000 7f6b0affd700 1 -- 172.21.15.121:0/758434186 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f6b14105630 con 0x7f6b0006ce20 2024-04-05T21:28:09.735 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:28:09.735 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (8m) 32s ago 10m 26.2M - 0.25.0 c8568f914cd2 6594a12b2bc8 2024-04-05T21:28:09.735 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (10m) 32s ago 10m 7826k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:28:09.735 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (9m) 3s ago 9m 7801k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:28:09.735 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (10m) 32s ago 10m 6651k - 18.2.1 5be31c24972a 5bf35fe2ace4 2024-04-05T21:28:09.735 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (9m) 3s ago 9m 6651k - 18.2.1 5be31c24972a dcdfa6ae17bf 2024-04-05T21:28:09.735 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (8m) 32s ago 10m 92.0M - 9.4.7 954c08fa6188 ee38366c66e3 2024-04-05T21:28:09.735 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (4m) 32s ago 4m 101M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:28:09.735 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (4m) 32s ago 4m 171M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:28:09.735 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (4m) 3s ago 4m 115M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:28:09.735 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (4m) 3s ago 4m 65.5M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:28:09.735 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (60s) 32s ago 12m 471M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:28:09.735 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (7s) 3s ago 9m 48.9M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:28:09.736 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (12m) 32s ago 12m 51.0M 2048M 18.2.1 5be31c24972a 3c30ce99c006 2024-04-05T21:28:09.736 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (9m) 3s ago 9m 43.8M 2048M 18.2.1 5be31c24972a 8ecc9636670e 2024-04-05T21:28:09.736 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (10m) 32s ago 10m 18.3M - 1.5.0 0da6a335fe13 5ac35283f3e9 2024-04-05T21:28:09.736 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (9m) 3s ago 9m 16.4M - 1.5.0 0da6a335fe13 bc2ad104140f 2024-04-05T21:28:09.736 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (8m) 32s ago 8m 191M 1227M 18.2.1 5be31c24972a f6f3a45429a2 2024-04-05T21:28:09.736 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (7m) 32s ago 7m 215M 1227M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:28:09.736 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (7m) 32s ago 7m 165M 1227M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:28:09.736 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (6m) 3s ago 6m 285M 2251M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:28:09.736 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (6m) 3s ago 6m 243M 2251M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:28:09.736 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (5m) 3s ago 5m 220M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:28:09.736 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 starting - - - - 2024-04-05T21:28:09.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.735+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/758434186 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6b0006ce20 msgr2=0x7f6b0006f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:09.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.735+0000 7f6b1ba7b700 1 --2- 172.21.15.121:0/758434186 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6b0006ce20 0x7f6b0006f2e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6b14069eb0 tx=0x7f6b040073d0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.735+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/758434186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b140ffae0 msgr2=0x7f6b14069950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:09.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.735+0000 7f6b1ba7b700 1 --2- 172.21.15.121:0/758434186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b140ffae0 0x7f6b14069950 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000bda0 tx=0x7f6b1000bdd0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.735+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/758434186 shutdown_connections 2024-04-05T21:28:09.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.735+0000 7f6b1ba7b700 1 --2- 172.21.15.121:0/758434186 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6b0006ce20 0x7f6b0006f2e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.735+0000 7f6b1ba7b700 1 --2- 172.21.15.121:0/758434186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b140ffae0 0x7f6b14069950 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.735+0000 7f6b1ba7b700 1 --2- 172.21.15.121:0/758434186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6b14100ce0 0x7f6b1419afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.735+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/758434186 >> 172.21.15.121:0/758434186 conn(0x7f6b140fb080 msgr2=0x7f6b14103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:09.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.736+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/758434186 shutdown_connections 2024-04-05T21:28:09.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.736+0000 7f6b1ba7b700 1 -- 172.21.15.121:0/758434186 wait complete. 2024-04-05T21:28:09.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.840+0000 7f1ac54e7700 1 -- 172.21.15.121:0/3407379903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac0103f70 msgr2=0x7f1ac0106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:09.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.840+0000 7f1ac54e7700 1 --2- 172.21.15.121:0/3407379903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac0103f70 0x7f1ac0106360 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009910 tx=0x7f1ab4009c20 comp rx=0 tx=0).stop 2024-04-05T21:28:09.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.840+0000 7f1ac54e7700 1 -- 172.21.15.121:0/3407379903 shutdown_connections 2024-04-05T21:28:09.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.840+0000 7f1ac54e7700 1 --2- 172.21.15.121:0/3407379903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac0103f70 0x7f1ac0106360 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.840+0000 7f1ac54e7700 1 --2- 172.21.15.121:0/3407379903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ac0101640 0x7f1ac0103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.840+0000 7f1ac54e7700 1 -- 172.21.15.121:0/3407379903 >> 172.21.15.121:0/3407379903 conn(0x7f1ac00fb050 msgr2=0x7f1ac00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:09.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.841+0000 7f1ac54e7700 1 -- 172.21.15.121:0/3407379903 shutdown_connections 2024-04-05T21:28:09.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.841+0000 7f1ac54e7700 1 -- 172.21.15.121:0/3407379903 wait complete. 2024-04-05T21:28:09.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.841+0000 7f1ac54e7700 1 Processor -- start 2024-04-05T21:28:09.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.841+0000 7f1ac54e7700 1 -- start start 2024-04-05T21:28:09.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.841+0000 7f1ac54e7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ac0101640 0x7f1ac0196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:09.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.841+0000 7f1ac54e7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac0103f70 0x7f1ac01967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:09.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.841+0000 7f1ac54e7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0196dd0 con 0x7f1ac0103f70 2024-04-05T21:28:09.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.841+0000 7f1ac54e7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0196f10 con 0x7f1ac0101640 2024-04-05T21:28:09.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.842+0000 7f1abe7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac0103f70 0x7f1ac01967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:09.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.842+0000 7f1abe7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac0103f70 0x7f1ac01967b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44788/0 (socket says 172.21.15.121:44788) 2024-04-05T21:28:09.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.842+0000 7f1abeffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ac0101640 0x7f1ac0196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:09.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.842+0000 7f1abe7fc700 1 -- 172.21.15.121:0/2438548264 learned_addr learned my addr 172.21.15.121:0/2438548264 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:28:09.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.842+0000 7f1abe7fc700 1 -- 172.21.15.121:0/2438548264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ac0101640 msgr2=0x7f1ac0196270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:09.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.842+0000 7f1abe7fc700 1 --2- 172.21.15.121:0/2438548264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ac0101640 0x7f1ac0196270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:09.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.842+0000 7f1abe7fc700 1 -- 172.21.15.121:0/2438548264 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab40095f0 con 0x7f1ac0103f70 2024-04-05T21:28:09.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.842+0000 7f1abe7fc700 1 --2- 172.21.15.121:0/2438548264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac0103f70 0x7f1ac01967b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1ab400bde0 tx=0x7f1ab400be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:09.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.842+0000 7f1aa7fff700 1 -- 172.21.15.121:0/2438548264 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ab401d070 con 0x7f1ac0103f70 2024-04-05T21:28:09.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.842+0000 7f1ac54e7700 1 -- 172.21.15.121:0/2438548264 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac019b960 con 0x7f1ac0103f70 2024-04-05T21:28:09.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.842+0000 7f1aa7fff700 1 -- 172.21.15.121:0/2438548264 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f1ab4005dc0 con 0x7f1ac0103f70 2024-04-05T21:28:09.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.842+0000 7f1aa7fff700 1 -- 172.21.15.121:0/2438548264 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ab4003b10 con 0x7f1ac0103f70 2024-04-05T21:28:09.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.842+0000 7f1ac54e7700 1 -- 172.21.15.121:0/2438548264 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac019be70 con 0x7f1ac0103f70 2024-04-05T21:28:09.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.844+0000 7f1ac54e7700 1 -- 172.21.15.121:0/2438548264 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac01903e0 con 0x7f1ac0103f70 2024-04-05T21:28:09.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.845+0000 7f1aa7fff700 1 -- 172.21.15.121:0/2438548264 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 31) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f1ab40048e0 con 0x7f1ac0103f70 2024-04-05T21:28:09.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.845+0000 7f1aa7fff700 1 --2- 172.21.15.121:0/2438548264 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1aa806ce20 0x7f1aa806f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:09.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.845+0000 7f1aa7fff700 1 -- 172.21.15.121:0/2438548264 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f1ab408dee0 con 0x7f1ac0103f70 2024-04-05T21:28:09.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.845+0000 7f1abeffd700 1 --2- 172.21.15.121:0/2438548264 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1aa806ce20 0x7f1aa806f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:09.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.846+0000 7f1abeffd700 1 --2- 172.21.15.121:0/2438548264 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1aa806ce20 0x7f1aa806f2e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009c60 tx=0x7f1ab0009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:09.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:09.854+0000 7f1aa7fff700 1 -- 172.21.15.121:0/2438548264 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f1ab405b870 con 0x7f1ac0103f70 2024-04-05T21:28:10.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.135+0000 7f1ac54e7700 1 -- 172.21.15.121:0/2438548264 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1ac002ce50 con 0x7f1ac0103f70 2024-04-05T21:28:10.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.135+0000 7f1aa7fff700 1 -- 172.21.15.121:0/2438548264 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+679 (secure 0 0 0) 0x7f1ab405b400 con 0x7f1ac0103f70 2024-04-05T21:28:10.136 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:28:10.136 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:28:10.136 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-05T21:28:10.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:28:10.136 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:28:10.136 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:28:10.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:28:10.137 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:28:10.137 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-05T21:28:10.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:28:10.137 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:28:10.137 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:28:10.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:28:10.137 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:28:10.137 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-05T21:28:10.137 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:28:10.137 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:28:10.137 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:28:10.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.137+0000 7f1ac54e7700 1 -- 172.21.15.121:0/2438548264 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1aa806ce20 msgr2=0x7f1aa806f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:10.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.137+0000 7f1ac54e7700 1 --2- 172.21.15.121:0/2438548264 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1aa806ce20 0x7f1aa806f2e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009c60 tx=0x7f1ab0009260 comp rx=0 tx=0).stop 2024-04-05T21:28:10.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.137+0000 7f1ac54e7700 1 -- 172.21.15.121:0/2438548264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac0103f70 msgr2=0x7f1ac01967b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:10.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.137+0000 7f1ac54e7700 1 --2- 172.21.15.121:0/2438548264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac0103f70 0x7f1ac01967b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1ab400bde0 tx=0x7f1ab400be10 comp rx=0 tx=0).stop 2024-04-05T21:28:10.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.137+0000 7f1ac54e7700 1 -- 172.21.15.121:0/2438548264 shutdown_connections 2024-04-05T21:28:10.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.137+0000 7f1ac54e7700 1 --2- 172.21.15.121:0/2438548264 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1aa806ce20 0x7f1aa806f2e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:10.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.137+0000 7f1ac54e7700 1 --2- 172.21.15.121:0/2438548264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1ac0101640 0x7f1ac0196270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:10.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.137+0000 7f1ac54e7700 1 --2- 172.21.15.121:0/2438548264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac0103f70 0x7f1ac01967b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:10.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.137+0000 7f1ac54e7700 1 -- 172.21.15.121:0/2438548264 >> 172.21.15.121:0/2438548264 conn(0x7f1ac00fb050 msgr2=0x7f1ac00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:10.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.137+0000 7f1ac54e7700 1 -- 172.21.15.121:0/2438548264 shutdown_connections 2024-04-05T21:28:10.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.137+0000 7f1ac54e7700 1 -- 172.21.15.121:0/2438548264 wait complete. 2024-04-05T21:28:10.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.244+0000 7f97c27d3700 1 -- 172.21.15.121:0/1321690194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97bc100cb0 msgr2=0x7f97bc101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:10.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.244+0000 7f97c27d3700 1 --2- 172.21.15.121:0/1321690194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97bc100cb0 0x7f97bc101110 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f97b0009910 tx=0x7f97b0009c20 comp rx=0 tx=0).stop 2024-04-05T21:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.244+0000 7f97c27d3700 1 -- 172.21.15.121:0/1321690194 shutdown_connections 2024-04-05T21:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.244+0000 7f97c27d3700 1 --2- 172.21.15.121:0/1321690194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97bc100cb0 0x7f97bc101110 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.244+0000 7f97c27d3700 1 --2- 172.21.15.121:0/1321690194 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97bc0ffab0 0x7f97bc0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.244+0000 7f97c27d3700 1 -- 172.21.15.121:0/1321690194 >> 172.21.15.121:0/1321690194 conn(0x7f97bc0fb030 msgr2=0x7f97bc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.244+0000 7f97c27d3700 1 -- 172.21.15.121:0/1321690194 shutdown_connections 2024-04-05T21:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.244+0000 7f97c27d3700 1 -- 172.21.15.121:0/1321690194 wait complete. 2024-04-05T21:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.245+0000 7f97c27d3700 1 Processor -- start 2024-04-05T21:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.245+0000 7f97c27d3700 1 -- start start 2024-04-05T21:28:10.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.245+0000 7f97c27d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97bc0ffab0 0x7f97bc068f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:10.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.245+0000 7f97c27d3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97bc100cb0 0x7f97bc19afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:10.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.245+0000 7f97c27d3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc19b580 con 0x7f97bc0ffab0 2024-04-05T21:28:10.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.245+0000 7f97c27d3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc19b6c0 con 0x7f97bc100cb0 2024-04-05T21:28:10.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.245+0000 7f97bbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97bc0ffab0 0x7f97bc068f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:10.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.245+0000 7f97bbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97bc0ffab0 0x7f97bc068f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44810/0 (socket says 172.21.15.121:44810) 2024-04-05T21:28:10.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.245+0000 7f97bbfff700 1 -- 172.21.15.121:0/1171036022 learned_addr learned my addr 172.21.15.121:0/1171036022 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:28:10.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.245+0000 7f97bbfff700 1 -- 172.21.15.121:0/1171036022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97bc100cb0 msgr2=0x7f97bc19afb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-05T21:28:10.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.245+0000 7f97bbfff700 1 --2- 172.21.15.121:0/1171036022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97bc100cb0 0x7f97bc19afb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:10.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.245+0000 7f97bbfff700 1 -- 172.21.15.121:0/1171036022 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b00095f0 con 0x7f97bc0ffab0 2024-04-05T21:28:10.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.246+0000 7f97bbfff700 1 --2- 172.21.15.121:0/1171036022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97bc0ffab0 0x7f97bc068f70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00e8f0 tx=0x7f97ac00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:10.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.246+0000 7f97b97fa700 1 -- 172.21.15.121:0/1171036022 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97ac009790 con 0x7f97bc0ffab0 2024-04-05T21:28:10.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.246+0000 7f97c27d3700 1 -- 172.21.15.121:0/1171036022 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97bc19b9a0 con 0x7f97bc0ffab0 2024-04-05T21:28:10.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.246+0000 7f97b97fa700 1 -- 172.21.15.121:0/1171036022 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f97ac0044b0 con 0x7f97bc0ffab0 2024-04-05T21:28:10.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.246+0000 7f97b97fa700 1 -- 172.21.15.121:0/1171036022 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97ac003d20 con 0x7f97bc0ffab0 2024-04-05T21:28:10.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.246+0000 7f97c27d3700 1 -- 172.21.15.121:0/1171036022 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97bc19bf70 con 0x7f97bc0ffab0 2024-04-05T21:28:10.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.247+0000 7f97c27d3700 1 -- 172.21.15.121:0/1171036022 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97bc1084d0 con 0x7f97bc0ffab0 2024-04-05T21:28:10.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.247+0000 7f97b97fa700 1 -- 172.21.15.121:0/1171036022 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 31) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f97ac004020 con 0x7f97bc0ffab0 2024-04-05T21:28:10.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.247+0000 7f97b97fa700 1 --2- 172.21.15.121:0/1171036022 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f97a406ce10 0x7f97a406f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:10.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.248+0000 7f97b97fa700 1 -- 172.21.15.121:0/1171036022 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f97ac014070 con 0x7f97bc0ffab0 2024-04-05T21:28:10.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.248+0000 7f97bb7fe700 1 --2- 172.21.15.121:0/1171036022 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f97a406ce10 0x7f97a406f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:10.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.248+0000 7f97bb7fe700 1 --2- 172.21.15.121:0/1171036022 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f97a406ce10 0x7f97a406f2d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f97b0005f50 tx=0x7f97b0005dc0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:10.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.252+0000 7f97b97fa700 1 -- 172.21.15.121:0/1171036022 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f97ac05a440 con 0x7f97bc0ffab0 2024-04-05T21:28:10.417 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:10 smithi188 ceph-mon[49715]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:10.417 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:10 smithi188 ceph-mon[49715]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:10.417 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:10 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2438548264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:10.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:10 smithi121 ceph-mon[39723]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:10.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:10 smithi121 ceph-mon[39723]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:10.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:10 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2438548264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:10.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.493+0000 7f97c27d3700 1 -- 172.21.15.121:0/1171036022 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f97bc105600 con 0x7f97a406ce10 2024-04-05T21:28:10.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.494+0000 7f97b97fa700 1 -- 172.21.15.121:0/1171036022 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f97bc105600 con 0x7f97a406ce10 2024-04-05T21:28:10.495 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:28:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:28:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:28:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T21:28:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:28:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr" 2024-04-05T21:28:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:28:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "2/2 daemons upgraded", 2024-04-05T21:28:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mgr daemons", 2024-04-05T21:28:10.495 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:28:10.495 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:28:10.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.496+0000 7f97c27d3700 1 -- 172.21.15.121:0/1171036022 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f97a406ce10 msgr2=0x7f97a406f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:10.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.496+0000 7f97c27d3700 1 --2- 172.21.15.121:0/1171036022 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f97a406ce10 0x7f97a406f2d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f97b0005f50 tx=0x7f97b0005dc0 comp rx=0 tx=0).stop 2024-04-05T21:28:10.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.496+0000 7f97c27d3700 1 -- 172.21.15.121:0/1171036022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97bc0ffab0 msgr2=0x7f97bc068f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:10.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.496+0000 7f97c27d3700 1 --2- 172.21.15.121:0/1171036022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97bc0ffab0 0x7f97bc068f70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00e8f0 tx=0x7f97ac00ecb0 comp rx=0 tx=0).stop 2024-04-05T21:28:10.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.496+0000 7f97c27d3700 1 -- 172.21.15.121:0/1171036022 shutdown_connections 2024-04-05T21:28:10.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.496+0000 7f97c27d3700 1 --2- 172.21.15.121:0/1171036022 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f97a406ce10 0x7f97a406f2d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:10.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.496+0000 7f97c27d3700 1 --2- 172.21.15.121:0/1171036022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97bc0ffab0 0x7f97bc068f70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:10.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.496+0000 7f97c27d3700 1 --2- 172.21.15.121:0/1171036022 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97bc100cb0 0x7f97bc19afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:10.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.496+0000 7f97c27d3700 1 -- 172.21.15.121:0/1171036022 >> 172.21.15.121:0/1171036022 conn(0x7f97bc0fb030 msgr2=0x7f97bc103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:10.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.496+0000 7f97c27d3700 1 -- 172.21.15.121:0/1171036022 shutdown_connections 2024-04-05T21:28:10.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:10.496+0000 7f97c27d3700 1 -- 172.21.15.121:0/1171036022 wait complete. 2024-04-05T21:28:11.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:11 smithi121 ceph-mon[39723]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:11.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:11 smithi121 ceph-mon[39723]: pgmap v23: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 391 op/s 2024-04-05T21:28:11.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:11 smithi121 ceph-mon[39723]: Standby manager daemon smithi188.tgcuir restarted 2024-04-05T21:28:11.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:11 smithi121 ceph-mon[39723]: Standby manager daemon smithi188.tgcuir started 2024-04-05T21:28:11.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:11 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/1616832610' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/crt"}]: dispatch 2024-04-05T21:28:11.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:11 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/1616832610' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T21:28:11.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:11 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/1616832610' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/key"}]: dispatch 2024-04-05T21:28:11.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:11 smithi121 ceph-mon[39723]: from='mgr.? 172.21.15.188:0/1616832610' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T21:28:11.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:11 smithi121 ceph-mon[39723]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:11.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:11 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:11.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:11 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:11.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:11 smithi188 ceph-mon[49715]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:11.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:11 smithi188 ceph-mon[49715]: pgmap v23: 65 pgs: 65 active+clean; 275 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 391 op/s 2024-04-05T21:28:11.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:11 smithi188 ceph-mon[49715]: Standby manager daemon smithi188.tgcuir restarted 2024-04-05T21:28:11.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:11 smithi188 ceph-mon[49715]: Standby manager daemon smithi188.tgcuir started 2024-04-05T21:28:11.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:11 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/1616832610' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/crt"}]: dispatch 2024-04-05T21:28:11.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:11 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/1616832610' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-05T21:28:11.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:11 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/1616832610' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi188.tgcuir/key"}]: dispatch 2024-04-05T21:28:11.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:11 smithi188 ceph-mon[49715]: from='mgr.? 172.21.15.188:0/1616832610' entity='mgr.smithi188.tgcuir' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-05T21:28:11.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:11 smithi188 ceph-mon[49715]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:11.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:11 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:11.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:11 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:12.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:12 smithi121 ceph-mon[39723]: mgrmap e32: smithi121.etbquh(active, since 41s), standbys: smithi188.tgcuir 2024-04-05T21:28:12.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:12 smithi188 ceph-mon[49715]: mgrmap e32: smithi121.etbquh(active, since 41s), standbys: smithi188.tgcuir 2024-04-05T21:28:13.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:13 smithi121 ceph-mon[39723]: pgmap v24: 65 pgs: 65 active+clean; 269 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 460 op/s 2024-04-05T21:28:13.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:13 smithi188 ceph-mon[49715]: pgmap v24: 65 pgs: 65 active+clean; 269 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 460 op/s 2024-04-05T21:28:15.468 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:15 smithi188 ceph-mon[49715]: pgmap v25: 65 pgs: 65 active+clean; 269 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 761 KiB/s rd, 705 KiB/s wr, 246 op/s 2024-04-05T21:28:15.468 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:15 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:28:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:15 smithi121 ceph-mon[39723]: pgmap v25: 65 pgs: 65 active+clean; 269 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 761 KiB/s rd, 705 KiB/s wr, 246 op/s 2024-04-05T21:28:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:15 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:28:17.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: pgmap v26: 65 pgs: 65 active+clean; 269 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 761 KiB/s rd, 705 KiB/s wr, 247 op/s 2024-04-05T21:28:17.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi121.etbquh"}]: dispatch 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi121.etbquh"}]': finished 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi188.tgcuir"}]: dispatch 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi188.tgcuir"}]': finished 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:17 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: pgmap v26: 65 pgs: 65 active+clean; 269 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 761 KiB/s rd, 705 KiB/s wr, 247 op/s 2024-04-05T21:28:17.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:28:17.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:28:17.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:28:17.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi121.etbquh"}]: dispatch 2024-04-05T21:28:17.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi121.etbquh"}]': finished 2024-04-05T21:28:17.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi188.tgcuir"}]: dispatch 2024-04-05T21:28:17.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi188.tgcuir"}]': finished 2024-04-05T21:28:17.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:17.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:17.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:17 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:18.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:18 smithi121 ceph-mon[39723]: Upgrade: Setting container_image for all mgr 2024-04-05T21:28:18.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:18 smithi121 ceph-mon[39723]: Upgrade: Setting container_image for all rgw 2024-04-05T21:28:18.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:18 smithi121 ceph-mon[39723]: Upgrade: Setting container_image for all rbd-mirror 2024-04-05T21:28:18.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:18 smithi121 ceph-mon[39723]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-05T21:28:18.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:18 smithi121 ceph-mon[39723]: Upgrade: Setting container_image for all iscsi 2024-04-05T21:28:18.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:18 smithi121 ceph-mon[39723]: Upgrade: Setting container_image for all nfs 2024-04-05T21:28:18.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:18 smithi121 ceph-mon[39723]: Upgrade: Setting container_image for all nvmeof 2024-04-05T21:28:18.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:18 smithi188 ceph-mon[49715]: Upgrade: Setting container_image for all mgr 2024-04-05T21:28:18.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:18 smithi188 ceph-mon[49715]: Upgrade: Setting container_image for all rgw 2024-04-05T21:28:18.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:18 smithi188 ceph-mon[49715]: Upgrade: Setting container_image for all rbd-mirror 2024-04-05T21:28:18.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:18 smithi188 ceph-mon[49715]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-05T21:28:18.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:18 smithi188 ceph-mon[49715]: Upgrade: Setting container_image for all iscsi 2024-04-05T21:28:18.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:18 smithi188 ceph-mon[49715]: Upgrade: Setting container_image for all nfs 2024-04-05T21:28:18.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:18 smithi188 ceph-mon[49715]: Upgrade: Setting container_image for all nvmeof 2024-04-05T21:28:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:19 smithi121 ceph-mon[39723]: pgmap v27: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 470 KiB/s rd, 415 KiB/s wr, 211 op/s 2024-04-05T21:28:19.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:19 smithi188 ceph-mon[49715]: pgmap v27: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 470 KiB/s rd, 415 KiB/s wr, 211 op/s 2024-04-05T21:28:20.638 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:20 smithi121 ceph-mon[39723]: Upgrade: Updating node-exporter.smithi121 (1/2) 2024-04-05T21:28:20.638 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:20 smithi121 ceph-mon[39723]: Deploying daemon node-exporter.smithi121 on smithi121 2024-04-05T21:28:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:20 smithi188 ceph-mon[49715]: Upgrade: Updating node-exporter.smithi121 (1/2) 2024-04-05T21:28:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:20 smithi188 ceph-mon[49715]: Deploying daemon node-exporter.smithi121 on smithi121 2024-04-05T21:28:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:21 smithi121 ceph-mon[39723]: pgmap v28: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 180 KiB/s rd, 123 KiB/s wr, 72 op/s 2024-04-05T21:28:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:21 smithi188 ceph-mon[49715]: pgmap v28: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 180 KiB/s rd, 123 KiB/s wr, 72 op/s 2024-04-05T21:28:22.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:22 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:22.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:22 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:22.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:22 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:22.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:22 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:23.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:23 smithi188 ceph-mon[49715]: pgmap v29: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 181 KiB/s rd, 123 KiB/s wr, 75 op/s 2024-04-05T21:28:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:23 smithi121 ceph-mon[39723]: pgmap v29: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 181 KiB/s rd, 123 KiB/s wr, 75 op/s 2024-04-05T21:28:24.774 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:24 smithi188 ceph-mon[49715]: Upgrade: Updating node-exporter.smithi188 (2/2) 2024-04-05T21:28:24.775 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:24 smithi188 ceph-mon[49715]: Deploying daemon node-exporter.smithi188 on smithi188 2024-04-05T21:28:24.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:24 smithi121 ceph-mon[39723]: Upgrade: Updating node-exporter.smithi188 (2/2) 2024-04-05T21:28:24.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:24 smithi121 ceph-mon[39723]: Deploying daemon node-exporter.smithi188 on smithi188 2024-04-05T21:28:25.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:25 smithi121 ceph-mon[39723]: pgmap v30: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2024-04-05T21:28:26.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:25 smithi188 ceph-mon[49715]: pgmap v30: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2024-04-05T21:28:26.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:26 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:26.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:26 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:26.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:26 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:28:26.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:26 smithi121 ceph-mon[39723]: pgmap v31: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2024-04-05T21:28:26.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:26 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:26.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:26 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:27.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:26 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:27.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:26 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:27.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:26 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:28:27.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:26 smithi188 ceph-mon[49715]: pgmap v31: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2024-04-05T21:28:27.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:26 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:27.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:26 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:29.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:29 smithi121 ceph-mon[39723]: pgmap v32: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2024-04-05T21:28:29.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:29 smithi188 ceph-mon[49715]: pgmap v32: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2024-04-05T21:28:30.474 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:30.474 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:30.474 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:30.474 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:30.474 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:28:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:28:31.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:31 smithi121 ceph-mon[39723]: pgmap v33: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-05T21:28:31.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:31 smithi188 ceph-mon[49715]: pgmap v33: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-05T21:28:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:32 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:32 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:32 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:32 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:32.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:32 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:32.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:32 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:32.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:32 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:32.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:32 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:33.570 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:33 smithi121 ceph-mon[39723]: pgmap v34: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-05T21:28:33.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:33 smithi188 ceph-mon[49715]: pgmap v34: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-05T21:28:35.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:35 smithi121 ceph-mon[39723]: pgmap v35: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T21:28:35.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:35 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:35.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:35 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:35.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:35 smithi188 ceph-mon[49715]: pgmap v35: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T21:28:35.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:35 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:35.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:35 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:37.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: pgmap v36: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-04-05T21:28:37.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:37.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:37.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:28:37.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:28:37.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:37.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:28:37.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:37.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:37.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:37.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:37.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:37.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:37.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:37.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:37 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:38.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: pgmap v36: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-04-05T21:28:38.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:38.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:38.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:28:38.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:28:38.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:38.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:28:38.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:38.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:38.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:38.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:38.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:38.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:38.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:38.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:37 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:39.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:39 smithi188 ceph-mon[49715]: pgmap v37: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-05T21:28:39.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:39 smithi121 ceph-mon[39723]: pgmap v37: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-05T21:28:40.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.615+0000 7f1a5b516700 1 -- 172.21.15.121:0/923011903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a54103f70 msgr2=0x7f1a54106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:40.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.615+0000 7f1a5b516700 1 --2- 172.21.15.121:0/923011903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a54103f70 0x7f1a54106360 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009910 tx=0x7f1a50009c20 comp rx=0 tx=0).stop 2024-04-05T21:28:40.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.615+0000 7f1a5b516700 1 -- 172.21.15.121:0/923011903 shutdown_connections 2024-04-05T21:28:40.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.615+0000 7f1a5b516700 1 --2- 172.21.15.121:0/923011903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a54103f70 0x7f1a54106360 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:40.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.615+0000 7f1a5b516700 1 --2- 172.21.15.121:0/923011903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a54101640 0x7f1a54103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:40.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.615+0000 7f1a5b516700 1 -- 172.21.15.121:0/923011903 >> 172.21.15.121:0/923011903 conn(0x7f1a540fb050 msgr2=0x7f1a540fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.615+0000 7f1a5b516700 1 -- 172.21.15.121:0/923011903 shutdown_connections 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.615+0000 7f1a5b516700 1 -- 172.21.15.121:0/923011903 wait complete. 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.616+0000 7f1a5b516700 1 Processor -- start 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.616+0000 7f1a5b516700 1 -- start start 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.616+0000 7f1a5b516700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a54101640 0x7f1a54196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.616+0000 7f1a5b516700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a54103f70 0x7f1a541967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.616+0000 7f1a5b516700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a54196dd0 con 0x7f1a54101640 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.616+0000 7f1a5b516700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a54196f10 con 0x7f1a54103f70 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.616+0000 7f1a592b2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a54101640 0x7f1a54196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.616+0000 7f1a592b2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a54101640 0x7f1a54196270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35490/0 (socket says 172.21.15.121:35490) 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.616+0000 7f1a592b2700 1 -- 172.21.15.121:0/3913861592 learned_addr learned my addr 172.21.15.121:0/3913861592 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.616+0000 7f1a58ab1700 1 --2- 172.21.15.121:0/3913861592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a54103f70 0x7f1a541967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.616+0000 7f1a592b2700 1 -- 172.21.15.121:0/3913861592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a54103f70 msgr2=0x7f1a541967b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:40.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.617+0000 7f1a592b2700 1 --2- 172.21.15.121:0/3913861592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a54103f70 0x7f1a541967b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:40.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.617+0000 7f1a592b2700 1 -- 172.21.15.121:0/3913861592 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a500095f0 con 0x7f1a54101640 2024-04-05T21:28:40.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.617+0000 7f1a592b2700 1 --2- 172.21.15.121:0/3913861592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a54101640 0x7f1a54196270 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c8a0 tx=0x7f1a4400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:40.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.617+0000 7f1a4a7fc700 1 -- 172.21.15.121:0/3913861592 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a4400cea0 con 0x7f1a54101640 2024-04-05T21:28:40.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.617+0000 7f1a5b516700 1 -- 172.21.15.121:0/3913861592 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5419b9c0 con 0x7f1a54101640 2024-04-05T21:28:40.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.617+0000 7f1a4a7fc700 1 -- 172.21.15.121:0/3913861592 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f1a44004830 con 0x7f1a54101640 2024-04-05T21:28:40.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.617+0000 7f1a4a7fc700 1 -- 172.21.15.121:0/3913861592 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a44005540 con 0x7f1a54101640 2024-04-05T21:28:40.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.617+0000 7f1a5b516700 1 -- 172.21.15.121:0/3913861592 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5419bf90 con 0x7f1a54101640 2024-04-05T21:28:40.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.618+0000 7f1a5b516700 1 -- 172.21.15.121:0/3913861592 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a541903e0 con 0x7f1a54101640 2024-04-05T21:28:40.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.618+0000 7f1a4a7fc700 1 -- 172.21.15.121:0/3913861592 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 32) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f1a440049a0 con 0x7f1a54101640 2024-04-05T21:28:40.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.619+0000 7f1a4a7fc700 1 --2- 172.21.15.121:0/3913861592 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a4006ce20 0x7f1a4006f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:40.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.619+0000 7f1a4a7fc700 1 -- 172.21.15.121:0/3913861592 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f1a4408c440 con 0x7f1a54101640 2024-04-05T21:28:40.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.619+0000 7f1a58ab1700 1 --2- 172.21.15.121:0/3913861592 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a4006ce20 0x7f1a4006f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:40.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.619+0000 7f1a58ab1700 1 --2- 172.21.15.121:0/3913861592 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a4006ce20 0x7f1a4006f2e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1a500059d0 tx=0x7f1a50005920 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:40.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.623+0000 7f1a4a7fc700 1 -- 172.21.15.121:0/3913861592 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f1a44059dd0 con 0x7f1a54101640 2024-04-05T21:28:40.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.837+0000 7f1a5b516700 1 -- 172.21.15.121:0/3913861592 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a540611d0 con 0x7f1a4006ce20 2024-04-05T21:28:40.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.838+0000 7f1a4a7fc700 1 -- 172.21.15.121:0/3913861592 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f1a540611d0 con 0x7f1a4006ce20 2024-04-05T21:28:40.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.840+0000 7f1a5b516700 1 -- 172.21.15.121:0/3913861592 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a4006ce20 msgr2=0x7f1a4006f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:40.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.840+0000 7f1a5b516700 1 --2- 172.21.15.121:0/3913861592 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a4006ce20 0x7f1a4006f2e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1a500059d0 tx=0x7f1a50005920 comp rx=0 tx=0).stop 2024-04-05T21:28:40.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.840+0000 7f1a5b516700 1 -- 172.21.15.121:0/3913861592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a54101640 msgr2=0x7f1a54196270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:40.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.840+0000 7f1a5b516700 1 --2- 172.21.15.121:0/3913861592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a54101640 0x7f1a54196270 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c8a0 tx=0x7f1a4400cc60 comp rx=0 tx=0).stop 2024-04-05T21:28:40.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.840+0000 7f1a5b516700 1 -- 172.21.15.121:0/3913861592 shutdown_connections 2024-04-05T21:28:40.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.840+0000 7f1a5b516700 1 --2- 172.21.15.121:0/3913861592 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a4006ce20 0x7f1a4006f2e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:40.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.840+0000 7f1a5b516700 1 --2- 172.21.15.121:0/3913861592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a54101640 0x7f1a54196270 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:40.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.840+0000 7f1a5b516700 1 --2- 172.21.15.121:0/3913861592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a54103f70 0x7f1a541967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:40.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.840+0000 7f1a5b516700 1 -- 172.21.15.121:0/3913861592 >> 172.21.15.121:0/3913861592 conn(0x7f1a540fb050 msgr2=0x7f1a540fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:40.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.840+0000 7f1a5b516700 1 -- 172.21.15.121:0/3913861592 shutdown_connections 2024-04-05T21:28:40.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.840+0000 7f1a5b516700 1 -- 172.21.15.121:0/3913861592 wait complete. 2024-04-05T21:28:40.852 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:28:40.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.945+0000 7f0ad4595700 1 -- 172.21.15.121:0/3363227372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0acc103f70 msgr2=0x7f0acc106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:40.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.945+0000 7f0ad4595700 1 --2- 172.21.15.121:0/3363227372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0acc103f70 0x7f0acc106360 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0abc009910 tx=0x7f0abc009c20 comp rx=0 tx=0).stop 2024-04-05T21:28:40.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.945+0000 7f0ad4595700 1 -- 172.21.15.121:0/3363227372 shutdown_connections 2024-04-05T21:28:40.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.945+0000 7f0ad4595700 1 --2- 172.21.15.121:0/3363227372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0acc103f70 0x7f0acc106360 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.945+0000 7f0ad4595700 1 --2- 172.21.15.121:0/3363227372 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0acc101640 0x7f0acc103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.945+0000 7f0ad4595700 1 -- 172.21.15.121:0/3363227372 >> 172.21.15.121:0/3363227372 conn(0x7f0acc0fb030 msgr2=0x7f0acc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.945+0000 7f0ad4595700 1 -- 172.21.15.121:0/3363227372 shutdown_connections 2024-04-05T21:28:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.945+0000 7f0ad4595700 1 -- 172.21.15.121:0/3363227372 wait complete. 2024-04-05T21:28:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad4595700 1 Processor -- start 2024-04-05T21:28:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad4595700 1 -- start start 2024-04-05T21:28:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad4595700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0acc101640 0x7f0acc196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad4595700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0acc103f70 0x7f0acc1968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad4595700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc196ec0 con 0x7f0acc103f70 2024-04-05T21:28:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad4595700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc197000 con 0x7f0acc101640 2024-04-05T21:28:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad1b30700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0acc103f70 0x7f0acc1968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad1b30700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0acc103f70 0x7f0acc1968a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35498/0 (socket says 172.21.15.121:35498) 2024-04-05T21:28:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad1b30700 1 -- 172.21.15.121:0/1601974739 learned_addr learned my addr 172.21.15.121:0/1601974739 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad2331700 1 --2- 172.21.15.121:0/1601974739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0acc101640 0x7f0acc196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad1b30700 1 -- 172.21.15.121:0/1601974739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0acc101640 msgr2=0x7f0acc196360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad1b30700 1 --2- 172.21.15.121:0/1601974739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0acc101640 0x7f0acc196360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad1b30700 1 -- 172.21.15.121:0/1601974739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc0095f0 con 0x7f0acc103f70 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad1b30700 1 --2- 172.21.15.121:0/1601974739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0acc103f70 0x7f0acc1968a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0abc000c00 tx=0x7f0abc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ac37fe700 1 -- 172.21.15.121:0/1601974739 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0abc01d070 con 0x7f0acc103f70 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ac37fe700 1 -- 172.21.15.121:0/1601974739 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f0abc022470 con 0x7f0acc103f70 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.946+0000 7f0ad4595700 1 -- 172.21.15.121:0/1601974739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0acc19ba50 con 0x7f0acc103f70 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.947+0000 7f0ac37fe700 1 -- 172.21.15.121:0/1601974739 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0abc00f690 con 0x7f0acc103f70 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.947+0000 7f0ad4595700 1 -- 172.21.15.121:0/1601974739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0acc19bfc0 con 0x7f0acc103f70 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.947+0000 7f0ad4595700 1 -- 172.21.15.121:0/1601974739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0acc190480 con 0x7f0acc103f70 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.948+0000 7f0ac37fe700 1 -- 172.21.15.121:0/1601974739 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 32) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f0abc0225e0 con 0x7f0acc103f70 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.948+0000 7f0ac37fe700 1 --2- 172.21.15.121:0/1601974739 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ab806ce10 0x7f0ab806f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.948+0000 7f0ac37fe700 1 -- 172.21.15.121:0/1601974739 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f0abc08e080 con 0x7f0acc103f70 2024-04-05T21:28:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.949+0000 7f0ad2331700 1 --2- 172.21.15.121:0/1601974739 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ab806ce10 0x7f0ab806f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:40.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.949+0000 7f0ad2331700 1 --2- 172.21.15.121:0/1601974739 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ab806ce10 0x7f0ab806f2d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009910 tx=0x7f0ac8008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:40.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:40.953+0000 7f0ac37fe700 1 -- 172.21.15.121:0/1601974739 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f0abc05bbe0 con 0x7f0acc103f70 2024-04-05T21:28:41.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.162+0000 7f0ad4595700 1 -- 172.21.15.121:0/1601974739 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0acc0611d0 con 0x7f0ab806ce10 2024-04-05T21:28:41.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.164+0000 7f0ac37fe700 1 -- 172.21.15.121:0/1601974739 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f0acc0611d0 con 0x7f0ab806ce10 2024-04-05T21:28:41.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.166+0000 7f0ad4595700 1 -- 172.21.15.121:0/1601974739 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ab806ce10 msgr2=0x7f0ab806f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:41.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.166+0000 7f0ad4595700 1 --2- 172.21.15.121:0/1601974739 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ab806ce10 0x7f0ab806f2d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009910 tx=0x7f0ac8008040 comp rx=0 tx=0).stop 2024-04-05T21:28:41.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.166+0000 7f0ad4595700 1 -- 172.21.15.121:0/1601974739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0acc103f70 msgr2=0x7f0acc1968a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:41.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.166+0000 7f0ad4595700 1 --2- 172.21.15.121:0/1601974739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0acc103f70 0x7f0acc1968a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0abc000c00 tx=0x7f0abc00be20 comp rx=0 tx=0).stop 2024-04-05T21:28:41.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.166+0000 7f0ad4595700 1 -- 172.21.15.121:0/1601974739 shutdown_connections 2024-04-05T21:28:41.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.166+0000 7f0ad4595700 1 --2- 172.21.15.121:0/1601974739 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ab806ce10 0x7f0ab806f2d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.166+0000 7f0ad4595700 1 --2- 172.21.15.121:0/1601974739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0acc101640 0x7f0acc196360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.166+0000 7f0ad4595700 1 --2- 172.21.15.121:0/1601974739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0acc103f70 0x7f0acc1968a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.166+0000 7f0ad4595700 1 -- 172.21.15.121:0/1601974739 >> 172.21.15.121:0/1601974739 conn(0x7f0acc0fb030 msgr2=0x7f0acc0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:41.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.166+0000 7f0ad4595700 1 -- 172.21.15.121:0/1601974739 shutdown_connections 2024-04-05T21:28:41.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.166+0000 7f0ad4595700 1 -- 172.21.15.121:0/1601974739 wait complete. 2024-04-05T21:28:41.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.269+0000 7f055be60700 1 -- 172.21.15.121:0/3974387575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0554100bb0 msgr2=0x7f0554100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:41.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.269+0000 7f055be60700 1 --2- 172.21.15.121:0/3974387575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0554100bb0 0x7f0554100fd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0548009910 tx=0x7f0548009c20 comp rx=0 tx=0).stop 2024-04-05T21:28:41.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.269+0000 7f055be60700 1 -- 172.21.15.121:0/3974387575 shutdown_connections 2024-04-05T21:28:41.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.269+0000 7f055be60700 1 --2- 172.21.15.121:0/3974387575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0554101db0 0x7f0554102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.269+0000 7f055be60700 1 --2- 172.21.15.121:0/3974387575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0554100bb0 0x7f0554100fd0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.269+0000 7f055be60700 1 -- 172.21.15.121:0/3974387575 >> 172.21.15.121:0/3974387575 conn(0x7f05540fc110 msgr2=0x7f05540fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:41.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.269+0000 7f055be60700 1 -- 172.21.15.121:0/3974387575 shutdown_connections 2024-04-05T21:28:41.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.269+0000 7f055be60700 1 -- 172.21.15.121:0/3974387575 wait complete. 2024-04-05T21:28:41.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.269+0000 7f055be60700 1 Processor -- start 2024-04-05T21:28:41.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.269+0000 7f055be60700 1 -- start start 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f055be60700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0554100bb0 0x7f0554196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f055be60700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0554101db0 0x7f05541967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f055be60700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0554196dd0 con 0x7f0554101db0 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f055be60700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0554196f10 con 0x7f0554100bb0 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f05593fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0554101db0 0x7f05541967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f05593fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0554101db0 0x7f05541967b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35526/0 (socket says 172.21.15.121:35526) 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f05593fb700 1 -- 172.21.15.121:0/1866154320 learned_addr learned my addr 172.21.15.121:0/1866154320 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f0559bfc700 1 --2- 172.21.15.121:0/1866154320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0554100bb0 0x7f0554196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f05593fb700 1 -- 172.21.15.121:0/1866154320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0554100bb0 msgr2=0x7f0554196270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f05593fb700 1 --2- 172.21.15.121:0/1866154320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0554100bb0 0x7f0554196270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f05593fb700 1 -- 172.21.15.121:0/1866154320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05480095f0 con 0x7f0554101db0 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f0559bfc700 1 --2- 172.21.15.121:0/1866154320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0554100bb0 0x7f0554196270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f05593fb700 1 --2- 172.21.15.121:0/1866154320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0554101db0 0x7f05541967b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f055000d8d0 tx=0x7f055000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f0546ffd700 1 -- 172.21.15.121:0/1866154320 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0550009940 con 0x7f0554101db0 2024-04-05T21:28:41.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f0546ffd700 1 -- 172.21.15.121:0/1866154320 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f0550010460 con 0x7f0554101db0 2024-04-05T21:28:41.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f055be60700 1 -- 172.21.15.121:0/1866154320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055419b9c0 con 0x7f0554101db0 2024-04-05T21:28:41.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f0546ffd700 1 -- 172.21.15.121:0/1866154320 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f055000f5f0 con 0x7f0554101db0 2024-04-05T21:28:41.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.270+0000 7f055be60700 1 -- 172.21.15.121:0/1866154320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055419bf90 con 0x7f0554101db0 2024-04-05T21:28:41.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.271+0000 7f055be60700 1 -- 172.21.15.121:0/1866154320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0554109720 con 0x7f0554101db0 2024-04-05T21:28:41.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.272+0000 7f0546ffd700 1 -- 172.21.15.121:0/1866154320 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 32) v1 ==== 90946+0+0 (secure 0 0 0) 0x7f05500105d0 con 0x7f0554101db0 2024-04-05T21:28:41.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.272+0000 7f0546ffd700 1 --2- 172.21.15.121:0/1866154320 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f054006ce20 0x7f054006f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:41.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.272+0000 7f0546ffd700 1 -- 172.21.15.121:0/1866154320 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f055008c6c0 con 0x7f0554101db0 2024-04-05T21:28:41.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.272+0000 7f0559bfc700 1 --2- 172.21.15.121:0/1866154320 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f054006ce20 0x7f054006f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:41.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.273+0000 7f0559bfc700 1 --2- 172.21.15.121:0/1866154320 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f054006ce20 0x7f054006f2e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0548006010 tx=0x7f0548005a90 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:41.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.277+0000 7f0546ffd700 1 -- 172.21.15.121:0/1866154320 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f055005a0d0 con 0x7f0554101db0 2024-04-05T21:28:41.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.478+0000 7f055be60700 1 -- 172.21.15.121:0/1866154320 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0554106700 con 0x7f054006ce20 2024-04-05T21:28:41.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.488+0000 7f0546ffd700 1 -- 172.21.15.121:0/1866154320 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f0554106700 con 0x7f054006ce20 2024-04-05T21:28:41.489 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:28:41.490 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (9m) 10s ago 10m 26.2M - 0.25.0 c8568f914cd2 6594a12b2bc8 2024-04-05T21:28:41.490 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (11m) 10s ago 11m 7944k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:28:41.490 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (10m) 12s ago 10m 7834k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:28:41.490 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (11m) 10s ago 11m 6651k - 18.2.1 5be31c24972a 5bf35fe2ace4 2024-04-05T21:28:41.490 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (10m) 12s ago 10m 6651k - 18.2.1 5be31c24972a dcdfa6ae17bf 2024-04-05T21:28:41.490 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (9m) 10s ago 10m 89.9M - 9.4.7 954c08fa6188 ee38366c66e3 2024-04-05T21:28:41.490 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (4m) 10s ago 4m 64.2M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:28:41.490 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (4m) 10s ago 4m 134M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:28:41.490 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (4m) 12s ago 4m 110M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:28:41.490 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (4m) 12s ago 4m 62.4M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:28:41.490 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (92s) 10s ago 12m 493M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:28:41.490 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (39s) 12s ago 9m 448M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:28:41.490 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (12m) 10s ago 12m 55.3M 2048M 18.2.1 5be31c24972a 3c30ce99c006 2024-04-05T21:28:41.491 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (9m) 12s ago 9m 45.2M 2048M 18.2.1 5be31c24972a 8ecc9636670e 2024-04-05T21:28:41.491 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (20s) 10s ago 11m 3347k - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:28:41.491 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (16s) 12s ago 9m 3653k - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:28:41.491 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (8m) 10s ago 8m 213M 1227M 18.2.1 5be31c24972a f6f3a45429a2 2024-04-05T21:28:41.491 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (8m) 10s ago 8m 241M 1227M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:28:41.491 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (7m) 10s ago 7m 180M 1227M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:28:41.491 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (7m) 12s ago 7m 286M 2251M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:28:41.491 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (7m) 12s ago 6m 244M 2251M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:28:41.491 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (6m) 12s ago 6m 223M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:28:41.491 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (52s) 10s ago 10m 47.4M - 2.43.0 a07b618ecd1d b18fa8a85cef 2024-04-05T21:28:41.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.490+0000 7f055be60700 1 -- 172.21.15.121:0/1866154320 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f054006ce20 msgr2=0x7f054006f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:41.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.490+0000 7f055be60700 1 --2- 172.21.15.121:0/1866154320 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f054006ce20 0x7f054006f2e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0548006010 tx=0x7f0548005a90 comp rx=0 tx=0).stop 2024-04-05T21:28:41.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.490+0000 7f055be60700 1 -- 172.21.15.121:0/1866154320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0554101db0 msgr2=0x7f05541967b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:41.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.490+0000 7f055be60700 1 --2- 172.21.15.121:0/1866154320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0554101db0 0x7f05541967b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f055000d8d0 tx=0x7f055000dc90 comp rx=0 tx=0).stop 2024-04-05T21:28:41.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.490+0000 7f055be60700 1 -- 172.21.15.121:0/1866154320 shutdown_connections 2024-04-05T21:28:41.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.490+0000 7f055be60700 1 --2- 172.21.15.121:0/1866154320 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f054006ce20 0x7f054006f2e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.490+0000 7f055be60700 1 --2- 172.21.15.121:0/1866154320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0554100bb0 0x7f0554196270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.490+0000 7f055be60700 1 --2- 172.21.15.121:0/1866154320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0554101db0 0x7f05541967b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.491+0000 7f055be60700 1 -- 172.21.15.121:0/1866154320 >> 172.21.15.121:0/1866154320 conn(0x7f05540fc110 msgr2=0x7f0554104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:41.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.491+0000 7f055be60700 1 -- 172.21.15.121:0/1866154320 shutdown_connections 2024-04-05T21:28:41.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.491+0000 7f055be60700 1 -- 172.21.15.121:0/1866154320 wait complete. 2024-04-05T21:28:41.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.592+0000 7fe029593700 1 -- 172.21.15.121:0/1048388309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe024101f70 msgr2=0x7fe0241023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:41.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.592+0000 7fe029593700 1 --2- 172.21.15.121:0/1048388309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe024101f70 0x7fe0241023d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe018009910 tx=0x7fe018009c20 comp rx=0 tx=0).stop 2024-04-05T21:28:41.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.592+0000 7fe029593700 1 -- 172.21.15.121:0/1048388309 shutdown_connections 2024-04-05T21:28:41.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.592+0000 7fe029593700 1 --2- 172.21.15.121:0/1048388309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe024101f70 0x7fe0241023d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.592+0000 7fe029593700 1 --2- 172.21.15.121:0/1048388309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe0240feef0 0x7fe0240ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.592+0000 7fe029593700 1 -- 172.21.15.121:0/1048388309 >> 172.21.15.121:0/1048388309 conn(0x7fe0240fa810 msgr2=0x7fe0240fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.592+0000 7fe029593700 1 -- 172.21.15.121:0/1048388309 shutdown_connections 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.592+0000 7fe029593700 1 -- 172.21.15.121:0/1048388309 wait complete. 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.593+0000 7fe029593700 1 Processor -- start 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.593+0000 7fe029593700 1 -- start start 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.593+0000 7fe029593700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0240feef0 0x7fe024196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.593+0000 7fe029593700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe024101f70 0x7fe024196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.593+0000 7fe029593700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe024196e90 con 0x7fe0240feef0 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.593+0000 7fe029593700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe024196fd0 con 0x7fe024101f70 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.593+0000 7fe022ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0240feef0 0x7fe024196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.593+0000 7fe022ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0240feef0 0x7fe024196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35550/0 (socket says 172.21.15.121:35550) 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.593+0000 7fe022ffd700 1 -- 172.21.15.121:0/444535893 learned_addr learned my addr 172.21.15.121:0/444535893 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.593+0000 7fe0227fc700 1 --2- 172.21.15.121:0/444535893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe024101f70 0x7fe024196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.594+0000 7fe022ffd700 1 -- 172.21.15.121:0/444535893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe024101f70 msgr2=0x7fe024196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.594+0000 7fe022ffd700 1 --2- 172.21.15.121:0/444535893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe024101f70 0x7fe024196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.594+0000 7fe022ffd700 1 -- 172.21.15.121:0/444535893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0180095f0 con 0x7fe0240feef0 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.594+0000 7fe0227fc700 1 --2- 172.21.15.121:0/444535893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe024101f70 0x7fe024196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.594+0000 7fe022ffd700 1 --2- 172.21.15.121:0/444535893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0240feef0 0x7fe024196330 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe01400d900 tx=0x7fe01400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.594+0000 7fe00bfff700 1 -- 172.21.15.121:0/444535893 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0140041f0 con 0x7fe0240feef0 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.594+0000 7fe00bfff700 1 -- 172.21.15.121:0/444535893 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fe014004350 con 0x7fe0240feef0 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.594+0000 7fe00bfff700 1 -- 172.21.15.121:0/444535893 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe014003cf0 con 0x7fe0240feef0 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.594+0000 7fe029593700 1 -- 172.21.15.121:0/444535893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02419ba80 con 0x7fe0240feef0 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.594+0000 7fe029593700 1 -- 172.21.15.121:0/444535893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02419c050 con 0x7fe0240feef0 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.595+0000 7fe029593700 1 -- 172.21.15.121:0/444535893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe024109630 con 0x7fe0240feef0 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.595+0000 7fe00bfff700 1 -- 172.21.15.121:0/444535893 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 32) v1 ==== 90946+0+0 (secure 0 0 0) 0x7fe014009730 con 0x7fe0240feef0 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.596+0000 7fe00bfff700 1 --2- 172.21.15.121:0/444535893 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe00c06ce20 0x7fe00c06f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.596+0000 7fe00bfff700 1 -- 172.21.15.121:0/444535893 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fe01408c990 con 0x7fe0240feef0 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.596+0000 7fe0227fc700 1 --2- 172.21.15.121:0/444535893 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe00c06ce20 0x7fe00c06f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.596+0000 7fe0227fc700 1 --2- 172.21.15.121:0/444535893 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe00c06ce20 0x7fe00c06f2e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe01800b5c0 tx=0x7fe018005a90 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:41.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.600+0000 7fe00bfff700 1 -- 172.21.15.121:0/444535893 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7fe01405a280 con 0x7fe0240feef0 2024-04-05T21:28:41.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.880+0000 7fe029593700 1 -- 172.21.15.121:0/444535893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe02419c720 con 0x7fe0240feef0 2024-04-05T21:28:41.883 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:28:41.883 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:28:41.883 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-05T21:28:41.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:28:41.884 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:28:41.884 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:28:41.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:28:41.884 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:28:41.884 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-05T21:28:41.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:28:41.884 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:28:41.884 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:28:41.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:28:41.884 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:28:41.884 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-05T21:28:41.884 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:28:41.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:28:41.885 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:28:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.880+0000 7fe00bfff700 1 -- 172.21.15.121:0/444535893 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+679 (secure 0 0 0) 0x7fe014059e10 con 0x7fe0240feef0 2024-04-05T21:28:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.882+0000 7fe029593700 1 -- 172.21.15.121:0/444535893 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe00c06ce20 msgr2=0x7fe00c06f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.882+0000 7fe029593700 1 --2- 172.21.15.121:0/444535893 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe00c06ce20 0x7fe00c06f2e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe01800b5c0 tx=0x7fe018005a90 comp rx=0 tx=0).stop 2024-04-05T21:28:41.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.882+0000 7fe029593700 1 -- 172.21.15.121:0/444535893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0240feef0 msgr2=0x7fe024196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.882+0000 7fe029593700 1 --2- 172.21.15.121:0/444535893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0240feef0 0x7fe024196330 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe01400d900 tx=0x7fe01400dcc0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.882+0000 7fe029593700 1 -- 172.21.15.121:0/444535893 shutdown_connections 2024-04-05T21:28:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.882+0000 7fe029593700 1 --2- 172.21.15.121:0/444535893 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe00c06ce20 0x7fe00c06f2e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.882+0000 7fe029593700 1 --2- 172.21.15.121:0/444535893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0240feef0 0x7fe024196330 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.882+0000 7fe029593700 1 --2- 172.21.15.121:0/444535893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe024101f70 0x7fe024196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.882+0000 7fe029593700 1 -- 172.21.15.121:0/444535893 >> 172.21.15.121:0/444535893 conn(0x7fe0240fa810 msgr2=0x7fe0241051a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.882+0000 7fe029593700 1 -- 172.21.15.121:0/444535893 shutdown_connections 2024-04-05T21:28:41.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.882+0000 7fe029593700 1 -- 172.21.15.121:0/444535893 wait complete. 2024-04-05T21:28:41.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 -- 172.21.15.121:0/3130543478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde380ffab0 msgr2=0x7fde380ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:41.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 --2- 172.21.15.121:0/3130543478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde380ffab0 0x7fde380ffed0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fde28009910 tx=0x7fde28009c20 comp rx=0 tx=0).stop 2024-04-05T21:28:41.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 -- 172.21.15.121:0/3130543478 shutdown_connections 2024-04-05T21:28:41.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 --2- 172.21.15.121:0/3130543478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde38100cb0 0x7fde38101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 --2- 172.21.15.121:0/3130543478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde380ffab0 0x7fde380ffed0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 -- 172.21.15.121:0/3130543478 >> 172.21.15.121:0/3130543478 conn(0x7fde380fb030 msgr2=0x7fde380fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:41.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 -- 172.21.15.121:0/3130543478 shutdown_connections 2024-04-05T21:28:41.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 -- 172.21.15.121:0/3130543478 wait complete. 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 Processor -- start 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 -- start start 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde380ffab0 0x7fde38103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde38100cb0 0x7fde38102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde381025d0 con 0x7fde380ffab0 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.986+0000 7fde40788700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde38102710 con 0x7fde38100cb0 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.987+0000 7fde3e524700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde380ffab0 0x7fde38103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.987+0000 7fde3e524700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde380ffab0 0x7fde38103a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35562/0 (socket says 172.21.15.121:35562) 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.987+0000 7fde3e524700 1 -- 172.21.15.121:0/60325765 learned_addr learned my addr 172.21.15.121:0/60325765 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.987+0000 7fde3e524700 1 -- 172.21.15.121:0/60325765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde38100cb0 msgr2=0x7fde38102090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.987+0000 7fde3e524700 1 --2- 172.21.15.121:0/60325765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde38100cb0 0x7fde38102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.987+0000 7fde3e524700 1 -- 172.21.15.121:0/60325765 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde280095f0 con 0x7fde380ffab0 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.987+0000 7fde3e524700 1 --2- 172.21.15.121:0/60325765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde380ffab0 0x7fde38103a40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fde2800bda0 tx=0x7fde2800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:41.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.987+0000 7fde2f7fe700 1 -- 172.21.15.121:0/60325765 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde2801d070 con 0x7fde380ffab0 2024-04-05T21:28:41.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.987+0000 7fde40788700 1 -- 172.21.15.121:0/60325765 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde38102990 con 0x7fde380ffab0 2024-04-05T21:28:41.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.987+0000 7fde2f7fe700 1 -- 172.21.15.121:0/60325765 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fde28022470 con 0x7fde380ffab0 2024-04-05T21:28:41.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.987+0000 7fde2f7fe700 1 -- 172.21.15.121:0/60325765 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde2800f690 con 0x7fde380ffab0 2024-04-05T21:28:41.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:41 smithi121 ceph-mon[39723]: pgmap v38: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T21:28:41.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.987+0000 7fde40788700 1 -- 172.21.15.121:0/60325765 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde38102e80 con 0x7fde380ffab0 2024-04-05T21:28:41.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.988+0000 7fde40788700 1 -- 172.21.15.121:0/60325765 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde38108550 con 0x7fde380ffab0 2024-04-05T21:28:41.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.988+0000 7fde2f7fe700 1 -- 172.21.15.121:0/60325765 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 32) v1 ==== 90946+0+0 (secure 0 0 0) 0x7fde280229c0 con 0x7fde380ffab0 2024-04-05T21:28:41.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.989+0000 7fde2f7fe700 1 --2- 172.21.15.121:0/60325765 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde2406ce20 0x7fde2406f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:28:41.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.989+0000 7fde2f7fe700 1 -- 172.21.15.121:0/60325765 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fde2808dfc0 con 0x7fde380ffab0 2024-04-05T21:28:41.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.989+0000 7fde3dd23700 1 --2- 172.21.15.121:0/60325765 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde2406ce20 0x7fde2406f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:28:41.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.989+0000 7fde3dd23700 1 --2- 172.21.15.121:0/60325765 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde2406ce20 0x7fde2406f2e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fde340097b0 tx=0x7fde34006d20 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:28:41.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:41.993+0000 7fde2f7fe700 1 -- 172.21.15.121:0/60325765 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7fde2805bb20 con 0x7fde380ffab0 2024-04-05T21:28:42.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:41 smithi188 ceph-mon[49715]: pgmap v38: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T21:28:42.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:42.206+0000 7fde40788700 1 -- 172.21.15.121:0/60325765 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde38068b20 con 0x7fde2406ce20 2024-04-05T21:28:42.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:42.207+0000 7fde2f7fe700 1 -- 172.21.15.121:0/60325765 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fde38068b20 con 0x7fde2406ce20 2024-04-05T21:28:42.208 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:28:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:28:42.208 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:28:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T21:28:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:28:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr" 2024-04-05T21:28:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:28:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "2/2 daemons upgraded", 2024-04-05T21:28:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-05T21:28:42.209 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:28:42.209 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:28:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:42.209+0000 7fde40788700 1 -- 172.21.15.121:0/60325765 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde2406ce20 msgr2=0x7fde2406f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:42.209+0000 7fde40788700 1 --2- 172.21.15.121:0/60325765 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde2406ce20 0x7fde2406f2e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fde340097b0 tx=0x7fde34006d20 comp rx=0 tx=0).stop 2024-04-05T21:28:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:42.209+0000 7fde40788700 1 -- 172.21.15.121:0/60325765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde380ffab0 msgr2=0x7fde38103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:28:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:42.209+0000 7fde40788700 1 --2- 172.21.15.121:0/60325765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde380ffab0 0x7fde38103a40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fde2800bda0 tx=0x7fde2800be80 comp rx=0 tx=0).stop 2024-04-05T21:28:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:42.209+0000 7fde40788700 1 -- 172.21.15.121:0/60325765 shutdown_connections 2024-04-05T21:28:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:42.209+0000 7fde40788700 1 --2- 172.21.15.121:0/60325765 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde2406ce20 0x7fde2406f2e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:42.209+0000 7fde40788700 1 --2- 172.21.15.121:0/60325765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde380ffab0 0x7fde38103a40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:42.209+0000 7fde40788700 1 --2- 172.21.15.121:0/60325765 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde38100cb0 0x7fde38102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:28:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:42.209+0000 7fde40788700 1 -- 172.21.15.121:0/60325765 >> 172.21.15.121:0/60325765 conn(0x7fde380fb030 msgr2=0x7fde38068410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:28:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:42.210+0000 7fde40788700 1 -- 172.21.15.121:0/60325765 shutdown_connections 2024-04-05T21:28:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:28:42.210+0000 7fde40788700 1 -- 172.21.15.121:0/60325765 wait complete. 2024-04-05T21:28:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:42 smithi121 ceph-mon[39723]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:42 smithi121 ceph-mon[39723]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:42 smithi121 ceph-mon[39723]: from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:42 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/444535893' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:43.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:42 smithi188 ceph-mon[49715]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:43.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:42 smithi188 ceph-mon[49715]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:43.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:42 smithi188 ceph-mon[49715]: from='client.14732 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:43.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:42 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/444535893' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:28:43.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:43 smithi121 ceph-mon[39723]: pgmap v39: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-05T21:28:43.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:43 smithi121 ceph-mon[39723]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:43.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:43 smithi121 ceph-mon[39723]: Upgrade: Updating prometheus.smithi121 2024-04-05T21:28:44.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:43 smithi188 ceph-mon[49715]: pgmap v39: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-05T21:28:44.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:43 smithi188 ceph-mon[49715]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:28:44.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:43 smithi188 ceph-mon[49715]: Upgrade: Updating prometheus.smithi121 2024-04-05T21:28:44.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:44 smithi121 ceph-mon[39723]: Deploying daemon prometheus.smithi121 on smithi121 2024-04-05T21:28:45.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:44 smithi188 ceph-mon[49715]: Deploying daemon prometheus.smithi121 on smithi121 2024-04-05T21:28:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:45 smithi121 ceph-mon[39723]: pgmap v40: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T21:28:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:45 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:28:46.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:45 smithi188 ceph-mon[49715]: pgmap v40: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T21:28:46.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:45 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:28:47.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:47 smithi121 ceph-mon[39723]: pgmap v41: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-04-05T21:28:48.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:47 smithi188 ceph-mon[49715]: pgmap v41: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-04-05T21:28:48.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:48 smithi121 ceph-mon[39723]: pgmap v42: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-05T21:28:49.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:48 smithi188 ceph-mon[49715]: pgmap v42: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-05T21:28:51.408 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:51 smithi121 ceph-mon[39723]: pgmap v43: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T21:28:51.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:51 smithi188 ceph-mon[49715]: pgmap v43: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T21:28:52.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:52 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:52.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:52 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:52.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:52 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:28:53.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:52 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:53.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:52 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:53.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:52 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:28:53.826 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:53 smithi121 ceph-mon[39723]: pgmap v44: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-05T21:28:54.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:53 smithi188 ceph-mon[49715]: pgmap v44: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-05T21:28:55.893 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:55 smithi188 ceph-mon[49715]: pgmap v45: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T21:28:55.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:55 smithi121 ceph-mon[39723]: pgmap v45: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T21:28:57.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:57 smithi121 ceph-mon[39723]: pgmap v46: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-04-05T21:28:57.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:57 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:57.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:57 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:58.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:57 smithi188 ceph-mon[49715]: pgmap v46: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 5 op/s 2024-04-05T21:28:58.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:57 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:58.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:57 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:58.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:58 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:58.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:58 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:58.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:28:58 smithi121 ceph-mon[39723]: pgmap v47: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-05T21:28:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:58 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:59.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:58 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:28:59.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:28:58 smithi188 ceph-mon[49715]: pgmap v47: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 7 op/s 2024-04-05T21:29:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:00 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:29:00.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:00 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:29:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:01 smithi121 ceph-mon[39723]: pgmap v48: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T21:29:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:01 smithi121 ceph-mon[39723]: mgrmap e33: smithi121.etbquh(active, since 91s), standbys: smithi188.tgcuir 2024-04-05T21:29:01.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:01 smithi188 ceph-mon[49715]: pgmap v48: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-05T21:29:01.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:01 smithi188 ceph-mon[49715]: mgrmap e33: smithi121.etbquh(active, since 91s), standbys: smithi188.tgcuir 2024-04-05T21:29:03.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:03.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:03.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:29:03.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:29:03.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:03.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:29:03.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:29:03.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:03.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:02 smithi121 ceph-mon[39723]: pgmap v49: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-05T21:29:03.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:03.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:03.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:29:03.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:29:03.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:03.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:29:03.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-05T21:29:03.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:03.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:03.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:02 smithi188 ceph-mon[49715]: pgmap v49: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-05T21:29:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:05 smithi121 ceph-mon[39723]: Upgrade: Updating alertmanager.smithi121 2024-04-05T21:29:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:05 smithi121 ceph-mon[39723]: Deploying daemon alertmanager.smithi121 on smithi121 2024-04-05T21:29:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:05 smithi121 ceph-mon[39723]: pgmap v50: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-05T21:29:05.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:05 smithi188 ceph-mon[49715]: Upgrade: Updating alertmanager.smithi121 2024-04-05T21:29:05.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:05 smithi188 ceph-mon[49715]: Deploying daemon alertmanager.smithi121 on smithi121 2024-04-05T21:29:05.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:05 smithi188 ceph-mon[49715]: pgmap v50: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-05T21:29:07.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:07 smithi121 ceph-mon[39723]: pgmap v51: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-05T21:29:07.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:07 smithi188 ceph-mon[49715]: pgmap v51: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-05T21:29:09.736 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:09 smithi121 ceph-mon[39723]: pgmap v52: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-05T21:29:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:09 smithi188 ceph-mon[49715]: pgmap v52: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-05T21:29:11.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:10 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:11.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:10 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:11.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:10 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:11.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:10 smithi121 ceph-mon[39723]: pgmap v53: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T21:29:11.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:10 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:11.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:10 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:11.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:10 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:11.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:10 smithi188 ceph-mon[49715]: pgmap v53: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-04-05T21:29:12.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.323+0000 7f49ce33d700 1 -- 172.21.15.121:0/887299629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49c8101090 msgr2=0x7f49c81014b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:12.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.323+0000 7f49ce33d700 1 --2- 172.21.15.121:0/887299629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49c8101090 0x7f49c81014b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f49b8009910 tx=0x7f49b8009c20 comp rx=0 tx=0).stop 2024-04-05T21:29:12.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.323+0000 7f49ce33d700 1 -- 172.21.15.121:0/887299629 shutdown_connections 2024-04-05T21:29:12.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.323+0000 7f49ce33d700 1 --2- 172.21.15.121:0/887299629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49c80ffcf0 0x7f49c8100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.323+0000 7f49ce33d700 1 --2- 172.21.15.121:0/887299629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49c8101090 0x7f49c81014b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.323+0000 7f49ce33d700 1 -- 172.21.15.121:0/887299629 >> 172.21.15.121:0/887299629 conn(0x7f49c80fa780 msgr2=0x7f49c80fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.323+0000 7f49ce33d700 1 -- 172.21.15.121:0/887299629 shutdown_connections 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.323+0000 7f49ce33d700 1 -- 172.21.15.121:0/887299629 wait complete. 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.323+0000 7f49ce33d700 1 Processor -- start 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.323+0000 7f49ce33d700 1 -- start start 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49ce33d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49c80ffcf0 0x7f49c8103a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49ce33d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49c8101090 0x7f49c81020e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49ce33d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c8102620 con 0x7f49c80ffcf0 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49ce33d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c8102760 con 0x7f49c8101090 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49c7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49c80ffcf0 0x7f49c8103a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49c7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49c80ffcf0 0x7f49c8103a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60084/0 (socket says 172.21.15.121:60084) 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49c7fff700 1 -- 172.21.15.121:0/1938252701 learned_addr learned my addr 172.21.15.121:0/1938252701 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:29:12.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49c77fe700 1 --2- 172.21.15.121:0/1938252701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49c8101090 0x7f49c81020e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:12.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49c7fff700 1 -- 172.21.15.121:0/1938252701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49c8101090 msgr2=0x7f49c81020e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:12.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49c7fff700 1 --2- 172.21.15.121:0/1938252701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49c8101090 0x7f49c81020e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49c7fff700 1 -- 172.21.15.121:0/1938252701 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b80095f0 con 0x7f49c80ffcf0 2024-04-05T21:29:12.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49c7fff700 1 --2- 172.21.15.121:0/1938252701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49c80ffcf0 0x7f49c8103a90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f49b800bda0 tx=0x7f49b800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:12.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49c57fa700 1 -- 172.21.15.121:0/1938252701 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49b801d070 con 0x7f49c80ffcf0 2024-04-05T21:29:12.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49c57fa700 1 -- 172.21.15.121:0/1938252701 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f49b800f460 con 0x7f49c80ffcf0 2024-04-05T21:29:12.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49ce33d700 1 -- 172.21.15.121:0/1938252701 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c81029e0 con 0x7f49c80ffcf0 2024-04-05T21:29:12.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49c57fa700 1 -- 172.21.15.121:0/1938252701 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49b8022650 con 0x7f49c80ffcf0 2024-04-05T21:29:12.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.324+0000 7f49ce33d700 1 -- 172.21.15.121:0/1938252701 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c8102ed0 con 0x7f49c80ffcf0 2024-04-05T21:29:12.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.326+0000 7f49ce33d700 1 -- 172.21.15.121:0/1938252701 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c81084b0 con 0x7f49c80ffcf0 2024-04-05T21:29:12.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.326+0000 7f49c57fa700 1 -- 172.21.15.121:0/1938252701 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f49b8003a60 con 0x7f49c80ffcf0 2024-04-05T21:29:12.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.326+0000 7f49c57fa700 1 --2- 172.21.15.121:0/1938252701 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f49b006cce0 0x7f49b006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:12.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.326+0000 7f49c57fa700 1 -- 172.21.15.121:0/1938252701 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f49b808d880 con 0x7f49c80ffcf0 2024-04-05T21:29:12.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.326+0000 7f49c77fe700 1 --2- 172.21.15.121:0/1938252701 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f49b006cce0 0x7f49b006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:12.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.327+0000 7f49c77fe700 1 --2- 172.21.15.121:0/1938252701 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f49b006cce0 0x7f49b006f1a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f49bc005950 tx=0x7f49bc00b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:12.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.331+0000 7f49c57fa700 1 -- 172.21.15.121:0/1938252701 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f49b805b430 con 0x7f49c80ffcf0 2024-04-05T21:29:12.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.547+0000 7f49ce33d700 1 -- 172.21.15.121:0/1938252701 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f49c80ff2f0 con 0x7f49b006cce0 2024-04-05T21:29:12.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.548+0000 7f49c57fa700 1 -- 172.21.15.121:0/1938252701 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f49c80ff2f0 con 0x7f49b006cce0 2024-04-05T21:29:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.550+0000 7f49ce33d700 1 -- 172.21.15.121:0/1938252701 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f49b006cce0 msgr2=0x7f49b006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.550+0000 7f49ce33d700 1 --2- 172.21.15.121:0/1938252701 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f49b006cce0 0x7f49b006f1a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f49bc005950 tx=0x7f49bc00b410 comp rx=0 tx=0).stop 2024-04-05T21:29:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.550+0000 7f49ce33d700 1 -- 172.21.15.121:0/1938252701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49c80ffcf0 msgr2=0x7f49c8103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.550+0000 7f49ce33d700 1 --2- 172.21.15.121:0/1938252701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49c80ffcf0 0x7f49c8103a90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f49b800bda0 tx=0x7f49b800bdd0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.550+0000 7f49ce33d700 1 -- 172.21.15.121:0/1938252701 shutdown_connections 2024-04-05T21:29:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.550+0000 7f49ce33d700 1 --2- 172.21.15.121:0/1938252701 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f49b006cce0 0x7f49b006f1a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.550+0000 7f49ce33d700 1 --2- 172.21.15.121:0/1938252701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49c80ffcf0 0x7f49c8103a90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.550+0000 7f49ce33d700 1 --2- 172.21.15.121:0/1938252701 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49c8101090 0x7f49c81020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.550+0000 7f49ce33d700 1 -- 172.21.15.121:0/1938252701 >> 172.21.15.121:0/1938252701 conn(0x7f49c80fa780 msgr2=0x7f49c80febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.550+0000 7f49ce33d700 1 -- 172.21.15.121:0/1938252701 shutdown_connections 2024-04-05T21:29:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.550+0000 7f49ce33d700 1 -- 172.21.15.121:0/1938252701 wait complete. 2024-04-05T21:29:12.562 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:29:12.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.656+0000 7f9649af3700 1 -- 172.21.15.121:0/3430786181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9644101db0 msgr2=0x7f9644102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:12.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.656+0000 7f9649af3700 1 --2- 172.21.15.121:0/3430786181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9644101db0 0x7f9644102210 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9634009910 tx=0x7f9634009c20 comp rx=0 tx=0).stop 2024-04-05T21:29:12.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.656+0000 7f9649af3700 1 -- 172.21.15.121:0/3430786181 shutdown_connections 2024-04-05T21:29:12.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.656+0000 7f9649af3700 1 --2- 172.21.15.121:0/3430786181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9644101db0 0x7f9644102210 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.656+0000 7f9649af3700 1 --2- 172.21.15.121:0/3430786181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9644100bb0 0x7f9644100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.656+0000 7f9649af3700 1 -- 172.21.15.121:0/3430786181 >> 172.21.15.121:0/3430786181 conn(0x7f96440fc110 msgr2=0x7f96440fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:12.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.656+0000 7f9649af3700 1 -- 172.21.15.121:0/3430786181 shutdown_connections 2024-04-05T21:29:12.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.656+0000 7f9649af3700 1 -- 172.21.15.121:0/3430786181 wait complete. 2024-04-05T21:29:12.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f9649af3700 1 Processor -- start 2024-04-05T21:29:12.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f9649af3700 1 -- start start 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f9649af3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9644100bb0 0x7f9644198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f9649af3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9644101db0 0x7f96441989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f9649af3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9644198ff0 con 0x7f9644101db0 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f9649af3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9644199130 con 0x7f9644100bb0 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f9642ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9644101db0 0x7f96441989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f9642ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9644101db0 0x7f96441989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60116/0 (socket says 172.21.15.121:60116) 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f9642ffd700 1 -- 172.21.15.121:0/3027858792 learned_addr learned my addr 172.21.15.121:0/3027858792 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f96437fe700 1 --2- 172.21.15.121:0/3027858792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9644100bb0 0x7f9644198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f9642ffd700 1 -- 172.21.15.121:0/3027858792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9644100bb0 msgr2=0x7f9644198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f9642ffd700 1 --2- 172.21.15.121:0/3027858792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9644100bb0 0x7f9644198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f9642ffd700 1 -- 172.21.15.121:0/3027858792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96340095f0 con 0x7f9644101db0 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.657+0000 7f96437fe700 1 --2- 172.21.15.121:0/3027858792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9644100bb0 0x7f9644198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.658+0000 7f9642ffd700 1 --2- 172.21.15.121:0/3027858792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9644101db0 0x7f96441989d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f963400bde0 tx=0x7f963400bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.658+0000 7f9640ff9700 1 -- 172.21.15.121:0/3027858792 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f963401d070 con 0x7f9644101db0 2024-04-05T21:29:12.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.658+0000 7f9640ff9700 1 -- 172.21.15.121:0/3027858792 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f9634005710 con 0x7f9644101db0 2024-04-05T21:29:12.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.658+0000 7f9649af3700 1 -- 172.21.15.121:0/3027858792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964419db80 con 0x7f9644101db0 2024-04-05T21:29:12.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.658+0000 7f9640ff9700 1 -- 172.21.15.121:0/3027858792 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9634003ae0 con 0x7f9644101db0 2024-04-05T21:29:12.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.658+0000 7f9649af3700 1 -- 172.21.15.121:0/3027858792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964419e090 con 0x7f9644101db0 2024-04-05T21:29:12.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.659+0000 7f9649af3700 1 -- 172.21.15.121:0/3027858792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9644105e90 con 0x7f9644101db0 2024-04-05T21:29:12.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.659+0000 7f9640ff9700 1 -- 172.21.15.121:0/3027858792 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f9634003d50 con 0x7f9644101db0 2024-04-05T21:29:12.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.659+0000 7f9640ff9700 1 --2- 172.21.15.121:0/3027858792 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f963006cce0 0x7f963006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:12.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.660+0000 7f9640ff9700 1 -- 172.21.15.121:0/3027858792 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f963408dcd0 con 0x7f9644101db0 2024-04-05T21:29:12.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.660+0000 7f96437fe700 1 --2- 172.21.15.121:0/3027858792 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f963006cce0 0x7f963006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:12.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.660+0000 7f96437fe700 1 --2- 172.21.15.121:0/3027858792 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f963006cce0 0x7f963006f1a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9644101c10 tx=0x7f962c009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:12.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.664+0000 7f9640ff9700 1 -- 172.21.15.121:0/3027858792 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f963405b880 con 0x7f9644101db0 2024-04-05T21:29:12.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.898+0000 7f9649af3700 1 -- 172.21.15.121:0/3027858792 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f96440611d0 con 0x7f963006cce0 2024-04-05T21:29:12.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.899+0000 7f9640ff9700 1 -- 172.21.15.121:0/3027858792 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f96440611d0 con 0x7f963006cce0 2024-04-05T21:29:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.901+0000 7f9649af3700 1 -- 172.21.15.121:0/3027858792 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f963006cce0 msgr2=0x7f963006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.901+0000 7f9649af3700 1 --2- 172.21.15.121:0/3027858792 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f963006cce0 0x7f963006f1a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9644101c10 tx=0x7f962c009260 comp rx=0 tx=0).stop 2024-04-05T21:29:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.901+0000 7f9649af3700 1 -- 172.21.15.121:0/3027858792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9644101db0 msgr2=0x7f96441989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.901+0000 7f9649af3700 1 --2- 172.21.15.121:0/3027858792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9644101db0 0x7f96441989d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f963400bde0 tx=0x7f963400bec0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.901+0000 7f9649af3700 1 -- 172.21.15.121:0/3027858792 shutdown_connections 2024-04-05T21:29:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.901+0000 7f9649af3700 1 --2- 172.21.15.121:0/3027858792 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f963006cce0 0x7f963006f1a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.902+0000 7f9649af3700 1 --2- 172.21.15.121:0/3027858792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9644100bb0 0x7f9644198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.902+0000 7f9649af3700 1 --2- 172.21.15.121:0/3027858792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9644101db0 0x7f96441989d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.902+0000 7f9649af3700 1 -- 172.21.15.121:0/3027858792 >> 172.21.15.121:0/3027858792 conn(0x7f96440fc110 msgr2=0x7f9644102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.902+0000 7f9649af3700 1 -- 172.21.15.121:0/3027858792 shutdown_connections 2024-04-05T21:29:12.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:12.902+0000 7f9649af3700 1 -- 172.21.15.121:0/3027858792 wait complete. 2024-04-05T21:29:13.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.012+0000 7fb8135d4700 1 -- 172.21.15.121:0/1588228350 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb80c103ff0 msgr2=0x7fb80c1063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:13.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.012+0000 7fb8135d4700 1 --2- 172.21.15.121:0/1588228350 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb80c103ff0 0x7fb80c1063e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb808009910 tx=0x7fb808009c20 comp rx=0 tx=0).stop 2024-04-05T21:29:13.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.012+0000 7fb8135d4700 1 -- 172.21.15.121:0/1588228350 shutdown_connections 2024-04-05T21:29:13.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.012+0000 7fb8135d4700 1 --2- 172.21.15.121:0/1588228350 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb80c103ff0 0x7fb80c1063e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.012+0000 7fb8135d4700 1 --2- 172.21.15.121:0/1588228350 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb80c1016c0 0x7fb80c103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.012+0000 7fb8135d4700 1 -- 172.21.15.121:0/1588228350 >> 172.21.15.121:0/1588228350 conn(0x7fb80c0fb030 msgr2=0x7fb80c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:13.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.012+0000 7fb8135d4700 1 -- 172.21.15.121:0/1588228350 shutdown_connections 2024-04-05T21:29:13.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.012+0000 7fb8135d4700 1 -- 172.21.15.121:0/1588228350 wait complete. 2024-04-05T21:29:13.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.012+0000 7fb8135d4700 1 Processor -- start 2024-04-05T21:29:13.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.012+0000 7fb8135d4700 1 -- start start 2024-04-05T21:29:13.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb8135d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb80c1016c0 0x7fb80c196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:13.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb8135d4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb80c103ff0 0x7fb80c196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:13.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb8135d4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c196d20 con 0x7fb80c1016c0 2024-04-05T21:29:13.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb8135d4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c196e60 con 0x7fb80c103ff0 2024-04-05T21:29:13.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb811370700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb80c1016c0 0x7fb80c196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:13.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb811370700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb80c1016c0 0x7fb80c196250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60144/0 (socket says 172.21.15.121:60144) 2024-04-05T21:29:13.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb811370700 1 -- 172.21.15.121:0/896959330 learned_addr learned my addr 172.21.15.121:0/896959330 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:29:13.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb810b6f700 1 --2- 172.21.15.121:0/896959330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb80c103ff0 0x7fb80c196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:13.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb811370700 1 -- 172.21.15.121:0/896959330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb80c103ff0 msgr2=0x7fb80c196790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:13.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb811370700 1 --2- 172.21.15.121:0/896959330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb80c103ff0 0x7fb80c196790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb811370700 1 -- 172.21.15.121:0/896959330 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8080095f0 con 0x7fb80c1016c0 2024-04-05T21:29:13.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb811370700 1 --2- 172.21.15.121:0/896959330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb80c1016c0 0x7fb80c196250 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00c930 tx=0x7fb7fc00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:13.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb8027fc700 1 -- 172.21.15.121:0/896959330 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7fc007ad0 con 0x7fb80c1016c0 2024-04-05T21:29:13.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb8027fc700 1 -- 172.21.15.121:0/896959330 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fb7fc007c30 con 0x7fb80c1016c0 2024-04-05T21:29:13.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb8135d4700 1 -- 172.21.15.121:0/896959330 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80c19b920 con 0x7fb80c1016c0 2024-04-05T21:29:13.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb8027fc700 1 -- 172.21.15.121:0/896959330 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7fc018630 con 0x7fb80c1016c0 2024-04-05T21:29:13.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.013+0000 7fb8135d4700 1 -- 172.21.15.121:0/896959330 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80c19bec0 con 0x7fb80c1016c0 2024-04-05T21:29:13.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.014+0000 7fb8135d4700 1 -- 172.21.15.121:0/896959330 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80c1903e0 con 0x7fb80c1016c0 2024-04-05T21:29:13.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.015+0000 7fb8027fc700 1 -- 172.21.15.121:0/896959330 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fb7fc00f450 con 0x7fb80c1016c0 2024-04-05T21:29:13.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.015+0000 7fb8027fc700 1 --2- 172.21.15.121:0/896959330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb7f806cce0 0x7fb7f806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:13.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.015+0000 7fb8027fc700 1 -- 172.21.15.121:0/896959330 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fb7fc08c6f0 con 0x7fb80c1016c0 2024-04-05T21:29:13.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.015+0000 7fb810b6f700 1 --2- 172.21.15.121:0/896959330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb7f806cce0 0x7fb7f806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:13.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.016+0000 7fb810b6f700 1 --2- 172.21.15.121:0/896959330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb7f806cce0 0x7fb7f806f1a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb8080059d0 tx=0x7fb808005920 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:13.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.020+0000 7fb8027fc700 1 -- 172.21.15.121:0/896959330 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7fb7fc05a2a0 con 0x7fb80c1016c0 2024-04-05T21:29:13.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.224+0000 7fb8135d4700 1 -- 172.21.15.121:0/896959330 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb80c0611d0 con 0x7fb7f806cce0 2024-04-05T21:29:13.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.236+0000 7fb8027fc700 1 -- 172.21.15.121:0/896959330 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb80c0611d0 con 0x7fb7f806cce0 2024-04-05T21:29:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:29:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 starting - - - - 2024-04-05T21:29:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (11m) 16s ago 11m 7977k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:29:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (10m) 44s ago 10m 7834k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:29:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (11m) 16s ago 11m 6651k - 18.2.1 5be31c24972a 5bf35fe2ace4 2024-04-05T21:29:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (10m) 44s ago 10m 6651k - 18.2.1 5be31c24972a dcdfa6ae17bf 2024-04-05T21:29:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (9m) 16s ago 11m 90.1M - 9.4.7 954c08fa6188 ee38366c66e3 2024-04-05T21:29:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (5m) 16s ago 5m 64.5M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:29:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (5m) 16s ago 5m 134M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:29:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (5m) 44s ago 5m 110M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:29:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (5m) 44s ago 5m 62.4M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:29:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (2m) 16s ago 13m 497M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:29:13.237 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (71s) 44s ago 10m 448M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:29:13.238 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (13m) 16s ago 13m 56.2M 2048M 18.2.1 5be31c24972a 3c30ce99c006 2024-04-05T21:29:13.238 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (10m) 44s ago 10m 45.2M 2048M 18.2.1 5be31c24972a 8ecc9636670e 2024-04-05T21:29:13.238 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (52s) 16s ago 11m 8036k - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:29:13.238 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (48s) 44s ago 10m 3653k - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:29:13.238 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (9m) 16s ago 9m 214M 1227M 18.2.1 5be31c24972a f6f3a45429a2 2024-04-05T21:29:13.238 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (8m) 16s ago 8m 242M 1227M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:29:13.238 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (8m) 16s ago 8m 180M 1227M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:29:13.238 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (8m) 44s ago 7m 286M 2251M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:29:13.238 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (7m) 44s ago 7m 244M 2251M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:29:13.238 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (7m) 44s ago 7m 223M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:29:13.238 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (22s) 16s ago 10m 40.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:29:13.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.238+0000 7fb8135d4700 1 -- 172.21.15.121:0/896959330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb7f806cce0 msgr2=0x7fb7f806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:13.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.238+0000 7fb8135d4700 1 --2- 172.21.15.121:0/896959330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb7f806cce0 0x7fb7f806f1a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb8080059d0 tx=0x7fb808005920 comp rx=0 tx=0).stop 2024-04-05T21:29:13.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.238+0000 7fb8135d4700 1 -- 172.21.15.121:0/896959330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb80c1016c0 msgr2=0x7fb80c196250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:13.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.238+0000 7fb8135d4700 1 --2- 172.21.15.121:0/896959330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb80c1016c0 0x7fb80c196250 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00c930 tx=0x7fb7fc00ccf0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.238+0000 7fb8135d4700 1 -- 172.21.15.121:0/896959330 shutdown_connections 2024-04-05T21:29:13.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.238+0000 7fb8135d4700 1 --2- 172.21.15.121:0/896959330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb7f806cce0 0x7fb7f806f1a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.238+0000 7fb8135d4700 1 --2- 172.21.15.121:0/896959330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb80c1016c0 0x7fb80c196250 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.238+0000 7fb8135d4700 1 --2- 172.21.15.121:0/896959330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb80c103ff0 0x7fb80c196790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.238+0000 7fb8135d4700 1 -- 172.21.15.121:0/896959330 >> 172.21.15.121:0/896959330 conn(0x7fb80c0fb030 msgr2=0x7fb80c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:13.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.238+0000 7fb8135d4700 1 -- 172.21.15.121:0/896959330 shutdown_connections 2024-04-05T21:29:13.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.238+0000 7fb8135d4700 1 -- 172.21.15.121:0/896959330 wait complete. 2024-04-05T21:29:13.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.351+0000 7f42e8ed2700 1 -- 172.21.15.121:0/4097846614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e4101640 msgr2=0x7f42e4103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:13.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.351+0000 7f42e8ed2700 1 --2- 172.21.15.121:0/4097846614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e4101640 0x7f42e4103a30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009910 tx=0x7f42d4009c20 comp rx=0 tx=0).stop 2024-04-05T21:29:13.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.352+0000 7f42e8ed2700 1 -- 172.21.15.121:0/4097846614 shutdown_connections 2024-04-05T21:29:13.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.352+0000 7f42e8ed2700 1 --2- 172.21.15.121:0/4097846614 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e4103f70 0x7f42e4106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.352+0000 7f42e8ed2700 1 --2- 172.21.15.121:0/4097846614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e4101640 0x7f42e4103a30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.352+0000 7f42e8ed2700 1 -- 172.21.15.121:0/4097846614 >> 172.21.15.121:0/4097846614 conn(0x7f42e40fb010 msgr2=0x7f42e40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.352+0000 7f42e8ed2700 1 -- 172.21.15.121:0/4097846614 shutdown_connections 2024-04-05T21:29:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.352+0000 7f42e8ed2700 1 -- 172.21.15.121:0/4097846614 wait complete. 2024-04-05T21:29:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.352+0000 7f42e8ed2700 1 Processor -- start 2024-04-05T21:29:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.352+0000 7f42e8ed2700 1 -- start start 2024-04-05T21:29:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.352+0000 7f42e8ed2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e4101640 0x7f42e4196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42e8ed2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e4103f70 0x7f42e41967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42e8ed2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e4196d40 con 0x7f42e4101640 2024-04-05T21:29:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42e8ed2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e4196e80 con 0x7f42e4103f70 2024-04-05T21:29:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42e259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e4101640 0x7f42e4196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42e259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e4101640 0x7f42e4196270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60166/0 (socket says 172.21.15.121:60166) 2024-04-05T21:29:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42e259c700 1 -- 172.21.15.121:0/2217176761 learned_addr learned my addr 172.21.15.121:0/2217176761 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:29:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42e1d9b700 1 --2- 172.21.15.121:0/2217176761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e4103f70 0x7f42e41967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42e259c700 1 -- 172.21.15.121:0/2217176761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e4103f70 msgr2=0x7f42e41967b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42e259c700 1 --2- 172.21.15.121:0/2217176761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e4103f70 0x7f42e41967b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42e259c700 1 -- 172.21.15.121:0/2217176761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d40095f0 con 0x7f42e4101640 2024-04-05T21:29:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42e259c700 1 --2- 172.21.15.121:0/2217176761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e4101640 0x7f42e4196270 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f42d400bd70 tx=0x7f42d400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42d37fe700 1 -- 172.21.15.121:0/2217176761 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42d401d070 con 0x7f42e4101640 2024-04-05T21:29:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42d37fe700 1 -- 172.21.15.121:0/2217176761 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7f42d4005940 con 0x7f42e4101640 2024-04-05T21:29:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42e8ed2700 1 -- 172.21.15.121:0/2217176761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e419b8e0 con 0x7f42e4101640 2024-04-05T21:29:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42d37fe700 1 -- 172.21.15.121:0/2217176761 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42d400f460 con 0x7f42e4101640 2024-04-05T21:29:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.353+0000 7f42e8ed2700 1 -- 172.21.15.121:0/2217176761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e419be20 con 0x7f42e4101640 2024-04-05T21:29:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.354+0000 7f42e8ed2700 1 -- 172.21.15.121:0/2217176761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e41903e0 con 0x7f42e4101640 2024-04-05T21:29:13.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.355+0000 7f42d37fe700 1 -- 172.21.15.121:0/2217176761 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f42d4022800 con 0x7f42e4101640 2024-04-05T21:29:13.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.356+0000 7f42d37fe700 1 --2- 172.21.15.121:0/2217176761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f42cc06cce0 0x7f42cc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:13.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.356+0000 7f42d37fe700 1 -- 172.21.15.121:0/2217176761 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f42d408dc20 con 0x7f42e4101640 2024-04-05T21:29:13.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.356+0000 7f42e1d9b700 1 --2- 172.21.15.121:0/2217176761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f42cc06cce0 0x7f42cc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:13.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.356+0000 7f42e1d9b700 1 --2- 172.21.15.121:0/2217176761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f42cc06cce0 0x7f42cc06f1a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f42d8005ea0 tx=0x7f42d8005e30 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:13.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.362+0000 7f42d37fe700 1 -- 172.21.15.121:0/2217176761 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f42d405b9a0 con 0x7f42e4101640 2024-04-05T21:29:13.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:13 smithi121 ceph-mon[39723]: pgmap v54: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-05T21:29:13.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:13 smithi121 ceph-mon[39723]: from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:29:13.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:13 smithi188 ceph-mon[49715]: pgmap v54: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2024-04-05T21:29:13.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:13 smithi188 ceph-mon[49715]: from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:29:13.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.638+0000 7f42e8ed2700 1 -- 172.21.15.121:0/2217176761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f42e402ced0 con 0x7f42e4101640 2024-04-05T21:29:13.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.638+0000 7f42d37fe700 1 -- 172.21.15.121:0/2217176761 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+679 (secure 0 0 0) 0x7f42d40272c0 con 0x7f42e4101640 2024-04-05T21:29:13.639 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:29:13.639 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:29:13.640 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-05T21:29:13.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:29:13.640 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:29:13.640 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:29:13.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:29:13.640 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:29:13.640 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-05T21:29:13.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:29:13.640 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:29:13.640 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:29:13.640 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:29:13.641 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:29:13.641 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-05T21:29:13.641 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:29:13.641 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:29:13.641 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:29:13.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.640+0000 7f42e8ed2700 1 -- 172.21.15.121:0/2217176761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f42cc06cce0 msgr2=0x7f42cc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:13.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.640+0000 7f42e8ed2700 1 --2- 172.21.15.121:0/2217176761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f42cc06cce0 0x7f42cc06f1a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f42d8005ea0 tx=0x7f42d8005e30 comp rx=0 tx=0).stop 2024-04-05T21:29:13.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.640+0000 7f42e8ed2700 1 -- 172.21.15.121:0/2217176761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e4101640 msgr2=0x7f42e4196270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:13.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.640+0000 7f42e8ed2700 1 --2- 172.21.15.121:0/2217176761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e4101640 0x7f42e4196270 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f42d400bd70 tx=0x7f42d400be50 comp rx=0 tx=0).stop 2024-04-05T21:29:13.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.640+0000 7f42e8ed2700 1 -- 172.21.15.121:0/2217176761 shutdown_connections 2024-04-05T21:29:13.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.640+0000 7f42e8ed2700 1 --2- 172.21.15.121:0/2217176761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f42cc06cce0 0x7f42cc06f1a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.640+0000 7f42e8ed2700 1 --2- 172.21.15.121:0/2217176761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e4101640 0x7f42e4196270 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.640+0000 7f42e8ed2700 1 --2- 172.21.15.121:0/2217176761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e4103f70 0x7f42e41967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.640+0000 7f42e8ed2700 1 -- 172.21.15.121:0/2217176761 >> 172.21.15.121:0/2217176761 conn(0x7f42e40fb010 msgr2=0x7f42e40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:13.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.640+0000 7f42e8ed2700 1 -- 172.21.15.121:0/2217176761 shutdown_connections 2024-04-05T21:29:13.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.641+0000 7f42e8ed2700 1 -- 172.21.15.121:0/2217176761 wait complete. 2024-04-05T21:29:13.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.753+0000 7fdb5acfb700 1 -- 172.21.15.121:0/1332589876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb540ffcf0 msgr2=0x7fdb54100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:13.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.753+0000 7fdb5acfb700 1 --2- 172.21.15.121:0/1332589876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb540ffcf0 0x7fdb54100170 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009910 tx=0x7fdb40009c20 comp rx=0 tx=0).stop 2024-04-05T21:29:13.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.753+0000 7fdb5acfb700 1 -- 172.21.15.121:0/1332589876 shutdown_connections 2024-04-05T21:29:13.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.753+0000 7fdb5acfb700 1 --2- 172.21.15.121:0/1332589876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb540ffcf0 0x7fdb54100170 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.753+0000 7fdb5acfb700 1 --2- 172.21.15.121:0/1332589876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb54101090 0x7fdb541014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.753+0000 7fdb5acfb700 1 -- 172.21.15.121:0/1332589876 >> 172.21.15.121:0/1332589876 conn(0x7fdb540fa7c0 msgr2=0x7fdb540fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:13.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.753+0000 7fdb5acfb700 1 -- 172.21.15.121:0/1332589876 shutdown_connections 2024-04-05T21:29:13.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.753+0000 7fdb5acfb700 1 -- 172.21.15.121:0/1332589876 wait complete. 2024-04-05T21:29:13.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.753+0000 7fdb5acfb700 1 Processor -- start 2024-04-05T21:29:13.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.753+0000 7fdb5acfb700 1 -- start start 2024-04-05T21:29:13.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb5acfb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb540ffcf0 0x7fdb540ff740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:13.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb5acfb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb54101090 0x7fdb5419b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:13.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb5acfb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5419b610 con 0x7fdb54101090 2024-04-05T21:29:13.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb5acfb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5419b750 con 0x7fdb540ffcf0 2024-04-05T21:29:13.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb54101090 0x7fdb5419b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:13.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb54101090 0x7fdb5419b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60188/0 (socket says 172.21.15.121:60188) 2024-04-05T21:29:13.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb53fff700 1 -- 172.21.15.121:0/2357096813 learned_addr learned my addr 172.21.15.121:0/2357096813 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:29:13.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb58a97700 1 --2- 172.21.15.121:0/2357096813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb540ffcf0 0x7fdb540ff740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:13.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb53fff700 1 -- 172.21.15.121:0/2357096813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb540ffcf0 msgr2=0x7fdb540ff740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:13.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb53fff700 1 --2- 172.21.15.121:0/2357096813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb540ffcf0 0x7fdb540ff740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb53fff700 1 -- 172.21.15.121:0/2357096813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb400095f0 con 0x7fdb54101090 2024-04-05T21:29:13.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb53fff700 1 --2- 172.21.15.121:0/2357096813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb54101090 0x7fdb5419b040 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000bda0 tx=0x7fdb4000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:13.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb51ffb700 1 -- 172.21.15.121:0/2357096813 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb4001d070 con 0x7fdb54101090 2024-04-05T21:29:13.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb51ffb700 1 -- 172.21.15.121:0/2357096813 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1066+0+0 (secure 0 0 0) 0x7fdb4000f460 con 0x7fdb54101090 2024-04-05T21:29:13.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb5acfb700 1 -- 172.21.15.121:0/2357096813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5419b9d0 con 0x7fdb54101090 2024-04-05T21:29:13.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.754+0000 7fdb51ffb700 1 -- 172.21.15.121:0/2357096813 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb40021640 con 0x7fdb54101090 2024-04-05T21:29:13.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.755+0000 7fdb5acfb700 1 -- 172.21.15.121:0/2357096813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5419bf40 con 0x7fdb54101090 2024-04-05T21:29:13.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.756+0000 7fdb3f7fe700 1 -- 172.21.15.121:0/2357096813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb340052f0 con 0x7fdb54101090 2024-04-05T21:29:13.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.757+0000 7fdb51ffb700 1 -- 172.21.15.121:0/2357096813 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fdb4000f9d0 con 0x7fdb54101090 2024-04-05T21:29:13.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.758+0000 7fdb51ffb700 1 --2- 172.21.15.121:0/2357096813 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb4406cd30 0x7fdb4406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:13.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.758+0000 7fdb51ffb700 1 -- 172.21.15.121:0/2357096813 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fdb4008dc60 con 0x7fdb54101090 2024-04-05T21:29:13.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.758+0000 7fdb58a97700 1 --2- 172.21.15.121:0/2357096813 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb4406cd30 0x7fdb4406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:13.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.758+0000 7fdb58a97700 1 --2- 172.21.15.121:0/2357096813 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb4406cd30 0x7fdb4406f1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdb48005950 tx=0x7fdb480058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:13.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.762+0000 7fdb51ffb700 1 -- 172.21.15.121:0/2357096813 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7fdb4005b870 con 0x7fdb54101090 2024-04-05T21:29:13.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.973+0000 7fdb3f7fe700 1 -- 172.21.15.121:0/2357096813 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb34000bc0 con 0x7fdb4406cd30 2024-04-05T21:29:13.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.974+0000 7fdb51ffb700 1 -- 172.21.15.121:0/2357096813 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7fdb34000bc0 con 0x7fdb4406cd30 2024-04-05T21:29:13.975 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:29:13.975 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:29:13.975 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:29:13.975 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-05T21:29:13.976 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:29:13.976 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr" 2024-04-05T21:29:13.976 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:29:13.976 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "2/2 daemons upgraded", 2024-04-05T21:29:13.976 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading alertmanager daemons", 2024-04-05T21:29:13.976 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:29:13.976 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:29:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.976+0000 7fdb3f7fe700 1 -- 172.21.15.121:0/2357096813 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb4406cd30 msgr2=0x7fdb4406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.976+0000 7fdb3f7fe700 1 --2- 172.21.15.121:0/2357096813 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb4406cd30 0x7fdb4406f1f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdb48005950 tx=0x7fdb480058e0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.976+0000 7fdb3f7fe700 1 -- 172.21.15.121:0/2357096813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb54101090 msgr2=0x7fdb5419b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.976+0000 7fdb3f7fe700 1 --2- 172.21.15.121:0/2357096813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb54101090 0x7fdb5419b040 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000bda0 tx=0x7fdb4000be80 comp rx=0 tx=0).stop 2024-04-05T21:29:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.976+0000 7fdb3f7fe700 1 -- 172.21.15.121:0/2357096813 shutdown_connections 2024-04-05T21:29:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.977+0000 7fdb3f7fe700 1 --2- 172.21.15.121:0/2357096813 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb4406cd30 0x7fdb4406f1f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.977+0000 7fdb3f7fe700 1 --2- 172.21.15.121:0/2357096813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb540ffcf0 0x7fdb540ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.977+0000 7fdb3f7fe700 1 --2- 172.21.15.121:0/2357096813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb54101090 0x7fdb5419b040 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.977+0000 7fdb3f7fe700 1 -- 172.21.15.121:0/2357096813 >> 172.21.15.121:0/2357096813 conn(0x7fdb540fa7c0 msgr2=0x7fdb54103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:13.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.977+0000 7fdb3f7fe700 1 -- 172.21.15.121:0/2357096813 shutdown_connections 2024-04-05T21:29:13.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:13.977+0000 7fdb3f7fe700 1 -- 172.21.15.121:0/2357096813 wait complete. 2024-04-05T21:29:14.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:14 smithi121 ceph-mon[39723]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:29:14.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:14 smithi121 ceph-mon[39723]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:29:14.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:14 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2217176761' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:14.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:14 smithi188 ceph-mon[49715]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:29:14.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:14 smithi188 ceph-mon[49715]: from='client.14752 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:29:14.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:14 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2217176761' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:15 smithi121 ceph-mon[39723]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:29:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:15 smithi121 ceph-mon[39723]: pgmap v55: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:15 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:29:15.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:15 smithi188 ceph-mon[49715]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:29:15.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:15 smithi188 ceph-mon[49715]: pgmap v55: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:15.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:15 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:29:16.878 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:16 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:16.878 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:16 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:16.878 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:16 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:16.878 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:16 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:17.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:16 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:17.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:16 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:17.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:16 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:17.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:16 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:17.963 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:17 smithi121 ceph-mon[39723]: pgmap v56: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:29:18.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:17 smithi188 ceph-mon[49715]: pgmap v56: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:29:19.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:19 smithi121 ceph-mon[39723]: pgmap v57: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:20.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:19 smithi188 ceph-mon[49715]: pgmap v57: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:21.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: pgmap v58: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:21.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:21.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:21.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:29:21.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:29:21.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:21.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T21:29:21.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T21:29:21.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:21.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: pgmap v58: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:21.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:21.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:21.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:29:21.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:29:21.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:21.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T21:29:21.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-05T21:29:21.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:21.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:21.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:23.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:23 smithi121 ceph-mon[39723]: Upgrade: Updating grafana.smithi121 2024-04-05T21:29:23.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:23 smithi121 ceph-mon[39723]: Deploying daemon grafana.smithi121 on smithi121 2024-04-05T21:29:23.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:23 smithi121 ceph-mon[39723]: pgmap v59: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:23.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:23 smithi188 ceph-mon[49715]: Upgrade: Updating grafana.smithi121 2024-04-05T21:29:23.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:23 smithi188 ceph-mon[49715]: Deploying daemon grafana.smithi121 on smithi121 2024-04-05T21:29:23.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:23 smithi188 ceph-mon[49715]: pgmap v59: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:25.406 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:25 smithi121 ceph-mon[39723]: pgmap v60: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:25.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:25 smithi188 ceph-mon[49715]: pgmap v60: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:27 smithi121 ceph-mon[39723]: pgmap v61: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:29:27.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:27 smithi188 ceph-mon[49715]: pgmap v61: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:29:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:28 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:28 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:28 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:28.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:28 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:28.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:28 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:28.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:28 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:29 smithi121 ceph-mon[39723]: pgmap v62: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:29:29.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:29 smithi188 ceph-mon[49715]: pgmap v62: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:29:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:30 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:29:30.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:29:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:31 smithi188 ceph-mon[49715]: pgmap v63: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:31.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:31 smithi121 ceph-mon[39723]: pgmap v63: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:33 smithi121 ceph-mon[39723]: pgmap v64: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:33 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:33 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:33 smithi188 ceph-mon[49715]: pgmap v64: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:33.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:33 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:33.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:33 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:34.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:34 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:34.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:34 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:34.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:34 smithi121 ceph-mon[39723]: pgmap v65: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:35.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:34 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:35.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:34 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:35.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:34 smithi188 ceph-mon[49715]: pgmap v65: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:37.665 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:37 smithi121 ceph-mon[39723]: pgmap v66: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:37.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:37 smithi188 ceph-mon[49715]: pgmap v66: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:38.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:38.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:38.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:29:38.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:29:38.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:38.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T21:29:38.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T21:29:38.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:38.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:38.992 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:38.992 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:38.992 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:38.992 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:38.992 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:38.992 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:38.993 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:38.993 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:38.993 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:38.993 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:38.993 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:38.993 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:38.994 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: Upgrade: Finalizing container_image settings 2024-04-05T21:29:38.994 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:38.994 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-05T21:29:38.994 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-05T21:29:38.994 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:38.994 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-05T21:29:38.995 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-05T21:29:38.995 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-05T21:29:38.995 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-05T21:29:38.995 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-05T21:29:38.995 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-05T21:29:38.995 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-05T21:29:38.996 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-05T21:29:38.996 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:38.996 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-05T21:29:38.996 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-05T21:29:38.996 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-05T21:29:38.997 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-05T21:29:38.997 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-05T21:29:38.997 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-05T21:29:38.997 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-05T21:29:38.997 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:38.997 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:38.997 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:38.998 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:38.998 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:38.998 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:38.998 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-05T21:29:38.998 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-05T21:29:38.998 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: Upgrade: Complete! 2024-04-05T21:29:38.999 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-05T21:29:38.999 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-05T21:29:38.999 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:38.999 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: pgmap v67: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:29:39.000 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:29:39.000 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:29:39.000 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:39.000 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:39.000 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:29:39.000 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:29:39.000 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:38 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:29:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:29:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:39.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:39.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: Upgrade: Finalizing container_image settings 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-05T21:29:39.032 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:39.033 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: Upgrade: Complete! 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: pgmap v67: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:29:39.034 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:38 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:41.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:41 smithi121 ceph-mon[39723]: pgmap v68: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:41.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:41 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:41.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:41 smithi188 ceph-mon[49715]: pgmap v68: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:41.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:41 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:29:43.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:43 smithi121 ceph-mon[39723]: pgmap v69: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:43.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:43 smithi188 ceph-mon[49715]: pgmap v69: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 -- 172.21.15.121:0/319146348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07c0101da0 msgr2=0x7f07c0104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 --2- 172.21.15.121:0/319146348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07c0101da0 0x7f07c0104230 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009910 tx=0x7f07b0009c20 comp rx=0 tx=0).stop 2024-04-05T21:29:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 -- 172.21.15.121:0/319146348 shutdown_connections 2024-04-05T21:29:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 --2- 172.21.15.121:0/319146348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07c0101da0 0x7f07c0104230 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 --2- 172.21.15.121:0/319146348 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07c00ff430 0x7f07c0101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 -- 172.21.15.121:0/319146348 >> 172.21.15.121:0/319146348 conn(0x7f07c00fb030 msgr2=0x7f07c00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 -- 172.21.15.121:0/319146348 shutdown_connections 2024-04-05T21:29:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 -- 172.21.15.121:0/319146348 wait complete. 2024-04-05T21:29:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 Processor -- start 2024-04-05T21:29:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 -- start start 2024-04-05T21:29:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07c00ff430 0x7f07c019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07c0101da0 0x7f07c019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c019d310 con 0x7f07c00ff430 2024-04-05T21:29:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.089+0000 7f07c787e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c019d450 con 0x7f07c0101da0 2024-04-05T21:29:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07c561a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07c00ff430 0x7f07c019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07c561a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07c00ff430 0x7f07c019c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55060/0 (socket says 172.21.15.121:55060) 2024-04-05T21:29:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07c561a700 1 -- 172.21.15.121:0/756564668 learned_addr learned my addr 172.21.15.121:0/756564668 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:29:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07c4e19700 1 --2- 172.21.15.121:0/756564668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07c0101da0 0x7f07c019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07c561a700 1 -- 172.21.15.121:0/756564668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07c0101da0 msgr2=0x7f07c019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07c561a700 1 --2- 172.21.15.121:0/756564668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07c0101da0 0x7f07c019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07c561a700 1 -- 172.21.15.121:0/756564668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b00095f0 con 0x7f07c00ff430 2024-04-05T21:29:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07c4e19700 1 --2- 172.21.15.121:0/756564668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07c0101da0 0x7f07c019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:29:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07c561a700 1 --2- 172.21.15.121:0/756564668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07c00ff430 0x7f07c019c840 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00c930 tx=0x7f07bc00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07b67fc700 1 -- 172.21.15.121:0/756564668 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07bc007ad0 con 0x7f07c00ff430 2024-04-05T21:29:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07b67fc700 1 -- 172.21.15.121:0/756564668 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07bc007c30 con 0x7f07c00ff430 2024-04-05T21:29:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07c787e700 1 -- 172.21.15.121:0/756564668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c01a1f10 con 0x7f07c00ff430 2024-04-05T21:29:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07b67fc700 1 -- 172.21.15.121:0/756564668 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07bc018630 con 0x7f07c00ff430 2024-04-05T21:29:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.090+0000 7f07c787e700 1 -- 172.21.15.121:0/756564668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c01a2450 con 0x7f07c00ff430 2024-04-05T21:29:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.091+0000 7f07c787e700 1 -- 172.21.15.121:0/756564668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c01969d0 con 0x7f07c00ff430 2024-04-05T21:29:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.092+0000 7f07b67fc700 1 -- 172.21.15.121:0/756564668 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f07bc00f450 con 0x7f07c00ff430 2024-04-05T21:29:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.092+0000 7f07b67fc700 1 --2- 172.21.15.121:0/756564668 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f07ac06cce0 0x7f07ac06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.092+0000 7f07b67fc700 1 -- 172.21.15.121:0/756564668 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f07bc08c6f0 con 0x7f07c00ff430 2024-04-05T21:29:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.092+0000 7f07c4e19700 1 --2- 172.21.15.121:0/756564668 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f07ac06cce0 0x7f07ac06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.093+0000 7f07c4e19700 1 --2- 172.21.15.121:0/756564668 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f07ac06cce0 0x7f07ac06f1a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f07b000b5c0 tx=0x7f07b0005e20 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.097+0000 7f07b67fc700 1 -- 172.21.15.121:0/756564668 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f07bc05a350 con 0x7f07c00ff430 2024-04-05T21:29:44.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.308+0000 7f07c787e700 1 -- 172.21.15.121:0/756564668 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f07c00611d0 con 0x7f07ac06cce0 2024-04-05T21:29:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.309+0000 7f07b67fc700 1 -- 172.21.15.121:0/756564668 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f07c00611d0 con 0x7f07ac06cce0 2024-04-05T21:29:44.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.311+0000 7f07c787e700 1 -- 172.21.15.121:0/756564668 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f07ac06cce0 msgr2=0x7f07ac06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:44.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.311+0000 7f07c787e700 1 --2- 172.21.15.121:0/756564668 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f07ac06cce0 0x7f07ac06f1a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f07b000b5c0 tx=0x7f07b0005e20 comp rx=0 tx=0).stop 2024-04-05T21:29:44.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.311+0000 7f07c787e700 1 -- 172.21.15.121:0/756564668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07c00ff430 msgr2=0x7f07c019c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:44.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.311+0000 7f07c787e700 1 --2- 172.21.15.121:0/756564668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07c00ff430 0x7f07c019c840 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00c930 tx=0x7f07bc00ccf0 comp rx=0 tx=0).stop 2024-04-05T21:29:44.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.311+0000 7f07c787e700 1 -- 172.21.15.121:0/756564668 shutdown_connections 2024-04-05T21:29:44.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.311+0000 7f07c787e700 1 --2- 172.21.15.121:0/756564668 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f07ac06cce0 0x7f07ac06f1a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:44.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.311+0000 7f07c787e700 1 --2- 172.21.15.121:0/756564668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07c00ff430 0x7f07c019c840 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:44.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.311+0000 7f07c787e700 1 --2- 172.21.15.121:0/756564668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07c0101da0 0x7f07c019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:44.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.311+0000 7f07c787e700 1 -- 172.21.15.121:0/756564668 >> 172.21.15.121:0/756564668 conn(0x7f07c00fb030 msgr2=0x7f07c00fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:44.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.311+0000 7f07c787e700 1 -- 172.21.15.121:0/756564668 shutdown_connections 2024-04-05T21:29:44.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:44.311+0000 7f07c787e700 1 -- 172.21.15.121:0/756564668 wait complete. 2024-04-05T21:29:45.144 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-05T21:29:45.414 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:29:45.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:45 smithi121 ceph-mon[39723]: pgmap v70: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:45.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:45 smithi121 ceph-mon[39723]: from='client.14764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:29:45.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:45 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:29:45.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:45 smithi188 ceph-mon[49715]: pgmap v70: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:45.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:45 smithi188 ceph-mon[49715]: from='client.14764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:29:45.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:45 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:29:47.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.358+0000 7feabc2c0700 1 -- 172.21.15.121:0/2232104150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feab40ff530 msgr2=0x7feab40ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:47.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.358+0000 7feabc2c0700 1 --2- 172.21.15.121:0/2232104150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feab40ff530 0x7feab40ff910 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7feab0009910 tx=0x7feab0009c20 comp rx=0 tx=0).stop 2024-04-05T21:29:47.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.358+0000 7feabc2c0700 1 -- 172.21.15.121:0/2232104150 shutdown_connections 2024-04-05T21:29:47.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.358+0000 7feabc2c0700 1 --2- 172.21.15.121:0/2232104150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feab40ffee0 0x7feab4105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:47.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.358+0000 7feabc2c0700 1 --2- 172.21.15.121:0/2232104150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feab40ff530 0x7feab40ff910 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:47.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.358+0000 7feabc2c0700 1 -- 172.21.15.121:0/2232104150 >> 172.21.15.121:0/2232104150 conn(0x7feab4074df0 msgr2=0x7feab4075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.358+0000 7feabc2c0700 1 -- 172.21.15.121:0/2232104150 shutdown_connections 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.358+0000 7feabc2c0700 1 -- 172.21.15.121:0/2232104150 wait complete. 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feabc2c0700 1 Processor -- start 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feabc2c0700 1 -- start start 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feabc2c0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feab40ff530 0x7feab41004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feabc2c0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feab40ffee0 0x7feab41009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feabc2c0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab4100f30 con 0x7feab40ff530 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feabc2c0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab4101070 con 0x7feab40ffee0 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feaba05c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feab40ff530 0x7feab41004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feaba05c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feab40ff530 0x7feab41004b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38866/0 (socket says 172.21.15.121:38866) 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feaba05c700 1 -- 172.21.15.121:0/2457544910 learned_addr learned my addr 172.21.15.121:0/2457544910 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feab985b700 1 --2- 172.21.15.121:0/2457544910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feab40ffee0 0x7feab41009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feaba05c700 1 -- 172.21.15.121:0/2457544910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feab40ffee0 msgr2=0x7feab41009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:47.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feaba05c700 1 --2- 172.21.15.121:0/2457544910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feab40ffee0 0x7feab41009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:47.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feaba05c700 1 -- 172.21.15.121:0/2457544910 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab00095f0 con 0x7feab40ff530 2024-04-05T21:29:47.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.359+0000 7feaba05c700 1 --2- 172.21.15.121:0/2457544910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feab40ff530 0x7feab41004b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7feab0006010 tx=0x7feab000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:47.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.360+0000 7feaab7fe700 1 -- 172.21.15.121:0/2457544910 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feab001d070 con 0x7feab40ff530 2024-04-05T21:29:47.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.360+0000 7feaab7fe700 1 -- 172.21.15.121:0/2457544910 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feab0005710 con 0x7feab40ff530 2024-04-05T21:29:47.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.360+0000 7feabc2c0700 1 -- 172.21.15.121:0/2457544910 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab41012f0 con 0x7feab40ff530 2024-04-05T21:29:47.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.360+0000 7feaab7fe700 1 -- 172.21.15.121:0/2457544910 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feab000fe20 con 0x7feab40ff530 2024-04-05T21:29:47.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.360+0000 7feabc2c0700 1 -- 172.21.15.121:0/2457544910 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab41a2c50 con 0x7feab40ff530 2024-04-05T21:29:47.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.361+0000 7feabc2c0700 1 -- 172.21.15.121:0/2457544910 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab41091e0 con 0x7feab40ff530 2024-04-05T21:29:47.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.362+0000 7feaab7fe700 1 -- 172.21.15.121:0/2457544910 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7feab0003a60 con 0x7feab40ff530 2024-04-05T21:29:47.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.362+0000 7feaab7fe700 1 --2- 172.21.15.121:0/2457544910 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feaa006cce0 0x7feaa006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:47.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.362+0000 7feaab7fe700 1 -- 172.21.15.121:0/2457544910 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7feab008d640 con 0x7feab40ff530 2024-04-05T21:29:47.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.362+0000 7feab985b700 1 --2- 172.21.15.121:0/2457544910 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feaa006cce0 0x7feaa006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:47.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.362+0000 7feab985b700 1 --2- 172.21.15.121:0/2457544910 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feaa006cce0 0x7feaa006f1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feab4101ca0 tx=0x7feaa400b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:47.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.367+0000 7feaab7fe700 1 -- 172.21.15.121:0/2457544910 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7feab0057760 con 0x7feab40ff530 2024-04-05T21:29:47.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.640+0000 7feabc2c0700 1 -- 172.21.15.121:0/2457544910 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7feab404ea90 con 0x7feab40ff530 2024-04-05T21:29:47.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.641+0000 7feaab7fe700 1 -- 172.21.15.121:0/2457544910 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+679 (secure 0 0 0) 0x7feab0061020 con 0x7feab40ff530 2024-04-05T21:29:47.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.643+0000 7feabc2c0700 1 -- 172.21.15.121:0/2457544910 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feaa006cce0 msgr2=0x7feaa006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:47.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.643+0000 7feabc2c0700 1 --2- 172.21.15.121:0/2457544910 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feaa006cce0 0x7feaa006f1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feab4101ca0 tx=0x7feaa400b410 comp rx=0 tx=0).stop 2024-04-05T21:29:47.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.643+0000 7feabc2c0700 1 -- 172.21.15.121:0/2457544910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feab40ff530 msgr2=0x7feab41004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:47.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.643+0000 7feabc2c0700 1 --2- 172.21.15.121:0/2457544910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feab40ff530 0x7feab41004b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7feab0006010 tx=0x7feab000bd70 comp rx=0 tx=0).stop 2024-04-05T21:29:47.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.643+0000 7feabc2c0700 1 -- 172.21.15.121:0/2457544910 shutdown_connections 2024-04-05T21:29:47.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.643+0000 7feabc2c0700 1 --2- 172.21.15.121:0/2457544910 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feaa006cce0 0x7feaa006f1a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:47.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.643+0000 7feabc2c0700 1 --2- 172.21.15.121:0/2457544910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feab40ff530 0x7feab41004b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:47.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.643+0000 7feabc2c0700 1 --2- 172.21.15.121:0/2457544910 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feab40ffee0 0x7feab41009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:47.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.643+0000 7feabc2c0700 1 -- 172.21.15.121:0/2457544910 >> 172.21.15.121:0/2457544910 conn(0x7feab4074df0 msgr2=0x7feab40fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:47.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.643+0000 7feabc2c0700 1 -- 172.21.15.121:0/2457544910 shutdown_connections 2024-04-05T21:29:47.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:47.643+0000 7feabc2c0700 1 -- 172.21.15.121:0/2457544910 wait complete. 2024-04-05T21:29:47.655 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:29:47.669 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:47 smithi121 ceph-mon[39723]: pgmap v71: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:47.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:47 smithi188 ceph-mon[49715]: pgmap v71: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:48.450 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-04-05T21:29:48.680 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:29:48.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:48 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2457544910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:48.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:48 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2457544910' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:49.674 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:49 smithi121 ceph-mon[39723]: pgmap v72: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:49.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:49 smithi188 ceph-mon[49715]: pgmap v72: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:50.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.545+0000 7feb6098e700 1 -- 172.21.15.121:0/3501388491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb5c105a30 msgr2=0x7feb5c105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:50.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.545+0000 7feb6098e700 1 --2- 172.21.15.121:0/3501388491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb5c105a30 0x7feb5c105e10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7feb4c009910 tx=0x7feb4c009c20 comp rx=0 tx=0).stop 2024-04-05T21:29:50.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.545+0000 7feb6098e700 1 -- 172.21.15.121:0/3501388491 shutdown_connections 2024-04-05T21:29:50.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.545+0000 7feb6098e700 1 --2- 172.21.15.121:0/3501388491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb5c0ffb70 0x7feb5c0fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:50.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.545+0000 7feb6098e700 1 --2- 172.21.15.121:0/3501388491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb5c105a30 0x7feb5c105e10 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:50.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.545+0000 7feb6098e700 1 -- 172.21.15.121:0/3501388491 >> 172.21.15.121:0/3501388491 conn(0x7feb5c0754f0 msgr2=0x7feb5c075900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:50.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.545+0000 7feb6098e700 1 -- 172.21.15.121:0/3501388491 shutdown_connections 2024-04-05T21:29:50.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.545+0000 7feb6098e700 1 -- 172.21.15.121:0/3501388491 wait complete. 2024-04-05T21:29:50.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.545+0000 7feb6098e700 1 Processor -- start 2024-04-05T21:29:50.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb6098e700 1 -- start start 2024-04-05T21:29:50.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb6098e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb5c0ffb70 0x7feb5c19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:50.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb6098e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb5c105a30 0x7feb5c19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:50.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb6098e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c19f1a0 con 0x7feb5c0ffb70 2024-04-05T21:29:50.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb6098e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c198650 con 0x7feb5c105a30 2024-04-05T21:29:50.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb5a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb5c0ffb70 0x7feb5c19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:50.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb5a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb5c0ffb70 0x7feb5c19e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38888/0 (socket says 172.21.15.121:38888) 2024-04-05T21:29:50.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb5a59c700 1 -- 172.21.15.121:0/3923868438 learned_addr learned my addr 172.21.15.121:0/3923868438 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:29:50.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb59d9b700 1 --2- 172.21.15.121:0/3923868438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb5c105a30 0x7feb5c19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:50.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb5a59c700 1 -- 172.21.15.121:0/3923868438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb5c105a30 msgr2=0x7feb5c19eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb5a59c700 1 --2- 172.21.15.121:0/3923868438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb5c105a30 0x7feb5c19eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb5a59c700 1 -- 172.21.15.121:0/3923868438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c0095f0 con 0x7feb5c0ffb70 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb5a59c700 1 --2- 172.21.15.121:0/3923868438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb5c0ffb70 0x7feb5c19e5d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7feb4c006010 tx=0x7feb4c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb4b7fe700 1 -- 172.21.15.121:0/3923868438 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb4c01d070 con 0x7feb5c0ffb70 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb4b7fe700 1 -- 172.21.15.121:0/3923868438 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb4c00f460 con 0x7feb5c0ffb70 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb6098e700 1 -- 172.21.15.121:0/3923868438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5c1988d0 con 0x7feb5c0ffb70 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb4b7fe700 1 -- 172.21.15.121:0/3923868438 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb4c021620 con 0x7feb5c0ffb70 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.546+0000 7feb6098e700 1 -- 172.21.15.121:0/3923868438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5c198dc0 con 0x7feb5c0ffb70 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.547+0000 7feb6098e700 1 -- 172.21.15.121:0/3923868438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5c04ea90 con 0x7feb5c0ffb70 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.548+0000 7feb4b7fe700 1 -- 172.21.15.121:0/3923868438 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7feb4c00f5d0 con 0x7feb5c0ffb70 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.549+0000 7feb4b7fe700 1 --2- 172.21.15.121:0/3923868438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feb4406cce0 0x7feb4406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.549+0000 7feb4b7fe700 1 -- 172.21.15.121:0/3923868438 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7feb4c08db90 con 0x7feb5c0ffb70 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.549+0000 7feb59d9b700 1 --2- 172.21.15.121:0/3923868438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feb4406cce0 0x7feb4406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:50.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.549+0000 7feb59d9b700 1 --2- 172.21.15.121:0/3923868438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feb4406cce0 0x7feb4406f1a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7feb5c0ff9a0 tx=0x7feb5000a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:50.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.553+0000 7feb4b7fe700 1 -- 172.21.15.121:0/3923868438 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7feb4c05b740 con 0x7feb5c0ffb70 2024-04-05T21:29:50.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.821+0000 7feb6098e700 1 -- 172.21.15.121:0/3923868438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7feb5c199b60 con 0x7feb5c0ffb70 2024-04-05T21:29:50.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.822+0000 7feb4b7fe700 1 -- 172.21.15.121:0/3923868438 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+679 (secure 0 0 0) 0x7feb4c05b2d0 con 0x7feb5c0ffb70 2024-04-05T21:29:50.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.824+0000 7feb6098e700 1 -- 172.21.15.121:0/3923868438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feb4406cce0 msgr2=0x7feb4406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:50.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.824+0000 7feb6098e700 1 --2- 172.21.15.121:0/3923868438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feb4406cce0 0x7feb4406f1a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7feb5c0ff9a0 tx=0x7feb5000a300 comp rx=0 tx=0).stop 2024-04-05T21:29:50.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.824+0000 7feb6098e700 1 -- 172.21.15.121:0/3923868438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb5c0ffb70 msgr2=0x7feb5c19e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:50.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.824+0000 7feb6098e700 1 --2- 172.21.15.121:0/3923868438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb5c0ffb70 0x7feb5c19e5d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7feb4c006010 tx=0x7feb4c00bd70 comp rx=0 tx=0).stop 2024-04-05T21:29:50.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.824+0000 7feb6098e700 1 -- 172.21.15.121:0/3923868438 shutdown_connections 2024-04-05T21:29:50.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.824+0000 7feb6098e700 1 --2- 172.21.15.121:0/3923868438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feb4406cce0 0x7feb4406f1a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:50.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.824+0000 7feb6098e700 1 --2- 172.21.15.121:0/3923868438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb5c0ffb70 0x7feb5c19e5d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:50.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.824+0000 7feb6098e700 1 --2- 172.21.15.121:0/3923868438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feb5c105a30 0x7feb5c19eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:50.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.824+0000 7feb6098e700 1 -- 172.21.15.121:0/3923868438 >> 172.21.15.121:0/3923868438 conn(0x7feb5c0754f0 msgr2=0x7feb5c068590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:50.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.824+0000 7feb6098e700 1 -- 172.21.15.121:0/3923868438 shutdown_connections 2024-04-05T21:29:50.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:50.824+0000 7feb6098e700 1 -- 172.21.15.121:0/3923868438 wait complete. 2024-04-05T21:29:50.836 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)" 2024-04-05T21:29:51.545 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:51 smithi121 ceph-mon[39723]: pgmap v73: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:51.546 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:51 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/3923868438' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:51.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-04-05T21:29:51.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:51 smithi188 ceph-mon[49715]: pgmap v73: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:51.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:51 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/3923868438' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:51.798 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:29:53.476 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:53 smithi121 ceph-mon[39723]: pgmap v74: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:53.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.755+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2774630174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc100de0 msgr2=0x7f28bc1011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:53.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.755+0000 7f28c2ac8700 1 --2- 172.21.15.121:0/2774630174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc100de0 0x7f28bc1011c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f28b0009910 tx=0x7f28b0009c20 comp rx=0 tx=0).stop 2024-04-05T21:29:53.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.755+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2774630174 shutdown_connections 2024-04-05T21:29:53.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.755+0000 7f28c2ac8700 1 --2- 172.21.15.121:0/2774630174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28bc101790 0x7f28bc1057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:53.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.755+0000 7f28c2ac8700 1 --2- 172.21.15.121:0/2774630174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc100de0 0x7f28bc1011c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:53.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.755+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2774630174 >> 172.21.15.121:0/2774630174 conn(0x7f28bc0fc670 msgr2=0x7f28bc0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:53.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.755+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2774630174 shutdown_connections 2024-04-05T21:29:53.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.755+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2774630174 wait complete. 2024-04-05T21:29:53.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28c2ac8700 1 Processor -- start 2024-04-05T21:29:53.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28c2ac8700 1 -- start start 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28c2ac8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28bc100de0 0x7f28bc19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28c2ac8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc101790 0x7f28bc19eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28c2ac8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc19f1e0 con 0x7f28bc101790 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28c2ac8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc198690 con 0x7f28bc100de0 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28bbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc101790 0x7f28bc19eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28bbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc101790 0x7f28bc19eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38908/0 (socket says 172.21.15.121:38908) 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28bbfff700 1 -- 172.21.15.121:0/2760960816 learned_addr learned my addr 172.21.15.121:0/2760960816 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28c0864700 1 --2- 172.21.15.121:0/2760960816 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28bc100de0 0x7f28bc19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28bbfff700 1 -- 172.21.15.121:0/2760960816 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28bc100de0 msgr2=0x7f28bc19e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28bbfff700 1 --2- 172.21.15.121:0/2760960816 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28bc100de0 0x7f28bc19e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28bbfff700 1 -- 172.21.15.121:0/2760960816 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b00095f0 con 0x7f28bc101790 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.756+0000 7f28bbfff700 1 --2- 172.21.15.121:0/2760960816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc101790 0x7f28bc19eb50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f28a800c8f0 tx=0x7f28a800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.757+0000 7f28b9ffb700 1 -- 172.21.15.121:0/2760960816 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28a8007a50 con 0x7f28bc101790 2024-04-05T21:29:53.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.757+0000 7f28b9ffb700 1 -- 172.21.15.121:0/2760960816 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28a8007bb0 con 0x7f28bc101790 2024-04-05T21:29:53.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.757+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2760960816 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28bc198970 con 0x7f28bc101790 2024-04-05T21:29:53.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.757+0000 7f28b9ffb700 1 -- 172.21.15.121:0/2760960816 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28a8018630 con 0x7f28bc101790 2024-04-05T21:29:53.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.757+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2760960816 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28bc198ec0 con 0x7f28bc101790 2024-04-05T21:29:53.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.758+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2760960816 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28bc104f60 con 0x7f28bc101790 2024-04-05T21:29:53.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.758+0000 7f28b9ffb700 1 -- 172.21.15.121:0/2760960816 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f28a8007d20 con 0x7f28bc101790 2024-04-05T21:29:53.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.758+0000 7f28b9ffb700 1 --2- 172.21.15.121:0/2760960816 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28ac06cce0 0x7f28ac06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:53.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.759+0000 7f28b9ffb700 1 -- 172.21.15.121:0/2760960816 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f28a808c3f0 con 0x7f28bc101790 2024-04-05T21:29:53.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.759+0000 7f28c0864700 1 --2- 172.21.15.121:0/2760960816 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28ac06cce0 0x7f28ac06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:53.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.759+0000 7f28c0864700 1 --2- 172.21.15.121:0/2760960816 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28ac06cce0 0x7f28ac06f1a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f28b0006010 tx=0x7f28b001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:53.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:53.763+0000 7f28b9ffb700 1 -- 172.21.15.121:0/2760960816 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f28a805a020 con 0x7f28bc101790 2024-04-05T21:29:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:53 smithi188 ceph-mon[49715]: pgmap v74: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:54.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:54.034+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2760960816 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f28bc04ea90 con 0x7f28bc101790 2024-04-05T21:29:54.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:54.034+0000 7f28b9ffb700 1 -- 172.21.15.121:0/2760960816 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+679 (secure 0 0 0) 0x7f28a8054e60 con 0x7f28bc101790 2024-04-05T21:29:54.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:54.036+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2760960816 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28ac06cce0 msgr2=0x7f28ac06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:54.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:54.036+0000 7f28c2ac8700 1 --2- 172.21.15.121:0/2760960816 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28ac06cce0 0x7f28ac06f1a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f28b0006010 tx=0x7f28b001a040 comp rx=0 tx=0).stop 2024-04-05T21:29:54.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:54.036+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2760960816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc101790 msgr2=0x7f28bc19eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:54.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:54.036+0000 7f28c2ac8700 1 --2- 172.21.15.121:0/2760960816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc101790 0x7f28bc19eb50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f28a800c8f0 tx=0x7f28a800ccb0 comp rx=0 tx=0).stop 2024-04-05T21:29:54.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:54.036+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2760960816 shutdown_connections 2024-04-05T21:29:54.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:54.036+0000 7f28c2ac8700 1 --2- 172.21.15.121:0/2760960816 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28ac06cce0 0x7f28ac06f1a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:54.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:54.036+0000 7f28c2ac8700 1 --2- 172.21.15.121:0/2760960816 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28bc100de0 0x7f28bc19e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:54.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:54.036+0000 7f28c2ac8700 1 --2- 172.21.15.121:0/2760960816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28bc101790 0x7f28bc19eb50 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:54.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:54.036+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2760960816 >> 172.21.15.121:0/2760960816 conn(0x7f28bc0fc670 msgr2=0x7f28bc0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:54.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:54.037+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2760960816 shutdown_connections 2024-04-05T21:29:54.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:54.037+0000 7f28c2ac8700 1 -- 172.21.15.121:0/2760960816 wait complete. 2024-04-05T21:29:54.048 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:29:54.735 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:54 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/2760960816' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:54.755 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-04-05T21:29:54.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:54 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/2760960816' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:29:54.983 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:29:55.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:55 smithi121 ceph-mon[39723]: pgmap v75: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:55 smithi188 ceph-mon[49715]: pgmap v75: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:57.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.002+0000 7f8309bd3700 1 -- 172.21.15.121:0/825111218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83040ffbd0 msgr2=0x7f8304100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:57.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.002+0000 7f8309bd3700 1 --2- 172.21.15.121:0/825111218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83040ffbd0 0x7f8304100050 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f82f8009910 tx=0x7f82f8009c20 comp rx=0 tx=0).stop 2024-04-05T21:29:57.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.002+0000 7f8309bd3700 1 -- 172.21.15.121:0/825111218 shutdown_connections 2024-04-05T21:29:57.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.002+0000 7f8309bd3700 1 --2- 172.21.15.121:0/825111218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83040ffbd0 0x7f8304100050 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:57.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.002+0000 7f8309bd3700 1 --2- 172.21.15.121:0/825111218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8304105a40 0x7f8304105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:57.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.002+0000 7f8309bd3700 1 -- 172.21.15.121:0/825111218 >> 172.21.15.121:0/825111218 conn(0x7f83040755a0 msgr2=0x7f83040759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:57.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.003+0000 7f8309bd3700 1 -- 172.21.15.121:0/825111218 shutdown_connections 2024-04-05T21:29:57.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.003+0000 7f8309bd3700 1 -- 172.21.15.121:0/825111218 wait complete. 2024-04-05T21:29:57.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.003+0000 7f8309bd3700 1 Processor -- start 2024-04-05T21:29:57.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.003+0000 7f8309bd3700 1 -- start start 2024-04-05T21:29:57.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.003+0000 7f8309bd3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83040ffbd0 0x7f8304197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:57.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.003+0000 7f8309bd3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8304105a40 0x7f83041979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:57.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8309bd3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8304197ff0 con 0x7f8304105a40 2024-04-05T21:29:57.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8309bd3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8304198160 con 0x7f83040ffbd0 2024-04-05T21:29:57.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8302ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8304105a40 0x7f83041979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:57.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8302ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8304105a40 0x7f83041979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41312/0 (socket says 172.21.15.121:41312) 2024-04-05T21:29:57.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8302ffd700 1 -- 172.21.15.121:0/4081844825 learned_addr learned my addr 172.21.15.121:0/4081844825 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:29:57.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f83037fe700 1 --2- 172.21.15.121:0/4081844825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83040ffbd0 0x7f8304197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:57.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8302ffd700 1 -- 172.21.15.121:0/4081844825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83040ffbd0 msgr2=0x7f8304197490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:57.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8302ffd700 1 --2- 172.21.15.121:0/4081844825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83040ffbd0 0x7f8304197490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:57.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8302ffd700 1 -- 172.21.15.121:0/4081844825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f80095f0 con 0x7f8304105a40 2024-04-05T21:29:57.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8302ffd700 1 --2- 172.21.15.121:0/4081844825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8304105a40 0x7f83041979d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f82f8006010 tx=0x7f82f800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:57.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8300ff9700 1 -- 172.21.15.121:0/4081844825 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82f801d070 con 0x7f8304105a40 2024-04-05T21:29:57.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8300ff9700 1 -- 172.21.15.121:0/4081844825 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82f800f460 con 0x7f8304105a40 2024-04-05T21:29:57.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8309bd3700 1 -- 172.21.15.121:0/4081844825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83041a0500 con 0x7f8304105a40 2024-04-05T21:29:57.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8300ff9700 1 -- 172.21.15.121:0/4081844825 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82f8021620 con 0x7f8304105a40 2024-04-05T21:29:57.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.004+0000 7f8309bd3700 1 -- 172.21.15.121:0/4081844825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83041a0970 con 0x7f8304105a40 2024-04-05T21:29:57.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.005+0000 7f8309bd3700 1 -- 172.21.15.121:0/4081844825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8304107fd0 con 0x7f8304105a40 2024-04-05T21:29:57.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.006+0000 7f8300ff9700 1 -- 172.21.15.121:0/4081844825 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f82f800fa40 con 0x7f8304105a40 2024-04-05T21:29:57.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.006+0000 7f8300ff9700 1 --2- 172.21.15.121:0/4081844825 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:29:57.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.006+0000 7f8300ff9700 1 -- 172.21.15.121:0/4081844825 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f82f808d6d0 con 0x7f8304105a40 2024-04-05T21:29:57.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.006+0000 7f83037fe700 1 --2- 172.21.15.121:0/4081844825 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:29:57.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.007+0000 7f83037fe700 1 --2- 172.21.15.121:0/4081844825 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f82f4005950 tx=0x7f82f40058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:29:57.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.011+0000 7f8300ff9700 1 -- 172.21.15.121:0/4081844825 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f82f80577f0 con 0x7f8304105a40 2024-04-05T21:29:57.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:57.220+0000 7f8309bd3700 1 -- 172.21.15.121:0/4081844825 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e", "target": ["mon-mgr", ""]}) v1 -- 0x7f830402cdd0 con 0x7f82ec06cce0 2024-04-05T21:29:57.707 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:57 smithi121 ceph-mon[39723]: pgmap v76: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:57 smithi188 ceph-mon[49715]: pgmap v76: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:29:58.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:58 smithi121 ceph-mon[39723]: from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:29:58.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:58 smithi188 ceph-mon[49715]: from='client.14780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:29:58.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:58.889+0000 7f8300ff9700 1 -- 172.21.15.121:0/4081844825 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5393 (secure 0 0 0) 0x7f830402cdd0 con 0x7f82ec06cce0 2024-04-05T21:29:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:58.892+0000 7f8309bd3700 1 -- 172.21.15.121:0/4081844825 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 msgr2=0x7f82ec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:58.892+0000 7f8309bd3700 1 --2- 172.21.15.121:0/4081844825 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f82f4005950 tx=0x7f82f40058e0 comp rx=0 tx=0).stop 2024-04-05T21:29:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:58.892+0000 7f8309bd3700 1 -- 172.21.15.121:0/4081844825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8304105a40 msgr2=0x7f83041979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:29:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:58.892+0000 7f8309bd3700 1 --2- 172.21.15.121:0/4081844825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8304105a40 0x7f83041979d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f82f8006010 tx=0x7f82f800be20 comp rx=0 tx=0).stop 2024-04-05T21:29:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:58.892+0000 7f8309bd3700 1 -- 172.21.15.121:0/4081844825 shutdown_connections 2024-04-05T21:29:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:58.892+0000 7f8309bd3700 1 --2- 172.21.15.121:0/4081844825 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:58.892+0000 7f8309bd3700 1 --2- 172.21.15.121:0/4081844825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83040ffbd0 0x7f8304197490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:58.892+0000 7f8309bd3700 1 --2- 172.21.15.121:0/4081844825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8304105a40 0x7f83041979d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:29:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:58.892+0000 7f8309bd3700 1 -- 172.21.15.121:0/4081844825 >> 172.21.15.121:0/4081844825 conn(0x7f83040755a0 msgr2=0x7f83040fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:29:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:58.892+0000 7f8309bd3700 1 -- 172.21.15.121:0/4081844825 shutdown_connections 2024-04-05T21:29:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:29:58.892+0000 7f8309bd3700 1 -- 172.21.15.121:0/4081844825 wait complete. 2024-04-05T21:29:58.904 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:29:59.674 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:29:59 smithi121 ceph-mon[39723]: pgmap v77: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:59.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph orch ps' 2024-04-05T21:29:59.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:29:59 smithi188 ceph-mon[49715]: pgmap v77: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:29:59.931 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:30:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:00 smithi121 ceph-mon[39723]: overall HEALTH_OK 2024-04-05T21:30:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:00 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:30:00.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:00 smithi188 ceph-mon[49715]: overall HEALTH_OK 2024-04-05T21:30:00.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:00 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:30:01.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:01 smithi121 ceph-mon[39723]: pgmap v78: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:01.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:01 smithi188 ceph-mon[49715]: pgmap v78: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.805+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/2514645241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c880ffbd0 msgr2=0x7f7c88100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.805+0000 7f7c8e7ee700 1 --2- 172.21.15.121:0/2514645241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c880ffbd0 0x7f7c88100050 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c009910 tx=0x7f7c7c009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.806+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/2514645241 shutdown_connections 2024-04-05T21:30:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.806+0000 7f7c8e7ee700 1 --2- 172.21.15.121:0/2514645241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c880ffbd0 0x7f7c88100050 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.806+0000 7f7c8e7ee700 1 --2- 172.21.15.121:0/2514645241 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c88105a40 0x7f7c88105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.806+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/2514645241 >> 172.21.15.121:0/2514645241 conn(0x7f7c880755a0 msgr2=0x7f7c880759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.806+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/2514645241 shutdown_connections 2024-04-05T21:30:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.806+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/2514645241 wait complete. 2024-04-05T21:30:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.806+0000 7f7c8e7ee700 1 Processor -- start 2024-04-05T21:30:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.806+0000 7f7c8e7ee700 1 -- start start 2024-04-05T21:30:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c8e7ee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c880ffbd0 0x7f7c8819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c8e7ee700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c88105a40 0x7f7c8819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c8e7ee700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8819d0e0 con 0x7f7c880ffbd0 2024-04-05T21:30:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c8e7ee700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c88196590 con 0x7f7c88105a40 2024-04-05T21:30:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c87fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c880ffbd0 0x7f7c8819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c87fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c880ffbd0 0x7f7c8819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41344/0 (socket says 172.21.15.121:41344) 2024-04-05T21:30:01.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c87fff700 1 -- 172.21.15.121:0/4243339124 learned_addr learned my addr 172.21.15.121:0/4243339124 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c877fe700 1 --2- 172.21.15.121:0/4243339124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c88105a40 0x7f7c8819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c87fff700 1 -- 172.21.15.121:0/4243339124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c88105a40 msgr2=0x7f7c8819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c87fff700 1 --2- 172.21.15.121:0/4243339124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c88105a40 0x7f7c8819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c87fff700 1 -- 172.21.15.121:0/4243339124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c0095f0 con 0x7f7c880ffbd0 2024-04-05T21:30:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c87fff700 1 --2- 172.21.15.121:0/4243339124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c880ffbd0 0x7f7c8819c510 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800d8d0 tx=0x7f7c7800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:01.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c857fa700 1 -- 172.21.15.121:0/4243339124 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c78009940 con 0x7f7c880ffbd0 2024-04-05T21:30:01.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c857fa700 1 -- 172.21.15.121:0/4243339124 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c78010460 con 0x7f7c880ffbd0 2024-04-05T21:30:01.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/4243339124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c88196870 con 0x7f7c880ffbd0 2024-04-05T21:30:01.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c857fa700 1 -- 172.21.15.121:0/4243339124 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c7800f5f0 con 0x7f7c880ffbd0 2024-04-05T21:30:01.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.807+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/4243339124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c88196dc0 con 0x7f7c880ffbd0 2024-04-05T21:30:01.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.808+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/4243339124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c88108fe0 con 0x7f7c880ffbd0 2024-04-05T21:30:01.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.809+0000 7f7c857fa700 1 -- 172.21.15.121:0/4243339124 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7c780105d0 con 0x7f7c880ffbd0 2024-04-05T21:30:01.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.809+0000 7f7c857fa700 1 --2- 172.21.15.121:0/4243339124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7c7006cce0 0x7f7c7006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:01.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.809+0000 7f7c857fa700 1 -- 172.21.15.121:0/4243339124 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f7c7808c520 con 0x7f7c880ffbd0 2024-04-05T21:30:01.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.809+0000 7f7c877fe700 1 --2- 172.21.15.121:0/4243339124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7c7006cce0 0x7f7c7006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:01.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.809+0000 7f7c877fe700 1 --2- 172.21.15.121:0/4243339124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7c7006cce0 0x7f7c7006f1a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7c88104ef0 tx=0x7f7c7c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:01.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:01.814+0000 7f7c857fa700 1 -- 172.21.15.121:0/4243339124 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f7c7805a0d0 con 0x7f7c880ffbd0 2024-04-05T21:30:02.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:02.013+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/4243339124 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c88196fe0 con 0x7f7c7006cce0 2024-04-05T21:30:02.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:02.023+0000 7f7c857fa700 1 -- 172.21.15.121:0/4243339124 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f7c88196fe0 con 0x7f7c7006cce0 2024-04-05T21:30:02.024 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:30:02.024 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (52s) 28s ago 12m 16.6M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:30:02.024 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (12m) 28s ago 12m 8024k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:30:02.024 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (11m) 92s ago 11m 7834k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:30:02.024 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (12m) 28s ago 12m 6651k - 18.2.1 5be31c24972a 5bf35fe2ace4 2024-04-05T21:30:02.024 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (11m) 92s ago 11m 6651k - 18.2.1 5be31c24972a dcdfa6ae17bf 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (35s) 28s ago 12m 47.8M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (6m) 28s ago 6m 64.6M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (6m) 28s ago 6m 135M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (6m) 92s ago 6m 110M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (6m) 92s ago 6m 62.4M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (2m) 28s ago 13m 498M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (2m) 92s ago 11m 448M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (13m) 28s ago 14m 58.6M 2048M 18.2.1 5be31c24972a 3c30ce99c006 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (11m) 92s ago 11m 45.2M 2048M 18.2.1 5be31c24972a 8ecc9636670e 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (101s) 28s ago 12m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (97s) 92s ago 11m 3653k - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (10m) 28s ago 10m 215M 1227M 18.2.1 5be31c24972a f6f3a45429a2 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (9m) 28s ago 9m 241M 1227M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:30:02.025 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (9m) 28s ago 9m 182M 1227M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:30:02.026 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (8m) 92s ago 8m 286M 2251M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:30:02.026 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (8m) 92s ago 8m 244M 2251M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:30:02.026 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (7m) 92s ago 7m 223M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:30:02.026 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (71s) 28s ago 11m 48.7M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:30:02.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:02.025+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/4243339124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7c7006cce0 msgr2=0x7f7c7006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:02.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:02.025+0000 7f7c8e7ee700 1 --2- 172.21.15.121:0/4243339124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7c7006cce0 0x7f7c7006f1a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7c88104ef0 tx=0x7f7c7c01a040 comp rx=0 tx=0).stop 2024-04-05T21:30:02.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:02.025+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/4243339124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c880ffbd0 msgr2=0x7f7c8819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:02.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:02.025+0000 7f7c8e7ee700 1 --2- 172.21.15.121:0/4243339124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c880ffbd0 0x7f7c8819c510 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7c7800d8d0 tx=0x7f7c7800dc90 comp rx=0 tx=0).stop 2024-04-05T21:30:02.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:02.025+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/4243339124 shutdown_connections 2024-04-05T21:30:02.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:02.025+0000 7f7c8e7ee700 1 --2- 172.21.15.121:0/4243339124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7c7006cce0 0x7f7c7006f1a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:02.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:02.025+0000 7f7c8e7ee700 1 --2- 172.21.15.121:0/4243339124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c880ffbd0 0x7f7c8819c510 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:02.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:02.025+0000 7f7c8e7ee700 1 --2- 172.21.15.121:0/4243339124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c88105a40 0x7f7c8819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:02.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:02.025+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/4243339124 >> 172.21.15.121:0/4243339124 conn(0x7f7c880755a0 msgr2=0x7f7c88103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:02.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:02.025+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/4243339124 shutdown_connections 2024-04-05T21:30:02.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:02.025+0000 7f7c8e7ee700 1 -- 172.21.15.121:0/4243339124 wait complete. 2024-04-05T21:30:02.776 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-05T21:30:02.777 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T21:30:02.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-04-05T21:30:03.009 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:30:03.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:03 smithi121 ceph-mon[39723]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:03.479 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:03 smithi121 ceph-mon[39723]: pgmap v79: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:03 smithi188 ceph-mon[49715]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:03 smithi188 ceph-mon[49715]: pgmap v79: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.025+0000 7f49b219e700 1 -- 172.21.15.121:0/348842733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49ac068410 msgr2=0x7f49ac0687f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.025+0000 7f49b219e700 1 --2- 172.21.15.121:0/348842733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49ac068410 0x7f49ac0687f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009910 tx=0x7f49a0009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.025+0000 7f49b219e700 1 -- 172.21.15.121:0/348842733 shutdown_connections 2024-04-05T21:30:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.025+0000 7f49b219e700 1 --2- 172.21.15.121:0/348842733 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49ac068d30 0x7f49ac10d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.025+0000 7f49b219e700 1 --2- 172.21.15.121:0/348842733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49ac068410 0x7f49ac0687f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.025+0000 7f49b219e700 1 -- 172.21.15.121:0/348842733 >> 172.21.15.121:0/348842733 conn(0x7f49ac0755a0 msgr2=0x7f49ac0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.025+0000 7f49b219e700 1 -- 172.21.15.121:0/348842733 shutdown_connections 2024-04-05T21:30:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.025+0000 7f49b219e700 1 -- 172.21.15.121:0/348842733 wait complete. 2024-04-05T21:30:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.025+0000 7f49b219e700 1 Processor -- start 2024-04-05T21:30:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.025+0000 7f49b219e700 1 -- start start 2024-04-05T21:30:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49b219e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49ac068410 0x7f49ac19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:05.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49b219e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49ac068d30 0x7f49ac19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:05.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49b219e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac19f2c0 con 0x7f49ac068d30 2024-04-05T21:30:05.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49b219e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac198770 con 0x7f49ac068410 2024-04-05T21:30:05.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49aaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49ac068d30 0x7f49ac19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:05.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49ab7fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49ac068410 0x7f49ac19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:05.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49aaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49ac068d30 0x7f49ac19ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45568/0 (socket says 172.21.15.121:45568) 2024-04-05T21:30:05.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49aaffd700 1 -- 172.21.15.121:0/1806016057 learned_addr learned my addr 172.21.15.121:0/1806016057 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:05.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49aaffd700 1 -- 172.21.15.121:0/1806016057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49ac068410 msgr2=0x7f49ac19e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:05.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49aaffd700 1 --2- 172.21.15.121:0/1806016057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49ac068410 0x7f49ac19e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:05.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49aaffd700 1 -- 172.21.15.121:0/1806016057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a00095f0 con 0x7f49ac068d30 2024-04-05T21:30:05.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49aaffd700 1 --2- 172.21.15.121:0/1806016057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49ac068d30 0x7f49ac19ec30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f499c00b4b0 tx=0x7f499c00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:05.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49a8ff9700 1 -- 172.21.15.121:0/1806016057 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f499c011800 con 0x7f49ac068d30 2024-04-05T21:30:05.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49a8ff9700 1 -- 172.21.15.121:0/1806016057 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f499c011e00 con 0x7f49ac068d30 2024-04-05T21:30:05.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49b219e700 1 -- 172.21.15.121:0/1806016057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac198a50 con 0x7f49ac068d30 2024-04-05T21:30:05.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.026+0000 7f49a8ff9700 1 -- 172.21.15.121:0/1806016057 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f499c010e60 con 0x7f49ac068d30 2024-04-05T21:30:05.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.027+0000 7f49b219e700 1 -- 172.21.15.121:0/1806016057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac198fa0 con 0x7f49ac068d30 2024-04-05T21:30:05.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.028+0000 7f49b219e700 1 -- 172.21.15.121:0/1806016057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac10a9b0 con 0x7f49ac068d30 2024-04-05T21:30:05.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.028+0000 7f49a8ff9700 1 -- 172.21.15.121:0/1806016057 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f499c01a460 con 0x7f49ac068d30 2024-04-05T21:30:05.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.028+0000 7f49a8ff9700 1 --2- 172.21.15.121:0/1806016057 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f499406cce0 0x7f499406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:05.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.028+0000 7f49ab7fe700 1 --2- 172.21.15.121:0/1806016057 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f499406cce0 0x7f499406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:05.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.028+0000 7f49a8ff9700 1 -- 172.21.15.121:0/1806016057 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f499c08bfc0 con 0x7f49ac068d30 2024-04-05T21:30:05.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.029+0000 7f49ab7fe700 1 --2- 172.21.15.121:0/1806016057 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f499406cce0 0x7f499406f1a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f49a0006010 tx=0x7f49a001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:05.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.033+0000 7f49a8ff9700 1 -- 172.21.15.121:0/1806016057 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f499c059c60 con 0x7f49ac068d30 2024-04-05T21:30:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.227+0000 7f49b219e700 1 -- 172.21.15.121:0/1806016057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f49ac04ea90 con 0x7f49ac068d30 2024-04-05T21:30:05.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.231+0000 7f49a8ff9700 1 -- 172.21.15.121:0/1806016057 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v45)=0 v45) v1 ==== 125+0+0 (secure 0 0 0) 0x7f499c0597f0 con 0x7f49ac068d30 2024-04-05T21:30:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.233+0000 7f49b219e700 1 -- 172.21.15.121:0/1806016057 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f499406cce0 msgr2=0x7f499406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.233+0000 7f49b219e700 1 --2- 172.21.15.121:0/1806016057 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f499406cce0 0x7f499406f1a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f49a0006010 tx=0x7f49a001a040 comp rx=0 tx=0).stop 2024-04-05T21:30:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.233+0000 7f49b219e700 1 -- 172.21.15.121:0/1806016057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49ac068d30 msgr2=0x7f49ac19ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.233+0000 7f49b219e700 1 --2- 172.21.15.121:0/1806016057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49ac068d30 0x7f49ac19ec30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f499c00b4b0 tx=0x7f499c00b7c0 comp rx=0 tx=0).stop 2024-04-05T21:30:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.233+0000 7f49b219e700 1 -- 172.21.15.121:0/1806016057 shutdown_connections 2024-04-05T21:30:05.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.233+0000 7f49b219e700 1 --2- 172.21.15.121:0/1806016057 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f499406cce0 0x7f499406f1a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:05.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.233+0000 7f49b219e700 1 --2- 172.21.15.121:0/1806016057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49ac068410 0x7f49ac19e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:05.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.233+0000 7f49b219e700 1 --2- 172.21.15.121:0/1806016057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49ac068d30 0x7f49ac19ec30 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:05.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.233+0000 7f49b219e700 1 -- 172.21.15.121:0/1806016057 >> 172.21.15.121:0/1806016057 conn(0x7f49ac0755a0 msgr2=0x7f49ac0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:05.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.233+0000 7f49b219e700 1 -- 172.21.15.121:0/1806016057 shutdown_connections 2024-04-05T21:30:05.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:05.233+0000 7f49b219e700 1 -- 172.21.15.121:0/1806016057 wait complete. 2024-04-05T21:30:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:05 smithi121 ceph-mon[39723]: pgmap v80: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:05 smithi121 ceph-mon[39723]: from='client.? 172.21.15.121:0/1806016057' entity='client.admin' 2024-04-05T21:30:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:05 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:05 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:05 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:30:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:05 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:05.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:05 smithi188 ceph-mon[49715]: pgmap v80: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:05.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:05 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1806016057' entity='client.admin' 2024-04-05T21:30:05.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:05 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:05.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:05 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:05.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:05 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:30:05.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:05 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:06.031 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-05T21:30:06.031 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T21:30:06.032 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-05T21:30:06.262 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:30:07.567 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:07 smithi121 ceph-mon[39723]: pgmap v81: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:07.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:07 smithi188 ceph-mon[49715]: pgmap v81: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:08.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.196+0000 7fa0923aa700 1 -- 172.21.15.121:0/100161961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa08c101770 msgr2=0x7fa08c1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:08.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.196+0000 7fa0923aa700 1 --2- 172.21.15.121:0/100161961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa08c101770 0x7fa08c1057c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009910 tx=0x7fa07c009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:08.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.196+0000 7fa0923aa700 1 -- 172.21.15.121:0/100161961 shutdown_connections 2024-04-05T21:30:08.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.196+0000 7fa0923aa700 1 --2- 172.21.15.121:0/100161961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa08c101770 0x7fa08c1057c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:08.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.196+0000 7fa0923aa700 1 --2- 172.21.15.121:0/100161961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa08c100dc0 0x7fa08c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:08.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.196+0000 7fa0923aa700 1 -- 172.21.15.121:0/100161961 >> 172.21.15.121:0/100161961 conn(0x7fa08c0fc670 msgr2=0x7fa08c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:08.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.196+0000 7fa0923aa700 1 -- 172.21.15.121:0/100161961 shutdown_connections 2024-04-05T21:30:08.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.196+0000 7fa0923aa700 1 -- 172.21.15.121:0/100161961 wait complete. 2024-04-05T21:30:08.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa0923aa700 1 Processor -- start 2024-04-05T21:30:08.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa0923aa700 1 -- start start 2024-04-05T21:30:08.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa0923aa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa08c100dc0 0x7fa08c196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:08.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa0923aa700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa08c101770 0x7fa08c196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:08.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa0923aa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c1971e0 con 0x7fa08c100dc0 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa0923aa700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c197320 con 0x7fa08c101770 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa08bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa08c100dc0 0x7fa08c196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa08bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa08c100dc0 0x7fa08c196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45590/0 (socket says 172.21.15.121:45590) 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa08bfff700 1 -- 172.21.15.121:0/27151618 learned_addr learned my addr 172.21.15.121:0/27151618 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa08b7fe700 1 --2- 172.21.15.121:0/27151618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa08c101770 0x7fa08c196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa08bfff700 1 -- 172.21.15.121:0/27151618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa08c101770 msgr2=0x7fa08c196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa08bfff700 1 --2- 172.21.15.121:0/27151618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa08c101770 0x7fa08c196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa08bfff700 1 -- 172.21.15.121:0/27151618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07c0095f0 con 0x7fa08c100dc0 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.197+0000 7fa08b7fe700 1 --2- 172.21.15.121:0/27151618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa08c101770 0x7fa08c196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.198+0000 7fa08bfff700 1 --2- 172.21.15.121:0/27151618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa08c100dc0 0x7fa08c196650 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa08000c930 tx=0x7fa08000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.198+0000 7fa0897fa700 1 -- 172.21.15.121:0/27151618 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa080007ad0 con 0x7fa08c100dc0 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.198+0000 7fa0923aa700 1 -- 172.21.15.121:0/27151618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08c19b170 con 0x7fa08c100dc0 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.198+0000 7fa0897fa700 1 -- 172.21.15.121:0/27151618 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa080007c30 con 0x7fa08c100dc0 2024-04-05T21:30:08.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.198+0000 7fa0897fa700 1 -- 172.21.15.121:0/27151618 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa080018630 con 0x7fa08c100dc0 2024-04-05T21:30:08.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.198+0000 7fa0923aa700 1 -- 172.21.15.121:0/27151618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08c19b6c0 con 0x7fa08c100dc0 2024-04-05T21:30:08.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.199+0000 7fa0923aa700 1 -- 172.21.15.121:0/27151618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08c1091c0 con 0x7fa08c100dc0 2024-04-05T21:30:08.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.199+0000 7fa0897fa700 1 -- 172.21.15.121:0/27151618 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fa08000f450 con 0x7fa08c100dc0 2024-04-05T21:30:08.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.200+0000 7fa0897fa700 1 --2- 172.21.15.121:0/27151618 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa07406cce0 0x7fa07406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:08.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.200+0000 7fa08b7fe700 1 --2- 172.21.15.121:0/27151618 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa07406cce0 0x7fa07406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:08.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.200+0000 7fa0897fa700 1 -- 172.21.15.121:0/27151618 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fa08008c710 con 0x7fa08c100dc0 2024-04-05T21:30:08.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.200+0000 7fa08b7fe700 1 --2- 172.21.15.121:0/27151618 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa07406cce0 0x7fa07406f1a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa07c006010 tx=0x7fa07c005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:08.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.204+0000 7fa0897fa700 1 -- 172.21.15.121:0/27151618 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7fa0800568e0 con 0x7fa08c100dc0 2024-04-05T21:30:08.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.401+0000 7fa0923aa700 1 -- 172.21.15.121:0/27151618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fa08c04ea90 con 0x7fa08c100dc0 2024-04-05T21:30:08.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.402+0000 7fa0897fa700 1 -- 172.21.15.121:0/27151618 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v45)=0 v45) v1 ==== 155+0+0 (secure 0 0 0) 0x7fa080059f00 con 0x7fa08c100dc0 2024-04-05T21:30:08.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.404+0000 7fa0923aa700 1 -- 172.21.15.121:0/27151618 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa07406cce0 msgr2=0x7fa07406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:08.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.404+0000 7fa0923aa700 1 --2- 172.21.15.121:0/27151618 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa07406cce0 0x7fa07406f1a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa07c006010 tx=0x7fa07c005960 comp rx=0 tx=0).stop 2024-04-05T21:30:08.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.404+0000 7fa0923aa700 1 -- 172.21.15.121:0/27151618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa08c100dc0 msgr2=0x7fa08c196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:08.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.404+0000 7fa0923aa700 1 --2- 172.21.15.121:0/27151618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa08c100dc0 0x7fa08c196650 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa08000c930 tx=0x7fa08000ccf0 comp rx=0 tx=0).stop 2024-04-05T21:30:08.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.404+0000 7fa0923aa700 1 -- 172.21.15.121:0/27151618 shutdown_connections 2024-04-05T21:30:08.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.404+0000 7fa0923aa700 1 --2- 172.21.15.121:0/27151618 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa07406cce0 0x7fa07406f1a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:08.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.404+0000 7fa0923aa700 1 --2- 172.21.15.121:0/27151618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa08c100dc0 0x7fa08c196650 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:08.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.404+0000 7fa0923aa700 1 --2- 172.21.15.121:0/27151618 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa08c101770 0x7fa08c196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:08.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.404+0000 7fa0923aa700 1 -- 172.21.15.121:0/27151618 >> 172.21.15.121:0/27151618 conn(0x7fa08c0fc670 msgr2=0x7fa08c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:08.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.404+0000 7fa0923aa700 1 -- 172.21.15.121:0/27151618 shutdown_connections 2024-04-05T21:30:08.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:08.404+0000 7fa0923aa700 1 -- 172.21.15.121:0/27151618 wait complete. 2024-04-05T21:30:09.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-05T21:30:09.367 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:30:09.670 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:09 smithi121 ceph-mon[39723]: pgmap v82: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:09 smithi188 ceph-mon[49715]: pgmap v82: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:11.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.354+0000 7ff82ea0e700 1 -- 172.21.15.121:0/2470318999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff828101750 msgr2=0x7ff8281057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:11.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.354+0000 7ff82ea0e700 1 --2- 172.21.15.121:0/2470318999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff828101750 0x7ff8281057a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff818009910 tx=0x7ff818009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:11.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.354+0000 7ff82ea0e700 1 -- 172.21.15.121:0/2470318999 shutdown_connections 2024-04-05T21:30:11.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.354+0000 7ff82ea0e700 1 --2- 172.21.15.121:0/2470318999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff828101750 0x7ff8281057a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:11.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.354+0000 7ff82ea0e700 1 --2- 172.21.15.121:0/2470318999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff828100da0 0x7ff828101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:11.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.354+0000 7ff82ea0e700 1 -- 172.21.15.121:0/2470318999 >> 172.21.15.121:0/2470318999 conn(0x7ff8280fc630 msgr2=0x7ff8280fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:11.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.354+0000 7ff82ea0e700 1 -- 172.21.15.121:0/2470318999 shutdown_connections 2024-04-05T21:30:11.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.354+0000 7ff82ea0e700 1 -- 172.21.15.121:0/2470318999 wait complete. 2024-04-05T21:30:11.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff82ea0e700 1 Processor -- start 2024-04-05T21:30:11.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff82ea0e700 1 -- start start 2024-04-05T21:30:11.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff82ea0e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff828100da0 0x7ff828073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff82ea0e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff828101750 0x7ff8280735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff82ea0e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff828073b10 con 0x7ff828101750 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff82ea0e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff828073c50 con 0x7ff828100da0 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff8277fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff828101750 0x7ff8280735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff8277fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff828101750 0x7ff8280735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45602/0 (socket says 172.21.15.121:45602) 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff8277fe700 1 -- 172.21.15.121:0/3404690279 learned_addr learned my addr 172.21.15.121:0/3404690279 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff827fff700 1 --2- 172.21.15.121:0/3404690279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff828100da0 0x7ff828073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff8277fe700 1 -- 172.21.15.121:0/3404690279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff828100da0 msgr2=0x7ff828073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff8277fe700 1 --2- 172.21.15.121:0/3404690279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff828100da0 0x7ff828073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff8277fe700 1 -- 172.21.15.121:0/3404690279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8180095f0 con 0x7ff828101750 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.355+0000 7ff827fff700 1 --2- 172.21.15.121:0/3404690279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff828100da0 0x7ff828073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.356+0000 7ff8277fe700 1 --2- 172.21.15.121:0/3404690279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff828101750 0x7ff8280735d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff81800bda0 tx=0x7ff81800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.356+0000 7ff8257fa700 1 -- 172.21.15.121:0/3404690279 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff81801d070 con 0x7ff828101750 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.356+0000 7ff8257fa700 1 -- 172.21.15.121:0/3404690279 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff81800f460 con 0x7ff828101750 2024-04-05T21:30:11.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.356+0000 7ff8257fa700 1 -- 172.21.15.121:0/3404690279 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff818021640 con 0x7ff828101750 2024-04-05T21:30:11.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.356+0000 7ff82ea0e700 1 -- 172.21.15.121:0/3404690279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff828073ed0 con 0x7ff828101750 2024-04-05T21:30:11.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.356+0000 7ff82ea0e700 1 -- 172.21.15.121:0/3404690279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8281a2aa0 con 0x7ff828101750 2024-04-05T21:30:11.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.357+0000 7ff82ea0e700 1 -- 172.21.15.121:0/3404690279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8281091a0 con 0x7ff828101750 2024-04-05T21:30:11.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.357+0000 7ff8257fa700 1 -- 172.21.15.121:0/3404690279 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7ff81800f9b0 con 0x7ff828101750 2024-04-05T21:30:11.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.357+0000 7ff8257fa700 1 --2- 172.21.15.121:0/3404690279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff81006cce0 0x7ff81006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:11.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.358+0000 7ff8257fa700 1 -- 172.21.15.121:0/3404690279 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7ff81808d770 con 0x7ff828101750 2024-04-05T21:30:11.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.358+0000 7ff827fff700 1 --2- 172.21.15.121:0/3404690279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff81006cce0 0x7ff81006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:11.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.358+0000 7ff827fff700 1 --2- 172.21.15.121:0/3404690279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff81006cce0 0x7ff81006f1a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff81c005fd0 tx=0x7ff81c005e30 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:11.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.362+0000 7ff8257fa700 1 -- 172.21.15.121:0/3404690279 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7ff818057890 con 0x7ff828101750 2024-04-05T21:30:11.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.556+0000 7ff82ea0e700 1 -- 172.21.15.121:0/3404690279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7ff82804ea90 con 0x7ff828101750 2024-04-05T21:30:11.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.557+0000 7ff8257fa700 1 -- 172.21.15.121:0/3404690279 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v45)=0 v45) v1 ==== 163+0+0 (secure 0 0 0) 0x7ff82804ea90 con 0x7ff828101750 2024-04-05T21:30:11.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.559+0000 7ff82ea0e700 1 -- 172.21.15.121:0/3404690279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff81006cce0 msgr2=0x7ff81006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:11.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.559+0000 7ff82ea0e700 1 --2- 172.21.15.121:0/3404690279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff81006cce0 0x7ff81006f1a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff81c005fd0 tx=0x7ff81c005e30 comp rx=0 tx=0).stop 2024-04-05T21:30:11.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.559+0000 7ff82ea0e700 1 -- 172.21.15.121:0/3404690279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff828101750 msgr2=0x7ff8280735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:11.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.559+0000 7ff82ea0e700 1 --2- 172.21.15.121:0/3404690279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff828101750 0x7ff8280735d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff81800bda0 tx=0x7ff81800be80 comp rx=0 tx=0).stop 2024-04-05T21:30:11.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.559+0000 7ff82ea0e700 1 -- 172.21.15.121:0/3404690279 shutdown_connections 2024-04-05T21:30:11.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.559+0000 7ff82ea0e700 1 --2- 172.21.15.121:0/3404690279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff81006cce0 0x7ff81006f1a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:11.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.559+0000 7ff82ea0e700 1 --2- 172.21.15.121:0/3404690279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff828100da0 0x7ff828073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:11.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.559+0000 7ff82ea0e700 1 --2- 172.21.15.121:0/3404690279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff828101750 0x7ff8280735d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:11.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.559+0000 7ff82ea0e700 1 -- 172.21.15.121:0/3404690279 >> 172.21.15.121:0/3404690279 conn(0x7ff8280fc630 msgr2=0x7ff8280fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:11.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.559+0000 7ff82ea0e700 1 -- 172.21.15.121:0/3404690279 shutdown_connections 2024-04-05T21:30:11.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:11.559+0000 7ff82ea0e700 1 -- 172.21.15.121:0/3404690279 wait complete. 2024-04-05T21:30:11.703 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:11 smithi188 ceph-mon[49715]: pgmap v83: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:11 smithi121 ceph-mon[39723]: pgmap v83: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:12.358 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-05T21:30:12.591 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:30:13.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:13 smithi121 ceph-mon[39723]: pgmap v84: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:13.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:13 smithi188 ceph-mon[49715]: pgmap v84: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:14.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.478+0000 7f5feda7b700 1 -- 172.21.15.121:0/17805511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe8102620 msgr2=0x7f5fe810ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:14.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.478+0000 7f5feda7b700 1 --2- 172.21.15.121:0/17805511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe8102620 0x7f5fe810ab10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009910 tx=0x7f5fd0009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:14.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.478+0000 7f5feda7b700 1 -- 172.21.15.121:0/17805511 shutdown_connections 2024-04-05T21:30:14.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.478+0000 7f5feda7b700 1 --2- 172.21.15.121:0/17805511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe8102620 0x7f5fe810ab10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:14.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.478+0000 7f5feda7b700 1 --2- 172.21.15.121:0/17805511 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fe8101d00 0x7f5fe81020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:14.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.478+0000 7f5feda7b700 1 -- 172.21.15.121:0/17805511 >> 172.21.15.121:0/17805511 conn(0x7f5fe80fb570 msgr2=0x7f5fe80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:14.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.478+0000 7f5feda7b700 1 -- 172.21.15.121:0/17805511 shutdown_connections 2024-04-05T21:30:14.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.478+0000 7f5feda7b700 1 -- 172.21.15.121:0/17805511 wait complete. 2024-04-05T21:30:14.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.479+0000 7f5feda7b700 1 Processor -- start 2024-04-05T21:30:14.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.479+0000 7f5feda7b700 1 -- start start 2024-04-05T21:30:14.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.479+0000 7f5feda7b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe8101d00 0x7f5fe81966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:14.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.479+0000 7f5feda7b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fe8102620 0x7f5fe8196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:14.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.479+0000 7f5feda7b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe81972a0 con 0x7f5fe8101d00 2024-04-05T21:30:14.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.479+0000 7f5feda7b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe8199fd0 con 0x7f5fe8102620 2024-04-05T21:30:14.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.479+0000 7f5fe77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe8101d00 0x7f5fe81966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:14.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.479+0000 7f5fe77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe8101d00 0x7f5fe81966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48296/0 (socket says 172.21.15.121:48296) 2024-04-05T21:30:14.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.479+0000 7f5fe77fe700 1 -- 172.21.15.121:0/3908243731 learned_addr learned my addr 172.21.15.121:0/3908243731 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:14.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.480+0000 7f5fe6ffd700 1 --2- 172.21.15.121:0/3908243731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fe8102620 0x7f5fe8196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:14.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.480+0000 7f5fe77fe700 1 -- 172.21.15.121:0/3908243731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fe8102620 msgr2=0x7f5fe8196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:14.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.480+0000 7f5fe77fe700 1 --2- 172.21.15.121:0/3908243731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fe8102620 0x7f5fe8196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:14.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.480+0000 7f5fe77fe700 1 -- 172.21.15.121:0/3908243731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd00095f0 con 0x7f5fe8101d00 2024-04-05T21:30:14.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.480+0000 7f5fe77fe700 1 --2- 172.21.15.121:0/3908243731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe8101d00 0x7f5fe81966d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800d8d0 tx=0x7f5fd800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:14.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.480+0000 7f5fe4ff9700 1 -- 172.21.15.121:0/3908243731 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fd8009940 con 0x7f5fe8101d00 2024-04-05T21:30:14.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.480+0000 7f5feda7b700 1 -- 172.21.15.121:0/3908243731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe819a2b0 con 0x7f5fe8101d00 2024-04-05T21:30:14.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.480+0000 7f5fe4ff9700 1 -- 172.21.15.121:0/3908243731 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fd8010460 con 0x7f5fe8101d00 2024-04-05T21:30:14.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.480+0000 7f5fe4ff9700 1 -- 172.21.15.121:0/3908243731 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fd800f5f0 con 0x7f5fe8101d00 2024-04-05T21:30:14.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.480+0000 7f5feda7b700 1 -- 172.21.15.121:0/3908243731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe819a7d0 con 0x7f5fe8101d00 2024-04-05T21:30:14.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.481+0000 7f5feda7b700 1 -- 172.21.15.121:0/3908243731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe8108210 con 0x7f5fe8101d00 2024-04-05T21:30:14.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.481+0000 7f5fe4ff9700 1 -- 172.21.15.121:0/3908243731 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f5fd80105d0 con 0x7f5fe8101d00 2024-04-05T21:30:14.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.482+0000 7f5fe4ff9700 1 --2- 172.21.15.121:0/3908243731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fd406cce0 0x7f5fd406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:14.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.482+0000 7f5fe4ff9700 1 -- 172.21.15.121:0/3908243731 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f5fd808c520 con 0x7f5fe8101d00 2024-04-05T21:30:14.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.482+0000 7f5fe6ffd700 1 --2- 172.21.15.121:0/3908243731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fd406cce0 0x7f5fd406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:14.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.482+0000 7f5fe6ffd700 1 --2- 172.21.15.121:0/3908243731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fd406cce0 0x7f5fd406f1a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8197df0 tx=0x7f5fd001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:14.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.486+0000 7f5fe4ff9700 1 -- 172.21.15.121:0/3908243731 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f5fd8056640 con 0x7f5fe8101d00 2024-04-05T21:30:14.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.676+0000 7f5feda7b700 1 -- 172.21.15.121:0/3908243731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f5fe804ea90 con 0x7f5fe8101d00 2024-04-05T21:30:14.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.676+0000 7f5fe4ff9700 1 -- 172.21.15.121:0/3908243731 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v45)=0 v45) v1 ==== 135+0+0 (secure 0 0 0) 0x7f5fd8010880 con 0x7f5fe8101d00 2024-04-05T21:30:14.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.678+0000 7f5feda7b700 1 -- 172.21.15.121:0/3908243731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fd406cce0 msgr2=0x7f5fd406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:14.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.678+0000 7f5feda7b700 1 --2- 172.21.15.121:0/3908243731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fd406cce0 0x7f5fd406f1a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8197df0 tx=0x7f5fd001a040 comp rx=0 tx=0).stop 2024-04-05T21:30:14.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.678+0000 7f5feda7b700 1 -- 172.21.15.121:0/3908243731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe8101d00 msgr2=0x7f5fe81966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:14.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.678+0000 7f5feda7b700 1 --2- 172.21.15.121:0/3908243731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe8101d00 0x7f5fe81966d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800d8d0 tx=0x7f5fd800dc90 comp rx=0 tx=0).stop 2024-04-05T21:30:14.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.678+0000 7f5feda7b700 1 -- 172.21.15.121:0/3908243731 shutdown_connections 2024-04-05T21:30:14.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.678+0000 7f5feda7b700 1 --2- 172.21.15.121:0/3908243731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fd406cce0 0x7f5fd406f1a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:14.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.678+0000 7f5feda7b700 1 --2- 172.21.15.121:0/3908243731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe8101d00 0x7f5fe81966d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:14.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.678+0000 7f5feda7b700 1 --2- 172.21.15.121:0/3908243731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fe8102620 0x7f5fe8196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:14.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.678+0000 7f5feda7b700 1 -- 172.21.15.121:0/3908243731 >> 172.21.15.121:0/3908243731 conn(0x7f5fe80fb570 msgr2=0x7f5fe8105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:14.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.678+0000 7f5feda7b700 1 -- 172.21.15.121:0/3908243731 shutdown_connections 2024-04-05T21:30:14.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:14.678+0000 7f5feda7b700 1 -- 172.21.15.121:0/3908243731 wait complete. 2024-04-05T21:30:15.506 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-05T21:30:15.734 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:30:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:15 smithi121 ceph-mon[39723]: pgmap v85: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:15 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:30:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:15 smithi188 ceph-mon[49715]: pgmap v85: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:15 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:30:17.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.672+0000 7f1253f01700 1 -- 172.21.15.121:0/2001046017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f124c0ffbd0 msgr2=0x7f124c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:17.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.672+0000 7f1253f01700 1 --2- 172.21.15.121:0/2001046017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f124c0ffbd0 0x7f124c100050 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f123c009910 tx=0x7f123c009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:17.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.672+0000 7f1253f01700 1 -- 172.21.15.121:0/2001046017 shutdown_connections 2024-04-05T21:30:17.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.672+0000 7f1253f01700 1 --2- 172.21.15.121:0/2001046017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f124c0ffbd0 0x7f124c100050 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:17.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.672+0000 7f1253f01700 1 --2- 172.21.15.121:0/2001046017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f124c105a40 0x7f124c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:17.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.672+0000 7f1253f01700 1 -- 172.21.15.121:0/2001046017 >> 172.21.15.121:0/2001046017 conn(0x7f124c0755a0 msgr2=0x7f124c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:17.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.672+0000 7f1253f01700 1 -- 172.21.15.121:0/2001046017 shutdown_connections 2024-04-05T21:30:17.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.672+0000 7f1253f01700 1 -- 172.21.15.121:0/2001046017 wait complete. 2024-04-05T21:30:17.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.672+0000 7f1253f01700 1 Processor -- start 2024-04-05T21:30:17.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.672+0000 7f1253f01700 1 -- start start 2024-04-05T21:30:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f1253f01700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f124c0ffbd0 0x7f124c197450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f1253f01700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f124c105a40 0x7f124c197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f1253f01700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c197fe0 con 0x7f124c105a40 2024-04-05T21:30:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f1253f01700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c198120 con 0x7f124c0ffbd0 2024-04-05T21:30:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f125149c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f124c105a40 0x7f124c197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f125149c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f124c105a40 0x7f124c197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48304/0 (socket says 172.21.15.121:48304) 2024-04-05T21:30:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f125149c700 1 -- 172.21.15.121:0/1913756455 learned_addr learned my addr 172.21.15.121:0/1913756455 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f1251c9d700 1 --2- 172.21.15.121:0/1913756455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f124c0ffbd0 0x7f124c197450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f125149c700 1 -- 172.21.15.121:0/1913756455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f124c0ffbd0 msgr2=0x7f124c197450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f125149c700 1 --2- 172.21.15.121:0/1913756455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f124c0ffbd0 0x7f124c197450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f125149c700 1 -- 172.21.15.121:0/1913756455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c0095f0 con 0x7f124c105a40 2024-04-05T21:30:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f125149c700 1 --2- 172.21.15.121:0/1913756455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f124c105a40 0x7f124c197990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f123c006010 tx=0x7f123c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f1242ffd700 1 -- 172.21.15.121:0/1913756455 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f123c01d070 con 0x7f124c105a40 2024-04-05T21:30:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f1253f01700 1 -- 172.21.15.121:0/1913756455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124c1a04c0 con 0x7f124c105a40 2024-04-05T21:30:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f1242ffd700 1 -- 172.21.15.121:0/1913756455 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f123c005940 con 0x7f124c105a40 2024-04-05T21:30:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f1242ffd700 1 -- 172.21.15.121:0/1913756455 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f123c021410 con 0x7f124c105a40 2024-04-05T21:30:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.673+0000 7f1253f01700 1 -- 172.21.15.121:0/1913756455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124c1a08d0 con 0x7f124c105a40 2024-04-05T21:30:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.674+0000 7f1253f01700 1 -- 172.21.15.121:0/1913756455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124c107fd0 con 0x7f124c105a40 2024-04-05T21:30:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.675+0000 7f1242ffd700 1 -- 172.21.15.121:0/1913756455 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f123c003a60 con 0x7f124c105a40 2024-04-05T21:30:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.675+0000 7f1242ffd700 1 --2- 172.21.15.121:0/1913756455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f123806cce0 0x7f123806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.675+0000 7f1242ffd700 1 -- 172.21.15.121:0/1913756455 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f123c08d750 con 0x7f124c105a40 2024-04-05T21:30:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.675+0000 7f1251c9d700 1 --2- 172.21.15.121:0/1913756455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f123806cce0 0x7f123806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:17.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.676+0000 7f1251c9d700 1 --2- 172.21.15.121:0/1913756455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f123806cce0 0x7f123806f1a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f124800ac50 tx=0x7f124800a380 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:17.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.680+0000 7f1242ffd700 1 -- 172.21.15.121:0/1913756455 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f123c057870 con 0x7f124c105a40 2024-04-05T21:30:17.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:17 smithi121 ceph-mon[39723]: pgmap v86: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:17.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:17 smithi188 ceph-mon[49715]: pgmap v86: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:17.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.890+0000 7f1253f01700 1 -- 172.21.15.121:0/1913756455 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e", "target": ["mon-mgr", ""]}) v1 -- 0x7f124c02cdf0 con 0x7f123806cce0 2024-04-05T21:30:17.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.895+0000 7f1242ffd700 1 -- 172.21.15.121:0/1913756455 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f124c02cdf0 con 0x7f123806cce0 2024-04-05T21:30:17.896 INFO:teuthology.orchestra.run.smithi121.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:30:17.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.897+0000 7f1253f01700 1 -- 172.21.15.121:0/1913756455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f123806cce0 msgr2=0x7f123806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:17.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.897+0000 7f1253f01700 1 --2- 172.21.15.121:0/1913756455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f123806cce0 0x7f123806f1a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f124800ac50 tx=0x7f124800a380 comp rx=0 tx=0).stop 2024-04-05T21:30:17.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.897+0000 7f1253f01700 1 -- 172.21.15.121:0/1913756455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f124c105a40 msgr2=0x7f124c197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:17.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.897+0000 7f1253f01700 1 --2- 172.21.15.121:0/1913756455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f124c105a40 0x7f124c197990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f123c006010 tx=0x7f123c00bd70 comp rx=0 tx=0).stop 2024-04-05T21:30:17.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.897+0000 7f1253f01700 1 -- 172.21.15.121:0/1913756455 shutdown_connections 2024-04-05T21:30:17.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.897+0000 7f1253f01700 1 --2- 172.21.15.121:0/1913756455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f123806cce0 0x7f123806f1a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:17.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.897+0000 7f1253f01700 1 --2- 172.21.15.121:0/1913756455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f124c0ffbd0 0x7f124c197450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:17.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.897+0000 7f1253f01700 1 --2- 172.21.15.121:0/1913756455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f124c105a40 0x7f124c197990 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:17.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.897+0000 7f1253f01700 1 -- 172.21.15.121:0/1913756455 >> 172.21.15.121:0/1913756455 conn(0x7f124c0755a0 msgr2=0x7f124c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:17.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.897+0000 7f1253f01700 1 -- 172.21.15.121:0/1913756455 shutdown_connections 2024-04-05T21:30:17.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:17.897+0000 7f1253f01700 1 -- 172.21.15.121:0/1913756455 wait complete. 2024-04-05T21:30:18.691 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-05T21:30:18.692 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T21:30:18.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-05T21:30:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:18 smithi121 ceph-mon[39723]: from='client.14804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:18 smithi121 ceph-mon[39723]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:30:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:18 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:18 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:18 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:18 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:30:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:18 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:18 smithi121 ceph-mon[39723]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:30:18.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:18 smithi121 ceph-mon[39723]: pgmap v87: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:18 smithi188 ceph-mon[49715]: from='client.14804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:18 smithi188 ceph-mon[49715]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:30:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:18 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:18 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:18 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:18 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:30:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:18 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:18 smithi188 ceph-mon[49715]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8919605b0a5dc1f1650c8aec3051c77c0e3e240e 2024-04-05T21:30:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:18 smithi188 ceph-mon[49715]: pgmap v87: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:19.762 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T21:30:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:21 smithi121 ceph-mon[39723]: pgmap v88: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:21 smithi121 ceph-mon[39723]: Upgrade: Target is version 18.2.2-1173-g8919605b (reef) 2024-04-05T21:30:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:21 smithi121 ceph-mon[39723]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da'] 2024-04-05T21:30:21.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:21.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:21.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:30:21.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:21 smithi121 ceph-mon[39723]: Upgrade: Setting container_image for all mgr 2024-04-05T21:30:21.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:21.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:21 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-05T21:30:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:21 smithi188 ceph-mon[49715]: pgmap v88: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:21 smithi188 ceph-mon[49715]: Upgrade: Target is version 18.2.2-1173-g8919605b (reef) 2024-04-05T21:30:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:21 smithi188 ceph-mon[49715]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da'] 2024-04-05T21:30:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:30:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:21 smithi188 ceph-mon[49715]: Upgrade: Setting container_image for all mgr 2024-04-05T21:30:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:21 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-05T21:30:23.407 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:23 smithi121 ceph-mon[39723]: pgmap v89: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:23.407 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:23 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:23.407 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:23 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T21:30:23.407 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:23 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T21:30:23.407 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:23 smithi121 ceph-mon[39723]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:23.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.448+0000 7fbaa2b47700 1 -- 172.21.15.121:0/1563419845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba9c1026f0 msgr2=0x7fba9c10abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:23.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.448+0000 7fbaa2b47700 1 --2- 172.21.15.121:0/1563419845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba9c1026f0 0x7fba9c10abe0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fba88009940 tx=0x7fba88009c50 comp rx=0 tx=0).stop 2024-04-05T21:30:23.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.448+0000 7fbaa2b47700 1 -- 172.21.15.121:0/1563419845 shutdown_connections 2024-04-05T21:30:23.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.448+0000 7fbaa2b47700 1 --2- 172.21.15.121:0/1563419845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba9c1026f0 0x7fba9c10abe0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:23.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.448+0000 7fbaa2b47700 1 --2- 172.21.15.121:0/1563419845 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba9c101dd0 0x7fba9c1021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:23.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.448+0000 7fbaa2b47700 1 -- 172.21.15.121:0/1563419845 >> 172.21.15.121:0/1563419845 conn(0x7fba9c0fb540 msgr2=0x7fba9c0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:23.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.448+0000 7fbaa2b47700 1 -- 172.21.15.121:0/1563419845 shutdown_connections 2024-04-05T21:30:23.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.448+0000 7fbaa2b47700 1 -- 172.21.15.121:0/1563419845 wait complete. 2024-04-05T21:30:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.449+0000 7fbaa2b47700 1 Processor -- start 2024-04-05T21:30:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.449+0000 7fbaa2b47700 1 -- start start 2024-04-05T21:30:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.449+0000 7fbaa2b47700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba9c101dd0 0x7fba9c196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.449+0000 7fbaa2b47700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba9c1026f0 0x7fba9c196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.449+0000 7fbaa2b47700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c197280 con 0x7fba9c101dd0 2024-04-05T21:30:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.449+0000 7fbaa2b47700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c19b010 con 0x7fba9c1026f0 2024-04-05T21:30:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.449+0000 7fbaa08e3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba9c101dd0 0x7fba9c196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.449+0000 7fbaa08e3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba9c101dd0 0x7fba9c196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48322/0 (socket says 172.21.15.121:48322) 2024-04-05T21:30:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.449+0000 7fbaa08e3700 1 -- 172.21.15.121:0/276723455 learned_addr learned my addr 172.21.15.121:0/276723455 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.449+0000 7fba9bfff700 1 --2- 172.21.15.121:0/276723455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba9c1026f0 0x7fba9c196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.450+0000 7fbaa08e3700 1 -- 172.21.15.121:0/276723455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba9c1026f0 msgr2=0x7fba9c196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.450+0000 7fbaa08e3700 1 --2- 172.21.15.121:0/276723455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba9c1026f0 0x7fba9c196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.450+0000 7fbaa08e3700 1 -- 172.21.15.121:0/276723455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba880095f0 con 0x7fba9c101dd0 2024-04-05T21:30:23.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.450+0000 7fbaa08e3700 1 --2- 172.21.15.121:0/276723455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba9c101dd0 0x7fba9c196660 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fba9000c8a0 tx=0x7fba9000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.450+0000 7fba99ffb700 1 -- 172.21.15.121:0/276723455 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba900078e0 con 0x7fba9c101dd0 2024-04-05T21:30:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.450+0000 7fbaa2b47700 1 -- 172.21.15.121:0/276723455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9c19b2f0 con 0x7fba9c101dd0 2024-04-05T21:30:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.450+0000 7fba99ffb700 1 -- 172.21.15.121:0/276723455 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba9000f450 con 0x7fba9c101dd0 2024-04-05T21:30:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.450+0000 7fba99ffb700 1 -- 172.21.15.121:0/276723455 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba9000e5e0 con 0x7fba9c101dd0 2024-04-05T21:30:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.450+0000 7fbaa2b47700 1 -- 172.21.15.121:0/276723455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9c19b840 con 0x7fba9c101dd0 2024-04-05T21:30:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.451+0000 7fba99ffb700 1 -- 172.21.15.121:0/276723455 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fba9000f5c0 con 0x7fba9c101dd0 2024-04-05T21:30:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.452+0000 7fbaa2b47700 1 -- 172.21.15.121:0/276723455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9c1082e0 con 0x7fba9c101dd0 2024-04-05T21:30:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.452+0000 7fba99ffb700 1 --2- 172.21.15.121:0/276723455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fba8c06cce0 0x7fba8c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.452+0000 7fba99ffb700 1 -- 172.21.15.121:0/276723455 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fba9008c1b0 con 0x7fba9c101dd0 2024-04-05T21:30:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.452+0000 7fba9bfff700 1 --2- 172.21.15.121:0/276723455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fba8c06cce0 0x7fba8c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:23.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.452+0000 7fba9bfff700 1 --2- 172.21.15.121:0/276723455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fba8c06cce0 0x7fba8c06f1a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fba9c197c80 tx=0x7fba88005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:23.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.457+0000 7fba99ffb700 1 -- 172.21.15.121:0/276723455 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7fba90056230 con 0x7fba9c101dd0 2024-04-05T21:30:23.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.675+0000 7fbaa2b47700 1 -- 172.21.15.121:0/276723455 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba9c19bbd0 con 0x7fba8c06cce0 2024-04-05T21:30:23.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.676+0000 7fba99ffb700 1 -- 172.21.15.121:0/276723455 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fba9c19bbd0 con 0x7fba8c06cce0 2024-04-05T21:30:23.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.678+0000 7fbaa2b47700 1 -- 172.21.15.121:0/276723455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fba8c06cce0 msgr2=0x7fba8c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:23.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.678+0000 7fbaa2b47700 1 --2- 172.21.15.121:0/276723455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fba8c06cce0 0x7fba8c06f1a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fba9c197c80 tx=0x7fba88005fb0 comp rx=0 tx=0).stop 2024-04-05T21:30:23.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.679+0000 7fbaa2b47700 1 -- 172.21.15.121:0/276723455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba9c101dd0 msgr2=0x7fba9c196660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:23.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.679+0000 7fbaa2b47700 1 --2- 172.21.15.121:0/276723455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba9c101dd0 0x7fba9c196660 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fba9000c8a0 tx=0x7fba9000cbb0 comp rx=0 tx=0).stop 2024-04-05T21:30:23.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.679+0000 7fbaa2b47700 1 -- 172.21.15.121:0/276723455 shutdown_connections 2024-04-05T21:30:23.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.679+0000 7fbaa2b47700 1 --2- 172.21.15.121:0/276723455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fba8c06cce0 0x7fba8c06f1a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:23.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.679+0000 7fbaa2b47700 1 --2- 172.21.15.121:0/276723455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba9c101dd0 0x7fba9c196660 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:23.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.679+0000 7fbaa2b47700 1 --2- 172.21.15.121:0/276723455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba9c1026f0 0x7fba9c196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:23.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.679+0000 7fbaa2b47700 1 -- 172.21.15.121:0/276723455 >> 172.21.15.121:0/276723455 conn(0x7fba9c0fb540 msgr2=0x7fba9c105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:23.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.679+0000 7fbaa2b47700 1 -- 172.21.15.121:0/276723455 shutdown_connections 2024-04-05T21:30:23.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.679+0000 7fbaa2b47700 1 -- 172.21.15.121:0/276723455 wait complete. 2024-04-05T21:30:23.694 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:30:23.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:23 smithi188 ceph-mon[49715]: pgmap v89: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:23.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:23 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:23.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:23 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T21:30:23.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:23 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T21:30:23.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:23 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:23.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.790+0000 7fce920a2700 1 -- 172.21.15.121:0/1333987450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce8c0ffa40 msgr2=0x7fce8c0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:23.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.790+0000 7fce920a2700 1 --2- 172.21.15.121:0/1333987450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce8c0ffa40 0x7fce8c0ffec0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fce80009910 tx=0x7fce80009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:23.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.790+0000 7fce920a2700 1 -- 172.21.15.121:0/1333987450 shutdown_connections 2024-04-05T21:30:23.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.790+0000 7fce920a2700 1 --2- 172.21.15.121:0/1333987450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce8c0ffa40 0x7fce8c0ffec0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:23.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.790+0000 7fce920a2700 1 --2- 172.21.15.121:0/1333987450 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce8c105a40 0x7fce8c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:23.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.790+0000 7fce920a2700 1 -- 172.21.15.121:0/1333987450 >> 172.21.15.121:0/1333987450 conn(0x7fce8c0fb570 msgr2=0x7fce8c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:23.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.790+0000 7fce920a2700 1 -- 172.21.15.121:0/1333987450 shutdown_connections 2024-04-05T21:30:23.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.790+0000 7fce920a2700 1 -- 172.21.15.121:0/1333987450 wait complete. 2024-04-05T21:30:23.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.791+0000 7fce920a2700 1 Processor -- start 2024-04-05T21:30:23.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.791+0000 7fce920a2700 1 -- start start 2024-04-05T21:30:23.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.791+0000 7fce920a2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce8c0ffa40 0x7fce8c196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:23.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.791+0000 7fce920a2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce8c105a40 0x7fce8c196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:23.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.791+0000 7fce920a2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c197230 con 0x7fce8c105a40 2024-04-05T21:30:23.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.791+0000 7fce920a2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c19afc0 con 0x7fce8c0ffa40 2024-04-05T21:30:23.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.791+0000 7fce8affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce8c105a40 0x7fce8c196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.791+0000 7fce8affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce8c105a40 0x7fce8c196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48328/0 (socket says 172.21.15.121:48328) 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.791+0000 7fce8affd700 1 -- 172.21.15.121:0/2969768600 learned_addr learned my addr 172.21.15.121:0/2969768600 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.792+0000 7fce8b7fe700 1 --2- 172.21.15.121:0/2969768600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce8c0ffa40 0x7fce8c196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.792+0000 7fce8affd700 1 -- 172.21.15.121:0/2969768600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce8c0ffa40 msgr2=0x7fce8c196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.792+0000 7fce8affd700 1 --2- 172.21.15.121:0/2969768600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce8c0ffa40 0x7fce8c196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.792+0000 7fce8affd700 1 -- 172.21.15.121:0/2969768600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce800095f0 con 0x7fce8c105a40 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.792+0000 7fce8b7fe700 1 --2- 172.21.15.121:0/2969768600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce8c0ffa40 0x7fce8c196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.792+0000 7fce8affd700 1 --2- 172.21.15.121:0/2969768600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce8c105a40 0x7fce8c196b50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fce800092e0 tx=0x7fce8000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.792+0000 7fce88ff9700 1 -- 172.21.15.121:0/2969768600 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce8001d070 con 0x7fce8c105a40 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.792+0000 7fce920a2700 1 -- 172.21.15.121:0/2969768600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8c19b240 con 0x7fce8c105a40 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.792+0000 7fce88ff9700 1 -- 172.21.15.121:0/2969768600 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce80005e00 con 0x7fce8c105a40 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.792+0000 7fce88ff9700 1 -- 172.21.15.121:0/2969768600 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce8000fe60 con 0x7fce8c105a40 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.792+0000 7fce920a2700 1 -- 172.21.15.121:0/2969768600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8c19b730 con 0x7fce8c105a40 2024-04-05T21:30:23.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.793+0000 7fce920a2700 1 -- 172.21.15.121:0/2969768600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8c04ea90 con 0x7fce8c105a40 2024-04-05T21:30:23.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.793+0000 7fce88ff9700 1 -- 172.21.15.121:0/2969768600 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7fce8000f6d0 con 0x7fce8c105a40 2024-04-05T21:30:23.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.794+0000 7fce88ff9700 1 --2- 172.21.15.121:0/2969768600 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce7406cce0 0x7fce7406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:23.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.794+0000 7fce88ff9700 1 -- 172.21.15.121:0/2969768600 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fce8008d7f0 con 0x7fce8c105a40 2024-04-05T21:30:23.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.794+0000 7fce8b7fe700 1 --2- 172.21.15.121:0/2969768600 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce7406cce0 0x7fce7406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:23.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.794+0000 7fce8b7fe700 1 --2- 172.21.15.121:0/2969768600 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce7406cce0 0x7fce7406f1a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fce7c005fd0 tx=0x7fce7c005ee0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:23.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:23.798+0000 7fce88ff9700 1 -- 172.21.15.121:0/2969768600 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7fce80057910 con 0x7fce8c105a40 2024-04-05T21:30:24.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.010+0000 7fce920a2700 1 -- 172.21.15.121:0/2969768600 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce8c19bb50 con 0x7fce7406cce0 2024-04-05T21:30:24.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.011+0000 7fce88ff9700 1 -- 172.21.15.121:0/2969768600 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fce8c19bb50 con 0x7fce7406cce0 2024-04-05T21:30:24.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.014+0000 7fce920a2700 1 -- 172.21.15.121:0/2969768600 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce7406cce0 msgr2=0x7fce7406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:24.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.014+0000 7fce920a2700 1 --2- 172.21.15.121:0/2969768600 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce7406cce0 0x7fce7406f1a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fce7c005fd0 tx=0x7fce7c005ee0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.014+0000 7fce920a2700 1 -- 172.21.15.121:0/2969768600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce8c105a40 msgr2=0x7fce8c196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:24.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.014+0000 7fce920a2700 1 --2- 172.21.15.121:0/2969768600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce8c105a40 0x7fce8c196b50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fce800092e0 tx=0x7fce8000be20 comp rx=0 tx=0).stop 2024-04-05T21:30:24.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.014+0000 7fce920a2700 1 -- 172.21.15.121:0/2969768600 shutdown_connections 2024-04-05T21:30:24.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.014+0000 7fce920a2700 1 --2- 172.21.15.121:0/2969768600 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce7406cce0 0x7fce7406f1a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.014+0000 7fce920a2700 1 --2- 172.21.15.121:0/2969768600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce8c0ffa40 0x7fce8c196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.014+0000 7fce920a2700 1 --2- 172.21.15.121:0/2969768600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce8c105a40 0x7fce8c196b50 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.014+0000 7fce920a2700 1 -- 172.21.15.121:0/2969768600 >> 172.21.15.121:0/2969768600 conn(0x7fce8c0fb570 msgr2=0x7fce8c1037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:24.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.014+0000 7fce920a2700 1 -- 172.21.15.121:0/2969768600 shutdown_connections 2024-04-05T21:30:24.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.014+0000 7fce920a2700 1 -- 172.21.15.121:0/2969768600 wait complete. 2024-04-05T21:30:24.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.120+0000 7f7979bcd700 1 -- 172.21.15.121:0/3052011319 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7974105a40 msgr2=0x7f7974105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:24.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.120+0000 7f7979bcd700 1 --2- 172.21.15.121:0/3052011319 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7974105a40 0x7f7974105e20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7964009910 tx=0x7f7964009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:24.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.120+0000 7f7979bcd700 1 -- 172.21.15.121:0/3052011319 shutdown_connections 2024-04-05T21:30:24.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.120+0000 7f7979bcd700 1 --2- 172.21.15.121:0/3052011319 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79740ffbd0 0x7f7974100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.120+0000 7f7979bcd700 1 --2- 172.21.15.121:0/3052011319 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7974105a40 0x7f7974105e20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.120+0000 7f7979bcd700 1 -- 172.21.15.121:0/3052011319 >> 172.21.15.121:0/3052011319 conn(0x7f79740755a0 msgr2=0x7f79740759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:24.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.120+0000 7f7979bcd700 1 -- 172.21.15.121:0/3052011319 shutdown_connections 2024-04-05T21:30:24.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.120+0000 7f7979bcd700 1 -- 172.21.15.121:0/3052011319 wait complete. 2024-04-05T21:30:24.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.121+0000 7f7979bcd700 1 Processor -- start 2024-04-05T21:30:24.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.121+0000 7f7979bcd700 1 -- start start 2024-04-05T21:30:24.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.121+0000 7f7979bcd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79740ffbd0 0x7f7974195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:24.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.121+0000 7f7979bcd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7974105a40 0x7f7974195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:24.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.121+0000 7f7979bcd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7974196170 con 0x7f79740ffbd0 2024-04-05T21:30:24.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.121+0000 7f7979bcd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7974199f00 con 0x7f7974105a40 2024-04-05T21:30:24.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.121+0000 7f79737fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79740ffbd0 0x7f7974195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:24.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.122+0000 7f79737fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79740ffbd0 0x7f7974195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48352/0 (socket says 172.21.15.121:48352) 2024-04-05T21:30:24.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.122+0000 7f79737fe700 1 -- 172.21.15.121:0/4179428781 learned_addr learned my addr 172.21.15.121:0/4179428781 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:24.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.122+0000 7f79737fe700 1 -- 172.21.15.121:0/4179428781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7974105a40 msgr2=0x7f7974195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:24.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.122+0000 7f7972ffd700 1 --2- 172.21.15.121:0/4179428781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7974105a40 0x7f7974195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:24.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.122+0000 7f79737fe700 1 --2- 172.21.15.121:0/4179428781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7974105a40 0x7f7974195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.122+0000 7f79737fe700 1 -- 172.21.15.121:0/4179428781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79640095f0 con 0x7f79740ffbd0 2024-04-05T21:30:24.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.122+0000 7f79737fe700 1 --2- 172.21.15.121:0/4179428781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79740ffbd0 0x7f7974195550 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7964006010 tx=0x7f796400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:24.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.122+0000 7f7970ff9700 1 -- 172.21.15.121:0/4179428781 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f796401f070 con 0x7f79740ffbd0 2024-04-05T21:30:24.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.122+0000 7f7979bcd700 1 -- 172.21.15.121:0/4179428781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797419a180 con 0x7f79740ffbd0 2024-04-05T21:30:24.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.122+0000 7f7970ff9700 1 -- 172.21.15.121:0/4179428781 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7964019610 con 0x7f79740ffbd0 2024-04-05T21:30:24.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.122+0000 7f7970ff9700 1 -- 172.21.15.121:0/4179428781 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f796400f670 con 0x7f79740ffbd0 2024-04-05T21:30:24.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.122+0000 7f7979bcd700 1 -- 172.21.15.121:0/4179428781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797419a670 con 0x7f79740ffbd0 2024-04-05T21:30:24.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.123+0000 7f7979bcd700 1 -- 172.21.15.121:0/4179428781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f797404ea90 con 0x7f79740ffbd0 2024-04-05T21:30:24.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.124+0000 7f7970ff9700 1 -- 172.21.15.121:0/4179428781 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f7964019bd0 con 0x7f79740ffbd0 2024-04-05T21:30:24.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.124+0000 7f7970ff9700 1 --2- 172.21.15.121:0/4179428781 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f795c06cce0 0x7f795c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:24.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.124+0000 7f7970ff9700 1 -- 172.21.15.121:0/4179428781 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f796408f950 con 0x7f79740ffbd0 2024-04-05T21:30:24.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.124+0000 7f7972ffd700 1 --2- 172.21.15.121:0/4179428781 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f795c06cce0 0x7f795c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:24.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.124+0000 7f7972ffd700 1 --2- 172.21.15.121:0/4179428781 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f795c06cce0 0x7f795c06f1a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7968005950 tx=0x7f79680058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:24.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.128+0000 7f7970ff9700 1 -- 172.21.15.121:0/4179428781 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f7964059a70 con 0x7f79740ffbd0 2024-04-05T21:30:24.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.333+0000 7f7979bcd700 1 -- 172.21.15.121:0/4179428781 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f797419ab70 con 0x7f795c06cce0 2024-04-05T21:30:24.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.343+0000 7f7970ff9700 1 -- 172.21.15.121:0/4179428781 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f797419ab70 con 0x7f795c06cce0 2024-04-05T21:30:24.344 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:30:24.344 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (75s) 51s ago 12m 16.6M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (12m) 51s ago 12m 8024k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (11m) 115s ago 11m 7834k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (12m) 51s ago 12m 6651k - 18.2.1 5be31c24972a 5bf35fe2ace4 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (11m) 115s ago 11m 6651k - 18.2.1 5be31c24972a dcdfa6ae17bf 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (57s) 51s ago 12m 47.8M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (6m) 51s ago 6m 64.6M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (6m) 51s ago 6m 135M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (6m) 115s ago 6m 110M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (6m) 115s ago 6m 62.4M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (3m) 51s ago 14m 498M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (2m) 115s ago 11m 448M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (14m) 51s ago 14m 58.6M 2048M 18.2.1 5be31c24972a 3c30ce99c006 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (11m) 115s ago 11m 45.2M 2048M 18.2.1 5be31c24972a 8ecc9636670e 2024-04-05T21:30:24.345 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (2m) 51s ago 12m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:30:24.346 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (119s) 115s ago 11m 3653k - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:30:24.346 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (10m) 51s ago 10m 215M 1227M 18.2.1 5be31c24972a f6f3a45429a2 2024-04-05T21:30:24.346 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (10m) 51s ago 10m 241M 1227M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:30:24.346 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (9m) 51s ago 9m 182M 1227M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:30:24.346 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (9m) 115s ago 9m 286M 2251M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:30:24.346 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (8m) 115s ago 8m 244M 2251M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:30:24.346 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (8m) 115s ago 8m 223M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:30:24.346 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (93s) 51s ago 12m 48.7M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:30:24.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.345+0000 7f7979bcd700 1 -- 172.21.15.121:0/4179428781 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f795c06cce0 msgr2=0x7f795c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:24.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.345+0000 7f7979bcd700 1 --2- 172.21.15.121:0/4179428781 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f795c06cce0 0x7f795c06f1a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7968005950 tx=0x7f79680058e0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.345+0000 7f7979bcd700 1 -- 172.21.15.121:0/4179428781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79740ffbd0 msgr2=0x7f7974195550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:24.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.345+0000 7f7979bcd700 1 --2- 172.21.15.121:0/4179428781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79740ffbd0 0x7f7974195550 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7964006010 tx=0x7f796400be20 comp rx=0 tx=0).stop 2024-04-05T21:30:24.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.345+0000 7f7979bcd700 1 -- 172.21.15.121:0/4179428781 shutdown_connections 2024-04-05T21:30:24.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.345+0000 7f7979bcd700 1 --2- 172.21.15.121:0/4179428781 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f795c06cce0 0x7f795c06f1a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.345+0000 7f7979bcd700 1 --2- 172.21.15.121:0/4179428781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79740ffbd0 0x7f7974195550 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.345+0000 7f7979bcd700 1 --2- 172.21.15.121:0/4179428781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7974105a40 0x7f7974195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.345+0000 7f7979bcd700 1 -- 172.21.15.121:0/4179428781 >> 172.21.15.121:0/4179428781 conn(0x7f79740755a0 msgr2=0x7f79740fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:24.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.345+0000 7f7979bcd700 1 -- 172.21.15.121:0/4179428781 shutdown_connections 2024-04-05T21:30:24.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.345+0000 7f7979bcd700 1 -- 172.21.15.121:0/4179428781 wait complete. 2024-04-05T21:30:24.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.450+0000 7f0085fe6700 1 -- 172.21.15.121:0/2137203679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00800ff400 msgr2=0x7f00800ff7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.450+0000 7f0085fe6700 1 --2- 172.21.15.121:0/2137203679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00800ff400 0x7f00800ff7e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0070009910 tx=0x7f0070009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.450+0000 7f0085fe6700 1 -- 172.21.15.121:0/2137203679 shutdown_connections 2024-04-05T21:30:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.450+0000 7f0085fe6700 1 --2- 172.21.15.121:0/2137203679 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00800ffdb0 0x7f0080103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.450+0000 7f0085fe6700 1 --2- 172.21.15.121:0/2137203679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00800ff400 0x7f00800ff7e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.450+0000 7f0085fe6700 1 -- 172.21.15.121:0/2137203679 >> 172.21.15.121:0/2137203679 conn(0x7f0080075e30 msgr2=0x7f00800fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.450+0000 7f0085fe6700 1 -- 172.21.15.121:0/2137203679 shutdown_connections 2024-04-05T21:30:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.450+0000 7f0085fe6700 1 -- 172.21.15.121:0/2137203679 wait complete. 2024-04-05T21:30:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.451+0000 7f0085fe6700 1 Processor -- start 2024-04-05T21:30:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.451+0000 7f0085fe6700 1 -- start start 2024-04-05T21:30:24.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.451+0000 7f0085fe6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00800ff400 0x7f0080196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:24.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.451+0000 7f0085fe6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00800ffdb0 0x7f0080196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:24.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.451+0000 7f0085fe6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00801971c0 con 0x7f00800ff400 2024-04-05T21:30:24.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.451+0000 7f0085fe6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0080197300 con 0x7f00800ffdb0 2024-04-05T21:30:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.451+0000 7f007f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00800ff400 0x7f0080196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.451+0000 7f007f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00800ff400 0x7f0080196630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41004/0 (socket says 172.21.15.121:41004) 2024-04-05T21:30:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.451+0000 7f007f7fe700 1 -- 172.21.15.121:0/1908534990 learned_addr learned my addr 172.21.15.121:0/1908534990 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.451+0000 7f007effd700 1 --2- 172.21.15.121:0/1908534990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00800ffdb0 0x7f0080196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.452+0000 7f007f7fe700 1 -- 172.21.15.121:0/1908534990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00800ffdb0 msgr2=0x7f0080196b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.452+0000 7f007f7fe700 1 --2- 172.21.15.121:0/1908534990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00800ffdb0 0x7f0080196b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.452+0000 7f007f7fe700 1 -- 172.21.15.121:0/1908534990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00700095f0 con 0x7f00800ff400 2024-04-05T21:30:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.452+0000 7f007effd700 1 --2- 172.21.15.121:0/1908534990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00800ffdb0 0x7f0080196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:30:24.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.452+0000 7f007f7fe700 1 --2- 172.21.15.121:0/1908534990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00800ff400 0x7f0080196630 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0070006010 tx=0x7f007000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.452+0000 7f007cff9700 1 -- 172.21.15.121:0/1908534990 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f007001d070 con 0x7f00800ff400 2024-04-05T21:30:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.452+0000 7f007cff9700 1 -- 172.21.15.121:0/1908534990 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f007000f460 con 0x7f00800ff400 2024-04-05T21:30:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.452+0000 7f0085fe6700 1 -- 172.21.15.121:0/1908534990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f008019b0f0 con 0x7f00800ff400 2024-04-05T21:30:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.452+0000 7f007cff9700 1 -- 172.21.15.121:0/1908534990 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0070021640 con 0x7f00800ff400 2024-04-05T21:30:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.452+0000 7f0085fe6700 1 -- 172.21.15.121:0/1908534990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008019b5e0 con 0x7f00800ff400 2024-04-05T21:30:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.453+0000 7f0085fe6700 1 -- 172.21.15.121:0/1908534990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00801097b0 con 0x7f00800ff400 2024-04-05T21:30:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.453+0000 7f007cff9700 1 -- 172.21.15.121:0/1908534990 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f007002b440 con 0x7f00800ff400 2024-04-05T21:30:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.454+0000 7f007cff9700 1 --2- 172.21.15.121:0/1908534990 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f006806cce0 0x7f006806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.454+0000 7f007cff9700 1 -- 172.21.15.121:0/1908534990 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f007008d710 con 0x7f00800ff400 2024-04-05T21:30:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.454+0000 7f007effd700 1 --2- 172.21.15.121:0/1908534990 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f006806cce0 0x7f006806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:24.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.454+0000 7f007effd700 1 --2- 172.21.15.121:0/1908534990 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f006806cce0 0x7f006806f1a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0074005fd0 tx=0x7f0074005e30 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:24.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.458+0000 7f007cff9700 1 -- 172.21.15.121:0/1908534990 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f0070057a70 con 0x7f00800ff400 2024-04-05T21:30:24.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:24 smithi121 ceph-mon[39723]: Upgrade: Updating mon.smithi121 2024-04-05T21:30:24.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:24 smithi121 ceph-mon[39723]: Deploying daemon mon.smithi121 on smithi121 2024-04-05T21:30:24.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:24 smithi121 ceph-mon[39723]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:24.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.741+0000 7f0085fe6700 1 -- 172.21.15.121:0/1908534990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f008004ea90 con 0x7f00800ff400 2024-04-05T21:30:24.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.741+0000 7f007cff9700 1 -- 172.21.15.121:0/1908534990 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+679 (secure 0 0 0) 0x7f007005b090 con 0x7f00800ff400 2024-04-05T21:30:24.742 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:30:24.742 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:30:24.742 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2 2024-04-05T21:30:24.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:30:24.742 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:30:24.742 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:30:24.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:30:24.743 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:30:24.743 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-05T21:30:24.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:30:24.743 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:30:24.743 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:30:24.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:30:24.743 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:30:24.743 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 12, 2024-04-05T21:30:24.743 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:30:24.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:30:24.743 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:30:24.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.743+0000 7f0085fe6700 1 -- 172.21.15.121:0/1908534990 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f006806cce0 msgr2=0x7f006806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:24.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.743+0000 7f0085fe6700 1 --2- 172.21.15.121:0/1908534990 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f006806cce0 0x7f006806f1a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0074005fd0 tx=0x7f0074005e30 comp rx=0 tx=0).stop 2024-04-05T21:30:24.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.743+0000 7f0085fe6700 1 -- 172.21.15.121:0/1908534990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00800ff400 msgr2=0x7f0080196630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:24.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.743+0000 7f0085fe6700 1 --2- 172.21.15.121:0/1908534990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00800ff400 0x7f0080196630 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0070006010 tx=0x7f007000be20 comp rx=0 tx=0).stop 2024-04-05T21:30:24.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.743+0000 7f0085fe6700 1 -- 172.21.15.121:0/1908534990 shutdown_connections 2024-04-05T21:30:24.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.743+0000 7f0085fe6700 1 --2- 172.21.15.121:0/1908534990 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f006806cce0 0x7f006806f1a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.743+0000 7f0085fe6700 1 --2- 172.21.15.121:0/1908534990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00800ff400 0x7f0080196630 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.743+0000 7f0085fe6700 1 --2- 172.21.15.121:0/1908534990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00800ffdb0 0x7f0080196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.743+0000 7f0085fe6700 1 -- 172.21.15.121:0/1908534990 >> 172.21.15.121:0/1908534990 conn(0x7f0080075e30 msgr2=0x7f0080102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:24.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.743+0000 7f0085fe6700 1 -- 172.21.15.121:0/1908534990 shutdown_connections 2024-04-05T21:30:24.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.743+0000 7f0085fe6700 1 -- 172.21.15.121:0/1908534990 wait complete. 2024-04-05T21:30:24.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:24 smithi188 ceph-mon[49715]: Upgrade: Updating mon.smithi121 2024-04-05T21:30:24.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:24 smithi188 ceph-mon[49715]: Deploying daemon mon.smithi121 on smithi121 2024-04-05T21:30:24.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:24 smithi188 ceph-mon[49715]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:24.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.847+0000 7f99e98cc700 1 -- 172.21.15.121:0/1605006199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e41001c0 msgr2=0x7f99e4100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:24.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.847+0000 7f99e98cc700 1 --2- 172.21.15.121:0/1605006199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e41001c0 0x7f99e4100640 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f99d4009910 tx=0x7f99d4009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:24.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.847+0000 7f99e98cc700 1 -- 172.21.15.121:0/1605006199 shutdown_connections 2024-04-05T21:30:24.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.847+0000 7f99e98cc700 1 --2- 172.21.15.121:0/1605006199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e41001c0 0x7f99e4100640 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.847+0000 7f99e98cc700 1 --2- 172.21.15.121:0/1605006199 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e4104a10 0x7f99e4104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.847+0000 7f99e98cc700 1 -- 172.21.15.121:0/1605006199 >> 172.21.15.121:0/1605006199 conn(0x7f99e4075e30 msgr2=0x7f99e40fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:24.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.847+0000 7f99e98cc700 1 -- 172.21.15.121:0/1605006199 shutdown_connections 2024-04-05T21:30:24.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.847+0000 7f99e98cc700 1 -- 172.21.15.121:0/1605006199 wait complete. 2024-04-05T21:30:24.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e98cc700 1 Processor -- start 2024-04-05T21:30:24.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e98cc700 1 -- start start 2024-04-05T21:30:24.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e98cc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e41001c0 0x7f99e41966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:24.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e98cc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e4104a10 0x7f99e4196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e98cc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e41972a0 con 0x7f99e4104a10 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e98cc700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e4199fd0 con 0x7f99e41001c0 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e27fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e4104a10 0x7f99e4196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e27fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e4104a10 0x7f99e4196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41026/0 (socket says 172.21.15.121:41026) 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e27fc700 1 -- 172.21.15.121:0/1260119112 learned_addr learned my addr 172.21.15.121:0/1260119112 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e2ffd700 1 --2- 172.21.15.121:0/1260119112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e41001c0 0x7f99e41966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e27fc700 1 -- 172.21.15.121:0/1260119112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e41001c0 msgr2=0x7f99e41966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e27fc700 1 --2- 172.21.15.121:0/1260119112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e41001c0 0x7f99e41966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e27fc700 1 -- 172.21.15.121:0/1260119112 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d40095f0 con 0x7f99e4104a10 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.848+0000 7f99e27fc700 1 --2- 172.21.15.121:0/1260119112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e4104a10 0x7f99e4196c10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f99d400bde0 tx=0x7f99d400bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.849+0000 7f99e88ca700 1 -- 172.21.15.121:0/1260119112 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99d401d070 con 0x7f99e4104a10 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.849+0000 7f99e98cc700 1 -- 172.21.15.121:0/1260119112 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e419a250 con 0x7f99e4104a10 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.849+0000 7f99e88ca700 1 -- 172.21.15.121:0/1260119112 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99d4005dc0 con 0x7f99e4104a10 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.849+0000 7f99e88ca700 1 -- 172.21.15.121:0/1260119112 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99d4003ad0 con 0x7f99e4104a10 2024-04-05T21:30:24.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.849+0000 7f99e98cc700 1 -- 172.21.15.121:0/1260119112 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e419a6e0 con 0x7f99e4104a10 2024-04-05T21:30:24.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.850+0000 7f99e98cc700 1 -- 172.21.15.121:0/1260119112 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e4108f30 con 0x7f99e4104a10 2024-04-05T21:30:24.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.850+0000 7f99e88ca700 1 -- 172.21.15.121:0/1260119112 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f99d4003d50 con 0x7f99e4104a10 2024-04-05T21:30:24.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.850+0000 7f99e88ca700 1 --2- 172.21.15.121:0/1260119112 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f99cc06cce0 0x7f99cc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:24.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.851+0000 7f99e88ca700 1 -- 172.21.15.121:0/1260119112 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f99d408dd10 con 0x7f99e4104a10 2024-04-05T21:30:24.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.851+0000 7f99e2ffd700 1 --2- 172.21.15.121:0/1260119112 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f99cc06cce0 0x7f99cc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:24.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.851+0000 7f99e2ffd700 1 --2- 172.21.15.121:0/1260119112 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f99cc06cce0 0x7f99cc06f1a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009bf0 tx=0x7f99d8009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:24.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:24.855+0000 7f99e88ca700 1 -- 172.21.15.121:0/1260119112 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f99d4057e30 con 0x7f99e4104a10 2024-04-05T21:30:25.124 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:25 smithi121 systemd[1]: Stopping Ceph mon.smithi121 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:30:25.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.123+0000 7f99e98cc700 1 -- 172.21.15.121:0/1260119112 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f99e4197ad0 con 0x7f99e4104a10 2024-04-05T21:30:25.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.124+0000 7f99e88ca700 1 -- 172.21.15.121:0/1260119112 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+1887 (secure 0 0 0) 0x7f99d4026070 con 0x7f99e4104a10 2024-04-05T21:30:25.125 INFO:teuthology.orchestra.run.smithi121.stdout:e20 2024-04-05T21:30:25.125 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:30:25.125 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:30:25.125 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:30:25.125 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:30:25.125 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:30:25.125 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:30:25.125 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 20 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:27:26.656630+0000 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 46 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:30:25.126 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:up {0=14476,1=24255} 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:failed 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{0:14476} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{0:14488} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/4279084130,v1:172.21.15.121:6827/4279084130] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{1:24255} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{1:24265} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:25.127 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:30:25.128 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:30:25.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.126+0000 7f99e98cc700 1 -- 172.21.15.121:0/1260119112 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f99cc06cce0 msgr2=0x7f99cc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:25.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.126+0000 7f99e98cc700 1 --2- 172.21.15.121:0/1260119112 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f99cc06cce0 0x7f99cc06f1a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009bf0 tx=0x7f99d8009260 comp rx=0 tx=0).stop 2024-04-05T21:30:25.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.127+0000 7f99e98cc700 1 -- 172.21.15.121:0/1260119112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e4104a10 msgr2=0x7f99e4196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:25.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.127+0000 7f99e98cc700 1 --2- 172.21.15.121:0/1260119112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e4104a10 0x7f99e4196c10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f99d400bde0 tx=0x7f99d400bec0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.127+0000 7f99e98cc700 1 -- 172.21.15.121:0/1260119112 shutdown_connections 2024-04-05T21:30:25.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.127+0000 7f99e98cc700 1 --2- 172.21.15.121:0/1260119112 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f99cc06cce0 0x7f99cc06f1a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.127+0000 7f99e98cc700 1 --2- 172.21.15.121:0/1260119112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e41001c0 0x7f99e41966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.127+0000 7f99e98cc700 1 --2- 172.21.15.121:0/1260119112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e4104a10 0x7f99e4196c10 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.127+0000 7f99e98cc700 1 -- 172.21.15.121:0/1260119112 >> 172.21.15.121:0/1260119112 conn(0x7f99e4075e30 msgr2=0x7f99e40ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:25.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.127+0000 7f99e98cc700 1 -- 172.21.15.121:0/1260119112 shutdown_connections 2024-04-05T21:30:25.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.127+0000 7f99e98cc700 1 -- 172.21.15.121:0/1260119112 wait complete. 2024-04-05T21:30:25.129 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 20 2024-04-05T21:30:25.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.241+0000 7f3289b90700 1 -- 172.21.15.121:0/1607484854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3284100dc0 msgr2=0x7f32841011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:25.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.241+0000 7f3289b90700 1 --2- 172.21.15.121:0/1607484854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3284100dc0 0x7f32841011a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3278009910 tx=0x7f3278009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:25.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.241+0000 7f3289b90700 1 -- 172.21.15.121:0/1607484854 shutdown_connections 2024-04-05T21:30:25.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.241+0000 7f3289b90700 1 --2- 172.21.15.121:0/1607484854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3284101770 0x7f32841057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.241+0000 7f3289b90700 1 --2- 172.21.15.121:0/1607484854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3284100dc0 0x7f32841011a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.241+0000 7f3289b90700 1 -- 172.21.15.121:0/1607484854 >> 172.21.15.121:0/1607484854 conn(0x7f32840fc650 msgr2=0x7f32840fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:25.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.242+0000 7f3289b90700 1 -- 172.21.15.121:0/1607484854 shutdown_connections 2024-04-05T21:30:25.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.242+0000 7f3289b90700 1 -- 172.21.15.121:0/1607484854 wait complete. 2024-04-05T21:30:25.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.242+0000 7f3289b90700 1 Processor -- start 2024-04-05T21:30:25.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f3289b90700 1 -- start start 2024-04-05T21:30:25.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f3289b90700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3284100dc0 0x7f32841966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:25.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f3289b90700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3284101770 0x7f3284196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:25.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f3289b90700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32841972a0 con 0x7f3284101770 2024-04-05T21:30:25.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f3289b90700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3284199fd0 con 0x7f3284100dc0 2024-04-05T21:30:25.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f3282ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3284101770 0x7f3284196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:25.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f3282ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3284101770 0x7f3284196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41044/0 (socket says 172.21.15.121:41044) 2024-04-05T21:30:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f3282ffd700 1 -- 172.21.15.121:0/3738916554 learned_addr learned my addr 172.21.15.121:0/3738916554 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f32837fe700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3284100dc0 0x7f32841966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f3282ffd700 1 -- 172.21.15.121:0/3738916554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3284100dc0 msgr2=0x7f32841966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f3282ffd700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3284100dc0 0x7f32841966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f3282ffd700 1 -- 172.21.15.121:0/3738916554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780095f0 con 0x7f3284101770 2024-04-05T21:30:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f3282ffd700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3284101770 0x7f3284196c10 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f327400c8f0 tx=0x7f327400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.243+0000 7f3280ff9700 1 -- 172.21.15.121:0/3738916554 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32740078a0 con 0x7f3284101770 2024-04-05T21:30:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.244+0000 7f3280ff9700 1 -- 172.21.15.121:0/3738916554 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f327400f450 con 0x7f3284101770 2024-04-05T21:30:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.244+0000 7f3289b90700 1 -- 172.21.15.121:0/3738916554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328419a2b0 con 0x7f3284101770 2024-04-05T21:30:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.244+0000 7f3280ff9700 1 -- 172.21.15.121:0/3738916554 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3274007ea0 con 0x7f3284101770 2024-04-05T21:30:25.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.244+0000 7f3289b90700 1 -- 172.21.15.121:0/3738916554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328419a800 con 0x7f3284101770 2024-04-05T21:30:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.245+0000 7f3289b90700 1 -- 172.21.15.121:0/3738916554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3284109160 con 0x7f3284101770 2024-04-05T21:30:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.245+0000 7f3280ff9700 1 -- 172.21.15.121:0/3738916554 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f327400f5c0 con 0x7f3284101770 2024-04-05T21:30:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.245+0000 7f3280ff9700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f326c06cce0 0x7f326c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:25.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.246+0000 7f3280ff9700 1 -- 172.21.15.121:0/3738916554 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f327408c410 con 0x7f3284101770 2024-04-05T21:30:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.246+0000 7f32837fe700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f326c06cce0 0x7f326c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:25.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.246+0000 7f32837fe700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f326c06cce0 0x7f326c06f1a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3278006010 tx=0x7f327801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:25.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.250+0000 7f3280ff9700 1 -- 172.21.15.121:0/3738916554 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f3274056530 con 0x7f3284101770 2024-04-05T21:30:25.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.348+0000 7f3282ffd700 1 -- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3284101770 msgr2=0x7f3284196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-05T21:30:25.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.348+0000 7f3282ffd700 1 -- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3284101770 msgr2=0x7f3284196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-05T21:30:25.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.348+0000 7f3282ffd700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3284101770 0x7f3284196c10 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f327400c8f0 tx=0x7f327400cc00 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-05T21:30:25.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.348+0000 7f3282ffd700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3284101770 0x7f3284196c10 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f327400c8f0 tx=0x7f327400cc00 comp rx=0 tx=0).stop 2024-04-05T21:30:25.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.348+0000 7f3280ff9700 1 -- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3284101770 msgr2=0x7f3284196c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-05T21:30:25.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.348+0000 7f3280ff9700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3284101770 0x7f3284196c10 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.348+0000 7f3280ff9700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f326c074940 0x7f326c076d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:25.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.348+0000 7f3280ff9700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3284100dc0 0x7f326c079580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:25.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.348+0000 7f3280ff9700 1 -- 172.21.15.121:0/3738916554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c07e310 con 0x7f326c074940 2024-04-05T21:30:25.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.348+0000 7f3280ff9700 1 -- 172.21.15.121:0/3738916554 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c07e450 con 0x7f3284100dc0 2024-04-05T21:30:25.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.348+0000 7f3282ffd700 1 -- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f326c074940 msgr2=0x7f326c076d30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-04-05T21:30:25.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.348+0000 7f3282ffd700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f326c074940 0x7f326c076d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T21:30:25.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.349+0000 7f3283fff700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3284100dc0 0x7f326c079580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:25.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.349+0000 7f3283fff700 1 -- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f326c074940 msgr2=0x7f326c076d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T21:30:25.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.349+0000 7f3283fff700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f326c074940 0x7f326c076d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.349+0000 7f3283fff700 1 -- 172.21.15.121:0/3738916554 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7f326c07e5f0 con 0x7f3284100dc0 2024-04-05T21:30:25.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.349+0000 7f3283fff700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3284100dc0 0x7f326c079580 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f327c009dc0 tx=0x7f327400a3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:25.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.350+0000 7f3280ff9700 1 -- 172.21.15.121:0/3738916554 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f327c0098b0 con 0x7f3284100dc0 2024-04-05T21:30:25.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.350+0000 7f3280ff9700 1 -- 172.21.15.121:0/3738916554 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f327c00bd30 con 0x7f3284100dc0 2024-04-05T21:30:25.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.352+0000 7f3280ff9700 1 -- 172.21.15.121:0/3738916554 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f327c015420 con 0x7f3284100dc0 2024-04-05T21:30:25.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:25 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[39700]: 2024-04-05T21:30:25.347+0000 7f995803e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi121 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T21:30:25.471 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:25 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[39700]: 2024-04-05T21:30:25.347+0000 7f995803e700 -1 mon.smithi121@0(leader) e2 *** Got Signal Terminated *** 2024-04-05T21:30:25.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.473+0000 7f3289b90700 1 -- 172.21.15.121:0/3738916554 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3264000d50 con 0x7f326c06cce0 2024-04-05T21:30:25.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.474+0000 7f3280ff9700 1 -- 172.21.15.121:0/3738916554 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f3264000d50 con 0x7f326c06cce0 2024-04-05T21:30:25.475 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:30:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:30:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:30:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:30:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:30:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr" 2024-04-05T21:30:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:30:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "2/23 daemons upgraded", 2024-04-05T21:30:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mon daemons", 2024-04-05T21:30:25.476 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:30:25.476 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:30:25.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.477+0000 7f3289b90700 1 -- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f326c06cce0 msgr2=0x7f326c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:25.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.477+0000 7f3289b90700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f326c06cce0 0x7f326c06f1a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3278006010 tx=0x7f327801a040 comp rx=0 tx=0).stop 2024-04-05T21:30:25.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.477+0000 7f3289b90700 1 -- 172.21.15.121:0/3738916554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3284100dc0 msgr2=0x7f326c079580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:25.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.477+0000 7f3289b90700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3284100dc0 0x7f326c079580 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f327c009dc0 tx=0x7f327400a3b0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.478+0000 7f3289b90700 1 -- 172.21.15.121:0/3738916554 shutdown_connections 2024-04-05T21:30:25.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.478+0000 7f3289b90700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3284100dc0 0x7f326c079580 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.478+0000 7f3289b90700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f326c074940 0x7f326c076d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.478+0000 7f3289b90700 1 --2- 172.21.15.121:0/3738916554 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f326c06cce0 0x7f326c06f1a0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3278006010 tx=0x7f327801a040 comp rx=0 tx=0).stop 2024-04-05T21:30:25.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.478+0000 7f3289b90700 1 -- 172.21.15.121:0/3738916554 >> 172.21.15.121:0/3738916554 conn(0x7f32840fc650 msgr2=0x7f32840fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:25.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.478+0000 7f3289b90700 1 -- 172.21.15.121:0/3738916554 shutdown_connections 2024-04-05T21:30:25.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.478+0000 7f3289b90700 1 -- 172.21.15.121:0/3738916554 wait complete. 2024-04-05T21:30:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.586+0000 7f25174ee700 1 -- 172.21.15.121:0/1336278096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25100ffbd0 msgr2=0x7f2510100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.586+0000 7f25174ee700 1 --2- 172.21.15.121:0/1336278096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25100ffbd0 0x7f2510100050 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2500009910 tx=0x7f2500009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.586+0000 7f25174ee700 1 -- 172.21.15.121:0/1336278096 shutdown_connections 2024-04-05T21:30:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.586+0000 7f25174ee700 1 --2- 172.21.15.121:0/1336278096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25100ffbd0 0x7f2510100050 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.586+0000 7f25174ee700 1 --2- 172.21.15.121:0/1336278096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2510105a40 0x7f2510105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.586+0000 7f25174ee700 1 -- 172.21.15.121:0/1336278096 >> 172.21.15.121:0/1336278096 conn(0x7f25100755a0 msgr2=0x7f25100759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.586+0000 7f25174ee700 1 -- 172.21.15.121:0/1336278096 shutdown_connections 2024-04-05T21:30:25.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.586+0000 7f25174ee700 1 -- 172.21.15.121:0/1336278096 wait complete. 2024-04-05T21:30:25.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.586+0000 7f25174ee700 1 Processor -- start 2024-04-05T21:30:25.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.586+0000 7f25174ee700 1 -- start start 2024-04-05T21:30:25.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.587+0000 7f25174ee700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25100ffbd0 0x7f251019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:25.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.587+0000 7f25174ee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2510105a40 0x7f251019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:25.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.587+0000 7f25174ee700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251019d130 con 0x7f2510105a40 2024-04-05T21:30:25.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.587+0000 7f25174ee700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25101965e0 con 0x7f25100ffbd0 2024-04-05T21:30:25.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.587+0000 7f2514a89700 1 -- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2510105a40 msgr2=0x7f251019caa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:3300/0 2024-04-05T21:30:25.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.587+0000 7f2514a89700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2510105a40 0x7f251019caa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-05T21:30:25.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.587+0000 7f251528a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25100ffbd0 0x7f251019c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:25.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.587+0000 7f251528a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25100ffbd0 0x7f251019c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.121:33614/0 (socket says 172.21.15.121:33614) 2024-04-05T21:30:25.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.587+0000 7f251528a700 1 -- 172.21.15.121:0/1533716330 learned_addr learned my addr 172.21.15.121:0/1533716330 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:25.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.587+0000 7f251528a700 1 -- 172.21.15.121:0/1533716330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2510105a40 msgr2=0x7f251019caa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-05T21:30:25.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.587+0000 7f251528a700 1 --2- 172.21.15.121:0/1533716330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2510105a40 0x7f251019caa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.587+0000 7f251528a700 1 -- 172.21.15.121:0/1533716330 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25000095f0 con 0x7f25100ffbd0 2024-04-05T21:30:25.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.587+0000 7f251528a700 1 --2- 172.21.15.121:0/1533716330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25100ffbd0 0x7f251019c560 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f250c00b700 tx=0x7f250c00bac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:25.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.588+0000 7f25067fc700 1 -- 172.21.15.121:0/1533716330 <== mon.1 v2:172.21.15.188:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f250c011800 con 0x7f25100ffbd0 2024-04-05T21:30:25.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.588+0000 7f25174ee700 1 -- 172.21.15.121:0/1533716330 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25101968c0 con 0x7f25100ffbd0 2024-04-05T21:30:25.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.588+0000 7f25067fc700 1 -- 172.21.15.121:0/1533716330 <== mon.1 v2:172.21.15.188:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f250c011e00 con 0x7f25100ffbd0 2024-04-05T21:30:25.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.588+0000 7f25067fc700 1 -- 172.21.15.121:0/1533716330 <== mon.1 v2:172.21.15.188:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f250c00f2e0 con 0x7f25100ffbd0 2024-04-05T21:30:25.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.588+0000 7f25174ee700 1 -- 172.21.15.121:0/1533716330 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2510196e10 con 0x7f25100ffbd0 2024-04-05T21:30:25.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.588+0000 7f25174ee700 1 -- 172.21.15.121:0/1533716330 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251004ea90 con 0x7f25100ffbd0 2024-04-05T21:30:25.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.589+0000 7f25067fc700 1 -- 172.21.15.121:0/1533716330 <== mon.1 v2:172.21.15.188:3300/0 4 ==== mgrmap(e 33) v1 ==== 90844+0+0 (secure 0 0 0) 0x7f250c010380 con 0x7f25100ffbd0 2024-04-05T21:30:25.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.590+0000 7f25067fc700 1 --2- 172.21.15.121:0/1533716330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f24fc06cc40 0x7f24fc06f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:25.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.590+0000 7f25067fc700 1 -- 172.21.15.121:0/1533716330 <== mon.1 v2:172.21.15.188:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f250c08c260 con 0x7f25100ffbd0 2024-04-05T21:30:25.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.590+0000 7f2514a89700 1 --2- 172.21.15.121:0/1533716330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f24fc06cc40 0x7f24fc06f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:25.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.590+0000 7f2514a89700 1 --2- 172.21.15.121:0/1533716330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f24fc06cc40 0x7f24fc06f100 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f250000b540 tx=0x7f2500009fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:25.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.594+0000 7f25067fc700 1 -- 172.21.15.121:0/1533716330 <== mon.1 v2:172.21.15.188:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182676 (secure 0 0 0) 0x7f250c059df0 con 0x7f25100ffbd0 2024-04-05T21:30:25.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.876+0000 7f25174ee700 1 -- 172.21.15.121:0/1533716330 --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2510197af0 con 0x7f25100ffbd0 2024-04-05T21:30:25.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.877+0000 7f25067fc700 1 -- 172.21.15.121:0/1533716330 <== mon.1 v2:172.21.15.188:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f250c059980 con 0x7f25100ffbd0 2024-04-05T21:30:25.878 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_OK 2024-04-05T21:30:25.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.879+0000 7f25174ee700 1 -- 172.21.15.121:0/1533716330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f24fc06cc40 msgr2=0x7f24fc06f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:25.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.879+0000 7f25174ee700 1 --2- 172.21.15.121:0/1533716330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f24fc06cc40 0x7f24fc06f100 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f250000b540 tx=0x7f2500009fd0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.879+0000 7f25174ee700 1 -- 172.21.15.121:0/1533716330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25100ffbd0 msgr2=0x7f251019c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:25.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.879+0000 7f25174ee700 1 --2- 172.21.15.121:0/1533716330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25100ffbd0 0x7f251019c560 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f250c00b700 tx=0x7f250c00bac0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.879+0000 7f25174ee700 1 -- 172.21.15.121:0/1533716330 shutdown_connections 2024-04-05T21:30:25.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.879+0000 7f25174ee700 1 --2- 172.21.15.121:0/1533716330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f24fc06cc40 0x7f24fc06f100 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.879+0000 7f25174ee700 1 --2- 172.21.15.121:0/1533716330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25100ffbd0 0x7f251019c560 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.879+0000 7f25174ee700 1 --2- 172.21.15.121:0/1533716330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2510105a40 0x7f251019caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:25.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.879+0000 7f25174ee700 1 -- 172.21.15.121:0/1533716330 >> 172.21.15.121:0/1533716330 conn(0x7f25100755a0 msgr2=0x7f2510109110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:25.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.879+0000 7f25174ee700 1 -- 172.21.15.121:0/1533716330 shutdown_connections 2024-04-05T21:30:25.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:25.879+0000 7f25174ee700 1 -- 172.21.15.121:0/1533716330 wait complete. 2024-04-05T21:30:26.206 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:25 smithi121 bash[145617]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121 2024-04-05T21:30:26.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 systemd[1]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mon.smithi121.service: Succeeded. 2024-04-05T21:30:26.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 systemd[1]: Stopped Ceph mon.smithi121 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:30:26.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 systemd[1]: Starting Ceph mon.smithi121 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:30:27.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: set uid:gid to 167:167 (ceph:ceph) 2024-04-05T21:30:27.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable), process ceph-mon, pid 8 2024-04-05T21:30:27.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: pidfile_write: ignore empty --pid-file 2024-04-05T21:30:27.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: load: jerasure load: lrc 2024-04-05T21:30:27.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: RocksDB version: 7.9.2 2024-04-05T21:30:27.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Git sha 0 2024-04-05T21:30:27.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Compile date 2024-04-05 12:53:35 2024-04-05T21:30:27.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: DB SUMMARY 2024-04-05T21:30:27.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: DB Session ID: HYGBQPKRDW80UUDPOPLL 2024-04-05T21:30:27.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: CURRENT file: CURRENT 2024-04-05T21:30:27.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: IDENTITY file: IDENTITY 2024-04-05T21:30:27.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1517 Bytes 2024-04-05T21:30:27.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi121/store.db dir, Total Num: 1, files: 000032.sst 2024-04-05T21:30:27.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi121/store.db: 000030.log size: 5097835 ; 2024-04-05T21:30:27.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.error_if_exists: 0 2024-04-05T21:30:27.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.create_if_missing: 0 2024-04-05T21:30:27.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.paranoid_checks: 1 2024-04-05T21:30:27.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-05T21:30:27.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-05T21:30:27.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-05T21:30:27.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.env: 0x563a9e5d57a0 2024-04-05T21:30:27.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.fs: PosixFileSystem 2024-04-05T21:30:27.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.info_log: 0x563aa11ed3a0 2024-04-05T21:30:27.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_file_opening_threads: 16 2024-04-05T21:30:27.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.statistics: (nil) 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.use_fsync: 0 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_log_file_size: 0 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.keep_log_file_num: 1000 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.recycle_log_file_num: 0 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.allow_fallocate: 1 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.allow_mmap_reads: 0 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.allow_mmap_writes: 0 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.use_direct_reads: 0 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.create_missing_column_families: 0 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.db_log_dir: 2024-04-05T21:30:27.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.wal_dir: 2024-04-05T21:30:27.245 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-05T21:30:27.245 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-05T21:30:27.245 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-05T21:30:27.245 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-05T21:30:27.245 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-05T21:30:27.245 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-05T21:30:27.245 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.advise_random_on_open: 1 2024-04-05T21:30:27.246 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.db_write_buffer_size: 0 2024-04-05T21:30:27.246 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.write_buffer_manager: 0x563aa0462140 2024-04-05T21:30:27.246 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-05T21:30:27.246 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-05T21:30:27.246 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-05T21:30:27.246 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.rate_limiter: (nil) 2024-04-05T21:30:27.246 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-05T21:30:27.246 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.wal_recovery_mode: 2 2024-04-05T21:30:27.246 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.enable_thread_tracking: 0 2024-04-05T21:30:27.246 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.enable_pipelined_write: 0 2024-04-05T21:30:27.246 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.unordered_write: 0 2024-04-05T21:30:27.246 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-05T21:30:27.246 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.row_cache: None 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.wal_filter: None 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.allow_ingest_behind: 0 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.two_write_queues: 0 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.manual_wal_flush: 0 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.wal_compression: 0 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.atomic_flush: 0 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-05T21:30:27.247 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.log_readahead_size: 0 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.best_efforts_recovery: 0 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.allow_data_in_errors: 0 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.db_host_id: __hostname__ 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_background_jobs: 2 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_background_compactions: -1 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_subcompactions: 1 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_total_wal_size: 0 2024-04-05T21:30:27.248 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_open_files: -1 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.bytes_per_sync: 0 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compaction_readahead_size: 0 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_background_flushes: -1 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Compression algorithms supported: 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: kZSTD supported: 0 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: kXpressCompression supported: 0 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: kLZ4HCCompression supported: 1 2024-04-05T21:30:27.249 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: kZlibCompression supported: 1 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: kSnappyCompression supported: 1 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: kLZ4Compression supported: 1 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: kBZip2Compression supported: 0 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi121/store.db/MANIFEST-000015 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.merge_operator: 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compaction_filter: None 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compaction_filter_factory: None 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.sst_partitioner_factory: None 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-05T21:30:27.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563aa11ed4c0) 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: cache_index_and_filter_blocks: 1 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: pin_top_level_index_and_filter: 1 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: index_type: 0 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: data_block_index_type: 0 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: index_shortening: 1 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: checksum: 4 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: no_block_cache: 0 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_cache: 0x563aa05031f0 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_cache_name: BinnedLRUCache 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_cache_options: 2024-04-05T21:30:27.251 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: capacity : 536870912 2024-04-05T21:30:27.252 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: num_shard_bits : 4 2024-04-05T21:30:27.252 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: strict_capacity_limit : 0 2024-04-05T21:30:27.252 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: high_pri_pool_ratio: 0.000 2024-04-05T21:30:27.252 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_cache_compressed: (nil) 2024-04-05T21:30:27.252 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: persistent_cache: (nil) 2024-04-05T21:30:27.252 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_size: 4096 2024-04-05T21:30:27.252 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_size_deviation: 10 2024-04-05T21:30:27.252 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_restart_interval: 16 2024-04-05T21:30:27.252 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: index_block_restart_interval: 1 2024-04-05T21:30:27.252 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: metadata_block_size: 4096 2024-04-05T21:30:27.252 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: partition_filters: 0 2024-04-05T21:30:27.252 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: use_delta_encoding: 1 2024-04-05T21:30:27.252 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: filter_policy: bloomfilter 2024-04-05T21:30:27.253 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: whole_key_filtering: 1 2024-04-05T21:30:27.253 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: verify_compression: 0 2024-04-05T21:30:27.253 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: read_amp_bytes_per_bit: 0 2024-04-05T21:30:27.253 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: format_version: 5 2024-04-05T21:30:27.253 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: enable_index_compression: 1 2024-04-05T21:30:27.253 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: block_align: 0 2024-04-05T21:30:27.253 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: max_auto_readahead_size: 262144 2024-04-05T21:30:27.253 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: prepopulate_block_cache: 0 2024-04-05T21:30:27.253 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: initial_auto_readahead_size: 8192 2024-04-05T21:30:27.253 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: num_file_reads_for_auto_readahead: 2 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.write_buffer_size: 33554432 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_write_buffer_number: 2 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compression: NoCompression 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.bottommost_compression: Disabled 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.prefix_extractor: nullptr 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.num_levels: 7 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-05T21:30:27.254 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-05T21:30:27.255 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-05T21:30:27.255 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-05T21:30:27.255 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-05T21:30:27.255 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-05T21:30:27.255 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-05T21:30:27.255 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-05T21:30:27.255 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compression_opts.level: 32767 2024-04-05T21:30:27.255 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compression_opts.strategy: 0 2024-04-05T21:30:27.255 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-05T21:30:27.255 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-05T21:30:27.255 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-05T21:30:27.255 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-05T21:30:27.255 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compression_opts.enabled: false 2024-04-05T21:30:27.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-05T21:30:27.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-05T21:30:27.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-05T21:30:27.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-05T21:30:27.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.target_file_size_base: 67108864 2024-04-05T21:30:27.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-05T21:30:27.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-05T21:30:27.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-05T21:30:27.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-05T21:30:27.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-05T21:30:27.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-05T21:30:27.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.arena_block_size: 1048576 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.disable_auto_compactions: 0 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-05T21:30:27.257 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.table_properties_collectors: 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.inplace_update_support: 0 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.bloom_locality: 0 2024-04-05T21:30:27.258 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.max_successive_merges: 0 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.paranoid_file_checks: 0 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.force_consistency_checks: 1 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.report_bg_io_stats: 0 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.ttl: 2592000 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.enable_blob_files: false 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.min_blob_size: 0 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.blob_file_size: 268435456 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-05T21:30:27.259 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.blob_file_starting_level: 0 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi121/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 34, last_sequence is 12023, log_number is 30,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 30 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 30 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 31f70f03-b4f1-420b-ad02-7abefe01d6b5 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712352626910994, "job": 1, "event": "recovery_started", "wal_files": [30]} 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #30 mode 2 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712352626926847, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 35, "file_size": 4523817, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12024, "largest_seqno": 15499, "table_properties": {"data_size": 4510313, "index_size": 8579, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3909, "raw_key_size": 34380, "raw_average_key_size": 22, "raw_value_size": 4479565, "raw_average_value_size": 2878, "num_data_blocks": 388, "num_entries": 1556, "num_filter_entries": 1556, "num_deletions": 259, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712352626, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "31f70f03-b4f1-420b-ad02-7abefe01d6b5", "db_session_id": "HYGBQPKRDW80UUDPOPLL", "orig_file_number": 35, "seqno_to_time_mapping": "N/A"}} 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712352626926970, "job": 1, "event": "recovery_finished"} 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: [db/version_set.cc:5047] Creating manifest 37 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-05T21:30:27.260 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi121/store.db/000030.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563aa05a2000 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: DB pointer 0x563aa0590000 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: ** DB Stats ** 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: ** Compaction Stats [default] ** 2024-04-05T21:30:27.261 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: L0 1/0 4.31 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 535.5 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: L6 1/0 4.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Sum 2/0 9.13 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 535.5 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 535.5 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: ** Compaction Stats [default] ** 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 535.5 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:30:27.262 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Cumulative compaction: 0.00 GB write, 233.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Interval compaction: 0.00 GB write, 233.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Block cache BinnedLRUCache@0x563aa05031f0#8 capacity: 512.00 MB usage: 13.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.03 KB,0.0007689%) IndexBlock(1,9.08 KB,0.00173151%) Misc(1,0.00 KB,0%) 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: starting mon.smithi121 rank 0 at public addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] at bind addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi121 fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:30:27.263 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: mon.smithi121@-1(???) e2 preinit fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: mon.smithi121@-1(???).mds e20 new map 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: mon.smithi121@-1(???).mds e20 print_map 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: e20 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: legacy client fscid: 1 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: Filesystem 'cephfs' (1) 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: fs_name cephfs 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: epoch 20 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: modified 2024-04-05T21:27:26.656630+0000 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: tableserver 0 2024-04-05T21:30:27.264 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: root 0 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: session_timeout 60 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: session_autoclose 300 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: max_file_size 1099511627776 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: max_xattr_size 65536 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: required_client_features {} 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: last_failure 0 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: last_failure_osd_epoch 46 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: max_mds 2 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: in 0,1 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: up {0=14476,1=24255} 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: failed 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: damaged 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: stopped 2024-04-05T21:30:27.265 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: data_pools [3] 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: metadata_pool 2 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: inline_data disabled 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: balancer 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: bal_rank_mask -1 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: standby_count_wanted 1 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: [mds.cephfs.smithi121.ifoxmw{0:14476} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: [mds.cephfs.smithi121.gxheqt{0:14488} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/4279084130,v1:172.21.15.121:6827/4279084130] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: [mds.cephfs.smithi188.bjmisu{1:24255} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: [mds.cephfs.smithi188.latnft{1:24265} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout: 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: mon.smithi121@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: mon.smithi121@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-05T21:30:27.266 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: mon.smithi121@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-05T21:30:27.267 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: mon.smithi121@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-05T21:30:27.267 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 ceph-mon[145859]: mon.smithi121@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-04-05T21:30:27.267 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 bash[145820]: 1a9b277ce0e7aab36a227fa90a02278186973dcf69390eaecd861e5a1e058f48 2024-04-05T21:30:27.267 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:26 smithi121 systemd[1]: Started Ceph mon.smithi121 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:30:30.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:30:30.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:30:30.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:30.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: from='client.? 172.21.15.121:0/1533716330' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:30:30.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: pgmap v91: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:30.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: pgmap v92: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:30.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: mon.smithi121 calling monitor election 2024-04-05T21:30:30.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: mon.smithi121 is new leader, mons smithi121,smithi188 in quorum (ranks 0,1) 2024-04-05T21:30:30.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: monmap e2: 2 mons at {smithi121=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],smithi188=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T21:30:30.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:30:30.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: osdmap e48: 6 total, 6 up, 6 in 2024-04-05T21:30:30.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: mgrmap e33: smithi121.etbquh(active, since 2m), standbys: smithi188.tgcuir 2024-04-05T21:30:30.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: overall HEALTH_OK 2024-04-05T21:30:30.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: from='mgr.14656 ' entity='' 2024-04-05T21:30:30.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:30 smithi188 ceph-mon[49715]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-04-05T21:30:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mon metadata", "id": "smithi188"}]: dispatch 2024-04-05T21:30:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: from='client.14828 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1533716330' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:30:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: pgmap v91: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: pgmap v92: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: mon.smithi121 calling monitor election 2024-04-05T21:30:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: mon.smithi121 is new leader, mons smithi121,smithi188 in quorum (ranks 0,1) 2024-04-05T21:30:30.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: monmap e2: 2 mons at {smithi121=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],smithi188=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T21:30:30.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:30:30.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: osdmap e48: 6 total, 6 up, 6 in 2024-04-05T21:30:30.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: mgrmap e33: smithi121.etbquh(active, since 2m), standbys: smithi188.tgcuir 2024-04-05T21:30:30.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: overall HEALTH_OK 2024-04-05T21:30:30.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: from='mgr.14656 ' entity='' 2024-04-05T21:30:30.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:30 smithi121 ceph-mon[145859]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:31.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:31 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:31.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:31 smithi188 ceph-mon[49715]: pgmap v93: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:31.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:31 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:30:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:31 smithi121 ceph-mon[145859]: pgmap v93: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:30:33.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:33 smithi121 ceph-mon[145859]: pgmap v94: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:33.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:33 smithi188 ceph-mon[49715]: pgmap v94: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:35 smithi121 ceph-mon[145859]: pgmap v95: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:35 smithi188 ceph-mon[49715]: pgmap v95: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:36.974 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:36 smithi188 ceph-mon[49715]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:36.974 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:36 smithi188 ceph-mon[49715]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:36.974 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:36 smithi188 ceph-mon[49715]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:36.974 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:36 smithi188 ceph-mon[49715]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:36.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:36 smithi121 ceph-mon[145859]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:36.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:36 smithi121 ceph-mon[145859]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:36.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:36 smithi121 ceph-mon[145859]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:36.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:36 smithi121 ceph-mon[145859]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:37.774 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:37 smithi121 ceph-mon[145859]: pgmap v96: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:38.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:37 smithi188 ceph-mon[49715]: pgmap v96: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:39.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:39 smithi121 ceph-mon[145859]: pgmap v97: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:40.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:39 smithi188 ceph-mon[49715]: pgmap v97: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:41.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:40 smithi121 ceph-mon[145859]: Detected new or changed devices on smithi121 2024-04-05T21:30:41.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:40 smithi121 ceph-mon[145859]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:41.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:40 smithi121 ceph-mon[145859]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:41.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:40 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:41.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:40 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:30:41.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:40 smithi121 ceph-mon[145859]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:41.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:40 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:41.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:40 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:30:41.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:40 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-05T21:30:41.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:40 smithi121 ceph-mon[145859]: pgmap v98: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:41.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:40 smithi188 ceph-mon[49715]: Detected new or changed devices on smithi121 2024-04-05T21:30:41.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:40 smithi188 ceph-mon[49715]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:41.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:40 smithi188 ceph-mon[49715]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:41.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:40 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:41.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:40 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:30:41.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:40 smithi188 ceph-mon[49715]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:41.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:40 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:41.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:40 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:30:41.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:40 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-05T21:30:41.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:40 smithi188 ceph-mon[49715]: pgmap v98: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:42 smithi121 ceph-mon[145859]: Upgrade: Updating mon.smithi188 2024-04-05T21:30:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:42 smithi121 ceph-mon[145859]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:42 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T21:30:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:42 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T21:30:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:42 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:42 smithi121 ceph-mon[145859]: Deploying daemon mon.smithi188 on smithi188 2024-04-05T21:30:43.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:42 smithi188 ceph-mon[49715]: Upgrade: Updating mon.smithi188 2024-04-05T21:30:43.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:42 smithi188 ceph-mon[49715]: from='mgr.14656 ' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:43.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:42 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-05T21:30:43.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:42 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-05T21:30:43.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:42 smithi188 ceph-mon[49715]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:43.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:42 smithi188 ceph-mon[49715]: Deploying daemon mon.smithi188 on smithi188 2024-04-05T21:30:43.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:43 smithi188 systemd[1]: Stopping Ceph mon.smithi188 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:30:43.886 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:43 smithi121 ceph-mon[145859]: pgmap v99: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:44.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:43 smithi188 ceph-mon[49715]: pgmap v99: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:44.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:43 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi188[49635]: 2024-04-05T21:30:43.766+0000 7fd5074e5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi188 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T21:30:44.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:43 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi188[49635]: 2024-04-05T21:30:43.766+0000 7fd5074e5700 -1 mon.smithi188@1(peon) e2 *** Got Signal Terminated *** 2024-04-05T21:30:44.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:44 smithi188 bash[107604]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi188 2024-04-05T21:30:45.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:44 smithi188 systemd[1]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mon.smithi188.service: Succeeded. 2024-04-05T21:30:45.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:44 smithi188 systemd[1]: Stopped Ceph mon.smithi188 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:30:45.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:44 smithi188 systemd[1]: Starting Ceph mon.smithi188 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:30:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: set uid:gid to 167:167 (ceph:ceph) 2024-04-05T21:30:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable), process ceph-mon, pid 7 2024-04-05T21:30:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: pidfile_write: ignore empty --pid-file 2024-04-05T21:30:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: load: jerasure load: lrc 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: RocksDB version: 7.9.2 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Git sha 0 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Compile date 2024-04-05 12:53:35 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: DB SUMMARY 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: DB Session ID: OY9R13C170TCVTMYSAJP 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: CURRENT file: CURRENT 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: IDENTITY file: IDENTITY 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: MANIFEST file: MANIFEST-000010 size: 1411 Bytes 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi188/store.db dir, Total Num: 1, files: 000027.sst 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi188/store.db: 000025.log size: 8223014 ; 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.error_if_exists: 0 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.create_if_missing: 0 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.paranoid_checks: 1 2024-04-05T21:30:45.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-05T21:30:45.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-05T21:30:45.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-05T21:30:45.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.env: 0x55bad2c317a0 2024-04-05T21:30:45.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.fs: PosixFileSystem 2024-04-05T21:30:45.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.info_log: 0x55bad4c8d3a0 2024-04-05T21:30:45.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_file_opening_threads: 16 2024-04-05T21:30:45.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.statistics: (nil) 2024-04-05T21:30:45.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.use_fsync: 0 2024-04-05T21:30:45.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_log_file_size: 0 2024-04-05T21:30:45.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-05T21:30:45.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-05T21:30:45.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.keep_log_file_num: 1000 2024-04-05T21:30:45.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.recycle_log_file_num: 0 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.allow_fallocate: 1 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.allow_mmap_reads: 0 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.allow_mmap_writes: 0 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.use_direct_reads: 0 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.create_missing_column_families: 0 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.db_log_dir: 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.wal_dir: 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-05T21:30:45.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.advise_random_on_open: 1 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.db_write_buffer_size: 0 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.write_buffer_manager: 0x55bad3f02140 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.rate_limiter: (nil) 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.wal_recovery_mode: 2 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.enable_thread_tracking: 0 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.enable_pipelined_write: 0 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.unordered_write: 0 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-05T21:30:45.784 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.row_cache: None 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.wal_filter: None 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.allow_ingest_behind: 0 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.two_write_queues: 0 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.manual_wal_flush: 0 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.wal_compression: 0 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.atomic_flush: 0 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.log_readahead_size: 0 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-05T21:30:45.785 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.best_efforts_recovery: 0 2024-04-05T21:30:45.786 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-05T21:30:45.786 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-05T21:30:45.786 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.allow_data_in_errors: 0 2024-04-05T21:30:45.786 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.db_host_id: __hostname__ 2024-04-05T21:30:45.787 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-05T21:30:45.787 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_background_jobs: 2 2024-04-05T21:30:45.787 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_background_compactions: -1 2024-04-05T21:30:45.787 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_subcompactions: 1 2024-04-05T21:30:45.787 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-05T21:30:45.787 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_total_wal_size: 0 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_open_files: -1 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.bytes_per_sync: 0 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compaction_readahead_size: 0 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_background_flushes: -1 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Compression algorithms supported: 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-05T21:30:45.788 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: kZSTD supported: 0 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: kXpressCompression supported: 0 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: kLZ4HCCompression supported: 1 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: kZlibCompression supported: 1 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: kSnappyCompression supported: 1 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: kLZ4Compression supported: 1 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: kBZip2Compression supported: 0 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi188/store.db/MANIFEST-000010 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.merge_operator: 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compaction_filter: None 2024-04-05T21:30:45.789 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compaction_filter_factory: None 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.sst_partitioner_factory: None 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bad4c8d4c0) 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: cache_index_and_filter_blocks: 1 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: pin_top_level_index_and_filter: 1 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: index_type: 0 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: data_block_index_type: 0 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: index_shortening: 1 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: checksum: 4 2024-04-05T21:30:45.790 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: no_block_cache: 0 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_cache: 0x55bad3fa31f0 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_cache_name: BinnedLRUCache 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_cache_options: 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: capacity : 536870912 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: num_shard_bits : 4 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: strict_capacity_limit : 0 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: high_pri_pool_ratio: 0.000 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_cache_compressed: (nil) 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: persistent_cache: (nil) 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_size: 4096 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_size_deviation: 10 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_restart_interval: 16 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: index_block_restart_interval: 1 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: metadata_block_size: 4096 2024-04-05T21:30:45.791 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: partition_filters: 0 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: use_delta_encoding: 1 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: filter_policy: bloomfilter 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: whole_key_filtering: 1 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: verify_compression: 0 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: read_amp_bytes_per_bit: 0 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: format_version: 5 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: enable_index_compression: 1 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: block_align: 0 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: max_auto_readahead_size: 262144 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: prepopulate_block_cache: 0 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: initial_auto_readahead_size: 8192 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: num_file_reads_for_auto_readahead: 2 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.write_buffer_size: 33554432 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_write_buffer_number: 2 2024-04-05T21:30:45.792 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compression: NoCompression 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.bottommost_compression: Disabled 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.prefix_extractor: nullptr 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.num_levels: 7 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-05T21:30:45.793 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compression_opts.level: 32767 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compression_opts.strategy: 0 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compression_opts.enabled: false 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-05T21:30:45.794 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.target_file_size_base: 67108864 2024-04-05T21:30:45.795 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-05T21:30:45.795 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-05T21:30:45.795 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-05T21:30:45.795 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-05T21:30:45.795 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-05T21:30:45.795 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-05T21:30:45.795 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-05T21:30:45.795 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-05T21:30:45.795 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-05T21:30:45.795 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.arena_block_size: 1048576 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.disable_auto_compactions: 0 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-05T21:30:45.796 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.table_properties_collectors: 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.inplace_update_support: 0 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.bloom_locality: 0 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.max_successive_merges: 0 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-05T21:30:45.797 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.paranoid_file_checks: 0 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.force_consistency_checks: 1 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.report_bg_io_stats: 0 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.ttl: 2592000 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.enable_blob_files: false 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.min_blob_size: 0 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.blob_file_size: 268435456 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-05T21:30:45.798 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.blob_file_starting_level: 0 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi188/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 29, last_sequence is 12017, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 25 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f84aa254-bc15-4e06-bc4b-53d1aa1f6248 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712352645412850, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712352645435894, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 4840579, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12022, "largest_seqno": 16152, "table_properties": {"data_size": 4826081, "index_size": 9317, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4165, "raw_key_size": 36829, "raw_average_key_size": 22, "raw_value_size": 4793103, "raw_average_value_size": 2882, "num_data_blocks": 421, "num_entries": 1663, "num_filter_entries": 1663, "num_deletions": 259, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712352645, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f84aa254-bc15-4e06-bc4b-53d1aa1f6248", "db_session_id": "OY9R13C170TCVTMYSAJP", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712352645435999, "job": 1, "event": "recovery_finished"} 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi188/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-05T21:30:45.799 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bad4042000 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: DB pointer 0x55bad4030000 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: ** DB Stats ** 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T21:30:45.800 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: L0 1/0 4.62 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 552.3 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: L6 1/0 4.81 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Sum 2/0 9.43 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 552.3 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 552.3 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: ** Compaction Stats [default] ** 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 552.3 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-05T21:30:45.801 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Flush(GB): cumulative 0.005, interval 0.005 2024-04-05T21:30:45.802 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-05T21:30:45.802 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-05T21:30:45.802 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-05T21:30:45.802 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-05T21:30:45.802 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Cumulative compaction: 0.00 GB write, 177.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T21:30:45.802 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Interval compaction: 0.00 GB write, 177.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-05T21:30:45.802 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-05T21:30:45.802 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Block cache BinnedLRUCache@0x55bad3fa31f0#7 capacity: 512.00 MB usage: 14.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-05T21:30:45.802 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.53 KB,0.000864267%) IndexBlock(1,10.08 KB,0.00192225%) Misc(1,0.00 KB,0%) 2024-04-05T21:30:45.802 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-04-05T21:30:45.802 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-05T21:30:45.802 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: starting mon.smithi188 rank 1 at public addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] at bind addrs [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi188 fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:30:45.802 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: mon.smithi188@-1(???) e2 preinit fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: mon.smithi188@-1(???).mds e20 new map 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: mon.smithi188@-1(???).mds e20 print_map 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: e20 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: legacy client fscid: 1 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: Filesystem 'cephfs' (1) 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: fs_name cephfs 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: epoch 20 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: modified 2024-04-05T21:27:26.656630+0000 2024-04-05T21:30:45.803 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: tableserver 0 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: root 0 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: session_timeout 60 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: session_autoclose 300 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: max_file_size 1099511627776 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: max_xattr_size 65536 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: required_client_features {} 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: last_failure 0 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: last_failure_osd_epoch 46 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: max_mds 2 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: in 0,1 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: up {0=14476,1=24255} 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: failed 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: damaged 2024-04-05T21:30:45.804 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: stopped 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: data_pools [3] 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: metadata_pool 2 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: inline_data disabled 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: balancer 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: bal_rank_mask -1 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: standby_count_wanted 1 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: [mds.cephfs.smithi121.ifoxmw{0:14476} state up:active seq 6 export targets 1 join_fscid=1 addr [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: [mds.cephfs.smithi121.gxheqt{0:14488} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/4279084130,v1:172.21.15.121:6827/4279084130] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: [mds.cephfs.smithi188.bjmisu{1:24255} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: [mds.cephfs.smithi188.latnft{1:24265} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout: 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: mon.smithi188@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-05T21:30:45.805 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: mon.smithi188@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-05T21:30:45.806 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: mon.smithi188@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-05T21:30:45.806 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: mon.smithi188@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-05T21:30:45.806 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 ceph-mon[107793]: mon.smithi188@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-04-05T21:30:45.806 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 bash[107755]: 1a1944725b7d344fcb52fc9f6fb75b013e6ac653b33f06e290f99cce83b7da6c 2024-04-05T21:30:45.806 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:45 smithi188 systemd[1]: Started Ceph mon.smithi188 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:30:48.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: pgmap v100: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:48.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:30:48.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: pgmap v101: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:48.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: mon.smithi188 calling monitor election 2024-04-05T21:30:48.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: mon.smithi121 calling monitor election 2024-04-05T21:30:48.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: mon.smithi121 is new leader, mons smithi121,smithi188 in quorum (ranks 0,1) 2024-04-05T21:30:48.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: monmap e2: 2 mons at {smithi121=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],smithi188=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T21:30:48.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:30:48.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: osdmap e48: 6 total, 6 up, 6 in 2024-04-05T21:30:48.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: mgrmap e33: smithi121.etbquh(active, since 3m), standbys: smithi188.tgcuir 2024-04-05T21:30:48.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: overall HEALTH_OK 2024-04-05T21:30:48.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:48.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:48.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:47 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:48.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: pgmap v100: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:48.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:30:48.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: pgmap v101: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:48.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: mon.smithi188 calling monitor election 2024-04-05T21:30:48.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: mon.smithi121 calling monitor election 2024-04-05T21:30:48.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: mon.smithi121 is new leader, mons smithi121,smithi188 in quorum (ranks 0,1) 2024-04-05T21:30:48.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: monmap e2: 2 mons at {smithi121=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],smithi188=[v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-05T21:30:48.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:30:48.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: osdmap e48: 6 total, 6 up, 6 in 2024-04-05T21:30:48.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: mgrmap e33: smithi121.etbquh(active, since 3m), standbys: smithi188.tgcuir 2024-04-05T21:30:48.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: overall HEALTH_OK 2024-04-05T21:30:48.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:48.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:48.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:47 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:49.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:48 smithi121 ceph-mon[145859]: pgmap v102: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:48 smithi188 ceph-mon[107793]: pgmap v102: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:51.511 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:51 smithi188 ceph-mon[107793]: pgmap v103: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:51.511 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:51 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:51.511 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:51 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:51.511 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:51 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:51.512 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:51 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:51 smithi121 ceph-mon[145859]: pgmap v103: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:51 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:51 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:51 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:51 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:53.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:53 smithi188 ceph-mon[107793]: pgmap v104: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:53.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:53 smithi188 ceph-mon[107793]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:30:53.734 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:53 smithi121 ceph-mon[145859]: pgmap v104: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:53.734 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:53 smithi121 ceph-mon[145859]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2 up:standby-replay 2024-04-05T21:30:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: pgmap v105: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:30:55.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:55.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:55.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:30:55.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:30:55.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:55.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi121"}]: dispatch 2024-04-05T21:30:55.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi121"}]': finished 2024-04-05T21:30:55.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi188"}]: dispatch 2024-04-05T21:30:55.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi188"}]': finished 2024-04-05T21:30:55.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: pgmap v105: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:30:55.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:55.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:55.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:55.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:30:55.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:55.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:30:55.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:30:55.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:30:55.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:55.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi121"}]: dispatch 2024-04-05T21:30:55.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi121"}]': finished 2024-04-05T21:30:55.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi188"}]: dispatch 2024-04-05T21:30:55.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi188"}]': finished 2024-04-05T21:30:55.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.990+0000 7f7bd9096700 1 -- 172.21.15.121:0/3475398684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd4068670 msgr2=0x7f7bd4068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:55.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.990+0000 7f7bd9096700 1 --2- 172.21.15.121:0/3475398684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd4068670 0x7f7bd4068a50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4009910 tx=0x7f7bc4009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:55.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.990+0000 7f7bd9096700 1 -- 172.21.15.121:0/3475398684 shutdown_connections 2024-04-05T21:30:55.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.990+0000 7f7bd9096700 1 --2- 172.21.15.121:0/3475398684 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bd4069020 0x7f7bd4105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:55.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.990+0000 7f7bd9096700 1 --2- 172.21.15.121:0/3475398684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd4068670 0x7f7bd4068a50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:55.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.990+0000 7f7bd9096700 1 -- 172.21.15.121:0/3475398684 >> 172.21.15.121:0/3475398684 conn(0x7f7bd40755a0 msgr2=0x7f7bd40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:55.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.990+0000 7f7bd9096700 1 -- 172.21.15.121:0/3475398684 shutdown_connections 2024-04-05T21:30:55.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.990+0000 7f7bd9096700 1 -- 172.21.15.121:0/3475398684 wait complete. 2024-04-05T21:30:55.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.991+0000 7f7bd9096700 1 Processor -- start 2024-04-05T21:30:55.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.991+0000 7f7bd9096700 1 -- start start 2024-04-05T21:30:55.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd9096700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bd4068670 0x7f7bd40ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:55.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd9096700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd4069020 0x7f7bd41001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:55.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd9096700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd4100720 con 0x7f7bd4069020 2024-04-05T21:30:55.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd9096700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd4100860 con 0x7f7bd4068670 2024-04-05T21:30:55.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd4069020 0x7f7bd41001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:55.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd4069020 0x7f7bd41001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45610/0 (socket says 172.21.15.121:45610) 2024-04-05T21:30:55.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd259c700 1 -- 172.21.15.121:0/2342426235 learned_addr learned my addr 172.21.15.121:0/2342426235 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:55.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd2d9d700 1 --2- 172.21.15.121:0/2342426235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bd4068670 0x7f7bd40ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:55.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd259c700 1 -- 172.21.15.121:0/2342426235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bd4068670 msgr2=0x7f7bd40ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:55.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd259c700 1 --2- 172.21.15.121:0/2342426235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bd4068670 0x7f7bd40ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:55.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd259c700 1 -- 172.21.15.121:0/2342426235 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc40095f0 con 0x7f7bd4069020 2024-04-05T21:30:55.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd259c700 1 --2- 172.21.15.121:0/2342426235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd4069020 0x7f7bd41001e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800c960 tx=0x7f7bc800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:55.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bbbfff700 1 -- 172.21.15.121:0/2342426235 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bc8007a70 con 0x7f7bd4069020 2024-04-05T21:30:55.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bbbfff700 1 -- 172.21.15.121:0/2342426235 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bc8007bd0 con 0x7f7bd4069020 2024-04-05T21:30:55.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd9096700 1 -- 172.21.15.121:0/2342426235 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd41a2840 con 0x7f7bd4069020 2024-04-05T21:30:55.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bbbfff700 1 -- 172.21.15.121:0/2342426235 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bc8018630 con 0x7f7bd4069020 2024-04-05T21:30:55.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.992+0000 7f7bd9096700 1 -- 172.21.15.121:0/2342426235 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd41a2ba0 con 0x7f7bd4069020 2024-04-05T21:30:55.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.993+0000 7f7bd9096700 1 -- 172.21.15.121:0/2342426235 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd4109130 con 0x7f7bd4069020 2024-04-05T21:30:55.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.994+0000 7f7bbbfff700 1 -- 172.21.15.121:0/2342426235 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7bc800f450 con 0x7f7bd4069020 2024-04-05T21:30:55.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.994+0000 7f7bbbfff700 1 --2- 172.21.15.121:0/2342426235 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7bbc06cce0 0x7f7bbc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:55.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.994+0000 7f7bbbfff700 1 -- 172.21.15.121:0/2342426235 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f7bc808c540 con 0x7f7bd4069020 2024-04-05T21:30:55.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.994+0000 7f7bd2d9d700 1 --2- 172.21.15.121:0/2342426235 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7bbc06cce0 0x7f7bbc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:55.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.995+0000 7f7bd2d9d700 1 --2- 172.21.15.121:0/2342426235 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7bbc06cce0 0x7f7bbc06f1a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4006010 tx=0x7f7bc401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:56.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:55.999+0000 7f7bbbfff700 1 -- 172.21.15.121:0/2342426235 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7bc805a180 con 0x7f7bd4069020 2024-04-05T21:30:56.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.227+0000 7f7bd9096700 1 -- 172.21.15.121:0/2342426235 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bd402cd60 con 0x7f7bbc06cce0 2024-04-05T21:30:56.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.228+0000 7f7bbbfff700 1 -- 172.21.15.121:0/2342426235 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f7bd402cd60 con 0x7f7bbc06cce0 2024-04-05T21:30:56.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.230+0000 7f7bd9096700 1 -- 172.21.15.121:0/2342426235 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7bbc06cce0 msgr2=0x7f7bbc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:56.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.230+0000 7f7bd9096700 1 --2- 172.21.15.121:0/2342426235 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7bbc06cce0 0x7f7bbc06f1a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4006010 tx=0x7f7bc401a040 comp rx=0 tx=0).stop 2024-04-05T21:30:56.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.230+0000 7f7bd9096700 1 -- 172.21.15.121:0/2342426235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd4069020 msgr2=0x7f7bd41001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:56.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.230+0000 7f7bd9096700 1 --2- 172.21.15.121:0/2342426235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd4069020 0x7f7bd41001e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800c960 tx=0x7f7bc800cd20 comp rx=0 tx=0).stop 2024-04-05T21:30:56.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.230+0000 7f7bd9096700 1 -- 172.21.15.121:0/2342426235 shutdown_connections 2024-04-05T21:30:56.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.230+0000 7f7bd9096700 1 --2- 172.21.15.121:0/2342426235 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7bbc06cce0 0x7f7bbc06f1a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.230+0000 7f7bd9096700 1 --2- 172.21.15.121:0/2342426235 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bd4068670 0x7f7bd40ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.230+0000 7f7bd9096700 1 --2- 172.21.15.121:0/2342426235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bd4069020 0x7f7bd41001e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.230+0000 7f7bd9096700 1 -- 172.21.15.121:0/2342426235 >> 172.21.15.121:0/2342426235 conn(0x7f7bd40755a0 msgr2=0x7f7bd40fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:56.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.230+0000 7f7bd9096700 1 -- 172.21.15.121:0/2342426235 shutdown_connections 2024-04-05T21:30:56.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.230+0000 7f7bd9096700 1 -- 172.21.15.121:0/2342426235 wait complete. 2024-04-05T21:30:56.241 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:30:56.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.337+0000 7fa5f93ef700 1 -- 172.21.15.121:0/4208475453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f4101770 msgr2=0x7fa5f41057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:56.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.337+0000 7fa5f93ef700 1 --2- 172.21.15.121:0/4208475453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f4101770 0x7fa5f41057c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009910 tx=0x7fa5e8009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:56.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.337+0000 7fa5f93ef700 1 -- 172.21.15.121:0/4208475453 shutdown_connections 2024-04-05T21:30:56.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.337+0000 7fa5f93ef700 1 --2- 172.21.15.121:0/4208475453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f4101770 0x7fa5f41057c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.337+0000 7fa5f93ef700 1 --2- 172.21.15.121:0/4208475453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f4100dc0 0x7fa5f41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.337+0000 7fa5f93ef700 1 -- 172.21.15.121:0/4208475453 >> 172.21.15.121:0/4208475453 conn(0x7fa5f40fc650 msgr2=0x7fa5f40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:56.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.338+0000 7fa5f93ef700 1 -- 172.21.15.121:0/4208475453 shutdown_connections 2024-04-05T21:30:56.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.338+0000 7fa5f93ef700 1 -- 172.21.15.121:0/4208475453 wait complete. 2024-04-05T21:30:56.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.338+0000 7fa5f93ef700 1 Processor -- start 2024-04-05T21:30:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.338+0000 7fa5f93ef700 1 -- start start 2024-04-05T21:30:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.338+0000 7fa5f93ef700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f4100dc0 0x7fa5f41966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.338+0000 7fa5f93ef700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f4101770 0x7fa5f4196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.338+0000 7fa5f93ef700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f41972a0 con 0x7fa5f4101770 2024-04-05T21:30:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.338+0000 7fa5f93ef700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f4199fd0 con 0x7fa5f4100dc0 2024-04-05T21:30:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.339+0000 7fa5f27fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f4101770 0x7fa5f4196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.339+0000 7fa5f27fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f4101770 0x7fa5f4196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45622/0 (socket says 172.21.15.121:45622) 2024-04-05T21:30:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.339+0000 7fa5f27fc700 1 -- 172.21.15.121:0/2844806296 learned_addr learned my addr 172.21.15.121:0/2844806296 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.339+0000 7fa5f2ffd700 1 --2- 172.21.15.121:0/2844806296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f4100dc0 0x7fa5f41966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.339+0000 7fa5f27fc700 1 -- 172.21.15.121:0/2844806296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f4100dc0 msgr2=0x7fa5f41966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.339+0000 7fa5f27fc700 1 --2- 172.21.15.121:0/2844806296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f4100dc0 0x7fa5f41966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.339+0000 7fa5f27fc700 1 -- 172.21.15.121:0/2844806296 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e80095f0 con 0x7fa5f4101770 2024-04-05T21:30:56.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.339+0000 7fa5f27fc700 1 --2- 172.21.15.121:0/2844806296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f4101770 0x7fa5f4196c10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa5e80092e0 tx=0x7fa5e800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:56.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.339+0000 7fa5dbfff700 1 -- 172.21.15.121:0/2844806296 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5e801d070 con 0x7fa5f4101770 2024-04-05T21:30:56.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.339+0000 7fa5dbfff700 1 -- 172.21.15.121:0/2844806296 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5e8005940 con 0x7fa5f4101770 2024-04-05T21:30:56.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.339+0000 7fa5f93ef700 1 -- 172.21.15.121:0/2844806296 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f419a250 con 0x7fa5f4101770 2024-04-05T21:30:56.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.339+0000 7fa5dbfff700 1 -- 172.21.15.121:0/2844806296 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5e800fe50 con 0x7fa5f4101770 2024-04-05T21:30:56.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.339+0000 7fa5f93ef700 1 -- 172.21.15.121:0/2844806296 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f419a740 con 0x7fa5f4101770 2024-04-05T21:30:56.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.340+0000 7fa5f93ef700 1 -- 172.21.15.121:0/2844806296 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f41091c0 con 0x7fa5f4101770 2024-04-05T21:30:56.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.341+0000 7fa5dbfff700 1 -- 172.21.15.121:0/2844806296 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa5e8003a60 con 0x7fa5f4101770 2024-04-05T21:30:56.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.341+0000 7fa5dbfff700 1 --2- 172.21.15.121:0/2844806296 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5dc06cd30 0x7fa5dc06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:56.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.341+0000 7fa5dbfff700 1 -- 172.21.15.121:0/2844806296 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fa5e808d730 con 0x7fa5f4101770 2024-04-05T21:30:56.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.341+0000 7fa5f2ffd700 1 --2- 172.21.15.121:0/2844806296 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5dc06cd30 0x7fa5dc06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:56.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.342+0000 7fa5f2ffd700 1 --2- 172.21.15.121:0/2844806296 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5dc06cd30 0x7fa5dc06f1f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4005950 tx=0x7fa5e40058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:56.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.346+0000 7fa5dbfff700 1 -- 172.21.15.121:0/2844806296 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa5e80577b0 con 0x7fa5f4101770 2024-04-05T21:30:56.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:56 smithi188 ceph-mon[107793]: Detected new or changed devices on smithi188 2024-04-05T21:30:56.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:56 smithi188 ceph-mon[107793]: Upgrade: Setting container_image for all mon 2024-04-05T21:30:56.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.561+0000 7fa5f93ef700 1 -- 172.21.15.121:0/2844806296 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5f402cd30 con 0x7fa5dc06cd30 2024-04-05T21:30:56.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.562+0000 7fa5dbfff700 1 -- 172.21.15.121:0/2844806296 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7fa5f402cd30 con 0x7fa5dc06cd30 2024-04-05T21:30:56.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.564+0000 7fa5f93ef700 1 -- 172.21.15.121:0/2844806296 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5dc06cd30 msgr2=0x7fa5dc06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:56.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.564+0000 7fa5f93ef700 1 --2- 172.21.15.121:0/2844806296 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5dc06cd30 0x7fa5dc06f1f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4005950 tx=0x7fa5e40058e0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.564+0000 7fa5f93ef700 1 -- 172.21.15.121:0/2844806296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f4101770 msgr2=0x7fa5f4196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:56.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.564+0000 7fa5f93ef700 1 --2- 172.21.15.121:0/2844806296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f4101770 0x7fa5f4196c10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa5e80092e0 tx=0x7fa5e800be20 comp rx=0 tx=0).stop 2024-04-05T21:30:56.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.564+0000 7fa5f93ef700 1 -- 172.21.15.121:0/2844806296 shutdown_connections 2024-04-05T21:30:56.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.564+0000 7fa5f93ef700 1 --2- 172.21.15.121:0/2844806296 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5dc06cd30 0x7fa5dc06f1f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.564+0000 7fa5f93ef700 1 --2- 172.21.15.121:0/2844806296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f4100dc0 0x7fa5f41966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.564+0000 7fa5f93ef700 1 --2- 172.21.15.121:0/2844806296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f4101770 0x7fa5f4196c10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.564+0000 7fa5f93ef700 1 -- 172.21.15.121:0/2844806296 >> 172.21.15.121:0/2844806296 conn(0x7fa5f40fc650 msgr2=0x7fa5f40fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:56.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.565+0000 7fa5f93ef700 1 -- 172.21.15.121:0/2844806296 shutdown_connections 2024-04-05T21:30:56.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.565+0000 7fa5f93ef700 1 -- 172.21.15.121:0/2844806296 wait complete. 2024-04-05T21:30:56.660 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:56 smithi121 ceph-mon[145859]: Detected new or changed devices on smithi188 2024-04-05T21:30:56.660 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:56 smithi121 ceph-mon[145859]: Upgrade: Setting container_image for all mon 2024-04-05T21:30:56.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.669+0000 7f4371954700 1 -- 172.21.15.121:0/1332652626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f436c102620 msgr2=0x7f436c10ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:56.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.669+0000 7f4371954700 1 --2- 172.21.15.121:0/1332652626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f436c102620 0x7f436c10ab10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4360009910 tx=0x7f4360009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:56.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.670+0000 7f4371954700 1 -- 172.21.15.121:0/1332652626 shutdown_connections 2024-04-05T21:30:56.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.670+0000 7f4371954700 1 --2- 172.21.15.121:0/1332652626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f436c102620 0x7f436c10ab10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.670+0000 7f4371954700 1 --2- 172.21.15.121:0/1332652626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f436c101d00 0x7f436c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.670+0000 7f4371954700 1 -- 172.21.15.121:0/1332652626 >> 172.21.15.121:0/1332652626 conn(0x7f436c0fb570 msgr2=0x7f436c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:56.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.670+0000 7f4371954700 1 -- 172.21.15.121:0/1332652626 shutdown_connections 2024-04-05T21:30:56.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.670+0000 7f4371954700 1 -- 172.21.15.121:0/1332652626 wait complete. 2024-04-05T21:30:56.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.670+0000 7f4371954700 1 Processor -- start 2024-04-05T21:30:56.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.670+0000 7f4371954700 1 -- start start 2024-04-05T21:30:56.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.671+0000 7f4371954700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f436c101d00 0x7f436c1966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:56.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.671+0000 7f4371954700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f436c102620 0x7f436c196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:56.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.671+0000 7f4371954700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c1972a0 con 0x7f436c102620 2024-04-05T21:30:56.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.671+0000 7f4371954700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c199fd0 con 0x7f436c101d00 2024-04-05T21:30:56.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.671+0000 7f436a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f436c102620 0x7f436c196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:56.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.671+0000 7f436a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f436c102620 0x7f436c196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45644/0 (socket says 172.21.15.121:45644) 2024-04-05T21:30:56.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.671+0000 7f436a7fc700 1 -- 172.21.15.121:0/2763239995 learned_addr learned my addr 172.21.15.121:0/2763239995 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:56.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.671+0000 7f436affd700 1 --2- 172.21.15.121:0/2763239995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f436c101d00 0x7f436c1966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:56.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.671+0000 7f436a7fc700 1 -- 172.21.15.121:0/2763239995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f436c101d00 msgr2=0x7f436c1966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:56.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.671+0000 7f436a7fc700 1 --2- 172.21.15.121:0/2763239995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f436c101d00 0x7f436c1966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.671+0000 7f436a7fc700 1 -- 172.21.15.121:0/2763239995 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43600095f0 con 0x7f436c102620 2024-04-05T21:30:56.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.671+0000 7f436affd700 1 --2- 172.21.15.121:0/2763239995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f436c101d00 0x7f436c1966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:30:56.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.671+0000 7f436a7fc700 1 --2- 172.21.15.121:0/2763239995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f436c102620 0x7f436c196c10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4360000c00 tx=0x7f436000bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:56.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.672+0000 7f4370952700 1 -- 172.21.15.121:0/2763239995 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f436001d070 con 0x7f436c102620 2024-04-05T21:30:56.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.672+0000 7f4370952700 1 -- 172.21.15.121:0/2763239995 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4360003cf0 con 0x7f436c102620 2024-04-05T21:30:56.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.672+0000 7f4371954700 1 -- 172.21.15.121:0/2763239995 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436c19a250 con 0x7f436c102620 2024-04-05T21:30:56.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.672+0000 7f4370952700 1 -- 172.21.15.121:0/2763239995 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f436000fdd0 con 0x7f436c102620 2024-04-05T21:30:56.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.672+0000 7f4371954700 1 -- 172.21.15.121:0/2763239995 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436c19a6e0 con 0x7f436c102620 2024-04-05T21:30:56.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.673+0000 7f4371954700 1 -- 172.21.15.121:0/2763239995 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4358005320 con 0x7f436c102620 2024-04-05T21:30:56.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.673+0000 7f4370952700 1 -- 172.21.15.121:0/2763239995 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4360021430 con 0x7f436c102620 2024-04-05T21:30:56.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.673+0000 7f4370952700 1 --2- 172.21.15.121:0/2763239995 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f435406cce0 0x7f435406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:56.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.673+0000 7f4370952700 1 -- 172.21.15.121:0/2763239995 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f436008d600 con 0x7f436c102620 2024-04-05T21:30:56.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.674+0000 7f436affd700 1 --2- 172.21.15.121:0/2763239995 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f435406cce0 0x7f435406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:56.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.674+0000 7f436affd700 1 --2- 172.21.15.121:0/2763239995 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f435406cce0 0x7f435406f1a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f435c009cb0 tx=0x7f435c009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:56.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.678+0000 7f4370952700 1 -- 172.21.15.121:0/2763239995 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f436005b1b0 con 0x7f436c102620 2024-04-05T21:30:56.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.905+0000 7f4371954700 1 -- 172.21.15.121:0/2763239995 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4358000bf0 con 0x7f435406cce0 2024-04-05T21:30:56.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.916+0000 7f4370952700 1 -- 172.21.15.121:0/2763239995 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f4358000bf0 con 0x7f435406cce0 2024-04-05T21:30:56.917 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:30:56.917 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (107s) 21s ago 13m 24.4M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:30:56.918 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (13m) 21s ago 13m 8112k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:30:56.918 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (12m) 6s ago 12m 8024k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:30:56.918 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (13m) 21s ago 13m 6651k - 18.2.1 5be31c24972a 5bf35fe2ace4 2024-04-05T21:30:56.918 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (12m) 6s ago 12m 6656k - 18.2.1 5be31c24972a dcdfa6ae17bf 2024-04-05T21:30:56.918 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (90s) 21s ago 12m 80.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:30:56.918 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (7m) 21s ago 7m 64.8M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:30:56.918 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (7m) 21s ago 7m 134M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:30:56.918 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (7m) 6s ago 7m 110M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:30:56.918 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (6m) 6s ago 6m 62.5M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:30:56.918 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (3m) 21s ago 14m 503M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:30:56.918 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (2m) 6s ago 12m 450M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:30:56.918 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (30s) 21s ago 14m 23.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:30:56.919 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (11s) 6s ago 12m 23.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:30:56.919 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (2m) 21s ago 13m 17.8M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:30:56.919 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (2m) 6s ago 12m 16.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:30:56.919 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (11m) 21s ago 11m 216M 1227M 18.2.1 5be31c24972a f6f3a45429a2 2024-04-05T21:30:56.919 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (10m) 21s ago 10m 243M 1227M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:30:56.919 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (10m) 21s ago 10m 182M 1227M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:30:56.919 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (9m) 6s ago 9m 288M 2251M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:30:56.919 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (9m) 6s ago 9m 246M 2251M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:30:56.919 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (8m) 6s ago 8m 224M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:30:56.919 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (2m) 21s ago 12m 56.4M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:30:56.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.919+0000 7f4371954700 1 -- 172.21.15.121:0/2763239995 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f435406cce0 msgr2=0x7f435406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:56.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.919+0000 7f4371954700 1 --2- 172.21.15.121:0/2763239995 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f435406cce0 0x7f435406f1a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f435c009cb0 tx=0x7f435c009260 comp rx=0 tx=0).stop 2024-04-05T21:30:56.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.919+0000 7f4371954700 1 -- 172.21.15.121:0/2763239995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f436c102620 msgr2=0x7f436c196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:56.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.919+0000 7f4371954700 1 --2- 172.21.15.121:0/2763239995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f436c102620 0x7f436c196c10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4360000c00 tx=0x7f436000bb20 comp rx=0 tx=0).stop 2024-04-05T21:30:56.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.919+0000 7f4371954700 1 -- 172.21.15.121:0/2763239995 shutdown_connections 2024-04-05T21:30:56.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.919+0000 7f4371954700 1 --2- 172.21.15.121:0/2763239995 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f435406cce0 0x7f435406f1a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.919+0000 7f4371954700 1 --2- 172.21.15.121:0/2763239995 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f436c101d00 0x7f436c1966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.919+0000 7f4371954700 1 --2- 172.21.15.121:0/2763239995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f436c102620 0x7f436c196c10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:56.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.919+0000 7f4371954700 1 -- 172.21.15.121:0/2763239995 >> 172.21.15.121:0/2763239995 conn(0x7f436c0fb570 msgr2=0x7f436c105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:56.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.919+0000 7f4371954700 1 -- 172.21.15.121:0/2763239995 shutdown_connections 2024-04-05T21:30:56.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:56.919+0000 7f4371954700 1 -- 172.21.15.121:0/2763239995 wait complete. 2024-04-05T21:30:57.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.030+0000 7f7b2fffd700 1 -- 172.21.15.121:0/3949443333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b28068670 msgr2=0x7f7b28068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:57.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.030+0000 7f7b2fffd700 1 --2- 172.21.15.121:0/3949443333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b28068670 0x7f7b28068a50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7b18009910 tx=0x7f7b18009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:57.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.030+0000 7f7b2fffd700 1 -- 172.21.15.121:0/3949443333 shutdown_connections 2024-04-05T21:30:57.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.030+0000 7f7b2fffd700 1 --2- 172.21.15.121:0/3949443333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b28069020 0x7f7b28105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.030+0000 7f7b2fffd700 1 --2- 172.21.15.121:0/3949443333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b28068670 0x7f7b28068a50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.030+0000 7f7b2fffd700 1 -- 172.21.15.121:0/3949443333 >> 172.21.15.121:0/3949443333 conn(0x7f7b280755a0 msgr2=0x7f7b280759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:57.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.030+0000 7f7b2fffd700 1 -- 172.21.15.121:0/3949443333 shutdown_connections 2024-04-05T21:30:57.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.030+0000 7f7b2fffd700 1 -- 172.21.15.121:0/3949443333 wait complete. 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.031+0000 7f7b2fffd700 1 Processor -- start 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.031+0000 7f7b2fffd700 1 -- start start 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.031+0000 7f7b2fffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b28068670 0x7f7b28196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.031+0000 7f7b2fffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b28069020 0x7f7b28196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.031+0000 7f7b2fffd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b28197290 con 0x7f7b28069020 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.031+0000 7f7b2fffd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2819afb0 con 0x7f7b28068670 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.031+0000 7f7b2d598700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b28069020 0x7f7b28196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.031+0000 7f7b2d598700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b28069020 0x7f7b28196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45668/0 (socket says 172.21.15.121:45668) 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.031+0000 7f7b2d598700 1 -- 172.21.15.121:0/214897865 learned_addr learned my addr 172.21.15.121:0/214897865 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.032+0000 7f7b2dd99700 1 --2- 172.21.15.121:0/214897865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b28068670 0x7f7b28196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.032+0000 7f7b2d598700 1 -- 172.21.15.121:0/214897865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b28068670 msgr2=0x7f7b28196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.032+0000 7f7b2d598700 1 --2- 172.21.15.121:0/214897865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b28068670 0x7f7b28196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.032+0000 7f7b2d598700 1 -- 172.21.15.121:0/214897865 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b180095f0 con 0x7f7b28069020 2024-04-05T21:30:57.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.032+0000 7f7b2d598700 1 --2- 172.21.15.121:0/214897865 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b28069020 0x7f7b28196bb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7b2400c8f0 tx=0x7f7b2400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:57.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.032+0000 7f7b1effd700 1 -- 172.21.15.121:0/214897865 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b24007a50 con 0x7f7b28069020 2024-04-05T21:30:57.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.032+0000 7f7b1effd700 1 -- 172.21.15.121:0/214897865 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b24007bb0 con 0x7f7b28069020 2024-04-05T21:30:57.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.032+0000 7f7b2fffd700 1 -- 172.21.15.121:0/214897865 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2819b290 con 0x7f7b28069020 2024-04-05T21:30:57.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.032+0000 7f7b1effd700 1 -- 172.21.15.121:0/214897865 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b24018630 con 0x7f7b28069020 2024-04-05T21:30:57.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.032+0000 7f7b2fffd700 1 -- 172.21.15.121:0/214897865 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2819b7e0 con 0x7f7b28069020 2024-04-05T21:30:57.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.033+0000 7f7b2fffd700 1 -- 172.21.15.121:0/214897865 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b28109130 con 0x7f7b28069020 2024-04-05T21:30:57.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.034+0000 7f7b1effd700 1 -- 172.21.15.121:0/214897865 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7b24007d20 con 0x7f7b28069020 2024-04-05T21:30:57.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.034+0000 7f7b1effd700 1 --2- 172.21.15.121:0/214897865 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7b1406cce0 0x7f7b1406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:57.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.034+0000 7f7b1effd700 1 -- 172.21.15.121:0/214897865 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f7b2408c400 con 0x7f7b28069020 2024-04-05T21:30:57.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.034+0000 7f7b2dd99700 1 --2- 172.21.15.121:0/214897865 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7b1406cce0 0x7f7b1406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:57.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.035+0000 7f7b2dd99700 1 --2- 172.21.15.121:0/214897865 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7b1406cce0 0x7f7b1406f1a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800b5c0 tx=0x7f7b18005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:57.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.039+0000 7f7b1effd700 1 -- 172.21.15.121:0/214897865 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7b240565a0 con 0x7f7b28069020 2024-04-05T21:30:57.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.329+0000 7f7b2fffd700 1 -- 172.21.15.121:0/214897865 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7b2804ea90 con 0x7f7b28069020 2024-04-05T21:30:57.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.330+0000 7f7b1effd700 1 -- 172.21.15.121:0/214897865 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+694 (secure 0 0 0) 0x7f7b24059bc0 con 0x7f7b28069020 2024-04-05T21:30:57.331 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:30:57.331 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:30:57.331 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:30:57.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:30:57.331 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:30:57.331 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:30:57.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:30:57.331 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:30:57.331 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6 2024-04-05T21:30:57.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:30:57.332 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:30:57.332 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:30:57.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:30:57.332 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:30:57.332 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 10, 2024-04-05T21:30:57.332 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:30:57.332 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:30:57.332 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:30:57.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.332+0000 7f7b2fffd700 1 -- 172.21.15.121:0/214897865 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7b1406cce0 msgr2=0x7f7b1406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:57.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.332+0000 7f7b2fffd700 1 --2- 172.21.15.121:0/214897865 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7b1406cce0 0x7f7b1406f1a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800b5c0 tx=0x7f7b18005fb0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.332+0000 7f7b2fffd700 1 -- 172.21.15.121:0/214897865 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b28069020 msgr2=0x7f7b28196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:57.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.332+0000 7f7b2fffd700 1 --2- 172.21.15.121:0/214897865 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b28069020 0x7f7b28196bb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7b2400c8f0 tx=0x7f7b2400ccb0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.332+0000 7f7b2fffd700 1 -- 172.21.15.121:0/214897865 shutdown_connections 2024-04-05T21:30:57.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.332+0000 7f7b2fffd700 1 --2- 172.21.15.121:0/214897865 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7b1406cce0 0x7f7b1406f1a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.332+0000 7f7b2fffd700 1 --2- 172.21.15.121:0/214897865 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b28068670 0x7f7b28196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.332+0000 7f7b2fffd700 1 --2- 172.21.15.121:0/214897865 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b28069020 0x7f7b28196bb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.332+0000 7f7b2fffd700 1 -- 172.21.15.121:0/214897865 >> 172.21.15.121:0/214897865 conn(0x7f7b280755a0 msgr2=0x7f7b280fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.332+0000 7f7b2fffd700 1 -- 172.21.15.121:0/214897865 shutdown_connections 2024-04-05T21:30:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.332+0000 7f7b2fffd700 1 -- 172.21.15.121:0/214897865 wait complete. 2024-04-05T21:30:57.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.436+0000 7fe3b41b5700 1 -- 172.21.15.121:0/647072930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ac0ffbd0 msgr2=0x7fe3ac100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:57.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.436+0000 7fe3b41b5700 1 --2- 172.21.15.121:0/647072930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ac0ffbd0 0x7fe3ac100050 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe39c009910 tx=0x7fe39c009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:57.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.436+0000 7fe3b41b5700 1 -- 172.21.15.121:0/647072930 shutdown_connections 2024-04-05T21:30:57.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.436+0000 7fe3b41b5700 1 --2- 172.21.15.121:0/647072930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ac0ffbd0 0x7fe3ac100050 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.436+0000 7fe3b41b5700 1 --2- 172.21.15.121:0/647072930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe3ac105a40 0x7fe3ac105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.437+0000 7fe3b41b5700 1 -- 172.21.15.121:0/647072930 >> 172.21.15.121:0/647072930 conn(0x7fe3ac0755a0 msgr2=0x7fe3ac0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:57.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.437+0000 7fe3b41b5700 1 -- 172.21.15.121:0/647072930 shutdown_connections 2024-04-05T21:30:57.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.437+0000 7fe3b41b5700 1 -- 172.21.15.121:0/647072930 wait complete. 2024-04-05T21:30:57.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.437+0000 7fe3b41b5700 1 Processor -- start 2024-04-05T21:30:57.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.437+0000 7fe3b41b5700 1 -- start start 2024-04-05T21:30:57.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.437+0000 7fe3b41b5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe3ac0ffbd0 0x7fe3ac101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:57.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.437+0000 7fe3b41b5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ac105a40 0x7fe3ac101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:57.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.437+0000 7fe3b41b5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac102090 con 0x7fe3ac105a40 2024-04-05T21:30:57.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.437+0000 7fe3b41b5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac1021d0 con 0x7fe3ac0ffbd0 2024-04-05T21:30:57.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.438+0000 7fe3b1750700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ac105a40 0x7fe3ac101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:57.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.438+0000 7fe3b1750700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ac105a40 0x7fe3ac101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45684/0 (socket says 172.21.15.121:45684) 2024-04-05T21:30:57.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.438+0000 7fe3b1750700 1 -- 172.21.15.121:0/3275756144 learned_addr learned my addr 172.21.15.121:0/3275756144 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:57.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.438+0000 7fe3b1f51700 1 --2- 172.21.15.121:0/3275756144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe3ac0ffbd0 0x7fe3ac101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:57.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.438+0000 7fe3b1750700 1 -- 172.21.15.121:0/3275756144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe3ac0ffbd0 msgr2=0x7fe3ac101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:57.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.438+0000 7fe3b1750700 1 --2- 172.21.15.121:0/3275756144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe3ac0ffbd0 0x7fe3ac101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.438+0000 7fe3b1750700 1 -- 172.21.15.121:0/3275756144 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c0095f0 con 0x7fe3ac105a40 2024-04-05T21:30:57.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.438+0000 7fe3b1750700 1 --2- 172.21.15.121:0/3275756144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ac105a40 0x7fe3ac101b50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe39c006010 tx=0x7fe39c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:57.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.438+0000 7fe3a2ffd700 1 -- 172.21.15.121:0/3275756144 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe39c01d070 con 0x7fe3ac105a40 2024-04-05T21:30:57.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.438+0000 7fe3a2ffd700 1 -- 172.21.15.121:0/3275756144 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe39c00f460 con 0x7fe3ac105a40 2024-04-05T21:30:57.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.438+0000 7fe3b41b5700 1 -- 172.21.15.121:0/3275756144 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ac102450 con 0x7fe3ac105a40 2024-04-05T21:30:57.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.438+0000 7fe3a2ffd700 1 -- 172.21.15.121:0/3275756144 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe39c021640 con 0x7fe3ac105a40 2024-04-05T21:30:57.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.438+0000 7fe3b41b5700 1 -- 172.21.15.121:0/3275756144 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ac1a2bf0 con 0x7fe3ac105a40 2024-04-05T21:30:57.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.439+0000 7fe3b41b5700 1 -- 172.21.15.121:0/3275756144 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ac108fe0 con 0x7fe3ac105a40 2024-04-05T21:30:57.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.440+0000 7fe3a2ffd700 1 -- 172.21.15.121:0/3275756144 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe39c00fa20 con 0x7fe3ac105a40 2024-04-05T21:30:57.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.440+0000 7fe3a2ffd700 1 --2- 172.21.15.121:0/3275756144 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe39806cce0 0x7fe39806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:57.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.440+0000 7fe3a2ffd700 1 -- 172.21.15.121:0/3275756144 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fe39c08d810 con 0x7fe3ac105a40 2024-04-05T21:30:57.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.440+0000 7fe3b1f51700 1 --2- 172.21.15.121:0/3275756144 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe39806cce0 0x7fe39806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:57.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.440+0000 7fe3b1f51700 1 --2- 172.21.15.121:0/3275756144 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe39806cce0 0x7fe39806f1a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800ba60 tx=0x7fe3a8005d50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:57.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.445+0000 7fe3a2ffd700 1 -- 172.21.15.121:0/3275756144 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe39c057930 con 0x7fe3ac105a40 2024-04-05T21:30:57.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.686+0000 7fe3b41b5700 1 -- 172.21.15.121:0/3275756144 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe3ac1089c0 con 0x7fe3ac105a40 2024-04-05T21:30:57.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.687+0000 7fe3a2ffd700 1 -- 172.21.15.121:0/3275756144 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1874 (secure 0 0 0) 0x7fe39c05af50 con 0x7fe3ac105a40 2024-04-05T21:30:57.688 INFO:teuthology.orchestra.run.smithi121.stdout:e21 2024-04-05T21:30:57.688 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:30:57.688 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:30:57.688 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:30:57.688 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 21 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:30:52.246236+0000 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:30:57.689 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 46 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:up {0=14476,1=24255} 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:failed 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{0:14476} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{0:14488} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/4279084130,v1:172.21.15.121:6827/4279084130] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:57.691 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{1:24255} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:57.691 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{1:24265} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:30:57.691 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:30:57.691 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:30:57.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.689+0000 7fe3b41b5700 1 -- 172.21.15.121:0/3275756144 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe39806cce0 msgr2=0x7fe39806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:57.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.689+0000 7fe3b41b5700 1 --2- 172.21.15.121:0/3275756144 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe39806cce0 0x7fe39806f1a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800ba60 tx=0x7fe3a8005d50 comp rx=0 tx=0).stop 2024-04-05T21:30:57.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.689+0000 7fe3b41b5700 1 -- 172.21.15.121:0/3275756144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ac105a40 msgr2=0x7fe3ac101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:57.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.689+0000 7fe3b41b5700 1 --2- 172.21.15.121:0/3275756144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ac105a40 0x7fe3ac101b50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe39c006010 tx=0x7fe39c00be20 comp rx=0 tx=0).stop 2024-04-05T21:30:57.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.689+0000 7fe3b41b5700 1 -- 172.21.15.121:0/3275756144 shutdown_connections 2024-04-05T21:30:57.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.689+0000 7fe3b41b5700 1 --2- 172.21.15.121:0/3275756144 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe39806cce0 0x7fe39806f1a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.689+0000 7fe3b41b5700 1 --2- 172.21.15.121:0/3275756144 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe3ac0ffbd0 0x7fe3ac101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.689+0000 7fe3b41b5700 1 --2- 172.21.15.121:0/3275756144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ac105a40 0x7fe3ac101b50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.689+0000 7fe3b41b5700 1 -- 172.21.15.121:0/3275756144 >> 172.21.15.121:0/3275756144 conn(0x7fe3ac0755a0 msgr2=0x7fe3ac103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:57.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.689+0000 7fe3b41b5700 1 -- 172.21.15.121:0/3275756144 shutdown_connections 2024-04-05T21:30:57.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.689+0000 7fe3b41b5700 1 -- 172.21.15.121:0/3275756144 wait complete. 2024-04-05T21:30:57.692 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 21 2024-04-05T21:30:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.794+0000 7f64159a1700 1 -- 172.21.15.121:0/2674972652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64100ffa40 msgr2=0x7f64100ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.794+0000 7f64159a1700 1 --2- 172.21.15.121:0/2674972652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64100ffa40 0x7f64100ffec0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6400009910 tx=0x7f6400009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.794+0000 7f64159a1700 1 -- 172.21.15.121:0/2674972652 shutdown_connections 2024-04-05T21:30:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.794+0000 7f64159a1700 1 --2- 172.21.15.121:0/2674972652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64100ffa40 0x7f64100ffec0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.794+0000 7f64159a1700 1 --2- 172.21.15.121:0/2674972652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6410105a40 0x7f6410105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.794+0000 7f64159a1700 1 -- 172.21.15.121:0/2674972652 >> 172.21.15.121:0/2674972652 conn(0x7f64100fb570 msgr2=0x7f64100fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.794+0000 7f64159a1700 1 -- 172.21.15.121:0/2674972652 shutdown_connections 2024-04-05T21:30:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.794+0000 7f64159a1700 1 -- 172.21.15.121:0/2674972652 wait complete. 2024-04-05T21:30:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.794+0000 7f64159a1700 1 Processor -- start 2024-04-05T21:30:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f64159a1700 1 -- start start 2024-04-05T21:30:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f64159a1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64100ffa40 0x7f64100fbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f64159a1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6410105a40 0x7f64100fc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f64159a1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64100fc9e0 con 0x7f6410105a40 2024-04-05T21:30:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f64159a1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64100fcb20 con 0x7f64100ffa40 2024-04-05T21:30:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f640e7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6410105a40 0x7f64100fc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f640e7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6410105a40 0x7f64100fc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45698/0 (socket says 172.21.15.121:45698) 2024-04-05T21:30:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f640e7fc700 1 -- 172.21.15.121:0/2766227895 learned_addr learned my addr 172.21.15.121:0/2766227895 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f640effd700 1 --2- 172.21.15.121:0/2766227895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64100ffa40 0x7f64100fbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f640e7fc700 1 -- 172.21.15.121:0/2766227895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64100ffa40 msgr2=0x7f64100fbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f640e7fc700 1 --2- 172.21.15.121:0/2766227895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64100ffa40 0x7f64100fbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f640e7fc700 1 -- 172.21.15.121:0/2766227895 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64000095f0 con 0x7f6410105a40 2024-04-05T21:30:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f640effd700 1 --2- 172.21.15.121:0/2766227895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64100ffa40 0x7f64100fbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:30:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f640e7fc700 1 --2- 172.21.15.121:0/2766227895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6410105a40 0x7f64100fc4a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6400006010 tx=0x7f640000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f641499f700 1 -- 172.21.15.121:0/2766227895 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f640001d070 con 0x7f6410105a40 2024-04-05T21:30:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f641499f700 1 -- 172.21.15.121:0/2766227895 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f640000f460 con 0x7f6410105a40 2024-04-05T21:30:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f64159a1700 1 -- 172.21.15.121:0/2766227895 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64100fcda0 con 0x7f6410105a40 2024-04-05T21:30:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f641499f700 1 -- 172.21.15.121:0/2766227895 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6400021620 con 0x7f6410105a40 2024-04-05T21:30:57.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.795+0000 7f64159a1700 1 -- 172.21.15.121:0/2766227895 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64101a2b10 con 0x7f6410105a40 2024-04-05T21:30:57.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.796+0000 7f64159a1700 1 -- 172.21.15.121:0/2766227895 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6410108fe0 con 0x7f6410105a40 2024-04-05T21:30:57.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.797+0000 7f641499f700 1 -- 172.21.15.121:0/2766227895 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f640000fa40 con 0x7f6410105a40 2024-04-05T21:30:57.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.797+0000 7f641499f700 1 --2- 172.21.15.121:0/2766227895 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f63f806cce0 0x7f63f806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:57.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.797+0000 7f641499f700 1 -- 172.21.15.121:0/2766227895 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f640008d6e0 con 0x7f6410105a40 2024-04-05T21:30:57.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.797+0000 7f640effd700 1 --2- 172.21.15.121:0/2766227895 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f63f806cce0 0x7f63f806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:57.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.798+0000 7f640effd700 1 --2- 172.21.15.121:0/2766227895 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f63f806cce0 0x7f63f806f1a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6404005fd0 tx=0x7f6404005ee0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:57.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:57.802+0000 7f641499f700 1 -- 172.21.15.121:0/2766227895 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6400057800 con 0x7f6410105a40 2024-04-05T21:30:57.978 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:57 smithi121 ceph-mon[145859]: pgmap v106: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:57.979 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:57 smithi121 ceph-mon[145859]: from='client.34100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:57.979 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:57 smithi121 ceph-mon[145859]: from='client.34104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:57.979 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:57 smithi121 ceph-mon[145859]: Upgrade: Updating crash.smithi121 (1/2) 2024-04-05T21:30:57.979 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:57.979 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi121", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T21:30:57.979 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:57.979 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:57 smithi121 ceph-mon[145859]: Deploying daemon crash.smithi121 on smithi121 2024-04-05T21:30:57.979 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:57 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/214897865' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:30:57.979 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:57 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3275756144' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:30:58.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.018+0000 7f64159a1700 1 -- 172.21.15.121:0/2766227895 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f641002cce0 con 0x7f63f806cce0 2024-04-05T21:30:58.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.019+0000 7f641499f700 1 -- 172.21.15.121:0/2766227895 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f641002cce0 con 0x7f63f806cce0 2024-04-05T21:30:58.020 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:30:58.021 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:30:58.021 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:30:58.021 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:30:58.021 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:30:58.021 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:30:58.021 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:30:58.021 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:30:58.022 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "4/23 daemons upgraded", 2024-04-05T21:30:58.022 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading crash daemons", 2024-04-05T21:30:58.022 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:30:58.022 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:30:58.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.021+0000 7f64159a1700 1 -- 172.21.15.121:0/2766227895 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f63f806cce0 msgr2=0x7f63f806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:58.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.021+0000 7f64159a1700 1 --2- 172.21.15.121:0/2766227895 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f63f806cce0 0x7f63f806f1a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6404005fd0 tx=0x7f6404005ee0 comp rx=0 tx=0).stop 2024-04-05T21:30:58.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.021+0000 7f64159a1700 1 -- 172.21.15.121:0/2766227895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6410105a40 msgr2=0x7f64100fc4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:58.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.021+0000 7f64159a1700 1 --2- 172.21.15.121:0/2766227895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6410105a40 0x7f64100fc4a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6400006010 tx=0x7f640000be20 comp rx=0 tx=0).stop 2024-04-05T21:30:58.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.021+0000 7f64159a1700 1 -- 172.21.15.121:0/2766227895 shutdown_connections 2024-04-05T21:30:58.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.021+0000 7f64159a1700 1 --2- 172.21.15.121:0/2766227895 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f63f806cce0 0x7f63f806f1a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:58.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.021+0000 7f64159a1700 1 --2- 172.21.15.121:0/2766227895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f64100ffa40 0x7f64100fbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:58.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.021+0000 7f64159a1700 1 --2- 172.21.15.121:0/2766227895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6410105a40 0x7f64100fc4a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:58.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.021+0000 7f64159a1700 1 -- 172.21.15.121:0/2766227895 >> 172.21.15.121:0/2766227895 conn(0x7f64100fb570 msgr2=0x7f6410068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:58.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.021+0000 7f64159a1700 1 -- 172.21.15.121:0/2766227895 shutdown_connections 2024-04-05T21:30:58.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.022+0000 7f64159a1700 1 -- 172.21.15.121:0/2766227895 wait complete. 2024-04-05T21:30:58.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:57 smithi188 ceph-mon[107793]: pgmap v106: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:30:58.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:57 smithi188 ceph-mon[107793]: from='client.34100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:58.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:57 smithi188 ceph-mon[107793]: from='client.34104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:58.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:57 smithi188 ceph-mon[107793]: Upgrade: Updating crash.smithi121 (1/2) 2024-04-05T21:30:58.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:30:58.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi121", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T21:30:58.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:30:58.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:57 smithi188 ceph-mon[107793]: Deploying daemon crash.smithi121 on smithi121 2024-04-05T21:30:58.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:57 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/214897865' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:30:58.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:57 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3275756144' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:30:58.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.125+0000 7fc274551700 1 -- 172.21.15.121:0/2952694721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc26c105a30 msgr2=0x7fc26c105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:58.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.125+0000 7fc274551700 1 --2- 172.21.15.121:0/2952694721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc26c105a30 0x7fc26c105e10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009910 tx=0x7fc25c009c20 comp rx=0 tx=0).stop 2024-04-05T21:30:58.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.125+0000 7fc274551700 1 -- 172.21.15.121:0/2952694721 shutdown_connections 2024-04-05T21:30:58.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.125+0000 7fc274551700 1 --2- 172.21.15.121:0/2952694721 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc26c0ffb70 0x7fc26c0fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:58.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.125+0000 7fc274551700 1 --2- 172.21.15.121:0/2952694721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc26c105a30 0x7fc26c105e10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:58.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.125+0000 7fc274551700 1 -- 172.21.15.121:0/2952694721 >> 172.21.15.121:0/2952694721 conn(0x7fc26c0754f0 msgr2=0x7fc26c075900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:58.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.125+0000 7fc274551700 1 -- 172.21.15.121:0/2952694721 shutdown_connections 2024-04-05T21:30:58.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.125+0000 7fc274551700 1 -- 172.21.15.121:0/2952694721 wait complete. 2024-04-05T21:30:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc274551700 1 Processor -- start 2024-04-05T21:30:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc274551700 1 -- start start 2024-04-05T21:30:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc274551700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc26c0ffb70 0x7fc26c196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc274551700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc26c105a30 0x7fc26c196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc274551700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c197220 con 0x7fc26c105a30 2024-04-05T21:30:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc274551700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c19afb0 con 0x7fc26c0ffb70 2024-04-05T21:30:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc271aec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc26c105a30 0x7fc26c196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc271aec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc26c105a30 0x7fc26c196b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45712/0 (socket says 172.21.15.121:45712) 2024-04-05T21:30:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc271aec700 1 -- 172.21.15.121:0/4112632173 learned_addr learned my addr 172.21.15.121:0/4112632173 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:30:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc2722ed700 1 --2- 172.21.15.121:0/4112632173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc26c0ffb70 0x7fc26c196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc271aec700 1 -- 172.21.15.121:0/4112632173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc26c0ffb70 msgr2=0x7fc26c196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc271aec700 1 --2- 172.21.15.121:0/4112632173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc26c0ffb70 0x7fc26c196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:58.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc271aec700 1 -- 172.21.15.121:0/4112632173 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c0095f0 con 0x7fc26c105a30 2024-04-05T21:30:58.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc271aec700 1 --2- 172.21.15.121:0/4112632173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc26c105a30 0x7fc26c196b40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc26800d8d0 tx=0x7fc26800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:58.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.126+0000 7fc2637fe700 1 -- 172.21.15.121:0/4112632173 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2680098a0 con 0x7fc26c105a30 2024-04-05T21:30:58.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.127+0000 7fc2637fe700 1 -- 172.21.15.121:0/4112632173 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc268010460 con 0x7fc26c105a30 2024-04-05T21:30:58.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.127+0000 7fc2637fe700 1 -- 172.21.15.121:0/4112632173 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc268009ea0 con 0x7fc26c105a30 2024-04-05T21:30:58.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.127+0000 7fc274551700 1 -- 172.21.15.121:0/4112632173 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26c19b290 con 0x7fc26c105a30 2024-04-05T21:30:58.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.127+0000 7fc274551700 1 -- 172.21.15.121:0/4112632173 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26c19b7e0 con 0x7fc26c105a30 2024-04-05T21:30:58.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.127+0000 7fc274551700 1 -- 172.21.15.121:0/4112632173 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26c04ea90 con 0x7fc26c105a30 2024-04-05T21:30:58.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.128+0000 7fc2637fe700 1 -- 172.21.15.121:0/4112632173 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc2680105d0 con 0x7fc26c105a30 2024-04-05T21:30:58.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.128+0000 7fc2637fe700 1 --2- 172.21.15.121:0/4112632173 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc25806cce0 0x7fc25806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:30:58.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.128+0000 7fc2637fe700 1 -- 172.21.15.121:0/4112632173 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fc26808c2f0 con 0x7fc26c105a30 2024-04-05T21:30:58.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.128+0000 7fc2722ed700 1 --2- 172.21.15.121:0/4112632173 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc25806cce0 0x7fc25806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:30:58.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.129+0000 7fc2722ed700 1 --2- 172.21.15.121:0/4112632173 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc25806cce0 0x7fc25806f1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc25c006010 tx=0x7fc25c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:30:58.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.133+0000 7fc2637fe700 1 -- 172.21.15.121:0/4112632173 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc268056490 con 0x7fc26c105a30 2024-04-05T21:30:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.415+0000 7fc274551700 1 -- 172.21.15.121:0/4112632173 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc26c19bbd0 con 0x7fc26c105a30 2024-04-05T21:30:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.416+0000 7fc2637fe700 1 -- 172.21.15.121:0/4112632173 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fc268010880 con 0x7fc26c105a30 2024-04-05T21:30:58.417 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_OK 2024-04-05T21:30:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.418+0000 7fc274551700 1 -- 172.21.15.121:0/4112632173 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc25806cce0 msgr2=0x7fc25806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:58.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.418+0000 7fc274551700 1 --2- 172.21.15.121:0/4112632173 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc25806cce0 0x7fc25806f1a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc25c006010 tx=0x7fc25c01a040 comp rx=0 tx=0).stop 2024-04-05T21:30:58.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.418+0000 7fc274551700 1 -- 172.21.15.121:0/4112632173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc26c105a30 msgr2=0x7fc26c196b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:30:58.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.418+0000 7fc274551700 1 --2- 172.21.15.121:0/4112632173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc26c105a30 0x7fc26c196b40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc26800d8d0 tx=0x7fc26800dbe0 comp rx=0 tx=0).stop 2024-04-05T21:30:58.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.418+0000 7fc274551700 1 -- 172.21.15.121:0/4112632173 shutdown_connections 2024-04-05T21:30:58.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.418+0000 7fc274551700 1 --2- 172.21.15.121:0/4112632173 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc25806cce0 0x7fc25806f1a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:58.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.418+0000 7fc274551700 1 --2- 172.21.15.121:0/4112632173 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc26c0ffb70 0x7fc26c196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:58.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.418+0000 7fc274551700 1 --2- 172.21.15.121:0/4112632173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc26c105a30 0x7fc26c196b40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:30:58.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.418+0000 7fc274551700 1 -- 172.21.15.121:0/4112632173 >> 172.21.15.121:0/4112632173 conn(0x7fc26c0754f0 msgr2=0x7fc26c068530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:30:58.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.418+0000 7fc274551700 1 -- 172.21.15.121:0/4112632173 shutdown_connections 2024-04-05T21:30:58.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:30:58.418+0000 7fc274551700 1 -- 172.21.15.121:0/4112632173 wait complete. 2024-04-05T21:30:58.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:58 smithi121 ceph-mon[145859]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:58.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:58 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4112632173' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:30:59.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:58 smithi188 ceph-mon[107793]: from='client.34108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:59.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:58 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4112632173' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:30:59.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:59 smithi121 ceph-mon[145859]: from='client.34120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:30:59.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:30:59 smithi121 ceph-mon[145859]: pgmap v107: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:00.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:59 smithi188 ceph-mon[107793]: from='client.34120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:31:00.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:30:59 smithi188 ceph-mon[107793]: pgmap v107: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:00 smithi121 ceph-mon[145859]: pgmap v108: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:31:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:00 smithi188 ceph-mon[107793]: pgmap v108: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:31:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:03 smithi121 ceph-mon[145859]: pgmap v109: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:03 smithi121 ceph-mon[145859]: Upgrade: Updating crash.smithi188 (2/2) 2024-04-05T21:31:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:03 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:03 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi188", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T21:31:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:03 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:31:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:03 smithi121 ceph-mon[145859]: Deploying daemon crash.smithi188 on smithi188 2024-04-05T21:31:04.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:03 smithi188 ceph-mon[107793]: pgmap v109: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:04.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:03 smithi188 ceph-mon[107793]: Upgrade: Updating crash.smithi188 (2/2) 2024-04-05T21:31:04.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:03 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:04.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:03 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi188", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-05T21:31:04.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:03 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:31:04.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:03 smithi188 ceph-mon[107793]: Deploying daemon crash.smithi188 on smithi188 2024-04-05T21:31:05.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:05 smithi121 ceph-mon[145859]: pgmap v110: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:06.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:05 smithi188 ceph-mon[107793]: pgmap v110: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:07 smithi121 ceph-mon[145859]: pgmap v111: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:07 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:07 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:07 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:31:08.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:07 smithi188 ceph-mon[107793]: pgmap v111: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:08.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:07 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:08.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:07 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:08.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:07 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:31:09.924 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:09 smithi121 ceph-mon[145859]: pgmap v112: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:10.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:09 smithi188 ceph-mon[107793]: pgmap v112: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:10.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:10 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:10.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:10 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:10.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:10 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:10.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:10 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:11.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:10 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:11.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:10 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:11.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:10 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:11.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:10 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:11.907 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:11 smithi188 ceph-mon[107793]: pgmap v113: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:11.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:11 smithi121 ceph-mon[145859]: pgmap v113: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:13.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:13 smithi188 ceph-mon[107793]: pgmap v114: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:13.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:13 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:13.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:13 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:13.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:13 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:13.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:13 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:13.691 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:13 smithi121 ceph-mon[145859]: pgmap v114: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:13.691 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:13 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:13.692 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:13 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:13.692 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:13 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:13.692 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:13 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:15 smithi121 ceph-mon[145859]: pgmap v115: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:15.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:15.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:15.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:15 smithi188 ceph-mon[107793]: pgmap v115: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:16.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:31:16.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:31:17.402 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: pgmap v116: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:17.402 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:17.402 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:17.402 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:31:17.402 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:31:17.402 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:17.402 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:31:17.402 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:17.402 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:17.403 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:17.403 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:17.403 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi121"}]: dispatch 2024-04-05T21:31:17.403 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi121"}]': finished 2024-04-05T21:31:17.403 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi188"}]: dispatch 2024-04-05T21:31:17.403 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi188"}]': finished 2024-04-05T21:31:17.403 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-05T21:31:17.453 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: pgmap v116: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:17.453 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:17.453 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:17.453 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:31:17.453 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:31:17.453 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:17.453 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:31:17.453 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:17.454 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:17.454 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:17.454 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:17.454 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi121"}]: dispatch 2024-04-05T21:31:17.454 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi121"}]': finished 2024-04-05T21:31:17.454 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi188"}]: dispatch 2024-04-05T21:31:17.454 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi188"}]': finished 2024-04-05T21:31:17.454 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-05T21:31:18.442 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:18 smithi121 ceph-mon[145859]: Upgrade: Setting container_image for all crash 2024-04-05T21:31:18.442 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:18 smithi121 ceph-mon[145859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-05T21:31:18.442 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:18 smithi121 ceph-mon[145859]: Upgrade: osd.0 is safe to restart 2024-04-05T21:31:18.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:18 smithi188 ceph-mon[107793]: Upgrade: Setting container_image for all crash 2024-04-05T21:31:18.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:18 smithi188 ceph-mon[107793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-05T21:31:18.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:18 smithi188 ceph-mon[107793]: Upgrade: osd.0 is safe to restart 2024-04-05T21:31:19.659 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:19 smithi121 ceph-mon[145859]: pgmap v117: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:19.659 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:19 smithi121 ceph-mon[145859]: Upgrade: Updating osd.0 2024-04-05T21:31:19.659 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:19 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:19.659 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:19 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-05T21:31:19.660 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:19 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:31:19.660 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:19 smithi121 ceph-mon[145859]: Deploying daemon osd.0 on smithi121 2024-04-05T21:31:19.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:19 smithi188 ceph-mon[107793]: pgmap v117: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:19.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:19 smithi188 ceph-mon[107793]: Upgrade: Updating osd.0 2024-04-05T21:31:19.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:19 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:19.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:19 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-05T21:31:19.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:19 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:31:19.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:19 smithi188 ceph-mon[107793]: Deploying daemon osd.0 on smithi121 2024-04-05T21:31:21.653 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:21 smithi121 ceph-mon[145859]: pgmap v118: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:21 smithi188 ceph-mon[107793]: pgmap v118: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:21.989 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:21 smithi121 systemd[1]: Stopping Ceph osd.0 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:31:21.989 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:21 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-0[63821]: 2024-04-05T21:31:21.893+0000 7f0d36553700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T21:31:21.989 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:21 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-0[63821]: 2024-04-05T21:31:21.893+0000 7f0d36553700 -1 osd.0 48 *** Got signal Terminated *** 2024-04-05T21:31:21.989 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:21 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-0[63821]: 2024-04-05T21:31:21.893+0000 7f0d36553700 -1 osd.0 48 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T21:31:22.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:22 smithi188 ceph-mon[107793]: osd.0 marked itself down and dead 2024-04-05T21:31:22.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:22 smithi121 ceph-mon[145859]: osd.0 marked itself down and dead 2024-04-05T21:31:23.741 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:23 smithi121 bash[157049]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-0 2024-04-05T21:31:23.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:23 smithi121 ceph-mon[145859]: pgmap v119: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:23.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:23 smithi121 ceph-mon[145859]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T21:31:23.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:23 smithi121 ceph-mon[145859]: osdmap e49: 6 total, 5 up, 6 in 2024-04-05T21:31:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:23 smithi188 ceph-mon[107793]: pgmap v119: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:23.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:23 smithi188 ceph-mon[107793]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T21:31:23.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:23 smithi188 ceph-mon[107793]: osdmap e49: 6 total, 5 up, 6 in 2024-04-05T21:31:24.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:24 smithi188 ceph-mon[107793]: osdmap e50: 6 total, 5 up, 6 in 2024-04-05T21:31:24.945 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:24 smithi121 ceph-mon[145859]: osdmap e50: 6 total, 5 up, 6 in 2024-04-05T21:31:25.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:25 smithi121 ceph-mon[145859]: pgmap v122: 65 pgs: 9 stale+active+clean, 56 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T21:31:25.739 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:25 smithi121 systemd[1]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.0.service: Succeeded. 2024-04-05T21:31:25.739 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:25 smithi121 systemd[1]: Stopped Ceph osd.0 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:31:25.739 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:25 smithi121 systemd[1]: Starting Ceph osd.0 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:31:25.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:25 smithi188 ceph-mon[107793]: pgmap v122: 65 pgs: 9 stale+active+clean, 56 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T21:31:26.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:26 smithi121 ceph-mon[145859]: Health check failed: Degraded data redundancy: 67/333 objects degraded (20.120%), 19 pgs degraded (PG_DEGRADED) 2024-04-05T21:31:26.989 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:26 smithi121 bash[157324]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-05T21:31:26.989 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:26 smithi121 bash[157324]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-05T21:31:26.989 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:26 smithi121 bash[157324]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-05T21:31:26.990 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:26 smithi121 bash[157324]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-05T21:31:26.990 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:26 smithi121 bash[157324]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-05T21:31:26.990 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:26 smithi121 bash[157324]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-05T21:31:26.990 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:26 smithi121 bash[157324]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-05T21:31:27.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:26 smithi188 ceph-mon[107793]: Health check failed: Degraded data redundancy: 67/333 objects degraded (20.120%), 19 pgs degraded (PG_DEGRADED) 2024-04-05T21:31:27.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:27 smithi188 ceph-mon[107793]: pgmap v123: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 67/333 objects degraded (20.120%) 2024-04-05T21:31:27.845 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:27 smithi121 ceph-mon[145859]: pgmap v123: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 67/333 objects degraded (20.120%) 2024-04-05T21:31:28.108 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:27 smithi121 ceph-osd[157652]: -- 172.21.15.121:0/3614406012 <== mon.0 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561eb1deb1e0 con 0x561eb1dbac00 2024-04-05T21:31:28.489 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:28 smithi121 bash[157613]: 698c6ac230411c8daacbf35c22b1897ee39ddc49991e7a1ce032f9b87fee8e6f 2024-04-05T21:31:28.489 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:28 smithi121 systemd[1]: Started Ceph osd.0 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:31:28.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.541+0000 7f5da9c47700 1 -- 172.21.15.121:0/2476176893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5da4073a80 msgr2=0x7f5da410cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:28.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.541+0000 7f5da9c47700 1 --2- 172.21.15.121:0/2476176893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5da4073a80 0x7f5da410cee0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009910 tx=0x7f5d94009c20 comp rx=0 tx=0).stop 2024-04-05T21:31:28.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.542+0000 7f5da9c47700 1 -- 172.21.15.121:0/2476176893 shutdown_connections 2024-04-05T21:31:28.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.542+0000 7f5da9c47700 1 --2- 172.21.15.121:0/2476176893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5da4073a80 0x7f5da410cee0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:28.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.542+0000 7f5da9c47700 1 --2- 172.21.15.121:0/2476176893 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5da4073160 0x7f5da4073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:28.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.542+0000 7f5da9c47700 1 -- 172.21.15.121:0/2476176893 >> 172.21.15.121:0/2476176893 conn(0x7f5da40fc670 msgr2=0x7f5da40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:28.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.542+0000 7f5da9c47700 1 -- 172.21.15.121:0/2476176893 shutdown_connections 2024-04-05T21:31:28.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.542+0000 7f5da9c47700 1 -- 172.21.15.121:0/2476176893 wait complete. 2024-04-05T21:31:28.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.543+0000 7f5da9c47700 1 Processor -- start 2024-04-05T21:31:28.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.543+0000 7f5da9c47700 1 -- start start 2024-04-05T21:31:28.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.544+0000 7f5da9c47700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5da4073160 0x7f5da410c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:28.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.544+0000 7f5da9c47700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5da4073a80 0x7f5da410cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:28.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.544+0000 7f5da9c47700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da41068f0 con 0x7f5da4073a80 2024-04-05T21:31:28.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.544+0000 7f5da9c47700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da4106a60 con 0x7f5da4073160 2024-04-05T21:31:28.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.544+0000 7f5da2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5da4073a80 0x7f5da410cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:28.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.544+0000 7f5da2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5da4073a80 0x7f5da410cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50082/0 (socket says 172.21.15.121:50082) 2024-04-05T21:31:28.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.544+0000 7f5da2ffd700 1 -- 172.21.15.121:0/2104313761 learned_addr learned my addr 172.21.15.121:0/2104313761 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:31:28.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.544+0000 7f5da37fe700 1 --2- 172.21.15.121:0/2104313761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5da4073160 0x7f5da410c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:28.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.544+0000 7f5da2ffd700 1 -- 172.21.15.121:0/2104313761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5da4073160 msgr2=0x7f5da410c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:28.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.544+0000 7f5da2ffd700 1 --2- 172.21.15.121:0/2104313761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5da4073160 0x7f5da410c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:28.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.544+0000 7f5da2ffd700 1 -- 172.21.15.121:0/2104313761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d940095f0 con 0x7f5da4073a80 2024-04-05T21:31:28.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.545+0000 7f5da2ffd700 1 --2- 172.21.15.121:0/2104313761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5da4073a80 0x7f5da410cdb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5d94000c00 tx=0x7f5d9400be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:28.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.545+0000 7f5da0ff9700 1 -- 172.21.15.121:0/2104313761 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d9401d070 con 0x7f5da4073a80 2024-04-05T21:31:28.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.545+0000 7f5da9c47700 1 -- 172.21.15.121:0/2104313761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da4106ce0 con 0x7f5da4073a80 2024-04-05T21:31:28.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.545+0000 7f5da9c47700 1 -- 172.21.15.121:0/2104313761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da4107170 con 0x7f5da4073a80 2024-04-05T21:31:28.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.545+0000 7f5da0ff9700 1 -- 172.21.15.121:0/2104313761 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d9400b5c0 con 0x7f5da4073a80 2024-04-05T21:31:28.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.545+0000 7f5da0ff9700 1 -- 172.21.15.121:0/2104313761 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d940039d0 con 0x7f5da4073a80 2024-04-05T21:31:28.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.546+0000 7f5da9c47700 1 -- 172.21.15.121:0/2104313761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d90005320 con 0x7f5da4073a80 2024-04-05T21:31:28.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.547+0000 7f5da0ff9700 1 -- 172.21.15.121:0/2104313761 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5d94003c00 con 0x7f5da4073a80 2024-04-05T21:31:28.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.548+0000 7f5da0ff9700 1 --2- 172.21.15.121:0/2104313761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5d8c06cd30 0x7f5d8c06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:28.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.548+0000 7f5da0ff9700 1 -- 172.21.15.121:0/2104313761 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f5d9408e150 con 0x7f5da4073a80 2024-04-05T21:31:28.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.548+0000 7f5da37fe700 1 --2- 172.21.15.121:0/2104313761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5d8c06cd30 0x7f5d8c06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:28.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.548+0000 7f5da37fe700 1 --2- 172.21.15.121:0/2104313761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5d8c06cd30 0x7f5d8c06f1f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5d98009c60 tx=0x7f5d98009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:28.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.553+0000 7f5da0ff9700 1 -- 172.21.15.121:0/2104313761 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5d940582d0 con 0x7f5da4073a80 2024-04-05T21:31:28.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.777+0000 7f5da9c47700 1 -- 172.21.15.121:0/2104313761 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d90000bf0 con 0x7f5d8c06cd30 2024-04-05T21:31:28.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.779+0000 7f5da0ff9700 1 -- 172.21.15.121:0/2104313761 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f5d90000bf0 con 0x7f5d8c06cd30 2024-04-05T21:31:28.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.781+0000 7f5da9c47700 1 -- 172.21.15.121:0/2104313761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5d8c06cd30 msgr2=0x7f5d8c06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:28.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.781+0000 7f5da9c47700 1 --2- 172.21.15.121:0/2104313761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5d8c06cd30 0x7f5d8c06f1f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5d98009c60 tx=0x7f5d98009260 comp rx=0 tx=0).stop 2024-04-05T21:31:28.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.781+0000 7f5da9c47700 1 -- 172.21.15.121:0/2104313761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5da4073a80 msgr2=0x7f5da410cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:28.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.781+0000 7f5da9c47700 1 --2- 172.21.15.121:0/2104313761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5da4073a80 0x7f5da410cdb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5d94000c00 tx=0x7f5d9400be10 comp rx=0 tx=0).stop 2024-04-05T21:31:28.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.781+0000 7f5da9c47700 1 -- 172.21.15.121:0/2104313761 shutdown_connections 2024-04-05T21:31:28.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.781+0000 7f5da9c47700 1 --2- 172.21.15.121:0/2104313761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5d8c06cd30 0x7f5d8c06f1f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:28.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.781+0000 7f5da9c47700 1 --2- 172.21.15.121:0/2104313761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5da4073160 0x7f5da410c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:28.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.781+0000 7f5da9c47700 1 --2- 172.21.15.121:0/2104313761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5da4073a80 0x7f5da410cdb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:28.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.781+0000 7f5da9c47700 1 -- 172.21.15.121:0/2104313761 >> 172.21.15.121:0/2104313761 conn(0x7f5da40fc670 msgr2=0x7f5da41101f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:28.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.781+0000 7f5da9c47700 1 -- 172.21.15.121:0/2104313761 shutdown_connections 2024-04-05T21:31:28.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.781+0000 7f5da9c47700 1 -- 172.21.15.121:0/2104313761 wait complete. 2024-04-05T21:31:28.793 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:31:28.845 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:28 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-0[157629]: 2024-04-05T21:31:28.513+0000 7fd0ea0427c0 -1 Falling back to public interface 2024-04-05T21:31:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.892+0000 7f4548f67700 1 -- 172.21.15.121:0/3231349762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45441026b0 msgr2=0x7f454410aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.892+0000 7f4548f67700 1 --2- 172.21.15.121:0/3231349762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45441026b0 0x7f454410aba0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4534009910 tx=0x7f4534009c20 comp rx=0 tx=0).stop 2024-04-05T21:31:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.892+0000 7f4548f67700 1 -- 172.21.15.121:0/3231349762 shutdown_connections 2024-04-05T21:31:28.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.892+0000 7f4548f67700 1 --2- 172.21.15.121:0/3231349762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45441026b0 0x7f454410aba0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:28.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.892+0000 7f4548f67700 1 --2- 172.21.15.121:0/3231349762 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4544101d90 0x7f4544102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:28.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.892+0000 7f4548f67700 1 -- 172.21.15.121:0/3231349762 >> 172.21.15.121:0/3231349762 conn(0x7f45440fb540 msgr2=0x7f45440fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:28.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.892+0000 7f4548f67700 1 -- 172.21.15.121:0/3231349762 shutdown_connections 2024-04-05T21:31:28.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.892+0000 7f4548f67700 1 -- 172.21.15.121:0/3231349762 wait complete. 2024-04-05T21:31:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.893+0000 7f4548f67700 1 Processor -- start 2024-04-05T21:31:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.893+0000 7f4548f67700 1 -- start start 2024-04-05T21:31:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.893+0000 7f4548f67700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4544101d90 0x7f45440ff9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.893+0000 7f4548f67700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45441026b0 0x7f45440fff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:28.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.893+0000 7f4548f67700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4544100470 con 0x7f4544101d90 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.893+0000 7f4548f67700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441005b0 con 0x7f45441026b0 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.893+0000 7f454259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4544101d90 0x7f45440ff9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.893+0000 7f454259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4544101d90 0x7f45440ff9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50094/0 (socket says 172.21.15.121:50094) 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.893+0000 7f454259c700 1 -- 172.21.15.121:0/3140435913 learned_addr learned my addr 172.21.15.121:0/3140435913 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.893+0000 7f4541d9b700 1 --2- 172.21.15.121:0/3140435913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45441026b0 0x7f45440fff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.894+0000 7f454259c700 1 -- 172.21.15.121:0/3140435913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45441026b0 msgr2=0x7f45440fff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.894+0000 7f454259c700 1 --2- 172.21.15.121:0/3140435913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45441026b0 0x7f45440fff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.894+0000 7f454259c700 1 -- 172.21.15.121:0/3140435913 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45340095f0 con 0x7f4544101d90 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.894+0000 7f454259c700 1 --2- 172.21.15.121:0/3140435913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4544101d90 0x7f45440ff9f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f453800c8a0 tx=0x7f453800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.894+0000 7f45337fe700 1 -- 172.21.15.121:0/3140435913 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45380078e0 con 0x7f4544101d90 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.894+0000 7f4548f67700 1 -- 172.21.15.121:0/3140435913 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45441a2840 con 0x7f4544101d90 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.894+0000 7f45337fe700 1 -- 172.21.15.121:0/3140435913 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f453800f450 con 0x7f4544101d90 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.894+0000 7f45337fe700 1 -- 172.21.15.121:0/3140435913 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f453800e5e0 con 0x7f4544101d90 2024-04-05T21:31:28.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.894+0000 7f4548f67700 1 -- 172.21.15.121:0/3140435913 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45441a2c50 con 0x7f4544101d90 2024-04-05T21:31:28.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.895+0000 7f4548f67700 1 -- 172.21.15.121:0/3140435913 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45441082a0 con 0x7f4544101d90 2024-04-05T21:31:28.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.895+0000 7f45337fe700 1 -- 172.21.15.121:0/3140435913 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f453800f5c0 con 0x7f4544101d90 2024-04-05T21:31:28.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.896+0000 7f45337fe700 1 --2- 172.21.15.121:0/3140435913 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f452c06cce0 0x7f452c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:28.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.896+0000 7f45337fe700 1 -- 172.21.15.121:0/3140435913 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f453808c120 con 0x7f4544101d90 2024-04-05T21:31:28.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.896+0000 7f4541d9b700 1 --2- 172.21.15.121:0/3140435913 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f452c06cce0 0x7f452c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:28.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.896+0000 7f4541d9b700 1 --2- 172.21.15.121:0/3140435913 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f452c06cce0 0x7f452c06f1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f45441011e0 tx=0x7f453401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:28.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:28.900+0000 7f45337fe700 1 -- 172.21.15.121:0/3140435913 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4538056240 con 0x7f4544101d90 2024-04-05T21:31:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.119+0000 7f4548f67700 1 -- 172.21.15.121:0/3140435913 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f454402cda0 con 0x7f452c06cce0 2024-04-05T21:31:29.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.121+0000 7f45337fe700 1 -- 172.21.15.121:0/3140435913 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f454402cda0 con 0x7f452c06cce0 2024-04-05T21:31:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.123+0000 7f4548f67700 1 -- 172.21.15.121:0/3140435913 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f452c06cce0 msgr2=0x7f452c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.123+0000 7f4548f67700 1 --2- 172.21.15.121:0/3140435913 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f452c06cce0 0x7f452c06f1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f45441011e0 tx=0x7f453401a040 comp rx=0 tx=0).stop 2024-04-05T21:31:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.123+0000 7f4548f67700 1 -- 172.21.15.121:0/3140435913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4544101d90 msgr2=0x7f45440ff9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.123+0000 7f4548f67700 1 --2- 172.21.15.121:0/3140435913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4544101d90 0x7f45440ff9f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f453800c8a0 tx=0x7f453800cbb0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.123+0000 7f4548f67700 1 -- 172.21.15.121:0/3140435913 shutdown_connections 2024-04-05T21:31:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.123+0000 7f4548f67700 1 --2- 172.21.15.121:0/3140435913 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f452c06cce0 0x7f452c06f1a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.123+0000 7f4548f67700 1 --2- 172.21.15.121:0/3140435913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4544101d90 0x7f45440ff9f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.123+0000 7f4548f67700 1 --2- 172.21.15.121:0/3140435913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f45441026b0 0x7f45440fff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.123+0000 7f4548f67700 1 -- 172.21.15.121:0/3140435913 >> 172.21.15.121:0/3140435913 conn(0x7f45440fb540 msgr2=0x7f45441053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.123+0000 7f4548f67700 1 -- 172.21.15.121:0/3140435913 shutdown_connections 2024-04-05T21:31:29.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.123+0000 7f4548f67700 1 -- 172.21.15.121:0/3140435913 wait complete. 2024-04-05T21:31:29.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.230+0000 7f2ab2954700 1 -- 172.21.15.121:0/237242694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aac0ffbd0 msgr2=0x7f2aac100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:29.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.230+0000 7f2ab2954700 1 --2- 172.21.15.121:0/237242694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aac0ffbd0 0x7f2aac100050 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0009910 tx=0x7f2aa0009c20 comp rx=0 tx=0).stop 2024-04-05T21:31:29.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.230+0000 7f2ab2954700 1 -- 172.21.15.121:0/237242694 shutdown_connections 2024-04-05T21:31:29.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.230+0000 7f2ab2954700 1 --2- 172.21.15.121:0/237242694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aac0ffbd0 0x7f2aac100050 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.230+0000 7f2ab2954700 1 --2- 172.21.15.121:0/237242694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aac105a40 0x7f2aac105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.230+0000 7f2ab2954700 1 -- 172.21.15.121:0/237242694 >> 172.21.15.121:0/237242694 conn(0x7f2aac0755a0 msgr2=0x7f2aac0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:29.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.230+0000 7f2ab2954700 1 -- 172.21.15.121:0/237242694 shutdown_connections 2024-04-05T21:31:29.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.230+0000 7f2ab2954700 1 -- 172.21.15.121:0/237242694 wait complete. 2024-04-05T21:31:29.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.230+0000 7f2ab2954700 1 Processor -- start 2024-04-05T21:31:29.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.230+0000 7f2ab2954700 1 -- start start 2024-04-05T21:31:29.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.231+0000 7f2ab2954700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aac0ffbd0 0x7f2aac19c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:29.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.231+0000 7f2ab2954700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aac105a40 0x7f2aac19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:29.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.231+0000 7f2ab2954700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac19d170 con 0x7f2aac0ffbd0 2024-04-05T21:31:29.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.231+0000 7f2ab2954700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac196620 con 0x7f2aac105a40 2024-04-05T21:31:29.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.231+0000 7f2aabfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aac0ffbd0 0x7f2aac19c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:29.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.231+0000 7f2aabfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aac0ffbd0 0x7f2aac19c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50116/0 (socket says 172.21.15.121:50116) 2024-04-05T21:31:29.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.231+0000 7f2aabfff700 1 -- 172.21.15.121:0/892104411 learned_addr learned my addr 172.21.15.121:0/892104411 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:31:29.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.231+0000 7f2aab7fe700 1 --2- 172.21.15.121:0/892104411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aac105a40 0x7f2aac19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:29.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.231+0000 7f2aabfff700 1 -- 172.21.15.121:0/892104411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aac105a40 msgr2=0x7f2aac19cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:29.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.231+0000 7f2aabfff700 1 --2- 172.21.15.121:0/892104411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aac105a40 0x7f2aac19cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.231+0000 7f2aabfff700 1 -- 172.21.15.121:0/892104411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa00095f0 con 0x7f2aac0ffbd0 2024-04-05T21:31:29.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.231+0000 7f2aabfff700 1 --2- 172.21.15.121:0/892104411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aac0ffbd0 0x7f2aac19c5a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c00b4b0 tx=0x7f2a9c00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:29.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.232+0000 7f2aa97fa700 1 -- 172.21.15.121:0/892104411 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a9c011800 con 0x7f2aac0ffbd0 2024-04-05T21:31:29.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.232+0000 7f2ab2954700 1 -- 172.21.15.121:0/892104411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aac196900 con 0x7f2aac0ffbd0 2024-04-05T21:31:29.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.232+0000 7f2aa97fa700 1 -- 172.21.15.121:0/892104411 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a9c011e00 con 0x7f2aac0ffbd0 2024-04-05T21:31:29.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.232+0000 7f2aa97fa700 1 -- 172.21.15.121:0/892104411 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a9c010e60 con 0x7f2aac0ffbd0 2024-04-05T21:31:29.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.232+0000 7f2ab2954700 1 -- 172.21.15.121:0/892104411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aac196e50 con 0x7f2aac0ffbd0 2024-04-05T21:31:29.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.233+0000 7f2ab2954700 1 -- 172.21.15.121:0/892104411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aac04ea90 con 0x7f2aac0ffbd0 2024-04-05T21:31:29.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.234+0000 7f2aa97fa700 1 -- 172.21.15.121:0/892104411 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2a9c011960 con 0x7f2aac0ffbd0 2024-04-05T21:31:29.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.234+0000 7f2aa97fa700 1 --2- 172.21.15.121:0/892104411 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a9406cce0 0x7f2a9406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:29.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.234+0000 7f2aa97fa700 1 -- 172.21.15.121:0/892104411 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f2a9c08c4a0 con 0x7f2aac0ffbd0 2024-04-05T21:31:29.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.234+0000 7f2aab7fe700 1 --2- 172.21.15.121:0/892104411 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a9406cce0 0x7f2a9406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:29.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.234+0000 7f2aab7fe700 1 --2- 172.21.15.121:0/892104411 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a9406cce0 0x7f2a9406f1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2aac104e40 tx=0x7f2aa001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:29.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.238+0000 7f2aa97fa700 1 -- 172.21.15.121:0/892104411 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2a9c05a0d0 con 0x7f2aac0ffbd0 2024-04-05T21:31:29.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.456+0000 7f2ab2954700 1 -- 172.21.15.121:0/892104411 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2aac1972a0 con 0x7f2a9406cce0 2024-04-05T21:31:29.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.467+0000 7f2aa97fa700 1 -- 172.21.15.121:0/892104411 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2aac1972a0 con 0x7f2a9406cce0 2024-04-05T21:31:29.468 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:31:29.468 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (2m) 17s ago 13m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:31:29.468 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (14m) 17s ago 14m 8158k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:31:29.468 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (12m) 19s ago 12m 8044k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:31:29.468 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (29s) 17s ago 13m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:31:29.468 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (23s) 19s ago 12m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:31:29.468 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (2m) 17s ago 13m 80.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:31:29.468 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (7m) 17s ago 7m 64.5M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:31:29.468 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (7m) 17s ago 7m 134M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:31:29.468 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (7m) 19s ago 7m 110M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (7m) 19s ago 7m 62.2M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (4m) 17s ago 15m 503M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (3m) 19s ago 12m 450M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (62s) 17s ago 15m 38.2M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (44s) 19s ago 12m 27.9M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (3m) 17s ago 13m 17.6M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (3m) 19s ago 12m 16.4M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 starting - - - 1227M 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (11m) 17s ago 11m 243M 1227M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (10m) 17s ago 10m 182M 1227M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (10m) 19s ago 10m 288M 2251M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (9m) 19s ago 9m 246M 2251M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (9m) 19s ago 9m 224M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:31:29.469 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (2m) 17s ago 13m 62.8M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:31:29.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.469+0000 7f2ab2954700 1 -- 172.21.15.121:0/892104411 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a9406cce0 msgr2=0x7f2a9406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:29.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.469+0000 7f2ab2954700 1 --2- 172.21.15.121:0/892104411 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a9406cce0 0x7f2a9406f1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2aac104e40 tx=0x7f2aa001a040 comp rx=0 tx=0).stop 2024-04-05T21:31:29.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.469+0000 7f2ab2954700 1 -- 172.21.15.121:0/892104411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aac0ffbd0 msgr2=0x7f2aac19c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:29.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.469+0000 7f2ab2954700 1 --2- 172.21.15.121:0/892104411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aac0ffbd0 0x7f2aac19c5a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c00b4b0 tx=0x7f2a9c00b7c0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.469+0000 7f2ab2954700 1 -- 172.21.15.121:0/892104411 shutdown_connections 2024-04-05T21:31:29.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.469+0000 7f2ab2954700 1 --2- 172.21.15.121:0/892104411 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a9406cce0 0x7f2a9406f1a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.469+0000 7f2ab2954700 1 --2- 172.21.15.121:0/892104411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2aac0ffbd0 0x7f2aac19c5a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.469+0000 7f2ab2954700 1 --2- 172.21.15.121:0/892104411 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2aac105a40 0x7f2aac19cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.469+0000 7f2ab2954700 1 -- 172.21.15.121:0/892104411 >> 172.21.15.121:0/892104411 conn(0x7f2aac0755a0 msgr2=0x7f2aac103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:29.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.469+0000 7f2ab2954700 1 -- 172.21.15.121:0/892104411 shutdown_connections 2024-04-05T21:31:29.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.469+0000 7f2ab2954700 1 -- 172.21.15.121:0/892104411 wait complete. 2024-04-05T21:31:29.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:29 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:29.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:29 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:29.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:29 smithi121 ceph-mon[145859]: pgmap v124: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 67/333 objects degraded (20.120%) 2024-04-05T21:31:29.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:29 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:31:29.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:29 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:29.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:29 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:29.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:29 smithi188 ceph-mon[107793]: pgmap v124: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 67/333 objects degraded (20.120%) 2024-04-05T21:31:29.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:29 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:31:29.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.571+0000 7fc9bc168700 1 -- 172.21.15.121:0/884113652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9b41017a0 msgr2=0x7fc9b41057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:29.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.571+0000 7fc9bc168700 1 --2- 172.21.15.121:0/884113652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9b41017a0 0x7fc9b41057f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4009910 tx=0x7fc9a4009c20 comp rx=0 tx=0).stop 2024-04-05T21:31:29.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.571+0000 7fc9bc168700 1 -- 172.21.15.121:0/884113652 shutdown_connections 2024-04-05T21:31:29.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.571+0000 7fc9bc168700 1 --2- 172.21.15.121:0/884113652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9b41017a0 0x7fc9b41057f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.571+0000 7fc9bc168700 1 --2- 172.21.15.121:0/884113652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9b4100df0 0x7fc9b41011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.571+0000 7fc9bc168700 1 -- 172.21.15.121:0/884113652 >> 172.21.15.121:0/884113652 conn(0x7fc9b40fc6a0 msgr2=0x7fc9b40feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:29.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.571+0000 7fc9bc168700 1 -- 172.21.15.121:0/884113652 shutdown_connections 2024-04-05T21:31:29.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.572+0000 7fc9bc168700 1 -- 172.21.15.121:0/884113652 wait complete. 2024-04-05T21:31:29.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.572+0000 7fc9bc168700 1 Processor -- start 2024-04-05T21:31:29.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.572+0000 7fc9bc168700 1 -- start start 2024-04-05T21:31:29.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.572+0000 7fc9bc168700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9b4100df0 0x7fc9b4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:29.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.572+0000 7fc9bc168700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9b41017a0 0x7fc9b40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:29.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.572+0000 7fc9bc168700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b4073b10 con 0x7fc9b4100df0 2024-04-05T21:31:29.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.572+0000 7fc9bc168700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b4073c50 con 0x7fc9b41017a0 2024-04-05T21:31:29.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.572+0000 7fc9b9f04700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9b4100df0 0x7fc9b4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:29.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.572+0000 7fc9b9f04700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9b4100df0 0x7fc9b4073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50136/0 (socket says 172.21.15.121:50136) 2024-04-05T21:31:29.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.572+0000 7fc9b9f04700 1 -- 172.21.15.121:0/3456743262 learned_addr learned my addr 172.21.15.121:0/3456743262 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:31:29.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.573+0000 7fc9b9703700 1 --2- 172.21.15.121:0/3456743262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9b41017a0 0x7fc9b40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:29.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.573+0000 7fc9b9f04700 1 -- 172.21.15.121:0/3456743262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9b41017a0 msgr2=0x7fc9b40735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:29.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.573+0000 7fc9b9f04700 1 --2- 172.21.15.121:0/3456743262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9b41017a0 0x7fc9b40735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.573+0000 7fc9b9f04700 1 -- 172.21.15.121:0/3456743262 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a40095f0 con 0x7fc9b4100df0 2024-04-05T21:31:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.573+0000 7fc9b9f04700 1 --2- 172.21.15.121:0/3456743262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9b4100df0 0x7fc9b4073090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000c8f0 tx=0x7fc9b000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.573+0000 7fc9aaffd700 1 -- 172.21.15.121:0/3456743262 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9b0007a70 con 0x7fc9b4100df0 2024-04-05T21:31:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.573+0000 7fc9aaffd700 1 -- 172.21.15.121:0/3456743262 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc9b0004cd0 con 0x7fc9b4100df0 2024-04-05T21:31:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.573+0000 7fc9bc168700 1 -- 172.21.15.121:0/3456743262 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b41a28f0 con 0x7fc9b4100df0 2024-04-05T21:31:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.573+0000 7fc9aaffd700 1 -- 172.21.15.121:0/3456743262 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc9b0005580 con 0x7fc9b4100df0 2024-04-05T21:31:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.573+0000 7fc9bc168700 1 -- 172.21.15.121:0/3456743262 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b41a2d00 con 0x7fc9b4100df0 2024-04-05T21:31:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.574+0000 7fc9bc168700 1 -- 172.21.15.121:0/3456743262 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b41091f0 con 0x7fc9b4100df0 2024-04-05T21:31:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.574+0000 7fc9aaffd700 1 -- 172.21.15.121:0/3456743262 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc9b0004750 con 0x7fc9b4100df0 2024-04-05T21:31:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.575+0000 7fc9aaffd700 1 --2- 172.21.15.121:0/3456743262 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc9a006cce0 0x7fc9a006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.575+0000 7fc9aaffd700 1 -- 172.21.15.121:0/3456743262 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fc9b0013070 con 0x7fc9b4100df0 2024-04-05T21:31:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.575+0000 7fc9b9703700 1 --2- 172.21.15.121:0/3456743262 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc9a006cce0 0x7fc9a006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:29.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.575+0000 7fc9b9703700 1 --2- 172.21.15.121:0/3456743262 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc9a006cce0 0x7fc9a006f1a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4074880 tx=0x7fc9a401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:29.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.579+0000 7fc9aaffd700 1 -- 172.21.15.121:0/3456743262 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc9b0056440 con 0x7fc9b4100df0 2024-04-05T21:31:29.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.863+0000 7fc9bc168700 1 -- 172.21.15.121:0/3456743262 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc9b404ea90 con 0x7fc9b4100df0 2024-04-05T21:31:29.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.864+0000 7fc9aaffd700 1 -- 172.21.15.121:0/3456743262 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+693 (secure 0 0 0) 0x7fc9b0059a60 con 0x7fc9b4100df0 2024-04-05T21:31:29.865 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:31:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:31:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:31:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:31:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:31:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:31:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:31:29.865 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:31:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5 2024-04-05T21:31:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:31:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:31:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:31:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:31:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:31:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 9, 2024-04-05T21:31:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:31:29.866 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:31:29.866 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:31:29.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.866+0000 7fc9bc168700 1 -- 172.21.15.121:0/3456743262 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc9a006cce0 msgr2=0x7fc9a006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:29.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.866+0000 7fc9bc168700 1 --2- 172.21.15.121:0/3456743262 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc9a006cce0 0x7fc9a006f1a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4074880 tx=0x7fc9a401a040 comp rx=0 tx=0).stop 2024-04-05T21:31:29.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.866+0000 7fc9bc168700 1 -- 172.21.15.121:0/3456743262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9b4100df0 msgr2=0x7fc9b4073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:29.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.866+0000 7fc9bc168700 1 --2- 172.21.15.121:0/3456743262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9b4100df0 0x7fc9b4073090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000c8f0 tx=0x7fc9b000ccb0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.866+0000 7fc9bc168700 1 -- 172.21.15.121:0/3456743262 shutdown_connections 2024-04-05T21:31:29.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.866+0000 7fc9bc168700 1 --2- 172.21.15.121:0/3456743262 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc9a006cce0 0x7fc9a006f1a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.866+0000 7fc9bc168700 1 --2- 172.21.15.121:0/3456743262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9b4100df0 0x7fc9b4073090 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.866+0000 7fc9bc168700 1 --2- 172.21.15.121:0/3456743262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc9b41017a0 0x7fc9b40735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.866+0000 7fc9bc168700 1 -- 172.21.15.121:0/3456743262 >> 172.21.15.121:0/3456743262 conn(0x7fc9b40fc6a0 msgr2=0x7fc9b40fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:29.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.866+0000 7fc9bc168700 1 -- 172.21.15.121:0/3456743262 shutdown_connections 2024-04-05T21:31:29.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.866+0000 7fc9bc168700 1 -- 172.21.15.121:0/3456743262 wait complete. 2024-04-05T21:31:29.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.972+0000 7f50bb7d6700 1 -- 172.21.15.121:0/509619264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50b40fef70 msgr2=0x7f50b40ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:29.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.972+0000 7f50bb7d6700 1 --2- 172.21.15.121:0/509619264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50b40fef70 0x7f50b40ff3f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009910 tx=0x7f50b0009c20 comp rx=0 tx=0).stop 2024-04-05T21:31:29.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.972+0000 7f50bb7d6700 1 -- 172.21.15.121:0/509619264 shutdown_connections 2024-04-05T21:31:29.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.972+0000 7f50bb7d6700 1 --2- 172.21.15.121:0/509619264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50b40fef70 0x7f50b40ff3f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.972+0000 7f50bb7d6700 1 --2- 172.21.15.121:0/509619264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50b4105b50 0x7f50b4105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.972+0000 7f50bb7d6700 1 -- 172.21.15.121:0/509619264 >> 172.21.15.121:0/509619264 conn(0x7f50b4075e30 msgr2=0x7f50b40fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:29.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.972+0000 7f50bb7d6700 1 -- 172.21.15.121:0/509619264 shutdown_connections 2024-04-05T21:31:29.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.972+0000 7f50bb7d6700 1 -- 172.21.15.121:0/509619264 wait complete. 2024-04-05T21:31:29.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50bb7d6700 1 Processor -- start 2024-04-05T21:31:29.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50bb7d6700 1 -- start start 2024-04-05T21:31:29.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50bb7d6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50b40fef70 0x7f50b419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50bb7d6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50b4105b50 0x7f50b419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50bb7d6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b419d0e0 con 0x7f50b4105b50 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50bb7d6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4196590 con 0x7f50b40fef70 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50b8d71700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50b4105b50 0x7f50b419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50b8d71700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50b4105b50 0x7f50b419ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50148/0 (socket says 172.21.15.121:50148) 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50b8d71700 1 -- 172.21.15.121:0/2944345686 learned_addr learned my addr 172.21.15.121:0/2944345686 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50b9572700 1 --2- 172.21.15.121:0/2944345686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50b40fef70 0x7f50b419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50b8d71700 1 -- 172.21.15.121:0/2944345686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50b40fef70 msgr2=0x7f50b419c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50b8d71700 1 --2- 172.21.15.121:0/2944345686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50b40fef70 0x7f50b419c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50b8d71700 1 -- 172.21.15.121:0/2944345686 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b00095f0 con 0x7f50b4105b50 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50b8d71700 1 --2- 172.21.15.121:0/2944345686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50b4105b50 0x7f50b419ca50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f50b0006010 tx=0x7f50b000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.973+0000 7f50aa7fc700 1 -- 172.21.15.121:0/2944345686 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50b001d070 con 0x7f50b4105b50 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.974+0000 7f50aa7fc700 1 -- 172.21.15.121:0/2944345686 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50b000f460 con 0x7f50b4105b50 2024-04-05T21:31:29.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.974+0000 7f50bb7d6700 1 -- 172.21.15.121:0/2944345686 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b4196810 con 0x7f50b4105b50 2024-04-05T21:31:29.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.974+0000 7f50aa7fc700 1 -- 172.21.15.121:0/2944345686 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50b0021620 con 0x7f50b4105b50 2024-04-05T21:31:29.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.974+0000 7f50bb7d6700 1 -- 172.21.15.121:0/2944345686 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b4196d00 con 0x7f50b4105b50 2024-04-05T21:31:29.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.974+0000 7f50bb7d6700 1 -- 172.21.15.121:0/2944345686 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b4109120 con 0x7f50b4105b50 2024-04-05T21:31:29.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.975+0000 7f50aa7fc700 1 -- 172.21.15.121:0/2944345686 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f50b000f5d0 con 0x7f50b4105b50 2024-04-05T21:31:29.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.975+0000 7f50aa7fc700 1 --2- 172.21.15.121:0/2944345686 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f50a006cce0 0x7f50a006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:29.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.975+0000 7f50aa7fc700 1 -- 172.21.15.121:0/2944345686 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f50b008d7a0 con 0x7f50b4105b50 2024-04-05T21:31:29.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.975+0000 7f50b9572700 1 --2- 172.21.15.121:0/2944345686 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f50a006cce0 0x7f50a006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:29.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.976+0000 7f50b9572700 1 --2- 172.21.15.121:0/2944345686 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f50a006cce0 0x7f50a006f1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f50a4009a10 tx=0x7f50a4009190 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:29.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:29.980+0000 7f50aa7fc700 1 -- 172.21.15.121:0/2944345686 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f50b005b350 con 0x7f50b4105b50 2024-04-05T21:31:30.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.248+0000 7f50bb7d6700 1 -- 172.21.15.121:0/2944345686 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f50b404ea90 con 0x7f50b4105b50 2024-04-05T21:31:30.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.248+0000 7f50aa7fc700 1 -- 172.21.15.121:0/2944345686 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f50b005aee0 con 0x7f50b4105b50 2024-04-05T21:31:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:e21 2024-04-05T21:31:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:31:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:31:30.249 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:31:30.249 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 21 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:30:52.246236+0000 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:31:30.250 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 46 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:up {0=14476,1=24255} 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:failed 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{0:14476} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:31:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{0:14488} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/4279084130,v1:172.21.15.121:6827/4279084130] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:31:30.252 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{1:24255} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:31:30.252 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{1:24265} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:31:30.252 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:31:30.252 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:31:30.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.250+0000 7f50bb7d6700 1 -- 172.21.15.121:0/2944345686 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f50a006cce0 msgr2=0x7f50a006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:30.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.250+0000 7f50bb7d6700 1 --2- 172.21.15.121:0/2944345686 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f50a006cce0 0x7f50a006f1a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f50a4009a10 tx=0x7f50a4009190 comp rx=0 tx=0).stop 2024-04-05T21:31:30.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.250+0000 7f50bb7d6700 1 -- 172.21.15.121:0/2944345686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50b4105b50 msgr2=0x7f50b419ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:30.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.250+0000 7f50bb7d6700 1 --2- 172.21.15.121:0/2944345686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50b4105b50 0x7f50b419ca50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f50b0006010 tx=0x7f50b000bd70 comp rx=0 tx=0).stop 2024-04-05T21:31:30.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.250+0000 7f50bb7d6700 1 -- 172.21.15.121:0/2944345686 shutdown_connections 2024-04-05T21:31:30.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.250+0000 7f50bb7d6700 1 --2- 172.21.15.121:0/2944345686 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f50a006cce0 0x7f50a006f1a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.250+0000 7f50bb7d6700 1 --2- 172.21.15.121:0/2944345686 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50b40fef70 0x7f50b419c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.250+0000 7f50bb7d6700 1 --2- 172.21.15.121:0/2944345686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50b4105b50 0x7f50b419ca50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.250+0000 7f50bb7d6700 1 -- 172.21.15.121:0/2944345686 >> 172.21.15.121:0/2944345686 conn(0x7f50b4075e30 msgr2=0x7f50b4104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:30.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.251+0000 7f50bb7d6700 1 -- 172.21.15.121:0/2944345686 shutdown_connections 2024-04-05T21:31:30.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.251+0000 7f50bb7d6700 1 -- 172.21.15.121:0/2944345686 wait complete. 2024-04-05T21:31:30.253 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 21 2024-04-05T21:31:30.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.355+0000 7fe467114700 1 -- 172.21.15.121:0/3128749868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe460069020 msgr2=0x7fe460105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:30.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.355+0000 7fe467114700 1 --2- 172.21.15.121:0/3128749868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe460069020 0x7fe460105790 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe44c009910 tx=0x7fe44c009c20 comp rx=0 tx=0).stop 2024-04-05T21:31:30.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.355+0000 7fe467114700 1 -- 172.21.15.121:0/3128749868 shutdown_connections 2024-04-05T21:31:30.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.355+0000 7fe467114700 1 --2- 172.21.15.121:0/3128749868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe460069020 0x7fe460105790 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.355+0000 7fe467114700 1 --2- 172.21.15.121:0/3128749868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe460068670 0x7fe460068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.355+0000 7fe467114700 1 -- 172.21.15.121:0/3128749868 >> 172.21.15.121:0/3128749868 conn(0x7fe4600755a0 msgr2=0x7fe4600759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:30.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.355+0000 7fe467114700 1 -- 172.21.15.121:0/3128749868 shutdown_connections 2024-04-05T21:31:30.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.355+0000 7fe467114700 1 -- 172.21.15.121:0/3128749868 wait complete. 2024-04-05T21:31:30.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.356+0000 7fe467114700 1 Processor -- start 2024-04-05T21:31:30.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.356+0000 7fe467114700 1 -- start start 2024-04-05T21:31:30.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.356+0000 7fe467114700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe460068670 0x7fe4600ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:30.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.356+0000 7fe467114700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe460069020 0x7fe4601001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.356+0000 7fe467114700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe460100720 con 0x7fe460068670 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.356+0000 7fe467114700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe460100860 con 0x7fe460069020 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.356+0000 7fe464eb0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe460068670 0x7fe4600ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.356+0000 7fe464eb0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe460068670 0x7fe4600ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50158/0 (socket says 172.21.15.121:50158) 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.356+0000 7fe464eb0700 1 -- 172.21.15.121:0/118941384 learned_addr learned my addr 172.21.15.121:0/118941384 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.357+0000 7fe45ffff700 1 --2- 172.21.15.121:0/118941384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe460069020 0x7fe4601001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.357+0000 7fe464eb0700 1 -- 172.21.15.121:0/118941384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe460069020 msgr2=0x7fe4601001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.357+0000 7fe464eb0700 1 --2- 172.21.15.121:0/118941384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe460069020 0x7fe4601001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.357+0000 7fe464eb0700 1 -- 172.21.15.121:0/118941384 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44c0095f0 con 0x7fe460068670 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.357+0000 7fe464eb0700 1 --2- 172.21.15.121:0/118941384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe460068670 0x7fe4600ffca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe45400c930 tx=0x7fe45400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.357+0000 7fe45dffb700 1 -- 172.21.15.121:0/118941384 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe454007ad0 con 0x7fe460068670 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.357+0000 7fe467114700 1 -- 172.21.15.121:0/118941384 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4601a2810 con 0x7fe460068670 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.357+0000 7fe45dffb700 1 -- 172.21.15.121:0/118941384 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe454007c30 con 0x7fe460068670 2024-04-05T21:31:30.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.357+0000 7fe45dffb700 1 -- 172.21.15.121:0/118941384 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe454018630 con 0x7fe460068670 2024-04-05T21:31:30.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.357+0000 7fe467114700 1 -- 172.21.15.121:0/118941384 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4601a2b70 con 0x7fe460068670 2024-04-05T21:31:30.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.358+0000 7fe467114700 1 -- 172.21.15.121:0/118941384 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4601090a0 con 0x7fe460068670 2024-04-05T21:31:30.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.358+0000 7fe45dffb700 1 -- 172.21.15.121:0/118941384 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe45400f450 con 0x7fe460068670 2024-04-05T21:31:30.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.359+0000 7fe45dffb700 1 --2- 172.21.15.121:0/118941384 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe45006cce0 0x7fe45006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:30.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.359+0000 7fe45dffb700 1 -- 172.21.15.121:0/118941384 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fe45408c720 con 0x7fe460068670 2024-04-05T21:31:30.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.359+0000 7fe45ffff700 1 --2- 172.21.15.121:0/118941384 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe45006cce0 0x7fe45006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:30.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.359+0000 7fe45ffff700 1 --2- 172.21.15.121:0/118941384 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe45006cce0 0x7fe45006f1a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00b5c0 tx=0x7fe44c005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:30.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.363+0000 7fe45dffb700 1 -- 172.21.15.121:0/118941384 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe4540568f0 con 0x7fe460068670 2024-04-05T21:31:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:30 smithi121 ceph-mon[145859]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:31:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:30 smithi121 ceph-mon[145859]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:31:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:30 smithi121 ceph-mon[145859]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:31:30.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:30 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3456743262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:30.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:30 smithi188 ceph-mon[107793]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:31:30.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:30 smithi188 ceph-mon[107793]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:31:30.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:30 smithi188 ceph-mon[107793]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:31:30.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:30 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3456743262' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:30.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.598+0000 7fe467114700 1 -- 172.21.15.121:0/118941384 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe46002cd50 con 0x7fe45006cce0 2024-04-05T21:31:30.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.600+0000 7fe45dffb700 1 -- 172.21.15.121:0/118941384 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fe46002cd50 con 0x7fe45006cce0 2024-04-05T21:31:30.601 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:31:30.601 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:31:30.601 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:31:30.601 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:31:30.601 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:31:30.601 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:31:30.601 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:31:30.601 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:31:30.601 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:31:30.601 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "6/23 daemons upgraded", 2024-04-05T21:31:30.601 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading osd daemons", 2024-04-05T21:31:30.602 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:31:30.602 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:31:30.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.602+0000 7fe467114700 1 -- 172.21.15.121:0/118941384 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe45006cce0 msgr2=0x7fe45006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:30.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.602+0000 7fe467114700 1 --2- 172.21.15.121:0/118941384 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe45006cce0 0x7fe45006f1a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe44c00b5c0 tx=0x7fe44c005fb0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.602+0000 7fe467114700 1 -- 172.21.15.121:0/118941384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe460068670 msgr2=0x7fe4600ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:30.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.602+0000 7fe467114700 1 --2- 172.21.15.121:0/118941384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe460068670 0x7fe4600ffca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe45400c930 tx=0x7fe45400ccf0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.602+0000 7fe467114700 1 -- 172.21.15.121:0/118941384 shutdown_connections 2024-04-05T21:31:30.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.602+0000 7fe467114700 1 --2- 172.21.15.121:0/118941384 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe45006cce0 0x7fe45006f1a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.602+0000 7fe467114700 1 --2- 172.21.15.121:0/118941384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe460068670 0x7fe4600ffca0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.602+0000 7fe467114700 1 --2- 172.21.15.121:0/118941384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe460069020 0x7fe4601001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.602+0000 7fe467114700 1 -- 172.21.15.121:0/118941384 >> 172.21.15.121:0/118941384 conn(0x7fe4600755a0 msgr2=0x7fe460105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:30.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.602+0000 7fe467114700 1 -- 172.21.15.121:0/118941384 shutdown_connections 2024-04-05T21:31:30.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.602+0000 7fe467114700 1 -- 172.21.15.121:0/118941384 wait complete. 2024-04-05T21:31:30.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.708+0000 7f8f49bb3700 1 -- 172.21.15.121:0/2675352652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44105a40 msgr2=0x7f8f44105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:30.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.708+0000 7f8f49bb3700 1 --2- 172.21.15.121:0/2675352652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44105a40 0x7f8f44105e20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009910 tx=0x7f8f34009c20 comp rx=0 tx=0).stop 2024-04-05T21:31:30.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.708+0000 7f8f49bb3700 1 -- 172.21.15.121:0/2675352652 shutdown_connections 2024-04-05T21:31:30.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.708+0000 7f8f49bb3700 1 --2- 172.21.15.121:0/2675352652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f440ffa40 0x7f8f440ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.708+0000 7f8f49bb3700 1 --2- 172.21.15.121:0/2675352652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44105a40 0x7f8f44105e20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.708+0000 7f8f49bb3700 1 -- 172.21.15.121:0/2675352652 >> 172.21.15.121:0/2675352652 conn(0x7f8f440fb570 msgr2=0x7f8f440fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:30.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.708+0000 7f8f49bb3700 1 -- 172.21.15.121:0/2675352652 shutdown_connections 2024-04-05T21:31:30.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.708+0000 7f8f49bb3700 1 -- 172.21.15.121:0/2675352652 wait complete. 2024-04-05T21:31:30.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.709+0000 7f8f49bb3700 1 Processor -- start 2024-04-05T21:31:30.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.709+0000 7f8f49bb3700 1 -- start start 2024-04-05T21:31:30.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.709+0000 7f8f49bb3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f440ffa40 0x7f8f440fbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:30.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.709+0000 7f8f49bb3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44105a40 0x7f8f440fc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:30.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.709+0000 7f8f49bb3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f440fdca0 con 0x7f8f44105a40 2024-04-05T21:31:30.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.709+0000 7f8f49bb3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f440fde10 con 0x7f8f440ffa40 2024-04-05T21:31:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.709+0000 7f8f3bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44105a40 0x7f8f440fc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.709+0000 7f8f3bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44105a40 0x7f8f440fc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50174/0 (socket says 172.21.15.121:50174) 2024-04-05T21:31:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.709+0000 7f8f3bfff700 1 -- 172.21.15.121:0/570622438 learned_addr learned my addr 172.21.15.121:0/570622438 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:31:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.709+0000 7f8f437fe700 1 --2- 172.21.15.121:0/570622438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f440ffa40 0x7f8f440fbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.710+0000 7f8f3bfff700 1 -- 172.21.15.121:0/570622438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f440ffa40 msgr2=0x7f8f440fbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.710+0000 7f8f3bfff700 1 --2- 172.21.15.121:0/570622438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f440ffa40 0x7f8f440fbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.710+0000 7f8f3bfff700 1 -- 172.21.15.121:0/570622438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f340095f0 con 0x7f8f44105a40 2024-04-05T21:31:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.710+0000 7f8f3bfff700 1 --2- 172.21.15.121:0/570622438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44105a40 0x7f8f440fc430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00e920 tx=0x7f8f2c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.710+0000 7f8f417fa700 1 -- 172.21.15.121:0/570622438 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f2c00c9d0 con 0x7f8f44105a40 2024-04-05T21:31:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.710+0000 7f8f417fa700 1 -- 172.21.15.121:0/570622438 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f2c010430 con 0x7f8f44105a40 2024-04-05T21:31:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.710+0000 7f8f49bb3700 1 -- 172.21.15.121:0/570622438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f440fca30 con 0x7f8f44105a40 2024-04-05T21:31:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.710+0000 7f8f49bb3700 1 -- 172.21.15.121:0/570622438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4419a9b0 con 0x7f8f44105a40 2024-04-05T21:31:30.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.710+0000 7f8f417fa700 1 -- 172.21.15.121:0/570622438 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f2c018650 con 0x7f8f44105a40 2024-04-05T21:31:30.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.711+0000 7f8f417fa700 1 -- 172.21.15.121:0/570622438 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8f2c00cb30 con 0x7f8f44105a40 2024-04-05T21:31:30.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.712+0000 7f8f417fa700 1 --2- 172.21.15.121:0/570622438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8f3006cce0 0x7f8f3006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:31:30.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.712+0000 7f8f417fa700 1 -- 172.21.15.121:0/570622438 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f8f2c014070 con 0x7f8f44105a40 2024-04-05T21:31:30.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.712+0000 7f8f437fe700 1 --2- 172.21.15.121:0/570622438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8f3006cce0 0x7f8f3006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:31:30.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.712+0000 7f8f437fe700 1 --2- 172.21.15.121:0/570622438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8f3006cce0 0x7f8f3006f1a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009fd0 tx=0x7f8f34005940 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:31:30.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.712+0000 7f8f49bb3700 1 -- 172.21.15.121:0/570622438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f240052f0 con 0x7f8f44105a40 2024-04-05T21:31:30.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:30.717+0000 7f8f417fa700 1 -- 172.21.15.121:0/570622438 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8f2c052ab0 con 0x7f8f44105a40 2024-04-05T21:31:31.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:31.006+0000 7f8f49bb3700 1 -- 172.21.15.121:0/570622438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8f24005160 con 0x7f8f44105a40 2024-04-05T21:31:31.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:31.007+0000 7f8f417fa700 1 -- 172.21.15.121:0/570622438 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1328 (secure 0 0 0) 0x7f8f2c056860 con 0x7f8f44105a40 2024-04-05T21:31:31.008 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 67/333 objects degraded (20.120%), 19 pgs degraded 2024-04-05T21:31:31.008 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-05T21:31:31.008 INFO:teuthology.orchestra.run.smithi121.stdout: osd.0 (root=default,host=smithi121) is down 2024-04-05T21:31:31.008 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 67/333 objects degraded (20.120%), 19 pgs degraded 2024-04-05T21:31:31.008 INFO:teuthology.orchestra.run.smithi121.stdout: pg 1.0 is active+undersized+degraded, acting [3,1] 2024-04-05T21:31:31.008 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.0 is active+undersized+degraded, acting [3,1] 2024-04-05T21:31:31.008 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.1 is active+undersized+degraded, acting [2,1] 2024-04-05T21:31:31.009 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-04-05T21:31:31.009 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.4 is active+undersized+degraded, acting [1,4] 2024-04-05T21:31:31.009 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.5 is active+undersized+degraded, acting [3,4] 2024-04-05T21:31:31.009 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.8 is active+undersized+degraded, acting [3,5] 2024-04-05T21:31:31.009 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.9 is active+undersized+degraded, acting [1,4] 2024-04-05T21:31:31.009 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-04-05T21:31:31.009 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.f is active+undersized+degraded, acting [4,5] 2024-04-05T21:31:31.009 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-04-05T21:31:31.009 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.12 is active+undersized+degraded, acting [3,1] 2024-04-05T21:31:31.009 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.13 is active+undersized+degraded, acting [4,2] 2024-04-05T21:31:31.009 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.15 is active+undersized+degraded, acting [1,3] 2024-04-05T21:31:31.009 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.19 is active+undersized+degraded, acting [4,2] 2024-04-05T21:31:31.009 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.1b is active+undersized+degraded, acting [1,5] 2024-04-05T21:31:31.010 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.1d is active+undersized+degraded, acting [3,5] 2024-04-05T21:31:31.010 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.1e is active+undersized+degraded, acting [2,5] 2024-04-05T21:31:31.010 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.1f is active+undersized+degraded, acting [3,4] 2024-04-05T21:31:31.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:31.009+0000 7f8f49bb3700 1 -- 172.21.15.121:0/570622438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8f3006cce0 msgr2=0x7f8f3006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:31.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:31.009+0000 7f8f49bb3700 1 --2- 172.21.15.121:0/570622438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8f3006cce0 0x7f8f3006f1a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8f34009fd0 tx=0x7f8f34005940 comp rx=0 tx=0).stop 2024-04-05T21:31:31.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:31.009+0000 7f8f49bb3700 1 -- 172.21.15.121:0/570622438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44105a40 msgr2=0x7f8f440fc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:31:31.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:31.009+0000 7f8f49bb3700 1 --2- 172.21.15.121:0/570622438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44105a40 0x7f8f440fc430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00e920 tx=0x7f8f2c00ec30 comp rx=0 tx=0).stop 2024-04-05T21:31:31.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:31.009+0000 7f8f49bb3700 1 -- 172.21.15.121:0/570622438 shutdown_connections 2024-04-05T21:31:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:31.009+0000 7f8f49bb3700 1 --2- 172.21.15.121:0/570622438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8f3006cce0 0x7f8f3006f1a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:31.009+0000 7f8f49bb3700 1 --2- 172.21.15.121:0/570622438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f440ffa40 0x7f8f440fbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:31.009+0000 7f8f49bb3700 1 --2- 172.21.15.121:0/570622438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f44105a40 0x7f8f440fc430 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:31:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:31.009+0000 7f8f49bb3700 1 -- 172.21.15.121:0/570622438 >> 172.21.15.121:0/570622438 conn(0x7f8f440fb570 msgr2=0x7f8f44068f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:31:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:31.009+0000 7f8f49bb3700 1 -- 172.21.15.121:0/570622438 shutdown_connections 2024-04-05T21:31:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:31:31.009+0000 7f8f49bb3700 1 -- 172.21.15.121:0/570622438 wait complete. 2024-04-05T21:31:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:31 smithi121 ceph-mon[145859]: pgmap v125: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 67/333 objects degraded (20.120%) 2024-04-05T21:31:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:31:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:31 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2944345686' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:31:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:31 smithi121 ceph-mon[145859]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:31:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:31 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/570622438' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:31:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:31 smithi188 ceph-mon[107793]: pgmap v125: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 67/333 objects degraded (20.120%) 2024-04-05T21:31:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:31:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:31 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2944345686' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:31:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:31 smithi188 ceph-mon[107793]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:31:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:31 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/570622438' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:31:32.739 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:32 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-0[157629]: 2024-04-05T21:31:32.503+0000 7fd0ea0427c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-05T21:31:33.193 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:32 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-0[157629]: 2024-04-05T21:31:32.913+0000 7fd0ea0427c0 -1 osd.0 48 log_to_monitors true 2024-04-05T21:31:33.193 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:33 smithi121 ceph-mon[145859]: pgmap v126: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 67/333 objects degraded (20.120%) 2024-04-05T21:31:33.194 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:33 smithi121 ceph-mon[145859]: from='osd.0 [v2:172.21.15.121:6802/2472269034,v1:172.21.15.121:6803/2472269034]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-05T21:31:33.489 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 21:31:33 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-0[157629]: 2024-04-05T21:31:33.204+0000 7fd0db49c700 -1 osd.0 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T21:31:33.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:33 smithi188 ceph-mon[107793]: pgmap v126: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 67/333 objects degraded (20.120%) 2024-04-05T21:31:33.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:33 smithi188 ceph-mon[107793]: from='osd.0 [v2:172.21.15.121:6802/2472269034,v1:172.21.15.121:6803/2472269034]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-05T21:31:34.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:34 smithi121 ceph-mon[145859]: from='osd.0 [v2:172.21.15.121:6802/2472269034,v1:172.21.15.121:6803/2472269034]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-05T21:31:34.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:34 smithi121 ceph-mon[145859]: osdmap e51: 6 total, 5 up, 6 in 2024-04-05T21:31:34.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:34 smithi121 ceph-mon[145859]: from='osd.0 [v2:172.21.15.121:6802/2472269034,v1:172.21.15.121:6803/2472269034]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-05T21:31:34.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:34 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:34.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:34 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:34.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:34 smithi188 ceph-mon[107793]: from='osd.0 [v2:172.21.15.121:6802/2472269034,v1:172.21.15.121:6803/2472269034]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-05T21:31:34.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:34 smithi188 ceph-mon[107793]: osdmap e51: 6 total, 5 up, 6 in 2024-04-05T21:31:34.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:34 smithi188 ceph-mon[107793]: from='osd.0 [v2:172.21.15.121:6802/2472269034,v1:172.21.15.121:6803/2472269034]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-05T21:31:34.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:34 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:34.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:34 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:35.327 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:35 smithi121 ceph-mon[145859]: pgmap v128: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 67/333 objects degraded (20.120%) 2024-04-05T21:31:35.327 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:35 smithi121 ceph-mon[145859]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T21:31:35.327 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:35 smithi121 ceph-mon[145859]: osd.0 [v2:172.21.15.121:6802/2472269034,v1:172.21.15.121:6803/2472269034] boot 2024-04-05T21:31:35.327 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:35 smithi121 ceph-mon[145859]: osdmap e52: 6 total, 6 up, 6 in 2024-04-05T21:31:35.327 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:31:35.327 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:35.328 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:35.328 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:35 smithi121 ceph-mon[145859]: osdmap e53: 6 total, 6 up, 6 in 2024-04-05T21:31:35.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:35 smithi188 ceph-mon[107793]: pgmap v128: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 67/333 objects degraded (20.120%) 2024-04-05T21:31:35.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:35 smithi188 ceph-mon[107793]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T21:31:35.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:35 smithi188 ceph-mon[107793]: osd.0 [v2:172.21.15.121:6802/2472269034,v1:172.21.15.121:6803/2472269034] boot 2024-04-05T21:31:35.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:35 smithi188 ceph-mon[107793]: osdmap e52: 6 total, 6 up, 6 in 2024-04-05T21:31:35.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-05T21:31:35.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:35.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:35.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:35 smithi188 ceph-mon[107793]: osdmap e53: 6 total, 6 up, 6 in 2024-04-05T21:31:36.356 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:36 smithi121 ceph-mon[145859]: Health check update: Degraded data redundancy: 12/333 objects degraded (3.604%), 3 pgs degraded (PG_DEGRADED) 2024-04-05T21:31:36.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:36 smithi188 ceph-mon[107793]: Health check update: Degraded data redundancy: 12/333 objects degraded (3.604%), 3 pgs degraded (PG_DEGRADED) 2024-04-05T21:31:37.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:37 smithi188 ceph-mon[107793]: pgmap v131: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.0 KiB/s rd, 5 op/s; 12/333 objects degraded (3.604%) 2024-04-05T21:31:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:37 smithi121 ceph-mon[145859]: pgmap v131: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 3.0 KiB/s rd, 5 op/s; 12/333 objects degraded (3.604%) 2024-04-05T21:31:39.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:39 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:39.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:39 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:39.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:39 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:31:39.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:39 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:31:39.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:39 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:39.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:39 smithi121 ceph-mon[145859]: pgmap v132: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 12/333 objects degraded (3.604%) 2024-04-05T21:31:39.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:39 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:31:39.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:39 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:39.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:39 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:39.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:39 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:39.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:39 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T21:31:39.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:39 smithi121 ceph-mon[145859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T21:31:39.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:39 smithi121 ceph-mon[145859]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-05T21:31:39.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:39 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:39.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:39 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:39.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:39 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:31:39.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:39 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:31:39.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:39 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:39.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:39 smithi188 ceph-mon[107793]: pgmap v132: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 12/333 objects degraded (3.604%) 2024-04-05T21:31:39.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:39 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:31:39.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:39 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:39.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:39 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:39.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:39 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:31:39.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:39 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T21:31:39.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:39 smithi188 ceph-mon[107793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T21:31:39.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:39 smithi188 ceph-mon[107793]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-04-05T21:31:40.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:40 smithi188 ceph-mon[107793]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/333 objects degraded (3.604%), 3 pgs degraded) 2024-04-05T21:31:40.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:40 smithi188 ceph-mon[107793]: Cluster is now healthy 2024-04-05T21:31:40.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:40 smithi121 ceph-mon[145859]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/333 objects degraded (3.604%), 3 pgs degraded) 2024-04-05T21:31:40.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:40 smithi121 ceph-mon[145859]: Cluster is now healthy 2024-04-05T21:31:41.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:41 smithi188 ceph-mon[107793]: pgmap v133: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-05T21:31:41.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:41 smithi121 ceph-mon[145859]: pgmap v133: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-05T21:31:43.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:43 smithi188 ceph-mon[107793]: pgmap v134: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-05T21:31:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:43 smithi121 ceph-mon[145859]: pgmap v134: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-05T21:31:45.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:45 smithi121 ceph-mon[145859]: pgmap v135: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-05T21:31:45.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:45.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:31:45.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:45.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:45 smithi188 ceph-mon[107793]: pgmap v135: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-05T21:31:45.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:45.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:31:45.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:47.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:47 smithi188 ceph-mon[107793]: pgmap v136: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-05T21:31:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:47 smithi121 ceph-mon[145859]: pgmap v136: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 4 op/s 2024-04-05T21:31:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:49 smithi121 ceph-mon[145859]: pgmap v137: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:49.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:49 smithi188 ceph-mon[107793]: pgmap v137: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:51 smithi121 ceph-mon[145859]: pgmap v138: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:51 smithi188 ceph-mon[107793]: pgmap v138: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:53 smithi121 ceph-mon[145859]: pgmap v139: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:53 smithi188 ceph-mon[107793]: pgmap v139: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:54.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:54 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T21:31:54.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:54 smithi121 ceph-mon[145859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T21:31:54.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:54 smithi121 ceph-mon[145859]: Upgrade: osd.1 is safe to restart 2024-04-05T21:31:54.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:54 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T21:31:54.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:54 smithi188 ceph-mon[107793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-05T21:31:54.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:54 smithi188 ceph-mon[107793]: Upgrade: osd.1 is safe to restart 2024-04-05T21:31:55.389 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:55 smithi121 ceph-mon[145859]: pgmap v140: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:55.389 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:55.389 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-05T21:31:55.389 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:31:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:55 smithi188 ceph-mon[107793]: pgmap v140: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:55.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:31:55.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-05T21:31:55.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:31:56.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:56 smithi121 ceph-mon[145859]: Upgrade: Updating osd.1 2024-04-05T21:31:56.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:56 smithi121 ceph-mon[145859]: Deploying daemon osd.1 on smithi121 2024-04-05T21:31:56.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:56 smithi188 ceph-mon[107793]: Upgrade: Updating osd.1 2024-04-05T21:31:56.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:56 smithi188 ceph-mon[107793]: Deploying daemon osd.1 on smithi121 2024-04-05T21:31:57.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:57 smithi121 ceph-mon[145859]: pgmap v141: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:57 smithi188 ceph-mon[107793]: pgmap v141: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:31:58.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:31:58 smithi121 systemd[1]: Stopping Ceph osd.1 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:31:58.990 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:31:58 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1[73558]: 2024-04-05T21:31:58.898+0000 7fd9c85b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T21:31:58.990 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:31:58 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1[73558]: 2024-04-05T21:31:58.898+0000 7fd9c85b1700 -1 osd.1 53 *** Got signal Terminated *** 2024-04-05T21:31:58.990 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:31:58 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1[73558]: 2024-04-05T21:31:58.898+0000 7fd9c85b1700 -1 osd.1 53 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T21:31:59.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:59 smithi121 ceph-mon[145859]: pgmap v142: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:59.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:31:59 smithi121 ceph-mon[145859]: osd.1 marked itself down and dead 2024-04-05T21:31:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:59 smithi188 ceph-mon[107793]: pgmap v142: 65 pgs: 65 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:31:59.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:31:59 smithi188 ceph-mon[107793]: osd.1 marked itself down and dead 2024-04-05T21:32:00.323 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:00 smithi121 bash[163749]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1 2024-04-05T21:32:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:00 smithi121 ceph-mon[145859]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T21:32:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:00 smithi121 ceph-mon[145859]: osdmap e54: 6 total, 5 up, 6 in 2024-04-05T21:32:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:32:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:00 smithi121 ceph-mon[145859]: osdmap e55: 6 total, 5 up, 6 in 2024-04-05T21:32:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:00 smithi188 ceph-mon[107793]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T21:32:00.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:00 smithi188 ceph-mon[107793]: osdmap e54: 6 total, 5 up, 6 in 2024-04-05T21:32:00.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:00.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:32:00.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:00 smithi188 ceph-mon[107793]: osdmap e55: 6 total, 5 up, 6 in 2024-04-05T21:32:01.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.121+0000 7fea59262700 1 -- 172.21.15.121:0/1920895779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea54068670 msgr2=0x7fea54068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:01.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.122+0000 7fea59262700 1 --2- 172.21.15.121:0/1920895779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea54068670 0x7fea54068a50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fea44009910 tx=0x7fea44009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:01.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.122+0000 7fea59262700 1 -- 172.21.15.121:0/1920895779 shutdown_connections 2024-04-05T21:32:01.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.122+0000 7fea59262700 1 --2- 172.21.15.121:0/1920895779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea54069020 0x7fea54105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.122+0000 7fea59262700 1 --2- 172.21.15.121:0/1920895779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea54068670 0x7fea54068a50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.122+0000 7fea59262700 1 -- 172.21.15.121:0/1920895779 >> 172.21.15.121:0/1920895779 conn(0x7fea540755a0 msgr2=0x7fea540759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:01.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.122+0000 7fea59262700 1 -- 172.21.15.121:0/1920895779 shutdown_connections 2024-04-05T21:32:01.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.122+0000 7fea59262700 1 -- 172.21.15.121:0/1920895779 wait complete. 2024-04-05T21:32:01.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.122+0000 7fea59262700 1 Processor -- start 2024-04-05T21:32:01.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.122+0000 7fea59262700 1 -- start start 2024-04-05T21:32:01.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea59262700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea54068670 0x7fea540ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:01.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea59262700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea54069020 0x7fea541001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:01.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea59262700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea54100720 con 0x7fea54068670 2024-04-05T21:32:01.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea59262700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea54100860 con 0x7fea54069020 2024-04-05T21:32:01.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea52ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea54068670 0x7fea540ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:01.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea52ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea54068670 0x7fea540ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54004/0 (socket says 172.21.15.121:54004) 2024-04-05T21:32:01.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea52ffd700 1 -- 172.21.15.121:0/725684802 learned_addr learned my addr 172.21.15.121:0/725684802 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:01.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea527fc700 1 --2- 172.21.15.121:0/725684802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea54069020 0x7fea541001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:01.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea52ffd700 1 -- 172.21.15.121:0/725684802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea54069020 msgr2=0x7fea541001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:01.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea52ffd700 1 --2- 172.21.15.121:0/725684802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea54069020 0x7fea541001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea52ffd700 1 -- 172.21.15.121:0/725684802 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea440095f0 con 0x7fea54068670 2024-04-05T21:32:01.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea52ffd700 1 --2- 172.21.15.121:0/725684802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea54068670 0x7fea540ffca0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea440092e0 tx=0x7fea4400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:01.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea3bfff700 1 -- 172.21.15.121:0/725684802 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea4401d070 con 0x7fea54068670 2024-04-05T21:32:01.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea3bfff700 1 -- 172.21.15.121:0/725684802 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea44005e00 con 0x7fea54068670 2024-04-05T21:32:01.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea59262700 1 -- 172.21.15.121:0/725684802 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea54100ae0 con 0x7fea54068670 2024-04-05T21:32:01.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea3bfff700 1 -- 172.21.15.121:0/725684802 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea4400fe90 con 0x7fea54068670 2024-04-05T21:32:01.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.123+0000 7fea59262700 1 -- 172.21.15.121:0/725684802 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea541a2b70 con 0x7fea54068670 2024-04-05T21:32:01.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.124+0000 7fea59262700 1 -- 172.21.15.121:0/725684802 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea54109130 con 0x7fea54068670 2024-04-05T21:32:01.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.125+0000 7fea3bfff700 1 -- 172.21.15.121:0/725684802 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fea4400f6d0 con 0x7fea54068670 2024-04-05T21:32:01.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.125+0000 7fea3bfff700 1 --2- 172.21.15.121:0/725684802 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fea3c06cd30 0x7fea3c06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:01.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.125+0000 7fea527fc700 1 --2- 172.21.15.121:0/725684802 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fea3c06cd30 0x7fea3c06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:01.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.125+0000 7fea3bfff700 1 -- 172.21.15.121:0/725684802 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fea4408d7d0 con 0x7fea54068670 2024-04-05T21:32:01.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.126+0000 7fea527fc700 1 --2- 172.21.15.121:0/725684802 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fea3c06cd30 0x7fea3c06f1f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fea54101490 tx=0x7fea4800b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:01.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.130+0000 7fea3bfff700 1 -- 172.21.15.121:0/725684802 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fea44057850 con 0x7fea54068670 2024-04-05T21:32:01.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.350+0000 7fea59262700 1 -- 172.21.15.121:0/725684802 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fea540611d0 con 0x7fea3c06cd30 2024-04-05T21:32:01.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.352+0000 7fea3bfff700 1 -- 172.21.15.121:0/725684802 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fea540611d0 con 0x7fea3c06cd30 2024-04-05T21:32:01.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.354+0000 7fea59262700 1 -- 172.21.15.121:0/725684802 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fea3c06cd30 msgr2=0x7fea3c06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:01.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.354+0000 7fea59262700 1 --2- 172.21.15.121:0/725684802 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fea3c06cd30 0x7fea3c06f1f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fea54101490 tx=0x7fea4800b410 comp rx=0 tx=0).stop 2024-04-05T21:32:01.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.354+0000 7fea59262700 1 -- 172.21.15.121:0/725684802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea54068670 msgr2=0x7fea540ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:01.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.354+0000 7fea59262700 1 --2- 172.21.15.121:0/725684802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea54068670 0x7fea540ffca0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea440092e0 tx=0x7fea4400bd70 comp rx=0 tx=0).stop 2024-04-05T21:32:01.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.354+0000 7fea59262700 1 -- 172.21.15.121:0/725684802 shutdown_connections 2024-04-05T21:32:01.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.354+0000 7fea59262700 1 --2- 172.21.15.121:0/725684802 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fea3c06cd30 0x7fea3c06f1f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.354+0000 7fea59262700 1 --2- 172.21.15.121:0/725684802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea54068670 0x7fea540ffca0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.354+0000 7fea59262700 1 --2- 172.21.15.121:0/725684802 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fea54069020 0x7fea541001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.354+0000 7fea59262700 1 -- 172.21.15.121:0/725684802 >> 172.21.15.121:0/725684802 conn(0x7fea540755a0 msgr2=0x7fea540fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:01.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.354+0000 7fea59262700 1 -- 172.21.15.121:0/725684802 shutdown_connections 2024-04-05T21:32:01.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.354+0000 7fea59262700 1 -- 172.21.15.121:0/725684802 wait complete. 2024-04-05T21:32:01.365 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:32:01.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.462+0000 7f347f466700 1 -- 172.21.15.121:0/3338819685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3478069020 msgr2=0x7f3478105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:01.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.462+0000 7f347f466700 1 --2- 172.21.15.121:0/3338819685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3478069020 0x7f3478105790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3468009910 tx=0x7f3468009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:01.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.462+0000 7f347f466700 1 -- 172.21.15.121:0/3338819685 shutdown_connections 2024-04-05T21:32:01.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.462+0000 7f347f466700 1 --2- 172.21.15.121:0/3338819685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3478069020 0x7f3478105790 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.462+0000 7f347f466700 1 --2- 172.21.15.121:0/3338819685 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3478068670 0x7f3478068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.462+0000 7f347f466700 1 -- 172.21.15.121:0/3338819685 >> 172.21.15.121:0/3338819685 conn(0x7f34780755a0 msgr2=0x7f34780759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:01.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.462+0000 7f347f466700 1 -- 172.21.15.121:0/3338819685 shutdown_connections 2024-04-05T21:32:01.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.462+0000 7f347f466700 1 -- 172.21.15.121:0/3338819685 wait complete. 2024-04-05T21:32:01.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.463+0000 7f347f466700 1 Processor -- start 2024-04-05T21:32:01.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.463+0000 7f347f466700 1 -- start start 2024-04-05T21:32:01.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.463+0000 7f347f466700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3478068670 0x7f3478196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:01.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.463+0000 7f347f466700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3478069020 0x7f3478196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.463+0000 7f347f466700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3478197260 con 0x7f3478069020 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.463+0000 7f347f466700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347819aff0 con 0x7f3478068670 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.463+0000 7f347ca01700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3478069020 0x7f3478196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.463+0000 7f347ca01700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3478069020 0x7f3478196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54026/0 (socket says 172.21.15.121:54026) 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.463+0000 7f347ca01700 1 -- 172.21.15.121:0/1601720427 learned_addr learned my addr 172.21.15.121:0/1601720427 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.464+0000 7f347d202700 1 --2- 172.21.15.121:0/1601720427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3478068670 0x7f3478196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.464+0000 7f347ca01700 1 -- 172.21.15.121:0/1601720427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3478068670 msgr2=0x7f3478196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.464+0000 7f347ca01700 1 --2- 172.21.15.121:0/1601720427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3478068670 0x7f3478196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.464+0000 7f347ca01700 1 -- 172.21.15.121:0/1601720427 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34680095f0 con 0x7f3478069020 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.464+0000 7f347d202700 1 --2- 172.21.15.121:0/1601720427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3478068670 0x7f3478196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.464+0000 7f347ca01700 1 --2- 172.21.15.121:0/1601720427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3478069020 0x7f3478196b80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f34680092e0 tx=0x7f346800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.464+0000 7f346e7fc700 1 -- 172.21.15.121:0/1601720427 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f346801d070 con 0x7f3478069020 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.464+0000 7f346e7fc700 1 -- 172.21.15.121:0/1601720427 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3468005940 con 0x7f3478069020 2024-04-05T21:32:01.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.464+0000 7f346e7fc700 1 -- 172.21.15.121:0/1601720427 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f346800fe50 con 0x7f3478069020 2024-04-05T21:32:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.464+0000 7f347f466700 1 -- 172.21.15.121:0/1601720427 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347819b270 con 0x7f3478069020 2024-04-05T21:32:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.464+0000 7f347f466700 1 -- 172.21.15.121:0/1601720427 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347819b760 con 0x7f3478069020 2024-04-05T21:32:01.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.465+0000 7f347f466700 1 -- 172.21.15.121:0/1601720427 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34781090a0 con 0x7f3478069020 2024-04-05T21:32:01.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.466+0000 7f346e7fc700 1 -- 172.21.15.121:0/1601720427 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3468003a60 con 0x7f3478069020 2024-04-05T21:32:01.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.466+0000 7f346e7fc700 1 --2- 172.21.15.121:0/1601720427 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f346406cce0 0x7f346406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:01.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.466+0000 7f346e7fc700 1 -- 172.21.15.121:0/1601720427 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f346808d710 con 0x7f3478069020 2024-04-05T21:32:01.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.466+0000 7f347d202700 1 --2- 172.21.15.121:0/1601720427 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f346406cce0 0x7f346406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:01.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.467+0000 7f347d202700 1 --2- 172.21.15.121:0/1601720427 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f346406cce0 0x7f346406f1a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3474005fd0 tx=0x7f3474005ee0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:01.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.471+0000 7f346e7fc700 1 -- 172.21.15.121:0/1601720427 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3468057830 con 0x7f3478069020 2024-04-05T21:32:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:01 smithi121 ceph-mon[145859]: pgmap v144: 65 pgs: 12 stale+active+clean, 53 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T21:32:01.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.691+0000 7f347f466700 1 -- 172.21.15.121:0/1601720427 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f347802cd60 con 0x7f346406cce0 2024-04-05T21:32:01.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.693+0000 7f346e7fc700 1 -- 172.21.15.121:0/1601720427 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f347802cd60 con 0x7f346406cce0 2024-04-05T21:32:01.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.695+0000 7f347f466700 1 -- 172.21.15.121:0/1601720427 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f346406cce0 msgr2=0x7f346406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:01.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.695+0000 7f347f466700 1 --2- 172.21.15.121:0/1601720427 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f346406cce0 0x7f346406f1a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3474005fd0 tx=0x7f3474005ee0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.695+0000 7f347f466700 1 -- 172.21.15.121:0/1601720427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3478069020 msgr2=0x7f3478196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:01.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.695+0000 7f347f466700 1 --2- 172.21.15.121:0/1601720427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3478069020 0x7f3478196b80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f34680092e0 tx=0x7f346800be20 comp rx=0 tx=0).stop 2024-04-05T21:32:01.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.695+0000 7f347f466700 1 -- 172.21.15.121:0/1601720427 shutdown_connections 2024-04-05T21:32:01.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.695+0000 7f347f466700 1 --2- 172.21.15.121:0/1601720427 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f346406cce0 0x7f346406f1a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.695+0000 7f347f466700 1 --2- 172.21.15.121:0/1601720427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3478068670 0x7f3478196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.695+0000 7f347f466700 1 --2- 172.21.15.121:0/1601720427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3478069020 0x7f3478196b80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.695+0000 7f347f466700 1 -- 172.21.15.121:0/1601720427 >> 172.21.15.121:0/1601720427 conn(0x7f34780755a0 msgr2=0x7f3478105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:01.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.695+0000 7f347f466700 1 -- 172.21.15.121:0/1601720427 shutdown_connections 2024-04-05T21:32:01.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.695+0000 7f347f466700 1 -- 172.21.15.121:0/1601720427 wait complete. 2024-04-05T21:32:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:01 smithi188 ceph-mon[107793]: pgmap v144: 65 pgs: 12 stale+active+clean, 53 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T21:32:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.798+0000 7f7b4e40a700 1 -- 172.21.15.121:0/2053773993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b48101d00 msgr2=0x7f7b481020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.798+0000 7f7b4e40a700 1 --2- 172.21.15.121:0/2053773993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b48101d00 0x7f7b481020e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009910 tx=0x7f7b38009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.798+0000 7f7b4e40a700 1 -- 172.21.15.121:0/2053773993 shutdown_connections 2024-04-05T21:32:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.798+0000 7f7b4e40a700 1 --2- 172.21.15.121:0/2053773993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b48102620 0x7f7b4810ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.798+0000 7f7b4e40a700 1 --2- 172.21.15.121:0/2053773993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b48101d00 0x7f7b481020e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.798+0000 7f7b4e40a700 1 -- 172.21.15.121:0/2053773993 >> 172.21.15.121:0/2053773993 conn(0x7f7b480fb570 msgr2=0x7f7b480fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:01.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.798+0000 7f7b4e40a700 1 -- 172.21.15.121:0/2053773993 shutdown_connections 2024-04-05T21:32:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.798+0000 7f7b4e40a700 1 -- 172.21.15.121:0/2053773993 wait complete. 2024-04-05T21:32:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.799+0000 7f7b4e40a700 1 Processor -- start 2024-04-05T21:32:01.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.799+0000 7f7b4e40a700 1 -- start start 2024-04-05T21:32:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.799+0000 7f7b4e40a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b48101d00 0x7f7b480ff960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.799+0000 7f7b4e40a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b48102620 0x7f7b480ffea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:01.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.799+0000 7f7b4e40a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b481003e0 con 0x7f7b48102620 2024-04-05T21:32:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.799+0000 7f7b4e40a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b48100520 con 0x7f7b48101d00 2024-04-05T21:32:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.800+0000 7f7b477fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b48102620 0x7f7b480ffea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.800+0000 7f7b477fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b48102620 0x7f7b480ffea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54036/0 (socket says 172.21.15.121:54036) 2024-04-05T21:32:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.800+0000 7f7b477fe700 1 -- 172.21.15.121:0/3562954892 learned_addr learned my addr 172.21.15.121:0/3562954892 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.800+0000 7f7b47fff700 1 --2- 172.21.15.121:0/3562954892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b48101d00 0x7f7b480ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.800+0000 7f7b477fe700 1 -- 172.21.15.121:0/3562954892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b48101d00 msgr2=0x7f7b480ff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.800+0000 7f7b477fe700 1 --2- 172.21.15.121:0/3562954892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b48101d00 0x7f7b480ff960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.800+0000 7f7b477fe700 1 -- 172.21.15.121:0/3562954892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b380095f0 con 0x7f7b48102620 2024-04-05T21:32:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.800+0000 7f7b477fe700 1 --2- 172.21.15.121:0/3562954892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b48102620 0x7f7b480ffea0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00c8f0 tx=0x7f7b3c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.800+0000 7f7b457fa700 1 -- 172.21.15.121:0/3562954892 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b3c004a00 con 0x7f7b48102620 2024-04-05T21:32:01.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.800+0000 7f7b4e40a700 1 -- 172.21.15.121:0/3562954892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b481a2870 con 0x7f7b48102620 2024-04-05T21:32:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.800+0000 7f7b457fa700 1 -- 172.21.15.121:0/3562954892 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b3c005020 con 0x7f7b48102620 2024-04-05T21:32:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.800+0000 7f7b457fa700 1 -- 172.21.15.121:0/3562954892 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b3c00ebe0 con 0x7f7b48102620 2024-04-05T21:32:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.800+0000 7f7b4e40a700 1 -- 172.21.15.121:0/3562954892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b481a2c80 con 0x7f7b48102620 2024-04-05T21:32:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.801+0000 7f7b4e40a700 1 -- 172.21.15.121:0/3562954892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b48108210 con 0x7f7b48102620 2024-04-05T21:32:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.802+0000 7f7b457fa700 1 -- 172.21.15.121:0/3562954892 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7b3c00ed60 con 0x7f7b48102620 2024-04-05T21:32:01.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.802+0000 7f7b457fa700 1 --2- 172.21.15.121:0/3562954892 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7b3006cce0 0x7f7b3006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.802+0000 7f7b457fa700 1 -- 172.21.15.121:0/3562954892 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f7b3c08c020 con 0x7f7b48102620 2024-04-05T21:32:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.802+0000 7f7b47fff700 1 --2- 172.21.15.121:0/3562954892 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7b3006cce0 0x7f7b3006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:01.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.802+0000 7f7b47fff700 1 --2- 172.21.15.121:0/3562954892 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7b3006cce0 0x7f7b3006f1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7b38006010 tx=0x7f7b3801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:01.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:01.806+0000 7f7b457fa700 1 -- 172.21.15.121:0/3562954892 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7b3c056140 con 0x7f7b48102620 2024-04-05T21:32:02.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.016+0000 7f7b4e40a700 1 -- 172.21.15.121:0/3562954892 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b4802cda0 con 0x7f7b3006cce0 2024-04-05T21:32:02.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.026+0000 7f7b457fa700 1 -- 172.21.15.121:0/3562954892 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7b4802cda0 con 0x7f7b3006cce0 2024-04-05T21:32:02.027 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:32:02.027 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (2m) 28s ago 14m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:32:02.027 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (14m) 28s ago 14m 8182k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:32:02.027 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (13m) 52s ago 13m 8044k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:32:02.027 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (62s) 28s ago 14m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:32:02.027 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (56s) 52s ago 13m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:32:02.027 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (2m) 28s ago 14m 80.4M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:32:02.027 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (8m) 28s ago 8m 64.2M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:32:02.027 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (8m) 28s ago 8m 133M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:32:02.027 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (8m) 52s ago 8m 110M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (8m) 52s ago 8m 62.2M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (4m) 28s ago 15m 504M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (4m) 52s ago 13m 450M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (95s) 28s ago 16m 42.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (77s) 52s ago 13m 27.9M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (3m) 28s ago 14m 18.0M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (3m) 52s ago 13m 16.4M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (34s) 28s ago 12m 13.7M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (11m) 28s ago 11m 242M 1227M 18.2.1 5be31c24972a 17a73cce549c 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (11m) 28s ago 11m 182M 1227M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (10m) 52s ago 10m 288M 2251M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (10m) 52s ago 10m 246M 2251M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (9m) 52s ago 9m 224M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:32:02.028 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (3m) 28s ago 13m 62.8M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:32:02.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.028+0000 7f7b4e40a700 1 -- 172.21.15.121:0/3562954892 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7b3006cce0 msgr2=0x7f7b3006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:02.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.028+0000 7f7b4e40a700 1 --2- 172.21.15.121:0/3562954892 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7b3006cce0 0x7f7b3006f1a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7b38006010 tx=0x7f7b3801a040 comp rx=0 tx=0).stop 2024-04-05T21:32:02.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.028+0000 7f7b4e40a700 1 -- 172.21.15.121:0/3562954892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b48102620 msgr2=0x7f7b480ffea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:02.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.028+0000 7f7b4e40a700 1 --2- 172.21.15.121:0/3562954892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b48102620 0x7f7b480ffea0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00c8f0 tx=0x7f7b3c00cc00 comp rx=0 tx=0).stop 2024-04-05T21:32:02.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.028+0000 7f7b4e40a700 1 -- 172.21.15.121:0/3562954892 shutdown_connections 2024-04-05T21:32:02.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.028+0000 7f7b4e40a700 1 --2- 172.21.15.121:0/3562954892 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7b3006cce0 0x7f7b3006f1a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.028+0000 7f7b4e40a700 1 --2- 172.21.15.121:0/3562954892 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7b48101d00 0x7f7b480ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.028+0000 7f7b4e40a700 1 --2- 172.21.15.121:0/3562954892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b48102620 0x7f7b480ffea0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.028+0000 7f7b4e40a700 1 -- 172.21.15.121:0/3562954892 >> 172.21.15.121:0/3562954892 conn(0x7f7b480fb570 msgr2=0x7f7b48105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:02.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.028+0000 7f7b4e40a700 1 -- 172.21.15.121:0/3562954892 shutdown_connections 2024-04-05T21:32:02.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.028+0000 7f7b4e40a700 1 -- 172.21.15.121:0/3562954892 wait complete. 2024-04-05T21:32:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.132+0000 7f70d1ff2700 1 -- 172.21.15.121:0/673695904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70cc0ff400 msgr2=0x7f70cc0ff7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.132+0000 7f70d1ff2700 1 --2- 172.21.15.121:0/673695904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70cc0ff400 0x7f70cc0ff7e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009910 tx=0x7f70b4009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.132+0000 7f70d1ff2700 1 -- 172.21.15.121:0/673695904 shutdown_connections 2024-04-05T21:32:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.132+0000 7f70d1ff2700 1 --2- 172.21.15.121:0/673695904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc0ffdb0 0x7f70cc103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.132+0000 7f70d1ff2700 1 --2- 172.21.15.121:0/673695904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70cc0ff400 0x7f70cc0ff7e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.132+0000 7f70d1ff2700 1 -- 172.21.15.121:0/673695904 >> 172.21.15.121:0/673695904 conn(0x7f70cc075e30 msgr2=0x7f70cc0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.132+0000 7f70d1ff2700 1 -- 172.21.15.121:0/673695904 shutdown_connections 2024-04-05T21:32:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.132+0000 7f70d1ff2700 1 -- 172.21.15.121:0/673695904 wait complete. 2024-04-05T21:32:02.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.133+0000 7f70d1ff2700 1 Processor -- start 2024-04-05T21:32:02.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.133+0000 7f70d1ff2700 1 -- start start 2024-04-05T21:32:02.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.133+0000 7f70d1ff2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70cc0ff400 0x7f70cc19a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:02.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.133+0000 7f70d1ff2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc0ffdb0 0x7f70cc19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:02.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.133+0000 7f70d1ff2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc195b60 con 0x7f70cc0ff400 2024-04-05T21:32:02.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.133+0000 7f70d1ff2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc195ca0 con 0x7f70cc0ffdb0 2024-04-05T21:32:02.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.133+0000 7f70cb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70cc0ff400 0x7f70cc19a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.133+0000 7f70cb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70cc0ff400 0x7f70cc19a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54056/0 (socket says 172.21.15.121:54056) 2024-04-05T21:32:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.133+0000 7f70cb7fe700 1 -- 172.21.15.121:0/2796671374 learned_addr learned my addr 172.21.15.121:0/2796671374 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.133+0000 7f70caffd700 1 --2- 172.21.15.121:0/2796671374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc0ffdb0 0x7f70cc19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.133+0000 7f70cb7fe700 1 -- 172.21.15.121:0/2796671374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc0ffdb0 msgr2=0x7f70cc19a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.133+0000 7f70cb7fe700 1 --2- 172.21.15.121:0/2796671374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc0ffdb0 0x7f70cc19a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.134+0000 7f70cb7fe700 1 -- 172.21.15.121:0/2796671374 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b40095f0 con 0x7f70cc0ff400 2024-04-05T21:32:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.134+0000 7f70cb7fe700 1 --2- 172.21.15.121:0/2796671374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70cc0ff400 0x7f70cc19a2c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f70b4006010 tx=0x7f70b400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.134+0000 7f70c8ff9700 1 -- 172.21.15.121:0/2796671374 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70b401f070 con 0x7f70cc0ff400 2024-04-05T21:32:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.134+0000 7f70d1ff2700 1 -- 172.21.15.121:0/2796671374 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70cc195f20 con 0x7f70cc0ff400 2024-04-05T21:32:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.134+0000 7f70c8ff9700 1 -- 172.21.15.121:0/2796671374 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70b4019610 con 0x7f70cc0ff400 2024-04-05T21:32:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.134+0000 7f70c8ff9700 1 -- 172.21.15.121:0/2796671374 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70b400f670 con 0x7f70cc0ff400 2024-04-05T21:32:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.134+0000 7f70d1ff2700 1 -- 172.21.15.121:0/2796671374 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70cc196410 con 0x7f70cc0ff400 2024-04-05T21:32:02.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.135+0000 7f70d1ff2700 1 -- 172.21.15.121:0/2796671374 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70cc04ea90 con 0x7f70cc0ff400 2024-04-05T21:32:02.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.136+0000 7f70c8ff9700 1 -- 172.21.15.121:0/2796671374 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f70b4019c10 con 0x7f70cc0ff400 2024-04-05T21:32:02.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.136+0000 7f70c8ff9700 1 --2- 172.21.15.121:0/2796671374 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70b806cce0 0x7f70b806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:02.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.136+0000 7f70c8ff9700 1 -- 172.21.15.121:0/2796671374 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f70b408fcc0 con 0x7f70cc0ff400 2024-04-05T21:32:02.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.136+0000 7f70caffd700 1 --2- 172.21.15.121:0/2796671374 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70b806cce0 0x7f70b806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:02.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.136+0000 7f70caffd700 1 --2- 172.21.15.121:0/2796671374 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70b806cce0 0x7f70b806f1a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f70bc005fd0 tx=0x7f70bc00a560 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:02.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.140+0000 7f70c8ff9700 1 -- 172.21.15.121:0/2796671374 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f70b405da10 con 0x7f70cc0ff400 2024-04-05T21:32:02.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.416+0000 7f70d1ff2700 1 -- 172.21.15.121:0/2796671374 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f70cc0681d0 con 0x7f70cc0ff400 2024-04-05T21:32:02.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.416+0000 7f70c8ff9700 1 -- 172.21.15.121:0/2796671374 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+799 (secure 0 0 0) 0x7f70b405d5a0 con 0x7f70cc0ff400 2024-04-05T21:32:02.417 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:32:02.418 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:32:02.418 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:32:02.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:32:02.418 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:32:02.418 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:32:02.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:32:02.418 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:32:02.418 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4, 2024-04-05T21:32:02.418 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 1 2024-04-05T21:32:02.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:32:02.418 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:32:02.418 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:32:02.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:32:02.419 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:32:02.419 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 8, 2024-04-05T21:32:02.419 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 5 2024-04-05T21:32:02.419 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:32:02.419 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:32:02.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.418+0000 7f70d1ff2700 1 -- 172.21.15.121:0/2796671374 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70b806cce0 msgr2=0x7f70b806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:02.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.418+0000 7f70d1ff2700 1 --2- 172.21.15.121:0/2796671374 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70b806cce0 0x7f70b806f1a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f70bc005fd0 tx=0x7f70bc00a560 comp rx=0 tx=0).stop 2024-04-05T21:32:02.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.418+0000 7f70d1ff2700 1 -- 172.21.15.121:0/2796671374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70cc0ff400 msgr2=0x7f70cc19a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:02.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.418+0000 7f70d1ff2700 1 --2- 172.21.15.121:0/2796671374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70cc0ff400 0x7f70cc19a2c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f70b4006010 tx=0x7f70b400be20 comp rx=0 tx=0).stop 2024-04-05T21:32:02.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.418+0000 7f70d1ff2700 1 -- 172.21.15.121:0/2796671374 shutdown_connections 2024-04-05T21:32:02.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.418+0000 7f70d1ff2700 1 --2- 172.21.15.121:0/2796671374 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70b806cce0 0x7f70b806f1a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.418+0000 7f70d1ff2700 1 --2- 172.21.15.121:0/2796671374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70cc0ff400 0x7f70cc19a2c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.418+0000 7f70d1ff2700 1 --2- 172.21.15.121:0/2796671374 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70cc0ffdb0 0x7f70cc19a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.418+0000 7f70d1ff2700 1 -- 172.21.15.121:0/2796671374 >> 172.21.15.121:0/2796671374 conn(0x7f70cc075e30 msgr2=0x7f70cc103170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:02.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.419+0000 7f70d1ff2700 1 -- 172.21.15.121:0/2796671374 shutdown_connections 2024-04-05T21:32:02.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.419+0000 7f70d1ff2700 1 -- 172.21.15.121:0/2796671374 wait complete. 2024-04-05T21:32:02.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.523+0000 7f6007674700 1 -- 172.21.15.121:0/2885790049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60001052c0 msgr2=0x7f60001056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:02.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.523+0000 7f6007674700 1 --2- 172.21.15.121:0/2885790049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60001052c0 0x7f60001056a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009910 tx=0x7f5ffc009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:02.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.523+0000 7f6007674700 1 -- 172.21.15.121:0/2885790049 shutdown_connections 2024-04-05T21:32:02.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.523+0000 7f6007674700 1 --2- 172.21.15.121:0/2885790049 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60000ff2b0 0x7f60000ff710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.523+0000 7f6007674700 1 --2- 172.21.15.121:0/2885790049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60001052c0 0x7f60001056a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.523+0000 7f6007674700 1 -- 172.21.15.121:0/2885790049 >> 172.21.15.121:0/2885790049 conn(0x7f6000074e00 msgr2=0x7f6000075210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:02.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.523+0000 7f6007674700 1 -- 172.21.15.121:0/2885790049 shutdown_connections 2024-04-05T21:32:02.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.523+0000 7f6007674700 1 -- 172.21.15.121:0/2885790049 wait complete. 2024-04-05T21:32:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.524+0000 7f6007674700 1 Processor -- start 2024-04-05T21:32:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.524+0000 7f6007674700 1 -- start start 2024-04-05T21:32:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.524+0000 7f6007674700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60000ff2b0 0x7f6000196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.524+0000 7f6007674700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60001052c0 0x7f6000196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.524+0000 7f6007674700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6000197290 con 0x7f60000ff2b0 2024-04-05T21:32:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.524+0000 7f6007674700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600019b020 con 0x7f60001052c0 2024-04-05T21:32:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.524+0000 7f6005410700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60000ff2b0 0x7f6000196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.524+0000 7f6005410700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60000ff2b0 0x7f6000196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54084/0 (socket says 172.21.15.121:54084) 2024-04-05T21:32:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.524+0000 7f6005410700 1 -- 172.21.15.121:0/4210474612 learned_addr learned my addr 172.21.15.121:0/4210474612 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.524+0000 7f6004c0f700 1 --2- 172.21.15.121:0/4210474612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60001052c0 0x7f6000196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.525+0000 7f6005410700 1 -- 172.21.15.121:0/4210474612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60001052c0 msgr2=0x7f6000196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.525+0000 7f6005410700 1 --2- 172.21.15.121:0/4210474612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60001052c0 0x7f6000196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.525+0000 7f6005410700 1 -- 172.21.15.121:0/4210474612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc0095f0 con 0x7f60000ff2b0 2024-04-05T21:32:02.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.525+0000 7f6005410700 1 --2- 172.21.15.121:0/4210474612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60000ff2b0 0x7f6000196670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc000c00 tx=0x7f5ffc00bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:02.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.525+0000 7f5ff67fc700 1 -- 172.21.15.121:0/4210474612 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ffc01d070 con 0x7f60000ff2b0 2024-04-05T21:32:02.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.525+0000 7f6007674700 1 -- 172.21.15.121:0/4210474612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600019b2a0 con 0x7f60000ff2b0 2024-04-05T21:32:02.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.525+0000 7f6007674700 1 -- 172.21.15.121:0/4210474612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600019b730 con 0x7f60000ff2b0 2024-04-05T21:32:02.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.525+0000 7f5ff67fc700 1 -- 172.21.15.121:0/4210474612 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ffc005dc0 con 0x7f60000ff2b0 2024-04-05T21:32:02.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.525+0000 7f5ff67fc700 1 -- 172.21.15.121:0/4210474612 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ffc003940 con 0x7f60000ff2b0 2024-04-05T21:32:02.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.526+0000 7f6007674700 1 -- 172.21.15.121:0/4210474612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe4005320 con 0x7f60000ff2b0 2024-04-05T21:32:02.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.527+0000 7f5ff67fc700 1 -- 172.21.15.121:0/4210474612 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5ffc0048e0 con 0x7f60000ff2b0 2024-04-05T21:32:02.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.527+0000 7f5ff67fc700 1 --2- 172.21.15.121:0/4210474612 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fec06cce0 0x7f5fec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:02.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.527+0000 7f5ff67fc700 1 -- 172.21.15.121:0/4210474612 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f5ffc08d980 con 0x7f60000ff2b0 2024-04-05T21:32:02.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.527+0000 7f6004c0f700 1 --2- 172.21.15.121:0/4210474612 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fec06cce0 0x7f5fec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:02.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.528+0000 7f6004c0f700 1 --2- 172.21.15.121:0/4210474612 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fec06cce0 0x7f5fec06f1a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6000197c90 tx=0x7f5ff0010040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:02.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.532+0000 7f5ff67fc700 1 -- 172.21.15.121:0/4210474612 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5ffc057aa0 con 0x7f60000ff2b0 2024-04-05T21:32:02.673 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:02 smithi121 ceph-mon[145859]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:02.673 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:02 smithi121 ceph-mon[145859]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:02.673 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:02 smithi121 systemd[1]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.1.service: Succeeded. 2024-04-05T21:32:02.673 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:02 smithi121 systemd[1]: Stopped Ceph osd.1 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:32:02.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:02 smithi188 ceph-mon[107793]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:02.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:02 smithi188 ceph-mon[107793]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:02.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.809+0000 7f6007674700 1 -- 172.21.15.121:0/4210474612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5fe4006200 con 0x7f60000ff2b0 2024-04-05T21:32:02.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.810+0000 7f5ff67fc700 1 -- 172.21.15.121:0/4210474612 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f5ffc026770 con 0x7f60000ff2b0 2024-04-05T21:32:02.812 INFO:teuthology.orchestra.run.smithi121.stdout:e21 2024-04-05T21:32:02.812 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:32:02.812 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:32:02.812 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:32:02.812 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:32:02.813 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:32:02.813 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:32:02.813 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 21 2024-04-05T21:32:02.813 INFO:teuthology.orchestra.run.smithi121.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T21:32:02.813 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:32:02.813 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:30:52.246236+0000 2024-04-05T21:32:02.813 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:32:02.813 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:32:02.813 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:32:02.814 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:32:02.814 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:32:02.814 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:32:02.814 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:32:02.814 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:32:02.814 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 46 2024-04-05T21:32:02.814 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:32:02.814 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:32:02.814 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:32:02.814 INFO:teuthology.orchestra.run.smithi121.stdout:up {0=14476,1=24255} 2024-04-05T21:32:02.814 INFO:teuthology.orchestra.run.smithi121.stdout:failed 2024-04-05T21:32:02.815 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:32:02.815 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:32:02.815 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:32:02.815 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:32:02.815 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:32:02.815 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:32:02.815 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:32:02.815 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:32:02.815 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{0:14476} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:32:02.815 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{0:14488} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/4279084130,v1:172.21.15.121:6827/4279084130] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:32:02.815 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{1:24255} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:32:02.815 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{1:24265} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:32:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:32:02.816 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:32:02.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.812+0000 7f6007674700 1 -- 172.21.15.121:0/4210474612 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fec06cce0 msgr2=0x7f5fec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:02.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.812+0000 7f6007674700 1 --2- 172.21.15.121:0/4210474612 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fec06cce0 0x7f5fec06f1a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6000197c90 tx=0x7f5ff0010040 comp rx=0 tx=0).stop 2024-04-05T21:32:02.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.812+0000 7f6007674700 1 -- 172.21.15.121:0/4210474612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60000ff2b0 msgr2=0x7f6000196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:02.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.812+0000 7f6007674700 1 --2- 172.21.15.121:0/4210474612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60000ff2b0 0x7f6000196670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc000c00 tx=0x7f5ffc00bde0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.812+0000 7f6007674700 1 -- 172.21.15.121:0/4210474612 shutdown_connections 2024-04-05T21:32:02.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.812+0000 7f6007674700 1 --2- 172.21.15.121:0/4210474612 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fec06cce0 0x7f5fec06f1a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.812+0000 7f6007674700 1 --2- 172.21.15.121:0/4210474612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60000ff2b0 0x7f6000196670 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.812+0000 7f6007674700 1 --2- 172.21.15.121:0/4210474612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60001052c0 0x7f6000196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.812+0000 7f6007674700 1 -- 172.21.15.121:0/4210474612 >> 172.21.15.121:0/4210474612 conn(0x7f6000074e00 msgr2=0x7f60001037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:02.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.813+0000 7f6007674700 1 -- 172.21.15.121:0/4210474612 shutdown_connections 2024-04-05T21:32:02.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.813+0000 7f6007674700 1 -- 172.21.15.121:0/4210474612 wait complete. 2024-04-05T21:32:02.818 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 21 2024-04-05T21:32:02.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.918+0000 7fdc3de6b700 1 -- 172.21.15.121:0/28260163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc38100de0 msgr2=0x7fdc381011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:02.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.918+0000 7fdc3de6b700 1 --2- 172.21.15.121:0/28260163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc38100de0 0x7fdc381011c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009910 tx=0x7fdc2c009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:02.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.918+0000 7fdc3de6b700 1 -- 172.21.15.121:0/28260163 shutdown_connections 2024-04-05T21:32:02.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.918+0000 7fdc3de6b700 1 --2- 172.21.15.121:0/28260163 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc38101790 0x7fdc381057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.918+0000 7fdc3de6b700 1 --2- 172.21.15.121:0/28260163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc38100de0 0x7fdc381011c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.918+0000 7fdc3de6b700 1 -- 172.21.15.121:0/28260163 >> 172.21.15.121:0/28260163 conn(0x7fdc380fc670 msgr2=0x7fdc380fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.918+0000 7fdc3de6b700 1 -- 172.21.15.121:0/28260163 shutdown_connections 2024-04-05T21:32:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.918+0000 7fdc3de6b700 1 -- 172.21.15.121:0/28260163 wait complete. 2024-04-05T21:32:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.918+0000 7fdc3de6b700 1 Processor -- start 2024-04-05T21:32:02.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.919+0000 7fdc3de6b700 1 -- start start 2024-04-05T21:32:02.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.919+0000 7fdc3de6b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc38100de0 0x7fdc38073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:02.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.919+0000 7fdc3de6b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc38101790 0x7fdc380735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:02.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.919+0000 7fdc3de6b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38073b10 con 0x7fdc38101790 2024-04-05T21:32:02.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.919+0000 7fdc3de6b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38073c50 con 0x7fdc38100de0 2024-04-05T21:32:02.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.919+0000 7fdc36ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc38101790 0x7fdc380735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.919+0000 7fdc36ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc38101790 0x7fdc380735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54100/0 (socket says 172.21.15.121:54100) 2024-04-05T21:32:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.919+0000 7fdc36ffd700 1 -- 172.21.15.121:0/1943028455 learned_addr learned my addr 172.21.15.121:0/1943028455 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.919+0000 7fdc377fe700 1 --2- 172.21.15.121:0/1943028455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc38100de0 0x7fdc38073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.919+0000 7fdc36ffd700 1 -- 172.21.15.121:0/1943028455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc38100de0 msgr2=0x7fdc38073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.919+0000 7fdc36ffd700 1 --2- 172.21.15.121:0/1943028455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc38100de0 0x7fdc38073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.919+0000 7fdc36ffd700 1 -- 172.21.15.121:0/1943028455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c0095f0 con 0x7fdc38101790 2024-04-05T21:32:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.919+0000 7fdc36ffd700 1 --2- 172.21.15.121:0/1943028455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc38101790 0x7fdc380735d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800d900 tx=0x7fdc2800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.920+0000 7fdc34ff9700 1 -- 172.21.15.121:0/1943028455 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc28009900 con 0x7fdc38101790 2024-04-05T21:32:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.920+0000 7fdc3de6b700 1 -- 172.21.15.121:0/1943028455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc381a2840 con 0x7fdc38101790 2024-04-05T21:32:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.920+0000 7fdc34ff9700 1 -- 172.21.15.121:0/1943028455 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc28010460 con 0x7fdc38101790 2024-04-05T21:32:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.920+0000 7fdc34ff9700 1 -- 172.21.15.121:0/1943028455 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc2800f5f0 con 0x7fdc38101790 2024-04-05T21:32:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.920+0000 7fdc3de6b700 1 -- 172.21.15.121:0/1943028455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc381a2c50 con 0x7fdc38101790 2024-04-05T21:32:02.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.920+0000 7fdc3de6b700 1 -- 172.21.15.121:0/1943028455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc38109180 con 0x7fdc38101790 2024-04-05T21:32:02.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.921+0000 7fdc34ff9700 1 -- 172.21.15.121:0/1943028455 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdc280105d0 con 0x7fdc38101790 2024-04-05T21:32:02.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.921+0000 7fdc34ff9700 1 --2- 172.21.15.121:0/1943028455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdc2006cce0 0x7fdc2006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:02.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.921+0000 7fdc34ff9700 1 -- 172.21.15.121:0/1943028455 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fdc2808c030 con 0x7fdc38101790 2024-04-05T21:32:02.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.922+0000 7fdc377fe700 1 --2- 172.21.15.121:0/1943028455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdc2006cce0 0x7fdc2006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:02.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.922+0000 7fdc377fe700 1 --2- 172.21.15.121:0/1943028455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdc2006cce0 0x7fdc2006f1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c006010 tx=0x7fdc2c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:02.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:02.926+0000 7fdc34ff9700 1 -- 172.21.15.121:0/1943028455 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdc28056150 con 0x7fdc38101790 2024-04-05T21:32:02.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:02 smithi121 systemd[1]: Starting Ceph osd.1 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:32:03.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.144+0000 7fdc3de6b700 1 -- 172.21.15.121:0/1943028455 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc3802cda0 con 0x7fdc2006cce0 2024-04-05T21:32:03.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.145+0000 7fdc34ff9700 1 -- 172.21.15.121:0/1943028455 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fdc3802cda0 con 0x7fdc2006cce0 2024-04-05T21:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:32:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:32:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:32:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:32:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:32:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "7/23 daemons upgraded", 2024-04-05T21:32:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading osd daemons", 2024-04-05T21:32:03.147 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:32:03.147 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:32:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.147+0000 7fdc3de6b700 1 -- 172.21.15.121:0/1943028455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdc2006cce0 msgr2=0x7fdc2006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.147+0000 7fdc3de6b700 1 --2- 172.21.15.121:0/1943028455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdc2006cce0 0x7fdc2006f1a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c006010 tx=0x7fdc2c01a040 comp rx=0 tx=0).stop 2024-04-05T21:32:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.147+0000 7fdc3de6b700 1 -- 172.21.15.121:0/1943028455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc38101790 msgr2=0x7fdc380735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.147+0000 7fdc3de6b700 1 --2- 172.21.15.121:0/1943028455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc38101790 0x7fdc380735d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdc2800d900 tx=0x7fdc2800dc10 comp rx=0 tx=0).stop 2024-04-05T21:32:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.147+0000 7fdc3de6b700 1 -- 172.21.15.121:0/1943028455 shutdown_connections 2024-04-05T21:32:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.147+0000 7fdc3de6b700 1 --2- 172.21.15.121:0/1943028455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdc2006cce0 0x7fdc2006f1a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.147+0000 7fdc3de6b700 1 --2- 172.21.15.121:0/1943028455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc38100de0 0x7fdc38073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.147+0000 7fdc3de6b700 1 --2- 172.21.15.121:0/1943028455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc38101790 0x7fdc380735d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.147+0000 7fdc3de6b700 1 -- 172.21.15.121:0/1943028455 >> 172.21.15.121:0/1943028455 conn(0x7fdc380fc670 msgr2=0x7fdc380fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.148+0000 7fdc3de6b700 1 -- 172.21.15.121:0/1943028455 shutdown_connections 2024-04-05T21:32:03.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.148+0000 7fdc3de6b700 1 -- 172.21.15.121:0/1943028455 wait complete. 2024-04-05T21:32:03.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.252+0000 7f751fa81700 1 -- 172.21.15.121:0/78930481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7518068670 msgr2=0x7f7518068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:03.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.252+0000 7f751fa81700 1 --2- 172.21.15.121:0/78930481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7518068670 0x7f7518068a50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7514009910 tx=0x7f7514009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:03.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.252+0000 7f751fa81700 1 -- 172.21.15.121:0/78930481 shutdown_connections 2024-04-05T21:32:03.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.252+0000 7f751fa81700 1 --2- 172.21.15.121:0/78930481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7518069020 0x7f7518105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:03.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.252+0000 7f751fa81700 1 --2- 172.21.15.121:0/78930481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7518068670 0x7f7518068a50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:03.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.252+0000 7f751fa81700 1 -- 172.21.15.121:0/78930481 >> 172.21.15.121:0/78930481 conn(0x7f75180755a0 msgr2=0x7f75180759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:03.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.252+0000 7f751fa81700 1 -- 172.21.15.121:0/78930481 shutdown_connections 2024-04-05T21:32:03.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.252+0000 7f751fa81700 1 -- 172.21.15.121:0/78930481 wait complete. 2024-04-05T21:32:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.252+0000 7f751fa81700 1 Processor -- start 2024-04-05T21:32:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751fa81700 1 -- start start 2024-04-05T21:32:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751fa81700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7518068670 0x7f7518196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751fa81700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7518069020 0x7f7518196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751fa81700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518197260 con 0x7f7518069020 2024-04-05T21:32:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751fa81700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518199f90 con 0x7f7518068670 2024-04-05T21:32:03.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751d01c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7518069020 0x7f7518196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:03.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751d01c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7518069020 0x7f7518196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54112/0 (socket says 172.21.15.121:54112) 2024-04-05T21:32:03.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751d01c700 1 -- 172.21.15.121:0/2133271640 learned_addr learned my addr 172.21.15.121:0/2133271640 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:03.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751d81d700 1 --2- 172.21.15.121:0/2133271640 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7518068670 0x7f7518196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:03.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751d01c700 1 -- 172.21.15.121:0/2133271640 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7518068670 msgr2=0x7f7518196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:03.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751d01c700 1 --2- 172.21.15.121:0/2133271640 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7518068670 0x7f7518196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:03.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751d01c700 1 -- 172.21.15.121:0/2133271640 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75140095f0 con 0x7f7518069020 2024-04-05T21:32:03.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751d01c700 1 --2- 172.21.15.121:0/2133271640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7518069020 0x7f7518196bd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f750800d8d0 tx=0x7f750800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:03.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f750effd700 1 -- 172.21.15.121:0/2133271640 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7508004a00 con 0x7f7518069020 2024-04-05T21:32:03.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f750effd700 1 -- 172.21.15.121:0/2133271640 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7508005020 con 0x7f7518069020 2024-04-05T21:32:03.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f751fa81700 1 -- 172.21.15.121:0/2133271640 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751819a270 con 0x7f7518069020 2024-04-05T21:32:03.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.253+0000 7f750effd700 1 -- 172.21.15.121:0/2133271640 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7508009bd0 con 0x7f7518069020 2024-04-05T21:32:03.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.254+0000 7f751fa81700 1 -- 172.21.15.121:0/2133271640 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f751819a7c0 con 0x7f7518069020 2024-04-05T21:32:03.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.254+0000 7f751fa81700 1 -- 172.21.15.121:0/2133271640 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7518109130 con 0x7f7518069020 2024-04-05T21:32:03.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.255+0000 7f750effd700 1 -- 172.21.15.121:0/2133271640 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7508005190 con 0x7f7518069020 2024-04-05T21:32:03.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.255+0000 7f750effd700 1 --2- 172.21.15.121:0/2133271640 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f750406cce0 0x7f750406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:03.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.255+0000 7f750effd700 1 -- 172.21.15.121:0/2133271640 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f750808be40 con 0x7f7518069020 2024-04-05T21:32:03.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.255+0000 7f751d81d700 1 --2- 172.21.15.121:0/2133271640 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f750406cce0 0x7f750406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:03.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.256+0000 7f751d81d700 1 --2- 172.21.15.121:0/2133271640 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f750406cce0 0x7f750406f1a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f751400b5c0 tx=0x7f7514005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:03.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.260+0000 7f750effd700 1 -- 172.21.15.121:0/2133271640 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7508055f60 con 0x7f7518069020 2024-04-05T21:32:03.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:03 smithi121 ceph-mon[145859]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:03.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:03 smithi121 ceph-mon[145859]: pgmap v146: 65 pgs: 14 active+undersized+degraded, 3 stale+active+clean, 15 active+undersized, 33 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 47/333 objects degraded (14.114%) 2024-04-05T21:32:03.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:03 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2796671374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:03.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:03 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4210474612' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:32:03.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:03 smithi121 ceph-mon[145859]: Health check failed: Degraded data redundancy: 47/333 objects degraded (14.114%), 14 pgs degraded (PG_DEGRADED) 2024-04-05T21:32:03.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.543+0000 7f751fa81700 1 -- 172.21.15.121:0/2133271640 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f751819aaa0 con 0x7f7518069020 2024-04-05T21:32:03.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.543+0000 7f750effd700 1 -- 172.21.15.121:0/2133271640 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1049 (secure 0 0 0) 0x7f7508017430 con 0x7f7518069020 2024-04-05T21:32:03.544 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 47/333 objects degraded (14.114%), 14 pgs degraded 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout: osd.1 (root=default,host=smithi121) is down 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 47/333 objects degraded (14.114%), 14 pgs degraded 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout: pg 1.0 is active+undersized+degraded, acting [3,0] 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.0 is active+undersized+degraded, acting [3,0] 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.2 is active+undersized+degraded, acting [5,0] 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.6 is active+undersized+degraded, acting [3,4] 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.9 is active+undersized+degraded, acting [4,0] 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.a is active+undersized+degraded, acting [4,3] 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.d is active+undersized+degraded, acting [3,2] 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-04-05T21:32:03.545 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.11 is active+undersized+degraded, acting [4,3] 2024-04-05T21:32:03.546 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.12 is active+undersized+degraded, acting [3,0] 2024-04-05T21:32:03.546 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.15 is active+undersized+degraded, acting [3,0] 2024-04-05T21:32:03.546 INFO:teuthology.orchestra.run.smithi121.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-04-05T21:32:03.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.546+0000 7f751fa81700 1 -- 172.21.15.121:0/2133271640 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f750406cce0 msgr2=0x7f750406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:03.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.546+0000 7f751fa81700 1 --2- 172.21.15.121:0/2133271640 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f750406cce0 0x7f750406f1a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f751400b5c0 tx=0x7f7514005fd0 comp rx=0 tx=0).stop 2024-04-05T21:32:03.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.546+0000 7f751fa81700 1 -- 172.21.15.121:0/2133271640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7518069020 msgr2=0x7f7518196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:03.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.546+0000 7f751fa81700 1 --2- 172.21.15.121:0/2133271640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7518069020 0x7f7518196bd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f750800d8d0 tx=0x7f750800dc90 comp rx=0 tx=0).stop 2024-04-05T21:32:03.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.546+0000 7f751fa81700 1 -- 172.21.15.121:0/2133271640 shutdown_connections 2024-04-05T21:32:03.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.546+0000 7f751fa81700 1 --2- 172.21.15.121:0/2133271640 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f750406cce0 0x7f750406f1a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:03.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.546+0000 7f751fa81700 1 --2- 172.21.15.121:0/2133271640 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7518068670 0x7f7518196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:03.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.546+0000 7f751fa81700 1 --2- 172.21.15.121:0/2133271640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7518069020 0x7f7518196bd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:03.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.546+0000 7f751fa81700 1 -- 172.21.15.121:0/2133271640 >> 172.21.15.121:0/2133271640 conn(0x7f75180755a0 msgr2=0x7f75180fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:03.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.546+0000 7f751fa81700 1 -- 172.21.15.121:0/2133271640 shutdown_connections 2024-04-05T21:32:03.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:03.546+0000 7f751fa81700 1 -- 172.21.15.121:0/2133271640 wait complete. 2024-04-05T21:32:03.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:03 smithi188 ceph-mon[107793]: from='client.34168 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:03 smithi188 ceph-mon[107793]: pgmap v146: 65 pgs: 14 active+undersized+degraded, 3 stale+active+clean, 15 active+undersized, 33 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 47/333 objects degraded (14.114%) 2024-04-05T21:32:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:03 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2796671374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:03 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4210474612' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:32:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:03 smithi188 ceph-mon[107793]: Health check failed: Degraded data redundancy: 47/333 objects degraded (14.114%), 14 pgs degraded (PG_DEGRADED) 2024-04-05T21:32:03.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:03 smithi121 bash[164145]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-05T21:32:03.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:03 smithi121 bash[164145]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-05T21:32:03.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:03 smithi121 bash[164145]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-05T21:32:03.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:03 smithi121 bash[164145]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-05T21:32:03.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:03 smithi121 bash[164145]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-05T21:32:03.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:03 smithi121 bash[164145]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-05T21:32:03.989 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:03 smithi121 bash[164145]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-05T21:32:04.459 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:04 smithi121 ceph-mon[145859]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:04.459 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:04 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2133271640' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:32:04.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:04 smithi188 ceph-mon[107793]: from='client.34180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:04.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:04 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2133271640' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:32:05.489 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:05 smithi121 ceph-osd[164496]: -- 172.21.15.121:0/3914170549 <== mon.0 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5604fa0431e0 con 0x5604fa013000 2024-04-05T21:32:05.489 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:05 smithi121 bash[164457]: 0fd0bdd316f71a78c6ba79ba383447b9ec53a84f80ce5b02d4b6d4e59910cb22 2024-04-05T21:32:05.489 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:05 smithi121 systemd[1]: Started Ceph osd.1 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:32:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:05 smithi121 ceph-mon[145859]: pgmap v147: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 54/333 objects degraded (16.216%) 2024-04-05T21:32:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:05 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:05 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:05.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:05 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:32:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:05 smithi188 ceph-mon[107793]: pgmap v147: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 54/333 objects degraded (16.216%) 2024-04-05T21:32:05.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:05 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:05.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:05 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:05.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:05 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:32:05.981 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:05 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1[164473]: 2024-04-05T21:32:05.619+0000 7f638724a7c0 -1 Falling back to public interface 2024-04-05T21:32:07.651 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:07 smithi121 ceph-mon[145859]: pgmap v148: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 54/333 objects degraded (16.216%) 2024-04-05T21:32:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:07 smithi188 ceph-mon[107793]: pgmap v148: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 54/333 objects degraded (16.216%) 2024-04-05T21:32:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:09 smithi121 ceph-mon[145859]: pgmap v149: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 54/333 objects degraded (16.216%) 2024-04-05T21:32:09.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:09 smithi188 ceph-mon[107793]: pgmap v149: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 54/333 objects degraded (16.216%) 2024-04-05T21:32:10.489 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:10 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1[164473]: 2024-04-05T21:32:10.179+0000 7f638724a7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-05T21:32:10.990 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:10 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1[164473]: 2024-04-05T21:32:10.712+0000 7f638724a7c0 -1 osd.1 53 log_to_monitors true 2024-04-05T21:32:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:11 smithi121 ceph-mon[145859]: pgmap v150: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 54/333 objects degraded (16.216%) 2024-04-05T21:32:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:11 smithi121 ceph-mon[145859]: from='osd.1 [v2:172.21.15.121:6810/627590186,v1:172.21.15.121:6811/627590186]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T21:32:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:11 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:11 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:11.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:11 smithi188 ceph-mon[107793]: pgmap v150: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 54/333 objects degraded (16.216%) 2024-04-05T21:32:11.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:11 smithi188 ceph-mon[107793]: from='osd.1 [v2:172.21.15.121:6810/627590186,v1:172.21.15.121:6811/627590186]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-05T21:32:11.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:11 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:11.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:11 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:12.313 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 21:32:12 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1[164473]: 2024-04-05T21:32:12.197+0000 7f637469c700 -1 osd.1 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T21:32:12.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:12 smithi121 ceph-mon[145859]: from='osd.1 [v2:172.21.15.121:6810/627590186,v1:172.21.15.121:6811/627590186]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-05T21:32:12.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:12 smithi121 ceph-mon[145859]: osdmap e56: 6 total, 5 up, 6 in 2024-04-05T21:32:12.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:12 smithi121 ceph-mon[145859]: from='osd.1 [v2:172.21.15.121:6810/627590186,v1:172.21.15.121:6811/627590186]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-05T21:32:12.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:12 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:12.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:12 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:12.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:12 smithi121 ceph-mon[145859]: Health check update: Degraded data redundancy: 54/333 objects degraded (16.216%), 16 pgs degraded (PG_DEGRADED) 2024-04-05T21:32:12.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:12 smithi121 ceph-mon[145859]: from='osd.1 [v2:172.21.15.121:6810/627590186,v1:172.21.15.121:6811/627590186]' entity='osd.1' 2024-04-05T21:32:12.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:12 smithi188 ceph-mon[107793]: from='osd.1 [v2:172.21.15.121:6810/627590186,v1:172.21.15.121:6811/627590186]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-05T21:32:12.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:12 smithi188 ceph-mon[107793]: osdmap e56: 6 total, 5 up, 6 in 2024-04-05T21:32:12.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:12 smithi188 ceph-mon[107793]: from='osd.1 [v2:172.21.15.121:6810/627590186,v1:172.21.15.121:6811/627590186]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-05T21:32:12.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:12 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:12.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:12 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:12.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:12 smithi188 ceph-mon[107793]: Health check update: Degraded data redundancy: 54/333 objects degraded (16.216%), 16 pgs degraded (PG_DEGRADED) 2024-04-05T21:32:12.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:12 smithi188 ceph-mon[107793]: from='osd.1 [v2:172.21.15.121:6810/627590186,v1:172.21.15.121:6811/627590186]' entity='osd.1' 2024-04-05T21:32:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:13 smithi121 ceph-mon[145859]: pgmap v152: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3 op/s; 54/333 objects degraded (16.216%) 2024-04-05T21:32:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:13 smithi121 ceph-mon[145859]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T21:32:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:13 smithi121 ceph-mon[145859]: osd.1 [v2:172.21.15.121:6810/627590186,v1:172.21.15.121:6811/627590186] boot 2024-04-05T21:32:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:13 smithi121 ceph-mon[145859]: osdmap e57: 6 total, 6 up, 6 in 2024-04-05T21:32:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:13 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:32:13.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:13 smithi188 ceph-mon[107793]: pgmap v152: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3 op/s; 54/333 objects degraded (16.216%) 2024-04-05T21:32:13.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:13 smithi188 ceph-mon[107793]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T21:32:13.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:13 smithi188 ceph-mon[107793]: osd.1 [v2:172.21.15.121:6810/627590186,v1:172.21.15.121:6811/627590186] boot 2024-04-05T21:32:13.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:13 smithi188 ceph-mon[107793]: osdmap e57: 6 total, 6 up, 6 in 2024-04-05T21:32:13.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:13 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-05T21:32:15.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:15 smithi121 ceph-mon[145859]: pgmap v154: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 54/333 objects degraded (16.216%) 2024-04-05T21:32:15.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:15 smithi121 ceph-mon[145859]: osdmap e58: 6 total, 6 up, 6 in 2024-04-05T21:32:15.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:15.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:32:15.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:15 smithi188 ceph-mon[107793]: pgmap v154: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s; 54/333 objects degraded (16.216%) 2024-04-05T21:32:15.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:15 smithi188 ceph-mon[107793]: osdmap e58: 6 total, 6 up, 6 in 2024-04-05T21:32:15.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:15.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:32:16.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:16.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:16.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:32:16.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:32:16.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:16.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:32:16.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:16.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:16.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:16.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T21:32:16.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:16 smithi121 ceph-mon[145859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T21:32:16.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:16 smithi121 ceph-mon[145859]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-05T21:32:16.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:16.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:16.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:32:16.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:32:16.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:16.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:32:16.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:16.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:16.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:16.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T21:32:16.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:16 smithi188 ceph-mon[107793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T21:32:16.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:16 smithi188 ceph-mon[107793]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-05T21:32:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:17 smithi121 ceph-mon[145859]: pgmap v156: 65 pgs: 7 active+undersized+degraded, 7 active+undersized, 51 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 3.0 KiB/s rd, 5 op/s; 23/333 objects degraded (6.907%) 2024-04-05T21:32:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:17 smithi121 ceph-mon[145859]: Health check update: Degraded data redundancy: 23/333 objects degraded (6.907%), 7 pgs degraded (PG_DEGRADED) 2024-04-05T21:32:17.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:17 smithi188 ceph-mon[107793]: pgmap v156: 65 pgs: 7 active+undersized+degraded, 7 active+undersized, 51 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 3.0 KiB/s rd, 5 op/s; 23/333 objects degraded (6.907%) 2024-04-05T21:32:17.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:17 smithi188 ceph-mon[107793]: Health check update: Degraded data redundancy: 23/333 objects degraded (6.907%), 7 pgs degraded (PG_DEGRADED) 2024-04-05T21:32:18.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:18 smithi121 ceph-mon[145859]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/333 objects degraded (6.907%), 7 pgs degraded) 2024-04-05T21:32:18.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:18 smithi121 ceph-mon[145859]: Cluster is now healthy 2024-04-05T21:32:18.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:18 smithi188 ceph-mon[107793]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/333 objects degraded (6.907%), 7 pgs degraded) 2024-04-05T21:32:18.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:18 smithi188 ceph-mon[107793]: Cluster is now healthy 2024-04-05T21:32:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:19 smithi121 ceph-mon[145859]: pgmap v157: 65 pgs: 2 active+undersized, 63 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T21:32:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:19 smithi188 ceph-mon[107793]: pgmap v157: 65 pgs: 2 active+undersized, 63 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-05T21:32:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:21 smithi121 ceph-mon[145859]: pgmap v158: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-04-05T21:32:21.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:21 smithi188 ceph-mon[107793]: pgmap v158: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 1 op/s 2024-04-05T21:32:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:23 smithi121 ceph-mon[145859]: pgmap v159: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T21:32:23.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:23 smithi188 ceph-mon[107793]: pgmap v159: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T21:32:25.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:25 smithi121 ceph-mon[145859]: pgmap v160: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-05T21:32:25.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:25 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:25.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:25 smithi188 ceph-mon[107793]: pgmap v160: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-05T21:32:25.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:25 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:27 smithi121 ceph-mon[145859]: pgmap v161: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-05T21:32:27.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:27 smithi188 ceph-mon[107793]: pgmap v161: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-05T21:32:29.713 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:29 smithi121 ceph-mon[145859]: pgmap v162: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:32:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:29 smithi188 ceph-mon[107793]: pgmap v162: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:32:31.422 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:31 smithi121 ceph-mon[145859]: pgmap v163: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:32:31.422 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:31.422 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:32:31.422 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T21:32:31.422 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:31 smithi121 ceph-mon[145859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T21:32:31.422 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:31 smithi121 ceph-mon[145859]: Upgrade: osd.2 is safe to restart 2024-04-05T21:32:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:31 smithi188 ceph-mon[107793]: pgmap v163: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:32:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:32:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T21:32:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:31 smithi188 ceph-mon[107793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-05T21:32:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:31 smithi188 ceph-mon[107793]: Upgrade: osd.2 is safe to restart 2024-04-05T21:32:33.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:33 smithi188 ceph-mon[107793]: pgmap v164: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:32:33.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:33 smithi188 ceph-mon[107793]: Upgrade: Updating osd.2 2024-04-05T21:32:33.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:33 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:33.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:33 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-05T21:32:33.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:33 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:32:33.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:33 smithi188 ceph-mon[107793]: Deploying daemon osd.2 on smithi121 2024-04-05T21:32:33.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.654+0000 7f843339f700 1 -- 172.21.15.121:0/2238522886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f842c068670 msgr2=0x7f842c068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.654+0000 7f843339f700 1 --2- 172.21.15.121:0/2238522886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f842c068670 0x7f842c068a50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8428009910 tx=0x7f8428009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.654+0000 7f843339f700 1 -- 172.21.15.121:0/2238522886 shutdown_connections 2024-04-05T21:32:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.654+0000 7f843339f700 1 --2- 172.21.15.121:0/2238522886 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f842c069020 0x7f842c105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.654+0000 7f843339f700 1 --2- 172.21.15.121:0/2238522886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f842c068670 0x7f842c068a50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.654+0000 7f843339f700 1 -- 172.21.15.121:0/2238522886 >> 172.21.15.121:0/2238522886 conn(0x7f842c0755a0 msgr2=0x7f842c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.654+0000 7f843339f700 1 -- 172.21.15.121:0/2238522886 shutdown_connections 2024-04-05T21:32:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.654+0000 7f843339f700 1 -- 172.21.15.121:0/2238522886 wait complete. 2024-04-05T21:32:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.654+0000 7f843339f700 1 Processor -- start 2024-04-05T21:32:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.654+0000 7f843339f700 1 -- start start 2024-04-05T21:32:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843339f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f842c068670 0x7f842c19c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843339f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f842c069020 0x7f842c19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843339f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c19d170 con 0x7f842c069020 2024-04-05T21:32:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843339f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c196620 con 0x7f842c068670 2024-04-05T21:32:33.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843093a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f842c069020 0x7f842c19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:33.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843093a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f842c069020 0x7f842c19cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56094/0 (socket says 172.21.15.121:56094) 2024-04-05T21:32:33.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843093a700 1 -- 172.21.15.121:0/4259219647 learned_addr learned my addr 172.21.15.121:0/4259219647 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:33.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843113b700 1 --2- 172.21.15.121:0/4259219647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f842c068670 0x7f842c19c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:33.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843093a700 1 -- 172.21.15.121:0/4259219647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f842c068670 msgr2=0x7f842c19c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:33.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843093a700 1 --2- 172.21.15.121:0/4259219647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f842c068670 0x7f842c19c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:33.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843093a700 1 -- 172.21.15.121:0/4259219647 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84280095f0 con 0x7f842c069020 2024-04-05T21:32:33.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843113b700 1 --2- 172.21.15.121:0/4259219647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f842c068670 0x7f842c19c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:32:33.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843093a700 1 --2- 172.21.15.121:0/4259219647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f842c069020 0x7f842c19cae0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f841c00b480 tx=0x7f841c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:33.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f84227fc700 1 -- 172.21.15.121:0/4259219647 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f841c0118c0 con 0x7f842c069020 2024-04-05T21:32:33.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f84227fc700 1 -- 172.21.15.121:0/4259219647 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f841c010380 con 0x7f842c069020 2024-04-05T21:32:33.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843339f700 1 -- 172.21.15.121:0/4259219647 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842c196900 con 0x7f842c069020 2024-04-05T21:32:33.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f84227fc700 1 -- 172.21.15.121:0/4259219647 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f841c00f510 con 0x7f842c069020 2024-04-05T21:32:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.655+0000 7f843339f700 1 -- 172.21.15.121:0/4259219647 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842c196e50 con 0x7f842c069020 2024-04-05T21:32:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.656+0000 7f843339f700 1 -- 172.21.15.121:0/4259219647 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842c109130 con 0x7f842c069020 2024-04-05T21:32:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.657+0000 7f84227fc700 1 -- 172.21.15.121:0/4259219647 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f841c0104f0 con 0x7f842c069020 2024-04-05T21:32:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.657+0000 7f84227fc700 1 --2- 172.21.15.121:0/4259219647 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f841806cce0 0x7f841806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.657+0000 7f84227fc700 1 -- 172.21.15.121:0/4259219647 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f841c08c160 con 0x7f842c069020 2024-04-05T21:32:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.657+0000 7f843113b700 1 --2- 172.21.15.121:0/4259219647 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f841806cce0 0x7f841806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.658+0000 7f843113b700 1 --2- 172.21.15.121:0/4259219647 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f841806cce0 0x7f841806f1a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f842800b5c0 tx=0x7f842801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:33.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.662+0000 7f84227fc700 1 -- 172.21.15.121:0/4259219647 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f841c059d10 con 0x7f842c069020 2024-04-05T21:32:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:33 smithi121 ceph-mon[145859]: pgmap v164: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:32:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:33 smithi121 ceph-mon[145859]: Upgrade: Updating osd.2 2024-04-05T21:32:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:33 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:33 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-05T21:32:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:33 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:32:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:33 smithi121 ceph-mon[145859]: Deploying daemon osd.2 on smithi121 2024-04-05T21:32:33.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.879+0000 7f843339f700 1 -- 172.21.15.121:0/4259219647 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f842c197150 con 0x7f841806cce0 2024-04-05T21:32:33.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.880+0000 7f84227fc700 1 -- 172.21.15.121:0/4259219647 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f842c197150 con 0x7f841806cce0 2024-04-05T21:32:33.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.883+0000 7f843339f700 1 -- 172.21.15.121:0/4259219647 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f841806cce0 msgr2=0x7f841806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:33.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.883+0000 7f843339f700 1 --2- 172.21.15.121:0/4259219647 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f841806cce0 0x7f841806f1a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f842800b5c0 tx=0x7f842801a040 comp rx=0 tx=0).stop 2024-04-05T21:32:33.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.883+0000 7f843339f700 1 -- 172.21.15.121:0/4259219647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f842c069020 msgr2=0x7f842c19cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:33.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.883+0000 7f843339f700 1 --2- 172.21.15.121:0/4259219647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f842c069020 0x7f842c19cae0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f841c00b480 tx=0x7f841c00b840 comp rx=0 tx=0).stop 2024-04-05T21:32:33.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.883+0000 7f843339f700 1 -- 172.21.15.121:0/4259219647 shutdown_connections 2024-04-05T21:32:33.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.883+0000 7f843339f700 1 --2- 172.21.15.121:0/4259219647 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f841806cce0 0x7f841806f1a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:33.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.883+0000 7f843339f700 1 --2- 172.21.15.121:0/4259219647 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f842c068670 0x7f842c19c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:33.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.883+0000 7f843339f700 1 --2- 172.21.15.121:0/4259219647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f842c069020 0x7f842c19cae0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:33.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.883+0000 7f843339f700 1 -- 172.21.15.121:0/4259219647 >> 172.21.15.121:0/4259219647 conn(0x7f842c0755a0 msgr2=0x7f842c0fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:33.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.883+0000 7f843339f700 1 -- 172.21.15.121:0/4259219647 shutdown_connections 2024-04-05T21:32:33.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.883+0000 7f843339f700 1 -- 172.21.15.121:0/4259219647 wait complete. 2024-04-05T21:32:33.895 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:32:33.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.993+0000 7f2a07310700 1 -- 172.21.15.121:0/1334148890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000ffbd0 msgr2=0x7f2a00100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:33.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.993+0000 7f2a07310700 1 --2- 172.21.15.121:0/1334148890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000ffbd0 0x7f2a00100050 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009910 tx=0x7f29f4009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:33.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.993+0000 7f2a07310700 1 -- 172.21.15.121:0/1334148890 shutdown_connections 2024-04-05T21:32:33.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.993+0000 7f2a07310700 1 --2- 172.21.15.121:0/1334148890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000ffbd0 0x7f2a00100050 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:33.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.993+0000 7f2a07310700 1 --2- 172.21.15.121:0/1334148890 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a00105a40 0x7f2a00105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:33.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.993+0000 7f2a07310700 1 -- 172.21.15.121:0/1334148890 >> 172.21.15.121:0/1334148890 conn(0x7f2a000755a0 msgr2=0x7f2a000759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:33.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.993+0000 7f2a07310700 1 -- 172.21.15.121:0/1334148890 shutdown_connections 2024-04-05T21:32:33.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.993+0000 7f2a07310700 1 -- 172.21.15.121:0/1334148890 wait complete. 2024-04-05T21:32:33.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.993+0000 7f2a07310700 1 Processor -- start 2024-04-05T21:32:33.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a07310700 1 -- start start 2024-04-05T21:32:33.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a07310700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000ffbd0 0x7f2a0019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:33.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a07310700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a00105a40 0x7f2a0019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:33.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a07310700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0019d0e0 con 0x7f2a000ffbd0 2024-04-05T21:32:33.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a07310700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a00196590 con 0x7f2a00105a40 2024-04-05T21:32:33.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a050ac700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000ffbd0 0x7f2a0019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:33.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a050ac700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000ffbd0 0x7f2a0019c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56114/0 (socket says 172.21.15.121:56114) 2024-04-05T21:32:33.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a050ac700 1 -- 172.21.15.121:0/1064803908 learned_addr learned my addr 172.21.15.121:0/1064803908 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:33.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a048ab700 1 --2- 172.21.15.121:0/1064803908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a00105a40 0x7f2a0019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:33.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a050ac700 1 -- 172.21.15.121:0/1064803908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a00105a40 msgr2=0x7f2a0019ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:33.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a050ac700 1 --2- 172.21.15.121:0/1064803908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a00105a40 0x7f2a0019ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:33.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a050ac700 1 -- 172.21.15.121:0/1064803908 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f40095f0 con 0x7f2a000ffbd0 2024-04-05T21:32:33.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a050ac700 1 --2- 172.21.15.121:0/1064803908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000ffbd0 0x7f2a0019c510 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f29f800c8a0 tx=0x7f29f800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:33.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f29fe7fc700 1 -- 172.21.15.121:0/1064803908 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29f80078e0 con 0x7f2a000ffbd0 2024-04-05T21:32:33.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f29fe7fc700 1 -- 172.21.15.121:0/1064803908 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29f800f450 con 0x7f2a000ffbd0 2024-04-05T21:32:33.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a07310700 1 -- 172.21.15.121:0/1064803908 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a00196870 con 0x7f2a000ffbd0 2024-04-05T21:32:33.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f29fe7fc700 1 -- 172.21.15.121:0/1064803908 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29f800e5e0 con 0x7f2a000ffbd0 2024-04-05T21:32:33.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.994+0000 7f2a07310700 1 -- 172.21.15.121:0/1064803908 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a00196dc0 con 0x7f2a000ffbd0 2024-04-05T21:32:33.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.996+0000 7f2a07310700 1 -- 172.21.15.121:0/1064803908 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a00108fe0 con 0x7f2a000ffbd0 2024-04-05T21:32:33.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.996+0000 7f29fe7fc700 1 -- 172.21.15.121:0/1064803908 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f29f800e740 con 0x7f2a000ffbd0 2024-04-05T21:32:33.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.996+0000 7f29fe7fc700 1 --2- 172.21.15.121:0/1064803908 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f29e806cce0 0x7f29e806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:33.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.996+0000 7f2a048ab700 1 --2- 172.21.15.121:0/1064803908 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f29e806cce0 0x7f29e806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:33.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.996+0000 7f29fe7fc700 1 -- 172.21.15.121:0/1064803908 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f29f808c120 con 0x7f2a000ffbd0 2024-04-05T21:32:33.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:33.997+0000 7f2a048ab700 1 --2- 172.21.15.121:0/1064803908 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f29e806cce0 0x7f29e806f1a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2a00104ef0 tx=0x7f29f401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:34.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.001+0000 7f29fe7fc700 1 -- 172.21.15.121:0/1064803908 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f29f8059cd0 con 0x7f2a000ffbd0 2024-04-05T21:32:34.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.225+0000 7f2a07310700 1 -- 172.21.15.121:0/1064803908 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a00197040 con 0x7f29e806cce0 2024-04-05T21:32:34.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.226+0000 7f29fe7fc700 1 -- 172.21.15.121:0/1064803908 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2a00197040 con 0x7f29e806cce0 2024-04-05T21:32:34.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.228+0000 7f2a07310700 1 -- 172.21.15.121:0/1064803908 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f29e806cce0 msgr2=0x7f29e806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:34.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.228+0000 7f2a07310700 1 --2- 172.21.15.121:0/1064803908 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f29e806cce0 0x7f29e806f1a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2a00104ef0 tx=0x7f29f401a040 comp rx=0 tx=0).stop 2024-04-05T21:32:34.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.228+0000 7f2a07310700 1 -- 172.21.15.121:0/1064803908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000ffbd0 msgr2=0x7f2a0019c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:34.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.228+0000 7f2a07310700 1 --2- 172.21.15.121:0/1064803908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000ffbd0 0x7f2a0019c510 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f29f800c8a0 tx=0x7f29f800cbb0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.228+0000 7f2a07310700 1 -- 172.21.15.121:0/1064803908 shutdown_connections 2024-04-05T21:32:34.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.228+0000 7f2a07310700 1 --2- 172.21.15.121:0/1064803908 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f29e806cce0 0x7f29e806f1a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.228+0000 7f2a07310700 1 --2- 172.21.15.121:0/1064803908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a000ffbd0 0x7f2a0019c510 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.228+0000 7f2a07310700 1 --2- 172.21.15.121:0/1064803908 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a00105a40 0x7f2a0019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.228+0000 7f2a07310700 1 -- 172.21.15.121:0/1064803908 >> 172.21.15.121:0/1064803908 conn(0x7f2a000755a0 msgr2=0x7f2a00103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:34.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.228+0000 7f2a07310700 1 -- 172.21.15.121:0/1064803908 shutdown_connections 2024-04-05T21:32:34.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.228+0000 7f2a07310700 1 -- 172.21.15.121:0/1064803908 wait complete. 2024-04-05T21:32:34.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.335+0000 7f21dfb85700 1 -- 172.21.15.121:0/3296195997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d8068670 msgr2=0x7f21d8068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:34.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.335+0000 7f21dfb85700 1 --2- 172.21.15.121:0/3296195997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d8068670 0x7f21d8068a50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009910 tx=0x7f21cc009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:34.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.335+0000 7f21dfb85700 1 -- 172.21.15.121:0/3296195997 shutdown_connections 2024-04-05T21:32:34.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.335+0000 7f21dfb85700 1 --2- 172.21.15.121:0/3296195997 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21d8069020 0x7f21d8105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.335+0000 7f21dfb85700 1 --2- 172.21.15.121:0/3296195997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d8068670 0x7f21d8068a50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.335+0000 7f21dfb85700 1 -- 172.21.15.121:0/3296195997 >> 172.21.15.121:0/3296195997 conn(0x7f21d80755a0 msgr2=0x7f21d80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:34.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.335+0000 7f21dfb85700 1 -- 172.21.15.121:0/3296195997 shutdown_connections 2024-04-05T21:32:34.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.335+0000 7f21dfb85700 1 -- 172.21.15.121:0/3296195997 wait complete. 2024-04-05T21:32:34.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dfb85700 1 Processor -- start 2024-04-05T21:32:34.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dfb85700 1 -- start start 2024-04-05T21:32:34.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dfb85700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d8068670 0x7f21d8196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:34.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dfb85700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21d8069020 0x7f21d8196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:34.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dfb85700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d8197200 con 0x7f21d8068670 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dfb85700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d8197340 con 0x7f21d8069020 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dd921700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d8068670 0x7f21d8196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dd921700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d8068670 0x7f21d8196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40606/0 (socket says 172.21.15.121:40606) 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dd921700 1 -- 172.21.15.121:0/1205533253 learned_addr learned my addr 172.21.15.121:0/1205533253 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dd120700 1 --2- 172.21.15.121:0/1205533253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21d8069020 0x7f21d8196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dd921700 1 -- 172.21.15.121:0/1205533253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21d8069020 msgr2=0x7f21d8196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dd921700 1 --2- 172.21.15.121:0/1205533253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21d8069020 0x7f21d8196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dd921700 1 -- 172.21.15.121:0/1205533253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc0095f0 con 0x7f21d8068670 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dd120700 1 --2- 172.21.15.121:0/1205533253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21d8069020 0x7f21d8196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21dd921700 1 --2- 172.21.15.121:0/1205533253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d8068670 0x7f21d8196670 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f21cc006010 tx=0x7f21cc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.336+0000 7f21caffd700 1 -- 172.21.15.121:0/1205533253 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21cc01d070 con 0x7f21d8068670 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.337+0000 7f21caffd700 1 -- 172.21.15.121:0/1205533253 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21cc00f460 con 0x7f21d8068670 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.337+0000 7f21dfb85700 1 -- 172.21.15.121:0/1205533253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d819b130 con 0x7f21d8068670 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.337+0000 7f21caffd700 1 -- 172.21.15.121:0/1205533253 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21cc021640 con 0x7f21d8068670 2024-04-05T21:32:34.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.337+0000 7f21dfb85700 1 -- 172.21.15.121:0/1205533253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d819b620 con 0x7f21d8068670 2024-04-05T21:32:34.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.338+0000 7f21dfb85700 1 -- 172.21.15.121:0/1205533253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d8109130 con 0x7f21d8068670 2024-04-05T21:32:34.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.338+0000 7f21caffd700 1 -- 172.21.15.121:0/1205533253 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f21cc00fa20 con 0x7f21d8068670 2024-04-05T21:32:34.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.338+0000 7f21caffd700 1 --2- 172.21.15.121:0/1205533253 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f21c406cce0 0x7f21c406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:34.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.339+0000 7f21dd120700 1 --2- 172.21.15.121:0/1205533253 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f21c406cce0 0x7f21c406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:34.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.339+0000 7f21caffd700 1 -- 172.21.15.121:0/1205533253 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f21cc08d700 con 0x7f21d8068670 2024-04-05T21:32:34.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.339+0000 7f21dd120700 1 --2- 172.21.15.121:0/1205533253 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f21c406cce0 0x7f21c406f1a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f21d4005fd0 tx=0x7f21d4005e20 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:34.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.343+0000 7f21caffd700 1 -- 172.21.15.121:0/1205533253 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f21cc0578a0 con 0x7f21d8068670 2024-04-05T21:32:34.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.551+0000 7f21dfb85700 1 -- 172.21.15.121:0/1205533253 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f21d802cd60 con 0x7f21c406cce0 2024-04-05T21:32:34.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.562+0000 7f21caffd700 1 -- 172.21.15.121:0/1205533253 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f21d802cd60 con 0x7f21c406cce0 2024-04-05T21:32:34.563 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:32:34.563 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (3m) 23s ago 14m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:32:34.563 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (15m) 23s ago 15m 8238k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:32:34.563 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (13m) 84s ago 13m 8044k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:32:34.563 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (94s) 23s ago 15m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:32:34.563 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (88s) 84s ago 13m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:32:34.563 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (3m) 23s ago 14m 80.4M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:32:34.563 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (8m) 23s ago 8m 64.5M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:32:34.563 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (8m) 23s ago 8m 134M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:32:34.563 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (8m) 84s ago 8m 110M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:32:34.563 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (8m) 84s ago 8m 62.2M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:32:34.563 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (5m) 23s ago 16m 506M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:32:34.563 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (4m) 84s ago 13m 450M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:32:34.564 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (2m) 23s ago 16m 43.4M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:32:34.564 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (109s) 84s ago 13m 27.9M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:32:34.564 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (4m) 23s ago 15m 18.0M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:32:34.564 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (4m) 84s ago 13m 16.4M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:32:34.564 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (66s) 23s ago 12m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:32:34.564 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (29s) 23s ago 12m 11.3M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:32:34.564 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (11m) 23s ago 11m 182M 1227M 18.2.1 5be31c24972a c2902a84223d 2024-04-05T21:32:34.564 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (11m) 84s ago 11m 288M 2251M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:32:34.564 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (10m) 84s ago 10m 246M 2251M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:32:34.564 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (10m) 84s ago 10m 224M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:32:34.564 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (3m) 23s ago 14m 65.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:32:34.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.564+0000 7f21dfb85700 1 -- 172.21.15.121:0/1205533253 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f21c406cce0 msgr2=0x7f21c406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:34.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.564+0000 7f21dfb85700 1 --2- 172.21.15.121:0/1205533253 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f21c406cce0 0x7f21c406f1a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f21d4005fd0 tx=0x7f21d4005e20 comp rx=0 tx=0).stop 2024-04-05T21:32:34.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.564+0000 7f21dfb85700 1 -- 172.21.15.121:0/1205533253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d8068670 msgr2=0x7f21d8196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:34.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.564+0000 7f21dfb85700 1 --2- 172.21.15.121:0/1205533253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d8068670 0x7f21d8196670 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f21cc006010 tx=0x7f21cc00be20 comp rx=0 tx=0).stop 2024-04-05T21:32:34.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.564+0000 7f21dfb85700 1 -- 172.21.15.121:0/1205533253 shutdown_connections 2024-04-05T21:32:34.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.564+0000 7f21dfb85700 1 --2- 172.21.15.121:0/1205533253 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f21c406cce0 0x7f21c406f1a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.564+0000 7f21dfb85700 1 --2- 172.21.15.121:0/1205533253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d8068670 0x7f21d8196670 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.564+0000 7f21dfb85700 1 --2- 172.21.15.121:0/1205533253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21d8069020 0x7f21d8196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.564+0000 7f21dfb85700 1 -- 172.21.15.121:0/1205533253 >> 172.21.15.121:0/1205533253 conn(0x7f21d80755a0 msgr2=0x7f21d80fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:34.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.564+0000 7f21dfb85700 1 -- 172.21.15.121:0/1205533253 shutdown_connections 2024-04-05T21:32:34.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.564+0000 7f21dfb85700 1 -- 172.21.15.121:0/1205533253 wait complete. 2024-04-05T21:32:34.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.670+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/4138152488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a04105a40 msgr2=0x7f0a04105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:34.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.670+0000 7f0a0b8d8700 1 --2- 172.21.15.121:0/4138152488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a04105a40 0x7f0a04105e20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f09f4009910 tx=0x7f09f4009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:34.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.670+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/4138152488 shutdown_connections 2024-04-05T21:32:34.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.670+0000 7f0a0b8d8700 1 --2- 172.21.15.121:0/4138152488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a040ffa40 0x7f0a040ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.670+0000 7f0a0b8d8700 1 --2- 172.21.15.121:0/4138152488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a04105a40 0x7f0a04105e20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.670+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/4138152488 >> 172.21.15.121:0/4138152488 conn(0x7f0a040fb570 msgr2=0x7f0a040fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:34.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.670+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/4138152488 shutdown_connections 2024-04-05T21:32:34.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.670+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/4138152488 wait complete. 2024-04-05T21:32:34.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.671+0000 7f0a0b8d8700 1 Processor -- start 2024-04-05T21:32:34.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.671+0000 7f0a0b8d8700 1 -- start start 2024-04-05T21:32:34.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.671+0000 7f0a0b8d8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a040ffa40 0x7f0a04195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.671+0000 7f0a0b8d8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a04105a40 0x7f0a04195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.671+0000 7f0a0b8d8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a04196170 con 0x7f0a040ffa40 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.671+0000 7f0a0b8d8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a04199f00 con 0x7f0a04105a40 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.671+0000 7f0a09674700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a040ffa40 0x7f0a04195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.671+0000 7f0a09674700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a040ffa40 0x7f0a04195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40628/0 (socket says 172.21.15.121:40628) 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.671+0000 7f0a09674700 1 -- 172.21.15.121:0/3598666830 learned_addr learned my addr 172.21.15.121:0/3598666830 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.672+0000 7f0a08e73700 1 --2- 172.21.15.121:0/3598666830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a04105a40 0x7f0a04195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.672+0000 7f0a09674700 1 -- 172.21.15.121:0/3598666830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a04105a40 msgr2=0x7f0a04195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.672+0000 7f0a09674700 1 --2- 172.21.15.121:0/3598666830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a04105a40 0x7f0a04195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.672+0000 7f0a09674700 1 -- 172.21.15.121:0/3598666830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f40095f0 con 0x7f0a040ffa40 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.672+0000 7f0a09674700 1 --2- 172.21.15.121:0/3598666830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a040ffa40 0x7f0a04195550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f09f400b5c0 tx=0x7f09f400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.672+0000 7f09fa7fc700 1 -- 172.21.15.121:0/3598666830 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09f401d070 con 0x7f0a040ffa40 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.672+0000 7f09fa7fc700 1 -- 172.21.15.121:0/3598666830 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09f400f460 con 0x7f0a040ffa40 2024-04-05T21:32:34.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.672+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/3598666830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0419a180 con 0x7f0a040ffa40 2024-04-05T21:32:34.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.672+0000 7f09fa7fc700 1 -- 172.21.15.121:0/3598666830 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09f4021640 con 0x7f0a040ffa40 2024-04-05T21:32:34.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.672+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/3598666830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0419a670 con 0x7f0a040ffa40 2024-04-05T21:32:34.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.673+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/3598666830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0404ea90 con 0x7f0a040ffa40 2024-04-05T21:32:34.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.674+0000 7f09fa7fc700 1 -- 172.21.15.121:0/3598666830 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f09f400fa20 con 0x7f0a040ffa40 2024-04-05T21:32:34.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.675+0000 7f09fa7fc700 1 --2- 172.21.15.121:0/3598666830 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09f006cce0 0x7f09f006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:34.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.675+0000 7f0a08e73700 1 --2- 172.21.15.121:0/3598666830 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09f006cce0 0x7f09f006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:34.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.675+0000 7f09fa7fc700 1 -- 172.21.15.121:0/3598666830 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f09f408db40 con 0x7f0a040ffa40 2024-04-05T21:32:34.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.675+0000 7f0a08e73700 1 --2- 172.21.15.121:0/3598666830 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09f006cce0 0x7f09f006f1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a04196b70 tx=0x7f0a000073d0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:34.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.679+0000 7f09fa7fc700 1 -- 172.21.15.121:0/3598666830 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f09f4057c40 con 0x7f0a040ffa40 2024-04-05T21:32:34.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.995+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/3598666830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0a0419aad0 con 0x7f0a040ffa40 2024-04-05T21:32:34.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.996+0000 7f09fa7fc700 1 -- 172.21.15.121:0/3598666830 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+799 (secure 0 0 0) 0x7f09f405b260 con 0x7f0a040ffa40 2024-04-05T21:32:34.997 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:32:34.997 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:32:34.997 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:32:34.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:32:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:32:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:32:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:32:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:32:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4, 2024-04-05T21:32:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:32:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:32:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:32:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:32:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:32:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:32:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 8, 2024-04-05T21:32:34.999 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:32:34.999 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:32:34.999 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:32:34.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.998+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/3598666830 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09f006cce0 msgr2=0x7f09f006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:34.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.998+0000 7f0a0b8d8700 1 --2- 172.21.15.121:0/3598666830 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09f006cce0 0x7f09f006f1a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a04196b70 tx=0x7f0a000073d0 comp rx=0 tx=0).stop 2024-04-05T21:32:34.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.998+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/3598666830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a040ffa40 msgr2=0x7f0a04195550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.998+0000 7f0a0b8d8700 1 --2- 172.21.15.121:0/3598666830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a040ffa40 0x7f0a04195550 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f09f400b5c0 tx=0x7f09f400bd70 comp rx=0 tx=0).stop 2024-04-05T21:32:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.998+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/3598666830 shutdown_connections 2024-04-05T21:32:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.998+0000 7f0a0b8d8700 1 --2- 172.21.15.121:0/3598666830 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09f006cce0 0x7f09f006f1a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.998+0000 7f0a0b8d8700 1 --2- 172.21.15.121:0/3598666830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a040ffa40 0x7f0a04195550 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.998+0000 7f0a0b8d8700 1 --2- 172.21.15.121:0/3598666830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a04105a40 0x7f0a04195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.998+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/3598666830 >> 172.21.15.121:0/3598666830 conn(0x7f0a040fb570 msgr2=0x7f0a040fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.998+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/3598666830 shutdown_connections 2024-04-05T21:32:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:34.998+0000 7f0a0b8d8700 1 -- 172.21.15.121:0/3598666830 wait complete. 2024-04-05T21:32:35.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 -- 172.21.15.121:0/3850701960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86b0105b50 msgr2=0x7f86b0105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:35.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 --2- 172.21.15.121:0/3850701960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86b0105b50 0x7f86b0105f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f86a4009910 tx=0x7f86a4009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:35.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 -- 172.21.15.121:0/3850701960 shutdown_connections 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 --2- 172.21.15.121:0/3850701960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86b00fef70 0x7f86b00ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 --2- 172.21.15.121:0/3850701960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86b0105b50 0x7f86b0105f30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 -- 172.21.15.121:0/3850701960 >> 172.21.15.121:0/3850701960 conn(0x7f86b0075e30 msgr2=0x7f86b00fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 -- 172.21.15.121:0/3850701960 shutdown_connections 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 -- 172.21.15.121:0/3850701960 wait complete. 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 Processor -- start 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 -- start start 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86b00fef70 0x7f86b0100150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86b0105b50 0x7f86b0100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b0100bd0 con 0x7f86b0105b50 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.101+0000 7f86b70d2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b0100d10 con 0x7f86b00fef70 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86affff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86b0105b50 0x7f86b0100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86affff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86b0105b50 0x7f86b0100690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40636/0 (socket says 172.21.15.121:40636) 2024-04-05T21:32:35.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86affff700 1 -- 172.21.15.121:0/1827911050 learned_addr learned my addr 172.21.15.121:0/1827911050 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:35.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86b4e6e700 1 --2- 172.21.15.121:0/1827911050 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86b00fef70 0x7f86b0100150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:35.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86affff700 1 -- 172.21.15.121:0/1827911050 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86b00fef70 msgr2=0x7f86b0100150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:35.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86affff700 1 --2- 172.21.15.121:0/1827911050 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86b00fef70 0x7f86b0100150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86affff700 1 -- 172.21.15.121:0/1827911050 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a40095f0 con 0x7f86b0105b50 2024-04-05T21:32:35.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86b4e6e700 1 --2- 172.21.15.121:0/1827911050 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86b00fef70 0x7f86b0100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:32:35.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86affff700 1 --2- 172.21.15.121:0/1827911050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86b0105b50 0x7f86b0100690 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f869c00c8a0 tx=0x7f869c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:35.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86adffb700 1 -- 172.21.15.121:0/1827911050 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f869c00cea0 con 0x7f86b0105b50 2024-04-05T21:32:35.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86adffb700 1 -- 172.21.15.121:0/1827911050 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f869c004830 con 0x7f86b0105b50 2024-04-05T21:32:35.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86adffb700 1 -- 172.21.15.121:0/1827911050 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f869c005500 con 0x7f86b0105b50 2024-04-05T21:32:35.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86b70d2700 1 -- 172.21.15.121:0/1827911050 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b01a2910 con 0x7f86b0105b50 2024-04-05T21:32:35.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.102+0000 7f86b70d2700 1 -- 172.21.15.121:0/1827911050 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b01a2d20 con 0x7f86b0105b50 2024-04-05T21:32:35.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.103+0000 7f86b70d2700 1 -- 172.21.15.121:0/1827911050 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b0109140 con 0x7f86b0105b50 2024-04-05T21:32:35.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.104+0000 7f86adffb700 1 -- 172.21.15.121:0/1827911050 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f869c0049a0 con 0x7f86b0105b50 2024-04-05T21:32:35.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.104+0000 7f86adffb700 1 --2- 172.21.15.121:0/1827911050 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f86a006cce0 0x7f86a006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:35.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.104+0000 7f86adffb700 1 -- 172.21.15.121:0/1827911050 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f869c08c120 con 0x7f86b0105b50 2024-04-05T21:32:35.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.104+0000 7f86b4e6e700 1 --2- 172.21.15.121:0/1827911050 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f86a006cce0 0x7f86a006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:35.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.104+0000 7f86b4e6e700 1 --2- 172.21.15.121:0/1827911050 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f86a006cce0 0x7f86a006f1a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f86a4006010 tx=0x7f86a400b540 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:35.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.108+0000 7f86adffb700 1 -- 172.21.15.121:0/1827911050 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f869c0562c0 con 0x7f86b0105b50 2024-04-05T21:32:35.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.348+0000 7f86b70d2700 1 -- 172.21.15.121:0/1827911050 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f86b004ea90 con 0x7f86b0105b50 2024-04-05T21:32:35.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.349+0000 7f86adffb700 1 -- 172.21.15.121:0/1827911050 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f869c0598e0 con 0x7f86b0105b50 2024-04-05T21:32:35.350 INFO:teuthology.orchestra.run.smithi121.stdout:e21 2024-04-05T21:32:35.350 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:32:35.350 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:32:35.350 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:32:35.350 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:32:35.350 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:32:35.350 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:32:35.350 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 21 2024-04-05T21:32:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T21:32:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:32:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:30:52.246236+0000 2024-04-05T21:32:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:32:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:32:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:32:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:32:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:32:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:32:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:32:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:32:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 46 2024-04-05T21:32:35.351 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:up {0=14476,1=24255} 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:failed 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{0:14476} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:32:35.352 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{0:14488} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/4279084130,v1:172.21.15.121:6827/4279084130] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:32:35.353 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{1:24255} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:32:35.353 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{1:24265} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:32:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:32:35.353 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:32:35.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.351+0000 7f86b70d2700 1 -- 172.21.15.121:0/1827911050 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f86a006cce0 msgr2=0x7f86a006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:35.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.351+0000 7f86b70d2700 1 --2- 172.21.15.121:0/1827911050 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f86a006cce0 0x7f86a006f1a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f86a4006010 tx=0x7f86a400b540 comp rx=0 tx=0).stop 2024-04-05T21:32:35.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.351+0000 7f86b70d2700 1 -- 172.21.15.121:0/1827911050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86b0105b50 msgr2=0x7f86b0100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:35.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.351+0000 7f86b70d2700 1 --2- 172.21.15.121:0/1827911050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86b0105b50 0x7f86b0100690 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f869c00c8a0 tx=0x7f869c00cc60 comp rx=0 tx=0).stop 2024-04-05T21:32:35.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.351+0000 7f86b70d2700 1 -- 172.21.15.121:0/1827911050 shutdown_connections 2024-04-05T21:32:35.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.351+0000 7f86b70d2700 1 --2- 172.21.15.121:0/1827911050 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f86a006cce0 0x7f86a006f1a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.351+0000 7f86b70d2700 1 --2- 172.21.15.121:0/1827911050 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86b00fef70 0x7f86b0100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.351+0000 7f86b70d2700 1 --2- 172.21.15.121:0/1827911050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86b0105b50 0x7f86b0100690 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.351+0000 7f86b70d2700 1 -- 172.21.15.121:0/1827911050 >> 172.21.15.121:0/1827911050 conn(0x7f86b0075e30 msgr2=0x7f86b0104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:35.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.351+0000 7f86b70d2700 1 -- 172.21.15.121:0/1827911050 shutdown_connections 2024-04-05T21:32:35.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.351+0000 7f86b70d2700 1 -- 172.21.15.121:0/1827911050 wait complete. 2024-04-05T21:32:35.355 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 21 2024-04-05T21:32:35.427 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:35 smithi121 ceph-mon[145859]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:35.428 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:35 smithi121 ceph-mon[145859]: pgmap v165: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:32:35.428 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:35 smithi121 ceph-mon[145859]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:35.428 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:35 smithi121 ceph-mon[145859]: from='client.34200 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:35.428 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:35 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3598666830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:35.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.457+0000 7fb523533700 1 -- 172.21.15.121:0/4217394935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb51c069020 msgr2=0x7fb51c105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:35.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.457+0000 7fb523533700 1 --2- 172.21.15.121:0/4217394935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb51c069020 0x7fb51c105790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb50c009910 tx=0x7fb50c009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:35.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.457+0000 7fb523533700 1 -- 172.21.15.121:0/4217394935 shutdown_connections 2024-04-05T21:32:35.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.457+0000 7fb523533700 1 --2- 172.21.15.121:0/4217394935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb51c069020 0x7fb51c105790 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.457+0000 7fb523533700 1 --2- 172.21.15.121:0/4217394935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb51c068670 0x7fb51c068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.457+0000 7fb523533700 1 -- 172.21.15.121:0/4217394935 >> 172.21.15.121:0/4217394935 conn(0x7fb51c0755a0 msgr2=0x7fb51c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:35.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.457+0000 7fb523533700 1 -- 172.21.15.121:0/4217394935 shutdown_connections 2024-04-05T21:32:35.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.457+0000 7fb523533700 1 -- 172.21.15.121:0/4217394935 wait complete. 2024-04-05T21:32:35.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.458+0000 7fb523533700 1 Processor -- start 2024-04-05T21:32:35.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.458+0000 7fb523533700 1 -- start start 2024-04-05T21:32:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.458+0000 7fb523533700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb51c068670 0x7fb51c196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.458+0000 7fb523533700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb51c069020 0x7fb51c196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.458+0000 7fb523533700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c197290 con 0x7fb51c068670 2024-04-05T21:32:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.458+0000 7fb523533700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c19b020 con 0x7fb51c069020 2024-04-05T21:32:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.458+0000 7fb5212cf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb51c068670 0x7fb51c196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.458+0000 7fb5212cf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb51c068670 0x7fb51c196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40644/0 (socket says 172.21.15.121:40644) 2024-04-05T21:32:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.458+0000 7fb5212cf700 1 -- 172.21.15.121:0/3520871677 learned_addr learned my addr 172.21.15.121:0/3520871677 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.459+0000 7fb520ace700 1 --2- 172.21.15.121:0/3520871677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb51c069020 0x7fb51c196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.459+0000 7fb5212cf700 1 -- 172.21.15.121:0/3520871677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb51c069020 msgr2=0x7fb51c196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:35.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.459+0000 7fb5212cf700 1 --2- 172.21.15.121:0/3520871677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb51c069020 0x7fb51c196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.459+0000 7fb5212cf700 1 -- 172.21.15.121:0/3520871677 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50c0095f0 con 0x7fb51c068670 2024-04-05T21:32:35.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.459+0000 7fb5212cf700 1 --2- 172.21.15.121:0/3520871677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb51c068670 0x7fb51c196670 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb51800d900 tx=0x7fb51800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:35.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.459+0000 7fb5127fc700 1 -- 172.21.15.121:0/3520871677 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb518004a00 con 0x7fb51c068670 2024-04-05T21:32:35.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.459+0000 7fb5127fc700 1 -- 172.21.15.121:0/3520871677 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb518005020 con 0x7fb51c068670 2024-04-05T21:32:35.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.459+0000 7fb5127fc700 1 -- 172.21.15.121:0/3520871677 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb518009d00 con 0x7fb51c068670 2024-04-05T21:32:35.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.459+0000 7fb523533700 1 -- 172.21.15.121:0/3520871677 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb51c19b300 con 0x7fb51c068670 2024-04-05T21:32:35.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.459+0000 7fb523533700 1 -- 172.21.15.121:0/3520871677 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb51c19b850 con 0x7fb51c068670 2024-04-05T21:32:35.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.460+0000 7fb523533700 1 -- 172.21.15.121:0/3520871677 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb51c1090a0 con 0x7fb51c068670 2024-04-05T21:32:35.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.460+0000 7fb5127fc700 1 -- 172.21.15.121:0/3520871677 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb518005190 con 0x7fb51c068670 2024-04-05T21:32:35.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.461+0000 7fb5127fc700 1 --2- 172.21.15.121:0/3520871677 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb50806cce0 0x7fb50806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:35.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.461+0000 7fb5127fc700 1 -- 172.21.15.121:0/3520871677 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fb51808c650 con 0x7fb51c068670 2024-04-05T21:32:35.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.461+0000 7fb520ace700 1 --2- 172.21.15.121:0/3520871677 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb50806cce0 0x7fb50806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:35.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.461+0000 7fb520ace700 1 --2- 172.21.15.121:0/3520871677 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb50806cce0 0x7fb50806f1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00b5c0 tx=0x7fb50c005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:35.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.465+0000 7fb5127fc700 1 -- 172.21.15.121:0/3520871677 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb5180567f0 con 0x7fb51c068670 2024-04-05T21:32:35.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:35 smithi188 ceph-mon[107793]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:35.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:35 smithi188 ceph-mon[107793]: pgmap v165: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:32:35.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:35 smithi188 ceph-mon[107793]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:35.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:35 smithi188 ceph-mon[107793]: from='client.34200 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:35.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:35 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3598666830' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:35.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.690+0000 7fb523533700 1 -- 172.21.15.121:0/3520871677 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb51c19bc00 con 0x7fb50806cce0 2024-04-05T21:32:35.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.691+0000 7fb5127fc700 1 -- 172.21.15.121:0/3520871677 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fb51c19bc00 con 0x7fb50806cce0 2024-04-05T21:32:35.692 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:32:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:32:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:32:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:32:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:32:35.692 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:32:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:32:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:32:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:32:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "8/23 daemons upgraded", 2024-04-05T21:32:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading osd daemons", 2024-04-05T21:32:35.693 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:32:35.693 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:32:35.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.693+0000 7fb523533700 1 -- 172.21.15.121:0/3520871677 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb50806cce0 msgr2=0x7fb50806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:35.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.693+0000 7fb523533700 1 --2- 172.21.15.121:0/3520871677 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb50806cce0 0x7fb50806f1a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00b5c0 tx=0x7fb50c005fd0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.693+0000 7fb523533700 1 -- 172.21.15.121:0/3520871677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb51c068670 msgr2=0x7fb51c196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:35.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.693+0000 7fb523533700 1 --2- 172.21.15.121:0/3520871677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb51c068670 0x7fb51c196670 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb51800d900 tx=0x7fb51800dcc0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.693+0000 7fb523533700 1 -- 172.21.15.121:0/3520871677 shutdown_connections 2024-04-05T21:32:35.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.693+0000 7fb523533700 1 --2- 172.21.15.121:0/3520871677 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb50806cce0 0x7fb50806f1a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.693+0000 7fb523533700 1 --2- 172.21.15.121:0/3520871677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb51c068670 0x7fb51c196670 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.693+0000 7fb523533700 1 --2- 172.21.15.121:0/3520871677 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb51c069020 0x7fb51c196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.693+0000 7fb523533700 1 -- 172.21.15.121:0/3520871677 >> 172.21.15.121:0/3520871677 conn(0x7fb51c0755a0 msgr2=0x7fb51c105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:35.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.694+0000 7fb523533700 1 -- 172.21.15.121:0/3520871677 shutdown_connections 2024-04-05T21:32:35.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.694+0000 7fb523533700 1 -- 172.21.15.121:0/3520871677 wait complete. 2024-04-05T21:32:35.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.809+0000 7f671fe91700 1 -- 172.21.15.121:0/1528667472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67180ffa60 msgr2=0x7f67180ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:35.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.809+0000 7f671fe91700 1 --2- 172.21.15.121:0/1528667472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67180ffa60 0x7f67180ffee0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6708009910 tx=0x7f6708009c20 comp rx=0 tx=0).stop 2024-04-05T21:32:35.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.810+0000 7f671fe91700 1 -- 172.21.15.121:0/1528667472 shutdown_connections 2024-04-05T21:32:35.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.810+0000 7f671fe91700 1 --2- 172.21.15.121:0/1528667472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67180ffa60 0x7f67180ffee0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.810+0000 7f671fe91700 1 --2- 172.21.15.121:0/1528667472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6718105a60 0x7f6718105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.810+0000 7f671fe91700 1 -- 172.21.15.121:0/1528667472 >> 172.21.15.121:0/1528667472 conn(0x7f67180fb550 msgr2=0x7f67180fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:35.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.810+0000 7f671fe91700 1 -- 172.21.15.121:0/1528667472 shutdown_connections 2024-04-05T21:32:35.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.810+0000 7f671fe91700 1 -- 172.21.15.121:0/1528667472 wait complete. 2024-04-05T21:32:35.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.810+0000 7f671fe91700 1 Processor -- start 2024-04-05T21:32:35.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.810+0000 7f671fe91700 1 -- start start 2024-04-05T21:32:35.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.810+0000 7f671fe91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67180ffa60 0x7f6718195570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:35.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.810+0000 7f671fe91700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6718105a60 0x7f6718195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:35.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.810+0000 7f671fe91700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6718196190 con 0x7f67180ffa60 2024-04-05T21:32:35.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.810+0000 7f671fe91700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6718199f20 con 0x7f6718105a60 2024-04-05T21:32:35.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.811+0000 7f671dc2d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67180ffa60 0x7f6718195570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:35.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.811+0000 7f671dc2d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67180ffa60 0x7f6718195570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40656/0 (socket says 172.21.15.121:40656) 2024-04-05T21:32:35.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.811+0000 7f671dc2d700 1 -- 172.21.15.121:0/2824903842 learned_addr learned my addr 172.21.15.121:0/2824903842 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:32:35.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.811+0000 7f671d42c700 1 --2- 172.21.15.121:0/2824903842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6718105a60 0x7f6718195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:35.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.811+0000 7f671dc2d700 1 -- 172.21.15.121:0/2824903842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6718105a60 msgr2=0x7f6718195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:35.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.811+0000 7f671dc2d700 1 --2- 172.21.15.121:0/2824903842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6718105a60 0x7f6718195ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:35.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.811+0000 7f671dc2d700 1 -- 172.21.15.121:0/2824903842 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67080095f0 con 0x7f67180ffa60 2024-04-05T21:32:35.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.811+0000 7f671dc2d700 1 --2- 172.21.15.121:0/2824903842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67180ffa60 0x7f6718195570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f671400e920 tx=0x7f671400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:35.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.811+0000 7f670effd700 1 -- 172.21.15.121:0/2824903842 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f671400c9d0 con 0x7f67180ffa60 2024-04-05T21:32:35.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.811+0000 7f670effd700 1 -- 172.21.15.121:0/2824903842 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6714010430 con 0x7f67180ffa60 2024-04-05T21:32:35.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.811+0000 7f670effd700 1 -- 172.21.15.121:0/2824903842 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6714018650 con 0x7f67180ffa60 2024-04-05T21:32:35.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.811+0000 7f671fe91700 1 -- 172.21.15.121:0/2824903842 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671819a200 con 0x7f67180ffa60 2024-04-05T21:32:35.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.811+0000 7f671fe91700 1 -- 172.21.15.121:0/2824903842 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671819a750 con 0x7f67180ffa60 2024-04-05T21:32:35.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.812+0000 7f671fe91700 1 -- 172.21.15.121:0/2824903842 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6718107ff0 con 0x7f67180ffa60 2024-04-05T21:32:35.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.813+0000 7f670effd700 1 -- 172.21.15.121:0/2824903842 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f67140105a0 con 0x7f67180ffa60 2024-04-05T21:32:35.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.813+0000 7f670effd700 1 --2- 172.21.15.121:0/2824903842 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f670406cce0 0x7f670406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:32:35.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.813+0000 7f671d42c700 1 --2- 172.21.15.121:0/2824903842 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f670406cce0 0x7f670406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:32:35.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.813+0000 7f670effd700 1 -- 172.21.15.121:0/2824903842 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f6714014070 con 0x7f67180ffa60 2024-04-05T21:32:35.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.814+0000 7f671d42c700 1 --2- 172.21.15.121:0/2824903842 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f670406cce0 0x7f670406f1a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6718196b90 tx=0x7f6708005940 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:32:35.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:35.818+0000 7f670effd700 1 -- 172.21.15.121:0/2824903842 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6714052ab0 con 0x7f67180ffa60 2024-04-05T21:32:36.025 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:35 smithi121 systemd[1]: Stopping Ceph osd.2 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:32:36.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:36.114+0000 7f671fe91700 1 -- 172.21.15.121:0/2824903842 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f671804ea90 con 0x7f67180ffa60 2024-04-05T21:32:36.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:36.115+0000 7f670effd700 1 -- 172.21.15.121:0/2824903842 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f6714056910 con 0x7f67180ffa60 2024-04-05T21:32:36.116 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_OK 2024-04-05T21:32:36.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:36.117+0000 7f671fe91700 1 -- 172.21.15.121:0/2824903842 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f670406cce0 msgr2=0x7f670406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:36.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:36.117+0000 7f671fe91700 1 --2- 172.21.15.121:0/2824903842 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f670406cce0 0x7f670406f1a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6718196b90 tx=0x7f6708005940 comp rx=0 tx=0).stop 2024-04-05T21:32:36.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:36.117+0000 7f671fe91700 1 -- 172.21.15.121:0/2824903842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67180ffa60 msgr2=0x7f6718195570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:32:36.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:36.117+0000 7f671fe91700 1 --2- 172.21.15.121:0/2824903842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67180ffa60 0x7f6718195570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f671400e920 tx=0x7f671400ec30 comp rx=0 tx=0).stop 2024-04-05T21:32:36.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:36.117+0000 7f671fe91700 1 -- 172.21.15.121:0/2824903842 shutdown_connections 2024-04-05T21:32:36.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:36.117+0000 7f671fe91700 1 --2- 172.21.15.121:0/2824903842 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f670406cce0 0x7f670406f1a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:36.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:36.117+0000 7f671fe91700 1 --2- 172.21.15.121:0/2824903842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67180ffa60 0x7f6718195570 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:36.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:36.117+0000 7f671fe91700 1 --2- 172.21.15.121:0/2824903842 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6718105a60 0x7f6718195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:32:36.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:36.117+0000 7f671fe91700 1 -- 172.21.15.121:0/2824903842 >> 172.21.15.121:0/2824903842 conn(0x7f67180fb550 msgr2=0x7f67180fc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:32:36.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:36.117+0000 7f671fe91700 1 -- 172.21.15.121:0/2824903842 shutdown_connections 2024-04-05T21:32:36.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:32:36.117+0000 7f671fe91700 1 -- 172.21.15.121:0/2824903842 wait complete. 2024-04-05T21:32:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:36 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1827911050' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:32:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:36 smithi121 ceph-mon[145859]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:36 smithi121 ceph-mon[145859]: osd.2 marked itself down and dead 2024-04-05T21:32:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:36 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2824903842' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:32:36.490 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:36 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-2[83354]: 2024-04-05T21:32:36.023+0000 7f47f6bc3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T21:32:36.490 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:36 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-2[83354]: 2024-04-05T21:32:36.023+0000 7f47f6bc3700 -1 osd.2 58 *** Got signal Terminated *** 2024-04-05T21:32:36.490 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:36 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-2[83354]: 2024-04-05T21:32:36.023+0000 7f47f6bc3700 -1 osd.2 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T21:32:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:36 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1827911050' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:32:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:36 smithi188 ceph-mon[107793]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:32:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:36 smithi188 ceph-mon[107793]: osd.2 marked itself down and dead 2024-04-05T21:32:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:36 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2824903842' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:32:37.239 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:36 smithi121 bash[171220]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-2 2024-04-05T21:32:37.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:37 smithi188 ceph-mon[107793]: pgmap v166: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:32:37.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:37 smithi188 ceph-mon[107793]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T21:32:37.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:37 smithi188 ceph-mon[107793]: osdmap e59: 6 total, 5 up, 6 in 2024-04-05T21:32:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:37 smithi121 ceph-mon[145859]: pgmap v166: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:32:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:37 smithi121 ceph-mon[145859]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T21:32:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:37 smithi121 ceph-mon[145859]: osdmap e59: 6 total, 5 up, 6 in 2024-04-05T21:32:38.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:38 smithi188 ceph-mon[107793]: osdmap e60: 6 total, 5 up, 6 in 2024-04-05T21:32:38.610 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:38 smithi121 ceph-mon[145859]: osdmap e60: 6 total, 5 up, 6 in 2024-04-05T21:32:39.489 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:39 smithi121 systemd[1]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.2.service: Succeeded. 2024-04-05T21:32:39.489 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:39 smithi121 systemd[1]: Stopped Ceph osd.2 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:32:39.489 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:39 smithi121 systemd[1]: Starting Ceph osd.2 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:32:39.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:39 smithi121 ceph-mon[145859]: pgmap v169: 65 pgs: 7 peering, 5 stale+active+clean, 53 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:32:39.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:39 smithi121 ceph-mon[145859]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-04-05T21:32:39.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:39 smithi188 ceph-mon[107793]: pgmap v169: 65 pgs: 7 peering, 5 stale+active+clean, 53 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:32:39.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:39 smithi188 ceph-mon[107793]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-04-05T21:32:40.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:40 smithi121 ceph-mon[145859]: Health check failed: Degraded data redundancy: 11/333 objects degraded (3.303%), 4 pgs degraded (PG_DEGRADED) 2024-04-05T21:32:40.489 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:40 smithi121 bash[171495]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-05T21:32:40.489 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:40 smithi121 bash[171495]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-05T21:32:40.489 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:40 smithi121 bash[171495]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-05T21:32:40.489 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:40 smithi121 bash[171495]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-05T21:32:40.490 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:40 smithi121 bash[171495]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-05T21:32:40.490 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:40 smithi121 bash[171495]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-05T21:32:40.490 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:40 smithi121 bash[171495]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-05T21:32:40.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:40 smithi188 ceph-mon[107793]: Health check failed: Degraded data redundancy: 11/333 objects degraded (3.303%), 4 pgs degraded (PG_DEGRADED) 2024-04-05T21:32:41.711 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:41 smithi121 ceph-mon[145859]: pgmap v170: 65 pgs: 4 active+undersized+degraded, 7 peering, 2 stale+active+clean, 4 active+undersized, 48 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 11/333 objects degraded (3.303%) 2024-04-05T21:32:41.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:41 smithi188 ceph-mon[107793]: pgmap v170: 65 pgs: 4 active+undersized+degraded, 7 peering, 2 stale+active+clean, 4 active+undersized, 48 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 11/333 objects degraded (3.303%) 2024-04-05T21:32:41.989 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:41 smithi121 ceph-osd[171817]: -- 172.21.15.121:0/897660622 <== mon.0 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a6f8c6b1e0 con 0x55a6f8c3b000 2024-04-05T21:32:41.989 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:41 smithi121 bash[171779]: 6b8b7f7862d3cb614d4bb951905f41889aaaf45078045a0eb8c0092b262d76fd 2024-04-05T21:32:41.989 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:41 smithi121 systemd[1]: Started Ceph osd.2 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:32:42.489 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:42 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-2[171794]: 2024-04-05T21:32:42.225+0000 7f6649ca77c0 -1 Falling back to public interface 2024-04-05T21:32:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:42 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:42 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:42 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:32:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:42 smithi121 ceph-mon[145859]: pgmap v171: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 38/333 objects degraded (11.411%) 2024-04-05T21:32:43.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:42 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:43.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:42 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:43.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:42 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:32:43.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:42 smithi188 ceph-mon[107793]: pgmap v171: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 38/333 objects degraded (11.411%) 2024-04-05T21:32:44.230 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:43 smithi121 ceph-mon[145859]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-04-05T21:32:44.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:43 smithi188 ceph-mon[107793]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-04-05T21:32:45.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:44 smithi121 ceph-mon[145859]: pgmap v172: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 38/333 objects degraded (11.411%) 2024-04-05T21:32:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:44 smithi188 ceph-mon[107793]: pgmap v172: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 38/333 objects degraded (11.411%) 2024-04-05T21:32:46.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:46 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:46.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:46 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:32:46.489 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:46 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-2[171794]: 2024-04-05T21:32:46.148+0000 7f6649ca77c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-05T21:32:46.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:46 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:46.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:46 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:32:46.799 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:46 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-2[171794]: 2024-04-05T21:32:46.533+0000 7f6649ca77c0 -1 osd.2 58 log_to_monitors true 2024-04-05T21:32:47.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:47 smithi121 ceph-mon[145859]: pgmap v173: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 38/333 objects degraded (11.411%) 2024-04-05T21:32:47.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:47 smithi121 ceph-mon[145859]: from='osd.2 [v2:172.21.15.121:6818/2092662684,v1:172.21.15.121:6819/2092662684]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T21:32:47.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:47 smithi121 ceph-mon[145859]: Health check update: Degraded data redundancy: 38/333 objects degraded (11.411%), 12 pgs degraded (PG_DEGRADED) 2024-04-05T21:32:47.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:47 smithi188 ceph-mon[107793]: pgmap v173: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 38/333 objects degraded (11.411%) 2024-04-05T21:32:47.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:47 smithi188 ceph-mon[107793]: from='osd.2 [v2:172.21.15.121:6818/2092662684,v1:172.21.15.121:6819/2092662684]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-05T21:32:47.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:47 smithi188 ceph-mon[107793]: Health check update: Degraded data redundancy: 38/333 objects degraded (11.411%), 12 pgs degraded (PG_DEGRADED) 2024-04-05T21:32:48.489 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 21:32:48 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-2[171794]: 2024-04-05T21:32:48.132+0000 7f66370f9700 -1 osd.2 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T21:32:48.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:48 smithi121 ceph-mon[145859]: from='osd.2 [v2:172.21.15.121:6818/2092662684,v1:172.21.15.121:6819/2092662684]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-05T21:32:48.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:48 smithi121 ceph-mon[145859]: osdmap e61: 6 total, 5 up, 6 in 2024-04-05T21:32:48.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:48 smithi121 ceph-mon[145859]: from='osd.2 [v2:172.21.15.121:6818/2092662684,v1:172.21.15.121:6819/2092662684]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-05T21:32:48.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:48 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:48.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:48 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:48.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:48 smithi121 ceph-mon[145859]: from='osd.2 [v2:172.21.15.121:6818/2092662684,v1:172.21.15.121:6819/2092662684]' entity='osd.2' 2024-04-05T21:32:48.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:48 smithi188 ceph-mon[107793]: from='osd.2 [v2:172.21.15.121:6818/2092662684,v1:172.21.15.121:6819/2092662684]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-05T21:32:48.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:48 smithi188 ceph-mon[107793]: osdmap e61: 6 total, 5 up, 6 in 2024-04-05T21:32:48.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:48 smithi188 ceph-mon[107793]: from='osd.2 [v2:172.21.15.121:6818/2092662684,v1:172.21.15.121:6819/2092662684]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-04-05T21:32:48.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:48 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:48.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:48 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:48.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:48 smithi188 ceph-mon[107793]: from='osd.2 [v2:172.21.15.121:6818/2092662684,v1:172.21.15.121:6819/2092662684]' entity='osd.2' 2024-04-05T21:32:49.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:49 smithi121 ceph-mon[145859]: pgmap v175: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 38/333 objects degraded (11.411%) 2024-04-05T21:32:49.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:49 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:49.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:49 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:49.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:49 smithi121 ceph-mon[145859]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T21:32:49.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:49 smithi121 ceph-mon[145859]: osd.2 [v2:172.21.15.121:6818/2092662684,v1:172.21.15.121:6819/2092662684] boot 2024-04-05T21:32:49.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:49 smithi121 ceph-mon[145859]: osdmap e62: 6 total, 6 up, 6 in 2024-04-05T21:32:49.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:49 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:32:49.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:49 smithi188 ceph-mon[107793]: pgmap v175: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 38/333 objects degraded (11.411%) 2024-04-05T21:32:49.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:49 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:49.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:49 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:49.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:49 smithi188 ceph-mon[107793]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T21:32:49.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:49 smithi188 ceph-mon[107793]: osd.2 [v2:172.21.15.121:6818/2092662684,v1:172.21.15.121:6819/2092662684] boot 2024-04-05T21:32:49.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:49 smithi188 ceph-mon[107793]: osdmap e62: 6 total, 6 up, 6 in 2024-04-05T21:32:49.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:49 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-05T21:32:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:51 smithi121 ceph-mon[145859]: osdmap e63: 6 total, 6 up, 6 in 2024-04-05T21:32:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:51 smithi121 ceph-mon[145859]: pgmap v178: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 38/333 objects degraded (11.411%) 2024-04-05T21:32:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:51 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:51.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:51 smithi188 ceph-mon[107793]: osdmap e63: 6 total, 6 up, 6 in 2024-04-05T21:32:51.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:51 smithi188 ceph-mon[107793]: pgmap v178: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 38/333 objects degraded (11.411%) 2024-04-05T21:32:51.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:51 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:52.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:52 smithi121 ceph-mon[145859]: Health check update: Degraded data redundancy: 21/333 objects degraded (6.306%), 6 pgs degraded (PG_DEGRADED) 2024-04-05T21:32:52.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:52 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:52.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:52 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:52.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:52 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:32:52.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:52 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:32:52.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:52 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:52.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:52 smithi188 ceph-mon[107793]: Health check update: Degraded data redundancy: 21/333 objects degraded (6.306%), 6 pgs degraded (PG_DEGRADED) 2024-04-05T21:32:52.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:52 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:52.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:52 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:52.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:52 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:32:52.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:52 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:32:52.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:52 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:32:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:53 smithi121 ceph-mon[145859]: pgmap v179: 65 pgs: 6 active+undersized+degraded, 8 active+undersized, 51 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 21/333 objects degraded (6.306%) 2024-04-05T21:32:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:53 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:32:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:53 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:53 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:53 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:53 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T21:32:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:53 smithi121 ceph-mon[145859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T21:32:53.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:53 smithi121 ceph-mon[145859]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-05T21:32:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:53 smithi188 ceph-mon[107793]: pgmap v179: 65 pgs: 6 active+undersized+degraded, 8 active+undersized, 51 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 21/333 objects degraded (6.306%) 2024-04-05T21:32:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:53 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:32:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:53 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:53 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:53 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:32:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:53 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T21:32:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:53 smithi188 ceph-mon[107793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T21:32:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:53 smithi188 ceph-mon[107793]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-05T21:32:54.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:54 smithi121 ceph-mon[145859]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/333 objects degraded (6.306%), 6 pgs degraded) 2024-04-05T21:32:54.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:54 smithi121 ceph-mon[145859]: Cluster is now healthy 2024-04-05T21:32:54.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:54 smithi188 ceph-mon[107793]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/333 objects degraded (6.306%), 6 pgs degraded) 2024-04-05T21:32:54.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:54 smithi188 ceph-mon[107793]: Cluster is now healthy 2024-04-05T21:32:55.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:55 smithi121 ceph-mon[145859]: pgmap v180: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 op/s 2024-04-05T21:32:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:55 smithi188 ceph-mon[107793]: pgmap v180: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 op/s 2024-04-05T21:32:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:57 smithi121 ceph-mon[145859]: pgmap v181: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-04-05T21:32:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:57 smithi188 ceph-mon[107793]: pgmap v181: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-04-05T21:32:59.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:32:59 smithi121 ceph-mon[145859]: pgmap v182: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-04-05T21:32:59.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:32:59 smithi188 ceph-mon[107793]: pgmap v182: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-04-05T21:33:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:01 smithi121 ceph-mon[145859]: pgmap v183: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-05T21:33:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:33:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:01 smithi188 ceph-mon[107793]: pgmap v183: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 2 op/s 2024-04-05T21:33:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:33:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:03.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:03 smithi121 ceph-mon[145859]: pgmap v184: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-05T21:33:03.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:03 smithi188 ceph-mon[107793]: pgmap v184: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-04-05T21:33:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:05 smithi121 ceph-mon[145859]: pgmap v185: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-05T21:33:05.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:05 smithi188 ceph-mon[107793]: pgmap v185: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 2 op/s 2024-04-05T21:33:06.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.237+0000 7fd7f1237700 1 -- 172.21.15.121:0/730416595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ec105a50 msgr2=0x7fd7ec105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:06.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.237+0000 7fd7f1237700 1 --2- 172.21.15.121:0/730416595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ec105a50 0x7fd7ec105e30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0009910 tx=0x7fd7e0009c20 comp rx=0 tx=0).stop 2024-04-05T21:33:06.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.238+0000 7fd7f1237700 1 -- 172.21.15.121:0/730416595 shutdown_connections 2024-04-05T21:33:06.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.238+0000 7fd7f1237700 1 --2- 172.21.15.121:0/730416595 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7ec0ffa50 0x7fd7ec0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.238+0000 7fd7f1237700 1 --2- 172.21.15.121:0/730416595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ec105a50 0x7fd7ec105e30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.238+0000 7fd7f1237700 1 -- 172.21.15.121:0/730416595 >> 172.21.15.121:0/730416595 conn(0x7fd7ec0fb540 msgr2=0x7fd7ec0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:06.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.238+0000 7fd7f1237700 1 -- 172.21.15.121:0/730416595 shutdown_connections 2024-04-05T21:33:06.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.238+0000 7fd7f1237700 1 -- 172.21.15.121:0/730416595 wait complete. 2024-04-05T21:33:06.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.238+0000 7fd7f1237700 1 Processor -- start 2024-04-05T21:33:06.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.239+0000 7fd7f1237700 1 -- start start 2024-04-05T21:33:06.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.239+0000 7fd7f1237700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ec0ffa50 0x7fd7ec196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:06.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.239+0000 7fd7f1237700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7ec105a50 0x7fd7ec196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:06.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.239+0000 7fd7f1237700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec197290 con 0x7fd7ec0ffa50 2024-04-05T21:33:06.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.239+0000 7fd7f1237700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec19b020 con 0x7fd7ec105a50 2024-04-05T21:33:06.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.239+0000 7fd7ead9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ec0ffa50 0x7fd7ec196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:06.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.239+0000 7fd7ead9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ec0ffa50 0x7fd7ec196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59608/0 (socket says 172.21.15.121:59608) 2024-04-05T21:33:06.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.239+0000 7fd7ead9d700 1 -- 172.21.15.121:0/194054044 learned_addr learned my addr 172.21.15.121:0/194054044 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:06.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.239+0000 7fd7ea59c700 1 --2- 172.21.15.121:0/194054044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7ec105a50 0x7fd7ec196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:06.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.239+0000 7fd7ead9d700 1 -- 172.21.15.121:0/194054044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7ec105a50 msgr2=0x7fd7ec196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:06.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.239+0000 7fd7ead9d700 1 --2- 172.21.15.121:0/194054044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7ec105a50 0x7fd7ec196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.239+0000 7fd7ead9d700 1 -- 172.21.15.121:0/194054044 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e00095f0 con 0x7fd7ec0ffa50 2024-04-05T21:33:06.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.239+0000 7fd7ead9d700 1 --2- 172.21.15.121:0/194054044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ec0ffa50 0x7fd7ec196670 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0000c00 tx=0x7fd7e000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:06.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.240+0000 7fd7d3fff700 1 -- 172.21.15.121:0/194054044 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7e001d070 con 0x7fd7ec0ffa50 2024-04-05T21:33:06.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.240+0000 7fd7d3fff700 1 -- 172.21.15.121:0/194054044 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7e0005dc0 con 0x7fd7ec0ffa50 2024-04-05T21:33:06.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.240+0000 7fd7f1237700 1 -- 172.21.15.121:0/194054044 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ec19b2a0 con 0x7fd7ec0ffa50 2024-04-05T21:33:06.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.240+0000 7fd7d3fff700 1 -- 172.21.15.121:0/194054044 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7e0003ae0 con 0x7fd7ec0ffa50 2024-04-05T21:33:06.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.240+0000 7fd7f1237700 1 -- 172.21.15.121:0/194054044 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ec19b730 con 0x7fd7ec0ffa50 2024-04-05T21:33:06.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.241+0000 7fd7f1237700 1 -- 172.21.15.121:0/194054044 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7ec109060 con 0x7fd7ec0ffa50 2024-04-05T21:33:06.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.241+0000 7fd7d3fff700 1 -- 172.21.15.121:0/194054044 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd7e0003d20 con 0x7fd7ec0ffa50 2024-04-05T21:33:06.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.242+0000 7fd7d3fff700 1 --2- 172.21.15.121:0/194054044 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d406cce0 0x7fd7d406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:06.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.242+0000 7fd7d3fff700 1 -- 172.21.15.121:0/194054044 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fd7e008dc30 con 0x7fd7ec0ffa50 2024-04-05T21:33:06.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.242+0000 7fd7ea59c700 1 --2- 172.21.15.121:0/194054044 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d406cce0 0x7fd7d406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:06.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.242+0000 7fd7ea59c700 1 --2- 172.21.15.121:0/194054044 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d406cce0 0x7fd7d406f1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec197c90 tx=0x7fd7dc009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:06.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.247+0000 7fd7d3fff700 1 -- 172.21.15.121:0/194054044 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd7e0057d50 con 0x7fd7ec0ffa50 2024-04-05T21:33:06.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.459+0000 7fd7f1237700 1 -- 172.21.15.121:0/194054044 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7ec02cce0 con 0x7fd7d406cce0 2024-04-05T21:33:06.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.460+0000 7fd7d3fff700 1 -- 172.21.15.121:0/194054044 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fd7ec02cce0 con 0x7fd7d406cce0 2024-04-05T21:33:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.462+0000 7fd7f1237700 1 -- 172.21.15.121:0/194054044 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d406cce0 msgr2=0x7fd7d406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:06.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.462+0000 7fd7f1237700 1 --2- 172.21.15.121:0/194054044 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d406cce0 0x7fd7d406f1a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec197c90 tx=0x7fd7dc009260 comp rx=0 tx=0).stop 2024-04-05T21:33:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.462+0000 7fd7f1237700 1 -- 172.21.15.121:0/194054044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ec0ffa50 msgr2=0x7fd7ec196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.462+0000 7fd7f1237700 1 --2- 172.21.15.121:0/194054044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ec0ffa50 0x7fd7ec196670 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0000c00 tx=0x7fd7e000be90 comp rx=0 tx=0).stop 2024-04-05T21:33:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.463+0000 7fd7f1237700 1 -- 172.21.15.121:0/194054044 shutdown_connections 2024-04-05T21:33:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.463+0000 7fd7f1237700 1 --2- 172.21.15.121:0/194054044 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d406cce0 0x7fd7d406f1a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.463+0000 7fd7f1237700 1 --2- 172.21.15.121:0/194054044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ec0ffa50 0x7fd7ec196670 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.463+0000 7fd7f1237700 1 --2- 172.21.15.121:0/194054044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7ec105a50 0x7fd7ec196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.463+0000 7fd7f1237700 1 -- 172.21.15.121:0/194054044 >> 172.21.15.121:0/194054044 conn(0x7fd7ec0fb540 msgr2=0x7fd7ec069010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.463+0000 7fd7f1237700 1 -- 172.21.15.121:0/194054044 shutdown_connections 2024-04-05T21:33:06.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.463+0000 7fd7f1237700 1 -- 172.21.15.121:0/194054044 wait complete. 2024-04-05T21:33:06.474 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:33:06.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.567+0000 7fad4be28700 1 -- 172.21.15.121:0/2931662336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad440730b0 msgr2=0x7fad44073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:06.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.567+0000 7fad4be28700 1 --2- 172.21.15.121:0/2931662336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad440730b0 0x7fad44073490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fad3400b0a0 tx=0x7fad3400b3b0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.567+0000 7fad4be28700 1 -- 172.21.15.121:0/2931662336 shutdown_connections 2024-04-05T21:33:06.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.567+0000 7fad4be28700 1 --2- 172.21.15.121:0/2931662336 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad440739d0 0x7fad4410cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.567+0000 7fad4be28700 1 --2- 172.21.15.121:0/2931662336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad440730b0 0x7fad44073490 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.567+0000 7fad4be28700 1 -- 172.21.15.121:0/2931662336 >> 172.21.15.121:0/2931662336 conn(0x7fad440fc630 msgr2=0x7fad440fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:06.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.567+0000 7fad4be28700 1 -- 172.21.15.121:0/2931662336 shutdown_connections 2024-04-05T21:33:06.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.567+0000 7fad4be28700 1 -- 172.21.15.121:0/2931662336 wait complete. 2024-04-05T21:33:06.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.568+0000 7fad4be28700 1 Processor -- start 2024-04-05T21:33:06.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.568+0000 7fad4be28700 1 -- start start 2024-04-05T21:33:06.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad4be28700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad440730b0 0x7fad4419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:06.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad4be28700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad440739d0 0x7fad4419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:06.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad4be28700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4419f250 con 0x7fad440730b0 2024-04-05T21:33:06.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad4be28700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad44198750 con 0x7fad440739d0 2024-04-05T21:33:06.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad49bc4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad440730b0 0x7fad4419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:06.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad49bc4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad440730b0 0x7fad4419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59622/0 (socket says 172.21.15.121:59622) 2024-04-05T21:33:06.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad49bc4700 1 -- 172.21.15.121:0/3111946718 learned_addr learned my addr 172.21.15.121:0/3111946718 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:06.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad493c3700 1 --2- 172.21.15.121:0/3111946718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad440739d0 0x7fad4419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:06.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad49bc4700 1 -- 172.21.15.121:0/3111946718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad440739d0 msgr2=0x7fad4419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:06.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad49bc4700 1 --2- 172.21.15.121:0/3111946718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad440739d0 0x7fad4419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad49bc4700 1 -- 172.21.15.121:0/3111946718 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad34009d00 con 0x7fad440730b0 2024-04-05T21:33:06.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad49bc4700 1 --2- 172.21.15.121:0/3111946718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad440730b0 0x7fad4419e680 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fad34015040 tx=0x7fad34003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:06.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad42ffd700 1 -- 172.21.15.121:0/3111946718 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad3400e040 con 0x7fad440730b0 2024-04-05T21:33:06.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad42ffd700 1 -- 172.21.15.121:0/3111946718 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad34007ce0 con 0x7fad440730b0 2024-04-05T21:33:06.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad4be28700 1 -- 172.21.15.121:0/3111946718 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad441989d0 con 0x7fad440730b0 2024-04-05T21:33:06.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad42ffd700 1 -- 172.21.15.121:0/3111946718 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad3401d7d0 con 0x7fad440730b0 2024-04-05T21:33:06.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.569+0000 7fad4be28700 1 -- 172.21.15.121:0/3111946718 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad44198ec0 con 0x7fad440730b0 2024-04-05T21:33:06.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.570+0000 7fad4be28700 1 -- 172.21.15.121:0/3111946718 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4410a5c0 con 0x7fad440730b0 2024-04-05T21:33:06.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.571+0000 7fad42ffd700 1 -- 172.21.15.121:0/3111946718 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fad340046e0 con 0x7fad440730b0 2024-04-05T21:33:06.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.571+0000 7fad42ffd700 1 --2- 172.21.15.121:0/3111946718 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fad2c06cce0 0x7fad2c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:06.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.571+0000 7fad42ffd700 1 -- 172.21.15.121:0/3111946718 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fad3408dca0 con 0x7fad440730b0 2024-04-05T21:33:06.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.571+0000 7fad493c3700 1 --2- 172.21.15.121:0/3111946718 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fad2c06cce0 0x7fad2c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:06.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.571+0000 7fad493c3700 1 --2- 172.21.15.121:0/3111946718 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fad2c06cce0 0x7fad2c06f1a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fad44199b70 tx=0x7fad3800a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:06.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.576+0000 7fad42ffd700 1 -- 172.21.15.121:0/3111946718 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fad34092070 con 0x7fad440730b0 2024-04-05T21:33:06.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.784+0000 7fad4be28700 1 -- 172.21.15.121:0/3111946718 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fad441990e0 con 0x7fad2c06cce0 2024-04-05T21:33:06.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.786+0000 7fad42ffd700 1 -- 172.21.15.121:0/3111946718 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fad441990e0 con 0x7fad2c06cce0 2024-04-05T21:33:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.788+0000 7fad4be28700 1 -- 172.21.15.121:0/3111946718 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fad2c06cce0 msgr2=0x7fad2c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.788+0000 7fad4be28700 1 --2- 172.21.15.121:0/3111946718 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fad2c06cce0 0x7fad2c06f1a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fad44199b70 tx=0x7fad3800a300 comp rx=0 tx=0).stop 2024-04-05T21:33:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.788+0000 7fad4be28700 1 -- 172.21.15.121:0/3111946718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad440730b0 msgr2=0x7fad4419e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.788+0000 7fad4be28700 1 --2- 172.21.15.121:0/3111946718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad440730b0 0x7fad4419e680 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fad34015040 tx=0x7fad34003c30 comp rx=0 tx=0).stop 2024-04-05T21:33:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.788+0000 7fad4be28700 1 -- 172.21.15.121:0/3111946718 shutdown_connections 2024-04-05T21:33:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.788+0000 7fad4be28700 1 --2- 172.21.15.121:0/3111946718 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fad2c06cce0 0x7fad2c06f1a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.788+0000 7fad4be28700 1 --2- 172.21.15.121:0/3111946718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad440730b0 0x7fad4419e680 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.788+0000 7fad4be28700 1 --2- 172.21.15.121:0/3111946718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad440739d0 0x7fad4419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.788+0000 7fad4be28700 1 -- 172.21.15.121:0/3111946718 >> 172.21.15.121:0/3111946718 conn(0x7fad440fc630 msgr2=0x7fad44107700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:06.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.788+0000 7fad4be28700 1 -- 172.21.15.121:0/3111946718 shutdown_connections 2024-04-05T21:33:06.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.788+0000 7fad4be28700 1 -- 172.21.15.121:0/3111946718 wait complete. 2024-04-05T21:33:06.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.891+0000 7fafdb530700 1 -- 172.21.15.121:0/2946052497 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd4101770 msgr2=0x7fafd41057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:06.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.891+0000 7fafdb530700 1 --2- 172.21.15.121:0/2946052497 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd4101770 0x7fafd41057c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009910 tx=0x7fafd0009c20 comp rx=0 tx=0).stop 2024-04-05T21:33:06.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.891+0000 7fafdb530700 1 -- 172.21.15.121:0/2946052497 shutdown_connections 2024-04-05T21:33:06.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.891+0000 7fafdb530700 1 --2- 172.21.15.121:0/2946052497 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd4101770 0x7fafd41057c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.891+0000 7fafdb530700 1 --2- 172.21.15.121:0/2946052497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafd4100dc0 0x7fafd41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.891+0000 7fafdb530700 1 -- 172.21.15.121:0/2946052497 >> 172.21.15.121:0/2946052497 conn(0x7fafd40fc650 msgr2=0x7fafd40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:06.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.891+0000 7fafdb530700 1 -- 172.21.15.121:0/2946052497 shutdown_connections 2024-04-05T21:33:06.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.891+0000 7fafdb530700 1 -- 172.21.15.121:0/2946052497 wait complete. 2024-04-05T21:33:06.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafdb530700 1 Processor -- start 2024-04-05T21:33:06.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafdb530700 1 -- start start 2024-04-05T21:33:06.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafdb530700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafd4100dc0 0x7fafd41966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:06.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafdb530700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd4101770 0x7fafd4196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:06.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafdb530700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd41972a0 con 0x7fafd4101770 2024-04-05T21:33:06.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafdb530700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd4199fd0 con 0x7fafd4100dc0 2024-04-05T21:33:06.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafd8acb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd4101770 0x7fafd4196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:06.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafd8acb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd4101770 0x7fafd4196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59646/0 (socket says 172.21.15.121:59646) 2024-04-05T21:33:06.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafd8acb700 1 -- 172.21.15.121:0/4241289467 learned_addr learned my addr 172.21.15.121:0/4241289467 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:06.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafd92cc700 1 --2- 172.21.15.121:0/4241289467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafd4100dc0 0x7fafd41966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:06.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafd8acb700 1 -- 172.21.15.121:0/4241289467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafd4100dc0 msgr2=0x7fafd41966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:06.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafd8acb700 1 --2- 172.21.15.121:0/4241289467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafd4100dc0 0x7fafd41966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:06.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafd8acb700 1 -- 172.21.15.121:0/4241289467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd00095f0 con 0x7fafd4101770 2024-04-05T21:33:06.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafd92cc700 1 --2- 172.21.15.121:0/4241289467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafd4100dc0 0x7fafd41966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:33:06.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.892+0000 7fafd8acb700 1 --2- 172.21.15.121:0/4241289467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd4101770 0x7fafd4196c10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fafd0000c00 tx=0x7fafd000bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:06.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.893+0000 7fafca7fc700 1 -- 172.21.15.121:0/4241289467 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafd001d070 con 0x7fafd4101770 2024-04-05T21:33:06.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.893+0000 7fafdb530700 1 -- 172.21.15.121:0/4241289467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd419a250 con 0x7fafd4101770 2024-04-05T21:33:06.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.893+0000 7fafca7fc700 1 -- 172.21.15.121:0/4241289467 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafd0003cf0 con 0x7fafd4101770 2024-04-05T21:33:06.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.893+0000 7fafca7fc700 1 -- 172.21.15.121:0/4241289467 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafd0021410 con 0x7fafd4101770 2024-04-05T21:33:06.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.893+0000 7fafdb530700 1 -- 172.21.15.121:0/4241289467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd419a6e0 con 0x7fafd4101770 2024-04-05T21:33:06.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.893+0000 7fafdb530700 1 -- 172.21.15.121:0/4241289467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd41091c0 con 0x7fafd4101770 2024-04-05T21:33:06.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.894+0000 7fafca7fc700 1 -- 172.21.15.121:0/4241289467 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fafd0003850 con 0x7fafd4101770 2024-04-05T21:33:06.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.894+0000 7fafca7fc700 1 --2- 172.21.15.121:0/4241289467 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fafc006cce0 0x7fafc006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:06.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.895+0000 7fafd92cc700 1 --2- 172.21.15.121:0/4241289467 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fafc006cce0 0x7fafc006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:06.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.895+0000 7fafca7fc700 1 -- 172.21.15.121:0/4241289467 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fafd008d760 con 0x7fafd4101770 2024-04-05T21:33:06.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.895+0000 7fafd92cc700 1 --2- 172.21.15.121:0/4241289467 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fafc006cce0 0x7fafc006f1a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fafc4009cb0 tx=0x7fafc4009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:06.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:06.899+0000 7fafca7fc700 1 -- 172.21.15.121:0/4241289467 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fafd0057880 con 0x7fafd4101770 2024-04-05T21:33:07.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.098+0000 7fafdb530700 1 -- 172.21.15.121:0/4241289467 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fafd402cce0 con 0x7fafc006cce0 2024-04-05T21:33:07.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.108+0000 7fafca7fc700 1 -- 172.21.15.121:0/4241289467 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fafd402cce0 con 0x7fafc006cce0 2024-04-05T21:33:07.109 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:33:07.110 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (3m) 19s ago 15m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:33:07.110 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (15m) 19s ago 15m 8284k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:33:07.110 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (14m) 117s ago 14m 8044k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:33:07.110 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (2m) 19s ago 15m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:33:07.110 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (2m) 117s ago 14m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:33:07.110 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (3m) 19s ago 15m 82.7M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:33:07.110 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (9m) 19s ago 9m 64.5M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:33:07.110 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (9m) 19s ago 9m 133M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:33:07.110 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (9m) 117s ago 9m 110M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:33:07.110 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (9m) 117s ago 9m 62.2M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:33:07.110 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (5m) 19s ago 16m 507M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:33:07.110 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (5m) 117s ago 14m 450M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:33:07.111 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (2m) 19s ago 17m 43.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:33:07.111 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (2m) 117s ago 14m 27.9M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:33:07.111 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (4m) 19s ago 15m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:33:07.111 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (4m) 117s ago 14m 16.4M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:33:07.111 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (99s) 19s ago 13m 88.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:33:07.111 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (62s) 19s ago 12m 99.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:33:07.111 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (25s) 19s ago 12m 11.2M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:33:07.111 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (11m) 117s ago 11m 288M 2251M 18.2.1 5be31c24972a ce847cf647f2 2024-04-05T21:33:07.111 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (11m) 117s ago 11m 246M 2251M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:33:07.111 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (10m) 117s ago 10m 224M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:33:07.111 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (4m) 19s ago 14m 65.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:33:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.111+0000 7fafdb530700 1 -- 172.21.15.121:0/4241289467 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fafc006cce0 msgr2=0x7fafc006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.111+0000 7fafdb530700 1 --2- 172.21.15.121:0/4241289467 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fafc006cce0 0x7fafc006f1a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fafc4009cb0 tx=0x7fafc4009260 comp rx=0 tx=0).stop 2024-04-05T21:33:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.111+0000 7fafdb530700 1 -- 172.21.15.121:0/4241289467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd4101770 msgr2=0x7fafd4196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:07.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.111+0000 7fafdb530700 1 --2- 172.21.15.121:0/4241289467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd4101770 0x7fafd4196c10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fafd0000c00 tx=0x7fafd000bb20 comp rx=0 tx=0).stop 2024-04-05T21:33:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.111+0000 7fafdb530700 1 -- 172.21.15.121:0/4241289467 shutdown_connections 2024-04-05T21:33:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.111+0000 7fafdb530700 1 --2- 172.21.15.121:0/4241289467 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fafc006cce0 0x7fafc006f1a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.111+0000 7fafdb530700 1 --2- 172.21.15.121:0/4241289467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafd4100dc0 0x7fafd41966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.111+0000 7fafdb530700 1 --2- 172.21.15.121:0/4241289467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd4101770 0x7fafd4196c10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.111+0000 7fafdb530700 1 -- 172.21.15.121:0/4241289467 >> 172.21.15.121:0/4241289467 conn(0x7fafd40fc650 msgr2=0x7fafd40fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.111+0000 7fafdb530700 1 -- 172.21.15.121:0/4241289467 shutdown_connections 2024-04-05T21:33:07.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.111+0000 7fafdb530700 1 -- 172.21.15.121:0/4241289467 wait complete. 2024-04-05T21:33:07.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.213+0000 7f652f291700 1 -- 172.21.15.121:0/3505586626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6528068670 msgr2=0x7f6528068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:07.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.213+0000 7f652f291700 1 --2- 172.21.15.121:0/3505586626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6528068670 0x7f6528068a50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6524009910 tx=0x7f6524009c20 comp rx=0 tx=0).stop 2024-04-05T21:33:07.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.213+0000 7f652f291700 1 -- 172.21.15.121:0/3505586626 shutdown_connections 2024-04-05T21:33:07.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.213+0000 7f652f291700 1 --2- 172.21.15.121:0/3505586626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6528069020 0x7f6528105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.213+0000 7f652f291700 1 --2- 172.21.15.121:0/3505586626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6528068670 0x7f6528068a50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.213+0000 7f652f291700 1 -- 172.21.15.121:0/3505586626 >> 172.21.15.121:0/3505586626 conn(0x7f65280755a0 msgr2=0x7f65280759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:07.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.213+0000 7f652f291700 1 -- 172.21.15.121:0/3505586626 shutdown_connections 2024-04-05T21:33:07.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.213+0000 7f652f291700 1 -- 172.21.15.121:0/3505586626 wait complete. 2024-04-05T21:33:07.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.214+0000 7f652f291700 1 Processor -- start 2024-04-05T21:33:07.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.214+0000 7f652f291700 1 -- start start 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.214+0000 7f652f291700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6528068670 0x7f6528196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.214+0000 7f652f291700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6528069020 0x7f6528196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.214+0000 7f652f291700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6528197290 con 0x7f6528069020 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.214+0000 7f652f291700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652819afb0 con 0x7f6528068670 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.214+0000 7f652c82c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6528069020 0x7f6528196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.214+0000 7f652c82c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6528069020 0x7f6528196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59652/0 (socket says 172.21.15.121:59652) 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.214+0000 7f652c82c700 1 -- 172.21.15.121:0/1475857309 learned_addr learned my addr 172.21.15.121:0/1475857309 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.214+0000 7f652d02d700 1 --2- 172.21.15.121:0/1475857309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6528068670 0x7f6528196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.215+0000 7f652c82c700 1 -- 172.21.15.121:0/1475857309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6528068670 msgr2=0x7f6528196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.215+0000 7f652c82c700 1 --2- 172.21.15.121:0/1475857309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6528068670 0x7f6528196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.215+0000 7f652c82c700 1 -- 172.21.15.121:0/1475857309 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65240095f0 con 0x7f6528069020 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.215+0000 7f652c82c700 1 --2- 172.21.15.121:0/1475857309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6528069020 0x7f6528196bb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f651800c8f0 tx=0x7f651800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.215+0000 7f651e7fc700 1 -- 172.21.15.121:0/1475857309 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6518007a50 con 0x7f6528069020 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.215+0000 7f651e7fc700 1 -- 172.21.15.121:0/1475857309 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6518007bb0 con 0x7f6528069020 2024-04-05T21:33:07.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.215+0000 7f652f291700 1 -- 172.21.15.121:0/1475857309 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f652819b290 con 0x7f6528069020 2024-04-05T21:33:07.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.215+0000 7f651e7fc700 1 -- 172.21.15.121:0/1475857309 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6518018630 con 0x7f6528069020 2024-04-05T21:33:07.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.215+0000 7f652f291700 1 -- 172.21.15.121:0/1475857309 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f652819b7e0 con 0x7f6528069020 2024-04-05T21:33:07.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.216+0000 7f652f291700 1 -- 172.21.15.121:0/1475857309 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6528109130 con 0x7f6528069020 2024-04-05T21:33:07.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.216+0000 7f651e7fc700 1 -- 172.21.15.121:0/1475857309 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6518007d20 con 0x7f6528069020 2024-04-05T21:33:07.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.217+0000 7f651e7fc700 1 --2- 172.21.15.121:0/1475857309 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f651406cce0 0x7f651406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:07.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.217+0000 7f651e7fc700 1 -- 172.21.15.121:0/1475857309 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f651808c400 con 0x7f6528069020 2024-04-05T21:33:07.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.217+0000 7f652d02d700 1 --2- 172.21.15.121:0/1475857309 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f651406cce0 0x7f651406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:07.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.217+0000 7f652d02d700 1 --2- 172.21.15.121:0/1475857309 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f651406cce0 0x7f651406f1a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6524006010 tx=0x7f652401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:07.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.221+0000 7f651e7fc700 1 -- 172.21.15.121:0/1475857309 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f65180565a0 con 0x7f6528069020 2024-04-05T21:33:07.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:07 smithi121 ceph-mon[145859]: pgmap v186: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-05T21:33:07.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:07 smithi121 ceph-mon[145859]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:07.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.497+0000 7f652f291700 1 -- 172.21.15.121:0/1475857309 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f652804ea90 con 0x7f6528069020 2024-04-05T21:33:07.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.498+0000 7f651e7fc700 1 -- 172.21.15.121:0/1475857309 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+799 (secure 0 0 0) 0x7f6518059bc0 con 0x7f6528069020 2024-04-05T21:33:07.499 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:33:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:33:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:33:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:33:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:33:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:33:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:33:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:33:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-04-05T21:33:07.499 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:33:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:33:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:33:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:33:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:33:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:33:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 7, 2024-04-05T21:33:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 7 2024-04-05T21:33:07.500 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:33:07.500 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:33:07.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.500+0000 7f652f291700 1 -- 172.21.15.121:0/1475857309 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f651406cce0 msgr2=0x7f651406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:07.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.500+0000 7f652f291700 1 --2- 172.21.15.121:0/1475857309 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f651406cce0 0x7f651406f1a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6524006010 tx=0x7f652401a040 comp rx=0 tx=0).stop 2024-04-05T21:33:07.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.500+0000 7f652f291700 1 -- 172.21.15.121:0/1475857309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6528069020 msgr2=0x7f6528196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:07.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.500+0000 7f652f291700 1 --2- 172.21.15.121:0/1475857309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6528069020 0x7f6528196bb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f651800c8f0 tx=0x7f651800ccb0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.500+0000 7f652f291700 1 -- 172.21.15.121:0/1475857309 shutdown_connections 2024-04-05T21:33:07.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.500+0000 7f652f291700 1 --2- 172.21.15.121:0/1475857309 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f651406cce0 0x7f651406f1a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.500+0000 7f652f291700 1 --2- 172.21.15.121:0/1475857309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6528068670 0x7f6528196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.500+0000 7f652f291700 1 --2- 172.21.15.121:0/1475857309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6528069020 0x7f6528196bb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.500+0000 7f652f291700 1 -- 172.21.15.121:0/1475857309 >> 172.21.15.121:0/1475857309 conn(0x7f65280755a0 msgr2=0x7f65280fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:07.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.500+0000 7f652f291700 1 -- 172.21.15.121:0/1475857309 shutdown_connections 2024-04-05T21:33:07.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.500+0000 7f652f291700 1 -- 172.21.15.121:0/1475857309 wait complete. 2024-04-05T21:33:07.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:07 smithi188 ceph-mon[107793]: pgmap v186: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-05T21:33:07.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:07 smithi188 ceph-mon[107793]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.604+0000 7f474dd15700 1 -- 172.21.15.121:0/3718071545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47480ffbd0 msgr2=0x7f4748100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.604+0000 7f474dd15700 1 --2- 172.21.15.121:0/3718071545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47480ffbd0 0x7f4748100050 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4730009910 tx=0x7f4730009c20 comp rx=0 tx=0).stop 2024-04-05T21:33:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.604+0000 7f474dd15700 1 -- 172.21.15.121:0/3718071545 shutdown_connections 2024-04-05T21:33:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.604+0000 7f474dd15700 1 --2- 172.21.15.121:0/3718071545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47480ffbd0 0x7f4748100050 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.604+0000 7f474dd15700 1 --2- 172.21.15.121:0/3718071545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4748105a40 0x7f4748105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.604+0000 7f474dd15700 1 -- 172.21.15.121:0/3718071545 >> 172.21.15.121:0/3718071545 conn(0x7f47480755a0 msgr2=0x7f47480759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.604+0000 7f474dd15700 1 -- 172.21.15.121:0/3718071545 shutdown_connections 2024-04-05T21:33:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.604+0000 7f474dd15700 1 -- 172.21.15.121:0/3718071545 wait complete. 2024-04-05T21:33:07.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.604+0000 7f474dd15700 1 Processor -- start 2024-04-05T21:33:07.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.605+0000 7f474dd15700 1 -- start start 2024-04-05T21:33:07.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.605+0000 7f474dd15700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47480ffbd0 0x7f4748195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:07.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.605+0000 7f474dd15700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4748105a40 0x7f4748195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:07.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.605+0000 7f474dd15700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4748196160 con 0x7f47480ffbd0 2024-04-05T21:33:07.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.605+0000 7f474dd15700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4748199ef0 con 0x7f4748105a40 2024-04-05T21:33:07.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.605+0000 7f47477fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47480ffbd0 0x7f4748195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:07.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.605+0000 7f47477fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47480ffbd0 0x7f4748195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59672/0 (socket says 172.21.15.121:59672) 2024-04-05T21:33:07.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.605+0000 7f47477fe700 1 -- 172.21.15.121:0/842353562 learned_addr learned my addr 172.21.15.121:0/842353562 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:07.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.605+0000 7f4746ffd700 1 --2- 172.21.15.121:0/842353562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4748105a40 0x7f4748195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:07.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.605+0000 7f47477fe700 1 -- 172.21.15.121:0/842353562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4748105a40 msgr2=0x7f4748195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:07.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.605+0000 7f47477fe700 1 --2- 172.21.15.121:0/842353562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4748105a40 0x7f4748195a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.605+0000 7f47477fe700 1 -- 172.21.15.121:0/842353562 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47300095f0 con 0x7f47480ffbd0 2024-04-05T21:33:07.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.605+0000 7f47477fe700 1 --2- 172.21.15.121:0/842353562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47480ffbd0 0x7f4748195540 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4738009970 tx=0x7f4738009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:07.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.606+0000 7f4744ff9700 1 -- 172.21.15.121:0/842353562 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4738012070 con 0x7f47480ffbd0 2024-04-05T21:33:07.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.606+0000 7f4744ff9700 1 -- 172.21.15.121:0/842353562 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4738007ad0 con 0x7f47480ffbd0 2024-04-05T21:33:07.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.606+0000 7f474dd15700 1 -- 172.21.15.121:0/842353562 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474819a170 con 0x7f47480ffbd0 2024-04-05T21:33:07.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.606+0000 7f4744ff9700 1 -- 172.21.15.121:0/842353562 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4738017bb0 con 0x7f47480ffbd0 2024-04-05T21:33:07.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.606+0000 7f474dd15700 1 -- 172.21.15.121:0/842353562 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f474819a6c0 con 0x7f47480ffbd0 2024-04-05T21:33:07.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.607+0000 7f474dd15700 1 -- 172.21.15.121:0/842353562 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474804ea90 con 0x7f47480ffbd0 2024-04-05T21:33:07.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.607+0000 7f4744ff9700 1 -- 172.21.15.121:0/842353562 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4738007c40 con 0x7f47480ffbd0 2024-04-05T21:33:07.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.607+0000 7f4744ff9700 1 --2- 172.21.15.121:0/842353562 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f473406cce0 0x7f473406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:07.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.607+0000 7f4744ff9700 1 -- 172.21.15.121:0/842353562 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f473808da30 con 0x7f47480ffbd0 2024-04-05T21:33:07.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.607+0000 7f4746ffd700 1 --2- 172.21.15.121:0/842353562 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f473406cce0 0x7f473406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:07.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.608+0000 7f4746ffd700 1 --2- 172.21.15.121:0/842353562 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f473406cce0 0x7f473406f1a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4748196b60 tx=0x7f473001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:07.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.612+0000 7f4744ff9700 1 -- 172.21.15.121:0/842353562 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4738057c00 con 0x7f47480ffbd0 2024-04-05T21:33:07.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.853+0000 7f474dd15700 1 -- 172.21.15.121:0/842353562 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f474819abb0 con 0x7f47480ffbd0 2024-04-05T21:33:07.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.854+0000 7f4744ff9700 1 -- 172.21.15.121:0/842353562 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f473805b220 con 0x7f47480ffbd0 2024-04-05T21:33:07.855 INFO:teuthology.orchestra.run.smithi121.stdout:e21 2024-04-05T21:33:07.855 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:33:07.855 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:33:07.855 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:33:07.855 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:33:07.855 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:33:07.855 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:33:07.855 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 21 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:30:52.246236+0000 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 46 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:33:07.856 INFO:teuthology.orchestra.run.smithi121.stdout:up {0=14476,1=24255} 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout:failed 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{0:14476} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{0:14488} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/4279084130,v1:172.21.15.121:6827/4279084130] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{1:24255} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{1:24265} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:33:07.857 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:33:07.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.856+0000 7f474dd15700 1 -- 172.21.15.121:0/842353562 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f473406cce0 msgr2=0x7f473406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:07.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.856+0000 7f474dd15700 1 --2- 172.21.15.121:0/842353562 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f473406cce0 0x7f473406f1a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4748196b60 tx=0x7f473001a040 comp rx=0 tx=0).stop 2024-04-05T21:33:07.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.856+0000 7f474dd15700 1 -- 172.21.15.121:0/842353562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47480ffbd0 msgr2=0x7f4748195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:07.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.856+0000 7f474dd15700 1 --2- 172.21.15.121:0/842353562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47480ffbd0 0x7f4748195540 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4738009970 tx=0x7f4738009c80 comp rx=0 tx=0).stop 2024-04-05T21:33:07.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.856+0000 7f474dd15700 1 -- 172.21.15.121:0/842353562 shutdown_connections 2024-04-05T21:33:07.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.856+0000 7f474dd15700 1 --2- 172.21.15.121:0/842353562 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f473406cce0 0x7f473406f1a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.856+0000 7f474dd15700 1 --2- 172.21.15.121:0/842353562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47480ffbd0 0x7f4748195540 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.856+0000 7f474dd15700 1 --2- 172.21.15.121:0/842353562 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4748105a40 0x7f4748195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.856+0000 7f474dd15700 1 -- 172.21.15.121:0/842353562 >> 172.21.15.121:0/842353562 conn(0x7f47480755a0 msgr2=0x7f47480fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:07.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.856+0000 7f474dd15700 1 -- 172.21.15.121:0/842353562 shutdown_connections 2024-04-05T21:33:07.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.856+0000 7f474dd15700 1 -- 172.21.15.121:0/842353562 wait complete. 2024-04-05T21:33:07.859 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 21 2024-04-05T21:33:07.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.960+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1896583142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2180ffa40 msgr2=0x7ff2180ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:07.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.960+0000 7ff21d1e6700 1 --2- 172.21.15.121:0/1896583142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2180ffa40 0x7ff2180ffec0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff208009910 tx=0x7ff208009c20 comp rx=0 tx=0).stop 2024-04-05T21:33:07.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.960+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1896583142 shutdown_connections 2024-04-05T21:33:07.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.960+0000 7ff21d1e6700 1 --2- 172.21.15.121:0/1896583142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2180ffa40 0x7ff2180ffec0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.960+0000 7ff21d1e6700 1 --2- 172.21.15.121:0/1896583142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff218105a40 0x7ff218105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.960+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1896583142 >> 172.21.15.121:0/1896583142 conn(0x7ff2180fb570 msgr2=0x7ff2180fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:07.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.960+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1896583142 shutdown_connections 2024-04-05T21:33:07.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.960+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1896583142 wait complete. 2024-04-05T21:33:07.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.960+0000 7ff21d1e6700 1 Processor -- start 2024-04-05T21:33:07.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.961+0000 7ff21d1e6700 1 -- start start 2024-04-05T21:33:07.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.961+0000 7ff21d1e6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2180ffa40 0x7ff218197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:07.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.961+0000 7ff21d1e6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff218105a40 0x7ff2181979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:07.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.961+0000 7ff21d1e6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff218197f60 con 0x7ff2180ffa40 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.961+0000 7ff21d1e6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2181980d0 con 0x7ff218105a40 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.961+0000 7ff216d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2180ffa40 0x7ff218197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.961+0000 7ff216d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2180ffa40 0x7ff218197490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59676/0 (socket says 172.21.15.121:59676) 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.961+0000 7ff216d9d700 1 -- 172.21.15.121:0/1747541638 learned_addr learned my addr 172.21.15.121:0/1747541638 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.961+0000 7ff21659c700 1 --2- 172.21.15.121:0/1747541638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff218105a40 0x7ff2181979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.961+0000 7ff216d9d700 1 -- 172.21.15.121:0/1747541638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff218105a40 msgr2=0x7ff2181979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.961+0000 7ff216d9d700 1 --2- 172.21.15.121:0/1747541638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff218105a40 0x7ff2181979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.961+0000 7ff216d9d700 1 -- 172.21.15.121:0/1747541638 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2080095f0 con 0x7ff2180ffa40 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.961+0000 7ff216d9d700 1 --2- 172.21.15.121:0/1747541638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2180ffa40 0x7ff218197490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00c930 tx=0x7ff20c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.962+0000 7ff1fffff700 1 -- 172.21.15.121:0/1747541638 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff20c007ad0 con 0x7ff2180ffa40 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.962+0000 7ff1fffff700 1 -- 172.21.15.121:0/1747541638 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff20c00ce80 con 0x7ff2180ffa40 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.962+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1747541638 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2181a0500 con 0x7ff2180ffa40 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.962+0000 7ff1fffff700 1 -- 172.21.15.121:0/1747541638 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff20c018630 con 0x7ff2180ffa40 2024-04-05T21:33:07.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.962+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1747541638 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2181a0970 con 0x7ff2180ffa40 2024-04-05T21:33:07.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.963+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1747541638 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff218107fd0 con 0x7ff2180ffa40 2024-04-05T21:33:07.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.963+0000 7ff1fffff700 1 -- 172.21.15.121:0/1747541638 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff20c00f450 con 0x7ff2180ffa40 2024-04-05T21:33:07.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.963+0000 7ff1fffff700 1 --2- 172.21.15.121:0/1747541638 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff20006cce0 0x7ff20006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:07.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.963+0000 7ff1fffff700 1 -- 172.21.15.121:0/1747541638 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6156+0+0 (secure 0 0 0) 0x7ff20c08c520 con 0x7ff2180ffa40 2024-04-05T21:33:07.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.963+0000 7ff21659c700 1 --2- 172.21.15.121:0/1747541638 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff20006cce0 0x7ff20006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:07.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.964+0000 7ff21659c700 1 --2- 172.21.15.121:0/1747541638 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff20006cce0 0x7ff20006f1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff218069570 tx=0x7ff20801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:07.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:07.968+0000 7ff1fffff700 1 -- 172.21.15.121:0/1747541638 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff20c056640 con 0x7ff2180ffa40 2024-04-05T21:33:08.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.175+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1747541638 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff21802cd80 con 0x7ff20006cce0 2024-04-05T21:33:08.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.176+0000 7ff1fffff700 1 -- 172.21.15.121:0/1747541638 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7ff21802cd80 con 0x7ff20006cce0 2024-04-05T21:33:08.177 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:33:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:33:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:33:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:33:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:33:08.177 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:33:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:33:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:33:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:33:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "9/23 daemons upgraded", 2024-04-05T21:33:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading osd daemons", 2024-04-05T21:33:08.178 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:33:08.178 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:33:08.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.178+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1747541638 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff20006cce0 msgr2=0x7ff20006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:08.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.178+0000 7ff21d1e6700 1 --2- 172.21.15.121:0/1747541638 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff20006cce0 0x7ff20006f1a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff218069570 tx=0x7ff20801a040 comp rx=0 tx=0).stop 2024-04-05T21:33:08.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.178+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1747541638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2180ffa40 msgr2=0x7ff218197490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:08.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.178+0000 7ff21d1e6700 1 --2- 172.21.15.121:0/1747541638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2180ffa40 0x7ff218197490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00c930 tx=0x7ff20c00cc40 comp rx=0 tx=0).stop 2024-04-05T21:33:08.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.178+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1747541638 shutdown_connections 2024-04-05T21:33:08.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.178+0000 7ff21d1e6700 1 --2- 172.21.15.121:0/1747541638 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff20006cce0 0x7ff20006f1a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:08.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.178+0000 7ff21d1e6700 1 --2- 172.21.15.121:0/1747541638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2180ffa40 0x7ff218197490 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:08.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.178+0000 7ff21d1e6700 1 --2- 172.21.15.121:0/1747541638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff218105a40 0x7ff2181979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:08.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.178+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1747541638 >> 172.21.15.121:0/1747541638 conn(0x7ff2180fb570 msgr2=0x7ff2180fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:08.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.178+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1747541638 shutdown_connections 2024-04-05T21:33:08.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.178+0000 7ff21d1e6700 1 -- 172.21.15.121:0/1747541638 wait complete. 2024-04-05T21:33:08.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.282+0000 7f9b1effd700 1 -- 172.21.15.121:0/3369546212 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b100053e0 con 0x7f9b20102620 2024-04-05T21:33:08.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.283+0000 7f9b269b8700 1 -- 172.21.15.121:0/3369546212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b20102620 msgr2=0x7f9b2010ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:08.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.283+0000 7f9b269b8700 1 --2- 172.21.15.121:0/3369546212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b20102620 0x7f9b2010ab10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9b10009910 tx=0x7f9b10009c20 comp rx=0 tx=0).stop 2024-04-05T21:33:08.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.283+0000 7f9b269b8700 1 -- 172.21.15.121:0/3369546212 shutdown_connections 2024-04-05T21:33:08.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.283+0000 7f9b269b8700 1 --2- 172.21.15.121:0/3369546212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b20102620 0x7f9b2010ab10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:08.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.283+0000 7f9b269b8700 1 --2- 172.21.15.121:0/3369546212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b20101d00 0x7f9b201020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:08.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.283+0000 7f9b269b8700 1 -- 172.21.15.121:0/3369546212 >> 172.21.15.121:0/3369546212 conn(0x7f9b200fb570 msgr2=0x7f9b200fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:08.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.283+0000 7f9b269b8700 1 -- 172.21.15.121:0/3369546212 shutdown_connections 2024-04-05T21:33:08.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.283+0000 7f9b269b8700 1 -- 172.21.15.121:0/3369546212 wait complete. 2024-04-05T21:33:08.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.283+0000 7f9b269b8700 1 Processor -- start 2024-04-05T21:33:08.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.283+0000 7f9b269b8700 1 -- start start 2024-04-05T21:33:08.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.284+0000 7f9b269b8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b20101d00 0x7f9b201966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:08.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.284+0000 7f9b269b8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b20102620 0x7f9b20196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:08.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.284+0000 7f9b269b8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b201972a0 con 0x7f9b20102620 2024-04-05T21:33:08.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.284+0000 7f9b269b8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b20199fd0 con 0x7f9b20101d00 2024-04-05T21:33:08.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.284+0000 7f9b1f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b20102620 0x7f9b20196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:08.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.284+0000 7f9b1f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b20102620 0x7f9b20196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59694/0 (socket says 172.21.15.121:59694) 2024-04-05T21:33:08.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.284+0000 7f9b1f7fe700 1 -- 172.21.15.121:0/2414783435 learned_addr learned my addr 172.21.15.121:0/2414783435 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.284+0000 7f9b1ffff700 1 --2- 172.21.15.121:0/2414783435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b20101d00 0x7f9b201966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.284+0000 7f9b1f7fe700 1 -- 172.21.15.121:0/2414783435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b20101d00 msgr2=0x7f9b201966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.284+0000 7f9b1f7fe700 1 --2- 172.21.15.121:0/2414783435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b20101d00 0x7f9b201966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.284+0000 7f9b1f7fe700 1 -- 172.21.15.121:0/2414783435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b100095f0 con 0x7f9b20102620 2024-04-05T21:33:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.284+0000 7f9b1f7fe700 1 --2- 172.21.15.121:0/2414783435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b20102620 0x7f9b20196c10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9b100092e0 tx=0x7f9b1000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.284+0000 7f9b1d7fa700 1 -- 172.21.15.121:0/2414783435 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b1001d070 con 0x7f9b20102620 2024-04-05T21:33:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.285+0000 7f9b269b8700 1 -- 172.21.15.121:0/2414783435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2019a250 con 0x7f9b20102620 2024-04-05T21:33:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.285+0000 7f9b1d7fa700 1 -- 172.21.15.121:0/2414783435 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b10003d30 con 0x7f9b20102620 2024-04-05T21:33:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.285+0000 7f9b1d7fa700 1 -- 172.21.15.121:0/2414783435 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b1000fdb0 con 0x7f9b20102620 2024-04-05T21:33:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.285+0000 7f9b269b8700 1 -- 172.21.15.121:0/2414783435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2019a740 con 0x7f9b20102620 2024-04-05T21:33:08.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.286+0000 7f9b269b8700 1 -- 172.21.15.121:0/2414783435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b20108210 con 0x7f9b20102620 2024-04-05T21:33:08.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.286+0000 7f9b1d7fa700 1 -- 172.21.15.121:0/2414783435 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9b10003890 con 0x7f9b20102620 2024-04-05T21:33:08.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.286+0000 7f9b1d7fa700 1 --2- 172.21.15.121:0/2414783435 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9b0806cce0 0x7f9b0806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:08.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.286+0000 7f9b1d7fa700 1 -- 172.21.15.121:0/2414783435 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f9b1008d670 con 0x7f9b20102620 2024-04-05T21:33:08.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.286+0000 7f9b1ffff700 1 --2- 172.21.15.121:0/2414783435 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9b0806cce0 0x7f9b0806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:08.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.287+0000 7f9b1ffff700 1 --2- 172.21.15.121:0/2414783435 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9b0806cce0 0x7f9b0806f1a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9b14007be0 tx=0x7f9b140073d0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:08.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.291+0000 7f9b1d7fa700 1 -- 172.21.15.121:0/2414783435 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9b10057790 con 0x7f9b20102620 2024-04-05T21:33:08.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:08 smithi121 ceph-mon[145859]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:08.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:08 smithi121 ceph-mon[145859]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:08.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:08 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T21:33:08.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:08 smithi121 ceph-mon[145859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T21:33:08.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:08 smithi121 ceph-mon[145859]: Upgrade: osd.3 is safe to restart 2024-04-05T21:33:08.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:08 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1475857309' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:33:08.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:08 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/842353562' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:33:08.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:08 smithi188 ceph-mon[107793]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:08.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:08 smithi188 ceph-mon[107793]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:08.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:08 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T21:33:08.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:08 smithi188 ceph-mon[107793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-05T21:33:08.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:08 smithi188 ceph-mon[107793]: Upgrade: osd.3 is safe to restart 2024-04-05T21:33:08.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:08 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1475857309' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:33:08.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:08 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/842353562' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:33:08.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.561+0000 7f9b269b8700 1 -- 172.21.15.121:0/2414783435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9b20197a30 con 0x7f9b20102620 2024-04-05T21:33:08.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.561+0000 7f9b1d7fa700 1 -- 172.21.15.121:0/2414783435 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f9b1005adb0 con 0x7f9b20102620 2024-04-05T21:33:08.563 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_OK 2024-04-05T21:33:08.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.563+0000 7f9b269b8700 1 -- 172.21.15.121:0/2414783435 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9b0806cce0 msgr2=0x7f9b0806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:08.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.563+0000 7f9b269b8700 1 --2- 172.21.15.121:0/2414783435 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9b0806cce0 0x7f9b0806f1a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9b14007be0 tx=0x7f9b140073d0 comp rx=0 tx=0).stop 2024-04-05T21:33:08.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.564+0000 7f9b269b8700 1 -- 172.21.15.121:0/2414783435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b20102620 msgr2=0x7f9b20196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:08.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.564+0000 7f9b269b8700 1 --2- 172.21.15.121:0/2414783435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b20102620 0x7f9b20196c10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9b100092e0 tx=0x7f9b1000bd70 comp rx=0 tx=0).stop 2024-04-05T21:33:08.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.564+0000 7f9b269b8700 1 -- 172.21.15.121:0/2414783435 shutdown_connections 2024-04-05T21:33:08.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.564+0000 7f9b269b8700 1 --2- 172.21.15.121:0/2414783435 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9b0806cce0 0x7f9b0806f1a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:08.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.564+0000 7f9b269b8700 1 --2- 172.21.15.121:0/2414783435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9b20101d00 0x7f9b201966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:08.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.564+0000 7f9b269b8700 1 --2- 172.21.15.121:0/2414783435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b20102620 0x7f9b20196c10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:08.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.564+0000 7f9b269b8700 1 -- 172.21.15.121:0/2414783435 >> 172.21.15.121:0/2414783435 conn(0x7f9b200fb570 msgr2=0x7f9b20105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:08.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.564+0000 7f9b269b8700 1 -- 172.21.15.121:0/2414783435 shutdown_connections 2024-04-05T21:33:08.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:08.564+0000 7f9b269b8700 1 -- 172.21.15.121:0/2414783435 wait complete. 2024-04-05T21:33:09.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:09 smithi188 ceph-mon[107793]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:09.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:09 smithi188 ceph-mon[107793]: pgmap v187: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T21:33:09.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:09 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2414783435' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:33:09.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:09 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:09.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:09 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-05T21:33:09.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:09 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:33:09.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:09 smithi121 ceph-mon[145859]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:09.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:09 smithi121 ceph-mon[145859]: pgmap v187: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T21:33:09.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:09 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2414783435' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:33:09.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:09 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:09.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:09 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-05T21:33:09.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:09 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:33:10.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:10 smithi121 ceph-mon[145859]: Upgrade: Updating osd.3 2024-04-05T21:33:10.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:10 smithi121 ceph-mon[145859]: Deploying daemon osd.3 on smithi188 2024-04-05T21:33:10.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:10 smithi188 ceph-mon[107793]: Upgrade: Updating osd.3 2024-04-05T21:33:10.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:10 smithi188 ceph-mon[107793]: Deploying daemon osd.3 on smithi188 2024-04-05T21:33:11.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:11 smithi121 ceph-mon[145859]: pgmap v188: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T21:33:11.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:11 smithi188 ceph-mon[107793]: pgmap v188: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T21:33:12.823 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:12 smithi188 systemd[1]: Stopping Ceph osd.3 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:33:13.213 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:12 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-3[56622]: 2024-04-05T21:33:12.819+0000 7f69e98f6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T21:33:13.213 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:12 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-3[56622]: 2024-04-05T21:33:12.819+0000 7f69e98f6700 -1 osd.3 63 *** Got signal Terminated *** 2024-04-05T21:33:13.213 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:12 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-3[56622]: 2024-04-05T21:33:12.819+0000 7f69e98f6700 -1 osd.3 63 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T21:33:13.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:13 smithi121 ceph-mon[145859]: pgmap v189: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-05T21:33:13.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:13 smithi121 ceph-mon[145859]: osd.3 marked itself down and dead 2024-04-05T21:33:13.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:13 smithi188 ceph-mon[107793]: pgmap v189: 65 pgs: 65 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-05T21:33:13.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:13 smithi188 ceph-mon[107793]: osd.3 marked itself down and dead 2024-04-05T21:33:14.280 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:14 smithi188 bash[117773]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-3 2024-04-05T21:33:14.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:14 smithi188 ceph-mon[107793]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T21:33:14.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:14 smithi188 ceph-mon[107793]: osdmap e64: 6 total, 5 up, 6 in 2024-04-05T21:33:14.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:14 smithi121 ceph-mon[145859]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T21:33:14.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:14 smithi121 ceph-mon[145859]: osdmap e64: 6 total, 5 up, 6 in 2024-04-05T21:33:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:15 smithi121 ceph-mon[145859]: pgmap v191: 65 pgs: 16 stale+active+clean, 49 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T21:33:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:15 smithi121 ceph-mon[145859]: osdmap e65: 6 total, 5 up, 6 in 2024-04-05T21:33:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:33:15.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:15 smithi188 ceph-mon[107793]: pgmap v191: 65 pgs: 16 stale+active+clean, 49 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T21:33:15.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:15 smithi188 ceph-mon[107793]: osdmap e65: 6 total, 5 up, 6 in 2024-04-05T21:33:15.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:15.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:33:16.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:16 smithi188 ceph-mon[107793]: Health check failed: Degraded data redundancy: 37/333 objects degraded (11.111%), 9 pgs degraded (PG_DEGRADED) 2024-04-05T21:33:16.530 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:16 smithi188 systemd[1]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.3.service: Succeeded. 2024-04-05T21:33:16.530 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:16 smithi188 systemd[1]: Stopped Ceph osd.3 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:33:16.530 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:16 smithi188 systemd[1]: Starting Ceph osd.3 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:33:16.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:16 smithi121 ceph-mon[145859]: Health check failed: Degraded data redundancy: 37/333 objects degraded (11.111%), 9 pgs degraded (PG_DEGRADED) 2024-04-05T21:33:17.528 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:17 smithi188 ceph-mon[107793]: pgmap v193: 65 pgs: 9 active+undersized+degraded, 6 stale+active+clean, 9 active+undersized, 41 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 37/333 objects degraded (11.111%) 2024-04-05T21:33:17.528 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:17 smithi188 bash[118045]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-05T21:33:17.528 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:17 smithi188 bash[118045]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-05T21:33:17.528 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:17 smithi188 bash[118045]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-05T21:33:17.528 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:17 smithi188 bash[118045]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-05T21:33:17.528 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:17 smithi188 bash[118045]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-05T21:33:17.528 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:17 smithi188 bash[118045]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-05T21:33:17.529 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:17 smithi188 bash[118045]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-05T21:33:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:17 smithi121 ceph-mon[145859]: pgmap v193: 65 pgs: 9 active+undersized+degraded, 6 stale+active+clean, 9 active+undersized, 41 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 37/333 objects degraded (11.111%) 2024-04-05T21:33:19.030 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:18 smithi188 bash[118305]: 8607be213a4a5b3df6aa51fc85657337f4b8f42379097f900f826bdbbda190ab 2024-04-05T21:33:19.030 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:18 smithi188 systemd[1]: Started Ceph osd.3 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:33:19.464 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:19 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-3[118320]: 2024-04-05T21:33:19.064+0000 7f3afeaf47c0 -1 Falling back to public interface 2024-04-05T21:33:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:19 smithi188 ceph-mon[107793]: pgmap v194: 65 pgs: 16 active+undersized+degraded, 3 stale+active+clean, 15 active+undersized, 31 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 59/333 objects degraded (17.718%) 2024-04-05T21:33:19.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:19 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:19.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:19 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:19.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:19 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:33:19.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:19 smithi121 ceph-mon[145859]: pgmap v194: 65 pgs: 16 active+undersized+degraded, 3 stale+active+clean, 15 active+undersized, 31 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 59/333 objects degraded (17.718%) 2024-04-05T21:33:19.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:19 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:19.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:19 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:19.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:19 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:33:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:21 smithi188 ceph-mon[107793]: pgmap v195: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 68/333 objects degraded (20.420%) 2024-04-05T21:33:21.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:21 smithi121 ceph-mon[145859]: pgmap v195: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 68/333 objects degraded (20.420%) 2024-04-05T21:33:22.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:22 smithi121 ceph-mon[145859]: Health check update: Degraded data redundancy: 68/333 objects degraded (20.420%), 19 pgs degraded (PG_DEGRADED) 2024-04-05T21:33:22.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:22 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:22.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:22 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:23.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:22 smithi188 ceph-mon[107793]: Health check update: Degraded data redundancy: 68/333 objects degraded (20.420%), 19 pgs degraded (PG_DEGRADED) 2024-04-05T21:33:23.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:22 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:23.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:22 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:23.737 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:23 smithi188 ceph-mon[107793]: pgmap v196: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 68/333 objects degraded (20.420%) 2024-04-05T21:33:23.737 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:23 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:23.737 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:23 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:23.738 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:23 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-3[118320]: 2024-04-05T21:33:23.646+0000 7f3afeaf47c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-05T21:33:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:23 smithi121 ceph-mon[145859]: pgmap v196: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 68/333 objects degraded (20.420%) 2024-04-05T21:33:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:23 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:23 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:24.529 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:24 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-3[118320]: 2024-04-05T21:33:24.178+0000 7f3afeaf47c0 -1 osd.3 63 log_to_monitors true 2024-04-05T21:33:24.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:24 smithi121 ceph-mon[145859]: from='osd.3 [v2:172.21.15.188:6800/83582134,v1:172.21.15.188:6801/83582134]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T21:33:24.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:24 smithi121 ceph-mon[145859]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T21:33:25.007 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:24 smithi188 ceph-mon[107793]: from='osd.3 [v2:172.21.15.188:6800/83582134,v1:172.21.15.188:6801/83582134]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T21:33:25.007 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:24 smithi188 ceph-mon[107793]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-05T21:33:25.779 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 21:33:25 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-3[118320]: 2024-04-05T21:33:25.498+0000 7f3aebf46700 -1 osd.3 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T21:33:25.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:25 smithi188 ceph-mon[107793]: pgmap v197: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 68/333 objects degraded (20.420%) 2024-04-05T21:33:25.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:25 smithi188 ceph-mon[107793]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-05T21:33:25.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:25 smithi188 ceph-mon[107793]: osdmap e66: 6 total, 5 up, 6 in 2024-04-05T21:33:25.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:25 smithi188 ceph-mon[107793]: from='osd.3 [v2:172.21.15.188:6800/83582134,v1:172.21.15.188:6801/83582134]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:33:25.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:25 smithi188 ceph-mon[107793]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:33:25.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:25 smithi188 ceph-mon[107793]: from='osd.3 ' entity='osd.3' 2024-04-05T21:33:25.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:25 smithi121 ceph-mon[145859]: pgmap v197: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.3 GiB used, 535 GiB / 536 GiB avail; 68/333 objects degraded (20.420%) 2024-04-05T21:33:25.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:25 smithi121 ceph-mon[145859]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-05T21:33:25.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:25 smithi121 ceph-mon[145859]: osdmap e66: 6 total, 5 up, 6 in 2024-04-05T21:33:25.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:25 smithi121 ceph-mon[145859]: from='osd.3 [v2:172.21.15.188:6800/83582134,v1:172.21.15.188:6801/83582134]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:33:25.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:25 smithi121 ceph-mon[145859]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:33:25.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:25 smithi121 ceph-mon[145859]: from='osd.3 ' entity='osd.3' 2024-04-05T21:33:26.941 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:26 smithi188 ceph-mon[107793]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T21:33:26.941 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:26 smithi188 ceph-mon[107793]: osd.3 [v2:172.21.15.188:6800/83582134,v1:172.21.15.188:6801/83582134] boot 2024-04-05T21:33:26.941 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:26 smithi188 ceph-mon[107793]: osdmap e67: 6 total, 6 up, 6 in 2024-04-05T21:33:26.941 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:26 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:33:26.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:26 smithi121 ceph-mon[145859]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T21:33:26.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:26 smithi121 ceph-mon[145859]: osd.3 [v2:172.21.15.188:6800/83582134,v1:172.21.15.188:6801/83582134] boot 2024-04-05T21:33:26.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:26 smithi121 ceph-mon[145859]: osdmap e67: 6 total, 6 up, 6 in 2024-04-05T21:33:26.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:26 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-05T21:33:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:27 smithi121 ceph-mon[145859]: pgmap v199: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 68/333 objects degraded (20.420%) 2024-04-05T21:33:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:33:27.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:33:27.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:27.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:33:27.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:33:27.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:33:27.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:33:27.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T21:33:27.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:27 smithi121 ceph-mon[145859]: osdmap e68: 6 total, 6 up, 6 in 2024-04-05T21:33:28.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:27 smithi188 ceph-mon[107793]: pgmap v199: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 68/333 objects degraded (20.420%) 2024-04-05T21:33:28.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:28.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:28.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:33:28.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:33:28.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:28.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:33:28.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:33:28.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:33:28.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:33:28.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T21:33:28.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:27 smithi188 ceph-mon[107793]: osdmap e68: 6 total, 6 up, 6 in 2024-04-05T21:33:28.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:28 smithi121 ceph-mon[145859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T21:33:28.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:28 smithi121 ceph-mon[145859]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-05T21:33:29.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:28 smithi188 ceph-mon[107793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T21:33:29.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:28 smithi188 ceph-mon[107793]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-04-05T21:33:29.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:29 smithi121 ceph-mon[145859]: pgmap v202: 65 pgs: 3 peering, 18 active+undersized+degraded, 17 active+undersized, 27 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s; 66/333 objects degraded (19.820%) 2024-04-05T21:33:29.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:29 smithi121 ceph-mon[145859]: Health check update: Degraded data redundancy: 66/333 objects degraded (19.820%), 18 pgs degraded (PG_DEGRADED) 2024-04-05T21:33:30.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:29 smithi188 ceph-mon[107793]: pgmap v202: 65 pgs: 3 peering, 18 active+undersized+degraded, 17 active+undersized, 27 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s; 66/333 objects degraded (19.820%) 2024-04-05T21:33:30.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:29 smithi188 ceph-mon[107793]: Health check update: Degraded data redundancy: 66/333 objects degraded (19.820%), 18 pgs degraded (PG_DEGRADED) 2024-04-05T21:33:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:33:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:31 smithi121 ceph-mon[145859]: pgmap v203: 65 pgs: 3 peering, 7 active+undersized+degraded, 8 active+undersized, 47 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 20/333 objects degraded (6.006%) 2024-04-05T21:33:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:33:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:31 smithi188 ceph-mon[107793]: pgmap v203: 65 pgs: 3 peering, 7 active+undersized+degraded, 8 active+undersized, 47 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 20/333 objects degraded (6.006%) 2024-04-05T21:33:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:32 smithi121 ceph-mon[145859]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/333 objects degraded (6.006%), 7 pgs degraded) 2024-04-05T21:33:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:32 smithi121 ceph-mon[145859]: Cluster is now healthy 2024-04-05T21:33:32.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:32 smithi188 ceph-mon[107793]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/333 objects degraded (6.006%), 7 pgs degraded) 2024-04-05T21:33:32.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:32 smithi188 ceph-mon[107793]: Cluster is now healthy 2024-04-05T21:33:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:33 smithi121 ceph-mon[145859]: pgmap v204: 65 pgs: 3 peering, 62 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-05T21:33:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:33 smithi188 ceph-mon[107793]: pgmap v204: 65 pgs: 3 peering, 62 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 3 op/s 2024-04-05T21:33:35.420 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:35 smithi121 ceph-mon[145859]: pgmap v205: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 895 B/s rd, 2 op/s 2024-04-05T21:33:35.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:35 smithi188 ceph-mon[107793]: pgmap v205: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 895 B/s rd, 2 op/s 2024-04-05T21:33:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:37 smithi121 ceph-mon[145859]: pgmap v206: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-05T21:33:37.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:37 smithi188 ceph-mon[107793]: pgmap v206: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-05T21:33:38.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.680+0000 7f411bb8f700 1 -- 172.21.15.121:0/2328855164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41140ffa40 msgr2=0x7f41140ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:38.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.680+0000 7f411bb8f700 1 --2- 172.21.15.121:0/2328855164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41140ffa40 0x7f41140ffec0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4110009910 tx=0x7f4110009c20 comp rx=0 tx=0).stop 2024-04-05T21:33:38.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.680+0000 7f411bb8f700 1 -- 172.21.15.121:0/2328855164 shutdown_connections 2024-04-05T21:33:38.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.680+0000 7f411bb8f700 1 --2- 172.21.15.121:0/2328855164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41140ffa40 0x7f41140ffec0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:38.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.680+0000 7f411bb8f700 1 --2- 172.21.15.121:0/2328855164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4114105a40 0x7f4114105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:38.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.680+0000 7f411bb8f700 1 -- 172.21.15.121:0/2328855164 >> 172.21.15.121:0/2328855164 conn(0x7f41140fb570 msgr2=0x7f41140fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:38.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.680+0000 7f411bb8f700 1 -- 172.21.15.121:0/2328855164 shutdown_connections 2024-04-05T21:33:38.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.680+0000 7f411bb8f700 1 -- 172.21.15.121:0/2328855164 wait complete. 2024-04-05T21:33:38.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.681+0000 7f411bb8f700 1 Processor -- start 2024-04-05T21:33:38.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.681+0000 7f411bb8f700 1 -- start start 2024-04-05T21:33:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.681+0000 7f411bb8f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41140ffa40 0x7f4114195660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.681+0000 7f411bb8f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4114105a40 0x7f4114195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.681+0000 7f411bb8f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4114196280 con 0x7f41140ffa40 2024-04-05T21:33:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.681+0000 7f411bb8f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411419a010 con 0x7f4114105a40 2024-04-05T21:33:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.681+0000 7f411992b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41140ffa40 0x7f4114195660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.681+0000 7f411992b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41140ffa40 0x7f4114195660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56650/0 (socket says 172.21.15.121:56650) 2024-04-05T21:33:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.681+0000 7f411992b700 1 -- 172.21.15.121:0/586200728 learned_addr learned my addr 172.21.15.121:0/586200728 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.682+0000 7f411912a700 1 --2- 172.21.15.121:0/586200728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4114105a40 0x7f4114195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.682+0000 7f411992b700 1 -- 172.21.15.121:0/586200728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4114105a40 msgr2=0x7f4114195ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.682+0000 7f411992b700 1 --2- 172.21.15.121:0/586200728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4114105a40 0x7f4114195ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:38.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.682+0000 7f411992b700 1 -- 172.21.15.121:0/586200728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41100095f0 con 0x7f41140ffa40 2024-04-05T21:33:38.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.682+0000 7f411992b700 1 --2- 172.21.15.121:0/586200728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41140ffa40 0x7f4114195660 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f410800b4b0 tx=0x7f410800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:38.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.682+0000 7f4106ffd700 1 -- 172.21.15.121:0/586200728 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4108011800 con 0x7f41140ffa40 2024-04-05T21:33:38.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.682+0000 7f4106ffd700 1 -- 172.21.15.121:0/586200728 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4108011e00 con 0x7f41140ffa40 2024-04-05T21:33:38.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.682+0000 7f411bb8f700 1 -- 172.21.15.121:0/586200728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411419a2f0 con 0x7f41140ffa40 2024-04-05T21:33:38.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.682+0000 7f4106ffd700 1 -- 172.21.15.121:0/586200728 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4108010e50 con 0x7f41140ffa40 2024-04-05T21:33:38.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.682+0000 7f411bb8f700 1 -- 172.21.15.121:0/586200728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f411419a840 con 0x7f41140ffa40 2024-04-05T21:33:38.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.683+0000 7f411bb8f700 1 -- 172.21.15.121:0/586200728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4114108060 con 0x7f41140ffa40 2024-04-05T21:33:38.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.683+0000 7f4106ffd700 1 -- 172.21.15.121:0/586200728 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4108010380 con 0x7f41140ffa40 2024-04-05T21:33:38.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.684+0000 7f4106ffd700 1 --2- 172.21.15.121:0/586200728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f410006cce0 0x7f410006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:38.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.684+0000 7f4106ffd700 1 -- 172.21.15.121:0/586200728 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f410808c040 con 0x7f41140ffa40 2024-04-05T21:33:38.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.684+0000 7f411912a700 1 --2- 172.21.15.121:0/586200728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f410006cce0 0x7f410006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:38.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.684+0000 7f411912a700 1 --2- 172.21.15.121:0/586200728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f410006cce0 0x7f410006f1a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4114196c80 tx=0x7f411001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:38.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.689+0000 7f4106ffd700 1 -- 172.21.15.121:0/586200728 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f41080561e0 con 0x7f41140ffa40 2024-04-05T21:33:38.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.896+0000 7f411bb8f700 1 -- 172.21.15.121:0/586200728 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f411419ab30 con 0x7f410006cce0 2024-04-05T21:33:38.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.897+0000 7f4106ffd700 1 -- 172.21.15.121:0/586200728 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f411419ab30 con 0x7f410006cce0 2024-04-05T21:33:38.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.899+0000 7f411bb8f700 1 -- 172.21.15.121:0/586200728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f410006cce0 msgr2=0x7f410006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:38.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.899+0000 7f411bb8f700 1 --2- 172.21.15.121:0/586200728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f410006cce0 0x7f410006f1a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4114196c80 tx=0x7f411001a040 comp rx=0 tx=0).stop 2024-04-05T21:33:38.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.899+0000 7f411bb8f700 1 -- 172.21.15.121:0/586200728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41140ffa40 msgr2=0x7f4114195660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:38.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.899+0000 7f411bb8f700 1 --2- 172.21.15.121:0/586200728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41140ffa40 0x7f4114195660 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f410800b4b0 tx=0x7f410800b870 comp rx=0 tx=0).stop 2024-04-05T21:33:38.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.899+0000 7f411bb8f700 1 -- 172.21.15.121:0/586200728 shutdown_connections 2024-04-05T21:33:38.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.899+0000 7f411bb8f700 1 --2- 172.21.15.121:0/586200728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f410006cce0 0x7f410006f1a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:38.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.899+0000 7f411bb8f700 1 --2- 172.21.15.121:0/586200728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41140ffa40 0x7f4114195660 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:38.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.899+0000 7f411bb8f700 1 --2- 172.21.15.121:0/586200728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4114105a40 0x7f4114195ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:38.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.899+0000 7f411bb8f700 1 -- 172.21.15.121:0/586200728 >> 172.21.15.121:0/586200728 conn(0x7f41140fb570 msgr2=0x7f41140fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:38.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.900+0000 7f411bb8f700 1 -- 172.21.15.121:0/586200728 shutdown_connections 2024-04-05T21:33:38.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:38.900+0000 7f411bb8f700 1 -- 172.21.15.121:0/586200728 wait complete. 2024-04-05T21:33:38.911 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:33:39.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.004+0000 7f8f0da3e700 1 -- 172.21.15.121:0/3540184625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f08101da0 msgr2=0x7f8f08102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:39.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.004+0000 7f8f0da3e700 1 --2- 172.21.15.121:0/3540184625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f08101da0 0x7f8f08102180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0009910 tx=0x7f8ef0009c20 comp rx=0 tx=0).stop 2024-04-05T21:33:39.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.004+0000 7f8f0da3e700 1 -- 172.21.15.121:0/3540184625 shutdown_connections 2024-04-05T21:33:39.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.004+0000 7f8f0da3e700 1 --2- 172.21.15.121:0/3540184625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f081026c0 0x7f8f0810abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.004+0000 7f8f0da3e700 1 --2- 172.21.15.121:0/3540184625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f08101da0 0x7f8f08102180 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.004+0000 7f8f0da3e700 1 -- 172.21.15.121:0/3540184625 >> 172.21.15.121:0/3540184625 conn(0x7f8f080fb570 msgr2=0x7f8f080fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.004+0000 7f8f0da3e700 1 -- 172.21.15.121:0/3540184625 shutdown_connections 2024-04-05T21:33:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.004+0000 7f8f0da3e700 1 -- 172.21.15.121:0/3540184625 wait complete. 2024-04-05T21:33:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.005+0000 7f8f0da3e700 1 Processor -- start 2024-04-05T21:33:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.005+0000 7f8f0da3e700 1 -- start start 2024-04-05T21:33:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.005+0000 7f8f0da3e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f08101da0 0x7f8f08196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.005+0000 7f8f0da3e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f081026c0 0x7f8f08196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.005+0000 7f8f0da3e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f081971f0 con 0x7f8f081026c0 2024-04-05T21:33:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.005+0000 7f8f0da3e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f08197330 con 0x7f8f08101da0 2024-04-05T21:33:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.005+0000 7f8f067fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f081026c0 0x7f8f08196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.005+0000 7f8f067fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f081026c0 0x7f8f08196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56674/0 (socket says 172.21.15.121:56674) 2024-04-05T21:33:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.005+0000 7f8f067fc700 1 -- 172.21.15.121:0/243401659 learned_addr learned my addr 172.21.15.121:0/243401659 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.005+0000 7f8f067fc700 1 -- 172.21.15.121:0/243401659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f08101da0 msgr2=0x7f8f08196660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:39.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.005+0000 7f8f06ffd700 1 --2- 172.21.15.121:0/243401659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f08101da0 0x7f8f08196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:39.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.005+0000 7f8f067fc700 1 --2- 172.21.15.121:0/243401659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f08101da0 0x7f8f08196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.005+0000 7f8f067fc700 1 -- 172.21.15.121:0/243401659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef00095f0 con 0x7f8f081026c0 2024-04-05T21:33:39.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.006+0000 7f8f067fc700 1 --2- 172.21.15.121:0/243401659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f081026c0 0x7f8f08196ba0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800c960 tx=0x7f8ef800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:39.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.006+0000 7f8f0ca3c700 1 -- 172.21.15.121:0/243401659 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ef8007a70 con 0x7f8f081026c0 2024-04-05T21:33:39.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.006+0000 7f8f0ca3c700 1 -- 172.21.15.121:0/243401659 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ef8007bd0 con 0x7f8f081026c0 2024-04-05T21:33:39.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.006+0000 7f8f0da3e700 1 -- 172.21.15.121:0/243401659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0819b180 con 0x7f8f081026c0 2024-04-05T21:33:39.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.006+0000 7f8f0ca3c700 1 -- 172.21.15.121:0/243401659 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ef8018630 con 0x7f8f081026c0 2024-04-05T21:33:39.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.006+0000 7f8f0da3e700 1 -- 172.21.15.121:0/243401659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0819b6d0 con 0x7f8f081026c0 2024-04-05T21:33:39.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.007+0000 7f8f0da3e700 1 -- 172.21.15.121:0/243401659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f081082b0 con 0x7f8f081026c0 2024-04-05T21:33:39.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.007+0000 7f8f0ca3c700 1 -- 172.21.15.121:0/243401659 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8ef800f9f0 con 0x7f8f081026c0 2024-04-05T21:33:39.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.007+0000 7f8f0ca3c700 1 --2- 172.21.15.121:0/243401659 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8ef406cce0 0x7f8ef406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:39.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.008+0000 7f8f0ca3c700 1 -- 172.21.15.121:0/243401659 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f8ef808c530 con 0x7f8f081026c0 2024-04-05T21:33:39.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.008+0000 7f8f06ffd700 1 --2- 172.21.15.121:0/243401659 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8ef406cce0 0x7f8ef406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:39.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.008+0000 7f8f06ffd700 1 --2- 172.21.15.121:0/243401659 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8ef406cce0 0x7f8ef406f1a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000b5c0 tx=0x7f8ef0005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:39.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.012+0000 7f8f0ca3c700 1 -- 172.21.15.121:0/243401659 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8ef805a0e0 con 0x7f8f081026c0 2024-04-05T21:33:39.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.223+0000 7f8f0da3e700 1 -- 172.21.15.121:0/243401659 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f0802cd60 con 0x7f8ef406cce0 2024-04-05T21:33:39.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.224+0000 7f8f0ca3c700 1 -- 172.21.15.121:0/243401659 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f8f0802cd60 con 0x7f8ef406cce0 2024-04-05T21:33:39.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.226+0000 7f8f0da3e700 1 -- 172.21.15.121:0/243401659 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8ef406cce0 msgr2=0x7f8ef406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:39.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.226+0000 7f8f0da3e700 1 --2- 172.21.15.121:0/243401659 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8ef406cce0 0x7f8ef406f1a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000b5c0 tx=0x7f8ef0005fb0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.226+0000 7f8f0da3e700 1 -- 172.21.15.121:0/243401659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f081026c0 msgr2=0x7f8f08196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:39.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.226+0000 7f8f0da3e700 1 --2- 172.21.15.121:0/243401659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f081026c0 0x7f8f08196ba0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800c960 tx=0x7f8ef800cd20 comp rx=0 tx=0).stop 2024-04-05T21:33:39.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.227+0000 7f8f0da3e700 1 -- 172.21.15.121:0/243401659 shutdown_connections 2024-04-05T21:33:39.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.227+0000 7f8f0da3e700 1 --2- 172.21.15.121:0/243401659 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8ef406cce0 0x7f8ef406f1a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.227+0000 7f8f0da3e700 1 --2- 172.21.15.121:0/243401659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8f08101da0 0x7f8f08196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.227+0000 7f8f0da3e700 1 --2- 172.21.15.121:0/243401659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f081026c0 0x7f8f08196ba0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.227+0000 7f8f0da3e700 1 -- 172.21.15.121:0/243401659 >> 172.21.15.121:0/243401659 conn(0x7f8f080fb570 msgr2=0x7f8f081053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:39.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.227+0000 7f8f0da3e700 1 -- 172.21.15.121:0/243401659 shutdown_connections 2024-04-05T21:33:39.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.227+0000 7f8f0da3e700 1 -- 172.21.15.121:0/243401659 wait complete. 2024-04-05T21:33:39.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.332+0000 7fd845cd9700 1 -- 172.21.15.121:0/3059711335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8401017a0 msgr2=0x7fd8401057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:39.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.332+0000 7fd845cd9700 1 --2- 172.21.15.121:0/3059711335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8401017a0 0x7fd8401057f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd828009910 tx=0x7fd828009c20 comp rx=0 tx=0).stop 2024-04-05T21:33:39.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.332+0000 7fd845cd9700 1 -- 172.21.15.121:0/3059711335 shutdown_connections 2024-04-05T21:33:39.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.332+0000 7fd845cd9700 1 --2- 172.21.15.121:0/3059711335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8401017a0 0x7fd8401057f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.332+0000 7fd845cd9700 1 --2- 172.21.15.121:0/3059711335 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd840100df0 0x7fd8401011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.332+0000 7fd845cd9700 1 -- 172.21.15.121:0/3059711335 >> 172.21.15.121:0/3059711335 conn(0x7fd8400fc6a0 msgr2=0x7fd8400feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:39.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.332+0000 7fd845cd9700 1 -- 172.21.15.121:0/3059711335 shutdown_connections 2024-04-05T21:33:39.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.332+0000 7fd845cd9700 1 -- 172.21.15.121:0/3059711335 wait complete. 2024-04-05T21:33:39.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.332+0000 7fd845cd9700 1 Processor -- start 2024-04-05T21:33:39.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd845cd9700 1 -- start start 2024-04-05T21:33:39.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd845cd9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd840100df0 0x7fd840073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:39.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd845cd9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8401017a0 0x7fd8400735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:39.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd845cd9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd840073b10 con 0x7fd840100df0 2024-04-05T21:33:39.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd845cd9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd840073c50 con 0x7fd8401017a0 2024-04-05T21:33:39.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd83f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd840100df0 0x7fd840073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:39.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd83f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd840100df0 0x7fd840073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56688/0 (socket says 172.21.15.121:56688) 2024-04-05T21:33:39.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd83f7fe700 1 -- 172.21.15.121:0/3329495215 learned_addr learned my addr 172.21.15.121:0/3329495215 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:39.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd83effd700 1 --2- 172.21.15.121:0/3329495215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8401017a0 0x7fd8400735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:39.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd83f7fe700 1 -- 172.21.15.121:0/3329495215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8401017a0 msgr2=0x7fd8400735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:39.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd83f7fe700 1 --2- 172.21.15.121:0/3329495215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8401017a0 0x7fd8400735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd83f7fe700 1 -- 172.21.15.121:0/3329495215 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8280095f0 con 0x7fd840100df0 2024-04-05T21:33:39.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd83effd700 1 --2- 172.21.15.121:0/3329495215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8401017a0 0x7fd8400735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:33:39.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd83f7fe700 1 --2- 172.21.15.121:0/3329495215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd840100df0 0x7fd840073090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd8380060b0 tx=0x7fd83800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:39.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.333+0000 7fd83cff9700 1 -- 172.21.15.121:0/3329495215 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd83800dda0 con 0x7fd840100df0 2024-04-05T21:33:39.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.334+0000 7fd845cd9700 1 -- 172.21.15.121:0/3329495215 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8401a28b0 con 0x7fd840100df0 2024-04-05T21:33:39.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.334+0000 7fd83cff9700 1 -- 172.21.15.121:0/3329495215 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd83800f040 con 0x7fd840100df0 2024-04-05T21:33:39.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.334+0000 7fd83cff9700 1 -- 172.21.15.121:0/3329495215 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8380146f0 con 0x7fd840100df0 2024-04-05T21:33:39.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.334+0000 7fd845cd9700 1 -- 172.21.15.121:0/3329495215 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8401a2cc0 con 0x7fd840100df0 2024-04-05T21:33:39.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.334+0000 7fd845cd9700 1 -- 172.21.15.121:0/3329495215 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8401091f0 con 0x7fd840100df0 2024-04-05T21:33:39.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.335+0000 7fd83cff9700 1 -- 172.21.15.121:0/3329495215 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd838015400 con 0x7fd840100df0 2024-04-05T21:33:39.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.335+0000 7fd83cff9700 1 --2- 172.21.15.121:0/3329495215 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd82c06cce0 0x7fd82c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:39.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.336+0000 7fd83effd700 1 --2- 172.21.15.121:0/3329495215 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd82c06cce0 0x7fd82c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:39.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.336+0000 7fd83cff9700 1 -- 172.21.15.121:0/3329495215 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fd83808bfd0 con 0x7fd840100df0 2024-04-05T21:33:39.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.336+0000 7fd83effd700 1 --2- 172.21.15.121:0/3329495215 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd82c06cce0 0x7fd82c06f1a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd840074880 tx=0x7fd82801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:39.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.340+0000 7fd83cff9700 1 -- 172.21.15.121:0/3329495215 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd8380561a0 con 0x7fd840100df0 2024-04-05T21:33:39.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.540+0000 7fd845cd9700 1 -- 172.21.15.121:0/3329495215 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd84002cdd0 con 0x7fd82c06cce0 2024-04-05T21:33:39.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.551+0000 7fd83cff9700 1 -- 172.21.15.121:0/3329495215 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd84002cdd0 con 0x7fd82c06cce0 2024-04-05T21:33:39.552 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:33:39.552 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (4m) 52s ago 15m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:33:39.552 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (16m) 52s ago 16m 8284k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:33:39.552 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (15m) 17s ago 15m 8246k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:33:39.552 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (2m) 52s ago 16m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:33:39.552 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (2m) 17s ago 14m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:33:39.552 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (4m) 52s ago 15m 82.7M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:33:39.553 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (9m) 52s ago 9m 64.5M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:33:39.553 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (9m) 52s ago 9m 133M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:33:39.553 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (9m) 17s ago 9m 110M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:33:39.553 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (9m) 17s ago 9m 62.4M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:33:39.553 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (6m) 52s ago 17m 507M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:33:39.553 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (5m) 17s ago 14m 452M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:33:39.553 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (3m) 52s ago 17m 43.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:33:39.553 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (2m) 17s ago 14m 37.2M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:33:39.553 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (5m) 52s ago 16m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:33:39.553 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (5m) 17s ago 14m 16.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:33:39.553 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (2m) 52s ago 13m 88.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:33:39.553 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (94s) 52s ago 13m 99.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:33:39.553 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (58s) 52s ago 12m 11.2M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:33:39.554 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (21s) 17s ago 12m 11.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:33:39.554 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (11m) 17s ago 11m 245M 2251M 18.2.1 5be31c24972a d6d9a0a0ec5b 2024-04-05T21:33:39.554 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (11m) 17s ago 11m 221M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:33:39.554 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (4m) 52s ago 15m 65.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:33:39.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.553+0000 7fd845cd9700 1 -- 172.21.15.121:0/3329495215 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd82c06cce0 msgr2=0x7fd82c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:39.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.553+0000 7fd845cd9700 1 --2- 172.21.15.121:0/3329495215 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd82c06cce0 0x7fd82c06f1a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd840074880 tx=0x7fd82801a040 comp rx=0 tx=0).stop 2024-04-05T21:33:39.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.553+0000 7fd845cd9700 1 -- 172.21.15.121:0/3329495215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd840100df0 msgr2=0x7fd840073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:39.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.553+0000 7fd845cd9700 1 --2- 172.21.15.121:0/3329495215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd840100df0 0x7fd840073090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd8380060b0 tx=0x7fd83800d3b0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.553+0000 7fd845cd9700 1 -- 172.21.15.121:0/3329495215 shutdown_connections 2024-04-05T21:33:39.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.553+0000 7fd845cd9700 1 --2- 172.21.15.121:0/3329495215 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd82c06cce0 0x7fd82c06f1a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.553+0000 7fd845cd9700 1 --2- 172.21.15.121:0/3329495215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd840100df0 0x7fd840073090 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.553+0000 7fd845cd9700 1 --2- 172.21.15.121:0/3329495215 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8401017a0 0x7fd8400735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.553+0000 7fd845cd9700 1 -- 172.21.15.121:0/3329495215 >> 172.21.15.121:0/3329495215 conn(0x7fd8400fc6a0 msgr2=0x7fd8400fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:39.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.553+0000 7fd845cd9700 1 -- 172.21.15.121:0/3329495215 shutdown_connections 2024-04-05T21:33:39.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.553+0000 7fd845cd9700 1 -- 172.21.15.121:0/3329495215 wait complete. 2024-04-05T21:33:39.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.654+0000 7f9dacf9a700 1 -- 172.21.15.121:0/1613961009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80ff270 msgr2=0x7f9da80ff650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:39.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.654+0000 7f9dacf9a700 1 --2- 172.21.15.121:0/1613961009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80ff270 0x7f9da80ff650 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9d98009910 tx=0x7f9d98009c20 comp rx=0 tx=0).stop 2024-04-05T21:33:39.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.655+0000 7f9dacf9a700 1 -- 172.21.15.121:0/1613961009 shutdown_connections 2024-04-05T21:33:39.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.655+0000 7f9dacf9a700 1 --2- 172.21.15.121:0/1613961009 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9da80ffc20 0x7f9da81057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.655+0000 7f9dacf9a700 1 --2- 172.21.15.121:0/1613961009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80ff270 0x7f9da80ff650 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.655+0000 7f9dacf9a700 1 -- 172.21.15.121:0/1613961009 >> 172.21.15.121:0/1613961009 conn(0x7f9da8075e30 msgr2=0x7f9da80fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:39.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.655+0000 7f9dacf9a700 1 -- 172.21.15.121:0/1613961009 shutdown_connections 2024-04-05T21:33:39.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.655+0000 7f9dacf9a700 1 -- 172.21.15.121:0/1613961009 wait complete. 2024-04-05T21:33:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.655+0000 7f9dacf9a700 1 Processor -- start 2024-04-05T21:33:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.655+0000 7f9dacf9a700 1 -- start start 2024-04-05T21:33:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9dacf9a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80ff270 0x7f9da8196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9dacf9a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9da80ffc20 0x7f9da8196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9dacf9a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da8197250 con 0x7f9da80ff270 2024-04-05T21:33:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9dacf9a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da8199f80 con 0x7f9da80ffc20 2024-04-05T21:33:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9da659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80ff270 0x7f9da8196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9da659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80ff270 0x7f9da8196680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56696/0 (socket says 172.21.15.121:56696) 2024-04-05T21:33:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9da659c700 1 -- 172.21.15.121:0/2037639770 learned_addr learned my addr 172.21.15.121:0/2037639770 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9da5d9b700 1 --2- 172.21.15.121:0/2037639770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9da80ffc20 0x7f9da8196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9da659c700 1 -- 172.21.15.121:0/2037639770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9da80ffc20 msgr2=0x7f9da8196bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9da659c700 1 --2- 172.21.15.121:0/2037639770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9da80ffc20 0x7f9da8196bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9da659c700 1 -- 172.21.15.121:0/2037639770 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d980095f0 con 0x7f9da80ff270 2024-04-05T21:33:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9da659c700 1 --2- 172.21.15.121:0/2037639770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80ff270 0x7f9da8196680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d98000c00 tx=0x7f9d9800bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9d977fe700 1 -- 172.21.15.121:0/2037639770 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d9801d070 con 0x7f9da80ff270 2024-04-05T21:33:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9d977fe700 1 -- 172.21.15.121:0/2037639770 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d98005dc0 con 0x7f9da80ff270 2024-04-05T21:33:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9dacf9a700 1 -- 172.21.15.121:0/2037639770 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da819a200 con 0x7f9da80ff270 2024-04-05T21:33:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9d977fe700 1 -- 172.21.15.121:0/2037639770 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d98003940 con 0x7f9da80ff270 2024-04-05T21:33:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.656+0000 7f9dacf9a700 1 -- 172.21.15.121:0/2037639770 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da819a690 con 0x7f9da80ff270 2024-04-05T21:33:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.657+0000 7f9dacf9a700 1 -- 172.21.15.121:0/2037639770 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da8109130 con 0x7f9da80ff270 2024-04-05T21:33:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.658+0000 7f9d977fe700 1 -- 172.21.15.121:0/2037639770 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9d98004ae0 con 0x7f9da80ff270 2024-04-05T21:33:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.658+0000 7f9d977fe700 1 --2- 172.21.15.121:0/2037639770 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9d9006cce0 0x7f9d9006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.658+0000 7f9d977fe700 1 -- 172.21.15.121:0/2037639770 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f9d9808da20 con 0x7f9da80ff270 2024-04-05T21:33:39.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.658+0000 7f9da5d9b700 1 --2- 172.21.15.121:0/2037639770 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9d9006cce0 0x7f9d9006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:39.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.659+0000 7f9da5d9b700 1 --2- 172.21.15.121:0/2037639770 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9d9006cce0 0x7f9d9006f1a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9da8197ca0 tx=0x7f9d9c009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:39.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.663+0000 7f9d977fe700 1 -- 172.21.15.121:0/2037639770 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9d98057b40 con 0x7f9da80ff270 2024-04-05T21:33:39.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:39 smithi121 ceph-mon[145859]: pgmap v207: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T21:33:39.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:39 smithi188 ceph-mon[107793]: pgmap v207: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T21:33:39.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.933+0000 7f9dacf9a700 1 -- 172.21.15.121:0/2037639770 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9da804ea90 con 0x7f9da80ff270 2024-04-05T21:33:39.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.933+0000 7f9d977fe700 1 -- 172.21.15.121:0/2037639770 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+799 (secure 0 0 0) 0x7f9d9805b160 con 0x7f9da80ff270 2024-04-05T21:33:39.934 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:33:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 6, 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 8 2024-04-05T21:33:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:33:39.936 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:33:39.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.935+0000 7f9dacf9a700 1 -- 172.21.15.121:0/2037639770 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9d9006cce0 msgr2=0x7f9d9006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:39.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.935+0000 7f9dacf9a700 1 --2- 172.21.15.121:0/2037639770 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9d9006cce0 0x7f9d9006f1a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9da8197ca0 tx=0x7f9d9c009260 comp rx=0 tx=0).stop 2024-04-05T21:33:39.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.935+0000 7f9dacf9a700 1 -- 172.21.15.121:0/2037639770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80ff270 msgr2=0x7f9da8196680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:39.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.935+0000 7f9dacf9a700 1 --2- 172.21.15.121:0/2037639770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80ff270 0x7f9da8196680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d98000c00 tx=0x7f9d9800bde0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.935+0000 7f9dacf9a700 1 -- 172.21.15.121:0/2037639770 shutdown_connections 2024-04-05T21:33:39.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.935+0000 7f9dacf9a700 1 --2- 172.21.15.121:0/2037639770 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9d9006cce0 0x7f9d9006f1a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.935+0000 7f9dacf9a700 1 --2- 172.21.15.121:0/2037639770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80ff270 0x7f9da8196680 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.935+0000 7f9dacf9a700 1 --2- 172.21.15.121:0/2037639770 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9da80ffc20 0x7f9da8196bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:39.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.935+0000 7f9dacf9a700 1 -- 172.21.15.121:0/2037639770 >> 172.21.15.121:0/2037639770 conn(0x7f9da8075e30 msgr2=0x7f9da80fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:39.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.935+0000 7f9dacf9a700 1 -- 172.21.15.121:0/2037639770 shutdown_connections 2024-04-05T21:33:39.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:39.935+0000 7f9dacf9a700 1 -- 172.21.15.121:0/2037639770 wait complete. 2024-04-05T21:33:40.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.037+0000 7f5f5c51c700 1 -- 172.21.15.121:0/4165084872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f54100da0 msgr2=0x7f5f54101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:40.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.037+0000 7f5f5c51c700 1 --2- 172.21.15.121:0/4165084872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f54100da0 0x7f5f54101180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400b0a0 tx=0x7f5f4400b3b0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.037+0000 7f5f5c51c700 1 -- 172.21.15.121:0/4165084872 shutdown_connections 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.037+0000 7f5f5c51c700 1 --2- 172.21.15.121:0/4165084872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f54101750 0x7f5f541057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.037+0000 7f5f5c51c700 1 --2- 172.21.15.121:0/4165084872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f54100da0 0x7f5f54101180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.037+0000 7f5f5c51c700 1 -- 172.21.15.121:0/4165084872 >> 172.21.15.121:0/4165084872 conn(0x7f5f540fc630 msgr2=0x7f5f540fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.037+0000 7f5f5c51c700 1 -- 172.21.15.121:0/4165084872 shutdown_connections 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.037+0000 7f5f5c51c700 1 -- 172.21.15.121:0/4165084872 wait complete. 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f5c51c700 1 Processor -- start 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f5c51c700 1 -- start start 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f5c51c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f54100da0 0x7f5f54196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f5c51c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f54101750 0x7f5f54196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f5c51c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f54197260 con 0x7f5f54101750 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f5c51c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f54199f90 con 0x7f5f54100da0 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f59ab7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f54101750 0x7f5f54196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f59ab7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f54101750 0x7f5f54196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56710/0 (socket says 172.21.15.121:56710) 2024-04-05T21:33:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f59ab7700 1 -- 172.21.15.121:0/3505501093 learned_addr learned my addr 172.21.15.121:0/3505501093 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f5a2b8700 1 --2- 172.21.15.121:0/3505501093 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f54100da0 0x7f5f54196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f59ab7700 1 -- 172.21.15.121:0/3505501093 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f54100da0 msgr2=0x7f5f54196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f59ab7700 1 --2- 172.21.15.121:0/3505501093 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f54100da0 0x7f5f54196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f59ab7700 1 -- 172.21.15.121:0/3505501093 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f44009d00 con 0x7f5f54101750 2024-04-05T21:33:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f5a2b8700 1 --2- 172.21.15.121:0/3505501093 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f54100da0 0x7f5f54196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:33:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f59ab7700 1 --2- 172.21.15.121:0/3505501093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f54101750 0x7f5f54196bd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800d900 tx=0x7f5f4800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.038+0000 7f5f537fe700 1 -- 172.21.15.121:0/3505501093 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f48009900 con 0x7f5f54101750 2024-04-05T21:33:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.039+0000 7f5f537fe700 1 -- 172.21.15.121:0/3505501093 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f48010460 con 0x7f5f54101750 2024-04-05T21:33:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.039+0000 7f5f5c51c700 1 -- 172.21.15.121:0/3505501093 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5419a270 con 0x7f5f54101750 2024-04-05T21:33:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.039+0000 7f5f537fe700 1 -- 172.21.15.121:0/3505501093 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f4800f5f0 con 0x7f5f54101750 2024-04-05T21:33:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.039+0000 7f5f5c51c700 1 -- 172.21.15.121:0/3505501093 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5419a7c0 con 0x7f5f54101750 2024-04-05T21:33:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.039+0000 7f5f5c51c700 1 -- 172.21.15.121:0/3505501093 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f54109140 con 0x7f5f54101750 2024-04-05T21:33:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.040+0000 7f5f537fe700 1 -- 172.21.15.121:0/3505501093 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5f480105d0 con 0x7f5f54101750 2024-04-05T21:33:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.040+0000 7f5f537fe700 1 --2- 172.21.15.121:0/3505501093 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5f3c06cce0 0x7f5f3c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.040+0000 7f5f537fe700 1 -- 172.21.15.121:0/3505501093 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f5f4808c310 con 0x7f5f54101750 2024-04-05T21:33:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.040+0000 7f5f5a2b8700 1 --2- 172.21.15.121:0/3505501093 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5f3c06cce0 0x7f5f3c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.041+0000 7f5f5a2b8700 1 --2- 172.21.15.121:0/3505501093 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5f3c06cce0 0x7f5f3c06f1a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400b830 tx=0x7f5f44006210 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:40.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.045+0000 7f5f537fe700 1 -- 172.21.15.121:0/3505501093 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5f48056430 con 0x7f5f54101750 2024-04-05T21:33:40.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.284+0000 7f5f5c51c700 1 -- 172.21.15.121:0/3505501093 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5f5404ea90 con 0x7f5f54101750 2024-04-05T21:33:40.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.284+0000 7f5f537fe700 1 -- 172.21.15.121:0/3505501093 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f5f48059a50 con 0x7f5f54101750 2024-04-05T21:33:40.286 INFO:teuthology.orchestra.run.smithi121.stdout:e21 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 21 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:30:52.246236+0000 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:33:40.287 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 46 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:up {0=14476,1=24255} 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:failed 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:33:40.288 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:33:40.289 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:33:40.289 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:33:40.289 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:33:40.289 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:33:40.289 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{0:14476} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:33:40.289 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{0:14488} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/4279084130,v1:172.21.15.121:6827/4279084130] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:33:40.289 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{1:24255} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:33:40.289 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{1:24265} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:33:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:33:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:33:40.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.286+0000 7f5f5c51c700 1 -- 172.21.15.121:0/3505501093 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5f3c06cce0 msgr2=0x7f5f3c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:40.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.286+0000 7f5f5c51c700 1 --2- 172.21.15.121:0/3505501093 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5f3c06cce0 0x7f5f3c06f1a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400b830 tx=0x7f5f44006210 comp rx=0 tx=0).stop 2024-04-05T21:33:40.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.286+0000 7f5f5c51c700 1 -- 172.21.15.121:0/3505501093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f54101750 msgr2=0x7f5f54196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:40.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.286+0000 7f5f5c51c700 1 --2- 172.21.15.121:0/3505501093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f54101750 0x7f5f54196bd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800d900 tx=0x7f5f4800dcc0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.286+0000 7f5f5c51c700 1 -- 172.21.15.121:0/3505501093 shutdown_connections 2024-04-05T21:33:40.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.286+0000 7f5f5c51c700 1 --2- 172.21.15.121:0/3505501093 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5f3c06cce0 0x7f5f3c06f1a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.286+0000 7f5f5c51c700 1 --2- 172.21.15.121:0/3505501093 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f54100da0 0x7f5f54196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.286+0000 7f5f5c51c700 1 --2- 172.21.15.121:0/3505501093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f54101750 0x7f5f54196bd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.286+0000 7f5f5c51c700 1 -- 172.21.15.121:0/3505501093 >> 172.21.15.121:0/3505501093 conn(0x7f5f540fc630 msgr2=0x7f5f540fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:40.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.286+0000 7f5f5c51c700 1 -- 172.21.15.121:0/3505501093 shutdown_connections 2024-04-05T21:33:40.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.287+0000 7f5f5c51c700 1 -- 172.21.15.121:0/3505501093 wait complete. 2024-04-05T21:33:40.291 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 21 2024-04-05T21:33:40.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.391+0000 7f361eb17700 1 -- 172.21.15.121:0/792686669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36181026a0 msgr2=0x7f361810ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:40.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.391+0000 7f361eb17700 1 --2- 172.21.15.121:0/792686669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36181026a0 0x7f361810ab90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3604009940 tx=0x7f3604009c50 comp rx=0 tx=0).stop 2024-04-05T21:33:40.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.391+0000 7f361eb17700 1 -- 172.21.15.121:0/792686669 shutdown_connections 2024-04-05T21:33:40.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.391+0000 7f361eb17700 1 --2- 172.21.15.121:0/792686669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36181026a0 0x7f361810ab90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.391+0000 7f361eb17700 1 --2- 172.21.15.121:0/792686669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3618101d80 0x7f3618102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.391+0000 7f361eb17700 1 -- 172.21.15.121:0/792686669 >> 172.21.15.121:0/792686669 conn(0x7f36180fb570 msgr2=0x7f36180fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:40.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.391+0000 7f361eb17700 1 -- 172.21.15.121:0/792686669 shutdown_connections 2024-04-05T21:33:40.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.391+0000 7f361eb17700 1 -- 172.21.15.121:0/792686669 wait complete. 2024-04-05T21:33:40.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.391+0000 7f361eb17700 1 Processor -- start 2024-04-05T21:33:40.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.391+0000 7f361eb17700 1 -- start start 2024-04-05T21:33:40.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f361eb17700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3618101d80 0x7f36181966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:40.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f361eb17700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36181026a0 0x7f3618196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f361eb17700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3618197280 con 0x7f36181026a0 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f361eb17700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3618199fb0 con 0x7f3618101d80 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f3617fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36181026a0 0x7f3618196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f3617fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36181026a0 0x7f3618196bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56732/0 (socket says 172.21.15.121:56732) 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f3617fff700 1 -- 172.21.15.121:0/3570485115 learned_addr learned my addr 172.21.15.121:0/3570485115 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f361c8b3700 1 --2- 172.21.15.121:0/3570485115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3618101d80 0x7f36181966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f3617fff700 1 -- 172.21.15.121:0/3570485115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3618101d80 msgr2=0x7f36181966b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f3617fff700 1 --2- 172.21.15.121:0/3570485115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3618101d80 0x7f36181966b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f3617fff700 1 -- 172.21.15.121:0/3570485115 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36040095f0 con 0x7f36181026a0 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f3617fff700 1 --2- 172.21.15.121:0/3570485115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36181026a0 0x7f3618196bf0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f36040059d0 tx=0x7f36040055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f3615ffb700 1 -- 172.21.15.121:0/3570485115 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f360401d070 con 0x7f36181026a0 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f3615ffb700 1 -- 172.21.15.121:0/3570485115 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f360400f460 con 0x7f36181026a0 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f3615ffb700 1 -- 172.21.15.121:0/3570485115 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f360400bee0 con 0x7f36181026a0 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.392+0000 7f361eb17700 1 -- 172.21.15.121:0/3570485115 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361819a230 con 0x7f36181026a0 2024-04-05T21:33:40.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.393+0000 7f361eb17700 1 -- 172.21.15.121:0/3570485115 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361819a720 con 0x7f36181026a0 2024-04-05T21:33:40.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.394+0000 7f361eb17700 1 -- 172.21.15.121:0/3570485115 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3618108290 con 0x7f36181026a0 2024-04-05T21:33:40.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.394+0000 7f3615ffb700 1 -- 172.21.15.121:0/3570485115 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f360400fa20 con 0x7f36181026a0 2024-04-05T21:33:40.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.395+0000 7f3615ffb700 1 --2- 172.21.15.121:0/3570485115 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f360806cce0 0x7f360806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:40.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.395+0000 7f3615ffb700 1 -- 172.21.15.121:0/3570485115 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f360408d660 con 0x7f36181026a0 2024-04-05T21:33:40.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.395+0000 7f361c8b3700 1 --2- 172.21.15.121:0/3570485115 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f360806cce0 0x7f360806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:40.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.395+0000 7f361c8b3700 1 --2- 172.21.15.121:0/3570485115 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f360806cce0 0x7f360806f1a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f360c005950 tx=0x7f360c00a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:40.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.399+0000 7f3615ffb700 1 -- 172.21.15.121:0/3570485115 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3604057800 con 0x7f36181026a0 2024-04-05T21:33:40.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.625+0000 7f361eb17700 1 -- 172.21.15.121:0/3570485115 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f361819aa50 con 0x7f360806cce0 2024-04-05T21:33:40.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.626+0000 7f3615ffb700 1 -- 172.21.15.121:0/3570485115 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f361819aa50 con 0x7f360806cce0 2024-04-05T21:33:40.628 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:33:40.628 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:33:40.628 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:33:40.628 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:33:40.628 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:33:40.628 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:33:40.628 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:33:40.628 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:33:40.628 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:33:40.628 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "10/23 daemons upgraded", 2024-04-05T21:33:40.628 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading osd daemons", 2024-04-05T21:33:40.629 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:33:40.629 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:33:40.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.628+0000 7f361eb17700 1 -- 172.21.15.121:0/3570485115 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f360806cce0 msgr2=0x7f360806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:40.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.628+0000 7f361eb17700 1 --2- 172.21.15.121:0/3570485115 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f360806cce0 0x7f360806f1a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f360c005950 tx=0x7f360c00a300 comp rx=0 tx=0).stop 2024-04-05T21:33:40.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.628+0000 7f361eb17700 1 -- 172.21.15.121:0/3570485115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36181026a0 msgr2=0x7f3618196bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:40.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.628+0000 7f361eb17700 1 --2- 172.21.15.121:0/3570485115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36181026a0 0x7f3618196bf0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f36040059d0 tx=0x7f36040055f0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.628+0000 7f361eb17700 1 -- 172.21.15.121:0/3570485115 shutdown_connections 2024-04-05T21:33:40.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.628+0000 7f361eb17700 1 --2- 172.21.15.121:0/3570485115 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f360806cce0 0x7f360806f1a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.628+0000 7f361eb17700 1 --2- 172.21.15.121:0/3570485115 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3618101d80 0x7f36181966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.628+0000 7f361eb17700 1 --2- 172.21.15.121:0/3570485115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36181026a0 0x7f3618196bf0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.628+0000 7f361eb17700 1 -- 172.21.15.121:0/3570485115 >> 172.21.15.121:0/3570485115 conn(0x7f36180fb570 msgr2=0x7f36181053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:40.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.628+0000 7f361eb17700 1 -- 172.21.15.121:0/3570485115 shutdown_connections 2024-04-05T21:33:40.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.628+0000 7f361eb17700 1 -- 172.21.15.121:0/3570485115 wait complete. 2024-04-05T21:33:40.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.730+0000 7efebd9fd700 1 -- 172.21.15.121:0/3954743285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb8100fd0 msgr2=0x7efeb8104720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:40.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.730+0000 7efebd9fd700 1 --2- 172.21.15.121:0/3954743285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb8100fd0 0x7efeb8104720 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efea8009910 tx=0x7efea8009c20 comp rx=0 tx=0).stop 2024-04-05T21:33:40.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.730+0000 7efebd9fd700 1 -- 172.21.15.121:0/3954743285 shutdown_connections 2024-04-05T21:33:40.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.730+0000 7efebd9fd700 1 --2- 172.21.15.121:0/3954743285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb8100fd0 0x7efeb8104720 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.730+0000 7efebd9fd700 1 --2- 172.21.15.121:0/3954743285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efeb8100620 0x7efeb8100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.730+0000 7efebd9fd700 1 -- 172.21.15.121:0/3954743285 >> 172.21.15.121:0/3954743285 conn(0x7efeb8074e00 msgr2=0x7efeb8075210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:40.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.730+0000 7efebd9fd700 1 -- 172.21.15.121:0/3954743285 shutdown_connections 2024-04-05T21:33:40.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.730+0000 7efebd9fd700 1 -- 172.21.15.121:0/3954743285 wait complete. 2024-04-05T21:33:40.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.731+0000 7efebd9fd700 1 Processor -- start 2024-04-05T21:33:40.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.731+0000 7efebd9fd700 1 -- start start 2024-04-05T21:33:40.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.731+0000 7efebd9fd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb8100620 0x7efeb8196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:40.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.731+0000 7efebd9fd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efeb8100fd0 0x7efeb8196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:40.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.731+0000 7efebd9fd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb8197270 con 0x7efeb8100620 2024-04-05T21:33:40.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.731+0000 7efebd9fd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb819b000 con 0x7efeb8100fd0 2024-04-05T21:33:40.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.731+0000 7efeb6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb8100620 0x7efeb8196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:40.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.731+0000 7efeb6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb8100620 0x7efeb8196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56752/0 (socket says 172.21.15.121:56752) 2024-04-05T21:33:40.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.731+0000 7efeb6ffd700 1 -- 172.21.15.121:0/4068206648 learned_addr learned my addr 172.21.15.121:0/4068206648 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:33:40.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.731+0000 7efeb6ffd700 1 -- 172.21.15.121:0/4068206648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efeb8100fd0 msgr2=0x7efeb8196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:40.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.731+0000 7efeb6ffd700 1 --2- 172.21.15.121:0/4068206648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efeb8100fd0 0x7efeb8196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:40.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.732+0000 7efeb6ffd700 1 -- 172.21.15.121:0/4068206648 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea80095f0 con 0x7efeb8100620 2024-04-05T21:33:40.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.732+0000 7efeb6ffd700 1 --2- 172.21.15.121:0/4068206648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb8100620 0x7efeb8196650 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efea000d8d0 tx=0x7efea000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:40.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.732+0000 7efebc9fb700 1 -- 172.21.15.121:0/4068206648 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efea0009940 con 0x7efeb8100620 2024-04-05T21:33:40.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.732+0000 7efebd9fd700 1 -- 172.21.15.121:0/4068206648 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb819b2e0 con 0x7efeb8100620 2024-04-05T21:33:40.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.732+0000 7efebc9fb700 1 -- 172.21.15.121:0/4068206648 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efea0010460 con 0x7efeb8100620 2024-04-05T21:33:40.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.732+0000 7efebc9fb700 1 -- 172.21.15.121:0/4068206648 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efea000f5f0 con 0x7efeb8100620 2024-04-05T21:33:40.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.732+0000 7efebd9fd700 1 -- 172.21.15.121:0/4068206648 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb819b830 con 0x7efeb8100620 2024-04-05T21:33:40.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.733+0000 7efebd9fd700 1 -- 172.21.15.121:0/4068206648 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb8109010 con 0x7efeb8100620 2024-04-05T21:33:40.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.733+0000 7efebc9fb700 1 -- 172.21.15.121:0/4068206648 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7efea00105d0 con 0x7efeb8100620 2024-04-05T21:33:40.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.734+0000 7efebc9fb700 1 --2- 172.21.15.121:0/4068206648 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efea406cce0 0x7efea406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:33:40.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.734+0000 7efebc9fb700 1 -- 172.21.15.121:0/4068206648 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6156+0+0 (secure 0 0 0) 0x7efea008c530 con 0x7efeb8100620 2024-04-05T21:33:40.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.734+0000 7efeb67fc700 1 --2- 172.21.15.121:0/4068206648 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efea406cce0 0x7efea406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:33:40.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.734+0000 7efeb67fc700 1 --2- 172.21.15.121:0/4068206648 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efea406cce0 0x7efea406f1a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efea800b5c0 tx=0x7efea8005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:33:40.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:40 smithi121 ceph-mon[145859]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:40.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:40 smithi121 ceph-mon[145859]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:40.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:40 smithi121 ceph-mon[145859]: from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:40.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:40 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2037639770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:33:40.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:40 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:40.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:40 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3505501093' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:33:40.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:40.738+0000 7efebc9fb700 1 -- 172.21.15.121:0/4068206648 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efea0056650 con 0x7efeb8100620 2024-04-05T21:33:40.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:40 smithi188 ceph-mon[107793]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:40.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:40 smithi188 ceph-mon[107793]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:40.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:40 smithi188 ceph-mon[107793]: from='client.34262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:40.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:40 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2037639770' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:33:40.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:40 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:40.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:40 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3505501093' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:33:41.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:41.009+0000 7efebd9fd700 1 -- 172.21.15.121:0/4068206648 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7efeb804ea90 con 0x7efeb8100620 2024-04-05T21:33:41.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:41.009+0000 7efebc9fb700 1 -- 172.21.15.121:0/4068206648 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7efea0010880 con 0x7efeb8100620 2024-04-05T21:33:41.011 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_OK 2024-04-05T21:33:41.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:41.011+0000 7efebd9fd700 1 -- 172.21.15.121:0/4068206648 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efea406cce0 msgr2=0x7efea406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:41.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:41.011+0000 7efebd9fd700 1 --2- 172.21.15.121:0/4068206648 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efea406cce0 0x7efea406f1a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efea800b5c0 tx=0x7efea8005960 comp rx=0 tx=0).stop 2024-04-05T21:33:41.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:41.011+0000 7efebd9fd700 1 -- 172.21.15.121:0/4068206648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb8100620 msgr2=0x7efeb8196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:33:41.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:41.011+0000 7efebd9fd700 1 --2- 172.21.15.121:0/4068206648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb8100620 0x7efeb8196650 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efea000d8d0 tx=0x7efea000dc90 comp rx=0 tx=0).stop 2024-04-05T21:33:41.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:41.011+0000 7efebd9fd700 1 -- 172.21.15.121:0/4068206648 shutdown_connections 2024-04-05T21:33:41.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:41.011+0000 7efebd9fd700 1 --2- 172.21.15.121:0/4068206648 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efea406cce0 0x7efea406f1a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:41.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:41.011+0000 7efebd9fd700 1 --2- 172.21.15.121:0/4068206648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb8100620 0x7efeb8196650 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:41.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:41.011+0000 7efebd9fd700 1 --2- 172.21.15.121:0/4068206648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efeb8100fd0 0x7efeb8196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:33:41.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:41.011+0000 7efebd9fd700 1 -- 172.21.15.121:0/4068206648 >> 172.21.15.121:0/4068206648 conn(0x7efeb8074e00 msgr2=0x7efeb8103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:33:41.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:41.011+0000 7efebd9fd700 1 -- 172.21.15.121:0/4068206648 shutdown_connections 2024-04-05T21:33:41.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:33:41.011+0000 7efebd9fd700 1 -- 172.21.15.121:0/4068206648 wait complete. 2024-04-05T21:33:41.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:41 smithi121 ceph-mon[145859]: pgmap v208: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-05T21:33:41.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:41 smithi121 ceph-mon[145859]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:41.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:41 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4068206648' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:33:41.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:41 smithi188 ceph-mon[107793]: pgmap v208: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-04-05T21:33:41.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:41 smithi188 ceph-mon[107793]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:33:41.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:41 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4068206648' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:33:42.667 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:42 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T21:33:42.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:42 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T21:33:43.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:43 smithi121 ceph-mon[145859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T21:33:43.743 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:43 smithi121 ceph-mon[145859]: Upgrade: osd.4 is safe to restart 2024-04-05T21:33:43.743 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:43 smithi121 ceph-mon[145859]: pgmap v209: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T21:33:43.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:43 smithi188 ceph-mon[107793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-05T21:33:43.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:43 smithi188 ceph-mon[107793]: Upgrade: osd.4 is safe to restart 2024-04-05T21:33:43.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:43 smithi188 ceph-mon[107793]: pgmap v209: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T21:33:45.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:44 smithi121 ceph-mon[145859]: Upgrade: Updating osd.4 2024-04-05T21:33:45.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:44 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:45.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:44 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-05T21:33:45.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:44 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:33:45.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:44 smithi121 ceph-mon[145859]: Deploying daemon osd.4 on smithi188 2024-04-05T21:33:45.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:44 smithi121 ceph-mon[145859]: pgmap v210: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:33:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:44 smithi188 ceph-mon[107793]: Upgrade: Updating osd.4 2024-04-05T21:33:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:44 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:45.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:44 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-05T21:33:45.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:44 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:33:45.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:44 smithi188 ceph-mon[107793]: Deploying daemon osd.4 on smithi188 2024-04-05T21:33:45.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:44 smithi188 ceph-mon[107793]: pgmap v210: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:33:46.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:46 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:46.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:46 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:33:46.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:46 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:46.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:46 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:33:47.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:47 smithi121 ceph-mon[145859]: pgmap v211: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:33:47.495 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:47 smithi188 ceph-mon[107793]: pgmap v211: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-05T21:33:47.495 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:47 smithi188 systemd[1]: Stopping Ceph osd.4 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:33:47.779 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:47 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[65356]: 2024-04-05T21:33:47.493+0000 7f3fc06e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T21:33:47.780 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:47 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[65356]: 2024-04-05T21:33:47.493+0000 7f3fc06e1700 -1 osd.4 68 *** Got signal Terminated *** 2024-04-05T21:33:47.780 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:47 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[65356]: 2024-04-05T21:33:47.493+0000 7f3fc06e1700 -1 osd.4 68 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T21:33:48.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:48 smithi121 ceph-mon[145859]: osd.4 marked itself down and dead 2024-04-05T21:33:48.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:48 smithi188 ceph-mon[107793]: osd.4 marked itself down and dead 2024-04-05T21:33:49.279 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:49 smithi188 bash[124574]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4 2024-04-05T21:33:49.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:49 smithi188 ceph-mon[107793]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T21:33:49.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:49 smithi188 ceph-mon[107793]: pgmap v212: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:33:49.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:49 smithi188 ceph-mon[107793]: osdmap e69: 6 total, 5 up, 6 in 2024-04-05T21:33:49.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:49 smithi121 ceph-mon[145859]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T21:33:49.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:49 smithi121 ceph-mon[145859]: pgmap v212: 65 pgs: 65 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:33:49.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:49 smithi121 ceph-mon[145859]: osdmap e69: 6 total, 5 up, 6 in 2024-04-05T21:33:50.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:50 smithi121 ceph-mon[145859]: osdmap e70: 6 total, 5 up, 6 in 2024-04-05T21:33:50.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:50 smithi188 ceph-mon[107793]: osdmap e70: 6 total, 5 up, 6 in 2024-04-05T21:33:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:51 smithi121 ceph-mon[145859]: pgmap v215: 65 pgs: 6 active+undersized+degraded, 8 peering, 3 stale+active+clean, 5 active+undersized, 43 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 25/333 objects degraded (7.508%) 2024-04-05T21:33:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:51 smithi121 ceph-mon[145859]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-05T21:33:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:51 smithi121 ceph-mon[145859]: Health check failed: Degraded data redundancy: 25/333 objects degraded (7.508%), 6 pgs degraded (PG_DEGRADED) 2024-04-05T21:33:51.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:51 smithi188 ceph-mon[107793]: pgmap v215: 65 pgs: 6 active+undersized+degraded, 8 peering, 3 stale+active+clean, 5 active+undersized, 43 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 25/333 objects degraded (7.508%) 2024-04-05T21:33:51.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:51 smithi188 ceph-mon[107793]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-05T21:33:51.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:51 smithi188 ceph-mon[107793]: Health check failed: Degraded data redundancy: 25/333 objects degraded (7.508%), 6 pgs degraded (PG_DEGRADED) 2024-04-05T21:33:51.530 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:51 smithi188 systemd[1]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.4.service: Succeeded. 2024-04-05T21:33:51.530 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:51 smithi188 systemd[1]: Stopped Ceph osd.4 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:33:51.530 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:51 smithi188 systemd[1]: Starting Ceph osd.4 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:33:52.530 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:52 smithi188 bash[124854]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-05T21:33:52.530 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:52 smithi188 bash[124854]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-05T21:33:52.530 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:52 smithi188 bash[124854]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-05T21:33:52.530 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:52 smithi188 bash[124854]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-05T21:33:52.530 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:52 smithi188 bash[124854]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-05T21:33:52.530 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:52 smithi188 bash[124854]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-05T21:33:52.530 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:52 smithi188 bash[124854]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-05T21:33:53.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:53 smithi121 ceph-mon[145859]: pgmap v216: 65 pgs: 12 active+undersized+degraded, 8 peering, 1 stale+active+clean, 12 active+undersized, 32 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 42/333 objects degraded (12.613%) 2024-04-05T21:33:53.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:53 smithi188 ceph-mon[107793]: pgmap v216: 65 pgs: 12 active+undersized+degraded, 8 peering, 1 stale+active+clean, 12 active+undersized, 32 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 42/333 objects degraded (12.613%) 2024-04-05T21:33:53.530 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:53 smithi188 ceph-osd[125170]: -- 172.21.15.188:0/1291749096 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559bf932b1e0 con 0x559bf92fac00 2024-04-05T21:33:53.982 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:53 smithi188 bash[125132]: e7985264d99e9faa6822babec9d17d9716d7f24eeb4056bf06eadc9d43c17889 2024-04-05T21:33:53.982 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:53 smithi188 systemd[1]: Started Ceph osd.4 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:33:54.280 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:53 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T21:33:53.980+0000 7f64fb4a47c0 -1 Falling back to public interface 2024-04-05T21:33:54.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:54 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:54.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:54 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:54.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:54 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:33:55.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:54 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:55.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:54 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:55.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:54 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:33:55.935 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:55 smithi188 ceph-mon[107793]: pgmap v217: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/333 objects degraded (16.817%) 2024-04-05T21:33:55.935 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:55 smithi188 ceph-mon[107793]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-05T21:33:55.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:55 smithi121 ceph-mon[145859]: pgmap v217: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/333 objects degraded (16.817%) 2024-04-05T21:33:55.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:55 smithi121 ceph-mon[145859]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-05T21:33:57.973 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:57 smithi188 ceph-mon[107793]: pgmap v218: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/333 objects degraded (16.817%) 2024-04-05T21:33:57.974 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:57.974 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:57.974 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:57 smithi188 ceph-mon[107793]: Health check update: Degraded data redundancy: 56/333 objects degraded (16.817%), 16 pgs degraded (PG_DEGRADED) 2024-04-05T21:33:57.974 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:57.974 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:57.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:57 smithi121 ceph-mon[145859]: pgmap v218: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/333 objects degraded (16.817%) 2024-04-05T21:33:57.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:57.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:57.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:57 smithi121 ceph-mon[145859]: Health check update: Degraded data redundancy: 56/333 objects degraded (16.817%), 16 pgs degraded (PG_DEGRADED) 2024-04-05T21:33:57.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:57.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:33:59.030 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:58 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T21:33:58.567+0000 7f64fb4a47c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-05T21:33:59.339 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:59 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T21:33:59.082+0000 7f64fb4a47c0 -1 osd.4 68 log_to_monitors true 2024-04-05T21:33:59.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:59 smithi121 ceph-mon[145859]: pgmap v219: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/333 objects degraded (16.817%) 2024-04-05T21:33:59.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:59 smithi121 ceph-mon[145859]: from='osd.4 [v2:172.21.15.188:6808/3966070828,v1:172.21.15.188:6809/3966070828]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T21:33:59.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:33:59 smithi121 ceph-mon[145859]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T21:34:00.029 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 21:33:59 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T21:33:59.746+0000 7f64ec8fe700 -1 osd.4 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T21:34:00.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:59 smithi188 ceph-mon[107793]: pgmap v219: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 56/333 objects degraded (16.817%) 2024-04-05T21:34:00.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:59 smithi188 ceph-mon[107793]: from='osd.4 [v2:172.21.15.188:6808/3966070828,v1:172.21.15.188:6809/3966070828]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T21:34:00.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:33:59 smithi188 ceph-mon[107793]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-05T21:34:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:00 smithi121 ceph-mon[145859]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-05T21:34:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:00 smithi121 ceph-mon[145859]: osdmap e71: 6 total, 5 up, 6 in 2024-04-05T21:34:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:00 smithi121 ceph-mon[145859]: from='osd.4 [v2:172.21.15.188:6808/3966070828,v1:172.21.15.188:6809/3966070828]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:34:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:00 smithi121 ceph-mon[145859]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:34:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:34:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:00 smithi188 ceph-mon[107793]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-05T21:34:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:00 smithi188 ceph-mon[107793]: osdmap e71: 6 total, 5 up, 6 in 2024-04-05T21:34:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:00 smithi188 ceph-mon[107793]: from='osd.4 [v2:172.21.15.188:6808/3966070828,v1:172.21.15.188:6809/3966070828]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:34:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:00 smithi188 ceph-mon[107793]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:34:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:34:02.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: pgmap v221: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 56/333 objects degraded (16.817%) 2024-04-05T21:34:02.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T21:34:02.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: osd.4 [v2:172.21.15.188:6808/3966070828,v1:172.21.15.188:6809/3966070828] boot 2024-04-05T21:34:02.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: osdmap e72: 6 total, 6 up, 6 in 2024-04-05T21:34:02.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:34:02.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:02.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:02.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:34:02.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:34:02.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:02.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:02.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:02.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:02.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:02.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T21:34:02.031 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:01 smithi188 ceph-mon[107793]: osdmap e73: 6 total, 6 up, 6 in 2024-04-05T21:34:02.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: pgmap v221: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 56/333 objects degraded (16.817%) 2024-04-05T21:34:02.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T21:34:02.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: osd.4 [v2:172.21.15.188:6808/3966070828,v1:172.21.15.188:6809/3966070828] boot 2024-04-05T21:34:02.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: osdmap e72: 6 total, 6 up, 6 in 2024-04-05T21:34:02.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-05T21:34:02.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:02.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:02.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:34:02.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:34:02.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:02.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:02.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:02.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:02.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:02.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T21:34:02.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:01 smithi121 ceph-mon[145859]: osdmap e73: 6 total, 6 up, 6 in 2024-04-05T21:34:03.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:02 smithi188 ceph-mon[107793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T21:34:03.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:02 smithi188 ceph-mon[107793]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-05T21:34:03.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:02 smithi121 ceph-mon[145859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T21:34:03.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:02 smithi121 ceph-mon[145859]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-04-05T21:34:04.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:03 smithi188 ceph-mon[107793]: pgmap v224: 65 pgs: 11 peering, 11 active+undersized+degraded, 12 active+undersized, 31 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 41/333 objects degraded (12.312%) 2024-04-05T21:34:04.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:03 smithi188 ceph-mon[107793]: Health check update: Degraded data redundancy: 41/333 objects degraded (12.312%), 11 pgs degraded (PG_DEGRADED) 2024-04-05T21:34:04.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:03 smithi121 ceph-mon[145859]: pgmap v224: 65 pgs: 11 peering, 11 active+undersized+degraded, 12 active+undersized, 31 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 41/333 objects degraded (12.312%) 2024-04-05T21:34:04.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:03 smithi121 ceph-mon[145859]: Health check update: Degraded data redundancy: 41/333 objects degraded (12.312%), 11 pgs degraded (PG_DEGRADED) 2024-04-05T21:34:06.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:05 smithi188 ceph-mon[107793]: pgmap v225: 65 pgs: 11 peering, 8 active+undersized+degraded, 8 active+undersized, 38 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 29/333 objects degraded (8.709%) 2024-04-05T21:34:06.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:05 smithi121 ceph-mon[145859]: pgmap v225: 65 pgs: 11 peering, 8 active+undersized+degraded, 8 active+undersized, 38 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 29/333 objects degraded (8.709%) 2024-04-05T21:34:07.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:06 smithi188 ceph-mon[107793]: pgmap v226: 65 pgs: 7 peering, 58 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 4 op/s 2024-04-05T21:34:07.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:06 smithi121 ceph-mon[145859]: pgmap v226: 65 pgs: 7 peering, 58 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 4 op/s 2024-04-05T21:34:08.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:07 smithi188 ceph-mon[107793]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/333 objects degraded (8.709%), 8 pgs degraded) 2024-04-05T21:34:08.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:07 smithi188 ceph-mon[107793]: Cluster is now healthy 2024-04-05T21:34:08.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:07 smithi121 ceph-mon[145859]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/333 objects degraded (8.709%), 8 pgs degraded) 2024-04-05T21:34:08.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:07 smithi121 ceph-mon[145859]: Cluster is now healthy 2024-04-05T21:34:09.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:08 smithi188 ceph-mon[107793]: pgmap v227: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T21:34:09.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:08 smithi121 ceph-mon[145859]: pgmap v227: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T21:34:11.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.124+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/112564999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd58101790 msgr2=0x7fdd581057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:11.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.124+0000 7fdd5ca5b700 1 --2- 172.21.15.121:0/112564999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd58101790 0x7fdd581057e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdd48009910 tx=0x7fdd48009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:11.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.125+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/112564999 shutdown_connections 2024-04-05T21:34:11.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.125+0000 7fdd5ca5b700 1 --2- 172.21.15.121:0/112564999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd58101790 0x7fdd581057e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.125+0000 7fdd5ca5b700 1 --2- 172.21.15.121:0/112564999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd58100de0 0x7fdd581011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.125+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/112564999 >> 172.21.15.121:0/112564999 conn(0x7fdd580fc6b0 msgr2=0x7fdd580fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:11.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.125+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/112564999 shutdown_connections 2024-04-05T21:34:11.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.125+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/112564999 wait complete. 2024-04-05T21:34:11.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.125+0000 7fdd5ca5b700 1 Processor -- start 2024-04-05T21:34:11.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.125+0000 7fdd5ca5b700 1 -- start start 2024-04-05T21:34:11.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd5ca5b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd58100de0 0x7fdd58196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:11.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd5ca5b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd58101790 0x7fdd58196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:11.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd5ca5b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd58197290 con 0x7fdd58100de0 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd5ca5b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5819b020 con 0x7fdd58101790 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd5659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd58100de0 0x7fdd58196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd5659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd58100de0 0x7fdd58196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45720/0 (socket says 172.21.15.121:45720) 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd5659c700 1 -- 172.21.15.121:0/1249133851 learned_addr learned my addr 172.21.15.121:0/1249133851 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd55d9b700 1 --2- 172.21.15.121:0/1249133851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd58101790 0x7fdd58196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd5659c700 1 -- 172.21.15.121:0/1249133851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd58101790 msgr2=0x7fdd58196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd5659c700 1 --2- 172.21.15.121:0/1249133851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd58101790 0x7fdd58196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd5659c700 1 -- 172.21.15.121:0/1249133851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd480095f0 con 0x7fdd58100de0 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd5659c700 1 --2- 172.21.15.121:0/1249133851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd58100de0 0x7fdd58196670 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00d8d0 tx=0x7fdd4c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd477fe700 1 -- 172.21.15.121:0/1249133851 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd4c0098a0 con 0x7fdd58100de0 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd477fe700 1 -- 172.21.15.121:0/1249133851 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd4c010460 con 0x7fdd58100de0 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/1249133851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5819b300 con 0x7fdd58100de0 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd477fe700 1 -- 172.21.15.121:0/1249133851 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd4c009ea0 con 0x7fdd58100de0 2024-04-05T21:34:11.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.126+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/1249133851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5819b820 con 0x7fdd58100de0 2024-04-05T21:34:11.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.128+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/1249133851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd581091e0 con 0x7fdd58100de0 2024-04-05T21:34:11.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.128+0000 7fdd477fe700 1 -- 172.21.15.121:0/1249133851 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdd4c00f6c0 con 0x7fdd58100de0 2024-04-05T21:34:11.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.128+0000 7fdd477fe700 1 --2- 172.21.15.121:0/1249133851 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdd4006cce0 0x7fdd4006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:11.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.128+0000 7fdd477fe700 1 -- 172.21.15.121:0/1249133851 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fdd4c08c3d0 con 0x7fdd58100de0 2024-04-05T21:34:11.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.128+0000 7fdd55d9b700 1 --2- 172.21.15.121:0/1249133851 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdd4006cce0 0x7fdd4006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:11.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.129+0000 7fdd55d9b700 1 --2- 172.21.15.121:0/1249133851 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdd4006cce0 0x7fdd4006f1a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdd48006010 tx=0x7fdd4801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:11.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.133+0000 7fdd477fe700 1 -- 172.21.15.121:0/1249133851 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdd4c056560 con 0x7fdd58100de0 2024-04-05T21:34:11.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.345+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/1249133851 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd5819bae0 con 0x7fdd4006cce0 2024-04-05T21:34:11.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.346+0000 7fdd477fe700 1 -- 172.21.15.121:0/1249133851 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fdd5819bae0 con 0x7fdd4006cce0 2024-04-05T21:34:11.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.348+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/1249133851 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdd4006cce0 msgr2=0x7fdd4006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:11.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.348+0000 7fdd5ca5b700 1 --2- 172.21.15.121:0/1249133851 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdd4006cce0 0x7fdd4006f1a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdd48006010 tx=0x7fdd4801a040 comp rx=0 tx=0).stop 2024-04-05T21:34:11.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.348+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/1249133851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd58100de0 msgr2=0x7fdd58196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:11.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.348+0000 7fdd5ca5b700 1 --2- 172.21.15.121:0/1249133851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd58100de0 0x7fdd58196670 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00d8d0 tx=0x7fdd4c00dbe0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.349+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/1249133851 shutdown_connections 2024-04-05T21:34:11.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.349+0000 7fdd5ca5b700 1 --2- 172.21.15.121:0/1249133851 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdd4006cce0 0x7fdd4006f1a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.349+0000 7fdd5ca5b700 1 --2- 172.21.15.121:0/1249133851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd58100de0 0x7fdd58196670 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.349+0000 7fdd5ca5b700 1 --2- 172.21.15.121:0/1249133851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd58101790 0x7fdd58196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.349+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/1249133851 >> 172.21.15.121:0/1249133851 conn(0x7fdd580fc6b0 msgr2=0x7fdd580fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:11.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.349+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/1249133851 shutdown_connections 2024-04-05T21:34:11.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.349+0000 7fdd5ca5b700 1 -- 172.21.15.121:0/1249133851 wait complete. 2024-04-05T21:34:11.360 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:34:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.452+0000 7f4c8379b700 1 -- 172.21.15.121:0/3248599553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c7c105a40 msgr2=0x7f4c7c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.452+0000 7f4c8379b700 1 --2- 172.21.15.121:0/3248599553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c7c105a40 0x7f4c7c105e20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009910 tx=0x7f4c78009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.452+0000 7f4c8379b700 1 -- 172.21.15.121:0/3248599553 shutdown_connections 2024-04-05T21:34:11.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.452+0000 7f4c8379b700 1 --2- 172.21.15.121:0/3248599553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c7c0ffbd0 0x7f4c7c100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.452+0000 7f4c8379b700 1 --2- 172.21.15.121:0/3248599553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c7c105a40 0x7f4c7c105e20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.452+0000 7f4c8379b700 1 -- 172.21.15.121:0/3248599553 >> 172.21.15.121:0/3248599553 conn(0x7f4c7c0755a0 msgr2=0x7f4c7c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.452+0000 7f4c8379b700 1 -- 172.21.15.121:0/3248599553 shutdown_connections 2024-04-05T21:34:11.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.452+0000 7f4c8379b700 1 -- 172.21.15.121:0/3248599553 wait complete. 2024-04-05T21:34:11.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.453+0000 7f4c8379b700 1 Processor -- start 2024-04-05T21:34:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.453+0000 7f4c8379b700 1 -- start start 2024-04-05T21:34:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.453+0000 7f4c8379b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c7c0ffbd0 0x7f4c7c101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.453+0000 7f4c8379b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c7c105a40 0x7f4c7c101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.453+0000 7f4c8379b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c102120 con 0x7f4c7c105a40 2024-04-05T21:34:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.453+0000 7f4c8379b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7c102290 con 0x7f4c7c0ffbd0 2024-04-05T21:34:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.453+0000 7f4c80d36700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c7c105a40 0x7f4c7c101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.453+0000 7f4c80d36700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c7c105a40 0x7f4c7c101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45736/0 (socket says 172.21.15.121:45736) 2024-04-05T21:34:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.453+0000 7f4c80d36700 1 -- 172.21.15.121:0/797044361 learned_addr learned my addr 172.21.15.121:0/797044361 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.453+0000 7f4c81537700 1 --2- 172.21.15.121:0/797044361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c7c0ffbd0 0x7f4c7c101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.454+0000 7f4c80d36700 1 -- 172.21.15.121:0/797044361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c7c0ffbd0 msgr2=0x7f4c7c101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.454+0000 7f4c80d36700 1 --2- 172.21.15.121:0/797044361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c7c0ffbd0 0x7f4c7c101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.454+0000 7f4c80d36700 1 -- 172.21.15.121:0/797044361 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c780095f0 con 0x7f4c7c105a40 2024-04-05T21:34:11.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.454+0000 7f4c80d36700 1 --2- 172.21.15.121:0/797044361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c7c105a40 0x7f4c7c101b50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000d8d0 tx=0x7f4c7000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:11.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.454+0000 7f4c6e7fc700 1 -- 172.21.15.121:0/797044361 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c70009940 con 0x7f4c7c105a40 2024-04-05T21:34:11.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.454+0000 7f4c6e7fc700 1 -- 172.21.15.121:0/797044361 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c70010460 con 0x7f4c7c105a40 2024-04-05T21:34:11.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.454+0000 7f4c8379b700 1 -- 172.21.15.121:0/797044361 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7c1a2810 con 0x7f4c7c105a40 2024-04-05T21:34:11.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.454+0000 7f4c6e7fc700 1 -- 172.21.15.121:0/797044361 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c7000f5f0 con 0x7f4c7c105a40 2024-04-05T21:34:11.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.454+0000 7f4c8379b700 1 -- 172.21.15.121:0/797044361 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c7c1a2c80 con 0x7f4c7c105a40 2024-04-05T21:34:11.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.455+0000 7f4c8379b700 1 -- 172.21.15.121:0/797044361 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c7c109050 con 0x7f4c7c105a40 2024-04-05T21:34:11.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.455+0000 7f4c6e7fc700 1 -- 172.21.15.121:0/797044361 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4c700105d0 con 0x7f4c7c105a40 2024-04-05T21:34:11.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.456+0000 7f4c6e7fc700 1 --2- 172.21.15.121:0/797044361 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4c6806cce0 0x7f4c6806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:11.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.456+0000 7f4c6e7fc700 1 -- 172.21.15.121:0/797044361 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f4c7008c440 con 0x7f4c7c105a40 2024-04-05T21:34:11.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.456+0000 7f4c81537700 1 --2- 172.21.15.121:0/797044361 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4c6806cce0 0x7f4c6806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:11.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.456+0000 7f4c81537700 1 --2- 172.21.15.121:0/797044361 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4c6806cce0 0x7f4c6806f1a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4c78006010 tx=0x7f4c7801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:11.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.460+0000 7f4c6e7fc700 1 -- 172.21.15.121:0/797044361 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4c700565e0 con 0x7f4c7c105a40 2024-04-05T21:34:11.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:11 smithi188 ceph-mon[107793]: pgmap v228: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-05T21:34:11.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.670+0000 7f4c8379b700 1 -- 172.21.15.121:0/797044361 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c7c02cce0 con 0x7f4c6806cce0 2024-04-05T21:34:11.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.671+0000 7f4c6e7fc700 1 -- 172.21.15.121:0/797044361 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f4c7c02cce0 con 0x7f4c6806cce0 2024-04-05T21:34:11.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.673+0000 7f4c8379b700 1 -- 172.21.15.121:0/797044361 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4c6806cce0 msgr2=0x7f4c6806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:11.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.673+0000 7f4c8379b700 1 --2- 172.21.15.121:0/797044361 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4c6806cce0 0x7f4c6806f1a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4c78006010 tx=0x7f4c7801a040 comp rx=0 tx=0).stop 2024-04-05T21:34:11.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.673+0000 7f4c8379b700 1 -- 172.21.15.121:0/797044361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c7c105a40 msgr2=0x7f4c7c101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:11.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.673+0000 7f4c8379b700 1 --2- 172.21.15.121:0/797044361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c7c105a40 0x7f4c7c101b50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000d8d0 tx=0x7f4c7000dc90 comp rx=0 tx=0).stop 2024-04-05T21:34:11.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.673+0000 7f4c8379b700 1 -- 172.21.15.121:0/797044361 shutdown_connections 2024-04-05T21:34:11.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.673+0000 7f4c8379b700 1 --2- 172.21.15.121:0/797044361 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4c6806cce0 0x7f4c6806f1a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.673+0000 7f4c8379b700 1 --2- 172.21.15.121:0/797044361 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c7c0ffbd0 0x7f4c7c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.673+0000 7f4c8379b700 1 --2- 172.21.15.121:0/797044361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c7c105a40 0x7f4c7c101b50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.673+0000 7f4c8379b700 1 -- 172.21.15.121:0/797044361 >> 172.21.15.121:0/797044361 conn(0x7f4c7c0755a0 msgr2=0x7f4c7c068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:11.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.673+0000 7f4c8379b700 1 -- 172.21.15.121:0/797044361 shutdown_connections 2024-04-05T21:34:11.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.673+0000 7f4c8379b700 1 -- 172.21.15.121:0/797044361 wait complete. 2024-04-05T21:34:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:11 smithi121 ceph-mon[145859]: pgmap v228: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-05T21:34:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.777+0000 7f7ee8e97700 1 -- 172.21.15.121:0/2746204913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4101770 msgr2=0x7f7ee41057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.777+0000 7f7ee8e97700 1 --2- 172.21.15.121:0/2746204913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4101770 0x7f7ee41057c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009910 tx=0x7f7ed8009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.777+0000 7f7ee8e97700 1 -- 172.21.15.121:0/2746204913 shutdown_connections 2024-04-05T21:34:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.777+0000 7f7ee8e97700 1 --2- 172.21.15.121:0/2746204913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4101770 0x7f7ee41057c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.777+0000 7f7ee8e97700 1 --2- 172.21.15.121:0/2746204913 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ee4100dc0 0x7f7ee41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.777+0000 7f7ee8e97700 1 -- 172.21.15.121:0/2746204913 >> 172.21.15.121:0/2746204913 conn(0x7f7ee40fc650 msgr2=0x7f7ee40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.777+0000 7f7ee8e97700 1 -- 172.21.15.121:0/2746204913 shutdown_connections 2024-04-05T21:34:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.777+0000 7f7ee8e97700 1 -- 172.21.15.121:0/2746204913 wait complete. 2024-04-05T21:34:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.777+0000 7f7ee8e97700 1 Processor -- start 2024-04-05T21:34:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee8e97700 1 -- start start 2024-04-05T21:34:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee8e97700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ee4100dc0 0x7f7ee4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee8e97700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4101770 0x7f7ee40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee8e97700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee4073b10 con 0x7f7ee4101770 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee8e97700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee4073c50 con 0x7f7ee4100dc0 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee1d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4101770 0x7f7ee40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee1d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4101770 0x7f7ee40735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45758/0 (socket says 172.21.15.121:45758) 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee1d9b700 1 -- 172.21.15.121:0/1218302392 learned_addr learned my addr 172.21.15.121:0/1218302392 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee259c700 1 --2- 172.21.15.121:0/1218302392 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ee4100dc0 0x7f7ee4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee1d9b700 1 -- 172.21.15.121:0/1218302392 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ee4100dc0 msgr2=0x7f7ee4073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee1d9b700 1 --2- 172.21.15.121:0/1218302392 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ee4100dc0 0x7f7ee4073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee1d9b700 1 -- 172.21.15.121:0/1218302392 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed80095f0 con 0x7f7ee4101770 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee1d9b700 1 --2- 172.21.15.121:0/1218302392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4101770 0x7f7ee40735d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8000c00 tx=0x7f7ed800bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ed37fe700 1 -- 172.21.15.121:0/1218302392 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ed801d070 con 0x7f7ee4101770 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ee8e97700 1 -- 172.21.15.121:0/1218302392 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee4073ed0 con 0x7f7ee4101770 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ed37fe700 1 -- 172.21.15.121:0/1218302392 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ed8003cf0 con 0x7f7ee4101770 2024-04-05T21:34:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.778+0000 7f7ed37fe700 1 -- 172.21.15.121:0/1218302392 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ed8021410 con 0x7f7ee4101770 2024-04-05T21:34:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.779+0000 7f7ee8e97700 1 -- 172.21.15.121:0/1218302392 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee41a2b20 con 0x7f7ee4101770 2024-04-05T21:34:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.779+0000 7f7ee8e97700 1 -- 172.21.15.121:0/1218302392 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee41091c0 con 0x7f7ee4101770 2024-04-05T21:34:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.780+0000 7f7ed37fe700 1 -- 172.21.15.121:0/1218302392 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7ed8003850 con 0x7f7ee4101770 2024-04-05T21:34:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.780+0000 7f7ed37fe700 1 --2- 172.21.15.121:0/1218302392 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ecc06cce0 0x7f7ecc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.780+0000 7f7ed37fe700 1 -- 172.21.15.121:0/1218302392 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f7ed808d750 con 0x7f7ee4101770 2024-04-05T21:34:11.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.780+0000 7f7ee259c700 1 --2- 172.21.15.121:0/1218302392 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ecc06cce0 0x7f7ecc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:11.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.781+0000 7f7ee259c700 1 --2- 172.21.15.121:0/1218302392 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ecc06cce0 0x7f7ecc06f1a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009bf0 tx=0x7f7ed4009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:11.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.785+0000 7f7ed37fe700 1 -- 172.21.15.121:0/1218302392 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7ed8057870 con 0x7f7ee4101770 2024-04-05T21:34:11.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.986+0000 7f7ee8e97700 1 -- 172.21.15.121:0/1218302392 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ee402cd30 con 0x7f7ecc06cce0 2024-04-05T21:34:11.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.996+0000 7f7ed37fe700 1 -- 172.21.15.121:0/1218302392 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7ee402cd30 con 0x7f7ecc06cce0 2024-04-05T21:34:11.997 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:34:11.997 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (5m) 84s ago 16m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:34:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (16m) 84s ago 16m 8284k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:34:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (15m) 15s ago 15m 8288k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:34:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (3m) 84s ago 16m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:34:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (3m) 15s ago 15m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:34:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (4m) 84s ago 16m 82.7M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:34:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (10m) 84s ago 10m 64.5M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:34:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (10m) 84s ago 10m 133M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:34:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (10m) 15s ago 10m 110M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:34:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (10m) 15s ago 10m 62.2M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:34:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (7m) 84s ago 18m 507M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:34:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (6m) 15s ago 15m 452M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:34:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (3m) 84s ago 18m 43.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:34:11.998 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (3m) 15s ago 15m 38.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:34:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (5m) 84s ago 16m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:34:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (5m) 15s ago 15m 17.5M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:34:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (2m) 84s ago 14m 88.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:34:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (2m) 84s ago 13m 99.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:34:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (90s) 84s ago 13m 11.2M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:34:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (53s) 15s ago 12m 114M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:34:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (18s) 15s ago 12m 10.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:34:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (12m) 15s ago 12m 220M 2251M 18.2.1 5be31c24972a 33f8c15e9af5 2024-04-05T21:34:11.999 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (5m) 84s ago 15m 65.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:34:12.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.998+0000 7f7ee8e97700 1 -- 172.21.15.121:0/1218302392 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ecc06cce0 msgr2=0x7f7ecc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:12.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.998+0000 7f7ee8e97700 1 --2- 172.21.15.121:0/1218302392 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ecc06cce0 0x7f7ecc06f1a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009bf0 tx=0x7f7ed4009260 comp rx=0 tx=0).stop 2024-04-05T21:34:12.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.998+0000 7f7ee8e97700 1 -- 172.21.15.121:0/1218302392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4101770 msgr2=0x7f7ee40735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:12.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.998+0000 7f7ee8e97700 1 --2- 172.21.15.121:0/1218302392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4101770 0x7f7ee40735d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8000c00 tx=0x7f7ed800bb20 comp rx=0 tx=0).stop 2024-04-05T21:34:12.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.998+0000 7f7ee8e97700 1 -- 172.21.15.121:0/1218302392 shutdown_connections 2024-04-05T21:34:12.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.998+0000 7f7ee8e97700 1 --2- 172.21.15.121:0/1218302392 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ecc06cce0 0x7f7ecc06f1a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.998+0000 7f7ee8e97700 1 --2- 172.21.15.121:0/1218302392 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ee4100dc0 0x7f7ee4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.998+0000 7f7ee8e97700 1 --2- 172.21.15.121:0/1218302392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4101770 0x7f7ee40735d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.998+0000 7f7ee8e97700 1 -- 172.21.15.121:0/1218302392 >> 172.21.15.121:0/1218302392 conn(0x7f7ee40fc650 msgr2=0x7f7ee40fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:12.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.999+0000 7f7ee8e97700 1 -- 172.21.15.121:0/1218302392 shutdown_connections 2024-04-05T21:34:12.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:11.999+0000 7f7ee8e97700 1 -- 172.21.15.121:0/1218302392 wait complete. 2024-04-05T21:34:12.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.100+0000 7f0723d39700 1 -- 172.21.15.121:0/332620104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f071c100de0 msgr2=0x7f071c1011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:12.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.100+0000 7f0723d39700 1 --2- 172.21.15.121:0/332620104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f071c100de0 0x7f071c1011c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f070c009910 tx=0x7f070c009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:12.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.100+0000 7f0723d39700 1 -- 172.21.15.121:0/332620104 shutdown_connections 2024-04-05T21:34:12.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.100+0000 7f0723d39700 1 --2- 172.21.15.121:0/332620104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f071c101790 0x7f071c1057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.100+0000 7f0723d39700 1 --2- 172.21.15.121:0/332620104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f071c100de0 0x7f071c1011c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.100+0000 7f0723d39700 1 -- 172.21.15.121:0/332620104 >> 172.21.15.121:0/332620104 conn(0x7f071c0fc670 msgr2=0x7f071c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:12.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.100+0000 7f0723d39700 1 -- 172.21.15.121:0/332620104 shutdown_connections 2024-04-05T21:34:12.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.100+0000 7f0723d39700 1 -- 172.21.15.121:0/332620104 wait complete. 2024-04-05T21:34:12.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0723d39700 1 Processor -- start 2024-04-05T21:34:12.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0723d39700 1 -- start start 2024-04-05T21:34:12.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0723d39700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f071c100de0 0x7f071c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:12.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0723d39700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f071c101790 0x7f071c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:12.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0723d39700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c073b10 con 0x7f071c100de0 2024-04-05T21:34:12.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0723d39700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c073c50 con 0x7f071c101790 2024-04-05T21:34:12.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0721ad5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f071c100de0 0x7f071c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:12.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0721ad5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f071c100de0 0x7f071c073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45786/0 (socket says 172.21.15.121:45786) 2024-04-05T21:34:12.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0721ad5700 1 -- 172.21.15.121:0/1255525234 learned_addr learned my addr 172.21.15.121:0/1255525234 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:12.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f07212d4700 1 --2- 172.21.15.121:0/1255525234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f071c101790 0x7f071c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:12.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0721ad5700 1 -- 172.21.15.121:0/1255525234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f071c101790 msgr2=0x7f071c0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:12.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0721ad5700 1 --2- 172.21.15.121:0/1255525234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f071c101790 0x7f071c0735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0721ad5700 1 -- 172.21.15.121:0/1255525234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070c0095f0 con 0x7f071c100de0 2024-04-05T21:34:12.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0721ad5700 1 --2- 172.21.15.121:0/1255525234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f071c100de0 0x7f071c073090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f070c006010 tx=0x7f070c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:12.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.101+0000 7f0712ffd700 1 -- 172.21.15.121:0/1255525234 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f070c01d070 con 0x7f071c100de0 2024-04-05T21:34:12.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.102+0000 7f0723d39700 1 -- 172.21.15.121:0/1255525234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071c073ed0 con 0x7f071c100de0 2024-04-05T21:34:12.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.102+0000 7f0712ffd700 1 -- 172.21.15.121:0/1255525234 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f070c00f460 con 0x7f071c100de0 2024-04-05T21:34:12.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.102+0000 7f0712ffd700 1 -- 172.21.15.121:0/1255525234 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f070c021640 con 0x7f071c100de0 2024-04-05T21:34:12.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.102+0000 7f0723d39700 1 -- 172.21.15.121:0/1255525234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f071c1a2b70 con 0x7f071c100de0 2024-04-05T21:34:12.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.102+0000 7f0723d39700 1 -- 172.21.15.121:0/1255525234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071c109180 con 0x7f071c100de0 2024-04-05T21:34:12.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.103+0000 7f0712ffd700 1 -- 172.21.15.121:0/1255525234 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f070c00fa60 con 0x7f071c100de0 2024-04-05T21:34:12.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.103+0000 7f0712ffd700 1 --2- 172.21.15.121:0/1255525234 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f070806cce0 0x7f070806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:12.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.103+0000 7f0712ffd700 1 -- 172.21.15.121:0/1255525234 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f070c08d7b0 con 0x7f071c100de0 2024-04-05T21:34:12.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.103+0000 7f07212d4700 1 --2- 172.21.15.121:0/1255525234 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f070806cce0 0x7f070806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:12.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.104+0000 7f07212d4700 1 --2- 172.21.15.121:0/1255525234 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f070806cce0 0x7f070806f1a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0718005fd0 tx=0x7f071800a560 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:12.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.108+0000 7f0712ffd700 1 -- 172.21.15.121:0/1255525234 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f070c0578d0 con 0x7f071c100de0 2024-04-05T21:34:12.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.385+0000 7f0723d39700 1 -- 172.21.15.121:0/1255525234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f071c04ea90 con 0x7f071c100de0 2024-04-05T21:34:12.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.385+0000 7f0712ffd700 1 -- 172.21.15.121:0/1255525234 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+799 (secure 0 0 0) 0x7f070c05aef0 con 0x7f071c100de0 2024-04-05T21:34:12.387 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:34:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:34:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:34:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:34:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:34:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:34:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:34:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:34:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:34:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 5 2024-04-05T21:34:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:34:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:34:12.387 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 4 2024-04-05T21:34:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:34:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:34:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 5, 2024-04-05T21:34:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 9 2024-04-05T21:34:12.388 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:34:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:34:12.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.387+0000 7f0723d39700 1 -- 172.21.15.121:0/1255525234 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f070806cce0 msgr2=0x7f070806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:12.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.387+0000 7f0723d39700 1 --2- 172.21.15.121:0/1255525234 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f070806cce0 0x7f070806f1a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0718005fd0 tx=0x7f071800a560 comp rx=0 tx=0).stop 2024-04-05T21:34:12.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.387+0000 7f0723d39700 1 -- 172.21.15.121:0/1255525234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f071c100de0 msgr2=0x7f071c073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:12.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.387+0000 7f0723d39700 1 --2- 172.21.15.121:0/1255525234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f071c100de0 0x7f071c073090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f070c006010 tx=0x7f070c00be20 comp rx=0 tx=0).stop 2024-04-05T21:34:12.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.388+0000 7f0723d39700 1 -- 172.21.15.121:0/1255525234 shutdown_connections 2024-04-05T21:34:12.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.388+0000 7f0723d39700 1 --2- 172.21.15.121:0/1255525234 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f070806cce0 0x7f070806f1a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.388+0000 7f0723d39700 1 --2- 172.21.15.121:0/1255525234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f071c100de0 0x7f071c073090 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.388+0000 7f0723d39700 1 --2- 172.21.15.121:0/1255525234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f071c101790 0x7f071c0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.388+0000 7f0723d39700 1 -- 172.21.15.121:0/1255525234 >> 172.21.15.121:0/1255525234 conn(0x7f071c0fc670 msgr2=0x7f071c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:12.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.388+0000 7f0723d39700 1 -- 172.21.15.121:0/1255525234 shutdown_connections 2024-04-05T21:34:12.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.388+0000 7f0723d39700 1 -- 172.21.15.121:0/1255525234 wait complete. 2024-04-05T21:34:12.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.490+0000 7fbf3a56d700 1 -- 172.21.15.121:0/2371946371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf340ffbd0 msgr2=0x7fbf34100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:12.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.490+0000 7fbf3a56d700 1 --2- 172.21.15.121:0/2371946371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf340ffbd0 0x7fbf34100050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009910 tx=0x7fbf1c009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:12.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.490+0000 7fbf3a56d700 1 -- 172.21.15.121:0/2371946371 shutdown_connections 2024-04-05T21:34:12.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.490+0000 7fbf3a56d700 1 --2- 172.21.15.121:0/2371946371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf340ffbd0 0x7fbf34100050 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.490+0000 7fbf3a56d700 1 --2- 172.21.15.121:0/2371946371 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf34105a40 0x7fbf34105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.490+0000 7fbf3a56d700 1 -- 172.21.15.121:0/2371946371 >> 172.21.15.121:0/2371946371 conn(0x7fbf340755a0 msgr2=0x7fbf340759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:12.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.490+0000 7fbf3a56d700 1 -- 172.21.15.121:0/2371946371 shutdown_connections 2024-04-05T21:34:12.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.490+0000 7fbf3a56d700 1 -- 172.21.15.121:0/2371946371 wait complete. 2024-04-05T21:34:12.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.490+0000 7fbf3a56d700 1 Processor -- start 2024-04-05T21:34:12.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.490+0000 7fbf3a56d700 1 -- start start 2024-04-05T21:34:12.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.491+0000 7fbf3a56d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf340ffbd0 0x7fbf34195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.491+0000 7fbf3a56d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf34105a40 0x7fbf34195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.491+0000 7fbf3a56d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf34196160 con 0x7fbf340ffbd0 2024-04-05T21:34:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.491+0000 7fbf3a56d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf34199ef0 con 0x7fbf34105a40 2024-04-05T21:34:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.491+0000 7fbf33fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf340ffbd0 0x7fbf34195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.491+0000 7fbf33fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf340ffbd0 0x7fbf34195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45796/0 (socket says 172.21.15.121:45796) 2024-04-05T21:34:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.491+0000 7fbf33fff700 1 -- 172.21.15.121:0/810855847 learned_addr learned my addr 172.21.15.121:0/810855847 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.491+0000 7fbf337fe700 1 --2- 172.21.15.121:0/810855847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf34105a40 0x7fbf34195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.491+0000 7fbf33fff700 1 -- 172.21.15.121:0/810855847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf34105a40 msgr2=0x7fbf34195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.491+0000 7fbf33fff700 1 --2- 172.21.15.121:0/810855847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf34105a40 0x7fbf34195a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.491+0000 7fbf33fff700 1 -- 172.21.15.121:0/810855847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1c0095f0 con 0x7fbf340ffbd0 2024-04-05T21:34:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.491+0000 7fbf33fff700 1 --2- 172.21.15.121:0/810855847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf340ffbd0 0x7fbf34195540 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009970 tx=0x7fbf24009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.492+0000 7fbf317fa700 1 -- 172.21.15.121:0/810855847 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf24012070 con 0x7fbf340ffbd0 2024-04-05T21:34:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.492+0000 7fbf3a56d700 1 -- 172.21.15.121:0/810855847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3419a170 con 0x7fbf340ffbd0 2024-04-05T21:34:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.492+0000 7fbf317fa700 1 -- 172.21.15.121:0/810855847 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf24007ad0 con 0x7fbf340ffbd0 2024-04-05T21:34:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.492+0000 7fbf317fa700 1 -- 172.21.15.121:0/810855847 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf24017c00 con 0x7fbf340ffbd0 2024-04-05T21:34:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.492+0000 7fbf3a56d700 1 -- 172.21.15.121:0/810855847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3419a6c0 con 0x7fbf340ffbd0 2024-04-05T21:34:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.492+0000 7fbf3a56d700 1 -- 172.21.15.121:0/810855847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3404ea90 con 0x7fbf340ffbd0 2024-04-05T21:34:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.493+0000 7fbf317fa700 1 -- 172.21.15.121:0/810855847 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbf24007c40 con 0x7fbf340ffbd0 2024-04-05T21:34:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.493+0000 7fbf317fa700 1 --2- 172.21.15.121:0/810855847 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbf2006cce0 0x7fbf2006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:12.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.494+0000 7fbf317fa700 1 -- 172.21.15.121:0/810855847 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fbf2408db10 con 0x7fbf340ffbd0 2024-04-05T21:34:12.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.494+0000 7fbf337fe700 1 --2- 172.21.15.121:0/810855847 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbf2006cce0 0x7fbf2006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:12.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.494+0000 7fbf337fe700 1 --2- 172.21.15.121:0/810855847 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbf2006cce0 0x7fbf2006f1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbf34196b60 tx=0x7fbf1c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:12.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.498+0000 7fbf317fa700 1 -- 172.21.15.121:0/810855847 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbf24057ce0 con 0x7fbf340ffbd0 2024-04-05T21:34:12.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:12 smithi188 ceph-mon[107793]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:12.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:12 smithi188 ceph-mon[107793]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:12.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.733+0000 7fbf3a56d700 1 -- 172.21.15.121:0/810855847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbf3419abb0 con 0x7fbf340ffbd0 2024-04-05T21:34:12.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.734+0000 7fbf317fa700 1 -- 172.21.15.121:0/810855847 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1874 (secure 0 0 0) 0x7fbf2405b300 con 0x7fbf340ffbd0 2024-04-05T21:34:12.735 INFO:teuthology.orchestra.run.smithi121.stdout:e21 2024-04-05T21:34:12.735 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:34:12.735 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:34:12.735 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:34:12.735 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:34:12.735 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:34:12.735 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:34:12.735 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 21 2024-04-05T21:34:12.735 INFO:teuthology.orchestra.run.smithi121.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T21:34:12.735 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:34:12.735 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:30:52.246236+0000 2024-04-05T21:34:12.735 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 46 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:up {0=14476,1=24255} 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:failed 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:34:12.736 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:34:12.737 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:34:12.737 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:34:12.737 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:34:12.737 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:34:12.737 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:34:12.737 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{0:14476} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.121:6828/3760584056,v1:172.21.15.121:6829/3760584056] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:34:12.737 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{0:14488} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/4279084130,v1:172.21.15.121:6827/4279084130] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:34:12.737 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{1:24255} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.188:6824/213881583,v1:172.21.15.188:6825/213881583] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:34:12.737 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{1:24265} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.188:6826/3730442150,v1:172.21.15.188:6827/3730442150] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:34:12.737 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:34:12.737 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:34:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.736+0000 7fbf3a56d700 1 -- 172.21.15.121:0/810855847 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbf2006cce0 msgr2=0x7fbf2006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.736+0000 7fbf3a56d700 1 --2- 172.21.15.121:0/810855847 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbf2006cce0 0x7fbf2006f1a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbf34196b60 tx=0x7fbf1c01a040 comp rx=0 tx=0).stop 2024-04-05T21:34:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.736+0000 7fbf3a56d700 1 -- 172.21.15.121:0/810855847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf340ffbd0 msgr2=0x7fbf34195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.736+0000 7fbf3a56d700 1 --2- 172.21.15.121:0/810855847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf340ffbd0 0x7fbf34195540 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009970 tx=0x7fbf24009d30 comp rx=0 tx=0).stop 2024-04-05T21:34:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.736+0000 7fbf3a56d700 1 -- 172.21.15.121:0/810855847 shutdown_connections 2024-04-05T21:34:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.736+0000 7fbf3a56d700 1 --2- 172.21.15.121:0/810855847 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbf2006cce0 0x7fbf2006f1a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.736+0000 7fbf3a56d700 1 --2- 172.21.15.121:0/810855847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf340ffbd0 0x7fbf34195540 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.736+0000 7fbf3a56d700 1 --2- 172.21.15.121:0/810855847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbf34105a40 0x7fbf34195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.736+0000 7fbf3a56d700 1 -- 172.21.15.121:0/810855847 >> 172.21.15.121:0/810855847 conn(0x7fbf340755a0 msgr2=0x7fbf340fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.736+0000 7fbf3a56d700 1 -- 172.21.15.121:0/810855847 shutdown_connections 2024-04-05T21:34:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.736+0000 7fbf3a56d700 1 -- 172.21.15.121:0/810855847 wait complete. 2024-04-05T21:34:12.739 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 21 2024-04-05T21:34:12.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:12 smithi121 ceph-mon[145859]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:12.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:12 smithi121 ceph-mon[145859]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:12.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.839+0000 7f128432e700 1 -- 172.21.15.121:0/1183374277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c069020 msgr2=0x7f127c105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:12.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.839+0000 7f128432e700 1 --2- 172.21.15.121:0/1183374277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c069020 0x7f127c105790 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1278009910 tx=0x7f1278009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:12.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.839+0000 7f128432e700 1 -- 172.21.15.121:0/1183374277 shutdown_connections 2024-04-05T21:34:12.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.839+0000 7f128432e700 1 --2- 172.21.15.121:0/1183374277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c069020 0x7f127c105790 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.839+0000 7f128432e700 1 --2- 172.21.15.121:0/1183374277 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f127c068670 0x7f127c068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.839+0000 7f128432e700 1 -- 172.21.15.121:0/1183374277 >> 172.21.15.121:0/1183374277 conn(0x7f127c0755a0 msgr2=0x7f127c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:12.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.839+0000 7f128432e700 1 -- 172.21.15.121:0/1183374277 shutdown_connections 2024-04-05T21:34:12.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.839+0000 7f128432e700 1 -- 172.21.15.121:0/1183374277 wait complete. 2024-04-05T21:34:12.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.840+0000 7f128432e700 1 Processor -- start 2024-04-05T21:34:12.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.840+0000 7f128432e700 1 -- start start 2024-04-05T21:34:12.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.840+0000 7f128432e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c068670 0x7f127c0ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:12.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.840+0000 7f128432e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f127c069020 0x7f127c1001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:12.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.840+0000 7f128432e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c100720 con 0x7f127c068670 2024-04-05T21:34:12.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.840+0000 7f128432e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c100860 con 0x7f127c069020 2024-04-05T21:34:12.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.840+0000 7f12820ca700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c068670 0x7f127c0ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:12.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.840+0000 7f12820ca700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c068670 0x7f127c0ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45828/0 (socket says 172.21.15.121:45828) 2024-04-05T21:34:12.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.840+0000 7f12820ca700 1 -- 172.21.15.121:0/582880728 learned_addr learned my addr 172.21.15.121:0/582880728 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:12.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.840+0000 7f12818c9700 1 --2- 172.21.15.121:0/582880728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f127c069020 0x7f127c1001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:12.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.840+0000 7f12820ca700 1 -- 172.21.15.121:0/582880728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f127c069020 msgr2=0x7f127c1001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:12.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.840+0000 7f12820ca700 1 --2- 172.21.15.121:0/582880728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f127c069020 0x7f127c1001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:12.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.840+0000 7f12820ca700 1 -- 172.21.15.121:0/582880728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12780095f0 con 0x7f127c068670 2024-04-05T21:34:12.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.841+0000 7f12820ca700 1 --2- 172.21.15.121:0/582880728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c068670 0x7f127c0ffca0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f126c00c8a0 tx=0x7f126c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:12.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.841+0000 7f12737fe700 1 -- 172.21.15.121:0/582880728 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f126c00cea0 con 0x7f127c068670 2024-04-05T21:34:12.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.841+0000 7f128432e700 1 -- 172.21.15.121:0/582880728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f127c1a2810 con 0x7f127c068670 2024-04-05T21:34:12.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.841+0000 7f12737fe700 1 -- 172.21.15.121:0/582880728 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f126c004830 con 0x7f127c068670 2024-04-05T21:34:12.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.841+0000 7f12737fe700 1 -- 172.21.15.121:0/582880728 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f126c005500 con 0x7f127c068670 2024-04-05T21:34:12.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.841+0000 7f128432e700 1 -- 172.21.15.121:0/582880728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f127c1a2c20 con 0x7f127c068670 2024-04-05T21:34:12.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.842+0000 7f128432e700 1 -- 172.21.15.121:0/582880728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f127c1090a0 con 0x7f127c068670 2024-04-05T21:34:12.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.842+0000 7f12737fe700 1 -- 172.21.15.121:0/582880728 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f126c0049a0 con 0x7f127c068670 2024-04-05T21:34:12.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.843+0000 7f12737fe700 1 --2- 172.21.15.121:0/582880728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f126806cce0 0x7f126806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:12.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.843+0000 7f12737fe700 1 -- 172.21.15.121:0/582880728 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6156+0+0 (secure 0 0 0) 0x7f126c08c200 con 0x7f127c068670 2024-04-05T21:34:12.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.843+0000 7f12818c9700 1 --2- 172.21.15.121:0/582880728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f126806cce0 0x7f126806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:12.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.843+0000 7f12818c9700 1 --2- 172.21.15.121:0/582880728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f126806cce0 0x7f126806f1a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f127800b5c0 tx=0x7f1278005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:12.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:12.847+0000 7f12737fe700 1 -- 172.21.15.121:0/582880728 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f126c056320 con 0x7f127c068670 2024-04-05T21:34:13.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.055+0000 7f128432e700 1 -- 172.21.15.121:0/582880728 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f127c02cdd0 con 0x7f126806cce0 2024-04-05T21:34:13.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.057+0000 7f12737fe700 1 -- 172.21.15.121:0/582880728 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f127c02cdd0 con 0x7f126806cce0 2024-04-05T21:34:13.058 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:34:13.058 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:34:13.058 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:34:13.058 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:34:13.058 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:34:13.058 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:34:13.058 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:34:13.058 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:34:13.058 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:34:13.058 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "11/23 daemons upgraded", 2024-04-05T21:34:13.058 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading osd daemons", 2024-04-05T21:34:13.059 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:34:13.059 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:34:13.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.059+0000 7f128432e700 1 -- 172.21.15.121:0/582880728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f126806cce0 msgr2=0x7f126806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:13.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.059+0000 7f128432e700 1 --2- 172.21.15.121:0/582880728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f126806cce0 0x7f126806f1a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f127800b5c0 tx=0x7f1278005fb0 comp rx=0 tx=0).stop 2024-04-05T21:34:13.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.059+0000 7f128432e700 1 -- 172.21.15.121:0/582880728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c068670 msgr2=0x7f127c0ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:13.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.059+0000 7f128432e700 1 --2- 172.21.15.121:0/582880728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c068670 0x7f127c0ffca0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f126c00c8a0 tx=0x7f126c00cc60 comp rx=0 tx=0).stop 2024-04-05T21:34:13.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.059+0000 7f128432e700 1 -- 172.21.15.121:0/582880728 shutdown_connections 2024-04-05T21:34:13.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.059+0000 7f128432e700 1 --2- 172.21.15.121:0/582880728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f126806cce0 0x7f126806f1a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:13.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.059+0000 7f128432e700 1 --2- 172.21.15.121:0/582880728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c068670 0x7f127c0ffca0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:13.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.059+0000 7f128432e700 1 --2- 172.21.15.121:0/582880728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f127c069020 0x7f127c1001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:13.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.059+0000 7f128432e700 1 -- 172.21.15.121:0/582880728 >> 172.21.15.121:0/582880728 conn(0x7f127c0755a0 msgr2=0x7f127c105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:13.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.059+0000 7f128432e700 1 -- 172.21.15.121:0/582880728 shutdown_connections 2024-04-05T21:34:13.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.059+0000 7f128432e700 1 -- 172.21.15.121:0/582880728 wait complete. 2024-04-05T21:34:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.162+0000 7fcc8fb48700 1 -- 172.21.15.121:0/589313723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc88105b50 msgr2=0x7fcc88105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.162+0000 7fcc8fb48700 1 --2- 172.21.15.121:0/589313723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc88105b50 0x7fcc88105f30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcc78009910 tx=0x7fcc78009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.162+0000 7fcc8fb48700 1 -- 172.21.15.121:0/589313723 shutdown_connections 2024-04-05T21:34:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.162+0000 7fcc8fb48700 1 --2- 172.21.15.121:0/589313723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc880fef70 0x7fcc880ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.162+0000 7fcc8fb48700 1 --2- 172.21.15.121:0/589313723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc88105b50 0x7fcc88105f30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.162+0000 7fcc8fb48700 1 -- 172.21.15.121:0/589313723 >> 172.21.15.121:0/589313723 conn(0x7fcc88075e30 msgr2=0x7fcc880fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.162+0000 7fcc8fb48700 1 -- 172.21.15.121:0/589313723 shutdown_connections 2024-04-05T21:34:13.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.162+0000 7fcc8fb48700 1 -- 172.21.15.121:0/589313723 wait complete. 2024-04-05T21:34:13.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8fb48700 1 Processor -- start 2024-04-05T21:34:13.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8fb48700 1 -- start start 2024-04-05T21:34:13.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8fb48700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc880fef70 0x7fcc88100150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:13.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8fb48700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc88105b50 0x7fcc88100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:13.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8fb48700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc88100bd0 con 0x7fcc880fef70 2024-04-05T21:34:13.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8fb48700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc88100d10 con 0x7fcc88105b50 2024-04-05T21:34:13.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8d8e4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc880fef70 0x7fcc88100150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:13.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8d8e4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc880fef70 0x7fcc88100150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45854/0 (socket says 172.21.15.121:45854) 2024-04-05T21:34:13.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8d8e4700 1 -- 172.21.15.121:0/2067801393 learned_addr learned my addr 172.21.15.121:0/2067801393 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:13.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8d0e3700 1 --2- 172.21.15.121:0/2067801393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc88105b50 0x7fcc88100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:13.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8d8e4700 1 -- 172.21.15.121:0/2067801393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc88105b50 msgr2=0x7fcc88100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:13.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8d8e4700 1 --2- 172.21.15.121:0/2067801393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc88105b50 0x7fcc88100690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:13.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8d8e4700 1 -- 172.21.15.121:0/2067801393 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc780095f0 con 0x7fcc880fef70 2024-04-05T21:34:13.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8d8e4700 1 --2- 172.21.15.121:0/2067801393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc880fef70 0x7fcc88100150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcc78000c00 tx=0x7fcc7800be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:13.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc7effd700 1 -- 172.21.15.121:0/2067801393 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc7801d070 con 0x7fcc880fef70 2024-04-05T21:34:13.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8fb48700 1 -- 172.21.15.121:0/2067801393 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc88100f90 con 0x7fcc880fef70 2024-04-05T21:34:13.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc7effd700 1 -- 172.21.15.121:0/2067801393 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc780044c0 con 0x7fcc880fef70 2024-04-05T21:34:13.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc7effd700 1 -- 172.21.15.121:0/2067801393 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc780039d0 con 0x7fcc880fef70 2024-04-05T21:34:13.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.163+0000 7fcc8fb48700 1 -- 172.21.15.121:0/2067801393 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc881a2be0 con 0x7fcc880fef70 2024-04-05T21:34:13.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.164+0000 7fcc8fb48700 1 -- 172.21.15.121:0/2067801393 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc88109120 con 0x7fcc880fef70 2024-04-05T21:34:13.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.165+0000 7fcc7effd700 1 -- 172.21.15.121:0/2067801393 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcc780048e0 con 0x7fcc880fef70 2024-04-05T21:34:13.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.165+0000 7fcc7effd700 1 --2- 172.21.15.121:0/2067801393 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcc7406cce0 0x7fcc7406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:13.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.165+0000 7fcc7effd700 1 -- 172.21.15.121:0/2067801393 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6156+0+0 (secure 0 0 0) 0x7fcc7808db60 con 0x7fcc880fef70 2024-04-05T21:34:13.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.165+0000 7fcc8d0e3700 1 --2- 172.21.15.121:0/2067801393 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcc7406cce0 0x7fcc7406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:13.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.166+0000 7fcc8d0e3700 1 --2- 172.21.15.121:0/2067801393 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcc7406cce0 0x7fcc7406f1a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcc88101940 tx=0x7fcc84009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:13.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.170+0000 7fcc7effd700 1 -- 172.21.15.121:0/2067801393 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcc78057c80 con 0x7fcc880fef70 2024-04-05T21:34:13.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.445+0000 7fcc8fb48700 1 -- 172.21.15.121:0/2067801393 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcc8804ea90 con 0x7fcc880fef70 2024-04-05T21:34:13.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.446+0000 7fcc7effd700 1 -- 172.21.15.121:0/2067801393 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fcc78004be0 con 0x7fcc880fef70 2024-04-05T21:34:13.450 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_OK 2024-04-05T21:34:13.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.447+0000 7fcc8fb48700 1 -- 172.21.15.121:0/2067801393 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcc7406cce0 msgr2=0x7fcc7406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:13.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.447+0000 7fcc8fb48700 1 --2- 172.21.15.121:0/2067801393 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcc7406cce0 0x7fcc7406f1a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcc88101940 tx=0x7fcc84009260 comp rx=0 tx=0).stop 2024-04-05T21:34:13.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.447+0000 7fcc8fb48700 1 -- 172.21.15.121:0/2067801393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc880fef70 msgr2=0x7fcc88100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:13.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.447+0000 7fcc8fb48700 1 --2- 172.21.15.121:0/2067801393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc880fef70 0x7fcc88100150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcc78000c00 tx=0x7fcc7800be90 comp rx=0 tx=0).stop 2024-04-05T21:34:13.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.448+0000 7fcc8fb48700 1 -- 172.21.15.121:0/2067801393 shutdown_connections 2024-04-05T21:34:13.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.448+0000 7fcc8fb48700 1 --2- 172.21.15.121:0/2067801393 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcc7406cce0 0x7fcc7406f1a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:13.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.448+0000 7fcc8fb48700 1 --2- 172.21.15.121:0/2067801393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc880fef70 0x7fcc88100150 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:13.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.448+0000 7fcc8fb48700 1 --2- 172.21.15.121:0/2067801393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcc88105b50 0x7fcc88100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:13.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.448+0000 7fcc8fb48700 1 -- 172.21.15.121:0/2067801393 >> 172.21.15.121:0/2067801393 conn(0x7fcc88075e30 msgr2=0x7fcc88104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:13.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.448+0000 7fcc8fb48700 1 -- 172.21.15.121:0/2067801393 shutdown_connections 2024-04-05T21:34:13.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:13.448+0000 7fcc8fb48700 1 -- 172.21.15.121:0/2067801393 wait complete. 2024-04-05T21:34:13.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:13 smithi188 ceph-mon[107793]: from='client.34294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:13.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:13 smithi188 ceph-mon[107793]: pgmap v229: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:34:13.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:13 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1255525234' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:13.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:13 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/810855847' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:34:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:13 smithi121 ceph-mon[145859]: from='client.34294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:13 smithi121 ceph-mon[145859]: pgmap v229: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:34:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:13 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1255525234' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:13 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/810855847' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:34:14.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:14 smithi188 ceph-mon[107793]: from='client.34306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:14.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:14 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2067801393' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:34:14.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:14 smithi121 ceph-mon[145859]: from='client.34306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:14.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:14 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2067801393' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:34:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:15 smithi121 ceph-mon[145859]: pgmap v230: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-05T21:34:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:34:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:15 smithi188 ceph-mon[107793]: pgmap v230: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 3 op/s 2024-04-05T21:34:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:34:16.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:16.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:17.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:17 smithi188 ceph-mon[107793]: pgmap v231: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-05T21:34:17.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T21:34:17.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:17 smithi188 ceph-mon[107793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T21:34:17.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:17 smithi188 ceph-mon[107793]: Upgrade: osd.5 is safe to restart 2024-04-05T21:34:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:17 smithi121 ceph-mon[145859]: pgmap v231: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-05T21:34:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T21:34:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:17 smithi121 ceph-mon[145859]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-05T21:34:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:17 smithi121 ceph-mon[145859]: Upgrade: osd.5 is safe to restart 2024-04-05T21:34:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:18 smithi188 ceph-mon[107793]: Upgrade: Updating osd.5 2024-04-05T21:34:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:18 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:18 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-05T21:34:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:18 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:34:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:18 smithi188 ceph-mon[107793]: Deploying daemon osd.5 on smithi188 2024-04-05T21:34:19.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:18 smithi188 ceph-mon[107793]: pgmap v232: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:34:19.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:18 smithi121 ceph-mon[145859]: Upgrade: Updating osd.5 2024-04-05T21:34:19.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:18 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:19.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:18 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-05T21:34:19.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:18 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:34:19.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:18 smithi121 ceph-mon[145859]: Deploying daemon osd.5 on smithi188 2024-04-05T21:34:19.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:18 smithi121 ceph-mon[145859]: pgmap v232: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T21:34:21.421 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:21 smithi188 ceph-mon[107793]: pgmap v233: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:34:21.730 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:21 smithi188 systemd[1]: Stopping Ceph osd.5 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:34:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:21 smithi121 ceph-mon[145859]: pgmap v233: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T21:34:22.029 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:21 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[74343]: 2024-04-05T21:34:21.728+0000 7f9df1a13700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T21:34:22.030 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:21 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[74343]: 2024-04-05T21:34:21.728+0000 7f9df1a13700 -1 osd.5 73 *** Got signal Terminated *** 2024-04-05T21:34:22.030 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:21 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[74343]: 2024-04-05T21:34:21.728+0000 7f9df1a13700 -1 osd.5 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T21:34:22.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:22 smithi188 ceph-mon[107793]: osd.5 marked itself down and dead 2024-04-05T21:34:22.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:22 smithi121 ceph-mon[145859]: osd.5 marked itself down and dead 2024-04-05T21:34:23.529 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:23 smithi188 bash[130893]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5 2024-04-05T21:34:23.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:23 smithi188 ceph-mon[107793]: pgmap v234: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T21:34:23.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:23 smithi188 ceph-mon[107793]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T21:34:23.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:23 smithi188 ceph-mon[107793]: osdmap e74: 6 total, 5 up, 6 in 2024-04-05T21:34:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:23 smithi121 ceph-mon[145859]: pgmap v234: 65 pgs: 65 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T21:34:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:23 smithi121 ceph-mon[145859]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-05T21:34:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:23 smithi121 ceph-mon[145859]: osdmap e74: 6 total, 5 up, 6 in 2024-04-05T21:34:24.381 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:24 smithi188 ceph-mon[107793]: osdmap e75: 6 total, 5 up, 6 in 2024-04-05T21:34:24.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:24 smithi121 ceph-mon[145859]: osdmap e75: 6 total, 5 up, 6 in 2024-04-05T21:34:25.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:25 smithi121 ceph-mon[145859]: pgmap v237: 65 pgs: 3 peering, 11 stale+active+clean, 51 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s 2024-04-05T21:34:25.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:25 smithi121 ceph-mon[145859]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-05T21:34:25.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:25 smithi188 ceph-mon[107793]: pgmap v237: 65 pgs: 3 peering, 11 stale+active+clean, 51 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s 2024-04-05T21:34:25.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:25 smithi188 ceph-mon[107793]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-04-05T21:34:25.530 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:25 smithi188 systemd[1]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.5.service: Succeeded. 2024-04-05T21:34:25.530 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:25 smithi188 systemd[1]: Stopped Ceph osd.5 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:34:25.530 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:25 smithi188 systemd[1]: Starting Ceph osd.5 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T21:34:26.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:26 smithi121 ceph-mon[145859]: Health check failed: Degraded data redundancy: 36/333 objects degraded (10.811%), 9 pgs degraded (PG_DEGRADED) 2024-04-05T21:34:26.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:26 smithi188 ceph-mon[107793]: Health check failed: Degraded data redundancy: 36/333 objects degraded (10.811%), 9 pgs degraded (PG_DEGRADED) 2024-04-05T21:34:26.780 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:26 smithi188 bash[131168]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-05T21:34:26.780 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:26 smithi188 bash[131168]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-05T21:34:26.780 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:26 smithi188 bash[131168]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-05T21:34:26.780 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:26 smithi188 bash[131168]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-05T21:34:26.780 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:26 smithi188 bash[131168]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-05T21:34:26.780 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:26 smithi188 bash[131168]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-05T21:34:26.780 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:26 smithi188 bash[131168]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-05T21:34:27.667 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:27 smithi188 ceph-mon[107793]: pgmap v238: 65 pgs: 9 active+undersized+degraded, 3 peering, 4 stale+active+clean, 8 active+undersized, 41 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 36/333 objects degraded (10.811%) 2024-04-05T21:34:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:27 smithi121 ceph-mon[145859]: pgmap v238: 65 pgs: 9 active+undersized+degraded, 3 peering, 4 stale+active+clean, 8 active+undersized, 41 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 36/333 objects degraded (10.811%) 2024-04-05T21:34:28.029 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:27 smithi188 ceph-osd[131464]: -- 172.21.15.188:0/2534984094 <== mon.1 v2:172.21.15.188:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564283d931e0 con 0x564283d63000 2024-04-05T21:34:28.030 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:27 smithi188 bash[131426]: 62b26c18a78cebbeb9b40c31c6344d167bffad8a931abe34afeb13b856713ed0 2024-04-05T21:34:28.030 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:27 smithi188 systemd[1]: Started Ceph osd.5 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T21:34:28.529 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:28 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T21:34:28.191+0000 7f5773c937c0 -1 Falling back to public interface 2024-04-05T21:34:29.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:28 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:29.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:28 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:29.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:28 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:29.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:28 smithi188 ceph-mon[107793]: pgmap v239: 65 pgs: 14 active+undersized+degraded, 3 peering, 11 active+undersized, 37 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 50/333 objects degraded (15.015%) 2024-04-05T21:34:29.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:28 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:29.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:28 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:29.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:28 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:29.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:28 smithi121 ceph-mon[145859]: pgmap v239: 65 pgs: 14 active+undersized+degraded, 3 peering, 11 active+undersized, 37 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 50/333 objects degraded (15.015%) 2024-04-05T21:34:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:34:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:31 smithi121 ceph-mon[145859]: pgmap v240: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 50/333 objects degraded (15.015%) 2024-04-05T21:34:31.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:34:31.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:31 smithi188 ceph-mon[107793]: pgmap v240: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 50/333 objects degraded (15.015%) 2024-04-05T21:34:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:32 smithi121 ceph-mon[145859]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-05T21:34:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:32 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:32 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:32 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:32 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:32 smithi121 ceph-mon[145859]: Health check update: Degraded data redundancy: 50/333 objects degraded (15.015%), 14 pgs degraded (PG_DEGRADED) 2024-04-05T21:34:32.522 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:32 smithi188 ceph-mon[107793]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-04-05T21:34:32.522 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:32 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:32.522 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:32 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:32.522 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:32 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:32.522 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:32 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:32.522 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:32 smithi188 ceph-mon[107793]: Health check update: Degraded data redundancy: 50/333 objects degraded (15.015%), 14 pgs degraded (PG_DEGRADED) 2024-04-05T21:34:32.779 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:32 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T21:34:32.661+0000 7f5773c937c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-05T21:34:33.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:33 smithi188 ceph-mon[107793]: pgmap v241: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 50/333 objects degraded (15.015%) 2024-04-05T21:34:33.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:33 smithi188 ceph-mon[107793]: from='osd.5 [v2:172.21.15.188:6816/3805501159,v1:172.21.15.188:6817/3805501159]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T21:34:33.334 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:33 smithi188 ceph-mon[107793]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T21:34:33.334 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:33 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T21:34:33.074+0000 7f5773c937c0 -1 osd.5 73 log_to_monitors true 2024-04-05T21:34:33.334 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 21:34:33 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T21:34:33.221+0000 7f57650ed700 -1 osd.5 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-05T21:34:33.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:33 smithi121 ceph-mon[145859]: pgmap v241: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1.1 GiB used, 535 GiB / 536 GiB avail; 50/333 objects degraded (15.015%) 2024-04-05T21:34:33.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:33 smithi121 ceph-mon[145859]: from='osd.5 [v2:172.21.15.188:6816/3805501159,v1:172.21.15.188:6817/3805501159]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T21:34:33.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:33 smithi121 ceph-mon[145859]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-05T21:34:34.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:34 smithi188 ceph-mon[107793]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-05T21:34:34.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:34 smithi188 ceph-mon[107793]: osdmap e76: 6 total, 5 up, 6 in 2024-04-05T21:34:34.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:34 smithi188 ceph-mon[107793]: from='osd.5 [v2:172.21.15.188:6816/3805501159,v1:172.21.15.188:6817/3805501159]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:34:34.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:34 smithi188 ceph-mon[107793]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:34:34.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:34 smithi121 ceph-mon[145859]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-05T21:34:34.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:34 smithi121 ceph-mon[145859]: osdmap e76: 6 total, 5 up, 6 in 2024-04-05T21:34:34.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:34 smithi121 ceph-mon[145859]: from='osd.5 [v2:172.21.15.188:6816/3805501159,v1:172.21.15.188:6817/3805501159]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:34:34.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:34 smithi121 ceph-mon[145859]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi188", "root=default"]}]: dispatch 2024-04-05T21:34:35.416 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:35 smithi188 ceph-mon[107793]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T21:34:35.416 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:35 smithi188 ceph-mon[107793]: osd.5 [v2:172.21.15.188:6816/3805501159,v1:172.21.15.188:6817/3805501159] boot 2024-04-05T21:34:35.416 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:35 smithi188 ceph-mon[107793]: osdmap e77: 6 total, 6 up, 6 in 2024-04-05T21:34:35.416 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:34:35.416 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:35 smithi188 ceph-mon[107793]: pgmap v244: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 50/333 objects degraded (15.015%) 2024-04-05T21:34:35.416 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:35 smithi188 ceph-mon[107793]: osdmap e78: 6 total, 6 up, 6 in 2024-04-05T21:34:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:35 smithi121 ceph-mon[145859]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-05T21:34:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:35 smithi121 ceph-mon[145859]: osd.5 [v2:172.21.15.188:6816/3805501159,v1:172.21.15.188:6817/3805501159] boot 2024-04-05T21:34:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:35 smithi121 ceph-mon[145859]: osdmap e77: 6 total, 6 up, 6 in 2024-04-05T21:34:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-05T21:34:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:35 smithi121 ceph-mon[145859]: pgmap v244: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 50/333 objects degraded (15.015%) 2024-04-05T21:34:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:35 smithi121 ceph-mon[145859]: osdmap e78: 6 total, 6 up, 6 in 2024-04-05T21:34:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-05T21:34:36.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-05T21:34:36.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-05T21:34:36.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-05T21:34:36.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-05T21:34:36.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-05T21:34:36.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-05T21:34:36.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-05T21:34:36.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-05T21:34:36.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:36.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-05T21:34:36.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:36 smithi121 ceph-mon[145859]: osdmap e79: 6 total, 6 up, 6 in 2024-04-05T21:34:36.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:36.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:36.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:36.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:34:36.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:34:36.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:36.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:36.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:36.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:36.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:36.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:36.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:36.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-05T21:34:36.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-05T21:34:36.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-05T21:34:36.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-05T21:34:36.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-05T21:34:36.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-05T21:34:36.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-05T21:34:36.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-05T21:34:36.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-05T21:34:36.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-05T21:34:36.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-05T21:34:36.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-05T21:34:36.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:36.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-04-05T21:34:36.782 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:36 smithi188 ceph-mon[107793]: osdmap e79: 6 total, 6 up, 6 in 2024-04-05T21:34:37.288 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:34:37.155+0000 7f914dea6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T21:34:37.685 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: Upgrade: Setting container_image for all osd 2024-04-05T21:34:37.685 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-05T21:34:37.685 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: pgmap v247: 65 pgs: 10 peering, 7 active+undersized+degraded, 11 active+undersized, 37 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s; 20/333 objects degraded (6.006%) 2024-04-05T21:34:37.685 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-05T21:34:37.685 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-05T21:34:37.686 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2024-04-05T21:34:37.686 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: Health check update: Degraded data redundancy: 20/333 objects degraded (6.006%), 7 pgs degraded (PG_DEGRADED) 2024-04-05T21:34:37.686 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:37.686 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:34:37.686 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:34:37.686 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:37.686 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:37.686 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:37.686 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:37.686 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:37.686 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:37.686 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-05T21:34:37.686 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-05T21:34:37.687 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T21:34:37.687 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-05T21:34:37.687 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: osdmap e80: 6 total, 6 up, 6 in 2024-04-05T21:34:37.687 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] up:boot 2024-04-05T21:34:37.687 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.121:6826/1474300111,v1:172.21.15.121:6827/1474300111] up:boot 2024-04-05T21:34:37.687 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-05T21:34:37.687 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-05T21:34:37.687 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:34:37.687 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.latnft"}]: dispatch 2024-04-05T21:34:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: Upgrade: Setting container_image for all osd 2024-04-05T21:34:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-04-05T21:34:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: pgmap v247: 65 pgs: 10 peering, 7 active+undersized+degraded, 11 active+undersized, 37 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s; 20/333 objects degraded (6.006%) 2024-04-05T21:34:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-04-05T21:34:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-04-05T21:34:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: fsmap cephfs:2 {0=cephfs.smithi121.ifoxmw=up:active,1=cephfs.smithi188.bjmisu=up:active} 2024-04-05T21:34:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: Health check update: Degraded data redundancy: 20/333 objects degraded (6.006%), 7 pgs degraded (PG_DEGRADED) 2024-04-05T21:34:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:34:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:34:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: osdmap e80: 6 total, 6 up, 6 in 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] up:boot 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.121:6826/1474300111,v1:172.21.15.121:6827/1474300111] up:boot 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:34:37.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.latnft"}]: dispatch 2024-04-05T21:34:38.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:38 smithi121 ceph-mon[145859]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-05T21:34:38.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:38 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.188:6824/2257429686,v1:172.21.15.188:6825/2257429686] up:boot 2024-04-05T21:34:38.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:38 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.121:6828/1553603485,v1:172.21.15.121:6829/1553603485] up:boot 2024-04-05T21:34:38.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:38 smithi121 ceph-mon[145859]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T21:34:38.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:38 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T21:34:38.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:38 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T21:34:38.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:38 smithi188 ceph-mon[107793]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-04-05T21:34:38.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:38 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.188:6824/2257429686,v1:172.21.15.188:6825/2257429686] up:boot 2024-04-05T21:34:38.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:38 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.121:6828/1553603485,v1:172.21.15.121:6829/1553603485] up:boot 2024-04-05T21:34:38.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:38 smithi188 ceph-mon[107793]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T21:34:38.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:38 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T21:34:38.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:38 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T21:34:39.547 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:39 smithi188 ceph-mon[107793]: pgmap v249: 65 pgs: 10 peering, 5 active+undersized+degraded, 9 active+undersized, 41 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 765 B/s rd, 2 op/s; 15/333 objects degraded (4.505%) 2024-04-05T21:34:39.547 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:39 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:39.547 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:39 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi121.gxheqt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:34:39.547 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:39 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:34:39.662 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:39 smithi121 ceph-mon[145859]: pgmap v249: 65 pgs: 10 peering, 5 active+undersized+degraded, 9 active+undersized, 41 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 765 B/s rd, 2 op/s; 15/333 objects degraded (4.505%) 2024-04-05T21:34:39.662 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:39 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:39.662 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:39 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi121.gxheqt", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:34:39.662 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:39 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:34:40.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:40 smithi121 ceph-mon[145859]: Upgrade: Updating mds.cephfs.smithi121.gxheqt 2024-04-05T21:34:40.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:40 smithi121 ceph-mon[145859]: Deploying daemon mds.cephfs.smithi121.gxheqt on smithi121 2024-04-05T21:34:40.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:40 smithi188 ceph-mon[107793]: Upgrade: Updating mds.cephfs.smithi121.gxheqt 2024-04-05T21:34:40.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:40 smithi188 ceph-mon[107793]: Deploying daemon mds.cephfs.smithi121.gxheqt on smithi121 2024-04-05T21:34:41.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:41 smithi121 ceph-mon[145859]: pgmap v250: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-05T21:34:41.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:41 smithi121 ceph-mon[145859]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/333 objects degraded (4.505%), 5 pgs degraded) 2024-04-05T21:34:41.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:41 smithi121 ceph-mon[145859]: osdmap e81: 6 total, 6 up, 6 in 2024-04-05T21:34:41.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:41 smithi188 ceph-mon[107793]: pgmap v250: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-04-05T21:34:41.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:41 smithi188 ceph-mon[107793]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/333 objects degraded (4.505%), 5 pgs degraded) 2024-04-05T21:34:41.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:41 smithi188 ceph-mon[107793]: osdmap e81: 6 total, 6 up, 6 in 2024-04-05T21:34:42.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:42 smithi121 ceph-mon[145859]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T21:34:42.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:42 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:34:42.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:42 smithi188 ceph-mon[107793]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T21:34:42.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:42 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:34:43.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.568+0000 7f25d7577700 1 -- 172.21.15.121:0/630998578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25d0105a50 msgr2=0x7f25d0105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:43.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.568+0000 7f25d7577700 1 --2- 172.21.15.121:0/630998578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25d0105a50 0x7f25d0105e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f25c0009910 tx=0x7f25c0009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:43.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.568+0000 7f25d7577700 1 -- 172.21.15.121:0/630998578 shutdown_connections 2024-04-05T21:34:43.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.568+0000 7f25d7577700 1 --2- 172.21.15.121:0/630998578 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25d00ffa50 0x7f25d00ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:43.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.568+0000 7f25d7577700 1 --2- 172.21.15.121:0/630998578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25d0105a50 0x7f25d0105e30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:43.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.568+0000 7f25d7577700 1 -- 172.21.15.121:0/630998578 >> 172.21.15.121:0/630998578 conn(0x7f25d00fb540 msgr2=0x7f25d00fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:43.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.568+0000 7f25d7577700 1 -- 172.21.15.121:0/630998578 shutdown_connections 2024-04-05T21:34:43.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.568+0000 7f25d7577700 1 -- 172.21.15.121:0/630998578 wait complete. 2024-04-05T21:34:43.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.569+0000 7f25d7577700 1 Processor -- start 2024-04-05T21:34:43.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.569+0000 7f25d7577700 1 -- start start 2024-04-05T21:34:43.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.569+0000 7f25d7577700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25d00ffa50 0x7f25d0195410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:43.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.569+0000 7f25d7577700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25d0105a50 0x7f25d0195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:43.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.569+0000 7f25d7577700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0195e90 con 0x7f25d0105a50 2024-04-05T21:34:43.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.569+0000 7f25d7577700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0196000 con 0x7f25d00ffa50 2024-04-05T21:34:43.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.569+0000 7f25d4b12700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25d0105a50 0x7f25d0195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:43.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.569+0000 7f25d4b12700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25d0105a50 0x7f25d0195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38660/0 (socket says 172.21.15.121:38660) 2024-04-05T21:34:43.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.569+0000 7f25d4b12700 1 -- 172.21.15.121:0/693824408 learned_addr learned my addr 172.21.15.121:0/693824408 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:43.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.570+0000 7f25d5313700 1 --2- 172.21.15.121:0/693824408 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25d00ffa50 0x7f25d0195410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:43.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.570+0000 7f25d4b12700 1 -- 172.21.15.121:0/693824408 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25d00ffa50 msgr2=0x7f25d0195410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:43.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.570+0000 7f25d4b12700 1 --2- 172.21.15.121:0/693824408 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25d00ffa50 0x7f25d0195410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:43.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.570+0000 7f25d4b12700 1 -- 172.21.15.121:0/693824408 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00095f0 con 0x7f25d0105a50 2024-04-05T21:34:43.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.570+0000 7f25d4b12700 1 --2- 172.21.15.121:0/693824408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25d0105a50 0x7f25d0195950 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00b4b0 tx=0x7f25cc00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:43.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.570+0000 7f25c67fc700 1 -- 172.21.15.121:0/693824408 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25cc011800 con 0x7f25d0105a50 2024-04-05T21:34:43.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.570+0000 7f25d7577700 1 -- 172.21.15.121:0/693824408 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d01a2a80 con 0x7f25d0105a50 2024-04-05T21:34:43.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.570+0000 7f25c67fc700 1 -- 172.21.15.121:0/693824408 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25cc011e00 con 0x7f25d0105a50 2024-04-05T21:34:43.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.570+0000 7f25c67fc700 1 -- 172.21.15.121:0/693824408 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25cc010e60 con 0x7f25d0105a50 2024-04-05T21:34:43.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.570+0000 7f25d7577700 1 -- 172.21.15.121:0/693824408 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d01a2de0 con 0x7f25d0105a50 2024-04-05T21:34:43.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.571+0000 7f25d7577700 1 -- 172.21.15.121:0/693824408 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d0108050 con 0x7f25d0105a50 2024-04-05T21:34:43.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.572+0000 7f25c67fc700 1 -- 172.21.15.121:0/693824408 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f25cc01a440 con 0x7f25d0105a50 2024-04-05T21:34:43.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.572+0000 7f25c67fc700 1 --2- 172.21.15.121:0/693824408 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f25bc06cce0 0x7f25bc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:43.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.572+0000 7f25c67fc700 1 -- 172.21.15.121:0/693824408 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f25cc08c070 con 0x7f25d0105a50 2024-04-05T21:34:43.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.572+0000 7f25d5313700 1 --2- 172.21.15.121:0/693824408 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f25bc06cce0 0x7f25bc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:43.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.572+0000 7f25d5313700 1 --2- 172.21.15.121:0/693824408 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f25bc06cce0 0x7f25bc06f1a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f25c0006010 tx=0x7f25c001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:43.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.577+0000 7f25c67fc700 1 -- 172.21.15.121:0/693824408 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f25cc059b70 con 0x7f25d0105a50 2024-04-05T21:34:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:43 smithi121 ceph-mon[145859]: pgmap v252: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-05T21:34:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:43 smithi188 ceph-mon[107793]: pgmap v252: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-05T21:34:43.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.789+0000 7f25d7577700 1 -- 172.21.15.121:0/693824408 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f25d002cce0 con 0x7f25bc06cce0 2024-04-05T21:34:43.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.791+0000 7f25c67fc700 1 -- 172.21.15.121:0/693824408 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f25d002cce0 con 0x7f25bc06cce0 2024-04-05T21:34:43.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.793+0000 7f25d7577700 1 -- 172.21.15.121:0/693824408 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f25bc06cce0 msgr2=0x7f25bc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:43.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.793+0000 7f25d7577700 1 --2- 172.21.15.121:0/693824408 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f25bc06cce0 0x7f25bc06f1a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f25c0006010 tx=0x7f25c001a040 comp rx=0 tx=0).stop 2024-04-05T21:34:43.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.793+0000 7f25d7577700 1 -- 172.21.15.121:0/693824408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25d0105a50 msgr2=0x7f25d0195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:43.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.793+0000 7f25d7577700 1 --2- 172.21.15.121:0/693824408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25d0105a50 0x7f25d0195950 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00b4b0 tx=0x7f25cc00b7c0 comp rx=0 tx=0).stop 2024-04-05T21:34:43.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.793+0000 7f25d7577700 1 -- 172.21.15.121:0/693824408 shutdown_connections 2024-04-05T21:34:43.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.793+0000 7f25d7577700 1 --2- 172.21.15.121:0/693824408 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f25bc06cce0 0x7f25bc06f1a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:43.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.793+0000 7f25d7577700 1 --2- 172.21.15.121:0/693824408 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f25d00ffa50 0x7f25d0195410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:43.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.793+0000 7f25d7577700 1 --2- 172.21.15.121:0/693824408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25d0105a50 0x7f25d0195950 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:43.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.793+0000 7f25d7577700 1 -- 172.21.15.121:0/693824408 >> 172.21.15.121:0/693824408 conn(0x7f25d00fb540 msgr2=0x7f25d00fc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:43.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.793+0000 7f25d7577700 1 -- 172.21.15.121:0/693824408 shutdown_connections 2024-04-05T21:34:43.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.793+0000 7f25d7577700 1 -- 172.21.15.121:0/693824408 wait complete. 2024-04-05T21:34:43.804 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:34:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.896+0000 7f686f98f700 1 -- 172.21.15.121:0/3750844813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6868105a40 msgr2=0x7f6868105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.896+0000 7f686f98f700 1 --2- 172.21.15.121:0/3750844813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6868105a40 0x7f6868105e20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6858009910 tx=0x7f6858009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.896+0000 7f686f98f700 1 -- 172.21.15.121:0/3750844813 shutdown_connections 2024-04-05T21:34:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.896+0000 7f686f98f700 1 --2- 172.21.15.121:0/3750844813 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68680ffbd0 0x7f6868100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.896+0000 7f686f98f700 1 --2- 172.21.15.121:0/3750844813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6868105a40 0x7f6868105e20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.896+0000 7f686f98f700 1 -- 172.21.15.121:0/3750844813 >> 172.21.15.121:0/3750844813 conn(0x7f68680755a0 msgr2=0x7f68680759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.896+0000 7f686f98f700 1 -- 172.21.15.121:0/3750844813 shutdown_connections 2024-04-05T21:34:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.896+0000 7f686f98f700 1 -- 172.21.15.121:0/3750844813 wait complete. 2024-04-05T21:34:43.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.897+0000 7f686f98f700 1 Processor -- start 2024-04-05T21:34:43.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.897+0000 7f686f98f700 1 -- start start 2024-04-05T21:34:43.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.897+0000 7f686f98f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68680ffbd0 0x7f6868195570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:43.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.897+0000 7f686f98f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6868105a40 0x7f6868195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:43.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.897+0000 7f686f98f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6868196190 con 0x7f68680ffbd0 2024-04-05T21:34:43.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.897+0000 7f686f98f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6868199f20 con 0x7f6868105a40 2024-04-05T21:34:43.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.897+0000 7f686d72b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68680ffbd0 0x7f6868195570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:43.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.897+0000 7f686d72b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68680ffbd0 0x7f6868195570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38674/0 (socket says 172.21.15.121:38674) 2024-04-05T21:34:43.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.897+0000 7f686d72b700 1 -- 172.21.15.121:0/1549702737 learned_addr learned my addr 172.21.15.121:0/1549702737 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:43.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.897+0000 7f686d72b700 1 -- 172.21.15.121:0/1549702737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6868105a40 msgr2=0x7f6868195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:43.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.897+0000 7f686d72b700 1 --2- 172.21.15.121:0/1549702737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6868105a40 0x7f6868195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:43.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.897+0000 7f686d72b700 1 -- 172.21.15.121:0/1549702737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68580095f0 con 0x7f68680ffbd0 2024-04-05T21:34:43.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.898+0000 7f686d72b700 1 --2- 172.21.15.121:0/1549702737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68680ffbd0 0x7f6868195570 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6858006010 tx=0x7f685800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:43.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.898+0000 7f685e7fc700 1 -- 172.21.15.121:0/1549702737 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f685801d070 con 0x7f68680ffbd0 2024-04-05T21:34:43.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.898+0000 7f685e7fc700 1 -- 172.21.15.121:0/1549702737 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f685800f460 con 0x7f68680ffbd0 2024-04-05T21:34:43.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.898+0000 7f685e7fc700 1 -- 172.21.15.121:0/1549702737 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6858021640 con 0x7f68680ffbd0 2024-04-05T21:34:43.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.898+0000 7f686f98f700 1 -- 172.21.15.121:0/1549702737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f686819a1a0 con 0x7f68680ffbd0 2024-04-05T21:34:43.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.898+0000 7f686f98f700 1 -- 172.21.15.121:0/1549702737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f686819a690 con 0x7f68680ffbd0 2024-04-05T21:34:43.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.899+0000 7f686f98f700 1 -- 172.21.15.121:0/1549702737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6868108040 con 0x7f68680ffbd0 2024-04-05T21:34:43.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.899+0000 7f685e7fc700 1 -- 172.21.15.121:0/1549702737 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f685800fa20 con 0x7f68680ffbd0 2024-04-05T21:34:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.899+0000 7f685e7fc700 1 --2- 172.21.15.121:0/1549702737 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f685406cce0 0x7f685406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.900+0000 7f685e7fc700 1 -- 172.21.15.121:0/1549702737 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f685808d740 con 0x7f68680ffbd0 2024-04-05T21:34:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.900+0000 7f686cf2a700 1 --2- 172.21.15.121:0/1549702737 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f685406cce0 0x7f685406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.900+0000 7f686cf2a700 1 --2- 172.21.15.121:0/1549702737 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f685406cce0 0x7f685406f1a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6864005950 tx=0x7f68640058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:43.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:43.904+0000 7f685e7fc700 1 -- 172.21.15.121:0/1549702737 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6858057830 con 0x7f68680ffbd0 2024-04-05T21:34:44.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.115+0000 7f686f98f700 1 -- 172.21.15.121:0/1549702737 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f686802cd30 con 0x7f685406cce0 2024-04-05T21:34:44.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.117+0000 7f685e7fc700 1 -- 172.21.15.121:0/1549702737 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f686802cd30 con 0x7f685406cce0 2024-04-05T21:34:44.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.119+0000 7f686f98f700 1 -- 172.21.15.121:0/1549702737 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f685406cce0 msgr2=0x7f685406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:44.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.119+0000 7f686f98f700 1 --2- 172.21.15.121:0/1549702737 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f685406cce0 0x7f685406f1a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6864005950 tx=0x7f68640058e0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.119+0000 7f686f98f700 1 -- 172.21.15.121:0/1549702737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68680ffbd0 msgr2=0x7f6868195570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:44.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.119+0000 7f686f98f700 1 --2- 172.21.15.121:0/1549702737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68680ffbd0 0x7f6868195570 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6858006010 tx=0x7f685800be20 comp rx=0 tx=0).stop 2024-04-05T21:34:44.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.119+0000 7f686f98f700 1 -- 172.21.15.121:0/1549702737 shutdown_connections 2024-04-05T21:34:44.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.119+0000 7f686f98f700 1 --2- 172.21.15.121:0/1549702737 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f685406cce0 0x7f685406f1a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.119+0000 7f686f98f700 1 --2- 172.21.15.121:0/1549702737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68680ffbd0 0x7f6868195570 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.119+0000 7f686f98f700 1 --2- 172.21.15.121:0/1549702737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6868105a40 0x7f6868195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.119+0000 7f686f98f700 1 -- 172.21.15.121:0/1549702737 >> 172.21.15.121:0/1549702737 conn(0x7f68680755a0 msgr2=0x7f68680fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:44.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.119+0000 7f686f98f700 1 -- 172.21.15.121:0/1549702737 shutdown_connections 2024-04-05T21:34:44.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.119+0000 7f686f98f700 1 -- 172.21.15.121:0/1549702737 wait complete. 2024-04-05T21:34:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.221+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/1388931453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e180ffbd0 msgr2=0x7f9e18100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.221+0000 7f9e1ebeb700 1 --2- 172.21.15.121:0/1388931453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e180ffbd0 0x7f9e18100050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009910 tx=0x7f9e04009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.221+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/1388931453 shutdown_connections 2024-04-05T21:34:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.221+0000 7f9e1ebeb700 1 --2- 172.21.15.121:0/1388931453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e180ffbd0 0x7f9e18100050 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.221+0000 7f9e1ebeb700 1 --2- 172.21.15.121:0/1388931453 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e18105a40 0x7f9e18105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.221+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/1388931453 >> 172.21.15.121:0/1388931453 conn(0x7f9e180755a0 msgr2=0x7f9e180759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.221+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/1388931453 shutdown_connections 2024-04-05T21:34:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.221+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/1388931453 wait complete. 2024-04-05T21:34:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.222+0000 7f9e1ebeb700 1 Processor -- start 2024-04-05T21:34:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.222+0000 7f9e1ebeb700 1 -- start start 2024-04-05T21:34:44.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.222+0000 7f9e1ebeb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e180ffbd0 0x7f9e1819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.222+0000 7f9e1ebeb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e18105a40 0x7f9e1819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.222+0000 7f9e1ebeb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1819d0d0 con 0x7f9e180ffbd0 2024-04-05T21:34:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.222+0000 7f9e1ebeb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e18196580 con 0x7f9e18105a40 2024-04-05T21:34:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.222+0000 7f9e1c987700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e180ffbd0 0x7f9e1819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.222+0000 7f9e1c987700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e180ffbd0 0x7f9e1819c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38690/0 (socket says 172.21.15.121:38690) 2024-04-05T21:34:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.222+0000 7f9e1c987700 1 -- 172.21.15.121:0/585949588 learned_addr learned my addr 172.21.15.121:0/585949588 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.223+0000 7f9e1c987700 1 -- 172.21.15.121:0/585949588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e18105a40 msgr2=0x7f9e1819ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.223+0000 7f9e1c987700 1 --2- 172.21.15.121:0/585949588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e18105a40 0x7f9e1819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.223+0000 7f9e1c987700 1 -- 172.21.15.121:0/585949588 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e040095f0 con 0x7f9e180ffbd0 2024-04-05T21:34:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.223+0000 7f9e1c987700 1 --2- 172.21.15.121:0/585949588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e180ffbd0 0x7f9e1819c500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00b4b0 tx=0x7f9e0c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.223+0000 7f9e15ffb700 1 -- 172.21.15.121:0/585949588 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e0c011800 con 0x7f9e180ffbd0 2024-04-05T21:34:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.223+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/585949588 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e18196860 con 0x7f9e180ffbd0 2024-04-05T21:34:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.223+0000 7f9e15ffb700 1 -- 172.21.15.121:0/585949588 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e0c011e00 con 0x7f9e180ffbd0 2024-04-05T21:34:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.223+0000 7f9e15ffb700 1 -- 172.21.15.121:0/585949588 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e0c010e50 con 0x7f9e180ffbd0 2024-04-05T21:34:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.223+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/585949588 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e18196db0 con 0x7f9e180ffbd0 2024-04-05T21:34:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.224+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/585949588 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e18109070 con 0x7f9e180ffbd0 2024-04-05T21:34:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.224+0000 7f9e15ffb700 1 -- 172.21.15.121:0/585949588 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9e0c010380 con 0x7f9e180ffbd0 2024-04-05T21:34:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.225+0000 7f9e15ffb700 1 --2- 172.21.15.121:0/585949588 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9e0806cce0 0x7f9e0806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.225+0000 7f9e15ffb700 1 -- 172.21.15.121:0/585949588 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9e0c08c0c0 con 0x7f9e180ffbd0 2024-04-05T21:34:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.225+0000 7f9e17fff700 1 --2- 172.21.15.121:0/585949588 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9e0806cce0 0x7f9e0806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.225+0000 7f9e17fff700 1 --2- 172.21.15.121:0/585949588 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9e0806cce0 0x7f9e0806f1a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400b5c0 tx=0x7f9e04005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:44.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.229+0000 7f9e15ffb700 1 -- 172.21.15.121:0/585949588 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9e0c059c40 con 0x7f9e180ffbd0 2024-04-05T21:34:44.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.435+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/585949588 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e181970b0 con 0x7f9e0806cce0 2024-04-05T21:34:44.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.445+0000 7f9e15ffb700 1 -- 172.21.15.121:0/585949588 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f9e181970b0 con 0x7f9e0806cce0 2024-04-05T21:34:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:34:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (5m) 116s ago 16m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:34:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (17m) 116s ago 17m 8284k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:34:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (16m) 13s ago 16m 8334k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:34:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (3m) 116s ago 17m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:34:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (3m) 13s ago 16m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:34:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (5m) 116s ago 16m 82.7M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:34:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (10m) 116s ago 10m 64.5M - 18.2.1 5be31c24972a e161c92e7351 2024-04-05T21:34:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (10m) 116s ago 10m 133M - 18.2.1 5be31c24972a 5979e5171172 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (10m) 13s ago 10m 110M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (10m) 13s ago 10m 62.4M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (7m) 116s ago 18m 507M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (6m) 13s ago 15m 452M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (4m) 116s ago 18m 43.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (3m) 13s ago 15m 38.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (6m) 116s ago 17m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (6m) 13s ago 15m 16.9M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (3m) 116s ago 15m 88.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (2m) 116s ago 14m 99.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (2m) 116s ago 13m 11.2M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (86s) 13s ago 13m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (51s) 13s ago 13m 95.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (17s) 13s ago 12m 11.5M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:34:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (5m) 116s ago 16m 65.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:34:44.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.447+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/585949588 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9e0806cce0 msgr2=0x7f9e0806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:44.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.447+0000 7f9e1ebeb700 1 --2- 172.21.15.121:0/585949588 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9e0806cce0 0x7f9e0806f1a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400b5c0 tx=0x7f9e04005960 comp rx=0 tx=0).stop 2024-04-05T21:34:44.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.447+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/585949588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e180ffbd0 msgr2=0x7f9e1819c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:44.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.447+0000 7f9e1ebeb700 1 --2- 172.21.15.121:0/585949588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e180ffbd0 0x7f9e1819c500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00b4b0 tx=0x7f9e0c00b870 comp rx=0 tx=0).stop 2024-04-05T21:34:44.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.447+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/585949588 shutdown_connections 2024-04-05T21:34:44.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.447+0000 7f9e1ebeb700 1 --2- 172.21.15.121:0/585949588 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9e0806cce0 0x7f9e0806f1a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.447+0000 7f9e1ebeb700 1 --2- 172.21.15.121:0/585949588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e180ffbd0 0x7f9e1819c500 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.447+0000 7f9e1ebeb700 1 --2- 172.21.15.121:0/585949588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9e18105a40 0x7f9e1819ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.447+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/585949588 >> 172.21.15.121:0/585949588 conn(0x7f9e180755a0 msgr2=0x7f9e18103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:44.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.447+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/585949588 shutdown_connections 2024-04-05T21:34:44.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.447+0000 7f9e1ebeb700 1 -- 172.21.15.121:0/585949588 wait complete. 2024-04-05T21:34:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.549+0000 7ff9eaae8700 1 -- 172.21.15.121:0/3263636953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9e4100de0 msgr2=0x7ff9e41011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.549+0000 7ff9eaae8700 1 --2- 172.21.15.121:0/3263636953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9e4100de0 0x7ff9e41011c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8009910 tx=0x7ff9d8009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.550+0000 7ff9eaae8700 1 -- 172.21.15.121:0/3263636953 shutdown_connections 2024-04-05T21:34:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.550+0000 7ff9eaae8700 1 --2- 172.21.15.121:0/3263636953 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e4101790 0x7ff9e41057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.550+0000 7ff9eaae8700 1 --2- 172.21.15.121:0/3263636953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9e4100de0 0x7ff9e41011c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.550+0000 7ff9eaae8700 1 -- 172.21.15.121:0/3263636953 >> 172.21.15.121:0/3263636953 conn(0x7ff9e40fc670 msgr2=0x7ff9e40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.550+0000 7ff9eaae8700 1 -- 172.21.15.121:0/3263636953 shutdown_connections 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.550+0000 7ff9eaae8700 1 -- 172.21.15.121:0/3263636953 wait complete. 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.550+0000 7ff9eaae8700 1 Processor -- start 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.550+0000 7ff9eaae8700 1 -- start start 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.550+0000 7ff9eaae8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e4100de0 0x7ff9e4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.550+0000 7ff9eaae8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9e4101790 0x7ff9e40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.550+0000 7ff9eaae8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e4073b10 con 0x7ff9e4101790 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.550+0000 7ff9eaae8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e4073c50 con 0x7ff9e4100de0 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.550+0000 7ff9e3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9e4101790 0x7ff9e40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.551+0000 7ff9e3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9e4101790 0x7ff9e40735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56800/0 (socket says 172.21.15.121:56800) 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.551+0000 7ff9e3fff700 1 -- 172.21.15.121:0/2287288443 learned_addr learned my addr 172.21.15.121:0/2287288443 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.551+0000 7ff9e8884700 1 --2- 172.21.15.121:0/2287288443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e4100de0 0x7ff9e4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.551+0000 7ff9e3fff700 1 -- 172.21.15.121:0/2287288443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e4100de0 msgr2=0x7ff9e4073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.551+0000 7ff9e3fff700 1 --2- 172.21.15.121:0/2287288443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e4100de0 0x7ff9e4073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.551+0000 7ff9e3fff700 1 -- 172.21.15.121:0/2287288443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d80095f0 con 0x7ff9e4101790 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.551+0000 7ff9e8884700 1 --2- 172.21.15.121:0/2287288443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e4100de0 0x7ff9e4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:34:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.551+0000 7ff9e3fff700 1 --2- 172.21.15.121:0/2287288443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9e4101790 0x7ff9e40735d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000d8d0 tx=0x7ff9d000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.551+0000 7ff9e1ffb700 1 -- 172.21.15.121:0/2287288443 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9d0004a00 con 0x7ff9e4101790 2024-04-05T21:34:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.551+0000 7ff9e1ffb700 1 -- 172.21.15.121:0/2287288443 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9d0005020 con 0x7ff9e4101790 2024-04-05T21:34:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.551+0000 7ff9eaae8700 1 -- 172.21.15.121:0/2287288443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e41a2860 con 0x7ff9e4101790 2024-04-05T21:34:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.551+0000 7ff9e1ffb700 1 -- 172.21.15.121:0/2287288443 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9d0009bd0 con 0x7ff9e4101790 2024-04-05T21:34:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.551+0000 7ff9eaae8700 1 -- 172.21.15.121:0/2287288443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e41a2c70 con 0x7ff9e4101790 2024-04-05T21:34:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.552+0000 7ff9eaae8700 1 -- 172.21.15.121:0/2287288443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e4109180 con 0x7ff9e4101790 2024-04-05T21:34:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.553+0000 7ff9e1ffb700 1 -- 172.21.15.121:0/2287288443 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff9d0005190 con 0x7ff9e4101790 2024-04-05T21:34:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.553+0000 7ff9e1ffb700 1 --2- 172.21.15.121:0/2287288443 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff9d406cce0 0x7ff9d406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.553+0000 7ff9e8884700 1 --2- 172.21.15.121:0/2287288443 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff9d406cce0 0x7ff9d406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.553+0000 7ff9e1ffb700 1 -- 172.21.15.121:0/2287288443 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff9d008bf10 con 0x7ff9e4101790 2024-04-05T21:34:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.553+0000 7ff9e8884700 1 --2- 172.21.15.121:0/2287288443 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff9d406cce0 0x7ff9d406f1a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800b5c0 tx=0x7ff9d801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:44.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.558+0000 7ff9e1ffb700 1 -- 172.21.15.121:0/2287288443 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff9d0055f80 con 0x7ff9e4101790 2024-04-05T21:34:44.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.834+0000 7ff9eaae8700 1 -- 172.21.15.121:0/2287288443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff9e404ea90 con 0x7ff9e4101790 2024-04-05T21:34:44.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.835+0000 7ff9e1ffb700 1 -- 172.21.15.121:0/2287288443 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+709 (secure 0 0 0) 0x7ff9d00202e0 con 0x7ff9e4101790 2024-04-05T21:34:44.836 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:34:44.836 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:34:44.836 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:34:44.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:34:44.836 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:34:44.836 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:34:44.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:34:44.837 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:34:44.837 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:34:44.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:34:44.837 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:34:44.837 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3 2024-04-05T21:34:44.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:34:44.837 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:34:44.837 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 3, 2024-04-05T21:34:44.837 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 10 2024-04-05T21:34:44.837 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:34:44.837 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:34:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.837+0000 7ff9eaae8700 1 -- 172.21.15.121:0/2287288443 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff9d406cce0 msgr2=0x7ff9d406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.837+0000 7ff9eaae8700 1 --2- 172.21.15.121:0/2287288443 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff9d406cce0 0x7ff9d406f1a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800b5c0 tx=0x7ff9d801a040 comp rx=0 tx=0).stop 2024-04-05T21:34:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.837+0000 7ff9eaae8700 1 -- 172.21.15.121:0/2287288443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9e4101790 msgr2=0x7ff9e40735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.837+0000 7ff9eaae8700 1 --2- 172.21.15.121:0/2287288443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9e4101790 0x7ff9e40735d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff9d000d8d0 tx=0x7ff9d000dc90 comp rx=0 tx=0).stop 2024-04-05T21:34:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.837+0000 7ff9eaae8700 1 -- 172.21.15.121:0/2287288443 shutdown_connections 2024-04-05T21:34:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.837+0000 7ff9eaae8700 1 --2- 172.21.15.121:0/2287288443 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff9d406cce0 0x7ff9d406f1a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.837+0000 7ff9eaae8700 1 --2- 172.21.15.121:0/2287288443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9e4100de0 0x7ff9e4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.837+0000 7ff9eaae8700 1 --2- 172.21.15.121:0/2287288443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9e4101790 0x7ff9e40735d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.837+0000 7ff9eaae8700 1 -- 172.21.15.121:0/2287288443 >> 172.21.15.121:0/2287288443 conn(0x7ff9e40fc670 msgr2=0x7ff9e40fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.837+0000 7ff9eaae8700 1 -- 172.21.15.121:0/2287288443 shutdown_connections 2024-04-05T21:34:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.837+0000 7ff9eaae8700 1 -- 172.21.15.121:0/2287288443 wait complete. 2024-04-05T21:34:44.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.940+0000 7f0bea4c4700 1 -- 172.21.15.121:0/409815298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0be40ffbd0 msgr2=0x7f0be4100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.940+0000 7f0bea4c4700 1 --2- 172.21.15.121:0/409815298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0be40ffbd0 0x7f0be4100050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009910 tx=0x7f0bcc009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.940+0000 7f0bea4c4700 1 -- 172.21.15.121:0/409815298 shutdown_connections 2024-04-05T21:34:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.940+0000 7f0bea4c4700 1 --2- 172.21.15.121:0/409815298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0be40ffbd0 0x7f0be4100050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.940+0000 7f0bea4c4700 1 --2- 172.21.15.121:0/409815298 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0be4105a40 0x7f0be4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.940+0000 7f0bea4c4700 1 -- 172.21.15.121:0/409815298 >> 172.21.15.121:0/409815298 conn(0x7f0be40755a0 msgr2=0x7f0be40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.940+0000 7f0bea4c4700 1 -- 172.21.15.121:0/409815298 shutdown_connections 2024-04-05T21:34:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.940+0000 7f0bea4c4700 1 -- 172.21.15.121:0/409815298 wait complete. 2024-04-05T21:34:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0bea4c4700 1 Processor -- start 2024-04-05T21:34:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0bea4c4700 1 -- start start 2024-04-05T21:34:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0bea4c4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0be40ffbd0 0x7f0be4195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0bea4c4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0be4105a40 0x7f0be4195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0bea4c4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be4196160 con 0x7f0be40ffbd0 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0bea4c4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be4199ef0 con 0x7f0be4105a40 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0be3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0be40ffbd0 0x7f0be4195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0be3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0be40ffbd0 0x7f0be4195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56830/0 (socket says 172.21.15.121:56830) 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0be3fff700 1 -- 172.21.15.121:0/2660154129 learned_addr learned my addr 172.21.15.121:0/2660154129 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0bdbfff700 1 --2- 172.21.15.121:0/2660154129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0be4105a40 0x7f0be4195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0be3fff700 1 -- 172.21.15.121:0/2660154129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0be4105a40 msgr2=0x7f0be4195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0be3fff700 1 --2- 172.21.15.121:0/2660154129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0be4105a40 0x7f0be4195a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0be3fff700 1 -- 172.21.15.121:0/2660154129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc0095f0 con 0x7f0be40ffbd0 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.941+0000 7f0be3fff700 1 --2- 172.21.15.121:0/2660154129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0be40ffbd0 0x7f0be4195540 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0bd400d8d0 tx=0x7f0bd400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.942+0000 7f0be1ffb700 1 -- 172.21.15.121:0/2660154129 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bd4009940 con 0x7f0be40ffbd0 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.942+0000 7f0be1ffb700 1 -- 172.21.15.121:0/2660154129 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bd4010460 con 0x7f0be40ffbd0 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.942+0000 7f0bea4c4700 1 -- 172.21.15.121:0/2660154129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be419a1d0 con 0x7f0be40ffbd0 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.942+0000 7f0be1ffb700 1 -- 172.21.15.121:0/2660154129 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bd400f5f0 con 0x7f0be40ffbd0 2024-04-05T21:34:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.942+0000 7f0bea4c4700 1 -- 172.21.15.121:0/2660154129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be419a720 con 0x7f0be40ffbd0 2024-04-05T21:34:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.943+0000 7f0bea4c4700 1 -- 172.21.15.121:0/2660154129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be404ea90 con 0x7f0be40ffbd0 2024-04-05T21:34:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.943+0000 7f0be1ffb700 1 -- 172.21.15.121:0/2660154129 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0bd40105d0 con 0x7f0be40ffbd0 2024-04-05T21:34:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.943+0000 7f0be1ffb700 1 --2- 172.21.15.121:0/2660154129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0bd006cce0 0x7f0bd006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.944+0000 7f0be1ffb700 1 -- 172.21.15.121:0/2660154129 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0bd408c5e0 con 0x7f0be40ffbd0 2024-04-05T21:34:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.944+0000 7f0bdbfff700 1 --2- 172.21.15.121:0/2660154129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0bd006cce0 0x7f0bd006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.944+0000 7f0bdbfff700 1 --2- 172.21.15.121:0/2660154129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0bd006cce0 0x7f0bd006f1a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0be4196b60 tx=0x7f0bcc01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:44.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:44.948+0000 7f0be1ffb700 1 -- 172.21.15.121:0/2660154129 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0bd4056650 con 0x7f0be40ffbd0 2024-04-05T21:34:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.202+0000 7f0bea4c4700 1 -- 172.21.15.121:0/2660154129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0be419abb0 con 0x7f0be40ffbd0 2024-04-05T21:34:45.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.203+0000 7f0be1ffb700 1 -- 172.21.15.121:0/2660154129 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) v1 ==== 76+0+1665 (secure 0 0 0) 0x7f0bd4010880 con 0x7f0be40ffbd0 2024-04-05T21:34:45.204 INFO:teuthology.orchestra.run.smithi121.stdout:e25 2024-04-05T21:34:45.204 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:34:45.204 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:34:45.204 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:34:45.204 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:34:45.204 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:34:45.204 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:34:45.204 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:34:45.204 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:34:45.205 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:34:45.206 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34328} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/1553603485,v1:172.21.15.121:6829/1553603485] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:34:45.207 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:34:45.207 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44255} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2257429686,v1:172.21.15.188:6825/2257429686] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:34:45.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.205+0000 7f0bea4c4700 1 -- 172.21.15.121:0/2660154129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0bd006cce0 msgr2=0x7f0bd006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:45.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.205+0000 7f0bea4c4700 1 --2- 172.21.15.121:0/2660154129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0bd006cce0 0x7f0bd006f1a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0be4196b60 tx=0x7f0bcc01a040 comp rx=0 tx=0).stop 2024-04-05T21:34:45.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.205+0000 7f0bea4c4700 1 -- 172.21.15.121:0/2660154129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0be40ffbd0 msgr2=0x7f0be4195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:45.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.205+0000 7f0bea4c4700 1 --2- 172.21.15.121:0/2660154129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0be40ffbd0 0x7f0be4195540 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0bd400d8d0 tx=0x7f0bd400dc90 comp rx=0 tx=0).stop 2024-04-05T21:34:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.205+0000 7f0bea4c4700 1 -- 172.21.15.121:0/2660154129 shutdown_connections 2024-04-05T21:34:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.205+0000 7f0bea4c4700 1 --2- 172.21.15.121:0/2660154129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0bd006cce0 0x7f0bd006f1a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.205+0000 7f0bea4c4700 1 --2- 172.21.15.121:0/2660154129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0be40ffbd0 0x7f0be4195540 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.205+0000 7f0bea4c4700 1 --2- 172.21.15.121:0/2660154129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0be4105a40 0x7f0be4195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.205+0000 7f0bea4c4700 1 -- 172.21.15.121:0/2660154129 >> 172.21.15.121:0/2660154129 conn(0x7f0be40755a0 msgr2=0x7f0be40fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.205+0000 7f0bea4c4700 1 -- 172.21.15.121:0/2660154129 shutdown_connections 2024-04-05T21:34:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.205+0000 7f0bea4c4700 1 -- 172.21.15.121:0/2660154129 wait complete. 2024-04-05T21:34:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 25 2024-04-05T21:34:45.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.312+0000 7fde67cde700 1 -- 172.21.15.121:0/4227558946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde60068670 msgr2=0x7fde60068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:45.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.312+0000 7fde67cde700 1 --2- 172.21.15.121:0/4227558946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde60068670 0x7fde60068a50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fde5c009910 tx=0x7fde5c009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:45.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.313+0000 7fde67cde700 1 -- 172.21.15.121:0/4227558946 shutdown_connections 2024-04-05T21:34:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.313+0000 7fde67cde700 1 --2- 172.21.15.121:0/4227558946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde60069020 0x7fde60105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.313+0000 7fde67cde700 1 --2- 172.21.15.121:0/4227558946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde60068670 0x7fde60068a50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.313+0000 7fde67cde700 1 -- 172.21.15.121:0/4227558946 >> 172.21.15.121:0/4227558946 conn(0x7fde600755a0 msgr2=0x7fde600759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.313+0000 7fde67cde700 1 -- 172.21.15.121:0/4227558946 shutdown_connections 2024-04-05T21:34:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.313+0000 7fde67cde700 1 -- 172.21.15.121:0/4227558946 wait complete. 2024-04-05T21:34:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.313+0000 7fde67cde700 1 Processor -- start 2024-04-05T21:34:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.313+0000 7fde67cde700 1 -- start start 2024-04-05T21:34:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.313+0000 7fde67cde700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde60068670 0x7fde60196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.313+0000 7fde67cde700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde60069020 0x7fde60196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.313+0000 7fde67cde700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde60197290 con 0x7fde60068670 2024-04-05T21:34:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.313+0000 7fde67cde700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6019afb0 con 0x7fde60069020 2024-04-05T21:34:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde65a7a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde60068670 0x7fde60196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde65a7a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde60068670 0x7fde60196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56838/0 (socket says 172.21.15.121:56838) 2024-04-05T21:34:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde65a7a700 1 -- 172.21.15.121:0/3008925082 learned_addr learned my addr 172.21.15.121:0/3008925082 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde65279700 1 --2- 172.21.15.121:0/3008925082 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde60069020 0x7fde60196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde65a7a700 1 -- 172.21.15.121:0/3008925082 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde60069020 msgr2=0x7fde60196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde65a7a700 1 --2- 172.21.15.121:0/3008925082 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde60069020 0x7fde60196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde65a7a700 1 -- 172.21.15.121:0/3008925082 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c0095f0 con 0x7fde60068670 2024-04-05T21:34:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde65279700 1 --2- 172.21.15.121:0/3008925082 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde60069020 0x7fde60196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:34:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde65a7a700 1 --2- 172.21.15.121:0/3008925082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde60068670 0x7fde60196670 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fde5c006010 tx=0x7fde5c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde56ffd700 1 -- 172.21.15.121:0/3008925082 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde5c01d070 con 0x7fde60068670 2024-04-05T21:34:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde67cde700 1 -- 172.21.15.121:0/3008925082 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6019b230 con 0x7fde60068670 2024-04-05T21:34:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde56ffd700 1 -- 172.21.15.121:0/3008925082 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde5c00f460 con 0x7fde60068670 2024-04-05T21:34:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde56ffd700 1 -- 172.21.15.121:0/3008925082 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde5c021620 con 0x7fde60068670 2024-04-05T21:34:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.314+0000 7fde67cde700 1 -- 172.21.15.121:0/3008925082 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6019b720 con 0x7fde60068670 2024-04-05T21:34:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.315+0000 7fde67cde700 1 -- 172.21.15.121:0/3008925082 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde60109130 con 0x7fde60068670 2024-04-05T21:34:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.316+0000 7fde56ffd700 1 -- 172.21.15.121:0/3008925082 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fde5c00fa40 con 0x7fde60068670 2024-04-05T21:34:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.316+0000 7fde56ffd700 1 --2- 172.21.15.121:0/3008925082 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde4c06cce0 0x7fde4c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.316+0000 7fde56ffd700 1 -- 172.21.15.121:0/3008925082 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fde5c08d790 con 0x7fde60068670 2024-04-05T21:34:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.316+0000 7fde65279700 1 --2- 172.21.15.121:0/3008925082 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde4c06cce0 0x7fde4c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.316+0000 7fde65279700 1 --2- 172.21.15.121:0/3008925082 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde4c06cce0 0x7fde4c06f1a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fde50005950 tx=0x7fde500058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:45.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.321+0000 7fde56ffd700 1 -- 172.21.15.121:0/3008925082 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fde5c057800 con 0x7fde60068670 2024-04-05T21:34:45.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:45 smithi121 ceph-mon[145859]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:45.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:45 smithi121 ceph-mon[145859]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:45.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:45 smithi121 ceph-mon[145859]: pgmap v253: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-05T21:34:45.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:45 smithi121 ceph-mon[145859]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:45.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:45 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2287288443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:45.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:45.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:34:45.477 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:45 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2660154129' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:34:45.478 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:45.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.537+0000 7fde67cde700 1 -- 172.21.15.121:0/3008925082 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde6002cd30 con 0x7fde4c06cce0 2024-04-05T21:34:45.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.538+0000 7fde56ffd700 1 -- 172.21.15.121:0/3008925082 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fde6002cd30 con 0x7fde4c06cce0 2024-04-05T21:34:45.539 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:34:45.539 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:34:45.539 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:34:45.540 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:34:45.540 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:34:45.540 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:34:45.540 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:34:45.540 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:34:45.540 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:34:45.540 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:34:45.540 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "12/23 daemons upgraded", 2024-04-05T21:34:45.540 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:34:45.540 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:34:45.540 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:34:45.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.540+0000 7fde67cde700 1 -- 172.21.15.121:0/3008925082 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde4c06cce0 msgr2=0x7fde4c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:45.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.540+0000 7fde67cde700 1 --2- 172.21.15.121:0/3008925082 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde4c06cce0 0x7fde4c06f1a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fde50005950 tx=0x7fde500058e0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.540+0000 7fde67cde700 1 -- 172.21.15.121:0/3008925082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde60068670 msgr2=0x7fde60196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:45.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.541+0000 7fde67cde700 1 --2- 172.21.15.121:0/3008925082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde60068670 0x7fde60196670 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fde5c006010 tx=0x7fde5c00be20 comp rx=0 tx=0).stop 2024-04-05T21:34:45.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.541+0000 7fde67cde700 1 -- 172.21.15.121:0/3008925082 shutdown_connections 2024-04-05T21:34:45.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.541+0000 7fde67cde700 1 --2- 172.21.15.121:0/3008925082 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde4c06cce0 0x7fde4c06f1a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.541+0000 7fde67cde700 1 --2- 172.21.15.121:0/3008925082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde60068670 0x7fde60196670 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.541+0000 7fde67cde700 1 --2- 172.21.15.121:0/3008925082 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde60069020 0x7fde60196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.541+0000 7fde67cde700 1 -- 172.21.15.121:0/3008925082 >> 172.21.15.121:0/3008925082 conn(0x7fde600755a0 msgr2=0x7fde600fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:45.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.541+0000 7fde67cde700 1 -- 172.21.15.121:0/3008925082 shutdown_connections 2024-04-05T21:34:45.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.541+0000 7fde67cde700 1 -- 172.21.15.121:0/3008925082 wait complete. 2024-04-05T21:34:45.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.645+0000 7fc97af82700 1 -- 172.21.15.121:0/82017535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974100da0 msgr2=0x7fc974101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:45.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.645+0000 7fc97af82700 1 --2- 172.21.15.121:0/82017535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974100da0 0x7fc974101180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc968009910 tx=0x7fc968009c20 comp rx=0 tx=0).stop 2024-04-05T21:34:45.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.645+0000 7fc97af82700 1 -- 172.21.15.121:0/82017535 shutdown_connections 2024-04-05T21:34:45.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.645+0000 7fc97af82700 1 --2- 172.21.15.121:0/82017535 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc974101750 0x7fc9741057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.645+0000 7fc97af82700 1 --2- 172.21.15.121:0/82017535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974100da0 0x7fc974101180 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.645+0000 7fc97af82700 1 -- 172.21.15.121:0/82017535 >> 172.21.15.121:0/82017535 conn(0x7fc9740fc630 msgr2=0x7fc9740fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:45.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.646+0000 7fc97af82700 1 -- 172.21.15.121:0/82017535 shutdown_connections 2024-04-05T21:34:45.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.646+0000 7fc97af82700 1 -- 172.21.15.121:0/82017535 wait complete. 2024-04-05T21:34:45.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.646+0000 7fc97af82700 1 Processor -- start 2024-04-05T21:34:45.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.646+0000 7fc97af82700 1 -- start start 2024-04-05T21:34:45.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc97af82700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974100da0 0x7fc974196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:45.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc97af82700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc974101750 0x7fc974196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:45.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc97af82700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc974197260 con 0x7fc974100da0 2024-04-05T21:34:45.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc97af82700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc974199f90 con 0x7fc974101750 2024-04-05T21:34:45.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc978d1e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974100da0 0x7fc974196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:45.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc978d1e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974100da0 0x7fc974196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56854/0 (socket says 172.21.15.121:56854) 2024-04-05T21:34:45.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc978d1e700 1 -- 172.21.15.121:0/1180680806 learned_addr learned my addr 172.21.15.121:0/1180680806 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:34:45.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc973fff700 1 --2- 172.21.15.121:0/1180680806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc974101750 0x7fc974196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:45.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc978d1e700 1 -- 172.21.15.121:0/1180680806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc974101750 msgr2=0x7fc974196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:45.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc978d1e700 1 --2- 172.21.15.121:0/1180680806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc974101750 0x7fc974196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc978d1e700 1 -- 172.21.15.121:0/1180680806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9680095f0 con 0x7fc974100da0 2024-04-05T21:34:45.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc978d1e700 1 --2- 172.21.15.121:0/1180680806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974100da0 0x7fc974196690 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc968006010 tx=0x7fc96800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:45.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc971ffb700 1 -- 172.21.15.121:0/1180680806 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc96801d070 con 0x7fc974100da0 2024-04-05T21:34:45.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc97af82700 1 -- 172.21.15.121:0/1180680806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97419a210 con 0x7fc974100da0 2024-04-05T21:34:45.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc971ffb700 1 -- 172.21.15.121:0/1180680806 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc96800f460 con 0x7fc974100da0 2024-04-05T21:34:45.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc971ffb700 1 -- 172.21.15.121:0/1180680806 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc968021640 con 0x7fc974100da0 2024-04-05T21:34:45.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.647+0000 7fc97af82700 1 -- 172.21.15.121:0/1180680806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97419a700 con 0x7fc974100da0 2024-04-05T21:34:45.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.648+0000 7fc97af82700 1 -- 172.21.15.121:0/1180680806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc974109140 con 0x7fc974100da0 2024-04-05T21:34:45.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.649+0000 7fc971ffb700 1 -- 172.21.15.121:0/1180680806 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc96800f9e0 con 0x7fc974100da0 2024-04-05T21:34:45.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.649+0000 7fc971ffb700 1 --2- 172.21.15.121:0/1180680806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc96406cce0 0x7fc96406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:34:45.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.649+0000 7fc971ffb700 1 -- 172.21.15.121:0/1180680806 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc96808d9a0 con 0x7fc974100da0 2024-04-05T21:34:45.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.649+0000 7fc973fff700 1 --2- 172.21.15.121:0/1180680806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc96406cce0 0x7fc96406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:34:45.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.650+0000 7fc973fff700 1 --2- 172.21.15.121:0/1180680806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc96406cce0 0x7fc96406f1a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc974197cb0 tx=0x7fc960005cf0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:34:45.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.654+0000 7fc971ffb700 1 -- 172.21.15.121:0/1180680806 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc968057970 con 0x7fc974100da0 2024-04-05T21:34:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:45 smithi188 ceph-mon[107793]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:45 smithi188 ceph-mon[107793]: from='client.34336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:45 smithi188 ceph-mon[107793]: pgmap v253: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-05T21:34:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:45 smithi188 ceph-mon[107793]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:45 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2287288443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:34:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:45 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2660154129' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:34:45.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:45.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.957+0000 7fc97af82700 1 -- 172.21.15.121:0/1180680806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc974108b20 con 0x7fc974100da0 2024-04-05T21:34:45.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.958+0000 7fc971ffb700 1 -- 172.21.15.121:0/1180680806 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc96805af90 con 0x7fc974100da0 2024-04-05T21:34:45.959 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:34:45.959 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:34:45.959 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:34:45.959 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:34:45.959 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:34:45.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.960+0000 7fc97af82700 1 -- 172.21.15.121:0/1180680806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc96406cce0 msgr2=0x7fc96406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:45.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.960+0000 7fc97af82700 1 --2- 172.21.15.121:0/1180680806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc96406cce0 0x7fc96406f1a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc974197cb0 tx=0x7fc960005cf0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.960+0000 7fc97af82700 1 -- 172.21.15.121:0/1180680806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974100da0 msgr2=0x7fc974196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:34:45.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.960+0000 7fc97af82700 1 --2- 172.21.15.121:0/1180680806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974100da0 0x7fc974196690 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc968006010 tx=0x7fc96800be20 comp rx=0 tx=0).stop 2024-04-05T21:34:45.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.960+0000 7fc97af82700 1 -- 172.21.15.121:0/1180680806 shutdown_connections 2024-04-05T21:34:45.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.960+0000 7fc97af82700 1 --2- 172.21.15.121:0/1180680806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc96406cce0 0x7fc96406f1a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.960+0000 7fc97af82700 1 --2- 172.21.15.121:0/1180680806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974100da0 0x7fc974196690 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.960+0000 7fc97af82700 1 --2- 172.21.15.121:0/1180680806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc974101750 0x7fc974196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:34:45.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.960+0000 7fc97af82700 1 -- 172.21.15.121:0/1180680806 >> 172.21.15.121:0/1180680806 conn(0x7fc9740fc630 msgr2=0x7fc9740fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:34:45.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.960+0000 7fc97af82700 1 -- 172.21.15.121:0/1180680806 shutdown_connections 2024-04-05T21:34:45.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:34:45.960+0000 7fc97af82700 1 -- 172.21.15.121:0/1180680806 wait complete. 2024-04-05T21:34:46.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:46 smithi121 ceph-mon[145859]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:46.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:46 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1180680806' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:34:46.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:46 smithi188 ceph-mon[107793]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:34:46.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:46 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1180680806' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:34:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:47 smithi121 ceph-mon[145859]: pgmap v254: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-05T21:34:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:47 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:47 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:47 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:47 smithi188 ceph-mon[107793]: pgmap v254: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-04-05T21:34:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:47 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:47.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:47 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:47.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:47 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:48.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:48 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] up:boot 2024-04-05T21:34:48.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:48 smithi121 ceph-mon[145859]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T21:34:48.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:48 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:34:48.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:48 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] up:boot 2024-04-05T21:34:48.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:48 smithi188 ceph-mon[107793]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T21:34:48.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:48 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T21:34:49.642 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:49 smithi188 ceph-mon[107793]: pgmap v255: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:34:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:49 smithi121 ceph-mon[145859]: pgmap v255: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:34:51.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:51 smithi121 ceph-mon[145859]: pgmap v256: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:34:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:51 smithi188 ceph-mon[107793]: pgmap v256: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:34:53.750 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:53 smithi121 ceph-mon[145859]: pgmap v257: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:34:53.750 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:53 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:53.750 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:53 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:53.750 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:53 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:53.750 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:53 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:53 smithi188 ceph-mon[107793]: pgmap v257: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:34:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:53 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:53 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:53 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:53 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:55 smithi188 ceph-mon[107793]: pgmap v258: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:34:55.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:55 smithi121 ceph-mon[145859]: pgmap v258: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:34:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:57 smithi188 ceph-mon[107793]: pgmap v259: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:34:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:34:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:34:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:57.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:57.782 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:57 smithi121 ceph-mon[145859]: pgmap v259: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:34:57.782 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:57.782 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:57.782 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:34:57.782 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:34:57.782 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:57.782 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:34:57.782 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:57.782 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:57.782 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:57.782 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:34:59.641 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:59 smithi121 ceph-mon[145859]: pgmap v260: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:34:59.641 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:59 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:59.641 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:59 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi121.ifoxmw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:34:59.641 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:34:59 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:34:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:59 smithi188 ceph-mon[107793]: pgmap v260: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:34:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:59 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:34:59.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:59 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi121.ifoxmw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:34:59.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:34:59 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:35:00.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:00 smithi188 ceph-mon[107793]: Upgrade: Updating mds.cephfs.smithi121.ifoxmw 2024-04-05T21:35:00.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:00 smithi188 ceph-mon[107793]: Deploying daemon mds.cephfs.smithi121.ifoxmw on smithi121 2024-04-05T21:35:00.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:35:00.976 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:00 smithi121 ceph-mon[145859]: Upgrade: Updating mds.cephfs.smithi121.ifoxmw 2024-04-05T21:35:00.976 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:00 smithi121 ceph-mon[145859]: Deploying daemon mds.cephfs.smithi121.ifoxmw on smithi121 2024-04-05T21:35:00.977 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:35:01.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:01 smithi121 ceph-mon[145859]: pgmap v261: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:01.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:01 smithi121 ceph-mon[145859]: osdmap e82: 6 total, 6 up, 6 in 2024-04-05T21:35:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:01 smithi188 ceph-mon[107793]: pgmap v261: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:01.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:01 smithi188 ceph-mon[107793]: osdmap e82: 6 total, 6 up, 6 in 2024-04-05T21:35:02.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:02 smithi188 ceph-mon[107793]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T21:35:02.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:02 smithi121 ceph-mon[145859]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T21:35:03.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:03 smithi188 ceph-mon[107793]: pgmap v263: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:03.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:03 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T21:35:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:03 smithi121 ceph-mon[145859]: pgmap v263: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:03 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T21:35:05.916 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:05 smithi121 ceph-mon[145859]: pgmap v264: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:06.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:05 smithi188 ceph-mon[107793]: pgmap v264: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:07.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:07 smithi121 ceph-mon[145859]: pgmap v265: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:07.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:07 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:07.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:07 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:07.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:07 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:35:08.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:07 smithi188 ceph-mon[107793]: pgmap v265: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:08.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:07 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:08.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:07 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:08.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:07 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:35:08.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:08 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] up:boot 2024-04-05T21:35:08.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:08 smithi121 ceph-mon[145859]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T21:35:08.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:08 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T21:35:09.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:08 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] up:boot 2024-04-05T21:35:09.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:08 smithi188 ceph-mon[107793]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T21:35:09.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:08 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T21:35:09.829 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:09 smithi188 ceph-mon[107793]: pgmap v266: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:09.957 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:09 smithi121 ceph-mon[145859]: pgmap v266: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:11 smithi121 ceph-mon[145859]: pgmap v267: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:12.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:11 smithi188 ceph-mon[107793]: pgmap v267: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:13 smithi121 ceph-mon[145859]: pgmap v268: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:13 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:13 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:13 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:13 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:13.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:13 smithi188 ceph-mon[107793]: pgmap v268: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:13.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:13 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:13.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:13 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:13.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:13 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:13.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:13 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:15 smithi121 ceph-mon[145859]: pgmap v269: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:35:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:15 smithi188 ceph-mon[107793]: pgmap v269: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:35:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.070+0000 7f20b1ef6700 1 -- 172.21.15.121:0/3959927579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac100da0 msgr2=0x7f20ac101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.070+0000 7f20b1ef6700 1 --2- 172.21.15.121:0/3959927579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac100da0 0x7f20ac101180 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f209c009910 tx=0x7f209c009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.070+0000 7f20b1ef6700 1 -- 172.21.15.121:0/3959927579 shutdown_connections 2024-04-05T21:35:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.070+0000 7f20b1ef6700 1 --2- 172.21.15.121:0/3959927579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20ac101750 0x7f20ac1057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.070+0000 7f20b1ef6700 1 --2- 172.21.15.121:0/3959927579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac100da0 0x7f20ac101180 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.070+0000 7f20b1ef6700 1 -- 172.21.15.121:0/3959927579 >> 172.21.15.121:0/3959927579 conn(0x7f20ac0fc630 msgr2=0x7f20ac0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.070+0000 7f20b1ef6700 1 -- 172.21.15.121:0/3959927579 shutdown_connections 2024-04-05T21:35:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.070+0000 7f20b1ef6700 1 -- 172.21.15.121:0/3959927579 wait complete. 2024-04-05T21:35:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.070+0000 7f20b1ef6700 1 Processor -- start 2024-04-05T21:35:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.071+0000 7f20b1ef6700 1 -- start start 2024-04-05T21:35:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.071+0000 7f20b1ef6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20ac100da0 0x7f20ac19c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.071+0000 7f20b1ef6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac101750 0x7f20ac19cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:16.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.071+0000 7f20b1ef6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac19d150 con 0x7f20ac101750 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.071+0000 7f20b1ef6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac196600 con 0x7f20ac100da0 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.071+0000 7f20aaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac101750 0x7f20ac19cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.071+0000 7f20aaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac101750 0x7f20ac19cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45368/0 (socket says 172.21.15.121:45368) 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.071+0000 7f20aaffd700 1 -- 172.21.15.121:0/1858030912 learned_addr learned my addr 172.21.15.121:0/1858030912 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.071+0000 7f20ab7fe700 1 --2- 172.21.15.121:0/1858030912 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20ac100da0 0x7f20ac19c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.071+0000 7f20aaffd700 1 -- 172.21.15.121:0/1858030912 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20ac100da0 msgr2=0x7f20ac19c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.071+0000 7f20aaffd700 1 --2- 172.21.15.121:0/1858030912 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20ac100da0 0x7f20ac19c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.071+0000 7f20aaffd700 1 -- 172.21.15.121:0/1858030912 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209c0095f0 con 0x7f20ac101750 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.072+0000 7f20aaffd700 1 --2- 172.21.15.121:0/1858030912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac101750 0x7f20ac19cac0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f20a000b480 tx=0x7f20a000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.072+0000 7f20a8ff9700 1 -- 172.21.15.121:0/1858030912 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20a00118c0 con 0x7f20ac101750 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.072+0000 7f20a8ff9700 1 -- 172.21.15.121:0/1858030912 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20a0010380 con 0x7f20ac101750 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.072+0000 7f20a8ff9700 1 -- 172.21.15.121:0/1858030912 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20a000f510 con 0x7f20ac101750 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.072+0000 7f20b1ef6700 1 -- 172.21.15.121:0/1858030912 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac1968e0 con 0x7f20ac101750 2024-04-05T21:35:16.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.072+0000 7f20b1ef6700 1 -- 172.21.15.121:0/1858030912 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac196e30 con 0x7f20ac101750 2024-04-05T21:35:16.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.073+0000 7f20b1ef6700 1 -- 172.21.15.121:0/1858030912 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ac109140 con 0x7f20ac101750 2024-04-05T21:35:16.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.073+0000 7f20a8ff9700 1 -- 172.21.15.121:0/1858030912 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f20a00104f0 con 0x7f20ac101750 2024-04-05T21:35:16.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.074+0000 7f20a8ff9700 1 --2- 172.21.15.121:0/1858030912 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f209406cce0 0x7f209406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:16.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.074+0000 7f20ab7fe700 1 --2- 172.21.15.121:0/1858030912 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f209406cce0 0x7f209406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:16.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.074+0000 7f20a8ff9700 1 -- 172.21.15.121:0/1858030912 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f20a008c130 con 0x7f20ac101750 2024-04-05T21:35:16.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.074+0000 7f20ab7fe700 1 --2- 172.21.15.121:0/1858030912 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f209406cce0 0x7f209406f1a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f209c006010 tx=0x7f209c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:16.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.079+0000 7f20a8ff9700 1 -- 172.21.15.121:0/1858030912 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f20a0059cb0 con 0x7f20ac101750 2024-04-05T21:35:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.294+0000 7f20b1ef6700 1 -- 172.21.15.121:0/1858030912 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f20ac197180 con 0x7f209406cce0 2024-04-05T21:35:16.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.295+0000 7f20a8ff9700 1 -- 172.21.15.121:0/1858030912 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f20ac197180 con 0x7f209406cce0 2024-04-05T21:35:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.297+0000 7f20b1ef6700 1 -- 172.21.15.121:0/1858030912 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f209406cce0 msgr2=0x7f209406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.297+0000 7f20b1ef6700 1 --2- 172.21.15.121:0/1858030912 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f209406cce0 0x7f209406f1a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f209c006010 tx=0x7f209c01a040 comp rx=0 tx=0).stop 2024-04-05T21:35:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.297+0000 7f20b1ef6700 1 -- 172.21.15.121:0/1858030912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac101750 msgr2=0x7f20ac19cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.297+0000 7f20b1ef6700 1 --2- 172.21.15.121:0/1858030912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac101750 0x7f20ac19cac0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f20a000b480 tx=0x7f20a000b840 comp rx=0 tx=0).stop 2024-04-05T21:35:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.297+0000 7f20b1ef6700 1 -- 172.21.15.121:0/1858030912 shutdown_connections 2024-04-05T21:35:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.297+0000 7f20b1ef6700 1 --2- 172.21.15.121:0/1858030912 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f209406cce0 0x7f209406f1a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.297+0000 7f20b1ef6700 1 --2- 172.21.15.121:0/1858030912 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20ac100da0 0x7f20ac19c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.297+0000 7f20b1ef6700 1 --2- 172.21.15.121:0/1858030912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac101750 0x7f20ac19cac0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.297+0000 7f20b1ef6700 1 -- 172.21.15.121:0/1858030912 >> 172.21.15.121:0/1858030912 conn(0x7f20ac0fc630 msgr2=0x7f20ac0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:16.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.297+0000 7f20b1ef6700 1 -- 172.21.15.121:0/1858030912 shutdown_connections 2024-04-05T21:35:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.297+0000 7f20b1ef6700 1 -- 172.21.15.121:0/1858030912 wait complete. 2024-04-05T21:35:16.308 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:35:16.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.405+0000 7f0990347700 1 -- 172.21.15.121:0/495273933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0988100dc0 msgr2=0x7f09881011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:16.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.405+0000 7f0990347700 1 --2- 172.21.15.121:0/495273933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0988100dc0 0x7f09881011a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0978009910 tx=0x7f0978009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:16.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.405+0000 7f0990347700 1 -- 172.21.15.121:0/495273933 shutdown_connections 2024-04-05T21:35:16.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.405+0000 7f0990347700 1 --2- 172.21.15.121:0/495273933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0988101770 0x7f09881057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.405+0000 7f0990347700 1 --2- 172.21.15.121:0/495273933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0988100dc0 0x7f09881011a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.405+0000 7f0990347700 1 -- 172.21.15.121:0/495273933 >> 172.21.15.121:0/495273933 conn(0x7f09880fc670 msgr2=0x7f09880fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:16.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.405+0000 7f0990347700 1 -- 172.21.15.121:0/495273933 shutdown_connections 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.405+0000 7f0990347700 1 -- 172.21.15.121:0/495273933 wait complete. 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f0990347700 1 Processor -- start 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f0990347700 1 -- start start 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f0990347700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0988100dc0 0x7f0988196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f0990347700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0988101770 0x7f0988196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f0990347700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09881972e0 con 0x7f0988100dc0 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f0990347700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098819a010 con 0x7f0988101770 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f098e0e3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0988100dc0 0x7f0988196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f098e0e3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0988100dc0 0x7f0988196710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45376/0 (socket says 172.21.15.121:45376) 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f098e0e3700 1 -- 172.21.15.121:0/446796119 learned_addr learned my addr 172.21.15.121:0/446796119 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f098d8e2700 1 --2- 172.21.15.121:0/446796119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0988101770 0x7f0988196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f098e0e3700 1 -- 172.21.15.121:0/446796119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0988101770 msgr2=0x7f0988196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f098e0e3700 1 --2- 172.21.15.121:0/446796119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0988101770 0x7f0988196c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f098e0e3700 1 -- 172.21.15.121:0/446796119 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09780095f0 con 0x7f0988100dc0 2024-04-05T21:35:16.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f098d8e2700 1 --2- 172.21.15.121:0/446796119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0988101770 0x7f0988196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:35:16.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f098e0e3700 1 --2- 172.21.15.121:0/446796119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0988100dc0 0x7f0988196710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0978006010 tx=0x7f097800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:16.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.406+0000 7f097f7fe700 1 -- 172.21.15.121:0/446796119 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f097801d070 con 0x7f0988100dc0 2024-04-05T21:35:16.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.407+0000 7f097f7fe700 1 -- 172.21.15.121:0/446796119 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f097800f460 con 0x7f0988100dc0 2024-04-05T21:35:16.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.407+0000 7f0990347700 1 -- 172.21.15.121:0/446796119 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098819a290 con 0x7f0988100dc0 2024-04-05T21:35:16.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.407+0000 7f097f7fe700 1 -- 172.21.15.121:0/446796119 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0978021640 con 0x7f0988100dc0 2024-04-05T21:35:16.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.407+0000 7f0990347700 1 -- 172.21.15.121:0/446796119 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098819a780 con 0x7f0988100dc0 2024-04-05T21:35:16.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.407+0000 7f0990347700 1 -- 172.21.15.121:0/446796119 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0988109160 con 0x7f0988100dc0 2024-04-05T21:35:16.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.408+0000 7f097f7fe700 1 -- 172.21.15.121:0/446796119 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f097800f9e0 con 0x7f0988100dc0 2024-04-05T21:35:16.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.408+0000 7f097f7fe700 1 --2- 172.21.15.121:0/446796119 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f097406cce0 0x7f097406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:16.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.408+0000 7f097f7fe700 1 -- 172.21.15.121:0/446796119 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f097808d900 con 0x7f0988100dc0 2024-04-05T21:35:16.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.408+0000 7f098d8e2700 1 --2- 172.21.15.121:0/446796119 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f097406cce0 0x7f097406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:16.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.409+0000 7f098d8e2700 1 --2- 172.21.15.121:0/446796119 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f097406cce0 0x7f097406f1a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0984005fd0 tx=0x7f0984005dc0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:16.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.413+0000 7f097f7fe700 1 -- 172.21.15.121:0/446796119 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0978057970 con 0x7f0988100dc0 2024-04-05T21:35:16.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.634+0000 7f0990347700 1 -- 172.21.15.121:0/446796119 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f098819a9a0 con 0x7f097406cce0 2024-04-05T21:35:16.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.635+0000 7f097f7fe700 1 -- 172.21.15.121:0/446796119 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f098819a9a0 con 0x7f097406cce0 2024-04-05T21:35:16.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.637+0000 7f0990347700 1 -- 172.21.15.121:0/446796119 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f097406cce0 msgr2=0x7f097406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:16.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.637+0000 7f0990347700 1 --2- 172.21.15.121:0/446796119 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f097406cce0 0x7f097406f1a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0984005fd0 tx=0x7f0984005dc0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.637+0000 7f0990347700 1 -- 172.21.15.121:0/446796119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0988100dc0 msgr2=0x7f0988196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.637+0000 7f0990347700 1 --2- 172.21.15.121:0/446796119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0988100dc0 0x7f0988196710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0978006010 tx=0x7f097800be20 comp rx=0 tx=0).stop 2024-04-05T21:35:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.637+0000 7f0990347700 1 -- 172.21.15.121:0/446796119 shutdown_connections 2024-04-05T21:35:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.637+0000 7f0990347700 1 --2- 172.21.15.121:0/446796119 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f097406cce0 0x7f097406f1a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.637+0000 7f0990347700 1 --2- 172.21.15.121:0/446796119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0988100dc0 0x7f0988196710 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.637+0000 7f0990347700 1 --2- 172.21.15.121:0/446796119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0988101770 0x7f0988196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.637+0000 7f0990347700 1 -- 172.21.15.121:0/446796119 >> 172.21.15.121:0/446796119 conn(0x7f09880fc670 msgr2=0x7f09880fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.638+0000 7f0990347700 1 -- 172.21.15.121:0/446796119 shutdown_connections 2024-04-05T21:35:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.638+0000 7f0990347700 1 -- 172.21.15.121:0/446796119 wait complete. 2024-04-05T21:35:16.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.741+0000 7f1aadae8700 1 -- 172.21.15.121:0/2295479001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8100dc0 msgr2=0x7f1aa81011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:16.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.741+0000 7f1aadae8700 1 --2- 172.21.15.121:0/2295479001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8100dc0 0x7f1aa81011a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1a90009910 tx=0x7f1a90009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:16.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.741+0000 7f1aadae8700 1 -- 172.21.15.121:0/2295479001 shutdown_connections 2024-04-05T21:35:16.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.741+0000 7f1aadae8700 1 --2- 172.21.15.121:0/2295479001 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aa8101770 0x7f1aa81057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.741+0000 7f1aadae8700 1 --2- 172.21.15.121:0/2295479001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8100dc0 0x7f1aa81011a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.741+0000 7f1aadae8700 1 -- 172.21.15.121:0/2295479001 >> 172.21.15.121:0/2295479001 conn(0x7f1aa80fc670 msgr2=0x7f1aa80fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:16.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.741+0000 7f1aadae8700 1 -- 172.21.15.121:0/2295479001 shutdown_connections 2024-04-05T21:35:16.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.741+0000 7f1aadae8700 1 -- 172.21.15.121:0/2295479001 wait complete. 2024-04-05T21:35:16.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.742+0000 7f1aadae8700 1 Processor -- start 2024-04-05T21:35:16.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.742+0000 7f1aadae8700 1 -- start start 2024-04-05T21:35:16.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.742+0000 7f1aadae8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aa8100dc0 0x7f1aa8196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.742+0000 7f1aadae8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8101770 0x7f1aa8196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.742+0000 7f1aadae8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa81972e0 con 0x7f1aa8101770 2024-04-05T21:35:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.742+0000 7f1aadae8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa819a010 con 0x7f1aa8100dc0 2024-04-05T21:35:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.742+0000 7f1aa6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8101770 0x7f1aa8196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.742+0000 7f1aa6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8101770 0x7f1aa8196c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45394/0 (socket says 172.21.15.121:45394) 2024-04-05T21:35:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.742+0000 7f1aa6ffd700 1 -- 172.21.15.121:0/1832449397 learned_addr learned my addr 172.21.15.121:0/1832449397 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.742+0000 7f1aa77fe700 1 --2- 172.21.15.121:0/1832449397 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aa8100dc0 0x7f1aa8196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.743+0000 7f1aa6ffd700 1 -- 172.21.15.121:0/1832449397 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aa8100dc0 msgr2=0x7f1aa8196710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.743+0000 7f1aa6ffd700 1 --2- 172.21.15.121:0/1832449397 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aa8100dc0 0x7f1aa8196710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.743+0000 7f1aa6ffd700 1 -- 172.21.15.121:0/1832449397 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a900095f0 con 0x7f1aa8101770 2024-04-05T21:35:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.743+0000 7f1aa6ffd700 1 --2- 172.21.15.121:0/1832449397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8101770 0x7f1aa8196c50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800d8d0 tx=0x7f1a9800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:16.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.743+0000 7f1aa4ff9700 1 -- 172.21.15.121:0/1832449397 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a980041f0 con 0x7f1aa8101770 2024-04-05T21:35:16.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.743+0000 7f1aadae8700 1 -- 172.21.15.121:0/1832449397 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa819a2f0 con 0x7f1aa8101770 2024-04-05T21:35:16.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.743+0000 7f1aa4ff9700 1 -- 172.21.15.121:0/1832449397 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a98007620 con 0x7f1aa8101770 2024-04-05T21:35:16.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.743+0000 7f1aa4ff9700 1 -- 172.21.15.121:0/1832449397 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a98005610 con 0x7f1aa8101770 2024-04-05T21:35:16.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.743+0000 7f1aadae8700 1 -- 172.21.15.121:0/1832449397 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa819a840 con 0x7f1aa8101770 2024-04-05T21:35:16.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.744+0000 7f1aadae8700 1 -- 172.21.15.121:0/1832449397 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa8109160 con 0x7f1aa8101770 2024-04-05T21:35:16.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.745+0000 7f1aa4ff9700 1 -- 172.21.15.121:0/1832449397 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1a98004750 con 0x7f1aa8101770 2024-04-05T21:35:16.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.745+0000 7f1aa4ff9700 1 --2- 172.21.15.121:0/1832449397 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a9406cce0 0x7f1a9406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:16.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.745+0000 7f1aa4ff9700 1 -- 172.21.15.121:0/1832449397 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1a9808c930 con 0x7f1aa8101770 2024-04-05T21:35:16.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.745+0000 7f1aa77fe700 1 --2- 172.21.15.121:0/1832449397 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a9406cce0 0x7f1a9406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:16.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.746+0000 7f1aa77fe700 1 --2- 172.21.15.121:0/1832449397 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a9406cce0 0x7f1a9406f1a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1a90006010 tx=0x7f1a9001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:16.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.749+0000 7f1aa4ff9700 1 -- 172.21.15.121:0/1832449397 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1a98056a20 con 0x7f1aa8101770 2024-04-05T21:35:16.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.953+0000 7f1aadae8700 1 -- 172.21.15.121:0/1832449397 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1aa819aad0 con 0x7f1a9406cce0 2024-04-05T21:35:16.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.963+0000 7f1aa4ff9700 1 -- 172.21.15.121:0/1832449397 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f1aa819aad0 con 0x7f1a9406cce0 2024-04-05T21:35:16.964 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:35:16.964 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (6m) 4s ago 17m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:35:16.964 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (17m) 4s ago 17m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (16m) 45s ago 16m 8334k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (4m) 4s ago 17m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (4m) 45s ago 16m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (5m) 4s ago 17m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (30s) 4s ago 11m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (10s) 4s ago 11m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (11m) 45s ago 11m 110M - 18.2.1 5be31c24972a a5ec30a5b214 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (11m) 45s ago 11m 62.4M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (8m) 4s ago 19m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (7m) 45s ago 16m 452M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (4m) 4s ago 19m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (4m) 45s ago 16m 38.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (6m) 4s ago 17m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:35:16.965 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (6m) 45s ago 16m 16.9M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:35:16.966 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (3m) 4s ago 15m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:35:16.966 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (3m) 4s ago 15m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:35:16.966 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (2m) 4s ago 14m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:35:16.966 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (118s) 45s ago 14m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:35:16.966 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (83s) 45s ago 13m 95.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:35:16.966 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (49s) 45s ago 13m 11.5M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:35:16.966 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (6m) 4s ago 16m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:35:16.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.965+0000 7f1aadae8700 1 -- 172.21.15.121:0/1832449397 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a9406cce0 msgr2=0x7f1a9406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:16.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.965+0000 7f1aadae8700 1 --2- 172.21.15.121:0/1832449397 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a9406cce0 0x7f1a9406f1a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1a90006010 tx=0x7f1a9001a040 comp rx=0 tx=0).stop 2024-04-05T21:35:16.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.965+0000 7f1aadae8700 1 -- 172.21.15.121:0/1832449397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8101770 msgr2=0x7f1aa8196c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:16.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.965+0000 7f1aadae8700 1 --2- 172.21.15.121:0/1832449397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8101770 0x7f1aa8196c50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800d8d0 tx=0x7f1a9800dbe0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.966+0000 7f1aadae8700 1 -- 172.21.15.121:0/1832449397 shutdown_connections 2024-04-05T21:35:16.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.966+0000 7f1aadae8700 1 --2- 172.21.15.121:0/1832449397 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a9406cce0 0x7f1a9406f1a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.966+0000 7f1aadae8700 1 --2- 172.21.15.121:0/1832449397 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aa8100dc0 0x7f1aa8196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.966+0000 7f1aadae8700 1 --2- 172.21.15.121:0/1832449397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8101770 0x7f1aa8196c50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:16.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.966+0000 7f1aadae8700 1 -- 172.21.15.121:0/1832449397 >> 172.21.15.121:0/1832449397 conn(0x7f1aa80fc670 msgr2=0x7f1aa80fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:16.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.966+0000 7f1aadae8700 1 -- 172.21.15.121:0/1832449397 shutdown_connections 2024-04-05T21:35:16.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:16.966+0000 7f1aadae8700 1 -- 172.21.15.121:0/1832449397 wait complete. 2024-04-05T21:35:17.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.074+0000 7fef2da9e700 1 -- 172.21.15.121:0/809660332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef28068670 msgr2=0x7fef28068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:17.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.074+0000 7fef2da9e700 1 --2- 172.21.15.121:0/809660332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef28068670 0x7fef28068a50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fef18009910 tx=0x7fef18009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:17.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.075+0000 7fef2da9e700 1 -- 172.21.15.121:0/809660332 shutdown_connections 2024-04-05T21:35:17.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.075+0000 7fef2da9e700 1 --2- 172.21.15.121:0/809660332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef28069020 0x7fef28105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.075+0000 7fef2da9e700 1 --2- 172.21.15.121:0/809660332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef28068670 0x7fef28068a50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.075+0000 7fef2da9e700 1 -- 172.21.15.121:0/809660332 >> 172.21.15.121:0/809660332 conn(0x7fef280755a0 msgr2=0x7fef280759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:17.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.075+0000 7fef2da9e700 1 -- 172.21.15.121:0/809660332 shutdown_connections 2024-04-05T21:35:17.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.075+0000 7fef2da9e700 1 -- 172.21.15.121:0/809660332 wait complete. 2024-04-05T21:35:17.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.075+0000 7fef2da9e700 1 Processor -- start 2024-04-05T21:35:17.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.075+0000 7fef2da9e700 1 -- start start 2024-04-05T21:35:17.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef2da9e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef28068670 0x7fef28196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:17.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef2da9e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef28069020 0x7fef28196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:17.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef2da9e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef28197290 con 0x7fef28068670 2024-04-05T21:35:17.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef2da9e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2819afb0 con 0x7fef28069020 2024-04-05T21:35:17.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef277fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef28068670 0x7fef28196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:17.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef277fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef28068670 0x7fef28196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45426/0 (socket says 172.21.15.121:45426) 2024-04-05T21:35:17.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef277fe700 1 -- 172.21.15.121:0/3785115188 learned_addr learned my addr 172.21.15.121:0/3785115188 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:17.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef26ffd700 1 --2- 172.21.15.121:0/3785115188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef28069020 0x7fef28196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:17.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef277fe700 1 -- 172.21.15.121:0/3785115188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef28069020 msgr2=0x7fef28196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:17.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef277fe700 1 --2- 172.21.15.121:0/3785115188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef28069020 0x7fef28196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef277fe700 1 -- 172.21.15.121:0/3785115188 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef180095f0 con 0x7fef28068670 2024-04-05T21:35:17.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef277fe700 1 --2- 172.21.15.121:0/3785115188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef28068670 0x7fef28196670 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fef18006010 tx=0x7fef1800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:17.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef24ff9700 1 -- 172.21.15.121:0/3785115188 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef1801d070 con 0x7fef28068670 2024-04-05T21:35:17.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.076+0000 7fef2da9e700 1 -- 172.21.15.121:0/3785115188 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2819b230 con 0x7fef28068670 2024-04-05T21:35:17.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.077+0000 7fef24ff9700 1 -- 172.21.15.121:0/3785115188 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef18005940 con 0x7fef28068670 2024-04-05T21:35:17.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.077+0000 7fef24ff9700 1 -- 172.21.15.121:0/3785115188 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef1800fd90 con 0x7fef28068670 2024-04-05T21:35:17.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.077+0000 7fef2da9e700 1 -- 172.21.15.121:0/3785115188 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2819b720 con 0x7fef28068670 2024-04-05T21:35:17.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.077+0000 7fef2da9e700 1 -- 172.21.15.121:0/3785115188 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef28109130 con 0x7fef28068670 2024-04-05T21:35:17.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.078+0000 7fef24ff9700 1 -- 172.21.15.121:0/3785115188 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fef1802b440 con 0x7fef28068670 2024-04-05T21:35:17.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.079+0000 7fef24ff9700 1 --2- 172.21.15.121:0/3785115188 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fef1006cce0 0x7fef1006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:17.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.079+0000 7fef24ff9700 1 -- 172.21.15.121:0/3785115188 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fef1808d750 con 0x7fef28068670 2024-04-05T21:35:17.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.079+0000 7fef26ffd700 1 --2- 172.21.15.121:0/3785115188 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fef1006cce0 0x7fef1006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:17.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.079+0000 7fef26ffd700 1 --2- 172.21.15.121:0/3785115188 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fef1006cce0 0x7fef1006f1a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fef28197c20 tx=0x7fef1c0073d0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:17.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.083+0000 7fef24ff9700 1 -- 172.21.15.121:0/3785115188 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fef18057720 con 0x7fef28068670 2024-04-05T21:35:17.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.366+0000 7fef2da9e700 1 -- 172.21.15.121:0/3785115188 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fef28108b10 con 0x7fef28068670 2024-04-05T21:35:17.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.367+0000 7fef24ff9700 1 -- 172.21.15.121:0/3785115188 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7fef1805ad40 con 0x7fef28068670 2024-04-05T21:35:17.368 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:35:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:35:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:35:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:35:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:35:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:35:17.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:35:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:35:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:35:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:35:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:35:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-04-05T21:35:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:35:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:35:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:35:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 2, 2024-04-05T21:35:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 12 2024-04-05T21:35:17.369 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:35:17.369 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:35:17.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.369+0000 7fef2da9e700 1 -- 172.21.15.121:0/3785115188 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fef1006cce0 msgr2=0x7fef1006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:17.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.369+0000 7fef2da9e700 1 --2- 172.21.15.121:0/3785115188 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fef1006cce0 0x7fef1006f1a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fef28197c20 tx=0x7fef1c0073d0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.369+0000 7fef2da9e700 1 -- 172.21.15.121:0/3785115188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef28068670 msgr2=0x7fef28196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:17.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.369+0000 7fef2da9e700 1 --2- 172.21.15.121:0/3785115188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef28068670 0x7fef28196670 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fef18006010 tx=0x7fef1800bd70 comp rx=0 tx=0).stop 2024-04-05T21:35:17.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.369+0000 7fef2da9e700 1 -- 172.21.15.121:0/3785115188 shutdown_connections 2024-04-05T21:35:17.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.369+0000 7fef2da9e700 1 --2- 172.21.15.121:0/3785115188 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fef1006cce0 0x7fef1006f1a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.369+0000 7fef2da9e700 1 --2- 172.21.15.121:0/3785115188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef28068670 0x7fef28196670 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.369+0000 7fef2da9e700 1 --2- 172.21.15.121:0/3785115188 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fef28069020 0x7fef28196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.369+0000 7fef2da9e700 1 -- 172.21.15.121:0/3785115188 >> 172.21.15.121:0/3785115188 conn(0x7fef280755a0 msgr2=0x7fef280fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:17.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.369+0000 7fef2da9e700 1 -- 172.21.15.121:0/3785115188 shutdown_connections 2024-04-05T21:35:17.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.369+0000 7fef2da9e700 1 -- 172.21.15.121:0/3785115188 wait complete. 2024-04-05T21:35:17.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.473+0000 7f3bea421700 1 -- 172.21.15.121:0/1744643609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3be4100df0 msgr2=0x7f3be41011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:17.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.473+0000 7f3bea421700 1 --2- 172.21.15.121:0/1744643609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3be4100df0 0x7f3be41011d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4009910 tx=0x7f3bd4009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:17.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.473+0000 7f3bea421700 1 -- 172.21.15.121:0/1744643609 shutdown_connections 2024-04-05T21:35:17.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.473+0000 7f3bea421700 1 --2- 172.21.15.121:0/1744643609 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be41017a0 0x7f3be41057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.473+0000 7f3bea421700 1 --2- 172.21.15.121:0/1744643609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3be4100df0 0x7f3be41011d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.473+0000 7f3bea421700 1 -- 172.21.15.121:0/1744643609 >> 172.21.15.121:0/1744643609 conn(0x7f3be40fc6a0 msgr2=0x7f3be40feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:17.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.473+0000 7f3bea421700 1 -- 172.21.15.121:0/1744643609 shutdown_connections 2024-04-05T21:35:17.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.473+0000 7f3bea421700 1 -- 172.21.15.121:0/1744643609 wait complete. 2024-04-05T21:35:17.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.474+0000 7f3bea421700 1 Processor -- start 2024-04-05T21:35:17.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.474+0000 7f3bea421700 1 -- start start 2024-04-05T21:35:17.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.474+0000 7f3bea421700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be4100df0 0x7f3be419c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:17.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.474+0000 7f3bea421700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3be41017a0 0x7f3be419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.474+0000 7f3bea421700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be419d110 con 0x7f3be41017a0 2024-04-05T21:35:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.474+0000 7f3bea421700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be41965c0 con 0x7f3be4100df0 2024-04-05T21:35:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.474+0000 7f3be37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3be41017a0 0x7f3be419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.474+0000 7f3be37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3be41017a0 0x7f3be419ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45442/0 (socket says 172.21.15.121:45442) 2024-04-05T21:35:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.474+0000 7f3be37fe700 1 -- 172.21.15.121:0/58271134 learned_addr learned my addr 172.21.15.121:0/58271134 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.475+0000 7f3be3fff700 1 --2- 172.21.15.121:0/58271134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be4100df0 0x7f3be419c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.475+0000 7f3be37fe700 1 -- 172.21.15.121:0/58271134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be4100df0 msgr2=0x7f3be419c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.475+0000 7f3be37fe700 1 --2- 172.21.15.121:0/58271134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be4100df0 0x7f3be419c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.475+0000 7f3be37fe700 1 -- 172.21.15.121:0/58271134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd40095f0 con 0x7f3be41017a0 2024-04-05T21:35:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.475+0000 7f3be37fe700 1 --2- 172.21.15.121:0/58271134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3be41017a0 0x7f3be419ca80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800b480 tx=0x7f3bd800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.475+0000 7f3be17fa700 1 -- 172.21.15.121:0/58271134 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bd80118c0 con 0x7f3be41017a0 2024-04-05T21:35:17.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.475+0000 7f3bea421700 1 -- 172.21.15.121:0/58271134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be41968a0 con 0x7f3be41017a0 2024-04-05T21:35:17.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.475+0000 7f3be17fa700 1 -- 172.21.15.121:0/58271134 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bd8010380 con 0x7f3be41017a0 2024-04-05T21:35:17.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.475+0000 7f3be17fa700 1 -- 172.21.15.121:0/58271134 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bd800f510 con 0x7f3be41017a0 2024-04-05T21:35:17.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.475+0000 7f3bea421700 1 -- 172.21.15.121:0/58271134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be4196df0 con 0x7f3be41017a0 2024-04-05T21:35:17.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.476+0000 7f3bea421700 1 -- 172.21.15.121:0/58271134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be4109190 con 0x7f3be41017a0 2024-04-05T21:35:17.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.477+0000 7f3be17fa700 1 -- 172.21.15.121:0/58271134 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3bd80104f0 con 0x7f3be41017a0 2024-04-05T21:35:17.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.477+0000 7f3be17fa700 1 --2- 172.21.15.121:0/58271134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3bcc06cce0 0x7f3bcc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:17.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.477+0000 7f3be17fa700 1 -- 172.21.15.121:0/58271134 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3bd808c210 con 0x7f3be41017a0 2024-04-05T21:35:17.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.477+0000 7f3be3fff700 1 --2- 172.21.15.121:0/58271134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3bcc06cce0 0x7f3bcc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:17.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.477+0000 7f3be3fff700 1 --2- 172.21.15.121:0/58271134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3bcc06cce0 0x7f3bcc06f1a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4006010 tx=0x7f3bd401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:17.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.481+0000 7f3be17fa700 1 -- 172.21.15.121:0/58271134 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3bd8059d10 con 0x7f3be41017a0 2024-04-05T21:35:17.679 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:17 smithi188 ceph-mon[107793]: pgmap v270: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:17.679 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:17 smithi188 ceph-mon[107793]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:17.680 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:17 smithi188 ceph-mon[107793]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:17.680 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:17.680 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:17.680 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:35:17.680 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:35:17.680 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:17.680 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:35:17.680 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:35:17.680 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:35:17.680 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:35:17.680 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:17 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:35:17.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.718+0000 7f3bea421700 1 -- 172.21.15.121:0/58271134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3be404ea90 con 0x7f3be41017a0 2024-04-05T21:35:17.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.718+0000 7f3be17fa700 1 -- 172.21.15.121:0/58271134 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f3bd80107a0 con 0x7f3be41017a0 2024-04-05T21:35:17.719 INFO:teuthology.orchestra.run.smithi121.stdout:e28 2024-04-05T21:35:17.719 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:35:17.719 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:35:17.719 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:35:17.719 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:35:17.719 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:35:17.719 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:35:17.720 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:35:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:35:17.722 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:35:17.722 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:35:17.722 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:35:17.722 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44255} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2257429686,v1:172.21.15.188:6825/2257429686] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:35:17.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.720+0000 7f3bea421700 1 -- 172.21.15.121:0/58271134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3bcc06cce0 msgr2=0x7f3bcc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:17.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.720+0000 7f3bea421700 1 --2- 172.21.15.121:0/58271134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3bcc06cce0 0x7f3bcc06f1a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4006010 tx=0x7f3bd401a040 comp rx=0 tx=0).stop 2024-04-05T21:35:17.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.720+0000 7f3bea421700 1 -- 172.21.15.121:0/58271134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3be41017a0 msgr2=0x7f3be419ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:17.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.720+0000 7f3bea421700 1 --2- 172.21.15.121:0/58271134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3be41017a0 0x7f3be419ca80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800b480 tx=0x7f3bd800b840 comp rx=0 tx=0).stop 2024-04-05T21:35:17.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.720+0000 7f3bea421700 1 -- 172.21.15.121:0/58271134 shutdown_connections 2024-04-05T21:35:17.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.720+0000 7f3bea421700 1 --2- 172.21.15.121:0/58271134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3bcc06cce0 0x7f3bcc06f1a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.720+0000 7f3bea421700 1 --2- 172.21.15.121:0/58271134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3be4100df0 0x7f3be419c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.720+0000 7f3bea421700 1 --2- 172.21.15.121:0/58271134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3be41017a0 0x7f3be419ca80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.720+0000 7f3bea421700 1 -- 172.21.15.121:0/58271134 >> 172.21.15.121:0/58271134 conn(0x7f3be40fc6a0 msgr2=0x7f3be40fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:17.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.721+0000 7f3bea421700 1 -- 172.21.15.121:0/58271134 shutdown_connections 2024-04-05T21:35:17.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.721+0000 7f3bea421700 1 -- 172.21.15.121:0/58271134 wait complete. 2024-04-05T21:35:17.723 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 28 2024-04-05T21:35:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:17 smithi121 ceph-mon[145859]: pgmap v270: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:17 smithi121 ceph-mon[145859]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:17 smithi121 ceph-mon[145859]: from='client.34372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:35:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:35:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:35:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:35:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:35:17.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:35:17.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:17 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:35:17.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.824+0000 7f41747b8700 1 -- 172.21.15.121:0/1392186996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c101790 msgr2=0x7f416c1057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:17.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.824+0000 7f41747b8700 1 --2- 172.21.15.121:0/1392186996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c101790 0x7f416c1057e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4168009910 tx=0x7f4168009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:17.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.824+0000 7f41747b8700 1 -- 172.21.15.121:0/1392186996 shutdown_connections 2024-04-05T21:35:17.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.824+0000 7f41747b8700 1 --2- 172.21.15.121:0/1392186996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c101790 0x7f416c1057e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.824+0000 7f41747b8700 1 --2- 172.21.15.121:0/1392186996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f416c100de0 0x7f416c1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.824+0000 7f41747b8700 1 -- 172.21.15.121:0/1392186996 >> 172.21.15.121:0/1392186996 conn(0x7f416c0fc670 msgr2=0x7f416c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:17.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.824+0000 7f41747b8700 1 -- 172.21.15.121:0/1392186996 shutdown_connections 2024-04-05T21:35:17.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.824+0000 7f41747b8700 1 -- 172.21.15.121:0/1392186996 wait complete. 2024-04-05T21:35:17.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.825+0000 7f41747b8700 1 Processor -- start 2024-04-05T21:35:17.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.825+0000 7f41747b8700 1 -- start start 2024-04-05T21:35:17.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.825+0000 7f41747b8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c100de0 0x7f416c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:17.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.825+0000 7f41747b8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f416c101790 0x7f416c198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:17.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.825+0000 7f41747b8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c199560 con 0x7f416c100de0 2024-04-05T21:35:17.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.825+0000 7f41747b8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c19d370 con 0x7f416c101790 2024-04-05T21:35:17.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.825+0000 7f4172554700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c100de0 0x7f416c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:17.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.825+0000 7f4172554700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c100de0 0x7f416c198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45452/0 (socket says 172.21.15.121:45452) 2024-04-05T21:35:17.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.825+0000 7f4172554700 1 -- 172.21.15.121:0/3926798664 learned_addr learned my addr 172.21.15.121:0/3926798664 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:17.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.826+0000 7f4171d53700 1 --2- 172.21.15.121:0/3926798664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f416c101790 0x7f416c198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:17.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.826+0000 7f4172554700 1 -- 172.21.15.121:0/3926798664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f416c101790 msgr2=0x7f416c198ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:17.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.826+0000 7f4172554700 1 --2- 172.21.15.121:0/3926798664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f416c101790 0x7f416c198ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:17.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.826+0000 7f4172554700 1 -- 172.21.15.121:0/3926798664 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41680095f0 con 0x7f416c100de0 2024-04-05T21:35:17.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.826+0000 7f4171d53700 1 --2- 172.21.15.121:0/3926798664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f416c101790 0x7f416c198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:35:17.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.826+0000 7f4172554700 1 --2- 172.21.15.121:0/3926798664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c100de0 0x7f416c198990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f415c00b4b0 tx=0x7f415c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:17.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.826+0000 7f41637fe700 1 -- 172.21.15.121:0/3926798664 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f415c011800 con 0x7f416c100de0 2024-04-05T21:35:17.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.826+0000 7f41637fe700 1 -- 172.21.15.121:0/3926798664 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f415c011e00 con 0x7f416c100de0 2024-04-05T21:35:17.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.826+0000 7f41747b8700 1 -- 172.21.15.121:0/3926798664 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c19d650 con 0x7f416c100de0 2024-04-05T21:35:17.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.826+0000 7f41637fe700 1 -- 172.21.15.121:0/3926798664 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f415c010e50 con 0x7f416c100de0 2024-04-05T21:35:17.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.826+0000 7f41747b8700 1 -- 172.21.15.121:0/3926798664 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416c19db70 con 0x7f416c100de0 2024-04-05T21:35:17.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.827+0000 7f41747b8700 1 -- 172.21.15.121:0/3926798664 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416c104f60 con 0x7f416c100de0 2024-04-05T21:35:17.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.827+0000 7f41637fe700 1 -- 172.21.15.121:0/3926798664 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f415c010380 con 0x7f416c100de0 2024-04-05T21:35:17.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.828+0000 7f41637fe700 1 --2- 172.21.15.121:0/3926798664 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f415806cce0 0x7f415806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:17.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.828+0000 7f4171d53700 1 --2- 172.21.15.121:0/3926798664 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f415806cce0 0x7f415806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:17.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.828+0000 7f41637fe700 1 -- 172.21.15.121:0/3926798664 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f415c08c190 con 0x7f416c100de0 2024-04-05T21:35:17.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.828+0000 7f4171d53700 1 --2- 172.21.15.121:0/3926798664 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f415806cce0 0x7f415806f1a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f416c199900 tx=0x7f416801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:17.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:17.832+0000 7f41637fe700 1 -- 172.21.15.121:0/3926798664 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f415c059c90 con 0x7f416c100de0 2024-04-05T21:35:18.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.042+0000 7f41747b8700 1 -- 172.21.15.121:0/3926798664 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f416c19ddf0 con 0x7f415806cce0 2024-04-05T21:35:18.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.043+0000 7f41637fe700 1 -- 172.21.15.121:0/3926798664 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f416c19ddf0 con 0x7f415806cce0 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "14/23 daemons upgraded", 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:35:18.044 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:35:18.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.045+0000 7f41747b8700 1 -- 172.21.15.121:0/3926798664 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f415806cce0 msgr2=0x7f415806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:18.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.045+0000 7f41747b8700 1 --2- 172.21.15.121:0/3926798664 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f415806cce0 0x7f415806f1a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f416c199900 tx=0x7f416801a040 comp rx=0 tx=0).stop 2024-04-05T21:35:18.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.045+0000 7f41747b8700 1 -- 172.21.15.121:0/3926798664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c100de0 msgr2=0x7f416c198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:18.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.045+0000 7f41747b8700 1 --2- 172.21.15.121:0/3926798664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c100de0 0x7f416c198990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f415c00b4b0 tx=0x7f415c00b870 comp rx=0 tx=0).stop 2024-04-05T21:35:18.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.045+0000 7f41747b8700 1 -- 172.21.15.121:0/3926798664 shutdown_connections 2024-04-05T21:35:18.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.045+0000 7f41747b8700 1 --2- 172.21.15.121:0/3926798664 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f415806cce0 0x7f415806f1a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:18.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.045+0000 7f41747b8700 1 --2- 172.21.15.121:0/3926798664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c100de0 0x7f416c198990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:18.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.045+0000 7f41747b8700 1 --2- 172.21.15.121:0/3926798664 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f416c101790 0x7f416c198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:18.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.045+0000 7f41747b8700 1 -- 172.21.15.121:0/3926798664 >> 172.21.15.121:0/3926798664 conn(0x7f416c0fc670 msgr2=0x7f416c0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:18.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.045+0000 7f41747b8700 1 -- 172.21.15.121:0/3926798664 shutdown_connections 2024-04-05T21:35:18.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.045+0000 7f41747b8700 1 -- 172.21.15.121:0/3926798664 wait complete. 2024-04-05T21:35:18.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.147+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2365578092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee780ff400 msgr2=0x7fee780ff7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:18.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.147+0000 7fee7d0f7700 1 --2- 172.21.15.121:0/2365578092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee780ff400 0x7fee780ff7e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fee6c009910 tx=0x7fee6c009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:18.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.147+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2365578092 shutdown_connections 2024-04-05T21:35:18.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.147+0000 7fee7d0f7700 1 --2- 172.21.15.121:0/2365578092 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee780ffdb0 0x7fee78103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:18.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.147+0000 7fee7d0f7700 1 --2- 172.21.15.121:0/2365578092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee780ff400 0x7fee780ff7e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:18.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.147+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2365578092 >> 172.21.15.121:0/2365578092 conn(0x7fee78075e30 msgr2=0x7fee780fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:18.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.147+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2365578092 shutdown_connections 2024-04-05T21:35:18.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.147+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2365578092 wait complete. 2024-04-05T21:35:18.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.148+0000 7fee7d0f7700 1 Processor -- start 2024-04-05T21:35:18.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.148+0000 7fee7d0f7700 1 -- start start 2024-04-05T21:35:18.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.148+0000 7fee7d0f7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee780ff400 0x7fee780fb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.148+0000 7fee7d0f7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee780ffdb0 0x7fee780fb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.148+0000 7fee7d0f7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee780fbdb0 con 0x7fee780ffdb0 2024-04-05T21:35:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.148+0000 7fee7d0f7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee780fbf20 con 0x7fee780ff400 2024-04-05T21:35:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.149+0000 7fee7659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee780ffdb0 0x7fee780fb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.149+0000 7fee7659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee780ffdb0 0x7fee780fb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45468/0 (socket says 172.21.15.121:45468) 2024-04-05T21:35:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.149+0000 7fee7659c700 1 -- 172.21.15.121:0/2301057530 learned_addr learned my addr 172.21.15.121:0/2301057530 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.149+0000 7fee76d9d700 1 --2- 172.21.15.121:0/2301057530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee780ff400 0x7fee780fb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.149+0000 7fee7659c700 1 -- 172.21.15.121:0/2301057530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee780ff400 msgr2=0x7fee780fb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.149+0000 7fee7659c700 1 --2- 172.21.15.121:0/2301057530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee780ff400 0x7fee780fb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.149+0000 7fee7659c700 1 -- 172.21.15.121:0/2301057530 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c0095f0 con 0x7fee780ffdb0 2024-04-05T21:35:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.149+0000 7fee7659c700 1 --2- 172.21.15.121:0/2301057530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee780ffdb0 0x7fee780fb7e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fee6800e8f0 tx=0x7fee6800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.149+0000 7fee5ffff700 1 -- 172.21.15.121:0/2301057530 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee68009790 con 0x7fee780ffdb0 2024-04-05T21:35:18.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.149+0000 7fee5ffff700 1 -- 172.21.15.121:0/2301057530 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee68004cd0 con 0x7fee780ffdb0 2024-04-05T21:35:18.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.149+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2301057530 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee781a28b0 con 0x7fee780ffdb0 2024-04-05T21:35:18.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.149+0000 7fee5ffff700 1 -- 172.21.15.121:0/2301057530 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee68005580 con 0x7fee780ffdb0 2024-04-05T21:35:18.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.149+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2301057530 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee781a2d20 con 0x7fee780ffdb0 2024-04-05T21:35:18.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.150+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2301057530 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee781097b0 con 0x7fee780ffdb0 2024-04-05T21:35:18.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.151+0000 7fee5ffff700 1 -- 172.21.15.121:0/2301057530 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fee68004750 con 0x7fee780ffdb0 2024-04-05T21:35:18.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.151+0000 7fee5ffff700 1 --2- 172.21.15.121:0/2301057530 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fee6006cce0 0x7fee6006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:18.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.151+0000 7fee76d9d700 1 --2- 172.21.15.121:0/2301057530 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fee6006cce0 0x7fee6006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:18.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.151+0000 7fee5ffff700 1 -- 172.21.15.121:0/2301057530 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fee68014070 con 0x7fee780ffdb0 2024-04-05T21:35:18.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.152+0000 7fee76d9d700 1 --2- 172.21.15.121:0/2301057530 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fee6006cce0 0x7fee6006f1a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fee6c000c00 tx=0x7fee6c009f70 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:18.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.156+0000 7fee5ffff700 1 -- 172.21.15.121:0/2301057530 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fee680569c0 con 0x7fee780ffdb0 2024-04-05T21:35:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.433+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2301057530 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fee7804ea90 con 0x7fee780ffdb0 2024-04-05T21:35:18.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.433+0000 7fee5ffff700 1 -- 172.21.15.121:0/2301057530 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fee68059fe0 con 0x7fee780ffdb0 2024-04-05T21:35:18.434 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:35:18.434 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:35:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:35:18.434 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:35:18.434 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:35:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.435+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2301057530 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fee6006cce0 msgr2=0x7fee6006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.435+0000 7fee7d0f7700 1 --2- 172.21.15.121:0/2301057530 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fee6006cce0 0x7fee6006f1a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fee6c000c00 tx=0x7fee6c009f70 comp rx=0 tx=0).stop 2024-04-05T21:35:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.435+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2301057530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee780ffdb0 msgr2=0x7fee780fb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.435+0000 7fee7d0f7700 1 --2- 172.21.15.121:0/2301057530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee780ffdb0 0x7fee780fb7e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fee6800e8f0 tx=0x7fee6800ecb0 comp rx=0 tx=0).stop 2024-04-05T21:35:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.435+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2301057530 shutdown_connections 2024-04-05T21:35:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.435+0000 7fee7d0f7700 1 --2- 172.21.15.121:0/2301057530 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fee6006cce0 0x7fee6006f1a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.435+0000 7fee7d0f7700 1 --2- 172.21.15.121:0/2301057530 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fee780ff400 0x7fee780fb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.435+0000 7fee7d0f7700 1 --2- 172.21.15.121:0/2301057530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee780ffdb0 0x7fee780fb7e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:18.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.435+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2301057530 >> 172.21.15.121:0/2301057530 conn(0x7fee78075e30 msgr2=0x7fee78103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.435+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2301057530 shutdown_connections 2024-04-05T21:35:18.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:18.435+0000 7fee7d0f7700 1 -- 172.21.15.121:0/2301057530 wait complete. 2024-04-05T21:35:18.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:18 smithi121 ceph-mon[145859]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:18.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:18 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3785115188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:35:18.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:18 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/58271134' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:35:18.747 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:18 smithi188 ceph-mon[107793]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:18.747 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:18 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3785115188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:35:18.748 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:18 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/58271134' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:35:19.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:19 smithi188 ceph-mon[107793]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:19.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:19 smithi188 ceph-mon[107793]: pgmap v271: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:19.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:19 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2301057530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:35:19.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:19 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:19.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:19 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi188.bjmisu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:35:19.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:19 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:35:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:19 smithi121 ceph-mon[145859]: from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:19 smithi121 ceph-mon[145859]: pgmap v271: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:19 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2301057530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:35:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:19 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:19 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi188.bjmisu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:35:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:19 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:35:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:20 smithi121 ceph-mon[145859]: Upgrade: Updating mds.cephfs.smithi188.bjmisu 2024-04-05T21:35:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:20 smithi121 ceph-mon[145859]: Deploying daemon mds.cephfs.smithi188.bjmisu on smithi188 2024-04-05T21:35:20.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:20 smithi188 ceph-mon[107793]: Upgrade: Updating mds.cephfs.smithi188.bjmisu 2024-04-05T21:35:20.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:20 smithi188 ceph-mon[107793]: Deploying daemon mds.cephfs.smithi188.bjmisu on smithi188 2024-04-05T21:35:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:21 smithi121 ceph-mon[145859]: pgmap v272: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:21 smithi121 ceph-mon[145859]: osdmap e83: 6 total, 6 up, 6 in 2024-04-05T21:35:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:21 smithi188 ceph-mon[107793]: pgmap v272: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:21 smithi188 ceph-mon[107793]: osdmap e83: 6 total, 6 up, 6 in 2024-04-05T21:35:22.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:22 smithi121 ceph-mon[145859]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T21:35:22.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:22 smithi188 ceph-mon[107793]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T21:35:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:23 smithi121 ceph-mon[145859]: pgmap v274: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:23 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T21:35:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:23 smithi188 ceph-mon[107793]: pgmap v274: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:23 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T21:35:25.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:25 smithi188 ceph-mon[107793]: pgmap v275: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:25.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:25 smithi121 ceph-mon[145859]: pgmap v275: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:27.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:27 smithi188 ceph-mon[107793]: pgmap v276: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:27.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:27.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:27.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:35:27.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:27 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] up:boot 2024-04-05T21:35:27.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:27 smithi188 ceph-mon[107793]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T21:35:27.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:27 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T21:35:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:27 smithi121 ceph-mon[145859]: pgmap v276: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:35:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:27 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] up:boot 2024-04-05T21:35:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:27 smithi121 ceph-mon[145859]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T21:35:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:27 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T21:35:29.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:29 smithi121 ceph-mon[145859]: pgmap v277: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:30.012 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:29 smithi188 ceph-mon[107793]: pgmap v277: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:31.441 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:31.442 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:31.442 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:35:31.442 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:31 smithi188 ceph-mon[107793]: pgmap v278: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:31.442 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:31.442 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:35:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:31 smithi121 ceph-mon[145859]: pgmap v278: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:31.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:31.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:35:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:33 smithi121 ceph-mon[145859]: pgmap v279: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:33.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:33 smithi188 ceph-mon[107793]: pgmap v279: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:35 smithi121 ceph-mon[145859]: pgmap v280: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:35 smithi188 ceph-mon[107793]: pgmap v280: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:37 smithi121 ceph-mon[145859]: pgmap v281: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:37.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:37 smithi188 ceph-mon[107793]: pgmap v281: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:39.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:39 smithi121 ceph-mon[145859]: pgmap v282: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:39.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:39 smithi188 ceph-mon[107793]: pgmap v282: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:41.677 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:41 smithi121 ceph-mon[145859]: pgmap v283: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:41.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:41 smithi188 ceph-mon[107793]: pgmap v283: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:43 smithi121 ceph-mon[145859]: pgmap v284: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:43.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:43 smithi188 ceph-mon[107793]: pgmap v284: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:45.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:45 smithi121 ceph-mon[145859]: pgmap v285: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:45.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:35:45.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:45 smithi188 ceph-mon[107793]: pgmap v285: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:45.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:35:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:47 smithi121 ceph-mon[145859]: pgmap v286: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:47 smithi188 ceph-mon[107793]: pgmap v286: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:48.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.552+0000 7ff490667700 1 -- 172.21.15.121:0/2162823408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff488105a50 msgr2=0x7ff488105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:48.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.552+0000 7ff490667700 1 --2- 172.21.15.121:0/2162823408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff488105a50 0x7ff488105e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff478009910 tx=0x7ff478009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:48.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.552+0000 7ff490667700 1 -- 172.21.15.121:0/2162823408 shutdown_connections 2024-04-05T21:35:48.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.552+0000 7ff490667700 1 --2- 172.21.15.121:0/2162823408 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4880ffa50 0x7ff4880ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:48.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.552+0000 7ff490667700 1 --2- 172.21.15.121:0/2162823408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff488105a50 0x7ff488105e30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:48.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.552+0000 7ff490667700 1 -- 172.21.15.121:0/2162823408 >> 172.21.15.121:0/2162823408 conn(0x7ff4880fb540 msgr2=0x7ff4880fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:48.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.552+0000 7ff490667700 1 -- 172.21.15.121:0/2162823408 shutdown_connections 2024-04-05T21:35:48.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.552+0000 7ff490667700 1 -- 172.21.15.121:0/2162823408 wait complete. 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.553+0000 7ff490667700 1 Processor -- start 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.553+0000 7ff490667700 1 -- start start 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.553+0000 7ff490667700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4880ffa50 0x7ff488196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.553+0000 7ff490667700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff488105a50 0x7ff488196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.553+0000 7ff490667700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff488197280 con 0x7ff488105a50 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.553+0000 7ff490667700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48819b010 con 0x7ff4880ffa50 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.553+0000 7ff48dc02700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff488105a50 0x7ff488196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.553+0000 7ff48dc02700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff488105a50 0x7ff488196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51500/0 (socket says 172.21.15.121:51500) 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.553+0000 7ff48dc02700 1 -- 172.21.15.121:0/3739072389 learned_addr learned my addr 172.21.15.121:0/3739072389 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.554+0000 7ff48e403700 1 --2- 172.21.15.121:0/3739072389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4880ffa50 0x7ff488196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.554+0000 7ff48dc02700 1 -- 172.21.15.121:0/3739072389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4880ffa50 msgr2=0x7ff488196660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.554+0000 7ff48dc02700 1 --2- 172.21.15.121:0/3739072389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4880ffa50 0x7ff488196660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.554+0000 7ff48dc02700 1 -- 172.21.15.121:0/3739072389 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4780095f0 con 0x7ff488105a50 2024-04-05T21:35:48.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.554+0000 7ff48dc02700 1 --2- 172.21.15.121:0/3739072389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff488105a50 0x7ff488196ba0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff48400d8d0 tx=0x7ff48400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:48.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.554+0000 7ff47f7fe700 1 -- 172.21.15.121:0/3739072389 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff484009940 con 0x7ff488105a50 2024-04-05T21:35:48.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.554+0000 7ff47f7fe700 1 -- 172.21.15.121:0/3739072389 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff484010460 con 0x7ff488105a50 2024-04-05T21:35:48.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.554+0000 7ff490667700 1 -- 172.21.15.121:0/3739072389 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48819b2f0 con 0x7ff488105a50 2024-04-05T21:35:48.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.554+0000 7ff47f7fe700 1 -- 172.21.15.121:0/3739072389 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff48400f5f0 con 0x7ff488105a50 2024-04-05T21:35:48.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.554+0000 7ff490667700 1 -- 172.21.15.121:0/3739072389 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48819b840 con 0x7ff488105a50 2024-04-05T21:35:48.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.555+0000 7ff490667700 1 -- 172.21.15.121:0/3739072389 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff48804ea90 con 0x7ff488105a50 2024-04-05T21:35:48.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.556+0000 7ff47f7fe700 1 -- 172.21.15.121:0/3739072389 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff4840105d0 con 0x7ff488105a50 2024-04-05T21:35:48.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.556+0000 7ff47f7fe700 1 --2- 172.21.15.121:0/3739072389 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff47406cce0 0x7ff47406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:48.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.556+0000 7ff47f7fe700 1 -- 172.21.15.121:0/3739072389 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff48408c4f0 con 0x7ff488105a50 2024-04-05T21:35:48.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.556+0000 7ff48e403700 1 --2- 172.21.15.121:0/3739072389 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff47406cce0 0x7ff47406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:48.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.556+0000 7ff48e403700 1 --2- 172.21.15.121:0/3739072389 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff47406cce0 0x7ff47406f1a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff478006010 tx=0x7ff47801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:48.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.561+0000 7ff47f7fe700 1 -- 172.21.15.121:0/3739072389 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff4840565e0 con 0x7ff488105a50 2024-04-05T21:35:48.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.772+0000 7ff490667700 1 -- 172.21.15.121:0/3739072389 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff48819bce0 con 0x7ff47406cce0 2024-04-05T21:35:48.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.774+0000 7ff47f7fe700 1 -- 172.21.15.121:0/3739072389 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff48819bce0 con 0x7ff47406cce0 2024-04-05T21:35:48.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.776+0000 7ff490667700 1 -- 172.21.15.121:0/3739072389 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff47406cce0 msgr2=0x7ff47406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:48.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.776+0000 7ff490667700 1 --2- 172.21.15.121:0/3739072389 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff47406cce0 0x7ff47406f1a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff478006010 tx=0x7ff47801a040 comp rx=0 tx=0).stop 2024-04-05T21:35:48.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.776+0000 7ff490667700 1 -- 172.21.15.121:0/3739072389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff488105a50 msgr2=0x7ff488196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:48.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.776+0000 7ff490667700 1 --2- 172.21.15.121:0/3739072389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff488105a50 0x7ff488196ba0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff48400d8d0 tx=0x7ff48400dc90 comp rx=0 tx=0).stop 2024-04-05T21:35:48.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.776+0000 7ff490667700 1 -- 172.21.15.121:0/3739072389 shutdown_connections 2024-04-05T21:35:48.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.776+0000 7ff490667700 1 --2- 172.21.15.121:0/3739072389 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff47406cce0 0x7ff47406f1a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:48.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.776+0000 7ff490667700 1 --2- 172.21.15.121:0/3739072389 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4880ffa50 0x7ff488196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:48.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.776+0000 7ff490667700 1 --2- 172.21.15.121:0/3739072389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff488105a50 0x7ff488196ba0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:48.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.776+0000 7ff490667700 1 -- 172.21.15.121:0/3739072389 >> 172.21.15.121:0/3739072389 conn(0x7ff4880fb540 msgr2=0x7ff4881038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:48.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.776+0000 7ff490667700 1 -- 172.21.15.121:0/3739072389 shutdown_connections 2024-04-05T21:35:48.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.776+0000 7ff490667700 1 -- 172.21.15.121:0/3739072389 wait complete. 2024-04-05T21:35:48.787 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:35:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.879+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/827555502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf340ffa40 msgr2=0x7fdf340ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.879+0000 7fdf3a1e1700 1 --2- 172.21.15.121:0/827555502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf340ffa40 0x7fdf340ffec0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009910 tx=0x7fdf28009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.879+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/827555502 shutdown_connections 2024-04-05T21:35:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.879+0000 7fdf3a1e1700 1 --2- 172.21.15.121:0/827555502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf340ffa40 0x7fdf340ffec0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.879+0000 7fdf3a1e1700 1 --2- 172.21.15.121:0/827555502 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf34105a40 0x7fdf34105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.879+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/827555502 >> 172.21.15.121:0/827555502 conn(0x7fdf340fb570 msgr2=0x7fdf340fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.879+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/827555502 shutdown_connections 2024-04-05T21:35:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.879+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/827555502 wait complete. 2024-04-05T21:35:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.880+0000 7fdf3a1e1700 1 Processor -- start 2024-04-05T21:35:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.880+0000 7fdf3a1e1700 1 -- start start 2024-04-05T21:35:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.880+0000 7fdf3a1e1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf340ffa40 0x7fdf340fbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.880+0000 7fdf3a1e1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf34105a40 0x7fdf340fc480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.880+0000 7fdf3a1e1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf340fc9c0 con 0x7fdf340ffa40 2024-04-05T21:35:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.880+0000 7fdf3a1e1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf340fcb00 con 0x7fdf34105a40 2024-04-05T21:35:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.880+0000 7fdf337fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf340ffa40 0x7fdf340fbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.880+0000 7fdf337fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf340ffa40 0x7fdf340fbf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51516/0 (socket says 172.21.15.121:51516) 2024-04-05T21:35:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.880+0000 7fdf337fe700 1 -- 172.21.15.121:0/454776529 learned_addr learned my addr 172.21.15.121:0/454776529 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.880+0000 7fdf32ffd700 1 --2- 172.21.15.121:0/454776529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf34105a40 0x7fdf340fc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.881+0000 7fdf337fe700 1 -- 172.21.15.121:0/454776529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf34105a40 msgr2=0x7fdf340fc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.881+0000 7fdf337fe700 1 --2- 172.21.15.121:0/454776529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf34105a40 0x7fdf340fc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.881+0000 7fdf337fe700 1 -- 172.21.15.121:0/454776529 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf280095f0 con 0x7fdf340ffa40 2024-04-05T21:35:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.881+0000 7fdf32ffd700 1 --2- 172.21.15.121:0/454776529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf34105a40 0x7fdf340fc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:35:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.881+0000 7fdf337fe700 1 --2- 172.21.15.121:0/454776529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf340ffa40 0x7fdf340fbf40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdf2400d8d0 tx=0x7fdf2400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.881+0000 7fdf30ff9700 1 -- 172.21.15.121:0/454776529 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf24009940 con 0x7fdf340ffa40 2024-04-05T21:35:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.881+0000 7fdf30ff9700 1 -- 172.21.15.121:0/454776529 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf24010460 con 0x7fdf340ffa40 2024-04-05T21:35:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.881+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/454776529 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf341a27e0 con 0x7fdf340ffa40 2024-04-05T21:35:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.881+0000 7fdf30ff9700 1 -- 172.21.15.121:0/454776529 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf2400f5f0 con 0x7fdf340ffa40 2024-04-05T21:35:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.881+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/454776529 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf341a2bf0 con 0x7fdf340ffa40 2024-04-05T21:35:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.882+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/454776529 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf34108fe0 con 0x7fdf340ffa40 2024-04-05T21:35:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.882+0000 7fdf30ff9700 1 -- 172.21.15.121:0/454776529 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdf240105d0 con 0x7fdf340ffa40 2024-04-05T21:35:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.883+0000 7fdf30ff9700 1 --2- 172.21.15.121:0/454776529 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdf1c06cce0 0x7fdf1c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.883+0000 7fdf30ff9700 1 -- 172.21.15.121:0/454776529 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdf2408c5e0 con 0x7fdf340ffa40 2024-04-05T21:35:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.883+0000 7fdf32ffd700 1 --2- 172.21.15.121:0/454776529 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdf1c06cce0 0x7fdf1c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.883+0000 7fdf32ffd700 1 --2- 172.21.15.121:0/454776529 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdf1c06cce0 0x7fdf1c06f1a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdf28005b40 tx=0x7fdf2801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:48.887+0000 7fdf30ff9700 1 -- 172.21.15.121:0/454776529 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdf24056650 con 0x7fdf340ffa40 2024-04-05T21:35:49.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.097+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/454776529 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf3402cdd0 con 0x7fdf1c06cce0 2024-04-05T21:35:49.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.098+0000 7fdf30ff9700 1 -- 172.21.15.121:0/454776529 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdf3402cdd0 con 0x7fdf1c06cce0 2024-04-05T21:35:49.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.100+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/454776529 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdf1c06cce0 msgr2=0x7fdf1c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:49.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.100+0000 7fdf3a1e1700 1 --2- 172.21.15.121:0/454776529 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdf1c06cce0 0x7fdf1c06f1a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdf28005b40 tx=0x7fdf2801a040 comp rx=0 tx=0).stop 2024-04-05T21:35:49.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.100+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/454776529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf340ffa40 msgr2=0x7fdf340fbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:49.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.100+0000 7fdf3a1e1700 1 --2- 172.21.15.121:0/454776529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf340ffa40 0x7fdf340fbf40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdf2400d8d0 tx=0x7fdf2400dc90 comp rx=0 tx=0).stop 2024-04-05T21:35:49.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.100+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/454776529 shutdown_connections 2024-04-05T21:35:49.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.100+0000 7fdf3a1e1700 1 --2- 172.21.15.121:0/454776529 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdf1c06cce0 0x7fdf1c06f1a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.100+0000 7fdf3a1e1700 1 --2- 172.21.15.121:0/454776529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf340ffa40 0x7fdf340fbf40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.100+0000 7fdf3a1e1700 1 --2- 172.21.15.121:0/454776529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf34105a40 0x7fdf340fc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.100+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/454776529 >> 172.21.15.121:0/454776529 conn(0x7fdf340fb570 msgr2=0x7fdf34068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:49.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.100+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/454776529 shutdown_connections 2024-04-05T21:35:49.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.100+0000 7fdf3a1e1700 1 -- 172.21.15.121:0/454776529 wait complete. 2024-04-05T21:35:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.203+0000 7fd296f4a700 1 -- 172.21.15.121:0/1638184261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2900ffbd0 msgr2=0x7fd290100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.203+0000 7fd296f4a700 1 --2- 172.21.15.121:0/1638184261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2900ffbd0 0x7fd290100050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd27c009910 tx=0x7fd27c009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.203+0000 7fd296f4a700 1 -- 172.21.15.121:0/1638184261 shutdown_connections 2024-04-05T21:35:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.203+0000 7fd296f4a700 1 --2- 172.21.15.121:0/1638184261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2900ffbd0 0x7fd290100050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.203+0000 7fd296f4a700 1 --2- 172.21.15.121:0/1638184261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd290105a40 0x7fd290105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.203+0000 7fd296f4a700 1 -- 172.21.15.121:0/1638184261 >> 172.21.15.121:0/1638184261 conn(0x7fd2900755a0 msgr2=0x7fd2900759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.203+0000 7fd296f4a700 1 -- 172.21.15.121:0/1638184261 shutdown_connections 2024-04-05T21:35:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.203+0000 7fd296f4a700 1 -- 172.21.15.121:0/1638184261 wait complete. 2024-04-05T21:35:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd296f4a700 1 Processor -- start 2024-04-05T21:35:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd296f4a700 1 -- start start 2024-04-05T21:35:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd296f4a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2900ffbd0 0x7fd29019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd296f4a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd290105a40 0x7fd29019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd296f4a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29019f1e0 con 0x7fd2900ffbd0 2024-04-05T21:35:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd296f4a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd290198690 con 0x7fd290105a40 2024-04-05T21:35:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd294ce6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2900ffbd0 0x7fd29019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd294ce6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2900ffbd0 0x7fd29019e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51532/0 (socket says 172.21.15.121:51532) 2024-04-05T21:35:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd294ce6700 1 -- 172.21.15.121:0/2273111769 learned_addr learned my addr 172.21.15.121:0/2273111769 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd28ffff700 1 --2- 172.21.15.121:0/2273111769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd290105a40 0x7fd29019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd294ce6700 1 -- 172.21.15.121:0/2273111769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd290105a40 msgr2=0x7fd29019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd294ce6700 1 --2- 172.21.15.121:0/2273111769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd290105a40 0x7fd29019eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd294ce6700 1 -- 172.21.15.121:0/2273111769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27c0095f0 con 0x7fd2900ffbd0 2024-04-05T21:35:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.204+0000 7fd294ce6700 1 --2- 172.21.15.121:0/2273111769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2900ffbd0 0x7fd29019e610 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd28400d8d0 tx=0x7fd28400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.205+0000 7fd28dffb700 1 -- 172.21.15.121:0/2273111769 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd284009940 con 0x7fd2900ffbd0 2024-04-05T21:35:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.205+0000 7fd28dffb700 1 -- 172.21.15.121:0/2273111769 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd284010460 con 0x7fd2900ffbd0 2024-04-05T21:35:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.205+0000 7fd296f4a700 1 -- 172.21.15.121:0/2273111769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd290198970 con 0x7fd2900ffbd0 2024-04-05T21:35:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.205+0000 7fd28dffb700 1 -- 172.21.15.121:0/2273111769 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd28400f5f0 con 0x7fd2900ffbd0 2024-04-05T21:35:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.205+0000 7fd296f4a700 1 -- 172.21.15.121:0/2273111769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd290198ec0 con 0x7fd2900ffbd0 2024-04-05T21:35:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.206+0000 7fd296f4a700 1 -- 172.21.15.121:0/2273111769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29004ea90 con 0x7fd2900ffbd0 2024-04-05T21:35:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.206+0000 7fd28dffb700 1 -- 172.21.15.121:0/2273111769 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd2840105d0 con 0x7fd2900ffbd0 2024-04-05T21:35:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.206+0000 7fd28dffb700 1 --2- 172.21.15.121:0/2273111769 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd28006cce0 0x7fd28006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.207+0000 7fd28dffb700 1 -- 172.21.15.121:0/2273111769 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd28408c5e0 con 0x7fd2900ffbd0 2024-04-05T21:35:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.207+0000 7fd28ffff700 1 --2- 172.21.15.121:0/2273111769 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd28006cce0 0x7fd28006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.207+0000 7fd28ffff700 1 --2- 172.21.15.121:0/2273111769 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd28006cce0 0x7fd28006f1a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00b5c0 tx=0x7fd27c005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.211+0000 7fd28dffb700 1 -- 172.21.15.121:0/2273111769 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd28405a0e0 con 0x7fd2900ffbd0 2024-04-05T21:35:49.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.424+0000 7fd296f4a700 1 -- 172.21.15.121:0/2273111769 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd290199340 con 0x7fd28006cce0 2024-04-05T21:35:49.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.434+0000 7fd28dffb700 1 -- 172.21.15.121:0/2273111769 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fd290199340 con 0x7fd28006cce0 2024-04-05T21:35:49.435 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:35:49.435 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (6m) 37s ago 18m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:35:49.435 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (18m) 37s ago 18m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:35:49.436 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (17m) 19s ago 17m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:35:49.436 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (4m) 37s ago 18m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:35:49.436 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (4m) 19s ago 17m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:35:49.436 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (6m) 37s ago 17m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:35:49.436 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (63s) 37s ago 12m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:35:49.436 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (43s) 37s ago 11m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:35:49.437 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (23s) 19s ago 11m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:35:49.437 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (11m) 19s ago 11m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:35:49.437 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (8m) 37s ago 19m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:35:49.437 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (7m) 19s ago 17m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:35:49.437 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (5m) 37s ago 19m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:35:49.437 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (5m) 19s ago 16m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:35:49.437 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (7m) 37s ago 18m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:35:49.438 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (7m) 19s ago 17m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:35:49.438 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (4m) 37s ago 16m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:35:49.438 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (3m) 37s ago 15m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:35:49.438 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (3m) 37s ago 15m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:35:49.438 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (2m) 19s ago 14m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:35:49.438 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (116s) 19s ago 14m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:35:49.438 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (82s) 19s ago 13m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:35:49.439 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (6m) 37s ago 17m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.436+0000 7fd296f4a700 1 -- 172.21.15.121:0/2273111769 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd28006cce0 msgr2=0x7fd28006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.436+0000 7fd296f4a700 1 --2- 172.21.15.121:0/2273111769 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd28006cce0 0x7fd28006f1a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00b5c0 tx=0x7fd27c005fd0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.436+0000 7fd296f4a700 1 -- 172.21.15.121:0/2273111769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2900ffbd0 msgr2=0x7fd29019e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.436+0000 7fd296f4a700 1 --2- 172.21.15.121:0/2273111769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2900ffbd0 0x7fd29019e610 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd28400d8d0 tx=0x7fd28400dc90 comp rx=0 tx=0).stop 2024-04-05T21:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.436+0000 7fd296f4a700 1 -- 172.21.15.121:0/2273111769 shutdown_connections 2024-04-05T21:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.436+0000 7fd296f4a700 1 --2- 172.21.15.121:0/2273111769 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd28006cce0 0x7fd28006f1a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.436+0000 7fd296f4a700 1 --2- 172.21.15.121:0/2273111769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2900ffbd0 0x7fd29019e610 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.436+0000 7fd296f4a700 1 --2- 172.21.15.121:0/2273111769 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd290105a40 0x7fd29019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.436+0000 7fd296f4a700 1 -- 172.21.15.121:0/2273111769 >> 172.21.15.121:0/2273111769 conn(0x7fd2900755a0 msgr2=0x7fd2900fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.436+0000 7fd296f4a700 1 -- 172.21.15.121:0/2273111769 shutdown_connections 2024-04-05T21:35:49.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.436+0000 7fd296f4a700 1 -- 172.21.15.121:0/2273111769 wait complete. 2024-04-05T21:35:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.539+0000 7f499f66e700 1 -- 172.21.15.121:0/2081868905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4998105a40 msgr2=0x7f4998105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.539+0000 7f499f66e700 1 --2- 172.21.15.121:0/2081868905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4998105a40 0x7f4998105e20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4988009910 tx=0x7f4988009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.539+0000 7f499f66e700 1 -- 172.21.15.121:0/2081868905 shutdown_connections 2024-04-05T21:35:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.539+0000 7f499f66e700 1 --2- 172.21.15.121:0/2081868905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49980ffbd0 0x7f4998100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.539+0000 7f499f66e700 1 --2- 172.21.15.121:0/2081868905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4998105a40 0x7f4998105e20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.539+0000 7f499f66e700 1 -- 172.21.15.121:0/2081868905 >> 172.21.15.121:0/2081868905 conn(0x7f49980755a0 msgr2=0x7f49980759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.539+0000 7f499f66e700 1 -- 172.21.15.121:0/2081868905 shutdown_connections 2024-04-05T21:35:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.539+0000 7f499f66e700 1 -- 172.21.15.121:0/2081868905 wait complete. 2024-04-05T21:35:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.540+0000 7f499f66e700 1 Processor -- start 2024-04-05T21:35:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.540+0000 7f499f66e700 1 -- start start 2024-04-05T21:35:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.540+0000 7f499f66e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49980ffbd0 0x7f4998196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.540+0000 7f499f66e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4998105a40 0x7f4998196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.540+0000 7f499f66e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4998197260 con 0x7f4998105a40 2024-04-05T21:35:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.540+0000 7f499f66e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499819aff0 con 0x7f49980ffbd0 2024-04-05T21:35:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.540+0000 7f499cc09700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4998105a40 0x7f4998196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.540+0000 7f499cc09700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4998105a40 0x7f4998196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51550/0 (socket says 172.21.15.121:51550) 2024-04-05T21:35:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.540+0000 7f499cc09700 1 -- 172.21.15.121:0/2521774597 learned_addr learned my addr 172.21.15.121:0/2521774597 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.540+0000 7f499d40a700 1 --2- 172.21.15.121:0/2521774597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49980ffbd0 0x7f4998196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.541+0000 7f499cc09700 1 -- 172.21.15.121:0/2521774597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49980ffbd0 msgr2=0x7f4998196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.541+0000 7f499cc09700 1 --2- 172.21.15.121:0/2521774597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49980ffbd0 0x7f4998196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.541+0000 7f499cc09700 1 -- 172.21.15.121:0/2521774597 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49880095f0 con 0x7f4998105a40 2024-04-05T21:35:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.541+0000 7f499cc09700 1 --2- 172.21.15.121:0/2521774597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4998105a40 0x7f4998196b80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f499400d8d0 tx=0x7f499400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.541+0000 7f498e7fc700 1 -- 172.21.15.121:0/2521774597 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49940098a0 con 0x7f4998105a40 2024-04-05T21:35:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.541+0000 7f499f66e700 1 -- 172.21.15.121:0/2521774597 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499819b2d0 con 0x7f4998105a40 2024-04-05T21:35:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.541+0000 7f498e7fc700 1 -- 172.21.15.121:0/2521774597 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4994010460 con 0x7f4998105a40 2024-04-05T21:35:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.541+0000 7f498e7fc700 1 -- 172.21.15.121:0/2521774597 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4994009ea0 con 0x7f4998105a40 2024-04-05T21:35:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.542+0000 7f499f66e700 1 -- 172.21.15.121:0/2521774597 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499819b820 con 0x7f4998105a40 2024-04-05T21:35:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.543+0000 7f499f66e700 1 -- 172.21.15.121:0/2521774597 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4998109050 con 0x7f4998105a40 2024-04-05T21:35:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.543+0000 7f498e7fc700 1 -- 172.21.15.121:0/2521774597 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f49940105d0 con 0x7f4998105a40 2024-04-05T21:35:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.543+0000 7f498e7fc700 1 --2- 172.21.15.121:0/2521774597 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f498406cce0 0x7f498406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.543+0000 7f498e7fc700 1 -- 172.21.15.121:0/2521774597 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f499408c480 con 0x7f4998105a40 2024-04-05T21:35:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.543+0000 7f499d40a700 1 --2- 172.21.15.121:0/2521774597 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f498406cce0 0x7f498406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.544+0000 7f499d40a700 1 --2- 172.21.15.121:0/2521774597 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f498406cce0 0x7f498406f1a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f498800b5c0 tx=0x7f498801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:49.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.548+0000 7f498e7fc700 1 -- 172.21.15.121:0/2521774597 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f49940564f0 con 0x7f4998105a40 2024-04-05T21:35:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:49 smithi121 ceph-mon[145859]: pgmap v287: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:49.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:49 smithi188 ceph-mon[107793]: pgmap v287: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:49.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.821+0000 7f499f66e700 1 -- 172.21.15.121:0/2521774597 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f499804ea90 con 0x7f4998105a40 2024-04-05T21:35:49.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.821+0000 7f498e7fc700 1 -- 172.21.15.121:0/2521774597 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f4994010880 con 0x7f4998105a40 2024-04-05T21:35:49.822 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:35:49.822 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:35:49.823 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:35:49.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:35:49.823 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:35:49.823 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:35:49.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:35:49.823 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:35:49.823 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:35:49.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:35:49.823 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:35:49.823 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:35:49.824 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:35:49.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:35:49.824 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:35:49.824 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:35:49.824 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:35:49.824 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:35:49.824 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:35:49.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.824+0000 7f499f66e700 1 -- 172.21.15.121:0/2521774597 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f498406cce0 msgr2=0x7f498406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:49.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.824+0000 7f499f66e700 1 --2- 172.21.15.121:0/2521774597 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f498406cce0 0x7f498406f1a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f498800b5c0 tx=0x7f498801a040 comp rx=0 tx=0).stop 2024-04-05T21:35:49.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.824+0000 7f499f66e700 1 -- 172.21.15.121:0/2521774597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4998105a40 msgr2=0x7f4998196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:49.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.824+0000 7f499f66e700 1 --2- 172.21.15.121:0/2521774597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4998105a40 0x7f4998196b80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f499400d8d0 tx=0x7f499400dbe0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.824+0000 7f499f66e700 1 -- 172.21.15.121:0/2521774597 shutdown_connections 2024-04-05T21:35:49.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.824+0000 7f499f66e700 1 --2- 172.21.15.121:0/2521774597 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f498406cce0 0x7f498406f1a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.824+0000 7f499f66e700 1 --2- 172.21.15.121:0/2521774597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f49980ffbd0 0x7f4998196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.824+0000 7f499f66e700 1 --2- 172.21.15.121:0/2521774597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4998105a40 0x7f4998196b80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.824+0000 7f499f66e700 1 -- 172.21.15.121:0/2521774597 >> 172.21.15.121:0/2521774597 conn(0x7f49980755a0 msgr2=0x7f4998068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:49.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.824+0000 7f499f66e700 1 -- 172.21.15.121:0/2521774597 shutdown_connections 2024-04-05T21:35:49.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.824+0000 7f499f66e700 1 -- 172.21.15.121:0/2521774597 wait complete. 2024-04-05T21:35:49.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.926+0000 7f6533238700 1 -- 172.21.15.121:0/1509537036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f652c105a40 msgr2=0x7f652c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:49.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.926+0000 7f6533238700 1 --2- 172.21.15.121:0/1509537036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f652c105a40 0x7f652c105e20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6520009910 tx=0x7f6520009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:49.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.926+0000 7f6533238700 1 -- 172.21.15.121:0/1509537036 shutdown_connections 2024-04-05T21:35:49.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.926+0000 7f6533238700 1 --2- 172.21.15.121:0/1509537036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652c0ffa40 0x7f652c0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.926+0000 7f6533238700 1 --2- 172.21.15.121:0/1509537036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f652c105a40 0x7f652c105e20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.926+0000 7f6533238700 1 -- 172.21.15.121:0/1509537036 >> 172.21.15.121:0/1509537036 conn(0x7f652c0fb570 msgr2=0x7f652c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.926+0000 7f6533238700 1 -- 172.21.15.121:0/1509537036 shutdown_connections 2024-04-05T21:35:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.926+0000 7f6533238700 1 -- 172.21.15.121:0/1509537036 wait complete. 2024-04-05T21:35:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.926+0000 7f6533238700 1 Processor -- start 2024-04-05T21:35:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.926+0000 7f6533238700 1 -- start start 2024-04-05T21:35:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6533238700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f652c0ffa40 0x7f652c0fbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6533238700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652c105a40 0x7f652c0fc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6533238700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652c0fca40 con 0x7f652c0ffa40 2024-04-05T21:35:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6533238700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652c0fcbb0 con 0x7f652c105a40 2024-04-05T21:35:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6530fd4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f652c0ffa40 0x7f652c0fbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:49.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6530fd4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f652c0ffa40 0x7f652c0fbf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51560/0 (socket says 172.21.15.121:51560) 2024-04-05T21:35:49.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6530fd4700 1 -- 172.21.15.121:0/1924279038 learned_addr learned my addr 172.21.15.121:0/1924279038 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:49.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f652bfff700 1 --2- 172.21.15.121:0/1924279038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652c105a40 0x7f652c0fc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:49.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6530fd4700 1 -- 172.21.15.121:0/1924279038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652c105a40 msgr2=0x7f652c0fc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:49.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6530fd4700 1 --2- 172.21.15.121:0/1924279038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652c105a40 0x7f652c0fc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:49.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6530fd4700 1 -- 172.21.15.121:0/1924279038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65200095f0 con 0x7f652c0ffa40 2024-04-05T21:35:49.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f652bfff700 1 --2- 172.21.15.121:0/1924279038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652c105a40 0x7f652c0fc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:35:49.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6530fd4700 1 --2- 172.21.15.121:0/1924279038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f652c0ffa40 0x7f652c0fbf30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6520000c00 tx=0x7f652000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:49.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6529ffb700 1 -- 172.21.15.121:0/1924279038 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f652001d070 con 0x7f652c0ffa40 2024-04-05T21:35:49.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6529ffb700 1 -- 172.21.15.121:0/1924279038 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6520005dc0 con 0x7f652c0ffa40 2024-04-05T21:35:49.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6533238700 1 -- 172.21.15.121:0/1924279038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f652c1a2860 con 0x7f652c0ffa40 2024-04-05T21:35:49.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.927+0000 7f6529ffb700 1 -- 172.21.15.121:0/1924279038 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65200039b0 con 0x7f652c0ffa40 2024-04-05T21:35:49.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.928+0000 7f6533238700 1 -- 172.21.15.121:0/1924279038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f652c1a2bc0 con 0x7f652c0ffa40 2024-04-05T21:35:49.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.928+0000 7f6533238700 1 -- 172.21.15.121:0/1924279038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f652c109050 con 0x7f652c0ffa40 2024-04-05T21:35:49.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.929+0000 7f6529ffb700 1 -- 172.21.15.121:0/1924279038 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f65200048e0 con 0x7f652c0ffa40 2024-04-05T21:35:49.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.929+0000 7f6529ffb700 1 --2- 172.21.15.121:0/1924279038 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f651c06cce0 0x7f651c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:49.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.929+0000 7f6529ffb700 1 -- 172.21.15.121:0/1924279038 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f652008dd20 con 0x7f652c0ffa40 2024-04-05T21:35:49.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.929+0000 7f652bfff700 1 --2- 172.21.15.121:0/1924279038 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f651c06cce0 0x7f651c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:49.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.930+0000 7f652bfff700 1 --2- 172.21.15.121:0/1924279038 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f651c06cce0 0x7f651c06f1a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f652c0fd720 tx=0x7f6518009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:49.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:49.934+0000 7f6529ffb700 1 -- 172.21.15.121:0/1924279038 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6520057cf0 con 0x7f652c0ffa40 2024-04-05T21:35:50.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.170+0000 7f6533238700 1 -- 172.21.15.121:0/1924279038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f652c108a30 con 0x7f652c0ffa40 2024-04-05T21:35:50.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.171+0000 7f6529ffb700 1 -- 172.21.15.121:0/1924279038 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f6520004be0 con 0x7f652c0ffa40 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:35:50.172 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:35:50.173 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:35:50.174 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:35:50.174 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:35:50.174 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:35:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:35:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:35:50.174 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:35:50.174 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:35:50.174 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:35:50.174 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:35:50.174 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:35:50.174 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:35:50.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.173+0000 7f6533238700 1 -- 172.21.15.121:0/1924279038 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f651c06cce0 msgr2=0x7f651c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:50.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.173+0000 7f6533238700 1 --2- 172.21.15.121:0/1924279038 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f651c06cce0 0x7f651c06f1a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f652c0fd720 tx=0x7f6518009260 comp rx=0 tx=0).stop 2024-04-05T21:35:50.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.173+0000 7f6533238700 1 -- 172.21.15.121:0/1924279038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f652c0ffa40 msgr2=0x7f652c0fbf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:50.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.173+0000 7f6533238700 1 --2- 172.21.15.121:0/1924279038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f652c0ffa40 0x7f652c0fbf30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6520000c00 tx=0x7f652000be90 comp rx=0 tx=0).stop 2024-04-05T21:35:50.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.173+0000 7f6533238700 1 -- 172.21.15.121:0/1924279038 shutdown_connections 2024-04-05T21:35:50.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.173+0000 7f6533238700 1 --2- 172.21.15.121:0/1924279038 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f651c06cce0 0x7f651c06f1a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.173+0000 7f6533238700 1 --2- 172.21.15.121:0/1924279038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f652c0ffa40 0x7f652c0fbf30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.173+0000 7f6533238700 1 --2- 172.21.15.121:0/1924279038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f652c105a40 0x7f652c0fc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.173+0000 7f6533238700 1 -- 172.21.15.121:0/1924279038 >> 172.21.15.121:0/1924279038 conn(0x7f652c0fb570 msgr2=0x7f652c068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:50.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.173+0000 7f6533238700 1 -- 172.21.15.121:0/1924279038 shutdown_connections 2024-04-05T21:35:50.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.173+0000 7f6533238700 1 -- 172.21.15.121:0/1924279038 wait complete. 2024-04-05T21:35:50.176 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:35:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.277+0000 7f0604660700 1 -- 172.21.15.121:0/142078262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05fc101750 msgr2=0x7f05fc1057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.277+0000 7f0604660700 1 --2- 172.21.15.121:0/142078262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05fc101750 0x7f05fc1057a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009910 tx=0x7f05f8009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.277+0000 7f0604660700 1 -- 172.21.15.121:0/142078262 shutdown_connections 2024-04-05T21:35:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.277+0000 7f0604660700 1 --2- 172.21.15.121:0/142078262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05fc101750 0x7f05fc1057a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.277+0000 7f0604660700 1 --2- 172.21.15.121:0/142078262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05fc100da0 0x7f05fc101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.277+0000 7f0604660700 1 -- 172.21.15.121:0/142078262 >> 172.21.15.121:0/142078262 conn(0x7f05fc0fc630 msgr2=0x7f05fc0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.277+0000 7f0604660700 1 -- 172.21.15.121:0/142078262 shutdown_connections 2024-04-05T21:35:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.277+0000 7f0604660700 1 -- 172.21.15.121:0/142078262 wait complete. 2024-04-05T21:35:50.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f0604660700 1 Processor -- start 2024-04-05T21:35:50.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f0604660700 1 -- start start 2024-04-05T21:35:50.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f0604660700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05fc100da0 0x7f05fc196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:50.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f0604660700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05fc101750 0x7f05fc196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f0604660700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc197270 con 0x7f05fc100da0 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f0604660700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc19b000 con 0x7f05fc101750 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f06023fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05fc100da0 0x7f05fc196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f06023fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05fc100da0 0x7f05fc196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51572/0 (socket says 172.21.15.121:51572) 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f06023fc700 1 -- 172.21.15.121:0/2304224472 learned_addr learned my addr 172.21.15.121:0/2304224472 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f0601bfb700 1 --2- 172.21.15.121:0/2304224472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05fc101750 0x7f05fc196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f06023fc700 1 -- 172.21.15.121:0/2304224472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05fc101750 msgr2=0x7f05fc196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f06023fc700 1 --2- 172.21.15.121:0/2304224472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05fc101750 0x7f05fc196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f06023fc700 1 -- 172.21.15.121:0/2304224472 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f80095f0 con 0x7f05fc100da0 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.278+0000 7f06023fc700 1 --2- 172.21.15.121:0/2304224472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05fc100da0 0x7f05fc196650 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f05ec00d8d0 tx=0x7f05ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.279+0000 7f05f37fe700 1 -- 172.21.15.121:0/2304224472 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05ec009940 con 0x7f05fc100da0 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.279+0000 7f0604660700 1 -- 172.21.15.121:0/2304224472 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc19b2e0 con 0x7f05fc100da0 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.279+0000 7f05f37fe700 1 -- 172.21.15.121:0/2304224472 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05ec010460 con 0x7f05fc100da0 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.279+0000 7f05f37fe700 1 -- 172.21.15.121:0/2304224472 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05ec00f5f0 con 0x7f05fc100da0 2024-04-05T21:35:50.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.279+0000 7f0604660700 1 -- 172.21.15.121:0/2304224472 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc19b830 con 0x7f05fc100da0 2024-04-05T21:35:50.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.280+0000 7f0604660700 1 -- 172.21.15.121:0/2304224472 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05fc1091a0 con 0x7f05fc100da0 2024-04-05T21:35:50.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.280+0000 7f05f37fe700 1 -- 172.21.15.121:0/2304224472 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f05ec0105d0 con 0x7f05fc100da0 2024-04-05T21:35:50.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.280+0000 7f05f37fe700 1 --2- 172.21.15.121:0/2304224472 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f05e806cce0 0x7f05e806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:50.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.281+0000 7f05f37fe700 1 -- 172.21.15.121:0/2304224472 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f05ec08c5e0 con 0x7f05fc100da0 2024-04-05T21:35:50.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.281+0000 7f0601bfb700 1 --2- 172.21.15.121:0/2304224472 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f05e806cce0 0x7f05e806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.281+0000 7f0601bfb700 1 --2- 172.21.15.121:0/2304224472 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f05e806cce0 0x7f05e806f1a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f05f800b5c0 tx=0x7f05f8005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.285+0000 7f05f37fe700 1 -- 172.21.15.121:0/2304224472 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f05ec056650 con 0x7f05fc100da0 2024-04-05T21:35:50.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.496+0000 7f0604660700 1 -- 172.21.15.121:0/2304224472 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f05fc19bb10 con 0x7f05e806cce0 2024-04-05T21:35:50.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.497+0000 7f05f37fe700 1 -- 172.21.15.121:0/2304224472 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f05fc19bb10 con 0x7f05e806cce0 2024-04-05T21:35:50.498 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:35:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:35:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:35:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:35:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:35:50.498 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:35:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:35:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:35:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:35:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:35:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:35:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:35:50.499 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:35:50.499 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:35:50.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.499+0000 7f0604660700 1 -- 172.21.15.121:0/2304224472 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f05e806cce0 msgr2=0x7f05e806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:50.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.499+0000 7f0604660700 1 --2- 172.21.15.121:0/2304224472 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f05e806cce0 0x7f05e806f1a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f05f800b5c0 tx=0x7f05f8005fd0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.499+0000 7f0604660700 1 -- 172.21.15.121:0/2304224472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05fc100da0 msgr2=0x7f05fc196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:50.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.499+0000 7f0604660700 1 --2- 172.21.15.121:0/2304224472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05fc100da0 0x7f05fc196650 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f05ec00d8d0 tx=0x7f05ec00dc90 comp rx=0 tx=0).stop 2024-04-05T21:35:50.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.499+0000 7f0604660700 1 -- 172.21.15.121:0/2304224472 shutdown_connections 2024-04-05T21:35:50.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.499+0000 7f0604660700 1 --2- 172.21.15.121:0/2304224472 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f05e806cce0 0x7f05e806f1a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.499+0000 7f0604660700 1 --2- 172.21.15.121:0/2304224472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05fc100da0 0x7f05fc196650 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.499+0000 7f0604660700 1 --2- 172.21.15.121:0/2304224472 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f05fc101750 0x7f05fc196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.499+0000 7f0604660700 1 -- 172.21.15.121:0/2304224472 >> 172.21.15.121:0/2304224472 conn(0x7f05fc0fc630 msgr2=0x7f05fc0fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:50.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.500+0000 7f0604660700 1 -- 172.21.15.121:0/2304224472 shutdown_connections 2024-04-05T21:35:50.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.500+0000 7f0604660700 1 -- 172.21.15.121:0/2304224472 wait complete. 2024-04-05T21:35:50.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:50 smithi188 ceph-mon[107793]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:50.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:50 smithi188 ceph-mon[107793]: from='client.34402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:50.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:50 smithi188 ceph-mon[107793]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:50.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:50 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2521774597' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:35:50.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:50 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1924279038' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:35:50.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.603+0000 7fdb354ed700 1 -- 172.21.15.121:0/1905729181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb300fecf0 msgr2=0x7fdb300ff150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:50.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.603+0000 7fdb354ed700 1 --2- 172.21.15.121:0/1905729181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb300fecf0 0x7fdb300ff150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdb20009910 tx=0x7fdb20009c20 comp rx=0 tx=0).stop 2024-04-05T21:35:50.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.603+0000 7fdb354ed700 1 -- 172.21.15.121:0/1905729181 shutdown_connections 2024-04-05T21:35:50.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.603+0000 7fdb354ed700 1 --2- 172.21.15.121:0/1905729181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb300fecf0 0x7fdb300ff150 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.603+0000 7fdb354ed700 1 --2- 172.21.15.121:0/1905729181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb30106da0 0x7fdb30107180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.603+0000 7fdb354ed700 1 -- 172.21.15.121:0/1905729181 >> 172.21.15.121:0/1905729181 conn(0x7fdb30075dd0 msgr2=0x7fdb300761e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:50.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.603+0000 7fdb354ed700 1 -- 172.21.15.121:0/1905729181 shutdown_connections 2024-04-05T21:35:50.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.603+0000 7fdb354ed700 1 -- 172.21.15.121:0/1905729181 wait complete. 2024-04-05T21:35:50.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.603+0000 7fdb354ed700 1 Processor -- start 2024-04-05T21:35:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.603+0000 7fdb354ed700 1 -- start start 2024-04-05T21:35:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb354ed700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb300fecf0 0x7fdb3019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb354ed700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb30106da0 0x7fdb3019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb354ed700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3019f130 con 0x7fdb30106da0 2024-04-05T21:35:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb354ed700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb301985e0 con 0x7fdb300fecf0 2024-04-05T21:35:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb2e7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb30106da0 0x7fdb3019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb2e7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb30106da0 0x7fdb3019eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51592/0 (socket says 172.21.15.121:51592) 2024-04-05T21:35:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb2e7fc700 1 -- 172.21.15.121:0/3203835068 learned_addr learned my addr 172.21.15.121:0/3203835068 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:35:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb2effd700 1 --2- 172.21.15.121:0/3203835068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb300fecf0 0x7fdb3019e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb2e7fc700 1 -- 172.21.15.121:0/3203835068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb300fecf0 msgr2=0x7fdb3019e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb2e7fc700 1 --2- 172.21.15.121:0/3203835068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb300fecf0 0x7fdb3019e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb2e7fc700 1 -- 172.21.15.121:0/3203835068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb200095f0 con 0x7fdb30106da0 2024-04-05T21:35:50.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb2e7fc700 1 --2- 172.21.15.121:0/3203835068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb30106da0 0x7fdb3019eaa0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdb20006010 tx=0x7fdb2000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:50.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb17fff700 1 -- 172.21.15.121:0/3203835068 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb2001d070 con 0x7fdb30106da0 2024-04-05T21:35:50.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb17fff700 1 -- 172.21.15.121:0/3203835068 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb20005710 con 0x7fdb30106da0 2024-04-05T21:35:50.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb17fff700 1 -- 172.21.15.121:0/3203835068 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb2000fe20 con 0x7fdb30106da0 2024-04-05T21:35:50.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb354ed700 1 -- 172.21.15.121:0/3203835068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb30198860 con 0x7fdb30106da0 2024-04-05T21:35:50.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.604+0000 7fdb354ed700 1 -- 172.21.15.121:0/3203835068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb30198d80 con 0x7fdb30106da0 2024-04-05T21:35:50.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.605+0000 7fdb354ed700 1 -- 172.21.15.121:0/3203835068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3004ea90 con 0x7fdb30106da0 2024-04-05T21:35:50.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.606+0000 7fdb17fff700 1 -- 172.21.15.121:0/3203835068 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdb20003a60 con 0x7fdb30106da0 2024-04-05T21:35:50.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.606+0000 7fdb17fff700 1 --2- 172.21.15.121:0/3203835068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb1806cce0 0x7fdb1806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:35:50.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.606+0000 7fdb17fff700 1 -- 172.21.15.121:0/3203835068 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdb2008d750 con 0x7fdb30106da0 2024-04-05T21:35:50.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.606+0000 7fdb2effd700 1 --2- 172.21.15.121:0/3203835068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb1806cce0 0x7fdb1806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:35:50.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.607+0000 7fdb2effd700 1 --2- 172.21.15.121:0/3203835068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb1806cce0 0x7fdb1806f1a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdb3006a2c0 tx=0x7fdb2400b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:35:50.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.611+0000 7fdb17fff700 1 -- 172.21.15.121:0/3203835068 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdb2005b250 con 0x7fdb30106da0 2024-04-05T21:35:50.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:50 smithi121 ceph-mon[145859]: from='client.34398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:50.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:50 smithi121 ceph-mon[145859]: from='client.34402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:50.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:50 smithi121 ceph-mon[145859]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:50.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:50 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2521774597' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:35:50.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:50 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1924279038' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:35:50.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.880+0000 7fdb354ed700 1 -- 172.21.15.121:0/3203835068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdb30199af0 con 0x7fdb30106da0 2024-04-05T21:35:50.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.881+0000 7fdb17fff700 1 -- 172.21.15.121:0/3203835068 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fdb2005ade0 con 0x7fdb30106da0 2024-04-05T21:35:50.882 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:35:50.882 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:35:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:35:50.882 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:35:50.882 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:35:50.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.883+0000 7fdb354ed700 1 -- 172.21.15.121:0/3203835068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb1806cce0 msgr2=0x7fdb1806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:50.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.883+0000 7fdb354ed700 1 --2- 172.21.15.121:0/3203835068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb1806cce0 0x7fdb1806f1a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdb3006a2c0 tx=0x7fdb2400b410 comp rx=0 tx=0).stop 2024-04-05T21:35:50.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.883+0000 7fdb354ed700 1 -- 172.21.15.121:0/3203835068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb30106da0 msgr2=0x7fdb3019eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:35:50.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.883+0000 7fdb354ed700 1 --2- 172.21.15.121:0/3203835068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb30106da0 0x7fdb3019eaa0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdb20006010 tx=0x7fdb2000bd70 comp rx=0 tx=0).stop 2024-04-05T21:35:50.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.883+0000 7fdb354ed700 1 -- 172.21.15.121:0/3203835068 shutdown_connections 2024-04-05T21:35:50.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.883+0000 7fdb354ed700 1 --2- 172.21.15.121:0/3203835068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb1806cce0 0x7fdb1806f1a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.883+0000 7fdb354ed700 1 --2- 172.21.15.121:0/3203835068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb300fecf0 0x7fdb3019e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.883+0000 7fdb354ed700 1 --2- 172.21.15.121:0/3203835068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb30106da0 0x7fdb3019eaa0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:35:50.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.883+0000 7fdb354ed700 1 -- 172.21.15.121:0/3203835068 >> 172.21.15.121:0/3203835068 conn(0x7fdb30075dd0 msgr2=0x7fdb300fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:35:50.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.883+0000 7fdb354ed700 1 -- 172.21.15.121:0/3203835068 shutdown_connections 2024-04-05T21:35:50.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:35:50.883+0000 7fdb354ed700 1 -- 172.21.15.121:0/3203835068 wait complete. 2024-04-05T21:35:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:51 smithi121 ceph-mon[145859]: pgmap v288: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:51 smithi121 ceph-mon[145859]: from='client.34418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:51 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3203835068' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:35:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:51 smithi188 ceph-mon[107793]: pgmap v288: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:51 smithi188 ceph-mon[107793]: from='client.34418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:35:51.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:51 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3203835068' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:35:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:53 smithi121 ceph-mon[145859]: pgmap v289: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:53 smithi188 ceph-mon[107793]: pgmap v289: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:55.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:55 smithi121 ceph-mon[145859]: pgmap v290: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:55.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:55 smithi188 ceph-mon[107793]: pgmap v290: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:57 smithi121 ceph-mon[145859]: pgmap v291: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:57 smithi188 ceph-mon[107793]: pgmap v291: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:59.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:35:59 smithi121 ceph-mon[145859]: pgmap v292: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:35:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:35:59 smithi188 ceph-mon[107793]: pgmap v292: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:36:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:36:01.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:01 smithi121 ceph-mon[145859]: pgmap v293: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:01 smithi188 ceph-mon[107793]: pgmap v293: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:03 smithi188 ceph-mon[107793]: pgmap v294: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:03.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:03 smithi121 ceph-mon[145859]: pgmap v294: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:05 smithi121 ceph-mon[145859]: pgmap v295: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:05.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:05 smithi188 ceph-mon[107793]: pgmap v295: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:07 smithi121 ceph-mon[145859]: pgmap v296: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:07 smithi188 ceph-mon[107793]: pgmap v296: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:09 smithi121 ceph-mon[145859]: pgmap v297: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:09 smithi188 ceph-mon[107793]: pgmap v297: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:11 smithi121 ceph-mon[145859]: pgmap v298: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:11.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:11 smithi188 ceph-mon[107793]: pgmap v298: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:13 smithi121 ceph-mon[145859]: pgmap v299: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:13.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:13 smithi188 ceph-mon[107793]: pgmap v299: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:15 smithi121 ceph-mon[145859]: pgmap v300: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:36:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:15 smithi188 ceph-mon[107793]: pgmap v300: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:36:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:17 smithi121 ceph-mon[145859]: pgmap v301: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:17.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:17 smithi188 ceph-mon[107793]: pgmap v301: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:19 smithi121 ceph-mon[145859]: pgmap v302: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:19 smithi188 ceph-mon[107793]: pgmap v302: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:21.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:20.999+0000 7f22f66b3700 1 -- 172.21.15.121:0/3399763135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22f0068670 msgr2=0x7f22f0068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:20.999+0000 7f22f66b3700 1 --2- 172.21.15.121:0/3399763135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22f0068670 0x7f22f0068a50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f22e4009910 tx=0x7f22e4009c20 comp rx=0 tx=0).stop 2024-04-05T21:36:21.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:20.999+0000 7f22f66b3700 1 -- 172.21.15.121:0/3399763135 shutdown_connections 2024-04-05T21:36:21.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:20.999+0000 7f22f66b3700 1 --2- 172.21.15.121:0/3399763135 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22f0069020 0x7f22f0105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:20.999+0000 7f22f66b3700 1 --2- 172.21.15.121:0/3399763135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22f0068670 0x7f22f0068a50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:20.999+0000 7f22f66b3700 1 -- 172.21.15.121:0/3399763135 >> 172.21.15.121:0/3399763135 conn(0x7f22f00755a0 msgr2=0x7f22f00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:21.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:20.999+0000 7f22f66b3700 1 -- 172.21.15.121:0/3399763135 shutdown_connections 2024-04-05T21:36:21.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:20.999+0000 7f22f66b3700 1 -- 172.21.15.121:0/3399763135 wait complete. 2024-04-05T21:36:21.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.000+0000 7f22f66b3700 1 Processor -- start 2024-04-05T21:36:21.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.000+0000 7f22f66b3700 1 -- start start 2024-04-05T21:36:21.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.000+0000 7f22f66b3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22f0068670 0x7f22f019c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:21.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.000+0000 7f22f66b3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22f0069020 0x7f22f019cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.000+0000 7f22f66b3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f019d170 con 0x7f22f0068670 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.000+0000 7f22f66b3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f0196620 con 0x7f22f0069020 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.000+0000 7f22effff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22f0068670 0x7f22f019c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.000+0000 7f22effff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22f0068670 0x7f22f019c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43220/0 (socket says 172.21.15.121:43220) 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.000+0000 7f22effff700 1 -- 172.21.15.121:0/1067859016 learned_addr learned my addr 172.21.15.121:0/1067859016 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.001+0000 7f22ef7fe700 1 --2- 172.21.15.121:0/1067859016 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22f0069020 0x7f22f019cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.001+0000 7f22effff700 1 -- 172.21.15.121:0/1067859016 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22f0069020 msgr2=0x7f22f019cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.001+0000 7f22effff700 1 --2- 172.21.15.121:0/1067859016 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22f0069020 0x7f22f019cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.001+0000 7f22effff700 1 -- 172.21.15.121:0/1067859016 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e40095f0 con 0x7f22f0068670 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.001+0000 7f22effff700 1 --2- 172.21.15.121:0/1067859016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22f0068670 0x7f22f019c5a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f22e4006010 tx=0x7f22e400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.001+0000 7f22ed7fa700 1 -- 172.21.15.121:0/1067859016 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22e401d070 con 0x7f22f0068670 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.001+0000 7f22f66b3700 1 -- 172.21.15.121:0/1067859016 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f01968a0 con 0x7f22f0068670 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.001+0000 7f22ed7fa700 1 -- 172.21.15.121:0/1067859016 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22e400f460 con 0x7f22f0068670 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.001+0000 7f22ed7fa700 1 -- 172.21.15.121:0/1067859016 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22e4021640 con 0x7f22f0068670 2024-04-05T21:36:21.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.001+0000 7f22f66b3700 1 -- 172.21.15.121:0/1067859016 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f0196d90 con 0x7f22f0068670 2024-04-05T21:36:21.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.002+0000 7f22f66b3700 1 -- 172.21.15.121:0/1067859016 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f0109130 con 0x7f22f0068670 2024-04-05T21:36:21.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.002+0000 7f22ed7fa700 1 -- 172.21.15.121:0/1067859016 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f22e400fa20 con 0x7f22f0068670 2024-04-05T21:36:21.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.003+0000 7f22ed7fa700 1 --2- 172.21.15.121:0/1067859016 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f22d806cce0 0x7f22d806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:21.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.003+0000 7f22ef7fe700 1 --2- 172.21.15.121:0/1067859016 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f22d806cce0 0x7f22d806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:21.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.003+0000 7f22ed7fa700 1 -- 172.21.15.121:0/1067859016 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f22e408d8c0 con 0x7f22f0068670 2024-04-05T21:36:21.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.004+0000 7f22ef7fe700 1 --2- 172.21.15.121:0/1067859016 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f22d806cce0 0x7f22d806f1a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f22e0006fd0 tx=0x7f22e0008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:21.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.008+0000 7f22ed7fa700 1 -- 172.21.15.121:0/1067859016 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f22e405b3c0 con 0x7f22f0068670 2024-04-05T21:36:21.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.216+0000 7f22f66b3700 1 -- 172.21.15.121:0/1067859016 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f22f0196fb0 con 0x7f22d806cce0 2024-04-05T21:36:21.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.217+0000 7f22ed7fa700 1 -- 172.21.15.121:0/1067859016 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f22f0196fb0 con 0x7f22d806cce0 2024-04-05T21:36:21.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.219+0000 7f22f66b3700 1 -- 172.21.15.121:0/1067859016 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f22d806cce0 msgr2=0x7f22d806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.219+0000 7f22f66b3700 1 --2- 172.21.15.121:0/1067859016 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f22d806cce0 0x7f22d806f1a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f22e0006fd0 tx=0x7f22e0008040 comp rx=0 tx=0).stop 2024-04-05T21:36:21.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.219+0000 7f22f66b3700 1 -- 172.21.15.121:0/1067859016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22f0068670 msgr2=0x7f22f019c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.219+0000 7f22f66b3700 1 --2- 172.21.15.121:0/1067859016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22f0068670 0x7f22f019c5a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f22e4006010 tx=0x7f22e400be20 comp rx=0 tx=0).stop 2024-04-05T21:36:21.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.219+0000 7f22f66b3700 1 -- 172.21.15.121:0/1067859016 shutdown_connections 2024-04-05T21:36:21.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.219+0000 7f22f66b3700 1 --2- 172.21.15.121:0/1067859016 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f22d806cce0 0x7f22d806f1a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.219+0000 7f22f66b3700 1 --2- 172.21.15.121:0/1067859016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22f0068670 0x7f22f019c5a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.219+0000 7f22f66b3700 1 --2- 172.21.15.121:0/1067859016 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22f0069020 0x7f22f019cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.219+0000 7f22f66b3700 1 -- 172.21.15.121:0/1067859016 >> 172.21.15.121:0/1067859016 conn(0x7f22f00755a0 msgr2=0x7f22f00fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:21.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.219+0000 7f22f66b3700 1 -- 172.21.15.121:0/1067859016 shutdown_connections 2024-04-05T21:36:21.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.219+0000 7f22f66b3700 1 -- 172.21.15.121:0/1067859016 wait complete. 2024-04-05T21:36:21.231 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:36:21.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.323+0000 7fd36919f700 1 -- 172.21.15.121:0/368927532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364102680 msgr2=0x7fd36410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.323+0000 7fd36919f700 1 --2- 172.21.15.121:0/368927532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364102680 0x7fd36410ab70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd354009940 tx=0x7fd354009c50 comp rx=0 tx=0).stop 2024-04-05T21:36:21.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.323+0000 7fd36919f700 1 -- 172.21.15.121:0/368927532 shutdown_connections 2024-04-05T21:36:21.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.323+0000 7fd36919f700 1 --2- 172.21.15.121:0/368927532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364102680 0x7fd36410ab70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.323+0000 7fd36919f700 1 --2- 172.21.15.121:0/368927532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd364101d60 0x7fd364102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.323+0000 7fd36919f700 1 -- 172.21.15.121:0/368927532 >> 172.21.15.121:0/368927532 conn(0x7fd3640fb570 msgr2=0x7fd3640fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:21.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.323+0000 7fd36919f700 1 -- 172.21.15.121:0/368927532 shutdown_connections 2024-04-05T21:36:21.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.324+0000 7fd36919f700 1 -- 172.21.15.121:0/368927532 wait complete. 2024-04-05T21:36:21.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.324+0000 7fd36919f700 1 Processor -- start 2024-04-05T21:36:21.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.324+0000 7fd36919f700 1 -- start start 2024-04-05T21:36:21.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd36919f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101d60 0x7fd3641966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:21.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd36919f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd364102680 0x7fd364196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd36919f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3641972c0 con 0x7fd364101d60 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd36919f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd364199ff0 con 0x7fd364102680 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd362d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101d60 0x7fd3641966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd362d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101d60 0x7fd3641966f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43244/0 (socket says 172.21.15.121:43244) 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd362d9d700 1 -- 172.21.15.121:0/894775838 learned_addr learned my addr 172.21.15.121:0/894775838 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd36259c700 1 --2- 172.21.15.121:0/894775838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd364102680 0x7fd364196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd362d9d700 1 -- 172.21.15.121:0/894775838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd364102680 msgr2=0x7fd364196c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd362d9d700 1 --2- 172.21.15.121:0/894775838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd364102680 0x7fd364196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd362d9d700 1 -- 172.21.15.121:0/894775838 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3540095f0 con 0x7fd364101d60 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd362d9d700 1 --2- 172.21.15.121:0/894775838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101d60 0x7fd3641966f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd35800d8d0 tx=0x7fd35800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd34bfff700 1 -- 172.21.15.121:0/894775838 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd358009940 con 0x7fd364101d60 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd36919f700 1 -- 172.21.15.121:0/894775838 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36419a2d0 con 0x7fd364101d60 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd34bfff700 1 -- 172.21.15.121:0/894775838 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd358010460 con 0x7fd364101d60 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd34bfff700 1 -- 172.21.15.121:0/894775838 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd35800f5f0 con 0x7fd364101d60 2024-04-05T21:36:21.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.325+0000 7fd36919f700 1 -- 172.21.15.121:0/894775838 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36419a820 con 0x7fd364101d60 2024-04-05T21:36:21.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.326+0000 7fd36919f700 1 -- 172.21.15.121:0/894775838 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd364108270 con 0x7fd364101d60 2024-04-05T21:36:21.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.327+0000 7fd34bfff700 1 -- 172.21.15.121:0/894775838 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd3580105d0 con 0x7fd364101d60 2024-04-05T21:36:21.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.327+0000 7fd34bfff700 1 --2- 172.21.15.121:0/894775838 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd34c06cce0 0x7fd34c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:21.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.327+0000 7fd34bfff700 1 -- 172.21.15.121:0/894775838 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd35808c5e0 con 0x7fd364101d60 2024-04-05T21:36:21.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.327+0000 7fd36259c700 1 --2- 172.21.15.121:0/894775838 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd34c06cce0 0x7fd34c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:21.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.328+0000 7fd36259c700 1 --2- 172.21.15.121:0/894775838 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd34c06cce0 0x7fd34c06f1a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd35400b7d0 tx=0x7fd354005e60 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:21.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.333+0000 7fd34bfff700 1 -- 172.21.15.121:0/894775838 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd358056650 con 0x7fd364101d60 2024-04-05T21:36:21.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.542+0000 7fd36919f700 1 -- 172.21.15.121:0/894775838 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd36419aa40 con 0x7fd34c06cce0 2024-04-05T21:36:21.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.543+0000 7fd34bfff700 1 -- 172.21.15.121:0/894775838 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd36419aa40 con 0x7fd34c06cce0 2024-04-05T21:36:21.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.546+0000 7fd36919f700 1 -- 172.21.15.121:0/894775838 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd34c06cce0 msgr2=0x7fd34c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.546+0000 7fd36919f700 1 --2- 172.21.15.121:0/894775838 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd34c06cce0 0x7fd34c06f1a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd35400b7d0 tx=0x7fd354005e60 comp rx=0 tx=0).stop 2024-04-05T21:36:21.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.546+0000 7fd36919f700 1 -- 172.21.15.121:0/894775838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101d60 msgr2=0x7fd3641966f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.546+0000 7fd36919f700 1 --2- 172.21.15.121:0/894775838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101d60 0x7fd3641966f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd35800d8d0 tx=0x7fd35800dc90 comp rx=0 tx=0).stop 2024-04-05T21:36:21.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.546+0000 7fd36919f700 1 -- 172.21.15.121:0/894775838 shutdown_connections 2024-04-05T21:36:21.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.546+0000 7fd36919f700 1 --2- 172.21.15.121:0/894775838 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd34c06cce0 0x7fd34c06f1a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.546+0000 7fd36919f700 1 --2- 172.21.15.121:0/894775838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101d60 0x7fd3641966f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.546+0000 7fd36919f700 1 --2- 172.21.15.121:0/894775838 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd364102680 0x7fd364196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.546+0000 7fd36919f700 1 -- 172.21.15.121:0/894775838 >> 172.21.15.121:0/894775838 conn(0x7fd3640fb570 msgr2=0x7fd3641053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:21.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.546+0000 7fd36919f700 1 -- 172.21.15.121:0/894775838 shutdown_connections 2024-04-05T21:36:21.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.546+0000 7fd36919f700 1 -- 172.21.15.121:0/894775838 wait complete. 2024-04-05T21:36:21.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.649+0000 7fa5deb47700 1 -- 172.21.15.121:0/3415941785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5d80ffa50 msgr2=0x7fa5d80ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.649+0000 7fa5deb47700 1 --2- 172.21.15.121:0/3415941785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5d80ffa50 0x7fa5d80ffed0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009910 tx=0x7fa5c4009c20 comp rx=0 tx=0).stop 2024-04-05T21:36:21.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.649+0000 7fa5deb47700 1 -- 172.21.15.121:0/3415941785 shutdown_connections 2024-04-05T21:36:21.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.649+0000 7fa5deb47700 1 --2- 172.21.15.121:0/3415941785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5d80ffa50 0x7fa5d80ffed0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.649+0000 7fa5deb47700 1 --2- 172.21.15.121:0/3415941785 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5d8105a50 0x7fa5d8105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.649+0000 7fa5deb47700 1 -- 172.21.15.121:0/3415941785 >> 172.21.15.121:0/3415941785 conn(0x7fa5d80fb540 msgr2=0x7fa5d80fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:21.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.649+0000 7fa5deb47700 1 -- 172.21.15.121:0/3415941785 shutdown_connections 2024-04-05T21:36:21.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.649+0000 7fa5deb47700 1 -- 172.21.15.121:0/3415941785 wait complete. 2024-04-05T21:36:21.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.649+0000 7fa5deb47700 1 Processor -- start 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.649+0000 7fa5deb47700 1 -- start start 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5deb47700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5d80ffa50 0x7fa5d80fbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5deb47700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5d8105a50 0x7fa5d80fc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5deb47700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80fc960 con 0x7fa5d8105a50 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5deb47700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80fcad0 con 0x7fa5d80ffa50 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5d7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5d8105a50 0x7fa5d80fc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5d7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5d8105a50 0x7fa5d80fc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43264/0 (socket says 172.21.15.121:43264) 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5d7fff700 1 -- 172.21.15.121:0/1595029264 learned_addr learned my addr 172.21.15.121:0/1595029264 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5dc8e3700 1 --2- 172.21.15.121:0/1595029264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5d80ffa50 0x7fa5d80fbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5d7fff700 1 -- 172.21.15.121:0/1595029264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5d80ffa50 msgr2=0x7fa5d80fbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5d7fff700 1 --2- 172.21.15.121:0/1595029264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5d80ffa50 0x7fa5d80fbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5d7fff700 1 -- 172.21.15.121:0/1595029264 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c40095f0 con 0x7fa5d8105a50 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5d7fff700 1 --2- 172.21.15.121:0/1595029264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5d8105a50 0x7fa5d80fc390 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4006010 tx=0x7fa5c400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5d5ffb700 1 -- 172.21.15.121:0/1595029264 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5c401d070 con 0x7fa5d8105a50 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5deb47700 1 -- 172.21.15.121:0/1595029264 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d81a2820 con 0x7fa5d8105a50 2024-04-05T21:36:21.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5d5ffb700 1 -- 172.21.15.121:0/1595029264 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5c4005710 con 0x7fa5d8105a50 2024-04-05T21:36:21.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.650+0000 7fa5d5ffb700 1 -- 172.21.15.121:0/1595029264 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5c400fe20 con 0x7fa5d8105a50 2024-04-05T21:36:21.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.651+0000 7fa5deb47700 1 -- 172.21.15.121:0/1595029264 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d81a2bd0 con 0x7fa5d8105a50 2024-04-05T21:36:21.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.651+0000 7fa5deb47700 1 -- 172.21.15.121:0/1595029264 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d8108ff0 con 0x7fa5d8105a50 2024-04-05T21:36:21.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.652+0000 7fa5d5ffb700 1 -- 172.21.15.121:0/1595029264 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa5c4003a60 con 0x7fa5d8105a50 2024-04-05T21:36:21.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.652+0000 7fa5d5ffb700 1 --2- 172.21.15.121:0/1595029264 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5c806cce0 0x7fa5c806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:21.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.652+0000 7fa5d5ffb700 1 -- 172.21.15.121:0/1595029264 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa5c408d7c0 con 0x7fa5d8105a50 2024-04-05T21:36:21.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.652+0000 7fa5dc8e3700 1 --2- 172.21.15.121:0/1595029264 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5c806cce0 0x7fa5c806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:21.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.653+0000 7fa5dc8e3700 1 --2- 172.21.15.121:0/1595029264 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5c806cce0 0x7fa5c806f1a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc005950 tx=0x7fa5cc00b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:21.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.657+0000 7fa5d5ffb700 1 -- 172.21.15.121:0/1595029264 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa5c4057830 con 0x7fa5d8105a50 2024-04-05T21:36:21.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:21 smithi121 ceph-mon[145859]: pgmap v303: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:21 smithi188 ceph-mon[107793]: pgmap v303: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:21.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.857+0000 7fa5deb47700 1 -- 172.21.15.121:0/1595029264 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5d80611d0 con 0x7fa5c806cce0 2024-04-05T21:36:21.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.867+0000 7fa5d5ffb700 1 -- 172.21.15.121:0/1595029264 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fa5d80611d0 con 0x7fa5c806cce0 2024-04-05T21:36:21.867 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (7m) 69s ago 18m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (18m) 69s ago 18m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (17m) 51s ago 17m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (5m) 69s ago 18m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (5m) 51s ago 17m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (6m) 69s ago 18m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (95s) 69s ago 12m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (75s) 69s ago 12m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (55s) 51s ago 12m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (12m) 51s ago 12m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (9m) 69s ago 20m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (8m) 51s ago 17m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (5m) 69s ago 20m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (5m) 51s ago 17m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (8m) 69s ago 18m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:36:21.868 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (7m) 51s ago 17m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:36:21.869 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (4m) 69s ago 16m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:36:21.869 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (4m) 69s ago 16m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:36:21.869 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (3m) 69s ago 15m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:36:21.869 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (3m) 51s ago 15m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:36:21.869 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (2m) 51s ago 14m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:36:21.869 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (114s) 51s ago 14m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:36:21.869 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (7m) 69s ago 18m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:36:21.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.869+0000 7fa5deb47700 1 -- 172.21.15.121:0/1595029264 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5c806cce0 msgr2=0x7fa5c806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.869+0000 7fa5deb47700 1 --2- 172.21.15.121:0/1595029264 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5c806cce0 0x7fa5c806f1a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc005950 tx=0x7fa5cc00b410 comp rx=0 tx=0).stop 2024-04-05T21:36:21.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.869+0000 7fa5deb47700 1 -- 172.21.15.121:0/1595029264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5d8105a50 msgr2=0x7fa5d80fc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.869+0000 7fa5deb47700 1 --2- 172.21.15.121:0/1595029264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5d8105a50 0x7fa5d80fc390 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4006010 tx=0x7fa5c400bd70 comp rx=0 tx=0).stop 2024-04-05T21:36:21.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.869+0000 7fa5deb47700 1 -- 172.21.15.121:0/1595029264 shutdown_connections 2024-04-05T21:36:21.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.869+0000 7fa5deb47700 1 --2- 172.21.15.121:0/1595029264 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5c806cce0 0x7fa5c806f1a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.869+0000 7fa5deb47700 1 --2- 172.21.15.121:0/1595029264 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5d80ffa50 0x7fa5d80fbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.869+0000 7fa5deb47700 1 --2- 172.21.15.121:0/1595029264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5d8105a50 0x7fa5d80fc390 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.869+0000 7fa5deb47700 1 -- 172.21.15.121:0/1595029264 >> 172.21.15.121:0/1595029264 conn(0x7fa5d80fb540 msgr2=0x7fa5d8069050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:21.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.869+0000 7fa5deb47700 1 -- 172.21.15.121:0/1595029264 shutdown_connections 2024-04-05T21:36:21.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.869+0000 7fa5deb47700 1 -- 172.21.15.121:0/1595029264 wait complete. 2024-04-05T21:36:21.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.971+0000 7fa1d05f8700 1 -- 172.21.15.121:0/109041522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c8105a40 msgr2=0x7fa1c8105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.971+0000 7fa1d05f8700 1 --2- 172.21.15.121:0/109041522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c8105a40 0x7fa1c8105e20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc009910 tx=0x7fa1bc009c20 comp rx=0 tx=0).stop 2024-04-05T21:36:21.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.971+0000 7fa1d05f8700 1 -- 172.21.15.121:0/109041522 shutdown_connections 2024-04-05T21:36:21.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.971+0000 7fa1d05f8700 1 --2- 172.21.15.121:0/109041522 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1c80ffa40 0x7fa1c80ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.971+0000 7fa1d05f8700 1 --2- 172.21.15.121:0/109041522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c8105a40 0x7fa1c8105e20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.971+0000 7fa1d05f8700 1 -- 172.21.15.121:0/109041522 >> 172.21.15.121:0/109041522 conn(0x7fa1c80fb570 msgr2=0x7fa1c80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.971+0000 7fa1d05f8700 1 -- 172.21.15.121:0/109041522 shutdown_connections 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.971+0000 7fa1d05f8700 1 -- 172.21.15.121:0/109041522 wait complete. 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.971+0000 7fa1d05f8700 1 Processor -- start 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.971+0000 7fa1d05f8700 1 -- start start 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1d05f8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1c80ffa40 0x7fa1c819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1d05f8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c8105a40 0x7fa1c819ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1d05f8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c819d100 con 0x7fa1c8105a40 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1d05f8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c81965b0 con 0x7fa1c80ffa40 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1cdb93700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c8105a40 0x7fa1c819ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1cdb93700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c8105a40 0x7fa1c819ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43278/0 (socket says 172.21.15.121:43278) 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1cdb93700 1 -- 172.21.15.121:0/3404663941 learned_addr learned my addr 172.21.15.121:0/3404663941 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1ce394700 1 --2- 172.21.15.121:0/3404663941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1c80ffa40 0x7fa1c819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:21.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1cdb93700 1 -- 172.21.15.121:0/3404663941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1c80ffa40 msgr2=0x7fa1c819c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:21.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1cdb93700 1 --2- 172.21.15.121:0/3404663941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1c80ffa40 0x7fa1c819c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:21.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1cdb93700 1 -- 172.21.15.121:0/3404663941 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc0095f0 con 0x7fa1c8105a40 2024-04-05T21:36:21.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1cdb93700 1 --2- 172.21.15.121:0/3404663941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c8105a40 0x7fa1c819ca70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400b4b0 tx=0x7fa1c400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:21.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1bb7fe700 1 -- 172.21.15.121:0/3404663941 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1c4011800 con 0x7fa1c8105a40 2024-04-05T21:36:21.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1bb7fe700 1 -- 172.21.15.121:0/3404663941 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1c4011e00 con 0x7fa1c8105a40 2024-04-05T21:36:21.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1d05f8700 1 -- 172.21.15.121:0/3404663941 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c8196890 con 0x7fa1c8105a40 2024-04-05T21:36:21.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1bb7fe700 1 -- 172.21.15.121:0/3404663941 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1c4010e60 con 0x7fa1c8105a40 2024-04-05T21:36:21.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.972+0000 7fa1d05f8700 1 -- 172.21.15.121:0/3404663941 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c8196de0 con 0x7fa1c8105a40 2024-04-05T21:36:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.973+0000 7fa1d05f8700 1 -- 172.21.15.121:0/3404663941 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c8109050 con 0x7fa1c8105a40 2024-04-05T21:36:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.974+0000 7fa1bb7fe700 1 -- 172.21.15.121:0/3404663941 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa1c4011960 con 0x7fa1c8105a40 2024-04-05T21:36:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.974+0000 7fa1bb7fe700 1 --2- 172.21.15.121:0/3404663941 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1b406cce0 0x7fa1b406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.974+0000 7fa1ce394700 1 --2- 172.21.15.121:0/3404663941 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1b406cce0 0x7fa1b406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.974+0000 7fa1bb7fe700 1 -- 172.21.15.121:0/3404663941 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa1c408c0f0 con 0x7fa1c8105a40 2024-04-05T21:36:21.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.975+0000 7fa1ce394700 1 --2- 172.21.15.121:0/3404663941 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1b406cce0 0x7fa1b406f1a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc006010 tx=0x7fa1bc01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:21.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:21.979+0000 7fa1bb7fe700 1 -- 172.21.15.121:0/3404663941 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa1c4059bf0 con 0x7fa1c8105a40 2024-04-05T21:36:22.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.255+0000 7fa1d05f8700 1 -- 172.21.15.121:0/3404663941 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa1c804ea90 con 0x7fa1c8105a40 2024-04-05T21:36:22.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.256+0000 7fa1bb7fe700 1 -- 172.21.15.121:0/3404663941 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7fa1c4059780 con 0x7fa1c8105a40 2024-04-05T21:36:22.257 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:36:22.257 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:36:22.257 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:36:22.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:36:22.257 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:36:22.257 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:36:22.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:36:22.257 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:36:22.257 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:36:22.257 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:36:22.257 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:36:22.257 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:36:22.257 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:36:22.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:36:22.258 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:36:22.258 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:36:22.258 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:36:22.258 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:36:22.258 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:36:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.258+0000 7fa1d05f8700 1 -- 172.21.15.121:0/3404663941 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1b406cce0 msgr2=0x7fa1b406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.258+0000 7fa1d05f8700 1 --2- 172.21.15.121:0/3404663941 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1b406cce0 0x7fa1b406f1a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc006010 tx=0x7fa1bc01a040 comp rx=0 tx=0).stop 2024-04-05T21:36:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.258+0000 7fa1d05f8700 1 -- 172.21.15.121:0/3404663941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c8105a40 msgr2=0x7fa1c819ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.258+0000 7fa1d05f8700 1 --2- 172.21.15.121:0/3404663941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c8105a40 0x7fa1c819ca70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400b4b0 tx=0x7fa1c400b7c0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.258+0000 7fa1d05f8700 1 -- 172.21.15.121:0/3404663941 shutdown_connections 2024-04-05T21:36:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.258+0000 7fa1d05f8700 1 --2- 172.21.15.121:0/3404663941 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1b406cce0 0x7fa1b406f1a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.258+0000 7fa1d05f8700 1 --2- 172.21.15.121:0/3404663941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1c80ffa40 0x7fa1c819c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.258+0000 7fa1d05f8700 1 --2- 172.21.15.121:0/3404663941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c8105a40 0x7fa1c819ca70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.258+0000 7fa1d05f8700 1 -- 172.21.15.121:0/3404663941 >> 172.21.15.121:0/3404663941 conn(0x7fa1c80fb570 msgr2=0x7fa1c80690f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.258+0000 7fa1d05f8700 1 -- 172.21.15.121:0/3404663941 shutdown_connections 2024-04-05T21:36:22.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.258+0000 7fa1d05f8700 1 -- 172.21.15.121:0/3404663941 wait complete. 2024-04-05T21:36:22.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.362+0000 7f23bf180700 1 -- 172.21.15.121:0/4240770449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b8100de0 msgr2=0x7f23b81011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:22.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.362+0000 7f23bf180700 1 --2- 172.21.15.121:0/4240770449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b8100de0 0x7f23b81011c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009910 tx=0x7f23ac009c20 comp rx=0 tx=0).stop 2024-04-05T21:36:22.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.362+0000 7f23bf180700 1 -- 172.21.15.121:0/4240770449 shutdown_connections 2024-04-05T21:36:22.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.362+0000 7f23bf180700 1 --2- 172.21.15.121:0/4240770449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23b8101790 0x7f23b81057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.362+0000 7f23bf180700 1 --2- 172.21.15.121:0/4240770449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b8100de0 0x7f23b81011c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.362+0000 7f23bf180700 1 -- 172.21.15.121:0/4240770449 >> 172.21.15.121:0/4240770449 conn(0x7f23b80fc6b0 msgr2=0x7f23b80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:22.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.362+0000 7f23bf180700 1 -- 172.21.15.121:0/4240770449 shutdown_connections 2024-04-05T21:36:22.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.362+0000 7f23bf180700 1 -- 172.21.15.121:0/4240770449 wait complete. 2024-04-05T21:36:22.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bf180700 1 Processor -- start 2024-04-05T21:36:22.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bf180700 1 -- start start 2024-04-05T21:36:22.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bf180700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b8100de0 0x7f23b81966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:22.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bf180700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23b8101790 0x7f23b8196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:22.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bf180700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b8197280 con 0x7f23b8100de0 2024-04-05T21:36:22.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bf180700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b8199fb0 con 0x7f23b8101790 2024-04-05T21:36:22.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bcf1c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b8100de0 0x7f23b81966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:22.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bcf1c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b8100de0 0x7f23b81966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43296/0 (socket says 172.21.15.121:43296) 2024-04-05T21:36:22.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bcf1c700 1 -- 172.21.15.121:0/2587728023 learned_addr learned my addr 172.21.15.121:0/2587728023 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:22.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23b7fff700 1 --2- 172.21.15.121:0/2587728023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23b8101790 0x7f23b8196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:22.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bcf1c700 1 -- 172.21.15.121:0/2587728023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23b8101790 msgr2=0x7f23b8196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:22.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bcf1c700 1 --2- 172.21.15.121:0/2587728023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23b8101790 0x7f23b8196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bcf1c700 1 -- 172.21.15.121:0/2587728023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac0095f0 con 0x7f23b8100de0 2024-04-05T21:36:22.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bcf1c700 1 --2- 172.21.15.121:0/2587728023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b8100de0 0x7f23b81966b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0092e0 tx=0x7f23ac00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:22.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23b5ffb700 1 -- 172.21.15.121:0/2587728023 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23ac01d070 con 0x7f23b8100de0 2024-04-05T21:36:22.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23b5ffb700 1 -- 172.21.15.121:0/2587728023 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23ac005e00 con 0x7f23b8100de0 2024-04-05T21:36:22.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.363+0000 7f23bf180700 1 -- 172.21.15.121:0/2587728023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b819a230 con 0x7f23b8100de0 2024-04-05T21:36:22.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.364+0000 7f23b5ffb700 1 -- 172.21.15.121:0/2587728023 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23ac00fe90 con 0x7f23b8100de0 2024-04-05T21:36:22.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.364+0000 7f23bf180700 1 -- 172.21.15.121:0/2587728023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b819a720 con 0x7f23b8100de0 2024-04-05T21:36:22.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.365+0000 7f23bf180700 1 -- 172.21.15.121:0/2587728023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b8109180 con 0x7f23b8100de0 2024-04-05T21:36:22.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.365+0000 7f23b5ffb700 1 -- 172.21.15.121:0/2587728023 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f23ac00f6d0 con 0x7f23b8100de0 2024-04-05T21:36:22.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.365+0000 7f23b5ffb700 1 --2- 172.21.15.121:0/2587728023 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f23a806cce0 0x7f23a806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:22.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.366+0000 7f23b5ffb700 1 -- 172.21.15.121:0/2587728023 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f23ac08d7e0 con 0x7f23b8100de0 2024-04-05T21:36:22.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.366+0000 7f23b7fff700 1 --2- 172.21.15.121:0/2587728023 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f23a806cce0 0x7f23a806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:22.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.366+0000 7f23b7fff700 1 --2- 172.21.15.121:0/2587728023 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f23a806cce0 0x7f23a806f1a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f23b8197cd0 tx=0x7f23a400b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:22.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.370+0000 7f23b5ffb700 1 -- 172.21.15.121:0/2587728023 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f23ac057850 con 0x7f23b8100de0 2024-04-05T21:36:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.606+0000 7f23bf180700 1 -- 172.21.15.121:0/2587728023 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f23b8108b60 con 0x7f23b8100de0 2024-04-05T21:36:22.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.607+0000 7f23b5ffb700 1 -- 172.21.15.121:0/2587728023 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f23ac05ae70 con 0x7f23b8100de0 2024-04-05T21:36:22.608 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:36:22.608 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:36:22.608 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:36:22.608 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:36:22.608 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:36:22.608 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:36:22.608 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:36:22.608 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:36:22.608 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:36:22.608 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:36:22.608 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:36:22.608 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:36:22.608 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:36:22.609 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:36:22.609 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:36:22.609 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:36:22.609 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:36:22.609 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:36:22.609 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:36:22.609 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:36:22.609 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:36:22.609 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:36:22.609 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:36:22.609 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:36:22.609 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:36:22.609 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:36:22.610 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:36:22.611 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:36:22.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.609+0000 7f23bf180700 1 -- 172.21.15.121:0/2587728023 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f23a806cce0 msgr2=0x7f23a806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:22.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.609+0000 7f23bf180700 1 --2- 172.21.15.121:0/2587728023 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f23a806cce0 0x7f23a806f1a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f23b8197cd0 tx=0x7f23a400b410 comp rx=0 tx=0).stop 2024-04-05T21:36:22.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.609+0000 7f23bf180700 1 -- 172.21.15.121:0/2587728023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b8100de0 msgr2=0x7f23b81966b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:22.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.609+0000 7f23bf180700 1 --2- 172.21.15.121:0/2587728023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b8100de0 0x7f23b81966b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0092e0 tx=0x7f23ac00bd70 comp rx=0 tx=0).stop 2024-04-05T21:36:22.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.609+0000 7f23bf180700 1 -- 172.21.15.121:0/2587728023 shutdown_connections 2024-04-05T21:36:22.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.609+0000 7f23bf180700 1 --2- 172.21.15.121:0/2587728023 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f23a806cce0 0x7f23a806f1a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.609+0000 7f23bf180700 1 --2- 172.21.15.121:0/2587728023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23b8100de0 0x7f23b81966b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.609+0000 7f23bf180700 1 --2- 172.21.15.121:0/2587728023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23b8101790 0x7f23b8196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.609+0000 7f23bf180700 1 -- 172.21.15.121:0/2587728023 >> 172.21.15.121:0/2587728023 conn(0x7f23b80fc6b0 msgr2=0x7f23b80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:22.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.609+0000 7f23bf180700 1 -- 172.21.15.121:0/2587728023 shutdown_connections 2024-04-05T21:36:22.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.609+0000 7f23bf180700 1 -- 172.21.15.121:0/2587728023 wait complete. 2024-04-05T21:36:22.612 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:36:22.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.712+0000 7fe79296a700 1 -- 172.21.15.121:0/2983339332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c101790 msgr2=0x7fe78c1057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:22.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.712+0000 7fe79296a700 1 --2- 172.21.15.121:0/2983339332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c101790 0x7fe78c1057e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe780009910 tx=0x7fe780009c20 comp rx=0 tx=0).stop 2024-04-05T21:36:22.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.712+0000 7fe79296a700 1 -- 172.21.15.121:0/2983339332 shutdown_connections 2024-04-05T21:36:22.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.712+0000 7fe79296a700 1 --2- 172.21.15.121:0/2983339332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c101790 0x7fe78c1057e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.712+0000 7fe79296a700 1 --2- 172.21.15.121:0/2983339332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe78c100de0 0x7fe78c1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.712+0000 7fe79296a700 1 -- 172.21.15.121:0/2983339332 >> 172.21.15.121:0/2983339332 conn(0x7fe78c0fc670 msgr2=0x7fe78c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:22.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.713+0000 7fe79296a700 1 -- 172.21.15.121:0/2983339332 shutdown_connections 2024-04-05T21:36:22.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.713+0000 7fe79296a700 1 -- 172.21.15.121:0/2983339332 wait complete. 2024-04-05T21:36:22.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.713+0000 7fe79296a700 1 Processor -- start 2024-04-05T21:36:22.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.713+0000 7fe79296a700 1 -- start start 2024-04-05T21:36:22.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.713+0000 7fe79296a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe78c100de0 0x7fe78c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:22.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.713+0000 7fe79296a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c101790 0x7fe78c19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:22.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.713+0000 7fe79296a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c19f230 con 0x7fe78c101790 2024-04-05T21:36:22.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.713+0000 7fe79296a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c1986e0 con 0x7fe78c100de0 2024-04-05T21:36:22.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.714+0000 7fe78b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c101790 0x7fe78c19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:22.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.714+0000 7fe78b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c101790 0x7fe78c19eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43326/0 (socket says 172.21.15.121:43326) 2024-04-05T21:36:22.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.714+0000 7fe78b7fe700 1 -- 172.21.15.121:0/1849663451 learned_addr learned my addr 172.21.15.121:0/1849663451 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:22.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.714+0000 7fe78bfff700 1 --2- 172.21.15.121:0/1849663451 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe78c100de0 0x7fe78c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:22.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.714+0000 7fe78b7fe700 1 -- 172.21.15.121:0/1849663451 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe78c100de0 msgr2=0x7fe78c19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.714+0000 7fe78b7fe700 1 --2- 172.21.15.121:0/1849663451 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe78c100de0 0x7fe78c19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.714+0000 7fe78b7fe700 1 -- 172.21.15.121:0/1849663451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7800095f0 con 0x7fe78c101790 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.714+0000 7fe78b7fe700 1 --2- 172.21.15.121:0/1849663451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c101790 0x7fe78c19eba0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe780000c00 tx=0x7fe78000ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.714+0000 7fe7897fa700 1 -- 172.21.15.121:0/1849663451 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe78001d070 con 0x7fe78c101790 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.714+0000 7fe79296a700 1 -- 172.21.15.121:0/1849663451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe78c198960 con 0x7fe78c101790 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.714+0000 7fe79296a700 1 -- 172.21.15.121:0/1849663451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe78c198df0 con 0x7fe78c101790 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.714+0000 7fe7897fa700 1 -- 172.21.15.121:0/1849663451 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe78000bed0 con 0x7fe78c101790 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.714+0000 7fe7897fa700 1 -- 172.21.15.121:0/1849663451 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe78000fc70 con 0x7fe78c101790 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.715+0000 7fe79296a700 1 -- 172.21.15.121:0/1849663451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe778005320 con 0x7fe78c101790 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.716+0000 7fe7897fa700 1 -- 172.21.15.121:0/1849663451 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe780003850 con 0x7fe78c101790 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.716+0000 7fe7897fa700 1 --2- 172.21.15.121:0/1849663451 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe77406cce0 0x7fe77406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.716+0000 7fe7897fa700 1 -- 172.21.15.121:0/1849663451 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe78008d670 con 0x7fe78c101790 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.716+0000 7fe78bfff700 1 --2- 172.21.15.121:0/1849663451 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe77406cce0 0x7fe77406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:22.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.716+0000 7fe78bfff700 1 --2- 172.21.15.121:0/1849663451 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe77406cce0 0x7fe77406f1a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe77c009c60 tx=0x7fe77c009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:22.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.720+0000 7fe7897fa700 1 -- 172.21.15.121:0/1849663451 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe78005b170 con 0x7fe78c101790 2024-04-05T21:36:22.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:22 smithi121 ceph-mon[145859]: from='client.34426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:22.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:22 smithi121 ceph-mon[145859]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:22.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:22 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3404663941' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:36:22.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:22 smithi188 ceph-mon[107793]: from='client.34426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:22.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:22 smithi188 ceph-mon[107793]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:22.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:22 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3404663941' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:36:22.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.928+0000 7fe79296a700 1 -- 172.21.15.121:0/1849663451 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe778000bf0 con 0x7fe77406cce0 2024-04-05T21:36:22.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.930+0000 7fe7897fa700 1 -- 172.21.15.121:0/1849663451 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe778000bf0 con 0x7fe77406cce0 2024-04-05T21:36:22.930 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:36:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:36:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:36:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:36:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:36:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:36:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:36:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:36:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:36:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:36:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:36:22.931 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:36:22.932 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:36:22.932 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:36:22.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.932+0000 7fe79296a700 1 -- 172.21.15.121:0/1849663451 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe77406cce0 msgr2=0x7fe77406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:22.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.932+0000 7fe79296a700 1 --2- 172.21.15.121:0/1849663451 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe77406cce0 0x7fe77406f1a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe77c009c60 tx=0x7fe77c009260 comp rx=0 tx=0).stop 2024-04-05T21:36:22.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.932+0000 7fe79296a700 1 -- 172.21.15.121:0/1849663451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c101790 msgr2=0x7fe78c19eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:22.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.932+0000 7fe79296a700 1 --2- 172.21.15.121:0/1849663451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c101790 0x7fe78c19eba0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe780000c00 tx=0x7fe78000ba70 comp rx=0 tx=0).stop 2024-04-05T21:36:22.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.932+0000 7fe79296a700 1 -- 172.21.15.121:0/1849663451 shutdown_connections 2024-04-05T21:36:22.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.932+0000 7fe79296a700 1 --2- 172.21.15.121:0/1849663451 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe77406cce0 0x7fe77406f1a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.932+0000 7fe79296a700 1 --2- 172.21.15.121:0/1849663451 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe78c100de0 0x7fe78c19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.932+0000 7fe79296a700 1 --2- 172.21.15.121:0/1849663451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c101790 0x7fe78c19eba0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:22.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.932+0000 7fe79296a700 1 -- 172.21.15.121:0/1849663451 >> 172.21.15.121:0/1849663451 conn(0x7fe78c0fc670 msgr2=0x7fe78c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:22.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.932+0000 7fe79296a700 1 -- 172.21.15.121:0/1849663451 shutdown_connections 2024-04-05T21:36:22.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:22.932+0000 7fe79296a700 1 -- 172.21.15.121:0/1849663451 wait complete. 2024-04-05T21:36:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.035+0000 7f895e293700 1 -- 172.21.15.121:0/4033300854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8958101750 msgr2=0x7f89581057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.035+0000 7f895e293700 1 --2- 172.21.15.121:0/4033300854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8958101750 0x7f89581057a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8948009910 tx=0x7f8948009c20 comp rx=0 tx=0).stop 2024-04-05T21:36:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.036+0000 7f895e293700 1 -- 172.21.15.121:0/4033300854 shutdown_connections 2024-04-05T21:36:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.036+0000 7f895e293700 1 --2- 172.21.15.121:0/4033300854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8958101750 0x7f89581057a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.036+0000 7f895e293700 1 --2- 172.21.15.121:0/4033300854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8958100da0 0x7f8958101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.036+0000 7f895e293700 1 -- 172.21.15.121:0/4033300854 >> 172.21.15.121:0/4033300854 conn(0x7f89580fc630 msgr2=0x7f89580fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:23.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.036+0000 7f895e293700 1 -- 172.21.15.121:0/4033300854 shutdown_connections 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.036+0000 7f895e293700 1 -- 172.21.15.121:0/4033300854 wait complete. 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.036+0000 7f895e293700 1 Processor -- start 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.036+0000 7f895e293700 1 -- start start 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f895e293700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8958100da0 0x7f8958073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f895e293700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8958101750 0x7f89580735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f895e293700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8958073b10 con 0x7f8958100da0 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f895e293700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8958073c50 con 0x7f8958101750 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f8957fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8958100da0 0x7f8958073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f8957fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8958100da0 0x7f8958073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43338/0 (socket says 172.21.15.121:43338) 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f8957fff700 1 -- 172.21.15.121:0/4261711636 learned_addr learned my addr 172.21.15.121:0/4261711636 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f89577fe700 1 --2- 172.21.15.121:0/4261711636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8958101750 0x7f89580735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f8957fff700 1 -- 172.21.15.121:0/4261711636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8958101750 msgr2=0x7f89580735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f8957fff700 1 --2- 172.21.15.121:0/4261711636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8958101750 0x7f89580735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:23.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f8957fff700 1 -- 172.21.15.121:0/4261711636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89480095f0 con 0x7f8958100da0 2024-04-05T21:36:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f8957fff700 1 --2- 172.21.15.121:0/4261711636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8958100da0 0x7f8958073090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f894000d8d0 tx=0x7f894000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f89557fa700 1 -- 172.21.15.121:0/4261711636 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8940009940 con 0x7f8958100da0 2024-04-05T21:36:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f89557fa700 1 -- 172.21.15.121:0/4261711636 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8940010460 con 0x7f8958100da0 2024-04-05T21:36:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f895e293700 1 -- 172.21.15.121:0/4261711636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89581a2820 con 0x7f8958100da0 2024-04-05T21:36:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f89557fa700 1 -- 172.21.15.121:0/4261711636 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f894000f5f0 con 0x7f8958100da0 2024-04-05T21:36:23.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.037+0000 7f895e293700 1 -- 172.21.15.121:0/4261711636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89581a2c30 con 0x7f8958100da0 2024-04-05T21:36:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.038+0000 7f895e293700 1 -- 172.21.15.121:0/4261711636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89581091a0 con 0x7f8958100da0 2024-04-05T21:36:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.039+0000 7f89557fa700 1 -- 172.21.15.121:0/4261711636 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f89400105d0 con 0x7f8958100da0 2024-04-05T21:36:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.039+0000 7f89557fa700 1 --2- 172.21.15.121:0/4261711636 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f894406cce0 0x7f894406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.039+0000 7f89557fa700 1 -- 172.21.15.121:0/4261711636 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f894008c5e0 con 0x7f8958100da0 2024-04-05T21:36:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.039+0000 7f89577fe700 1 --2- 172.21.15.121:0/4261711636 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f894406cce0 0x7f894406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:23.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.039+0000 7f89577fe700 1 --2- 172.21.15.121:0/4261711636 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f894406cce0 0x7f894406f1a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8958074880 tx=0x7f894801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:23.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.044+0000 7f89557fa700 1 -- 172.21.15.121:0/4261711636 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8940056650 con 0x7f8958100da0 2024-04-05T21:36:23.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.312+0000 7f895e293700 1 -- 172.21.15.121:0/4261711636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f895804ea90 con 0x7f8958100da0 2024-04-05T21:36:23.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.312+0000 7f89557fa700 1 -- 172.21.15.121:0/4261711636 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f8940010880 con 0x7f8958100da0 2024-04-05T21:36:23.315 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:36:23.315 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:36:23.315 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:36:23.315 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:36:23.315 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:36:23.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.314+0000 7f895e293700 1 -- 172.21.15.121:0/4261711636 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f894406cce0 msgr2=0x7f894406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:23.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.314+0000 7f895e293700 1 --2- 172.21.15.121:0/4261711636 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f894406cce0 0x7f894406f1a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8958074880 tx=0x7f894801a040 comp rx=0 tx=0).stop 2024-04-05T21:36:23.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.314+0000 7f895e293700 1 -- 172.21.15.121:0/4261711636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8958100da0 msgr2=0x7f8958073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:23.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.314+0000 7f895e293700 1 --2- 172.21.15.121:0/4261711636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8958100da0 0x7f8958073090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f894000d8d0 tx=0x7f894000dc90 comp rx=0 tx=0).stop 2024-04-05T21:36:23.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.315+0000 7f895e293700 1 -- 172.21.15.121:0/4261711636 shutdown_connections 2024-04-05T21:36:23.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.315+0000 7f895e293700 1 --2- 172.21.15.121:0/4261711636 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f894406cce0 0x7f894406f1a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:23.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.315+0000 7f895e293700 1 --2- 172.21.15.121:0/4261711636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8958100da0 0x7f8958073090 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:23.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.315+0000 7f895e293700 1 --2- 172.21.15.121:0/4261711636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8958101750 0x7f89580735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:23.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.315+0000 7f895e293700 1 -- 172.21.15.121:0/4261711636 >> 172.21.15.121:0/4261711636 conn(0x7f89580fc630 msgr2=0x7f89580fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:23.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.315+0000 7f895e293700 1 -- 172.21.15.121:0/4261711636 shutdown_connections 2024-04-05T21:36:23.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:23.315+0000 7f895e293700 1 -- 172.21.15.121:0/4261711636 wait complete. 2024-04-05T21:36:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:23 smithi121 ceph-mon[145859]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:23 smithi121 ceph-mon[145859]: pgmap v304: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:23 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2587728023' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:36:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:23 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4261711636' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:36:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:23 smithi188 ceph-mon[107793]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:23 smithi188 ceph-mon[107793]: pgmap v304: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:23 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2587728023' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:36:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:23 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4261711636' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:36:24.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:24 smithi121 ceph-mon[145859]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:24.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:24 smithi188 ceph-mon[107793]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:25.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:25 smithi121 ceph-mon[145859]: pgmap v305: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:25.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:25 smithi188 ceph-mon[107793]: pgmap v305: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:27.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:27 smithi121 ceph-mon[145859]: pgmap v306: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:27.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:27 smithi188 ceph-mon[107793]: pgmap v306: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:29 smithi121 ceph-mon[145859]: pgmap v307: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:29 smithi188 ceph-mon[107793]: pgmap v307: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:36:30.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:36:31.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:31 smithi121 ceph-mon[145859]: pgmap v308: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:31.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:31 smithi188 ceph-mon[107793]: pgmap v308: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:33 smithi121 ceph-mon[145859]: pgmap v309: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:33 smithi188 ceph-mon[107793]: pgmap v309: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:35 smithi121 ceph-mon[145859]: pgmap v310: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:35.617 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:35 smithi188 ceph-mon[107793]: pgmap v310: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:37.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:37 smithi121 ceph-mon[145859]: pgmap v311: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:37.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:37 smithi188 ceph-mon[107793]: pgmap v311: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:39.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:39 smithi121 ceph-mon[145859]: pgmap v312: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:39.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:39 smithi188 ceph-mon[107793]: pgmap v312: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:41.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:41 smithi121 ceph-mon[145859]: pgmap v313: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:41.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:41 smithi188 ceph-mon[107793]: pgmap v313: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:43 smithi121 ceph-mon[145859]: pgmap v314: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:43.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:43 smithi188 ceph-mon[107793]: pgmap v314: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:45.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:45 smithi121 ceph-mon[145859]: pgmap v315: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:45.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:36:45.709 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:45 smithi188 ceph-mon[107793]: pgmap v315: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:45.710 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:36:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:47 smithi121 ceph-mon[145859]: pgmap v316: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:47 smithi188 ceph-mon[107793]: pgmap v316: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:49 smithi121 ceph-mon[145859]: pgmap v317: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:49.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:49 smithi188 ceph-mon[107793]: pgmap v317: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:51 smithi121 ceph-mon[145859]: pgmap v318: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:51 smithi188 ceph-mon[107793]: pgmap v318: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.432+0000 7fb1aa234700 1 -- 172.21.15.121:0/4203366007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1a4100da0 msgr2=0x7fb1a4101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.432+0000 7fb1aa234700 1 --2- 172.21.15.121:0/4203366007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1a4100da0 0x7fb1a4101180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb194009910 tx=0x7fb194009c20 comp rx=0 tx=0).stop 2024-04-05T21:36:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.433+0000 7fb1aa234700 1 -- 172.21.15.121:0/4203366007 shutdown_connections 2024-04-05T21:36:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.433+0000 7fb1aa234700 1 --2- 172.21.15.121:0/4203366007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1a4101750 0x7fb1a41057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.433+0000 7fb1aa234700 1 --2- 172.21.15.121:0/4203366007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1a4100da0 0x7fb1a4101180 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:53.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.433+0000 7fb1aa234700 1 -- 172.21.15.121:0/4203366007 >> 172.21.15.121:0/4203366007 conn(0x7fb1a40fc630 msgr2=0x7fb1a40fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:53.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.433+0000 7fb1aa234700 1 -- 172.21.15.121:0/4203366007 shutdown_connections 2024-04-05T21:36:53.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.433+0000 7fb1aa234700 1 -- 172.21.15.121:0/4203366007 wait complete. 2024-04-05T21:36:53.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.433+0000 7fb1aa234700 1 Processor -- start 2024-04-05T21:36:53.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.433+0000 7fb1aa234700 1 -- start start 2024-04-05T21:36:53.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1aa234700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1a4100da0 0x7fb1a4196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:53.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1aa234700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1a4101750 0x7fb1a4196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:53.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1aa234700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a4197260 con 0x7fb1a4100da0 2024-04-05T21:36:53.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1aa234700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a4199f90 con 0x7fb1a4101750 2024-04-05T21:36:53.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1a37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1a4100da0 0x7fb1a4196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:53.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1a37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1a4100da0 0x7fb1a4196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42644/0 (socket says 172.21.15.121:42644) 2024-04-05T21:36:53.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1a37fe700 1 -- 172.21.15.121:0/2004032034 learned_addr learned my addr 172.21.15.121:0/2004032034 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:53.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1a2ffd700 1 --2- 172.21.15.121:0/2004032034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1a4101750 0x7fb1a4196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:53.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1a37fe700 1 -- 172.21.15.121:0/2004032034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1a4101750 msgr2=0x7fb1a4196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:53.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1a37fe700 1 --2- 172.21.15.121:0/2004032034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1a4101750 0x7fb1a4196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:53.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1a37fe700 1 -- 172.21.15.121:0/2004032034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1940095f0 con 0x7fb1a4100da0 2024-04-05T21:36:53.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1a37fe700 1 --2- 172.21.15.121:0/2004032034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1a4100da0 0x7fb1a4196690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb194006010 tx=0x7fb19400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:53.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1a0ff9700 1 -- 172.21.15.121:0/2004032034 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb19401d070 con 0x7fb1a4100da0 2024-04-05T21:36:53.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1a0ff9700 1 -- 172.21.15.121:0/2004032034 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb19400f460 con 0x7fb1a4100da0 2024-04-05T21:36:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1aa234700 1 -- 172.21.15.121:0/2004032034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a419a210 con 0x7fb1a4100da0 2024-04-05T21:36:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.434+0000 7fb1a0ff9700 1 -- 172.21.15.121:0/2004032034 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb194021640 con 0x7fb1a4100da0 2024-04-05T21:36:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.435+0000 7fb1aa234700 1 -- 172.21.15.121:0/2004032034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a419a700 con 0x7fb1a4100da0 2024-04-05T21:36:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.435+0000 7fb1aa234700 1 -- 172.21.15.121:0/2004032034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a4109140 con 0x7fb1a4100da0 2024-04-05T21:36:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.436+0000 7fb1a0ff9700 1 -- 172.21.15.121:0/2004032034 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb19400f9e0 con 0x7fb1a4100da0 2024-04-05T21:36:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.436+0000 7fb1a0ff9700 1 --2- 172.21.15.121:0/2004032034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb18c06cce0 0x7fb18c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.436+0000 7fb1a0ff9700 1 -- 172.21.15.121:0/2004032034 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb19408d810 con 0x7fb1a4100da0 2024-04-05T21:36:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.436+0000 7fb1a2ffd700 1 --2- 172.21.15.121:0/2004032034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb18c06cce0 0x7fb18c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.437+0000 7fb1a2ffd700 1 --2- 172.21.15.121:0/2004032034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb18c06cce0 0x7fb18c06f1a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4197cb0 tx=0x7fb19800a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:53.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.441+0000 7fb1a0ff9700 1 -- 172.21.15.121:0/2004032034 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb194057880 con 0x7fb1a4100da0 2024-04-05T21:36:53.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.655+0000 7fb1aa234700 1 -- 172.21.15.121:0/2004032034 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1a419a920 con 0x7fb18c06cce0 2024-04-05T21:36:53.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.656+0000 7fb1a0ff9700 1 -- 172.21.15.121:0/2004032034 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb1a419a920 con 0x7fb18c06cce0 2024-04-05T21:36:53.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.658+0000 7fb1aa234700 1 -- 172.21.15.121:0/2004032034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb18c06cce0 msgr2=0x7fb18c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:53.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.658+0000 7fb1aa234700 1 --2- 172.21.15.121:0/2004032034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb18c06cce0 0x7fb18c06f1a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4197cb0 tx=0x7fb19800a300 comp rx=0 tx=0).stop 2024-04-05T21:36:53.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.658+0000 7fb1aa234700 1 -- 172.21.15.121:0/2004032034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1a4100da0 msgr2=0x7fb1a4196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:53.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.658+0000 7fb1aa234700 1 --2- 172.21.15.121:0/2004032034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1a4100da0 0x7fb1a4196690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb194006010 tx=0x7fb19400bd70 comp rx=0 tx=0).stop 2024-04-05T21:36:53.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.658+0000 7fb1aa234700 1 -- 172.21.15.121:0/2004032034 shutdown_connections 2024-04-05T21:36:53.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.658+0000 7fb1aa234700 1 --2- 172.21.15.121:0/2004032034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb18c06cce0 0x7fb18c06f1a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:53.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.658+0000 7fb1aa234700 1 --2- 172.21.15.121:0/2004032034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1a4100da0 0x7fb1a4196690 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:53.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.658+0000 7fb1aa234700 1 --2- 172.21.15.121:0/2004032034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb1a4101750 0x7fb1a4196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:53.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.658+0000 7fb1aa234700 1 -- 172.21.15.121:0/2004032034 >> 172.21.15.121:0/2004032034 conn(0x7fb1a40fc630 msgr2=0x7fb1a40fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:53.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.658+0000 7fb1aa234700 1 -- 172.21.15.121:0/2004032034 shutdown_connections 2024-04-05T21:36:53.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.658+0000 7fb1aa234700 1 -- 172.21.15.121:0/2004032034 wait complete. 2024-04-05T21:36:53.669 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:36:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:53 smithi121 ceph-mon[145859]: pgmap v319: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:53.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.763+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2769661163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ae4105a40 msgr2=0x7f9ae4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:53.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.763+0000 7f9aeae4a700 1 --2- 172.21.15.121:0/2769661163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ae4105a40 0x7f9ae4105e20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8009910 tx=0x7f9ad8009c20 comp rx=0 tx=0).stop 2024-04-05T21:36:53.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.763+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2769661163 shutdown_connections 2024-04-05T21:36:53.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.763+0000 7f9aeae4a700 1 --2- 172.21.15.121:0/2769661163 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ae40ffbd0 0x7f9ae4100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:53.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.763+0000 7f9aeae4a700 1 --2- 172.21.15.121:0/2769661163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ae4105a40 0x7f9ae4105e20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:53.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.763+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2769661163 >> 172.21.15.121:0/2769661163 conn(0x7f9ae40755a0 msgr2=0x7f9ae40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:53.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.763+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2769661163 shutdown_connections 2024-04-05T21:36:53.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.763+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2769661163 wait complete. 2024-04-05T21:36:53.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9aeae4a700 1 Processor -- start 2024-04-05T21:36:53.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9aeae4a700 1 -- start start 2024-04-05T21:36:53.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9aeae4a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ae40ffbd0 0x7f9ae4195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:53.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9aeae4a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ae4105a40 0x7f9ae4195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:53.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9aeae4a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae4196170 con 0x7f9ae40ffbd0 2024-04-05T21:36:53.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9aeae4a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae4199f00 con 0x7f9ae4105a40 2024-04-05T21:36:53.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9ae8be6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ae40ffbd0 0x7f9ae4195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:53.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9ae8be6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ae40ffbd0 0x7f9ae4195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42658/0 (socket says 172.21.15.121:42658) 2024-04-05T21:36:53.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9ae8be6700 1 -- 172.21.15.121:0/2470568680 learned_addr learned my addr 172.21.15.121:0/2470568680 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:53.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9ae3fff700 1 --2- 172.21.15.121:0/2470568680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ae4105a40 0x7f9ae4195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:53.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9ae8be6700 1 -- 172.21.15.121:0/2470568680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ae4105a40 msgr2=0x7f9ae4195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:53.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9ae8be6700 1 --2- 172.21.15.121:0/2470568680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ae4105a40 0x7f9ae4195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:53.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9ae8be6700 1 -- 172.21.15.121:0/2470568680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad80095f0 con 0x7f9ae40ffbd0 2024-04-05T21:36:53.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.764+0000 7f9ae8be6700 1 --2- 172.21.15.121:0/2470568680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ae40ffbd0 0x7f9ae4195550 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8006010 tx=0x7f9ad800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:53.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.765+0000 7f9ae1ffb700 1 -- 172.21.15.121:0/2470568680 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ad801d070 con 0x7f9ae40ffbd0 2024-04-05T21:36:53.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.765+0000 7f9ae1ffb700 1 -- 172.21.15.121:0/2470568680 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ad8005710 con 0x7f9ae40ffbd0 2024-04-05T21:36:53.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.765+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2470568680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae419a180 con 0x7f9ae40ffbd0 2024-04-05T21:36:53.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.765+0000 7f9ae1ffb700 1 -- 172.21.15.121:0/2470568680 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ad800fe20 con 0x7f9ae40ffbd0 2024-04-05T21:36:53.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.765+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2470568680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae419a670 con 0x7f9ae40ffbd0 2024-04-05T21:36:53.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.766+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2470568680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae4108040 con 0x7f9ae40ffbd0 2024-04-05T21:36:53.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.766+0000 7f9ae1ffb700 1 -- 172.21.15.121:0/2470568680 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9ad8003a60 con 0x7f9ae40ffbd0 2024-04-05T21:36:53.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.767+0000 7f9ae1ffb700 1 --2- 172.21.15.121:0/2470568680 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9ad406cce0 0x7f9ad406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:53.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.767+0000 7f9ae1ffb700 1 -- 172.21.15.121:0/2470568680 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9ad808d7b0 con 0x7f9ae40ffbd0 2024-04-05T21:36:53.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.767+0000 7f9ae3fff700 1 --2- 172.21.15.121:0/2470568680 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9ad406cce0 0x7f9ad406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:53.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.767+0000 7f9ae3fff700 1 --2- 172.21.15.121:0/2470568680 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9ad406cce0 0x7f9ad406f1a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4196b70 tx=0x7f9ad000b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:53.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.771+0000 7f9ae1ffb700 1 -- 172.21.15.121:0/2470568680 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9ad80577c0 con 0x7f9ae40ffbd0 2024-04-05T21:36:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:53 smithi188 ceph-mon[107793]: pgmap v319: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:53.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.982+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2470568680 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ae402cd30 con 0x7f9ad406cce0 2024-04-05T21:36:53.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.984+0000 7f9ae1ffb700 1 -- 172.21.15.121:0/2470568680 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9ae402cd30 con 0x7f9ad406cce0 2024-04-05T21:36:53.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.986+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2470568680 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9ad406cce0 msgr2=0x7f9ad406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:53.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.986+0000 7f9aeae4a700 1 --2- 172.21.15.121:0/2470568680 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9ad406cce0 0x7f9ad406f1a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4196b70 tx=0x7f9ad000b410 comp rx=0 tx=0).stop 2024-04-05T21:36:53.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.986+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2470568680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ae40ffbd0 msgr2=0x7f9ae4195550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:53.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.986+0000 7f9aeae4a700 1 --2- 172.21.15.121:0/2470568680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ae40ffbd0 0x7f9ae4195550 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8006010 tx=0x7f9ad800bd70 comp rx=0 tx=0).stop 2024-04-05T21:36:53.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.986+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2470568680 shutdown_connections 2024-04-05T21:36:53.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.986+0000 7f9aeae4a700 1 --2- 172.21.15.121:0/2470568680 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9ad406cce0 0x7f9ad406f1a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:53.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.986+0000 7f9aeae4a700 1 --2- 172.21.15.121:0/2470568680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ae40ffbd0 0x7f9ae4195550 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:53.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.986+0000 7f9aeae4a700 1 --2- 172.21.15.121:0/2470568680 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ae4105a40 0x7f9ae4195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:53.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.986+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2470568680 >> 172.21.15.121:0/2470568680 conn(0x7f9ae40755a0 msgr2=0x7f9ae40fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:53.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.986+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2470568680 shutdown_connections 2024-04-05T21:36:53.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:53.986+0000 7f9aeae4a700 1 -- 172.21.15.121:0/2470568680 wait complete. 2024-04-05T21:36:54.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.089+0000 7fc37c883700 1 -- 172.21.15.121:0/3132700290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc378100da0 msgr2=0x7fc378101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.089+0000 7fc37c883700 1 --2- 172.21.15.121:0/3132700290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc378100da0 0x7fc378101180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009910 tx=0x7fc36c009c20 comp rx=0 tx=0).stop 2024-04-05T21:36:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.089+0000 7fc37c883700 1 -- 172.21.15.121:0/3132700290 shutdown_connections 2024-04-05T21:36:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.089+0000 7fc37c883700 1 --2- 172.21.15.121:0/3132700290 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc378101750 0x7fc3781057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.089+0000 7fc37c883700 1 --2- 172.21.15.121:0/3132700290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc378100da0 0x7fc378101180 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.089+0000 7fc37c883700 1 -- 172.21.15.121:0/3132700290 >> 172.21.15.121:0/3132700290 conn(0x7fc3780fc630 msgr2=0x7fc3780fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.089+0000 7fc37c883700 1 -- 172.21.15.121:0/3132700290 shutdown_connections 2024-04-05T21:36:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.089+0000 7fc37c883700 1 -- 172.21.15.121:0/3132700290 wait complete. 2024-04-05T21:36:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.090+0000 7fc37c883700 1 Processor -- start 2024-04-05T21:36:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.090+0000 7fc37c883700 1 -- start start 2024-04-05T21:36:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.090+0000 7fc37c883700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc378100da0 0x7fc378073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.090+0000 7fc37c883700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc378101750 0x7fc3780735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.090+0000 7fc37c883700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc378073b10 con 0x7fc378101750 2024-04-05T21:36:54.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.090+0000 7fc37c883700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc378073c50 con 0x7fc378100da0 2024-04-05T21:36:54.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.090+0000 7fc375d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc378101750 0x7fc3780735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:54.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.090+0000 7fc375d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc378101750 0x7fc3780735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42684/0 (socket says 172.21.15.121:42684) 2024-04-05T21:36:54.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.090+0000 7fc375d9b700 1 -- 172.21.15.121:0/608085248 learned_addr learned my addr 172.21.15.121:0/608085248 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:54.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.090+0000 7fc37659c700 1 --2- 172.21.15.121:0/608085248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc378100da0 0x7fc378073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:54.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.090+0000 7fc375d9b700 1 -- 172.21.15.121:0/608085248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc378100da0 msgr2=0x7fc378073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:54.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.090+0000 7fc375d9b700 1 --2- 172.21.15.121:0/608085248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc378100da0 0x7fc378073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.090+0000 7fc375d9b700 1 -- 172.21.15.121:0/608085248 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c0095f0 con 0x7fc378101750 2024-04-05T21:36:54.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.091+0000 7fc375d9b700 1 --2- 172.21.15.121:0/608085248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc378101750 0x7fc3780735d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc36800e8f0 tx=0x7fc36800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:54.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.091+0000 7fc3677fe700 1 -- 172.21.15.121:0/608085248 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc368009790 con 0x7fc378101750 2024-04-05T21:36:54.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.091+0000 7fc37c883700 1 -- 172.21.15.121:0/608085248 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3781a2860 con 0x7fc378101750 2024-04-05T21:36:54.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.091+0000 7fc3677fe700 1 -- 172.21.15.121:0/608085248 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc368004cd0 con 0x7fc378101750 2024-04-05T21:36:54.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.091+0000 7fc3677fe700 1 -- 172.21.15.121:0/608085248 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc368005580 con 0x7fc378101750 2024-04-05T21:36:54.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.091+0000 7fc37c883700 1 -- 172.21.15.121:0/608085248 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3781a2c70 con 0x7fc378101750 2024-04-05T21:36:54.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.092+0000 7fc37c883700 1 -- 172.21.15.121:0/608085248 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc378109140 con 0x7fc378101750 2024-04-05T21:36:54.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.092+0000 7fc3677fe700 1 -- 172.21.15.121:0/608085248 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc368004750 con 0x7fc378101750 2024-04-05T21:36:54.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.093+0000 7fc3677fe700 1 --2- 172.21.15.121:0/608085248 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc36006cce0 0x7fc36006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:54.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.093+0000 7fc3677fe700 1 -- 172.21.15.121:0/608085248 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc368014070 con 0x7fc378101750 2024-04-05T21:36:54.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.093+0000 7fc37659c700 1 --2- 172.21.15.121:0/608085248 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc36006cce0 0x7fc36006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:54.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.093+0000 7fc37659c700 1 --2- 172.21.15.121:0/608085248 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc36006cce0 0x7fc36006f1a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc36c006010 tx=0x7fc36c00b5a0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:54.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.097+0000 7fc3677fe700 1 -- 172.21.15.121:0/608085248 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc3680569c0 con 0x7fc378101750 2024-04-05T21:36:54.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.297+0000 7fc37c883700 1 -- 172.21.15.121:0/608085248 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc37802cda0 con 0x7fc36006cce0 2024-04-05T21:36:54.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.308+0000 7fc3677fe700 1 -- 172.21.15.121:0/608085248 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7fc37802cda0 con 0x7fc36006cce0 2024-04-05T21:36:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:36:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (7m) 101s ago 19m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:36:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (19m) 101s ago 19m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:36:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (18m) 84s ago 18m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:36:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (5m) 101s ago 19m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:36:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (5m) 84s ago 18m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:36:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (7m) 101s ago 18m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:36:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (2m) 101s ago 13m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:36:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (108s) 101s ago 12m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:36:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (88s) 84s ago 13m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:36:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (12m) 84s ago 12m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:36:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (9m) 101s ago 20m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:36:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (8m) 84s ago 18m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:36:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (6m) 101s ago 20m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:36:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (6m) 84s ago 18m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:36:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (8m) 101s ago 19m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:36:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (8m) 84s ago 18m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:36:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (5m) 101s ago 17m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:36:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (4m) 101s ago 16m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:36:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (4m) 101s ago 16m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:36:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (3m) 84s ago 15m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:36:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (3m) 84s ago 15m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:36:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (2m) 84s ago 14m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:36:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (8m) 101s ago 18m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:36:54.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.310+0000 7fc37c883700 1 -- 172.21.15.121:0/608085248 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc36006cce0 msgr2=0x7fc36006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:54.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.310+0000 7fc37c883700 1 --2- 172.21.15.121:0/608085248 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc36006cce0 0x7fc36006f1a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc36c006010 tx=0x7fc36c00b5a0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.310+0000 7fc37c883700 1 -- 172.21.15.121:0/608085248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc378101750 msgr2=0x7fc3780735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:54.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.310+0000 7fc37c883700 1 --2- 172.21.15.121:0/608085248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc378101750 0x7fc3780735d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc36800e8f0 tx=0x7fc36800ecb0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.310+0000 7fc37c883700 1 -- 172.21.15.121:0/608085248 shutdown_connections 2024-04-05T21:36:54.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.310+0000 7fc37c883700 1 --2- 172.21.15.121:0/608085248 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc36006cce0 0x7fc36006f1a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.310+0000 7fc37c883700 1 --2- 172.21.15.121:0/608085248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc378100da0 0x7fc378073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.310+0000 7fc37c883700 1 --2- 172.21.15.121:0/608085248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc378101750 0x7fc3780735d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.310+0000 7fc37c883700 1 -- 172.21.15.121:0/608085248 >> 172.21.15.121:0/608085248 conn(0x7fc3780fc630 msgr2=0x7fc3780fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:54.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.310+0000 7fc37c883700 1 -- 172.21.15.121:0/608085248 shutdown_connections 2024-04-05T21:36:54.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.310+0000 7fc37c883700 1 -- 172.21.15.121:0/608085248 wait complete. 2024-04-05T21:36:54.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.412+0000 7f86e6f89700 1 -- 172.21.15.121:0/4071032427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e0101770 msgr2=0x7f86e01057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:54.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.412+0000 7f86e6f89700 1 --2- 172.21.15.121:0/4071032427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e0101770 0x7f86e01057c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f86cc009910 tx=0x7f86cc009c20 comp rx=0 tx=0).stop 2024-04-05T21:36:54.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.412+0000 7f86e6f89700 1 -- 172.21.15.121:0/4071032427 shutdown_connections 2024-04-05T21:36:54.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.412+0000 7f86e6f89700 1 --2- 172.21.15.121:0/4071032427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e0101770 0x7f86e01057c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.412+0000 7f86e6f89700 1 --2- 172.21.15.121:0/4071032427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86e0100dc0 0x7f86e01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.412+0000 7f86e6f89700 1 -- 172.21.15.121:0/4071032427 >> 172.21.15.121:0/4071032427 conn(0x7f86e00fc670 msgr2=0x7f86e00fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:54.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.412+0000 7f86e6f89700 1 -- 172.21.15.121:0/4071032427 shutdown_connections 2024-04-05T21:36:54.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.412+0000 7f86e6f89700 1 -- 172.21.15.121:0/4071032427 wait complete. 2024-04-05T21:36:54.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.413+0000 7f86e6f89700 1 Processor -- start 2024-04-05T21:36:54.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.413+0000 7f86e6f89700 1 -- start start 2024-04-05T21:36:54.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.413+0000 7f86e6f89700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e0100dc0 0x7f86e0196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:54.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.413+0000 7f86e6f89700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86e0101770 0x7f86e0196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:54.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.413+0000 7f86e6f89700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e0197270 con 0x7f86e0100dc0 2024-04-05T21:36:54.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.413+0000 7f86e6f89700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e019b000 con 0x7f86e0101770 2024-04-05T21:36:54.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.413+0000 7f86e4d25700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e0100dc0 0x7f86e0196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:54.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.413+0000 7f86e4d25700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e0100dc0 0x7f86e0196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33648/0 (socket says 172.21.15.121:33648) 2024-04-05T21:36:54.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.413+0000 7f86e4d25700 1 -- 172.21.15.121:0/2868357958 learned_addr learned my addr 172.21.15.121:0/2868357958 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:54.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.413+0000 7f86dffff700 1 --2- 172.21.15.121:0/2868357958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86e0101770 0x7f86e0196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:54.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.414+0000 7f86e4d25700 1 -- 172.21.15.121:0/2868357958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86e0101770 msgr2=0x7f86e0196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:54.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.414+0000 7f86e4d25700 1 --2- 172.21.15.121:0/2868357958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86e0101770 0x7f86e0196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.414+0000 7f86e4d25700 1 -- 172.21.15.121:0/2868357958 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86cc0095f0 con 0x7f86e0100dc0 2024-04-05T21:36:54.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.414+0000 7f86e4d25700 1 --2- 172.21.15.121:0/2868357958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e0100dc0 0x7f86e0196650 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f86d400c8a0 tx=0x7f86d400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:54.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.414+0000 7f86ddffb700 1 -- 172.21.15.121:0/2868357958 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86d40078e0 con 0x7f86e0100dc0 2024-04-05T21:36:54.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.414+0000 7f86ddffb700 1 -- 172.21.15.121:0/2868357958 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86d400f450 con 0x7f86e0100dc0 2024-04-05T21:36:54.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.414+0000 7f86e6f89700 1 -- 172.21.15.121:0/2868357958 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e019b2e0 con 0x7f86e0100dc0 2024-04-05T21:36:54.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.414+0000 7f86ddffb700 1 -- 172.21.15.121:0/2868357958 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86d400e5e0 con 0x7f86e0100dc0 2024-04-05T21:36:54.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.414+0000 7f86e6f89700 1 -- 172.21.15.121:0/2868357958 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e019b830 con 0x7f86e0100dc0 2024-04-05T21:36:54.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.415+0000 7f86e6f89700 1 -- 172.21.15.121:0/2868357958 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e01091c0 con 0x7f86e0100dc0 2024-04-05T21:36:54.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.415+0000 7f86ddffb700 1 -- 172.21.15.121:0/2868357958 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f86d400f5c0 con 0x7f86e0100dc0 2024-04-05T21:36:54.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.416+0000 7f86ddffb700 1 --2- 172.21.15.121:0/2868357958 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f86d006cce0 0x7f86d006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:54.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.416+0000 7f86dffff700 1 --2- 172.21.15.121:0/2868357958 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f86d006cce0 0x7f86d006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:54.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.416+0000 7f86ddffb700 1 -- 172.21.15.121:0/2868357958 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f86d408c480 con 0x7f86e0100dc0 2024-04-05T21:36:54.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.416+0000 7f86dffff700 1 --2- 172.21.15.121:0/2868357958 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f86d006cce0 0x7f86d006f1a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f86e0197c70 tx=0x7f86cc01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:54.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.420+0000 7f86ddffb700 1 -- 172.21.15.121:0/2868357958 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f86d4056570 con 0x7f86e0100dc0 2024-04-05T21:36:54.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.693+0000 7f86e6f89700 1 -- 172.21.15.121:0/2868357958 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f86e019bbf0 con 0x7f86e0100dc0 2024-04-05T21:36:54.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.693+0000 7f86ddffb700 1 -- 172.21.15.121:0/2868357958 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f86d4059b90 con 0x7f86e0100dc0 2024-04-05T21:36:54.694 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:36:54.694 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:36:54.695 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:36:54.696 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:36:54.696 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:36:54.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.695+0000 7f86e6f89700 1 -- 172.21.15.121:0/2868357958 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f86d006cce0 msgr2=0x7f86d006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:54.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.695+0000 7f86e6f89700 1 --2- 172.21.15.121:0/2868357958 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f86d006cce0 0x7f86d006f1a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f86e0197c70 tx=0x7f86cc01a040 comp rx=0 tx=0).stop 2024-04-05T21:36:54.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.695+0000 7f86e6f89700 1 -- 172.21.15.121:0/2868357958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e0100dc0 msgr2=0x7f86e0196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.695+0000 7f86e6f89700 1 --2- 172.21.15.121:0/2868357958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e0100dc0 0x7f86e0196650 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f86d400c8a0 tx=0x7f86d400cbb0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.695+0000 7f86e6f89700 1 -- 172.21.15.121:0/2868357958 shutdown_connections 2024-04-05T21:36:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.695+0000 7f86e6f89700 1 --2- 172.21.15.121:0/2868357958 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f86d006cce0 0x7f86d006f1a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.695+0000 7f86e6f89700 1 --2- 172.21.15.121:0/2868357958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86e0100dc0 0x7f86e0196650 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.695+0000 7f86e6f89700 1 --2- 172.21.15.121:0/2868357958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f86e0101770 0x7f86e0196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.696+0000 7f86e6f89700 1 -- 172.21.15.121:0/2868357958 >> 172.21.15.121:0/2868357958 conn(0x7f86e00fc670 msgr2=0x7f86e00fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.696+0000 7f86e6f89700 1 -- 172.21.15.121:0/2868357958 shutdown_connections 2024-04-05T21:36:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.696+0000 7f86e6f89700 1 -- 172.21.15.121:0/2868357958 wait complete. 2024-04-05T21:36:54.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:54 smithi121 ceph-mon[145859]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:54.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:54 smithi188 ceph-mon[107793]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:54.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.799+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3163391047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dd8105a40 msgr2=0x7f0dd8105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:54.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.799+0000 7f0ddfe26700 1 --2- 172.21.15.121:0/3163391047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dd8105a40 0x7f0dd8105e20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8009910 tx=0x7f0dc8009c20 comp rx=0 tx=0).stop 2024-04-05T21:36:54.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.799+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3163391047 shutdown_connections 2024-04-05T21:36:54.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.799+0000 7f0ddfe26700 1 --2- 172.21.15.121:0/3163391047 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dd80ffa40 0x7f0dd80ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.799+0000 7f0ddfe26700 1 --2- 172.21.15.121:0/3163391047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dd8105a40 0x7f0dd8105e20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.799+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3163391047 >> 172.21.15.121:0/3163391047 conn(0x7f0dd80fb570 msgr2=0x7f0dd80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:54.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.799+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3163391047 shutdown_connections 2024-04-05T21:36:54.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.799+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3163391047 wait complete. 2024-04-05T21:36:54.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.800+0000 7f0ddfe26700 1 Processor -- start 2024-04-05T21:36:54.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.800+0000 7f0ddfe26700 1 -- start start 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.800+0000 7f0ddfe26700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dd80ffa40 0x7f0dd80fbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.800+0000 7f0ddfe26700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dd8105a40 0x7f0dd80fc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.800+0000 7f0ddfe26700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd80fc990 con 0x7f0dd8105a40 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.800+0000 7f0ddfe26700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd80fcad0 con 0x7f0dd80ffa40 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.800+0000 7f0ddd3c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dd8105a40 0x7f0dd80fc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.800+0000 7f0ddd3c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dd8105a40 0x7f0dd80fc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33676/0 (socket says 172.21.15.121:33676) 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.800+0000 7f0ddd3c1700 1 -- 172.21.15.121:0/3374497219 learned_addr learned my addr 172.21.15.121:0/3374497219 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.800+0000 7f0dddbc2700 1 --2- 172.21.15.121:0/3374497219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dd80ffa40 0x7f0dd80fbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.801+0000 7f0ddd3c1700 1 -- 172.21.15.121:0/3374497219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dd80ffa40 msgr2=0x7f0dd80fbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.801+0000 7f0ddd3c1700 1 --2- 172.21.15.121:0/3374497219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dd80ffa40 0x7f0dd80fbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.801+0000 7f0ddd3c1700 1 -- 172.21.15.121:0/3374497219 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc80095f0 con 0x7f0dd8105a40 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.801+0000 7f0ddd3c1700 1 --2- 172.21.15.121:0/3374497219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dd8105a40 0x7f0dd80fc450 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400d8d0 tx=0x7f0dd400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.801+0000 7f0dceffd700 1 -- 172.21.15.121:0/3374497219 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dd40098a0 con 0x7f0dd8105a40 2024-04-05T21:36:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.801+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3374497219 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd81a2840 con 0x7f0dd8105a40 2024-04-05T21:36:54.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.801+0000 7f0dceffd700 1 -- 172.21.15.121:0/3374497219 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0dd4010460 con 0x7f0dd8105a40 2024-04-05T21:36:54.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.801+0000 7f0dceffd700 1 -- 172.21.15.121:0/3374497219 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dd4009ea0 con 0x7f0dd8105a40 2024-04-05T21:36:54.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.801+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3374497219 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd81a2ba0 con 0x7f0dd8105a40 2024-04-05T21:36:54.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.802+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3374497219 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd8109050 con 0x7f0dd8105a40 2024-04-05T21:36:54.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.802+0000 7f0dceffd700 1 -- 172.21.15.121:0/3374497219 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0dd40105d0 con 0x7f0dd8105a40 2024-04-05T21:36:54.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.803+0000 7f0dceffd700 1 --2- 172.21.15.121:0/3374497219 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0dc406cce0 0x7f0dc406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:54.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.803+0000 7f0dddbc2700 1 --2- 172.21.15.121:0/3374497219 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0dc406cce0 0x7f0dc406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:54.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.803+0000 7f0dceffd700 1 -- 172.21.15.121:0/3374497219 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0dd408c480 con 0x7f0dd8105a40 2024-04-05T21:36:54.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.803+0000 7f0dddbc2700 1 --2- 172.21.15.121:0/3374497219 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0dc406cce0 0x7f0dc406f1a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8006010 tx=0x7f0dc801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:54.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:54.808+0000 7f0dceffd700 1 -- 172.21.15.121:0/3374497219 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0dd40564f0 con 0x7f0dd8105a40 2024-04-05T21:36:55.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.042+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3374497219 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0dd802cc70 con 0x7f0dd8105a40 2024-04-05T21:36:55.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.043+0000 7f0dceffd700 1 -- 172.21.15.121:0/3374497219 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f0dd4010880 con 0x7f0dd8105a40 2024-04-05T21:36:55.044 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:36:55.044 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:36:55.044 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:36:55.044 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:36:55.044 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:36:55.044 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:36:55.044 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:36:55.044 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:36:55.045 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:36:55.046 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:36:55.047 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:36:55.047 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:36:55.047 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:36:55.047 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:36:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.045+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3374497219 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0dc406cce0 msgr2=0x7f0dc406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.045+0000 7f0ddfe26700 1 --2- 172.21.15.121:0/3374497219 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0dc406cce0 0x7f0dc406f1a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8006010 tx=0x7f0dc801a040 comp rx=0 tx=0).stop 2024-04-05T21:36:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.045+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3374497219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dd8105a40 msgr2=0x7f0dd80fc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.045+0000 7f0ddfe26700 1 --2- 172.21.15.121:0/3374497219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dd8105a40 0x7f0dd80fc450 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400d8d0 tx=0x7f0dd400dbe0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.045+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3374497219 shutdown_connections 2024-04-05T21:36:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.045+0000 7f0ddfe26700 1 --2- 172.21.15.121:0/3374497219 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0dc406cce0 0x7f0dc406f1a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.045+0000 7f0ddfe26700 1 --2- 172.21.15.121:0/3374497219 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0dd80ffa40 0x7f0dd80fbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.045+0000 7f0ddfe26700 1 --2- 172.21.15.121:0/3374497219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dd8105a40 0x7f0dd80fc450 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.045+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3374497219 >> 172.21.15.121:0/3374497219 conn(0x7f0dd80fb570 msgr2=0x7f0dd8068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.045+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3374497219 shutdown_connections 2024-04-05T21:36:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.045+0000 7f0ddfe26700 1 -- 172.21.15.121:0/3374497219 wait complete. 2024-04-05T21:36:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:36:55.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.149+0000 7fc3a98c5700 1 -- 172.21.15.121:0/303510957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a41003e0 msgr2=0x7fc3a4100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:55.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.149+0000 7fc3a98c5700 1 --2- 172.21.15.121:0/303510957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a41003e0 0x7fc3a4100860 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc394009910 tx=0x7fc394009c20 comp rx=0 tx=0).stop 2024-04-05T21:36:55.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.149+0000 7fc3a98c5700 1 -- 172.21.15.121:0/303510957 shutdown_connections 2024-04-05T21:36:55.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.149+0000 7fc3a98c5700 1 --2- 172.21.15.121:0/303510957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a41003e0 0x7fc3a4100860 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.149+0000 7fc3a98c5700 1 --2- 172.21.15.121:0/303510957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3a4104ae0 0x7fc3a4104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.149+0000 7fc3a98c5700 1 -- 172.21.15.121:0/303510957 >> 172.21.15.121:0/303510957 conn(0x7fc3a4074df0 msgr2=0x7fc3a4075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:55.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.149+0000 7fc3a98c5700 1 -- 172.21.15.121:0/303510957 shutdown_connections 2024-04-05T21:36:55.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.149+0000 7fc3a98c5700 1 -- 172.21.15.121:0/303510957 wait complete. 2024-04-05T21:36:55.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.150+0000 7fc3a98c5700 1 Processor -- start 2024-04-05T21:36:55.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.150+0000 7fc3a98c5700 1 -- start start 2024-04-05T21:36:55.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.150+0000 7fc3a98c5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3a41003e0 0x7fc3a419c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:55.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.150+0000 7fc3a98c5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a4104ae0 0x7fc3a419ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:55.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.150+0000 7fc3a98c5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a419d0f0 con 0x7fc3a4104ae0 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.150+0000 7fc3a98c5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a41965a0 con 0x7fc3a41003e0 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.150+0000 7fc3a27fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a4104ae0 0x7fc3a419ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.150+0000 7fc3a27fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a4104ae0 0x7fc3a419ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33690/0 (socket says 172.21.15.121:33690) 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.150+0000 7fc3a27fc700 1 -- 172.21.15.121:0/1975848055 learned_addr learned my addr 172.21.15.121:0/1975848055 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.150+0000 7fc3a2ffd700 1 --2- 172.21.15.121:0/1975848055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3a41003e0 0x7fc3a419c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.150+0000 7fc3a27fc700 1 -- 172.21.15.121:0/1975848055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3a41003e0 msgr2=0x7fc3a419c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.150+0000 7fc3a27fc700 1 --2- 172.21.15.121:0/1975848055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3a41003e0 0x7fc3a419c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.150+0000 7fc3a27fc700 1 -- 172.21.15.121:0/1975848055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3940095f0 con 0x7fc3a4104ae0 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.151+0000 7fc3a27fc700 1 --2- 172.21.15.121:0/1975848055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a4104ae0 0x7fc3a419ca60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc394000c00 tx=0x7fc39400be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.151+0000 7fc3a88c3700 1 -- 172.21.15.121:0/1975848055 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc39401d070 con 0x7fc3a4104ae0 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.151+0000 7fc3a98c5700 1 -- 172.21.15.121:0/1975848055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a4196820 con 0x7fc3a4104ae0 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.151+0000 7fc3a88c3700 1 -- 172.21.15.121:0/1975848055 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3940044c0 con 0x7fc3a4104ae0 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.151+0000 7fc3a88c3700 1 -- 172.21.15.121:0/1975848055 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3940039d0 con 0x7fc3a4104ae0 2024-04-05T21:36:55.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.151+0000 7fc3a98c5700 1 -- 172.21.15.121:0/1975848055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a4196cb0 con 0x7fc3a4104ae0 2024-04-05T21:36:55.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.152+0000 7fc3a98c5700 1 -- 172.21.15.121:0/1975848055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a4109070 con 0x7fc3a4104ae0 2024-04-05T21:36:55.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.152+0000 7fc3a88c3700 1 -- 172.21.15.121:0/1975848055 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc3940048e0 con 0x7fc3a4104ae0 2024-04-05T21:36:55.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.152+0000 7fc3a88c3700 1 --2- 172.21.15.121:0/1975848055 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc38c06cce0 0x7fc38c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:55.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.153+0000 7fc3a88c3700 1 -- 172.21.15.121:0/1975848055 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc39408dbb0 con 0x7fc3a4104ae0 2024-04-05T21:36:55.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.153+0000 7fc3a2ffd700 1 --2- 172.21.15.121:0/1975848055 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc38c06cce0 0x7fc38c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:55.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.153+0000 7fc3a2ffd700 1 --2- 172.21.15.121:0/1975848055 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc38c06cce0 0x7fc38c06f1a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc398009bf0 tx=0x7fc398009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:55.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.157+0000 7fc3a88c3700 1 -- 172.21.15.121:0/1975848055 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc394094080 con 0x7fc3a4104ae0 2024-04-05T21:36:55.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.368+0000 7fc3a98c5700 1 -- 172.21.15.121:0/1975848055 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3a4196f80 con 0x7fc38c06cce0 2024-04-05T21:36:55.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.370+0000 7fc3a88c3700 1 -- 172.21.15.121:0/1975848055 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc3a4196f80 con 0x7fc38c06cce0 2024-04-05T21:36:55.370 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:36:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:36:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:36:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:36:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:36:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:36:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:36:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:36:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:36:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:36:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:36:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:36:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:36:55.371 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:36:55.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.372+0000 7fc3a98c5700 1 -- 172.21.15.121:0/1975848055 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc38c06cce0 msgr2=0x7fc38c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:55.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.372+0000 7fc3a98c5700 1 --2- 172.21.15.121:0/1975848055 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc38c06cce0 0x7fc38c06f1a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc398009bf0 tx=0x7fc398009260 comp rx=0 tx=0).stop 2024-04-05T21:36:55.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.372+0000 7fc3a98c5700 1 -- 172.21.15.121:0/1975848055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a4104ae0 msgr2=0x7fc3a419ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:55.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.372+0000 7fc3a98c5700 1 --2- 172.21.15.121:0/1975848055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a4104ae0 0x7fc3a419ca60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc394000c00 tx=0x7fc39400be90 comp rx=0 tx=0).stop 2024-04-05T21:36:55.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.372+0000 7fc3a98c5700 1 -- 172.21.15.121:0/1975848055 shutdown_connections 2024-04-05T21:36:55.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.372+0000 7fc3a98c5700 1 --2- 172.21.15.121:0/1975848055 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc38c06cce0 0x7fc38c06f1a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.372+0000 7fc3a98c5700 1 --2- 172.21.15.121:0/1975848055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3a41003e0 0x7fc3a419c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.372+0000 7fc3a98c5700 1 --2- 172.21.15.121:0/1975848055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a4104ae0 0x7fc3a419ca60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.372+0000 7fc3a98c5700 1 -- 172.21.15.121:0/1975848055 >> 172.21.15.121:0/1975848055 conn(0x7fc3a4074df0 msgr2=0x7fc3a40ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:55.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.372+0000 7fc3a98c5700 1 -- 172.21.15.121:0/1975848055 shutdown_connections 2024-04-05T21:36:55.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.372+0000 7fc3a98c5700 1 -- 172.21.15.121:0/1975848055 wait complete. 2024-04-05T21:36:55.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.475+0000 7fb275b2b700 1 -- 172.21.15.121:0/2495513852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb270101790 msgr2=0x7fb2701057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:55.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.475+0000 7fb275b2b700 1 --2- 172.21.15.121:0/2495513852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb270101790 0x7fb2701057e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb258009910 tx=0x7fb258009c20 comp rx=0 tx=0).stop 2024-04-05T21:36:55.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.475+0000 7fb275b2b700 1 -- 172.21.15.121:0/2495513852 shutdown_connections 2024-04-05T21:36:55.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.475+0000 7fb275b2b700 1 --2- 172.21.15.121:0/2495513852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb270101790 0x7fb2701057e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.475+0000 7fb275b2b700 1 --2- 172.21.15.121:0/2495513852 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb270100de0 0x7fb2701011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.475+0000 7fb275b2b700 1 -- 172.21.15.121:0/2495513852 >> 172.21.15.121:0/2495513852 conn(0x7fb2700fc6b0 msgr2=0x7fb2700fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:55.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.475+0000 7fb275b2b700 1 -- 172.21.15.121:0/2495513852 shutdown_connections 2024-04-05T21:36:55.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.475+0000 7fb275b2b700 1 -- 172.21.15.121:0/2495513852 wait complete. 2024-04-05T21:36:55.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.476+0000 7fb275b2b700 1 Processor -- start 2024-04-05T21:36:55.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.476+0000 7fb275b2b700 1 -- start start 2024-04-05T21:36:55.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.476+0000 7fb275b2b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb270100de0 0x7fb270196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:55.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.476+0000 7fb275b2b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb270101790 0x7fb270196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:55.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.476+0000 7fb275b2b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb270197290 con 0x7fb270100de0 2024-04-05T21:36:55.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.476+0000 7fb275b2b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27019b020 con 0x7fb270101790 2024-04-05T21:36:55.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.476+0000 7fb26f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb270100de0 0x7fb270196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:55.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.476+0000 7fb26f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb270100de0 0x7fb270196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33704/0 (socket says 172.21.15.121:33704) 2024-04-05T21:36:55.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.476+0000 7fb26f7fe700 1 -- 172.21.15.121:0/4066026669 learned_addr learned my addr 172.21.15.121:0/4066026669 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:36:55.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.477+0000 7fb26f7fe700 1 -- 172.21.15.121:0/4066026669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb270101790 msgr2=0x7fb270196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:55.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.477+0000 7fb26f7fe700 1 --2- 172.21.15.121:0/4066026669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb270101790 0x7fb270196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.477+0000 7fb26f7fe700 1 -- 172.21.15.121:0/4066026669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2580095f0 con 0x7fb270100de0 2024-04-05T21:36:55.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.477+0000 7fb26f7fe700 1 --2- 172.21.15.121:0/4066026669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb270100de0 0x7fb270196670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb26000d8d0 tx=0x7fb26000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:55.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.477+0000 7fb26d7fa700 1 -- 172.21.15.121:0/4066026669 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb260009940 con 0x7fb270100de0 2024-04-05T21:36:55.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.477+0000 7fb275b2b700 1 -- 172.21.15.121:0/4066026669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27019b300 con 0x7fb270100de0 2024-04-05T21:36:55.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.477+0000 7fb26d7fa700 1 -- 172.21.15.121:0/4066026669 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb260010460 con 0x7fb270100de0 2024-04-05T21:36:55.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.477+0000 7fb26d7fa700 1 -- 172.21.15.121:0/4066026669 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb26000f5f0 con 0x7fb270100de0 2024-04-05T21:36:55.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.477+0000 7fb275b2b700 1 -- 172.21.15.121:0/4066026669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27019b850 con 0x7fb270100de0 2024-04-05T21:36:55.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.478+0000 7fb275b2b700 1 -- 172.21.15.121:0/4066026669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2701091e0 con 0x7fb270100de0 2024-04-05T21:36:55.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.479+0000 7fb26d7fa700 1 -- 172.21.15.121:0/4066026669 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb2600105d0 con 0x7fb270100de0 2024-04-05T21:36:55.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.479+0000 7fb26d7fa700 1 --2- 172.21.15.121:0/4066026669 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb25c06cce0 0x7fb25c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:36:55.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.480+0000 7fb26d7fa700 1 -- 172.21.15.121:0/4066026669 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb26008c5f0 con 0x7fb270100de0 2024-04-05T21:36:55.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.480+0000 7fb267fff700 1 --2- 172.21.15.121:0/4066026669 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb25c06cce0 0x7fb25c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:36:55.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.480+0000 7fb267fff700 1 --2- 172.21.15.121:0/4066026669 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb25c06cce0 0x7fb25c06f1a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb258006010 tx=0x7fb25800b540 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:36:55.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.483+0000 7fb26d7fa700 1 -- 172.21.15.121:0/4066026669 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb2600566f0 con 0x7fb270100de0 2024-04-05T21:36:55.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:55 smithi121 ceph-mon[145859]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:55.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:55 smithi121 ceph-mon[145859]: pgmap v320: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:55.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:55 smithi121 ceph-mon[145859]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:55.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:55 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2868357958' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:36:55.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:55 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3374497219' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:36:55.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.755+0000 7fb275b2b700 1 -- 172.21.15.121:0/4066026669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb2701979d0 con 0x7fb270100de0 2024-04-05T21:36:55.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.755+0000 7fb26d7fa700 1 -- 172.21.15.121:0/4066026669 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fb260010880 con 0x7fb270100de0 2024-04-05T21:36:55.756 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:36:55.756 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:36:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:36:55.756 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:36:55.756 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:36:55.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.757+0000 7fb275b2b700 1 -- 172.21.15.121:0/4066026669 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb25c06cce0 msgr2=0x7fb25c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:55.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.757+0000 7fb275b2b700 1 --2- 172.21.15.121:0/4066026669 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb25c06cce0 0x7fb25c06f1a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb258006010 tx=0x7fb25800b540 comp rx=0 tx=0).stop 2024-04-05T21:36:55.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.757+0000 7fb275b2b700 1 -- 172.21.15.121:0/4066026669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb270100de0 msgr2=0x7fb270196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:36:55.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.757+0000 7fb275b2b700 1 --2- 172.21.15.121:0/4066026669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb270100de0 0x7fb270196670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb26000d8d0 tx=0x7fb26000dc90 comp rx=0 tx=0).stop 2024-04-05T21:36:55.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.757+0000 7fb275b2b700 1 -- 172.21.15.121:0/4066026669 shutdown_connections 2024-04-05T21:36:55.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.757+0000 7fb275b2b700 1 --2- 172.21.15.121:0/4066026669 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb25c06cce0 0x7fb25c06f1a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.757+0000 7fb275b2b700 1 --2- 172.21.15.121:0/4066026669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb270100de0 0x7fb270196670 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.757+0000 7fb275b2b700 1 --2- 172.21.15.121:0/4066026669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb270101790 0x7fb270196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:36:55.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.757+0000 7fb275b2b700 1 -- 172.21.15.121:0/4066026669 >> 172.21.15.121:0/4066026669 conn(0x7fb2700fc6b0 msgr2=0x7fb2700fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:36:55.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.757+0000 7fb275b2b700 1 -- 172.21.15.121:0/4066026669 shutdown_connections 2024-04-05T21:36:55.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:36:55.757+0000 7fb275b2b700 1 -- 172.21.15.121:0/4066026669 wait complete. 2024-04-05T21:36:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:55 smithi188 ceph-mon[107793]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:55 smithi188 ceph-mon[107793]: pgmap v320: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:55.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:55 smithi188 ceph-mon[107793]: from='client.34462 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:55.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:55 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2868357958' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:36:55.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:55 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3374497219' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:36:56.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:56 smithi121 ceph-mon[145859]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:56.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:56 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4066026669' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:36:56.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:56 smithi188 ceph-mon[107793]: from='client.34474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:36:56.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:56 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4066026669' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:36:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:57 smithi121 ceph-mon[145859]: pgmap v321: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:57 smithi188 ceph-mon[107793]: pgmap v321: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:59.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:36:59 smithi121 ceph-mon[145859]: pgmap v322: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:36:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:36:59 smithi188 ceph-mon[107793]: pgmap v322: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:37:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:37:01.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:01 smithi121 ceph-mon[145859]: pgmap v323: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:01 smithi188 ceph-mon[107793]: pgmap v323: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:03.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:03 smithi121 ceph-mon[145859]: pgmap v324: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:03.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:03 smithi188 ceph-mon[107793]: pgmap v324: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:05.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:05 smithi121 ceph-mon[145859]: pgmap v325: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:05 smithi188 ceph-mon[107793]: pgmap v325: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:07 smithi121 ceph-mon[145859]: pgmap v326: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:07 smithi188 ceph-mon[107793]: pgmap v326: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:09 smithi121 ceph-mon[145859]: pgmap v327: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:09 smithi188 ceph-mon[107793]: pgmap v327: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:11 smithi121 ceph-mon[145859]: pgmap v328: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:11.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:11 smithi188 ceph-mon[107793]: pgmap v328: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:13 smithi121 ceph-mon[145859]: pgmap v329: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:13.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:13 smithi188 ceph-mon[107793]: pgmap v329: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:15 smithi121 ceph-mon[145859]: pgmap v330: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:37:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:15 smithi188 ceph-mon[107793]: pgmap v330: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:37:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:17 smithi121 ceph-mon[145859]: pgmap v331: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:17.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:17 smithi188 ceph-mon[107793]: pgmap v331: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:19 smithi121 ceph-mon[145859]: pgmap v332: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:19 smithi188 ceph-mon[107793]: pgmap v332: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:21 smithi121 ceph-mon[145859]: pgmap v333: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:21 smithi188 ceph-mon[107793]: pgmap v333: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:23 smithi121 ceph-mon[145859]: pgmap v334: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:23 smithi188 ceph-mon[107793]: pgmap v334: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:25.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:25 smithi121 ceph-mon[145859]: pgmap v335: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:25.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:25 smithi188 ceph-mon[107793]: pgmap v335: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:25.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.873+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1722887800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc0ffa40 msgr2=0x7f2dcc0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:25.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.873+0000 7f2dd22a1700 1 --2- 172.21.15.121:0/1722887800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc0ffa40 0x7f2dcc0ffec0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009910 tx=0x7f2dbc009c20 comp rx=0 tx=0).stop 2024-04-05T21:37:25.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.873+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1722887800 shutdown_connections 2024-04-05T21:37:25.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.873+0000 7f2dd22a1700 1 --2- 172.21.15.121:0/1722887800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc0ffa40 0x7f2dcc0ffec0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:25.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.873+0000 7f2dd22a1700 1 --2- 172.21.15.121:0/1722887800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2dcc105a40 0x7f2dcc105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:25.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.873+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1722887800 >> 172.21.15.121:0/1722887800 conn(0x7f2dcc0fb570 msgr2=0x7f2dcc0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:25.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.873+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1722887800 shutdown_connections 2024-04-05T21:37:25.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.873+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1722887800 wait complete. 2024-04-05T21:37:25.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.873+0000 7f2dd22a1700 1 Processor -- start 2024-04-05T21:37:25.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.873+0000 7f2dd22a1700 1 -- start start 2024-04-05T21:37:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.874+0000 7f2dd22a1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2dcc0ffa40 0x7f2dcc19e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.874+0000 7f2dd22a1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc105a40 0x7f2dcc19ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.874+0000 7f2dd22a1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc19f370 con 0x7f2dcc105a40 2024-04-05T21:37:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.874+0000 7f2dd22a1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc198820 con 0x7f2dcc0ffa40 2024-04-05T21:37:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.874+0000 7f2dcb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc105a40 0x7f2dcc19ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.874+0000 7f2dcb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc105a40 0x7f2dcc19ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33892/0 (socket says 172.21.15.121:33892) 2024-04-05T21:37:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.874+0000 7f2dcb7fe700 1 -- 172.21.15.121:0/1436320276 learned_addr learned my addr 172.21.15.121:0/1436320276 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:37:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.874+0000 7f2dcbfff700 1 --2- 172.21.15.121:0/1436320276 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2dcc0ffa40 0x7f2dcc19e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.874+0000 7f2dcb7fe700 1 -- 172.21.15.121:0/1436320276 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2dcc0ffa40 msgr2=0x7f2dcc19e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:25.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.874+0000 7f2dcb7fe700 1 --2- 172.21.15.121:0/1436320276 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2dcc0ffa40 0x7f2dcc19e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:25.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.874+0000 7f2dcb7fe700 1 -- 172.21.15.121:0/1436320276 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc0095f0 con 0x7f2dcc105a40 2024-04-05T21:37:25.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.874+0000 7f2dcbfff700 1 --2- 172.21.15.121:0/1436320276 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2dcc0ffa40 0x7f2dcc19e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:37:25.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.874+0000 7f2dcb7fe700 1 --2- 172.21.15.121:0/1436320276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc105a40 0x7f2dcc19ece0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc006010 tx=0x7f2dbc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:25.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.875+0000 7f2dc97fa700 1 -- 172.21.15.121:0/1436320276 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dbc01d070 con 0x7f2dcc105a40 2024-04-05T21:37:25.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.875+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1436320276 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dcc198aa0 con 0x7f2dcc105a40 2024-04-05T21:37:25.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.875+0000 7f2dc97fa700 1 -- 172.21.15.121:0/1436320276 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2dbc00f460 con 0x7f2dcc105a40 2024-04-05T21:37:25.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.875+0000 7f2dc97fa700 1 -- 172.21.15.121:0/1436320276 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2dbc021640 con 0x7f2dcc105a40 2024-04-05T21:37:25.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.875+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1436320276 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dcc198f90 con 0x7f2dcc105a40 2024-04-05T21:37:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.876+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1436320276 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dcc04ea90 con 0x7f2dcc105a40 2024-04-05T21:37:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.876+0000 7f2dc97fa700 1 -- 172.21.15.121:0/1436320276 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2dbc00fa20 con 0x7f2dcc105a40 2024-04-05T21:37:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.877+0000 7f2dc97fa700 1 --2- 172.21.15.121:0/1436320276 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2db806cce0 0x7f2db806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.877+0000 7f2dc97fa700 1 -- 172.21.15.121:0/1436320276 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2dbc08d8c0 con 0x7f2dcc105a40 2024-04-05T21:37:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.877+0000 7f2dcbfff700 1 --2- 172.21.15.121:0/1436320276 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2db806cce0 0x7f2db806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.877+0000 7f2dcbfff700 1 --2- 172.21.15.121:0/1436320276 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2db806cce0 0x7f2db806f1a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2db4007900 tx=0x7f2db4008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:25.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:25.881+0000 7f2dc97fa700 1 -- 172.21.15.121:0/1436320276 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2dbc05b3c0 con 0x7f2dcc105a40 2024-04-05T21:37:26.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.093+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1436320276 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2dcc199430 con 0x7f2db806cce0 2024-04-05T21:37:26.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.094+0000 7f2dc97fa700 1 -- 172.21.15.121:0/1436320276 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2dcc199430 con 0x7f2db806cce0 2024-04-05T21:37:26.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.096+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1436320276 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2db806cce0 msgr2=0x7f2db806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:26.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.096+0000 7f2dd22a1700 1 --2- 172.21.15.121:0/1436320276 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2db806cce0 0x7f2db806f1a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2db4007900 tx=0x7f2db4008040 comp rx=0 tx=0).stop 2024-04-05T21:37:26.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.096+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1436320276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc105a40 msgr2=0x7f2dcc19ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:26.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.096+0000 7f2dd22a1700 1 --2- 172.21.15.121:0/1436320276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc105a40 0x7f2dcc19ece0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc006010 tx=0x7f2dbc00be20 comp rx=0 tx=0).stop 2024-04-05T21:37:26.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.096+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1436320276 shutdown_connections 2024-04-05T21:37:26.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.096+0000 7f2dd22a1700 1 --2- 172.21.15.121:0/1436320276 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2db806cce0 0x7f2db806f1a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.096+0000 7f2dd22a1700 1 --2- 172.21.15.121:0/1436320276 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2dcc0ffa40 0x7f2dcc19e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.096+0000 7f2dd22a1700 1 --2- 172.21.15.121:0/1436320276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2dcc105a40 0x7f2dcc19ece0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.096+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1436320276 >> 172.21.15.121:0/1436320276 conn(0x7f2dcc0fb570 msgr2=0x7f2dcc0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:26.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.096+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1436320276 shutdown_connections 2024-04-05T21:37:26.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.096+0000 7f2dd22a1700 1 -- 172.21.15.121:0/1436320276 wait complete. 2024-04-05T21:37:26.107 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:37:26.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.200+0000 7f4659773700 1 -- 172.21.15.121:0/3833730118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46540ffbd0 msgr2=0x7f4654100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:26.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.200+0000 7f4659773700 1 --2- 172.21.15.121:0/3833730118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46540ffbd0 0x7f4654100050 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4644009910 tx=0x7f4644009c20 comp rx=0 tx=0).stop 2024-04-05T21:37:26.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.200+0000 7f4659773700 1 -- 172.21.15.121:0/3833730118 shutdown_connections 2024-04-05T21:37:26.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.200+0000 7f4659773700 1 --2- 172.21.15.121:0/3833730118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46540ffbd0 0x7f4654100050 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.200+0000 7f4659773700 1 --2- 172.21.15.121:0/3833730118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4654105a40 0x7f4654105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.200+0000 7f4659773700 1 -- 172.21.15.121:0/3833730118 >> 172.21.15.121:0/3833730118 conn(0x7f46540755a0 msgr2=0x7f46540759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:26.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.200+0000 7f4659773700 1 -- 172.21.15.121:0/3833730118 shutdown_connections 2024-04-05T21:37:26.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.200+0000 7f4659773700 1 -- 172.21.15.121:0/3833730118 wait complete. 2024-04-05T21:37:26.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.200+0000 7f4659773700 1 Processor -- start 2024-04-05T21:37:26.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f4659773700 1 -- start start 2024-04-05T21:37:26.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f4659773700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46540ffbd0 0x7f46541974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:26.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f4659773700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4654105a40 0x7f46541979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:26.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f4659773700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4654197f80 con 0x7f46540ffbd0 2024-04-05T21:37:26.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f4659773700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46541980c0 con 0x7f4654105a40 2024-04-05T21:37:26.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f4652ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46540ffbd0 0x7f46541974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:26.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f4652ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46540ffbd0 0x7f46541974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33916/0 (socket says 172.21.15.121:33916) 2024-04-05T21:37:26.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f4652ffd700 1 -- 172.21.15.121:0/1521565637 learned_addr learned my addr 172.21.15.121:0/1521565637 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:37:26.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f46527fc700 1 --2- 172.21.15.121:0/1521565637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4654105a40 0x7f46541979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:26.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f4652ffd700 1 -- 172.21.15.121:0/1521565637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4654105a40 msgr2=0x7f46541979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:26.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f4652ffd700 1 --2- 172.21.15.121:0/1521565637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4654105a40 0x7f46541979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f4652ffd700 1 -- 172.21.15.121:0/1521565637 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46440095f0 con 0x7f46540ffbd0 2024-04-05T21:37:26.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f4652ffd700 1 --2- 172.21.15.121:0/1521565637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46540ffbd0 0x7f46541974b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f464800c930 tx=0x7f464800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:26.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f463bfff700 1 -- 172.21.15.121:0/1521565637 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4648007ad0 con 0x7f46540ffbd0 2024-04-05T21:37:26.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f463bfff700 1 -- 172.21.15.121:0/1521565637 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4648007c30 con 0x7f46540ffbd0 2024-04-05T21:37:26.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.201+0000 7f4659773700 1 -- 172.21.15.121:0/1521565637 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46541a0520 con 0x7f46540ffbd0 2024-04-05T21:37:26.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.202+0000 7f463bfff700 1 -- 172.21.15.121:0/1521565637 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4648018630 con 0x7f46540ffbd0 2024-04-05T21:37:26.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.202+0000 7f4659773700 1 -- 172.21.15.121:0/1521565637 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46541a0990 con 0x7f46540ffbd0 2024-04-05T21:37:26.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.202+0000 7f4659773700 1 -- 172.21.15.121:0/1521565637 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4654107fd0 con 0x7f46540ffbd0 2024-04-05T21:37:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.203+0000 7f463bfff700 1 -- 172.21.15.121:0/1521565637 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f464800f450 con 0x7f46540ffbd0 2024-04-05T21:37:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.204+0000 7f463bfff700 1 --2- 172.21.15.121:0/1521565637 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f463c06cce0 0x7f463c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.204+0000 7f463bfff700 1 -- 172.21.15.121:0/1521565637 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f464808c8c0 con 0x7f46540ffbd0 2024-04-05T21:37:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.204+0000 7f46527fc700 1 --2- 172.21.15.121:0/1521565637 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f463c06cce0 0x7f463c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:26.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.204+0000 7f46527fc700 1 --2- 172.21.15.121:0/1521565637 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f463c06cce0 0x7f463c06f1a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4654068cc0 tx=0x7f464401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:26.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.208+0000 7f463bfff700 1 -- 172.21.15.121:0/1521565637 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4648056a50 con 0x7f46540ffbd0 2024-04-05T21:37:26.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.416+0000 7f4659773700 1 -- 172.21.15.121:0/1521565637 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f465402cd80 con 0x7f463c06cce0 2024-04-05T21:37:26.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.417+0000 7f463bfff700 1 -- 172.21.15.121:0/1521565637 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f465402cd80 con 0x7f463c06cce0 2024-04-05T21:37:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.419+0000 7f4659773700 1 -- 172.21.15.121:0/1521565637 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f463c06cce0 msgr2=0x7f463c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.419+0000 7f4659773700 1 --2- 172.21.15.121:0/1521565637 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f463c06cce0 0x7f463c06f1a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4654068cc0 tx=0x7f464401a040 comp rx=0 tx=0).stop 2024-04-05T21:37:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.419+0000 7f4659773700 1 -- 172.21.15.121:0/1521565637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46540ffbd0 msgr2=0x7f46541974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.419+0000 7f4659773700 1 --2- 172.21.15.121:0/1521565637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46540ffbd0 0x7f46541974b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f464800c930 tx=0x7f464800ccf0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.419+0000 7f4659773700 1 -- 172.21.15.121:0/1521565637 shutdown_connections 2024-04-05T21:37:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.419+0000 7f4659773700 1 --2- 172.21.15.121:0/1521565637 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f463c06cce0 0x7f463c06f1a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.419+0000 7f4659773700 1 --2- 172.21.15.121:0/1521565637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46540ffbd0 0x7f46541974b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.419+0000 7f4659773700 1 --2- 172.21.15.121:0/1521565637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4654105a40 0x7f46541979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.419+0000 7f4659773700 1 -- 172.21.15.121:0/1521565637 >> 172.21.15.121:0/1521565637 conn(0x7f46540755a0 msgr2=0x7f46540fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.419+0000 7f4659773700 1 -- 172.21.15.121:0/1521565637 shutdown_connections 2024-04-05T21:37:26.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.419+0000 7f4659773700 1 -- 172.21.15.121:0/1521565637 wait complete. 2024-04-05T21:37:26.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.522+0000 7fc6c0894700 1 -- 172.21.15.121:0/1552888145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6bc105a40 msgr2=0x7fc6bc105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:26.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.522+0000 7fc6c0894700 1 --2- 172.21.15.121:0/1552888145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6bc105a40 0x7fc6bc105e20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0009910 tx=0x7fc6b0009c20 comp rx=0 tx=0).stop 2024-04-05T21:37:26.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.522+0000 7fc6c0894700 1 -- 172.21.15.121:0/1552888145 shutdown_connections 2024-04-05T21:37:26.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.522+0000 7fc6c0894700 1 --2- 172.21.15.121:0/1552888145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6bc0ffbd0 0x7fc6bc100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.522+0000 7fc6c0894700 1 --2- 172.21.15.121:0/1552888145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6bc105a40 0x7fc6bc105e20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.522+0000 7fc6c0894700 1 -- 172.21.15.121:0/1552888145 >> 172.21.15.121:0/1552888145 conn(0x7fc6bc0755a0 msgr2=0x7fc6bc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:26.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.522+0000 7fc6c0894700 1 -- 172.21.15.121:0/1552888145 shutdown_connections 2024-04-05T21:37:26.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.523+0000 7fc6c0894700 1 -- 172.21.15.121:0/1552888145 wait complete. 2024-04-05T21:37:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.523+0000 7fc6c0894700 1 Processor -- start 2024-04-05T21:37:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.523+0000 7fc6c0894700 1 -- start start 2024-04-05T21:37:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6c0894700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6bc0ffbd0 0x7fc6bc196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6c0894700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6bc105a40 0x7fc6bc196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6c0894700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc197260 con 0x7fc6bc0ffbd0 2024-04-05T21:37:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6c0894700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc19aff0 con 0x7fc6bc105a40 2024-04-05T21:37:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6ba59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6bc0ffbd0 0x7fc6bc196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6ba59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6bc0ffbd0 0x7fc6bc196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33938/0 (socket says 172.21.15.121:33938) 2024-04-05T21:37:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6ba59c700 1 -- 172.21.15.121:0/1292657482 learned_addr learned my addr 172.21.15.121:0/1292657482 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:37:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6b9d9b700 1 --2- 172.21.15.121:0/1292657482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6bc105a40 0x7fc6bc196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6ba59c700 1 -- 172.21.15.121:0/1292657482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6bc105a40 msgr2=0x7fc6bc196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:26.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6ba59c700 1 --2- 172.21.15.121:0/1292657482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6bc105a40 0x7fc6bc196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6ba59c700 1 -- 172.21.15.121:0/1292657482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b00095f0 con 0x7fc6bc0ffbd0 2024-04-05T21:37:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6ba59c700 1 --2- 172.21.15.121:0/1292657482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6bc0ffbd0 0x7fc6bc196640 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc6b00092e0 tx=0x7fc6b000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6ab7fe700 1 -- 172.21.15.121:0/1292657482 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6b001d070 con 0x7fc6bc0ffbd0 2024-04-05T21:37:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6ab7fe700 1 -- 172.21.15.121:0/1292657482 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6b0005e00 con 0x7fc6bc0ffbd0 2024-04-05T21:37:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6c0894700 1 -- 172.21.15.121:0/1292657482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6bc19b270 con 0x7fc6bc0ffbd0 2024-04-05T21:37:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6ab7fe700 1 -- 172.21.15.121:0/1292657482 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6b000fe90 con 0x7fc6bc0ffbd0 2024-04-05T21:37:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.524+0000 7fc6c0894700 1 -- 172.21.15.121:0/1292657482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6bc19b760 con 0x7fc6bc0ffbd0 2024-04-05T21:37:26.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.525+0000 7fc6c0894700 1 -- 172.21.15.121:0/1292657482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6bc109050 con 0x7fc6bc0ffbd0 2024-04-05T21:37:26.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.526+0000 7fc6ab7fe700 1 -- 172.21.15.121:0/1292657482 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc6b000f6d0 con 0x7fc6bc0ffbd0 2024-04-05T21:37:26.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.526+0000 7fc6ab7fe700 1 --2- 172.21.15.121:0/1292657482 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc6a406cce0 0x7fc6a406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:26.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.526+0000 7fc6b9d9b700 1 --2- 172.21.15.121:0/1292657482 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc6a406cce0 0x7fc6a406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:26.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.526+0000 7fc6ab7fe700 1 -- 172.21.15.121:0/1292657482 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc6b008d7e0 con 0x7fc6bc0ffbd0 2024-04-05T21:37:26.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.527+0000 7fc6b9d9b700 1 --2- 172.21.15.121:0/1292657482 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc6a406cce0 0x7fc6a406f1a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc197c60 tx=0x7fc6ac00b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:26.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.531+0000 7fc6ab7fe700 1 -- 172.21.15.121:0/1292657482 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc6b0057850 con 0x7fc6bc0ffbd0 2024-04-05T21:37:26.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.730+0000 7fc6c0894700 1 -- 172.21.15.121:0/1292657482 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6bc02cd80 con 0x7fc6a406cce0 2024-04-05T21:37:26.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.741+0000 7fc6ab7fe700 1 -- 172.21.15.121:0/1292657482 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc6bc02cd80 con 0x7fc6a406cce0 2024-04-05T21:37:26.741 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:37:26.741 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (8m) 2m ago 19m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:37:26.742 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (20m) 2m ago 19m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:37:26.742 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (18m) 116s ago 18m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:37:26.742 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (6m) 2m ago 19m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:37:26.742 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (6m) 116s ago 18m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:37:26.742 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (8m) 2m ago 19m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:37:26.742 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (2m) 2m ago 13m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:37:26.742 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (2m) 2m ago 13m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:37:26.742 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (2m) 116s ago 13m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:37:26.742 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (13m) 116s ago 13m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:37:26.742 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (10m) 2m ago 21m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:37:26.742 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (9m) 116s ago 18m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:37:26.742 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (7m) 2m ago 21m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:37:26.743 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (6m) 116s ago 18m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:37:26.743 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (9m) 2m ago 19m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:37:26.743 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (9m) 116s ago 18m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:37:26.743 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (5m) 2m ago 17m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:37:26.743 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (5m) 2m ago 17m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:37:26.743 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (4m) 2m ago 16m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:37:26.743 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (4m) 116s ago 16m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:37:26.743 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (3m) 116s ago 15m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:37:26.743 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (2m) 116s ago 15m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:37:26.743 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (8m) 2m ago 19m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:37:26.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.743+0000 7fc6c0894700 1 -- 172.21.15.121:0/1292657482 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc6a406cce0 msgr2=0x7fc6a406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:26.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.743+0000 7fc6c0894700 1 --2- 172.21.15.121:0/1292657482 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc6a406cce0 0x7fc6a406f1a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc197c60 tx=0x7fc6ac00b410 comp rx=0 tx=0).stop 2024-04-05T21:37:26.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.743+0000 7fc6c0894700 1 -- 172.21.15.121:0/1292657482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6bc0ffbd0 msgr2=0x7fc6bc196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:26.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.743+0000 7fc6c0894700 1 --2- 172.21.15.121:0/1292657482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6bc0ffbd0 0x7fc6bc196640 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc6b00092e0 tx=0x7fc6b000bd70 comp rx=0 tx=0).stop 2024-04-05T21:37:26.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.743+0000 7fc6c0894700 1 -- 172.21.15.121:0/1292657482 shutdown_connections 2024-04-05T21:37:26.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.743+0000 7fc6c0894700 1 --2- 172.21.15.121:0/1292657482 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc6a406cce0 0x7fc6a406f1a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.743+0000 7fc6c0894700 1 --2- 172.21.15.121:0/1292657482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6bc0ffbd0 0x7fc6bc196640 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.743+0000 7fc6c0894700 1 --2- 172.21.15.121:0/1292657482 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6bc105a40 0x7fc6bc196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.743+0000 7fc6c0894700 1 -- 172.21.15.121:0/1292657482 >> 172.21.15.121:0/1292657482 conn(0x7fc6bc0755a0 msgr2=0x7fc6bc068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:26.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.743+0000 7fc6c0894700 1 -- 172.21.15.121:0/1292657482 shutdown_connections 2024-04-05T21:37:26.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.743+0000 7fc6c0894700 1 -- 172.21.15.121:0/1292657482 wait complete. 2024-04-05T21:37:26.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.844+0000 7f2656992700 1 -- 172.21.15.121:0/3041135226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2650101750 msgr2=0x7f26501057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:26.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.844+0000 7f2656992700 1 --2- 172.21.15.121:0/3041135226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2650101750 0x7f26501057a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2644009910 tx=0x7f2644009c20 comp rx=0 tx=0).stop 2024-04-05T21:37:26.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.844+0000 7f2656992700 1 -- 172.21.15.121:0/3041135226 shutdown_connections 2024-04-05T21:37:26.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.844+0000 7f2656992700 1 --2- 172.21.15.121:0/3041135226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2650101750 0x7f26501057a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.844+0000 7f2656992700 1 --2- 172.21.15.121:0/3041135226 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2650100da0 0x7f2650101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.844+0000 7f2656992700 1 -- 172.21.15.121:0/3041135226 >> 172.21.15.121:0/3041135226 conn(0x7f26500fc630 msgr2=0x7f26500fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:26.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.844+0000 7f2656992700 1 -- 172.21.15.121:0/3041135226 shutdown_connections 2024-04-05T21:37:26.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.844+0000 7f2656992700 1 -- 172.21.15.121:0/3041135226 wait complete. 2024-04-05T21:37:26.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.845+0000 7f2656992700 1 Processor -- start 2024-04-05T21:37:26.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.845+0000 7f2656992700 1 -- start start 2024-04-05T21:37:26.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.845+0000 7f2656992700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2650100da0 0x7f2650196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:26.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.845+0000 7f2656992700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2650101750 0x7f2650196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:26.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.845+0000 7f2656992700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2650197270 con 0x7f2650101750 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.845+0000 7f2656992700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265019b000 con 0x7f2650100da0 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.846+0000 7f264f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2650101750 0x7f2650196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.846+0000 7f264f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2650101750 0x7f2650196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33962/0 (socket says 172.21.15.121:33962) 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.846+0000 7f264f7fe700 1 -- 172.21.15.121:0/2128495462 learned_addr learned my addr 172.21.15.121:0/2128495462 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.846+0000 7f264ffff700 1 --2- 172.21.15.121:0/2128495462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2650100da0 0x7f2650196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.846+0000 7f264f7fe700 1 -- 172.21.15.121:0/2128495462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2650100da0 msgr2=0x7f2650196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.846+0000 7f264f7fe700 1 --2- 172.21.15.121:0/2128495462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2650100da0 0x7f2650196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.846+0000 7f264f7fe700 1 -- 172.21.15.121:0/2128495462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26440095f0 con 0x7f2650101750 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.846+0000 7f264f7fe700 1 --2- 172.21.15.121:0/2128495462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2650101750 0x7f2650196b90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f264400bda0 tx=0x7f264400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.846+0000 7f264d7fa700 1 -- 172.21.15.121:0/2128495462 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f264401d070 con 0x7f2650101750 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.846+0000 7f2656992700 1 -- 172.21.15.121:0/2128495462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265019b280 con 0x7f2650101750 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.846+0000 7f264d7fa700 1 -- 172.21.15.121:0/2128495462 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f264400f460 con 0x7f2650101750 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.846+0000 7f264d7fa700 1 -- 172.21.15.121:0/2128495462 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2644021640 con 0x7f2650101750 2024-04-05T21:37:26.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.846+0000 7f2656992700 1 -- 172.21.15.121:0/2128495462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265019b770 con 0x7f2650101750 2024-04-05T21:37:26.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.847+0000 7f2656992700 1 -- 172.21.15.121:0/2128495462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26501091a0 con 0x7f2650101750 2024-04-05T21:37:26.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.847+0000 7f264d7fa700 1 -- 172.21.15.121:0/2128495462 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f264400fa60 con 0x7f2650101750 2024-04-05T21:37:26.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.848+0000 7f264d7fa700 1 --2- 172.21.15.121:0/2128495462 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f263806cce0 0x7f263806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:26.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.848+0000 7f264d7fa700 1 -- 172.21.15.121:0/2128495462 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f264408d7a0 con 0x7f2650101750 2024-04-05T21:37:26.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.848+0000 7f264ffff700 1 --2- 172.21.15.121:0/2128495462 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f263806cce0 0x7f263806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:26.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.848+0000 7f264ffff700 1 --2- 172.21.15.121:0/2128495462 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f263806cce0 0x7f263806f1a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2640005950 tx=0x7f264000a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:26.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:26.852+0000 7f264d7fa700 1 -- 172.21.15.121:0/2128495462 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2644057810 con 0x7f2650101750 2024-04-05T21:37:27.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.125+0000 7f2656992700 1 -- 172.21.15.121:0/2128495462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f265004ea90 con 0x7f2650101750 2024-04-05T21:37:27.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.125+0000 7f264d7fa700 1 -- 172.21.15.121:0/2128495462 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f264405ae30 con 0x7f2650101750 2024-04-05T21:37:27.126 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:37:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:37:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:37:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:37:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:37:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:37:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:37:27.126 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:37:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:37:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:37:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:37:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:37:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:37:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:37:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:37:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:37:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:37:27.127 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:37:27.127 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:37:27.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.127+0000 7f2656992700 1 -- 172.21.15.121:0/2128495462 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f263806cce0 msgr2=0x7f263806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:27.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.127+0000 7f2656992700 1 --2- 172.21.15.121:0/2128495462 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f263806cce0 0x7f263806f1a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2640005950 tx=0x7f264000a300 comp rx=0 tx=0).stop 2024-04-05T21:37:27.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.127+0000 7f2656992700 1 -- 172.21.15.121:0/2128495462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2650101750 msgr2=0x7f2650196b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:27.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.127+0000 7f2656992700 1 --2- 172.21.15.121:0/2128495462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2650101750 0x7f2650196b90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f264400bda0 tx=0x7f264400bdd0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.127+0000 7f2656992700 1 -- 172.21.15.121:0/2128495462 shutdown_connections 2024-04-05T21:37:27.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.127+0000 7f2656992700 1 --2- 172.21.15.121:0/2128495462 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f263806cce0 0x7f263806f1a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.127+0000 7f2656992700 1 --2- 172.21.15.121:0/2128495462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2650100da0 0x7f2650196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.127+0000 7f2656992700 1 --2- 172.21.15.121:0/2128495462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2650101750 0x7f2650196b90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.127+0000 7f2656992700 1 -- 172.21.15.121:0/2128495462 >> 172.21.15.121:0/2128495462 conn(0x7f26500fc630 msgr2=0x7f26500fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:27.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.127+0000 7f2656992700 1 -- 172.21.15.121:0/2128495462 shutdown_connections 2024-04-05T21:37:27.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.127+0000 7f2656992700 1 -- 172.21.15.121:0/2128495462 wait complete. 2024-04-05T21:37:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.229+0000 7f4ab4344700 1 -- 172.21.15.121:0/1870007259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aac0ffbd0 msgr2=0x7f4aac100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.229+0000 7f4ab4344700 1 --2- 172.21.15.121:0/1870007259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aac0ffbd0 0x7f4aac100050 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009910 tx=0x7f4aa8009c20 comp rx=0 tx=0).stop 2024-04-05T21:37:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.229+0000 7f4ab4344700 1 -- 172.21.15.121:0/1870007259 shutdown_connections 2024-04-05T21:37:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.229+0000 7f4ab4344700 1 --2- 172.21.15.121:0/1870007259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aac0ffbd0 0x7f4aac100050 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.229+0000 7f4ab4344700 1 --2- 172.21.15.121:0/1870007259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4aac105a40 0x7f4aac105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.229+0000 7f4ab4344700 1 -- 172.21.15.121:0/1870007259 >> 172.21.15.121:0/1870007259 conn(0x7f4aac0755a0 msgr2=0x7f4aac0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:27.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.229+0000 7f4ab4344700 1 -- 172.21.15.121:0/1870007259 shutdown_connections 2024-04-05T21:37:27.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.229+0000 7f4ab4344700 1 -- 172.21.15.121:0/1870007259 wait complete. 2024-04-05T21:37:27.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.230+0000 7f4ab4344700 1 Processor -- start 2024-04-05T21:37:27.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.230+0000 7f4ab4344700 1 -- start start 2024-04-05T21:37:27.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.230+0000 7f4ab4344700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4aac0ffbd0 0x7f4aac19c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:27.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.230+0000 7f4ab4344700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aac105a40 0x7f4aac19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:27.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.230+0000 7f4ab4344700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac19cff0 con 0x7f4aac105a40 2024-04-05T21:37:27.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.230+0000 7f4ab4344700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac1964b0 con 0x7f4aac0ffbd0 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.230+0000 7f4ab18df700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aac105a40 0x7f4aac19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.230+0000 7f4ab18df700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aac105a40 0x7f4aac19c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33980/0 (socket says 172.21.15.121:33980) 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.230+0000 7f4ab18df700 1 -- 172.21.15.121:0/331804905 learned_addr learned my addr 172.21.15.121:0/331804905 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.230+0000 7f4ab20e0700 1 --2- 172.21.15.121:0/331804905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4aac0ffbd0 0x7f4aac19c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.230+0000 7f4ab18df700 1 -- 172.21.15.121:0/331804905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4aac0ffbd0 msgr2=0x7f4aac19c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.230+0000 7f4ab18df700 1 --2- 172.21.15.121:0/331804905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4aac0ffbd0 0x7f4aac19c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.230+0000 7f4ab18df700 1 -- 172.21.15.121:0/331804905 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa80095f0 con 0x7f4aac105a40 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.231+0000 7f4ab18df700 1 --2- 172.21.15.121:0/331804905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aac105a40 0x7f4aac19c910 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8006010 tx=0x7f4aa800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.231+0000 7f4a9f7fe700 1 -- 172.21.15.121:0/331804905 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4aa801d070 con 0x7f4aac105a40 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.231+0000 7f4a9f7fe700 1 -- 172.21.15.121:0/331804905 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4aa800f460 con 0x7f4aac105a40 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.231+0000 7f4ab4344700 1 -- 172.21.15.121:0/331804905 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aac196730 con 0x7f4aac105a40 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.231+0000 7f4a9f7fe700 1 -- 172.21.15.121:0/331804905 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4aa8021620 con 0x7f4aac105a40 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.231+0000 7f4ab4344700 1 -- 172.21.15.121:0/331804905 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aac196c20 con 0x7f4aac105a40 2024-04-05T21:37:27.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.232+0000 7f4ab4344700 1 -- 172.21.15.121:0/331804905 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aac108060 con 0x7f4aac105a40 2024-04-05T21:37:27.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.232+0000 7f4a9f7fe700 1 -- 172.21.15.121:0/331804905 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4aa800fa40 con 0x7f4aac105a40 2024-04-05T21:37:27.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.232+0000 7f4a9f7fe700 1 --2- 172.21.15.121:0/331804905 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4a9806cce0 0x7f4a9806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:27.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.233+0000 7f4a9f7fe700 1 -- 172.21.15.121:0/331804905 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4aa808d790 con 0x7f4aac105a40 2024-04-05T21:37:27.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.233+0000 7f4ab20e0700 1 --2- 172.21.15.121:0/331804905 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4a9806cce0 0x7f4a9806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:27.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.233+0000 7f4ab20e0700 1 --2- 172.21.15.121:0/331804905 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4a9806cce0 0x7f4a9806f1a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0005950 tx=0x7f4aa00058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:27.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.237+0000 7f4a9f7fe700 1 -- 172.21.15.121:0/331804905 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4aa805b290 con 0x7f4aac105a40 2024-04-05T21:37:27.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.473+0000 7f4ab4344700 1 -- 172.21.15.121:0/331804905 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4aac04ea90 con 0x7f4aac105a40 2024-04-05T21:37:27.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.473+0000 7f4a9f7fe700 1 -- 172.21.15.121:0/331804905 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f4aa805ae20 con 0x7f4aac105a40 2024-04-05T21:37:27.474 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:37:27.474 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:37:27.474 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:37:27.474 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:37:27.474 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:37:27.474 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:37:27.474 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:37:27.475 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:37:27.476 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:37:27.477 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:37:27.477 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:37:27.477 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:37:27.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.475+0000 7f4ab4344700 1 -- 172.21.15.121:0/331804905 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4a9806cce0 msgr2=0x7f4a9806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:27.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.475+0000 7f4ab4344700 1 --2- 172.21.15.121:0/331804905 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4a9806cce0 0x7f4a9806f1a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0005950 tx=0x7f4aa00058e0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.475+0000 7f4ab4344700 1 -- 172.21.15.121:0/331804905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aac105a40 msgr2=0x7f4aac19c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:27.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.475+0000 7f4ab4344700 1 --2- 172.21.15.121:0/331804905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aac105a40 0x7f4aac19c910 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8006010 tx=0x7f4aa800be20 comp rx=0 tx=0).stop 2024-04-05T21:37:27.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.475+0000 7f4ab4344700 1 -- 172.21.15.121:0/331804905 shutdown_connections 2024-04-05T21:37:27.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.475+0000 7f4ab4344700 1 --2- 172.21.15.121:0/331804905 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4a9806cce0 0x7f4a9806f1a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.475+0000 7f4ab4344700 1 --2- 172.21.15.121:0/331804905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4aac0ffbd0 0x7f4aac19c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.475+0000 7f4ab4344700 1 --2- 172.21.15.121:0/331804905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aac105a40 0x7f4aac19c910 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.475+0000 7f4ab4344700 1 -- 172.21.15.121:0/331804905 >> 172.21.15.121:0/331804905 conn(0x7f4aac0755a0 msgr2=0x7f4aac0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:27.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.475+0000 7f4ab4344700 1 -- 172.21.15.121:0/331804905 shutdown_connections 2024-04-05T21:37:27.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.475+0000 7f4ab4344700 1 -- 172.21.15.121:0/331804905 wait complete. 2024-04-05T21:37:27.478 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:37:27.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.581+0000 7f973320c700 1 -- 172.21.15.121:0/3446243527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c0ffbd0 msgr2=0x7f972c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:27.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.581+0000 7f973320c700 1 --2- 172.21.15.121:0/3446243527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c0ffbd0 0x7f972c100050 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9720009910 tx=0x7f9720009c20 comp rx=0 tx=0).stop 2024-04-05T21:37:27.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.581+0000 7f973320c700 1 -- 172.21.15.121:0/3446243527 shutdown_connections 2024-04-05T21:37:27.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.581+0000 7f973320c700 1 --2- 172.21.15.121:0/3446243527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c0ffbd0 0x7f972c100050 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.581+0000 7f973320c700 1 --2- 172.21.15.121:0/3446243527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f972c105a40 0x7f972c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.581+0000 7f973320c700 1 -- 172.21.15.121:0/3446243527 >> 172.21.15.121:0/3446243527 conn(0x7f972c0755a0 msgr2=0x7f972c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:27.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.581+0000 7f973320c700 1 -- 172.21.15.121:0/3446243527 shutdown_connections 2024-04-05T21:37:27.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.581+0000 7f973320c700 1 -- 172.21.15.121:0/3446243527 wait complete. 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.582+0000 7f973320c700 1 Processor -- start 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.582+0000 7f973320c700 1 -- start start 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.582+0000 7f973320c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c0ffbd0 0x7f972c101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.582+0000 7f973320c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f972c105a40 0x7f972c101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.582+0000 7f973320c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c102090 con 0x7f972c0ffbd0 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.582+0000 7f973320c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c102200 con 0x7f972c105a40 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.582+0000 7f9730fa8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c0ffbd0 0x7f972c101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.582+0000 7f9730fa8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c0ffbd0 0x7f972c101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33998/0 (socket says 172.21.15.121:33998) 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.582+0000 7f9730fa8700 1 -- 172.21.15.121:0/3876389682 learned_addr learned my addr 172.21.15.121:0/3876389682 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.582+0000 7f972bfff700 1 --2- 172.21.15.121:0/3876389682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f972c105a40 0x7f972c101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.582+0000 7f9730fa8700 1 -- 172.21.15.121:0/3876389682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f972c105a40 msgr2=0x7f972c101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.582+0000 7f9730fa8700 1 --2- 172.21.15.121:0/3876389682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f972c105a40 0x7f972c101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.582+0000 7f9730fa8700 1 -- 172.21.15.121:0/3876389682 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97200095f0 con 0x7f972c0ffbd0 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.583+0000 7f9730fa8700 1 --2- 172.21.15.121:0/3876389682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c0ffbd0 0x7f972c101610 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f972400ef10 tx=0x7f972400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:27.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.583+0000 7f9729ffb700 1 -- 172.21.15.121:0/3876389682 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f972400ca50 con 0x7f972c0ffbd0 2024-04-05T21:37:27.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.583+0000 7f9729ffb700 1 -- 172.21.15.121:0/3876389682 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f972400f040 con 0x7f972c0ffbd0 2024-04-05T21:37:27.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.583+0000 7f973320c700 1 -- 172.21.15.121:0/3876389682 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972c102480 con 0x7f972c0ffbd0 2024-04-05T21:37:27.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.583+0000 7f9729ffb700 1 -- 172.21.15.121:0/3876389682 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97240146b0 con 0x7f972c0ffbd0 2024-04-05T21:37:27.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.583+0000 7f973320c700 1 -- 172.21.15.121:0/3876389682 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972c1a2ca0 con 0x7f972c0ffbd0 2024-04-05T21:37:27.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.584+0000 7f973320c700 1 -- 172.21.15.121:0/3876389682 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f972c108fe0 con 0x7f972c0ffbd0 2024-04-05T21:37:27.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.584+0000 7f9729ffb700 1 -- 172.21.15.121:0/3876389682 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f972400cbb0 con 0x7f972c0ffbd0 2024-04-05T21:37:27.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.585+0000 7f9729ffb700 1 --2- 172.21.15.121:0/3876389682 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f971406cce0 0x7f971406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:27.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.585+0000 7f9729ffb700 1 -- 172.21.15.121:0/3876389682 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f972408c860 con 0x7f972c0ffbd0 2024-04-05T21:37:27.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.585+0000 7f972bfff700 1 --2- 172.21.15.121:0/3876389682 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f971406cce0 0x7f971406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:27.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.585+0000 7f972bfff700 1 --2- 172.21.15.121:0/3876389682 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f971406cce0 0x7f971406f1a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f97200059d0 tx=0x7f9720005940 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:27.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.589+0000 7f9729ffb700 1 -- 172.21.15.121:0/3876389682 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f97240568d0 con 0x7f972c0ffbd0 2024-04-05T21:37:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:27 smithi121 ceph-mon[145859]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:37:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:27 smithi121 ceph-mon[145859]: pgmap v336: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:27 smithi121 ceph-mon[145859]: from='client.34486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:37:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:27 smithi121 ceph-mon[145859]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:37:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:27 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2128495462' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:37:27.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:27 smithi188 ceph-mon[107793]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:37:27.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:27 smithi188 ceph-mon[107793]: pgmap v336: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:27.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:27 smithi188 ceph-mon[107793]: from='client.34486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:37:27.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:27 smithi188 ceph-mon[107793]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:37:27.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:27 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2128495462' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:37:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.802+0000 7f973320c700 1 -- 172.21.15.121:0/3876389682 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f972c102480 con 0x7f971406cce0 2024-04-05T21:37:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.804+0000 7f9729ffb700 1 -- 172.21.15.121:0/3876389682 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f972c102480 con 0x7f971406cce0 2024-04-05T21:37:27.804 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:37:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:37:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:37:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:37:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:37:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:37:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:37:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:37:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:37:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:37:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:37:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:37:27.805 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:37:27.805 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:37:27.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.806+0000 7f973320c700 1 -- 172.21.15.121:0/3876389682 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f971406cce0 msgr2=0x7f971406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:27.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.806+0000 7f973320c700 1 --2- 172.21.15.121:0/3876389682 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f971406cce0 0x7f971406f1a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f97200059d0 tx=0x7f9720005940 comp rx=0 tx=0).stop 2024-04-05T21:37:27.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.806+0000 7f973320c700 1 -- 172.21.15.121:0/3876389682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c0ffbd0 msgr2=0x7f972c101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:27.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.806+0000 7f973320c700 1 --2- 172.21.15.121:0/3876389682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c0ffbd0 0x7f972c101610 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f972400ef10 tx=0x7f972400c3a0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.806+0000 7f973320c700 1 -- 172.21.15.121:0/3876389682 shutdown_connections 2024-04-05T21:37:27.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.806+0000 7f973320c700 1 --2- 172.21.15.121:0/3876389682 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f971406cce0 0x7f971406f1a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.806+0000 7f973320c700 1 --2- 172.21.15.121:0/3876389682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f972c0ffbd0 0x7f972c101610 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.806+0000 7f973320c700 1 --2- 172.21.15.121:0/3876389682 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f972c105a40 0x7f972c101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.806+0000 7f973320c700 1 -- 172.21.15.121:0/3876389682 >> 172.21.15.121:0/3876389682 conn(0x7f972c0755a0 msgr2=0x7f972c103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:27.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.806+0000 7f973320c700 1 -- 172.21.15.121:0/3876389682 shutdown_connections 2024-04-05T21:37:27.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.806+0000 7f973320c700 1 -- 172.21.15.121:0/3876389682 wait complete. 2024-04-05T21:37:27.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.909+0000 7f74f09e4700 1 -- 172.21.15.121:0/2480683800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74ec069020 msgr2=0x7f74ec105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:27.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.909+0000 7f74f09e4700 1 --2- 172.21.15.121:0/2480683800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74ec069020 0x7f74ec105790 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f74e0009910 tx=0x7f74e0009c20 comp rx=0 tx=0).stop 2024-04-05T21:37:27.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.909+0000 7f74f09e4700 1 -- 172.21.15.121:0/2480683800 shutdown_connections 2024-04-05T21:37:27.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.909+0000 7f74f09e4700 1 --2- 172.21.15.121:0/2480683800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74ec069020 0x7f74ec105790 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.909+0000 7f74f09e4700 1 --2- 172.21.15.121:0/2480683800 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74ec068670 0x7f74ec068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.909+0000 7f74f09e4700 1 -- 172.21.15.121:0/2480683800 >> 172.21.15.121:0/2480683800 conn(0x7f74ec0755a0 msgr2=0x7f74ec0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:27.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.909+0000 7f74f09e4700 1 -- 172.21.15.121:0/2480683800 shutdown_connections 2024-04-05T21:37:27.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.909+0000 7f74f09e4700 1 -- 172.21.15.121:0/2480683800 wait complete. 2024-04-05T21:37:27.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.909+0000 7f74f09e4700 1 Processor -- start 2024-04-05T21:37:27.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.909+0000 7f74f09e4700 1 -- start start 2024-04-05T21:37:27.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74f09e4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74ec068670 0x7f74ec196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:27.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74f09e4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74ec069020 0x7f74ec196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:27.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74f09e4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec197260 con 0x7f74ec069020 2024-04-05T21:37:27.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74f09e4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec19aff0 con 0x7f74ec068670 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74e9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74ec069020 0x7f74ec196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74e9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74ec069020 0x7f74ec196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34004/0 (socket says 172.21.15.121:34004) 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74e9d9b700 1 -- 172.21.15.121:0/1350594422 learned_addr learned my addr 172.21.15.121:0/1350594422 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74ea59c700 1 --2- 172.21.15.121:0/1350594422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74ec068670 0x7f74ec196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74e9d9b700 1 -- 172.21.15.121:0/1350594422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74ec068670 msgr2=0x7f74ec196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74e9d9b700 1 --2- 172.21.15.121:0/1350594422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74ec068670 0x7f74ec196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74e9d9b700 1 -- 172.21.15.121:0/1350594422 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e00095f0 con 0x7f74ec069020 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74e9d9b700 1 --2- 172.21.15.121:0/1350594422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74ec069020 0x7f74ec196b80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f74e00092e0 tx=0x7f74e000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74db7fe700 1 -- 172.21.15.121:0/1350594422 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74e001d070 con 0x7f74ec069020 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74db7fe700 1 -- 172.21.15.121:0/1350594422 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74e0005dc0 con 0x7f74ec069020 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74f09e4700 1 -- 172.21.15.121:0/1350594422 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ec19b270 con 0x7f74ec069020 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74db7fe700 1 -- 172.21.15.121:0/1350594422 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74e000fea0 con 0x7f74ec069020 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.910+0000 7f74f09e4700 1 -- 172.21.15.121:0/1350594422 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ec19b760 con 0x7f74ec069020 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.911+0000 7f74f09e4700 1 -- 172.21.15.121:0/1350594422 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ec1090a0 con 0x7f74ec069020 2024-04-05T21:37:27.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.912+0000 7f74db7fe700 1 -- 172.21.15.121:0/1350594422 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f74e0003a60 con 0x7f74ec069020 2024-04-05T21:37:27.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.912+0000 7f74db7fe700 1 --2- 172.21.15.121:0/1350594422 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f74d406cce0 0x7f74d406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:27.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.912+0000 7f74db7fe700 1 -- 172.21.15.121:0/1350594422 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f74e008d740 con 0x7f74ec069020 2024-04-05T21:37:27.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.912+0000 7f74ea59c700 1 --2- 172.21.15.121:0/1350594422 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f74d406cce0 0x7f74d406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:27.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.913+0000 7f74ea59c700 1 --2- 172.21.15.121:0/1350594422 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f74d406cce0 0x7f74d406f1a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f74dc005950 tx=0x7f74dc00b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:27.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:27.917+0000 7f74db7fe700 1 -- 172.21.15.121:0/1350594422 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f74e00577b0 con 0x7f74ec069020 2024-04-05T21:37:28.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:28.185+0000 7f74f09e4700 1 -- 172.21.15.121:0/1350594422 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f74ec02cd60 con 0x7f74ec069020 2024-04-05T21:37:28.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:28.186+0000 7f74db7fe700 1 -- 172.21.15.121:0/1350594422 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f74e0003d10 con 0x7f74ec069020 2024-04-05T21:37:28.187 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:37:28.187 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:37:28.187 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:37:28.187 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:37:28.187 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:37:28.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:28.188+0000 7f74f09e4700 1 -- 172.21.15.121:0/1350594422 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f74d406cce0 msgr2=0x7f74d406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:28.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:28.188+0000 7f74f09e4700 1 --2- 172.21.15.121:0/1350594422 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f74d406cce0 0x7f74d406f1a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f74dc005950 tx=0x7f74dc00b410 comp rx=0 tx=0).stop 2024-04-05T21:37:28.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:28.188+0000 7f74f09e4700 1 -- 172.21.15.121:0/1350594422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74ec069020 msgr2=0x7f74ec196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:28.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:28.188+0000 7f74f09e4700 1 --2- 172.21.15.121:0/1350594422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74ec069020 0x7f74ec196b80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f74e00092e0 tx=0x7f74e000bd70 comp rx=0 tx=0).stop 2024-04-05T21:37:28.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:28.188+0000 7f74f09e4700 1 -- 172.21.15.121:0/1350594422 shutdown_connections 2024-04-05T21:37:28.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:28.188+0000 7f74f09e4700 1 --2- 172.21.15.121:0/1350594422 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f74d406cce0 0x7f74d406f1a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:28.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:28.188+0000 7f74f09e4700 1 --2- 172.21.15.121:0/1350594422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f74ec068670 0x7f74ec196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:28.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:28.188+0000 7f74f09e4700 1 --2- 172.21.15.121:0/1350594422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74ec069020 0x7f74ec196b80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:28.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:28.188+0000 7f74f09e4700 1 -- 172.21.15.121:0/1350594422 >> 172.21.15.121:0/1350594422 conn(0x7f74ec0755a0 msgr2=0x7f74ec105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:28.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:28.188+0000 7f74f09e4700 1 -- 172.21.15.121:0/1350594422 shutdown_connections 2024-04-05T21:37:28.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:28.188+0000 7f74f09e4700 1 -- 172.21.15.121:0/1350594422 wait complete. 2024-04-05T21:37:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:28 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/331804905' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:37:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:28 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1350594422' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:37:28.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:28 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/331804905' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:37:28.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:28 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1350594422' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:37:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:29 smithi121 ceph-mon[145859]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:37:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:29 smithi121 ceph-mon[145859]: pgmap v337: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:29 smithi188 ceph-mon[107793]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:37:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:29 smithi188 ceph-mon[107793]: pgmap v337: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:37:30.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:37:31.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:31 smithi121 ceph-mon[145859]: pgmap v338: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:31.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:31 smithi188 ceph-mon[107793]: pgmap v338: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:33 smithi121 ceph-mon[145859]: pgmap v339: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:33 smithi188 ceph-mon[107793]: pgmap v339: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:35.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:35 smithi121 ceph-mon[145859]: pgmap v340: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:35.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:35 smithi188 ceph-mon[107793]: pgmap v340: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:37 smithi121 ceph-mon[145859]: pgmap v341: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:37.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:37 smithi188 ceph-mon[107793]: pgmap v341: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:39.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:39 smithi121 ceph-mon[145859]: pgmap v342: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:39.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:39 smithi188 ceph-mon[107793]: pgmap v342: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:41.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:41 smithi188 ceph-mon[107793]: pgmap v343: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:41.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:41 smithi121 ceph-mon[145859]: pgmap v343: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:43 smithi121 ceph-mon[145859]: pgmap v344: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:43.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:43 smithi188 ceph-mon[107793]: pgmap v344: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:45 smithi121 ceph-mon[145859]: pgmap v345: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:37:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:45 smithi188 ceph-mon[107793]: pgmap v345: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:37:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:47 smithi121 ceph-mon[145859]: pgmap v346: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:47 smithi188 ceph-mon[107793]: pgmap v346: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:49 smithi121 ceph-mon[145859]: pgmap v347: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:49.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:49 smithi188 ceph-mon[107793]: pgmap v347: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:51 smithi121 ceph-mon[145859]: pgmap v348: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:51 smithi188 ceph-mon[107793]: pgmap v348: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:53 smithi121 ceph-mon[145859]: pgmap v349: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:53 smithi188 ceph-mon[107793]: pgmap v349: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:55.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:55 smithi121 ceph-mon[145859]: pgmap v350: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:55 smithi188 ceph-mon[107793]: pgmap v350: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:57 smithi121 ceph-mon[145859]: pgmap v351: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:57 smithi188 ceph-mon[107793]: pgmap v351: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:58.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.305+0000 7fadbb95c700 1 -- 172.21.15.121:0/1461551325 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4100df0 msgr2=0x7fadb41011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:58.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.305+0000 7fadbb95c700 1 --2- 172.21.15.121:0/1461551325 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4100df0 0x7fadb41011d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fadb0009910 tx=0x7fadb0009c20 comp rx=0 tx=0).stop 2024-04-05T21:37:58.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.305+0000 7fadbb95c700 1 -- 172.21.15.121:0/1461551325 shutdown_connections 2024-04-05T21:37:58.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.305+0000 7fadbb95c700 1 --2- 172.21.15.121:0/1461551325 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadb41017a0 0x7fadb41057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.305+0000 7fadbb95c700 1 --2- 172.21.15.121:0/1461551325 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4100df0 0x7fadb41011d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.305+0000 7fadbb95c700 1 -- 172.21.15.121:0/1461551325 >> 172.21.15.121:0/1461551325 conn(0x7fadb40fc6a0 msgr2=0x7fadb40feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:58.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.305+0000 7fadbb95c700 1 -- 172.21.15.121:0/1461551325 shutdown_connections 2024-04-05T21:37:58.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.305+0000 7fadbb95c700 1 -- 172.21.15.121:0/1461551325 wait complete. 2024-04-05T21:37:58.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.306+0000 7fadbb95c700 1 Processor -- start 2024-04-05T21:37:58.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.306+0000 7fadbb95c700 1 -- start start 2024-04-05T21:37:58.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.306+0000 7fadbb95c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4100df0 0x7fadb419c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:58.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.306+0000 7fadbb95c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadb41017a0 0x7fadb419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:58.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.306+0000 7fadbb95c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb419d110 con 0x7fadb4100df0 2024-04-05T21:37:58.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.306+0000 7fadbb95c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb41965c0 con 0x7fadb41017a0 2024-04-05T21:37:58.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.306+0000 7fadb96f8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4100df0 0x7fadb419c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:58.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.306+0000 7fadb96f8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4100df0 0x7fadb419c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48700/0 (socket says 172.21.15.121:48700) 2024-04-05T21:37:58.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.306+0000 7fadb96f8700 1 -- 172.21.15.121:0/2834661410 learned_addr learned my addr 172.21.15.121:0/2834661410 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:37:58.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.306+0000 7fadb8ef7700 1 --2- 172.21.15.121:0/2834661410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadb41017a0 0x7fadb419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:58.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.307+0000 7fadb96f8700 1 -- 172.21.15.121:0/2834661410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadb41017a0 msgr2=0x7fadb419ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:58.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.307+0000 7fadb96f8700 1 --2- 172.21.15.121:0/2834661410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadb41017a0 0x7fadb419ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.307+0000 7fadb96f8700 1 -- 172.21.15.121:0/2834661410 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb00095f0 con 0x7fadb4100df0 2024-04-05T21:37:58.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.307+0000 7fadb96f8700 1 --2- 172.21.15.121:0/2834661410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4100df0 0x7fadb419c540 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fadb000b5c0 tx=0x7fadb000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:58.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.307+0000 7fada67fc700 1 -- 172.21.15.121:0/2834661410 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadb001d070 con 0x7fadb4100df0 2024-04-05T21:37:58.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.307+0000 7fada67fc700 1 -- 172.21.15.121:0/2834661410 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadb000f460 con 0x7fadb4100df0 2024-04-05T21:37:58.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.307+0000 7fada67fc700 1 -- 172.21.15.121:0/2834661410 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadb0021640 con 0x7fadb4100df0 2024-04-05T21:37:58.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.307+0000 7fadbb95c700 1 -- 172.21.15.121:0/2834661410 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb4196840 con 0x7fadb4100df0 2024-04-05T21:37:58.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.307+0000 7fadbb95c700 1 -- 172.21.15.121:0/2834661410 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb4196d30 con 0x7fadb4100df0 2024-04-05T21:37:58.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.308+0000 7fadbb95c700 1 -- 172.21.15.121:0/2834661410 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb4109190 con 0x7fadb4100df0 2024-04-05T21:37:58.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.309+0000 7fada67fc700 1 -- 172.21.15.121:0/2834661410 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fadb000f9b0 con 0x7fadb4100df0 2024-04-05T21:37:58.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.309+0000 7fada67fc700 1 --2- 172.21.15.121:0/2834661410 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fada006cce0 0x7fada006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:58.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.309+0000 7fadb8ef7700 1 --2- 172.21.15.121:0/2834661410 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fada006cce0 0x7fada006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:58.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.309+0000 7fada67fc700 1 -- 172.21.15.121:0/2834661410 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fadb008d880 con 0x7fadb4100df0 2024-04-05T21:37:58.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.310+0000 7fadb8ef7700 1 --2- 172.21.15.121:0/2834661410 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fada006cce0 0x7fada006f1a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fada8006fd0 tx=0x7fada8008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:58.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.314+0000 7fada67fc700 1 -- 172.21.15.121:0/2834661410 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fadb005b2e0 con 0x7fadb4100df0 2024-04-05T21:37:58.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.529+0000 7fadbb95c700 1 -- 172.21.15.121:0/2834661410 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fadb402cd00 con 0x7fada006cce0 2024-04-05T21:37:58.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.530+0000 7fada67fc700 1 -- 172.21.15.121:0/2834661410 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fadb402cd00 con 0x7fada006cce0 2024-04-05T21:37:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.532+0000 7fadbb95c700 1 -- 172.21.15.121:0/2834661410 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fada006cce0 msgr2=0x7fada006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.532+0000 7fadbb95c700 1 --2- 172.21.15.121:0/2834661410 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fada006cce0 0x7fada006f1a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fada8006fd0 tx=0x7fada8008040 comp rx=0 tx=0).stop 2024-04-05T21:37:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.532+0000 7fadbb95c700 1 -- 172.21.15.121:0/2834661410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4100df0 msgr2=0x7fadb419c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.532+0000 7fadbb95c700 1 --2- 172.21.15.121:0/2834661410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4100df0 0x7fadb419c540 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fadb000b5c0 tx=0x7fadb000be20 comp rx=0 tx=0).stop 2024-04-05T21:37:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.532+0000 7fadbb95c700 1 -- 172.21.15.121:0/2834661410 shutdown_connections 2024-04-05T21:37:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.532+0000 7fadbb95c700 1 --2- 172.21.15.121:0/2834661410 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fada006cce0 0x7fada006f1a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.532+0000 7fadbb95c700 1 --2- 172.21.15.121:0/2834661410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4100df0 0x7fadb419c540 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.532+0000 7fadbb95c700 1 --2- 172.21.15.121:0/2834661410 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadb41017a0 0x7fadb419ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.532+0000 7fadbb95c700 1 -- 172.21.15.121:0/2834661410 >> 172.21.15.121:0/2834661410 conn(0x7fadb40fc6a0 msgr2=0x7fadb40fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.532+0000 7fadbb95c700 1 -- 172.21.15.121:0/2834661410 shutdown_connections 2024-04-05T21:37:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.532+0000 7fadbb95c700 1 -- 172.21.15.121:0/2834661410 wait complete. 2024-04-05T21:37:58.543 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:37:58.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.635+0000 7f75e1c1e700 1 -- 172.21.15.121:0/3561117343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75dc0ffbd0 msgr2=0x7f75dc100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:58.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.635+0000 7f75e1c1e700 1 --2- 172.21.15.121:0/3561117343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75dc0ffbd0 0x7f75dc100050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009910 tx=0x7f75d0009c20 comp rx=0 tx=0).stop 2024-04-05T21:37:58.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.635+0000 7f75e1c1e700 1 -- 172.21.15.121:0/3561117343 shutdown_connections 2024-04-05T21:37:58.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.635+0000 7f75e1c1e700 1 --2- 172.21.15.121:0/3561117343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75dc0ffbd0 0x7f75dc100050 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.635+0000 7f75e1c1e700 1 --2- 172.21.15.121:0/3561117343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75dc105a40 0x7f75dc105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.635+0000 7f75e1c1e700 1 -- 172.21.15.121:0/3561117343 >> 172.21.15.121:0/3561117343 conn(0x7f75dc0755a0 msgr2=0x7f75dc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.635+0000 7f75e1c1e700 1 -- 172.21.15.121:0/3561117343 shutdown_connections 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.635+0000 7f75e1c1e700 1 -- 172.21.15.121:0/3561117343 wait complete. 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.636+0000 7f75e1c1e700 1 Processor -- start 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.636+0000 7f75e1c1e700 1 -- start start 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.636+0000 7f75e1c1e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75dc0ffbd0 0x7f75dc101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.636+0000 7f75e1c1e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75dc105a40 0x7f75dc101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.636+0000 7f75e1c1e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc102090 con 0x7f75dc105a40 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.636+0000 7f75e1c1e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc1021d0 con 0x7f75dc0ffbd0 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.636+0000 7f75daffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75dc105a40 0x7f75dc101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.636+0000 7f75daffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75dc105a40 0x7f75dc101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48720/0 (socket says 172.21.15.121:48720) 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.636+0000 7f75daffd700 1 -- 172.21.15.121:0/2287112364 learned_addr learned my addr 172.21.15.121:0/2287112364 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.637+0000 7f75db7fe700 1 --2- 172.21.15.121:0/2287112364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75dc0ffbd0 0x7f75dc101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.637+0000 7f75daffd700 1 -- 172.21.15.121:0/2287112364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75dc0ffbd0 msgr2=0x7f75dc101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.637+0000 7f75daffd700 1 --2- 172.21.15.121:0/2287112364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75dc0ffbd0 0x7f75dc101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.637+0000 7f75daffd700 1 -- 172.21.15.121:0/2287112364 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d00095f0 con 0x7f75dc105a40 2024-04-05T21:37:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.637+0000 7f75daffd700 1 --2- 172.21.15.121:0/2287112364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75dc105a40 0x7f75dc101b50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f75d0006010 tx=0x7f75d000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.637+0000 7f75d8ff9700 1 -- 172.21.15.121:0/2287112364 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75d001d070 con 0x7f75dc105a40 2024-04-05T21:37:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.637+0000 7f75e1c1e700 1 -- 172.21.15.121:0/2287112364 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75dc102450 con 0x7f75dc105a40 2024-04-05T21:37:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.637+0000 7f75d8ff9700 1 -- 172.21.15.121:0/2287112364 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75d000f460 con 0x7f75dc105a40 2024-04-05T21:37:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.637+0000 7f75d8ff9700 1 -- 172.21.15.121:0/2287112364 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75d0021640 con 0x7f75dc105a40 2024-04-05T21:37:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.637+0000 7f75e1c1e700 1 -- 172.21.15.121:0/2287112364 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75dc1a2c40 con 0x7f75dc105a40 2024-04-05T21:37:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.638+0000 7f75e1c1e700 1 -- 172.21.15.121:0/2287112364 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75dc108fe0 con 0x7f75dc105a40 2024-04-05T21:37:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.638+0000 7f75d8ff9700 1 -- 172.21.15.121:0/2287112364 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f75d000fa60 con 0x7f75dc105a40 2024-04-05T21:37:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.639+0000 7f75d8ff9700 1 --2- 172.21.15.121:0/2287112364 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f75c406cce0 0x7f75c406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.639+0000 7f75d8ff9700 1 -- 172.21.15.121:0/2287112364 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f75d008d8b0 con 0x7f75dc105a40 2024-04-05T21:37:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.639+0000 7f75db7fe700 1 --2- 172.21.15.121:0/2287112364 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f75c406cce0 0x7f75c406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.639+0000 7f75db7fe700 1 --2- 172.21.15.121:0/2287112364 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f75c406cce0 0x7f75c406f1a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f75cc005950 tx=0x7f75cc005cf0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:58.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.643+0000 7f75d8ff9700 1 -- 172.21.15.121:0/2287112364 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f75d0057920 con 0x7f75dc105a40 2024-04-05T21:37:58.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.854+0000 7f75e1c1e700 1 -- 172.21.15.121:0/2287112364 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f75dc102c50 con 0x7f75c406cce0 2024-04-05T21:37:58.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.855+0000 7f75d8ff9700 1 -- 172.21.15.121:0/2287112364 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f75dc102c50 con 0x7f75c406cce0 2024-04-05T21:37:58.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.858+0000 7f75e1c1e700 1 -- 172.21.15.121:0/2287112364 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f75c406cce0 msgr2=0x7f75c406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:58.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.858+0000 7f75e1c1e700 1 --2- 172.21.15.121:0/2287112364 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f75c406cce0 0x7f75c406f1a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f75cc005950 tx=0x7f75cc005cf0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.858+0000 7f75e1c1e700 1 -- 172.21.15.121:0/2287112364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75dc105a40 msgr2=0x7f75dc101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:58.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.858+0000 7f75e1c1e700 1 --2- 172.21.15.121:0/2287112364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75dc105a40 0x7f75dc101b50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f75d0006010 tx=0x7f75d000be20 comp rx=0 tx=0).stop 2024-04-05T21:37:58.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.858+0000 7f75e1c1e700 1 -- 172.21.15.121:0/2287112364 shutdown_connections 2024-04-05T21:37:58.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.858+0000 7f75e1c1e700 1 --2- 172.21.15.121:0/2287112364 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f75c406cce0 0x7f75c406f1a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.858+0000 7f75e1c1e700 1 --2- 172.21.15.121:0/2287112364 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75dc0ffbd0 0x7f75dc101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.858+0000 7f75e1c1e700 1 --2- 172.21.15.121:0/2287112364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75dc105a40 0x7f75dc101b50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.858+0000 7f75e1c1e700 1 -- 172.21.15.121:0/2287112364 >> 172.21.15.121:0/2287112364 conn(0x7f75dc0755a0 msgr2=0x7f75dc103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:58.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.858+0000 7f75e1c1e700 1 -- 172.21.15.121:0/2287112364 shutdown_connections 2024-04-05T21:37:58.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.858+0000 7f75e1c1e700 1 -- 172.21.15.121:0/2287112364 wait complete. 2024-04-05T21:37:58.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.959+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/2931498676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e0100de0 msgr2=0x7ff4e01011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:58.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.959+0000 7ff4e6a0f700 1 --2- 172.21.15.121:0/2931498676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e0100de0 0x7ff4e01011c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4009910 tx=0x7ff4d4009c20 comp rx=0 tx=0).stop 2024-04-05T21:37:58.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.959+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/2931498676 shutdown_connections 2024-04-05T21:37:58.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.959+0000 7ff4e6a0f700 1 --2- 172.21.15.121:0/2931498676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e0101790 0x7ff4e01057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.960+0000 7ff4e6a0f700 1 --2- 172.21.15.121:0/2931498676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e0100de0 0x7ff4e01011c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.960+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/2931498676 >> 172.21.15.121:0/2931498676 conn(0x7ff4e00fc6b0 msgr2=0x7ff4e00fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:58.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.960+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/2931498676 shutdown_connections 2024-04-05T21:37:58.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.960+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/2931498676 wait complete. 2024-04-05T21:37:58.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.960+0000 7ff4e6a0f700 1 Processor -- start 2024-04-05T21:37:58.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.960+0000 7ff4e6a0f700 1 -- start start 2024-04-05T21:37:58.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4e6a0f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e0100de0 0x7ff4e0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:58.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4e6a0f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e0101790 0x7ff4e00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:58.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4e6a0f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e0073b10 con 0x7ff4e0101790 2024-04-05T21:37:58.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4e6a0f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e0073c50 con 0x7ff4e0100de0 2024-04-05T21:37:58.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4df7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e0101790 0x7ff4e00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:58.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4df7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e0101790 0x7ff4e00735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48750/0 (socket says 172.21.15.121:48750) 2024-04-05T21:37:58.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4df7fe700 1 -- 172.21.15.121:0/556656728 learned_addr learned my addr 172.21.15.121:0/556656728 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:37:58.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4dffff700 1 --2- 172.21.15.121:0/556656728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e0100de0 0x7ff4e0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:58.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4df7fe700 1 -- 172.21.15.121:0/556656728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e0100de0 msgr2=0x7ff4e0073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:58.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4df7fe700 1 --2- 172.21.15.121:0/556656728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e0100de0 0x7ff4e0073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:58.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4df7fe700 1 -- 172.21.15.121:0/556656728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d40095f0 con 0x7ff4e0101790 2024-04-05T21:37:58.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4df7fe700 1 --2- 172.21.15.121:0/556656728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e0101790 0x7ff4e00735d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000d900 tx=0x7ff4d000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:58.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4dd7fa700 1 -- 172.21.15.121:0/556656728 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4d0009900 con 0x7ff4e0101790 2024-04-05T21:37:58.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4dd7fa700 1 -- 172.21.15.121:0/556656728 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4d0010460 con 0x7ff4e0101790 2024-04-05T21:37:58.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/556656728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e01a2880 con 0x7ff4e0101790 2024-04-05T21:37:58.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4dd7fa700 1 -- 172.21.15.121:0/556656728 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4d000f5f0 con 0x7ff4e0101790 2024-04-05T21:37:58.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.961+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/556656728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e01a2c90 con 0x7ff4e0101790 2024-04-05T21:37:58.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.962+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/556656728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e0109180 con 0x7ff4e0101790 2024-04-05T21:37:58.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.964+0000 7ff4dd7fa700 1 -- 172.21.15.121:0/556656728 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff4d00105d0 con 0x7ff4e0101790 2024-04-05T21:37:58.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.964+0000 7ff4dd7fa700 1 --2- 172.21.15.121:0/556656728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff4c806cce0 0x7ff4c806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:58.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.964+0000 7ff4dd7fa700 1 -- 172.21.15.121:0/556656728 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff4d008c070 con 0x7ff4e0101790 2024-04-05T21:37:58.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.964+0000 7ff4dffff700 1 --2- 172.21.15.121:0/556656728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff4c806cce0 0x7ff4c806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:58.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.965+0000 7ff4dffff700 1 --2- 172.21.15.121:0/556656728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff4c806cce0 0x7ff4c806f1a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4006010 tx=0x7ff4d401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:58.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:58.968+0000 7ff4dd7fa700 1 -- 172.21.15.121:0/556656728 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff4d0056200 con 0x7ff4e0101790 2024-04-05T21:37:59.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.168+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/556656728 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4e002cdb0 con 0x7ff4c806cce0 2024-04-05T21:37:59.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.179+0000 7ff4dd7fa700 1 -- 172.21.15.121:0/556656728 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff4e002cdb0 con 0x7ff4c806cce0 2024-04-05T21:37:59.179 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:37:59.180 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (8m) 2m ago 20m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:37:59.180 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (20m) 2m ago 20m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:37:59.180 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (19m) 2m ago 19m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:37:59.180 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (6m) 2m ago 20m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:37:59.180 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (6m) 2m ago 19m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:37:59.180 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (8m) 2m ago 20m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:37:59.180 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (3m) 2m ago 14m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:37:59.180 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (2m) 2m ago 14m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:37:59.180 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (2m) 2m ago 14m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:37:59.180 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (14m) 2m ago 14m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:37:59.180 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (10m) 2m ago 21m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:37:59.180 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (9m) 2m ago 19m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:37:59.180 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (7m) 2m ago 22m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:37:59.181 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (7m) 2m ago 19m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:37:59.181 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (9m) 2m ago 20m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:37:59.181 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (9m) 2m ago 19m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:37:59.181 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (6m) 2m ago 18m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:37:59.181 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (5m) 2m ago 17m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:37:59.181 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (5m) 2m ago 17m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:37:59.181 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (4m) 2m ago 16m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:37:59.181 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (4m) 2m ago 16m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:37:59.181 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (3m) 2m ago 15m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:37:59.181 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (9m) 2m ago 19m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:37:59.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.181+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/556656728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff4c806cce0 msgr2=0x7ff4c806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:59.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.181+0000 7ff4e6a0f700 1 --2- 172.21.15.121:0/556656728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff4c806cce0 0x7ff4c806f1a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4006010 tx=0x7ff4d401a040 comp rx=0 tx=0).stop 2024-04-05T21:37:59.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.181+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/556656728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e0101790 msgr2=0x7ff4e00735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:59.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.181+0000 7ff4e6a0f700 1 --2- 172.21.15.121:0/556656728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e0101790 0x7ff4e00735d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000d900 tx=0x7ff4d000dc10 comp rx=0 tx=0).stop 2024-04-05T21:37:59.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.181+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/556656728 shutdown_connections 2024-04-05T21:37:59.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.181+0000 7ff4e6a0f700 1 --2- 172.21.15.121:0/556656728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff4c806cce0 0x7ff4c806f1a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.181+0000 7ff4e6a0f700 1 --2- 172.21.15.121:0/556656728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e0100de0 0x7ff4e0073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.181+0000 7ff4e6a0f700 1 --2- 172.21.15.121:0/556656728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e0101790 0x7ff4e00735d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.181+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/556656728 >> 172.21.15.121:0/556656728 conn(0x7ff4e00fc6b0 msgr2=0x7ff4e00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:59.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.181+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/556656728 shutdown_connections 2024-04-05T21:37:59.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.181+0000 7ff4e6a0f700 1 -- 172.21.15.121:0/556656728 wait complete. 2024-04-05T21:37:59.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.282+0000 7f187c5bc700 1 -- 172.21.15.121:0/501847931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1874101d70 msgr2=0x7f1874102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:59.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.282+0000 7f187c5bc700 1 --2- 172.21.15.121:0/501847931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1874101d70 0x7f1874102150 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1864009940 tx=0x7f1864009c50 comp rx=0 tx=0).stop 2024-04-05T21:37:59.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.282+0000 7f187c5bc700 1 -- 172.21.15.121:0/501847931 shutdown_connections 2024-04-05T21:37:59.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.282+0000 7f187c5bc700 1 --2- 172.21.15.121:0/501847931 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1874102690 0x7f187410ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.282+0000 7f187c5bc700 1 --2- 172.21.15.121:0/501847931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1874101d70 0x7f1874102150 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.282+0000 7f187c5bc700 1 -- 172.21.15.121:0/501847931 >> 172.21.15.121:0/501847931 conn(0x7f18740fb540 msgr2=0x7f18740fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:59.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.282+0000 7f187c5bc700 1 -- 172.21.15.121:0/501847931 shutdown_connections 2024-04-05T21:37:59.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.282+0000 7f187c5bc700 1 -- 172.21.15.121:0/501847931 wait complete. 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.282+0000 7f187c5bc700 1 Processor -- start 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.282+0000 7f187c5bc700 1 -- start start 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f187c5bc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1874101d70 0x7f1874196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f187c5bc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1874102690 0x7f1874196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f187c5bc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1874197270 con 0x7f1874101d70 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f187c5bc700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187419b000 con 0x7f1874102690 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f187a358700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1874101d70 0x7f1874196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f187a358700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1874101d70 0x7f1874196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48766/0 (socket says 172.21.15.121:48766) 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f187a358700 1 -- 172.21.15.121:0/2281774118 learned_addr learned my addr 172.21.15.121:0/2281774118 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f1879b57700 1 --2- 172.21.15.121:0/2281774118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1874102690 0x7f1874196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f187a358700 1 -- 172.21.15.121:0/2281774118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1874102690 msgr2=0x7f1874196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f187a358700 1 --2- 172.21.15.121:0/2281774118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1874102690 0x7f1874196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f187a358700 1 -- 172.21.15.121:0/2281774118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18640095f0 con 0x7f1874101d70 2024-04-05T21:37:59.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f187a358700 1 --2- 172.21.15.121:0/2281774118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1874101d70 0x7f1874196650 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f186400b580 tx=0x7f18640055f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:59.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f186b7fe700 1 -- 172.21.15.121:0/2281774118 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f186401d070 con 0x7f1874101d70 2024-04-05T21:37:59.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f187c5bc700 1 -- 172.21.15.121:0/2281774118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187419b280 con 0x7f1874101d70 2024-04-05T21:37:59.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.283+0000 7f186b7fe700 1 -- 172.21.15.121:0/2281774118 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f186400bea0 con 0x7f1874101d70 2024-04-05T21:37:59.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.284+0000 7f186b7fe700 1 -- 172.21.15.121:0/2281774118 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1864021410 con 0x7f1874101d70 2024-04-05T21:37:59.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.284+0000 7f187c5bc700 1 -- 172.21.15.121:0/2281774118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187419b770 con 0x7f1874101d70 2024-04-05T21:37:59.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.285+0000 7f187c5bc700 1 -- 172.21.15.121:0/2281774118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1858005320 con 0x7f1874101d70 2024-04-05T21:37:59.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.285+0000 7f186b7fe700 1 -- 172.21.15.121:0/2281774118 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f186400f6f0 con 0x7f1874101d70 2024-04-05T21:37:59.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.285+0000 7f186b7fe700 1 --2- 172.21.15.121:0/2281774118 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f186006cce0 0x7f186006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:59.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.285+0000 7f186b7fe700 1 -- 172.21.15.121:0/2281774118 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f186408d6e0 con 0x7f1874101d70 2024-04-05T21:37:59.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.285+0000 7f1879b57700 1 --2- 172.21.15.121:0/2281774118 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f186006cce0 0x7f186006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:59.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.286+0000 7f1879b57700 1 --2- 172.21.15.121:0/2281774118 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f186006cce0 0x7f186006f1a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1874197c70 tx=0x7f187000b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.290+0000 7f186b7fe700 1 -- 172.21.15.121:0/2281774118 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1864057810 con 0x7f1874101d70 2024-04-05T21:37:59.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.572+0000 7f187c5bc700 1 -- 172.21.15.121:0/2281774118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1858006200 con 0x7f1874101d70 2024-04-05T21:37:59.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.573+0000 7f186b7fe700 1 -- 172.21.15.121:0/2281774118 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f186405ae30 con 0x7f1874101d70 2024-04-05T21:37:59.573 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:37:59.574 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:37:59.574 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:37:59.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:37:59.574 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:37:59.574 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:37:59.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:37:59.574 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:37:59.574 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:37:59.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:37:59.574 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:37:59.574 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:37:59.574 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:37:59.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:37:59.575 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:37:59.575 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:37:59.575 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:37:59.575 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:37:59.575 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.575+0000 7f187c5bc700 1 -- 172.21.15.121:0/2281774118 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f186006cce0 msgr2=0x7f186006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.575+0000 7f187c5bc700 1 --2- 172.21.15.121:0/2281774118 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f186006cce0 0x7f186006f1a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1874197c70 tx=0x7f187000b410 comp rx=0 tx=0).stop 2024-04-05T21:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.575+0000 7f187c5bc700 1 -- 172.21.15.121:0/2281774118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1874101d70 msgr2=0x7f1874196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.575+0000 7f187c5bc700 1 --2- 172.21.15.121:0/2281774118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1874101d70 0x7f1874196650 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f186400b580 tx=0x7f18640055f0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.575+0000 7f187c5bc700 1 -- 172.21.15.121:0/2281774118 shutdown_connections 2024-04-05T21:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.575+0000 7f187c5bc700 1 --2- 172.21.15.121:0/2281774118 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f186006cce0 0x7f186006f1a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.575+0000 7f187c5bc700 1 --2- 172.21.15.121:0/2281774118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1874101d70 0x7f1874196650 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.575+0000 7f187c5bc700 1 --2- 172.21.15.121:0/2281774118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1874102690 0x7f1874196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.575+0000 7f187c5bc700 1 -- 172.21.15.121:0/2281774118 >> 172.21.15.121:0/2281774118 conn(0x7f18740fb540 msgr2=0x7f18741053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.575+0000 7f187c5bc700 1 -- 172.21.15.121:0/2281774118 shutdown_connections 2024-04-05T21:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.575+0000 7f187c5bc700 1 -- 172.21.15.121:0/2281774118 wait complete. 2024-04-05T21:37:59.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.676+0000 7f626c150700 1 -- 172.21.15.121:0/2493246773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62640ff050 msgr2=0x7f62640ff4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:59.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.676+0000 7f626c150700 1 --2- 172.21.15.121:0/2493246773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62640ff050 0x7f62640ff4d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6254009910 tx=0x7f6254009c20 comp rx=0 tx=0).stop 2024-04-05T21:37:59.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.676+0000 7f626c150700 1 -- 172.21.15.121:0/2493246773 shutdown_connections 2024-04-05T21:37:59.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.676+0000 7f626c150700 1 --2- 172.21.15.121:0/2493246773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62640ff050 0x7f62640ff4d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.676+0000 7f626c150700 1 --2- 172.21.15.121:0/2493246773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6264105b20 0x7f6264105f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.676+0000 7f626c150700 1 -- 172.21.15.121:0/2493246773 >> 172.21.15.121:0/2493246773 conn(0x7f6264075e30 msgr2=0x7f62640fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:59.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.677+0000 7f626c150700 1 -- 172.21.15.121:0/2493246773 shutdown_connections 2024-04-05T21:37:59.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.677+0000 7f626c150700 1 -- 172.21.15.121:0/2493246773 wait complete. 2024-04-05T21:37:59.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.677+0000 7f626c150700 1 Processor -- start 2024-04-05T21:37:59.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.677+0000 7f626c150700 1 -- start start 2024-04-05T21:37:59.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.677+0000 7f626c150700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62640ff050 0x7f62641966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:59.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.677+0000 7f626c150700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6264105b20 0x7f6264196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:59.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.677+0000 7f626c150700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6264197290 con 0x7f6264105b20 2024-04-05T21:37:59.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.677+0000 7f626c150700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6264199fc0 con 0x7f62640ff050 2024-04-05T21:37:59.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.677+0000 7f62696eb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6264105b20 0x7f6264196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:59.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.677+0000 7f62696eb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6264105b20 0x7f6264196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48786/0 (socket says 172.21.15.121:48786) 2024-04-05T21:37:59.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.677+0000 7f62696eb700 1 -- 172.21.15.121:0/1435246321 learned_addr learned my addr 172.21.15.121:0/1435246321 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:37:59.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.678+0000 7f62696eb700 1 -- 172.21.15.121:0/1435246321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62640ff050 msgr2=0x7f62641966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:59.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.678+0000 7f6269eec700 1 --2- 172.21.15.121:0/1435246321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62640ff050 0x7f62641966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:59.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.678+0000 7f62696eb700 1 --2- 172.21.15.121:0/1435246321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62640ff050 0x7f62641966c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.678+0000 7f62696eb700 1 -- 172.21.15.121:0/1435246321 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62540095f0 con 0x7f6264105b20 2024-04-05T21:37:59.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.678+0000 7f62696eb700 1 --2- 172.21.15.121:0/1435246321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6264105b20 0x7f6264196c00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6254006010 tx=0x7f625400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:59.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.678+0000 7f625affd700 1 -- 172.21.15.121:0/1435246321 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f625401d070 con 0x7f6264105b20 2024-04-05T21:37:59.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.678+0000 7f626c150700 1 -- 172.21.15.121:0/1435246321 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626419a240 con 0x7f6264105b20 2024-04-05T21:37:59.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.678+0000 7f625affd700 1 -- 172.21.15.121:0/1435246321 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f625400f460 con 0x7f6264105b20 2024-04-05T21:37:59.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.678+0000 7f625affd700 1 -- 172.21.15.121:0/1435246321 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6254021640 con 0x7f6264105b20 2024-04-05T21:37:59.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.678+0000 7f626c150700 1 -- 172.21.15.121:0/1435246321 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626419a730 con 0x7f6264105b20 2024-04-05T21:37:59.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.679+0000 7f626c150700 1 -- 172.21.15.121:0/1435246321 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62641090c0 con 0x7f6264105b20 2024-04-05T21:37:59.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.679+0000 7f625affd700 1 -- 172.21.15.121:0/1435246321 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f625400f5d0 con 0x7f6264105b20 2024-04-05T21:37:59.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.680+0000 7f625affd700 1 --2- 172.21.15.121:0/1435246321 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f625006cce0 0x7f625006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:37:59.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.680+0000 7f625affd700 1 -- 172.21.15.121:0/1435246321 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f625408d720 con 0x7f6264105b20 2024-04-05T21:37:59.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.680+0000 7f6269eec700 1 --2- 172.21.15.121:0/1435246321 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f625006cce0 0x7f625006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:37:59.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.680+0000 7f6269eec700 1 --2- 172.21.15.121:0/1435246321 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f625006cce0 0x7f625006f1a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6260005950 tx=0x7f62600058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:37:59.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.684+0000 7f625affd700 1 -- 172.21.15.121:0/1435246321 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6254057840 con 0x7f6264105b20 2024-04-05T21:37:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:59 smithi188 ceph-mon[107793]: pgmap v352: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:59.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:37:59 smithi188 ceph-mon[107793]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:37:59.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.921+0000 7f626c150700 1 -- 172.21.15.121:0/1435246321 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6264197af0 con 0x7f6264105b20 2024-04-05T21:37:59.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.921+0000 7f625affd700 1 -- 172.21.15.121:0/1435246321 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f6254005740 con 0x7f6264105b20 2024-04-05T21:37:59.922 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:37:59.922 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:37:59.922 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:37:59.922 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:37:59.922 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:37:59.922 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:37:59.922 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:37:59.922 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:37:59.922 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:37:59.923 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:37:59.924 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:37:59.925 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:37:59.925 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:37:59.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.923+0000 7f626c150700 1 -- 172.21.15.121:0/1435246321 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f625006cce0 msgr2=0x7f625006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:59.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.923+0000 7f626c150700 1 --2- 172.21.15.121:0/1435246321 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f625006cce0 0x7f625006f1a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6260005950 tx=0x7f62600058e0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.923+0000 7f626c150700 1 -- 172.21.15.121:0/1435246321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6264105b20 msgr2=0x7f6264196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:37:59.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.923+0000 7f626c150700 1 --2- 172.21.15.121:0/1435246321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6264105b20 0x7f6264196c00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6254006010 tx=0x7f625400be20 comp rx=0 tx=0).stop 2024-04-05T21:37:59.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.923+0000 7f626c150700 1 -- 172.21.15.121:0/1435246321 shutdown_connections 2024-04-05T21:37:59.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.923+0000 7f626c150700 1 --2- 172.21.15.121:0/1435246321 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f625006cce0 0x7f625006f1a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.923+0000 7f626c150700 1 --2- 172.21.15.121:0/1435246321 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f62640ff050 0x7f62641966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.923+0000 7f626c150700 1 --2- 172.21.15.121:0/1435246321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6264105b20 0x7f6264196c00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:37:59.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.923+0000 7f626c150700 1 -- 172.21.15.121:0/1435246321 >> 172.21.15.121:0/1435246321 conn(0x7f6264075e30 msgr2=0x7f6264104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:37:59.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.923+0000 7f626c150700 1 -- 172.21.15.121:0/1435246321 shutdown_connections 2024-04-05T21:37:59.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:37:59.923+0000 7f626c150700 1 -- 172.21.15.121:0/1435246321 wait complete. 2024-04-05T21:37:59.926 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:37:59.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:59 smithi121 ceph-mon[145859]: pgmap v352: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:37:59.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:37:59 smithi121 ceph-mon[145859]: from='client.34510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:00.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.027+0000 7f30e9bf2700 1 -- 172.21.15.121:0/1381635260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30e4069020 msgr2=0x7f30e4105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:00.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.027+0000 7f30e9bf2700 1 --2- 172.21.15.121:0/1381635260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30e4069020 0x7f30e4105790 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f30d8009910 tx=0x7f30d8009c20 comp rx=0 tx=0).stop 2024-04-05T21:38:00.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.027+0000 7f30e9bf2700 1 -- 172.21.15.121:0/1381635260 shutdown_connections 2024-04-05T21:38:00.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.027+0000 7f30e9bf2700 1 --2- 172.21.15.121:0/1381635260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30e4069020 0x7f30e4105790 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:00.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.027+0000 7f30e9bf2700 1 --2- 172.21.15.121:0/1381635260 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30e4068670 0x7f30e4068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:00.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.027+0000 7f30e9bf2700 1 -- 172.21.15.121:0/1381635260 >> 172.21.15.121:0/1381635260 conn(0x7f30e40755a0 msgr2=0x7f30e40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:00.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.027+0000 7f30e9bf2700 1 -- 172.21.15.121:0/1381635260 shutdown_connections 2024-04-05T21:38:00.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.027+0000 7f30e9bf2700 1 -- 172.21.15.121:0/1381635260 wait complete. 2024-04-05T21:38:00.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.028+0000 7f30e9bf2700 1 Processor -- start 2024-04-05T21:38:00.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.028+0000 7f30e9bf2700 1 -- start start 2024-04-05T21:38:00.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e9bf2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30e4068670 0x7f30e4196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:00.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e9bf2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30e4069020 0x7f30e4196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:00.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e9bf2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e4197260 con 0x7f30e4069020 2024-04-05T21:38:00.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e9bf2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e419aff0 con 0x7f30e4068670 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30e4069020 0x7f30e4196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30e4069020 0x7f30e4196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48806/0 (socket says 172.21.15.121:48806) 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e2ffd700 1 -- 172.21.15.121:0/4220770279 learned_addr learned my addr 172.21.15.121:0/4220770279 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e37fe700 1 --2- 172.21.15.121:0/4220770279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30e4068670 0x7f30e4196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e2ffd700 1 -- 172.21.15.121:0/4220770279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30e4068670 msgr2=0x7f30e4196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e2ffd700 1 --2- 172.21.15.121:0/4220770279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30e4068670 0x7f30e4196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e2ffd700 1 -- 172.21.15.121:0/4220770279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d80095f0 con 0x7f30e4069020 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e2ffd700 1 --2- 172.21.15.121:0/4220770279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30e4069020 0x7f30e4196b80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f30d8005f50 tx=0x7f30d800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e0ff9700 1 -- 172.21.15.121:0/4220770279 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30d801d070 con 0x7f30e4069020 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e0ff9700 1 -- 172.21.15.121:0/4220770279 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30d800f460 con 0x7f30e4069020 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e9bf2700 1 -- 172.21.15.121:0/4220770279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e419b270 con 0x7f30e4069020 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e0ff9700 1 -- 172.21.15.121:0/4220770279 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30d8021640 con 0x7f30e4069020 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.029+0000 7f30e9bf2700 1 -- 172.21.15.121:0/4220770279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e419b760 con 0x7f30e4069020 2024-04-05T21:38:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.030+0000 7f30e9bf2700 1 -- 172.21.15.121:0/4220770279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e41090a0 con 0x7f30e4069020 2024-04-05T21:38:00.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.031+0000 7f30e0ff9700 1 -- 172.21.15.121:0/4220770279 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f30d800f9e0 con 0x7f30e4069020 2024-04-05T21:38:00.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.031+0000 7f30e0ff9700 1 --2- 172.21.15.121:0/4220770279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f30cc06cce0 0x7f30cc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:00.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.031+0000 7f30e0ff9700 1 -- 172.21.15.121:0/4220770279 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f30d808d7e0 con 0x7f30e4069020 2024-04-05T21:38:00.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.031+0000 7f30e37fe700 1 --2- 172.21.15.121:0/4220770279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f30cc06cce0 0x7f30cc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:00.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.031+0000 7f30e37fe700 1 --2- 172.21.15.121:0/4220770279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f30cc06cce0 0x7f30cc06f1a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f30d4005950 tx=0x7f30d400a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:00.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.036+0000 7f30e0ff9700 1 -- 172.21.15.121:0/4220770279 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f30d8057850 con 0x7f30e4069020 2024-04-05T21:38:00.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.247+0000 7f30e9bf2700 1 -- 172.21.15.121:0/4220770279 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f30e402cd60 con 0x7f30cc06cce0 2024-04-05T21:38:00.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.248+0000 7f30e0ff9700 1 -- 172.21.15.121:0/4220770279 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f30e402cd60 con 0x7f30cc06cce0 2024-04-05T21:38:00.249 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:38:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:38:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:38:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:38:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:38:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:38:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:38:00.249 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:38:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:38:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:38:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:38:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:38:00.250 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:38:00.250 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:38:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.250+0000 7f30e9bf2700 1 -- 172.21.15.121:0/4220770279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f30cc06cce0 msgr2=0x7f30cc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.250+0000 7f30e9bf2700 1 --2- 172.21.15.121:0/4220770279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f30cc06cce0 0x7f30cc06f1a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f30d4005950 tx=0x7f30d400a300 comp rx=0 tx=0).stop 2024-04-05T21:38:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.250+0000 7f30e9bf2700 1 -- 172.21.15.121:0/4220770279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30e4069020 msgr2=0x7f30e4196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.250+0000 7f30e9bf2700 1 --2- 172.21.15.121:0/4220770279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30e4069020 0x7f30e4196b80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f30d8005f50 tx=0x7f30d800bd70 comp rx=0 tx=0).stop 2024-04-05T21:38:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.250+0000 7f30e9bf2700 1 -- 172.21.15.121:0/4220770279 shutdown_connections 2024-04-05T21:38:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.250+0000 7f30e9bf2700 1 --2- 172.21.15.121:0/4220770279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f30cc06cce0 0x7f30cc06f1a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:00.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.250+0000 7f30e9bf2700 1 --2- 172.21.15.121:0/4220770279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30e4068670 0x7f30e4196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.250+0000 7f30e9bf2700 1 --2- 172.21.15.121:0/4220770279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30e4069020 0x7f30e4196b80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.250+0000 7f30e9bf2700 1 -- 172.21.15.121:0/4220770279 >> 172.21.15.121:0/4220770279 conn(0x7f30e40755a0 msgr2=0x7f30e4105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.250+0000 7f30e9bf2700 1 -- 172.21.15.121:0/4220770279 shutdown_connections 2024-04-05T21:38:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.251+0000 7f30e9bf2700 1 -- 172.21.15.121:0/4220770279 wait complete. 2024-04-05T21:38:00.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.353+0000 7f7c3717c700 1 -- 172.21.15.121:0/1187066856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c30100de0 msgr2=0x7f7c301011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:00.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.353+0000 7f7c3717c700 1 --2- 172.21.15.121:0/1187066856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c30100de0 0x7f7c301011c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009910 tx=0x7f7c24009c20 comp rx=0 tx=0).stop 2024-04-05T21:38:00.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.354+0000 7f7c3717c700 1 -- 172.21.15.121:0/1187066856 shutdown_connections 2024-04-05T21:38:00.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.354+0000 7f7c3717c700 1 --2- 172.21.15.121:0/1187066856 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c30101790 0x7f7c301057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:00.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.354+0000 7f7c3717c700 1 --2- 172.21.15.121:0/1187066856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c30100de0 0x7f7c301011c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:00.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.354+0000 7f7c3717c700 1 -- 172.21.15.121:0/1187066856 >> 172.21.15.121:0/1187066856 conn(0x7f7c300fc670 msgr2=0x7f7c300fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:00.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.354+0000 7f7c3717c700 1 -- 172.21.15.121:0/1187066856 shutdown_connections 2024-04-05T21:38:00.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.354+0000 7f7c3717c700 1 -- 172.21.15.121:0/1187066856 wait complete. 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.354+0000 7f7c3717c700 1 Processor -- start 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.354+0000 7f7c3717c700 1 -- start start 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.354+0000 7f7c3717c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c30100de0 0x7f7c30196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.354+0000 7f7c3717c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c30101790 0x7f7c30196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.354+0000 7f7c3717c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c30197290 con 0x7f7c30100de0 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.354+0000 7f7c3717c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3019b020 con 0x7f7c30101790 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.354+0000 7f7c34f18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c30100de0 0x7f7c30196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.355+0000 7f7c34f18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c30100de0 0x7f7c30196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48832/0 (socket says 172.21.15.121:48832) 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.355+0000 7f7c34f18700 1 -- 172.21.15.121:0/2254592966 learned_addr learned my addr 172.21.15.121:0/2254592966 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.355+0000 7f7c2ffff700 1 --2- 172.21.15.121:0/2254592966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c30101790 0x7f7c30196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.355+0000 7f7c34f18700 1 -- 172.21.15.121:0/2254592966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c30101790 msgr2=0x7f7c30196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.355+0000 7f7c34f18700 1 --2- 172.21.15.121:0/2254592966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c30101790 0x7f7c30196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.355+0000 7f7c34f18700 1 -- 172.21.15.121:0/2254592966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c240095f0 con 0x7f7c30100de0 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.355+0000 7f7c34f18700 1 --2- 172.21.15.121:0/2254592966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c30100de0 0x7f7c30196670 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7c24006010 tx=0x7f7c2400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:00.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.355+0000 7f7c2dffb700 1 -- 172.21.15.121:0/2254592966 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c2401d070 con 0x7f7c30100de0 2024-04-05T21:38:00.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.355+0000 7f7c2dffb700 1 -- 172.21.15.121:0/2254592966 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c24005710 con 0x7f7c30100de0 2024-04-05T21:38:00.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.355+0000 7f7c3717c700 1 -- 172.21.15.121:0/2254592966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c3019b2a0 con 0x7f7c30100de0 2024-04-05T21:38:00.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.355+0000 7f7c2dffb700 1 -- 172.21.15.121:0/2254592966 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c2400fe20 con 0x7f7c30100de0 2024-04-05T21:38:00.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.355+0000 7f7c3717c700 1 -- 172.21.15.121:0/2254592966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3019b790 con 0x7f7c30100de0 2024-04-05T21:38:00.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.356+0000 7f7c3717c700 1 -- 172.21.15.121:0/2254592966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c30109180 con 0x7f7c30100de0 2024-04-05T21:38:00.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.357+0000 7f7c2dffb700 1 -- 172.21.15.121:0/2254592966 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7c24003a60 con 0x7f7c30100de0 2024-04-05T21:38:00.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.357+0000 7f7c2dffb700 1 --2- 172.21.15.121:0/2254592966 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7c2006cce0 0x7f7c2006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:00.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.357+0000 7f7c2dffb700 1 -- 172.21.15.121:0/2254592966 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7c2408d750 con 0x7f7c30100de0 2024-04-05T21:38:00.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.357+0000 7f7c2ffff700 1 --2- 172.21.15.121:0/2254592966 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7c2006cce0 0x7f7c2006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:00.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.358+0000 7f7c2ffff700 1 --2- 172.21.15.121:0/2254592966 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7c2006cce0 0x7f7c2006f1a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7c30197c90 tx=0x7f7c1c00b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:00.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.362+0000 7f7c2dffb700 1 -- 172.21.15.121:0/2254592966 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7c240577c0 con 0x7f7c30100de0 2024-04-05T21:38:00.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.632+0000 7f7c3717c700 1 -- 172.21.15.121:0/2254592966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7c3002cd30 con 0x7f7c30100de0 2024-04-05T21:38:00.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.632+0000 7f7c2dffb700 1 -- 172.21.15.121:0/2254592966 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f7c2405ade0 con 0x7f7c30100de0 2024-04-05T21:38:00.633 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:38:00.633 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:38:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:38:00.633 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:38:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:38:00.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.634+0000 7f7c3717c700 1 -- 172.21.15.121:0/2254592966 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7c2006cce0 msgr2=0x7f7c2006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:00.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.634+0000 7f7c3717c700 1 --2- 172.21.15.121:0/2254592966 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7c2006cce0 0x7f7c2006f1a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7c30197c90 tx=0x7f7c1c00b410 comp rx=0 tx=0).stop 2024-04-05T21:38:00.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.634+0000 7f7c3717c700 1 -- 172.21.15.121:0/2254592966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c30100de0 msgr2=0x7f7c30196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:00.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.634+0000 7f7c3717c700 1 --2- 172.21.15.121:0/2254592966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c30100de0 0x7f7c30196670 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7c24006010 tx=0x7f7c2400bd70 comp rx=0 tx=0).stop 2024-04-05T21:38:00.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.634+0000 7f7c3717c700 1 -- 172.21.15.121:0/2254592966 shutdown_connections 2024-04-05T21:38:00.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.634+0000 7f7c3717c700 1 --2- 172.21.15.121:0/2254592966 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7c2006cce0 0x7f7c2006f1a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:00.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.634+0000 7f7c3717c700 1 --2- 172.21.15.121:0/2254592966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c30100de0 0x7f7c30196670 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:00.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.634+0000 7f7c3717c700 1 --2- 172.21.15.121:0/2254592966 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c30101790 0x7f7c30196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:00.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.634+0000 7f7c3717c700 1 -- 172.21.15.121:0/2254592966 >> 172.21.15.121:0/2254592966 conn(0x7f7c300fc670 msgr2=0x7f7c300fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:00.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.634+0000 7f7c3717c700 1 -- 172.21.15.121:0/2254592966 shutdown_connections 2024-04-05T21:38:00.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:00.634+0000 7f7c3717c700 1 -- 172.21.15.121:0/2254592966 wait complete. 2024-04-05T21:38:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:00 smithi188 ceph-mon[107793]: from='client.34514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:00 smithi188 ceph-mon[107793]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:00 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2281774118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:38:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:00 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1435246321' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:38:00.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:38:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:00 smithi121 ceph-mon[145859]: from='client.34514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:00 smithi121 ceph-mon[145859]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:00.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:00 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2281774118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:38:00.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:00 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1435246321' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:38:00.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:38:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:01 smithi188 ceph-mon[107793]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:01.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:01 smithi188 ceph-mon[107793]: pgmap v353: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:01.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:01 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2254592966' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:38:01.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:01 smithi121 ceph-mon[145859]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:01.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:01 smithi121 ceph-mon[145859]: pgmap v353: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:01.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:01 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2254592966' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:38:03.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:03 smithi121 ceph-mon[145859]: pgmap v354: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:03 smithi188 ceph-mon[107793]: pgmap v354: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:05 smithi121 ceph-mon[145859]: pgmap v355: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:05 smithi188 ceph-mon[107793]: pgmap v355: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:07 smithi121 ceph-mon[145859]: pgmap v356: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:07 smithi188 ceph-mon[107793]: pgmap v356: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:09 smithi121 ceph-mon[145859]: pgmap v357: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:09 smithi188 ceph-mon[107793]: pgmap v357: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:11 smithi121 ceph-mon[145859]: pgmap v358: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:11.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:11 smithi188 ceph-mon[107793]: pgmap v358: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:13 smithi121 ceph-mon[145859]: pgmap v359: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:13.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:13 smithi188 ceph-mon[107793]: pgmap v359: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:15 smithi121 ceph-mon[145859]: pgmap v360: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:38:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:15 smithi188 ceph-mon[107793]: pgmap v360: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:38:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:17 smithi121 ceph-mon[145859]: pgmap v361: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:17.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:17 smithi188 ceph-mon[107793]: pgmap v361: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:19 smithi121 ceph-mon[145859]: pgmap v362: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:19 smithi188 ceph-mon[107793]: pgmap v362: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:21 smithi121 ceph-mon[145859]: pgmap v363: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:21 smithi188 ceph-mon[107793]: pgmap v363: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:23.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:23 smithi121 ceph-mon[145859]: pgmap v364: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:23 smithi188 ceph-mon[107793]: pgmap v364: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:25.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:25 smithi121 ceph-mon[145859]: pgmap v365: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:25.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:25 smithi188 ceph-mon[107793]: pgmap v365: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:27 smithi121 ceph-mon[145859]: pgmap v366: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:27.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:27 smithi188 ceph-mon[107793]: pgmap v366: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:29 smithi121 ceph-mon[145859]: pgmap v367: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:29 smithi188 ceph-mon[107793]: pgmap v367: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:38:30.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.750+0000 7f3ee673a700 1 -- 172.21.15.121:0/4175082955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee0105a40 msgr2=0x7f3ee0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:30.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.750+0000 7f3ee673a700 1 --2- 172.21.15.121:0/4175082955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee0105a40 0x7f3ee0105e20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4009910 tx=0x7f3ed4009c20 comp rx=0 tx=0).stop 2024-04-05T21:38:30.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.750+0000 7f3ee673a700 1 -- 172.21.15.121:0/4175082955 shutdown_connections 2024-04-05T21:38:30.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.750+0000 7f3ee673a700 1 --2- 172.21.15.121:0/4175082955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ee00ffa40 0x7f3ee00ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:30.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.750+0000 7f3ee673a700 1 --2- 172.21.15.121:0/4175082955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee0105a40 0x7f3ee0105e20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:30.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.750+0000 7f3ee673a700 1 -- 172.21.15.121:0/4175082955 >> 172.21.15.121:0/4175082955 conn(0x7f3ee00fb570 msgr2=0x7f3ee00fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:30.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.750+0000 7f3ee673a700 1 -- 172.21.15.121:0/4175082955 shutdown_connections 2024-04-05T21:38:30.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.750+0000 7f3ee673a700 1 -- 172.21.15.121:0/4175082955 wait complete. 2024-04-05T21:38:30.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.751+0000 7f3ee673a700 1 Processor -- start 2024-04-05T21:38:30.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.751+0000 7f3ee673a700 1 -- start start 2024-04-05T21:38:30.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.752+0000 7f3ee673a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee00ffa40 0x7f3ee0195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:30.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.752+0000 7f3ee673a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ee0105a40 0x7f3ee0195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:30.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.752+0000 7f3ee673a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee0196170 con 0x7f3ee00ffa40 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.752+0000 7f3ee673a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee0199f00 con 0x7f3ee0105a40 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.752+0000 7f3edffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee00ffa40 0x7f3ee0195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.752+0000 7f3edffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee00ffa40 0x7f3ee0195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56362/0 (socket says 172.21.15.121:56362) 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.752+0000 7f3edffff700 1 -- 172.21.15.121:0/2297134440 learned_addr learned my addr 172.21.15.121:0/2297134440 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.752+0000 7f3edf7fe700 1 --2- 172.21.15.121:0/2297134440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ee0105a40 0x7f3ee0195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.752+0000 7f3edffff700 1 -- 172.21.15.121:0/2297134440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ee0105a40 msgr2=0x7f3ee0195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.752+0000 7f3edffff700 1 --2- 172.21.15.121:0/2297134440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ee0105a40 0x7f3ee0195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.752+0000 7f3edffff700 1 -- 172.21.15.121:0/2297134440 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed40095f0 con 0x7f3ee00ffa40 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.752+0000 7f3edffff700 1 --2- 172.21.15.121:0/2297134440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee00ffa40 0x7f3ee0195550 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4006010 tx=0x7f3ed400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.752+0000 7f3edd7fa700 1 -- 172.21.15.121:0/2297134440 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ed401f070 con 0x7f3ee00ffa40 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.753+0000 7f3edd7fa700 1 -- 172.21.15.121:0/2297134440 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ed4019610 con 0x7f3ee00ffa40 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.753+0000 7f3edd7fa700 1 -- 172.21.15.121:0/2297134440 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ed400f670 con 0x7f3ee00ffa40 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.753+0000 7f3ee673a700 1 -- 172.21.15.121:0/2297134440 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee019a180 con 0x7f3ee00ffa40 2024-04-05T21:38:30.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.753+0000 7f3ee673a700 1 -- 172.21.15.121:0/2297134440 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee019a670 con 0x7f3ee00ffa40 2024-04-05T21:38:30.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.754+0000 7f3ee673a700 1 -- 172.21.15.121:0/2297134440 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee004ea90 con 0x7f3ee00ffa40 2024-04-05T21:38:30.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.754+0000 7f3edd7fa700 1 -- 172.21.15.121:0/2297134440 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3ed4019bf0 con 0x7f3ee00ffa40 2024-04-05T21:38:30.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.754+0000 7f3edd7fa700 1 --2- 172.21.15.121:0/2297134440 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ec806cce0 0x7f3ec806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:30.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.754+0000 7f3edd7fa700 1 -- 172.21.15.121:0/2297134440 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3ed408fa60 con 0x7f3ee00ffa40 2024-04-05T21:38:30.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.754+0000 7f3edf7fe700 1 --2- 172.21.15.121:0/2297134440 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ec806cce0 0x7f3ec806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:30.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.755+0000 7f3edf7fe700 1 --2- 172.21.15.121:0/2297134440 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ec806cce0 0x7f3ec806f1a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0196b70 tx=0x7f3ed00094f0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:30.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.759+0000 7f3edd7fa700 1 -- 172.21.15.121:0/2297134440 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3ed4059ad0 con 0x7f3ee00ffa40 2024-04-05T21:38:30.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:38:30.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.970+0000 7f3ee673a700 1 -- 172.21.15.121:0/2297134440 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ee019abb0 con 0x7f3ec806cce0 2024-04-05T21:38:30.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.972+0000 7f3edd7fa700 1 -- 172.21.15.121:0/2297134440 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3ee019abb0 con 0x7f3ec806cce0 2024-04-05T21:38:30.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.974+0000 7f3ee673a700 1 -- 172.21.15.121:0/2297134440 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ec806cce0 msgr2=0x7f3ec806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:30.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.974+0000 7f3ee673a700 1 --2- 172.21.15.121:0/2297134440 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ec806cce0 0x7f3ec806f1a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0196b70 tx=0x7f3ed00094f0 comp rx=0 tx=0).stop 2024-04-05T21:38:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.974+0000 7f3ee673a700 1 -- 172.21.15.121:0/2297134440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee00ffa40 msgr2=0x7f3ee0195550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.974+0000 7f3ee673a700 1 --2- 172.21.15.121:0/2297134440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee00ffa40 0x7f3ee0195550 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4006010 tx=0x7f3ed400be20 comp rx=0 tx=0).stop 2024-04-05T21:38:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.974+0000 7f3ee673a700 1 -- 172.21.15.121:0/2297134440 shutdown_connections 2024-04-05T21:38:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.974+0000 7f3ee673a700 1 --2- 172.21.15.121:0/2297134440 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ec806cce0 0x7f3ec806f1a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.974+0000 7f3ee673a700 1 --2- 172.21.15.121:0/2297134440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee00ffa40 0x7f3ee0195550 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.974+0000 7f3ee673a700 1 --2- 172.21.15.121:0/2297134440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ee0105a40 0x7f3ee0195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.974+0000 7f3ee673a700 1 -- 172.21.15.121:0/2297134440 >> 172.21.15.121:0/2297134440 conn(0x7f3ee00fb570 msgr2=0x7f3ee00fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.974+0000 7f3ee673a700 1 -- 172.21.15.121:0/2297134440 shutdown_connections 2024-04-05T21:38:30.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:30.974+0000 7f3ee673a700 1 -- 172.21.15.121:0/2297134440 wait complete. 2024-04-05T21:38:30.985 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:38:31.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.078+0000 7f4481d52700 1 -- 172.21.15.121:0/2773965861 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447c101790 msgr2=0x7f447c1057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:31.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.078+0000 7f4481d52700 1 --2- 172.21.15.121:0/2773965861 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447c101790 0x7f447c1057e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f446c009910 tx=0x7f446c009c20 comp rx=0 tx=0).stop 2024-04-05T21:38:31.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.078+0000 7f4481d52700 1 -- 172.21.15.121:0/2773965861 shutdown_connections 2024-04-05T21:38:31.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.078+0000 7f4481d52700 1 --2- 172.21.15.121:0/2773965861 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447c101790 0x7f447c1057e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.078+0000 7f4481d52700 1 --2- 172.21.15.121:0/2773965861 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f447c100de0 0x7f447c1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.078+0000 7f4481d52700 1 -- 172.21.15.121:0/2773965861 >> 172.21.15.121:0/2773965861 conn(0x7f447c0fc670 msgr2=0x7f447c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:31.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.078+0000 7f4481d52700 1 -- 172.21.15.121:0/2773965861 shutdown_connections 2024-04-05T21:38:31.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.078+0000 7f4481d52700 1 -- 172.21.15.121:0/2773965861 wait complete. 2024-04-05T21:38:31.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.078+0000 7f4481d52700 1 Processor -- start 2024-04-05T21:38:31.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.079+0000 7f4481d52700 1 -- start start 2024-04-05T21:38:31.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.079+0000 7f4481d52700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447c100de0 0x7f447c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:31.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.079+0000 7f4481d52700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f447c101790 0x7f447c19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:31.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.079+0000 7f4481d52700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c19f230 con 0x7f447c100de0 2024-04-05T21:38:31.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.079+0000 7f4481d52700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c1986e0 con 0x7f447c101790 2024-04-05T21:38:31.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.079+0000 7f447b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447c100de0 0x7f447c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:31.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.079+0000 7f447b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447c100de0 0x7f447c19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56392/0 (socket says 172.21.15.121:56392) 2024-04-05T21:38:31.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.079+0000 7f447b7fe700 1 -- 172.21.15.121:0/2296181011 learned_addr learned my addr 172.21.15.121:0/2296181011 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:38:31.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.079+0000 7f447affd700 1 --2- 172.21.15.121:0/2296181011 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f447c101790 0x7f447c19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.079+0000 7f447b7fe700 1 -- 172.21.15.121:0/2296181011 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f447c101790 msgr2=0x7f447c19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.079+0000 7f447b7fe700 1 --2- 172.21.15.121:0/2296181011 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f447c101790 0x7f447c19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.079+0000 7f447b7fe700 1 -- 172.21.15.121:0/2296181011 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c0095f0 con 0x7f447c100de0 2024-04-05T21:38:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.079+0000 7f447b7fe700 1 --2- 172.21.15.121:0/2296181011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447c100de0 0x7f447c19e660 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f446400d8d0 tx=0x7f446400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.080+0000 7f4478ff9700 1 -- 172.21.15.121:0/2296181011 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44640098a0 con 0x7f447c100de0 2024-04-05T21:38:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.080+0000 7f4478ff9700 1 -- 172.21.15.121:0/2296181011 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4464010460 con 0x7f447c100de0 2024-04-05T21:38:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.080+0000 7f4481d52700 1 -- 172.21.15.121:0/2296181011 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447c1989c0 con 0x7f447c100de0 2024-04-05T21:38:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.080+0000 7f4478ff9700 1 -- 172.21.15.121:0/2296181011 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4464009ea0 con 0x7f447c100de0 2024-04-05T21:38:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.080+0000 7f4481d52700 1 -- 172.21.15.121:0/2296181011 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447c198ee0 con 0x7f447c100de0 2024-04-05T21:38:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.081+0000 7f4481d52700 1 -- 172.21.15.121:0/2296181011 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f447c104f60 con 0x7f447c100de0 2024-04-05T21:38:31.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.081+0000 7f4478ff9700 1 -- 172.21.15.121:0/2296181011 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f44640105d0 con 0x7f447c100de0 2024-04-05T21:38:31.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.081+0000 7f4478ff9700 1 --2- 172.21.15.121:0/2296181011 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f446806cce0 0x7f446806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:31.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.081+0000 7f4478ff9700 1 -- 172.21.15.121:0/2296181011 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f446408c480 con 0x7f447c100de0 2024-04-05T21:38:31.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.082+0000 7f447affd700 1 --2- 172.21.15.121:0/2296181011 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f446806cce0 0x7f446806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:31.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.082+0000 7f447affd700 1 --2- 172.21.15.121:0/2296181011 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f446806cce0 0x7f446806f1a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f446c006010 tx=0x7f446c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:31.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.086+0000 7f4478ff9700 1 -- 172.21.15.121:0/2296181011 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4464059f80 con 0x7f447c100de0 2024-04-05T21:38:31.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.296+0000 7f4481d52700 1 -- 172.21.15.121:0/2296181011 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f447c199100 con 0x7f446806cce0 2024-04-05T21:38:31.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.297+0000 7f4478ff9700 1 -- 172.21.15.121:0/2296181011 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f447c199100 con 0x7f446806cce0 2024-04-05T21:38:31.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.299+0000 7f4481d52700 1 -- 172.21.15.121:0/2296181011 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f446806cce0 msgr2=0x7f446806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:31.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.299+0000 7f4481d52700 1 --2- 172.21.15.121:0/2296181011 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f446806cce0 0x7f446806f1a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f446c006010 tx=0x7f446c01a040 comp rx=0 tx=0).stop 2024-04-05T21:38:31.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.299+0000 7f4481d52700 1 -- 172.21.15.121:0/2296181011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447c100de0 msgr2=0x7f447c19e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:31.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.299+0000 7f4481d52700 1 --2- 172.21.15.121:0/2296181011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447c100de0 0x7f447c19e660 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f446400d8d0 tx=0x7f446400dbe0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.299+0000 7f4481d52700 1 -- 172.21.15.121:0/2296181011 shutdown_connections 2024-04-05T21:38:31.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.299+0000 7f4481d52700 1 --2- 172.21.15.121:0/2296181011 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f446806cce0 0x7f446806f1a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.299+0000 7f4481d52700 1 --2- 172.21.15.121:0/2296181011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447c100de0 0x7f447c19e660 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.299+0000 7f4481d52700 1 --2- 172.21.15.121:0/2296181011 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f447c101790 0x7f447c19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.299+0000 7f4481d52700 1 -- 172.21.15.121:0/2296181011 >> 172.21.15.121:0/2296181011 conn(0x7f447c0fc670 msgr2=0x7f447c0fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:31.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.299+0000 7f4481d52700 1 -- 172.21.15.121:0/2296181011 shutdown_connections 2024-04-05T21:38:31.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.299+0000 7f4481d52700 1 -- 172.21.15.121:0/2296181011 wait complete. 2024-04-05T21:38:31.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.403+0000 7fde905fe700 1 -- 172.21.15.121:0/747512815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde88100dc0 msgr2=0x7fde881011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:31.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.403+0000 7fde905fe700 1 --2- 172.21.15.121:0/747512815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde88100dc0 0x7fde881011a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fde84009910 tx=0x7fde84009c20 comp rx=0 tx=0).stop 2024-04-05T21:38:31.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.403+0000 7fde905fe700 1 -- 172.21.15.121:0/747512815 shutdown_connections 2024-04-05T21:38:31.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.403+0000 7fde905fe700 1 --2- 172.21.15.121:0/747512815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde88101770 0x7fde881057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.403+0000 7fde905fe700 1 --2- 172.21.15.121:0/747512815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde88100dc0 0x7fde881011a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.403+0000 7fde905fe700 1 -- 172.21.15.121:0/747512815 >> 172.21.15.121:0/747512815 conn(0x7fde880fc650 msgr2=0x7fde880fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:31.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.403+0000 7fde905fe700 1 -- 172.21.15.121:0/747512815 shutdown_connections 2024-04-05T21:38:31.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.404+0000 7fde905fe700 1 -- 172.21.15.121:0/747512815 wait complete. 2024-04-05T21:38:31.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.404+0000 7fde905fe700 1 Processor -- start 2024-04-05T21:38:31.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.404+0000 7fde905fe700 1 -- start start 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde905fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde88100dc0 0x7fde88196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde905fe700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde88101770 0x7fde88196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde905fe700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde88197290 con 0x7fde88100dc0 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde905fe700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8819b020 con 0x7fde88101770 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde8e39a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde88100dc0 0x7fde88196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde8e39a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde88100dc0 0x7fde88196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56404/0 (socket says 172.21.15.121:56404) 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde8e39a700 1 -- 172.21.15.121:0/1909884627 learned_addr learned my addr 172.21.15.121:0/1909884627 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde8db99700 1 --2- 172.21.15.121:0/1909884627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde88101770 0x7fde88196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde8e39a700 1 -- 172.21.15.121:0/1909884627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde88101770 msgr2=0x7fde88196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde8e39a700 1 --2- 172.21.15.121:0/1909884627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde88101770 0x7fde88196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde8e39a700 1 -- 172.21.15.121:0/1909884627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde840095f0 con 0x7fde88100dc0 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde8e39a700 1 --2- 172.21.15.121:0/1909884627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde88100dc0 0x7fde88196670 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fde84006010 tx=0x7fde8400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde7f7fe700 1 -- 172.21.15.121:0/1909884627 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde8401d070 con 0x7fde88100dc0 2024-04-05T21:38:31.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde7f7fe700 1 -- 172.21.15.121:0/1909884627 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde8400f460 con 0x7fde88100dc0 2024-04-05T21:38:31.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde905fe700 1 -- 172.21.15.121:0/1909884627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8819b2a0 con 0x7fde88100dc0 2024-04-05T21:38:31.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde7f7fe700 1 -- 172.21.15.121:0/1909884627 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde84021620 con 0x7fde88100dc0 2024-04-05T21:38:31.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.405+0000 7fde905fe700 1 -- 172.21.15.121:0/1909884627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8819b790 con 0x7fde88100dc0 2024-04-05T21:38:31.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.406+0000 7fde905fe700 1 -- 172.21.15.121:0/1909884627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde88109160 con 0x7fde88100dc0 2024-04-05T21:38:31.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.407+0000 7fde7f7fe700 1 -- 172.21.15.121:0/1909884627 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fde8400fa40 con 0x7fde88100dc0 2024-04-05T21:38:31.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.407+0000 7fde7f7fe700 1 --2- 172.21.15.121:0/1909884627 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde7406cce0 0x7fde7406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:31.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.407+0000 7fde7f7fe700 1 -- 172.21.15.121:0/1909884627 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fde8408d790 con 0x7fde88100dc0 2024-04-05T21:38:31.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.407+0000 7fde8db99700 1 --2- 172.21.15.121:0/1909884627 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde7406cce0 0x7fde7406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:31.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.407+0000 7fde8db99700 1 --2- 172.21.15.121:0/1909884627 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde7406cce0 0x7fde7406f1a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fde78005950 tx=0x7fde780058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:31.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.411+0000 7fde7f7fe700 1 -- 172.21.15.121:0/1909884627 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fde84057800 con 0x7fde88100dc0 2024-04-05T21:38:31.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.616+0000 7fde905fe700 1 -- 172.21.15.121:0/1909884627 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fde8802cd30 con 0x7fde7406cce0 2024-04-05T21:38:31.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.626+0000 7fde7f7fe700 1 -- 172.21.15.121:0/1909884627 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fde8802cd30 con 0x7fde7406cce0 2024-04-05T21:38:31.627 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:38:31.627 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (9m) 3m ago 20m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:38:31.627 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (21m) 3m ago 21m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:38:31.627 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (19m) 3m ago 19m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:38:31.627 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (7m) 3m ago 21m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (7m) 3m ago 19m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (9m) 3m ago 20m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (3m) 3m ago 14m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (3m) 3m ago 14m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (3m) 3m ago 14m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (14m) 3m ago 14m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (11m) 3m ago 22m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (10m) 3m ago 19m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (8m) 3m ago 22m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (7m) 3m ago 19m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (10m) 3m ago 20m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (10m) 3m ago 19m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (7m) 3m ago 18m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:38:31.628 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (6m) 3m ago 18m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:38:31.629 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (5m) 3m ago 17m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:38:31.629 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (5m) 3m ago 17m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:38:31.629 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (4m) 3m ago 16m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:38:31.629 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (4m) 3m ago 16m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:38:31.629 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (9m) 3m ago 20m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:38:31.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.629+0000 7fde905fe700 1 -- 172.21.15.121:0/1909884627 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde7406cce0 msgr2=0x7fde7406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:31.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.629+0000 7fde905fe700 1 --2- 172.21.15.121:0/1909884627 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde7406cce0 0x7fde7406f1a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fde78005950 tx=0x7fde780058e0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.629+0000 7fde905fe700 1 -- 172.21.15.121:0/1909884627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde88100dc0 msgr2=0x7fde88196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:31.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.629+0000 7fde905fe700 1 --2- 172.21.15.121:0/1909884627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde88100dc0 0x7fde88196670 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fde84006010 tx=0x7fde8400be20 comp rx=0 tx=0).stop 2024-04-05T21:38:31.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.629+0000 7fde905fe700 1 -- 172.21.15.121:0/1909884627 shutdown_connections 2024-04-05T21:38:31.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.629+0000 7fde905fe700 1 --2- 172.21.15.121:0/1909884627 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fde7406cce0 0x7fde7406f1a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.629+0000 7fde905fe700 1 --2- 172.21.15.121:0/1909884627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde88100dc0 0x7fde88196670 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.629+0000 7fde905fe700 1 --2- 172.21.15.121:0/1909884627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fde88101770 0x7fde88196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.629+0000 7fde905fe700 1 -- 172.21.15.121:0/1909884627 >> 172.21.15.121:0/1909884627 conn(0x7fde880fc650 msgr2=0x7fde880fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:31.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.629+0000 7fde905fe700 1 -- 172.21.15.121:0/1909884627 shutdown_connections 2024-04-05T21:38:31.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.629+0000 7fde905fe700 1 -- 172.21.15.121:0/1909884627 wait complete. 2024-04-05T21:38:31.680 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:31 smithi188 ceph-mon[107793]: pgmap v368: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.730+0000 7f0aee3ce700 1 -- 172.21.15.121:0/1101504074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae8105a40 msgr2=0x7f0ae8105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.730+0000 7f0aee3ce700 1 --2- 172.21.15.121:0/1101504074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae8105a40 0x7f0ae8105e20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009910 tx=0x7f0ad8009c20 comp rx=0 tx=0).stop 2024-04-05T21:38:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.730+0000 7f0aee3ce700 1 -- 172.21.15.121:0/1101504074 shutdown_connections 2024-04-05T21:38:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.730+0000 7f0aee3ce700 1 --2- 172.21.15.121:0/1101504074 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ae80ffbd0 0x7f0ae8100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.730+0000 7f0aee3ce700 1 --2- 172.21.15.121:0/1101504074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae8105a40 0x7f0ae8105e20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.730+0000 7f0aee3ce700 1 -- 172.21.15.121:0/1101504074 >> 172.21.15.121:0/1101504074 conn(0x7f0ae80755a0 msgr2=0x7f0ae80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.730+0000 7f0aee3ce700 1 -- 172.21.15.121:0/1101504074 shutdown_connections 2024-04-05T21:38:31.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.730+0000 7f0aee3ce700 1 -- 172.21.15.121:0/1101504074 wait complete. 2024-04-05T21:38:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.730+0000 7f0aee3ce700 1 Processor -- start 2024-04-05T21:38:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.731+0000 7f0aee3ce700 1 -- start start 2024-04-05T21:38:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.731+0000 7f0aee3ce700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ae80ffbd0 0x7f0ae81974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.731+0000 7f0aee3ce700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae8105a40 0x7f0ae81979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.731+0000 7f0aee3ce700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae8198000 con 0x7f0ae8105a40 2024-04-05T21:38:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.731+0000 7f0aee3ce700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae8198170 con 0x7f0ae80ffbd0 2024-04-05T21:38:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.731+0000 7f0ae77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae8105a40 0x7f0ae81979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.731+0000 7f0ae77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae8105a40 0x7f0ae81979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56428/0 (socket says 172.21.15.121:56428) 2024-04-05T21:38:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.731+0000 7f0ae77fe700 1 -- 172.21.15.121:0/3054265449 learned_addr learned my addr 172.21.15.121:0/3054265449 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:38:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.731+0000 7f0ae7fff700 1 --2- 172.21.15.121:0/3054265449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ae80ffbd0 0x7f0ae81974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.731+0000 7f0ae77fe700 1 -- 172.21.15.121:0/3054265449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ae80ffbd0 msgr2=0x7f0ae81974a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.731+0000 7f0ae77fe700 1 --2- 172.21.15.121:0/3054265449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ae80ffbd0 0x7f0ae81974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:31.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.731+0000 7f0ae77fe700 1 -- 172.21.15.121:0/3054265449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad80095f0 con 0x7f0ae8105a40 2024-04-05T21:38:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.731+0000 7f0ae77fe700 1 --2- 172.21.15.121:0/3054265449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae8105a40 0x7f0ae81979e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00d8d0 tx=0x7f0adc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.732+0000 7f0ae57fa700 1 -- 172.21.15.121:0/3054265449 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0adc009940 con 0x7f0ae8105a40 2024-04-05T21:38:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.732+0000 7f0ae57fa700 1 -- 172.21.15.121:0/3054265449 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0adc010460 con 0x7f0ae8105a40 2024-04-05T21:38:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.732+0000 7f0aee3ce700 1 -- 172.21.15.121:0/3054265449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae81a0510 con 0x7f0ae8105a40 2024-04-05T21:38:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.732+0000 7f0ae57fa700 1 -- 172.21.15.121:0/3054265449 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0adc00f5f0 con 0x7f0ae8105a40 2024-04-05T21:38:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.732+0000 7f0aee3ce700 1 -- 172.21.15.121:0/3054265449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae81a0a00 con 0x7f0ae8105a40 2024-04-05T21:38:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.732+0000 7f0aee3ce700 1 -- 172.21.15.121:0/3054265449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae8108040 con 0x7f0ae8105a40 2024-04-05T21:38:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.733+0000 7f0ae57fa700 1 -- 172.21.15.121:0/3054265449 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0adc0105d0 con 0x7f0ae8105a40 2024-04-05T21:38:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.733+0000 7f0ae57fa700 1 --2- 172.21.15.121:0/3054265449 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ad006cce0 0x7f0ad006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.734+0000 7f0ae57fa700 1 -- 172.21.15.121:0/3054265449 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0adc08c4f0 con 0x7f0ae8105a40 2024-04-05T21:38:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.734+0000 7f0ae7fff700 1 --2- 172.21.15.121:0/3054265449 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ad006cce0 0x7f0ad006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:31.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.734+0000 7f0ae7fff700 1 --2- 172.21.15.121:0/3054265449 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ad006cce0 0x7f0ad006f1a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0ad800b5c0 tx=0x7f0ad8005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:31.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:31 smithi121 ceph-mon[145859]: pgmap v368: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:31.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:31.738+0000 7f0ae57fa700 1 -- 172.21.15.121:0/3054265449 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0adc0565e0 con 0x7f0ae8105a40 2024-04-05T21:38:32.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.009+0000 7f0aee3ce700 1 -- 172.21.15.121:0/3054265449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0ae804ea90 con 0x7f0ae8105a40 2024-04-05T21:38:32.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.009+0000 7f0ae57fa700 1 -- 172.21.15.121:0/3054265449 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f0adc010880 con 0x7f0ae8105a40 2024-04-05T21:38:32.010 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:38:32.011 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:38:32.012 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:38:32.012 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:38:32.012 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:38:32.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.012+0000 7f0aee3ce700 1 -- 172.21.15.121:0/3054265449 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ad006cce0 msgr2=0x7f0ad006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:32.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.012+0000 7f0aee3ce700 1 --2- 172.21.15.121:0/3054265449 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ad006cce0 0x7f0ad006f1a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0ad800b5c0 tx=0x7f0ad8005fb0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.012+0000 7f0aee3ce700 1 -- 172.21.15.121:0/3054265449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae8105a40 msgr2=0x7f0ae81979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:32.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.012+0000 7f0aee3ce700 1 --2- 172.21.15.121:0/3054265449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae8105a40 0x7f0ae81979e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00d8d0 tx=0x7f0adc00dc90 comp rx=0 tx=0).stop 2024-04-05T21:38:32.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.012+0000 7f0aee3ce700 1 -- 172.21.15.121:0/3054265449 shutdown_connections 2024-04-05T21:38:32.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.012+0000 7f0aee3ce700 1 --2- 172.21.15.121:0/3054265449 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ad006cce0 0x7f0ad006f1a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.012+0000 7f0aee3ce700 1 --2- 172.21.15.121:0/3054265449 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ae80ffbd0 0x7f0ae81974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.012+0000 7f0aee3ce700 1 --2- 172.21.15.121:0/3054265449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ae8105a40 0x7f0ae81979e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.012+0000 7f0aee3ce700 1 -- 172.21.15.121:0/3054265449 >> 172.21.15.121:0/3054265449 conn(0x7f0ae80755a0 msgr2=0x7f0ae80fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:32.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.012+0000 7f0aee3ce700 1 -- 172.21.15.121:0/3054265449 shutdown_connections 2024-04-05T21:38:32.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.012+0000 7f0aee3ce700 1 -- 172.21.15.121:0/3054265449 wait complete. 2024-04-05T21:38:32.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.115+0000 7f5d81e5e700 1 -- 172.21.15.121:0/2856471208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d7c100da0 msgr2=0x7f5d7c101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:32.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.115+0000 7f5d81e5e700 1 --2- 172.21.15.121:0/2856471208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d7c100da0 0x7f5d7c101180 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c009910 tx=0x7f5d6c009c20 comp rx=0 tx=0).stop 2024-04-05T21:38:32.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.115+0000 7f5d81e5e700 1 -- 172.21.15.121:0/2856471208 shutdown_connections 2024-04-05T21:38:32.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.115+0000 7f5d81e5e700 1 --2- 172.21.15.121:0/2856471208 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d7c101750 0x7f5d7c1057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.115+0000 7f5d81e5e700 1 --2- 172.21.15.121:0/2856471208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d7c100da0 0x7f5d7c101180 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.115+0000 7f5d81e5e700 1 -- 172.21.15.121:0/2856471208 >> 172.21.15.121:0/2856471208 conn(0x7f5d7c0fc630 msgr2=0x7f5d7c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:32.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.115+0000 7f5d81e5e700 1 -- 172.21.15.121:0/2856471208 shutdown_connections 2024-04-05T21:38:32.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.115+0000 7f5d81e5e700 1 -- 172.21.15.121:0/2856471208 wait complete. 2024-04-05T21:38:32.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.115+0000 7f5d81e5e700 1 Processor -- start 2024-04-05T21:38:32.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.115+0000 7f5d81e5e700 1 -- start start 2024-04-05T21:38:32.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d81e5e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d7c100da0 0x7f5d7c196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d81e5e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d7c101750 0x7f5d7c196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d81e5e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c197260 con 0x7f5d7c101750 2024-04-05T21:38:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d81e5e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c199f90 con 0x7f5d7c100da0 2024-04-05T21:38:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d7affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d7c101750 0x7f5d7c196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d7affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d7c101750 0x7f5d7c196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56454/0 (socket says 172.21.15.121:56454) 2024-04-05T21:38:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d7affd700 1 -- 172.21.15.121:0/370308901 learned_addr learned my addr 172.21.15.121:0/370308901 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:38:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d7b7fe700 1 --2- 172.21.15.121:0/370308901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d7c100da0 0x7f5d7c196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d7affd700 1 -- 172.21.15.121:0/370308901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d7c100da0 msgr2=0x7f5d7c196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d7affd700 1 --2- 172.21.15.121:0/370308901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d7c100da0 0x7f5d7c196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d7affd700 1 -- 172.21.15.121:0/370308901 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c0095f0 con 0x7f5d7c101750 2024-04-05T21:38:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d7affd700 1 --2- 172.21.15.121:0/370308901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d7c101750 0x7f5d7c196bd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000d900 tx=0x7f5d7000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d78ff9700 1 -- 172.21.15.121:0/370308901 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d70009900 con 0x7f5d7c101750 2024-04-05T21:38:32.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d78ff9700 1 -- 172.21.15.121:0/370308901 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d70010460 con 0x7f5d7c101750 2024-04-05T21:38:32.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d81e5e700 1 -- 172.21.15.121:0/370308901 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c19a270 con 0x7f5d7c101750 2024-04-05T21:38:32.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d78ff9700 1 -- 172.21.15.121:0/370308901 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5d7000f5f0 con 0x7f5d7c101750 2024-04-05T21:38:32.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.116+0000 7f5d81e5e700 1 -- 172.21.15.121:0/370308901 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c19a7c0 con 0x7f5d7c101750 2024-04-05T21:38:32.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.117+0000 7f5d81e5e700 1 -- 172.21.15.121:0/370308901 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c109140 con 0x7f5d7c101750 2024-04-05T21:38:32.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.118+0000 7f5d78ff9700 1 -- 172.21.15.121:0/370308901 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5d700105d0 con 0x7f5d7c101750 2024-04-05T21:38:32.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.118+0000 7f5d78ff9700 1 --2- 172.21.15.121:0/370308901 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5d6406cce0 0x7f5d6406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:32.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.118+0000 7f5d78ff9700 1 -- 172.21.15.121:0/370308901 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5d7008c0e0 con 0x7f5d7c101750 2024-04-05T21:38:32.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.118+0000 7f5d7b7fe700 1 --2- 172.21.15.121:0/370308901 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5d6406cce0 0x7f5d6406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:32.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.119+0000 7f5d7b7fe700 1 --2- 172.21.15.121:0/370308901 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5d6406cce0 0x7f5d6406f1a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c006010 tx=0x7f5d6c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:32.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.123+0000 7f5d78ff9700 1 -- 172.21.15.121:0/370308901 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5d70056150 con 0x7f5d7c101750 2024-04-05T21:38:32.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.358+0000 7f5d81e5e700 1 -- 172.21.15.121:0/370308901 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5d7c04ea90 con 0x7f5d7c101750 2024-04-05T21:38:32.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.358+0000 7f5d78ff9700 1 -- 172.21.15.121:0/370308901 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f5d70059770 con 0x7f5d7c101750 2024-04-05T21:38:32.359 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:38:32.359 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:38:32.359 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:38:32.359 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:38:32.360 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:38:32.361 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:38:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:38:32.362 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:38:32.362 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:38:32.362 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:38:32.362 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:38:32.362 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:38:32.362 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:38:32.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.360+0000 7f5d81e5e700 1 -- 172.21.15.121:0/370308901 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5d6406cce0 msgr2=0x7f5d6406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:32.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.360+0000 7f5d81e5e700 1 --2- 172.21.15.121:0/370308901 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5d6406cce0 0x7f5d6406f1a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c006010 tx=0x7f5d6c01a040 comp rx=0 tx=0).stop 2024-04-05T21:38:32.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.360+0000 7f5d81e5e700 1 -- 172.21.15.121:0/370308901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d7c101750 msgr2=0x7f5d7c196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:32.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.360+0000 7f5d81e5e700 1 --2- 172.21.15.121:0/370308901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d7c101750 0x7f5d7c196bd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000d900 tx=0x7f5d7000dc10 comp rx=0 tx=0).stop 2024-04-05T21:38:32.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.360+0000 7f5d81e5e700 1 -- 172.21.15.121:0/370308901 shutdown_connections 2024-04-05T21:38:32.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.360+0000 7f5d81e5e700 1 --2- 172.21.15.121:0/370308901 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5d6406cce0 0x7f5d6406f1a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.361+0000 7f5d81e5e700 1 --2- 172.21.15.121:0/370308901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5d7c100da0 0x7f5d7c196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.361+0000 7f5d81e5e700 1 --2- 172.21.15.121:0/370308901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d7c101750 0x7f5d7c196bd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.361+0000 7f5d81e5e700 1 -- 172.21.15.121:0/370308901 >> 172.21.15.121:0/370308901 conn(0x7f5d7c0fc630 msgr2=0x7f5d7c0fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:32.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.361+0000 7f5d81e5e700 1 -- 172.21.15.121:0/370308901 shutdown_connections 2024-04-05T21:38:32.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.361+0000 7f5d81e5e700 1 -- 172.21.15.121:0/370308901 wait complete. 2024-04-05T21:38:32.363 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:38:32.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.464+0000 7f4d102d5700 1 -- 172.21.15.121:0/3483919249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08105a40 msgr2=0x7f4d08105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:32.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.464+0000 7f4d102d5700 1 --2- 172.21.15.121:0/3483919249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08105a40 0x7f4d08105e20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4d04009910 tx=0x7f4d04009c20 comp rx=0 tx=0).stop 2024-04-05T21:38:32.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.465+0000 7f4d102d5700 1 -- 172.21.15.121:0/3483919249 shutdown_connections 2024-04-05T21:38:32.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.465+0000 7f4d102d5700 1 --2- 172.21.15.121:0/3483919249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d080ffbd0 0x7f4d08100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.465+0000 7f4d102d5700 1 --2- 172.21.15.121:0/3483919249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08105a40 0x7f4d08105e20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.465+0000 7f4d102d5700 1 -- 172.21.15.121:0/3483919249 >> 172.21.15.121:0/3483919249 conn(0x7f4d080755a0 msgr2=0x7f4d080759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:32.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.465+0000 7f4d102d5700 1 -- 172.21.15.121:0/3483919249 shutdown_connections 2024-04-05T21:38:32.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.465+0000 7f4d102d5700 1 -- 172.21.15.121:0/3483919249 wait complete. 2024-04-05T21:38:32.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.465+0000 7f4d102d5700 1 Processor -- start 2024-04-05T21:38:32.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.465+0000 7f4d102d5700 1 -- start start 2024-04-05T21:38:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.465+0000 7f4d102d5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d080ffbd0 0x7f4d08101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.465+0000 7f4d102d5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08105a40 0x7f4d08101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.465+0000 7f4d102d5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d08102120 con 0x7f4d08105a40 2024-04-05T21:38:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.465+0000 7f4d102d5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d08102290 con 0x7f4d080ffbd0 2024-04-05T21:38:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.466+0000 7f4d0d870700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08105a40 0x7f4d08101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.466+0000 7f4d0d870700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08105a40 0x7f4d08101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56468/0 (socket says 172.21.15.121:56468) 2024-04-05T21:38:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.466+0000 7f4d0d870700 1 -- 172.21.15.121:0/2380466854 learned_addr learned my addr 172.21.15.121:0/2380466854 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:38:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.466+0000 7f4d0e071700 1 --2- 172.21.15.121:0/2380466854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d080ffbd0 0x7f4d08101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.466+0000 7f4d0d870700 1 -- 172.21.15.121:0/2380466854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d080ffbd0 msgr2=0x7f4d08101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.466+0000 7f4d0d870700 1 --2- 172.21.15.121:0/2380466854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d080ffbd0 0x7f4d08101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.466+0000 7f4d0d870700 1 -- 172.21.15.121:0/2380466854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d040095f0 con 0x7f4d08105a40 2024-04-05T21:38:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.466+0000 7f4d0d870700 1 --2- 172.21.15.121:0/2380466854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08105a40 0x7f4d08101b50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4cf800d8d0 tx=0x7f4cf800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.466+0000 7f4cff7fe700 1 -- 172.21.15.121:0/2380466854 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cf8009940 con 0x7f4d08105a40 2024-04-05T21:38:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.466+0000 7f4cff7fe700 1 -- 172.21.15.121:0/2380466854 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4cf8010460 con 0x7f4d08105a40 2024-04-05T21:38:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.466+0000 7f4d102d5700 1 -- 172.21.15.121:0/2380466854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d081a2810 con 0x7f4d08105a40 2024-04-05T21:38:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.466+0000 7f4cff7fe700 1 -- 172.21.15.121:0/2380466854 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cf800f5f0 con 0x7f4d08105a40 2024-04-05T21:38:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.466+0000 7f4d102d5700 1 -- 172.21.15.121:0/2380466854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d081a2c80 con 0x7f4d08105a40 2024-04-05T21:38:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.467+0000 7f4d102d5700 1 -- 172.21.15.121:0/2380466854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d08109050 con 0x7f4d08105a40 2024-04-05T21:38:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.468+0000 7f4cff7fe700 1 -- 172.21.15.121:0/2380466854 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4cf80105d0 con 0x7f4d08105a40 2024-04-05T21:38:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.468+0000 7f4cff7fe700 1 --2- 172.21.15.121:0/2380466854 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4cf406cce0 0x7f4cf406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.468+0000 7f4cff7fe700 1 -- 172.21.15.121:0/2380466854 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4cf808c5e0 con 0x7f4d08105a40 2024-04-05T21:38:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.468+0000 7f4d0e071700 1 --2- 172.21.15.121:0/2380466854 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4cf406cce0 0x7f4cf406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.468+0000 7f4d0e071700 1 --2- 172.21.15.121:0/2380466854 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4cf406cce0 0x7f4cf406f1a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4d04006010 tx=0x7f4d0401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:32.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.473+0000 7f4cff7fe700 1 -- 172.21.15.121:0/2380466854 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4cf8056650 con 0x7f4d08105a40 2024-04-05T21:38:32.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.682+0000 7f4d102d5700 1 -- 172.21.15.121:0/2380466854 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d0802cce0 con 0x7f4cf406cce0 2024-04-05T21:38:32.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.683+0000 7f4cff7fe700 1 -- 172.21.15.121:0/2380466854 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4d0802cce0 con 0x7f4cf406cce0 2024-04-05T21:38:32.684 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:38:32.684 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:38:32.684 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:38:32.684 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:38:32.684 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:38:32.684 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:38:32.684 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:38:32.684 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:38:32.684 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:38:32.684 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:38:32.684 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:38:32.685 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:38:32.685 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:38:32.685 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:38:32.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.685+0000 7f4d102d5700 1 -- 172.21.15.121:0/2380466854 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4cf406cce0 msgr2=0x7f4cf406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:32.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.685+0000 7f4d102d5700 1 --2- 172.21.15.121:0/2380466854 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4cf406cce0 0x7f4cf406f1a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4d04006010 tx=0x7f4d0401a040 comp rx=0 tx=0).stop 2024-04-05T21:38:32.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.685+0000 7f4d102d5700 1 -- 172.21.15.121:0/2380466854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08105a40 msgr2=0x7f4d08101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:32.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.685+0000 7f4d102d5700 1 --2- 172.21.15.121:0/2380466854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08105a40 0x7f4d08101b50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4cf800d8d0 tx=0x7f4cf800dc90 comp rx=0 tx=0).stop 2024-04-05T21:38:32.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.685+0000 7f4d102d5700 1 -- 172.21.15.121:0/2380466854 shutdown_connections 2024-04-05T21:38:32.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.685+0000 7f4d102d5700 1 --2- 172.21.15.121:0/2380466854 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4cf406cce0 0x7f4cf406f1a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.685+0000 7f4d102d5700 1 --2- 172.21.15.121:0/2380466854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d080ffbd0 0x7f4d08101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.685+0000 7f4d102d5700 1 --2- 172.21.15.121:0/2380466854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d08105a40 0x7f4d08101b50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.685+0000 7f4d102d5700 1 -- 172.21.15.121:0/2380466854 >> 172.21.15.121:0/2380466854 conn(0x7f4d080755a0 msgr2=0x7f4d08068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:32.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.685+0000 7f4d102d5700 1 -- 172.21.15.121:0/2380466854 shutdown_connections 2024-04-05T21:38:32.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.685+0000 7f4d102d5700 1 -- 172.21.15.121:0/2380466854 wait complete. 2024-04-05T21:38:32.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:32 smithi121 ceph-mon[145859]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:32 smithi121 ceph-mon[145859]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:32 smithi121 ceph-mon[145859]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:32 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3054265449' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:38:32.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:32 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/370308901' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:38:32.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:32 smithi188 ceph-mon[107793]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:32.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:32 smithi188 ceph-mon[107793]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:32.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:32 smithi188 ceph-mon[107793]: from='client.34546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:32.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:32 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3054265449' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:38:32.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:32 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/370308901' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:38:32.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.788+0000 7f635aa69700 1 -- 172.21.15.121:0/3983350575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63540fef70 msgr2=0x7f63540ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:32.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.788+0000 7f635aa69700 1 --2- 172.21.15.121:0/3983350575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63540fef70 0x7f63540ff3f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6340009910 tx=0x7f6340009c20 comp rx=0 tx=0).stop 2024-04-05T21:38:32.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.788+0000 7f635aa69700 1 -- 172.21.15.121:0/3983350575 shutdown_connections 2024-04-05T21:38:32.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.788+0000 7f635aa69700 1 --2- 172.21.15.121:0/3983350575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63540fef70 0x7f63540ff3f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.788+0000 7f635aa69700 1 --2- 172.21.15.121:0/3983350575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6354105b50 0x7f6354105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.788+0000 7f635aa69700 1 -- 172.21.15.121:0/3983350575 >> 172.21.15.121:0/3983350575 conn(0x7f6354075e30 msgr2=0x7f63540fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:32.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.788+0000 7f635aa69700 1 -- 172.21.15.121:0/3983350575 shutdown_connections 2024-04-05T21:38:32.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.788+0000 7f635aa69700 1 -- 172.21.15.121:0/3983350575 wait complete. 2024-04-05T21:38:32.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.789+0000 7f635aa69700 1 Processor -- start 2024-04-05T21:38:32.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.789+0000 7f635aa69700 1 -- start start 2024-04-05T21:38:32.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.789+0000 7f635aa69700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63540fef70 0x7f6354196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:32.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.789+0000 7f635aa69700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6354105b50 0x7f6354196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:32.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.789+0000 7f635aa69700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63541972d0 con 0x7f63540fef70 2024-04-05T21:38:32.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.789+0000 7f635aa69700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635419a000 con 0x7f6354105b50 2024-04-05T21:38:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.789+0000 7f6358805700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63540fef70 0x7f6354196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.789+0000 7f6358805700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63540fef70 0x7f6354196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56482/0 (socket says 172.21.15.121:56482) 2024-04-05T21:38:32.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.789+0000 7f6358805700 1 -- 172.21.15.121:0/750349282 learned_addr learned my addr 172.21.15.121:0/750349282 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:38:32.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.789+0000 7f6353fff700 1 --2- 172.21.15.121:0/750349282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6354105b50 0x7f6354196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:32.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.789+0000 7f6358805700 1 -- 172.21.15.121:0/750349282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6354105b50 msgr2=0x7f6354196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:32.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.789+0000 7f6358805700 1 --2- 172.21.15.121:0/750349282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6354105b50 0x7f6354196c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:32.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.789+0000 7f6358805700 1 -- 172.21.15.121:0/750349282 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63400095f0 con 0x7f63540fef70 2024-04-05T21:38:32.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.790+0000 7f6358805700 1 --2- 172.21.15.121:0/750349282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63540fef70 0x7f6354196700 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f634800d900 tx=0x7f634800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:32.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.790+0000 7f6351ffb700 1 -- 172.21.15.121:0/750349282 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63480041f0 con 0x7f63540fef70 2024-04-05T21:38:32.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.790+0000 7f6351ffb700 1 -- 172.21.15.121:0/750349282 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6348004cd0 con 0x7f63540fef70 2024-04-05T21:38:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.790+0000 7f635aa69700 1 -- 172.21.15.121:0/750349282 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635419a2e0 con 0x7f63540fef70 2024-04-05T21:38:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.790+0000 7f6351ffb700 1 -- 172.21.15.121:0/750349282 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6348005660 con 0x7f63540fef70 2024-04-05T21:38:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.790+0000 7f635aa69700 1 -- 172.21.15.121:0/750349282 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635419a830 con 0x7f63540fef70 2024-04-05T21:38:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.791+0000 7f635aa69700 1 -- 172.21.15.121:0/750349282 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63541090d0 con 0x7f63540fef70 2024-04-05T21:38:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.792+0000 7f6351ffb700 1 -- 172.21.15.121:0/750349282 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6348004e40 con 0x7f63540fef70 2024-04-05T21:38:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.792+0000 7f6351ffb700 1 --2- 172.21.15.121:0/750349282 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f634406cce0 0x7f634406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:38:32.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.792+0000 7f6351ffb700 1 -- 172.21.15.121:0/750349282 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f634808c8f0 con 0x7f63540fef70 2024-04-05T21:38:32.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.792+0000 7f6353fff700 1 --2- 172.21.15.121:0/750349282 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f634406cce0 0x7f634406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:38:32.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.792+0000 7f6353fff700 1 --2- 172.21.15.121:0/750349282 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f634406cce0 0x7f634406f1a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6354197d20 tx=0x7f634001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:38:32.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:32.796+0000 7f6351ffb700 1 -- 172.21.15.121:0/750349282 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f63480569e0 con 0x7f63540fef70 2024-04-05T21:38:33.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:33.066+0000 7f635aa69700 1 -- 172.21.15.121:0/750349282 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f635404ea90 con 0x7f63540fef70 2024-04-05T21:38:33.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:33.066+0000 7f6351ffb700 1 -- 172.21.15.121:0/750349282 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f6348016090 con 0x7f63540fef70 2024-04-05T21:38:33.068 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:38:33.068 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:38:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:38:33.068 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:38:33.068 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:38:33.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:33.068+0000 7f635aa69700 1 -- 172.21.15.121:0/750349282 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f634406cce0 msgr2=0x7f634406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:33.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:33.068+0000 7f635aa69700 1 --2- 172.21.15.121:0/750349282 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f634406cce0 0x7f634406f1a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6354197d20 tx=0x7f634001a040 comp rx=0 tx=0).stop 2024-04-05T21:38:33.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:33.068+0000 7f635aa69700 1 -- 172.21.15.121:0/750349282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63540fef70 msgr2=0x7f6354196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:38:33.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:33.068+0000 7f635aa69700 1 --2- 172.21.15.121:0/750349282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63540fef70 0x7f6354196700 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f634800d900 tx=0x7f634800dc10 comp rx=0 tx=0).stop 2024-04-05T21:38:33.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:33.068+0000 7f635aa69700 1 -- 172.21.15.121:0/750349282 shutdown_connections 2024-04-05T21:38:33.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:33.068+0000 7f635aa69700 1 --2- 172.21.15.121:0/750349282 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f634406cce0 0x7f634406f1a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:33.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:33.068+0000 7f635aa69700 1 --2- 172.21.15.121:0/750349282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63540fef70 0x7f6354196700 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:33.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:33.068+0000 7f635aa69700 1 --2- 172.21.15.121:0/750349282 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6354105b50 0x7f6354196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:38:33.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:33.068+0000 7f635aa69700 1 -- 172.21.15.121:0/750349282 >> 172.21.15.121:0/750349282 conn(0x7f6354075e30 msgr2=0x7f63541045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:38:33.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:33.069+0000 7f635aa69700 1 -- 172.21.15.121:0/750349282 shutdown_connections 2024-04-05T21:38:33.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:38:33.069+0000 7f635aa69700 1 -- 172.21.15.121:0/750349282 wait complete. 2024-04-05T21:38:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:33 smithi121 ceph-mon[145859]: pgmap v369: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:33 smithi121 ceph-mon[145859]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:33 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/750349282' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:38:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:33 smithi188 ceph-mon[107793]: pgmap v369: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:33 smithi188 ceph-mon[107793]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:38:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:33 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/750349282' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:38:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:35 smithi121 ceph-mon[145859]: pgmap v370: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:35.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:35 smithi188 ceph-mon[107793]: pgmap v370: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:37 smithi121 ceph-mon[145859]: pgmap v371: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:37.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:37 smithi188 ceph-mon[107793]: pgmap v371: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:39.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:39 smithi121 ceph-mon[145859]: pgmap v372: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:39.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:39 smithi188 ceph-mon[107793]: pgmap v372: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:41.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:41 smithi121 ceph-mon[145859]: pgmap v373: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:41.772 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:41 smithi188 ceph-mon[107793]: pgmap v373: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:43 smithi121 ceph-mon[145859]: pgmap v374: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:43.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:43 smithi188 ceph-mon[107793]: pgmap v374: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:45 smithi121 ceph-mon[145859]: pgmap v375: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:38:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:45 smithi188 ceph-mon[107793]: pgmap v375: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:38:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:47 smithi121 ceph-mon[145859]: pgmap v376: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:47 smithi188 ceph-mon[107793]: pgmap v376: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:49 smithi121 ceph-mon[145859]: pgmap v377: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:49.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:49 smithi188 ceph-mon[107793]: pgmap v377: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:51 smithi121 ceph-mon[145859]: pgmap v378: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:51 smithi188 ceph-mon[107793]: pgmap v378: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:53 smithi121 ceph-mon[145859]: pgmap v379: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:53 smithi188 ceph-mon[107793]: pgmap v379: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:55.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:55 smithi121 ceph-mon[145859]: pgmap v380: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:55 smithi188 ceph-mon[107793]: pgmap v380: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:57 smithi121 ceph-mon[145859]: pgmap v381: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:57 smithi188 ceph-mon[107793]: pgmap v381: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:59.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:38:59 smithi121 ceph-mon[145859]: pgmap v382: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:38:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:38:59 smithi188 ceph-mon[107793]: pgmap v382: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:00.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:39:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:39:01.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:01 smithi121 ceph-mon[145859]: pgmap v383: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:01 smithi188 ceph-mon[107793]: pgmap v383: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:03.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.184+0000 7fc34bad3700 1 -- 172.21.15.121:0/1659850344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc344105a40 msgr2=0x7fc344105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:03.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.184+0000 7fc34bad3700 1 --2- 172.21.15.121:0/1659850344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc344105a40 0x7fc344105e20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc340009910 tx=0x7fc340009c20 comp rx=0 tx=0).stop 2024-04-05T21:39:03.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.184+0000 7fc34bad3700 1 -- 172.21.15.121:0/1659850344 shutdown_connections 2024-04-05T21:39:03.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.184+0000 7fc34bad3700 1 --2- 172.21.15.121:0/1659850344 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3440ffa40 0x7fc3440ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.184+0000 7fc34bad3700 1 --2- 172.21.15.121:0/1659850344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc344105a40 0x7fc344105e20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.184+0000 7fc34bad3700 1 -- 172.21.15.121:0/1659850344 >> 172.21.15.121:0/1659850344 conn(0x7fc3440fb570 msgr2=0x7fc3440fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:03.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.184+0000 7fc34bad3700 1 -- 172.21.15.121:0/1659850344 shutdown_connections 2024-04-05T21:39:03.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.184+0000 7fc34bad3700 1 -- 172.21.15.121:0/1659850344 wait complete. 2024-04-05T21:39:03.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.185+0000 7fc34bad3700 1 Processor -- start 2024-04-05T21:39:03.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.185+0000 7fc34bad3700 1 -- start start 2024-04-05T21:39:03.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.185+0000 7fc34bad3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3440ffa40 0x7fc344195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:03.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.185+0000 7fc34bad3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc344105a40 0x7fc344195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:03.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.185+0000 7fc34bad3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc344196170 con 0x7fc344105a40 2024-04-05T21:39:03.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.185+0000 7fc34bad3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc344199f00 con 0x7fc3440ffa40 2024-04-05T21:39:03.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.185+0000 7fc34906e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc344105a40 0x7fc344195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:03.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.185+0000 7fc34906e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc344105a40 0x7fc344195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33456/0 (socket says 172.21.15.121:33456) 2024-04-05T21:39:03.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.185+0000 7fc34906e700 1 -- 172.21.15.121:0/3520964592 learned_addr learned my addr 172.21.15.121:0/3520964592 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:03.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.185+0000 7fc34986f700 1 --2- 172.21.15.121:0/3520964592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3440ffa40 0x7fc344195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:03.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.186+0000 7fc34906e700 1 -- 172.21.15.121:0/3520964592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3440ffa40 msgr2=0x7fc344195550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:03.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.186+0000 7fc34906e700 1 --2- 172.21.15.121:0/3520964592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3440ffa40 0x7fc344195550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.186+0000 7fc34906e700 1 -- 172.21.15.121:0/3520964592 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3400095f0 con 0x7fc344105a40 2024-04-05T21:39:03.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.186+0000 7fc34906e700 1 --2- 172.21.15.121:0/3520964592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc344105a40 0x7fc344195a90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc33400d8d0 tx=0x7fc33400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:03.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.186+0000 7fc33affd700 1 -- 172.21.15.121:0/3520964592 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3340098a0 con 0x7fc344105a40 2024-04-05T21:39:03.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.186+0000 7fc33affd700 1 -- 172.21.15.121:0/3520964592 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc334010460 con 0x7fc344105a40 2024-04-05T21:39:03.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.186+0000 7fc34bad3700 1 -- 172.21.15.121:0/3520964592 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc34419a1e0 con 0x7fc344105a40 2024-04-05T21:39:03.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.186+0000 7fc33affd700 1 -- 172.21.15.121:0/3520964592 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc334009ea0 con 0x7fc344105a40 2024-04-05T21:39:03.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.186+0000 7fc34bad3700 1 -- 172.21.15.121:0/3520964592 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc34419a730 con 0x7fc344105a40 2024-04-05T21:39:03.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.187+0000 7fc34bad3700 1 -- 172.21.15.121:0/3520964592 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc34404ea90 con 0x7fc344105a40 2024-04-05T21:39:03.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.187+0000 7fc33affd700 1 -- 172.21.15.121:0/3520964592 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc3340105d0 con 0x7fc344105a40 2024-04-05T21:39:03.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.188+0000 7fc33affd700 1 --2- 172.21.15.121:0/3520964592 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc33006cce0 0x7fc33006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:03.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.188+0000 7fc33affd700 1 -- 172.21.15.121:0/3520964592 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc33408c520 con 0x7fc344105a40 2024-04-05T21:39:03.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.188+0000 7fc34986f700 1 --2- 172.21.15.121:0/3520964592 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc33006cce0 0x7fc33006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:03.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.188+0000 7fc34986f700 1 --2- 172.21.15.121:0/3520964592 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc33006cce0 0x7fc33006f1a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc340006010 tx=0x7fc34001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:03.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.192+0000 7fc33affd700 1 -- 172.21.15.121:0/3520964592 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc3340564f0 con 0x7fc344105a40 2024-04-05T21:39:03.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.402+0000 7fc34bad3700 1 -- 172.21.15.121:0/3520964592 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc34419ac10 con 0x7fc33006cce0 2024-04-05T21:39:03.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.403+0000 7fc33affd700 1 -- 172.21.15.121:0/3520964592 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc34419ac10 con 0x7fc33006cce0 2024-04-05T21:39:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.405+0000 7fc34bad3700 1 -- 172.21.15.121:0/3520964592 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc33006cce0 msgr2=0x7fc33006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.405+0000 7fc34bad3700 1 --2- 172.21.15.121:0/3520964592 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc33006cce0 0x7fc33006f1a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc340006010 tx=0x7fc34001a040 comp rx=0 tx=0).stop 2024-04-05T21:39:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.405+0000 7fc34bad3700 1 -- 172.21.15.121:0/3520964592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc344105a40 msgr2=0x7fc344195a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.405+0000 7fc34bad3700 1 --2- 172.21.15.121:0/3520964592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc344105a40 0x7fc344195a90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc33400d8d0 tx=0x7fc33400dbe0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.405+0000 7fc34bad3700 1 -- 172.21.15.121:0/3520964592 shutdown_connections 2024-04-05T21:39:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.405+0000 7fc34bad3700 1 --2- 172.21.15.121:0/3520964592 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc33006cce0 0x7fc33006f1a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.405+0000 7fc34bad3700 1 --2- 172.21.15.121:0/3520964592 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3440ffa40 0x7fc344195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.405+0000 7fc34bad3700 1 --2- 172.21.15.121:0/3520964592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc344105a40 0x7fc344195a90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.405+0000 7fc34bad3700 1 -- 172.21.15.121:0/3520964592 >> 172.21.15.121:0/3520964592 conn(0x7fc3440fb570 msgr2=0x7fc3440fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.405+0000 7fc34bad3700 1 -- 172.21.15.121:0/3520964592 shutdown_connections 2024-04-05T21:39:03.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.405+0000 7fc34bad3700 1 -- 172.21.15.121:0/3520964592 wait complete. 2024-04-05T21:39:03.416 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:39:03.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.509+0000 7f574364e700 1 -- 172.21.15.121:0/55182621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c0ffbd0 msgr2=0x7f573c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:03.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.509+0000 7f574364e700 1 --2- 172.21.15.121:0/55182621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c0ffbd0 0x7f573c100050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f572c009910 tx=0x7f572c009c20 comp rx=0 tx=0).stop 2024-04-05T21:39:03.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.509+0000 7f574364e700 1 -- 172.21.15.121:0/55182621 shutdown_connections 2024-04-05T21:39:03.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.509+0000 7f574364e700 1 --2- 172.21.15.121:0/55182621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c0ffbd0 0x7f573c100050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.509+0000 7f574364e700 1 --2- 172.21.15.121:0/55182621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f573c105a40 0x7f573c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.509+0000 7f574364e700 1 -- 172.21.15.121:0/55182621 >> 172.21.15.121:0/55182621 conn(0x7f573c0755a0 msgr2=0x7f573c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:03.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.509+0000 7f574364e700 1 -- 172.21.15.121:0/55182621 shutdown_connections 2024-04-05T21:39:03.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.509+0000 7f574364e700 1 -- 172.21.15.121:0/55182621 wait complete. 2024-04-05T21:39:03.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.509+0000 7f574364e700 1 Processor -- start 2024-04-05T21:39:03.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.510+0000 7f574364e700 1 -- start start 2024-04-05T21:39:03.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.510+0000 7f574364e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c0ffbd0 0x7f573c195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:03.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.510+0000 7f574364e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f573c105a40 0x7f573c195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:03.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.510+0000 7f574364e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c196160 con 0x7f573c0ffbd0 2024-04-05T21:39:03.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.510+0000 7f574364e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c199ef0 con 0x7f573c105a40 2024-04-05T21:39:03.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.510+0000 7f57413ea700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c0ffbd0 0x7f573c195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:03.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.510+0000 7f57413ea700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c0ffbd0 0x7f573c195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33470/0 (socket says 172.21.15.121:33470) 2024-04-05T21:39:03.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.510+0000 7f57413ea700 1 -- 172.21.15.121:0/783114466 learned_addr learned my addr 172.21.15.121:0/783114466 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:03.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.510+0000 7f5740be9700 1 --2- 172.21.15.121:0/783114466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f573c105a40 0x7f573c195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:03.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.510+0000 7f57413ea700 1 -- 172.21.15.121:0/783114466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f573c105a40 msgr2=0x7f573c195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:03.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.510+0000 7f57413ea700 1 --2- 172.21.15.121:0/783114466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f573c105a40 0x7f573c195a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.510+0000 7f57413ea700 1 -- 172.21.15.121:0/783114466 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572c0095f0 con 0x7f573c0ffbd0 2024-04-05T21:39:03.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.510+0000 7f57413ea700 1 --2- 172.21.15.121:0/783114466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c0ffbd0 0x7f573c195540 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5738009970 tx=0x7f5738009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:03.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.511+0000 7f57327fc700 1 -- 172.21.15.121:0/783114466 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5738012070 con 0x7f573c0ffbd0 2024-04-05T21:39:03.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.511+0000 7f57327fc700 1 -- 172.21.15.121:0/783114466 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5738007ad0 con 0x7f573c0ffbd0 2024-04-05T21:39:03.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.511+0000 7f574364e700 1 -- 172.21.15.121:0/783114466 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c19a170 con 0x7f573c0ffbd0 2024-04-05T21:39:03.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.511+0000 7f57327fc700 1 -- 172.21.15.121:0/783114466 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5738017c00 con 0x7f573c0ffbd0 2024-04-05T21:39:03.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.511+0000 7f574364e700 1 -- 172.21.15.121:0/783114466 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c19a6c0 con 0x7f573c0ffbd0 2024-04-05T21:39:03.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.511+0000 7f574364e700 1 -- 172.21.15.121:0/783114466 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c04ea90 con 0x7f573c0ffbd0 2024-04-05T21:39:03.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.512+0000 7f57327fc700 1 -- 172.21.15.121:0/783114466 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5738007c40 con 0x7f573c0ffbd0 2024-04-05T21:39:03.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.512+0000 7f57327fc700 1 --2- 172.21.15.121:0/783114466 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f572806cce0 0x7f572806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:03.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.512+0000 7f57327fc700 1 -- 172.21.15.121:0/783114466 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f573808dbc0 con 0x7f573c0ffbd0 2024-04-05T21:39:03.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.512+0000 7f5740be9700 1 --2- 172.21.15.121:0/783114466 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f572806cce0 0x7f572806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:03.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.513+0000 7f5740be9700 1 --2- 172.21.15.121:0/783114466 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f572806cce0 0x7f572806f1a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f573c196b60 tx=0x7f572c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:03.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.517+0000 7f57327fc700 1 -- 172.21.15.121:0/783114466 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5738057ce0 con 0x7f573c0ffbd0 2024-04-05T21:39:03.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.737+0000 7f574364e700 1 -- 172.21.15.121:0/783114466 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f573c19ac20 con 0x7f572806cce0 2024-04-05T21:39:03.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:03 smithi121 ceph-mon[145859]: pgmap v384: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:03.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.738+0000 7f57327fc700 1 -- 172.21.15.121:0/783114466 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f573c19ac20 con 0x7f572806cce0 2024-04-05T21:39:03.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.741+0000 7f574364e700 1 -- 172.21.15.121:0/783114466 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f572806cce0 msgr2=0x7f572806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:03.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.741+0000 7f574364e700 1 --2- 172.21.15.121:0/783114466 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f572806cce0 0x7f572806f1a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f573c196b60 tx=0x7f572c01a040 comp rx=0 tx=0).stop 2024-04-05T21:39:03.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.741+0000 7f574364e700 1 -- 172.21.15.121:0/783114466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c0ffbd0 msgr2=0x7f573c195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:03.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.741+0000 7f574364e700 1 --2- 172.21.15.121:0/783114466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c0ffbd0 0x7f573c195540 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5738009970 tx=0x7f5738009d30 comp rx=0 tx=0).stop 2024-04-05T21:39:03.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.741+0000 7f574364e700 1 -- 172.21.15.121:0/783114466 shutdown_connections 2024-04-05T21:39:03.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.741+0000 7f574364e700 1 --2- 172.21.15.121:0/783114466 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f572806cce0 0x7f572806f1a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.741+0000 7f574364e700 1 --2- 172.21.15.121:0/783114466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f573c0ffbd0 0x7f573c195540 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.741+0000 7f574364e700 1 --2- 172.21.15.121:0/783114466 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f573c105a40 0x7f573c195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.741+0000 7f574364e700 1 -- 172.21.15.121:0/783114466 >> 172.21.15.121:0/783114466 conn(0x7f573c0755a0 msgr2=0x7f573c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:03.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.741+0000 7f574364e700 1 -- 172.21.15.121:0/783114466 shutdown_connections 2024-04-05T21:39:03.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.741+0000 7f574364e700 1 -- 172.21.15.121:0/783114466 wait complete. 2024-04-05T21:39:03.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:03 smithi188 ceph-mon[107793]: pgmap v384: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:03.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.844+0000 7f0593fa2700 1 -- 172.21.15.121:0/739686248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f058c0ffbd0 msgr2=0x7f058c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:03.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.844+0000 7f0593fa2700 1 --2- 172.21.15.121:0/739686248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f058c0ffbd0 0x7f058c100050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0588009910 tx=0x7f0588009c20 comp rx=0 tx=0).stop 2024-04-05T21:39:03.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.844+0000 7f0593fa2700 1 -- 172.21.15.121:0/739686248 shutdown_connections 2024-04-05T21:39:03.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.844+0000 7f0593fa2700 1 --2- 172.21.15.121:0/739686248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f058c0ffbd0 0x7f058c100050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.844+0000 7f0593fa2700 1 --2- 172.21.15.121:0/739686248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f058c105a40 0x7f058c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.844+0000 7f0593fa2700 1 -- 172.21.15.121:0/739686248 >> 172.21.15.121:0/739686248 conn(0x7f058c0755a0 msgr2=0x7f058c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:03.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.844+0000 7f0593fa2700 1 -- 172.21.15.121:0/739686248 shutdown_connections 2024-04-05T21:39:03.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.844+0000 7f0593fa2700 1 -- 172.21.15.121:0/739686248 wait complete. 2024-04-05T21:39:03.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f0593fa2700 1 Processor -- start 2024-04-05T21:39:03.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f0593fa2700 1 -- start start 2024-04-05T21:39:03.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f0593fa2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f058c0ffbd0 0x7f058c1974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:03.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f0593fa2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f058c105a40 0x7f058c1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:03.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f0593fa2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c198010 con 0x7f058c105a40 2024-04-05T21:39:03.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f0593fa2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c198150 con 0x7f058c0ffbd0 2024-04-05T21:39:03.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f059153d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f058c105a40 0x7f058c1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:03.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f059153d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f058c105a40 0x7f058c1979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33494/0 (socket says 172.21.15.121:33494) 2024-04-05T21:39:03.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f059153d700 1 -- 172.21.15.121:0/2645838608 learned_addr learned my addr 172.21.15.121:0/2645838608 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:03.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f0591d3e700 1 --2- 172.21.15.121:0/2645838608 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f058c0ffbd0 0x7f058c1974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:03.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f059153d700 1 -- 172.21.15.121:0/2645838608 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f058c0ffbd0 msgr2=0x7f058c1974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:03.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f059153d700 1 --2- 172.21.15.121:0/2645838608 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f058c0ffbd0 0x7f058c1974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:03.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f059153d700 1 -- 172.21.15.121:0/2645838608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05880095f0 con 0x7f058c105a40 2024-04-05T21:39:03.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.845+0000 7f059153d700 1 --2- 172.21.15.121:0/2645838608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f058c105a40 0x7f058c1979f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0588006010 tx=0x7f058800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:03.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.846+0000 7f057effd700 1 -- 172.21.15.121:0/2645838608 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f058801d070 con 0x7f058c105a40 2024-04-05T21:39:03.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.846+0000 7f0593fa2700 1 -- 172.21.15.121:0/2645838608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058c1a0520 con 0x7f058c105a40 2024-04-05T21:39:03.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.846+0000 7f057effd700 1 -- 172.21.15.121:0/2645838608 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0588005710 con 0x7f058c105a40 2024-04-05T21:39:03.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.846+0000 7f057effd700 1 -- 172.21.15.121:0/2645838608 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f058800fe20 con 0x7f058c105a40 2024-04-05T21:39:03.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.846+0000 7f0593fa2700 1 -- 172.21.15.121:0/2645838608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058c1a0930 con 0x7f058c105a40 2024-04-05T21:39:03.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.847+0000 7f0593fa2700 1 -- 172.21.15.121:0/2645838608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058c107fd0 con 0x7f058c105a40 2024-04-05T21:39:03.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.847+0000 7f057effd700 1 -- 172.21.15.121:0/2645838608 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0588003a60 con 0x7f058c105a40 2024-04-05T21:39:03.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.847+0000 7f057effd700 1 --2- 172.21.15.121:0/2645838608 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f057806cce0 0x7f057806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:03.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.847+0000 7f057effd700 1 -- 172.21.15.121:0/2645838608 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f058808d6f0 con 0x7f058c105a40 2024-04-05T21:39:03.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.848+0000 7f0591d3e700 1 --2- 172.21.15.121:0/2645838608 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f057806cce0 0x7f057806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:03.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.848+0000 7f0591d3e700 1 --2- 172.21.15.121:0/2645838608 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f057806cce0 0x7f057806f1a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0580005950 tx=0x7f058000b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:03.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:03.852+0000 7f057effd700 1 -- 172.21.15.121:0/2645838608 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0588057760 con 0x7f058c105a40 2024-04-05T21:39:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.054+0000 7f0593fa2700 1 -- 172.21.15.121:0/2645838608 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f058c02cd30 con 0x7f057806cce0 2024-04-05T21:39:04.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.064+0000 7f057effd700 1 -- 172.21.15.121:0/2645838608 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f058c02cd30 con 0x7f057806cce0 2024-04-05T21:39:04.065 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:39:04.065 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (9m) 3m ago 21m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:39:04.065 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (21m) 3m ago 21m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:39:04.065 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (20m) 3m ago 20m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:39:04.065 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (8m) 3m ago 21m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:39:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (7m) 3m ago 20m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:39:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (9m) 3m ago 21m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:39:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (4m) 3m ago 15m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:39:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (3m) 3m ago 15m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:39:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (3m) 3m ago 15m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:39:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (15m) 3m ago 15m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:39:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (11m) 3m ago 22m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:39:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (11m) 3m ago 20m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:39:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (8m) 3m ago 23m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:39:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (8m) 3m ago 20m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:39:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (10m) 3m ago 21m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:39:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (10m) 3m ago 20m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:39:04.066 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (7m) 3m ago 19m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:39:04.067 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (6m) 3m ago 18m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:39:04.067 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (6m) 3m ago 18m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:39:04.067 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (5m) 3m ago 17m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:39:04.067 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (5m) 3m ago 17m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:39:04.067 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (4m) 3m ago 16m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:39:04.067 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (10m) 3m ago 20m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:39:04.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.066+0000 7f0593fa2700 1 -- 172.21.15.121:0/2645838608 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f057806cce0 msgr2=0x7f057806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:04.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.066+0000 7f0593fa2700 1 --2- 172.21.15.121:0/2645838608 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f057806cce0 0x7f057806f1a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0580005950 tx=0x7f058000b410 comp rx=0 tx=0).stop 2024-04-05T21:39:04.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.066+0000 7f0593fa2700 1 -- 172.21.15.121:0/2645838608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f058c105a40 msgr2=0x7f058c1979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:04.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.066+0000 7f0593fa2700 1 --2- 172.21.15.121:0/2645838608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f058c105a40 0x7f058c1979f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0588006010 tx=0x7f058800bd70 comp rx=0 tx=0).stop 2024-04-05T21:39:04.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.067+0000 7f0593fa2700 1 -- 172.21.15.121:0/2645838608 shutdown_connections 2024-04-05T21:39:04.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.067+0000 7f0593fa2700 1 --2- 172.21.15.121:0/2645838608 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f057806cce0 0x7f057806f1a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.067+0000 7f0593fa2700 1 --2- 172.21.15.121:0/2645838608 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f058c0ffbd0 0x7f058c1974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.067+0000 7f0593fa2700 1 --2- 172.21.15.121:0/2645838608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f058c105a40 0x7f058c1979f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.067+0000 7f0593fa2700 1 -- 172.21.15.121:0/2645838608 >> 172.21.15.121:0/2645838608 conn(0x7f058c0755a0 msgr2=0x7f058c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:04.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.067+0000 7f0593fa2700 1 -- 172.21.15.121:0/2645838608 shutdown_connections 2024-04-05T21:39:04.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.067+0000 7f0593fa2700 1 -- 172.21.15.121:0/2645838608 wait complete. 2024-04-05T21:39:04.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.167+0000 7fe158949700 1 -- 172.21.15.121:0/275839626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1540fef70 msgr2=0x7fe1540ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:04.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.168+0000 7fe158949700 1 --2- 172.21.15.121:0/275839626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1540fef70 0x7fe1540ff3f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe148009910 tx=0x7fe148009c20 comp rx=0 tx=0).stop 2024-04-05T21:39:04.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.168+0000 7fe158949700 1 -- 172.21.15.121:0/275839626 shutdown_connections 2024-04-05T21:39:04.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.168+0000 7fe158949700 1 --2- 172.21.15.121:0/275839626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1540fef70 0x7fe1540ff3f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.168+0000 7fe158949700 1 --2- 172.21.15.121:0/275839626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe154105b50 0x7fe154105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.168+0000 7fe158949700 1 -- 172.21.15.121:0/275839626 >> 172.21.15.121:0/275839626 conn(0x7fe154075e30 msgr2=0x7fe1540fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.168+0000 7fe158949700 1 -- 172.21.15.121:0/275839626 shutdown_connections 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.168+0000 7fe158949700 1 -- 172.21.15.121:0/275839626 wait complete. 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.168+0000 7fe158949700 1 Processor -- start 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.168+0000 7fe158949700 1 -- start start 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe158949700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1540fef70 0x7fe154100150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe158949700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe154105b50 0x7fe154100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe158949700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe154100bd0 con 0x7fe1540fef70 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe158949700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe154100d10 con 0x7fe154105b50 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe15259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1540fef70 0x7fe154100150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe15259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1540fef70 0x7fe154100150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33512/0 (socket says 172.21.15.121:33512) 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe15259c700 1 -- 172.21.15.121:0/1551089096 learned_addr learned my addr 172.21.15.121:0/1551089096 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:04.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe151d9b700 1 --2- 172.21.15.121:0/1551089096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe154105b50 0x7fe154100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:04.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe15259c700 1 -- 172.21.15.121:0/1551089096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe154105b50 msgr2=0x7fe154100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:04.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe15259c700 1 --2- 172.21.15.121:0/1551089096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe154105b50 0x7fe154100690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe15259c700 1 -- 172.21.15.121:0/1551089096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1480095f0 con 0x7fe1540fef70 2024-04-05T21:39:04.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe15259c700 1 --2- 172.21.15.121:0/1551089096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1540fef70 0x7fe154100150 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe14400c8a0 tx=0x7fe14400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:04.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe1437fe700 1 -- 172.21.15.121:0/1551089096 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe14400cea0 con 0x7fe1540fef70 2024-04-05T21:39:04.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe1437fe700 1 -- 172.21.15.121:0/1551089096 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe144004830 con 0x7fe1540fef70 2024-04-05T21:39:04.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe158949700 1 -- 172.21.15.121:0/1551089096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1541a28d0 con 0x7fe1540fef70 2024-04-05T21:39:04.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe1437fe700 1 -- 172.21.15.121:0/1551089096 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe144005500 con 0x7fe1540fef70 2024-04-05T21:39:04.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.169+0000 7fe158949700 1 -- 172.21.15.121:0/1551089096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1541a2ce0 con 0x7fe1540fef70 2024-04-05T21:39:04.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.171+0000 7fe158949700 1 -- 172.21.15.121:0/1551089096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1541090d0 con 0x7fe1540fef70 2024-04-05T21:39:04.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.171+0000 7fe1437fe700 1 -- 172.21.15.121:0/1551089096 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe14400f450 con 0x7fe1540fef70 2024-04-05T21:39:04.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.171+0000 7fe1437fe700 1 --2- 172.21.15.121:0/1551089096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe13c06cce0 0x7fe13c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:04.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.171+0000 7fe151d9b700 1 --2- 172.21.15.121:0/1551089096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe13c06cce0 0x7fe13c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:04.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.171+0000 7fe1437fe700 1 -- 172.21.15.121:0/1551089096 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe14408c1a0 con 0x7fe1540fef70 2024-04-05T21:39:04.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.172+0000 7fe151d9b700 1 --2- 172.21.15.121:0/1551089096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe13c06cce0 0x7fe13c06f1a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe14800b5c0 tx=0x7fe148005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:04.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.176+0000 7fe1437fe700 1 -- 172.21.15.121:0/1551089096 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe144056300 con 0x7fe1540fef70 2024-04-05T21:39:04.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.449+0000 7fe158949700 1 -- 172.21.15.121:0/1551089096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe15402ccc0 con 0x7fe1540fef70 2024-04-05T21:39:04.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.450+0000 7fe1437fe700 1 -- 172.21.15.121:0/1551089096 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7fe144059920 con 0x7fe1540fef70 2024-04-05T21:39:04.451 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:39:04.451 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:39:04.451 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:39:04.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:39:04.451 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:39:04.451 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:39:04.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:39:04.451 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:39:04.451 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:39:04.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:39:04.451 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:39:04.451 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:39:04.451 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:39:04.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:39:04.452 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:39:04.452 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:39:04.452 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:39:04.452 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:39:04.452 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:39:04.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.452+0000 7fe158949700 1 -- 172.21.15.121:0/1551089096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe13c06cce0 msgr2=0x7fe13c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:04.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.452+0000 7fe158949700 1 --2- 172.21.15.121:0/1551089096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe13c06cce0 0x7fe13c06f1a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe14800b5c0 tx=0x7fe148005fd0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.452+0000 7fe158949700 1 -- 172.21.15.121:0/1551089096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1540fef70 msgr2=0x7fe154100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:04.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.452+0000 7fe158949700 1 --2- 172.21.15.121:0/1551089096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1540fef70 0x7fe154100150 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe14400c8a0 tx=0x7fe14400cc60 comp rx=0 tx=0).stop 2024-04-05T21:39:04.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.452+0000 7fe158949700 1 -- 172.21.15.121:0/1551089096 shutdown_connections 2024-04-05T21:39:04.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.452+0000 7fe158949700 1 --2- 172.21.15.121:0/1551089096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe13c06cce0 0x7fe13c06f1a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.452+0000 7fe158949700 1 --2- 172.21.15.121:0/1551089096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1540fef70 0x7fe154100150 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.452+0000 7fe158949700 1 --2- 172.21.15.121:0/1551089096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe154105b50 0x7fe154100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.452+0000 7fe158949700 1 -- 172.21.15.121:0/1551089096 >> 172.21.15.121:0/1551089096 conn(0x7fe154075e30 msgr2=0x7fe1541045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:04.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.452+0000 7fe158949700 1 -- 172.21.15.121:0/1551089096 shutdown_connections 2024-04-05T21:39:04.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.452+0000 7fe158949700 1 -- 172.21.15.121:0/1551089096 wait complete. 2024-04-05T21:39:04.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.554+0000 7fb0244db700 1 -- 172.21.15.121:0/3068524400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb01c100dc0 msgr2=0x7fb01c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:04.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.554+0000 7fb0244db700 1 --2- 172.21.15.121:0/3068524400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb01c100dc0 0x7fb01c1011a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb018009910 tx=0x7fb018009c20 comp rx=0 tx=0).stop 2024-04-05T21:39:04.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.555+0000 7fb0244db700 1 -- 172.21.15.121:0/3068524400 shutdown_connections 2024-04-05T21:39:04.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.555+0000 7fb0244db700 1 --2- 172.21.15.121:0/3068524400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb01c101770 0x7fb01c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.555+0000 7fb0244db700 1 --2- 172.21.15.121:0/3068524400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb01c100dc0 0x7fb01c1011a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.555+0000 7fb0244db700 1 -- 172.21.15.121:0/3068524400 >> 172.21.15.121:0/3068524400 conn(0x7fb01c0fc670 msgr2=0x7fb01c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.555+0000 7fb0244db700 1 -- 172.21.15.121:0/3068524400 shutdown_connections 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.555+0000 7fb0244db700 1 -- 172.21.15.121:0/3068524400 wait complete. 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.555+0000 7fb0244db700 1 Processor -- start 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.555+0000 7fb0244db700 1 -- start start 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.555+0000 7fb0244db700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb01c100dc0 0x7fb01c196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.555+0000 7fb0244db700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb01c101770 0x7fb01c196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.555+0000 7fb0244db700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c1972e0 con 0x7fb01c101770 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.555+0000 7fb0244db700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c19a010 con 0x7fb01c100dc0 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.556+0000 7fb021a76700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb01c101770 0x7fb01c196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.556+0000 7fb021a76700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb01c101770 0x7fb01c196c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49358/0 (socket says 172.21.15.121:49358) 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.556+0000 7fb021a76700 1 -- 172.21.15.121:0/4273648057 learned_addr learned my addr 172.21.15.121:0/4273648057 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.556+0000 7fb022277700 1 --2- 172.21.15.121:0/4273648057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb01c100dc0 0x7fb01c196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.556+0000 7fb021a76700 1 -- 172.21.15.121:0/4273648057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb01c100dc0 msgr2=0x7fb01c196710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:04.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.556+0000 7fb021a76700 1 --2- 172.21.15.121:0/4273648057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb01c100dc0 0x7fb01c196710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.556+0000 7fb021a76700 1 -- 172.21.15.121:0/4273648057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180095f0 con 0x7fb01c101770 2024-04-05T21:39:04.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.556+0000 7fb021a76700 1 --2- 172.21.15.121:0/4273648057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb01c101770 0x7fb01c196c50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00d900 tx=0x7fb00c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:04.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.556+0000 7fb0137fe700 1 -- 172.21.15.121:0/4273648057 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb00c009900 con 0x7fb01c101770 2024-04-05T21:39:04.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.556+0000 7fb0244db700 1 -- 172.21.15.121:0/4273648057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01c19a2f0 con 0x7fb01c101770 2024-04-05T21:39:04.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.556+0000 7fb0137fe700 1 -- 172.21.15.121:0/4273648057 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb00c010460 con 0x7fb01c101770 2024-04-05T21:39:04.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.556+0000 7fb0137fe700 1 -- 172.21.15.121:0/4273648057 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb00c00f5f0 con 0x7fb01c101770 2024-04-05T21:39:04.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.556+0000 7fb0244db700 1 -- 172.21.15.121:0/4273648057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01c19a840 con 0x7fb01c101770 2024-04-05T21:39:04.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.557+0000 7fb0244db700 1 -- 172.21.15.121:0/4273648057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01c109160 con 0x7fb01c101770 2024-04-05T21:39:04.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.558+0000 7fb0137fe700 1 -- 172.21.15.121:0/4273648057 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb00c0105d0 con 0x7fb01c101770 2024-04-05T21:39:04.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.558+0000 7fb0137fe700 1 --2- 172.21.15.121:0/4273648057 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb00806cce0 0x7fb00806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:04.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.558+0000 7fb0137fe700 1 -- 172.21.15.121:0/4273648057 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb00c08c3c0 con 0x7fb01c101770 2024-04-05T21:39:04.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.558+0000 7fb022277700 1 --2- 172.21.15.121:0/4273648057 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb00806cce0 0x7fb00806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:04.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.558+0000 7fb022277700 1 --2- 172.21.15.121:0/4273648057 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb00806cce0 0x7fb00806f1a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb01800b5c0 tx=0x7fb018005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:04.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.562+0000 7fb0137fe700 1 -- 172.21.15.121:0/4273648057 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb00c056430 con 0x7fb01c101770 2024-04-05T21:39:04.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:04 smithi121 ceph-mon[145859]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:04.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:04 smithi121 ceph-mon[145859]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:04.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:04 smithi188 ceph-mon[107793]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:04.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:04 smithi188 ceph-mon[107793]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:04.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.795+0000 7fb0244db700 1 -- 172.21.15.121:0/4273648057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb01c19aa60 con 0x7fb01c101770 2024-04-05T21:39:04.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.795+0000 7fb0137fe700 1 -- 172.21.15.121:0/4273648057 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fb00c059a50 con 0x7fb01c101770 2024-04-05T21:39:04.796 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:39:04.796 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:39:04.796 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:39:04.796 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:39:04.796 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:39:04.796 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:39:04.796 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:39:04.796 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:39:04.796 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:39:04.796 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:39:04.796 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:39:04.797 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:39:04.798 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:39:04.799 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:39:04.799 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:39:04.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.797+0000 7fb0244db700 1 -- 172.21.15.121:0/4273648057 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb00806cce0 msgr2=0x7fb00806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:04.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.797+0000 7fb0244db700 1 --2- 172.21.15.121:0/4273648057 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb00806cce0 0x7fb00806f1a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb01800b5c0 tx=0x7fb018005fd0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.797+0000 7fb0244db700 1 -- 172.21.15.121:0/4273648057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb01c101770 msgr2=0x7fb01c196c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:04.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.797+0000 7fb0244db700 1 --2- 172.21.15.121:0/4273648057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb01c101770 0x7fb01c196c50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00d900 tx=0x7fb00c00dcc0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.797+0000 7fb0244db700 1 -- 172.21.15.121:0/4273648057 shutdown_connections 2024-04-05T21:39:04.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.797+0000 7fb0244db700 1 --2- 172.21.15.121:0/4273648057 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb00806cce0 0x7fb00806f1a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.797+0000 7fb0244db700 1 --2- 172.21.15.121:0/4273648057 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb01c100dc0 0x7fb01c196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.797+0000 7fb0244db700 1 --2- 172.21.15.121:0/4273648057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb01c101770 0x7fb01c196c50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.797+0000 7fb0244db700 1 -- 172.21.15.121:0/4273648057 >> 172.21.15.121:0/4273648057 conn(0x7fb01c0fc670 msgr2=0x7fb01c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:04.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.797+0000 7fb0244db700 1 -- 172.21.15.121:0/4273648057 shutdown_connections 2024-04-05T21:39:04.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.797+0000 7fb0244db700 1 -- 172.21.15.121:0/4273648057 wait complete. 2024-04-05T21:39:04.800 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:39:04.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.900+0000 7f722b6ae700 1 -- 172.21.15.121:0/3404624994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7224105a40 msgr2=0x7f7224105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:04.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.900+0000 7f722b6ae700 1 --2- 172.21.15.121:0/3404624994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7224105a40 0x7f7224105e20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7218009910 tx=0x7f7218009c20 comp rx=0 tx=0).stop 2024-04-05T21:39:04.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.900+0000 7f722b6ae700 1 -- 172.21.15.121:0/3404624994 shutdown_connections 2024-04-05T21:39:04.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.900+0000 7f722b6ae700 1 --2- 172.21.15.121:0/3404624994 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f72240ffbd0 0x7f7224100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.900+0000 7f722b6ae700 1 --2- 172.21.15.121:0/3404624994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7224105a40 0x7f7224105e20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.900+0000 7f722b6ae700 1 -- 172.21.15.121:0/3404624994 >> 172.21.15.121:0/3404624994 conn(0x7f72240755a0 msgr2=0x7f72240759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:04.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.900+0000 7f722b6ae700 1 -- 172.21.15.121:0/3404624994 shutdown_connections 2024-04-05T21:39:04.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.900+0000 7f722b6ae700 1 -- 172.21.15.121:0/3404624994 wait complete. 2024-04-05T21:39:04.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.901+0000 7f722b6ae700 1 Processor -- start 2024-04-05T21:39:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.901+0000 7f722b6ae700 1 -- start start 2024-04-05T21:39:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.901+0000 7f722b6ae700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72240ffbd0 0x7f7224195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f722b6ae700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7224105a40 0x7f7224195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f722b6ae700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7224196170 con 0x7f72240ffbd0 2024-04-05T21:39:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f722b6ae700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7224199f00 con 0x7f7224105a40 2024-04-05T21:39:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f722944a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72240ffbd0 0x7f7224195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f722944a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72240ffbd0 0x7f7224195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49372/0 (socket says 172.21.15.121:49372) 2024-04-05T21:39:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f722944a700 1 -- 172.21.15.121:0/596607695 learned_addr learned my addr 172.21.15.121:0/596607695 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f7228c49700 1 --2- 172.21.15.121:0/596607695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7224105a40 0x7f7224195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f722944a700 1 -- 172.21.15.121:0/596607695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7224105a40 msgr2=0x7f7224195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f722944a700 1 --2- 172.21.15.121:0/596607695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7224105a40 0x7f7224195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f722944a700 1 -- 172.21.15.121:0/596607695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72180095f0 con 0x7f72240ffbd0 2024-04-05T21:39:04.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f7228c49700 1 --2- 172.21.15.121:0/596607695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7224105a40 0x7f7224195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:39:04.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f722944a700 1 --2- 172.21.15.121:0/596607695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72240ffbd0 0x7f7224195550 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7218006010 tx=0x7f721800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:04.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f72167fc700 1 -- 172.21.15.121:0/596607695 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f721801d070 con 0x7f72240ffbd0 2024-04-05T21:39:04.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f722b6ae700 1 -- 172.21.15.121:0/596607695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722419a180 con 0x7f72240ffbd0 2024-04-05T21:39:04.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f72167fc700 1 -- 172.21.15.121:0/596607695 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f721800f460 con 0x7f72240ffbd0 2024-04-05T21:39:04.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f72167fc700 1 -- 172.21.15.121:0/596607695 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7218021640 con 0x7f72240ffbd0 2024-04-05T21:39:04.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.902+0000 7f722b6ae700 1 -- 172.21.15.121:0/596607695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722419a670 con 0x7f72240ffbd0 2024-04-05T21:39:04.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.903+0000 7f722b6ae700 1 -- 172.21.15.121:0/596607695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7224108040 con 0x7f72240ffbd0 2024-04-05T21:39:04.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.904+0000 7f72167fc700 1 -- 172.21.15.121:0/596607695 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f721800fa20 con 0x7f72240ffbd0 2024-04-05T21:39:04.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.904+0000 7f72167fc700 1 --2- 172.21.15.121:0/596607695 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f721006cce0 0x7f721006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:04.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.904+0000 7f72167fc700 1 -- 172.21.15.121:0/596607695 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f721808d830 con 0x7f72240ffbd0 2024-04-05T21:39:04.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.904+0000 7f7228c49700 1 --2- 172.21.15.121:0/596607695 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f721006cce0 0x7f721006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:04.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.904+0000 7f7228c49700 1 --2- 172.21.15.121:0/596607695 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f721006cce0 0x7f721006f1a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7220005d90 tx=0x7f7220005d00 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:04.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:04.908+0000 7f72167fc700 1 -- 172.21.15.121:0/596607695 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f72180578a0 con 0x7f72240ffbd0 2024-04-05T21:39:05.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.117+0000 7f722b6ae700 1 -- 172.21.15.121:0/596607695 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f722402cce0 con 0x7f721006cce0 2024-04-05T21:39:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.119+0000 7f72167fc700 1 -- 172.21.15.121:0/596607695 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f722402cce0 con 0x7f721006cce0 2024-04-05T21:39:05.119 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:39:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:39:05.119 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:39:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:39:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:39:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:39:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:39:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:39:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:39:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:39:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:39:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:39:05.120 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:39:05.120 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:39:05.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.121+0000 7f722b6ae700 1 -- 172.21.15.121:0/596607695 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f721006cce0 msgr2=0x7f721006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:05.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.121+0000 7f722b6ae700 1 --2- 172.21.15.121:0/596607695 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f721006cce0 0x7f721006f1a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7220005d90 tx=0x7f7220005d00 comp rx=0 tx=0).stop 2024-04-05T21:39:05.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.121+0000 7f722b6ae700 1 -- 172.21.15.121:0/596607695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72240ffbd0 msgr2=0x7f7224195550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:05.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.121+0000 7f722b6ae700 1 --2- 172.21.15.121:0/596607695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72240ffbd0 0x7f7224195550 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7218006010 tx=0x7f721800be20 comp rx=0 tx=0).stop 2024-04-05T21:39:05.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.121+0000 7f722b6ae700 1 -- 172.21.15.121:0/596607695 shutdown_connections 2024-04-05T21:39:05.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.121+0000 7f722b6ae700 1 --2- 172.21.15.121:0/596607695 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f721006cce0 0x7f721006f1a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:05.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.121+0000 7f722b6ae700 1 --2- 172.21.15.121:0/596607695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72240ffbd0 0x7f7224195550 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:05.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.121+0000 7f722b6ae700 1 --2- 172.21.15.121:0/596607695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7224105a40 0x7f7224195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:05.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.121+0000 7f722b6ae700 1 -- 172.21.15.121:0/596607695 >> 172.21.15.121:0/596607695 conn(0x7f72240755a0 msgr2=0x7f72240fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:05.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.121+0000 7f722b6ae700 1 -- 172.21.15.121:0/596607695 shutdown_connections 2024-04-05T21:39:05.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.121+0000 7f722b6ae700 1 -- 172.21.15.121:0/596607695 wait complete. 2024-04-05T21:39:05.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.223+0000 7fca84a67700 1 -- 172.21.15.121:0/3453922280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca800ff2f0 msgr2=0x7fca800ff6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:05.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.223+0000 7fca84a67700 1 --2- 172.21.15.121:0/3453922280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca800ff2f0 0x7fca800ff6d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fca70009910 tx=0x7fca70009c20 comp rx=0 tx=0).stop 2024-04-05T21:39:05.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.223+0000 7fca84a67700 1 -- 172.21.15.121:0/3453922280 shutdown_connections 2024-04-05T21:39:05.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.223+0000 7fca84a67700 1 --2- 172.21.15.121:0/3453922280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca800ffca0 0x7fca801057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:05.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.223+0000 7fca84a67700 1 --2- 172.21.15.121:0/3453922280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca800ff2f0 0x7fca800ff6d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:05.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.223+0000 7fca84a67700 1 -- 172.21.15.121:0/3453922280 >> 172.21.15.121:0/3453922280 conn(0x7fca80075e30 msgr2=0x7fca800fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:05.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.223+0000 7fca84a67700 1 -- 172.21.15.121:0/3453922280 shutdown_connections 2024-04-05T21:39:05.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.223+0000 7fca84a67700 1 -- 172.21.15.121:0/3453922280 wait complete. 2024-04-05T21:39:05.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.223+0000 7fca84a67700 1 Processor -- start 2024-04-05T21:39:05.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca84a67700 1 -- start start 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca84a67700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca800ff2f0 0x7fca80196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca84a67700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca800ffca0 0x7fca80196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca84a67700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca801971a0 con 0x7fca800ff2f0 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca84a67700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8019af30 con 0x7fca800ffca0 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca7e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca800ff2f0 0x7fca80196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca7e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca800ff2f0 0x7fca80196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49378/0 (socket says 172.21.15.121:49378) 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca7e59c700 1 -- 172.21.15.121:0/4041400885 learned_addr learned my addr 172.21.15.121:0/4041400885 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca7dd9b700 1 --2- 172.21.15.121:0/4041400885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca800ffca0 0x7fca80196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca7e59c700 1 -- 172.21.15.121:0/4041400885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca800ffca0 msgr2=0x7fca80196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca7e59c700 1 --2- 172.21.15.121:0/4041400885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca800ffca0 0x7fca80196ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca7e59c700 1 -- 172.21.15.121:0/4041400885 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca700095f0 con 0x7fca800ff2f0 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca7e59c700 1 --2- 172.21.15.121:0/4041400885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca800ff2f0 0x7fca80196580 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fca700092e0 tx=0x7fca7000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca6f7fe700 1 -- 172.21.15.121:0/4041400885 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca7001d070 con 0x7fca800ff2f0 2024-04-05T21:39:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca6f7fe700 1 -- 172.21.15.121:0/4041400885 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca70005e00 con 0x7fca800ff2f0 2024-04-05T21:39:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca84a67700 1 -- 172.21.15.121:0/4041400885 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8019b1b0 con 0x7fca800ff2f0 2024-04-05T21:39:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca6f7fe700 1 -- 172.21.15.121:0/4041400885 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca7000fe90 con 0x7fca800ff2f0 2024-04-05T21:39:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.224+0000 7fca84a67700 1 -- 172.21.15.121:0/4041400885 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8019b720 con 0x7fca800ff2f0 2024-04-05T21:39:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.226+0000 7fca84a67700 1 -- 172.21.15.121:0/4041400885 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca80109110 con 0x7fca800ff2f0 2024-04-05T21:39:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.226+0000 7fca6f7fe700 1 -- 172.21.15.121:0/4041400885 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fca7000f6d0 con 0x7fca800ff2f0 2024-04-05T21:39:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.226+0000 7fca6f7fe700 1 --2- 172.21.15.121:0/4041400885 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fca6806cce0 0x7fca6806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.227+0000 7fca6f7fe700 1 -- 172.21.15.121:0/4041400885 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fca7008d7e0 con 0x7fca800ff2f0 2024-04-05T21:39:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.227+0000 7fca7dd9b700 1 --2- 172.21.15.121:0/4041400885 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fca6806cce0 0x7fca6806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.227+0000 7fca7dd9b700 1 --2- 172.21.15.121:0/4041400885 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fca6806cce0 0x7fca6806f1a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fca80197ba0 tx=0x7fca7400b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:05.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.231+0000 7fca6f7fe700 1 -- 172.21.15.121:0/4041400885 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fca70057850 con 0x7fca800ff2f0 2024-04-05T21:39:05.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.506+0000 7fca84a67700 1 -- 172.21.15.121:0/4041400885 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fca8019ba00 con 0x7fca800ff2f0 2024-04-05T21:39:05.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.506+0000 7fca6f7fe700 1 -- 172.21.15.121:0/4041400885 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fca7005ae70 con 0x7fca800ff2f0 2024-04-05T21:39:05.507 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:39:05.507 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:39:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:39:05.507 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:39:05.507 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:39:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.508+0000 7fca84a67700 1 -- 172.21.15.121:0/4041400885 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fca6806cce0 msgr2=0x7fca6806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.508+0000 7fca84a67700 1 --2- 172.21.15.121:0/4041400885 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fca6806cce0 0x7fca6806f1a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fca80197ba0 tx=0x7fca7400b410 comp rx=0 tx=0).stop 2024-04-05T21:39:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.508+0000 7fca84a67700 1 -- 172.21.15.121:0/4041400885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca800ff2f0 msgr2=0x7fca80196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.508+0000 7fca84a67700 1 --2- 172.21.15.121:0/4041400885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca800ff2f0 0x7fca80196580 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fca700092e0 tx=0x7fca7000bd70 comp rx=0 tx=0).stop 2024-04-05T21:39:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.508+0000 7fca84a67700 1 -- 172.21.15.121:0/4041400885 shutdown_connections 2024-04-05T21:39:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.508+0000 7fca84a67700 1 --2- 172.21.15.121:0/4041400885 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fca6806cce0 0x7fca6806f1a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:05.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.508+0000 7fca84a67700 1 --2- 172.21.15.121:0/4041400885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca800ff2f0 0x7fca80196580 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:05.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.508+0000 7fca84a67700 1 --2- 172.21.15.121:0/4041400885 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca800ffca0 0x7fca80196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:05.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.508+0000 7fca84a67700 1 -- 172.21.15.121:0/4041400885 >> 172.21.15.121:0/4041400885 conn(0x7fca80075e30 msgr2=0x7fca800fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:05.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.509+0000 7fca84a67700 1 -- 172.21.15.121:0/4041400885 shutdown_connections 2024-04-05T21:39:05.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:05.509+0000 7fca84a67700 1 -- 172.21.15.121:0/4041400885 wait complete. 2024-04-05T21:39:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:05 smithi121 ceph-mon[145859]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:05 smithi121 ceph-mon[145859]: pgmap v385: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:05 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1551089096' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:39:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:05 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4273648057' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:39:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:05 smithi188 ceph-mon[107793]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:05 smithi188 ceph-mon[107793]: pgmap v385: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:05 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1551089096' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:39:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:05 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4273648057' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:39:06.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:06 smithi121 ceph-mon[145859]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:06.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:06 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4041400885' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:39:06.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:06 smithi188 ceph-mon[107793]: from='client.34586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:06.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:06 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4041400885' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:39:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:07 smithi121 ceph-mon[145859]: pgmap v386: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:07 smithi188 ceph-mon[107793]: pgmap v386: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:09 smithi121 ceph-mon[145859]: pgmap v387: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:09 smithi188 ceph-mon[107793]: pgmap v387: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:11 smithi121 ceph-mon[145859]: pgmap v388: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:11.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:11 smithi188 ceph-mon[107793]: pgmap v388: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:13 smithi121 ceph-mon[145859]: pgmap v389: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:13.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:13 smithi188 ceph-mon[107793]: pgmap v389: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:15 smithi121 ceph-mon[145859]: pgmap v390: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:39:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:15 smithi188 ceph-mon[107793]: pgmap v390: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:39:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:17 smithi121 ceph-mon[145859]: pgmap v391: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:17.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:17 smithi188 ceph-mon[107793]: pgmap v391: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:19 smithi121 ceph-mon[145859]: pgmap v392: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:19 smithi188 ceph-mon[107793]: pgmap v392: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:21 smithi121 ceph-mon[145859]: pgmap v393: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:21 smithi188 ceph-mon[107793]: pgmap v393: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:23.718 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:23 smithi121 ceph-mon[145859]: pgmap v394: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:23 smithi188 ceph-mon[107793]: pgmap v394: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:25.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:25 smithi121 ceph-mon[145859]: pgmap v395: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:25.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:25 smithi188 ceph-mon[107793]: pgmap v395: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:27 smithi121 ceph-mon[145859]: pgmap v396: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:27.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:27 smithi188 ceph-mon[107793]: pgmap v396: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:29 smithi121 ceph-mon[145859]: pgmap v397: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:29 smithi188 ceph-mon[107793]: pgmap v397: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:39:30.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:39:31.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:31 smithi121 ceph-mon[145859]: pgmap v398: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:31.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:31 smithi188 ceph-mon[107793]: pgmap v398: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:33.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:33 smithi121 ceph-mon[145859]: pgmap v399: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:33 smithi188 ceph-mon[107793]: pgmap v399: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:35.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.626+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/2022157104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1f8105b50 msgr2=0x7fa1f8068410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:35.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.626+0000 7fa1ff2dd700 1 --2- 172.21.15.121:0/2022157104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1f8105b50 0x7fa1f8068410 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009910 tx=0x7fa1f4009c20 comp rx=0 tx=0).stop 2024-04-05T21:39:35.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.626+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/2022157104 shutdown_connections 2024-04-05T21:39:35.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.626+0000 7fa1ff2dd700 1 --2- 172.21.15.121:0/2022157104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1f8068950 0x7fa1f8068dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:35.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.626+0000 7fa1ff2dd700 1 --2- 172.21.15.121:0/2022157104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1f8105b50 0x7fa1f8068410 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:35.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.626+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/2022157104 >> 172.21.15.121:0/2022157104 conn(0x7fa1f80755a0 msgr2=0x7fa1f80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:35.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.626+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/2022157104 shutdown_connections 2024-04-05T21:39:35.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.626+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/2022157104 wait complete. 2024-04-05T21:39:35.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.626+0000 7fa1ff2dd700 1 Processor -- start 2024-04-05T21:39:35.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.626+0000 7fa1ff2dd700 1 -- start start 2024-04-05T21:39:35.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1ff2dd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1f8068950 0x7fa1f81a29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:35.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1ff2dd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1f8105b50 0x7fa1f81a2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:35.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1ff2dd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f81a35c0 con 0x7fa1f8105b50 2024-04-05T21:39:35.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1ff2dd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f819ca70 con 0x7fa1f8068950 2024-04-05T21:39:35.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1fc878700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1f8105b50 0x7fa1f81a2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:35.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1fc878700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1f8105b50 0x7fa1f81a2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59824/0 (socket says 172.21.15.121:59824) 2024-04-05T21:39:35.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1fc878700 1 -- 172.21.15.121:0/1782581834 learned_addr learned my addr 172.21.15.121:0/1782581834 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:35.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1fd079700 1 --2- 172.21.15.121:0/1782581834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1f8068950 0x7fa1f81a29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:35.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1fc878700 1 -- 172.21.15.121:0/1782581834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1f8068950 msgr2=0x7fa1f81a29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:35.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1fc878700 1 --2- 172.21.15.121:0/1782581834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1f8068950 0x7fa1f81a29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:35.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1fc878700 1 -- 172.21.15.121:0/1782581834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f40095f0 con 0x7fa1f8105b50 2024-04-05T21:39:35.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1fc878700 1 --2- 172.21.15.121:0/1782581834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1f8105b50 0x7fa1f81a2f30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa1e800d8d0 tx=0x7fa1e800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:35.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1ee7fc700 1 -- 172.21.15.121:0/1782581834 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1e80098a0 con 0x7fa1f8105b50 2024-04-05T21:39:35.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/1782581834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f819cd50 con 0x7fa1f8105b50 2024-04-05T21:39:35.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1ee7fc700 1 -- 172.21.15.121:0/1782581834 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1e8010460 con 0x7fa1f8105b50 2024-04-05T21:39:35.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1ee7fc700 1 -- 172.21.15.121:0/1782581834 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1e8009ea0 con 0x7fa1f8105b50 2024-04-05T21:39:35.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.627+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/1782581834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f819d320 con 0x7fa1f8105b50 2024-04-05T21:39:35.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.628+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/1782581834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f804ea90 con 0x7fa1f8105b50 2024-04-05T21:39:35.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.629+0000 7fa1ee7fc700 1 -- 172.21.15.121:0/1782581834 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa1e80105d0 con 0x7fa1f8105b50 2024-04-05T21:39:35.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.629+0000 7fa1ee7fc700 1 --2- 172.21.15.121:0/1782581834 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1e406cce0 0x7fa1e406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:35.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.629+0000 7fa1ee7fc700 1 -- 172.21.15.121:0/1782581834 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa1e808c440 con 0x7fa1f8105b50 2024-04-05T21:39:35.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.629+0000 7fa1fd079700 1 --2- 172.21.15.121:0/1782581834 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1e406cce0 0x7fa1e406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:35.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.630+0000 7fa1fd079700 1 --2- 172.21.15.121:0/1782581834 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1e406cce0 0x7fa1e406f1a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4006010 tx=0x7fa1f401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:35.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.634+0000 7fa1ee7fc700 1 -- 172.21.15.121:0/1782581834 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa1e8059f20 con 0x7fa1f8105b50 2024-04-05T21:39:35.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:35 smithi121 ceph-mon[145859]: pgmap v400: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:35.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:35 smithi188 ceph-mon[107793]: pgmap v400: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:35.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.844+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/1782581834 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1f819d770 con 0x7fa1e406cce0 2024-04-05T21:39:35.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.845+0000 7fa1ee7fc700 1 -- 172.21.15.121:0/1782581834 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa1f819d770 con 0x7fa1e406cce0 2024-04-05T21:39:35.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.847+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/1782581834 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1e406cce0 msgr2=0x7fa1e406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:35.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.847+0000 7fa1ff2dd700 1 --2- 172.21.15.121:0/1782581834 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1e406cce0 0x7fa1e406f1a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4006010 tx=0x7fa1f401a040 comp rx=0 tx=0).stop 2024-04-05T21:39:35.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.847+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/1782581834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1f8105b50 msgr2=0x7fa1f81a2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:35.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.847+0000 7fa1ff2dd700 1 --2- 172.21.15.121:0/1782581834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1f8105b50 0x7fa1f81a2f30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa1e800d8d0 tx=0x7fa1e800dbe0 comp rx=0 tx=0).stop 2024-04-05T21:39:35.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.847+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/1782581834 shutdown_connections 2024-04-05T21:39:35.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.847+0000 7fa1ff2dd700 1 --2- 172.21.15.121:0/1782581834 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1e406cce0 0x7fa1e406f1a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:35.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.847+0000 7fa1ff2dd700 1 --2- 172.21.15.121:0/1782581834 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1f8068950 0x7fa1f81a29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:35.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.847+0000 7fa1ff2dd700 1 --2- 172.21.15.121:0/1782581834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1f8105b50 0x7fa1f81a2f30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:35.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.847+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/1782581834 >> 172.21.15.121:0/1782581834 conn(0x7fa1f80755a0 msgr2=0x7fa1f8101510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:35.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.847+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/1782581834 shutdown_connections 2024-04-05T21:39:35.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.847+0000 7fa1ff2dd700 1 -- 172.21.15.121:0/1782581834 wait complete. 2024-04-05T21:39:35.858 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:39:35.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.949+0000 7fc3a6986700 1 -- 172.21.15.121:0/2038340261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a0100dc0 msgr2=0x7fc3a01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:35.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.949+0000 7fc3a6986700 1 --2- 172.21.15.121:0/2038340261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a0100dc0 0x7fc3a01011a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc390009910 tx=0x7fc390009c20 comp rx=0 tx=0).stop 2024-04-05T21:39:35.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.949+0000 7fc3a6986700 1 -- 172.21.15.121:0/2038340261 shutdown_connections 2024-04-05T21:39:35.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.949+0000 7fc3a6986700 1 --2- 172.21.15.121:0/2038340261 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3a0101770 0x7fc3a01057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:35.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.949+0000 7fc3a6986700 1 --2- 172.21.15.121:0/2038340261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a0100dc0 0x7fc3a01011a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:35.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.949+0000 7fc3a6986700 1 -- 172.21.15.121:0/2038340261 >> 172.21.15.121:0/2038340261 conn(0x7fc3a00fc650 msgr2=0x7fc3a00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:35.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.949+0000 7fc3a6986700 1 -- 172.21.15.121:0/2038340261 shutdown_connections 2024-04-05T21:39:35.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.949+0000 7fc3a6986700 1 -- 172.21.15.121:0/2038340261 wait complete. 2024-04-05T21:39:35.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.950+0000 7fc3a6986700 1 Processor -- start 2024-04-05T21:39:35.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.950+0000 7fc3a6986700 1 -- start start 2024-04-05T21:39:35.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.950+0000 7fc3a6986700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a0100dc0 0x7fc3a0196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:35.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.950+0000 7fc3a6986700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3a0101770 0x7fc3a0196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:35.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.950+0000 7fc3a6986700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a0197280 con 0x7fc3a0100dc0 2024-04-05T21:39:35.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.950+0000 7fc3a6986700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a019b010 con 0x7fc3a0101770 2024-04-05T21:39:35.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.951+0000 7fc39ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a0100dc0 0x7fc3a0196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:35.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.951+0000 7fc39ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a0100dc0 0x7fc3a0196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59854/0 (socket says 172.21.15.121:59854) 2024-04-05T21:39:35.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.951+0000 7fc39ffff700 1 -- 172.21.15.121:0/2410295253 learned_addr learned my addr 172.21.15.121:0/2410295253 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:35.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.951+0000 7fc39f7fe700 1 --2- 172.21.15.121:0/2410295253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3a0101770 0x7fc3a0196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:35.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.951+0000 7fc39ffff700 1 -- 172.21.15.121:0/2410295253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3a0101770 msgr2=0x7fc3a0196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:35.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.951+0000 7fc39ffff700 1 --2- 172.21.15.121:0/2410295253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3a0101770 0x7fc3a0196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:35.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.951+0000 7fc39ffff700 1 -- 172.21.15.121:0/2410295253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3900095f0 con 0x7fc3a0100dc0 2024-04-05T21:39:35.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.951+0000 7fc39ffff700 1 --2- 172.21.15.121:0/2410295253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a0100dc0 0x7fc3a0196660 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc390006010 tx=0x7fc39000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:35.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.951+0000 7fc39d7fa700 1 -- 172.21.15.121:0/2410295253 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc39001d070 con 0x7fc3a0100dc0 2024-04-05T21:39:35.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.951+0000 7fc39d7fa700 1 -- 172.21.15.121:0/2410295253 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc390005710 con 0x7fc3a0100dc0 2024-04-05T21:39:35.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.951+0000 7fc3a6986700 1 -- 172.21.15.121:0/2410295253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a019b290 con 0x7fc3a0100dc0 2024-04-05T21:39:35.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.951+0000 7fc39d7fa700 1 -- 172.21.15.121:0/2410295253 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc39000fe20 con 0x7fc3a0100dc0 2024-04-05T21:39:35.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.951+0000 7fc3a6986700 1 -- 172.21.15.121:0/2410295253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a019b780 con 0x7fc3a0100dc0 2024-04-05T21:39:35.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.952+0000 7fc3a6986700 1 -- 172.21.15.121:0/2410295253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a0109160 con 0x7fc3a0100dc0 2024-04-05T21:39:35.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.952+0000 7fc39d7fa700 1 -- 172.21.15.121:0/2410295253 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc390003a60 con 0x7fc3a0100dc0 2024-04-05T21:39:35.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.953+0000 7fc39d7fa700 1 --2- 172.21.15.121:0/2410295253 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc38806cce0 0x7fc38806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:35.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.953+0000 7fc39d7fa700 1 -- 172.21.15.121:0/2410295253 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc39008d750 con 0x7fc3a0100dc0 2024-04-05T21:39:35.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.953+0000 7fc39f7fe700 1 --2- 172.21.15.121:0/2410295253 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc38806cce0 0x7fc38806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:35.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.953+0000 7fc39f7fe700 1 --2- 172.21.15.121:0/2410295253 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc38806cce0 0x7fc38806f1a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0197c80 tx=0x7fc3940073d0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:35.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:35.957+0000 7fc39d7fa700 1 -- 172.21.15.121:0/2410295253 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc3900577c0 con 0x7fc3a0100dc0 2024-04-05T21:39:36.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.166+0000 7fc3a6986700 1 -- 172.21.15.121:0/2410295253 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3a002cd30 con 0x7fc38806cce0 2024-04-05T21:39:36.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.167+0000 7fc39d7fa700 1 -- 172.21.15.121:0/2410295253 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fc3a002cd30 con 0x7fc38806cce0 2024-04-05T21:39:36.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.169+0000 7fc3a6986700 1 -- 172.21.15.121:0/2410295253 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc38806cce0 msgr2=0x7fc38806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:36.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.169+0000 7fc3a6986700 1 --2- 172.21.15.121:0/2410295253 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc38806cce0 0x7fc38806f1a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0197c80 tx=0x7fc3940073d0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.169+0000 7fc3a6986700 1 -- 172.21.15.121:0/2410295253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a0100dc0 msgr2=0x7fc3a0196660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:36.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.169+0000 7fc3a6986700 1 --2- 172.21.15.121:0/2410295253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a0100dc0 0x7fc3a0196660 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc390006010 tx=0x7fc39000bd70 comp rx=0 tx=0).stop 2024-04-05T21:39:36.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.169+0000 7fc3a6986700 1 -- 172.21.15.121:0/2410295253 shutdown_connections 2024-04-05T21:39:36.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.169+0000 7fc3a6986700 1 --2- 172.21.15.121:0/2410295253 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc38806cce0 0x7fc38806f1a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.169+0000 7fc3a6986700 1 --2- 172.21.15.121:0/2410295253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a0100dc0 0x7fc3a0196660 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.169+0000 7fc3a6986700 1 --2- 172.21.15.121:0/2410295253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3a0101770 0x7fc3a0196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.169+0000 7fc3a6986700 1 -- 172.21.15.121:0/2410295253 >> 172.21.15.121:0/2410295253 conn(0x7fc3a00fc650 msgr2=0x7fc3a00fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:36.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.169+0000 7fc3a6986700 1 -- 172.21.15.121:0/2410295253 shutdown_connections 2024-04-05T21:39:36.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.169+0000 7fc3a6986700 1 -- 172.21.15.121:0/2410295253 wait complete. 2024-04-05T21:39:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.271+0000 7f99e879e700 1 -- 172.21.15.121:0/3566733087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e00ffbd0 msgr2=0x7f99e0100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.271+0000 7f99e879e700 1 --2- 172.21.15.121:0/3566733087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e00ffbd0 0x7f99e0100050 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f99d4009910 tx=0x7f99d4009c20 comp rx=0 tx=0).stop 2024-04-05T21:39:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.271+0000 7f99e879e700 1 -- 172.21.15.121:0/3566733087 shutdown_connections 2024-04-05T21:39:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.271+0000 7f99e879e700 1 --2- 172.21.15.121:0/3566733087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e00ffbd0 0x7f99e0100050 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.271+0000 7f99e879e700 1 --2- 172.21.15.121:0/3566733087 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e0105a40 0x7f99e0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.271+0000 7f99e879e700 1 -- 172.21.15.121:0/3566733087 >> 172.21.15.121:0/3566733087 conn(0x7f99e00755a0 msgr2=0x7f99e00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.271+0000 7f99e879e700 1 -- 172.21.15.121:0/3566733087 shutdown_connections 2024-04-05T21:39:36.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.271+0000 7f99e879e700 1 -- 172.21.15.121:0/3566733087 wait complete. 2024-04-05T21:39:36.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.272+0000 7f99e879e700 1 Processor -- start 2024-04-05T21:39:36.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.272+0000 7f99e879e700 1 -- start start 2024-04-05T21:39:36.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.272+0000 7f99e879e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e00ffbd0 0x7f99e01956b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:36.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.272+0000 7f99e879e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e0105a40 0x7f99e0195bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:36.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.272+0000 7f99e879e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e0196280 con 0x7f99e0105a40 2024-04-05T21:39:36.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.272+0000 7f99e879e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e0198fb0 con 0x7f99e00ffbd0 2024-04-05T21:39:36.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.272+0000 7f99e5d39700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e0105a40 0x7f99e0195bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:36.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.272+0000 7f99e5d39700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e0105a40 0x7f99e0195bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59872/0 (socket says 172.21.15.121:59872) 2024-04-05T21:39:36.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.272+0000 7f99e5d39700 1 -- 172.21.15.121:0/1879284060 learned_addr learned my addr 172.21.15.121:0/1879284060 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:36.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.273+0000 7f99e5d39700 1 -- 172.21.15.121:0/1879284060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e00ffbd0 msgr2=0x7f99e01956b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:36.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.273+0000 7f99e653a700 1 --2- 172.21.15.121:0/1879284060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e00ffbd0 0x7f99e01956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:36.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.273+0000 7f99e5d39700 1 --2- 172.21.15.121:0/1879284060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e00ffbd0 0x7f99e01956b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.273+0000 7f99e5d39700 1 -- 172.21.15.121:0/1879284060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d40095f0 con 0x7f99e0105a40 2024-04-05T21:39:36.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.273+0000 7f99e5d39700 1 --2- 172.21.15.121:0/1879284060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e0105a40 0x7f99e0195bf0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f99d4006010 tx=0x7f99d400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:36.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.273+0000 7f99df7fe700 1 -- 172.21.15.121:0/1879284060 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99d401d070 con 0x7f99e0105a40 2024-04-05T21:39:36.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.273+0000 7f99df7fe700 1 -- 172.21.15.121:0/1879284060 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99d400f460 con 0x7f99e0105a40 2024-04-05T21:39:36.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.273+0000 7f99e879e700 1 -- 172.21.15.121:0/1879284060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e0199230 con 0x7f99e0105a40 2024-04-05T21:39:36.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.273+0000 7f99df7fe700 1 -- 172.21.15.121:0/1879284060 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99d4021640 con 0x7f99e0105a40 2024-04-05T21:39:36.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.273+0000 7f99e879e700 1 -- 172.21.15.121:0/1879284060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e0199720 con 0x7f99e0105a40 2024-04-05T21:39:36.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.274+0000 7f99e879e700 1 -- 172.21.15.121:0/1879284060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e0108060 con 0x7f99e0105a40 2024-04-05T21:39:36.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.274+0000 7f99df7fe700 1 -- 172.21.15.121:0/1879284060 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f99d400f9e0 con 0x7f99e0105a40 2024-04-05T21:39:36.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.275+0000 7f99df7fe700 1 --2- 172.21.15.121:0/1879284060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f99c806cce0 0x7f99c806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:36.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.275+0000 7f99df7fe700 1 -- 172.21.15.121:0/1879284060 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f99d408d900 con 0x7f99e0105a40 2024-04-05T21:39:36.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.275+0000 7f99e653a700 1 --2- 172.21.15.121:0/1879284060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f99c806cce0 0x7f99c806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:36.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.275+0000 7f99e653a700 1 --2- 172.21.15.121:0/1879284060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f99c806cce0 0x7f99c806f1a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99d0007dc0 tx=0x7f99d00072f0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:36.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.279+0000 7f99df7fe700 1 -- 172.21.15.121:0/1879284060 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f99d4057970 con 0x7f99e0105a40 2024-04-05T21:39:36.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.482+0000 7f99e879e700 1 -- 172.21.15.121:0/1879284060 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f99e0196b20 con 0x7f99c806cce0 2024-04-05T21:39:36.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.492+0000 7f99df7fe700 1 -- 172.21.15.121:0/1879284060 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f99e0196b20 con 0x7f99c806cce0 2024-04-05T21:39:36.493 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:39:36.493 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (10m) 4m ago 21m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:39:36.493 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (22m) 4m ago 22m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:39:36.493 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (21m) 4m ago 21m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (8m) 4m ago 22m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (8m) 4m ago 20m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (10m) 4m ago 21m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (4m) 4m ago 15m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (4m) 4m ago 15m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (4m) 4m ago 15m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (15m) 4m ago 15m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (12m) 4m ago 23m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (11m) 4m ago 20m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (9m) 4m ago 23m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (8m) 4m ago 20m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (11m) 4m ago 22m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (11m) 4m ago 20m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (8m) 4m ago 19m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:39:36.494 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (7m) 4m ago 19m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:39:36.495 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (6m) 4m ago 18m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:39:36.495 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (6m) 4m ago 18m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:39:36.495 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (5m) 4m ago 17m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:39:36.495 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (5m) 4m ago 17m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:39:36.495 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (10m) 4m ago 21m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:39:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.494+0000 7f99e879e700 1 -- 172.21.15.121:0/1879284060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f99c806cce0 msgr2=0x7f99c806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.494+0000 7f99e879e700 1 --2- 172.21.15.121:0/1879284060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f99c806cce0 0x7f99c806f1a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f99d0007dc0 tx=0x7f99d00072f0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.494+0000 7f99e879e700 1 -- 172.21.15.121:0/1879284060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e0105a40 msgr2=0x7f99e0195bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.494+0000 7f99e879e700 1 --2- 172.21.15.121:0/1879284060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e0105a40 0x7f99e0195bf0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f99d4006010 tx=0x7f99d400be20 comp rx=0 tx=0).stop 2024-04-05T21:39:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.495+0000 7f99e879e700 1 -- 172.21.15.121:0/1879284060 shutdown_connections 2024-04-05T21:39:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.495+0000 7f99e879e700 1 --2- 172.21.15.121:0/1879284060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f99c806cce0 0x7f99c806f1a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.495+0000 7f99e879e700 1 --2- 172.21.15.121:0/1879284060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99e00ffbd0 0x7f99e01956b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.495+0000 7f99e879e700 1 --2- 172.21.15.121:0/1879284060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99e0105a40 0x7f99e0195bf0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.495+0000 7f99e879e700 1 -- 172.21.15.121:0/1879284060 >> 172.21.15.121:0/1879284060 conn(0x7f99e00755a0 msgr2=0x7f99e00fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.495+0000 7f99e879e700 1 -- 172.21.15.121:0/1879284060 shutdown_connections 2024-04-05T21:39:36.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.495+0000 7f99e879e700 1 -- 172.21.15.121:0/1879284060 wait complete. 2024-04-05T21:39:36.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.596+0000 7ff8b1175700 1 -- 172.21.15.121:0/784805871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac100da0 msgr2=0x7ff8ac101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:36.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.596+0000 7ff8b1175700 1 --2- 172.21.15.121:0/784805871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac100da0 0x7ff8ac101180 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00b0a0 tx=0x7ff89c00b3b0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.596+0000 7ff8b1175700 1 -- 172.21.15.121:0/784805871 shutdown_connections 2024-04-05T21:39:36.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.596+0000 7ff8b1175700 1 --2- 172.21.15.121:0/784805871 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac101750 0x7ff8ac1057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.596+0000 7ff8b1175700 1 --2- 172.21.15.121:0/784805871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac100da0 0x7ff8ac101180 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.596+0000 7ff8b1175700 1 -- 172.21.15.121:0/784805871 >> 172.21.15.121:0/784805871 conn(0x7ff8ac0fc630 msgr2=0x7ff8ac0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:36.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.596+0000 7ff8b1175700 1 -- 172.21.15.121:0/784805871 shutdown_connections 2024-04-05T21:39:36.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.596+0000 7ff8b1175700 1 -- 172.21.15.121:0/784805871 wait complete. 2024-04-05T21:39:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.596+0000 7ff8b1175700 1 Processor -- start 2024-04-05T21:39:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.597+0000 7ff8b1175700 1 -- start start 2024-04-05T21:39:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.597+0000 7ff8b1175700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac100da0 0x7ff8ac1966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.597+0000 7ff8b1175700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac101750 0x7ff8ac196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.597+0000 7ff8b1175700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac1972c0 con 0x7ff8ac101750 2024-04-05T21:39:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.597+0000 7ff8b1175700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac199ff0 con 0x7ff8ac100da0 2024-04-05T21:39:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.597+0000 7ff8aa59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac101750 0x7ff8ac196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.597+0000 7ff8aa59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac101750 0x7ff8ac196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59902/0 (socket says 172.21.15.121:59902) 2024-04-05T21:39:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.597+0000 7ff8aa59c700 1 -- 172.21.15.121:0/562433195 learned_addr learned my addr 172.21.15.121:0/562433195 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.597+0000 7ff8aad9d700 1 --2- 172.21.15.121:0/562433195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac100da0 0x7ff8ac1966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.597+0000 7ff8aa59c700 1 -- 172.21.15.121:0/562433195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac100da0 msgr2=0x7ff8ac1966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.597+0000 7ff8aa59c700 1 --2- 172.21.15.121:0/562433195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac100da0 0x7ff8ac1966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.597+0000 7ff8aa59c700 1 -- 172.21.15.121:0/562433195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89c009d00 con 0x7ff8ac101750 2024-04-05T21:39:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.597+0000 7ff8aa59c700 1 --2- 172.21.15.121:0/562433195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac101750 0x7ff8ac196c30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff89400e8f0 tx=0x7ff89400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.598+0000 7ff8a3fff700 1 -- 172.21.15.121:0/562433195 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff894009790 con 0x7ff8ac101750 2024-04-05T21:39:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.598+0000 7ff8b1175700 1 -- 172.21.15.121:0/562433195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ac19a2d0 con 0x7ff8ac101750 2024-04-05T21:39:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.598+0000 7ff8a3fff700 1 -- 172.21.15.121:0/562433195 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff89400c810 con 0x7ff8ac101750 2024-04-05T21:39:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.598+0000 7ff8b1175700 1 -- 172.21.15.121:0/562433195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac19a820 con 0x7ff8ac101750 2024-04-05T21:39:36.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.598+0000 7ff8a3fff700 1 -- 172.21.15.121:0/562433195 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff894005540 con 0x7ff8ac101750 2024-04-05T21:39:36.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.599+0000 7ff8b1175700 1 -- 172.21.15.121:0/562433195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ac109140 con 0x7ff8ac101750 2024-04-05T21:39:36.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.599+0000 7ff8a3fff700 1 -- 172.21.15.121:0/562433195 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff89400c980 con 0x7ff8ac101750 2024-04-05T21:39:36.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.600+0000 7ff8a3fff700 1 --2- 172.21.15.121:0/562433195 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff89806cce0 0x7ff89806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:36.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.600+0000 7ff8a3fff700 1 -- 172.21.15.121:0/562433195 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff894014070 con 0x7ff8ac101750 2024-04-05T21:39:36.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.600+0000 7ff8aad9d700 1 --2- 172.21.15.121:0/562433195 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff89806cce0 0x7ff89806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:36.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.600+0000 7ff8aad9d700 1 --2- 172.21.15.121:0/562433195 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff89806cce0 0x7ff89806f1a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff89c000f80 tx=0x7ff89c0061f0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:36.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.604+0000 7ff8a3fff700 1 -- 172.21.15.121:0/562433195 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff894056a20 con 0x7ff8ac101750 2024-04-05T21:39:36.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.876+0000 7ff8b1175700 1 -- 172.21.15.121:0/562433195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff8ac04ea90 con 0x7ff8ac101750 2024-04-05T21:39:36.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.876+0000 7ff8a3fff700 1 -- 172.21.15.121:0/562433195 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7ff89400cc30 con 0x7ff8ac101750 2024-04-05T21:39:36.877 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:39:36.877 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:39:36.877 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:39:36.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:39:36.877 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:39:36.877 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:39:36.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:39:36.877 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:39:36.878 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:39:36.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:39:36.878 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:39:36.878 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:39:36.878 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:39:36.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:39:36.878 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:39:36.878 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:39:36.878 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:39:36.878 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:39:36.878 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:39:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.878+0000 7ff8b1175700 1 -- 172.21.15.121:0/562433195 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff89806cce0 msgr2=0x7ff89806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.878+0000 7ff8b1175700 1 --2- 172.21.15.121:0/562433195 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff89806cce0 0x7ff89806f1a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff89c000f80 tx=0x7ff89c0061f0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.878+0000 7ff8b1175700 1 -- 172.21.15.121:0/562433195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac101750 msgr2=0x7ff8ac196c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.878+0000 7ff8b1175700 1 --2- 172.21.15.121:0/562433195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac101750 0x7ff8ac196c30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff89400e8f0 tx=0x7ff89400ec00 comp rx=0 tx=0).stop 2024-04-05T21:39:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.878+0000 7ff8b1175700 1 -- 172.21.15.121:0/562433195 shutdown_connections 2024-04-05T21:39:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.878+0000 7ff8b1175700 1 --2- 172.21.15.121:0/562433195 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff89806cce0 0x7ff89806f1a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.878+0000 7ff8b1175700 1 --2- 172.21.15.121:0/562433195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8ac100da0 0x7ff8ac1966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.878+0000 7ff8b1175700 1 --2- 172.21.15.121:0/562433195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac101750 0x7ff8ac196c30 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.878+0000 7ff8b1175700 1 -- 172.21.15.121:0/562433195 >> 172.21.15.121:0/562433195 conn(0x7ff8ac0fc630 msgr2=0x7ff8ac0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:36.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.879+0000 7ff8b1175700 1 -- 172.21.15.121:0/562433195 shutdown_connections 2024-04-05T21:39:36.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.879+0000 7ff8b1175700 1 -- 172.21.15.121:0/562433195 wait complete. 2024-04-05T21:39:36.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.981+0000 7f5fe81db700 1 -- 172.21.15.121:0/1079137961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe0101d60 msgr2=0x7f5fe0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:36.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.981+0000 7f5fe81db700 1 --2- 172.21.15.121:0/1079137961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe0101d60 0x7f5fe0102140 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009940 tx=0x7f5fd0009c50 comp rx=0 tx=0).stop 2024-04-05T21:39:36.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.981+0000 7f5fe81db700 1 -- 172.21.15.121:0/1079137961 shutdown_connections 2024-04-05T21:39:36.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.981+0000 7f5fe81db700 1 --2- 172.21.15.121:0/1079137961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fe0102680 0x7f5fe010ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.981+0000 7f5fe81db700 1 --2- 172.21.15.121:0/1079137961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe0101d60 0x7f5fe0102140 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.981+0000 7f5fe81db700 1 -- 172.21.15.121:0/1079137961 >> 172.21.15.121:0/1079137961 conn(0x7f5fe00fb570 msgr2=0x7f5fe00fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:36.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.981+0000 7f5fe81db700 1 -- 172.21.15.121:0/1079137961 shutdown_connections 2024-04-05T21:39:36.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.981+0000 7f5fe81db700 1 -- 172.21.15.121:0/1079137961 wait complete. 2024-04-05T21:39:36.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.981+0000 7f5fe81db700 1 Processor -- start 2024-04-05T21:39:36.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.981+0000 7f5fe81db700 1 -- start start 2024-04-05T21:39:36.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe81db700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe0101d60 0x7f5fe0196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:36.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe81db700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fe0102680 0x7f5fe0196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:36.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe81db700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe0197250 con 0x7f5fe0101d60 2024-04-05T21:39:36.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe81db700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe0199f80 con 0x7f5fe0102680 2024-04-05T21:39:36.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe5f77700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe0101d60 0x7f5fe0196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:36.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe5f77700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe0101d60 0x7f5fe0196680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59918/0 (socket says 172.21.15.121:59918) 2024-04-05T21:39:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe5f77700 1 -- 172.21.15.121:0/1298658580 learned_addr learned my addr 172.21.15.121:0/1298658580 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe5776700 1 --2- 172.21.15.121:0/1298658580 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fe0102680 0x7f5fe0196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe5f77700 1 -- 172.21.15.121:0/1298658580 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fe0102680 msgr2=0x7f5fe0196bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe5f77700 1 --2- 172.21.15.121:0/1298658580 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fe0102680 0x7f5fe0196bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe5f77700 1 -- 172.21.15.121:0/1298658580 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd00095f0 con 0x7f5fe0101d60 2024-04-05T21:39:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe5f77700 1 --2- 172.21.15.121:0/1298658580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe0101d60 0x7f5fe0196680 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0006010 tx=0x7f5fd00056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fd6ffd700 1 -- 172.21.15.121:0/1298658580 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fd001d070 con 0x7f5fe0101d60 2024-04-05T21:39:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fd6ffd700 1 -- 172.21.15.121:0/1298658580 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fd000f460 con 0x7f5fe0101d60 2024-04-05T21:39:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe81db700 1 -- 172.21.15.121:0/1298658580 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe019a200 con 0x7f5fe0101d60 2024-04-05T21:39:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fd6ffd700 1 -- 172.21.15.121:0/1298658580 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fd0021620 con 0x7f5fe0101d60 2024-04-05T21:39:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.982+0000 7f5fe81db700 1 -- 172.21.15.121:0/1298658580 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe019a6f0 con 0x7f5fe0101d60 2024-04-05T21:39:36.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.983+0000 7f5fe81db700 1 -- 172.21.15.121:0/1298658580 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe0108270 con 0x7f5fe0101d60 2024-04-05T21:39:36.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.984+0000 7f5fd6ffd700 1 -- 172.21.15.121:0/1298658580 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5fd000f5d0 con 0x7f5fe0101d60 2024-04-05T21:39:36.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.984+0000 7f5fd6ffd700 1 --2- 172.21.15.121:0/1298658580 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fcc06cce0 0x7f5fcc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:36.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.984+0000 7f5fd6ffd700 1 -- 172.21.15.121:0/1298658580 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5fd008d860 con 0x7f5fe0101d60 2024-04-05T21:39:36.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.984+0000 7f5fe5776700 1 --2- 172.21.15.121:0/1298658580 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fcc06cce0 0x7f5fcc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:36.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.985+0000 7f5fe5776700 1 --2- 172.21.15.121:0/1298658580 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fcc06cce0 0x7f5fcc06f1a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc005fd0 tx=0x7f5fdc00a560 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:36.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:36.989+0000 7f5fd6ffd700 1 -- 172.21.15.121:0/1298658580 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5fd00578d0 con 0x7f5fe0101d60 2024-04-05T21:39:37.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.224+0000 7f5fe81db700 1 -- 172.21.15.121:0/1298658580 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5fe0107c50 con 0x7f5fe0101d60 2024-04-05T21:39:37.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.224+0000 7f5fd6ffd700 1 -- 172.21.15.121:0/1298658580 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f5fd005aef0 con 0x7f5fe0101d60 2024-04-05T21:39:37.225 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:39:37.225 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:39:37.225 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:39:37.225 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:39:37.225 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:39:37.225 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:39:37.225 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:39:37.225 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:39:37.226 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:39:37.227 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:39:37.228 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:39:37.228 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:39:37.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.226+0000 7f5fe81db700 1 -- 172.21.15.121:0/1298658580 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fcc06cce0 msgr2=0x7f5fcc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:37.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.226+0000 7f5fe81db700 1 --2- 172.21.15.121:0/1298658580 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fcc06cce0 0x7f5fcc06f1a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc005fd0 tx=0x7f5fdc00a560 comp rx=0 tx=0).stop 2024-04-05T21:39:37.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.226+0000 7f5fe81db700 1 -- 172.21.15.121:0/1298658580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe0101d60 msgr2=0x7f5fe0196680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:37.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.226+0000 7f5fe81db700 1 --2- 172.21.15.121:0/1298658580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe0101d60 0x7f5fe0196680 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0006010 tx=0x7f5fd00056a0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.226+0000 7f5fe81db700 1 -- 172.21.15.121:0/1298658580 shutdown_connections 2024-04-05T21:39:37.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.226+0000 7f5fe81db700 1 --2- 172.21.15.121:0/1298658580 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fcc06cce0 0x7f5fcc06f1a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.226+0000 7f5fe81db700 1 --2- 172.21.15.121:0/1298658580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fe0101d60 0x7f5fe0196680 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.226+0000 7f5fe81db700 1 --2- 172.21.15.121:0/1298658580 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fe0102680 0x7f5fe0196bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.226+0000 7f5fe81db700 1 -- 172.21.15.121:0/1298658580 >> 172.21.15.121:0/1298658580 conn(0x7f5fe00fb570 msgr2=0x7f5fe01053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:37.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.226+0000 7f5fe81db700 1 -- 172.21.15.121:0/1298658580 shutdown_connections 2024-04-05T21:39:37.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.226+0000 7f5fe81db700 1 -- 172.21.15.121:0/1298658580 wait complete. 2024-04-05T21:39:37.229 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:39:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.330+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2255228425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb4105a40 msgr2=0x7f6cb4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.330+0000 7f6cb8f39700 1 --2- 172.21.15.121:0/2255228425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb4105a40 0x7f6cb4105e20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009910 tx=0x7f6ca8009c20 comp rx=0 tx=0).stop 2024-04-05T21:39:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.330+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2255228425 shutdown_connections 2024-04-05T21:39:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.330+0000 7f6cb8f39700 1 --2- 172.21.15.121:0/2255228425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cb40ffa40 0x7f6cb40ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.330+0000 7f6cb8f39700 1 --2- 172.21.15.121:0/2255228425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb4105a40 0x7f6cb4105e20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.330+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2255228425 >> 172.21.15.121:0/2255228425 conn(0x7f6cb40fb570 msgr2=0x7f6cb40fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.330+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2255228425 shutdown_connections 2024-04-05T21:39:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.330+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2255228425 wait complete. 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb8f39700 1 Processor -- start 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb8f39700 1 -- start start 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb8f39700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cb40ffa40 0x7f6cb40fbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb8f39700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb4105a40 0x7f6cb40fc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb8f39700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb40fca00 con 0x7f6cb4105a40 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb8f39700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb40fcb70 con 0x7f6cb40ffa40 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb1d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb4105a40 0x7f6cb40fc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb1d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb4105a40 0x7f6cb40fc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59926/0 (socket says 172.21.15.121:59926) 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb1d9b700 1 -- 172.21.15.121:0/2328883469 learned_addr learned my addr 172.21.15.121:0/2328883469 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb259c700 1 --2- 172.21.15.121:0/2328883469 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cb40ffa40 0x7f6cb40fbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb1d9b700 1 -- 172.21.15.121:0/2328883469 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cb40ffa40 msgr2=0x7f6cb40fbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb1d9b700 1 --2- 172.21.15.121:0/2328883469 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cb40ffa40 0x7f6cb40fbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb1d9b700 1 -- 172.21.15.121:0/2328883469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca80095f0 con 0x7f6cb4105a40 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6cb1d9b700 1 --2- 172.21.15.121:0/2328883469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb4105a40 0x7f6cb40fc430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400c8a0 tx=0x7f6ca400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6ca37fe700 1 -- 172.21.15.121:0/2328883469 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ca400cea0 con 0x7f6cb4105a40 2024-04-05T21:39:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6ca37fe700 1 -- 172.21.15.121:0/2328883469 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ca4004830 con 0x7f6cb4105a40 2024-04-05T21:39:37.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.331+0000 7f6ca37fe700 1 -- 172.21.15.121:0/2328883469 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ca4005500 con 0x7f6cb4105a40 2024-04-05T21:39:37.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.332+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2328883469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb41a2840 con 0x7f6cb4105a40 2024-04-05T21:39:37.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.332+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2328883469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb41a2cb0 con 0x7f6cb4105a40 2024-04-05T21:39:37.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.333+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2328883469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb4109050 con 0x7f6cb4105a40 2024-04-05T21:39:37.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.333+0000 7f6ca37fe700 1 -- 172.21.15.121:0/2328883469 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6ca40049a0 con 0x7f6cb4105a40 2024-04-05T21:39:37.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.333+0000 7f6ca37fe700 1 --2- 172.21.15.121:0/2328883469 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6c9c06cce0 0x7f6c9c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:37.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.334+0000 7f6ca37fe700 1 -- 172.21.15.121:0/2328883469 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6ca408c250 con 0x7f6cb4105a40 2024-04-05T21:39:37.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.334+0000 7f6cb259c700 1 --2- 172.21.15.121:0/2328883469 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6c9c06cce0 0x7f6c9c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:37.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.334+0000 7f6cb259c700 1 --2- 172.21.15.121:0/2328883469 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6c9c06cce0 0x7f6c9c06f1a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8006010 tx=0x7f6ca801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:37.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.338+0000 7f6ca37fe700 1 -- 172.21.15.121:0/2328883469 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6ca40562c0 con 0x7f6cb4105a40 2024-04-05T21:39:37.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.548+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2328883469 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cb402cce0 con 0x7f6c9c06cce0 2024-04-05T21:39:37.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.549+0000 7f6ca37fe700 1 -- 172.21.15.121:0/2328883469 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6cb402cce0 con 0x7f6c9c06cce0 2024-04-05T21:39:37.550 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:39:37.550 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:39:37.550 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:39:37.550 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:39:37.550 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:39:37.550 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:39:37.550 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:39:37.551 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:39:37.551 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:39:37.551 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:39:37.551 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:39:37.551 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:39:37.551 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:39:37.551 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:39:37.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.551+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2328883469 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6c9c06cce0 msgr2=0x7f6c9c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:37.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.551+0000 7f6cb8f39700 1 --2- 172.21.15.121:0/2328883469 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6c9c06cce0 0x7f6c9c06f1a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8006010 tx=0x7f6ca801a040 comp rx=0 tx=0).stop 2024-04-05T21:39:37.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.551+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2328883469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb4105a40 msgr2=0x7f6cb40fc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:37.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.551+0000 7f6cb8f39700 1 --2- 172.21.15.121:0/2328883469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb4105a40 0x7f6cb40fc430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6ca400c8a0 tx=0x7f6ca400cc60 comp rx=0 tx=0).stop 2024-04-05T21:39:37.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.551+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2328883469 shutdown_connections 2024-04-05T21:39:37.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.551+0000 7f6cb8f39700 1 --2- 172.21.15.121:0/2328883469 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6c9c06cce0 0x7f6c9c06f1a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.551+0000 7f6cb8f39700 1 --2- 172.21.15.121:0/2328883469 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cb40ffa40 0x7f6cb40fbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.551+0000 7f6cb8f39700 1 --2- 172.21.15.121:0/2328883469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb4105a40 0x7f6cb40fc430 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.551+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2328883469 >> 172.21.15.121:0/2328883469 conn(0x7f6cb40fb570 msgr2=0x7f6cb4068f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:37.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.552+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2328883469 shutdown_connections 2024-04-05T21:39:37.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.552+0000 7f6cb8f39700 1 -- 172.21.15.121:0/2328883469 wait complete. 2024-04-05T21:39:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.653+0000 7f3ef5042700 1 -- 172.21.15.121:0/3016516061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ef00ffa50 msgr2=0x7f3ef00ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.653+0000 7f3ef5042700 1 --2- 172.21.15.121:0/3016516061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ef00ffa50 0x7f3ef00ffed0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4009910 tx=0x7f3ee4009c20 comp rx=0 tx=0).stop 2024-04-05T21:39:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.653+0000 7f3ef5042700 1 -- 172.21.15.121:0/3016516061 shutdown_connections 2024-04-05T21:39:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.653+0000 7f3ef5042700 1 --2- 172.21.15.121:0/3016516061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ef00ffa50 0x7f3ef00ffed0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.653+0000 7f3ef5042700 1 --2- 172.21.15.121:0/3016516061 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef0105a50 0x7f3ef0105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.653+0000 7f3ef5042700 1 -- 172.21.15.121:0/3016516061 >> 172.21.15.121:0/3016516061 conn(0x7f3ef00fb540 msgr2=0x7f3ef00fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.653+0000 7f3ef5042700 1 -- 172.21.15.121:0/3016516061 shutdown_connections 2024-04-05T21:39:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.654+0000 7f3ef5042700 1 -- 172.21.15.121:0/3016516061 wait complete. 2024-04-05T21:39:37.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.654+0000 7f3ef5042700 1 Processor -- start 2024-04-05T21:39:37.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.655+0000 7f3ef5042700 1 -- start start 2024-04-05T21:39:37.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.655+0000 7f3ef5042700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ef00ffa50 0x7f3ef01955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:37.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.655+0000 7f3ef5042700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef0105a50 0x7f3ef0195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:37.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.655+0000 7f3ef5042700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef0196200 con 0x7f3ef00ffa50 2024-04-05T21:39:37.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.655+0000 7f3ef5042700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef0199f90 con 0x7f3ef0105a50 2024-04-05T21:39:37.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.655+0000 7f3eeed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ef00ffa50 0x7f3ef01955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:37.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.655+0000 7f3eeed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ef00ffa50 0x7f3ef01955e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59936/0 (socket says 172.21.15.121:59936) 2024-04-05T21:39:37.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.655+0000 7f3eeed9d700 1 -- 172.21.15.121:0/1580448745 learned_addr learned my addr 172.21.15.121:0/1580448745 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:39:37.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.655+0000 7f3eee59c700 1 --2- 172.21.15.121:0/1580448745 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef0105a50 0x7f3ef0195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:37.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.655+0000 7f3eeed9d700 1 -- 172.21.15.121:0/1580448745 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef0105a50 msgr2=0x7f3ef0195b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:37.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.655+0000 7f3eeed9d700 1 --2- 172.21.15.121:0/1580448745 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef0105a50 0x7f3ef0195b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.655+0000 7f3eeed9d700 1 -- 172.21.15.121:0/1580448745 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee40095f0 con 0x7f3ef00ffa50 2024-04-05T21:39:37.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.655+0000 7f3eee59c700 1 --2- 172.21.15.121:0/1580448745 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef0105a50 0x7f3ef0195b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:39:37.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.656+0000 7f3eeed9d700 1 --2- 172.21.15.121:0/1580448745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ef00ffa50 0x7f3ef01955e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3ee000d8d0 tx=0x7f3ee000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:37.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.656+0000 7f3ed7fff700 1 -- 172.21.15.121:0/1580448745 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ee0009940 con 0x7f3ef00ffa50 2024-04-05T21:39:37.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.656+0000 7f3ed7fff700 1 -- 172.21.15.121:0/1580448745 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ee0010460 con 0x7f3ef00ffa50 2024-04-05T21:39:37.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.656+0000 7f3ed7fff700 1 -- 172.21.15.121:0/1580448745 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ee000f5f0 con 0x7f3ef00ffa50 2024-04-05T21:39:37.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.656+0000 7f3ef5042700 1 -- 172.21.15.121:0/1580448745 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef019a270 con 0x7f3ef00ffa50 2024-04-05T21:39:37.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.656+0000 7f3ef5042700 1 -- 172.21.15.121:0/1580448745 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef019a7c0 con 0x7f3ef00ffa50 2024-04-05T21:39:37.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.657+0000 7f3ef5042700 1 -- 172.21.15.121:0/1580448745 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef0107fe0 con 0x7f3ef00ffa50 2024-04-05T21:39:37.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.657+0000 7f3ed7fff700 1 -- 172.21.15.121:0/1580448745 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3ee00105d0 con 0x7f3ef00ffa50 2024-04-05T21:39:37.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.658+0000 7f3ed7fff700 1 --2- 172.21.15.121:0/1580448745 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ed806cce0 0x7f3ed806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:39:37.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.658+0000 7f3ed7fff700 1 -- 172.21.15.121:0/1580448745 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3ee008c560 con 0x7f3ef00ffa50 2024-04-05T21:39:37.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.658+0000 7f3eee59c700 1 --2- 172.21.15.121:0/1580448745 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ed806cce0 0x7f3ed806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:39:37.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.658+0000 7f3eee59c700 1 --2- 172.21.15.121:0/1580448745 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ed806cce0 0x7f3ed806f1a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400b5c0 tx=0x7f3ee4005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:39:37.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.662+0000 7f3ed7fff700 1 -- 172.21.15.121:0/1580448745 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3ee00566e0 con 0x7f3ef00ffa50 2024-04-05T21:39:37.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:37 smithi121 ceph-mon[145859]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:37 smithi121 ceph-mon[145859]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:37 smithi121 ceph-mon[145859]: pgmap v401: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:37 smithi121 ceph-mon[145859]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:37 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/562433195' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:39:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:37 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1298658580' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:39:37.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:37 smithi188 ceph-mon[107793]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:37 smithi188 ceph-mon[107793]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:37 smithi188 ceph-mon[107793]: pgmap v401: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:37 smithi188 ceph-mon[107793]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:37 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/562433195' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:39:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:37 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1298658580' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:39:37.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.935+0000 7f3ef5042700 1 -- 172.21.15.121:0/1580448745 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3ef004ea90 con 0x7f3ef00ffa50 2024-04-05T21:39:37.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.936+0000 7f3ed7fff700 1 -- 172.21.15.121:0/1580448745 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f3ee0010880 con 0x7f3ef00ffa50 2024-04-05T21:39:37.936 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:39:37.937 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:39:37.937 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:39:37.937 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:39:37.937 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:39:37.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.938+0000 7f3ef5042700 1 -- 172.21.15.121:0/1580448745 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ed806cce0 msgr2=0x7f3ed806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:37.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.938+0000 7f3ef5042700 1 --2- 172.21.15.121:0/1580448745 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ed806cce0 0x7f3ed806f1a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400b5c0 tx=0x7f3ee4005960 comp rx=0 tx=0).stop 2024-04-05T21:39:37.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.938+0000 7f3ef5042700 1 -- 172.21.15.121:0/1580448745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ef00ffa50 msgr2=0x7f3ef01955e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:39:37.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.938+0000 7f3ef5042700 1 --2- 172.21.15.121:0/1580448745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ef00ffa50 0x7f3ef01955e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3ee000d8d0 tx=0x7f3ee000dc90 comp rx=0 tx=0).stop 2024-04-05T21:39:37.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.938+0000 7f3ef5042700 1 -- 172.21.15.121:0/1580448745 shutdown_connections 2024-04-05T21:39:37.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.938+0000 7f3ef5042700 1 --2- 172.21.15.121:0/1580448745 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ed806cce0 0x7f3ed806f1a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.938+0000 7f3ef5042700 1 --2- 172.21.15.121:0/1580448745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ef00ffa50 0x7f3ef01955e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.938+0000 7f3ef5042700 1 --2- 172.21.15.121:0/1580448745 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3ef0105a50 0x7f3ef0195b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:39:37.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.938+0000 7f3ef5042700 1 -- 172.21.15.121:0/1580448745 >> 172.21.15.121:0/1580448745 conn(0x7f3ef00fb540 msgr2=0x7f3ef00fc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:39:37.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.938+0000 7f3ef5042700 1 -- 172.21.15.121:0/1580448745 shutdown_connections 2024-04-05T21:39:37.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:39:37.938+0000 7f3ef5042700 1 -- 172.21.15.121:0/1580448745 wait complete. 2024-04-05T21:39:38.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:38 smithi121 ceph-mon[145859]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:38.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:38 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1580448745' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:39:38.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:38 smithi188 ceph-mon[107793]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:39:38.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:38 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1580448745' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:39:39.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:39 smithi121 ceph-mon[145859]: pgmap v402: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:39.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:39 smithi188 ceph-mon[107793]: pgmap v402: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:41.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:41 smithi121 ceph-mon[145859]: pgmap v403: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:41.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:41 smithi188 ceph-mon[107793]: pgmap v403: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:43.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:43 smithi188 ceph-mon[107793]: pgmap v404: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:43.905 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:43 smithi121 ceph-mon[145859]: pgmap v404: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:45 smithi188 ceph-mon[107793]: pgmap v405: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:39:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:45 smithi121 ceph-mon[145859]: pgmap v405: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:39:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:47 smithi188 ceph-mon[107793]: pgmap v406: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:47.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:47 smithi121 ceph-mon[145859]: pgmap v406: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:49.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:49 smithi188 ceph-mon[107793]: pgmap v407: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:49 smithi121 ceph-mon[145859]: pgmap v407: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:51 smithi188 ceph-mon[107793]: pgmap v408: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:51.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:51 smithi121 ceph-mon[145859]: pgmap v408: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:53 smithi188 ceph-mon[107793]: pgmap v409: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:53.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:53 smithi121 ceph-mon[145859]: pgmap v409: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:55 smithi188 ceph-mon[107793]: pgmap v410: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:55.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:55 smithi121 ceph-mon[145859]: pgmap v410: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:57 smithi188 ceph-mon[107793]: pgmap v411: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:57.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:57 smithi121 ceph-mon[145859]: pgmap v411: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:39:59 smithi188 ceph-mon[107793]: pgmap v412: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:39:59.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:39:59 smithi121 ceph-mon[145859]: pgmap v412: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:39:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:40:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:39:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:40:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:39:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-05T21:40:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:39:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:40:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:39:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-05T21:40:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:00 smithi121 ceph-mon[145859]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:40:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:00 smithi121 ceph-mon[145859]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:40:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:00 smithi121 ceph-mon[145859]: fs cephfs is degraded 2024-04-05T21:40:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:00 smithi121 ceph-mon[145859]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:40:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:00 smithi121 ceph-mon[145859]: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:40:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:40:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:00 smithi188 ceph-mon[107793]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:40:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:00 smithi188 ceph-mon[107793]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:40:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:00 smithi188 ceph-mon[107793]: fs cephfs is degraded 2024-04-05T21:40:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:00 smithi188 ceph-mon[107793]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:40:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:00 smithi188 ceph-mon[107793]: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:40:01.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:40:01.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:01 smithi121 ceph-mon[145859]: pgmap v413: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:02.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:01 smithi188 ceph-mon[107793]: pgmap v413: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:03.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:03 smithi121 ceph-mon[145859]: pgmap v414: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:03.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:03 smithi188 ceph-mon[107793]: pgmap v414: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:05.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:05 smithi121 ceph-mon[145859]: pgmap v415: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:05 smithi188 ceph-mon[107793]: pgmap v415: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:07 smithi121 ceph-mon[145859]: pgmap v416: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:07 smithi188 ceph-mon[107793]: pgmap v416: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:08.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.053+0000 7f7e3222a700 1 -- 172.21.15.121:0/4037868202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e2c101790 msgr2=0x7f7e2c1057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:08.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.053+0000 7f7e3222a700 1 --2- 172.21.15.121:0/4037868202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e2c101790 0x7f7e2c1057e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009910 tx=0x7f7e14009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:08.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.053+0000 7f7e3222a700 1 -- 172.21.15.121:0/4037868202 shutdown_connections 2024-04-05T21:40:08.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.053+0000 7f7e3222a700 1 --2- 172.21.15.121:0/4037868202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e2c101790 0x7f7e2c1057e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.053+0000 7f7e3222a700 1 --2- 172.21.15.121:0/4037868202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e2c100de0 0x7f7e2c1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.053+0000 7f7e3222a700 1 -- 172.21.15.121:0/4037868202 >> 172.21.15.121:0/4037868202 conn(0x7f7e2c0fc6b0 msgr2=0x7f7e2c0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:08.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.053+0000 7f7e3222a700 1 -- 172.21.15.121:0/4037868202 shutdown_connections 2024-04-05T21:40:08.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.053+0000 7f7e3222a700 1 -- 172.21.15.121:0/4037868202 wait complete. 2024-04-05T21:40:08.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.053+0000 7f7e3222a700 1 Processor -- start 2024-04-05T21:40:08.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.053+0000 7f7e3222a700 1 -- start start 2024-04-05T21:40:08.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.054+0000 7f7e3222a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e2c100de0 0x7f7e2c073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:08.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.054+0000 7f7e3222a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e2c101790 0x7f7e2c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:08.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.054+0000 7f7e3222a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c073b10 con 0x7f7e2c101790 2024-04-05T21:40:08.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.054+0000 7f7e3222a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c073c50 con 0x7f7e2c100de0 2024-04-05T21:40:08.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.054+0000 7f7e22ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e2c101790 0x7f7e2c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:08.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.054+0000 7f7e22ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e2c101790 0x7f7e2c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37506/0 (socket says 172.21.15.121:37506) 2024-04-05T21:40:08.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.054+0000 7f7e22ffd700 1 -- 172.21.15.121:0/2477215285 learned_addr learned my addr 172.21.15.121:0/2477215285 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:08.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.054+0000 7f7e2b7fe700 1 --2- 172.21.15.121:0/2477215285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e2c100de0 0x7f7e2c073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:08.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.054+0000 7f7e22ffd700 1 -- 172.21.15.121:0/2477215285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e2c100de0 msgr2=0x7f7e2c073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:08.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.054+0000 7f7e22ffd700 1 --2- 172.21.15.121:0/2477215285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e2c100de0 0x7f7e2c073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.054+0000 7f7e22ffd700 1 -- 172.21.15.121:0/2477215285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e140095f0 con 0x7f7e2c101790 2024-04-05T21:40:08.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.054+0000 7f7e2b7fe700 1 --2- 172.21.15.121:0/2477215285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e2c100de0 0x7f7e2c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:40:08.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.054+0000 7f7e22ffd700 1 --2- 172.21.15.121:0/2477215285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e2c101790 0x7f7e2c0735d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7e140092e0 tx=0x7f7e1400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:08.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.055+0000 7f7e297fa700 1 -- 172.21.15.121:0/2477215285 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e1401d070 con 0x7f7e2c101790 2024-04-05T21:40:08.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.055+0000 7f7e297fa700 1 -- 172.21.15.121:0/2477215285 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e1400f460 con 0x7f7e2c101790 2024-04-05T21:40:08.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.055+0000 7f7e3222a700 1 -- 172.21.15.121:0/2477215285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c073ed0 con 0x7f7e2c101790 2024-04-05T21:40:08.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.055+0000 7f7e297fa700 1 -- 172.21.15.121:0/2477215285 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e14021600 con 0x7f7e2c101790 2024-04-05T21:40:08.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.055+0000 7f7e3222a700 1 -- 172.21.15.121:0/2477215285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c1a2b70 con 0x7f7e2c101790 2024-04-05T21:40:08.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.056+0000 7f7e3222a700 1 -- 172.21.15.121:0/2477215285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c1091e0 con 0x7f7e2c101790 2024-04-05T21:40:08.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.056+0000 7f7e297fa700 1 -- 172.21.15.121:0/2477215285 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7e1400f5d0 con 0x7f7e2c101790 2024-04-05T21:40:08.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.057+0000 7f7e297fa700 1 --2- 172.21.15.121:0/2477215285 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7e1806cce0 0x7f7e1806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:08.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.057+0000 7f7e2b7fe700 1 --2- 172.21.15.121:0/2477215285 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7e1806cce0 0x7f7e1806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:08.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.057+0000 7f7e297fa700 1 -- 172.21.15.121:0/2477215285 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7e1408d7b0 con 0x7f7e2c101790 2024-04-05T21:40:08.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.057+0000 7f7e2b7fe700 1 --2- 172.21.15.121:0/2477215285 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7e1806cce0 0x7f7e1806f1a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c005950 tx=0x7f7e1c005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:08.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.061+0000 7f7e297fa700 1 -- 172.21.15.121:0/2477215285 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7e14057820 con 0x7f7e2c101790 2024-04-05T21:40:08.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.269+0000 7f7e3222a700 1 -- 172.21.15.121:0/2477215285 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e2c02cce0 con 0x7f7e1806cce0 2024-04-05T21:40:08.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.270+0000 7f7e297fa700 1 -- 172.21.15.121:0/2477215285 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7e2c02cce0 con 0x7f7e1806cce0 2024-04-05T21:40:08.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.273+0000 7f7e3222a700 1 -- 172.21.15.121:0/2477215285 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7e1806cce0 msgr2=0x7f7e1806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:08.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.273+0000 7f7e3222a700 1 --2- 172.21.15.121:0/2477215285 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7e1806cce0 0x7f7e1806f1a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c005950 tx=0x7f7e1c005f50 comp rx=0 tx=0).stop 2024-04-05T21:40:08.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.273+0000 7f7e3222a700 1 -- 172.21.15.121:0/2477215285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e2c101790 msgr2=0x7f7e2c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:08.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.273+0000 7f7e3222a700 1 --2- 172.21.15.121:0/2477215285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e2c101790 0x7f7e2c0735d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7e140092e0 tx=0x7f7e1400be20 comp rx=0 tx=0).stop 2024-04-05T21:40:08.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.273+0000 7f7e3222a700 1 -- 172.21.15.121:0/2477215285 shutdown_connections 2024-04-05T21:40:08.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.273+0000 7f7e3222a700 1 --2- 172.21.15.121:0/2477215285 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7e1806cce0 0x7f7e1806f1a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.273+0000 7f7e3222a700 1 --2- 172.21.15.121:0/2477215285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7e2c100de0 0x7f7e2c073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.273+0000 7f7e3222a700 1 --2- 172.21.15.121:0/2477215285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e2c101790 0x7f7e2c0735d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.273+0000 7f7e3222a700 1 -- 172.21.15.121:0/2477215285 >> 172.21.15.121:0/2477215285 conn(0x7f7e2c0fc6b0 msgr2=0x7f7e2c0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:08.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.273+0000 7f7e3222a700 1 -- 172.21.15.121:0/2477215285 shutdown_connections 2024-04-05T21:40:08.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.273+0000 7f7e3222a700 1 -- 172.21.15.121:0/2477215285 wait complete. 2024-04-05T21:40:08.285 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:40:08.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.378+0000 7ff1f9b52700 1 -- 172.21.15.121:0/3386099150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f4101790 msgr2=0x7ff1f41057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:08.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.378+0000 7ff1f9b52700 1 --2- 172.21.15.121:0/3386099150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f4101790 0x7ff1f41057e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4009910 tx=0x7ff1e4009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:08.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.378+0000 7ff1f9b52700 1 -- 172.21.15.121:0/3386099150 shutdown_connections 2024-04-05T21:40:08.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.378+0000 7ff1f9b52700 1 --2- 172.21.15.121:0/3386099150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f4101790 0x7ff1f41057e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.378+0000 7ff1f9b52700 1 --2- 172.21.15.121:0/3386099150 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1f4100de0 0x7ff1f41011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.378+0000 7ff1f9b52700 1 -- 172.21.15.121:0/3386099150 >> 172.21.15.121:0/3386099150 conn(0x7ff1f40fc6b0 msgr2=0x7ff1f40fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:08.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.378+0000 7ff1f9b52700 1 -- 172.21.15.121:0/3386099150 shutdown_connections 2024-04-05T21:40:08.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.378+0000 7ff1f9b52700 1 -- 172.21.15.121:0/3386099150 wait complete. 2024-04-05T21:40:08.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.378+0000 7ff1f9b52700 1 Processor -- start 2024-04-05T21:40:08.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f9b52700 1 -- start start 2024-04-05T21:40:08.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f9b52700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1f4100de0 0x7ff1f4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:08.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f9b52700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f4101790 0x7ff1f40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:08.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f9b52700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f4073b10 con 0x7ff1f4101790 2024-04-05T21:40:08.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f9b52700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f4073c50 con 0x7ff1f4100de0 2024-04-05T21:40:08.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f4101790 0x7ff1f40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:08.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f4101790 0x7ff1f40735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37528/0 (socket says 172.21.15.121:37528) 2024-04-05T21:40:08.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f2ffd700 1 -- 172.21.15.121:0/2475649818 learned_addr learned my addr 172.21.15.121:0/2475649818 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:08.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f37fe700 1 --2- 172.21.15.121:0/2475649818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1f4100de0 0x7ff1f4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:08.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f2ffd700 1 -- 172.21.15.121:0/2475649818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1f4100de0 msgr2=0x7ff1f4073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:08.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f2ffd700 1 --2- 172.21.15.121:0/2475649818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1f4100de0 0x7ff1f4073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f2ffd700 1 -- 172.21.15.121:0/2475649818 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e40095f0 con 0x7ff1f4101790 2024-04-05T21:40:08.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f2ffd700 1 --2- 172.21.15.121:0/2475649818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f4101790 0x7ff1f40735d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4000c00 tx=0x7ff1e400ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:08.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f0ff9700 1 -- 172.21.15.121:0/2475649818 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1e401d070 con 0x7ff1f4101790 2024-04-05T21:40:08.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f9b52700 1 -- 172.21.15.121:0/2475649818 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f4073ed0 con 0x7ff1f4101790 2024-04-05T21:40:08.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f0ff9700 1 -- 172.21.15.121:0/2475649818 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1e400bed0 con 0x7ff1f4101790 2024-04-05T21:40:08.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f0ff9700 1 -- 172.21.15.121:0/2475649818 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1e400fd90 con 0x7ff1f4101790 2024-04-05T21:40:08.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.379+0000 7ff1f9b52700 1 -- 172.21.15.121:0/2475649818 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f41a2b10 con 0x7ff1f4101790 2024-04-05T21:40:08.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.381+0000 7ff1f9b52700 1 -- 172.21.15.121:0/2475649818 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f41091e0 con 0x7ff1f4101790 2024-04-05T21:40:08.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.381+0000 7ff1f0ff9700 1 -- 172.21.15.121:0/2475649818 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff1e4021410 con 0x7ff1f4101790 2024-04-05T21:40:08.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.381+0000 7ff1f0ff9700 1 --2- 172.21.15.121:0/2475649818 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1dc06cce0 0x7ff1dc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:08.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.381+0000 7ff1f0ff9700 1 -- 172.21.15.121:0/2475649818 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff1e408d760 con 0x7ff1f4101790 2024-04-05T21:40:08.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.381+0000 7ff1f37fe700 1 --2- 172.21.15.121:0/2475649818 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1dc06cce0 0x7ff1dc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:08.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.382+0000 7ff1f37fe700 1 --2- 172.21.15.121:0/2475649818 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1dc06cce0 0x7ff1dc06f1a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8009c60 tx=0x7ff1e8009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:08.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.386+0000 7ff1f0ff9700 1 -- 172.21.15.121:0/2475649818 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff1e40577d0 con 0x7ff1f4101790 2024-04-05T21:40:08.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.595+0000 7ff1f9b52700 1 -- 172.21.15.121:0/2475649818 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1f402cd30 con 0x7ff1dc06cce0 2024-04-05T21:40:08.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.597+0000 7ff1f0ff9700 1 -- 172.21.15.121:0/2475649818 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff1f402cd30 con 0x7ff1dc06cce0 2024-04-05T21:40:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.599+0000 7ff1f9b52700 1 -- 172.21.15.121:0/2475649818 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1dc06cce0 msgr2=0x7ff1dc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.599+0000 7ff1f9b52700 1 --2- 172.21.15.121:0/2475649818 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1dc06cce0 0x7ff1dc06f1a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8009c60 tx=0x7ff1e8009260 comp rx=0 tx=0).stop 2024-04-05T21:40:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.599+0000 7ff1f9b52700 1 -- 172.21.15.121:0/2475649818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f4101790 msgr2=0x7ff1f40735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.599+0000 7ff1f9b52700 1 --2- 172.21.15.121:0/2475649818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f4101790 0x7ff1f40735d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4000c00 tx=0x7ff1e400ba70 comp rx=0 tx=0).stop 2024-04-05T21:40:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.599+0000 7ff1f9b52700 1 -- 172.21.15.121:0/2475649818 shutdown_connections 2024-04-05T21:40:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.599+0000 7ff1f9b52700 1 --2- 172.21.15.121:0/2475649818 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1dc06cce0 0x7ff1dc06f1a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.599+0000 7ff1f9b52700 1 --2- 172.21.15.121:0/2475649818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1f4100de0 0x7ff1f4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.599+0000 7ff1f9b52700 1 --2- 172.21.15.121:0/2475649818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f4101790 0x7ff1f40735d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.599+0000 7ff1f9b52700 1 -- 172.21.15.121:0/2475649818 >> 172.21.15.121:0/2475649818 conn(0x7ff1f40fc6b0 msgr2=0x7ff1f40fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.599+0000 7ff1f9b52700 1 -- 172.21.15.121:0/2475649818 shutdown_connections 2024-04-05T21:40:08.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.599+0000 7ff1f9b52700 1 -- 172.21.15.121:0/2475649818 wait complete. 2024-04-05T21:40:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.702+0000 7fa110a35700 1 -- 172.21.15.121:0/3885537539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10c0ffbd0 msgr2=0x7fa10c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.702+0000 7fa110a35700 1 --2- 172.21.15.121:0/3885537539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10c0ffbd0 0x7fa10c100050 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc009910 tx=0x7fa0fc009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.702+0000 7fa110a35700 1 -- 172.21.15.121:0/3885537539 shutdown_connections 2024-04-05T21:40:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.702+0000 7fa110a35700 1 --2- 172.21.15.121:0/3885537539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10c0ffbd0 0x7fa10c100050 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.702+0000 7fa110a35700 1 --2- 172.21.15.121:0/3885537539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa10c105a40 0x7fa10c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.702+0000 7fa110a35700 1 -- 172.21.15.121:0/3885537539 >> 172.21.15.121:0/3885537539 conn(0x7fa10c0755a0 msgr2=0x7fa10c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.702+0000 7fa110a35700 1 -- 172.21.15.121:0/3885537539 shutdown_connections 2024-04-05T21:40:08.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.702+0000 7fa110a35700 1 -- 172.21.15.121:0/3885537539 wait complete. 2024-04-05T21:40:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.703+0000 7fa110a35700 1 Processor -- start 2024-04-05T21:40:08.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.703+0000 7fa110a35700 1 -- start start 2024-04-05T21:40:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.703+0000 7fa110a35700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10c0ffbd0 0x7fa10c195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.703+0000 7fa110a35700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa10c105a40 0x7fa10c195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.703+0000 7fa110a35700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10c196160 con 0x7fa10c0ffbd0 2024-04-05T21:40:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.703+0000 7fa110a35700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10c199ef0 con 0x7fa10c105a40 2024-04-05T21:40:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.703+0000 7fa10a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10c0ffbd0 0x7fa10c195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.703+0000 7fa10a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10c0ffbd0 0x7fa10c195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37544/0 (socket says 172.21.15.121:37544) 2024-04-05T21:40:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.703+0000 7fa10a59c700 1 -- 172.21.15.121:0/4150821732 learned_addr learned my addr 172.21.15.121:0/4150821732 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.703+0000 7fa10a59c700 1 -- 172.21.15.121:0/4150821732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa10c105a40 msgr2=0x7fa10c195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.703+0000 7fa109d9b700 1 --2- 172.21.15.121:0/4150821732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa10c105a40 0x7fa10c195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.703+0000 7fa10a59c700 1 --2- 172.21.15.121:0/4150821732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa10c105a40 0x7fa10c195a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.703+0000 7fa10a59c700 1 -- 172.21.15.121:0/4150821732 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0fc0095f0 con 0x7fa10c0ffbd0 2024-04-05T21:40:08.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.704+0000 7fa10a59c700 1 --2- 172.21.15.121:0/4150821732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10c0ffbd0 0x7fa10c195540 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa100007b50 tx=0x7fa100007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.704+0000 7fa0fb7fe700 1 -- 172.21.15.121:0/4150821732 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa100011070 con 0x7fa10c0ffbd0 2024-04-05T21:40:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.704+0000 7fa0fb7fe700 1 -- 172.21.15.121:0/4150821732 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa10000c2e0 con 0x7fa10c0ffbd0 2024-04-05T21:40:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.704+0000 7fa110a35700 1 -- 172.21.15.121:0/4150821732 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa10c19a170 con 0x7fa10c0ffbd0 2024-04-05T21:40:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.704+0000 7fa0fb7fe700 1 -- 172.21.15.121:0/4150821732 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa10000e770 con 0x7fa10c0ffbd0 2024-04-05T21:40:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.704+0000 7fa110a35700 1 -- 172.21.15.121:0/4150821732 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa10c19a6c0 con 0x7fa10c0ffbd0 2024-04-05T21:40:08.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.705+0000 7fa110a35700 1 -- 172.21.15.121:0/4150821732 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa10c04ea90 con 0x7fa10c0ffbd0 2024-04-05T21:40:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.705+0000 7fa0fb7fe700 1 -- 172.21.15.121:0/4150821732 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa100004a00 con 0x7fa10c0ffbd0 2024-04-05T21:40:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.706+0000 7fa0fb7fe700 1 --2- 172.21.15.121:0/4150821732 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa0f406cce0 0x7fa0f406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.706+0000 7fa109d9b700 1 --2- 172.21.15.121:0/4150821732 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa0f406cce0 0x7fa0f406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.706+0000 7fa0fb7fe700 1 -- 172.21.15.121:0/4150821732 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa10008dd10 con 0x7fa10c0ffbd0 2024-04-05T21:40:08.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.706+0000 7fa109d9b700 1 --2- 172.21.15.121:0/4150821732 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa0f406cce0 0x7fa0f406f1a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00b5c0 tx=0x7fa0fc005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:08.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.710+0000 7fa0fb7fe700 1 -- 172.21.15.121:0/4150821732 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa100057d80 con 0x7fa10c0ffbd0 2024-04-05T21:40:08.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.911+0000 7fa110a35700 1 -- 172.21.15.121:0/4150821732 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa10c19ab90 con 0x7fa0f406cce0 2024-04-05T21:40:08.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.921+0000 7fa0fb7fe700 1 -- 172.21.15.121:0/4150821732 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fa10c19ab90 con 0x7fa0f406cce0 2024-04-05T21:40:08.923 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:40:08.923 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (10m) 4m ago 22m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:40:08.923 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (22m) 4m ago 22m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:40:08.923 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (21m) 4m ago 21m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:40:08.923 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (9m) 4m ago 22m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:40:08.923 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (9m) 4m ago 21m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:40:08.923 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (10m) 4m ago 22m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:40:08.923 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (5m) 4m ago 16m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:40:08.923 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (5m) 4m ago 16m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:40:08.923 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (4m) 4m ago 16m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:40:08.923 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (16m) 4m ago 16m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:40:08.924 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (12m) 4m ago 24m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:40:08.924 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (12m) 4m ago 21m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:40:08.924 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (9m) 4m ago 24m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:40:08.924 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (9m) 4m ago 21m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:40:08.924 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (11m) 4m ago 22m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:40:08.924 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (11m) 4m ago 21m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:40:08.924 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (8m) 4m ago 20m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:40:08.924 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (8m) 4m ago 19m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:40:08.924 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (7m) 4m ago 19m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:40:08.924 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (6m) 4m ago 18m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:40:08.924 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (6m) 4m ago 18m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:40:08.924 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (5m) 4m ago 17m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:40:08.924 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (11m) 4m ago 21m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:40:08.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.923+0000 7fa110a35700 1 -- 172.21.15.121:0/4150821732 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa0f406cce0 msgr2=0x7fa0f406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:08.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.923+0000 7fa110a35700 1 --2- 172.21.15.121:0/4150821732 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa0f406cce0 0x7fa0f406f1a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc00b5c0 tx=0x7fa0fc005fb0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.923+0000 7fa110a35700 1 -- 172.21.15.121:0/4150821732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10c0ffbd0 msgr2=0x7fa10c195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:08.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.923+0000 7fa110a35700 1 --2- 172.21.15.121:0/4150821732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10c0ffbd0 0x7fa10c195540 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa100007b50 tx=0x7fa100007f10 comp rx=0 tx=0).stop 2024-04-05T21:40:08.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.923+0000 7fa110a35700 1 -- 172.21.15.121:0/4150821732 shutdown_connections 2024-04-05T21:40:08.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.923+0000 7fa110a35700 1 --2- 172.21.15.121:0/4150821732 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa0f406cce0 0x7fa0f406f1a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.923+0000 7fa110a35700 1 --2- 172.21.15.121:0/4150821732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10c0ffbd0 0x7fa10c195540 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.923+0000 7fa110a35700 1 --2- 172.21.15.121:0/4150821732 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa10c105a40 0x7fa10c195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:08.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.923+0000 7fa110a35700 1 -- 172.21.15.121:0/4150821732 >> 172.21.15.121:0/4150821732 conn(0x7fa10c0755a0 msgr2=0x7fa10c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:08.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.923+0000 7fa110a35700 1 -- 172.21.15.121:0/4150821732 shutdown_connections 2024-04-05T21:40:08.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:08.923+0000 7fa110a35700 1 -- 172.21.15.121:0/4150821732 wait complete. 2024-04-05T21:40:09.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.024+0000 7fd00d8b1700 1 -- 172.21.15.121:0/3563140539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0080ff310 msgr2=0x7fd0080ff770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.024+0000 7fd00d8b1700 1 --2- 172.21.15.121:0/3563140539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0080ff310 0x7fd0080ff770 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009910 tx=0x7fcffc009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.024+0000 7fd00d8b1700 1 -- 172.21.15.121:0/3563140539 shutdown_connections 2024-04-05T21:40:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.024+0000 7fd00d8b1700 1 --2- 172.21.15.121:0/3563140539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0080ff310 0x7fd0080ff770 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.024+0000 7fd00d8b1700 1 --2- 172.21.15.121:0/3563140539 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd008106c10 0x7fd008106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.024+0000 7fd00d8b1700 1 -- 172.21.15.121:0/3563140539 >> 172.21.15.121:0/3563140539 conn(0x7fd008074bf0 msgr2=0x7fd008075000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.025+0000 7fd00d8b1700 1 -- 172.21.15.121:0/3563140539 shutdown_connections 2024-04-05T21:40:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.025+0000 7fd00d8b1700 1 -- 172.21.15.121:0/3563140539 wait complete. 2024-04-05T21:40:09.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.025+0000 7fd00d8b1700 1 Processor -- start 2024-04-05T21:40:09.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.025+0000 7fd00d8b1700 1 -- start start 2024-04-05T21:40:09.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.025+0000 7fd00d8b1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0080ff310 0x7fd0081004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:09.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.025+0000 7fd00d8b1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd008106c10 0x7fd008100a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:09.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd00d8b1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0081022a0 con 0x7fd008106c10 2024-04-05T21:40:09.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd00d8b1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd008102410 con 0x7fd0080ff310 2024-04-05T21:40:09.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd0067fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd008106c10 0x7fd008100a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:09.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd0067fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd008106c10 0x7fd008100a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37548/0 (socket says 172.21.15.121:37548) 2024-04-05T21:40:09.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd0067fc700 1 -- 172.21.15.121:0/2093491760 learned_addr learned my addr 172.21.15.121:0/2093491760 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:09.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd006ffd700 1 --2- 172.21.15.121:0/2093491760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0080ff310 0x7fd0081004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:09.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd0067fc700 1 -- 172.21.15.121:0/2093491760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0080ff310 msgr2=0x7fd0081004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:09.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd0067fc700 1 --2- 172.21.15.121:0/2093491760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0080ff310 0x7fd0081004f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd0067fc700 1 -- 172.21.15.121:0/2093491760 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc0095f0 con 0x7fd008106c10 2024-04-05T21:40:09.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd0067fc700 1 --2- 172.21.15.121:0/2093491760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd008106c10 0x7fd008100a30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcffc006010 tx=0x7fcffc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:09.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd00c8af700 1 -- 172.21.15.121:0/2093491760 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcffc01d070 con 0x7fd008106c10 2024-04-05T21:40:09.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd00c8af700 1 -- 172.21.15.121:0/2093491760 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcffc005710 con 0x7fd008106c10 2024-04-05T21:40:09.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd00d8b1700 1 -- 172.21.15.121:0/2093491760 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd008100fd0 con 0x7fd008106c10 2024-04-05T21:40:09.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd00c8af700 1 -- 172.21.15.121:0/2093491760 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcffc00fe20 con 0x7fd008106c10 2024-04-05T21:40:09.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.026+0000 7fd00d8b1700 1 -- 172.21.15.121:0/2093491760 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0081a7010 con 0x7fd008106c10 2024-04-05T21:40:09.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.027+0000 7fd00d8b1700 1 -- 172.21.15.121:0/2093491760 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd008109110 con 0x7fd008106c10 2024-04-05T21:40:09.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.028+0000 7fd00c8af700 1 -- 172.21.15.121:0/2093491760 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcffc003a60 con 0x7fd008106c10 2024-04-05T21:40:09.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.028+0000 7fd00c8af700 1 --2- 172.21.15.121:0/2093491760 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcff006cce0 0x7fcff006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:09.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.028+0000 7fd00c8af700 1 -- 172.21.15.121:0/2093491760 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fcffc08d790 con 0x7fd008106c10 2024-04-05T21:40:09.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.028+0000 7fd006ffd700 1 --2- 172.21.15.121:0/2093491760 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcff006cce0 0x7fcff006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:09.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.029+0000 7fd006ffd700 1 --2- 172.21.15.121:0/2093491760 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcff006cce0 0x7fcff006f1a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcff8005950 tx=0x7fcff800b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:09.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.033+0000 7fd00c8af700 1 -- 172.21.15.121:0/2093491760 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcffc057800 con 0x7fd008106c10 2024-04-05T21:40:09.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.311+0000 7fd00d8b1700 1 -- 172.21.15.121:0/2093491760 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd00804ea90 con 0x7fd008106c10 2024-04-05T21:40:09.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.312+0000 7fd00c8af700 1 -- 172.21.15.121:0/2093491760 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7fcffc003d10 con 0x7fd008106c10 2024-04-05T21:40:09.314 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:40:09.314 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:40:09.314 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:40:09.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:40:09.314 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:40:09.314 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:40:09.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:40:09.314 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:40:09.314 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:40:09.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:40:09.314 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:40:09.315 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:40:09.315 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:40:09.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:40:09.315 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:40:09.315 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:40:09.315 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:40:09.315 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:40:09.315 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:40:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.314+0000 7fd00d8b1700 1 -- 172.21.15.121:0/2093491760 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcff006cce0 msgr2=0x7fcff006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.314+0000 7fd00d8b1700 1 --2- 172.21.15.121:0/2093491760 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcff006cce0 0x7fcff006f1a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcff8005950 tx=0x7fcff800b410 comp rx=0 tx=0).stop 2024-04-05T21:40:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.314+0000 7fd00d8b1700 1 -- 172.21.15.121:0/2093491760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd008106c10 msgr2=0x7fd008100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.314+0000 7fd00d8b1700 1 --2- 172.21.15.121:0/2093491760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd008106c10 0x7fd008100a30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcffc006010 tx=0x7fcffc00bd70 comp rx=0 tx=0).stop 2024-04-05T21:40:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.314+0000 7fd00d8b1700 1 -- 172.21.15.121:0/2093491760 shutdown_connections 2024-04-05T21:40:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.314+0000 7fd00d8b1700 1 --2- 172.21.15.121:0/2093491760 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcff006cce0 0x7fcff006f1a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.314+0000 7fd00d8b1700 1 --2- 172.21.15.121:0/2093491760 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0080ff310 0x7fd0081004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.314+0000 7fd00d8b1700 1 --2- 172.21.15.121:0/2093491760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd008106c10 0x7fd008100a30 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.314+0000 7fd00d8b1700 1 -- 172.21.15.121:0/2093491760 >> 172.21.15.121:0/2093491760 conn(0x7fd008074bf0 msgr2=0x7fd0080fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:09.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.314+0000 7fd00d8b1700 1 -- 172.21.15.121:0/2093491760 shutdown_connections 2024-04-05T21:40:09.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.314+0000 7fd00d8b1700 1 -- 172.21.15.121:0/2093491760 wait complete. 2024-04-05T21:40:09.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.418+0000 7f1f5733d700 1 -- 172.21.15.121:0/2943691655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f50105a40 msgr2=0x7f1f50105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:09.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.418+0000 7f1f5733d700 1 --2- 172.21.15.121:0/2943691655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f50105a40 0x7f1f50105e20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c009910 tx=0x7f1f4c009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:09.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.418+0000 7f1f5733d700 1 -- 172.21.15.121:0/2943691655 shutdown_connections 2024-04-05T21:40:09.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.418+0000 7f1f5733d700 1 --2- 172.21.15.121:0/2943691655 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f500ffbd0 0x7f1f50100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.418+0000 7f1f5733d700 1 --2- 172.21.15.121:0/2943691655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f50105a40 0x7f1f50105e20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.418+0000 7f1f5733d700 1 -- 172.21.15.121:0/2943691655 >> 172.21.15.121:0/2943691655 conn(0x7f1f500755a0 msgr2=0x7f1f500759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:09.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.419+0000 7f1f5733d700 1 -- 172.21.15.121:0/2943691655 shutdown_connections 2024-04-05T21:40:09.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.419+0000 7f1f5733d700 1 -- 172.21.15.121:0/2943691655 wait complete. 2024-04-05T21:40:09.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.419+0000 7f1f5733d700 1 Processor -- start 2024-04-05T21:40:09.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.419+0000 7f1f5733d700 1 -- start start 2024-04-05T21:40:09.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.419+0000 7f1f5733d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f500ffbd0 0x7f1f501974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:09.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.419+0000 7f1f5733d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f50105a40 0x7f1f501979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:09.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.419+0000 7f1f5733d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f50198000 con 0x7f1f500ffbd0 2024-04-05T21:40:09.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.419+0000 7f1f5733d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f50198170 con 0x7f1f50105a40 2024-04-05T21:40:09.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.419+0000 7f1f550d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f500ffbd0 0x7f1f501974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:09.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.420+0000 7f1f550d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f500ffbd0 0x7f1f501974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37568/0 (socket says 172.21.15.121:37568) 2024-04-05T21:40:09.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.420+0000 7f1f550d9700 1 -- 172.21.15.121:0/792131579 learned_addr learned my addr 172.21.15.121:0/792131579 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:09.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.420+0000 7f1f548d8700 1 --2- 172.21.15.121:0/792131579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f50105a40 0x7f1f501979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:09.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.420+0000 7f1f550d9700 1 -- 172.21.15.121:0/792131579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f50105a40 msgr2=0x7f1f501979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:09.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.420+0000 7f1f550d9700 1 --2- 172.21.15.121:0/792131579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f50105a40 0x7f1f501979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.420+0000 7f1f550d9700 1 -- 172.21.15.121:0/792131579 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c0095f0 con 0x7f1f500ffbd0 2024-04-05T21:40:09.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.420+0000 7f1f550d9700 1 --2- 172.21.15.121:0/792131579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f500ffbd0 0x7f1f501974a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c006010 tx=0x7f1f4c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:09.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.420+0000 7f1f467fc700 1 -- 172.21.15.121:0/792131579 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f4c01d070 con 0x7f1f500ffbd0 2024-04-05T21:40:09.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.420+0000 7f1f5733d700 1 -- 172.21.15.121:0/792131579 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f501a0510 con 0x7f1f500ffbd0 2024-04-05T21:40:09.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.420+0000 7f1f467fc700 1 -- 172.21.15.121:0/792131579 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f4c005710 con 0x7f1f500ffbd0 2024-04-05T21:40:09.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.420+0000 7f1f467fc700 1 -- 172.21.15.121:0/792131579 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f4c00fe20 con 0x7f1f500ffbd0 2024-04-05T21:40:09.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.420+0000 7f1f5733d700 1 -- 172.21.15.121:0/792131579 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f501a0980 con 0x7f1f500ffbd0 2024-04-05T21:40:09.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.421+0000 7f1f5733d700 1 -- 172.21.15.121:0/792131579 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f50108040 con 0x7f1f500ffbd0 2024-04-05T21:40:09.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.421+0000 7f1f467fc700 1 -- 172.21.15.121:0/792131579 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1f4c003a60 con 0x7f1f500ffbd0 2024-04-05T21:40:09.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.422+0000 7f1f467fc700 1 --2- 172.21.15.121:0/792131579 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1f3c06cce0 0x7f1f3c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:09.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.422+0000 7f1f467fc700 1 -- 172.21.15.121:0/792131579 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1f4c08d830 con 0x7f1f500ffbd0 2024-04-05T21:40:09.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.422+0000 7f1f548d8700 1 --2- 172.21.15.121:0/792131579 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1f3c06cce0 0x7f1f3c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:09.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.422+0000 7f1f548d8700 1 --2- 172.21.15.121:0/792131579 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1f3c06cce0 0x7f1f3c06f1a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1f50068ce0 tx=0x7f1f4000b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:09.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.426+0000 7f1f467fc700 1 -- 172.21.15.121:0/792131579 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1f4c057800 con 0x7f1f500ffbd0 2024-04-05T21:40:09.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:09 smithi121 ceph-mon[145859]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:09.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:09 smithi121 ceph-mon[145859]: pgmap v417: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:09.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:09 smithi121 ceph-mon[145859]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:09.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:09 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2093491760' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:40:09.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.663+0000 7f1f5733d700 1 -- 172.21.15.121:0/792131579 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1f5002cd30 con 0x7f1f500ffbd0 2024-04-05T21:40:09.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.663+0000 7f1f467fc700 1 -- 172.21.15.121:0/792131579 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f1f4c003d10 con 0x7f1f500ffbd0 2024-04-05T21:40:09.665 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:40:09.665 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:40:09.665 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:40:09.665 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:40:09.665 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:40:09.665 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:40:09.666 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:40:09.667 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:40:09.668 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:40:09.668 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:40:09.668 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:40:09.668 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:40:09.668 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:40:09.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.665+0000 7f1f5733d700 1 -- 172.21.15.121:0/792131579 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1f3c06cce0 msgr2=0x7f1f3c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:09.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.665+0000 7f1f5733d700 1 --2- 172.21.15.121:0/792131579 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1f3c06cce0 0x7f1f3c06f1a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1f50068ce0 tx=0x7f1f4000b410 comp rx=0 tx=0).stop 2024-04-05T21:40:09.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.665+0000 7f1f5733d700 1 -- 172.21.15.121:0/792131579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f500ffbd0 msgr2=0x7f1f501974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:09.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.665+0000 7f1f5733d700 1 --2- 172.21.15.121:0/792131579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f500ffbd0 0x7f1f501974a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c006010 tx=0x7f1f4c00bd70 comp rx=0 tx=0).stop 2024-04-05T21:40:09.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.665+0000 7f1f5733d700 1 -- 172.21.15.121:0/792131579 shutdown_connections 2024-04-05T21:40:09.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.665+0000 7f1f5733d700 1 --2- 172.21.15.121:0/792131579 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1f3c06cce0 0x7f1f3c06f1a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.665+0000 7f1f5733d700 1 --2- 172.21.15.121:0/792131579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f500ffbd0 0x7f1f501974a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.665+0000 7f1f5733d700 1 --2- 172.21.15.121:0/792131579 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f50105a40 0x7f1f501979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.665+0000 7f1f5733d700 1 -- 172.21.15.121:0/792131579 >> 172.21.15.121:0/792131579 conn(0x7f1f500755a0 msgr2=0x7f1f500fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:09.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.666+0000 7f1f5733d700 1 -- 172.21.15.121:0/792131579 shutdown_connections 2024-04-05T21:40:09.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.666+0000 7f1f5733d700 1 -- 172.21.15.121:0/792131579 wait complete. 2024-04-05T21:40:09.670 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:40:09.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.770+0000 7f8858372700 1 -- 172.21.15.121:0/646620851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8850100de0 msgr2=0x7f88501011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:09.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.770+0000 7f8858372700 1 --2- 172.21.15.121:0/646620851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8850100de0 0x7f88501011c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8840009910 tx=0x7f8840009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:09.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.770+0000 7f8858372700 1 -- 172.21.15.121:0/646620851 shutdown_connections 2024-04-05T21:40:09.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.770+0000 7f8858372700 1 --2- 172.21.15.121:0/646620851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8850101790 0x7f88501057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.770+0000 7f8858372700 1 --2- 172.21.15.121:0/646620851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8850100de0 0x7f88501011c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.770+0000 7f8858372700 1 -- 172.21.15.121:0/646620851 >> 172.21.15.121:0/646620851 conn(0x7f88500fc670 msgr2=0x7f88500fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.770+0000 7f8858372700 1 -- 172.21.15.121:0/646620851 shutdown_connections 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.770+0000 7f8858372700 1 -- 172.21.15.121:0/646620851 wait complete. 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.770+0000 7f8858372700 1 Processor -- start 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f8858372700 1 -- start start 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f8858372700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8850100de0 0x7f8850196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f8858372700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8850101790 0x7f8850196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f8858372700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8850197290 con 0x7f8850100de0 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f8858372700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885019b020 con 0x7f8850101790 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f885610e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8850100de0 0x7f8850196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f885610e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8850100de0 0x7f8850196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37582/0 (socket says 172.21.15.121:37582) 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f885610e700 1 -- 172.21.15.121:0/1411828070 learned_addr learned my addr 172.21.15.121:0/1411828070 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f885590d700 1 --2- 172.21.15.121:0/1411828070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8850101790 0x7f8850196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:09.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f885610e700 1 -- 172.21.15.121:0/1411828070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8850101790 msgr2=0x7f8850196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:09.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f885610e700 1 --2- 172.21.15.121:0/1411828070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8850101790 0x7f8850196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f885610e700 1 -- 172.21.15.121:0/1411828070 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88400095f0 con 0x7f8850100de0 2024-04-05T21:40:09.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f885610e700 1 --2- 172.21.15.121:0/1411828070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8850100de0 0x7f8850196670 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8840006010 tx=0x7f884000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:09.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f88477fe700 1 -- 172.21.15.121:0/1411828070 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f884001d070 con 0x7f8850100de0 2024-04-05T21:40:09.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f8858372700 1 -- 172.21.15.121:0/1411828070 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885019b2a0 con 0x7f8850100de0 2024-04-05T21:40:09.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f88477fe700 1 -- 172.21.15.121:0/1411828070 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f884000f460 con 0x7f8850100de0 2024-04-05T21:40:09.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.771+0000 7f88477fe700 1 -- 172.21.15.121:0/1411828070 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8840021640 con 0x7f8850100de0 2024-04-05T21:40:09.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.772+0000 7f8858372700 1 -- 172.21.15.121:0/1411828070 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f885019b790 con 0x7f8850100de0 2024-04-05T21:40:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.772+0000 7f8858372700 1 -- 172.21.15.121:0/1411828070 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8850109180 con 0x7f8850100de0 2024-04-05T21:40:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.773+0000 7f88477fe700 1 -- 172.21.15.121:0/1411828070 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f884000fa20 con 0x7f8850100de0 2024-04-05T21:40:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.773+0000 7f88477fe700 1 --2- 172.21.15.121:0/1411828070 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f883c06cce0 0x7f883c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.773+0000 7f88477fe700 1 -- 172.21.15.121:0/1411828070 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f884008d830 con 0x7f8850100de0 2024-04-05T21:40:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.773+0000 7f885590d700 1 --2- 172.21.15.121:0/1411828070 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f883c06cce0 0x7f883c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.774+0000 7f885590d700 1 --2- 172.21.15.121:0/1411828070 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f883c06cce0 0x7f883c06f1a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f884c005950 tx=0x7f884c0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:09 smithi188 ceph-mon[107793]: from='client.34622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:09 smithi188 ceph-mon[107793]: pgmap v417: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:09 smithi188 ceph-mon[107793]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:09 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2093491760' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:40:09.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.778+0000 7f88477fe700 1 -- 172.21.15.121:0/1411828070 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f88400578a0 con 0x7f8850100de0 2024-04-05T21:40:09.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.988+0000 7f8858372700 1 -- 172.21.15.121:0/1411828070 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f885019b9b0 con 0x7f883c06cce0 2024-04-05T21:40:09.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.990+0000 7f88477fe700 1 -- 172.21.15.121:0/1411828070 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f885019b9b0 con 0x7f883c06cce0 2024-04-05T21:40:09.991 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:40:09.991 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:40:09.991 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:40:09.992 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:40:09.992 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:40:09.992 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:40:09.992 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:40:09.992 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:40:09.992 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:40:09.992 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:40:09.992 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:40:09.992 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:40:09.992 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:40:09.992 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:40:09.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.992+0000 7f8858372700 1 -- 172.21.15.121:0/1411828070 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f883c06cce0 msgr2=0x7f883c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:09.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.992+0000 7f8858372700 1 --2- 172.21.15.121:0/1411828070 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f883c06cce0 0x7f883c06f1a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f884c005950 tx=0x7f884c0058e0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.992+0000 7f8858372700 1 -- 172.21.15.121:0/1411828070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8850100de0 msgr2=0x7f8850196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:09.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.992+0000 7f8858372700 1 --2- 172.21.15.121:0/1411828070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8850100de0 0x7f8850196670 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8840006010 tx=0x7f884000be20 comp rx=0 tx=0).stop 2024-04-05T21:40:09.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.992+0000 7f8858372700 1 -- 172.21.15.121:0/1411828070 shutdown_connections 2024-04-05T21:40:09.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.992+0000 7f8858372700 1 --2- 172.21.15.121:0/1411828070 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f883c06cce0 0x7f883c06f1a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.992+0000 7f8858372700 1 --2- 172.21.15.121:0/1411828070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8850100de0 0x7f8850196670 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.992+0000 7f8858372700 1 --2- 172.21.15.121:0/1411828070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8850101790 0x7f8850196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:09.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.992+0000 7f8858372700 1 -- 172.21.15.121:0/1411828070 >> 172.21.15.121:0/1411828070 conn(0x7f88500fc670 msgr2=0x7f88500fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:09.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.992+0000 7f8858372700 1 -- 172.21.15.121:0/1411828070 shutdown_connections 2024-04-05T21:40:09.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:09.992+0000 7f8858372700 1 -- 172.21.15.121:0/1411828070 wait complete. 2024-04-05T21:40:10.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.096+0000 7f37e7055700 1 -- 172.21.15.121:0/3195187102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37e0105a40 msgr2=0x7f37e0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:10.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.096+0000 7f37e7055700 1 --2- 172.21.15.121:0/3195187102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37e0105a40 0x7f37e0105e20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f37d4009910 tx=0x7f37d4009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:10.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.096+0000 7f37e7055700 1 -- 172.21.15.121:0/3195187102 shutdown_connections 2024-04-05T21:40:10.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.096+0000 7f37e7055700 1 --2- 172.21.15.121:0/3195187102 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37e00ffbd0 0x7f37e0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:10.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.096+0000 7f37e7055700 1 --2- 172.21.15.121:0/3195187102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37e0105a40 0x7f37e0105e20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:10.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.096+0000 7f37e7055700 1 -- 172.21.15.121:0/3195187102 >> 172.21.15.121:0/3195187102 conn(0x7f37e00755a0 msgr2=0x7f37e00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:10.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.096+0000 7f37e7055700 1 -- 172.21.15.121:0/3195187102 shutdown_connections 2024-04-05T21:40:10.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.096+0000 7f37e7055700 1 -- 172.21.15.121:0/3195187102 wait complete. 2024-04-05T21:40:10.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.096+0000 7f37e7055700 1 Processor -- start 2024-04-05T21:40:10.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.096+0000 7f37e7055700 1 -- start start 2024-04-05T21:40:10.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37e7055700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37e00ffbd0 0x7f37e0195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37e7055700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37e0105a40 0x7f37e0195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37e7055700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e0196170 con 0x7f37e00ffbd0 2024-04-05T21:40:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37e7055700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e0199f00 con 0x7f37e0105a40 2024-04-05T21:40:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37e4df1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37e00ffbd0 0x7f37e0195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37e4df1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37e00ffbd0 0x7f37e0195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37598/0 (socket says 172.21.15.121:37598) 2024-04-05T21:40:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37e4df1700 1 -- 172.21.15.121:0/743125269 learned_addr learned my addr 172.21.15.121:0/743125269 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37dffff700 1 --2- 172.21.15.121:0/743125269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37e0105a40 0x7f37e0195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37e4df1700 1 -- 172.21.15.121:0/743125269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37e0105a40 msgr2=0x7f37e0195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37e4df1700 1 --2- 172.21.15.121:0/743125269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37e0105a40 0x7f37e0195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37e4df1700 1 -- 172.21.15.121:0/743125269 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d40095f0 con 0x7f37e00ffbd0 2024-04-05T21:40:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37dffff700 1 --2- 172.21.15.121:0/743125269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37e0105a40 0x7f37e0195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:40:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37e4df1700 1 --2- 172.21.15.121:0/743125269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37e00ffbd0 0x7f37e0195550 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f37d4006010 tx=0x7f37d400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37ddffb700 1 -- 172.21.15.121:0/743125269 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37d401d070 con 0x7f37e00ffbd0 2024-04-05T21:40:10.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37ddffb700 1 -- 172.21.15.121:0/743125269 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37d400f460 con 0x7f37e00ffbd0 2024-04-05T21:40:10.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37ddffb700 1 -- 172.21.15.121:0/743125269 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37d4021620 con 0x7f37e00ffbd0 2024-04-05T21:40:10.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37e7055700 1 -- 172.21.15.121:0/743125269 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e019a180 con 0x7f37e00ffbd0 2024-04-05T21:40:10.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.097+0000 7f37e7055700 1 -- 172.21.15.121:0/743125269 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e019a670 con 0x7f37e00ffbd0 2024-04-05T21:40:10.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.098+0000 7f37e7055700 1 -- 172.21.15.121:0/743125269 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e0108040 con 0x7f37e00ffbd0 2024-04-05T21:40:10.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.099+0000 7f37ddffb700 1 -- 172.21.15.121:0/743125269 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f37d400fa40 con 0x7f37e00ffbd0 2024-04-05T21:40:10.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.099+0000 7f37ddffb700 1 --2- 172.21.15.121:0/743125269 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37d006cce0 0x7f37d006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:10.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.099+0000 7f37ddffb700 1 -- 172.21.15.121:0/743125269 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f37d408d760 con 0x7f37e00ffbd0 2024-04-05T21:40:10.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.099+0000 7f37dffff700 1 --2- 172.21.15.121:0/743125269 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37d006cce0 0x7f37d006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:10.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.100+0000 7f37dffff700 1 --2- 172.21.15.121:0/743125269 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37d006cce0 0x7f37d006f1a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f37cc005950 tx=0x7f37cc0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:10.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.104+0000 7f37ddffb700 1 -- 172.21.15.121:0/743125269 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f37d40577b0 con 0x7f37e00ffbd0 2024-04-05T21:40:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.381+0000 7f37e7055700 1 -- 172.21.15.121:0/743125269 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f37e002cd30 con 0x7f37e00ffbd0 2024-04-05T21:40:10.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.381+0000 7f37ddffb700 1 -- 172.21.15.121:0/743125269 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f37d405add0 con 0x7f37e00ffbd0 2024-04-05T21:40:10.383 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:40:10.383 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:40:10.383 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:40:10.384 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:40:10.384 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:40:10.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.383+0000 7f37e7055700 1 -- 172.21.15.121:0/743125269 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37d006cce0 msgr2=0x7f37d006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:10.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.383+0000 7f37e7055700 1 --2- 172.21.15.121:0/743125269 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37d006cce0 0x7f37d006f1a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f37cc005950 tx=0x7f37cc0058e0 comp rx=0 tx=0).stop 2024-04-05T21:40:10.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.383+0000 7f37e7055700 1 -- 172.21.15.121:0/743125269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37e00ffbd0 msgr2=0x7f37e0195550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:10.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.383+0000 7f37e7055700 1 --2- 172.21.15.121:0/743125269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37e00ffbd0 0x7f37e0195550 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f37d4006010 tx=0x7f37d400be20 comp rx=0 tx=0).stop 2024-04-05T21:40:10.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.383+0000 7f37e7055700 1 -- 172.21.15.121:0/743125269 shutdown_connections 2024-04-05T21:40:10.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.383+0000 7f37e7055700 1 --2- 172.21.15.121:0/743125269 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37d006cce0 0x7f37d006f1a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:10.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.383+0000 7f37e7055700 1 --2- 172.21.15.121:0/743125269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37e00ffbd0 0x7f37e0195550 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:10.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.383+0000 7f37e7055700 1 --2- 172.21.15.121:0/743125269 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37e0105a40 0x7f37e0195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:10.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.383+0000 7f37e7055700 1 -- 172.21.15.121:0/743125269 >> 172.21.15.121:0/743125269 conn(0x7f37e00755a0 msgr2=0x7f37e00fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:10.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.384+0000 7f37e7055700 1 -- 172.21.15.121:0/743125269 shutdown_connections 2024-04-05T21:40:10.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:10.384+0000 7f37e7055700 1 -- 172.21.15.121:0/743125269 wait complete. 2024-04-05T21:40:10.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:10 smithi121 ceph-mon[145859]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:10.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:10 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/792131579' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:40:10.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:10 smithi188 ceph-mon[107793]: from='client.34630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:10.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:10 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/792131579' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:40:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:11 smithi121 ceph-mon[145859]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:11 smithi121 ceph-mon[145859]: pgmap v418: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:11 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/743125269' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:40:11.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:11 smithi188 ceph-mon[107793]: from='client.34642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:11.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:11 smithi188 ceph-mon[107793]: pgmap v418: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:11.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:11 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/743125269' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:40:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:13 smithi121 ceph-mon[145859]: pgmap v419: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:13.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:13 smithi188 ceph-mon[107793]: pgmap v419: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:15.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:15 smithi121 ceph-mon[145859]: pgmap v420: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:15.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:40:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:15 smithi188 ceph-mon[107793]: pgmap v420: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:40:17.654 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:17 smithi188 ceph-mon[107793]: pgmap v421: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:17 smithi121 ceph-mon[145859]: pgmap v421: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:19.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:19 smithi121 ceph-mon[145859]: pgmap v422: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:19 smithi188 ceph-mon[107793]: pgmap v422: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:21 smithi121 ceph-mon[145859]: pgmap v423: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:21 smithi188 ceph-mon[107793]: pgmap v423: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:23.632 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:23 smithi188 ceph-mon[107793]: pgmap v424: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:23.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:23 smithi121 ceph-mon[145859]: pgmap v424: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:25.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:25 smithi121 ceph-mon[145859]: pgmap v425: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:25.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:25 smithi188 ceph-mon[107793]: pgmap v425: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:27 smithi121 ceph-mon[145859]: pgmap v426: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:27.747 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:27 smithi188 ceph-mon[107793]: pgmap v426: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:29 smithi121 ceph-mon[145859]: pgmap v427: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:29 smithi188 ceph-mon[107793]: pgmap v427: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:40:30.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:40:31.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:31 smithi121 ceph-mon[145859]: pgmap v428: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:31.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:31 smithi188 ceph-mon[107793]: pgmap v428: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:33.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:33 smithi121 ceph-mon[145859]: pgmap v429: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:33 smithi188 ceph-mon[107793]: pgmap v429: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:35.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:35 smithi121 ceph-mon[145859]: pgmap v430: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:35.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:35 smithi188 ceph-mon[107793]: pgmap v430: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:37 smithi121 ceph-mon[145859]: pgmap v431: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:37.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:37 smithi188 ceph-mon[107793]: pgmap v431: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:39.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:39 smithi121 ceph-mon[145859]: pgmap v432: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:39.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:39 smithi188 ceph-mon[107793]: pgmap v432: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:40.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.506+0000 7f2048f55700 1 -- 172.21.15.121:0/2517038111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2044101750 msgr2=0x7f20441057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:40.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.506+0000 7f2048f55700 1 --2- 172.21.15.121:0/2517038111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2044101750 0x7f20441057a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2034009910 tx=0x7f2034009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:40.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.506+0000 7f2048f55700 1 -- 172.21.15.121:0/2517038111 shutdown_connections 2024-04-05T21:40:40.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.506+0000 7f2048f55700 1 --2- 172.21.15.121:0/2517038111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2044101750 0x7f20441057a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:40.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.506+0000 7f2048f55700 1 --2- 172.21.15.121:0/2517038111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2044100da0 0x7f2044101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:40.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.506+0000 7f2048f55700 1 -- 172.21.15.121:0/2517038111 >> 172.21.15.121:0/2517038111 conn(0x7f20440fc630 msgr2=0x7f20440fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:40.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.506+0000 7f2048f55700 1 -- 172.21.15.121:0/2517038111 shutdown_connections 2024-04-05T21:40:40.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.506+0000 7f2048f55700 1 -- 172.21.15.121:0/2517038111 wait complete. 2024-04-05T21:40:40.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.507+0000 7f2048f55700 1 Processor -- start 2024-04-05T21:40:40.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.507+0000 7f2048f55700 1 -- start start 2024-04-05T21:40:40.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.507+0000 7f2048f55700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2044100da0 0x7f2044196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:40.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.507+0000 7f2048f55700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2044101750 0x7f2044196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:40.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.507+0000 7f2048f55700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2044197270 con 0x7f2044100da0 2024-04-05T21:40:40.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.507+0000 7f2048f55700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204419b000 con 0x7f2044101750 2024-04-05T21:40:40.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.507+0000 7f204259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2044100da0 0x7f2044196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:40.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.507+0000 7f204259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2044100da0 0x7f2044196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60740/0 (socket says 172.21.15.121:60740) 2024-04-05T21:40:40.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.507+0000 7f204259c700 1 -- 172.21.15.121:0/3006205716 learned_addr learned my addr 172.21.15.121:0/3006205716 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:40.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.508+0000 7f2041d9b700 1 --2- 172.21.15.121:0/3006205716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2044101750 0x7f2044196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:40.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.508+0000 7f204259c700 1 -- 172.21.15.121:0/3006205716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2044101750 msgr2=0x7f2044196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:40.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.508+0000 7f204259c700 1 --2- 172.21.15.121:0/3006205716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2044101750 0x7f2044196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:40.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.508+0000 7f204259c700 1 -- 172.21.15.121:0/3006205716 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20340095f0 con 0x7f2044100da0 2024-04-05T21:40:40.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.508+0000 7f204259c700 1 --2- 172.21.15.121:0/3006205716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2044100da0 0x7f2044196650 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f203800e920 tx=0x7f203800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:40.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.508+0000 7f20337fe700 1 -- 172.21.15.121:0/3006205716 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f203800c9d0 con 0x7f2044100da0 2024-04-05T21:40:40.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.508+0000 7f2048f55700 1 -- 172.21.15.121:0/3006205716 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204419b2e0 con 0x7f2044100da0 2024-04-05T21:40:40.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.508+0000 7f20337fe700 1 -- 172.21.15.121:0/3006205716 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2038010430 con 0x7f2044100da0 2024-04-05T21:40:40.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.508+0000 7f20337fe700 1 -- 172.21.15.121:0/3006205716 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2038018650 con 0x7f2044100da0 2024-04-05T21:40:40.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.508+0000 7f2048f55700 1 -- 172.21.15.121:0/3006205716 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204419b800 con 0x7f2044100da0 2024-04-05T21:40:40.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.509+0000 7f2048f55700 1 -- 172.21.15.121:0/3006205716 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20441091a0 con 0x7f2044100da0 2024-04-05T21:40:40.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.510+0000 7f20337fe700 1 -- 172.21.15.121:0/3006205716 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f20380105a0 con 0x7f2044100da0 2024-04-05T21:40:40.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.510+0000 7f20337fe700 1 --2- 172.21.15.121:0/3006205716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f202c06cce0 0x7f202c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:40.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.510+0000 7f20337fe700 1 -- 172.21.15.121:0/3006205716 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2038014070 con 0x7f2044100da0 2024-04-05T21:40:40.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.510+0000 7f2041d9b700 1 --2- 172.21.15.121:0/3006205716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f202c06cce0 0x7f202c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:40.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.511+0000 7f2041d9b700 1 --2- 172.21.15.121:0/3006205716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f202c06cce0 0x7f202c06f1a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2034009fd0 tx=0x7f2034005940 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:40.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.515+0000 7f20337fe700 1 -- 172.21.15.121:0/3006205716 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2038052ab0 con 0x7f2044100da0 2024-04-05T21:40:40.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.723+0000 7f2048f55700 1 -- 172.21.15.121:0/3006205716 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f204419bac0 con 0x7f202c06cce0 2024-04-05T21:40:40.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.725+0000 7f20337fe700 1 -- 172.21.15.121:0/3006205716 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f204419bac0 con 0x7f202c06cce0 2024-04-05T21:40:40.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.727+0000 7f2048f55700 1 -- 172.21.15.121:0/3006205716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f202c06cce0 msgr2=0x7f202c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:40.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.727+0000 7f2048f55700 1 --2- 172.21.15.121:0/3006205716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f202c06cce0 0x7f202c06f1a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2034009fd0 tx=0x7f2034005940 comp rx=0 tx=0).stop 2024-04-05T21:40:40.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.727+0000 7f2048f55700 1 -- 172.21.15.121:0/3006205716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2044100da0 msgr2=0x7f2044196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:40.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.727+0000 7f2048f55700 1 --2- 172.21.15.121:0/3006205716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2044100da0 0x7f2044196650 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f203800e920 tx=0x7f203800ec30 comp rx=0 tx=0).stop 2024-04-05T21:40:40.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.727+0000 7f2048f55700 1 -- 172.21.15.121:0/3006205716 shutdown_connections 2024-04-05T21:40:40.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.727+0000 7f2048f55700 1 --2- 172.21.15.121:0/3006205716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f202c06cce0 0x7f202c06f1a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:40.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.727+0000 7f2048f55700 1 --2- 172.21.15.121:0/3006205716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2044100da0 0x7f2044196650 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:40.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.727+0000 7f2048f55700 1 --2- 172.21.15.121:0/3006205716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2044101750 0x7f2044196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:40.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.727+0000 7f2048f55700 1 -- 172.21.15.121:0/3006205716 >> 172.21.15.121:0/3006205716 conn(0x7f20440fc630 msgr2=0x7f20440fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:40.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.727+0000 7f2048f55700 1 -- 172.21.15.121:0/3006205716 shutdown_connections 2024-04-05T21:40:40.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.727+0000 7f2048f55700 1 -- 172.21.15.121:0/3006205716 wait complete. 2024-04-05T21:40:40.738 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:40:40.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.831+0000 7fa5f286c700 1 -- 172.21.15.121:0/3886181056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec102620 msgr2=0x7fa5ec10ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:40.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.831+0000 7fa5f286c700 1 --2- 172.21.15.121:0/3886181056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec102620 0x7fa5ec10ab10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc009910 tx=0x7fa5dc009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:40.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.831+0000 7fa5f286c700 1 -- 172.21.15.121:0/3886181056 shutdown_connections 2024-04-05T21:40:40.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.831+0000 7fa5f286c700 1 --2- 172.21.15.121:0/3886181056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec102620 0x7fa5ec10ab10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:40.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.831+0000 7fa5f286c700 1 --2- 172.21.15.121:0/3886181056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5ec101d00 0x7fa5ec1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:40.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.831+0000 7fa5f286c700 1 -- 172.21.15.121:0/3886181056 >> 172.21.15.121:0/3886181056 conn(0x7fa5ec0fb570 msgr2=0x7fa5ec0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:40.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.831+0000 7fa5f286c700 1 -- 172.21.15.121:0/3886181056 shutdown_connections 2024-04-05T21:40:40.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.831+0000 7fa5f286c700 1 -- 172.21.15.121:0/3886181056 wait complete. 2024-04-05T21:40:40.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.831+0000 7fa5f286c700 1 Processor -- start 2024-04-05T21:40:40.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.831+0000 7fa5f286c700 1 -- start start 2024-04-05T21:40:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5f286c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5ec101d00 0x7fa5ec1966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5f286c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec102620 0x7fa5ec196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5f286c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec1972a0 con 0x7fa5ec102620 2024-04-05T21:40:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5f286c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec199fd0 con 0x7fa5ec101d00 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5eb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec102620 0x7fa5ec196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5eb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec102620 0x7fa5ec196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60754/0 (socket says 172.21.15.121:60754) 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5eb7fe700 1 -- 172.21.15.121:0/1894664460 learned_addr learned my addr 172.21.15.121:0/1894664460 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5ebfff700 1 --2- 172.21.15.121:0/1894664460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5ec101d00 0x7fa5ec1966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5eb7fe700 1 -- 172.21.15.121:0/1894664460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5ec101d00 msgr2=0x7fa5ec1966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5eb7fe700 1 --2- 172.21.15.121:0/1894664460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5ec101d00 0x7fa5ec1966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5eb7fe700 1 -- 172.21.15.121:0/1894664460 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5dc0095f0 con 0x7fa5ec102620 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5eb7fe700 1 --2- 172.21.15.121:0/1894664460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec102620 0x7fa5ec196c10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0092e0 tx=0x7fa5dc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5e97fa700 1 -- 172.21.15.121:0/1894664460 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5dc01d070 con 0x7fa5ec102620 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5e97fa700 1 -- 172.21.15.121:0/1894664460 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5dc003d30 con 0x7fa5ec102620 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.832+0000 7fa5e97fa700 1 -- 172.21.15.121:0/1894664460 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5dc00fdb0 con 0x7fa5ec102620 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.833+0000 7fa5f286c700 1 -- 172.21.15.121:0/1894664460 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ec19a250 con 0x7fa5ec102620 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.833+0000 7fa5f286c700 1 -- 172.21.15.121:0/1894664460 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ec19a740 con 0x7fa5ec102620 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.834+0000 7fa5f286c700 1 -- 172.21.15.121:0/1894664460 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ec108210 con 0x7fa5ec102620 2024-04-05T21:40:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.834+0000 7fa5e97fa700 1 -- 172.21.15.121:0/1894664460 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa5dc003890 con 0x7fa5ec102620 2024-04-05T21:40:40.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.834+0000 7fa5e97fa700 1 --2- 172.21.15.121:0/1894664460 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5d406cce0 0x7fa5d406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:40.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.834+0000 7fa5e97fa700 1 -- 172.21.15.121:0/1894664460 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa5dc08d720 con 0x7fa5ec102620 2024-04-05T21:40:40.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.834+0000 7fa5ebfff700 1 --2- 172.21.15.121:0/1894664460 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5d406cce0 0x7fa5d406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:40.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.835+0000 7fa5ebfff700 1 --2- 172.21.15.121:0/1894664460 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5d406cce0 0x7fa5d406f1a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0005950 tx=0x7fa5e000b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:40.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:40.839+0000 7fa5e97fa700 1 -- 172.21.15.121:0/1894664460 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa5dc057790 con 0x7fa5ec102620 2024-04-05T21:40:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.050+0000 7fa5f286c700 1 -- 172.21.15.121:0/1894664460 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5ec19a960 con 0x7fa5d406cce0 2024-04-05T21:40:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.051+0000 7fa5e97fa700 1 -- 172.21.15.121:0/1894664460 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa5ec19a960 con 0x7fa5d406cce0 2024-04-05T21:40:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.054+0000 7fa5f286c700 1 -- 172.21.15.121:0/1894664460 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5d406cce0 msgr2=0x7fa5d406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.054+0000 7fa5f286c700 1 --2- 172.21.15.121:0/1894664460 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5d406cce0 0x7fa5d406f1a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0005950 tx=0x7fa5e000b410 comp rx=0 tx=0).stop 2024-04-05T21:40:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.054+0000 7fa5f286c700 1 -- 172.21.15.121:0/1894664460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec102620 msgr2=0x7fa5ec196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.054+0000 7fa5f286c700 1 --2- 172.21.15.121:0/1894664460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec102620 0x7fa5ec196c10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0092e0 tx=0x7fa5dc00bd70 comp rx=0 tx=0).stop 2024-04-05T21:40:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.054+0000 7fa5f286c700 1 -- 172.21.15.121:0/1894664460 shutdown_connections 2024-04-05T21:40:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.054+0000 7fa5f286c700 1 --2- 172.21.15.121:0/1894664460 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5d406cce0 0x7fa5d406f1a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.054+0000 7fa5f286c700 1 --2- 172.21.15.121:0/1894664460 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5ec101d00 0x7fa5ec1966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.054+0000 7fa5f286c700 1 --2- 172.21.15.121:0/1894664460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5ec102620 0x7fa5ec196c10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.054+0000 7fa5f286c700 1 -- 172.21.15.121:0/1894664460 >> 172.21.15.121:0/1894664460 conn(0x7fa5ec0fb570 msgr2=0x7fa5ec105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.054+0000 7fa5f286c700 1 -- 172.21.15.121:0/1894664460 shutdown_connections 2024-04-05T21:40:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.054+0000 7fa5f286c700 1 -- 172.21.15.121:0/1894664460 wait complete. 2024-04-05T21:40:41.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.156+0000 7f7392cd9700 1 -- 172.21.15.121:0/312562683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f738c0fef70 msgr2=0x7f738c0ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:41.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.156+0000 7f7392cd9700 1 --2- 172.21.15.121:0/312562683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f738c0fef70 0x7f738c0ff3f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7378009910 tx=0x7f7378009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:41.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.156+0000 7f7392cd9700 1 -- 172.21.15.121:0/312562683 shutdown_connections 2024-04-05T21:40:41.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.156+0000 7f7392cd9700 1 --2- 172.21.15.121:0/312562683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f738c0fef70 0x7f738c0ff3f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.156+0000 7f7392cd9700 1 --2- 172.21.15.121:0/312562683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f738c105b50 0x7f738c105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.156+0000 7f7392cd9700 1 -- 172.21.15.121:0/312562683 >> 172.21.15.121:0/312562683 conn(0x7f738c075e30 msgr2=0x7f738c0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:41.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.156+0000 7f7392cd9700 1 -- 172.21.15.121:0/312562683 shutdown_connections 2024-04-05T21:40:41.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.156+0000 7f7392cd9700 1 -- 172.21.15.121:0/312562683 wait complete. 2024-04-05T21:40:41.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.157+0000 7f7392cd9700 1 Processor -- start 2024-04-05T21:40:41.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.157+0000 7f7392cd9700 1 -- start start 2024-04-05T21:40:41.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.157+0000 7f7392cd9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f738c0fef70 0x7f738c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:41.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.157+0000 7f7392cd9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f738c105b50 0x7f738c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:41.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.157+0000 7f7392cd9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c19f270 con 0x7f738c105b50 2024-04-05T21:40:41.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.157+0000 7f7392cd9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c198720 con 0x7f738c0fef70 2024-04-05T21:40:41.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.157+0000 7f738bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f738c105b50 0x7f738c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:41.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.157+0000 7f738bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f738c105b50 0x7f738c19ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60772/0 (socket says 172.21.15.121:60772) 2024-04-05T21:40:41.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.157+0000 7f738bfff700 1 -- 172.21.15.121:0/2611092139 learned_addr learned my addr 172.21.15.121:0/2611092139 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:41.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.157+0000 7f7390a75700 1 --2- 172.21.15.121:0/2611092139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f738c0fef70 0x7f738c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:41.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.158+0000 7f738bfff700 1 -- 172.21.15.121:0/2611092139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f738c0fef70 msgr2=0x7f738c19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:41.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.158+0000 7f738bfff700 1 --2- 172.21.15.121:0/2611092139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f738c0fef70 0x7f738c19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.158+0000 7f738bfff700 1 -- 172.21.15.121:0/2611092139 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73780095f0 con 0x7f738c105b50 2024-04-05T21:40:41.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.158+0000 7f738bfff700 1 --2- 172.21.15.121:0/2611092139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f738c105b50 0x7f738c19ebe0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7378006010 tx=0x7f737800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:41.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.158+0000 7f7389ffb700 1 -- 172.21.15.121:0/2611092139 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f737801d070 con 0x7f738c105b50 2024-04-05T21:40:41.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.158+0000 7f7392cd9700 1 -- 172.21.15.121:0/2611092139 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738c1989a0 con 0x7f738c105b50 2024-04-05T21:40:41.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.158+0000 7f7389ffb700 1 -- 172.21.15.121:0/2611092139 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7378005710 con 0x7f738c105b50 2024-04-05T21:40:41.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.158+0000 7f7389ffb700 1 -- 172.21.15.121:0/2611092139 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f737800fe20 con 0x7f738c105b50 2024-04-05T21:40:41.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.158+0000 7f7392cd9700 1 -- 172.21.15.121:0/2611092139 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738c198e90 con 0x7f738c105b50 2024-04-05T21:40:41.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.159+0000 7f7392cd9700 1 -- 172.21.15.121:0/2611092139 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738c0fd980 con 0x7f738c105b50 2024-04-05T21:40:41.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.160+0000 7f7389ffb700 1 -- 172.21.15.121:0/2611092139 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7378003a60 con 0x7f738c105b50 2024-04-05T21:40:41.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.160+0000 7f7389ffb700 1 --2- 172.21.15.121:0/2611092139 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f737c06cce0 0x7f737c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:41.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.160+0000 7f7389ffb700 1 -- 172.21.15.121:0/2611092139 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f737808d750 con 0x7f738c105b50 2024-04-05T21:40:41.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.160+0000 7f7390a75700 1 --2- 172.21.15.121:0/2611092139 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f737c06cce0 0x7f737c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:41.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.160+0000 7f7390a75700 1 --2- 172.21.15.121:0/2611092139 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f737c06cce0 0x7f737c06f1a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7380005950 tx=0x7f738000b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:41.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.164+0000 7f7389ffb700 1 -- 172.21.15.121:0/2611092139 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f737805b250 con 0x7f738c105b50 2024-04-05T21:40:41.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.365+0000 7f7392cd9700 1 -- 172.21.15.121:0/2611092139 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f738c02cce0 con 0x7f737c06cce0 2024-04-05T21:40:41.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.375+0000 7f7389ffb700 1 -- 172.21.15.121:0/2611092139 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f738c02cce0 con 0x7f737c06cce0 2024-04-05T21:40:41.375 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:40:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (11m) 5m ago 22m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:40:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (23m) 5m ago 23m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:40:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (22m) 5m ago 22m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:40:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (9m) 5m ago 23m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:40:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (9m) 5m ago 22m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:40:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (11m) 5m ago 22m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:40:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (5m) 5m ago 16m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:40:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (5m) 5m ago 16m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:40:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (5m) 5m ago 16m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:40:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (16m) 5m ago 16m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:40:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (13m) 5m ago 24m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:40:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (12m) 5m ago 21m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:40:41.376 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (10m) 5m ago 24m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:40:41.377 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (9m) 5m ago 21m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:40:41.377 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (12m) 5m ago 23m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:40:41.377 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (12m) 5m ago 21m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:40:41.377 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (9m) 5m ago 20m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:40:41.377 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (8m) 5m ago 20m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:40:41.377 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (8m) 5m ago 19m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:40:41.377 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (7m) 5m ago 19m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:40:41.377 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (6m) 5m ago 18m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:40:41.377 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (6m) 5m ago 18m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:40:41.377 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (11m) 5m ago 22m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:40:41.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.377+0000 7f7392cd9700 1 -- 172.21.15.121:0/2611092139 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f737c06cce0 msgr2=0x7f737c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:41.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.377+0000 7f7392cd9700 1 --2- 172.21.15.121:0/2611092139 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f737c06cce0 0x7f737c06f1a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7380005950 tx=0x7f738000b410 comp rx=0 tx=0).stop 2024-04-05T21:40:41.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.377+0000 7f7392cd9700 1 -- 172.21.15.121:0/2611092139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f738c105b50 msgr2=0x7f738c19ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:41.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.377+0000 7f7392cd9700 1 --2- 172.21.15.121:0/2611092139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f738c105b50 0x7f738c19ebe0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7378006010 tx=0x7f737800bd70 comp rx=0 tx=0).stop 2024-04-05T21:40:41.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.377+0000 7f7392cd9700 1 -- 172.21.15.121:0/2611092139 shutdown_connections 2024-04-05T21:40:41.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.377+0000 7f7392cd9700 1 --2- 172.21.15.121:0/2611092139 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f737c06cce0 0x7f737c06f1a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.377+0000 7f7392cd9700 1 --2- 172.21.15.121:0/2611092139 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f738c0fef70 0x7f738c19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.377+0000 7f7392cd9700 1 --2- 172.21.15.121:0/2611092139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f738c105b50 0x7f738c19ebe0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.377+0000 7f7392cd9700 1 -- 172.21.15.121:0/2611092139 >> 172.21.15.121:0/2611092139 conn(0x7f738c075e30 msgr2=0x7f738c104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:41.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.377+0000 7f7392cd9700 1 -- 172.21.15.121:0/2611092139 shutdown_connections 2024-04-05T21:40:41.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.377+0000 7f7392cd9700 1 -- 172.21.15.121:0/2611092139 wait complete. 2024-04-05T21:40:41.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.479+0000 7f149d1bd700 1 -- 172.21.15.121:0/2959072416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1498105a40 msgr2=0x7f1498105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:41.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.479+0000 7f149d1bd700 1 --2- 172.21.15.121:0/2959072416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1498105a40 0x7f1498105e20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1488009910 tx=0x7f1488009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:41.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.479+0000 7f149d1bd700 1 -- 172.21.15.121:0/2959072416 shutdown_connections 2024-04-05T21:40:41.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.479+0000 7f149d1bd700 1 --2- 172.21.15.121:0/2959072416 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14980ffbd0 0x7f1498100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.479+0000 7f149d1bd700 1 --2- 172.21.15.121:0/2959072416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1498105a40 0x7f1498105e20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.479+0000 7f149d1bd700 1 -- 172.21.15.121:0/2959072416 >> 172.21.15.121:0/2959072416 conn(0x7f14980755a0 msgr2=0x7f14980759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:41.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.479+0000 7f149d1bd700 1 -- 172.21.15.121:0/2959072416 shutdown_connections 2024-04-05T21:40:41.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.479+0000 7f149d1bd700 1 -- 172.21.15.121:0/2959072416 wait complete. 2024-04-05T21:40:41.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.480+0000 7f149d1bd700 1 Processor -- start 2024-04-05T21:40:41.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.480+0000 7f149d1bd700 1 -- start start 2024-04-05T21:40:41.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.480+0000 7f149d1bd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14980ffbd0 0x7f1498196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:41.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.480+0000 7f149d1bd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1498105a40 0x7f1498196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:41.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.480+0000 7f149d1bd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1498197260 con 0x7f14980ffbd0 2024-04-05T21:40:41.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.480+0000 7f149d1bd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149819aff0 con 0x7f1498105a40 2024-04-05T21:40:41.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.481+0000 7f1496d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14980ffbd0 0x7f1498196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:41.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.481+0000 7f1496d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14980ffbd0 0x7f1498196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60790/0 (socket says 172.21.15.121:60790) 2024-04-05T21:40:41.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.481+0000 7f1496d9d700 1 -- 172.21.15.121:0/1386206077 learned_addr learned my addr 172.21.15.121:0/1386206077 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:41.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.481+0000 7f149659c700 1 --2- 172.21.15.121:0/1386206077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1498105a40 0x7f1498196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:41.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.481+0000 7f1496d9d700 1 -- 172.21.15.121:0/1386206077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1498105a40 msgr2=0x7f1498196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:41.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.481+0000 7f1496d9d700 1 --2- 172.21.15.121:0/1386206077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1498105a40 0x7f1498196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.481+0000 7f1496d9d700 1 -- 172.21.15.121:0/1386206077 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14880095f0 con 0x7f14980ffbd0 2024-04-05T21:40:41.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.481+0000 7f1496d9d700 1 --2- 172.21.15.121:0/1386206077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14980ffbd0 0x7f1498196640 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1488006010 tx=0x7f148800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:41.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.481+0000 7f147ffff700 1 -- 172.21.15.121:0/1386206077 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f148801d070 con 0x7f14980ffbd0 2024-04-05T21:40:41.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.481+0000 7f149d1bd700 1 -- 172.21.15.121:0/1386206077 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149819b270 con 0x7f14980ffbd0 2024-04-05T21:40:41.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.481+0000 7f147ffff700 1 -- 172.21.15.121:0/1386206077 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f148800f460 con 0x7f14980ffbd0 2024-04-05T21:40:41.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.481+0000 7f147ffff700 1 -- 172.21.15.121:0/1386206077 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1488021620 con 0x7f14980ffbd0 2024-04-05T21:40:41.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.482+0000 7f149d1bd700 1 -- 172.21.15.121:0/1386206077 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149819b760 con 0x7f14980ffbd0 2024-04-05T21:40:41.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.482+0000 7f149d1bd700 1 -- 172.21.15.121:0/1386206077 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1498109050 con 0x7f14980ffbd0 2024-04-05T21:40:41.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.483+0000 7f147ffff700 1 -- 172.21.15.121:0/1386206077 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1488003a60 con 0x7f14980ffbd0 2024-04-05T21:40:41.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.483+0000 7f147ffff700 1 --2- 172.21.15.121:0/1386206077 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f148006cce0 0x7f148006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:41.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.483+0000 7f147ffff700 1 -- 172.21.15.121:0/1386206077 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f148808d810 con 0x7f14980ffbd0 2024-04-05T21:40:41.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.483+0000 7f149659c700 1 --2- 172.21.15.121:0/1386206077 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f148006cce0 0x7f148006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:41.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.483+0000 7f149659c700 1 --2- 172.21.15.121:0/1386206077 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f148006cce0 0x7f148006f1a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1498197c60 tx=0x7f148c00a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:41.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.488+0000 7f147ffff700 1 -- 172.21.15.121:0/1386206077 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1488057880 con 0x7f14980ffbd0 2024-04-05T21:40:41.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:41 smithi121 ceph-mon[145859]: pgmap v433: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:41.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:41 smithi121 ceph-mon[145859]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:41.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.759+0000 7f149d1bd700 1 -- 172.21.15.121:0/1386206077 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1498108a30 con 0x7f14980ffbd0 2024-04-05T21:40:41.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.760+0000 7f147ffff700 1 -- 172.21.15.121:0/1386206077 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f148805aea0 con 0x7f14980ffbd0 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:40:41.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:40:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:40:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:40:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:40:41.762 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:40:41.762 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:40:41.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.762+0000 7f149d1bd700 1 -- 172.21.15.121:0/1386206077 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f148006cce0 msgr2=0x7f148006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:41.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.762+0000 7f149d1bd700 1 --2- 172.21.15.121:0/1386206077 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f148006cce0 0x7f148006f1a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1498197c60 tx=0x7f148c00a300 comp rx=0 tx=0).stop 2024-04-05T21:40:41.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.762+0000 7f149d1bd700 1 -- 172.21.15.121:0/1386206077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14980ffbd0 msgr2=0x7f1498196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:41.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.762+0000 7f149d1bd700 1 --2- 172.21.15.121:0/1386206077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14980ffbd0 0x7f1498196640 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1488006010 tx=0x7f148800bd70 comp rx=0 tx=0).stop 2024-04-05T21:40:41.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.762+0000 7f149d1bd700 1 -- 172.21.15.121:0/1386206077 shutdown_connections 2024-04-05T21:40:41.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.762+0000 7f149d1bd700 1 --2- 172.21.15.121:0/1386206077 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f148006cce0 0x7f148006f1a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.762+0000 7f149d1bd700 1 --2- 172.21.15.121:0/1386206077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14980ffbd0 0x7f1498196640 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.762+0000 7f149d1bd700 1 --2- 172.21.15.121:0/1386206077 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1498105a40 0x7f1498196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.762+0000 7f149d1bd700 1 -- 172.21.15.121:0/1386206077 >> 172.21.15.121:0/1386206077 conn(0x7f14980755a0 msgr2=0x7f1498068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:41.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.762+0000 7f149d1bd700 1 -- 172.21.15.121:0/1386206077 shutdown_connections 2024-04-05T21:40:41.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.762+0000 7f149d1bd700 1 -- 172.21.15.121:0/1386206077 wait complete. 2024-04-05T21:40:41.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:41 smithi188 ceph-mon[107793]: pgmap v433: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:41.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:41 smithi188 ceph-mon[107793]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:41.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.865+0000 7fe900da9700 1 -- 172.21.15.121:0/4008915464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8fc105a50 msgr2=0x7fe8fc105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:41.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.865+0000 7fe900da9700 1 --2- 172.21.15.121:0/4008915464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8fc105a50 0x7fe8fc105e30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009910 tx=0x7fe8ec009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:41.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.865+0000 7fe900da9700 1 -- 172.21.15.121:0/4008915464 shutdown_connections 2024-04-05T21:40:41.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.865+0000 7fe900da9700 1 --2- 172.21.15.121:0/4008915464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8fc0ffa50 0x7fe8fc0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.865+0000 7fe900da9700 1 --2- 172.21.15.121:0/4008915464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8fc105a50 0x7fe8fc105e30 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.866+0000 7fe900da9700 1 -- 172.21.15.121:0/4008915464 >> 172.21.15.121:0/4008915464 conn(0x7fe8fc0fb540 msgr2=0x7fe8fc0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:41.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.866+0000 7fe900da9700 1 -- 172.21.15.121:0/4008915464 shutdown_connections 2024-04-05T21:40:41.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.866+0000 7fe900da9700 1 -- 172.21.15.121:0/4008915464 wait complete. 2024-04-05T21:40:41.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.866+0000 7fe900da9700 1 Processor -- start 2024-04-05T21:40:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.866+0000 7fe900da9700 1 -- start start 2024-04-05T21:40:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.866+0000 7fe900da9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8fc0ffa50 0x7fe8fc0fbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe900da9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8fc105a50 0x7fe8fc0fc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe900da9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc0fc940 con 0x7fe8fc105a50 2024-04-05T21:40:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe900da9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc0fcab0 con 0x7fe8fc0ffa50 2024-04-05T21:40:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe8f9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8fc105a50 0x7fe8fc0fc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe8f9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8fc105a50 0x7fe8fc0fc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60796/0 (socket says 172.21.15.121:60796) 2024-04-05T21:40:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe8f9d9b700 1 -- 172.21.15.121:0/1586316055 learned_addr learned my addr 172.21.15.121:0/1586316055 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe8fa59c700 1 --2- 172.21.15.121:0/1586316055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8fc0ffa50 0x7fe8fc0fbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe8f9d9b700 1 -- 172.21.15.121:0/1586316055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8fc0ffa50 msgr2=0x7fe8fc0fbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe8f9d9b700 1 --2- 172.21.15.121:0/1586316055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8fc0ffa50 0x7fe8fc0fbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe8f9d9b700 1 -- 172.21.15.121:0/1586316055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec0095f0 con 0x7fe8fc105a50 2024-04-05T21:40:41.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe8f9d9b700 1 --2- 172.21.15.121:0/1586316055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8fc105a50 0x7fe8fc0fc370 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000e920 tx=0x7fe8f000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe8eb7fe700 1 -- 172.21.15.121:0/1586316055 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8f0009790 con 0x7fe8fc105a50 2024-04-05T21:40:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe8eb7fe700 1 -- 172.21.15.121:0/1586316055 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8f000cba0 con 0x7fe8fc105a50 2024-04-05T21:40:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe900da9700 1 -- 172.21.15.121:0/1586316055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8fc1a2810 con 0x7fe8fc105a50 2024-04-05T21:40:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe8eb7fe700 1 -- 172.21.15.121:0/1586316055 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8f0018650 con 0x7fe8fc105a50 2024-04-05T21:40:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.867+0000 7fe900da9700 1 -- 172.21.15.121:0/1586316055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8fc1a2c80 con 0x7fe8fc105a50 2024-04-05T21:40:41.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.868+0000 7fe900da9700 1 -- 172.21.15.121:0/1586316055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8fc109060 con 0x7fe8fc105a50 2024-04-05T21:40:41.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.869+0000 7fe8eb7fe700 1 -- 172.21.15.121:0/1586316055 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe8f000cd10 con 0x7fe8fc105a50 2024-04-05T21:40:41.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.869+0000 7fe8eb7fe700 1 --2- 172.21.15.121:0/1586316055 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe8e406cce0 0x7fe8e406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:41.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.869+0000 7fe8eb7fe700 1 -- 172.21.15.121:0/1586316055 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe8f0014070 con 0x7fe8fc105a50 2024-04-05T21:40:41.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.869+0000 7fe8fa59c700 1 --2- 172.21.15.121:0/1586316055 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe8e406cce0 0x7fe8e406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:41.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.870+0000 7fe8fa59c700 1 --2- 172.21.15.121:0/1586316055 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe8e406cce0 0x7fe8e406f1a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009fd0 tx=0x7fe8ec005940 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:41.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:41.874+0000 7fe8eb7fe700 1 -- 172.21.15.121:0/1586316055 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe8f0052ab0 con 0x7fe8fc105a50 2024-04-05T21:40:42.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.108+0000 7fe900da9700 1 -- 172.21.15.121:0/1586316055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe8fc02cc70 con 0x7fe8fc105a50 2024-04-05T21:40:42.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.108+0000 7fe8eb7fe700 1 -- 172.21.15.121:0/1586316055 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fe8f005a450 con 0x7fe8fc105a50 2024-04-05T21:40:42.109 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:40:42.109 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:40:42.110 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:40:42.110 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:40:42.110 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:40:42.110 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:40:42.110 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:40:42.110 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:40:42.110 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:40:42.110 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:40:42.110 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:40:42.110 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:40:42.110 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:40:42.110 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:40:42.110 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:40:42.111 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:40:42.112 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:40:42.112 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:40:42.112 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:40:42.112 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:40:42.112 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:40:42.112 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:40:42.112 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:40:42.112 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:40:42.112 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:40:42.112 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:40:42.112 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:40:42.112 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:40:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.111+0000 7fe900da9700 1 -- 172.21.15.121:0/1586316055 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe8e406cce0 msgr2=0x7fe8e406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.111+0000 7fe900da9700 1 --2- 172.21.15.121:0/1586316055 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe8e406cce0 0x7fe8e406f1a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009fd0 tx=0x7fe8ec005940 comp rx=0 tx=0).stop 2024-04-05T21:40:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.111+0000 7fe900da9700 1 -- 172.21.15.121:0/1586316055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8fc105a50 msgr2=0x7fe8fc0fc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.111+0000 7fe900da9700 1 --2- 172.21.15.121:0/1586316055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8fc105a50 0x7fe8fc0fc370 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000e920 tx=0x7fe8f000ece0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.111+0000 7fe900da9700 1 -- 172.21.15.121:0/1586316055 shutdown_connections 2024-04-05T21:40:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.111+0000 7fe900da9700 1 --2- 172.21.15.121:0/1586316055 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe8e406cce0 0x7fe8e406f1a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.111+0000 7fe900da9700 1 --2- 172.21.15.121:0/1586316055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe8fc0ffa50 0x7fe8fc0fbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.111+0000 7fe900da9700 1 --2- 172.21.15.121:0/1586316055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8fc105a50 0x7fe8fc0fc370 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.111+0000 7fe900da9700 1 -- 172.21.15.121:0/1586316055 >> 172.21.15.121:0/1586316055 conn(0x7fe8fc0fb540 msgr2=0x7fe8fc068f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.111+0000 7fe900da9700 1 -- 172.21.15.121:0/1586316055 shutdown_connections 2024-04-05T21:40:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.111+0000 7fe900da9700 1 -- 172.21.15.121:0/1586316055 wait complete. 2024-04-05T21:40:42.114 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:40:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.215+0000 7f314d995700 1 -- 172.21.15.121:0/451930675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3148069020 msgr2=0x7f3148105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.215+0000 7f314d995700 1 --2- 172.21.15.121:0/451930675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3148069020 0x7f3148105790 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3138009910 tx=0x7f3138009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.215+0000 7f314d995700 1 -- 172.21.15.121:0/451930675 shutdown_connections 2024-04-05T21:40:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.215+0000 7f314d995700 1 --2- 172.21.15.121:0/451930675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3148069020 0x7f3148105790 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.215+0000 7f314d995700 1 --2- 172.21.15.121:0/451930675 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3148068670 0x7f3148068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.215+0000 7f314d995700 1 -- 172.21.15.121:0/451930675 >> 172.21.15.121:0/451930675 conn(0x7f31480755a0 msgr2=0x7f31480759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.215+0000 7f314d995700 1 -- 172.21.15.121:0/451930675 shutdown_connections 2024-04-05T21:40:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.215+0000 7f314d995700 1 -- 172.21.15.121:0/451930675 wait complete. 2024-04-05T21:40:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.215+0000 7f314d995700 1 Processor -- start 2024-04-05T21:40:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.215+0000 7f314d995700 1 -- start start 2024-04-05T21:40:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f314d995700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3148068670 0x7f31480ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f314d995700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3148069020 0x7f31481001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f314d995700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3148100720 con 0x7f3148069020 2024-04-05T21:40:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f314d995700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3148100860 con 0x7f3148068670 2024-04-05T21:40:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f31467fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3148069020 0x7f31481001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f31467fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3148069020 0x7f31481001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60814/0 (socket says 172.21.15.121:60814) 2024-04-05T21:40:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f31467fc700 1 -- 172.21.15.121:0/3682331747 learned_addr learned my addr 172.21.15.121:0/3682331747 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f3146ffd700 1 --2- 172.21.15.121:0/3682331747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3148068670 0x7f31480ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f31467fc700 1 -- 172.21.15.121:0/3682331747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3148068670 msgr2=0x7f31480ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f31467fc700 1 --2- 172.21.15.121:0/3682331747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3148068670 0x7f31480ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f31467fc700 1 -- 172.21.15.121:0/3682331747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31380095f0 con 0x7f3148069020 2024-04-05T21:40:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f31467fc700 1 --2- 172.21.15.121:0/3682331747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3148069020 0x7f31481001e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f313800bda0 tx=0x7f313800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f314c993700 1 -- 172.21.15.121:0/3682331747 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f313801d070 con 0x7f3148069020 2024-04-05T21:40:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f314c993700 1 -- 172.21.15.121:0/3682331747 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f313800f460 con 0x7f3148069020 2024-04-05T21:40:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f314d995700 1 -- 172.21.15.121:0/3682331747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3148100ae0 con 0x7f3148069020 2024-04-05T21:40:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f314c993700 1 -- 172.21.15.121:0/3682331747 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3138021640 con 0x7f3148069020 2024-04-05T21:40:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.216+0000 7f314d995700 1 -- 172.21.15.121:0/3682331747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31481a2b40 con 0x7f3148069020 2024-04-05T21:40:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.217+0000 7f314d995700 1 -- 172.21.15.121:0/3682331747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31481090a0 con 0x7f3148069020 2024-04-05T21:40:42.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.218+0000 7f314c993700 1 -- 172.21.15.121:0/3682331747 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f313800fa60 con 0x7f3148069020 2024-04-05T21:40:42.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.218+0000 7f314c993700 1 --2- 172.21.15.121:0/3682331747 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f313006cce0 0x7f313006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:42.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.218+0000 7f314c993700 1 -- 172.21.15.121:0/3682331747 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f313808d5f0 con 0x7f3148069020 2024-04-05T21:40:42.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.218+0000 7f3146ffd700 1 --2- 172.21.15.121:0/3682331747 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f313006cce0 0x7f313006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:42.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.219+0000 7f3146ffd700 1 --2- 172.21.15.121:0/3682331747 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f313006cce0 0x7f313006f1a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f313c005950 tx=0x7f313c00b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:42.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.223+0000 7f314c993700 1 -- 172.21.15.121:0/3682331747 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f31380576e0 con 0x7f3148069020 2024-04-05T21:40:42.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.435+0000 7f314d995700 1 -- 172.21.15.121:0/3682331747 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f314802cce0 con 0x7f313006cce0 2024-04-05T21:40:42.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.436+0000 7f314c993700 1 -- 172.21.15.121:0/3682331747 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f314802cce0 con 0x7f313006cce0 2024-04-05T21:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:40:42.437 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:40:42.438 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:40:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.438+0000 7f314d995700 1 -- 172.21.15.121:0/3682331747 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f313006cce0 msgr2=0x7f313006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.438+0000 7f314d995700 1 --2- 172.21.15.121:0/3682331747 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f313006cce0 0x7f313006f1a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f313c005950 tx=0x7f313c00b410 comp rx=0 tx=0).stop 2024-04-05T21:40:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.438+0000 7f314d995700 1 -- 172.21.15.121:0/3682331747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3148069020 msgr2=0x7f31481001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.438+0000 7f314d995700 1 --2- 172.21.15.121:0/3682331747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3148069020 0x7f31481001e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f313800bda0 tx=0x7f313800bdd0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.438+0000 7f314d995700 1 -- 172.21.15.121:0/3682331747 shutdown_connections 2024-04-05T21:40:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.438+0000 7f314d995700 1 --2- 172.21.15.121:0/3682331747 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f313006cce0 0x7f313006f1a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.438+0000 7f314d995700 1 --2- 172.21.15.121:0/3682331747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3148068670 0x7f31480ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.438+0000 7f314d995700 1 --2- 172.21.15.121:0/3682331747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3148069020 0x7f31481001e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.438+0000 7f314d995700 1 -- 172.21.15.121:0/3682331747 >> 172.21.15.121:0/3682331747 conn(0x7f31480755a0 msgr2=0x7f3148105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:42.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.438+0000 7f314d995700 1 -- 172.21.15.121:0/3682331747 shutdown_connections 2024-04-05T21:40:42.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.438+0000 7f314d995700 1 -- 172.21.15.121:0/3682331747 wait complete. 2024-04-05T21:40:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.541+0000 7f4730ccb700 1 -- 172.21.15.121:0/837047773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f472c101770 msgr2=0x7f472c1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.541+0000 7f4730ccb700 1 --2- 172.21.15.121:0/837047773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f472c101770 0x7f472c1057c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f471c009910 tx=0x7f471c009c20 comp rx=0 tx=0).stop 2024-04-05T21:40:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.541+0000 7f4730ccb700 1 -- 172.21.15.121:0/837047773 shutdown_connections 2024-04-05T21:40:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.541+0000 7f4730ccb700 1 --2- 172.21.15.121:0/837047773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f472c101770 0x7f472c1057c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.541+0000 7f4730ccb700 1 --2- 172.21.15.121:0/837047773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f472c100dc0 0x7f472c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.541+0000 7f4730ccb700 1 -- 172.21.15.121:0/837047773 >> 172.21.15.121:0/837047773 conn(0x7f472c0fc650 msgr2=0x7f472c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.541+0000 7f4730ccb700 1 -- 172.21.15.121:0/837047773 shutdown_connections 2024-04-05T21:40:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.541+0000 7f4730ccb700 1 -- 172.21.15.121:0/837047773 wait complete. 2024-04-05T21:40:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.542+0000 7f4730ccb700 1 Processor -- start 2024-04-05T21:40:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.542+0000 7f4730ccb700 1 -- start start 2024-04-05T21:40:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.542+0000 7f4730ccb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f472c100dc0 0x7f472c1966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.542+0000 7f4730ccb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f472c101770 0x7f472c196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.542+0000 7f4730ccb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c1972a0 con 0x7f472c100dc0 2024-04-05T21:40:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.542+0000 7f4730ccb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c199fd0 con 0x7f472c101770 2024-04-05T21:40:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.542+0000 7f472a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f472c100dc0 0x7f472c1966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.542+0000 7f472a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f472c100dc0 0x7f472c1966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60838/0 (socket says 172.21.15.121:60838) 2024-04-05T21:40:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.542+0000 7f472a59c700 1 -- 172.21.15.121:0/1453133683 learned_addr learned my addr 172.21.15.121:0/1453133683 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:40:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.543+0000 7f4729d9b700 1 --2- 172.21.15.121:0/1453133683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f472c101770 0x7f472c196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.543+0000 7f472a59c700 1 -- 172.21.15.121:0/1453133683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f472c101770 msgr2=0x7f472c196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.543+0000 7f472a59c700 1 --2- 172.21.15.121:0/1453133683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f472c101770 0x7f472c196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.543+0000 7f472a59c700 1 -- 172.21.15.121:0/1453133683 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c0095f0 con 0x7f472c100dc0 2024-04-05T21:40:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.543+0000 7f472a59c700 1 --2- 172.21.15.121:0/1453133683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f472c100dc0 0x7f472c1966d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f472000d8d0 tx=0x7f472000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.543+0000 7f471b7fe700 1 -- 172.21.15.121:0/1453133683 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4720009940 con 0x7f472c100dc0 2024-04-05T21:40:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.543+0000 7f471b7fe700 1 -- 172.21.15.121:0/1453133683 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4720010460 con 0x7f472c100dc0 2024-04-05T21:40:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.543+0000 7f4730ccb700 1 -- 172.21.15.121:0/1453133683 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c19a2b0 con 0x7f472c100dc0 2024-04-05T21:40:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.543+0000 7f471b7fe700 1 -- 172.21.15.121:0/1453133683 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f472000f5f0 con 0x7f472c100dc0 2024-04-05T21:40:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.543+0000 7f4730ccb700 1 -- 172.21.15.121:0/1453133683 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c19a7d0 con 0x7f472c100dc0 2024-04-05T21:40:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.544+0000 7f4730ccb700 1 -- 172.21.15.121:0/1453133683 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c1091c0 con 0x7f472c100dc0 2024-04-05T21:40:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.544+0000 7f471b7fe700 1 -- 172.21.15.121:0/1453133683 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f47200105d0 con 0x7f472c100dc0 2024-04-05T21:40:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.545+0000 7f471b7fe700 1 --2- 172.21.15.121:0/1453133683 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f471406cce0 0x7f471406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:40:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.545+0000 7f471b7fe700 1 -- 172.21.15.121:0/1453133683 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f472008c5e0 con 0x7f472c100dc0 2024-04-05T21:40:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.545+0000 7f4729d9b700 1 --2- 172.21.15.121:0/1453133683 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f471406cce0 0x7f471406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:40:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.545+0000 7f4729d9b700 1 --2- 172.21.15.121:0/1453133683 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f471406cce0 0x7f471406f1a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f472c197df0 tx=0x7f471c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:40:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.549+0000 7f471b7fe700 1 -- 172.21.15.121:0/1453133683 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4720056650 con 0x7f472c100dc0 2024-04-05T21:40:42.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:42 smithi121 ceph-mon[145859]: from='client.34654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:42.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:42 smithi121 ceph-mon[145859]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:42.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:42 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1386206077' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:40:42.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:42 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1586316055' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:40:42.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:42 smithi188 ceph-mon[107793]: from='client.34654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:42.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:42 smithi188 ceph-mon[107793]: from='client.34658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:42.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:42 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1386206077' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:40:42.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:42 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1586316055' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:40:42.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.821+0000 7f4730ccb700 1 -- 172.21.15.121:0/1453133683 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f472c04ea90 con 0x7f472c100dc0 2024-04-05T21:40:42.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.821+0000 7f471b7fe700 1 -- 172.21.15.121:0/1453133683 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f4720010880 con 0x7f472c100dc0 2024-04-05T21:40:42.822 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:40:42.822 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:40:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:40:42.822 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:40:42.822 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:40:42.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.823+0000 7f4730ccb700 1 -- 172.21.15.121:0/1453133683 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f471406cce0 msgr2=0x7f471406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:42.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.823+0000 7f4730ccb700 1 --2- 172.21.15.121:0/1453133683 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f471406cce0 0x7f471406f1a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f472c197df0 tx=0x7f471c01a040 comp rx=0 tx=0).stop 2024-04-05T21:40:42.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.823+0000 7f4730ccb700 1 -- 172.21.15.121:0/1453133683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f472c100dc0 msgr2=0x7f472c1966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:40:42.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.823+0000 7f4730ccb700 1 --2- 172.21.15.121:0/1453133683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f472c100dc0 0x7f472c1966d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f472000d8d0 tx=0x7f472000dc90 comp rx=0 tx=0).stop 2024-04-05T21:40:42.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.823+0000 7f4730ccb700 1 -- 172.21.15.121:0/1453133683 shutdown_connections 2024-04-05T21:40:42.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.823+0000 7f4730ccb700 1 --2- 172.21.15.121:0/1453133683 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f471406cce0 0x7f471406f1a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.823+0000 7f4730ccb700 1 --2- 172.21.15.121:0/1453133683 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f472c100dc0 0x7f472c1966d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.823+0000 7f4730ccb700 1 --2- 172.21.15.121:0/1453133683 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f472c101770 0x7f472c196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:40:42.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.823+0000 7f4730ccb700 1 -- 172.21.15.121:0/1453133683 >> 172.21.15.121:0/1453133683 conn(0x7f472c0fc650 msgr2=0x7f472c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:40:42.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.823+0000 7f4730ccb700 1 -- 172.21.15.121:0/1453133683 shutdown_connections 2024-04-05T21:40:42.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:40:42.823+0000 7f4730ccb700 1 -- 172.21.15.121:0/1453133683 wait complete. 2024-04-05T21:40:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:43 smithi121 ceph-mon[145859]: pgmap v434: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:43 smithi121 ceph-mon[145859]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:43 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1453133683' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:40:43.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:43 smithi188 ceph-mon[107793]: pgmap v434: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:43 smithi188 ceph-mon[107793]: from='client.34670 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:40:43.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:43 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1453133683' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:40:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:45 smithi121 ceph-mon[145859]: pgmap v435: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:40:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:45 smithi188 ceph-mon[107793]: pgmap v435: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:40:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:47 smithi121 ceph-mon[145859]: pgmap v436: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:47 smithi188 ceph-mon[107793]: pgmap v436: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:49 smithi121 ceph-mon[145859]: pgmap v437: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:49.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:49 smithi188 ceph-mon[107793]: pgmap v437: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:51 smithi121 ceph-mon[145859]: pgmap v438: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:51 smithi188 ceph-mon[107793]: pgmap v438: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:53 smithi121 ceph-mon[145859]: pgmap v439: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:53 smithi188 ceph-mon[107793]: pgmap v439: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:55.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:55 smithi121 ceph-mon[145859]: pgmap v440: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:55 smithi188 ceph-mon[107793]: pgmap v440: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:57 smithi121 ceph-mon[145859]: pgmap v441: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:57 smithi188 ceph-mon[107793]: pgmap v441: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:59.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:40:59 smithi121 ceph-mon[145859]: pgmap v442: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:40:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:40:59 smithi188 ceph-mon[107793]: pgmap v442: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:00.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:41:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:41:01.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:01 smithi121 ceph-mon[145859]: pgmap v443: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:01 smithi188 ceph-mon[107793]: pgmap v443: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:03.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:03 smithi121 ceph-mon[145859]: pgmap v444: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:03.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:03 smithi188 ceph-mon[107793]: pgmap v444: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:05 smithi121 ceph-mon[145859]: pgmap v445: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:05 smithi188 ceph-mon[107793]: pgmap v445: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:07.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:07 smithi121 ceph-mon[145859]: pgmap v446: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:07.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:07 smithi188 ceph-mon[107793]: pgmap v446: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:09.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:09 smithi121 ceph-mon[145859]: pgmap v447: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:09 smithi188 ceph-mon[107793]: pgmap v447: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:11 smithi121 ceph-mon[145859]: pgmap v448: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:11.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:11 smithi188 ceph-mon[107793]: pgmap v448: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:12.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.934+0000 7f026b4e6700 1 -- 172.21.15.121:0/4278287338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0264100da0 msgr2=0x7f0264101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:12.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.934+0000 7f026b4e6700 1 --2- 172.21.15.121:0/4278287338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0264100da0 0x7f0264101180 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0254009910 tx=0x7f0254009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:12.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.934+0000 7f026b4e6700 1 -- 172.21.15.121:0/4278287338 shutdown_connections 2024-04-05T21:41:12.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.934+0000 7f026b4e6700 1 --2- 172.21.15.121:0/4278287338 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0264101750 0x7f02641057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:12.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.934+0000 7f026b4e6700 1 --2- 172.21.15.121:0/4278287338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0264100da0 0x7f0264101180 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:12.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.934+0000 7f026b4e6700 1 -- 172.21.15.121:0/4278287338 >> 172.21.15.121:0/4278287338 conn(0x7f02640fc630 msgr2=0x7f02640fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:12.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.934+0000 7f026b4e6700 1 -- 172.21.15.121:0/4278287338 shutdown_connections 2024-04-05T21:41:12.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.934+0000 7f026b4e6700 1 -- 172.21.15.121:0/4278287338 wait complete. 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.935+0000 7f026b4e6700 1 Processor -- start 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.935+0000 7f026b4e6700 1 -- start start 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.935+0000 7f026b4e6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0264100da0 0x7f0264073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.935+0000 7f026b4e6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0264101750 0x7f02640735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.935+0000 7f026b4e6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0264073b10 con 0x7f0264100da0 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.935+0000 7f026b4e6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0264073c50 con 0x7f0264101750 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.935+0000 7f0269282700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0264100da0 0x7f0264073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.935+0000 7f0269282700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0264100da0 0x7f0264073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52710/0 (socket says 172.21.15.121:52710) 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.935+0000 7f0269282700 1 -- 172.21.15.121:0/3065701373 learned_addr learned my addr 172.21.15.121:0/3065701373 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.935+0000 7f0268a81700 1 --2- 172.21.15.121:0/3065701373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0264101750 0x7f02640735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.936+0000 7f0269282700 1 -- 172.21.15.121:0/3065701373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0264101750 msgr2=0x7f02640735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.936+0000 7f0269282700 1 --2- 172.21.15.121:0/3065701373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0264101750 0x7f02640735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.936+0000 7f0269282700 1 -- 172.21.15.121:0/3065701373 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02540095f0 con 0x7f0264100da0 2024-04-05T21:41:12.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.936+0000 7f0269282700 1 --2- 172.21.15.121:0/3065701373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0264100da0 0x7f0264073090 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0254000c00 tx=0x7f025400bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:12.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.936+0000 7f025a7fc700 1 -- 172.21.15.121:0/3065701373 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f025401d070 con 0x7f0264100da0 2024-04-05T21:41:12.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.936+0000 7f025a7fc700 1 -- 172.21.15.121:0/3065701373 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0254005dc0 con 0x7f0264100da0 2024-04-05T21:41:12.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.936+0000 7f026b4e6700 1 -- 172.21.15.121:0/3065701373 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0264073ed0 con 0x7f0264100da0 2024-04-05T21:41:12.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.936+0000 7f025a7fc700 1 -- 172.21.15.121:0/3065701373 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0254003940 con 0x7f0264100da0 2024-04-05T21:41:12.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.936+0000 7f026b4e6700 1 -- 172.21.15.121:0/3065701373 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02641a2b30 con 0x7f0264100da0 2024-04-05T21:41:12.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.937+0000 7f026b4e6700 1 -- 172.21.15.121:0/3065701373 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0264109140 con 0x7f0264100da0 2024-04-05T21:41:12.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.937+0000 7f025a7fc700 1 -- 172.21.15.121:0/3065701373 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0254022470 con 0x7f0264100da0 2024-04-05T21:41:12.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.938+0000 7f025a7fc700 1 --2- 172.21.15.121:0/3065701373 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f025006cce0 0x7f025006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:12.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.938+0000 7f025a7fc700 1 -- 172.21.15.121:0/3065701373 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f025408daf0 con 0x7f0264100da0 2024-04-05T21:41:12.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.938+0000 7f0268a81700 1 --2- 172.21.15.121:0/3065701373 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f025006cce0 0x7f025006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:12.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.938+0000 7f0268a81700 1 --2- 172.21.15.121:0/3065701373 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f025006cce0 0x7f025006f1a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0264074880 tx=0x7f0260009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:12.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:12.943+0000 7f025a7fc700 1 -- 172.21.15.121:0/3065701373 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0254057b00 con 0x7f0264100da0 2024-04-05T21:41:13.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.152+0000 7f026b4e6700 1 -- 172.21.15.121:0/3065701373 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f02640611d0 con 0x7f025006cce0 2024-04-05T21:41:13.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.153+0000 7f025a7fc700 1 -- 172.21.15.121:0/3065701373 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f02640611d0 con 0x7f025006cce0 2024-04-05T21:41:13.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.155+0000 7f026b4e6700 1 -- 172.21.15.121:0/3065701373 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f025006cce0 msgr2=0x7f025006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:13.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.155+0000 7f026b4e6700 1 --2- 172.21.15.121:0/3065701373 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f025006cce0 0x7f025006f1a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0264074880 tx=0x7f0260009260 comp rx=0 tx=0).stop 2024-04-05T21:41:13.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.155+0000 7f026b4e6700 1 -- 172.21.15.121:0/3065701373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0264100da0 msgr2=0x7f0264073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:13.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.155+0000 7f026b4e6700 1 --2- 172.21.15.121:0/3065701373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0264100da0 0x7f0264073090 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0254000c00 tx=0x7f025400bde0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.155+0000 7f026b4e6700 1 -- 172.21.15.121:0/3065701373 shutdown_connections 2024-04-05T21:41:13.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.155+0000 7f026b4e6700 1 --2- 172.21.15.121:0/3065701373 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f025006cce0 0x7f025006f1a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.155+0000 7f026b4e6700 1 --2- 172.21.15.121:0/3065701373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0264100da0 0x7f0264073090 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.155+0000 7f026b4e6700 1 --2- 172.21.15.121:0/3065701373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0264101750 0x7f02640735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.155+0000 7f026b4e6700 1 -- 172.21.15.121:0/3065701373 >> 172.21.15.121:0/3065701373 conn(0x7f02640fc630 msgr2=0x7f02640fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:13.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.155+0000 7f026b4e6700 1 -- 172.21.15.121:0/3065701373 shutdown_connections 2024-04-05T21:41:13.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.155+0000 7f026b4e6700 1 -- 172.21.15.121:0/3065701373 wait complete. 2024-04-05T21:41:13.166 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:41:13.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.259+0000 7f47bb55c700 1 -- 172.21.15.121:0/1911728903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b4101750 msgr2=0x7f47b41057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:13.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.259+0000 7f47bb55c700 1 --2- 172.21.15.121:0/1911728903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b4101750 0x7f47b41057a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009910 tx=0x7f47b0009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:13.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.259+0000 7f47bb55c700 1 -- 172.21.15.121:0/1911728903 shutdown_connections 2024-04-05T21:41:13.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.259+0000 7f47bb55c700 1 --2- 172.21.15.121:0/1911728903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b4101750 0x7f47b41057a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.259+0000 7f47bb55c700 1 --2- 172.21.15.121:0/1911728903 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b4100da0 0x7f47b4101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.259+0000 7f47bb55c700 1 -- 172.21.15.121:0/1911728903 >> 172.21.15.121:0/1911728903 conn(0x7f47b40fc630 msgr2=0x7f47b40fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:13.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.259+0000 7f47bb55c700 1 -- 172.21.15.121:0/1911728903 shutdown_connections 2024-04-05T21:41:13.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.259+0000 7f47bb55c700 1 -- 172.21.15.121:0/1911728903 wait complete. 2024-04-05T21:41:13.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.259+0000 7f47bb55c700 1 Processor -- start 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.259+0000 7f47bb55c700 1 -- start start 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47bb55c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b4100da0 0x7f47b419c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47bb55c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b4101750 0x7f47b419cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47bb55c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b419d140 con 0x7f47b4100da0 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47bb55c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b41965f0 con 0x7f47b4101750 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47b92f8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b4100da0 0x7f47b419c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47b92f8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b4100da0 0x7f47b419c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52740/0 (socket says 172.21.15.121:52740) 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47b92f8700 1 -- 172.21.15.121:0/2410331174 learned_addr learned my addr 172.21.15.121:0/2410331174 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47b8af7700 1 --2- 172.21.15.121:0/2410331174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b4101750 0x7f47b419cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47b92f8700 1 -- 172.21.15.121:0/2410331174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b4101750 msgr2=0x7f47b419cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47b92f8700 1 --2- 172.21.15.121:0/2410331174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b4101750 0x7f47b419cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47b92f8700 1 -- 172.21.15.121:0/2410331174 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b00095f0 con 0x7f47b4100da0 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47b92f8700 1 --2- 172.21.15.121:0/2410331174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b4100da0 0x7f47b419c570 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f47a400b4b0 tx=0x7f47a400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:13.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47aa7fc700 1 -- 172.21.15.121:0/2410331174 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a4011800 con 0x7f47b4100da0 2024-04-05T21:41:13.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47aa7fc700 1 -- 172.21.15.121:0/2410331174 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47a4011e00 con 0x7f47b4100da0 2024-04-05T21:41:13.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47bb55c700 1 -- 172.21.15.121:0/2410331174 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b41968d0 con 0x7f47b4100da0 2024-04-05T21:41:13.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47aa7fc700 1 -- 172.21.15.121:0/2410331174 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a4010e60 con 0x7f47b4100da0 2024-04-05T21:41:13.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.260+0000 7f47bb55c700 1 -- 172.21.15.121:0/2410331174 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b4196e20 con 0x7f47b4100da0 2024-04-05T21:41:13.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.261+0000 7f47bb55c700 1 -- 172.21.15.121:0/2410331174 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b41090d0 con 0x7f47b4100da0 2024-04-05T21:41:13.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.262+0000 7f47aa7fc700 1 -- 172.21.15.121:0/2410331174 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f47a4011960 con 0x7f47b4100da0 2024-04-05T21:41:13.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.262+0000 7f47aa7fc700 1 --2- 172.21.15.121:0/2410331174 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f47a006cce0 0x7f47a006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:13.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.262+0000 7f47aa7fc700 1 -- 172.21.15.121:0/2410331174 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f47a408c040 con 0x7f47b4100da0 2024-04-05T21:41:13.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.262+0000 7f47b8af7700 1 --2- 172.21.15.121:0/2410331174 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f47a006cce0 0x7f47a006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:13.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.262+0000 7f47b8af7700 1 --2- 172.21.15.121:0/2410331174 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f47a006cce0 0x7f47a006f1a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f47b40fd040 tx=0x7f47b001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:13.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.267+0000 7f47aa7fc700 1 -- 172.21.15.121:0/2410331174 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f47a4059bc0 con 0x7f47b4100da0 2024-04-05T21:41:13.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.480+0000 7f47bb55c700 1 -- 172.21.15.121:0/2410331174 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f47b41970a0 con 0x7f47a006cce0 2024-04-05T21:41:13.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.482+0000 7f47aa7fc700 1 -- 172.21.15.121:0/2410331174 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f47b41970a0 con 0x7f47a006cce0 2024-04-05T21:41:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.484+0000 7f47bb55c700 1 -- 172.21.15.121:0/2410331174 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f47a006cce0 msgr2=0x7f47a006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.484+0000 7f47bb55c700 1 --2- 172.21.15.121:0/2410331174 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f47a006cce0 0x7f47a006f1a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f47b40fd040 tx=0x7f47b001a040 comp rx=0 tx=0).stop 2024-04-05T21:41:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.484+0000 7f47bb55c700 1 -- 172.21.15.121:0/2410331174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b4100da0 msgr2=0x7f47b419c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.484+0000 7f47bb55c700 1 --2- 172.21.15.121:0/2410331174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b4100da0 0x7f47b419c570 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f47a400b4b0 tx=0x7f47a400b7c0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.484+0000 7f47bb55c700 1 -- 172.21.15.121:0/2410331174 shutdown_connections 2024-04-05T21:41:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.484+0000 7f47bb55c700 1 --2- 172.21.15.121:0/2410331174 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f47a006cce0 0x7f47a006f1a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.484+0000 7f47bb55c700 1 --2- 172.21.15.121:0/2410331174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b4100da0 0x7f47b419c570 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.484+0000 7f47bb55c700 1 --2- 172.21.15.121:0/2410331174 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b4101750 0x7f47b419cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.484+0000 7f47bb55c700 1 -- 172.21.15.121:0/2410331174 >> 172.21.15.121:0/2410331174 conn(0x7f47b40fc630 msgr2=0x7f47b40fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.485+0000 7f47bb55c700 1 -- 172.21.15.121:0/2410331174 shutdown_connections 2024-04-05T21:41:13.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.485+0000 7f47bb55c700 1 -- 172.21.15.121:0/2410331174 wait complete. 2024-04-05T21:41:13.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.587+0000 7fdbadf79700 1 -- 172.21.15.121:0/2845791821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdba80ffa40 msgr2=0x7fdba80ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:13.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.587+0000 7fdbadf79700 1 --2- 172.21.15.121:0/2845791821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdba80ffa40 0x7fdba80ffec0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009910 tx=0x7fdb9c009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:13.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.587+0000 7fdbadf79700 1 -- 172.21.15.121:0/2845791821 shutdown_connections 2024-04-05T21:41:13.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.587+0000 7fdbadf79700 1 --2- 172.21.15.121:0/2845791821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdba80ffa40 0x7fdba80ffec0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.587+0000 7fdbadf79700 1 --2- 172.21.15.121:0/2845791821 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba8105a40 0x7fdba8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.587+0000 7fdbadf79700 1 -- 172.21.15.121:0/2845791821 >> 172.21.15.121:0/2845791821 conn(0x7fdba80fb570 msgr2=0x7fdba80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:13.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.587+0000 7fdbadf79700 1 -- 172.21.15.121:0/2845791821 shutdown_connections 2024-04-05T21:41:13.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.587+0000 7fdbadf79700 1 -- 172.21.15.121:0/2845791821 wait complete. 2024-04-05T21:41:13.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.588+0000 7fdbadf79700 1 Processor -- start 2024-04-05T21:41:13.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.588+0000 7fdbadf79700 1 -- start start 2024-04-05T21:41:13.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.588+0000 7fdbadf79700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba80ffa40 0x7fdba81974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:13.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.588+0000 7fdbadf79700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdba8105a40 0x7fdba81979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:13.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.588+0000 7fdbadf79700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba8198010 con 0x7fdba8105a40 2024-04-05T21:41:13.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.588+0000 7fdbadf79700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba8198150 con 0x7fdba80ffa40 2024-04-05T21:41:13.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.589+0000 7fdba6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdba8105a40 0x7fdba81979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:13.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.589+0000 7fdba6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdba8105a40 0x7fdba81979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52766/0 (socket says 172.21.15.121:52766) 2024-04-05T21:41:13.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.589+0000 7fdba6ffd700 1 -- 172.21.15.121:0/1587916979 learned_addr learned my addr 172.21.15.121:0/1587916979 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:13.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.589+0000 7fdba77fe700 1 --2- 172.21.15.121:0/1587916979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba80ffa40 0x7fdba81974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:13.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.589+0000 7fdba6ffd700 1 -- 172.21.15.121:0/1587916979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba80ffa40 msgr2=0x7fdba81974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:13.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.589+0000 7fdba6ffd700 1 --2- 172.21.15.121:0/1587916979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba80ffa40 0x7fdba81974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.589+0000 7fdba6ffd700 1 -- 172.21.15.121:0/1587916979 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9c0095f0 con 0x7fdba8105a40 2024-04-05T21:41:13.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.589+0000 7fdba6ffd700 1 --2- 172.21.15.121:0/1587916979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdba8105a40 0x7fdba81979f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c006010 tx=0x7fdb9c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:13.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.589+0000 7fdba4ff9700 1 -- 172.21.15.121:0/1587916979 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb9c01d070 con 0x7fdba8105a40 2024-04-05T21:41:13.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.589+0000 7fdbadf79700 1 -- 172.21.15.121:0/1587916979 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba81a0520 con 0x7fdba8105a40 2024-04-05T21:41:13.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.589+0000 7fdba4ff9700 1 -- 172.21.15.121:0/1587916979 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb9c005710 con 0x7fdba8105a40 2024-04-05T21:41:13.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.589+0000 7fdba4ff9700 1 -- 172.21.15.121:0/1587916979 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb9c00fe20 con 0x7fdba8105a40 2024-04-05T21:41:13.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.589+0000 7fdbadf79700 1 -- 172.21.15.121:0/1587916979 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba81a0930 con 0x7fdba8105a40 2024-04-05T21:41:13.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.590+0000 7fdbadf79700 1 -- 172.21.15.121:0/1587916979 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba8107fd0 con 0x7fdba8105a40 2024-04-05T21:41:13.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.591+0000 7fdba4ff9700 1 -- 172.21.15.121:0/1587916979 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdb9c003a60 con 0x7fdba8105a40 2024-04-05T21:41:13.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.591+0000 7fdba4ff9700 1 --2- 172.21.15.121:0/1587916979 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb9006cce0 0x7fdb9006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:13.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.591+0000 7fdba4ff9700 1 -- 172.21.15.121:0/1587916979 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdb9c08d750 con 0x7fdba8105a40 2024-04-05T21:41:13.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.591+0000 7fdba77fe700 1 --2- 172.21.15.121:0/1587916979 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb9006cce0 0x7fdb9006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:13.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.592+0000 7fdba77fe700 1 --2- 172.21.15.121:0/1587916979 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb9006cce0 0x7fdb9006f1a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdb98005950 tx=0x7fdb9800b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:13.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.596+0000 7fdba4ff9700 1 -- 172.21.15.121:0/1587916979 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdb9c0577c0 con 0x7fdba8105a40 2024-04-05T21:41:13.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:13 smithi121 ceph-mon[145859]: pgmap v449: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:13.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:13 smithi188 ceph-mon[107793]: pgmap v449: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:13.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.798+0000 7fdbadf79700 1 -- 172.21.15.121:0/1587916979 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdba802cd30 con 0x7fdb9006cce0 2024-04-05T21:41:13.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.809+0000 7fdba4ff9700 1 -- 172.21.15.121:0/1587916979 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fdba802cd30 con 0x7fdb9006cce0 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (12m) 6m ago 23m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (23m) 6m ago 23m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (22m) 5m ago 22m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (10m) 6m ago 23m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (10m) 5m ago 22m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (11m) 6m ago 23m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (6m) 6m ago 17m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (6m) 6m ago 17m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (5m) 5m ago 17m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (17m) 5m ago 17m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (14m) 6m ago 25m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (13m) 5m ago 22m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:41:13.810 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (10m) 6m ago 25m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:41:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (10m) 5m ago 22m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:41:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (12m) 6m ago 23m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:41:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (12m) 5m ago 22m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:41:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (9m) 6m ago 21m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:41:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (9m) 6m ago 20m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:41:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (8m) 6m ago 20m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:41:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (7m) 5m ago 20m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:41:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (7m) 5m ago 19m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:41:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (6m) 5m ago 19m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:41:13.811 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (12m) 6m ago 22m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:41:13.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.811+0000 7fdbadf79700 1 -- 172.21.15.121:0/1587916979 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb9006cce0 msgr2=0x7fdb9006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:13.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.811+0000 7fdbadf79700 1 --2- 172.21.15.121:0/1587916979 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb9006cce0 0x7fdb9006f1a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdb98005950 tx=0x7fdb9800b410 comp rx=0 tx=0).stop 2024-04-05T21:41:13.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.811+0000 7fdbadf79700 1 -- 172.21.15.121:0/1587916979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdba8105a40 msgr2=0x7fdba81979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:13.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.811+0000 7fdbadf79700 1 --2- 172.21.15.121:0/1587916979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdba8105a40 0x7fdba81979f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c006010 tx=0x7fdb9c00bd70 comp rx=0 tx=0).stop 2024-04-05T21:41:13.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.811+0000 7fdbadf79700 1 -- 172.21.15.121:0/1587916979 shutdown_connections 2024-04-05T21:41:13.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.811+0000 7fdbadf79700 1 --2- 172.21.15.121:0/1587916979 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb9006cce0 0x7fdb9006f1a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.811+0000 7fdbadf79700 1 --2- 172.21.15.121:0/1587916979 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdba80ffa40 0x7fdba81974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.811+0000 7fdbadf79700 1 --2- 172.21.15.121:0/1587916979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdba8105a40 0x7fdba81979f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.811+0000 7fdbadf79700 1 -- 172.21.15.121:0/1587916979 >> 172.21.15.121:0/1587916979 conn(0x7fdba80fb570 msgr2=0x7fdba80fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:13.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.811+0000 7fdbadf79700 1 -- 172.21.15.121:0/1587916979 shutdown_connections 2024-04-05T21:41:13.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.811+0000 7fdbadf79700 1 -- 172.21.15.121:0/1587916979 wait complete. 2024-04-05T21:41:13.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.913+0000 7f79ef1dd700 1 -- 172.21.15.121:0/3775493614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79e80ffbd0 msgr2=0x7f79e8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:13.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.913+0000 7f79ef1dd700 1 --2- 172.21.15.121:0/3775493614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79e80ffbd0 0x7f79e8100050 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009910 tx=0x7f79d4009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:13.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.913+0000 7f79ef1dd700 1 -- 172.21.15.121:0/3775493614 shutdown_connections 2024-04-05T21:41:13.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.913+0000 7f79ef1dd700 1 --2- 172.21.15.121:0/3775493614 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79e80ffbd0 0x7f79e8100050 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.913+0000 7f79ef1dd700 1 --2- 172.21.15.121:0/3775493614 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79e8105a40 0x7f79e8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.913+0000 7f79ef1dd700 1 -- 172.21.15.121:0/3775493614 >> 172.21.15.121:0/3775493614 conn(0x7f79e80755a0 msgr2=0x7f79e80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:13.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.913+0000 7f79ef1dd700 1 -- 172.21.15.121:0/3775493614 shutdown_connections 2024-04-05T21:41:13.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.913+0000 7f79ef1dd700 1 -- 172.21.15.121:0/3775493614 wait complete. 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.913+0000 7f79ef1dd700 1 Processor -- start 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.913+0000 7f79ef1dd700 1 -- start start 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79ef1dd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79e80ffbd0 0x7f79e8101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79ef1dd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79e8105a40 0x7f79e8101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79ef1dd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8102090 con 0x7f79e80ffbd0 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79ef1dd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e81021d0 con 0x7f79e8105a40 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79ecf79700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79e80ffbd0 0x7f79e8101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79ecf79700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79e80ffbd0 0x7f79e8101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52784/0 (socket says 172.21.15.121:52784) 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79ecf79700 1 -- 172.21.15.121:0/301828328 learned_addr learned my addr 172.21.15.121:0/301828328 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79e7fff700 1 --2- 172.21.15.121:0/301828328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79e8105a40 0x7f79e8101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79ecf79700 1 -- 172.21.15.121:0/301828328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79e8105a40 msgr2=0x7f79e8101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79ecf79700 1 --2- 172.21.15.121:0/301828328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79e8105a40 0x7f79e8101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79ecf79700 1 -- 172.21.15.121:0/301828328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d40095f0 con 0x7f79e80ffbd0 2024-04-05T21:41:13.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79ecf79700 1 --2- 172.21.15.121:0/301828328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79e80ffbd0 0x7f79e8101610 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f79dc00c8a0 tx=0x7f79dc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79e5ffb700 1 -- 172.21.15.121:0/301828328 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79dc00cea0 con 0x7f79e80ffbd0 2024-04-05T21:41:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79e5ffb700 1 -- 172.21.15.121:0/301828328 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79dc004830 con 0x7f79e80ffbd0 2024-04-05T21:41:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79ef1dd700 1 -- 172.21.15.121:0/301828328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e81a2910 con 0x7f79e80ffbd0 2024-04-05T21:41:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79e5ffb700 1 -- 172.21.15.121:0/301828328 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79dc005500 con 0x7f79e80ffbd0 2024-04-05T21:41:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.914+0000 7f79ef1dd700 1 -- 172.21.15.121:0/301828328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e81a2d20 con 0x7f79e80ffbd0 2024-04-05T21:41:13.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.915+0000 7f79ef1dd700 1 -- 172.21.15.121:0/301828328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e8108fe0 con 0x7f79e80ffbd0 2024-04-05T21:41:13.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.916+0000 7f79e5ffb700 1 -- 172.21.15.121:0/301828328 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f79dc0049a0 con 0x7f79e80ffbd0 2024-04-05T21:41:13.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.916+0000 7f79e5ffb700 1 --2- 172.21.15.121:0/301828328 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f79d806cce0 0x7f79d806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:13.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.916+0000 7f79e5ffb700 1 -- 172.21.15.121:0/301828328 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f79dc08c250 con 0x7f79e80ffbd0 2024-04-05T21:41:13.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.916+0000 7f79e7fff700 1 --2- 172.21.15.121:0/301828328 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f79d806cce0 0x7f79d806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:13.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.916+0000 7f79e7fff700 1 --2- 172.21.15.121:0/301828328 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f79d806cce0 0x7f79d806f1a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f79d400b5c0 tx=0x7f79d4005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:13.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:13.921+0000 7f79e5ffb700 1 -- 172.21.15.121:0/301828328 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f79dc0562c0 con 0x7f79e80ffbd0 2024-04-05T21:41:14.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.193+0000 7f79ef1dd700 1 -- 172.21.15.121:0/301828328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f79e81089c0 con 0x7f79e80ffbd0 2024-04-05T21:41:14.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.193+0000 7f79e5ffb700 1 -- 172.21.15.121:0/301828328 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f79dc0598e0 con 0x7f79e80ffbd0 2024-04-05T21:41:14.194 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:41:14.194 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:41:14.194 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:41:14.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:41:14.194 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:41:14.194 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:41:14.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:41:14.195 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:41:14.195 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:41:14.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:41:14.195 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:41:14.195 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:41:14.195 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:41:14.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:41:14.195 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:41:14.195 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:41:14.195 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:41:14.195 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:41:14.195 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:41:14.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.195+0000 7f79ef1dd700 1 -- 172.21.15.121:0/301828328 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f79d806cce0 msgr2=0x7f79d806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:14.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.195+0000 7f79ef1dd700 1 --2- 172.21.15.121:0/301828328 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f79d806cce0 0x7f79d806f1a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f79d400b5c0 tx=0x7f79d4005fd0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.195+0000 7f79ef1dd700 1 -- 172.21.15.121:0/301828328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79e80ffbd0 msgr2=0x7f79e8101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:14.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.195+0000 7f79ef1dd700 1 --2- 172.21.15.121:0/301828328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79e80ffbd0 0x7f79e8101610 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f79dc00c8a0 tx=0x7f79dc00cc60 comp rx=0 tx=0).stop 2024-04-05T21:41:14.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.195+0000 7f79ef1dd700 1 -- 172.21.15.121:0/301828328 shutdown_connections 2024-04-05T21:41:14.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.195+0000 7f79ef1dd700 1 --2- 172.21.15.121:0/301828328 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f79d806cce0 0x7f79d806f1a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.195+0000 7f79ef1dd700 1 --2- 172.21.15.121:0/301828328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79e80ffbd0 0x7f79e8101610 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.195+0000 7f79ef1dd700 1 --2- 172.21.15.121:0/301828328 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79e8105a40 0x7f79e8101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.195+0000 7f79ef1dd700 1 -- 172.21.15.121:0/301828328 >> 172.21.15.121:0/301828328 conn(0x7f79e80755a0 msgr2=0x7f79e8103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:14.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.195+0000 7f79ef1dd700 1 -- 172.21.15.121:0/301828328 shutdown_connections 2024-04-05T21:41:14.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.195+0000 7f79ef1dd700 1 -- 172.21.15.121:0/301828328 wait complete. 2024-04-05T21:41:14.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.299+0000 7f6bddbad700 1 -- 172.21.15.121:0/61498557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd8105a40 msgr2=0x7f6bd8105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:14.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.299+0000 7f6bddbad700 1 --2- 172.21.15.121:0/61498557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd8105a40 0x7f6bd8105e20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009910 tx=0x7f6bc8009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:14.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.299+0000 7f6bddbad700 1 -- 172.21.15.121:0/61498557 shutdown_connections 2024-04-05T21:41:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.299+0000 7f6bddbad700 1 --2- 172.21.15.121:0/61498557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd80ffbd0 0x7f6bd8100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.299+0000 7f6bddbad700 1 --2- 172.21.15.121:0/61498557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd8105a40 0x7f6bd8105e20 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.299+0000 7f6bddbad700 1 -- 172.21.15.121:0/61498557 >> 172.21.15.121:0/61498557 conn(0x7f6bd80755a0 msgr2=0x7f6bd80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.299+0000 7f6bddbad700 1 -- 172.21.15.121:0/61498557 shutdown_connections 2024-04-05T21:41:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.299+0000 7f6bddbad700 1 -- 172.21.15.121:0/61498557 wait complete. 2024-04-05T21:41:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bddbad700 1 Processor -- start 2024-04-05T21:41:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bddbad700 1 -- start start 2024-04-05T21:41:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bddbad700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd80ffbd0 0x7f6bd8196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bddbad700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd8105a40 0x7f6bd8196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bddbad700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd8197290 con 0x7f6bd8105a40 2024-04-05T21:41:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bddbad700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd819b020 con 0x7f6bd80ffbd0 2024-04-05T21:41:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bcffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd8105a40 0x7f6bd8196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:14.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bcffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd8105a40 0x7f6bd8196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52808/0 (socket says 172.21.15.121:52808) 2024-04-05T21:41:14.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bcffff700 1 -- 172.21.15.121:0/3002268280 learned_addr learned my addr 172.21.15.121:0/3002268280 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:14.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bd77fe700 1 --2- 172.21.15.121:0/3002268280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd80ffbd0 0x7f6bd8196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:14.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bcffff700 1 -- 172.21.15.121:0/3002268280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd80ffbd0 msgr2=0x7f6bd8196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:14.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bcffff700 1 --2- 172.21.15.121:0/3002268280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd80ffbd0 0x7f6bd8196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bcffff700 1 -- 172.21.15.121:0/3002268280 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc80095f0 con 0x7f6bd8105a40 2024-04-05T21:41:14.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bd77fe700 1 --2- 172.21.15.121:0/3002268280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd80ffbd0 0x7f6bd8196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:41:14.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.300+0000 7f6bcffff700 1 --2- 172.21.15.121:0/3002268280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd8105a40 0x7f6bd8196bb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6bc000e920 tx=0x7f6bc000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:14.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.301+0000 7f6bd57fa700 1 -- 172.21.15.121:0/3002268280 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bc0009790 con 0x7f6bd8105a40 2024-04-05T21:41:14.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.301+0000 7f6bd57fa700 1 -- 172.21.15.121:0/3002268280 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bc000cba0 con 0x7f6bd8105a40 2024-04-05T21:41:14.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.301+0000 7f6bd57fa700 1 -- 172.21.15.121:0/3002268280 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bc0018650 con 0x7f6bd8105a40 2024-04-05T21:41:14.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.301+0000 7f6bddbad700 1 -- 172.21.15.121:0/3002268280 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd819b300 con 0x7f6bd8105a40 2024-04-05T21:41:14.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.301+0000 7f6bddbad700 1 -- 172.21.15.121:0/3002268280 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd819b850 con 0x7f6bd8105a40 2024-04-05T21:41:14.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.302+0000 7f6bddbad700 1 -- 172.21.15.121:0/3002268280 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd804ea90 con 0x7f6bd8105a40 2024-04-05T21:41:14.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.302+0000 7f6bd57fa700 1 -- 172.21.15.121:0/3002268280 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6bc000cd10 con 0x7f6bd8105a40 2024-04-05T21:41:14.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.303+0000 7f6bd57fa700 1 --2- 172.21.15.121:0/3002268280 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6bc406cce0 0x7f6bc406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:14.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.303+0000 7f6bd57fa700 1 -- 172.21.15.121:0/3002268280 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6bc0014070 con 0x7f6bd8105a40 2024-04-05T21:41:14.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.303+0000 7f6bd77fe700 1 --2- 172.21.15.121:0/3002268280 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6bc406cce0 0x7f6bc406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:14.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.303+0000 7f6bd77fe700 1 --2- 172.21.15.121:0/3002268280 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6bc406cce0 0x7f6bc406f1a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8000c00 tx=0x7f6bc8009f90 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:14.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.307+0000 7f6bd57fa700 1 -- 172.21.15.121:0/3002268280 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6bc0052ab0 con 0x7f6bd8105a40 2024-04-05T21:41:14.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.546+0000 7f6bddbad700 1 -- 172.21.15.121:0/3002268280 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6bd819bc40 con 0x7f6bd8105a40 2024-04-05T21:41:14.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.546+0000 7f6bd57fa700 1 -- 172.21.15.121:0/3002268280 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f6bc005a3f0 con 0x7f6bd8105a40 2024-04-05T21:41:14.547 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:41:14.547 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:41:14.547 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:41:14.547 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:41:14.547 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:41:14.547 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:41:14.547 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:41:14.547 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:41:14.547 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:41:14.548 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:41:14.549 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:41:14.550 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:41:14.550 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:41:14.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.548+0000 7f6bddbad700 1 -- 172.21.15.121:0/3002268280 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6bc406cce0 msgr2=0x7f6bc406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:14.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.548+0000 7f6bddbad700 1 --2- 172.21.15.121:0/3002268280 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6bc406cce0 0x7f6bc406f1a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8000c00 tx=0x7f6bc8009f90 comp rx=0 tx=0).stop 2024-04-05T21:41:14.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.548+0000 7f6bddbad700 1 -- 172.21.15.121:0/3002268280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd8105a40 msgr2=0x7f6bd8196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:14.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.548+0000 7f6bddbad700 1 --2- 172.21.15.121:0/3002268280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd8105a40 0x7f6bd8196bb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6bc000e920 tx=0x7f6bc000ece0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.548+0000 7f6bddbad700 1 -- 172.21.15.121:0/3002268280 shutdown_connections 2024-04-05T21:41:14.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.548+0000 7f6bddbad700 1 --2- 172.21.15.121:0/3002268280 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6bc406cce0 0x7f6bc406f1a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.548+0000 7f6bddbad700 1 --2- 172.21.15.121:0/3002268280 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd80ffbd0 0x7f6bd8196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.548+0000 7f6bddbad700 1 --2- 172.21.15.121:0/3002268280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd8105a40 0x7f6bd8196bb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.548+0000 7f6bddbad700 1 -- 172.21.15.121:0/3002268280 >> 172.21.15.121:0/3002268280 conn(0x7f6bd80755a0 msgr2=0x7f6bd81037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:14.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.548+0000 7f6bddbad700 1 -- 172.21.15.121:0/3002268280 shutdown_connections 2024-04-05T21:41:14.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.548+0000 7f6bddbad700 1 -- 172.21.15.121:0/3002268280 wait complete. 2024-04-05T21:41:14.551 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:41:14.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.653+0000 7f6986854700 1 -- 172.21.15.121:0/1519714248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6980105a40 msgr2=0x7f6980105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:14.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.653+0000 7f6986854700 1 --2- 172.21.15.121:0/1519714248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6980105a40 0x7f6980105e20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6974009910 tx=0x7f6974009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:14.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.653+0000 7f6986854700 1 -- 172.21.15.121:0/1519714248 shutdown_connections 2024-04-05T21:41:14.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.653+0000 7f6986854700 1 --2- 172.21.15.121:0/1519714248 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69800ffa40 0x7f69800ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.653+0000 7f6986854700 1 --2- 172.21.15.121:0/1519714248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6980105a40 0x7f6980105e20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.653+0000 7f6986854700 1 -- 172.21.15.121:0/1519714248 >> 172.21.15.121:0/1519714248 conn(0x7f69800fb570 msgr2=0x7f69800fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:14.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.653+0000 7f6986854700 1 -- 172.21.15.121:0/1519714248 shutdown_connections 2024-04-05T21:41:14.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.653+0000 7f6986854700 1 -- 172.21.15.121:0/1519714248 wait complete. 2024-04-05T21:41:14.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.653+0000 7f6986854700 1 Processor -- start 2024-04-05T21:41:14.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.653+0000 7f6986854700 1 -- start start 2024-04-05T21:41:14.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f6986854700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69800ffa40 0x7f6980195660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:14.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f6986854700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6980105a40 0x7f6980195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:14.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f6986854700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6980196280 con 0x7f6980105a40 2024-04-05T21:41:14.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f6986854700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698019a010 con 0x7f69800ffa40 2024-04-05T21:41:14.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f697f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6980105a40 0x7f6980195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:14.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f697f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6980105a40 0x7f6980195ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35262/0 (socket says 172.21.15.121:35262) 2024-04-05T21:41:14.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f697f7fe700 1 -- 172.21.15.121:0/534331110 learned_addr learned my addr 172.21.15.121:0/534331110 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:14.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f697ffff700 1 --2- 172.21.15.121:0/534331110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69800ffa40 0x7f6980195660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:14.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f697f7fe700 1 -- 172.21.15.121:0/534331110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69800ffa40 msgr2=0x7f6980195660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:14.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f697f7fe700 1 --2- 172.21.15.121:0/534331110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69800ffa40 0x7f6980195660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f697f7fe700 1 -- 172.21.15.121:0/534331110 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69740095f0 con 0x7f6980105a40 2024-04-05T21:41:14.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f697f7fe700 1 --2- 172.21.15.121:0/534331110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6980105a40 0x7f6980195ba0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f697000b4b0 tx=0x7f697000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:14.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f697d7fa700 1 -- 172.21.15.121:0/534331110 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6970011800 con 0x7f6980105a40 2024-04-05T21:41:14.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f697d7fa700 1 -- 172.21.15.121:0/534331110 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6970011e00 con 0x7f6980105a40 2024-04-05T21:41:14.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f697d7fa700 1 -- 172.21.15.121:0/534331110 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6970010e60 con 0x7f6980105a40 2024-04-05T21:41:14.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f6986854700 1 -- 172.21.15.121:0/534331110 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698019a2f0 con 0x7f6980105a40 2024-04-05T21:41:14.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.654+0000 7f6986854700 1 -- 172.21.15.121:0/534331110 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698019a840 con 0x7f6980105a40 2024-04-05T21:41:14.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.655+0000 7f6986854700 1 -- 172.21.15.121:0/534331110 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6980108040 con 0x7f6980105a40 2024-04-05T21:41:14.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.656+0000 7f697d7fa700 1 -- 172.21.15.121:0/534331110 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6970011960 con 0x7f6980105a40 2024-04-05T21:41:14.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.656+0000 7f697d7fa700 1 --2- 172.21.15.121:0/534331110 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f696806cce0 0x7f696806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:14.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.656+0000 7f697ffff700 1 --2- 172.21.15.121:0/534331110 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f696806cce0 0x7f696806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:14.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.656+0000 7f697d7fa700 1 -- 172.21.15.121:0/534331110 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f697008c0d0 con 0x7f6980105a40 2024-04-05T21:41:14.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.657+0000 7f697ffff700 1 --2- 172.21.15.121:0/534331110 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f696806cce0 0x7f696806f1a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6974006010 tx=0x7f697401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:14.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.661+0000 7f697d7fa700 1 -- 172.21.15.121:0/534331110 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6970056140 con 0x7f6980105a40 2024-04-05T21:41:14.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:14 smithi121 ceph-mon[145859]: from='client.34678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:14.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:14 smithi121 ceph-mon[145859]: from='client.34682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:14.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:14 smithi121 ceph-mon[145859]: from='client.34686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:14.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:14 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/301828328' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:41:14.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:14 smithi188 ceph-mon[107793]: from='client.34678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:14.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:14 smithi188 ceph-mon[107793]: from='client.34682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:14.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:14 smithi188 ceph-mon[107793]: from='client.34686 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:14.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:14 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/301828328' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:41:14.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.879+0000 7f6986854700 1 -- 172.21.15.121:0/534331110 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f698019ab80 con 0x7f696806cce0 2024-04-05T21:41:14.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.880+0000 7f697d7fa700 1 -- 172.21.15.121:0/534331110 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f698019ab80 con 0x7f696806cce0 2024-04-05T21:41:14.881 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:41:14.881 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:41:14.881 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:41:14.881 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:41:14.881 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:41:14.881 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:41:14.881 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:41:14.881 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:41:14.881 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:41:14.882 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:41:14.882 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:41:14.882 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:41:14.882 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:41:14.882 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:41:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.882+0000 7f6986854700 1 -- 172.21.15.121:0/534331110 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f696806cce0 msgr2=0x7f696806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.882+0000 7f6986854700 1 --2- 172.21.15.121:0/534331110 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f696806cce0 0x7f696806f1a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6974006010 tx=0x7f697401a040 comp rx=0 tx=0).stop 2024-04-05T21:41:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.882+0000 7f6986854700 1 -- 172.21.15.121:0/534331110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6980105a40 msgr2=0x7f6980195ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.882+0000 7f6986854700 1 --2- 172.21.15.121:0/534331110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6980105a40 0x7f6980195ba0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f697000b4b0 tx=0x7f697000b7c0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.882+0000 7f6986854700 1 -- 172.21.15.121:0/534331110 shutdown_connections 2024-04-05T21:41:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.882+0000 7f6986854700 1 --2- 172.21.15.121:0/534331110 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f696806cce0 0x7f696806f1a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.882+0000 7f6986854700 1 --2- 172.21.15.121:0/534331110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f69800ffa40 0x7f6980195660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.882+0000 7f6986854700 1 --2- 172.21.15.121:0/534331110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6980105a40 0x7f6980195ba0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.882+0000 7f6986854700 1 -- 172.21.15.121:0/534331110 >> 172.21.15.121:0/534331110 conn(0x7f69800fb570 msgr2=0x7f69800fc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:14.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.882+0000 7f6986854700 1 -- 172.21.15.121:0/534331110 shutdown_connections 2024-04-05T21:41:14.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.882+0000 7f6986854700 1 -- 172.21.15.121:0/534331110 wait complete. 2024-04-05T21:41:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.984+0000 7f93ded7b700 1 -- 172.21.15.121:0/1333887639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93d8103990 msgr2=0x7f93d81079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.984+0000 7f93ded7b700 1 --2- 172.21.15.121:0/1333887639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93d8103990 0x7f93d81079e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f93c4009910 tx=0x7f93c4009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.984+0000 7f93ded7b700 1 -- 172.21.15.121:0/1333887639 shutdown_connections 2024-04-05T21:41:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.984+0000 7f93ded7b700 1 --2- 172.21.15.121:0/1333887639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93d8103990 0x7f93d81079e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.984+0000 7f93ded7b700 1 --2- 172.21.15.121:0/1333887639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93d8102fe0 0x7f93d81033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.984+0000 7f93ded7b700 1 -- 172.21.15.121:0/1333887639 >> 172.21.15.121:0/1333887639 conn(0x7f93d80fe870 msgr2=0x7f93d8100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.985+0000 7f93ded7b700 1 -- 172.21.15.121:0/1333887639 shutdown_connections 2024-04-05T21:41:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.985+0000 7f93ded7b700 1 -- 172.21.15.121:0/1333887639 wait complete. 2024-04-05T21:41:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.985+0000 7f93ded7b700 1 Processor -- start 2024-04-05T21:41:14.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.985+0000 7f93ded7b700 1 -- start start 2024-04-05T21:41:14.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.985+0000 7f93ded7b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93d8102fe0 0x7f93d819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:14.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.985+0000 7f93ded7b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93d8103990 0x7f93d819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:14.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.985+0000 7f93ded7b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d819f2b0 con 0x7f93d8103990 2024-04-05T21:41:14.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.985+0000 7f93ded7b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d8198760 con 0x7f93d8102fe0 2024-04-05T21:41:14.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.985+0000 7f93d7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93d8103990 0x7f93d819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:14.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.985+0000 7f93d7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93d8103990 0x7f93d819ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35272/0 (socket says 172.21.15.121:35272) 2024-04-05T21:41:14.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.985+0000 7f93d7fff700 1 -- 172.21.15.121:0/1513836894 learned_addr learned my addr 172.21.15.121:0/1513836894 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.986+0000 7f93d7fff700 1 -- 172.21.15.121:0/1513836894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93d8102fe0 msgr2=0x7f93d819e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.986+0000 7f93d7fff700 1 --2- 172.21.15.121:0/1513836894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93d8102fe0 0x7f93d819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.986+0000 7f93d7fff700 1 -- 172.21.15.121:0/1513836894 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c40095f0 con 0x7f93d8103990 2024-04-05T21:41:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.986+0000 7f93d7fff700 1 --2- 172.21.15.121:0/1513836894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93d8103990 0x7f93d819ec20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f93c40092e0 tx=0x7f93c400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.986+0000 7f93d5ffb700 1 -- 172.21.15.121:0/1513836894 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c401d070 con 0x7f93d8103990 2024-04-05T21:41:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.986+0000 7f93d5ffb700 1 -- 172.21.15.121:0/1513836894 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93c400f460 con 0x7f93d8103990 2024-04-05T21:41:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.986+0000 7f93ded7b700 1 -- 172.21.15.121:0/1513836894 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d81989e0 con 0x7f93d8103990 2024-04-05T21:41:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.986+0000 7f93d5ffb700 1 -- 172.21.15.121:0/1513836894 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c4021600 con 0x7f93d8103990 2024-04-05T21:41:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.986+0000 7f93ded7b700 1 -- 172.21.15.121:0/1513836894 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d8198ed0 con 0x7f93d8103990 2024-04-05T21:41:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.987+0000 7f93ded7b700 1 -- 172.21.15.121:0/1513836894 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d810b3e0 con 0x7f93d8103990 2024-04-05T21:41:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.988+0000 7f93d5ffb700 1 -- 172.21.15.121:0/1513836894 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f93c400f5d0 con 0x7f93d8103990 2024-04-05T21:41:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.988+0000 7f93d5ffb700 1 --2- 172.21.15.121:0/1513836894 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f93c806cce0 0x7f93c806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.988+0000 7f93d5ffb700 1 -- 172.21.15.121:0/1513836894 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f93c408d7b0 con 0x7f93d8103990 2024-04-05T21:41:14.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.988+0000 7f93dcb17700 1 --2- 172.21.15.121:0/1513836894 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f93c806cce0 0x7f93c806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:14.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.988+0000 7f93dcb17700 1 --2- 172.21.15.121:0/1513836894 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f93c806cce0 0x7f93c806f1a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00a990 tx=0x7f93cc005c80 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:14.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:14.992+0000 7f93d5ffb700 1 -- 172.21.15.121:0/1513836894 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f93c405b2b0 con 0x7f93d8103990 2024-04-05T21:41:15.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:15.268+0000 7f93ded7b700 1 -- 172.21.15.121:0/1513836894 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f93d802cc70 con 0x7f93d8103990 2024-04-05T21:41:15.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:15.268+0000 7f93d5ffb700 1 -- 172.21.15.121:0/1513836894 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f93c405ae40 con 0x7f93d8103990 2024-04-05T21:41:15.269 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:41:15.269 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:41:15.269 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:41:15.269 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:41:15.269 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:41:15.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:15.270+0000 7f93ded7b700 1 -- 172.21.15.121:0/1513836894 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f93c806cce0 msgr2=0x7f93c806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:15.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:15.270+0000 7f93ded7b700 1 --2- 172.21.15.121:0/1513836894 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f93c806cce0 0x7f93c806f1a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f93cc00a990 tx=0x7f93cc005c80 comp rx=0 tx=0).stop 2024-04-05T21:41:15.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:15.270+0000 7f93ded7b700 1 -- 172.21.15.121:0/1513836894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93d8103990 msgr2=0x7f93d819ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:15.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:15.270+0000 7f93ded7b700 1 --2- 172.21.15.121:0/1513836894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93d8103990 0x7f93d819ec20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f93c40092e0 tx=0x7f93c400be20 comp rx=0 tx=0).stop 2024-04-05T21:41:15.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:15.270+0000 7f93ded7b700 1 -- 172.21.15.121:0/1513836894 shutdown_connections 2024-04-05T21:41:15.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:15.270+0000 7f93ded7b700 1 --2- 172.21.15.121:0/1513836894 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f93c806cce0 0x7f93c806f1a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:15.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:15.270+0000 7f93ded7b700 1 --2- 172.21.15.121:0/1513836894 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93d8102fe0 0x7f93d819e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:15.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:15.270+0000 7f93ded7b700 1 --2- 172.21.15.121:0/1513836894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93d8103990 0x7f93d819ec20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:15.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:15.270+0000 7f93ded7b700 1 -- 172.21.15.121:0/1513836894 >> 172.21.15.121:0/1513836894 conn(0x7f93d80fe870 msgr2=0x7f93d80ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:15.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:15.270+0000 7f93ded7b700 1 -- 172.21.15.121:0/1513836894 shutdown_connections 2024-04-05T21:41:15.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:15.270+0000 7f93ded7b700 1 -- 172.21.15.121:0/1513836894 wait complete. 2024-04-05T21:41:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:15 smithi121 ceph-mon[145859]: pgmap v450: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:15 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3002268280' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:41:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:41:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:15 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1513836894' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:41:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:15 smithi188 ceph-mon[107793]: pgmap v450: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:15 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3002268280' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:41:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:41:15.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:15 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1513836894' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:41:16.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:16 smithi121 ceph-mon[145859]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:16.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:16 smithi188 ceph-mon[107793]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:17 smithi121 ceph-mon[145859]: pgmap v451: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:17.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:17 smithi188 ceph-mon[107793]: pgmap v451: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:19 smithi121 ceph-mon[145859]: pgmap v452: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:19 smithi188 ceph-mon[107793]: pgmap v452: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:21 smithi121 ceph-mon[145859]: pgmap v453: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:21 smithi188 ceph-mon[107793]: pgmap v453: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:23.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:23 smithi188 ceph-mon[107793]: pgmap v454: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:23 smithi121 ceph-mon[145859]: pgmap v454: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:25.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:25 smithi188 ceph-mon[107793]: pgmap v455: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:25.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:25 smithi121 ceph-mon[145859]: pgmap v455: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:27.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:27 smithi188 ceph-mon[107793]: pgmap v456: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:27 smithi121 ceph-mon[145859]: pgmap v456: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:29 smithi188 ceph-mon[107793]: pgmap v457: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:29.882 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:29 smithi121 ceph-mon[145859]: pgmap v457: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:30.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:41:30.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:41:31.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:31 smithi188 ceph-mon[107793]: pgmap v458: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:31.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:31 smithi121 ceph-mon[145859]: pgmap v458: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:33 smithi188 ceph-mon[107793]: pgmap v459: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:33.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:33 smithi121 ceph-mon[145859]: pgmap v459: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:35.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:35 smithi188 ceph-mon[107793]: pgmap v460: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:35.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:35 smithi121 ceph-mon[145859]: pgmap v460: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:37.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:37 smithi188 ceph-mon[107793]: pgmap v461: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:37.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:37 smithi121 ceph-mon[145859]: pgmap v461: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:39.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:39 smithi188 ceph-mon[107793]: pgmap v462: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:39.976 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:39 smithi121 ceph-mon[145859]: pgmap v462: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:41.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:41 smithi188 ceph-mon[107793]: pgmap v463: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:41.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:41 smithi121 ceph-mon[145859]: pgmap v463: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:43.443 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:43 smithi188 ceph-mon[107793]: pgmap v464: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:43.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:43 smithi121 ceph-mon[145859]: pgmap v464: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:45.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.382+0000 7f8af77ea700 1 -- 172.21.15.121:0/3959560781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8af0102670 msgr2=0x7f8af010ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:45.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.382+0000 7f8af77ea700 1 --2- 172.21.15.121:0/3959560781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8af0102670 0x7f8af010ab60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8aec009910 tx=0x7f8aec009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:45.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.382+0000 7f8af77ea700 1 -- 172.21.15.121:0/3959560781 shutdown_connections 2024-04-05T21:41:45.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.382+0000 7f8af77ea700 1 --2- 172.21.15.121:0/3959560781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8af0102670 0x7f8af010ab60 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:45.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.382+0000 7f8af77ea700 1 --2- 172.21.15.121:0/3959560781 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8af0101d50 0x7f8af0102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:45.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.382+0000 7f8af77ea700 1 -- 172.21.15.121:0/3959560781 >> 172.21.15.121:0/3959560781 conn(0x7f8af00fb540 msgr2=0x7f8af00fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:45.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.382+0000 7f8af77ea700 1 -- 172.21.15.121:0/3959560781 shutdown_connections 2024-04-05T21:41:45.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.383+0000 7f8af77ea700 1 -- 172.21.15.121:0/3959560781 wait complete. 2024-04-05T21:41:45.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.383+0000 7f8af77ea700 1 Processor -- start 2024-04-05T21:41:45.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.383+0000 7f8af77ea700 1 -- start start 2024-04-05T21:41:45.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.383+0000 7f8af77ea700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8af0101d50 0x7f8af0196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:45.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.383+0000 7f8af77ea700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8af0102670 0x7f8af0196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:45.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.383+0000 7f8af77ea700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af0197280 con 0x7f8af0101d50 2024-04-05T21:41:45.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.383+0000 7f8af77ea700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af019b010 con 0x7f8af0102670 2024-04-05T21:41:45.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.383+0000 7f8af5586700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8af0101d50 0x7f8af0196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:45.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.383+0000 7f8af5586700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8af0101d50 0x7f8af0196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51218/0 (socket says 172.21.15.121:51218) 2024-04-05T21:41:45.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.384+0000 7f8af5586700 1 -- 172.21.15.121:0/3556020572 learned_addr learned my addr 172.21.15.121:0/3556020572 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:45.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.384+0000 7f8af4d85700 1 --2- 172.21.15.121:0/3556020572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8af0102670 0x7f8af0196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:45.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.384+0000 7f8af5586700 1 -- 172.21.15.121:0/3556020572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8af0102670 msgr2=0x7f8af0196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:45.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.384+0000 7f8af5586700 1 --2- 172.21.15.121:0/3556020572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8af0102670 0x7f8af0196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:45.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.384+0000 7f8af5586700 1 -- 172.21.15.121:0/3556020572 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec0095f0 con 0x7f8af0101d50 2024-04-05T21:41:45.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.384+0000 7f8af5586700 1 --2- 172.21.15.121:0/3556020572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8af0101d50 0x7f8af0196660 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400d8d0 tx=0x7f8ae400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:45.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.384+0000 7f8ae27fc700 1 -- 172.21.15.121:0/3556020572 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ae4009940 con 0x7f8af0101d50 2024-04-05T21:41:45.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.384+0000 7f8af77ea700 1 -- 172.21.15.121:0/3556020572 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af019b2f0 con 0x7f8af0101d50 2024-04-05T21:41:45.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.384+0000 7f8ae27fc700 1 -- 172.21.15.121:0/3556020572 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ae4010460 con 0x7f8af0101d50 2024-04-05T21:41:45.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.384+0000 7f8ae27fc700 1 -- 172.21.15.121:0/3556020572 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ae400f5f0 con 0x7f8af0101d50 2024-04-05T21:41:45.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.384+0000 7f8af77ea700 1 -- 172.21.15.121:0/3556020572 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af019b840 con 0x7f8af0101d50 2024-04-05T21:41:45.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.385+0000 7f8af77ea700 1 -- 172.21.15.121:0/3556020572 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af0108260 con 0x7f8af0101d50 2024-04-05T21:41:45.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.385+0000 7f8ae27fc700 1 -- 172.21.15.121:0/3556020572 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8ae40105d0 con 0x7f8af0101d50 2024-04-05T21:41:45.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.386+0000 7f8ae27fc700 1 --2- 172.21.15.121:0/3556020572 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8adc06cce0 0x7f8adc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:45.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.386+0000 7f8af4d85700 1 --2- 172.21.15.121:0/3556020572 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8adc06cce0 0x7f8adc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:45.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.386+0000 7f8ae27fc700 1 -- 172.21.15.121:0/3556020572 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8ae408c5e0 con 0x7f8af0101d50 2024-04-05T21:41:45.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.386+0000 7f8af4d85700 1 --2- 172.21.15.121:0/3556020572 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8adc06cce0 0x7f8adc06f1a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8af0197c80 tx=0x7f8aec01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:45.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.391+0000 7f8ae27fc700 1 -- 172.21.15.121:0/3556020572 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8ae4056650 con 0x7f8af0101d50 2024-04-05T21:41:45.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.599+0000 7f8af77ea700 1 -- 172.21.15.121:0/3556020572 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8af019bb70 con 0x7f8adc06cce0 2024-04-05T21:41:45.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.600+0000 7f8ae27fc700 1 -- 172.21.15.121:0/3556020572 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8af019bb70 con 0x7f8adc06cce0 2024-04-05T21:41:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.602+0000 7f8af77ea700 1 -- 172.21.15.121:0/3556020572 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8adc06cce0 msgr2=0x7f8adc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.602+0000 7f8af77ea700 1 --2- 172.21.15.121:0/3556020572 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8adc06cce0 0x7f8adc06f1a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8af0197c80 tx=0x7f8aec01a040 comp rx=0 tx=0).stop 2024-04-05T21:41:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.602+0000 7f8af77ea700 1 -- 172.21.15.121:0/3556020572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8af0101d50 msgr2=0x7f8af0196660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.602+0000 7f8af77ea700 1 --2- 172.21.15.121:0/3556020572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8af0101d50 0x7f8af0196660 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400d8d0 tx=0x7f8ae400dc90 comp rx=0 tx=0).stop 2024-04-05T21:41:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.602+0000 7f8af77ea700 1 -- 172.21.15.121:0/3556020572 shutdown_connections 2024-04-05T21:41:45.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.602+0000 7f8af77ea700 1 --2- 172.21.15.121:0/3556020572 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8adc06cce0 0x7f8adc06f1a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:45.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.602+0000 7f8af77ea700 1 --2- 172.21.15.121:0/3556020572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8af0101d50 0x7f8af0196660 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:45.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.602+0000 7f8af77ea700 1 --2- 172.21.15.121:0/3556020572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8af0102670 0x7f8af0196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:45.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.602+0000 7f8af77ea700 1 -- 172.21.15.121:0/3556020572 >> 172.21.15.121:0/3556020572 conn(0x7f8af00fb540 msgr2=0x7f8af01053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:45.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.602+0000 7f8af77ea700 1 -- 172.21.15.121:0/3556020572 shutdown_connections 2024-04-05T21:41:45.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.602+0000 7f8af77ea700 1 -- 172.21.15.121:0/3556020572 wait complete. 2024-04-05T21:41:45.613 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:41:45.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.705+0000 7f2d626d7700 1 -- 172.21.15.121:0/3691333525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d5c101770 msgr2=0x7f2d5c1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:45.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.705+0000 7f2d626d7700 1 --2- 172.21.15.121:0/3691333525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d5c101770 0x7f2d5c1057c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2d44009910 tx=0x7f2d44009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:45.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.705+0000 7f2d626d7700 1 -- 172.21.15.121:0/3691333525 shutdown_connections 2024-04-05T21:41:45.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.705+0000 7f2d626d7700 1 --2- 172.21.15.121:0/3691333525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d5c101770 0x7f2d5c1057c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:45.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.705+0000 7f2d626d7700 1 --2- 172.21.15.121:0/3691333525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d5c100dc0 0x7f2d5c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:45.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.705+0000 7f2d626d7700 1 -- 172.21.15.121:0/3691333525 >> 172.21.15.121:0/3691333525 conn(0x7f2d5c0fc650 msgr2=0x7f2d5c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:45.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.705+0000 7f2d626d7700 1 -- 172.21.15.121:0/3691333525 shutdown_connections 2024-04-05T21:41:45.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.705+0000 7f2d626d7700 1 -- 172.21.15.121:0/3691333525 wait complete. 2024-04-05T21:41:45.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.706+0000 7f2d626d7700 1 Processor -- start 2024-04-05T21:41:45.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.706+0000 7f2d626d7700 1 -- start start 2024-04-05T21:41:45.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.706+0000 7f2d626d7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d5c100dc0 0x7f2d5c1966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:45.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.706+0000 7f2d626d7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d5c101770 0x7f2d5c196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:45.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.706+0000 7f2d626d7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c1972a0 con 0x7f2d5c100dc0 2024-04-05T21:41:45.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.706+0000 7f2d626d7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c199fd0 con 0x7f2d5c101770 2024-04-05T21:41:45.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.706+0000 7f2d5bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d5c100dc0 0x7f2d5c1966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:45.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.706+0000 7f2d5bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d5c100dc0 0x7f2d5c1966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51240/0 (socket says 172.21.15.121:51240) 2024-04-05T21:41:45.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.706+0000 7f2d5bfff700 1 -- 172.21.15.121:0/997573708 learned_addr learned my addr 172.21.15.121:0/997573708 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:45.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.706+0000 7f2d5b7fe700 1 --2- 172.21.15.121:0/997573708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d5c101770 0x7f2d5c196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:45.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.706+0000 7f2d5bfff700 1 -- 172.21.15.121:0/997573708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d5c101770 msgr2=0x7f2d5c196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:45.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.706+0000 7f2d5bfff700 1 --2- 172.21.15.121:0/997573708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d5c101770 0x7f2d5c196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:45.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.707+0000 7f2d5bfff700 1 -- 172.21.15.121:0/997573708 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d440095f0 con 0x7f2d5c100dc0 2024-04-05T21:41:45.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.707+0000 7f2d5bfff700 1 --2- 172.21.15.121:0/997573708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d5c100dc0 0x7f2d5c1966d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00d8d0 tx=0x7f2d4c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:45.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.707+0000 7f2d597fa700 1 -- 172.21.15.121:0/997573708 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d4c009940 con 0x7f2d5c100dc0 2024-04-05T21:41:45.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.707+0000 7f2d597fa700 1 -- 172.21.15.121:0/997573708 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d4c010460 con 0x7f2d5c100dc0 2024-04-05T21:41:45.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.707+0000 7f2d626d7700 1 -- 172.21.15.121:0/997573708 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5c19a2b0 con 0x7f2d5c100dc0 2024-04-05T21:41:45.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.707+0000 7f2d597fa700 1 -- 172.21.15.121:0/997573708 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d4c00f5f0 con 0x7f2d5c100dc0 2024-04-05T21:41:45.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.707+0000 7f2d626d7700 1 -- 172.21.15.121:0/997573708 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5c19a800 con 0x7f2d5c100dc0 2024-04-05T21:41:45.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.708+0000 7f2d626d7700 1 -- 172.21.15.121:0/997573708 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5c1091c0 con 0x7f2d5c100dc0 2024-04-05T21:41:45.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.708+0000 7f2d597fa700 1 -- 172.21.15.121:0/997573708 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2d4c0105d0 con 0x7f2d5c100dc0 2024-04-05T21:41:45.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.709+0000 7f2d597fa700 1 --2- 172.21.15.121:0/997573708 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2d4806cce0 0x7f2d4806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:45.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.709+0000 7f2d5b7fe700 1 --2- 172.21.15.121:0/997573708 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2d4806cce0 0x7f2d4806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:45.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.709+0000 7f2d597fa700 1 -- 172.21.15.121:0/997573708 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2d4c08c5e0 con 0x7f2d5c100dc0 2024-04-05T21:41:45.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.709+0000 7f2d5b7fe700 1 --2- 172.21.15.121:0/997573708 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2d4806cce0 0x7f2d4806f1a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400b5c0 tx=0x7f2d44005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:45.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.713+0000 7f2d597fa700 1 -- 172.21.15.121:0/997573708 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2d4c056650 con 0x7f2d5c100dc0 2024-04-05T21:41:45.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:45 smithi121 ceph-mon[145859]: pgmap v465: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:45.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:41:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:45 smithi188 ceph-mon[107793]: pgmap v465: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:41:45.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.924+0000 7f2d626d7700 1 -- 172.21.15.121:0/997573708 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d5c19aa20 con 0x7f2d4806cce0 2024-04-05T21:41:45.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.925+0000 7f2d597fa700 1 -- 172.21.15.121:0/997573708 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f2d5c19aa20 con 0x7f2d4806cce0 2024-04-05T21:41:45.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.927+0000 7f2d626d7700 1 -- 172.21.15.121:0/997573708 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2d4806cce0 msgr2=0x7f2d4806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:45.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.927+0000 7f2d626d7700 1 --2- 172.21.15.121:0/997573708 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2d4806cce0 0x7f2d4806f1a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400b5c0 tx=0x7f2d44005fd0 comp rx=0 tx=0).stop 2024-04-05T21:41:45.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.927+0000 7f2d626d7700 1 -- 172.21.15.121:0/997573708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d5c100dc0 msgr2=0x7f2d5c1966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:45.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.928+0000 7f2d626d7700 1 --2- 172.21.15.121:0/997573708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d5c100dc0 0x7f2d5c1966d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00d8d0 tx=0x7f2d4c00dc90 comp rx=0 tx=0).stop 2024-04-05T21:41:45.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.928+0000 7f2d626d7700 1 -- 172.21.15.121:0/997573708 shutdown_connections 2024-04-05T21:41:45.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.928+0000 7f2d626d7700 1 --2- 172.21.15.121:0/997573708 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2d4806cce0 0x7f2d4806f1a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:45.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.928+0000 7f2d626d7700 1 --2- 172.21.15.121:0/997573708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d5c100dc0 0x7f2d5c1966d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:45.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.928+0000 7f2d626d7700 1 --2- 172.21.15.121:0/997573708 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d5c101770 0x7f2d5c196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:45.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.928+0000 7f2d626d7700 1 -- 172.21.15.121:0/997573708 >> 172.21.15.121:0/997573708 conn(0x7f2d5c0fc650 msgr2=0x7f2d5c0fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:45.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.928+0000 7f2d626d7700 1 -- 172.21.15.121:0/997573708 shutdown_connections 2024-04-05T21:41:45.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:45.928+0000 7f2d626d7700 1 -- 172.21.15.121:0/997573708 wait complete. 2024-04-05T21:41:46.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.030+0000 7fc31cc37700 1 -- 172.21.15.121:0/2863674716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc318101790 msgr2=0x7fc3181057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:46.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.030+0000 7fc31cc37700 1 --2- 172.21.15.121:0/2863674716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc318101790 0x7fc3181057e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc308009910 tx=0x7fc308009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:46.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.030+0000 7fc31cc37700 1 -- 172.21.15.121:0/2863674716 shutdown_connections 2024-04-05T21:41:46.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.030+0000 7fc31cc37700 1 --2- 172.21.15.121:0/2863674716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc318101790 0x7fc3181057e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.030+0000 7fc31cc37700 1 --2- 172.21.15.121:0/2863674716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc318100de0 0x7fc3181011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.030+0000 7fc31cc37700 1 -- 172.21.15.121:0/2863674716 >> 172.21.15.121:0/2863674716 conn(0x7fc3180fc6b0 msgr2=0x7fc3180fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:46.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.030+0000 7fc31cc37700 1 -- 172.21.15.121:0/2863674716 shutdown_connections 2024-04-05T21:41:46.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.030+0000 7fc31cc37700 1 -- 172.21.15.121:0/2863674716 wait complete. 2024-04-05T21:41:46.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.031+0000 7fc31cc37700 1 Processor -- start 2024-04-05T21:41:46.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.031+0000 7fc31cc37700 1 -- start start 2024-04-05T21:41:46.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.031+0000 7fc31cc37700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc318100de0 0x7fc318196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:46.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.031+0000 7fc31cc37700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc318101790 0x7fc318196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.031+0000 7fc31cc37700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc318197290 con 0x7fc318101790 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.031+0000 7fc31cc37700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31819b020 con 0x7fc318100de0 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.031+0000 7fc315d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc318101790 0x7fc318196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.031+0000 7fc315d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc318101790 0x7fc318196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51256/0 (socket says 172.21.15.121:51256) 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.031+0000 7fc315d9b700 1 -- 172.21.15.121:0/4008226186 learned_addr learned my addr 172.21.15.121:0/4008226186 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.031+0000 7fc31659c700 1 --2- 172.21.15.121:0/4008226186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc318100de0 0x7fc318196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.032+0000 7fc315d9b700 1 -- 172.21.15.121:0/4008226186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc318100de0 msgr2=0x7fc318196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.032+0000 7fc315d9b700 1 --2- 172.21.15.121:0/4008226186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc318100de0 0x7fc318196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.032+0000 7fc315d9b700 1 -- 172.21.15.121:0/4008226186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3080095f0 con 0x7fc318101790 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.032+0000 7fc315d9b700 1 --2- 172.21.15.121:0/4008226186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc318101790 0x7fc318196bb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc308000c00 tx=0x7fc30800bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.032+0000 7fc3077fe700 1 -- 172.21.15.121:0/4008226186 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc30801d070 con 0x7fc318101790 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.032+0000 7fc3077fe700 1 -- 172.21.15.121:0/4008226186 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc308003cf0 con 0x7fc318101790 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.032+0000 7fc31cc37700 1 -- 172.21.15.121:0/4008226186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc31819b2a0 con 0x7fc318101790 2024-04-05T21:41:46.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.032+0000 7fc3077fe700 1 -- 172.21.15.121:0/4008226186 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc308021410 con 0x7fc318101790 2024-04-05T21:41:46.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.032+0000 7fc31cc37700 1 -- 172.21.15.121:0/4008226186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc31819b730 con 0x7fc318101790 2024-04-05T21:41:46.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.033+0000 7fc31cc37700 1 -- 172.21.15.121:0/4008226186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3181091e0 con 0x7fc318101790 2024-04-05T21:41:46.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.033+0000 7fc3077fe700 1 -- 172.21.15.121:0/4008226186 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc308003850 con 0x7fc318101790 2024-04-05T21:41:46.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.034+0000 7fc3077fe700 1 --2- 172.21.15.121:0/4008226186 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc30006cce0 0x7fc30006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:46.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.034+0000 7fc3077fe700 1 -- 172.21.15.121:0/4008226186 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc30808d840 con 0x7fc318101790 2024-04-05T21:41:46.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.034+0000 7fc31659c700 1 --2- 172.21.15.121:0/4008226186 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc30006cce0 0x7fc30006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:46.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.034+0000 7fc31659c700 1 --2- 172.21.15.121:0/4008226186 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc30006cce0 0x7fc30006f1a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009b30 tx=0x7fc30c009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:46.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.038+0000 7fc3077fe700 1 -- 172.21.15.121:0/4008226186 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc3080578b0 con 0x7fc318101790 2024-04-05T21:41:46.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.240+0000 7fc31cc37700 1 -- 172.21.15.121:0/4008226186 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc31802cda0 con 0x7fc30006cce0 2024-04-05T21:41:46.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.250+0000 7fc3077fe700 1 -- 172.21.15.121:0/4008226186 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc31802cda0 con 0x7fc30006cce0 2024-04-05T21:41:46.251 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:41:46.251 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (12m) 6m ago 24m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:41:46.251 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (24m) 6m ago 24m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:41:46.251 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (23m) 6m ago 23m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:41:46.251 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (10m) 6m ago 24m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:41:46.251 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (10m) 6m ago 23m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (12m) 6m ago 23m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (7m) 6m ago 17m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (6m) 6m ago 17m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (6m) 6m ago 17m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (17m) 6m ago 17m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (14m) 6m ago 25m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (13m) 6m ago 22m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (11m) 6m ago 25m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (11m) 6m ago 22m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (13m) 6m ago 24m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (13m) 6m ago 23m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (10m) 6m ago 22m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (9m) 6m ago 21m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (9m) 6m ago 20m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (8m) 6m ago 20m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:41:46.252 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (7m) 6m ago 20m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:41:46.253 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (7m) 6m ago 19m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:41:46.253 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (12m) 6m ago 23m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:41:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.252+0000 7fc31cc37700 1 -- 172.21.15.121:0/4008226186 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc30006cce0 msgr2=0x7fc30006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.252+0000 7fc31cc37700 1 --2- 172.21.15.121:0/4008226186 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc30006cce0 0x7fc30006f1a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc30c009b30 tx=0x7fc30c009260 comp rx=0 tx=0).stop 2024-04-05T21:41:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.252+0000 7fc31cc37700 1 -- 172.21.15.121:0/4008226186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc318101790 msgr2=0x7fc318196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:46.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.252+0000 7fc31cc37700 1 --2- 172.21.15.121:0/4008226186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc318101790 0x7fc318196bb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc308000c00 tx=0x7fc30800bb20 comp rx=0 tx=0).stop 2024-04-05T21:41:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.252+0000 7fc31cc37700 1 -- 172.21.15.121:0/4008226186 shutdown_connections 2024-04-05T21:41:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.252+0000 7fc31cc37700 1 --2- 172.21.15.121:0/4008226186 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc30006cce0 0x7fc30006f1a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.252+0000 7fc31cc37700 1 --2- 172.21.15.121:0/4008226186 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc318100de0 0x7fc318196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.252+0000 7fc31cc37700 1 --2- 172.21.15.121:0/4008226186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc318101790 0x7fc318196bb0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.252+0000 7fc31cc37700 1 -- 172.21.15.121:0/4008226186 >> 172.21.15.121:0/4008226186 conn(0x7fc3180fc6b0 msgr2=0x7fc3180fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.253+0000 7fc31cc37700 1 -- 172.21.15.121:0/4008226186 shutdown_connections 2024-04-05T21:41:46.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.253+0000 7fc31cc37700 1 -- 172.21.15.121:0/4008226186 wait complete. 2024-04-05T21:41:46.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.354+0000 7f961c0f1700 1 -- 172.21.15.121:0/1605743669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9614101750 msgr2=0x7f96141057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.354+0000 7f961c0f1700 1 --2- 172.21.15.121:0/1605743669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9614101750 0x7f96141057a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9610009910 tx=0x7f9610009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.354+0000 7f961c0f1700 1 -- 172.21.15.121:0/1605743669 shutdown_connections 2024-04-05T21:41:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.354+0000 7f961c0f1700 1 --2- 172.21.15.121:0/1605743669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9614101750 0x7f96141057a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.354+0000 7f961c0f1700 1 --2- 172.21.15.121:0/1605743669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9614100da0 0x7f9614101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.354+0000 7f961c0f1700 1 -- 172.21.15.121:0/1605743669 >> 172.21.15.121:0/1605743669 conn(0x7f96140fc630 msgr2=0x7f96140fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.354+0000 7f961c0f1700 1 -- 172.21.15.121:0/1605743669 shutdown_connections 2024-04-05T21:41:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.354+0000 7f961c0f1700 1 -- 172.21.15.121:0/1605743669 wait complete. 2024-04-05T21:41:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.355+0000 7f961c0f1700 1 Processor -- start 2024-04-05T21:41:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.355+0000 7f961c0f1700 1 -- start start 2024-04-05T21:41:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.355+0000 7f961c0f1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9614100da0 0x7f9614196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.355+0000 7f961c0f1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9614101750 0x7f9614196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.355+0000 7f961c0f1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9614197270 con 0x7f9614100da0 2024-04-05T21:41:46.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.355+0000 7f961c0f1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961419b000 con 0x7f9614101750 2024-04-05T21:41:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.355+0000 7f9619e8d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9614100da0 0x7f9614196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.355+0000 7f9619e8d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9614100da0 0x7f9614196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51272/0 (socket says 172.21.15.121:51272) 2024-04-05T21:41:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.355+0000 7f9619e8d700 1 -- 172.21.15.121:0/1811934734 learned_addr learned my addr 172.21.15.121:0/1811934734 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.355+0000 7f961968c700 1 --2- 172.21.15.121:0/1811934734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9614101750 0x7f9614196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.356+0000 7f9619e8d700 1 -- 172.21.15.121:0/1811934734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9614101750 msgr2=0x7f9614196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.356+0000 7f9619e8d700 1 --2- 172.21.15.121:0/1811934734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9614101750 0x7f9614196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.356+0000 7f9619e8d700 1 -- 172.21.15.121:0/1811934734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96100095f0 con 0x7f9614100da0 2024-04-05T21:41:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.356+0000 7f9619e8d700 1 --2- 172.21.15.121:0/1811934734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9614100da0 0x7f9614196650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f960400d8d0 tx=0x7f960400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.356+0000 7f960affd700 1 -- 172.21.15.121:0/1811934734 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9604009940 con 0x7f9614100da0 2024-04-05T21:41:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.356+0000 7f960affd700 1 -- 172.21.15.121:0/1811934734 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9604010460 con 0x7f9614100da0 2024-04-05T21:41:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.356+0000 7f961c0f1700 1 -- 172.21.15.121:0/1811934734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961419b2e0 con 0x7f9614100da0 2024-04-05T21:41:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.356+0000 7f960affd700 1 -- 172.21.15.121:0/1811934734 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f960400f5f0 con 0x7f9614100da0 2024-04-05T21:41:46.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.356+0000 7f961c0f1700 1 -- 172.21.15.121:0/1811934734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961419b830 con 0x7f9614100da0 2024-04-05T21:41:46.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.357+0000 7f961c0f1700 1 -- 172.21.15.121:0/1811934734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96141091a0 con 0x7f9614100da0 2024-04-05T21:41:46.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.357+0000 7f960affd700 1 -- 172.21.15.121:0/1811934734 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f96040105d0 con 0x7f9614100da0 2024-04-05T21:41:46.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.358+0000 7f960affd700 1 --2- 172.21.15.121:0/1811934734 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f960006cce0 0x7f960006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:46.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.358+0000 7f960affd700 1 -- 172.21.15.121:0/1811934734 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f960408c5e0 con 0x7f9614100da0 2024-04-05T21:41:46.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.358+0000 7f961968c700 1 --2- 172.21.15.121:0/1811934734 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f960006cce0 0x7f960006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:46.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.358+0000 7f961968c700 1 --2- 172.21.15.121:0/1811934734 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f960006cce0 0x7f960006f1a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9614197c70 tx=0x7f961001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:46.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.362+0000 7f960affd700 1 -- 172.21.15.121:0/1811934734 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9604056650 con 0x7f9614100da0 2024-04-05T21:41:46.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.646+0000 7f961c0f1700 1 -- 172.21.15.121:0/1811934734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f961419bb40 con 0x7f9614100da0 2024-04-05T21:41:46.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.647+0000 7f960affd700 1 -- 172.21.15.121:0/1811934734 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f9604010880 con 0x7f9614100da0 2024-04-05T21:41:46.647 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:41:46.647 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:41:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:41:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:41:46.649 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:41:46.649 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:41:46.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.649+0000 7f961c0f1700 1 -- 172.21.15.121:0/1811934734 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f960006cce0 msgr2=0x7f960006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:46.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.649+0000 7f961c0f1700 1 --2- 172.21.15.121:0/1811934734 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f960006cce0 0x7f960006f1a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9614197c70 tx=0x7f961001a040 comp rx=0 tx=0).stop 2024-04-05T21:41:46.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.649+0000 7f961c0f1700 1 -- 172.21.15.121:0/1811934734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9614100da0 msgr2=0x7f9614196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:46.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.649+0000 7f961c0f1700 1 --2- 172.21.15.121:0/1811934734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9614100da0 0x7f9614196650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f960400d8d0 tx=0x7f960400dc90 comp rx=0 tx=0).stop 2024-04-05T21:41:46.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.649+0000 7f961c0f1700 1 -- 172.21.15.121:0/1811934734 shutdown_connections 2024-04-05T21:41:46.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.649+0000 7f961c0f1700 1 --2- 172.21.15.121:0/1811934734 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f960006cce0 0x7f960006f1a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.649+0000 7f961c0f1700 1 --2- 172.21.15.121:0/1811934734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9614100da0 0x7f9614196650 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.649+0000 7f961c0f1700 1 --2- 172.21.15.121:0/1811934734 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9614101750 0x7f9614196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.649+0000 7f961c0f1700 1 -- 172.21.15.121:0/1811934734 >> 172.21.15.121:0/1811934734 conn(0x7f96140fc630 msgr2=0x7f96140fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:46.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.649+0000 7f961c0f1700 1 -- 172.21.15.121:0/1811934734 shutdown_connections 2024-04-05T21:41:46.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.649+0000 7f961c0f1700 1 -- 172.21.15.121:0/1811934734 wait complete. 2024-04-05T21:41:46.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:46 smithi121 ceph-mon[145859]: from='client.34706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:46.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.752+0000 7fca42506700 1 -- 172.21.15.121:0/1867571887 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3c102620 msgr2=0x7fca3c10ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:46.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.752+0000 7fca42506700 1 --2- 172.21.15.121:0/1867571887 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3c102620 0x7fca3c10ab10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fca2c009910 tx=0x7fca2c009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:46.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.752+0000 7fca42506700 1 -- 172.21.15.121:0/1867571887 shutdown_connections 2024-04-05T21:41:46.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.752+0000 7fca42506700 1 --2- 172.21.15.121:0/1867571887 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3c102620 0x7fca3c10ab10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.752+0000 7fca42506700 1 --2- 172.21.15.121:0/1867571887 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca3c101d00 0x7fca3c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.752+0000 7fca42506700 1 -- 172.21.15.121:0/1867571887 >> 172.21.15.121:0/1867571887 conn(0x7fca3c0fb570 msgr2=0x7fca3c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:46.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.752+0000 7fca42506700 1 -- 172.21.15.121:0/1867571887 shutdown_connections 2024-04-05T21:41:46.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.752+0000 7fca42506700 1 -- 172.21.15.121:0/1867571887 wait complete. 2024-04-05T21:41:46.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.752+0000 7fca42506700 1 Processor -- start 2024-04-05T21:41:46.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.752+0000 7fca42506700 1 -- start start 2024-04-05T21:41:46.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca42506700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3c101d00 0x7fca3c1966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:46.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca42506700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca3c102620 0x7fca3c196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:46.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca42506700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c1972a0 con 0x7fca3c101d00 2024-04-05T21:41:46.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca42506700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c199fd0 con 0x7fca3c102620 2024-04-05T21:41:46.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca3bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3c101d00 0x7fca3c1966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:46.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca3bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3c101d00 0x7fca3c1966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51286/0 (socket says 172.21.15.121:51286) 2024-04-05T21:41:46.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca3bfff700 1 -- 172.21.15.121:0/187609855 learned_addr learned my addr 172.21.15.121:0/187609855 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:46.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca3b7fe700 1 --2- 172.21.15.121:0/187609855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca3c102620 0x7fca3c196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:46.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca3bfff700 1 -- 172.21.15.121:0/187609855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca3c102620 msgr2=0x7fca3c196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:46.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca3bfff700 1 --2- 172.21.15.121:0/187609855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca3c102620 0x7fca3c196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:46.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca3bfff700 1 -- 172.21.15.121:0/187609855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c0095f0 con 0x7fca3c101d00 2024-04-05T21:41:46.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca3bfff700 1 --2- 172.21.15.121:0/187609855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3c101d00 0x7fca3c1966d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fca3000d900 tx=0x7fca3000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:46.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca397fa700 1 -- 172.21.15.121:0/187609855 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca300041f0 con 0x7fca3c101d00 2024-04-05T21:41:46.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca397fa700 1 -- 172.21.15.121:0/187609855 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca30004cd0 con 0x7fca3c101d00 2024-04-05T21:41:46.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca42506700 1 -- 172.21.15.121:0/187609855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3c19a2b0 con 0x7fca3c101d00 2024-04-05T21:41:46.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca397fa700 1 -- 172.21.15.121:0/187609855 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca300056d0 con 0x7fca3c101d00 2024-04-05T21:41:46.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.753+0000 7fca42506700 1 -- 172.21.15.121:0/187609855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3c19a7d0 con 0x7fca3c101d00 2024-04-05T21:41:46.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.754+0000 7fca42506700 1 -- 172.21.15.121:0/187609855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3c108210 con 0x7fca3c101d00 2024-04-05T21:41:46.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.755+0000 7fca397fa700 1 -- 172.21.15.121:0/187609855 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fca30004e40 con 0x7fca3c101d00 2024-04-05T21:41:46.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.755+0000 7fca397fa700 1 --2- 172.21.15.121:0/187609855 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fca2406cc90 0x7fca2406f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:46.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.755+0000 7fca397fa700 1 -- 172.21.15.121:0/187609855 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fca3008c710 con 0x7fca3c101d00 2024-04-05T21:41:46.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.755+0000 7fca3b7fe700 1 --2- 172.21.15.121:0/187609855 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fca2406cc90 0x7fca2406f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:46.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.756+0000 7fca3b7fe700 1 --2- 172.21.15.121:0/187609855 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fca2406cc90 0x7fca2406f150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fca3c197df0 tx=0x7fca2c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:46.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.760+0000 7fca397fa700 1 -- 172.21.15.121:0/187609855 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fca30056780 con 0x7fca3c101d00 2024-04-05T21:41:46.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:46 smithi188 ceph-mon[107793]: from='client.34706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:46.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.994+0000 7fca42506700 1 -- 172.21.15.121:0/187609855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fca3c19a9f0 con 0x7fca3c101d00 2024-04-05T21:41:46.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.995+0000 7fca397fa700 1 -- 172.21.15.121:0/187609855 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fca30016750 con 0x7fca3c101d00 2024-04-05T21:41:46.996 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:41:46.996 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:41:46.996 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:41:46.996 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:41:46.996 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:41:46.996 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:41:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:41:46.998 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:41:46.999 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:41:46.999 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:41:46.999 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:41:46.999 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:41:46.999 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:41:46.999 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:41:47.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.997+0000 7fca42506700 1 -- 172.21.15.121:0/187609855 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fca2406cc90 msgr2=0x7fca2406f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:47.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.997+0000 7fca42506700 1 --2- 172.21.15.121:0/187609855 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fca2406cc90 0x7fca2406f150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fca3c197df0 tx=0x7fca2c01a040 comp rx=0 tx=0).stop 2024-04-05T21:41:47.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.997+0000 7fca42506700 1 -- 172.21.15.121:0/187609855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3c101d00 msgr2=0x7fca3c1966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:47.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.997+0000 7fca42506700 1 --2- 172.21.15.121:0/187609855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3c101d00 0x7fca3c1966d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fca3000d900 tx=0x7fca3000dcc0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.997+0000 7fca42506700 1 -- 172.21.15.121:0/187609855 shutdown_connections 2024-04-05T21:41:47.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.997+0000 7fca42506700 1 --2- 172.21.15.121:0/187609855 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fca2406cc90 0x7fca2406f150 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.997+0000 7fca42506700 1 --2- 172.21.15.121:0/187609855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3c101d00 0x7fca3c1966d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.997+0000 7fca42506700 1 --2- 172.21.15.121:0/187609855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fca3c102620 0x7fca3c196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.997+0000 7fca42506700 1 -- 172.21.15.121:0/187609855 >> 172.21.15.121:0/187609855 conn(0x7fca3c0fb570 msgr2=0x7fca3c105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:47.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.997+0000 7fca42506700 1 -- 172.21.15.121:0/187609855 shutdown_connections 2024-04-05T21:41:47.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:46.997+0000 7fca42506700 1 -- 172.21.15.121:0/187609855 wait complete. 2024-04-05T21:41:47.000 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:41:47.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.100+0000 7f37a29cb700 1 -- 172.21.15.121:0/161845933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f379c101790 msgr2=0x7f379c1057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:47.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.100+0000 7f37a29cb700 1 --2- 172.21.15.121:0/161845933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f379c101790 0x7f379c1057e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3790009910 tx=0x7f3790009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:47.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.100+0000 7f37a29cb700 1 -- 172.21.15.121:0/161845933 shutdown_connections 2024-04-05T21:41:47.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.100+0000 7f37a29cb700 1 --2- 172.21.15.121:0/161845933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f379c101790 0x7f379c1057e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.100+0000 7f37a29cb700 1 --2- 172.21.15.121:0/161845933 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f379c100de0 0x7f379c1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.100+0000 7f37a29cb700 1 -- 172.21.15.121:0/161845933 >> 172.21.15.121:0/161845933 conn(0x7f379c0fc670 msgr2=0x7f379c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:47.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.100+0000 7f37a29cb700 1 -- 172.21.15.121:0/161845933 shutdown_connections 2024-04-05T21:41:47.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.100+0000 7f37a29cb700 1 -- 172.21.15.121:0/161845933 wait complete. 2024-04-05T21:41:47.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.101+0000 7f37a29cb700 1 Processor -- start 2024-04-05T21:41:47.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.101+0000 7f37a29cb700 1 -- start start 2024-04-05T21:41:47.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.101+0000 7f37a29cb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f379c100de0 0x7f379c1966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:47.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.101+0000 7f37a29cb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f379c101790 0x7f379c196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:47.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.101+0000 7f37a29cb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c1972a0 con 0x7f379c101790 2024-04-05T21:41:47.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.101+0000 7f37a29cb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c199fd0 con 0x7f379c100de0 2024-04-05T21:41:47.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.102+0000 7f379b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f379c101790 0x7f379c196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:47.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.102+0000 7f379b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f379c101790 0x7f379c196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51300/0 (socket says 172.21.15.121:51300) 2024-04-05T21:41:47.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.102+0000 7f379b7fe700 1 -- 172.21.15.121:0/1277356440 learned_addr learned my addr 172.21.15.121:0/1277356440 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:47.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.102+0000 7f379bfff700 1 --2- 172.21.15.121:0/1277356440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f379c100de0 0x7f379c1966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:47.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.102+0000 7f379b7fe700 1 -- 172.21.15.121:0/1277356440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f379c100de0 msgr2=0x7f379c1966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:47.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.102+0000 7f379b7fe700 1 --2- 172.21.15.121:0/1277356440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f379c100de0 0x7f379c1966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.102+0000 7f379b7fe700 1 -- 172.21.15.121:0/1277356440 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37900095f0 con 0x7f379c101790 2024-04-05T21:41:47.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.102+0000 7f379b7fe700 1 --2- 172.21.15.121:0/1277356440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f379c101790 0x7f379c196c10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f379000bda0 tx=0x7f379000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:47.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.102+0000 7f37997fa700 1 -- 172.21.15.121:0/1277356440 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f379001d070 con 0x7f379c101790 2024-04-05T21:41:47.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.102+0000 7f37997fa700 1 -- 172.21.15.121:0/1277356440 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f379000f460 con 0x7f379c101790 2024-04-05T21:41:47.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.102+0000 7f37997fa700 1 -- 172.21.15.121:0/1277356440 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3790021620 con 0x7f379c101790 2024-04-05T21:41:47.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.102+0000 7f37a29cb700 1 -- 172.21.15.121:0/1277356440 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379c19a250 con 0x7f379c101790 2024-04-05T21:41:47.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.102+0000 7f37a29cb700 1 -- 172.21.15.121:0/1277356440 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379c19a740 con 0x7f379c101790 2024-04-05T21:41:47.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.103+0000 7f37a29cb700 1 -- 172.21.15.121:0/1277356440 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f379c1091e0 con 0x7f379c101790 2024-04-05T21:41:47.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.104+0000 7f37997fa700 1 -- 172.21.15.121:0/1277356440 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3790003a60 con 0x7f379c101790 2024-04-05T21:41:47.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.104+0000 7f37997fa700 1 --2- 172.21.15.121:0/1277356440 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f378406cce0 0x7f378406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:47.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.104+0000 7f37997fa700 1 -- 172.21.15.121:0/1277356440 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f379008d7e0 con 0x7f379c101790 2024-04-05T21:41:47.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.104+0000 7f379bfff700 1 --2- 172.21.15.121:0/1277356440 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f378406cce0 0x7f378406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:47.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.104+0000 7f379bfff700 1 --2- 172.21.15.121:0/1277356440 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f378406cce0 0x7f378406f1a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f378c005950 tx=0x7f378c00a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:47.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.108+0000 7f37997fa700 1 -- 172.21.15.121:0/1277356440 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3790057850 con 0x7f379c101790 2024-04-05T21:41:47.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.321+0000 7f37a29cb700 1 -- 172.21.15.121:0/1277356440 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f379c02cd30 con 0x7f378406cce0 2024-04-05T21:41:47.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.322+0000 7f37997fa700 1 -- 172.21.15.121:0/1277356440 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f379c02cd30 con 0x7f378406cce0 2024-04-05T21:41:47.322 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:41:47.323 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:41:47.323 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:41:47.323 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:41:47.323 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:41:47.323 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:41:47.323 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:41:47.323 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:41:47.323 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:41:47.323 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:41:47.323 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:41:47.323 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:41:47.323 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:41:47.323 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:41:47.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.324+0000 7f37a29cb700 1 -- 172.21.15.121:0/1277356440 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f378406cce0 msgr2=0x7f378406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:47.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.324+0000 7f37a29cb700 1 --2- 172.21.15.121:0/1277356440 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f378406cce0 0x7f378406f1a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f378c005950 tx=0x7f378c00a300 comp rx=0 tx=0).stop 2024-04-05T21:41:47.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.324+0000 7f37a29cb700 1 -- 172.21.15.121:0/1277356440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f379c101790 msgr2=0x7f379c196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:47.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.324+0000 7f37a29cb700 1 --2- 172.21.15.121:0/1277356440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f379c101790 0x7f379c196c10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f379000bda0 tx=0x7f379000bdd0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.324+0000 7f37a29cb700 1 -- 172.21.15.121:0/1277356440 shutdown_connections 2024-04-05T21:41:47.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.324+0000 7f37a29cb700 1 --2- 172.21.15.121:0/1277356440 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f378406cce0 0x7f378406f1a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.324+0000 7f37a29cb700 1 --2- 172.21.15.121:0/1277356440 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f379c100de0 0x7f379c1966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.324+0000 7f37a29cb700 1 --2- 172.21.15.121:0/1277356440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f379c101790 0x7f379c196c10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.324+0000 7f37a29cb700 1 -- 172.21.15.121:0/1277356440 >> 172.21.15.121:0/1277356440 conn(0x7f379c0fc670 msgr2=0x7f379c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:47.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.324+0000 7f37a29cb700 1 -- 172.21.15.121:0/1277356440 shutdown_connections 2024-04-05T21:41:47.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.324+0000 7f37a29cb700 1 -- 172.21.15.121:0/1277356440 wait complete. 2024-04-05T21:41:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.428+0000 7f7d19368700 1 -- 172.21.15.121:0/1686276977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d14105a40 msgr2=0x7f7d14105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.428+0000 7f7d19368700 1 --2- 172.21.15.121:0/1686276977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d14105a40 0x7f7d14105e20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009910 tx=0x7f7d08009c20 comp rx=0 tx=0).stop 2024-04-05T21:41:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.428+0000 7f7d19368700 1 -- 172.21.15.121:0/1686276977 shutdown_connections 2024-04-05T21:41:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.428+0000 7f7d19368700 1 --2- 172.21.15.121:0/1686276977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d140ffbd0 0x7f7d14100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.428+0000 7f7d19368700 1 --2- 172.21.15.121:0/1686276977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d14105a40 0x7f7d14105e20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.428+0000 7f7d19368700 1 -- 172.21.15.121:0/1686276977 >> 172.21.15.121:0/1686276977 conn(0x7f7d140755a0 msgr2=0x7f7d140759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.428+0000 7f7d19368700 1 -- 172.21.15.121:0/1686276977 shutdown_connections 2024-04-05T21:41:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.428+0000 7f7d19368700 1 -- 172.21.15.121:0/1686276977 wait complete. 2024-04-05T21:41:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d19368700 1 Processor -- start 2024-04-05T21:41:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d19368700 1 -- start start 2024-04-05T21:41:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d19368700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d140ffbd0 0x7f7d141975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d19368700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d14105a40 0x7f7d14197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d19368700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d14198100 con 0x7f7d140ffbd0 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d19368700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d14198270 con 0x7f7d14105a40 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d12ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d140ffbd0 0x7f7d141975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d12ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d140ffbd0 0x7f7d141975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51326/0 (socket says 172.21.15.121:51326) 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d12ffd700 1 -- 172.21.15.121:0/2586629230 learned_addr learned my addr 172.21.15.121:0/2586629230 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d127fc700 1 --2- 172.21.15.121:0/2586629230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d14105a40 0x7f7d14197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d12ffd700 1 -- 172.21.15.121:0/2586629230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d14105a40 msgr2=0x7f7d14197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d12ffd700 1 --2- 172.21.15.121:0/2586629230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d14105a40 0x7f7d14197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d12ffd700 1 -- 172.21.15.121:0/2586629230 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d080095f0 con 0x7f7d140ffbd0 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.429+0000 7f7d127fc700 1 --2- 172.21.15.121:0/2586629230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d14105a40 0x7f7d14197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.430+0000 7f7d12ffd700 1 --2- 172.21.15.121:0/2586629230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d140ffbd0 0x7f7d141975a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7d08006010 tx=0x7f7d0800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.430+0000 7f7cfbfff700 1 -- 172.21.15.121:0/2586629230 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d0801d070 con 0x7f7d140ffbd0 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.430+0000 7f7d19368700 1 -- 172.21.15.121:0/2586629230 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d141a0610 con 0x7f7d140ffbd0 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.430+0000 7f7cfbfff700 1 -- 172.21.15.121:0/2586629230 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d0800f460 con 0x7f7d140ffbd0 2024-04-05T21:41:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.430+0000 7f7cfbfff700 1 -- 172.21.15.121:0/2586629230 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d08021640 con 0x7f7d140ffbd0 2024-04-05T21:41:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.430+0000 7f7d19368700 1 -- 172.21.15.121:0/2586629230 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d141a0a80 con 0x7f7d140ffbd0 2024-04-05T21:41:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.431+0000 7f7d19368700 1 -- 172.21.15.121:0/2586629230 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d14108040 con 0x7f7d140ffbd0 2024-04-05T21:41:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.431+0000 7f7cfbfff700 1 -- 172.21.15.121:0/2586629230 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7d08003a60 con 0x7f7d140ffbd0 2024-04-05T21:41:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.432+0000 7f7cfbfff700 1 --2- 172.21.15.121:0/2586629230 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cfc06cce0 0x7f7cfc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:41:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.432+0000 7f7cfbfff700 1 -- 172.21.15.121:0/2586629230 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7d0808d8f0 con 0x7f7d140ffbd0 2024-04-05T21:41:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.432+0000 7f7d127fc700 1 --2- 172.21.15.121:0/2586629230 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cfc06cce0 0x7f7cfc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:41:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.432+0000 7f7d127fc700 1 --2- 172.21.15.121:0/2586629230 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cfc06cce0 0x7f7cfc06f1a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7d14068ce0 tx=0x7f7d04008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:41:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.436+0000 7f7cfbfff700 1 -- 172.21.15.121:0/2586629230 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7d08057960 con 0x7f7d140ffbd0 2024-04-05T21:41:47.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.704+0000 7f7d19368700 1 -- 172.21.15.121:0/2586629230 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7d1404ea90 con 0x7f7d140ffbd0 2024-04-05T21:41:47.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.705+0000 7f7cfbfff700 1 -- 172.21.15.121:0/2586629230 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f7d0805af80 con 0x7f7d140ffbd0 2024-04-05T21:41:47.706 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:41:47.706 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:41:47.706 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:41:47.706 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:41:47.706 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:41:47.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.707+0000 7f7d19368700 1 -- 172.21.15.121:0/2586629230 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cfc06cce0 msgr2=0x7f7cfc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:47.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.707+0000 7f7d19368700 1 --2- 172.21.15.121:0/2586629230 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cfc06cce0 0x7f7cfc06f1a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7d14068ce0 tx=0x7f7d04008040 comp rx=0 tx=0).stop 2024-04-05T21:41:47.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.707+0000 7f7d19368700 1 -- 172.21.15.121:0/2586629230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d140ffbd0 msgr2=0x7f7d141975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:41:47.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.707+0000 7f7d19368700 1 --2- 172.21.15.121:0/2586629230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d140ffbd0 0x7f7d141975a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7d08006010 tx=0x7f7d0800be20 comp rx=0 tx=0).stop 2024-04-05T21:41:47.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.707+0000 7f7d19368700 1 -- 172.21.15.121:0/2586629230 shutdown_connections 2024-04-05T21:41:47.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.707+0000 7f7d19368700 1 --2- 172.21.15.121:0/2586629230 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cfc06cce0 0x7f7cfc06f1a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.707+0000 7f7d19368700 1 --2- 172.21.15.121:0/2586629230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d140ffbd0 0x7f7d141975a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.707+0000 7f7d19368700 1 --2- 172.21.15.121:0/2586629230 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d14105a40 0x7f7d14197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:41:47.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.707+0000 7f7d19368700 1 -- 172.21.15.121:0/2586629230 >> 172.21.15.121:0/2586629230 conn(0x7f7d140755a0 msgr2=0x7f7d140fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:41:47.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.707+0000 7f7d19368700 1 -- 172.21.15.121:0/2586629230 shutdown_connections 2024-04-05T21:41:47.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:41:47.707+0000 7f7d19368700 1 -- 172.21.15.121:0/2586629230 wait complete. 2024-04-05T21:41:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:47 smithi121 ceph-mon[145859]: from='client.34710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:47 smithi121 ceph-mon[145859]: from='client.34714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:47 smithi121 ceph-mon[145859]: pgmap v466: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:47 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1811934734' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:41:47.740 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:47 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/187609855' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:41:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:47 smithi188 ceph-mon[107793]: from='client.34710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:47.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:47 smithi188 ceph-mon[107793]: from='client.34714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:47.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:47 smithi188 ceph-mon[107793]: pgmap v466: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:47.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:47 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1811934734' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:41:47.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:47 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/187609855' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:41:48.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:48 smithi121 ceph-mon[145859]: from='client.34726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:48.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:48 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2586629230' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:41:48.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:48 smithi188 ceph-mon[107793]: from='client.34726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:41:48.783 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:48 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2586629230' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:41:49.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:49 smithi121 ceph-mon[145859]: pgmap v467: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:49.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:49 smithi188 ceph-mon[107793]: pgmap v467: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:51.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:51 smithi121 ceph-mon[145859]: pgmap v468: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:51.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:51 smithi188 ceph-mon[107793]: pgmap v468: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:53.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:53 smithi121 ceph-mon[145859]: pgmap v469: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:53 smithi188 ceph-mon[107793]: pgmap v469: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:55.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:55 smithi121 ceph-mon[145859]: pgmap v470: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:55 smithi188 ceph-mon[107793]: pgmap v470: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:57 smithi121 ceph-mon[145859]: pgmap v471: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:57 smithi188 ceph-mon[107793]: pgmap v471: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:59.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:41:59 smithi121 ceph-mon[145859]: pgmap v472: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:41:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:41:59 smithi188 ceph-mon[107793]: pgmap v472: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:42:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:42:01.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:01 smithi121 ceph-mon[145859]: pgmap v473: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:01 smithi188 ceph-mon[107793]: pgmap v473: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:03.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:03 smithi121 ceph-mon[145859]: pgmap v474: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:03.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:03 smithi188 ceph-mon[107793]: pgmap v474: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:05 smithi121 ceph-mon[145859]: pgmap v475: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:05 smithi188 ceph-mon[107793]: pgmap v475: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:07 smithi121 ceph-mon[145859]: pgmap v476: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:07 smithi188 ceph-mon[107793]: pgmap v476: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:09.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:09 smithi121 ceph-mon[145859]: pgmap v477: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:09 smithi188 ceph-mon[107793]: pgmap v477: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:11 smithi121 ceph-mon[145859]: pgmap v478: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:11.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:11 smithi188 ceph-mon[107793]: pgmap v478: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:13.722 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:13 smithi188 ceph-mon[107793]: pgmap v479: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:13.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:13 smithi121 ceph-mon[145859]: pgmap v479: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:15 smithi121 ceph-mon[145859]: pgmap v480: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:42:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:15 smithi188 ceph-mon[107793]: pgmap v480: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:42:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:17 smithi121 ceph-mon[145859]: pgmap v481: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:17.998 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:17 smithi188 ceph-mon[107793]: pgmap v481: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:18.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.825+0000 7f5fe1520700 1 -- 172.21.15.121:0/4058875690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fdc0ffee0 msgr2=0x7f5fdc103eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.825+0000 7f5fe1520700 1 --2- 172.21.15.121:0/4058875690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fdc0ffee0 0x7f5fdc103eb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009910 tx=0x7f5fcc009c20 comp rx=0 tx=0).stop 2024-04-05T21:42:18.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.825+0000 7f5fe1520700 1 -- 172.21.15.121:0/4058875690 shutdown_connections 2024-04-05T21:42:18.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.825+0000 7f5fe1520700 1 --2- 172.21.15.121:0/4058875690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fdc0ffee0 0x7f5fdc103eb0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.825+0000 7f5fe1520700 1 --2- 172.21.15.121:0/4058875690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fdc0ff530 0x7f5fdc0ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.825+0000 7f5fe1520700 1 -- 172.21.15.121:0/4058875690 >> 172.21.15.121:0/4058875690 conn(0x7f5fdc074df0 msgr2=0x7f5fdc075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:18.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.825+0000 7f5fe1520700 1 -- 172.21.15.121:0/4058875690 shutdown_connections 2024-04-05T21:42:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.825+0000 7f5fe1520700 1 -- 172.21.15.121:0/4058875690 wait complete. 2024-04-05T21:42:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.826+0000 7f5fe1520700 1 Processor -- start 2024-04-05T21:42:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.826+0000 7f5fe1520700 1 -- start start 2024-04-05T21:42:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.826+0000 7f5fe1520700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fdc0ff530 0x7f5fdc197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.826+0000 7f5fe1520700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fdc0ffee0 0x7f5fdc197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.826+0000 7f5fe1520700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc198090 con 0x7f5fdc0ff530 2024-04-05T21:42:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.826+0000 7f5fe1520700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc1981d0 con 0x7f5fdc0ffee0 2024-04-05T21:42:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.826+0000 7f5fdaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fdc0ff530 0x7f5fdc197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.826+0000 7f5fdaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fdc0ff530 0x7f5fdc197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47772/0 (socket says 172.21.15.121:47772) 2024-04-05T21:42:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.826+0000 7f5fdaffd700 1 -- 172.21.15.121:0/864045737 learned_addr learned my addr 172.21.15.121:0/864045737 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.827+0000 7f5fda7fc700 1 --2- 172.21.15.121:0/864045737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fdc0ffee0 0x7f5fdc197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.827+0000 7f5fdaffd700 1 -- 172.21.15.121:0/864045737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fdc0ffee0 msgr2=0x7f5fdc197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.827+0000 7f5fdaffd700 1 --2- 172.21.15.121:0/864045737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fdc0ffee0 0x7f5fdc197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.827+0000 7f5fdaffd700 1 -- 172.21.15.121:0/864045737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc0095f0 con 0x7f5fdc0ff530 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.827+0000 7f5fdaffd700 1 --2- 172.21.15.121:0/864045737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fdc0ff530 0x7f5fdc197530 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400d8d0 tx=0x7f5fc400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.827+0000 7f5fd3fff700 1 -- 172.21.15.121:0/864045737 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fc4009940 con 0x7f5fdc0ff530 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.827+0000 7f5fd3fff700 1 -- 172.21.15.121:0/864045737 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fc4010460 con 0x7f5fdc0ff530 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.827+0000 7f5fe1520700 1 -- 172.21.15.121:0/864045737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc1a05a0 con 0x7f5fdc0ff530 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.827+0000 7f5fd3fff700 1 -- 172.21.15.121:0/864045737 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fc400f5f0 con 0x7f5fdc0ff530 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.827+0000 7f5fe1520700 1 -- 172.21.15.121:0/864045737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc1a0a60 con 0x7f5fdc0ff530 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.828+0000 7f5fe1520700 1 -- 172.21.15.121:0/864045737 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc107790 con 0x7f5fdc0ff530 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.829+0000 7f5fd3fff700 1 -- 172.21.15.121:0/864045737 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5fc40105d0 con 0x7f5fdc0ff530 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.829+0000 7f5fd3fff700 1 --2- 172.21.15.121:0/864045737 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fc806cce0 0x7f5fc806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.829+0000 7f5fd3fff700 1 -- 172.21.15.121:0/864045737 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5fc408c5e0 con 0x7f5fdc0ff530 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.829+0000 7f5fda7fc700 1 --2- 172.21.15.121:0/864045737 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fc806cce0 0x7f5fc806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.829+0000 7f5fda7fc700 1 --2- 172.21.15.121:0/864045737 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fc806cce0 0x7f5fc806f1a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc00b5c0 tx=0x7f5fcc005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:18.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:17.834+0000 7f5fd3fff700 1 -- 172.21.15.121:0/864045737 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5fc4056650 con 0x7f5fdc0ff530 2024-04-05T21:42:18.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.045+0000 7f5fe1520700 1 -- 172.21.15.121:0/864045737 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fdc02cd80 con 0x7f5fc806cce0 2024-04-05T21:42:18.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.046+0000 7f5fd3fff700 1 -- 172.21.15.121:0/864045737 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5fdc02cd80 con 0x7f5fc806cce0 2024-04-05T21:42:18.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.048+0000 7f5fe1520700 1 -- 172.21.15.121:0/864045737 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fc806cce0 msgr2=0x7f5fc806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.048+0000 7f5fe1520700 1 --2- 172.21.15.121:0/864045737 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fc806cce0 0x7f5fc806f1a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc00b5c0 tx=0x7f5fcc005fd0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.048+0000 7f5fe1520700 1 -- 172.21.15.121:0/864045737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fdc0ff530 msgr2=0x7f5fdc197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.049+0000 7f5fe1520700 1 --2- 172.21.15.121:0/864045737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fdc0ff530 0x7f5fdc197530 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400d8d0 tx=0x7f5fc400dc90 comp rx=0 tx=0).stop 2024-04-05T21:42:18.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.049+0000 7f5fe1520700 1 -- 172.21.15.121:0/864045737 shutdown_connections 2024-04-05T21:42:18.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.049+0000 7f5fe1520700 1 --2- 172.21.15.121:0/864045737 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5fc806cce0 0x7f5fc806f1a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.049+0000 7f5fe1520700 1 --2- 172.21.15.121:0/864045737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fdc0ff530 0x7f5fdc197530 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.049+0000 7f5fe1520700 1 --2- 172.21.15.121:0/864045737 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5fdc0ffee0 0x7f5fdc197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.049+0000 7f5fe1520700 1 -- 172.21.15.121:0/864045737 >> 172.21.15.121:0/864045737 conn(0x7f5fdc074df0 msgr2=0x7f5fdc0fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:18.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.049+0000 7f5fe1520700 1 -- 172.21.15.121:0/864045737 shutdown_connections 2024-04-05T21:42:18.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.049+0000 7f5fe1520700 1 -- 172.21.15.121:0/864045737 wait complete. 2024-04-05T21:42:18.060 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:42:18.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.153+0000 7fe348d89700 1 -- 172.21.15.121:0/2023167718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe344068670 msgr2=0x7fe344068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.153+0000 7fe348d89700 1 --2- 172.21.15.121:0/2023167718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe344068670 0x7fe344068a50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe334009910 tx=0x7fe334009c20 comp rx=0 tx=0).stop 2024-04-05T21:42:18.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.153+0000 7fe348d89700 1 -- 172.21.15.121:0/2023167718 shutdown_connections 2024-04-05T21:42:18.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.153+0000 7fe348d89700 1 --2- 172.21.15.121:0/2023167718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe344069020 0x7fe344105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.153+0000 7fe348d89700 1 --2- 172.21.15.121:0/2023167718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe344068670 0x7fe344068a50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.153+0000 7fe348d89700 1 -- 172.21.15.121:0/2023167718 >> 172.21.15.121:0/2023167718 conn(0x7fe3440755a0 msgr2=0x7fe3440759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:18.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.153+0000 7fe348d89700 1 -- 172.21.15.121:0/2023167718 shutdown_connections 2024-04-05T21:42:18.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.153+0000 7fe348d89700 1 -- 172.21.15.121:0/2023167718 wait complete. 2024-04-05T21:42:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.154+0000 7fe348d89700 1 Processor -- start 2024-04-05T21:42:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.154+0000 7fe348d89700 1 -- start start 2024-04-05T21:42:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.154+0000 7fe348d89700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe344068670 0x7fe34419c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.154+0000 7fe348d89700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe344069020 0x7fe34419cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.154+0000 7fe348d89700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34419d170 con 0x7fe344068670 2024-04-05T21:42:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.154+0000 7fe348d89700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe344196620 con 0x7fe344069020 2024-04-05T21:42:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.154+0000 7fe34259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe344068670 0x7fe34419c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.154+0000 7fe34259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe344068670 0x7fe34419c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47778/0 (socket says 172.21.15.121:47778) 2024-04-05T21:42:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.154+0000 7fe34259c700 1 -- 172.21.15.121:0/1878636204 learned_addr learned my addr 172.21.15.121:0/1878636204 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.154+0000 7fe341d9b700 1 --2- 172.21.15.121:0/1878636204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe344069020 0x7fe34419cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.155+0000 7fe34259c700 1 -- 172.21.15.121:0/1878636204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe344069020 msgr2=0x7fe34419cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.155+0000 7fe34259c700 1 --2- 172.21.15.121:0/1878636204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe344069020 0x7fe34419cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.155+0000 7fe34259c700 1 -- 172.21.15.121:0/1878636204 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3340095f0 con 0x7fe344068670 2024-04-05T21:42:18.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.155+0000 7fe341d9b700 1 --2- 172.21.15.121:0/1878636204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe344069020 0x7fe34419cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:42:18.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.155+0000 7fe34259c700 1 --2- 172.21.15.121:0/1878636204 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe344068670 0x7fe34419c5a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe334006010 tx=0x7fe33400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:18.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.155+0000 7fe3337fe700 1 -- 172.21.15.121:0/1878636204 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe33401d070 con 0x7fe344068670 2024-04-05T21:42:18.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.155+0000 7fe3337fe700 1 -- 172.21.15.121:0/1878636204 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe33400f460 con 0x7fe344068670 2024-04-05T21:42:18.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.155+0000 7fe348d89700 1 -- 172.21.15.121:0/1878636204 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3441968a0 con 0x7fe344068670 2024-04-05T21:42:18.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.155+0000 7fe3337fe700 1 -- 172.21.15.121:0/1878636204 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe334021640 con 0x7fe344068670 2024-04-05T21:42:18.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.155+0000 7fe348d89700 1 -- 172.21.15.121:0/1878636204 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe344196d90 con 0x7fe344068670 2024-04-05T21:42:18.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.156+0000 7fe348d89700 1 -- 172.21.15.121:0/1878636204 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe344109130 con 0x7fe344068670 2024-04-05T21:42:18.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.156+0000 7fe3337fe700 1 -- 172.21.15.121:0/1878636204 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe33400fa60 con 0x7fe344068670 2024-04-05T21:42:18.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.157+0000 7fe3337fe700 1 --2- 172.21.15.121:0/1878636204 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe32c06cce0 0x7fe32c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:18.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.157+0000 7fe3337fe700 1 -- 172.21.15.121:0/1878636204 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe33408d880 con 0x7fe344068670 2024-04-05T21:42:18.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.157+0000 7fe341d9b700 1 --2- 172.21.15.121:0/1878636204 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe32c06cce0 0x7fe32c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:18.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.157+0000 7fe341d9b700 1 --2- 172.21.15.121:0/1878636204 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe32c06cce0 0x7fe32c06f1a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe344107730 tx=0x7fe338008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:18.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.161+0000 7fe3337fe700 1 -- 172.21.15.121:0/1878636204 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe33405b380 con 0x7fe344068670 2024-04-05T21:42:18.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.373+0000 7fe348d89700 1 -- 172.21.15.121:0/1878636204 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe344196fb0 con 0x7fe32c06cce0 2024-04-05T21:42:18.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.374+0000 7fe3337fe700 1 -- 172.21.15.121:0/1878636204 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe344196fb0 con 0x7fe32c06cce0 2024-04-05T21:42:18.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.376+0000 7fe348d89700 1 -- 172.21.15.121:0/1878636204 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe32c06cce0 msgr2=0x7fe32c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.376+0000 7fe348d89700 1 --2- 172.21.15.121:0/1878636204 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe32c06cce0 0x7fe32c06f1a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe344107730 tx=0x7fe338008040 comp rx=0 tx=0).stop 2024-04-05T21:42:18.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.376+0000 7fe348d89700 1 -- 172.21.15.121:0/1878636204 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe344068670 msgr2=0x7fe34419c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.376+0000 7fe348d89700 1 --2- 172.21.15.121:0/1878636204 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe344068670 0x7fe34419c5a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe334006010 tx=0x7fe33400be20 comp rx=0 tx=0).stop 2024-04-05T21:42:18.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.376+0000 7fe348d89700 1 -- 172.21.15.121:0/1878636204 shutdown_connections 2024-04-05T21:42:18.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.376+0000 7fe348d89700 1 --2- 172.21.15.121:0/1878636204 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe32c06cce0 0x7fe32c06f1a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.376+0000 7fe348d89700 1 --2- 172.21.15.121:0/1878636204 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe344068670 0x7fe34419c5a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.376+0000 7fe348d89700 1 --2- 172.21.15.121:0/1878636204 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe344069020 0x7fe34419cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.376+0000 7fe348d89700 1 -- 172.21.15.121:0/1878636204 >> 172.21.15.121:0/1878636204 conn(0x7fe3440755a0 msgr2=0x7fe3440fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:18.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.376+0000 7fe348d89700 1 -- 172.21.15.121:0/1878636204 shutdown_connections 2024-04-05T21:42:18.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.376+0000 7fe348d89700 1 -- 172.21.15.121:0/1878636204 wait complete. 2024-04-05T21:42:18.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.479+0000 7f3e896a2700 1 -- 172.21.15.121:0/1199323071 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e84105b50 msgr2=0x7f3e84105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.479+0000 7f3e896a2700 1 --2- 172.21.15.121:0/1199323071 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e84105b50 0x7f3e84105f30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009910 tx=0x7f3e78009c20 comp rx=0 tx=0).stop 2024-04-05T21:42:18.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.479+0000 7f3e896a2700 1 -- 172.21.15.121:0/1199323071 shutdown_connections 2024-04-05T21:42:18.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.479+0000 7f3e896a2700 1 --2- 172.21.15.121:0/1199323071 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e840fef70 0x7f3e840ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.479+0000 7f3e896a2700 1 --2- 172.21.15.121:0/1199323071 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e84105b50 0x7f3e84105f30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.479+0000 7f3e896a2700 1 -- 172.21.15.121:0/1199323071 >> 172.21.15.121:0/1199323071 conn(0x7f3e84075e30 msgr2=0x7f3e840fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:18.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.479+0000 7f3e896a2700 1 -- 172.21.15.121:0/1199323071 shutdown_connections 2024-04-05T21:42:18.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.479+0000 7f3e896a2700 1 -- 172.21.15.121:0/1199323071 wait complete. 2024-04-05T21:42:18.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.480+0000 7f3e896a2700 1 Processor -- start 2024-04-05T21:42:18.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.480+0000 7f3e896a2700 1 -- start start 2024-04-05T21:42:18.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.480+0000 7f3e896a2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e840fef70 0x7f3e84100150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.480+0000 7f3e896a2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e84105b50 0x7f3e84100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.480+0000 7f3e896a2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e84100bd0 con 0x7f3e84105b50 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.480+0000 7f3e896a2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e84100d10 con 0x7f3e840fef70 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.480+0000 7f3e827fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e84105b50 0x7f3e84100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.480+0000 7f3e827fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e84105b50 0x7f3e84100690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47798/0 (socket says 172.21.15.121:47798) 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.480+0000 7f3e827fc700 1 -- 172.21.15.121:0/2029929917 learned_addr learned my addr 172.21.15.121:0/2029929917 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.481+0000 7f3e82ffd700 1 --2- 172.21.15.121:0/2029929917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e840fef70 0x7f3e84100150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.481+0000 7f3e827fc700 1 -- 172.21.15.121:0/2029929917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e840fef70 msgr2=0x7f3e84100150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.481+0000 7f3e827fc700 1 --2- 172.21.15.121:0/2029929917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e840fef70 0x7f3e84100150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.481+0000 7f3e827fc700 1 -- 172.21.15.121:0/2029929917 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e780095f0 con 0x7f3e84105b50 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.481+0000 7f3e827fc700 1 --2- 172.21.15.121:0/2029929917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e84105b50 0x7f3e84100690 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400d900 tx=0x7f3e7400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.481+0000 7f3e6bfff700 1 -- 172.21.15.121:0/2029929917 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e740041f0 con 0x7f3e84105b50 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.481+0000 7f3e896a2700 1 -- 172.21.15.121:0/2029929917 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e841a2910 con 0x7f3e84105b50 2024-04-05T21:42:18.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.481+0000 7f3e6bfff700 1 -- 172.21.15.121:0/2029929917 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e74004cd0 con 0x7f3e84105b50 2024-04-05T21:42:18.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.481+0000 7f3e6bfff700 1 -- 172.21.15.121:0/2029929917 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e740056d0 con 0x7f3e84105b50 2024-04-05T21:42:18.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.481+0000 7f3e896a2700 1 -- 172.21.15.121:0/2029929917 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e841a2d20 con 0x7f3e84105b50 2024-04-05T21:42:18.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.482+0000 7f3e896a2700 1 -- 172.21.15.121:0/2029929917 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e84109140 con 0x7f3e84105b50 2024-04-05T21:42:18.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.482+0000 7f3e6bfff700 1 -- 172.21.15.121:0/2029929917 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3e74004e40 con 0x7f3e84105b50 2024-04-05T21:42:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.483+0000 7f3e6bfff700 1 --2- 172.21.15.121:0/2029929917 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3e6c06cc80 0x7f3e6c06f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.483+0000 7f3e6bfff700 1 -- 172.21.15.121:0/2029929917 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3e7408c710 con 0x7f3e84105b50 2024-04-05T21:42:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.483+0000 7f3e82ffd700 1 --2- 172.21.15.121:0/2029929917 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3e6c06cc80 0x7f3e6c06f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:18.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.483+0000 7f3e82ffd700 1 --2- 172.21.15.121:0/2029929917 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3e6c06cc80 0x7f3e6c06f140 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3e78000c00 tx=0x7f3e78005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:18.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.487+0000 7f3e6bfff700 1 -- 172.21.15.121:0/2029929917 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3e74056780 con 0x7f3e84105b50 2024-04-05T21:42:18.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.691+0000 7f3e896a2700 1 -- 172.21.15.121:0/2029929917 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e8402cd30 con 0x7f3e6c06cc80 2024-04-05T21:42:18.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.701+0000 7f3e6bfff700 1 -- 172.21.15.121:0/2029929917 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f3e8402cd30 con 0x7f3e6c06cc80 2024-04-05T21:42:18.702 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:42:18.702 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (13m) 7m ago 24m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:42:18.702 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (24m) 7m ago 24m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:42:18.702 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (23m) 6m ago 23m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:42:18.702 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (11m) 7m ago 24m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:42:18.702 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (11m) 6m ago 23m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:42:18.702 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (12m) 7m ago 24m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:42:18.702 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (7m) 7m ago 18m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:42:18.702 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (7m) 7m ago 18m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (6m) 6m ago 18m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (18m) 6m ago 18m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (15m) 7m ago 26m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (14m) 6m ago 23m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (11m) 7m ago 26m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (11m) 6m ago 23m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (13m) 7m ago 24m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (13m) 6m ago 23m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (10m) 7m ago 22m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (10m) 7m ago 22m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (9m) 7m ago 21m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (9m) 6m ago 21m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (8m) 6m ago 20m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (7m) 6m ago 20m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (13m) 7m ago 24m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:42:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.703+0000 7f3e896a2700 1 -- 172.21.15.121:0/2029929917 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3e6c06cc80 msgr2=0x7f3e6c06f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.703+0000 7f3e896a2700 1 --2- 172.21.15.121:0/2029929917 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3e6c06cc80 0x7f3e6c06f140 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3e78000c00 tx=0x7f3e78005fb0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.703+0000 7f3e896a2700 1 -- 172.21.15.121:0/2029929917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e84105b50 msgr2=0x7f3e84100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.703+0000 7f3e896a2700 1 --2- 172.21.15.121:0/2029929917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e84105b50 0x7f3e84100690 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3e7400d900 tx=0x7f3e7400dcc0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.703+0000 7f3e896a2700 1 -- 172.21.15.121:0/2029929917 shutdown_connections 2024-04-05T21:42:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.703+0000 7f3e896a2700 1 --2- 172.21.15.121:0/2029929917 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3e6c06cc80 0x7f3e6c06f140 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.703+0000 7f3e896a2700 1 --2- 172.21.15.121:0/2029929917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e840fef70 0x7f3e84100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.703+0000 7f3e896a2700 1 --2- 172.21.15.121:0/2029929917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e84105b50 0x7f3e84100690 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.703+0000 7f3e896a2700 1 -- 172.21.15.121:0/2029929917 >> 172.21.15.121:0/2029929917 conn(0x7f3e84075e30 msgr2=0x7f3e84104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.703+0000 7f3e896a2700 1 -- 172.21.15.121:0/2029929917 shutdown_connections 2024-04-05T21:42:18.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.703+0000 7f3e896a2700 1 -- 172.21.15.121:0/2029929917 wait complete. 2024-04-05T21:42:18.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.805+0000 7f97d4a78700 1 -- 172.21.15.121:0/3160091798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d0100dc0 msgr2=0x7f97d01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.805+0000 7f97d4a78700 1 --2- 172.21.15.121:0/3160091798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d0100dc0 0x7f97d01011a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009910 tx=0x7f97c4009c20 comp rx=0 tx=0).stop 2024-04-05T21:42:18.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.805+0000 7f97d4a78700 1 -- 172.21.15.121:0/3160091798 shutdown_connections 2024-04-05T21:42:18.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.805+0000 7f97d4a78700 1 --2- 172.21.15.121:0/3160091798 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97d0101770 0x7f97d01057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.805+0000 7f97d4a78700 1 --2- 172.21.15.121:0/3160091798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d0100dc0 0x7f97d01011a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.805+0000 7f97d4a78700 1 -- 172.21.15.121:0/3160091798 >> 172.21.15.121:0/3160091798 conn(0x7f97d00fc670 msgr2=0x7f97d00fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:18.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.805+0000 7f97d4a78700 1 -- 172.21.15.121:0/3160091798 shutdown_connections 2024-04-05T21:42:18.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.805+0000 7f97d4a78700 1 -- 172.21.15.121:0/3160091798 wait complete. 2024-04-05T21:42:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.806+0000 7f97d4a78700 1 Processor -- start 2024-04-05T21:42:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.806+0000 7f97d4a78700 1 -- start start 2024-04-05T21:42:18.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.806+0000 7f97d4a78700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97d0100dc0 0x7f97d019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.806+0000 7f97d4a78700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d0101770 0x7f97d019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.806+0000 7f97d4a78700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d019d0d0 con 0x7f97d0101770 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.806+0000 7f97d4a78700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d0196580 con 0x7f97d0100dc0 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.806+0000 7f97cdd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d0101770 0x7f97d019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.806+0000 7f97cdd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d0101770 0x7f97d019ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47818/0 (socket says 172.21.15.121:47818) 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.806+0000 7f97cdd9b700 1 -- 172.21.15.121:0/4089742181 learned_addr learned my addr 172.21.15.121:0/4089742181 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.806+0000 7f97ce59c700 1 --2- 172.21.15.121:0/4089742181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97d0100dc0 0x7f97d019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.807+0000 7f97cdd9b700 1 -- 172.21.15.121:0/4089742181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97d0100dc0 msgr2=0x7f97d019c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.807+0000 7f97cdd9b700 1 --2- 172.21.15.121:0/4089742181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97d0100dc0 0x7f97d019c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.807+0000 7f97cdd9b700 1 -- 172.21.15.121:0/4089742181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c40095f0 con 0x7f97d0101770 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.807+0000 7f97cdd9b700 1 --2- 172.21.15.121:0/4089742181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d0101770 0x7f97d019ca40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f97c000d8d0 tx=0x7f97c000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.807+0000 7f97bf7fe700 1 -- 172.21.15.121:0/4089742181 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97c0004a00 con 0x7f97d0101770 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.807+0000 7f97bf7fe700 1 -- 172.21.15.121:0/4089742181 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97c0005020 con 0x7f97d0101770 2024-04-05T21:42:18.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.807+0000 7f97d4a78700 1 -- 172.21.15.121:0/4089742181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d0196860 con 0x7f97d0101770 2024-04-05T21:42:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.807+0000 7f97bf7fe700 1 -- 172.21.15.121:0/4089742181 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97c000fb80 con 0x7f97d0101770 2024-04-05T21:42:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.807+0000 7f97d4a78700 1 -- 172.21.15.121:0/4089742181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d0196db0 con 0x7f97d0101770 2024-04-05T21:42:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.808+0000 7f97d4a78700 1 -- 172.21.15.121:0/4089742181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d0109160 con 0x7f97d0101770 2024-04-05T21:42:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.808+0000 7f97bf7fe700 1 -- 172.21.15.121:0/4089742181 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f97c000fce0 con 0x7f97d0101770 2024-04-05T21:42:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.809+0000 7f97bf7fe700 1 --2- 172.21.15.121:0/4089742181 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f97b806cce0 0x7f97b806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.809+0000 7f97bf7fe700 1 -- 172.21.15.121:0/4089742181 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f97c008bdf0 con 0x7f97d0101770 2024-04-05T21:42:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.809+0000 7f97ce59c700 1 --2- 172.21.15.121:0/4089742181 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f97b806cce0 0x7f97b806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.809+0000 7f97ce59c700 1 --2- 172.21.15.121:0/4089742181 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f97b806cce0 0x7f97b806f1a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f97c4006010 tx=0x7f97c401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:18.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:18.813+0000 7f97bf7fe700 1 -- 172.21.15.121:0/4089742181 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f97c0059970 con 0x7f97d0101770 2024-04-05T21:42:19.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.085+0000 7f97d4a78700 1 -- 172.21.15.121:0/4089742181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f97d0197020 con 0x7f97d0101770 2024-04-05T21:42:19.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.086+0000 7f97bf7fe700 1 -- 172.21.15.121:0/4089742181 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f97c0017340 con 0x7f97d0101770 2024-04-05T21:42:19.087 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:42:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:42:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:42:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:42:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:42:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:42:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:42:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:42:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:42:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:42:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:42:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:42:19.087 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:42:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:42:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:42:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:42:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:42:19.088 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:42:19.088 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:42:19.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.088+0000 7f97d4a78700 1 -- 172.21.15.121:0/4089742181 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f97b806cce0 msgr2=0x7f97b806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:19.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.088+0000 7f97d4a78700 1 --2- 172.21.15.121:0/4089742181 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f97b806cce0 0x7f97b806f1a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f97c4006010 tx=0x7f97c401a040 comp rx=0 tx=0).stop 2024-04-05T21:42:19.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.088+0000 7f97d4a78700 1 -- 172.21.15.121:0/4089742181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d0101770 msgr2=0x7f97d019ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:19.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.088+0000 7f97d4a78700 1 --2- 172.21.15.121:0/4089742181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d0101770 0x7f97d019ca40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f97c000d8d0 tx=0x7f97c000dbe0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.088+0000 7f97d4a78700 1 -- 172.21.15.121:0/4089742181 shutdown_connections 2024-04-05T21:42:19.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.088+0000 7f97d4a78700 1 --2- 172.21.15.121:0/4089742181 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f97b806cce0 0x7f97b806f1a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.088+0000 7f97d4a78700 1 --2- 172.21.15.121:0/4089742181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f97d0100dc0 0x7f97d019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.088+0000 7f97d4a78700 1 --2- 172.21.15.121:0/4089742181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97d0101770 0x7f97d019ca40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.088+0000 7f97d4a78700 1 -- 172.21.15.121:0/4089742181 >> 172.21.15.121:0/4089742181 conn(0x7f97d00fc670 msgr2=0x7f97d00fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:19.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.088+0000 7f97d4a78700 1 -- 172.21.15.121:0/4089742181 shutdown_connections 2024-04-05T21:42:19.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.088+0000 7f97d4a78700 1 -- 172.21.15.121:0/4089742181 wait complete. 2024-04-05T21:42:19.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.190+0000 7ff754991700 1 -- 172.21.15.121:0/1613456015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff750105a40 msgr2=0x7ff750105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:19.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.190+0000 7ff754991700 1 --2- 172.21.15.121:0/1613456015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff750105a40 0x7ff750105e20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff744009910 tx=0x7ff744009c20 comp rx=0 tx=0).stop 2024-04-05T21:42:19.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.190+0000 7ff754991700 1 -- 172.21.15.121:0/1613456015 shutdown_connections 2024-04-05T21:42:19.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.190+0000 7ff754991700 1 --2- 172.21.15.121:0/1613456015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7500ffbd0 0x7ff750100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.190+0000 7ff754991700 1 --2- 172.21.15.121:0/1613456015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff750105a40 0x7ff750105e20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.190+0000 7ff754991700 1 -- 172.21.15.121:0/1613456015 >> 172.21.15.121:0/1613456015 conn(0x7ff7500755a0 msgr2=0x7ff7500759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:19.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.190+0000 7ff754991700 1 -- 172.21.15.121:0/1613456015 shutdown_connections 2024-04-05T21:42:19.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.190+0000 7ff754991700 1 -- 172.21.15.121:0/1613456015 wait complete. 2024-04-05T21:42:19.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff754991700 1 Processor -- start 2024-04-05T21:42:19.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff754991700 1 -- start start 2024-04-05T21:42:19.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff754991700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7500ffbd0 0x7ff750196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:19.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff754991700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff750105a40 0x7ff750196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:19.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff754991700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff750197260 con 0x7ff750105a40 2024-04-05T21:42:19.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff754991700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75019aff0 con 0x7ff7500ffbd0 2024-04-05T21:42:19.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff74dd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff750105a40 0x7ff750196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:19.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff74dd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff750105a40 0x7ff750196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47830/0 (socket says 172.21.15.121:47830) 2024-04-05T21:42:19.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff74dd9b700 1 -- 172.21.15.121:0/560797025 learned_addr learned my addr 172.21.15.121:0/560797025 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:19.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff74e59c700 1 --2- 172.21.15.121:0/560797025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7500ffbd0 0x7ff750196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:19.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff74dd9b700 1 -- 172.21.15.121:0/560797025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7500ffbd0 msgr2=0x7ff750196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:19.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff74dd9b700 1 --2- 172.21.15.121:0/560797025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7500ffbd0 0x7ff750196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff74dd9b700 1 -- 172.21.15.121:0/560797025 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7440095f0 con 0x7ff750105a40 2024-04-05T21:42:19.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff74dd9b700 1 --2- 172.21.15.121:0/560797025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff750105a40 0x7ff750196b80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff74000d8d0 tx=0x7ff74000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:19.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff73f7fe700 1 -- 172.21.15.121:0/560797025 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff740009940 con 0x7ff750105a40 2024-04-05T21:42:19.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff754991700 1 -- 172.21.15.121:0/560797025 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff75019b2d0 con 0x7ff750105a40 2024-04-05T21:42:19.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.191+0000 7ff73f7fe700 1 -- 172.21.15.121:0/560797025 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff740010460 con 0x7ff750105a40 2024-04-05T21:42:19.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.192+0000 7ff73f7fe700 1 -- 172.21.15.121:0/560797025 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff74000f5f0 con 0x7ff750105a40 2024-04-05T21:42:19.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.192+0000 7ff754991700 1 -- 172.21.15.121:0/560797025 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff75019b820 con 0x7ff750105a40 2024-04-05T21:42:19.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.192+0000 7ff754991700 1 -- 172.21.15.121:0/560797025 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff750109050 con 0x7ff750105a40 2024-04-05T21:42:19.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.193+0000 7ff73f7fe700 1 -- 172.21.15.121:0/560797025 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff740009aa0 con 0x7ff750105a40 2024-04-05T21:42:19.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.193+0000 7ff73f7fe700 1 --2- 172.21.15.121:0/560797025 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff73806cce0 0x7ff73806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:19.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.193+0000 7ff73f7fe700 1 -- 172.21.15.121:0/560797025 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff74008c5a0 con 0x7ff750105a40 2024-04-05T21:42:19.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.193+0000 7ff74e59c700 1 --2- 172.21.15.121:0/560797025 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff73806cce0 0x7ff73806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:19.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.194+0000 7ff74e59c700 1 --2- 172.21.15.121:0/560797025 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff73806cce0 0x7ff73806f1a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff744006010 tx=0x7ff74401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:19.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.198+0000 7ff73f7fe700 1 -- 172.21.15.121:0/560797025 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff740056610 con 0x7ff750105a40 2024-04-05T21:42:19.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.436+0000 7ff754991700 1 -- 172.21.15.121:0/560797025 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff75004ea90 con 0x7ff750105a40 2024-04-05T21:42:19.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.439+0000 7ff73f7fe700 1 -- 172.21.15.121:0/560797025 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7ff740059c30 con 0x7ff750105a40 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:42:19.440 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:42:19.441 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:42:19.441 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:42:19.441 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:42:19.441 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:42:19.441 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:42:19.441 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:42:19.441 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:42:19.441 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:42:19.441 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:42:19.441 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:42:19.441 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:42:19.442 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:42:19.443 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:42:19.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.441+0000 7ff754991700 1 -- 172.21.15.121:0/560797025 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff73806cce0 msgr2=0x7ff73806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:19.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.441+0000 7ff754991700 1 --2- 172.21.15.121:0/560797025 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff73806cce0 0x7ff73806f1a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff744006010 tx=0x7ff74401a040 comp rx=0 tx=0).stop 2024-04-05T21:42:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.441+0000 7ff754991700 1 -- 172.21.15.121:0/560797025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff750105a40 msgr2=0x7ff750196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.441+0000 7ff754991700 1 --2- 172.21.15.121:0/560797025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff750105a40 0x7ff750196b80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff74000d8d0 tx=0x7ff74000dc90 comp rx=0 tx=0).stop 2024-04-05T21:42:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.441+0000 7ff754991700 1 -- 172.21.15.121:0/560797025 shutdown_connections 2024-04-05T21:42:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.441+0000 7ff754991700 1 --2- 172.21.15.121:0/560797025 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff73806cce0 0x7ff73806f1a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.441+0000 7ff754991700 1 --2- 172.21.15.121:0/560797025 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff7500ffbd0 0x7ff750196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.441+0000 7ff754991700 1 --2- 172.21.15.121:0/560797025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff750105a40 0x7ff750196b80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.441+0000 7ff754991700 1 -- 172.21.15.121:0/560797025 >> 172.21.15.121:0/560797025 conn(0x7ff7500755a0 msgr2=0x7ff750068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.441+0000 7ff754991700 1 -- 172.21.15.121:0/560797025 shutdown_connections 2024-04-05T21:42:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.441+0000 7ff754991700 1 -- 172.21.15.121:0/560797025 wait complete. 2024-04-05T21:42:19.444 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:42:19.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.550+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/4110579571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbec1026a0 msgr2=0x7fcbec10ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:19.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.550+0000 7fcbf0b4b700 1 --2- 172.21.15.121:0/4110579571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbec1026a0 0x7fcbec10ab90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009940 tx=0x7fcbe0009c50 comp rx=0 tx=0).stop 2024-04-05T21:42:19.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.550+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/4110579571 shutdown_connections 2024-04-05T21:42:19.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.550+0000 7fcbf0b4b700 1 --2- 172.21.15.121:0/4110579571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbec1026a0 0x7fcbec10ab90 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.550+0000 7fcbf0b4b700 1 --2- 172.21.15.121:0/4110579571 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbec101d80 0x7fcbec102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.550+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/4110579571 >> 172.21.15.121:0/4110579571 conn(0x7fcbec0fb570 msgr2=0x7fcbec0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:19.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.551+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/4110579571 shutdown_connections 2024-04-05T21:42:19.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.551+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/4110579571 wait complete. 2024-04-05T21:42:19.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.551+0000 7fcbf0b4b700 1 Processor -- start 2024-04-05T21:42:19.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.551+0000 7fcbf0b4b700 1 -- start start 2024-04-05T21:42:19.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.551+0000 7fcbf0b4b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbec101d80 0x7fcbec0ff9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:19.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.551+0000 7fcbf0b4b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbec1026a0 0x7fcbec0fff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:19.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.551+0000 7fcbf0b4b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec100460 con 0x7fcbec101d80 2024-04-05T21:42:19.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.551+0000 7fcbf0b4b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec1005a0 con 0x7fcbec1026a0 2024-04-05T21:42:19.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbea59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbec101d80 0x7fcbec0ff9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:19.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbea59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbec101d80 0x7fcbec0ff9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47844/0 (socket says 172.21.15.121:47844) 2024-04-05T21:42:19.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbea59c700 1 -- 172.21.15.121:0/3907768759 learned_addr learned my addr 172.21.15.121:0/3907768759 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:19.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbe9d9b700 1 --2- 172.21.15.121:0/3907768759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbec1026a0 0x7fcbec0fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:19.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbea59c700 1 -- 172.21.15.121:0/3907768759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbec1026a0 msgr2=0x7fcbec0fff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:19.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbea59c700 1 --2- 172.21.15.121:0/3907768759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbec1026a0 0x7fcbec0fff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbea59c700 1 -- 172.21.15.121:0/3907768759 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe00095f0 con 0x7fcbec101d80 2024-04-05T21:42:19.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbe9d9b700 1 --2- 172.21.15.121:0/3907768759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbec1026a0 0x7fcbec0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:42:19.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbea59c700 1 --2- 172.21.15.121:0/3907768759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbec101d80 0x7fcbec0ff9e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00d8d0 tx=0x7fcbdc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:19.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbdb7fe700 1 -- 172.21.15.121:0/3907768759 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbdc009940 con 0x7fcbec101d80 2024-04-05T21:42:19.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/3907768759 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbec1a2880 con 0x7fcbec101d80 2024-04-05T21:42:19.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbdb7fe700 1 -- 172.21.15.121:0/3907768759 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbdc010460 con 0x7fcbec101d80 2024-04-05T21:42:19.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbdb7fe700 1 -- 172.21.15.121:0/3907768759 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbdc00f5f0 con 0x7fcbec101d80 2024-04-05T21:42:19.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.552+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/3907768759 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbec1a2c30 con 0x7fcbec101d80 2024-04-05T21:42:19.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.553+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/3907768759 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbec108290 con 0x7fcbec101d80 2024-04-05T21:42:19.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.554+0000 7fcbdb7fe700 1 -- 172.21.15.121:0/3907768759 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcbdc0105d0 con 0x7fcbec101d80 2024-04-05T21:42:19.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.554+0000 7fcbdb7fe700 1 --2- 172.21.15.121:0/3907768759 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcbd406cce0 0x7fcbd406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:19.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.554+0000 7fcbdb7fe700 1 -- 172.21.15.121:0/3907768759 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fcbdc08c5e0 con 0x7fcbec101d80 2024-04-05T21:42:19.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.554+0000 7fcbe9d9b700 1 --2- 172.21.15.121:0/3907768759 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcbd406cce0 0x7fcbd406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:19.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.555+0000 7fcbe9d9b700 1 --2- 172.21.15.121:0/3907768759 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcbd406cce0 0x7fcbd406f1a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009910 tx=0x7fcbe0005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:19.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.559+0000 7fcbdb7fe700 1 -- 172.21.15.121:0/3907768759 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcbdc056650 con 0x7fcbec101d80 2024-04-05T21:42:19.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:19 smithi121 ceph-mon[145859]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:19 smithi121 ceph-mon[145859]: pgmap v482: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:19 smithi121 ceph-mon[145859]: from='client.34738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:19 smithi121 ceph-mon[145859]: from='client.34742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:19 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4089742181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:42:19.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.769+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/3907768759 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbec0611d0 con 0x7fcbd406cce0 2024-04-05T21:42:19.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.770+0000 7fcbdb7fe700 1 -- 172.21.15.121:0/3907768759 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcbec0611d0 con 0x7fcbd406cce0 2024-04-05T21:42:19.771 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:42:19.771 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:42:19.771 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:42:19.771 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:42:19.771 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:42:19.771 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:42:19.771 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:42:19.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:42:19.771 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:42:19.771 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:42:19.771 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:42:19.771 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:42:19.772 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:42:19.772 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:42:19.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.772+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/3907768759 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcbd406cce0 msgr2=0x7fcbd406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:19.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.772+0000 7fcbf0b4b700 1 --2- 172.21.15.121:0/3907768759 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcbd406cce0 0x7fcbd406f1a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009910 tx=0x7fcbe0005960 comp rx=0 tx=0).stop 2024-04-05T21:42:19.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.772+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/3907768759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbec101d80 msgr2=0x7fcbec0ff9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:19.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.772+0000 7fcbf0b4b700 1 --2- 172.21.15.121:0/3907768759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbec101d80 0x7fcbec0ff9e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00d8d0 tx=0x7fcbdc00dc90 comp rx=0 tx=0).stop 2024-04-05T21:42:19.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.772+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/3907768759 shutdown_connections 2024-04-05T21:42:19.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.772+0000 7fcbf0b4b700 1 --2- 172.21.15.121:0/3907768759 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcbd406cce0 0x7fcbd406f1a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.772+0000 7fcbf0b4b700 1 --2- 172.21.15.121:0/3907768759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbec101d80 0x7fcbec0ff9e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.772+0000 7fcbf0b4b700 1 --2- 172.21.15.121:0/3907768759 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbec1026a0 0x7fcbec0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.772+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/3907768759 >> 172.21.15.121:0/3907768759 conn(0x7fcbec0fb570 msgr2=0x7fcbec1053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:19.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.772+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/3907768759 shutdown_connections 2024-04-05T21:42:19.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.772+0000 7fcbf0b4b700 1 -- 172.21.15.121:0/3907768759 wait complete. 2024-04-05T21:42:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:19 smithi188 ceph-mon[107793]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:19 smithi188 ceph-mon[107793]: pgmap v482: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:19 smithi188 ceph-mon[107793]: from='client.34738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:19 smithi188 ceph-mon[107793]: from='client.34742 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:19 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4089742181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:42:19.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.875+0000 7fbd68917700 1 -- 172.21.15.121:0/3608570286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd64105a40 msgr2=0x7fbd64105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:19.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.875+0000 7fbd68917700 1 --2- 172.21.15.121:0/3608570286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd64105a40 0x7fbd64105e20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00b0a0 tx=0x7fbd5c00b3b0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.875+0000 7fbd68917700 1 -- 172.21.15.121:0/3608570286 shutdown_connections 2024-04-05T21:42:19.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.875+0000 7fbd68917700 1 --2- 172.21.15.121:0/3608570286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd640ffa40 0x7fbd640ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.875+0000 7fbd68917700 1 --2- 172.21.15.121:0/3608570286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd64105a40 0x7fbd64105e20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.875+0000 7fbd68917700 1 -- 172.21.15.121:0/3608570286 >> 172.21.15.121:0/3608570286 conn(0x7fbd640fb570 msgr2=0x7fbd640fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:19.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.875+0000 7fbd68917700 1 -- 172.21.15.121:0/3608570286 shutdown_connections 2024-04-05T21:42:19.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.875+0000 7fbd68917700 1 -- 172.21.15.121:0/3608570286 wait complete. 2024-04-05T21:42:19.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd68917700 1 Processor -- start 2024-04-05T21:42:19.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd68917700 1 -- start start 2024-04-05T21:42:19.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd68917700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd640ffa40 0x7fbd64196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:19.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd68917700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd64105a40 0x7fbd64196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:19.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd68917700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd64197200 con 0x7fbd640ffa40 2024-04-05T21:42:19.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd68917700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd64197340 con 0x7fbd64105a40 2024-04-05T21:42:19.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd6259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd640ffa40 0x7fbd64196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:19.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd6259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd640ffa40 0x7fbd64196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47852/0 (socket says 172.21.15.121:47852) 2024-04-05T21:42:19.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd6259c700 1 -- 172.21.15.121:0/129707975 learned_addr learned my addr 172.21.15.121:0/129707975 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd61d9b700 1 --2- 172.21.15.121:0/129707975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd64105a40 0x7fbd64196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd6259c700 1 -- 172.21.15.121:0/129707975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd64105a40 msgr2=0x7fbd64196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd6259c700 1 --2- 172.21.15.121:0/129707975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd64105a40 0x7fbd64196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd6259c700 1 -- 172.21.15.121:0/129707975 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c009d00 con 0x7fbd640ffa40 2024-04-05T21:42:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.876+0000 7fbd6259c700 1 --2- 172.21.15.121:0/129707975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd640ffa40 0x7fbd64196670 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00d010 tx=0x7fbd5c007be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.877+0000 7fbd537fe700 1 -- 172.21.15.121:0/129707975 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd5c00e040 con 0x7fbd640ffa40 2024-04-05T21:42:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.877+0000 7fbd68917700 1 -- 172.21.15.121:0/129707975 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6419b130 con 0x7fbd640ffa40 2024-04-05T21:42:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.877+0000 7fbd537fe700 1 -- 172.21.15.121:0/129707975 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd5c003e50 con 0x7fbd640ffa40 2024-04-05T21:42:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.877+0000 7fbd537fe700 1 -- 172.21.15.121:0/129707975 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd5c01da10 con 0x7fbd640ffa40 2024-04-05T21:42:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.877+0000 7fbd68917700 1 -- 172.21.15.121:0/129707975 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6419b620 con 0x7fbd640ffa40 2024-04-05T21:42:19.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.877+0000 7fbd68917700 1 -- 172.21.15.121:0/129707975 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6404ea90 con 0x7fbd640ffa40 2024-04-05T21:42:19.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.878+0000 7fbd537fe700 1 -- 172.21.15.121:0/129707975 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbd5c004210 con 0x7fbd640ffa40 2024-04-05T21:42:19.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.878+0000 7fbd537fe700 1 --2- 172.21.15.121:0/129707975 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd4c06cce0 0x7fbd4c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:19.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.879+0000 7fbd61d9b700 1 --2- 172.21.15.121:0/129707975 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd4c06cce0 0x7fbd4c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:19.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.879+0000 7fbd537fe700 1 -- 172.21.15.121:0/129707975 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbd5c08e050 con 0x7fbd640ffa40 2024-04-05T21:42:19.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.879+0000 7fbd61d9b700 1 --2- 172.21.15.121:0/129707975 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd4c06cce0 0x7fbd4c06f1a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbd58005ea0 tx=0x7fbd58005e30 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:19.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:19.883+0000 7fbd537fe700 1 -- 172.21.15.121:0/129707975 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbd5c0580c0 con 0x7fbd640ffa40 2024-04-05T21:42:20.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:20.156+0000 7fbd68917700 1 -- 172.21.15.121:0/129707975 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbd64108c30 con 0x7fbd640ffa40 2024-04-05T21:42:20.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:20.156+0000 7fbd537fe700 1 -- 172.21.15.121:0/129707975 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fbd5c05b6e0 con 0x7fbd640ffa40 2024-04-05T21:42:20.157 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:42:20.157 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:42:20.157 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:42:20.157 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:42:20.157 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:42:20.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:20.158+0000 7fbd68917700 1 -- 172.21.15.121:0/129707975 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd4c06cce0 msgr2=0x7fbd4c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:20.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:20.158+0000 7fbd68917700 1 --2- 172.21.15.121:0/129707975 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd4c06cce0 0x7fbd4c06f1a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbd58005ea0 tx=0x7fbd58005e30 comp rx=0 tx=0).stop 2024-04-05T21:42:20.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:20.158+0000 7fbd68917700 1 -- 172.21.15.121:0/129707975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd640ffa40 msgr2=0x7fbd64196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:20.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:20.158+0000 7fbd68917700 1 --2- 172.21.15.121:0/129707975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd640ffa40 0x7fbd64196670 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00d010 tx=0x7fbd5c007be0 comp rx=0 tx=0).stop 2024-04-05T21:42:20.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:20.158+0000 7fbd68917700 1 -- 172.21.15.121:0/129707975 shutdown_connections 2024-04-05T21:42:20.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:20.158+0000 7fbd68917700 1 --2- 172.21.15.121:0/129707975 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd4c06cce0 0x7fbd4c06f1a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:20.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:20.158+0000 7fbd68917700 1 --2- 172.21.15.121:0/129707975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd640ffa40 0x7fbd64196670 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:20.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:20.158+0000 7fbd68917700 1 --2- 172.21.15.121:0/129707975 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd64105a40 0x7fbd64196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:20.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:20.158+0000 7fbd68917700 1 -- 172.21.15.121:0/129707975 >> 172.21.15.121:0/129707975 conn(0x7fbd640fb570 msgr2=0x7fbd64068f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:20.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:20.159+0000 7fbd68917700 1 -- 172.21.15.121:0/129707975 shutdown_connections 2024-04-05T21:42:20.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:20.159+0000 7fbd68917700 1 -- 172.21.15.121:0/129707975 wait complete. 2024-04-05T21:42:20.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:20 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/560797025' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:42:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:20 smithi121 ceph-mon[145859]: from='client.34754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:20.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:20 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/129707975' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:42:20.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:20 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/560797025' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:42:20.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:20 smithi188 ceph-mon[107793]: from='client.34754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:20.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:20 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/129707975' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:42:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:21 smithi121 ceph-mon[145859]: pgmap v483: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:21 smithi188 ceph-mon[107793]: pgmap v483: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:23.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:23 smithi121 ceph-mon[145859]: pgmap v484: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:23 smithi188 ceph-mon[107793]: pgmap v484: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:25.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:25 smithi121 ceph-mon[145859]: pgmap v485: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:25.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:25 smithi188 ceph-mon[107793]: pgmap v485: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:27.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:27 smithi121 ceph-mon[145859]: pgmap v486: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:27.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:27 smithi188 ceph-mon[107793]: pgmap v486: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:29.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:29 smithi121 ceph-mon[145859]: pgmap v487: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:29 smithi188 ceph-mon[107793]: pgmap v487: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:42:30.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:42:31.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:31 smithi121 ceph-mon[145859]: pgmap v488: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:31.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:31 smithi188 ceph-mon[107793]: pgmap v488: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:33.741 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:33 smithi121 ceph-mon[145859]: pgmap v489: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:33 smithi188 ceph-mon[107793]: pgmap v489: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:35.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:35 smithi121 ceph-mon[145859]: pgmap v490: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:35.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:35 smithi188 ceph-mon[107793]: pgmap v490: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:37 smithi121 ceph-mon[145859]: pgmap v491: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:37.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:37 smithi188 ceph-mon[107793]: pgmap v491: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:39.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:39 smithi121 ceph-mon[145859]: pgmap v492: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:39.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:39 smithi188 ceph-mon[107793]: pgmap v492: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:41.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:41 smithi121 ceph-mon[145859]: pgmap v493: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:41.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:41 smithi188 ceph-mon[107793]: pgmap v493: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:43.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:43 smithi121 ceph-mon[145859]: pgmap v494: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:43.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:43 smithi188 ceph-mon[107793]: pgmap v494: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:45.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:45 smithi121 ceph-mon[145859]: pgmap v495: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:42:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:45 smithi188 ceph-mon[107793]: pgmap v495: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:42:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:47 smithi121 ceph-mon[145859]: pgmap v496: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:47 smithi188 ceph-mon[107793]: pgmap v496: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:49.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:49 smithi188 ceph-mon[107793]: pgmap v497: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:49 smithi121 ceph-mon[145859]: pgmap v497: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:50.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.274+0000 7fcdace12700 1 -- 172.21.15.121:0/50418459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcda8105a40 msgr2=0x7fcda8105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:50.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.274+0000 7fcdace12700 1 --2- 172.21.15.121:0/50418459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcda8105a40 0x7fcda8105e20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c009910 tx=0x7fcd9c009c20 comp rx=0 tx=0).stop 2024-04-05T21:42:50.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.274+0000 7fcdace12700 1 -- 172.21.15.121:0/50418459 shutdown_connections 2024-04-05T21:42:50.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.274+0000 7fcdace12700 1 --2- 172.21.15.121:0/50418459 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcda80ffa40 0x7fcda80ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.274+0000 7fcdace12700 1 --2- 172.21.15.121:0/50418459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcda8105a40 0x7fcda8105e20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.274+0000 7fcdace12700 1 -- 172.21.15.121:0/50418459 >> 172.21.15.121:0/50418459 conn(0x7fcda80fb570 msgr2=0x7fcda80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:50.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.274+0000 7fcdace12700 1 -- 172.21.15.121:0/50418459 shutdown_connections 2024-04-05T21:42:50.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.274+0000 7fcdace12700 1 -- 172.21.15.121:0/50418459 wait complete. 2024-04-05T21:42:50.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.275+0000 7fcdace12700 1 Processor -- start 2024-04-05T21:42:50.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.275+0000 7fcdace12700 1 -- start start 2024-04-05T21:42:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.275+0000 7fcdace12700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcda80ffa40 0x7fcda8197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.275+0000 7fcdace12700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcda8105a40 0x7fcda81979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.275+0000 7fcdace12700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda8197ff0 con 0x7fcda80ffa40 2024-04-05T21:42:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.275+0000 7fcdace12700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda8198130 con 0x7fcda8105a40 2024-04-05T21:42:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.275+0000 7fcda659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcda80ffa40 0x7fcda8197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.275+0000 7fcda659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcda80ffa40 0x7fcda8197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45734/0 (socket says 172.21.15.121:45734) 2024-04-05T21:42:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.275+0000 7fcda659c700 1 -- 172.21.15.121:0/2555579929 learned_addr learned my addr 172.21.15.121:0/2555579929 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.276+0000 7fcda5d9b700 1 --2- 172.21.15.121:0/2555579929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcda8105a40 0x7fcda81979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.276+0000 7fcda659c700 1 -- 172.21.15.121:0/2555579929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcda8105a40 msgr2=0x7fcda81979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.276+0000 7fcda659c700 1 --2- 172.21.15.121:0/2555579929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcda8105a40 0x7fcda81979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.276+0000 7fcda659c700 1 -- 172.21.15.121:0/2555579929 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c0095f0 con 0x7fcda80ffa40 2024-04-05T21:42:50.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.276+0000 7fcda659c700 1 --2- 172.21.15.121:0/2555579929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcda80ffa40 0x7fcda8197460 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c006010 tx=0x7fcd9c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:50.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.276+0000 7fcd977fe700 1 -- 172.21.15.121:0/2555579929 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd9c01d070 con 0x7fcda80ffa40 2024-04-05T21:42:50.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.276+0000 7fcd977fe700 1 -- 172.21.15.121:0/2555579929 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd9c00f460 con 0x7fcda80ffa40 2024-04-05T21:42:50.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.276+0000 7fcdace12700 1 -- 172.21.15.121:0/2555579929 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda81a04d0 con 0x7fcda80ffa40 2024-04-05T21:42:50.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.276+0000 7fcd977fe700 1 -- 172.21.15.121:0/2555579929 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd9c021640 con 0x7fcda80ffa40 2024-04-05T21:42:50.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.276+0000 7fcdace12700 1 -- 172.21.15.121:0/2555579929 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda81a08e0 con 0x7fcda80ffa40 2024-04-05T21:42:50.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.277+0000 7fcdace12700 1 -- 172.21.15.121:0/2555579929 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda804ea90 con 0x7fcda80ffa40 2024-04-05T21:42:50.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.278+0000 7fcd977fe700 1 -- 172.21.15.121:0/2555579929 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcd9c00fa20 con 0x7fcda80ffa40 2024-04-05T21:42:50.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.278+0000 7fcd977fe700 1 --2- 172.21.15.121:0/2555579929 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcd9006cce0 0x7fcd9006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:50.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.278+0000 7fcd977fe700 1 -- 172.21.15.121:0/2555579929 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fcd9c08d8c0 con 0x7fcda80ffa40 2024-04-05T21:42:50.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.278+0000 7fcda5d9b700 1 --2- 172.21.15.121:0/2555579929 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcd9006cce0 0x7fcd9006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:50.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.278+0000 7fcda5d9b700 1 --2- 172.21.15.121:0/2555579929 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcd9006cce0 0x7fcd9006f1a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcd98005ea0 tx=0x7fcd98005e30 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.283+0000 7fcd977fe700 1 -- 172.21.15.121:0/2555579929 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcd9c057930 con 0x7fcda80ffa40 2024-04-05T21:42:50.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.494+0000 7fcdace12700 1 -- 172.21.15.121:0/2555579929 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcda80fc6f0 con 0x7fcd9006cce0 2024-04-05T21:42:50.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.495+0000 7fcd977fe700 1 -- 172.21.15.121:0/2555579929 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fcda80fc6f0 con 0x7fcd9006cce0 2024-04-05T21:42:50.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.497+0000 7fcdace12700 1 -- 172.21.15.121:0/2555579929 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcd9006cce0 msgr2=0x7fcd9006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:50.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.497+0000 7fcdace12700 1 --2- 172.21.15.121:0/2555579929 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcd9006cce0 0x7fcd9006f1a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcd98005ea0 tx=0x7fcd98005e30 comp rx=0 tx=0).stop 2024-04-05T21:42:50.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.497+0000 7fcdace12700 1 -- 172.21.15.121:0/2555579929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcda80ffa40 msgr2=0x7fcda8197460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:50.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.497+0000 7fcdace12700 1 --2- 172.21.15.121:0/2555579929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcda80ffa40 0x7fcda8197460 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c006010 tx=0x7fcd9c00be20 comp rx=0 tx=0).stop 2024-04-05T21:42:50.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.497+0000 7fcdace12700 1 -- 172.21.15.121:0/2555579929 shutdown_connections 2024-04-05T21:42:50.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.497+0000 7fcdace12700 1 --2- 172.21.15.121:0/2555579929 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcd9006cce0 0x7fcd9006f1a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.497+0000 7fcdace12700 1 --2- 172.21.15.121:0/2555579929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcda80ffa40 0x7fcda8197460 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.497+0000 7fcdace12700 1 --2- 172.21.15.121:0/2555579929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcda8105a40 0x7fcda81979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.497+0000 7fcdace12700 1 -- 172.21.15.121:0/2555579929 >> 172.21.15.121:0/2555579929 conn(0x7fcda80fb570 msgr2=0x7fcda80fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:50.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.497+0000 7fcdace12700 1 -- 172.21.15.121:0/2555579929 shutdown_connections 2024-04-05T21:42:50.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.497+0000 7fcdace12700 1 -- 172.21.15.121:0/2555579929 wait complete. 2024-04-05T21:42:50.508 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:42:50.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.601+0000 7feffa186700 1 -- 172.21.15.121:0/311497763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feff4105a40 msgr2=0x7feff4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:50.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.601+0000 7feffa186700 1 --2- 172.21.15.121:0/311497763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feff4105a40 0x7feff4105e20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009910 tx=0x7fefe4009c20 comp rx=0 tx=0).stop 2024-04-05T21:42:50.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.601+0000 7feffa186700 1 -- 172.21.15.121:0/311497763 shutdown_connections 2024-04-05T21:42:50.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.601+0000 7feffa186700 1 --2- 172.21.15.121:0/311497763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feff40ffbd0 0x7feff4100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.601+0000 7feffa186700 1 --2- 172.21.15.121:0/311497763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feff4105a40 0x7feff4105e20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.601+0000 7feffa186700 1 -- 172.21.15.121:0/311497763 >> 172.21.15.121:0/311497763 conn(0x7feff40755a0 msgr2=0x7feff40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:50.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.601+0000 7feffa186700 1 -- 172.21.15.121:0/311497763 shutdown_connections 2024-04-05T21:42:50.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.601+0000 7feffa186700 1 -- 172.21.15.121:0/311497763 wait complete. 2024-04-05T21:42:50.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.602+0000 7feffa186700 1 Processor -- start 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.602+0000 7feffa186700 1 -- start start 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.602+0000 7feffa186700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feff40ffbd0 0x7feff4195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.602+0000 7feffa186700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feff4105a40 0x7feff4195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.602+0000 7feffa186700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff4196270 con 0x7feff4105a40 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.602+0000 7feffa186700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff419a000 con 0x7feff40ffbd0 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.602+0000 7feff2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feff4105a40 0x7feff4195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.602+0000 7feff2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feff4105a40 0x7feff4195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45754/0 (socket says 172.21.15.121:45754) 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.602+0000 7feff2ffd700 1 -- 172.21.15.121:0/4286748007 learned_addr learned my addr 172.21.15.121:0/4286748007 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.603+0000 7feff2ffd700 1 -- 172.21.15.121:0/4286748007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feff40ffbd0 msgr2=0x7feff4195650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.603+0000 7feff37fe700 1 --2- 172.21.15.121:0/4286748007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feff40ffbd0 0x7feff4195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.603+0000 7feff2ffd700 1 --2- 172.21.15.121:0/4286748007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feff40ffbd0 0x7feff4195650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.603+0000 7feff2ffd700 1 -- 172.21.15.121:0/4286748007 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe40095f0 con 0x7feff4105a40 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.603+0000 7feff37fe700 1 --2- 172.21.15.121:0/4286748007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feff40ffbd0 0x7feff4195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:42:50.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.603+0000 7feff2ffd700 1 --2- 172.21.15.121:0/4286748007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feff4105a40 0x7feff4195b90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fefe800b4b0 tx=0x7fefe800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.603+0000 7feff0ff9700 1 -- 172.21.15.121:0/4286748007 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefe8011800 con 0x7feff4105a40 2024-04-05T21:42:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.603+0000 7feffa186700 1 -- 172.21.15.121:0/4286748007 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff419a2e0 con 0x7feff4105a40 2024-04-05T21:42:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.603+0000 7feff0ff9700 1 -- 172.21.15.121:0/4286748007 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefe8011e00 con 0x7feff4105a40 2024-04-05T21:42:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.603+0000 7feff0ff9700 1 -- 172.21.15.121:0/4286748007 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefe8010e50 con 0x7feff4105a40 2024-04-05T21:42:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.603+0000 7feffa186700 1 -- 172.21.15.121:0/4286748007 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff419a830 con 0x7feff4105a40 2024-04-05T21:42:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.604+0000 7feffa186700 1 -- 172.21.15.121:0/4286748007 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff404ea90 con 0x7feff4105a40 2024-04-05T21:42:50.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.604+0000 7feff0ff9700 1 -- 172.21.15.121:0/4286748007 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fefe801a440 con 0x7feff4105a40 2024-04-05T21:42:50.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.605+0000 7feff0ff9700 1 --2- 172.21.15.121:0/4286748007 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fefdc06eef0 0x7fefdc0713b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:50.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.605+0000 7feff0ff9700 1 -- 172.21.15.121:0/4286748007 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fefe808c1a0 con 0x7feff4105a40 2024-04-05T21:42:50.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.605+0000 7feff37fe700 1 --2- 172.21.15.121:0/4286748007 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fefdc06eef0 0x7fefdc0713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:50.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.605+0000 7feff37fe700 1 --2- 172.21.15.121:0/4286748007 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fefdc06eef0 0x7fefdc0713b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fefe400b5c0 tx=0x7fefe4005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:50.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.609+0000 7feff0ff9700 1 -- 172.21.15.121:0/4286748007 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fefe8056280 con 0x7feff4105a40 2024-04-05T21:42:50.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.822+0000 7feffa186700 1 -- 172.21.15.121:0/4286748007 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7feff419ac70 con 0x7fefdc06eef0 2024-04-05T21:42:50.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.823+0000 7feff0ff9700 1 -- 172.21.15.121:0/4286748007 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7feff419ac70 con 0x7fefdc06eef0 2024-04-05T21:42:50.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.825+0000 7feffa186700 1 -- 172.21.15.121:0/4286748007 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fefdc06eef0 msgr2=0x7fefdc0713b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:50.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.825+0000 7feffa186700 1 --2- 172.21.15.121:0/4286748007 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fefdc06eef0 0x7fefdc0713b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fefe400b5c0 tx=0x7fefe4005960 comp rx=0 tx=0).stop 2024-04-05T21:42:50.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.825+0000 7feffa186700 1 -- 172.21.15.121:0/4286748007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feff4105a40 msgr2=0x7feff4195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:50.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.825+0000 7feffa186700 1 --2- 172.21.15.121:0/4286748007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feff4105a40 0x7feff4195b90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fefe800b4b0 tx=0x7fefe800b870 comp rx=0 tx=0).stop 2024-04-05T21:42:50.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.825+0000 7feffa186700 1 -- 172.21.15.121:0/4286748007 shutdown_connections 2024-04-05T21:42:50.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.825+0000 7feffa186700 1 --2- 172.21.15.121:0/4286748007 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fefdc06eef0 0x7fefdc0713b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.825+0000 7feffa186700 1 --2- 172.21.15.121:0/4286748007 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feff40ffbd0 0x7feff4195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.825+0000 7feffa186700 1 --2- 172.21.15.121:0/4286748007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feff4105a40 0x7feff4195b90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.825+0000 7feffa186700 1 -- 172.21.15.121:0/4286748007 >> 172.21.15.121:0/4286748007 conn(0x7feff40755a0 msgr2=0x7feff40fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:50.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.825+0000 7feffa186700 1 -- 172.21.15.121:0/4286748007 shutdown_connections 2024-04-05T21:42:50.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.825+0000 7feffa186700 1 -- 172.21.15.121:0/4286748007 wait complete. 2024-04-05T21:42:50.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.928+0000 7fa5a3736700 1 -- 172.21.15.121:0/2606462463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa59c105b50 msgr2=0x7fa59c105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:50.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.928+0000 7fa5a3736700 1 --2- 172.21.15.121:0/2606462463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa59c105b50 0x7fa59c105f30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa598009910 tx=0x7fa598009c20 comp rx=0 tx=0).stop 2024-04-05T21:42:50.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.928+0000 7fa5a3736700 1 -- 172.21.15.121:0/2606462463 shutdown_connections 2024-04-05T21:42:50.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.928+0000 7fa5a3736700 1 --2- 172.21.15.121:0/2606462463 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa59c0fef70 0x7fa59c0ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.928+0000 7fa5a3736700 1 --2- 172.21.15.121:0/2606462463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa59c105b50 0x7fa59c105f30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.928+0000 7fa5a3736700 1 -- 172.21.15.121:0/2606462463 >> 172.21.15.121:0/2606462463 conn(0x7fa59c075e30 msgr2=0x7fa59c0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:50.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.929+0000 7fa5a3736700 1 -- 172.21.15.121:0/2606462463 shutdown_connections 2024-04-05T21:42:50.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.929+0000 7fa5a3736700 1 -- 172.21.15.121:0/2606462463 wait complete. 2024-04-05T21:42:50.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.929+0000 7fa5a3736700 1 Processor -- start 2024-04-05T21:42:50.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.929+0000 7fa5a3736700 1 -- start start 2024-04-05T21:42:50.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a3736700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa59c0fef70 0x7fa59c19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:50.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a3736700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa59c105b50 0x7fa59c19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:50.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a3736700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c19d070 con 0x7fa59c0fef70 2024-04-05T21:42:50.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a3736700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c196590 con 0x7fa59c105b50 2024-04-05T21:42:50.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a14d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa59c0fef70 0x7fa59c19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:50.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a14d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa59c0fef70 0x7fa59c19c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45770/0 (socket says 172.21.15.121:45770) 2024-04-05T21:42:50.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a14d2700 1 -- 172.21.15.121:0/1263411123 learned_addr learned my addr 172.21.15.121:0/1263411123 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:50.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a0cd1700 1 --2- 172.21.15.121:0/1263411123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa59c105b50 0x7fa59c19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:50.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a14d2700 1 -- 172.21.15.121:0/1263411123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa59c105b50 msgr2=0x7fa59c19ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:50.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a14d2700 1 --2- 172.21.15.121:0/1263411123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa59c105b50 0x7fa59c19ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:50.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a14d2700 1 -- 172.21.15.121:0/1263411123 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5980095f0 con 0x7fa59c0fef70 2024-04-05T21:42:50.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a0cd1700 1 --2- 172.21.15.121:0/1263411123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa59c105b50 0x7fa59c19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:42:50.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a14d2700 1 --2- 172.21.15.121:0/1263411123 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa59c0fef70 0x7fa59c19c510 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa598006010 tx=0x7fa59800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:50.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5927fc700 1 -- 172.21.15.121:0/1263411123 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa59801d070 con 0x7fa59c0fef70 2024-04-05T21:42:50.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a3736700 1 -- 172.21.15.121:0/1263411123 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa59c196810 con 0x7fa59c0fef70 2024-04-05T21:42:50.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5927fc700 1 -- 172.21.15.121:0/1263411123 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa59800f460 con 0x7fa59c0fef70 2024-04-05T21:42:50.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5927fc700 1 -- 172.21.15.121:0/1263411123 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa598021620 con 0x7fa59c0fef70 2024-04-05T21:42:50.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.930+0000 7fa5a3736700 1 -- 172.21.15.121:0/1263411123 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59c196d00 con 0x7fa59c0fef70 2024-04-05T21:42:50.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.931+0000 7fa5a3736700 1 -- 172.21.15.121:0/1263411123 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59c109120 con 0x7fa59c0fef70 2024-04-05T21:42:50.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.932+0000 7fa5927fc700 1 -- 172.21.15.121:0/1263411123 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa59800fa40 con 0x7fa59c0fef70 2024-04-05T21:42:50.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.932+0000 7fa5927fc700 1 --2- 172.21.15.121:0/1263411123 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa58806cce0 0x7fa58806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:50.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.932+0000 7fa5927fc700 1 -- 172.21.15.121:0/1263411123 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa59808d790 con 0x7fa59c0fef70 2024-04-05T21:42:50.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.932+0000 7fa5a0cd1700 1 --2- 172.21.15.121:0/1263411123 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa58806cce0 0x7fa58806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:50.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.933+0000 7fa5a0cd1700 1 --2- 172.21.15.121:0/1263411123 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa58806cce0 0x7fa58806f1a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa58c005950 tx=0x7fa58c0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:50.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:50.937+0000 7fa5927fc700 1 -- 172.21.15.121:0/1263411123 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa59805b290 con 0x7fa59c0fef70 2024-04-05T21:42:51.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.138+0000 7fa5a3736700 1 -- 172.21.15.121:0/1263411123 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa59c196f20 con 0x7fa58806cce0 2024-04-05T21:42:51.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.148+0000 7fa5927fc700 1 -- 172.21.15.121:0/1263411123 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fa59c196f20 con 0x7fa58806cce0 2024-04-05T21:42:51.149 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:42:51.149 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (13m) 7m ago 25m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:42:51.150 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (25m) 7m ago 25m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:42:51.150 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (24m) 7m ago 24m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:42:51.150 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (11m) 7m ago 25m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:42:51.150 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (11m) 7m ago 24m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:42:51.150 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (13m) 7m ago 24m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:42:51.150 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (8m) 7m ago 19m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:42:51.150 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (7m) 7m ago 18m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:42:51.150 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (7m) 7m ago 18m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:42:51.150 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (18m) 7m ago 18m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:42:51.150 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (15m) 7m ago 26m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:42:51.150 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (14m) 7m ago 24m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:42:51.151 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (12m) 7m ago 26m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:42:51.151 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (12m) 7m ago 23m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:42:51.151 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (14m) 7m ago 25m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:42:51.151 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (14m) 7m ago 24m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:42:51.151 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (11m) 7m ago 23m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:42:51.151 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (10m) 7m ago 22m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:42:51.151 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (10m) 7m ago 22m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:42:51.151 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (9m) 7m ago 21m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:42:51.151 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (8m) 7m ago 21m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:42:51.151 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (8m) 7m ago 20m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:42:51.151 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (14m) 7m ago 24m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:42:51.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.151+0000 7fa5a3736700 1 -- 172.21.15.121:0/1263411123 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa58806cce0 msgr2=0x7fa58806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:51.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.151+0000 7fa5a3736700 1 --2- 172.21.15.121:0/1263411123 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa58806cce0 0x7fa58806f1a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa58c005950 tx=0x7fa58c0058e0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.151+0000 7fa5a3736700 1 -- 172.21.15.121:0/1263411123 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa59c0fef70 msgr2=0x7fa59c19c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:51.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.151+0000 7fa5a3736700 1 --2- 172.21.15.121:0/1263411123 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa59c0fef70 0x7fa59c19c510 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa598006010 tx=0x7fa59800be20 comp rx=0 tx=0).stop 2024-04-05T21:42:51.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.151+0000 7fa5a3736700 1 -- 172.21.15.121:0/1263411123 shutdown_connections 2024-04-05T21:42:51.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.151+0000 7fa5a3736700 1 --2- 172.21.15.121:0/1263411123 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa58806cce0 0x7fa58806f1a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.151+0000 7fa5a3736700 1 --2- 172.21.15.121:0/1263411123 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa59c0fef70 0x7fa59c19c510 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.151+0000 7fa5a3736700 1 --2- 172.21.15.121:0/1263411123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa59c105b50 0x7fa59c19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.151+0000 7fa5a3736700 1 -- 172.21.15.121:0/1263411123 >> 172.21.15.121:0/1263411123 conn(0x7fa59c075e30 msgr2=0x7fa59c104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:51.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.151+0000 7fa5a3736700 1 -- 172.21.15.121:0/1263411123 shutdown_connections 2024-04-05T21:42:51.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.151+0000 7fa5a3736700 1 -- 172.21.15.121:0/1263411123 wait complete. 2024-04-05T21:42:51.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.253+0000 7f131b9dc700 1 -- 172.21.15.121:0/3217351304 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1314105a40 msgr2=0x7f1314105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.253+0000 7f131b9dc700 1 --2- 172.21.15.121:0/3217351304 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1314105a40 0x7f1314105e20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1310009910 tx=0x7f1310009c20 comp rx=0 tx=0).stop 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.253+0000 7f131b9dc700 1 -- 172.21.15.121:0/3217351304 shutdown_connections 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.253+0000 7f131b9dc700 1 --2- 172.21.15.121:0/3217351304 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13140ffbd0 0x7f1314100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.253+0000 7f131b9dc700 1 --2- 172.21.15.121:0/3217351304 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1314105a40 0x7f1314105e20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.253+0000 7f131b9dc700 1 -- 172.21.15.121:0/3217351304 >> 172.21.15.121:0/3217351304 conn(0x7f13140755a0 msgr2=0x7f13140759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.253+0000 7f131b9dc700 1 -- 172.21.15.121:0/3217351304 shutdown_connections 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.253+0000 7f131b9dc700 1 -- 172.21.15.121:0/3217351304 wait complete. 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.253+0000 7f131b9dc700 1 Processor -- start 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.253+0000 7f131b9dc700 1 -- start start 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f131b9dc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13140ffbd0 0x7f13141974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f131b9dc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1314105a40 0x7f13141979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f131b9dc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1314198000 con 0x7f13140ffbd0 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f131b9dc700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1314198170 con 0x7f1314105a40 2024-04-05T21:42:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f1319778700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13140ffbd0 0x7f13141974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f1319778700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13140ffbd0 0x7f13141974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45798/0 (socket says 172.21.15.121:45798) 2024-04-05T21:42:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f1319778700 1 -- 172.21.15.121:0/1398245955 learned_addr learned my addr 172.21.15.121:0/1398245955 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f1318f77700 1 --2- 172.21.15.121:0/1398245955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1314105a40 0x7f13141979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f1319778700 1 -- 172.21.15.121:0/1398245955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1314105a40 msgr2=0x7f13141979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f1319778700 1 --2- 172.21.15.121:0/1398245955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1314105a40 0x7f13141979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f1319778700 1 -- 172.21.15.121:0/1398245955 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13100095f0 con 0x7f13140ffbd0 2024-04-05T21:42:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f1319778700 1 --2- 172.21.15.121:0/1398245955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13140ffbd0 0x7f13141974a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1310006010 tx=0x7f131000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f130a7fc700 1 -- 172.21.15.121:0/1398245955 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f131001d070 con 0x7f13140ffbd0 2024-04-05T21:42:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f130a7fc700 1 -- 172.21.15.121:0/1398245955 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f131000f460 con 0x7f13140ffbd0 2024-04-05T21:42:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f131b9dc700 1 -- 172.21.15.121:0/1398245955 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13141a0510 con 0x7f13140ffbd0 2024-04-05T21:42:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f130a7fc700 1 -- 172.21.15.121:0/1398245955 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1310021620 con 0x7f13140ffbd0 2024-04-05T21:42:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.254+0000 7f131b9dc700 1 -- 172.21.15.121:0/1398245955 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13141a0980 con 0x7f13140ffbd0 2024-04-05T21:42:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.255+0000 7f131b9dc700 1 -- 172.21.15.121:0/1398245955 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1314108040 con 0x7f13140ffbd0 2024-04-05T21:42:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.256+0000 7f130a7fc700 1 -- 172.21.15.121:0/1398245955 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f131000f5d0 con 0x7f13140ffbd0 2024-04-05T21:42:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.256+0000 7f130a7fc700 1 --2- 172.21.15.121:0/1398245955 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f130006cce0 0x7f130006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.256+0000 7f130a7fc700 1 -- 172.21.15.121:0/1398245955 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f131008d740 con 0x7f13140ffbd0 2024-04-05T21:42:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.256+0000 7f1318f77700 1 --2- 172.21.15.121:0/1398245955 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f130006cce0 0x7f130006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.257+0000 7f1318f77700 1 --2- 172.21.15.121:0/1398245955 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f130006cce0 0x7f130006f1a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1314068ce0 tx=0x7f130400a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:51.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.261+0000 7f130a7fc700 1 -- 172.21.15.121:0/1398245955 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f13100577b0 con 0x7f13140ffbd0 2024-04-05T21:42:51.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.537+0000 7f131b9dc700 1 -- 172.21.15.121:0/1398245955 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f131402cd30 con 0x7f13140ffbd0 2024-04-05T21:42:51.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.537+0000 7f130a7fc700 1 -- 172.21.15.121:0/1398245955 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f131005add0 con 0x7f13140ffbd0 2024-04-05T21:42:51.538 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:42:51.538 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:42:51.539 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:42:51.540 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:42:51.540 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:42:51.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.540+0000 7f131b9dc700 1 -- 172.21.15.121:0/1398245955 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f130006cce0 msgr2=0x7f130006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:51.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.540+0000 7f131b9dc700 1 --2- 172.21.15.121:0/1398245955 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f130006cce0 0x7f130006f1a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1314068ce0 tx=0x7f130400a300 comp rx=0 tx=0).stop 2024-04-05T21:42:51.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.540+0000 7f131b9dc700 1 -- 172.21.15.121:0/1398245955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13140ffbd0 msgr2=0x7f13141974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:51.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.540+0000 7f131b9dc700 1 --2- 172.21.15.121:0/1398245955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13140ffbd0 0x7f13141974a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1310006010 tx=0x7f131000bd70 comp rx=0 tx=0).stop 2024-04-05T21:42:51.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.540+0000 7f131b9dc700 1 -- 172.21.15.121:0/1398245955 shutdown_connections 2024-04-05T21:42:51.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.540+0000 7f131b9dc700 1 --2- 172.21.15.121:0/1398245955 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f130006cce0 0x7f130006f1a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.540+0000 7f131b9dc700 1 --2- 172.21.15.121:0/1398245955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13140ffbd0 0x7f13141974a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.540+0000 7f131b9dc700 1 --2- 172.21.15.121:0/1398245955 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1314105a40 0x7f13141979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.540+0000 7f131b9dc700 1 -- 172.21.15.121:0/1398245955 >> 172.21.15.121:0/1398245955 conn(0x7f13140755a0 msgr2=0x7f13140fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:51.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.540+0000 7f131b9dc700 1 -- 172.21.15.121:0/1398245955 shutdown_connections 2024-04-05T21:42:51.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.540+0000 7f131b9dc700 1 -- 172.21.15.121:0/1398245955 wait complete. 2024-04-05T21:42:51.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.643+0000 7fb509496700 1 -- 172.21.15.121:0/1558022657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb504101750 msgr2=0x7fb5041057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:51.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.643+0000 7fb509496700 1 --2- 172.21.15.121:0/1558022657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb504101750 0x7fb5041057a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4009910 tx=0x7fb4f4009c20 comp rx=0 tx=0).stop 2024-04-05T21:42:51.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.643+0000 7fb509496700 1 -- 172.21.15.121:0/1558022657 shutdown_connections 2024-04-05T21:42:51.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.643+0000 7fb509496700 1 --2- 172.21.15.121:0/1558022657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb504101750 0x7fb5041057a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.643+0000 7fb509496700 1 --2- 172.21.15.121:0/1558022657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb504100da0 0x7fb504101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.643+0000 7fb509496700 1 -- 172.21.15.121:0/1558022657 >> 172.21.15.121:0/1558022657 conn(0x7fb5040fc630 msgr2=0x7fb5040fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:51.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.643+0000 7fb509496700 1 -- 172.21.15.121:0/1558022657 shutdown_connections 2024-04-05T21:42:51.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.643+0000 7fb509496700 1 -- 172.21.15.121:0/1558022657 wait complete. 2024-04-05T21:42:51.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.643+0000 7fb509496700 1 Processor -- start 2024-04-05T21:42:51.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.643+0000 7fb509496700 1 -- start start 2024-04-05T21:42:51.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb509496700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb504100da0 0x7fb504196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:51.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb509496700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb504101750 0x7fb504196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:51.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb509496700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb504197270 con 0x7fb504100da0 2024-04-05T21:42:51.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb509496700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50419b000 con 0x7fb504101750 2024-04-05T21:42:51.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb502ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb504100da0 0x7fb504196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:51.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb502ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb504100da0 0x7fb504196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45808/0 (socket says 172.21.15.121:45808) 2024-04-05T21:42:51.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb502ffd700 1 -- 172.21.15.121:0/372789249 learned_addr learned my addr 172.21.15.121:0/372789249 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:51.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb5027fc700 1 --2- 172.21.15.121:0/372789249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb504101750 0x7fb504196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:51.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb502ffd700 1 -- 172.21.15.121:0/372789249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb504101750 msgr2=0x7fb504196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:51.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb502ffd700 1 --2- 172.21.15.121:0/372789249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb504101750 0x7fb504196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb502ffd700 1 -- 172.21.15.121:0/372789249 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f40095f0 con 0x7fb504100da0 2024-04-05T21:42:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb502ffd700 1 --2- 172.21.15.121:0/372789249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb504100da0 0x7fb504196650 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb4f800d8d0 tx=0x7fb4f800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb4ebfff700 1 -- 172.21.15.121:0/372789249 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4f80098a0 con 0x7fb504100da0 2024-04-05T21:42:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb4ebfff700 1 -- 172.21.15.121:0/372789249 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4f8010460 con 0x7fb504100da0 2024-04-05T21:42:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb509496700 1 -- 172.21.15.121:0/372789249 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50419b2e0 con 0x7fb504100da0 2024-04-05T21:42:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb4ebfff700 1 -- 172.21.15.121:0/372789249 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4f8009ea0 con 0x7fb504100da0 2024-04-05T21:42:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.644+0000 7fb509496700 1 -- 172.21.15.121:0/372789249 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb50419b830 con 0x7fb504100da0 2024-04-05T21:42:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.645+0000 7fb509496700 1 -- 172.21.15.121:0/372789249 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5041091a0 con 0x7fb504100da0 2024-04-05T21:42:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.646+0000 7fb4ebfff700 1 -- 172.21.15.121:0/372789249 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb4f80105d0 con 0x7fb504100da0 2024-04-05T21:42:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.646+0000 7fb4ebfff700 1 --2- 172.21.15.121:0/372789249 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb4ec06cce0 0x7fb4ec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.646+0000 7fb4ebfff700 1 -- 172.21.15.121:0/372789249 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb4f808c480 con 0x7fb504100da0 2024-04-05T21:42:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.646+0000 7fb5027fc700 1 --2- 172.21.15.121:0/372789249 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb4ec06cce0 0x7fb4ec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:51.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.647+0000 7fb5027fc700 1 --2- 172.21.15.121:0/372789249 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb4ec06cce0 0x7fb4ec06f1a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb504197c70 tx=0x7fb4f401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:51.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.651+0000 7fb4ebfff700 1 -- 172.21.15.121:0/372789249 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb4f80564f0 con 0x7fb504100da0 2024-04-05T21:42:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:51 smithi188 ceph-mon[107793]: pgmap v498: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:51 smithi188 ceph-mon[107793]: from='client.34762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:51 smithi188 ceph-mon[107793]: from='client.34766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:51.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.883+0000 7fb509496700 1 -- 172.21.15.121:0/372789249 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb50419bb40 con 0x7fb504100da0 2024-04-05T21:42:51.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.883+0000 7fb4ebfff700 1 -- 172.21.15.121:0/372789249 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fb4f8010880 con 0x7fb504100da0 2024-04-05T21:42:51.884 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:42:51.884 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:42:51.884 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:42:51.885 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:42:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:42:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:42:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:42:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:42:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:42:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:42:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:42:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:42:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:42:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:42:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:42:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:42:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:42:51.887 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:42:51.887 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:42:51.887 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:42:51.887 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:42:51.887 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:42:51.887 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:42:51.887 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:42:51.887 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:42:51.887 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:42:51.887 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:42:51.887 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:42:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.885+0000 7fb509496700 1 -- 172.21.15.121:0/372789249 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb4ec06cce0 msgr2=0x7fb4ec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.885+0000 7fb509496700 1 --2- 172.21.15.121:0/372789249 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb4ec06cce0 0x7fb4ec06f1a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb504197c70 tx=0x7fb4f401a040 comp rx=0 tx=0).stop 2024-04-05T21:42:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.885+0000 7fb509496700 1 -- 172.21.15.121:0/372789249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb504100da0 msgr2=0x7fb504196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.885+0000 7fb509496700 1 --2- 172.21.15.121:0/372789249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb504100da0 0x7fb504196650 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb4f800d8d0 tx=0x7fb4f800dbe0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.886+0000 7fb509496700 1 -- 172.21.15.121:0/372789249 shutdown_connections 2024-04-05T21:42:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.886+0000 7fb509496700 1 --2- 172.21.15.121:0/372789249 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb4ec06cce0 0x7fb4ec06f1a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.886+0000 7fb509496700 1 --2- 172.21.15.121:0/372789249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb504100da0 0x7fb504196650 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.886+0000 7fb509496700 1 --2- 172.21.15.121:0/372789249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb504101750 0x7fb504196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.886+0000 7fb509496700 1 -- 172.21.15.121:0/372789249 >> 172.21.15.121:0/372789249 conn(0x7fb5040fc630 msgr2=0x7fb5040fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.886+0000 7fb509496700 1 -- 172.21.15.121:0/372789249 shutdown_connections 2024-04-05T21:42:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.886+0000 7fb509496700 1 -- 172.21.15.121:0/372789249 wait complete. 2024-04-05T21:42:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:42:51.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:51 smithi121 ceph-mon[145859]: pgmap v498: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:51.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:51 smithi121 ceph-mon[145859]: from='client.34762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:51.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:51 smithi121 ceph-mon[145859]: from='client.34766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:51.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.990+0000 7ff13ada5700 1 -- 172.21.15.121:0/3347784542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1340ffa40 msgr2=0x7ff1340ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:51.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.990+0000 7ff13ada5700 1 --2- 172.21.15.121:0/3347784542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1340ffa40 0x7ff1340ffec0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff120009910 tx=0x7ff120009c20 comp rx=0 tx=0).stop 2024-04-05T21:42:51.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.990+0000 7ff13ada5700 1 -- 172.21.15.121:0/3347784542 shutdown_connections 2024-04-05T21:42:51.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.990+0000 7ff13ada5700 1 --2- 172.21.15.121:0/3347784542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1340ffa40 0x7ff1340ffec0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.990+0000 7ff13ada5700 1 --2- 172.21.15.121:0/3347784542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff134105a40 0x7ff134105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.990+0000 7ff13ada5700 1 -- 172.21.15.121:0/3347784542 >> 172.21.15.121:0/3347784542 conn(0x7ff1340fb570 msgr2=0x7ff1340fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:51.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.990+0000 7ff13ada5700 1 -- 172.21.15.121:0/3347784542 shutdown_connections 2024-04-05T21:42:51.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.990+0000 7ff13ada5700 1 -- 172.21.15.121:0/3347784542 wait complete. 2024-04-05T21:42:51.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.990+0000 7ff13ada5700 1 Processor -- start 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.990+0000 7ff13ada5700 1 -- start start 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff13ada5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1340ffa40 0x7ff134195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff13ada5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff134105a40 0x7ff134195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff13ada5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1341960d0 con 0x7ff1340ffa40 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff13ada5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff134196210 con 0x7ff134105a40 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff138b41700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1340ffa40 0x7ff134195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff138b41700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1340ffa40 0x7ff134195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45820/0 (socket says 172.21.15.121:45820) 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff138b41700 1 -- 172.21.15.121:0/2546493003 learned_addr learned my addr 172.21.15.121:0/2546493003 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff133fff700 1 --2- 172.21.15.121:0/2546493003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff134105a40 0x7ff134195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff138b41700 1 -- 172.21.15.121:0/2546493003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff134105a40 msgr2=0x7ff134195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff138b41700 1 --2- 172.21.15.121:0/2546493003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff134105a40 0x7ff134195a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff138b41700 1 -- 172.21.15.121:0/2546493003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1200095f0 con 0x7ff1340ffa40 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff138b41700 1 --2- 172.21.15.121:0/2546493003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1340ffa40 0x7ff134195540 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff12800c930 tx=0x7ff12800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff131ffb700 1 -- 172.21.15.121:0/2546493003 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff128007ad0 con 0x7ff1340ffa40 2024-04-05T21:42:51.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff13ada5700 1 -- 172.21.15.121:0/2546493003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13419a060 con 0x7ff1340ffa40 2024-04-05T21:42:51.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff131ffb700 1 -- 172.21.15.121:0/2546493003 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff128007c30 con 0x7ff1340ffa40 2024-04-05T21:42:51.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.991+0000 7ff131ffb700 1 -- 172.21.15.121:0/2546493003 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff128018630 con 0x7ff1340ffa40 2024-04-05T21:42:51.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.992+0000 7ff13ada5700 1 -- 172.21.15.121:0/2546493003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13419a5b0 con 0x7ff1340ffa40 2024-04-05T21:42:51.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.992+0000 7ff13ada5700 1 -- 172.21.15.121:0/2546493003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff134107fd0 con 0x7ff1340ffa40 2024-04-05T21:42:51.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.993+0000 7ff131ffb700 1 -- 172.21.15.121:0/2546493003 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff12800f450 con 0x7ff1340ffa40 2024-04-05T21:42:51.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.993+0000 7ff131ffb700 1 --2- 172.21.15.121:0/2546493003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff12406cce0 0x7ff12406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:51.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.993+0000 7ff131ffb700 1 -- 172.21.15.121:0/2546493003 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff12808c7d0 con 0x7ff1340ffa40 2024-04-05T21:42:51.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.993+0000 7ff133fff700 1 --2- 172.21.15.121:0/2546493003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff12406cce0 0x7ff12406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:51.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.994+0000 7ff133fff700 1 --2- 172.21.15.121:0/2546493003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff12406cce0 0x7ff12406f1a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff134196b60 tx=0x7ff12001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:51.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:51.998+0000 7ff131ffb700 1 -- 172.21.15.121:0/2546493003 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff1280568f0 con 0x7ff1340ffa40 2024-04-05T21:42:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.210+0000 7ff13ada5700 1 -- 172.21.15.121:0/2546493003 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff13402cd80 con 0x7ff12406cce0 2024-04-05T21:42:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.211+0000 7ff131ffb700 1 -- 172.21.15.121:0/2546493003 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff13402cd80 con 0x7ff12406cce0 2024-04-05T21:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:42:52.212 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:42:52.213 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:42:52.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.213+0000 7ff13ada5700 1 -- 172.21.15.121:0/2546493003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff12406cce0 msgr2=0x7ff12406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:52.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.213+0000 7ff13ada5700 1 --2- 172.21.15.121:0/2546493003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff12406cce0 0x7ff12406f1a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff134196b60 tx=0x7ff12001a040 comp rx=0 tx=0).stop 2024-04-05T21:42:52.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.213+0000 7ff13ada5700 1 -- 172.21.15.121:0/2546493003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1340ffa40 msgr2=0x7ff134195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:52.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.213+0000 7ff13ada5700 1 --2- 172.21.15.121:0/2546493003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1340ffa40 0x7ff134195540 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff12800c930 tx=0x7ff12800ccf0 comp rx=0 tx=0).stop 2024-04-05T21:42:52.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.214+0000 7ff13ada5700 1 -- 172.21.15.121:0/2546493003 shutdown_connections 2024-04-05T21:42:52.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.214+0000 7ff13ada5700 1 --2- 172.21.15.121:0/2546493003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff12406cce0 0x7ff12406f1a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:52.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.214+0000 7ff13ada5700 1 --2- 172.21.15.121:0/2546493003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1340ffa40 0x7ff134195540 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:52.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.214+0000 7ff13ada5700 1 --2- 172.21.15.121:0/2546493003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff134105a40 0x7ff134195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:52.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.214+0000 7ff13ada5700 1 -- 172.21.15.121:0/2546493003 >> 172.21.15.121:0/2546493003 conn(0x7ff1340fb570 msgr2=0x7ff1340fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:52.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.214+0000 7ff13ada5700 1 -- 172.21.15.121:0/2546493003 shutdown_connections 2024-04-05T21:42:52.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.214+0000 7ff13ada5700 1 -- 172.21.15.121:0/2546493003 wait complete. 2024-04-05T21:42:52.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.316+0000 7fcf406a1700 1 -- 172.21.15.121:0/580350512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf38101770 msgr2=0x7fcf381057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:52.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.316+0000 7fcf406a1700 1 --2- 172.21.15.121:0/580350512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf38101770 0x7fcf381057c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcf28009910 tx=0x7fcf28009c20 comp rx=0 tx=0).stop 2024-04-05T21:42:52.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.317+0000 7fcf406a1700 1 -- 172.21.15.121:0/580350512 shutdown_connections 2024-04-05T21:42:52.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.317+0000 7fcf406a1700 1 --2- 172.21.15.121:0/580350512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf38101770 0x7fcf381057c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:52.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.317+0000 7fcf406a1700 1 --2- 172.21.15.121:0/580350512 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf38100dc0 0x7fcf381011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:52.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.317+0000 7fcf406a1700 1 -- 172.21.15.121:0/580350512 >> 172.21.15.121:0/580350512 conn(0x7fcf380fc650 msgr2=0x7fcf380fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:52.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.317+0000 7fcf406a1700 1 -- 172.21.15.121:0/580350512 shutdown_connections 2024-04-05T21:42:52.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.317+0000 7fcf406a1700 1 -- 172.21.15.121:0/580350512 wait complete. 2024-04-05T21:42:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.317+0000 7fcf406a1700 1 Processor -- start 2024-04-05T21:42:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.317+0000 7fcf406a1700 1 -- start start 2024-04-05T21:42:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf406a1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf38100dc0 0x7fcf3819c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf406a1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf38101770 0x7fcf3819ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf406a1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3819d120 con 0x7fcf38101770 2024-04-05T21:42:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf406a1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf381965d0 con 0x7fcf38100dc0 2024-04-05T21:42:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf3dc3c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf38101770 0x7fcf3819ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf3dc3c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf38101770 0x7fcf3819ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45838/0 (socket says 172.21.15.121:45838) 2024-04-05T21:42:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf3dc3c700 1 -- 172.21.15.121:0/1825806008 learned_addr learned my addr 172.21.15.121:0/1825806008 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:42:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf3e43d700 1 --2- 172.21.15.121:0/1825806008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf38100dc0 0x7fcf3819c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf3dc3c700 1 -- 172.21.15.121:0/1825806008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf38100dc0 msgr2=0x7fcf3819c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf3dc3c700 1 --2- 172.21.15.121:0/1825806008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf38100dc0 0x7fcf3819c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:52.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf3dc3c700 1 -- 172.21.15.121:0/1825806008 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf280095f0 con 0x7fcf38101770 2024-04-05T21:42:52.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf3e43d700 1 --2- 172.21.15.121:0/1825806008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf38100dc0 0x7fcf3819c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:42:52.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf3dc3c700 1 --2- 172.21.15.121:0/1825806008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf38101770 0x7fcf3819ca90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800bd70 tx=0x7fcf2800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:52.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf2f7fe700 1 -- 172.21.15.121:0/1825806008 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf2801d070 con 0x7fcf38101770 2024-04-05T21:42:52.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf406a1700 1 -- 172.21.15.121:0/1825806008 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf38196850 con 0x7fcf38101770 2024-04-05T21:42:52.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf2f7fe700 1 -- 172.21.15.121:0/1825806008 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf2800f460 con 0x7fcf38101770 2024-04-05T21:42:52.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.318+0000 7fcf2f7fe700 1 -- 172.21.15.121:0/1825806008 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf28021630 con 0x7fcf38101770 2024-04-05T21:42:52.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.319+0000 7fcf406a1700 1 -- 172.21.15.121:0/1825806008 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf38196d40 con 0x7fcf38101770 2024-04-05T21:42:52.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.319+0000 7fcf406a1700 1 -- 172.21.15.121:0/1825806008 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf381091c0 con 0x7fcf38101770 2024-04-05T21:42:52.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.320+0000 7fcf2f7fe700 1 -- 172.21.15.121:0/1825806008 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcf2800f9d0 con 0x7fcf38101770 2024-04-05T21:42:52.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.320+0000 7fcf2f7fe700 1 --2- 172.21.15.121:0/1825806008 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcf2406cce0 0x7fcf2406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:42:52.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.320+0000 7fcf2f7fe700 1 -- 172.21.15.121:0/1825806008 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fcf2808d6f0 con 0x7fcf38101770 2024-04-05T21:42:52.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.320+0000 7fcf3e43d700 1 --2- 172.21.15.121:0/1825806008 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcf2406cce0 0x7fcf2406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:42:52.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.321+0000 7fcf3e43d700 1 --2- 172.21.15.121:0/1825806008 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcf2406cce0 0x7fcf2406f1a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcf34006fd0 tx=0x7fcf34008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:42:52.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.325+0000 7fcf2f7fe700 1 -- 172.21.15.121:0/1825806008 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcf2805b1f0 con 0x7fcf38101770 2024-04-05T21:42:52.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.605+0000 7fcf406a1700 1 -- 172.21.15.121:0/1825806008 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcf3802ccf0 con 0x7fcf38101770 2024-04-05T21:42:52.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.605+0000 7fcf2f7fe700 1 -- 172.21.15.121:0/1825806008 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fcf28061020 con 0x7fcf38101770 2024-04-05T21:42:52.606 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:42:52.606 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:42:52.607 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:42:52.607 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:42:52.607 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:42:52.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.608+0000 7fcf406a1700 1 -- 172.21.15.121:0/1825806008 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcf2406cce0 msgr2=0x7fcf2406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:52.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.608+0000 7fcf406a1700 1 --2- 172.21.15.121:0/1825806008 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcf2406cce0 0x7fcf2406f1a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcf34006fd0 tx=0x7fcf34008040 comp rx=0 tx=0).stop 2024-04-05T21:42:52.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.608+0000 7fcf406a1700 1 -- 172.21.15.121:0/1825806008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf38101770 msgr2=0x7fcf3819ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:42:52.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.608+0000 7fcf406a1700 1 --2- 172.21.15.121:0/1825806008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf38101770 0x7fcf3819ca90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800bd70 tx=0x7fcf2800be50 comp rx=0 tx=0).stop 2024-04-05T21:42:52.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.608+0000 7fcf406a1700 1 -- 172.21.15.121:0/1825806008 shutdown_connections 2024-04-05T21:42:52.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.608+0000 7fcf406a1700 1 --2- 172.21.15.121:0/1825806008 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcf2406cce0 0x7fcf2406f1a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:52.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.608+0000 7fcf406a1700 1 --2- 172.21.15.121:0/1825806008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf38100dc0 0x7fcf3819c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:52.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.608+0000 7fcf406a1700 1 --2- 172.21.15.121:0/1825806008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf38101770 0x7fcf3819ca90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:42:52.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.608+0000 7fcf406a1700 1 -- 172.21.15.121:0/1825806008 >> 172.21.15.121:0/1825806008 conn(0x7fcf380fc650 msgr2=0x7fcf380fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:42:52.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.608+0000 7fcf406a1700 1 -- 172.21.15.121:0/1825806008 shutdown_connections 2024-04-05T21:42:52.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:42:52.608+0000 7fcf406a1700 1 -- 172.21.15.121:0/1825806008 wait complete. 2024-04-05T21:42:52.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:52 smithi188 ceph-mon[107793]: from='client.34770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:52.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:52 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1398245955' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:42:52.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:52 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/372789249' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:42:52.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:52 smithi121 ceph-mon[145859]: from='client.34770 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:52.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:52 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1398245955' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:42:52.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:52 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/372789249' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:42:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:53 smithi188 ceph-mon[107793]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:53 smithi188 ceph-mon[107793]: pgmap v499: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:53.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:53 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1825806008' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:42:53.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:53 smithi121 ceph-mon[145859]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:42:53.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:53 smithi121 ceph-mon[145859]: pgmap v499: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:53.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:53 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1825806008' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:42:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:55 smithi188 ceph-mon[107793]: pgmap v500: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:55.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:55 smithi121 ceph-mon[145859]: pgmap v500: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:57 smithi188 ceph-mon[107793]: pgmap v501: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:57.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:57 smithi121 ceph-mon[145859]: pgmap v501: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:42:59 smithi188 ceph-mon[107793]: pgmap v502: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:42:59.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:42:59 smithi121 ceph-mon[145859]: pgmap v502: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:43:00.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:43:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:01 smithi188 ceph-mon[107793]: pgmap v503: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:01.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:01 smithi121 ceph-mon[145859]: pgmap v503: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:03.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:03 smithi121 ceph-mon[145859]: pgmap v504: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:03.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:03 smithi188 ceph-mon[107793]: pgmap v504: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:05 smithi121 ceph-mon[145859]: pgmap v505: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:05 smithi188 ceph-mon[107793]: pgmap v505: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:07 smithi121 ceph-mon[145859]: pgmap v506: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:07 smithi188 ceph-mon[107793]: pgmap v506: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:09.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:09 smithi188 ceph-mon[107793]: pgmap v507: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:09.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:09 smithi121 ceph-mon[145859]: pgmap v507: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:11 smithi121 ceph-mon[145859]: pgmap v508: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:11.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:11 smithi188 ceph-mon[107793]: pgmap v508: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:13.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:13 smithi121 ceph-mon[145859]: pgmap v509: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:13.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:13 smithi188 ceph-mon[107793]: pgmap v509: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:15 smithi121 ceph-mon[145859]: pgmap v510: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:43:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:15 smithi188 ceph-mon[107793]: pgmap v510: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:43:17.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:17 smithi121 ceph-mon[145859]: pgmap v511: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:17.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:17 smithi188 ceph-mon[107793]: pgmap v511: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:19 smithi121 ceph-mon[145859]: pgmap v512: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:19 smithi188 ceph-mon[107793]: pgmap v512: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:21 smithi121 ceph-mon[145859]: pgmap v513: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:21 smithi188 ceph-mon[107793]: pgmap v513: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:22.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 -- 172.21.15.121:0/977773036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce88101750 msgr2=0x7fce881057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:22.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 --2- 172.21.15.121:0/977773036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce88101750 0x7fce881057a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fce74009910 tx=0x7fce74009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:22.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 -- 172.21.15.121:0/977773036 shutdown_connections 2024-04-05T21:43:22.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 --2- 172.21.15.121:0/977773036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce88101750 0x7fce881057a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:22.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 --2- 172.21.15.121:0/977773036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce88100da0 0x7fce88101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:22.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 -- 172.21.15.121:0/977773036 >> 172.21.15.121:0/977773036 conn(0x7fce880fc630 msgr2=0x7fce880fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:22.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 -- 172.21.15.121:0/977773036 shutdown_connections 2024-04-05T21:43:22.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 -- 172.21.15.121:0/977773036 wait complete. 2024-04-05T21:43:22.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 Processor -- start 2024-04-05T21:43:22.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 -- start start 2024-04-05T21:43:22.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce88100da0 0x7fce88196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:22.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce88101750 0x7fce88196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:22.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce881971e0 con 0x7fce88100da0 2024-04-05T21:43:22.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.724+0000 7fce8f0d4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce88197320 con 0x7fce88101750 2024-04-05T21:43:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.725+0000 7fce8ce70700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce88100da0 0x7fce88196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.725+0000 7fce8ce70700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce88100da0 0x7fce88196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60406/0 (socket says 172.21.15.121:60406) 2024-04-05T21:43:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.725+0000 7fce8ce70700 1 -- 172.21.15.121:0/739470118 learned_addr learned my addr 172.21.15.121:0/739470118 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.725+0000 7fce87fff700 1 --2- 172.21.15.121:0/739470118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce88101750 0x7fce88196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.725+0000 7fce8ce70700 1 -- 172.21.15.121:0/739470118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce88101750 msgr2=0x7fce88196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.725+0000 7fce8ce70700 1 --2- 172.21.15.121:0/739470118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce88101750 0x7fce88196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.725+0000 7fce8ce70700 1 -- 172.21.15.121:0/739470118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce740095f0 con 0x7fce88100da0 2024-04-05T21:43:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.725+0000 7fce8ce70700 1 --2- 172.21.15.121:0/739470118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce88100da0 0x7fce88196650 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00c930 tx=0x7fce7c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.725+0000 7fce85ffb700 1 -- 172.21.15.121:0/739470118 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce7c007ad0 con 0x7fce88100da0 2024-04-05T21:43:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.725+0000 7fce85ffb700 1 -- 172.21.15.121:0/739470118 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce7c007c30 con 0x7fce88100da0 2024-04-05T21:43:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.725+0000 7fce8f0d4700 1 -- 172.21.15.121:0/739470118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8819b170 con 0x7fce88100da0 2024-04-05T21:43:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.725+0000 7fce85ffb700 1 -- 172.21.15.121:0/739470118 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce7c018630 con 0x7fce88100da0 2024-04-05T21:43:22.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.726+0000 7fce8f0d4700 1 -- 172.21.15.121:0/739470118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8819b6c0 con 0x7fce88100da0 2024-04-05T21:43:22.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.727+0000 7fce8f0d4700 1 -- 172.21.15.121:0/739470118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce881091a0 con 0x7fce88100da0 2024-04-05T21:43:22.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.727+0000 7fce85ffb700 1 -- 172.21.15.121:0/739470118 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fce7c00f450 con 0x7fce88100da0 2024-04-05T21:43:22.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.727+0000 7fce85ffb700 1 --2- 172.21.15.121:0/739470118 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce7806cce0 0x7fce7806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:22.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.727+0000 7fce85ffb700 1 -- 172.21.15.121:0/739470118 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fce7c08c780 con 0x7fce88100da0 2024-04-05T21:43:22.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.727+0000 7fce87fff700 1 --2- 172.21.15.121:0/739470118 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce7806cce0 0x7fce7806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:22.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.728+0000 7fce87fff700 1 --2- 172.21.15.121:0/739470118 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce7806cce0 0x7fce7806f1a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fce88197c70 tx=0x7fce7401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:22.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.732+0000 7fce85ffb700 1 -- 172.21.15.121:0/739470118 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fce7c056920 con 0x7fce88100da0 2024-04-05T21:43:22.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.944+0000 7fce8f0d4700 1 -- 172.21.15.121:0/739470118 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce8802cd60 con 0x7fce7806cce0 2024-04-05T21:43:22.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.945+0000 7fce85ffb700 1 -- 172.21.15.121:0/739470118 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fce8802cd60 con 0x7fce7806cce0 2024-04-05T21:43:22.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.947+0000 7fce8f0d4700 1 -- 172.21.15.121:0/739470118 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce7806cce0 msgr2=0x7fce7806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:22.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.947+0000 7fce8f0d4700 1 --2- 172.21.15.121:0/739470118 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce7806cce0 0x7fce7806f1a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fce88197c70 tx=0x7fce7401a040 comp rx=0 tx=0).stop 2024-04-05T21:43:22.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.947+0000 7fce8f0d4700 1 -- 172.21.15.121:0/739470118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce88100da0 msgr2=0x7fce88196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:22.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.947+0000 7fce8f0d4700 1 --2- 172.21.15.121:0/739470118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce88100da0 0x7fce88196650 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00c930 tx=0x7fce7c00ccf0 comp rx=0 tx=0).stop 2024-04-05T21:43:22.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.947+0000 7fce8f0d4700 1 -- 172.21.15.121:0/739470118 shutdown_connections 2024-04-05T21:43:22.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.947+0000 7fce8f0d4700 1 --2- 172.21.15.121:0/739470118 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce7806cce0 0x7fce7806f1a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:22.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.947+0000 7fce8f0d4700 1 --2- 172.21.15.121:0/739470118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce88100da0 0x7fce88196650 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:22.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.947+0000 7fce8f0d4700 1 --2- 172.21.15.121:0/739470118 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce88101750 0x7fce88196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:22.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.947+0000 7fce8f0d4700 1 -- 172.21.15.121:0/739470118 >> 172.21.15.121:0/739470118 conn(0x7fce880fc630 msgr2=0x7fce880fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:22.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.947+0000 7fce8f0d4700 1 -- 172.21.15.121:0/739470118 shutdown_connections 2024-04-05T21:43:22.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:22.948+0000 7fce8f0d4700 1 -- 172.21.15.121:0/739470118 wait complete. 2024-04-05T21:43:22.958 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:43:23.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.051+0000 7f994deef700 1 -- 172.21.15.121:0/2253906145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99480ffee0 msgr2=0x7f9948105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:23.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.051+0000 7f994deef700 1 --2- 172.21.15.121:0/2253906145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99480ffee0 0x7f9948105810 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f993c009910 tx=0x7f993c009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:23.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.051+0000 7f994deef700 1 -- 172.21.15.121:0/2253906145 shutdown_connections 2024-04-05T21:43:23.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.051+0000 7f994deef700 1 --2- 172.21.15.121:0/2253906145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99480ffee0 0x7f9948105810 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.051+0000 7f994deef700 1 --2- 172.21.15.121:0/2253906145 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99480ff530 0x7f99480ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.051+0000 7f994deef700 1 -- 172.21.15.121:0/2253906145 >> 172.21.15.121:0/2253906145 conn(0x7f9948074d20 msgr2=0x7f9948075130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:23.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.051+0000 7f994deef700 1 -- 172.21.15.121:0/2253906145 shutdown_connections 2024-04-05T21:43:23.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.051+0000 7f994deef700 1 -- 172.21.15.121:0/2253906145 wait complete. 2024-04-05T21:43:23.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f994deef700 1 Processor -- start 2024-04-05T21:43:23.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f994deef700 1 -- start start 2024-04-05T21:43:23.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f994deef700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99480ff530 0x7f9948196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:23.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f994deef700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99480ffee0 0x7f9948196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:23.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f994deef700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9948197290 con 0x7f99480ff530 2024-04-05T21:43:23.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f994deef700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994819b020 con 0x7f99480ffee0 2024-04-05T21:43:23.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f99477fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99480ff530 0x7f9948196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f99477fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99480ff530 0x7f9948196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60422/0 (socket says 172.21.15.121:60422) 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f99477fe700 1 -- 172.21.15.121:0/1242172702 learned_addr learned my addr 172.21.15.121:0/1242172702 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f9946ffd700 1 --2- 172.21.15.121:0/1242172702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99480ffee0 0x7f9948196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f99477fe700 1 -- 172.21.15.121:0/1242172702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99480ffee0 msgr2=0x7f9948196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f99477fe700 1 --2- 172.21.15.121:0/1242172702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99480ffee0 0x7f9948196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f99477fe700 1 -- 172.21.15.121:0/1242172702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c0095f0 con 0x7f99480ff530 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.052+0000 7f9946ffd700 1 --2- 172.21.15.121:0/1242172702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99480ffee0 0x7f9948196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.053+0000 7f99477fe700 1 --2- 172.21.15.121:0/1242172702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99480ff530 0x7f9948196670 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f993800c8a0 tx=0x7f993800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.053+0000 7f9944ff9700 1 -- 172.21.15.121:0/1242172702 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f993800cea0 con 0x7f99480ff530 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.053+0000 7f994deef700 1 -- 172.21.15.121:0/1242172702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994819b300 con 0x7f99480ff530 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.053+0000 7f9944ff9700 1 -- 172.21.15.121:0/1242172702 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9938004830 con 0x7f99480ff530 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.053+0000 7f9944ff9700 1 -- 172.21.15.121:0/1242172702 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9938005500 con 0x7f99480ff530 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.053+0000 7f994deef700 1 -- 172.21.15.121:0/1242172702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994819b850 con 0x7f99480ff530 2024-04-05T21:43:23.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.054+0000 7f994deef700 1 -- 172.21.15.121:0/1242172702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9948109110 con 0x7f99480ff530 2024-04-05T21:43:23.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.054+0000 7f9944ff9700 1 -- 172.21.15.121:0/1242172702 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f99380049a0 con 0x7f99480ff530 2024-04-05T21:43:23.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.054+0000 7f9944ff9700 1 --2- 172.21.15.121:0/1242172702 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f993006cce0 0x7f993006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:23.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.055+0000 7f9944ff9700 1 -- 172.21.15.121:0/1242172702 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f993808c250 con 0x7f99480ff530 2024-04-05T21:43:23.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.055+0000 7f9946ffd700 1 --2- 172.21.15.121:0/1242172702 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f993006cce0 0x7f993006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:23.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.055+0000 7f9946ffd700 1 --2- 172.21.15.121:0/1242172702 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f993006cce0 0x7f993006f1a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f993c0059d0 tx=0x7f993c005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:23.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.059+0000 7f9944ff9700 1 -- 172.21.15.121:0/1242172702 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f99380562c0 con 0x7f99480ff530 2024-04-05T21:43:23.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.270+0000 7f994deef700 1 -- 172.21.15.121:0/1242172702 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f994819bbb0 con 0x7f993006cce0 2024-04-05T21:43:23.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.271+0000 7f9944ff9700 1 -- 172.21.15.121:0/1242172702 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f994819bbb0 con 0x7f993006cce0 2024-04-05T21:43:23.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.273+0000 7f994deef700 1 -- 172.21.15.121:0/1242172702 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f993006cce0 msgr2=0x7f993006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:23.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.273+0000 7f994deef700 1 --2- 172.21.15.121:0/1242172702 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f993006cce0 0x7f993006f1a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f993c0059d0 tx=0x7f993c005960 comp rx=0 tx=0).stop 2024-04-05T21:43:23.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.273+0000 7f994deef700 1 -- 172.21.15.121:0/1242172702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99480ff530 msgr2=0x7f9948196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:23.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.273+0000 7f994deef700 1 --2- 172.21.15.121:0/1242172702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99480ff530 0x7f9948196670 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f993800c8a0 tx=0x7f993800cc60 comp rx=0 tx=0).stop 2024-04-05T21:43:23.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.273+0000 7f994deef700 1 -- 172.21.15.121:0/1242172702 shutdown_connections 2024-04-05T21:43:23.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.273+0000 7f994deef700 1 --2- 172.21.15.121:0/1242172702 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f993006cce0 0x7f993006f1a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.273+0000 7f994deef700 1 --2- 172.21.15.121:0/1242172702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99480ff530 0x7f9948196670 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.273+0000 7f994deef700 1 --2- 172.21.15.121:0/1242172702 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99480ffee0 0x7f9948196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.273+0000 7f994deef700 1 -- 172.21.15.121:0/1242172702 >> 172.21.15.121:0/1242172702 conn(0x7f9948074d20 msgr2=0x7f99480fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:23.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.273+0000 7f994deef700 1 -- 172.21.15.121:0/1242172702 shutdown_connections 2024-04-05T21:43:23.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.273+0000 7f994deef700 1 -- 172.21.15.121:0/1242172702 wait complete. 2024-04-05T21:43:23.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.376+0000 7fa196f3d700 1 -- 172.21.15.121:0/1342003660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1900ffbd0 msgr2=0x7fa190100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:23.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.376+0000 7fa196f3d700 1 --2- 172.21.15.121:0/1342003660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1900ffbd0 0x7fa190100050 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa17c009910 tx=0x7fa17c009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:23.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.376+0000 7fa196f3d700 1 -- 172.21.15.121:0/1342003660 shutdown_connections 2024-04-05T21:43:23.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.376+0000 7fa196f3d700 1 --2- 172.21.15.121:0/1342003660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1900ffbd0 0x7fa190100050 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.376+0000 7fa196f3d700 1 --2- 172.21.15.121:0/1342003660 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa190105a40 0x7fa190105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.376+0000 7fa196f3d700 1 -- 172.21.15.121:0/1342003660 >> 172.21.15.121:0/1342003660 conn(0x7fa1900755a0 msgr2=0x7fa1900759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:23.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.377+0000 7fa196f3d700 1 -- 172.21.15.121:0/1342003660 shutdown_connections 2024-04-05T21:43:23.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.377+0000 7fa196f3d700 1 -- 172.21.15.121:0/1342003660 wait complete. 2024-04-05T21:43:23.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.377+0000 7fa196f3d700 1 Processor -- start 2024-04-05T21:43:23.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.377+0000 7fa196f3d700 1 -- start start 2024-04-05T21:43:23.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.377+0000 7fa196f3d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1900ffbd0 0x7fa190197450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:23.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.377+0000 7fa196f3d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa190105a40 0x7fa190197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:23.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.377+0000 7fa196f3d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa190198070 con 0x7fa190105a40 2024-04-05T21:43:23.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.377+0000 7fa196f3d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1901981b0 con 0x7fa1900ffbd0 2024-04-05T21:43:23.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.377+0000 7fa18ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa190105a40 0x7fa190197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:23.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.377+0000 7fa18ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa190105a40 0x7fa190197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60434/0 (socket says 172.21.15.121:60434) 2024-04-05T21:43:23.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.377+0000 7fa18ffff700 1 -- 172.21.15.121:0/3156125694 learned_addr learned my addr 172.21.15.121:0/3156125694 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:23.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.377+0000 7fa194cd9700 1 --2- 172.21.15.121:0/3156125694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1900ffbd0 0x7fa190197450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:23.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.378+0000 7fa18ffff700 1 -- 172.21.15.121:0/3156125694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1900ffbd0 msgr2=0x7fa190197450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:23.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.378+0000 7fa18ffff700 1 --2- 172.21.15.121:0/3156125694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1900ffbd0 0x7fa190197450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.378+0000 7fa18ffff700 1 -- 172.21.15.121:0/3156125694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c0095f0 con 0x7fa190105a40 2024-04-05T21:43:23.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.378+0000 7fa194cd9700 1 --2- 172.21.15.121:0/3156125694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1900ffbd0 0x7fa190197450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:43:23.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.378+0000 7fa18ffff700 1 --2- 172.21.15.121:0/3156125694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa190105a40 0x7fa190197990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa17c006010 tx=0x7fa17c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:23.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.378+0000 7fa18dffb700 1 -- 172.21.15.121:0/3156125694 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17c01d070 con 0x7fa190105a40 2024-04-05T21:43:23.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.378+0000 7fa18dffb700 1 -- 172.21.15.121:0/3156125694 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa17c00f460 con 0x7fa190105a40 2024-04-05T21:43:23.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.378+0000 7fa196f3d700 1 -- 172.21.15.121:0/3156125694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1901a04c0 con 0x7fa190105a40 2024-04-05T21:43:23.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.378+0000 7fa18dffb700 1 -- 172.21.15.121:0/3156125694 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17c021640 con 0x7fa190105a40 2024-04-05T21:43:23.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.378+0000 7fa196f3d700 1 -- 172.21.15.121:0/3156125694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1901a0930 con 0x7fa190105a40 2024-04-05T21:43:23.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.379+0000 7fa196f3d700 1 -- 172.21.15.121:0/3156125694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa190107fd0 con 0x7fa190105a40 2024-04-05T21:43:23.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.380+0000 7fa18dffb700 1 -- 172.21.15.121:0/3156125694 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa17c00fa20 con 0x7fa190105a40 2024-04-05T21:43:23.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.380+0000 7fa18dffb700 1 --2- 172.21.15.121:0/3156125694 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa18006cce0 0x7fa18006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:23.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.380+0000 7fa18dffb700 1 -- 172.21.15.121:0/3156125694 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa17c08dd60 con 0x7fa190105a40 2024-04-05T21:43:23.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.380+0000 7fa194cd9700 1 --2- 172.21.15.121:0/3156125694 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa18006cce0 0x7fa18006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:23.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.381+0000 7fa194cd9700 1 --2- 172.21.15.121:0/3156125694 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa18006cce0 0x7fa18006f1a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa184005950 tx=0x7fa184005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:23.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.384+0000 7fa18dffb700 1 -- 172.21.15.121:0/3156125694 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa17c057ed0 con 0x7fa190105a40 2024-04-05T21:43:23.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.591+0000 7fa196f3d700 1 -- 172.21.15.121:0/3156125694 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa19002cd50 con 0x7fa18006cce0 2024-04-05T21:43:23.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.601+0000 7fa18dffb700 1 -- 172.21.15.121:0/3156125694 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fa19002cd50 con 0x7fa18006cce0 2024-04-05T21:43:23.602 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:43:23.602 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (14m) 8m ago 25m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:43:23.602 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (25m) 8m ago 25m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:43:23.602 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (24m) 7m ago 24m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:43:23.602 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (12m) 8m ago 25m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:43:23.603 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (12m) 7m ago 24m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:43:23.603 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (13m) 8m ago 25m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:43:23.603 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (8m) 8m ago 19m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:43:23.603 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (8m) 8m ago 19m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:43:23.603 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (7m) 7m ago 19m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:43:23.603 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (19m) 7m ago 19m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:43:23.603 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (16m) 8m ago 27m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:43:23.603 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (15m) 7m ago 24m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:43:23.603 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (12m) 8m ago 27m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:43:23.603 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (12m) 7m ago 24m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:43:23.603 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (15m) 8m ago 25m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:43:23.603 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (14m) 7m ago 24m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:43:23.604 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (11m) 8m ago 23m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:43:23.604 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (11m) 8m ago 23m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:43:23.604 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (10m) 8m ago 22m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:43:23.604 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (10m) 7m ago 22m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:43:23.604 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (9m) 7m ago 21m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:43:23.604 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (8m) 7m ago 21m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:43:23.604 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (14m) 8m ago 25m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:43:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.603+0000 7fa196f3d700 1 -- 172.21.15.121:0/3156125694 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa18006cce0 msgr2=0x7fa18006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.603+0000 7fa196f3d700 1 --2- 172.21.15.121:0/3156125694 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa18006cce0 0x7fa18006f1a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa184005950 tx=0x7fa184005f50 comp rx=0 tx=0).stop 2024-04-05T21:43:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.603+0000 7fa196f3d700 1 -- 172.21.15.121:0/3156125694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa190105a40 msgr2=0x7fa190197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.603+0000 7fa196f3d700 1 --2- 172.21.15.121:0/3156125694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa190105a40 0x7fa190197990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa17c006010 tx=0x7fa17c00be20 comp rx=0 tx=0).stop 2024-04-05T21:43:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.603+0000 7fa196f3d700 1 -- 172.21.15.121:0/3156125694 shutdown_connections 2024-04-05T21:43:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.603+0000 7fa196f3d700 1 --2- 172.21.15.121:0/3156125694 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa18006cce0 0x7fa18006f1a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.603+0000 7fa196f3d700 1 --2- 172.21.15.121:0/3156125694 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1900ffbd0 0x7fa190197450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.603+0000 7fa196f3d700 1 --2- 172.21.15.121:0/3156125694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa190105a40 0x7fa190197990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.603+0000 7fa196f3d700 1 -- 172.21.15.121:0/3156125694 >> 172.21.15.121:0/3156125694 conn(0x7fa1900755a0 msgr2=0x7fa1900fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.604+0000 7fa196f3d700 1 -- 172.21.15.121:0/3156125694 shutdown_connections 2024-04-05T21:43:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.604+0000 7fa196f3d700 1 -- 172.21.15.121:0/3156125694 wait complete. 2024-04-05T21:43:23.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.705+0000 7feef19fa700 1 -- 172.21.15.121:0/3135130713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeec1025e0 msgr2=0x7feeec10aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.705+0000 7feef19fa700 1 --2- 172.21.15.121:0/3135130713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeec1025e0 0x7feeec10aad0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7feedc009910 tx=0x7feedc009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.705+0000 7feef19fa700 1 -- 172.21.15.121:0/3135130713 shutdown_connections 2024-04-05T21:43:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.705+0000 7feef19fa700 1 --2- 172.21.15.121:0/3135130713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeec1025e0 0x7feeec10aad0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.705+0000 7feef19fa700 1 --2- 172.21.15.121:0/3135130713 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeec101cc0 0x7feeec1020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.705+0000 7feef19fa700 1 -- 172.21.15.121:0/3135130713 >> 172.21.15.121:0/3135130713 conn(0x7feeec0fb4f0 msgr2=0x7feeec0fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.705+0000 7feef19fa700 1 -- 172.21.15.121:0/3135130713 shutdown_connections 2024-04-05T21:43:23.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.705+0000 7feef19fa700 1 -- 172.21.15.121:0/3135130713 wait complete. 2024-04-05T21:43:23.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feef19fa700 1 Processor -- start 2024-04-05T21:43:23.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feef19fa700 1 -- start start 2024-04-05T21:43:23.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feef19fa700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeec101cc0 0x7feeec196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:23.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feef19fa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeec1025e0 0x7feeec196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:23.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feef19fa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec197260 con 0x7feeec1025e0 2024-04-05T21:43:23.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feef19fa700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeec19aff0 con 0x7feeec101cc0 2024-04-05T21:43:23.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feeea7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeec1025e0 0x7feeec196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:23.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feeea7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeec1025e0 0x7feeec196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60448/0 (socket says 172.21.15.121:60448) 2024-04-05T21:43:23.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feeea7fc700 1 -- 172.21.15.121:0/2594698627 learned_addr learned my addr 172.21.15.121:0/2594698627 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:23.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feeeaffd700 1 --2- 172.21.15.121:0/2594698627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeec101cc0 0x7feeec196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:23.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feeea7fc700 1 -- 172.21.15.121:0/2594698627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeec101cc0 msgr2=0x7feeec196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:23.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feeea7fc700 1 --2- 172.21.15.121:0/2594698627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeec101cc0 0x7feeec196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feeea7fc700 1 -- 172.21.15.121:0/2594698627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc0095f0 con 0x7feeec1025e0 2024-04-05T21:43:23.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.706+0000 7feeeaffd700 1 --2- 172.21.15.121:0/2594698627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeec101cc0 0x7feeec196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:43:23.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.707+0000 7feeea7fc700 1 --2- 172.21.15.121:0/2594698627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeec1025e0 0x7feeec196b80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7feedc005b40 tx=0x7feedc00bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:23.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.707+0000 7feef09f8700 1 -- 172.21.15.121:0/2594698627 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feedc01d070 con 0x7feeec1025e0 2024-04-05T21:43:23.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.707+0000 7feef19fa700 1 -- 172.21.15.121:0/2594698627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeec19b270 con 0x7feeec1025e0 2024-04-05T21:43:23.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.707+0000 7feef09f8700 1 -- 172.21.15.121:0/2594698627 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feedc00bed0 con 0x7feeec1025e0 2024-04-05T21:43:23.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.707+0000 7feef09f8700 1 -- 172.21.15.121:0/2594698627 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feedc00fd80 con 0x7feeec1025e0 2024-04-05T21:43:23.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.707+0000 7feef19fa700 1 -- 172.21.15.121:0/2594698627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeec19b700 con 0x7feeec1025e0 2024-04-05T21:43:23.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.708+0000 7feef19fa700 1 -- 172.21.15.121:0/2594698627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeec1081d0 con 0x7feeec1025e0 2024-04-05T21:43:23.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.708+0000 7feef09f8700 1 -- 172.21.15.121:0/2594698627 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7feedc021410 con 0x7feeec1025e0 2024-04-05T21:43:23.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.709+0000 7feef09f8700 1 --2- 172.21.15.121:0/2594698627 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feed406cce0 0x7feed406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:23.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.709+0000 7feef09f8700 1 -- 172.21.15.121:0/2594698627 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7feedc08d770 con 0x7feeec1025e0 2024-04-05T21:43:23.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.709+0000 7feeeaffd700 1 --2- 172.21.15.121:0/2594698627 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feed406cce0 0x7feed406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:23.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.709+0000 7feeeaffd700 1 --2- 172.21.15.121:0/2594698627 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feed406cce0 0x7feed406f1a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7feee0009be0 tx=0x7feee0009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:23.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.713+0000 7feef09f8700 1 -- 172.21.15.121:0/2594698627 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7feedc0577e0 con 0x7feeec1025e0 2024-04-05T21:43:23.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:23 smithi121 ceph-mon[145859]: pgmap v514: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:23 smithi188 ceph-mon[107793]: pgmap v514: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:23.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.988+0000 7feef19fa700 1 -- 172.21.15.121:0/2594698627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7feeec04ea90 con 0x7feeec1025e0 2024-04-05T21:43:23.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.989+0000 7feef09f8700 1 -- 172.21.15.121:0/2594698627 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7feedc05ae00 con 0x7feeec1025e0 2024-04-05T21:43:23.990 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:43:23.990 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:43:23.990 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:43:23.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:43:23.990 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:43:23.990 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:43:23.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:43:23.991 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:43:23.991 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:43:23.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:43:23.991 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:43:23.991 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:43:23.991 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:43:23.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:43:23.991 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:43:23.991 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:43:23.992 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:43:23.992 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:43:23.992 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:43:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.991+0000 7feef19fa700 1 -- 172.21.15.121:0/2594698627 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feed406cce0 msgr2=0x7feed406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.991+0000 7feef19fa700 1 --2- 172.21.15.121:0/2594698627 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feed406cce0 0x7feed406f1a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7feee0009be0 tx=0x7feee0009260 comp rx=0 tx=0).stop 2024-04-05T21:43:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.991+0000 7feef19fa700 1 -- 172.21.15.121:0/2594698627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeec1025e0 msgr2=0x7feeec196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.991+0000 7feef19fa700 1 --2- 172.21.15.121:0/2594698627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeec1025e0 0x7feeec196b80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7feedc005b40 tx=0x7feedc00bb20 comp rx=0 tx=0).stop 2024-04-05T21:43:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.991+0000 7feef19fa700 1 -- 172.21.15.121:0/2594698627 shutdown_connections 2024-04-05T21:43:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.991+0000 7feef19fa700 1 --2- 172.21.15.121:0/2594698627 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feed406cce0 0x7feed406f1a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.991+0000 7feef19fa700 1 --2- 172.21.15.121:0/2594698627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7feeec101cc0 0x7feeec196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.991+0000 7feef19fa700 1 --2- 172.21.15.121:0/2594698627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feeec1025e0 0x7feeec196b80 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.991+0000 7feef19fa700 1 -- 172.21.15.121:0/2594698627 >> 172.21.15.121:0/2594698627 conn(0x7feeec0fb4f0 msgr2=0x7feeec105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.991+0000 7feef19fa700 1 -- 172.21.15.121:0/2594698627 shutdown_connections 2024-04-05T21:43:23.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:23.991+0000 7feef19fa700 1 -- 172.21.15.121:0/2594698627 wait complete. 2024-04-05T21:43:24.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.093+0000 7f421d44c700 1 -- 172.21.15.121:0/2264395851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4218101780 msgr2=0x7f42181057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:24.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.093+0000 7f421d44c700 1 --2- 172.21.15.121:0/2264395851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4218101780 0x7f42181057d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f420c009910 tx=0x7f420c009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:24.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.093+0000 7f421d44c700 1 -- 172.21.15.121:0/2264395851 shutdown_connections 2024-04-05T21:43:24.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.093+0000 7f421d44c700 1 --2- 172.21.15.121:0/2264395851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4218101780 0x7f42181057d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.093+0000 7f421d44c700 1 --2- 172.21.15.121:0/2264395851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4218100dd0 0x7f42181011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.093+0000 7f421d44c700 1 -- 172.21.15.121:0/2264395851 >> 172.21.15.121:0/2264395851 conn(0x7f42180fc640 msgr2=0x7f42180fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:24.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.093+0000 7f421d44c700 1 -- 172.21.15.121:0/2264395851 shutdown_connections 2024-04-05T21:43:24.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.093+0000 7f421d44c700 1 -- 172.21.15.121:0/2264395851 wait complete. 2024-04-05T21:43:24.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.094+0000 7f421d44c700 1 Processor -- start 2024-04-05T21:43:24.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.094+0000 7f421d44c700 1 -- start start 2024-04-05T21:43:24.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f421d44c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4218100dd0 0x7f4218073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:24.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f421d44c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4218101780 0x7f42180735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:24.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f421d44c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4218073b10 con 0x7f4218101780 2024-04-05T21:43:24.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f421d44c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4218073c50 con 0x7f4218100dd0 2024-04-05T21:43:24.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f42167fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4218101780 0x7f42180735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:24.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f42167fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4218101780 0x7f42180735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60458/0 (socket says 172.21.15.121:60458) 2024-04-05T21:43:24.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f42167fc700 1 -- 172.21.15.121:0/3279441339 learned_addr learned my addr 172.21.15.121:0/3279441339 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:24.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f4216ffd700 1 --2- 172.21.15.121:0/3279441339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4218100dd0 0x7f4218073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f42167fc700 1 -- 172.21.15.121:0/3279441339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4218100dd0 msgr2=0x7f4218073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f42167fc700 1 --2- 172.21.15.121:0/3279441339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4218100dd0 0x7f4218073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f42167fc700 1 -- 172.21.15.121:0/3279441339 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c0095f0 con 0x7f4218101780 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f42167fc700 1 --2- 172.21.15.121:0/3279441339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4218101780 0x7f42180735d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f420c00bda0 tx=0x7f420c00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f41fffff700 1 -- 172.21.15.121:0/3279441339 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f420c01d070 con 0x7f4218101780 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f421d44c700 1 -- 172.21.15.121:0/3279441339 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4218073ed0 con 0x7f4218101780 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f41fffff700 1 -- 172.21.15.121:0/3279441339 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f420c00f460 con 0x7f4218101780 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f41fffff700 1 -- 172.21.15.121:0/3279441339 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f420c021640 con 0x7f4218101780 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.095+0000 7f421d44c700 1 -- 172.21.15.121:0/3279441339 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42181a2c00 con 0x7f4218101780 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.096+0000 7f421d44c700 1 -- 172.21.15.121:0/3279441339 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42181091d0 con 0x7f4218101780 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.097+0000 7f41fffff700 1 -- 172.21.15.121:0/3279441339 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f420c00fa60 con 0x7f4218101780 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.097+0000 7f41fffff700 1 --2- 172.21.15.121:0/3279441339 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f420006cce0 0x7f420006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.097+0000 7f41fffff700 1 -- 172.21.15.121:0/3279441339 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f420c08d7c0 con 0x7f4218101780 2024-04-05T21:43:24.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.097+0000 7f4216ffd700 1 --2- 172.21.15.121:0/3279441339 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f420006cce0 0x7f420006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:24.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.098+0000 7f4216ffd700 1 --2- 172.21.15.121:0/3279441339 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f420006cce0 0x7f420006f1a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4208005950 tx=0x7f420800a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:24.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.102+0000 7f41fffff700 1 -- 172.21.15.121:0/3279441339 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f420c057830 con 0x7f4218101780 2024-04-05T21:43:24.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.332+0000 7f421d44c700 1 -- 172.21.15.121:0/3279441339 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f421804ea90 con 0x7f4218101780 2024-04-05T21:43:24.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.333+0000 7f41fffff700 1 -- 172.21.15.121:0/3279441339 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f420c05ae50 con 0x7f4218101780 2024-04-05T21:43:24.334 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:43:24.334 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:43:24.334 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:43:24.334 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:43:24.334 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:43:24.334 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:43:24.334 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:43:24.334 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:43:24.334 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:43:24.334 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:43:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:43:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:43:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:43:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:43:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:43:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:43:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:43:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:43:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:43:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:43:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:43:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:43:24.335 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:43:24.336 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:43:24.337 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:43:24.337 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:43:24.337 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:43:24.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.335+0000 7f421d44c700 1 -- 172.21.15.121:0/3279441339 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f420006cce0 msgr2=0x7f420006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:24.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.335+0000 7f421d44c700 1 --2- 172.21.15.121:0/3279441339 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f420006cce0 0x7f420006f1a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4208005950 tx=0x7f420800a300 comp rx=0 tx=0).stop 2024-04-05T21:43:24.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.335+0000 7f421d44c700 1 -- 172.21.15.121:0/3279441339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4218101780 msgr2=0x7f42180735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:24.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.335+0000 7f421d44c700 1 --2- 172.21.15.121:0/3279441339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4218101780 0x7f42180735d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f420c00bda0 tx=0x7f420c00bdd0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.335+0000 7f421d44c700 1 -- 172.21.15.121:0/3279441339 shutdown_connections 2024-04-05T21:43:24.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.335+0000 7f421d44c700 1 --2- 172.21.15.121:0/3279441339 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f420006cce0 0x7f420006f1a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.335+0000 7f421d44c700 1 --2- 172.21.15.121:0/3279441339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4218100dd0 0x7f4218073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.335+0000 7f421d44c700 1 --2- 172.21.15.121:0/3279441339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4218101780 0x7f42180735d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.335+0000 7f421d44c700 1 -- 172.21.15.121:0/3279441339 >> 172.21.15.121:0/3279441339 conn(0x7f42180fc640 msgr2=0x7f42180fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:24.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.335+0000 7f421d44c700 1 -- 172.21.15.121:0/3279441339 shutdown_connections 2024-04-05T21:43:24.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.335+0000 7f421d44c700 1 -- 172.21.15.121:0/3279441339 wait complete. 2024-04-05T21:43:24.338 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:43:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.440+0000 7f04b4888700 1 -- 172.21.15.121:0/2927048084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b0105a40 msgr2=0x7f04b0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.440+0000 7f04b4888700 1 --2- 172.21.15.121:0/2927048084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b0105a40 0x7f04b0105e20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009910 tx=0x7f04a4009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.440+0000 7f04b4888700 1 -- 172.21.15.121:0/2927048084 shutdown_connections 2024-04-05T21:43:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.440+0000 7f04b4888700 1 --2- 172.21.15.121:0/2927048084 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b00ffbd0 0x7f04b0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.440+0000 7f04b4888700 1 --2- 172.21.15.121:0/2927048084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b0105a40 0x7f04b0105e20 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.440+0000 7f04b4888700 1 -- 172.21.15.121:0/2927048084 >> 172.21.15.121:0/2927048084 conn(0x7f04b00755a0 msgr2=0x7f04b00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.440+0000 7f04b4888700 1 -- 172.21.15.121:0/2927048084 shutdown_connections 2024-04-05T21:43:24.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.440+0000 7f04b4888700 1 -- 172.21.15.121:0/2927048084 wait complete. 2024-04-05T21:43:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04b4888700 1 Processor -- start 2024-04-05T21:43:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04b4888700 1 -- start start 2024-04-05T21:43:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04b4888700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b00ffbd0 0x7f04b01952d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04b4888700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b0105a40 0x7f04b0195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04b4888700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0195d50 con 0x7f04b0105a40 2024-04-05T21:43:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04b4888700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b0195ec0 con 0x7f04b00ffbd0 2024-04-05T21:43:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04add9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b0105a40 0x7f04b0195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04add9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b0105a40 0x7f04b0195810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55088/0 (socket says 172.21.15.121:55088) 2024-04-05T21:43:24.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04add9b700 1 -- 172.21.15.121:0/3489281739 learned_addr learned my addr 172.21.15.121:0/3489281739 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04ae59c700 1 --2- 172.21.15.121:0/3489281739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b00ffbd0 0x7f04b01952d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04add9b700 1 -- 172.21.15.121:0/3489281739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b00ffbd0 msgr2=0x7f04b01952d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04add9b700 1 --2- 172.21.15.121:0/3489281739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b00ffbd0 0x7f04b01952d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04add9b700 1 -- 172.21.15.121:0/3489281739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a40095f0 con 0x7f04b0105a40 2024-04-05T21:43:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.441+0000 7f04add9b700 1 --2- 172.21.15.121:0/3489281739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b0105a40 0x7f04b0195810 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f04a000d900 tx=0x7f04a000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.442+0000 7f049f7fe700 1 -- 172.21.15.121:0/3489281739 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04a0004a00 con 0x7f04b0105a40 2024-04-05T21:43:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.442+0000 7f049f7fe700 1 -- 172.21.15.121:0/3489281739 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04a0005020 con 0x7f04b0105a40 2024-04-05T21:43:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.442+0000 7f04b4888700 1 -- 172.21.15.121:0/3489281739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b01a2940 con 0x7f04b0105a40 2024-04-05T21:43:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.442+0000 7f049f7fe700 1 -- 172.21.15.121:0/3489281739 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04a0009b90 con 0x7f04b0105a40 2024-04-05T21:43:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.442+0000 7f04b4888700 1 -- 172.21.15.121:0/3489281739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b01a2ca0 con 0x7f04b0105a40 2024-04-05T21:43:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.443+0000 7f04b4888700 1 -- 172.21.15.121:0/3489281739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b0108040 con 0x7f04b0105a40 2024-04-05T21:43:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.443+0000 7f049f7fe700 1 -- 172.21.15.121:0/3489281739 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f04a0009cf0 con 0x7f04b0105a40 2024-04-05T21:43:24.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.443+0000 7f049f7fe700 1 --2- 172.21.15.121:0/3489281739 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f049806cce0 0x7f049806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.443+0000 7f049f7fe700 1 -- 172.21.15.121:0/3489281739 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f04a008c360 con 0x7f04b0105a40 2024-04-05T21:43:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.443+0000 7f04ae59c700 1 --2- 172.21.15.121:0/3489281739 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f049806cce0 0x7f049806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:24.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.444+0000 7f04ae59c700 1 --2- 172.21.15.121:0/3489281739 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f049806cce0 0x7f049806f1a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f04a4006010 tx=0x7f04a401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:24.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.448+0000 7f049f7fe700 1 -- 172.21.15.121:0/3489281739 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f04a0059ee0 con 0x7f04b0105a40 2024-04-05T21:43:24.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.658+0000 7f04b4888700 1 -- 172.21.15.121:0/3489281739 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f04b002cce0 con 0x7f049806cce0 2024-04-05T21:43:24.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.659+0000 7f049f7fe700 1 -- 172.21.15.121:0/3489281739 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f04b002cce0 con 0x7f049806cce0 2024-04-05T21:43:24.660 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:43:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:43:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:43:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:43:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:43:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:43:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:43:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:43:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:43:24.660 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:43:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:43:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:43:24.661 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:43:24.661 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:43:24.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.661+0000 7f04b4888700 1 -- 172.21.15.121:0/3489281739 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f049806cce0 msgr2=0x7f049806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:24.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.661+0000 7f04b4888700 1 --2- 172.21.15.121:0/3489281739 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f049806cce0 0x7f049806f1a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f04a4006010 tx=0x7f04a401a040 comp rx=0 tx=0).stop 2024-04-05T21:43:24.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.661+0000 7f04b4888700 1 -- 172.21.15.121:0/3489281739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b0105a40 msgr2=0x7f04b0195810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:24.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.661+0000 7f04b4888700 1 --2- 172.21.15.121:0/3489281739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b0105a40 0x7f04b0195810 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f04a000d900 tx=0x7f04a000dc10 comp rx=0 tx=0).stop 2024-04-05T21:43:24.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.661+0000 7f04b4888700 1 -- 172.21.15.121:0/3489281739 shutdown_connections 2024-04-05T21:43:24.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.662+0000 7f04b4888700 1 --2- 172.21.15.121:0/3489281739 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f049806cce0 0x7f049806f1a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.662+0000 7f04b4888700 1 --2- 172.21.15.121:0/3489281739 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f04b00ffbd0 0x7f04b01952d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.662+0000 7f04b4888700 1 --2- 172.21.15.121:0/3489281739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b0105a40 0x7f04b0195810 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.662+0000 7f04b4888700 1 -- 172.21.15.121:0/3489281739 >> 172.21.15.121:0/3489281739 conn(0x7f04b00755a0 msgr2=0x7f04b00fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:24.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.662+0000 7f04b4888700 1 -- 172.21.15.121:0/3489281739 shutdown_connections 2024-04-05T21:43:24.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.662+0000 7f04b4888700 1 -- 172.21.15.121:0/3489281739 wait complete. 2024-04-05T21:43:24.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:24 smithi121 ceph-mon[145859]: from='client.34790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:24.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:24 smithi121 ceph-mon[145859]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:24.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:24 smithi121 ceph-mon[145859]: from='client.34798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:24.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:24 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2594698627' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:43:24.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:24 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3279441339' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:43:24.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.764+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/1284594454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e98068670 msgr2=0x7f1e98068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:24.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.764+0000 7f1e9e5cd700 1 --2- 172.21.15.121:0/1284594454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e98068670 0x7f1e98068a50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1e88009910 tx=0x7f1e88009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:24.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.764+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/1284594454 shutdown_connections 2024-04-05T21:43:24.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.764+0000 7f1e9e5cd700 1 --2- 172.21.15.121:0/1284594454 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e98069020 0x7f1e98105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.764+0000 7f1e9e5cd700 1 --2- 172.21.15.121:0/1284594454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e98068670 0x7f1e98068a50 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.764+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/1284594454 >> 172.21.15.121:0/1284594454 conn(0x7f1e980755a0 msgr2=0x7f1e980759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:24.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.764+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/1284594454 shutdown_connections 2024-04-05T21:43:24.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.764+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/1284594454 wait complete. 2024-04-05T21:43:24.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.765+0000 7f1e9e5cd700 1 Processor -- start 2024-04-05T21:43:24.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.765+0000 7f1e9e5cd700 1 -- start start 2024-04-05T21:43:24.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.765+0000 7f1e9e5cd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e98068670 0x7f1e98196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:24.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.765+0000 7f1e9e5cd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e98069020 0x7f1e98196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:24.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.765+0000 7f1e9e5cd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e98197290 con 0x7f1e98068670 2024-04-05T21:43:24.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.765+0000 7f1e9e5cd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9819b020 con 0x7f1e98069020 2024-04-05T21:43:24.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.765+0000 7f1e97fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e98068670 0x7f1e98196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:24.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.765+0000 7f1e97fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e98068670 0x7f1e98196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55114/0 (socket says 172.21.15.121:55114) 2024-04-05T21:43:24.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.765+0000 7f1e97fff700 1 -- 172.21.15.121:0/3803173915 learned_addr learned my addr 172.21.15.121:0/3803173915 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:24.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.765+0000 7f1e977fe700 1 --2- 172.21.15.121:0/3803173915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e98069020 0x7f1e98196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:24.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.765+0000 7f1e97fff700 1 -- 172.21.15.121:0/3803173915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e98069020 msgr2=0x7f1e98196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:24.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.765+0000 7f1e97fff700 1 --2- 172.21.15.121:0/3803173915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e98069020 0x7f1e98196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:24.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.766+0000 7f1e97fff700 1 -- 172.21.15.121:0/3803173915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e880095f0 con 0x7f1e98068670 2024-04-05T21:43:24.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.766+0000 7f1e977fe700 1 --2- 172.21.15.121:0/3803173915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e98069020 0x7f1e98196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:43:24.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.766+0000 7f1e97fff700 1 --2- 172.21.15.121:0/3803173915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e98068670 0x7f1e98196670 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1e88006010 tx=0x7f1e8800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:24.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.766+0000 7f1e957fa700 1 -- 172.21.15.121:0/3803173915 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e8801d070 con 0x7f1e98068670 2024-04-05T21:43:24.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.766+0000 7f1e957fa700 1 -- 172.21.15.121:0/3803173915 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e8800f460 con 0x7f1e98068670 2024-04-05T21:43:24.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.766+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/3803173915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9819b2a0 con 0x7f1e98068670 2024-04-05T21:43:24.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.766+0000 7f1e957fa700 1 -- 172.21.15.121:0/3803173915 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e88021620 con 0x7f1e98068670 2024-04-05T21:43:24.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.766+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/3803173915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e9819b790 con 0x7f1e98068670 2024-04-05T21:43:24.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.767+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/3803173915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e98109130 con 0x7f1e98068670 2024-04-05T21:43:24.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.767+0000 7f1e957fa700 1 -- 172.21.15.121:0/3803173915 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1e88003a60 con 0x7f1e98068670 2024-04-05T21:43:24.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.768+0000 7f1e957fa700 1 --2- 172.21.15.121:0/3803173915 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1e8006cce0 0x7f1e8006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:24.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.768+0000 7f1e957fa700 1 -- 172.21.15.121:0/3803173915 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1e8808d730 con 0x7f1e98068670 2024-04-05T21:43:24.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.768+0000 7f1e977fe700 1 --2- 172.21.15.121:0/3803173915 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1e8006cce0 0x7f1e8006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:24.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.768+0000 7f1e977fe700 1 --2- 172.21.15.121:0/3803173915 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1e8006cce0 0x7f1e8006f1a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c005950 tx=0x7f1e8c0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:24.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:24.772+0000 7f1e957fa700 1 -- 172.21.15.121:0/3803173915 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1e88057820 con 0x7f1e98068670 2024-04-05T21:43:24.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:24 smithi188 ceph-mon[107793]: from='client.34790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:24.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:24 smithi188 ceph-mon[107793]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:24.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:24 smithi188 ceph-mon[107793]: from='client.34798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:24.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:24 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2594698627' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:43:24.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:24 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3279441339' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:43:25.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:25.045+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/3803173915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1e9802cd30 con 0x7f1e98068670 2024-04-05T21:43:25.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:25.046+0000 7f1e957fa700 1 -- 172.21.15.121:0/3803173915 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f1e8805ae40 con 0x7f1e98068670 2024-04-05T21:43:25.047 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:43:25.047 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:43:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:43:25.047 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:43:25.047 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:43:25.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:25.048+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/3803173915 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1e8006cce0 msgr2=0x7f1e8006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:25.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:25.048+0000 7f1e9e5cd700 1 --2- 172.21.15.121:0/3803173915 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1e8006cce0 0x7f1e8006f1a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c005950 tx=0x7f1e8c0058e0 comp rx=0 tx=0).stop 2024-04-05T21:43:25.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:25.048+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/3803173915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e98068670 msgr2=0x7f1e98196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:25.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:25.048+0000 7f1e9e5cd700 1 --2- 172.21.15.121:0/3803173915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e98068670 0x7f1e98196670 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1e88006010 tx=0x7f1e8800be20 comp rx=0 tx=0).stop 2024-04-05T21:43:25.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:25.048+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/3803173915 shutdown_connections 2024-04-05T21:43:25.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:25.048+0000 7f1e9e5cd700 1 --2- 172.21.15.121:0/3803173915 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1e8006cce0 0x7f1e8006f1a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:25.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:25.048+0000 7f1e9e5cd700 1 --2- 172.21.15.121:0/3803173915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e98068670 0x7f1e98196670 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:25.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:25.048+0000 7f1e9e5cd700 1 --2- 172.21.15.121:0/3803173915 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1e98069020 0x7f1e98196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:25.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:25.048+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/3803173915 >> 172.21.15.121:0/3803173915 conn(0x7f1e980755a0 msgr2=0x7f1e980fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:25.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:25.048+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/3803173915 shutdown_connections 2024-04-05T21:43:25.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:25.048+0000 7f1e9e5cd700 1 -- 172.21.15.121:0/3803173915 wait complete. 2024-04-05T21:43:25.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:25 smithi121 ceph-mon[145859]: pgmap v515: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:25.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:25 smithi121 ceph-mon[145859]: from='client.34810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:25.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:25 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3803173915' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:43:25.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:25 smithi188 ceph-mon[107793]: pgmap v515: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:25.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:25 smithi188 ceph-mon[107793]: from='client.34810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:25.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:25 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3803173915' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:43:27.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:27 smithi121 ceph-mon[145859]: pgmap v516: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:27.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:27 smithi188 ceph-mon[107793]: pgmap v516: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:29 smithi121 ceph-mon[145859]: pgmap v517: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:29 smithi188 ceph-mon[107793]: pgmap v517: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:43:30.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:43:31.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:31 smithi121 ceph-mon[145859]: pgmap v518: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:31.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:31 smithi188 ceph-mon[107793]: pgmap v518: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:33.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:33 smithi121 ceph-mon[145859]: pgmap v519: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:33 smithi188 ceph-mon[107793]: pgmap v519: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:35.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:35 smithi121 ceph-mon[145859]: pgmap v520: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:35.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:35 smithi188 ceph-mon[107793]: pgmap v520: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:37 smithi121 ceph-mon[145859]: pgmap v521: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:37.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:37 smithi188 ceph-mon[107793]: pgmap v521: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:39.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:39 smithi121 ceph-mon[145859]: pgmap v522: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:39.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:39 smithi188 ceph-mon[107793]: pgmap v522: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:41.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:41 smithi121 ceph-mon[145859]: pgmap v523: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:41.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:41 smithi188 ceph-mon[107793]: pgmap v523: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:43.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:43 smithi121 ceph-mon[145859]: pgmap v524: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:43.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:43 smithi188 ceph-mon[107793]: pgmap v524: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:45 smithi121 ceph-mon[145859]: pgmap v525: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:43:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:45 smithi188 ceph-mon[107793]: pgmap v525: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:43:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:47 smithi121 ceph-mon[145859]: pgmap v526: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:47 smithi188 ceph-mon[107793]: pgmap v526: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:49.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:49 smithi121 ceph-mon[145859]: pgmap v527: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:49.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:49 smithi188 ceph-mon[107793]: pgmap v527: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:51.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:51 smithi121 ceph-mon[145859]: pgmap v528: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:51 smithi188 ceph-mon[107793]: pgmap v528: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:53.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:53 smithi121 ceph-mon[145859]: pgmap v529: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:53 smithi188 ceph-mon[107793]: pgmap v529: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:55.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.166+0000 7f542a5a9700 1 -- 172.21.15.121:0/2263298074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54240ffa40 msgr2=0x7f54240ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:55.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.166+0000 7f542a5a9700 1 --2- 172.21.15.121:0/2263298074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54240ffa40 0x7f54240ffec0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f540c009910 tx=0x7f540c009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:55.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.166+0000 7f542a5a9700 1 -- 172.21.15.121:0/2263298074 shutdown_connections 2024-04-05T21:43:55.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.166+0000 7f542a5a9700 1 --2- 172.21.15.121:0/2263298074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54240ffa40 0x7f54240ffec0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.166+0000 7f542a5a9700 1 --2- 172.21.15.121:0/2263298074 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5424105a40 0x7f5424105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.166+0000 7f542a5a9700 1 -- 172.21.15.121:0/2263298074 >> 172.21.15.121:0/2263298074 conn(0x7f54240fb570 msgr2=0x7f54240fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:55.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.166+0000 7f542a5a9700 1 -- 172.21.15.121:0/2263298074 shutdown_connections 2024-04-05T21:43:55.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.166+0000 7f542a5a9700 1 -- 172.21.15.121:0/2263298074 wait complete. 2024-04-05T21:43:55.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.166+0000 7f542a5a9700 1 Processor -- start 2024-04-05T21:43:55.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.166+0000 7f542a5a9700 1 -- start start 2024-04-05T21:43:55.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.167+0000 7f542a5a9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54240ffa40 0x7f5424195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:55.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.167+0000 7f542a5a9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5424105a40 0x7f5424195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:55.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.167+0000 7f542a5a9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54241960d0 con 0x7f5424105a40 2024-04-05T21:43:55.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.167+0000 7f542a5a9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5424196210 con 0x7f54240ffa40 2024-04-05T21:43:55.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.167+0000 7f54237fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5424105a40 0x7f5424195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.167+0000 7f54237fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5424105a40 0x7f5424195a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51536/0 (socket says 172.21.15.121:51536) 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.167+0000 7f54237fe700 1 -- 172.21.15.121:0/1824703218 learned_addr learned my addr 172.21.15.121:0/1824703218 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.167+0000 7f5423fff700 1 --2- 172.21.15.121:0/1824703218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54240ffa40 0x7f5424195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.167+0000 7f54237fe700 1 -- 172.21.15.121:0/1824703218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54240ffa40 msgr2=0x7f5424195540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.167+0000 7f54237fe700 1 --2- 172.21.15.121:0/1824703218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54240ffa40 0x7f5424195540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.167+0000 7f54237fe700 1 -- 172.21.15.121:0/1824703218 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c0095f0 con 0x7f5424105a40 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.167+0000 7f54237fe700 1 --2- 172.21.15.121:0/1824703218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5424105a40 0x7f5424195a80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f540c00b5c0 tx=0x7f540c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.168+0000 7f54217fa700 1 -- 172.21.15.121:0/1824703218 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f540c01d070 con 0x7f5424105a40 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.168+0000 7f542a5a9700 1 -- 172.21.15.121:0/1824703218 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f542419a000 con 0x7f5424105a40 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.168+0000 7f54217fa700 1 -- 172.21.15.121:0/1824703218 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f540c00f460 con 0x7f5424105a40 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.168+0000 7f54217fa700 1 -- 172.21.15.121:0/1824703218 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f540c021640 con 0x7f5424105a40 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.168+0000 7f542a5a9700 1 -- 172.21.15.121:0/1824703218 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542419a4f0 con 0x7f5424105a40 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.169+0000 7f542a5a9700 1 -- 172.21.15.121:0/1824703218 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542404ea90 con 0x7f5424105a40 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.169+0000 7f54217fa700 1 -- 172.21.15.121:0/1824703218 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f540c00fa20 con 0x7f5424105a40 2024-04-05T21:43:55.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.170+0000 7f54217fa700 1 --2- 172.21.15.121:0/1824703218 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f541006cce0 0x7f541006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:55.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.170+0000 7f5423fff700 1 --2- 172.21.15.121:0/1824703218 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f541006cce0 0x7f541006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:55.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.170+0000 7f54217fa700 1 -- 172.21.15.121:0/1824703218 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f540c08d7c0 con 0x7f5424105a40 2024-04-05T21:43:55.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.170+0000 7f5423fff700 1 --2- 172.21.15.121:0/1824703218 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f541006cce0 0x7f541006f1a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f541400ac50 tx=0x7f541400a380 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:55.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.174+0000 7f54217fa700 1 -- 172.21.15.121:0/1824703218 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f540c057830 con 0x7f5424105a40 2024-04-05T21:43:55.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.386+0000 7f542a5a9700 1 -- 172.21.15.121:0/1824703218 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5424107ba0 con 0x7f541006cce0 2024-04-05T21:43:55.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.387+0000 7f54217fa700 1 -- 172.21.15.121:0/1824703218 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f5424107ba0 con 0x7f541006cce0 2024-04-05T21:43:55.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.389+0000 7f542a5a9700 1 -- 172.21.15.121:0/1824703218 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f541006cce0 msgr2=0x7f541006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:55.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.389+0000 7f542a5a9700 1 --2- 172.21.15.121:0/1824703218 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f541006cce0 0x7f541006f1a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f541400ac50 tx=0x7f541400a380 comp rx=0 tx=0).stop 2024-04-05T21:43:55.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.389+0000 7f542a5a9700 1 -- 172.21.15.121:0/1824703218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5424105a40 msgr2=0x7f5424195a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:55.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.389+0000 7f542a5a9700 1 --2- 172.21.15.121:0/1824703218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5424105a40 0x7f5424195a80 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f540c00b5c0 tx=0x7f540c00bd70 comp rx=0 tx=0).stop 2024-04-05T21:43:55.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.390+0000 7f542a5a9700 1 -- 172.21.15.121:0/1824703218 shutdown_connections 2024-04-05T21:43:55.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.390+0000 7f542a5a9700 1 --2- 172.21.15.121:0/1824703218 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f541006cce0 0x7f541006f1a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.390+0000 7f542a5a9700 1 --2- 172.21.15.121:0/1824703218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54240ffa40 0x7f5424195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.390+0000 7f542a5a9700 1 --2- 172.21.15.121:0/1824703218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5424105a40 0x7f5424195a80 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.390+0000 7f542a5a9700 1 -- 172.21.15.121:0/1824703218 >> 172.21.15.121:0/1824703218 conn(0x7f54240fb570 msgr2=0x7f54240fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:55.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.390+0000 7f542a5a9700 1 -- 172.21.15.121:0/1824703218 shutdown_connections 2024-04-05T21:43:55.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.390+0000 7f542a5a9700 1 -- 172.21.15.121:0/1824703218 wait complete. 2024-04-05T21:43:55.401 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:43:55.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.496+0000 7f22901ac700 1 -- 172.21.15.121:0/3272773617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22880fef70 msgr2=0x7f22880ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:55.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.496+0000 7f22901ac700 1 --2- 172.21.15.121:0/3272773617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22880fef70 0x7f22880ff3f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2284009910 tx=0x7f2284009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:55.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.496+0000 7f22901ac700 1 -- 172.21.15.121:0/3272773617 shutdown_connections 2024-04-05T21:43:55.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.496+0000 7f22901ac700 1 --2- 172.21.15.121:0/3272773617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22880fef70 0x7f22880ff3f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.496+0000 7f22901ac700 1 --2- 172.21.15.121:0/3272773617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2288105b50 0x7f2288105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.496+0000 7f22901ac700 1 -- 172.21.15.121:0/3272773617 >> 172.21.15.121:0/3272773617 conn(0x7f2288075e30 msgr2=0x7f22880fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:55.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.496+0000 7f22901ac700 1 -- 172.21.15.121:0/3272773617 shutdown_connections 2024-04-05T21:43:55.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.496+0000 7f22901ac700 1 -- 172.21.15.121:0/3272773617 wait complete. 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.496+0000 7f22901ac700 1 Processor -- start 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.496+0000 7f22901ac700 1 -- start start 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f22901ac700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22880fef70 0x7f2288196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f22901ac700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2288105b50 0x7f2288196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f22901ac700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22881972d0 con 0x7f22880fef70 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f22901ac700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228819a000 con 0x7f2288105b50 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f228df48700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22880fef70 0x7f2288196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f228df48700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22880fef70 0x7f2288196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51562/0 (socket says 172.21.15.121:51562) 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f228df48700 1 -- 172.21.15.121:0/1741732378 learned_addr learned my addr 172.21.15.121:0/1741732378 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f228df48700 1 -- 172.21.15.121:0/1741732378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2288105b50 msgr2=0x7f2288196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f228d747700 1 --2- 172.21.15.121:0/1741732378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2288105b50 0x7f2288196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f228df48700 1 --2- 172.21.15.121:0/1741732378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2288105b50 0x7f2288196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f228df48700 1 -- 172.21.15.121:0/1741732378 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22840095f0 con 0x7f22880fef70 2024-04-05T21:43:55.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f228df48700 1 --2- 172.21.15.121:0/1741732378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22880fef70 0x7f2288196700 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f227c00d8d0 tx=0x7f227c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:55.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f227affd700 1 -- 172.21.15.121:0/1741732378 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f227c009940 con 0x7f22880fef70 2024-04-05T21:43:55.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f227affd700 1 -- 172.21.15.121:0/1741732378 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f227c010460 con 0x7f22880fef70 2024-04-05T21:43:55.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f227affd700 1 -- 172.21.15.121:0/1741732378 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f227c00f5f0 con 0x7f22880fef70 2024-04-05T21:43:55.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f22901ac700 1 -- 172.21.15.121:0/1741732378 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228819a2e0 con 0x7f22880fef70 2024-04-05T21:43:55.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.497+0000 7f22901ac700 1 -- 172.21.15.121:0/1741732378 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228819a830 con 0x7f22880fef70 2024-04-05T21:43:55.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.498+0000 7f22901ac700 1 -- 172.21.15.121:0/1741732378 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22881090d0 con 0x7f22880fef70 2024-04-05T21:43:55.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.499+0000 7f227affd700 1 -- 172.21.15.121:0/1741732378 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f227c0105d0 con 0x7f22880fef70 2024-04-05T21:43:55.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.499+0000 7f227affd700 1 --2- 172.21.15.121:0/1741732378 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f227406cce0 0x7f227406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:55.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.499+0000 7f228d747700 1 --2- 172.21.15.121:0/1741732378 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f227406cce0 0x7f227406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:55.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.499+0000 7f227affd700 1 -- 172.21.15.121:0/1741732378 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f227c08c4f0 con 0x7f22880fef70 2024-04-05T21:43:55.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.500+0000 7f228d747700 1 --2- 172.21.15.121:0/1741732378 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f227406cce0 0x7f227406f1a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f228400b5c0 tx=0x7f2284005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:55.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.504+0000 7f227affd700 1 -- 172.21.15.121:0/1741732378 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f227c0565e0 con 0x7f22880fef70 2024-04-05T21:43:55.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.715+0000 7f22901ac700 1 -- 172.21.15.121:0/1741732378 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f228819aaa0 con 0x7f227406cce0 2024-04-05T21:43:55.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.716+0000 7f227affd700 1 -- 172.21.15.121:0/1741732378 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f228819aaa0 con 0x7f227406cce0 2024-04-05T21:43:55.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.718+0000 7f22901ac700 1 -- 172.21.15.121:0/1741732378 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f227406cce0 msgr2=0x7f227406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:55.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.718+0000 7f22901ac700 1 --2- 172.21.15.121:0/1741732378 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f227406cce0 0x7f227406f1a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f228400b5c0 tx=0x7f2284005fd0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.718+0000 7f22901ac700 1 -- 172.21.15.121:0/1741732378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22880fef70 msgr2=0x7f2288196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:55.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.718+0000 7f22901ac700 1 --2- 172.21.15.121:0/1741732378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22880fef70 0x7f2288196700 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f227c00d8d0 tx=0x7f227c00dc90 comp rx=0 tx=0).stop 2024-04-05T21:43:55.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.718+0000 7f22901ac700 1 -- 172.21.15.121:0/1741732378 shutdown_connections 2024-04-05T21:43:55.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.718+0000 7f22901ac700 1 --2- 172.21.15.121:0/1741732378 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f227406cce0 0x7f227406f1a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.718+0000 7f22901ac700 1 --2- 172.21.15.121:0/1741732378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22880fef70 0x7f2288196700 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.718+0000 7f22901ac700 1 --2- 172.21.15.121:0/1741732378 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2288105b50 0x7f2288196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.718+0000 7f22901ac700 1 -- 172.21.15.121:0/1741732378 >> 172.21.15.121:0/1741732378 conn(0x7f2288075e30 msgr2=0x7f22881045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:55.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.718+0000 7f22901ac700 1 -- 172.21.15.121:0/1741732378 shutdown_connections 2024-04-05T21:43:55.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.718+0000 7f22901ac700 1 -- 172.21.15.121:0/1741732378 wait complete. 2024-04-05T21:43:55.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:55 smithi121 ceph-mon[145859]: pgmap v530: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:55 smithi188 ceph-mon[107793]: pgmap v530: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:55.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.820+0000 7f0bdf493700 1 -- 172.21.15.121:0/1738783112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd80ffbd0 msgr2=0x7f0bd8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.820+0000 7f0bdf493700 1 --2- 172.21.15.121:0/1738783112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd80ffbd0 0x7f0bd8100050 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009910 tx=0x7f0bc8009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.820+0000 7f0bdf493700 1 -- 172.21.15.121:0/1738783112 shutdown_connections 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.820+0000 7f0bdf493700 1 --2- 172.21.15.121:0/1738783112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd80ffbd0 0x7f0bd8100050 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.820+0000 7f0bdf493700 1 --2- 172.21.15.121:0/1738783112 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bd8105a40 0x7f0bd8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.820+0000 7f0bdf493700 1 -- 172.21.15.121:0/1738783112 >> 172.21.15.121:0/1738783112 conn(0x7f0bd80755a0 msgr2=0x7f0bd80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.820+0000 7f0bdf493700 1 -- 172.21.15.121:0/1738783112 shutdown_connections 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.820+0000 7f0bdf493700 1 -- 172.21.15.121:0/1738783112 wait complete. 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.821+0000 7f0bdf493700 1 Processor -- start 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.821+0000 7f0bdf493700 1 -- start start 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.821+0000 7f0bdf493700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd80ffbd0 0x7f0bd819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.821+0000 7f0bdf493700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bd8105a40 0x7f0bd819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.821+0000 7f0bdf493700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd819d0e0 con 0x7f0bd80ffbd0 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.821+0000 7f0bdf493700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd8196590 con 0x7f0bd8105a40 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.821+0000 7f0bdd22f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd80ffbd0 0x7f0bd819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:55.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.821+0000 7f0bdd22f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd80ffbd0 0x7f0bd819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51592/0 (socket says 172.21.15.121:51592) 2024-04-05T21:43:55.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.821+0000 7f0bdd22f700 1 -- 172.21.15.121:0/2750508427 learned_addr learned my addr 172.21.15.121:0/2750508427 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:55.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.821+0000 7f0bdca2e700 1 --2- 172.21.15.121:0/2750508427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bd8105a40 0x7f0bd819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:55.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.822+0000 7f0bdd22f700 1 -- 172.21.15.121:0/2750508427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bd8105a40 msgr2=0x7f0bd819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:55.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.822+0000 7f0bdd22f700 1 --2- 172.21.15.121:0/2750508427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bd8105a40 0x7f0bd819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:55.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.822+0000 7f0bdd22f700 1 -- 172.21.15.121:0/2750508427 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc80095f0 con 0x7f0bd80ffbd0 2024-04-05T21:43:55.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.822+0000 7f0bdca2e700 1 --2- 172.21.15.121:0/2750508427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bd8105a40 0x7f0bd819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:43:55.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.822+0000 7f0bdd22f700 1 --2- 172.21.15.121:0/2750508427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd80ffbd0 0x7f0bd819c510 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0bd400d8d0 tx=0x7f0bd400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:55.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.822+0000 7f0bce7fc700 1 -- 172.21.15.121:0/2750508427 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bd4009940 con 0x7f0bd80ffbd0 2024-04-05T21:43:55.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.822+0000 7f0bce7fc700 1 -- 172.21.15.121:0/2750508427 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bd4010460 con 0x7f0bd80ffbd0 2024-04-05T21:43:55.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.822+0000 7f0bdf493700 1 -- 172.21.15.121:0/2750508427 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd8196870 con 0x7f0bd80ffbd0 2024-04-05T21:43:55.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.822+0000 7f0bce7fc700 1 -- 172.21.15.121:0/2750508427 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bd400f5f0 con 0x7f0bd80ffbd0 2024-04-05T21:43:55.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.822+0000 7f0bdf493700 1 -- 172.21.15.121:0/2750508427 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd8196dc0 con 0x7f0bd80ffbd0 2024-04-05T21:43:55.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.823+0000 7f0bdf493700 1 -- 172.21.15.121:0/2750508427 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd8108fe0 con 0x7f0bd80ffbd0 2024-04-05T21:43:55.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.824+0000 7f0bce7fc700 1 -- 172.21.15.121:0/2750508427 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0bd40105d0 con 0x7f0bd80ffbd0 2024-04-05T21:43:55.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.824+0000 7f0bce7fc700 1 --2- 172.21.15.121:0/2750508427 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0bc406cce0 0x7f0bc406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:55.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.824+0000 7f0bce7fc700 1 -- 172.21.15.121:0/2750508427 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0bd408c5e0 con 0x7f0bd80ffbd0 2024-04-05T21:43:55.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.824+0000 7f0bdca2e700 1 --2- 172.21.15.121:0/2750508427 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0bc406cce0 0x7f0bc406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:55.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.825+0000 7f0bdca2e700 1 --2- 172.21.15.121:0/2750508427 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0bc406cce0 0x7f0bc406f1a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8104ef0 tx=0x7f0bc801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:55.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:55.829+0000 7f0bce7fc700 1 -- 172.21.15.121:0/2750508427 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0bd405a0e0 con 0x7f0bd80ffbd0 2024-04-05T21:43:56.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.030+0000 7f0bdf493700 1 -- 172.21.15.121:0/2750508427 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bd8196fe0 con 0x7f0bc406cce0 2024-04-05T21:43:56.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.041+0000 7f0bce7fc700 1 -- 172.21.15.121:0/2750508427 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f0bd8196fe0 con 0x7f0bc406cce0 2024-04-05T21:43:56.041 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:43:56.042 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (14m) 8m ago 26m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:43:56.042 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (26m) 8m ago 26m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:43:56.042 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (25m) 8m ago 25m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:43:56.042 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (12m) 8m ago 26m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:43:56.042 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (12m) 8m ago 25m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:43:56.043 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (14m) 8m ago 25m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:43:56.043 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (9m) 8m ago 20m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:43:56.043 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (8m) 8m ago 19m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:43:56.043 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (8m) 8m ago 20m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:43:56.043 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (19m) 8m ago 19m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:43:56.043 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (16m) 8m ago 27m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:43:56.044 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (15m) 8m ago 25m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:43:56.044 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (13m) 8m ago 27m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:43:56.044 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (13m) 8m ago 25m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:43:56.044 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (15m) 8m ago 26m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:43:56.044 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (15m) 8m ago 25m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:43:56.044 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (12m) 8m ago 24m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:43:56.045 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (11m) 8m ago 23m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:43:56.045 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (11m) 8m ago 23m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:43:56.045 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (10m) 8m ago 22m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:43:56.045 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (10m) 8m ago 22m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:43:56.045 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (9m) 8m ago 21m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:43:56.045 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (15m) 8m ago 25m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:43:56.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.043+0000 7f0bdf493700 1 -- 172.21.15.121:0/2750508427 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0bc406cce0 msgr2=0x7f0bc406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:56.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.043+0000 7f0bdf493700 1 --2- 172.21.15.121:0/2750508427 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0bc406cce0 0x7f0bc406f1a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8104ef0 tx=0x7f0bc801a040 comp rx=0 tx=0).stop 2024-04-05T21:43:56.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.043+0000 7f0bdf493700 1 -- 172.21.15.121:0/2750508427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd80ffbd0 msgr2=0x7f0bd819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:56.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.043+0000 7f0bdf493700 1 --2- 172.21.15.121:0/2750508427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd80ffbd0 0x7f0bd819c510 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0bd400d8d0 tx=0x7f0bd400dc90 comp rx=0 tx=0).stop 2024-04-05T21:43:56.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.043+0000 7f0bdf493700 1 -- 172.21.15.121:0/2750508427 shutdown_connections 2024-04-05T21:43:56.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.043+0000 7f0bdf493700 1 --2- 172.21.15.121:0/2750508427 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0bc406cce0 0x7f0bc406f1a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.043+0000 7f0bdf493700 1 --2- 172.21.15.121:0/2750508427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd80ffbd0 0x7f0bd819c510 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.043+0000 7f0bdf493700 1 --2- 172.21.15.121:0/2750508427 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bd8105a40 0x7f0bd819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.043+0000 7f0bdf493700 1 -- 172.21.15.121:0/2750508427 >> 172.21.15.121:0/2750508427 conn(0x7f0bd80755a0 msgr2=0x7f0bd8103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:56.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.043+0000 7f0bdf493700 1 -- 172.21.15.121:0/2750508427 shutdown_connections 2024-04-05T21:43:56.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.043+0000 7f0bdf493700 1 -- 172.21.15.121:0/2750508427 wait complete. 2024-04-05T21:43:56.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.144+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/3713370003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4cc0ffa50 msgr2=0x7fa4cc0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:56.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.144+0000 7fa4d3bd0700 1 --2- 172.21.15.121:0/3713370003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4cc0ffa50 0x7fa4cc0ffed0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009910 tx=0x7fa4c8009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:56.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.144+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/3713370003 shutdown_connections 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.144+0000 7fa4d3bd0700 1 --2- 172.21.15.121:0/3713370003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4cc0ffa50 0x7fa4cc0ffed0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.144+0000 7fa4d3bd0700 1 --2- 172.21.15.121:0/3713370003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4cc105a50 0x7fa4cc105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.144+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/3713370003 >> 172.21.15.121:0/3713370003 conn(0x7fa4cc0fb540 msgr2=0x7fa4cc0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.144+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/3713370003 shutdown_connections 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.144+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/3713370003 wait complete. 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.145+0000 7fa4d3bd0700 1 Processor -- start 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.145+0000 7fa4d3bd0700 1 -- start start 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.145+0000 7fa4d3bd0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4cc0ffa50 0x7fa4cc0fbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.145+0000 7fa4d3bd0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4cc105a50 0x7fa4cc0fc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.145+0000 7fa4d3bd0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc0fc960 con 0x7fa4cc105a50 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.145+0000 7fa4d3bd0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc0fcad0 con 0x7fa4cc0ffa50 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.145+0000 7fa4d116b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4cc105a50 0x7fa4cc0fc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.145+0000 7fa4d116b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4cc105a50 0x7fa4cc0fc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51608/0 (socket says 172.21.15.121:51608) 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.145+0000 7fa4d116b700 1 -- 172.21.15.121:0/2602979263 learned_addr learned my addr 172.21.15.121:0/2602979263 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:56.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.146+0000 7fa4d116b700 1 -- 172.21.15.121:0/2602979263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4cc0ffa50 msgr2=0x7fa4cc0fbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:56.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.146+0000 7fa4d116b700 1 --2- 172.21.15.121:0/2602979263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4cc0ffa50 0x7fa4cc0fbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.146+0000 7fa4d116b700 1 -- 172.21.15.121:0/2602979263 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c80095f0 con 0x7fa4cc105a50 2024-04-05T21:43:56.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.146+0000 7fa4d116b700 1 --2- 172.21.15.121:0/2602979263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4cc105a50 0x7fa4cc0fc390 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8006010 tx=0x7fa4c800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:56.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.146+0000 7fa4c2ffd700 1 -- 172.21.15.121:0/2602979263 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4c801d070 con 0x7fa4cc105a50 2024-04-05T21:43:56.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.146+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/2602979263 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4cc1a2820 con 0x7fa4cc105a50 2024-04-05T21:43:56.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.146+0000 7fa4c2ffd700 1 -- 172.21.15.121:0/2602979263 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4c800f460 con 0x7fa4cc105a50 2024-04-05T21:43:56.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.146+0000 7fa4c2ffd700 1 -- 172.21.15.121:0/2602979263 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4c8021620 con 0x7fa4cc105a50 2024-04-05T21:43:56.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.146+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/2602979263 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4cc1a2bd0 con 0x7fa4cc105a50 2024-04-05T21:43:56.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.147+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/2602979263 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4cc108ff0 con 0x7fa4cc105a50 2024-04-05T21:43:56.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.148+0000 7fa4c2ffd700 1 -- 172.21.15.121:0/2602979263 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa4c800fa40 con 0x7fa4cc105a50 2024-04-05T21:43:56.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.148+0000 7fa4c2ffd700 1 --2- 172.21.15.121:0/2602979263 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa4b806cce0 0x7fa4b806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:56.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.148+0000 7fa4c2ffd700 1 -- 172.21.15.121:0/2602979263 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa4c808dbf0 con 0x7fa4cc105a50 2024-04-05T21:43:56.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.148+0000 7fa4d196c700 1 --2- 172.21.15.121:0/2602979263 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa4b806cce0 0x7fa4b806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:56.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.149+0000 7fa4d196c700 1 --2- 172.21.15.121:0/2602979263 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa4b806cce0 0x7fa4b806f1a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc005950 tx=0x7fa4bc0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:56.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.152+0000 7fa4c2ffd700 1 -- 172.21.15.121:0/2602979263 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa4c8057ce0 con 0x7fa4cc105a50 2024-04-05T21:43:56.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.426+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/2602979263 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa4cc0fd380 con 0x7fa4cc105a50 2024-04-05T21:43:56.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.427+0000 7fa4c2ffd700 1 -- 172.21.15.121:0/2602979263 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7fa4c805b300 con 0x7fa4cc105a50 2024-04-05T21:43:56.427 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:43:56.428 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:43:56.429 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:43:56.429 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:43:56.429 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:43:56.429 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:43:56.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.429+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/2602979263 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa4b806cce0 msgr2=0x7fa4b806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:56.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.429+0000 7fa4d3bd0700 1 --2- 172.21.15.121:0/2602979263 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa4b806cce0 0x7fa4b806f1a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc005950 tx=0x7fa4bc0058e0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.429+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/2602979263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4cc105a50 msgr2=0x7fa4cc0fc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:56.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.429+0000 7fa4d3bd0700 1 --2- 172.21.15.121:0/2602979263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4cc105a50 0x7fa4cc0fc390 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8006010 tx=0x7fa4c800be20 comp rx=0 tx=0).stop 2024-04-05T21:43:56.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.429+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/2602979263 shutdown_connections 2024-04-05T21:43:56.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.429+0000 7fa4d3bd0700 1 --2- 172.21.15.121:0/2602979263 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa4b806cce0 0x7fa4b806f1a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.429+0000 7fa4d3bd0700 1 --2- 172.21.15.121:0/2602979263 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa4cc0ffa50 0x7fa4cc0fbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.429+0000 7fa4d3bd0700 1 --2- 172.21.15.121:0/2602979263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4cc105a50 0x7fa4cc0fc390 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.429+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/2602979263 >> 172.21.15.121:0/2602979263 conn(0x7fa4cc0fb540 msgr2=0x7fa4cc069050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:56.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.429+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/2602979263 shutdown_connections 2024-04-05T21:43:56.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.429+0000 7fa4d3bd0700 1 -- 172.21.15.121:0/2602979263 wait complete. 2024-04-05T21:43:56.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.533+0000 7f7112ca6700 1 -- 172.21.15.121:0/2437933876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f710c0fef70 msgr2=0x7f710c0ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:56.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.533+0000 7f7112ca6700 1 --2- 172.21.15.121:0/2437933876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f710c0fef70 0x7f710c0ff3f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f70f8009910 tx=0x7f70f8009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:56.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.533+0000 7f7112ca6700 1 -- 172.21.15.121:0/2437933876 shutdown_connections 2024-04-05T21:43:56.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.533+0000 7f7112ca6700 1 --2- 172.21.15.121:0/2437933876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f710c0fef70 0x7f710c0ff3f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.533+0000 7f7112ca6700 1 --2- 172.21.15.121:0/2437933876 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f710c105b50 0x7f710c105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.533+0000 7f7112ca6700 1 -- 172.21.15.121:0/2437933876 >> 172.21.15.121:0/2437933876 conn(0x7f710c075e30 msgr2=0x7f710c0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:56.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.533+0000 7f7112ca6700 1 -- 172.21.15.121:0/2437933876 shutdown_connections 2024-04-05T21:43:56.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.533+0000 7f7112ca6700 1 -- 172.21.15.121:0/2437933876 wait complete. 2024-04-05T21:43:56.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.534+0000 7f7112ca6700 1 Processor -- start 2024-04-05T21:43:56.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.534+0000 7f7112ca6700 1 -- start start 2024-04-05T21:43:56.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.534+0000 7f7112ca6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f710c0fef70 0x7f710c196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:56.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.534+0000 7f7112ca6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f710c105b50 0x7f710c196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:56.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.534+0000 7f7112ca6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c1972d0 con 0x7f710c105b50 2024-04-05T21:43:56.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.534+0000 7f7112ca6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c19a000 con 0x7f710c0fef70 2024-04-05T21:43:56.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.534+0000 7f710bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f710c105b50 0x7f710c196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:56.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.534+0000 7f710bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f710c105b50 0x7f710c196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51614/0 (socket says 172.21.15.121:51614) 2024-04-05T21:43:56.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.534+0000 7f710bfff700 1 -- 172.21.15.121:0/1665332830 learned_addr learned my addr 172.21.15.121:0/1665332830 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:56.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.534+0000 7f7110a42700 1 --2- 172.21.15.121:0/1665332830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f710c0fef70 0x7f710c196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:56.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.534+0000 7f710bfff700 1 -- 172.21.15.121:0/1665332830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f710c0fef70 msgr2=0x7f710c196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:56.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.534+0000 7f710bfff700 1 --2- 172.21.15.121:0/1665332830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f710c0fef70 0x7f710c196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.534+0000 7f710bfff700 1 -- 172.21.15.121:0/1665332830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f80095f0 con 0x7f710c105b50 2024-04-05T21:43:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.535+0000 7f710bfff700 1 --2- 172.21.15.121:0/1665332830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f710c105b50 0x7f710c196c40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f70f8006010 tx=0x7f70f800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.535+0000 7f7109ffb700 1 -- 172.21.15.121:0/1665332830 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70f801d070 con 0x7f710c105b50 2024-04-05T21:43:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.535+0000 7f7112ca6700 1 -- 172.21.15.121:0/1665332830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710c19a280 con 0x7f710c105b50 2024-04-05T21:43:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.535+0000 7f7109ffb700 1 -- 172.21.15.121:0/1665332830 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70f8005710 con 0x7f710c105b50 2024-04-05T21:43:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.535+0000 7f7109ffb700 1 -- 172.21.15.121:0/1665332830 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70f800fe20 con 0x7f710c105b50 2024-04-05T21:43:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.535+0000 7f7112ca6700 1 -- 172.21.15.121:0/1665332830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710c19a770 con 0x7f710c105b50 2024-04-05T21:43:56.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.536+0000 7f7112ca6700 1 -- 172.21.15.121:0/1665332830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710c1090d0 con 0x7f710c105b50 2024-04-05T21:43:56.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.536+0000 7f7109ffb700 1 -- 172.21.15.121:0/1665332830 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f70f8003a60 con 0x7f710c105b50 2024-04-05T21:43:56.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.537+0000 7f7109ffb700 1 --2- 172.21.15.121:0/1665332830 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70fc06cce0 0x7f70fc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:56.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.537+0000 7f7109ffb700 1 -- 172.21.15.121:0/1665332830 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f70f808d7c0 con 0x7f710c105b50 2024-04-05T21:43:56.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.537+0000 7f7110a42700 1 --2- 172.21.15.121:0/1665332830 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70fc06cce0 0x7f70fc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:56.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.537+0000 7f7110a42700 1 --2- 172.21.15.121:0/1665332830 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70fc06cce0 0x7f70fc06f1a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7100005950 tx=0x7f710000b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:56.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.541+0000 7f7109ffb700 1 -- 172.21.15.121:0/1665332830 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f70f8057830 con 0x7f710c105b50 2024-04-05T21:43:56.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:56 smithi121 ceph-mon[145859]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:56.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:56 smithi121 ceph-mon[145859]: from='client.34822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:56.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:56 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2602979263' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:43:56.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.773+0000 7f7112ca6700 1 -- 172.21.15.121:0/1665332830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f710c197a60 con 0x7f710c105b50 2024-04-05T21:43:56.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.774+0000 7f7109ffb700 1 -- 172.21.15.121:0/1665332830 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f70f805ae50 con 0x7f710c105b50 2024-04-05T21:43:56.774 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:43:56.775 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:43:56.776 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:43:56.777 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:43:56.777 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:43:56.777 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:43:56.777 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:43:56.777 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:43:56.777 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:43:56.777 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:43:56.777 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:43:56.777 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:43:56.777 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:43:56.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.776+0000 7f7112ca6700 1 -- 172.21.15.121:0/1665332830 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70fc06cce0 msgr2=0x7f70fc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:56.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.776+0000 7f7112ca6700 1 --2- 172.21.15.121:0/1665332830 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70fc06cce0 0x7f70fc06f1a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7100005950 tx=0x7f710000b410 comp rx=0 tx=0).stop 2024-04-05T21:43:56.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.776+0000 7f7112ca6700 1 -- 172.21.15.121:0/1665332830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f710c105b50 msgr2=0x7f710c196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:56.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.776+0000 7f7112ca6700 1 --2- 172.21.15.121:0/1665332830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f710c105b50 0x7f710c196c40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f70f8006010 tx=0x7f70f800bd70 comp rx=0 tx=0).stop 2024-04-05T21:43:56.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.776+0000 7f7112ca6700 1 -- 172.21.15.121:0/1665332830 shutdown_connections 2024-04-05T21:43:56.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.776+0000 7f7112ca6700 1 --2- 172.21.15.121:0/1665332830 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70fc06cce0 0x7f70fc06f1a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.776+0000 7f7112ca6700 1 --2- 172.21.15.121:0/1665332830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f710c0fef70 0x7f710c196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.776+0000 7f7112ca6700 1 --2- 172.21.15.121:0/1665332830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f710c105b50 0x7f710c196c40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.776+0000 7f7112ca6700 1 -- 172.21.15.121:0/1665332830 >> 172.21.15.121:0/1665332830 conn(0x7f710c075e30 msgr2=0x7f710c1045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:56.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.776+0000 7f7112ca6700 1 -- 172.21.15.121:0/1665332830 shutdown_connections 2024-04-05T21:43:56.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.776+0000 7f7112ca6700 1 -- 172.21.15.121:0/1665332830 wait complete. 2024-04-05T21:43:56.779 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:43:56.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:56 smithi188 ceph-mon[107793]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:56.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:56 smithi188 ceph-mon[107793]: from='client.34822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:56.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:56 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2602979263' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:43:56.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.880+0000 7fd456751700 1 -- 172.21.15.121:0/228425418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd450100dc0 msgr2=0x7fd4501011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:56.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.880+0000 7fd456751700 1 --2- 172.21.15.121:0/228425418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd450100dc0 0x7fd4501011a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd440009910 tx=0x7fd440009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:56.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.880+0000 7fd456751700 1 -- 172.21.15.121:0/228425418 shutdown_connections 2024-04-05T21:43:56.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.880+0000 7fd456751700 1 --2- 172.21.15.121:0/228425418 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd450101770 0x7fd4501057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.880+0000 7fd456751700 1 --2- 172.21.15.121:0/228425418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd450100dc0 0x7fd4501011a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.880+0000 7fd456751700 1 -- 172.21.15.121:0/228425418 >> 172.21.15.121:0/228425418 conn(0x7fd4500fc650 msgr2=0x7fd4500fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:56.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.880+0000 7fd456751700 1 -- 172.21.15.121:0/228425418 shutdown_connections 2024-04-05T21:43:56.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.880+0000 7fd456751700 1 -- 172.21.15.121:0/228425418 wait complete. 2024-04-05T21:43:56.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.881+0000 7fd456751700 1 Processor -- start 2024-04-05T21:43:56.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.881+0000 7fd456751700 1 -- start start 2024-04-05T21:43:56.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.881+0000 7fd456751700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd450100dc0 0x7fd450196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:56.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.881+0000 7fd456751700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd450101770 0x7fd450196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:56.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.881+0000 7fd456751700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd450197280 con 0x7fd450101770 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.881+0000 7fd456751700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45019b010 con 0x7fd450100dc0 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.881+0000 7fd44f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd450101770 0x7fd450196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.881+0000 7fd44f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd450101770 0x7fd450196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51622/0 (socket says 172.21.15.121:51622) 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.881+0000 7fd44f7fe700 1 -- 172.21.15.121:0/3838409124 learned_addr learned my addr 172.21.15.121:0/3838409124 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.881+0000 7fd44ffff700 1 --2- 172.21.15.121:0/3838409124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd450100dc0 0x7fd450196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.882+0000 7fd44f7fe700 1 -- 172.21.15.121:0/3838409124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd450100dc0 msgr2=0x7fd450196660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.882+0000 7fd44f7fe700 1 --2- 172.21.15.121:0/3838409124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd450100dc0 0x7fd450196660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.882+0000 7fd44f7fe700 1 -- 172.21.15.121:0/3838409124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4400095f0 con 0x7fd450101770 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.882+0000 7fd44f7fe700 1 --2- 172.21.15.121:0/3838409124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd450101770 0x7fd450196ba0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd44400d900 tx=0x7fd44400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.882+0000 7fd44d7fa700 1 -- 172.21.15.121:0/3838409124 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd444009900 con 0x7fd450101770 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.882+0000 7fd44d7fa700 1 -- 172.21.15.121:0/3838409124 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd444010460 con 0x7fd450101770 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.882+0000 7fd456751700 1 -- 172.21.15.121:0/3838409124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45019b2f0 con 0x7fd450101770 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.882+0000 7fd44d7fa700 1 -- 172.21.15.121:0/3838409124 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd44400f5f0 con 0x7fd450101770 2024-04-05T21:43:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.882+0000 7fd456751700 1 -- 172.21.15.121:0/3838409124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45019b840 con 0x7fd450101770 2024-04-05T21:43:56.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.883+0000 7fd456751700 1 -- 172.21.15.121:0/3838409124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd450109160 con 0x7fd450101770 2024-04-05T21:43:56.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.883+0000 7fd44d7fa700 1 -- 172.21.15.121:0/3838409124 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd444009a60 con 0x7fd450101770 2024-04-05T21:43:56.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.884+0000 7fd44d7fa700 1 --2- 172.21.15.121:0/3838409124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd43806cce0 0x7fd43806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:56.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.884+0000 7fd44ffff700 1 --2- 172.21.15.121:0/3838409124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd43806cce0 0x7fd43806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:56.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.884+0000 7fd44d7fa700 1 -- 172.21.15.121:0/3838409124 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd44408c000 con 0x7fd450101770 2024-04-05T21:43:56.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.884+0000 7fd44ffff700 1 --2- 172.21.15.121:0/3838409124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd43806cce0 0x7fd43806f1a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd440006010 tx=0x7fd44001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:56.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:56.888+0000 7fd44d7fa700 1 -- 172.21.15.121:0/3838409124 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd4440560f0 con 0x7fd450101770 2024-04-05T21:43:57.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.102+0000 7fd456751700 1 -- 172.21.15.121:0/3838409124 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd45019bba0 con 0x7fd43806cce0 2024-04-05T21:43:57.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.103+0000 7fd44d7fa700 1 -- 172.21.15.121:0/3838409124 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd45019bba0 con 0x7fd43806cce0 2024-04-05T21:43:57.103 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:43:57.103 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:43:57.103 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:43:57.104 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:43:57.104 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:43:57.104 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:43:57.104 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:43:57.104 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:43:57.104 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:43:57.104 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:43:57.104 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:43:57.104 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:43:57.104 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:43:57.104 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:43:57.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.105+0000 7fd456751700 1 -- 172.21.15.121:0/3838409124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd43806cce0 msgr2=0x7fd43806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:57.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.105+0000 7fd456751700 1 --2- 172.21.15.121:0/3838409124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd43806cce0 0x7fd43806f1a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd440006010 tx=0x7fd44001a040 comp rx=0 tx=0).stop 2024-04-05T21:43:57.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.105+0000 7fd456751700 1 -- 172.21.15.121:0/3838409124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd450101770 msgr2=0x7fd450196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:57.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.105+0000 7fd456751700 1 --2- 172.21.15.121:0/3838409124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd450101770 0x7fd450196ba0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd44400d900 tx=0x7fd44400dc10 comp rx=0 tx=0).stop 2024-04-05T21:43:57.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.105+0000 7fd456751700 1 -- 172.21.15.121:0/3838409124 shutdown_connections 2024-04-05T21:43:57.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.105+0000 7fd456751700 1 --2- 172.21.15.121:0/3838409124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd43806cce0 0x7fd43806f1a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:57.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.105+0000 7fd456751700 1 --2- 172.21.15.121:0/3838409124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd450100dc0 0x7fd450196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:57.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.105+0000 7fd456751700 1 --2- 172.21.15.121:0/3838409124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd450101770 0x7fd450196ba0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:57.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.105+0000 7fd456751700 1 -- 172.21.15.121:0/3838409124 >> 172.21.15.121:0/3838409124 conn(0x7fd4500fc650 msgr2=0x7fd4500fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:57.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.105+0000 7fd456751700 1 -- 172.21.15.121:0/3838409124 shutdown_connections 2024-04-05T21:43:57.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.105+0000 7fd456751700 1 -- 172.21.15.121:0/3838409124 wait complete. 2024-04-05T21:43:57.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.207+0000 7f2be3625700 1 -- 172.21.15.121:0/3935044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ffbd0 msgr2=0x7f2bdc100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:57.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.207+0000 7f2be3625700 1 --2- 172.21.15.121:0/3935044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ffbd0 0x7f2bdc100050 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc009910 tx=0x7f2bcc009c20 comp rx=0 tx=0).stop 2024-04-05T21:43:57.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.207+0000 7f2be3625700 1 -- 172.21.15.121:0/3935044 shutdown_connections 2024-04-05T21:43:57.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.207+0000 7f2be3625700 1 --2- 172.21.15.121:0/3935044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ffbd0 0x7f2bdc100050 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:57.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.208+0000 7f2be3625700 1 --2- 172.21.15.121:0/3935044 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bdc105a40 0x7f2bdc105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:57.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.208+0000 7f2be3625700 1 -- 172.21.15.121:0/3935044 >> 172.21.15.121:0/3935044 conn(0x7f2bdc0755a0 msgr2=0x7f2bdc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:57.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.208+0000 7f2be3625700 1 -- 172.21.15.121:0/3935044 shutdown_connections 2024-04-05T21:43:57.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.208+0000 7f2be3625700 1 -- 172.21.15.121:0/3935044 wait complete. 2024-04-05T21:43:57.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.208+0000 7f2be3625700 1 Processor -- start 2024-04-05T21:43:57.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.208+0000 7f2be3625700 1 -- start start 2024-04-05T21:43:57.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.208+0000 7f2be3625700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ffbd0 0x7f2bdc196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:57.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.208+0000 7f2be3625700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bdc105a40 0x7f2bdc196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:57.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.208+0000 7f2be3625700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc197290 con 0x7f2bdc0ffbd0 2024-04-05T21:43:57.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.208+0000 7f2be3625700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc19b020 con 0x7f2bdc105a40 2024-04-05T21:43:57.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.209+0000 7f2be13c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ffbd0 0x7f2bdc196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.209+0000 7f2be13c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ffbd0 0x7f2bdc196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51632/0 (socket says 172.21.15.121:51632) 2024-04-05T21:43:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.209+0000 7f2be13c1700 1 -- 172.21.15.121:0/1408405038 learned_addr learned my addr 172.21.15.121:0/1408405038 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:43:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.209+0000 7f2be0bc0700 1 --2- 172.21.15.121:0/1408405038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bdc105a40 0x7f2bdc196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.209+0000 7f2be13c1700 1 -- 172.21.15.121:0/1408405038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bdc105a40 msgr2=0x7f2bdc196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.209+0000 7f2be13c1700 1 --2- 172.21.15.121:0/1408405038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bdc105a40 0x7f2bdc196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.209+0000 7f2be13c1700 1 -- 172.21.15.121:0/1408405038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc0095f0 con 0x7f2bdc0ffbd0 2024-04-05T21:43:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.209+0000 7f2be13c1700 1 --2- 172.21.15.121:0/1408405038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ffbd0 0x7f2bdc196670 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800e920 tx=0x7f2bd800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.209+0000 7f2bd27fc700 1 -- 172.21.15.121:0/1408405038 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bd8009790 con 0x7f2bdc0ffbd0 2024-04-05T21:43:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.209+0000 7f2bd27fc700 1 -- 172.21.15.121:0/1408405038 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bd800cba0 con 0x7f2bdc0ffbd0 2024-04-05T21:43:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.209+0000 7f2be3625700 1 -- 172.21.15.121:0/1408405038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bdc19b300 con 0x7f2bdc0ffbd0 2024-04-05T21:43:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.209+0000 7f2bd27fc700 1 -- 172.21.15.121:0/1408405038 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bd8018650 con 0x7f2bdc0ffbd0 2024-04-05T21:43:57.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.209+0000 7f2be3625700 1 -- 172.21.15.121:0/1408405038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bdc19b850 con 0x7f2bdc0ffbd0 2024-04-05T21:43:57.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.210+0000 7f2be3625700 1 -- 172.21.15.121:0/1408405038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bdc04ea90 con 0x7f2bdc0ffbd0 2024-04-05T21:43:57.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.210+0000 7f2bd27fc700 1 -- 172.21.15.121:0/1408405038 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2bd800cd10 con 0x7f2bdc0ffbd0 2024-04-05T21:43:57.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.211+0000 7f2bd27fc700 1 --2- 172.21.15.121:0/1408405038 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2bc806cce0 0x7f2bc806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:43:57.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.211+0000 7f2bd27fc700 1 -- 172.21.15.121:0/1408405038 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2bd8014070 con 0x7f2bdc0ffbd0 2024-04-05T21:43:57.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.211+0000 7f2be0bc0700 1 --2- 172.21.15.121:0/1408405038 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2bc806cce0 0x7f2bc806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:43:57.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.211+0000 7f2be0bc0700 1 --2- 172.21.15.121:0/1408405038 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2bc806cce0 0x7f2bc806f1a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc006010 tx=0x7f2bcc00b5a0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:43:57.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.216+0000 7f2bd27fc700 1 -- 172.21.15.121:0/1408405038 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2bd8052ab0 con 0x7f2bdc0ffbd0 2024-04-05T21:43:57.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.486+0000 7f2be3625700 1 -- 172.21.15.121:0/1408405038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2bdc1979d0 con 0x7f2bdc0ffbd0 2024-04-05T21:43:57.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.486+0000 7f2bd27fc700 1 -- 172.21.15.121:0/1408405038 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f2bd805a450 con 0x7f2bdc0ffbd0 2024-04-05T21:43:57.487 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:43:57.487 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:43:57.487 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:43:57.487 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:43:57.487 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:43:57.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.488+0000 7f2be3625700 1 -- 172.21.15.121:0/1408405038 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2bc806cce0 msgr2=0x7f2bc806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:57.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.488+0000 7f2be3625700 1 --2- 172.21.15.121:0/1408405038 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2bc806cce0 0x7f2bc806f1a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc006010 tx=0x7f2bcc00b5a0 comp rx=0 tx=0).stop 2024-04-05T21:43:57.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.488+0000 7f2be3625700 1 -- 172.21.15.121:0/1408405038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ffbd0 msgr2=0x7f2bdc196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:43:57.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.488+0000 7f2be3625700 1 --2- 172.21.15.121:0/1408405038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ffbd0 0x7f2bdc196670 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800e920 tx=0x7f2bd800ece0 comp rx=0 tx=0).stop 2024-04-05T21:43:57.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.488+0000 7f2be3625700 1 -- 172.21.15.121:0/1408405038 shutdown_connections 2024-04-05T21:43:57.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.488+0000 7f2be3625700 1 --2- 172.21.15.121:0/1408405038 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2bc806cce0 0x7f2bc806f1a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:57.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.488+0000 7f2be3625700 1 --2- 172.21.15.121:0/1408405038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2bdc0ffbd0 0x7f2bdc196670 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:57.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.488+0000 7f2be3625700 1 --2- 172.21.15.121:0/1408405038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2bdc105a40 0x7f2bdc196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:43:57.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.488+0000 7f2be3625700 1 -- 172.21.15.121:0/1408405038 >> 172.21.15.121:0/1408405038 conn(0x7f2bdc0755a0 msgr2=0x7f2bdc1037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:43:57.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.488+0000 7f2be3625700 1 -- 172.21.15.121:0/1408405038 shutdown_connections 2024-04-05T21:43:57.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:43:57.488+0000 7f2be3625700 1 -- 172.21.15.121:0/1408405038 wait complete. 2024-04-05T21:43:57.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:57 smithi121 ceph-mon[145859]: from='client.34826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:57 smithi121 ceph-mon[145859]: pgmap v531: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:57 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1665332830' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:43:57.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:57 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1408405038' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:43:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:57 smithi188 ceph-mon[107793]: from='client.34826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:57 smithi188 ceph-mon[107793]: pgmap v531: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:57 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1665332830' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:43:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:57 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1408405038' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:43:58.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:58 smithi188 ceph-mon[107793]: from='client.34838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:58.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:58 smithi121 ceph-mon[145859]: from='client.34838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:43:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:43:59 smithi188 ceph-mon[107793]: pgmap v532: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:43:59.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:43:59 smithi121 ceph-mon[145859]: pgmap v532: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:44:00.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:44:01.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:01 smithi121 ceph-mon[145859]: pgmap v533: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:01.781 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:01 smithi188 ceph-mon[107793]: pgmap v533: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:03.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:03 smithi121 ceph-mon[145859]: pgmap v534: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:03.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:03 smithi188 ceph-mon[107793]: pgmap v534: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:05.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:05 smithi121 ceph-mon[145859]: pgmap v535: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:05 smithi188 ceph-mon[107793]: pgmap v535: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:07 smithi121 ceph-mon[145859]: pgmap v536: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:07 smithi188 ceph-mon[107793]: pgmap v536: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:09.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:09 smithi121 ceph-mon[145859]: pgmap v537: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:09 smithi188 ceph-mon[107793]: pgmap v537: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:11 smithi121 ceph-mon[145859]: pgmap v538: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:11.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:11 smithi188 ceph-mon[107793]: pgmap v538: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:13.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:13 smithi121 ceph-mon[145859]: pgmap v539: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:13.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:13 smithi188 ceph-mon[107793]: pgmap v539: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:15 smithi121 ceph-mon[145859]: pgmap v540: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:15.772 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:44:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:15 smithi188 ceph-mon[107793]: pgmap v540: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:44:17.742 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:17 smithi121 ceph-mon[145859]: pgmap v541: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:17.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:17 smithi188 ceph-mon[107793]: pgmap v541: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:19.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:19 smithi121 ceph-mon[145859]: pgmap v542: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:19 smithi188 ceph-mon[107793]: pgmap v542: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:21.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:21 smithi121 ceph-mon[145859]: pgmap v543: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:21 smithi188 ceph-mon[107793]: pgmap v543: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:23.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:23 smithi121 ceph-mon[145859]: pgmap v544: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:23 smithi188 ceph-mon[107793]: pgmap v544: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:25.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:25 smithi121 ceph-mon[145859]: pgmap v545: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:25.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:25 smithi188 ceph-mon[107793]: pgmap v545: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.604+0000 7fae1967f700 1 -- 172.21.15.121:0/818533669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae14102610 msgr2=0x7fae1410ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.604+0000 7fae1967f700 1 --2- 172.21.15.121:0/818533669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae14102610 0x7fae1410ab00 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fae08009910 tx=0x7fae08009c20 comp rx=0 tx=0).stop 2024-04-05T21:44:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.604+0000 7fae1967f700 1 -- 172.21.15.121:0/818533669 shutdown_connections 2024-04-05T21:44:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.604+0000 7fae1967f700 1 --2- 172.21.15.121:0/818533669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae14102610 0x7fae1410ab00 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.604+0000 7fae1967f700 1 --2- 172.21.15.121:0/818533669 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae14101c60 0x7fae14102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.604+0000 7fae1967f700 1 -- 172.21.15.121:0/818533669 >> 172.21.15.121:0/818533669 conn(0x7fae140fb520 msgr2=0x7fae140fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.604+0000 7fae1967f700 1 -- 172.21.15.121:0/818533669 shutdown_connections 2024-04-05T21:44:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.604+0000 7fae1967f700 1 -- 172.21.15.121:0/818533669 wait complete. 2024-04-05T21:44:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.605+0000 7fae1967f700 1 Processor -- start 2024-04-05T21:44:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.605+0000 7fae1967f700 1 -- start start 2024-04-05T21:44:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.605+0000 7fae1967f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae14101c60 0x7fae140ff950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fae1967f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae14102610 0x7fae140ffe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fae1967f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae141003d0 con 0x7fae14102610 2024-04-05T21:44:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fae1967f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae14100510 con 0x7fae14101c60 2024-04-05T21:44:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fae127fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae14102610 0x7fae140ffe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fae127fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae14102610 0x7fae140ffe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46558/0 (socket says 172.21.15.121:46558) 2024-04-05T21:44:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fae127fc700 1 -- 172.21.15.121:0/503165220 learned_addr learned my addr 172.21.15.121:0/503165220 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:44:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fae12ffd700 1 --2- 172.21.15.121:0/503165220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae14101c60 0x7fae140ff950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fae127fc700 1 -- 172.21.15.121:0/503165220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae14101c60 msgr2=0x7fae140ff950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:27.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fae127fc700 1 --2- 172.21.15.121:0/503165220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae14101c60 0x7fae140ff950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:27.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fae127fc700 1 -- 172.21.15.121:0/503165220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae080095f0 con 0x7fae14102610 2024-04-05T21:44:27.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fae127fc700 1 --2- 172.21.15.121:0/503165220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae14102610 0x7fae140ffe90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fae08000c00 tx=0x7fae0800ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:27.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fadfbfff700 1 -- 172.21.15.121:0/503165220 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae0801d070 con 0x7fae14102610 2024-04-05T21:44:27.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fae1967f700 1 -- 172.21.15.121:0/503165220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae14100790 con 0x7fae14102610 2024-04-05T21:44:27.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fadfbfff700 1 -- 172.21.15.121:0/503165220 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae0800bed0 con 0x7fae14102610 2024-04-05T21:44:27.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fadfbfff700 1 -- 172.21.15.121:0/503165220 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae0800fd90 con 0x7fae14102610 2024-04-05T21:44:27.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.606+0000 7fae1967f700 1 -- 172.21.15.121:0/503165220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae141a2ba0 con 0x7fae14102610 2024-04-05T21:44:27.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.607+0000 7fae1967f700 1 -- 172.21.15.121:0/503165220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae14108200 con 0x7fae14102610 2024-04-05T21:44:27.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.608+0000 7fadfbfff700 1 -- 172.21.15.121:0/503165220 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fae08021410 con 0x7fae14102610 2024-04-05T21:44:27.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.608+0000 7fadfbfff700 1 --2- 172.21.15.121:0/503165220 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fadfc06cce0 0x7fadfc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:27.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.608+0000 7fae12ffd700 1 --2- 172.21.15.121:0/503165220 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fadfc06cce0 0x7fadfc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:27.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.609+0000 7fadfbfff700 1 -- 172.21.15.121:0/503165220 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fae0808d760 con 0x7fae14102610 2024-04-05T21:44:27.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.609+0000 7fae12ffd700 1 --2- 172.21.15.121:0/503165220 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fadfc06cce0 0x7fadfc06f1a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fae04009c60 tx=0x7fae04009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:27.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.613+0000 7fadfbfff700 1 -- 172.21.15.121:0/503165220 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fae080577d0 con 0x7fae14102610 2024-04-05T21:44:27.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:27 smithi121 ceph-mon[145859]: pgmap v546: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:27.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:27 smithi188 ceph-mon[107793]: pgmap v546: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:27.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.822+0000 7fae1967f700 1 -- 172.21.15.121:0/503165220 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fae1402cce0 con 0x7fadfc06cce0 2024-04-05T21:44:27.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.823+0000 7fadfbfff700 1 -- 172.21.15.121:0/503165220 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fae1402cce0 con 0x7fadfc06cce0 2024-04-05T21:44:27.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.825+0000 7fae1967f700 1 -- 172.21.15.121:0/503165220 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fadfc06cce0 msgr2=0x7fadfc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:27.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.825+0000 7fae1967f700 1 --2- 172.21.15.121:0/503165220 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fadfc06cce0 0x7fadfc06f1a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fae04009c60 tx=0x7fae04009260 comp rx=0 tx=0).stop 2024-04-05T21:44:27.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.825+0000 7fae1967f700 1 -- 172.21.15.121:0/503165220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae14102610 msgr2=0x7fae140ffe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:27.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.825+0000 7fae1967f700 1 --2- 172.21.15.121:0/503165220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae14102610 0x7fae140ffe90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fae08000c00 tx=0x7fae0800ba70 comp rx=0 tx=0).stop 2024-04-05T21:44:27.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.825+0000 7fae1967f700 1 -- 172.21.15.121:0/503165220 shutdown_connections 2024-04-05T21:44:27.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.825+0000 7fae1967f700 1 --2- 172.21.15.121:0/503165220 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fadfc06cce0 0x7fadfc06f1a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:27.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.825+0000 7fae1967f700 1 --2- 172.21.15.121:0/503165220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae14101c60 0x7fae140ff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:27.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.825+0000 7fae1967f700 1 --2- 172.21.15.121:0/503165220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae14102610 0x7fae140ffe90 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:27.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.825+0000 7fae1967f700 1 -- 172.21.15.121:0/503165220 >> 172.21.15.121:0/503165220 conn(0x7fae140fb520 msgr2=0x7fae14105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:27.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.826+0000 7fae1967f700 1 -- 172.21.15.121:0/503165220 shutdown_connections 2024-04-05T21:44:27.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.826+0000 7fae1967f700 1 -- 172.21.15.121:0/503165220 wait complete. 2024-04-05T21:44:27.836 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:44:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.929+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2632243565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e88100da0 msgr2=0x7f8e88101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.929+0000 7f8e8f5d9700 1 --2- 172.21.15.121:0/2632243565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e88100da0 0x7f8e88101180 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009910 tx=0x7f8e78009c20 comp rx=0 tx=0).stop 2024-04-05T21:44:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.929+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2632243565 shutdown_connections 2024-04-05T21:44:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.929+0000 7f8e8f5d9700 1 --2- 172.21.15.121:0/2632243565 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e88101750 0x7f8e881057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.929+0000 7f8e8f5d9700 1 --2- 172.21.15.121:0/2632243565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e88100da0 0x7f8e88101180 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.929+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2632243565 >> 172.21.15.121:0/2632243565 conn(0x7f8e880fc630 msgr2=0x7f8e880fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.929+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2632243565 shutdown_connections 2024-04-05T21:44:27.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.929+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2632243565 wait complete. 2024-04-05T21:44:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.929+0000 7f8e8f5d9700 1 Processor -- start 2024-04-05T21:44:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8f5d9700 1 -- start start 2024-04-05T21:44:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8f5d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e88100da0 0x7f8e88196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8f5d9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e88101750 0x7f8e88196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8f5d9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e88197260 con 0x7f8e88100da0 2024-04-05T21:44:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8f5d9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e88199f90 con 0x7f8e88101750 2024-04-05T21:44:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8d375700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e88100da0 0x7f8e88196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8d375700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e88100da0 0x7f8e88196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46582/0 (socket says 172.21.15.121:46582) 2024-04-05T21:44:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8d375700 1 -- 172.21.15.121:0/2855081925 learned_addr learned my addr 172.21.15.121:0/2855081925 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:44:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8cb74700 1 --2- 172.21.15.121:0/2855081925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e88101750 0x7f8e88196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:27.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8d375700 1 -- 172.21.15.121:0/2855081925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e88101750 msgr2=0x7f8e88196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8d375700 1 --2- 172.21.15.121:0/2855081925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e88101750 0x7f8e88196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8d375700 1 -- 172.21.15.121:0/2855081925 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e780095f0 con 0x7f8e88100da0 2024-04-05T21:44:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8d375700 1 --2- 172.21.15.121:0/2855081925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e88100da0 0x7f8e88196690 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8e78006010 tx=0x7f8e7800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e7e7fc700 1 -- 172.21.15.121:0/2855081925 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e7801d070 con 0x7f8e88100da0 2024-04-05T21:44:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e7e7fc700 1 -- 172.21.15.121:0/2855081925 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e7800f460 con 0x7f8e88100da0 2024-04-05T21:44:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e7e7fc700 1 -- 172.21.15.121:0/2855081925 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e78021620 con 0x7f8e88100da0 2024-04-05T21:44:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2855081925 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8819a210 con 0x7f8e88100da0 2024-04-05T21:44:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.930+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2855081925 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8819a700 con 0x7f8e88100da0 2024-04-05T21:44:27.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.931+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2855081925 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e88109140 con 0x7f8e88100da0 2024-04-05T21:44:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.932+0000 7f8e7e7fc700 1 -- 172.21.15.121:0/2855081925 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8e7800fa40 con 0x7f8e88100da0 2024-04-05T21:44:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.932+0000 7f8e7e7fc700 1 --2- 172.21.15.121:0/2855081925 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e7406cce0 0x7f8e7406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.932+0000 7f8e7e7fc700 1 -- 172.21.15.121:0/2855081925 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8e7808d6c0 con 0x7f8e88100da0 2024-04-05T21:44:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.932+0000 7f8e8cb74700 1 --2- 172.21.15.121:0/2855081925 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e7406cce0 0x7f8e7406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:27.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.933+0000 7f8e8cb74700 1 --2- 172.21.15.121:0/2855081925 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e7406cce0 0x7f8e7406f1a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8e84005950 tx=0x7f8e840058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:27.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:27.937+0000 7f8e7e7fc700 1 -- 172.21.15.121:0/2855081925 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8e780577b0 con 0x7f8e88100da0 2024-04-05T21:44:28.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.146+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2855081925 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e8819a970 con 0x7f8e7406cce0 2024-04-05T21:44:28.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.147+0000 7f8e7e7fc700 1 -- 172.21.15.121:0/2855081925 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8e8819a970 con 0x7f8e7406cce0 2024-04-05T21:44:28.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.149+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2855081925 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e7406cce0 msgr2=0x7f8e7406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:28.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.149+0000 7f8e8f5d9700 1 --2- 172.21.15.121:0/2855081925 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e7406cce0 0x7f8e7406f1a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8e84005950 tx=0x7f8e840058e0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.149+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2855081925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e88100da0 msgr2=0x7f8e88196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:28.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.149+0000 7f8e8f5d9700 1 --2- 172.21.15.121:0/2855081925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e88100da0 0x7f8e88196690 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8e78006010 tx=0x7f8e7800be20 comp rx=0 tx=0).stop 2024-04-05T21:44:28.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.150+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2855081925 shutdown_connections 2024-04-05T21:44:28.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.150+0000 7f8e8f5d9700 1 --2- 172.21.15.121:0/2855081925 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e7406cce0 0x7f8e7406f1a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.150+0000 7f8e8f5d9700 1 --2- 172.21.15.121:0/2855081925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e88100da0 0x7f8e88196690 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.150+0000 7f8e8f5d9700 1 --2- 172.21.15.121:0/2855081925 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e88101750 0x7f8e88196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.150+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2855081925 >> 172.21.15.121:0/2855081925 conn(0x7f8e880fc630 msgr2=0x7f8e880fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:28.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.150+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2855081925 shutdown_connections 2024-04-05T21:44:28.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.150+0000 7f8e8f5d9700 1 -- 172.21.15.121:0/2855081925 wait complete. 2024-04-05T21:44:28.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.252+0000 7fddc4280700 1 -- 172.21.15.121:0/3808536322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddbc105a40 msgr2=0x7fddbc105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:28.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.252+0000 7fddc4280700 1 --2- 172.21.15.121:0/3808536322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddbc105a40 0x7fddbc105e20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009910 tx=0x7fddb8009c20 comp rx=0 tx=0).stop 2024-04-05T21:44:28.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.252+0000 7fddc4280700 1 -- 172.21.15.121:0/3808536322 shutdown_connections 2024-04-05T21:44:28.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.252+0000 7fddc4280700 1 --2- 172.21.15.121:0/3808536322 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddbc0ffa40 0x7fddbc0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.252+0000 7fddc4280700 1 --2- 172.21.15.121:0/3808536322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddbc105a40 0x7fddbc105e20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.252+0000 7fddc4280700 1 -- 172.21.15.121:0/3808536322 >> 172.21.15.121:0/3808536322 conn(0x7fddbc0fb570 msgr2=0x7fddbc0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.252+0000 7fddc4280700 1 -- 172.21.15.121:0/3808536322 shutdown_connections 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.252+0000 7fddc4280700 1 -- 172.21.15.121:0/3808536322 wait complete. 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.252+0000 7fddc4280700 1 Processor -- start 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.252+0000 7fddc4280700 1 -- start start 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc4280700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddbc0ffa40 0x7fddbc1974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc4280700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddbc105a40 0x7fddbc197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc4280700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc198020 con 0x7fddbc105a40 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc4280700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc198160 con 0x7fddbc0ffa40 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc181b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddbc105a40 0x7fddbc197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc181b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddbc105a40 0x7fddbc197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46612/0 (socket says 172.21.15.121:46612) 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc181b700 1 -- 172.21.15.121:0/879353425 learned_addr learned my addr 172.21.15.121:0/879353425 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc201c700 1 --2- 172.21.15.121:0/879353425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddbc0ffa40 0x7fddbc1974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:28.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc181b700 1 -- 172.21.15.121:0/879353425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddbc0ffa40 msgr2=0x7fddbc1974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc181b700 1 --2- 172.21.15.121:0/879353425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddbc0ffa40 0x7fddbc1974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc181b700 1 -- 172.21.15.121:0/879353425 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb80095f0 con 0x7fddbc105a40 2024-04-05T21:44:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc201c700 1 --2- 172.21.15.121:0/879353425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddbc0ffa40 0x7fddbc1974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:44:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc181b700 1 --2- 172.21.15.121:0/879353425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddbc105a40 0x7fddbc197a00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fddac00d8d0 tx=0x7fddac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddb37fe700 1 -- 172.21.15.121:0/879353425 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddac009940 con 0x7fddbc105a40 2024-04-05T21:44:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddb37fe700 1 -- 172.21.15.121:0/879353425 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fddac010460 con 0x7fddbc105a40 2024-04-05T21:44:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc4280700 1 -- 172.21.15.121:0/879353425 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddbc1a0530 con 0x7fddbc105a40 2024-04-05T21:44:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddb37fe700 1 -- 172.21.15.121:0/879353425 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddac00f5f0 con 0x7fddbc105a40 2024-04-05T21:44:28.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.253+0000 7fddc4280700 1 -- 172.21.15.121:0/879353425 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddbc1a09f0 con 0x7fddbc105a40 2024-04-05T21:44:28.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.254+0000 7fddc4280700 1 -- 172.21.15.121:0/879353425 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddbc04ea90 con 0x7fddbc105a40 2024-04-05T21:44:28.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.255+0000 7fddb37fe700 1 -- 172.21.15.121:0/879353425 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fddac0105d0 con 0x7fddbc105a40 2024-04-05T21:44:28.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.255+0000 7fddb37fe700 1 --2- 172.21.15.121:0/879353425 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdda806cce0 0x7fdda806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:28.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.255+0000 7fddb37fe700 1 -- 172.21.15.121:0/879353425 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fddac08c5e0 con 0x7fddbc105a40 2024-04-05T21:44:28.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.255+0000 7fddc201c700 1 --2- 172.21.15.121:0/879353425 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdda806cce0 0x7fdda806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:28.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.256+0000 7fddc201c700 1 --2- 172.21.15.121:0/879353425 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdda806cce0 0x7fdda806f1a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fddb800b5c0 tx=0x7fddb8005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:28.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.260+0000 7fddb37fe700 1 -- 172.21.15.121:0/879353425 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fddac056650 con 0x7fddbc105a40 2024-04-05T21:44:28.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.462+0000 7fddc4280700 1 -- 172.21.15.121:0/879353425 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fddbc0fc6f0 con 0x7fdda806cce0 2024-04-05T21:44:28.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.472+0000 7fddb37fe700 1 -- 172.21.15.121:0/879353425 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fddbc0fc6f0 con 0x7fdda806cce0 2024-04-05T21:44:28.473 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:44:28.473 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (15m) 9m ago 26m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:44:28.473 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (27m) 9m ago 27m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:44:28.473 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (25m) 8m ago 25m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:44:28.473 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (13m) 9m ago 26m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:44:28.473 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (13m) 8m ago 25m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:44:28.473 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (15m) 9m ago 26m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (9m) 9m ago 20m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (9m) 9m ago 20m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (9m) 8m ago 20m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (20m) 8m ago 20m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (17m) 9m ago 28m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (16m) 8m ago 25m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (14m) 9m ago 28m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (13m) 8m ago 25m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (16m) 9m ago 26m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (16m) 8m ago 25m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (13m) 9m ago 24m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (12m) 9m ago 24m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (11m) 9m ago 23m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:44:28.474 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (11m) 8m ago 23m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:44:28.475 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (10m) 8m ago 22m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:44:28.475 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (10m) 8m ago 22m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:44:28.475 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (15m) 9m ago 26m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:44:28.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.474+0000 7fddc4280700 1 -- 172.21.15.121:0/879353425 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdda806cce0 msgr2=0x7fdda806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:28.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.474+0000 7fddc4280700 1 --2- 172.21.15.121:0/879353425 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdda806cce0 0x7fdda806f1a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fddb800b5c0 tx=0x7fddb8005960 comp rx=0 tx=0).stop 2024-04-05T21:44:28.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.474+0000 7fddc4280700 1 -- 172.21.15.121:0/879353425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddbc105a40 msgr2=0x7fddbc197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:28.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.474+0000 7fddc4280700 1 --2- 172.21.15.121:0/879353425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddbc105a40 0x7fddbc197a00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fddac00d8d0 tx=0x7fddac00dc90 comp rx=0 tx=0).stop 2024-04-05T21:44:28.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.475+0000 7fddc4280700 1 -- 172.21.15.121:0/879353425 shutdown_connections 2024-04-05T21:44:28.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.475+0000 7fddc4280700 1 --2- 172.21.15.121:0/879353425 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdda806cce0 0x7fdda806f1a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.475+0000 7fddc4280700 1 --2- 172.21.15.121:0/879353425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fddbc0ffa40 0x7fddbc1974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.475+0000 7fddc4280700 1 --2- 172.21.15.121:0/879353425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddbc105a40 0x7fddbc197a00 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.475+0000 7fddc4280700 1 -- 172.21.15.121:0/879353425 >> 172.21.15.121:0/879353425 conn(0x7fddbc0fb570 msgr2=0x7fddbc0fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:28.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.475+0000 7fddc4280700 1 -- 172.21.15.121:0/879353425 shutdown_connections 2024-04-05T21:44:28.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.475+0000 7fddc4280700 1 -- 172.21.15.121:0/879353425 wait complete. 2024-04-05T21:44:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.577+0000 7fc964277700 1 -- 172.21.15.121:0/2491595586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c0ffa50 msgr2=0x7fc95c0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.577+0000 7fc964277700 1 --2- 172.21.15.121:0/2491595586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c0ffa50 0x7fc95c0ffed0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc958009910 tx=0x7fc958009c20 comp rx=0 tx=0).stop 2024-04-05T21:44:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.577+0000 7fc964277700 1 -- 172.21.15.121:0/2491595586 shutdown_connections 2024-04-05T21:44:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.577+0000 7fc964277700 1 --2- 172.21.15.121:0/2491595586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c0ffa50 0x7fc95c0ffed0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.577+0000 7fc964277700 1 --2- 172.21.15.121:0/2491595586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc95c105a50 0x7fc95c105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.577+0000 7fc964277700 1 -- 172.21.15.121:0/2491595586 >> 172.21.15.121:0/2491595586 conn(0x7fc95c0fb540 msgr2=0x7fc95c0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.577+0000 7fc964277700 1 -- 172.21.15.121:0/2491595586 shutdown_connections 2024-04-05T21:44:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.577+0000 7fc964277700 1 -- 172.21.15.121:0/2491595586 wait complete. 2024-04-05T21:44:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.577+0000 7fc964277700 1 Processor -- start 2024-04-05T21:44:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.577+0000 7fc964277700 1 -- start start 2024-04-05T21:44:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc964277700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc95c0ffa50 0x7fc95c0fbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc964277700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c105a50 0x7fc95c0fc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc964277700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c0fc930 con 0x7fc95c105a50 2024-04-05T21:44:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc964277700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c0fca70 con 0x7fc95c0ffa50 2024-04-05T21:44:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc961812700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c105a50 0x7fc95c0fc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc961812700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c105a50 0x7fc95c0fc3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46640/0 (socket says 172.21.15.121:46640) 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc961812700 1 -- 172.21.15.121:0/60219690 learned_addr learned my addr 172.21.15.121:0/60219690 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc962013700 1 --2- 172.21.15.121:0/60219690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc95c0ffa50 0x7fc95c0fbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc961812700 1 -- 172.21.15.121:0/60219690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc95c0ffa50 msgr2=0x7fc95c0fbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc961812700 1 --2- 172.21.15.121:0/60219690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc95c0ffa50 0x7fc95c0fbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc961812700 1 -- 172.21.15.121:0/60219690 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9580095f0 con 0x7fc95c105a50 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc962013700 1 --2- 172.21.15.121:0/60219690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc95c0ffa50 0x7fc95c0fbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc961812700 1 --2- 172.21.15.121:0/60219690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c105a50 0x7fc95c0fc3f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc958006010 tx=0x7fc95800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc9537fe700 1 -- 172.21.15.121:0/60219690 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc95801d070 con 0x7fc95c105a50 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc9537fe700 1 -- 172.21.15.121:0/60219690 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc95800f460 con 0x7fc95c105a50 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc964277700 1 -- 172.21.15.121:0/60219690 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95c0fccf0 con 0x7fc95c105a50 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc9537fe700 1 -- 172.21.15.121:0/60219690 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc958021640 con 0x7fc95c105a50 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.578+0000 7fc964277700 1 -- 172.21.15.121:0/60219690 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95c1a2c50 con 0x7fc95c105a50 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.579+0000 7fc964277700 1 -- 172.21.15.121:0/60219690 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95c109080 con 0x7fc95c105a50 2024-04-05T21:44:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.580+0000 7fc9537fe700 1 -- 172.21.15.121:0/60219690 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc95800f9e0 con 0x7fc95c105a50 2024-04-05T21:44:28.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.580+0000 7fc9537fe700 1 --2- 172.21.15.121:0/60219690 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc94806cce0 0x7fc94806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:28.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.580+0000 7fc9537fe700 1 -- 172.21.15.121:0/60219690 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc95808d830 con 0x7fc95c105a50 2024-04-05T21:44:28.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.580+0000 7fc962013700 1 --2- 172.21.15.121:0/60219690 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc94806cce0 0x7fc94806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:28.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.581+0000 7fc962013700 1 --2- 172.21.15.121:0/60219690 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc94806cce0 0x7fc94806f1a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc94c007900 tx=0x7fc94c008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:28.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.585+0000 7fc9537fe700 1 -- 172.21.15.121:0/60219690 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc9580578a0 con 0x7fc95c105a50 2024-04-05T21:44:28.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:28 smithi121 ceph-mon[145859]: from='client.34846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:44:28.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:28 smithi188 ceph-mon[107793]: from='client.34846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:44:28.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.858+0000 7fc964277700 1 -- 172.21.15.121:0/60219690 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc95c02cc70 con 0x7fc95c105a50 2024-04-05T21:44:28.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.858+0000 7fc9537fe700 1 -- 172.21.15.121:0/60219690 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7fc958057030 con 0x7fc95c105a50 2024-04-05T21:44:28.859 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:44:28.859 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:44:28.859 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:44:28.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:44:28.859 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:44:28.859 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:44:28.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:44:28.859 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:44:28.859 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:44:28.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:44:28.860 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:44:28.860 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:44:28.860 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:44:28.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:44:28.860 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:44:28.860 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:44:28.860 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:44:28.860 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:44:28.860 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:44:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.860+0000 7fc964277700 1 -- 172.21.15.121:0/60219690 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc94806cce0 msgr2=0x7fc94806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.860+0000 7fc964277700 1 --2- 172.21.15.121:0/60219690 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc94806cce0 0x7fc94806f1a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc94c007900 tx=0x7fc94c008040 comp rx=0 tx=0).stop 2024-04-05T21:44:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.860+0000 7fc964277700 1 -- 172.21.15.121:0/60219690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c105a50 msgr2=0x7fc95c0fc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.860+0000 7fc964277700 1 --2- 172.21.15.121:0/60219690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c105a50 0x7fc95c0fc3f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc958006010 tx=0x7fc95800be20 comp rx=0 tx=0).stop 2024-04-05T21:44:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.860+0000 7fc964277700 1 -- 172.21.15.121:0/60219690 shutdown_connections 2024-04-05T21:44:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.860+0000 7fc964277700 1 --2- 172.21.15.121:0/60219690 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc94806cce0 0x7fc94806f1a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.860+0000 7fc964277700 1 --2- 172.21.15.121:0/60219690 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc95c0ffa50 0x7fc95c0fbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.860+0000 7fc964277700 1 --2- 172.21.15.121:0/60219690 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c105a50 0x7fc95c0fc3f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.860+0000 7fc964277700 1 -- 172.21.15.121:0/60219690 >> 172.21.15.121:0/60219690 conn(0x7fc95c0fb540 msgr2=0x7fc95c069070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:28.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.860+0000 7fc964277700 1 -- 172.21.15.121:0/60219690 shutdown_connections 2024-04-05T21:44:28.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.860+0000 7fc964277700 1 -- 172.21.15.121:0/60219690 wait complete. 2024-04-05T21:44:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.963+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3084225097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e8c105a40 msgr2=0x7f0e8c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.963+0000 7f0e93e8e700 1 --2- 172.21.15.121:0/3084225097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e8c105a40 0x7f0e8c105e20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c009910 tx=0x7f0e7c009c20 comp rx=0 tx=0).stop 2024-04-05T21:44:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.963+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3084225097 shutdown_connections 2024-04-05T21:44:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.963+0000 7f0e93e8e700 1 --2- 172.21.15.121:0/3084225097 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e8c0ffbd0 0x7f0e8c100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.963+0000 7f0e93e8e700 1 --2- 172.21.15.121:0/3084225097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e8c105a40 0x7f0e8c105e20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.963+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3084225097 >> 172.21.15.121:0/3084225097 conn(0x7f0e8c0755a0 msgr2=0x7f0e8c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.963+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3084225097 shutdown_connections 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.963+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3084225097 wait complete. 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.963+0000 7f0e93e8e700 1 Processor -- start 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.963+0000 7f0e93e8e700 1 -- start start 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e93e8e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e8c0ffbd0 0x7f0e8c195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e93e8e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e8c105a40 0x7f0e8c195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e93e8e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c196170 con 0x7f0e8c0ffbd0 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e93e8e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c199f00 con 0x7f0e8c105a40 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e91c2a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e8c0ffbd0 0x7f0e8c195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e91c2a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e8c0ffbd0 0x7f0e8c195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46652/0 (socket says 172.21.15.121:46652) 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e91c2a700 1 -- 172.21.15.121:0/3402987175 learned_addr learned my addr 172.21.15.121:0/3402987175 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e91c2a700 1 -- 172.21.15.121:0/3402987175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e8c105a40 msgr2=0x7f0e8c195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:28.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e91c2a700 1 --2- 172.21.15.121:0/3402987175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e8c105a40 0x7f0e8c195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:28.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e91c2a700 1 -- 172.21.15.121:0/3402987175 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7c0095f0 con 0x7f0e8c0ffbd0 2024-04-05T21:44:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e91c2a700 1 --2- 172.21.15.121:0/3402987175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e8c0ffbd0 0x7f0e8c195550 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c006010 tx=0x7f0e7c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e82ffd700 1 -- 172.21.15.121:0/3402987175 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e7c01d070 con 0x7f0e8c0ffbd0 2024-04-05T21:44:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e82ffd700 1 -- 172.21.15.121:0/3402987175 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e7c00f460 con 0x7f0e8c0ffbd0 2024-04-05T21:44:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e82ffd700 1 -- 172.21.15.121:0/3402987175 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e7c021620 con 0x7f0e8c0ffbd0 2024-04-05T21:44:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3402987175 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8c19a180 con 0x7f0e8c0ffbd0 2024-04-05T21:44:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.964+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3402987175 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8c19a670 con 0x7f0e8c0ffbd0 2024-04-05T21:44:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.965+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3402987175 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8c108040 con 0x7f0e8c0ffbd0 2024-04-05T21:44:28.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.966+0000 7f0e82ffd700 1 -- 172.21.15.121:0/3402987175 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0e7c00fa40 con 0x7f0e8c0ffbd0 2024-04-05T21:44:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.966+0000 7f0e82ffd700 1 --2- 172.21.15.121:0/3402987175 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0e7806cce0 0x7f0e7806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.966+0000 7f0e82ffd700 1 -- 172.21.15.121:0/3402987175 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0e7c08d790 con 0x7f0e8c0ffbd0 2024-04-05T21:44:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.966+0000 7f0e91429700 1 --2- 172.21.15.121:0/3402987175 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0e7806cce0 0x7f0e7806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:28.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.967+0000 7f0e91429700 1 --2- 172.21.15.121:0/3402987175 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0e7806cce0 0x7f0e7806f1a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0e88005950 tx=0x7f0e880058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:28.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:28.971+0000 7f0e82ffd700 1 -- 172.21.15.121:0/3402987175 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0e7c057800 con 0x7f0e8c0ffbd0 2024-04-05T21:44:29.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.205+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3402987175 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0e8c02cdd0 con 0x7f0e8c0ffbd0 2024-04-05T21:44:29.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.206+0000 7f0e82ffd700 1 -- 172.21.15.121:0/3402987175 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f0e7c05ae20 con 0x7f0e8c0ffbd0 2024-04-05T21:44:29.207 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:44:29.207 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:44:29.207 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:44:29.207 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:44:29.207 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:44:29.207 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:44:29.207 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:44:29.207 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:44:29.207 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:44:29.207 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:44:29.207 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:44:29.207 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:44:29.207 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:44:29.208 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:44:29.209 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:44:29.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.208+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3402987175 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0e7806cce0 msgr2=0x7f0e7806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:29.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.208+0000 7f0e93e8e700 1 --2- 172.21.15.121:0/3402987175 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0e7806cce0 0x7f0e7806f1a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0e88005950 tx=0x7f0e880058e0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.208+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3402987175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e8c0ffbd0 msgr2=0x7f0e8c195550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:29.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.208+0000 7f0e93e8e700 1 --2- 172.21.15.121:0/3402987175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e8c0ffbd0 0x7f0e8c195550 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c006010 tx=0x7f0e7c00be20 comp rx=0 tx=0).stop 2024-04-05T21:44:29.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.208+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3402987175 shutdown_connections 2024-04-05T21:44:29.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.208+0000 7f0e93e8e700 1 --2- 172.21.15.121:0/3402987175 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0e7806cce0 0x7f0e7806f1a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.208+0000 7f0e93e8e700 1 --2- 172.21.15.121:0/3402987175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e8c0ffbd0 0x7f0e8c195550 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.208+0000 7f0e93e8e700 1 --2- 172.21.15.121:0/3402987175 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0e8c105a40 0x7f0e8c195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.208+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3402987175 >> 172.21.15.121:0/3402987175 conn(0x7f0e8c0755a0 msgr2=0x7f0e8c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:29.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.208+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3402987175 shutdown_connections 2024-04-05T21:44:29.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.208+0000 7f0e93e8e700 1 -- 172.21.15.121:0/3402987175 wait complete. 2024-04-05T21:44:29.211 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:44:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.312+0000 7fe43baf6700 1 -- 172.21.15.121:0/2569496625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe434105a40 msgr2=0x7fe434105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.312+0000 7fe43baf6700 1 --2- 172.21.15.121:0/2569496625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe434105a40 0x7fe434105e20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009910 tx=0x7fe42c009c20 comp rx=0 tx=0).stop 2024-04-05T21:44:29.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.312+0000 7fe43baf6700 1 -- 172.21.15.121:0/2569496625 shutdown_connections 2024-04-05T21:44:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.312+0000 7fe43baf6700 1 --2- 172.21.15.121:0/2569496625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4340ffbd0 0x7fe434100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.312+0000 7fe43baf6700 1 --2- 172.21.15.121:0/2569496625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe434105a40 0x7fe434105e20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.312+0000 7fe43baf6700 1 -- 172.21.15.121:0/2569496625 >> 172.21.15.121:0/2569496625 conn(0x7fe4340755a0 msgr2=0x7fe4340759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.312+0000 7fe43baf6700 1 -- 172.21.15.121:0/2569496625 shutdown_connections 2024-04-05T21:44:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.312+0000 7fe43baf6700 1 -- 172.21.15.121:0/2569496625 wait complete. 2024-04-05T21:44:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe43baf6700 1 Processor -- start 2024-04-05T21:44:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe43baf6700 1 -- start start 2024-04-05T21:44:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe43baf6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4340ffbd0 0x7fe4341974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe43baf6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe434105a40 0x7fe4341979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe43baf6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe434198000 con 0x7fe4340ffbd0 2024-04-05T21:44:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe43baf6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe434198170 con 0x7fe434105a40 2024-04-05T21:44:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe439892700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4340ffbd0 0x7fe4341974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:29.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe439892700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4340ffbd0 0x7fe4341974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46664/0 (socket says 172.21.15.121:46664) 2024-04-05T21:44:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe439892700 1 -- 172.21.15.121:0/1457161158 learned_addr learned my addr 172.21.15.121:0/1457161158 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:44:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe439091700 1 --2- 172.21.15.121:0/1457161158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe434105a40 0x7fe4341979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe439892700 1 -- 172.21.15.121:0/1457161158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe434105a40 msgr2=0x7fe4341979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe439892700 1 --2- 172.21.15.121:0/1457161158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe434105a40 0x7fe4341979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe439892700 1 -- 172.21.15.121:0/1457161158 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c0095f0 con 0x7fe4340ffbd0 2024-04-05T21:44:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.313+0000 7fe439091700 1 --2- 172.21.15.121:0/1457161158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe434105a40 0x7fe4341979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:44:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.314+0000 7fe439892700 1 --2- 172.21.15.121:0/1457161158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4340ffbd0 0x7fe4341974a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe42c006010 tx=0x7fe42c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.314+0000 7fe432ffd700 1 -- 172.21.15.121:0/1457161158 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe42c01d070 con 0x7fe4340ffbd0 2024-04-05T21:44:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.314+0000 7fe432ffd700 1 -- 172.21.15.121:0/1457161158 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe42c00f460 con 0x7fe4340ffbd0 2024-04-05T21:44:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.314+0000 7fe43baf6700 1 -- 172.21.15.121:0/1457161158 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4341a0510 con 0x7fe4340ffbd0 2024-04-05T21:44:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.314+0000 7fe432ffd700 1 -- 172.21.15.121:0/1457161158 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe42c021620 con 0x7fe4340ffbd0 2024-04-05T21:44:29.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.314+0000 7fe43baf6700 1 -- 172.21.15.121:0/1457161158 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4341a0980 con 0x7fe4340ffbd0 2024-04-05T21:44:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.315+0000 7fe43baf6700 1 -- 172.21.15.121:0/1457161158 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe434108040 con 0x7fe4340ffbd0 2024-04-05T21:44:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.315+0000 7fe432ffd700 1 -- 172.21.15.121:0/1457161158 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe42c00fa40 con 0x7fe4340ffbd0 2024-04-05T21:44:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.316+0000 7fe432ffd700 1 --2- 172.21.15.121:0/1457161158 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe41c06cce0 0x7fe41c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.316+0000 7fe432ffd700 1 -- 172.21.15.121:0/1457161158 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe42c08d6c0 con 0x7fe4340ffbd0 2024-04-05T21:44:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.316+0000 7fe439091700 1 --2- 172.21.15.121:0/1457161158 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe41c06cce0 0x7fe41c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:29.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.316+0000 7fe439091700 1 --2- 172.21.15.121:0/1457161158 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe41c06cce0 0x7fe41c06f1a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe434068ce0 tx=0x7fe4280092a0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:29.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.320+0000 7fe432ffd700 1 -- 172.21.15.121:0/1457161158 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe42c0577b0 con 0x7fe4340ffbd0 2024-04-05T21:44:29.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.528+0000 7fe43baf6700 1 -- 172.21.15.121:0/1457161158 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe43402cd80 con 0x7fe41c06cce0 2024-04-05T21:44:29.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.529+0000 7fe432ffd700 1 -- 172.21.15.121:0/1457161158 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe43402cd80 con 0x7fe41c06cce0 2024-04-05T21:44:29.530 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:44:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:44:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:44:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:44:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:44:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:44:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:44:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:44:29.530 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:44:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:44:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:44:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:44:29.531 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:44:29.531 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:44:29.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.531+0000 7fe43baf6700 1 -- 172.21.15.121:0/1457161158 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe41c06cce0 msgr2=0x7fe41c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:29.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.531+0000 7fe43baf6700 1 --2- 172.21.15.121:0/1457161158 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe41c06cce0 0x7fe41c06f1a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe434068ce0 tx=0x7fe4280092a0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.531+0000 7fe43baf6700 1 -- 172.21.15.121:0/1457161158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4340ffbd0 msgr2=0x7fe4341974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:29.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.531+0000 7fe43baf6700 1 --2- 172.21.15.121:0/1457161158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4340ffbd0 0x7fe4341974a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe42c006010 tx=0x7fe42c00be20 comp rx=0 tx=0).stop 2024-04-05T21:44:29.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.531+0000 7fe43baf6700 1 -- 172.21.15.121:0/1457161158 shutdown_connections 2024-04-05T21:44:29.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.531+0000 7fe43baf6700 1 --2- 172.21.15.121:0/1457161158 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe41c06cce0 0x7fe41c06f1a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.531+0000 7fe43baf6700 1 --2- 172.21.15.121:0/1457161158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4340ffbd0 0x7fe4341974a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.531+0000 7fe43baf6700 1 --2- 172.21.15.121:0/1457161158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe434105a40 0x7fe4341979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.531+0000 7fe43baf6700 1 -- 172.21.15.121:0/1457161158 >> 172.21.15.121:0/1457161158 conn(0x7fe4340755a0 msgr2=0x7fe4340fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:29.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.531+0000 7fe43baf6700 1 -- 172.21.15.121:0/1457161158 shutdown_connections 2024-04-05T21:44:29.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.531+0000 7fe43baf6700 1 -- 172.21.15.121:0/1457161158 wait complete. 2024-04-05T21:44:29.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.634+0000 7f00bce66700 1 -- 172.21.15.121:0/2013695902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b81026c0 msgr2=0x7f00b810abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:29.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.634+0000 7f00bce66700 1 --2- 172.21.15.121:0/2013695902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b81026c0 0x7f00b810abb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f00ac009910 tx=0x7f00ac009c20 comp rx=0 tx=0).stop 2024-04-05T21:44:29.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.634+0000 7f00bce66700 1 -- 172.21.15.121:0/2013695902 shutdown_connections 2024-04-05T21:44:29.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.634+0000 7f00bce66700 1 --2- 172.21.15.121:0/2013695902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b81026c0 0x7f00b810abb0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.634+0000 7f00bce66700 1 --2- 172.21.15.121:0/2013695902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00b8101da0 0x7f00b8102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.634+0000 7f00bce66700 1 -- 172.21.15.121:0/2013695902 >> 172.21.15.121:0/2013695902 conn(0x7f00b80fb570 msgr2=0x7f00b80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:29.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.634+0000 7f00bce66700 1 -- 172.21.15.121:0/2013695902 shutdown_connections 2024-04-05T21:44:29.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.634+0000 7f00bce66700 1 -- 172.21.15.121:0/2013695902 wait complete. 2024-04-05T21:44:29.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.634+0000 7f00bce66700 1 Processor -- start 2024-04-05T21:44:29.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00bce66700 1 -- start start 2024-04-05T21:44:29.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00bce66700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b8101da0 0x7f00b81966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00bce66700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00b81026c0 0x7f00b8196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00bce66700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b8197290 con 0x7f00b8101da0 2024-04-05T21:44:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00bce66700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b8199fc0 con 0x7f00b81026c0 2024-04-05T21:44:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00b659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b8101da0 0x7f00b81966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00b659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b8101da0 0x7f00b81966c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46678/0 (socket says 172.21.15.121:46678) 2024-04-05T21:44:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00b659c700 1 -- 172.21.15.121:0/1168855597 learned_addr learned my addr 172.21.15.121:0/1168855597 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:44:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00b5d9b700 1 --2- 172.21.15.121:0/1168855597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00b81026c0 0x7f00b8196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00b659c700 1 -- 172.21.15.121:0/1168855597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00b81026c0 msgr2=0x7f00b8196c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00b659c700 1 --2- 172.21.15.121:0/1168855597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00b81026c0 0x7f00b8196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00b659c700 1 -- 172.21.15.121:0/1168855597 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac0095f0 con 0x7f00b8101da0 2024-04-05T21:44:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00b659c700 1 --2- 172.21.15.121:0/1168855597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b8101da0 0x7f00b81966c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f00a800e920 tx=0x7f00a800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00a77fe700 1 -- 172.21.15.121:0/1168855597 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00a8009790 con 0x7f00b8101da0 2024-04-05T21:44:29.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00a77fe700 1 -- 172.21.15.121:0/1168855597 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00a800cba0 con 0x7f00b8101da0 2024-04-05T21:44:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00bce66700 1 -- 172.21.15.121:0/1168855597 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b819a2a0 con 0x7f00b8101da0 2024-04-05T21:44:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00a77fe700 1 -- 172.21.15.121:0/1168855597 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00a8018650 con 0x7f00b8101da0 2024-04-05T21:44:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.635+0000 7f00bce66700 1 -- 172.21.15.121:0/1168855597 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b819a7f0 con 0x7f00b8101da0 2024-04-05T21:44:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.636+0000 7f00bce66700 1 -- 172.21.15.121:0/1168855597 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b81082b0 con 0x7f00b8101da0 2024-04-05T21:44:29.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.637+0000 7f00a77fe700 1 -- 172.21.15.121:0/1168855597 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f00a800cd10 con 0x7f00b8101da0 2024-04-05T21:44:29.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.637+0000 7f00a77fe700 1 --2- 172.21.15.121:0/1168855597 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f00a006cce0 0x7f00a006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:44:29.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.637+0000 7f00a77fe700 1 -- 172.21.15.121:0/1168855597 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f00a8014070 con 0x7f00b8101da0 2024-04-05T21:44:29.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.637+0000 7f00b5d9b700 1 --2- 172.21.15.121:0/1168855597 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f00a006cce0 0x7f00a006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:44:29.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.638+0000 7f00b5d9b700 1 --2- 172.21.15.121:0/1168855597 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f00a006cce0 0x7f00a006f1a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f00ac009fd0 tx=0x7f00ac00b5c0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:44:29.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.642+0000 7f00a77fe700 1 -- 172.21.15.121:0/1168855597 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f00a8052ab0 con 0x7f00b8101da0 2024-04-05T21:44:29.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:29 smithi121 ceph-mon[145859]: from='client.34850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:44:29.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:29 smithi121 ceph-mon[145859]: pgmap v547: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:29 smithi121 ceph-mon[145859]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:44:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:29 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/60219690' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:44:29.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:29 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3402987175' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:44:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:29 smithi188 ceph-mon[107793]: from='client.34850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:44:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:29 smithi188 ceph-mon[107793]: pgmap v547: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:29 smithi188 ceph-mon[107793]: from='client.34854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:44:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:29 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/60219690' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:44:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:29 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3402987175' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:44:29.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.912+0000 7f00bce66700 1 -- 172.21.15.121:0/1168855597 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f00b804ea90 con 0x7f00b8101da0 2024-04-05T21:44:29.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.913+0000 7f00a77fe700 1 -- 172.21.15.121:0/1168855597 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f00a805a450 con 0x7f00b8101da0 2024-04-05T21:44:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:44:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:44:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:44:29.914 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:44:29.914 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:44:29.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.915+0000 7f00bce66700 1 -- 172.21.15.121:0/1168855597 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f00a006cce0 msgr2=0x7f00a006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:29.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.915+0000 7f00bce66700 1 --2- 172.21.15.121:0/1168855597 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f00a006cce0 0x7f00a006f1a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f00ac009fd0 tx=0x7f00ac00b5c0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.915+0000 7f00bce66700 1 -- 172.21.15.121:0/1168855597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b8101da0 msgr2=0x7f00b81966c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:44:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.915+0000 7f00bce66700 1 --2- 172.21.15.121:0/1168855597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b8101da0 0x7f00b81966c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f00a800e920 tx=0x7f00a800ece0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.915+0000 7f00bce66700 1 -- 172.21.15.121:0/1168855597 shutdown_connections 2024-04-05T21:44:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.915+0000 7f00bce66700 1 --2- 172.21.15.121:0/1168855597 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f00a006cce0 0x7f00a006f1a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.915+0000 7f00bce66700 1 --2- 172.21.15.121:0/1168855597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b8101da0 0x7f00b81966c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.915+0000 7f00bce66700 1 --2- 172.21.15.121:0/1168855597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f00b81026c0 0x7f00b8196c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:44:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.915+0000 7f00bce66700 1 -- 172.21.15.121:0/1168855597 >> 172.21.15.121:0/1168855597 conn(0x7f00b80fb570 msgr2=0x7f00b81053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:44:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.915+0000 7f00bce66700 1 -- 172.21.15.121:0/1168855597 shutdown_connections 2024-04-05T21:44:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:44:29.915+0000 7f00bce66700 1 -- 172.21.15.121:0/1168855597 wait complete. 2024-04-05T21:44:30.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:30 smithi121 ceph-mon[145859]: from='client.34866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:44:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:30 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1168855597' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:44:30.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:44:30.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:30 smithi188 ceph-mon[107793]: from='client.34866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:44:30.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:30 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1168855597' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:44:30.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:44:31.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:31 smithi121 ceph-mon[145859]: pgmap v548: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:31.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:31 smithi188 ceph-mon[107793]: pgmap v548: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:33.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:33 smithi121 ceph-mon[145859]: pgmap v549: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:33 smithi188 ceph-mon[107793]: pgmap v549: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:35.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:35 smithi121 ceph-mon[145859]: pgmap v550: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:35.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:35 smithi188 ceph-mon[107793]: pgmap v550: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:37.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:37 smithi121 ceph-mon[145859]: pgmap v551: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:37.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:37 smithi188 ceph-mon[107793]: pgmap v551: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:39.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:39 smithi121 ceph-mon[145859]: pgmap v552: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:39.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:39 smithi188 ceph-mon[107793]: pgmap v552: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:41.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:41 smithi121 ceph-mon[145859]: pgmap v553: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:41.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:41 smithi188 ceph-mon[107793]: pgmap v553: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:43.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:43 smithi121 ceph-mon[145859]: pgmap v554: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:43.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:43 smithi188 ceph-mon[107793]: pgmap v554: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:45.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:45 smithi121 ceph-mon[145859]: pgmap v555: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:45.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:44:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:45 smithi188 ceph-mon[107793]: pgmap v555: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:45.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:44:47.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:47 smithi121 ceph-mon[145859]: pgmap v556: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:47 smithi188 ceph-mon[107793]: pgmap v556: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:49.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:49 smithi188 ceph-mon[107793]: pgmap v557: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:49.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:49 smithi121 ceph-mon[145859]: pgmap v557: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:51.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:51 smithi188 ceph-mon[107793]: pgmap v558: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:51.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:51 smithi121 ceph-mon[145859]: pgmap v558: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:53 smithi188 ceph-mon[107793]: pgmap v559: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:53.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:53 smithi121 ceph-mon[145859]: pgmap v559: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:55.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:55 smithi188 ceph-mon[107793]: pgmap v560: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:55.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:55 smithi121 ceph-mon[145859]: pgmap v560: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:57 smithi188 ceph-mon[107793]: pgmap v561: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:57.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:57 smithi121 ceph-mon[145859]: pgmap v561: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:44:59 smithi188 ceph-mon[107793]: pgmap v562: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:44:59.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:44:59 smithi121 ceph-mon[145859]: pgmap v562: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.031+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3876621637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ccc0ffa50 msgr2=0x7f7ccc0ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.031+0000 7f7cd2d62700 1 --2- 172.21.15.121:0/3876621637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ccc0ffa50 0x7f7ccc0ffed0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009910 tx=0x7f7cb8009c20 comp rx=0 tx=0).stop 2024-04-05T21:45:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.031+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3876621637 shutdown_connections 2024-04-05T21:45:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.031+0000 7f7cd2d62700 1 --2- 172.21.15.121:0/3876621637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ccc0ffa50 0x7f7ccc0ffed0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.031+0000 7f7cd2d62700 1 --2- 172.21.15.121:0/3876621637 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ccc105a50 0x7f7ccc105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.031+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3876621637 >> 172.21.15.121:0/3876621637 conn(0x7f7ccc0fb540 msgr2=0x7f7ccc0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.031+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3876621637 shutdown_connections 2024-04-05T21:45:00.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.031+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3876621637 wait complete. 2024-04-05T21:45:00.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.032+0000 7f7cd2d62700 1 Processor -- start 2024-04-05T21:45:00.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.032+0000 7f7cd2d62700 1 -- start start 2024-04-05T21:45:00.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.032+0000 7f7cd2d62700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ccc0ffa50 0x7f7ccc1955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:00.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.032+0000 7f7cd2d62700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ccc105a50 0x7f7ccc195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:00.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.032+0000 7f7cd2d62700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc196200 con 0x7f7ccc105a50 2024-04-05T21:45:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.032+0000 7f7cd2d62700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc199f90 con 0x7f7ccc0ffa50 2024-04-05T21:45:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.033+0000 7f7ccbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ccc105a50 0x7f7ccc195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.033+0000 7f7ccbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ccc105a50 0x7f7ccc195b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44362/0 (socket says 172.21.15.121:44362) 2024-04-05T21:45:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.033+0000 7f7ccbfff700 1 -- 172.21.15.121:0/3788987699 learned_addr learned my addr 172.21.15.121:0/3788987699 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.033+0000 7f7cd0afe700 1 --2- 172.21.15.121:0/3788987699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ccc0ffa50 0x7f7ccc1955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.033+0000 7f7ccbfff700 1 -- 172.21.15.121:0/3788987699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ccc0ffa50 msgr2=0x7f7ccc1955e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.033+0000 7f7ccbfff700 1 --2- 172.21.15.121:0/3788987699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ccc0ffa50 0x7f7ccc1955e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.033+0000 7f7ccbfff700 1 -- 172.21.15.121:0/3788987699 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb80095f0 con 0x7f7ccc105a50 2024-04-05T21:45:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.033+0000 7f7ccbfff700 1 --2- 172.21.15.121:0/3788987699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ccc105a50 0x7f7ccc195b20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8000c00 tx=0x7f7cb800bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.033+0000 7f7cc9ffb700 1 -- 172.21.15.121:0/3788987699 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cb801d070 con 0x7f7ccc105a50 2024-04-05T21:45:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.033+0000 7f7cc9ffb700 1 -- 172.21.15.121:0/3788987699 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cb8005dc0 con 0x7f7ccc105a50 2024-04-05T21:45:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.033+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3788987699 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc19a210 con 0x7f7ccc105a50 2024-04-05T21:45:00.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.033+0000 7f7cc9ffb700 1 -- 172.21.15.121:0/3788987699 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cb8003940 con 0x7f7ccc105a50 2024-04-05T21:45:00.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.034+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3788987699 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ccc19a6a0 con 0x7f7ccc105a50 2024-04-05T21:45:00.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.034+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3788987699 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ccc107fe0 con 0x7f7ccc105a50 2024-04-05T21:45:00.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.035+0000 7f7cc9ffb700 1 -- 172.21.15.121:0/3788987699 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7cb8004ae0 con 0x7f7ccc105a50 2024-04-05T21:45:00.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.035+0000 7f7cc9ffb700 1 --2- 172.21.15.121:0/3788987699 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cbc06cce0 0x7f7cbc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:00.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.035+0000 7f7cc9ffb700 1 -- 172.21.15.121:0/3788987699 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7cb808dc30 con 0x7f7ccc105a50 2024-04-05T21:45:00.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.036+0000 7f7cd0afe700 1 --2- 172.21.15.121:0/3788987699 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cbc06cce0 0x7f7cbc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:00.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.036+0000 7f7cd0afe700 1 --2- 172.21.15.121:0/3788987699 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cbc06cce0 0x7f7cbc06f1a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009c60 tx=0x7f7cc0009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:00.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.040+0000 7f7cc9ffb700 1 -- 172.21.15.121:0/3788987699 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7cb8057c00 con 0x7f7ccc105a50 2024-04-05T21:45:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.256+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3788987699 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ccc19a920 con 0x7f7cbc06cce0 2024-04-05T21:45:00.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.258+0000 7f7cc9ffb700 1 -- 172.21.15.121:0/3788987699 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7ccc19a920 con 0x7f7cbc06cce0 2024-04-05T21:45:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.260+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3788987699 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cbc06cce0 msgr2=0x7f7cbc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.260+0000 7f7cd2d62700 1 --2- 172.21.15.121:0/3788987699 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cbc06cce0 0x7f7cbc06f1a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009c60 tx=0x7f7cc0009260 comp rx=0 tx=0).stop 2024-04-05T21:45:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.260+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3788987699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ccc105a50 msgr2=0x7f7ccc195b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.260+0000 7f7cd2d62700 1 --2- 172.21.15.121:0/3788987699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ccc105a50 0x7f7ccc195b20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8000c00 tx=0x7f7cb800bde0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.260+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3788987699 shutdown_connections 2024-04-05T21:45:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.260+0000 7f7cd2d62700 1 --2- 172.21.15.121:0/3788987699 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cbc06cce0 0x7f7cbc06f1a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.260+0000 7f7cd2d62700 1 --2- 172.21.15.121:0/3788987699 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ccc0ffa50 0x7f7ccc1955e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.260+0000 7f7cd2d62700 1 --2- 172.21.15.121:0/3788987699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ccc105a50 0x7f7ccc195b20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.260+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3788987699 >> 172.21.15.121:0/3788987699 conn(0x7f7ccc0fb540 msgr2=0x7f7ccc0fc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.260+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3788987699 shutdown_connections 2024-04-05T21:45:00.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.260+0000 7f7cd2d62700 1 -- 172.21.15.121:0/3788987699 wait complete. 2024-04-05T21:45:00.271 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:45:00.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.364+0000 7f674a645700 1 -- 172.21.15.121:0/91249465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6744105a40 msgr2=0x7f6744105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:00.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.364+0000 7f674a645700 1 --2- 172.21.15.121:0/91249465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6744105a40 0x7f6744105e20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6734009910 tx=0x7f6734009c20 comp rx=0 tx=0).stop 2024-04-05T21:45:00.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.364+0000 7f674a645700 1 -- 172.21.15.121:0/91249465 shutdown_connections 2024-04-05T21:45:00.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.364+0000 7f674a645700 1 --2- 172.21.15.121:0/91249465 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67440ffbd0 0x7f6744100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.364+0000 7f674a645700 1 --2- 172.21.15.121:0/91249465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6744105a40 0x7f6744105e20 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.364+0000 7f674a645700 1 -- 172.21.15.121:0/91249465 >> 172.21.15.121:0/91249465 conn(0x7f67440755a0 msgr2=0x7f67440759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:00.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.364+0000 7f674a645700 1 -- 172.21.15.121:0/91249465 shutdown_connections 2024-04-05T21:45:00.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.364+0000 7f674a645700 1 -- 172.21.15.121:0/91249465 wait complete. 2024-04-05T21:45:00.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.365+0000 7f674a645700 1 Processor -- start 2024-04-05T21:45:00.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.365+0000 7f674a645700 1 -- start start 2024-04-05T21:45:00.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.365+0000 7f674a645700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67440ffbd0 0x7f6744196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.365+0000 7f674a645700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6744105a40 0x7f6744196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.365+0000 7f674a645700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6744197270 con 0x7f6744105a40 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.365+0000 7f674a645700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674419b000 con 0x7f67440ffbd0 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.365+0000 7f67437fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6744105a40 0x7f6744196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.365+0000 7f67437fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6744105a40 0x7f6744196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44370/0 (socket says 172.21.15.121:44370) 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.365+0000 7f67437fe700 1 -- 172.21.15.121:0/1985664542 learned_addr learned my addr 172.21.15.121:0/1985664542 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.366+0000 7f6743fff700 1 --2- 172.21.15.121:0/1985664542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67440ffbd0 0x7f6744196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.366+0000 7f67437fe700 1 -- 172.21.15.121:0/1985664542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67440ffbd0 msgr2=0x7f6744196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.366+0000 7f67437fe700 1 --2- 172.21.15.121:0/1985664542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67440ffbd0 0x7f6744196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.366+0000 7f67437fe700 1 -- 172.21.15.121:0/1985664542 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67340095f0 con 0x7f6744105a40 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.366+0000 7f67437fe700 1 --2- 172.21.15.121:0/1985664542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6744105a40 0x7f6744196b90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f673800d8d0 tx=0x7f673800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.366+0000 7f67417fa700 1 -- 172.21.15.121:0/1985664542 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67380098a0 con 0x7f6744105a40 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.366+0000 7f67417fa700 1 -- 172.21.15.121:0/1985664542 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6738010460 con 0x7f6744105a40 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.366+0000 7f674a645700 1 -- 172.21.15.121:0/1985664542 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674419b2e0 con 0x7f6744105a40 2024-04-05T21:45:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.366+0000 7f67417fa700 1 -- 172.21.15.121:0/1985664542 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6738009ea0 con 0x7f6744105a40 2024-04-05T21:45:00.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.366+0000 7f674a645700 1 -- 172.21.15.121:0/1985664542 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674419b830 con 0x7f6744105a40 2024-04-05T21:45:00.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.367+0000 7f674a645700 1 -- 172.21.15.121:0/1985664542 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6744108040 con 0x7f6744105a40 2024-04-05T21:45:00.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.368+0000 7f67417fa700 1 -- 172.21.15.121:0/1985664542 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f67380105d0 con 0x7f6744105a40 2024-04-05T21:45:00.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.368+0000 7f67417fa700 1 --2- 172.21.15.121:0/1985664542 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f672c06cce0 0x7f672c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:00.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.368+0000 7f67417fa700 1 -- 172.21.15.121:0/1985664542 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f673808c480 con 0x7f6744105a40 2024-04-05T21:45:00.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.368+0000 7f6743fff700 1 --2- 172.21.15.121:0/1985664542 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f672c06cce0 0x7f672c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:00.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.368+0000 7f6743fff700 1 --2- 172.21.15.121:0/1985664542 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f672c06cce0 0x7f672c06f1a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6734006010 tx=0x7f673401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:00.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.372+0000 7f67417fa700 1 -- 172.21.15.121:0/1985664542 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f67380564f0 con 0x7f6744105a40 2024-04-05T21:45:00.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.584+0000 7f674a645700 1 -- 172.21.15.121:0/1985664542 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f674419bb40 con 0x7f672c06cce0 2024-04-05T21:45:00.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.585+0000 7f67417fa700 1 -- 172.21.15.121:0/1985664542 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f674419bb40 con 0x7f672c06cce0 2024-04-05T21:45:00.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.587+0000 7f674a645700 1 -- 172.21.15.121:0/1985664542 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f672c06cce0 msgr2=0x7f672c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:00.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.587+0000 7f674a645700 1 --2- 172.21.15.121:0/1985664542 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f672c06cce0 0x7f672c06f1a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6734006010 tx=0x7f673401a040 comp rx=0 tx=0).stop 2024-04-05T21:45:00.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.587+0000 7f674a645700 1 -- 172.21.15.121:0/1985664542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6744105a40 msgr2=0x7f6744196b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:00.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.587+0000 7f674a645700 1 --2- 172.21.15.121:0/1985664542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6744105a40 0x7f6744196b90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f673800d8d0 tx=0x7f673800dbe0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.587+0000 7f674a645700 1 -- 172.21.15.121:0/1985664542 shutdown_connections 2024-04-05T21:45:00.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.587+0000 7f674a645700 1 --2- 172.21.15.121:0/1985664542 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f672c06cce0 0x7f672c06f1a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.587+0000 7f674a645700 1 --2- 172.21.15.121:0/1985664542 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67440ffbd0 0x7f6744196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.587+0000 7f674a645700 1 --2- 172.21.15.121:0/1985664542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6744105a40 0x7f6744196b90 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.587+0000 7f674a645700 1 -- 172.21.15.121:0/1985664542 >> 172.21.15.121:0/1985664542 conn(0x7f67440755a0 msgr2=0x7f67440fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:00.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.587+0000 7f674a645700 1 -- 172.21.15.121:0/1985664542 shutdown_connections 2024-04-05T21:45:00.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.588+0000 7f674a645700 1 -- 172.21.15.121:0/1985664542 wait complete. 2024-04-05T21:45:00.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.689+0000 7f21d080e700 1 -- 172.21.15.121:0/1424758497 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc0ffbd0 msgr2=0x7f21cc100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:00.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.689+0000 7f21d080e700 1 --2- 172.21.15.121:0/1424758497 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc0ffbd0 0x7f21cc100050 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f21c0009910 tx=0x7f21c0009c20 comp rx=0 tx=0).stop 2024-04-05T21:45:00.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.689+0000 7f21d080e700 1 -- 172.21.15.121:0/1424758497 shutdown_connections 2024-04-05T21:45:00.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.689+0000 7f21d080e700 1 --2- 172.21.15.121:0/1424758497 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc0ffbd0 0x7f21cc100050 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.689+0000 7f21d080e700 1 --2- 172.21.15.121:0/1424758497 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21cc105a40 0x7f21cc105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.689+0000 7f21d080e700 1 -- 172.21.15.121:0/1424758497 >> 172.21.15.121:0/1424758497 conn(0x7f21cc0755a0 msgr2=0x7f21cc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:00.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.689+0000 7f21d080e700 1 -- 172.21.15.121:0/1424758497 shutdown_connections 2024-04-05T21:45:00.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.689+0000 7f21d080e700 1 -- 172.21.15.121:0/1424758497 wait complete. 2024-04-05T21:45:00.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.690+0000 7f21d080e700 1 Processor -- start 2024-04-05T21:45:00.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.690+0000 7f21d080e700 1 -- start start 2024-04-05T21:45:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.690+0000 7f21d080e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21cc0ffbd0 0x7f21cc195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.690+0000 7f21d080e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc105a40 0x7f21cc195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.690+0000 7f21d080e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc196160 con 0x7f21cc105a40 2024-04-05T21:45:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.690+0000 7f21d080e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc199ef0 con 0x7f21cc0ffbd0 2024-04-05T21:45:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.691+0000 7f21c9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc105a40 0x7f21cc195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.691+0000 7f21c9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc105a40 0x7f21cc195a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44388/0 (socket says 172.21.15.121:44388) 2024-04-05T21:45:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.691+0000 7f21c9d9b700 1 -- 172.21.15.121:0/2359079538 learned_addr learned my addr 172.21.15.121:0/2359079538 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.691+0000 7f21ca59c700 1 --2- 172.21.15.121:0/2359079538 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21cc0ffbd0 0x7f21cc195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.691+0000 7f21c9d9b700 1 -- 172.21.15.121:0/2359079538 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21cc0ffbd0 msgr2=0x7f21cc195540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.691+0000 7f21c9d9b700 1 --2- 172.21.15.121:0/2359079538 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21cc0ffbd0 0x7f21cc195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.691+0000 7f21c9d9b700 1 -- 172.21.15.121:0/2359079538 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c00095f0 con 0x7f21cc105a40 2024-04-05T21:45:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.691+0000 7f21c9d9b700 1 --2- 172.21.15.121:0/2359079538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc105a40 0x7f21cc195a80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f21c0006010 tx=0x7f21c000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:00.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.691+0000 7f21bb7fe700 1 -- 172.21.15.121:0/2359079538 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21c001f070 con 0x7f21cc105a40 2024-04-05T21:45:00.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.691+0000 7f21bb7fe700 1 -- 172.21.15.121:0/2359079538 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21c0019610 con 0x7f21cc105a40 2024-04-05T21:45:00.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.691+0000 7f21d080e700 1 -- 172.21.15.121:0/2359079538 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc19a170 con 0x7f21cc105a40 2024-04-05T21:45:00.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.691+0000 7f21bb7fe700 1 -- 172.21.15.121:0/2359079538 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21c000f670 con 0x7f21cc105a40 2024-04-05T21:45:00.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.691+0000 7f21d080e700 1 -- 172.21.15.121:0/2359079538 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc19a660 con 0x7f21cc105a40 2024-04-05T21:45:00.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.692+0000 7f21d080e700 1 -- 172.21.15.121:0/2359079538 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21cc04ea90 con 0x7f21cc105a40 2024-04-05T21:45:00.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.693+0000 7f21bb7fe700 1 -- 172.21.15.121:0/2359079538 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f21c0019bf0 con 0x7f21cc105a40 2024-04-05T21:45:00.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.693+0000 7f21bb7fe700 1 --2- 172.21.15.121:0/2359079538 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f21b406cce0 0x7f21b406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:00.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.693+0000 7f21bb7fe700 1 -- 172.21.15.121:0/2359079538 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f21c008fbe0 con 0x7f21cc105a40 2024-04-05T21:45:00.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.693+0000 7f21ca59c700 1 --2- 172.21.15.121:0/2359079538 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f21b406cce0 0x7f21b406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:00.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.694+0000 7f21ca59c700 1 --2- 172.21.15.121:0/2359079538 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f21b406cce0 0x7f21b406f1a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f21bc005950 tx=0x7f21bc0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:00.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.697+0000 7f21bb7fe700 1 -- 172.21.15.121:0/2359079538 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f21c0059cd0 con 0x7f21cc105a40 2024-04-05T21:45:00.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:45:00.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.898+0000 7f21d080e700 1 -- 172.21.15.121:0/2359079538 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f21cc19ab70 con 0x7f21b406cce0 2024-04-05T21:45:00.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.909+0000 7f21bb7fe700 1 -- 172.21.15.121:0/2359079538 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f21cc19ab70 con 0x7f21b406cce0 2024-04-05T21:45:00.910 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:45:00.910 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (15m) 9m ago 27m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:45:00.910 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (27m) 9m ago 27m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:45:00.910 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (26m) 9m ago 26m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:45:00.910 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (14m) 9m ago 27m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:45:00.910 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (13m) 9m ago 26m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:45:00.910 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (15m) 9m ago 27m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:45:00.910 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (10m) 9m ago 21m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:45:00.910 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (9m) 9m ago 21m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:45:00.910 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (9m) 9m ago 21m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:45:00.910 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (21m) 9m ago 21m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:45:00.910 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (17m) 9m ago 28m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:45:00.910 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (16m) 9m ago 26m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:45:00.911 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (14m) 9m ago 29m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:45:00.911 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (14m) 9m ago 26m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:45:00.911 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (16m) 9m ago 27m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:45:00.911 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (16m) 9m ago 26m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:45:00.911 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (13m) 9m ago 25m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:45:00.911 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (12m) 9m ago 24m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:45:00.911 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (12m) 9m ago 24m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:45:00.911 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (11m) 9m ago 23m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:45:00.911 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (11m) 9m ago 23m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:45:00.911 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (10m) 9m ago 22m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:45:00.911 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (16m) 9m ago 26m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:45:00.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.911+0000 7f21d080e700 1 -- 172.21.15.121:0/2359079538 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f21b406cce0 msgr2=0x7f21b406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:00.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.911+0000 7f21d080e700 1 --2- 172.21.15.121:0/2359079538 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f21b406cce0 0x7f21b406f1a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f21bc005950 tx=0x7f21bc0058e0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.911+0000 7f21d080e700 1 -- 172.21.15.121:0/2359079538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc105a40 msgr2=0x7f21cc195a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:00.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.911+0000 7f21d080e700 1 --2- 172.21.15.121:0/2359079538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc105a40 0x7f21cc195a80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f21c0006010 tx=0x7f21c000be20 comp rx=0 tx=0).stop 2024-04-05T21:45:00.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.911+0000 7f21d080e700 1 -- 172.21.15.121:0/2359079538 shutdown_connections 2024-04-05T21:45:00.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.911+0000 7f21d080e700 1 --2- 172.21.15.121:0/2359079538 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f21b406cce0 0x7f21b406f1a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.911+0000 7f21d080e700 1 --2- 172.21.15.121:0/2359079538 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21cc0ffbd0 0x7f21cc195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.911+0000 7f21d080e700 1 --2- 172.21.15.121:0/2359079538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc105a40 0x7f21cc195a80 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:00.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.911+0000 7f21d080e700 1 -- 172.21.15.121:0/2359079538 >> 172.21.15.121:0/2359079538 conn(0x7f21cc0755a0 msgr2=0x7f21cc0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:00.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.911+0000 7f21d080e700 1 -- 172.21.15.121:0/2359079538 shutdown_connections 2024-04-05T21:45:00.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:00.911+0000 7f21d080e700 1 -- 172.21.15.121:0/2359079538 wait complete. 2024-04-05T21:45:00.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:45:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.012+0000 7f6a0604c700 1 -- 172.21.15.121:0/1810279017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a000ffa40 msgr2=0x7f6a000ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.012+0000 7f6a0604c700 1 --2- 172.21.15.121:0/1810279017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a000ffa40 0x7f6a000ffec0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f69e8009910 tx=0x7f69e8009c20 comp rx=0 tx=0).stop 2024-04-05T21:45:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.012+0000 7f6a0604c700 1 -- 172.21.15.121:0/1810279017 shutdown_connections 2024-04-05T21:45:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.012+0000 7f6a0604c700 1 --2- 172.21.15.121:0/1810279017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a000ffa40 0x7f6a000ffec0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.012+0000 7f6a0604c700 1 --2- 172.21.15.121:0/1810279017 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a00105a40 0x7f6a00105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.012+0000 7f6a0604c700 1 -- 172.21.15.121:0/1810279017 >> 172.21.15.121:0/1810279017 conn(0x7f6a000fb570 msgr2=0x7f6a000fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.012+0000 7f6a0604c700 1 -- 172.21.15.121:0/1810279017 shutdown_connections 2024-04-05T21:45:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.012+0000 7f6a0604c700 1 -- 172.21.15.121:0/1810279017 wait complete. 2024-04-05T21:45:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.013+0000 7f6a0604c700 1 Processor -- start 2024-04-05T21:45:01.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.013+0000 7f6a0604c700 1 -- start start 2024-04-05T21:45:01.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.013+0000 7f6a0604c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a000ffa40 0x7f6a001944a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:01.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.013+0000 7f6a0604c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a00105a40 0x7f6a001949e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:01.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.013+0000 7f6a0604c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a001950c0 con 0x7f6a000ffa40 2024-04-05T21:45:01.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.013+0000 7f6a0604c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a00198e50 con 0x7f6a00105a40 2024-04-05T21:45:01.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.013+0000 7f69ff7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a000ffa40 0x7f6a001944a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:01.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.013+0000 7f69ff7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a000ffa40 0x7f6a001944a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44402/0 (socket says 172.21.15.121:44402) 2024-04-05T21:45:01.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.013+0000 7f69ff7fe700 1 -- 172.21.15.121:0/2844811373 learned_addr learned my addr 172.21.15.121:0/2844811373 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:01.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.013+0000 7f69feffd700 1 --2- 172.21.15.121:0/2844811373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a00105a40 0x7f6a001949e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:01.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.014+0000 7f69ff7fe700 1 -- 172.21.15.121:0/2844811373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a00105a40 msgr2=0x7f6a001949e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:01.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.014+0000 7f69ff7fe700 1 --2- 172.21.15.121:0/2844811373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a00105a40 0x7f6a001949e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.014+0000 7f69ff7fe700 1 -- 172.21.15.121:0/2844811373 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e80095f0 con 0x7f6a000ffa40 2024-04-05T21:45:01.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.014+0000 7f69ff7fe700 1 --2- 172.21.15.121:0/2844811373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a000ffa40 0x7f6a001944a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009fd0 tx=0x7f69f000e5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:01.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.014+0000 7f69fcff9700 1 -- 172.21.15.121:0/2844811373 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69f0004bb0 con 0x7f6a000ffa40 2024-04-05T21:45:01.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.014+0000 7f69fcff9700 1 -- 172.21.15.121:0/2844811373 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69f0004d10 con 0x7f6a000ffa40 2024-04-05T21:45:01.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.014+0000 7f6a0604c700 1 -- 172.21.15.121:0/2844811373 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a00199130 con 0x7f6a000ffa40 2024-04-05T21:45:01.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.014+0000 7f69fcff9700 1 -- 172.21.15.121:0/2844811373 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69f0005270 con 0x7f6a000ffa40 2024-04-05T21:45:01.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.014+0000 7f6a0604c700 1 -- 172.21.15.121:0/2844811373 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a00199680 con 0x7f6a000ffa40 2024-04-05T21:45:01.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.015+0000 7f6a0604c700 1 -- 172.21.15.121:0/2844811373 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a00107fd0 con 0x7f6a000ffa40 2024-04-05T21:45:01.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.016+0000 7f69fcff9700 1 -- 172.21.15.121:0/2844811373 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f69f0011450 con 0x7f6a000ffa40 2024-04-05T21:45:01.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.016+0000 7f69fcff9700 1 --2- 172.21.15.121:0/2844811373 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f69ec06cce0 0x7f69ec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:01.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.016+0000 7f69fcff9700 1 -- 172.21.15.121:0/2844811373 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f69f008c870 con 0x7f6a000ffa40 2024-04-05T21:45:01.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.016+0000 7f69feffd700 1 --2- 172.21.15.121:0/2844811373 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f69ec06cce0 0x7f69ec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:01.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.016+0000 7f69feffd700 1 --2- 172.21.15.121:0/2844811373 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f69ec06cce0 0x7f69ec06f1a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6a00195ac0 tx=0x7f69e8012040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:01.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.020+0000 7f69fcff9700 1 -- 172.21.15.121:0/2844811373 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f69f00568e0 con 0x7f6a000ffa40 2024-04-05T21:45:01.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.294+0000 7f6a0604c700 1 -- 172.21.15.121:0/2844811373 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6a00199940 con 0x7f6a000ffa40 2024-04-05T21:45:01.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.294+0000 7f69fcff9700 1 -- 172.21.15.121:0/2844811373 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f69f0004e80 con 0x7f6a000ffa40 2024-04-05T21:45:01.295 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:45:01.295 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:45:01.295 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:45:01.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:45:01.295 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:45:01.295 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:45:01.295 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:45:01.296 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:45:01.296 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:45:01.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:45:01.296 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:45:01.296 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:45:01.296 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:45:01.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:45:01.296 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:45:01.296 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:45:01.296 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:45:01.296 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:45:01.296 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:45:01.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.296+0000 7f6a0604c700 1 -- 172.21.15.121:0/2844811373 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f69ec06cce0 msgr2=0x7f69ec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:01.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.296+0000 7f6a0604c700 1 --2- 172.21.15.121:0/2844811373 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f69ec06cce0 0x7f69ec06f1a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6a00195ac0 tx=0x7f69e8012040 comp rx=0 tx=0).stop 2024-04-05T21:45:01.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.296+0000 7f6a0604c700 1 -- 172.21.15.121:0/2844811373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a000ffa40 msgr2=0x7f6a001944a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:01.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.296+0000 7f6a0604c700 1 --2- 172.21.15.121:0/2844811373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a000ffa40 0x7f6a001944a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009fd0 tx=0x7f69f000e5b0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.296+0000 7f6a0604c700 1 -- 172.21.15.121:0/2844811373 shutdown_connections 2024-04-05T21:45:01.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.296+0000 7f6a0604c700 1 --2- 172.21.15.121:0/2844811373 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f69ec06cce0 0x7f69ec06f1a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.296+0000 7f6a0604c700 1 --2- 172.21.15.121:0/2844811373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a000ffa40 0x7f6a001944a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.296+0000 7f6a0604c700 1 --2- 172.21.15.121:0/2844811373 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6a00105a40 0x7f6a001949e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.296+0000 7f6a0604c700 1 -- 172.21.15.121:0/2844811373 >> 172.21.15.121:0/2844811373 conn(0x7f6a000fb570 msgr2=0x7f6a000fc260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:01.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.297+0000 7f6a0604c700 1 -- 172.21.15.121:0/2844811373 shutdown_connections 2024-04-05T21:45:01.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.297+0000 7f6a0604c700 1 -- 172.21.15.121:0/2844811373 wait complete. 2024-04-05T21:45:01.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.399+0000 7fbea9d6c700 1 -- 172.21.15.121:0/2831362481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea40ffc40 msgr2=0x7fbea4105780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:01.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.399+0000 7fbea9d6c700 1 --2- 172.21.15.121:0/2831362481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea40ffc40 0x7fbea4105780 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fbe94009910 tx=0x7fbe94009c20 comp rx=0 tx=0).stop 2024-04-05T21:45:01.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.399+0000 7fbea9d6c700 1 -- 172.21.15.121:0/2831362481 shutdown_connections 2024-04-05T21:45:01.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.399+0000 7fbea9d6c700 1 --2- 172.21.15.121:0/2831362481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea40ffc40 0x7fbea4105780 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.399+0000 7fbea9d6c700 1 --2- 172.21.15.121:0/2831362481 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbea40ff290 0x7fbea40ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.399+0000 7fbea9d6c700 1 -- 172.21.15.121:0/2831362481 >> 172.21.15.121:0/2831362481 conn(0x7fbea4075e30 msgr2=0x7fbea40fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:01.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.399+0000 7fbea9d6c700 1 -- 172.21.15.121:0/2831362481 shutdown_connections 2024-04-05T21:45:01.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.399+0000 7fbea9d6c700 1 -- 172.21.15.121:0/2831362481 wait complete. 2024-04-05T21:45:01.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.399+0000 7fbea9d6c700 1 Processor -- start 2024-04-05T21:45:01.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea9d6c700 1 -- start start 2024-04-05T21:45:01.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea9d6c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea40ff290 0x7fbea4196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:01.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea9d6c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbea40ffc40 0x7fbea4196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:01.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea9d6c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea4197270 con 0x7fbea40ff290 2024-04-05T21:45:01.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea9d6c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea419b000 con 0x7fbea40ffc40 2024-04-05T21:45:01.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea40ff290 0x7fbea4196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:01.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea40ff290 0x7fbea4196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44418/0 (socket says 172.21.15.121:44418) 2024-04-05T21:45:01.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea37fe700 1 -- 172.21.15.121:0/1109247794 learned_addr learned my addr 172.21.15.121:0/1109247794 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:01.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea2ffd700 1 --2- 172.21.15.121:0/1109247794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbea40ffc40 0x7fbea4196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:01.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea37fe700 1 -- 172.21.15.121:0/1109247794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbea40ffc40 msgr2=0x7fbea4196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:01.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea37fe700 1 --2- 172.21.15.121:0/1109247794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbea40ffc40 0x7fbea4196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea37fe700 1 -- 172.21.15.121:0/1109247794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe940095f0 con 0x7fbea40ff290 2024-04-05T21:45:01.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea2ffd700 1 --2- 172.21.15.121:0/1109247794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbea40ffc40 0x7fbea4196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:45:01.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.400+0000 7fbea37fe700 1 --2- 172.21.15.121:0/1109247794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea40ff290 0x7fbea4196650 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00c8a0 tx=0x7fbe8c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:01.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.401+0000 7fbea0ff9700 1 -- 172.21.15.121:0/1109247794 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe8c00cea0 con 0x7fbea40ff290 2024-04-05T21:45:01.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.401+0000 7fbea0ff9700 1 -- 172.21.15.121:0/1109247794 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe8c004830 con 0x7fbea40ff290 2024-04-05T21:45:01.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.401+0000 7fbea0ff9700 1 -- 172.21.15.121:0/1109247794 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe8c005500 con 0x7fbea40ff290 2024-04-05T21:45:01.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.401+0000 7fbea9d6c700 1 -- 172.21.15.121:0/1109247794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea419b2e0 con 0x7fbea40ff290 2024-04-05T21:45:01.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.401+0000 7fbea9d6c700 1 -- 172.21.15.121:0/1109247794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea419b830 con 0x7fbea40ff290 2024-04-05T21:45:01.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.402+0000 7fbea9d6c700 1 -- 172.21.15.121:0/1109247794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea4109180 con 0x7fbea40ff290 2024-04-05T21:45:01.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.402+0000 7fbea0ff9700 1 -- 172.21.15.121:0/1109247794 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbe8c0049a0 con 0x7fbea40ff290 2024-04-05T21:45:01.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.402+0000 7fbea0ff9700 1 --2- 172.21.15.121:0/1109247794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbe9006cce0 0x7fbe9006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:01.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.403+0000 7fbea0ff9700 1 -- 172.21.15.121:0/1109247794 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbe8c08c1d0 con 0x7fbea40ff290 2024-04-05T21:45:01.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.403+0000 7fbea2ffd700 1 --2- 172.21.15.121:0/1109247794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbe9006cce0 0x7fbe9006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:01.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.403+0000 7fbea2ffd700 1 --2- 172.21.15.121:0/1109247794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbe9006cce0 0x7fbe9006f1a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbea4197c70 tx=0x7fbe9401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:01.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.407+0000 7fbea0ff9700 1 -- 172.21.15.121:0/1109247794 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbe8c0562c0 con 0x7fbea40ff290 2024-04-05T21:45:01.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.644+0000 7fbea9d6c700 1 -- 172.21.15.121:0/1109247794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbea419bbf0 con 0x7fbea40ff290 2024-04-05T21:45:01.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.644+0000 7fbea0ff9700 1 -- 172.21.15.121:0/1109247794 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fbe8c01d750 con 0x7fbea40ff290 2024-04-05T21:45:01.645 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:45:01.645 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:45:01.645 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:45:01.645 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:45:01.645 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:45:01.645 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:45:01.645 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:45:01.645 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:45:01.645 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:45:01.646 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:45:01.647 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:45:01.648 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:45:01.648 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:45:01.648 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:45:01.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.646+0000 7fbea9d6c700 1 -- 172.21.15.121:0/1109247794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbe9006cce0 msgr2=0x7fbe9006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:01.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.646+0000 7fbea9d6c700 1 --2- 172.21.15.121:0/1109247794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbe9006cce0 0x7fbe9006f1a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbea4197c70 tx=0x7fbe9401a040 comp rx=0 tx=0).stop 2024-04-05T21:45:01.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.646+0000 7fbea9d6c700 1 -- 172.21.15.121:0/1109247794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea40ff290 msgr2=0x7fbea4196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:01.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.646+0000 7fbea9d6c700 1 --2- 172.21.15.121:0/1109247794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea40ff290 0x7fbea4196650 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00c8a0 tx=0x7fbe8c00cc60 comp rx=0 tx=0).stop 2024-04-05T21:45:01.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.646+0000 7fbea9d6c700 1 -- 172.21.15.121:0/1109247794 shutdown_connections 2024-04-05T21:45:01.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.646+0000 7fbea9d6c700 1 --2- 172.21.15.121:0/1109247794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbe9006cce0 0x7fbe9006f1a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.646+0000 7fbea9d6c700 1 --2- 172.21.15.121:0/1109247794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbea40ff290 0x7fbea4196650 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.646+0000 7fbea9d6c700 1 --2- 172.21.15.121:0/1109247794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbea40ffc40 0x7fbea4196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.646+0000 7fbea9d6c700 1 -- 172.21.15.121:0/1109247794 >> 172.21.15.121:0/1109247794 conn(0x7fbea4075e30 msgr2=0x7fbea4104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:01.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.646+0000 7fbea9d6c700 1 -- 172.21.15.121:0/1109247794 shutdown_connections 2024-04-05T21:45:01.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.646+0000 7fbea9d6c700 1 -- 172.21.15.121:0/1109247794 wait complete. 2024-04-05T21:45:01.649 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:45:01.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.750+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/932723752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a28105a50 msgr2=0x7f8a28105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:01.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.750+0000 7f8a2d8b0700 1 --2- 172.21.15.121:0/932723752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a28105a50 0x7f8a28105e30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009910 tx=0x7f8a1c009c20 comp rx=0 tx=0).stop 2024-04-05T21:45:01.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.750+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/932723752 shutdown_connections 2024-04-05T21:45:01.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.750+0000 7f8a2d8b0700 1 --2- 172.21.15.121:0/932723752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a280ffa50 0x7f8a280ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.750+0000 7f8a2d8b0700 1 --2- 172.21.15.121:0/932723752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a28105a50 0x7f8a28105e30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.750+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/932723752 >> 172.21.15.121:0/932723752 conn(0x7f8a280fb540 msgr2=0x7f8a280fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:01.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.750+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/932723752 shutdown_connections 2024-04-05T21:45:01.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.750+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/932723752 wait complete. 2024-04-05T21:45:01.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.750+0000 7f8a2d8b0700 1 Processor -- start 2024-04-05T21:45:01.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.750+0000 7f8a2d8b0700 1 -- start start 2024-04-05T21:45:01.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a2d8b0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a280ffa50 0x7f8a28197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:01.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a2d8b0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a28105a50 0x7f8a28197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:01.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a2d8b0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a28198060 con 0x7f8a280ffa50 2024-04-05T21:45:01.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a2d8b0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a281981a0 con 0x7f8a28105a50 2024-04-05T21:45:01.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a26ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a280ffa50 0x7f8a28197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:01.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a26ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a280ffa50 0x7f8a28197500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44440/0 (socket says 172.21.15.121:44440) 2024-04-05T21:45:01.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a26ffd700 1 -- 172.21.15.121:0/1321321638 learned_addr learned my addr 172.21.15.121:0/1321321638 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a267fc700 1 --2- 172.21.15.121:0/1321321638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a28105a50 0x7f8a28197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a26ffd700 1 -- 172.21.15.121:0/1321321638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a28105a50 msgr2=0x7f8a28197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a26ffd700 1 --2- 172.21.15.121:0/1321321638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a28105a50 0x7f8a28197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a26ffd700 1 -- 172.21.15.121:0/1321321638 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c0095f0 con 0x7f8a280ffa50 2024-04-05T21:45:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a267fc700 1 --2- 172.21.15.121:0/1321321638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a28105a50 0x7f8a28197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:45:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a26ffd700 1 --2- 172.21.15.121:0/1321321638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a280ffa50 0x7f8a28197500 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c006010 tx=0x7f8a1c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a2c8ae700 1 -- 172.21.15.121:0/1321321638 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a1c01d070 con 0x7f8a280ffa50 2024-04-05T21:45:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/1321321638 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a281a0570 con 0x7f8a280ffa50 2024-04-05T21:45:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a2c8ae700 1 -- 172.21.15.121:0/1321321638 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a1c00f460 con 0x7f8a280ffa50 2024-04-05T21:45:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a2c8ae700 1 -- 172.21.15.121:0/1321321638 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a1c021640 con 0x7f8a280ffa50 2024-04-05T21:45:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.751+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/1321321638 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a281a0980 con 0x7f8a280ffa50 2024-04-05T21:45:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.752+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/1321321638 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a28108050 con 0x7f8a280ffa50 2024-04-05T21:45:01.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.753+0000 7f8a2c8ae700 1 -- 172.21.15.121:0/1321321638 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8a1c00fa60 con 0x7f8a280ffa50 2024-04-05T21:45:01.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.753+0000 7f8a2c8ae700 1 --2- 172.21.15.121:0/1321321638 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8a1006cce0 0x7f8a1006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:01.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.753+0000 7f8a2c8ae700 1 -- 172.21.15.121:0/1321321638 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8a1c08d880 con 0x7f8a280ffa50 2024-04-05T21:45:01.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.753+0000 7f8a267fc700 1 --2- 172.21.15.121:0/1321321638 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8a1006cce0 0x7f8a1006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:01.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.754+0000 7f8a267fc700 1 --2- 172.21.15.121:0/1321321638 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8a1006cce0 0x7f8a1006f1a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8a280696a0 tx=0x7f8a18005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:01.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.758+0000 7f8a2c8ae700 1 -- 172.21.15.121:0/1321321638 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8a1c0578f0 con 0x7f8a280ffa50 2024-04-05T21:45:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:01 smithi188 ceph-mon[107793]: from='client.34874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:01 smithi188 ceph-mon[107793]: pgmap v563: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:01 smithi188 ceph-mon[107793]: from='client.34878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:01.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:01 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2844811373' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:45:01.817 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:01 smithi121 ceph-mon[145859]: from='client.34874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:01.817 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:01 smithi121 ceph-mon[145859]: pgmap v563: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:01.817 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:01 smithi121 ceph-mon[145859]: from='client.34878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:01.818 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:01 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2844811373' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:45:01.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.971+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/1321321638 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a2802cd30 con 0x7f8a1006cce0 2024-04-05T21:45:01.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.972+0000 7f8a2c8ae700 1 -- 172.21.15.121:0/1321321638 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f8a2802cd30 con 0x7f8a1006cce0 2024-04-05T21:45:01.973 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:45:01.973 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:45:01.973 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:45:01.973 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:45:01.973 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:45:01.973 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:45:01.973 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:45:01.973 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:45:01.973 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:45:01.973 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:45:01.973 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:45:01.973 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:45:01.973 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:45:01.974 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:45:01.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.974+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/1321321638 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8a1006cce0 msgr2=0x7f8a1006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:01.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.974+0000 7f8a2d8b0700 1 --2- 172.21.15.121:0/1321321638 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8a1006cce0 0x7f8a1006f1a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8a280696a0 tx=0x7f8a18005f50 comp rx=0 tx=0).stop 2024-04-05T21:45:01.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.974+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/1321321638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a280ffa50 msgr2=0x7f8a28197500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:01.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.974+0000 7f8a2d8b0700 1 --2- 172.21.15.121:0/1321321638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a280ffa50 0x7f8a28197500 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c006010 tx=0x7f8a1c00be20 comp rx=0 tx=0).stop 2024-04-05T21:45:01.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.974+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/1321321638 shutdown_connections 2024-04-05T21:45:01.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.974+0000 7f8a2d8b0700 1 --2- 172.21.15.121:0/1321321638 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8a1006cce0 0x7f8a1006f1a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.974+0000 7f8a2d8b0700 1 --2- 172.21.15.121:0/1321321638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a280ffa50 0x7f8a28197500 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.974+0000 7f8a2d8b0700 1 --2- 172.21.15.121:0/1321321638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8a28105a50 0x7f8a28197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:01.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.974+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/1321321638 >> 172.21.15.121:0/1321321638 conn(0x7f8a280fb540 msgr2=0x7f8a280fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:01.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.974+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/1321321638 shutdown_connections 2024-04-05T21:45:01.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:01.975+0000 7f8a2d8b0700 1 -- 172.21.15.121:0/1321321638 wait complete. 2024-04-05T21:45:02.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.076+0000 7f1aa96c8700 1 -- 172.21.15.121:0/2549823818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa40ffa40 msgr2=0x7f1aa40ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:02.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.076+0000 7f1aa96c8700 1 --2- 172.21.15.121:0/2549823818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa40ffa40 0x7f1aa40ffec0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009910 tx=0x7f1a94009c20 comp rx=0 tx=0).stop 2024-04-05T21:45:02.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.076+0000 7f1aa96c8700 1 -- 172.21.15.121:0/2549823818 shutdown_connections 2024-04-05T21:45:02.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.076+0000 7f1aa96c8700 1 --2- 172.21.15.121:0/2549823818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa40ffa40 0x7f1aa40ffec0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:02.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.076+0000 7f1aa96c8700 1 --2- 172.21.15.121:0/2549823818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aa4105a40 0x7f1aa4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:02.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.076+0000 7f1aa96c8700 1 -- 172.21.15.121:0/2549823818 >> 172.21.15.121:0/2549823818 conn(0x7f1aa40fb550 msgr2=0x7f1aa40fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:02.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.076+0000 7f1aa96c8700 1 -- 172.21.15.121:0/2549823818 shutdown_connections 2024-04-05T21:45:02.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.076+0000 7f1aa96c8700 1 -- 172.21.15.121:0/2549823818 wait complete. 2024-04-05T21:45:02.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.077+0000 7f1aa96c8700 1 Processor -- start 2024-04-05T21:45:02.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.077+0000 7f1aa96c8700 1 -- start start 2024-04-05T21:45:02.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.077+0000 7f1aa96c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa40ffa40 0x7f1aa4197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:02.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.077+0000 7f1aa96c8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aa4105a40 0x7f1aa4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.077+0000 7f1aa96c8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa4198140 con 0x7f1aa40ffa40 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.077+0000 7f1aa96c8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa41982b0 con 0x7f1aa4105a40 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.077+0000 7f1aa2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa40ffa40 0x7f1aa4197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.077+0000 7f1aa2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa40ffa40 0x7f1aa4197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44458/0 (socket says 172.21.15.121:44458) 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.077+0000 7f1aa2ffd700 1 -- 172.21.15.121:0/587645586 learned_addr learned my addr 172.21.15.121:0/587645586 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.077+0000 7f1aa27fc700 1 --2- 172.21.15.121:0/587645586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aa4105a40 0x7f1aa4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.078+0000 7f1aa2ffd700 1 -- 172.21.15.121:0/587645586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aa4105a40 msgr2=0x7f1aa4197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.078+0000 7f1aa2ffd700 1 --2- 172.21.15.121:0/587645586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aa4105a40 0x7f1aa4197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.078+0000 7f1aa2ffd700 1 -- 172.21.15.121:0/587645586 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a940095f0 con 0x7f1aa40ffa40 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.078+0000 7f1aa2ffd700 1 --2- 172.21.15.121:0/587645586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa40ffa40 0x7f1aa4197520 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00e940 tx=0x7f1a8c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.078+0000 7f1a9bfff700 1 -- 172.21.15.121:0/587645586 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a8c00cba0 con 0x7f1aa40ffa40 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.078+0000 7f1a9bfff700 1 -- 172.21.15.121:0/587645586 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a8c00ee90 con 0x7f1aa40ffa40 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.078+0000 7f1aa96c8700 1 -- 172.21.15.121:0/587645586 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa41a0590 con 0x7f1aa40ffa40 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.078+0000 7f1a9bfff700 1 -- 172.21.15.121:0/587645586 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a8c0055f0 con 0x7f1aa40ffa40 2024-04-05T21:45:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.078+0000 7f1aa96c8700 1 -- 172.21.15.121:0/587645586 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa41a0ab0 con 0x7f1aa40ffa40 2024-04-05T21:45:02.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.079+0000 7f1aa96c8700 1 -- 172.21.15.121:0/587645586 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa4107fd0 con 0x7f1aa40ffa40 2024-04-05T21:45:02.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.080+0000 7f1a9bfff700 1 -- 172.21.15.121:0/587645586 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1a8c004750 con 0x7f1aa40ffa40 2024-04-05T21:45:02.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.080+0000 7f1a9bfff700 1 --2- 172.21.15.121:0/587645586 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a9006cce0 0x7f1a9006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:02.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.080+0000 7f1a9bfff700 1 -- 172.21.15.121:0/587645586 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1a8c014070 con 0x7f1aa40ffa40 2024-04-05T21:45:02.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.080+0000 7f1aa27fc700 1 --2- 172.21.15.121:0/587645586 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a9006cce0 0x7f1a9006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:02.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.081+0000 7f1aa27fc700 1 --2- 172.21.15.121:0/587645586 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a9006cce0 0x7f1a9006f1a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40699e0 tx=0x7f1a94005940 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:02.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.084+0000 7f1a9bfff700 1 -- 172.21.15.121:0/587645586 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1a8c056b30 con 0x7f1aa40ffa40 2024-04-05T21:45:02.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.358+0000 7f1aa96c8700 1 -- 172.21.15.121:0/587645586 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1aa404ea90 con 0x7f1aa40ffa40 2024-04-05T21:45:02.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.359+0000 7f1a9bfff700 1 -- 172.21.15.121:0/587645586 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f1a8c05a150 con 0x7f1aa40ffa40 2024-04-05T21:45:02.360 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:45:02.360 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:45:02.361 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:45:02.361 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:45:02.361 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:45:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.361+0000 7f1aa96c8700 1 -- 172.21.15.121:0/587645586 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a9006cce0 msgr2=0x7f1a9006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.361+0000 7f1aa96c8700 1 --2- 172.21.15.121:0/587645586 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a9006cce0 0x7f1a9006f1a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40699e0 tx=0x7f1a94005940 comp rx=0 tx=0).stop 2024-04-05T21:45:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.361+0000 7f1aa96c8700 1 -- 172.21.15.121:0/587645586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa40ffa40 msgr2=0x7f1aa4197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.361+0000 7f1aa96c8700 1 --2- 172.21.15.121:0/587645586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa40ffa40 0x7f1aa4197520 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c00e940 tx=0x7f1a8c00ec50 comp rx=0 tx=0).stop 2024-04-05T21:45:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.361+0000 7f1aa96c8700 1 -- 172.21.15.121:0/587645586 shutdown_connections 2024-04-05T21:45:02.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.361+0000 7f1aa96c8700 1 --2- 172.21.15.121:0/587645586 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a9006cce0 0x7f1a9006f1a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:02.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.361+0000 7f1aa96c8700 1 --2- 172.21.15.121:0/587645586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa40ffa40 0x7f1aa4197520 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:02.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.361+0000 7f1aa96c8700 1 --2- 172.21.15.121:0/587645586 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1aa4105a40 0x7f1aa4197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:02.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.361+0000 7f1aa96c8700 1 -- 172.21.15.121:0/587645586 >> 172.21.15.121:0/587645586 conn(0x7f1aa40fb550 msgr2=0x7f1aa40fc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:02.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.361+0000 7f1aa96c8700 1 -- 172.21.15.121:0/587645586 shutdown_connections 2024-04-05T21:45:02.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:02.361+0000 7f1aa96c8700 1 -- 172.21.15.121:0/587645586 wait complete. 2024-04-05T21:45:02.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:02 smithi188 ceph-mon[107793]: from='client.34882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:02.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:02 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1109247794' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:45:02.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:02 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/587645586' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:45:02.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:02 smithi121 ceph-mon[145859]: from='client.34882 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:02.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:02 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1109247794' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:45:02.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:02 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/587645586' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:45:03.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:03 smithi188 ceph-mon[107793]: from='client.34894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:03.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:03 smithi188 ceph-mon[107793]: pgmap v564: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:03.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:03 smithi121 ceph-mon[145859]: from='client.34894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:03 smithi121 ceph-mon[145859]: pgmap v564: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:05 smithi188 ceph-mon[107793]: pgmap v565: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:05.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:05 smithi121 ceph-mon[145859]: pgmap v565: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:07 smithi188 ceph-mon[107793]: pgmap v566: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:07.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:07 smithi121 ceph-mon[145859]: pgmap v566: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:09 smithi188 ceph-mon[107793]: pgmap v567: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:09.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:09 smithi121 ceph-mon[145859]: pgmap v567: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:11.908 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:11 smithi121 ceph-mon[145859]: pgmap v568: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:12.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:11 smithi188 ceph-mon[107793]: pgmap v568: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:13.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:13 smithi121 ceph-mon[145859]: pgmap v569: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:14.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:13 smithi188 ceph-mon[107793]: pgmap v569: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:15 smithi188 ceph-mon[107793]: pgmap v570: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:45:15.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:15 smithi121 ceph-mon[145859]: pgmap v570: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:15.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:45:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:17 smithi121 ceph-mon[145859]: pgmap v571: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:18.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:17 smithi188 ceph-mon[107793]: pgmap v571: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:19.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:19 smithi121 ceph-mon[145859]: pgmap v572: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:20.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:19 smithi188 ceph-mon[107793]: pgmap v572: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:21.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:21 smithi121 ceph-mon[145859]: pgmap v573: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:22.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:21 smithi188 ceph-mon[107793]: pgmap v573: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:23 smithi121 ceph-mon[145859]: pgmap v574: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:24.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:23 smithi188 ceph-mon[107793]: pgmap v574: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:25.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:25 smithi188 ceph-mon[107793]: pgmap v575: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:25.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:25 smithi121 ceph-mon[145859]: pgmap v575: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:27 smithi121 ceph-mon[145859]: pgmap v576: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:28.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:27 smithi188 ceph-mon[107793]: pgmap v576: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:29.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:29 smithi121 ceph-mon[145859]: pgmap v577: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:30.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:29 smithi188 ceph-mon[107793]: pgmap v577: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:30.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:45:30.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:45:31.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:31 smithi121 ceph-mon[145859]: pgmap v578: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:32.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:31 smithi188 ceph-mon[107793]: pgmap v578: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:32.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.477+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/1199388423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8f8105a40 msgr2=0x7ff8f8105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:32.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.477+0000 7ff8ff8b4700 1 --2- 172.21.15.121:0/1199388423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8f8105a40 0x7ff8f8105e20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4009910 tx=0x7ff8f4009c20 comp rx=0 tx=0).stop 2024-04-05T21:45:32.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.477+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/1199388423 shutdown_connections 2024-04-05T21:45:32.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.477+0000 7ff8ff8b4700 1 --2- 172.21.15.121:0/1199388423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8f80ffbd0 0x7ff8f8100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:32.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.477+0000 7ff8ff8b4700 1 --2- 172.21.15.121:0/1199388423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8f8105a40 0x7ff8f8105e20 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:32.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.477+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/1199388423 >> 172.21.15.121:0/1199388423 conn(0x7ff8f80755a0 msgr2=0x7ff8f80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:32.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.477+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/1199388423 shutdown_connections 2024-04-05T21:45:32.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.477+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/1199388423 wait complete. 2024-04-05T21:45:32.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.477+0000 7ff8ff8b4700 1 Processor -- start 2024-04-05T21:45:32.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.477+0000 7ff8ff8b4700 1 -- start start 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8ff8b4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8f80ffbd0 0x7ff8f8196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8ff8b4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8f8105a40 0x7ff8f8196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8ff8b4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f81971d0 con 0x7ff8f80ffbd0 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8ff8b4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8197310 con 0x7ff8f8105a40 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8fd650700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8f80ffbd0 0x7ff8f8196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8fd650700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8f80ffbd0 0x7ff8f8196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45932/0 (socket says 172.21.15.121:45932) 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8fd650700 1 -- 172.21.15.121:0/2096969646 learned_addr learned my addr 172.21.15.121:0/2096969646 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8fce4f700 1 --2- 172.21.15.121:0/2096969646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8f8105a40 0x7ff8f8196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8fd650700 1 -- 172.21.15.121:0/2096969646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8f8105a40 msgr2=0x7ff8f8196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8fd650700 1 --2- 172.21.15.121:0/2096969646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8f8105a40 0x7ff8f8196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8fd650700 1 -- 172.21.15.121:0/2096969646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f40095f0 con 0x7ff8f80ffbd0 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8fd650700 1 --2- 172.21.15.121:0/2096969646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8f80ffbd0 0x7ff8f8196640 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff8f400b5c0 tx=0x7ff8f400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8ea7fc700 1 -- 172.21.15.121:0/2096969646 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8f401d070 con 0x7ff8f80ffbd0 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8ea7fc700 1 -- 172.21.15.121:0/2096969646 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8f400f460 con 0x7ff8f80ffbd0 2024-04-05T21:45:32.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/2096969646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f819b100 con 0x7ff8f80ffbd0 2024-04-05T21:45:32.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8ea7fc700 1 -- 172.21.15.121:0/2096969646 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8f4021640 con 0x7ff8f80ffbd0 2024-04-05T21:45:32.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.478+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/2096969646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f819b5f0 con 0x7ff8f80ffbd0 2024-04-05T21:45:32.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.479+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/2096969646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f804ea90 con 0x7ff8f80ffbd0 2024-04-05T21:45:32.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.480+0000 7ff8ea7fc700 1 -- 172.21.15.121:0/2096969646 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff8f400f9b0 con 0x7ff8f80ffbd0 2024-04-05T21:45:32.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.480+0000 7ff8ea7fc700 1 --2- 172.21.15.121:0/2096969646 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff8e406cce0 0x7ff8e406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:32.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.480+0000 7ff8ea7fc700 1 -- 172.21.15.121:0/2096969646 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff8f408d8d0 con 0x7ff8f80ffbd0 2024-04-05T21:45:32.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.480+0000 7ff8fce4f700 1 --2- 172.21.15.121:0/2096969646 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff8e406cce0 0x7ff8e406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:32.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.481+0000 7ff8fce4f700 1 --2- 172.21.15.121:0/2096969646 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff8e406cce0 0x7ff8e406f1a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec005fd0 tx=0x7ff8ec005e30 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:32.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.485+0000 7ff8ea7fc700 1 -- 172.21.15.121:0/2096969646 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff8f4057940 con 0x7ff8f80ffbd0 2024-04-05T21:45:32.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.698+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/2096969646 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8f8108ce0 con 0x7ff8e406cce0 2024-04-05T21:45:32.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.699+0000 7ff8ea7fc700 1 -- 172.21.15.121:0/2096969646 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff8f8108ce0 con 0x7ff8e406cce0 2024-04-05T21:45:32.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.701+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/2096969646 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff8e406cce0 msgr2=0x7ff8e406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:32.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.701+0000 7ff8ff8b4700 1 --2- 172.21.15.121:0/2096969646 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff8e406cce0 0x7ff8e406f1a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec005fd0 tx=0x7ff8ec005e30 comp rx=0 tx=0).stop 2024-04-05T21:45:32.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.701+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/2096969646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8f80ffbd0 msgr2=0x7ff8f8196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:32.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.701+0000 7ff8ff8b4700 1 --2- 172.21.15.121:0/2096969646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8f80ffbd0 0x7ff8f8196640 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff8f400b5c0 tx=0x7ff8f400be20 comp rx=0 tx=0).stop 2024-04-05T21:45:32.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.701+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/2096969646 shutdown_connections 2024-04-05T21:45:32.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.701+0000 7ff8ff8b4700 1 --2- 172.21.15.121:0/2096969646 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff8e406cce0 0x7ff8e406f1a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:32.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.701+0000 7ff8ff8b4700 1 --2- 172.21.15.121:0/2096969646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8f80ffbd0 0x7ff8f8196640 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:32.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.701+0000 7ff8ff8b4700 1 --2- 172.21.15.121:0/2096969646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff8f8105a40 0x7ff8f8196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:32.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.701+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/2096969646 >> 172.21.15.121:0/2096969646 conn(0x7ff8f80755a0 msgr2=0x7ff8f8068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:32.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.702+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/2096969646 shutdown_connections 2024-04-05T21:45:32.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.702+0000 7ff8ff8b4700 1 -- 172.21.15.121:0/2096969646 wait complete. 2024-04-05T21:45:32.712 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:45:32.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.804+0000 7f0c517bb700 1 -- 172.21.15.121:0/2851435598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c4c0ffeb0 msgr2=0x7f0c4c105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:32.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.804+0000 7f0c517bb700 1 --2- 172.21.15.121:0/2851435598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c4c0ffeb0 0x7f0c4c105840 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c009910 tx=0x7f0c3c009c20 comp rx=0 tx=0).stop 2024-04-05T21:45:32.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.804+0000 7f0c517bb700 1 -- 172.21.15.121:0/2851435598 shutdown_connections 2024-04-05T21:45:32.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.804+0000 7f0c517bb700 1 --2- 172.21.15.121:0/2851435598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c4c0ffeb0 0x7f0c4c105840 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:32.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.804+0000 7f0c517bb700 1 --2- 172.21.15.121:0/2851435598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c4c0ff500 0x7f0c4c0ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:32.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.804+0000 7f0c517bb700 1 -- 172.21.15.121:0/2851435598 >> 172.21.15.121:0/2851435598 conn(0x7f0c4c074d80 msgr2=0x7f0c4c075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:32.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.805+0000 7f0c517bb700 1 -- 172.21.15.121:0/2851435598 shutdown_connections 2024-04-05T21:45:32.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.805+0000 7f0c517bb700 1 -- 172.21.15.121:0/2851435598 wait complete. 2024-04-05T21:45:32.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.805+0000 7f0c517bb700 1 Processor -- start 2024-04-05T21:45:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.805+0000 7f0c517bb700 1 -- start start 2024-04-05T21:45:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.805+0000 7f0c517bb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c4c0ff500 0x7f0c4c19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.805+0000 7f0c517bb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c4c0ffeb0 0x7f0c4c19cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c517bb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c19d1f0 con 0x7f0c4c0ffeb0 2024-04-05T21:45:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c517bb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c1966a0 con 0x7f0c4c0ff500 2024-04-05T21:45:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c4a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c4c0ffeb0 0x7f0c4c19cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c4a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c4c0ffeb0 0x7f0c4c19cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45938/0 (socket says 172.21.15.121:45938) 2024-04-05T21:45:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c4a7fc700 1 -- 172.21.15.121:0/1021785725 learned_addr learned my addr 172.21.15.121:0/1021785725 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c4affd700 1 --2- 172.21.15.121:0/1021785725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c4c0ff500 0x7f0c4c19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c4a7fc700 1 -- 172.21.15.121:0/1021785725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c4c0ff500 msgr2=0x7f0c4c19c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c4a7fc700 1 --2- 172.21.15.121:0/1021785725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c4c0ff500 0x7f0c4c19c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c4a7fc700 1 -- 172.21.15.121:0/1021785725 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3c0095f0 con 0x7f0c4c0ffeb0 2024-04-05T21:45:32.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c4a7fc700 1 --2- 172.21.15.121:0/1021785725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c4c0ffeb0 0x7f0c4c19cb60 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00bda0 tx=0x7f0c3c00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:32.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c33fff700 1 -- 172.21.15.121:0/1021785725 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c3c01d070 con 0x7f0c4c0ffeb0 2024-04-05T21:45:32.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c33fff700 1 -- 172.21.15.121:0/1021785725 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c3c00f460 con 0x7f0c4c0ffeb0 2024-04-05T21:45:32.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c517bb700 1 -- 172.21.15.121:0/1021785725 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4c196920 con 0x7f0c4c0ffeb0 2024-04-05T21:45:32.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c33fff700 1 -- 172.21.15.121:0/1021785725 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c3c021640 con 0x7f0c4c0ffeb0 2024-04-05T21:45:32.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.806+0000 7f0c517bb700 1 -- 172.21.15.121:0/1021785725 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4c196e10 con 0x7f0c4c0ffeb0 2024-04-05T21:45:32.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.807+0000 7f0c517bb700 1 -- 172.21.15.121:0/1021785725 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4c1091b0 con 0x7f0c4c0ffeb0 2024-04-05T21:45:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.808+0000 7f0c33fff700 1 -- 172.21.15.121:0/1021785725 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0c3c00fa60 con 0x7f0c4c0ffeb0 2024-04-05T21:45:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.808+0000 7f0c33fff700 1 --2- 172.21.15.121:0/1021785725 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0c3406cce0 0x7f0c3406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.808+0000 7f0c33fff700 1 -- 172.21.15.121:0/1021785725 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0c3c08d7e0 con 0x7f0c4c0ffeb0 2024-04-05T21:45:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.808+0000 7f0c4affd700 1 --2- 172.21.15.121:0/1021785725 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0c3406cce0 0x7f0c3406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:32.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.809+0000 7f0c4affd700 1 --2- 172.21.15.121:0/1021785725 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0c3406cce0 0x7f0c3406f1a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009a10 tx=0x7f0c40009190 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:32.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:32.813+0000 7f0c33fff700 1 -- 172.21.15.121:0/1021785725 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0c3c05b2e0 con 0x7f0c4c0ffeb0 2024-04-05T21:45:33.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.022+0000 7f0c517bb700 1 -- 172.21.15.121:0/1021785725 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c4c1970f0 con 0x7f0c3406cce0 2024-04-05T21:45:33.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.023+0000 7f0c33fff700 1 -- 172.21.15.121:0/1021785725 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f0c4c1970f0 con 0x7f0c3406cce0 2024-04-05T21:45:33.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.025+0000 7f0c517bb700 1 -- 172.21.15.121:0/1021785725 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0c3406cce0 msgr2=0x7f0c3406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:33.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.025+0000 7f0c517bb700 1 --2- 172.21.15.121:0/1021785725 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0c3406cce0 0x7f0c3406f1a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009a10 tx=0x7f0c40009190 comp rx=0 tx=0).stop 2024-04-05T21:45:33.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.025+0000 7f0c517bb700 1 -- 172.21.15.121:0/1021785725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c4c0ffeb0 msgr2=0x7f0c4c19cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:33.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.025+0000 7f0c517bb700 1 --2- 172.21.15.121:0/1021785725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c4c0ffeb0 0x7f0c4c19cb60 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00bda0 tx=0x7f0c3c00bdd0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.025+0000 7f0c517bb700 1 -- 172.21.15.121:0/1021785725 shutdown_connections 2024-04-05T21:45:33.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.025+0000 7f0c517bb700 1 --2- 172.21.15.121:0/1021785725 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0c3406cce0 0x7f0c3406f1a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.025+0000 7f0c517bb700 1 --2- 172.21.15.121:0/1021785725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0c4c0ff500 0x7f0c4c19c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.025+0000 7f0c517bb700 1 --2- 172.21.15.121:0/1021785725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c4c0ffeb0 0x7f0c4c19cb60 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.025+0000 7f0c517bb700 1 -- 172.21.15.121:0/1021785725 >> 172.21.15.121:0/1021785725 conn(0x7f0c4c074d80 msgr2=0x7f0c4c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:33.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.025+0000 7f0c517bb700 1 -- 172.21.15.121:0/1021785725 shutdown_connections 2024-04-05T21:45:33.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.025+0000 7f0c517bb700 1 -- 172.21.15.121:0/1021785725 wait complete. 2024-04-05T21:45:33.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.127+0000 7f2e98d74700 1 -- 172.21.15.121:0/3597515741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e94105a40 msgr2=0x7f2e94105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:33.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.127+0000 7f2e98d74700 1 --2- 172.21.15.121:0/3597515741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e94105a40 0x7f2e94105e20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2e84009910 tx=0x7f2e84009c20 comp rx=0 tx=0).stop 2024-04-05T21:45:33.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.127+0000 7f2e98d74700 1 -- 172.21.15.121:0/3597515741 shutdown_connections 2024-04-05T21:45:33.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.127+0000 7f2e98d74700 1 --2- 172.21.15.121:0/3597515741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e940ffbd0 0x7f2e94100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.127+0000 7f2e98d74700 1 --2- 172.21.15.121:0/3597515741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e94105a40 0x7f2e94105e20 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.127+0000 7f2e98d74700 1 -- 172.21.15.121:0/3597515741 >> 172.21.15.121:0/3597515741 conn(0x7f2e940755a0 msgr2=0x7f2e940759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:33.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.127+0000 7f2e98d74700 1 -- 172.21.15.121:0/3597515741 shutdown_connections 2024-04-05T21:45:33.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.127+0000 7f2e98d74700 1 -- 172.21.15.121:0/3597515741 wait complete. 2024-04-05T21:45:33.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.128+0000 7f2e98d74700 1 Processor -- start 2024-04-05T21:45:33.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.128+0000 7f2e98d74700 1 -- start start 2024-04-05T21:45:33.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.128+0000 7f2e98d74700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e940ffbd0 0x7f2e94101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:33.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.128+0000 7f2e98d74700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e94105a40 0x7f2e94101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.128+0000 7f2e98d74700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e94103330 con 0x7f2e94105a40 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.128+0000 7f2e98d74700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e941034a0 con 0x7f2e940ffbd0 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e91d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e94105a40 0x7f2e94101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e91d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e94105a40 0x7f2e94101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45958/0 (socket says 172.21.15.121:45958) 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e91d9b700 1 -- 172.21.15.121:0/3200445464 learned_addr learned my addr 172.21.15.121:0/3200445464 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e9259c700 1 --2- 172.21.15.121:0/3200445464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e940ffbd0 0x7f2e94101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e91d9b700 1 -- 172.21.15.121:0/3200445464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e940ffbd0 msgr2=0x7f2e94101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e91d9b700 1 --2- 172.21.15.121:0/3200445464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e940ffbd0 0x7f2e94101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e91d9b700 1 -- 172.21.15.121:0/3200445464 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e840095f0 con 0x7f2e94105a40 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e9259c700 1 --2- 172.21.15.121:0/3200445464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e940ffbd0 0x7f2e94101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e91d9b700 1 --2- 172.21.15.121:0/3200445464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e94105a40 0x7f2e94101b50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c00c930 tx=0x7f2e7c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e8b7fe700 1 -- 172.21.15.121:0/3200445464 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e7c007ad0 con 0x7f2e94105a40 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e98d74700 1 -- 172.21.15.121:0/3200445464 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e941020f0 con 0x7f2e94105a40 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e8b7fe700 1 -- 172.21.15.121:0/3200445464 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e7c007c30 con 0x7f2e94105a40 2024-04-05T21:45:33.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e8b7fe700 1 -- 172.21.15.121:0/3200445464 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e7c018630 con 0x7f2e94105a40 2024-04-05T21:45:33.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.129+0000 7f2e98d74700 1 -- 172.21.15.121:0/3200445464 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e941a6fe0 con 0x7f2e94105a40 2024-04-05T21:45:33.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.130+0000 7f2e98d74700 1 -- 172.21.15.121:0/3200445464 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e94109050 con 0x7f2e94105a40 2024-04-05T21:45:33.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.130+0000 7f2e8b7fe700 1 -- 172.21.15.121:0/3200445464 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2e7c018790 con 0x7f2e94105a40 2024-04-05T21:45:33.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.131+0000 7f2e8b7fe700 1 --2- 172.21.15.121:0/3200445464 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2e8006cce0 0x7f2e8006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:33.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.131+0000 7f2e8b7fe700 1 -- 172.21.15.121:0/3200445464 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2e7c08c7c0 con 0x7f2e94105a40 2024-04-05T21:45:33.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.131+0000 7f2e9259c700 1 --2- 172.21.15.121:0/3200445464 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2e8006cce0 0x7f2e8006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:33.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.131+0000 7f2e9259c700 1 --2- 172.21.15.121:0/3200445464 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2e8006cce0 0x7f2e8006f1a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400b5c0 tx=0x7f2e84005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:33.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.135+0000 7f2e8b7fe700 1 -- 172.21.15.121:0/3200445464 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2e7c0568a0 con 0x7f2e94105a40 2024-04-05T21:45:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.337+0000 7f2e98d74700 1 -- 172.21.15.121:0/3200445464 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e94102280 con 0x7f2e8006cce0 2024-04-05T21:45:33.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.348+0000 7f2e8b7fe700 1 -- 172.21.15.121:0/3200445464 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2e94102280 con 0x7f2e8006cce0 2024-04-05T21:45:33.348 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:45:33.348 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (16m) 10m ago 27m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:45:33.348 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (28m) 10m ago 28m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:45:33.348 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (26m) 10m ago 26m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:45:33.349 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (14m) 10m ago 28m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:45:33.349 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (14m) 10m ago 26m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:45:33.349 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (16m) 10m ago 27m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:45:33.349 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (10m) 10m ago 21m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:45:33.349 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (10m) 10m ago 21m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:45:33.349 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (10m) 10m ago 21m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:45:33.349 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (21m) 10m ago 21m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:45:33.349 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (18m) 10m ago 29m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:45:33.349 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (17m) 10m ago 26m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:45:33.349 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (15m) 10m ago 29m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:45:33.349 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (14m) 10m ago 26m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:45:33.349 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (17m) 10m ago 27m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:45:33.349 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (17m) 10m ago 26m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:45:33.350 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (14m) 10m ago 25m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:45:33.350 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (13m) 10m ago 25m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:45:33.350 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (12m) 10m ago 24m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:45:33.350 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (12m) 10m ago 24m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:45:33.350 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (11m) 10m ago 23m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:45:33.350 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (11m) 10m ago 23m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:45:33.350 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (16m) 10m ago 27m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:45:33.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.350+0000 7f2e98d74700 1 -- 172.21.15.121:0/3200445464 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2e8006cce0 msgr2=0x7f2e8006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:33.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.350+0000 7f2e98d74700 1 --2- 172.21.15.121:0/3200445464 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2e8006cce0 0x7f2e8006f1a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400b5c0 tx=0x7f2e84005fd0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.350+0000 7f2e98d74700 1 -- 172.21.15.121:0/3200445464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e94105a40 msgr2=0x7f2e94101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:33.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.350+0000 7f2e98d74700 1 --2- 172.21.15.121:0/3200445464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e94105a40 0x7f2e94101b50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c00c930 tx=0x7f2e7c00ccf0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.350+0000 7f2e98d74700 1 -- 172.21.15.121:0/3200445464 shutdown_connections 2024-04-05T21:45:33.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.350+0000 7f2e98d74700 1 --2- 172.21.15.121:0/3200445464 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2e8006cce0 0x7f2e8006f1a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.350+0000 7f2e98d74700 1 --2- 172.21.15.121:0/3200445464 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2e940ffbd0 0x7f2e94101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.350+0000 7f2e98d74700 1 --2- 172.21.15.121:0/3200445464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e94105a40 0x7f2e94101b50 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.350+0000 7f2e98d74700 1 -- 172.21.15.121:0/3200445464 >> 172.21.15.121:0/3200445464 conn(0x7f2e940755a0 msgr2=0x7f2e94068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:33.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.350+0000 7f2e98d74700 1 -- 172.21.15.121:0/3200445464 shutdown_connections 2024-04-05T21:45:33.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.350+0000 7f2e98d74700 1 -- 172.21.15.121:0/3200445464 wait complete. 2024-04-05T21:45:33.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.451+0000 7f2247ce1700 1 -- 172.21.15.121:0/1639456823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2240100de0 msgr2=0x7f22401011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:33.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.451+0000 7f2247ce1700 1 --2- 172.21.15.121:0/1639456823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2240100de0 0x7f22401011c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f223800b0a0 tx=0x7f223800b3b0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.451+0000 7f2247ce1700 1 -- 172.21.15.121:0/1639456823 shutdown_connections 2024-04-05T21:45:33.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.451+0000 7f2247ce1700 1 --2- 172.21.15.121:0/1639456823 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2240101790 0x7f22401057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.451+0000 7f2247ce1700 1 --2- 172.21.15.121:0/1639456823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2240100de0 0x7f22401011c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.451+0000 7f2247ce1700 1 -- 172.21.15.121:0/1639456823 >> 172.21.15.121:0/1639456823 conn(0x7f22400fc6b0 msgr2=0x7f22400fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:33.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.451+0000 7f2247ce1700 1 -- 172.21.15.121:0/1639456823 shutdown_connections 2024-04-05T21:45:33.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.451+0000 7f2247ce1700 1 -- 172.21.15.121:0/1639456823 wait complete. 2024-04-05T21:45:33.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f2247ce1700 1 Processor -- start 2024-04-05T21:45:33.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f2247ce1700 1 -- start start 2024-04-05T21:45:33.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f2247ce1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2240100de0 0x7f22401966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:33.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f2247ce1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2240101790 0x7f2240196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:33.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f2247ce1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2240197280 con 0x7f2240100de0 2024-04-05T21:45:33.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f2247ce1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2240199fb0 con 0x7f2240101790 2024-04-05T21:45:33.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f2245a7d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2240100de0 0x7f22401966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:33.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f2245a7d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2240100de0 0x7f22401966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45972/0 (socket says 172.21.15.121:45972) 2024-04-05T21:45:33.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f2245a7d700 1 -- 172.21.15.121:0/3127917134 learned_addr learned my addr 172.21.15.121:0/3127917134 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:33.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f224527c700 1 --2- 172.21.15.121:0/3127917134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2240101790 0x7f2240196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:33.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f2245a7d700 1 -- 172.21.15.121:0/3127917134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2240101790 msgr2=0x7f2240196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:33.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f2245a7d700 1 --2- 172.21.15.121:0/3127917134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2240101790 0x7f2240196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f2245a7d700 1 -- 172.21.15.121:0/3127917134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2238009d00 con 0x7f2240100de0 2024-04-05T21:45:33.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f224527c700 1 --2- 172.21.15.121:0/3127917134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2240101790 0x7f2240196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:45:33.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.452+0000 7f2245a7d700 1 --2- 172.21.15.121:0/3127917134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2240100de0 0x7f22401966b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f223800d010 tx=0x7f2238007be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:33.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.453+0000 7f223effd700 1 -- 172.21.15.121:0/3127917134 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f223800e040 con 0x7f2240100de0 2024-04-05T21:45:33.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.453+0000 7f223effd700 1 -- 172.21.15.121:0/3127917134 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2238003e60 con 0x7f2240100de0 2024-04-05T21:45:33.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.453+0000 7f2247ce1700 1 -- 172.21.15.121:0/3127917134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224019a230 con 0x7f2240100de0 2024-04-05T21:45:33.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.453+0000 7f223effd700 1 -- 172.21.15.121:0/3127917134 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f223801d7c0 con 0x7f2240100de0 2024-04-05T21:45:33.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.453+0000 7f2247ce1700 1 -- 172.21.15.121:0/3127917134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224019a720 con 0x7f2240100de0 2024-04-05T21:45:33.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.454+0000 7f2247ce1700 1 -- 172.21.15.121:0/3127917134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2240109180 con 0x7f2240100de0 2024-04-05T21:45:33.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.454+0000 7f223effd700 1 -- 172.21.15.121:0/3127917134 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f22380046e0 con 0x7f2240100de0 2024-04-05T21:45:33.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.454+0000 7f223effd700 1 --2- 172.21.15.121:0/3127917134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f222806cce0 0x7f222806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:33.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.455+0000 7f223effd700 1 -- 172.21.15.121:0/3127917134 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f223808df80 con 0x7f2240100de0 2024-04-05T21:45:33.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.455+0000 7f224527c700 1 --2- 172.21.15.121:0/3127917134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f222806cce0 0x7f222806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:33.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.455+0000 7f224527c700 1 --2- 172.21.15.121:0/3127917134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f222806cce0 0x7f222806f1a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2234005950 tx=0x7f22340058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:33.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.459+0000 7f223effd700 1 -- 172.21.15.121:0/3127917134 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2238057ff0 con 0x7f2240100de0 2024-04-05T21:45:33.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.732+0000 7f2247ce1700 1 -- 172.21.15.121:0/3127917134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f224004ea90 con 0x7f2240100de0 2024-04-05T21:45:33.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.732+0000 7f223effd700 1 -- 172.21.15.121:0/3127917134 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f223805b610 con 0x7f2240100de0 2024-04-05T21:45:33.733 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:45:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:45:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:45:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:45:33.733 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:45:33.734 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:45:33.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.734+0000 7f2247ce1700 1 -- 172.21.15.121:0/3127917134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f222806cce0 msgr2=0x7f222806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:33.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.734+0000 7f2247ce1700 1 --2- 172.21.15.121:0/3127917134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f222806cce0 0x7f222806f1a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2234005950 tx=0x7f22340058e0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.734+0000 7f2247ce1700 1 -- 172.21.15.121:0/3127917134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2240100de0 msgr2=0x7f22401966b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:33.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.734+0000 7f2247ce1700 1 --2- 172.21.15.121:0/3127917134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2240100de0 0x7f22401966b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f223800d010 tx=0x7f2238007be0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.734+0000 7f2247ce1700 1 -- 172.21.15.121:0/3127917134 shutdown_connections 2024-04-05T21:45:33.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.734+0000 7f2247ce1700 1 --2- 172.21.15.121:0/3127917134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f222806cce0 0x7f222806f1a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.734+0000 7f2247ce1700 1 --2- 172.21.15.121:0/3127917134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2240100de0 0x7f22401966b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.734+0000 7f2247ce1700 1 --2- 172.21.15.121:0/3127917134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2240101790 0x7f2240196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.734+0000 7f2247ce1700 1 -- 172.21.15.121:0/3127917134 >> 172.21.15.121:0/3127917134 conn(0x7f22400fc6b0 msgr2=0x7f22400fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:33.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.735+0000 7f2247ce1700 1 -- 172.21.15.121:0/3127917134 shutdown_connections 2024-04-05T21:45:33.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.735+0000 7f2247ce1700 1 -- 172.21.15.121:0/3127917134 wait complete. 2024-04-05T21:45:33.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.837+0000 7f8304f9d700 1 -- 172.21.15.121:0/2866141255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83000ffbd0 msgr2=0x7f8300100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:33.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.837+0000 7f8304f9d700 1 --2- 172.21.15.121:0/2866141255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83000ffbd0 0x7f8300100050 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009910 tx=0x7f82f0009c20 comp rx=0 tx=0).stop 2024-04-05T21:45:33.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.837+0000 7f8304f9d700 1 -- 172.21.15.121:0/2866141255 shutdown_connections 2024-04-05T21:45:33.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.837+0000 7f8304f9d700 1 --2- 172.21.15.121:0/2866141255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83000ffbd0 0x7f8300100050 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.837+0000 7f8304f9d700 1 --2- 172.21.15.121:0/2866141255 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8300105a40 0x7f8300105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.837+0000 7f8304f9d700 1 -- 172.21.15.121:0/2866141255 >> 172.21.15.121:0/2866141255 conn(0x7f83000755a0 msgr2=0x7f83000759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:33.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.837+0000 7f8304f9d700 1 -- 172.21.15.121:0/2866141255 shutdown_connections 2024-04-05T21:45:33.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.837+0000 7f8304f9d700 1 -- 172.21.15.121:0/2866141255 wait complete. 2024-04-05T21:45:33.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.838+0000 7f8304f9d700 1 Processor -- start 2024-04-05T21:45:33.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.838+0000 7f8304f9d700 1 -- start start 2024-04-05T21:45:33.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f8304f9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83000ffbd0 0x7f83001974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:33.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f8304f9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8300105a40 0x7f83001979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:33.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f8304f9d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8300198010 con 0x7f83000ffbd0 2024-04-05T21:45:33.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f8304f9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8300198150 con 0x7f8300105a40 2024-04-05T21:45:33.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f82fe59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83000ffbd0 0x7f83001974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:33.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f82fe59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83000ffbd0 0x7f83001974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45996/0 (socket says 172.21.15.121:45996) 2024-04-05T21:45:33.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f82fe59c700 1 -- 172.21.15.121:0/2472858859 learned_addr learned my addr 172.21.15.121:0/2472858859 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:33.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f82fdd9b700 1 --2- 172.21.15.121:0/2472858859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8300105a40 0x7f83001979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:33.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f82fe59c700 1 -- 172.21.15.121:0/2472858859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8300105a40 msgr2=0x7f83001979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:33.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f82fe59c700 1 --2- 172.21.15.121:0/2472858859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8300105a40 0x7f83001979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:33.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f82fe59c700 1 -- 172.21.15.121:0/2472858859 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f00095f0 con 0x7f83000ffbd0 2024-04-05T21:45:33.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f82fe59c700 1 --2- 172.21.15.121:0/2472858859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83000ffbd0 0x7f83001974b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f82e800c8a0 tx=0x7f82e800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:33.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f82f77fe700 1 -- 172.21.15.121:0/2472858859 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e800cea0 con 0x7f83000ffbd0 2024-04-05T21:45:33.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f82f77fe700 1 -- 172.21.15.121:0/2472858859 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82e8004830 con 0x7f83000ffbd0 2024-04-05T21:45:33.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f8304f9d700 1 -- 172.21.15.121:0/2472858859 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83001a0520 con 0x7f83000ffbd0 2024-04-05T21:45:33.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f82f77fe700 1 -- 172.21.15.121:0/2472858859 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e8005500 con 0x7f83000ffbd0 2024-04-05T21:45:33.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.839+0000 7f8304f9d700 1 -- 172.21.15.121:0/2472858859 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83001a09e0 con 0x7f83000ffbd0 2024-04-05T21:45:33.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.840+0000 7f8304f9d700 1 -- 172.21.15.121:0/2472858859 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8300107fd0 con 0x7f83000ffbd0 2024-04-05T21:45:33.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.841+0000 7f82f77fe700 1 -- 172.21.15.121:0/2472858859 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f82e80049a0 con 0x7f83000ffbd0 2024-04-05T21:45:33.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.841+0000 7f82f77fe700 1 --2- 172.21.15.121:0/2472858859 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:33.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.841+0000 7f82f77fe700 1 -- 172.21.15.121:0/2472858859 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f82e808c2b0 con 0x7f83000ffbd0 2024-04-05T21:45:33.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.841+0000 7f82fdd9b700 1 --2- 172.21.15.121:0/2472858859 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:33.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.842+0000 7f82fdd9b700 1 --2- 172.21.15.121:0/2472858859 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8300068cc0 tx=0x7f82f001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:33.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:33.846+0000 7f82f77fe700 1 -- 172.21.15.121:0/2472858859 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f82e8056320 con 0x7f83000ffbd0 2024-04-05T21:45:33.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:33 smithi121 ceph-mon[145859]: pgmap v579: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:33.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:33 smithi121 ceph-mon[145859]: from='client.34902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:34.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:33 smithi188 ceph-mon[107793]: pgmap v579: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:34.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:33 smithi188 ceph-mon[107793]: from='client.34902 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:34.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.078+0000 7f8304f9d700 1 -- 172.21.15.121:0/2472858859 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f830002cd10 con 0x7f83000ffbd0 2024-04-05T21:45:34.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.078+0000 7f82f77fe700 1 -- 172.21.15.121:0/2472858859 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f82e8059940 con 0x7f83000ffbd0 2024-04-05T21:45:34.079 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:45:34.079 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:45:34.079 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:45:34.079 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:45:34.079 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:45:34.079 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:45:34.079 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:45:34.080 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:45:34.081 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:45:34.082 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:45:34.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.080+0000 7f8304f9d700 1 -- 172.21.15.121:0/2472858859 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 msgr2=0x7f82ec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:34.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.080+0000 7f8304f9d700 1 --2- 172.21.15.121:0/2472858859 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8300068cc0 tx=0x7f82f001a040 comp rx=0 tx=0).stop 2024-04-05T21:45:34.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.080+0000 7f8304f9d700 1 -- 172.21.15.121:0/2472858859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83000ffbd0 msgr2=0x7f83001974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:34.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.080+0000 7f8304f9d700 1 --2- 172.21.15.121:0/2472858859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83000ffbd0 0x7f83001974b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f82e800c8a0 tx=0x7f82e800cc60 comp rx=0 tx=0).stop 2024-04-05T21:45:34.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.080+0000 7f8304f9d700 1 -- 172.21.15.121:0/2472858859 shutdown_connections 2024-04-05T21:45:34.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.080+0000 7f8304f9d700 1 --2- 172.21.15.121:0/2472858859 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.080+0000 7f8304f9d700 1 --2- 172.21.15.121:0/2472858859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83000ffbd0 0x7f83001974b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.080+0000 7f8304f9d700 1 --2- 172.21.15.121:0/2472858859 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8300105a40 0x7f83001979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.080+0000 7f8304f9d700 1 -- 172.21.15.121:0/2472858859 >> 172.21.15.121:0/2472858859 conn(0x7f83000755a0 msgr2=0x7f83000fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:34.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.080+0000 7f8304f9d700 1 -- 172.21.15.121:0/2472858859 shutdown_connections 2024-04-05T21:45:34.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.080+0000 7f8304f9d700 1 -- 172.21.15.121:0/2472858859 wait complete. 2024-04-05T21:45:34.083 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:45:34.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.184+0000 7fed073fc700 1 -- 172.21.15.121:0/1001726868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed00101770 msgr2=0x7fed001057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:34.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.184+0000 7fed073fc700 1 --2- 172.21.15.121:0/1001726868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed00101770 0x7fed001057c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009910 tx=0x7fecfc009c20 comp rx=0 tx=0).stop 2024-04-05T21:45:34.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.184+0000 7fed073fc700 1 -- 172.21.15.121:0/1001726868 shutdown_connections 2024-04-05T21:45:34.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.184+0000 7fed073fc700 1 --2- 172.21.15.121:0/1001726868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed00101770 0x7fed001057c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.184+0000 7fed073fc700 1 --2- 172.21.15.121:0/1001726868 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed00100dc0 0x7fed001011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.184+0000 7fed073fc700 1 -- 172.21.15.121:0/1001726868 >> 172.21.15.121:0/1001726868 conn(0x7fed000fc670 msgr2=0x7fed000fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:34.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.184+0000 7fed073fc700 1 -- 172.21.15.121:0/1001726868 shutdown_connections 2024-04-05T21:45:34.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.184+0000 7fed073fc700 1 -- 172.21.15.121:0/1001726868 wait complete. 2024-04-05T21:45:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.184+0000 7fed073fc700 1 Processor -- start 2024-04-05T21:45:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.184+0000 7fed073fc700 1 -- start start 2024-04-05T21:45:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed073fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed00100dc0 0x7fed00073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed073fc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed00101770 0x7fed000735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed073fc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed00073b10 con 0x7fed00100dc0 2024-04-05T21:45:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed073fc700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed00073c50 con 0x7fed00101770 2024-04-05T21:45:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed05198700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed00100dc0 0x7fed00073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed05198700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed00100dc0 0x7fed00073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46014/0 (socket says 172.21.15.121:46014) 2024-04-05T21:45:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed05198700 1 -- 172.21.15.121:0/3378286988 learned_addr learned my addr 172.21.15.121:0/3378286988 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed04997700 1 --2- 172.21.15.121:0/3378286988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed00101770 0x7fed000735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed05198700 1 -- 172.21.15.121:0/3378286988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed00101770 msgr2=0x7fed000735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed05198700 1 --2- 172.21.15.121:0/3378286988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed00101770 0x7fed000735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed05198700 1 -- 172.21.15.121:0/3378286988 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc0095f0 con 0x7fed00100dc0 2024-04-05T21:45:34.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed05198700 1 --2- 172.21.15.121:0/3378286988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed00100dc0 0x7fed00073090 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fecf000c8a0 tx=0x7fecf000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:34.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fecf67fc700 1 -- 172.21.15.121:0/3378286988 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecf00078e0 con 0x7fed00100dc0 2024-04-05T21:45:34.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fecf67fc700 1 -- 172.21.15.121:0/3378286988 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecf000f450 con 0x7fed00100dc0 2024-04-05T21:45:34.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed073fc700 1 -- 172.21.15.121:0/3378286988 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed001a2820 con 0x7fed00100dc0 2024-04-05T21:45:34.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fecf67fc700 1 -- 172.21.15.121:0/3378286988 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecf000e5e0 con 0x7fed00100dc0 2024-04-05T21:45:34.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.185+0000 7fed073fc700 1 -- 172.21.15.121:0/3378286988 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed001a2c30 con 0x7fed00100dc0 2024-04-05T21:45:34.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.186+0000 7fed073fc700 1 -- 172.21.15.121:0/3378286988 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed001091c0 con 0x7fed00100dc0 2024-04-05T21:45:34.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.187+0000 7fecf67fc700 1 -- 172.21.15.121:0/3378286988 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fecf000f5c0 con 0x7fed00100dc0 2024-04-05T21:45:34.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.187+0000 7fecf67fc700 1 --2- 172.21.15.121:0/3378286988 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fecec06cce0 0x7fecec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:34.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.187+0000 7fecf67fc700 1 -- 172.21.15.121:0/3378286988 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fecf008c1d0 con 0x7fed00100dc0 2024-04-05T21:45:34.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.187+0000 7fed04997700 1 --2- 172.21.15.121:0/3378286988 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fecec06cce0 0x7fecec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:34.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.188+0000 7fed04997700 1 --2- 172.21.15.121:0/3378286988 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fecec06cce0 0x7fecec06f1a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fed00074880 tx=0x7fecfc01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:34.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.192+0000 7fecf67fc700 1 -- 172.21.15.121:0/3378286988 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fecf0056240 con 0x7fed00100dc0 2024-04-05T21:45:34.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.401+0000 7fed073fc700 1 -- 172.21.15.121:0/3378286988 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fed0002cdd0 con 0x7fecec06cce0 2024-04-05T21:45:34.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.402+0000 7fecf67fc700 1 -- 172.21.15.121:0/3378286988 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fed0002cdd0 con 0x7fecec06cce0 2024-04-05T21:45:34.403 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:45:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:45:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:45:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:45:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:45:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:45:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:45:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:45:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:45:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:45:34.403 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:45:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:45:34.404 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:45:34.404 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:45:34.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.404+0000 7fed073fc700 1 -- 172.21.15.121:0/3378286988 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fecec06cce0 msgr2=0x7fecec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:34.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.404+0000 7fed073fc700 1 --2- 172.21.15.121:0/3378286988 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fecec06cce0 0x7fecec06f1a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fed00074880 tx=0x7fecfc01a040 comp rx=0 tx=0).stop 2024-04-05T21:45:34.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.404+0000 7fed073fc700 1 -- 172.21.15.121:0/3378286988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed00100dc0 msgr2=0x7fed00073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:34.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.404+0000 7fed073fc700 1 --2- 172.21.15.121:0/3378286988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed00100dc0 0x7fed00073090 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fecf000c8a0 tx=0x7fecf000cbb0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.404+0000 7fed073fc700 1 -- 172.21.15.121:0/3378286988 shutdown_connections 2024-04-05T21:45:34.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.404+0000 7fed073fc700 1 --2- 172.21.15.121:0/3378286988 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fecec06cce0 0x7fecec06f1a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.404+0000 7fed073fc700 1 --2- 172.21.15.121:0/3378286988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed00100dc0 0x7fed00073090 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.404+0000 7fed073fc700 1 --2- 172.21.15.121:0/3378286988 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed00101770 0x7fed000735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.404+0000 7fed073fc700 1 -- 172.21.15.121:0/3378286988 >> 172.21.15.121:0/3378286988 conn(0x7fed000fc670 msgr2=0x7fed000fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:34.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.404+0000 7fed073fc700 1 -- 172.21.15.121:0/3378286988 shutdown_connections 2024-04-05T21:45:34.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.404+0000 7fed073fc700 1 -- 172.21.15.121:0/3378286988 wait complete. 2024-04-05T21:45:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.509+0000 7f3e0802e700 1 -- 172.21.15.121:0/2050860227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00069020 msgr2=0x7f3e00105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.509+0000 7f3e0802e700 1 --2- 172.21.15.121:0/2050860227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00069020 0x7f3e00105790 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009910 tx=0x7f3df0009c20 comp rx=0 tx=0).stop 2024-04-05T21:45:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.509+0000 7f3e0802e700 1 -- 172.21.15.121:0/2050860227 shutdown_connections 2024-04-05T21:45:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.509+0000 7f3e0802e700 1 --2- 172.21.15.121:0/2050860227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00069020 0x7f3e00105790 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.509+0000 7f3e0802e700 1 --2- 172.21.15.121:0/2050860227 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e00068670 0x7f3e00068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.509+0000 7f3e0802e700 1 -- 172.21.15.121:0/2050860227 >> 172.21.15.121:0/2050860227 conn(0x7f3e000755a0 msgr2=0x7f3e000759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.509+0000 7f3e0802e700 1 -- 172.21.15.121:0/2050860227 shutdown_connections 2024-04-05T21:45:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.509+0000 7f3e0802e700 1 -- 172.21.15.121:0/2050860227 wait complete. 2024-04-05T21:45:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.509+0000 7f3e0802e700 1 Processor -- start 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.509+0000 7f3e0802e700 1 -- start start 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e0802e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e00068670 0x7f3e00196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e0802e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00069020 0x7f3e00196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e0802e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e00197260 con 0x7f3e00069020 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e0802e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0019aff0 con 0x7f3e00068670 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e055c9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00069020 0x7f3e00196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e055c9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00069020 0x7f3e00196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45262/0 (socket says 172.21.15.121:45262) 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e055c9700 1 -- 172.21.15.121:0/2717069661 learned_addr learned my addr 172.21.15.121:0/2717069661 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e05dca700 1 --2- 172.21.15.121:0/2717069661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e00068670 0x7f3e00196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e055c9700 1 -- 172.21.15.121:0/2717069661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e00068670 msgr2=0x7f3e00196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e055c9700 1 --2- 172.21.15.121:0/2717069661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e00068670 0x7f3e00196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e055c9700 1 -- 172.21.15.121:0/2717069661 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df00095f0 con 0x7f3e00069020 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e05dca700 1 --2- 172.21.15.121:0/2717069661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e00068670 0x7f3e00196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e055c9700 1 --2- 172.21.15.121:0/2717069661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00069020 0x7f3e00196b80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3df00092e0 tx=0x7f3df000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3df6ffd700 1 -- 172.21.15.121:0/2717069661 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df001d070 con 0x7f3e00069020 2024-04-05T21:45:34.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3df6ffd700 1 -- 172.21.15.121:0/2717069661 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3df0005940 con 0x7f3e00069020 2024-04-05T21:45:34.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e0802e700 1 -- 172.21.15.121:0/2717069661 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0019b270 con 0x7f3e00069020 2024-04-05T21:45:34.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3df6ffd700 1 -- 172.21.15.121:0/2717069661 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3df000fe50 con 0x7f3e00069020 2024-04-05T21:45:34.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.510+0000 7f3e0802e700 1 -- 172.21.15.121:0/2717069661 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0019b760 con 0x7f3e00069020 2024-04-05T21:45:34.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.511+0000 7f3e0802e700 1 -- 172.21.15.121:0/2717069661 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e001090a0 con 0x7f3e00069020 2024-04-05T21:45:34.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.512+0000 7f3df6ffd700 1 -- 172.21.15.121:0/2717069661 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3df0003a60 con 0x7f3e00069020 2024-04-05T21:45:34.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.512+0000 7f3df6ffd700 1 --2- 172.21.15.121:0/2717069661 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3dec06cce0 0x7f3dec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:45:34.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.512+0000 7f3df6ffd700 1 -- 172.21.15.121:0/2717069661 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3df008d740 con 0x7f3e00069020 2024-04-05T21:45:34.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.512+0000 7f3e05dca700 1 --2- 172.21.15.121:0/2717069661 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3dec06cce0 0x7f3dec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:45:34.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.513+0000 7f3e05dca700 1 --2- 172.21.15.121:0/2717069661 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3dec06cce0 0x7f3dec06f1a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc005950 tx=0x7f3dfc0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:45:34.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.517+0000 7f3df6ffd700 1 -- 172.21.15.121:0/2717069661 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3df00577b0 con 0x7f3e00069020 2024-04-05T21:45:34.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.789+0000 7f3e0802e700 1 -- 172.21.15.121:0/2717069661 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3e0004ea90 con 0x7f3e00069020 2024-04-05T21:45:34.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.790+0000 7f3df6ffd700 1 -- 172.21.15.121:0/2717069661 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f3df005add0 con 0x7f3e00069020 2024-04-05T21:45:34.790 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:45:34.791 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:45:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:45:34.791 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:45:34.791 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:45:34.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.792+0000 7f3e0802e700 1 -- 172.21.15.121:0/2717069661 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3dec06cce0 msgr2=0x7f3dec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:34.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.792+0000 7f3e0802e700 1 --2- 172.21.15.121:0/2717069661 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3dec06cce0 0x7f3dec06f1a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc005950 tx=0x7f3dfc0058e0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.792+0000 7f3e0802e700 1 -- 172.21.15.121:0/2717069661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00069020 msgr2=0x7f3e00196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:45:34.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.792+0000 7f3e0802e700 1 --2- 172.21.15.121:0/2717069661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00069020 0x7f3e00196b80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3df00092e0 tx=0x7f3df000be20 comp rx=0 tx=0).stop 2024-04-05T21:45:34.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.792+0000 7f3e0802e700 1 -- 172.21.15.121:0/2717069661 shutdown_connections 2024-04-05T21:45:34.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.792+0000 7f3e0802e700 1 --2- 172.21.15.121:0/2717069661 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3dec06cce0 0x7f3dec06f1a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.792+0000 7f3e0802e700 1 --2- 172.21.15.121:0/2717069661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3e00068670 0x7f3e00196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.792+0000 7f3e0802e700 1 --2- 172.21.15.121:0/2717069661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00069020 0x7f3e00196b80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:45:34.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.792+0000 7f3e0802e700 1 -- 172.21.15.121:0/2717069661 >> 172.21.15.121:0/2717069661 conn(0x7f3e000755a0 msgr2=0x7f3e00105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:45:34.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.792+0000 7f3e0802e700 1 -- 172.21.15.121:0/2717069661 shutdown_connections 2024-04-05T21:45:34.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:45:34.792+0000 7f3e0802e700 1 -- 172.21.15.121:0/2717069661 wait complete. 2024-04-05T21:45:34.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:34 smithi121 ceph-mon[145859]: from='client.34906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:34.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:34 smithi121 ceph-mon[145859]: from='client.34910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:34.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:34 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3127917134' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:45:34.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:34 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2472858859' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:45:35.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:34 smithi188 ceph-mon[107793]: from='client.34906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:35.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:34 smithi188 ceph-mon[107793]: from='client.34910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:35.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:34 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3127917134' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:45:35.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:34 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2472858859' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:45:35.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:35 smithi121 ceph-mon[145859]: pgmap v580: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:35.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:35 smithi121 ceph-mon[145859]: from='client.34922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:35.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:35 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2717069661' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:45:36.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:35 smithi188 ceph-mon[107793]: pgmap v580: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:36.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:35 smithi188 ceph-mon[107793]: from='client.34922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:45:36.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:35 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2717069661' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:45:37.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:37 smithi121 ceph-mon[145859]: pgmap v581: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:38.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:37 smithi188 ceph-mon[107793]: pgmap v581: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:39.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:39 smithi121 ceph-mon[145859]: pgmap v582: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:40.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:39 smithi188 ceph-mon[107793]: pgmap v582: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:41.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:41 smithi121 ceph-mon[145859]: pgmap v583: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:42.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:41 smithi188 ceph-mon[107793]: pgmap v583: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:43.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:43 smithi121 ceph-mon[145859]: pgmap v584: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:44.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:43 smithi188 ceph-mon[107793]: pgmap v584: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:45 smithi121 ceph-mon[145859]: pgmap v585: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:45:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:45 smithi188 ceph-mon[107793]: pgmap v585: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:45:47.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:47 smithi121 ceph-mon[145859]: pgmap v586: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:48.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:47 smithi188 ceph-mon[107793]: pgmap v586: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:49.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:49 smithi121 ceph-mon[145859]: pgmap v587: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:50.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:49 smithi188 ceph-mon[107793]: pgmap v587: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:51.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:51 smithi121 ceph-mon[145859]: pgmap v588: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:52.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:51 smithi188 ceph-mon[107793]: pgmap v588: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:53.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:53 smithi121 ceph-mon[145859]: pgmap v589: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:54.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:53 smithi188 ceph-mon[107793]: pgmap v589: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:55.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:55 smithi121 ceph-mon[145859]: pgmap v590: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:56.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:55 smithi188 ceph-mon[107793]: pgmap v590: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:57.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:57 smithi121 ceph-mon[145859]: pgmap v591: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:58.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:57 smithi188 ceph-mon[107793]: pgmap v591: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:45:59.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:45:59 smithi121 ceph-mon[145859]: pgmap v592: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:00.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:45:59 smithi188 ceph-mon[107793]: pgmap v592: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:46:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:46:01.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:01 smithi121 ceph-mon[145859]: pgmap v593: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:02.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:01 smithi188 ceph-mon[107793]: pgmap v593: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:03.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:03 smithi121 ceph-mon[145859]: pgmap v594: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:04.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:03 smithi188 ceph-mon[107793]: pgmap v594: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:04.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.908+0000 7f47b8a1c700 1 -- 172.21.15.121:0/419821572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b40ff290 msgr2=0x7f47b40ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:04.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.908+0000 7f47b8a1c700 1 --2- 172.21.15.121:0/419821572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b40ff290 0x7f47b40ff670 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009910 tx=0x7f47a8009c20 comp rx=0 tx=0).stop 2024-04-05T21:46:04.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.908+0000 7f47b8a1c700 1 -- 172.21.15.121:0/419821572 shutdown_connections 2024-04-05T21:46:04.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.908+0000 7f47b8a1c700 1 --2- 172.21.15.121:0/419821572 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b40ffc40 0x7f47b4105780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:04.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.908+0000 7f47b8a1c700 1 --2- 172.21.15.121:0/419821572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b40ff290 0x7f47b40ff670 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:04.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.908+0000 7f47b8a1c700 1 -- 172.21.15.121:0/419821572 >> 172.21.15.121:0/419821572 conn(0x7f47b4075e30 msgr2=0x7f47b40fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:04.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.908+0000 7f47b8a1c700 1 -- 172.21.15.121:0/419821572 shutdown_connections 2024-04-05T21:46:04.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.908+0000 7f47b8a1c700 1 -- 172.21.15.121:0/419821572 wait complete. 2024-04-05T21:46:04.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.909+0000 7f47b8a1c700 1 Processor -- start 2024-04-05T21:46:04.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.909+0000 7f47b8a1c700 1 -- start start 2024-04-05T21:46:04.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.909+0000 7f47b8a1c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b40ff290 0x7f47b4100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:04.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.909+0000 7f47b8a1c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b40ffc40 0x7f47b4100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:04.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.909+0000 7f47b8a1c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b4100c90 con 0x7f47b40ff290 2024-04-05T21:46:04.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.909+0000 7f47b8a1c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b4100dd0 con 0x7f47b40ffc40 2024-04-05T21:46:04.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.910+0000 7f47b259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b40ff290 0x7f47b4100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:04.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.910+0000 7f47b259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b40ff290 0x7f47b4100210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33966/0 (socket says 172.21.15.121:33966) 2024-04-05T21:46:04.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.910+0000 7f47b259c700 1 -- 172.21.15.121:0/3049169121 learned_addr learned my addr 172.21.15.121:0/3049169121 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:04.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.910+0000 7f47b1d9b700 1 --2- 172.21.15.121:0/3049169121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b40ffc40 0x7f47b4100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:04.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.910+0000 7f47b259c700 1 -- 172.21.15.121:0/3049169121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b40ffc40 msgr2=0x7f47b4100750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:04.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.910+0000 7f47b259c700 1 --2- 172.21.15.121:0/3049169121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b40ffc40 0x7f47b4100750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:04.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.910+0000 7f47b259c700 1 -- 172.21.15.121:0/3049169121 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a80095f0 con 0x7f47b40ff290 2024-04-05T21:46:04.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.910+0000 7f47b259c700 1 --2- 172.21.15.121:0/3049169121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b40ff290 0x7f47b4100210 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f47a8006010 tx=0x7f47a800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:04.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.910+0000 7f47a37fe700 1 -- 172.21.15.121:0/3049169121 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a801d070 con 0x7f47b40ff290 2024-04-05T21:46:04.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.910+0000 7f47b8a1c700 1 -- 172.21.15.121:0/3049169121 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b4101050 con 0x7f47b40ff290 2024-04-05T21:46:04.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.910+0000 7f47a37fe700 1 -- 172.21.15.121:0/3049169121 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47a8005710 con 0x7f47b40ff290 2024-04-05T21:46:04.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.910+0000 7f47a37fe700 1 -- 172.21.15.121:0/3049169121 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47a800fe20 con 0x7f47b40ff290 2024-04-05T21:46:04.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.910+0000 7f47b8a1c700 1 -- 172.21.15.121:0/3049169121 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b41a2bd0 con 0x7f47b40ff290 2024-04-05T21:46:04.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.911+0000 7f47b8a1c700 1 -- 172.21.15.121:0/3049169121 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b4109120 con 0x7f47b40ff290 2024-04-05T21:46:04.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.912+0000 7f47a37fe700 1 -- 172.21.15.121:0/3049169121 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f47a8003a60 con 0x7f47b40ff290 2024-04-05T21:46:04.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.912+0000 7f47a37fe700 1 --2- 172.21.15.121:0/3049169121 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f479c06cce0 0x7f479c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:04.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.912+0000 7f47a37fe700 1 -- 172.21.15.121:0/3049169121 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f47a808d720 con 0x7f47b40ff290 2024-04-05T21:46:04.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.912+0000 7f47b1d9b700 1 --2- 172.21.15.121:0/3049169121 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f479c06cce0 0x7f479c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:04.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.913+0000 7f47b1d9b700 1 --2- 172.21.15.121:0/3049169121 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f479c06cce0 0x7f479c06f1a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f47b4101a00 tx=0x7f47a400b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:04.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:04.917+0000 7f47a37fe700 1 -- 172.21.15.121:0/3049169121 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f47a8057790 con 0x7f47b40ff290 2024-04-05T21:46:05.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.125+0000 7f47b8a1c700 1 -- 172.21.15.121:0/3049169121 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f47b402cce0 con 0x7f479c06cce0 2024-04-05T21:46:05.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.126+0000 7f47a37fe700 1 -- 172.21.15.121:0/3049169121 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f47b402cce0 con 0x7f479c06cce0 2024-04-05T21:46:05.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.128+0000 7f47b8a1c700 1 -- 172.21.15.121:0/3049169121 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f479c06cce0 msgr2=0x7f479c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:05.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.128+0000 7f47b8a1c700 1 --2- 172.21.15.121:0/3049169121 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f479c06cce0 0x7f479c06f1a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f47b4101a00 tx=0x7f47a400b410 comp rx=0 tx=0).stop 2024-04-05T21:46:05.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.128+0000 7f47b8a1c700 1 -- 172.21.15.121:0/3049169121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b40ff290 msgr2=0x7f47b4100210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:05.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.128+0000 7f47b8a1c700 1 --2- 172.21.15.121:0/3049169121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b40ff290 0x7f47b4100210 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f47a8006010 tx=0x7f47a800bd70 comp rx=0 tx=0).stop 2024-04-05T21:46:05.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.128+0000 7f47b8a1c700 1 -- 172.21.15.121:0/3049169121 shutdown_connections 2024-04-05T21:46:05.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.128+0000 7f47b8a1c700 1 --2- 172.21.15.121:0/3049169121 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f479c06cce0 0x7f479c06f1a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.128+0000 7f47b8a1c700 1 --2- 172.21.15.121:0/3049169121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47b40ff290 0x7f47b4100210 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.128+0000 7f47b8a1c700 1 --2- 172.21.15.121:0/3049169121 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f47b40ffc40 0x7f47b4100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.128+0000 7f47b8a1c700 1 -- 172.21.15.121:0/3049169121 >> 172.21.15.121:0/3049169121 conn(0x7f47b4075e30 msgr2=0x7f47b4104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:05.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.128+0000 7f47b8a1c700 1 -- 172.21.15.121:0/3049169121 shutdown_connections 2024-04-05T21:46:05.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.128+0000 7f47b8a1c700 1 -- 172.21.15.121:0/3049169121 wait complete. 2024-04-05T21:46:05.139 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:46:05.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.231+0000 7f205755e700 1 -- 172.21.15.121:0/2739213109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2050068670 msgr2=0x7f2050068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:05.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.231+0000 7f205755e700 1 --2- 172.21.15.121:0/2739213109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2050068670 0x7f2050068a50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2044009910 tx=0x7f2044009c20 comp rx=0 tx=0).stop 2024-04-05T21:46:05.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.231+0000 7f205755e700 1 -- 172.21.15.121:0/2739213109 shutdown_connections 2024-04-05T21:46:05.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.231+0000 7f205755e700 1 --2- 172.21.15.121:0/2739213109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2050069020 0x7f2050105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.231+0000 7f205755e700 1 --2- 172.21.15.121:0/2739213109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2050068670 0x7f2050068a50 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.231+0000 7f205755e700 1 -- 172.21.15.121:0/2739213109 >> 172.21.15.121:0/2739213109 conn(0x7f20500755a0 msgr2=0x7f20500759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:05.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.231+0000 7f205755e700 1 -- 172.21.15.121:0/2739213109 shutdown_connections 2024-04-05T21:46:05.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.231+0000 7f205755e700 1 -- 172.21.15.121:0/2739213109 wait complete. 2024-04-05T21:46:05.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.232+0000 7f205755e700 1 Processor -- start 2024-04-05T21:46:05.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.232+0000 7f205755e700 1 -- start start 2024-04-05T21:46:05.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.232+0000 7f205755e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2050068670 0x7f2050196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:05.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.232+0000 7f205755e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2050069020 0x7f2050196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:05.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.232+0000 7f205755e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2050197290 con 0x7f2050069020 2024-04-05T21:46:05.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.232+0000 7f205755e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205019b020 con 0x7f2050068670 2024-04-05T21:46:05.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.232+0000 7f2054af9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2050069020 0x7f2050196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:05.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.232+0000 7f2054af9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2050069020 0x7f2050196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33984/0 (socket says 172.21.15.121:33984) 2024-04-05T21:46:05.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.232+0000 7f2054af9700 1 -- 172.21.15.121:0/3264148754 learned_addr learned my addr 172.21.15.121:0/3264148754 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:05.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.233+0000 7f20552fa700 1 --2- 172.21.15.121:0/3264148754 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2050068670 0x7f2050196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:05.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.233+0000 7f2054af9700 1 -- 172.21.15.121:0/3264148754 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2050068670 msgr2=0x7f2050196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:05.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.233+0000 7f2054af9700 1 --2- 172.21.15.121:0/3264148754 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2050068670 0x7f2050196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.233+0000 7f2054af9700 1 -- 172.21.15.121:0/3264148754 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20440095f0 con 0x7f2050069020 2024-04-05T21:46:05.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.233+0000 7f20552fa700 1 --2- 172.21.15.121:0/3264148754 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2050068670 0x7f2050196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:46:05.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.233+0000 7f2054af9700 1 --2- 172.21.15.121:0/3264148754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2050069020 0x7f2050196bb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f204c00d900 tx=0x7f204c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:05.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.233+0000 7f20427fc700 1 -- 172.21.15.121:0/3264148754 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f204c009900 con 0x7f2050069020 2024-04-05T21:46:05.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.233+0000 7f205755e700 1 -- 172.21.15.121:0/3264148754 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205019b300 con 0x7f2050069020 2024-04-05T21:46:05.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.233+0000 7f20427fc700 1 -- 172.21.15.121:0/3264148754 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f204c010460 con 0x7f2050069020 2024-04-05T21:46:05.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.233+0000 7f20427fc700 1 -- 172.21.15.121:0/3264148754 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f204c00f5f0 con 0x7f2050069020 2024-04-05T21:46:05.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.233+0000 7f205755e700 1 -- 172.21.15.121:0/3264148754 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205019b850 con 0x7f2050069020 2024-04-05T21:46:05.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.234+0000 7f205755e700 1 -- 172.21.15.121:0/3264148754 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2050109130 con 0x7f2050069020 2024-04-05T21:46:05.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.234+0000 7f20427fc700 1 -- 172.21.15.121:0/3264148754 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f204c0105d0 con 0x7f2050069020 2024-04-05T21:46:05.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.235+0000 7f20427fc700 1 --2- 172.21.15.121:0/3264148754 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f203c06cce0 0x7f203c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:05.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.235+0000 7f20427fc700 1 -- 172.21.15.121:0/3264148754 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f204c08c3c0 con 0x7f2050069020 2024-04-05T21:46:05.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.235+0000 7f20552fa700 1 --2- 172.21.15.121:0/3264148754 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f203c06cce0 0x7f203c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:05.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.235+0000 7f20552fa700 1 --2- 172.21.15.121:0/3264148754 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f203c06cce0 0x7f203c06f1a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f204400b5c0 tx=0x7f2044005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:05.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.239+0000 7f20427fc700 1 -- 172.21.15.121:0/3264148754 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f204c056430 con 0x7f2050069020 2024-04-05T21:46:05.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.449+0000 7f205755e700 1 -- 172.21.15.121:0/3264148754 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f205019bb60 con 0x7f203c06cce0 2024-04-05T21:46:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.451+0000 7f20427fc700 1 -- 172.21.15.121:0/3264148754 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f205019bb60 con 0x7f203c06cce0 2024-04-05T21:46:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.453+0000 7f205755e700 1 -- 172.21.15.121:0/3264148754 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f203c06cce0 msgr2=0x7f203c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.453+0000 7f205755e700 1 --2- 172.21.15.121:0/3264148754 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f203c06cce0 0x7f203c06f1a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f204400b5c0 tx=0x7f2044005960 comp rx=0 tx=0).stop 2024-04-05T21:46:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.453+0000 7f205755e700 1 -- 172.21.15.121:0/3264148754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2050069020 msgr2=0x7f2050196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.453+0000 7f205755e700 1 --2- 172.21.15.121:0/3264148754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2050069020 0x7f2050196bb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f204c00d900 tx=0x7f204c00dcc0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.453+0000 7f205755e700 1 -- 172.21.15.121:0/3264148754 shutdown_connections 2024-04-05T21:46:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.453+0000 7f205755e700 1 --2- 172.21.15.121:0/3264148754 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f203c06cce0 0x7f203c06f1a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.453+0000 7f205755e700 1 --2- 172.21.15.121:0/3264148754 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2050068670 0x7f2050196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.453+0000 7f205755e700 1 --2- 172.21.15.121:0/3264148754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2050069020 0x7f2050196bb0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.453+0000 7f205755e700 1 -- 172.21.15.121:0/3264148754 >> 172.21.15.121:0/3264148754 conn(0x7f20500755a0 msgr2=0x7f20500fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.453+0000 7f205755e700 1 -- 172.21.15.121:0/3264148754 shutdown_connections 2024-04-05T21:46:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.453+0000 7f205755e700 1 -- 172.21.15.121:0/3264148754 wait complete. 2024-04-05T21:46:05.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.556+0000 7f1f9c41b700 1 -- 172.21.15.121:0/83002233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f94101790 msgr2=0x7f1f941057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:05.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.556+0000 7f1f9c41b700 1 --2- 172.21.15.121:0/83002233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f94101790 0x7f1f941057e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f1f90009910 tx=0x7f1f90009c20 comp rx=0 tx=0).stop 2024-04-05T21:46:05.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.556+0000 7f1f9c41b700 1 -- 172.21.15.121:0/83002233 shutdown_connections 2024-04-05T21:46:05.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.556+0000 7f1f9c41b700 1 --2- 172.21.15.121:0/83002233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f94101790 0x7f1f941057e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.556+0000 7f1f9c41b700 1 --2- 172.21.15.121:0/83002233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f94100de0 0x7f1f941011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.556+0000 7f1f9c41b700 1 -- 172.21.15.121:0/83002233 >> 172.21.15.121:0/83002233 conn(0x7f1f940fc6b0 msgr2=0x7f1f940fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:05.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.556+0000 7f1f9c41b700 1 -- 172.21.15.121:0/83002233 shutdown_connections 2024-04-05T21:46:05.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.556+0000 7f1f9c41b700 1 -- 172.21.15.121:0/83002233 wait complete. 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.556+0000 7f1f9c41b700 1 Processor -- start 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.556+0000 7f1f9c41b700 1 -- start start 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f9c41b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f94100de0 0x7f1f94196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f9c41b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f94101790 0x7f1f94196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f9c41b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f94197290 con 0x7f1f94101790 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f9c41b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9419b020 con 0x7f1f94100de0 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f999b6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f94101790 0x7f1f94196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f999b6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f94101790 0x7f1f94196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33988/0 (socket says 172.21.15.121:33988) 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f999b6700 1 -- 172.21.15.121:0/2546922129 learned_addr learned my addr 172.21.15.121:0/2546922129 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f9a1b7700 1 --2- 172.21.15.121:0/2546922129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f94100de0 0x7f1f94196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f999b6700 1 -- 172.21.15.121:0/2546922129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f94100de0 msgr2=0x7f1f94196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f999b6700 1 --2- 172.21.15.121:0/2546922129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f94100de0 0x7f1f94196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f999b6700 1 -- 172.21.15.121:0/2546922129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f900095f0 con 0x7f1f94101790 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f999b6700 1 --2- 172.21.15.121:0/2546922129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f94101790 0x7f1f94196bb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1f900092e0 tx=0x7f1f9000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:05.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f8b7fe700 1 -- 172.21.15.121:0/2546922129 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f9001d070 con 0x7f1f94101790 2024-04-05T21:46:05.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f8b7fe700 1 -- 172.21.15.121:0/2546922129 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f90005dc0 con 0x7f1f94101790 2024-04-05T21:46:05.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f8b7fe700 1 -- 172.21.15.121:0/2546922129 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f9000fea0 con 0x7f1f94101790 2024-04-05T21:46:05.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f9c41b700 1 -- 172.21.15.121:0/2546922129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9419b2a0 con 0x7f1f94101790 2024-04-05T21:46:05.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.557+0000 7f1f9c41b700 1 -- 172.21.15.121:0/2546922129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9419b790 con 0x7f1f94101790 2024-04-05T21:46:05.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.558+0000 7f1f9c41b700 1 -- 172.21.15.121:0/2546922129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f941091e0 con 0x7f1f94101790 2024-04-05T21:46:05.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.559+0000 7f1f8b7fe700 1 -- 172.21.15.121:0/2546922129 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1f90003a60 con 0x7f1f94101790 2024-04-05T21:46:05.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.559+0000 7f1f8b7fe700 1 --2- 172.21.15.121:0/2546922129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1f8006cce0 0x7f1f8006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:05.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.559+0000 7f1f8b7fe700 1 -- 172.21.15.121:0/2546922129 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1f9008d6c0 con 0x7f1f94101790 2024-04-05T21:46:05.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.559+0000 7f1f9a1b7700 1 --2- 172.21.15.121:0/2546922129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1f8006cce0 0x7f1f8006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:05.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.560+0000 7f1f9a1b7700 1 --2- 172.21.15.121:0/2546922129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1f8006cce0 0x7f1f8006f1a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1f84005950 tx=0x7f1f8400b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:05.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.564+0000 7f1f8b7fe700 1 -- 172.21.15.121:0/2546922129 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1f900577b0 con 0x7f1f94101790 2024-04-05T21:46:05.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.766+0000 7f1f9c41b700 1 -- 172.21.15.121:0/2546922129 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f9402cd60 con 0x7f1f8006cce0 2024-04-05T21:46:05.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.776+0000 7f1f8b7fe700 1 -- 172.21.15.121:0/2546922129 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1f9402cd60 con 0x7f1f8006cce0 2024-04-05T21:46:05.777 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:46:05.777 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (16m) 10m ago 28m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:46:05.777 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (28m) 10m ago 28m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:46:05.777 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (27m) 10m ago 27m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:46:05.777 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (15m) 10m ago 28m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:46:05.777 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (14m) 10m ago 27m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:46:05.777 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (16m) 10m ago 28m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:46:05.777 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (11m) 10m ago 22m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:46:05.777 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (10m) 10m ago 22m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:46:05.777 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (10m) 10m ago 22m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:46:05.778 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (22m) 10m ago 22m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:46:05.778 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (18m) 10m ago 29m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:46:05.778 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (18m) 10m ago 27m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:46:05.778 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (15m) 10m ago 30m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:46:05.778 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (15m) 10m ago 27m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:46:05.778 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (17m) 10m ago 28m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:46:05.778 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (17m) 10m ago 27m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:46:05.778 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (14m) 10m ago 26m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:46:05.778 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (14m) 10m ago 25m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:46:05.778 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (13m) 10m ago 25m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:46:05.778 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (12m) 10m ago 24m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:46:05.778 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (12m) 10m ago 24m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:46:05.778 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (11m) 10m ago 23m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:46:05.779 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (17m) 10m ago 27m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:46:05.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.778+0000 7f1f9c41b700 1 -- 172.21.15.121:0/2546922129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1f8006cce0 msgr2=0x7f1f8006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:05.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.778+0000 7f1f9c41b700 1 --2- 172.21.15.121:0/2546922129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1f8006cce0 0x7f1f8006f1a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1f84005950 tx=0x7f1f8400b410 comp rx=0 tx=0).stop 2024-04-05T21:46:05.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.778+0000 7f1f9c41b700 1 -- 172.21.15.121:0/2546922129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f94101790 msgr2=0x7f1f94196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:05.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.778+0000 7f1f9c41b700 1 --2- 172.21.15.121:0/2546922129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f94101790 0x7f1f94196bb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1f900092e0 tx=0x7f1f9000bd70 comp rx=0 tx=0).stop 2024-04-05T21:46:05.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.778+0000 7f1f9c41b700 1 -- 172.21.15.121:0/2546922129 shutdown_connections 2024-04-05T21:46:05.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.778+0000 7f1f9c41b700 1 --2- 172.21.15.121:0/2546922129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1f8006cce0 0x7f1f8006f1a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.778+0000 7f1f9c41b700 1 --2- 172.21.15.121:0/2546922129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1f94100de0 0x7f1f94196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.778+0000 7f1f9c41b700 1 --2- 172.21.15.121:0/2546922129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f94101790 0x7f1f94196bb0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.778+0000 7f1f9c41b700 1 -- 172.21.15.121:0/2546922129 >> 172.21.15.121:0/2546922129 conn(0x7f1f940fc6b0 msgr2=0x7f1f940fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:05.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.779+0000 7f1f9c41b700 1 -- 172.21.15.121:0/2546922129 shutdown_connections 2024-04-05T21:46:05.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.779+0000 7f1f9c41b700 1 -- 172.21.15.121:0/2546922129 wait complete. 2024-04-05T21:46:05.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.879+0000 7f55715f9700 1 -- 172.21.15.121:0/2981117763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f556c105a40 msgr2=0x7f556c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:05.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.879+0000 7f55715f9700 1 --2- 172.21.15.121:0/2981117763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f556c105a40 0x7f556c105e20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f555c009910 tx=0x7f555c009c20 comp rx=0 tx=0).stop 2024-04-05T21:46:05.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.879+0000 7f55715f9700 1 -- 172.21.15.121:0/2981117763 shutdown_connections 2024-04-05T21:46:05.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.879+0000 7f55715f9700 1 --2- 172.21.15.121:0/2981117763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f556c0ffa40 0x7f556c0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.879+0000 7f55715f9700 1 --2- 172.21.15.121:0/2981117763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f556c105a40 0x7f556c105e20 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.879+0000 7f55715f9700 1 -- 172.21.15.121:0/2981117763 >> 172.21.15.121:0/2981117763 conn(0x7f556c0fb570 msgr2=0x7f556c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:05.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.879+0000 7f55715f9700 1 -- 172.21.15.121:0/2981117763 shutdown_connections 2024-04-05T21:46:05.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.879+0000 7f55715f9700 1 -- 172.21.15.121:0/2981117763 wait complete. 2024-04-05T21:46:05.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.880+0000 7f55715f9700 1 Processor -- start 2024-04-05T21:46:05.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.880+0000 7f55715f9700 1 -- start start 2024-04-05T21:46:05.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.880+0000 7f55715f9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f556c0ffa40 0x7f556c1952b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:05.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.880+0000 7f55715f9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f556c105a40 0x7f556c1957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:05.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.880+0000 7f55715f9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c195d30 con 0x7f556c105a40 2024-04-05T21:46:05.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.880+0000 7f55715f9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c195ea0 con 0x7f556c0ffa40 2024-04-05T21:46:05.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.881+0000 7f556a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f556c105a40 0x7f556c1957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:05.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.881+0000 7f556a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f556c105a40 0x7f556c1957f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34006/0 (socket says 172.21.15.121:34006) 2024-04-05T21:46:05.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.881+0000 7f556a7fc700 1 -- 172.21.15.121:0/1124668224 learned_addr learned my addr 172.21.15.121:0/1124668224 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:05.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.881+0000 7f556affd700 1 --2- 172.21.15.121:0/1124668224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f556c0ffa40 0x7f556c1952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:05.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.881+0000 7f556a7fc700 1 -- 172.21.15.121:0/1124668224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f556c0ffa40 msgr2=0x7f556c1952b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:05.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.881+0000 7f556a7fc700 1 --2- 172.21.15.121:0/1124668224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f556c0ffa40 0x7f556c1952b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:05.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.881+0000 7f556a7fc700 1 -- 172.21.15.121:0/1124668224 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c0095f0 con 0x7f556c105a40 2024-04-05T21:46:05.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.881+0000 7f556a7fc700 1 --2- 172.21.15.121:0/1124668224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f556c105a40 0x7f556c1957f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f556000d900 tx=0x7f556000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:05.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.881+0000 7f5553fff700 1 -- 172.21.15.121:0/1124668224 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55600041f0 con 0x7f556c105a40 2024-04-05T21:46:05.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.881+0000 7f55715f9700 1 -- 172.21.15.121:0/1124668224 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556c1a2920 con 0x7f556c105a40 2024-04-05T21:46:05.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.881+0000 7f5553fff700 1 -- 172.21.15.121:0/1124668224 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5560004cd0 con 0x7f556c105a40 2024-04-05T21:46:05.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.881+0000 7f5553fff700 1 -- 172.21.15.121:0/1124668224 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5560005660 con 0x7f556c105a40 2024-04-05T21:46:05.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.881+0000 7f55715f9700 1 -- 172.21.15.121:0/1124668224 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556c1a2c80 con 0x7f556c105a40 2024-04-05T21:46:05.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.882+0000 7f55715f9700 1 -- 172.21.15.121:0/1124668224 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556c04ea90 con 0x7f556c105a40 2024-04-05T21:46:05.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.883+0000 7f5553fff700 1 -- 172.21.15.121:0/1124668224 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5560004e40 con 0x7f556c105a40 2024-04-05T21:46:05.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.883+0000 7f5553fff700 1 --2- 172.21.15.121:0/1124668224 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f555406cce0 0x7f555406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:05.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.883+0000 7f5553fff700 1 -- 172.21.15.121:0/1124668224 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f556008c350 con 0x7f556c105a40 2024-04-05T21:46:05.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.883+0000 7f556affd700 1 --2- 172.21.15.121:0/1124668224 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f555406cce0 0x7f555406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:05.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.883+0000 7f556affd700 1 --2- 172.21.15.121:0/1124668224 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f555406cce0 0x7f555406f1a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f555c006010 tx=0x7f555c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:05.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:05.888+0000 7f5553fff700 1 -- 172.21.15.121:0/1124668224 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5560059e50 con 0x7f556c105a40 2024-04-05T21:46:05.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:05 smithi121 ceph-mon[145859]: pgmap v595: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:06.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:05 smithi188 ceph-mon[107793]: pgmap v595: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:06.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.161+0000 7f55715f9700 1 -- 172.21.15.121:0/1124668224 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f556c1a3070 con 0x7f556c105a40 2024-04-05T21:46:06.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.161+0000 7f5553fff700 1 -- 172.21.15.121:0/1124668224 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f55600599e0 con 0x7f556c105a40 2024-04-05T21:46:06.162 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:46:06.162 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:46:06.162 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:46:06.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:46:06.162 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:46:06.162 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:46:06.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:46:06.162 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:46:06.162 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:46:06.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:46:06.163 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:46:06.163 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:46:06.163 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:46:06.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:46:06.163 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:46:06.163 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:46:06.163 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:46:06.163 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:46:06.163 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:46:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.163+0000 7f55715f9700 1 -- 172.21.15.121:0/1124668224 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f555406cce0 msgr2=0x7f555406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.163+0000 7f55715f9700 1 --2- 172.21.15.121:0/1124668224 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f555406cce0 0x7f555406f1a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f555c006010 tx=0x7f555c01a040 comp rx=0 tx=0).stop 2024-04-05T21:46:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.163+0000 7f55715f9700 1 -- 172.21.15.121:0/1124668224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f556c105a40 msgr2=0x7f556c1957f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.163+0000 7f55715f9700 1 --2- 172.21.15.121:0/1124668224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f556c105a40 0x7f556c1957f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f556000d900 tx=0x7f556000dc10 comp rx=0 tx=0).stop 2024-04-05T21:46:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.163+0000 7f55715f9700 1 -- 172.21.15.121:0/1124668224 shutdown_connections 2024-04-05T21:46:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.163+0000 7f55715f9700 1 --2- 172.21.15.121:0/1124668224 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f555406cce0 0x7f555406f1a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.163+0000 7f55715f9700 1 --2- 172.21.15.121:0/1124668224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f556c0ffa40 0x7f556c1952b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.163+0000 7f55715f9700 1 --2- 172.21.15.121:0/1124668224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f556c105a40 0x7f556c1957f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.163+0000 7f55715f9700 1 -- 172.21.15.121:0/1124668224 >> 172.21.15.121:0/1124668224 conn(0x7f556c0fb570 msgr2=0x7f556c0fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.163+0000 7f55715f9700 1 -- 172.21.15.121:0/1124668224 shutdown_connections 2024-04-05T21:46:06.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.163+0000 7f55715f9700 1 -- 172.21.15.121:0/1124668224 wait complete. 2024-04-05T21:46:06.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.265+0000 7fce50480700 1 -- 172.21.15.121:0/4274369661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48068670 msgr2=0x7fce48068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:06.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.265+0000 7fce50480700 1 --2- 172.21.15.121:0/4274369661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48068670 0x7fce48068a50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fce3800b0a0 tx=0x7fce3800b3b0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.265+0000 7fce50480700 1 -- 172.21.15.121:0/4274369661 shutdown_connections 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.265+0000 7fce50480700 1 --2- 172.21.15.121:0/4274369661 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce48069020 0x7fce48105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.265+0000 7fce50480700 1 --2- 172.21.15.121:0/4274369661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48068670 0x7fce48068a50 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.265+0000 7fce50480700 1 -- 172.21.15.121:0/4274369661 >> 172.21.15.121:0/4274369661 conn(0x7fce480755a0 msgr2=0x7fce480759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.265+0000 7fce50480700 1 -- 172.21.15.121:0/4274369661 shutdown_connections 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.265+0000 7fce50480700 1 -- 172.21.15.121:0/4274369661 wait complete. 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.266+0000 7fce50480700 1 Processor -- start 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.266+0000 7fce50480700 1 -- start start 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.266+0000 7fce50480700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce48068670 0x7fce48196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.266+0000 7fce50480700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48069020 0x7fce48196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.266+0000 7fce50480700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce48197290 con 0x7fce48069020 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.266+0000 7fce50480700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4819b020 con 0x7fce48068670 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.266+0000 7fce4da1b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48069020 0x7fce48196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.266+0000 7fce4da1b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48069020 0x7fce48196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34026/0 (socket says 172.21.15.121:34026) 2024-04-05T21:46:06.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.266+0000 7fce4da1b700 1 -- 172.21.15.121:0/1794732354 learned_addr learned my addr 172.21.15.121:0/1794732354 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:06.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.267+0000 7fce4e21c700 1 --2- 172.21.15.121:0/1794732354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce48068670 0x7fce48196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:06.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.267+0000 7fce4da1b700 1 -- 172.21.15.121:0/1794732354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce48068670 msgr2=0x7fce48196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:06.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.267+0000 7fce4da1b700 1 --2- 172.21.15.121:0/1794732354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce48068670 0x7fce48196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.267+0000 7fce4da1b700 1 -- 172.21.15.121:0/1794732354 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce38009d00 con 0x7fce48069020 2024-04-05T21:46:06.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.267+0000 7fce4da1b700 1 --2- 172.21.15.121:0/1794732354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48069020 0x7fce48196bb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00d900 tx=0x7fce3c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:06.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.267+0000 7fce477fe700 1 -- 172.21.15.121:0/1794732354 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce3c009900 con 0x7fce48069020 2024-04-05T21:46:06.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.267+0000 7fce50480700 1 -- 172.21.15.121:0/1794732354 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4819b300 con 0x7fce48069020 2024-04-05T21:46:06.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.267+0000 7fce477fe700 1 -- 172.21.15.121:0/1794732354 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce3c010460 con 0x7fce48069020 2024-04-05T21:46:06.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.267+0000 7fce477fe700 1 -- 172.21.15.121:0/1794732354 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce3c00f5f0 con 0x7fce48069020 2024-04-05T21:46:06.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.267+0000 7fce50480700 1 -- 172.21.15.121:0/1794732354 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce4819b850 con 0x7fce48069020 2024-04-05T21:46:06.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.268+0000 7fce50480700 1 -- 172.21.15.121:0/1794732354 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce48109130 con 0x7fce48069020 2024-04-05T21:46:06.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.268+0000 7fce477fe700 1 -- 172.21.15.121:0/1794732354 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fce3c0105d0 con 0x7fce48069020 2024-04-05T21:46:06.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.269+0000 7fce477fe700 1 --2- 172.21.15.121:0/1794732354 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce3006cce0 0x7fce3006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:06.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.269+0000 7fce477fe700 1 -- 172.21.15.121:0/1794732354 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fce3c08c3c0 con 0x7fce48069020 2024-04-05T21:46:06.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.269+0000 7fce4e21c700 1 --2- 172.21.15.121:0/1794732354 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce3006cce0 0x7fce3006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:06.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.269+0000 7fce4e21c700 1 --2- 172.21.15.121:0/1794732354 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce3006cce0 0x7fce3006f1a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fce3800b830 tx=0x7fce38006000 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:06.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.273+0000 7fce477fe700 1 -- 172.21.15.121:0/1794732354 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fce3c056430 con 0x7fce48069020 2024-04-05T21:46:06.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.508+0000 7fce50480700 1 -- 172.21.15.121:0/1794732354 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fce4804ea90 con 0x7fce48069020 2024-04-05T21:46:06.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.508+0000 7fce477fe700 1 -- 172.21.15.121:0/1794732354 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fce3c059a50 con 0x7fce48069020 2024-04-05T21:46:06.509 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:46:06.509 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:46:06.509 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:46:06.509 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:46:06.509 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:46:06.509 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:46:06.509 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:46:06.509 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:46:06.509 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:46:06.510 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:46:06.510 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:46:06.510 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:46:06.510 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:46:06.510 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:46:06.510 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:46:06.510 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:46:06.510 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:46:06.510 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:46:06.510 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:46:06.510 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:46:06.510 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:46:06.510 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:46:06.511 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:46:06.512 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:46:06.512 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:46:06.512 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:46:06.512 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:46:06.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.510+0000 7fce50480700 1 -- 172.21.15.121:0/1794732354 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce3006cce0 msgr2=0x7fce3006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:06.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.510+0000 7fce50480700 1 --2- 172.21.15.121:0/1794732354 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce3006cce0 0x7fce3006f1a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fce3800b830 tx=0x7fce38006000 comp rx=0 tx=0).stop 2024-04-05T21:46:06.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.510+0000 7fce50480700 1 -- 172.21.15.121:0/1794732354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48069020 msgr2=0x7fce48196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:06.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.510+0000 7fce50480700 1 --2- 172.21.15.121:0/1794732354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48069020 0x7fce48196bb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00d900 tx=0x7fce3c00dcc0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.510+0000 7fce50480700 1 -- 172.21.15.121:0/1794732354 shutdown_connections 2024-04-05T21:46:06.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.510+0000 7fce50480700 1 --2- 172.21.15.121:0/1794732354 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce3006cce0 0x7fce3006f1a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.510+0000 7fce50480700 1 --2- 172.21.15.121:0/1794732354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce48068670 0x7fce48196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.510+0000 7fce50480700 1 --2- 172.21.15.121:0/1794732354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce48069020 0x7fce48196bb0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.510+0000 7fce50480700 1 -- 172.21.15.121:0/1794732354 >> 172.21.15.121:0/1794732354 conn(0x7fce480755a0 msgr2=0x7fce480fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:06.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.511+0000 7fce50480700 1 -- 172.21.15.121:0/1794732354 shutdown_connections 2024-04-05T21:46:06.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.511+0000 7fce50480700 1 -- 172.21.15.121:0/1794732354 wait complete. 2024-04-05T21:46:06.514 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:46:06.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.614+0000 7fa228f21700 1 -- 172.21.15.121:0/4142205963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2240ffbd0 msgr2=0x7fa224100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:06.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.614+0000 7fa228f21700 1 --2- 172.21.15.121:0/4142205963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2240ffbd0 0x7fa224100050 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa214009910 tx=0x7fa214009c20 comp rx=0 tx=0).stop 2024-04-05T21:46:06.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.614+0000 7fa228f21700 1 -- 172.21.15.121:0/4142205963 shutdown_connections 2024-04-05T21:46:06.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.614+0000 7fa228f21700 1 --2- 172.21.15.121:0/4142205963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2240ffbd0 0x7fa224100050 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.614+0000 7fa228f21700 1 --2- 172.21.15.121:0/4142205963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa224105a40 0x7fa224105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.614+0000 7fa228f21700 1 -- 172.21.15.121:0/4142205963 >> 172.21.15.121:0/4142205963 conn(0x7fa2240755a0 msgr2=0x7fa2240759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:06.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.614+0000 7fa228f21700 1 -- 172.21.15.121:0/4142205963 shutdown_connections 2024-04-05T21:46:06.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.614+0000 7fa228f21700 1 -- 172.21.15.121:0/4142205963 wait complete. 2024-04-05T21:46:06.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.615+0000 7fa228f21700 1 Processor -- start 2024-04-05T21:46:06.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.615+0000 7fa228f21700 1 -- start start 2024-04-05T21:46:06.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.615+0000 7fa228f21700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2240ffbd0 0x7fa224197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:06.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.615+0000 7fa228f21700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa224105a40 0x7fa2241979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:06.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.615+0000 7fa228f21700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa224197ff0 con 0x7fa2240ffbd0 2024-04-05T21:46:06.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.615+0000 7fa228f21700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa224198160 con 0x7fa224105a40 2024-04-05T21:46:06.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.615+0000 7fa22259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2240ffbd0 0x7fa224197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:06.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.615+0000 7fa22259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2240ffbd0 0x7fa224197490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34038/0 (socket says 172.21.15.121:34038) 2024-04-05T21:46:06.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.615+0000 7fa22259c700 1 -- 172.21.15.121:0/3368661158 learned_addr learned my addr 172.21.15.121:0/3368661158 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:06.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.615+0000 7fa221d9b700 1 --2- 172.21.15.121:0/3368661158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa224105a40 0x7fa2241979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:06.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.616+0000 7fa22259c700 1 -- 172.21.15.121:0/3368661158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa224105a40 msgr2=0x7fa2241979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:06.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.616+0000 7fa22259c700 1 --2- 172.21.15.121:0/3368661158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa224105a40 0x7fa2241979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.616+0000 7fa22259c700 1 -- 172.21.15.121:0/3368661158 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2140095f0 con 0x7fa2240ffbd0 2024-04-05T21:46:06.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.616+0000 7fa221d9b700 1 --2- 172.21.15.121:0/3368661158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa224105a40 0x7fa2241979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:46:06.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.616+0000 7fa22259c700 1 --2- 172.21.15.121:0/3368661158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2240ffbd0 0x7fa224197490 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa21800c8a0 tx=0x7fa21800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:06.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.616+0000 7fa2137fe700 1 -- 172.21.15.121:0/3368661158 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa21800cea0 con 0x7fa2240ffbd0 2024-04-05T21:46:06.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.616+0000 7fa2137fe700 1 -- 172.21.15.121:0/3368661158 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa218004830 con 0x7fa2240ffbd0 2024-04-05T21:46:06.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.616+0000 7fa228f21700 1 -- 172.21.15.121:0/3368661158 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2241a0500 con 0x7fa2240ffbd0 2024-04-05T21:46:06.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.616+0000 7fa2137fe700 1 -- 172.21.15.121:0/3368661158 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa218005500 con 0x7fa2240ffbd0 2024-04-05T21:46:06.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.616+0000 7fa228f21700 1 -- 172.21.15.121:0/3368661158 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2241a09f0 con 0x7fa2240ffbd0 2024-04-05T21:46:06.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.617+0000 7fa228f21700 1 -- 172.21.15.121:0/3368661158 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa224107fd0 con 0x7fa2240ffbd0 2024-04-05T21:46:06.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.617+0000 7fa2137fe700 1 -- 172.21.15.121:0/3368661158 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa2180049a0 con 0x7fa2240ffbd0 2024-04-05T21:46:06.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.618+0000 7fa2137fe700 1 --2- 172.21.15.121:0/3368661158 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa20c06cce0 0x7fa20c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:06.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.618+0000 7fa2137fe700 1 -- 172.21.15.121:0/3368661158 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa21808c1b0 con 0x7fa2240ffbd0 2024-04-05T21:46:06.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.618+0000 7fa221d9b700 1 --2- 172.21.15.121:0/3368661158 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa20c06cce0 0x7fa20c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:06.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.618+0000 7fa221d9b700 1 --2- 172.21.15.121:0/3368661158 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa20c06cce0 0x7fa20c06f1a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa214006010 tx=0x7fa21400b540 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.622+0000 7fa2137fe700 1 -- 172.21.15.121:0/3368661158 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa2180562a0 con 0x7fa2240ffbd0 2024-04-05T21:46:06.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.833+0000 7fa228f21700 1 -- 172.21.15.121:0/3368661158 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa22402cd80 con 0x7fa20c06cce0 2024-04-05T21:46:06.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.834+0000 7fa2137fe700 1 -- 172.21.15.121:0/3368661158 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa22402cd80 con 0x7fa20c06cce0 2024-04-05T21:46:06.835 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:46:06.835 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:46:06.835 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:46:06.835 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:46:06.835 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:46:06.835 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:46:06.835 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:46:06.835 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:46:06.835 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:46:06.836 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:46:06.836 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:46:06.836 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:46:06.836 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:46:06.836 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:46:06.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.836+0000 7fa228f21700 1 -- 172.21.15.121:0/3368661158 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa20c06cce0 msgr2=0x7fa20c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:06.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.836+0000 7fa228f21700 1 --2- 172.21.15.121:0/3368661158 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa20c06cce0 0x7fa20c06f1a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa214006010 tx=0x7fa21400b540 comp rx=0 tx=0).stop 2024-04-05T21:46:06.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.837+0000 7fa228f21700 1 -- 172.21.15.121:0/3368661158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2240ffbd0 msgr2=0x7fa224197490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:06.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.837+0000 7fa228f21700 1 --2- 172.21.15.121:0/3368661158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2240ffbd0 0x7fa224197490 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa21800c8a0 tx=0x7fa21800cc60 comp rx=0 tx=0).stop 2024-04-05T21:46:06.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.837+0000 7fa228f21700 1 -- 172.21.15.121:0/3368661158 shutdown_connections 2024-04-05T21:46:06.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.837+0000 7fa228f21700 1 --2- 172.21.15.121:0/3368661158 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa20c06cce0 0x7fa20c06f1a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.837+0000 7fa228f21700 1 --2- 172.21.15.121:0/3368661158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2240ffbd0 0x7fa224197490 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.837+0000 7fa228f21700 1 --2- 172.21.15.121:0/3368661158 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa224105a40 0x7fa2241979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.837+0000 7fa228f21700 1 -- 172.21.15.121:0/3368661158 >> 172.21.15.121:0/3368661158 conn(0x7fa2240755a0 msgr2=0x7fa2240fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:06.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.837+0000 7fa228f21700 1 -- 172.21.15.121:0/3368661158 shutdown_connections 2024-04-05T21:46:06.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.837+0000 7fa228f21700 1 -- 172.21.15.121:0/3368661158 wait complete. 2024-04-05T21:46:06.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.940+0000 7fdecaa65700 1 -- 172.21.15.121:0/2860561111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec40ffbd0 msgr2=0x7fdec4100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:06.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.940+0000 7fdecaa65700 1 --2- 172.21.15.121:0/2860561111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec40ffbd0 0x7fdec4100050 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0009910 tx=0x7fdeb0009c20 comp rx=0 tx=0).stop 2024-04-05T21:46:06.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.940+0000 7fdecaa65700 1 -- 172.21.15.121:0/2860561111 shutdown_connections 2024-04-05T21:46:06.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.940+0000 7fdecaa65700 1 --2- 172.21.15.121:0/2860561111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec40ffbd0 0x7fdec4100050 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.940+0000 7fdecaa65700 1 --2- 172.21.15.121:0/2860561111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdec4105a40 0x7fdec4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.940+0000 7fdecaa65700 1 -- 172.21.15.121:0/2860561111 >> 172.21.15.121:0/2860561111 conn(0x7fdec40755a0 msgr2=0x7fdec40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:06.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.940+0000 7fdecaa65700 1 -- 172.21.15.121:0/2860561111 shutdown_connections 2024-04-05T21:46:06.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.940+0000 7fdecaa65700 1 -- 172.21.15.121:0/2860561111 wait complete. 2024-04-05T21:46:06.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.940+0000 7fdecaa65700 1 Processor -- start 2024-04-05T21:46:06.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.940+0000 7fdecaa65700 1 -- start start 2024-04-05T21:46:06.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdecaa65700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdec40ffbd0 0x7fdec419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:06.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdecaa65700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec4105a40 0x7fdec419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:06.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdecaa65700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec419d0e0 con 0x7fdec4105a40 2024-04-05T21:46:06.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdecaa65700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec4196590 con 0x7fdec40ffbd0 2024-04-05T21:46:06.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdec3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec4105a40 0x7fdec419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:06.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdec3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec4105a40 0x7fdec419ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34060/0 (socket says 172.21.15.121:34060) 2024-04-05T21:46:06.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdec3fff700 1 -- 172.21.15.121:0/3487682382 learned_addr learned my addr 172.21.15.121:0/3487682382 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:06.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdec8801700 1 --2- 172.21.15.121:0/3487682382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdec40ffbd0 0x7fdec419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:06.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdec3fff700 1 -- 172.21.15.121:0/3487682382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdec40ffbd0 msgr2=0x7fdec419c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:06.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdec3fff700 1 --2- 172.21.15.121:0/3487682382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdec40ffbd0 0x7fdec419c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:06.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdec3fff700 1 -- 172.21.15.121:0/3487682382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb00095f0 con 0x7fdec4105a40 2024-04-05T21:46:06.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdec3fff700 1 --2- 172.21.15.121:0/3487682382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec4105a40 0x7fdec419ca50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0006010 tx=0x7fdeb000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:06.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdec1ffb700 1 -- 172.21.15.121:0/3487682382 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdeb001d070 con 0x7fdec4105a40 2024-04-05T21:46:06.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdecaa65700 1 -- 172.21.15.121:0/3487682382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec4196810 con 0x7fdec4105a40 2024-04-05T21:46:06.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdec1ffb700 1 -- 172.21.15.121:0/3487682382 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdeb000f460 con 0x7fdec4105a40 2024-04-05T21:46:06.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdec1ffb700 1 -- 172.21.15.121:0/3487682382 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdeb0021620 con 0x7fdec4105a40 2024-04-05T21:46:06.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.941+0000 7fdecaa65700 1 -- 172.21.15.121:0/3487682382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec4196d00 con 0x7fdec4105a40 2024-04-05T21:46:06.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.942+0000 7fdecaa65700 1 -- 172.21.15.121:0/3487682382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec4108fe0 con 0x7fdec4105a40 2024-04-05T21:46:06.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.943+0000 7fdec1ffb700 1 -- 172.21.15.121:0/3487682382 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdeb000fa40 con 0x7fdec4105a40 2024-04-05T21:46:06.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.943+0000 7fdec1ffb700 1 --2- 172.21.15.121:0/3487682382 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdeb406cce0 0x7fdeb406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:06.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.944+0000 7fdec8801700 1 --2- 172.21.15.121:0/3487682382 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdeb406cce0 0x7fdeb406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:06.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.944+0000 7fdec1ffb700 1 -- 172.21.15.121:0/3487682382 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdeb008d790 con 0x7fdec4105a40 2024-04-05T21:46:06.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.944+0000 7fdec8801700 1 --2- 172.21.15.121:0/3487682382 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdeb406cce0 0x7fdeb406f1a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8005fd0 tx=0x7fdeb8005f60 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:06.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:06.948+0000 7fdec1ffb700 1 -- 172.21.15.121:0/3487682382 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdeb005b290 con 0x7fdec4105a40 2024-04-05T21:46:06.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:06 smithi121 ceph-mon[145859]: from='client.34930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:06.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:06 smithi121 ceph-mon[145859]: from='client.34934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:06.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:06 smithi121 ceph-mon[145859]: from='client.34938 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:06.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:06 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1124668224' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:46:06.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:06 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1794732354' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:46:07.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:06 smithi188 ceph-mon[107793]: from='client.34930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:07.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:06 smithi188 ceph-mon[107793]: from='client.34934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:07.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:06 smithi188 ceph-mon[107793]: from='client.34938 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:07.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:06 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1124668224' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:46:07.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:06 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1794732354' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:46:07.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:07.219+0000 7fdecaa65700 1 -- 172.21.15.121:0/3487682382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdec41089c0 con 0x7fdec4105a40 2024-04-05T21:46:07.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:07.220+0000 7fdec1ffb700 1 -- 172.21.15.121:0/3487682382 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fdeb005ae20 con 0x7fdec4105a40 2024-04-05T21:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:46:07.222 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:46:07.223 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:46:07.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:07.222+0000 7fdecaa65700 1 -- 172.21.15.121:0/3487682382 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdeb406cce0 msgr2=0x7fdeb406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:07.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:07.222+0000 7fdecaa65700 1 --2- 172.21.15.121:0/3487682382 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdeb406cce0 0x7fdeb406f1a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8005fd0 tx=0x7fdeb8005f60 comp rx=0 tx=0).stop 2024-04-05T21:46:07.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:07.222+0000 7fdecaa65700 1 -- 172.21.15.121:0/3487682382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec4105a40 msgr2=0x7fdec419ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:07.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:07.222+0000 7fdecaa65700 1 --2- 172.21.15.121:0/3487682382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec4105a40 0x7fdec419ca50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0006010 tx=0x7fdeb000be20 comp rx=0 tx=0).stop 2024-04-05T21:46:07.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:07.222+0000 7fdecaa65700 1 -- 172.21.15.121:0/3487682382 shutdown_connections 2024-04-05T21:46:07.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:07.222+0000 7fdecaa65700 1 --2- 172.21.15.121:0/3487682382 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdeb406cce0 0x7fdeb406f1a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:07.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:07.222+0000 7fdecaa65700 1 --2- 172.21.15.121:0/3487682382 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdec40ffbd0 0x7fdec419c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:07.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:07.222+0000 7fdecaa65700 1 --2- 172.21.15.121:0/3487682382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec4105a40 0x7fdec419ca50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:07.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:07.222+0000 7fdecaa65700 1 -- 172.21.15.121:0/3487682382 >> 172.21.15.121:0/3487682382 conn(0x7fdec40755a0 msgr2=0x7fdec4103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:07.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:07.222+0000 7fdecaa65700 1 -- 172.21.15.121:0/3487682382 shutdown_connections 2024-04-05T21:46:07.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:07.222+0000 7fdecaa65700 1 -- 172.21.15.121:0/3487682382 wait complete. 2024-04-05T21:46:07.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:07 smithi121 ceph-mon[145859]: pgmap v596: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:07 smithi121 ceph-mon[145859]: from='client.34950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:07.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:07 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3487682382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:46:08.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:07 smithi188 ceph-mon[107793]: pgmap v596: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:08.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:07 smithi188 ceph-mon[107793]: from='client.34950 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:08.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:07 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3487682382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:46:09.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:09 smithi121 ceph-mon[145859]: pgmap v597: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:10.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:09 smithi188 ceph-mon[107793]: pgmap v597: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:11.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:11 smithi121 ceph-mon[145859]: pgmap v598: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:12.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:11 smithi188 ceph-mon[107793]: pgmap v598: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:13.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:13 smithi121 ceph-mon[145859]: pgmap v599: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:14.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:13 smithi188 ceph-mon[107793]: pgmap v599: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:15.943 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:15 smithi188 ceph-mon[107793]: pgmap v600: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:15.944 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:46:15.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:15 smithi121 ceph-mon[145859]: pgmap v600: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:15.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:46:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:17 smithi121 ceph-mon[145859]: pgmap v601: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:18.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:17 smithi188 ceph-mon[107793]: pgmap v601: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:19.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:19 smithi121 ceph-mon[145859]: pgmap v602: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:20.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:19 smithi188 ceph-mon[107793]: pgmap v602: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:21.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:21 smithi121 ceph-mon[145859]: pgmap v603: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:22.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:21 smithi188 ceph-mon[107793]: pgmap v603: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:23.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:23 smithi121 ceph-mon[145859]: pgmap v604: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:24.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:23 smithi188 ceph-mon[107793]: pgmap v604: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:25.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:25 smithi121 ceph-mon[145859]: pgmap v605: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:26.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:25 smithi188 ceph-mon[107793]: pgmap v605: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:27.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:27 smithi121 ceph-mon[145859]: pgmap v606: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:28.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:27 smithi188 ceph-mon[107793]: pgmap v606: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:29.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:29 smithi121 ceph-mon[145859]: pgmap v607: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:30.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:29 smithi188 ceph-mon[107793]: pgmap v607: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:30.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:46:31.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:46:31.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:31 smithi121 ceph-mon[145859]: pgmap v608: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:32.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:31 smithi188 ceph-mon[107793]: pgmap v608: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:33.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:33 smithi121 ceph-mon[145859]: pgmap v609: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:34.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:33 smithi188 ceph-mon[107793]: pgmap v609: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:35.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:35 smithi121 ceph-mon[145859]: pgmap v610: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:35 smithi188 ceph-mon[107793]: pgmap v610: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:37.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.336+0000 7f7cc8d20700 1 -- 172.21.15.121:0/1142363109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc4105bb0 msgr2=0x7f7cc4105f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:37.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.336+0000 7f7cc8d20700 1 --2- 172.21.15.121:0/1142363109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc4105bb0 0x7f7cc4105f90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009910 tx=0x7f7cb8009c20 comp rx=0 tx=0).stop 2024-04-05T21:46:37.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.336+0000 7f7cc8d20700 1 -- 172.21.15.121:0/1142363109 shutdown_connections 2024-04-05T21:46:37.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.336+0000 7f7cc8d20700 1 --2- 172.21.15.121:0/1142363109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7cc4100120 0x7f7cc41005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.336+0000 7f7cc8d20700 1 --2- 172.21.15.121:0/1142363109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc4105bb0 0x7f7cc4105f90 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.336+0000 7f7cc8d20700 1 -- 172.21.15.121:0/1142363109 >> 172.21.15.121:0/1142363109 conn(0x7f7cc4075e30 msgr2=0x7f7cc40fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:37.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.336+0000 7f7cc8d20700 1 -- 172.21.15.121:0/1142363109 shutdown_connections 2024-04-05T21:46:37.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.337+0000 7f7cc8d20700 1 -- 172.21.15.121:0/1142363109 wait complete. 2024-04-05T21:46:37.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.337+0000 7f7cc8d20700 1 Processor -- start 2024-04-05T21:46:37.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.338+0000 7f7cc8d20700 1 -- start start 2024-04-05T21:46:37.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.338+0000 7f7cc8d20700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc4100120 0x7f7cc4195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:37.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.338+0000 7f7cc8d20700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7cc4105bb0 0x7f7cc4195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:37.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.338+0000 7f7cc8d20700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc4196270 con 0x7f7cc4100120 2024-04-05T21:46:37.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.338+0000 7f7cc8d20700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc419a000 con 0x7f7cc4105bb0 2024-04-05T21:46:37.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.338+0000 7f7cc259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc4100120 0x7f7cc4195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.338+0000 7f7cc259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc4100120 0x7f7cc4195650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35338/0 (socket says 172.21.15.121:35338) 2024-04-05T21:46:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.338+0000 7f7cc259c700 1 -- 172.21.15.121:0/3234181090 learned_addr learned my addr 172.21.15.121:0/3234181090 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.338+0000 7f7cc259c700 1 -- 172.21.15.121:0/3234181090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7cc4105bb0 msgr2=0x7f7cc4195b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.338+0000 7f7cc259c700 1 --2- 172.21.15.121:0/3234181090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7cc4105bb0 0x7f7cc4195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.338+0000 7f7cc259c700 1 -- 172.21.15.121:0/3234181090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb80095f0 con 0x7f7cc4100120 2024-04-05T21:46:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.338+0000 7f7cc259c700 1 --2- 172.21.15.121:0/3234181090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc4100120 0x7f7cc4195650 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8006010 tx=0x7f7cb800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.339+0000 7f7cb37fe700 1 -- 172.21.15.121:0/3234181090 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cb801f070 con 0x7f7cc4100120 2024-04-05T21:46:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.339+0000 7f7cc8d20700 1 -- 172.21.15.121:0/3234181090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc419a280 con 0x7f7cc4100120 2024-04-05T21:46:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.339+0000 7f7cb37fe700 1 -- 172.21.15.121:0/3234181090 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cb8019610 con 0x7f7cc4100120 2024-04-05T21:46:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.339+0000 7f7cb37fe700 1 -- 172.21.15.121:0/3234181090 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cb800f670 con 0x7f7cc4100120 2024-04-05T21:46:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.339+0000 7f7cc8d20700 1 -- 172.21.15.121:0/3234181090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc419a770 con 0x7f7cc4100120 2024-04-05T21:46:37.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.340+0000 7f7cc8d20700 1 -- 172.21.15.121:0/3234181090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc404ea90 con 0x7f7cc4100120 2024-04-05T21:46:37.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.340+0000 7f7cb37fe700 1 -- 172.21.15.121:0/3234181090 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7cb8019bd0 con 0x7f7cc4100120 2024-04-05T21:46:37.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.341+0000 7f7cb37fe700 1 --2- 172.21.15.121:0/3234181090 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cac06cce0 0x7f7cac06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:37.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.341+0000 7f7cc1d9b700 1 --2- 172.21.15.121:0/3234181090 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cac06cce0 0x7f7cac06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:37.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.341+0000 7f7cb37fe700 1 -- 172.21.15.121:0/3234181090 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7cb808fa10 con 0x7f7cc4100120 2024-04-05T21:46:37.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.341+0000 7f7cc1d9b700 1 --2- 172.21.15.121:0/3234181090 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cac06cce0 0x7f7cac06f1a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4005fd0 tx=0x7f7cb4005f60 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:37.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.346+0000 7f7cb37fe700 1 -- 172.21.15.121:0/3234181090 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7cb8059a80 con 0x7f7cc4100120 2024-04-05T21:46:37.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.559+0000 7f7cc8d20700 1 -- 172.21.15.121:0/3234181090 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7cc419ab10 con 0x7f7cac06cce0 2024-04-05T21:46:37.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.560+0000 7f7cb37fe700 1 -- 172.21.15.121:0/3234181090 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7cc419ab10 con 0x7f7cac06cce0 2024-04-05T21:46:37.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.562+0000 7f7cc8d20700 1 -- 172.21.15.121:0/3234181090 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cac06cce0 msgr2=0x7f7cac06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:37.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.562+0000 7f7cc8d20700 1 --2- 172.21.15.121:0/3234181090 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cac06cce0 0x7f7cac06f1a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4005fd0 tx=0x7f7cb4005f60 comp rx=0 tx=0).stop 2024-04-05T21:46:37.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.562+0000 7f7cc8d20700 1 -- 172.21.15.121:0/3234181090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc4100120 msgr2=0x7f7cc4195650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:37.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.562+0000 7f7cc8d20700 1 --2- 172.21.15.121:0/3234181090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc4100120 0x7f7cc4195650 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8006010 tx=0x7f7cb800be20 comp rx=0 tx=0).stop 2024-04-05T21:46:37.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.562+0000 7f7cc8d20700 1 -- 172.21.15.121:0/3234181090 shutdown_connections 2024-04-05T21:46:37.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.562+0000 7f7cc8d20700 1 --2- 172.21.15.121:0/3234181090 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7cac06cce0 0x7f7cac06f1a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.562+0000 7f7cc8d20700 1 --2- 172.21.15.121:0/3234181090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cc4100120 0x7f7cc4195650 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.562+0000 7f7cc8d20700 1 --2- 172.21.15.121:0/3234181090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7cc4105bb0 0x7f7cc4195b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.562+0000 7f7cc8d20700 1 -- 172.21.15.121:0/3234181090 >> 172.21.15.121:0/3234181090 conn(0x7f7cc4075e30 msgr2=0x7f7cc4103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:37.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.563+0000 7f7cc8d20700 1 -- 172.21.15.121:0/3234181090 shutdown_connections 2024-04-05T21:46:37.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.563+0000 7f7cc8d20700 1 -- 172.21.15.121:0/3234181090 wait complete. 2024-04-05T21:46:37.574 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:46:37.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.666+0000 7f95d83f4700 1 -- 172.21.15.121:0/1718274313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 msgr2=0x7f95d0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:37.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.666+0000 7f95d83f4700 1 --2- 172.21.15.121:0/1718274313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d0105e20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009910 tx=0x7f95c0009c20 comp rx=0 tx=0).stop 2024-04-05T21:46:37.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.666+0000 7f95d83f4700 1 -- 172.21.15.121:0/1718274313 shutdown_connections 2024-04-05T21:46:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.666+0000 7f95d83f4700 1 --2- 172.21.15.121:0/1718274313 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95d00ffbd0 0x7f95d0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.666+0000 7f95d83f4700 1 --2- 172.21.15.121:0/1718274313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d0105e20 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.666+0000 7f95d83f4700 1 -- 172.21.15.121:0/1718274313 >> 172.21.15.121:0/1718274313 conn(0x7f95d00755a0 msgr2=0x7f95d00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.666+0000 7f95d83f4700 1 -- 172.21.15.121:0/1718274313 shutdown_connections 2024-04-05T21:46:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.666+0000 7f95d83f4700 1 -- 172.21.15.121:0/1718274313 wait complete. 2024-04-05T21:46:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d83f4700 1 Processor -- start 2024-04-05T21:46:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d83f4700 1 -- start start 2024-04-05T21:46:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d83f4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95d00ffbd0 0x7f95d0196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d83f4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d0196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d83f4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d0197260 con 0x7f95d0105a40 2024-04-05T21:46:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d83f4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d019aff0 con 0x7f95d00ffbd0 2024-04-05T21:46:37.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d598f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d0196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d598f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d0196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35344/0 (socket says 172.21.15.121:35344) 2024-04-05T21:46:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d598f700 1 -- 172.21.15.121:0/1298319478 learned_addr learned my addr 172.21.15.121:0/1298319478 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d6190700 1 --2- 172.21.15.121:0/1298319478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95d00ffbd0 0x7f95d0196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d598f700 1 -- 172.21.15.121:0/1298319478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95d00ffbd0 msgr2=0x7f95d0196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d598f700 1 --2- 172.21.15.121:0/1298319478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95d00ffbd0 0x7f95d0196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d598f700 1 -- 172.21.15.121:0/1298319478 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c00095f0 con 0x7f95d0105a40 2024-04-05T21:46:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.667+0000 7f95d598f700 1 --2- 172.21.15.121:0/1298319478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d0196b80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f95cc00d8d0 tx=0x7f95cc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.668+0000 7f95c77fe700 1 -- 172.21.15.121:0/1298319478 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95cc0098a0 con 0x7f95d0105a40 2024-04-05T21:46:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.668+0000 7f95c77fe700 1 -- 172.21.15.121:0/1298319478 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95cc010460 con 0x7f95d0105a40 2024-04-05T21:46:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.668+0000 7f95d83f4700 1 -- 172.21.15.121:0/1298319478 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d019b2d0 con 0x7f95d0105a40 2024-04-05T21:46:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.668+0000 7f95c77fe700 1 -- 172.21.15.121:0/1298319478 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95cc009ea0 con 0x7f95d0105a40 2024-04-05T21:46:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.668+0000 7f95d83f4700 1 -- 172.21.15.121:0/1298319478 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d019b820 con 0x7f95d0105a40 2024-04-05T21:46:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.669+0000 7f95d83f4700 1 -- 172.21.15.121:0/1298319478 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d0109050 con 0x7f95d0105a40 2024-04-05T21:46:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.669+0000 7f95c77fe700 1 -- 172.21.15.121:0/1298319478 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f95cc0105d0 con 0x7f95d0105a40 2024-04-05T21:46:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.669+0000 7f95c77fe700 1 --2- 172.21.15.121:0/1298319478 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f95bc06cce0 0x7f95bc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.670+0000 7f95c77fe700 1 -- 172.21.15.121:0/1298319478 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f95cc08c3a0 con 0x7f95d0105a40 2024-04-05T21:46:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.670+0000 7f95d6190700 1 --2- 172.21.15.121:0/1298319478 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f95bc06cce0 0x7f95bc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:37.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.670+0000 7f95d6190700 1 --2- 172.21.15.121:0/1298319478 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f95bc06cce0 0x7f95bc06f1a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f95c0006010 tx=0x7f95c001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:37.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.675+0000 7f95c77fe700 1 -- 172.21.15.121:0/1298319478 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f95cc056490 con 0x7f95d0105a40 2024-04-05T21:46:37.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.891+0000 7f95d83f4700 1 -- 172.21.15.121:0/1298319478 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f95d019bb80 con 0x7f95bc06cce0 2024-04-05T21:46:37.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.892+0000 7f95c77fe700 1 -- 172.21.15.121:0/1298319478 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f95d019bb80 con 0x7f95bc06cce0 2024-04-05T21:46:37.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.894+0000 7f95d83f4700 1 -- 172.21.15.121:0/1298319478 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f95bc06cce0 msgr2=0x7f95bc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:37.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.894+0000 7f95d83f4700 1 --2- 172.21.15.121:0/1298319478 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f95bc06cce0 0x7f95bc06f1a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f95c0006010 tx=0x7f95c001a040 comp rx=0 tx=0).stop 2024-04-05T21:46:37.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.894+0000 7f95d83f4700 1 -- 172.21.15.121:0/1298319478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 msgr2=0x7f95d0196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:37.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.894+0000 7f95d83f4700 1 --2- 172.21.15.121:0/1298319478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d0196b80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f95cc00d8d0 tx=0x7f95cc00dbe0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.894+0000 7f95d83f4700 1 -- 172.21.15.121:0/1298319478 shutdown_connections 2024-04-05T21:46:37.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.894+0000 7f95d83f4700 1 --2- 172.21.15.121:0/1298319478 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f95bc06cce0 0x7f95bc06f1a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.894+0000 7f95d83f4700 1 --2- 172.21.15.121:0/1298319478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95d00ffbd0 0x7f95d0196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.894+0000 7f95d83f4700 1 --2- 172.21.15.121:0/1298319478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d0196b80 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.894+0000 7f95d83f4700 1 -- 172.21.15.121:0/1298319478 >> 172.21.15.121:0/1298319478 conn(0x7f95d00755a0 msgr2=0x7f95d0068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:37.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.894+0000 7f95d83f4700 1 -- 172.21.15.121:0/1298319478 shutdown_connections 2024-04-05T21:46:37.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.894+0000 7f95d83f4700 1 -- 172.21.15.121:0/1298319478 wait complete. 2024-04-05T21:46:37.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:37 smithi121 ceph-mon[145859]: pgmap v611: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:37.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.996+0000 7fbfe56cb700 1 -- 172.21.15.121:0/75768555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe0101780 msgr2=0x7fbfe01057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:37.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.996+0000 7fbfe56cb700 1 --2- 172.21.15.121:0/75768555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe0101780 0x7fbfe01057d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009910 tx=0x7fbfd0009c20 comp rx=0 tx=0).stop 2024-04-05T21:46:37.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.997+0000 7fbfe56cb700 1 -- 172.21.15.121:0/75768555 shutdown_connections 2024-04-05T21:46:37.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.997+0000 7fbfe56cb700 1 --2- 172.21.15.121:0/75768555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe0101780 0x7fbfe01057d0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.997+0000 7fbfe56cb700 1 --2- 172.21.15.121:0/75768555 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfe0100dd0 0x7fbfe01011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:37.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.997+0000 7fbfe56cb700 1 -- 172.21.15.121:0/75768555 >> 172.21.15.121:0/75768555 conn(0x7fbfe00fc640 msgr2=0x7fbfe00fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:37.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.997+0000 7fbfe56cb700 1 -- 172.21.15.121:0/75768555 shutdown_connections 2024-04-05T21:46:37.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.997+0000 7fbfe56cb700 1 -- 172.21.15.121:0/75768555 wait complete. 2024-04-05T21:46:38.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.997+0000 7fbfe56cb700 1 Processor -- start 2024-04-05T21:46:38.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfe56cb700 1 -- start start 2024-04-05T21:46:38.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfe56cb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe0100dd0 0x7fbfe0196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:38.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfe56cb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfe0101780 0x7fbfe0196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfe56cb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe01972d0 con 0x7fbfe0100dd0 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfe56cb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe019a000 con 0x7fbfe0101780 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfdeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe0100dd0 0x7fbfe0196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfdeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe0100dd0 0x7fbfe0196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35358/0 (socket says 172.21.15.121:35358) 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfdeffd700 1 -- 172.21.15.121:0/1411802514 learned_addr learned my addr 172.21.15.121:0/1411802514 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfde7fc700 1 --2- 172.21.15.121:0/1411802514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfe0101780 0x7fbfe0196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfdeffd700 1 -- 172.21.15.121:0/1411802514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfe0101780 msgr2=0x7fbfe0196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfdeffd700 1 --2- 172.21.15.121:0/1411802514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfe0101780 0x7fbfe0196c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfdeffd700 1 -- 172.21.15.121:0/1411802514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd00095f0 con 0x7fbfe0100dd0 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfde7fc700 1 --2- 172.21.15.121:0/1411802514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfe0101780 0x7fbfe0196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.998+0000 7fbfdeffd700 1 --2- 172.21.15.121:0/1411802514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe0100dd0 0x7fbfe0196700 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400d900 tx=0x7fbfd400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.999+0000 7fbfc7fff700 1 -- 172.21.15.121:0/1411802514 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd40041f0 con 0x7fbfe0100dd0 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.999+0000 7fbfe56cb700 1 -- 172.21.15.121:0/1411802514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe019a2e0 con 0x7fbfe0100dd0 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.999+0000 7fbfc7fff700 1 -- 172.21.15.121:0/1411802514 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfd4004cd0 con 0x7fbfe0100dd0 2024-04-05T21:46:38.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.999+0000 7fbfc7fff700 1 -- 172.21.15.121:0/1411802514 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfd40056d0 con 0x7fbfe0100dd0 2024-04-05T21:46:38.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:37.999+0000 7fbfe56cb700 1 -- 172.21.15.121:0/1411802514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe019a830 con 0x7fbfe0100dd0 2024-04-05T21:46:38.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.000+0000 7fbfe56cb700 1 -- 172.21.15.121:0/1411802514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe01091d0 con 0x7fbfe0100dd0 2024-04-05T21:46:38.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.000+0000 7fbfc7fff700 1 -- 172.21.15.121:0/1411802514 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbfd4004e40 con 0x7fbfe0100dd0 2024-04-05T21:46:38.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.000+0000 7fbfc7fff700 1 --2- 172.21.15.121:0/1411802514 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbfc806cc90 0x7fbfc806f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:38.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.001+0000 7fbfc7fff700 1 -- 172.21.15.121:0/1411802514 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbfd408c710 con 0x7fbfe0100dd0 2024-04-05T21:46:38.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.001+0000 7fbfde7fc700 1 --2- 172.21.15.121:0/1411802514 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbfc806cc90 0x7fbfc806f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:38.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.001+0000 7fbfde7fc700 1 --2- 172.21.15.121:0/1411802514 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbfc806cc90 0x7fbfc806f150 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00059d0 tx=0x7fbfd0005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:38.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.006+0000 7fbfc7fff700 1 -- 172.21.15.121:0/1411802514 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbfd4056780 con 0x7fbfe0100dd0 2024-04-05T21:46:38.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:37 smithi188 ceph-mon[107793]: pgmap v611: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:38.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.203+0000 7fbfe56cb700 1 -- 172.21.15.121:0/1411802514 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbfe019aa50 con 0x7fbfc806cc90 2024-04-05T21:46:38.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.213+0000 7fbfc7fff700 1 -- 172.21.15.121:0/1411802514 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fbfe019aa50 con 0x7fbfc806cc90 2024-04-05T21:46:38.215 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:46:38.215 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (17m) 11m ago 28m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:46:38.215 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (29m) 11m ago 29m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:46:38.215 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (28m) 11m ago 28m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:46:38.215 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (15m) 11m ago 29m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:46:38.215 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (15m) 11m ago 27m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:46:38.215 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (17m) 11m ago 28m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:46:38.215 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (11m) 11m ago 22m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:46:38.215 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (11m) 11m ago 22m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (11m) 11m ago 22m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (22m) 11m ago 22m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (19m) 11m ago 30m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (18m) 11m ago 27m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (16m) 11m ago 30m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (15m) 11m ago 27m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (18m) 11m ago 29m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (18m) 11m ago 27m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (15m) 11m ago 26m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (14m) 11m ago 26m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (13m) 11m ago 25m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (13m) 11m ago 25m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (12m) 11m ago 24m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (12m) 11m ago 24m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:46:38.216 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (17m) 11m ago 28m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:46:38.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.215+0000 7fbfe56cb700 1 -- 172.21.15.121:0/1411802514 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbfc806cc90 msgr2=0x7fbfc806f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:38.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.215+0000 7fbfe56cb700 1 --2- 172.21.15.121:0/1411802514 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbfc806cc90 0x7fbfc806f150 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00059d0 tx=0x7fbfd0005960 comp rx=0 tx=0).stop 2024-04-05T21:46:38.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.215+0000 7fbfe56cb700 1 -- 172.21.15.121:0/1411802514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe0100dd0 msgr2=0x7fbfe0196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:38.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.215+0000 7fbfe56cb700 1 --2- 172.21.15.121:0/1411802514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe0100dd0 0x7fbfe0196700 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400d900 tx=0x7fbfd400dcc0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.215+0000 7fbfe56cb700 1 -- 172.21.15.121:0/1411802514 shutdown_connections 2024-04-05T21:46:38.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.215+0000 7fbfe56cb700 1 --2- 172.21.15.121:0/1411802514 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbfc806cc90 0x7fbfc806f150 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.215+0000 7fbfe56cb700 1 --2- 172.21.15.121:0/1411802514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfe0100dd0 0x7fbfe0196700 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.215+0000 7fbfe56cb700 1 --2- 172.21.15.121:0/1411802514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbfe0101780 0x7fbfe0196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.215+0000 7fbfe56cb700 1 -- 172.21.15.121:0/1411802514 >> 172.21.15.121:0/1411802514 conn(0x7fbfe00fc640 msgr2=0x7fbfe00fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:38.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.216+0000 7fbfe56cb700 1 -- 172.21.15.121:0/1411802514 shutdown_connections 2024-04-05T21:46:38.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.216+0000 7fbfe56cb700 1 -- 172.21.15.121:0/1411802514 wait complete. 2024-04-05T21:46:38.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.318+0000 7f6cd66e4700 1 -- 172.21.15.121:0/1828601557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd0100b50 msgr2=0x7f6cd0100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:38.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.318+0000 7f6cd66e4700 1 --2- 172.21.15.121:0/1828601557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd0100b50 0x7f6cd0100fd0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4009910 tx=0x7f6cc4009c20 comp rx=0 tx=0).stop 2024-04-05T21:46:38.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.318+0000 7f6cd66e4700 1 -- 172.21.15.121:0/1828601557 shutdown_connections 2024-04-05T21:46:38.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.318+0000 7f6cd66e4700 1 --2- 172.21.15.121:0/1828601557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd0100b50 0x7f6cd0100fd0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.318+0000 7f6cd66e4700 1 --2- 172.21.15.121:0/1828601557 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cd0106c00 0x7f6cd0106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.318+0000 7f6cd66e4700 1 -- 172.21.15.121:0/1828601557 >> 172.21.15.121:0/1828601557 conn(0x7f6cd00fc640 msgr2=0x7f6cd00fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:38.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.318+0000 7f6cd66e4700 1 -- 172.21.15.121:0/1828601557 shutdown_connections 2024-04-05T21:46:38.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.318+0000 7f6cd66e4700 1 -- 172.21.15.121:0/1828601557 wait complete. 2024-04-05T21:46:38.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.318+0000 7f6cd66e4700 1 Processor -- start 2024-04-05T21:46:38.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.319+0000 7f6cd66e4700 1 -- start start 2024-04-05T21:46:38.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.319+0000 7f6cd66e4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd0100b50 0x7f6cd01965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:38.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.319+0000 7f6cd66e4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cd0106c00 0x7f6cd0196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:38.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.319+0000 7f6cd66e4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd01971e0 con 0x7f6cd0100b50 2024-04-05T21:46:38.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.319+0000 7f6cd66e4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd019af70 con 0x7f6cd0106c00 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.319+0000 7f6ccffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd0100b50 0x7f6cd01965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.319+0000 7f6ccffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd0100b50 0x7f6cd01965c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35380/0 (socket says 172.21.15.121:35380) 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.319+0000 7f6ccffff700 1 -- 172.21.15.121:0/541551524 learned_addr learned my addr 172.21.15.121:0/541551524 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.319+0000 7f6ccf7fe700 1 --2- 172.21.15.121:0/541551524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cd0106c00 0x7f6cd0196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.319+0000 7f6ccffff700 1 -- 172.21.15.121:0/541551524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cd0106c00 msgr2=0x7f6cd0196b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.319+0000 7f6ccffff700 1 --2- 172.21.15.121:0/541551524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cd0106c00 0x7f6cd0196b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.319+0000 7f6ccffff700 1 -- 172.21.15.121:0/541551524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc40095f0 con 0x7f6cd0100b50 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.319+0000 7f6ccffff700 1 --2- 172.21.15.121:0/541551524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd0100b50 0x7f6cd01965c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6cc000d8d0 tx=0x7f6cc000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.320+0000 7f6ccd7fa700 1 -- 172.21.15.121:0/541551524 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cc0009940 con 0x7f6cd0100b50 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.320+0000 7f6cd66e4700 1 -- 172.21.15.121:0/541551524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd019b250 con 0x7f6cd0100b50 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.320+0000 7f6ccd7fa700 1 -- 172.21.15.121:0/541551524 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6cc0010460 con 0x7f6cd0100b50 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.320+0000 7f6ccd7fa700 1 -- 172.21.15.121:0/541551524 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cc000f5f0 con 0x7f6cd0100b50 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.320+0000 7f6cd66e4700 1 -- 172.21.15.121:0/541551524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd019b820 con 0x7f6cd0100b50 2024-04-05T21:46:38.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.321+0000 7f6cd66e4700 1 -- 172.21.15.121:0/541551524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd01090b0 con 0x7f6cd0100b50 2024-04-05T21:46:38.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.321+0000 7f6ccd7fa700 1 -- 172.21.15.121:0/541551524 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6cc00105d0 con 0x7f6cd0100b50 2024-04-05T21:46:38.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.321+0000 7f6ccd7fa700 1 --2- 172.21.15.121:0/541551524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6cb806cce0 0x7f6cb806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:38.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.321+0000 7f6ccd7fa700 1 -- 172.21.15.121:0/541551524 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6cc008c5e0 con 0x7f6cd0100b50 2024-04-05T21:46:38.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.322+0000 7f6ccf7fe700 1 --2- 172.21.15.121:0/541551524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6cb806cce0 0x7f6cb806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:38.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.322+0000 7f6ccf7fe700 1 --2- 172.21.15.121:0/541551524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6cb806cce0 0x7f6cb806f1a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400b5c0 tx=0x7f6cc4005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:38.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.326+0000 7f6ccd7fa700 1 -- 172.21.15.121:0/541551524 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6cc0056650 con 0x7f6cd0100b50 2024-04-05T21:46:38.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.594+0000 7f6cd66e4700 1 -- 172.21.15.121:0/541551524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6cd019bae0 con 0x7f6cd0100b50 2024-04-05T21:46:38.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.594+0000 7f6ccd7fa700 1 -- 172.21.15.121:0/541551524 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f6cc0010880 con 0x7f6cd0100b50 2024-04-05T21:46:38.595 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:46:38.595 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:46:38.595 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:46:38.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:46:38.595 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:46:38.595 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:46:38.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:46:38.595 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:46:38.596 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:46:38.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:46:38.596 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:46:38.596 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:46:38.596 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:46:38.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:46:38.596 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:46:38.596 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:46:38.596 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:46:38.596 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:46:38.596 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:46:38.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.596+0000 7f6cd66e4700 1 -- 172.21.15.121:0/541551524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6cb806cce0 msgr2=0x7f6cb806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:38.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.596+0000 7f6cd66e4700 1 --2- 172.21.15.121:0/541551524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6cb806cce0 0x7f6cb806f1a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400b5c0 tx=0x7f6cc4005fb0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.596+0000 7f6cd66e4700 1 -- 172.21.15.121:0/541551524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd0100b50 msgr2=0x7f6cd01965c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:38.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.596+0000 7f6cd66e4700 1 --2- 172.21.15.121:0/541551524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd0100b50 0x7f6cd01965c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6cc000d8d0 tx=0x7f6cc000dc90 comp rx=0 tx=0).stop 2024-04-05T21:46:38.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.596+0000 7f6cd66e4700 1 -- 172.21.15.121:0/541551524 shutdown_connections 2024-04-05T21:46:38.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.596+0000 7f6cd66e4700 1 --2- 172.21.15.121:0/541551524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6cb806cce0 0x7f6cb806f1a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.596+0000 7f6cd66e4700 1 --2- 172.21.15.121:0/541551524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd0100b50 0x7f6cd01965c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.596+0000 7f6cd66e4700 1 --2- 172.21.15.121:0/541551524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cd0106c00 0x7f6cd0196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.596+0000 7f6cd66e4700 1 -- 172.21.15.121:0/541551524 >> 172.21.15.121:0/541551524 conn(0x7f6cd00fc640 msgr2=0x7f6cd00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:38.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.596+0000 7f6cd66e4700 1 -- 172.21.15.121:0/541551524 shutdown_connections 2024-04-05T21:46:38.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.596+0000 7f6cd66e4700 1 -- 172.21.15.121:0/541551524 wait complete. 2024-04-05T21:46:38.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.701+0000 7f577ca40700 1 -- 172.21.15.121:0/3143513725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780ffee0 msgr2=0x7f5778105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:38.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.701+0000 7f577ca40700 1 --2- 172.21.15.121:0/3143513725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780ffee0 0x7f5778105810 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f576c009910 tx=0x7f576c009c20 comp rx=0 tx=0).stop 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.701+0000 7f577ca40700 1 -- 172.21.15.121:0/3143513725 shutdown_connections 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.701+0000 7f577ca40700 1 --2- 172.21.15.121:0/3143513725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780ffee0 0x7f5778105810 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.701+0000 7f577ca40700 1 --2- 172.21.15.121:0/3143513725 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57780ff530 0x7f57780ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.701+0000 7f577ca40700 1 -- 172.21.15.121:0/3143513725 >> 172.21.15.121:0/3143513725 conn(0x7f5778074d20 msgr2=0x7f5778075130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.701+0000 7f577ca40700 1 -- 172.21.15.121:0/3143513725 shutdown_connections 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.701+0000 7f577ca40700 1 -- 172.21.15.121:0/3143513725 wait complete. 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.701+0000 7f577ca40700 1 Processor -- start 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.701+0000 7f577ca40700 1 -- start start 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f577ca40700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57780ff530 0x7f5778196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f577ca40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780ffee0 0x7f5778196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f577ca40700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5778197290 con 0x7f57780ffee0 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f577ca40700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577819b020 con 0x7f57780ff530 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f5775d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780ffee0 0x7f5778196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f5775d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780ffee0 0x7f5778196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35412/0 (socket says 172.21.15.121:35412) 2024-04-05T21:46:38.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f5775d9b700 1 -- 172.21.15.121:0/1484932409 learned_addr learned my addr 172.21.15.121:0/1484932409 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:38.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f577659c700 1 --2- 172.21.15.121:0/1484932409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57780ff530 0x7f5778196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:38.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f5775d9b700 1 -- 172.21.15.121:0/1484932409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57780ff530 msgr2=0x7f5778196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:38.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f5775d9b700 1 --2- 172.21.15.121:0/1484932409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57780ff530 0x7f5778196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f5775d9b700 1 -- 172.21.15.121:0/1484932409 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c0095f0 con 0x7f57780ffee0 2024-04-05T21:46:38.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f5775d9b700 1 --2- 172.21.15.121:0/1484932409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780ffee0 0x7f5778196bb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f576c0092e0 tx=0x7f576c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:38.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f57677fe700 1 -- 172.21.15.121:0/1484932409 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f576c01d070 con 0x7f57780ffee0 2024-04-05T21:46:38.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f57677fe700 1 -- 172.21.15.121:0/1484932409 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f576c005dc0 con 0x7f57780ffee0 2024-04-05T21:46:38.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f577ca40700 1 -- 172.21.15.121:0/1484932409 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577819b2a0 con 0x7f57780ffee0 2024-04-05T21:46:38.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f57677fe700 1 -- 172.21.15.121:0/1484932409 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f576c00fea0 con 0x7f57780ffee0 2024-04-05T21:46:38.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.702+0000 7f577ca40700 1 -- 172.21.15.121:0/1484932409 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577819b790 con 0x7f57780ffee0 2024-04-05T21:46:38.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.703+0000 7f577ca40700 1 -- 172.21.15.121:0/1484932409 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5778109110 con 0x7f57780ffee0 2024-04-05T21:46:38.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.704+0000 7f57677fe700 1 -- 172.21.15.121:0/1484932409 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f576c003a60 con 0x7f57780ffee0 2024-04-05T21:46:38.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.704+0000 7f57677fe700 1 --2- 172.21.15.121:0/1484932409 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f576006cce0 0x7f576006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:38.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.704+0000 7f57677fe700 1 -- 172.21.15.121:0/1484932409 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f576c08d6c0 con 0x7f57780ffee0 2024-04-05T21:46:38.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.704+0000 7f577659c700 1 --2- 172.21.15.121:0/1484932409 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f576006cce0 0x7f576006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:38.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.705+0000 7f577659c700 1 --2- 172.21.15.121:0/1484932409 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f576006cce0 0x7f576006f1a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5768005950 tx=0x7f576800b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:38.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.709+0000 7f57677fe700 1 -- 172.21.15.121:0/1484932409 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f576c0577b0 con 0x7f57780ffee0 2024-04-05T21:46:38.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.944+0000 7f577ca40700 1 -- 172.21.15.121:0/1484932409 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f577804ea90 con 0x7f57780ffee0 2024-04-05T21:46:38.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.944+0000 7f57677fe700 1 -- 172.21.15.121:0/1484932409 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f576c05add0 con 0x7f57780ffee0 2024-04-05T21:46:38.945 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:46:38.945 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:46:38.945 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:46:38.945 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:46:38.945 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:46:38.945 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:46:38.945 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:46:38.945 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:46:38.945 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:46:38.945 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:46:38.946 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:46:38.946 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:46:38.946 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:46:38.946 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:46:38.946 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:46:38.946 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:46:38.946 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:46:38.946 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:46:38.946 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:46:38.946 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:46:38.947 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:46:38.948 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:46:38.948 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:46:38.948 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:46:38.948 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:46:38.948 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:46:38.948 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:46:38.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.946+0000 7f577ca40700 1 -- 172.21.15.121:0/1484932409 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f576006cce0 msgr2=0x7f576006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:38.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.946+0000 7f577ca40700 1 --2- 172.21.15.121:0/1484932409 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f576006cce0 0x7f576006f1a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5768005950 tx=0x7f576800b410 comp rx=0 tx=0).stop 2024-04-05T21:46:38.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.946+0000 7f577ca40700 1 -- 172.21.15.121:0/1484932409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780ffee0 msgr2=0x7f5778196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:38.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.946+0000 7f577ca40700 1 --2- 172.21.15.121:0/1484932409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780ffee0 0x7f5778196bb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f576c0092e0 tx=0x7f576c00bd70 comp rx=0 tx=0).stop 2024-04-05T21:46:38.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.946+0000 7f577ca40700 1 -- 172.21.15.121:0/1484932409 shutdown_connections 2024-04-05T21:46:38.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.946+0000 7f577ca40700 1 --2- 172.21.15.121:0/1484932409 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f576006cce0 0x7f576006f1a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.946+0000 7f577ca40700 1 --2- 172.21.15.121:0/1484932409 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f57780ff530 0x7f5778196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.946+0000 7f577ca40700 1 --2- 172.21.15.121:0/1484932409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780ffee0 0x7f5778196bb0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:38.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.946+0000 7f577ca40700 1 -- 172.21.15.121:0/1484932409 >> 172.21.15.121:0/1484932409 conn(0x7f5778074d20 msgr2=0x7f57780fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:38.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.946+0000 7f577ca40700 1 -- 172.21.15.121:0/1484932409 shutdown_connections 2024-04-05T21:46:38.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:38.946+0000 7f577ca40700 1 -- 172.21.15.121:0/1484932409 wait complete. 2024-04-05T21:46:38.950 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:46:38.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:38 smithi121 ceph-mon[145859]: from='client.34958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:38.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:38 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/541551524' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:46:39.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:38 smithi188 ceph-mon[107793]: from='client.34958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:39.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:38 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/541551524' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:46:39.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.049+0000 7f30c80d6700 1 -- 172.21.15.121:0/865947008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c0101790 msgr2=0x7f30c01057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:39.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.049+0000 7f30c80d6700 1 --2- 172.21.15.121:0/865947008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c0101790 0x7f30c01057e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f30b0009910 tx=0x7f30b0009c20 comp rx=0 tx=0).stop 2024-04-05T21:46:39.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.049+0000 7f30c80d6700 1 -- 172.21.15.121:0/865947008 shutdown_connections 2024-04-05T21:46:39.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.049+0000 7f30c80d6700 1 --2- 172.21.15.121:0/865947008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c0101790 0x7f30c01057e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:39.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.049+0000 7f30c80d6700 1 --2- 172.21.15.121:0/865947008 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30c0100de0 0x7f30c01011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:39.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.049+0000 7f30c80d6700 1 -- 172.21.15.121:0/865947008 >> 172.21.15.121:0/865947008 conn(0x7f30c00fc670 msgr2=0x7f30c00fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:39.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.050+0000 7f30c80d6700 1 -- 172.21.15.121:0/865947008 shutdown_connections 2024-04-05T21:46:39.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.050+0000 7f30c80d6700 1 -- 172.21.15.121:0/865947008 wait complete. 2024-04-05T21:46:39.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.050+0000 7f30c80d6700 1 Processor -- start 2024-04-05T21:46:39.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.050+0000 7f30c80d6700 1 -- start start 2024-04-05T21:46:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.050+0000 7f30c80d6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c0100de0 0x7f30c0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.050+0000 7f30c80d6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30c0101790 0x7f30c00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.050+0000 7f30c80d6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0073b10 con 0x7f30c0100de0 2024-04-05T21:46:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.050+0000 7f30c80d6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0073c50 con 0x7f30c0101790 2024-04-05T21:46:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.050+0000 7f30c5e72700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c0100de0 0x7f30c0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.050+0000 7f30c5e72700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c0100de0 0x7f30c0073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35426/0 (socket says 172.21.15.121:35426) 2024-04-05T21:46:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.050+0000 7f30c5e72700 1 -- 172.21.15.121:0/2589833152 learned_addr learned my addr 172.21.15.121:0/2589833152 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.051+0000 7f30c5671700 1 --2- 172.21.15.121:0/2589833152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30c0101790 0x7f30c00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.051+0000 7f30c5e72700 1 -- 172.21.15.121:0/2589833152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30c0101790 msgr2=0x7f30c00735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.051+0000 7f30c5e72700 1 --2- 172.21.15.121:0/2589833152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30c0101790 0x7f30c00735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.051+0000 7f30c5e72700 1 -- 172.21.15.121:0/2589833152 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b00095f0 con 0x7f30c0100de0 2024-04-05T21:46:39.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.051+0000 7f30c5671700 1 --2- 172.21.15.121:0/2589833152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30c0101790 0x7f30c00735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:46:39.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.051+0000 7f30c5e72700 1 --2- 172.21.15.121:0/2589833152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c0100de0 0x7f30c0073090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f30bc00d900 tx=0x7f30bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:39.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.051+0000 7f30b6ffd700 1 -- 172.21.15.121:0/2589833152 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30bc0041f0 con 0x7f30c0100de0 2024-04-05T21:46:39.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.051+0000 7f30c80d6700 1 -- 172.21.15.121:0/2589833152 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c01a28a0 con 0x7f30c0100de0 2024-04-05T21:46:39.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.051+0000 7f30b6ffd700 1 -- 172.21.15.121:0/2589833152 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30bc004cd0 con 0x7f30c0100de0 2024-04-05T21:46:39.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.051+0000 7f30b6ffd700 1 -- 172.21.15.121:0/2589833152 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30bc0056d0 con 0x7f30c0100de0 2024-04-05T21:46:39.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.051+0000 7f30c80d6700 1 -- 172.21.15.121:0/2589833152 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c01a2cb0 con 0x7f30c0100de0 2024-04-05T21:46:39.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.052+0000 7f30c80d6700 1 -- 172.21.15.121:0/2589833152 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c01091e0 con 0x7f30c0100de0 2024-04-05T21:46:39.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.052+0000 7f30b6ffd700 1 -- 172.21.15.121:0/2589833152 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f30bc004e40 con 0x7f30c0100de0 2024-04-05T21:46:39.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.053+0000 7f30b6ffd700 1 --2- 172.21.15.121:0/2589833152 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f30ac06cc90 0x7f30ac06f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:39.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.053+0000 7f30b6ffd700 1 -- 172.21.15.121:0/2589833152 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f30bc08c710 con 0x7f30c0100de0 2024-04-05T21:46:39.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.053+0000 7f30c5671700 1 --2- 172.21.15.121:0/2589833152 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f30ac06cc90 0x7f30ac06f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:39.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.053+0000 7f30c5671700 1 --2- 172.21.15.121:0/2589833152 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f30ac06cc90 0x7f30ac06f150 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f30b00059d0 tx=0x7f30b0005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:39.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.057+0000 7f30b6ffd700 1 -- 172.21.15.121:0/2589833152 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f30bc056780 con 0x7f30c0100de0 2024-04-05T21:46:39.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.267+0000 7f30c80d6700 1 -- 172.21.15.121:0/2589833152 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f30c002cdd0 con 0x7f30ac06cc90 2024-04-05T21:46:39.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.269+0000 7f30b6ffd700 1 -- 172.21.15.121:0/2589833152 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f30c002cdd0 con 0x7f30ac06cc90 2024-04-05T21:46:39.269 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:46:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:46:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:46:39.269 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:46:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:46:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:46:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:46:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:46:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:46:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:46:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:46:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:46:39.270 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:46:39.270 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:46:39.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.271+0000 7f30c80d6700 1 -- 172.21.15.121:0/2589833152 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f30ac06cc90 msgr2=0x7f30ac06f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:39.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.271+0000 7f30c80d6700 1 --2- 172.21.15.121:0/2589833152 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f30ac06cc90 0x7f30ac06f150 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f30b00059d0 tx=0x7f30b0005960 comp rx=0 tx=0).stop 2024-04-05T21:46:39.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.271+0000 7f30c80d6700 1 -- 172.21.15.121:0/2589833152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c0100de0 msgr2=0x7f30c0073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:39.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.271+0000 7f30c80d6700 1 --2- 172.21.15.121:0/2589833152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c0100de0 0x7f30c0073090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f30bc00d900 tx=0x7f30bc00dcc0 comp rx=0 tx=0).stop 2024-04-05T21:46:39.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.271+0000 7f30c80d6700 1 -- 172.21.15.121:0/2589833152 shutdown_connections 2024-04-05T21:46:39.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.271+0000 7f30c80d6700 1 --2- 172.21.15.121:0/2589833152 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f30ac06cc90 0x7f30ac06f150 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:39.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.271+0000 7f30c80d6700 1 --2- 172.21.15.121:0/2589833152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c0100de0 0x7f30c0073090 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:39.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.271+0000 7f30c80d6700 1 --2- 172.21.15.121:0/2589833152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f30c0101790 0x7f30c00735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:39.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.271+0000 7f30c80d6700 1 -- 172.21.15.121:0/2589833152 >> 172.21.15.121:0/2589833152 conn(0x7f30c00fc670 msgr2=0x7f30c00fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:39.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.271+0000 7f30c80d6700 1 -- 172.21.15.121:0/2589833152 shutdown_connections 2024-04-05T21:46:39.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.271+0000 7f30c80d6700 1 -- 172.21.15.121:0/2589833152 wait complete. 2024-04-05T21:46:39.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.375+0000 7ff1c2974700 1 -- 172.21.15.121:0/486938354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1bc105a40 msgr2=0x7ff1bc105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:39.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.375+0000 7ff1c2974700 1 --2- 172.21.15.121:0/486938354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1bc105a40 0x7ff1bc105e20 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac009910 tx=0x7ff1ac009c20 comp rx=0 tx=0).stop 2024-04-05T21:46:39.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.375+0000 7ff1c2974700 1 -- 172.21.15.121:0/486938354 shutdown_connections 2024-04-05T21:46:39.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.375+0000 7ff1c2974700 1 --2- 172.21.15.121:0/486938354 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1bc0ffbd0 0x7ff1bc100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:39.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.375+0000 7ff1c2974700 1 --2- 172.21.15.121:0/486938354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1bc105a40 0x7ff1bc105e20 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:39.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.375+0000 7ff1c2974700 1 -- 172.21.15.121:0/486938354 >> 172.21.15.121:0/486938354 conn(0x7ff1bc0755a0 msgr2=0x7ff1bc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:39.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.375+0000 7ff1c2974700 1 -- 172.21.15.121:0/486938354 shutdown_connections 2024-04-05T21:46:39.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.375+0000 7ff1c2974700 1 -- 172.21.15.121:0/486938354 wait complete. 2024-04-05T21:46:39.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.375+0000 7ff1c2974700 1 Processor -- start 2024-04-05T21:46:39.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.376+0000 7ff1c2974700 1 -- start start 2024-04-05T21:46:39.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.376+0000 7ff1c2974700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1bc0ffbd0 0x7ff1bc196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:39.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.376+0000 7ff1c2974700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1bc105a40 0x7ff1bc196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:39.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.376+0000 7ff1c2974700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc197290 con 0x7ff1bc0ffbd0 2024-04-05T21:46:39.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.376+0000 7ff1c2974700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc19b020 con 0x7ff1bc105a40 2024-04-05T21:46:39.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.376+0000 7ff1bbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1bc0ffbd0 0x7ff1bc196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:39.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.376+0000 7ff1bbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1bc0ffbd0 0x7ff1bc196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35446/0 (socket says 172.21.15.121:35446) 2024-04-05T21:46:39.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.376+0000 7ff1bbfff700 1 -- 172.21.15.121:0/693500918 learned_addr learned my addr 172.21.15.121:0/693500918 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:46:39.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.376+0000 7ff1bbfff700 1 -- 172.21.15.121:0/693500918 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1bc105a40 msgr2=0x7ff1bc196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:39.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.376+0000 7ff1bbfff700 1 --2- 172.21.15.121:0/693500918 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1bc105a40 0x7ff1bc196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:39.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.376+0000 7ff1bbfff700 1 -- 172.21.15.121:0/693500918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac0095f0 con 0x7ff1bc0ffbd0 2024-04-05T21:46:39.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.376+0000 7ff1bbfff700 1 --2- 172.21.15.121:0/693500918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1bc0ffbd0 0x7ff1bc196670 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac006010 tx=0x7ff1ac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:39.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.377+0000 7ff1b97fa700 1 -- 172.21.15.121:0/693500918 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1ac01d070 con 0x7ff1bc0ffbd0 2024-04-05T21:46:39.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.377+0000 7ff1b97fa700 1 -- 172.21.15.121:0/693500918 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1ac00f460 con 0x7ff1bc0ffbd0 2024-04-05T21:46:39.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.377+0000 7ff1c2974700 1 -- 172.21.15.121:0/693500918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1bc19b2a0 con 0x7ff1bc0ffbd0 2024-04-05T21:46:39.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.377+0000 7ff1b97fa700 1 -- 172.21.15.121:0/693500918 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1ac021640 con 0x7ff1bc0ffbd0 2024-04-05T21:46:39.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.377+0000 7ff1c2974700 1 -- 172.21.15.121:0/693500918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc19b790 con 0x7ff1bc0ffbd0 2024-04-05T21:46:39.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.378+0000 7ff1c2974700 1 -- 172.21.15.121:0/693500918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc04ea90 con 0x7ff1bc0ffbd0 2024-04-05T21:46:39.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.378+0000 7ff1b97fa700 1 -- 172.21.15.121:0/693500918 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff1ac00fa20 con 0x7ff1bc0ffbd0 2024-04-05T21:46:39.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.378+0000 7ff1b97fa700 1 --2- 172.21.15.121:0/693500918 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1a806cce0 0x7ff1a806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:46:39.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.378+0000 7ff1b97fa700 1 -- 172.21.15.121:0/693500918 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff1ac08d8c0 con 0x7ff1bc0ffbd0 2024-04-05T21:46:39.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.379+0000 7ff1bb7fe700 1 --2- 172.21.15.121:0/693500918 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1a806cce0 0x7ff1a806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:46:39.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.379+0000 7ff1bb7fe700 1 --2- 172.21.15.121:0/693500918 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1a806cce0 0x7ff1a806f1a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc197c90 tx=0x7ff1a4005c80 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:46:39.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.383+0000 7ff1b97fa700 1 -- 172.21.15.121:0/693500918 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff1ac057930 con 0x7ff1bc0ffbd0 2024-04-05T21:46:39.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.655+0000 7ff1c2974700 1 -- 172.21.15.121:0/693500918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff1bc1979d0 con 0x7ff1bc0ffbd0 2024-04-05T21:46:39.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.656+0000 7ff1b97fa700 1 -- 172.21.15.121:0/693500918 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ff1ac05af50 con 0x7ff1bc0ffbd0 2024-04-05T21:46:39.656 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:46:39.656 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:46:39.656 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:46:39.657 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:46:39.657 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:46:39.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.658+0000 7ff1c2974700 1 -- 172.21.15.121:0/693500918 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1a806cce0 msgr2=0x7ff1a806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:39.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.658+0000 7ff1c2974700 1 --2- 172.21.15.121:0/693500918 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1a806cce0 0x7ff1a806f1a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc197c90 tx=0x7ff1a4005c80 comp rx=0 tx=0).stop 2024-04-05T21:46:39.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.658+0000 7ff1c2974700 1 -- 172.21.15.121:0/693500918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1bc0ffbd0 msgr2=0x7ff1bc196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:46:39.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.658+0000 7ff1c2974700 1 --2- 172.21.15.121:0/693500918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1bc0ffbd0 0x7ff1bc196670 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac006010 tx=0x7ff1ac00be20 comp rx=0 tx=0).stop 2024-04-05T21:46:39.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.658+0000 7ff1c2974700 1 -- 172.21.15.121:0/693500918 shutdown_connections 2024-04-05T21:46:39.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.658+0000 7ff1c2974700 1 --2- 172.21.15.121:0/693500918 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1a806cce0 0x7ff1a806f1a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:39.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.658+0000 7ff1c2974700 1 --2- 172.21.15.121:0/693500918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1bc0ffbd0 0x7ff1bc196670 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.658+0000 7ff1c2974700 1 --2- 172.21.15.121:0/693500918 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1bc105a40 0x7ff1bc196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:46:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.658+0000 7ff1c2974700 1 -- 172.21.15.121:0/693500918 >> 172.21.15.121:0/693500918 conn(0x7ff1bc0755a0 msgr2=0x7ff1bc1037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:46:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.658+0000 7ff1c2974700 1 -- 172.21.15.121:0/693500918 shutdown_connections 2024-04-05T21:46:39.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:46:39.658+0000 7ff1c2974700 1 -- 172.21.15.121:0/693500918 wait complete. 2024-04-05T21:46:39.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:39 smithi121 ceph-mon[145859]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:39.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:39 smithi121 ceph-mon[145859]: from='client.34966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:39.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:39 smithi121 ceph-mon[145859]: pgmap v612: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:39.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:39 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1484932409' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:46:39.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:39 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/693500918' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:46:40.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:39 smithi188 ceph-mon[107793]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:40.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:39 smithi188 ceph-mon[107793]: from='client.34966 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:40.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:39 smithi188 ceph-mon[107793]: pgmap v612: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:40.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:39 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1484932409' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:46:40.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:39 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/693500918' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:46:40.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:40 smithi121 ceph-mon[145859]: from='client.34978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:41.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:40 smithi188 ceph-mon[107793]: from='client.34978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:46:41.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:41 smithi121 ceph-mon[145859]: pgmap v613: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:42.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:41 smithi188 ceph-mon[107793]: pgmap v613: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:43.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:43 smithi121 ceph-mon[145859]: pgmap v614: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:44.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:43 smithi188 ceph-mon[107793]: pgmap v614: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:45 smithi121 ceph-mon[145859]: pgmap v615: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:46:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:45 smithi188 ceph-mon[107793]: pgmap v615: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:46:47.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:47 smithi121 ceph-mon[145859]: pgmap v616: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:48.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:47 smithi188 ceph-mon[107793]: pgmap v616: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:49.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:49 smithi121 ceph-mon[145859]: pgmap v617: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:50.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:49 smithi188 ceph-mon[107793]: pgmap v617: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:51.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:51 smithi121 ceph-mon[145859]: pgmap v618: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:52.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:51 smithi188 ceph-mon[107793]: pgmap v618: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:53.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:53 smithi121 ceph-mon[145859]: pgmap v619: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:54.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:53 smithi188 ceph-mon[107793]: pgmap v619: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:55.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:55 smithi121 ceph-mon[145859]: pgmap v620: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:56.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:55 smithi188 ceph-mon[107793]: pgmap v620: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:57.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:57 smithi121 ceph-mon[145859]: pgmap v621: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:58.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:57 smithi188 ceph-mon[107793]: pgmap v621: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:46:59.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:46:59 smithi121 ceph-mon[145859]: pgmap v622: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:00.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:46:59 smithi188 ceph-mon[107793]: pgmap v622: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:00.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:47:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:47:01.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:01 smithi121 ceph-mon[145859]: pgmap v623: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:02.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:01 smithi188 ceph-mon[107793]: pgmap v623: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:03.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:03 smithi121 ceph-mon[145859]: pgmap v624: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:04.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:03 smithi188 ceph-mon[107793]: pgmap v624: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:05.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:05 smithi121 ceph-mon[145859]: pgmap v625: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:06.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:05 smithi188 ceph-mon[107793]: pgmap v625: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:07.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:07 smithi121 ceph-mon[145859]: pgmap v626: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:08.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:07 smithi188 ceph-mon[107793]: pgmap v626: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:09.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.773+0000 7f7d1f532700 1 -- 172.21.15.121:0/3821953397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d18101790 msgr2=0x7f7d181057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:09.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.773+0000 7f7d1f532700 1 --2- 172.21.15.121:0/3821953397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d18101790 0x7f7d181057e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009910 tx=0x7f7d08009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:09.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.774+0000 7f7d1f532700 1 -- 172.21.15.121:0/3821953397 shutdown_connections 2024-04-05T21:47:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.774+0000 7f7d1f532700 1 --2- 172.21.15.121:0/3821953397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d18101790 0x7f7d181057e0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.774+0000 7f7d1f532700 1 --2- 172.21.15.121:0/3821953397 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d18100de0 0x7f7d181011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.774+0000 7f7d1f532700 1 -- 172.21.15.121:0/3821953397 >> 172.21.15.121:0/3821953397 conn(0x7f7d180fc670 msgr2=0x7f7d180fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.774+0000 7f7d1f532700 1 -- 172.21.15.121:0/3821953397 shutdown_connections 2024-04-05T21:47:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.774+0000 7f7d1f532700 1 -- 172.21.15.121:0/3821953397 wait complete. 2024-04-05T21:47:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.774+0000 7f7d1f532700 1 Processor -- start 2024-04-05T21:47:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.775+0000 7f7d1f532700 1 -- start start 2024-04-05T21:47:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.775+0000 7f7d1f532700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d18100de0 0x7f7d181966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.775+0000 7f7d1f532700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d18101790 0x7f7d18196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.775+0000 7f7d1f532700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d181972a0 con 0x7f7d18101790 2024-04-05T21:47:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.775+0000 7f7d1f532700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d18199fd0 con 0x7f7d18100de0 2024-04-05T21:47:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.775+0000 7f7d1cacd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d18101790 0x7f7d18196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:09.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.775+0000 7f7d1cacd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d18101790 0x7f7d18196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60004/0 (socket says 172.21.15.121:60004) 2024-04-05T21:47:09.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.775+0000 7f7d1cacd700 1 -- 172.21.15.121:0/1476976493 learned_addr learned my addr 172.21.15.121:0/1476976493 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:09.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.775+0000 7f7d1d2ce700 1 --2- 172.21.15.121:0/1476976493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d18100de0 0x7f7d181966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:09.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.775+0000 7f7d1cacd700 1 -- 172.21.15.121:0/1476976493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d18100de0 msgr2=0x7f7d181966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:09.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.775+0000 7f7d1cacd700 1 --2- 172.21.15.121:0/1476976493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d18100de0 0x7f7d181966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:09.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.775+0000 7f7d1cacd700 1 -- 172.21.15.121:0/1476976493 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d080095f0 con 0x7f7d18101790 2024-04-05T21:47:09.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.775+0000 7f7d1cacd700 1 --2- 172.21.15.121:0/1476976493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d18101790 0x7f7d18196c10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7d080092e0 tx=0x7f7d0800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:09.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.776+0000 7f7d0e7fc700 1 -- 172.21.15.121:0/1476976493 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d0801d070 con 0x7f7d18101790 2024-04-05T21:47:09.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.776+0000 7f7d0e7fc700 1 -- 172.21.15.121:0/1476976493 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d08005dc0 con 0x7f7d18101790 2024-04-05T21:47:09.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.776+0000 7f7d1f532700 1 -- 172.21.15.121:0/1476976493 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d1819a250 con 0x7f7d18101790 2024-04-05T21:47:09.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.776+0000 7f7d0e7fc700 1 -- 172.21.15.121:0/1476976493 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d0800fea0 con 0x7f7d18101790 2024-04-05T21:47:09.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.776+0000 7f7d1f532700 1 -- 172.21.15.121:0/1476976493 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d1819a740 con 0x7f7d18101790 2024-04-05T21:47:09.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.777+0000 7f7d1f532700 1 -- 172.21.15.121:0/1476976493 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d181091e0 con 0x7f7d18101790 2024-04-05T21:47:09.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.777+0000 7f7d0e7fc700 1 -- 172.21.15.121:0/1476976493 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7d08003a60 con 0x7f7d18101790 2024-04-05T21:47:09.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.778+0000 7f7d0e7fc700 1 --2- 172.21.15.121:0/1476976493 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7d0406cce0 0x7f7d0406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:09.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.778+0000 7f7d0e7fc700 1 -- 172.21.15.121:0/1476976493 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7d0808d6c0 con 0x7f7d18101790 2024-04-05T21:47:09.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.778+0000 7f7d1d2ce700 1 --2- 172.21.15.121:0/1476976493 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7d0406cce0 0x7f7d0406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:09.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.778+0000 7f7d1d2ce700 1 --2- 172.21.15.121:0/1476976493 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7d0406cce0 0x7f7d0406f1a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7d14005950 tx=0x7f7d1400b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:09.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.782+0000 7f7d0e7fc700 1 -- 172.21.15.121:0/1476976493 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7d080577b0 con 0x7f7d18101790 2024-04-05T21:47:09.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.992+0000 7f7d1f532700 1 -- 172.21.15.121:0/1476976493 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d1802cd30 con 0x7f7d0406cce0 2024-04-05T21:47:09.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.994+0000 7f7d0e7fc700 1 -- 172.21.15.121:0/1476976493 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f7d1802cd30 con 0x7f7d0406cce0 2024-04-05T21:47:09.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.996+0000 7f7d1f532700 1 -- 172.21.15.121:0/1476976493 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7d0406cce0 msgr2=0x7f7d0406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:09.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.996+0000 7f7d1f532700 1 --2- 172.21.15.121:0/1476976493 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7d0406cce0 0x7f7d0406f1a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7d14005950 tx=0x7f7d1400b410 comp rx=0 tx=0).stop 2024-04-05T21:47:09.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.996+0000 7f7d1f532700 1 -- 172.21.15.121:0/1476976493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d18101790 msgr2=0x7f7d18196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:09.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.996+0000 7f7d1f532700 1 --2- 172.21.15.121:0/1476976493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d18101790 0x7f7d18196c10 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7d080092e0 tx=0x7f7d0800bd70 comp rx=0 tx=0).stop 2024-04-05T21:47:09.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.996+0000 7f7d1f532700 1 -- 172.21.15.121:0/1476976493 shutdown_connections 2024-04-05T21:47:09.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.996+0000 7f7d1f532700 1 --2- 172.21.15.121:0/1476976493 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7d0406cce0 0x7f7d0406f1a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:09.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.996+0000 7f7d1f532700 1 --2- 172.21.15.121:0/1476976493 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d18100de0 0x7f7d181966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:09.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.996+0000 7f7d1f532700 1 --2- 172.21.15.121:0/1476976493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d18101790 0x7f7d18196c10 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:09.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.996+0000 7f7d1f532700 1 -- 172.21.15.121:0/1476976493 >> 172.21.15.121:0/1476976493 conn(0x7f7d180fc670 msgr2=0x7f7d180fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:09.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.996+0000 7f7d1f532700 1 -- 172.21.15.121:0/1476976493 shutdown_connections 2024-04-05T21:47:09.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:09.996+0000 7f7d1f532700 1 -- 172.21.15.121:0/1476976493 wait complete. 2024-04-05T21:47:10.006 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:47:10.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:09 smithi188 ceph-mon[107793]: pgmap v627: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.099+0000 7f8308d18700 1 -- 172.21.15.121:0/3246795048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83040739d0 msgr2=0x7f830410cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.099+0000 7f8308d18700 1 --2- 172.21.15.121:0/3246795048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83040739d0 0x7f830410cf30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f82f8009910 tx=0x7f82f8009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.099+0000 7f8308d18700 1 -- 172.21.15.121:0/3246795048 shutdown_connections 2024-04-05T21:47:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.099+0000 7f8308d18700 1 --2- 172.21.15.121:0/3246795048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83040739d0 0x7f830410cf30 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.099+0000 7f8308d18700 1 --2- 172.21.15.121:0/3246795048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83040730b0 0x7f8304073490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.099+0000 7f8308d18700 1 -- 172.21.15.121:0/3246795048 >> 172.21.15.121:0/3246795048 conn(0x7f83040fc6c0 msgr2=0x7f83040feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:10.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.099+0000 7f8308d18700 1 -- 172.21.15.121:0/3246795048 shutdown_connections 2024-04-05T21:47:10.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.099+0000 7f8308d18700 1 -- 172.21.15.121:0/3246795048 wait complete. 2024-04-05T21:47:10.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.100+0000 7f8308d18700 1 Processor -- start 2024-04-05T21:47:10.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.100+0000 7f8308d18700 1 -- start start 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.100+0000 7f8308d18700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83040730b0 0x7f830419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.100+0000 7f8308d18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83040739d0 0x7f830419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.100+0000 7f8308d18700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830419f1e0 con 0x7f83040739d0 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.100+0000 7f8308d18700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8304198690 con 0x7f83040730b0 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.100+0000 7f8301d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83040739d0 0x7f830419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.101+0000 7f8301d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83040739d0 0x7f830419eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60036/0 (socket says 172.21.15.121:60036) 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.101+0000 7f8301d9b700 1 -- 172.21.15.121:0/120347396 learned_addr learned my addr 172.21.15.121:0/120347396 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.101+0000 7f830259c700 1 --2- 172.21.15.121:0/120347396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83040730b0 0x7f830419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.101+0000 7f8301d9b700 1 -- 172.21.15.121:0/120347396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83040730b0 msgr2=0x7f830419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.101+0000 7f8301d9b700 1 --2- 172.21.15.121:0/120347396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83040730b0 0x7f830419e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.101+0000 7f8301d9b700 1 -- 172.21.15.121:0/120347396 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f80095f0 con 0x7f83040739d0 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.101+0000 7f8301d9b700 1 --2- 172.21.15.121:0/120347396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83040739d0 0x7f830419eb50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f82f8006010 tx=0x7f82f800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.101+0000 7f82f37fe700 1 -- 172.21.15.121:0/120347396 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82f801d070 con 0x7f83040739d0 2024-04-05T21:47:10.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.101+0000 7f8308d18700 1 -- 172.21.15.121:0/120347396 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8304198910 con 0x7f83040739d0 2024-04-05T21:47:10.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.101+0000 7f82f37fe700 1 -- 172.21.15.121:0/120347396 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82f800f460 con 0x7f83040739d0 2024-04-05T21:47:10.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.101+0000 7f82f37fe700 1 -- 172.21.15.121:0/120347396 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82f8021620 con 0x7f83040739d0 2024-04-05T21:47:10.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.101+0000 7f8308d18700 1 -- 172.21.15.121:0/120347396 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8304198e00 con 0x7f83040739d0 2024-04-05T21:47:10.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.102+0000 7f8308d18700 1 -- 172.21.15.121:0/120347396 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830410a630 con 0x7f83040739d0 2024-04-05T21:47:10.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.103+0000 7f82f37fe700 1 -- 172.21.15.121:0/120347396 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f82f800fa40 con 0x7f83040739d0 2024-04-05T21:47:10.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.103+0000 7f82f37fe700 1 --2- 172.21.15.121:0/120347396 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:10.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.103+0000 7f82f37fe700 1 -- 172.21.15.121:0/120347396 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f82f808d790 con 0x7f83040739d0 2024-04-05T21:47:10.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.103+0000 7f830259c700 1 --2- 172.21.15.121:0/120347396 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:10.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.103+0000 7f830259c700 1 --2- 172.21.15.121:0/120347396 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f82f4005950 tx=0x7f82f40058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:10.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.107+0000 7f82f37fe700 1 -- 172.21.15.121:0/120347396 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f82f805b290 con 0x7f83040739d0 2024-04-05T21:47:10.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:09 smithi121 ceph-mon[145859]: pgmap v627: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:10.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.318+0000 7f8308d18700 1 -- 172.21.15.121:0/120347396 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f830402cd80 con 0x7f82ec06cce0 2024-04-05T21:47:10.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.320+0000 7f82f37fe700 1 -- 172.21.15.121:0/120347396 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f830402cd80 con 0x7f82ec06cce0 2024-04-05T21:47:10.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.322+0000 7f8308d18700 1 -- 172.21.15.121:0/120347396 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 msgr2=0x7f82ec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:10.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.322+0000 7f8308d18700 1 --2- 172.21.15.121:0/120347396 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f82f4005950 tx=0x7f82f40058e0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.322+0000 7f8308d18700 1 -- 172.21.15.121:0/120347396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83040739d0 msgr2=0x7f830419eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:10.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.322+0000 7f8308d18700 1 --2- 172.21.15.121:0/120347396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83040739d0 0x7f830419eb50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f82f8006010 tx=0x7f82f800be20 comp rx=0 tx=0).stop 2024-04-05T21:47:10.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.322+0000 7f8308d18700 1 -- 172.21.15.121:0/120347396 shutdown_connections 2024-04-05T21:47:10.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.322+0000 7f8308d18700 1 --2- 172.21.15.121:0/120347396 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82ec06cce0 0x7f82ec06f1a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.322+0000 7f8308d18700 1 --2- 172.21.15.121:0/120347396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83040730b0 0x7f830419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.322+0000 7f8308d18700 1 --2- 172.21.15.121:0/120347396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83040739d0 0x7f830419eb50 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.322+0000 7f8308d18700 1 -- 172.21.15.121:0/120347396 >> 172.21.15.121:0/120347396 conn(0x7f83040fc6c0 msgr2=0x7f8304107770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:10.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.322+0000 7f8308d18700 1 -- 172.21.15.121:0/120347396 shutdown_connections 2024-04-05T21:47:10.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.322+0000 7f8308d18700 1 -- 172.21.15.121:0/120347396 wait complete. 2024-04-05T21:47:10.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.424+0000 7f8165fe7700 1 -- 172.21.15.121:0/2443601095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8160105a50 msgr2=0x7f8160105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:10.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.424+0000 7f8165fe7700 1 --2- 172.21.15.121:0/2443601095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8160105a50 0x7f8160105e30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8154009910 tx=0x7f8154009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:10.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.424+0000 7f8165fe7700 1 -- 172.21.15.121:0/2443601095 shutdown_connections 2024-04-05T21:47:10.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.424+0000 7f8165fe7700 1 --2- 172.21.15.121:0/2443601095 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81600ffa50 0x7f81600ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.424+0000 7f8165fe7700 1 --2- 172.21.15.121:0/2443601095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8160105a50 0x7f8160105e30 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.424+0000 7f8165fe7700 1 -- 172.21.15.121:0/2443601095 >> 172.21.15.121:0/2443601095 conn(0x7f81600fb540 msgr2=0x7f81600fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:10.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.425+0000 7f8165fe7700 1 -- 172.21.15.121:0/2443601095 shutdown_connections 2024-04-05T21:47:10.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.425+0000 7f8165fe7700 1 -- 172.21.15.121:0/2443601095 wait complete. 2024-04-05T21:47:10.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.425+0000 7f8165fe7700 1 Processor -- start 2024-04-05T21:47:10.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.425+0000 7f8165fe7700 1 -- start start 2024-04-05T21:47:10.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.425+0000 7f8165fe7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81600ffa50 0x7f816019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.425+0000 7f8165fe7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8160105a50 0x7f816019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.425+0000 7f8165fe7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816019f200 con 0x7f81600ffa50 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.425+0000 7f8165fe7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81601986b0 con 0x7f8160105a50 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.426+0000 7f815f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81600ffa50 0x7f816019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.426+0000 7f815f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81600ffa50 0x7f816019e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60042/0 (socket says 172.21.15.121:60042) 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.426+0000 7f815f7fe700 1 -- 172.21.15.121:0/2512510914 learned_addr learned my addr 172.21.15.121:0/2512510914 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.426+0000 7f815f7fe700 1 -- 172.21.15.121:0/2512510914 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8160105a50 msgr2=0x7f816019eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.426+0000 7f815f7fe700 1 --2- 172.21.15.121:0/2512510914 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8160105a50 0x7f816019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.426+0000 7f815f7fe700 1 -- 172.21.15.121:0/2512510914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81540095f0 con 0x7f81600ffa50 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.426+0000 7f815f7fe700 1 --2- 172.21.15.121:0/2512510914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81600ffa50 0x7f816019e630 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8154006010 tx=0x7f815400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.426+0000 7f815cff9700 1 -- 172.21.15.121:0/2512510914 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f815401d070 con 0x7f81600ffa50 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.426+0000 7f8165fe7700 1 -- 172.21.15.121:0/2512510914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8160198930 con 0x7f81600ffa50 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.426+0000 7f815cff9700 1 -- 172.21.15.121:0/2512510914 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f815400f460 con 0x7f81600ffa50 2024-04-05T21:47:10.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.426+0000 7f815cff9700 1 -- 172.21.15.121:0/2512510914 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8154021600 con 0x7f81600ffa50 2024-04-05T21:47:10.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.426+0000 7f8165fe7700 1 -- 172.21.15.121:0/2512510914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8160198e20 con 0x7f81600ffa50 2024-04-05T21:47:10.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.427+0000 7f8165fe7700 1 -- 172.21.15.121:0/2512510914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f816004ea90 con 0x7f81600ffa50 2024-04-05T21:47:10.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.428+0000 7f815cff9700 1 -- 172.21.15.121:0/2512510914 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f815400f5d0 con 0x7f81600ffa50 2024-04-05T21:47:10.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.428+0000 7f815cff9700 1 --2- 172.21.15.121:0/2512510914 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f814806cce0 0x7f814806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:10.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.428+0000 7f815cff9700 1 -- 172.21.15.121:0/2512510914 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f815408d850 con 0x7f81600ffa50 2024-04-05T21:47:10.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.428+0000 7f815effd700 1 --2- 172.21.15.121:0/2512510914 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f814806cce0 0x7f814806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:10.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.428+0000 7f815effd700 1 --2- 172.21.15.121:0/2512510914 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f814806cce0 0x7f814806f1a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8150005950 tx=0x7f81500058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:10.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.433+0000 7f815cff9700 1 -- 172.21.15.121:0/2512510914 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f815405b350 con 0x7f81600ffa50 2024-04-05T21:47:10.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.643+0000 7f8165fe7700 1 -- 172.21.15.121:0/2512510914 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f81601991c0 con 0x7f814806cce0 2024-04-05T21:47:10.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.653+0000 7f815cff9700 1 -- 172.21.15.121:0/2512510914 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f81601991c0 con 0x7f814806cce0 2024-04-05T21:47:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:47:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (18m) 11m ago 29m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:47:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (29m) 11m ago 29m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:47:10.654 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (28m) 11m ago 28m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (16m) 11m ago 29m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (16m) 11m ago 28m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (17m) 11m ago 29m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (12m) 11m ago 23m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (12m) 11m ago 23m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (11m) 11m ago 23m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (23m) 11m ago 23m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (20m) 11m ago 31m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (19m) 11m ago 28m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (16m) 11m ago 31m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (16m) 11m ago 28m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (18m) 11m ago 29m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (18m) 11m ago 28m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:47:10.655 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (15m) 11m ago 27m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:47:10.656 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (15m) 11m ago 26m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:47:10.656 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (14m) 11m ago 26m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:47:10.656 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (13m) 11m ago 25m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:47:10.656 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (13m) 11m ago 25m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:47:10.656 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (12m) 11m ago 25m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:47:10.656 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (18m) 11m ago 28m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:47:10.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.655+0000 7f8165fe7700 1 -- 172.21.15.121:0/2512510914 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f814806cce0 msgr2=0x7f814806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:10.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.655+0000 7f8165fe7700 1 --2- 172.21.15.121:0/2512510914 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f814806cce0 0x7f814806f1a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8150005950 tx=0x7f81500058e0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.655+0000 7f8165fe7700 1 -- 172.21.15.121:0/2512510914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81600ffa50 msgr2=0x7f816019e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:10.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.655+0000 7f8165fe7700 1 --2- 172.21.15.121:0/2512510914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81600ffa50 0x7f816019e630 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8154006010 tx=0x7f815400be20 comp rx=0 tx=0).stop 2024-04-05T21:47:10.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.655+0000 7f8165fe7700 1 -- 172.21.15.121:0/2512510914 shutdown_connections 2024-04-05T21:47:10.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.655+0000 7f8165fe7700 1 --2- 172.21.15.121:0/2512510914 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f814806cce0 0x7f814806f1a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.655+0000 7f8165fe7700 1 --2- 172.21.15.121:0/2512510914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81600ffa50 0x7f816019e630 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.655+0000 7f8165fe7700 1 --2- 172.21.15.121:0/2512510914 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8160105a50 0x7f816019eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.655+0000 7f8165fe7700 1 -- 172.21.15.121:0/2512510914 >> 172.21.15.121:0/2512510914 conn(0x7f81600fb540 msgr2=0x7f8160068d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:10.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.656+0000 7f8165fe7700 1 -- 172.21.15.121:0/2512510914 shutdown_connections 2024-04-05T21:47:10.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.656+0000 7f8165fe7700 1 -- 172.21.15.121:0/2512510914 wait complete. 2024-04-05T21:47:10.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.759+0000 7faa7ada6700 1 -- 172.21.15.121:0/615389649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa740ffa40 msgr2=0x7faa740ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:10.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.759+0000 7faa7ada6700 1 --2- 172.21.15.121:0/615389649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa740ffa40 0x7faa740ffec0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7faa60009910 tx=0x7faa60009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:10.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.759+0000 7faa7ada6700 1 -- 172.21.15.121:0/615389649 shutdown_connections 2024-04-05T21:47:10.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.759+0000 7faa7ada6700 1 --2- 172.21.15.121:0/615389649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa740ffa40 0x7faa740ffec0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.759+0000 7faa7ada6700 1 --2- 172.21.15.121:0/615389649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa74105a40 0x7faa74105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.759+0000 7faa7ada6700 1 -- 172.21.15.121:0/615389649 >> 172.21.15.121:0/615389649 conn(0x7faa740fb570 msgr2=0x7faa740fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.759+0000 7faa7ada6700 1 -- 172.21.15.121:0/615389649 shutdown_connections 2024-04-05T21:47:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.759+0000 7faa7ada6700 1 -- 172.21.15.121:0/615389649 wait complete. 2024-04-05T21:47:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.759+0000 7faa7ada6700 1 Processor -- start 2024-04-05T21:47:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.759+0000 7faa7ada6700 1 -- start start 2024-04-05T21:47:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa7ada6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa740ffa40 0x7faa741974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa7ada6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa74105a40 0x7faa741979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa7ada6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa74198010 con 0x7faa74105a40 2024-04-05T21:47:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa7ada6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa74198150 con 0x7faa740ffa40 2024-04-05T21:47:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa73fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa74105a40 0x7faa741979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa73fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa74105a40 0x7faa741979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60070/0 (socket says 172.21.15.121:60070) 2024-04-05T21:47:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa73fff700 1 -- 172.21.15.121:0/3081358467 learned_addr learned my addr 172.21.15.121:0/3081358467 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa78b42700 1 --2- 172.21.15.121:0/3081358467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa740ffa40 0x7faa741974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa73fff700 1 -- 172.21.15.121:0/3081358467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa740ffa40 msgr2=0x7faa741974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:10.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa73fff700 1 --2- 172.21.15.121:0/3081358467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa740ffa40 0x7faa741974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:10.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa73fff700 1 -- 172.21.15.121:0/3081358467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa600095f0 con 0x7faa74105a40 2024-04-05T21:47:10.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa78b42700 1 --2- 172.21.15.121:0/3081358467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa740ffa40 0x7faa741974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:47:10.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa73fff700 1 --2- 172.21.15.121:0/3081358467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa74105a40 0x7faa741979f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7faa60006010 tx=0x7faa6000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:10.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa71ffb700 1 -- 172.21.15.121:0/3081358467 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa6001d070 con 0x7faa74105a40 2024-04-05T21:47:10.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa71ffb700 1 -- 172.21.15.121:0/3081358467 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa6000f460 con 0x7faa74105a40 2024-04-05T21:47:10.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa7ada6700 1 -- 172.21.15.121:0/3081358467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa741a0520 con 0x7faa74105a40 2024-04-05T21:47:10.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.760+0000 7faa71ffb700 1 -- 172.21.15.121:0/3081358467 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa60021640 con 0x7faa74105a40 2024-04-05T21:47:10.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.761+0000 7faa7ada6700 1 -- 172.21.15.121:0/3081358467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa741a0930 con 0x7faa74105a40 2024-04-05T21:47:10.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.762+0000 7faa7ada6700 1 -- 172.21.15.121:0/3081358467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa74107fd0 con 0x7faa74105a40 2024-04-05T21:47:10.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.762+0000 7faa71ffb700 1 -- 172.21.15.121:0/3081358467 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7faa6000f9e0 con 0x7faa74105a40 2024-04-05T21:47:10.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.762+0000 7faa71ffb700 1 --2- 172.21.15.121:0/3081358467 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faa6406cce0 0x7faa6406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:10.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.762+0000 7faa71ffb700 1 -- 172.21.15.121:0/3081358467 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7faa6008d900 con 0x7faa74105a40 2024-04-05T21:47:10.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.762+0000 7faa78b42700 1 --2- 172.21.15.121:0/3081358467 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faa6406cce0 0x7faa6406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:10.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.763+0000 7faa78b42700 1 --2- 172.21.15.121:0/3081358467 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faa6406cce0 0x7faa6406f1a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7faa68005950 tx=0x7faa68005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:10.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:10.767+0000 7faa71ffb700 1 -- 172.21.15.121:0/3081358467 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faa60057970 con 0x7faa74105a40 2024-04-05T21:47:11.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.040+0000 7faa7ada6700 1 -- 172.21.15.121:0/3081358467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faa7402cc70 con 0x7faa74105a40 2024-04-05T21:47:11.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.040+0000 7faa71ffb700 1 -- 172.21.15.121:0/3081358467 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7faa6005af90 con 0x7faa74105a40 2024-04-05T21:47:11.041 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:47:11.041 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:47:11.041 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:47:11.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:47:11.041 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:47:11.041 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:47:11.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:47:11.042 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:47:11.042 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:47:11.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:47:11.042 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:47:11.042 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:47:11.042 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:47:11.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:47:11.042 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:47:11.042 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:47:11.042 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:47:11.042 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:47:11.042 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:47:11.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.042+0000 7faa7ada6700 1 -- 172.21.15.121:0/3081358467 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faa6406cce0 msgr2=0x7faa6406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:11.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.042+0000 7faa7ada6700 1 --2- 172.21.15.121:0/3081358467 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faa6406cce0 0x7faa6406f1a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7faa68005950 tx=0x7faa68005f50 comp rx=0 tx=0).stop 2024-04-05T21:47:11.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.042+0000 7faa7ada6700 1 -- 172.21.15.121:0/3081358467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa74105a40 msgr2=0x7faa741979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:11.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.042+0000 7faa7ada6700 1 --2- 172.21.15.121:0/3081358467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa74105a40 0x7faa741979f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7faa60006010 tx=0x7faa6000be20 comp rx=0 tx=0).stop 2024-04-05T21:47:11.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.042+0000 7faa7ada6700 1 -- 172.21.15.121:0/3081358467 shutdown_connections 2024-04-05T21:47:11.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.042+0000 7faa7ada6700 1 --2- 172.21.15.121:0/3081358467 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faa6406cce0 0x7faa6406f1a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.042+0000 7faa7ada6700 1 --2- 172.21.15.121:0/3081358467 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa740ffa40 0x7faa741974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.042+0000 7faa7ada6700 1 --2- 172.21.15.121:0/3081358467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa74105a40 0x7faa741979f0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.042+0000 7faa7ada6700 1 -- 172.21.15.121:0/3081358467 >> 172.21.15.121:0/3081358467 conn(0x7faa740fb570 msgr2=0x7faa740fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:11.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.043+0000 7faa7ada6700 1 -- 172.21.15.121:0/3081358467 shutdown_connections 2024-04-05T21:47:11.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.043+0000 7faa7ada6700 1 -- 172.21.15.121:0/3081358467 wait complete. 2024-04-05T21:47:11.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.144+0000 7f3fc636c700 1 -- 172.21.15.121:0/3622714384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fc0100de0 msgr2=0x7f3fc01011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:11.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.144+0000 7f3fc636c700 1 --2- 172.21.15.121:0/3622714384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fc0100de0 0x7f3fc01011c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0009910 tx=0x7f3fb0009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:11.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.144+0000 7f3fc636c700 1 -- 172.21.15.121:0/3622714384 shutdown_connections 2024-04-05T21:47:11.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.144+0000 7f3fc636c700 1 --2- 172.21.15.121:0/3622714384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fc0101790 0x7f3fc01057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.144+0000 7f3fc636c700 1 --2- 172.21.15.121:0/3622714384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fc0100de0 0x7f3fc01011c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.144+0000 7f3fc636c700 1 -- 172.21.15.121:0/3622714384 >> 172.21.15.121:0/3622714384 conn(0x7f3fc00fc6b0 msgr2=0x7f3fc00fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:11.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.144+0000 7f3fc636c700 1 -- 172.21.15.121:0/3622714384 shutdown_connections 2024-04-05T21:47:11.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.144+0000 7f3fc636c700 1 -- 172.21.15.121:0/3622714384 wait complete. 2024-04-05T21:47:11.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.145+0000 7f3fc636c700 1 Processor -- start 2024-04-05T21:47:11.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.145+0000 7f3fc636c700 1 -- start start 2024-04-05T21:47:11.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fc636c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fc0100de0 0x7f3fc01966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:11.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fc636c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fc0101790 0x7f3fc0196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:11.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fc636c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc01971f0 con 0x7f3fc0101790 2024-04-05T21:47:11.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fc636c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc0197360 con 0x7f3fc0100de0 2024-04-05T21:47:11.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fbf7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fc0101790 0x7f3fc0196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:11.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fbf7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fc0101790 0x7f3fc0196bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60086/0 (socket says 172.21.15.121:60086) 2024-04-05T21:47:11.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fbf7fe700 1 -- 172.21.15.121:0/3885376437 learned_addr learned my addr 172.21.15.121:0/3885376437 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:11.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fbffff700 1 --2- 172.21.15.121:0/3885376437 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fc0100de0 0x7f3fc01966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:11.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fbf7fe700 1 -- 172.21.15.121:0/3885376437 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fc0100de0 msgr2=0x7f3fc01966b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:11.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fbf7fe700 1 --2- 172.21.15.121:0/3885376437 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fc0100de0 0x7f3fc01966b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fbf7fe700 1 -- 172.21.15.121:0/3885376437 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb00095f0 con 0x7f3fc0101790 2024-04-05T21:47:11.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fbf7fe700 1 --2- 172.21.15.121:0/3885376437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fc0101790 0x7f3fc0196bf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400c960 tx=0x7f3fb400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:11.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fbd7fa700 1 -- 172.21.15.121:0/3885376437 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fb4007a70 con 0x7f3fc0101790 2024-04-05T21:47:11.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fc636c700 1 -- 172.21.15.121:0/3885376437 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc019a100 con 0x7f3fc0101790 2024-04-05T21:47:11.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fbd7fa700 1 -- 172.21.15.121:0/3885376437 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fb4007bd0 con 0x7f3fc0101790 2024-04-05T21:47:11.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fbd7fa700 1 -- 172.21.15.121:0/3885376437 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fb4018630 con 0x7f3fc0101790 2024-04-05T21:47:11.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.146+0000 7f3fc636c700 1 -- 172.21.15.121:0/3885376437 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc019a650 con 0x7f3fc0101790 2024-04-05T21:47:11.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.147+0000 7f3fc636c700 1 -- 172.21.15.121:0/3885376437 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc0109180 con 0x7f3fc0101790 2024-04-05T21:47:11.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.148+0000 7f3fbd7fa700 1 -- 172.21.15.121:0/3885376437 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3fb400f450 con 0x7f3fc0101790 2024-04-05T21:47:11.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.148+0000 7f3fbd7fa700 1 --2- 172.21.15.121:0/3885376437 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3fa806cce0 0x7f3fa806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:11.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.148+0000 7f3fbffff700 1 --2- 172.21.15.121:0/3885376437 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3fa806cce0 0x7f3fa806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:11.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.148+0000 7f3fbd7fa700 1 -- 172.21.15.121:0/3885376437 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3fb408c570 con 0x7f3fc0101790 2024-04-05T21:47:11.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.149+0000 7f3fbffff700 1 --2- 172.21.15.121:0/3885376437 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3fa806cce0 0x7f3fa806f1a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006010 tx=0x7f3fb001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:11.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.153+0000 7f3fbd7fa700 1 -- 172.21.15.121:0/3885376437 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3fb40565e0 con 0x7f3fc0101790 2024-04-05T21:47:11.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.385+0000 7f3fc636c700 1 -- 172.21.15.121:0/3885376437 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3fc004ea90 con 0x7f3fc0101790 2024-04-05T21:47:11.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.386+0000 7f3fbd7fa700 1 -- 172.21.15.121:0/3885376437 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f3fb4059c00 con 0x7f3fc0101790 2024-04-05T21:47:11.386 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:47:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:47:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:47:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:47:11.387 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:47:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:47:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:47:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:47:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:47:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:47:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:47:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:47:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:47:11.387 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:47:11.388 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:47:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:47:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:47:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:47:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:47:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:47:11.389 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:47:11.389 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:47:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:47:11.389 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:47:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:47:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:47:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:47:11.389 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:47:11.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.388+0000 7f3fc636c700 1 -- 172.21.15.121:0/3885376437 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3fa806cce0 msgr2=0x7f3fa806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:11.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.388+0000 7f3fc636c700 1 --2- 172.21.15.121:0/3885376437 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3fa806cce0 0x7f3fa806f1a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006010 tx=0x7f3fb001a040 comp rx=0 tx=0).stop 2024-04-05T21:47:11.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.388+0000 7f3fc636c700 1 -- 172.21.15.121:0/3885376437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fc0101790 msgr2=0x7f3fc0196bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:11.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.388+0000 7f3fc636c700 1 --2- 172.21.15.121:0/3885376437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fc0101790 0x7f3fc0196bf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400c960 tx=0x7f3fb400cc70 comp rx=0 tx=0).stop 2024-04-05T21:47:11.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.388+0000 7f3fc636c700 1 -- 172.21.15.121:0/3885376437 shutdown_connections 2024-04-05T21:47:11.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.388+0000 7f3fc636c700 1 --2- 172.21.15.121:0/3885376437 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3fa806cce0 0x7f3fa806f1a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.388+0000 7f3fc636c700 1 --2- 172.21.15.121:0/3885376437 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3fc0100de0 0x7f3fc01966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.388+0000 7f3fc636c700 1 --2- 172.21.15.121:0/3885376437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fc0101790 0x7f3fc0196bf0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.388+0000 7f3fc636c700 1 -- 172.21.15.121:0/3885376437 >> 172.21.15.121:0/3885376437 conn(0x7f3fc00fc6b0 msgr2=0x7f3fc00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:11.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.388+0000 7f3fc636c700 1 -- 172.21.15.121:0/3885376437 shutdown_connections 2024-04-05T21:47:11.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.388+0000 7f3fc636c700 1 -- 172.21.15.121:0/3885376437 wait complete. 2024-04-05T21:47:11.391 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:47:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.491+0000 7efc2367f700 1 -- 172.21.15.121:0/2372422947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c101770 msgr2=0x7efc1c1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.491+0000 7efc2367f700 1 --2- 172.21.15.121:0/2372422947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c101770 0x7efc1c1057c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7efc18009910 tx=0x7efc18009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.491+0000 7efc2367f700 1 -- 172.21.15.121:0/2372422947 shutdown_connections 2024-04-05T21:47:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.491+0000 7efc2367f700 1 --2- 172.21.15.121:0/2372422947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c101770 0x7efc1c1057c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.491+0000 7efc2367f700 1 --2- 172.21.15.121:0/2372422947 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc1c100dc0 0x7efc1c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.491+0000 7efc2367f700 1 -- 172.21.15.121:0/2372422947 >> 172.21.15.121:0/2372422947 conn(0x7efc1c0fc650 msgr2=0x7efc1c0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.491+0000 7efc2367f700 1 -- 172.21.15.121:0/2372422947 shutdown_connections 2024-04-05T21:47:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.491+0000 7efc2367f700 1 -- 172.21.15.121:0/2372422947 wait complete. 2024-04-05T21:47:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.491+0000 7efc2367f700 1 Processor -- start 2024-04-05T21:47:11.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.491+0000 7efc2367f700 1 -- start start 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc2367f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc1c100dc0 0x7efc1c1966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc2367f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c101770 0x7efc1c196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc2367f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c1972a0 con 0x7efc1c101770 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc2367f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c199fd0 con 0x7efc1c100dc0 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc20c1a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c101770 0x7efc1c196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc20c1a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c101770 0x7efc1c196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60104/0 (socket says 172.21.15.121:60104) 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc20c1a700 1 -- 172.21.15.121:0/1588310247 learned_addr learned my addr 172.21.15.121:0/1588310247 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc2141b700 1 --2- 172.21.15.121:0/1588310247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc1c100dc0 0x7efc1c1966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc20c1a700 1 -- 172.21.15.121:0/1588310247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc1c100dc0 msgr2=0x7efc1c1966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc20c1a700 1 --2- 172.21.15.121:0/1588310247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc1c100dc0 0x7efc1c1966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc20c1a700 1 -- 172.21.15.121:0/1588310247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc180095f0 con 0x7efc1c101770 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc2141b700 1 --2- 172.21.15.121:0/1588310247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc1c100dc0 0x7efc1c1966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc20c1a700 1 --2- 172.21.15.121:0/1588310247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c101770 0x7efc1c196c10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7efc180092e0 tx=0x7efc1800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:11.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc127fc700 1 -- 172.21.15.121:0/1588310247 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc1801d070 con 0x7efc1c101770 2024-04-05T21:47:11.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc2367f700 1 -- 172.21.15.121:0/1588310247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c19a250 con 0x7efc1c101770 2024-04-05T21:47:11.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc127fc700 1 -- 172.21.15.121:0/1588310247 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc18005940 con 0x7efc1c101770 2024-04-05T21:47:11.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc127fc700 1 -- 172.21.15.121:0/1588310247 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc1800fe50 con 0x7efc1c101770 2024-04-05T21:47:11.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.492+0000 7efc2367f700 1 -- 172.21.15.121:0/1588310247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c19a740 con 0x7efc1c101770 2024-04-05T21:47:11.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.493+0000 7efc2367f700 1 -- 172.21.15.121:0/1588310247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c1091c0 con 0x7efc1c101770 2024-04-05T21:47:11.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.494+0000 7efc127fc700 1 -- 172.21.15.121:0/1588310247 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7efc18003a60 con 0x7efc1c101770 2024-04-05T21:47:11.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.494+0000 7efc127fc700 1 --2- 172.21.15.121:0/1588310247 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efc0806cce0 0x7efc0806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:11.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.494+0000 7efc127fc700 1 -- 172.21.15.121:0/1588310247 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7efc1808d7e0 con 0x7efc1c101770 2024-04-05T21:47:11.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.494+0000 7efc2141b700 1 --2- 172.21.15.121:0/1588310247 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efc0806cce0 0x7efc0806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:11.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.495+0000 7efc2141b700 1 --2- 172.21.15.121:0/1588310247 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efc0806cce0 0x7efc0806f1a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efc0c005950 tx=0x7efc0c0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:11.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.499+0000 7efc127fc700 1 -- 172.21.15.121:0/1588310247 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efc18057850 con 0x7efc1c101770 2024-04-05T21:47:11.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.709+0000 7efc2367f700 1 -- 172.21.15.121:0/1588310247 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7efc1c02cd30 con 0x7efc0806cce0 2024-04-05T21:47:11.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.710+0000 7efc127fc700 1 -- 172.21.15.121:0/1588310247 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7efc1c02cd30 con 0x7efc0806cce0 2024-04-05T21:47:11.711 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:47:11.711 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:47:11.711 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:47:11.711 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:47:11.711 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:47:11.711 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:47:11.711 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:47:11.711 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:47:11.711 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:47:11.711 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:47:11.711 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:47:11.711 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:47:11.711 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:47:11.712 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:47:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.712+0000 7efc2367f700 1 -- 172.21.15.121:0/1588310247 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efc0806cce0 msgr2=0x7efc0806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.712+0000 7efc2367f700 1 --2- 172.21.15.121:0/1588310247 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efc0806cce0 0x7efc0806f1a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efc0c005950 tx=0x7efc0c0058e0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.712+0000 7efc2367f700 1 -- 172.21.15.121:0/1588310247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c101770 msgr2=0x7efc1c196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.712+0000 7efc2367f700 1 --2- 172.21.15.121:0/1588310247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c101770 0x7efc1c196c10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7efc180092e0 tx=0x7efc1800be20 comp rx=0 tx=0).stop 2024-04-05T21:47:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.712+0000 7efc2367f700 1 -- 172.21.15.121:0/1588310247 shutdown_connections 2024-04-05T21:47:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.712+0000 7efc2367f700 1 --2- 172.21.15.121:0/1588310247 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efc0806cce0 0x7efc0806f1a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.712+0000 7efc2367f700 1 --2- 172.21.15.121:0/1588310247 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efc1c100dc0 0x7efc1c1966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.712+0000 7efc2367f700 1 --2- 172.21.15.121:0/1588310247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc1c101770 0x7efc1c196c10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.712+0000 7efc2367f700 1 -- 172.21.15.121:0/1588310247 >> 172.21.15.121:0/1588310247 conn(0x7efc1c0fc650 msgr2=0x7efc1c0fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.712+0000 7efc2367f700 1 -- 172.21.15.121:0/1588310247 shutdown_connections 2024-04-05T21:47:11.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.712+0000 7efc2367f700 1 -- 172.21.15.121:0/1588310247 wait complete. 2024-04-05T21:47:11.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.815+0000 7f934bd45700 1 -- 172.21.15.121:0/3571097425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9344100dd0 msgr2=0x7f93441011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:11.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.815+0000 7f934bd45700 1 --2- 172.21.15.121:0/3571097425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9344100dd0 0x7f93441011b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9334009910 tx=0x7f9334009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:11.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.815+0000 7f934bd45700 1 -- 172.21.15.121:0/3571097425 shutdown_connections 2024-04-05T21:47:11.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.815+0000 7f934bd45700 1 --2- 172.21.15.121:0/3571097425 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9344101780 0x7f93441057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.815+0000 7f934bd45700 1 --2- 172.21.15.121:0/3571097425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9344100dd0 0x7f93441011b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.815+0000 7f934bd45700 1 -- 172.21.15.121:0/3571097425 >> 172.21.15.121:0/3571097425 conn(0x7f93440fc640 msgr2=0x7f93440fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:11.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.815+0000 7f934bd45700 1 -- 172.21.15.121:0/3571097425 shutdown_connections 2024-04-05T21:47:11.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.815+0000 7f934bd45700 1 -- 172.21.15.121:0/3571097425 wait complete. 2024-04-05T21:47:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.815+0000 7f934bd45700 1 Processor -- start 2024-04-05T21:47:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.815+0000 7f934bd45700 1 -- start start 2024-04-05T21:47:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f934bd45700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9344100dd0 0x7f934419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f934bd45700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9344101780 0x7f934419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f934bd45700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934419f250 con 0x7f9344100dd0 2024-04-05T21:47:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f934bd45700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9344198700 con 0x7f9344101780 2024-04-05T21:47:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f9349ae1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9344100dd0 0x7f934419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f9349ae1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9344100dd0 0x7f934419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60120/0 (socket says 172.21.15.121:60120) 2024-04-05T21:47:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f9349ae1700 1 -- 172.21.15.121:0/204698059 learned_addr learned my addr 172.21.15.121:0/204698059 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f93492e0700 1 --2- 172.21.15.121:0/204698059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9344101780 0x7f934419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f9349ae1700 1 -- 172.21.15.121:0/204698059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9344101780 msgr2=0x7f934419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f9349ae1700 1 --2- 172.21.15.121:0/204698059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9344101780 0x7f934419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:11.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f9349ae1700 1 -- 172.21.15.121:0/204698059 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93340095f0 con 0x7f9344100dd0 2024-04-05T21:47:11.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f9349ae1700 1 --2- 172.21.15.121:0/204698059 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9344100dd0 0x7f934419e680 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9334006010 tx=0x7f933400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:11.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f933affd700 1 -- 172.21.15.121:0/204698059 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f933401d070 con 0x7f9344100dd0 2024-04-05T21:47:11.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f934bd45700 1 -- 172.21.15.121:0/204698059 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9344198980 con 0x7f9344100dd0 2024-04-05T21:47:11.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f933affd700 1 -- 172.21.15.121:0/204698059 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f933400f460 con 0x7f9344100dd0 2024-04-05T21:47:11.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f933affd700 1 -- 172.21.15.121:0/204698059 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9334021620 con 0x7f9344100dd0 2024-04-05T21:47:11.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.816+0000 7f934bd45700 1 -- 172.21.15.121:0/204698059 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9344198e70 con 0x7f9344100dd0 2024-04-05T21:47:11.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.817+0000 7f934bd45700 1 -- 172.21.15.121:0/204698059 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9344104f50 con 0x7f9344100dd0 2024-04-05T21:47:11.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.818+0000 7f933affd700 1 -- 172.21.15.121:0/204698059 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f933400fa40 con 0x7f9344100dd0 2024-04-05T21:47:11.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.818+0000 7f933affd700 1 --2- 172.21.15.121:0/204698059 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f933006cce0 0x7f933006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:11.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.818+0000 7f933affd700 1 -- 172.21.15.121:0/204698059 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f933408d790 con 0x7f9344100dd0 2024-04-05T21:47:11.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.818+0000 7f93492e0700 1 --2- 172.21.15.121:0/204698059 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f933006cce0 0x7f933006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:11.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.819+0000 7f93492e0700 1 --2- 172.21.15.121:0/204698059 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f933006cce0 0x7f933006f1a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9340005950 tx=0x7f93400058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:11.823+0000 7f933affd700 1 -- 172.21.15.121:0/204698059 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f933405b290 con 0x7f9344100dd0 2024-04-05T21:47:12.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:11 smithi188 ceph-mon[107793]: from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:12.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:11 smithi188 ceph-mon[107793]: from='client.34990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:12.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:11 smithi188 ceph-mon[107793]: pgmap v628: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:11 smithi188 ceph-mon[107793]: from='client.34994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:11 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3081358467' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:47:12.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:11 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3885376437' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:47:12.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:12.095+0000 7f934bd45700 1 -- 172.21.15.121:0/204698059 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f934404ea90 con 0x7f9344100dd0 2024-04-05T21:47:12.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:12.096+0000 7f933affd700 1 -- 172.21.15.121:0/204698059 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f933405ae20 con 0x7f9344100dd0 2024-04-05T21:47:12.097 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:47:12.097 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:47:12.097 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:47:12.097 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:47:12.097 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:47:12.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:12.098+0000 7f934bd45700 1 -- 172.21.15.121:0/204698059 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f933006cce0 msgr2=0x7f933006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:12.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:12.098+0000 7f934bd45700 1 --2- 172.21.15.121:0/204698059 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f933006cce0 0x7f933006f1a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9340005950 tx=0x7f93400058e0 comp rx=0 tx=0).stop 2024-04-05T21:47:12.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:12.098+0000 7f934bd45700 1 -- 172.21.15.121:0/204698059 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9344100dd0 msgr2=0x7f934419e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:12.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:12.098+0000 7f934bd45700 1 --2- 172.21.15.121:0/204698059 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9344100dd0 0x7f934419e680 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9334006010 tx=0x7f933400be20 comp rx=0 tx=0).stop 2024-04-05T21:47:12.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:12.098+0000 7f934bd45700 1 -- 172.21.15.121:0/204698059 shutdown_connections 2024-04-05T21:47:12.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:12.098+0000 7f934bd45700 1 --2- 172.21.15.121:0/204698059 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f933006cce0 0x7f933006f1a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:12.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:12.098+0000 7f934bd45700 1 --2- 172.21.15.121:0/204698059 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9344100dd0 0x7f934419e680 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:12.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:12.098+0000 7f934bd45700 1 --2- 172.21.15.121:0/204698059 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9344101780 0x7f934419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:12.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:12.098+0000 7f934bd45700 1 -- 172.21.15.121:0/204698059 >> 172.21.15.121:0/204698059 conn(0x7f93440fc640 msgr2=0x7f93440fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:12.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:12.098+0000 7f934bd45700 1 -- 172.21.15.121:0/204698059 shutdown_connections 2024-04-05T21:47:12.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:12.098+0000 7f934bd45700 1 -- 172.21.15.121:0/204698059 wait complete. 2024-04-05T21:47:12.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:11 smithi121 ceph-mon[145859]: from='client.34986 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:12.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:11 smithi121 ceph-mon[145859]: from='client.34990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:12.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:11 smithi121 ceph-mon[145859]: pgmap v628: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:12.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:11 smithi121 ceph-mon[145859]: from='client.34994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:12.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:11 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3081358467' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:47:12.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:11 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3885376437' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:47:13.025 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:12 smithi121 ceph-mon[145859]: from='client.35006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:13.025 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:12 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/204698059' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:47:13.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:12 smithi188 ceph-mon[107793]: from='client.35006 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:13.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:12 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/204698059' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:47:14.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:13 smithi188 ceph-mon[107793]: pgmap v629: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:14.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:13 smithi121 ceph-mon[145859]: pgmap v629: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:16.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:15 smithi188 ceph-mon[107793]: pgmap v630: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:16.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:47:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:15 smithi121 ceph-mon[145859]: pgmap v630: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:16.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:47:18.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:17 smithi188 ceph-mon[107793]: pgmap v631: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:18.069 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:17 smithi121 ceph-mon[145859]: pgmap v631: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:20.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:19 smithi188 ceph-mon[107793]: pgmap v632: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:20.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:19 smithi121 ceph-mon[145859]: pgmap v632: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:22.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:21 smithi188 ceph-mon[107793]: pgmap v633: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:22.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:21 smithi121 ceph-mon[145859]: pgmap v633: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:24.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:23 smithi188 ceph-mon[107793]: pgmap v634: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:24.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:23 smithi121 ceph-mon[145859]: pgmap v634: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:26.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:25 smithi188 ceph-mon[107793]: pgmap v635: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:26.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:25 smithi121 ceph-mon[145859]: pgmap v635: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:28.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:27 smithi188 ceph-mon[107793]: pgmap v636: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:28.162 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:27 smithi121 ceph-mon[145859]: pgmap v636: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:30.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:29 smithi188 ceph-mon[107793]: pgmap v637: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:30.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:29 smithi121 ceph-mon[145859]: pgmap v637: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:47:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:47:32.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:31 smithi188 ceph-mon[107793]: pgmap v638: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:32.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:31 smithi121 ceph-mon[145859]: pgmap v638: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:34.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:33 smithi121 ceph-mon[145859]: pgmap v639: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:34.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:33 smithi188 ceph-mon[107793]: pgmap v639: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:36.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:35 smithi121 ceph-mon[145859]: pgmap v640: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:36.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:35 smithi188 ceph-mon[107793]: pgmap v640: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:38.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:37 smithi121 ceph-mon[145859]: pgmap v641: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:38.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:37 smithi188 ceph-mon[107793]: pgmap v641: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:40.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:39 smithi121 ceph-mon[145859]: pgmap v642: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:40.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:39 smithi188 ceph-mon[107793]: pgmap v642: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:42.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:41 smithi188 ceph-mon[107793]: pgmap v643: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.213+0000 7fa12e3c8700 1 -- 172.21.15.121:0/15660755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa128105a40 msgr2=0x7fa128105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.213+0000 7fa12e3c8700 1 --2- 172.21.15.121:0/15660755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa128105a40 0x7fa128105e20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa118009910 tx=0x7fa118009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.214+0000 7fa12e3c8700 1 -- 172.21.15.121:0/15660755 shutdown_connections 2024-04-05T21:47:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.214+0000 7fa12e3c8700 1 --2- 172.21.15.121:0/15660755 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1280ffa40 0x7fa1280ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.214+0000 7fa12e3c8700 1 --2- 172.21.15.121:0/15660755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa128105a40 0x7fa128105e20 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.214+0000 7fa12e3c8700 1 -- 172.21.15.121:0/15660755 >> 172.21.15.121:0/15660755 conn(0x7fa1280fb550 msgr2=0x7fa1280fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.214+0000 7fa12e3c8700 1 -- 172.21.15.121:0/15660755 shutdown_connections 2024-04-05T21:47:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.214+0000 7fa12e3c8700 1 -- 172.21.15.121:0/15660755 wait complete. 2024-04-05T21:47:42.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.214+0000 7fa12e3c8700 1 Processor -- start 2024-04-05T21:47:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.214+0000 7fa12e3c8700 1 -- start start 2024-04-05T21:47:42.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa12e3c8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1280ffa40 0x7fa128197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:42.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa12e3c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa128105a40 0x7fa128197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:42.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa12e3c8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa128198080 con 0x7fa128105a40 2024-04-05T21:47:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa12e3c8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1281981c0 con 0x7fa1280ffa40 2024-04-05T21:47:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa1277fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa128105a40 0x7fa128197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa1277fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa128105a40 0x7fa128197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47890/0 (socket says 172.21.15.121:47890) 2024-04-05T21:47:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa1277fe700 1 -- 172.21.15.121:0/679424816 learned_addr learned my addr 172.21.15.121:0/679424816 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa127fff700 1 --2- 172.21.15.121:0/679424816 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1280ffa40 0x7fa128197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa1277fe700 1 -- 172.21.15.121:0/679424816 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1280ffa40 msgr2=0x7fa128197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa1277fe700 1 --2- 172.21.15.121:0/679424816 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1280ffa40 0x7fa128197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa1277fe700 1 -- 172.21.15.121:0/679424816 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1180095f0 con 0x7fa128105a40 2024-04-05T21:47:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa1277fe700 1 --2- 172.21.15.121:0/679424816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa128105a40 0x7fa128197a60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00d900 tx=0x7fa11c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa1257fa700 1 -- 172.21.15.121:0/679424816 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa11c004a00 con 0x7fa128105a40 2024-04-05T21:47:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa1257fa700 1 -- 172.21.15.121:0/679424816 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa11c005020 con 0x7fa128105a40 2024-04-05T21:47:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.215+0000 7fa12e3c8700 1 -- 172.21.15.121:0/679424816 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1281a0590 con 0x7fa128105a40 2024-04-05T21:47:42.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.216+0000 7fa1257fa700 1 -- 172.21.15.121:0/679424816 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa11c009b90 con 0x7fa128105a40 2024-04-05T21:47:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.216+0000 7fa12e3c8700 1 -- 172.21.15.121:0/679424816 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1281a0a50 con 0x7fa128105a40 2024-04-05T21:47:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.217+0000 7fa12e3c8700 1 -- 172.21.15.121:0/679424816 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa128108040 con 0x7fa128105a40 2024-04-05T21:47:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.217+0000 7fa1257fa700 1 -- 172.21.15.121:0/679424816 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa11c009cf0 con 0x7fa128105a40 2024-04-05T21:47:42.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.217+0000 7fa1257fa700 1 --2- 172.21.15.121:0/679424816 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa11006cce0 0x7fa11006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:42.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.218+0000 7fa1257fa700 1 -- 172.21.15.121:0/679424816 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa11c08c3f0 con 0x7fa128105a40 2024-04-05T21:47:42.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.218+0000 7fa127fff700 1 --2- 172.21.15.121:0/679424816 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa11006cce0 0x7fa11006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:42.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.218+0000 7fa127fff700 1 --2- 172.21.15.121:0/679424816 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa11006cce0 0x7fa11006f1a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa118006010 tx=0x7fa11801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:42.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.222+0000 7fa1257fa700 1 -- 172.21.15.121:0/679424816 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa11c0564e0 con 0x7fa128105a40 2024-04-05T21:47:42.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:41 smithi121 ceph-mon[145859]: pgmap v643: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:42.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.433+0000 7fa12e3c8700 1 -- 172.21.15.121:0/679424816 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa12802cce0 con 0x7fa11006cce0 2024-04-05T21:47:42.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.434+0000 7fa1257fa700 1 -- 172.21.15.121:0/679424816 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa12802cce0 con 0x7fa11006cce0 2024-04-05T21:47:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.436+0000 7fa12e3c8700 1 -- 172.21.15.121:0/679424816 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa11006cce0 msgr2=0x7fa11006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.436+0000 7fa12e3c8700 1 --2- 172.21.15.121:0/679424816 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa11006cce0 0x7fa11006f1a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa118006010 tx=0x7fa11801a040 comp rx=0 tx=0).stop 2024-04-05T21:47:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.436+0000 7fa12e3c8700 1 -- 172.21.15.121:0/679424816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa128105a40 msgr2=0x7fa128197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.436+0000 7fa12e3c8700 1 --2- 172.21.15.121:0/679424816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa128105a40 0x7fa128197a60 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00d900 tx=0x7fa11c00dc10 comp rx=0 tx=0).stop 2024-04-05T21:47:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.436+0000 7fa12e3c8700 1 -- 172.21.15.121:0/679424816 shutdown_connections 2024-04-05T21:47:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.436+0000 7fa12e3c8700 1 --2- 172.21.15.121:0/679424816 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa11006cce0 0x7fa11006f1a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.436+0000 7fa12e3c8700 1 --2- 172.21.15.121:0/679424816 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa1280ffa40 0x7fa128197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.436+0000 7fa12e3c8700 1 --2- 172.21.15.121:0/679424816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa128105a40 0x7fa128197a60 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.436+0000 7fa12e3c8700 1 -- 172.21.15.121:0/679424816 >> 172.21.15.121:0/679424816 conn(0x7fa1280fb550 msgr2=0x7fa1280fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.436+0000 7fa12e3c8700 1 -- 172.21.15.121:0/679424816 shutdown_connections 2024-04-05T21:47:42.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.436+0000 7fa12e3c8700 1 -- 172.21.15.121:0/679424816 wait complete. 2024-04-05T21:47:42.447 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:47:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.540+0000 7fe590ed1700 1 -- 172.21.15.121:0/3002122498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe58c102640 msgr2=0x7fe58c10ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.540+0000 7fe590ed1700 1 --2- 172.21.15.121:0/3002122498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe58c102640 0x7fe58c10ab30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe580009910 tx=0x7fe580009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.540+0000 7fe590ed1700 1 -- 172.21.15.121:0/3002122498 shutdown_connections 2024-04-05T21:47:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.540+0000 7fe590ed1700 1 --2- 172.21.15.121:0/3002122498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe58c102640 0x7fe58c10ab30 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.540+0000 7fe590ed1700 1 --2- 172.21.15.121:0/3002122498 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe58c101d20 0x7fe58c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.540+0000 7fe590ed1700 1 -- 172.21.15.121:0/3002122498 >> 172.21.15.121:0/3002122498 conn(0x7fe58c0fb550 msgr2=0x7fe58c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.540+0000 7fe590ed1700 1 -- 172.21.15.121:0/3002122498 shutdown_connections 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.540+0000 7fe590ed1700 1 -- 172.21.15.121:0/3002122498 wait complete. 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.540+0000 7fe590ed1700 1 Processor -- start 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe590ed1700 1 -- start start 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe590ed1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe58c101d20 0x7fe58c0ff980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe590ed1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe58c102640 0x7fe58c0ffec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe590ed1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c100400 con 0x7fe58c102640 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe590ed1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c100540 con 0x7fe58c101d20 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe589d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe58c102640 0x7fe58c0ffec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe589d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe58c102640 0x7fe58c0ffec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47912/0 (socket says 172.21.15.121:47912) 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe589d9b700 1 -- 172.21.15.121:0/222118596 learned_addr learned my addr 172.21.15.121:0/222118596 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe58a59c700 1 --2- 172.21.15.121:0/222118596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe58c101d20 0x7fe58c0ff980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe589d9b700 1 -- 172.21.15.121:0/222118596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe58c101d20 msgr2=0x7fe58c0ff980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe589d9b700 1 --2- 172.21.15.121:0/222118596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe58c101d20 0x7fe58c0ff980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe589d9b700 1 -- 172.21.15.121:0/222118596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5800095f0 con 0x7fe58c102640 2024-04-05T21:47:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe589d9b700 1 --2- 172.21.15.121:0/222118596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe58c102640 0x7fe58c0ffec0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe58000bd70 tx=0x7fe58000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe57b7fe700 1 -- 172.21.15.121:0/222118596 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe58001d070 con 0x7fe58c102640 2024-04-05T21:47:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe57b7fe700 1 -- 172.21.15.121:0/222118596 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe58000f460 con 0x7fe58c102640 2024-04-05T21:47:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe590ed1700 1 -- 172.21.15.121:0/222118596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58c1007c0 con 0x7fe58c102640 2024-04-05T21:47:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.541+0000 7fe57b7fe700 1 -- 172.21.15.121:0/222118596 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe580021640 con 0x7fe58c102640 2024-04-05T21:47:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.542+0000 7fe590ed1700 1 -- 172.21.15.121:0/222118596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58c1a2bd0 con 0x7fe58c102640 2024-04-05T21:47:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.542+0000 7fe590ed1700 1 -- 172.21.15.121:0/222118596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58c108230 con 0x7fe58c102640 2024-04-05T21:47:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.543+0000 7fe57b7fe700 1 -- 172.21.15.121:0/222118596 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe58000fa20 con 0x7fe58c102640 2024-04-05T21:47:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.543+0000 7fe57b7fe700 1 --2- 172.21.15.121:0/222118596 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe57406cce0 0x7fe57406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.543+0000 7fe58a59c700 1 --2- 172.21.15.121:0/222118596 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe57406cce0 0x7fe57406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.543+0000 7fe57b7fe700 1 -- 172.21.15.121:0/222118596 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe58008d600 con 0x7fe58c102640 2024-04-05T21:47:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.544+0000 7fe58a59c700 1 --2- 172.21.15.121:0/222118596 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe57406cce0 0x7fe57406f1a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe57c005950 tx=0x7fe57c00b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.548+0000 7fe57b7fe700 1 -- 172.21.15.121:0/222118596 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe5800576f0 con 0x7fe58c102640 2024-04-05T21:47:42.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.758+0000 7fe590ed1700 1 -- 172.21.15.121:0/222118596 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe58c02cce0 con 0x7fe57406cce0 2024-04-05T21:47:42.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.759+0000 7fe57b7fe700 1 -- 172.21.15.121:0/222118596 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fe58c02cce0 con 0x7fe57406cce0 2024-04-05T21:47:42.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.761+0000 7fe590ed1700 1 -- 172.21.15.121:0/222118596 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe57406cce0 msgr2=0x7fe57406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:42.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.761+0000 7fe590ed1700 1 --2- 172.21.15.121:0/222118596 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe57406cce0 0x7fe57406f1a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe57c005950 tx=0x7fe57c00b410 comp rx=0 tx=0).stop 2024-04-05T21:47:42.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.761+0000 7fe590ed1700 1 -- 172.21.15.121:0/222118596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe58c102640 msgr2=0x7fe58c0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:42.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.761+0000 7fe590ed1700 1 --2- 172.21.15.121:0/222118596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe58c102640 0x7fe58c0ffec0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe58000bd70 tx=0x7fe58000bda0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.761+0000 7fe590ed1700 1 -- 172.21.15.121:0/222118596 shutdown_connections 2024-04-05T21:47:42.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.761+0000 7fe590ed1700 1 --2- 172.21.15.121:0/222118596 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe57406cce0 0x7fe57406f1a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.761+0000 7fe590ed1700 1 --2- 172.21.15.121:0/222118596 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe58c101d20 0x7fe58c0ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.761+0000 7fe590ed1700 1 --2- 172.21.15.121:0/222118596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe58c102640 0x7fe58c0ffec0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.761+0000 7fe590ed1700 1 -- 172.21.15.121:0/222118596 >> 172.21.15.121:0/222118596 conn(0x7fe58c0fb550 msgr2=0x7fe58c105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:42.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.761+0000 7fe590ed1700 1 -- 172.21.15.121:0/222118596 shutdown_connections 2024-04-05T21:47:42.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.761+0000 7fe590ed1700 1 -- 172.21.15.121:0/222118596 wait complete. 2024-04-05T21:47:42.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.865+0000 7fc0b8db8700 1 -- 172.21.15.121:0/2730396847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b4100de0 msgr2=0x7fc0b41011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.865+0000 7fc0b8db8700 1 --2- 172.21.15.121:0/2730396847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b4100de0 0x7fc0b41011c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009910 tx=0x7fc0a4009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.866+0000 7fc0b8db8700 1 -- 172.21.15.121:0/2730396847 shutdown_connections 2024-04-05T21:47:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.866+0000 7fc0b8db8700 1 --2- 172.21.15.121:0/2730396847 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0b4101790 0x7fc0b41057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.866+0000 7fc0b8db8700 1 --2- 172.21.15.121:0/2730396847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b4100de0 0x7fc0b41011c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.866+0000 7fc0b8db8700 1 -- 172.21.15.121:0/2730396847 >> 172.21.15.121:0/2730396847 conn(0x7fc0b40fc6b0 msgr2=0x7fc0b40fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.866+0000 7fc0b8db8700 1 -- 172.21.15.121:0/2730396847 shutdown_connections 2024-04-05T21:47:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.866+0000 7fc0b8db8700 1 -- 172.21.15.121:0/2730396847 wait complete. 2024-04-05T21:47:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.866+0000 7fc0b8db8700 1 Processor -- start 2024-04-05T21:47:42.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.866+0000 7fc0b8db8700 1 -- start start 2024-04-05T21:47:42.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b8db8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0b4100de0 0x7fc0b4073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:42.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b8db8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b4101790 0x7fc0b40735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:42.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b8db8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b4073b10 con 0x7fc0b4101790 2024-04-05T21:47:42.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b8db8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b4073c50 con 0x7fc0b4100de0 2024-04-05T21:47:42.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b1d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b4101790 0x7fc0b40735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:42.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b1d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b4101790 0x7fc0b40735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47942/0 (socket says 172.21.15.121:47942) 2024-04-05T21:47:42.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b1d9b700 1 -- 172.21.15.121:0/3740239256 learned_addr learned my addr 172.21.15.121:0/3740239256 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:42.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b259c700 1 --2- 172.21.15.121:0/3740239256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0b4100de0 0x7fc0b4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:42.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b1d9b700 1 -- 172.21.15.121:0/3740239256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0b4100de0 msgr2=0x7fc0b4073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:42.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b1d9b700 1 --2- 172.21.15.121:0/3740239256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0b4100de0 0x7fc0b4073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:42.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b1d9b700 1 -- 172.21.15.121:0/3740239256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a40095f0 con 0x7fc0b4101790 2024-04-05T21:47:42.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b1d9b700 1 --2- 172.21.15.121:0/3740239256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b4101790 0x7fc0b40735d0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800d8d0 tx=0x7fc0a800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:42.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0a37fe700 1 -- 172.21.15.121:0/3740239256 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0a8004a00 con 0x7fc0b4101790 2024-04-05T21:47:42.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0a37fe700 1 -- 172.21.15.121:0/3740239256 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0a8005020 con 0x7fc0b4101790 2024-04-05T21:47:42.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b8db8700 1 -- 172.21.15.121:0/3740239256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b41a28a0 con 0x7fc0b4101790 2024-04-05T21:47:42.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0a37fe700 1 -- 172.21.15.121:0/3740239256 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0a800fb80 con 0x7fc0b4101790 2024-04-05T21:47:42.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.867+0000 7fc0b8db8700 1 -- 172.21.15.121:0/3740239256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b41a2cb0 con 0x7fc0b4101790 2024-04-05T21:47:42.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.868+0000 7fc0b8db8700 1 -- 172.21.15.121:0/3740239256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b4109180 con 0x7fc0b4101790 2024-04-05T21:47:42.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.869+0000 7fc0a37fe700 1 -- 172.21.15.121:0/3740239256 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc0a8005190 con 0x7fc0b4101790 2024-04-05T21:47:42.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.869+0000 7fc0a37fe700 1 --2- 172.21.15.121:0/3740239256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc09c06cce0 0x7fc09c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:42.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.869+0000 7fc0a37fe700 1 -- 172.21.15.121:0/3740239256 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc0a808beb0 con 0x7fc0b4101790 2024-04-05T21:47:42.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.869+0000 7fc0b259c700 1 --2- 172.21.15.121:0/3740239256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc09c06cce0 0x7fc09c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:42.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.870+0000 7fc0b259c700 1 --2- 172.21.15.121:0/3740239256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc09c06cce0 0x7fc09c06f1a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4006010 tx=0x7fc0a401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:42.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:42.874+0000 7fc0a37fe700 1 -- 172.21.15.121:0/3740239256 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc0a8055f20 con 0x7fc0b4101790 2024-04-05T21:47:43.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.073+0000 7fc0b8db8700 1 -- 172.21.15.121:0/3740239256 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0b402cda0 con 0x7fc09c06cce0 2024-04-05T21:47:43.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.083+0000 7fc0a37fe700 1 -- 172.21.15.121:0/3740239256 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fc0b402cda0 con 0x7fc09c06cce0 2024-04-05T21:47:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:47:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (18m) 12m ago 29m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:47:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (30m) 12m ago 30m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:47:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (29m) 12m ago 29m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:47:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (16m) 12m ago 30m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:47:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (16m) 12m ago 29m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:47:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (18m) 12m ago 29m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:47:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (12m) 12m ago 23m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:47:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (12m) 12m ago 23m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:47:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (12m) 12m ago 23m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:47:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (23m) 12m ago 23m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:47:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (20m) 12m ago 31m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:47:43.084 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (19m) 12m ago 28m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:47:43.085 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (17m) 12m ago 31m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:47:43.085 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (16m) 12m ago 28m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:47:43.085 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (19m) 12m ago 30m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:47:43.085 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (19m) 12m ago 28m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:47:43.085 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (16m) 12m ago 27m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:47:43.085 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (15m) 12m ago 27m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:47:43.085 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (15m) 12m ago 26m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:47:43.085 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (14m) 12m ago 26m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:47:43.085 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (13m) 12m ago 26m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:47:43.085 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (13m) 12m ago 25m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:47:43.085 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (18m) 12m ago 29m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:47:43.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.085+0000 7fc0b8db8700 1 -- 172.21.15.121:0/3740239256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc09c06cce0 msgr2=0x7fc09c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:43.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.085+0000 7fc0b8db8700 1 --2- 172.21.15.121:0/3740239256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc09c06cce0 0x7fc09c06f1a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4006010 tx=0x7fc0a401a040 comp rx=0 tx=0).stop 2024-04-05T21:47:43.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.085+0000 7fc0b8db8700 1 -- 172.21.15.121:0/3740239256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b4101790 msgr2=0x7fc0b40735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:43.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.085+0000 7fc0b8db8700 1 --2- 172.21.15.121:0/3740239256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b4101790 0x7fc0b40735d0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800d8d0 tx=0x7fc0a800dbe0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.085+0000 7fc0b8db8700 1 -- 172.21.15.121:0/3740239256 shutdown_connections 2024-04-05T21:47:43.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.085+0000 7fc0b8db8700 1 --2- 172.21.15.121:0/3740239256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc09c06cce0 0x7fc09c06f1a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.085+0000 7fc0b8db8700 1 --2- 172.21.15.121:0/3740239256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc0b4100de0 0x7fc0b4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.085+0000 7fc0b8db8700 1 --2- 172.21.15.121:0/3740239256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b4101790 0x7fc0b40735d0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.085+0000 7fc0b8db8700 1 -- 172.21.15.121:0/3740239256 >> 172.21.15.121:0/3740239256 conn(0x7fc0b40fc6b0 msgr2=0x7fc0b40fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:43.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.085+0000 7fc0b8db8700 1 -- 172.21.15.121:0/3740239256 shutdown_connections 2024-04-05T21:47:43.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.085+0000 7fc0b8db8700 1 -- 172.21.15.121:0/3740239256 wait complete. 2024-04-05T21:47:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.187+0000 7f7348465700 1 -- 172.21.15.121:0/3862978784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7340105a40 msgr2=0x7f7340105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.187+0000 7f7348465700 1 --2- 172.21.15.121:0/3862978784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7340105a40 0x7f7340105e20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f733c009910 tx=0x7f733c009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.187+0000 7f7348465700 1 -- 172.21.15.121:0/3862978784 shutdown_connections 2024-04-05T21:47:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.187+0000 7f7348465700 1 --2- 172.21.15.121:0/3862978784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73400ffbd0 0x7f7340100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.187+0000 7f7348465700 1 --2- 172.21.15.121:0/3862978784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7340105a40 0x7f7340105e20 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.187+0000 7f7348465700 1 -- 172.21.15.121:0/3862978784 >> 172.21.15.121:0/3862978784 conn(0x7f73400755a0 msgr2=0x7f73400759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.188+0000 7f7348465700 1 -- 172.21.15.121:0/3862978784 shutdown_connections 2024-04-05T21:47:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.188+0000 7f7348465700 1 -- 172.21.15.121:0/3862978784 wait complete. 2024-04-05T21:47:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.188+0000 7f7348465700 1 Processor -- start 2024-04-05T21:47:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.188+0000 7f7348465700 1 -- start start 2024-04-05T21:47:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.188+0000 7f7348465700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73400ffbd0 0x7f7340101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.188+0000 7f7348465700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7340105a40 0x7f7340101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.188+0000 7f7348465700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7340102120 con 0x7f7340105a40 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.188+0000 7f7348465700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7340102290 con 0x7f73400ffbd0 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.188+0000 7f7345a00700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7340105a40 0x7f7340101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.189+0000 7f7345a00700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7340105a40 0x7f7340101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47962/0 (socket says 172.21.15.121:47962) 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.189+0000 7f7345a00700 1 -- 172.21.15.121:0/2022912854 learned_addr learned my addr 172.21.15.121:0/2022912854 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.189+0000 7f7346201700 1 --2- 172.21.15.121:0/2022912854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73400ffbd0 0x7f7340101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.189+0000 7f7345a00700 1 -- 172.21.15.121:0/2022912854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73400ffbd0 msgr2=0x7f7340101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.189+0000 7f7345a00700 1 --2- 172.21.15.121:0/2022912854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73400ffbd0 0x7f7340101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.189+0000 7f7345a00700 1 -- 172.21.15.121:0/2022912854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c0095f0 con 0x7f7340105a40 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.189+0000 7f7345a00700 1 --2- 172.21.15.121:0/2022912854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7340105a40 0x7f7340101b50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f733000d8d0 tx=0x7f733000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.189+0000 7f73377fe700 1 -- 172.21.15.121:0/2022912854 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73300098a0 con 0x7f7340105a40 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.189+0000 7f73377fe700 1 -- 172.21.15.121:0/2022912854 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7330010460 con 0x7f7340105a40 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.189+0000 7f7348465700 1 -- 172.21.15.121:0/2022912854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73401a2810 con 0x7f7340105a40 2024-04-05T21:47:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.189+0000 7f73377fe700 1 -- 172.21.15.121:0/2022912854 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7330009ea0 con 0x7f7340105a40 2024-04-05T21:47:43.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.189+0000 7f7348465700 1 -- 172.21.15.121:0/2022912854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73401a2c80 con 0x7f7340105a40 2024-04-05T21:47:43.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.190+0000 7f7348465700 1 -- 172.21.15.121:0/2022912854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7340109050 con 0x7f7340105a40 2024-04-05T21:47:43.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.190+0000 7f73377fe700 1 -- 172.21.15.121:0/2022912854 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7330009a00 con 0x7f7340105a40 2024-04-05T21:47:43.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.191+0000 7f73377fe700 1 --2- 172.21.15.121:0/2022912854 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f732c06cce0 0x7f732c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:43.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.191+0000 7f73377fe700 1 -- 172.21.15.121:0/2022912854 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f733008c450 con 0x7f7340105a40 2024-04-05T21:47:43.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.191+0000 7f7346201700 1 --2- 172.21.15.121:0/2022912854 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f732c06cce0 0x7f732c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:43.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.191+0000 7f7346201700 1 --2- 172.21.15.121:0/2022912854 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f732c06cce0 0x7f732c06f1a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f733c006010 tx=0x7f733c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:43.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.195+0000 7f73377fe700 1 -- 172.21.15.121:0/2022912854 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f73300564c0 con 0x7f7340105a40 2024-04-05T21:47:43.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.475+0000 7f7348465700 1 -- 172.21.15.121:0/2022912854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f734002cc70 con 0x7f7340105a40 2024-04-05T21:47:43.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.476+0000 7f73377fe700 1 -- 172.21.15.121:0/2022912854 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f7330059ae0 con 0x7f7340105a40 2024-04-05T21:47:43.476 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:47:43.477 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:47:43.477 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:47:43.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:47:43.477 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:47:43.477 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:47:43.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:47:43.477 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:47:43.477 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:47:43.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:47:43.477 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:47:43.477 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:47:43.477 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:47:43.478 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:47:43.478 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:47:43.478 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:47:43.478 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:47:43.478 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:47:43.478 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:47:43.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.478+0000 7f7348465700 1 -- 172.21.15.121:0/2022912854 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f732c06cce0 msgr2=0x7f732c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:43.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.478+0000 7f7348465700 1 --2- 172.21.15.121:0/2022912854 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f732c06cce0 0x7f732c06f1a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f733c006010 tx=0x7f733c01a040 comp rx=0 tx=0).stop 2024-04-05T21:47:43.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.478+0000 7f7348465700 1 -- 172.21.15.121:0/2022912854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7340105a40 msgr2=0x7f7340101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:43.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.478+0000 7f7348465700 1 --2- 172.21.15.121:0/2022912854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7340105a40 0x7f7340101b50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f733000d8d0 tx=0x7f733000dbe0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.478+0000 7f7348465700 1 -- 172.21.15.121:0/2022912854 shutdown_connections 2024-04-05T21:47:43.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.478+0000 7f7348465700 1 --2- 172.21.15.121:0/2022912854 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f732c06cce0 0x7f732c06f1a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.478+0000 7f7348465700 1 --2- 172.21.15.121:0/2022912854 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f73400ffbd0 0x7f7340101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.478+0000 7f7348465700 1 --2- 172.21.15.121:0/2022912854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7340105a40 0x7f7340101b50 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.478+0000 7f7348465700 1 -- 172.21.15.121:0/2022912854 >> 172.21.15.121:0/2022912854 conn(0x7f73400755a0 msgr2=0x7f7340068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:43.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.478+0000 7f7348465700 1 -- 172.21.15.121:0/2022912854 shutdown_connections 2024-04-05T21:47:43.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.478+0000 7f7348465700 1 -- 172.21.15.121:0/2022912854 wait complete. 2024-04-05T21:47:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.582+0000 7fcbfccc4700 1 -- 172.21.15.121:0/1480633930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf80ffbd0 msgr2=0x7fcbf8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.582+0000 7fcbfccc4700 1 --2- 172.21.15.121:0/1480633930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf80ffbd0 0x7fcbf8100050 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009910 tx=0x7fcbec009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.582+0000 7fcbfccc4700 1 -- 172.21.15.121:0/1480633930 shutdown_connections 2024-04-05T21:47:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.582+0000 7fcbfccc4700 1 --2- 172.21.15.121:0/1480633930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf80ffbd0 0x7fcbf8100050 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.582+0000 7fcbfccc4700 1 --2- 172.21.15.121:0/1480633930 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbf8105a40 0x7fcbf8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.582+0000 7fcbfccc4700 1 -- 172.21.15.121:0/1480633930 >> 172.21.15.121:0/1480633930 conn(0x7fcbf80755a0 msgr2=0x7fcbf80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.582+0000 7fcbfccc4700 1 -- 172.21.15.121:0/1480633930 shutdown_connections 2024-04-05T21:47:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.582+0000 7fcbfccc4700 1 -- 172.21.15.121:0/1480633930 wait complete. 2024-04-05T21:47:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.583+0000 7fcbfccc4700 1 Processor -- start 2024-04-05T21:47:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.583+0000 7fcbfccc4700 1 -- start start 2024-04-05T21:47:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.583+0000 7fcbfccc4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbf80ffbd0 0x7fcbf819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.583+0000 7fcbfccc4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf8105a40 0x7fcbf819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.583+0000 7fcbfccc4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf819d0e0 con 0x7fcbf8105a40 2024-04-05T21:47:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.583+0000 7fcbfccc4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf8196590 con 0x7fcbf80ffbd0 2024-04-05T21:47:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.583+0000 7fcbf5d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf8105a40 0x7fcbf819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.583+0000 7fcbf5d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf8105a40 0x7fcbf819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47982/0 (socket says 172.21.15.121:47982) 2024-04-05T21:47:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.583+0000 7fcbf5d9b700 1 -- 172.21.15.121:0/2474059577 learned_addr learned my addr 172.21.15.121:0/2474059577 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.583+0000 7fcbf659c700 1 --2- 172.21.15.121:0/2474059577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbf80ffbd0 0x7fcbf819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.583+0000 7fcbf5d9b700 1 -- 172.21.15.121:0/2474059577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbf80ffbd0 msgr2=0x7fcbf819c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.583+0000 7fcbf5d9b700 1 --2- 172.21.15.121:0/2474059577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbf80ffbd0 0x7fcbf819c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.583+0000 7fcbf5d9b700 1 -- 172.21.15.121:0/2474059577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec0095f0 con 0x7fcbf8105a40 2024-04-05T21:47:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.584+0000 7fcbf5d9b700 1 --2- 172.21.15.121:0/2474059577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf8105a40 0x7fcbf819ca50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcbec006010 tx=0x7fcbec00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.584+0000 7fcbe77fe700 1 -- 172.21.15.121:0/2474059577 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbec01d070 con 0x7fcbf8105a40 2024-04-05T21:47:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.584+0000 7fcbe77fe700 1 -- 172.21.15.121:0/2474059577 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbec005710 con 0x7fcbf8105a40 2024-04-05T21:47:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.584+0000 7fcbfccc4700 1 -- 172.21.15.121:0/2474059577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf8196810 con 0x7fcbf8105a40 2024-04-05T21:47:43.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.584+0000 7fcbe77fe700 1 -- 172.21.15.121:0/2474059577 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbec00fe20 con 0x7fcbf8105a40 2024-04-05T21:47:43.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.584+0000 7fcbfccc4700 1 -- 172.21.15.121:0/2474059577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf8196d00 con 0x7fcbf8105a40 2024-04-05T21:47:43.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.585+0000 7fcbfccc4700 1 -- 172.21.15.121:0/2474059577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf8108fe0 con 0x7fcbf8105a40 2024-04-05T21:47:43.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.585+0000 7fcbe77fe700 1 -- 172.21.15.121:0/2474059577 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcbec003a60 con 0x7fcbf8105a40 2024-04-05T21:47:43.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.586+0000 7fcbe77fe700 1 --2- 172.21.15.121:0/2474059577 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcbe006cce0 0x7fcbe006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:43.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.586+0000 7fcbf659c700 1 --2- 172.21.15.121:0/2474059577 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcbe006cce0 0x7fcbe006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:43.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.586+0000 7fcbe77fe700 1 -- 172.21.15.121:0/2474059577 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fcbec08d6f0 con 0x7fcbf8105a40 2024-04-05T21:47:43.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.586+0000 7fcbf659c700 1 --2- 172.21.15.121:0/2474059577 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcbe006cce0 0x7fcbe006f1a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8005950 tx=0x7fcbe800b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.590+0000 7fcbe77fe700 1 -- 172.21.15.121:0/2474059577 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcbec05b1f0 con 0x7fcbf8105a40 2024-04-05T21:47:43.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.828+0000 7fcbfccc4700 1 -- 172.21.15.121:0/2474059577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcbf804ea90 con 0x7fcbf8105a40 2024-04-05T21:47:43.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.829+0000 7fcbe77fe700 1 -- 172.21.15.121:0/2474059577 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fcbec061070 con 0x7fcbf8105a40 2024-04-05T21:47:43.829 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:47:43.830 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:47:43.830 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:47:43.830 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:47:43.830 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:47:43.830 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:47:43.830 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:47:43.830 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:47:43.830 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:47:43.830 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:47:43.830 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:47:43.830 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:47:43.830 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:47:43.830 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:47:43.831 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:47:43.832 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:47:43.832 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:47:43.832 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:47:43.832 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:47:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:47:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:47:43.832 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:47:43.832 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:47:43.832 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:47:43.832 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:47:43.832 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:47:43.832 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:47:43.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.831+0000 7fcbfccc4700 1 -- 172.21.15.121:0/2474059577 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcbe006cce0 msgr2=0x7fcbe006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:43.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.831+0000 7fcbfccc4700 1 --2- 172.21.15.121:0/2474059577 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcbe006cce0 0x7fcbe006f1a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8005950 tx=0x7fcbe800b410 comp rx=0 tx=0).stop 2024-04-05T21:47:43.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.831+0000 7fcbfccc4700 1 -- 172.21.15.121:0/2474059577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf8105a40 msgr2=0x7fcbf819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:43.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.831+0000 7fcbfccc4700 1 --2- 172.21.15.121:0/2474059577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf8105a40 0x7fcbf819ca50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcbec006010 tx=0x7fcbec00bd70 comp rx=0 tx=0).stop 2024-04-05T21:47:43.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.831+0000 7fcbfccc4700 1 -- 172.21.15.121:0/2474059577 shutdown_connections 2024-04-05T21:47:43.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.831+0000 7fcbfccc4700 1 --2- 172.21.15.121:0/2474059577 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcbe006cce0 0x7fcbe006f1a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.831+0000 7fcbfccc4700 1 --2- 172.21.15.121:0/2474059577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcbf80ffbd0 0x7fcbf819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.831+0000 7fcbfccc4700 1 --2- 172.21.15.121:0/2474059577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf8105a40 0x7fcbf819ca50 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.831+0000 7fcbfccc4700 1 -- 172.21.15.121:0/2474059577 >> 172.21.15.121:0/2474059577 conn(0x7fcbf80755a0 msgr2=0x7fcbf8103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:43.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.831+0000 7fcbfccc4700 1 -- 172.21.15.121:0/2474059577 shutdown_connections 2024-04-05T21:47:43.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.831+0000 7fcbfccc4700 1 -- 172.21.15.121:0/2474059577 wait complete. 2024-04-05T21:47:43.834 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:47:43.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.935+0000 7f5237d9c700 1 -- 172.21.15.121:0/771139030 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52300ffa40 msgr2=0x7f52300ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:43.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.935+0000 7f5237d9c700 1 --2- 172.21.15.121:0/771139030 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52300ffa40 0x7f52300ffec0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f522c009910 tx=0x7f522c009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:43.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.935+0000 7f5237d9c700 1 -- 172.21.15.121:0/771139030 shutdown_connections 2024-04-05T21:47:43.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.935+0000 7f5237d9c700 1 --2- 172.21.15.121:0/771139030 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52300ffa40 0x7f52300ffec0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.935+0000 7f5237d9c700 1 --2- 172.21.15.121:0/771139030 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5230105a40 0x7f5230105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.935+0000 7f5237d9c700 1 -- 172.21.15.121:0/771139030 >> 172.21.15.121:0/771139030 conn(0x7f52300fb570 msgr2=0x7f52300fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:43.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.935+0000 7f5237d9c700 1 -- 172.21.15.121:0/771139030 shutdown_connections 2024-04-05T21:47:43.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.936+0000 7f5237d9c700 1 -- 172.21.15.121:0/771139030 wait complete. 2024-04-05T21:47:43.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.936+0000 7f5237d9c700 1 Processor -- start 2024-04-05T21:47:43.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.936+0000 7f5237d9c700 1 -- start start 2024-04-05T21:47:43.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.936+0000 7f5237d9c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52300ffa40 0x7f5230196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:43.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.936+0000 7f5237d9c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5230105a40 0x7f5230196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:43.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.936+0000 7f5237d9c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5230197230 con 0x7f52300ffa40 2024-04-05T21:47:43.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.936+0000 7f5237d9c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523019afc0 con 0x7f5230105a40 2024-04-05T21:47:43.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.936+0000 7f5235b38700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52300ffa40 0x7f5230196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:43.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.936+0000 7f5235b38700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52300ffa40 0x7f5230196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48004/0 (socket says 172.21.15.121:48004) 2024-04-05T21:47:43.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.936+0000 7f5235b38700 1 -- 172.21.15.121:0/1522790855 learned_addr learned my addr 172.21.15.121:0/1522790855 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:43.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.937+0000 7f5235337700 1 --2- 172.21.15.121:0/1522790855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5230105a40 0x7f5230196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:43.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.937+0000 7f5235b38700 1 -- 172.21.15.121:0/1522790855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5230105a40 msgr2=0x7f5230196b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:43.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.937+0000 7f5235b38700 1 --2- 172.21.15.121:0/1522790855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5230105a40 0x7f5230196b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:43.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.937+0000 7f5235b38700 1 -- 172.21.15.121:0/1522790855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c0095f0 con 0x7f52300ffa40 2024-04-05T21:47:43.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.937+0000 7f5235337700 1 --2- 172.21.15.121:0/1522790855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5230105a40 0x7f5230196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:47:43.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.937+0000 7f5235b38700 1 --2- 172.21.15.121:0/1522790855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52300ffa40 0x7f5230196610 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f522000c8a0 tx=0x7f522000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:43.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.937+0000 7f5226ffd700 1 -- 172.21.15.121:0/1522790855 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f522000cea0 con 0x7f52300ffa40 2024-04-05T21:47:43.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.937+0000 7f5226ffd700 1 -- 172.21.15.121:0/1522790855 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5220004830 con 0x7f52300ffa40 2024-04-05T21:47:43.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.937+0000 7f5237d9c700 1 -- 172.21.15.121:0/1522790855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523019b2a0 con 0x7f52300ffa40 2024-04-05T21:47:43.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.937+0000 7f5226ffd700 1 -- 172.21.15.121:0/1522790855 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5220005500 con 0x7f52300ffa40 2024-04-05T21:47:43.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.937+0000 7f5237d9c700 1 -- 172.21.15.121:0/1522790855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523019b7f0 con 0x7f52300ffa40 2024-04-05T21:47:43.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.938+0000 7f5237d9c700 1 -- 172.21.15.121:0/1522790855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f523004ea90 con 0x7f52300ffa40 2024-04-05T21:47:43.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.938+0000 7f5226ffd700 1 -- 172.21.15.121:0/1522790855 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f52200049a0 con 0x7f52300ffa40 2024-04-05T21:47:43.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.939+0000 7f5226ffd700 1 --2- 172.21.15.121:0/1522790855 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f521c06cce0 0x7f521c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:43.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.939+0000 7f5226ffd700 1 -- 172.21.15.121:0/1522790855 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f522008c1b0 con 0x7f52300ffa40 2024-04-05T21:47:43.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.939+0000 7f5235337700 1 --2- 172.21.15.121:0/1522790855 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f521c06cce0 0x7f521c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:43.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.939+0000 7f5235337700 1 --2- 172.21.15.121:0/1522790855 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f521c06cce0 0x7f521c06f1a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f522c006010 tx=0x7f522c00b540 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:43.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:43.943+0000 7f5226ffd700 1 -- 172.21.15.121:0/1522790855 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f52200562a0 con 0x7f52300ffa40 2024-04-05T21:47:44.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.154+0000 7f5237d9c700 1 -- 172.21.15.121:0/1522790855 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f523019bd70 con 0x7f521c06cce0 2024-04-05T21:47:44.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.155+0000 7f5226ffd700 1 -- 172.21.15.121:0/1522790855 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f523019bd70 con 0x7f521c06cce0 2024-04-05T21:47:44.155 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:47:44.155 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:47:44.156 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:47:44.156 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:47:44.156 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:47:44.156 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:47:44.156 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:47:44.156 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:47:44.156 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:47:44.156 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:47:44.156 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:47:44.156 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:47:44.156 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:47:44.156 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:47:44.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.157+0000 7f5237d9c700 1 -- 172.21.15.121:0/1522790855 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f521c06cce0 msgr2=0x7f521c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:44.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.157+0000 7f5237d9c700 1 --2- 172.21.15.121:0/1522790855 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f521c06cce0 0x7f521c06f1a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f522c006010 tx=0x7f522c00b540 comp rx=0 tx=0).stop 2024-04-05T21:47:44.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.157+0000 7f5237d9c700 1 -- 172.21.15.121:0/1522790855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52300ffa40 msgr2=0x7f5230196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:44.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.157+0000 7f5237d9c700 1 --2- 172.21.15.121:0/1522790855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52300ffa40 0x7f5230196610 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f522000c8a0 tx=0x7f522000cc60 comp rx=0 tx=0).stop 2024-04-05T21:47:44.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.157+0000 7f5237d9c700 1 -- 172.21.15.121:0/1522790855 shutdown_connections 2024-04-05T21:47:44.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.157+0000 7f5237d9c700 1 --2- 172.21.15.121:0/1522790855 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f521c06cce0 0x7f521c06f1a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:44.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.157+0000 7f5237d9c700 1 --2- 172.21.15.121:0/1522790855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52300ffa40 0x7f5230196610 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:44.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.157+0000 7f5237d9c700 1 --2- 172.21.15.121:0/1522790855 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5230105a40 0x7f5230196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:44.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.157+0000 7f5237d9c700 1 -- 172.21.15.121:0/1522790855 >> 172.21.15.121:0/1522790855 conn(0x7f52300fb570 msgr2=0x7f52301037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:44.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.157+0000 7f5237d9c700 1 -- 172.21.15.121:0/1522790855 shutdown_connections 2024-04-05T21:47:44.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.157+0000 7f5237d9c700 1 -- 172.21.15.121:0/1522790855 wait complete. 2024-04-05T21:47:44.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:43 smithi121 ceph-mon[145859]: pgmap v644: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:44.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:43 smithi121 ceph-mon[145859]: from='client.35014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:44.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:43 smithi121 ceph-mon[145859]: from='client.35018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:44.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:43 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2022912854' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:47:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.260+0000 7f70c9929700 1 -- 172.21.15.121:0/2710798976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c40ffbd0 msgr2=0x7f70c4100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.260+0000 7f70c9929700 1 --2- 172.21.15.121:0/2710798976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c40ffbd0 0x7f70c4100050 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009910 tx=0x7f70b4009c20 comp rx=0 tx=0).stop 2024-04-05T21:47:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.260+0000 7f70c9929700 1 -- 172.21.15.121:0/2710798976 shutdown_connections 2024-04-05T21:47:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.260+0000 7f70c9929700 1 --2- 172.21.15.121:0/2710798976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c40ffbd0 0x7f70c4100050 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.260+0000 7f70c9929700 1 --2- 172.21.15.121:0/2710798976 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70c4105a40 0x7f70c4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.260+0000 7f70c9929700 1 -- 172.21.15.121:0/2710798976 >> 172.21.15.121:0/2710798976 conn(0x7f70c40755a0 msgr2=0x7f70c40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.260+0000 7f70c9929700 1 -- 172.21.15.121:0/2710798976 shutdown_connections 2024-04-05T21:47:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.260+0000 7f70c9929700 1 -- 172.21.15.121:0/2710798976 wait complete. 2024-04-05T21:47:44.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c9929700 1 Processor -- start 2024-04-05T21:47:44.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c9929700 1 -- start start 2024-04-05T21:47:44.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c9929700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70c40ffbd0 0x7f70c4195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:44.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c9929700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c4105a40 0x7f70c4195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:44.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c9929700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4196160 con 0x7f70c4105a40 2024-04-05T21:47:44.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c9929700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4199ef0 con 0x7f70c40ffbd0 2024-04-05T21:47:44.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c27fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c4105a40 0x7f70c4195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:44.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c27fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c4105a40 0x7f70c4195a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48022/0 (socket says 172.21.15.121:48022) 2024-04-05T21:47:44.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c27fc700 1 -- 172.21.15.121:0/3063715688 learned_addr learned my addr 172.21.15.121:0/3063715688 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:47:44.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c2ffd700 1 --2- 172.21.15.121:0/3063715688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70c40ffbd0 0x7f70c4195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:44.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c27fc700 1 -- 172.21.15.121:0/3063715688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70c40ffbd0 msgr2=0x7f70c4195540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:44.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c27fc700 1 --2- 172.21.15.121:0/3063715688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70c40ffbd0 0x7f70c4195540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:44.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c27fc700 1 -- 172.21.15.121:0/3063715688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b40095f0 con 0x7f70c4105a40 2024-04-05T21:47:44.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.261+0000 7f70c27fc700 1 --2- 172.21.15.121:0/3063715688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c4105a40 0x7f70c4195a80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f70b4006010 tx=0x7f70b400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:44.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.262+0000 7f70c8927700 1 -- 172.21.15.121:0/3063715688 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70b401d070 con 0x7f70c4105a40 2024-04-05T21:47:44.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.262+0000 7f70c9929700 1 -- 172.21.15.121:0/3063715688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c419a170 con 0x7f70c4105a40 2024-04-05T21:47:44.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.262+0000 7f70c8927700 1 -- 172.21.15.121:0/3063715688 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70b400f460 con 0x7f70c4105a40 2024-04-05T21:47:44.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.262+0000 7f70c8927700 1 -- 172.21.15.121:0/3063715688 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70b4021620 con 0x7f70c4105a40 2024-04-05T21:47:44.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.262+0000 7f70c9929700 1 -- 172.21.15.121:0/3063715688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c419a660 con 0x7f70c4105a40 2024-04-05T21:47:44.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.263+0000 7f70c9929700 1 -- 172.21.15.121:0/3063715688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c4107fd0 con 0x7f70c4105a40 2024-04-05T21:47:44.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.263+0000 7f70c8927700 1 -- 172.21.15.121:0/3063715688 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f70b400f5d0 con 0x7f70c4105a40 2024-04-05T21:47:44.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.263+0000 7f70c8927700 1 --2- 172.21.15.121:0/3063715688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70b006cd30 0x7f70b006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:47:44.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.264+0000 7f70c8927700 1 -- 172.21.15.121:0/3063715688 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f70b408d8f0 con 0x7f70c4105a40 2024-04-05T21:47:44.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.264+0000 7f70c2ffd700 1 --2- 172.21.15.121:0/3063715688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70b006cd30 0x7f70b006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:47:44.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.264+0000 7f70c2ffd700 1 --2- 172.21.15.121:0/3063715688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70b006cd30 0x7f70b006f1f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f70ac005950 tx=0x7f70ac00a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:47:44.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.268+0000 7f70c8927700 1 -- 172.21.15.121:0/3063715688 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f70b40578c0 con 0x7f70c4105a40 2024-04-05T21:47:44.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:43 smithi188 ceph-mon[107793]: pgmap v644: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:44.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:43 smithi188 ceph-mon[107793]: from='client.35014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:44.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:43 smithi188 ceph-mon[107793]: from='client.35018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:44.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:43 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2022912854' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:47:44.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.539+0000 7f70c9929700 1 -- 172.21.15.121:0/3063715688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f70c402cd60 con 0x7f70c4105a40 2024-04-05T21:47:44.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.540+0000 7f70c8927700 1 -- 172.21.15.121:0/3063715688 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f70b405aee0 con 0x7f70c4105a40 2024-04-05T21:47:44.540 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:47:44.540 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:47:44.540 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:47:44.541 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:47:44.541 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:47:44.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.542+0000 7f70c9929700 1 -- 172.21.15.121:0/3063715688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70b006cd30 msgr2=0x7f70b006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:44.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.542+0000 7f70c9929700 1 --2- 172.21.15.121:0/3063715688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70b006cd30 0x7f70b006f1f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f70ac005950 tx=0x7f70ac00a300 comp rx=0 tx=0).stop 2024-04-05T21:47:44.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.542+0000 7f70c9929700 1 -- 172.21.15.121:0/3063715688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c4105a40 msgr2=0x7f70c4195a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:47:44.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.542+0000 7f70c9929700 1 --2- 172.21.15.121:0/3063715688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c4105a40 0x7f70c4195a80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f70b4006010 tx=0x7f70b400bd70 comp rx=0 tx=0).stop 2024-04-05T21:47:44.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.542+0000 7f70c9929700 1 -- 172.21.15.121:0/3063715688 shutdown_connections 2024-04-05T21:47:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.542+0000 7f70c9929700 1 --2- 172.21.15.121:0/3063715688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f70b006cd30 0x7f70b006f1f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.542+0000 7f70c9929700 1 --2- 172.21.15.121:0/3063715688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f70c40ffbd0 0x7f70c4195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.542+0000 7f70c9929700 1 --2- 172.21.15.121:0/3063715688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c4105a40 0x7f70c4195a80 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:47:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.542+0000 7f70c9929700 1 -- 172.21.15.121:0/3063715688 >> 172.21.15.121:0/3063715688 conn(0x7f70c40755a0 msgr2=0x7f70c40fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:47:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.542+0000 7f70c9929700 1 -- 172.21.15.121:0/3063715688 shutdown_connections 2024-04-05T21:47:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:47:44.542+0000 7f70c9929700 1 -- 172.21.15.121:0/3063715688 wait complete. 2024-04-05T21:47:45.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:44 smithi121 ceph-mon[145859]: from='client.35022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:45.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:44 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2474059577' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:47:45.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:44 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3063715688' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:47:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:44 smithi188 ceph-mon[107793]: from='client.35022 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:44 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2474059577' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:47:45.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:44 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3063715688' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:47:46.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:45 smithi121 ceph-mon[145859]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:46.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:45 smithi121 ceph-mon[145859]: pgmap v645: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:46.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:47:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:45 smithi188 ceph-mon[107793]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:47:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:45 smithi188 ceph-mon[107793]: pgmap v645: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:47:48.241 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:47 smithi121 ceph-mon[145859]: pgmap v646: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:48.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:47 smithi188 ceph-mon[107793]: pgmap v646: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:50.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:49 smithi121 ceph-mon[145859]: pgmap v647: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:50.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:49 smithi188 ceph-mon[107793]: pgmap v647: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:52.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:51 smithi121 ceph-mon[145859]: pgmap v648: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:51 smithi188 ceph-mon[107793]: pgmap v648: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:54.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:53 smithi121 ceph-mon[145859]: pgmap v649: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:54.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:53 smithi188 ceph-mon[107793]: pgmap v649: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:56.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:55 smithi121 ceph-mon[145859]: pgmap v650: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:56.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:55 smithi188 ceph-mon[107793]: pgmap v650: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:58.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:57 smithi121 ceph-mon[145859]: pgmap v651: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:47:58.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:57 smithi188 ceph-mon[107793]: pgmap v651: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:00.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:47:59 smithi121 ceph-mon[145859]: pgmap v652: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:00.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:47:59 smithi188 ceph-mon[107793]: pgmap v652: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:48:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:48:02.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:01 smithi121 ceph-mon[145859]: pgmap v653: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:02.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:01 smithi188 ceph-mon[107793]: pgmap v653: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:03.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:03 smithi121 ceph-mon[145859]: pgmap v654: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:03.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:03 smithi188 ceph-mon[107793]: pgmap v654: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:05.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:05 smithi121 ceph-mon[145859]: pgmap v655: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:05 smithi188 ceph-mon[107793]: pgmap v655: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:07.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:07 smithi121 ceph-mon[145859]: pgmap v656: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:07 smithi188 ceph-mon[107793]: pgmap v656: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:09.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:09 smithi121 ceph-mon[145859]: pgmap v657: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:09 smithi188 ceph-mon[107793]: pgmap v657: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:11.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:11 smithi121 ceph-mon[145859]: pgmap v658: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:11.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:11 smithi188 ceph-mon[107793]: pgmap v658: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:13.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:13 smithi121 ceph-mon[145859]: pgmap v659: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:13.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:13 smithi188 ceph-mon[107793]: pgmap v659: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:14.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.658+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/3971150316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd88101770 msgr2=0x7fdd881057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:14.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.658+0000 7fdd8f5d7700 1 --2- 172.21.15.121:0/3971150316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd88101770 0x7fdd881057c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdd78009910 tx=0x7fdd78009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:14.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.658+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/3971150316 shutdown_connections 2024-04-05T21:48:14.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.658+0000 7fdd8f5d7700 1 --2- 172.21.15.121:0/3971150316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd88101770 0x7fdd881057c0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:14.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.658+0000 7fdd8f5d7700 1 --2- 172.21.15.121:0/3971150316 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd88100dc0 0x7fdd881011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:14.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.658+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/3971150316 >> 172.21.15.121:0/3971150316 conn(0x7fdd880fc650 msgr2=0x7fdd880fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:14.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.658+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/3971150316 shutdown_connections 2024-04-05T21:48:14.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.658+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/3971150316 wait complete. 2024-04-05T21:48:14.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.658+0000 7fdd8f5d7700 1 Processor -- start 2024-04-05T21:48:14.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.658+0000 7fdd8f5d7700 1 -- start start 2024-04-05T21:48:14.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8f5d7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd88100dc0 0x7fdd881966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:14.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8f5d7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd88101770 0x7fdd88196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:14.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8f5d7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd881972a0 con 0x7fdd88100dc0 2024-04-05T21:48:14.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8f5d7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd88199fd0 con 0x7fdd88101770 2024-04-05T21:48:14.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8d373700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd88100dc0 0x7fdd881966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:14.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8d373700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd88100dc0 0x7fdd881966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58382/0 (socket says 172.21.15.121:58382) 2024-04-05T21:48:14.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8d373700 1 -- 172.21.15.121:0/1587574639 learned_addr learned my addr 172.21.15.121:0/1587574639 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:14.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8d373700 1 -- 172.21.15.121:0/1587574639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd88101770 msgr2=0x7fdd88196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:14.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8cb72700 1 --2- 172.21.15.121:0/1587574639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd88101770 0x7fdd88196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:14.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8d373700 1 --2- 172.21.15.121:0/1587574639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd88101770 0x7fdd88196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:14.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8d373700 1 -- 172.21.15.121:0/1587574639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd780095f0 con 0x7fdd88100dc0 2024-04-05T21:48:14.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8cb72700 1 --2- 172.21.15.121:0/1587574639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd88101770 0x7fdd88196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:48:14.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8d373700 1 --2- 172.21.15.121:0/1587574639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd88100dc0 0x7fdd881966d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400c8a0 tx=0x7fdd8400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:14.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd7e7fc700 1 -- 172.21.15.121:0/1587574639 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd8400cea0 con 0x7fdd88100dc0 2024-04-05T21:48:14.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/1587574639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8819a2b0 con 0x7fdd88100dc0 2024-04-05T21:48:14.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd7e7fc700 1 -- 172.21.15.121:0/1587574639 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd84004830 con 0x7fdd88100dc0 2024-04-05T21:48:14.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd7e7fc700 1 -- 172.21.15.121:0/1587574639 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd84005500 con 0x7fdd88100dc0 2024-04-05T21:48:14.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.659+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/1587574639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8819a800 con 0x7fdd88100dc0 2024-04-05T21:48:14.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.660+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/1587574639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd881091c0 con 0x7fdd88100dc0 2024-04-05T21:48:14.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.661+0000 7fdd7e7fc700 1 -- 172.21.15.121:0/1587574639 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdd840049a0 con 0x7fdd88100dc0 2024-04-05T21:48:14.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.661+0000 7fdd7e7fc700 1 --2- 172.21.15.121:0/1587574639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdd7406cce0 0x7fdd7406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:14.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.661+0000 7fdd7e7fc700 1 -- 172.21.15.121:0/1587574639 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdd8408c2f0 con 0x7fdd88100dc0 2024-04-05T21:48:14.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.661+0000 7fdd8cb72700 1 --2- 172.21.15.121:0/1587574639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdd7406cce0 0x7fdd7406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:14.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.662+0000 7fdd8cb72700 1 --2- 172.21.15.121:0/1587574639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdd7406cce0 0x7fdd7406f1a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdd78006010 tx=0x7fdd78005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:14.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.666+0000 7fdd7e7fc700 1 -- 172.21.15.121:0/1587574639 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdd840562c0 con 0x7fdd88100dc0 2024-04-05T21:48:14.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.874+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/1587574639 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd8819aaf0 con 0x7fdd7406cce0 2024-04-05T21:48:14.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.875+0000 7fdd7e7fc700 1 -- 172.21.15.121:0/1587574639 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdd8819aaf0 con 0x7fdd7406cce0 2024-04-05T21:48:14.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.877+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/1587574639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdd7406cce0 msgr2=0x7fdd7406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:14.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.877+0000 7fdd8f5d7700 1 --2- 172.21.15.121:0/1587574639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdd7406cce0 0x7fdd7406f1a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdd78006010 tx=0x7fdd78005960 comp rx=0 tx=0).stop 2024-04-05T21:48:14.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.877+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/1587574639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd88100dc0 msgr2=0x7fdd881966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:14.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.877+0000 7fdd8f5d7700 1 --2- 172.21.15.121:0/1587574639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd88100dc0 0x7fdd881966d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400c8a0 tx=0x7fdd8400cc60 comp rx=0 tx=0).stop 2024-04-05T21:48:14.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.877+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/1587574639 shutdown_connections 2024-04-05T21:48:14.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.877+0000 7fdd8f5d7700 1 --2- 172.21.15.121:0/1587574639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdd7406cce0 0x7fdd7406f1a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:14.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.877+0000 7fdd8f5d7700 1 --2- 172.21.15.121:0/1587574639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd88100dc0 0x7fdd881966d0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:14.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.877+0000 7fdd8f5d7700 1 --2- 172.21.15.121:0/1587574639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdd88101770 0x7fdd88196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:14.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.877+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/1587574639 >> 172.21.15.121:0/1587574639 conn(0x7fdd880fc650 msgr2=0x7fdd880fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:14.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.877+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/1587574639 shutdown_connections 2024-04-05T21:48:14.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.877+0000 7fdd8f5d7700 1 -- 172.21.15.121:0/1587574639 wait complete. 2024-04-05T21:48:14.888 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:48:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.980+0000 7f2987c59700 1 -- 172.21.15.121:0/2812597423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2980100e10 msgr2=0x7f29801011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.981+0000 7f2987c59700 1 --2- 172.21.15.121:0/2812597423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2980100e10 0x7f29801011f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f297c009910 tx=0x7f297c009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.981+0000 7f2987c59700 1 -- 172.21.15.121:0/2812597423 shutdown_connections 2024-04-05T21:48:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.981+0000 7f2987c59700 1 --2- 172.21.15.121:0/2812597423 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29801017c0 0x7f2980105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.981+0000 7f2987c59700 1 --2- 172.21.15.121:0/2812597423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2980100e10 0x7f29801011f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.981+0000 7f2987c59700 1 -- 172.21.15.121:0/2812597423 >> 172.21.15.121:0/2812597423 conn(0x7f29800fc6c0 msgr2=0x7f29800feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.981+0000 7f2987c59700 1 -- 172.21.15.121:0/2812597423 shutdown_connections 2024-04-05T21:48:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.981+0000 7f2987c59700 1 -- 172.21.15.121:0/2812597423 wait complete. 2024-04-05T21:48:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.981+0000 7f2987c59700 1 Processor -- start 2024-04-05T21:48:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.981+0000 7f2987c59700 1 -- start start 2024-04-05T21:48:14.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.981+0000 7f2987c59700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2980100e10 0x7f298019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.981+0000 7f2987c59700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29801017c0 0x7f298019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.981+0000 7f2987c59700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298019d0d0 con 0x7f2980100e10 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.981+0000 7f2987c59700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2980196580 con 0x7f29801017c0 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.982+0000 7f29859f5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2980100e10 0x7f298019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.982+0000 7f29859f5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2980100e10 0x7f298019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58388/0 (socket says 172.21.15.121:58388) 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.982+0000 7f29859f5700 1 -- 172.21.15.121:0/3220419127 learned_addr learned my addr 172.21.15.121:0/3220419127 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.982+0000 7f29859f5700 1 -- 172.21.15.121:0/3220419127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29801017c0 msgr2=0x7f298019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.982+0000 7f29859f5700 1 --2- 172.21.15.121:0/3220419127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29801017c0 0x7f298019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.982+0000 7f29859f5700 1 -- 172.21.15.121:0/3220419127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297c0095f0 con 0x7f2980100e10 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.982+0000 7f29859f5700 1 --2- 172.21.15.121:0/3220419127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2980100e10 0x7f298019c500 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f297c006010 tx=0x7f297c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.982+0000 7f2976ffd700 1 -- 172.21.15.121:0/3220419127 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f297c01d070 con 0x7f2980100e10 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.982+0000 7f2976ffd700 1 -- 172.21.15.121:0/3220419127 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f297c00f460 con 0x7f2980100e10 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.982+0000 7f2987c59700 1 -- 172.21.15.121:0/3220419127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2980196800 con 0x7f2980100e10 2024-04-05T21:48:14.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.982+0000 7f2976ffd700 1 -- 172.21.15.121:0/3220419127 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f297c021640 con 0x7f2980100e10 2024-04-05T21:48:14.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.982+0000 7f2987c59700 1 -- 172.21.15.121:0/3220419127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2980196cf0 con 0x7f2980100e10 2024-04-05T21:48:14.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.983+0000 7f2987c59700 1 -- 172.21.15.121:0/3220419127 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29801091b0 con 0x7f2980100e10 2024-04-05T21:48:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.984+0000 7f2976ffd700 1 -- 172.21.15.121:0/3220419127 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f297c00fa60 con 0x7f2980100e10 2024-04-05T21:48:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.984+0000 7f2976ffd700 1 --2- 172.21.15.121:0/3220419127 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f296c06cce0 0x7f296c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.984+0000 7f2976ffd700 1 -- 172.21.15.121:0/3220419127 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f297c08dba0 con 0x7f2980100e10 2024-04-05T21:48:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.984+0000 7f29851f4700 1 --2- 172.21.15.121:0/3220419127 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f296c06cce0 0x7f296c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:14.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.985+0000 7f29851f4700 1 --2- 172.21.15.121:0/3220419127 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f296c06cce0 0x7f296c06f1a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f29800fce90 tx=0x7f2970005c80 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:14.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:14.988+0000 7f2976ffd700 1 -- 172.21.15.121:0/3220419127 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f297c05b7c0 con 0x7f2980100e10 2024-04-05T21:48:15.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.200+0000 7f2987c59700 1 -- 172.21.15.121:0/3220419127 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f298002cd00 con 0x7f296c06cce0 2024-04-05T21:48:15.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.204+0000 7f2976ffd700 1 -- 172.21.15.121:0/3220419127 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f298002cd00 con 0x7f296c06cce0 2024-04-05T21:48:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.206+0000 7f2987c59700 1 -- 172.21.15.121:0/3220419127 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f296c06cce0 msgr2=0x7f296c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.206+0000 7f2987c59700 1 --2- 172.21.15.121:0/3220419127 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f296c06cce0 0x7f296c06f1a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f29800fce90 tx=0x7f2970005c80 comp rx=0 tx=0).stop 2024-04-05T21:48:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.206+0000 7f2987c59700 1 -- 172.21.15.121:0/3220419127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2980100e10 msgr2=0x7f298019c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.206+0000 7f2987c59700 1 --2- 172.21.15.121:0/3220419127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2980100e10 0x7f298019c500 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f297c006010 tx=0x7f297c00be20 comp rx=0 tx=0).stop 2024-04-05T21:48:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.206+0000 7f2987c59700 1 -- 172.21.15.121:0/3220419127 shutdown_connections 2024-04-05T21:48:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.206+0000 7f2987c59700 1 --2- 172.21.15.121:0/3220419127 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f296c06cce0 0x7f296c06f1a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.206+0000 7f2987c59700 1 --2- 172.21.15.121:0/3220419127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2980100e10 0x7f298019c500 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.206+0000 7f2987c59700 1 --2- 172.21.15.121:0/3220419127 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f29801017c0 0x7f298019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.206+0000 7f2987c59700 1 -- 172.21.15.121:0/3220419127 >> 172.21.15.121:0/3220419127 conn(0x7f29800fc6c0 msgr2=0x7f29800fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:15.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.207+0000 7f2987c59700 1 -- 172.21.15.121:0/3220419127 shutdown_connections 2024-04-05T21:48:15.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.207+0000 7f2987c59700 1 -- 172.21.15.121:0/3220419127 wait complete. 2024-04-05T21:48:15.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.309+0000 7f934597a700 1 -- 172.21.15.121:0/690827983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400ffa30 msgr2=0x7f93400ffeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:15.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.309+0000 7f934597a700 1 --2- 172.21.15.121:0/690827983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400ffa30 0x7f93400ffeb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9334009910 tx=0x7f9334009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:15.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.309+0000 7f934597a700 1 -- 172.21.15.121:0/690827983 shutdown_connections 2024-04-05T21:48:15.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.309+0000 7f934597a700 1 --2- 172.21.15.121:0/690827983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400ffa30 0x7f93400ffeb0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.309+0000 7f934597a700 1 --2- 172.21.15.121:0/690827983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9340105a30 0x7f9340105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.309+0000 7f934597a700 1 -- 172.21.15.121:0/690827983 >> 172.21.15.121:0/690827983 conn(0x7f93400fb520 msgr2=0x7f93400fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:15.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.309+0000 7f934597a700 1 -- 172.21.15.121:0/690827983 shutdown_connections 2024-04-05T21:48:15.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.309+0000 7f934597a700 1 -- 172.21.15.121:0/690827983 wait complete. 2024-04-05T21:48:15.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.310+0000 7f934597a700 1 Processor -- start 2024-04-05T21:48:15.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.310+0000 7f934597a700 1 -- start start 2024-04-05T21:48:15.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.310+0000 7f934597a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400ffa30 0x7f9340195560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:15.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.310+0000 7f934597a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9340105a30 0x7f9340195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:15.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.310+0000 7f934597a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9340196180 con 0x7f93400ffa30 2024-04-05T21:48:15.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.310+0000 7f934597a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9340199f10 con 0x7f9340105a30 2024-04-05T21:48:15.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.310+0000 7f933effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400ffa30 0x7f9340195560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:15.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.310+0000 7f933effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400ffa30 0x7f9340195560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58406/0 (socket says 172.21.15.121:58406) 2024-04-05T21:48:15.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.310+0000 7f933effd700 1 -- 172.21.15.121:0/4075995540 learned_addr learned my addr 172.21.15.121:0/4075995540 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:15.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.311+0000 7f933e7fc700 1 --2- 172.21.15.121:0/4075995540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9340105a30 0x7f9340195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:15.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.311+0000 7f933effd700 1 -- 172.21.15.121:0/4075995540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9340105a30 msgr2=0x7f9340195aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:15.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.311+0000 7f933effd700 1 --2- 172.21.15.121:0/4075995540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9340105a30 0x7f9340195aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.311+0000 7f933effd700 1 -- 172.21.15.121:0/4075995540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93340095f0 con 0x7f93400ffa30 2024-04-05T21:48:15.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.311+0000 7f933effd700 1 --2- 172.21.15.121:0/4075995540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400ffa30 0x7f9340195560 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f933000d8d0 tx=0x7f933000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:15.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.311+0000 7f9344978700 1 -- 172.21.15.121:0/4075995540 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93300098a0 con 0x7f93400ffa30 2024-04-05T21:48:15.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.311+0000 7f9344978700 1 -- 172.21.15.121:0/4075995540 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9330010460 con 0x7f93400ffa30 2024-04-05T21:48:15.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.311+0000 7f934597a700 1 -- 172.21.15.121:0/4075995540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934019a1f0 con 0x7f93400ffa30 2024-04-05T21:48:15.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.311+0000 7f9344978700 1 -- 172.21.15.121:0/4075995540 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9330009ea0 con 0x7f93400ffa30 2024-04-05T21:48:15.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.311+0000 7f934597a700 1 -- 172.21.15.121:0/4075995540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f934019a710 con 0x7f93400ffa30 2024-04-05T21:48:15.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.312+0000 7f934597a700 1 -- 172.21.15.121:0/4075995540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9340107fc0 con 0x7f93400ffa30 2024-04-05T21:48:15.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.313+0000 7f9344978700 1 -- 172.21.15.121:0/4075995540 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f93300105d0 con 0x7f93400ffa30 2024-04-05T21:48:15.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.313+0000 7f9344978700 1 --2- 172.21.15.121:0/4075995540 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f932806cce0 0x7f932806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:15.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.313+0000 7f9344978700 1 -- 172.21.15.121:0/4075995540 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f933008c480 con 0x7f93400ffa30 2024-04-05T21:48:15.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.313+0000 7f933e7fc700 1 --2- 172.21.15.121:0/4075995540 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f932806cce0 0x7f932806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:15.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.313+0000 7f933e7fc700 1 --2- 172.21.15.121:0/4075995540 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f932806cce0 0x7f932806f1a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f933400b5c0 tx=0x7f933401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:15.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.317+0000 7f9344978700 1 -- 172.21.15.121:0/4075995540 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f93300564f0 con 0x7f93400ffa30 2024-04-05T21:48:15.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.519+0000 7f934597a700 1 -- 172.21.15.121:0/4075995540 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f934019a9d0 con 0x7f932806cce0 2024-04-05T21:48:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.529+0000 7f9344978700 1 -- 172.21.15.121:0/4075995540 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f934019a9d0 con 0x7f932806cce0 2024-04-05T21:48:15.530 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:48:15.530 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (19m) 13m ago 30m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:48:15.530 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (30m) 13m ago 30m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:48:15.530 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (29m) 12m ago 29m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:48:15.530 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (17m) 13m ago 30m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:48:15.530 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (17m) 12m ago 29m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (18m) 13m ago 30m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (13m) 13m ago 24m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (13m) 13m ago 24m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (12m) 12m ago 24m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (24m) 12m ago 24m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (21m) 13m ago 32m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (20m) 12m ago 29m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (17m) 13m ago 32m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (17m) 12m ago 29m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (19m) 13m ago 30m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (19m) 12m ago 29m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (16m) 13m ago 28m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (16m) 13m ago 28m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:48:15.531 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (15m) 13m ago 27m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:48:15.532 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (14m) 12m ago 27m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:48:15.532 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (14m) 12m ago 26m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:48:15.532 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (13m) 12m ago 26m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:48:15.532 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (19m) 13m ago 29m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:48:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.531+0000 7f934597a700 1 -- 172.21.15.121:0/4075995540 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f932806cce0 msgr2=0x7f932806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.531+0000 7f934597a700 1 --2- 172.21.15.121:0/4075995540 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f932806cce0 0x7f932806f1a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f933400b5c0 tx=0x7f933401a040 comp rx=0 tx=0).stop 2024-04-05T21:48:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.531+0000 7f934597a700 1 -- 172.21.15.121:0/4075995540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400ffa30 msgr2=0x7f9340195560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.531+0000 7f934597a700 1 --2- 172.21.15.121:0/4075995540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400ffa30 0x7f9340195560 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f933000d8d0 tx=0x7f933000dbe0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.531+0000 7f934597a700 1 -- 172.21.15.121:0/4075995540 shutdown_connections 2024-04-05T21:48:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.531+0000 7f934597a700 1 --2- 172.21.15.121:0/4075995540 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f932806cce0 0x7f932806f1a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.531+0000 7f934597a700 1 --2- 172.21.15.121:0/4075995540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400ffa30 0x7f9340195560 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.531+0000 7f934597a700 1 --2- 172.21.15.121:0/4075995540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9340105a30 0x7f9340195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.531+0000 7f934597a700 1 -- 172.21.15.121:0/4075995540 >> 172.21.15.121:0/4075995540 conn(0x7f93400fb520 msgr2=0x7f93400fc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.532+0000 7f934597a700 1 -- 172.21.15.121:0/4075995540 shutdown_connections 2024-04-05T21:48:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.532+0000 7f934597a700 1 -- 172.21.15.121:0/4075995540 wait complete. 2024-04-05T21:48:15.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.633+0000 7f3c46029700 1 -- 172.21.15.121:0/2103240038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c40101d00 msgr2=0x7f3c401020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:15.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.633+0000 7f3c46029700 1 --2- 172.21.15.121:0/2103240038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c40101d00 0x7f3c401020e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3c30009910 tx=0x7f3c30009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:15.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.633+0000 7f3c46029700 1 -- 172.21.15.121:0/2103240038 shutdown_connections 2024-04-05T21:48:15.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.633+0000 7f3c46029700 1 --2- 172.21.15.121:0/2103240038 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c40102620 0x7f3c4010ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.633+0000 7f3c46029700 1 --2- 172.21.15.121:0/2103240038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c40101d00 0x7f3c401020e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.633+0000 7f3c46029700 1 -- 172.21.15.121:0/2103240038 >> 172.21.15.121:0/2103240038 conn(0x7f3c400fb570 msgr2=0x7f3c400fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:15.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.633+0000 7f3c46029700 1 -- 172.21.15.121:0/2103240038 shutdown_connections 2024-04-05T21:48:15.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.633+0000 7f3c46029700 1 -- 172.21.15.121:0/2103240038 wait complete. 2024-04-05T21:48:15.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.634+0000 7f3c46029700 1 Processor -- start 2024-04-05T21:48:15.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.634+0000 7f3c46029700 1 -- start start 2024-04-05T21:48:15.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.634+0000 7f3c46029700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c40101d00 0x7f3c40196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:15.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.634+0000 7f3c46029700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c40102620 0x7f3c40196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:15.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.634+0000 7f3c46029700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c40197250 con 0x7f3c40102620 2024-04-05T21:48:15.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.634+0000 7f3c46029700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c40199f80 con 0x7f3c40101d00 2024-04-05T21:48:15.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.634+0000 7f3c3effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c40102620 0x7f3c40196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:15.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.634+0000 7f3c3effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c40102620 0x7f3c40196bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58420/0 (socket says 172.21.15.121:58420) 2024-04-05T21:48:15.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.634+0000 7f3c3effd700 1 -- 172.21.15.121:0/2579524056 learned_addr learned my addr 172.21.15.121:0/2579524056 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:15.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.634+0000 7f3c3f7fe700 1 --2- 172.21.15.121:0/2579524056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c40101d00 0x7f3c40196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:15.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.634+0000 7f3c3effd700 1 -- 172.21.15.121:0/2579524056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c40101d00 msgr2=0x7f3c40196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.634+0000 7f3c3effd700 1 --2- 172.21.15.121:0/2579524056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c40101d00 0x7f3c40196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.634+0000 7f3c3effd700 1 -- 172.21.15.121:0/2579524056 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c300095f0 con 0x7f3c40102620 2024-04-05T21:48:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.635+0000 7f3c3effd700 1 --2- 172.21.15.121:0/2579524056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c40102620 0x7f3c40196bc0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3c2800e940 tx=0x7f3c2800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.635+0000 7f3c3cff9700 1 -- 172.21.15.121:0/2579524056 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c280096e0 con 0x7f3c40102620 2024-04-05T21:48:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.635+0000 7f3c3cff9700 1 -- 172.21.15.121:0/2579524056 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c2800c7c0 con 0x7f3c40102620 2024-04-05T21:48:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.635+0000 7f3c3cff9700 1 -- 172.21.15.121:0/2579524056 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c28005420 con 0x7f3c40102620 2024-04-05T21:48:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.635+0000 7f3c46029700 1 -- 172.21.15.121:0/2579524056 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c4019a260 con 0x7f3c40102620 2024-04-05T21:48:15.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.635+0000 7f3c46029700 1 -- 172.21.15.121:0/2579524056 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4019a7b0 con 0x7f3c40102620 2024-04-05T21:48:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.636+0000 7f3c46029700 1 -- 172.21.15.121:0/2579524056 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c40108210 con 0x7f3c40102620 2024-04-05T21:48:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.636+0000 7f3c3cff9700 1 -- 172.21.15.121:0/2579524056 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3c2800c930 con 0x7f3c40102620 2024-04-05T21:48:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.637+0000 7f3c3cff9700 1 --2- 172.21.15.121:0/2579524056 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3c2c06cce0 0x7f3c2c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.637+0000 7f3c3cff9700 1 -- 172.21.15.121:0/2579524056 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3c28014070 con 0x7f3c40102620 2024-04-05T21:48:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.637+0000 7f3c3f7fe700 1 --2- 172.21.15.121:0/2579524056 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3c2c06cce0 0x7f3c2c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:15.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.637+0000 7f3c3f7fe700 1 --2- 172.21.15.121:0/2579524056 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3c2c06cce0 0x7f3c2c06f1a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3c30009fd0 tx=0x7f3c30005940 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:15.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.641+0000 7f3c3cff9700 1 -- 172.21.15.121:0/2579524056 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3c28056830 con 0x7f3c40102620 2024-04-05T21:48:15.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:15 smithi121 ceph-mon[145859]: pgmap v660: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:15 smithi121 ceph-mon[145859]: from='client.35042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:15.739 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:48:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:15 smithi188 ceph-mon[107793]: pgmap v660: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:15 smithi188 ceph-mon[107793]: from='client.35042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:48:15.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.912+0000 7f3c46029700 1 -- 172.21.15.121:0/2579524056 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3c4004ea90 con 0x7f3c40102620 2024-04-05T21:48:15.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.913+0000 7f3c3cff9700 1 -- 172.21.15.121:0/2579524056 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f3c2800cbe0 con 0x7f3c40102620 2024-04-05T21:48:15.913 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:48:15.913 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:48:15.913 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:48:15.914 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:48:15.915 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:48:15.915 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:48:15.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.915+0000 7f3c46029700 1 -- 172.21.15.121:0/2579524056 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3c2c06cce0 msgr2=0x7f3c2c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:15.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.915+0000 7f3c46029700 1 --2- 172.21.15.121:0/2579524056 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3c2c06cce0 0x7f3c2c06f1a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3c30009fd0 tx=0x7f3c30005940 comp rx=0 tx=0).stop 2024-04-05T21:48:15.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.915+0000 7f3c46029700 1 -- 172.21.15.121:0/2579524056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c40102620 msgr2=0x7f3c40196bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:15.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.915+0000 7f3c46029700 1 --2- 172.21.15.121:0/2579524056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c40102620 0x7f3c40196bc0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3c2800e940 tx=0x7f3c2800ec50 comp rx=0 tx=0).stop 2024-04-05T21:48:15.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.915+0000 7f3c46029700 1 -- 172.21.15.121:0/2579524056 shutdown_connections 2024-04-05T21:48:15.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.915+0000 7f3c46029700 1 --2- 172.21.15.121:0/2579524056 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3c2c06cce0 0x7f3c2c06f1a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.915+0000 7f3c46029700 1 --2- 172.21.15.121:0/2579524056 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3c40101d00 0x7f3c40196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.915+0000 7f3c46029700 1 --2- 172.21.15.121:0/2579524056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c40102620 0x7f3c40196bc0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:15.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.915+0000 7f3c46029700 1 -- 172.21.15.121:0/2579524056 >> 172.21.15.121:0/2579524056 conn(0x7f3c400fb570 msgr2=0x7f3c40105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:15.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.915+0000 7f3c46029700 1 -- 172.21.15.121:0/2579524056 shutdown_connections 2024-04-05T21:48:15.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:15.915+0000 7f3c46029700 1 -- 172.21.15.121:0/2579524056 wait complete. 2024-04-05T21:48:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.016+0000 7f1df9fcc700 1 -- 172.21.15.121:0/239348964 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1df4100de0 msgr2=0x7f1df41011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.016+0000 7f1df9fcc700 1 --2- 172.21.15.121:0/239348964 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1df4100de0 0x7f1df41011c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009910 tx=0x7f1de4009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.016+0000 7f1df9fcc700 1 -- 172.21.15.121:0/239348964 shutdown_connections 2024-04-05T21:48:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.016+0000 7f1df9fcc700 1 --2- 172.21.15.121:0/239348964 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1df4101790 0x7f1df41057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.016+0000 7f1df9fcc700 1 --2- 172.21.15.121:0/239348964 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1df4100de0 0x7f1df41011c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.016+0000 7f1df9fcc700 1 -- 172.21.15.121:0/239348964 >> 172.21.15.121:0/239348964 conn(0x7f1df40fc670 msgr2=0x7f1df40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.016+0000 7f1df9fcc700 1 -- 172.21.15.121:0/239348964 shutdown_connections 2024-04-05T21:48:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.016+0000 7f1df9fcc700 1 -- 172.21.15.121:0/239348964 wait complete. 2024-04-05T21:48:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.017+0000 7f1df9fcc700 1 Processor -- start 2024-04-05T21:48:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.017+0000 7f1df9fcc700 1 -- start start 2024-04-05T21:48:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.017+0000 7f1df9fcc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1df4100de0 0x7f1df419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.017+0000 7f1df9fcc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1df4101790 0x7f1df419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.017+0000 7f1df9fcc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df419f1f0 con 0x7f1df4101790 2024-04-05T21:48:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.017+0000 7f1df9fcc700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df41986a0 con 0x7f1df4100de0 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.017+0000 7f1df2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1df4101790 0x7f1df419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.017+0000 7f1df2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1df4101790 0x7f1df419eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58426/0 (socket says 172.21.15.121:58426) 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.017+0000 7f1df2ffd700 1 -- 172.21.15.121:0/2753591148 learned_addr learned my addr 172.21.15.121:0/2753591148 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.018+0000 7f1df37fe700 1 --2- 172.21.15.121:0/2753591148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1df4100de0 0x7f1df419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.018+0000 7f1df2ffd700 1 -- 172.21.15.121:0/2753591148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1df4100de0 msgr2=0x7f1df419e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.018+0000 7f1df2ffd700 1 --2- 172.21.15.121:0/2753591148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1df4100de0 0x7f1df419e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.018+0000 7f1df2ffd700 1 -- 172.21.15.121:0/2753591148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de40095f0 con 0x7f1df4101790 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.018+0000 7f1df2ffd700 1 --2- 172.21.15.121:0/2753591148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1df4101790 0x7f1df419eb60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1de800c8f0 tx=0x7f1de800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.018+0000 7f1df0ff9700 1 -- 172.21.15.121:0/2753591148 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1de8007a50 con 0x7f1df4101790 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.018+0000 7f1df9fcc700 1 -- 172.21.15.121:0/2753591148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df4198980 con 0x7f1df4101790 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.018+0000 7f1df0ff9700 1 -- 172.21.15.121:0/2753591148 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1de8007bb0 con 0x7f1df4101790 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.018+0000 7f1df0ff9700 1 -- 172.21.15.121:0/2753591148 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1de8018630 con 0x7f1df4101790 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.018+0000 7f1df9fcc700 1 -- 172.21.15.121:0/2753591148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df4198ed0 con 0x7f1df4101790 2024-04-05T21:48:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.019+0000 7f1df9fcc700 1 -- 172.21.15.121:0/2753591148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df4104f60 con 0x7f1df4101790 2024-04-05T21:48:16.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.020+0000 7f1df0ff9700 1 -- 172.21.15.121:0/2753591148 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1de8007d20 con 0x7f1df4101790 2024-04-05T21:48:16.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.020+0000 7f1df0ff9700 1 --2- 172.21.15.121:0/2753591148 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1ddc06cce0 0x7f1ddc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:16.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.020+0000 7f1df0ff9700 1 -- 172.21.15.121:0/2753591148 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1de808c590 con 0x7f1df4101790 2024-04-05T21:48:16.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.020+0000 7f1df37fe700 1 --2- 172.21.15.121:0/2753591148 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1ddc06cce0 0x7f1ddc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:16.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.020+0000 7f1df37fe700 1 --2- 172.21.15.121:0/2753591148 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1ddc06cce0 0x7f1ddc06f1a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1de400b5c0 tx=0x7f1de4005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.024+0000 7f1df0ff9700 1 -- 172.21.15.121:0/2753591148 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1de805a090 con 0x7f1df4101790 2024-04-05T21:48:16.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.258+0000 7f1df9fcc700 1 -- 172.21.15.121:0/2753591148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1df404ea90 con 0x7f1df4101790 2024-04-05T21:48:16.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.258+0000 7f1df0ff9700 1 -- 172.21.15.121:0/2753591148 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f1de8059c20 con 0x7f1df4101790 2024-04-05T21:48:16.259 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:48:16.259 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:48:16.259 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:48:16.259 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:48:16.259 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:48:16.259 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:48:16.259 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:48:16.260 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:48:16.261 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:48:16.262 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:48:16.262 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:48:16.262 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:48:16.262 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:48:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.260+0000 7f1df9fcc700 1 -- 172.21.15.121:0/2753591148 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1ddc06cce0 msgr2=0x7f1ddc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.260+0000 7f1df9fcc700 1 --2- 172.21.15.121:0/2753591148 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1ddc06cce0 0x7f1ddc06f1a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1de400b5c0 tx=0x7f1de4005fd0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.260+0000 7f1df9fcc700 1 -- 172.21.15.121:0/2753591148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1df4101790 msgr2=0x7f1df419eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.260+0000 7f1df9fcc700 1 --2- 172.21.15.121:0/2753591148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1df4101790 0x7f1df419eb60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1de800c8f0 tx=0x7f1de800ccb0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.260+0000 7f1df9fcc700 1 -- 172.21.15.121:0/2753591148 shutdown_connections 2024-04-05T21:48:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.260+0000 7f1df9fcc700 1 --2- 172.21.15.121:0/2753591148 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1ddc06cce0 0x7f1ddc06f1a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.260+0000 7f1df9fcc700 1 --2- 172.21.15.121:0/2753591148 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1df4100de0 0x7f1df419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.260+0000 7f1df9fcc700 1 --2- 172.21.15.121:0/2753591148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1df4101790 0x7f1df419eb60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.260+0000 7f1df9fcc700 1 -- 172.21.15.121:0/2753591148 >> 172.21.15.121:0/2753591148 conn(0x7f1df40fc670 msgr2=0x7f1df40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.261+0000 7f1df9fcc700 1 -- 172.21.15.121:0/2753591148 shutdown_connections 2024-04-05T21:48:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.261+0000 7f1df9fcc700 1 -- 172.21.15.121:0/2753591148 wait complete. 2024-04-05T21:48:16.263 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:48:16.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.363+0000 7fa34eb0a700 1 -- 172.21.15.121:0/3805811810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3480ffa40 msgr2=0x7fa3480ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:16.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.363+0000 7fa34eb0a700 1 --2- 172.21.15.121:0/3805811810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3480ffa40 0x7fa3480ffec0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fa334009910 tx=0x7fa334009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:16.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.363+0000 7fa34eb0a700 1 -- 172.21.15.121:0/3805811810 shutdown_connections 2024-04-05T21:48:16.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.363+0000 7fa34eb0a700 1 --2- 172.21.15.121:0/3805811810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3480ffa40 0x7fa3480ffec0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.363+0000 7fa34eb0a700 1 --2- 172.21.15.121:0/3805811810 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa348105a40 0x7fa348105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.363+0000 7fa34eb0a700 1 -- 172.21.15.121:0/3805811810 >> 172.21.15.121:0/3805811810 conn(0x7fa3480fb570 msgr2=0x7fa3480fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:16.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.364+0000 7fa34eb0a700 1 -- 172.21.15.121:0/3805811810 shutdown_connections 2024-04-05T21:48:16.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.364+0000 7fa34eb0a700 1 -- 172.21.15.121:0/3805811810 wait complete. 2024-04-05T21:48:16.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.364+0000 7fa34eb0a700 1 Processor -- start 2024-04-05T21:48:16.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.364+0000 7fa34eb0a700 1 -- start start 2024-04-05T21:48:16.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa34eb0a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3480ffa40 0x7fa348197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:16.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa34eb0a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa348105a40 0x7fa3481979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:16.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa34eb0a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa348197ff0 con 0x7fa348105a40 2024-04-05T21:48:16.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa34eb0a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa348198160 con 0x7fa3480ffa40 2024-04-05T21:48:16.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa347fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa348105a40 0x7fa3481979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:16.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa347fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa348105a40 0x7fa3481979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58456/0 (socket says 172.21.15.121:58456) 2024-04-05T21:48:16.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa347fff700 1 -- 172.21.15.121:0/757330212 learned_addr learned my addr 172.21.15.121:0/757330212 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:16.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa34c8a6700 1 --2- 172.21.15.121:0/757330212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3480ffa40 0x7fa348197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:16.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa347fff700 1 -- 172.21.15.121:0/757330212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3480ffa40 msgr2=0x7fa348197490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:16.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa347fff700 1 --2- 172.21.15.121:0/757330212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3480ffa40 0x7fa348197490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa347fff700 1 -- 172.21.15.121:0/757330212 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3340095f0 con 0x7fa348105a40 2024-04-05T21:48:16.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa347fff700 1 --2- 172.21.15.121:0/757330212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa348105a40 0x7fa3481979d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa334006010 tx=0x7fa33400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:16.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa345ffb700 1 -- 172.21.15.121:0/757330212 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa33401d070 con 0x7fa348105a40 2024-04-05T21:48:16.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa345ffb700 1 -- 172.21.15.121:0/757330212 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa33400f460 con 0x7fa348105a40 2024-04-05T21:48:16.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa34eb0a700 1 -- 172.21.15.121:0/757330212 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3481a0500 con 0x7fa348105a40 2024-04-05T21:48:16.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.365+0000 7fa345ffb700 1 -- 172.21.15.121:0/757330212 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa334021640 con 0x7fa348105a40 2024-04-05T21:48:16.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.366+0000 7fa34eb0a700 1 -- 172.21.15.121:0/757330212 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3481a0970 con 0x7fa348105a40 2024-04-05T21:48:16.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.367+0000 7fa34eb0a700 1 -- 172.21.15.121:0/757330212 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa348107fd0 con 0x7fa348105a40 2024-04-05T21:48:16.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.367+0000 7fa345ffb700 1 -- 172.21.15.121:0/757330212 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa33400f9e0 con 0x7fa348105a40 2024-04-05T21:48:16.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.367+0000 7fa345ffb700 1 --2- 172.21.15.121:0/757330212 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa33806cce0 0x7fa33806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:16.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.367+0000 7fa345ffb700 1 -- 172.21.15.121:0/757330212 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa33408d900 con 0x7fa348105a40 2024-04-05T21:48:16.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.367+0000 7fa34c8a6700 1 --2- 172.21.15.121:0/757330212 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa33806cce0 0x7fa33806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:16.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.368+0000 7fa34c8a6700 1 --2- 172.21.15.121:0/757330212 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa33806cce0 0x7fa33806f1a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa33c005fd0 tx=0x7fa33c00a560 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:16.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.372+0000 7fa345ffb700 1 -- 172.21.15.121:0/757330212 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa334057970 con 0x7fa348105a40 2024-04-05T21:48:16.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.584+0000 7fa34eb0a700 1 -- 172.21.15.121:0/757330212 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa34802cce0 con 0x7fa33806cce0 2024-04-05T21:48:16.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.585+0000 7fa345ffb700 1 -- 172.21.15.121:0/757330212 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fa34802cce0 con 0x7fa33806cce0 2024-04-05T21:48:16.661 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:48:16.661 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:48:16.661 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:48:16.661 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:48:16.661 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:48:16.661 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:48:16.661 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:48:16.661 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:48:16.661 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:48:16.661 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:48:16.661 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:48:16.661 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:48:16.662 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:48:16.662 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:48:16.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.587+0000 7fa34eb0a700 1 -- 172.21.15.121:0/757330212 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa33806cce0 msgr2=0x7fa33806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:16.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.587+0000 7fa34eb0a700 1 --2- 172.21.15.121:0/757330212 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa33806cce0 0x7fa33806f1a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa33c005fd0 tx=0x7fa33c00a560 comp rx=0 tx=0).stop 2024-04-05T21:48:16.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.587+0000 7fa34eb0a700 1 -- 172.21.15.121:0/757330212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa348105a40 msgr2=0x7fa3481979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:16.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.587+0000 7fa34eb0a700 1 --2- 172.21.15.121:0/757330212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa348105a40 0x7fa3481979d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa334006010 tx=0x7fa33400be20 comp rx=0 tx=0).stop 2024-04-05T21:48:16.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.587+0000 7fa34eb0a700 1 -- 172.21.15.121:0/757330212 shutdown_connections 2024-04-05T21:48:16.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.587+0000 7fa34eb0a700 1 --2- 172.21.15.121:0/757330212 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa33806cce0 0x7fa33806f1a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.587+0000 7fa34eb0a700 1 --2- 172.21.15.121:0/757330212 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3480ffa40 0x7fa348197490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.587+0000 7fa34eb0a700 1 --2- 172.21.15.121:0/757330212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa348105a40 0x7fa3481979d0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.587+0000 7fa34eb0a700 1 -- 172.21.15.121:0/757330212 >> 172.21.15.121:0/757330212 conn(0x7fa3480fb570 msgr2=0x7fa3480fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:16.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.587+0000 7fa34eb0a700 1 -- 172.21.15.121:0/757330212 shutdown_connections 2024-04-05T21:48:16.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.587+0000 7fa34eb0a700 1 -- 172.21.15.121:0/757330212 wait complete. 2024-04-05T21:48:16.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.689+0000 7fed2b157700 1 -- 172.21.15.121:0/1590101107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed240fef70 msgr2=0x7fed240ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:16.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.689+0000 7fed2b157700 1 --2- 172.21.15.121:0/1590101107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed240fef70 0x7fed240ff3f0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fed10009910 tx=0x7fed10009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:16.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.689+0000 7fed2b157700 1 -- 172.21.15.121:0/1590101107 shutdown_connections 2024-04-05T21:48:16.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.689+0000 7fed2b157700 1 --2- 172.21.15.121:0/1590101107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed240fef70 0x7fed240ff3f0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.689+0000 7fed2b157700 1 --2- 172.21.15.121:0/1590101107 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed24105b50 0x7fed24105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.690+0000 7fed2b157700 1 -- 172.21.15.121:0/1590101107 >> 172.21.15.121:0/1590101107 conn(0x7fed24075e30 msgr2=0x7fed240fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:16.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.690+0000 7fed2b157700 1 -- 172.21.15.121:0/1590101107 shutdown_connections 2024-04-05T21:48:16.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.690+0000 7fed2b157700 1 -- 172.21.15.121:0/1590101107 wait complete. 2024-04-05T21:48:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.690+0000 7fed2b157700 1 Processor -- start 2024-04-05T21:48:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.690+0000 7fed2b157700 1 -- start start 2024-04-05T21:48:16.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed2b157700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed240fef70 0x7fed24196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed2b157700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed24105b50 0x7fed24196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed2b157700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed241972d0 con 0x7fed24105b50 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed2b157700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2419a000 con 0x7fed240fef70 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed23fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed24105b50 0x7fed24196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed23fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed24105b50 0x7fed24196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58486/0 (socket says 172.21.15.121:58486) 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed23fff700 1 -- 172.21.15.121:0/22759122 learned_addr learned my addr 172.21.15.121:0/22759122 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed28ef3700 1 --2- 172.21.15.121:0/22759122 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed240fef70 0x7fed24196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed23fff700 1 -- 172.21.15.121:0/22759122 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed240fef70 msgr2=0x7fed24196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed23fff700 1 --2- 172.21.15.121:0/22759122 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed240fef70 0x7fed24196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed23fff700 1 -- 172.21.15.121:0/22759122 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed100095f0 con 0x7fed24105b50 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed23fff700 1 --2- 172.21.15.121:0/22759122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed24105b50 0x7fed24196c40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fed10006010 tx=0x7fed1000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed21ffb700 1 -- 172.21.15.121:0/22759122 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed1001d070 con 0x7fed24105b50 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed21ffb700 1 -- 172.21.15.121:0/22759122 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed1000f460 con 0x7fed24105b50 2024-04-05T21:48:16.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed2b157700 1 -- 172.21.15.121:0/22759122 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed2419a280 con 0x7fed24105b50 2024-04-05T21:48:16.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.691+0000 7fed21ffb700 1 -- 172.21.15.121:0/22759122 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed10021620 con 0x7fed24105b50 2024-04-05T21:48:16.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.692+0000 7fed2b157700 1 -- 172.21.15.121:0/22759122 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed2419a770 con 0x7fed24105b50 2024-04-05T21:48:16.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.693+0000 7fed21ffb700 1 -- 172.21.15.121:0/22759122 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fed1000fa40 con 0x7fed24105b50 2024-04-05T21:48:16.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.693+0000 7fed2b157700 1 -- 172.21.15.121:0/22759122 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed241090d0 con 0x7fed24105b50 2024-04-05T21:48:16.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.693+0000 7fed21ffb700 1 --2- 172.21.15.121:0/22759122 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fed1406cce0 0x7fed1406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:16.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.694+0000 7fed21ffb700 1 -- 172.21.15.121:0/22759122 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fed1008d6c0 con 0x7fed24105b50 2024-04-05T21:48:16.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.694+0000 7fed28ef3700 1 --2- 172.21.15.121:0/22759122 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fed1406cce0 0x7fed1406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:16.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.694+0000 7fed28ef3700 1 --2- 172.21.15.121:0/22759122 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fed1406cce0 0x7fed1406f1a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fed18005950 tx=0x7fed180058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:16.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.698+0000 7fed21ffb700 1 -- 172.21.15.121:0/22759122 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fed100577b0 con 0x7fed24105b50 2024-04-05T21:48:16.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:16 smithi188 ceph-mon[107793]: from='client.35046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:16.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:16 smithi188 ceph-mon[107793]: from='client.35050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:16.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:16 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2579524056' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:48:16.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:16 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2753591148' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:48:16.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.971+0000 7fed2b157700 1 -- 172.21.15.121:0/22759122 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fed24108ab0 con 0x7fed24105b50 2024-04-05T21:48:16.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.971+0000 7fed21ffb700 1 -- 172.21.15.121:0/22759122 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fed1005add0 con 0x7fed24105b50 2024-04-05T21:48:16.972 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:48:16.972 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:48:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:48:16.972 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:48:16.972 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:48:16.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.973+0000 7fed2b157700 1 -- 172.21.15.121:0/22759122 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fed1406cce0 msgr2=0x7fed1406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:16.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.973+0000 7fed2b157700 1 --2- 172.21.15.121:0/22759122 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fed1406cce0 0x7fed1406f1a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fed18005950 tx=0x7fed180058e0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.973+0000 7fed2b157700 1 -- 172.21.15.121:0/22759122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed24105b50 msgr2=0x7fed24196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:16.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.973+0000 7fed2b157700 1 --2- 172.21.15.121:0/22759122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed24105b50 0x7fed24196c40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fed10006010 tx=0x7fed1000be20 comp rx=0 tx=0).stop 2024-04-05T21:48:16.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.973+0000 7fed2b157700 1 -- 172.21.15.121:0/22759122 shutdown_connections 2024-04-05T21:48:16.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.973+0000 7fed2b157700 1 --2- 172.21.15.121:0/22759122 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fed1406cce0 0x7fed1406f1a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.973+0000 7fed2b157700 1 --2- 172.21.15.121:0/22759122 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fed240fef70 0x7fed24196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.973+0000 7fed2b157700 1 --2- 172.21.15.121:0/22759122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed24105b50 0x7fed24196c40 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:16.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.973+0000 7fed2b157700 1 -- 172.21.15.121:0/22759122 >> 172.21.15.121:0/22759122 conn(0x7fed24075e30 msgr2=0x7fed241045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:16.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.973+0000 7fed2b157700 1 -- 172.21.15.121:0/22759122 shutdown_connections 2024-04-05T21:48:16.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:16.973+0000 7fed2b157700 1 -- 172.21.15.121:0/22759122 wait complete. 2024-04-05T21:48:16.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:16 smithi121 ceph-mon[145859]: from='client.35046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:16.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:16 smithi121 ceph-mon[145859]: from='client.35050 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:16.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:16 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2579524056' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:48:16.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:16 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2753591148' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:48:17.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:17 smithi188 ceph-mon[107793]: pgmap v661: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:17.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:17 smithi188 ceph-mon[107793]: from='client.35062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:17.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:17 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/22759122' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:48:17.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:17 smithi121 ceph-mon[145859]: pgmap v661: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:17 smithi121 ceph-mon[145859]: from='client.35062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:17.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:17 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/22759122' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:48:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:19 smithi188 ceph-mon[107793]: pgmap v662: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:19.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:19 smithi121 ceph-mon[145859]: pgmap v662: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:21 smithi188 ceph-mon[107793]: pgmap v663: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:21.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:21 smithi121 ceph-mon[145859]: pgmap v663: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:23 smithi188 ceph-mon[107793]: pgmap v664: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:23 smithi121 ceph-mon[145859]: pgmap v664: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:25.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:25 smithi188 ceph-mon[107793]: pgmap v665: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:25.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:25 smithi121 ceph-mon[145859]: pgmap v665: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:27.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:27 smithi188 ceph-mon[107793]: pgmap v666: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:27.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:27 smithi121 ceph-mon[145859]: pgmap v666: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:29 smithi188 ceph-mon[107793]: pgmap v667: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:29.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:29 smithi121 ceph-mon[145859]: pgmap v667: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:30.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:48:30.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:48:31.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:31 smithi188 ceph-mon[107793]: pgmap v668: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:31.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:31 smithi121 ceph-mon[145859]: pgmap v668: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:33.770 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:33 smithi121 ceph-mon[145859]: pgmap v669: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:33.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:33 smithi188 ceph-mon[107793]: pgmap v669: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:35.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:35 smithi188 ceph-mon[107793]: pgmap v670: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:35.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:35 smithi121 ceph-mon[145859]: pgmap v670: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:37.780 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:37 smithi188 ceph-mon[107793]: pgmap v671: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:37.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:37 smithi121 ceph-mon[145859]: pgmap v671: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:39.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:39 smithi121 ceph-mon[145859]: pgmap v672: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:40.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:39 smithi188 ceph-mon[107793]: pgmap v672: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:41.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:41 smithi121 ceph-mon[145859]: pgmap v673: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:42.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:41 smithi188 ceph-mon[107793]: pgmap v673: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:43.863 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:43 smithi121 ceph-mon[145859]: pgmap v674: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:44.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:43 smithi188 ceph-mon[107793]: pgmap v674: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:45.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:45 smithi121 ceph-mon[145859]: pgmap v675: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:45.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:48:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:45 smithi188 ceph-mon[107793]: pgmap v675: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:48:47.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.088+0000 7f71acf4d700 1 -- 172.21.15.121:0/2778030253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71a8100da0 msgr2=0x7f71a8101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:47.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.088+0000 7f71acf4d700 1 --2- 172.21.15.121:0/2778030253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71a8100da0 0x7f71a8101180 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f719c009910 tx=0x7f719c009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:47.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.088+0000 7f71acf4d700 1 -- 172.21.15.121:0/2778030253 shutdown_connections 2024-04-05T21:48:47.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.088+0000 7f71acf4d700 1 --2- 172.21.15.121:0/2778030253 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f71a8101750 0x7f71a81057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.088+0000 7f71acf4d700 1 --2- 172.21.15.121:0/2778030253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71a8100da0 0x7f71a8101180 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.088+0000 7f71acf4d700 1 -- 172.21.15.121:0/2778030253 >> 172.21.15.121:0/2778030253 conn(0x7f71a80fc630 msgr2=0x7f71a80fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.088+0000 7f71acf4d700 1 -- 172.21.15.121:0/2778030253 shutdown_connections 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.088+0000 7f71acf4d700 1 -- 172.21.15.121:0/2778030253 wait complete. 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.089+0000 7f71acf4d700 1 Processor -- start 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.089+0000 7f71acf4d700 1 -- start start 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.089+0000 7f71acf4d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f71a8100da0 0x7f71a8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.089+0000 7f71acf4d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71a8101750 0x7f71a80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.089+0000 7f71acf4d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a8073b10 con 0x7f71a8101750 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.089+0000 7f71acf4d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a8073c50 con 0x7f71a8100da0 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.089+0000 7f71a5d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71a8101750 0x7f71a80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.089+0000 7f71a5d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71a8101750 0x7f71a80735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55798/0 (socket says 172.21.15.121:55798) 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.089+0000 7f71a5d9b700 1 -- 172.21.15.121:0/3977102648 learned_addr learned my addr 172.21.15.121:0/3977102648 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.089+0000 7f71a659c700 1 --2- 172.21.15.121:0/3977102648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f71a8100da0 0x7f71a8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.089+0000 7f71a5d9b700 1 -- 172.21.15.121:0/3977102648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f71a8100da0 msgr2=0x7f71a8073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.089+0000 7f71a5d9b700 1 --2- 172.21.15.121:0/3977102648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f71a8100da0 0x7f71a8073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.089+0000 7f71a5d9b700 1 -- 172.21.15.121:0/3977102648 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c0095f0 con 0x7f71a8101750 2024-04-05T21:48:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.090+0000 7f71a5d9b700 1 --2- 172.21.15.121:0/3977102648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71a8101750 0x7f71a80735d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f719800b480 tx=0x7f719800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.090+0000 7f71977fe700 1 -- 172.21.15.121:0/3977102648 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71980118c0 con 0x7f71a8101750 2024-04-05T21:48:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.090+0000 7f71acf4d700 1 -- 172.21.15.121:0/3977102648 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a81a2980 con 0x7f71a8101750 2024-04-05T21:48:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.090+0000 7f71977fe700 1 -- 172.21.15.121:0/3977102648 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7198010380 con 0x7f71a8101750 2024-04-05T21:48:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.090+0000 7f71977fe700 1 -- 172.21.15.121:0/3977102648 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f719800f510 con 0x7f71a8101750 2024-04-05T21:48:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.090+0000 7f71acf4d700 1 -- 172.21.15.121:0/3977102648 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a81a2ce0 con 0x7f71a8101750 2024-04-05T21:48:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.091+0000 7f71acf4d700 1 -- 172.21.15.121:0/3977102648 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a8109140 con 0x7f71a8101750 2024-04-05T21:48:47.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.091+0000 7f71977fe700 1 -- 172.21.15.121:0/3977102648 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f71980104f0 con 0x7f71a8101750 2024-04-05T21:48:47.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.092+0000 7f71977fe700 1 --2- 172.21.15.121:0/3977102648 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f719006cce0 0x7f719006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:47.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.092+0000 7f71977fe700 1 -- 172.21.15.121:0/3977102648 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f719808c210 con 0x7f71a8101750 2024-04-05T21:48:47.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.092+0000 7f71a659c700 1 --2- 172.21.15.121:0/3977102648 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f719006cce0 0x7f719006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:47.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.092+0000 7f71a659c700 1 --2- 172.21.15.121:0/3977102648 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f719006cce0 0x7f719006f1a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f719c006010 tx=0x7f719c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:47.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.097+0000 7f71977fe700 1 -- 172.21.15.121:0/3977102648 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7198056280 con 0x7f71a8101750 2024-04-05T21:48:47.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.306+0000 7f71acf4d700 1 -- 172.21.15.121:0/3977102648 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f71a8073e10 con 0x7f719006cce0 2024-04-05T21:48:47.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.308+0000 7f71977fe700 1 -- 172.21.15.121:0/3977102648 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f71a8073e10 con 0x7f719006cce0 2024-04-05T21:48:47.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.310+0000 7f71acf4d700 1 -- 172.21.15.121:0/3977102648 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f719006cce0 msgr2=0x7f719006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:47.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.310+0000 7f71acf4d700 1 --2- 172.21.15.121:0/3977102648 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f719006cce0 0x7f719006f1a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f719c006010 tx=0x7f719c01a040 comp rx=0 tx=0).stop 2024-04-05T21:48:47.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.310+0000 7f71acf4d700 1 -- 172.21.15.121:0/3977102648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71a8101750 msgr2=0x7f71a80735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:47.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.310+0000 7f71acf4d700 1 --2- 172.21.15.121:0/3977102648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71a8101750 0x7f71a80735d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f719800b480 tx=0x7f719800b840 comp rx=0 tx=0).stop 2024-04-05T21:48:47.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.310+0000 7f71acf4d700 1 -- 172.21.15.121:0/3977102648 shutdown_connections 2024-04-05T21:48:47.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.310+0000 7f71acf4d700 1 --2- 172.21.15.121:0/3977102648 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f719006cce0 0x7f719006f1a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.310+0000 7f71acf4d700 1 --2- 172.21.15.121:0/3977102648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f71a8100da0 0x7f71a8073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.310+0000 7f71acf4d700 1 --2- 172.21.15.121:0/3977102648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71a8101750 0x7f71a80735d0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.310+0000 7f71acf4d700 1 -- 172.21.15.121:0/3977102648 >> 172.21.15.121:0/3977102648 conn(0x7f71a80fc630 msgr2=0x7f71a80fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:47.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.310+0000 7f71acf4d700 1 -- 172.21.15.121:0/3977102648 shutdown_connections 2024-04-05T21:48:47.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.310+0000 7f71acf4d700 1 -- 172.21.15.121:0/3977102648 wait complete. 2024-04-05T21:48:47.321 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:48:47.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.413+0000 7faf7cf44700 1 -- 172.21.15.121:0/3501547094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf780ffbd0 msgr2=0x7faf78100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:47.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.413+0000 7faf7cf44700 1 --2- 172.21.15.121:0/3501547094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf780ffbd0 0x7faf78100050 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7faf68009910 tx=0x7faf68009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:47.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.413+0000 7faf7cf44700 1 -- 172.21.15.121:0/3501547094 shutdown_connections 2024-04-05T21:48:47.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.413+0000 7faf7cf44700 1 --2- 172.21.15.121:0/3501547094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf780ffbd0 0x7faf78100050 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.413+0000 7faf7cf44700 1 --2- 172.21.15.121:0/3501547094 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf78105a40 0x7faf78105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.413+0000 7faf7cf44700 1 -- 172.21.15.121:0/3501547094 >> 172.21.15.121:0/3501547094 conn(0x7faf780755a0 msgr2=0x7faf780759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:47.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.413+0000 7faf7cf44700 1 -- 172.21.15.121:0/3501547094 shutdown_connections 2024-04-05T21:48:47.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.413+0000 7faf7cf44700 1 -- 172.21.15.121:0/3501547094 wait complete. 2024-04-05T21:48:47.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.414+0000 7faf7cf44700 1 Processor -- start 2024-04-05T21:48:47.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.414+0000 7faf7cf44700 1 -- start start 2024-04-05T21:48:47.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.414+0000 7faf7cf44700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf780ffbd0 0x7faf7819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:47.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.414+0000 7faf7cf44700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf78105a40 0x7faf7819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.414+0000 7faf7cf44700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7819d0e0 con 0x7faf780ffbd0 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.414+0000 7faf7cf44700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf78196590 con 0x7faf78105a40 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.414+0000 7faf7659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf780ffbd0 0x7faf7819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.414+0000 7faf7659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf780ffbd0 0x7faf7819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55816/0 (socket says 172.21.15.121:55816) 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.414+0000 7faf7659c700 1 -- 172.21.15.121:0/2261873971 learned_addr learned my addr 172.21.15.121:0/2261873971 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.414+0000 7faf75d9b700 1 --2- 172.21.15.121:0/2261873971 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf78105a40 0x7faf7819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.415+0000 7faf7659c700 1 -- 172.21.15.121:0/2261873971 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf78105a40 msgr2=0x7faf7819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.415+0000 7faf7659c700 1 --2- 172.21.15.121:0/2261873971 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf78105a40 0x7faf7819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.415+0000 7faf7659c700 1 -- 172.21.15.121:0/2261873971 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf680095f0 con 0x7faf780ffbd0 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.415+0000 7faf7659c700 1 --2- 172.21.15.121:0/2261873971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf780ffbd0 0x7faf7819c510 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00d8d0 tx=0x7faf6c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.415+0000 7faf677fe700 1 -- 172.21.15.121:0/2261873971 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf6c0098a0 con 0x7faf780ffbd0 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.415+0000 7faf7cf44700 1 -- 172.21.15.121:0/2261873971 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf78196870 con 0x7faf780ffbd0 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.415+0000 7faf677fe700 1 -- 172.21.15.121:0/2261873971 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf6c010460 con 0x7faf780ffbd0 2024-04-05T21:48:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.415+0000 7faf677fe700 1 -- 172.21.15.121:0/2261873971 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf6c009ea0 con 0x7faf780ffbd0 2024-04-05T21:48:47.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.415+0000 7faf7cf44700 1 -- 172.21.15.121:0/2261873971 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf78196dc0 con 0x7faf780ffbd0 2024-04-05T21:48:47.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.416+0000 7faf7cf44700 1 -- 172.21.15.121:0/2261873971 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf78108fe0 con 0x7faf780ffbd0 2024-04-05T21:48:47.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.417+0000 7faf677fe700 1 -- 172.21.15.121:0/2261873971 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7faf6c0105d0 con 0x7faf780ffbd0 2024-04-05T21:48:47.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.417+0000 7faf677fe700 1 --2- 172.21.15.121:0/2261873971 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faf6006cce0 0x7faf6006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:47.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.418+0000 7faf677fe700 1 -- 172.21.15.121:0/2261873971 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7faf6c08c550 con 0x7faf780ffbd0 2024-04-05T21:48:47.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.418+0000 7faf75d9b700 1 --2- 172.21.15.121:0/2261873971 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faf6006cce0 0x7faf6006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:47.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.418+0000 7faf75d9b700 1 --2- 172.21.15.121:0/2261873971 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faf6006cce0 0x7faf6006f1a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7faf78104ef0 tx=0x7faf6801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:47.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.422+0000 7faf677fe700 1 -- 172.21.15.121:0/2261873971 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faf6c05a0e0 con 0x7faf780ffbd0 2024-04-05T21:48:47.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.634+0000 7faf7cf44700 1 -- 172.21.15.121:0/2261873971 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faf78196fe0 con 0x7faf6006cce0 2024-04-05T21:48:47.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.635+0000 7faf677fe700 1 -- 172.21.15.121:0/2261873971 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7faf78196fe0 con 0x7faf6006cce0 2024-04-05T21:48:47.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.637+0000 7faf7cf44700 1 -- 172.21.15.121:0/2261873971 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faf6006cce0 msgr2=0x7faf6006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:47.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.637+0000 7faf7cf44700 1 --2- 172.21.15.121:0/2261873971 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faf6006cce0 0x7faf6006f1a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7faf78104ef0 tx=0x7faf6801a040 comp rx=0 tx=0).stop 2024-04-05T21:48:47.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.637+0000 7faf7cf44700 1 -- 172.21.15.121:0/2261873971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf780ffbd0 msgr2=0x7faf7819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:47.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.637+0000 7faf7cf44700 1 --2- 172.21.15.121:0/2261873971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf780ffbd0 0x7faf7819c510 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00d8d0 tx=0x7faf6c00dbe0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.637+0000 7faf7cf44700 1 -- 172.21.15.121:0/2261873971 shutdown_connections 2024-04-05T21:48:47.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.637+0000 7faf7cf44700 1 --2- 172.21.15.121:0/2261873971 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faf6006cce0 0x7faf6006f1a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.637+0000 7faf7cf44700 1 --2- 172.21.15.121:0/2261873971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf780ffbd0 0x7faf7819c510 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.637+0000 7faf7cf44700 1 --2- 172.21.15.121:0/2261873971 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf78105a40 0x7faf7819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.637+0000 7faf7cf44700 1 -- 172.21.15.121:0/2261873971 >> 172.21.15.121:0/2261873971 conn(0x7faf780755a0 msgr2=0x7faf78103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:47.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.637+0000 7faf7cf44700 1 -- 172.21.15.121:0/2261873971 shutdown_connections 2024-04-05T21:48:47.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.637+0000 7faf7cf44700 1 -- 172.21.15.121:0/2261873971 wait complete. 2024-04-05T21:48:47.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.740+0000 7f086b81c700 1 -- 172.21.15.121:0/4045654981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0864105b50 msgr2=0x7f0864105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:47.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.740+0000 7f086b81c700 1 --2- 172.21.15.121:0/4045654981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0864105b50 0x7f0864105f30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0860009910 tx=0x7f0860009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:47.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.740+0000 7f086b81c700 1 -- 172.21.15.121:0/4045654981 shutdown_connections 2024-04-05T21:48:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.740+0000 7f086b81c700 1 --2- 172.21.15.121:0/4045654981 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08640fef70 0x7f08640ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.740+0000 7f086b81c700 1 --2- 172.21.15.121:0/4045654981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0864105b50 0x7f0864105f30 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.740+0000 7f086b81c700 1 -- 172.21.15.121:0/4045654981 >> 172.21.15.121:0/4045654981 conn(0x7f0864075e30 msgr2=0x7f08640fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.741+0000 7f086b81c700 1 -- 172.21.15.121:0/4045654981 shutdown_connections 2024-04-05T21:48:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.741+0000 7f086b81c700 1 -- 172.21.15.121:0/4045654981 wait complete. 2024-04-05T21:48:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.741+0000 7f086b81c700 1 Processor -- start 2024-04-05T21:48:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.741+0000 7f086b81c700 1 -- start start 2024-04-05T21:48:47.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.741+0000 7f086b81c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08640fef70 0x7f086419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:47.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.741+0000 7f086b81c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0864105b50 0x7f086419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:47.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.741+0000 7f086b81c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086419d070 con 0x7f0864105b50 2024-04-05T21:48:47.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.741+0000 7f086b81c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0864196590 con 0x7f08640fef70 2024-04-05T21:48:47.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.741+0000 7f0868db7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0864105b50 0x7f086419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:47.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.741+0000 7f0868db7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0864105b50 0x7f086419ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55826/0 (socket says 172.21.15.121:55826) 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.742+0000 7f0868db7700 1 -- 172.21.15.121:0/2832017422 learned_addr learned my addr 172.21.15.121:0/2832017422 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.742+0000 7f0868db7700 1 -- 172.21.15.121:0/2832017422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08640fef70 msgr2=0x7f086419c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.742+0000 7f0868db7700 1 --2- 172.21.15.121:0/2832017422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08640fef70 0x7f086419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.742+0000 7f0868db7700 1 -- 172.21.15.121:0/2832017422 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08600095f0 con 0x7f0864105b50 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.742+0000 7f0868db7700 1 --2- 172.21.15.121:0/2832017422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0864105b50 0x7f086419ca50 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f085400d8d0 tx=0x7f085400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.742+0000 7f085a7fc700 1 -- 172.21.15.121:0/2832017422 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0854009940 con 0x7f0864105b50 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.742+0000 7f085a7fc700 1 -- 172.21.15.121:0/2832017422 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0854010460 con 0x7f0864105b50 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.742+0000 7f086b81c700 1 -- 172.21.15.121:0/2832017422 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0864196870 con 0x7f0864105b50 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.742+0000 7f085a7fc700 1 -- 172.21.15.121:0/2832017422 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f085400f5f0 con 0x7f0864105b50 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.742+0000 7f086b81c700 1 -- 172.21.15.121:0/2832017422 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0864196dc0 con 0x7f0864105b50 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.743+0000 7f086b81c700 1 -- 172.21.15.121:0/2832017422 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0864109140 con 0x7f0864105b50 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.743+0000 7f085a7fc700 1 -- 172.21.15.121:0/2832017422 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f08540105d0 con 0x7f0864105b50 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.744+0000 7f085a7fc700 1 --2- 172.21.15.121:0/2832017422 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f085006cce0 0x7f085006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.744+0000 7f085a7fc700 1 -- 172.21.15.121:0/2832017422 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f085408c4f0 con 0x7f0864105b50 2024-04-05T21:48:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.744+0000 7f08695b8700 1 --2- 172.21.15.121:0/2832017422 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f085006cce0 0x7f085006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.744+0000 7f08695b8700 1 --2- 172.21.15.121:0/2832017422 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f085006cce0 0x7f085006f1a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f086000b5c0 tx=0x7f0860005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.748+0000 7f085a7fc700 1 -- 172.21.15.121:0/2832017422 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f085405a070 con 0x7f0864105b50 2024-04-05T21:48:47.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:47 smithi188 ceph-mon[107793]: pgmap v676: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:47.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.954+0000 7f086b81c700 1 -- 172.21.15.121:0/2832017422 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0864197030 con 0x7f085006cce0 2024-04-05T21:48:47.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.965+0000 7f085a7fc700 1 -- 172.21.15.121:0/2832017422 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f0864197030 con 0x7f085006cce0 2024-04-05T21:48:47.965 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:48:47.965 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (19m) 13m ago 31m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:48:47.965 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (31m) 13m ago 31m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:48:47.965 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (30m) 13m ago 30m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:48:47.965 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (17m) 13m ago 31m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (17m) 13m ago 30m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (19m) 13m ago 30m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (14m) 13m ago 24m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (13m) 13m ago 24m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (13m) 13m ago 24m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (24m) 13m ago 24m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (21m) 13m ago 32m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (20m) 13m ago 29m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (18m) 13m ago 32m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (18m) 13m ago 29m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (20m) 13m ago 31m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (20m) 13m ago 30m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (17m) 13m ago 29m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (16m) 13m ago 28m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:48:47.966 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (16m) 13m ago 28m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:48:47.967 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (15m) 13m ago 27m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:48:47.967 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (14m) 13m ago 27m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:48:47.967 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (14m) 13m ago 26m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:48:47.967 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (19m) 13m ago 30m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:48:47.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.967+0000 7f086b81c700 1 -- 172.21.15.121:0/2832017422 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f085006cce0 msgr2=0x7f085006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:47.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.967+0000 7f086b81c700 1 --2- 172.21.15.121:0/2832017422 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f085006cce0 0x7f085006f1a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f086000b5c0 tx=0x7f0860005960 comp rx=0 tx=0).stop 2024-04-05T21:48:47.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.967+0000 7f086b81c700 1 -- 172.21.15.121:0/2832017422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0864105b50 msgr2=0x7f086419ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:47.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.967+0000 7f086b81c700 1 --2- 172.21.15.121:0/2832017422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0864105b50 0x7f086419ca50 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f085400d8d0 tx=0x7f085400dc90 comp rx=0 tx=0).stop 2024-04-05T21:48:47.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.967+0000 7f086b81c700 1 -- 172.21.15.121:0/2832017422 shutdown_connections 2024-04-05T21:48:47.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.967+0000 7f086b81c700 1 --2- 172.21.15.121:0/2832017422 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f085006cce0 0x7f085006f1a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.967+0000 7f086b81c700 1 --2- 172.21.15.121:0/2832017422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f08640fef70 0x7f086419c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.967+0000 7f086b81c700 1 --2- 172.21.15.121:0/2832017422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0864105b50 0x7f086419ca50 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:47.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.967+0000 7f086b81c700 1 -- 172.21.15.121:0/2832017422 >> 172.21.15.121:0/2832017422 conn(0x7f0864075e30 msgr2=0x7f0864104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:47.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.967+0000 7f086b81c700 1 -- 172.21.15.121:0/2832017422 shutdown_connections 2024-04-05T21:48:47.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:47.967+0000 7f086b81c700 1 -- 172.21.15.121:0/2832017422 wait complete. 2024-04-05T21:48:47.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:47 smithi121 ceph-mon[145859]: pgmap v676: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:48.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.068+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1543561544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76c0100140 msgr2=0x7f76c01005c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:48.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.068+0000 7f76c6a5a700 1 --2- 172.21.15.121:0/1543561544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76c0100140 0x7f76c01005c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f76b0009910 tx=0x7f76b0009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:48.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.068+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1543561544 shutdown_connections 2024-04-05T21:48:48.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.068+0000 7f76c6a5a700 1 --2- 172.21.15.121:0/1543561544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76c0100140 0x7f76c01005c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.068+0000 7f76c6a5a700 1 --2- 172.21.15.121:0/1543561544 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76c0104a50 0x7f76c0104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.068+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1543561544 >> 172.21.15.121:0/1543561544 conn(0x7f76c0075e30 msgr2=0x7f76c00fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:48.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.068+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1543561544 shutdown_connections 2024-04-05T21:48:48.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.068+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1543561544 wait complete. 2024-04-05T21:48:48.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.068+0000 7f76c6a5a700 1 Processor -- start 2024-04-05T21:48:48.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.068+0000 7f76c6a5a700 1 -- start start 2024-04-05T21:48:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76c6a5a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76c0100140 0x7f76c01954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76c6a5a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76c0104a50 0x7f76c01959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76c6a5a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c0196040 con 0x7f76c0100140 2024-04-05T21:48:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76c6a5a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c0196180 con 0x7f76c0104a50 2024-04-05T21:48:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76bffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76c0100140 0x7f76c01954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76bffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76c0100140 0x7f76c01954b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55840/0 (socket says 172.21.15.121:55840) 2024-04-05T21:48:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76bffff700 1 -- 172.21.15.121:0/1010014000 learned_addr learned my addr 172.21.15.121:0/1010014000 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76bf7fe700 1 --2- 172.21.15.121:0/1010014000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76c0104a50 0x7f76c01959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76bffff700 1 -- 172.21.15.121:0/1010014000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76c0104a50 msgr2=0x7f76c01959f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76bffff700 1 --2- 172.21.15.121:0/1010014000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76c0104a50 0x7f76c01959f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76bffff700 1 -- 172.21.15.121:0/1010014000 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b00095f0 con 0x7f76c0100140 2024-04-05T21:48:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76bffff700 1 --2- 172.21.15.121:0/1010014000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76c0100140 0x7f76c01954b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f76b4007b50 tx=0x7f76b4007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:48.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76bd7fa700 1 -- 172.21.15.121:0/1010014000 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76b4011070 con 0x7f76c0100140 2024-04-05T21:48:48.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.069+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1010014000 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c0199fd0 con 0x7f76c0100140 2024-04-05T21:48:48.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.070+0000 7f76bd7fa700 1 -- 172.21.15.121:0/1010014000 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76b400c2e0 con 0x7f76c0100140 2024-04-05T21:48:48.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.070+0000 7f76bd7fa700 1 -- 172.21.15.121:0/1010014000 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76b400ac60 con 0x7f76c0100140 2024-04-05T21:48:48.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.070+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1010014000 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c019a5a0 con 0x7f76c0100140 2024-04-05T21:48:48.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.070+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1010014000 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c004ea90 con 0x7f76c0100140 2024-04-05T21:48:48.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.071+0000 7f76bd7fa700 1 -- 172.21.15.121:0/1010014000 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f76b400e710 con 0x7f76c0100140 2024-04-05T21:48:48.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.071+0000 7f76bd7fa700 1 --2- 172.21.15.121:0/1010014000 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f76a806cd30 0x7f76a806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:48.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.071+0000 7f76bd7fa700 1 -- 172.21.15.121:0/1010014000 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f76b408d7c0 con 0x7f76c0100140 2024-04-05T21:48:48.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.071+0000 7f76bf7fe700 1 --2- 172.21.15.121:0/1010014000 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f76a806cd30 0x7f76a806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:48.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.072+0000 7f76bf7fe700 1 --2- 172.21.15.121:0/1010014000 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f76a806cd30 0x7f76a806f1f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f76c0196ad0 tx=0x7f76b001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:48.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.076+0000 7f76bd7fa700 1 -- 172.21.15.121:0/1010014000 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f76b4057840 con 0x7f76c0100140 2024-04-05T21:48:48.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.347+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1010014000 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f76c0196810 con 0x7f76c0100140 2024-04-05T21:48:48.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.348+0000 7f76bd7fa700 1 -- 172.21.15.121:0/1010014000 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f76b405ae60 con 0x7f76c0100140 2024-04-05T21:48:48.348 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:48:48.348 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:48:48.349 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:48:48.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:48:48.349 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:48:48.349 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:48:48.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:48:48.349 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:48:48.349 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:48:48.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:48:48.349 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:48:48.349 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:48:48.349 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:48:48.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:48:48.349 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:48:48.350 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:48:48.350 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:48:48.350 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:48:48.350 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:48:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.350+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1010014000 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f76a806cd30 msgr2=0x7f76a806f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.350+0000 7f76c6a5a700 1 --2- 172.21.15.121:0/1010014000 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f76a806cd30 0x7f76a806f1f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f76c0196ad0 tx=0x7f76b001a040 comp rx=0 tx=0).stop 2024-04-05T21:48:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.350+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1010014000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76c0100140 msgr2=0x7f76c01954b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.350+0000 7f76c6a5a700 1 --2- 172.21.15.121:0/1010014000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76c0100140 0x7f76c01954b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f76b4007b50 tx=0x7f76b4007f10 comp rx=0 tx=0).stop 2024-04-05T21:48:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.350+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1010014000 shutdown_connections 2024-04-05T21:48:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.350+0000 7f76c6a5a700 1 --2- 172.21.15.121:0/1010014000 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f76a806cd30 0x7f76a806f1f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.350+0000 7f76c6a5a700 1 --2- 172.21.15.121:0/1010014000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76c0100140 0x7f76c01954b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.350+0000 7f76c6a5a700 1 --2- 172.21.15.121:0/1010014000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76c0104a50 0x7f76c01959f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.350+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1010014000 >> 172.21.15.121:0/1010014000 conn(0x7f76c0075e30 msgr2=0x7f76c0103440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.350+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1010014000 shutdown_connections 2024-04-05T21:48:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.350+0000 7f76c6a5a700 1 -- 172.21.15.121:0/1010014000 wait complete. 2024-04-05T21:48:48.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.452+0000 7fdf6740b700 1 -- 172.21.15.121:0/549986743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf60101790 msgr2=0x7fdf601057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:48.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.452+0000 7fdf6740b700 1 --2- 172.21.15.121:0/549986743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf60101790 0x7fdf601057e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009910 tx=0x7fdf5c009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:48.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.452+0000 7fdf6740b700 1 -- 172.21.15.121:0/549986743 shutdown_connections 2024-04-05T21:48:48.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.452+0000 7fdf6740b700 1 --2- 172.21.15.121:0/549986743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf60101790 0x7fdf601057e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.452+0000 7fdf6740b700 1 --2- 172.21.15.121:0/549986743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf60100de0 0x7fdf601011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.452+0000 7fdf6740b700 1 -- 172.21.15.121:0/549986743 >> 172.21.15.121:0/549986743 conn(0x7fdf600fc670 msgr2=0x7fdf600fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:48.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.452+0000 7fdf6740b700 1 -- 172.21.15.121:0/549986743 shutdown_connections 2024-04-05T21:48:48.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.452+0000 7fdf6740b700 1 -- 172.21.15.121:0/549986743 wait complete. 2024-04-05T21:48:48.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf6740b700 1 Processor -- start 2024-04-05T21:48:48.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf6740b700 1 -- start start 2024-04-05T21:48:48.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf6740b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf60100de0 0x7fdf601966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:48.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf6740b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf60101790 0x7fdf60196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf6740b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf601972a0 con 0x7fdf60100de0 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf6740b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf60199fd0 con 0x7fdf60101790 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf651a7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf60100de0 0x7fdf601966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf651a7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf60100de0 0x7fdf601966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55850/0 (socket says 172.21.15.121:55850) 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf651a7700 1 -- 172.21.15.121:0/697164996 learned_addr learned my addr 172.21.15.121:0/697164996 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf649a6700 1 --2- 172.21.15.121:0/697164996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf60101790 0x7fdf60196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf651a7700 1 -- 172.21.15.121:0/697164996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf60101790 msgr2=0x7fdf60196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf651a7700 1 --2- 172.21.15.121:0/697164996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf60101790 0x7fdf60196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf651a7700 1 -- 172.21.15.121:0/697164996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c0095f0 con 0x7fdf60100de0 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.453+0000 7fdf649a6700 1 --2- 172.21.15.121:0/697164996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf60101790 0x7fdf60196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.454+0000 7fdf651a7700 1 --2- 172.21.15.121:0/697164996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf60100de0 0x7fdf601966d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000c8a0 tx=0x7fdf5000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.454+0000 7fdf567fc700 1 -- 172.21.15.121:0/697164996 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf5000cea0 con 0x7fdf60100de0 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.454+0000 7fdf567fc700 1 -- 172.21.15.121:0/697164996 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf50004830 con 0x7fdf60100de0 2024-04-05T21:48:48.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.454+0000 7fdf6740b700 1 -- 172.21.15.121:0/697164996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6019a2b0 con 0x7fdf60100de0 2024-04-05T21:48:48.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.454+0000 7fdf567fc700 1 -- 172.21.15.121:0/697164996 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf50005500 con 0x7fdf60100de0 2024-04-05T21:48:48.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.454+0000 7fdf6740b700 1 -- 172.21.15.121:0/697164996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6019a7d0 con 0x7fdf60100de0 2024-04-05T21:48:48.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.455+0000 7fdf6740b700 1 -- 172.21.15.121:0/697164996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf601091e0 con 0x7fdf60100de0 2024-04-05T21:48:48.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.455+0000 7fdf567fc700 1 -- 172.21.15.121:0/697164996 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdf500049a0 con 0x7fdf60100de0 2024-04-05T21:48:48.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.455+0000 7fdf567fc700 1 --2- 172.21.15.121:0/697164996 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdf4c06cce0 0x7fdf4c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:48.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.456+0000 7fdf567fc700 1 -- 172.21.15.121:0/697164996 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdf5008c250 con 0x7fdf60100de0 2024-04-05T21:48:48.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.456+0000 7fdf649a6700 1 --2- 172.21.15.121:0/697164996 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdf4c06cce0 0x7fdf4c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:48.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.456+0000 7fdf649a6700 1 --2- 172.21.15.121:0/697164996 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdf4c06cce0 0x7fdf4c06f1a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdf60197df0 tx=0x7fdf5c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:48.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.460+0000 7fdf567fc700 1 -- 172.21.15.121:0/697164996 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdf500562c0 con 0x7fdf60100de0 2024-04-05T21:48:48.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.696+0000 7fdf6740b700 1 -- 172.21.15.121:0/697164996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdf6004ea90 con 0x7fdf60100de0 2024-04-05T21:48:48.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.696+0000 7fdf567fc700 1 -- 172.21.15.121:0/697164996 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fdf500598e0 con 0x7fdf60100de0 2024-04-05T21:48:48.697 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:48:48.697 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:48:48.697 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:48:48.697 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:48:48.697 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:48:48.697 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:48:48.697 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:48:48.697 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:48:48.698 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:48:48.698 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:48:48.698 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:48:48.698 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:48:48.698 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:48:48.698 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:48:48.698 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:48:48.698 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:48:48.698 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:48:48.698 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:48:48.698 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:48:48.698 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:48:48.699 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:48:48.699 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:48:48.699 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:48:48.699 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:48:48.699 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:48:48.699 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:48:48.699 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:48:48.699 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:48:48.699 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:48:48.699 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:48:48.699 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:48:48.699 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:48:48.699 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:48:48.700 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:48:48.700 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:48:48.700 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:48:48.700 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:48:48.700 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:48:48.700 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:48:48.700 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:48:48.700 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:48:48.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.698+0000 7fdf6740b700 1 -- 172.21.15.121:0/697164996 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdf4c06cce0 msgr2=0x7fdf4c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:48.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.698+0000 7fdf6740b700 1 --2- 172.21.15.121:0/697164996 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdf4c06cce0 0x7fdf4c06f1a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdf60197df0 tx=0x7fdf5c01a040 comp rx=0 tx=0).stop 2024-04-05T21:48:48.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.698+0000 7fdf6740b700 1 -- 172.21.15.121:0/697164996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf60100de0 msgr2=0x7fdf601966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:48.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.698+0000 7fdf6740b700 1 --2- 172.21.15.121:0/697164996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf60100de0 0x7fdf601966d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000c8a0 tx=0x7fdf5000cc60 comp rx=0 tx=0).stop 2024-04-05T21:48:48.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.698+0000 7fdf6740b700 1 -- 172.21.15.121:0/697164996 shutdown_connections 2024-04-05T21:48:48.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.698+0000 7fdf6740b700 1 --2- 172.21.15.121:0/697164996 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdf4c06cce0 0x7fdf4c06f1a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.698+0000 7fdf6740b700 1 --2- 172.21.15.121:0/697164996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf60100de0 0x7fdf601966d0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.698+0000 7fdf6740b700 1 --2- 172.21.15.121:0/697164996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdf60101790 0x7fdf60196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.698+0000 7fdf6740b700 1 -- 172.21.15.121:0/697164996 >> 172.21.15.121:0/697164996 conn(0x7fdf600fc670 msgr2=0x7fdf600fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:48.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.698+0000 7fdf6740b700 1 -- 172.21.15.121:0/697164996 shutdown_connections 2024-04-05T21:48:48.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.698+0000 7fdf6740b700 1 -- 172.21.15.121:0/697164996 wait complete. 2024-04-05T21:48:48.702 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:48:48.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.801+0000 7f9c0e587700 1 -- 172.21.15.121:0/483911190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c08101d00 msgr2=0x7f9c081020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:48.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.801+0000 7f9c0e587700 1 --2- 172.21.15.121:0/483911190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c08101d00 0x7f9c081020e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc009910 tx=0x7f9bfc009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:48.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.801+0000 7f9c0e587700 1 -- 172.21.15.121:0/483911190 shutdown_connections 2024-04-05T21:48:48.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.801+0000 7f9c0e587700 1 --2- 172.21.15.121:0/483911190 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c08102620 0x7f9c0810ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.801+0000 7f9c0e587700 1 --2- 172.21.15.121:0/483911190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c08101d00 0x7f9c081020e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.801+0000 7f9c0e587700 1 -- 172.21.15.121:0/483911190 >> 172.21.15.121:0/483911190 conn(0x7f9c080fb570 msgr2=0x7f9c080fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:48.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.801+0000 7f9c0e587700 1 -- 172.21.15.121:0/483911190 shutdown_connections 2024-04-05T21:48:48.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.801+0000 7f9c0e587700 1 -- 172.21.15.121:0/483911190 wait complete. 2024-04-05T21:48:48.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c0e587700 1 Processor -- start 2024-04-05T21:48:48.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c0e587700 1 -- start start 2024-04-05T21:48:48.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c0e587700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c08102620 0x7f9c080ff960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:48.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c0e587700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c080ffea0 0x7f9c08100320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c0e587700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c081a2ad0 con 0x7f9c08102620 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c0e587700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c081a2c10 con 0x7f9c080ffea0 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c07fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c08102620 0x7f9c080ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c07fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c08102620 0x7f9c080ff960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55870/0 (socket says 172.21.15.121:55870) 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c07fff700 1 -- 172.21.15.121:0/214963131 learned_addr learned my addr 172.21.15.121:0/214963131 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c077fe700 1 --2- 172.21.15.121:0/214963131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c080ffea0 0x7f9c08100320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c07fff700 1 -- 172.21.15.121:0/214963131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c080ffea0 msgr2=0x7f9c08100320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c07fff700 1 --2- 172.21.15.121:0/214963131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c080ffea0 0x7f9c08100320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c07fff700 1 -- 172.21.15.121:0/214963131 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc0095f0 con 0x7f9c08102620 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.802+0000 7f9c077fe700 1 --2- 172.21.15.121:0/214963131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c080ffea0 0x7f9c08100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.803+0000 7f9c07fff700 1 --2- 172.21.15.121:0/214963131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c08102620 0x7f9c080ff960 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00bd70 tx=0x7f9bfc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.803+0000 7f9c057fa700 1 -- 172.21.15.121:0/214963131 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bfc01d070 con 0x7f9c08102620 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.803+0000 7f9c057fa700 1 -- 172.21.15.121:0/214963131 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bfc005710 con 0x7f9c08102620 2024-04-05T21:48:48.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.803+0000 7f9c057fa700 1 -- 172.21.15.121:0/214963131 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bfc00fe10 con 0x7f9c08102620 2024-04-05T21:48:48.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.803+0000 7f9c0e587700 1 -- 172.21.15.121:0/214963131 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c081a2d50 con 0x7f9c08102620 2024-04-05T21:48:48.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.803+0000 7f9c0e587700 1 -- 172.21.15.121:0/214963131 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c081a31a0 con 0x7f9c08102620 2024-04-05T21:48:48.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.804+0000 7f9c0e587700 1 -- 172.21.15.121:0/214963131 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c08108210 con 0x7f9c08102620 2024-04-05T21:48:48.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.804+0000 7f9c057fa700 1 -- 172.21.15.121:0/214963131 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9bfc00f460 con 0x7f9c08102620 2024-04-05T21:48:48.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.805+0000 7f9c057fa700 1 --2- 172.21.15.121:0/214963131 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9bf006cce0 0x7f9bf006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:48.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.805+0000 7f9c057fa700 1 -- 172.21.15.121:0/214963131 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9bfc08d8e0 con 0x7f9c08102620 2024-04-05T21:48:48.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.805+0000 7f9c077fe700 1 --2- 172.21.15.121:0/214963131 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9bf006cce0 0x7f9bf006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:48.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.805+0000 7f9c077fe700 1 --2- 172.21.15.121:0/214963131 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9bf006cce0 0x7f9bf006f1a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9c08102a80 tx=0x7f9bf80092c0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:48.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:48.809+0000 7f9c057fa700 1 -- 172.21.15.121:0/214963131 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9bfc057970 con 0x7f9c08102620 2024-04-05T21:48:48.909 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:48 smithi121 ceph-mon[145859]: from='client.35070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:48.910 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:48 smithi121 ceph-mon[145859]: from='client.35074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:48.910 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:48 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1010014000' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:48:49.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:48 smithi188 ceph-mon[107793]: from='client.35070 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:49.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:48 smithi188 ceph-mon[107793]: from='client.35074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:49.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:48 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1010014000' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:48:49.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.029+0000 7f9c0e587700 1 -- 172.21.15.121:0/214963131 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c0802cce0 con 0x7f9bf006cce0 2024-04-05T21:48:49.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.030+0000 7f9c057fa700 1 -- 172.21.15.121:0/214963131 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9c0802cce0 con 0x7f9bf006cce0 2024-04-05T21:48:49.031 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:48:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:48:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:48:49.031 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:48:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:48:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:48:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:48:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:48:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:48:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:48:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:48:49.032 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:48:49.033 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:48:49.033 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:48:49.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.032+0000 7f9c0e587700 1 -- 172.21.15.121:0/214963131 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9bf006cce0 msgr2=0x7f9bf006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:49.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.032+0000 7f9c0e587700 1 --2- 172.21.15.121:0/214963131 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9bf006cce0 0x7f9bf006f1a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9c08102a80 tx=0x7f9bf80092c0 comp rx=0 tx=0).stop 2024-04-05T21:48:49.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.032+0000 7f9c0e587700 1 -- 172.21.15.121:0/214963131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c08102620 msgr2=0x7f9c080ff960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:49.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.032+0000 7f9c0e587700 1 --2- 172.21.15.121:0/214963131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c08102620 0x7f9c080ff960 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00bd70 tx=0x7f9bfc00be50 comp rx=0 tx=0).stop 2024-04-05T21:48:49.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.032+0000 7f9c0e587700 1 -- 172.21.15.121:0/214963131 shutdown_connections 2024-04-05T21:48:49.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.032+0000 7f9c0e587700 1 --2- 172.21.15.121:0/214963131 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9bf006cce0 0x7f9bf006f1a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:49.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.032+0000 7f9c0e587700 1 --2- 172.21.15.121:0/214963131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c08102620 0x7f9c080ff960 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:49.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.032+0000 7f9c0e587700 1 --2- 172.21.15.121:0/214963131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9c080ffea0 0x7f9c08100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:49.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.032+0000 7f9c0e587700 1 -- 172.21.15.121:0/214963131 >> 172.21.15.121:0/214963131 conn(0x7f9c080fb570 msgr2=0x7f9c08105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:49.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.032+0000 7f9c0e587700 1 -- 172.21.15.121:0/214963131 shutdown_connections 2024-04-05T21:48:49.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.032+0000 7f9c0e587700 1 -- 172.21.15.121:0/214963131 wait complete. 2024-04-05T21:48:49.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.134+0000 7f95d5ab2700 1 -- 172.21.15.121:0/3455690610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d00ffa40 msgr2=0x7f95d00ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:49.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.134+0000 7f95d5ab2700 1 --2- 172.21.15.121:0/3455690610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d00ffa40 0x7f95d00ffec0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009910 tx=0x7f95c4009c20 comp rx=0 tx=0).stop 2024-04-05T21:48:49.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.134+0000 7f95d5ab2700 1 -- 172.21.15.121:0/3455690610 shutdown_connections 2024-04-05T21:48:49.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.134+0000 7f95d5ab2700 1 --2- 172.21.15.121:0/3455690610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d00ffa40 0x7f95d00ffec0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:49.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.134+0000 7f95d5ab2700 1 --2- 172.21.15.121:0/3455690610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95d0105a40 0x7f95d0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:49.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.134+0000 7f95d5ab2700 1 -- 172.21.15.121:0/3455690610 >> 172.21.15.121:0/3455690610 conn(0x7f95d00fb570 msgr2=0x7f95d00fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:49.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.134+0000 7f95d5ab2700 1 -- 172.21.15.121:0/3455690610 shutdown_connections 2024-04-05T21:48:49.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.134+0000 7f95d5ab2700 1 -- 172.21.15.121:0/3455690610 wait complete. 2024-04-05T21:48:49.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.135+0000 7f95d5ab2700 1 Processor -- start 2024-04-05T21:48:49.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.135+0000 7f95d5ab2700 1 -- start start 2024-04-05T21:48:49.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.135+0000 7f95d5ab2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95d00ffa40 0x7f95d01974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:49.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.135+0000 7f95d5ab2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d01979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:49.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.135+0000 7f95d5ab2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d0198010 con 0x7f95d0105a40 2024-04-05T21:48:49.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.135+0000 7f95d5ab2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d0198150 con 0x7f95d00ffa40 2024-04-05T21:48:49.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.135+0000 7f95ceffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d01979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:49.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.135+0000 7f95ceffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d01979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55878/0 (socket says 172.21.15.121:55878) 2024-04-05T21:48:49.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.135+0000 7f95ceffd700 1 -- 172.21.15.121:0/4104921895 learned_addr learned my addr 172.21.15.121:0/4104921895 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:48:49.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.136+0000 7f95cf7fe700 1 --2- 172.21.15.121:0/4104921895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95d00ffa40 0x7f95d01974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:49.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.136+0000 7f95ceffd700 1 -- 172.21.15.121:0/4104921895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95d00ffa40 msgr2=0x7f95d01974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:49.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.136+0000 7f95ceffd700 1 --2- 172.21.15.121:0/4104921895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95d00ffa40 0x7f95d01974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:49.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.136+0000 7f95ceffd700 1 -- 172.21.15.121:0/4104921895 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c40095f0 con 0x7f95d0105a40 2024-04-05T21:48:49.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.136+0000 7f95ceffd700 1 --2- 172.21.15.121:0/4104921895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d01979f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f95c4006010 tx=0x7f95c400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:49.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.136+0000 7f95ccff9700 1 -- 172.21.15.121:0/4104921895 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95c401d070 con 0x7f95d0105a40 2024-04-05T21:48:49.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.136+0000 7f95d5ab2700 1 -- 172.21.15.121:0/4104921895 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d01a0520 con 0x7f95d0105a40 2024-04-05T21:48:49.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.136+0000 7f95ccff9700 1 -- 172.21.15.121:0/4104921895 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95c4005710 con 0x7f95d0105a40 2024-04-05T21:48:49.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.136+0000 7f95ccff9700 1 -- 172.21.15.121:0/4104921895 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95c400fe20 con 0x7f95d0105a40 2024-04-05T21:48:49.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.136+0000 7f95d5ab2700 1 -- 172.21.15.121:0/4104921895 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d01a0930 con 0x7f95d0105a40 2024-04-05T21:48:49.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.137+0000 7f95d5ab2700 1 -- 172.21.15.121:0/4104921895 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d0107fd0 con 0x7f95d0105a40 2024-04-05T21:48:49.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.138+0000 7f95ccff9700 1 -- 172.21.15.121:0/4104921895 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f95c4003a60 con 0x7f95d0105a40 2024-04-05T21:48:49.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.138+0000 7f95ccff9700 1 --2- 172.21.15.121:0/4104921895 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f95b806cce0 0x7f95b806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:48:49.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.138+0000 7f95ccff9700 1 -- 172.21.15.121:0/4104921895 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f95c408d750 con 0x7f95d0105a40 2024-04-05T21:48:49.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.138+0000 7f95cf7fe700 1 --2- 172.21.15.121:0/4104921895 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f95b806cce0 0x7f95b806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:48:49.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.139+0000 7f95cf7fe700 1 --2- 172.21.15.121:0/4104921895 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f95b806cce0 0x7f95b806f1a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f95c0005950 tx=0x7f95c000b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:48:49.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.143+0000 7f95ccff9700 1 -- 172.21.15.121:0/4104921895 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f95c40577c0 con 0x7f95d0105a40 2024-04-05T21:48:49.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.423+0000 7f95d5ab2700 1 -- 172.21.15.121:0/4104921895 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f95d004ea90 con 0x7f95d0105a40 2024-04-05T21:48:49.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.424+0000 7f95ccff9700 1 -- 172.21.15.121:0/4104921895 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f95c405ade0 con 0x7f95d0105a40 2024-04-05T21:48:49.424 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:48:49.424 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:48:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:48:49.425 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:48:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:48:49.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.426+0000 7f95d5ab2700 1 -- 172.21.15.121:0/4104921895 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f95b806cce0 msgr2=0x7f95b806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:49.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.426+0000 7f95d5ab2700 1 --2- 172.21.15.121:0/4104921895 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f95b806cce0 0x7f95b806f1a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f95c0005950 tx=0x7f95c000b410 comp rx=0 tx=0).stop 2024-04-05T21:48:49.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.426+0000 7f95d5ab2700 1 -- 172.21.15.121:0/4104921895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 msgr2=0x7f95d01979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:48:49.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.426+0000 7f95d5ab2700 1 --2- 172.21.15.121:0/4104921895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d01979f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f95c4006010 tx=0x7f95c400bd70 comp rx=0 tx=0).stop 2024-04-05T21:48:49.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.426+0000 7f95d5ab2700 1 -- 172.21.15.121:0/4104921895 shutdown_connections 2024-04-05T21:48:49.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.426+0000 7f95d5ab2700 1 --2- 172.21.15.121:0/4104921895 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f95b806cce0 0x7f95b806f1a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:49.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.426+0000 7f95d5ab2700 1 --2- 172.21.15.121:0/4104921895 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95d00ffa40 0x7f95d01974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:49.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.426+0000 7f95d5ab2700 1 --2- 172.21.15.121:0/4104921895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95d0105a40 0x7f95d01979f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:48:49.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.426+0000 7f95d5ab2700 1 -- 172.21.15.121:0/4104921895 >> 172.21.15.121:0/4104921895 conn(0x7f95d00fb570 msgr2=0x7f95d00fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:48:49.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.426+0000 7f95d5ab2700 1 -- 172.21.15.121:0/4104921895 shutdown_connections 2024-04-05T21:48:49.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:48:49.426+0000 7f95d5ab2700 1 -- 172.21.15.121:0/4104921895 wait complete. 2024-04-05T21:48:49.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:49 smithi121 ceph-mon[145859]: from='client.35078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:49 smithi121 ceph-mon[145859]: pgmap v677: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:49 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/697164996' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:48:49.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:49 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4104921895' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:48:50.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:49 smithi188 ceph-mon[107793]: from='client.35078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:50.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:49 smithi188 ceph-mon[107793]: pgmap v677: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:50.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:49 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/697164996' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:48:50.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:49 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4104921895' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:48:50.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:50 smithi121 ceph-mon[145859]: from='client.35090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:51.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:50 smithi188 ceph-mon[107793]: from='client.35090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:48:51.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:51 smithi121 ceph-mon[145859]: pgmap v678: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:52.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:51 smithi188 ceph-mon[107793]: pgmap v678: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:53.957 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:53 smithi121 ceph-mon[145859]: pgmap v679: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:54.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:53 smithi188 ceph-mon[107793]: pgmap v679: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:55.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:55 smithi121 ceph-mon[145859]: pgmap v680: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:56.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:55 smithi188 ceph-mon[107793]: pgmap v680: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:57.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:57 smithi188 ceph-mon[107793]: pgmap v681: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:57.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:57 smithi121 ceph-mon[145859]: pgmap v681: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:48:59.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:48:59 smithi121 ceph-mon[145859]: pgmap v682: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:00.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:48:59 smithi188 ceph-mon[107793]: pgmap v682: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:00.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:49:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:49:01.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:01 smithi121 ceph-mon[145859]: pgmap v683: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:02.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:01 smithi188 ceph-mon[107793]: pgmap v683: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:03.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:03 smithi121 ceph-mon[145859]: pgmap v684: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:03.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:03 smithi188 ceph-mon[107793]: pgmap v684: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:05.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:05 smithi121 ceph-mon[145859]: pgmap v685: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:05 smithi188 ceph-mon[107793]: pgmap v685: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:07.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:07 smithi188 ceph-mon[107793]: pgmap v686: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:07.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:07 smithi121 ceph-mon[145859]: pgmap v686: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:09.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:09 smithi188 ceph-mon[107793]: pgmap v687: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:09.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:09 smithi121 ceph-mon[145859]: pgmap v687: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:11.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:11 smithi188 ceph-mon[107793]: pgmap v688: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:11.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:11 smithi121 ceph-mon[145859]: pgmap v688: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:13.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:13 smithi188 ceph-mon[107793]: pgmap v689: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:13.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:13 smithi121 ceph-mon[145859]: pgmap v689: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:15 smithi188 ceph-mon[107793]: pgmap v690: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:15.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:49:15.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:15 smithi121 ceph-mon[145859]: pgmap v690: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:15.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:49:17.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:17 smithi188 ceph-mon[107793]: pgmap v691: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:17.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:17 smithi121 ceph-mon[145859]: pgmap v691: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:19.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.541+0000 7f9ec3036700 1 -- 172.21.15.121:0/3623523381 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ebc068670 msgr2=0x7f9ebc068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:19.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.541+0000 7f9ec3036700 1 --2- 172.21.15.121:0/3623523381 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ebc068670 0x7f9ebc068a50 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009910 tx=0x7f9eb0009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:19.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.541+0000 7f9ec3036700 1 -- 172.21.15.121:0/3623523381 shutdown_connections 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.541+0000 7f9ec3036700 1 --2- 172.21.15.121:0/3623523381 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ebc069020 0x7f9ebc105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.541+0000 7f9ec3036700 1 --2- 172.21.15.121:0/3623523381 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ebc068670 0x7f9ebc068a50 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.541+0000 7f9ec3036700 1 -- 172.21.15.121:0/3623523381 >> 172.21.15.121:0/3623523381 conn(0x7f9ebc0755a0 msgr2=0x7f9ebc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.541+0000 7f9ec3036700 1 -- 172.21.15.121:0/3623523381 shutdown_connections 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.541+0000 7f9ec3036700 1 -- 172.21.15.121:0/3623523381 wait complete. 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ec3036700 1 Processor -- start 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ec3036700 1 -- start start 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ec3036700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ebc068670 0x7f9ebc196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ec3036700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ebc069020 0x7f9ebc196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ec3036700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc197290 con 0x7f9ebc069020 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ec3036700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc19b020 con 0x7f9ebc068670 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ebbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ebc069020 0x7f9ebc196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ebbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ebc069020 0x7f9ebc196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52376/0 (socket says 172.21.15.121:52376) 2024-04-05T21:49:19.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ebbfff700 1 -- 172.21.15.121:0/3156978339 learned_addr learned my addr 172.21.15.121:0/3156978339 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ec0dd2700 1 --2- 172.21.15.121:0/3156978339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ebc068670 0x7f9ebc196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ebbfff700 1 -- 172.21.15.121:0/3156978339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ebc068670 msgr2=0x7f9ebc196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ebbfff700 1 --2- 172.21.15.121:0/3156978339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ebc068670 0x7f9ebc196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ebbfff700 1 -- 172.21.15.121:0/3156978339 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb00095f0 con 0x7f9ebc069020 2024-04-05T21:49:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9ebbfff700 1 --2- 172.21.15.121:0/3156978339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ebc069020 0x7f9ebc196bb0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800c8f0 tx=0x7f9ea800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.542+0000 7f9eb9ffb700 1 -- 172.21.15.121:0/3156978339 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ea80078a0 con 0x7f9ebc069020 2024-04-05T21:49:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.543+0000 7f9eb9ffb700 1 -- 172.21.15.121:0/3156978339 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ea800f450 con 0x7f9ebc069020 2024-04-05T21:49:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.543+0000 7f9ec3036700 1 -- 172.21.15.121:0/3156978339 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ebc19b300 con 0x7f9ebc069020 2024-04-05T21:49:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.543+0000 7f9eb9ffb700 1 -- 172.21.15.121:0/3156978339 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ea8007ea0 con 0x7f9ebc069020 2024-04-05T21:49:19.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.543+0000 7f9ec3036700 1 -- 172.21.15.121:0/3156978339 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ebc19b850 con 0x7f9ebc069020 2024-04-05T21:49:19.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.544+0000 7f9ec3036700 1 -- 172.21.15.121:0/3156978339 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ebc109130 con 0x7f9ebc069020 2024-04-05T21:49:19.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.544+0000 7f9eb9ffb700 1 -- 172.21.15.121:0/3156978339 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9ea800f5c0 con 0x7f9ebc069020 2024-04-05T21:49:19.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.544+0000 7f9eb9ffb700 1 --2- 172.21.15.121:0/3156978339 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9eac06cce0 0x7f9eac06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:19.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.545+0000 7f9eb9ffb700 1 -- 172.21.15.121:0/3156978339 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9ea808c570 con 0x7f9ebc069020 2024-04-05T21:49:19.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.545+0000 7f9ec0dd2700 1 --2- 172.21.15.121:0/3156978339 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9eac06cce0 0x7f9eac06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:19.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.545+0000 7f9ec0dd2700 1 --2- 172.21.15.121:0/3156978339 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9eac06cce0 0x7f9eac06f1a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0006010 tx=0x7f9eb001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:19.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.549+0000 7f9eb9ffb700 1 -- 172.21.15.121:0/3156978339 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9ea8056540 con 0x7f9ebc069020 2024-04-05T21:49:19.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.760+0000 7f9ec3036700 1 -- 172.21.15.121:0/3156978339 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ebc19bbc0 con 0x7f9eac06cce0 2024-04-05T21:49:19.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.761+0000 7f9eb9ffb700 1 -- 172.21.15.121:0/3156978339 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9ebc19bbc0 con 0x7f9eac06cce0 2024-04-05T21:49:19.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.763+0000 7f9ec3036700 1 -- 172.21.15.121:0/3156978339 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9eac06cce0 msgr2=0x7f9eac06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:19.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.763+0000 7f9ec3036700 1 --2- 172.21.15.121:0/3156978339 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9eac06cce0 0x7f9eac06f1a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0006010 tx=0x7f9eb001a040 comp rx=0 tx=0).stop 2024-04-05T21:49:19.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.763+0000 7f9ec3036700 1 -- 172.21.15.121:0/3156978339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ebc069020 msgr2=0x7f9ebc196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:19.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.763+0000 7f9ec3036700 1 --2- 172.21.15.121:0/3156978339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ebc069020 0x7f9ebc196bb0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800c8f0 tx=0x7f9ea800cc00 comp rx=0 tx=0).stop 2024-04-05T21:49:19.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.763+0000 7f9ec3036700 1 -- 172.21.15.121:0/3156978339 shutdown_connections 2024-04-05T21:49:19.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.763+0000 7f9ec3036700 1 --2- 172.21.15.121:0/3156978339 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9eac06cce0 0x7f9eac06f1a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:19.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.763+0000 7f9ec3036700 1 --2- 172.21.15.121:0/3156978339 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9ebc068670 0x7f9ebc196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:19.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.763+0000 7f9ec3036700 1 --2- 172.21.15.121:0/3156978339 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ebc069020 0x7f9ebc196bb0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:19.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.763+0000 7f9ec3036700 1 -- 172.21.15.121:0/3156978339 >> 172.21.15.121:0/3156978339 conn(0x7f9ebc0755a0 msgr2=0x7f9ebc0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:19.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.763+0000 7f9ec3036700 1 -- 172.21.15.121:0/3156978339 shutdown_connections 2024-04-05T21:49:19.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.763+0000 7f9ec3036700 1 -- 172.21.15.121:0/3156978339 wait complete. 2024-04-05T21:49:19.774 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:49:19.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:19 smithi188 ceph-mon[107793]: pgmap v692: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:19.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.867+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/2318328201 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdaf0105a40 msgr2=0x7fdaf0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:19.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.867+0000 7fdaf4bbd700 1 --2- 172.21.15.121:0/2318328201 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdaf0105a40 0x7fdaf0105e20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fdae4009910 tx=0x7fdae4009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:19.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.867+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/2318328201 shutdown_connections 2024-04-05T21:49:19.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.867+0000 7fdaf4bbd700 1 --2- 172.21.15.121:0/2318328201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaf00ffbd0 0x7fdaf0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:19.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.867+0000 7fdaf4bbd700 1 --2- 172.21.15.121:0/2318328201 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdaf0105a40 0x7fdaf0105e20 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:19.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.867+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/2318328201 >> 172.21.15.121:0/2318328201 conn(0x7fdaf00755a0 msgr2=0x7fdaf00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:19.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.867+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/2318328201 shutdown_connections 2024-04-05T21:49:19.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.867+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/2318328201 wait complete. 2024-04-05T21:49:19.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.868+0000 7fdaf4bbd700 1 Processor -- start 2024-04-05T21:49:19.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.868+0000 7fdaf4bbd700 1 -- start start 2024-04-05T21:49:19.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.868+0000 7fdaf4bbd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdaf00ffbd0 0x7fdaf0196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:19.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.868+0000 7fdaf4bbd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaf0105a40 0x7fdaf0196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:19.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.868+0000 7fdaf4bbd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf0197260 con 0x7fdaf00ffbd0 2024-04-05T21:49:19.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.868+0000 7fdaf4bbd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf019aff0 con 0x7fdaf0105a40 2024-04-05T21:49:19.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.868+0000 7fdaee59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdaf00ffbd0 0x7fdaf0196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:19.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.868+0000 7fdaee59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdaf00ffbd0 0x7fdaf0196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52400/0 (socket says 172.21.15.121:52400) 2024-04-05T21:49:19.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.868+0000 7fdaee59c700 1 -- 172.21.15.121:0/3633461256 learned_addr learned my addr 172.21.15.121:0/3633461256 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:19.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.868+0000 7fdaedd9b700 1 --2- 172.21.15.121:0/3633461256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaf0105a40 0x7fdaf0196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.868+0000 7fdaee59c700 1 -- 172.21.15.121:0/3633461256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaf0105a40 msgr2=0x7fdaf0196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.868+0000 7fdaee59c700 1 --2- 172.21.15.121:0/3633461256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaf0105a40 0x7fdaf0196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.868+0000 7fdaee59c700 1 -- 172.21.15.121:0/3633461256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae40095f0 con 0x7fdaf00ffbd0 2024-04-05T21:49:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.869+0000 7fdaee59c700 1 --2- 172.21.15.121:0/3633461256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdaf00ffbd0 0x7fdaf0196640 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdae4006010 tx=0x7fdae400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.869+0000 7fdadf7fe700 1 -- 172.21.15.121:0/3633461256 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdae401d070 con 0x7fdaf00ffbd0 2024-04-05T21:49:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.869+0000 7fdadf7fe700 1 -- 172.21.15.121:0/3633461256 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdae4005710 con 0x7fdaf00ffbd0 2024-04-05T21:49:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.869+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/3633461256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf019b270 con 0x7fdaf00ffbd0 2024-04-05T21:49:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.869+0000 7fdadf7fe700 1 -- 172.21.15.121:0/3633461256 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdae400fe20 con 0x7fdaf00ffbd0 2024-04-05T21:49:19.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.869+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/3633461256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf019b760 con 0x7fdaf00ffbd0 2024-04-05T21:49:19.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.870+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/3633461256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf0109050 con 0x7fdaf00ffbd0 2024-04-05T21:49:19.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.870+0000 7fdadf7fe700 1 -- 172.21.15.121:0/3633461256 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdae4003a60 con 0x7fdaf00ffbd0 2024-04-05T21:49:19.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.871+0000 7fdadf7fe700 1 --2- 172.21.15.121:0/3633461256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdad806cce0 0x7fdad806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:19.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.871+0000 7fdadf7fe700 1 -- 172.21.15.121:0/3633461256 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdae408d750 con 0x7fdaf00ffbd0 2024-04-05T21:49:19.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.871+0000 7fdaedd9b700 1 --2- 172.21.15.121:0/3633461256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdad806cce0 0x7fdad806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:19.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.871+0000 7fdaedd9b700 1 --2- 172.21.15.121:0/3633461256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdad806cce0 0x7fdad806f1a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0197c60 tx=0x7fdae000b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:19.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:19.875+0000 7fdadf7fe700 1 -- 172.21.15.121:0/3633461256 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdae40577c0 con 0x7fdaf00ffbd0 2024-04-05T21:49:19.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:19 smithi121 ceph-mon[145859]: pgmap v692: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:20.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.086+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/3633461256 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdaf002cd30 con 0x7fdad806cce0 2024-04-05T21:49:20.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.088+0000 7fdadf7fe700 1 -- 172.21.15.121:0/3633461256 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdaf002cd30 con 0x7fdad806cce0 2024-04-05T21:49:20.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.090+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/3633461256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdad806cce0 msgr2=0x7fdad806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:20.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.090+0000 7fdaf4bbd700 1 --2- 172.21.15.121:0/3633461256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdad806cce0 0x7fdad806f1a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0197c60 tx=0x7fdae000b410 comp rx=0 tx=0).stop 2024-04-05T21:49:20.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.090+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/3633461256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdaf00ffbd0 msgr2=0x7fdaf0196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:20.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.090+0000 7fdaf4bbd700 1 --2- 172.21.15.121:0/3633461256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdaf00ffbd0 0x7fdaf0196640 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fdae4006010 tx=0x7fdae400bd70 comp rx=0 tx=0).stop 2024-04-05T21:49:20.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.090+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/3633461256 shutdown_connections 2024-04-05T21:49:20.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.090+0000 7fdaf4bbd700 1 --2- 172.21.15.121:0/3633461256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdad806cce0 0x7fdad806f1a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.090+0000 7fdaf4bbd700 1 --2- 172.21.15.121:0/3633461256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdaf00ffbd0 0x7fdaf0196640 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.090+0000 7fdaf4bbd700 1 --2- 172.21.15.121:0/3633461256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdaf0105a40 0x7fdaf0196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.090+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/3633461256 >> 172.21.15.121:0/3633461256 conn(0x7fdaf00755a0 msgr2=0x7fdaf0068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:20.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.090+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/3633461256 shutdown_connections 2024-04-05T21:49:20.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.090+0000 7fdaf4bbd700 1 -- 172.21.15.121:0/3633461256 wait complete. 2024-04-05T21:49:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.192+0000 7f372d2e4700 1 -- 172.21.15.121:0/478792629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37280ffec0 msgr2=0x7f37281057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.192+0000 7f372d2e4700 1 --2- 172.21.15.121:0/478792629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37280ffec0 0x7f37281057b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f371c009910 tx=0x7f371c009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.192+0000 7f372d2e4700 1 -- 172.21.15.121:0/478792629 shutdown_connections 2024-04-05T21:49:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.192+0000 7f372d2e4700 1 --2- 172.21.15.121:0/478792629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37280ffec0 0x7f37281057b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.192+0000 7f372d2e4700 1 --2- 172.21.15.121:0/478792629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37280ff510 0x7f37280ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.192+0000 7f372d2e4700 1 -- 172.21.15.121:0/478792629 >> 172.21.15.121:0/478792629 conn(0x7f3728074e00 msgr2=0x7f3728075210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.192+0000 7f372d2e4700 1 -- 172.21.15.121:0/478792629 shutdown_connections 2024-04-05T21:49:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.192+0000 7f372d2e4700 1 -- 172.21.15.121:0/478792629 wait complete. 2024-04-05T21:49:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.193+0000 7f372d2e4700 1 Processor -- start 2024-04-05T21:49:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.193+0000 7f372d2e4700 1 -- start start 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.193+0000 7f372d2e4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37280ff510 0x7f37281966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.193+0000 7f372d2e4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37280ffec0 0x7f3728196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.193+0000 7f372d2e4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37281972c0 con 0x7f37280ffec0 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.193+0000 7f372d2e4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3728199ff0 con 0x7f37280ff510 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.193+0000 7f37267fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37280ffec0 0x7f3728196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.193+0000 7f37267fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37280ffec0 0x7f3728196c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52426/0 (socket says 172.21.15.121:52426) 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.193+0000 7f37267fc700 1 -- 172.21.15.121:0/2505082794 learned_addr learned my addr 172.21.15.121:0/2505082794 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.194+0000 7f3726ffd700 1 --2- 172.21.15.121:0/2505082794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37280ff510 0x7f37281966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.194+0000 7f37267fc700 1 -- 172.21.15.121:0/2505082794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37280ff510 msgr2=0x7f37281966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.194+0000 7f37267fc700 1 --2- 172.21.15.121:0/2505082794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37280ff510 0x7f37281966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.194+0000 7f37267fc700 1 -- 172.21.15.121:0/2505082794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371c0095f0 con 0x7f37280ffec0 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.194+0000 7f3726ffd700 1 --2- 172.21.15.121:0/2505082794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37280ff510 0x7f37281966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.194+0000 7f37267fc700 1 --2- 172.21.15.121:0/2505082794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37280ffec0 0x7f3728196c30 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f371c00bda0 tx=0x7f371c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.194+0000 7f370ffff700 1 -- 172.21.15.121:0/2505082794 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f371c01d070 con 0x7f37280ffec0 2024-04-05T21:49:20.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.194+0000 7f370ffff700 1 -- 172.21.15.121:0/2505082794 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f371c00f460 con 0x7f37280ffec0 2024-04-05T21:49:20.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.194+0000 7f372d2e4700 1 -- 172.21.15.121:0/2505082794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372819a270 con 0x7f37280ffec0 2024-04-05T21:49:20.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.194+0000 7f370ffff700 1 -- 172.21.15.121:0/2505082794 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f371c021640 con 0x7f37280ffec0 2024-04-05T21:49:20.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.194+0000 7f372d2e4700 1 -- 172.21.15.121:0/2505082794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372819a760 con 0x7f37280ffec0 2024-04-05T21:49:20.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.195+0000 7f372d2e4700 1 -- 172.21.15.121:0/2505082794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3728109170 con 0x7f37280ffec0 2024-04-05T21:49:20.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.195+0000 7f370ffff700 1 -- 172.21.15.121:0/2505082794 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f371c003a60 con 0x7f37280ffec0 2024-04-05T21:49:20.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.196+0000 7f370ffff700 1 --2- 172.21.15.121:0/2505082794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f371006cce0 0x7f371006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:20.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.196+0000 7f3726ffd700 1 --2- 172.21.15.121:0/2505082794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f371006cce0 0x7f371006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:20.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.196+0000 7f370ffff700 1 -- 172.21.15.121:0/2505082794 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f371c08d8b0 con 0x7f37280ffec0 2024-04-05T21:49:20.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.196+0000 7f3726ffd700 1 --2- 172.21.15.121:0/2505082794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f371006cce0 0x7f371006f1a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3718005950 tx=0x7f3718005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:20.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.200+0000 7f370ffff700 1 -- 172.21.15.121:0/2505082794 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f371c057920 con 0x7f37280ffec0 2024-04-05T21:49:20.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.403+0000 7f372d2e4700 1 -- 172.21.15.121:0/2505082794 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f372819aa40 con 0x7f371006cce0 2024-04-05T21:49:20.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.413+0000 7f370ffff700 1 -- 172.21.15.121:0/2505082794 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f372819aa40 con 0x7f371006cce0 2024-04-05T21:49:20.414 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:49:20.414 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (20m) 14m ago 31m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:49:20.414 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (31m) 14m ago 31m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:49:20.414 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (30m) 13m ago 30m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:49:20.414 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (18m) 14m ago 31m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:49:20.414 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (18m) 13m ago 30m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:49:20.414 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (19m) 14m ago 31m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:49:20.414 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (14m) 14m ago 25m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:49:20.414 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (14m) 14m ago 25m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:49:20.414 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (13m) 13m ago 25m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:49:20.414 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (25m) 13m ago 25m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:49:20.414 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (22m) 14m ago 33m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:49:20.415 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (21m) 13m ago 30m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:49:20.415 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (18m) 14m ago 33m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:49:20.415 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (18m) 13m ago 30m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:49:20.415 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (20m) 14m ago 31m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:49:20.415 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (20m) 13m ago 30m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:49:20.415 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (17m) 14m ago 29m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:49:20.415 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (17m) 14m ago 29m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:49:20.415 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (16m) 14m ago 28m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:49:20.415 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (16m) 13m ago 28m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:49:20.415 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (15m) 13m ago 27m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:49:20.415 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (14m) 13m ago 27m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:49:20.415 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (20m) 14m ago 31m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:49:20.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.415+0000 7f372d2e4700 1 -- 172.21.15.121:0/2505082794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f371006cce0 msgr2=0x7f371006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:20.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.415+0000 7f372d2e4700 1 --2- 172.21.15.121:0/2505082794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f371006cce0 0x7f371006f1a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3718005950 tx=0x7f3718005f50 comp rx=0 tx=0).stop 2024-04-05T21:49:20.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.415+0000 7f372d2e4700 1 -- 172.21.15.121:0/2505082794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37280ffec0 msgr2=0x7f3728196c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:20.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.415+0000 7f372d2e4700 1 --2- 172.21.15.121:0/2505082794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37280ffec0 0x7f3728196c30 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f371c00bda0 tx=0x7f371c00be80 comp rx=0 tx=0).stop 2024-04-05T21:49:20.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.415+0000 7f372d2e4700 1 -- 172.21.15.121:0/2505082794 shutdown_connections 2024-04-05T21:49:20.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.415+0000 7f372d2e4700 1 --2- 172.21.15.121:0/2505082794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f371006cce0 0x7f371006f1a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.415+0000 7f372d2e4700 1 --2- 172.21.15.121:0/2505082794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f37280ff510 0x7f37281966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.415+0000 7f372d2e4700 1 --2- 172.21.15.121:0/2505082794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37280ffec0 0x7f3728196c30 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.415+0000 7f372d2e4700 1 -- 172.21.15.121:0/2505082794 >> 172.21.15.121:0/2505082794 conn(0x7f3728074e00 msgr2=0x7f37280fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:20.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.415+0000 7f372d2e4700 1 -- 172.21.15.121:0/2505082794 shutdown_connections 2024-04-05T21:49:20.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.415+0000 7f372d2e4700 1 -- 172.21.15.121:0/2505082794 wait complete. 2024-04-05T21:49:20.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.516+0000 7f20cd482700 1 -- 172.21.15.121:0/703776251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20c80ffa30 msgr2=0x7f20c80ffeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:20.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.516+0000 7f20cd482700 1 --2- 172.21.15.121:0/703776251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20c80ffa30 0x7f20c80ffeb0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f20b0009910 tx=0x7f20b0009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:20.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.516+0000 7f20cd482700 1 -- 172.21.15.121:0/703776251 shutdown_connections 2024-04-05T21:49:20.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.516+0000 7f20cd482700 1 --2- 172.21.15.121:0/703776251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20c80ffa30 0x7f20c80ffeb0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.516+0000 7f20cd482700 1 --2- 172.21.15.121:0/703776251 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20c8105a30 0x7f20c8105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.516+0000 7f20cd482700 1 -- 172.21.15.121:0/703776251 >> 172.21.15.121:0/703776251 conn(0x7f20c80fb520 msgr2=0x7f20c80fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:20.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.516+0000 7f20cd482700 1 -- 172.21.15.121:0/703776251 shutdown_connections 2024-04-05T21:49:20.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.516+0000 7f20cd482700 1 -- 172.21.15.121:0/703776251 wait complete. 2024-04-05T21:49:20.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.517+0000 7f20cd482700 1 Processor -- start 2024-04-05T21:49:20.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.517+0000 7f20cd482700 1 -- start start 2024-04-05T21:49:20.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.517+0000 7f20cd482700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20c80ffa30 0x7f20c8196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:20.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.517+0000 7f20cd482700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20c8105a30 0x7f20c8196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:20.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.517+0000 7f20cd482700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c8197270 con 0x7f20c8105a30 2024-04-05T21:49:20.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.517+0000 7f20cd482700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c819b000 con 0x7f20c80ffa30 2024-04-05T21:49:20.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.517+0000 7f20c67fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20c8105a30 0x7f20c8196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:20.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.517+0000 7f20c67fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20c8105a30 0x7f20c8196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52448/0 (socket says 172.21.15.121:52448) 2024-04-05T21:49:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.517+0000 7f20c67fc700 1 -- 172.21.15.121:0/2388089797 learned_addr learned my addr 172.21.15.121:0/2388089797 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.517+0000 7f20c6ffd700 1 --2- 172.21.15.121:0/2388089797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20c80ffa30 0x7f20c8196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.517+0000 7f20c67fc700 1 -- 172.21.15.121:0/2388089797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20c80ffa30 msgr2=0x7f20c8196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.517+0000 7f20c67fc700 1 --2- 172.21.15.121:0/2388089797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20c80ffa30 0x7f20c8196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.517+0000 7f20c67fc700 1 -- 172.21.15.121:0/2388089797 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b00095f0 con 0x7f20c8105a30 2024-04-05T21:49:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.518+0000 7f20c67fc700 1 --2- 172.21.15.121:0/2388089797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20c8105a30 0x7f20c8196b90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f20b0000c00 tx=0x7f20b000bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.518+0000 7f20bffff700 1 -- 172.21.15.121:0/2388089797 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20b001d070 con 0x7f20c8105a30 2024-04-05T21:49:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.518+0000 7f20bffff700 1 -- 172.21.15.121:0/2388089797 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20b0005710 con 0x7f20c8105a30 2024-04-05T21:49:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.518+0000 7f20cd482700 1 -- 172.21.15.121:0/2388089797 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c819b280 con 0x7f20c8105a30 2024-04-05T21:49:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.518+0000 7f20bffff700 1 -- 172.21.15.121:0/2388089797 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20b00039d0 con 0x7f20c8105a30 2024-04-05T21:49:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.518+0000 7f20cd482700 1 -- 172.21.15.121:0/2388089797 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c819b710 con 0x7f20c8105a30 2024-04-05T21:49:20.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.519+0000 7f20cd482700 1 -- 172.21.15.121:0/2388089797 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c8108fd0 con 0x7f20c8105a30 2024-04-05T21:49:20.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.519+0000 7f20bffff700 1 -- 172.21.15.121:0/2388089797 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f20b0003c00 con 0x7f20c8105a30 2024-04-05T21:49:20.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.520+0000 7f20bffff700 1 --2- 172.21.15.121:0/2388089797 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f20b406cce0 0x7f20b406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:20.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.520+0000 7f20bffff700 1 -- 172.21.15.121:0/2388089797 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f20b008dad0 con 0x7f20c8105a30 2024-04-05T21:49:20.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.520+0000 7f20c6ffd700 1 --2- 172.21.15.121:0/2388089797 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f20b406cce0 0x7f20b406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:20.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.520+0000 7f20c6ffd700 1 --2- 172.21.15.121:0/2388089797 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f20b406cce0 0x7f20b406f1a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f20b8009c60 tx=0x7f20b8009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:20.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.524+0000 7f20bffff700 1 -- 172.21.15.121:0/2388089797 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f20b0057bc0 con 0x7f20c8105a30 2024-04-05T21:49:20.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:20 smithi188 ceph-mon[107793]: from='client.35098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:20.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.793+0000 7f20cd482700 1 -- 172.21.15.121:0/2388089797 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f20c804ea90 con 0x7f20c8105a30 2024-04-05T21:49:20.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.794+0000 7f20bffff700 1 -- 172.21.15.121:0/2388089797 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f20b0026070 con 0x7f20c8105a30 2024-04-05T21:49:20.794 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:49:20.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:49:20.794 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:49:20.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:49:20.795 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:49:20.796 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:49:20.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.796+0000 7f20cd482700 1 -- 172.21.15.121:0/2388089797 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f20b406cce0 msgr2=0x7f20b406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:20.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.796+0000 7f20cd482700 1 --2- 172.21.15.121:0/2388089797 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f20b406cce0 0x7f20b406f1a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f20b8009c60 tx=0x7f20b8009260 comp rx=0 tx=0).stop 2024-04-05T21:49:20.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.796+0000 7f20cd482700 1 -- 172.21.15.121:0/2388089797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20c8105a30 msgr2=0x7f20c8196b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:20.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.796+0000 7f20cd482700 1 --2- 172.21.15.121:0/2388089797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20c8105a30 0x7f20c8196b90 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f20b0000c00 tx=0x7f20b000bde0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.796+0000 7f20cd482700 1 -- 172.21.15.121:0/2388089797 shutdown_connections 2024-04-05T21:49:20.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.796+0000 7f20cd482700 1 --2- 172.21.15.121:0/2388089797 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f20b406cce0 0x7f20b406f1a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.796+0000 7f20cd482700 1 --2- 172.21.15.121:0/2388089797 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f20c80ffa30 0x7f20c8196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.796+0000 7f20cd482700 1 --2- 172.21.15.121:0/2388089797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20c8105a30 0x7f20c8196b90 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.796+0000 7f20cd482700 1 -- 172.21.15.121:0/2388089797 >> 172.21.15.121:0/2388089797 conn(0x7f20c80fb520 msgr2=0x7f20c8069170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:20.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.796+0000 7f20cd482700 1 -- 172.21.15.121:0/2388089797 shutdown_connections 2024-04-05T21:49:20.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.796+0000 7f20cd482700 1 -- 172.21.15.121:0/2388089797 wait complete. 2024-04-05T21:49:20.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.898+0000 7f1d58d1d700 1 -- 172.21.15.121:0/373033808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d540ffbd0 msgr2=0x7f1d54100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:20.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.898+0000 7f1d58d1d700 1 --2- 172.21.15.121:0/373033808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d540ffbd0 0x7f1d54100050 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1d44009910 tx=0x7f1d44009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:20.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.898+0000 7f1d58d1d700 1 -- 172.21.15.121:0/373033808 shutdown_connections 2024-04-05T21:49:20.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.898+0000 7f1d58d1d700 1 --2- 172.21.15.121:0/373033808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d540ffbd0 0x7f1d54100050 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.898+0000 7f1d58d1d700 1 --2- 172.21.15.121:0/373033808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d54105a40 0x7f1d54105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.898+0000 7f1d58d1d700 1 -- 172.21.15.121:0/373033808 >> 172.21.15.121:0/373033808 conn(0x7f1d540755a0 msgr2=0x7f1d540759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:20.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.898+0000 7f1d58d1d700 1 -- 172.21.15.121:0/373033808 shutdown_connections 2024-04-05T21:49:20.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.898+0000 7f1d58d1d700 1 -- 172.21.15.121:0/373033808 wait complete. 2024-04-05T21:49:20.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.899+0000 7f1d58d1d700 1 Processor -- start 2024-04-05T21:49:20.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.899+0000 7f1d58d1d700 1 -- start start 2024-04-05T21:49:20.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.899+0000 7f1d58d1d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d540ffbd0 0x7f1d541974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:20.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.899+0000 7f1d58d1d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d54105a40 0x7f1d541979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:20.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.899+0000 7f1d58d1d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d54198010 con 0x7f1d54105a40 2024-04-05T21:49:20.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.899+0000 7f1d58d1d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d54198150 con 0x7f1d540ffbd0 2024-04-05T21:49:20.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.900+0000 7f1d51d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d54105a40 0x7f1d541979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:20.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.900+0000 7f1d51d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d54105a40 0x7f1d541979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52478/0 (socket says 172.21.15.121:52478) 2024-04-05T21:49:20.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.900+0000 7f1d51d9b700 1 -- 172.21.15.121:0/1848147998 learned_addr learned my addr 172.21.15.121:0/1848147998 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:20.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.900+0000 7f1d51d9b700 1 -- 172.21.15.121:0/1848147998 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d540ffbd0 msgr2=0x7f1d541974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:20.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.900+0000 7f1d51d9b700 1 --2- 172.21.15.121:0/1848147998 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d540ffbd0 0x7f1d541974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:20.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.900+0000 7f1d51d9b700 1 -- 172.21.15.121:0/1848147998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d440095f0 con 0x7f1d54105a40 2024-04-05T21:49:20.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.900+0000 7f1d51d9b700 1 --2- 172.21.15.121:0/1848147998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d54105a40 0x7f1d541979f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1d44006010 tx=0x7f1d4400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:20.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.900+0000 7f1d4b7fe700 1 -- 172.21.15.121:0/1848147998 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d4401d070 con 0x7f1d54105a40 2024-04-05T21:49:20.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.900+0000 7f1d4b7fe700 1 -- 172.21.15.121:0/1848147998 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d4400f460 con 0x7f1d54105a40 2024-04-05T21:49:20.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.900+0000 7f1d58d1d700 1 -- 172.21.15.121:0/1848147998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d541a0520 con 0x7f1d54105a40 2024-04-05T21:49:20.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.900+0000 7f1d4b7fe700 1 -- 172.21.15.121:0/1848147998 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d44021620 con 0x7f1d54105a40 2024-04-05T21:49:20.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.900+0000 7f1d58d1d700 1 -- 172.21.15.121:0/1848147998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d541a0930 con 0x7f1d54105a40 2024-04-05T21:49:20.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.901+0000 7f1d58d1d700 1 -- 172.21.15.121:0/1848147998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d54107fd0 con 0x7f1d54105a40 2024-04-05T21:49:20.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.902+0000 7f1d4b7fe700 1 -- 172.21.15.121:0/1848147998 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1d4400fa40 con 0x7f1d54105a40 2024-04-05T21:49:20.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.902+0000 7f1d4b7fe700 1 --2- 172.21.15.121:0/1848147998 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1d4006cce0 0x7f1d4006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:20.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.902+0000 7f1d4b7fe700 1 -- 172.21.15.121:0/1848147998 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1d4408d6a0 con 0x7f1d54105a40 2024-04-05T21:49:20.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.902+0000 7f1d5259c700 1 --2- 172.21.15.121:0/1848147998 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1d4006cce0 0x7f1d4006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:20.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.902+0000 7f1d5259c700 1 --2- 172.21.15.121:0/1848147998 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1d4006cce0 0x7f1d4006f1a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c005950 tx=0x7f1d3c0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:20.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:20.906+0000 7f1d4b7fe700 1 -- 172.21.15.121:0/1848147998 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1d44057790 con 0x7f1d54105a40 2024-04-05T21:49:20.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:20 smithi121 ceph-mon[145859]: from='client.35098 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:21.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.144+0000 7f1d58d1d700 1 -- 172.21.15.121:0/1848147998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1d5404ea90 con 0x7f1d54105a40 2024-04-05T21:49:21.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.145+0000 7f1d4b7fe700 1 -- 172.21.15.121:0/1848147998 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f1d4405adb0 con 0x7f1d54105a40 2024-04-05T21:49:21.145 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:49:21.145 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:49:21.145 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:49:21.146 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:49:21.147 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:49:21.148 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:49:21.148 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:49:21.148 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:49:21.148 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:49:21.148 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:49:21.148 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:49:21.148 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:49:21.148 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:49:21.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.147+0000 7f1d58d1d700 1 -- 172.21.15.121:0/1848147998 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1d4006cce0 msgr2=0x7f1d4006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:21.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.147+0000 7f1d58d1d700 1 --2- 172.21.15.121:0/1848147998 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1d4006cce0 0x7f1d4006f1a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c005950 tx=0x7f1d3c0058e0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.147+0000 7f1d58d1d700 1 -- 172.21.15.121:0/1848147998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d54105a40 msgr2=0x7f1d541979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:21.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.147+0000 7f1d58d1d700 1 --2- 172.21.15.121:0/1848147998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d54105a40 0x7f1d541979f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1d44006010 tx=0x7f1d4400be20 comp rx=0 tx=0).stop 2024-04-05T21:49:21.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.147+0000 7f1d58d1d700 1 -- 172.21.15.121:0/1848147998 shutdown_connections 2024-04-05T21:49:21.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.147+0000 7f1d58d1d700 1 --2- 172.21.15.121:0/1848147998 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1d4006cce0 0x7f1d4006f1a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.147+0000 7f1d58d1d700 1 --2- 172.21.15.121:0/1848147998 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d540ffbd0 0x7f1d541974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.147+0000 7f1d58d1d700 1 --2- 172.21.15.121:0/1848147998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d54105a40 0x7f1d541979f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.147+0000 7f1d58d1d700 1 -- 172.21.15.121:0/1848147998 >> 172.21.15.121:0/1848147998 conn(0x7f1d540755a0 msgr2=0x7f1d540fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:21.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.147+0000 7f1d58d1d700 1 -- 172.21.15.121:0/1848147998 shutdown_connections 2024-04-05T21:49:21.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.147+0000 7f1d58d1d700 1 -- 172.21.15.121:0/1848147998 wait complete. 2024-04-05T21:49:21.150 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:49:21.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.250+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/4021323133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ecc105a40 msgr2=0x7f6ecc105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:21.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.250+0000 7f6ed0fd9700 1 --2- 172.21.15.121:0/4021323133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ecc105a40 0x7f6ecc105e20 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0009910 tx=0x7f6ec0009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:21.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.250+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/4021323133 shutdown_connections 2024-04-05T21:49:21.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.250+0000 7f6ed0fd9700 1 --2- 172.21.15.121:0/4021323133 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ecc0ffa40 0x7f6ecc0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.250+0000 7f6ed0fd9700 1 --2- 172.21.15.121:0/4021323133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ecc105a40 0x7f6ecc105e20 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.250+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/4021323133 >> 172.21.15.121:0/4021323133 conn(0x7f6ecc0fb570 msgr2=0x7f6ecc0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:21.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.250+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/4021323133 shutdown_connections 2024-04-05T21:49:21.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.250+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/4021323133 wait complete. 2024-04-05T21:49:21.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.251+0000 7f6ed0fd9700 1 Processor -- start 2024-04-05T21:49:21.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.251+0000 7f6ed0fd9700 1 -- start start 2024-04-05T21:49:21.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.251+0000 7f6ed0fd9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ecc0ffa40 0x7f6ecc1974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:21.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.251+0000 7f6ed0fd9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ecc105a40 0x7f6ecc197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:21.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.251+0000 7f6ed0fd9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc197f90 con 0x7f6ecc0ffa40 2024-04-05T21:49:21.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.251+0000 7f6ed0fd9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc1980d0 con 0x7f6ecc105a40 2024-04-05T21:49:21.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.251+0000 7f6eca59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ecc0ffa40 0x7f6ecc1974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:21.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.251+0000 7f6eca59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ecc0ffa40 0x7f6ecc1974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52504/0 (socket says 172.21.15.121:52504) 2024-04-05T21:49:21.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.251+0000 7f6eca59c700 1 -- 172.21.15.121:0/2778821266 learned_addr learned my addr 172.21.15.121:0/2778821266 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:21.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.251+0000 7f6eca59c700 1 -- 172.21.15.121:0/2778821266 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ecc105a40 msgr2=0x7f6ecc197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:21.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.251+0000 7f6eca59c700 1 --2- 172.21.15.121:0/2778821266 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ecc105a40 0x7f6ecc197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.251+0000 7f6eca59c700 1 -- 172.21.15.121:0/2778821266 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec00095f0 con 0x7f6ecc0ffa40 2024-04-05T21:49:21.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.252+0000 7f6eca59c700 1 --2- 172.21.15.121:0/2778821266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ecc0ffa40 0x7f6ecc1974c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0006010 tx=0x7f6ec000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:21.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.252+0000 7f6ebb7fe700 1 -- 172.21.15.121:0/2778821266 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ec001d070 con 0x7f6ecc0ffa40 2024-04-05T21:49:21.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.252+0000 7f6ebb7fe700 1 -- 172.21.15.121:0/2778821266 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ec000f460 con 0x7f6ecc0ffa40 2024-04-05T21:49:21.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.252+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/2778821266 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ecc1a0530 con 0x7f6ecc0ffa40 2024-04-05T21:49:21.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.252+0000 7f6ebb7fe700 1 -- 172.21.15.121:0/2778821266 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ec0021640 con 0x7f6ecc0ffa40 2024-04-05T21:49:21.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.252+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/2778821266 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ecc1a08e0 con 0x7f6ecc0ffa40 2024-04-05T21:49:21.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.253+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/2778821266 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ecc04ea90 con 0x7f6ecc0ffa40 2024-04-05T21:49:21.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.253+0000 7f6ebb7fe700 1 -- 172.21.15.121:0/2778821266 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6ec000fa20 con 0x7f6ecc0ffa40 2024-04-05T21:49:21.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.254+0000 7f6ebb7fe700 1 --2- 172.21.15.121:0/2778821266 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6eb406cce0 0x7f6eb406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:21.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.254+0000 7f6ec9d9b700 1 --2- 172.21.15.121:0/2778821266 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6eb406cce0 0x7f6eb406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:21.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.254+0000 7f6ebb7fe700 1 -- 172.21.15.121:0/2778821266 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6ec008d8c0 con 0x7f6ecc0ffa40 2024-04-05T21:49:21.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.254+0000 7f6ec9d9b700 1 --2- 172.21.15.121:0/2778821266 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6eb406cce0 0x7f6eb406f1a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc005fd0 tx=0x7f6ebc005d00 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.258+0000 7f6ebb7fe700 1 -- 172.21.15.121:0/2778821266 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6ec0057930 con 0x7f6ecc0ffa40 2024-04-05T21:49:21.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.467+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/2778821266 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ecc0fc6f0 con 0x7f6eb406cce0 2024-04-05T21:49:21.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.469+0000 7f6ebb7fe700 1 -- 172.21.15.121:0/2778821266 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6ecc0fc6f0 con 0x7f6eb406cce0 2024-04-05T21:49:21.469 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:49:21.469 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:49:21.469 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:49:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:49:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:49:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:49:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:49:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:49:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:49:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:49:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:49:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:49:21.470 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:49:21.470 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:49:21.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.471+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/2778821266 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6eb406cce0 msgr2=0x7f6eb406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:21.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.471+0000 7f6ed0fd9700 1 --2- 172.21.15.121:0/2778821266 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6eb406cce0 0x7f6eb406f1a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc005fd0 tx=0x7f6ebc005d00 comp rx=0 tx=0).stop 2024-04-05T21:49:21.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.471+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/2778821266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ecc0ffa40 msgr2=0x7f6ecc1974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:21.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.471+0000 7f6ed0fd9700 1 --2- 172.21.15.121:0/2778821266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ecc0ffa40 0x7f6ecc1974c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0006010 tx=0x7f6ec000be20 comp rx=0 tx=0).stop 2024-04-05T21:49:21.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.471+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/2778821266 shutdown_connections 2024-04-05T21:49:21.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.471+0000 7f6ed0fd9700 1 --2- 172.21.15.121:0/2778821266 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6eb406cce0 0x7f6eb406f1a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.471+0000 7f6ed0fd9700 1 --2- 172.21.15.121:0/2778821266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ecc0ffa40 0x7f6ecc1974c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.471+0000 7f6ed0fd9700 1 --2- 172.21.15.121:0/2778821266 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ecc105a40 0x7f6ecc197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.471+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/2778821266 >> 172.21.15.121:0/2778821266 conn(0x7f6ecc0fb570 msgr2=0x7f6ecc0fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:21.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.471+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/2778821266 shutdown_connections 2024-04-05T21:49:21.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.471+0000 7f6ed0fd9700 1 -- 172.21.15.121:0/2778821266 wait complete. 2024-04-05T21:49:21.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.573+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3067702262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44105a40 msgr2=0x7fbc44105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:21.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.573+0000 7fbc4b2f0700 1 --2- 172.21.15.121:0/3067702262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44105a40 0x7fbc44105e20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbc34009910 tx=0x7fbc34009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:21.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.573+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3067702262 shutdown_connections 2024-04-05T21:49:21.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.573+0000 7fbc4b2f0700 1 --2- 172.21.15.121:0/3067702262 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc440ffa40 0x7fbc440ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.573+0000 7fbc4b2f0700 1 --2- 172.21.15.121:0/3067702262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44105a40 0x7fbc44105e20 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.573+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3067702262 >> 172.21.15.121:0/3067702262 conn(0x7fbc440fb570 msgr2=0x7fbc440fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:21.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.573+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3067702262 shutdown_connections 2024-04-05T21:49:21.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.573+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3067702262 wait complete. 2024-04-05T21:49:21.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.574+0000 7fbc4b2f0700 1 Processor -- start 2024-04-05T21:49:21.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.574+0000 7fbc4b2f0700 1 -- start start 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.574+0000 7fbc4b2f0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc440ffa40 0x7fbc44196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.574+0000 7fbc4b2f0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44105a40 0x7fbc44196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.574+0000 7fbc4b2f0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc44197290 con 0x7fbc44105a40 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.574+0000 7fbc4b2f0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4419b020 con 0x7fbc440ffa40 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.574+0000 7fbc4888b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44105a40 0x7fbc44196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.574+0000 7fbc4888b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44105a40 0x7fbc44196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52514/0 (socket says 172.21.15.121:52514) 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.574+0000 7fbc4888b700 1 -- 172.21.15.121:0/3217934951 learned_addr learned my addr 172.21.15.121:0/3217934951 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.574+0000 7fbc4908c700 1 --2- 172.21.15.121:0/3217934951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc440ffa40 0x7fbc44196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.575+0000 7fbc4888b700 1 -- 172.21.15.121:0/3217934951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc440ffa40 msgr2=0x7fbc44196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.575+0000 7fbc4888b700 1 --2- 172.21.15.121:0/3217934951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc440ffa40 0x7fbc44196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.575+0000 7fbc4888b700 1 -- 172.21.15.121:0/3217934951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc340095f0 con 0x7fbc44105a40 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.575+0000 7fbc4888b700 1 --2- 172.21.15.121:0/3217934951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44105a40 0x7fbc44196bb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000e920 tx=0x7fbc4000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.575+0000 7fbc3a7fc700 1 -- 172.21.15.121:0/3217934951 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc40009790 con 0x7fbc44105a40 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.575+0000 7fbc3a7fc700 1 -- 172.21.15.121:0/3217934951 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc4000cba0 con 0x7fbc44105a40 2024-04-05T21:49:21.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.575+0000 7fbc3a7fc700 1 -- 172.21.15.121:0/3217934951 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc40018650 con 0x7fbc44105a40 2024-04-05T21:49:21.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.575+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3217934951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4419b300 con 0x7fbc44105a40 2024-04-05T21:49:21.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.575+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3217934951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4419b850 con 0x7fbc44105a40 2024-04-05T21:49:21.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.576+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3217934951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc44109050 con 0x7fbc44105a40 2024-04-05T21:49:21.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.576+0000 7fbc3a7fc700 1 -- 172.21.15.121:0/3217934951 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbc4000cd10 con 0x7fbc44105a40 2024-04-05T21:49:21.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.577+0000 7fbc3a7fc700 1 --2- 172.21.15.121:0/3217934951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbc3006cce0 0x7fbc3006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:21.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.577+0000 7fbc3a7fc700 1 -- 172.21.15.121:0/3217934951 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbc40014070 con 0x7fbc44105a40 2024-04-05T21:49:21.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.577+0000 7fbc4908c700 1 --2- 172.21.15.121:0/3217934951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbc3006cce0 0x7fbc3006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:21.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.577+0000 7fbc4908c700 1 --2- 172.21.15.121:0/3217934951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbc3006cce0 0x7fbc3006f1a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbc34006010 tx=0x7fbc3400b5a0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:21.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.581+0000 7fbc3a7fc700 1 -- 172.21.15.121:0/3217934951 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbc40052ab0 con 0x7fbc44105a40 2024-04-05T21:49:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:21 smithi188 ceph-mon[107793]: from='client.35102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:21 smithi188 ceph-mon[107793]: from='client.35106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:21 smithi188 ceph-mon[107793]: pgmap v693: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:21 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2388089797' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:49:21.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:21 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1848147998' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:49:21.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.854+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3217934951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbc4419bbb0 con 0x7fbc44105a40 2024-04-05T21:49:21.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.855+0000 7fbc3a7fc700 1 -- 172.21.15.121:0/3217934951 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fbc4005a3f0 con 0x7fbc44105a40 2024-04-05T21:49:21.855 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:49:21.856 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:49:21.856 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:49:21.856 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:49:21.856 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:49:21.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.857+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3217934951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbc3006cce0 msgr2=0x7fbc3006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:21.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.857+0000 7fbc4b2f0700 1 --2- 172.21.15.121:0/3217934951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbc3006cce0 0x7fbc3006f1a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbc34006010 tx=0x7fbc3400b5a0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.857+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3217934951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44105a40 msgr2=0x7fbc44196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:21.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.857+0000 7fbc4b2f0700 1 --2- 172.21.15.121:0/3217934951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44105a40 0x7fbc44196bb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000e920 tx=0x7fbc4000ece0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.857+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3217934951 shutdown_connections 2024-04-05T21:49:21.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.857+0000 7fbc4b2f0700 1 --2- 172.21.15.121:0/3217934951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbc3006cce0 0x7fbc3006f1a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.857+0000 7fbc4b2f0700 1 --2- 172.21.15.121:0/3217934951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc440ffa40 0x7fbc44196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.857+0000 7fbc4b2f0700 1 --2- 172.21.15.121:0/3217934951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc44105a40 0x7fbc44196bb0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:21.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.857+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3217934951 >> 172.21.15.121:0/3217934951 conn(0x7fbc440fb570 msgr2=0x7fbc44068f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:21.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.857+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3217934951 shutdown_connections 2024-04-05T21:49:21.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:21.857+0000 7fbc4b2f0700 1 -- 172.21.15.121:0/3217934951 wait complete. 2024-04-05T21:49:21.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:21 smithi121 ceph-mon[145859]: from='client.35102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:21.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:21 smithi121 ceph-mon[145859]: from='client.35106 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:21.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:21 smithi121 ceph-mon[145859]: pgmap v693: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:21.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:21 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2388089797' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:49:21.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:21 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1848147998' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:49:22.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:22 smithi188 ceph-mon[107793]: from='client.35118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:22.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:22 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3217934951' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:49:22.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:22 smithi121 ceph-mon[145859]: from='client.35118 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:22.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:22 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3217934951' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:49:23.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:23 smithi188 ceph-mon[107793]: pgmap v694: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:23.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:23 smithi121 ceph-mon[145859]: pgmap v694: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:25.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:25 smithi188 ceph-mon[107793]: pgmap v695: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:25.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:25 smithi121 ceph-mon[145859]: pgmap v695: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:27.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:27 smithi188 ceph-mon[107793]: pgmap v696: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:27.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:27 smithi121 ceph-mon[145859]: pgmap v696: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:29.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:29 smithi121 ceph-mon[145859]: pgmap v697: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:29.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:29 smithi188 ceph-mon[107793]: pgmap v697: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:30.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:49:31.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:49:31.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:31 smithi121 ceph-mon[145859]: pgmap v698: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:32.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:31 smithi188 ceph-mon[107793]: pgmap v698: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:33.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:33 smithi121 ceph-mon[145859]: pgmap v699: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:34.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:33 smithi188 ceph-mon[107793]: pgmap v699: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:35.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:35 smithi121 ceph-mon[145859]: pgmap v700: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:36.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:35 smithi188 ceph-mon[107793]: pgmap v700: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:37.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:37 smithi121 ceph-mon[145859]: pgmap v701: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:38.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:37 smithi188 ceph-mon[107793]: pgmap v701: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:39.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:39 smithi121 ceph-mon[145859]: pgmap v702: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:40.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:39 smithi188 ceph-mon[107793]: pgmap v702: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:41.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:41 smithi121 ceph-mon[145859]: pgmap v703: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:42.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:41 smithi188 ceph-mon[107793]: pgmap v703: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:43.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:43 smithi121 ceph-mon[145859]: pgmap v704: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:44.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:43 smithi188 ceph-mon[107793]: pgmap v704: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:45.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:45 smithi121 ceph-mon[145859]: pgmap v705: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:45.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:49:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:45 smithi188 ceph-mon[107793]: pgmap v705: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:49:47.873 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:47 smithi188 ceph-mon[107793]: pgmap v706: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:47.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:47 smithi121 ceph-mon[145859]: pgmap v706: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:49.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:49 smithi121 ceph-mon[145859]: pgmap v707: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:50.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:49 smithi188 ceph-mon[107793]: pgmap v707: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:51.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.972+0000 7f993693b700 1 -- 172.21.15.121:0/2844974231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99300ffbd0 msgr2=0x7f9930100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:51.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.972+0000 7f993693b700 1 --2- 172.21.15.121:0/2844974231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99300ffbd0 0x7f9930100050 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9920009910 tx=0x7f9920009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:51.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.972+0000 7f993693b700 1 -- 172.21.15.121:0/2844974231 shutdown_connections 2024-04-05T21:49:51.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.972+0000 7f993693b700 1 --2- 172.21.15.121:0/2844974231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99300ffbd0 0x7f9930100050 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:51.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.972+0000 7f993693b700 1 --2- 172.21.15.121:0/2844974231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9930105a40 0x7f9930105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:51.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.972+0000 7f993693b700 1 -- 172.21.15.121:0/2844974231 >> 172.21.15.121:0/2844974231 conn(0x7f99300755a0 msgr2=0x7f99300759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:51.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.972+0000 7f993693b700 1 -- 172.21.15.121:0/2844974231 shutdown_connections 2024-04-05T21:49:51.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.972+0000 7f993693b700 1 -- 172.21.15.121:0/2844974231 wait complete. 2024-04-05T21:49:51.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.973+0000 7f993693b700 1 Processor -- start 2024-04-05T21:49:51.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.973+0000 7f993693b700 1 -- start start 2024-04-05T21:49:51.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.973+0000 7f993693b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99300ffbd0 0x7f993019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.973+0000 7f993693b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9930105a40 0x7f993019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.973+0000 7f993693b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993019d0e0 con 0x7f9930105a40 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.973+0000 7f993693b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9930196590 con 0x7f99300ffbd0 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.973+0000 7f992f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9930105a40 0x7f993019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.973+0000 7f992f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9930105a40 0x7f993019ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51136/0 (socket says 172.21.15.121:51136) 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.974+0000 7f992f7fe700 1 -- 172.21.15.121:0/4166357179 learned_addr learned my addr 172.21.15.121:0/4166357179 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.974+0000 7f992ffff700 1 --2- 172.21.15.121:0/4166357179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99300ffbd0 0x7f993019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.974+0000 7f992f7fe700 1 -- 172.21.15.121:0/4166357179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99300ffbd0 msgr2=0x7f993019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.974+0000 7f992f7fe700 1 --2- 172.21.15.121:0/4166357179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99300ffbd0 0x7f993019c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.974+0000 7f992f7fe700 1 -- 172.21.15.121:0/4166357179 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99200095f0 con 0x7f9930105a40 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.974+0000 7f992f7fe700 1 --2- 172.21.15.121:0/4166357179 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9930105a40 0x7f993019ca50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f99200092e0 tx=0x7f992000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.974+0000 7f992d7fa700 1 -- 172.21.15.121:0/4166357179 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f992001d070 con 0x7f9930105a40 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.974+0000 7f992d7fa700 1 -- 172.21.15.121:0/4166357179 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9920005e00 con 0x7f9930105a40 2024-04-05T21:49:51.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.974+0000 7f993693b700 1 -- 172.21.15.121:0/4166357179 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9930196810 con 0x7f9930105a40 2024-04-05T21:49:51.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.974+0000 7f992d7fa700 1 -- 172.21.15.121:0/4166357179 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f992000fe60 con 0x7f9930105a40 2024-04-05T21:49:51.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.974+0000 7f993693b700 1 -- 172.21.15.121:0/4166357179 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9930196d00 con 0x7f9930105a40 2024-04-05T21:49:51.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.975+0000 7f993693b700 1 -- 172.21.15.121:0/4166357179 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9930108fe0 con 0x7f9930105a40 2024-04-05T21:49:51.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.976+0000 7f992d7fa700 1 -- 172.21.15.121:0/4166357179 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f992000f6d0 con 0x7f9930105a40 2024-04-05T21:49:51.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.976+0000 7f992d7fa700 1 --2- 172.21.15.121:0/4166357179 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f991806cce0 0x7f991806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:51.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.976+0000 7f992d7fa700 1 -- 172.21.15.121:0/4166357179 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f992008d7e0 con 0x7f9930105a40 2024-04-05T21:49:51.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.976+0000 7f992ffff700 1 --2- 172.21.15.121:0/4166357179 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f991806cce0 0x7f991806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:51.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.977+0000 7f992ffff700 1 --2- 172.21.15.121:0/4166357179 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f991806cce0 0x7f991806f1a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9924005950 tx=0x7f99240058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:51.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:51.981+0000 7f992d7fa700 1 -- 172.21.15.121:0/4166357179 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f992005b2e0 con 0x7f9930105a40 2024-04-05T21:49:51.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:51 smithi121 ceph-mon[145859]: pgmap v708: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:52.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:51 smithi188 ceph-mon[107793]: pgmap v708: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:52.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.191+0000 7f993693b700 1 -- 172.21.15.121:0/4166357179 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9930196f70 con 0x7f991806cce0 2024-04-05T21:49:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.192+0000 7f992d7fa700 1 -- 172.21.15.121:0/4166357179 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9930196f70 con 0x7f991806cce0 2024-04-05T21:49:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.194+0000 7f993693b700 1 -- 172.21.15.121:0/4166357179 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f991806cce0 msgr2=0x7f991806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.194+0000 7f993693b700 1 --2- 172.21.15.121:0/4166357179 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f991806cce0 0x7f991806f1a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9924005950 tx=0x7f99240058e0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.194+0000 7f993693b700 1 -- 172.21.15.121:0/4166357179 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9930105a40 msgr2=0x7f993019ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.194+0000 7f993693b700 1 --2- 172.21.15.121:0/4166357179 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9930105a40 0x7f993019ca50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f99200092e0 tx=0x7f992000be20 comp rx=0 tx=0).stop 2024-04-05T21:49:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.194+0000 7f993693b700 1 -- 172.21.15.121:0/4166357179 shutdown_connections 2024-04-05T21:49:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.194+0000 7f993693b700 1 --2- 172.21.15.121:0/4166357179 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f991806cce0 0x7f991806f1a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.194+0000 7f993693b700 1 --2- 172.21.15.121:0/4166357179 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f99300ffbd0 0x7f993019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.194+0000 7f993693b700 1 --2- 172.21.15.121:0/4166357179 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9930105a40 0x7f993019ca50 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.194+0000 7f993693b700 1 -- 172.21.15.121:0/4166357179 >> 172.21.15.121:0/4166357179 conn(0x7f99300755a0 msgr2=0x7f9930103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.194+0000 7f993693b700 1 -- 172.21.15.121:0/4166357179 shutdown_connections 2024-04-05T21:49:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.194+0000 7f993693b700 1 -- 172.21.15.121:0/4166357179 wait complete. 2024-04-05T21:49:52.205 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:49:52.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.297+0000 7fd516db1700 1 -- 172.21.15.121:0/2192367871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5100ffa50 msgr2=0x7fd5100ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:52.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.297+0000 7fd516db1700 1 --2- 172.21.15.121:0/2192367871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5100ffa50 0x7fd5100ffed0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc009910 tx=0x7fd4fc009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:52.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.297+0000 7fd516db1700 1 -- 172.21.15.121:0/2192367871 shutdown_connections 2024-04-05T21:49:52.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.297+0000 7fd516db1700 1 --2- 172.21.15.121:0/2192367871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5100ffa50 0x7fd5100ffed0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.297+0000 7fd516db1700 1 --2- 172.21.15.121:0/2192367871 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd510105a50 0x7fd510105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.297+0000 7fd516db1700 1 -- 172.21.15.121:0/2192367871 >> 172.21.15.121:0/2192367871 conn(0x7fd5100fb540 msgr2=0x7fd5100fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:52.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.298+0000 7fd516db1700 1 -- 172.21.15.121:0/2192367871 shutdown_connections 2024-04-05T21:49:52.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.298+0000 7fd516db1700 1 -- 172.21.15.121:0/2192367871 wait complete. 2024-04-05T21:49:52.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.298+0000 7fd516db1700 1 Processor -- start 2024-04-05T21:49:52.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.298+0000 7fd516db1700 1 -- start start 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.298+0000 7fd516db1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5100ffa50 0x7fd510197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.298+0000 7fd516db1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd510105a50 0x7fd510197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.298+0000 7fd516db1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd510198090 con 0x7fd5100ffa50 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.298+0000 7fd516db1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd510198200 con 0x7fd510105a50 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.298+0000 7fd514b4d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5100ffa50 0x7fd510197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.298+0000 7fd514b4d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5100ffa50 0x7fd510197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51162/0 (socket says 172.21.15.121:51162) 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.298+0000 7fd514b4d700 1 -- 172.21.15.121:0/3642554492 learned_addr learned my addr 172.21.15.121:0/3642554492 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.299+0000 7fd514b4d700 1 -- 172.21.15.121:0/3642554492 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd510105a50 msgr2=0x7fd510197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.299+0000 7fd514b4d700 1 --2- 172.21.15.121:0/3642554492 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd510105a50 0x7fd510197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.299+0000 7fd514b4d700 1 -- 172.21.15.121:0/3642554492 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc0095f0 con 0x7fd5100ffa50 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.299+0000 7fd514b4d700 1 --2- 172.21.15.121:0/3642554492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5100ffa50 0x7fd510197530 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd50400c8a0 tx=0x7fd50400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.299+0000 7fd50dffb700 1 -- 172.21.15.121:0/3642554492 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd50400cea0 con 0x7fd5100ffa50 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.299+0000 7fd516db1700 1 -- 172.21.15.121:0/3642554492 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5101a05a0 con 0x7fd5100ffa50 2024-04-05T21:49:52.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.299+0000 7fd50dffb700 1 -- 172.21.15.121:0/3642554492 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd504004830 con 0x7fd5100ffa50 2024-04-05T21:49:52.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.299+0000 7fd50dffb700 1 -- 172.21.15.121:0/3642554492 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd504005500 con 0x7fd5100ffa50 2024-04-05T21:49:52.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.299+0000 7fd516db1700 1 -- 172.21.15.121:0/3642554492 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5101a0a90 con 0x7fd5100ffa50 2024-04-05T21:49:52.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.300+0000 7fd516db1700 1 -- 172.21.15.121:0/3642554492 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd510107fe0 con 0x7fd5100ffa50 2024-04-05T21:49:52.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.300+0000 7fd50dffb700 1 -- 172.21.15.121:0/3642554492 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd5040049a0 con 0x7fd5100ffa50 2024-04-05T21:49:52.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.301+0000 7fd50dffb700 1 --2- 172.21.15.121:0/3642554492 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd50006cce0 0x7fd50006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:52.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.301+0000 7fd50dffb700 1 -- 172.21.15.121:0/3642554492 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd50408c250 con 0x7fd5100ffa50 2024-04-05T21:49:52.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.301+0000 7fd50ffff700 1 --2- 172.21.15.121:0/3642554492 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd50006cce0 0x7fd50006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:52.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.301+0000 7fd50ffff700 1 --2- 172.21.15.121:0/3642554492 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd50006cce0 0x7fd50006f1a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc006010 tx=0x7fd4fc00b540 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:52.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.306+0000 7fd50dffb700 1 -- 172.21.15.121:0/3642554492 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd5040562c0 con 0x7fd5100ffa50 2024-04-05T21:49:52.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.518+0000 7fd516db1700 1 -- 172.21.15.121:0/3642554492 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd51002cce0 con 0x7fd50006cce0 2024-04-05T21:49:52.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.519+0000 7fd50dffb700 1 -- 172.21.15.121:0/3642554492 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd51002cce0 con 0x7fd50006cce0 2024-04-05T21:49:52.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.521+0000 7fd516db1700 1 -- 172.21.15.121:0/3642554492 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd50006cce0 msgr2=0x7fd50006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:52.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.521+0000 7fd516db1700 1 --2- 172.21.15.121:0/3642554492 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd50006cce0 0x7fd50006f1a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc006010 tx=0x7fd4fc00b540 comp rx=0 tx=0).stop 2024-04-05T21:49:52.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.521+0000 7fd516db1700 1 -- 172.21.15.121:0/3642554492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5100ffa50 msgr2=0x7fd510197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:52.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.521+0000 7fd516db1700 1 --2- 172.21.15.121:0/3642554492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5100ffa50 0x7fd510197530 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd50400c8a0 tx=0x7fd50400cc60 comp rx=0 tx=0).stop 2024-04-05T21:49:52.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.521+0000 7fd516db1700 1 -- 172.21.15.121:0/3642554492 shutdown_connections 2024-04-05T21:49:52.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.521+0000 7fd516db1700 1 --2- 172.21.15.121:0/3642554492 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd50006cce0 0x7fd50006f1a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.521+0000 7fd516db1700 1 --2- 172.21.15.121:0/3642554492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5100ffa50 0x7fd510197530 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.521+0000 7fd516db1700 1 --2- 172.21.15.121:0/3642554492 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd510105a50 0x7fd510197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.521+0000 7fd516db1700 1 -- 172.21.15.121:0/3642554492 >> 172.21.15.121:0/3642554492 conn(0x7fd5100fb540 msgr2=0x7fd5100fc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:52.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.521+0000 7fd516db1700 1 -- 172.21.15.121:0/3642554492 shutdown_connections 2024-04-05T21:49:52.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.521+0000 7fd516db1700 1 -- 172.21.15.121:0/3642554492 wait complete. 2024-04-05T21:49:52.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.626+0000 7f06e7fd7700 1 -- 172.21.15.121:0/4043164243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06e0101790 msgr2=0x7f06e01057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:52.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.626+0000 7f06e7fd7700 1 --2- 172.21.15.121:0/4043164243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06e0101790 0x7f06e01057e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009910 tx=0x7f06d0009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:52.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.626+0000 7f06e7fd7700 1 -- 172.21.15.121:0/4043164243 shutdown_connections 2024-04-05T21:49:52.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.626+0000 7f06e7fd7700 1 --2- 172.21.15.121:0/4043164243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06e0101790 0x7f06e01057e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.626+0000 7f06e7fd7700 1 --2- 172.21.15.121:0/4043164243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06e0100de0 0x7f06e01011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.626+0000 7f06e7fd7700 1 -- 172.21.15.121:0/4043164243 >> 172.21.15.121:0/4043164243 conn(0x7f06e00fc6b0 msgr2=0x7f06e00fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:52.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.626+0000 7f06e7fd7700 1 -- 172.21.15.121:0/4043164243 shutdown_connections 2024-04-05T21:49:52.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.626+0000 7f06e7fd7700 1 -- 172.21.15.121:0/4043164243 wait complete. 2024-04-05T21:49:52.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.626+0000 7f06e7fd7700 1 Processor -- start 2024-04-05T21:49:52.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.626+0000 7f06e7fd7700 1 -- start start 2024-04-05T21:49:52.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.626+0000 7f06e7fd7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06e0100de0 0x7f06e0196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:52.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06e7fd7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06e0101790 0x7f06e0196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:52.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06e7fd7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e0197290 con 0x7f06e0101790 2024-04-05T21:49:52.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06e7fd7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e019b020 con 0x7f06e0100de0 2024-04-05T21:49:52.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06e5572700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06e0101790 0x7f06e0196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:52.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06e5572700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06e0101790 0x7f06e0196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51178/0 (socket says 172.21.15.121:51178) 2024-04-05T21:49:52.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06e5572700 1 -- 172.21.15.121:0/2593718034 learned_addr learned my addr 172.21.15.121:0/2593718034 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:52.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06e5d73700 1 --2- 172.21.15.121:0/2593718034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06e0100de0 0x7f06e0196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:52.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06e5572700 1 -- 172.21.15.121:0/2593718034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06e0100de0 msgr2=0x7f06e0196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:52.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06e5572700 1 --2- 172.21.15.121:0/2593718034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06e0100de0 0x7f06e0196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06e5572700 1 -- 172.21.15.121:0/2593718034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d00095f0 con 0x7f06e0101790 2024-04-05T21:49:52.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06e5572700 1 --2- 172.21.15.121:0/2593718034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06e0101790 0x7f06e0196bb0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f06d000bda0 tx=0x7f06d000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:52.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06d6ffd700 1 -- 172.21.15.121:0/2593718034 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06d001d070 con 0x7f06e0101790 2024-04-05T21:49:52.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06d6ffd700 1 -- 172.21.15.121:0/2593718034 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06d000f460 con 0x7f06e0101790 2024-04-05T21:49:52.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06d6ffd700 1 -- 172.21.15.121:0/2593718034 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06d0021640 con 0x7f06e0101790 2024-04-05T21:49:52.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06e7fd7700 1 -- 172.21.15.121:0/2593718034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e019b2a0 con 0x7f06e0101790 2024-04-05T21:49:52.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.627+0000 7f06e7fd7700 1 -- 172.21.15.121:0/2593718034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e019b790 con 0x7f06e0101790 2024-04-05T21:49:52.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.628+0000 7f06e7fd7700 1 -- 172.21.15.121:0/2593718034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06e01091e0 con 0x7f06e0101790 2024-04-05T21:49:52.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.629+0000 7f06d6ffd700 1 -- 172.21.15.121:0/2593718034 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f06d000fa20 con 0x7f06e0101790 2024-04-05T21:49:52.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.629+0000 7f06d6ffd700 1 --2- 172.21.15.121:0/2593718034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f06cc06cce0 0x7f06cc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:52.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.629+0000 7f06d6ffd700 1 -- 172.21.15.121:0/2593718034 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f06d008d780 con 0x7f06e0101790 2024-04-05T21:49:52.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.629+0000 7f06e5d73700 1 --2- 172.21.15.121:0/2593718034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f06cc06cce0 0x7f06cc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:52.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.629+0000 7f06e5d73700 1 --2- 172.21.15.121:0/2593718034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f06cc06cce0 0x7f06cc06f1a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f06dc005950 tx=0x7f06dc0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:52.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.633+0000 7f06d6ffd700 1 -- 172.21.15.121:0/2593718034 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f06d00577f0 con 0x7f06e0101790 2024-04-05T21:49:52.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.832+0000 7f06e7fd7700 1 -- 172.21.15.121:0/2593718034 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f06e002cd60 con 0x7f06cc06cce0 2024-04-05T21:49:52.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.842+0000 7f06d6ffd700 1 -- 172.21.15.121:0/2593718034 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f06e002cd60 con 0x7f06cc06cce0 2024-04-05T21:49:52.843 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:49:52.843 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (20m) 14m ago 32m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:49:52.843 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (32m) 14m ago 32m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:49:52.843 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (31m) 14m ago 31m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:49:52.843 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (18m) 14m ago 32m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:49:52.843 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (18m) 14m ago 31m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (20m) 14m ago 31m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (15m) 14m ago 26m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (14m) 14m ago 25m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (14m) 14m ago 26m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (25m) 14m ago 25m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (22m) 14m ago 33m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (21m) 14m ago 31m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (19m) 14m ago 33m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (19m) 14m ago 30m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (21m) 14m ago 32m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (21m) 14m ago 31m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (18m) 14m ago 30m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (17m) 14m ago 29m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:49:52.844 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (17m) 14m ago 29m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:49:52.845 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (16m) 14m ago 28m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:49:52.845 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (15m) 14m ago 28m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:49:52.845 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (15m) 14m ago 27m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:49:52.845 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (21m) 14m ago 31m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:49:52.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.845+0000 7f06e7fd7700 1 -- 172.21.15.121:0/2593718034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f06cc06cce0 msgr2=0x7f06cc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:52.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.845+0000 7f06e7fd7700 1 --2- 172.21.15.121:0/2593718034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f06cc06cce0 0x7f06cc06f1a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f06dc005950 tx=0x7f06dc0058e0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.845+0000 7f06e7fd7700 1 -- 172.21.15.121:0/2593718034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06e0101790 msgr2=0x7f06e0196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:52.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.845+0000 7f06e7fd7700 1 --2- 172.21.15.121:0/2593718034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06e0101790 0x7f06e0196bb0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f06d000bda0 tx=0x7f06d000be80 comp rx=0 tx=0).stop 2024-04-05T21:49:52.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.845+0000 7f06e7fd7700 1 -- 172.21.15.121:0/2593718034 shutdown_connections 2024-04-05T21:49:52.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.845+0000 7f06e7fd7700 1 --2- 172.21.15.121:0/2593718034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f06cc06cce0 0x7f06cc06f1a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.845+0000 7f06e7fd7700 1 --2- 172.21.15.121:0/2593718034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06e0100de0 0x7f06e0196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.845+0000 7f06e7fd7700 1 --2- 172.21.15.121:0/2593718034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06e0101790 0x7f06e0196bb0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.845+0000 7f06e7fd7700 1 -- 172.21.15.121:0/2593718034 >> 172.21.15.121:0/2593718034 conn(0x7f06e00fc6b0 msgr2=0x7f06e00fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:52.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.845+0000 7f06e7fd7700 1 -- 172.21.15.121:0/2593718034 shutdown_connections 2024-04-05T21:49:52.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.845+0000 7f06e7fd7700 1 -- 172.21.15.121:0/2593718034 wait complete. 2024-04-05T21:49:52.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.946+0000 7f756f3c4700 1 -- 172.21.15.121:0/1660887529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7568068670 msgr2=0x7f7568068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:52.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.946+0000 7f756f3c4700 1 --2- 172.21.15.121:0/1660887529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7568068670 0x7f7568068a50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f7564009910 tx=0x7f7564009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:52.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.946+0000 7f756f3c4700 1 -- 172.21.15.121:0/1660887529 shutdown_connections 2024-04-05T21:49:52.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.946+0000 7f756f3c4700 1 --2- 172.21.15.121:0/1660887529 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7568069020 0x7f7568105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.946+0000 7f756f3c4700 1 --2- 172.21.15.121:0/1660887529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7568068670 0x7f7568068a50 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.946+0000 7f756f3c4700 1 -- 172.21.15.121:0/1660887529 >> 172.21.15.121:0/1660887529 conn(0x7f75680755a0 msgr2=0x7f75680759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.946+0000 7f756f3c4700 1 -- 172.21.15.121:0/1660887529 shutdown_connections 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.947+0000 7f756f3c4700 1 -- 172.21.15.121:0/1660887529 wait complete. 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.947+0000 7f756f3c4700 1 Processor -- start 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.947+0000 7f756f3c4700 1 -- start start 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756f3c4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7568068670 0x7f7568196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756f3c4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7568069020 0x7f7568196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756f3c4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7568197290 con 0x7f7568068670 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756f3c4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756819b020 con 0x7f7568069020 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756d160700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7568068670 0x7f7568196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756d160700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7568068670 0x7f7568196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51200/0 (socket says 172.21.15.121:51200) 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756d160700 1 -- 172.21.15.121:0/1983640034 learned_addr learned my addr 172.21.15.121:0/1983640034 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756c95f700 1 --2- 172.21.15.121:0/1983640034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7568069020 0x7f7568196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756d160700 1 -- 172.21.15.121:0/1983640034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7568069020 msgr2=0x7f7568196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:52.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756d160700 1 --2- 172.21.15.121:0/1983640034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7568069020 0x7f7568196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:52.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756d160700 1 -- 172.21.15.121:0/1983640034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75640095f0 con 0x7f7568068670 2024-04-05T21:49:52.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756d160700 1 --2- 172.21.15.121:0/1983640034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7568068670 0x7f7568196670 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7564006010 tx=0x7f756400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:52.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f755e7fc700 1 -- 172.21.15.121:0/1983640034 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f756401d070 con 0x7f7568068670 2024-04-05T21:49:52.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756f3c4700 1 -- 172.21.15.121:0/1983640034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f756819b2a0 con 0x7f7568068670 2024-04-05T21:49:52.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f755e7fc700 1 -- 172.21.15.121:0/1983640034 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7564005dc0 con 0x7f7568068670 2024-04-05T21:49:52.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f755e7fc700 1 -- 172.21.15.121:0/1983640034 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f756400fe20 con 0x7f7568068670 2024-04-05T21:49:52.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.948+0000 7f756f3c4700 1 -- 172.21.15.121:0/1983640034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756819b790 con 0x7f7568068670 2024-04-05T21:49:52.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.949+0000 7f756f3c4700 1 -- 172.21.15.121:0/1983640034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7568109130 con 0x7f7568068670 2024-04-05T21:49:52.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.950+0000 7f755e7fc700 1 -- 172.21.15.121:0/1983640034 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f756400f660 con 0x7f7568068670 2024-04-05T21:49:52.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.950+0000 7f755e7fc700 1 --2- 172.21.15.121:0/1983640034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f755406cce0 0x7f755406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:52.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.950+0000 7f755e7fc700 1 -- 172.21.15.121:0/1983640034 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f756408d740 con 0x7f7568068670 2024-04-05T21:49:52.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.950+0000 7f756c95f700 1 --2- 172.21.15.121:0/1983640034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f755406cce0 0x7f755406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:52.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.951+0000 7f756c95f700 1 --2- 172.21.15.121:0/1983640034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f755406cce0 0x7f755406f1a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7568197c90 tx=0x7f755800b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:52.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:52.955+0000 7f755e7fc700 1 -- 172.21.15.121:0/1983640034 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f75640577b0 con 0x7f7568068670 2024-04-05T21:49:53.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.224+0000 7f756f3c4700 1 -- 172.21.15.121:0/1983640034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f756802cd60 con 0x7f7568068670 2024-04-05T21:49:53.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.224+0000 7f755e7fc700 1 -- 172.21.15.121:0/1983640034 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7f756405add0 con 0x7f7568068670 2024-04-05T21:49:53.225 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:49:53.225 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:49:53.225 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:49:53.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:49:53.225 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:49:53.226 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:49:53.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.226+0000 7f756f3c4700 1 -- 172.21.15.121:0/1983640034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f755406cce0 msgr2=0x7f755406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:53.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.226+0000 7f756f3c4700 1 --2- 172.21.15.121:0/1983640034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f755406cce0 0x7f755406f1a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7568197c90 tx=0x7f755800b410 comp rx=0 tx=0).stop 2024-04-05T21:49:53.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.226+0000 7f756f3c4700 1 -- 172.21.15.121:0/1983640034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7568068670 msgr2=0x7f7568196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:53.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.226+0000 7f756f3c4700 1 --2- 172.21.15.121:0/1983640034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7568068670 0x7f7568196670 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7564006010 tx=0x7f756400bd70 comp rx=0 tx=0).stop 2024-04-05T21:49:53.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.226+0000 7f756f3c4700 1 -- 172.21.15.121:0/1983640034 shutdown_connections 2024-04-05T21:49:53.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.226+0000 7f756f3c4700 1 --2- 172.21.15.121:0/1983640034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f755406cce0 0x7f755406f1a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.226+0000 7f756f3c4700 1 --2- 172.21.15.121:0/1983640034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7568068670 0x7f7568196670 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.226+0000 7f756f3c4700 1 --2- 172.21.15.121:0/1983640034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7568069020 0x7f7568196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.226+0000 7f756f3c4700 1 -- 172.21.15.121:0/1983640034 >> 172.21.15.121:0/1983640034 conn(0x7f75680755a0 msgr2=0x7f75680fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:53.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.227+0000 7f756f3c4700 1 -- 172.21.15.121:0/1983640034 shutdown_connections 2024-04-05T21:49:53.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.227+0000 7f756f3c4700 1 -- 172.21.15.121:0/1983640034 wait complete. 2024-04-05T21:49:53.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.328+0000 7fdcc91fd700 1 -- 172.21.15.121:0/2922149013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcc4105a40 msgr2=0x7fdcc4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:53.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.328+0000 7fdcc91fd700 1 --2- 172.21.15.121:0/2922149013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcc4105a40 0x7fdcc4105e20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009910 tx=0x7fdcac009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:53.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.328+0000 7fdcc91fd700 1 -- 172.21.15.121:0/2922149013 shutdown_connections 2024-04-05T21:49:53.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.328+0000 7fdcc91fd700 1 --2- 172.21.15.121:0/2922149013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdcc40ffbd0 0x7fdcc4100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.328+0000 7fdcc91fd700 1 --2- 172.21.15.121:0/2922149013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcc4105a40 0x7fdcc4105e20 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.328+0000 7fdcc91fd700 1 -- 172.21.15.121:0/2922149013 >> 172.21.15.121:0/2922149013 conn(0x7fdcc40755a0 msgr2=0x7fdcc40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:53.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.328+0000 7fdcc91fd700 1 -- 172.21.15.121:0/2922149013 shutdown_connections 2024-04-05T21:49:53.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.328+0000 7fdcc91fd700 1 -- 172.21.15.121:0/2922149013 wait complete. 2024-04-05T21:49:53.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.329+0000 7fdcc91fd700 1 Processor -- start 2024-04-05T21:49:53.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.329+0000 7fdcc91fd700 1 -- start start 2024-04-05T21:49:53.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.329+0000 7fdcc91fd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdcc40ffbd0 0x7fdcc419c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:53.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.329+0000 7fdcc91fd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcc4105a40 0x7fdcc419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:53.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.329+0000 7fdcc91fd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc419d0d0 con 0x7fdcc4105a40 2024-04-05T21:49:53.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.329+0000 7fdcc91fd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc4196580 con 0x7fdcc40ffbd0 2024-04-05T21:49:53.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.329+0000 7fdcc259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcc4105a40 0x7fdcc419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:53.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.329+0000 7fdcc259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcc4105a40 0x7fdcc419ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51208/0 (socket says 172.21.15.121:51208) 2024-04-05T21:49:53.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.329+0000 7fdcc259c700 1 -- 172.21.15.121:0/435826176 learned_addr learned my addr 172.21.15.121:0/435826176 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:53.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.329+0000 7fdcc2d9d700 1 --2- 172.21.15.121:0/435826176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdcc40ffbd0 0x7fdcc419c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:53.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.330+0000 7fdcc259c700 1 -- 172.21.15.121:0/435826176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdcc40ffbd0 msgr2=0x7fdcc419c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:53.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.330+0000 7fdcc259c700 1 --2- 172.21.15.121:0/435826176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdcc40ffbd0 0x7fdcc419c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.330+0000 7fdcc259c700 1 -- 172.21.15.121:0/435826176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac0095f0 con 0x7fdcc4105a40 2024-04-05T21:49:53.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.330+0000 7fdcc259c700 1 --2- 172.21.15.121:0/435826176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcc4105a40 0x7fdcc419ca40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400b4b0 tx=0x7fdcb400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:53.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.330+0000 7fdcbbfff700 1 -- 172.21.15.121:0/435826176 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcb4011800 con 0x7fdcc4105a40 2024-04-05T21:49:53.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.330+0000 7fdcc91fd700 1 -- 172.21.15.121:0/435826176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc4196860 con 0x7fdcc4105a40 2024-04-05T21:49:53.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.330+0000 7fdcbbfff700 1 -- 172.21.15.121:0/435826176 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcb4011e00 con 0x7fdcc4105a40 2024-04-05T21:49:53.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.330+0000 7fdcbbfff700 1 -- 172.21.15.121:0/435826176 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcb4010e60 con 0x7fdcc4105a40 2024-04-05T21:49:53.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.330+0000 7fdcc91fd700 1 -- 172.21.15.121:0/435826176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc4196db0 con 0x7fdcc4105a40 2024-04-05T21:49:53.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.331+0000 7fdcc91fd700 1 -- 172.21.15.121:0/435826176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc4109050 con 0x7fdcc4105a40 2024-04-05T21:49:53.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.331+0000 7fdcbbfff700 1 -- 172.21.15.121:0/435826176 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdcb4011960 con 0x7fdcc4105a40 2024-04-05T21:49:53.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.332+0000 7fdcbbfff700 1 --2- 172.21.15.121:0/435826176 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdcb006cce0 0x7fdcb006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:53.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.332+0000 7fdcbbfff700 1 -- 172.21.15.121:0/435826176 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdcb408c080 con 0x7fdcc4105a40 2024-04-05T21:49:53.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.332+0000 7fdcc2d9d700 1 --2- 172.21.15.121:0/435826176 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdcb006cce0 0x7fdcb006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:53.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.332+0000 7fdcc2d9d700 1 --2- 172.21.15.121:0/435826176 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdcb006cce0 0x7fdcb006f1a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdcac006010 tx=0x7fdcac01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:53.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.336+0000 7fdcbbfff700 1 -- 172.21.15.121:0/435826176 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdcb4059c00 con 0x7fdcc4105a40 2024-04-05T21:49:53.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.575+0000 7fdcc91fd700 1 -- 172.21.15.121:0/435826176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fdcc404ea90 con 0x7fdcc4105a40 2024-04-05T21:49:53.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.575+0000 7fdcbbfff700 1 -- 172.21.15.121:0/435826176 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7fdcb4059790 con 0x7fdcc4105a40 2024-04-05T21:49:53.576 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:49:53.576 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:49:53.576 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:49:53.576 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:49:53.576 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:49:53.576 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:49:53.576 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:49:53.576 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:49:53.576 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:49:53.577 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:49:53.578 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:49:53.578 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:49:53.578 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:49:53.578 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:49:53.578 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:49:53.578 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:49:53.578 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:49:53.578 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:49:53.578 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:49:53.578 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:49:53.578 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:49:53.578 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:49:53.578 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:49:53.579 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:49:53.579 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:49:53.579 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:49:53.579 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:49:53.579 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:49:53.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.577+0000 7fdcc91fd700 1 -- 172.21.15.121:0/435826176 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdcb006cce0 msgr2=0x7fdcb006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:53.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.577+0000 7fdcc91fd700 1 --2- 172.21.15.121:0/435826176 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdcb006cce0 0x7fdcb006f1a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdcac006010 tx=0x7fdcac01a040 comp rx=0 tx=0).stop 2024-04-05T21:49:53.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.577+0000 7fdcc91fd700 1 -- 172.21.15.121:0/435826176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcc4105a40 msgr2=0x7fdcc419ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:53.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.577+0000 7fdcc91fd700 1 --2- 172.21.15.121:0/435826176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcc4105a40 0x7fdcc419ca40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400b4b0 tx=0x7fdcb400b7c0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.577+0000 7fdcc91fd700 1 -- 172.21.15.121:0/435826176 shutdown_connections 2024-04-05T21:49:53.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.577+0000 7fdcc91fd700 1 --2- 172.21.15.121:0/435826176 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdcb006cce0 0x7fdcb006f1a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.577+0000 7fdcc91fd700 1 --2- 172.21.15.121:0/435826176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdcc40ffbd0 0x7fdcc419c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.577+0000 7fdcc91fd700 1 --2- 172.21.15.121:0/435826176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcc4105a40 0x7fdcc419ca40 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.577+0000 7fdcc91fd700 1 -- 172.21.15.121:0/435826176 >> 172.21.15.121:0/435826176 conn(0x7fdcc40755a0 msgr2=0x7fdcc4103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:53.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.578+0000 7fdcc91fd700 1 -- 172.21.15.121:0/435826176 shutdown_connections 2024-04-05T21:49:53.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.578+0000 7fdcc91fd700 1 -- 172.21.15.121:0/435826176 wait complete. 2024-04-05T21:49:53.580 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:49:53.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.680+0000 7f09ccd67700 1 -- 172.21.15.121:0/2206554111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c8105a50 msgr2=0x7f09c8105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:53.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.680+0000 7f09ccd67700 1 --2- 172.21.15.121:0/2206554111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c8105a50 0x7f09c8105e30 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009910 tx=0x7f09bc009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:53.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.680+0000 7f09ccd67700 1 -- 172.21.15.121:0/2206554111 shutdown_connections 2024-04-05T21:49:53.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.680+0000 7f09ccd67700 1 --2- 172.21.15.121:0/2206554111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09c80ffa50 0x7f09c80ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.680+0000 7f09ccd67700 1 --2- 172.21.15.121:0/2206554111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c8105a50 0x7f09c8105e30 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.680+0000 7f09ccd67700 1 -- 172.21.15.121:0/2206554111 >> 172.21.15.121:0/2206554111 conn(0x7f09c80fb540 msgr2=0x7f09c80fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:53.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.680+0000 7f09ccd67700 1 -- 172.21.15.121:0/2206554111 shutdown_connections 2024-04-05T21:49:53.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.680+0000 7f09ccd67700 1 -- 172.21.15.121:0/2206554111 wait complete. 2024-04-05T21:49:53.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.681+0000 7f09ccd67700 1 Processor -- start 2024-04-05T21:49:53.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.681+0000 7f09ccd67700 1 -- start start 2024-04-05T21:49:53.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.681+0000 7f09ccd67700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09c80ffa50 0x7f09c80fbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:53.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.681+0000 7f09ccd67700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c8105a50 0x7f09c80fc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:53.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.681+0000 7f09ccd67700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c80fc940 con 0x7f09c8105a50 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.681+0000 7f09ccd67700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c80fcab0 con 0x7f09c80ffa50 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.681+0000 7f09c5d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c8105a50 0x7f09c80fc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.681+0000 7f09c5d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c8105a50 0x7f09c80fc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51240/0 (socket says 172.21.15.121:51240) 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.681+0000 7f09c5d9b700 1 -- 172.21.15.121:0/916711153 learned_addr learned my addr 172.21.15.121:0/916711153 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.682+0000 7f09c659c700 1 --2- 172.21.15.121:0/916711153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09c80ffa50 0x7f09c80fbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.682+0000 7f09c5d9b700 1 -- 172.21.15.121:0/916711153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09c80ffa50 msgr2=0x7f09c80fbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.682+0000 7f09c5d9b700 1 --2- 172.21.15.121:0/916711153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09c80ffa50 0x7f09c80fbe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.682+0000 7f09c5d9b700 1 -- 172.21.15.121:0/916711153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc0095f0 con 0x7f09c8105a50 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.682+0000 7f09c5d9b700 1 --2- 172.21.15.121:0/916711153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c8105a50 0x7f09c80fc370 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f09b800c8a0 tx=0x7f09b800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.682+0000 7f09b77fe700 1 -- 172.21.15.121:0/916711153 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09b800cea0 con 0x7f09c8105a50 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.682+0000 7f09ccd67700 1 -- 172.21.15.121:0/916711153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c81a2810 con 0x7f09c8105a50 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.682+0000 7f09b77fe700 1 -- 172.21.15.121:0/916711153 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09b8004830 con 0x7f09c8105a50 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.682+0000 7f09b77fe700 1 -- 172.21.15.121:0/916711153 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09b8005500 con 0x7f09c8105a50 2024-04-05T21:49:53.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.682+0000 7f09ccd67700 1 -- 172.21.15.121:0/916711153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c81a2c80 con 0x7f09c8105a50 2024-04-05T21:49:53.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.683+0000 7f09ccd67700 1 -- 172.21.15.121:0/916711153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c8109060 con 0x7f09c8105a50 2024-04-05T21:49:53.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.683+0000 7f09b77fe700 1 -- 172.21.15.121:0/916711153 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f09b80049a0 con 0x7f09c8105a50 2024-04-05T21:49:53.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.684+0000 7f09b77fe700 1 --2- 172.21.15.121:0/916711153 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09b006cce0 0x7f09b006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:53.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.684+0000 7f09b77fe700 1 -- 172.21.15.121:0/916711153 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f09b808c2b0 con 0x7f09c8105a50 2024-04-05T21:49:53.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.684+0000 7f09c659c700 1 --2- 172.21.15.121:0/916711153 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09b006cce0 0x7f09b006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:53.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.684+0000 7f09c659c700 1 --2- 172.21.15.121:0/916711153 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09b006cce0 0x7f09b006f1a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00b5c0 tx=0x7f09bc005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:53.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.688+0000 7f09b77fe700 1 -- 172.21.15.121:0/916711153 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f09b8056320 con 0x7f09c8105a50 2024-04-05T21:49:53.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.901+0000 7f09ccd67700 1 -- 172.21.15.121:0/916711153 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f09c802cd30 con 0x7f09b006cce0 2024-04-05T21:49:53.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.902+0000 7f09b77fe700 1 -- 172.21.15.121:0/916711153 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f09c802cd30 con 0x7f09b006cce0 2024-04-05T21:49:53.903 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:49:53.903 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:49:53.903 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:49:53.903 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:49:53.903 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:49:53.903 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:49:53.903 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:49:53.903 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:49:53.903 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:49:53.904 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:49:53.904 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:49:53.904 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:49:53.904 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:49:53.904 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:49:53.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.904+0000 7f09ccd67700 1 -- 172.21.15.121:0/916711153 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09b006cce0 msgr2=0x7f09b006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:53.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.904+0000 7f09ccd67700 1 --2- 172.21.15.121:0/916711153 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09b006cce0 0x7f09b006f1a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00b5c0 tx=0x7f09bc005fd0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.904+0000 7f09ccd67700 1 -- 172.21.15.121:0/916711153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c8105a50 msgr2=0x7f09c80fc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:53.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.904+0000 7f09ccd67700 1 --2- 172.21.15.121:0/916711153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c8105a50 0x7f09c80fc370 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f09b800c8a0 tx=0x7f09b800cc60 comp rx=0 tx=0).stop 2024-04-05T21:49:53.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.905+0000 7f09ccd67700 1 -- 172.21.15.121:0/916711153 shutdown_connections 2024-04-05T21:49:53.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.905+0000 7f09ccd67700 1 --2- 172.21.15.121:0/916711153 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09b006cce0 0x7f09b006f1a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.905+0000 7f09ccd67700 1 --2- 172.21.15.121:0/916711153 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09c80ffa50 0x7f09c80fbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.905+0000 7f09ccd67700 1 --2- 172.21.15.121:0/916711153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c8105a50 0x7f09c80fc370 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:53.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.905+0000 7f09ccd67700 1 -- 172.21.15.121:0/916711153 >> 172.21.15.121:0/916711153 conn(0x7f09c80fb540 msgr2=0x7f09c8068f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:53.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.905+0000 7f09ccd67700 1 -- 172.21.15.121:0/916711153 shutdown_connections 2024-04-05T21:49:53.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:53.905+0000 7f09ccd67700 1 -- 172.21.15.121:0/916711153 wait complete. 2024-04-05T21:49:53.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:53 smithi121 ceph-mon[145859]: from='client.35126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:53.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:53 smithi121 ceph-mon[145859]: pgmap v709: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:53.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:53 smithi121 ceph-mon[145859]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:53.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:53 smithi121 ceph-mon[145859]: from='client.35134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:53.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:53 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1983640034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:49:54.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.006+0000 7f09d81d3700 1 -- 172.21.15.121:0/1014522773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09d0101750 msgr2=0x7f09d01057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:54.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.006+0000 7f09d81d3700 1 --2- 172.21.15.121:0/1014522773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09d0101750 0x7f09d01057a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f09cc009910 tx=0x7f09cc009c20 comp rx=0 tx=0).stop 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.006+0000 7f09d81d3700 1 -- 172.21.15.121:0/1014522773 shutdown_connections 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.006+0000 7f09d81d3700 1 --2- 172.21.15.121:0/1014522773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09d0101750 0x7f09d01057a0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.006+0000 7f09d81d3700 1 --2- 172.21.15.121:0/1014522773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09d0100da0 0x7f09d0101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.006+0000 7f09d81d3700 1 -- 172.21.15.121:0/1014522773 >> 172.21.15.121:0/1014522773 conn(0x7f09d00fc630 msgr2=0x7f09d00fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.006+0000 7f09d81d3700 1 -- 172.21.15.121:0/1014522773 shutdown_connections 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.006+0000 7f09d81d3700 1 -- 172.21.15.121:0/1014522773 wait complete. 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.007+0000 7f09d81d3700 1 Processor -- start 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.007+0000 7f09d81d3700 1 -- start start 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.007+0000 7f09d81d3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09d0100da0 0x7f09d0196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.007+0000 7f09d81d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09d0101750 0x7f09d0196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.007+0000 7f09d81d3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0197270 con 0x7f09d0101750 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.007+0000 7f09d81d3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d019b000 con 0x7f09d0100da0 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.007+0000 7f09d576e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09d0101750 0x7f09d0196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:54.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.007+0000 7f09d576e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09d0101750 0x7f09d0196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51260/0 (socket says 172.21.15.121:51260) 2024-04-05T21:49:54.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.007+0000 7f09d576e700 1 -- 172.21.15.121:0/1073844036 learned_addr learned my addr 172.21.15.121:0/1073844036 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:49:54.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.007+0000 7f09d5f6f700 1 --2- 172.21.15.121:0/1073844036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09d0100da0 0x7f09d0196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:54.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.008+0000 7f09d576e700 1 -- 172.21.15.121:0/1073844036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09d0100da0 msgr2=0x7f09d0196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:54.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.008+0000 7f09d576e700 1 --2- 172.21.15.121:0/1073844036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09d0100da0 0x7f09d0196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:54.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.008+0000 7f09d576e700 1 -- 172.21.15.121:0/1073844036 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc0095f0 con 0x7f09d0101750 2024-04-05T21:49:54.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.008+0000 7f09d5f6f700 1 --2- 172.21.15.121:0/1073844036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09d0100da0 0x7f09d0196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:49:54.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.008+0000 7f09d576e700 1 --2- 172.21.15.121:0/1073844036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09d0101750 0x7f09d0196b90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0092e0 tx=0x7f09cc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:54.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.008+0000 7f09c6ffd700 1 -- 172.21.15.121:0/1073844036 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09cc01d070 con 0x7f09d0101750 2024-04-05T21:49:54.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.008+0000 7f09c6ffd700 1 -- 172.21.15.121:0/1073844036 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09cc005940 con 0x7f09d0101750 2024-04-05T21:49:54.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.008+0000 7f09d81d3700 1 -- 172.21.15.121:0/1073844036 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d019b280 con 0x7f09d0101750 2024-04-05T21:49:54.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.008+0000 7f09c6ffd700 1 -- 172.21.15.121:0/1073844036 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09cc00fe50 con 0x7f09d0101750 2024-04-05T21:49:54.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.008+0000 7f09d81d3700 1 -- 172.21.15.121:0/1073844036 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d019b770 con 0x7f09d0101750 2024-04-05T21:49:54.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.009+0000 7f09d81d3700 1 -- 172.21.15.121:0/1073844036 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d01091a0 con 0x7f09d0101750 2024-04-05T21:49:54.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.009+0000 7f09c6ffd700 1 -- 172.21.15.121:0/1073844036 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f09cc003a60 con 0x7f09d0101750 2024-04-05T21:49:54.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.010+0000 7f09c6ffd700 1 --2- 172.21.15.121:0/1073844036 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09bc06cce0 0x7f09bc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:49:54.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.010+0000 7f09c6ffd700 1 -- 172.21.15.121:0/1073844036 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f09cc08d720 con 0x7f09d0101750 2024-04-05T21:49:54.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.010+0000 7f09d5f6f700 1 --2- 172.21.15.121:0/1073844036 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09bc06cce0 0x7f09bc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:49:54.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.010+0000 7f09d5f6f700 1 --2- 172.21.15.121:0/1073844036 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09bc06cce0 0x7f09bc06f1a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f09c0005950 tx=0x7f09c00058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:49:54.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.014+0000 7f09c6ffd700 1 -- 172.21.15.121:0/1073844036 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f09cc057790 con 0x7f09d0101750 2024-04-05T21:49:54.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:53 smithi188 ceph-mon[107793]: from='client.35126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:54.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:53 smithi188 ceph-mon[107793]: pgmap v709: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:54.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:53 smithi188 ceph-mon[107793]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:54.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:53 smithi188 ceph-mon[107793]: from='client.35134 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:54.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:53 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1983640034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:49:54.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.283+0000 7f09d81d3700 1 -- 172.21.15.121:0/1073844036 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f09d002cc70 con 0x7f09d0101750 2024-04-05T21:49:54.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.284+0000 7f09c6ffd700 1 -- 172.21.15.121:0/1073844036 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f09cc05adb0 con 0x7f09d0101750 2024-04-05T21:49:54.285 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:49:54.285 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:49:54.285 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:49:54.285 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:49:54.285 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:49:54.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.286+0000 7f09d81d3700 1 -- 172.21.15.121:0/1073844036 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09bc06cce0 msgr2=0x7f09bc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:54.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.286+0000 7f09d81d3700 1 --2- 172.21.15.121:0/1073844036 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09bc06cce0 0x7f09bc06f1a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f09c0005950 tx=0x7f09c00058e0 comp rx=0 tx=0).stop 2024-04-05T21:49:54.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.286+0000 7f09d81d3700 1 -- 172.21.15.121:0/1073844036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09d0101750 msgr2=0x7f09d0196b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:49:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.286+0000 7f09d81d3700 1 --2- 172.21.15.121:0/1073844036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09d0101750 0x7f09d0196b90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0092e0 tx=0x7f09cc00be20 comp rx=0 tx=0).stop 2024-04-05T21:49:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.286+0000 7f09d81d3700 1 -- 172.21.15.121:0/1073844036 shutdown_connections 2024-04-05T21:49:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.286+0000 7f09d81d3700 1 --2- 172.21.15.121:0/1073844036 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f09bc06cce0 0x7f09bc06f1a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.286+0000 7f09d81d3700 1 --2- 172.21.15.121:0/1073844036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f09d0100da0 0x7f09d0196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.286+0000 7f09d81d3700 1 --2- 172.21.15.121:0/1073844036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09d0101750 0x7f09d0196b90 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:49:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.286+0000 7f09d81d3700 1 -- 172.21.15.121:0/1073844036 >> 172.21.15.121:0/1073844036 conn(0x7f09d00fc630 msgr2=0x7f09d00fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:49:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.286+0000 7f09d81d3700 1 -- 172.21.15.121:0/1073844036 shutdown_connections 2024-04-05T21:49:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:49:54.286+0000 7f09d81d3700 1 -- 172.21.15.121:0/1073844036 wait complete. 2024-04-05T21:49:54.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:54 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/435826176' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:49:54.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:54 smithi121 ceph-mon[145859]: from='client.35146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:54.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:54 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1073844036' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:49:55.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:54 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/435826176' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:49:55.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:54 smithi188 ceph-mon[107793]: from='client.35146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:49:55.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:54 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1073844036' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:49:55.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:55 smithi121 ceph-mon[145859]: pgmap v710: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:56.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:55 smithi188 ceph-mon[107793]: pgmap v710: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:57.966 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:57 smithi188 ceph-mon[107793]: pgmap v711: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:57.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:57 smithi121 ceph-mon[145859]: pgmap v711: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:49:59.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:49:59 smithi121 ceph-mon[145859]: pgmap v712: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:00.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:49:59 smithi188 ceph-mon[107793]: pgmap v712: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:00.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:49:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:50:00.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:00 smithi121 ceph-mon[145859]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:50:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:50:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:00 smithi188 ceph-mon[107793]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:50:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:50:01.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:01 smithi121 ceph-mon[145859]: pgmap v713: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:02.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:01 smithi188 ceph-mon[107793]: pgmap v713: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:03.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:03 smithi121 ceph-mon[145859]: pgmap v714: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:04.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:03 smithi188 ceph-mon[107793]: pgmap v714: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:05.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:05 smithi121 ceph-mon[145859]: pgmap v715: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:06.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:05 smithi188 ceph-mon[107793]: pgmap v715: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:07.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:07 smithi121 ceph-mon[145859]: pgmap v716: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:08.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:07 smithi188 ceph-mon[107793]: pgmap v716: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:09.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:09 smithi121 ceph-mon[145859]: pgmap v717: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:10.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:09 smithi188 ceph-mon[107793]: pgmap v717: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:11.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:11 smithi121 ceph-mon[145859]: pgmap v718: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:12.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:11 smithi188 ceph-mon[107793]: pgmap v718: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:13.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:13 smithi121 ceph-mon[145859]: pgmap v719: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:14.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:13 smithi188 ceph-mon[107793]: pgmap v719: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:15.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:15 smithi121 ceph-mon[145859]: pgmap v720: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:15.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:50:16.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:15 smithi188 ceph-mon[107793]: pgmap v720: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:16.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:50:17.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:17 smithi121 ceph-mon[145859]: pgmap v721: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:18.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:17 smithi188 ceph-mon[107793]: pgmap v721: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:19.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:19 smithi121 ceph-mon[145859]: pgmap v722: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:20.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:19 smithi188 ceph-mon[107793]: pgmap v722: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:21.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:21 smithi121 ceph-mon[145859]: pgmap v723: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:22.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:21 smithi188 ceph-mon[107793]: pgmap v723: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:23.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:23 smithi121 ceph-mon[145859]: pgmap v724: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:24.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:23 smithi188 ceph-mon[107793]: pgmap v724: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:24.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.400+0000 7ff1de50b700 1 -- 172.21.15.121:0/708075957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1d8101790 msgr2=0x7ff1d81057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:24.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.400+0000 7ff1de50b700 1 --2- 172.21.15.121:0/708075957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1d8101790 0x7ff1d81057e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc009910 tx=0x7ff1cc009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:24.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.400+0000 7ff1de50b700 1 -- 172.21.15.121:0/708075957 shutdown_connections 2024-04-05T21:50:24.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.400+0000 7ff1de50b700 1 --2- 172.21.15.121:0/708075957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1d8101790 0x7ff1d81057e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:24.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.400+0000 7ff1de50b700 1 --2- 172.21.15.121:0/708075957 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1d8100de0 0x7ff1d81011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:24.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.400+0000 7ff1de50b700 1 -- 172.21.15.121:0/708075957 >> 172.21.15.121:0/708075957 conn(0x7ff1d80fc6b0 msgr2=0x7ff1d80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:24.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.400+0000 7ff1de50b700 1 -- 172.21.15.121:0/708075957 shutdown_connections 2024-04-05T21:50:24.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.400+0000 7ff1de50b700 1 -- 172.21.15.121:0/708075957 wait complete. 2024-04-05T21:50:24.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.401+0000 7ff1de50b700 1 Processor -- start 2024-04-05T21:50:24.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.401+0000 7ff1de50b700 1 -- start start 2024-04-05T21:50:24.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.401+0000 7ff1de50b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1d8100de0 0x7ff1d8196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:24.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.401+0000 7ff1de50b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1d8101790 0x7ff1d8196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:24.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.401+0000 7ff1de50b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d8197290 con 0x7ff1d8100de0 2024-04-05T21:50:24.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.401+0000 7ff1de50b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d819b020 con 0x7ff1d8101790 2024-04-05T21:50:24.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.401+0000 7ff1d7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1d8100de0 0x7ff1d8196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:24.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.401+0000 7ff1d7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1d8100de0 0x7ff1d8196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45454/0 (socket says 172.21.15.121:45454) 2024-04-05T21:50:24.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.401+0000 7ff1d7fff700 1 -- 172.21.15.121:0/1728020621 learned_addr learned my addr 172.21.15.121:0/1728020621 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:24.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.402+0000 7ff1d77fe700 1 --2- 172.21.15.121:0/1728020621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1d8101790 0x7ff1d8196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:24.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.402+0000 7ff1d7fff700 1 -- 172.21.15.121:0/1728020621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1d8101790 msgr2=0x7ff1d8196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:24.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.402+0000 7ff1d7fff700 1 --2- 172.21.15.121:0/1728020621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1d8101790 0x7ff1d8196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:24.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.402+0000 7ff1d7fff700 1 -- 172.21.15.121:0/1728020621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc0095f0 con 0x7ff1d8100de0 2024-04-05T21:50:24.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.402+0000 7ff1d77fe700 1 --2- 172.21.15.121:0/1728020621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1d8101790 0x7ff1d8196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:50:24.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.402+0000 7ff1d7fff700 1 --2- 172.21.15.121:0/1728020621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1d8100de0 0x7ff1d8196670 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800c8a0 tx=0x7ff1c800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:24.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.402+0000 7ff1d57fa700 1 -- 172.21.15.121:0/1728020621 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1c800cea0 con 0x7ff1d8100de0 2024-04-05T21:50:24.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.402+0000 7ff1d57fa700 1 -- 172.21.15.121:0/1728020621 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1c8004830 con 0x7ff1d8100de0 2024-04-05T21:50:24.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.402+0000 7ff1de50b700 1 -- 172.21.15.121:0/1728020621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d819b300 con 0x7ff1d8100de0 2024-04-05T21:50:24.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.402+0000 7ff1d57fa700 1 -- 172.21.15.121:0/1728020621 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1c8005500 con 0x7ff1d8100de0 2024-04-05T21:50:24.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.402+0000 7ff1de50b700 1 -- 172.21.15.121:0/1728020621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d819b850 con 0x7ff1d8100de0 2024-04-05T21:50:24.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.403+0000 7ff1de50b700 1 -- 172.21.15.121:0/1728020621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d81091e0 con 0x7ff1d8100de0 2024-04-05T21:50:24.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.403+0000 7ff1d57fa700 1 -- 172.21.15.121:0/1728020621 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff1c80049a0 con 0x7ff1d8100de0 2024-04-05T21:50:24.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.404+0000 7ff1d57fa700 1 --2- 172.21.15.121:0/1728020621 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1c006cce0 0x7ff1c006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:24.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.404+0000 7ff1d77fe700 1 --2- 172.21.15.121:0/1728020621 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1c006cce0 0x7ff1c006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:24.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.404+0000 7ff1d57fa700 1 -- 172.21.15.121:0/1728020621 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff1c808c2a0 con 0x7ff1d8100de0 2024-04-05T21:50:24.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.404+0000 7ff1d77fe700 1 --2- 172.21.15.121:0/1728020621 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1c006cce0 0x7ff1c006f1a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc006010 tx=0x7ff1cc005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:24.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.409+0000 7ff1d57fa700 1 -- 172.21.15.121:0/1728020621 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff1c8056310 con 0x7ff1d8100de0 2024-04-05T21:50:24.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.616+0000 7ff1de50b700 1 -- 172.21.15.121:0/1728020621 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1d819bbb0 con 0x7ff1c006cce0 2024-04-05T21:50:24.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.617+0000 7ff1d57fa700 1 -- 172.21.15.121:0/1728020621 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ff1d819bbb0 con 0x7ff1c006cce0 2024-04-05T21:50:24.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.619+0000 7ff1de50b700 1 -- 172.21.15.121:0/1728020621 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1c006cce0 msgr2=0x7ff1c006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:24.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.619+0000 7ff1de50b700 1 --2- 172.21.15.121:0/1728020621 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1c006cce0 0x7ff1c006f1a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc006010 tx=0x7ff1cc005960 comp rx=0 tx=0).stop 2024-04-05T21:50:24.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.619+0000 7ff1de50b700 1 -- 172.21.15.121:0/1728020621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1d8100de0 msgr2=0x7ff1d8196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:24.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.619+0000 7ff1de50b700 1 --2- 172.21.15.121:0/1728020621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1d8100de0 0x7ff1d8196670 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800c8a0 tx=0x7ff1c800cc60 comp rx=0 tx=0).stop 2024-04-05T21:50:24.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.619+0000 7ff1de50b700 1 -- 172.21.15.121:0/1728020621 shutdown_connections 2024-04-05T21:50:24.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.619+0000 7ff1de50b700 1 --2- 172.21.15.121:0/1728020621 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff1c006cce0 0x7ff1c006f1a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:24.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.619+0000 7ff1de50b700 1 --2- 172.21.15.121:0/1728020621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1d8100de0 0x7ff1d8196670 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:24.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.619+0000 7ff1de50b700 1 --2- 172.21.15.121:0/1728020621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff1d8101790 0x7ff1d8196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:24.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.619+0000 7ff1de50b700 1 -- 172.21.15.121:0/1728020621 >> 172.21.15.121:0/1728020621 conn(0x7ff1d80fc6b0 msgr2=0x7ff1d80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:24.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.620+0000 7ff1de50b700 1 -- 172.21.15.121:0/1728020621 shutdown_connections 2024-04-05T21:50:24.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.620+0000 7ff1de50b700 1 -- 172.21.15.121:0/1728020621 wait complete. 2024-04-05T21:50:24.660 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:50:24.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.724+0000 7f687d331700 1 -- 172.21.15.121:0/2792857956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6878100dd0 msgr2=0x7f68781011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:24.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.724+0000 7f687d331700 1 --2- 172.21.15.121:0/2792857956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6878100dd0 0x7f68781011b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f686c009910 tx=0x7f686c009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:24.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.724+0000 7f687d331700 1 -- 172.21.15.121:0/2792857956 shutdown_connections 2024-04-05T21:50:24.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.724+0000 7f687d331700 1 --2- 172.21.15.121:0/2792857956 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6878101780 0x7f68781057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:24.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.724+0000 7f687d331700 1 --2- 172.21.15.121:0/2792857956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6878100dd0 0x7f68781011b0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:24.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.724+0000 7f687d331700 1 -- 172.21.15.121:0/2792857956 >> 172.21.15.121:0/2792857956 conn(0x7f68780fc640 msgr2=0x7f68780fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:24.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.724+0000 7f687d331700 1 -- 172.21.15.121:0/2792857956 shutdown_connections 2024-04-05T21:50:24.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.724+0000 7f687d331700 1 -- 172.21.15.121:0/2792857956 wait complete. 2024-04-05T21:50:24.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.725+0000 7f687d331700 1 Processor -- start 2024-04-05T21:50:24.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.725+0000 7f687d331700 1 -- start start 2024-04-05T21:50:24.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.725+0000 7f687d331700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6878100dd0 0x7f687819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:24.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.725+0000 7f687d331700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6878101780 0x7f687819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:24.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.725+0000 7f687d331700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687819d030 con 0x7f6878100dd0 2024-04-05T21:50:24.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.725+0000 7f687d331700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6878196570 con 0x7f6878101780 2024-04-05T21:50:24.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.725+0000 7f6876ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6878100dd0 0x7f687819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:24.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.725+0000 7f6876ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6878100dd0 0x7f687819c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45472/0 (socket says 172.21.15.121:45472) 2024-04-05T21:50:24.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.725+0000 7f6876ffd700 1 -- 172.21.15.121:0/905220123 learned_addr learned my addr 172.21.15.121:0/905220123 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:24.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.725+0000 7f6876ffd700 1 -- 172.21.15.121:0/905220123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6878101780 msgr2=0x7f687819ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:24.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.725+0000 7f6876ffd700 1 --2- 172.21.15.121:0/905220123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6878101780 0x7f687819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:24.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.725+0000 7f6876ffd700 1 -- 172.21.15.121:0/905220123 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c0095f0 con 0x7f6878100dd0 2024-04-05T21:50:24.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.726+0000 7f6876ffd700 1 --2- 172.21.15.121:0/905220123 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6878100dd0 0x7f687819c4f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f686c006010 tx=0x7f686c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:24.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.726+0000 7f685ffff700 1 -- 172.21.15.121:0/905220123 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f686c01d070 con 0x7f6878100dd0 2024-04-05T21:50:24.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.726+0000 7f685ffff700 1 -- 172.21.15.121:0/905220123 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f686c00f460 con 0x7f6878100dd0 2024-04-05T21:50:24.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.726+0000 7f685ffff700 1 -- 172.21.15.121:0/905220123 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f686c021640 con 0x7f6878100dd0 2024-04-05T21:50:24.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.726+0000 7f687d331700 1 -- 172.21.15.121:0/905220123 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68781967f0 con 0x7f6878100dd0 2024-04-05T21:50:24.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.726+0000 7f687d331700 1 -- 172.21.15.121:0/905220123 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6878196ce0 con 0x7f6878100dd0 2024-04-05T21:50:24.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.727+0000 7f687d331700 1 -- 172.21.15.121:0/905220123 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6878109170 con 0x7f6878100dd0 2024-04-05T21:50:24.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.727+0000 7f685ffff700 1 -- 172.21.15.121:0/905220123 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f686c00fa20 con 0x7f6878100dd0 2024-04-05T21:50:24.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.728+0000 7f685ffff700 1 --2- 172.21.15.121:0/905220123 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f686006cce0 0x7f686006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:24.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.728+0000 7f685ffff700 1 -- 172.21.15.121:0/905220123 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f686c08d7b0 con 0x7f6878100dd0 2024-04-05T21:50:24.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.728+0000 7f68767fc700 1 --2- 172.21.15.121:0/905220123 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f686006cce0 0x7f686006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:24.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.728+0000 7f68767fc700 1 --2- 172.21.15.121:0/905220123 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f686006cce0 0x7f686006f1a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6868005fd0 tx=0x7f6868005d00 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:24.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.732+0000 7f685ffff700 1 -- 172.21.15.121:0/905220123 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f686c05b330 con 0x7f6878100dd0 2024-04-05T21:50:24.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.951+0000 7f687d331700 1 -- 172.21.15.121:0/905220123 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6878196f50 con 0x7f686006cce0 2024-04-05T21:50:24.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.952+0000 7f685ffff700 1 -- 172.21.15.121:0/905220123 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6878196f50 con 0x7f686006cce0 2024-04-05T21:50:24.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.954+0000 7f687d331700 1 -- 172.21.15.121:0/905220123 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f686006cce0 msgr2=0x7f686006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:24.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.954+0000 7f687d331700 1 --2- 172.21.15.121:0/905220123 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f686006cce0 0x7f686006f1a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6868005fd0 tx=0x7f6868005d00 comp rx=0 tx=0).stop 2024-04-05T21:50:24.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.954+0000 7f687d331700 1 -- 172.21.15.121:0/905220123 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6878100dd0 msgr2=0x7f687819c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:24.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.954+0000 7f687d331700 1 --2- 172.21.15.121:0/905220123 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6878100dd0 0x7f687819c4f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f686c006010 tx=0x7f686c00be20 comp rx=0 tx=0).stop 2024-04-05T21:50:24.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.954+0000 7f687d331700 1 -- 172.21.15.121:0/905220123 shutdown_connections 2024-04-05T21:50:24.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.954+0000 7f687d331700 1 --2- 172.21.15.121:0/905220123 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f686006cce0 0x7f686006f1a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:24.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.954+0000 7f687d331700 1 --2- 172.21.15.121:0/905220123 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6878100dd0 0x7f687819c4f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:24.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.954+0000 7f687d331700 1 --2- 172.21.15.121:0/905220123 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6878101780 0x7f687819ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:24.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.954+0000 7f687d331700 1 -- 172.21.15.121:0/905220123 >> 172.21.15.121:0/905220123 conn(0x7f68780fc640 msgr2=0x7f68780fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:24.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.954+0000 7f687d331700 1 -- 172.21.15.121:0/905220123 shutdown_connections 2024-04-05T21:50:24.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:24.954+0000 7f687d331700 1 -- 172.21.15.121:0/905220123 wait complete. 2024-04-05T21:50:25.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.056+0000 7fe553e7a700 1 -- 172.21.15.121:0/2413333649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe54c0ffa40 msgr2=0x7fe54c0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:25.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.056+0000 7fe553e7a700 1 --2- 172.21.15.121:0/2413333649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe54c0ffa40 0x7fe54c0ffec0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009910 tx=0x7fe53c009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:25.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.056+0000 7fe553e7a700 1 -- 172.21.15.121:0/2413333649 shutdown_connections 2024-04-05T21:50:25.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.056+0000 7fe553e7a700 1 --2- 172.21.15.121:0/2413333649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe54c0ffa40 0x7fe54c0ffec0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.056+0000 7fe553e7a700 1 --2- 172.21.15.121:0/2413333649 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe54c105a40 0x7fe54c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.056+0000 7fe553e7a700 1 -- 172.21.15.121:0/2413333649 >> 172.21.15.121:0/2413333649 conn(0x7fe54c0fb570 msgr2=0x7fe54c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:25.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.057+0000 7fe553e7a700 1 -- 172.21.15.121:0/2413333649 shutdown_connections 2024-04-05T21:50:25.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.057+0000 7fe553e7a700 1 -- 172.21.15.121:0/2413333649 wait complete. 2024-04-05T21:50:25.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.057+0000 7fe553e7a700 1 Processor -- start 2024-04-05T21:50:25.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.057+0000 7fe553e7a700 1 -- start start 2024-04-05T21:50:25.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.057+0000 7fe553e7a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe54c0ffa40 0x7fe54c195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.057+0000 7fe553e7a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe54c105a40 0x7fe54c195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.057+0000 7fe553e7a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c1960d0 con 0x7fe54c0ffa40 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.057+0000 7fe553e7a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c196210 con 0x7fe54c105a40 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.057+0000 7fe551c16700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe54c0ffa40 0x7fe54c195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.057+0000 7fe551c16700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe54c0ffa40 0x7fe54c195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45484/0 (socket says 172.21.15.121:45484) 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.057+0000 7fe551c16700 1 -- 172.21.15.121:0/2325038445 learned_addr learned my addr 172.21.15.121:0/2325038445 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.058+0000 7fe551415700 1 --2- 172.21.15.121:0/2325038445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe54c105a40 0x7fe54c195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.058+0000 7fe551c16700 1 -- 172.21.15.121:0/2325038445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe54c105a40 msgr2=0x7fe54c195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.058+0000 7fe551c16700 1 --2- 172.21.15.121:0/2325038445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe54c105a40 0x7fe54c195a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.058+0000 7fe551c16700 1 -- 172.21.15.121:0/2325038445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c0095f0 con 0x7fe54c0ffa40 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.058+0000 7fe551415700 1 --2- 172.21.15.121:0/2325038445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe54c105a40 0x7fe54c195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.058+0000 7fe551c16700 1 --2- 172.21.15.121:0/2325038445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe54c0ffa40 0x7fe54c195540 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe54800c930 tx=0x7fe54800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.058+0000 7fe542ffd700 1 -- 172.21.15.121:0/2325038445 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe548007ad0 con 0x7fe54c0ffa40 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.058+0000 7fe542ffd700 1 -- 172.21.15.121:0/2325038445 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe548007c30 con 0x7fe54c0ffa40 2024-04-05T21:50:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.058+0000 7fe553e7a700 1 -- 172.21.15.121:0/2325038445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c19a060 con 0x7fe54c0ffa40 2024-04-05T21:50:25.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.058+0000 7fe542ffd700 1 -- 172.21.15.121:0/2325038445 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe548018630 con 0x7fe54c0ffa40 2024-04-05T21:50:25.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.058+0000 7fe553e7a700 1 -- 172.21.15.121:0/2325038445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c19a5b0 con 0x7fe54c0ffa40 2024-04-05T21:50:25.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.059+0000 7fe553e7a700 1 -- 172.21.15.121:0/2325038445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c107fd0 con 0x7fe54c0ffa40 2024-04-05T21:50:25.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.059+0000 7fe542ffd700 1 -- 172.21.15.121:0/2325038445 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe54800f450 con 0x7fe54c0ffa40 2024-04-05T21:50:25.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.060+0000 7fe542ffd700 1 --2- 172.21.15.121:0/2325038445 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe53806cce0 0x7fe53806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:25.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.060+0000 7fe542ffd700 1 -- 172.21.15.121:0/2325038445 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe54808c780 con 0x7fe54c0ffa40 2024-04-05T21:50:25.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.060+0000 7fe551415700 1 --2- 172.21.15.121:0/2325038445 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe53806cce0 0x7fe53806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:25.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.060+0000 7fe551415700 1 --2- 172.21.15.121:0/2325038445 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe53806cce0 0x7fe53806f1a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00b5c0 tx=0x7fe53c005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:25.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.064+0000 7fe542ffd700 1 -- 172.21.15.121:0/2325038445 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe548056920 con 0x7fe54c0ffa40 2024-04-05T21:50:25.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.269+0000 7fe553e7a700 1 -- 172.21.15.121:0/2325038445 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe54c02cd50 con 0x7fe53806cce0 2024-04-05T21:50:25.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.280+0000 7fe542ffd700 1 -- 172.21.15.121:0/2325038445 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fe54c02cd50 con 0x7fe53806cce0 2024-04-05T21:50:25.281 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:50:25.281 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (21m) 15m ago 32m 26.5M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:50:25.282 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (32m) 15m ago 32m 8472k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:50:25.282 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (31m) 14m ago 31m 8409k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:50:25.282 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (19m) 15m ago 32m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:50:25.282 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (19m) 14m ago 31m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:50:25.282 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (20m) 15m ago 32m 88.2M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:50:25.282 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (15m) 15m ago 26m 14.7M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:50:25.282 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (15m) 15m ago 26m 12.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:50:25.282 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (14m) 14m ago 26m 13.4M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:50:25.282 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (26m) 14m ago 26m 12.0M - 18.2.1 5be31c24972a 559fc8c0b7b5 2024-04-05T21:50:25.282 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (23m) 15m ago 34m 511M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:50:25.282 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (22m) 14m ago 31m 453M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:50:25.282 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (19m) 15m ago 34m 50.6M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:50:25.282 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (19m) 14m ago 31m 41.0M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:50:25.283 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (22m) 15m ago 32m 17.9M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:50:25.283 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (22m) 14m ago 31m 17.0M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:50:25.283 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (18m) 15m ago 30m 93.5M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:50:25.283 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (18m) 15m ago 30m 102M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:50:25.283 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (17m) 15m ago 29m 88.0M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:50:25.283 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (17m) 14m ago 29m 116M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:50:25.283 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (16m) 14m ago 28m 95.8M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:50:25.283 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (15m) 14m ago 28m 86.0M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:50:25.283 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (21m) 15m ago 32m 67.1M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:50:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.282+0000 7fe553e7a700 1 -- 172.21.15.121:0/2325038445 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe53806cce0 msgr2=0x7fe53806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.282+0000 7fe553e7a700 1 --2- 172.21.15.121:0/2325038445 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe53806cce0 0x7fe53806f1a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00b5c0 tx=0x7fe53c005960 comp rx=0 tx=0).stop 2024-04-05T21:50:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.282+0000 7fe553e7a700 1 -- 172.21.15.121:0/2325038445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe54c0ffa40 msgr2=0x7fe54c195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.282+0000 7fe553e7a700 1 --2- 172.21.15.121:0/2325038445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe54c0ffa40 0x7fe54c195540 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe54800c930 tx=0x7fe54800ccf0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.282+0000 7fe553e7a700 1 -- 172.21.15.121:0/2325038445 shutdown_connections 2024-04-05T21:50:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.282+0000 7fe553e7a700 1 --2- 172.21.15.121:0/2325038445 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe53806cce0 0x7fe53806f1a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.282+0000 7fe553e7a700 1 --2- 172.21.15.121:0/2325038445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe54c0ffa40 0x7fe54c195540 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.282+0000 7fe553e7a700 1 --2- 172.21.15.121:0/2325038445 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe54c105a40 0x7fe54c195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.282+0000 7fe553e7a700 1 -- 172.21.15.121:0/2325038445 >> 172.21.15.121:0/2325038445 conn(0x7fe54c0fb570 msgr2=0x7fe54c0fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.282+0000 7fe553e7a700 1 -- 172.21.15.121:0/2325038445 shutdown_connections 2024-04-05T21:50:25.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.282+0000 7fe553e7a700 1 -- 172.21.15.121:0/2325038445 wait complete. 2024-04-05T21:50:25.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.384+0000 7fba72847700 1 -- 172.21.15.121:0/2497852773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba6c104a50 msgr2=0x7fba6c104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:25.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.384+0000 7fba72847700 1 --2- 172.21.15.121:0/2497852773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba6c104a50 0x7fba6c104e30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009910 tx=0x7fba5c009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:25.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.384+0000 7fba72847700 1 -- 172.21.15.121:0/2497852773 shutdown_connections 2024-04-05T21:50:25.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.384+0000 7fba72847700 1 --2- 172.21.15.121:0/2497852773 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba6c100140 0x7fba6c1005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.384+0000 7fba72847700 1 --2- 172.21.15.121:0/2497852773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba6c104a50 0x7fba6c104e30 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.384+0000 7fba72847700 1 -- 172.21.15.121:0/2497852773 >> 172.21.15.121:0/2497852773 conn(0x7fba6c075e30 msgr2=0x7fba6c0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:25.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.384+0000 7fba72847700 1 -- 172.21.15.121:0/2497852773 shutdown_connections 2024-04-05T21:50:25.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.384+0000 7fba72847700 1 -- 172.21.15.121:0/2497852773 wait complete. 2024-04-05T21:50:25.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.384+0000 7fba72847700 1 Processor -- start 2024-04-05T21:50:25.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.384+0000 7fba72847700 1 -- start start 2024-04-05T21:50:25.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba72847700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba6c100140 0x7fba6c1954c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:25.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba72847700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba6c104a50 0x7fba6c195a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:25.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba72847700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c1960e0 con 0x7fba6c104a50 2024-04-05T21:50:25.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba72847700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c199e70 con 0x7fba6c100140 2024-04-05T21:50:25.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba6b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba6c104a50 0x7fba6c195a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:25.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba6b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba6c104a50 0x7fba6c195a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45498/0 (socket says 172.21.15.121:45498) 2024-04-05T21:50:25.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba6b7fe700 1 -- 172.21.15.121:0/2160076195 learned_addr learned my addr 172.21.15.121:0/2160076195 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:25.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba6bfff700 1 --2- 172.21.15.121:0/2160076195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba6c100140 0x7fba6c1954c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:25.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba6b7fe700 1 -- 172.21.15.121:0/2160076195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba6c100140 msgr2=0x7fba6c1954c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:25.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba6b7fe700 1 --2- 172.21.15.121:0/2160076195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba6c100140 0x7fba6c1954c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba6b7fe700 1 -- 172.21.15.121:0/2160076195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c0095f0 con 0x7fba6c104a50 2024-04-05T21:50:25.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba6b7fe700 1 --2- 172.21.15.121:0/2160076195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba6c104a50 0x7fba6c195a00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fba6000c8a0 tx=0x7fba6000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:25.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba697fa700 1 -- 172.21.15.121:0/2160076195 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba600078e0 con 0x7fba6c104a50 2024-04-05T21:50:25.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba72847700 1 -- 172.21.15.121:0/2160076195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6c19a150 con 0x7fba6c104a50 2024-04-05T21:50:25.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba697fa700 1 -- 172.21.15.121:0/2160076195 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba6000f450 con 0x7fba6c104a50 2024-04-05T21:50:25.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba697fa700 1 -- 172.21.15.121:0/2160076195 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba6000e5e0 con 0x7fba6c104a50 2024-04-05T21:50:25.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.385+0000 7fba72847700 1 -- 172.21.15.121:0/2160076195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6c19a720 con 0x7fba6c104a50 2024-04-05T21:50:25.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.386+0000 7fba72847700 1 -- 172.21.15.121:0/2160076195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6c108020 con 0x7fba6c104a50 2024-04-05T21:50:25.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.387+0000 7fba697fa700 1 -- 172.21.15.121:0/2160076195 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fba6000f5c0 con 0x7fba6c104a50 2024-04-05T21:50:25.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.387+0000 7fba697fa700 1 --2- 172.21.15.121:0/2160076195 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fba5406cce0 0x7fba5406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:25.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.387+0000 7fba697fa700 1 -- 172.21.15.121:0/2160076195 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fba6008c1d0 con 0x7fba6c104a50 2024-04-05T21:50:25.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.387+0000 7fba6bfff700 1 --2- 172.21.15.121:0/2160076195 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fba5406cce0 0x7fba5406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:25.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.388+0000 7fba6bfff700 1 --2- 172.21.15.121:0/2160076195 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fba5406cce0 0x7fba5406f1a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fba5c006010 tx=0x7fba5c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:25.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.392+0000 7fba697fa700 1 -- 172.21.15.121:0/2160076195 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fba60056240 con 0x7fba6c104a50 2024-04-05T21:50:25.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.669+0000 7fba72847700 1 -- 172.21.15.121:0/2160076195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fba6c04ea90 con 0x7fba6c104a50 2024-04-05T21:50:25.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.669+0000 7fba697fa700 1 -- 172.21.15.121:0/2160076195 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+815 (secure 0 0 0) 0x7fba60059860 con 0x7fba6c104a50 2024-04-05T21:50:25.671 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:50:25.671 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:50:25.671 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:50:25.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:50:25.671 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:50:25.671 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:50:25.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:50:25.671 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:50:25.671 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:50:25.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:50:25.671 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:50:25.671 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:50:25.672 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 3 2024-04-05T21:50:25.672 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:50:25.672 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:50:25.672 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)": 1, 2024-04-05T21:50:25.672 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 13 2024-04-05T21:50:25.672 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:50:25.672 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:50:25.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.671+0000 7fba72847700 1 -- 172.21.15.121:0/2160076195 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fba5406cce0 msgr2=0x7fba5406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:25.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.671+0000 7fba72847700 1 --2- 172.21.15.121:0/2160076195 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fba5406cce0 0x7fba5406f1a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fba5c006010 tx=0x7fba5c01a040 comp rx=0 tx=0).stop 2024-04-05T21:50:25.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.671+0000 7fba72847700 1 -- 172.21.15.121:0/2160076195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba6c104a50 msgr2=0x7fba6c195a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:25.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.671+0000 7fba72847700 1 --2- 172.21.15.121:0/2160076195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba6c104a50 0x7fba6c195a00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fba6000c8a0 tx=0x7fba6000cbb0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.671+0000 7fba72847700 1 -- 172.21.15.121:0/2160076195 shutdown_connections 2024-04-05T21:50:25.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.671+0000 7fba72847700 1 --2- 172.21.15.121:0/2160076195 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fba5406cce0 0x7fba5406f1a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.671+0000 7fba72847700 1 --2- 172.21.15.121:0/2160076195 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fba6c100140 0x7fba6c1954c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.671+0000 7fba72847700 1 --2- 172.21.15.121:0/2160076195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba6c104a50 0x7fba6c195a00 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.671+0000 7fba72847700 1 -- 172.21.15.121:0/2160076195 >> 172.21.15.121:0/2160076195 conn(0x7fba6c075e30 msgr2=0x7fba6c103400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:25.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.671+0000 7fba72847700 1 -- 172.21.15.121:0/2160076195 shutdown_connections 2024-04-05T21:50:25.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.671+0000 7fba72847700 1 -- 172.21.15.121:0/2160076195 wait complete. 2024-04-05T21:50:25.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.773+0000 7f5a7d065700 1 -- 172.21.15.121:0/2202088588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a780ffbd0 msgr2=0x7f5a78100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:25.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.773+0000 7f5a7d065700 1 --2- 172.21.15.121:0/2202088588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a780ffbd0 0x7f5a78100050 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009910 tx=0x7f5a68009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:25.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.773+0000 7f5a7d065700 1 -- 172.21.15.121:0/2202088588 shutdown_connections 2024-04-05T21:50:25.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.773+0000 7f5a7d065700 1 --2- 172.21.15.121:0/2202088588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a780ffbd0 0x7f5a78100050 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.773+0000 7f5a7d065700 1 --2- 172.21.15.121:0/2202088588 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a78105a40 0x7f5a78105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.773+0000 7f5a7d065700 1 -- 172.21.15.121:0/2202088588 >> 172.21.15.121:0/2202088588 conn(0x7f5a780755a0 msgr2=0x7f5a780759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:25.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.773+0000 7f5a7d065700 1 -- 172.21.15.121:0/2202088588 shutdown_connections 2024-04-05T21:50:25.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.773+0000 7f5a7d065700 1 -- 172.21.15.121:0/2202088588 wait complete. 2024-04-05T21:50:25.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.774+0000 7f5a7d065700 1 Processor -- start 2024-04-05T21:50:25.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.774+0000 7f5a7d065700 1 -- start start 2024-04-05T21:50:25.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.774+0000 7f5a7d065700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a780ffbd0 0x7f5a78195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:25.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.774+0000 7f5a7d065700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a78105a40 0x7f5a78195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:25.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.774+0000 7f5a7d065700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a78196160 con 0x7f5a780ffbd0 2024-04-05T21:50:25.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.774+0000 7f5a7d065700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a78199ef0 con 0x7f5a78105a40 2024-04-05T21:50:25.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.775+0000 7f5a76d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a780ffbd0 0x7f5a78195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:25.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.775+0000 7f5a76d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a780ffbd0 0x7f5a78195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45506/0 (socket says 172.21.15.121:45506) 2024-04-05T21:50:25.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.775+0000 7f5a76d9d700 1 -- 172.21.15.121:0/1034426896 learned_addr learned my addr 172.21.15.121:0/1034426896 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:25.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.775+0000 7f5a7659c700 1 --2- 172.21.15.121:0/1034426896 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a78105a40 0x7f5a78195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:25.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.775+0000 7f5a76d9d700 1 -- 172.21.15.121:0/1034426896 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a78105a40 msgr2=0x7f5a78195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:25.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.775+0000 7f5a76d9d700 1 --2- 172.21.15.121:0/1034426896 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a78105a40 0x7f5a78195a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:25.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.775+0000 7f5a76d9d700 1 -- 172.21.15.121:0/1034426896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a680095f0 con 0x7f5a780ffbd0 2024-04-05T21:50:25.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.775+0000 7f5a76d9d700 1 --2- 172.21.15.121:0/1034426896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a780ffbd0 0x7f5a78195540 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009970 tx=0x7f5a6c009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:25.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.775+0000 7f5a5ffff700 1 -- 172.21.15.121:0/1034426896 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a6c012070 con 0x7f5a780ffbd0 2024-04-05T21:50:25.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.775+0000 7f5a7d065700 1 -- 172.21.15.121:0/1034426896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7819a170 con 0x7f5a780ffbd0 2024-04-05T21:50:25.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.775+0000 7f5a5ffff700 1 -- 172.21.15.121:0/1034426896 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a6c007ad0 con 0x7f5a780ffbd0 2024-04-05T21:50:25.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.775+0000 7f5a5ffff700 1 -- 172.21.15.121:0/1034426896 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a6c017c00 con 0x7f5a780ffbd0 2024-04-05T21:50:25.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.775+0000 7f5a7d065700 1 -- 172.21.15.121:0/1034426896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7819a6c0 con 0x7f5a780ffbd0 2024-04-05T21:50:25.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.776+0000 7f5a7d065700 1 -- 172.21.15.121:0/1034426896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7804ea90 con 0x7f5a780ffbd0 2024-04-05T21:50:25.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.777+0000 7f5a5ffff700 1 -- 172.21.15.121:0/1034426896 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5a6c007c40 con 0x7f5a780ffbd0 2024-04-05T21:50:25.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.777+0000 7f5a5ffff700 1 --2- 172.21.15.121:0/1034426896 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5a6006cce0 0x7f5a6006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:25.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.777+0000 7f5a5ffff700 1 -- 172.21.15.121:0/1034426896 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5a6c08dc60 con 0x7f5a780ffbd0 2024-04-05T21:50:25.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.777+0000 7f5a7659c700 1 --2- 172.21.15.121:0/1034426896 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5a6006cce0 0x7f5a6006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:25.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.778+0000 7f5a7659c700 1 --2- 172.21.15.121:0/1034426896 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5a6006cce0 0x7f5a6006f1a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5a78196b60 tx=0x7f5a6801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:25.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:25.782+0000 7f5a5ffff700 1 -- 172.21.15.121:0/1034426896 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5a6c057ce0 con 0x7f5a780ffbd0 2024-04-05T21:50:25.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:25 smithi121 ceph-mon[145859]: pgmap v725: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:25.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:25 smithi121 ceph-mon[145859]: from='client.35154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:26.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.020+0000 7f5a7d065700 1 -- 172.21.15.121:0/1034426896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5a7819abf0 con 0x7f5a780ffbd0 2024-04-05T21:50:26.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.021+0000 7f5a5ffff700 1 -- 172.21.15.121:0/1034426896 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f5a6c05b300 con 0x7f5a780ffbd0 2024-04-05T21:50:26.022 INFO:teuthology.orchestra.run.smithi121.stdout:e30 2024-04-05T21:50:26.022 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:50:26.022 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:50:26.023 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:50:26.024 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:50:26.025 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:50:26.025 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:50:26.025 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:50:26.025 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:50:26.025 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:50:26.025 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:50:26.025 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:50:26.025 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44245} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/963169033,v1:172.21.15.188:6827/963169033] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:50:26.025 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:50:26.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.023+0000 7f5a7d065700 1 -- 172.21.15.121:0/1034426896 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5a6006cce0 msgr2=0x7f5a6006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:26.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.023+0000 7f5a7d065700 1 --2- 172.21.15.121:0/1034426896 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5a6006cce0 0x7f5a6006f1a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5a78196b60 tx=0x7f5a6801a040 comp rx=0 tx=0).stop 2024-04-05T21:50:26.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.023+0000 7f5a7d065700 1 -- 172.21.15.121:0/1034426896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a780ffbd0 msgr2=0x7f5a78195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:26.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.023+0000 7f5a7d065700 1 --2- 172.21.15.121:0/1034426896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a780ffbd0 0x7f5a78195540 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009970 tx=0x7f5a6c009d30 comp rx=0 tx=0).stop 2024-04-05T21:50:26.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.023+0000 7f5a7d065700 1 -- 172.21.15.121:0/1034426896 shutdown_connections 2024-04-05T21:50:26.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.023+0000 7f5a7d065700 1 --2- 172.21.15.121:0/1034426896 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5a6006cce0 0x7f5a6006f1a0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.023+0000 7f5a7d065700 1 --2- 172.21.15.121:0/1034426896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a780ffbd0 0x7f5a78195540 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.023+0000 7f5a7d065700 1 --2- 172.21.15.121:0/1034426896 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5a78105a40 0x7f5a78195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.023+0000 7f5a7d065700 1 -- 172.21.15.121:0/1034426896 >> 172.21.15.121:0/1034426896 conn(0x7f5a780755a0 msgr2=0x7f5a780fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:26.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.023+0000 7f5a7d065700 1 -- 172.21.15.121:0/1034426896 shutdown_connections 2024-04-05T21:50:26.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.023+0000 7f5a7d065700 1 -- 172.21.15.121:0/1034426896 wait complete. 2024-04-05T21:50:26.027 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T21:50:26.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:25 smithi188 ceph-mon[107793]: pgmap v725: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:26.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:25 smithi188 ceph-mon[107793]: from='client.35154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:26.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.126+0000 7f40df088700 1 -- 172.21.15.121:0/1331191352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40d8105a30 msgr2=0x7f40d8105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:26.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.126+0000 7f40df088700 1 --2- 172.21.15.121:0/1331191352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40d8105a30 0x7f40d8105e10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009910 tx=0x7f40cc009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:26.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.126+0000 7f40df088700 1 -- 172.21.15.121:0/1331191352 shutdown_connections 2024-04-05T21:50:26.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.126+0000 7f40df088700 1 --2- 172.21.15.121:0/1331191352 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40d80ffb70 0x7f40d80fffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.126+0000 7f40df088700 1 --2- 172.21.15.121:0/1331191352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40d8105a30 0x7f40d8105e10 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.126+0000 7f40df088700 1 -- 172.21.15.121:0/1331191352 >> 172.21.15.121:0/1331191352 conn(0x7f40d80754f0 msgr2=0x7f40d8075900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:26.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.126+0000 7f40df088700 1 -- 172.21.15.121:0/1331191352 shutdown_connections 2024-04-05T21:50:26.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.126+0000 7f40df088700 1 -- 172.21.15.121:0/1331191352 wait complete. 2024-04-05T21:50:26.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.126+0000 7f40df088700 1 Processor -- start 2024-04-05T21:50:26.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.126+0000 7f40df088700 1 -- start start 2024-04-05T21:50:26.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40df088700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40d80ffb70 0x7f40d81976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:26.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40df088700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40d8105a30 0x7f40d8197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:26.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40df088700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d81982e0 con 0x7f40d80ffb70 2024-04-05T21:50:26.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40df088700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d8198450 con 0x7f40d8105a30 2024-04-05T21:50:26.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40dce24700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40d80ffb70 0x7f40d81976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:26.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40dce24700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40d80ffb70 0x7f40d81976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45534/0 (socket says 172.21.15.121:45534) 2024-04-05T21:50:26.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40dce24700 1 -- 172.21.15.121:0/1927516806 learned_addr learned my addr 172.21.15.121:0/1927516806 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:26.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40d7fff700 1 --2- 172.21.15.121:0/1927516806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40d8105a30 0x7f40d8197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:26.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40dce24700 1 -- 172.21.15.121:0/1927516806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40d8105a30 msgr2=0x7f40d8197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:26.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40dce24700 1 --2- 172.21.15.121:0/1927516806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40d8105a30 0x7f40d8197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40dce24700 1 -- 172.21.15.121:0/1927516806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40cc0095f0 con 0x7f40d80ffb70 2024-04-05T21:50:26.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40d7fff700 1 --2- 172.21.15.121:0/1927516806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40d8105a30 0x7f40d8197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:50:26.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40dce24700 1 --2- 172.21.15.121:0/1927516806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40d80ffb70 0x7f40d81976c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f40cc006010 tx=0x7f40cc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:26.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40d5ffb700 1 -- 172.21.15.121:0/1927516806 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40cc01d070 con 0x7f40d80ffb70 2024-04-05T21:50:26.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40df088700 1 -- 172.21.15.121:0/1927516806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d81a0730 con 0x7f40d80ffb70 2024-04-05T21:50:26.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40d5ffb700 1 -- 172.21.15.121:0/1927516806 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40cc00f460 con 0x7f40d80ffb70 2024-04-05T21:50:26.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.127+0000 7f40d5ffb700 1 -- 172.21.15.121:0/1927516806 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40cc021640 con 0x7f40d80ffb70 2024-04-05T21:50:26.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.128+0000 7f40df088700 1 -- 172.21.15.121:0/1927516806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d81a0bd0 con 0x7f40d80ffb70 2024-04-05T21:50:26.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.129+0000 7f40df088700 1 -- 172.21.15.121:0/1927516806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d8107fe0 con 0x7f40d80ffb70 2024-04-05T21:50:26.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.129+0000 7f40d5ffb700 1 -- 172.21.15.121:0/1927516806 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f40cc00fa60 con 0x7f40d80ffb70 2024-04-05T21:50:26.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.130+0000 7f40d5ffb700 1 --2- 172.21.15.121:0/1927516806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f40c806cce0 0x7f40c806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:26.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.130+0000 7f40d7fff700 1 --2- 172.21.15.121:0/1927516806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f40c806cce0 0x7f40c806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:26.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.130+0000 7f40d5ffb700 1 -- 172.21.15.121:0/1927516806 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f40cc08d8c0 con 0x7f40d80ffb70 2024-04-05T21:50:26.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.130+0000 7f40d7fff700 1 --2- 172.21.15.121:0/1927516806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f40c806cce0 0x7f40c806f1a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f40d8069060 tx=0x7f40c4008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:26.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.135+0000 7f40d5ffb700 1 -- 172.21.15.121:0/1927516806 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f40cc057930 con 0x7f40d80ffb70 2024-04-05T21:50:26.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.345+0000 7f40df088700 1 -- 172.21.15.121:0/1927516806 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f40d8068b00 con 0x7f40c806cce0 2024-04-05T21:50:26.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.346+0000 7f40d5ffb700 1 -- 172.21.15.121:0/1927516806 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f40d8068b00 con 0x7f40c806cce0 2024-04-05T21:50:26.347 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:50:26.347 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:50:26.348 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:50:26.348 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:50:26.348 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:50:26.348 INFO:teuthology.orchestra.run.smithi121.stdout: "osd", 2024-04-05T21:50:26.348 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:50:26.348 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:50:26.348 INFO:teuthology.orchestra.run.smithi121.stdout: "mon" 2024-04-05T21:50:26.348 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:50:26.348 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "15/23 daemons upgraded", 2024-04-05T21:50:26.348 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:50:26.348 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:50:26.348 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:50:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.349+0000 7f40df088700 1 -- 172.21.15.121:0/1927516806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f40c806cce0 msgr2=0x7f40c806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.349+0000 7f40df088700 1 --2- 172.21.15.121:0/1927516806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f40c806cce0 0x7f40c806f1a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f40d8069060 tx=0x7f40c4008040 comp rx=0 tx=0).stop 2024-04-05T21:50:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.349+0000 7f40df088700 1 -- 172.21.15.121:0/1927516806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40d80ffb70 msgr2=0x7f40d81976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.349+0000 7f40df088700 1 --2- 172.21.15.121:0/1927516806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40d80ffb70 0x7f40d81976c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f40cc006010 tx=0x7f40cc00be20 comp rx=0 tx=0).stop 2024-04-05T21:50:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.349+0000 7f40df088700 1 -- 172.21.15.121:0/1927516806 shutdown_connections 2024-04-05T21:50:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.349+0000 7f40df088700 1 --2- 172.21.15.121:0/1927516806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f40c806cce0 0x7f40c806f1a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.349+0000 7f40df088700 1 --2- 172.21.15.121:0/1927516806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40d80ffb70 0x7f40d81976c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.349+0000 7f40df088700 1 --2- 172.21.15.121:0/1927516806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40d8105a30 0x7f40d8197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.349+0000 7f40df088700 1 -- 172.21.15.121:0/1927516806 >> 172.21.15.121:0/1927516806 conn(0x7f40d80754f0 msgr2=0x7f40d80fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:26.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.349+0000 7f40df088700 1 -- 172.21.15.121:0/1927516806 shutdown_connections 2024-04-05T21:50:26.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.349+0000 7f40df088700 1 -- 172.21.15.121:0/1927516806 wait complete. 2024-04-05T21:50:26.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.450+0000 7faa04c6e700 1 -- 172.21.15.121:0/2799637062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa00105a40 msgr2=0x7faa00105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:26.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.450+0000 7faa04c6e700 1 --2- 172.21.15.121:0/2799637062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa00105a40 0x7faa00105e20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009910 tx=0x7fa9f0009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:26.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.451+0000 7faa04c6e700 1 -- 172.21.15.121:0/2799637062 shutdown_connections 2024-04-05T21:50:26.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.451+0000 7faa04c6e700 1 --2- 172.21.15.121:0/2799637062 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa000ffbd0 0x7faa00100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.451+0000 7faa04c6e700 1 --2- 172.21.15.121:0/2799637062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa00105a40 0x7faa00105e20 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.451+0000 7faa04c6e700 1 -- 172.21.15.121:0/2799637062 >> 172.21.15.121:0/2799637062 conn(0x7faa000755a0 msgr2=0x7faa000759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:26.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.451+0000 7faa04c6e700 1 -- 172.21.15.121:0/2799637062 shutdown_connections 2024-04-05T21:50:26.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.451+0000 7faa04c6e700 1 -- 172.21.15.121:0/2799637062 wait complete. 2024-04-05T21:50:26.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.451+0000 7faa04c6e700 1 Processor -- start 2024-04-05T21:50:26.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.451+0000 7faa04c6e700 1 -- start start 2024-04-05T21:50:26.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7faa04c6e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa000ffbd0 0x7faa00101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:26.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7faa04c6e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa00105a40 0x7faa00101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:26.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7faa04c6e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa00102090 con 0x7faa000ffbd0 2024-04-05T21:50:26.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7faa04c6e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa001021d0 con 0x7faa00105a40 2024-04-05T21:50:26.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7fa9fe59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa000ffbd0 0x7faa00101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:26.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7fa9fe59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa000ffbd0 0x7faa00101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45544/0 (socket says 172.21.15.121:45544) 2024-04-05T21:50:26.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7fa9fe59c700 1 -- 172.21.15.121:0/3608116581 learned_addr learned my addr 172.21.15.121:0/3608116581 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:26.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7fa9fdd9b700 1 --2- 172.21.15.121:0/3608116581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa00105a40 0x7faa00101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:26.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7fa9fe59c700 1 -- 172.21.15.121:0/3608116581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa00105a40 msgr2=0x7faa00101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:26.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7fa9fe59c700 1 --2- 172.21.15.121:0/3608116581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa00105a40 0x7faa00101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7fa9fe59c700 1 -- 172.21.15.121:0/3608116581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f00095f0 con 0x7faa000ffbd0 2024-04-05T21:50:26.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7fa9fe59c700 1 --2- 172.21.15.121:0/3608116581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa000ffbd0 0x7faa00101610 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0006010 tx=0x7fa9f000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:26.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7fa9ef7fe700 1 -- 172.21.15.121:0/3608116581 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9f001d070 con 0x7faa000ffbd0 2024-04-05T21:50:26.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7faa04c6e700 1 -- 172.21.15.121:0/3608116581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa00102450 con 0x7faa000ffbd0 2024-04-05T21:50:26.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7fa9ef7fe700 1 -- 172.21.15.121:0/3608116581 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9f000f460 con 0x7faa000ffbd0 2024-04-05T21:50:26.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7fa9ef7fe700 1 -- 172.21.15.121:0/3608116581 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9f0021620 con 0x7faa000ffbd0 2024-04-05T21:50:26.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.452+0000 7faa04c6e700 1 -- 172.21.15.121:0/3608116581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa001a2b20 con 0x7faa000ffbd0 2024-04-05T21:50:26.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.453+0000 7faa04c6e700 1 -- 172.21.15.121:0/3608116581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0004ea90 con 0x7faa000ffbd0 2024-04-05T21:50:26.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.454+0000 7fa9ef7fe700 1 -- 172.21.15.121:0/3608116581 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa9f0003a60 con 0x7faa000ffbd0 2024-04-05T21:50:26.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.454+0000 7fa9ef7fe700 1 --2- 172.21.15.121:0/3608116581 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa9e806cce0 0x7fa9e806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:26.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.454+0000 7fa9ef7fe700 1 -- 172.21.15.121:0/3608116581 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa9f008d610 con 0x7faa000ffbd0 2024-04-05T21:50:26.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.454+0000 7fa9fdd9b700 1 --2- 172.21.15.121:0/3608116581 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa9e806cce0 0x7fa9e806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:26.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.455+0000 7fa9fdd9b700 1 --2- 172.21.15.121:0/3608116581 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa9e806cce0 0x7fa9e806f1a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4005950 tx=0x7fa9f40058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:26.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.459+0000 7fa9ef7fe700 1 -- 172.21.15.121:0/3608116581 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa9f00575f0 con 0x7faa000ffbd0 2024-04-05T21:50:26.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.732+0000 7faa04c6e700 1 -- 172.21.15.121:0/3608116581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faa001a2ee0 con 0x7faa000ffbd0 2024-04-05T21:50:26.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.732+0000 7fa9ef7fe700 1 -- 172.21.15.121:0/3608116581 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fa9f005ac10 con 0x7faa000ffbd0 2024-04-05T21:50:26.734 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:50:26.734 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:50:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:50:26.734 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:50:26.734 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:50:26.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.734+0000 7faa04c6e700 1 -- 172.21.15.121:0/3608116581 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa9e806cce0 msgr2=0x7fa9e806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:26.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.734+0000 7faa04c6e700 1 --2- 172.21.15.121:0/3608116581 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa9e806cce0 0x7fa9e806f1a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4005950 tx=0x7fa9f40058e0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.734+0000 7faa04c6e700 1 -- 172.21.15.121:0/3608116581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa000ffbd0 msgr2=0x7faa00101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:26.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.734+0000 7faa04c6e700 1 --2- 172.21.15.121:0/3608116581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa000ffbd0 0x7faa00101610 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0006010 tx=0x7fa9f000be20 comp rx=0 tx=0).stop 2024-04-05T21:50:26.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.734+0000 7faa04c6e700 1 -- 172.21.15.121:0/3608116581 shutdown_connections 2024-04-05T21:50:26.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.734+0000 7faa04c6e700 1 --2- 172.21.15.121:0/3608116581 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa9e806cce0 0x7fa9e806f1a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.734+0000 7faa04c6e700 1 --2- 172.21.15.121:0/3608116581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa000ffbd0 0x7faa00101610 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.734+0000 7faa04c6e700 1 --2- 172.21.15.121:0/3608116581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa00105a40 0x7faa00101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:26.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.734+0000 7faa04c6e700 1 -- 172.21.15.121:0/3608116581 >> 172.21.15.121:0/3608116581 conn(0x7faa000755a0 msgr2=0x7faa001037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:26.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.734+0000 7faa04c6e700 1 -- 172.21.15.121:0/3608116581 shutdown_connections 2024-04-05T21:50:26.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:26.734+0000 7faa04c6e700 1 -- 172.21.15.121:0/3608116581 wait complete. 2024-04-05T21:50:26.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:26 smithi121 ceph-mon[145859]: from='client.35158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:26.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:26 smithi121 ceph-mon[145859]: from='client.35162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:26.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:26 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2160076195' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:50:26.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:26 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1034426896' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:50:27.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:26 smithi188 ceph-mon[107793]: from='client.35158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:27.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:26 smithi188 ceph-mon[107793]: from='client.35162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:27.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:26 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2160076195' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:50:27.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:26 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1034426896' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:50:27.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:27 smithi121 ceph-mon[145859]: from='client.35174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:27 smithi121 ceph-mon[145859]: pgmap v726: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:27.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:27 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3608116581' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:50:28.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:27 smithi188 ceph-mon[107793]: from='client.35174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:28.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:27 smithi188 ceph-mon[107793]: pgmap v726: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:28.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:27 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3608116581' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:50:28.902 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:28 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:50:28.902 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:28 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:50:28.902 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:28 smithi188 ceph-mon[107793]: pgmap v727: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:28.903 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:28 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:28.903 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:28 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:50:28.903 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:28 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:50:28.903 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:28 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:50:28.903 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:28 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:50:28.903 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:28 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:50:28.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:28 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:50:28.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:28 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T21:50:28.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:28 smithi121 ceph-mon[145859]: pgmap v727: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:28.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:28 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:28.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:28 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:50:28.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:28 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:50:28.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:28 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:50:28.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:28 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:50:28.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:28 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:50:29.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:29 smithi121 ceph-mon[145859]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-05T21:50:29.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:29 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:29.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:29 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi188.latnft", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:50:29.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:29 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:50:29.997 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:29 smithi188 ceph-mon[107793]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-04-05T21:50:29.998 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:29 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:29.998 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:29 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi188.latnft", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-05T21:50:29.998 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:29 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T21:50:30.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:30 smithi121 ceph-mon[145859]: Upgrade: Updating mds.cephfs.smithi188.latnft 2024-04-05T21:50:30.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:30 smithi121 ceph-mon[145859]: Deploying daemon mds.cephfs.smithi188.latnft on smithi188 2024-04-05T21:50:30.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:30 smithi121 ceph-mon[145859]: pgmap v728: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:30.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:30.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:50:31.003 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:30 smithi188 ceph-mon[107793]: Upgrade: Updating mds.cephfs.smithi188.latnft 2024-04-05T21:50:31.003 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:30 smithi188 ceph-mon[107793]: Deploying daemon mds.cephfs.smithi188.latnft on smithi188 2024-04-05T21:50:31.003 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:30 smithi188 ceph-mon[107793]: pgmap v728: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:31.003 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:31.003 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:50:32.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:32 smithi188 ceph-mon[107793]: osdmap e84: 6 total, 6 up, 6 in 2024-04-05T21:50:32.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:32 smithi188 ceph-mon[107793]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T21:50:32.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:32 smithi188 ceph-mon[107793]: pgmap v730: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:32.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:32 smithi121 ceph-mon[145859]: osdmap e84: 6 total, 6 up, 6 in 2024-04-05T21:50:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:32 smithi121 ceph-mon[145859]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-04-05T21:50:32.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:32 smithi121 ceph-mon[145859]: pgmap v730: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:33.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:33 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.latnft"}]: dispatch 2024-04-05T21:50:33.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:33 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.latnft"}]: dispatch 2024-04-05T21:50:34.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:34 smithi188 ceph-mon[107793]: pgmap v731: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:34.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:34 smithi121 ceph-mon[145859]: pgmap v731: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:36.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:36.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:36.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:50:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:35 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] up:boot 2024-04-05T21:50:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:35 smithi188 ceph-mon[107793]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T21:50:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.latnft"}]: dispatch 2024-04-05T21:50:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:36.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:36.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:36.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:36.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T21:50:36.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:35 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] up:boot 2024-04-05T21:50:36.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:35 smithi121 ceph-mon[145859]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T21:50:36.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi188.latnft"}]: dispatch 2024-04-05T21:50:36.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:36.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:36.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:36.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:36.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:36 smithi121 ceph-mon[145859]: pgmap v732: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:37.025 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:36 smithi188 ceph-mon[107793]: pgmap v732: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:39.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:38 smithi188 ceph-mon[107793]: pgmap v733: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:39.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:38 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:39.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:38 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:39.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:38 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:39.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:38 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:39.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:38 smithi121 ceph-mon[145859]: pgmap v733: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:39.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:38 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:39.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:38 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:39.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:38 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:39.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:38 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:41.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:40 smithi188 ceph-mon[107793]: pgmap v734: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:41.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:40 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:41.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:40 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:40 smithi121 ceph-mon[145859]: pgmap v734: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:41.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:40 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:41.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:40 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T21:50:43.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:42 smithi188 ceph-mon[107793]: pgmap v735: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:43.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:42 smithi121 ceph-mon[145859]: pgmap v735: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:45.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:44 smithi188 ceph-mon[107793]: pgmap v736: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:45.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:44 smithi121 ceph-mon[145859]: pgmap v736: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:50:46.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:50:47.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:46 smithi188 ceph-mon[107793]: pgmap v737: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:47.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:46 smithi121 ceph-mon[145859]: pgmap v737: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:49.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:48 smithi121 ceph-mon[145859]: pgmap v738: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:48 smithi188 ceph-mon[107793]: pgmap v738: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:51.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:50 smithi121 ceph-mon[145859]: pgmap v739: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:50 smithi188 ceph-mon[107793]: pgmap v739: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:53.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:52 smithi121 ceph-mon[145859]: pgmap v740: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:52 smithi188 ceph-mon[107793]: pgmap v740: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:55.071 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:54 smithi121 ceph-mon[145859]: pgmap v741: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:54 smithi188 ceph-mon[107793]: pgmap v741: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:56.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.852+0000 7f8d3b219700 1 -- 172.21.15.121:0/3246859890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d340ff290 msgr2=0x7f8d340ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:56.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.852+0000 7f8d3b219700 1 --2- 172.21.15.121:0/3246859890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d340ff290 0x7f8d340ff670 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8d28009910 tx=0x7f8d28009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:56.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.852+0000 7f8d3b219700 1 -- 172.21.15.121:0/3246859890 shutdown_connections 2024-04-05T21:50:56.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.853+0000 7f8d3b219700 1 --2- 172.21.15.121:0/3246859890 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d340ffc40 0x7f8d34105780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:56.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.853+0000 7f8d3b219700 1 --2- 172.21.15.121:0/3246859890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d340ff290 0x7f8d340ff670 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:56.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.853+0000 7f8d3b219700 1 -- 172.21.15.121:0/3246859890 >> 172.21.15.121:0/3246859890 conn(0x7f8d34075e30 msgr2=0x7f8d340fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:56.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.853+0000 7f8d3b219700 1 -- 172.21.15.121:0/3246859890 shutdown_connections 2024-04-05T21:50:56.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.853+0000 7f8d3b219700 1 -- 172.21.15.121:0/3246859890 wait complete. 2024-04-05T21:50:56.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.853+0000 7f8d3b219700 1 Processor -- start 2024-04-05T21:50:56.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.853+0000 7f8d3b219700 1 -- start start 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.853+0000 7f8d3b219700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d340ff290 0x7f8d34100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.853+0000 7f8d3b219700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d340ffc40 0x7f8d34100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.853+0000 7f8d3b219700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d34100c90 con 0x7f8d340ff290 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.853+0000 7f8d3b219700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d34100dd0 con 0x7f8d340ffc40 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.854+0000 7f8d38fb5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d340ff290 0x7f8d34100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.854+0000 7f8d38fb5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d340ff290 0x7f8d34100210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32808/0 (socket says 172.21.15.121:32808) 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.854+0000 7f8d38fb5700 1 -- 172.21.15.121:0/4281914034 learned_addr learned my addr 172.21.15.121:0/4281914034 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.854+0000 7f8d33fff700 1 --2- 172.21.15.121:0/4281914034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d340ffc40 0x7f8d34100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.854+0000 7f8d38fb5700 1 -- 172.21.15.121:0/4281914034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d340ffc40 msgr2=0x7f8d34100750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.854+0000 7f8d38fb5700 1 --2- 172.21.15.121:0/4281914034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d340ffc40 0x7f8d34100750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.854+0000 7f8d38fb5700 1 -- 172.21.15.121:0/4281914034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d280095f0 con 0x7f8d340ff290 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.854+0000 7f8d38fb5700 1 --2- 172.21.15.121:0/4281914034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d340ff290 0x7f8d34100210 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8d28006010 tx=0x7f8d2800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.854+0000 7f8d31ffb700 1 -- 172.21.15.121:0/4281914034 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d2801d070 con 0x7f8d340ff290 2024-04-05T21:50:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.854+0000 7f8d31ffb700 1 -- 172.21.15.121:0/4281914034 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d2800f460 con 0x7f8d340ff290 2024-04-05T21:50:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.854+0000 7f8d3b219700 1 -- 172.21.15.121:0/4281914034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d34101050 con 0x7f8d340ff290 2024-04-05T21:50:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.854+0000 7f8d31ffb700 1 -- 172.21.15.121:0/4281914034 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d28021620 con 0x7f8d340ff290 2024-04-05T21:50:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.854+0000 7f8d3b219700 1 -- 172.21.15.121:0/4281914034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d341a2bd0 con 0x7f8d340ff290 2024-04-05T21:50:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.856+0000 7f8d3b219700 1 -- 172.21.15.121:0/4281914034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d34109120 con 0x7f8d340ff290 2024-04-05T21:50:56.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.856+0000 7f8d31ffb700 1 -- 172.21.15.121:0/4281914034 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8d28003a60 con 0x7f8d340ff290 2024-04-05T21:50:56.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.856+0000 7f8d31ffb700 1 --2- 172.21.15.121:0/4281914034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d2406cce0 0x7f8d2406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:56.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.856+0000 7f8d31ffb700 1 -- 172.21.15.121:0/4281914034 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8d2808d710 con 0x7f8d340ff290 2024-04-05T21:50:56.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.856+0000 7f8d33fff700 1 --2- 172.21.15.121:0/4281914034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d2406cce0 0x7f8d2406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:56.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.857+0000 7f8d33fff700 1 --2- 172.21.15.121:0/4281914034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d2406cce0 0x7f8d2406f1a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8d20005950 tx=0x7f8d200058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:56.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:56.861+0000 7f8d31ffb700 1 -- 172.21.15.121:0/4281914034 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8d28057800 con 0x7f8d340ff290 2024-04-05T21:50:57.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.070+0000 7f8d3b219700 1 -- 172.21.15.121:0/4281914034 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d3402cce0 con 0x7f8d2406cce0 2024-04-05T21:50:57.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.071+0000 7f8d31ffb700 1 -- 172.21.15.121:0/4281914034 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8d3402cce0 con 0x7f8d2406cce0 2024-04-05T21:50:57.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.073+0000 7f8d3b219700 1 -- 172.21.15.121:0/4281914034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d2406cce0 msgr2=0x7f8d2406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:57.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.073+0000 7f8d3b219700 1 --2- 172.21.15.121:0/4281914034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d2406cce0 0x7f8d2406f1a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8d20005950 tx=0x7f8d200058e0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.073+0000 7f8d3b219700 1 -- 172.21.15.121:0/4281914034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d340ff290 msgr2=0x7f8d34100210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:57.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.073+0000 7f8d3b219700 1 --2- 172.21.15.121:0/4281914034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d340ff290 0x7f8d34100210 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8d28006010 tx=0x7f8d2800be20 comp rx=0 tx=0).stop 2024-04-05T21:50:57.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.073+0000 7f8d3b219700 1 -- 172.21.15.121:0/4281914034 shutdown_connections 2024-04-05T21:50:57.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.074+0000 7f8d3b219700 1 --2- 172.21.15.121:0/4281914034 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d2406cce0 0x7f8d2406f1a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.074+0000 7f8d3b219700 1 --2- 172.21.15.121:0/4281914034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d340ff290 0x7f8d34100210 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.074+0000 7f8d3b219700 1 --2- 172.21.15.121:0/4281914034 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d340ffc40 0x7f8d34100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.074+0000 7f8d3b219700 1 -- 172.21.15.121:0/4281914034 >> 172.21.15.121:0/4281914034 conn(0x7f8d34075e30 msgr2=0x7f8d34104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:57.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.074+0000 7f8d3b219700 1 -- 172.21.15.121:0/4281914034 shutdown_connections 2024-04-05T21:50:57.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.074+0000 7f8d3b219700 1 -- 172.21.15.121:0/4281914034 wait complete. 2024-04-05T21:50:57.084 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:50:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.176+0000 7fd1860e3700 1 -- 172.21.15.121:0/3054794217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1800ffb70 msgr2=0x7fd1800fffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.176+0000 7fd1860e3700 1 --2- 172.21.15.121:0/3054794217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1800ffb70 0x7fd1800fffd0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd174009910 tx=0x7fd174009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.176+0000 7fd1860e3700 1 -- 172.21.15.121:0/3054794217 shutdown_connections 2024-04-05T21:50:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.176+0000 7fd1860e3700 1 --2- 172.21.15.121:0/3054794217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1800ffb70 0x7fd1800fffd0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.176+0000 7fd1860e3700 1 --2- 172.21.15.121:0/3054794217 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd180105a30 0x7fd180105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.176+0000 7fd1860e3700 1 -- 172.21.15.121:0/3054794217 >> 172.21.15.121:0/3054794217 conn(0x7fd1800754f0 msgr2=0x7fd180075900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.176+0000 7fd1860e3700 1 -- 172.21.15.121:0/3054794217 shutdown_connections 2024-04-05T21:50:57.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.176+0000 7fd1860e3700 1 -- 172.21.15.121:0/3054794217 wait complete. 2024-04-05T21:50:57.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.177+0000 7fd1860e3700 1 Processor -- start 2024-04-05T21:50:57.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.177+0000 7fd1860e3700 1 -- start start 2024-04-05T21:50:57.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.177+0000 7fd1860e3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1800ffb70 0x7fd180196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.177+0000 7fd1860e3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd180105a30 0x7fd180196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.177+0000 7fd1860e3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd180197250 con 0x7fd180105a30 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.177+0000 7fd1860e3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18019afe0 con 0x7fd1800ffb70 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.178+0000 7fd17effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd180105a30 0x7fd180196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.178+0000 7fd17effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd180105a30 0x7fd180196b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32824/0 (socket says 172.21.15.121:32824) 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.178+0000 7fd17effd700 1 -- 172.21.15.121:0/2714742968 learned_addr learned my addr 172.21.15.121:0/2714742968 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.178+0000 7fd17f7fe700 1 --2- 172.21.15.121:0/2714742968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1800ffb70 0x7fd180196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.178+0000 7fd17effd700 1 -- 172.21.15.121:0/2714742968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1800ffb70 msgr2=0x7fd180196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.178+0000 7fd17effd700 1 --2- 172.21.15.121:0/2714742968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1800ffb70 0x7fd180196630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.178+0000 7fd17effd700 1 -- 172.21.15.121:0/2714742968 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1740095f0 con 0x7fd180105a30 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.178+0000 7fd17effd700 1 --2- 172.21.15.121:0/2714742968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd180105a30 0x7fd180196b70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd174000c00 tx=0x7fd17400bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.178+0000 7fd17cff9700 1 -- 172.21.15.121:0/2714742968 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd17401d070 con 0x7fd180105a30 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.178+0000 7fd17cff9700 1 -- 172.21.15.121:0/2714742968 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd174005dc0 con 0x7fd180105a30 2024-04-05T21:50:57.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.178+0000 7fd17cff9700 1 -- 172.21.15.121:0/2714742968 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd174003940 con 0x7fd180105a30 2024-04-05T21:50:57.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.178+0000 7fd1860e3700 1 -- 172.21.15.121:0/2714742968 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18019b260 con 0x7fd180105a30 2024-04-05T21:50:57.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.178+0000 7fd1860e3700 1 -- 172.21.15.121:0/2714742968 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18019b6f0 con 0x7fd180105a30 2024-04-05T21:50:57.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.179+0000 7fd1860e3700 1 -- 172.21.15.121:0/2714742968 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd180108fd0 con 0x7fd180105a30 2024-04-05T21:50:57.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.180+0000 7fd17cff9700 1 -- 172.21.15.121:0/2714742968 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd174004ae0 con 0x7fd180105a30 2024-04-05T21:50:57.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.180+0000 7fd17cff9700 1 --2- 172.21.15.121:0/2714742968 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd16806cce0 0x7fd16806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:57.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.180+0000 7fd17cff9700 1 -- 172.21.15.121:0/2714742968 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd17408db90 con 0x7fd180105a30 2024-04-05T21:50:57.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.180+0000 7fd17f7fe700 1 --2- 172.21.15.121:0/2714742968 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd16806cce0 0x7fd16806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:57.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.180+0000 7fd17f7fe700 1 --2- 172.21.15.121:0/2714742968 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd16806cce0 0x7fd16806f1a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd170009c60 tx=0x7fd170009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:57.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.185+0000 7fd17cff9700 1 -- 172.21.15.121:0/2714742968 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd174057c00 con 0x7fd180105a30 2024-04-05T21:50:57.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:56 smithi121 ceph-mon[145859]: pgmap v742: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:56 smithi188 ceph-mon[107793]: pgmap v742: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:57.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.397+0000 7fd1860e3700 1 -- 172.21.15.121:0/2714742968 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd18002cd80 con 0x7fd16806cce0 2024-04-05T21:50:57.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.398+0000 7fd17cff9700 1 -- 172.21.15.121:0/2714742968 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd18002cd80 con 0x7fd16806cce0 2024-04-05T21:50:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.401+0000 7fd1860e3700 1 -- 172.21.15.121:0/2714742968 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd16806cce0 msgr2=0x7fd16806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.401+0000 7fd1860e3700 1 --2- 172.21.15.121:0/2714742968 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd16806cce0 0x7fd16806f1a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd170009c60 tx=0x7fd170009260 comp rx=0 tx=0).stop 2024-04-05T21:50:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.401+0000 7fd1860e3700 1 -- 172.21.15.121:0/2714742968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd180105a30 msgr2=0x7fd180196b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.401+0000 7fd1860e3700 1 --2- 172.21.15.121:0/2714742968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd180105a30 0x7fd180196b70 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd174000c00 tx=0x7fd17400bde0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.401+0000 7fd1860e3700 1 -- 172.21.15.121:0/2714742968 shutdown_connections 2024-04-05T21:50:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.401+0000 7fd1860e3700 1 --2- 172.21.15.121:0/2714742968 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd16806cce0 0x7fd16806f1a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.401+0000 7fd1860e3700 1 --2- 172.21.15.121:0/2714742968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd1800ffb70 0x7fd180196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.401+0000 7fd1860e3700 1 --2- 172.21.15.121:0/2714742968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd180105a30 0x7fd180196b70 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.401+0000 7fd1860e3700 1 -- 172.21.15.121:0/2714742968 >> 172.21.15.121:0/2714742968 conn(0x7fd1800754f0 msgr2=0x7fd180068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:57.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.401+0000 7fd1860e3700 1 -- 172.21.15.121:0/2714742968 shutdown_connections 2024-04-05T21:50:57.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.401+0000 7fd1860e3700 1 -- 172.21.15.121:0/2714742968 wait complete. 2024-04-05T21:50:57.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.504+0000 7fb42ff1c700 1 -- 172.21.15.121:0/111987911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb428105a40 msgr2=0x7fb428105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:57.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.504+0000 7fb42ff1c700 1 --2- 172.21.15.121:0/111987911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb428105a40 0x7fb428105e20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb418009910 tx=0x7fb418009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:57.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.504+0000 7fb42ff1c700 1 -- 172.21.15.121:0/111987911 shutdown_connections 2024-04-05T21:50:57.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.504+0000 7fb42ff1c700 1 --2- 172.21.15.121:0/111987911 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4280ffa40 0x7fb4280ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.504+0000 7fb42ff1c700 1 --2- 172.21.15.121:0/111987911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb428105a40 0x7fb428105e20 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.504+0000 7fb42ff1c700 1 -- 172.21.15.121:0/111987911 >> 172.21.15.121:0/111987911 conn(0x7fb4280fb570 msgr2=0x7fb4280fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:57.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.504+0000 7fb42ff1c700 1 -- 172.21.15.121:0/111987911 shutdown_connections 2024-04-05T21:50:57.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.504+0000 7fb42ff1c700 1 -- 172.21.15.121:0/111987911 wait complete. 2024-04-05T21:50:57.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.504+0000 7fb42ff1c700 1 Processor -- start 2024-04-05T21:50:57.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.504+0000 7fb42ff1c700 1 -- start start 2024-04-05T21:50:57.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42ff1c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4280ffa40 0x7fb42819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:57.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42ff1c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb428105a40 0x7fb42819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:57.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42ff1c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42819d0f0 con 0x7fb428105a40 2024-04-05T21:50:57.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42ff1c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4281965a0 con 0x7fb4280ffa40 2024-04-05T21:50:57.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42d4b7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb428105a40 0x7fb42819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:57.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42d4b7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb428105a40 0x7fb42819ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32834/0 (socket says 172.21.15.121:32834) 2024-04-05T21:50:57.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42d4b7700 1 -- 172.21.15.121:0/2235650288 learned_addr learned my addr 172.21.15.121:0/2235650288 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:57.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42dcb8700 1 --2- 172.21.15.121:0/2235650288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4280ffa40 0x7fb42819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:57.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42d4b7700 1 -- 172.21.15.121:0/2235650288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4280ffa40 msgr2=0x7fb42819c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:57.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42d4b7700 1 --2- 172.21.15.121:0/2235650288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4280ffa40 0x7fb42819c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42d4b7700 1 -- 172.21.15.121:0/2235650288 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4180095f0 con 0x7fb428105a40 2024-04-05T21:50:57.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42d4b7700 1 --2- 172.21.15.121:0/2235650288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb428105a40 0x7fb42819ca60 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb42400b4b0 tx=0x7fb42400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:57.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb41effd700 1 -- 172.21.15.121:0/2235650288 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb424011800 con 0x7fb428105a40 2024-04-05T21:50:57.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb41effd700 1 -- 172.21.15.121:0/2235650288 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb424011e00 con 0x7fb428105a40 2024-04-05T21:50:57.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42ff1c700 1 -- 172.21.15.121:0/2235650288 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb428196880 con 0x7fb428105a40 2024-04-05T21:50:57.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb41effd700 1 -- 172.21.15.121:0/2235650288 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb424010e60 con 0x7fb428105a40 2024-04-05T21:50:57.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.505+0000 7fb42ff1c700 1 -- 172.21.15.121:0/2235650288 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb428196dd0 con 0x7fb428105a40 2024-04-05T21:50:57.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.506+0000 7fb42ff1c700 1 -- 172.21.15.121:0/2235650288 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb428109050 con 0x7fb428105a40 2024-04-05T21:50:57.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.507+0000 7fb41effd700 1 -- 172.21.15.121:0/2235650288 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb424011960 con 0x7fb428105a40 2024-04-05T21:50:57.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.507+0000 7fb41effd700 1 --2- 172.21.15.121:0/2235650288 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb41406cce0 0x7fb41406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:57.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.507+0000 7fb41effd700 1 -- 172.21.15.121:0/2235650288 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb42408c0d0 con 0x7fb428105a40 2024-04-05T21:50:57.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.507+0000 7fb42dcb8700 1 --2- 172.21.15.121:0/2235650288 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb41406cce0 0x7fb41406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:57.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.508+0000 7fb42dcb8700 1 --2- 172.21.15.121:0/2235650288 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb41406cce0 0x7fb41406f1a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb418005f50 tx=0x7fb41801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:57.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.512+0000 7fb41effd700 1 -- 172.21.15.121:0/2235650288 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb424059bd0 con 0x7fb428105a40 2024-04-05T21:50:57.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.716+0000 7fb42ff1c700 1 -- 172.21.15.121:0/2235650288 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4281970e0 con 0x7fb41406cce0 2024-04-05T21:50:57.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.726+0000 7fb41effd700 1 -- 172.21.15.121:0/2235650288 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb4281970e0 con 0x7fb41406cce0 2024-04-05T21:50:57.726 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (21m) 17s ago 33m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (33m) 17s ago 33m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (32m) 19s ago 32m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (19m) 17s ago 33m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (19m) 19s ago 32m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (21m) 17s ago 32m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (16m) 17s ago 27m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (15m) 17s ago 27m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (15m) 19s ago 27m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (23s) 19s ago 26m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (23m) 17s ago 34m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (22m) 19s ago 32m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (20m) 17s ago 34m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:50:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (20m) 19s ago 32m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:50:57.728 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (22m) 17s ago 33m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:50:57.728 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (22m) 19s ago 32m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:50:57.728 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (19m) 17s ago 31m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:50:57.728 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (18m) 17s ago 30m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:50:57.728 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (18m) 17s ago 30m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:50:57.728 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (17m) 19s ago 29m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:50:57.728 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (17m) 19s ago 29m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:50:57.728 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (16m) 19s ago 28m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:50:57.728 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (22m) 17s ago 32m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:50:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.728+0000 7fb42ff1c700 1 -- 172.21.15.121:0/2235650288 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb41406cce0 msgr2=0x7fb41406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.728+0000 7fb42ff1c700 1 --2- 172.21.15.121:0/2235650288 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb41406cce0 0x7fb41406f1a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb418005f50 tx=0x7fb41801a040 comp rx=0 tx=0).stop 2024-04-05T21:50:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.728+0000 7fb42ff1c700 1 -- 172.21.15.121:0/2235650288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb428105a40 msgr2=0x7fb42819ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.728+0000 7fb42ff1c700 1 --2- 172.21.15.121:0/2235650288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb428105a40 0x7fb42819ca60 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb42400b4b0 tx=0x7fb42400b7c0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.728+0000 7fb42ff1c700 1 -- 172.21.15.121:0/2235650288 shutdown_connections 2024-04-05T21:50:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.728+0000 7fb42ff1c700 1 --2- 172.21.15.121:0/2235650288 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb41406cce0 0x7fb41406f1a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.728+0000 7fb42ff1c700 1 --2- 172.21.15.121:0/2235650288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4280ffa40 0x7fb42819c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.728+0000 7fb42ff1c700 1 --2- 172.21.15.121:0/2235650288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb428105a40 0x7fb42819ca60 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.728+0000 7fb42ff1c700 1 -- 172.21.15.121:0/2235650288 >> 172.21.15.121:0/2235650288 conn(0x7fb4280fb570 msgr2=0x7fb428069040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.728+0000 7fb42ff1c700 1 -- 172.21.15.121:0/2235650288 shutdown_connections 2024-04-05T21:50:57.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.728+0000 7fb42ff1c700 1 -- 172.21.15.121:0/2235650288 wait complete. 2024-04-05T21:50:57.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.830+0000 7f0808f3d700 1 -- 172.21.15.121:0/1357282641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804068670 msgr2=0x7f0804068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:57.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.830+0000 7f0808f3d700 1 --2- 172.21.15.121:0/1357282641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804068670 0x7f0804068a50 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009910 tx=0x7f07f4009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:57.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.831+0000 7f0808f3d700 1 -- 172.21.15.121:0/1357282641 shutdown_connections 2024-04-05T21:50:57.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.831+0000 7f0808f3d700 1 --2- 172.21.15.121:0/1357282641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0804069020 0x7f0804105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.831+0000 7f0808f3d700 1 --2- 172.21.15.121:0/1357282641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804068670 0x7f0804068a50 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.831+0000 7f0808f3d700 1 -- 172.21.15.121:0/1357282641 >> 172.21.15.121:0/1357282641 conn(0x7f08040755a0 msgr2=0x7f08040759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:57.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.831+0000 7f0808f3d700 1 -- 172.21.15.121:0/1357282641 shutdown_connections 2024-04-05T21:50:57.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.831+0000 7f0808f3d700 1 -- 172.21.15.121:0/1357282641 wait complete. 2024-04-05T21:50:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.831+0000 7f0808f3d700 1 Processor -- start 2024-04-05T21:50:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.831+0000 7f0808f3d700 1 -- start start 2024-04-05T21:50:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f0808f3d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804068670 0x7f080419c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f0808f3d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0804069020 0x7f080419cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f0808f3d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080419d170 con 0x7f0804068670 2024-04-05T21:50:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f0808f3d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0804196620 con 0x7f0804069020 2024-04-05T21:50:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f080259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804068670 0x7f080419c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f080259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804068670 0x7f080419c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32854/0 (socket says 172.21.15.121:32854) 2024-04-05T21:50:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f080259c700 1 -- 172.21.15.121:0/2090131581 learned_addr learned my addr 172.21.15.121:0/2090131581 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:57.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f0801d9b700 1 --2- 172.21.15.121:0/2090131581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0804069020 0x7f080419cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f080259c700 1 -- 172.21.15.121:0/2090131581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0804069020 msgr2=0x7f080419cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f080259c700 1 --2- 172.21.15.121:0/2090131581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0804069020 0x7f080419cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f080259c700 1 -- 172.21.15.121:0/2090131581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f40095f0 con 0x7f0804068670 2024-04-05T21:50:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f080259c700 1 --2- 172.21.15.121:0/2090131581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804068670 0x7f080419c5a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f07f4006010 tx=0x7f07f400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f07f37fe700 1 -- 172.21.15.121:0/2090131581 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07f401d070 con 0x7f0804068670 2024-04-05T21:50:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f07f37fe700 1 -- 172.21.15.121:0/2090131581 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07f400f460 con 0x7f0804068670 2024-04-05T21:50:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f0808f3d700 1 -- 172.21.15.121:0/2090131581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08041968a0 con 0x7f0804068670 2024-04-05T21:50:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f07f37fe700 1 -- 172.21.15.121:0/2090131581 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07f4021640 con 0x7f0804068670 2024-04-05T21:50:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.832+0000 7f0808f3d700 1 -- 172.21.15.121:0/2090131581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0804196d90 con 0x7f0804068670 2024-04-05T21:50:57.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.833+0000 7f0808f3d700 1 -- 172.21.15.121:0/2090131581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0804109130 con 0x7f0804068670 2024-04-05T21:50:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.834+0000 7f07f37fe700 1 -- 172.21.15.121:0/2090131581 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f07f400f9e0 con 0x7f0804068670 2024-04-05T21:50:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.834+0000 7f07f37fe700 1 --2- 172.21.15.121:0/2090131581 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f07ec06cce0 0x7f07ec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.834+0000 7f07f37fe700 1 -- 172.21.15.121:0/2090131581 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f07f408d900 con 0x7f0804068670 2024-04-05T21:50:57.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.834+0000 7f0801d9b700 1 --2- 172.21.15.121:0/2090131581 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f07ec06cce0 0x7f07ec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:57.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.834+0000 7f0801d9b700 1 --2- 172.21.15.121:0/2090131581 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f07ec06cce0 0x7f07ec06f1a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f07f8006fd0 tx=0x7f07f8008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:57.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:57.838+0000 7f07f37fe700 1 -- 172.21.15.121:0/2090131581 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f07f405b400 con 0x7f0804068670 2024-04-05T21:50:58.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.112+0000 7f0808f3d700 1 -- 172.21.15.121:0/2090131581 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0804108b10 con 0x7f0804068670 2024-04-05T21:50:58.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.113+0000 7f07f37fe700 1 -- 172.21.15.121:0/2090131581 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f07f405af90 con 0x7f0804068670 2024-04-05T21:50:58.114 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:50:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:50:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:50:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:50:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:50:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:50:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:50:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:50:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:50:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:50:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:50:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:50:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:50:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:50:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:50:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:50:58.115 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:50:58.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.115+0000 7f0808f3d700 1 -- 172.21.15.121:0/2090131581 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f07ec06cce0 msgr2=0x7f07ec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:58.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.115+0000 7f0808f3d700 1 --2- 172.21.15.121:0/2090131581 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f07ec06cce0 0x7f07ec06f1a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f07f8006fd0 tx=0x7f07f8008040 comp rx=0 tx=0).stop 2024-04-05T21:50:58.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.115+0000 7f0808f3d700 1 -- 172.21.15.121:0/2090131581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804068670 msgr2=0x7f080419c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:58.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.115+0000 7f0808f3d700 1 --2- 172.21.15.121:0/2090131581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804068670 0x7f080419c5a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f07f4006010 tx=0x7f07f400be20 comp rx=0 tx=0).stop 2024-04-05T21:50:58.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.115+0000 7f0808f3d700 1 -- 172.21.15.121:0/2090131581 shutdown_connections 2024-04-05T21:50:58.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.115+0000 7f0808f3d700 1 --2- 172.21.15.121:0/2090131581 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f07ec06cce0 0x7f07ec06f1a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.115+0000 7f0808f3d700 1 --2- 172.21.15.121:0/2090131581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804068670 0x7f080419c5a0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.115+0000 7f0808f3d700 1 --2- 172.21.15.121:0/2090131581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0804069020 0x7f080419cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.115+0000 7f0808f3d700 1 -- 172.21.15.121:0/2090131581 >> 172.21.15.121:0/2090131581 conn(0x7f08040755a0 msgr2=0x7f08040fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:58.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.115+0000 7f0808f3d700 1 -- 172.21.15.121:0/2090131581 shutdown_connections 2024-04-05T21:50:58.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.115+0000 7f0808f3d700 1 -- 172.21.15.121:0/2090131581 wait complete. 2024-04-05T21:50:58.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.217+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/284944111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb84105a40 msgr2=0x7ffb84105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:58.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.217+0000 7ffb8c0ee700 1 --2- 172.21.15.121:0/284944111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb84105a40 0x7ffb84105e20 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ffb80009910 tx=0x7ffb80009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:58.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.217+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/284944111 shutdown_connections 2024-04-05T21:50:58.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.217+0000 7ffb8c0ee700 1 --2- 172.21.15.121:0/284944111 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb840ffbd0 0x7ffb84100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.217+0000 7ffb8c0ee700 1 --2- 172.21.15.121:0/284944111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb84105a40 0x7ffb84105e20 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.217+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/284944111 >> 172.21.15.121:0/284944111 conn(0x7ffb840755a0 msgr2=0x7ffb840759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:58.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.218+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/284944111 shutdown_connections 2024-04-05T21:50:58.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.218+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/284944111 wait complete. 2024-04-05T21:50:58.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.218+0000 7ffb8c0ee700 1 Processor -- start 2024-04-05T21:50:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.218+0000 7ffb8c0ee700 1 -- start start 2024-04-05T21:50:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.218+0000 7ffb8c0ee700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb840ffbd0 0x7ffb841974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.218+0000 7ffb8c0ee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb84105a40 0x7ffb84197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.218+0000 7ffb8c0ee700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb84198020 con 0x7ffb84105a40 2024-04-05T21:50:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.218+0000 7ffb8c0ee700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb84198160 con 0x7ffb840ffbd0 2024-04-05T21:50:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.218+0000 7ffb89689700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb84105a40 0x7ffb84197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.218+0000 7ffb89689700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb84105a40 0x7ffb84197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32876/0 (socket says 172.21.15.121:32876) 2024-04-05T21:50:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.218+0000 7ffb89689700 1 -- 172.21.15.121:0/647615068 learned_addr learned my addr 172.21.15.121:0/647615068 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.219+0000 7ffb89689700 1 -- 172.21.15.121:0/647615068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb840ffbd0 msgr2=0x7ffb841974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.219+0000 7ffb89e8a700 1 --2- 172.21.15.121:0/647615068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb840ffbd0 0x7ffb841974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.219+0000 7ffb89689700 1 --2- 172.21.15.121:0/647615068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb840ffbd0 0x7ffb841974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.219+0000 7ffb89689700 1 -- 172.21.15.121:0/647615068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb800095f0 con 0x7ffb84105a40 2024-04-05T21:50:58.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.219+0000 7ffb89e8a700 1 --2- 172.21.15.121:0/647615068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb840ffbd0 0x7ffb841974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:50:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.219+0000 7ffb89689700 1 --2- 172.21.15.121:0/647615068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb84105a40 0x7ffb84197a00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400c8a0 tx=0x7ffb7400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.219+0000 7ffb7affd700 1 -- 172.21.15.121:0/647615068 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb7400cea0 con 0x7ffb84105a40 2024-04-05T21:50:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.219+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/647615068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb841a0530 con 0x7ffb84105a40 2024-04-05T21:50:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.219+0000 7ffb7affd700 1 -- 172.21.15.121:0/647615068 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb74004830 con 0x7ffb84105a40 2024-04-05T21:50:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.219+0000 7ffb7affd700 1 -- 172.21.15.121:0/647615068 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb74005500 con 0x7ffb84105a40 2024-04-05T21:50:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.219+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/647615068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb841a09f0 con 0x7ffb84105a40 2024-04-05T21:50:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.220+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/647615068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb84108040 con 0x7ffb84105a40 2024-04-05T21:50:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.220+0000 7ffb7affd700 1 -- 172.21.15.121:0/647615068 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ffb740049a0 con 0x7ffb84105a40 2024-04-05T21:50:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.221+0000 7ffb7affd700 1 --2- 172.21.15.121:0/647615068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ffb7006cce0 0x7ffb7006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.221+0000 7ffb89e8a700 1 --2- 172.21.15.121:0/647615068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ffb7006cce0 0x7ffb7006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:58.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.221+0000 7ffb7affd700 1 -- 172.21.15.121:0/647615068 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ffb7408c2b0 con 0x7ffb84105a40 2024-04-05T21:50:58.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.221+0000 7ffb89e8a700 1 --2- 172.21.15.121:0/647615068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ffb7006cce0 0x7ffb7006f1a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ffb80006010 tx=0x7ffb8000b540 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:58.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.226+0000 7ffb7affd700 1 -- 172.21.15.121:0/647615068 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ffb74056320 con 0x7ffb84105a40 2024-04-05T21:50:58.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.460+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/647615068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ffb8404ea90 con 0x7ffb84105a40 2024-04-05T21:50:58.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.460+0000 7ffb7affd700 1 -- 172.21.15.121:0/647615068 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7ffb74059940 con 0x7ffb84105a40 2024-04-05T21:50:58.461 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:50:58.461 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:50:58.461 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:50:58.461 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:50:58.461 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:50:58.461 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:50:58.461 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:50:58.461 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:50:58.461 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:50:58.461 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:50:58.462 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:50:58.462 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:50:58.462 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:50:58.462 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:50:58.462 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:50:58.462 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:50:58.462 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:50:58.462 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:50:58.462 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:50:58.462 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:50:58.462 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:50:58.462 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:50:58.463 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:50:58.464 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:50:58.464 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:50:58.464 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:50:58.465 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:50:58.465 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:50:58.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.462+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/647615068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ffb7006cce0 msgr2=0x7ffb7006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:58.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.462+0000 7ffb8c0ee700 1 --2- 172.21.15.121:0/647615068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ffb7006cce0 0x7ffb7006f1a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ffb80006010 tx=0x7ffb8000b540 comp rx=0 tx=0).stop 2024-04-05T21:50:58.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.462+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/647615068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb84105a40 msgr2=0x7ffb84197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:58.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.462+0000 7ffb8c0ee700 1 --2- 172.21.15.121:0/647615068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb84105a40 0x7ffb84197a00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400c8a0 tx=0x7ffb7400cc60 comp rx=0 tx=0).stop 2024-04-05T21:50:58.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.462+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/647615068 shutdown_connections 2024-04-05T21:50:58.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.462+0000 7ffb8c0ee700 1 --2- 172.21.15.121:0/647615068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ffb7006cce0 0x7ffb7006f1a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.462+0000 7ffb8c0ee700 1 --2- 172.21.15.121:0/647615068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ffb840ffbd0 0x7ffb841974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.462+0000 7ffb8c0ee700 1 --2- 172.21.15.121:0/647615068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb84105a40 0x7ffb84197a00 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.462+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/647615068 >> 172.21.15.121:0/647615068 conn(0x7ffb840755a0 msgr2=0x7ffb840fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:58.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.462+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/647615068 shutdown_connections 2024-04-05T21:50:58.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.462+0000 7ffb8c0ee700 1 -- 172.21.15.121:0/647615068 wait complete. 2024-04-05T21:50:58.467 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:50:58.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.565+0000 7fa3603eb700 1 -- 172.21.15.121:0/1972883828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580ffbd0 msgr2=0x7fa358100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.565+0000 7fa3603eb700 1 --2- 172.21.15.121:0/1972883828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580ffbd0 0x7fa358100050 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa354009910 tx=0x7fa354009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.565+0000 7fa3603eb700 1 -- 172.21.15.121:0/1972883828 shutdown_connections 2024-04-05T21:50:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.565+0000 7fa3603eb700 1 --2- 172.21.15.121:0/1972883828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580ffbd0 0x7fa358100050 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.565+0000 7fa3603eb700 1 --2- 172.21.15.121:0/1972883828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa358105a40 0x7fa358105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.565+0000 7fa3603eb700 1 -- 172.21.15.121:0/1972883828 >> 172.21.15.121:0/1972883828 conn(0x7fa3580755a0 msgr2=0x7fa3580759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.565+0000 7fa3603eb700 1 -- 172.21.15.121:0/1972883828 shutdown_connections 2024-04-05T21:50:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.565+0000 7fa3603eb700 1 -- 172.21.15.121:0/1972883828 wait complete. 2024-04-05T21:50:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.566+0000 7fa3603eb700 1 Processor -- start 2024-04-05T21:50:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.566+0000 7fa3603eb700 1 -- start start 2024-04-05T21:50:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.566+0000 7fa3603eb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3580ffbd0 0x7fa35819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.566+0000 7fa3603eb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa358105a40 0x7fa35819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.566+0000 7fa3603eb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35819d0e0 con 0x7fa358105a40 2024-04-05T21:50:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.566+0000 7fa3603eb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa358196590 con 0x7fa3580ffbd0 2024-04-05T21:50:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.566+0000 7fa35d986700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa358105a40 0x7fa35819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.566+0000 7fa35d986700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa358105a40 0x7fa35819ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32902/0 (socket says 172.21.15.121:32902) 2024-04-05T21:50:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.566+0000 7fa35d986700 1 -- 172.21.15.121:0/4290886806 learned_addr learned my addr 172.21.15.121:0/4290886806 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.566+0000 7fa35e187700 1 --2- 172.21.15.121:0/4290886806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3580ffbd0 0x7fa35819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.567+0000 7fa35d986700 1 -- 172.21.15.121:0/4290886806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3580ffbd0 msgr2=0x7fa35819c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.567+0000 7fa35d986700 1 --2- 172.21.15.121:0/4290886806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3580ffbd0 0x7fa35819c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.567+0000 7fa35d986700 1 -- 172.21.15.121:0/4290886806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3540095f0 con 0x7fa358105a40 2024-04-05T21:50:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.567+0000 7fa35d986700 1 --2- 172.21.15.121:0/4290886806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa358105a40 0x7fa35819ca50 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa354000c00 tx=0x7fa35400bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.567+0000 7fa34f7fe700 1 -- 172.21.15.121:0/4290886806 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa35401d070 con 0x7fa358105a40 2024-04-05T21:50:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.567+0000 7fa34f7fe700 1 -- 172.21.15.121:0/4290886806 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa354005dc0 con 0x7fa358105a40 2024-04-05T21:50:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.567+0000 7fa3603eb700 1 -- 172.21.15.121:0/4290886806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa358196810 con 0x7fa358105a40 2024-04-05T21:50:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.567+0000 7fa34f7fe700 1 -- 172.21.15.121:0/4290886806 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa354003940 con 0x7fa358105a40 2024-04-05T21:50:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.567+0000 7fa3603eb700 1 -- 172.21.15.121:0/4290886806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa358196ca0 con 0x7fa358105a40 2024-04-05T21:50:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.568+0000 7fa3603eb700 1 -- 172.21.15.121:0/4290886806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa358108fe0 con 0x7fa358105a40 2024-04-05T21:50:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.568+0000 7fa34f7fe700 1 -- 172.21.15.121:0/4290886806 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa354004ae0 con 0x7fa358105a40 2024-04-05T21:50:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.569+0000 7fa34f7fe700 1 --2- 172.21.15.121:0/4290886806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa34406cce0 0x7fa34406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.569+0000 7fa34f7fe700 1 -- 172.21.15.121:0/4290886806 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa35408db70 con 0x7fa358105a40 2024-04-05T21:50:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.569+0000 7fa35e187700 1 --2- 172.21.15.121:0/4290886806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa34406cce0 0x7fa34406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.569+0000 7fa35e187700 1 --2- 172.21.15.121:0/4290886806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa34406cce0 0x7fa34406f1a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa348009c60 tx=0x7fa348009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.573+0000 7fa34f7fe700 1 -- 172.21.15.121:0/4290886806 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa35405b5d0 con 0x7fa358105a40 2024-04-05T21:50:58.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.782+0000 7fa3603eb700 1 -- 172.21.15.121:0/4290886806 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa358196f70 con 0x7fa34406cce0 2024-04-05T21:50:58.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.783+0000 7fa34f7fe700 1 -- 172.21.15.121:0/4290886806 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa358196f70 con 0x7fa34406cce0 2024-04-05T21:50:58.783 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:50:58.784 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:50:58.784 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:50:58.784 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:50:58.784 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:50:58.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:50:58.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:50:58.784 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:50:58.784 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:50:58.784 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:50:58.784 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:50:58.784 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:50:58.784 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:50:58.784 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:50:58.785 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:50:58.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.785+0000 7fa3603eb700 1 -- 172.21.15.121:0/4290886806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa34406cce0 msgr2=0x7fa34406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:58.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.785+0000 7fa3603eb700 1 --2- 172.21.15.121:0/4290886806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa34406cce0 0x7fa34406f1a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa348009c60 tx=0x7fa348009260 comp rx=0 tx=0).stop 2024-04-05T21:50:58.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.785+0000 7fa3603eb700 1 -- 172.21.15.121:0/4290886806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa358105a40 msgr2=0x7fa35819ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:58.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.785+0000 7fa3603eb700 1 --2- 172.21.15.121:0/4290886806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa358105a40 0x7fa35819ca50 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa354000c00 tx=0x7fa35400bde0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.785+0000 7fa3603eb700 1 -- 172.21.15.121:0/4290886806 shutdown_connections 2024-04-05T21:50:58.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.785+0000 7fa3603eb700 1 --2- 172.21.15.121:0/4290886806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa34406cce0 0x7fa34406f1a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.785+0000 7fa3603eb700 1 --2- 172.21.15.121:0/4290886806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa3580ffbd0 0x7fa35819c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.785+0000 7fa3603eb700 1 --2- 172.21.15.121:0/4290886806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa358105a40 0x7fa35819ca50 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.785+0000 7fa3603eb700 1 -- 172.21.15.121:0/4290886806 >> 172.21.15.121:0/4290886806 conn(0x7fa3580755a0 msgr2=0x7fa358103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:58.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.785+0000 7fa3603eb700 1 -- 172.21.15.121:0/4290886806 shutdown_connections 2024-04-05T21:50:58.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.785+0000 7fa3603eb700 1 -- 172.21.15.121:0/4290886806 wait complete. 2024-04-05T21:50:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.887+0000 7f5de1335700 1 -- 172.21.15.121:0/1344467169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ddc0ffee0 msgr2=0x7f5ddc103eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.887+0000 7f5de1335700 1 --2- 172.21.15.121:0/1344467169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ddc0ffee0 0x7f5ddc103eb0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009910 tx=0x7f5dd0009c20 comp rx=0 tx=0).stop 2024-04-05T21:50:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.887+0000 7f5de1335700 1 -- 172.21.15.121:0/1344467169 shutdown_connections 2024-04-05T21:50:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.887+0000 7f5de1335700 1 --2- 172.21.15.121:0/1344467169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ddc0ffee0 0x7f5ddc103eb0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.887+0000 7f5de1335700 1 --2- 172.21.15.121:0/1344467169 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ddc0ff530 0x7f5ddc0ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.887+0000 7f5de1335700 1 -- 172.21.15.121:0/1344467169 >> 172.21.15.121:0/1344467169 conn(0x7f5ddc074df0 msgr2=0x7f5ddc075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.887+0000 7f5de1335700 1 -- 172.21.15.121:0/1344467169 shutdown_connections 2024-04-05T21:50:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.887+0000 7f5de1335700 1 -- 172.21.15.121:0/1344467169 wait complete. 2024-04-05T21:50:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.888+0000 7f5de1335700 1 Processor -- start 2024-04-05T21:50:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.888+0000 7f5de1335700 1 -- start start 2024-04-05T21:50:58.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.888+0000 7f5de1335700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ddc0ff530 0x7f5ddc100f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:58.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.888+0000 7f5de1335700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ddc0ffee0 0x7f5ddc101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:58.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.888+0000 7f5de1335700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc1019d0 con 0x7f5ddc0ff530 2024-04-05T21:50:58.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.888+0000 7f5de1335700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ddc101b10 con 0x7f5ddc0ffee0 2024-04-05T21:50:58.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.888+0000 7f5ddaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ddc0ff530 0x7f5ddc100f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:58.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.888+0000 7f5ddaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ddc0ff530 0x7f5ddc100f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32906/0 (socket says 172.21.15.121:32906) 2024-04-05T21:50:58.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.888+0000 7f5ddaffd700 1 -- 172.21.15.121:0/1936339944 learned_addr learned my addr 172.21.15.121:0/1936339944 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:50:58.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.889+0000 7f5dda7fc700 1 --2- 172.21.15.121:0/1936339944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ddc0ffee0 0x7f5ddc101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:58.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.889+0000 7f5ddaffd700 1 -- 172.21.15.121:0/1936339944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ddc0ffee0 msgr2=0x7f5ddc101490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:58.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.889+0000 7f5ddaffd700 1 --2- 172.21.15.121:0/1936339944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ddc0ffee0 0x7f5ddc101490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:58.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.889+0000 7f5ddaffd700 1 -- 172.21.15.121:0/1936339944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd00095f0 con 0x7f5ddc0ff530 2024-04-05T21:50:58.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.889+0000 7f5ddaffd700 1 --2- 172.21.15.121:0/1936339944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ddc0ff530 0x7f5ddc100f50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00b4b0 tx=0x7f5dcc00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:58.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.889+0000 7f5dc3fff700 1 -- 172.21.15.121:0/1936339944 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dcc011800 con 0x7f5ddc0ff530 2024-04-05T21:50:58.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.889+0000 7f5de1335700 1 -- 172.21.15.121:0/1936339944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ddc1a29b0 con 0x7f5ddc0ff530 2024-04-05T21:50:58.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.889+0000 7f5dc3fff700 1 -- 172.21.15.121:0/1936339944 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5dcc011e00 con 0x7f5ddc0ff530 2024-04-05T21:50:58.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.889+0000 7f5dc3fff700 1 -- 172.21.15.121:0/1936339944 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dcc010e50 con 0x7f5ddc0ff530 2024-04-05T21:50:58.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.889+0000 7f5de1335700 1 -- 172.21.15.121:0/1936339944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ddc1a2dc0 con 0x7f5ddc0ff530 2024-04-05T21:50:58.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.890+0000 7f5de1335700 1 -- 172.21.15.121:0/1936339944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ddc04ea90 con 0x7f5ddc0ff530 2024-04-05T21:50:58.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.890+0000 7f5dc3fff700 1 -- 172.21.15.121:0/1936339944 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5dcc010380 con 0x7f5ddc0ff530 2024-04-05T21:50:58.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.891+0000 7f5dc3fff700 1 --2- 172.21.15.121:0/1936339944 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5dc406cce0 0x7f5dc406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:50:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.891+0000 7f5dc3fff700 1 -- 172.21.15.121:0/1936339944 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5dcc08c150 con 0x7f5ddc0ff530 2024-04-05T21:50:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.891+0000 7f5dda7fc700 1 --2- 172.21.15.121:0/1936339944 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5dc406cce0 0x7f5dc406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:50:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.891+0000 7f5dda7fc700 1 --2- 172.21.15.121:0/1936339944 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5dc406cce0 0x7f5dc406f1a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000b5c0 tx=0x7f5dd0005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:50:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:58.895+0000 7f5dc3fff700 1 -- 172.21.15.121:0/1936339944 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5dcc0561c0 con 0x7f5ddc0ff530 2024-04-05T21:50:59.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:59.166+0000 7f5de1335700 1 -- 172.21.15.121:0/1936339944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5ddc1a31a0 con 0x7f5ddc0ff530 2024-04-05T21:50:59.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:59.166+0000 7f5dc3fff700 1 -- 172.21.15.121:0/1936339944 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f5dcc015e10 con 0x7f5ddc0ff530 2024-04-05T21:50:59.167 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:50:59.167 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:50:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:50:59.167 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:50:59.167 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:50:59.167 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:50:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:50:59.167 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:50:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:50:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:50:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:50:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:50:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:50:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:50:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:50:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:50:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:50:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:50:59.168 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:50:59.168 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:50:59.168 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:50:59.169 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:50:59.169 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:50:59.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:59.168+0000 7f5de1335700 1 -- 172.21.15.121:0/1936339944 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5dc406cce0 msgr2=0x7f5dc406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:59.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:59.168+0000 7f5de1335700 1 --2- 172.21.15.121:0/1936339944 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5dc406cce0 0x7f5dc406f1a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000b5c0 tx=0x7f5dd0005fd0 comp rx=0 tx=0).stop 2024-04-05T21:50:59.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:59.168+0000 7f5de1335700 1 -- 172.21.15.121:0/1936339944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ddc0ff530 msgr2=0x7f5ddc100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:50:59.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:59.168+0000 7f5de1335700 1 --2- 172.21.15.121:0/1936339944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ddc0ff530 0x7f5ddc100f50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00b4b0 tx=0x7f5dcc00b870 comp rx=0 tx=0).stop 2024-04-05T21:50:59.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:59.169+0000 7f5de1335700 1 -- 172.21.15.121:0/1936339944 shutdown_connections 2024-04-05T21:50:59.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:59.169+0000 7f5de1335700 1 --2- 172.21.15.121:0/1936339944 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5dc406cce0 0x7f5dc406f1a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:59.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:59.169+0000 7f5de1335700 1 --2- 172.21.15.121:0/1936339944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ddc0ff530 0x7f5ddc100f50 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:59.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:59.169+0000 7f5de1335700 1 --2- 172.21.15.121:0/1936339944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5ddc0ffee0 0x7f5ddc101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:50:59.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:59.169+0000 7f5de1335700 1 -- 172.21.15.121:0/1936339944 >> 172.21.15.121:0/1936339944 conn(0x7f5ddc074df0 msgr2=0x7f5ddc103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:50:59.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:59.169+0000 7f5de1335700 1 -- 172.21.15.121:0/1936339944 shutdown_connections 2024-04-05T21:50:59.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:50:59.169+0000 7f5de1335700 1 -- 172.21.15.121:0/1936339944 wait complete. 2024-04-05T21:50:59.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:58 smithi121 ceph-mon[145859]: from='client.35186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:59.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:58 smithi121 ceph-mon[145859]: from='client.35190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:59.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:58 smithi121 ceph-mon[145859]: pgmap v743: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:59.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:58 smithi121 ceph-mon[145859]: from='client.35194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:59.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:58 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2090131581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:50:59.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:58 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/647615068' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:50:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:58 smithi188 ceph-mon[107793]: from='client.35186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:58 smithi188 ceph-mon[107793]: from='client.35190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:58 smithi188 ceph-mon[107793]: pgmap v743: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:50:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:58 smithi188 ceph-mon[107793]: from='client.35194 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:50:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:58 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2090131581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:50:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:58 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/647615068' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:51:00.118 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:59 smithi121 ceph-mon[145859]: from='client.35206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:51:00.118 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:50:59 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1936339944' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:51:00.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:59 smithi188 ceph-mon[107793]: from='client.35206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:51:00.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:50:59 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1936339944' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:51:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:00 smithi121 ceph-mon[145859]: pgmap v744: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:51:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:00 smithi188 ceph-mon[107793]: pgmap v744: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:51:03.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:02 smithi121 ceph-mon[145859]: pgmap v745: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:02 smithi188 ceph-mon[107793]: pgmap v745: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:05.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:04 smithi121 ceph-mon[145859]: pgmap v746: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:04 smithi188 ceph-mon[107793]: pgmap v746: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:07.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:06 smithi121 ceph-mon[145859]: pgmap v747: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:06 smithi188 ceph-mon[107793]: pgmap v747: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:09.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:08 smithi188 ceph-mon[107793]: pgmap v748: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:09.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:08 smithi121 ceph-mon[145859]: pgmap v748: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:11.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:10 smithi121 ceph-mon[145859]: pgmap v749: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:10 smithi188 ceph-mon[107793]: pgmap v749: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:13.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:12 smithi121 ceph-mon[145859]: pgmap v750: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:12 smithi188 ceph-mon[107793]: pgmap v750: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:15.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:14 smithi121 ceph-mon[145859]: pgmap v751: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:14 smithi188 ceph-mon[107793]: pgmap v751: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:16.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:51:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:51:17.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:16 smithi121 ceph-mon[145859]: pgmap v752: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:17.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:16 smithi188 ceph-mon[107793]: pgmap v752: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:19.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:18 smithi188 ceph-mon[107793]: pgmap v753: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:19.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:18 smithi121 ceph-mon[145859]: pgmap v753: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:21.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:20 smithi121 ceph-mon[145859]: pgmap v754: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:20 smithi188 ceph-mon[107793]: pgmap v754: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:23.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:22 smithi121 ceph-mon[145859]: pgmap v755: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:23.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:22 smithi188 ceph-mon[107793]: pgmap v755: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:25.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:24 smithi121 ceph-mon[145859]: pgmap v756: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:24 smithi188 ceph-mon[107793]: pgmap v756: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:27.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:26 smithi121 ceph-mon[145859]: pgmap v757: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:27.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:26 smithi188 ceph-mon[107793]: pgmap v757: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:29.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:28 smithi121 ceph-mon[145859]: pgmap v758: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:28 smithi188 ceph-mon[107793]: pgmap v758: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:29.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.286+0000 7f06c1894700 1 -- 172.21.15.121:0/693495288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06bc068d30 msgr2=0x7f06bc10d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:29.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.286+0000 7f06c1894700 1 --2- 172.21.15.121:0/693495288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06bc068d30 0x7f06bc10d240 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f06b0009910 tx=0x7f06b0009c20 comp rx=0 tx=0).stop 2024-04-05T21:51:29.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.286+0000 7f06c1894700 1 -- 172.21.15.121:0/693495288 shutdown_connections 2024-04-05T21:51:29.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.286+0000 7f06c1894700 1 --2- 172.21.15.121:0/693495288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06bc068d30 0x7f06bc10d240 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.286+0000 7f06c1894700 1 --2- 172.21.15.121:0/693495288 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06bc068410 0x7f06bc0687f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.286+0000 7f06c1894700 1 -- 172.21.15.121:0/693495288 >> 172.21.15.121:0/693495288 conn(0x7f06bc0755a0 msgr2=0x7f06bc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:29.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.286+0000 7f06c1894700 1 -- 172.21.15.121:0/693495288 shutdown_connections 2024-04-05T21:51:29.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.286+0000 7f06c1894700 1 -- 172.21.15.121:0/693495288 wait complete. 2024-04-05T21:51:29.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.286+0000 7f06c1894700 1 Processor -- start 2024-04-05T21:51:29.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.287+0000 7f06c1894700 1 -- start start 2024-04-05T21:51:29.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.287+0000 7f06c1894700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06bc068410 0x7f06bc19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:29.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.287+0000 7f06c1894700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06bc068d30 0x7f06bc19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:29.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.287+0000 7f06c1894700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc19f1f0 con 0x7f06bc068d30 2024-04-05T21:51:29.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.287+0000 7f06c1894700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc1986a0 con 0x7f06bc068410 2024-04-05T21:51:29.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.287+0000 7f06ba7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06bc068d30 0x7f06bc19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:29.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.287+0000 7f06ba7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06bc068d30 0x7f06bc19eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55028/0 (socket says 172.21.15.121:55028) 2024-04-05T21:51:29.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.287+0000 7f06ba7fc700 1 -- 172.21.15.121:0/1614801952 learned_addr learned my addr 172.21.15.121:0/1614801952 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:51:29.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.287+0000 7f06baffd700 1 --2- 172.21.15.121:0/1614801952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06bc068410 0x7f06bc19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:29.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.287+0000 7f06ba7fc700 1 -- 172.21.15.121:0/1614801952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06bc068410 msgr2=0x7f06bc19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:29.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.287+0000 7f06ba7fc700 1 --2- 172.21.15.121:0/1614801952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06bc068410 0x7f06bc19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.287+0000 7f06ba7fc700 1 -- 172.21.15.121:0/1614801952 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b00095f0 con 0x7f06bc068d30 2024-04-05T21:51:29.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.288+0000 7f06ba7fc700 1 --2- 172.21.15.121:0/1614801952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06bc068d30 0x7f06bc19eb60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f06b0006010 tx=0x7f06b000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:29.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.288+0000 7f06c0892700 1 -- 172.21.15.121:0/1614801952 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06b001d070 con 0x7f06bc068d30 2024-04-05T21:51:29.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.288+0000 7f06c0892700 1 -- 172.21.15.121:0/1614801952 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06b000f460 con 0x7f06bc068d30 2024-04-05T21:51:29.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.288+0000 7f06c1894700 1 -- 172.21.15.121:0/1614801952 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06bc198980 con 0x7f06bc068d30 2024-04-05T21:51:29.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.288+0000 7f06c0892700 1 -- 172.21.15.121:0/1614801952 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06b0021620 con 0x7f06bc068d30 2024-04-05T21:51:29.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.288+0000 7f06c1894700 1 -- 172.21.15.121:0/1614801952 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06bc198ed0 con 0x7f06bc068d30 2024-04-05T21:51:29.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.289+0000 7f06c1894700 1 -- 172.21.15.121:0/1614801952 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06bc10a9b0 con 0x7f06bc068d30 2024-04-05T21:51:29.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.290+0000 7f06c0892700 1 -- 172.21.15.121:0/1614801952 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f06b000f5d0 con 0x7f06bc068d30 2024-04-05T21:51:29.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.290+0000 7f06c0892700 1 --2- 172.21.15.121:0/1614801952 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f06a406cce0 0x7f06a406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:29.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.290+0000 7f06c0892700 1 -- 172.21.15.121:0/1614801952 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f06b008d850 con 0x7f06bc068d30 2024-04-05T21:51:29.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.290+0000 7f06baffd700 1 --2- 172.21.15.121:0/1614801952 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f06a406cce0 0x7f06a406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:29.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.291+0000 7f06baffd700 1 --2- 172.21.15.121:0/1614801952 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f06a406cce0 0x7f06a406f1a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f06bc076b00 tx=0x7f06ac00a380 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:29.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.295+0000 7f06c0892700 1 -- 172.21.15.121:0/1614801952 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f06b005b350 con 0x7f06bc068d30 2024-04-05T21:51:29.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.505+0000 7f06c1894700 1 -- 172.21.15.121:0/1614801952 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f06bc1991b0 con 0x7f06a406cce0 2024-04-05T21:51:29.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.506+0000 7f06c0892700 1 -- 172.21.15.121:0/1614801952 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f06bc1991b0 con 0x7f06a406cce0 2024-04-05T21:51:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.508+0000 7f06c1894700 1 -- 172.21.15.121:0/1614801952 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f06a406cce0 msgr2=0x7f06a406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.508+0000 7f06c1894700 1 --2- 172.21.15.121:0/1614801952 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f06a406cce0 0x7f06a406f1a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f06bc076b00 tx=0x7f06ac00a380 comp rx=0 tx=0).stop 2024-04-05T21:51:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.508+0000 7f06c1894700 1 -- 172.21.15.121:0/1614801952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06bc068d30 msgr2=0x7f06bc19eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.508+0000 7f06c1894700 1 --2- 172.21.15.121:0/1614801952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06bc068d30 0x7f06bc19eb60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f06b0006010 tx=0x7f06b000bd70 comp rx=0 tx=0).stop 2024-04-05T21:51:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.508+0000 7f06c1894700 1 -- 172.21.15.121:0/1614801952 shutdown_connections 2024-04-05T21:51:29.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.508+0000 7f06c1894700 1 --2- 172.21.15.121:0/1614801952 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f06a406cce0 0x7f06a406f1a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.508+0000 7f06c1894700 1 --2- 172.21.15.121:0/1614801952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f06bc068410 0x7f06bc19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.508+0000 7f06c1894700 1 --2- 172.21.15.121:0/1614801952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06bc068d30 0x7f06bc19eb60 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.508+0000 7f06c1894700 1 -- 172.21.15.121:0/1614801952 >> 172.21.15.121:0/1614801952 conn(0x7f06bc0755a0 msgr2=0x7f06bc0fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.509+0000 7f06c1894700 1 -- 172.21.15.121:0/1614801952 shutdown_connections 2024-04-05T21:51:29.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.509+0000 7f06c1894700 1 -- 172.21.15.121:0/1614801952 wait complete. 2024-04-05T21:51:29.519 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:51:29.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.612+0000 7fd8b4071700 1 -- 172.21.15.121:0/4164408943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ac068670 msgr2=0x7fd8ac068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:29.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.612+0000 7fd8b4071700 1 --2- 172.21.15.121:0/4164408943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ac068670 0x7fd8ac068a50 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009910 tx=0x7fd8a8009c20 comp rx=0 tx=0).stop 2024-04-05T21:51:29.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.612+0000 7fd8b4071700 1 -- 172.21.15.121:0/4164408943 shutdown_connections 2024-04-05T21:51:29.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.612+0000 7fd8b4071700 1 --2- 172.21.15.121:0/4164408943 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8ac069020 0x7fd8ac105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.612+0000 7fd8b4071700 1 --2- 172.21.15.121:0/4164408943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ac068670 0x7fd8ac068a50 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.612+0000 7fd8b4071700 1 -- 172.21.15.121:0/4164408943 >> 172.21.15.121:0/4164408943 conn(0x7fd8ac0755a0 msgr2=0x7fd8ac0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:29.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.613+0000 7fd8b4071700 1 -- 172.21.15.121:0/4164408943 shutdown_connections 2024-04-05T21:51:29.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.613+0000 7fd8b4071700 1 -- 172.21.15.121:0/4164408943 wait complete. 2024-04-05T21:51:29.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.613+0000 7fd8b4071700 1 Processor -- start 2024-04-05T21:51:29.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.613+0000 7fd8b4071700 1 -- start start 2024-04-05T21:51:29.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.613+0000 7fd8b4071700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ac068670 0x7fd8ac0ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:29.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.613+0000 7fd8b4071700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8ac069020 0x7fd8ac1001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:29.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.613+0000 7fd8b4071700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac100720 con 0x7fd8ac068670 2024-04-05T21:51:29.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.613+0000 7fd8b4071700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac100860 con 0x7fd8ac069020 2024-04-05T21:51:29.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.613+0000 7fd8b1e0d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ac068670 0x7fd8ac0ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:29.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.613+0000 7fd8b1e0d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ac068670 0x7fd8ac0ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55058/0 (socket says 172.21.15.121:55058) 2024-04-05T21:51:29.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.613+0000 7fd8b1e0d700 1 -- 172.21.15.121:0/2276609477 learned_addr learned my addr 172.21.15.121:0/2276609477 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:51:29.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.614+0000 7fd8b1e0d700 1 -- 172.21.15.121:0/2276609477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8ac069020 msgr2=0x7fd8ac1001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:29.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.614+0000 7fd8b1e0d700 1 --2- 172.21.15.121:0/2276609477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8ac069020 0x7fd8ac1001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.614+0000 7fd8b1e0d700 1 -- 172.21.15.121:0/2276609477 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a80095f0 con 0x7fd8ac068670 2024-04-05T21:51:29.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.614+0000 7fd8b1e0d700 1 --2- 172.21.15.121:0/2276609477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ac068670 0x7fd8ac0ffca0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8006010 tx=0x7fd8a800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:29.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.614+0000 7fd8a2ffd700 1 -- 172.21.15.121:0/2276609477 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8a801d070 con 0x7fd8ac068670 2024-04-05T21:51:29.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.614+0000 7fd8b4071700 1 -- 172.21.15.121:0/2276609477 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ac100ae0 con 0x7fd8ac068670 2024-04-05T21:51:29.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.614+0000 7fd8a2ffd700 1 -- 172.21.15.121:0/2276609477 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8a800f460 con 0x7fd8ac068670 2024-04-05T21:51:29.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.614+0000 7fd8a2ffd700 1 -- 172.21.15.121:0/2276609477 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8a8021640 con 0x7fd8ac068670 2024-04-05T21:51:29.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.614+0000 7fd8b4071700 1 -- 172.21.15.121:0/2276609477 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ac1a2b70 con 0x7fd8ac068670 2024-04-05T21:51:29.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.615+0000 7fd8b4071700 1 -- 172.21.15.121:0/2276609477 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ac109130 con 0x7fd8ac068670 2024-04-05T21:51:29.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.615+0000 7fd8a2ffd700 1 -- 172.21.15.121:0/2276609477 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd8a800fa60 con 0x7fd8ac068670 2024-04-05T21:51:29.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.616+0000 7fd8a2ffd700 1 --2- 172.21.15.121:0/2276609477 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd89806cce0 0x7fd89806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:29.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.616+0000 7fd8a2ffd700 1 -- 172.21.15.121:0/2276609477 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd8a808d860 con 0x7fd8ac068670 2024-04-05T21:51:29.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.616+0000 7fd8b160c700 1 --2- 172.21.15.121:0/2276609477 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd89806cce0 0x7fd89806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:29.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.616+0000 7fd8b160c700 1 --2- 172.21.15.121:0/2276609477 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd89806cce0 0x7fd89806f1a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac101490 tx=0x7fd89c005c80 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:29.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.620+0000 7fd8a2ffd700 1 -- 172.21.15.121:0/2276609477 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd8a80578d0 con 0x7fd8ac068670 2024-04-05T21:51:29.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.835+0000 7fd8b4071700 1 -- 172.21.15.121:0/2276609477 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8ac02cd10 con 0x7fd89806cce0 2024-04-05T21:51:29.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.836+0000 7fd8a2ffd700 1 -- 172.21.15.121:0/2276609477 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd8ac02cd10 con 0x7fd89806cce0 2024-04-05T21:51:29.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.838+0000 7fd8b4071700 1 -- 172.21.15.121:0/2276609477 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd89806cce0 msgr2=0x7fd89806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:29.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.838+0000 7fd8b4071700 1 --2- 172.21.15.121:0/2276609477 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd89806cce0 0x7fd89806f1a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac101490 tx=0x7fd89c005c80 comp rx=0 tx=0).stop 2024-04-05T21:51:29.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.838+0000 7fd8b4071700 1 -- 172.21.15.121:0/2276609477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ac068670 msgr2=0x7fd8ac0ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:29.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.838+0000 7fd8b4071700 1 --2- 172.21.15.121:0/2276609477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ac068670 0x7fd8ac0ffca0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8006010 tx=0x7fd8a800be20 comp rx=0 tx=0).stop 2024-04-05T21:51:29.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.838+0000 7fd8b4071700 1 -- 172.21.15.121:0/2276609477 shutdown_connections 2024-04-05T21:51:29.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.838+0000 7fd8b4071700 1 --2- 172.21.15.121:0/2276609477 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd89806cce0 0x7fd89806f1a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.838+0000 7fd8b4071700 1 --2- 172.21.15.121:0/2276609477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ac068670 0x7fd8ac0ffca0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.838+0000 7fd8b4071700 1 --2- 172.21.15.121:0/2276609477 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd8ac069020 0x7fd8ac1001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.838+0000 7fd8b4071700 1 -- 172.21.15.121:0/2276609477 >> 172.21.15.121:0/2276609477 conn(0x7fd8ac0755a0 msgr2=0x7fd8ac0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:29.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.838+0000 7fd8b4071700 1 -- 172.21.15.121:0/2276609477 shutdown_connections 2024-04-05T21:51:29.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.838+0000 7fd8b4071700 1 -- 172.21.15.121:0/2276609477 wait complete. 2024-04-05T21:51:29.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.941+0000 7fd5f3743700 1 -- 172.21.15.121:0/1794448674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5ec0ffbd0 msgr2=0x7fd5ec100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:29.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.941+0000 7fd5f3743700 1 --2- 172.21.15.121:0/1794448674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5ec0ffbd0 0x7fd5ec100050 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009910 tx=0x7fd5e8009c20 comp rx=0 tx=0).stop 2024-04-05T21:51:29.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.941+0000 7fd5f3743700 1 -- 172.21.15.121:0/1794448674 shutdown_connections 2024-04-05T21:51:29.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.941+0000 7fd5f3743700 1 --2- 172.21.15.121:0/1794448674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5ec0ffbd0 0x7fd5ec100050 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.941+0000 7fd5f3743700 1 --2- 172.21.15.121:0/1794448674 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5ec105a40 0x7fd5ec105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.941+0000 7fd5f3743700 1 -- 172.21.15.121:0/1794448674 >> 172.21.15.121:0/1794448674 conn(0x7fd5ec0755a0 msgr2=0x7fd5ec0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:29.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.941+0000 7fd5f3743700 1 -- 172.21.15.121:0/1794448674 shutdown_connections 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.941+0000 7fd5f3743700 1 -- 172.21.15.121:0/1794448674 wait complete. 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f3743700 1 Processor -- start 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f3743700 1 -- start start 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f3743700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5ec0ffbd0 0x7fd5ec197450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f3743700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5ec105a40 0x7fd5ec197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f3743700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec198070 con 0x7fd5ec0ffbd0 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f3743700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ec1981b0 con 0x7fd5ec105a40 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f14df700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5ec0ffbd0 0x7fd5ec197450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f14df700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5ec0ffbd0 0x7fd5ec197450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55076/0 (socket says 172.21.15.121:55076) 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f14df700 1 -- 172.21.15.121:0/3655432791 learned_addr learned my addr 172.21.15.121:0/3655432791 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f0cde700 1 --2- 172.21.15.121:0/3655432791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5ec105a40 0x7fd5ec197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f14df700 1 -- 172.21.15.121:0/3655432791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5ec105a40 msgr2=0x7fd5ec197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f14df700 1 --2- 172.21.15.121:0/3655432791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5ec105a40 0x7fd5ec197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f14df700 1 -- 172.21.15.121:0/3655432791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e80095f0 con 0x7fd5ec0ffbd0 2024-04-05T21:51:29.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.942+0000 7fd5f14df700 1 --2- 172.21.15.121:0/3655432791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5ec0ffbd0 0x7fd5ec197450 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00d8d0 tx=0x7fd5dc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:29.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.943+0000 7fd5e27fc700 1 -- 172.21.15.121:0/3655432791 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5dc009940 con 0x7fd5ec0ffbd0 2024-04-05T21:51:29.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.943+0000 7fd5e27fc700 1 -- 172.21.15.121:0/3655432791 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5dc010460 con 0x7fd5ec0ffbd0 2024-04-05T21:51:29.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.943+0000 7fd5f3743700 1 -- 172.21.15.121:0/3655432791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ec1a04c0 con 0x7fd5ec0ffbd0 2024-04-05T21:51:29.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.943+0000 7fd5e27fc700 1 -- 172.21.15.121:0/3655432791 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5dc00f5f0 con 0x7fd5ec0ffbd0 2024-04-05T21:51:29.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.943+0000 7fd5f3743700 1 -- 172.21.15.121:0/3655432791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ec1a09e0 con 0x7fd5ec0ffbd0 2024-04-05T21:51:29.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.943+0000 7fd5f3743700 1 -- 172.21.15.121:0/3655432791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ec107fd0 con 0x7fd5ec0ffbd0 2024-04-05T21:51:29.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.944+0000 7fd5e27fc700 1 -- 172.21.15.121:0/3655432791 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd5dc0105d0 con 0x7fd5ec0ffbd0 2024-04-05T21:51:29.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.944+0000 7fd5e27fc700 1 --2- 172.21.15.121:0/3655432791 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd5d806cce0 0x7fd5d806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:29.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.944+0000 7fd5e27fc700 1 -- 172.21.15.121:0/3655432791 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd5dc08c5e0 con 0x7fd5ec0ffbd0 2024-04-05T21:51:29.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.944+0000 7fd5f0cde700 1 --2- 172.21.15.121:0/3655432791 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd5d806cce0 0x7fd5d806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:29.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.945+0000 7fd5f0cde700 1 --2- 172.21.15.121:0/3655432791 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd5d806cce0 0x7fd5d806f1a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0694f0 tx=0x7fd5e801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:29.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:29.949+0000 7fd5e27fc700 1 -- 172.21.15.121:0/3655432791 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd5dc056650 con 0x7fd5ec0ffbd0 2024-04-05T21:51:30.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.150+0000 7fd5f3743700 1 -- 172.21.15.121:0/3655432791 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5ec02cd80 con 0x7fd5d806cce0 2024-04-05T21:51:30.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.160+0000 7fd5e27fc700 1 -- 172.21.15.121:0/3655432791 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fd5ec02cd80 con 0x7fd5d806cce0 2024-04-05T21:51:30.160 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:51:30.161 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (22m) 49s ago 33m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:51:30.161 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (34m) 49s ago 34m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:51:30.161 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (32m) 52s ago 32m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:51:30.161 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (20m) 49s ago 34m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:51:30.161 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (20m) 52s ago 32m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:51:30.161 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (22m) 49s ago 33m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:51:30.161 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (16m) 49s ago 27m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:51:30.161 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (16m) 49s ago 27m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:51:30.161 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (16m) 52s ago 27m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:51:30.161 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (55s) 52s ago 27m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:51:30.161 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (24m) 49s ago 35m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:51:30.161 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (23m) 52s ago 32m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:51:30.162 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (21m) 49s ago 35m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:51:30.162 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (20m) 52s ago 32m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:51:30.162 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (23m) 49s ago 33m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:51:30.162 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (23m) 52s ago 32m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:51:30.162 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (20m) 49s ago 31m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:51:30.162 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (19m) 49s ago 31m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:51:30.162 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (18m) 49s ago 30m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:51:30.162 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (18m) 52s ago 30m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:51:30.162 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (17m) 52s ago 29m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:51:30.162 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (17m) 52s ago 29m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:51:30.162 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (22m) 49s ago 33m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:51:30.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.162+0000 7fd5f3743700 1 -- 172.21.15.121:0/3655432791 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd5d806cce0 msgr2=0x7fd5d806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:30.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.162+0000 7fd5f3743700 1 --2- 172.21.15.121:0/3655432791 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd5d806cce0 0x7fd5d806f1a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0694f0 tx=0x7fd5e801a040 comp rx=0 tx=0).stop 2024-04-05T21:51:30.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.162+0000 7fd5f3743700 1 -- 172.21.15.121:0/3655432791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5ec0ffbd0 msgr2=0x7fd5ec197450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:30.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.162+0000 7fd5f3743700 1 --2- 172.21.15.121:0/3655432791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5ec0ffbd0 0x7fd5ec197450 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc00d8d0 tx=0x7fd5dc00dc90 comp rx=0 tx=0).stop 2024-04-05T21:51:30.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.162+0000 7fd5f3743700 1 -- 172.21.15.121:0/3655432791 shutdown_connections 2024-04-05T21:51:30.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.162+0000 7fd5f3743700 1 --2- 172.21.15.121:0/3655432791 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd5d806cce0 0x7fd5d806f1a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.162+0000 7fd5f3743700 1 --2- 172.21.15.121:0/3655432791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5ec0ffbd0 0x7fd5ec197450 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.162+0000 7fd5f3743700 1 --2- 172.21.15.121:0/3655432791 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5ec105a40 0x7fd5ec197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.162+0000 7fd5f3743700 1 -- 172.21.15.121:0/3655432791 >> 172.21.15.121:0/3655432791 conn(0x7fd5ec0755a0 msgr2=0x7fd5ec0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:30.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.162+0000 7fd5f3743700 1 -- 172.21.15.121:0/3655432791 shutdown_connections 2024-04-05T21:51:30.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.162+0000 7fd5f3743700 1 -- 172.21.15.121:0/3655432791 wait complete. 2024-04-05T21:51:30.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.265+0000 7fc4504b9700 1 -- 172.21.15.121:0/2865498604 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480ffa40 msgr2=0x7fc4480ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:30.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.265+0000 7fc4504b9700 1 --2- 172.21.15.121:0/2865498604 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480ffa40 0x7fc4480ffec0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc444009910 tx=0x7fc444009c20 comp rx=0 tx=0).stop 2024-04-05T21:51:30.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.265+0000 7fc4504b9700 1 -- 172.21.15.121:0/2865498604 shutdown_connections 2024-04-05T21:51:30.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.265+0000 7fc4504b9700 1 --2- 172.21.15.121:0/2865498604 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480ffa40 0x7fc4480ffec0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.265+0000 7fc4504b9700 1 --2- 172.21.15.121:0/2865498604 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc448105a40 0x7fc448105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.265+0000 7fc4504b9700 1 -- 172.21.15.121:0/2865498604 >> 172.21.15.121:0/2865498604 conn(0x7fc4480fb570 msgr2=0x7fc4480fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:30.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.265+0000 7fc4504b9700 1 -- 172.21.15.121:0/2865498604 shutdown_connections 2024-04-05T21:51:30.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.265+0000 7fc4504b9700 1 -- 172.21.15.121:0/2865498604 wait complete. 2024-04-05T21:51:30.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.266+0000 7fc4504b9700 1 Processor -- start 2024-04-05T21:51:30.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.266+0000 7fc4504b9700 1 -- start start 2024-04-05T21:51:30.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.266+0000 7fc4504b9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480ffa40 0x7fc448196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:30.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.266+0000 7fc4504b9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc448105a40 0x7fc448196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:30.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.266+0000 7fc4504b9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4481971a0 con 0x7fc4480ffa40 2024-04-05T21:51:30.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.266+0000 7fc4504b9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4481972e0 con 0x7fc448105a40 2024-04-05T21:51:30.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.266+0000 7fc44e255700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480ffa40 0x7fc448196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:30.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.266+0000 7fc44e255700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480ffa40 0x7fc448196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55106/0 (socket says 172.21.15.121:55106) 2024-04-05T21:51:30.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.266+0000 7fc44e255700 1 -- 172.21.15.121:0/840108487 learned_addr learned my addr 172.21.15.121:0/840108487 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:51:30.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.266+0000 7fc44da54700 1 --2- 172.21.15.121:0/840108487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc448105a40 0x7fc448196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:30.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.266+0000 7fc44e255700 1 -- 172.21.15.121:0/840108487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc448105a40 msgr2=0x7fc448196b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:30.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.266+0000 7fc44e255700 1 --2- 172.21.15.121:0/840108487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc448105a40 0x7fc448196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.266+0000 7fc44e255700 1 -- 172.21.15.121:0/840108487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4440095f0 con 0x7fc4480ffa40 2024-04-05T21:51:30.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.267+0000 7fc44e255700 1 --2- 172.21.15.121:0/840108487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480ffa40 0x7fc448196610 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c930 tx=0x7fc43800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:30.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.267+0000 7fc43f7fe700 1 -- 172.21.15.121:0/840108487 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc438007ad0 con 0x7fc4480ffa40 2024-04-05T21:51:30.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.267+0000 7fc43f7fe700 1 -- 172.21.15.121:0/840108487 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc438007c30 con 0x7fc4480ffa40 2024-04-05T21:51:30.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.267+0000 7fc4504b9700 1 -- 172.21.15.121:0/840108487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44819b130 con 0x7fc4480ffa40 2024-04-05T21:51:30.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.267+0000 7fc43f7fe700 1 -- 172.21.15.121:0/840108487 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc438018630 con 0x7fc4480ffa40 2024-04-05T21:51:30.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.267+0000 7fc4504b9700 1 -- 172.21.15.121:0/840108487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44819b680 con 0x7fc4480ffa40 2024-04-05T21:51:30.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.268+0000 7fc4504b9700 1 -- 172.21.15.121:0/840108487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc448108fe0 con 0x7fc4480ffa40 2024-04-05T21:51:30.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.268+0000 7fc43f7fe700 1 -- 172.21.15.121:0/840108487 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc43800f450 con 0x7fc4480ffa40 2024-04-05T21:51:30.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.269+0000 7fc43f7fe700 1 --2- 172.21.15.121:0/840108487 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc43406cd30 0x7fc43406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:30.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.269+0000 7fc43f7fe700 1 -- 172.21.15.121:0/840108487 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc43808c820 con 0x7fc4480ffa40 2024-04-05T21:51:30.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.269+0000 7fc44da54700 1 --2- 172.21.15.121:0/840108487 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc43406cd30 0x7fc43406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:30.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.269+0000 7fc44da54700 1 --2- 172.21.15.121:0/840108487 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc43406cd30 0x7fc43406f1f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc44400b5c0 tx=0x7fc444005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:30.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.273+0000 7fc43f7fe700 1 -- 172.21.15.121:0/840108487 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc438056920 con 0x7fc4480ffa40 2024-04-05T21:51:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.545+0000 7fc4504b9700 1 -- 172.21.15.121:0/840108487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc44802cc70 con 0x7fc4480ffa40 2024-04-05T21:51:30.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.546+0000 7fc43f7fe700 1 -- 172.21.15.121:0/840108487 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fc438059f40 con 0x7fc4480ffa40 2024-04-05T21:51:30.546 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:51:30.546 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:51:30.547 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:51:30.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:51:30.547 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:51:30.547 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:51:30.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:51:30.547 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:51:30.547 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:51:30.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:51:30.547 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:51:30.547 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:51:30.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:51:30.547 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:51:30.547 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:51:30.548 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:51:30.548 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:51:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.548+0000 7fc4504b9700 1 -- 172.21.15.121:0/840108487 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc43406cd30 msgr2=0x7fc43406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.548+0000 7fc4504b9700 1 --2- 172.21.15.121:0/840108487 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc43406cd30 0x7fc43406f1f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc44400b5c0 tx=0x7fc444005fb0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.548+0000 7fc4504b9700 1 -- 172.21.15.121:0/840108487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480ffa40 msgr2=0x7fc448196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.548+0000 7fc4504b9700 1 --2- 172.21.15.121:0/840108487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480ffa40 0x7fc448196610 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c930 tx=0x7fc43800ccf0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.548+0000 7fc4504b9700 1 -- 172.21.15.121:0/840108487 shutdown_connections 2024-04-05T21:51:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.548+0000 7fc4504b9700 1 --2- 172.21.15.121:0/840108487 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc43406cd30 0x7fc43406f1f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.548+0000 7fc4504b9700 1 --2- 172.21.15.121:0/840108487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480ffa40 0x7fc448196610 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.548+0000 7fc4504b9700 1 --2- 172.21.15.121:0/840108487 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc448105a40 0x7fc448196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.548+0000 7fc4504b9700 1 -- 172.21.15.121:0/840108487 >> 172.21.15.121:0/840108487 conn(0x7fc4480fb570 msgr2=0x7fc448068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.548+0000 7fc4504b9700 1 -- 172.21.15.121:0/840108487 shutdown_connections 2024-04-05T21:51:30.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.548+0000 7fc4504b9700 1 -- 172.21.15.121:0/840108487 wait complete. 2024-04-05T21:51:30.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.650+0000 7f16ebffd700 1 -- 172.21.15.121:0/4172489126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16e40ffbd0 msgr2=0x7f16e4100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:30.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.650+0000 7f16ebffd700 1 --2- 172.21.15.121:0/4172489126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16e40ffbd0 0x7f16e4100050 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009910 tx=0x7f16e0009c20 comp rx=0 tx=0).stop 2024-04-05T21:51:30.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.650+0000 7f16ebffd700 1 -- 172.21.15.121:0/4172489126 shutdown_connections 2024-04-05T21:51:30.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.650+0000 7f16ebffd700 1 --2- 172.21.15.121:0/4172489126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16e40ffbd0 0x7f16e4100050 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.650+0000 7f16ebffd700 1 --2- 172.21.15.121:0/4172489126 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16e4105a40 0x7f16e4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.650+0000 7f16ebffd700 1 -- 172.21.15.121:0/4172489126 >> 172.21.15.121:0/4172489126 conn(0x7f16e40755a0 msgr2=0x7f16e40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:30.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.650+0000 7f16ebffd700 1 -- 172.21.15.121:0/4172489126 shutdown_connections 2024-04-05T21:51:30.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.650+0000 7f16ebffd700 1 -- 172.21.15.121:0/4172489126 wait complete. 2024-04-05T21:51:30.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16ebffd700 1 Processor -- start 2024-04-05T21:51:30.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16ebffd700 1 -- start start 2024-04-05T21:51:30.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16ebffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16e40ffbd0 0x7f16e4101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:30.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16ebffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16e4105a40 0x7f16e4101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:30.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16ebffd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e4102090 con 0x7f16e4105a40 2024-04-05T21:51:30.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16ebffd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e41021d0 con 0x7f16e40ffbd0 2024-04-05T21:51:30.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16e9598700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16e4105a40 0x7f16e4101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:30.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16e9598700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16e4105a40 0x7f16e4101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55136/0 (socket says 172.21.15.121:55136) 2024-04-05T21:51:30.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16e9598700 1 -- 172.21.15.121:0/2610115438 learned_addr learned my addr 172.21.15.121:0/2610115438 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:51:30.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16e9d99700 1 --2- 172.21.15.121:0/2610115438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16e40ffbd0 0x7f16e4101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:30.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16e9598700 1 -- 172.21.15.121:0/2610115438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16e40ffbd0 msgr2=0x7f16e4101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:30.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16e9598700 1 --2- 172.21.15.121:0/2610115438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16e40ffbd0 0x7f16e4101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16e9598700 1 -- 172.21.15.121:0/2610115438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e00095f0 con 0x7f16e4105a40 2024-04-05T21:51:30.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16e9d99700 1 --2- 172.21.15.121:0/2610115438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16e40ffbd0 0x7f16e4101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:51:30.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16e9598700 1 --2- 172.21.15.121:0/2610115438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16e4105a40 0x7f16e4101b50 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f16e0006010 tx=0x7f16e000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:30.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.651+0000 7f16daffd700 1 -- 172.21.15.121:0/2610115438 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16e001d070 con 0x7f16e4105a40 2024-04-05T21:51:30.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.652+0000 7f16daffd700 1 -- 172.21.15.121:0/2610115438 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16e000f460 con 0x7f16e4105a40 2024-04-05T21:51:30.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.652+0000 7f16daffd700 1 -- 172.21.15.121:0/2610115438 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16e0021640 con 0x7f16e4105a40 2024-04-05T21:51:30.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.652+0000 7f16ebffd700 1 -- 172.21.15.121:0/2610115438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e4102450 con 0x7f16e4105a40 2024-04-05T21:51:30.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.652+0000 7f16ebffd700 1 -- 172.21.15.121:0/2610115438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e41a2bf0 con 0x7f16e4105a40 2024-04-05T21:51:30.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.653+0000 7f16ebffd700 1 -- 172.21.15.121:0/2610115438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e4108fe0 con 0x7f16e4105a40 2024-04-05T21:51:30.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.653+0000 7f16daffd700 1 -- 172.21.15.121:0/2610115438 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f16e000f9e0 con 0x7f16e4105a40 2024-04-05T21:51:30.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.653+0000 7f16daffd700 1 --2- 172.21.15.121:0/2610115438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f16d006cce0 0x7f16d006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:30.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.653+0000 7f16daffd700 1 -- 172.21.15.121:0/2610115438 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f16e008d900 con 0x7f16e4105a40 2024-04-05T21:51:30.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.653+0000 7f16e9d99700 1 --2- 172.21.15.121:0/2610115438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f16d006cce0 0x7f16d006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:30.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.654+0000 7f16e9d99700 1 --2- 172.21.15.121:0/2610115438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f16d006cce0 0x7f16d006f1a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005950 tx=0x7f16d4005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:30.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.658+0000 7f16daffd700 1 -- 172.21.15.121:0/2610115438 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f16e0057970 con 0x7f16e4105a40 2024-04-05T21:51:30.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.896+0000 7f16ebffd700 1 -- 172.21.15.121:0/2610115438 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f16e404ea90 con 0x7f16e4105a40 2024-04-05T21:51:30.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.896+0000 7f16daffd700 1 -- 172.21.15.121:0/2610115438 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f16e005af90 con 0x7f16e4105a40 2024-04-05T21:51:30.897 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:51:30.897 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:51:30.897 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:51:30.897 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:51:30.897 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:51:30.897 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:51:30.897 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:51:30.897 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:51:30.898 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:51:30.898 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:51:30.898 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:51:30.898 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:51:30.898 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:51:30.898 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:51:30.898 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:51:30.898 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:51:30.898 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:51:30.898 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:51:30.898 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:51:30.898 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:51:30.898 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:51:30.899 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:51:30.899 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:51:30.899 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:51:30.899 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:51:30.899 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:51:30.899 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:51:30.899 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:51:30.899 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:51:30.899 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:51:30.899 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:51:30.899 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:51:30.899 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:51:30.899 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:51:30.900 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:51:30.900 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:51:30.900 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:51:30.900 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:51:30.900 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:51:30.900 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:51:30.900 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:51:30.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.898+0000 7f16ebffd700 1 -- 172.21.15.121:0/2610115438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f16d006cce0 msgr2=0x7f16d006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:30.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.898+0000 7f16ebffd700 1 --2- 172.21.15.121:0/2610115438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f16d006cce0 0x7f16d006f1a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005950 tx=0x7f16d4005f50 comp rx=0 tx=0).stop 2024-04-05T21:51:30.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.898+0000 7f16ebffd700 1 -- 172.21.15.121:0/2610115438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16e4105a40 msgr2=0x7f16e4101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:30.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.898+0000 7f16ebffd700 1 --2- 172.21.15.121:0/2610115438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16e4105a40 0x7f16e4101b50 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f16e0006010 tx=0x7f16e000be20 comp rx=0 tx=0).stop 2024-04-05T21:51:30.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.898+0000 7f16ebffd700 1 -- 172.21.15.121:0/2610115438 shutdown_connections 2024-04-05T21:51:30.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.898+0000 7f16ebffd700 1 --2- 172.21.15.121:0/2610115438 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f16d006cce0 0x7f16d006f1a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.898+0000 7f16ebffd700 1 --2- 172.21.15.121:0/2610115438 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16e40ffbd0 0x7f16e4101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.898+0000 7f16ebffd700 1 --2- 172.21.15.121:0/2610115438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16e4105a40 0x7f16e4101b50 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:30.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.898+0000 7f16ebffd700 1 -- 172.21.15.121:0/2610115438 >> 172.21.15.121:0/2610115438 conn(0x7f16e40755a0 msgr2=0x7f16e4103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:30.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.899+0000 7f16ebffd700 1 -- 172.21.15.121:0/2610115438 shutdown_connections 2024-04-05T21:51:30.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:30.899+0000 7f16ebffd700 1 -- 172.21.15.121:0/2610115438 wait complete. 2024-04-05T21:51:30.901 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:51:31.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.001+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/1516135302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fc40ffdb0 msgr2=0x7f4fc4103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:31.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.001+0000 7f4fcc7c8700 1 --2- 172.21.15.121:0/1516135302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fc40ffdb0 0x7f4fc4103e00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0009910 tx=0x7f4fc0009c20 comp rx=0 tx=0).stop 2024-04-05T21:51:31.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.001+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/1516135302 shutdown_connections 2024-04-05T21:51:31.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.001+0000 7f4fcc7c8700 1 --2- 172.21.15.121:0/1516135302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fc40ffdb0 0x7f4fc4103e00 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.001+0000 7f4fcc7c8700 1 --2- 172.21.15.121:0/1516135302 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fc40ff400 0x7f4fc40ff7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.001+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/1516135302 >> 172.21.15.121:0/1516135302 conn(0x7f4fc4075e30 msgr2=0x7f4fc40fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:31.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.001+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/1516135302 shutdown_connections 2024-04-05T21:51:31.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.001+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/1516135302 wait complete. 2024-04-05T21:51:31.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.002+0000 7f4fcc7c8700 1 Processor -- start 2024-04-05T21:51:31.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.002+0000 7f4fcc7c8700 1 -- start start 2024-04-05T21:51:31.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.002+0000 7f4fcc7c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fc40ff400 0x7f4fc4197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:31.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.002+0000 7f4fcc7c8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fc40ffdb0 0x7f4fc4197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.002+0000 7f4fcc7c8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc4197ff0 con 0x7f4fc40ff400 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.002+0000 7f4fcc7c8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc4198130 con 0x7f4fc40ffdb0 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.002+0000 7f4fca564700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fc40ff400 0x7f4fc4197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.002+0000 7f4fca564700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fc40ff400 0x7f4fc4197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55144/0 (socket says 172.21.15.121:55144) 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.002+0000 7f4fca564700 1 -- 172.21.15.121:0/3919892265 learned_addr learned my addr 172.21.15.121:0/3919892265 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.003+0000 7f4fc9d63700 1 --2- 172.21.15.121:0/3919892265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fc40ffdb0 0x7f4fc4197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.003+0000 7f4fca564700 1 -- 172.21.15.121:0/3919892265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fc40ffdb0 msgr2=0x7f4fc4197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.003+0000 7f4fca564700 1 --2- 172.21.15.121:0/3919892265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fc40ffdb0 0x7f4fc4197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.003+0000 7f4fca564700 1 -- 172.21.15.121:0/3919892265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc00095f0 con 0x7f4fc40ff400 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.003+0000 7f4fc9d63700 1 --2- 172.21.15.121:0/3919892265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fc40ffdb0 0x7f4fc4197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.003+0000 7f4fca564700 1 --2- 172.21.15.121:0/3919892265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fc40ff400 0x7f4fc4197520 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400c930 tx=0x7f4fb400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.003+0000 7f4fbb7fe700 1 -- 172.21.15.121:0/3919892265 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fb4007ad0 con 0x7f4fc40ff400 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.003+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/3919892265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc41a0590 con 0x7f4fc40ff400 2024-04-05T21:51:31.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.003+0000 7f4fbb7fe700 1 -- 172.21.15.121:0/3919892265 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4fb4007c30 con 0x7f4fc40ff400 2024-04-05T21:51:31.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.003+0000 7f4fbb7fe700 1 -- 172.21.15.121:0/3919892265 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fb4018630 con 0x7f4fc40ff400 2024-04-05T21:51:31.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.003+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/3919892265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc41a09a0 con 0x7f4fc40ff400 2024-04-05T21:51:31.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.004+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/3919892265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc404ea90 con 0x7f4fc40ff400 2024-04-05T21:51:31.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.004+0000 7f4fbb7fe700 1 -- 172.21.15.121:0/3919892265 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4fb400f450 con 0x7f4fc40ff400 2024-04-05T21:51:31.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.005+0000 7f4fbb7fe700 1 --2- 172.21.15.121:0/3919892265 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4fb006cce0 0x7f4fb006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:31.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.005+0000 7f4fbb7fe700 1 -- 172.21.15.121:0/3919892265 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4fb408c7d0 con 0x7f4fc40ff400 2024-04-05T21:51:31.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.005+0000 7f4fc9d63700 1 --2- 172.21.15.121:0/3919892265 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4fb006cce0 0x7f4fb006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:31.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.005+0000 7f4fc9d63700 1 --2- 172.21.15.121:0/3919892265 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4fb006cce0 0x7f4fb006f1a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0006010 tx=0x7f4fc001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:31.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.009+0000 7f4fbb7fe700 1 -- 172.21.15.121:0/3919892265 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4fb40568f0 con 0x7f4fc40ff400 2024-04-05T21:51:31.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.219+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/3919892265 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fc40fd8f0 con 0x7f4fb006cce0 2024-04-05T21:51:31.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.220+0000 7f4fbb7fe700 1 -- 172.21.15.121:0/3919892265 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4fc40fd8f0 con 0x7f4fb006cce0 2024-04-05T21:51:31.220 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:51:31.220 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:51:31.220 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:51:31.220 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:51:31.220 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:51:31.220 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:51:31.221 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:51:31.221 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:51:31.221 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:51:31.221 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:51:31.221 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:51:31.221 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:51:31.221 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:51:31.221 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:51:31.221 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:51:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.222+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/3919892265 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4fb006cce0 msgr2=0x7f4fb006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.222+0000 7f4fcc7c8700 1 --2- 172.21.15.121:0/3919892265 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4fb006cce0 0x7f4fb006f1a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0006010 tx=0x7f4fc001a040 comp rx=0 tx=0).stop 2024-04-05T21:51:31.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.222+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/3919892265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fc40ff400 msgr2=0x7f4fc4197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.222+0000 7f4fcc7c8700 1 --2- 172.21.15.121:0/3919892265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fc40ff400 0x7f4fc4197520 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400c930 tx=0x7f4fb400ccf0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.222+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/3919892265 shutdown_connections 2024-04-05T21:51:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.222+0000 7f4fcc7c8700 1 --2- 172.21.15.121:0/3919892265 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4fb006cce0 0x7f4fb006f1a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.222+0000 7f4fcc7c8700 1 --2- 172.21.15.121:0/3919892265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fc40ff400 0x7f4fc4197520 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.222+0000 7f4fcc7c8700 1 --2- 172.21.15.121:0/3919892265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4fc40ffdb0 0x7f4fc4197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.222+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/3919892265 >> 172.21.15.121:0/3919892265 conn(0x7f4fc4075e30 msgr2=0x7f4fc40fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:31.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.222+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/3919892265 shutdown_connections 2024-04-05T21:51:31.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.222+0000 7f4fcc7c8700 1 -- 172.21.15.121:0/3919892265 wait complete. 2024-04-05T21:51:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:30 smithi121 ceph-mon[145859]: from='client.35214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:51:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:30 smithi121 ceph-mon[145859]: pgmap v759: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:31.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:30 smithi121 ceph-mon[145859]: from='client.35218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:51:31.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:51:31.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:30 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/840108487' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:51:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:30 smithi188 ceph-mon[107793]: from='client.35214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:51:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:30 smithi188 ceph-mon[107793]: pgmap v759: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:30 smithi188 ceph-mon[107793]: from='client.35218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:51:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:51:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:30 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/840108487' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:51:31.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.326+0000 7f133b92e700 1 -- 172.21.15.121:0/34567638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13340ffbd0 msgr2=0x7f1334100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:31.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.326+0000 7f133b92e700 1 --2- 172.21.15.121:0/34567638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13340ffbd0 0x7f1334100050 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1330009910 tx=0x7f1330009c20 comp rx=0 tx=0).stop 2024-04-05T21:51:31.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.326+0000 7f133b92e700 1 -- 172.21.15.121:0/34567638 shutdown_connections 2024-04-05T21:51:31.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.326+0000 7f133b92e700 1 --2- 172.21.15.121:0/34567638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13340ffbd0 0x7f1334100050 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.326+0000 7f133b92e700 1 --2- 172.21.15.121:0/34567638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1334105a40 0x7f1334105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.326+0000 7f133b92e700 1 -- 172.21.15.121:0/34567638 >> 172.21.15.121:0/34567638 conn(0x7f13340755a0 msgr2=0x7f13340759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:31.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.326+0000 7f133b92e700 1 -- 172.21.15.121:0/34567638 shutdown_connections 2024-04-05T21:51:31.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.326+0000 7f133b92e700 1 -- 172.21.15.121:0/34567638 wait complete. 2024-04-05T21:51:31.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f133b92e700 1 Processor -- start 2024-04-05T21:51:31.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f133b92e700 1 -- start start 2024-04-05T21:51:31.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f133b92e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13340ffbd0 0x7f1334195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:31.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f133b92e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1334105a40 0x7f1334195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:31.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f133b92e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1334196160 con 0x7f1334105a40 2024-04-05T21:51:31.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f133b92e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1334199ef0 con 0x7f13340ffbd0 2024-04-05T21:51:31.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f1338ec9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1334105a40 0x7f1334195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:31.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f1338ec9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1334105a40 0x7f1334195a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55152/0 (socket says 172.21.15.121:55152) 2024-04-05T21:51:31.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f1338ec9700 1 -- 172.21.15.121:0/3849804954 learned_addr learned my addr 172.21.15.121:0/3849804954 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:51:31.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f13396ca700 1 --2- 172.21.15.121:0/3849804954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13340ffbd0 0x7f1334195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:31.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f1338ec9700 1 -- 172.21.15.121:0/3849804954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13340ffbd0 msgr2=0x7f1334195540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:31.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f1338ec9700 1 --2- 172.21.15.121:0/3849804954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13340ffbd0 0x7f1334195540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f1338ec9700 1 -- 172.21.15.121:0/3849804954 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13300095f0 con 0x7f1334105a40 2024-04-05T21:51:31.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.327+0000 7f13396ca700 1 --2- 172.21.15.121:0/3849804954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13340ffbd0 0x7f1334195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:51:31.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.328+0000 7f1338ec9700 1 --2- 172.21.15.121:0/3849804954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1334105a40 0x7f1334195a80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1330006010 tx=0x7f133000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:31.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.328+0000 7f13267fc700 1 -- 172.21.15.121:0/3849804954 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f133001d070 con 0x7f1334105a40 2024-04-05T21:51:31.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.328+0000 7f13267fc700 1 -- 172.21.15.121:0/3849804954 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f133000f460 con 0x7f1334105a40 2024-04-05T21:51:31.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.328+0000 7f133b92e700 1 -- 172.21.15.121:0/3849804954 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133419a170 con 0x7f1334105a40 2024-04-05T21:51:31.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.328+0000 7f13267fc700 1 -- 172.21.15.121:0/3849804954 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1330021620 con 0x7f1334105a40 2024-04-05T21:51:31.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.328+0000 7f133b92e700 1 -- 172.21.15.121:0/3849804954 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133419a660 con 0x7f1334105a40 2024-04-05T21:51:31.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.329+0000 7f133b92e700 1 -- 172.21.15.121:0/3849804954 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1334107fd0 con 0x7f1334105a40 2024-04-05T21:51:31.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.329+0000 7f13267fc700 1 -- 172.21.15.121:0/3849804954 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f133000fa40 con 0x7f1334105a40 2024-04-05T21:51:31.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.330+0000 7f13267fc700 1 --2- 172.21.15.121:0/3849804954 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f132006cce0 0x7f132006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:51:31.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.330+0000 7f13267fc700 1 -- 172.21.15.121:0/3849804954 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f133008d760 con 0x7f1334105a40 2024-04-05T21:51:31.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.330+0000 7f13396ca700 1 --2- 172.21.15.121:0/3849804954 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f132006cce0 0x7f132006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:51:31.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.330+0000 7f13396ca700 1 --2- 172.21.15.121:0/3849804954 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f132006cce0 0x7f132006f1a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1328005950 tx=0x7f13280058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:51:31.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.334+0000 7f13267fc700 1 -- 172.21.15.121:0/3849804954 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f13300577b0 con 0x7f1334105a40 2024-04-05T21:51:31.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.615+0000 7f133b92e700 1 -- 172.21.15.121:0/3849804954 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f133402cd60 con 0x7f1334105a40 2024-04-05T21:51:31.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.616+0000 7f13267fc700 1 -- 172.21.15.121:0/3849804954 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f1330026e10 con 0x7f1334105a40 2024-04-05T21:51:31.616 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:51:31.616 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:51:31.617 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:51:31.618 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:51:31.618 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:51:31.618 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:51:31.618 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:51:31.618 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:51:31.618 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:51:31.618 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:51:31.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.618+0000 7f133b92e700 1 -- 172.21.15.121:0/3849804954 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f132006cce0 msgr2=0x7f132006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:31.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.618+0000 7f133b92e700 1 --2- 172.21.15.121:0/3849804954 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f132006cce0 0x7f132006f1a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1328005950 tx=0x7f13280058e0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.618+0000 7f133b92e700 1 -- 172.21.15.121:0/3849804954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1334105a40 msgr2=0x7f1334195a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:51:31.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.618+0000 7f133b92e700 1 --2- 172.21.15.121:0/3849804954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1334105a40 0x7f1334195a80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1330006010 tx=0x7f133000be20 comp rx=0 tx=0).stop 2024-04-05T21:51:31.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.618+0000 7f133b92e700 1 -- 172.21.15.121:0/3849804954 shutdown_connections 2024-04-05T21:51:31.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.618+0000 7f133b92e700 1 --2- 172.21.15.121:0/3849804954 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f132006cce0 0x7f132006f1a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.618+0000 7f133b92e700 1 --2- 172.21.15.121:0/3849804954 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13340ffbd0 0x7f1334195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.618+0000 7f133b92e700 1 --2- 172.21.15.121:0/3849804954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1334105a40 0x7f1334195a80 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:51:31.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.618+0000 7f133b92e700 1 -- 172.21.15.121:0/3849804954 >> 172.21.15.121:0/3849804954 conn(0x7f13340755a0 msgr2=0x7f13340fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:51:31.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.618+0000 7f133b92e700 1 -- 172.21.15.121:0/3849804954 shutdown_connections 2024-04-05T21:51:31.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:51:31.618+0000 7f133b92e700 1 -- 172.21.15.121:0/3849804954 wait complete. 2024-04-05T21:51:32.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:31 smithi121 ceph-mon[145859]: from='client.35222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:51:32.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:31 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2610115438' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:51:32.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:31 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3849804954' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:51:32.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:31 smithi188 ceph-mon[107793]: from='client.35222 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:51:32.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:31 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2610115438' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:51:32.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:31 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3849804954' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:51:33.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:32 smithi121 ceph-mon[145859]: from='client.35234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:51:33.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:32 smithi121 ceph-mon[145859]: pgmap v760: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:33.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:32 smithi188 ceph-mon[107793]: from='client.35234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:51:33.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:32 smithi188 ceph-mon[107793]: pgmap v760: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:35.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:34 smithi121 ceph-mon[145859]: pgmap v761: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:35.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:34 smithi188 ceph-mon[107793]: pgmap v761: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:37.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:36 smithi121 ceph-mon[145859]: pgmap v762: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:36 smithi188 ceph-mon[107793]: pgmap v762: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:39.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:38 smithi121 ceph-mon[145859]: pgmap v763: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:38 smithi188 ceph-mon[107793]: pgmap v763: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:40 smithi121 ceph-mon[145859]: pgmap v764: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:40 smithi188 ceph-mon[107793]: pgmap v764: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:43.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:42 smithi121 ceph-mon[145859]: pgmap v765: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:42 smithi188 ceph-mon[107793]: pgmap v765: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:45.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:44 smithi121 ceph-mon[145859]: pgmap v766: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:44 smithi188 ceph-mon[107793]: pgmap v766: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:51:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:51:47.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:46 smithi121 ceph-mon[145859]: pgmap v767: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:46 smithi188 ceph-mon[107793]: pgmap v767: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:49.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:48 smithi121 ceph-mon[145859]: pgmap v768: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:48 smithi188 ceph-mon[107793]: pgmap v768: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:51.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:50 smithi121 ceph-mon[145859]: pgmap v769: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:50 smithi188 ceph-mon[107793]: pgmap v769: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:53.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:52 smithi121 ceph-mon[145859]: pgmap v770: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:52 smithi188 ceph-mon[107793]: pgmap v770: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:55.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:54 smithi121 ceph-mon[145859]: pgmap v771: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:54 smithi188 ceph-mon[107793]: pgmap v771: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:56 smithi121 ceph-mon[145859]: pgmap v772: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:56 smithi188 ceph-mon[107793]: pgmap v772: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:59.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:51:58 smithi121 ceph-mon[145859]: pgmap v773: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:51:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:51:58 smithi188 ceph-mon[107793]: pgmap v773: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:00.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:00 smithi121 ceph-mon[145859]: pgmap v774: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:00.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:52:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:00 smithi188 ceph-mon[107793]: pgmap v774: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:52:01.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.735+0000 7fac3e760700 1 -- 172.21.15.121:0/73208106 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac380ffbd0 msgr2=0x7fac38100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:01.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.735+0000 7fac3e760700 1 --2- 172.21.15.121:0/73208106 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac380ffbd0 0x7fac38100050 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fac28009910 tx=0x7fac28009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:01.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.735+0000 7fac3e760700 1 -- 172.21.15.121:0/73208106 shutdown_connections 2024-04-05T21:52:01.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.735+0000 7fac3e760700 1 --2- 172.21.15.121:0/73208106 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac380ffbd0 0x7fac38100050 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:01.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.735+0000 7fac3e760700 1 --2- 172.21.15.121:0/73208106 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac38105a40 0x7fac38105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:01.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.735+0000 7fac3e760700 1 -- 172.21.15.121:0/73208106 >> 172.21.15.121:0/73208106 conn(0x7fac380755a0 msgr2=0x7fac380759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:01.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.735+0000 7fac3e760700 1 -- 172.21.15.121:0/73208106 shutdown_connections 2024-04-05T21:52:01.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.735+0000 7fac3e760700 1 -- 172.21.15.121:0/73208106 wait complete. 2024-04-05T21:52:01.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.735+0000 7fac3e760700 1 Processor -- start 2024-04-05T21:52:01.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.736+0000 7fac3e760700 1 -- start start 2024-04-05T21:52:01.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.736+0000 7fac3e760700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac380ffbd0 0x7fac38195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:01.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.736+0000 7fac3e760700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac38105a40 0x7fac38195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:01.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.736+0000 7fac3e760700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac381960d0 con 0x7fac380ffbd0 2024-04-05T21:52:01.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.736+0000 7fac3e760700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac38196210 con 0x7fac38105a40 2024-04-05T21:52:01.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.736+0000 7fac37fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac380ffbd0 0x7fac38195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:01.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.736+0000 7fac37fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac380ffbd0 0x7fac38195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36334/0 (socket says 172.21.15.121:36334) 2024-04-05T21:52:01.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.736+0000 7fac37fff700 1 -- 172.21.15.121:0/3743904817 learned_addr learned my addr 172.21.15.121:0/3743904817 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:01.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.736+0000 7fac377fe700 1 --2- 172.21.15.121:0/3743904817 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac38105a40 0x7fac38195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:01.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.736+0000 7fac37fff700 1 -- 172.21.15.121:0/3743904817 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac38105a40 msgr2=0x7fac38195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:01.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.736+0000 7fac37fff700 1 --2- 172.21.15.121:0/3743904817 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac38105a40 0x7fac38195a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:01.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.736+0000 7fac37fff700 1 -- 172.21.15.121:0/3743904817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac280095f0 con 0x7fac380ffbd0 2024-04-05T21:52:01.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.737+0000 7fac37fff700 1 --2- 172.21.15.121:0/3743904817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac380ffbd0 0x7fac38195540 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fac2000c930 tx=0x7fac2000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:01.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.737+0000 7fac357fa700 1 -- 172.21.15.121:0/3743904817 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac20007ad0 con 0x7fac380ffbd0 2024-04-05T21:52:01.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.737+0000 7fac357fa700 1 -- 172.21.15.121:0/3743904817 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac20007c30 con 0x7fac380ffbd0 2024-04-05T21:52:01.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.737+0000 7fac357fa700 1 -- 172.21.15.121:0/3743904817 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac20018630 con 0x7fac380ffbd0 2024-04-05T21:52:01.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.737+0000 7fac3e760700 1 -- 172.21.15.121:0/3743904817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac3819a060 con 0x7fac380ffbd0 2024-04-05T21:52:01.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.737+0000 7fac3e760700 1 -- 172.21.15.121:0/3743904817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac3819a5b0 con 0x7fac380ffbd0 2024-04-05T21:52:01.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.738+0000 7fac3e760700 1 -- 172.21.15.121:0/3743904817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac38107fd0 con 0x7fac380ffbd0 2024-04-05T21:52:01.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.738+0000 7fac357fa700 1 -- 172.21.15.121:0/3743904817 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fac2000f450 con 0x7fac380ffbd0 2024-04-05T21:52:01.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.739+0000 7fac357fa700 1 --2- 172.21.15.121:0/3743904817 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fac2406cce0 0x7fac2406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:01.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.739+0000 7fac357fa700 1 -- 172.21.15.121:0/3743904817 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fac2008c780 con 0x7fac380ffbd0 2024-04-05T21:52:01.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.739+0000 7fac377fe700 1 --2- 172.21.15.121:0/3743904817 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fac2406cce0 0x7fac2406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:01.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.739+0000 7fac377fe700 1 --2- 172.21.15.121:0/3743904817 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fac2406cce0 0x7fac2406f1a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fac38196b60 tx=0x7fac2801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:01.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.744+0000 7fac357fa700 1 -- 172.21.15.121:0/3743904817 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fac20056920 con 0x7fac380ffbd0 2024-04-05T21:52:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.959+0000 7fac3e760700 1 -- 172.21.15.121:0/3743904817 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fac3802cd80 con 0x7fac2406cce0 2024-04-05T21:52:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.960+0000 7fac357fa700 1 -- 172.21.15.121:0/3743904817 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fac3802cd80 con 0x7fac2406cce0 2024-04-05T21:52:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.962+0000 7fac3e760700 1 -- 172.21.15.121:0/3743904817 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fac2406cce0 msgr2=0x7fac2406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.962+0000 7fac3e760700 1 --2- 172.21.15.121:0/3743904817 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fac2406cce0 0x7fac2406f1a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fac38196b60 tx=0x7fac2801a040 comp rx=0 tx=0).stop 2024-04-05T21:52:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.962+0000 7fac3e760700 1 -- 172.21.15.121:0/3743904817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac380ffbd0 msgr2=0x7fac38195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.962+0000 7fac3e760700 1 --2- 172.21.15.121:0/3743904817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac380ffbd0 0x7fac38195540 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fac2000c930 tx=0x7fac2000ccf0 comp rx=0 tx=0).stop 2024-04-05T21:52:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.962+0000 7fac3e760700 1 -- 172.21.15.121:0/3743904817 shutdown_connections 2024-04-05T21:52:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.962+0000 7fac3e760700 1 --2- 172.21.15.121:0/3743904817 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fac2406cce0 0x7fac2406f1a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.962+0000 7fac3e760700 1 --2- 172.21.15.121:0/3743904817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac380ffbd0 0x7fac38195540 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.962+0000 7fac3e760700 1 --2- 172.21.15.121:0/3743904817 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac38105a40 0x7fac38195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.962+0000 7fac3e760700 1 -- 172.21.15.121:0/3743904817 >> 172.21.15.121:0/3743904817 conn(0x7fac380755a0 msgr2=0x7fac380fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.962+0000 7fac3e760700 1 -- 172.21.15.121:0/3743904817 shutdown_connections 2024-04-05T21:52:01.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:01.962+0000 7fac3e760700 1 -- 172.21.15.121:0/3743904817 wait complete. 2024-04-05T21:52:01.973 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:52:02.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.065+0000 7f325956c700 1 -- 172.21.15.121:0/4195338268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3254100dc0 msgr2=0x7f32541011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:02.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.065+0000 7f325956c700 1 --2- 172.21.15.121:0/4195338268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3254100dc0 0x7f32541011a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3248009910 tx=0x7f3248009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:02.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.065+0000 7f325956c700 1 -- 172.21.15.121:0/4195338268 shutdown_connections 2024-04-05T21:52:02.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.065+0000 7f325956c700 1 --2- 172.21.15.121:0/4195338268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3254101770 0x7f32541057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.065+0000 7f325956c700 1 --2- 172.21.15.121:0/4195338268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3254100dc0 0x7f32541011a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.065+0000 7f325956c700 1 -- 172.21.15.121:0/4195338268 >> 172.21.15.121:0/4195338268 conn(0x7f32540fc670 msgr2=0x7f32540fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:02.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.065+0000 7f325956c700 1 -- 172.21.15.121:0/4195338268 shutdown_connections 2024-04-05T21:52:02.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.065+0000 7f325956c700 1 -- 172.21.15.121:0/4195338268 wait complete. 2024-04-05T21:52:02.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.065+0000 7f325956c700 1 Processor -- start 2024-04-05T21:52:02.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f325956c700 1 -- start start 2024-04-05T21:52:02.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f325956c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3254100dc0 0x7f3254073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:02.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f325956c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3254101770 0x7f32540735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:02.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f325956c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3254073b10 con 0x7f3254100dc0 2024-04-05T21:52:02.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f325956c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3254073c50 con 0x7f3254101770 2024-04-05T21:52:02.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f3252ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3254100dc0 0x7f3254073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:02.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f3252ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3254100dc0 0x7f3254073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36360/0 (socket says 172.21.15.121:36360) 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f3252ffd700 1 -- 172.21.15.121:0/712212977 learned_addr learned my addr 172.21.15.121:0/712212977 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f32527fc700 1 --2- 172.21.15.121:0/712212977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3254101770 0x7f32540735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f3252ffd700 1 -- 172.21.15.121:0/712212977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3254101770 msgr2=0x7f32540735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f3252ffd700 1 --2- 172.21.15.121:0/712212977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3254101770 0x7f32540735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f3252ffd700 1 -- 172.21.15.121:0/712212977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32480095f0 con 0x7f3254100dc0 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f32527fc700 1 --2- 172.21.15.121:0/712212977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3254101770 0x7f32540735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.066+0000 7f3252ffd700 1 --2- 172.21.15.121:0/712212977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3254100dc0 0x7f3254073090 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3248006010 tx=0x7f324800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.067+0000 7f323bfff700 1 -- 172.21.15.121:0/712212977 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f324801d070 con 0x7f3254100dc0 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.067+0000 7f323bfff700 1 -- 172.21.15.121:0/712212977 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f324800f460 con 0x7f3254100dc0 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.067+0000 7f323bfff700 1 -- 172.21.15.121:0/712212977 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3248021620 con 0x7f3254100dc0 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.067+0000 7f325956c700 1 -- 172.21.15.121:0/712212977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3254073ed0 con 0x7f3254100dc0 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.067+0000 7f325956c700 1 -- 172.21.15.121:0/712212977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32541a2c10 con 0x7f3254100dc0 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.068+0000 7f325956c700 1 -- 172.21.15.121:0/712212977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3254109160 con 0x7f3254100dc0 2024-04-05T21:52:02.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.068+0000 7f323bfff700 1 -- 172.21.15.121:0/712212977 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3248003a60 con 0x7f3254100dc0 2024-04-05T21:52:02.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.068+0000 7f323bfff700 1 --2- 172.21.15.121:0/712212977 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f323c06cce0 0x7f323c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:02.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.068+0000 7f323bfff700 1 -- 172.21.15.121:0/712212977 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f324808d710 con 0x7f3254100dc0 2024-04-05T21:52:02.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.068+0000 7f32527fc700 1 --2- 172.21.15.121:0/712212977 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f323c06cce0 0x7f323c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:02.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.069+0000 7f32527fc700 1 --2- 172.21.15.121:0/712212977 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f323c06cce0 0x7f323c06f1a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3244005950 tx=0x7f32440058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:02.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.073+0000 7f323bfff700 1 -- 172.21.15.121:0/712212977 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3248057800 con 0x7f3254100dc0 2024-04-05T21:52:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.282+0000 7f325956c700 1 -- 172.21.15.121:0/712212977 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f325402cce0 con 0x7f323c06cce0 2024-04-05T21:52:02.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.284+0000 7f323bfff700 1 -- 172.21.15.121:0/712212977 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f325402cce0 con 0x7f323c06cce0 2024-04-05T21:52:02.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.286+0000 7f325956c700 1 -- 172.21.15.121:0/712212977 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f323c06cce0 msgr2=0x7f323c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:02.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.286+0000 7f325956c700 1 --2- 172.21.15.121:0/712212977 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f323c06cce0 0x7f323c06f1a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3244005950 tx=0x7f32440058e0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.286+0000 7f325956c700 1 -- 172.21.15.121:0/712212977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3254100dc0 msgr2=0x7f3254073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:02.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.286+0000 7f325956c700 1 --2- 172.21.15.121:0/712212977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3254100dc0 0x7f3254073090 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3248006010 tx=0x7f324800be20 comp rx=0 tx=0).stop 2024-04-05T21:52:02.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.286+0000 7f325956c700 1 -- 172.21.15.121:0/712212977 shutdown_connections 2024-04-05T21:52:02.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.286+0000 7f325956c700 1 --2- 172.21.15.121:0/712212977 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f323c06cce0 0x7f323c06f1a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.286+0000 7f325956c700 1 --2- 172.21.15.121:0/712212977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3254100dc0 0x7f3254073090 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.286+0000 7f325956c700 1 --2- 172.21.15.121:0/712212977 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3254101770 0x7f32540735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.286+0000 7f325956c700 1 -- 172.21.15.121:0/712212977 >> 172.21.15.121:0/712212977 conn(0x7f32540fc670 msgr2=0x7f32540fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:02.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.286+0000 7f325956c700 1 -- 172.21.15.121:0/712212977 shutdown_connections 2024-04-05T21:52:02.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.286+0000 7f325956c700 1 -- 172.21.15.121:0/712212977 wait complete. 2024-04-05T21:52:02.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.390+0000 7fbe031b9700 1 -- 172.21.15.121:0/343884871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdfc105b50 msgr2=0x7fbdfc105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:02.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.391+0000 7fbe031b9700 1 --2- 172.21.15.121:0/343884871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdfc105b50 0x7fbdfc105f30 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0009910 tx=0x7fbdf0009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:02.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.391+0000 7fbe031b9700 1 -- 172.21.15.121:0/343884871 shutdown_connections 2024-04-05T21:52:02.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.391+0000 7fbe031b9700 1 --2- 172.21.15.121:0/343884871 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbdfc0fef70 0x7fbdfc0ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.391+0000 7fbe031b9700 1 --2- 172.21.15.121:0/343884871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdfc105b50 0x7fbdfc105f30 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.391+0000 7fbe031b9700 1 -- 172.21.15.121:0/343884871 >> 172.21.15.121:0/343884871 conn(0x7fbdfc075e30 msgr2=0x7fbdfc0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:02.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.391+0000 7fbe031b9700 1 -- 172.21.15.121:0/343884871 shutdown_connections 2024-04-05T21:52:02.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.391+0000 7fbe031b9700 1 -- 172.21.15.121:0/343884871 wait complete. 2024-04-05T21:52:02.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.391+0000 7fbe031b9700 1 Processor -- start 2024-04-05T21:52:02.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.391+0000 7fbe031b9700 1 -- start start 2024-04-05T21:52:02.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.391+0000 7fbe031b9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbdfc0fef70 0x7fbdfc100150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:02.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.391+0000 7fbe031b9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdfc105b50 0x7fbdfc100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:02.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.391+0000 7fbe031b9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc100bd0 con 0x7fbdfc105b50 2024-04-05T21:52:02.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.391+0000 7fbe031b9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc100d10 con 0x7fbdfc0fef70 2024-04-05T21:52:02.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.392+0000 7fbdfbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdfc105b50 0x7fbdfc100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:02.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.392+0000 7fbdfbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdfc105b50 0x7fbdfc100690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36366/0 (socket says 172.21.15.121:36366) 2024-04-05T21:52:02.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.392+0000 7fbdfbfff700 1 -- 172.21.15.121:0/368246060 learned_addr learned my addr 172.21.15.121:0/368246060 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:02.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.392+0000 7fbe00f55700 1 --2- 172.21.15.121:0/368246060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbdfc0fef70 0x7fbdfc100150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:02.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.392+0000 7fbdfbfff700 1 -- 172.21.15.121:0/368246060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbdfc0fef70 msgr2=0x7fbdfc100150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:02.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.392+0000 7fbdfbfff700 1 --2- 172.21.15.121:0/368246060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbdfc0fef70 0x7fbdfc100150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.392+0000 7fbdfbfff700 1 -- 172.21.15.121:0/368246060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf00095f0 con 0x7fbdfc105b50 2024-04-05T21:52:02.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.392+0000 7fbdfbfff700 1 --2- 172.21.15.121:0/368246060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdfc105b50 0x7fbdfc100690 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbde800c8a0 tx=0x7fbde800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:02.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.392+0000 7fbdf9ffb700 1 -- 172.21.15.121:0/368246060 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbde80078e0 con 0x7fbdfc105b50 2024-04-05T21:52:02.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.392+0000 7fbdf9ffb700 1 -- 172.21.15.121:0/368246060 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbde800f450 con 0x7fbdfc105b50 2024-04-05T21:52:02.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.392+0000 7fbe031b9700 1 -- 172.21.15.121:0/368246060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdfc1a2910 con 0x7fbdfc105b50 2024-04-05T21:52:02.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.392+0000 7fbdf9ffb700 1 -- 172.21.15.121:0/368246060 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbde800e5e0 con 0x7fbdfc105b50 2024-04-05T21:52:02.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.392+0000 7fbe031b9700 1 -- 172.21.15.121:0/368246060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdfc1a2d20 con 0x7fbdfc105b50 2024-04-05T21:52:02.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.394+0000 7fbe031b9700 1 -- 172.21.15.121:0/368246060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdfc109120 con 0x7fbdfc105b50 2024-04-05T21:52:02.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.394+0000 7fbdf9ffb700 1 -- 172.21.15.121:0/368246060 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbde800f5c0 con 0x7fbdfc105b50 2024-04-05T21:52:02.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.394+0000 7fbdf9ffb700 1 --2- 172.21.15.121:0/368246060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbdec06cce0 0x7fbdec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:02.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.394+0000 7fbdf9ffb700 1 -- 172.21.15.121:0/368246060 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbde808c1d0 con 0x7fbdfc105b50 2024-04-05T21:52:02.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.394+0000 7fbe00f55700 1 --2- 172.21.15.121:0/368246060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbdec06cce0 0x7fbdec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:02.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.395+0000 7fbe00f55700 1 --2- 172.21.15.121:0/368246060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbdec06cce0 0x7fbdec06f1a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0006010 tx=0x7fbdf001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:02.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.399+0000 7fbdf9ffb700 1 -- 172.21.15.121:0/368246060 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbde8056240 con 0x7fbdfc105b50 2024-04-05T21:52:02.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.599+0000 7fbe031b9700 1 -- 172.21.15.121:0/368246060 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdfc02cd30 con 0x7fbdec06cce0 2024-04-05T21:52:02.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.609+0000 7fbdf9ffb700 1 -- 172.21.15.121:0/368246060 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fbdfc02cd30 con 0x7fbdec06cce0 2024-04-05T21:52:02.610 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:52:02.610 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (22m) 82s ago 34m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:52:02.610 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (34m) 82s ago 34m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:52:02.610 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (33m) 84s ago 33m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:52:02.610 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (21m) 82s ago 34m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:52:02.610 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (20m) 84s ago 33m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:52:02.610 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (22m) 82s ago 34m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:52:02.610 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (17m) 82s ago 28m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:52:02.610 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (16m) 82s ago 28m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:52:02.610 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (16m) 84s ago 28m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:52:02.610 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (88s) 84s ago 28m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:52:02.610 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (24m) 82s ago 35m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:52:02.610 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (24m) 84s ago 33m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:52:02.611 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (21m) 82s ago 36m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:52:02.611 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (21m) 84s ago 33m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:52:02.611 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (23m) 82s ago 34m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:52:02.611 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (23m) 84s ago 33m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:52:02.611 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (20m) 82s ago 32m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:52:02.611 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (19m) 82s ago 31m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:52:02.611 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (19m) 82s ago 31m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:52:02.611 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (18m) 84s ago 30m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:52:02.611 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (18m) 84s ago 30m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:52:02.611 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (17m) 84s ago 29m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:52:02.611 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (23m) 82s ago 33m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:52:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.611+0000 7fbe031b9700 1 -- 172.21.15.121:0/368246060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbdec06cce0 msgr2=0x7fbdec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.611+0000 7fbe031b9700 1 --2- 172.21.15.121:0/368246060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbdec06cce0 0x7fbdec06f1a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0006010 tx=0x7fbdf001a040 comp rx=0 tx=0).stop 2024-04-05T21:52:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.611+0000 7fbe031b9700 1 -- 172.21.15.121:0/368246060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdfc105b50 msgr2=0x7fbdfc100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.611+0000 7fbe031b9700 1 --2- 172.21.15.121:0/368246060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdfc105b50 0x7fbdfc100690 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbde800c8a0 tx=0x7fbde800cbb0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.611+0000 7fbe031b9700 1 -- 172.21.15.121:0/368246060 shutdown_connections 2024-04-05T21:52:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.611+0000 7fbe031b9700 1 --2- 172.21.15.121:0/368246060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbdec06cce0 0x7fbdec06f1a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.611+0000 7fbe031b9700 1 --2- 172.21.15.121:0/368246060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbdfc0fef70 0x7fbdfc100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.611+0000 7fbe031b9700 1 --2- 172.21.15.121:0/368246060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdfc105b50 0x7fbdfc100690 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.611+0000 7fbe031b9700 1 -- 172.21.15.121:0/368246060 >> 172.21.15.121:0/368246060 conn(0x7fbdfc075e30 msgr2=0x7fbdfc104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.611+0000 7fbe031b9700 1 -- 172.21.15.121:0/368246060 shutdown_connections 2024-04-05T21:52:02.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.611+0000 7fbe031b9700 1 -- 172.21.15.121:0/368246060 wait complete. 2024-04-05T21:52:02.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.713+0000 7f8942795700 1 -- 172.21.15.121:0/2679025999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f893c068670 msgr2=0x7f893c068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:02.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.713+0000 7f8942795700 1 --2- 172.21.15.121:0/2679025999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f893c068670 0x7f893c068a50 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f8930009910 tx=0x7f8930009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:02.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.713+0000 7f8942795700 1 -- 172.21.15.121:0/2679025999 shutdown_connections 2024-04-05T21:52:02.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.713+0000 7f8942795700 1 --2- 172.21.15.121:0/2679025999 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f893c069020 0x7f893c105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.713+0000 7f8942795700 1 --2- 172.21.15.121:0/2679025999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f893c068670 0x7f893c068a50 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.713+0000 7f8942795700 1 -- 172.21.15.121:0/2679025999 >> 172.21.15.121:0/2679025999 conn(0x7f893c0755a0 msgr2=0x7f893c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:02.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.713+0000 7f8942795700 1 -- 172.21.15.121:0/2679025999 shutdown_connections 2024-04-05T21:52:02.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.713+0000 7f8942795700 1 -- 172.21.15.121:0/2679025999 wait complete. 2024-04-05T21:52:02.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.713+0000 7f8942795700 1 Processor -- start 2024-04-05T21:52:02.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.713+0000 7f8942795700 1 -- start start 2024-04-05T21:52:02.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f8942795700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f893c068670 0x7f893c196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:02.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f8942795700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f893c069020 0x7f893c196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:02.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f8942795700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c197290 con 0x7f893c069020 2024-04-05T21:52:02.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f8942795700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c19b020 con 0x7f893c068670 2024-04-05T21:52:02.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f893c069020 0x7f893c196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:02.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f893c069020 0x7f893c196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36378/0 (socket says 172.21.15.121:36378) 2024-04-05T21:52:02.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f893b7fe700 1 -- 172.21.15.121:0/4029645257 learned_addr learned my addr 172.21.15.121:0/4029645257 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:02.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f893bfff700 1 --2- 172.21.15.121:0/4029645257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f893c068670 0x7f893c196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:02.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f893b7fe700 1 -- 172.21.15.121:0/4029645257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f893c068670 msgr2=0x7f893c196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:02.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f893b7fe700 1 --2- 172.21.15.121:0/4029645257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f893c068670 0x7f893c196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f893b7fe700 1 -- 172.21.15.121:0/4029645257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89300095f0 con 0x7f893c069020 2024-04-05T21:52:02.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f893b7fe700 1 --2- 172.21.15.121:0/4029645257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f893c069020 0x7f893c196bb0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f892c00e8f0 tx=0x7f892c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:02.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f89397fa700 1 -- 172.21.15.121:0/4029645257 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f892c009790 con 0x7f893c069020 2024-04-05T21:52:02.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f8942795700 1 -- 172.21.15.121:0/4029645257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893c19b300 con 0x7f893c069020 2024-04-05T21:52:02.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f89397fa700 1 -- 172.21.15.121:0/4029645257 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f892c004cd0 con 0x7f893c069020 2024-04-05T21:52:02.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f89397fa700 1 -- 172.21.15.121:0/4029645257 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f892c005580 con 0x7f893c069020 2024-04-05T21:52:02.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.714+0000 7f8942795700 1 -- 172.21.15.121:0/4029645257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893c19b850 con 0x7f893c069020 2024-04-05T21:52:02.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.716+0000 7f8942795700 1 -- 172.21.15.121:0/4029645257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893c109130 con 0x7f893c069020 2024-04-05T21:52:02.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.716+0000 7f89397fa700 1 -- 172.21.15.121:0/4029645257 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f892c004750 con 0x7f893c069020 2024-04-05T21:52:02.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.716+0000 7f89397fa700 1 --2- 172.21.15.121:0/4029645257 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f892406cce0 0x7f892406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:02.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.717+0000 7f893bfff700 1 --2- 172.21.15.121:0/4029645257 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f892406cce0 0x7f892406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:02.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.717+0000 7f89397fa700 1 -- 172.21.15.121:0/4029645257 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f892c014070 con 0x7f893c069020 2024-04-05T21:52:02.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.717+0000 7f893bfff700 1 --2- 172.21.15.121:0/4029645257 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f892406cce0 0x7f892406f1a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8930009fd0 tx=0x7f8930005940 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:02.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.721+0000 7f89397fa700 1 -- 172.21.15.121:0/4029645257 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f892c0569c0 con 0x7f893c069020 2024-04-05T21:52:02.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.994+0000 7f8942795700 1 -- 172.21.15.121:0/4029645257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f893c04ea90 con 0x7f893c069020 2024-04-05T21:52:02.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.994+0000 7f89397fa700 1 -- 172.21.15.121:0/4029645257 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f892c059fe0 con 0x7f893c069020 2024-04-05T21:52:02.995 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:52:02.995 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:52:02.995 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:52:02.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:52:02.995 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:52:02.995 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:52:02.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:52:02.995 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:52:02.996 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:52:02.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:52:02.996 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:52:02.996 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:52:02.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:52:02.996 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:52:02.996 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:52:02.996 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:52:02.996 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:52:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.996+0000 7f8942795700 1 -- 172.21.15.121:0/4029645257 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f892406cce0 msgr2=0x7f892406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.996+0000 7f8942795700 1 --2- 172.21.15.121:0/4029645257 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f892406cce0 0x7f892406f1a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8930009fd0 tx=0x7f8930005940 comp rx=0 tx=0).stop 2024-04-05T21:52:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.996+0000 7f8942795700 1 -- 172.21.15.121:0/4029645257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f893c069020 msgr2=0x7f893c196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.996+0000 7f8942795700 1 --2- 172.21.15.121:0/4029645257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f893c069020 0x7f893c196bb0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f892c00e8f0 tx=0x7f892c00ecb0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.997+0000 7f8942795700 1 -- 172.21.15.121:0/4029645257 shutdown_connections 2024-04-05T21:52:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.997+0000 7f8942795700 1 --2- 172.21.15.121:0/4029645257 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f892406cce0 0x7f892406f1a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.997+0000 7f8942795700 1 --2- 172.21.15.121:0/4029645257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f893c068670 0x7f893c196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.997+0000 7f8942795700 1 --2- 172.21.15.121:0/4029645257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f893c069020 0x7f893c196bb0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:02.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.997+0000 7f8942795700 1 -- 172.21.15.121:0/4029645257 >> 172.21.15.121:0/4029645257 conn(0x7f893c0755a0 msgr2=0x7f893c0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:02.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.997+0000 7f8942795700 1 -- 172.21.15.121:0/4029645257 shutdown_connections 2024-04-05T21:52:02.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:02.997+0000 7f8942795700 1 -- 172.21.15.121:0/4029645257 wait complete. 2024-04-05T21:52:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.098+0000 7f3816480700 1 -- 172.21.15.121:0/1240798949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38100fef70 msgr2=0x7f38100ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.098+0000 7f3816480700 1 --2- 172.21.15.121:0/1240798949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38100fef70 0x7f38100ff3f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3804009910 tx=0x7f3804009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:03.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.098+0000 7f3816480700 1 -- 172.21.15.121:0/1240798949 shutdown_connections 2024-04-05T21:52:03.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.098+0000 7f3816480700 1 --2- 172.21.15.121:0/1240798949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38100fef70 0x7f38100ff3f0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.098+0000 7f3816480700 1 --2- 172.21.15.121:0/1240798949 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3810105b50 0x7f3810105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.098+0000 7f3816480700 1 -- 172.21.15.121:0/1240798949 >> 172.21.15.121:0/1240798949 conn(0x7f3810075e30 msgr2=0x7f38100fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:03.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.098+0000 7f3816480700 1 -- 172.21.15.121:0/1240798949 shutdown_connections 2024-04-05T21:52:03.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.098+0000 7f3816480700 1 -- 172.21.15.121:0/1240798949 wait complete. 2024-04-05T21:52:03.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.099+0000 7f3816480700 1 Processor -- start 2024-04-05T21:52:03.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.099+0000 7f3816480700 1 -- start start 2024-04-05T21:52:03.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.099+0000 7f3816480700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38100fef70 0x7f381019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.099+0000 7f3816480700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3810105b50 0x7f381019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.099+0000 7f3816480700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381019d0e0 con 0x7f38100fef70 2024-04-05T21:52:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.099+0000 7f3816480700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3810196590 con 0x7f3810105b50 2024-04-05T21:52:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.100+0000 7f380ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38100fef70 0x7f381019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.100+0000 7f380ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38100fef70 0x7f381019c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36402/0 (socket says 172.21.15.121:36402) 2024-04-05T21:52:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.100+0000 7f380ffff700 1 -- 172.21.15.121:0/128696747 learned_addr learned my addr 172.21.15.121:0/128696747 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.100+0000 7f380f7fe700 1 --2- 172.21.15.121:0/128696747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3810105b50 0x7f381019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.100+0000 7f380ffff700 1 -- 172.21.15.121:0/128696747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3810105b50 msgr2=0x7f381019ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.100+0000 7f380ffff700 1 --2- 172.21.15.121:0/128696747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3810105b50 0x7f381019ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.100+0000 7f380ffff700 1 -- 172.21.15.121:0/128696747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38040095f0 con 0x7f38100fef70 2024-04-05T21:52:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.100+0000 7f380ffff700 1 --2- 172.21.15.121:0/128696747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38100fef70 0x7f381019c510 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f380000b4b0 tx=0x7f380000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.100+0000 7f380d7fa700 1 -- 172.21.15.121:0/128696747 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3800011800 con 0x7f38100fef70 2024-04-05T21:52:03.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.100+0000 7f380d7fa700 1 -- 172.21.15.121:0/128696747 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3800011e00 con 0x7f38100fef70 2024-04-05T21:52:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.100+0000 7f3816480700 1 -- 172.21.15.121:0/128696747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3810196870 con 0x7f38100fef70 2024-04-05T21:52:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.100+0000 7f380d7fa700 1 -- 172.21.15.121:0/128696747 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3800010e50 con 0x7f38100fef70 2024-04-05T21:52:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.100+0000 7f3816480700 1 -- 172.21.15.121:0/128696747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3810196dc0 con 0x7f38100fef70 2024-04-05T21:52:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.101+0000 7f3816480700 1 -- 172.21.15.121:0/128696747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3810109120 con 0x7f38100fef70 2024-04-05T21:52:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.101+0000 7f380d7fa700 1 -- 172.21.15.121:0/128696747 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f380001a400 con 0x7f38100fef70 2024-04-05T21:52:03.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.102+0000 7f380d7fa700 1 --2- 172.21.15.121:0/128696747 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37f806cce0 0x7f37f806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.102+0000 7f380d7fa700 1 -- 172.21.15.121:0/128696747 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f380008c1c0 con 0x7f38100fef70 2024-04-05T21:52:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.102+0000 7f380f7fe700 1 --2- 172.21.15.121:0/128696747 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37f806cce0 0x7f37f806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:03.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.102+0000 7f380f7fe700 1 --2- 172.21.15.121:0/128696747 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37f806cce0 0x7f37f806f1a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f380400b5c0 tx=0x7f3804005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:03.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.106+0000 7f380d7fa700 1 -- 172.21.15.121:0/128696747 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3800059d30 con 0x7f38100fef70 2024-04-05T21:52:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:02 smithi121 ceph-mon[145859]: pgmap v775: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:02 smithi188 ceph-mon[107793]: pgmap v775: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.345+0000 7f3816480700 1 -- 172.21.15.121:0/128696747 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f381004ea90 con 0x7f38100fef70 2024-04-05T21:52:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.345+0000 7f380d7fa700 1 -- 172.21.15.121:0/128696747 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f38000598c0 con 0x7f38100fef70 2024-04-05T21:52:03.346 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:52:03.346 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:52:03.346 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:52:03.346 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:52:03.346 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:52:03.346 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:52:03.347 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:52:03.347 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:52:03.347 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:52:03.347 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:52:03.347 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:52:03.347 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:52:03.347 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:52:03.347 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:52:03.347 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:52:03.347 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:52:03.347 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:52:03.347 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:52:03.348 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:52:03.349 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:52:03.349 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:52:03.349 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:52:03.349 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:52:03.349 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:52:03.349 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:52:03.349 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:52:03.349 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:52:03.349 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:52:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.347+0000 7f3816480700 1 -- 172.21.15.121:0/128696747 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37f806cce0 msgr2=0x7f37f806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.347+0000 7f3816480700 1 --2- 172.21.15.121:0/128696747 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37f806cce0 0x7f37f806f1a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f380400b5c0 tx=0x7f3804005fd0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.347+0000 7f3816480700 1 -- 172.21.15.121:0/128696747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38100fef70 msgr2=0x7f381019c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.347+0000 7f3816480700 1 --2- 172.21.15.121:0/128696747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38100fef70 0x7f381019c510 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f380000b4b0 tx=0x7f380000b870 comp rx=0 tx=0).stop 2024-04-05T21:52:03.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.347+0000 7f3816480700 1 -- 172.21.15.121:0/128696747 shutdown_connections 2024-04-05T21:52:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.347+0000 7f3816480700 1 --2- 172.21.15.121:0/128696747 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37f806cce0 0x7f37f806f1a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.347+0000 7f3816480700 1 --2- 172.21.15.121:0/128696747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38100fef70 0x7f381019c510 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.347+0000 7f3816480700 1 --2- 172.21.15.121:0/128696747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3810105b50 0x7f381019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.347+0000 7f3816480700 1 -- 172.21.15.121:0/128696747 >> 172.21.15.121:0/128696747 conn(0x7f3810075e30 msgr2=0x7f3810104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.347+0000 7f3816480700 1 -- 172.21.15.121:0/128696747 shutdown_connections 2024-04-05T21:52:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.347+0000 7f3816480700 1 -- 172.21.15.121:0/128696747 wait complete. 2024-04-05T21:52:03.351 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:52:03.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.451+0000 7fadff89e700 1 -- 172.21.15.121:0/1564461641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadf8100da0 msgr2=0x7fadf8101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:03.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.451+0000 7fadff89e700 1 --2- 172.21.15.121:0/1564461641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadf8100da0 0x7fadf8101180 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fade8009910 tx=0x7fade8009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:03.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.451+0000 7fadff89e700 1 -- 172.21.15.121:0/1564461641 shutdown_connections 2024-04-05T21:52:03.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.451+0000 7fadff89e700 1 --2- 172.21.15.121:0/1564461641 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadf8101750 0x7fadf81057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.451+0000 7fadff89e700 1 --2- 172.21.15.121:0/1564461641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadf8100da0 0x7fadf8101180 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.451+0000 7fadff89e700 1 -- 172.21.15.121:0/1564461641 >> 172.21.15.121:0/1564461641 conn(0x7fadf80fc630 msgr2=0x7fadf80fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:03.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.451+0000 7fadff89e700 1 -- 172.21.15.121:0/1564461641 shutdown_connections 2024-04-05T21:52:03.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.451+0000 7fadff89e700 1 -- 172.21.15.121:0/1564461641 wait complete. 2024-04-05T21:52:03.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.451+0000 7fadff89e700 1 Processor -- start 2024-04-05T21:52:03.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadff89e700 1 -- start start 2024-04-05T21:52:03.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadff89e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadf8100da0 0x7fadf8196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:03.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadff89e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadf8101750 0x7fadf8196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:03.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadff89e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf8197260 con 0x7fadf8101750 2024-04-05T21:52:03.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadff89e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf8199f90 con 0x7fadf8100da0 2024-04-05T21:52:03.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadfce39700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadf8101750 0x7fadf8196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:03.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadfce39700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadf8101750 0x7fadf8196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36406/0 (socket says 172.21.15.121:36406) 2024-04-05T21:52:03.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadfce39700 1 -- 172.21.15.121:0/4242344443 learned_addr learned my addr 172.21.15.121:0/4242344443 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:03.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadfd63a700 1 --2- 172.21.15.121:0/4242344443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadf8100da0 0x7fadf8196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:03.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadfce39700 1 -- 172.21.15.121:0/4242344443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadf8100da0 msgr2=0x7fadf8196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:03.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadfce39700 1 --2- 172.21.15.121:0/4242344443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadf8100da0 0x7fadf8196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadfce39700 1 -- 172.21.15.121:0/4242344443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade80095f0 con 0x7fadf8101750 2024-04-05T21:52:03.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadfce39700 1 --2- 172.21.15.121:0/4242344443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadf8101750 0x7fadf8196bd0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fadf400d900 tx=0x7fadf400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:03.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadee7fc700 1 -- 172.21.15.121:0/4242344443 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadf4009900 con 0x7fadf8101750 2024-04-05T21:52:03.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadee7fc700 1 -- 172.21.15.121:0/4242344443 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadf4010460 con 0x7fadf8101750 2024-04-05T21:52:03.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadff89e700 1 -- 172.21.15.121:0/4242344443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf819a270 con 0x7fadf8101750 2024-04-05T21:52:03.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadee7fc700 1 -- 172.21.15.121:0/4242344443 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadf400f5f0 con 0x7fadf8101750 2024-04-05T21:52:03.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.452+0000 7fadff89e700 1 -- 172.21.15.121:0/4242344443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf819a7c0 con 0x7fadf8101750 2024-04-05T21:52:03.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.453+0000 7fadff89e700 1 -- 172.21.15.121:0/4242344443 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf8109140 con 0x7fadf8101750 2024-04-05T21:52:03.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.454+0000 7fadee7fc700 1 -- 172.21.15.121:0/4242344443 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fadf40105d0 con 0x7fadf8101750 2024-04-05T21:52:03.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.454+0000 7fadee7fc700 1 --2- 172.21.15.121:0/4242344443 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fade406cce0 0x7fade406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:03.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.454+0000 7fadee7fc700 1 -- 172.21.15.121:0/4242344443 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fadf408c3c0 con 0x7fadf8101750 2024-04-05T21:52:03.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.454+0000 7fadfd63a700 1 --2- 172.21.15.121:0/4242344443 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fade406cce0 0x7fade406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:03.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.455+0000 7fadfd63a700 1 --2- 172.21.15.121:0/4242344443 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fade406cce0 0x7fade406f1a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fade800b5c0 tx=0x7fade8005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:03.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.459+0000 7fadee7fc700 1 -- 172.21.15.121:0/4242344443 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fadf4056430 con 0x7fadf8101750 2024-04-05T21:52:03.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.672+0000 7fadff89e700 1 -- 172.21.15.121:0/4242344443 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fadf819aaa0 con 0x7fade406cce0 2024-04-05T21:52:03.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.673+0000 7fadee7fc700 1 -- 172.21.15.121:0/4242344443 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fadf819aaa0 con 0x7fade406cce0 2024-04-05T21:52:03.674 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:52:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:52:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:52:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:52:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:52:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:52:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:52:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:52:03.674 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:52:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:52:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:52:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:52:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:52:03.675 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:52:03.675 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:52:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.675+0000 7fadff89e700 1 -- 172.21.15.121:0/4242344443 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fade406cce0 msgr2=0x7fade406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.675+0000 7fadff89e700 1 --2- 172.21.15.121:0/4242344443 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fade406cce0 0x7fade406f1a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fade800b5c0 tx=0x7fade8005fd0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.675+0000 7fadff89e700 1 -- 172.21.15.121:0/4242344443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadf8101750 msgr2=0x7fadf8196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.675+0000 7fadff89e700 1 --2- 172.21.15.121:0/4242344443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadf8101750 0x7fadf8196bd0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fadf400d900 tx=0x7fadf400dcc0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.676+0000 7fadff89e700 1 -- 172.21.15.121:0/4242344443 shutdown_connections 2024-04-05T21:52:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.676+0000 7fadff89e700 1 --2- 172.21.15.121:0/4242344443 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fade406cce0 0x7fade406f1a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.676+0000 7fadff89e700 1 --2- 172.21.15.121:0/4242344443 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fadf8100da0 0x7fadf8196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.676+0000 7fadff89e700 1 --2- 172.21.15.121:0/4242344443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadf8101750 0x7fadf8196bd0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.676+0000 7fadff89e700 1 -- 172.21.15.121:0/4242344443 >> 172.21.15.121:0/4242344443 conn(0x7fadf80fc630 msgr2=0x7fadf80fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.676+0000 7fadff89e700 1 -- 172.21.15.121:0/4242344443 shutdown_connections 2024-04-05T21:52:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.676+0000 7fadff89e700 1 -- 172.21.15.121:0/4242344443 wait complete. 2024-04-05T21:52:03.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.778+0000 7f90c1ae6700 1 -- 172.21.15.121:0/1135554306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90bc068410 msgr2=0x7f90bc0687f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:03.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.778+0000 7f90c1ae6700 1 --2- 172.21.15.121:0/1135554306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90bc068410 0x7f90bc0687f0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f90ac009910 tx=0x7f90ac009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:03.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.778+0000 7f90c1ae6700 1 -- 172.21.15.121:0/1135554306 shutdown_connections 2024-04-05T21:52:03.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.778+0000 7f90c1ae6700 1 --2- 172.21.15.121:0/1135554306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90bc068d30 0x7f90bc10d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.778+0000 7f90c1ae6700 1 --2- 172.21.15.121:0/1135554306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90bc068410 0x7f90bc0687f0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.778+0000 7f90c1ae6700 1 -- 172.21.15.121:0/1135554306 >> 172.21.15.121:0/1135554306 conn(0x7f90bc0755a0 msgr2=0x7f90bc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:03.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.778+0000 7f90c1ae6700 1 -- 172.21.15.121:0/1135554306 shutdown_connections 2024-04-05T21:52:03.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.778+0000 7f90c1ae6700 1 -- 172.21.15.121:0/1135554306 wait complete. 2024-04-05T21:52:03.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90c1ae6700 1 Processor -- start 2024-04-05T21:52:03.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90c1ae6700 1 -- start start 2024-04-05T21:52:03.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90c1ae6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90bc068410 0x7f90bc19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:03.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90c1ae6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90bc068d30 0x7f90bc19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:03.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90c1ae6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc19f1c0 con 0x7f90bc068d30 2024-04-05T21:52:03.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90c1ae6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc198670 con 0x7f90bc068410 2024-04-05T21:52:03.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90b3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90bc068d30 0x7f90bc19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:03.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90b3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90bc068d30 0x7f90bc19eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36424/0 (socket says 172.21.15.121:36424) 2024-04-05T21:52:03.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90b3fff700 1 -- 172.21.15.121:0/4125332239 learned_addr learned my addr 172.21.15.121:0/4125332239 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:03.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90bb7fe700 1 --2- 172.21.15.121:0/4125332239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90bc068410 0x7f90bc19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:03.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90b3fff700 1 -- 172.21.15.121:0/4125332239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90bc068410 msgr2=0x7f90bc19e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:03.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90b3fff700 1 --2- 172.21.15.121:0/4125332239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90bc068410 0x7f90bc19e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:03.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90b3fff700 1 -- 172.21.15.121:0/4125332239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac0095f0 con 0x7f90bc068d30 2024-04-05T21:52:03.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.779+0000 7f90b3fff700 1 --2- 172.21.15.121:0/4125332239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90bc068d30 0x7f90bc19eb30 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f90a400c8a0 tx=0x7f90a400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:03.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.780+0000 7f90b97fa700 1 -- 172.21.15.121:0/4125332239 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90a40078e0 con 0x7f90bc068d30 2024-04-05T21:52:03.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.780+0000 7f90b97fa700 1 -- 172.21.15.121:0/4125332239 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90a400f450 con 0x7f90bc068d30 2024-04-05T21:52:03.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.780+0000 7f90c1ae6700 1 -- 172.21.15.121:0/4125332239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90bc198950 con 0x7f90bc068d30 2024-04-05T21:52:03.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.780+0000 7f90b97fa700 1 -- 172.21.15.121:0/4125332239 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90a400e5e0 con 0x7f90bc068d30 2024-04-05T21:52:03.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.780+0000 7f90c1ae6700 1 -- 172.21.15.121:0/4125332239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90bc198ea0 con 0x7f90bc068d30 2024-04-05T21:52:03.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.781+0000 7f90c1ae6700 1 -- 172.21.15.121:0/4125332239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90bc10a9b0 con 0x7f90bc068d30 2024-04-05T21:52:03.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.781+0000 7f90b97fa700 1 -- 172.21.15.121:0/4125332239 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f90a400f5c0 con 0x7f90bc068d30 2024-04-05T21:52:03.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.781+0000 7f90b97fa700 1 --2- 172.21.15.121:0/4125332239 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f90a806cce0 0x7f90a806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:03.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.782+0000 7f90b97fa700 1 -- 172.21.15.121:0/4125332239 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f90a408c480 con 0x7f90bc068d30 2024-04-05T21:52:03.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.782+0000 7f90bb7fe700 1 --2- 172.21.15.121:0/4125332239 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f90a806cce0 0x7f90a806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:03.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.782+0000 7f90bb7fe700 1 --2- 172.21.15.121:0/4125332239 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f90a806cce0 0x7f90a806f1a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f90ac006010 tx=0x7f90ac01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:03.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:03.786+0000 7f90b97fa700 1 -- 172.21.15.121:0/4125332239 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f90a405a000 con 0x7f90bc068d30 2024-04-05T21:52:04.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:04.058+0000 7f90c1ae6700 1 -- 172.21.15.121:0/4125332239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f90bc04ea90 con 0x7f90bc068d30 2024-04-05T21:52:04.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:04.058+0000 7f90b97fa700 1 -- 172.21.15.121:0/4125332239 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f90a4059b90 con 0x7f90bc068d30 2024-04-05T21:52:04.059 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:52:04.059 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:52:04.059 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:52:04.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:52:04.059 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:52:04.059 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:52:04.059 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:52:04.059 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:52:04.059 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:52:04.059 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:52:04.060 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:52:04.060 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:52:04.060 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:52:04.060 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:52:04.060 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:52:04.060 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:52:04.060 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:52:04.060 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:52:04.060 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:52:04.060 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:52:04.060 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:52:04.060 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:52:04.060 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:52:04.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:04.060+0000 7f90c1ae6700 1 -- 172.21.15.121:0/4125332239 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f90a806cce0 msgr2=0x7f90a806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:04.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:04.060+0000 7f90c1ae6700 1 --2- 172.21.15.121:0/4125332239 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f90a806cce0 0x7f90a806f1a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f90ac006010 tx=0x7f90ac01a040 comp rx=0 tx=0).stop 2024-04-05T21:52:04.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:04.060+0000 7f90c1ae6700 1 -- 172.21.15.121:0/4125332239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90bc068d30 msgr2=0x7f90bc19eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:04.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:04.060+0000 7f90c1ae6700 1 --2- 172.21.15.121:0/4125332239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90bc068d30 0x7f90bc19eb30 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f90a400c8a0 tx=0x7f90a400cbb0 comp rx=0 tx=0).stop 2024-04-05T21:52:04.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:04.060+0000 7f90c1ae6700 1 -- 172.21.15.121:0/4125332239 shutdown_connections 2024-04-05T21:52:04.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:04.060+0000 7f90c1ae6700 1 --2- 172.21.15.121:0/4125332239 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f90a806cce0 0x7f90a806f1a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:04.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:04.060+0000 7f90c1ae6700 1 --2- 172.21.15.121:0/4125332239 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f90bc068410 0x7f90bc19e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:04.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:04.060+0000 7f90c1ae6700 1 --2- 172.21.15.121:0/4125332239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90bc068d30 0x7f90bc19eb30 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:04.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:04.060+0000 7f90c1ae6700 1 -- 172.21.15.121:0/4125332239 >> 172.21.15.121:0/4125332239 conn(0x7f90bc0755a0 msgr2=0x7f90bc0fe7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:04.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:04.060+0000 7f90c1ae6700 1 -- 172.21.15.121:0/4125332239 shutdown_connections 2024-04-05T21:52:04.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:04.060+0000 7f90c1ae6700 1 -- 172.21.15.121:0/4125332239 wait complete. 2024-04-05T21:52:04.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:03 smithi121 ceph-mon[145859]: from='client.35242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:04.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:03 smithi121 ceph-mon[145859]: from='client.35246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:04.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:03 smithi121 ceph-mon[145859]: from='client.35250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:04.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:03 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4029645257' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:52:04.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:03 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/128696747' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:52:04.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:03 smithi188 ceph-mon[107793]: from='client.35242 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:04.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:03 smithi188 ceph-mon[107793]: from='client.35246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:04.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:03 smithi188 ceph-mon[107793]: from='client.35250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:04.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:03 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4029645257' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:52:04.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:03 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/128696747' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:52:05.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:04 smithi121 ceph-mon[145859]: from='client.35262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:05.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:04 smithi121 ceph-mon[145859]: pgmap v776: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:05.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:04 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4125332239' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:52:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:04 smithi188 ceph-mon[107793]: from='client.35262 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:04 smithi188 ceph-mon[107793]: pgmap v776: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:04 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4125332239' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:52:07.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:06 smithi121 ceph-mon[145859]: pgmap v777: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:06 smithi188 ceph-mon[107793]: pgmap v777: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:09.188 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:08 smithi188 ceph-mon[107793]: pgmap v778: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:09.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:08 smithi121 ceph-mon[145859]: pgmap v778: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:11.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:10 smithi121 ceph-mon[145859]: pgmap v779: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:10 smithi188 ceph-mon[107793]: pgmap v779: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:12.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:11 smithi121 ceph-mon[145859]: pgmap v780: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:12.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:11 smithi188 ceph-mon[107793]: pgmap v780: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:15.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:14 smithi188 ceph-mon[107793]: pgmap v781: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:15.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:14 smithi121 ceph-mon[145859]: pgmap v781: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:16.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:52:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:52:17.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:16 smithi188 ceph-mon[107793]: pgmap v782: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:17.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:16 smithi121 ceph-mon[145859]: pgmap v782: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:19.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:18 smithi188 ceph-mon[107793]: pgmap v783: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:19.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:18 smithi121 ceph-mon[145859]: pgmap v783: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:21.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:20 smithi188 ceph-mon[107793]: pgmap v784: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:21.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:20 smithi121 ceph-mon[145859]: pgmap v784: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:23.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:22 smithi188 ceph-mon[107793]: pgmap v785: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:23.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:22 smithi121 ceph-mon[145859]: pgmap v785: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:25.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:24 smithi188 ceph-mon[107793]: pgmap v786: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:25.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:24 smithi121 ceph-mon[145859]: pgmap v786: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:27.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:26 smithi188 ceph-mon[107793]: pgmap v787: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:27.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:26 smithi121 ceph-mon[145859]: pgmap v787: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:29.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:28 smithi188 ceph-mon[107793]: pgmap v788: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:29.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:28 smithi121 ceph-mon[145859]: pgmap v788: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:31.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:30 smithi188 ceph-mon[107793]: pgmap v789: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:31.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:52:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:30 smithi121 ceph-mon[145859]: pgmap v789: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:31.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:52:33.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:32 smithi188 ceph-mon[107793]: pgmap v790: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:33.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:32 smithi121 ceph-mon[145859]: pgmap v790: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:34.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.176+0000 7faa81b9a700 1 -- 172.21.15.121:0/199030185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa7c105a40 msgr2=0x7faa7c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:34.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.176+0000 7faa81b9a700 1 --2- 172.21.15.121:0/199030185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa7c105a40 0x7faa7c105e20 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7faa70009910 tx=0x7faa70009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:34.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.176+0000 7faa81b9a700 1 -- 172.21.15.121:0/199030185 shutdown_connections 2024-04-05T21:52:34.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.176+0000 7faa81b9a700 1 --2- 172.21.15.121:0/199030185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa7c0ffa40 0x7faa7c0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.176+0000 7faa81b9a700 1 --2- 172.21.15.121:0/199030185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa7c105a40 0x7faa7c105e20 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.176+0000 7faa81b9a700 1 -- 172.21.15.121:0/199030185 >> 172.21.15.121:0/199030185 conn(0x7faa7c0fb570 msgr2=0x7faa7c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:34.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.176+0000 7faa81b9a700 1 -- 172.21.15.121:0/199030185 shutdown_connections 2024-04-05T21:52:34.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.176+0000 7faa81b9a700 1 -- 172.21.15.121:0/199030185 wait complete. 2024-04-05T21:52:34.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.176+0000 7faa81b9a700 1 Processor -- start 2024-04-05T21:52:34.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.177+0000 7faa81b9a700 1 -- start start 2024-04-05T21:52:34.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.177+0000 7faa81b9a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa7c0ffa40 0x7faa7c1974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:34.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.177+0000 7faa81b9a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa7c105a40 0x7faa7c1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:34.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.177+0000 7faa81b9a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c198000 con 0x7faa7c105a40 2024-04-05T21:52:34.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.177+0000 7faa81b9a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c198170 con 0x7faa7c0ffa40 2024-04-05T21:52:34.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.177+0000 7faa7affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa7c105a40 0x7faa7c1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:34.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.177+0000 7faa7affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa7c105a40 0x7faa7c1979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57454/0 (socket says 172.21.15.121:57454) 2024-04-05T21:52:34.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.177+0000 7faa7affd700 1 -- 172.21.15.121:0/2712302285 learned_addr learned my addr 172.21.15.121:0/2712302285 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:34.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.177+0000 7faa7b7fe700 1 --2- 172.21.15.121:0/2712302285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa7c0ffa40 0x7faa7c1974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:34.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.177+0000 7faa7affd700 1 -- 172.21.15.121:0/2712302285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa7c0ffa40 msgr2=0x7faa7c1974a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:34.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.177+0000 7faa7affd700 1 --2- 172.21.15.121:0/2712302285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa7c0ffa40 0x7faa7c1974a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.177+0000 7faa7affd700 1 -- 172.21.15.121:0/2712302285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa700095f0 con 0x7faa7c105a40 2024-04-05T21:52:34.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.177+0000 7faa7b7fe700 1 --2- 172.21.15.121:0/2712302285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa7c0ffa40 0x7faa7c1974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:52:34.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.178+0000 7faa7affd700 1 --2- 172.21.15.121:0/2712302285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa7c105a40 0x7faa7c1979e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00d8d0 tx=0x7faa6c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:34.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.178+0000 7faa78ff9700 1 -- 172.21.15.121:0/2712302285 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa6c009940 con 0x7faa7c105a40 2024-04-05T21:52:34.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.178+0000 7faa78ff9700 1 -- 172.21.15.121:0/2712302285 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa6c010460 con 0x7faa7c105a40 2024-04-05T21:52:34.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.178+0000 7faa81b9a700 1 -- 172.21.15.121:0/2712302285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7c1a0510 con 0x7faa7c105a40 2024-04-05T21:52:34.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.178+0000 7faa78ff9700 1 -- 172.21.15.121:0/2712302285 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa6c00f5f0 con 0x7faa7c105a40 2024-04-05T21:52:34.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.178+0000 7faa81b9a700 1 -- 172.21.15.121:0/2712302285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7c1a0a00 con 0x7faa7c105a40 2024-04-05T21:52:34.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.179+0000 7faa81b9a700 1 -- 172.21.15.121:0/2712302285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7c04ea90 con 0x7faa7c105a40 2024-04-05T21:52:34.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.179+0000 7faa78ff9700 1 -- 172.21.15.121:0/2712302285 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7faa6c0105d0 con 0x7faa7c105a40 2024-04-05T21:52:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.179+0000 7faa78ff9700 1 --2- 172.21.15.121:0/2712302285 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faa6406cce0 0x7faa6406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.180+0000 7faa78ff9700 1 -- 172.21.15.121:0/2712302285 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7faa6c08c5e0 con 0x7faa7c105a40 2024-04-05T21:52:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.180+0000 7faa7b7fe700 1 --2- 172.21.15.121:0/2712302285 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faa6406cce0 0x7faa6406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:34.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.180+0000 7faa7b7fe700 1 --2- 172.21.15.121:0/2712302285 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faa6406cce0 0x7faa6406f1a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7faa7000b5c0 tx=0x7faa7001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:34.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.184+0000 7faa78ff9700 1 -- 172.21.15.121:0/2712302285 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faa6c056650 con 0x7faa7c105a40 2024-04-05T21:52:34.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.393+0000 7faa81b9a700 1 -- 172.21.15.121:0/2712302285 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faa7c0fc6f0 con 0x7faa6406cce0 2024-04-05T21:52:34.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.395+0000 7faa78ff9700 1 -- 172.21.15.121:0/2712302285 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7faa7c0fc6f0 con 0x7faa6406cce0 2024-04-05T21:52:34.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.397+0000 7faa81b9a700 1 -- 172.21.15.121:0/2712302285 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faa6406cce0 msgr2=0x7faa6406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:34.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.397+0000 7faa81b9a700 1 --2- 172.21.15.121:0/2712302285 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faa6406cce0 0x7faa6406f1a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7faa7000b5c0 tx=0x7faa7001a040 comp rx=0 tx=0).stop 2024-04-05T21:52:34.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.397+0000 7faa81b9a700 1 -- 172.21.15.121:0/2712302285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa7c105a40 msgr2=0x7faa7c1979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:34.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.397+0000 7faa81b9a700 1 --2- 172.21.15.121:0/2712302285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa7c105a40 0x7faa7c1979e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00d8d0 tx=0x7faa6c00dc90 comp rx=0 tx=0).stop 2024-04-05T21:52:34.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.397+0000 7faa81b9a700 1 -- 172.21.15.121:0/2712302285 shutdown_connections 2024-04-05T21:52:34.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.397+0000 7faa81b9a700 1 --2- 172.21.15.121:0/2712302285 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faa6406cce0 0x7faa6406f1a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.397+0000 7faa81b9a700 1 --2- 172.21.15.121:0/2712302285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faa7c0ffa40 0x7faa7c1974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.397+0000 7faa81b9a700 1 --2- 172.21.15.121:0/2712302285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa7c105a40 0x7faa7c1979e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.397+0000 7faa81b9a700 1 -- 172.21.15.121:0/2712302285 >> 172.21.15.121:0/2712302285 conn(0x7faa7c0fb570 msgr2=0x7faa7c0fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:34.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.397+0000 7faa81b9a700 1 -- 172.21.15.121:0/2712302285 shutdown_connections 2024-04-05T21:52:34.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.397+0000 7faa81b9a700 1 -- 172.21.15.121:0/2712302285 wait complete. 2024-04-05T21:52:34.407 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:52:34.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.503+0000 7fc98a0f2700 1 -- 172.21.15.121:0/3811295587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc984068670 msgr2=0x7fc984068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:34.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.503+0000 7fc98a0f2700 1 --2- 172.21.15.121:0/3811295587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc984068670 0x7fc984068a50 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc974009910 tx=0x7fc974009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:34.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.503+0000 7fc98a0f2700 1 -- 172.21.15.121:0/3811295587 shutdown_connections 2024-04-05T21:52:34.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.503+0000 7fc98a0f2700 1 --2- 172.21.15.121:0/3811295587 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc984069020 0x7fc984105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.503+0000 7fc98a0f2700 1 --2- 172.21.15.121:0/3811295587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc984068670 0x7fc984068a50 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.503+0000 7fc98a0f2700 1 -- 172.21.15.121:0/3811295587 >> 172.21.15.121:0/3811295587 conn(0x7fc9840755a0 msgr2=0x7fc9840759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:34.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.503+0000 7fc98a0f2700 1 -- 172.21.15.121:0/3811295587 shutdown_connections 2024-04-05T21:52:34.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.503+0000 7fc98a0f2700 1 -- 172.21.15.121:0/3811295587 wait complete. 2024-04-05T21:52:34.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.504+0000 7fc98a0f2700 1 Processor -- start 2024-04-05T21:52:34.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.504+0000 7fc98a0f2700 1 -- start start 2024-04-05T21:52:34.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.504+0000 7fc98a0f2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc984068670 0x7fc984196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:34.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.504+0000 7fc98a0f2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc984069020 0x7fc984196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:34.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.504+0000 7fc98a0f2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc984197290 con 0x7fc984068670 2024-04-05T21:52:34.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.504+0000 7fc98a0f2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98419b020 con 0x7fc984069020 2024-04-05T21:52:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.504+0000 7fc9837fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc984068670 0x7fc984196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.504+0000 7fc9837fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc984068670 0x7fc984196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43830/0 (socket says 172.21.15.121:43830) 2024-04-05T21:52:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.504+0000 7fc9837fe700 1 -- 172.21.15.121:0/252997616 learned_addr learned my addr 172.21.15.121:0/252997616 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.504+0000 7fc9837fe700 1 -- 172.21.15.121:0/252997616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc984069020 msgr2=0x7fc984196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.504+0000 7fc9837fe700 1 --2- 172.21.15.121:0/252997616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc984069020 0x7fc984196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.504+0000 7fc9837fe700 1 -- 172.21.15.121:0/252997616 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9740095f0 con 0x7fc984068670 2024-04-05T21:52:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.505+0000 7fc9837fe700 1 --2- 172.21.15.121:0/252997616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc984068670 0x7fc984196670 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc974006010 tx=0x7fc97400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.505+0000 7fc980ff9700 1 -- 172.21.15.121:0/252997616 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc97401d070 con 0x7fc984068670 2024-04-05T21:52:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.505+0000 7fc98a0f2700 1 -- 172.21.15.121:0/252997616 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc98419b2a0 con 0x7fc984068670 2024-04-05T21:52:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.505+0000 7fc980ff9700 1 -- 172.21.15.121:0/252997616 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc97400f460 con 0x7fc984068670 2024-04-05T21:52:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.505+0000 7fc980ff9700 1 -- 172.21.15.121:0/252997616 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc974021640 con 0x7fc984068670 2024-04-05T21:52:34.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.505+0000 7fc98a0f2700 1 -- 172.21.15.121:0/252997616 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98419b790 con 0x7fc984068670 2024-04-05T21:52:34.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.506+0000 7fc98a0f2700 1 -- 172.21.15.121:0/252997616 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc984109130 con 0x7fc984068670 2024-04-05T21:52:34.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.506+0000 7fc980ff9700 1 -- 172.21.15.121:0/252997616 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc97400fa60 con 0x7fc984068670 2024-04-05T21:52:34.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.506+0000 7fc980ff9700 1 --2- 172.21.15.121:0/252997616 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc97006cce0 0x7fc97006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:34.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.507+0000 7fc980ff9700 1 -- 172.21.15.121:0/252997616 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc97408d880 con 0x7fc984068670 2024-04-05T21:52:34.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.507+0000 7fc982ffd700 1 --2- 172.21.15.121:0/252997616 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc97006cce0 0x7fc97006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:34.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.507+0000 7fc982ffd700 1 --2- 172.21.15.121:0/252997616 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc97006cce0 0x7fc97006f1a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc984197c90 tx=0x7fc96c005c80 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:34.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.511+0000 7fc980ff9700 1 -- 172.21.15.121:0/252997616 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc9740578f0 con 0x7fc984068670 2024-04-05T21:52:34.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.720+0000 7fc98a0f2700 1 -- 172.21.15.121:0/252997616 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc98402cdd0 con 0x7fc97006cce0 2024-04-05T21:52:34.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.721+0000 7fc980ff9700 1 -- 172.21.15.121:0/252997616 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc98402cdd0 con 0x7fc97006cce0 2024-04-05T21:52:34.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.723+0000 7fc98a0f2700 1 -- 172.21.15.121:0/252997616 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc97006cce0 msgr2=0x7fc97006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:34.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.723+0000 7fc98a0f2700 1 --2- 172.21.15.121:0/252997616 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc97006cce0 0x7fc97006f1a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc984197c90 tx=0x7fc96c005c80 comp rx=0 tx=0).stop 2024-04-05T21:52:34.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.723+0000 7fc98a0f2700 1 -- 172.21.15.121:0/252997616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc984068670 msgr2=0x7fc984196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:34.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.723+0000 7fc98a0f2700 1 --2- 172.21.15.121:0/252997616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc984068670 0x7fc984196670 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc974006010 tx=0x7fc97400be20 comp rx=0 tx=0).stop 2024-04-05T21:52:34.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.723+0000 7fc98a0f2700 1 -- 172.21.15.121:0/252997616 shutdown_connections 2024-04-05T21:52:34.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.723+0000 7fc98a0f2700 1 --2- 172.21.15.121:0/252997616 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc97006cce0 0x7fc97006f1a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.723+0000 7fc98a0f2700 1 --2- 172.21.15.121:0/252997616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc984068670 0x7fc984196670 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.723+0000 7fc98a0f2700 1 --2- 172.21.15.121:0/252997616 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc984069020 0x7fc984196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.723+0000 7fc98a0f2700 1 -- 172.21.15.121:0/252997616 >> 172.21.15.121:0/252997616 conn(0x7fc9840755a0 msgr2=0x7fc9840fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:34.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.724+0000 7fc98a0f2700 1 -- 172.21.15.121:0/252997616 shutdown_connections 2024-04-05T21:52:34.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.724+0000 7fc98a0f2700 1 -- 172.21.15.121:0/252997616 wait complete. 2024-04-05T21:52:34.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.827+0000 7f02d11c4700 1 -- 172.21.15.121:0/114324189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02cc0ffbd0 msgr2=0x7f02cc100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:34.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.827+0000 7f02d11c4700 1 --2- 172.21.15.121:0/114324189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02cc0ffbd0 0x7f02cc100050 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009910 tx=0x7f02bc009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:34.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.827+0000 7f02d11c4700 1 -- 172.21.15.121:0/114324189 shutdown_connections 2024-04-05T21:52:34.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.827+0000 7f02d11c4700 1 --2- 172.21.15.121:0/114324189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02cc0ffbd0 0x7f02cc100050 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.827+0000 7f02d11c4700 1 --2- 172.21.15.121:0/114324189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02cc105a40 0x7f02cc105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.827+0000 7f02d11c4700 1 -- 172.21.15.121:0/114324189 >> 172.21.15.121:0/114324189 conn(0x7f02cc0755a0 msgr2=0x7f02cc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:34.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.828+0000 7f02d11c4700 1 -- 172.21.15.121:0/114324189 shutdown_connections 2024-04-05T21:52:34.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.828+0000 7f02d11c4700 1 -- 172.21.15.121:0/114324189 wait complete. 2024-04-05T21:52:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.828+0000 7f02d11c4700 1 Processor -- start 2024-04-05T21:52:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.829+0000 7f02d11c4700 1 -- start start 2024-04-05T21:52:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.829+0000 7f02d11c4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02cc0ffbd0 0x7f02cc195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.829+0000 7f02d11c4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02cc105a40 0x7f02cc195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.829+0000 7f02d11c4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc1960d0 con 0x7f02cc105a40 2024-04-05T21:52:34.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.829+0000 7f02d11c4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc196210 con 0x7f02cc0ffbd0 2024-04-05T21:52:34.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.829+0000 7f02ca59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02cc105a40 0x7f02cc195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:34.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.829+0000 7f02ca59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02cc105a40 0x7f02cc195a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43854/0 (socket says 172.21.15.121:43854) 2024-04-05T21:52:34.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.829+0000 7f02ca59c700 1 -- 172.21.15.121:0/4002013973 learned_addr learned my addr 172.21.15.121:0/4002013973 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:34.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.829+0000 7f02cad9d700 1 --2- 172.21.15.121:0/4002013973 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02cc0ffbd0 0x7f02cc195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:34.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.829+0000 7f02ca59c700 1 -- 172.21.15.121:0/4002013973 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02cc0ffbd0 msgr2=0x7f02cc195540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:34.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.829+0000 7f02ca59c700 1 --2- 172.21.15.121:0/4002013973 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02cc0ffbd0 0x7f02cc195540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:34.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.829+0000 7f02ca59c700 1 -- 172.21.15.121:0/4002013973 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc0095f0 con 0x7f02cc105a40 2024-04-05T21:52:34.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.830+0000 7f02ca59c700 1 --2- 172.21.15.121:0/4002013973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02cc105a40 0x7f02cc195a80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f02bc006010 tx=0x7f02bc00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:34.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.830+0000 7f02b3fff700 1 -- 172.21.15.121:0/4002013973 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02bc01d070 con 0x7f02cc105a40 2024-04-05T21:52:34.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.830+0000 7f02d11c4700 1 -- 172.21.15.121:0/4002013973 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02cc19a000 con 0x7f02cc105a40 2024-04-05T21:52:34.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.830+0000 7f02b3fff700 1 -- 172.21.15.121:0/4002013973 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02bc005940 con 0x7f02cc105a40 2024-04-05T21:52:34.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.830+0000 7f02b3fff700 1 -- 172.21.15.121:0/4002013973 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02bc021410 con 0x7f02cc105a40 2024-04-05T21:52:34.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.830+0000 7f02d11c4700 1 -- 172.21.15.121:0/4002013973 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02cc19a4f0 con 0x7f02cc105a40 2024-04-05T21:52:34.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.831+0000 7f02d11c4700 1 -- 172.21.15.121:0/4002013973 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02cc04ea90 con 0x7f02cc105a40 2024-04-05T21:52:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.831+0000 7f02b3fff700 1 -- 172.21.15.121:0/4002013973 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f02bc003a60 con 0x7f02cc105a40 2024-04-05T21:52:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.832+0000 7f02b3fff700 1 --2- 172.21.15.121:0/4002013973 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f02b406cce0 0x7f02b406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.832+0000 7f02b3fff700 1 -- 172.21.15.121:0/4002013973 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f02bc08d810 con 0x7f02cc105a40 2024-04-05T21:52:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.832+0000 7f02cad9d700 1 --2- 172.21.15.121:0/4002013973 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f02b406cce0 0x7f02b406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:34.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.832+0000 7f02cad9d700 1 --2- 172.21.15.121:0/4002013973 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f02b406cce0 0x7f02b406f1a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f02c000ac50 tx=0x7f02c000a380 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:34.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:34.836+0000 7f02b3fff700 1 -- 172.21.15.121:0/4002013973 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f02bc057880 con 0x7f02cc105a40 2024-04-05T21:52:35.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:34 smithi188 ceph-mon[107793]: pgmap v791: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:35.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.037+0000 7f02d11c4700 1 -- 172.21.15.121:0/4002013973 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f02cc107ba0 con 0x7f02b406cce0 2024-04-05T21:52:35.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.048+0000 7f02b3fff700 1 -- 172.21.15.121:0/4002013973 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f02cc107ba0 con 0x7f02b406cce0 2024-04-05T21:52:35.049 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:52:35.049 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (23m) 114s ago 34m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:52:35.049 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (35m) 114s ago 35m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:52:35.049 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (33m) 117s ago 33m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:52:35.049 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (21m) 114s ago 35m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:52:35.049 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (21m) 117s ago 33m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:52:35.050 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (23m) 114s ago 34m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:52:35.050 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (17m) 114s ago 28m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:52:35.050 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (17m) 114s ago 28m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:52:35.050 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (17m) 117s ago 28m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:52:35.050 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (2m) 117s ago 28m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:52:35.051 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (25m) 114s ago 36m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:52:35.051 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (24m) 117s ago 33m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:52:35.051 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (22m) 114s ago 36m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:52:35.051 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (21m) 117s ago 33m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:52:35.051 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (24m) 114s ago 35m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:52:35.051 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (24m) 117s ago 33m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:52:35.052 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (21m) 114s ago 32m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:52:35.052 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (20m) 114s ago 32m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:52:35.052 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (19m) 114s ago 31m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:52:35.052 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (19m) 117s ago 31m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:52:35.052 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (18m) 117s ago 30m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:52:35.052 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (18m) 117s ago 30m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:52:35.053 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (23m) 114s ago 34m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:52:35.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.050+0000 7f02d11c4700 1 -- 172.21.15.121:0/4002013973 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f02b406cce0 msgr2=0x7f02b406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:35.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.050+0000 7f02d11c4700 1 --2- 172.21.15.121:0/4002013973 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f02b406cce0 0x7f02b406f1a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f02c000ac50 tx=0x7f02c000a380 comp rx=0 tx=0).stop 2024-04-05T21:52:35.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.050+0000 7f02d11c4700 1 -- 172.21.15.121:0/4002013973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02cc105a40 msgr2=0x7f02cc195a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:35.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.050+0000 7f02d11c4700 1 --2- 172.21.15.121:0/4002013973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02cc105a40 0x7f02cc195a80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f02bc006010 tx=0x7f02bc00bd70 comp rx=0 tx=0).stop 2024-04-05T21:52:35.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.050+0000 7f02d11c4700 1 -- 172.21.15.121:0/4002013973 shutdown_connections 2024-04-05T21:52:35.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.050+0000 7f02d11c4700 1 --2- 172.21.15.121:0/4002013973 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f02b406cce0 0x7f02b406f1a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.050+0000 7f02d11c4700 1 --2- 172.21.15.121:0/4002013973 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02cc0ffbd0 0x7f02cc195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.050+0000 7f02d11c4700 1 --2- 172.21.15.121:0/4002013973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02cc105a40 0x7f02cc195a80 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.050+0000 7f02d11c4700 1 -- 172.21.15.121:0/4002013973 >> 172.21.15.121:0/4002013973 conn(0x7f02cc0755a0 msgr2=0x7f02cc0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:35.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.050+0000 7f02d11c4700 1 -- 172.21.15.121:0/4002013973 shutdown_connections 2024-04-05T21:52:35.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.050+0000 7f02d11c4700 1 -- 172.21.15.121:0/4002013973 wait complete. 2024-04-05T21:52:35.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:34 smithi121 ceph-mon[145859]: pgmap v791: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:35.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.152+0000 7fcb0e11b700 1 -- 172.21.15.121:0/3338295929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb081003c0 msgr2=0x7fcb081007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:35.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.152+0000 7fcb0e11b700 1 --2- 172.21.15.121:0/3338295929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb081003c0 0x7fcb081007a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009910 tx=0x7fcafc009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:35.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.152+0000 7fcb0e11b700 1 -- 172.21.15.121:0/3338295929 shutdown_connections 2024-04-05T21:52:35.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.152+0000 7fcb0e11b700 1 --2- 172.21.15.121:0/3338295929 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb08100d70 0x7fcb081048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.152+0000 7fcb0e11b700 1 --2- 172.21.15.121:0/3338295929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb081003c0 0x7fcb081007a0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.152+0000 7fcb0e11b700 1 -- 172.21.15.121:0/3338295929 >> 172.21.15.121:0/3338295929 conn(0x7fcb08075e30 msgr2=0x7fcb080fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.152+0000 7fcb0e11b700 1 -- 172.21.15.121:0/3338295929 shutdown_connections 2024-04-05T21:52:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.152+0000 7fcb0e11b700 1 -- 172.21.15.121:0/3338295929 wait complete. 2024-04-05T21:52:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.152+0000 7fcb0e11b700 1 Processor -- start 2024-04-05T21:52:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb0e11b700 1 -- start start 2024-04-05T21:52:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb0e11b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb081003c0 0x7fcb0819e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb0e11b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb08100d70 0x7fcb0819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb0e11b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0819f430 con 0x7fcb08100d70 2024-04-05T21:52:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb0e11b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb081988e0 con 0x7fcb081003c0 2024-04-05T21:52:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb06ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb08100d70 0x7fcb0819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb06ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb08100d70 0x7fcb0819eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43866/0 (socket says 172.21.15.121:43866) 2024-04-05T21:52:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb06ffd700 1 -- 172.21.15.121:0/2033986776 learned_addr learned my addr 172.21.15.121:0/2033986776 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb077fe700 1 --2- 172.21.15.121:0/2033986776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb081003c0 0x7fcb0819e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:35.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb06ffd700 1 -- 172.21.15.121:0/2033986776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb081003c0 msgr2=0x7fcb0819e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:35.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb06ffd700 1 --2- 172.21.15.121:0/2033986776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb081003c0 0x7fcb0819e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb06ffd700 1 -- 172.21.15.121:0/2033986776 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc0095f0 con 0x7fcb08100d70 2024-04-05T21:52:35.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb077fe700 1 --2- 172.21.15.121:0/2033986776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb081003c0 0x7fcb0819e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:52:35.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb06ffd700 1 --2- 172.21.15.121:0/2033986776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb08100d70 0x7fcb0819eda0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800c8f0 tx=0x7fcaf800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:35.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb04ff9700 1 -- 172.21.15.121:0/2033986776 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaf8007a50 con 0x7fcb08100d70 2024-04-05T21:52:35.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.153+0000 7fcb0e11b700 1 -- 172.21.15.121:0/2033986776 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb08198bc0 con 0x7fcb08100d70 2024-04-05T21:52:35.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.154+0000 7fcb04ff9700 1 -- 172.21.15.121:0/2033986776 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcaf8007bb0 con 0x7fcb08100d70 2024-04-05T21:52:35.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.154+0000 7fcb04ff9700 1 -- 172.21.15.121:0/2033986776 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaf8018630 con 0x7fcb08100d70 2024-04-05T21:52:35.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.154+0000 7fcb0e11b700 1 -- 172.21.15.121:0/2033986776 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb08199110 con 0x7fcb08100d70 2024-04-05T21:52:35.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.154+0000 7fcb0e11b700 1 -- 172.21.15.121:0/2033986776 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb0804ea90 con 0x7fcb08100d70 2024-04-05T21:52:35.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.155+0000 7fcb04ff9700 1 -- 172.21.15.121:0/2033986776 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcaf8007d20 con 0x7fcb08100d70 2024-04-05T21:52:35.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.155+0000 7fcb04ff9700 1 --2- 172.21.15.121:0/2033986776 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcaf006cce0 0x7fcaf006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:35.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.155+0000 7fcb04ff9700 1 -- 172.21.15.121:0/2033986776 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fcaf808c4b0 con 0x7fcb08100d70 2024-04-05T21:52:35.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.156+0000 7fcb077fe700 1 --2- 172.21.15.121:0/2033986776 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcaf006cce0 0x7fcaf006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:35.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.156+0000 7fcb077fe700 1 --2- 172.21.15.121:0/2033986776 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcaf006cce0 0x7fcaf006f1a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00b5c0 tx=0x7fcafc01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:35.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.160+0000 7fcb04ff9700 1 -- 172.21.15.121:0/2033986776 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcaf805a030 con 0x7fcb08100d70 2024-04-05T21:52:35.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.435+0000 7fcb0e11b700 1 -- 172.21.15.121:0/2033986776 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcb08199df0 con 0x7fcb08100d70 2024-04-05T21:52:35.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.435+0000 7fcb04ff9700 1 -- 172.21.15.121:0/2033986776 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fcaf8059bc0 con 0x7fcb08100d70 2024-04-05T21:52:35.436 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:52:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:52:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:52:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:52:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:52:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:52:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:52:35.436 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:52:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:52:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:52:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:52:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:52:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:52:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:52:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:52:35.437 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:52:35.437 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:52:35.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.437+0000 7fcb0e11b700 1 -- 172.21.15.121:0/2033986776 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcaf006cce0 msgr2=0x7fcaf006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:35.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.437+0000 7fcb0e11b700 1 --2- 172.21.15.121:0/2033986776 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcaf006cce0 0x7fcaf006f1a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcafc00b5c0 tx=0x7fcafc01a040 comp rx=0 tx=0).stop 2024-04-05T21:52:35.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.437+0000 7fcb0e11b700 1 -- 172.21.15.121:0/2033986776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb08100d70 msgr2=0x7fcb0819eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:35.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.437+0000 7fcb0e11b700 1 --2- 172.21.15.121:0/2033986776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb08100d70 0x7fcb0819eda0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800c8f0 tx=0x7fcaf800ccb0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.437+0000 7fcb0e11b700 1 -- 172.21.15.121:0/2033986776 shutdown_connections 2024-04-05T21:52:35.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.437+0000 7fcb0e11b700 1 --2- 172.21.15.121:0/2033986776 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcaf006cce0 0x7fcaf006f1a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.437+0000 7fcb0e11b700 1 --2- 172.21.15.121:0/2033986776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcb081003c0 0x7fcb0819e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.437+0000 7fcb0e11b700 1 --2- 172.21.15.121:0/2033986776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb08100d70 0x7fcb0819eda0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.437+0000 7fcb0e11b700 1 -- 172.21.15.121:0/2033986776 >> 172.21.15.121:0/2033986776 conn(0x7fcb08075e30 msgr2=0x7fcb08103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:35.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.437+0000 7fcb0e11b700 1 -- 172.21.15.121:0/2033986776 shutdown_connections 2024-04-05T21:52:35.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.437+0000 7fcb0e11b700 1 -- 172.21.15.121:0/2033986776 wait complete. 2024-04-05T21:52:35.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.540+0000 7fe193b6c700 1 -- 172.21.15.121:0/4071473784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe18c105b50 msgr2=0x7fe18c105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:35.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.540+0000 7fe193b6c700 1 --2- 172.21.15.121:0/4071473784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe18c105b50 0x7fe18c105f30 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fe17c009910 tx=0x7fe17c009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:35.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.540+0000 7fe193b6c700 1 -- 172.21.15.121:0/4071473784 shutdown_connections 2024-04-05T21:52:35.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.540+0000 7fe193b6c700 1 --2- 172.21.15.121:0/4071473784 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe18c0fef70 0x7fe18c0ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.540+0000 7fe193b6c700 1 --2- 172.21.15.121:0/4071473784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe18c105b50 0x7fe18c105f30 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.540+0000 7fe193b6c700 1 -- 172.21.15.121:0/4071473784 >> 172.21.15.121:0/4071473784 conn(0x7fe18c075e30 msgr2=0x7fe18c0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:35.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.540+0000 7fe193b6c700 1 -- 172.21.15.121:0/4071473784 shutdown_connections 2024-04-05T21:52:35.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.540+0000 7fe193b6c700 1 -- 172.21.15.121:0/4071473784 wait complete. 2024-04-05T21:52:35.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.540+0000 7fe193b6c700 1 Processor -- start 2024-04-05T21:52:35.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe193b6c700 1 -- start start 2024-04-05T21:52:35.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe193b6c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe18c0fef70 0x7fe18c19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:35.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe193b6c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe18c105b50 0x7fe18c19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:35.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe193b6c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c19d0e0 con 0x7fe18c105b50 2024-04-05T21:52:35.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe193b6c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c196590 con 0x7fe18c0fef70 2024-04-05T21:52:35.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe191107700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe18c105b50 0x7fe18c19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:35.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe191107700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe18c105b50 0x7fe18c19ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43878/0 (socket says 172.21.15.121:43878) 2024-04-05T21:52:35.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe191107700 1 -- 172.21.15.121:0/2889503314 learned_addr learned my addr 172.21.15.121:0/2889503314 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:35.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe191908700 1 --2- 172.21.15.121:0/2889503314 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe18c0fef70 0x7fe18c19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:35.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe191107700 1 -- 172.21.15.121:0/2889503314 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe18c0fef70 msgr2=0x7fe18c19c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:35.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe191107700 1 --2- 172.21.15.121:0/2889503314 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe18c0fef70 0x7fe18c19c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe191107700 1 -- 172.21.15.121:0/2889503314 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c0095f0 con 0x7fe18c105b50 2024-04-05T21:52:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe191107700 1 --2- 172.21.15.121:0/2889503314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe18c105b50 0x7fe18c19ca50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe18800b4b0 tx=0x7fe18800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.541+0000 7fe182ffd700 1 -- 172.21.15.121:0/2889503314 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe188011800 con 0x7fe18c105b50 2024-04-05T21:52:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.542+0000 7fe193b6c700 1 -- 172.21.15.121:0/2889503314 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18c196870 con 0x7fe18c105b50 2024-04-05T21:52:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.542+0000 7fe182ffd700 1 -- 172.21.15.121:0/2889503314 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe188011e00 con 0x7fe18c105b50 2024-04-05T21:52:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.542+0000 7fe182ffd700 1 -- 172.21.15.121:0/2889503314 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe188010e50 con 0x7fe18c105b50 2024-04-05T21:52:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.542+0000 7fe193b6c700 1 -- 172.21.15.121:0/2889503314 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18c196dc0 con 0x7fe18c105b50 2024-04-05T21:52:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.543+0000 7fe193b6c700 1 -- 172.21.15.121:0/2889503314 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18c109120 con 0x7fe18c105b50 2024-04-05T21:52:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.543+0000 7fe182ffd700 1 -- 172.21.15.121:0/2889503314 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe188010380 con 0x7fe18c105b50 2024-04-05T21:52:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.543+0000 7fe182ffd700 1 --2- 172.21.15.121:0/2889503314 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe17806cce0 0x7fe17806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.544+0000 7fe191908700 1 --2- 172.21.15.121:0/2889503314 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe17806cce0 0x7fe17806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.544+0000 7fe182ffd700 1 -- 172.21.15.121:0/2889503314 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe18808c170 con 0x7fe18c105b50 2024-04-05T21:52:35.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.544+0000 7fe191908700 1 --2- 172.21.15.121:0/2889503314 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe17806cce0 0x7fe17806f1a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe17c006010 tx=0x7fe17c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:35.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.548+0000 7fe182ffd700 1 -- 172.21.15.121:0/2889503314 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe188059c70 con 0x7fe18c105b50 2024-04-05T21:52:35.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.785+0000 7fe193b6c700 1 -- 172.21.15.121:0/2889503314 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe18c04ea90 con 0x7fe18c105b50 2024-04-05T21:52:35.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.785+0000 7fe182ffd700 1 -- 172.21.15.121:0/2889503314 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fe188059800 con 0x7fe18c105b50 2024-04-05T21:52:35.786 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:52:35.786 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:52:35.786 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:52:35.786 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:52:35.786 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:52:35.786 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:52:35.786 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:52:35.786 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:52:35.786 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:52:35.786 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:52:35.786 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:52:35.787 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:52:35.788 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:52:35.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.787+0000 7fe193b6c700 1 -- 172.21.15.121:0/2889503314 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe17806cce0 msgr2=0x7fe17806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:35.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.787+0000 7fe193b6c700 1 --2- 172.21.15.121:0/2889503314 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe17806cce0 0x7fe17806f1a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe17c006010 tx=0x7fe17c01a040 comp rx=0 tx=0).stop 2024-04-05T21:52:35.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.787+0000 7fe193b6c700 1 -- 172.21.15.121:0/2889503314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe18c105b50 msgr2=0x7fe18c19ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:35.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.787+0000 7fe193b6c700 1 --2- 172.21.15.121:0/2889503314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe18c105b50 0x7fe18c19ca50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fe18800b4b0 tx=0x7fe18800b870 comp rx=0 tx=0).stop 2024-04-05T21:52:35.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.787+0000 7fe193b6c700 1 -- 172.21.15.121:0/2889503314 shutdown_connections 2024-04-05T21:52:35.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.787+0000 7fe193b6c700 1 --2- 172.21.15.121:0/2889503314 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe17806cce0 0x7fe17806f1a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.787+0000 7fe193b6c700 1 --2- 172.21.15.121:0/2889503314 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe18c0fef70 0x7fe18c19c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.787+0000 7fe193b6c700 1 --2- 172.21.15.121:0/2889503314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe18c105b50 0x7fe18c19ca50 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.787+0000 7fe193b6c700 1 -- 172.21.15.121:0/2889503314 >> 172.21.15.121:0/2889503314 conn(0x7fe18c075e30 msgr2=0x7fe18c104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:35.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.787+0000 7fe193b6c700 1 -- 172.21.15.121:0/2889503314 shutdown_connections 2024-04-05T21:52:35.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.787+0000 7fe193b6c700 1 -- 172.21.15.121:0/2889503314 wait complete. 2024-04-05T21:52:35.790 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:52:35.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.890+0000 7f848001e700 1 -- 172.21.15.121:0/3100812629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8478100da0 msgr2=0x7f8478101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:35.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.890+0000 7f848001e700 1 --2- 172.21.15.121:0/3100812629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8478100da0 0x7f8478101180 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8474009910 tx=0x7f8474009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:35.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.891+0000 7f848001e700 1 -- 172.21.15.121:0/3100812629 shutdown_connections 2024-04-05T21:52:35.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.891+0000 7f848001e700 1 --2- 172.21.15.121:0/3100812629 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8478101750 0x7f84781057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.891+0000 7f848001e700 1 --2- 172.21.15.121:0/3100812629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8478100da0 0x7f8478101180 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.891+0000 7f848001e700 1 -- 172.21.15.121:0/3100812629 >> 172.21.15.121:0/3100812629 conn(0x7f84780fc630 msgr2=0x7f84780fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:35.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.891+0000 7f848001e700 1 -- 172.21.15.121:0/3100812629 shutdown_connections 2024-04-05T21:52:35.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.891+0000 7f848001e700 1 -- 172.21.15.121:0/3100812629 wait complete. 2024-04-05T21:52:35.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.891+0000 7f848001e700 1 Processor -- start 2024-04-05T21:52:35.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.891+0000 7f848001e700 1 -- start start 2024-04-05T21:52:35.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f848001e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8478100da0 0x7f847819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:35.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f848001e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8478101750 0x7f847819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:35.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f848001e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847819f220 con 0x7f8478100da0 2024-04-05T21:52:35.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f848001e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84781986d0 con 0x7f8478101750 2024-04-05T21:52:35.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f847ddba700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8478100da0 0x7f847819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:35.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f847ddba700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8478100da0 0x7f847819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43896/0 (socket says 172.21.15.121:43896) 2024-04-05T21:52:35.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f847ddba700 1 -- 172.21.15.121:0/1100129851 learned_addr learned my addr 172.21.15.121:0/1100129851 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:35.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f847d5b9700 1 --2- 172.21.15.121:0/1100129851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8478101750 0x7f847819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:35.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f847ddba700 1 -- 172.21.15.121:0/1100129851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8478101750 msgr2=0x7f847819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:35.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f847ddba700 1 --2- 172.21.15.121:0/1100129851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8478101750 0x7f847819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:35.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f847ddba700 1 -- 172.21.15.121:0/1100129851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84740095f0 con 0x7f8478100da0 2024-04-05T21:52:35.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f847d5b9700 1 --2- 172.21.15.121:0/1100129851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8478101750 0x7f847819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:52:35.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f847ddba700 1 --2- 172.21.15.121:0/1100129851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8478100da0 0x7f847819e650 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f8474006010 tx=0x7f847400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:35.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f846effd700 1 -- 172.21.15.121:0/1100129851 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f847401d070 con 0x7f8478100da0 2024-04-05T21:52:35.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f848001e700 1 -- 172.21.15.121:0/1100129851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8478198950 con 0x7f8478100da0 2024-04-05T21:52:35.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f846effd700 1 -- 172.21.15.121:0/1100129851 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f847400f460 con 0x7f8478100da0 2024-04-05T21:52:35.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f846effd700 1 -- 172.21.15.121:0/1100129851 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8474021620 con 0x7f8478100da0 2024-04-05T21:52:35.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.892+0000 7f848001e700 1 -- 172.21.15.121:0/1100129851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8478198e40 con 0x7f8478100da0 2024-04-05T21:52:35.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.893+0000 7f848001e700 1 -- 172.21.15.121:0/1100129851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8478104f20 con 0x7f8478100da0 2024-04-05T21:52:35.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.894+0000 7f846effd700 1 -- 172.21.15.121:0/1100129851 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f847400fa40 con 0x7f8478100da0 2024-04-05T21:52:35.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.894+0000 7f846effd700 1 --2- 172.21.15.121:0/1100129851 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f846406cce0 0x7f846406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:35.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.894+0000 7f846effd700 1 -- 172.21.15.121:0/1100129851 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f847408d790 con 0x7f8478100da0 2024-04-05T21:52:35.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.894+0000 7f847d5b9700 1 --2- 172.21.15.121:0/1100129851 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f846406cce0 0x7f846406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:35.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.895+0000 7f847d5b9700 1 --2- 172.21.15.121:0/1100129851 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f846406cce0 0x7f846406f1a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8468005fd0 tx=0x7f8468005ee0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:35.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:35.899+0000 7f846effd700 1 -- 172.21.15.121:0/1100129851 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f847405b290 con 0x7f8478100da0 2024-04-05T21:52:36.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.116+0000 7f848001e700 1 -- 172.21.15.121:0/1100129851 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8478199170 con 0x7f846406cce0 2024-04-05T21:52:36.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.117+0000 7f846effd700 1 -- 172.21.15.121:0/1100129851 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8478199170 con 0x7f846406cce0 2024-04-05T21:52:36.118 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:52:36.118 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:52:36.118 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:52:36.118 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:52:36.118 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:52:36.118 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:52:36.118 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:52:36.118 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:52:36.118 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:52:36.118 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:52:36.118 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:52:36.118 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:52:36.119 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:52:36.119 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:52:36.119 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:52:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.119+0000 7f848001e700 1 -- 172.21.15.121:0/1100129851 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f846406cce0 msgr2=0x7f846406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.119+0000 7f848001e700 1 --2- 172.21.15.121:0/1100129851 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f846406cce0 0x7f846406f1a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8468005fd0 tx=0x7f8468005ee0 comp rx=0 tx=0).stop 2024-04-05T21:52:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.119+0000 7f848001e700 1 -- 172.21.15.121:0/1100129851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8478100da0 msgr2=0x7f847819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.119+0000 7f848001e700 1 --2- 172.21.15.121:0/1100129851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8478100da0 0x7f847819e650 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f8474006010 tx=0x7f847400be20 comp rx=0 tx=0).stop 2024-04-05T21:52:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.119+0000 7f848001e700 1 -- 172.21.15.121:0/1100129851 shutdown_connections 2024-04-05T21:52:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.119+0000 7f848001e700 1 --2- 172.21.15.121:0/1100129851 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f846406cce0 0x7f846406f1a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.119+0000 7f848001e700 1 --2- 172.21.15.121:0/1100129851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8478100da0 0x7f847819e650 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.119+0000 7f848001e700 1 --2- 172.21.15.121:0/1100129851 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8478101750 0x7f847819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.119+0000 7f848001e700 1 -- 172.21.15.121:0/1100129851 >> 172.21.15.121:0/1100129851 conn(0x7f84780fc630 msgr2=0x7f84780fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.119+0000 7f848001e700 1 -- 172.21.15.121:0/1100129851 shutdown_connections 2024-04-05T21:52:36.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.119+0000 7f848001e700 1 -- 172.21.15.121:0/1100129851 wait complete. 2024-04-05T21:52:36.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.222+0000 7f9a38a6f700 1 -- 172.21.15.121:0/2176425807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a340ffa50 msgr2=0x7f9a340ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:36.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.222+0000 7f9a38a6f700 1 --2- 172.21.15.121:0/2176425807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a340ffa50 0x7f9a340ffed0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9a28009910 tx=0x7f9a28009c20 comp rx=0 tx=0).stop 2024-04-05T21:52:36.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.222+0000 7f9a38a6f700 1 -- 172.21.15.121:0/2176425807 shutdown_connections 2024-04-05T21:52:36.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.222+0000 7f9a38a6f700 1 --2- 172.21.15.121:0/2176425807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a340ffa50 0x7f9a340ffed0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:36.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.222+0000 7f9a38a6f700 1 --2- 172.21.15.121:0/2176425807 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a34105a50 0x7f9a34105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:36.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.222+0000 7f9a38a6f700 1 -- 172.21.15.121:0/2176425807 >> 172.21.15.121:0/2176425807 conn(0x7f9a340fb540 msgr2=0x7f9a340fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:36.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.222+0000 7f9a38a6f700 1 -- 172.21.15.121:0/2176425807 shutdown_connections 2024-04-05T21:52:36.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.222+0000 7f9a38a6f700 1 -- 172.21.15.121:0/2176425807 wait complete. 2024-04-05T21:52:36.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.222+0000 7f9a38a6f700 1 Processor -- start 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.222+0000 7f9a38a6f700 1 -- start start 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a38a6f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a340ffa50 0x7f9a3419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a38a6f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a34105a50 0x7f9a3419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a38a6f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3419f290 con 0x7f9a340ffa50 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a38a6f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a34198740 con 0x7f9a34105a50 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a3259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a340ffa50 0x7f9a3419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a3259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a340ffa50 0x7f9a3419e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43912/0 (socket says 172.21.15.121:43912) 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a3259c700 1 -- 172.21.15.121:0/1236450181 learned_addr learned my addr 172.21.15.121:0/1236450181 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a31d9b700 1 --2- 172.21.15.121:0/1236450181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a34105a50 0x7f9a3419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a3259c700 1 -- 172.21.15.121:0/1236450181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a34105a50 msgr2=0x7f9a3419ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a3259c700 1 --2- 172.21.15.121:0/1236450181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a34105a50 0x7f9a3419ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a3259c700 1 -- 172.21.15.121:0/1236450181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a280095f0 con 0x7f9a340ffa50 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a3259c700 1 --2- 172.21.15.121:0/1236450181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a340ffa50 0x7f9a3419e6c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400d8d0 tx=0x7f9a2400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a237fe700 1 -- 172.21.15.121:0/1236450181 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a24009940 con 0x7f9a340ffa50 2024-04-05T21:52:36.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a237fe700 1 -- 172.21.15.121:0/1236450181 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a24010460 con 0x7f9a340ffa50 2024-04-05T21:52:36.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a237fe700 1 -- 172.21.15.121:0/1236450181 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a2400f5f0 con 0x7f9a340ffa50 2024-04-05T21:52:36.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a38a6f700 1 -- 172.21.15.121:0/1236450181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a34198a20 con 0x7f9a340ffa50 2024-04-05T21:52:36.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.223+0000 7f9a38a6f700 1 -- 172.21.15.121:0/1236450181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a34198f70 con 0x7f9a340ffa50 2024-04-05T21:52:36.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.224+0000 7f9a38a6f700 1 -- 172.21.15.121:0/1236450181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3404ea90 con 0x7f9a340ffa50 2024-04-05T21:52:36.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.225+0000 7f9a237fe700 1 -- 172.21.15.121:0/1236450181 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9a240105d0 con 0x7f9a340ffa50 2024-04-05T21:52:36.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.225+0000 7f9a237fe700 1 --2- 172.21.15.121:0/1236450181 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a1c06cce0 0x7f9a1c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:52:36.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.225+0000 7f9a31d9b700 1 --2- 172.21.15.121:0/1236450181 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a1c06cce0 0x7f9a1c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:52:36.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.225+0000 7f9a237fe700 1 -- 172.21.15.121:0/1236450181 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9a2408c5e0 con 0x7f9a340ffa50 2024-04-05T21:52:36.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.226+0000 7f9a31d9b700 1 --2- 172.21.15.121:0/1236450181 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a1c06cce0 0x7f9a1c06f1a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9a34199bb0 tx=0x7f9a2801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:52:36.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.230+0000 7f9a237fe700 1 -- 172.21.15.121:0/1236450181 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9a2405a0e0 con 0x7f9a340ffa50 2024-04-05T21:52:36.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:35 smithi121 ceph-mon[145859]: from='client.35270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:36.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:35 smithi121 ceph-mon[145859]: from='client.35274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:36.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:35 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2033986776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:52:36.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:35 smithi188 ceph-mon[107793]: from='client.35270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:36.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:35 smithi188 ceph-mon[107793]: from='client.35274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:36.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:35 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2033986776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:52:36.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.500+0000 7f9a38a6f700 1 -- 172.21.15.121:0/1236450181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9a34199c50 con 0x7f9a340ffa50 2024-04-05T21:52:36.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.500+0000 7f9a237fe700 1 -- 172.21.15.121:0/1236450181 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f9a24020e10 con 0x7f9a340ffa50 2024-04-05T21:52:36.501 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:52:36.502 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:52:36.502 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:52:36.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:52:36.502 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:52:36.502 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:52:36.502 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:52:36.502 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:52:36.502 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:52:36.503 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:52:36.503 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:52:36.503 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:52:36.503 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:52:36.503 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:52:36.503 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:52:36.503 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:52:36.503 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:52:36.503 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:52:36.503 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:52:36.504 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:52:36.504 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:52:36.504 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:52:36.504 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:52:36.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.502+0000 7f9a38a6f700 1 -- 172.21.15.121:0/1236450181 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a1c06cce0 msgr2=0x7f9a1c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:36.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.502+0000 7f9a38a6f700 1 --2- 172.21.15.121:0/1236450181 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a1c06cce0 0x7f9a1c06f1a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9a34199bb0 tx=0x7f9a2801a040 comp rx=0 tx=0).stop 2024-04-05T21:52:36.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.502+0000 7f9a38a6f700 1 -- 172.21.15.121:0/1236450181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a340ffa50 msgr2=0x7f9a3419e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:52:36.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.502+0000 7f9a38a6f700 1 --2- 172.21.15.121:0/1236450181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a340ffa50 0x7f9a3419e6c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400d8d0 tx=0x7f9a2400dc90 comp rx=0 tx=0).stop 2024-04-05T21:52:36.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.503+0000 7f9a38a6f700 1 -- 172.21.15.121:0/1236450181 shutdown_connections 2024-04-05T21:52:36.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.503+0000 7f9a38a6f700 1 --2- 172.21.15.121:0/1236450181 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a1c06cce0 0x7f9a1c06f1a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:36.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.503+0000 7f9a38a6f700 1 --2- 172.21.15.121:0/1236450181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a340ffa50 0x7f9a3419e6c0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:36.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.503+0000 7f9a38a6f700 1 --2- 172.21.15.121:0/1236450181 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a34105a50 0x7f9a3419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:52:36.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.503+0000 7f9a38a6f700 1 -- 172.21.15.121:0/1236450181 >> 172.21.15.121:0/1236450181 conn(0x7f9a340fb540 msgr2=0x7f9a34068f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:52:36.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.503+0000 7f9a38a6f700 1 -- 172.21.15.121:0/1236450181 shutdown_connections 2024-04-05T21:52:36.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:52:36.503+0000 7f9a38a6f700 1 -- 172.21.15.121:0/1236450181 wait complete. 2024-04-05T21:52:37.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:36 smithi121 ceph-mon[145859]: from='client.35278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:37.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:36 smithi121 ceph-mon[145859]: pgmap v792: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:37.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:36 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2889503314' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:52:37.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:36 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1236450181' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:52:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:36 smithi188 ceph-mon[107793]: from='client.35278 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:36 smithi188 ceph-mon[107793]: pgmap v792: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:36 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2889503314' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:52:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:36 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1236450181' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:52:38.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:37 smithi121 ceph-mon[145859]: from='client.35290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:38.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:37 smithi188 ceph-mon[107793]: from='client.35290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:52:39.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:38 smithi121 ceph-mon[145859]: pgmap v793: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:38 smithi188 ceph-mon[107793]: pgmap v793: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:40 smithi121 ceph-mon[145859]: pgmap v794: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:40 smithi188 ceph-mon[107793]: pgmap v794: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:43.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:42 smithi121 ceph-mon[145859]: pgmap v795: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:42 smithi188 ceph-mon[107793]: pgmap v795: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:45.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:44 smithi121 ceph-mon[145859]: pgmap v796: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:44 smithi188 ceph-mon[107793]: pgmap v796: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:46.086 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:52:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:52:47.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:46 smithi121 ceph-mon[145859]: pgmap v797: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:46 smithi188 ceph-mon[107793]: pgmap v797: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:49.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:48 smithi121 ceph-mon[145859]: pgmap v798: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:48 smithi188 ceph-mon[107793]: pgmap v798: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:51.133 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:50 smithi121 ceph-mon[145859]: pgmap v799: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:50 smithi188 ceph-mon[107793]: pgmap v799: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:53.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:52 smithi121 ceph-mon[145859]: pgmap v800: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:52 smithi188 ceph-mon[107793]: pgmap v800: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:55.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:54 smithi188 ceph-mon[107793]: pgmap v801: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:55.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:54 smithi121 ceph-mon[145859]: pgmap v801: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:56 smithi121 ceph-mon[145859]: pgmap v802: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:56 smithi188 ceph-mon[107793]: pgmap v802: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:59.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:52:58 smithi121 ceph-mon[145859]: pgmap v803: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:52:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:52:58 smithi188 ceph-mon[107793]: pgmap v803: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:01.225 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:00 smithi121 ceph-mon[145859]: pgmap v804: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:01.225 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:53:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:00 smithi188 ceph-mon[107793]: pgmap v804: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:53:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:02 smithi121 ceph-mon[145859]: pgmap v805: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:02 smithi188 ceph-mon[107793]: pgmap v805: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:05.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:04 smithi188 ceph-mon[107793]: pgmap v806: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:05.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:04 smithi121 ceph-mon[145859]: pgmap v806: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:06.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.619+0000 7f83978ea700 1 -- 172.21.15.121:0/157319116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83900ff050 msgr2=0x7f83900ff4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:06.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.619+0000 7f83978ea700 1 --2- 172.21.15.121:0/157319116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83900ff050 0x7f83900ff4d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f838c009910 tx=0x7f838c009c20 comp rx=0 tx=0).stop 2024-04-05T21:53:06.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.619+0000 7f83978ea700 1 -- 172.21.15.121:0/157319116 shutdown_connections 2024-04-05T21:53:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.619+0000 7f83978ea700 1 --2- 172.21.15.121:0/157319116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83900ff050 0x7f83900ff4d0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.619+0000 7f83978ea700 1 --2- 172.21.15.121:0/157319116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8390105b20 0x7f8390105f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.619+0000 7f83978ea700 1 -- 172.21.15.121:0/157319116 >> 172.21.15.121:0/157319116 conn(0x7f8390075e30 msgr2=0x7f83900fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.619+0000 7f83978ea700 1 -- 172.21.15.121:0/157319116 shutdown_connections 2024-04-05T21:53:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.619+0000 7f83978ea700 1 -- 172.21.15.121:0/157319116 wait complete. 2024-04-05T21:53:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.620+0000 7f83978ea700 1 Processor -- start 2024-04-05T21:53:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.620+0000 7f83978ea700 1 -- start start 2024-04-05T21:53:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.620+0000 7f83978ea700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83900ff050 0x7f83901966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.620+0000 7f83978ea700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8390105b20 0x7f8390196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.620+0000 7f83978ea700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8390197200 con 0x7f8390105b20 2024-04-05T21:53:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.620+0000 7f83978ea700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8390197370 con 0x7f83900ff050 2024-04-05T21:53:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.620+0000 7f8394e85700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8390105b20 0x7f8390196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:06.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.620+0000 7f8394e85700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8390105b20 0x7f8390196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55082/0 (socket says 172.21.15.121:55082) 2024-04-05T21:53:06.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.620+0000 7f8394e85700 1 -- 172.21.15.121:0/1619510407 learned_addr learned my addr 172.21.15.121:0/1619510407 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:06.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.620+0000 7f8395686700 1 --2- 172.21.15.121:0/1619510407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83900ff050 0x7f83901966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:06.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.621+0000 7f8394e85700 1 -- 172.21.15.121:0/1619510407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83900ff050 msgr2=0x7f83901966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:06.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.621+0000 7f8394e85700 1 --2- 172.21.15.121:0/1619510407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83900ff050 0x7f83901966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:06.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.621+0000 7f8394e85700 1 -- 172.21.15.121:0/1619510407 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c0095f0 con 0x7f8390105b20 2024-04-05T21:53:06.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.621+0000 7f8394e85700 1 --2- 172.21.15.121:0/1619510407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8390105b20 0x7f8390196c00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f838c006010 tx=0x7f838c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:06.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.621+0000 7f83867fc700 1 -- 172.21.15.121:0/1619510407 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f838c01d070 con 0x7f8390105b20 2024-04-05T21:53:06.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.621+0000 7f83867fc700 1 -- 172.21.15.121:0/1619510407 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f838c00f460 con 0x7f8390105b20 2024-04-05T21:53:06.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.621+0000 7f83978ea700 1 -- 172.21.15.121:0/1619510407 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839019a0b0 con 0x7f8390105b20 2024-04-05T21:53:06.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.621+0000 7f83867fc700 1 -- 172.21.15.121:0/1619510407 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f838c021620 con 0x7f8390105b20 2024-04-05T21:53:06.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.621+0000 7f83978ea700 1 -- 172.21.15.121:0/1619510407 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839019a5a0 con 0x7f8390105b20 2024-04-05T21:53:06.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.622+0000 7f83978ea700 1 -- 172.21.15.121:0/1619510407 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83901090c0 con 0x7f8390105b20 2024-04-05T21:53:06.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.623+0000 7f83867fc700 1 -- 172.21.15.121:0/1619510407 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f838c00f5d0 con 0x7f8390105b20 2024-04-05T21:53:06.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.623+0000 7f83867fc700 1 --2- 172.21.15.121:0/1619510407 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f837c06cce0 0x7f837c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:06.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.623+0000 7f83867fc700 1 -- 172.21.15.121:0/1619510407 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f838c08d6f0 con 0x7f8390105b20 2024-04-05T21:53:06.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.623+0000 7f8395686700 1 --2- 172.21.15.121:0/1619510407 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f837c06cce0 0x7f837c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:06.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.623+0000 7f8395686700 1 --2- 172.21.15.121:0/1619510407 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f837c06cce0 0x7f837c06f1a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f838000ac50 tx=0x7f838000a380 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:06.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.628+0000 7f83867fc700 1 -- 172.21.15.121:0/1619510407 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f838c0577e0 con 0x7f8390105b20 2024-04-05T21:53:06.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.840+0000 7f83978ea700 1 -- 172.21.15.121:0/1619510407 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f839002cce0 con 0x7f837c06cce0 2024-04-05T21:53:06.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.842+0000 7f83867fc700 1 -- 172.21.15.121:0/1619510407 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f839002cce0 con 0x7f837c06cce0 2024-04-05T21:53:06.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.844+0000 7f83978ea700 1 -- 172.21.15.121:0/1619510407 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f837c06cce0 msgr2=0x7f837c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:06.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.844+0000 7f83978ea700 1 --2- 172.21.15.121:0/1619510407 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f837c06cce0 0x7f837c06f1a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f838000ac50 tx=0x7f838000a380 comp rx=0 tx=0).stop 2024-04-05T21:53:06.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.844+0000 7f83978ea700 1 -- 172.21.15.121:0/1619510407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8390105b20 msgr2=0x7f8390196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:06.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.844+0000 7f83978ea700 1 --2- 172.21.15.121:0/1619510407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8390105b20 0x7f8390196c00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f838c006010 tx=0x7f838c00bd70 comp rx=0 tx=0).stop 2024-04-05T21:53:06.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.844+0000 7f83978ea700 1 -- 172.21.15.121:0/1619510407 shutdown_connections 2024-04-05T21:53:06.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.844+0000 7f83978ea700 1 --2- 172.21.15.121:0/1619510407 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f837c06cce0 0x7f837c06f1a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:06.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.844+0000 7f83978ea700 1 --2- 172.21.15.121:0/1619510407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83900ff050 0x7f83901966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:06.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.844+0000 7f83978ea700 1 --2- 172.21.15.121:0/1619510407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8390105b20 0x7f8390196c00 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:06.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.844+0000 7f83978ea700 1 -- 172.21.15.121:0/1619510407 >> 172.21.15.121:0/1619510407 conn(0x7f8390075e30 msgr2=0x7f8390104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:06.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.844+0000 7f83978ea700 1 -- 172.21.15.121:0/1619510407 shutdown_connections 2024-04-05T21:53:06.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.844+0000 7f83978ea700 1 -- 172.21.15.121:0/1619510407 wait complete. 2024-04-05T21:53:06.854 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:53:06.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.948+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/1205771935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6ac102630 msgr2=0x7fa6ac10ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:06.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.948+0000 7fa6b1cbf700 1 --2- 172.21.15.121:0/1205771935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6ac102630 0x7fa6ac10ab20 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009910 tx=0x7fa69c009c20 comp rx=0 tx=0).stop 2024-04-05T21:53:06.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.948+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/1205771935 shutdown_connections 2024-04-05T21:53:06.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.948+0000 7fa6b1cbf700 1 --2- 172.21.15.121:0/1205771935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6ac102630 0x7fa6ac10ab20 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:06.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.948+0000 7fa6b1cbf700 1 --2- 172.21.15.121:0/1205771935 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6ac101d10 0x7fa6ac1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:06.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.948+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/1205771935 >> 172.21.15.121:0/1205771935 conn(0x7fa6ac0fb540 msgr2=0x7fa6ac0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:06.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.948+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/1205771935 shutdown_connections 2024-04-05T21:53:06.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.948+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/1205771935 wait complete. 2024-04-05T21:53:06.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.948+0000 7fa6b1cbf700 1 Processor -- start 2024-04-05T21:53:06.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.948+0000 7fa6b1cbf700 1 -- start start 2024-04-05T21:53:06.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6b1cbf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6ac101d10 0x7fa6ac196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6b1cbf700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6ac102630 0x7fa6ac196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6b1cbf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac197260 con 0x7fa6ac101d10 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6b1cbf700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ac199f90 con 0x7fa6ac102630 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6ab7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6ac101d10 0x7fa6ac196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6ab7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6ac101d10 0x7fa6ac196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55100/0 (socket says 172.21.15.121:55100) 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6ab7fe700 1 -- 172.21.15.121:0/2373664857 learned_addr learned my addr 172.21.15.121:0/2373664857 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6aaffd700 1 --2- 172.21.15.121:0/2373664857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6ac102630 0x7fa6ac196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6ab7fe700 1 -- 172.21.15.121:0/2373664857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6ac102630 msgr2=0x7fa6ac196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6ab7fe700 1 --2- 172.21.15.121:0/2373664857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6ac102630 0x7fa6ac196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6ab7fe700 1 -- 172.21.15.121:0/2373664857 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c0095f0 con 0x7fa6ac101d10 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6ab7fe700 1 --2- 172.21.15.121:0/2373664857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6ac101d10 0x7fa6ac196690 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000c8a0 tx=0x7fa6a000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6a8ff9700 1 -- 172.21.15.121:0/2373664857 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6a000cea0 con 0x7fa6ac101d10 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6a8ff9700 1 -- 172.21.15.121:0/2373664857 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6a0004830 con 0x7fa6ac101d10 2024-04-05T21:53:06.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/2373664857 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ac19a270 con 0x7fa6ac101d10 2024-04-05T21:53:06.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6a8ff9700 1 -- 172.21.15.121:0/2373664857 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6a0005500 con 0x7fa6ac101d10 2024-04-05T21:53:06.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.949+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/2373664857 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ac19a790 con 0x7fa6ac101d10 2024-04-05T21:53:06.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.950+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/2373664857 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ac108220 con 0x7fa6ac101d10 2024-04-05T21:53:06.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.951+0000 7fa6a8ff9700 1 -- 172.21.15.121:0/2373664857 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa6a00049a0 con 0x7fa6ac101d10 2024-04-05T21:53:06.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.951+0000 7fa6a8ff9700 1 --2- 172.21.15.121:0/2373664857 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa69406cce0 0x7fa69406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:06.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.951+0000 7fa6a8ff9700 1 -- 172.21.15.121:0/2373664857 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa6a008c2b0 con 0x7fa6ac101d10 2024-04-05T21:53:06.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.951+0000 7fa6aaffd700 1 --2- 172.21.15.121:0/2373664857 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa69406cce0 0x7fa69406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:06.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.952+0000 7fa6aaffd700 1 --2- 172.21.15.121:0/2373664857 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa69406cce0 0x7fa69406f1a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac197db0 tx=0x7fa69c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:06.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:06.956+0000 7fa6a8ff9700 1 -- 172.21.15.121:0/2373664857 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa6a0056320 con 0x7fa6ac101d10 2024-04-05T21:53:07.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.164+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/2373664857 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6ac19aa10 con 0x7fa69406cce0 2024-04-05T21:53:07.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.165+0000 7fa6a8ff9700 1 -- 172.21.15.121:0/2373664857 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa6ac19aa10 con 0x7fa69406cce0 2024-04-05T21:53:07.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.167+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/2373664857 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa69406cce0 msgr2=0x7fa69406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:07.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.167+0000 7fa6b1cbf700 1 --2- 172.21.15.121:0/2373664857 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa69406cce0 0x7fa69406f1a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac197db0 tx=0x7fa69c01a040 comp rx=0 tx=0).stop 2024-04-05T21:53:07.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.167+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/2373664857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6ac101d10 msgr2=0x7fa6ac196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:07.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.167+0000 7fa6b1cbf700 1 --2- 172.21.15.121:0/2373664857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6ac101d10 0x7fa6ac196690 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000c8a0 tx=0x7fa6a000cc60 comp rx=0 tx=0).stop 2024-04-05T21:53:07.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.167+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/2373664857 shutdown_connections 2024-04-05T21:53:07.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.167+0000 7fa6b1cbf700 1 --2- 172.21.15.121:0/2373664857 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa69406cce0 0x7fa69406f1a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.167+0000 7fa6b1cbf700 1 --2- 172.21.15.121:0/2373664857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6ac101d10 0x7fa6ac196690 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.167+0000 7fa6b1cbf700 1 --2- 172.21.15.121:0/2373664857 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa6ac102630 0x7fa6ac196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.167+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/2373664857 >> 172.21.15.121:0/2373664857 conn(0x7fa6ac0fb540 msgr2=0x7fa6ac105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:07.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.167+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/2373664857 shutdown_connections 2024-04-05T21:53:07.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.167+0000 7fa6b1cbf700 1 -- 172.21.15.121:0/2373664857 wait complete. 2024-04-05T21:53:07.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:06 smithi121 ceph-mon[145859]: pgmap v807: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:07.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.271+0000 7fcdc9df5700 1 -- 172.21.15.121:0/803907197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdc40ffbd0 msgr2=0x7fcdc4100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:07.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.271+0000 7fcdc9df5700 1 --2- 172.21.15.121:0/803907197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdc40ffbd0 0x7fcdc4100050 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4009910 tx=0x7fcdb4009c20 comp rx=0 tx=0).stop 2024-04-05T21:53:07.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.271+0000 7fcdc9df5700 1 -- 172.21.15.121:0/803907197 shutdown_connections 2024-04-05T21:53:07.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.271+0000 7fcdc9df5700 1 --2- 172.21.15.121:0/803907197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdc40ffbd0 0x7fcdc4100050 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.271+0000 7fcdc9df5700 1 --2- 172.21.15.121:0/803907197 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdc4105a40 0x7fcdc4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.271+0000 7fcdc9df5700 1 -- 172.21.15.121:0/803907197 >> 172.21.15.121:0/803907197 conn(0x7fcdc40755a0 msgr2=0x7fcdc40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:07.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.271+0000 7fcdc9df5700 1 -- 172.21.15.121:0/803907197 shutdown_connections 2024-04-05T21:53:07.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.271+0000 7fcdc9df5700 1 -- 172.21.15.121:0/803907197 wait complete. 2024-04-05T21:53:07.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.272+0000 7fcdc9df5700 1 Processor -- start 2024-04-05T21:53:07.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.272+0000 7fcdc9df5700 1 -- start start 2024-04-05T21:53:07.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.272+0000 7fcdc9df5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdc40ffbd0 0x7fcdc4195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:07.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.272+0000 7fcdc9df5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdc4105a40 0x7fcdc4195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:07.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.272+0000 7fcdc9df5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc4196160 con 0x7fcdc4105a40 2024-04-05T21:53:07.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.272+0000 7fcdc9df5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc4199ef0 con 0x7fcdc40ffbd0 2024-04-05T21:53:07.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.272+0000 7fcdc2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdc4105a40 0x7fcdc4195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:07.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.272+0000 7fcdc2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdc4105a40 0x7fcdc4195a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55126/0 (socket says 172.21.15.121:55126) 2024-04-05T21:53:07.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.272+0000 7fcdc2ffd700 1 -- 172.21.15.121:0/686561728 learned_addr learned my addr 172.21.15.121:0/686561728 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:07.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.272+0000 7fcdc37fe700 1 --2- 172.21.15.121:0/686561728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdc40ffbd0 0x7fcdc4195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:07.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.272+0000 7fcdc2ffd700 1 -- 172.21.15.121:0/686561728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdc40ffbd0 msgr2=0x7fcdc4195540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:07.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.272+0000 7fcdc2ffd700 1 --2- 172.21.15.121:0/686561728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdc40ffbd0 0x7fcdc4195540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.273+0000 7fcdc2ffd700 1 -- 172.21.15.121:0/686561728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb40095f0 con 0x7fcdc4105a40 2024-04-05T21:53:07.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.273+0000 7fcdc2ffd700 1 --2- 172.21.15.121:0/686561728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdc4105a40 0x7fcdc4195a80 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40092e0 tx=0x7fcdb400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:07.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.273+0000 7fcdc0ff9700 1 -- 172.21.15.121:0/686561728 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdb401d070 con 0x7fcdc4105a40 2024-04-05T21:53:07.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.273+0000 7fcdc9df5700 1 -- 172.21.15.121:0/686561728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc419a170 con 0x7fcdc4105a40 2024-04-05T21:53:07.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.273+0000 7fcdc0ff9700 1 -- 172.21.15.121:0/686561728 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdb400f460 con 0x7fcdc4105a40 2024-04-05T21:53:07.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.273+0000 7fcdc0ff9700 1 -- 172.21.15.121:0/686561728 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdb4021620 con 0x7fcdc4105a40 2024-04-05T21:53:07.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.273+0000 7fcdc9df5700 1 -- 172.21.15.121:0/686561728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc419a660 con 0x7fcdc4105a40 2024-04-05T21:53:07.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.274+0000 7fcdc9df5700 1 -- 172.21.15.121:0/686561728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc4107fd0 con 0x7fcdc4105a40 2024-04-05T21:53:07.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.274+0000 7fcdc0ff9700 1 -- 172.21.15.121:0/686561728 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcdb400fa00 con 0x7fcdc4105a40 2024-04-05T21:53:07.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.275+0000 7fcdc0ff9700 1 --2- 172.21.15.121:0/686561728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcdac06cce0 0x7fcdac06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:07.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.275+0000 7fcdc0ff9700 1 -- 172.21.15.121:0/686561728 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fcdb408d810 con 0x7fcdc4105a40 2024-04-05T21:53:07.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.275+0000 7fcdc37fe700 1 --2- 172.21.15.121:0/686561728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcdac06cce0 0x7fcdac06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:07.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.275+0000 7fcdc37fe700 1 --2- 172.21.15.121:0/686561728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcdac06cce0 0x7fcdac06f1a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8005950 tx=0x7fcdb800a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:06 smithi188 ceph-mon[107793]: pgmap v807: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:07.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.279+0000 7fcdc0ff9700 1 -- 172.21.15.121:0/686561728 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcdb4057880 con 0x7fcdc4105a40 2024-04-05T21:53:07.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.487+0000 7fcdc9df5700 1 -- 172.21.15.121:0/686561728 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdc402cd60 con 0x7fcdac06cce0 2024-04-05T21:53:07.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.498+0000 7fcdc0ff9700 1 -- 172.21.15.121:0/686561728 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fcdc402cd60 con 0x7fcdac06cce0 2024-04-05T21:53:07.498 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:53:07.499 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (23m) 2m ago 35m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:53:07.499 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (35m) 2m ago 35m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:53:07.499 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (34m) 2m ago 34m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:53:07.499 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (22m) 2m ago 35m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:53:07.499 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (22m) 2m ago 34m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:53:07.499 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (23m) 2m ago 35m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:53:07.499 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (18m) 2m ago 29m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:53:07.499 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (18m) 2m ago 29m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:53:07.499 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (17m) 2m ago 29m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:53:07.500 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (2m) 2m ago 29m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:53:07.500 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (25m) 2m ago 37m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:53:07.500 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (25m) 2m ago 34m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:53:07.500 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (22m) 2m ago 37m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:53:07.500 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (22m) 2m ago 34m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:53:07.500 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (24m) 2m ago 35m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:53:07.501 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (24m) 2m ago 34m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:53:07.501 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (21m) 2m ago 33m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:53:07.501 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (21m) 2m ago 32m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:53:07.501 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (20m) 2m ago 32m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:53:07.501 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (19m) 2m ago 31m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:53:07.501 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (19m) 2m ago 31m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:53:07.501 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (18m) 2m ago 30m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:53:07.502 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (24m) 2m ago 34m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:53:07.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.500+0000 7fcdc9df5700 1 -- 172.21.15.121:0/686561728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcdac06cce0 msgr2=0x7fcdac06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:07.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.500+0000 7fcdc9df5700 1 --2- 172.21.15.121:0/686561728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcdac06cce0 0x7fcdac06f1a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8005950 tx=0x7fcdb800a300 comp rx=0 tx=0).stop 2024-04-05T21:53:07.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.500+0000 7fcdc9df5700 1 -- 172.21.15.121:0/686561728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdc4105a40 msgr2=0x7fcdc4195a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:07.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.500+0000 7fcdc9df5700 1 --2- 172.21.15.121:0/686561728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdc4105a40 0x7fcdc4195a80 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40092e0 tx=0x7fcdb400bd70 comp rx=0 tx=0).stop 2024-04-05T21:53:07.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.500+0000 7fcdc9df5700 1 -- 172.21.15.121:0/686561728 shutdown_connections 2024-04-05T21:53:07.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.500+0000 7fcdc9df5700 1 --2- 172.21.15.121:0/686561728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcdac06cce0 0x7fcdac06f1a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.500+0000 7fcdc9df5700 1 --2- 172.21.15.121:0/686561728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcdc40ffbd0 0x7fcdc4195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.500+0000 7fcdc9df5700 1 --2- 172.21.15.121:0/686561728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdc4105a40 0x7fcdc4195a80 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.500+0000 7fcdc9df5700 1 -- 172.21.15.121:0/686561728 >> 172.21.15.121:0/686561728 conn(0x7fcdc40755a0 msgr2=0x7fcdc40fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:07.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.500+0000 7fcdc9df5700 1 -- 172.21.15.121:0/686561728 shutdown_connections 2024-04-05T21:53:07.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.500+0000 7fcdc9df5700 1 -- 172.21.15.121:0/686561728 wait complete. 2024-04-05T21:53:07.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.601+0000 7f54dfb2d700 1 -- 172.21.15.121:0/2220978227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54d8069020 msgr2=0x7f54d8105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:07.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.601+0000 7f54dfb2d700 1 --2- 172.21.15.121:0/2220978227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54d8069020 0x7f54d8105790 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009910 tx=0x7f54d4009c20 comp rx=0 tx=0).stop 2024-04-05T21:53:07.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.601+0000 7f54dfb2d700 1 -- 172.21.15.121:0/2220978227 shutdown_connections 2024-04-05T21:53:07.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.601+0000 7f54dfb2d700 1 --2- 172.21.15.121:0/2220978227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54d8069020 0x7f54d8105790 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.601+0000 7f54dfb2d700 1 --2- 172.21.15.121:0/2220978227 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54d8068670 0x7f54d8068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.601+0000 7f54dfb2d700 1 -- 172.21.15.121:0/2220978227 >> 172.21.15.121:0/2220978227 conn(0x7f54d80755a0 msgr2=0x7f54d80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:07.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.601+0000 7f54dfb2d700 1 -- 172.21.15.121:0/2220978227 shutdown_connections 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.601+0000 7f54dfb2d700 1 -- 172.21.15.121:0/2220978227 wait complete. 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dfb2d700 1 Processor -- start 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dfb2d700 1 -- start start 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dfb2d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54d8068670 0x7f54d819c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dfb2d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54d8069020 0x7f54d819ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dfb2d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d819d120 con 0x7f54d8069020 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dfb2d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d81965d0 con 0x7f54d8068670 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dd0c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54d8069020 0x7f54d819ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dd0c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54d8069020 0x7f54d819ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55140/0 (socket says 172.21.15.121:55140) 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dd0c8700 1 -- 172.21.15.121:0/3245387657 learned_addr learned my addr 172.21.15.121:0/3245387657 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dd8c9700 1 --2- 172.21.15.121:0/3245387657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54d8068670 0x7f54d819c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dd0c8700 1 -- 172.21.15.121:0/3245387657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54d8068670 msgr2=0x7f54d819c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dd0c8700 1 --2- 172.21.15.121:0/3245387657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54d8068670 0x7f54d819c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dd0c8700 1 -- 172.21.15.121:0/3245387657 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d40095f0 con 0x7f54d8069020 2024-04-05T21:53:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.602+0000 7f54dd0c8700 1 --2- 172.21.15.121:0/3245387657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54d8069020 0x7f54d819ca90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f54d4005f50 tx=0x7f54d400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.603+0000 7f54ceffd700 1 -- 172.21.15.121:0/3245387657 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54d401d070 con 0x7f54d8069020 2024-04-05T21:53:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.603+0000 7f54ceffd700 1 -- 172.21.15.121:0/3245387657 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54d400f460 con 0x7f54d8069020 2024-04-05T21:53:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.603+0000 7f54dfb2d700 1 -- 172.21.15.121:0/3245387657 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d8196850 con 0x7f54d8069020 2024-04-05T21:53:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.603+0000 7f54ceffd700 1 -- 172.21.15.121:0/3245387657 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54d4021640 con 0x7f54d8069020 2024-04-05T21:53:07.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.603+0000 7f54dfb2d700 1 -- 172.21.15.121:0/3245387657 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d8196d40 con 0x7f54d8069020 2024-04-05T21:53:07.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.604+0000 7f54dfb2d700 1 -- 172.21.15.121:0/3245387657 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d81090e0 con 0x7f54d8069020 2024-04-05T21:53:07.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.604+0000 7f54ceffd700 1 -- 172.21.15.121:0/3245387657 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f54d400f9e0 con 0x7f54d8069020 2024-04-05T21:53:07.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.604+0000 7f54ceffd700 1 --2- 172.21.15.121:0/3245387657 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f54c406cce0 0x7f54c406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:07.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.604+0000 7f54ceffd700 1 -- 172.21.15.121:0/3245387657 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f54d408d7f0 con 0x7f54d8069020 2024-04-05T21:53:07.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.605+0000 7f54dd8c9700 1 --2- 172.21.15.121:0/3245387657 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f54c406cce0 0x7f54c406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:07.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.605+0000 7f54dd8c9700 1 --2- 172.21.15.121:0/3245387657 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f54c406cce0 0x7f54c406f1a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f54c8006fd0 tx=0x7f54c8008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:07.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.609+0000 7f54ceffd700 1 -- 172.21.15.121:0/3245387657 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f54d405b370 con 0x7f54d8069020 2024-04-05T21:53:07.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.883+0000 7f54dfb2d700 1 -- 172.21.15.121:0/3245387657 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f54d804ea90 con 0x7f54d8069020 2024-04-05T21:53:07.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.884+0000 7f54ceffd700 1 -- 172.21.15.121:0/3245387657 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f54d405af00 con 0x7f54d8069020 2024-04-05T21:53:07.884 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:53:07.884 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:53:07.884 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:53:07.885 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:53:07.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.886+0000 7f54dfb2d700 1 -- 172.21.15.121:0/3245387657 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f54c406cce0 msgr2=0x7f54c406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:07.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.886+0000 7f54dfb2d700 1 --2- 172.21.15.121:0/3245387657 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f54c406cce0 0x7f54c406f1a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f54c8006fd0 tx=0x7f54c8008040 comp rx=0 tx=0).stop 2024-04-05T21:53:07.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.886+0000 7f54dfb2d700 1 -- 172.21.15.121:0/3245387657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54d8069020 msgr2=0x7f54d819ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:07.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.886+0000 7f54dfb2d700 1 --2- 172.21.15.121:0/3245387657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54d8069020 0x7f54d819ca90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f54d4005f50 tx=0x7f54d400be20 comp rx=0 tx=0).stop 2024-04-05T21:53:07.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.886+0000 7f54dfb2d700 1 -- 172.21.15.121:0/3245387657 shutdown_connections 2024-04-05T21:53:07.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.886+0000 7f54dfb2d700 1 --2- 172.21.15.121:0/3245387657 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f54c406cce0 0x7f54c406f1a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.886+0000 7f54dfb2d700 1 --2- 172.21.15.121:0/3245387657 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54d8068670 0x7f54d819c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.886+0000 7f54dfb2d700 1 --2- 172.21.15.121:0/3245387657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54d8069020 0x7f54d819ca90 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.886+0000 7f54dfb2d700 1 -- 172.21.15.121:0/3245387657 >> 172.21.15.121:0/3245387657 conn(0x7f54d80755a0 msgr2=0x7f54d80fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:07.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.886+0000 7f54dfb2d700 1 -- 172.21.15.121:0/3245387657 shutdown_connections 2024-04-05T21:53:07.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.886+0000 7f54dfb2d700 1 -- 172.21.15.121:0/3245387657 wait complete. 2024-04-05T21:53:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.988+0000 7ff2b7b51700 1 -- 172.21.15.121:0/707804612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2b0105a40 msgr2=0x7ff2b0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:07.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.988+0000 7ff2b7b51700 1 --2- 172.21.15.121:0/707804612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2b0105a40 0x7ff2b0105e20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009910 tx=0x7ff2a4009c20 comp rx=0 tx=0).stop 2024-04-05T21:53:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.988+0000 7ff2b7b51700 1 -- 172.21.15.121:0/707804612 shutdown_connections 2024-04-05T21:53:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.988+0000 7ff2b7b51700 1 --2- 172.21.15.121:0/707804612 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2b00ffbd0 0x7ff2b0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.988+0000 7ff2b7b51700 1 --2- 172.21.15.121:0/707804612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2b0105a40 0x7ff2b0105e20 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.988+0000 7ff2b7b51700 1 -- 172.21.15.121:0/707804612 >> 172.21.15.121:0/707804612 conn(0x7ff2b00755a0 msgr2=0x7ff2b00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.988+0000 7ff2b7b51700 1 -- 172.21.15.121:0/707804612 shutdown_connections 2024-04-05T21:53:07.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.988+0000 7ff2b7b51700 1 -- 172.21.15.121:0/707804612 wait complete. 2024-04-05T21:53:07.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.989+0000 7ff2b7b51700 1 Processor -- start 2024-04-05T21:53:07.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.989+0000 7ff2b7b51700 1 -- start start 2024-04-05T21:53:07.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.989+0000 7ff2b7b51700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2b00ffbd0 0x7ff2b0195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:07.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.989+0000 7ff2b7b51700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2b0105a40 0x7ff2b0195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:07.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.989+0000 7ff2b7b51700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b0196170 con 0x7ff2b0105a40 2024-04-05T21:53:07.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.989+0000 7ff2b7b51700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b0199f00 con 0x7ff2b00ffbd0 2024-04-05T21:53:07.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.989+0000 7ff2b50ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2b0105a40 0x7ff2b0195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:07.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.989+0000 7ff2b50ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2b0105a40 0x7ff2b0195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55156/0 (socket says 172.21.15.121:55156) 2024-04-05T21:53:07.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.989+0000 7ff2b50ec700 1 -- 172.21.15.121:0/1497435724 learned_addr learned my addr 172.21.15.121:0/1497435724 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:07.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.990+0000 7ff2b50ec700 1 -- 172.21.15.121:0/1497435724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2b00ffbd0 msgr2=0x7ff2b0195550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:07.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.990+0000 7ff2b58ed700 1 --2- 172.21.15.121:0/1497435724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2b00ffbd0 0x7ff2b0195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:07.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.990+0000 7ff2b50ec700 1 --2- 172.21.15.121:0/1497435724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2b00ffbd0 0x7ff2b0195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:07.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.990+0000 7ff2b50ec700 1 -- 172.21.15.121:0/1497435724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a40095f0 con 0x7ff2b0105a40 2024-04-05T21:53:07.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.990+0000 7ff2b50ec700 1 --2- 172.21.15.121:0/1497435724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2b0105a40 0x7ff2b0195a90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac007910 tx=0x7ff2ac007cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:07.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.990+0000 7ff2a2ffd700 1 -- 172.21.15.121:0/1497435724 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2ac012070 con 0x7ff2b0105a40 2024-04-05T21:53:07.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.990+0000 7ff2a2ffd700 1 -- 172.21.15.121:0/1497435724 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2ac00d810 con 0x7ff2b0105a40 2024-04-05T21:53:07.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.990+0000 7ff2a2ffd700 1 -- 172.21.15.121:0/1497435724 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2ac0053d0 con 0x7ff2b0105a40 2024-04-05T21:53:07.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.990+0000 7ff2b7b51700 1 -- 172.21.15.121:0/1497435724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b019a180 con 0x7ff2b0105a40 2024-04-05T21:53:07.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.990+0000 7ff2b7b51700 1 -- 172.21.15.121:0/1497435724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b019a6d0 con 0x7ff2b0105a40 2024-04-05T21:53:07.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.991+0000 7ff2b7b51700 1 -- 172.21.15.121:0/1497435724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b004ea90 con 0x7ff2b0105a40 2024-04-05T21:53:07.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.991+0000 7ff2a2ffd700 1 -- 172.21.15.121:0/1497435724 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff2ac016070 con 0x7ff2b0105a40 2024-04-05T21:53:07.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.992+0000 7ff2a2ffd700 1 --2- 172.21.15.121:0/1497435724 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff29c06cce0 0x7ff29c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:07.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.992+0000 7ff2a2ffd700 1 -- 172.21.15.121:0/1497435724 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff2ac08d800 con 0x7ff2b0105a40 2024-04-05T21:53:07.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.992+0000 7ff2b58ed700 1 --2- 172.21.15.121:0/1497435724 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff29c06cce0 0x7ff29c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:07.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.992+0000 7ff2b58ed700 1 --2- 172.21.15.121:0/1497435724 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff29c06cce0 0x7ff29c06f1a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400b5c0 tx=0x7ff2a4005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:07.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:07.996+0000 7ff2a2ffd700 1 -- 172.21.15.121:0/1497435724 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff2ac057850 con 0x7ff2b0105a40 2024-04-05T21:53:08.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.232+0000 7ff2b7b51700 1 -- 172.21.15.121:0/1497435724 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff2b019aad0 con 0x7ff2b0105a40 2024-04-05T21:53:08.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.233+0000 7ff2a2ffd700 1 -- 172.21.15.121:0/1497435724 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7ff2ac01c750 con 0x7ff2b0105a40 2024-04-05T21:53:08.233 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:53:08.233 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:53:08.233 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:53:08.233 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:53:08.234 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:53:08.235 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:53:08.236 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:53:08.236 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:53:08.236 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:53:08.236 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:53:08.236 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:53:08.236 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:53:08.236 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:53:08.236 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:53:08.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.235+0000 7ff2b7b51700 1 -- 172.21.15.121:0/1497435724 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff29c06cce0 msgr2=0x7ff29c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:08.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.235+0000 7ff2b7b51700 1 --2- 172.21.15.121:0/1497435724 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff29c06cce0 0x7ff29c06f1a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400b5c0 tx=0x7ff2a4005fd0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.235+0000 7ff2b7b51700 1 -- 172.21.15.121:0/1497435724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2b0105a40 msgr2=0x7ff2b0195a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:08.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.235+0000 7ff2b7b51700 1 --2- 172.21.15.121:0/1497435724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2b0105a40 0x7ff2b0195a90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac007910 tx=0x7ff2ac007cd0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.235+0000 7ff2b7b51700 1 -- 172.21.15.121:0/1497435724 shutdown_connections 2024-04-05T21:53:08.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.235+0000 7ff2b7b51700 1 --2- 172.21.15.121:0/1497435724 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff29c06cce0 0x7ff29c06f1a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.235+0000 7ff2b7b51700 1 --2- 172.21.15.121:0/1497435724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2b00ffbd0 0x7ff2b0195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.235+0000 7ff2b7b51700 1 --2- 172.21.15.121:0/1497435724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2b0105a40 0x7ff2b0195a90 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.235+0000 7ff2b7b51700 1 -- 172.21.15.121:0/1497435724 >> 172.21.15.121:0/1497435724 conn(0x7ff2b00755a0 msgr2=0x7ff2b00fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:08.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.235+0000 7ff2b7b51700 1 -- 172.21.15.121:0/1497435724 shutdown_connections 2024-04-05T21:53:08.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.235+0000 7ff2b7b51700 1 -- 172.21.15.121:0/1497435724 wait complete. 2024-04-05T21:53:08.238 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:53:08.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:07 smithi121 ceph-mon[145859]: from='client.35298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:08.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:07 smithi188 ceph-mon[107793]: from='client.35298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:08.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.338+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/1853356875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d680ffa40 msgr2=0x7f0d680ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:08.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.338+0000 7f0d6ec9f700 1 --2- 172.21.15.121:0/1853356875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d680ffa40 0x7f0d680ffec0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009910 tx=0x7f0d54009c20 comp rx=0 tx=0).stop 2024-04-05T21:53:08.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.338+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/1853356875 shutdown_connections 2024-04-05T21:53:08.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.338+0000 7f0d6ec9f700 1 --2- 172.21.15.121:0/1853356875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d680ffa40 0x7f0d680ffec0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.338+0000 7f0d6ec9f700 1 --2- 172.21.15.121:0/1853356875 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d68105a40 0x7f0d68105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.338+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/1853356875 >> 172.21.15.121:0/1853356875 conn(0x7f0d680fb570 msgr2=0x7f0d680fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:08.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.338+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/1853356875 shutdown_connections 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.338+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/1853356875 wait complete. 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.338+0000 7f0d6ec9f700 1 Processor -- start 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d6ec9f700 1 -- start start 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d6ec9f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d680ffa40 0x7f0d68196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d6ec9f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d68105a40 0x7f0d68196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d6ec9f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d68197230 con 0x7f0d68105a40 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d6ec9f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6819afc0 con 0x7f0d680ffa40 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d67fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d68105a40 0x7f0d68196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d67fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d68105a40 0x7f0d68196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55166/0 (socket says 172.21.15.121:55166) 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d67fff700 1 -- 172.21.15.121:0/784038507 learned_addr learned my addr 172.21.15.121:0/784038507 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d6ca3b700 1 --2- 172.21.15.121:0/784038507 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d680ffa40 0x7f0d68196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d67fff700 1 -- 172.21.15.121:0/784038507 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d680ffa40 msgr2=0x7f0d68196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d67fff700 1 --2- 172.21.15.121:0/784038507 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d680ffa40 0x7f0d68196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d67fff700 1 -- 172.21.15.121:0/784038507 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d540095f0 con 0x7f0d68105a40 2024-04-05T21:53:08.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d67fff700 1 --2- 172.21.15.121:0/784038507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d68105a40 0x7f0d68196b50 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0d54000c00 tx=0x7f0d5400bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:08.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d65ffb700 1 -- 172.21.15.121:0/784038507 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d5401d070 con 0x7f0d68105a40 2024-04-05T21:53:08.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d65ffb700 1 -- 172.21.15.121:0/784038507 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d54005dc0 con 0x7f0d68105a40 2024-04-05T21:53:08.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/784038507 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6819b240 con 0x7f0d68105a40 2024-04-05T21:53:08.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.339+0000 7f0d65ffb700 1 -- 172.21.15.121:0/784038507 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d54003940 con 0x7f0d68105a40 2024-04-05T21:53:08.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.340+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/784038507 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6819b6d0 con 0x7f0d68105a40 2024-04-05T21:53:08.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.341+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/784038507 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d68108fe0 con 0x7f0d68105a40 2024-04-05T21:53:08.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.341+0000 7f0d65ffb700 1 -- 172.21.15.121:0/784038507 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0d54004ae0 con 0x7f0d68105a40 2024-04-05T21:53:08.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.341+0000 7f0d65ffb700 1 --2- 172.21.15.121:0/784038507 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0d5806cce0 0x7f0d5806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:08.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.341+0000 7f0d65ffb700 1 -- 172.21.15.121:0/784038507 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0d5408db90 con 0x7f0d68105a40 2024-04-05T21:53:08.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.341+0000 7f0d6ca3b700 1 --2- 172.21.15.121:0/784038507 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0d5806cce0 0x7f0d5806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:08.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.342+0000 7f0d6ca3b700 1 --2- 172.21.15.121:0/784038507 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0d5806cce0 0x7f0d5806f1a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c009c60 tx=0x7f0d5c009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:08.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.346+0000 7f0d65ffb700 1 -- 172.21.15.121:0/784038507 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0d54057c00 con 0x7f0d68105a40 2024-04-05T21:53:08.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.555+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/784038507 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d6802cd30 con 0x7f0d5806cce0 2024-04-05T21:53:08.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.556+0000 7f0d65ffb700 1 -- 172.21.15.121:0/784038507 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0d6802cd30 con 0x7f0d5806cce0 2024-04-05T21:53:08.557 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:53:08.557 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:53:08.557 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:53:08.557 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:53:08.557 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:53:08.557 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:53:08.557 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:53:08.557 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:53:08.557 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:53:08.557 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:53:08.557 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:53:08.558 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:53:08.558 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:53:08.558 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:53:08.558 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:53:08.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.558+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/784038507 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0d5806cce0 msgr2=0x7f0d5806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:08.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.558+0000 7f0d6ec9f700 1 --2- 172.21.15.121:0/784038507 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0d5806cce0 0x7f0d5806f1a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c009c60 tx=0x7f0d5c009260 comp rx=0 tx=0).stop 2024-04-05T21:53:08.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.558+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/784038507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d68105a40 msgr2=0x7f0d68196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:08.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.558+0000 7f0d6ec9f700 1 --2- 172.21.15.121:0/784038507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d68105a40 0x7f0d68196b50 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0d54000c00 tx=0x7f0d5400bde0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.558+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/784038507 shutdown_connections 2024-04-05T21:53:08.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.559+0000 7f0d6ec9f700 1 --2- 172.21.15.121:0/784038507 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0d5806cce0 0x7f0d5806f1a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.559+0000 7f0d6ec9f700 1 --2- 172.21.15.121:0/784038507 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0d680ffa40 0x7f0d68196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.559+0000 7f0d6ec9f700 1 --2- 172.21.15.121:0/784038507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d68105a40 0x7f0d68196b50 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.559+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/784038507 >> 172.21.15.121:0/784038507 conn(0x7f0d680fb570 msgr2=0x7f0d68068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:08.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.559+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/784038507 shutdown_connections 2024-04-05T21:53:08.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.559+0000 7f0d6ec9f700 1 -- 172.21.15.121:0/784038507 wait complete. 2024-04-05T21:53:08.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.661+0000 7f42e72c7700 1 -- 172.21.15.121:0/3070720627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e0105a50 msgr2=0x7f42e0105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:08.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.661+0000 7f42e72c7700 1 --2- 172.21.15.121:0/3070720627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e0105a50 0x7f42e0105e30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00b0a0 tx=0x7f42dc00b3b0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.661+0000 7f42e72c7700 1 -- 172.21.15.121:0/3070720627 shutdown_connections 2024-04-05T21:53:08.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.661+0000 7f42e72c7700 1 --2- 172.21.15.121:0/3070720627 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e00ffa50 0x7f42e00ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.661+0000 7f42e72c7700 1 --2- 172.21.15.121:0/3070720627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e0105a50 0x7f42e0105e30 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.661+0000 7f42e72c7700 1 -- 172.21.15.121:0/3070720627 >> 172.21.15.121:0/3070720627 conn(0x7f42e00fb540 msgr2=0x7f42e00fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.661+0000 7f42e72c7700 1 -- 172.21.15.121:0/3070720627 shutdown_connections 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.661+0000 7f42e72c7700 1 -- 172.21.15.121:0/3070720627 wait complete. 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.661+0000 7f42e72c7700 1 Processor -- start 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.661+0000 7f42e72c7700 1 -- start start 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e72c7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e00ffa50 0x7f42e0196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e72c7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e0105a50 0x7f42e0196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e72c7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e0197290 con 0x7f42e00ffa50 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e72c7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e019b020 con 0x7f42e0105a50 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e5063700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e00ffa50 0x7f42e0196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e5063700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e00ffa50 0x7f42e0196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55178/0 (socket says 172.21.15.121:55178) 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e5063700 1 -- 172.21.15.121:0/3009031790 learned_addr learned my addr 172.21.15.121:0/3009031790 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e4862700 1 --2- 172.21.15.121:0/3009031790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e0105a50 0x7f42e0196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:08.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e5063700 1 -- 172.21.15.121:0/3009031790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e0105a50 msgr2=0x7f42e0196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:08.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e5063700 1 --2- 172.21.15.121:0/3009031790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e0105a50 0x7f42e0196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e5063700 1 -- 172.21.15.121:0/3009031790 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc009d00 con 0x7f42e00ffa50 2024-04-05T21:53:08.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e4862700 1 --2- 172.21.15.121:0/3009031790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e0105a50 0x7f42e0196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:53:08.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e5063700 1 --2- 172.21.15.121:0/3009031790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e00ffa50 0x7f42e0196670 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00d010 tx=0x7f42dc003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:08.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42d67fc700 1 -- 172.21.15.121:0/3009031790 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42dc00e040 con 0x7f42e00ffa50 2024-04-05T21:53:08.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e72c7700 1 -- 172.21.15.121:0/3009031790 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e019b2a0 con 0x7f42e00ffa50 2024-04-05T21:53:08.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42d67fc700 1 -- 172.21.15.121:0/3009031790 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42dc0046f0 con 0x7f42e00ffa50 2024-04-05T21:53:08.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42d67fc700 1 -- 172.21.15.121:0/3009031790 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42dc01daf0 con 0x7f42e00ffa50 2024-04-05T21:53:08.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.662+0000 7f42e72c7700 1 -- 172.21.15.121:0/3009031790 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e019b790 con 0x7f42e00ffa50 2024-04-05T21:53:08.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.664+0000 7f42e72c7700 1 -- 172.21.15.121:0/3009031790 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e0109060 con 0x7f42e00ffa50 2024-04-05T21:53:08.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.664+0000 7f42d67fc700 1 -- 172.21.15.121:0/3009031790 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f42dc004860 con 0x7f42e00ffa50 2024-04-05T21:53:08.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.664+0000 7f42d67fc700 1 --2- 172.21.15.121:0/3009031790 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f42cc06cce0 0x7f42cc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:08.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.664+0000 7f42d67fc700 1 -- 172.21.15.121:0/3009031790 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f42dc08e0e0 con 0x7f42e00ffa50 2024-04-05T21:53:08.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.664+0000 7f42e4862700 1 --2- 172.21.15.121:0/3009031790 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f42cc06cce0 0x7f42cc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:08.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.665+0000 7f42e4862700 1 --2- 172.21.15.121:0/3009031790 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f42cc06cce0 0x7f42cc06f1a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f42e0197c90 tx=0x7f42d0005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:08.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.669+0000 7f42d67fc700 1 -- 172.21.15.121:0/3009031790 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f42dc058150 con 0x7f42e00ffa50 2024-04-05T21:53:08.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.944+0000 7f42e72c7700 1 -- 172.21.15.121:0/3009031790 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f42e004ea90 con 0x7f42e00ffa50 2024-04-05T21:53:08.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.944+0000 7f42d67fc700 1 -- 172.21.15.121:0/3009031790 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f42dc017e10 con 0x7f42e00ffa50 2024-04-05T21:53:08.945 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:53:08.945 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:53:08.945 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:53:08.945 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:53:08.945 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:53:08.945 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:53:08.945 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:53:08.945 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:53:08.945 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:53:08.946 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:53:08.946 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:53:08.946 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:53:08.946 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:53:08.946 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:53:08.946 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:53:08.946 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:53:08.946 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:53:08.946 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:53:08.946 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:53:08.946 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:53:08.946 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:53:08.946 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:53:08.947 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:53:08.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.946+0000 7f42e72c7700 1 -- 172.21.15.121:0/3009031790 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f42cc06cce0 msgr2=0x7f42cc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:08.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.946+0000 7f42e72c7700 1 --2- 172.21.15.121:0/3009031790 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f42cc06cce0 0x7f42cc06f1a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f42e0197c90 tx=0x7f42d0005f50 comp rx=0 tx=0).stop 2024-04-05T21:53:08.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.946+0000 7f42e72c7700 1 -- 172.21.15.121:0/3009031790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e00ffa50 msgr2=0x7f42e0196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:08.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.946+0000 7f42e72c7700 1 --2- 172.21.15.121:0/3009031790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e00ffa50 0x7f42e0196670 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00d010 tx=0x7f42dc003ce0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.946+0000 7f42e72c7700 1 -- 172.21.15.121:0/3009031790 shutdown_connections 2024-04-05T21:53:08.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.946+0000 7f42e72c7700 1 --2- 172.21.15.121:0/3009031790 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f42cc06cce0 0x7f42cc06f1a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.946+0000 7f42e72c7700 1 --2- 172.21.15.121:0/3009031790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42e00ffa50 0x7f42e0196670 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.946+0000 7f42e72c7700 1 --2- 172.21.15.121:0/3009031790 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42e0105a50 0x7f42e0196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:08.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.946+0000 7f42e72c7700 1 -- 172.21.15.121:0/3009031790 >> 172.21.15.121:0/3009031790 conn(0x7f42e00fb540 msgr2=0x7f42e0069010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:08.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.946+0000 7f42e72c7700 1 -- 172.21.15.121:0/3009031790 shutdown_connections 2024-04-05T21:53:08.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:08.946+0000 7f42e72c7700 1 -- 172.21.15.121:0/3009031790 wait complete. 2024-04-05T21:53:09.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:08 smithi121 ceph-mon[145859]: from='client.35302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:09.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:08 smithi121 ceph-mon[145859]: from='client.35306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:09.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:08 smithi121 ceph-mon[145859]: pgmap v808: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:09.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:08 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3245387657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:53:09.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:08 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1497435724' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:53:09.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:08 smithi188 ceph-mon[107793]: from='client.35302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:09.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:08 smithi188 ceph-mon[107793]: from='client.35306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:09.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:08 smithi188 ceph-mon[107793]: pgmap v808: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:09.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:08 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3245387657' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:53:09.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:08 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1497435724' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:53:10.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:09 smithi188 ceph-mon[107793]: from='client.35318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:10.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:09 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3009031790' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:53:10.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:09 smithi121 ceph-mon[145859]: from='client.35318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:10.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:09 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3009031790' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:53:11.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:10 smithi121 ceph-mon[145859]: pgmap v809: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:10 smithi188 ceph-mon[107793]: pgmap v809: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:13.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:12 smithi121 ceph-mon[145859]: pgmap v810: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:12 smithi188 ceph-mon[107793]: pgmap v810: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:15.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:14 smithi121 ceph-mon[145859]: pgmap v811: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:14 smithi188 ceph-mon[107793]: pgmap v811: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:53:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:53:17.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:16 smithi121 ceph-mon[145859]: pgmap v812: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:17.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:16 smithi188 ceph-mon[107793]: pgmap v812: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:19.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:18 smithi121 ceph-mon[145859]: pgmap v813: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:19.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:18 smithi188 ceph-mon[107793]: pgmap v813: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:21.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:20 smithi121 ceph-mon[145859]: pgmap v814: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:20 smithi188 ceph-mon[107793]: pgmap v814: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:23.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:22 smithi121 ceph-mon[145859]: pgmap v815: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:23.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:22 smithi188 ceph-mon[107793]: pgmap v815: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:25.126 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:24 smithi121 ceph-mon[145859]: pgmap v816: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:24 smithi188 ceph-mon[107793]: pgmap v816: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:27.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:26 smithi121 ceph-mon[145859]: pgmap v817: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:27.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:26 smithi188 ceph-mon[107793]: pgmap v817: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:29.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:28 smithi121 ceph-mon[145859]: pgmap v818: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:28 smithi188 ceph-mon[107793]: pgmap v818: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:30 smithi121 ceph-mon[145859]: pgmap v819: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:31.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:53:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:30 smithi188 ceph-mon[107793]: pgmap v819: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:31.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:53:33.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:32 smithi121 ceph-mon[145859]: pgmap v820: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:33.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:32 smithi188 ceph-mon[107793]: pgmap v820: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:35.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:34 smithi121 ceph-mon[145859]: pgmap v821: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:35.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:34 smithi188 ceph-mon[107793]: pgmap v821: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:37.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:36 smithi121 ceph-mon[145859]: pgmap v822: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:36 smithi188 ceph-mon[107793]: pgmap v822: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:39.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.062+0000 7f021966f700 1 -- 172.21.15.121:0/1328676687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02140ffa40 msgr2=0x7f02140ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.062+0000 7f021966f700 1 --2- 172.21.15.121:0/1328676687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02140ffa40 0x7f02140ffec0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0204009910 tx=0x7f0204009c20 comp rx=0 tx=0).stop 2024-04-05T21:53:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.062+0000 7f021966f700 1 -- 172.21.15.121:0/1328676687 shutdown_connections 2024-04-05T21:53:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.062+0000 7f021966f700 1 --2- 172.21.15.121:0/1328676687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02140ffa40 0x7f02140ffec0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.062+0000 7f021966f700 1 --2- 172.21.15.121:0/1328676687 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0214105a40 0x7f0214105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.062+0000 7f021966f700 1 -- 172.21.15.121:0/1328676687 >> 172.21.15.121:0/1328676687 conn(0x7f02140fb570 msgr2=0x7f02140fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.062+0000 7f021966f700 1 -- 172.21.15.121:0/1328676687 shutdown_connections 2024-04-05T21:53:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.062+0000 7f021966f700 1 -- 172.21.15.121:0/1328676687 wait complete. 2024-04-05T21:53:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f021966f700 1 Processor -- start 2024-04-05T21:53:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f021966f700 1 -- start start 2024-04-05T21:53:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f021966f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02140ffa40 0x7f02141952a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f021966f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0214105a40 0x7f02141957e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f021966f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0214195d20 con 0x7f0214105a40 2024-04-05T21:53:39.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f021966f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0214195e90 con 0x7f02140ffa40 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f02127fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0214105a40 0x7f02141957e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f02127fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0214105a40 0x7f02141957e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40944/0 (socket says 172.21.15.121:40944) 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f02127fc700 1 -- 172.21.15.121:0/1947442003 learned_addr learned my addr 172.21.15.121:0/1947442003 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f0212ffd700 1 --2- 172.21.15.121:0/1947442003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02140ffa40 0x7f02141952a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f02127fc700 1 -- 172.21.15.121:0/1947442003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02140ffa40 msgr2=0x7f02141952a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f02127fc700 1 --2- 172.21.15.121:0/1947442003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02140ffa40 0x7f02141952a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f02127fc700 1 -- 172.21.15.121:0/1947442003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02040095f0 con 0x7f0214105a40 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.063+0000 7f0212ffd700 1 --2- 172.21.15.121:0/1947442003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02140ffa40 0x7f02141952a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.064+0000 7f02127fc700 1 --2- 172.21.15.121:0/1947442003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0214105a40 0x7f02141957e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0204006010 tx=0x7f020400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.064+0000 7f01fbfff700 1 -- 172.21.15.121:0/1947442003 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f020401d070 con 0x7f0214105a40 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.064+0000 7f01fbfff700 1 -- 172.21.15.121:0/1947442003 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f020400f460 con 0x7f0214105a40 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.064+0000 7f021966f700 1 -- 172.21.15.121:0/1947442003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0214196110 con 0x7f0214105a40 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.064+0000 7f01fbfff700 1 -- 172.21.15.121:0/1947442003 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0204021620 con 0x7f0214105a40 2024-04-05T21:53:39.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.064+0000 7f021966f700 1 -- 172.21.15.121:0/1947442003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02141a2b90 con 0x7f0214105a40 2024-04-05T21:53:39.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.065+0000 7f021966f700 1 -- 172.21.15.121:0/1947442003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0214107fd0 con 0x7f0214105a40 2024-04-05T21:53:39.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.065+0000 7f01fbfff700 1 -- 172.21.15.121:0/1947442003 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f020400fa40 con 0x7f0214105a40 2024-04-05T21:53:39.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.066+0000 7f01fbfff700 1 --2- 172.21.15.121:0/1947442003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f01fc06cce0 0x7f01fc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:39.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.066+0000 7f01fbfff700 1 -- 172.21.15.121:0/1947442003 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f020408d620 con 0x7f0214105a40 2024-04-05T21:53:39.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.066+0000 7f0212ffd700 1 --2- 172.21.15.121:0/1947442003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f01fc06cce0 0x7f01fc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:39.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.066+0000 7f0212ffd700 1 --2- 172.21.15.121:0/1947442003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f01fc06cce0 0x7f01fc06f1a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0208005fd0 tx=0x7f0208005ee0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:39.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.070+0000 7f01fbfff700 1 -- 172.21.15.121:0/1947442003 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f020405b120 con 0x7f0214105a40 2024-04-05T21:53:39.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:38 smithi121 ceph-mon[145859]: pgmap v823: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:38 smithi188 ceph-mon[107793]: pgmap v823: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:39.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.283+0000 7f021966f700 1 -- 172.21.15.121:0/1947442003 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0214196110 con 0x7f01fc06cce0 2024-04-05T21:53:39.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.284+0000 7f01fbfff700 1 -- 172.21.15.121:0/1947442003 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f0214196110 con 0x7f01fc06cce0 2024-04-05T21:53:39.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.286+0000 7f021966f700 1 -- 172.21.15.121:0/1947442003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f01fc06cce0 msgr2=0x7f01fc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:39.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.287+0000 7f021966f700 1 --2- 172.21.15.121:0/1947442003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f01fc06cce0 0x7f01fc06f1a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0208005fd0 tx=0x7f0208005ee0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.287+0000 7f021966f700 1 -- 172.21.15.121:0/1947442003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0214105a40 msgr2=0x7f02141957e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:39.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.287+0000 7f021966f700 1 --2- 172.21.15.121:0/1947442003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0214105a40 0x7f02141957e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0204006010 tx=0x7f020400be20 comp rx=0 tx=0).stop 2024-04-05T21:53:39.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.287+0000 7f021966f700 1 -- 172.21.15.121:0/1947442003 shutdown_connections 2024-04-05T21:53:39.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.287+0000 7f021966f700 1 --2- 172.21.15.121:0/1947442003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f01fc06cce0 0x7f01fc06f1a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.287+0000 7f021966f700 1 --2- 172.21.15.121:0/1947442003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f02140ffa40 0x7f02141952a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.287+0000 7f021966f700 1 --2- 172.21.15.121:0/1947442003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0214105a40 0x7f02141957e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.287+0000 7f021966f700 1 -- 172.21.15.121:0/1947442003 >> 172.21.15.121:0/1947442003 conn(0x7f02140fb570 msgr2=0x7f02140fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:39.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.287+0000 7f021966f700 1 -- 172.21.15.121:0/1947442003 shutdown_connections 2024-04-05T21:53:39.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.287+0000 7f021966f700 1 -- 172.21.15.121:0/1947442003 wait complete. 2024-04-05T21:53:39.300 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:53:39.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.395+0000 7ff930db4700 1 -- 172.21.15.121:0/3557855688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c0ffbd0 msgr2=0x7ff92c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:39.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.395+0000 7ff930db4700 1 --2- 172.21.15.121:0/3557855688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c0ffbd0 0x7ff92c100050 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff920009910 tx=0x7ff920009c20 comp rx=0 tx=0).stop 2024-04-05T21:53:39.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.395+0000 7ff930db4700 1 -- 172.21.15.121:0/3557855688 shutdown_connections 2024-04-05T21:53:39.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.395+0000 7ff930db4700 1 --2- 172.21.15.121:0/3557855688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c0ffbd0 0x7ff92c100050 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.395+0000 7ff930db4700 1 --2- 172.21.15.121:0/3557855688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff92c105a40 0x7ff92c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.395+0000 7ff930db4700 1 -- 172.21.15.121:0/3557855688 >> 172.21.15.121:0/3557855688 conn(0x7ff92c0755a0 msgr2=0x7ff92c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:39.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.395+0000 7ff930db4700 1 -- 172.21.15.121:0/3557855688 shutdown_connections 2024-04-05T21:53:39.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.395+0000 7ff930db4700 1 -- 172.21.15.121:0/3557855688 wait complete. 2024-04-05T21:53:39.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.395+0000 7ff930db4700 1 Processor -- start 2024-04-05T21:53:39.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.395+0000 7ff930db4700 1 -- start start 2024-04-05T21:53:39.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff930db4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c0ffbd0 0x7ff92c1974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:39.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff930db4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff92c105a40 0x7ff92c1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:39.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff930db4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c197fc0 con 0x7ff92c0ffbd0 2024-04-05T21:53:39.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff930db4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c198100 con 0x7ff92c105a40 2024-04-05T21:53:39.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff92a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c0ffbd0 0x7ff92c1974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:39.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff92a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c0ffbd0 0x7ff92c1974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40962/0 (socket says 172.21.15.121:40962) 2024-04-05T21:53:39.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff92a59c700 1 -- 172.21.15.121:0/4029102610 learned_addr learned my addr 172.21.15.121:0/4029102610 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:39.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff929d9b700 1 --2- 172.21.15.121:0/4029102610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff92c105a40 0x7ff92c1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:39.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff92a59c700 1 -- 172.21.15.121:0/4029102610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff92c105a40 msgr2=0x7ff92c1979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:39.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff92a59c700 1 --2- 172.21.15.121:0/4029102610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff92c105a40 0x7ff92c1979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff92a59c700 1 -- 172.21.15.121:0/4029102610 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9200095f0 con 0x7ff92c0ffbd0 2024-04-05T21:53:39.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff92a59c700 1 --2- 172.21.15.121:0/4029102610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c0ffbd0 0x7ff92c1974b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff91c00c8a0 tx=0x7ff91c00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:39.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff91b7fe700 1 -- 172.21.15.121:0/4029102610 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff91c0078e0 con 0x7ff92c0ffbd0 2024-04-05T21:53:39.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff91b7fe700 1 -- 172.21.15.121:0/4029102610 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff91c00f450 con 0x7ff92c0ffbd0 2024-04-05T21:53:39.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff930db4700 1 -- 172.21.15.121:0/4029102610 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92c1a0520 con 0x7ff92c0ffbd0 2024-04-05T21:53:39.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff91b7fe700 1 -- 172.21.15.121:0/4029102610 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff91c00e5e0 con 0x7ff92c0ffbd0 2024-04-05T21:53:39.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.396+0000 7ff930db4700 1 -- 172.21.15.121:0/4029102610 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92c1a0990 con 0x7ff92c0ffbd0 2024-04-05T21:53:39.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.398+0000 7ff930db4700 1 -- 172.21.15.121:0/4029102610 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92c107fd0 con 0x7ff92c0ffbd0 2024-04-05T21:53:39.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.398+0000 7ff91b7fe700 1 -- 172.21.15.121:0/4029102610 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff91c007a40 con 0x7ff92c0ffbd0 2024-04-05T21:53:39.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.398+0000 7ff91b7fe700 1 --2- 172.21.15.121:0/4029102610 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff91406cce0 0x7ff91406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:39.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.398+0000 7ff91b7fe700 1 -- 172.21.15.121:0/4029102610 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff91c08c1a0 con 0x7ff92c0ffbd0 2024-04-05T21:53:39.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.398+0000 7ff929d9b700 1 --2- 172.21.15.121:0/4029102610 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff91406cce0 0x7ff91406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:39.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.399+0000 7ff929d9b700 1 --2- 172.21.15.121:0/4029102610 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff91406cce0 0x7ff91406f1a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff92c068c50 tx=0x7ff92001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:39.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.403+0000 7ff91b7fe700 1 -- 172.21.15.121:0/4029102610 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff91c056210 con 0x7ff92c0ffbd0 2024-04-05T21:53:39.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.616+0000 7ff930db4700 1 -- 172.21.15.121:0/4029102610 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff92c02cce0 con 0x7ff91406cce0 2024-04-05T21:53:39.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.617+0000 7ff91b7fe700 1 -- 172.21.15.121:0/4029102610 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff92c02cce0 con 0x7ff91406cce0 2024-04-05T21:53:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.619+0000 7ff930db4700 1 -- 172.21.15.121:0/4029102610 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff91406cce0 msgr2=0x7ff91406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.619+0000 7ff930db4700 1 --2- 172.21.15.121:0/4029102610 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff91406cce0 0x7ff91406f1a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff92c068c50 tx=0x7ff92001a040 comp rx=0 tx=0).stop 2024-04-05T21:53:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.619+0000 7ff930db4700 1 -- 172.21.15.121:0/4029102610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c0ffbd0 msgr2=0x7ff92c1974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.619+0000 7ff930db4700 1 --2- 172.21.15.121:0/4029102610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c0ffbd0 0x7ff92c1974b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff91c00c8a0 tx=0x7ff91c00cbb0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.619+0000 7ff930db4700 1 -- 172.21.15.121:0/4029102610 shutdown_connections 2024-04-05T21:53:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.619+0000 7ff930db4700 1 --2- 172.21.15.121:0/4029102610 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff91406cce0 0x7ff91406f1a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.619+0000 7ff930db4700 1 --2- 172.21.15.121:0/4029102610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c0ffbd0 0x7ff92c1974b0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.619+0000 7ff930db4700 1 --2- 172.21.15.121:0/4029102610 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff92c105a40 0x7ff92c1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.619+0000 7ff930db4700 1 -- 172.21.15.121:0/4029102610 >> 172.21.15.121:0/4029102610 conn(0x7ff92c0755a0 msgr2=0x7ff92c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.619+0000 7ff930db4700 1 -- 172.21.15.121:0/4029102610 shutdown_connections 2024-04-05T21:53:39.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.619+0000 7ff930db4700 1 -- 172.21.15.121:0/4029102610 wait complete. 2024-04-05T21:53:39.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.721+0000 7f7eeab95700 1 -- 172.21.15.121:0/1657840249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee40ffbd0 msgr2=0x7f7ee4100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:39.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.721+0000 7f7eeab95700 1 --2- 172.21.15.121:0/1657840249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee40ffbd0 0x7f7ee4100050 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009910 tx=0x7f7ed0009c20 comp rx=0 tx=0).stop 2024-04-05T21:53:39.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.721+0000 7f7eeab95700 1 -- 172.21.15.121:0/1657840249 shutdown_connections 2024-04-05T21:53:39.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.721+0000 7f7eeab95700 1 --2- 172.21.15.121:0/1657840249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee40ffbd0 0x7f7ee4100050 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.721+0000 7f7eeab95700 1 --2- 172.21.15.121:0/1657840249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ee4105a40 0x7f7ee4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.721+0000 7f7eeab95700 1 -- 172.21.15.121:0/1657840249 >> 172.21.15.121:0/1657840249 conn(0x7f7ee40755a0 msgr2=0x7f7ee40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:39.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.722+0000 7f7eeab95700 1 -- 172.21.15.121:0/1657840249 shutdown_connections 2024-04-05T21:53:39.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.722+0000 7f7eeab95700 1 -- 172.21.15.121:0/1657840249 wait complete. 2024-04-05T21:53:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.722+0000 7f7eeab95700 1 Processor -- start 2024-04-05T21:53:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.722+0000 7f7eeab95700 1 -- start start 2024-04-05T21:53:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.722+0000 7f7eeab95700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ee40ffbd0 0x7f7ee4195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.722+0000 7f7eeab95700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4105a40 0x7f7ee4195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.722+0000 7f7eeab95700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee4196160 con 0x7f7ee4105a40 2024-04-05T21:53:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.722+0000 7f7eeab95700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee4199ef0 con 0x7f7ee40ffbd0 2024-04-05T21:53:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.722+0000 7f7ee3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4105a40 0x7f7ee4195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:39.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.722+0000 7f7ee3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4105a40 0x7f7ee4195a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40986/0 (socket says 172.21.15.121:40986) 2024-04-05T21:53:39.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.723+0000 7f7ee3fff700 1 -- 172.21.15.121:0/2634367904 learned_addr learned my addr 172.21.15.121:0/2634367904 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:39.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.723+0000 7f7ee8931700 1 --2- 172.21.15.121:0/2634367904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ee40ffbd0 0x7f7ee4195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:39.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.723+0000 7f7ee3fff700 1 -- 172.21.15.121:0/2634367904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ee40ffbd0 msgr2=0x7f7ee4195540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:39.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.723+0000 7f7ee3fff700 1 --2- 172.21.15.121:0/2634367904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ee40ffbd0 0x7f7ee4195540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.723+0000 7f7ee3fff700 1 -- 172.21.15.121:0/2634367904 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed00095f0 con 0x7f7ee4105a40 2024-04-05T21:53:39.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.723+0000 7f7ee3fff700 1 --2- 172.21.15.121:0/2634367904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4105a40 0x7f7ee4195a80 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0006010 tx=0x7f7ed000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:39.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.723+0000 7f7ee1ffb700 1 -- 172.21.15.121:0/2634367904 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ed001d070 con 0x7f7ee4105a40 2024-04-05T21:53:39.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.723+0000 7f7eeab95700 1 -- 172.21.15.121:0/2634367904 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee419a170 con 0x7f7ee4105a40 2024-04-05T21:53:39.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.723+0000 7f7ee1ffb700 1 -- 172.21.15.121:0/2634367904 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ed000f460 con 0x7f7ee4105a40 2024-04-05T21:53:39.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.723+0000 7f7ee1ffb700 1 -- 172.21.15.121:0/2634367904 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ed0021620 con 0x7f7ee4105a40 2024-04-05T21:53:39.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.723+0000 7f7eeab95700 1 -- 172.21.15.121:0/2634367904 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee419a660 con 0x7f7ee4105a40 2024-04-05T21:53:39.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.724+0000 7f7eeab95700 1 -- 172.21.15.121:0/2634367904 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee4107fd0 con 0x7f7ee4105a40 2024-04-05T21:53:39.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.725+0000 7f7ee1ffb700 1 -- 172.21.15.121:0/2634367904 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7ed000fa40 con 0x7f7ee4105a40 2024-04-05T21:53:39.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.725+0000 7f7ee1ffb700 1 --2- 172.21.15.121:0/2634367904 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ed406cce0 0x7f7ed406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:39.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.726+0000 7f7ee1ffb700 1 -- 172.21.15.121:0/2634367904 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7ed008d830 con 0x7f7ee4105a40 2024-04-05T21:53:39.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.726+0000 7f7ee8931700 1 --2- 172.21.15.121:0/2634367904 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ed406cce0 0x7f7ed406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:39.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.726+0000 7f7ee8931700 1 --2- 172.21.15.121:0/2634367904 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ed406cce0 0x7f7ed406f1a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8005950 tx=0x7f7ed80058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:39.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.730+0000 7f7ee1ffb700 1 -- 172.21.15.121:0/2634367904 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7ed0057800 con 0x7f7ee4105a40 2024-04-05T21:53:39.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.932+0000 7f7eeab95700 1 -- 172.21.15.121:0/2634367904 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ee402cdb0 con 0x7f7ed406cce0 2024-04-05T21:53:39.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.942+0000 7f7ee1ffb700 1 -- 172.21.15.121:0/2634367904 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f7ee402cdb0 con 0x7f7ed406cce0 2024-04-05T21:53:39.943 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:53:39.943 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (24m) 2m ago 35m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:53:39.943 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (36m) 2m ago 36m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:53:39.943 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (35m) 3m ago 35m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:53:39.943 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (22m) 2m ago 36m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:53:39.943 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (22m) 3m ago 35m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:53:39.943 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (24m) 2m ago 35m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:53:39.943 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (18m) 2m ago 29m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (18m) 2m ago 29m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (18m) 3m ago 29m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (3m) 3m ago 29m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (26m) 2m ago 37m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (25m) 3m ago 34m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (23m) 2m ago 37m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (22m) 3m ago 34m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (25m) 2m ago 36m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (25m) 3m ago 34m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (22m) 2m ago 33m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (21m) 2m ago 33m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (20m) 2m ago 32m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (20m) 3m ago 32m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (19m) 3m ago 31m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:53:39.944 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (19m) 3m ago 31m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:53:39.945 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (24m) 2m ago 35m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:53:39.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.944+0000 7f7eeab95700 1 -- 172.21.15.121:0/2634367904 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ed406cce0 msgr2=0x7f7ed406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:39.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.944+0000 7f7eeab95700 1 --2- 172.21.15.121:0/2634367904 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ed406cce0 0x7f7ed406f1a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8005950 tx=0x7f7ed80058e0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.944+0000 7f7eeab95700 1 -- 172.21.15.121:0/2634367904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4105a40 msgr2=0x7f7ee4195a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:39.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.944+0000 7f7eeab95700 1 --2- 172.21.15.121:0/2634367904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4105a40 0x7f7ee4195a80 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0006010 tx=0x7f7ed000be20 comp rx=0 tx=0).stop 2024-04-05T21:53:39.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.944+0000 7f7eeab95700 1 -- 172.21.15.121:0/2634367904 shutdown_connections 2024-04-05T21:53:39.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.945+0000 7f7eeab95700 1 --2- 172.21.15.121:0/2634367904 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ed406cce0 0x7f7ed406f1a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.945+0000 7f7eeab95700 1 --2- 172.21.15.121:0/2634367904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ee40ffbd0 0x7f7ee4195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.945+0000 7f7eeab95700 1 --2- 172.21.15.121:0/2634367904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee4105a40 0x7f7ee4195a80 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:39.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.945+0000 7f7eeab95700 1 -- 172.21.15.121:0/2634367904 >> 172.21.15.121:0/2634367904 conn(0x7f7ee40755a0 msgr2=0x7f7ee40fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:39.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.945+0000 7f7eeab95700 1 -- 172.21.15.121:0/2634367904 shutdown_connections 2024-04-05T21:53:39.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:39.945+0000 7f7eeab95700 1 -- 172.21.15.121:0/2634367904 wait complete. 2024-04-05T21:53:40.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.045+0000 7f814eb18700 1 -- 172.21.15.121:0/1591028849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8148105a40 msgr2=0x7f8148105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.045+0000 7f814eb18700 1 --2- 172.21.15.121:0/1591028849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8148105a40 0x7f8148105e20 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f813c009910 tx=0x7f813c009c20 comp rx=0 tx=0).stop 2024-04-05T21:53:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.045+0000 7f814eb18700 1 -- 172.21.15.121:0/1591028849 shutdown_connections 2024-04-05T21:53:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.045+0000 7f814eb18700 1 --2- 172.21.15.121:0/1591028849 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81480ffa40 0x7f81480ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.045+0000 7f814eb18700 1 --2- 172.21.15.121:0/1591028849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8148105a40 0x7f8148105e20 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.045+0000 7f814eb18700 1 -- 172.21.15.121:0/1591028849 >> 172.21.15.121:0/1591028849 conn(0x7f81480fb570 msgr2=0x7f81480fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.046+0000 7f814eb18700 1 -- 172.21.15.121:0/1591028849 shutdown_connections 2024-04-05T21:53:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.046+0000 7f814eb18700 1 -- 172.21.15.121:0/1591028849 wait complete. 2024-04-05T21:53:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.046+0000 7f814eb18700 1 Processor -- start 2024-04-05T21:53:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.046+0000 7f814eb18700 1 -- start start 2024-04-05T21:53:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.046+0000 7f814eb18700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81480ffa40 0x7f81481974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.046+0000 7f814eb18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8148105a40 0x7f8148197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.046+0000 7f814eb18700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8148198040 con 0x7f8148105a40 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.046+0000 7f814eb18700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8148198180 con 0x7f81480ffa40 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.046+0000 7f8147fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8148105a40 0x7f8148197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.046+0000 7f8147fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8148105a40 0x7f8148197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41006/0 (socket says 172.21.15.121:41006) 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.046+0000 7f8147fff700 1 -- 172.21.15.121:0/1200501832 learned_addr learned my addr 172.21.15.121:0/1200501832 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.047+0000 7f814c8b4700 1 --2- 172.21.15.121:0/1200501832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81480ffa40 0x7f81481974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.047+0000 7f8147fff700 1 -- 172.21.15.121:0/1200501832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81480ffa40 msgr2=0x7f81481974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.047+0000 7f8147fff700 1 --2- 172.21.15.121:0/1200501832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81480ffa40 0x7f81481974e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.047+0000 7f8147fff700 1 -- 172.21.15.121:0/1200501832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c0095f0 con 0x7f8148105a40 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.047+0000 7f8147fff700 1 --2- 172.21.15.121:0/1200501832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8148105a40 0x7f8148197a20 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f813400d900 tx=0x7f813400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.047+0000 7f8145ffb700 1 -- 172.21.15.121:0/1200501832 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8134004a00 con 0x7f8148105a40 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.047+0000 7f8145ffb700 1 -- 172.21.15.121:0/1200501832 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8134005020 con 0x7f8148105a40 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.047+0000 7f8145ffb700 1 -- 172.21.15.121:0/1200501832 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8134009b90 con 0x7f8148105a40 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.047+0000 7f814eb18700 1 -- 172.21.15.121:0/1200501832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81481a0550 con 0x7f8148105a40 2024-04-05T21:53:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.047+0000 7f814eb18700 1 -- 172.21.15.121:0/1200501832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81481a0a10 con 0x7f8148105a40 2024-04-05T21:53:40.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.048+0000 7f814eb18700 1 -- 172.21.15.121:0/1200501832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f814804ea90 con 0x7f8148105a40 2024-04-05T21:53:40.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.049+0000 7f8145ffb700 1 -- 172.21.15.121:0/1200501832 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8134009cf0 con 0x7f8148105a40 2024-04-05T21:53:40.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.049+0000 7f8145ffb700 1 --2- 172.21.15.121:0/1200501832 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f813806cce0 0x7f813806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:40.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.049+0000 7f8145ffb700 1 -- 172.21.15.121:0/1200501832 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f813408c3f0 con 0x7f8148105a40 2024-04-05T21:53:40.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.049+0000 7f814c8b4700 1 --2- 172.21.15.121:0/1200501832 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f813806cce0 0x7f813806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:40.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.049+0000 7f814c8b4700 1 --2- 172.21.15.121:0/1200501832 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f813806cce0 0x7f813806f1a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f813c006010 tx=0x7f813c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:40.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.053+0000 7f8145ffb700 1 -- 172.21.15.121:0/1200501832 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f81340564e0 con 0x7f8148105a40 2024-04-05T21:53:40.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.326+0000 7f814eb18700 1 -- 172.21.15.121:0/1200501832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f8148069290 con 0x7f8148105a40 2024-04-05T21:53:40.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.326+0000 7f8145ffb700 1 -- 172.21.15.121:0/1200501832 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f8134059b00 con 0x7f8148105a40 2024-04-05T21:53:40.326 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:53:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:53:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:53:40.328 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:53:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.328+0000 7f814eb18700 1 -- 172.21.15.121:0/1200501832 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f813806cce0 msgr2=0x7f813806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.328+0000 7f814eb18700 1 --2- 172.21.15.121:0/1200501832 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f813806cce0 0x7f813806f1a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f813c006010 tx=0x7f813c01a040 comp rx=0 tx=0).stop 2024-04-05T21:53:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.328+0000 7f814eb18700 1 -- 172.21.15.121:0/1200501832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8148105a40 msgr2=0x7f8148197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.328+0000 7f814eb18700 1 --2- 172.21.15.121:0/1200501832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8148105a40 0x7f8148197a20 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f813400d900 tx=0x7f813400dc10 comp rx=0 tx=0).stop 2024-04-05T21:53:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.328+0000 7f814eb18700 1 -- 172.21.15.121:0/1200501832 shutdown_connections 2024-04-05T21:53:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.328+0000 7f814eb18700 1 --2- 172.21.15.121:0/1200501832 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f813806cce0 0x7f813806f1a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.328+0000 7f814eb18700 1 --2- 172.21.15.121:0/1200501832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81480ffa40 0x7f81481974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.328+0000 7f814eb18700 1 --2- 172.21.15.121:0/1200501832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8148105a40 0x7f8148197a20 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.328+0000 7f814eb18700 1 -- 172.21.15.121:0/1200501832 >> 172.21.15.121:0/1200501832 conn(0x7f81480fb570 msgr2=0x7f81480fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.328+0000 7f814eb18700 1 -- 172.21.15.121:0/1200501832 shutdown_connections 2024-04-05T21:53:40.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.328+0000 7f814eb18700 1 -- 172.21.15.121:0/1200501832 wait complete. 2024-04-05T21:53:40.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.431+0000 7fbecc014700 1 -- 172.21.15.121:0/4238436945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbec40ffbd0 msgr2=0x7fbec4100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:40.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.431+0000 7fbecc014700 1 --2- 172.21.15.121:0/4238436945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbec40ffbd0 0x7fbec4100050 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009910 tx=0x7fbeb4009c20 comp rx=0 tx=0).stop 2024-04-05T21:53:40.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.431+0000 7fbecc014700 1 -- 172.21.15.121:0/4238436945 shutdown_connections 2024-04-05T21:53:40.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.431+0000 7fbecc014700 1 --2- 172.21.15.121:0/4238436945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbec40ffbd0 0x7fbec4100050 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.431+0000 7fbecc014700 1 --2- 172.21.15.121:0/4238436945 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbec4105a40 0x7fbec4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.431+0000 7fbecc014700 1 -- 172.21.15.121:0/4238436945 >> 172.21.15.121:0/4238436945 conn(0x7fbec40755a0 msgr2=0x7fbec40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:40.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.431+0000 7fbecc014700 1 -- 172.21.15.121:0/4238436945 shutdown_connections 2024-04-05T21:53:40.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.431+0000 7fbecc014700 1 -- 172.21.15.121:0/4238436945 wait complete. 2024-04-05T21:53:40.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.431+0000 7fbecc014700 1 Processor -- start 2024-04-05T21:53:40.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.431+0000 7fbecc014700 1 -- start start 2024-04-05T21:53:40.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.431+0000 7fbecc014700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbec40ffbd0 0x7fbec419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:40.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.431+0000 7fbecc014700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbec4105a40 0x7fbec419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:40.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.431+0000 7fbecc014700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec419d0e0 con 0x7fbec4105a40 2024-04-05T21:53:40.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.432+0000 7fbecc014700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec4196590 con 0x7fbec40ffbd0 2024-04-05T21:53:40.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.432+0000 7fbec95af700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbec4105a40 0x7fbec419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:40.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.432+0000 7fbec95af700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbec4105a40 0x7fbec419ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41024/0 (socket says 172.21.15.121:41024) 2024-04-05T21:53:40.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.432+0000 7fbec95af700 1 -- 172.21.15.121:0/2401743100 learned_addr learned my addr 172.21.15.121:0/2401743100 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:40.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.432+0000 7fbec95af700 1 -- 172.21.15.121:0/2401743100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbec40ffbd0 msgr2=0x7fbec419c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:40.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.432+0000 7fbec95af700 1 --2- 172.21.15.121:0/2401743100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbec40ffbd0 0x7fbec419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.432+0000 7fbec95af700 1 -- 172.21.15.121:0/2401743100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb40095f0 con 0x7fbec4105a40 2024-04-05T21:53:40.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.432+0000 7fbec95af700 1 --2- 172.21.15.121:0/2401743100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbec4105a40 0x7fbec419ca50 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4006010 tx=0x7fbeb400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:40.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.432+0000 7fbebaffd700 1 -- 172.21.15.121:0/2401743100 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb401d070 con 0x7fbec4105a40 2024-04-05T21:53:40.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.432+0000 7fbecc014700 1 -- 172.21.15.121:0/2401743100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec4196810 con 0x7fbec4105a40 2024-04-05T21:53:40.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.432+0000 7fbebaffd700 1 -- 172.21.15.121:0/2401743100 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbeb400f460 con 0x7fbec4105a40 2024-04-05T21:53:40.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.432+0000 7fbebaffd700 1 -- 172.21.15.121:0/2401743100 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb4021620 con 0x7fbec4105a40 2024-04-05T21:53:40.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.432+0000 7fbecc014700 1 -- 172.21.15.121:0/2401743100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec4196d00 con 0x7fbec4105a40 2024-04-05T21:53:40.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.433+0000 7fbecc014700 1 -- 172.21.15.121:0/2401743100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec4108fe0 con 0x7fbec4105a40 2024-04-05T21:53:40.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.434+0000 7fbebaffd700 1 -- 172.21.15.121:0/2401743100 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbeb400fa40 con 0x7fbec4105a40 2024-04-05T21:53:40.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.434+0000 7fbebaffd700 1 --2- 172.21.15.121:0/2401743100 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbeb006cce0 0x7fbeb006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:40.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.434+0000 7fbebaffd700 1 -- 172.21.15.121:0/2401743100 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbeb408d790 con 0x7fbec4105a40 2024-04-05T21:53:40.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.434+0000 7fbec9db0700 1 --2- 172.21.15.121:0/2401743100 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbeb006cce0 0x7fbeb006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:40.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.434+0000 7fbec9db0700 1 --2- 172.21.15.121:0/2401743100 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbeb006cce0 0x7fbeb006f1a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbec0005950 tx=0x7fbec00058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:40.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.439+0000 7fbebaffd700 1 -- 172.21.15.121:0/2401743100 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbeb405b290 con 0x7fbec4105a40 2024-04-05T21:53:40.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.676+0000 7fbecc014700 1 -- 172.21.15.121:0/2401743100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbec404ea90 con 0x7fbec4105a40 2024-04-05T21:53:40.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.676+0000 7fbebaffd700 1 -- 172.21.15.121:0/2401743100 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fbeb405ae20 con 0x7fbec4105a40 2024-04-05T21:53:40.677 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:53:40.677 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:53:40.677 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:53:40.677 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:53:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:53:40.677 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:53:40.677 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:53:40.677 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:53:40.677 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:53:40.677 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:53:40.677 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:53:40.678 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:53:40.679 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:53:40.680 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:53:40.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.678+0000 7fbecc014700 1 -- 172.21.15.121:0/2401743100 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbeb006cce0 msgr2=0x7fbeb006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:40.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.678+0000 7fbecc014700 1 --2- 172.21.15.121:0/2401743100 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbeb006cce0 0x7fbeb006f1a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbec0005950 tx=0x7fbec00058e0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.678+0000 7fbecc014700 1 -- 172.21.15.121:0/2401743100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbec4105a40 msgr2=0x7fbec419ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:40.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.678+0000 7fbecc014700 1 --2- 172.21.15.121:0/2401743100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbec4105a40 0x7fbec419ca50 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4006010 tx=0x7fbeb400be20 comp rx=0 tx=0).stop 2024-04-05T21:53:40.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.678+0000 7fbecc014700 1 -- 172.21.15.121:0/2401743100 shutdown_connections 2024-04-05T21:53:40.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.678+0000 7fbecc014700 1 --2- 172.21.15.121:0/2401743100 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbeb006cce0 0x7fbeb006f1a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.678+0000 7fbecc014700 1 --2- 172.21.15.121:0/2401743100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbec40ffbd0 0x7fbec419c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.678+0000 7fbecc014700 1 --2- 172.21.15.121:0/2401743100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbec4105a40 0x7fbec419ca50 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.678+0000 7fbecc014700 1 -- 172.21.15.121:0/2401743100 >> 172.21.15.121:0/2401743100 conn(0x7fbec40755a0 msgr2=0x7fbec4103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:40.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.678+0000 7fbecc014700 1 -- 172.21.15.121:0/2401743100 shutdown_connections 2024-04-05T21:53:40.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.678+0000 7fbecc014700 1 -- 172.21.15.121:0/2401743100 wait complete. 2024-04-05T21:53:40.681 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:53:40.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.781+0000 7f9a6d886700 1 -- 172.21.15.121:0/4290189763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68106c00 msgr2=0x7f9a68106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:40.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.781+0000 7f9a6d886700 1 --2- 172.21.15.121:0/4290189763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68106c00 0x7f9a68106fe0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009910 tx=0x7f9a5c009c20 comp rx=0 tx=0).stop 2024-04-05T21:53:40.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.781+0000 7f9a6d886700 1 -- 172.21.15.121:0/4290189763 shutdown_connections 2024-04-05T21:53:40.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.781+0000 7f9a6d886700 1 --2- 172.21.15.121:0/4290189763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a68100b50 0x7f9a68100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.781+0000 7f9a6d886700 1 --2- 172.21.15.121:0/4290189763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68106c00 0x7f9a68106fe0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.781+0000 7f9a6d886700 1 -- 172.21.15.121:0/4290189763 >> 172.21.15.121:0/4290189763 conn(0x7f9a680fc640 msgr2=0x7f9a680fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:40.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.781+0000 7f9a6d886700 1 -- 172.21.15.121:0/4290189763 shutdown_connections 2024-04-05T21:53:40.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.781+0000 7f9a6d886700 1 -- 172.21.15.121:0/4290189763 wait complete. 2024-04-05T21:53:40.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.781+0000 7f9a6d886700 1 Processor -- start 2024-04-05T21:53:40.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a6d886700 1 -- start start 2024-04-05T21:53:40.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a6d886700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a68100b50 0x7f9a68196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:40.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a6d886700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68106c00 0x7f9a68196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:40.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a6d886700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a681972d0 con 0x7f9a68106c00 2024-04-05T21:53:40.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a6d886700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6819a000 con 0x7f9a68100b50 2024-04-05T21:53:40.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a667fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68106c00 0x7f9a68196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:40.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a667fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68106c00 0x7f9a68196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41034/0 (socket says 172.21.15.121:41034) 2024-04-05T21:53:40.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a667fc700 1 -- 172.21.15.121:0/2372122600 learned_addr learned my addr 172.21.15.121:0/2372122600 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:40.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a66ffd700 1 --2- 172.21.15.121:0/2372122600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a68100b50 0x7f9a68196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:40.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a667fc700 1 -- 172.21.15.121:0/2372122600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a68100b50 msgr2=0x7f9a68196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:40.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a667fc700 1 --2- 172.21.15.121:0/2372122600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a68100b50 0x7f9a68196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:40.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a667fc700 1 -- 172.21.15.121:0/2372122600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c0095f0 con 0x7f9a68106c00 2024-04-05T21:53:40.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a667fc700 1 --2- 172.21.15.121:0/2372122600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68106c00 0x7f9a68196c40 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800b4b0 tx=0x7f9a5800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:40.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a6c884700 1 -- 172.21.15.121:0/2372122600 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a58011800 con 0x7f9a68106c00 2024-04-05T21:53:40.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a6c884700 1 -- 172.21.15.121:0/2372122600 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a58011e00 con 0x7f9a68106c00 2024-04-05T21:53:40.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a6d886700 1 -- 172.21.15.121:0/2372122600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6819a2e0 con 0x7f9a68106c00 2024-04-05T21:53:40.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.782+0000 7f9a6c884700 1 -- 172.21.15.121:0/2372122600 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a58010e50 con 0x7f9a68106c00 2024-04-05T21:53:40.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.783+0000 7f9a6d886700 1 -- 172.21.15.121:0/2372122600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6819a830 con 0x7f9a68106c00 2024-04-05T21:53:40.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.783+0000 7f9a6d886700 1 -- 172.21.15.121:0/2372122600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a68109030 con 0x7f9a68106c00 2024-04-05T21:53:40.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.784+0000 7f9a6c884700 1 -- 172.21.15.121:0/2372122600 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9a58010380 con 0x7f9a68106c00 2024-04-05T21:53:40.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.784+0000 7f9a6c884700 1 --2- 172.21.15.121:0/2372122600 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a5006cce0 0x7f9a5006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:40.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.784+0000 7f9a6c884700 1 -- 172.21.15.121:0/2372122600 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9a5808c190 con 0x7f9a68106c00 2024-04-05T21:53:40.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.784+0000 7f9a66ffd700 1 --2- 172.21.15.121:0/2372122600 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a5006cce0 0x7f9a5006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:40.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.785+0000 7f9a66ffd700 1 --2- 172.21.15.121:0/2372122600 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a5006cce0 0x7f9a5006f1a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c006010 tx=0x7f9a5c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:40.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.789+0000 7f9a6c884700 1 -- 172.21.15.121:0/2372122600 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9a58056200 con 0x7f9a68106c00 2024-04-05T21:53:40.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.998+0000 7f9a6d886700 1 -- 172.21.15.121:0/2372122600 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a6819aab0 con 0x7f9a5006cce0 2024-04-05T21:53:40.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:40.999+0000 7f9a6c884700 1 -- 172.21.15.121:0/2372122600 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9a6819aab0 con 0x7f9a5006cce0 2024-04-05T21:53:41.000 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:53:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:53:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:53:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:53:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:53:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:53:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:53:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:53:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:53:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:53:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:53:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:53:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:53:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:53:41.001 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:53:41.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.001+0000 7f9a6d886700 1 -- 172.21.15.121:0/2372122600 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a5006cce0 msgr2=0x7f9a5006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:41.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.001+0000 7f9a6d886700 1 --2- 172.21.15.121:0/2372122600 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a5006cce0 0x7f9a5006f1a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c006010 tx=0x7f9a5c01a040 comp rx=0 tx=0).stop 2024-04-05T21:53:41.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.001+0000 7f9a6d886700 1 -- 172.21.15.121:0/2372122600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68106c00 msgr2=0x7f9a68196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:41.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.001+0000 7f9a6d886700 1 --2- 172.21.15.121:0/2372122600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68106c00 0x7f9a68196c40 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800b4b0 tx=0x7f9a5800b870 comp rx=0 tx=0).stop 2024-04-05T21:53:41.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.001+0000 7f9a6d886700 1 -- 172.21.15.121:0/2372122600 shutdown_connections 2024-04-05T21:53:41.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.001+0000 7f9a6d886700 1 --2- 172.21.15.121:0/2372122600 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a5006cce0 0x7f9a5006f1a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:41.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.001+0000 7f9a6d886700 1 --2- 172.21.15.121:0/2372122600 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a68100b50 0x7f9a68196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:41.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.001+0000 7f9a6d886700 1 --2- 172.21.15.121:0/2372122600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68106c00 0x7f9a68196c40 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:41.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.001+0000 7f9a6d886700 1 -- 172.21.15.121:0/2372122600 >> 172.21.15.121:0/2372122600 conn(0x7f9a680fc640 msgr2=0x7f9a680fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:41.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.001+0000 7f9a6d886700 1 -- 172.21.15.121:0/2372122600 shutdown_connections 2024-04-05T21:53:41.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.001+0000 7f9a6d886700 1 -- 172.21.15.121:0/2372122600 wait complete. 2024-04-05T21:53:41.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.103+0000 7f769acd5700 1 -- 172.21.15.121:0/1278519711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7694105a40 msgr2=0x7f7694105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:41.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.103+0000 7f769acd5700 1 --2- 172.21.15.121:0/1278519711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7694105a40 0x7f7694105e20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7688009910 tx=0x7f7688009c20 comp rx=0 tx=0).stop 2024-04-05T21:53:41.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.103+0000 7f769acd5700 1 -- 172.21.15.121:0/1278519711 shutdown_connections 2024-04-05T21:53:41.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.103+0000 7f769acd5700 1 --2- 172.21.15.121:0/1278519711 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76940ffbd0 0x7f7694100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:41.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.103+0000 7f769acd5700 1 --2- 172.21.15.121:0/1278519711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7694105a40 0x7f7694105e20 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:41.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.103+0000 7f769acd5700 1 -- 172.21.15.121:0/1278519711 >> 172.21.15.121:0/1278519711 conn(0x7f76940755a0 msgr2=0x7f76940759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:41.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.103+0000 7f769acd5700 1 -- 172.21.15.121:0/1278519711 shutdown_connections 2024-04-05T21:53:41.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.103+0000 7f769acd5700 1 -- 172.21.15.121:0/1278519711 wait complete. 2024-04-05T21:53:41.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.104+0000 7f769acd5700 1 Processor -- start 2024-04-05T21:53:41.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.104+0000 7f769acd5700 1 -- start start 2024-04-05T21:53:41.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.104+0000 7f769acd5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76940ffbd0 0x7f76941974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.104+0000 7f769acd5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7694105a40 0x7f7694197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.104+0000 7f769acd5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7694198020 con 0x7f7694105a40 2024-04-05T21:53:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.104+0000 7f769acd5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7694198160 con 0x7f76940ffbd0 2024-04-05T21:53:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.104+0000 7f7693fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7694105a40 0x7f7694197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.104+0000 7f7693fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7694105a40 0x7f7694197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41058/0 (socket says 172.21.15.121:41058) 2024-04-05T21:53:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.104+0000 7f7693fff700 1 -- 172.21.15.121:0/3022047332 learned_addr learned my addr 172.21.15.121:0/3022047332 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:53:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.104+0000 7f7698a71700 1 --2- 172.21.15.121:0/3022047332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76940ffbd0 0x7f76941974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.105+0000 7f7693fff700 1 -- 172.21.15.121:0/3022047332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76940ffbd0 msgr2=0x7f76941974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.105+0000 7f7693fff700 1 --2- 172.21.15.121:0/3022047332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76940ffbd0 0x7f76941974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.105+0000 7f7693fff700 1 -- 172.21.15.121:0/3022047332 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76880095f0 con 0x7f7694105a40 2024-04-05T21:53:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.105+0000 7f7693fff700 1 --2- 172.21.15.121:0/3022047332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7694105a40 0x7f7694197a00 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f768000d8d0 tx=0x7f768000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.105+0000 7f7691ffb700 1 -- 172.21.15.121:0/3022047332 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76800098a0 con 0x7f7694105a40 2024-04-05T21:53:41.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.105+0000 7f7691ffb700 1 -- 172.21.15.121:0/3022047332 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7680010460 con 0x7f7694105a40 2024-04-05T21:53:41.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.105+0000 7f769acd5700 1 -- 172.21.15.121:0/3022047332 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76941a0530 con 0x7f7694105a40 2024-04-05T21:53:41.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.105+0000 7f7691ffb700 1 -- 172.21.15.121:0/3022047332 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7680009ea0 con 0x7f7694105a40 2024-04-05T21:53:41.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.105+0000 7f769acd5700 1 -- 172.21.15.121:0/3022047332 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76941a09f0 con 0x7f7694105a40 2024-04-05T21:53:41.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.106+0000 7f769acd5700 1 -- 172.21.15.121:0/3022047332 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7694108040 con 0x7f7694105a40 2024-04-05T21:53:41.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.107+0000 7f7691ffb700 1 -- 172.21.15.121:0/3022047332 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f76800105d0 con 0x7f7694105a40 2024-04-05T21:53:41.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.107+0000 7f7691ffb700 1 --2- 172.21.15.121:0/3022047332 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f768406cce0 0x7f768406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:53:41.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.107+0000 7f7698a71700 1 --2- 172.21.15.121:0/3022047332 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f768406cce0 0x7f768406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:53:41.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.107+0000 7f7691ffb700 1 -- 172.21.15.121:0/3022047332 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f768008c3a0 con 0x7f7694105a40 2024-04-05T21:53:41.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.108+0000 7f7698a71700 1 --2- 172.21.15.121:0/3022047332 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f768406cce0 0x7f768406f1a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7688006010 tx=0x7f768801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:53:41.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.112+0000 7f7691ffb700 1 -- 172.21.15.121:0/3022047332 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7680056490 con 0x7f7694105a40 2024-04-05T21:53:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:40 smithi121 ceph-mon[145859]: from='client.35326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:40 smithi121 ceph-mon[145859]: from='client.35330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:41.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:40 smithi121 ceph-mon[145859]: pgmap v824: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:41.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:40 smithi121 ceph-mon[145859]: from='client.35334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:41.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:40 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1200501832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:53:41.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:40 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2401743100' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:53:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:40 smithi188 ceph-mon[107793]: from='client.35326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:40 smithi188 ceph-mon[107793]: from='client.35330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:40 smithi188 ceph-mon[107793]: pgmap v824: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:40 smithi188 ceph-mon[107793]: from='client.35334 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:40 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1200501832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:53:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:40 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2401743100' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:53:41.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.387+0000 7f769acd5700 1 -- 172.21.15.121:0/3022047332 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f769404ea90 con 0x7f7694105a40 2024-04-05T21:53:41.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.387+0000 7f7691ffb700 1 -- 172.21.15.121:0/3022047332 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f7680016e10 con 0x7f7694105a40 2024-04-05T21:53:41.387 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:53:41.388 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:53:41.389 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:53:41.389 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:53:41.389 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:53:41.389 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:53:41.389 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:53:41.389 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:53:41.389 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:53:41.389 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:53:41.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.389+0000 7f769acd5700 1 -- 172.21.15.121:0/3022047332 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f768406cce0 msgr2=0x7f768406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:41.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.389+0000 7f769acd5700 1 --2- 172.21.15.121:0/3022047332 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f768406cce0 0x7f768406f1a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7688006010 tx=0x7f768801a040 comp rx=0 tx=0).stop 2024-04-05T21:53:41.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.389+0000 7f769acd5700 1 -- 172.21.15.121:0/3022047332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7694105a40 msgr2=0x7f7694197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:53:41.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.389+0000 7f769acd5700 1 --2- 172.21.15.121:0/3022047332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7694105a40 0x7f7694197a00 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f768000d8d0 tx=0x7f768000dbe0 comp rx=0 tx=0).stop 2024-04-05T21:53:41.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.389+0000 7f769acd5700 1 -- 172.21.15.121:0/3022047332 shutdown_connections 2024-04-05T21:53:41.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.389+0000 7f769acd5700 1 --2- 172.21.15.121:0/3022047332 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f768406cce0 0x7f768406f1a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:41.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.389+0000 7f769acd5700 1 --2- 172.21.15.121:0/3022047332 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f76940ffbd0 0x7f76941974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:41.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.389+0000 7f769acd5700 1 --2- 172.21.15.121:0/3022047332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7694105a40 0x7f7694197a00 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:53:41.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.389+0000 7f769acd5700 1 -- 172.21.15.121:0/3022047332 >> 172.21.15.121:0/3022047332 conn(0x7f76940755a0 msgr2=0x7f76940fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:53:41.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.389+0000 7f769acd5700 1 -- 172.21.15.121:0/3022047332 shutdown_connections 2024-04-05T21:53:41.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:53:41.389+0000 7f769acd5700 1 -- 172.21.15.121:0/3022047332 wait complete. 2024-04-05T21:53:42.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:41 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3022047332' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:53:42.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:41 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3022047332' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:53:43.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:42 smithi121 ceph-mon[145859]: from='client.35346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:43.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:42 smithi121 ceph-mon[145859]: pgmap v825: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:42 smithi188 ceph-mon[107793]: from='client.35346 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:53:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:42 smithi188 ceph-mon[107793]: pgmap v825: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:45.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:44 smithi121 ceph-mon[145859]: pgmap v826: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:44 smithi188 ceph-mon[107793]: pgmap v826: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:53:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:53:47.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:46 smithi121 ceph-mon[145859]: pgmap v827: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:46 smithi188 ceph-mon[107793]: pgmap v827: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:49.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:48 smithi121 ceph-mon[145859]: pgmap v828: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:48 smithi188 ceph-mon[107793]: pgmap v828: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:51.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:50 smithi121 ceph-mon[145859]: pgmap v829: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:50 smithi188 ceph-mon[107793]: pgmap v829: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:53.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:52 smithi121 ceph-mon[145859]: pgmap v830: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:52 smithi188 ceph-mon[107793]: pgmap v830: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:55.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:54 smithi121 ceph-mon[145859]: pgmap v831: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:54 smithi188 ceph-mon[107793]: pgmap v831: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:56.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:56 smithi121 ceph-mon[145859]: pgmap v832: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:56 smithi188 ceph-mon[107793]: pgmap v832: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:59.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:53:58 smithi121 ceph-mon[145859]: pgmap v833: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:53:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:53:58 smithi188 ceph-mon[107793]: pgmap v833: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:00 smithi121 ceph-mon[145859]: pgmap v834: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:54:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:00 smithi188 ceph-mon[107793]: pgmap v834: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:54:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:02 smithi121 ceph-mon[145859]: pgmap v835: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:02 smithi188 ceph-mon[107793]: pgmap v835: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:05.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:04 smithi121 ceph-mon[145859]: pgmap v836: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:05.256 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:04 smithi188 ceph-mon[107793]: pgmap v836: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:07.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:06 smithi121 ceph-mon[145859]: pgmap v837: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:06 smithi188 ceph-mon[107793]: pgmap v837: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:09.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:08 smithi121 ceph-mon[145859]: pgmap v838: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:09.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:08 smithi188 ceph-mon[107793]: pgmap v838: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:11.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:10 smithi121 ceph-mon[145859]: pgmap v839: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:10 smithi188 ceph-mon[107793]: pgmap v839: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:11.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.504+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/279174671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf4069020 msgr2=0x7f8bf4105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:11.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.504+0000 7f8bfa7a3700 1 --2- 172.21.15.121:0/279174671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf4069020 0x7f8bf4105790 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009910 tx=0x7f8be8009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:11.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.504+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/279174671 shutdown_connections 2024-04-05T21:54:11.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.505+0000 7f8bfa7a3700 1 --2- 172.21.15.121:0/279174671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf4069020 0x7f8bf4105790 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:11.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.505+0000 7f8bfa7a3700 1 --2- 172.21.15.121:0/279174671 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bf4068670 0x7f8bf4068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:11.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.505+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/279174671 >> 172.21.15.121:0/279174671 conn(0x7f8bf40755a0 msgr2=0x7f8bf40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:11.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.505+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/279174671 shutdown_connections 2024-04-05T21:54:11.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.505+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/279174671 wait complete. 2024-04-05T21:54:11.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.505+0000 7f8bfa7a3700 1 Processor -- start 2024-04-05T21:54:11.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.505+0000 7f8bfa7a3700 1 -- start start 2024-04-05T21:54:11.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bfa7a3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bf4068670 0x7f8bf419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:11.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bfa7a3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf4069020 0x7f8bf419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:11.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bfa7a3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf419f2e0 con 0x7f8bf4069020 2024-04-05T21:54:11.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bfa7a3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4198790 con 0x7f8bf4068670 2024-04-05T21:54:11.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bf37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf4069020 0x7f8bf419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:11.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bf37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf4069020 0x7f8bf419ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59274/0 (socket says 172.21.15.121:59274) 2024-04-05T21:54:11.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bf37fe700 1 -- 172.21.15.121:0/2488046129 learned_addr learned my addr 172.21.15.121:0/2488046129 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:11.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bf3fff700 1 --2- 172.21.15.121:0/2488046129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bf4068670 0x7f8bf419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:11.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bf37fe700 1 -- 172.21.15.121:0/2488046129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bf4068670 msgr2=0x7f8bf419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:11.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bf37fe700 1 --2- 172.21.15.121:0/2488046129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bf4068670 0x7f8bf419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:11.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bf37fe700 1 -- 172.21.15.121:0/2488046129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be80095f0 con 0x7f8bf4069020 2024-04-05T21:54:11.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bf37fe700 1 --2- 172.21.15.121:0/2488046129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf4069020 0x7f8bf419ec50 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8be800bda0 tx=0x7f8be800bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:11.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bf17fa700 1 -- 172.21.15.121:0/2488046129 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8be801d070 con 0x7f8bf4069020 2024-04-05T21:54:11.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bf17fa700 1 -- 172.21.15.121:0/2488046129 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8be8005710 con 0x7f8bf4069020 2024-04-05T21:54:11.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/2488046129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf4198a10 con 0x7f8bf4069020 2024-04-05T21:54:11.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bf17fa700 1 -- 172.21.15.121:0/2488046129 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8be800fe90 con 0x7f8bf4069020 2024-04-05T21:54:11.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.506+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/2488046129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf4198f00 con 0x7f8bf4069020 2024-04-05T21:54:11.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.507+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/2488046129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf404ea90 con 0x7f8bf4069020 2024-04-05T21:54:11.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.508+0000 7f8bf17fa700 1 -- 172.21.15.121:0/2488046129 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8be800f6f0 con 0x7f8bf4069020 2024-04-05T21:54:11.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.508+0000 7f8bf17fa700 1 --2- 172.21.15.121:0/2488046129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8bdc06cce0 0x7f8bdc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:11.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.509+0000 7f8bf3fff700 1 --2- 172.21.15.121:0/2488046129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8bdc06cce0 0x7f8bdc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:11.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.509+0000 7f8bf17fa700 1 -- 172.21.15.121:0/2488046129 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8be808d6d0 con 0x7f8bf4069020 2024-04-05T21:54:11.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.509+0000 7f8bf3fff700 1 --2- 172.21.15.121:0/2488046129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8bdc06cce0 0x7f8bdc06f1a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8be4005950 tx=0x7f8be400b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:11.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.513+0000 7f8bf17fa700 1 -- 172.21.15.121:0/2488046129 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8be805b1b0 con 0x7f8bf4069020 2024-04-05T21:54:11.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.724+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/2488046129 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bf41991e0 con 0x7f8bdc06cce0 2024-04-05T21:54:11.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.725+0000 7f8bf17fa700 1 -- 172.21.15.121:0/2488046129 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8bf41991e0 con 0x7f8bdc06cce0 2024-04-05T21:54:11.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.727+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/2488046129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8bdc06cce0 msgr2=0x7f8bdc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:11.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.727+0000 7f8bfa7a3700 1 --2- 172.21.15.121:0/2488046129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8bdc06cce0 0x7f8bdc06f1a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8be4005950 tx=0x7f8be400b410 comp rx=0 tx=0).stop 2024-04-05T21:54:11.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.727+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/2488046129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf4069020 msgr2=0x7f8bf419ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:11.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.727+0000 7f8bfa7a3700 1 --2- 172.21.15.121:0/2488046129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf4069020 0x7f8bf419ec50 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8be800bda0 tx=0x7f8be800bdd0 comp rx=0 tx=0).stop 2024-04-05T21:54:11.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.727+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/2488046129 shutdown_connections 2024-04-05T21:54:11.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.727+0000 7f8bfa7a3700 1 --2- 172.21.15.121:0/2488046129 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8bdc06cce0 0x7f8bdc06f1a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:11.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.727+0000 7f8bfa7a3700 1 --2- 172.21.15.121:0/2488046129 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8bf4068670 0x7f8bf419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:11.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.727+0000 7f8bfa7a3700 1 --2- 172.21.15.121:0/2488046129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bf4069020 0x7f8bf419ec50 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:11.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.727+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/2488046129 >> 172.21.15.121:0/2488046129 conn(0x7f8bf40755a0 msgr2=0x7f8bf4102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:11.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.727+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/2488046129 shutdown_connections 2024-04-05T21:54:11.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.727+0000 7f8bfa7a3700 1 -- 172.21.15.121:0/2488046129 wait complete. 2024-04-05T21:54:11.739 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:54:11.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.830+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/39907320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80ffa40 msgr2=0x7ff4e80ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:11.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.830+0000 7ff4ed8e8700 1 --2- 172.21.15.121:0/39907320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80ffa40 0x7ff4e80ffec0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009910 tx=0x7ff4dc009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:11.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.830+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/39907320 shutdown_connections 2024-04-05T21:54:11.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.830+0000 7ff4ed8e8700 1 --2- 172.21.15.121:0/39907320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80ffa40 0x7ff4e80ffec0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:11.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.830+0000 7ff4ed8e8700 1 --2- 172.21.15.121:0/39907320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e8105a40 0x7ff4e8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:11.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.830+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/39907320 >> 172.21.15.121:0/39907320 conn(0x7ff4e80fb570 msgr2=0x7ff4e80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:11.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.830+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/39907320 shutdown_connections 2024-04-05T21:54:11.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.830+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/39907320 wait complete. 2024-04-05T21:54:11.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.831+0000 7ff4ed8e8700 1 Processor -- start 2024-04-05T21:54:11.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.831+0000 7ff4ed8e8700 1 -- start start 2024-04-05T21:54:11.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.831+0000 7ff4ed8e8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80ffa40 0x7ff4e819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:11.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.831+0000 7ff4ed8e8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e8105a40 0x7ff4e819ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:11.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.831+0000 7ff4ed8e8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e819f260 con 0x7ff4e80ffa40 2024-04-05T21:54:11.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.831+0000 7ff4ed8e8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e8198710 con 0x7ff4e8105a40 2024-04-05T21:54:11.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.831+0000 7ff4e6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80ffa40 0x7ff4e819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:11.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.831+0000 7ff4e6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80ffa40 0x7ff4e819e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59294/0 (socket says 172.21.15.121:59294) 2024-04-05T21:54:11.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.831+0000 7ff4e6ffd700 1 -- 172.21.15.121:0/1306773189 learned_addr learned my addr 172.21.15.121:0/1306773189 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:11.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.831+0000 7ff4e67fc700 1 --2- 172.21.15.121:0/1306773189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e8105a40 0x7ff4e819ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:11.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.832+0000 7ff4e6ffd700 1 -- 172.21.15.121:0/1306773189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e8105a40 msgr2=0x7ff4e819ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:11.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.832+0000 7ff4e6ffd700 1 --2- 172.21.15.121:0/1306773189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e8105a40 0x7ff4e819ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:11.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.832+0000 7ff4e6ffd700 1 -- 172.21.15.121:0/1306773189 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc0095f0 con 0x7ff4e80ffa40 2024-04-05T21:54:11.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.832+0000 7ff4e6ffd700 1 --2- 172.21.15.121:0/1306773189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80ffa40 0x7ff4e819e690 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800d900 tx=0x7ff4d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:11.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.832+0000 7ff4ec8e6700 1 -- 172.21.15.121:0/1306773189 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4d8004a00 con 0x7ff4e80ffa40 2024-04-05T21:54:11.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.832+0000 7ff4ec8e6700 1 -- 172.21.15.121:0/1306773189 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4d8005020 con 0x7ff4e80ffa40 2024-04-05T21:54:11.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.832+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/1306773189 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e81989f0 con 0x7ff4e80ffa40 2024-04-05T21:54:11.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.832+0000 7ff4ec8e6700 1 -- 172.21.15.121:0/1306773189 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4d8009c00 con 0x7ff4e80ffa40 2024-04-05T21:54:11.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.832+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/1306773189 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e8198f40 con 0x7ff4e80ffa40 2024-04-05T21:54:11.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.833+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/1306773189 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e804ea90 con 0x7ff4e80ffa40 2024-04-05T21:54:11.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.833+0000 7ff4ec8e6700 1 -- 172.21.15.121:0/1306773189 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff4d8005190 con 0x7ff4e80ffa40 2024-04-05T21:54:11.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.834+0000 7ff4ec8e6700 1 --2- 172.21.15.121:0/1306773189 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff4d006cce0 0x7ff4d006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:11.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.834+0000 7ff4e67fc700 1 --2- 172.21.15.121:0/1306773189 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff4d006cce0 0x7ff4d006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:11.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.834+0000 7ff4ec8e6700 1 -- 172.21.15.121:0/1306773189 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff4d808c690 con 0x7ff4e80ffa40 2024-04-05T21:54:11.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.834+0000 7ff4e67fc700 1 --2- 172.21.15.121:0/1306773189 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff4d006cce0 0x7ff4d006f1a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8199b80 tx=0x7ff4dc01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:11.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:11.838+0000 7ff4ec8e6700 1 -- 172.21.15.121:0/1306773189 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff4d805a210 con 0x7ff4e80ffa40 2024-04-05T21:54:12.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.058+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/1306773189 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4e81993b0 con 0x7ff4d006cce0 2024-04-05T21:54:12.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.060+0000 7ff4ec8e6700 1 -- 172.21.15.121:0/1306773189 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff4e81993b0 con 0x7ff4d006cce0 2024-04-05T21:54:12.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.062+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/1306773189 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff4d006cce0 msgr2=0x7ff4d006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:12.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.062+0000 7ff4ed8e8700 1 --2- 172.21.15.121:0/1306773189 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff4d006cce0 0x7ff4d006f1a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff4e8199b80 tx=0x7ff4dc01a040 comp rx=0 tx=0).stop 2024-04-05T21:54:12.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.062+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/1306773189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80ffa40 msgr2=0x7ff4e819e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:12.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.062+0000 7ff4ed8e8700 1 --2- 172.21.15.121:0/1306773189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80ffa40 0x7ff4e819e690 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800d900 tx=0x7ff4d800dcc0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.062+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/1306773189 shutdown_connections 2024-04-05T21:54:12.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.062+0000 7ff4ed8e8700 1 --2- 172.21.15.121:0/1306773189 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff4d006cce0 0x7ff4d006f1a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.062+0000 7ff4ed8e8700 1 --2- 172.21.15.121:0/1306773189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4e80ffa40 0x7ff4e819e690 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.062+0000 7ff4ed8e8700 1 --2- 172.21.15.121:0/1306773189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff4e8105a40 0x7ff4e819ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.062+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/1306773189 >> 172.21.15.121:0/1306773189 conn(0x7ff4e80fb570 msgr2=0x7ff4e8068f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:12.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.062+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/1306773189 shutdown_connections 2024-04-05T21:54:12.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.062+0000 7ff4ed8e8700 1 -- 172.21.15.121:0/1306773189 wait complete. 2024-04-05T21:54:12.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.164+0000 7f2f47dcd700 1 -- 172.21.15.121:0/3066999300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f400ffa40 msgr2=0x7f2f400ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:12.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.164+0000 7f2f47dcd700 1 --2- 172.21.15.121:0/3066999300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f400ffa40 0x7f2f400ffec0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009910 tx=0x7f2f3c009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:12.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.164+0000 7f2f47dcd700 1 -- 172.21.15.121:0/3066999300 shutdown_connections 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.164+0000 7f2f47dcd700 1 --2- 172.21.15.121:0/3066999300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f400ffa40 0x7f2f400ffec0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.164+0000 7f2f47dcd700 1 --2- 172.21.15.121:0/3066999300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f40105a40 0x7f2f40105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.164+0000 7f2f47dcd700 1 -- 172.21.15.121:0/3066999300 >> 172.21.15.121:0/3066999300 conn(0x7f2f400fb550 msgr2=0x7f2f400fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.164+0000 7f2f47dcd700 1 -- 172.21.15.121:0/3066999300 shutdown_connections 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.164+0000 7f2f47dcd700 1 -- 172.21.15.121:0/3066999300 wait complete. 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f47dcd700 1 Processor -- start 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f47dcd700 1 -- start start 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f47dcd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f400ffa40 0x7f2f401974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f47dcd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f40105a40 0x7f2f40197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f47dcd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40197f90 con 0x7f2f40105a40 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f47dcd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40198100 con 0x7f2f400ffa40 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f45368700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f40105a40 0x7f2f40197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f45368700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f40105a40 0x7f2f40197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59308/0 (socket says 172.21.15.121:59308) 2024-04-05T21:54:12.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f45368700 1 -- 172.21.15.121:0/4207358808 learned_addr learned my addr 172.21.15.121:0/4207358808 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:12.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f45b69700 1 --2- 172.21.15.121:0/4207358808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f400ffa40 0x7f2f401974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:12.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f45368700 1 -- 172.21.15.121:0/4207358808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f400ffa40 msgr2=0x7f2f401974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:12.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f45368700 1 --2- 172.21.15.121:0/4207358808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f400ffa40 0x7f2f401974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f45368700 1 -- 172.21.15.121:0/4207358808 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3c0095f0 con 0x7f2f40105a40 2024-04-05T21:54:12.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.165+0000 7f2f45b69700 1 --2- 172.21.15.121:0/4207358808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f400ffa40 0x7f2f401974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:54:12.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.166+0000 7f2f45368700 1 --2- 172.21.15.121:0/4207358808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f40105a40 0x7f2f40197a00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c00b5c0 tx=0x7f2f3c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:12.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.166+0000 7f2f36ffd700 1 -- 172.21.15.121:0/4207358808 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f3c01d070 con 0x7f2f40105a40 2024-04-05T21:54:12.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.166+0000 7f2f36ffd700 1 -- 172.21.15.121:0/4207358808 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f3c00f460 con 0x7f2f40105a40 2024-04-05T21:54:12.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.166+0000 7f2f47dcd700 1 -- 172.21.15.121:0/4207358808 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f401a0530 con 0x7f2f40105a40 2024-04-05T21:54:12.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.166+0000 7f2f36ffd700 1 -- 172.21.15.121:0/4207358808 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f3c021640 con 0x7f2f40105a40 2024-04-05T21:54:12.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.166+0000 7f2f47dcd700 1 -- 172.21.15.121:0/4207358808 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f401a08e0 con 0x7f2f40105a40 2024-04-05T21:54:12.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.167+0000 7f2f47dcd700 1 -- 172.21.15.121:0/4207358808 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f40107fd0 con 0x7f2f40105a40 2024-04-05T21:54:12.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.167+0000 7f2f36ffd700 1 -- 172.21.15.121:0/4207358808 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2f3c00f9b0 con 0x7f2f40105a40 2024-04-05T21:54:12.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.168+0000 7f2f36ffd700 1 --2- 172.21.15.121:0/4207358808 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2f2c06cce0 0x7f2f2c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:12.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.168+0000 7f2f45b69700 1 --2- 172.21.15.121:0/4207358808 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2f2c06cce0 0x7f2f2c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:12.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.168+0000 7f2f36ffd700 1 -- 172.21.15.121:0/4207358808 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2f3c08d8d0 con 0x7f2f40105a40 2024-04-05T21:54:12.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.168+0000 7f2f45b69700 1 --2- 172.21.15.121:0/4207358808 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2f2c06cce0 0x7f2f2c06f1a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2f30005fd0 tx=0x7f2f30005e20 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:12.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.172+0000 7f2f36ffd700 1 -- 172.21.15.121:0/4207358808 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2f3c057940 con 0x7f2f40105a40 2024-04-05T21:54:12.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:11 smithi121 ceph-mon[145859]: pgmap v840: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:12.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:11 smithi121 ceph-mon[145859]: from='client.35354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:12.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:11 smithi188 ceph-mon[107793]: pgmap v840: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:12.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:11 smithi188 ceph-mon[107793]: from='client.35354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:12.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.376+0000 7f2f47dcd700 1 -- 172.21.15.121:0/4207358808 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f4002cd30 con 0x7f2f2c06cce0 2024-04-05T21:54:12.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.386+0000 7f2f36ffd700 1 -- 172.21.15.121:0/4207358808 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2f4002cd30 con 0x7f2f2c06cce0 2024-04-05T21:54:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:54:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (25m) 3m ago 36m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:54:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (36m) 3m ago 36m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:54:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (35m) 3m ago 35m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:54:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (23m) 3m ago 36m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:54:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (23m) 3m ago 35m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:54:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (24m) 3m ago 36m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:54:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (19m) 3m ago 30m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:54:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (19m) 3m ago 30m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:54:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (18m) 3m ago 30m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:54:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (3m) 3m ago 30m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:54:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (27m) 3m ago 38m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:54:12.388 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (26m) 3m ago 35m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:54:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (23m) 3m ago 38m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:54:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (23m) 3m ago 35m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:54:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (25m) 3m ago 36m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:54:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (25m) 3m ago 35m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:54:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (22m) 3m ago 34m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:54:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (22m) 3m ago 33m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:54:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (21m) 3m ago 33m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:54:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (20m) 3m ago 32m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:54:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (20m) 3m ago 32m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:54:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (19m) 3m ago 32m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:54:12.389 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (25m) 3m ago 35m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:54:12.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.388+0000 7f2f47dcd700 1 -- 172.21.15.121:0/4207358808 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2f2c06cce0 msgr2=0x7f2f2c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:12.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.388+0000 7f2f47dcd700 1 --2- 172.21.15.121:0/4207358808 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2f2c06cce0 0x7f2f2c06f1a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2f30005fd0 tx=0x7f2f30005e20 comp rx=0 tx=0).stop 2024-04-05T21:54:12.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.388+0000 7f2f47dcd700 1 -- 172.21.15.121:0/4207358808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f40105a40 msgr2=0x7f2f40197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:12.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.388+0000 7f2f47dcd700 1 --2- 172.21.15.121:0/4207358808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f40105a40 0x7f2f40197a00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c00b5c0 tx=0x7f2f3c00be20 comp rx=0 tx=0).stop 2024-04-05T21:54:12.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.389+0000 7f2f47dcd700 1 -- 172.21.15.121:0/4207358808 shutdown_connections 2024-04-05T21:54:12.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.389+0000 7f2f47dcd700 1 --2- 172.21.15.121:0/4207358808 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2f2c06cce0 0x7f2f2c06f1a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.389+0000 7f2f47dcd700 1 --2- 172.21.15.121:0/4207358808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f400ffa40 0x7f2f401974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.389+0000 7f2f47dcd700 1 --2- 172.21.15.121:0/4207358808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f40105a40 0x7f2f40197a00 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.389+0000 7f2f47dcd700 1 -- 172.21.15.121:0/4207358808 >> 172.21.15.121:0/4207358808 conn(0x7f2f400fb550 msgr2=0x7f2f400fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:12.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.389+0000 7f2f47dcd700 1 -- 172.21.15.121:0/4207358808 shutdown_connections 2024-04-05T21:54:12.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.389+0000 7f2f47dcd700 1 -- 172.21.15.121:0/4207358808 wait complete. 2024-04-05T21:54:12.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.490+0000 7f756df25700 1 -- 172.21.15.121:0/2492884164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75680ffa40 msgr2=0x7f75680ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.490+0000 7f756df25700 1 --2- 172.21.15.121:0/2492884164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75680ffa40 0x7f75680ffec0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f755c009910 tx=0x7f755c009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.490+0000 7f756df25700 1 -- 172.21.15.121:0/2492884164 shutdown_connections 2024-04-05T21:54:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.490+0000 7f756df25700 1 --2- 172.21.15.121:0/2492884164 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75680ffa40 0x7f75680ffec0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.490+0000 7f756df25700 1 --2- 172.21.15.121:0/2492884164 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7568105a40 0x7f7568105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.490+0000 7f756df25700 1 -- 172.21.15.121:0/2492884164 >> 172.21.15.121:0/2492884164 conn(0x7f75680fb570 msgr2=0x7f75680fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.491+0000 7f756df25700 1 -- 172.21.15.121:0/2492884164 shutdown_connections 2024-04-05T21:54:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.491+0000 7f756df25700 1 -- 172.21.15.121:0/2492884164 wait complete. 2024-04-05T21:54:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.491+0000 7f756df25700 1 Processor -- start 2024-04-05T21:54:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.491+0000 7f756df25700 1 -- start start 2024-04-05T21:54:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.491+0000 7f756df25700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75680ffa40 0x7f75681974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f756df25700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7568105a40 0x7f75681979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:12.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f756df25700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7568198010 con 0x7f75680ffa40 2024-04-05T21:54:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f756df25700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7568198150 con 0x7f7568105a40 2024-04-05T21:54:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f75677fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75680ffa40 0x7f75681974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f75677fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75680ffa40 0x7f75681974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59326/0 (socket says 172.21.15.121:59326) 2024-04-05T21:54:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f75677fe700 1 -- 172.21.15.121:0/4105333252 learned_addr learned my addr 172.21.15.121:0/4105333252 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f7566ffd700 1 --2- 172.21.15.121:0/4105333252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7568105a40 0x7f75681979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f75677fe700 1 -- 172.21.15.121:0/4105333252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7568105a40 msgr2=0x7f75681979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f75677fe700 1 --2- 172.21.15.121:0/4105333252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7568105a40 0x7f75681979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f75677fe700 1 -- 172.21.15.121:0/4105333252 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c0095f0 con 0x7f75680ffa40 2024-04-05T21:54:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f75677fe700 1 --2- 172.21.15.121:0/4105333252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75680ffa40 0x7f75681974b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f755800c8a0 tx=0x7f755800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f7564ff9700 1 -- 172.21.15.121:0/4105333252 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f755800cea0 con 0x7f75680ffa40 2024-04-05T21:54:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f7564ff9700 1 -- 172.21.15.121:0/4105333252 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7558004830 con 0x7f75680ffa40 2024-04-05T21:54:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f756df25700 1 -- 172.21.15.121:0/4105333252 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75681a0520 con 0x7f75680ffa40 2024-04-05T21:54:12.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f7564ff9700 1 -- 172.21.15.121:0/4105333252 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7558005500 con 0x7f75680ffa40 2024-04-05T21:54:12.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.492+0000 7f756df25700 1 -- 172.21.15.121:0/4105333252 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75681a09e0 con 0x7f75680ffa40 2024-04-05T21:54:12.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.493+0000 7f756df25700 1 -- 172.21.15.121:0/4105333252 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7568107fd0 con 0x7f75680ffa40 2024-04-05T21:54:12.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.494+0000 7f7564ff9700 1 -- 172.21.15.121:0/4105333252 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f75580049a0 con 0x7f75680ffa40 2024-04-05T21:54:12.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.494+0000 7f7564ff9700 1 --2- 172.21.15.121:0/4105333252 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f755006cce0 0x7f755006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:12.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.494+0000 7f7566ffd700 1 --2- 172.21.15.121:0/4105333252 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f755006cce0 0x7f755006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:12.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.494+0000 7f7564ff9700 1 -- 172.21.15.121:0/4105333252 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f755808c2b0 con 0x7f75680ffa40 2024-04-05T21:54:12.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.495+0000 7f7566ffd700 1 --2- 172.21.15.121:0/4105333252 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f755006cce0 0x7f755006f1a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7568069550 tx=0x7f755c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:12.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.499+0000 7f7564ff9700 1 -- 172.21.15.121:0/4105333252 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7558056320 con 0x7f75680ffa40 2024-04-05T21:54:12.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.770+0000 7f756df25700 1 -- 172.21.15.121:0/4105333252 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f756802cd10 con 0x7f75680ffa40 2024-04-05T21:54:12.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.771+0000 7f7564ff9700 1 -- 172.21.15.121:0/4105333252 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f7558059940 con 0x7f75680ffa40 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:54:12.773 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:54:12.774 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:54:12.774 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:54:12.774 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:54:12.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.773+0000 7f756df25700 1 -- 172.21.15.121:0/4105333252 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f755006cce0 msgr2=0x7f755006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:12.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.773+0000 7f756df25700 1 --2- 172.21.15.121:0/4105333252 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f755006cce0 0x7f755006f1a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7568069550 tx=0x7f755c01a040 comp rx=0 tx=0).stop 2024-04-05T21:54:12.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.773+0000 7f756df25700 1 -- 172.21.15.121:0/4105333252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75680ffa40 msgr2=0x7f75681974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:12.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.773+0000 7f756df25700 1 --2- 172.21.15.121:0/4105333252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75680ffa40 0x7f75681974b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f755800c8a0 tx=0x7f755800cc60 comp rx=0 tx=0).stop 2024-04-05T21:54:12.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.773+0000 7f756df25700 1 -- 172.21.15.121:0/4105333252 shutdown_connections 2024-04-05T21:54:12.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.773+0000 7f756df25700 1 --2- 172.21.15.121:0/4105333252 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f755006cce0 0x7f755006f1a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.773+0000 7f756df25700 1 --2- 172.21.15.121:0/4105333252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75680ffa40 0x7f75681974b0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.773+0000 7f756df25700 1 --2- 172.21.15.121:0/4105333252 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7568105a40 0x7f75681979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.773+0000 7f756df25700 1 -- 172.21.15.121:0/4105333252 >> 172.21.15.121:0/4105333252 conn(0x7f75680fb570 msgr2=0x7f75680fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:12.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.773+0000 7f756df25700 1 -- 172.21.15.121:0/4105333252 shutdown_connections 2024-04-05T21:54:12.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.773+0000 7f756df25700 1 -- 172.21.15.121:0/4105333252 wait complete. 2024-04-05T21:54:12.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.875+0000 7f48d70a2700 1 -- 172.21.15.121:0/1632001540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48d0101770 msgr2=0x7f48d01057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:12.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.875+0000 7f48d70a2700 1 --2- 172.21.15.121:0/1632001540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48d0101770 0x7f48d01057c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f48bc009910 tx=0x7f48bc009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:12.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.875+0000 7f48d70a2700 1 -- 172.21.15.121:0/1632001540 shutdown_connections 2024-04-05T21:54:12.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.875+0000 7f48d70a2700 1 --2- 172.21.15.121:0/1632001540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48d0101770 0x7f48d01057c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.875+0000 7f48d70a2700 1 --2- 172.21.15.121:0/1632001540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f48d0100dc0 0x7f48d01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.875+0000 7f48d70a2700 1 -- 172.21.15.121:0/1632001540 >> 172.21.15.121:0/1632001540 conn(0x7f48d00fc650 msgr2=0x7f48d00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:12.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.875+0000 7f48d70a2700 1 -- 172.21.15.121:0/1632001540 shutdown_connections 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.875+0000 7f48d70a2700 1 -- 172.21.15.121:0/1632001540 wait complete. 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.875+0000 7f48d70a2700 1 Processor -- start 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48d70a2700 1 -- start start 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48d70a2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f48d0100dc0 0x7f48d01966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48d70a2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48d0101770 0x7f48d0196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48d70a2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d01972a0 con 0x7f48d0101770 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48d70a2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d0199fd0 con 0x7f48d0100dc0 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48cffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48d0101770 0x7f48d0196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48cffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48d0101770 0x7f48d0196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59346/0 (socket says 172.21.15.121:59346) 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48cffff700 1 -- 172.21.15.121:0/306729096 learned_addr learned my addr 172.21.15.121:0/306729096 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48d4e3e700 1 --2- 172.21.15.121:0/306729096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f48d0100dc0 0x7f48d01966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48cffff700 1 -- 172.21.15.121:0/306729096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f48d0100dc0 msgr2=0x7f48d01966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48cffff700 1 --2- 172.21.15.121:0/306729096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f48d0100dc0 0x7f48d01966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48cffff700 1 -- 172.21.15.121:0/306729096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc0095f0 con 0x7f48d0101770 2024-04-05T21:54:12.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48d4e3e700 1 --2- 172.21.15.121:0/306729096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f48d0100dc0 0x7f48d01966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:54:12.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48cffff700 1 --2- 172.21.15.121:0/306729096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48d0101770 0x7f48d0196c10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f48bc0092e0 tx=0x7f48bc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:12.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48cdffb700 1 -- 172.21.15.121:0/306729096 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48bc01d070 con 0x7f48d0101770 2024-04-05T21:54:12.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48cdffb700 1 -- 172.21.15.121:0/306729096 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48bc00f460 con 0x7f48d0101770 2024-04-05T21:54:12.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.876+0000 7f48cdffb700 1 -- 172.21.15.121:0/306729096 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48bc021600 con 0x7f48d0101770 2024-04-05T21:54:12.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.877+0000 7f48d70a2700 1 -- 172.21.15.121:0/306729096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d019a250 con 0x7f48d0101770 2024-04-05T21:54:12.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.877+0000 7f48d70a2700 1 -- 172.21.15.121:0/306729096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d019a740 con 0x7f48d0101770 2024-04-05T21:54:12.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.878+0000 7f48d70a2700 1 -- 172.21.15.121:0/306729096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d01091c0 con 0x7f48d0101770 2024-04-05T21:54:12.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.878+0000 7f48cdffb700 1 -- 172.21.15.121:0/306729096 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f48bc00f5d0 con 0x7f48d0101770 2024-04-05T21:54:12.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.879+0000 7f48cdffb700 1 --2- 172.21.15.121:0/306729096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f48c006cce0 0x7f48c006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:12.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.879+0000 7f48cdffb700 1 -- 172.21.15.121:0/306729096 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f48bc08d7b0 con 0x7f48d0101770 2024-04-05T21:54:12.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.879+0000 7f48d4e3e700 1 --2- 172.21.15.121:0/306729096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f48c006cce0 0x7f48c006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:12.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.879+0000 7f48d4e3e700 1 --2- 172.21.15.121:0/306729096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f48c006cce0 0x7f48c006f1a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f48c4005950 tx=0x7f48c4005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:12.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:12.883+0000 7f48cdffb700 1 -- 172.21.15.121:0/306729096 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f48bc057820 con 0x7f48d0101770 2024-04-05T21:54:13.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.117+0000 7f48d70a2700 1 -- 172.21.15.121:0/306729096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f48d004ea90 con 0x7f48d0101770 2024-04-05T21:54:13.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.117+0000 7f48cdffb700 1 -- 172.21.15.121:0/306729096 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f48bc05ae40 con 0x7f48d0101770 2024-04-05T21:54:13.119 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:54:13.119 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:54:13.119 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:54:13.119 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:54:13.119 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:54:13.119 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:54:13.119 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:54:13.119 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:54:13.119 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:54:13.120 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:54:13.121 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:54:13.122 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:54:13.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.119+0000 7f48d70a2700 1 -- 172.21.15.121:0/306729096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f48c006cce0 msgr2=0x7f48c006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:13.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.119+0000 7f48d70a2700 1 --2- 172.21.15.121:0/306729096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f48c006cce0 0x7f48c006f1a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f48c4005950 tx=0x7f48c4005f50 comp rx=0 tx=0).stop 2024-04-05T21:54:13.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.119+0000 7f48d70a2700 1 -- 172.21.15.121:0/306729096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48d0101770 msgr2=0x7f48d0196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:13.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.119+0000 7f48d70a2700 1 --2- 172.21.15.121:0/306729096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48d0101770 0x7f48d0196c10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f48bc0092e0 tx=0x7f48bc00be20 comp rx=0 tx=0).stop 2024-04-05T21:54:13.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.119+0000 7f48d70a2700 1 -- 172.21.15.121:0/306729096 shutdown_connections 2024-04-05T21:54:13.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.119+0000 7f48d70a2700 1 --2- 172.21.15.121:0/306729096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f48c006cce0 0x7f48c006f1a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.119+0000 7f48d70a2700 1 --2- 172.21.15.121:0/306729096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f48d0100dc0 0x7f48d01966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.119+0000 7f48d70a2700 1 --2- 172.21.15.121:0/306729096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48d0101770 0x7f48d0196c10 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.119+0000 7f48d70a2700 1 -- 172.21.15.121:0/306729096 >> 172.21.15.121:0/306729096 conn(0x7f48d00fc650 msgr2=0x7f48d00fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:13.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.119+0000 7f48d70a2700 1 -- 172.21.15.121:0/306729096 shutdown_connections 2024-04-05T21:54:13.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.119+0000 7f48d70a2700 1 -- 172.21.15.121:0/306729096 wait complete. 2024-04-05T21:54:13.123 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:54:13.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.222+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/1267914836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd80ffbd0 msgr2=0x7f6fd8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:13.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.222+0000 7f6fdc9f3700 1 --2- 172.21.15.121:0/1267914836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd80ffbd0 0x7f6fd8100050 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009910 tx=0x7f6fc8009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:13.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.222+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/1267914836 shutdown_connections 2024-04-05T21:54:13.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.222+0000 7f6fdc9f3700 1 --2- 172.21.15.121:0/1267914836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd80ffbd0 0x7f6fd8100050 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.222+0000 7f6fdc9f3700 1 --2- 172.21.15.121:0/1267914836 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fd8105a40 0x7f6fd8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.222+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/1267914836 >> 172.21.15.121:0/1267914836 conn(0x7f6fd80755a0 msgr2=0x7f6fd80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:13.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.222+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/1267914836 shutdown_connections 2024-04-05T21:54:13.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.222+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/1267914836 wait complete. 2024-04-05T21:54:13.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.223+0000 7f6fdc9f3700 1 Processor -- start 2024-04-05T21:54:13.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.223+0000 7f6fdc9f3700 1 -- start start 2024-04-05T21:54:13.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.223+0000 7f6fdc9f3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fd80ffbd0 0x7f6fd8101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.223+0000 7f6fdc9f3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8105a40 0x7f6fd8101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.223+0000 7f6fdc9f3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd8102090 con 0x7f6fd8105a40 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.223+0000 7f6fdc9f3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd81021d0 con 0x7f6fd80ffbd0 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.223+0000 7f6fd5d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8105a40 0x7f6fd8101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.223+0000 7f6fd5d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8105a40 0x7f6fd8101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59366/0 (socket says 172.21.15.121:59366) 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.223+0000 7f6fd5d9b700 1 -- 172.21.15.121:0/3421302667 learned_addr learned my addr 172.21.15.121:0/3421302667 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.224+0000 7f6fd659c700 1 --2- 172.21.15.121:0/3421302667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fd80ffbd0 0x7f6fd8101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.224+0000 7f6fd5d9b700 1 -- 172.21.15.121:0/3421302667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fd80ffbd0 msgr2=0x7f6fd8101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.224+0000 7f6fd5d9b700 1 --2- 172.21.15.121:0/3421302667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fd80ffbd0 0x7f6fd8101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.224+0000 7f6fd5d9b700 1 -- 172.21.15.121:0/3421302667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc80095f0 con 0x7f6fd8105a40 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.224+0000 7f6fd5d9b700 1 --2- 172.21.15.121:0/3421302667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8105a40 0x7f6fd8101b50 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8000c00 tx=0x7f6fc800bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.224+0000 7f6fcf7fe700 1 -- 172.21.15.121:0/3421302667 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fc801d070 con 0x7f6fd8105a40 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.224+0000 7f6fcf7fe700 1 -- 172.21.15.121:0/3421302667 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6fc8005dc0 con 0x7f6fd8105a40 2024-04-05T21:54:13.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.224+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/3421302667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd8102450 con 0x7f6fd8105a40 2024-04-05T21:54:13.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.224+0000 7f6fcf7fe700 1 -- 172.21.15.121:0/3421302667 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fc8003940 con 0x7f6fd8105a40 2024-04-05T21:54:13.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.224+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/3421302667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd81a2be0 con 0x7f6fd8105a40 2024-04-05T21:54:13.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.225+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/3421302667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd8108fe0 con 0x7f6fd8105a40 2024-04-05T21:54:13.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.225+0000 7f6fcf7fe700 1 -- 172.21.15.121:0/3421302667 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6fc8003ac0 con 0x7f6fd8105a40 2024-04-05T21:54:13.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.226+0000 7f6fcf7fe700 1 --2- 172.21.15.121:0/3421302667 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6fc406cce0 0x7f6fc406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:13.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.226+0000 7f6fcf7fe700 1 -- 172.21.15.121:0/3421302667 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6fc808db90 con 0x7f6fd8105a40 2024-04-05T21:54:13.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.226+0000 7f6fd659c700 1 --2- 172.21.15.121:0/3421302667 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6fc406cce0 0x7f6fc406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:13.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.226+0000 7f6fd659c700 1 --2- 172.21.15.121:0/3421302667 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6fc406cce0 0x7f6fc406f1a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0009c60 tx=0x7f6fc0009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:13.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.230+0000 7f6fcf7fe700 1 -- 172.21.15.121:0/3421302667 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6fc8057c00 con 0x7f6fd8105a40 2024-04-05T21:54:13.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:12 smithi121 ceph-mon[145859]: from='client.35358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:13.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:12 smithi121 ceph-mon[145859]: from='client.35362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:13.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:12 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4105333252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:54:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:12 smithi188 ceph-mon[107793]: from='client.35358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:12 smithi188 ceph-mon[107793]: from='client.35362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:12 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4105333252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:54:13.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.440+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/3421302667 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fd80611d0 con 0x7f6fc406cce0 2024-04-05T21:54:13.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.441+0000 7f6fcf7fe700 1 -- 172.21.15.121:0/3421302667 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6fd80611d0 con 0x7f6fc406cce0 2024-04-05T21:54:13.442 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:54:13.442 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:54:13.442 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:54:13.442 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:54:13.442 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:54:13.443 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:54:13.443 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:54:13.443 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:54:13.443 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:54:13.443 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:54:13.443 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:54:13.443 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:54:13.443 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:54:13.443 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:54:13.443 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:54:13.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.443+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/3421302667 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6fc406cce0 msgr2=0x7f6fc406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:13.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.443+0000 7f6fdc9f3700 1 --2- 172.21.15.121:0/3421302667 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6fc406cce0 0x7f6fc406f1a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0009c60 tx=0x7f6fc0009260 comp rx=0 tx=0).stop 2024-04-05T21:54:13.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.443+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/3421302667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8105a40 msgr2=0x7f6fd8101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:13.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.443+0000 7f6fdc9f3700 1 --2- 172.21.15.121:0/3421302667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8105a40 0x7f6fd8101b50 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8000c00 tx=0x7f6fc800bde0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.443+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/3421302667 shutdown_connections 2024-04-05T21:54:13.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.443+0000 7f6fdc9f3700 1 --2- 172.21.15.121:0/3421302667 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6fc406cce0 0x7f6fc406f1a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.443+0000 7f6fdc9f3700 1 --2- 172.21.15.121:0/3421302667 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6fd80ffbd0 0x7f6fd8101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.443+0000 7f6fdc9f3700 1 --2- 172.21.15.121:0/3421302667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8105a40 0x7f6fd8101b50 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.443+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/3421302667 >> 172.21.15.121:0/3421302667 conn(0x7f6fd80755a0 msgr2=0x7f6fd8103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:13.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.443+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/3421302667 shutdown_connections 2024-04-05T21:54:13.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.443+0000 7f6fdc9f3700 1 -- 172.21.15.121:0/3421302667 wait complete. 2024-04-05T21:54:13.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 -- 172.21.15.121:0/1348341815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa45c105a40 msgr2=0x7fa45c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 --2- 172.21.15.121:0/1348341815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa45c105a40 0x7fa45c105e20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa450009910 tx=0x7fa450009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 -- 172.21.15.121:0/1348341815 shutdown_connections 2024-04-05T21:54:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 --2- 172.21.15.121:0/1348341815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa45c0ffbd0 0x7fa45c100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 --2- 172.21.15.121:0/1348341815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa45c105a40 0x7fa45c105e20 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 -- 172.21.15.121:0/1348341815 >> 172.21.15.121:0/1348341815 conn(0x7fa45c0755a0 msgr2=0x7fa45c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 -- 172.21.15.121:0/1348341815 shutdown_connections 2024-04-05T21:54:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 -- 172.21.15.121:0/1348341815 wait complete. 2024-04-05T21:54:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 Processor -- start 2024-04-05T21:54:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 -- start start 2024-04-05T21:54:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa45c0ffbd0 0x7fa45c196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa45c105a40 0x7fa45c196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c197260 con 0x7fa45c105a40 2024-04-05T21:54:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.549+0000 7fa4630e4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c19aff0 con 0x7fa45c0ffbd0 2024-04-05T21:54:13.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.550+0000 7fa45bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa45c105a40 0x7fa45c196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:13.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.550+0000 7fa45bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa45c105a40 0x7fa45c196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59388/0 (socket says 172.21.15.121:59388) 2024-04-05T21:54:13.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.550+0000 7fa45bfff700 1 -- 172.21.15.121:0/2615701779 learned_addr learned my addr 172.21.15.121:0/2615701779 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:13.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.550+0000 7fa45bfff700 1 -- 172.21.15.121:0/2615701779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa45c0ffbd0 msgr2=0x7fa45c196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:13.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.550+0000 7fa460e80700 1 --2- 172.21.15.121:0/2615701779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa45c0ffbd0 0x7fa45c196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:13.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.550+0000 7fa45bfff700 1 --2- 172.21.15.121:0/2615701779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa45c0ffbd0 0x7fa45c196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.550+0000 7fa45bfff700 1 -- 172.21.15.121:0/2615701779 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4500095f0 con 0x7fa45c105a40 2024-04-05T21:54:13.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.550+0000 7fa45bfff700 1 --2- 172.21.15.121:0/2615701779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa45c105a40 0x7fa45c196b80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa44800d8d0 tx=0x7fa44800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:13.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.550+0000 7fa459ffb700 1 -- 172.21.15.121:0/2615701779 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa448009940 con 0x7fa45c105a40 2024-04-05T21:54:13.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.550+0000 7fa4630e4700 1 -- 172.21.15.121:0/2615701779 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45c19b2d0 con 0x7fa45c105a40 2024-04-05T21:54:13.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.550+0000 7fa459ffb700 1 -- 172.21.15.121:0/2615701779 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa448010460 con 0x7fa45c105a40 2024-04-05T21:54:13.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.550+0000 7fa459ffb700 1 -- 172.21.15.121:0/2615701779 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa44800f5f0 con 0x7fa45c105a40 2024-04-05T21:54:13.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.550+0000 7fa4630e4700 1 -- 172.21.15.121:0/2615701779 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c19b820 con 0x7fa45c105a40 2024-04-05T21:54:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.551+0000 7fa4630e4700 1 -- 172.21.15.121:0/2615701779 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45c109050 con 0x7fa45c105a40 2024-04-05T21:54:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.552+0000 7fa459ffb700 1 -- 172.21.15.121:0/2615701779 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa4480105d0 con 0x7fa45c105a40 2024-04-05T21:54:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.552+0000 7fa459ffb700 1 --2- 172.21.15.121:0/2615701779 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa44c06cce0 0x7fa44c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.552+0000 7fa460e80700 1 --2- 172.21.15.121:0/2615701779 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa44c06cce0 0x7fa44c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:13.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.552+0000 7fa459ffb700 1 -- 172.21.15.121:0/2615701779 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa44808c5e0 con 0x7fa45c105a40 2024-04-05T21:54:13.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.553+0000 7fa460e80700 1 --2- 172.21.15.121:0/2615701779 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa44c06cce0 0x7fa44c06f1a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa45000b5c0 tx=0x7fa450005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:13.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.557+0000 7fa459ffb700 1 -- 172.21.15.121:0/2615701779 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa448056650 con 0x7fa45c105a40 2024-04-05T21:54:13.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.829+0000 7fa4630e4700 1 -- 172.21.15.121:0/2615701779 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa45c19bb30 con 0x7fa45c105a40 2024-04-05T21:54:13.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.829+0000 7fa459ffb700 1 -- 172.21.15.121:0/2615701779 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fa448020e10 con 0x7fa45c105a40 2024-04-05T21:54:13.831 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:54:13.831 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:54:13.831 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:54:13.831 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:54:13.831 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:54:13.831 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:54:13.831 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:54:13.831 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:54:13.831 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:54:13.832 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:54:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.832+0000 7fa4630e4700 1 -- 172.21.15.121:0/2615701779 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa44c06cce0 msgr2=0x7fa44c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.832+0000 7fa4630e4700 1 --2- 172.21.15.121:0/2615701779 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa44c06cce0 0x7fa44c06f1a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa45000b5c0 tx=0x7fa450005fd0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.832+0000 7fa4630e4700 1 -- 172.21.15.121:0/2615701779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa45c105a40 msgr2=0x7fa45c196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.832+0000 7fa4630e4700 1 --2- 172.21.15.121:0/2615701779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa45c105a40 0x7fa45c196b80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa44800d8d0 tx=0x7fa44800dc90 comp rx=0 tx=0).stop 2024-04-05T21:54:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.832+0000 7fa4630e4700 1 -- 172.21.15.121:0/2615701779 shutdown_connections 2024-04-05T21:54:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.832+0000 7fa4630e4700 1 --2- 172.21.15.121:0/2615701779 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa44c06cce0 0x7fa44c06f1a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.832+0000 7fa4630e4700 1 --2- 172.21.15.121:0/2615701779 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa45c0ffbd0 0x7fa45c196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.832+0000 7fa4630e4700 1 --2- 172.21.15.121:0/2615701779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa45c105a40 0x7fa45c196b80 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.832+0000 7fa4630e4700 1 -- 172.21.15.121:0/2615701779 >> 172.21.15.121:0/2615701779 conn(0x7fa45c0755a0 msgr2=0x7fa45c068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.832+0000 7fa4630e4700 1 -- 172.21.15.121:0/2615701779 shutdown_connections 2024-04-05T21:54:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:13.832+0000 7fa4630e4700 1 -- 172.21.15.121:0/2615701779 wait complete. 2024-04-05T21:54:14.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:13 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/306729096' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:54:14.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:13 smithi121 ceph-mon[145859]: from='client.35374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:14.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:13 smithi121 ceph-mon[145859]: pgmap v841: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:14.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:13 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2615701779' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:54:14.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:13 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/306729096' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:54:14.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:13 smithi188 ceph-mon[107793]: from='client.35374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:14.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:13 smithi188 ceph-mon[107793]: pgmap v841: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:14.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:13 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2615701779' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:54:15.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:54:15.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:54:16.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:16 smithi188 ceph-mon[107793]: pgmap v842: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:16.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:16 smithi121 ceph-mon[145859]: pgmap v842: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:19.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:18 smithi121 ceph-mon[145859]: pgmap v843: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:19.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:18 smithi188 ceph-mon[107793]: pgmap v843: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:21.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:20 smithi121 ceph-mon[145859]: pgmap v844: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:20 smithi188 ceph-mon[107793]: pgmap v844: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:23.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:22 smithi121 ceph-mon[145859]: pgmap v845: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:23.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:22 smithi188 ceph-mon[107793]: pgmap v845: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:25.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:24 smithi121 ceph-mon[145859]: pgmap v846: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:24 smithi188 ceph-mon[107793]: pgmap v846: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:27.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:26 smithi121 ceph-mon[145859]: pgmap v847: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:27.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:26 smithi188 ceph-mon[107793]: pgmap v847: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:29.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:28 smithi121 ceph-mon[145859]: pgmap v848: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:28 smithi188 ceph-mon[107793]: pgmap v848: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:30 smithi121 ceph-mon[145859]: pgmap v849: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:31.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:54:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:30 smithi188 ceph-mon[107793]: pgmap v849: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:54:33.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:32 smithi121 ceph-mon[145859]: pgmap v850: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:33.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:32 smithi188 ceph-mon[107793]: pgmap v850: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:35.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:34 smithi121 ceph-mon[145859]: pgmap v851: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:35.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:34 smithi188 ceph-mon[107793]: pgmap v851: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:37.100 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:36 smithi121 ceph-mon[145859]: pgmap v852: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:36 smithi188 ceph-mon[107793]: pgmap v852: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:39.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:38 smithi121 ceph-mon[145859]: pgmap v853: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:38 smithi188 ceph-mon[107793]: pgmap v853: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:41.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:40 smithi188 ceph-mon[107793]: pgmap v854: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:40 smithi121 ceph-mon[145859]: pgmap v854: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:43.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:42 smithi121 ceph-mon[145859]: pgmap v855: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:42 smithi188 ceph-mon[107793]: pgmap v855: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:43.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.948+0000 7f8e398ca700 1 -- 172.21.15.121:0/2646387070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e34101750 msgr2=0x7f8e341057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:43.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.948+0000 7f8e398ca700 1 --2- 172.21.15.121:0/2646387070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e34101750 0x7f8e341057a0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f8e24009910 tx=0x7f8e24009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:43.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.948+0000 7f8e398ca700 1 -- 172.21.15.121:0/2646387070 shutdown_connections 2024-04-05T21:54:43.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.948+0000 7f8e398ca700 1 --2- 172.21.15.121:0/2646387070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e34101750 0x7f8e341057a0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:43.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.948+0000 7f8e398ca700 1 --2- 172.21.15.121:0/2646387070 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e34100da0 0x7f8e34101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:43.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.948+0000 7f8e398ca700 1 -- 172.21.15.121:0/2646387070 >> 172.21.15.121:0/2646387070 conn(0x7f8e340fc650 msgr2=0x7f8e340fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:43.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.948+0000 7f8e398ca700 1 -- 172.21.15.121:0/2646387070 shutdown_connections 2024-04-05T21:54:43.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.948+0000 7f8e398ca700 1 -- 172.21.15.121:0/2646387070 wait complete. 2024-04-05T21:54:43.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.949+0000 7f8e398ca700 1 Processor -- start 2024-04-05T21:54:43.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.949+0000 7f8e398ca700 1 -- start start 2024-04-05T21:54:43.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.949+0000 7f8e398ca700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e34100da0 0x7f8e34073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:43.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.949+0000 7f8e398ca700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e34101750 0x7f8e340735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:43.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.949+0000 7f8e398ca700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e34073b10 con 0x7f8e34101750 2024-04-05T21:54:43.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.949+0000 7f8e398ca700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e34073c50 con 0x7f8e34100da0 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e32ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e34100da0 0x7f8e34073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e327fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e34101750 0x7f8e340735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e32ffd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e34100da0 0x7f8e34073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.188:3300/0 says I am v2:172.21.15.121:58270/0 (socket says 172.21.15.121:58270) 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e327fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e34101750 0x7f8e340735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46710/0 (socket says 172.21.15.121:46710) 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e32ffd700 1 -- 172.21.15.121:0/2510213626 learned_addr learned my addr 172.21.15.121:0/2510213626 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e327fc700 1 -- 172.21.15.121:0/2510213626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e34100da0 msgr2=0x7f8e34073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e327fc700 1 --2- 172.21.15.121:0/2510213626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e34100da0 0x7f8e34073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e327fc700 1 -- 172.21.15.121:0/2510213626 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e240095f0 con 0x7f8e34101750 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e327fc700 1 --2- 172.21.15.121:0/2510213626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e34101750 0x7f8e340735d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f8e24000c00 tx=0x7f8e2400ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e388c8700 1 -- 172.21.15.121:0/2510213626 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e2401d070 con 0x7f8e34101750 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e388c8700 1 -- 172.21.15.121:0/2510213626 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e2400bea0 con 0x7f8e34101750 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e398ca700 1 -- 172.21.15.121:0/2510213626 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e34073ed0 con 0x7f8e34101750 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e388c8700 1 -- 172.21.15.121:0/2510213626 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e2400fd00 con 0x7f8e34101750 2024-04-05T21:54:43.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.950+0000 7f8e398ca700 1 -- 172.21.15.121:0/2510213626 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e341a2b70 con 0x7f8e34101750 2024-04-05T21:54:43.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.951+0000 7f8e398ca700 1 -- 172.21.15.121:0/2510213626 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e341091a0 con 0x7f8e34101750 2024-04-05T21:54:43.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.952+0000 7f8e388c8700 1 -- 172.21.15.121:0/2510213626 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8e2400f560 con 0x7f8e34101750 2024-04-05T21:54:43.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.952+0000 7f8e388c8700 1 --2- 172.21.15.121:0/2510213626 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e2006cce0 0x7f8e2006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:43.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.952+0000 7f8e388c8700 1 -- 172.21.15.121:0/2510213626 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8e2408d5e0 con 0x7f8e34101750 2024-04-05T21:54:43.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.952+0000 7f8e32ffd700 1 --2- 172.21.15.121:0/2510213626 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e2006cce0 0x7f8e2006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:43.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.952+0000 7f8e32ffd700 1 --2- 172.21.15.121:0/2510213626 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e2006cce0 0x7f8e2006f1a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c009d20 tx=0x7f8e1c009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:43.957+0000 7f8e388c8700 1 -- 172.21.15.121:0/2510213626 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8e24057650 con 0x7f8e34101750 2024-04-05T21:54:44.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.167+0000 7f8e398ca700 1 -- 172.21.15.121:0/2510213626 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e3402cd30 con 0x7f8e2006cce0 2024-04-05T21:54:44.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.168+0000 7f8e388c8700 1 -- 172.21.15.121:0/2510213626 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8e3402cd30 con 0x7f8e2006cce0 2024-04-05T21:54:44.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.170+0000 7f8e398ca700 1 -- 172.21.15.121:0/2510213626 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e2006cce0 msgr2=0x7f8e2006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:44.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.170+0000 7f8e398ca700 1 --2- 172.21.15.121:0/2510213626 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e2006cce0 0x7f8e2006f1a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c009d20 tx=0x7f8e1c009260 comp rx=0 tx=0).stop 2024-04-05T21:54:44.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.170+0000 7f8e398ca700 1 -- 172.21.15.121:0/2510213626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e34101750 msgr2=0x7f8e340735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:44.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.170+0000 7f8e398ca700 1 --2- 172.21.15.121:0/2510213626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e34101750 0x7f8e340735d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f8e24000c00 tx=0x7f8e2400ba70 comp rx=0 tx=0).stop 2024-04-05T21:54:44.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.170+0000 7f8e398ca700 1 -- 172.21.15.121:0/2510213626 shutdown_connections 2024-04-05T21:54:44.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.170+0000 7f8e398ca700 1 --2- 172.21.15.121:0/2510213626 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e2006cce0 0x7f8e2006f1a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.170+0000 7f8e398ca700 1 --2- 172.21.15.121:0/2510213626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8e34100da0 0x7f8e34073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.170+0000 7f8e398ca700 1 --2- 172.21.15.121:0/2510213626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e34101750 0x7f8e340735d0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.170+0000 7f8e398ca700 1 -- 172.21.15.121:0/2510213626 >> 172.21.15.121:0/2510213626 conn(0x7f8e340fc650 msgr2=0x7f8e340fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:44.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.170+0000 7f8e398ca700 1 -- 172.21.15.121:0/2510213626 shutdown_connections 2024-04-05T21:54:44.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.170+0000 7f8e398ca700 1 -- 172.21.15.121:0/2510213626 wait complete. 2024-04-05T21:54:44.181 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:54:44.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.274+0000 7fac22e8e700 1 -- 172.21.15.121:0/2308884272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac1c100da0 msgr2=0x7fac1c101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:44.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.274+0000 7fac22e8e700 1 --2- 172.21.15.121:0/2308884272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac1c100da0 0x7fac1c101180 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fac10009910 tx=0x7fac10009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:44.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.274+0000 7fac22e8e700 1 -- 172.21.15.121:0/2308884272 shutdown_connections 2024-04-05T21:54:44.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.274+0000 7fac22e8e700 1 --2- 172.21.15.121:0/2308884272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac1c101750 0x7fac1c1057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.274+0000 7fac22e8e700 1 --2- 172.21.15.121:0/2308884272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac1c100da0 0x7fac1c101180 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.274+0000 7fac22e8e700 1 -- 172.21.15.121:0/2308884272 >> 172.21.15.121:0/2308884272 conn(0x7fac1c0fc630 msgr2=0x7fac1c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:44.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.274+0000 7fac22e8e700 1 -- 172.21.15.121:0/2308884272 shutdown_connections 2024-04-05T21:54:44.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.274+0000 7fac22e8e700 1 -- 172.21.15.121:0/2308884272 wait complete. 2024-04-05T21:54:44.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.274+0000 7fac22e8e700 1 Processor -- start 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.274+0000 7fac22e8e700 1 -- start start 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac22e8e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac1c100da0 0x7fac1c196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac22e8e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac1c101750 0x7fac1c196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac22e8e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c197260 con 0x7fac1c100da0 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac22e8e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c199f90 con 0x7fac1c101750 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac20c2a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac1c100da0 0x7fac1c196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac20c2a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac1c100da0 0x7fac1c196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46732/0 (socket says 172.21.15.121:46732) 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac20c2a700 1 -- 172.21.15.121:0/4126443320 learned_addr learned my addr 172.21.15.121:0/4126443320 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac1bfff700 1 --2- 172.21.15.121:0/4126443320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac1c101750 0x7fac1c196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac20c2a700 1 -- 172.21.15.121:0/4126443320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac1c101750 msgr2=0x7fac1c196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac20c2a700 1 --2- 172.21.15.121:0/4126443320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac1c101750 0x7fac1c196bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac20c2a700 1 -- 172.21.15.121:0/4126443320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac100095f0 con 0x7fac1c100da0 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac20c2a700 1 --2- 172.21.15.121:0/4126443320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac1c100da0 0x7fac1c196690 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fac10000c00 tx=0x7fac1000bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac19ffb700 1 -- 172.21.15.121:0/4126443320 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac1001d070 con 0x7fac1c100da0 2024-04-05T21:54:44.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac19ffb700 1 -- 172.21.15.121:0/4126443320 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac10005710 con 0x7fac1c100da0 2024-04-05T21:54:44.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac22e8e700 1 -- 172.21.15.121:0/4126443320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1c19a210 con 0x7fac1c100da0 2024-04-05T21:54:44.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.275+0000 7fac19ffb700 1 -- 172.21.15.121:0/4126443320 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac100039d0 con 0x7fac1c100da0 2024-04-05T21:54:44.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.276+0000 7fac22e8e700 1 -- 172.21.15.121:0/4126443320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c19a6a0 con 0x7fac1c100da0 2024-04-05T21:54:44.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.277+0000 7fac22e8e700 1 -- 172.21.15.121:0/4126443320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1c109140 con 0x7fac1c100da0 2024-04-05T21:54:44.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.277+0000 7fac19ffb700 1 -- 172.21.15.121:0/4126443320 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fac10003c50 con 0x7fac1c100da0 2024-04-05T21:54:44.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.278+0000 7fac19ffb700 1 --2- 172.21.15.121:0/4126443320 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fac0c06cce0 0x7fac0c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:44.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.278+0000 7fac1bfff700 1 --2- 172.21.15.121:0/4126443320 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fac0c06cce0 0x7fac0c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:44.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.278+0000 7fac19ffb700 1 -- 172.21.15.121:0/4126443320 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fac1008db70 con 0x7fac1c100da0 2024-04-05T21:54:44.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.278+0000 7fac1bfff700 1 --2- 172.21.15.121:0/4126443320 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fac0c06cce0 0x7fac0c06f1a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fac1c197cb0 tx=0x7fac08009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:44.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.282+0000 7fac19ffb700 1 -- 172.21.15.121:0/4126443320 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fac10057be0 con 0x7fac1c100da0 2024-04-05T21:54:44.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.491+0000 7fac22e8e700 1 -- 172.21.15.121:0/4126443320 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fac1c19a920 con 0x7fac0c06cce0 2024-04-05T21:54:44.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.492+0000 7fac19ffb700 1 -- 172.21.15.121:0/4126443320 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fac1c19a920 con 0x7fac0c06cce0 2024-04-05T21:54:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.494+0000 7fac22e8e700 1 -- 172.21.15.121:0/4126443320 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fac0c06cce0 msgr2=0x7fac0c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.494+0000 7fac22e8e700 1 --2- 172.21.15.121:0/4126443320 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fac0c06cce0 0x7fac0c06f1a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fac1c197cb0 tx=0x7fac08009260 comp rx=0 tx=0).stop 2024-04-05T21:54:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.494+0000 7fac22e8e700 1 -- 172.21.15.121:0/4126443320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac1c100da0 msgr2=0x7fac1c196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.494+0000 7fac22e8e700 1 --2- 172.21.15.121:0/4126443320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac1c100da0 0x7fac1c196690 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fac10000c00 tx=0x7fac1000bde0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.494+0000 7fac22e8e700 1 -- 172.21.15.121:0/4126443320 shutdown_connections 2024-04-05T21:54:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.494+0000 7fac22e8e700 1 --2- 172.21.15.121:0/4126443320 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fac0c06cce0 0x7fac0c06f1a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.494+0000 7fac22e8e700 1 --2- 172.21.15.121:0/4126443320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac1c100da0 0x7fac1c196690 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.494+0000 7fac22e8e700 1 --2- 172.21.15.121:0/4126443320 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fac1c101750 0x7fac1c196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.494+0000 7fac22e8e700 1 -- 172.21.15.121:0/4126443320 >> 172.21.15.121:0/4126443320 conn(0x7fac1c0fc630 msgr2=0x7fac1c0fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.494+0000 7fac22e8e700 1 -- 172.21.15.121:0/4126443320 shutdown_connections 2024-04-05T21:54:44.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.494+0000 7fac22e8e700 1 -- 172.21.15.121:0/4126443320 wait complete. 2024-04-05T21:54:44.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.596+0000 7f28b041c700 1 -- 172.21.15.121:0/2118427458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a80ffdb0 msgr2=0x7f28a8103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.596+0000 7f28b041c700 1 --2- 172.21.15.121:0/2118427458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a80ffdb0 0x7f28a8103e00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f28a4009910 tx=0x7f28a4009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.596+0000 7f28b041c700 1 -- 172.21.15.121:0/2118427458 shutdown_connections 2024-04-05T21:54:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.596+0000 7f28b041c700 1 --2- 172.21.15.121:0/2118427458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a80ffdb0 0x7f28a8103e00 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.596+0000 7f28b041c700 1 --2- 172.21.15.121:0/2118427458 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a80ff400 0x7f28a80ff7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.596+0000 7f28b041c700 1 -- 172.21.15.121:0/2118427458 >> 172.21.15.121:0/2118427458 conn(0x7f28a8075e30 msgr2=0x7f28a80fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.596+0000 7f28b041c700 1 -- 172.21.15.121:0/2118427458 shutdown_connections 2024-04-05T21:54:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.596+0000 7f28b041c700 1 -- 172.21.15.121:0/2118427458 wait complete. 2024-04-05T21:54:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.597+0000 7f28b041c700 1 Processor -- start 2024-04-05T21:54:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.597+0000 7f28b041c700 1 -- start start 2024-04-05T21:54:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.597+0000 7f28b041c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a80ff400 0x7f28a80fb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.597+0000 7f28b041c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a80ffdb0 0x7f28a80fb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.597+0000 7f28b041c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a80fbd20 con 0x7f28a80ffdb0 2024-04-05T21:54:44.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.597+0000 7f28b041c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a80fbe60 con 0x7f28a80ff400 2024-04-05T21:54:44.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.597+0000 7f28ad9b7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a80ffdb0 0x7f28a80fb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:44.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.597+0000 7f28ad9b7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a80ffdb0 0x7f28a80fb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45510/0 (socket says 172.21.15.121:45510) 2024-04-05T21:54:44.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.597+0000 7f28ad9b7700 1 -- 172.21.15.121:0/3467789639 learned_addr learned my addr 172.21.15.121:0/3467789639 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.598+0000 7f28ad9b7700 1 -- 172.21.15.121:0/3467789639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a80ff400 msgr2=0x7f28a80fb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.598+0000 7f28ad9b7700 1 --2- 172.21.15.121:0/3467789639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a80ff400 0x7f28a80fb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.598+0000 7f28ad9b7700 1 -- 172.21.15.121:0/3467789639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a40095f0 con 0x7f28a80ffdb0 2024-04-05T21:54:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.598+0000 7f28ad9b7700 1 --2- 172.21.15.121:0/3467789639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a80ffdb0 0x7f28a80fb7e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f28a40092e0 tx=0x7f28a400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.598+0000 7f289f7fe700 1 -- 172.21.15.121:0/3467789639 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28a401d070 con 0x7f28a80ffdb0 2024-04-05T21:54:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.598+0000 7f289f7fe700 1 -- 172.21.15.121:0/3467789639 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28a4005940 con 0x7f28a80ffdb0 2024-04-05T21:54:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.598+0000 7f289f7fe700 1 -- 172.21.15.121:0/3467789639 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28a400fe50 con 0x7f28a80ffdb0 2024-04-05T21:54:44.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.598+0000 7f28b041c700 1 -- 172.21.15.121:0/3467789639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a80fc0e0 con 0x7f28a80ffdb0 2024-04-05T21:54:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.598+0000 7f28b041c700 1 -- 172.21.15.121:0/3467789639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a81a2b20 con 0x7f28a80ffdb0 2024-04-05T21:54:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.599+0000 7f28b041c700 1 -- 172.21.15.121:0/3467789639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a8109810 con 0x7f28a80ffdb0 2024-04-05T21:54:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.599+0000 7f289f7fe700 1 -- 172.21.15.121:0/3467789639 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f28a4003a60 con 0x7f28a80ffdb0 2024-04-05T21:54:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.600+0000 7f289f7fe700 1 --2- 172.21.15.121:0/3467789639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f289406cce0 0x7f289406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.600+0000 7f289f7fe700 1 -- 172.21.15.121:0/3467789639 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f28a408d790 con 0x7f28a80ffdb0 2024-04-05T21:54:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.600+0000 7f28ae1b8700 1 --2- 172.21.15.121:0/3467789639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f289406cce0 0x7f289406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:44.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.600+0000 7f28ae1b8700 1 --2- 172.21.15.121:0/3467789639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f289406cce0 0x7f289406f1a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2898005950 tx=0x7f28980058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:44.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.604+0000 7f289f7fe700 1 -- 172.21.15.121:0/3467789639 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f28a4057800 con 0x7f28a80ffdb0 2024-04-05T21:54:44.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.808+0000 7f28b041c700 1 -- 172.21.15.121:0/3467789639 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f28a802cd50 con 0x7f289406cce0 2024-04-05T21:54:44.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.818+0000 7f289f7fe700 1 -- 172.21.15.121:0/3467789639 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f28a802cd50 con 0x7f289406cce0 2024-04-05T21:54:44.818 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (25m) 4m ago 36m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (37m) 4m ago 37m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (36m) 4m ago 36m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (23m) 4m ago 37m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (23m) 4m ago 36m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (25m) 4m ago 36m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (19m) 4m ago 30m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (19m) 4m ago 30m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (19m) 4m ago 30m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (4m) 4m ago 30m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (27m) 4m ago 38m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (26m) 4m ago 35m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (24m) 4m ago 38m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:54:44.819 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (23m) 4m ago 35m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:54:44.820 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (26m) 4m ago 37m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:54:44.820 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (26m) 4m ago 36m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:54:44.820 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (23m) 4m ago 35m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:54:44.820 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (22m) 4m ago 34m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:54:44.820 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (22m) 4m ago 33m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:54:44.820 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (21m) 4m ago 33m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:54:44.820 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (20m) 4m ago 33m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:54:44.820 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (20m) 4m ago 32m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:54:44.820 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (25m) 4m ago 36m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:54:44.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.820+0000 7f28b041c700 1 -- 172.21.15.121:0/3467789639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f289406cce0 msgr2=0x7f289406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:44.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.820+0000 7f28b041c700 1 --2- 172.21.15.121:0/3467789639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f289406cce0 0x7f289406f1a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2898005950 tx=0x7f28980058e0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.820+0000 7f28b041c700 1 -- 172.21.15.121:0/3467789639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a80ffdb0 msgr2=0x7f28a80fb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:44.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.820+0000 7f28b041c700 1 --2- 172.21.15.121:0/3467789639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a80ffdb0 0x7f28a80fb7e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f28a40092e0 tx=0x7f28a400be20 comp rx=0 tx=0).stop 2024-04-05T21:54:44.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.820+0000 7f28b041c700 1 -- 172.21.15.121:0/3467789639 shutdown_connections 2024-04-05T21:54:44.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.820+0000 7f28b041c700 1 --2- 172.21.15.121:0/3467789639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f289406cce0 0x7f289406f1a0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.820+0000 7f28b041c700 1 --2- 172.21.15.121:0/3467789639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28a80ff400 0x7f28a80fb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.820+0000 7f28b041c700 1 --2- 172.21.15.121:0/3467789639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28a80ffdb0 0x7f28a80fb7e0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.820+0000 7f28b041c700 1 -- 172.21.15.121:0/3467789639 >> 172.21.15.121:0/3467789639 conn(0x7f28a8075e30 msgr2=0x7f28a8102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:44.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.820+0000 7f28b041c700 1 -- 172.21.15.121:0/3467789639 shutdown_connections 2024-04-05T21:54:44.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.820+0000 7f28b041c700 1 -- 172.21.15.121:0/3467789639 wait complete. 2024-04-05T21:54:44.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.921+0000 7f1a1925c700 1 -- 172.21.15.121:0/1762580771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a14101790 msgr2=0x7f1a141057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:44.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.921+0000 7f1a1925c700 1 --2- 172.21.15.121:0/1762580771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a14101790 0x7f1a141057e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009910 tx=0x7f19fc009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:44.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.921+0000 7f1a1925c700 1 -- 172.21.15.121:0/1762580771 shutdown_connections 2024-04-05T21:54:44.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.921+0000 7f1a1925c700 1 --2- 172.21.15.121:0/1762580771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a14101790 0x7f1a141057e0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.921+0000 7f1a1925c700 1 --2- 172.21.15.121:0/1762580771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a14100de0 0x7f1a141011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.921+0000 7f1a1925c700 1 -- 172.21.15.121:0/1762580771 >> 172.21.15.121:0/1762580771 conn(0x7f1a140fc6b0 msgr2=0x7f1a140fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:44.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.921+0000 7f1a1925c700 1 -- 172.21.15.121:0/1762580771 shutdown_connections 2024-04-05T21:54:44.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.921+0000 7f1a1925c700 1 -- 172.21.15.121:0/1762580771 wait complete. 2024-04-05T21:54:44.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.921+0000 7f1a1925c700 1 Processor -- start 2024-04-05T21:54:44.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.922+0000 7f1a1925c700 1 -- start start 2024-04-05T21:54:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.922+0000 7f1a1925c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a14100de0 0x7f1a1419c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.922+0000 7f1a1925c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a14101790 0x7f1a1419caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.922+0000 7f1a1925c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1419d130 con 0x7f1a14100de0 2024-04-05T21:54:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.922+0000 7f1a1925c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a141965e0 con 0x7f1a14101790 2024-04-05T21:54:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.922+0000 7f1a12d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a14100de0 0x7f1a1419c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.922+0000 7f1a12d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a14100de0 0x7f1a1419c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45520/0 (socket says 172.21.15.121:45520) 2024-04-05T21:54:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.922+0000 7f1a12d9d700 1 -- 172.21.15.121:0/2312813751 learned_addr learned my addr 172.21.15.121:0/2312813751 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.922+0000 7f1a1259c700 1 --2- 172.21.15.121:0/2312813751 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a14101790 0x7f1a1419caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.922+0000 7f1a12d9d700 1 -- 172.21.15.121:0/2312813751 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a14101790 msgr2=0x7f1a1419caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.922+0000 7f1a12d9d700 1 --2- 172.21.15.121:0/2312813751 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a14101790 0x7f1a1419caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.922+0000 7f1a12d9d700 1 -- 172.21.15.121:0/2312813751 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc0095f0 con 0x7f1a14100de0 2024-04-05T21:54:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.922+0000 7f1a12d9d700 1 --2- 172.21.15.121:0/2312813751 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a14100de0 0x7f1a1419c560 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00ba50 tx=0x7f1a0c00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.923+0000 7f1a0bfff700 1 -- 172.21.15.121:0/2312813751 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a0c00ca00 con 0x7f1a14100de0 2024-04-05T21:54:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.923+0000 7f1a0bfff700 1 -- 172.21.15.121:0/2312813751 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a0c014440 con 0x7f1a14100de0 2024-04-05T21:54:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.923+0000 7f1a0bfff700 1 -- 172.21.15.121:0/2312813751 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a0c0075a0 con 0x7f1a14100de0 2024-04-05T21:54:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.923+0000 7f1a1925c700 1 -- 172.21.15.121:0/2312813751 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a141968c0 con 0x7f1a14100de0 2024-04-05T21:54:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.923+0000 7f1a1925c700 1 -- 172.21.15.121:0/2312813751 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a14196e10 con 0x7f1a14100de0 2024-04-05T21:54:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.924+0000 7f1a1925c700 1 -- 172.21.15.121:0/2312813751 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a14109130 con 0x7f1a14100de0 2024-04-05T21:54:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.924+0000 7f1a0bfff700 1 -- 172.21.15.121:0/2312813751 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1a0c0145b0 con 0x7f1a14100de0 2024-04-05T21:54:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.924+0000 7f1a0bfff700 1 --2- 172.21.15.121:0/2312813751 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a0006cce0 0x7f1a0006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.925+0000 7f1a1259c700 1 --2- 172.21.15.121:0/2312813751 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a0006cce0 0x7f1a0006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:44.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.925+0000 7f1a0bfff700 1 -- 172.21.15.121:0/2312813751 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1a0c006e40 con 0x7f1a14100de0 2024-04-05T21:54:44.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.925+0000 7f1a1259c700 1 --2- 172.21.15.121:0/2312813751 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a0006cce0 0x7f1a0006f1a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1a140fd020 tx=0x7f19fc01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:44.929+0000 7f1a0bfff700 1 -- 172.21.15.121:0/2312813751 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1a0c0598c0 con 0x7f1a14100de0 2024-04-05T21:54:45.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:44 smithi121 ceph-mon[145859]: pgmap v856: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:45.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.200+0000 7f1a1925c700 1 -- 172.21.15.121:0/2312813751 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1a1404ea90 con 0x7f1a14100de0 2024-04-05T21:54:45.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.201+0000 7f1a0bfff700 1 -- 172.21.15.121:0/2312813751 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f1a0c019290 con 0x7f1a14100de0 2024-04-05T21:54:45.201 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:54:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:54:45.201 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:54:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:54:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:54:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:54:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:54:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:54:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:54:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:54:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:54:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:54:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:54:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:54:45.202 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:54:45.203 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:54:45.203 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:54:45.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.203+0000 7f1a1925c700 1 -- 172.21.15.121:0/2312813751 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a0006cce0 msgr2=0x7f1a0006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.203+0000 7f1a1925c700 1 --2- 172.21.15.121:0/2312813751 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a0006cce0 0x7f1a0006f1a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1a140fd020 tx=0x7f19fc01a040 comp rx=0 tx=0).stop 2024-04-05T21:54:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.203+0000 7f1a1925c700 1 -- 172.21.15.121:0/2312813751 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a14100de0 msgr2=0x7f1a1419c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.203+0000 7f1a1925c700 1 --2- 172.21.15.121:0/2312813751 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a14100de0 0x7f1a1419c560 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00ba50 tx=0x7f1a0c00be10 comp rx=0 tx=0).stop 2024-04-05T21:54:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.203+0000 7f1a1925c700 1 -- 172.21.15.121:0/2312813751 shutdown_connections 2024-04-05T21:54:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.203+0000 7f1a1925c700 1 --2- 172.21.15.121:0/2312813751 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1a0006cce0 0x7f1a0006f1a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.203+0000 7f1a1925c700 1 --2- 172.21.15.121:0/2312813751 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a14100de0 0x7f1a1419c560 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.203+0000 7f1a1925c700 1 --2- 172.21.15.121:0/2312813751 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1a14101790 0x7f1a1419caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.203+0000 7f1a1925c700 1 -- 172.21.15.121:0/2312813751 >> 172.21.15.121:0/2312813751 conn(0x7f1a140fc6b0 msgr2=0x7f1a140fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.203+0000 7f1a1925c700 1 -- 172.21.15.121:0/2312813751 shutdown_connections 2024-04-05T21:54:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.203+0000 7f1a1925c700 1 -- 172.21.15.121:0/2312813751 wait complete. 2024-04-05T21:54:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:44 smithi188 ceph-mon[107793]: pgmap v856: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:45.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.306+0000 7fd065d28700 1 -- 172.21.15.121:0/3741055850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0600ffa40 msgr2=0x7fd0600ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.306+0000 7fd065d28700 1 --2- 172.21.15.121:0/3741055850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0600ffa40 0x7fd0600ffec0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd050009910 tx=0x7fd050009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.306+0000 7fd065d28700 1 -- 172.21.15.121:0/3741055850 shutdown_connections 2024-04-05T21:54:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.306+0000 7fd065d28700 1 --2- 172.21.15.121:0/3741055850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0600ffa40 0x7fd0600ffec0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.306+0000 7fd065d28700 1 --2- 172.21.15.121:0/3741055850 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd060105a40 0x7fd060105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.306+0000 7fd065d28700 1 -- 172.21.15.121:0/3741055850 >> 172.21.15.121:0/3741055850 conn(0x7fd0600fb570 msgr2=0x7fd0600fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.306+0000 7fd065d28700 1 -- 172.21.15.121:0/3741055850 shutdown_connections 2024-04-05T21:54:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.306+0000 7fd065d28700 1 -- 172.21.15.121:0/3741055850 wait complete. 2024-04-05T21:54:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.306+0000 7fd065d28700 1 Processor -- start 2024-04-05T21:54:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd065d28700 1 -- start start 2024-04-05T21:54:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd065d28700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0600ffa40 0x7fd060196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd065d28700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd060105a40 0x7fd060196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd065d28700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd060197230 con 0x7fd060105a40 2024-04-05T21:54:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd065d28700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06019afc0 con 0x7fd0600ffa40 2024-04-05T21:54:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd05effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd060105a40 0x7fd060196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd05effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd060105a40 0x7fd060196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45530/0 (socket says 172.21.15.121:45530) 2024-04-05T21:54:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd05effd700 1 -- 172.21.15.121:0/1239539060 learned_addr learned my addr 172.21.15.121:0/1239539060 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd05f7fe700 1 --2- 172.21.15.121:0/1239539060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0600ffa40 0x7fd060196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd05effd700 1 -- 172.21.15.121:0/1239539060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0600ffa40 msgr2=0x7fd060196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd05effd700 1 --2- 172.21.15.121:0/1239539060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0600ffa40 0x7fd060196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd05effd700 1 -- 172.21.15.121:0/1239539060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0500095f0 con 0x7fd060105a40 2024-04-05T21:54:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd05effd700 1 --2- 172.21.15.121:0/1239539060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd060105a40 0x7fd060196b50 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fd050000c00 tx=0x7fd05000bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd05cff9700 1 -- 172.21.15.121:0/1239539060 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd05001d070 con 0x7fd060105a40 2024-04-05T21:54:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd05cff9700 1 -- 172.21.15.121:0/1239539060 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd050005dc0 con 0x7fd060105a40 2024-04-05T21:54:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd065d28700 1 -- 172.21.15.121:0/1239539060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd06019b240 con 0x7fd060105a40 2024-04-05T21:54:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.307+0000 7fd05cff9700 1 -- 172.21.15.121:0/1239539060 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd050003940 con 0x7fd060105a40 2024-04-05T21:54:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.308+0000 7fd065d28700 1 -- 172.21.15.121:0/1239539060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd06019b6d0 con 0x7fd060105a40 2024-04-05T21:54:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.308+0000 7fd065d28700 1 -- 172.21.15.121:0/1239539060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd060108fe0 con 0x7fd060105a40 2024-04-05T21:54:45.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.309+0000 7fd05cff9700 1 -- 172.21.15.121:0/1239539060 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd050004ae0 con 0x7fd060105a40 2024-04-05T21:54:45.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.309+0000 7fd05cff9700 1 --2- 172.21.15.121:0/1239539060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd04806cce0 0x7fd04806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:45.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.310+0000 7fd05cff9700 1 -- 172.21.15.121:0/1239539060 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd05008db70 con 0x7fd060105a40 2024-04-05T21:54:45.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.310+0000 7fd05f7fe700 1 --2- 172.21.15.121:0/1239539060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd04806cce0 0x7fd04806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:45.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.310+0000 7fd05f7fe700 1 --2- 172.21.15.121:0/1239539060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd04806cce0 0x7fd04806f1a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd054009c60 tx=0x7fd054009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.314+0000 7fd05cff9700 1 -- 172.21.15.121:0/1239539060 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd050057be0 con 0x7fd060105a40 2024-04-05T21:54:45.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.550+0000 7fd065d28700 1 -- 172.21.15.121:0/1239539060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd06004ea90 con 0x7fd060105a40 2024-04-05T21:54:45.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.550+0000 7fd05cff9700 1 -- 172.21.15.121:0/1239539060 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fd050003b30 con 0x7fd060105a40 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:54:45.551 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:54:45.552 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:54:45.553 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:54:45.553 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:54:45.553 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:54:45.553 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:54:45.553 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:54:45.553 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:54:45.553 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:54:45.553 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:54:45.553 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:54:45.553 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:54:45.553 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:54:45.553 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:54:45.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.552+0000 7fd065d28700 1 -- 172.21.15.121:0/1239539060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd04806cce0 msgr2=0x7fd04806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:45.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.552+0000 7fd065d28700 1 --2- 172.21.15.121:0/1239539060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd04806cce0 0x7fd04806f1a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd054009c60 tx=0x7fd054009260 comp rx=0 tx=0).stop 2024-04-05T21:54:45.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.552+0000 7fd065d28700 1 -- 172.21.15.121:0/1239539060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd060105a40 msgr2=0x7fd060196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:45.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.552+0000 7fd065d28700 1 --2- 172.21.15.121:0/1239539060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd060105a40 0x7fd060196b50 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fd050000c00 tx=0x7fd05000bde0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.552+0000 7fd065d28700 1 -- 172.21.15.121:0/1239539060 shutdown_connections 2024-04-05T21:54:45.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.552+0000 7fd065d28700 1 --2- 172.21.15.121:0/1239539060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd04806cce0 0x7fd04806f1a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.552+0000 7fd065d28700 1 --2- 172.21.15.121:0/1239539060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0600ffa40 0x7fd060196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.552+0000 7fd065d28700 1 --2- 172.21.15.121:0/1239539060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd060105a40 0x7fd060196b50 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.552+0000 7fd065d28700 1 -- 172.21.15.121:0/1239539060 >> 172.21.15.121:0/1239539060 conn(0x7fd0600fb570 msgr2=0x7fd060068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:45.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.552+0000 7fd065d28700 1 -- 172.21.15.121:0/1239539060 shutdown_connections 2024-04-05T21:54:45.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.552+0000 7fd065d28700 1 -- 172.21.15.121:0/1239539060 wait complete. 2024-04-05T21:54:45.555 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:54:45.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.656+0000 7f41cedf8700 1 -- 172.21.15.121:0/1391450309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41c8106be0 msgr2=0x7f41c8106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:45.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.656+0000 7f41cedf8700 1 --2- 172.21.15.121:0/1391450309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41c8106be0 0x7f41c8106fc0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009910 tx=0x7f41bc009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:45.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.656+0000 7f41cedf8700 1 -- 172.21.15.121:0/1391450309 shutdown_connections 2024-04-05T21:54:45.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.656+0000 7f41cedf8700 1 --2- 172.21.15.121:0/1391450309 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41c80ff260 0x7f41c80ff6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.656+0000 7f41cedf8700 1 --2- 172.21.15.121:0/1391450309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41c8106be0 0x7f41c8106fc0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.656+0000 7f41cedf8700 1 -- 172.21.15.121:0/1391450309 >> 172.21.15.121:0/1391450309 conn(0x7f41c8074ce0 msgr2=0x7f41c80750f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:45.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.656+0000 7f41cedf8700 1 -- 172.21.15.121:0/1391450309 shutdown_connections 2024-04-05T21:54:45.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.656+0000 7f41cedf8700 1 -- 172.21.15.121:0/1391450309 wait complete. 2024-04-05T21:54:45.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.656+0000 7f41cedf8700 1 Processor -- start 2024-04-05T21:54:45.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.656+0000 7f41cedf8700 1 -- start start 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41cedf8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41c80ff260 0x7f41c8100440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41cedf8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41c8106be0 0x7f41c8100980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41cedf8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c8100ec0 con 0x7f41c80ff260 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41cedf8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c8101000 con 0x7f41c8106be0 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41ccb94700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41c80ff260 0x7f41c8100440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41ccb94700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41c80ff260 0x7f41c8100440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45546/0 (socket says 172.21.15.121:45546) 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41ccb94700 1 -- 172.21.15.121:0/1682569716 learned_addr learned my addr 172.21.15.121:0/1682569716 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41c7fff700 1 --2- 172.21.15.121:0/1682569716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41c8106be0 0x7f41c8100980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41ccb94700 1 -- 172.21.15.121:0/1682569716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41c8106be0 msgr2=0x7f41c8100980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41ccb94700 1 --2- 172.21.15.121:0/1682569716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41c8106be0 0x7f41c8100980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41ccb94700 1 -- 172.21.15.121:0/1682569716 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc0095f0 con 0x7f41c80ff260 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41ccb94700 1 --2- 172.21.15.121:0/1682569716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41c80ff260 0x7f41c8100440 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00be10 tx=0x7f41bc00be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41c5ffb700 1 -- 172.21.15.121:0/1682569716 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41bc01d070 con 0x7f41c80ff260 2024-04-05T21:54:45.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41c5ffb700 1 -- 172.21.15.121:0/1682569716 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41bc005710 con 0x7f41c80ff260 2024-04-05T21:54:45.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41c5ffb700 1 -- 172.21.15.121:0/1682569716 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41bc0039d0 con 0x7f41c80ff260 2024-04-05T21:54:45.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.657+0000 7f41cedf8700 1 -- 172.21.15.121:0/1682569716 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c8101280 con 0x7f41c80ff260 2024-04-05T21:54:45.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.658+0000 7f41cedf8700 1 -- 172.21.15.121:0/1682569716 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c81a2ab0 con 0x7f41c80ff260 2024-04-05T21:54:45.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.658+0000 7f41cedf8700 1 -- 172.21.15.121:0/1682569716 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c804ea90 con 0x7f41c80ff260 2024-04-05T21:54:45.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.659+0000 7f41c5ffb700 1 -- 172.21.15.121:0/1682569716 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f41bc0041f0 con 0x7f41c80ff260 2024-04-05T21:54:45.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.659+0000 7f41c5ffb700 1 --2- 172.21.15.121:0/1682569716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f41b806cce0 0x7f41b806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:45.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.659+0000 7f41c5ffb700 1 -- 172.21.15.121:0/1682569716 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f41bc08db30 con 0x7f41c80ff260 2024-04-05T21:54:45.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.659+0000 7f41c7fff700 1 --2- 172.21.15.121:0/1682569716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f41b806cce0 0x7f41b806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:45.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.660+0000 7f41c7fff700 1 --2- 172.21.15.121:0/1682569716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f41b806cce0 0x7f41b806f1a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f41c8101c30 tx=0x7f41b4009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:45.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.664+0000 7f41c5ffb700 1 -- 172.21.15.121:0/1682569716 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f41bc057c20 con 0x7f41c80ff260 2024-04-05T21:54:45.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.880+0000 7f41cedf8700 1 -- 172.21.15.121:0/1682569716 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f41c80fe810 con 0x7f41b806cce0 2024-04-05T21:54:45.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.881+0000 7f41c5ffb700 1 -- 172.21.15.121:0/1682569716 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f41c80fe810 con 0x7f41b806cce0 2024-04-05T21:54:45.882 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:54:45.882 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:54:45.882 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:54:45.883 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:54:45.883 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:54:45.883 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:54:45.883 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:54:45.883 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:54:45.883 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:54:45.883 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:54:45.883 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:54:45.883 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:54:45.883 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:54:45.883 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:54:45.883 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:54:45.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.883+0000 7f41cedf8700 1 -- 172.21.15.121:0/1682569716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f41b806cce0 msgr2=0x7f41b806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:45.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.883+0000 7f41cedf8700 1 --2- 172.21.15.121:0/1682569716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f41b806cce0 0x7f41b806f1a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f41c8101c30 tx=0x7f41b4009260 comp rx=0 tx=0).stop 2024-04-05T21:54:45.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.883+0000 7f41cedf8700 1 -- 172.21.15.121:0/1682569716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41c80ff260 msgr2=0x7f41c8100440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:45.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.883+0000 7f41cedf8700 1 --2- 172.21.15.121:0/1682569716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41c80ff260 0x7f41c8100440 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00be10 tx=0x7f41bc00be40 comp rx=0 tx=0).stop 2024-04-05T21:54:45.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.884+0000 7f41cedf8700 1 -- 172.21.15.121:0/1682569716 shutdown_connections 2024-04-05T21:54:45.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.884+0000 7f41cedf8700 1 --2- 172.21.15.121:0/1682569716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f41b806cce0 0x7f41b806f1a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.884+0000 7f41cedf8700 1 --2- 172.21.15.121:0/1682569716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41c80ff260 0x7f41c8100440 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.884+0000 7f41cedf8700 1 --2- 172.21.15.121:0/1682569716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41c8106be0 0x7f41c8100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.884+0000 7f41cedf8700 1 -- 172.21.15.121:0/1682569716 >> 172.21.15.121:0/1682569716 conn(0x7f41c8074ce0 msgr2=0x7f41c80fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:45.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.884+0000 7f41cedf8700 1 -- 172.21.15.121:0/1682569716 shutdown_connections 2024-04-05T21:54:45.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.884+0000 7f41cedf8700 1 -- 172.21.15.121:0/1682569716 wait complete. 2024-04-05T21:54:45.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.985+0000 7f5330190700 1 -- 172.21.15.121:0/4227006672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5328100de0 msgr2=0x7f53281011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.985+0000 7f5330190700 1 --2- 172.21.15.121:0/4227006672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5328100de0 0x7f53281011c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5324009910 tx=0x7f5324009c20 comp rx=0 tx=0).stop 2024-04-05T21:54:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.985+0000 7f5330190700 1 -- 172.21.15.121:0/4227006672 shutdown_connections 2024-04-05T21:54:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.985+0000 7f5330190700 1 --2- 172.21.15.121:0/4227006672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5328101790 0x7f53281057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.985+0000 7f5330190700 1 --2- 172.21.15.121:0/4227006672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5328100de0 0x7f53281011c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.985+0000 7f5330190700 1 -- 172.21.15.121:0/4227006672 >> 172.21.15.121:0/4227006672 conn(0x7f53280fc670 msgr2=0x7f53280fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.986+0000 7f5330190700 1 -- 172.21.15.121:0/4227006672 shutdown_connections 2024-04-05T21:54:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.986+0000 7f5330190700 1 -- 172.21.15.121:0/4227006672 wait complete. 2024-04-05T21:54:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.986+0000 7f5330190700 1 Processor -- start 2024-04-05T21:54:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.986+0000 7f5330190700 1 -- start start 2024-04-05T21:54:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.986+0000 7f5330190700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5328100de0 0x7f5328196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.986+0000 7f5330190700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5328101790 0x7f5328196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.986+0000 7f5330190700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5328197200 con 0x7f5328100de0 2024-04-05T21:54:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.986+0000 7f5330190700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5328197340 con 0x7f5328101790 2024-04-05T21:54:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.986+0000 7f532df2c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5328100de0 0x7f5328196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.986+0000 7f532df2c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5328100de0 0x7f5328196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45562/0 (socket says 172.21.15.121:45562) 2024-04-05T21:54:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.986+0000 7f532df2c700 1 -- 172.21.15.121:0/2635885815 learned_addr learned my addr 172.21.15.121:0/2635885815 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:54:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.987+0000 7f532d72b700 1 --2- 172.21.15.121:0/2635885815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5328101790 0x7f5328196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.987+0000 7f532df2c700 1 -- 172.21.15.121:0/2635885815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5328101790 msgr2=0x7f5328196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.987+0000 7f532df2c700 1 --2- 172.21.15.121:0/2635885815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5328101790 0x7f5328196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.987+0000 7f532df2c700 1 -- 172.21.15.121:0/2635885815 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53240095f0 con 0x7f5328100de0 2024-04-05T21:54:45.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.987+0000 7f532df2c700 1 --2- 172.21.15.121:0/2635885815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5328100de0 0x7f5328196670 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5324006010 tx=0x7f532400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:45.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.987+0000 7f531affd700 1 -- 172.21.15.121:0/2635885815 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f532401d070 con 0x7f5328100de0 2024-04-05T21:54:45.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.987+0000 7f531affd700 1 -- 172.21.15.121:0/2635885815 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f532400f460 con 0x7f5328100de0 2024-04-05T21:54:45.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.987+0000 7f5330190700 1 -- 172.21.15.121:0/2635885815 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532819b130 con 0x7f5328100de0 2024-04-05T21:54:45.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.987+0000 7f531affd700 1 -- 172.21.15.121:0/2635885815 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5324021640 con 0x7f5328100de0 2024-04-05T21:54:45.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.987+0000 7f5330190700 1 -- 172.21.15.121:0/2635885815 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532819b620 con 0x7f5328100de0 2024-04-05T21:54:45.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.988+0000 7f5330190700 1 -- 172.21.15.121:0/2635885815 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5328109180 con 0x7f5328100de0 2024-04-05T21:54:45.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.988+0000 7f531affd700 1 -- 172.21.15.121:0/2635885815 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f532400fa20 con 0x7f5328100de0 2024-04-05T21:54:45.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.989+0000 7f531affd700 1 --2- 172.21.15.121:0/2635885815 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f531406cce0 0x7f531406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:54:45.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.989+0000 7f531affd700 1 -- 172.21.15.121:0/2635885815 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f532408d7b0 con 0x7f5328100de0 2024-04-05T21:54:45.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.989+0000 7f532d72b700 1 --2- 172.21.15.121:0/2635885815 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f531406cce0 0x7f531406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:54:45.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.989+0000 7f532d72b700 1 --2- 172.21.15.121:0/2635885815 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f531406cce0 0x7f531406f1a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f531c005ea0 tx=0x7f531c005e30 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:54:45.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:45.993+0000 7f531affd700 1 -- 172.21.15.121:0/2635885815 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f53240578a0 con 0x7f5328100de0 2024-04-05T21:54:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:45 smithi188 ceph-mon[107793]: from='client.35382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:45 smithi188 ceph-mon[107793]: from='client.35386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:45 smithi188 ceph-mon[107793]: from='client.35390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:45 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2312813751' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:54:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:54:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:45 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1239539060' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:54:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:45 smithi121 ceph-mon[145859]: from='client.35382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:45 smithi121 ceph-mon[145859]: from='client.35386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:45 smithi121 ceph-mon[145859]: from='client.35390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:46.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:45 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2312813751' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:54:46.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:54:46.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:45 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1239539060' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:54:46.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:46.263+0000 7f5330190700 1 -- 172.21.15.121:0/2635885815 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5328108b60 con 0x7f5328100de0 2024-04-05T21:54:46.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:46.263+0000 7f531affd700 1 -- 172.21.15.121:0/2635885815 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f5324026e10 con 0x7f5328100de0 2024-04-05T21:54:46.264 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:54:46.264 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:54:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:54:46.264 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:54:46.264 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:54:46.264 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:54:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:54:46.264 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:54:46.265 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:54:46.266 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:54:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:46.265+0000 7f5330190700 1 -- 172.21.15.121:0/2635885815 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f531406cce0 msgr2=0x7f531406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:46.265+0000 7f5330190700 1 --2- 172.21.15.121:0/2635885815 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f531406cce0 0x7f531406f1a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f531c005ea0 tx=0x7f531c005e30 comp rx=0 tx=0).stop 2024-04-05T21:54:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:46.265+0000 7f5330190700 1 -- 172.21.15.121:0/2635885815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5328100de0 msgr2=0x7f5328196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:54:46.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:46.265+0000 7f5330190700 1 --2- 172.21.15.121:0/2635885815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5328100de0 0x7f5328196670 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5324006010 tx=0x7f532400be20 comp rx=0 tx=0).stop 2024-04-05T21:54:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:46.265+0000 7f5330190700 1 -- 172.21.15.121:0/2635885815 shutdown_connections 2024-04-05T21:54:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:46.265+0000 7f5330190700 1 --2- 172.21.15.121:0/2635885815 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f531406cce0 0x7f531406f1a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:46.265+0000 7f5330190700 1 --2- 172.21.15.121:0/2635885815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5328100de0 0x7f5328196670 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:46.265+0000 7f5330190700 1 --2- 172.21.15.121:0/2635885815 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5328101790 0x7f5328196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:54:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:46.265+0000 7f5330190700 1 -- 172.21.15.121:0/2635885815 >> 172.21.15.121:0/2635885815 conn(0x7f53280fc670 msgr2=0x7f53280fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:54:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:46.266+0000 7f5330190700 1 -- 172.21.15.121:0/2635885815 shutdown_connections 2024-04-05T21:54:46.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:54:46.266+0000 7f5330190700 1 -- 172.21.15.121:0/2635885815 wait complete. 2024-04-05T21:54:47.194 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:46 smithi121 ceph-mon[145859]: pgmap v857: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:47.194 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:46 smithi121 ceph-mon[145859]: from='client.35402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:47.194 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:46 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2635885815' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:54:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:46 smithi188 ceph-mon[107793]: pgmap v857: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:46 smithi188 ceph-mon[107793]: from='client.35402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:54:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:46 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2635885815' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:54:49.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:48 smithi121 ceph-mon[145859]: pgmap v858: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:48 smithi188 ceph-mon[107793]: pgmap v858: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:51.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:50 smithi188 ceph-mon[107793]: pgmap v859: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:51.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:50 smithi121 ceph-mon[145859]: pgmap v859: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:53.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:52 smithi121 ceph-mon[145859]: pgmap v860: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:52 smithi188 ceph-mon[107793]: pgmap v860: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:55.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:54 smithi121 ceph-mon[145859]: pgmap v861: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:54 smithi188 ceph-mon[107793]: pgmap v861: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:56 smithi121 ceph-mon[145859]: pgmap v862: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:56 smithi188 ceph-mon[107793]: pgmap v862: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:59.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:54:58 smithi121 ceph-mon[145859]: pgmap v863: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:54:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:54:58 smithi188 ceph-mon[107793]: pgmap v863: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:00 smithi188 ceph-mon[107793]: pgmap v864: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:55:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:00 smithi121 ceph-mon[145859]: pgmap v864: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:55:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:02 smithi121 ceph-mon[145859]: pgmap v865: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:02 smithi188 ceph-mon[107793]: pgmap v865: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:05.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:04 smithi121 ceph-mon[145859]: pgmap v866: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:04 smithi188 ceph-mon[107793]: pgmap v866: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:07.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:06 smithi121 ceph-mon[145859]: pgmap v867: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:06 smithi188 ceph-mon[107793]: pgmap v867: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:09.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:08 smithi121 ceph-mon[145859]: pgmap v868: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:09.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:08 smithi188 ceph-mon[107793]: pgmap v868: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:11.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:10 smithi121 ceph-mon[145859]: pgmap v869: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:10 smithi188 ceph-mon[107793]: pgmap v869: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:13.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:12 smithi121 ceph-mon[145859]: pgmap v870: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:12 smithi188 ceph-mon[107793]: pgmap v870: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:15.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:14 smithi121 ceph-mon[145859]: pgmap v871: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:14 smithi188 ceph-mon[107793]: pgmap v871: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:55:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:55:16.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.380+0000 7f616bce4700 1 -- 172.21.15.121:0/1649413869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61640ffbd0 msgr2=0x7f6164100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:16.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.381+0000 7f616bce4700 1 --2- 172.21.15.121:0/1649413869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61640ffbd0 0x7f6164100050 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6158009910 tx=0x7f6158009c20 comp rx=0 tx=0).stop 2024-04-05T21:55:16.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.381+0000 7f616bce4700 1 -- 172.21.15.121:0/1649413869 shutdown_connections 2024-04-05T21:55:16.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.381+0000 7f616bce4700 1 --2- 172.21.15.121:0/1649413869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61640ffbd0 0x7f6164100050 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.381+0000 7f616bce4700 1 --2- 172.21.15.121:0/1649413869 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6164105a40 0x7f6164105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.381+0000 7f616bce4700 1 -- 172.21.15.121:0/1649413869 >> 172.21.15.121:0/1649413869 conn(0x7f61640755a0 msgr2=0x7f61640759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:16.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.381+0000 7f616bce4700 1 -- 172.21.15.121:0/1649413869 shutdown_connections 2024-04-05T21:55:16.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.381+0000 7f616bce4700 1 -- 172.21.15.121:0/1649413869 wait complete. 2024-04-05T21:55:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.381+0000 7f616bce4700 1 Processor -- start 2024-04-05T21:55:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.382+0000 7f616bce4700 1 -- start start 2024-04-05T21:55:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.382+0000 7f616bce4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61640ffbd0 0x7f61641974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.382+0000 7f616bce4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6164105a40 0x7f61641979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.382+0000 7f616bce4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6164198010 con 0x7f61640ffbd0 2024-04-05T21:55:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.382+0000 7f616bce4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6164198150 con 0x7f6164105a40 2024-04-05T21:55:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.382+0000 7f6169a80700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61640ffbd0 0x7f61641974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.382+0000 7f6169a80700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61640ffbd0 0x7f61641974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38006/0 (socket says 172.21.15.121:38006) 2024-04-05T21:55:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.382+0000 7f6169a80700 1 -- 172.21.15.121:0/1351763809 learned_addr learned my addr 172.21.15.121:0/1351763809 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.382+0000 7f616927f700 1 --2- 172.21.15.121:0/1351763809 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6164105a40 0x7f61641979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.382+0000 7f6169a80700 1 -- 172.21.15.121:0/1351763809 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6164105a40 msgr2=0x7f61641979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.382+0000 7f6169a80700 1 --2- 172.21.15.121:0/1351763809 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6164105a40 0x7f61641979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.382+0000 7f6169a80700 1 -- 172.21.15.121:0/1351763809 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61580095f0 con 0x7f61640ffbd0 2024-04-05T21:55:16.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.382+0000 7f6169a80700 1 --2- 172.21.15.121:0/1351763809 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61640ffbd0 0x7f61641974b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f616000d8d0 tx=0x7f616000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:16.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.383+0000 7f6156ffd700 1 -- 172.21.15.121:0/1351763809 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61600098a0 con 0x7f61640ffbd0 2024-04-05T21:55:16.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.383+0000 7f6156ffd700 1 -- 172.21.15.121:0/1351763809 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6160010460 con 0x7f61640ffbd0 2024-04-05T21:55:16.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.383+0000 7f616bce4700 1 -- 172.21.15.121:0/1351763809 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61641a0520 con 0x7f61640ffbd0 2024-04-05T21:55:16.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.383+0000 7f6156ffd700 1 -- 172.21.15.121:0/1351763809 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6160009ea0 con 0x7f61640ffbd0 2024-04-05T21:55:16.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.383+0000 7f616bce4700 1 -- 172.21.15.121:0/1351763809 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61641a09e0 con 0x7f61640ffbd0 2024-04-05T21:55:16.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.384+0000 7f616bce4700 1 -- 172.21.15.121:0/1351763809 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6164107fd0 con 0x7f61640ffbd0 2024-04-05T21:55:16.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.384+0000 7f6156ffd700 1 -- 172.21.15.121:0/1351763809 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f61600105d0 con 0x7f61640ffbd0 2024-04-05T21:55:16.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.384+0000 7f6156ffd700 1 --2- 172.21.15.121:0/1351763809 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f615006cce0 0x7f615006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:16.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.384+0000 7f6156ffd700 1 -- 172.21.15.121:0/1351763809 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f616008c3a0 con 0x7f61640ffbd0 2024-04-05T21:55:16.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.385+0000 7f616927f700 1 --2- 172.21.15.121:0/1351763809 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f615006cce0 0x7f615006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:16.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.385+0000 7f616927f700 1 --2- 172.21.15.121:0/1351763809 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f615006cce0 0x7f615006f1a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6164068cc0 tx=0x7f615801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:16.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.389+0000 7f6156ffd700 1 -- 172.21.15.121:0/1351763809 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6160056490 con 0x7f61640ffbd0 2024-04-05T21:55:16.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.601+0000 7f616bce4700 1 -- 172.21.15.121:0/1351763809 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f616402cd80 con 0x7f615006cce0 2024-04-05T21:55:16.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.603+0000 7f6156ffd700 1 -- 172.21.15.121:0/1351763809 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f616402cd80 con 0x7f615006cce0 2024-04-05T21:55:16.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.605+0000 7f616bce4700 1 -- 172.21.15.121:0/1351763809 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f615006cce0 msgr2=0x7f615006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:16.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.605+0000 7f616bce4700 1 --2- 172.21.15.121:0/1351763809 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f615006cce0 0x7f615006f1a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6164068cc0 tx=0x7f615801a040 comp rx=0 tx=0).stop 2024-04-05T21:55:16.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.605+0000 7f616bce4700 1 -- 172.21.15.121:0/1351763809 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61640ffbd0 msgr2=0x7f61641974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:16.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.605+0000 7f616bce4700 1 --2- 172.21.15.121:0/1351763809 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61640ffbd0 0x7f61641974b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f616000d8d0 tx=0x7f616000dbe0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.605+0000 7f616bce4700 1 -- 172.21.15.121:0/1351763809 shutdown_connections 2024-04-05T21:55:16.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.605+0000 7f616bce4700 1 --2- 172.21.15.121:0/1351763809 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f615006cce0 0x7f615006f1a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.605+0000 7f616bce4700 1 --2- 172.21.15.121:0/1351763809 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f61640ffbd0 0x7f61641974b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.605+0000 7f616bce4700 1 --2- 172.21.15.121:0/1351763809 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6164105a40 0x7f61641979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.605+0000 7f616bce4700 1 -- 172.21.15.121:0/1351763809 >> 172.21.15.121:0/1351763809 conn(0x7f61640755a0 msgr2=0x7f61640fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:16.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.605+0000 7f616bce4700 1 -- 172.21.15.121:0/1351763809 shutdown_connections 2024-04-05T21:55:16.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.605+0000 7f616bce4700 1 -- 172.21.15.121:0/1351763809 wait complete. 2024-04-05T21:55:16.615 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:55:16.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.708+0000 7fd69fa9d700 1 -- 172.21.15.121:0/2092223552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd698105a40 msgr2=0x7fd698105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.708+0000 7fd69fa9d700 1 --2- 172.21.15.121:0/2092223552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd698105a40 0x7fd698105e20 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd69000b0a0 tx=0x7fd69000b3b0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.708+0000 7fd69fa9d700 1 -- 172.21.15.121:0/2092223552 shutdown_connections 2024-04-05T21:55:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.708+0000 7fd69fa9d700 1 --2- 172.21.15.121:0/2092223552 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6980ffbd0 0x7fd698100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.708+0000 7fd69fa9d700 1 --2- 172.21.15.121:0/2092223552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd698105a40 0x7fd698105e20 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.708+0000 7fd69fa9d700 1 -- 172.21.15.121:0/2092223552 >> 172.21.15.121:0/2092223552 conn(0x7fd6980755a0 msgr2=0x7fd6980759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.708+0000 7fd69fa9d700 1 -- 172.21.15.121:0/2092223552 shutdown_connections 2024-04-05T21:55:16.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.708+0000 7fd69fa9d700 1 -- 172.21.15.121:0/2092223552 wait complete. 2024-04-05T21:55:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.709+0000 7fd69fa9d700 1 Processor -- start 2024-04-05T21:55:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.709+0000 7fd69fa9d700 1 -- start start 2024-04-05T21:55:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.709+0000 7fd69fa9d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6980ffbd0 0x7fd6981974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.709+0000 7fd69fa9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd698105a40 0x7fd6981979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.709+0000 7fd69fa9d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd698198000 con 0x7fd698105a40 2024-04-05T21:55:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.709+0000 7fd69fa9d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd698198170 con 0x7fd6980ffbd0 2024-04-05T21:55:16.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.709+0000 7fd69d038700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd698105a40 0x7fd6981979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.709+0000 7fd69d038700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd698105a40 0x7fd6981979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38012/0 (socket says 172.21.15.121:38012) 2024-04-05T21:55:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.709+0000 7fd69d038700 1 -- 172.21.15.121:0/3295361652 learned_addr learned my addr 172.21.15.121:0/3295361652 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.709+0000 7fd69d839700 1 --2- 172.21.15.121:0/3295361652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6980ffbd0 0x7fd6981974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.709+0000 7fd69d038700 1 -- 172.21.15.121:0/3295361652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6980ffbd0 msgr2=0x7fd6981974a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.709+0000 7fd69d038700 1 --2- 172.21.15.121:0/3295361652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6980ffbd0 0x7fd6981974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.709+0000 7fd69d038700 1 -- 172.21.15.121:0/3295361652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd690009d00 con 0x7fd698105a40 2024-04-05T21:55:16.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.710+0000 7fd69d038700 1 --2- 172.21.15.121:0/3295361652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd698105a40 0x7fd6981979e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd68800d8d0 tx=0x7fd68800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:16.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.710+0000 7fd68effd700 1 -- 172.21.15.121:0/3295361652 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd688009940 con 0x7fd698105a40 2024-04-05T21:55:16.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.710+0000 7fd68effd700 1 -- 172.21.15.121:0/3295361652 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd688010460 con 0x7fd698105a40 2024-04-05T21:55:16.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.710+0000 7fd69fa9d700 1 -- 172.21.15.121:0/3295361652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6981a0510 con 0x7fd698105a40 2024-04-05T21:55:16.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.710+0000 7fd68effd700 1 -- 172.21.15.121:0/3295361652 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd68800f5f0 con 0x7fd698105a40 2024-04-05T21:55:16.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.710+0000 7fd69fa9d700 1 -- 172.21.15.121:0/3295361652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6981a0a00 con 0x7fd698105a40 2024-04-05T21:55:16.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.711+0000 7fd69fa9d700 1 -- 172.21.15.121:0/3295361652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd698108040 con 0x7fd698105a40 2024-04-05T21:55:16.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.711+0000 7fd68effd700 1 -- 172.21.15.121:0/3295361652 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd6880105d0 con 0x7fd698105a40 2024-04-05T21:55:16.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.712+0000 7fd68effd700 1 --2- 172.21.15.121:0/3295361652 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd68406cce0 0x7fd68406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:16.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.712+0000 7fd68effd700 1 -- 172.21.15.121:0/3295361652 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd68808c5e0 con 0x7fd698105a40 2024-04-05T21:55:16.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.712+0000 7fd69d839700 1 --2- 172.21.15.121:0/3295361652 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd68406cce0 0x7fd68406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:16.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.712+0000 7fd69d839700 1 --2- 172.21.15.121:0/3295361652 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd68406cce0 0x7fd68406f1a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd69000b830 tx=0x7fd690006000 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:16.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.716+0000 7fd68effd700 1 -- 172.21.15.121:0/3295361652 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd688056650 con 0x7fd698105a40 2024-04-05T21:55:16.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.926+0000 7fd69fa9d700 1 -- 172.21.15.121:0/3295361652 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd69802cd60 con 0x7fd68406cce0 2024-04-05T21:55:16.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.927+0000 7fd68effd700 1 -- 172.21.15.121:0/3295361652 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd69802cd60 con 0x7fd68406cce0 2024-04-05T21:55:16.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.929+0000 7fd69fa9d700 1 -- 172.21.15.121:0/3295361652 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd68406cce0 msgr2=0x7fd68406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:16.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.929+0000 7fd69fa9d700 1 --2- 172.21.15.121:0/3295361652 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd68406cce0 0x7fd68406f1a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd69000b830 tx=0x7fd690006000 comp rx=0 tx=0).stop 2024-04-05T21:55:16.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.929+0000 7fd69fa9d700 1 -- 172.21.15.121:0/3295361652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd698105a40 msgr2=0x7fd6981979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:16.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.929+0000 7fd69fa9d700 1 --2- 172.21.15.121:0/3295361652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd698105a40 0x7fd6981979e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd68800d8d0 tx=0x7fd68800dc90 comp rx=0 tx=0).stop 2024-04-05T21:55:16.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.929+0000 7fd69fa9d700 1 -- 172.21.15.121:0/3295361652 shutdown_connections 2024-04-05T21:55:16.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.929+0000 7fd69fa9d700 1 --2- 172.21.15.121:0/3295361652 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd68406cce0 0x7fd68406f1a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.929+0000 7fd69fa9d700 1 --2- 172.21.15.121:0/3295361652 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6980ffbd0 0x7fd6981974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.929+0000 7fd69fa9d700 1 --2- 172.21.15.121:0/3295361652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd698105a40 0x7fd6981979e0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:16.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.929+0000 7fd69fa9d700 1 -- 172.21.15.121:0/3295361652 >> 172.21.15.121:0/3295361652 conn(0x7fd6980755a0 msgr2=0x7fd6980fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:16.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.929+0000 7fd69fa9d700 1 -- 172.21.15.121:0/3295361652 shutdown_connections 2024-04-05T21:55:16.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:16.929+0000 7fd69fa9d700 1 -- 172.21.15.121:0/3295361652 wait complete. 2024-04-05T21:55:17.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.031+0000 7f5e09ccd700 1 -- 172.21.15.121:0/4240014026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e04100da0 msgr2=0x7f5e04101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:17.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.031+0000 7f5e09ccd700 1 --2- 172.21.15.121:0/4240014026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e04100da0 0x7f5e04101180 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5df8009910 tx=0x7f5df8009c20 comp rx=0 tx=0).stop 2024-04-05T21:55:17.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.031+0000 7f5e09ccd700 1 -- 172.21.15.121:0/4240014026 shutdown_connections 2024-04-05T21:55:17.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.031+0000 7f5e09ccd700 1 --2- 172.21.15.121:0/4240014026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e04101750 0x7f5e041057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.031+0000 7f5e09ccd700 1 --2- 172.21.15.121:0/4240014026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e04100da0 0x7f5e04101180 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.031+0000 7f5e09ccd700 1 -- 172.21.15.121:0/4240014026 >> 172.21.15.121:0/4240014026 conn(0x7f5e040fc630 msgr2=0x7f5e040fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:17.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.031+0000 7f5e09ccd700 1 -- 172.21.15.121:0/4240014026 shutdown_connections 2024-04-05T21:55:17.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.031+0000 7f5e09ccd700 1 -- 172.21.15.121:0/4240014026 wait complete. 2024-04-05T21:55:17.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.032+0000 7f5e09ccd700 1 Processor -- start 2024-04-05T21:55:17.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.032+0000 7f5e09ccd700 1 -- start start 2024-04-05T21:55:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.032+0000 7f5e09ccd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e04100da0 0x7f5e04196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.032+0000 7f5e09ccd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e04101750 0x7f5e04196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.032+0000 7f5e09ccd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e041971d0 con 0x7f5e04101750 2024-04-05T21:55:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.032+0000 7f5e09ccd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e04197340 con 0x7f5e04100da0 2024-04-05T21:55:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.032+0000 7f5e02ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e04101750 0x7f5e04196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.032+0000 7f5e02ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e04101750 0x7f5e04196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38020/0 (socket says 172.21.15.121:38020) 2024-04-05T21:55:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.032+0000 7f5e02ffd700 1 -- 172.21.15.121:0/2062387519 learned_addr learned my addr 172.21.15.121:0/2062387519 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.032+0000 7f5e037fe700 1 --2- 172.21.15.121:0/2062387519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e04100da0 0x7f5e04196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.033+0000 7f5e02ffd700 1 -- 172.21.15.121:0/2062387519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e04100da0 msgr2=0x7f5e04196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.033+0000 7f5e02ffd700 1 --2- 172.21.15.121:0/2062387519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e04100da0 0x7f5e04196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.033+0000 7f5e02ffd700 1 -- 172.21.15.121:0/2062387519 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df80095f0 con 0x7f5e04101750 2024-04-05T21:55:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.033+0000 7f5e02ffd700 1 --2- 172.21.15.121:0/2062387519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e04101750 0x7f5e04196bd0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5df400c960 tx=0x7f5df400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:17.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.033+0000 7f5e00ff9700 1 -- 172.21.15.121:0/2062387519 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5df4007a70 con 0x7f5e04101750 2024-04-05T21:55:17.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.033+0000 7f5e09ccd700 1 -- 172.21.15.121:0/2062387519 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0419a0e0 con 0x7f5e04101750 2024-04-05T21:55:17.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.033+0000 7f5e00ff9700 1 -- 172.21.15.121:0/2062387519 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5df4007bd0 con 0x7f5e04101750 2024-04-05T21:55:17.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.033+0000 7f5e00ff9700 1 -- 172.21.15.121:0/2062387519 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5df4018630 con 0x7f5e04101750 2024-04-05T21:55:17.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.033+0000 7f5e09ccd700 1 -- 172.21.15.121:0/2062387519 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0419a630 con 0x7f5e04101750 2024-04-05T21:55:17.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.034+0000 7f5e09ccd700 1 -- 172.21.15.121:0/2062387519 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e04109140 con 0x7f5e04101750 2024-04-05T21:55:17.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.034+0000 7f5e00ff9700 1 -- 172.21.15.121:0/2062387519 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5df400f450 con 0x7f5e04101750 2024-04-05T21:55:17.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.035+0000 7f5e00ff9700 1 --2- 172.21.15.121:0/2062387519 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5dec06cce0 0x7f5dec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:17.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.035+0000 7f5e00ff9700 1 -- 172.21.15.121:0/2062387519 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5df408c560 con 0x7f5e04101750 2024-04-05T21:55:17.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.035+0000 7f5e037fe700 1 --2- 172.21.15.121:0/2062387519 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5dec06cce0 0x7f5dec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:17.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.035+0000 7f5e037fe700 1 --2- 172.21.15.121:0/2062387519 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5dec06cce0 0x7f5dec06f1a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5df8006010 tx=0x7f5df801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:17.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.039+0000 7f5e00ff9700 1 -- 172.21.15.121:0/2062387519 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5df40565d0 con 0x7f5e04101750 2024-04-05T21:55:17.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.237+0000 7f5e09ccd700 1 -- 172.21.15.121:0/2062387519 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e0402cd60 con 0x7f5dec06cce0 2024-04-05T21:55:17.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:16 smithi121 ceph-mon[145859]: pgmap v872: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:17.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.248+0000 7f5e00ff9700 1 -- 172.21.15.121:0/2062387519 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f5e0402cd60 con 0x7f5dec06cce0 2024-04-05T21:55:17.249 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:55:17.249 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (26m) 4m ago 37m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:55:17.249 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (37m) 4m ago 37m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:55:17.249 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (36m) 4m ago 36m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:55:17.249 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (24m) 4m ago 37m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:55:17.249 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (24m) 4m ago 36m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:55:17.249 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (25m) 4m ago 37m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:55:17.249 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (20m) 4m ago 31m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:55:17.250 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (20m) 4m ago 31m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:55:17.250 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (19m) 4m ago 31m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:55:17.250 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (4m) 4m ago 31m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:55:17.250 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (28m) 4m ago 39m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:55:17.250 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (27m) 4m ago 36m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:55:17.250 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (24m) 4m ago 39m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:55:17.250 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (24m) 4m ago 36m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:55:17.250 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (26m) 4m ago 37m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:55:17.251 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (26m) 4m ago 36m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:55:17.251 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (23m) 4m ago 35m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:55:17.251 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (23m) 4m ago 35m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:55:17.251 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (22m) 4m ago 34m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:55:17.251 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (21m) 4m ago 34m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:55:17.251 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (21m) 4m ago 33m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:55:17.251 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (20m) 4m ago 33m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:55:17.251 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (26m) 4m ago 37m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:55:17.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.250+0000 7f5e09ccd700 1 -- 172.21.15.121:0/2062387519 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5dec06cce0 msgr2=0x7f5dec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:17.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.250+0000 7f5e09ccd700 1 --2- 172.21.15.121:0/2062387519 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5dec06cce0 0x7f5dec06f1a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5df8006010 tx=0x7f5df801a040 comp rx=0 tx=0).stop 2024-04-05T21:55:17.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.250+0000 7f5e09ccd700 1 -- 172.21.15.121:0/2062387519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e04101750 msgr2=0x7f5e04196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:17.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.250+0000 7f5e09ccd700 1 --2- 172.21.15.121:0/2062387519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e04101750 0x7f5e04196bd0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5df400c960 tx=0x7f5df400cc70 comp rx=0 tx=0).stop 2024-04-05T21:55:17.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.250+0000 7f5e09ccd700 1 -- 172.21.15.121:0/2062387519 shutdown_connections 2024-04-05T21:55:17.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.250+0000 7f5e09ccd700 1 --2- 172.21.15.121:0/2062387519 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5dec06cce0 0x7f5dec06f1a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.250+0000 7f5e09ccd700 1 --2- 172.21.15.121:0/2062387519 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e04100da0 0x7f5e04196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.250+0000 7f5e09ccd700 1 --2- 172.21.15.121:0/2062387519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e04101750 0x7f5e04196bd0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.250+0000 7f5e09ccd700 1 -- 172.21.15.121:0/2062387519 >> 172.21.15.121:0/2062387519 conn(0x7f5e040fc630 msgr2=0x7f5e040fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:17.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.250+0000 7f5e09ccd700 1 -- 172.21.15.121:0/2062387519 shutdown_connections 2024-04-05T21:55:17.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.250+0000 7f5e09ccd700 1 -- 172.21.15.121:0/2062387519 wait complete. 2024-04-05T21:55:17.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:16 smithi188 ceph-mon[107793]: pgmap v872: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:17.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.352+0000 7fd719df5700 1 -- 172.21.15.121:0/1098774719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd714069020 msgr2=0x7fd714105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:17.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.352+0000 7fd719df5700 1 --2- 172.21.15.121:0/1098774719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd714069020 0x7fd714105790 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fd704009910 tx=0x7fd704009c20 comp rx=0 tx=0).stop 2024-04-05T21:55:17.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.352+0000 7fd719df5700 1 -- 172.21.15.121:0/1098774719 shutdown_connections 2024-04-05T21:55:17.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.352+0000 7fd719df5700 1 --2- 172.21.15.121:0/1098774719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd714069020 0x7fd714105790 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.352+0000 7fd719df5700 1 --2- 172.21.15.121:0/1098774719 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd714068670 0x7fd714068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.352+0000 7fd719df5700 1 -- 172.21.15.121:0/1098774719 >> 172.21.15.121:0/1098774719 conn(0x7fd7140755a0 msgr2=0x7fd7140759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:17.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.352+0000 7fd719df5700 1 -- 172.21.15.121:0/1098774719 shutdown_connections 2024-04-05T21:55:17.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.352+0000 7fd719df5700 1 -- 172.21.15.121:0/1098774719 wait complete. 2024-04-05T21:55:17.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd719df5700 1 Processor -- start 2024-04-05T21:55:17.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd719df5700 1 -- start start 2024-04-05T21:55:17.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd719df5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd714068670 0x7fd7140ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:17.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd719df5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd714069020 0x7fd7141001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:17.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd719df5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd714100720 con 0x7fd714068670 2024-04-05T21:55:17.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd719df5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd714100860 con 0x7fd714069020 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd7137fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd714068670 0x7fd7140ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd7137fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd714068670 0x7fd7140ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38040/0 (socket says 172.21.15.121:38040) 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd7137fe700 1 -- 172.21.15.121:0/506102343 learned_addr learned my addr 172.21.15.121:0/506102343 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd712ffd700 1 --2- 172.21.15.121:0/506102343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd714069020 0x7fd7141001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd7137fe700 1 -- 172.21.15.121:0/506102343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd714069020 msgr2=0x7fd7141001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd7137fe700 1 --2- 172.21.15.121:0/506102343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd714069020 0x7fd7141001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd7137fe700 1 -- 172.21.15.121:0/506102343 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7040095f0 con 0x7fd714068670 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.353+0000 7fd712ffd700 1 --2- 172.21.15.121:0/506102343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd714069020 0x7fd7141001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.354+0000 7fd7137fe700 1 --2- 172.21.15.121:0/506102343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd714068670 0x7fd7140ffca0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00c930 tx=0x7fd6fc00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.354+0000 7fd710ff9700 1 -- 172.21.15.121:0/506102343 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6fc007ad0 con 0x7fd714068670 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.354+0000 7fd710ff9700 1 -- 172.21.15.121:0/506102343 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6fc007c30 con 0x7fd714068670 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.354+0000 7fd719df5700 1 -- 172.21.15.121:0/506102343 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7141a2810 con 0x7fd714068670 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.354+0000 7fd710ff9700 1 -- 172.21.15.121:0/506102343 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6fc018630 con 0x7fd714068670 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.354+0000 7fd719df5700 1 -- 172.21.15.121:0/506102343 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7141a2b70 con 0x7fd714068670 2024-04-05T21:55:17.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.355+0000 7fd719df5700 1 -- 172.21.15.121:0/506102343 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7141090a0 con 0x7fd714068670 2024-04-05T21:55:17.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.355+0000 7fd710ff9700 1 -- 172.21.15.121:0/506102343 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd6fc00f450 con 0x7fd714068670 2024-04-05T21:55:17.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.355+0000 7fd710ff9700 1 --2- 172.21.15.121:0/506102343 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd70006cce0 0x7fd70006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:17.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.356+0000 7fd710ff9700 1 -- 172.21.15.121:0/506102343 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd6fc08c7d0 con 0x7fd714068670 2024-04-05T21:55:17.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.356+0000 7fd712ffd700 1 --2- 172.21.15.121:0/506102343 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd70006cce0 0x7fd70006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:17.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.356+0000 7fd712ffd700 1 --2- 172.21.15.121:0/506102343 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd70006cce0 0x7fd70006f1a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd714101490 tx=0x7fd70401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:17.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.360+0000 7fd710ff9700 1 -- 172.21.15.121:0/506102343 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd6fc0568f0 con 0x7fd714068670 2024-04-05T21:55:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.642+0000 7fd719df5700 1 -- 172.21.15.121:0/506102343 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd71402cd60 con 0x7fd714068670 2024-04-05T21:55:17.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.643+0000 7fd710ff9700 1 -- 172.21.15.121:0/506102343 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fd6fc059f10 con 0x7fd714068670 2024-04-05T21:55:17.643 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:55:17.643 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:55:17.644 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:55:17.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.645+0000 7fd719df5700 1 -- 172.21.15.121:0/506102343 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd70006cce0 msgr2=0x7fd70006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:17.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.645+0000 7fd719df5700 1 --2- 172.21.15.121:0/506102343 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd70006cce0 0x7fd70006f1a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd714101490 tx=0x7fd70401a040 comp rx=0 tx=0).stop 2024-04-05T21:55:17.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.645+0000 7fd719df5700 1 -- 172.21.15.121:0/506102343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd714068670 msgr2=0x7fd7140ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:17.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.645+0000 7fd719df5700 1 --2- 172.21.15.121:0/506102343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd714068670 0x7fd7140ffca0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc00c930 tx=0x7fd6fc00ccf0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.645+0000 7fd719df5700 1 -- 172.21.15.121:0/506102343 shutdown_connections 2024-04-05T21:55:17.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.645+0000 7fd719df5700 1 --2- 172.21.15.121:0/506102343 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd70006cce0 0x7fd70006f1a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.645+0000 7fd719df5700 1 --2- 172.21.15.121:0/506102343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd714068670 0x7fd7140ffca0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.645+0000 7fd719df5700 1 --2- 172.21.15.121:0/506102343 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd714069020 0x7fd7141001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.645+0000 7fd719df5700 1 -- 172.21.15.121:0/506102343 >> 172.21.15.121:0/506102343 conn(0x7fd7140755a0 msgr2=0x7fd714105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:17.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.645+0000 7fd719df5700 1 -- 172.21.15.121:0/506102343 shutdown_connections 2024-04-05T21:55:17.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.645+0000 7fd719df5700 1 -- 172.21.15.121:0/506102343 wait complete. 2024-04-05T21:55:17.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.748+0000 7fa8ca462700 1 -- 172.21.15.121:0/3824572104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c4101770 msgr2=0x7fa8c41057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:17.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.748+0000 7fa8ca462700 1 --2- 172.21.15.121:0/3824572104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c4101770 0x7fa8c41057c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009910 tx=0x7fa8b8009c20 comp rx=0 tx=0).stop 2024-04-05T21:55:17.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.748+0000 7fa8ca462700 1 -- 172.21.15.121:0/3824572104 shutdown_connections 2024-04-05T21:55:17.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.748+0000 7fa8ca462700 1 --2- 172.21.15.121:0/3824572104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c4101770 0x7fa8c41057c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.748+0000 7fa8ca462700 1 --2- 172.21.15.121:0/3824572104 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8c4100dc0 0x7fa8c41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.748+0000 7fa8ca462700 1 -- 172.21.15.121:0/3824572104 >> 172.21.15.121:0/3824572104 conn(0x7fa8c40fc650 msgr2=0x7fa8c40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:17.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.749+0000 7fa8ca462700 1 -- 172.21.15.121:0/3824572104 shutdown_connections 2024-04-05T21:55:17.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.749+0000 7fa8ca462700 1 -- 172.21.15.121:0/3824572104 wait complete. 2024-04-05T21:55:17.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.749+0000 7fa8ca462700 1 Processor -- start 2024-04-05T21:55:17.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.749+0000 7fa8ca462700 1 -- start start 2024-04-05T21:55:17.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.749+0000 7fa8ca462700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8c4100dc0 0x7fa8c419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:17.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.749+0000 7fa8ca462700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c4101770 0x7fa8c419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:17.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.749+0000 7fa8ca462700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c419f220 con 0x7fa8c4101770 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.749+0000 7fa8ca462700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c41986d0 con 0x7fa8c4100dc0 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.750+0000 7fa8c37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c4101770 0x7fa8c419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.750+0000 7fa8c37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c4101770 0x7fa8c419eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38062/0 (socket says 172.21.15.121:38062) 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.750+0000 7fa8c37fe700 1 -- 172.21.15.121:0/2984670455 learned_addr learned my addr 172.21.15.121:0/2984670455 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.750+0000 7fa8c3fff700 1 --2- 172.21.15.121:0/2984670455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8c4100dc0 0x7fa8c419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.750+0000 7fa8c37fe700 1 -- 172.21.15.121:0/2984670455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8c4100dc0 msgr2=0x7fa8c419e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.750+0000 7fa8c37fe700 1 --2- 172.21.15.121:0/2984670455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8c4100dc0 0x7fa8c419e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.750+0000 7fa8c37fe700 1 -- 172.21.15.121:0/2984670455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b80095f0 con 0x7fa8c4101770 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.750+0000 7fa8c37fe700 1 --2- 172.21.15.121:0/2984670455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c4101770 0x7fa8c419eb90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8000c00 tx=0x7fa8b800bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.750+0000 7fa8c17fa700 1 -- 172.21.15.121:0/2984670455 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8b801d070 con 0x7fa8c4101770 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.750+0000 7fa8ca462700 1 -- 172.21.15.121:0/2984670455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c4198950 con 0x7fa8c4101770 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.750+0000 7fa8c17fa700 1 -- 172.21.15.121:0/2984670455 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8b8003cf0 con 0x7fa8c4101770 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.750+0000 7fa8c17fa700 1 -- 172.21.15.121:0/2984670455 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8b8021410 con 0x7fa8c4101770 2024-04-05T21:55:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.750+0000 7fa8ca462700 1 -- 172.21.15.121:0/2984670455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c4198de0 con 0x7fa8c4101770 2024-04-05T21:55:17.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.751+0000 7fa8ca462700 1 -- 172.21.15.121:0/2984670455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c4104f40 con 0x7fa8c4101770 2024-04-05T21:55:17.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.752+0000 7fa8c17fa700 1 -- 172.21.15.121:0/2984670455 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa8b8003850 con 0x7fa8c4101770 2024-04-05T21:55:17.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.752+0000 7fa8c17fa700 1 --2- 172.21.15.121:0/2984670455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa8ac06cce0 0x7fa8ac06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:17.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.752+0000 7fa8c17fa700 1 -- 172.21.15.121:0/2984670455 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa8b808d840 con 0x7fa8c4101770 2024-04-05T21:55:17.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.752+0000 7fa8c3fff700 1 --2- 172.21.15.121:0/2984670455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa8ac06cce0 0x7fa8ac06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:17.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.753+0000 7fa8c3fff700 1 --2- 172.21.15.121:0/2984670455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa8ac06cce0 0x7fa8ac06f1a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009bf0 tx=0x7fa8b4009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:17.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.757+0000 7fa8c17fa700 1 -- 172.21.15.121:0/2984670455 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa8b805b340 con 0x7fa8c4101770 2024-04-05T21:55:17.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.991+0000 7fa8ca462700 1 -- 172.21.15.121:0/2984670455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa8c404ea90 con 0x7fa8c4101770 2024-04-05T21:55:17.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.992+0000 7fa8c17fa700 1 -- 172.21.15.121:0/2984670455 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fa8b805aed0 con 0x7fa8c4101770 2024-04-05T21:55:17.992 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:55:17.993 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:55:17.993 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:55:17.993 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:55:17.993 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:55:17.993 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:55:17.993 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:55:17.993 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:55:17.993 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:55:17.993 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:55:17.993 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:55:17.993 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:55:17.994 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:55:17.994 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:55:17.994 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:55:17.994 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:55:17.994 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:55:17.994 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:55:17.994 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:55:17.994 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:55:17.994 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:55:17.994 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:55:17.994 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:55:17.994 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:55:17.995 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:55:17.995 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:55:17.995 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:55:17.995 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:55:17.995 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:55:17.995 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:55:17.995 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:55:17.995 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:55:17.995 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:55:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:55:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:55:17.995 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:55:17.995 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:55:17.996 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:55:17.996 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:55:17.996 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:55:17.996 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:55:17.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.994+0000 7fa8ca462700 1 -- 172.21.15.121:0/2984670455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa8ac06cce0 msgr2=0x7fa8ac06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:17.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.994+0000 7fa8ca462700 1 --2- 172.21.15.121:0/2984670455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa8ac06cce0 0x7fa8ac06f1a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009bf0 tx=0x7fa8b4009260 comp rx=0 tx=0).stop 2024-04-05T21:55:17.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.994+0000 7fa8ca462700 1 -- 172.21.15.121:0/2984670455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c4101770 msgr2=0x7fa8c419eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:17.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.994+0000 7fa8ca462700 1 --2- 172.21.15.121:0/2984670455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c4101770 0x7fa8c419eb90 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8000c00 tx=0x7fa8b800bb20 comp rx=0 tx=0).stop 2024-04-05T21:55:17.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.994+0000 7fa8ca462700 1 -- 172.21.15.121:0/2984670455 shutdown_connections 2024-04-05T21:55:17.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.994+0000 7fa8ca462700 1 --2- 172.21.15.121:0/2984670455 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa8ac06cce0 0x7fa8ac06f1a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.994+0000 7fa8ca462700 1 --2- 172.21.15.121:0/2984670455 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa8c4100dc0 0x7fa8c419e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.994+0000 7fa8ca462700 1 --2- 172.21.15.121:0/2984670455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c4101770 0x7fa8c419eb90 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:17.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.994+0000 7fa8ca462700 1 -- 172.21.15.121:0/2984670455 >> 172.21.15.121:0/2984670455 conn(0x7fa8c40fc650 msgr2=0x7fa8c40fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:17.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.994+0000 7fa8ca462700 1 -- 172.21.15.121:0/2984670455 shutdown_connections 2024-04-05T21:55:17.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:17.994+0000 7fa8ca462700 1 -- 172.21.15.121:0/2984670455 wait complete. 2024-04-05T21:55:17.998 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:55:18.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.097+0000 7f9e04117700 1 -- 172.21.15.121:0/3101727068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0ff290 msgr2=0x7f9dfc0ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:18.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.097+0000 7f9e04117700 1 --2- 172.21.15.121:0/3101727068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0ff290 0x7f9dfc0ff670 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9dec009910 tx=0x7f9dec009c20 comp rx=0 tx=0).stop 2024-04-05T21:55:18.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.097+0000 7f9e04117700 1 -- 172.21.15.121:0/3101727068 shutdown_connections 2024-04-05T21:55:18.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.097+0000 7f9e04117700 1 --2- 172.21.15.121:0/3101727068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9dfc0ffc40 0x7f9dfc105780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.097+0000 7f9e04117700 1 --2- 172.21.15.121:0/3101727068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0ff290 0x7f9dfc0ff670 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.097+0000 7f9e04117700 1 -- 172.21.15.121:0/3101727068 >> 172.21.15.121:0/3101727068 conn(0x7f9dfc075e30 msgr2=0x7f9dfc0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:18.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.097+0000 7f9e04117700 1 -- 172.21.15.121:0/3101727068 shutdown_connections 2024-04-05T21:55:18.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.097+0000 7f9e04117700 1 -- 172.21.15.121:0/3101727068 wait complete. 2024-04-05T21:55:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.098+0000 7f9e04117700 1 Processor -- start 2024-04-05T21:55:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.098+0000 7f9e04117700 1 -- start start 2024-04-05T21:55:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.098+0000 7f9e04117700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9dfc0ff290 0x7f9dfc1966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.098+0000 7f9e04117700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0ffc40 0x7f9dfc196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.098+0000 7f9e04117700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc1972a0 con 0x7f9dfc0ffc40 2024-04-05T21:55:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.098+0000 7f9e04117700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc199fd0 con 0x7f9dfc0ff290 2024-04-05T21:55:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.098+0000 7f9e016b2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0ffc40 0x7f9dfc196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.098+0000 7f9e016b2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0ffc40 0x7f9dfc196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38078/0 (socket says 172.21.15.121:38078) 2024-04-05T21:55:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.098+0000 7f9e016b2700 1 -- 172.21.15.121:0/3290703356 learned_addr learned my addr 172.21.15.121:0/3290703356 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.098+0000 7f9e01eb3700 1 --2- 172.21.15.121:0/3290703356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9dfc0ff290 0x7f9dfc1966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.099+0000 7f9e016b2700 1 -- 172.21.15.121:0/3290703356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9dfc0ff290 msgr2=0x7f9dfc1966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.099+0000 7f9e016b2700 1 --2- 172.21.15.121:0/3290703356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9dfc0ff290 0x7f9dfc1966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.099+0000 7f9e016b2700 1 -- 172.21.15.121:0/3290703356 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dec0095f0 con 0x7f9dfc0ffc40 2024-04-05T21:55:18.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.099+0000 7f9e01eb3700 1 --2- 172.21.15.121:0/3290703356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9dfc0ff290 0x7f9dfc1966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:55:18.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.099+0000 7f9e016b2700 1 --2- 172.21.15.121:0/3290703356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0ffc40 0x7f9dfc196c10 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9df800d900 tx=0x7f9df800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:18.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.099+0000 7f9df2ffd700 1 -- 172.21.15.121:0/3290703356 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9df8009900 con 0x7f9dfc0ffc40 2024-04-05T21:55:18.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.099+0000 7f9df2ffd700 1 -- 172.21.15.121:0/3290703356 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9df8010460 con 0x7f9dfc0ffc40 2024-04-05T21:55:18.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.099+0000 7f9e04117700 1 -- 172.21.15.121:0/3290703356 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dfc19a2b0 con 0x7f9dfc0ffc40 2024-04-05T21:55:18.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.099+0000 7f9df2ffd700 1 -- 172.21.15.121:0/3290703356 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9df800f5f0 con 0x7f9dfc0ffc40 2024-04-05T21:55:18.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.099+0000 7f9e04117700 1 -- 172.21.15.121:0/3290703356 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dfc19a800 con 0x7f9dfc0ffc40 2024-04-05T21:55:18.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.100+0000 7f9e04117700 1 -- 172.21.15.121:0/3290703356 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dfc109120 con 0x7f9dfc0ffc40 2024-04-05T21:55:18.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.100+0000 7f9df2ffd700 1 -- 172.21.15.121:0/3290703356 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9df80105d0 con 0x7f9dfc0ffc40 2024-04-05T21:55:18.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.101+0000 7f9df2ffd700 1 --2- 172.21.15.121:0/3290703356 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9de806cce0 0x7f9de806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:18.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.101+0000 7f9df2ffd700 1 -- 172.21.15.121:0/3290703356 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9df808c3c0 con 0x7f9dfc0ffc40 2024-04-05T21:55:18.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.101+0000 7f9e01eb3700 1 --2- 172.21.15.121:0/3290703356 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9de806cce0 0x7f9de806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:18.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.101+0000 7f9e01eb3700 1 --2- 172.21.15.121:0/3290703356 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9de806cce0 0x7f9de806f1a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00b5c0 tx=0x7f9dec01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:18.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.105+0000 7f9df2ffd700 1 -- 172.21.15.121:0/3290703356 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9df8056430 con 0x7f9dfc0ffc40 2024-04-05T21:55:18.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:17 smithi121 ceph-mon[145859]: from='client.35410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:18.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:17 smithi121 ceph-mon[145859]: from='client.35414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:18.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:17 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/506102343' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:55:18.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:17 smithi188 ceph-mon[107793]: from='client.35410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:18.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:17 smithi188 ceph-mon[107793]: from='client.35414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:18.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:17 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/506102343' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:55:18.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.315+0000 7f9e04117700 1 -- 172.21.15.121:0/3290703356 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9dfc19aa20 con 0x7f9de806cce0 2024-04-05T21:55:18.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.316+0000 7f9df2ffd700 1 -- 172.21.15.121:0/3290703356 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9dfc19aa20 con 0x7f9de806cce0 2024-04-05T21:55:18.316 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:55:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:55:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:55:18.316 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:55:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:55:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:55:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:55:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:55:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:55:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:55:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:55:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:55:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:55:18.317 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:55:18.317 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:55:18.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.318+0000 7f9e04117700 1 -- 172.21.15.121:0/3290703356 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9de806cce0 msgr2=0x7f9de806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.318+0000 7f9e04117700 1 --2- 172.21.15.121:0/3290703356 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9de806cce0 0x7f9de806f1a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00b5c0 tx=0x7f9dec01a040 comp rx=0 tx=0).stop 2024-04-05T21:55:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.318+0000 7f9e04117700 1 -- 172.21.15.121:0/3290703356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0ffc40 msgr2=0x7f9dfc196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.318+0000 7f9e04117700 1 --2- 172.21.15.121:0/3290703356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0ffc40 0x7f9dfc196c10 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9df800d900 tx=0x7f9df800dcc0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.318+0000 7f9e04117700 1 -- 172.21.15.121:0/3290703356 shutdown_connections 2024-04-05T21:55:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.318+0000 7f9e04117700 1 --2- 172.21.15.121:0/3290703356 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9de806cce0 0x7f9de806f1a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.318+0000 7f9e04117700 1 --2- 172.21.15.121:0/3290703356 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9dfc0ff290 0x7f9dfc1966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.318+0000 7f9e04117700 1 --2- 172.21.15.121:0/3290703356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0ffc40 0x7f9dfc196c10 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.318+0000 7f9e04117700 1 -- 172.21.15.121:0/3290703356 >> 172.21.15.121:0/3290703356 conn(0x7f9dfc075e30 msgr2=0x7f9dfc104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.318+0000 7f9e04117700 1 -- 172.21.15.121:0/3290703356 shutdown_connections 2024-04-05T21:55:18.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.318+0000 7f9e04117700 1 -- 172.21.15.121:0/3290703356 wait complete. 2024-04-05T21:55:18.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.421+0000 7f28e62be700 1 -- 172.21.15.121:0/824664646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e0068670 msgr2=0x7f28e0068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:18.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.421+0000 7f28e62be700 1 --2- 172.21.15.121:0/824664646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e0068670 0x7f28e0068a50 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009910 tx=0x7f28d4009c20 comp rx=0 tx=0).stop 2024-04-05T21:55:18.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.421+0000 7f28e62be700 1 -- 172.21.15.121:0/824664646 shutdown_connections 2024-04-05T21:55:18.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.421+0000 7f28e62be700 1 --2- 172.21.15.121:0/824664646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28e0069020 0x7f28e0105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.421+0000 7f28e62be700 1 --2- 172.21.15.121:0/824664646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e0068670 0x7f28e0068a50 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.421+0000 7f28e62be700 1 -- 172.21.15.121:0/824664646 >> 172.21.15.121:0/824664646 conn(0x7f28e00755a0 msgr2=0x7f28e00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:18.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.421+0000 7f28e62be700 1 -- 172.21.15.121:0/824664646 shutdown_connections 2024-04-05T21:55:18.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.421+0000 7f28e62be700 1 -- 172.21.15.121:0/824664646 wait complete. 2024-04-05T21:55:18.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.421+0000 7f28e62be700 1 Processor -- start 2024-04-05T21:55:18.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.421+0000 7f28e62be700 1 -- start start 2024-04-05T21:55:18.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.422+0000 7f28e62be700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e0068670 0x7f28e00ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:18.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.422+0000 7f28e62be700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28e0069020 0x7f28e01001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:18.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.422+0000 7f28e62be700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e0100720 con 0x7f28e0068670 2024-04-05T21:55:18.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.422+0000 7f28e62be700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e0100860 con 0x7f28e0069020 2024-04-05T21:55:18.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.422+0000 7f28dffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e0068670 0x7f28e00ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:18.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.422+0000 7f28dffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e0068670 0x7f28e00ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38098/0 (socket says 172.21.15.121:38098) 2024-04-05T21:55:18.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.422+0000 7f28dffff700 1 -- 172.21.15.121:0/594567470 learned_addr learned my addr 172.21.15.121:0/594567470 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:18.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.422+0000 7f28df7fe700 1 --2- 172.21.15.121:0/594567470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28e0069020 0x7f28e01001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:18.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.422+0000 7f28dffff700 1 -- 172.21.15.121:0/594567470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28e0069020 msgr2=0x7f28e01001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:18.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.422+0000 7f28dffff700 1 --2- 172.21.15.121:0/594567470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28e0069020 0x7f28e01001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.422+0000 7f28dffff700 1 -- 172.21.15.121:0/594567470 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d40095f0 con 0x7f28e0068670 2024-04-05T21:55:18.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.422+0000 7f28dffff700 1 --2- 172.21.15.121:0/594567470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e0068670 0x7f28e00ffca0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f28d4006010 tx=0x7f28d400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:18.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.422+0000 7f28dd7fa700 1 -- 172.21.15.121:0/594567470 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28d401d070 con 0x7f28e0068670 2024-04-05T21:55:18.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.423+0000 7f28e62be700 1 -- 172.21.15.121:0/594567470 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e0100ae0 con 0x7f28e0068670 2024-04-05T21:55:18.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.423+0000 7f28dd7fa700 1 -- 172.21.15.121:0/594567470 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28d400f460 con 0x7f28e0068670 2024-04-05T21:55:18.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.423+0000 7f28e62be700 1 -- 172.21.15.121:0/594567470 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e01a2b20 con 0x7f28e0068670 2024-04-05T21:55:18.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.423+0000 7f28dd7fa700 1 -- 172.21.15.121:0/594567470 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28d4021620 con 0x7f28e0068670 2024-04-05T21:55:18.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.423+0000 7f28e62be700 1 -- 172.21.15.121:0/594567470 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e0109130 con 0x7f28e0068670 2024-04-05T21:55:18.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.424+0000 7f28dd7fa700 1 -- 172.21.15.121:0/594567470 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f28d400fa40 con 0x7f28e0068670 2024-04-05T21:55:18.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.425+0000 7f28dd7fa700 1 --2- 172.21.15.121:0/594567470 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28c806cce0 0x7f28c806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:18.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.425+0000 7f28df7fe700 1 --2- 172.21.15.121:0/594567470 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28c806cce0 0x7f28c806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:18.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.425+0000 7f28dd7fa700 1 -- 172.21.15.121:0/594567470 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f28d408dbe0 con 0x7f28e0068670 2024-04-05T21:55:18.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.425+0000 7f28df7fe700 1 --2- 172.21.15.121:0/594567470 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28c806cce0 0x7f28c806f1a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f28d0005950 tx=0x7f28d00058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:18.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.429+0000 7f28dd7fa700 1 -- 172.21.15.121:0/594567470 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f28d4057d50 con 0x7f28e0068670 2024-04-05T21:55:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.700+0000 7f28e62be700 1 -- 172.21.15.121:0/594567470 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f28e004ea90 con 0x7f28e0068670 2024-04-05T21:55:18.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.700+0000 7f28dd7fa700 1 -- 172.21.15.121:0/594567470 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f28d4026e10 con 0x7f28e0068670 2024-04-05T21:55:18.701 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:55:18.701 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:55:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:55:18.701 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:55:18.701 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:55:18.701 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:55:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:55:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:55:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:55:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:55:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:55:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:55:18.701 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:55:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:55:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:55:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:55:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:55:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:55:18.702 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:55:18.702 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:55:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:55:18.702 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:55:18.702 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:55:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.702+0000 7f28e62be700 1 -- 172.21.15.121:0/594567470 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28c806cce0 msgr2=0x7f28c806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.702+0000 7f28e62be700 1 --2- 172.21.15.121:0/594567470 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28c806cce0 0x7f28c806f1a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f28d0005950 tx=0x7f28d00058e0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.702+0000 7f28e62be700 1 -- 172.21.15.121:0/594567470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e0068670 msgr2=0x7f28e00ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.702+0000 7f28e62be700 1 --2- 172.21.15.121:0/594567470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e0068670 0x7f28e00ffca0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f28d4006010 tx=0x7f28d400be20 comp rx=0 tx=0).stop 2024-04-05T21:55:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.702+0000 7f28e62be700 1 -- 172.21.15.121:0/594567470 shutdown_connections 2024-04-05T21:55:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.702+0000 7f28e62be700 1 --2- 172.21.15.121:0/594567470 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28c806cce0 0x7f28c806f1a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.702+0000 7f28e62be700 1 --2- 172.21.15.121:0/594567470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e0068670 0x7f28e00ffca0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.702+0000 7f28e62be700 1 --2- 172.21.15.121:0/594567470 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28e0069020 0x7f28e01001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.702+0000 7f28e62be700 1 -- 172.21.15.121:0/594567470 >> 172.21.15.121:0/594567470 conn(0x7f28e00755a0 msgr2=0x7f28e00fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:18.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.702+0000 7f28e62be700 1 -- 172.21.15.121:0/594567470 shutdown_connections 2024-04-05T21:55:18.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:18.702+0000 7f28e62be700 1 -- 172.21.15.121:0/594567470 wait complete. 2024-04-05T21:55:19.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:18 smithi121 ceph-mon[145859]: from='client.35418 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:19.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:18 smithi121 ceph-mon[145859]: pgmap v873: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:19.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:18 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2984670455' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:55:19.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:18 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/594567470' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:55:19.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:18 smithi188 ceph-mon[107793]: from='client.35418 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:19.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:18 smithi188 ceph-mon[107793]: pgmap v873: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:19.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:18 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2984670455' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:55:19.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:18 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/594567470' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:55:20.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:19 smithi121 ceph-mon[145859]: from='client.35430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:20.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:19 smithi188 ceph-mon[107793]: from='client.35430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:21.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:20 smithi121 ceph-mon[145859]: pgmap v874: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:20 smithi188 ceph-mon[107793]: pgmap v874: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:23.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:22 smithi121 ceph-mon[145859]: pgmap v875: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:23.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:22 smithi188 ceph-mon[107793]: pgmap v875: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:25.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:24 smithi121 ceph-mon[145859]: pgmap v876: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:24 smithi188 ceph-mon[107793]: pgmap v876: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:27.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:26 smithi121 ceph-mon[145859]: pgmap v877: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:27.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:26 smithi188 ceph-mon[107793]: pgmap v877: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:29.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:28 smithi121 ceph-mon[145859]: pgmap v878: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:28 smithi188 ceph-mon[107793]: pgmap v878: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:30 smithi121 ceph-mon[145859]: pgmap v879: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:55:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:30 smithi188 ceph-mon[107793]: pgmap v879: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:55:33.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:32 smithi121 ceph-mon[145859]: pgmap v880: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:33.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:32 smithi188 ceph-mon[107793]: pgmap v880: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:35.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:34 smithi121 ceph-mon[145859]: pgmap v881: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:35.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:34 smithi188 ceph-mon[107793]: pgmap v881: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:37.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:36 smithi121 ceph-mon[145859]: pgmap v882: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:36 smithi188 ceph-mon[107793]: pgmap v882: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:39.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:38 smithi121 ceph-mon[145859]: pgmap v883: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:38 smithi188 ceph-mon[107793]: pgmap v883: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:40 smithi121 ceph-mon[145859]: pgmap v884: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:40 smithi188 ceph-mon[107793]: pgmap v884: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:42.989 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:42 smithi121 ceph-mon[145859]: pgmap v885: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:42 smithi188 ceph-mon[107793]: pgmap v885: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:45.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:44 smithi121 ceph-mon[145859]: pgmap v886: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:44 smithi188 ceph-mon[107793]: pgmap v886: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:46.182 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:55:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:55:47.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:46 smithi121 ceph-mon[145859]: pgmap v887: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:46 smithi188 ceph-mon[107793]: pgmap v887: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:48.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.817+0000 7fe17463d700 1 -- 172.21.15.121:0/4246024240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c101750 msgr2=0x7fe16c1057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:48.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.817+0000 7fe17463d700 1 --2- 172.21.15.121:0/4246024240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c101750 0x7fe16c1057a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe168009910 tx=0x7fe168009c20 comp rx=0 tx=0).stop 2024-04-05T21:55:48.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.817+0000 7fe17463d700 1 -- 172.21.15.121:0/4246024240 shutdown_connections 2024-04-05T21:55:48.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.817+0000 7fe17463d700 1 --2- 172.21.15.121:0/4246024240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c101750 0x7fe16c1057a0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:48.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.817+0000 7fe17463d700 1 --2- 172.21.15.121:0/4246024240 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe16c100da0 0x7fe16c101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:48.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.817+0000 7fe17463d700 1 -- 172.21.15.121:0/4246024240 >> 172.21.15.121:0/4246024240 conn(0x7fe16c0fc630 msgr2=0x7fe16c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:48.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.817+0000 7fe17463d700 1 -- 172.21.15.121:0/4246024240 shutdown_connections 2024-04-05T21:55:48.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.817+0000 7fe17463d700 1 -- 172.21.15.121:0/4246024240 wait complete. 2024-04-05T21:55:48.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.818+0000 7fe17463d700 1 Processor -- start 2024-04-05T21:55:48.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.818+0000 7fe17463d700 1 -- start start 2024-04-05T21:55:48.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.818+0000 7fe17463d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c100da0 0x7fe16c196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:48.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.818+0000 7fe17463d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe16c101750 0x7fe16c196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:48.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.818+0000 7fe17463d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c1971e0 con 0x7fe16c100da0 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.818+0000 7fe17463d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c197320 con 0x7fe16c101750 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.818+0000 7fe1723d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c100da0 0x7fe16c196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.818+0000 7fe1723d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c100da0 0x7fe16c196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37632/0 (socket says 172.21.15.121:37632) 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.818+0000 7fe1723d9700 1 -- 172.21.15.121:0/3535413334 learned_addr learned my addr 172.21.15.121:0/3535413334 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.818+0000 7fe171bd8700 1 --2- 172.21.15.121:0/3535413334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe16c101750 0x7fe16c196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.819+0000 7fe1723d9700 1 -- 172.21.15.121:0/3535413334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe16c101750 msgr2=0x7fe16c196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.819+0000 7fe1723d9700 1 --2- 172.21.15.121:0/3535413334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe16c101750 0x7fe16c196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.819+0000 7fe1723d9700 1 -- 172.21.15.121:0/3535413334 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1680095f0 con 0x7fe16c100da0 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.819+0000 7fe171bd8700 1 --2- 172.21.15.121:0/3535413334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe16c101750 0x7fe16c196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.819+0000 7fe1723d9700 1 --2- 172.21.15.121:0/3535413334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c100da0 0x7fe16c196650 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00c930 tx=0x7fe15c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.819+0000 7fe1637fe700 1 -- 172.21.15.121:0/3535413334 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe15c007ad0 con 0x7fe16c100da0 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.819+0000 7fe1637fe700 1 -- 172.21.15.121:0/3535413334 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe15c007c30 con 0x7fe16c100da0 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.819+0000 7fe1637fe700 1 -- 172.21.15.121:0/3535413334 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe15c018630 con 0x7fe16c100da0 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.819+0000 7fe17463d700 1 -- 172.21.15.121:0/3535413334 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c19b170 con 0x7fe16c100da0 2024-04-05T21:55:48.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.819+0000 7fe17463d700 1 -- 172.21.15.121:0/3535413334 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c19b6c0 con 0x7fe16c100da0 2024-04-05T21:55:48.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.820+0000 7fe17463d700 1 -- 172.21.15.121:0/3535413334 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe16c1091a0 con 0x7fe16c100da0 2024-04-05T21:55:48.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.820+0000 7fe1637fe700 1 -- 172.21.15.121:0/3535413334 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe15c00f450 con 0x7fe16c100da0 2024-04-05T21:55:48.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.821+0000 7fe1637fe700 1 --2- 172.21.15.121:0/3535413334 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe15806cce0 0x7fe15806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:48.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.821+0000 7fe1637fe700 1 -- 172.21.15.121:0/3535413334 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe15c08c6a0 con 0x7fe16c100da0 2024-04-05T21:55:48.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.821+0000 7fe171bd8700 1 --2- 172.21.15.121:0/3535413334 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe15806cce0 0x7fe15806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:48.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.821+0000 7fe171bd8700 1 --2- 172.21.15.121:0/3535413334 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe15806cce0 0x7fe15806f1a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe16c197c70 tx=0x7fe16801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:48.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:48.825+0000 7fe1637fe700 1 -- 172.21.15.121:0/3535413334 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe15c056790 con 0x7fe16c100da0 2024-04-05T21:55:49.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.038+0000 7fe17463d700 1 -- 172.21.15.121:0/3535413334 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe16c02cce0 con 0x7fe15806cce0 2024-04-05T21:55:49.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.040+0000 7fe1637fe700 1 -- 172.21.15.121:0/3535413334 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe16c02cce0 con 0x7fe15806cce0 2024-04-05T21:55:49.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.042+0000 7fe17463d700 1 -- 172.21.15.121:0/3535413334 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe15806cce0 msgr2=0x7fe15806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:49.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.042+0000 7fe17463d700 1 --2- 172.21.15.121:0/3535413334 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe15806cce0 0x7fe15806f1a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe16c197c70 tx=0x7fe16801a040 comp rx=0 tx=0).stop 2024-04-05T21:55:49.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.042+0000 7fe17463d700 1 -- 172.21.15.121:0/3535413334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c100da0 msgr2=0x7fe16c196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:49.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.042+0000 7fe17463d700 1 --2- 172.21.15.121:0/3535413334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c100da0 0x7fe16c196650 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fe15c00c930 tx=0x7fe15c00ccf0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.042+0000 7fe17463d700 1 -- 172.21.15.121:0/3535413334 shutdown_connections 2024-04-05T21:55:49.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.042+0000 7fe17463d700 1 --2- 172.21.15.121:0/3535413334 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe15806cce0 0x7fe15806f1a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.042+0000 7fe17463d700 1 --2- 172.21.15.121:0/3535413334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c100da0 0x7fe16c196650 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.042+0000 7fe17463d700 1 --2- 172.21.15.121:0/3535413334 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe16c101750 0x7fe16c196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.042+0000 7fe17463d700 1 -- 172.21.15.121:0/3535413334 >> 172.21.15.121:0/3535413334 conn(0x7fe16c0fc630 msgr2=0x7fe16c0fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.042+0000 7fe17463d700 1 -- 172.21.15.121:0/3535413334 shutdown_connections 2024-04-05T21:55:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.042+0000 7fe17463d700 1 -- 172.21.15.121:0/3535413334 wait complete. 2024-04-05T21:55:49.052 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:55:49.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.144+0000 7fda25223700 1 -- 172.21.15.121:0/3151538756 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20068670 msgr2=0x7fda20068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:49.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.144+0000 7fda25223700 1 --2- 172.21.15.121:0/3151538756 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20068670 0x7fda20068a50 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fda10009910 tx=0x7fda10009c20 comp rx=0 tx=0).stop 2024-04-05T21:55:49.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.144+0000 7fda25223700 1 -- 172.21.15.121:0/3151538756 shutdown_connections 2024-04-05T21:55:49.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.144+0000 7fda25223700 1 --2- 172.21.15.121:0/3151538756 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda20069020 0x7fda20105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.144+0000 7fda25223700 1 --2- 172.21.15.121:0/3151538756 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20068670 0x7fda20068a50 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.144+0000 7fda25223700 1 -- 172.21.15.121:0/3151538756 >> 172.21.15.121:0/3151538756 conn(0x7fda200755a0 msgr2=0x7fda200759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:49.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.144+0000 7fda25223700 1 -- 172.21.15.121:0/3151538756 shutdown_connections 2024-04-05T21:55:49.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.144+0000 7fda25223700 1 -- 172.21.15.121:0/3151538756 wait complete. 2024-04-05T21:55:49.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.145+0000 7fda25223700 1 Processor -- start 2024-04-05T21:55:49.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.145+0000 7fda25223700 1 -- start start 2024-04-05T21:55:49.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.145+0000 7fda25223700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda20068670 0x7fda200ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:49.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.145+0000 7fda25223700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20069020 0x7fda201001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:49.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.145+0000 7fda25223700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda20100720 con 0x7fda20069020 2024-04-05T21:55:49.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.145+0000 7fda25223700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda20100860 con 0x7fda20068670 2024-04-05T21:55:49.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda1e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20069020 0x7fda201001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:49.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda1e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20069020 0x7fda201001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37638/0 (socket says 172.21.15.121:37638) 2024-04-05T21:55:49.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda1e59c700 1 -- 172.21.15.121:0/1869810233 learned_addr learned my addr 172.21.15.121:0/1869810233 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:49.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda1ed9d700 1 --2- 172.21.15.121:0/1869810233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda20068670 0x7fda200ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:49.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda1e59c700 1 -- 172.21.15.121:0/1869810233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda20068670 msgr2=0x7fda200ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:49.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda1e59c700 1 --2- 172.21.15.121:0/1869810233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda20068670 0x7fda200ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda1e59c700 1 -- 172.21.15.121:0/1869810233 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda100095f0 con 0x7fda20069020 2024-04-05T21:55:49.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda1ed9d700 1 --2- 172.21.15.121:0/1869810233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda20068670 0x7fda200ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:55:49.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda1e59c700 1 --2- 172.21.15.121:0/1869810233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20069020 0x7fda201001e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fda1400d900 tx=0x7fda1400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:49.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda07fff700 1 -- 172.21.15.121:0/1869810233 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda14009900 con 0x7fda20069020 2024-04-05T21:55:49.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda07fff700 1 -- 172.21.15.121:0/1869810233 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda14010460 con 0x7fda20069020 2024-04-05T21:55:49.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda25223700 1 -- 172.21.15.121:0/1869810233 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda201a2840 con 0x7fda20069020 2024-04-05T21:55:49.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda07fff700 1 -- 172.21.15.121:0/1869810233 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda1400f5f0 con 0x7fda20069020 2024-04-05T21:55:49.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.146+0000 7fda25223700 1 -- 172.21.15.121:0/1869810233 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda201a2c50 con 0x7fda20069020 2024-04-05T21:55:49.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.147+0000 7fda25223700 1 -- 172.21.15.121:0/1869810233 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda20109130 con 0x7fda20069020 2024-04-05T21:55:49.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.148+0000 7fda07fff700 1 -- 172.21.15.121:0/1869810233 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fda1400fac0 con 0x7fda20069020 2024-04-05T21:55:49.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.148+0000 7fda07fff700 1 --2- 172.21.15.121:0/1869810233 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fda0806cce0 0x7fda0806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:49.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.148+0000 7fda1ed9d700 1 --2- 172.21.15.121:0/1869810233 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fda0806cce0 0x7fda0806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:49.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.148+0000 7fda07fff700 1 -- 172.21.15.121:0/1869810233 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fda1408c420 con 0x7fda20069020 2024-04-05T21:55:49.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.148+0000 7fda1ed9d700 1 --2- 172.21.15.121:0/1869810233 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fda0806cce0 0x7fda0806f1a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fda1000b5c0 tx=0x7fda1001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:49.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.153+0000 7fda07fff700 1 -- 172.21.15.121:0/1869810233 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fda14056490 con 0x7fda20069020 2024-04-05T21:55:49.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:48 smithi121 ceph-mon[145859]: pgmap v888: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:49.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:48 smithi188 ceph-mon[107793]: pgmap v888: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.368+0000 7fda25223700 1 -- 172.21.15.121:0/1869810233 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fda2002cda0 con 0x7fda0806cce0 2024-04-05T21:55:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.369+0000 7fda07fff700 1 -- 172.21.15.121:0/1869810233 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fda2002cda0 con 0x7fda0806cce0 2024-04-05T21:55:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.371+0000 7fda25223700 1 -- 172.21.15.121:0/1869810233 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fda0806cce0 msgr2=0x7fda0806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.371+0000 7fda25223700 1 --2- 172.21.15.121:0/1869810233 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fda0806cce0 0x7fda0806f1a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fda1000b5c0 tx=0x7fda1001a040 comp rx=0 tx=0).stop 2024-04-05T21:55:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.371+0000 7fda25223700 1 -- 172.21.15.121:0/1869810233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20069020 msgr2=0x7fda201001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.371+0000 7fda25223700 1 --2- 172.21.15.121:0/1869810233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20069020 0x7fda201001e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fda1400d900 tx=0x7fda1400dcc0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.371+0000 7fda25223700 1 -- 172.21.15.121:0/1869810233 shutdown_connections 2024-04-05T21:55:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.371+0000 7fda25223700 1 --2- 172.21.15.121:0/1869810233 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fda0806cce0 0x7fda0806f1a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.371+0000 7fda25223700 1 --2- 172.21.15.121:0/1869810233 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fda20068670 0x7fda200ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.371+0000 7fda25223700 1 --2- 172.21.15.121:0/1869810233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20069020 0x7fda201001e0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.371+0000 7fda25223700 1 -- 172.21.15.121:0/1869810233 >> 172.21.15.121:0/1869810233 conn(0x7fda200755a0 msgr2=0x7fda200fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.372+0000 7fda25223700 1 -- 172.21.15.121:0/1869810233 shutdown_connections 2024-04-05T21:55:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.372+0000 7fda25223700 1 -- 172.21.15.121:0/1869810233 wait complete. 2024-04-05T21:55:49.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.474+0000 7fa2ea903700 1 -- 172.21.15.121:0/2193784191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e40ffbd0 msgr2=0x7fa2e4100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:49.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.474+0000 7fa2ea903700 1 --2- 172.21.15.121:0/2193784191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e40ffbd0 0x7fa2e4100050 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4009910 tx=0x7fa2d4009c20 comp rx=0 tx=0).stop 2024-04-05T21:55:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.474+0000 7fa2ea903700 1 -- 172.21.15.121:0/2193784191 shutdown_connections 2024-04-05T21:55:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.474+0000 7fa2ea903700 1 --2- 172.21.15.121:0/2193784191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e40ffbd0 0x7fa2e4100050 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.474+0000 7fa2ea903700 1 --2- 172.21.15.121:0/2193784191 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2e4105a40 0x7fa2e4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.474+0000 7fa2ea903700 1 -- 172.21.15.121:0/2193784191 >> 172.21.15.121:0/2193784191 conn(0x7fa2e40755a0 msgr2=0x7fa2e40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.474+0000 7fa2ea903700 1 -- 172.21.15.121:0/2193784191 shutdown_connections 2024-04-05T21:55:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.474+0000 7fa2ea903700 1 -- 172.21.15.121:0/2193784191 wait complete. 2024-04-05T21:55:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.474+0000 7fa2ea903700 1 Processor -- start 2024-04-05T21:55:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.474+0000 7fa2ea903700 1 -- start start 2024-04-05T21:55:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.475+0000 7fa2ea903700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e40ffbd0 0x7fa2e41952a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.475+0000 7fa2ea903700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2e4105a40 0x7fa2e41957e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.475+0000 7fa2ea903700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e4195d20 con 0x7fa2e40ffbd0 2024-04-05T21:55:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.475+0000 7fa2ea903700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e4195e90 con 0x7fa2e4105a40 2024-04-05T21:55:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.475+0000 7fa2e3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e40ffbd0 0x7fa2e41952a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.475+0000 7fa2e3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e40ffbd0 0x7fa2e41952a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37662/0 (socket says 172.21.15.121:37662) 2024-04-05T21:55:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.475+0000 7fa2e3fff700 1 -- 172.21.15.121:0/1907933575 learned_addr learned my addr 172.21.15.121:0/1907933575 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.475+0000 7fa2e37fe700 1 --2- 172.21.15.121:0/1907933575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2e4105a40 0x7fa2e41957e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.475+0000 7fa2e3fff700 1 -- 172.21.15.121:0/1907933575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2e4105a40 msgr2=0x7fa2e41957e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.475+0000 7fa2e3fff700 1 --2- 172.21.15.121:0/1907933575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2e4105a40 0x7fa2e41957e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.475+0000 7fa2e3fff700 1 -- 172.21.15.121:0/1907933575 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d40095f0 con 0x7fa2e40ffbd0 2024-04-05T21:55:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.475+0000 7fa2e3fff700 1 --2- 172.21.15.121:0/1907933575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e40ffbd0 0x7fa2e41952a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800d8d0 tx=0x7fa2d800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.475+0000 7fa2e17fa700 1 -- 172.21.15.121:0/1907933575 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2d80098a0 con 0x7fa2e40ffbd0 2024-04-05T21:55:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.476+0000 7fa2e17fa700 1 -- 172.21.15.121:0/1907933575 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2d8010460 con 0x7fa2e40ffbd0 2024-04-05T21:55:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.476+0000 7fa2ea903700 1 -- 172.21.15.121:0/1907933575 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e41a2910 con 0x7fa2e40ffbd0 2024-04-05T21:55:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.476+0000 7fa2e17fa700 1 -- 172.21.15.121:0/1907933575 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2d8009ea0 con 0x7fa2e40ffbd0 2024-04-05T21:55:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.476+0000 7fa2ea903700 1 -- 172.21.15.121:0/1907933575 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e41a2c70 con 0x7fa2e40ffbd0 2024-04-05T21:55:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.477+0000 7fa2ea903700 1 -- 172.21.15.121:0/1907933575 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e4107fd0 con 0x7fa2e40ffbd0 2024-04-05T21:55:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.477+0000 7fa2e17fa700 1 -- 172.21.15.121:0/1907933575 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa2d80105d0 con 0x7fa2e40ffbd0 2024-04-05T21:55:49.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.477+0000 7fa2e17fa700 1 --2- 172.21.15.121:0/1907933575 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa2cc06cce0 0x7fa2cc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:49.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.478+0000 7fa2e17fa700 1 -- 172.21.15.121:0/1907933575 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa2d808c410 con 0x7fa2e40ffbd0 2024-04-05T21:55:49.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.478+0000 7fa2e37fe700 1 --2- 172.21.15.121:0/1907933575 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa2cc06cce0 0x7fa2cc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:49.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.478+0000 7fa2e37fe700 1 --2- 172.21.15.121:0/1907933575 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa2cc06cce0 0x7fa2cc06f1a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400b5c0 tx=0x7fa2d401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:49.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.482+0000 7fa2e17fa700 1 -- 172.21.15.121:0/1907933575 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa2d8059f10 con 0x7fa2e40ffbd0 2024-04-05T21:55:49.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.685+0000 7fa2ea903700 1 -- 172.21.15.121:0/1907933575 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2e402cce0 con 0x7fa2cc06cce0 2024-04-05T21:55:49.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.695+0000 7fa2e17fa700 1 -- 172.21.15.121:0/1907933575 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fa2e402cce0 con 0x7fa2cc06cce0 2024-04-05T21:55:49.696 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:55:49.696 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (26m) 5m ago 38m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:55:49.696 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (38m) 5m ago 38m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:55:49.696 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (37m) 5m ago 37m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:55:49.696 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (24m) 5m ago 38m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:55:49.696 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (24m) 5m ago 37m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:55:49.696 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (26m) 5m ago 37m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:55:49.696 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (21m) 5m ago 32m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:55:49.696 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (20m) 5m ago 31m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:55:49.696 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (20m) 5m ago 31m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:55:49.696 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (5m) 5m ago 31m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:55:49.696 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (28m) 5m ago 39m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:55:49.697 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (27m) 5m ago 37m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:55:49.697 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (25m) 5m ago 39m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:55:49.697 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (25m) 5m ago 36m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:55:49.697 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (27m) 5m ago 38m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:55:49.697 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (27m) 5m ago 37m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:55:49.697 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (24m) 5m ago 36m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:55:49.697 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (23m) 5m ago 35m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:55:49.697 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (23m) 5m ago 35m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:55:49.697 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (22m) 5m ago 34m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:55:49.697 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (21m) 5m ago 34m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:55:49.697 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (21m) 5m ago 33m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:55:49.697 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (26m) 5m ago 37m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:55:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.697+0000 7fa2ea903700 1 -- 172.21.15.121:0/1907933575 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa2cc06cce0 msgr2=0x7fa2cc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.697+0000 7fa2ea903700 1 --2- 172.21.15.121:0/1907933575 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa2cc06cce0 0x7fa2cc06f1a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400b5c0 tx=0x7fa2d401a040 comp rx=0 tx=0).stop 2024-04-05T21:55:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.697+0000 7fa2ea903700 1 -- 172.21.15.121:0/1907933575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e40ffbd0 msgr2=0x7fa2e41952a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.697+0000 7fa2ea903700 1 --2- 172.21.15.121:0/1907933575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e40ffbd0 0x7fa2e41952a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800d8d0 tx=0x7fa2d800dbe0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.697+0000 7fa2ea903700 1 -- 172.21.15.121:0/1907933575 shutdown_connections 2024-04-05T21:55:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.697+0000 7fa2ea903700 1 --2- 172.21.15.121:0/1907933575 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa2cc06cce0 0x7fa2cc06f1a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.697+0000 7fa2ea903700 1 --2- 172.21.15.121:0/1907933575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e40ffbd0 0x7fa2e41952a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.697+0000 7fa2ea903700 1 --2- 172.21.15.121:0/1907933575 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa2e4105a40 0x7fa2e41957e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.697+0000 7fa2ea903700 1 -- 172.21.15.121:0/1907933575 >> 172.21.15.121:0/1907933575 conn(0x7fa2e40755a0 msgr2=0x7fa2e40fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.697+0000 7fa2ea903700 1 -- 172.21.15.121:0/1907933575 shutdown_connections 2024-04-05T21:55:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.697+0000 7fa2ea903700 1 -- 172.21.15.121:0/1907933575 wait complete. 2024-04-05T21:55:49.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.798+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3624052638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e0105a40 msgr2=0x7fb0e0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:49.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.798+0000 7fb0e6b3d700 1 --2- 172.21.15.121:0/3624052638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e0105a40 0x7fb0e0105e20 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009910 tx=0x7fb0d4009c20 comp rx=0 tx=0).stop 2024-04-05T21:55:49.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.798+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3624052638 shutdown_connections 2024-04-05T21:55:49.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.798+0000 7fb0e6b3d700 1 --2- 172.21.15.121:0/3624052638 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e00ffa40 0x7fb0e00ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.798+0000 7fb0e6b3d700 1 --2- 172.21.15.121:0/3624052638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e0105a40 0x7fb0e0105e20 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.798+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3624052638 >> 172.21.15.121:0/3624052638 conn(0x7fb0e00fb530 msgr2=0x7fb0e00fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:49.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.798+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3624052638 shutdown_connections 2024-04-05T21:55:49.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.798+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3624052638 wait complete. 2024-04-05T21:55:49.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.799+0000 7fb0e6b3d700 1 Processor -- start 2024-04-05T21:55:49.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.799+0000 7fb0e6b3d700 1 -- start start 2024-04-05T21:55:49.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0e6b3d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e00ffa40 0x7fb0e01974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:49.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0e6b3d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e0105a40 0x7fb0e0197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:49.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0e6b3d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e0198050 con 0x7fb0e0105a40 2024-04-05T21:55:49.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0e6b3d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e0198190 con 0x7fb0e00ffa40 2024-04-05T21:55:49.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0dffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e0105a40 0x7fb0e0197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:49.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0dffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e0105a40 0x7fb0e0197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37676/0 (socket says 172.21.15.121:37676) 2024-04-05T21:55:49.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0dffff700 1 -- 172.21.15.121:0/3980726639 learned_addr learned my addr 172.21.15.121:0/3980726639 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:49.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0e48d9700 1 --2- 172.21.15.121:0/3980726639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e00ffa40 0x7fb0e01974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:49.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0dffff700 1 -- 172.21.15.121:0/3980726639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e00ffa40 msgr2=0x7fb0e01974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:49.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0dffff700 1 --2- 172.21.15.121:0/3980726639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e00ffa40 0x7fb0e01974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:49.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0dffff700 1 -- 172.21.15.121:0/3980726639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d40095f0 con 0x7fb0e0105a40 2024-04-05T21:55:49.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0e48d9700 1 --2- 172.21.15.121:0/3980726639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e00ffa40 0x7fb0e01974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:55:49.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0dffff700 1 --2- 172.21.15.121:0/3980726639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e0105a40 0x7fb0e0197a30 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00d900 tx=0x7fb0cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:49.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0ddffb700 1 -- 172.21.15.121:0/3980726639 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0cc0041f0 con 0x7fb0e0105a40 2024-04-05T21:55:49.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0ddffb700 1 -- 172.21.15.121:0/3980726639 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0cc004cd0 con 0x7fb0e0105a40 2024-04-05T21:55:49.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3980726639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e01a0560 con 0x7fb0e0105a40 2024-04-05T21:55:49.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0ddffb700 1 -- 172.21.15.121:0/3980726639 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0cc0056d0 con 0x7fb0e0105a40 2024-04-05T21:55:49.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.800+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3980726639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e01a0a20 con 0x7fb0e0105a40 2024-04-05T21:55:49.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.801+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3980726639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e0108040 con 0x7fb0e0105a40 2024-04-05T21:55:49.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.802+0000 7fb0ddffb700 1 -- 172.21.15.121:0/3980726639 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb0cc004e40 con 0x7fb0e0105a40 2024-04-05T21:55:49.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.802+0000 7fb0ddffb700 1 --2- 172.21.15.121:0/3980726639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb0d006cc90 0x7fb0d006f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:49.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.802+0000 7fb0ddffb700 1 -- 172.21.15.121:0/3980726639 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb0cc08c710 con 0x7fb0e0105a40 2024-04-05T21:55:49.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.802+0000 7fb0e48d9700 1 --2- 172.21.15.121:0/3980726639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb0d006cc90 0x7fb0d006f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:49.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.803+0000 7fb0e48d9700 1 --2- 172.21.15.121:0/3980726639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb0d006cc90 0x7fb0d006f150 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb0d400b5c0 tx=0x7fb0d4005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:49.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:49.807+0000 7fb0ddffb700 1 -- 172.21.15.121:0/3980726639 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb0cc056780 con 0x7fb0e0105a40 2024-04-05T21:55:50.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.082+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3980726639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb0e004ea90 con 0x7fb0e0105a40 2024-04-05T21:55:50.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.083+0000 7fb0ddffb700 1 -- 172.21.15.121:0/3980726639 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fb0cc059da0 con 0x7fb0e0105a40 2024-04-05T21:55:50.083 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:55:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:55:50.083 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:55:50.084 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:55:50.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.085+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3980726639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb0d006cc90 msgr2=0x7fb0d006f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:50.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.085+0000 7fb0e6b3d700 1 --2- 172.21.15.121:0/3980726639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb0d006cc90 0x7fb0d006f150 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb0d400b5c0 tx=0x7fb0d4005960 comp rx=0 tx=0).stop 2024-04-05T21:55:50.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.085+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3980726639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e0105a40 msgr2=0x7fb0e0197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:50.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.085+0000 7fb0e6b3d700 1 --2- 172.21.15.121:0/3980726639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e0105a40 0x7fb0e0197a30 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00d900 tx=0x7fb0cc00dcc0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.085+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3980726639 shutdown_connections 2024-04-05T21:55:50.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.085+0000 7fb0e6b3d700 1 --2- 172.21.15.121:0/3980726639 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb0d006cc90 0x7fb0d006f150 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.085+0000 7fb0e6b3d700 1 --2- 172.21.15.121:0/3980726639 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb0e00ffa40 0x7fb0e01974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.085+0000 7fb0e6b3d700 1 --2- 172.21.15.121:0/3980726639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e0105a40 0x7fb0e0197a30 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.085+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3980726639 >> 172.21.15.121:0/3980726639 conn(0x7fb0e00fb530 msgr2=0x7fb0e00fbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:50.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.085+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3980726639 shutdown_connections 2024-04-05T21:55:50.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.085+0000 7fb0e6b3d700 1 -- 172.21.15.121:0/3980726639 wait complete. 2024-04-05T21:55:50.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.188+0000 7f8c4c005700 1 -- 172.21.15.121:0/1263553250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c44105a40 msgr2=0x7f8c44105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:50.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.188+0000 7f8c4c005700 1 --2- 172.21.15.121:0/1263553250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c44105a40 0x7f8c44105e20 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00b0a0 tx=0x7f8c3c00b3b0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.188+0000 7f8c4c005700 1 -- 172.21.15.121:0/1263553250 shutdown_connections 2024-04-05T21:55:50.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.188+0000 7f8c4c005700 1 --2- 172.21.15.121:0/1263553250 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c440ffbd0 0x7f8c44100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.188+0000 7f8c4c005700 1 --2- 172.21.15.121:0/1263553250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c44105a40 0x7f8c44105e20 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.188+0000 7f8c4c005700 1 -- 172.21.15.121:0/1263553250 >> 172.21.15.121:0/1263553250 conn(0x7f8c440755a0 msgr2=0x7f8c440759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:50.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.188+0000 7f8c4c005700 1 -- 172.21.15.121:0/1263553250 shutdown_connections 2024-04-05T21:55:50.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.188+0000 7f8c4c005700 1 -- 172.21.15.121:0/1263553250 wait complete. 2024-04-05T21:55:50.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.188+0000 7f8c4c005700 1 Processor -- start 2024-04-05T21:55:50.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c4c005700 1 -- start start 2024-04-05T21:55:50.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c4c005700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c440ffbd0 0x7f8c44196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:50.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c4c005700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c44105a40 0x7f8c44196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:50.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c4c005700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c44197260 con 0x7f8c44105a40 2024-04-05T21:55:50.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c4c005700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4419aff0 con 0x7f8c440ffbd0 2024-04-05T21:55:50.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c495a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c44105a40 0x7f8c44196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:50.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c495a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c44105a40 0x7f8c44196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37704/0 (socket says 172.21.15.121:37704) 2024-04-05T21:55:50.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c495a0700 1 -- 172.21.15.121:0/1280538803 learned_addr learned my addr 172.21.15.121:0/1280538803 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:50.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c49da1700 1 --2- 172.21.15.121:0/1280538803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c440ffbd0 0x7f8c44196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:50.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c495a0700 1 -- 172.21.15.121:0/1280538803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c440ffbd0 msgr2=0x7f8c44196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:50.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c495a0700 1 --2- 172.21.15.121:0/1280538803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c440ffbd0 0x7f8c44196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c495a0700 1 -- 172.21.15.121:0/1280538803 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3c009d00 con 0x7f8c44105a40 2024-04-05T21:55:50.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c495a0700 1 --2- 172.21.15.121:0/1280538803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c44105a40 0x7f8c44196b80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800c8a0 tx=0x7f8c3800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:50.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c42ffd700 1 -- 172.21.15.121:0/1280538803 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c380078e0 con 0x7f8c44105a40 2024-04-05T21:55:50.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c42ffd700 1 -- 172.21.15.121:0/1280538803 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c3800f450 con 0x7f8c44105a40 2024-04-05T21:55:50.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c4c005700 1 -- 172.21.15.121:0/1280538803 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4419b2d0 con 0x7f8c44105a40 2024-04-05T21:55:50.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.189+0000 7f8c42ffd700 1 -- 172.21.15.121:0/1280538803 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3800e5e0 con 0x7f8c44105a40 2024-04-05T21:55:50.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.190+0000 7f8c4c005700 1 -- 172.21.15.121:0/1280538803 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4419b820 con 0x7f8c44105a40 2024-04-05T21:55:50.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.191+0000 7f8c4c005700 1 -- 172.21.15.121:0/1280538803 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c44109050 con 0x7f8c44105a40 2024-04-05T21:55:50.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.191+0000 7f8c42ffd700 1 -- 172.21.15.121:0/1280538803 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8c3800f5c0 con 0x7f8c44105a40 2024-04-05T21:55:50.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.191+0000 7f8c42ffd700 1 --2- 172.21.15.121:0/1280538803 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8c2c06cce0 0x7f8c2c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:50.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.192+0000 7f8c42ffd700 1 -- 172.21.15.121:0/1280538803 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8c3808c1d0 con 0x7f8c44105a40 2024-04-05T21:55:50.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.192+0000 7f8c49da1700 1 --2- 172.21.15.121:0/1280538803 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8c2c06cce0 0x7f8c2c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:50.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.192+0000 7f8c49da1700 1 --2- 172.21.15.121:0/1280538803 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8c2c06cce0 0x7f8c2c06f1a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c009fd0 tx=0x7f8c3c0060c0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:50.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.196+0000 7f8c42ffd700 1 -- 172.21.15.121:0/1280538803 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8c38056240 con 0x7f8c44105a40 2024-04-05T21:55:50.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.431+0000 7f8c4c005700 1 -- 172.21.15.121:0/1280538803 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8c4404ea90 con 0x7f8c44105a40 2024-04-05T21:55:50.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.431+0000 7f8c42ffd700 1 -- 172.21.15.121:0/1280538803 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f8c38059860 con 0x7f8c44105a40 2024-04-05T21:55:50.432 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:55:50.432 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:55:50.432 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:55:50.432 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:55:50.432 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:55:50.432 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:55:50.432 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:55:50.432 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:55:50.432 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:55:50.432 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:55:50.432 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:55:50.433 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:55:50.434 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:55:50.435 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:55:50.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.433+0000 7f8c4c005700 1 -- 172.21.15.121:0/1280538803 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8c2c06cce0 msgr2=0x7f8c2c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:50.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.433+0000 7f8c4c005700 1 --2- 172.21.15.121:0/1280538803 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8c2c06cce0 0x7f8c2c06f1a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c009fd0 tx=0x7f8c3c0060c0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.433+0000 7f8c4c005700 1 -- 172.21.15.121:0/1280538803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c44105a40 msgr2=0x7f8c44196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:50.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.433+0000 7f8c4c005700 1 --2- 172.21.15.121:0/1280538803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c44105a40 0x7f8c44196b80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800c8a0 tx=0x7f8c3800cbb0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.433+0000 7f8c4c005700 1 -- 172.21.15.121:0/1280538803 shutdown_connections 2024-04-05T21:55:50.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.433+0000 7f8c4c005700 1 --2- 172.21.15.121:0/1280538803 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8c2c06cce0 0x7f8c2c06f1a0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.433+0000 7f8c4c005700 1 --2- 172.21.15.121:0/1280538803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c440ffbd0 0x7f8c44196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.433+0000 7f8c4c005700 1 --2- 172.21.15.121:0/1280538803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c44105a40 0x7f8c44196b80 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.433+0000 7f8c4c005700 1 -- 172.21.15.121:0/1280538803 >> 172.21.15.121:0/1280538803 conn(0x7f8c440755a0 msgr2=0x7f8c44068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:50.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.433+0000 7f8c4c005700 1 -- 172.21.15.121:0/1280538803 shutdown_connections 2024-04-05T21:55:50.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.433+0000 7f8c4c005700 1 -- 172.21.15.121:0/1280538803 wait complete. 2024-04-05T21:55:50.436 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:55:50.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.536+0000 7f7068339700 1 -- 172.21.15.121:0/3723137514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7060100dc0 msgr2=0x7f70601011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:50.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.536+0000 7f7068339700 1 --2- 172.21.15.121:0/3723137514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7060100dc0 0x7f70601011a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f705c009910 tx=0x7f705c009c20 comp rx=0 tx=0).stop 2024-04-05T21:55:50.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.537+0000 7f7068339700 1 -- 172.21.15.121:0/3723137514 shutdown_connections 2024-04-05T21:55:50.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.537+0000 7f7068339700 1 --2- 172.21.15.121:0/3723137514 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7060101770 0x7f70601057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.537+0000 7f7068339700 1 --2- 172.21.15.121:0/3723137514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7060100dc0 0x7f70601011a0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.537+0000 7f7068339700 1 -- 172.21.15.121:0/3723137514 >> 172.21.15.121:0/3723137514 conn(0x7f70600fc670 msgr2=0x7f70600fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:50.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.537+0000 7f7068339700 1 -- 172.21.15.121:0/3723137514 shutdown_connections 2024-04-05T21:55:50.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.537+0000 7f7068339700 1 -- 172.21.15.121:0/3723137514 wait complete. 2024-04-05T21:55:50.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.537+0000 7f7068339700 1 Processor -- start 2024-04-05T21:55:50.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.537+0000 7f7068339700 1 -- start start 2024-04-05T21:55:50.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f7068339700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7060100dc0 0x7f7060196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f7068339700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7060101770 0x7f7060196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f7068339700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70601972e0 con 0x7f7060100dc0 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f7068339700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706019a010 con 0x7f7060101770 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f70660d5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7060100dc0 0x7f7060196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f70660d5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7060100dc0 0x7f7060196710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37722/0 (socket says 172.21.15.121:37722) 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f70660d5700 1 -- 172.21.15.121:0/2714227189 learned_addr learned my addr 172.21.15.121:0/2714227189 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f70658d4700 1 --2- 172.21.15.121:0/2714227189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7060101770 0x7f7060196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f70660d5700 1 -- 172.21.15.121:0/2714227189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7060101770 msgr2=0x7f7060196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f70660d5700 1 --2- 172.21.15.121:0/2714227189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7060101770 0x7f7060196c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f70660d5700 1 -- 172.21.15.121:0/2714227189 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c0095f0 con 0x7f7060100dc0 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f70660d5700 1 --2- 172.21.15.121:0/2714227189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7060100dc0 0x7f7060196710 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f705c006010 tx=0x7f705c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f70577fe700 1 -- 172.21.15.121:0/2714227189 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f705c01d070 con 0x7f7060100dc0 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f70577fe700 1 -- 172.21.15.121:0/2714227189 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f705c00f460 con 0x7f7060100dc0 2024-04-05T21:55:50.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f7068339700 1 -- 172.21.15.121:0/2714227189 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706019a290 con 0x7f7060100dc0 2024-04-05T21:55:50.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.538+0000 7f70577fe700 1 -- 172.21.15.121:0/2714227189 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f705c021620 con 0x7f7060100dc0 2024-04-05T21:55:50.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.539+0000 7f7068339700 1 -- 172.21.15.121:0/2714227189 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706019a780 con 0x7f7060100dc0 2024-04-05T21:55:50.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.540+0000 7f7068339700 1 -- 172.21.15.121:0/2714227189 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7060109160 con 0x7f7060100dc0 2024-04-05T21:55:50.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.540+0000 7f70577fe700 1 -- 172.21.15.121:0/2714227189 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f705c00fa40 con 0x7f7060100dc0 2024-04-05T21:55:50.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.540+0000 7f70577fe700 1 --2- 172.21.15.121:0/2714227189 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f704c06cce0 0x7f704c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:50.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.540+0000 7f70577fe700 1 -- 172.21.15.121:0/2714227189 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f705c08d790 con 0x7f7060100dc0 2024-04-05T21:55:50.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.540+0000 7f70658d4700 1 --2- 172.21.15.121:0/2714227189 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f704c06cce0 0x7f704c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:50.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.541+0000 7f70658d4700 1 --2- 172.21.15.121:0/2714227189 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f704c06cce0 0x7f704c06f1a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7060197d30 tx=0x7f70500094f0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:50.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.545+0000 7f70577fe700 1 -- 172.21.15.121:0/2714227189 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f705c057800 con 0x7f7060100dc0 2024-04-05T21:55:50.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.754+0000 7f7068339700 1 -- 172.21.15.121:0/2714227189 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f706019a9a0 con 0x7f704c06cce0 2024-04-05T21:55:50.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.756+0000 7f70577fe700 1 -- 172.21.15.121:0/2714227189 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f706019a9a0 con 0x7f704c06cce0 2024-04-05T21:55:50.756 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:55:50.756 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:55:50.756 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:55:50.756 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:55:50.756 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:55:50.757 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:55:50.757 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:55:50.757 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:55:50.757 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:55:50.757 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:55:50.757 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:55:50.757 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:55:50.757 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:55:50.758 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:55:50.758 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:55:50.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.758+0000 7f7068339700 1 -- 172.21.15.121:0/2714227189 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f704c06cce0 msgr2=0x7f704c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:50.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.758+0000 7f7068339700 1 --2- 172.21.15.121:0/2714227189 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f704c06cce0 0x7f704c06f1a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7060197d30 tx=0x7f70500094f0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.758+0000 7f7068339700 1 -- 172.21.15.121:0/2714227189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7060100dc0 msgr2=0x7f7060196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:50.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.758+0000 7f7068339700 1 --2- 172.21.15.121:0/2714227189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7060100dc0 0x7f7060196710 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f705c006010 tx=0x7f705c00be20 comp rx=0 tx=0).stop 2024-04-05T21:55:50.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.758+0000 7f7068339700 1 -- 172.21.15.121:0/2714227189 shutdown_connections 2024-04-05T21:55:50.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.758+0000 7f7068339700 1 --2- 172.21.15.121:0/2714227189 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f704c06cce0 0x7f704c06f1a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.758+0000 7f7068339700 1 --2- 172.21.15.121:0/2714227189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7060100dc0 0x7f7060196710 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.758+0000 7f7068339700 1 --2- 172.21.15.121:0/2714227189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7060101770 0x7f7060196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.758+0000 7f7068339700 1 -- 172.21.15.121:0/2714227189 >> 172.21.15.121:0/2714227189 conn(0x7f70600fc670 msgr2=0x7f70600fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:50.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.758+0000 7f7068339700 1 -- 172.21.15.121:0/2714227189 shutdown_connections 2024-04-05T21:55:50.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.758+0000 7f7068339700 1 -- 172.21.15.121:0/2714227189 wait complete. 2024-04-05T21:55:50.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.860+0000 7f9dfff9e700 1 -- 172.21.15.121:0/1768100936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9df8101750 msgr2=0x7f9df81057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:50.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.860+0000 7f9dfff9e700 1 --2- 172.21.15.121:0/1768100936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9df8101750 0x7f9df81057a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009910 tx=0x7f9df4009c20 comp rx=0 tx=0).stop 2024-04-05T21:55:50.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.860+0000 7f9dfff9e700 1 -- 172.21.15.121:0/1768100936 shutdown_connections 2024-04-05T21:55:50.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.860+0000 7f9dfff9e700 1 --2- 172.21.15.121:0/1768100936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9df8101750 0x7f9df81057a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.860+0000 7f9dfff9e700 1 --2- 172.21.15.121:0/1768100936 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9df8100da0 0x7f9df8101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.860+0000 7f9dfff9e700 1 -- 172.21.15.121:0/1768100936 >> 172.21.15.121:0/1768100936 conn(0x7f9df80fc630 msgr2=0x7f9df80fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:50.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.860+0000 7f9dfff9e700 1 -- 172.21.15.121:0/1768100936 shutdown_connections 2024-04-05T21:55:50.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.860+0000 7f9dfff9e700 1 -- 172.21.15.121:0/1768100936 wait complete. 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfff9e700 1 Processor -- start 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfff9e700 1 -- start start 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfff9e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9df8100da0 0x7f9df8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfff9e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9df8101750 0x7f9df80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfff9e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8073b10 con 0x7f9df8101750 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfff9e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8073c50 con 0x7f9df8100da0 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfd539700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9df8101750 0x7f9df80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfd539700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9df8101750 0x7f9df80735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37752/0 (socket says 172.21.15.121:37752) 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfd539700 1 -- 172.21.15.121:0/4098277516 learned_addr learned my addr 172.21.15.121:0/4098277516 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfdd3a700 1 --2- 172.21.15.121:0/4098277516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9df8100da0 0x7f9df8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfd539700 1 -- 172.21.15.121:0/4098277516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9df8100da0 msgr2=0x7f9df8073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfd539700 1 --2- 172.21.15.121:0/4098277516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9df8100da0 0x7f9df8073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfd539700 1 -- 172.21.15.121:0/4098277516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df40095f0 con 0x7f9df8101750 2024-04-05T21:55:50.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9dfd539700 1 --2- 172.21.15.121:0/4098277516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9df8101750 0x7f9df80735d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9df400bda0 tx=0x7f9df400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:50.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9deaffd700 1 -- 172.21.15.121:0/4098277516 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9df401d070 con 0x7f9df8101750 2024-04-05T21:55:50.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.861+0000 7f9deaffd700 1 -- 172.21.15.121:0/4098277516 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9df400f460 con 0x7f9df8101750 2024-04-05T21:55:50.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.862+0000 7f9dfff9e700 1 -- 172.21.15.121:0/4098277516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df8073ed0 con 0x7f9df8101750 2024-04-05T21:55:50.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.862+0000 7f9deaffd700 1 -- 172.21.15.121:0/4098277516 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9df4021640 con 0x7f9df8101750 2024-04-05T21:55:50.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.862+0000 7f9dfff9e700 1 -- 172.21.15.121:0/4098277516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df81a2b00 con 0x7f9df8101750 2024-04-05T21:55:50.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.862+0000 7f9dfff9e700 1 -- 172.21.15.121:0/4098277516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df81091a0 con 0x7f9df8101750 2024-04-05T21:55:50.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.863+0000 7f9deaffd700 1 -- 172.21.15.121:0/4098277516 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9df400f9b0 con 0x7f9df8101750 2024-04-05T21:55:50.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.863+0000 7f9deaffd700 1 --2- 172.21.15.121:0/4098277516 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9de406cce0 0x7f9de406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:55:50.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.863+0000 7f9deaffd700 1 -- 172.21.15.121:0/4098277516 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9df408d770 con 0x7f9df8101750 2024-04-05T21:55:50.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.863+0000 7f9dfdd3a700 1 --2- 172.21.15.121:0/4098277516 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9de406cce0 0x7f9de406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:55:50.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.864+0000 7f9dfdd3a700 1 --2- 172.21.15.121:0/4098277516 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9de406cce0 0x7f9de406f1a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9dec005ea0 tx=0x7f9dec005e30 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:55:50.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:50.868+0000 7f9deaffd700 1 -- 172.21.15.121:0/4098277516 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9df4057860 con 0x7f9df8101750 2024-04-05T21:55:51.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:51.138+0000 7f9dfff9e700 1 -- 172.21.15.121:0/4098277516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9df802cca0 con 0x7f9df8101750 2024-04-05T21:55:51.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:51.139+0000 7f9deaffd700 1 -- 172.21.15.121:0/4098277516 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f9df4026e10 con 0x7f9df8101750 2024-04-05T21:55:51.139 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:55:51.139 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:55:51.139 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:55:51.140 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:55:51.141 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:55:51.141 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:55:51.141 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:55:51.141 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:55:51.141 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:55:51.141 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:55:51.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:51.141+0000 7f9dfff9e700 1 -- 172.21.15.121:0/4098277516 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9de406cce0 msgr2=0x7f9de406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:51.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:51.141+0000 7f9dfff9e700 1 --2- 172.21.15.121:0/4098277516 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9de406cce0 0x7f9de406f1a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9dec005ea0 tx=0x7f9dec005e30 comp rx=0 tx=0).stop 2024-04-05T21:55:51.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:51.141+0000 7f9dfff9e700 1 -- 172.21.15.121:0/4098277516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9df8101750 msgr2=0x7f9df80735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:55:51.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:51.141+0000 7f9dfff9e700 1 --2- 172.21.15.121:0/4098277516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9df8101750 0x7f9df80735d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9df400bda0 tx=0x7f9df400be80 comp rx=0 tx=0).stop 2024-04-05T21:55:51.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:51.141+0000 7f9dfff9e700 1 -- 172.21.15.121:0/4098277516 shutdown_connections 2024-04-05T21:55:51.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:51.141+0000 7f9dfff9e700 1 --2- 172.21.15.121:0/4098277516 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9de406cce0 0x7f9de406f1a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:51.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:51.141+0000 7f9dfff9e700 1 --2- 172.21.15.121:0/4098277516 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9df8100da0 0x7f9df8073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:51.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:51.141+0000 7f9dfff9e700 1 --2- 172.21.15.121:0/4098277516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9df8101750 0x7f9df80735d0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:55:51.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:51.141+0000 7f9dfff9e700 1 -- 172.21.15.121:0/4098277516 >> 172.21.15.121:0/4098277516 conn(0x7f9df80fc630 msgr2=0x7f9df80fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:55:51.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:51.141+0000 7f9dfff9e700 1 -- 172.21.15.121:0/4098277516 shutdown_connections 2024-04-05T21:55:51.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:55:51.141+0000 7f9dfff9e700 1 -- 172.21.15.121:0/4098277516 wait complete. 2024-04-05T21:55:51.229 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:50 smithi188 ceph-mon[107793]: from='client.35438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:51.229 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:50 smithi188 ceph-mon[107793]: from='client.35442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:51.229 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:50 smithi188 ceph-mon[107793]: from='client.35446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:51.229 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:50 smithi188 ceph-mon[107793]: pgmap v889: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:51.229 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:50 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3980726639' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:55:51.229 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:50 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1280538803' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:55:51.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:50 smithi121 ceph-mon[145859]: from='client.35438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:51.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:50 smithi121 ceph-mon[145859]: from='client.35442 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:51.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:50 smithi121 ceph-mon[145859]: from='client.35446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:51.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:50 smithi121 ceph-mon[145859]: pgmap v889: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:51.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:50 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3980726639' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:55:51.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:50 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1280538803' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:55:52.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:51 smithi121 ceph-mon[145859]: from='client.35458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:52.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:51 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4098277516' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:55:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:51 smithi188 ceph-mon[107793]: from='client.35458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:55:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:51 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4098277516' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:55:53.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:52 smithi121 ceph-mon[145859]: pgmap v890: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:52 smithi188 ceph-mon[107793]: pgmap v890: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:55.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:54 smithi121 ceph-mon[145859]: pgmap v891: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:54 smithi188 ceph-mon[107793]: pgmap v891: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:56 smithi121 ceph-mon[145859]: pgmap v892: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:56 smithi188 ceph-mon[107793]: pgmap v892: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:59.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:55:58 smithi121 ceph-mon[145859]: pgmap v893: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:55:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:55:58 smithi188 ceph-mon[107793]: pgmap v893: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:00 smithi121 ceph-mon[145859]: pgmap v894: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:56:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:00 smithi188 ceph-mon[107793]: pgmap v894: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:56:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:02 smithi121 ceph-mon[145859]: pgmap v895: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:02 smithi188 ceph-mon[107793]: pgmap v895: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:05.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:04 smithi121 ceph-mon[145859]: pgmap v896: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:04 smithi188 ceph-mon[107793]: pgmap v896: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:07.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:06 smithi121 ceph-mon[145859]: pgmap v897: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:06 smithi188 ceph-mon[107793]: pgmap v897: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:09.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:08 smithi121 ceph-mon[145859]: pgmap v898: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:09.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:08 smithi188 ceph-mon[107793]: pgmap v898: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:11.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:10 smithi121 ceph-mon[145859]: pgmap v899: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:10 smithi188 ceph-mon[107793]: pgmap v899: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:12.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:12 smithi188 ceph-mon[107793]: pgmap v900: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:12.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:12 smithi121 ceph-mon[145859]: pgmap v900: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:15.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:14 smithi121 ceph-mon[145859]: pgmap v901: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:14 smithi188 ceph-mon[107793]: pgmap v901: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:56:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:56:17.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:16 smithi121 ceph-mon[145859]: pgmap v902: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:17.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:16 smithi188 ceph-mon[107793]: pgmap v902: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:18.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:18 smithi121 ceph-mon[145859]: pgmap v903: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:18.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:18 smithi188 ceph-mon[107793]: pgmap v903: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:21.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:20 smithi121 ceph-mon[145859]: pgmap v904: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.256+0000 7f6bdaea0700 1 -- 172.21.15.121:0/3397529780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd4101790 msgr2=0x7f6bd41057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.256+0000 7f6bdaea0700 1 --2- 172.21.15.121:0/3397529780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd4101790 0x7f6bd41057e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009910 tx=0x7f6bc0009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.256+0000 7f6bdaea0700 1 -- 172.21.15.121:0/3397529780 shutdown_connections 2024-04-05T21:56:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.256+0000 7f6bdaea0700 1 --2- 172.21.15.121:0/3397529780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd4101790 0x7f6bd41057e0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.256+0000 7f6bdaea0700 1 --2- 172.21.15.121:0/3397529780 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd4100de0 0x7f6bd41011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.256+0000 7f6bdaea0700 1 -- 172.21.15.121:0/3397529780 >> 172.21.15.121:0/3397529780 conn(0x7f6bd40fc6b0 msgr2=0x7f6bd40fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.256+0000 7f6bdaea0700 1 -- 172.21.15.121:0/3397529780 shutdown_connections 2024-04-05T21:56:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.256+0000 7f6bdaea0700 1 -- 172.21.15.121:0/3397529780 wait complete. 2024-04-05T21:56:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bdaea0700 1 Processor -- start 2024-04-05T21:56:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bdaea0700 1 -- start start 2024-04-05T21:56:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bdaea0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd4100de0 0x7f6bd419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bdaea0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd4101790 0x7f6bd419ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bdaea0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd419d050 con 0x7f6bd4100de0 2024-04-05T21:56:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bdaea0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd41965e0 con 0x7f6bd4101790 2024-04-05T21:56:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bd8c3c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd4100de0 0x7f6bd419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bd8c3c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd4100de0 0x7f6bd419c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41648/0 (socket says 172.21.15.121:41648) 2024-04-05T21:56:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bd8c3c700 1 -- 172.21.15.121:0/2183682697 learned_addr learned my addr 172.21.15.121:0/2183682697 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:21.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bd3fff700 1 --2- 172.21.15.121:0/2183682697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd4101790 0x7f6bd419ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:21.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bd8c3c700 1 -- 172.21.15.121:0/2183682697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd4101790 msgr2=0x7f6bd419ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:21.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bd8c3c700 1 --2- 172.21.15.121:0/2183682697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd4101790 0x7f6bd419ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bd8c3c700 1 -- 172.21.15.121:0/2183682697 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc00095f0 con 0x7f6bd4100de0 2024-04-05T21:56:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.257+0000 7f6bd3fff700 1 --2- 172.21.15.121:0/2183682697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd4101790 0x7f6bd419ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:56:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.258+0000 7f6bd8c3c700 1 --2- 172.21.15.121:0/2183682697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd4100de0 0x7f6bd419c4f0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800b4b0 tx=0x7f6bc800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.258+0000 7f6bd1ffb700 1 -- 172.21.15.121:0/2183682697 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bc8011800 con 0x7f6bd4100de0 2024-04-05T21:56:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.258+0000 7f6bdaea0700 1 -- 172.21.15.121:0/2183682697 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd41968c0 con 0x7f6bd4100de0 2024-04-05T21:56:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.258+0000 7f6bd1ffb700 1 -- 172.21.15.121:0/2183682697 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bc8011e00 con 0x7f6bd4100de0 2024-04-05T21:56:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.258+0000 7f6bd1ffb700 1 -- 172.21.15.121:0/2183682697 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bc8010e50 con 0x7f6bd4100de0 2024-04-05T21:56:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.258+0000 7f6bdaea0700 1 -- 172.21.15.121:0/2183682697 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd4196e10 con 0x7f6bd4100de0 2024-04-05T21:56:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.259+0000 7f6bdaea0700 1 -- 172.21.15.121:0/2183682697 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd4109110 con 0x7f6bd4100de0 2024-04-05T21:56:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.259+0000 7f6bd1ffb700 1 -- 172.21.15.121:0/2183682697 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6bc8010380 con 0x7f6bd4100de0 2024-04-05T21:56:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.260+0000 7f6bd1ffb700 1 --2- 172.21.15.121:0/2183682697 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6bc406cce0 0x7f6bc406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.260+0000 7f6bd1ffb700 1 -- 172.21.15.121:0/2183682697 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6bc808c190 con 0x7f6bd4100de0 2024-04-05T21:56:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.260+0000 7f6bd3fff700 1 --2- 172.21.15.121:0/2183682697 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6bc406cce0 0x7f6bc406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.260+0000 7f6bd3fff700 1 --2- 172.21.15.121:0/2183682697 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6bc406cce0 0x7f6bc406f1a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00059d0 tx=0x7f6bc0005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:21.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.265+0000 7f6bd1ffb700 1 -- 172.21.15.121:0/2183682697 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6bc8059c90 con 0x7f6bd4100de0 2024-04-05T21:56:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:20 smithi188 ceph-mon[107793]: pgmap v904: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:21.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.473+0000 7f6bdaea0700 1 -- 172.21.15.121:0/2183682697 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bd41970e0 con 0x7f6bc406cce0 2024-04-05T21:56:21.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.474+0000 7f6bd1ffb700 1 -- 172.21.15.121:0/2183682697 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6bd41970e0 con 0x7f6bc406cce0 2024-04-05T21:56:21.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.476+0000 7f6bdaea0700 1 -- 172.21.15.121:0/2183682697 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6bc406cce0 msgr2=0x7f6bc406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:21.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.477+0000 7f6bdaea0700 1 --2- 172.21.15.121:0/2183682697 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6bc406cce0 0x7f6bc406f1a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00059d0 tx=0x7f6bc0005960 comp rx=0 tx=0).stop 2024-04-05T21:56:21.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.477+0000 7f6bdaea0700 1 -- 172.21.15.121:0/2183682697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd4100de0 msgr2=0x7f6bd419c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:21.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.477+0000 7f6bdaea0700 1 --2- 172.21.15.121:0/2183682697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd4100de0 0x7f6bd419c4f0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800b4b0 tx=0x7f6bc800b870 comp rx=0 tx=0).stop 2024-04-05T21:56:21.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.477+0000 7f6bdaea0700 1 -- 172.21.15.121:0/2183682697 shutdown_connections 2024-04-05T21:56:21.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.477+0000 7f6bdaea0700 1 --2- 172.21.15.121:0/2183682697 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6bc406cce0 0x7f6bc406f1a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.477+0000 7f6bdaea0700 1 --2- 172.21.15.121:0/2183682697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bd4100de0 0x7f6bd419c4f0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.477+0000 7f6bdaea0700 1 --2- 172.21.15.121:0/2183682697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bd4101790 0x7f6bd419ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.477+0000 7f6bdaea0700 1 -- 172.21.15.121:0/2183682697 >> 172.21.15.121:0/2183682697 conn(0x7f6bd40fc6b0 msgr2=0x7f6bd40fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:21.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.477+0000 7f6bdaea0700 1 -- 172.21.15.121:0/2183682697 shutdown_connections 2024-04-05T21:56:21.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.477+0000 7f6bdaea0700 1 -- 172.21.15.121:0/2183682697 wait complete. 2024-04-05T21:56:21.487 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:56:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.580+0000 7fd794630700 1 -- 172.21.15.121:0/3378621119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd78c0ffeb0 msgr2=0x7fd78c105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.580+0000 7fd794630700 1 --2- 172.21.15.121:0/3378621119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd78c0ffeb0 0x7fd78c105840 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd788009910 tx=0x7fd788009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.580+0000 7fd794630700 1 -- 172.21.15.121:0/3378621119 shutdown_connections 2024-04-05T21:56:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.580+0000 7fd794630700 1 --2- 172.21.15.121:0/3378621119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd78c0ffeb0 0x7fd78c105840 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.580+0000 7fd794630700 1 --2- 172.21.15.121:0/3378621119 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd78c0ff500 0x7fd78c0ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.580+0000 7fd794630700 1 -- 172.21.15.121:0/3378621119 >> 172.21.15.121:0/3378621119 conn(0x7fd78c074d80 msgr2=0x7fd78c075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.580+0000 7fd794630700 1 -- 172.21.15.121:0/3378621119 shutdown_connections 2024-04-05T21:56:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.580+0000 7fd794630700 1 -- 172.21.15.121:0/3378621119 wait complete. 2024-04-05T21:56:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.580+0000 7fd794630700 1 Processor -- start 2024-04-05T21:56:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.580+0000 7fd794630700 1 -- start start 2024-04-05T21:56:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd794630700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd78c0ff500 0x7fd78c100480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:21.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd794630700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd78c0ffeb0 0x7fd78c1009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd794630700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c100f00 con 0x7fd78c0ffeb0 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd794630700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c101040 con 0x7fd78c0ff500 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd791bcb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd78c0ffeb0 0x7fd78c1009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd791bcb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd78c0ffeb0 0x7fd78c1009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41672/0 (socket says 172.21.15.121:41672) 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd791bcb700 1 -- 172.21.15.121:0/4144907010 learned_addr learned my addr 172.21.15.121:0/4144907010 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd7923cc700 1 --2- 172.21.15.121:0/4144907010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd78c0ff500 0x7fd78c100480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd791bcb700 1 -- 172.21.15.121:0/4144907010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd78c0ff500 msgr2=0x7fd78c100480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd791bcb700 1 --2- 172.21.15.121:0/4144907010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd78c0ff500 0x7fd78c100480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd791bcb700 1 -- 172.21.15.121:0/4144907010 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7880095f0 con 0x7fd78c0ffeb0 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd791bcb700 1 --2- 172.21.15.121:0/4144907010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd78c0ffeb0 0x7fd78c1009c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd7880092e0 tx=0x7fd78800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd7837fe700 1 -- 172.21.15.121:0/4144907010 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd78801d070 con 0x7fd78c0ffeb0 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd7837fe700 1 -- 172.21.15.121:0/4144907010 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd78800f460 con 0x7fd78c0ffeb0 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd794630700 1 -- 172.21.15.121:0/4144907010 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd78c1012c0 con 0x7fd78c0ffeb0 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd7837fe700 1 -- 172.21.15.121:0/4144907010 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd788021600 con 0x7fd78c0ffeb0 2024-04-05T21:56:21.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.581+0000 7fd794630700 1 -- 172.21.15.121:0/4144907010 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd78c1a2cb0 con 0x7fd78c0ffeb0 2024-04-05T21:56:21.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.582+0000 7fd794630700 1 -- 172.21.15.121:0/4144907010 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd78c1091b0 con 0x7fd78c0ffeb0 2024-04-05T21:56:21.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.583+0000 7fd7837fe700 1 -- 172.21.15.121:0/4144907010 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd78800f5d0 con 0x7fd78c0ffeb0 2024-04-05T21:56:21.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.583+0000 7fd7837fe700 1 --2- 172.21.15.121:0/4144907010 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd77806cce0 0x7fd77806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:21.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.583+0000 7fd7837fe700 1 -- 172.21.15.121:0/4144907010 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd78808d860 con 0x7fd78c0ffeb0 2024-04-05T21:56:21.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.583+0000 7fd7923cc700 1 --2- 172.21.15.121:0/4144907010 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd77806cce0 0x7fd77806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:21.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.583+0000 7fd7923cc700 1 --2- 172.21.15.121:0/4144907010 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd77806cce0 0x7fd77806f1a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd77c006fd0 tx=0x7fd77c008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:21.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.588+0000 7fd7837fe700 1 -- 172.21.15.121:0/4144907010 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd7880578d0 con 0x7fd78c0ffeb0 2024-04-05T21:56:21.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.797+0000 7fd794630700 1 -- 172.21.15.121:0/4144907010 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd78c02cd30 con 0x7fd77806cce0 2024-04-05T21:56:21.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.799+0000 7fd7837fe700 1 -- 172.21.15.121:0/4144907010 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd78c02cd30 con 0x7fd77806cce0 2024-04-05T21:56:21.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.801+0000 7fd794630700 1 -- 172.21.15.121:0/4144907010 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd77806cce0 msgr2=0x7fd77806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:21.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.801+0000 7fd794630700 1 --2- 172.21.15.121:0/4144907010 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd77806cce0 0x7fd77806f1a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd77c006fd0 tx=0x7fd77c008040 comp rx=0 tx=0).stop 2024-04-05T21:56:21.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.801+0000 7fd794630700 1 -- 172.21.15.121:0/4144907010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd78c0ffeb0 msgr2=0x7fd78c1009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:21.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.801+0000 7fd794630700 1 --2- 172.21.15.121:0/4144907010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd78c0ffeb0 0x7fd78c1009c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd7880092e0 tx=0x7fd78800be20 comp rx=0 tx=0).stop 2024-04-05T21:56:21.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.801+0000 7fd794630700 1 -- 172.21.15.121:0/4144907010 shutdown_connections 2024-04-05T21:56:21.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.801+0000 7fd794630700 1 --2- 172.21.15.121:0/4144907010 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd77806cce0 0x7fd77806f1a0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.801+0000 7fd794630700 1 --2- 172.21.15.121:0/4144907010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd78c0ff500 0x7fd78c100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.801+0000 7fd794630700 1 --2- 172.21.15.121:0/4144907010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd78c0ffeb0 0x7fd78c1009c0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.801+0000 7fd794630700 1 -- 172.21.15.121:0/4144907010 >> 172.21.15.121:0/4144907010 conn(0x7fd78c074d80 msgr2=0x7fd78c0fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:21.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.801+0000 7fd794630700 1 -- 172.21.15.121:0/4144907010 shutdown_connections 2024-04-05T21:56:21.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.801+0000 7fd794630700 1 -- 172.21.15.121:0/4144907010 wait complete. 2024-04-05T21:56:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.903+0000 7f1d0d081700 1 -- 172.21.15.121:0/3159079152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d08101d00 msgr2=0x7f1d081020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.903+0000 7f1d0d081700 1 --2- 172.21.15.121:0/3159079152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d08101d00 0x7f1d081020e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009910 tx=0x7f1cfc009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:21.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.903+0000 7f1d0d081700 1 -- 172.21.15.121:0/3159079152 shutdown_connections 2024-04-05T21:56:21.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.903+0000 7f1d0d081700 1 --2- 172.21.15.121:0/3159079152 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d08102620 0x7f1d0810ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.903+0000 7f1d0d081700 1 --2- 172.21.15.121:0/3159079152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d08101d00 0x7f1d081020e0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.903+0000 7f1d0d081700 1 -- 172.21.15.121:0/3159079152 >> 172.21.15.121:0/3159079152 conn(0x7f1d080fb570 msgr2=0x7f1d080fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:21.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.903+0000 7f1d0d081700 1 -- 172.21.15.121:0/3159079152 shutdown_connections 2024-04-05T21:56:21.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.903+0000 7f1d0d081700 1 -- 172.21.15.121:0/3159079152 wait complete. 2024-04-05T21:56:21.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.903+0000 7f1d0d081700 1 Processor -- start 2024-04-05T21:56:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1d0d081700 1 -- start start 2024-04-05T21:56:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1d0d081700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d08101d00 0x7f1d080ff960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1d0d081700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d08102620 0x7f1d080ffea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1d0d081700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d081003e0 con 0x7f1d08102620 2024-04-05T21:56:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1d0d081700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d08100520 con 0x7f1d08101d00 2024-04-05T21:56:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1d0659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d08102620 0x7f1d080ffea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1d0659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d08102620 0x7f1d080ffea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41684/0 (socket says 172.21.15.121:41684) 2024-04-05T21:56:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1d0659c700 1 -- 172.21.15.121:0/4145758256 learned_addr learned my addr 172.21.15.121:0/4145758256 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1d0659c700 1 -- 172.21.15.121:0/4145758256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d08101d00 msgr2=0x7f1d080ff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1d0659c700 1 --2- 172.21.15.121:0/4145758256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d08101d00 0x7f1d080ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1d0659c700 1 -- 172.21.15.121:0/4145758256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cfc0095f0 con 0x7f1d08102620 2024-04-05T21:56:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1d0659c700 1 --2- 172.21.15.121:0/4145758256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d08102620 0x7f1d080ffea0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800d6d0 tx=0x7f1cf800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1ceffff700 1 -- 172.21.15.121:0/4145758256 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cf800f860 con 0x7f1d08102620 2024-04-05T21:56:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.904+0000 7f1d0d081700 1 -- 172.21.15.121:0/4145758256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d081a2850 con 0x7f1d08102620 2024-04-05T21:56:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.905+0000 7f1ceffff700 1 -- 172.21.15.121:0/4145758256 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1cf800fe60 con 0x7f1d08102620 2024-04-05T21:56:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.905+0000 7f1d0d081700 1 -- 172.21.15.121:0/4145758256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d081a2c60 con 0x7f1d08102620 2024-04-05T21:56:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.905+0000 7f1ceffff700 1 -- 172.21.15.121:0/4145758256 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cf800e3b0 con 0x7f1d08102620 2024-04-05T21:56:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.905+0000 7f1d0d081700 1 -- 172.21.15.121:0/4145758256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d08108210 con 0x7f1d08102620 2024-04-05T21:56:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.906+0000 7f1ceffff700 1 -- 172.21.15.121:0/4145758256 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1cf8010460 con 0x7f1d08102620 2024-04-05T21:56:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.906+0000 7f1ceffff700 1 --2- 172.21.15.121:0/4145758256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1cf006cce0 0x7f1cf006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.906+0000 7f1ceffff700 1 -- 172.21.15.121:0/4145758256 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1cf808c050 con 0x7f1d08102620 2024-04-05T21:56:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.906+0000 7f1d06d9d700 1 --2- 172.21.15.121:0/4145758256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1cf006cce0 0x7f1cf006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.907+0000 7f1d06d9d700 1 --2- 172.21.15.121:0/4145758256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1cf006cce0 0x7f1cf006f1a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc00b5c0 tx=0x7f1cfc005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:21.911+0000 7f1ceffff700 1 -- 172.21.15.121:0/4145758256 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1cf8056140 con 0x7f1d08102620 2024-04-05T21:56:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.111+0000 7f1d0d081700 1 -- 172.21.15.121:0/4145758256 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d081006e0 con 0x7f1cf006cce0 2024-04-05T21:56:22.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.121+0000 7f1ceffff700 1 -- 172.21.15.121:0/4145758256 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f1d081006e0 con 0x7f1cf006cce0 2024-04-05T21:56:22.122 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:56:22.122 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (27m) 5m ago 38m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:56:22.122 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (38m) 5m ago 38m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:56:22.122 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (37m) 5m ago 37m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:56:22.122 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (25m) 5m ago 38m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:56:22.122 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (25m) 5m ago 37m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:56:22.122 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (26m) 5m ago 38m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:56:22.122 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (21m) 5m ago 32m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:56:22.122 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (21m) 5m ago 32m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:56:22.122 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (20m) 5m ago 32m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:56:22.122 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (5m) 5m ago 32m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:56:22.123 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (29m) 5m ago 40m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:56:22.123 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (28m) 5m ago 37m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:56:22.123 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (25m) 5m ago 40m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:56:22.123 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (25m) 5m ago 37m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:56:22.123 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (28m) 5m ago 38m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:56:22.123 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (27m) 5m ago 37m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:56:22.123 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (24m) 5m ago 36m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:56:22.123 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (24m) 5m ago 36m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:56:22.123 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (23m) 5m ago 35m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:56:22.123 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (23m) 5m ago 35m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:56:22.123 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (22m) 5m ago 34m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:56:22.123 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (21m) 5m ago 34m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:56:22.123 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (27m) 5m ago 38m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:56:22.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.123+0000 7f1d0d081700 1 -- 172.21.15.121:0/4145758256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1cf006cce0 msgr2=0x7f1cf006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:22.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.123+0000 7f1d0d081700 1 --2- 172.21.15.121:0/4145758256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1cf006cce0 0x7f1cf006f1a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc00b5c0 tx=0x7f1cfc005960 comp rx=0 tx=0).stop 2024-04-05T21:56:22.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.123+0000 7f1d0d081700 1 -- 172.21.15.121:0/4145758256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d08102620 msgr2=0x7f1d080ffea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:22.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.123+0000 7f1d0d081700 1 --2- 172.21.15.121:0/4145758256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d08102620 0x7f1d080ffea0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800d6d0 tx=0x7f1cf800da90 comp rx=0 tx=0).stop 2024-04-05T21:56:22.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.123+0000 7f1d0d081700 1 -- 172.21.15.121:0/4145758256 shutdown_connections 2024-04-05T21:56:22.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.123+0000 7f1d0d081700 1 --2- 172.21.15.121:0/4145758256 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1cf006cce0 0x7f1cf006f1a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.123+0000 7f1d0d081700 1 --2- 172.21.15.121:0/4145758256 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1d08101d00 0x7f1d080ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.123+0000 7f1d0d081700 1 --2- 172.21.15.121:0/4145758256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d08102620 0x7f1d080ffea0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.123+0000 7f1d0d081700 1 -- 172.21.15.121:0/4145758256 >> 172.21.15.121:0/4145758256 conn(0x7f1d080fb570 msgr2=0x7f1d08105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:22.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.123+0000 7f1d0d081700 1 -- 172.21.15.121:0/4145758256 shutdown_connections 2024-04-05T21:56:22.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.123+0000 7f1d0d081700 1 -- 172.21.15.121:0/4145758256 wait complete. 2024-04-05T21:56:22.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.225+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/1527336138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bbc069020 msgr2=0x7f7bbc105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:22.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.225+0000 7f7bc1d8d700 1 --2- 172.21.15.121:0/1527336138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bbc069020 0x7f7bbc105790 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009910 tx=0x7f7bac009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:22.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.225+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/1527336138 shutdown_connections 2024-04-05T21:56:22.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.225+0000 7f7bc1d8d700 1 --2- 172.21.15.121:0/1527336138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bbc069020 0x7f7bbc105790 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.225+0000 7f7bc1d8d700 1 --2- 172.21.15.121:0/1527336138 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bbc068670 0x7f7bbc068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.225+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/1527336138 >> 172.21.15.121:0/1527336138 conn(0x7f7bbc0755a0 msgr2=0x7f7bbc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:22.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.225+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/1527336138 shutdown_connections 2024-04-05T21:56:22.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.225+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/1527336138 wait complete. 2024-04-05T21:56:22.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.225+0000 7f7bc1d8d700 1 Processor -- start 2024-04-05T21:56:22.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bc1d8d700 1 -- start start 2024-04-05T21:56:22.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bc1d8d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bbc068670 0x7f7bbc19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:22.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bc1d8d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bbc069020 0x7f7bbc19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:22.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bc1d8d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc19f250 con 0x7f7bbc069020 2024-04-05T21:56:22.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bc1d8d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc198790 con 0x7f7bbc068670 2024-04-05T21:56:22.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bbaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bbc069020 0x7f7bbc19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:22.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bbaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bbc069020 0x7f7bbc19ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41700/0 (socket says 172.21.15.121:41700) 2024-04-05T21:56:22.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bbaffd700 1 -- 172.21.15.121:0/830628993 learned_addr learned my addr 172.21.15.121:0/830628993 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:22.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bbb7fe700 1 --2- 172.21.15.121:0/830628993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bbc068670 0x7f7bbc19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:22.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bbaffd700 1 -- 172.21.15.121:0/830628993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bbc068670 msgr2=0x7f7bbc19e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:22.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bbaffd700 1 --2- 172.21.15.121:0/830628993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bbc068670 0x7f7bbc19e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bbaffd700 1 -- 172.21.15.121:0/830628993 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bac0095f0 con 0x7f7bbc069020 2024-04-05T21:56:22.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bbaffd700 1 --2- 172.21.15.121:0/830628993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bbc069020 0x7f7bbc19ec50 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00bda0 tx=0x7f7bac00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:22.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bb8ff9700 1 -- 172.21.15.121:0/830628993 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bac01d070 con 0x7f7bbc069020 2024-04-05T21:56:22.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bb8ff9700 1 -- 172.21.15.121:0/830628993 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bac00f460 con 0x7f7bbc069020 2024-04-05T21:56:22.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/830628993 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bbc198a10 con 0x7f7bbc069020 2024-04-05T21:56:22.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bb8ff9700 1 -- 172.21.15.121:0/830628993 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bac021640 con 0x7f7bbc069020 2024-04-05T21:56:22.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.226+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/830628993 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bbc198f00 con 0x7f7bbc069020 2024-04-05T21:56:22.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.227+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/830628993 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bbc04ea90 con 0x7f7bbc069020 2024-04-05T21:56:22.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.228+0000 7f7bb8ff9700 1 -- 172.21.15.121:0/830628993 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7bac00fa60 con 0x7f7bbc069020 2024-04-05T21:56:22.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.228+0000 7f7bb8ff9700 1 --2- 172.21.15.121:0/830628993 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ba406cce0 0x7f7ba406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:22.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.228+0000 7f7bb8ff9700 1 -- 172.21.15.121:0/830628993 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7bac08d7a0 con 0x7f7bbc069020 2024-04-05T21:56:22.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.228+0000 7f7bbb7fe700 1 --2- 172.21.15.121:0/830628993 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ba406cce0 0x7f7ba406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:22.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.229+0000 7f7bbb7fe700 1 --2- 172.21.15.121:0/830628993 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ba406cce0 0x7f7ba406f1a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7bb000ac50 tx=0x7f7bb000a380 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:22.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.233+0000 7f7bb8ff9700 1 -- 172.21.15.121:0/830628993 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7bac05b2a0 con 0x7f7bbc069020 2024-04-05T21:56:22.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.510+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/830628993 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7bbc199ca0 con 0x7f7bbc069020 2024-04-05T21:56:22.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.510+0000 7f7bb8ff9700 1 -- 172.21.15.121:0/830628993 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f7bac05ae30 con 0x7f7bbc069020 2024-04-05T21:56:22.511 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:56:22.511 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:56:22.511 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:56:22.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:56:22.511 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:56:22.511 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:56:22.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:56:22.511 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:56:22.511 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:56:22.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:56:22.512 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:56:22.512 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:56:22.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:56:22.512 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:56:22.512 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:56:22.512 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:56:22.512 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:56:22.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.512+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/830628993 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ba406cce0 msgr2=0x7f7ba406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:22.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.512+0000 7f7bc1d8d700 1 --2- 172.21.15.121:0/830628993 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ba406cce0 0x7f7ba406f1a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7bb000ac50 tx=0x7f7bb000a380 comp rx=0 tx=0).stop 2024-04-05T21:56:22.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.512+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/830628993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bbc069020 msgr2=0x7f7bbc19ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:22.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.512+0000 7f7bc1d8d700 1 --2- 172.21.15.121:0/830628993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bbc069020 0x7f7bbc19ec50 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00bda0 tx=0x7f7bac00bdd0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.512+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/830628993 shutdown_connections 2024-04-05T21:56:22.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.512+0000 7f7bc1d8d700 1 --2- 172.21.15.121:0/830628993 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ba406cce0 0x7f7ba406f1a0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.512+0000 7f7bc1d8d700 1 --2- 172.21.15.121:0/830628993 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7bbc068670 0x7f7bbc19e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.512+0000 7f7bc1d8d700 1 --2- 172.21.15.121:0/830628993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bbc069020 0x7f7bbc19ec50 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.512+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/830628993 >> 172.21.15.121:0/830628993 conn(0x7f7bbc0755a0 msgr2=0x7f7bbc102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:22.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.513+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/830628993 shutdown_connections 2024-04-05T21:56:22.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.513+0000 7f7bc1d8d700 1 -- 172.21.15.121:0/830628993 wait complete. 2024-04-05T21:56:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.615+0000 7f0f978eb700 1 -- 172.21.15.121:0/3750024772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f90101790 msgr2=0x7f0f901057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.615+0000 7f0f978eb700 1 --2- 172.21.15.121:0/3750024772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f90101790 0x7f0f901057e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f0f80009910 tx=0x7f0f80009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:22.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.615+0000 7f0f978eb700 1 -- 172.21.15.121:0/3750024772 shutdown_connections 2024-04-05T21:56:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.615+0000 7f0f978eb700 1 --2- 172.21.15.121:0/3750024772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f90101790 0x7f0f901057e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.615+0000 7f0f978eb700 1 --2- 172.21.15.121:0/3750024772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f90100de0 0x7f0f901011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.615+0000 7f0f978eb700 1 -- 172.21.15.121:0/3750024772 >> 172.21.15.121:0/3750024772 conn(0x7f0f900fc670 msgr2=0x7f0f900fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.615+0000 7f0f978eb700 1 -- 172.21.15.121:0/3750024772 shutdown_connections 2024-04-05T21:56:22.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.615+0000 7f0f978eb700 1 -- 172.21.15.121:0/3750024772 wait complete. 2024-04-05T21:56:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f978eb700 1 Processor -- start 2024-04-05T21:56:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f978eb700 1 -- start start 2024-04-05T21:56:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f978eb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f90100de0 0x7f0f901966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f978eb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f90101790 0x7f0f90196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f978eb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f901972a0 con 0x7f0f90100de0 2024-04-05T21:56:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f978eb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f90199fd0 con 0x7f0f90101790 2024-04-05T21:56:22.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f95687700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f90100de0 0x7f0f901966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f95687700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f90100de0 0x7f0f901966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41710/0 (socket says 172.21.15.121:41710) 2024-04-05T21:56:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f95687700 1 -- 172.21.15.121:0/4287946889 learned_addr learned my addr 172.21.15.121:0/4287946889 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f94e86700 1 --2- 172.21.15.121:0/4287946889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f90101790 0x7f0f90196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f95687700 1 -- 172.21.15.121:0/4287946889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f90101790 msgr2=0x7f0f90196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f95687700 1 --2- 172.21.15.121:0/4287946889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f90101790 0x7f0f90196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f95687700 1 -- 172.21.15.121:0/4287946889 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f800095f0 con 0x7f0f90100de0 2024-04-05T21:56:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f94e86700 1 --2- 172.21.15.121:0/4287946889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f90101790 0x7f0f90196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:56:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f95687700 1 --2- 172.21.15.121:0/4287946889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f90100de0 0x7f0f901966d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c00d8d0 tx=0x7f0f8c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f867fc700 1 -- 172.21.15.121:0/4287946889 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f8c009940 con 0x7f0f90100de0 2024-04-05T21:56:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f867fc700 1 -- 172.21.15.121:0/4287946889 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f8c010460 con 0x7f0f90100de0 2024-04-05T21:56:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f867fc700 1 -- 172.21.15.121:0/4287946889 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f8c00f5f0 con 0x7f0f90100de0 2024-04-05T21:56:22.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.616+0000 7f0f978eb700 1 -- 172.21.15.121:0/4287946889 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9019a2b0 con 0x7f0f90100de0 2024-04-05T21:56:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.617+0000 7f0f978eb700 1 -- 172.21.15.121:0/4287946889 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9019a780 con 0x7f0f90100de0 2024-04-05T21:56:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.617+0000 7f0f978eb700 1 -- 172.21.15.121:0/4287946889 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f901091e0 con 0x7f0f90100de0 2024-04-05T21:56:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.618+0000 7f0f867fc700 1 -- 172.21.15.121:0/4287946889 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0f8c0105d0 con 0x7f0f90100de0 2024-04-05T21:56:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.618+0000 7f0f867fc700 1 --2- 172.21.15.121:0/4287946889 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0f7c06cce0 0x7f0f7c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:22.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.618+0000 7f0f94e86700 1 --2- 172.21.15.121:0/4287946889 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0f7c06cce0 0x7f0f7c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.618+0000 7f0f867fc700 1 -- 172.21.15.121:0/4287946889 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0f8c08c5e0 con 0x7f0f90100de0 2024-04-05T21:56:22.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.619+0000 7f0f94e86700 1 --2- 172.21.15.121:0/4287946889 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0f7c06cce0 0x7f0f7c06f1a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0f90197df0 tx=0x7f0f8001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:22.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.623+0000 7f0f867fc700 1 -- 172.21.15.121:0/4287946889 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0f8c056650 con 0x7f0f90100de0 2024-04-05T21:56:22.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.857+0000 7f0f978eb700 1 -- 172.21.15.121:0/4287946889 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0f9004ea90 con 0x7f0f90100de0 2024-04-05T21:56:22.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.857+0000 7f0f867fc700 1 -- 172.21.15.121:0/4287946889 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f0f8c010880 con 0x7f0f90100de0 2024-04-05T21:56:22.858 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:56:22.858 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:56:22.858 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:56:22.858 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:56:22.858 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:56:22.858 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:56:22.858 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:56:22.858 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:56:22.858 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:56:22.858 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:56:22.858 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:56:22.859 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:56:22.859 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:56:22.859 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:56:22.859 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:56:22.859 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:56:22.859 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:56:22.859 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:56:22.859 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:56:22.859 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:56:22.859 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:56:22.859 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:56:22.859 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:56:22.860 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:56:22.861 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:56:22.861 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:56:22.861 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:56:22.861 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:56:22.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.859+0000 7f0f978eb700 1 -- 172.21.15.121:0/4287946889 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0f7c06cce0 msgr2=0x7f0f7c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:22.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.859+0000 7f0f978eb700 1 --2- 172.21.15.121:0/4287946889 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0f7c06cce0 0x7f0f7c06f1a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0f90197df0 tx=0x7f0f8001a040 comp rx=0 tx=0).stop 2024-04-05T21:56:22.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.859+0000 7f0f978eb700 1 -- 172.21.15.121:0/4287946889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f90100de0 msgr2=0x7f0f901966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:22.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.859+0000 7f0f978eb700 1 --2- 172.21.15.121:0/4287946889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f90100de0 0x7f0f901966d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0f8c00d8d0 tx=0x7f0f8c00dc90 comp rx=0 tx=0).stop 2024-04-05T21:56:22.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.859+0000 7f0f978eb700 1 -- 172.21.15.121:0/4287946889 shutdown_connections 2024-04-05T21:56:22.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.859+0000 7f0f978eb700 1 --2- 172.21.15.121:0/4287946889 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0f7c06cce0 0x7f0f7c06f1a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.859+0000 7f0f978eb700 1 --2- 172.21.15.121:0/4287946889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f90100de0 0x7f0f901966d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.859+0000 7f0f978eb700 1 --2- 172.21.15.121:0/4287946889 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0f90101790 0x7f0f90196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.859+0000 7f0f978eb700 1 -- 172.21.15.121:0/4287946889 >> 172.21.15.121:0/4287946889 conn(0x7f0f900fc670 msgr2=0x7f0f900fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:22.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.859+0000 7f0f978eb700 1 -- 172.21.15.121:0/4287946889 shutdown_connections 2024-04-05T21:56:22.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.859+0000 7f0f978eb700 1 -- 172.21.15.121:0/4287946889 wait complete. 2024-04-05T21:56:22.862 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:56:22.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.963+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1609032023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb980ffa40 msgr2=0x7fbb980ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:22.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.963+0000 7fbb9fc0c700 1 --2- 172.21.15.121:0/1609032023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb980ffa40 0x7fbb980ffec0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009910 tx=0x7fbb94009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:22.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.963+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1609032023 shutdown_connections 2024-04-05T21:56:22.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.963+0000 7fbb9fc0c700 1 --2- 172.21.15.121:0/1609032023 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb980ffa40 0x7fbb980ffec0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.963+0000 7fbb9fc0c700 1 --2- 172.21.15.121:0/1609032023 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb98105a40 0x7fbb98105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.963+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1609032023 >> 172.21.15.121:0/1609032023 conn(0x7fbb980fb570 msgr2=0x7fbb980fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:22.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.963+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1609032023 shutdown_connections 2024-04-05T21:56:22.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.963+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1609032023 wait complete. 2024-04-05T21:56:22.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9fc0c700 1 Processor -- start 2024-04-05T21:56:22.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9fc0c700 1 -- start start 2024-04-05T21:56:22.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9fc0c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb980ffa40 0x7fbb98196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:22.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9fc0c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb98105a40 0x7fbb98196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:22.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9fc0c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb981971a0 con 0x7fbb98105a40 2024-04-05T21:56:22.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9fc0c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb981972e0 con 0x7fbb980ffa40 2024-04-05T21:56:22.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9d1a7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb98105a40 0x7fbb98196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:22.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9d1a7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb98105a40 0x7fbb98196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41718/0 (socket says 172.21.15.121:41718) 2024-04-05T21:56:22.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9d1a7700 1 -- 172.21.15.121:0/1872184532 learned_addr learned my addr 172.21.15.121:0/1872184532 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:22.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9d9a8700 1 --2- 172.21.15.121:0/1872184532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb980ffa40 0x7fbb98196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:22.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9d1a7700 1 -- 172.21.15.121:0/1872184532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb980ffa40 msgr2=0x7fbb98196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:22.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9d1a7700 1 --2- 172.21.15.121:0/1872184532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb980ffa40 0x7fbb98196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:22.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9d1a7700 1 -- 172.21.15.121:0/1872184532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb940095f0 con 0x7fbb98105a40 2024-04-05T21:56:22.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.964+0000 7fbb9d1a7700 1 --2- 172.21.15.121:0/1872184532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb98105a40 0x7fbb98196b50 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400b5c0 tx=0x7fbb9400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:22.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.965+0000 7fbb8effd700 1 -- 172.21.15.121:0/1872184532 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb9401d070 con 0x7fbb98105a40 2024-04-05T21:56:22.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.965+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1872184532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9819b0d0 con 0x7fbb98105a40 2024-04-05T21:56:22.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.965+0000 7fbb8effd700 1 -- 172.21.15.121:0/1872184532 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb9400f460 con 0x7fbb98105a40 2024-04-05T21:56:22.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.965+0000 7fbb8effd700 1 -- 172.21.15.121:0/1872184532 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb94021640 con 0x7fbb98105a40 2024-04-05T21:56:22.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.965+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1872184532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb9819b5c0 con 0x7fbb98105a40 2024-04-05T21:56:22.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.966+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1872184532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb98108fe0 con 0x7fbb98105a40 2024-04-05T21:56:22.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.966+0000 7fbb8effd700 1 -- 172.21.15.121:0/1872184532 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbb9400fa20 con 0x7fbb98105a40 2024-04-05T21:56:22.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.966+0000 7fbb8effd700 1 --2- 172.21.15.121:0/1872184532 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbb8406cce0 0x7fbb8406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:22.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.967+0000 7fbb8effd700 1 -- 172.21.15.121:0/1872184532 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbb9408d710 con 0x7fbb98105a40 2024-04-05T21:56:22.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.967+0000 7fbb9d9a8700 1 --2- 172.21.15.121:0/1872184532 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbb8406cce0 0x7fbb8406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:22.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.967+0000 7fbb9d9a8700 1 --2- 172.21.15.121:0/1872184532 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbb8406cce0 0x7fbb8406f1a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbb8800ac50 tx=0x7fbb8800a380 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:22.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:22.971+0000 7fbb8effd700 1 -- 172.21.15.121:0/1872184532 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbb94057800 con 0x7fbb98105a40 2024-04-05T21:56:23.074 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:22 smithi121 ceph-mon[145859]: from='client.35466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:23.075 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:22 smithi121 ceph-mon[145859]: pgmap v905: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:23.075 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:22 smithi121 ceph-mon[145859]: from='client.35470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:23.075 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:22 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/830628993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:56:23.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.193+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1872184532 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb9802cd50 con 0x7fbb8406cce0 2024-04-05T21:56:23.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.195+0000 7fbb8effd700 1 -- 172.21.15.121:0/1872184532 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbb9802cd50 con 0x7fbb8406cce0 2024-04-05T21:56:23.195 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:56:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:56:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:56:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:56:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:56:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:56:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:56:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:56:23.195 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:56:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:56:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:56:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:56:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:56:23.196 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:56:23.196 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:56:23.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.197+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1872184532 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbb8406cce0 msgr2=0x7fbb8406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:23.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.197+0000 7fbb9fc0c700 1 --2- 172.21.15.121:0/1872184532 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbb8406cce0 0x7fbb8406f1a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbb8800ac50 tx=0x7fbb8800a380 comp rx=0 tx=0).stop 2024-04-05T21:56:23.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.197+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1872184532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb98105a40 msgr2=0x7fbb98196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:23.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.197+0000 7fbb9fc0c700 1 --2- 172.21.15.121:0/1872184532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb98105a40 0x7fbb98196b50 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400b5c0 tx=0x7fbb9400bd70 comp rx=0 tx=0).stop 2024-04-05T21:56:23.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.197+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1872184532 shutdown_connections 2024-04-05T21:56:23.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.197+0000 7fbb9fc0c700 1 --2- 172.21.15.121:0/1872184532 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbb8406cce0 0x7fbb8406f1a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:23.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.197+0000 7fbb9fc0c700 1 --2- 172.21.15.121:0/1872184532 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb980ffa40 0x7fbb98196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:23.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.197+0000 7fbb9fc0c700 1 --2- 172.21.15.121:0/1872184532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb98105a40 0x7fbb98196b50 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:23.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.197+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1872184532 >> 172.21.15.121:0/1872184532 conn(0x7fbb980fb570 msgr2=0x7fbb98068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:23.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.197+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1872184532 shutdown_connections 2024-04-05T21:56:23.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.197+0000 7fbb9fc0c700 1 -- 172.21.15.121:0/1872184532 wait complete. 2024-04-05T21:56:23.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:22 smithi188 ceph-mon[107793]: from='client.35466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:23.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:22 smithi188 ceph-mon[107793]: pgmap v905: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:23.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:22 smithi188 ceph-mon[107793]: from='client.35470 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:23.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:22 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/830628993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:56:23.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.299+0000 7fb6b1614700 1 -- 172.21.15.121:0/1807018019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ac0ff510 msgr2=0x7fb6ac0ff8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:23.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.299+0000 7fb6b1614700 1 --2- 172.21.15.121:0/1807018019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ac0ff510 0x7fb6ac0ff8f0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0009910 tx=0x7fb6a0009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:23.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.299+0000 7fb6b1614700 1 -- 172.21.15.121:0/1807018019 shutdown_connections 2024-04-05T21:56:23.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.299+0000 7fb6b1614700 1 --2- 172.21.15.121:0/1807018019 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6ac0ffec0 0x7fb6ac105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:23.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.299+0000 7fb6b1614700 1 --2- 172.21.15.121:0/1807018019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ac0ff510 0x7fb6ac0ff8f0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:23.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.299+0000 7fb6b1614700 1 -- 172.21.15.121:0/1807018019 >> 172.21.15.121:0/1807018019 conn(0x7fb6ac074d80 msgr2=0x7fb6ac075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:23.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.299+0000 7fb6b1614700 1 -- 172.21.15.121:0/1807018019 shutdown_connections 2024-04-05T21:56:23.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.299+0000 7fb6b1614700 1 -- 172.21.15.121:0/1807018019 wait complete. 2024-04-05T21:56:23.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.300+0000 7fb6b1614700 1 Processor -- start 2024-04-05T21:56:23.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.300+0000 7fb6b1614700 1 -- start start 2024-04-05T21:56:23.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.300+0000 7fb6b1614700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ac0ff510 0x7fb6ac19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:23.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.300+0000 7fb6b1614700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6ac0ffec0 0x7fb6ac19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:23.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.300+0000 7fb6b1614700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac19f220 con 0x7fb6ac0ff510 2024-04-05T21:56:23.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.300+0000 7fb6b1614700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac1986d0 con 0x7fb6ac0ffec0 2024-04-05T21:56:23.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.300+0000 7fb6aaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ac0ff510 0x7fb6ac19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:23.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.300+0000 7fb6aaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ac0ff510 0x7fb6ac19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41750/0 (socket says 172.21.15.121:41750) 2024-04-05T21:56:23.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.300+0000 7fb6aaffd700 1 -- 172.21.15.121:0/2592592695 learned_addr learned my addr 172.21.15.121:0/2592592695 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:23.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.300+0000 7fb6aa7fc700 1 --2- 172.21.15.121:0/2592592695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6ac0ffec0 0x7fb6ac19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:23.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.301+0000 7fb6aaffd700 1 -- 172.21.15.121:0/2592592695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6ac0ffec0 msgr2=0x7fb6ac19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:23.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.301+0000 7fb6aaffd700 1 --2- 172.21.15.121:0/2592592695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6ac0ffec0 0x7fb6ac19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:23.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.301+0000 7fb6aaffd700 1 -- 172.21.15.121:0/2592592695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a00095f0 con 0x7fb6ac0ff510 2024-04-05T21:56:23.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.301+0000 7fb6aaffd700 1 --2- 172.21.15.121:0/2592592695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ac0ff510 0x7fb6ac19e650 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00092e0 tx=0x7fb6a000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:23.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.301+0000 7fb693fff700 1 -- 172.21.15.121:0/2592592695 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6a001d070 con 0x7fb6ac0ff510 2024-04-05T21:56:23.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.301+0000 7fb693fff700 1 -- 172.21.15.121:0/2592592695 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6a0005e00 con 0x7fb6ac0ff510 2024-04-05T21:56:23.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.301+0000 7fb693fff700 1 -- 172.21.15.121:0/2592592695 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6a000fe90 con 0x7fb6ac0ff510 2024-04-05T21:56:23.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.301+0000 7fb6b1614700 1 -- 172.21.15.121:0/2592592695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ac198950 con 0x7fb6ac0ff510 2024-04-05T21:56:23.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.301+0000 7fb6b1614700 1 -- 172.21.15.121:0/2592592695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ac198e40 con 0x7fb6ac0ff510 2024-04-05T21:56:23.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.302+0000 7fb6b1614700 1 -- 172.21.15.121:0/2592592695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ac104fd0 con 0x7fb6ac0ff510 2024-04-05T21:56:23.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.302+0000 7fb693fff700 1 -- 172.21.15.121:0/2592592695 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb6a000f6d0 con 0x7fb6ac0ff510 2024-04-05T21:56:23.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.303+0000 7fb693fff700 1 --2- 172.21.15.121:0/2592592695 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb69406cce0 0x7fb69406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:23.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.303+0000 7fb693fff700 1 -- 172.21.15.121:0/2592592695 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb6a008d760 con 0x7fb6ac0ff510 2024-04-05T21:56:23.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.303+0000 7fb6aa7fc700 1 --2- 172.21.15.121:0/2592592695 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb69406cce0 0x7fb69406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:23.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.303+0000 7fb6aa7fc700 1 --2- 172.21.15.121:0/2592592695 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb69406cce0 0x7fb69406f1a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac199b40 tx=0x7fb69c00b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:23.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.308+0000 7fb693fff700 1 -- 172.21.15.121:0/2592592695 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb6a005b2e0 con 0x7fb6ac0ff510 2024-04-05T21:56:23.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.579+0000 7fb6b1614700 1 -- 172.21.15.121:0/2592592695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb6ac02cc70 con 0x7fb6ac0ff510 2024-04-05T21:56:23.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.579+0000 7fb693fff700 1 -- 172.21.15.121:0/2592592695 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fb6a0026e10 con 0x7fb6ac0ff510 2024-04-05T21:56:23.580 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:56:23.580 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:56:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:56:23.580 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:56:23.580 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:56:23.580 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:56:23.580 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:56:23.581 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:56:23.582 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:56:23.582 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:56:23.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.582+0000 7fb6b1614700 1 -- 172.21.15.121:0/2592592695 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb69406cce0 msgr2=0x7fb69406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:23.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.582+0000 7fb6b1614700 1 --2- 172.21.15.121:0/2592592695 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb69406cce0 0x7fb69406f1a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac199b40 tx=0x7fb69c00b410 comp rx=0 tx=0).stop 2024-04-05T21:56:23.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.582+0000 7fb6b1614700 1 -- 172.21.15.121:0/2592592695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ac0ff510 msgr2=0x7fb6ac19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:23.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.582+0000 7fb6b1614700 1 --2- 172.21.15.121:0/2592592695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ac0ff510 0x7fb6ac19e650 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb6a00092e0 tx=0x7fb6a000bd70 comp rx=0 tx=0).stop 2024-04-05T21:56:23.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.582+0000 7fb6b1614700 1 -- 172.21.15.121:0/2592592695 shutdown_connections 2024-04-05T21:56:23.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.582+0000 7fb6b1614700 1 --2- 172.21.15.121:0/2592592695 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb69406cce0 0x7fb69406f1a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:23.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.582+0000 7fb6b1614700 1 --2- 172.21.15.121:0/2592592695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6ac0ff510 0x7fb6ac19e650 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:23.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.582+0000 7fb6b1614700 1 --2- 172.21.15.121:0/2592592695 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb6ac0ffec0 0x7fb6ac19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:23.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.582+0000 7fb6b1614700 1 -- 172.21.15.121:0/2592592695 >> 172.21.15.121:0/2592592695 conn(0x7fb6ac074d80 msgr2=0x7fb6ac0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:23.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.582+0000 7fb6b1614700 1 -- 172.21.15.121:0/2592592695 shutdown_connections 2024-04-05T21:56:23.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:23.582+0000 7fb6b1614700 1 -- 172.21.15.121:0/2592592695 wait complete. 2024-04-05T21:56:24.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:23 smithi121 ceph-mon[145859]: from='client.35474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:24.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:23 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4287946889' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:56:24.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:23 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2592592695' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:56:24.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:23 smithi188 ceph-mon[107793]: from='client.35474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:24.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:23 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4287946889' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:56:24.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:23 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2592592695' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:56:25.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:24 smithi121 ceph-mon[145859]: from='client.35486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:25.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:24 smithi121 ceph-mon[145859]: pgmap v906: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:24 smithi188 ceph-mon[107793]: from='client.35486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:24 smithi188 ceph-mon[107793]: pgmap v906: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:27.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:26 smithi121 ceph-mon[145859]: pgmap v907: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:27.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:26 smithi188 ceph-mon[107793]: pgmap v907: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:29.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:28 smithi121 ceph-mon[145859]: pgmap v908: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:28 smithi188 ceph-mon[107793]: pgmap v908: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:30 smithi121 ceph-mon[145859]: pgmap v909: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:31.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:56:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:30 smithi188 ceph-mon[107793]: pgmap v909: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:56:33.173 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:32 smithi121 ceph-mon[145859]: pgmap v910: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:33.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:32 smithi188 ceph-mon[107793]: pgmap v910: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:35.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:34 smithi121 ceph-mon[145859]: pgmap v911: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:35.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:34 smithi188 ceph-mon[107793]: pgmap v911: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:37.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:36 smithi188 ceph-mon[107793]: pgmap v912: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:37.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:36 smithi121 ceph-mon[145859]: pgmap v912: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:39.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:38 smithi121 ceph-mon[145859]: pgmap v913: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:38 smithi188 ceph-mon[107793]: pgmap v913: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:40 smithi121 ceph-mon[145859]: pgmap v914: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:40 smithi188 ceph-mon[107793]: pgmap v914: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:43.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:42 smithi121 ceph-mon[145859]: pgmap v915: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:42 smithi188 ceph-mon[107793]: pgmap v915: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:45.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:44 smithi121 ceph-mon[145859]: pgmap v916: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:44 smithi188 ceph-mon[107793]: pgmap v916: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:56:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:56:47.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:46 smithi188 ceph-mon[107793]: pgmap v917: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:47.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:46 smithi121 ceph-mon[145859]: pgmap v917: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:49.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:48 smithi121 ceph-mon[145859]: pgmap v918: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:49.282 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:48 smithi188 ceph-mon[107793]: pgmap v918: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:51.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:50 smithi121 ceph-mon[145859]: pgmap v919: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:50 smithi188 ceph-mon[107793]: pgmap v919: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:53.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:52 smithi121 ceph-mon[145859]: pgmap v920: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:52 smithi188 ceph-mon[107793]: pgmap v920: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:53.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.697+0000 7f03b582e700 1 -- 172.21.15.121:0/2171537550 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b0100fd0 msgr2=0x7f03b0104720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:53.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.697+0000 7f03b582e700 1 --2- 172.21.15.121:0/2171537550 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b0100fd0 0x7f03b0104720 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009910 tx=0x7f03a4009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:53.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.697+0000 7f03b582e700 1 -- 172.21.15.121:0/2171537550 shutdown_connections 2024-04-05T21:56:53.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.697+0000 7f03b582e700 1 --2- 172.21.15.121:0/2171537550 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b0100fd0 0x7f03b0104720 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.697+0000 7f03b582e700 1 --2- 172.21.15.121:0/2171537550 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b0100620 0x7f03b0100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.698+0000 7f03b582e700 1 -- 172.21.15.121:0/2171537550 >> 172.21.15.121:0/2171537550 conn(0x7f03b0074e00 msgr2=0x7f03b0075210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.698+0000 7f03b582e700 1 -- 172.21.15.121:0/2171537550 shutdown_connections 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.698+0000 7f03b582e700 1 -- 172.21.15.121:0/2171537550 wait complete. 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.698+0000 7f03b582e700 1 Processor -- start 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.698+0000 7f03b582e700 1 -- start start 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03b582e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b0100620 0x7f03b0196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03b582e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b0100fd0 0x7f03b0196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03b582e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b0197270 con 0x7f03b0100fd0 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03b582e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b019b000 con 0x7f03b0100620 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03ae7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b0100fd0 0x7f03b0196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03ae7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b0100fd0 0x7f03b0196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33326/0 (socket says 172.21.15.121:33326) 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03ae7fc700 1 -- 172.21.15.121:0/1797714013 learned_addr learned my addr 172.21.15.121:0/1797714013 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:53.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03aeffd700 1 --2- 172.21.15.121:0/1797714013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b0100620 0x7f03b0196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:53.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03ae7fc700 1 -- 172.21.15.121:0/1797714013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b0100620 msgr2=0x7f03b0196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:53.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03ae7fc700 1 --2- 172.21.15.121:0/1797714013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b0100620 0x7f03b0196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:53.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03ae7fc700 1 -- 172.21.15.121:0/1797714013 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a40095f0 con 0x7f03b0100fd0 2024-04-05T21:56:53.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03ae7fc700 1 --2- 172.21.15.121:0/1797714013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b0100fd0 0x7f03b0196b90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f03a40092e0 tx=0x7f03a400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:53.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03b482c700 1 -- 172.21.15.121:0/1797714013 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03a401d070 con 0x7f03b0100fd0 2024-04-05T21:56:53.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03b582e700 1 -- 172.21.15.121:0/1797714013 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b019b280 con 0x7f03b0100fd0 2024-04-05T21:56:53.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03b482c700 1 -- 172.21.15.121:0/1797714013 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03a4005dc0 con 0x7f03b0100fd0 2024-04-05T21:56:53.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03b482c700 1 -- 172.21.15.121:0/1797714013 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03a400fea0 con 0x7f03b0100fd0 2024-04-05T21:56:53.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.699+0000 7f03b582e700 1 -- 172.21.15.121:0/1797714013 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b019b770 con 0x7f03b0100fd0 2024-04-05T21:56:53.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.701+0000 7f03b582e700 1 -- 172.21.15.121:0/1797714013 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b0109010 con 0x7f03b0100fd0 2024-04-05T21:56:53.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.701+0000 7f03b482c700 1 -- 172.21.15.121:0/1797714013 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f03a4003a60 con 0x7f03b0100fd0 2024-04-05T21:56:53.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.701+0000 7f03b482c700 1 --2- 172.21.15.121:0/1797714013 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f039806cce0 0x7f039806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:53.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.702+0000 7f03b482c700 1 -- 172.21.15.121:0/1797714013 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f03a408d740 con 0x7f03b0100fd0 2024-04-05T21:56:53.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.702+0000 7f03aeffd700 1 --2- 172.21.15.121:0/1797714013 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f039806cce0 0x7f039806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:53.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.702+0000 7f03aeffd700 1 --2- 172.21.15.121:0/1797714013 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f039806cce0 0x7f039806f1a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f03a0005950 tx=0x7f03a000b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:53.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.707+0000 7f03b482c700 1 -- 172.21.15.121:0/1797714013 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f03a40577b0 con 0x7f03b0100fd0 2024-04-05T21:56:53.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.919+0000 7f03b582e700 1 -- 172.21.15.121:0/1797714013 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f03b002cd60 con 0x7f039806cce0 2024-04-05T21:56:53.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.920+0000 7f03b482c700 1 -- 172.21.15.121:0/1797714013 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f03b002cd60 con 0x7f039806cce0 2024-04-05T21:56:53.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.923+0000 7f03b582e700 1 -- 172.21.15.121:0/1797714013 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f039806cce0 msgr2=0x7f039806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:53.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.923+0000 7f03b582e700 1 --2- 172.21.15.121:0/1797714013 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f039806cce0 0x7f039806f1a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f03a0005950 tx=0x7f03a000b410 comp rx=0 tx=0).stop 2024-04-05T21:56:53.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.923+0000 7f03b582e700 1 -- 172.21.15.121:0/1797714013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b0100fd0 msgr2=0x7f03b0196b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:53.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.923+0000 7f03b582e700 1 --2- 172.21.15.121:0/1797714013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b0100fd0 0x7f03b0196b90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f03a40092e0 tx=0x7f03a400bd70 comp rx=0 tx=0).stop 2024-04-05T21:56:53.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.923+0000 7f03b582e700 1 -- 172.21.15.121:0/1797714013 shutdown_connections 2024-04-05T21:56:53.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.923+0000 7f03b582e700 1 --2- 172.21.15.121:0/1797714013 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f039806cce0 0x7f039806f1a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:53.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.923+0000 7f03b582e700 1 --2- 172.21.15.121:0/1797714013 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f03b0100620 0x7f03b0196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:53.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.923+0000 7f03b582e700 1 --2- 172.21.15.121:0/1797714013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b0100fd0 0x7f03b0196b90 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:53.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.923+0000 7f03b582e700 1 -- 172.21.15.121:0/1797714013 >> 172.21.15.121:0/1797714013 conn(0x7f03b0074e00 msgr2=0x7f03b0103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:53.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.923+0000 7f03b582e700 1 -- 172.21.15.121:0/1797714013 shutdown_connections 2024-04-05T21:56:53.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:53.923+0000 7f03b582e700 1 -- 172.21.15.121:0/1797714013 wait complete. 2024-04-05T21:56:53.933 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:56:54.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.026+0000 7f5058e8f700 1 -- 172.21.15.121:0/4011587125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50540ffa40 msgr2=0x7f50540ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:54.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.026+0000 7f5058e8f700 1 --2- 172.21.15.121:0/4011587125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50540ffa40 0x7f50540ffec0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5048009910 tx=0x7f5048009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:54.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.027+0000 7f5058e8f700 1 -- 172.21.15.121:0/4011587125 shutdown_connections 2024-04-05T21:56:54.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.027+0000 7f5058e8f700 1 --2- 172.21.15.121:0/4011587125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50540ffa40 0x7f50540ffec0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.027+0000 7f5058e8f700 1 --2- 172.21.15.121:0/4011587125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5054105a40 0x7f5054105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.027+0000 7f5058e8f700 1 -- 172.21.15.121:0/4011587125 >> 172.21.15.121:0/4011587125 conn(0x7f50540fb570 msgr2=0x7f50540fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:54.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.027+0000 7f5058e8f700 1 -- 172.21.15.121:0/4011587125 shutdown_connections 2024-04-05T21:56:54.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.027+0000 7f5058e8f700 1 -- 172.21.15.121:0/4011587125 wait complete. 2024-04-05T21:56:54.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.027+0000 7f5058e8f700 1 Processor -- start 2024-04-05T21:56:54.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.027+0000 7f5058e8f700 1 -- start start 2024-04-05T21:56:54.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f5058e8f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50540ffa40 0x7f50540fbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:54.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f5058e8f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5054105a40 0x7f50540fc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:54.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f5058e8f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50540fc960 con 0x7f5054105a40 2024-04-05T21:56:54.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f5058e8f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50540fcaa0 con 0x7f50540ffa40 2024-04-05T21:56:54.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f5051d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5054105a40 0x7f50540fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:54.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f5051d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5054105a40 0x7f50540fc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33346/0 (socket says 172.21.15.121:33346) 2024-04-05T21:56:54.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f5051d9b700 1 -- 172.21.15.121:0/1739106944 learned_addr learned my addr 172.21.15.121:0/1739106944 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:54.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f505259c700 1 --2- 172.21.15.121:0/1739106944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50540ffa40 0x7f50540fbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:54.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f5051d9b700 1 -- 172.21.15.121:0/1739106944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50540ffa40 msgr2=0x7f50540fbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:54.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f5051d9b700 1 --2- 172.21.15.121:0/1739106944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50540ffa40 0x7f50540fbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f5051d9b700 1 -- 172.21.15.121:0/1739106944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50480095f0 con 0x7f5054105a40 2024-04-05T21:56:54.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f5051d9b700 1 --2- 172.21.15.121:0/1739106944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5054105a40 0x7f50540fc420 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f5048006010 tx=0x7f504800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:54.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f50437fe700 1 -- 172.21.15.121:0/1739106944 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504801d070 con 0x7f5054105a40 2024-04-05T21:56:54.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f50437fe700 1 -- 172.21.15.121:0/1739106944 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f504800f460 con 0x7f5054105a40 2024-04-05T21:56:54.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f50437fe700 1 -- 172.21.15.121:0/1739106944 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5048021620 con 0x7f5054105a40 2024-04-05T21:56:54.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f5058e8f700 1 -- 172.21.15.121:0/1739106944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50540fcd20 con 0x7f5054105a40 2024-04-05T21:56:54.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.028+0000 7f5058e8f700 1 -- 172.21.15.121:0/1739106944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50541a2b10 con 0x7f5054105a40 2024-04-05T21:56:54.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.029+0000 7f5058e8f700 1 -- 172.21.15.121:0/1739106944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5054108fe0 con 0x7f5054105a40 2024-04-05T21:56:54.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.030+0000 7f50437fe700 1 -- 172.21.15.121:0/1739106944 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f504800f5d0 con 0x7f5054105a40 2024-04-05T21:56:54.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.030+0000 7f50437fe700 1 --2- 172.21.15.121:0/1739106944 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f503c06cce0 0x7f503c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:54.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.030+0000 7f50437fe700 1 -- 172.21.15.121:0/1739106944 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f504808d850 con 0x7f5054105a40 2024-04-05T21:56:54.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.030+0000 7f505259c700 1 --2- 172.21.15.121:0/1739106944 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f503c06cce0 0x7f503c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:54.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.031+0000 7f505259c700 1 --2- 172.21.15.121:0/1739106944 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f503c06cce0 0x7f503c06f1a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5044005950 tx=0x7f504400a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:54.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.035+0000 7f50437fe700 1 -- 172.21.15.121:0/1739106944 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f50480578c0 con 0x7f5054105a40 2024-04-05T21:56:54.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.245+0000 7f5058e8f700 1 -- 172.21.15.121:0/1739106944 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f505402cce0 con 0x7f503c06cce0 2024-04-05T21:56:54.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.246+0000 7f50437fe700 1 -- 172.21.15.121:0/1739106944 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f505402cce0 con 0x7f503c06cce0 2024-04-05T21:56:54.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.249+0000 7f5058e8f700 1 -- 172.21.15.121:0/1739106944 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f503c06cce0 msgr2=0x7f503c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:54.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.249+0000 7f5058e8f700 1 --2- 172.21.15.121:0/1739106944 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f503c06cce0 0x7f503c06f1a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5044005950 tx=0x7f504400a300 comp rx=0 tx=0).stop 2024-04-05T21:56:54.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.249+0000 7f5058e8f700 1 -- 172.21.15.121:0/1739106944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5054105a40 msgr2=0x7f50540fc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:54.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.249+0000 7f5058e8f700 1 --2- 172.21.15.121:0/1739106944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5054105a40 0x7f50540fc420 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f5048006010 tx=0x7f504800bd70 comp rx=0 tx=0).stop 2024-04-05T21:56:54.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.249+0000 7f5058e8f700 1 -- 172.21.15.121:0/1739106944 shutdown_connections 2024-04-05T21:56:54.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.249+0000 7f5058e8f700 1 --2- 172.21.15.121:0/1739106944 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f503c06cce0 0x7f503c06f1a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.249+0000 7f5058e8f700 1 --2- 172.21.15.121:0/1739106944 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f50540ffa40 0x7f50540fbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.249+0000 7f5058e8f700 1 --2- 172.21.15.121:0/1739106944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5054105a40 0x7f50540fc420 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.249+0000 7f5058e8f700 1 -- 172.21.15.121:0/1739106944 >> 172.21.15.121:0/1739106944 conn(0x7f50540fb570 msgr2=0x7f5054068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:54.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.249+0000 7f5058e8f700 1 -- 172.21.15.121:0/1739106944 shutdown_connections 2024-04-05T21:56:54.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.249+0000 7f5058e8f700 1 -- 172.21.15.121:0/1739106944 wait complete. 2024-04-05T21:56:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.351+0000 7f2a368d1700 1 -- 172.21.15.121:0/2782807594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a30105a00 msgr2=0x7f2a30105de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.351+0000 7f2a368d1700 1 --2- 172.21.15.121:0/2782807594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a30105a00 0x7f2a30105de0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009910 tx=0x7f2a24009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.351+0000 7f2a368d1700 1 -- 172.21.15.121:0/2782807594 shutdown_connections 2024-04-05T21:56:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.351+0000 7f2a368d1700 1 --2- 172.21.15.121:0/2782807594 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a300ffa00 0x7f2a300ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.351+0000 7f2a368d1700 1 --2- 172.21.15.121:0/2782807594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a30105a00 0x7f2a30105de0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.351+0000 7f2a368d1700 1 -- 172.21.15.121:0/2782807594 >> 172.21.15.121:0/2782807594 conn(0x7f2a300fb4f0 msgr2=0x7f2a300fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.351+0000 7f2a368d1700 1 -- 172.21.15.121:0/2782807594 shutdown_connections 2024-04-05T21:56:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.351+0000 7f2a368d1700 1 -- 172.21.15.121:0/2782807594 wait complete. 2024-04-05T21:56:54.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.351+0000 7f2a368d1700 1 Processor -- start 2024-04-05T21:56:54.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.351+0000 7f2a368d1700 1 -- start start 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a368d1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a300ffa00 0x7f2a30196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a368d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a30105a00 0x7f2a30196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a368d1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a30197260 con 0x7f2a30105a00 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a368d1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3019aff0 con 0x7f2a300ffa00 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a2f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a30105a00 0x7f2a30196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a2f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a30105a00 0x7f2a30196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39858/0 (socket says 172.21.15.121:39858) 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a2f7fe700 1 -- 172.21.15.121:0/3816750697 learned_addr learned my addr 172.21.15.121:0/3816750697 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a2ffff700 1 --2- 172.21.15.121:0/3816750697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a300ffa00 0x7f2a30196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a2f7fe700 1 -- 172.21.15.121:0/3816750697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a300ffa00 msgr2=0x7f2a30196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a2f7fe700 1 --2- 172.21.15.121:0/3816750697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a300ffa00 0x7f2a30196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a2f7fe700 1 -- 172.21.15.121:0/3816750697 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a240095f0 con 0x7f2a30105a00 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a2f7fe700 1 --2- 172.21.15.121:0/3816750697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a30105a00 0x7f2a30196b80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000c8a0 tx=0x7f2a2000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a2d7fa700 1 -- 172.21.15.121:0/3816750697 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a2000cea0 con 0x7f2a30105a00 2024-04-05T21:56:54.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a2d7fa700 1 -- 172.21.15.121:0/3816750697 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a20004830 con 0x7f2a30105a00 2024-04-05T21:56:54.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a368d1700 1 -- 172.21.15.121:0/3816750697 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3019b2d0 con 0x7f2a30105a00 2024-04-05T21:56:54.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a2d7fa700 1 -- 172.21.15.121:0/3816750697 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a20005500 con 0x7f2a30105a00 2024-04-05T21:56:54.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.352+0000 7f2a368d1700 1 -- 172.21.15.121:0/3816750697 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3019b820 con 0x7f2a30105a00 2024-04-05T21:56:54.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.353+0000 7f2a368d1700 1 -- 172.21.15.121:0/3816750697 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a30109010 con 0x7f2a30105a00 2024-04-05T21:56:54.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.354+0000 7f2a2d7fa700 1 -- 172.21.15.121:0/3816750697 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2a200049a0 con 0x7f2a30105a00 2024-04-05T21:56:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.354+0000 7f2a2d7fa700 1 --2- 172.21.15.121:0/3816750697 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a1806cce0 0x7f2a1806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.354+0000 7f2a2d7fa700 1 -- 172.21.15.121:0/3816750697 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2a2008c2b0 con 0x7f2a30105a00 2024-04-05T21:56:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.354+0000 7f2a2ffff700 1 --2- 172.21.15.121:0/3816750697 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a1806cce0 0x7f2a1806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.355+0000 7f2a2ffff700 1 --2- 172.21.15.121:0/3816750697 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a1806cce0 0x7f2a1806f1a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400b5c0 tx=0x7f2a24005920 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:54.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.359+0000 7f2a2d7fa700 1 -- 172.21.15.121:0/3816750697 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2a20056320 con 0x7f2a30105a00 2024-04-05T21:56:54.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.561+0000 7f2a368d1700 1 -- 172.21.15.121:0/3816750697 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a3019bb10 con 0x7f2a1806cce0 2024-04-05T21:56:54.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.571+0000 7f2a2d7fa700 1 -- 172.21.15.121:0/3816750697 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2a3019bb10 con 0x7f2a1806cce0 2024-04-05T21:56:54.571 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (27m) 6m ago 39m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (39m) 6m ago 39m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (38m) 6m ago 38m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (25m) 6m ago 39m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (25m) 6m ago 38m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (27m) 6m ago 38m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (22m) 6m ago 33m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (21m) 6m ago 32m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (21m) 6m ago 33m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (6m) 6m ago 32m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (29m) 6m ago 40m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (28m) 6m ago 38m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (26m) 6m ago 40m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:56:54.572 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (26m) 6m ago 38m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:56:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (28m) 6m ago 39m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:56:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (28m) 6m ago 38m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:56:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (25m) 6m ago 37m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:56:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (24m) 6m ago 36m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:56:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (24m) 6m ago 36m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:56:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (23m) 6m ago 35m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:56:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (23m) 6m ago 35m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:56:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (22m) 6m ago 34m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:56:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (28m) 6m ago 38m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:56:54.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.573+0000 7f2a368d1700 1 -- 172.21.15.121:0/3816750697 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a1806cce0 msgr2=0x7f2a1806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:54.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.573+0000 7f2a368d1700 1 --2- 172.21.15.121:0/3816750697 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a1806cce0 0x7f2a1806f1a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2a2400b5c0 tx=0x7f2a24005920 comp rx=0 tx=0).stop 2024-04-05T21:56:54.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.573+0000 7f2a368d1700 1 -- 172.21.15.121:0/3816750697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a30105a00 msgr2=0x7f2a30196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:54.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.573+0000 7f2a368d1700 1 --2- 172.21.15.121:0/3816750697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a30105a00 0x7f2a30196b80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000c8a0 tx=0x7f2a2000cc60 comp rx=0 tx=0).stop 2024-04-05T21:56:54.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.573+0000 7f2a368d1700 1 -- 172.21.15.121:0/3816750697 shutdown_connections 2024-04-05T21:56:54.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.573+0000 7f2a368d1700 1 --2- 172.21.15.121:0/3816750697 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a1806cce0 0x7f2a1806f1a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.573+0000 7f2a368d1700 1 --2- 172.21.15.121:0/3816750697 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a300ffa00 0x7f2a30196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.573+0000 7f2a368d1700 1 --2- 172.21.15.121:0/3816750697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a30105a00 0x7f2a30196b80 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.573+0000 7f2a368d1700 1 -- 172.21.15.121:0/3816750697 >> 172.21.15.121:0/3816750697 conn(0x7f2a300fb4f0 msgr2=0x7f2a30068f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:54.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.573+0000 7f2a368d1700 1 -- 172.21.15.121:0/3816750697 shutdown_connections 2024-04-05T21:56:54.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.573+0000 7f2a368d1700 1 -- 172.21.15.121:0/3816750697 wait complete. 2024-04-05T21:56:54.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.675+0000 7f6af7442700 1 -- 172.21.15.121:0/1482835718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6af0101770 msgr2=0x7f6af01057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:54.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.675+0000 7f6af7442700 1 --2- 172.21.15.121:0/1482835718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6af0101770 0x7f6af01057c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6aec009910 tx=0x7f6aec009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:54.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.675+0000 7f6af7442700 1 -- 172.21.15.121:0/1482835718 shutdown_connections 2024-04-05T21:56:54.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.675+0000 7f6af7442700 1 --2- 172.21.15.121:0/1482835718 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6af0101770 0x7f6af01057c0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.675+0000 7f6af7442700 1 --2- 172.21.15.121:0/1482835718 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6af0100dc0 0x7f6af01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.675+0000 7f6af7442700 1 -- 172.21.15.121:0/1482835718 >> 172.21.15.121:0/1482835718 conn(0x7f6af00fc650 msgr2=0x7f6af00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:54.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.675+0000 7f6af7442700 1 -- 172.21.15.121:0/1482835718 shutdown_connections 2024-04-05T21:56:54.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.675+0000 7f6af7442700 1 -- 172.21.15.121:0/1482835718 wait complete. 2024-04-05T21:56:54.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.675+0000 7f6af7442700 1 Processor -- start 2024-04-05T21:56:54.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af7442700 1 -- start start 2024-04-05T21:56:54.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af7442700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6af0100dc0 0x7f6af0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:54.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af7442700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6af0101770 0x7f6af00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af7442700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af0073b10 con 0x7f6af0100dc0 2024-04-05T21:56:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af7442700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af0073c50 con 0x7f6af0101770 2024-04-05T21:56:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af51de700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6af0100dc0 0x7f6af0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af51de700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6af0100dc0 0x7f6af0073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39874/0 (socket says 172.21.15.121:39874) 2024-04-05T21:56:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af51de700 1 -- 172.21.15.121:0/3623401989 learned_addr learned my addr 172.21.15.121:0/3623401989 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af49dd700 1 --2- 172.21.15.121:0/3623401989 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6af0101770 0x7f6af00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af51de700 1 -- 172.21.15.121:0/3623401989 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6af0101770 msgr2=0x7f6af00735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af51de700 1 --2- 172.21.15.121:0/3623401989 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6af0101770 0x7f6af00735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af51de700 1 -- 172.21.15.121:0/3623401989 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aec0095f0 con 0x7f6af0100dc0 2024-04-05T21:56:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af51de700 1 --2- 172.21.15.121:0/3623401989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6af0100dc0 0x7f6af0073090 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000d8d0 tx=0x7f6ae000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6ae67fc700 1 -- 172.21.15.121:0/3623401989 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ae00098a0 con 0x7f6af0100dc0 2024-04-05T21:56:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6af7442700 1 -- 172.21.15.121:0/3623401989 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af01a28a0 con 0x7f6af0100dc0 2024-04-05T21:56:54.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6ae67fc700 1 -- 172.21.15.121:0/3623401989 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ae0010460 con 0x7f6af0100dc0 2024-04-05T21:56:54.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.676+0000 7f6ae67fc700 1 -- 172.21.15.121:0/3623401989 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ae0009ea0 con 0x7f6af0100dc0 2024-04-05T21:56:54.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.677+0000 7f6af7442700 1 -- 172.21.15.121:0/3623401989 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af01a2cb0 con 0x7f6af0100dc0 2024-04-05T21:56:54.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.678+0000 7f6af7442700 1 -- 172.21.15.121:0/3623401989 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af01091c0 con 0x7f6af0100dc0 2024-04-05T21:56:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.678+0000 7f6ae67fc700 1 -- 172.21.15.121:0/3623401989 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6ae00105d0 con 0x7f6af0100dc0 2024-04-05T21:56:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.678+0000 7f6ae67fc700 1 --2- 172.21.15.121:0/3623401989 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6adc06cce0 0x7f6adc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.679+0000 7f6ae67fc700 1 -- 172.21.15.121:0/3623401989 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6ae008c480 con 0x7f6af0100dc0 2024-04-05T21:56:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.679+0000 7f6af49dd700 1 --2- 172.21.15.121:0/3623401989 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6adc06cce0 0x7f6adc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:54.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.679+0000 7f6af49dd700 1 --2- 172.21.15.121:0/3623401989 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6adc06cce0 0x7f6adc06f1a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6af0074880 tx=0x7f6aec01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:54.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.683+0000 7f6ae67fc700 1 -- 172.21.15.121:0/3623401989 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6ae00564f0 con 0x7f6af0100dc0 2024-04-05T21:56:54.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.956+0000 7f6af7442700 1 -- 172.21.15.121:0/3623401989 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6af004ea90 con 0x7f6af0100dc0 2024-04-05T21:56:54.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.957+0000 7f6ae67fc700 1 -- 172.21.15.121:0/3623401989 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f6ae0010880 con 0x7f6af0100dc0 2024-04-05T21:56:54.957 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:56:54.958 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:56:54.959 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:56:54.959 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:56:54.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.959+0000 7f6af7442700 1 -- 172.21.15.121:0/3623401989 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6adc06cce0 msgr2=0x7f6adc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:54.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.959+0000 7f6af7442700 1 --2- 172.21.15.121:0/3623401989 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6adc06cce0 0x7f6adc06f1a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6af0074880 tx=0x7f6aec01a040 comp rx=0 tx=0).stop 2024-04-05T21:56:54.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.959+0000 7f6af7442700 1 -- 172.21.15.121:0/3623401989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6af0100dc0 msgr2=0x7f6af0073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:54.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.959+0000 7f6af7442700 1 --2- 172.21.15.121:0/3623401989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6af0100dc0 0x7f6af0073090 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000d8d0 tx=0x7f6ae000dbe0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.959+0000 7f6af7442700 1 -- 172.21.15.121:0/3623401989 shutdown_connections 2024-04-05T21:56:54.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.959+0000 7f6af7442700 1 --2- 172.21.15.121:0/3623401989 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6adc06cce0 0x7f6adc06f1a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.959+0000 7f6af7442700 1 --2- 172.21.15.121:0/3623401989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6af0100dc0 0x7f6af0073090 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.959+0000 7f6af7442700 1 --2- 172.21.15.121:0/3623401989 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6af0101770 0x7f6af00735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:54.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.959+0000 7f6af7442700 1 -- 172.21.15.121:0/3623401989 >> 172.21.15.121:0/3623401989 conn(0x7f6af00fc650 msgr2=0x7f6af00fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:54.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.959+0000 7f6af7442700 1 -- 172.21.15.121:0/3623401989 shutdown_connections 2024-04-05T21:56:54.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:54.959+0000 7f6af7442700 1 -- 172.21.15.121:0/3623401989 wait complete. 2024-04-05T21:56:55.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.061+0000 7f6439a04700 1 -- 172.21.15.121:0/1477973249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6434068670 msgr2=0x7f6434068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:55.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.061+0000 7f6439a04700 1 --2- 172.21.15.121:0/1477973249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6434068670 0x7f6434068a50 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6424009910 tx=0x7f6424009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:55.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.061+0000 7f6439a04700 1 -- 172.21.15.121:0/1477973249 shutdown_connections 2024-04-05T21:56:55.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.061+0000 7f6439a04700 1 --2- 172.21.15.121:0/1477973249 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6434069020 0x7f6434105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.061+0000 7f6439a04700 1 --2- 172.21.15.121:0/1477973249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6434068670 0x7f6434068a50 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.061+0000 7f6439a04700 1 -- 172.21.15.121:0/1477973249 >> 172.21.15.121:0/1477973249 conn(0x7f64340755a0 msgr2=0x7f64340759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:55.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.061+0000 7f6439a04700 1 -- 172.21.15.121:0/1477973249 shutdown_connections 2024-04-05T21:56:55.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.061+0000 7f6439a04700 1 -- 172.21.15.121:0/1477973249 wait complete. 2024-04-05T21:56:55.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.062+0000 7f6439a04700 1 Processor -- start 2024-04-05T21:56:55.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.062+0000 7f6439a04700 1 -- start start 2024-04-05T21:56:55.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.062+0000 7f6439a04700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6434068670 0x7f6434196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:55.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.062+0000 7f6439a04700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6434069020 0x7f6434196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:55.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.062+0000 7f6439a04700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6434197200 con 0x7f6434069020 2024-04-05T21:56:55.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.062+0000 7f6439a04700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6434197340 con 0x7f6434068670 2024-04-05T21:56:55.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.062+0000 7f64327fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6434069020 0x7f6434196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:55.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.062+0000 7f64327fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6434069020 0x7f6434196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39890/0 (socket says 172.21.15.121:39890) 2024-04-05T21:56:55.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.063+0000 7f64327fc700 1 -- 172.21.15.121:0/3162498545 learned_addr learned my addr 172.21.15.121:0/3162498545 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:55.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.063+0000 7f6432ffd700 1 --2- 172.21.15.121:0/3162498545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6434068670 0x7f6434196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:55.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.063+0000 7f64327fc700 1 -- 172.21.15.121:0/3162498545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6434068670 msgr2=0x7f6434196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:55.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.063+0000 7f64327fc700 1 --2- 172.21.15.121:0/3162498545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6434068670 0x7f6434196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.063+0000 7f64327fc700 1 -- 172.21.15.121:0/3162498545 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64240095f0 con 0x7f6434069020 2024-04-05T21:56:55.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.063+0000 7f6432ffd700 1 --2- 172.21.15.121:0/3162498545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6434068670 0x7f6434196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:56:55.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.063+0000 7f64327fc700 1 --2- 172.21.15.121:0/3162498545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6434069020 0x7f6434196bb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f642800c960 tx=0x7f642800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:55.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.063+0000 7f6438a02700 1 -- 172.21.15.121:0/3162498545 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6428007a70 con 0x7f6434069020 2024-04-05T21:56:55.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.063+0000 7f6438a02700 1 -- 172.21.15.121:0/3162498545 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6428007bd0 con 0x7f6434069020 2024-04-05T21:56:55.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.063+0000 7f6439a04700 1 -- 172.21.15.121:0/3162498545 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643419b190 con 0x7f6434069020 2024-04-05T21:56:55.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.063+0000 7f6438a02700 1 -- 172.21.15.121:0/3162498545 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6428018630 con 0x7f6434069020 2024-04-05T21:56:55.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.063+0000 7f6439a04700 1 -- 172.21.15.121:0/3162498545 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643419b6e0 con 0x7f6434069020 2024-04-05T21:56:55.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.064+0000 7f6439a04700 1 -- 172.21.15.121:0/3162498545 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6434109130 con 0x7f6434069020 2024-04-05T21:56:55.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.065+0000 7f6438a02700 1 -- 172.21.15.121:0/3162498545 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f642800f9f0 con 0x7f6434069020 2024-04-05T21:56:55.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.065+0000 7f6438a02700 1 --2- 172.21.15.121:0/3162498545 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f641c06cce0 0x7f641c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:55.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.065+0000 7f6438a02700 1 -- 172.21.15.121:0/3162498545 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f642808c5e0 con 0x7f6434069020 2024-04-05T21:56:55.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.065+0000 7f6432ffd700 1 --2- 172.21.15.121:0/3162498545 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f641c06cce0 0x7f641c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:55.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.065+0000 7f6432ffd700 1 --2- 172.21.15.121:0/3162498545 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f641c06cce0 0x7f641c06f1a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6424006010 tx=0x7f642401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:55.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.069+0000 7f6438a02700 1 -- 172.21.15.121:0/3162498545 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f642805a0e0 con 0x7f6434069020 2024-04-05T21:56:55.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:54 smithi121 ceph-mon[145859]: pgmap v921: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:55.123 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:54 smithi121 ceph-mon[145859]: from='client.35494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:54 smithi188 ceph-mon[107793]: pgmap v921: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:54 smithi188 ceph-mon[107793]: from='client.35494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:55.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.304+0000 7f6439a04700 1 -- 172.21.15.121:0/3162498545 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f643404ea90 con 0x7f6434069020 2024-04-05T21:56:55.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.304+0000 7f6438a02700 1 -- 172.21.15.121:0/3162498545 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f6428059c70 con 0x7f6434069020 2024-04-05T21:56:55.305 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:56:55.305 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:56:55.305 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:56:55.305 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:56:55.305 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:56:55.305 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:56:55.305 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:56:55.305 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:56:55.306 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:56:55.307 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:56:55.308 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:56:55.308 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:56:55.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.306+0000 7f6439a04700 1 -- 172.21.15.121:0/3162498545 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f641c06cce0 msgr2=0x7f641c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:55.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.306+0000 7f6439a04700 1 --2- 172.21.15.121:0/3162498545 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f641c06cce0 0x7f641c06f1a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6424006010 tx=0x7f642401a040 comp rx=0 tx=0).stop 2024-04-05T21:56:55.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.306+0000 7f6439a04700 1 -- 172.21.15.121:0/3162498545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6434069020 msgr2=0x7f6434196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:55.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.306+0000 7f6439a04700 1 --2- 172.21.15.121:0/3162498545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6434069020 0x7f6434196bb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f642800c960 tx=0x7f642800cd20 comp rx=0 tx=0).stop 2024-04-05T21:56:55.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.307+0000 7f6439a04700 1 -- 172.21.15.121:0/3162498545 shutdown_connections 2024-04-05T21:56:55.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.307+0000 7f6439a04700 1 --2- 172.21.15.121:0/3162498545 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f641c06cce0 0x7f641c06f1a0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.307+0000 7f6439a04700 1 --2- 172.21.15.121:0/3162498545 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6434068670 0x7f6434196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.307+0000 7f6439a04700 1 --2- 172.21.15.121:0/3162498545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6434069020 0x7f6434196bb0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.307+0000 7f6439a04700 1 -- 172.21.15.121:0/3162498545 >> 172.21.15.121:0/3162498545 conn(0x7f64340755a0 msgr2=0x7f64340fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:55.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.307+0000 7f6439a04700 1 -- 172.21.15.121:0/3162498545 shutdown_connections 2024-04-05T21:56:55.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.307+0000 7f6439a04700 1 -- 172.21.15.121:0/3162498545 wait complete. 2024-04-05T21:56:55.309 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:56:55.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.410+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/4003909642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fac100da0 msgr2=0x7f7fac101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:55.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.410+0000 7f7fb0f3f700 1 --2- 172.21.15.121:0/4003909642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fac100da0 0x7f7fac101180 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7f94009910 tx=0x7f7f94009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:55.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.410+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/4003909642 shutdown_connections 2024-04-05T21:56:55.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.410+0000 7f7fb0f3f700 1 --2- 172.21.15.121:0/4003909642 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fac101750 0x7f7fac1057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.410+0000 7f7fb0f3f700 1 --2- 172.21.15.121:0/4003909642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fac100da0 0x7f7fac101180 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.410+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/4003909642 >> 172.21.15.121:0/4003909642 conn(0x7f7fac0fc630 msgr2=0x7f7fac0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:55.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.410+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/4003909642 shutdown_connections 2024-04-05T21:56:55.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.410+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/4003909642 wait complete. 2024-04-05T21:56:55.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.411+0000 7f7fb0f3f700 1 Processor -- start 2024-04-05T21:56:55.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.411+0000 7f7fb0f3f700 1 -- start start 2024-04-05T21:56:55.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.411+0000 7f7fb0f3f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fac100da0 0x7f7fac196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:55.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.411+0000 7f7fb0f3f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fac101750 0x7f7fac196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:55.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.411+0000 7f7fb0f3f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1971d0 con 0x7f7fac101750 2024-04-05T21:56:55.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.411+0000 7f7fb0f3f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac197340 con 0x7f7fac100da0 2024-04-05T21:56:55.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.412+0000 7f7fa9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fac101750 0x7f7fac196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.412+0000 7f7fa9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fac101750 0x7f7fac196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39908/0 (socket says 172.21.15.121:39908) 2024-04-05T21:56:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.412+0000 7f7fa9d9b700 1 -- 172.21.15.121:0/2181728026 learned_addr learned my addr 172.21.15.121:0/2181728026 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.412+0000 7f7faa59c700 1 --2- 172.21.15.121:0/2181728026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fac100da0 0x7f7fac196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.412+0000 7f7fa9d9b700 1 -- 172.21.15.121:0/2181728026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fac100da0 msgr2=0x7f7fac196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.412+0000 7f7fa9d9b700 1 --2- 172.21.15.121:0/2181728026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fac100da0 0x7f7fac196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.412+0000 7f7fa9d9b700 1 -- 172.21.15.121:0/2181728026 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f940095f0 con 0x7f7fac101750 2024-04-05T21:56:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.412+0000 7f7fa9d9b700 1 --2- 172.21.15.121:0/2181728026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fac101750 0x7f7fac196bd0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00c960 tx=0x7f7f9c00cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.412+0000 7f7fa37fe700 1 -- 172.21.15.121:0/2181728026 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f9c007a70 con 0x7f7fac101750 2024-04-05T21:56:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.412+0000 7f7fa37fe700 1 -- 172.21.15.121:0/2181728026 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f9c007bd0 con 0x7f7fac101750 2024-04-05T21:56:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.412+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/2181728026 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac19a0e0 con 0x7f7fac101750 2024-04-05T21:56:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.412+0000 7f7fa37fe700 1 -- 172.21.15.121:0/2181728026 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f9c018630 con 0x7f7fac101750 2024-04-05T21:56:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.412+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/2181728026 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac19a630 con 0x7f7fac101750 2024-04-05T21:56:55.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.413+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/2181728026 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fac109140 con 0x7f7fac101750 2024-04-05T21:56:55.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.414+0000 7f7fa37fe700 1 -- 172.21.15.121:0/2181728026 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7f9c00f9f0 con 0x7f7fac101750 2024-04-05T21:56:55.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.414+0000 7f7fa37fe700 1 --2- 172.21.15.121:0/2181728026 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7f9806cce0 0x7f7f9806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:55.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.414+0000 7f7fa37fe700 1 -- 172.21.15.121:0/2181728026 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7f9c08c750 con 0x7f7fac101750 2024-04-05T21:56:55.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.414+0000 7f7faa59c700 1 --2- 172.21.15.121:0/2181728026 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7f9806cce0 0x7f7f9806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:55.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.415+0000 7f7faa59c700 1 --2- 172.21.15.121:0/2181728026 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7f9806cce0 0x7f7f9806f1a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400b5c0 tx=0x7f7f94005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:55.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.418+0000 7f7fa37fe700 1 -- 172.21.15.121:0/2181728026 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7f9c05a370 con 0x7f7fac101750 2024-04-05T21:56:55.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.636+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/2181728026 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fac02cdb0 con 0x7f7f9806cce0 2024-04-05T21:56:55.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.637+0000 7f7fa37fe700 1 -- 172.21.15.121:0/2181728026 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7fac02cdb0 con 0x7f7f9806cce0 2024-04-05T21:56:55.638 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:56:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:56:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:56:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:56:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:56:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:56:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:56:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:56:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:56:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:56:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:56:55.638 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:56:55.639 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:56:55.639 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:56:55.639 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:56:55.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.639+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/2181728026 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7f9806cce0 msgr2=0x7f7f9806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:55.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.639+0000 7f7fb0f3f700 1 --2- 172.21.15.121:0/2181728026 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7f9806cce0 0x7f7f9806f1a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400b5c0 tx=0x7f7f94005fb0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.639+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/2181728026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fac101750 msgr2=0x7f7fac196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:55.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.639+0000 7f7fb0f3f700 1 --2- 172.21.15.121:0/2181728026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fac101750 0x7f7fac196bd0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00c960 tx=0x7f7f9c00cd20 comp rx=0 tx=0).stop 2024-04-05T21:56:55.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.640+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/2181728026 shutdown_connections 2024-04-05T21:56:55.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.640+0000 7f7fb0f3f700 1 --2- 172.21.15.121:0/2181728026 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7f9806cce0 0x7f7f9806f1a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.640+0000 7f7fb0f3f700 1 --2- 172.21.15.121:0/2181728026 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7fac100da0 0x7f7fac196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.640+0000 7f7fb0f3f700 1 --2- 172.21.15.121:0/2181728026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fac101750 0x7f7fac196bd0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.640+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/2181728026 >> 172.21.15.121:0/2181728026 conn(0x7f7fac0fc630 msgr2=0x7f7fac0fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:55.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.640+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/2181728026 shutdown_connections 2024-04-05T21:56:55.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.640+0000 7f7fb0f3f700 1 -- 172.21.15.121:0/2181728026 wait complete. 2024-04-05T21:56:55.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.742+0000 7f7d8863a700 1 -- 172.21.15.121:0/886111582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d80105a40 msgr2=0x7f7d80105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:55.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.742+0000 7f7d8863a700 1 --2- 172.21.15.121:0/886111582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d80105a40 0x7f7d80105e20 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f7d70009910 tx=0x7f7d70009c20 comp rx=0 tx=0).stop 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.742+0000 7f7d8863a700 1 -- 172.21.15.121:0/886111582 shutdown_connections 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.742+0000 7f7d8863a700 1 --2- 172.21.15.121:0/886111582 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d800ffbd0 0x7f7d80100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.742+0000 7f7d8863a700 1 --2- 172.21.15.121:0/886111582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d80105a40 0x7f7d80105e20 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.742+0000 7f7d8863a700 1 -- 172.21.15.121:0/886111582 >> 172.21.15.121:0/886111582 conn(0x7f7d800755a0 msgr2=0x7f7d800759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.742+0000 7f7d8863a700 1 -- 172.21.15.121:0/886111582 shutdown_connections 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.742+0000 7f7d8863a700 1 -- 172.21.15.121:0/886111582 wait complete. 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.742+0000 7f7d8863a700 1 Processor -- start 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.742+0000 7f7d8863a700 1 -- start start 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.742+0000 7f7d8863a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d800ffbd0 0x7f7d80196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d8863a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d80105a40 0x7f7d80196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d8863a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80197290 con 0x7f7d800ffbd0 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d8863a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8019afb0 con 0x7f7d80105a40 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d863d6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d800ffbd0 0x7f7d80196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:55.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d863d6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d800ffbd0 0x7f7d80196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39930/0 (socket says 172.21.15.121:39930) 2024-04-05T21:56:55.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d863d6700 1 -- 172.21.15.121:0/3425217996 learned_addr learned my addr 172.21.15.121:0/3425217996 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:56:55.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d85bd5700 1 --2- 172.21.15.121:0/3425217996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d80105a40 0x7f7d80196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:55.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d863d6700 1 -- 172.21.15.121:0/3425217996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d80105a40 msgr2=0x7f7d80196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:55.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d863d6700 1 --2- 172.21.15.121:0/3425217996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d80105a40 0x7f7d80196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:55.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d863d6700 1 -- 172.21.15.121:0/3425217996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d700095f0 con 0x7f7d800ffbd0 2024-04-05T21:56:55.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d863d6700 1 --2- 172.21.15.121:0/3425217996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d800ffbd0 0x7f7d80196670 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7d70006010 tx=0x7f7d7000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:55.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d777fe700 1 -- 172.21.15.121:0/3425217996 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d7001d070 con 0x7f7d800ffbd0 2024-04-05T21:56:55.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d777fe700 1 -- 172.21.15.121:0/3425217996 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d7000f460 con 0x7f7d800ffbd0 2024-04-05T21:56:55.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d8863a700 1 -- 172.21.15.121:0/3425217996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8019b230 con 0x7f7d800ffbd0 2024-04-05T21:56:55.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d777fe700 1 -- 172.21.15.121:0/3425217996 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d70021620 con 0x7f7d800ffbd0 2024-04-05T21:56:55.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.743+0000 7f7d8863a700 1 -- 172.21.15.121:0/3425217996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8019b720 con 0x7f7d800ffbd0 2024-04-05T21:56:55.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.744+0000 7f7d8863a700 1 -- 172.21.15.121:0/3425217996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8004ea90 con 0x7f7d800ffbd0 2024-04-05T21:56:55.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.745+0000 7f7d777fe700 1 -- 172.21.15.121:0/3425217996 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7d7000fa40 con 0x7f7d800ffbd0 2024-04-05T21:56:55.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.745+0000 7f7d777fe700 1 --2- 172.21.15.121:0/3425217996 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7d6c06cce0 0x7f7d6c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:56:55.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.745+0000 7f7d777fe700 1 -- 172.21.15.121:0/3425217996 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7d7008d6c0 con 0x7f7d800ffbd0 2024-04-05T21:56:55.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.745+0000 7f7d85bd5700 1 --2- 172.21.15.121:0/3425217996 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7d6c06cce0 0x7f7d6c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:56:55.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.745+0000 7f7d85bd5700 1 --2- 172.21.15.121:0/3425217996 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7d6c06cce0 0x7f7d6c06f1a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c005950 tx=0x7f7d7c0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:56:55.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:55.750+0000 7f7d777fe700 1 -- 172.21.15.121:0/3425217996 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7d700577b0 con 0x7f7d800ffbd0 2024-04-05T21:56:56.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:56.019+0000 7f7d8863a700 1 -- 172.21.15.121:0/3425217996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7d8019bbd0 con 0x7f7d800ffbd0 2024-04-05T21:56:56.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:56.019+0000 7f7d777fe700 1 -- 172.21.15.121:0/3425217996 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f7d70026e10 con 0x7f7d800ffbd0 2024-04-05T21:56:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:56:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:56:56.020 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:56:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:56:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:56:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:56:56.020 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:56:56.020 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:56:56.020 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:56:56.020 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:56:56.020 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:56:56.021 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:56:56.021 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:56:56.021 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:56:56.021 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:56:56.021 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:56:56.021 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:56:56.021 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:56:56.021 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:56:56.021 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:56:56.021 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:56:56.021 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:56:56.021 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:56:56.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:56.021+0000 7f7d8863a700 1 -- 172.21.15.121:0/3425217996 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7d6c06cce0 msgr2=0x7f7d6c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:56.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:56.021+0000 7f7d8863a700 1 --2- 172.21.15.121:0/3425217996 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7d6c06cce0 0x7f7d6c06f1a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c005950 tx=0x7f7d7c0058e0 comp rx=0 tx=0).stop 2024-04-05T21:56:56.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:56.021+0000 7f7d8863a700 1 -- 172.21.15.121:0/3425217996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d800ffbd0 msgr2=0x7f7d80196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:56:56.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:56.021+0000 7f7d8863a700 1 --2- 172.21.15.121:0/3425217996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d800ffbd0 0x7f7d80196670 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7d70006010 tx=0x7f7d7000be20 comp rx=0 tx=0).stop 2024-04-05T21:56:56.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:56.021+0000 7f7d8863a700 1 -- 172.21.15.121:0/3425217996 shutdown_connections 2024-04-05T21:56:56.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:56.021+0000 7f7d8863a700 1 --2- 172.21.15.121:0/3425217996 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7d6c06cce0 0x7f7d6c06f1a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:56.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:56.021+0000 7f7d8863a700 1 --2- 172.21.15.121:0/3425217996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d800ffbd0 0x7f7d80196670 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:56.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:56.021+0000 7f7d8863a700 1 --2- 172.21.15.121:0/3425217996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7d80105a40 0x7f7d80196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:56:56.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:56.021+0000 7f7d8863a700 1 -- 172.21.15.121:0/3425217996 >> 172.21.15.121:0/3425217996 conn(0x7f7d800755a0 msgr2=0x7f7d801037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:56:56.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:56.021+0000 7f7d8863a700 1 -- 172.21.15.121:0/3425217996 shutdown_connections 2024-04-05T21:56:56.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:56:56.021+0000 7f7d8863a700 1 -- 172.21.15.121:0/3425217996 wait complete. 2024-04-05T21:56:56.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:55 smithi121 ceph-mon[145859]: from='client.35498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:56.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:55 smithi121 ceph-mon[145859]: from='client.35502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:56.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:55 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3623401989' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:56:56.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:55 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3162498545' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:56:56.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:55 smithi188 ceph-mon[107793]: from='client.35498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:56.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:55 smithi188 ceph-mon[107793]: from='client.35502 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:56.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:55 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3623401989' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:56:56.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:55 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3162498545' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:56:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:56 smithi121 ceph-mon[145859]: from='client.35514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:57.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:56 smithi121 ceph-mon[145859]: pgmap v922: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:57.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:56 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3425217996' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:56:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:56 smithi188 ceph-mon[107793]: from='client.35514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:56:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:56 smithi188 ceph-mon[107793]: pgmap v922: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:56 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3425217996' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:56:59.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:56:58 smithi121 ceph-mon[145859]: pgmap v923: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:56:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:56:58 smithi188 ceph-mon[107793]: pgmap v923: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:00 smithi121 ceph-mon[145859]: pgmap v924: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:57:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:00 smithi188 ceph-mon[107793]: pgmap v924: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:57:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:02 smithi121 ceph-mon[145859]: pgmap v925: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:02 smithi188 ceph-mon[107793]: pgmap v925: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:05.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:04 smithi121 ceph-mon[145859]: pgmap v926: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:04 smithi188 ceph-mon[107793]: pgmap v926: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:07.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:06 smithi121 ceph-mon[145859]: pgmap v927: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:06 smithi188 ceph-mon[107793]: pgmap v927: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:09.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:08 smithi121 ceph-mon[145859]: pgmap v928: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:09.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:08 smithi188 ceph-mon[107793]: pgmap v928: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:11.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:10 smithi121 ceph-mon[145859]: pgmap v929: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:10 smithi188 ceph-mon[107793]: pgmap v929: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:13.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:12 smithi121 ceph-mon[145859]: pgmap v930: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:12 smithi188 ceph-mon[107793]: pgmap v930: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:15.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:14 smithi121 ceph-mon[145859]: pgmap v931: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:14 smithi188 ceph-mon[107793]: pgmap v931: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:57:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:57:17.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:16 smithi121 ceph-mon[145859]: pgmap v932: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:17.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:16 smithi188 ceph-mon[107793]: pgmap v932: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:19.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:18 smithi121 ceph-mon[145859]: pgmap v933: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:19.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:18 smithi188 ceph-mon[107793]: pgmap v933: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:21.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:20 smithi121 ceph-mon[145859]: pgmap v934: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:20 smithi188 ceph-mon[107793]: pgmap v934: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:23.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:22 smithi121 ceph-mon[145859]: pgmap v935: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:23.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:22 smithi188 ceph-mon[107793]: pgmap v935: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:25.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:24 smithi121 ceph-mon[145859]: pgmap v936: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:24 smithi188 ceph-mon[107793]: pgmap v936: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:26.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.137+0000 7f2a6db44700 1 -- 172.21.15.121:0/1100296648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a68068670 msgr2=0x7f2a68068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:26.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.137+0000 7f2a6db44700 1 --2- 172.21.15.121:0/1100296648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a68068670 0x7f2a68068a50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2a58009910 tx=0x7f2a58009c20 comp rx=0 tx=0).stop 2024-04-05T21:57:26.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.137+0000 7f2a6db44700 1 -- 172.21.15.121:0/1100296648 shutdown_connections 2024-04-05T21:57:26.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.137+0000 7f2a6db44700 1 --2- 172.21.15.121:0/1100296648 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a68069020 0x7f2a68105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.137+0000 7f2a6db44700 1 --2- 172.21.15.121:0/1100296648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a68068670 0x7f2a68068a50 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.137+0000 7f2a6db44700 1 -- 172.21.15.121:0/1100296648 >> 172.21.15.121:0/1100296648 conn(0x7f2a680755a0 msgr2=0x7f2a680759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:26.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.137+0000 7f2a6db44700 1 -- 172.21.15.121:0/1100296648 shutdown_connections 2024-04-05T21:57:26.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.137+0000 7f2a6db44700 1 -- 172.21.15.121:0/1100296648 wait complete. 2024-04-05T21:57:26.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.138+0000 7f2a6db44700 1 Processor -- start 2024-04-05T21:57:26.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.138+0000 7f2a6db44700 1 -- start start 2024-04-05T21:57:26.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.138+0000 7f2a6db44700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a68068670 0x7f2a68196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:26.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.138+0000 7f2a6db44700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a68069020 0x7f2a68196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:26.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.138+0000 7f2a6db44700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a68197200 con 0x7f2a68069020 2024-04-05T21:57:26.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.138+0000 7f2a6db44700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a68197340 con 0x7f2a68068670 2024-04-05T21:57:26.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a5effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a68069020 0x7f2a68196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:26.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a5effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a68069020 0x7f2a68196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50722/0 (socket says 172.21.15.121:50722) 2024-04-05T21:57:26.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a5effd700 1 -- 172.21.15.121:0/2051575100 learned_addr learned my addr 172.21.15.121:0/2051575100 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:57:26.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a677fe700 1 --2- 172.21.15.121:0/2051575100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a68068670 0x7f2a68196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:26.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a5effd700 1 -- 172.21.15.121:0/2051575100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a68068670 msgr2=0x7f2a68196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:26.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a5effd700 1 --2- 172.21.15.121:0/2051575100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a68068670 0x7f2a68196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a5effd700 1 -- 172.21.15.121:0/2051575100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a580095f0 con 0x7f2a68069020 2024-04-05T21:57:26.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a677fe700 1 --2- 172.21.15.121:0/2051575100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a68068670 0x7f2a68196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:57:26.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a5effd700 1 --2- 172.21.15.121:0/2051575100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a68069020 0x7f2a68196bb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000c960 tx=0x7f2a5000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:26.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a657fa700 1 -- 172.21.15.121:0/2051575100 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a50007a70 con 0x7f2a68069020 2024-04-05T21:57:26.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a657fa700 1 -- 172.21.15.121:0/2051575100 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a50007bd0 con 0x7f2a68069020 2024-04-05T21:57:26.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a6db44700 1 -- 172.21.15.121:0/2051575100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a6819b190 con 0x7f2a68069020 2024-04-05T21:57:26.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a657fa700 1 -- 172.21.15.121:0/2051575100 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a50018630 con 0x7f2a68069020 2024-04-05T21:57:26.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.139+0000 7f2a6db44700 1 -- 172.21.15.121:0/2051575100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6819b6e0 con 0x7f2a68069020 2024-04-05T21:57:26.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.140+0000 7f2a6db44700 1 -- 172.21.15.121:0/2051575100 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a68109130 con 0x7f2a68069020 2024-04-05T21:57:26.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.141+0000 7f2a657fa700 1 -- 172.21.15.121:0/2051575100 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2a5000f450 con 0x7f2a68069020 2024-04-05T21:57:26.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.141+0000 7f2a657fa700 1 --2- 172.21.15.121:0/2051575100 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a5406cce0 0x7f2a5406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:26.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.141+0000 7f2a657fa700 1 -- 172.21.15.121:0/2051575100 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2a5008c550 con 0x7f2a68069020 2024-04-05T21:57:26.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.141+0000 7f2a677fe700 1 --2- 172.21.15.121:0/2051575100 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a5406cce0 0x7f2a5406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:26.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.142+0000 7f2a677fe700 1 --2- 172.21.15.121:0/2051575100 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a5406cce0 0x7f2a5406f1a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2a5800b5c0 tx=0x7f2a58005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:26.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.146+0000 7f2a657fa700 1 -- 172.21.15.121:0/2051575100 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2a5005a180 con 0x7f2a68069020 2024-04-05T21:57:26.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.355+0000 7f2a6db44700 1 -- 172.21.15.121:0/2051575100 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a6802cd60 con 0x7f2a5406cce0 2024-04-05T21:57:26.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.356+0000 7f2a657fa700 1 -- 172.21.15.121:0/2051575100 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2a6802cd60 con 0x7f2a5406cce0 2024-04-05T21:57:26.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.358+0000 7f2a6db44700 1 -- 172.21.15.121:0/2051575100 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a5406cce0 msgr2=0x7f2a5406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:26.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.358+0000 7f2a6db44700 1 --2- 172.21.15.121:0/2051575100 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a5406cce0 0x7f2a5406f1a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2a5800b5c0 tx=0x7f2a58005960 comp rx=0 tx=0).stop 2024-04-05T21:57:26.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.358+0000 7f2a6db44700 1 -- 172.21.15.121:0/2051575100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a68069020 msgr2=0x7f2a68196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:26.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.358+0000 7f2a6db44700 1 --2- 172.21.15.121:0/2051575100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a68069020 0x7f2a68196bb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000c960 tx=0x7f2a5000cd20 comp rx=0 tx=0).stop 2024-04-05T21:57:26.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.358+0000 7f2a6db44700 1 -- 172.21.15.121:0/2051575100 shutdown_connections 2024-04-05T21:57:26.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.358+0000 7f2a6db44700 1 --2- 172.21.15.121:0/2051575100 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2a5406cce0 0x7f2a5406f1a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.358+0000 7f2a6db44700 1 --2- 172.21.15.121:0/2051575100 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2a68068670 0x7f2a68196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.358+0000 7f2a6db44700 1 --2- 172.21.15.121:0/2051575100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a68069020 0x7f2a68196bb0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.358+0000 7f2a6db44700 1 -- 172.21.15.121:0/2051575100 >> 172.21.15.121:0/2051575100 conn(0x7f2a680755a0 msgr2=0x7f2a680fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:26.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.358+0000 7f2a6db44700 1 -- 172.21.15.121:0/2051575100 shutdown_connections 2024-04-05T21:57:26.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.358+0000 7f2a6db44700 1 -- 172.21.15.121:0/2051575100 wait complete. 2024-04-05T21:57:26.369 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:57:26.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.461+0000 7ff299a59700 1 -- 172.21.15.121:0/3082436883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff294101770 msgr2=0x7ff2941057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:26.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.461+0000 7ff299a59700 1 --2- 172.21.15.121:0/3082436883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff294101770 0x7ff2941057c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff288009910 tx=0x7ff288009c20 comp rx=0 tx=0).stop 2024-04-05T21:57:26.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.461+0000 7ff299a59700 1 -- 172.21.15.121:0/3082436883 shutdown_connections 2024-04-05T21:57:26.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.461+0000 7ff299a59700 1 --2- 172.21.15.121:0/3082436883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff294101770 0x7ff2941057c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.461+0000 7ff299a59700 1 --2- 172.21.15.121:0/3082436883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff294100dc0 0x7ff2941011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.461+0000 7ff299a59700 1 -- 172.21.15.121:0/3082436883 >> 172.21.15.121:0/3082436883 conn(0x7ff2940fc650 msgr2=0x7ff2940fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:26.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.461+0000 7ff299a59700 1 -- 172.21.15.121:0/3082436883 shutdown_connections 2024-04-05T21:57:26.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.461+0000 7ff299a59700 1 -- 172.21.15.121:0/3082436883 wait complete. 2024-04-05T21:57:26.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.462+0000 7ff299a59700 1 Processor -- start 2024-04-05T21:57:26.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.462+0000 7ff299a59700 1 -- start start 2024-04-05T21:57:26.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff299a59700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff294100dc0 0x7ff2941966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff299a59700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff294101770 0x7ff294196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff299a59700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2941972a0 con 0x7ff294101770 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff299a59700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff294199fd0 con 0x7ff294100dc0 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff2927fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff294101770 0x7ff294196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff2927fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff294101770 0x7ff294196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50734/0 (socket says 172.21.15.121:50734) 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff2927fc700 1 -- 172.21.15.121:0/1698636176 learned_addr learned my addr 172.21.15.121:0/1698636176 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff292ffd700 1 --2- 172.21.15.121:0/1698636176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff294100dc0 0x7ff2941966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff2927fc700 1 -- 172.21.15.121:0/1698636176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff294100dc0 msgr2=0x7ff2941966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff2927fc700 1 --2- 172.21.15.121:0/1698636176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff294100dc0 0x7ff2941966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff2927fc700 1 -- 172.21.15.121:0/1698636176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2880095f0 con 0x7ff294101770 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff2927fc700 1 --2- 172.21.15.121:0/1698636176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff294101770 0x7ff294196c10 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff2880092e0 tx=0x7ff28800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff298a57700 1 -- 172.21.15.121:0/1698636176 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff28801d070 con 0x7ff294101770 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff299a59700 1 -- 172.21.15.121:0/1698636176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29419a250 con 0x7ff294101770 2024-04-05T21:57:26.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff298a57700 1 -- 172.21.15.121:0/1698636176 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff288005dc0 con 0x7ff294101770 2024-04-05T21:57:26.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.463+0000 7ff298a57700 1 -- 172.21.15.121:0/1698636176 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff28800fea0 con 0x7ff294101770 2024-04-05T21:57:26.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.464+0000 7ff299a59700 1 -- 172.21.15.121:0/1698636176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29419a740 con 0x7ff294101770 2024-04-05T21:57:26.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.465+0000 7ff299a59700 1 -- 172.21.15.121:0/1698636176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2941091c0 con 0x7ff294101770 2024-04-05T21:57:26.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.465+0000 7ff298a57700 1 -- 172.21.15.121:0/1698636176 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff288003a60 con 0x7ff294101770 2024-04-05T21:57:26.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.465+0000 7ff298a57700 1 --2- 172.21.15.121:0/1698636176 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff27c06cce0 0x7ff27c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:26.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.466+0000 7ff298a57700 1 -- 172.21.15.121:0/1698636176 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff28808d7e0 con 0x7ff294101770 2024-04-05T21:57:26.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.466+0000 7ff292ffd700 1 --2- 172.21.15.121:0/1698636176 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff27c06cce0 0x7ff27c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:26.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.466+0000 7ff292ffd700 1 --2- 172.21.15.121:0/1698636176 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff27c06cce0 0x7ff27c06f1a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff284007be0 tx=0x7ff2840073d0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:26.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.470+0000 7ff298a57700 1 -- 172.21.15.121:0/1698636176 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff2880577b0 con 0x7ff294101770 2024-04-05T21:57:26.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.678+0000 7ff299a59700 1 -- 172.21.15.121:0/1698636176 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff29402cd30 con 0x7ff27c06cce0 2024-04-05T21:57:26.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.679+0000 7ff298a57700 1 -- 172.21.15.121:0/1698636176 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff29402cd30 con 0x7ff27c06cce0 2024-04-05T21:57:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.681+0000 7ff299a59700 1 -- 172.21.15.121:0/1698636176 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff27c06cce0 msgr2=0x7ff27c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.681+0000 7ff299a59700 1 --2- 172.21.15.121:0/1698636176 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff27c06cce0 0x7ff27c06f1a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff284007be0 tx=0x7ff2840073d0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.681+0000 7ff299a59700 1 -- 172.21.15.121:0/1698636176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff294101770 msgr2=0x7ff294196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.681+0000 7ff299a59700 1 --2- 172.21.15.121:0/1698636176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff294101770 0x7ff294196c10 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff2880092e0 tx=0x7ff28800bd70 comp rx=0 tx=0).stop 2024-04-05T21:57:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.682+0000 7ff299a59700 1 -- 172.21.15.121:0/1698636176 shutdown_connections 2024-04-05T21:57:26.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.682+0000 7ff299a59700 1 --2- 172.21.15.121:0/1698636176 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff27c06cce0 0x7ff27c06f1a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.682+0000 7ff299a59700 1 --2- 172.21.15.121:0/1698636176 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff294100dc0 0x7ff2941966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.682+0000 7ff299a59700 1 --2- 172.21.15.121:0/1698636176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff294101770 0x7ff294196c10 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.682+0000 7ff299a59700 1 -- 172.21.15.121:0/1698636176 >> 172.21.15.121:0/1698636176 conn(0x7ff2940fc650 msgr2=0x7ff2940fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.682+0000 7ff299a59700 1 -- 172.21.15.121:0/1698636176 shutdown_connections 2024-04-05T21:57:26.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.682+0000 7ff299a59700 1 -- 172.21.15.121:0/1698636176 wait complete. 2024-04-05T21:57:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.784+0000 7fefe18ff700 1 -- 172.21.15.121:0/1080235251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefdc100dc0 msgr2=0x7fefdc1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.784+0000 7fefe18ff700 1 --2- 172.21.15.121:0/1080235251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefdc100dc0 0x7fefdc1011a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009910 tx=0x7fefd0009c20 comp rx=0 tx=0).stop 2024-04-05T21:57:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.784+0000 7fefe18ff700 1 -- 172.21.15.121:0/1080235251 shutdown_connections 2024-04-05T21:57:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.784+0000 7fefe18ff700 1 --2- 172.21.15.121:0/1080235251 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefdc101770 0x7fefdc1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.784+0000 7fefe18ff700 1 --2- 172.21.15.121:0/1080235251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefdc100dc0 0x7fefdc1011a0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.784+0000 7fefe18ff700 1 -- 172.21.15.121:0/1080235251 >> 172.21.15.121:0/1080235251 conn(0x7fefdc0fc650 msgr2=0x7fefdc0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.784+0000 7fefe18ff700 1 -- 172.21.15.121:0/1080235251 shutdown_connections 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.784+0000 7fefe18ff700 1 -- 172.21.15.121:0/1080235251 wait complete. 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefe18ff700 1 Processor -- start 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefe18ff700 1 -- start start 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefe18ff700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefdc100dc0 0x7fefdc19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefe18ff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefdc101770 0x7fefdc19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefe18ff700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc19f1f0 con 0x7fefdc101770 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefe18ff700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc1986a0 con 0x7fefdc100dc0 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefda7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefdc101770 0x7fefdc19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefda7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefdc101770 0x7fefdc19eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50750/0 (socket says 172.21.15.121:50750) 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefda7fc700 1 -- 172.21.15.121:0/1224219439 learned_addr learned my addr 172.21.15.121:0/1224219439 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefdaffd700 1 --2- 172.21.15.121:0/1224219439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefdc100dc0 0x7fefdc19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefda7fc700 1 -- 172.21.15.121:0/1224219439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefdc100dc0 msgr2=0x7fefdc19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:26.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefda7fc700 1 --2- 172.21.15.121:0/1224219439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefdc100dc0 0x7fefdc19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefda7fc700 1 -- 172.21.15.121:0/1224219439 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd00095f0 con 0x7fefdc101770 2024-04-05T21:57:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.785+0000 7fefda7fc700 1 --2- 172.21.15.121:0/1224219439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefdc101770 0x7fefdc19eb60 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00d900 tx=0x7fefcc00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.786+0000 7fefe08fd700 1 -- 172.21.15.121:0/1224219439 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefcc009900 con 0x7fefdc101770 2024-04-05T21:57:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.786+0000 7fefe18ff700 1 -- 172.21.15.121:0/1224219439 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefdc198980 con 0x7fefdc101770 2024-04-05T21:57:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.786+0000 7fefe08fd700 1 -- 172.21.15.121:0/1224219439 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefcc010460 con 0x7fefdc101770 2024-04-05T21:57:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.786+0000 7fefe08fd700 1 -- 172.21.15.121:0/1224219439 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefcc00f5f0 con 0x7fefdc101770 2024-04-05T21:57:26.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.786+0000 7fefe18ff700 1 -- 172.21.15.121:0/1224219439 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefdc198ed0 con 0x7fefdc101770 2024-04-05T21:57:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.787+0000 7fefe18ff700 1 -- 172.21.15.121:0/1224219439 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefdc104f40 con 0x7fefdc101770 2024-04-05T21:57:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.787+0000 7fefe08fd700 1 -- 172.21.15.121:0/1224219439 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fefcc0105d0 con 0x7fefdc101770 2024-04-05T21:57:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.788+0000 7fefe08fd700 1 --2- 172.21.15.121:0/1224219439 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fefc406cce0 0x7fefc406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.788+0000 7fefe08fd700 1 -- 172.21.15.121:0/1224219439 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fefcc08c0e0 con 0x7fefdc101770 2024-04-05T21:57:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.788+0000 7fefdaffd700 1 --2- 172.21.15.121:0/1224219439 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fefc406cce0 0x7fefc406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:26.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.788+0000 7fefdaffd700 1 --2- 172.21.15.121:0/1224219439 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fefc406cce0 0x7fefc406f1a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fefd0006010 tx=0x7fefd001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:26.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.792+0000 7fefe08fd700 1 -- 172.21.15.121:0/1224219439 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fefcc059be0 con 0x7fefdc101770 2024-04-05T21:57:26.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:26.991+0000 7fefe18ff700 1 -- 172.21.15.121:0/1224219439 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fefdc199160 con 0x7fefc406cce0 2024-04-05T21:57:27.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.001+0000 7fefe08fd700 1 -- 172.21.15.121:0/1224219439 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fefdc199160 con 0x7fefc406cce0 2024-04-05T21:57:27.001 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:57:27.001 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (28m) 6m ago 39m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (40m) 6m ago 40m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (38m) 6m ago 38m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (26m) 6m ago 39m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (26m) 6m ago 38m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (28m) 6m ago 39m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (22m) 6m ago 33m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (22m) 6m ago 33m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (22m) 6m ago 33m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (6m) 6m ago 33m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (30m) 6m ago 41m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (29m) 6m ago 38m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (27m) 6m ago 41m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (26m) 6m ago 38m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:57:27.002 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (29m) 6m ago 39m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:57:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (29m) 6m ago 38m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:57:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (25m) 6m ago 37m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:57:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (25m) 6m ago 37m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:57:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (24m) 6m ago 36m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:57:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (24m) 6m ago 36m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:57:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (23m) 6m ago 35m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:57:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (22m) 6m ago 35m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:57:27.003 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (28m) 6m ago 39m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:57:27.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.003+0000 7fefe18ff700 1 -- 172.21.15.121:0/1224219439 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fefc406cce0 msgr2=0x7fefc406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:27.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.003+0000 7fefe18ff700 1 --2- 172.21.15.121:0/1224219439 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fefc406cce0 0x7fefc406f1a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fefd0006010 tx=0x7fefd001a040 comp rx=0 tx=0).stop 2024-04-05T21:57:27.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.003+0000 7fefe18ff700 1 -- 172.21.15.121:0/1224219439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefdc101770 msgr2=0x7fefdc19eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:27.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.003+0000 7fefe18ff700 1 --2- 172.21.15.121:0/1224219439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefdc101770 0x7fefdc19eb60 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00d900 tx=0x7fefcc00dc10 comp rx=0 tx=0).stop 2024-04-05T21:57:27.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.003+0000 7fefe18ff700 1 -- 172.21.15.121:0/1224219439 shutdown_connections 2024-04-05T21:57:27.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.003+0000 7fefe18ff700 1 --2- 172.21.15.121:0/1224219439 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fefc406cce0 0x7fefc406f1a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.003+0000 7fefe18ff700 1 --2- 172.21.15.121:0/1224219439 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fefdc100dc0 0x7fefdc19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.003+0000 7fefe18ff700 1 --2- 172.21.15.121:0/1224219439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefdc101770 0x7fefdc19eb60 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.003+0000 7fefe18ff700 1 -- 172.21.15.121:0/1224219439 >> 172.21.15.121:0/1224219439 conn(0x7fefdc0fc650 msgr2=0x7fefdc0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:27.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.003+0000 7fefe18ff700 1 -- 172.21.15.121:0/1224219439 shutdown_connections 2024-04-05T21:57:27.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.003+0000 7fefe18ff700 1 -- 172.21.15.121:0/1224219439 wait complete. 2024-04-05T21:57:27.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.104+0000 7f5e7f48b700 1 -- 172.21.15.121:0/1366812380 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e78100de0 msgr2=0x7f5e781011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:27.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.104+0000 7f5e7f48b700 1 --2- 172.21.15.121:0/1366812380 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e78100de0 0x7f5e781011c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009910 tx=0x7f5e68009c20 comp rx=0 tx=0).stop 2024-04-05T21:57:27.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.104+0000 7f5e7f48b700 1 -- 172.21.15.121:0/1366812380 shutdown_connections 2024-04-05T21:57:27.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.104+0000 7f5e7f48b700 1 --2- 172.21.15.121:0/1366812380 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e78101790 0x7f5e781057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.104+0000 7f5e7f48b700 1 --2- 172.21.15.121:0/1366812380 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e78100de0 0x7f5e781011c0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.104+0000 7f5e7f48b700 1 -- 172.21.15.121:0/1366812380 >> 172.21.15.121:0/1366812380 conn(0x7f5e780fc670 msgr2=0x7f5e780fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:27.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.104+0000 7f5e7f48b700 1 -- 172.21.15.121:0/1366812380 shutdown_connections 2024-04-05T21:57:27.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.104+0000 7f5e7f48b700 1 -- 172.21.15.121:0/1366812380 wait complete. 2024-04-05T21:57:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7f48b700 1 Processor -- start 2024-04-05T21:57:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7f48b700 1 -- start start 2024-04-05T21:57:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7f48b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e78100de0 0x7f5e78196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7f48b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e78101790 0x7f5e78196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7f48b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e78197290 con 0x7f5e78100de0 2024-04-05T21:57:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7f48b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7819b020 con 0x7f5e78101790 2024-04-05T21:57:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7d227700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e78100de0 0x7f5e78196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7d227700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e78100de0 0x7f5e78196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50766/0 (socket says 172.21.15.121:50766) 2024-04-05T21:57:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7d227700 1 -- 172.21.15.121:0/936260688 learned_addr learned my addr 172.21.15.121:0/936260688 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:57:27.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7ca26700 1 --2- 172.21.15.121:0/936260688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e78101790 0x7f5e78196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7d227700 1 -- 172.21.15.121:0/936260688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e78101790 msgr2=0x7f5e78196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7d227700 1 --2- 172.21.15.121:0/936260688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e78101790 0x7f5e78196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7d227700 1 -- 172.21.15.121:0/936260688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e680095f0 con 0x7f5e78100de0 2024-04-05T21:57:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7ca26700 1 --2- 172.21.15.121:0/936260688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e78101790 0x7f5e78196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:57:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.105+0000 7f5e7d227700 1 --2- 172.21.15.121:0/936260688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e78100de0 0x7f5e78196670 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5e68000c00 tx=0x7f5e6800be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.106+0000 7f5e6e7fc700 1 -- 172.21.15.121:0/936260688 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e6801d070 con 0x7f5e78100de0 2024-04-05T21:57:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.106+0000 7f5e6e7fc700 1 -- 172.21.15.121:0/936260688 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e680044c0 con 0x7f5e78100de0 2024-04-05T21:57:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.106+0000 7f5e6e7fc700 1 -- 172.21.15.121:0/936260688 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e680039d0 con 0x7f5e78100de0 2024-04-05T21:57:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.106+0000 7f5e7f48b700 1 -- 172.21.15.121:0/936260688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7819b2a0 con 0x7f5e78100de0 2024-04-05T21:57:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.106+0000 7f5e7f48b700 1 -- 172.21.15.121:0/936260688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7819b730 con 0x7f5e78100de0 2024-04-05T21:57:27.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.107+0000 7f5e7f48b700 1 -- 172.21.15.121:0/936260688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e78109180 con 0x7f5e78100de0 2024-04-05T21:57:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.107+0000 7f5e6e7fc700 1 -- 172.21.15.121:0/936260688 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5e680048e0 con 0x7f5e78100de0 2024-04-05T21:57:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.107+0000 7f5e6e7fc700 1 --2- 172.21.15.121:0/936260688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5e6406cce0 0x7f5e6406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.107+0000 7f5e6e7fc700 1 -- 172.21.15.121:0/936260688 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5e6808db40 con 0x7f5e78100de0 2024-04-05T21:57:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.107+0000 7f5e7ca26700 1 --2- 172.21.15.121:0/936260688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5e6406cce0 0x7f5e6406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:27.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.108+0000 7f5e7ca26700 1 --2- 172.21.15.121:0/936260688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5e6406cce0 0x7f5e6406f1a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5e78197c90 tx=0x7f5e74009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:27.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.112+0000 7f5e6e7fc700 1 -- 172.21.15.121:0/936260688 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5e68057c30 con 0x7f5e78100de0 2024-04-05T21:57:27.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:26 smithi121 ceph-mon[145859]: pgmap v937: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:27.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:26 smithi188 ceph-mon[107793]: pgmap v937: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:27.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.384+0000 7f5e7f48b700 1 -- 172.21.15.121:0/936260688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5e7804ea90 con 0x7f5e78100de0 2024-04-05T21:57:27.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.385+0000 7f5e6e7fc700 1 -- 172.21.15.121:0/936260688 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f5e68026290 con 0x7f5e78100de0 2024-04-05T21:57:27.385 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:57:27.385 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:57:27.385 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:57:27.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:57:27.385 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:57:27.385 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:57:27.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:57:27.386 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:57:27.386 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:57:27.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:57:27.386 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:57:27.386 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:57:27.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:57:27.386 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:57:27.386 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:57:27.386 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:57:27.386 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:57:27.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.387+0000 7f5e7f48b700 1 -- 172.21.15.121:0/936260688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5e6406cce0 msgr2=0x7f5e6406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:27.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.387+0000 7f5e7f48b700 1 --2- 172.21.15.121:0/936260688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5e6406cce0 0x7f5e6406f1a0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5e78197c90 tx=0x7f5e74009260 comp rx=0 tx=0).stop 2024-04-05T21:57:27.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.387+0000 7f5e7f48b700 1 -- 172.21.15.121:0/936260688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e78100de0 msgr2=0x7f5e78196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:27.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.387+0000 7f5e7f48b700 1 --2- 172.21.15.121:0/936260688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e78100de0 0x7f5e78196670 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5e68000c00 tx=0x7f5e6800be90 comp rx=0 tx=0).stop 2024-04-05T21:57:27.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.387+0000 7f5e7f48b700 1 -- 172.21.15.121:0/936260688 shutdown_connections 2024-04-05T21:57:27.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.387+0000 7f5e7f48b700 1 --2- 172.21.15.121:0/936260688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5e6406cce0 0x7f5e6406f1a0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.387+0000 7f5e7f48b700 1 --2- 172.21.15.121:0/936260688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e78100de0 0x7f5e78196670 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.387+0000 7f5e7f48b700 1 --2- 172.21.15.121:0/936260688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5e78101790 0x7f5e78196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.387+0000 7f5e7f48b700 1 -- 172.21.15.121:0/936260688 >> 172.21.15.121:0/936260688 conn(0x7f5e780fc670 msgr2=0x7f5e780fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:27.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.387+0000 7f5e7f48b700 1 -- 172.21.15.121:0/936260688 shutdown_connections 2024-04-05T21:57:27.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.387+0000 7f5e7f48b700 1 -- 172.21.15.121:0/936260688 wait complete. 2024-04-05T21:57:27.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.489+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/438689422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4101da0 msgr2=0x7f8ea4102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:27.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.489+0000 7f8eaa2f1700 1 --2- 172.21.15.121:0/438689422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4101da0 0x7f8ea4102180 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f8e94009910 tx=0x7f8e94009c20 comp rx=0 tx=0).stop 2024-04-05T21:57:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.490+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/438689422 shutdown_connections 2024-04-05T21:57:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.490+0000 7f8eaa2f1700 1 --2- 172.21.15.121:0/438689422 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ea41026c0 0x7f8ea410abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.490+0000 7f8eaa2f1700 1 --2- 172.21.15.121:0/438689422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4101da0 0x7f8ea4102180 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.490+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/438689422 >> 172.21.15.121:0/438689422 conn(0x7f8ea40fb570 msgr2=0x7f8ea40fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.490+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/438689422 shutdown_connections 2024-04-05T21:57:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.490+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/438689422 wait complete. 2024-04-05T21:57:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.490+0000 7f8eaa2f1700 1 Processor -- start 2024-04-05T21:57:27.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.490+0000 7f8eaa2f1700 1 -- start start 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8eaa2f1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4101da0 0x7f8ea41966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8eaa2f1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ea41026c0 0x7f8ea4196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8eaa2f1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea4197270 con 0x7f8ea4101da0 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8eaa2f1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea4199fa0 con 0x7f8ea41026c0 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8ea3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4101da0 0x7f8ea41966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8ea3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4101da0 0x7f8ea41966a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50780/0 (socket says 172.21.15.121:50780) 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8ea3fff700 1 -- 172.21.15.121:0/544204566 learned_addr learned my addr 172.21.15.121:0/544204566 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8ea37fe700 1 --2- 172.21.15.121:0/544204566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ea41026c0 0x7f8ea4196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8ea3fff700 1 -- 172.21.15.121:0/544204566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ea41026c0 msgr2=0x7f8ea4196be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8ea3fff700 1 --2- 172.21.15.121:0/544204566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ea41026c0 0x7f8ea4196be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8ea3fff700 1 -- 172.21.15.121:0/544204566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e940095f0 con 0x7f8ea4101da0 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8ea3fff700 1 --2- 172.21.15.121:0/544204566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4101da0 0x7f8ea41966a0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f8e94006010 tx=0x7f8e9400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8ea17fa700 1 -- 172.21.15.121:0/544204566 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e9401d070 con 0x7f8ea4101da0 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8ea17fa700 1 -- 172.21.15.121:0/544204566 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e9400f460 con 0x7f8ea4101da0 2024-04-05T21:57:27.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/544204566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea419a220 con 0x7f8ea4101da0 2024-04-05T21:57:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8ea17fa700 1 -- 172.21.15.121:0/544204566 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e94021620 con 0x7f8ea4101da0 2024-04-05T21:57:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.491+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/544204566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea419a710 con 0x7f8ea4101da0 2024-04-05T21:57:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.492+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/544204566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea41082b0 con 0x7f8ea4101da0 2024-04-05T21:57:27.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.493+0000 7f8ea17fa700 1 -- 172.21.15.121:0/544204566 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8e9400f5d0 con 0x7f8ea4101da0 2024-04-05T21:57:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.494+0000 7f8ea17fa700 1 --2- 172.21.15.121:0/544204566 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e9006cce0 0x7f8e9006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.494+0000 7f8ea17fa700 1 -- 172.21.15.121:0/544204566 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8e9408dd50 con 0x7f8ea4101da0 2024-04-05T21:57:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.494+0000 7f8ea37fe700 1 --2- 172.21.15.121:0/544204566 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e9006cce0 0x7f8e9006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:27.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.494+0000 7f8ea37fe700 1 --2- 172.21.15.121:0/544204566 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e9006cce0 0x7f8e9006f1a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c005fd0 tx=0x7f8e8c00a560 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:27.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.498+0000 7f8ea17fa700 1 -- 172.21.15.121:0/544204566 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8e94057da0 con 0x7f8ea4101da0 2024-04-05T21:57:27.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.736+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/544204566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8ea404ea90 con 0x7f8ea4101da0 2024-04-05T21:57:27.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.737+0000 7f8ea17fa700 1 -- 172.21.15.121:0/544204566 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f8e9405b3c0 con 0x7f8ea4101da0 2024-04-05T21:57:27.737 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:57:27.737 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:57:27.738 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:57:27.738 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:57:27.738 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:57:27.738 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:57:27.738 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:57:27.738 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:57:27.738 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:57:27.738 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:57:27.738 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:57:27.738 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:57:27.738 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:57:27.738 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:57:27.738 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:57:27.739 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:57:27.739 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:57:27.739 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:57:27.739 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:57:27.739 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:57:27.739 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:57:27.739 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:57:27.739 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:57:27.739 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:57:27.739 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:57:27.739 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:57:27.739 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:57:27.741 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:57:27.741 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:57:27.741 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:57:27.741 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:57:27.741 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:57:27.741 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:57:27.741 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:57:27.742 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:57:27.742 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:57:27.742 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:57:27.742 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:57:27.742 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:57:27.742 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:57:27.742 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:57:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.739+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/544204566 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e9006cce0 msgr2=0x7f8e9006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.739+0000 7f8eaa2f1700 1 --2- 172.21.15.121:0/544204566 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e9006cce0 0x7f8e9006f1a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c005fd0 tx=0x7f8e8c00a560 comp rx=0 tx=0).stop 2024-04-05T21:57:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.739+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/544204566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4101da0 msgr2=0x7f8ea41966a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.739+0000 7f8eaa2f1700 1 --2- 172.21.15.121:0/544204566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4101da0 0x7f8ea41966a0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f8e94006010 tx=0x7f8e9400be20 comp rx=0 tx=0).stop 2024-04-05T21:57:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.739+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/544204566 shutdown_connections 2024-04-05T21:57:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.739+0000 7f8eaa2f1700 1 --2- 172.21.15.121:0/544204566 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8e9006cce0 0x7f8e9006f1a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.739+0000 7f8eaa2f1700 1 --2- 172.21.15.121:0/544204566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4101da0 0x7f8ea41966a0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.739+0000 7f8eaa2f1700 1 --2- 172.21.15.121:0/544204566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8ea41026c0 0x7f8ea4196be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.739+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/544204566 >> 172.21.15.121:0/544204566 conn(0x7f8ea40fb570 msgr2=0x7f8ea41053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:27.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.739+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/544204566 shutdown_connections 2024-04-05T21:57:27.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.739+0000 7f8eaa2f1700 1 -- 172.21.15.121:0/544204566 wait complete. 2024-04-05T21:57:27.744 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:57:27.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.843+0000 7f798ea18700 1 -- 172.21.15.121:0/1750328849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7988106be0 msgr2=0x7f7988106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:27.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.843+0000 7f798ea18700 1 --2- 172.21.15.121:0/1750328849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7988106be0 0x7f7988106fc0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7970009910 tx=0x7f7970009c20 comp rx=0 tx=0).stop 2024-04-05T21:57:27.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.843+0000 7f798ea18700 1 -- 172.21.15.121:0/1750328849 shutdown_connections 2024-04-05T21:57:27.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.843+0000 7f798ea18700 1 --2- 172.21.15.121:0/1750328849 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f79880ff260 0x7f79880ff6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.843+0000 7f798ea18700 1 --2- 172.21.15.121:0/1750328849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7988106be0 0x7f7988106fc0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.843+0000 7f798ea18700 1 -- 172.21.15.121:0/1750328849 >> 172.21.15.121:0/1750328849 conn(0x7f7988074ce0 msgr2=0x7f79880750f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:27.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.843+0000 7f798ea18700 1 -- 172.21.15.121:0/1750328849 shutdown_connections 2024-04-05T21:57:27.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.843+0000 7f798ea18700 1 -- 172.21.15.121:0/1750328849 wait complete. 2024-04-05T21:57:27.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.843+0000 7f798ea18700 1 Processor -- start 2024-04-05T21:57:27.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.843+0000 7f798ea18700 1 -- start start 2024-04-05T21:57:27.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f798ea18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79880ff260 0x7f798819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:27.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f798ea18700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7988106be0 0x7f798819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:27.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f798ea18700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798819f230 con 0x7f79880ff260 2024-04-05T21:57:27.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f798ea18700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79881986b0 con 0x7f7988106be0 2024-04-05T21:57:27.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f7987fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79880ff260 0x7f798819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:27.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f7987fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79880ff260 0x7f798819e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50788/0 (socket says 172.21.15.121:50788) 2024-04-05T21:57:27.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f7987fff700 1 -- 172.21.15.121:0/156846794 learned_addr learned my addr 172.21.15.121:0/156846794 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:57:27.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f79877fe700 1 --2- 172.21.15.121:0/156846794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7988106be0 0x7f798819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:27.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f7987fff700 1 -- 172.21.15.121:0/156846794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7988106be0 msgr2=0x7f798819eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:27.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f7987fff700 1 --2- 172.21.15.121:0/156846794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7988106be0 0x7f798819eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:27.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f7987fff700 1 -- 172.21.15.121:0/156846794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79700095f0 con 0x7f79880ff260 2024-04-05T21:57:27.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f7987fff700 1 --2- 172.21.15.121:0/156846794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79880ff260 0x7f798819e610 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7970006010 tx=0x7f797000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:27.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f79857fa700 1 -- 172.21.15.121:0/156846794 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f797001d070 con 0x7f79880ff260 2024-04-05T21:57:27.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f79857fa700 1 -- 172.21.15.121:0/156846794 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f797000f460 con 0x7f79880ff260 2024-04-05T21:57:27.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f79857fa700 1 -- 172.21.15.121:0/156846794 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7970021620 con 0x7f79880ff260 2024-04-05T21:57:27.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f798ea18700 1 -- 172.21.15.121:0/156846794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7988198930 con 0x7f79880ff260 2024-04-05T21:57:27.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.844+0000 7f798ea18700 1 -- 172.21.15.121:0/156846794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7988198ea0 con 0x7f79880ff260 2024-04-05T21:57:27.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.845+0000 7f798ea18700 1 -- 172.21.15.121:0/156846794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798804ea90 con 0x7f79880ff260 2024-04-05T21:57:27.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.846+0000 7f79857fa700 1 -- 172.21.15.121:0/156846794 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f797000f5d0 con 0x7f79880ff260 2024-04-05T21:57:27.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.846+0000 7f79857fa700 1 --2- 172.21.15.121:0/156846794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f797406cce0 0x7f797406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:27.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.846+0000 7f79857fa700 1 -- 172.21.15.121:0/156846794 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f797008d800 con 0x7f79880ff260 2024-04-05T21:57:27.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.846+0000 7f79877fe700 1 --2- 172.21.15.121:0/156846794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f797406cce0 0x7f797406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:27.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.847+0000 7f79877fe700 1 --2- 172.21.15.121:0/156846794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f797406cce0 0x7f797406f1a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7978006fd0 tx=0x7f7978008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:27.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:27.851+0000 7f79857fa700 1 -- 172.21.15.121:0/156846794 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f797005b380 con 0x7f79880ff260 2024-04-05T21:57:28.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.061+0000 7f798ea18700 1 -- 172.21.15.121:0/156846794 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7988199340 con 0x7f797406cce0 2024-04-05T21:57:28.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.062+0000 7f79857fa700 1 -- 172.21.15.121:0/156846794 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7988199340 con 0x7f797406cce0 2024-04-05T21:57:28.062 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:57:28.062 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:57:28.063 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:57:28.063 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:57:28.063 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:57:28.063 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:57:28.063 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:57:28.063 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:57:28.063 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:57:28.063 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:57:28.063 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:57:28.063 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:57:28.063 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:57:28.063 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:57:28.063 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:57:28.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.064+0000 7f798ea18700 1 -- 172.21.15.121:0/156846794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f797406cce0 msgr2=0x7f797406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:28.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.064+0000 7f798ea18700 1 --2- 172.21.15.121:0/156846794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f797406cce0 0x7f797406f1a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7978006fd0 tx=0x7f7978008040 comp rx=0 tx=0).stop 2024-04-05T21:57:28.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.064+0000 7f798ea18700 1 -- 172.21.15.121:0/156846794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79880ff260 msgr2=0x7f798819e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:28.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.064+0000 7f798ea18700 1 --2- 172.21.15.121:0/156846794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79880ff260 0x7f798819e610 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7970006010 tx=0x7f797000be20 comp rx=0 tx=0).stop 2024-04-05T21:57:28.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.064+0000 7f798ea18700 1 -- 172.21.15.121:0/156846794 shutdown_connections 2024-04-05T21:57:28.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.064+0000 7f798ea18700 1 --2- 172.21.15.121:0/156846794 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f797406cce0 0x7f797406f1a0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:28.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.064+0000 7f798ea18700 1 --2- 172.21.15.121:0/156846794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79880ff260 0x7f798819e610 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:28.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.064+0000 7f798ea18700 1 --2- 172.21.15.121:0/156846794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7988106be0 0x7f798819eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:28.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.064+0000 7f798ea18700 1 -- 172.21.15.121:0/156846794 >> 172.21.15.121:0/156846794 conn(0x7f7988074ce0 msgr2=0x7f79880fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:28.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.064+0000 7f798ea18700 1 -- 172.21.15.121:0/156846794 shutdown_connections 2024-04-05T21:57:28.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.064+0000 7f798ea18700 1 -- 172.21.15.121:0/156846794 wait complete. 2024-04-05T21:57:28.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.167+0000 7fe2917ce700 1 -- 172.21.15.121:0/2718549808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe28c0ffa30 msgr2=0x7fe28c0ffeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:28.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.167+0000 7fe2917ce700 1 --2- 172.21.15.121:0/2718549808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe28c0ffa30 0x7fe28c0ffeb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe27c009910 tx=0x7fe27c009c20 comp rx=0 tx=0).stop 2024-04-05T21:57:28.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.167+0000 7fe2917ce700 1 -- 172.21.15.121:0/2718549808 shutdown_connections 2024-04-05T21:57:28.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.167+0000 7fe2917ce700 1 --2- 172.21.15.121:0/2718549808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe28c0ffa30 0x7fe28c0ffeb0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:28.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.167+0000 7fe2917ce700 1 --2- 172.21.15.121:0/2718549808 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28c105a30 0x7fe28c105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:28.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.167+0000 7fe2917ce700 1 -- 172.21.15.121:0/2718549808 >> 172.21.15.121:0/2718549808 conn(0x7fe28c0fb540 msgr2=0x7fe28c0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:28.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.167+0000 7fe2917ce700 1 -- 172.21.15.121:0/2718549808 shutdown_connections 2024-04-05T21:57:28.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.167+0000 7fe2917ce700 1 -- 172.21.15.121:0/2718549808 wait complete. 2024-04-05T21:57:28.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.168+0000 7fe2917ce700 1 Processor -- start 2024-04-05T21:57:28.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.168+0000 7fe2917ce700 1 -- start start 2024-04-05T21:57:28.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.168+0000 7fe2917ce700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28c0ffa30 0x7fe28c1952c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:28.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.168+0000 7fe2917ce700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe28c105a30 0x7fe28c195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:28.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.168+0000 7fe2917ce700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c195d40 con 0x7fe28c105a30 2024-04-05T21:57:28.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.168+0000 7fe2917ce700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c195eb0 con 0x7fe28c0ffa30 2024-04-05T21:57:28.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.168+0000 7fe28a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe28c105a30 0x7fe28c195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:28.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.168+0000 7fe28a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe28c105a30 0x7fe28c195800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50794/0 (socket says 172.21.15.121:50794) 2024-04-05T21:57:28.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.168+0000 7fe28a7fc700 1 -- 172.21.15.121:0/2813782872 learned_addr learned my addr 172.21.15.121:0/2813782872 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:57:28.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.168+0000 7fe28affd700 1 --2- 172.21.15.121:0/2813782872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28c0ffa30 0x7fe28c1952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:28.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.169+0000 7fe28a7fc700 1 -- 172.21.15.121:0/2813782872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28c0ffa30 msgr2=0x7fe28c1952c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:28.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.169+0000 7fe28a7fc700 1 --2- 172.21.15.121:0/2813782872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28c0ffa30 0x7fe28c1952c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:28.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.169+0000 7fe28a7fc700 1 -- 172.21.15.121:0/2813782872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c0095f0 con 0x7fe28c105a30 2024-04-05T21:57:28.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.169+0000 7fe28a7fc700 1 --2- 172.21.15.121:0/2813782872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe28c105a30 0x7fe28c195800 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe27c000c00 tx=0x7fe27c00bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:28.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.169+0000 7fe273fff700 1 -- 172.21.15.121:0/2813782872 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe27c01d070 con 0x7fe28c105a30 2024-04-05T21:57:28.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.169+0000 7fe2917ce700 1 -- 172.21.15.121:0/2813782872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c196130 con 0x7fe28c105a30 2024-04-05T21:57:28.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.169+0000 7fe273fff700 1 -- 172.21.15.121:0/2813782872 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe27c005dc0 con 0x7fe28c105a30 2024-04-05T21:57:28.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.169+0000 7fe273fff700 1 -- 172.21.15.121:0/2813782872 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe27c003940 con 0x7fe28c105a30 2024-04-05T21:57:28.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.169+0000 7fe2917ce700 1 -- 172.21.15.121:0/2813782872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c1a2b50 con 0x7fe28c105a30 2024-04-05T21:57:28.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.170+0000 7fe2917ce700 1 -- 172.21.15.121:0/2813782872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28c107fc0 con 0x7fe28c105a30 2024-04-05T21:57:28.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.170+0000 7fe273fff700 1 -- 172.21.15.121:0/2813782872 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe27c004ae0 con 0x7fe28c105a30 2024-04-05T21:57:28.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.171+0000 7fe273fff700 1 --2- 172.21.15.121:0/2813782872 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe27406cce0 0x7fe27406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:28.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.171+0000 7fe273fff700 1 -- 172.21.15.121:0/2813782872 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe27c08da70 con 0x7fe28c105a30 2024-04-05T21:57:28.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.171+0000 7fe28affd700 1 --2- 172.21.15.121:0/2813782872 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe27406cce0 0x7fe27406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:28.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.171+0000 7fe28affd700 1 --2- 172.21.15.121:0/2813782872 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe27406cce0 0x7fe27406f1a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe280009c60 tx=0x7fe280009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.175+0000 7fe273fff700 1 -- 172.21.15.121:0/2813782872 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe27c08fe60 con 0x7fe28c105a30 2024-04-05T21:57:28.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:27 smithi121 ceph-mon[145859]: from='client.35522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:57:28.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:27 smithi121 ceph-mon[145859]: from='client.35526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:57:28.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:27 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/936260688' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:57:28.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:27 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/544204566' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:57:28.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:27 smithi188 ceph-mon[107793]: from='client.35522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:57:28.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:27 smithi188 ceph-mon[107793]: from='client.35526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:57:28.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:27 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/936260688' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:57:28.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:27 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/544204566' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:57:28.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.449+0000 7fe2917ce700 1 -- 172.21.15.121:0/2813782872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe28c196130 con 0x7fe28c105a30 2024-04-05T21:57:28.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.449+0000 7fe273fff700 1 -- 172.21.15.121:0/2813782872 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fe28c196130 con 0x7fe28c105a30 2024-04-05T21:57:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:57:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:57:28.450 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:57:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:57:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:57:28.450 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:57:28.450 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:57:28.450 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:57:28.450 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:57:28.451 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:57:28.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.451+0000 7fe2917ce700 1 -- 172.21.15.121:0/2813782872 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe27406cce0 msgr2=0x7fe27406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:28.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.451+0000 7fe2917ce700 1 --2- 172.21.15.121:0/2813782872 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe27406cce0 0x7fe27406f1a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe280009c60 tx=0x7fe280009260 comp rx=0 tx=0).stop 2024-04-05T21:57:28.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.451+0000 7fe2917ce700 1 -- 172.21.15.121:0/2813782872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe28c105a30 msgr2=0x7fe28c195800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:28.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.451+0000 7fe2917ce700 1 --2- 172.21.15.121:0/2813782872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe28c105a30 0x7fe28c195800 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fe27c000c00 tx=0x7fe27c00bde0 comp rx=0 tx=0).stop 2024-04-05T21:57:28.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.451+0000 7fe2917ce700 1 -- 172.21.15.121:0/2813782872 shutdown_connections 2024-04-05T21:57:28.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.451+0000 7fe2917ce700 1 --2- 172.21.15.121:0/2813782872 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe27406cce0 0x7fe27406f1a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:28.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.451+0000 7fe2917ce700 1 --2- 172.21.15.121:0/2813782872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe28c0ffa30 0x7fe28c1952c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:28.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.451+0000 7fe2917ce700 1 --2- 172.21.15.121:0/2813782872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe28c105a30 0x7fe28c195800 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:28.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.451+0000 7fe2917ce700 1 -- 172.21.15.121:0/2813782872 >> 172.21.15.121:0/2813782872 conn(0x7fe28c0fb540 msgr2=0x7fe28c0fc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:28.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.451+0000 7fe2917ce700 1 -- 172.21.15.121:0/2813782872 shutdown_connections 2024-04-05T21:57:28.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:28.452+0000 7fe2917ce700 1 -- 172.21.15.121:0/2813782872 wait complete. 2024-04-05T21:57:29.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:28 smithi121 ceph-mon[145859]: from='client.35530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:57:29.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:28 smithi121 ceph-mon[145859]: pgmap v938: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:29.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:28 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2813782872' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:57:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:28 smithi188 ceph-mon[107793]: from='client.35530 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:57:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:28 smithi188 ceph-mon[107793]: pgmap v938: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:28 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2813782872' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:57:30.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:29 smithi121 ceph-mon[145859]: from='client.35542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:57:30.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:29 smithi188 ceph-mon[107793]: from='client.35542 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:57:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:30 smithi121 ceph-mon[145859]: pgmap v939: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:31.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:57:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:30 smithi188 ceph-mon[107793]: pgmap v939: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:57:33.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:32 smithi121 ceph-mon[145859]: pgmap v940: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:33.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:32 smithi188 ceph-mon[107793]: pgmap v940: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:35.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:34 smithi121 ceph-mon[145859]: pgmap v941: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:35.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:34 smithi188 ceph-mon[107793]: pgmap v941: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:37.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:36 smithi121 ceph-mon[145859]: pgmap v942: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:36 smithi188 ceph-mon[107793]: pgmap v942: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:39.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:38 smithi121 ceph-mon[145859]: pgmap v943: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:38 smithi188 ceph-mon[107793]: pgmap v943: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:40 smithi121 ceph-mon[145859]: pgmap v944: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:40 smithi188 ceph-mon[107793]: pgmap v944: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:43.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:42 smithi121 ceph-mon[145859]: pgmap v945: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:42 smithi188 ceph-mon[107793]: pgmap v945: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:45.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:44 smithi121 ceph-mon[145859]: pgmap v946: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:44 smithi188 ceph-mon[107793]: pgmap v946: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:57:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:57:47.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:46 smithi121 ceph-mon[145859]: pgmap v947: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:46 smithi188 ceph-mon[107793]: pgmap v947: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:49.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:48 smithi121 ceph-mon[145859]: pgmap v948: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:48 smithi188 ceph-mon[107793]: pgmap v948: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:51.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:50 smithi121 ceph-mon[145859]: pgmap v949: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:50 smithi188 ceph-mon[107793]: pgmap v949: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:51 smithi188 ceph-mon[107793]: pgmap v950: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:52.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:51 smithi121 ceph-mon[145859]: pgmap v950: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:55.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:54 smithi121 ceph-mon[145859]: pgmap v951: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:54 smithi188 ceph-mon[107793]: pgmap v951: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:56 smithi121 ceph-mon[145859]: pgmap v952: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:56 smithi188 ceph-mon[107793]: pgmap v952: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.566+0000 7f166479f700 1 -- 172.21.15.121:0/2488281081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f165c105a40 msgr2=0x7f165c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.566+0000 7f166479f700 1 --2- 172.21.15.121:0/2488281081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f165c105a40 0x7f165c105e20 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f164c009910 tx=0x7f164c009c20 comp rx=0 tx=0).stop 2024-04-05T21:57:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.566+0000 7f166479f700 1 -- 172.21.15.121:0/2488281081 shutdown_connections 2024-04-05T21:57:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.566+0000 7f166479f700 1 --2- 172.21.15.121:0/2488281081 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f165c0ffbd0 0x7f165c100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.566+0000 7f166479f700 1 --2- 172.21.15.121:0/2488281081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f165c105a40 0x7f165c105e20 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.566+0000 7f166479f700 1 -- 172.21.15.121:0/2488281081 >> 172.21.15.121:0/2488281081 conn(0x7f165c0755a0 msgr2=0x7f165c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.566+0000 7f166479f700 1 -- 172.21.15.121:0/2488281081 shutdown_connections 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.566+0000 7f166479f700 1 -- 172.21.15.121:0/2488281081 wait complete. 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f166479f700 1 Processor -- start 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f166479f700 1 -- start start 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f166479f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f165c0ffbd0 0x7f165c1974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f166479f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f165c105a40 0x7f165c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f166479f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c198020 con 0x7f165c0ffbd0 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f166479f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c198160 con 0x7f165c105a40 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f166253b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f165c0ffbd0 0x7f165c1974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f166253b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f165c0ffbd0 0x7f165c1974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52030/0 (socket says 172.21.15.121:52030) 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f166253b700 1 -- 172.21.15.121:0/3027092803 learned_addr learned my addr 172.21.15.121:0/3027092803 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f1661d3a700 1 --2- 172.21.15.121:0/3027092803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f165c105a40 0x7f165c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f166253b700 1 -- 172.21.15.121:0/3027092803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f165c105a40 msgr2=0x7f165c197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f166253b700 1 --2- 172.21.15.121:0/3027092803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f165c105a40 0x7f165c197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f166253b700 1 -- 172.21.15.121:0/3027092803 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c0095f0 con 0x7f165c0ffbd0 2024-04-05T21:57:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.567+0000 7f166253b700 1 --2- 172.21.15.121:0/3027092803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f165c0ffbd0 0x7f165c1974c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f164c006010 tx=0x7f164c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.568+0000 7f16537fe700 1 -- 172.21.15.121:0/3027092803 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f164c01d070 con 0x7f165c0ffbd0 2024-04-05T21:57:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.568+0000 7f166479f700 1 -- 172.21.15.121:0/3027092803 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165c1a0530 con 0x7f165c0ffbd0 2024-04-05T21:57:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.568+0000 7f16537fe700 1 -- 172.21.15.121:0/3027092803 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f164c00f460 con 0x7f165c0ffbd0 2024-04-05T21:57:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.568+0000 7f16537fe700 1 -- 172.21.15.121:0/3027092803 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f164c021620 con 0x7f165c0ffbd0 2024-04-05T21:57:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.568+0000 7f166479f700 1 -- 172.21.15.121:0/3027092803 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165c1a0940 con 0x7f165c0ffbd0 2024-04-05T21:57:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.568+0000 7f166479f700 1 -- 172.21.15.121:0/3027092803 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f165c108040 con 0x7f165c0ffbd0 2024-04-05T21:57:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.569+0000 7f16537fe700 1 -- 172.21.15.121:0/3027092803 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f164c00fa40 con 0x7f165c0ffbd0 2024-04-05T21:57:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.569+0000 7f16537fe700 1 --2- 172.21.15.121:0/3027092803 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f164806cce0 0x7f164806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.570+0000 7f16537fe700 1 -- 172.21.15.121:0/3027092803 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f164c08d790 con 0x7f165c0ffbd0 2024-04-05T21:57:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.570+0000 7f1661d3a700 1 --2- 172.21.15.121:0/3027092803 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f164806cce0 0x7f164806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.570+0000 7f1661d3a700 1 --2- 172.21.15.121:0/3027092803 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f164806cce0 0x7f164806f1a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1658005950 tx=0x7f16580058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.574+0000 7f16537fe700 1 -- 172.21.15.121:0/3027092803 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f164c057800 con 0x7f165c0ffbd0 2024-04-05T21:57:58.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.787+0000 7f166479f700 1 -- 172.21.15.121:0/3027092803 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f165c02cd80 con 0x7f164806cce0 2024-04-05T21:57:58.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.789+0000 7f16537fe700 1 -- 172.21.15.121:0/3027092803 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f165c02cd80 con 0x7f164806cce0 2024-04-05T21:57:58.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.791+0000 7f166479f700 1 -- 172.21.15.121:0/3027092803 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f164806cce0 msgr2=0x7f164806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:58.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.791+0000 7f166479f700 1 --2- 172.21.15.121:0/3027092803 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f164806cce0 0x7f164806f1a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1658005950 tx=0x7f16580058e0 comp rx=0 tx=0).stop 2024-04-05T21:57:58.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.791+0000 7f166479f700 1 -- 172.21.15.121:0/3027092803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f165c0ffbd0 msgr2=0x7f165c1974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:58.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.791+0000 7f166479f700 1 --2- 172.21.15.121:0/3027092803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f165c0ffbd0 0x7f165c1974c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f164c006010 tx=0x7f164c00be20 comp rx=0 tx=0).stop 2024-04-05T21:57:58.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.791+0000 7f166479f700 1 -- 172.21.15.121:0/3027092803 shutdown_connections 2024-04-05T21:57:58.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.791+0000 7f166479f700 1 --2- 172.21.15.121:0/3027092803 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f164806cce0 0x7f164806f1a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:58.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.791+0000 7f166479f700 1 --2- 172.21.15.121:0/3027092803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f165c0ffbd0 0x7f165c1974c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:58.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.791+0000 7f166479f700 1 --2- 172.21.15.121:0/3027092803 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f165c105a40 0x7f165c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:58.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.791+0000 7f166479f700 1 -- 172.21.15.121:0/3027092803 >> 172.21.15.121:0/3027092803 conn(0x7f165c0755a0 msgr2=0x7f165c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:58.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.791+0000 7f166479f700 1 -- 172.21.15.121:0/3027092803 shutdown_connections 2024-04-05T21:57:58.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.791+0000 7f166479f700 1 -- 172.21.15.121:0/3027092803 wait complete. 2024-04-05T21:57:58.802 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:57:58.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.895+0000 7f34f8b6e700 1 -- 172.21.15.121:0/1904203294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f4068670 msgr2=0x7f34f4068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:58.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.895+0000 7f34f8b6e700 1 --2- 172.21.15.121:0/1904203294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f4068670 0x7f34f4068a50 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009910 tx=0x7f34dc009c20 comp rx=0 tx=0).stop 2024-04-05T21:57:58.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.895+0000 7f34f8b6e700 1 -- 172.21.15.121:0/1904203294 shutdown_connections 2024-04-05T21:57:58.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.895+0000 7f34f8b6e700 1 --2- 172.21.15.121:0/1904203294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34f4069020 0x7f34f4105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:58.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.895+0000 7f34f8b6e700 1 --2- 172.21.15.121:0/1904203294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f4068670 0x7f34f4068a50 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:58.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.895+0000 7f34f8b6e700 1 -- 172.21.15.121:0/1904203294 >> 172.21.15.121:0/1904203294 conn(0x7f34f40755a0 msgr2=0x7f34f40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:58.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.895+0000 7f34f8b6e700 1 -- 172.21.15.121:0/1904203294 shutdown_connections 2024-04-05T21:57:58.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.895+0000 7f34f8b6e700 1 -- 172.21.15.121:0/1904203294 wait complete. 2024-04-05T21:57:58.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f8b6e700 1 Processor -- start 2024-04-05T21:57:58.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f8b6e700 1 -- start start 2024-04-05T21:57:58.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f8b6e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f4068670 0x7f34f4196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:58.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f8b6e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34f4069020 0x7f34f4196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:58.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f8b6e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f4197290 con 0x7f34f4068670 2024-04-05T21:57:58.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f8b6e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f419b020 con 0x7f34f4069020 2024-04-05T21:57:58.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f4068670 0x7f34f4196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:58.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f4068670 0x7f34f4196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52058/0 (socket says 172.21.15.121:52058) 2024-04-05T21:57:58.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f259c700 1 -- 172.21.15.121:0/4100059698 learned_addr learned my addr 172.21.15.121:0/4100059698 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:57:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f1d9b700 1 --2- 172.21.15.121:0/4100059698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34f4069020 0x7f34f4196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f259c700 1 -- 172.21.15.121:0/4100059698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34f4069020 msgr2=0x7f34f4196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f259c700 1 --2- 172.21.15.121:0/4100059698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34f4069020 0x7f34f4196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f259c700 1 -- 172.21.15.121:0/4100059698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34dc0095f0 con 0x7f34f4068670 2024-04-05T21:57:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.896+0000 7f34f1d9b700 1 --2- 172.21.15.121:0/4100059698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34f4069020 0x7f34f4196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:57:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.897+0000 7f34f259c700 1 --2- 172.21.15.121:0/4100059698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f4068670 0x7f34f4196670 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f34dc006010 tx=0x7f34dc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.897+0000 7f34eb7fe700 1 -- 172.21.15.121:0/4100059698 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34dc01d070 con 0x7f34f4068670 2024-04-05T21:57:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.897+0000 7f34eb7fe700 1 -- 172.21.15.121:0/4100059698 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34dc00f460 con 0x7f34f4068670 2024-04-05T21:57:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.897+0000 7f34f8b6e700 1 -- 172.21.15.121:0/4100059698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f419b2a0 con 0x7f34f4068670 2024-04-05T21:57:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.897+0000 7f34eb7fe700 1 -- 172.21.15.121:0/4100059698 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34dc021640 con 0x7f34f4068670 2024-04-05T21:57:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.897+0000 7f34f8b6e700 1 -- 172.21.15.121:0/4100059698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f419b790 con 0x7f34f4068670 2024-04-05T21:57:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.898+0000 7f34f8b6e700 1 -- 172.21.15.121:0/4100059698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f4109130 con 0x7f34f4068670 2024-04-05T21:57:58.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.898+0000 7f34eb7fe700 1 -- 172.21.15.121:0/4100059698 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f34dc00fa60 con 0x7f34f4068670 2024-04-05T21:57:58.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.899+0000 7f34eb7fe700 1 --2- 172.21.15.121:0/4100059698 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f34e006cce0 0x7f34e006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:58.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.899+0000 7f34eb7fe700 1 -- 172.21.15.121:0/4100059698 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f34dc08d880 con 0x7f34f4068670 2024-04-05T21:57:58.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.899+0000 7f34f1d9b700 1 --2- 172.21.15.121:0/4100059698 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f34e006cce0 0x7f34e006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:58.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.899+0000 7f34f1d9b700 1 --2- 172.21.15.121:0/4100059698 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f34e006cce0 0x7f34e006f1a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f34e4005fd0 tx=0x7f34e4005dc0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:58.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:58.903+0000 7f34eb7fe700 1 -- 172.21.15.121:0/4100059698 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f34dc0578f0 con 0x7f34f4068670 2024-04-05T21:57:59.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.123+0000 7f34f8b6e700 1 -- 172.21.15.121:0/4100059698 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f34f402cd60 con 0x7f34e006cce0 2024-04-05T21:57:59.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.124+0000 7f34eb7fe700 1 -- 172.21.15.121:0/4100059698 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f34f402cd60 con 0x7f34e006cce0 2024-04-05T21:57:59.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.126+0000 7f34f8b6e700 1 -- 172.21.15.121:0/4100059698 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f34e006cce0 msgr2=0x7f34e006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:59.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.126+0000 7f34f8b6e700 1 --2- 172.21.15.121:0/4100059698 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f34e006cce0 0x7f34e006f1a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f34e4005fd0 tx=0x7f34e4005dc0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.126+0000 7f34f8b6e700 1 -- 172.21.15.121:0/4100059698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f4068670 msgr2=0x7f34f4196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:59.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.126+0000 7f34f8b6e700 1 --2- 172.21.15.121:0/4100059698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f4068670 0x7f34f4196670 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f34dc006010 tx=0x7f34dc00be20 comp rx=0 tx=0).stop 2024-04-05T21:57:59.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.126+0000 7f34f8b6e700 1 -- 172.21.15.121:0/4100059698 shutdown_connections 2024-04-05T21:57:59.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.126+0000 7f34f8b6e700 1 --2- 172.21.15.121:0/4100059698 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f34e006cce0 0x7f34e006f1a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.126+0000 7f34f8b6e700 1 --2- 172.21.15.121:0/4100059698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34f4068670 0x7f34f4196670 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.126+0000 7f34f8b6e700 1 --2- 172.21.15.121:0/4100059698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f34f4069020 0x7f34f4196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.126+0000 7f34f8b6e700 1 -- 172.21.15.121:0/4100059698 >> 172.21.15.121:0/4100059698 conn(0x7f34f40755a0 msgr2=0x7f34f40fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:59.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.126+0000 7f34f8b6e700 1 -- 172.21.15.121:0/4100059698 shutdown_connections 2024-04-05T21:57:59.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.126+0000 7f34f8b6e700 1 -- 172.21.15.121:0/4100059698 wait complete. 2024-04-05T21:57:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.228+0000 7efd013a0700 1 -- 172.21.15.121:0/1416793886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcfc100de0 msgr2=0x7efcfc1011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.228+0000 7efd013a0700 1 --2- 172.21.15.121:0/1416793886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcfc100de0 0x7efcfc1011c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7efcec00b0a0 tx=0x7efcec00b3b0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.228+0000 7efd013a0700 1 -- 172.21.15.121:0/1416793886 shutdown_connections 2024-04-05T21:57:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.228+0000 7efd013a0700 1 --2- 172.21.15.121:0/1416793886 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efcfc101790 0x7efcfc1057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.228+0000 7efd013a0700 1 --2- 172.21.15.121:0/1416793886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcfc100de0 0x7efcfc1011c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.228+0000 7efd013a0700 1 -- 172.21.15.121:0/1416793886 >> 172.21.15.121:0/1416793886 conn(0x7efcfc0fc6b0 msgr2=0x7efcfc0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.228+0000 7efd013a0700 1 -- 172.21.15.121:0/1416793886 shutdown_connections 2024-04-05T21:57:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.228+0000 7efd013a0700 1 -- 172.21.15.121:0/1416793886 wait complete. 2024-04-05T21:57:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.228+0000 7efd013a0700 1 Processor -- start 2024-04-05T21:57:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efd013a0700 1 -- start start 2024-04-05T21:57:59.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efd013a0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efcfc100de0 0x7efcfc1966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:59.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efd013a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcfc101790 0x7efcfc196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:59.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efd013a0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc1971f0 con 0x7efcfc101790 2024-04-05T21:57:59.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efd013a0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc197360 con 0x7efcfc100de0 2024-04-05T21:57:59.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efcfa7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcfc101790 0x7efcfc196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:59.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efcfa7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcfc101790 0x7efcfc196bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52066/0 (socket says 172.21.15.121:52066) 2024-04-05T21:57:59.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efcfa7fc700 1 -- 172.21.15.121:0/2585081832 learned_addr learned my addr 172.21.15.121:0/2585081832 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:57:59.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efcfaffd700 1 --2- 172.21.15.121:0/2585081832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efcfc100de0 0x7efcfc1966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:59.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efcfa7fc700 1 -- 172.21.15.121:0/2585081832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efcfc100de0 msgr2=0x7efcfc1966b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:59.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efcfa7fc700 1 --2- 172.21.15.121:0/2585081832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efcfc100de0 0x7efcfc1966b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efcfa7fc700 1 -- 172.21.15.121:0/2585081832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec009d00 con 0x7efcfc101790 2024-04-05T21:57:59.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efcfa7fc700 1 --2- 172.21.15.121:0/2585081832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcfc101790 0x7efcfc196bf0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7efce400c960 tx=0x7efce400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:59.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efcf3fff700 1 -- 172.21.15.121:0/2585081832 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efce4007a70 con 0x7efcfc101790 2024-04-05T21:57:59.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efd013a0700 1 -- 172.21.15.121:0/2585081832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcfc19a100 con 0x7efcfc101790 2024-04-05T21:57:59.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efcf3fff700 1 -- 172.21.15.121:0/2585081832 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efce4007bd0 con 0x7efcfc101790 2024-04-05T21:57:59.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.229+0000 7efcf3fff700 1 -- 172.21.15.121:0/2585081832 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efce4018630 con 0x7efcfc101790 2024-04-05T21:57:59.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.230+0000 7efd013a0700 1 -- 172.21.15.121:0/2585081832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcfc19a650 con 0x7efcfc101790 2024-04-05T21:57:59.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.230+0000 7efd013a0700 1 -- 172.21.15.121:0/2585081832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcfc109180 con 0x7efcfc101790 2024-04-05T21:57:59.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.231+0000 7efcf3fff700 1 -- 172.21.15.121:0/2585081832 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7efce400f450 con 0x7efcfc101790 2024-04-05T21:57:59.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.232+0000 7efcf3fff700 1 --2- 172.21.15.121:0/2585081832 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efce806cce0 0x7efce806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:59.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.232+0000 7efcf3fff700 1 -- 172.21.15.121:0/2585081832 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7efce408c6c0 con 0x7efcfc101790 2024-04-05T21:57:59.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.232+0000 7efcfaffd700 1 --2- 172.21.15.121:0/2585081832 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efce806cce0 0x7efce806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:59.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.232+0000 7efcfaffd700 1 --2- 172.21.15.121:0/2585081832 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efce806cce0 0x7efce806f1a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efcec009fd0 tx=0x7efcec0060c0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:59.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.236+0000 7efcf3fff700 1 -- 172.21.15.121:0/2585081832 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efce4056850 con 0x7efcfc101790 2024-04-05T21:57:59.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:58 smithi121 ceph-mon[145859]: pgmap v953: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:58 smithi188 ceph-mon[107793]: pgmap v953: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:57:59.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.438+0000 7efd013a0700 1 -- 172.21.15.121:0/2585081832 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efcfc02cdb0 con 0x7efce806cce0 2024-04-05T21:57:59.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.448+0000 7efcf3fff700 1 -- 172.21.15.121:0/2585081832 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7efcfc02cdb0 con 0x7efce806cce0 2024-04-05T21:57:59.450 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:57:59.450 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (28m) 7m ago 40m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:57:59.450 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (40m) 7m ago 40m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:57:59.450 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (39m) 7m ago 39m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:57:59.450 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (26m) 7m ago 40m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:57:59.450 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (26m) 7m ago 39m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:57:59.450 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (28m) 7m ago 40m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (23m) 7m ago 34m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (22m) 7m ago 34m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (22m) 7m ago 34m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (7m) 7m ago 34m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (30m) 7m ago 41m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (29m) 7m ago 39m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (27m) 7m ago 42m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (27m) 7m ago 39m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (29m) 7m ago 40m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (29m) 7m ago 39m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (26m) 7m ago 38m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (25m) 7m ago 37m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (25m) 7m ago 37m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (24m) 7m ago 36m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:57:59.451 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (24m) 7m ago 36m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:57:59.452 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (23m) 7m ago 35m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:57:59.452 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (29m) 7m ago 39m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:57:59.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.450+0000 7efd013a0700 1 -- 172.21.15.121:0/2585081832 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efce806cce0 msgr2=0x7efce806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:59.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.450+0000 7efd013a0700 1 --2- 172.21.15.121:0/2585081832 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efce806cce0 0x7efce806f1a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efcec009fd0 tx=0x7efcec0060c0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.450+0000 7efd013a0700 1 -- 172.21.15.121:0/2585081832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcfc101790 msgr2=0x7efcfc196bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:59.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.450+0000 7efd013a0700 1 --2- 172.21.15.121:0/2585081832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcfc101790 0x7efcfc196bf0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7efce400c960 tx=0x7efce400cc70 comp rx=0 tx=0).stop 2024-04-05T21:57:59.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.451+0000 7efd013a0700 1 -- 172.21.15.121:0/2585081832 shutdown_connections 2024-04-05T21:57:59.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.451+0000 7efd013a0700 1 --2- 172.21.15.121:0/2585081832 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efce806cce0 0x7efce806f1a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.451+0000 7efd013a0700 1 --2- 172.21.15.121:0/2585081832 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efcfc100de0 0x7efcfc1966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.451+0000 7efd013a0700 1 --2- 172.21.15.121:0/2585081832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcfc101790 0x7efcfc196bf0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.451+0000 7efd013a0700 1 -- 172.21.15.121:0/2585081832 >> 172.21.15.121:0/2585081832 conn(0x7efcfc0fc6b0 msgr2=0x7efcfc0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:59.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.451+0000 7efd013a0700 1 -- 172.21.15.121:0/2585081832 shutdown_connections 2024-04-05T21:57:59.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.451+0000 7efd013a0700 1 -- 172.21.15.121:0/2585081832 wait complete. 2024-04-05T21:57:59.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.551+0000 7f93a465d700 1 -- 172.21.15.121:0/765859917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c103990 msgr2=0x7f939c1079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:59.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.551+0000 7f93a465d700 1 --2- 172.21.15.121:0/765859917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c103990 0x7f939c1079e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f938c009910 tx=0x7f938c009c20 comp rx=0 tx=0).stop 2024-04-05T21:57:59.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.552+0000 7f93a465d700 1 -- 172.21.15.121:0/765859917 shutdown_connections 2024-04-05T21:57:59.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.552+0000 7f93a465d700 1 --2- 172.21.15.121:0/765859917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c103990 0x7f939c1079e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.552+0000 7f93a465d700 1 --2- 172.21.15.121:0/765859917 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f939c102fe0 0x7f939c1033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.552+0000 7f93a465d700 1 -- 172.21.15.121:0/765859917 >> 172.21.15.121:0/765859917 conn(0x7f939c0fe8d0 msgr2=0x7f939c100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:59.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.552+0000 7f93a465d700 1 -- 172.21.15.121:0/765859917 shutdown_connections 2024-04-05T21:57:59.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.552+0000 7f93a465d700 1 -- 172.21.15.121:0/765859917 wait complete. 2024-04-05T21:57:59.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.552+0000 7f93a465d700 1 Processor -- start 2024-04-05T21:57:59.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.552+0000 7f93a465d700 1 -- start start 2024-04-05T21:57:59.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a465d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f939c102fe0 0x7f939c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a465d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c103990 0x7f939c19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a465d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c19f220 con 0x7f939c103990 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a465d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c198760 con 0x7f939c102fe0 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a1bf8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c103990 0x7f939c19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a1bf8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c103990 0x7f939c19ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52080/0 (socket says 172.21.15.121:52080) 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a1bf8700 1 -- 172.21.15.121:0/1257225166 learned_addr learned my addr 172.21.15.121:0/1257225166 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a23f9700 1 --2- 172.21.15.121:0/1257225166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f939c102fe0 0x7f939c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a1bf8700 1 -- 172.21.15.121:0/1257225166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f939c102fe0 msgr2=0x7f939c19e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a1bf8700 1 --2- 172.21.15.121:0/1257225166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f939c102fe0 0x7f939c19e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a1bf8700 1 -- 172.21.15.121:0/1257225166 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c0095f0 con 0x7f939c103990 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a1bf8700 1 --2- 172.21.15.121:0/1257225166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c103990 0x7f939c19ec20 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f938c00bda0 tx=0x7f938c00be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93937fe700 1 -- 172.21.15.121:0/1257225166 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f938c01d070 con 0x7f939c103990 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93937fe700 1 -- 172.21.15.121:0/1257225166 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f938c00f460 con 0x7f939c103990 2024-04-05T21:57:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93937fe700 1 -- 172.21.15.121:0/1257225166 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f938c021640 con 0x7f939c103990 2024-04-05T21:57:59.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a465d700 1 -- 172.21.15.121:0/1257225166 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939c1989e0 con 0x7f939c103990 2024-04-05T21:57:59.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.553+0000 7f93a465d700 1 -- 172.21.15.121:0/1257225166 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939c198ed0 con 0x7f939c103990 2024-04-05T21:57:59.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.554+0000 7f93a465d700 1 -- 172.21.15.121:0/1257225166 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939c10b3e0 con 0x7f939c103990 2024-04-05T21:57:59.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.555+0000 7f93937fe700 1 -- 172.21.15.121:0/1257225166 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f938c00f9b0 con 0x7f939c103990 2024-04-05T21:57:59.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.555+0000 7f93937fe700 1 --2- 172.21.15.121:0/1257225166 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f938806cce0 0x7f938806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:59.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.555+0000 7f93937fe700 1 -- 172.21.15.121:0/1257225166 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f938c08d770 con 0x7f939c103990 2024-04-05T21:57:59.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.555+0000 7f93a23f9700 1 --2- 172.21.15.121:0/1257225166 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f938806cce0 0x7f938806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:59.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.556+0000 7f93a23f9700 1 --2- 172.21.15.121:0/1257225166 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f938806cce0 0x7f938806f1a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9398005ea0 tx=0x7f9398005e30 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:59.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.560+0000 7f93937fe700 1 -- 172.21.15.121:0/1257225166 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f938c05b2f0 con 0x7f939c103990 2024-04-05T21:57:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.834+0000 7f93a465d700 1 -- 172.21.15.121:0/1257225166 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f939c04ea90 con 0x7f939c103990 2024-04-05T21:57:59.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.834+0000 7f93937fe700 1 -- 172.21.15.121:0/1257225166 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f938c05ae80 con 0x7f939c103990 2024-04-05T21:57:59.835 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:57:59.836 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:57:59.837 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:57:59.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.836+0000 7f93a465d700 1 -- 172.21.15.121:0/1257225166 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f938806cce0 msgr2=0x7f938806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:59.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.836+0000 7f93a465d700 1 --2- 172.21.15.121:0/1257225166 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f938806cce0 0x7f938806f1a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9398005ea0 tx=0x7f9398005e30 comp rx=0 tx=0).stop 2024-04-05T21:57:59.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.836+0000 7f93a465d700 1 -- 172.21.15.121:0/1257225166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c103990 msgr2=0x7f939c19ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:59.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.836+0000 7f93a465d700 1 --2- 172.21.15.121:0/1257225166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c103990 0x7f939c19ec20 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f938c00bda0 tx=0x7f938c00be80 comp rx=0 tx=0).stop 2024-04-05T21:57:59.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.837+0000 7f93a465d700 1 -- 172.21.15.121:0/1257225166 shutdown_connections 2024-04-05T21:57:59.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.837+0000 7f93a465d700 1 --2- 172.21.15.121:0/1257225166 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f938806cce0 0x7f938806f1a0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.837+0000 7f93a465d700 1 --2- 172.21.15.121:0/1257225166 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f939c102fe0 0x7f939c19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.837+0000 7f93a465d700 1 --2- 172.21.15.121:0/1257225166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f939c103990 0x7f939c19ec20 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.837+0000 7f93a465d700 1 -- 172.21.15.121:0/1257225166 >> 172.21.15.121:0/1257225166 conn(0x7f939c0fe8d0 msgr2=0x7f939c0fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:59.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.837+0000 7f93a465d700 1 -- 172.21.15.121:0/1257225166 shutdown_connections 2024-04-05T21:57:59.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.837+0000 7f93a465d700 1 -- 172.21.15.121:0/1257225166 wait complete. 2024-04-05T21:57:59.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.939+0000 7faf081f2700 1 -- 172.21.15.121:0/4056986794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf00105b50 msgr2=0x7faf00105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:59.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.939+0000 7faf081f2700 1 --2- 172.21.15.121:0/4056986794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf00105b50 0x7faf00105f30 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7faefc009910 tx=0x7faefc009c20 comp rx=0 tx=0).stop 2024-04-05T21:57:59.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.939+0000 7faf081f2700 1 -- 172.21.15.121:0/4056986794 shutdown_connections 2024-04-05T21:57:59.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.939+0000 7faf081f2700 1 --2- 172.21.15.121:0/4056986794 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf000fef70 0x7faf000ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.939+0000 7faf081f2700 1 --2- 172.21.15.121:0/4056986794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf00105b50 0x7faf00105f30 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.939+0000 7faf081f2700 1 -- 172.21.15.121:0/4056986794 >> 172.21.15.121:0/4056986794 conn(0x7faf00075e30 msgr2=0x7faf000fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:57:59.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.939+0000 7faf081f2700 1 -- 172.21.15.121:0/4056986794 shutdown_connections 2024-04-05T21:57:59.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.939+0000 7faf081f2700 1 -- 172.21.15.121:0/4056986794 wait complete. 2024-04-05T21:57:59.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.939+0000 7faf081f2700 1 Processor -- start 2024-04-05T21:57:59.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.939+0000 7faf081f2700 1 -- start start 2024-04-05T21:57:59.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.939+0000 7faf081f2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf000fef70 0x7faf0019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:59.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.939+0000 7faf081f2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf00105b50 0x7faf0019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:59.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.939+0000 7faf081f2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0019d070 con 0x7faf00105b50 2024-04-05T21:57:59.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faf081f2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf00196590 con 0x7faf000fef70 2024-04-05T21:57:59.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faf0578d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf00105b50 0x7faf0019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:59.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faf0578d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf00105b50 0x7faf0019ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52094/0 (socket says 172.21.15.121:52094) 2024-04-05T21:57:59.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faf0578d700 1 -- 172.21.15.121:0/3385555911 learned_addr learned my addr 172.21.15.121:0/3385555911 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:57:59.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faf05f8e700 1 --2- 172.21.15.121:0/3385555911 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf000fef70 0x7faf0019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:59.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faf0578d700 1 -- 172.21.15.121:0/3385555911 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf000fef70 msgr2=0x7faf0019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:57:59.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faf0578d700 1 --2- 172.21.15.121:0/3385555911 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf000fef70 0x7faf0019c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:57:59.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faf0578d700 1 -- 172.21.15.121:0/3385555911 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc0095f0 con 0x7faf00105b50 2024-04-05T21:57:59.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faf0578d700 1 --2- 172.21.15.121:0/3385555911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf00105b50 0x7faf0019ca50 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7faef000c8a0 tx=0x7faef000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:59.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faef6ffd700 1 -- 172.21.15.121:0/3385555911 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faef000cea0 con 0x7faf00105b50 2024-04-05T21:57:59.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faef6ffd700 1 -- 172.21.15.121:0/3385555911 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faef0004830 con 0x7faf00105b50 2024-04-05T21:57:59.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faf081f2700 1 -- 172.21.15.121:0/3385555911 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf00196870 con 0x7faf00105b50 2024-04-05T21:57:59.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faef6ffd700 1 -- 172.21.15.121:0/3385555911 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faef0005500 con 0x7faf00105b50 2024-04-05T21:57:59.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.940+0000 7faf081f2700 1 -- 172.21.15.121:0/3385555911 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf00196dc0 con 0x7faf00105b50 2024-04-05T21:57:59.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.941+0000 7faf081f2700 1 -- 172.21.15.121:0/3385555911 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf00109120 con 0x7faf00105b50 2024-04-05T21:57:59.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.942+0000 7faef6ffd700 1 -- 172.21.15.121:0/3385555911 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7faef00049a0 con 0x7faf00105b50 2024-04-05T21:57:59.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.942+0000 7faef6ffd700 1 --2- 172.21.15.121:0/3385555911 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faeec06cce0 0x7faeec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:57:59.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.942+0000 7faf05f8e700 1 --2- 172.21.15.121:0/3385555911 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faeec06cce0 0x7faeec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:57:59.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.942+0000 7faef6ffd700 1 -- 172.21.15.121:0/3385555911 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7faef008c1d0 con 0x7faf00105b50 2024-04-05T21:57:59.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.943+0000 7faf05f8e700 1 --2- 172.21.15.121:0/3385555911 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faeec06cce0 0x7faeec06f1a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7faefc006010 tx=0x7faefc01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:57:59.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:57:59.947+0000 7faef6ffd700 1 -- 172.21.15.121:0/3385555911 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faef0059d50 con 0x7faf00105b50 2024-04-05T21:58:00.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.182+0000 7faf081f2700 1 -- 172.21.15.121:0/3385555911 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7faf0004ea90 con 0x7faf00105b50 2024-04-05T21:58:00.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.182+0000 7faef6ffd700 1 -- 172.21.15.121:0/3385555911 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7faef001d750 con 0x7faf00105b50 2024-04-05T21:58:00.184 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:58:00.184 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:58:00.184 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:58:00.184 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:58:00.184 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:58:00.184 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:58:00.184 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:58:00.184 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:58:00.185 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:58:00.186 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:58:00.187 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:58:00.187 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:58:00.187 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:58:00.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.184+0000 7faf081f2700 1 -- 172.21.15.121:0/3385555911 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faeec06cce0 msgr2=0x7faeec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:00.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.184+0000 7faf081f2700 1 --2- 172.21.15.121:0/3385555911 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faeec06cce0 0x7faeec06f1a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7faefc006010 tx=0x7faefc01a040 comp rx=0 tx=0).stop 2024-04-05T21:58:00.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.184+0000 7faf081f2700 1 -- 172.21.15.121:0/3385555911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf00105b50 msgr2=0x7faf0019ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:00.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.184+0000 7faf081f2700 1 --2- 172.21.15.121:0/3385555911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf00105b50 0x7faf0019ca50 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7faef000c8a0 tx=0x7faef000cc60 comp rx=0 tx=0).stop 2024-04-05T21:58:00.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.185+0000 7faf081f2700 1 -- 172.21.15.121:0/3385555911 shutdown_connections 2024-04-05T21:58:00.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.185+0000 7faf081f2700 1 --2- 172.21.15.121:0/3385555911 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faeec06cce0 0x7faeec06f1a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.185+0000 7faf081f2700 1 --2- 172.21.15.121:0/3385555911 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7faf000fef70 0x7faf0019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.185+0000 7faf081f2700 1 --2- 172.21.15.121:0/3385555911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf00105b50 0x7faf0019ca50 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.185+0000 7faf081f2700 1 -- 172.21.15.121:0/3385555911 >> 172.21.15.121:0/3385555911 conn(0x7faf00075e30 msgr2=0x7faf00104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:00.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.185+0000 7faf081f2700 1 -- 172.21.15.121:0/3385555911 shutdown_connections 2024-04-05T21:58:00.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.185+0000 7faf081f2700 1 -- 172.21.15.121:0/3385555911 wait complete. 2024-04-05T21:58:00.188 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:58:00.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:59 smithi121 ceph-mon[145859]: from='client.35550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:00.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:57:59 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1257225166' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:58:00.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:59 smithi188 ceph-mon[107793]: from='client.35550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:00.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:57:59 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1257225166' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:58:00.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.289+0000 7f9a308c9700 1 -- 172.21.15.121:0/4273507965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a2c0ffa40 msgr2=0x7f9a2c0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:00.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.289+0000 7f9a308c9700 1 --2- 172.21.15.121:0/4273507965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a2c0ffa40 0x7f9a2c0ffec0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009910 tx=0x7f9a1c009c20 comp rx=0 tx=0).stop 2024-04-05T21:58:00.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.289+0000 7f9a308c9700 1 -- 172.21.15.121:0/4273507965 shutdown_connections 2024-04-05T21:58:00.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.289+0000 7f9a308c9700 1 --2- 172.21.15.121:0/4273507965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a2c0ffa40 0x7f9a2c0ffec0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.289+0000 7f9a308c9700 1 --2- 172.21.15.121:0/4273507965 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a2c105a40 0x7f9a2c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.289+0000 7f9a308c9700 1 -- 172.21.15.121:0/4273507965 >> 172.21.15.121:0/4273507965 conn(0x7f9a2c0fb570 msgr2=0x7f9a2c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:00.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.289+0000 7f9a308c9700 1 -- 172.21.15.121:0/4273507965 shutdown_connections 2024-04-05T21:58:00.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.289+0000 7f9a308c9700 1 -- 172.21.15.121:0/4273507965 wait complete. 2024-04-05T21:58:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.290+0000 7f9a308c9700 1 Processor -- start 2024-04-05T21:58:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.290+0000 7f9a308c9700 1 -- start start 2024-04-05T21:58:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.290+0000 7f9a308c9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a2c0ffa40 0x7f9a2c1974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.290+0000 7f9a308c9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a2c105a40 0x7f9a2c1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.290+0000 7f9a308c9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c198010 con 0x7f9a2c0ffa40 2024-04-05T21:58:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.290+0000 7f9a308c9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c198150 con 0x7f9a2c105a40 2024-04-05T21:58:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.290+0000 7f9a2a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a2c0ffa40 0x7f9a2c1974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.290+0000 7f9a2a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a2c0ffa40 0x7f9a2c1974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52102/0 (socket says 172.21.15.121:52102) 2024-04-05T21:58:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.290+0000 7f9a2a59c700 1 -- 172.21.15.121:0/1663106883 learned_addr learned my addr 172.21.15.121:0/1663106883 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:58:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.291+0000 7f9a2a59c700 1 -- 172.21.15.121:0/1663106883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a2c105a40 msgr2=0x7f9a2c1979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.291+0000 7f9a2a59c700 1 --2- 172.21.15.121:0/1663106883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a2c105a40 0x7f9a2c1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.291+0000 7f9a2a59c700 1 -- 172.21.15.121:0/1663106883 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c0095f0 con 0x7f9a2c0ffa40 2024-04-05T21:58:00.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.291+0000 7f9a2a59c700 1 --2- 172.21.15.121:0/1663106883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a2c0ffa40 0x7f9a2c1974b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000d8d0 tx=0x7f9a2000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:00.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.291+0000 7f9a1b7fe700 1 -- 172.21.15.121:0/1663106883 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a20009940 con 0x7f9a2c0ffa40 2024-04-05T21:58:00.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.291+0000 7f9a308c9700 1 -- 172.21.15.121:0/1663106883 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2c1a0520 con 0x7f9a2c0ffa40 2024-04-05T21:58:00.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.291+0000 7f9a1b7fe700 1 -- 172.21.15.121:0/1663106883 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a20010460 con 0x7f9a2c0ffa40 2024-04-05T21:58:00.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.291+0000 7f9a1b7fe700 1 -- 172.21.15.121:0/1663106883 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a2000f5f0 con 0x7f9a2c0ffa40 2024-04-05T21:58:00.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.291+0000 7f9a308c9700 1 -- 172.21.15.121:0/1663106883 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2c1a09e0 con 0x7f9a2c0ffa40 2024-04-05T21:58:00.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.292+0000 7f9a308c9700 1 -- 172.21.15.121:0/1663106883 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2c107fd0 con 0x7f9a2c0ffa40 2024-04-05T21:58:00.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.292+0000 7f9a1b7fe700 1 -- 172.21.15.121:0/1663106883 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9a200105d0 con 0x7f9a2c0ffa40 2024-04-05T21:58:00.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.293+0000 7f9a1b7fe700 1 --2- 172.21.15.121:0/1663106883 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a1406cce0 0x7f9a1406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:00.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.293+0000 7f9a1b7fe700 1 -- 172.21.15.121:0/1663106883 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9a2008c5e0 con 0x7f9a2c0ffa40 2024-04-05T21:58:00.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.293+0000 7f9a29d9b700 1 --2- 172.21.15.121:0/1663106883 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a1406cce0 0x7f9a1406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:00.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.293+0000 7f9a29d9b700 1 --2- 172.21.15.121:0/1663106883 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a1406cce0 0x7f9a1406f1a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00b5c0 tx=0x7f9a1c005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:00.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.297+0000 7f9a1b7fe700 1 -- 172.21.15.121:0/1663106883 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9a20056650 con 0x7f9a2c0ffa40 2024-04-05T21:58:00.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.509+0000 7f9a308c9700 1 -- 172.21.15.121:0/1663106883 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a2c02cd80 con 0x7f9a1406cce0 2024-04-05T21:58:00.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.510+0000 7f9a1b7fe700 1 -- 172.21.15.121:0/1663106883 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9a2c02cd80 con 0x7f9a1406cce0 2024-04-05T21:58:00.512 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:58:00.512 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:58:00.512 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:58:00.512 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:58:00.512 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:58:00.512 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:58:00.512 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:58:00.512 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:58:00.512 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:58:00.512 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:58:00.513 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:58:00.513 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:58:00.513 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:58:00.513 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:58:00.513 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:58:00.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.512+0000 7f9a308c9700 1 -- 172.21.15.121:0/1663106883 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a1406cce0 msgr2=0x7f9a1406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:00.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.512+0000 7f9a308c9700 1 --2- 172.21.15.121:0/1663106883 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a1406cce0 0x7f9a1406f1a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00b5c0 tx=0x7f9a1c005960 comp rx=0 tx=0).stop 2024-04-05T21:58:00.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.512+0000 7f9a308c9700 1 -- 172.21.15.121:0/1663106883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a2c0ffa40 msgr2=0x7f9a2c1974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:00.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.512+0000 7f9a308c9700 1 --2- 172.21.15.121:0/1663106883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a2c0ffa40 0x7f9a2c1974b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000d8d0 tx=0x7f9a2000dc90 comp rx=0 tx=0).stop 2024-04-05T21:58:00.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.512+0000 7f9a308c9700 1 -- 172.21.15.121:0/1663106883 shutdown_connections 2024-04-05T21:58:00.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.512+0000 7f9a308c9700 1 --2- 172.21.15.121:0/1663106883 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a1406cce0 0x7f9a1406f1a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.512+0000 7f9a308c9700 1 --2- 172.21.15.121:0/1663106883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a2c0ffa40 0x7f9a2c1974b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.512+0000 7f9a308c9700 1 --2- 172.21.15.121:0/1663106883 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9a2c105a40 0x7f9a2c1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.512+0000 7f9a308c9700 1 -- 172.21.15.121:0/1663106883 >> 172.21.15.121:0/1663106883 conn(0x7f9a2c0fb570 msgr2=0x7f9a2c0fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:00.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.512+0000 7f9a308c9700 1 -- 172.21.15.121:0/1663106883 shutdown_connections 2024-04-05T21:58:00.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.513+0000 7f9a308c9700 1 -- 172.21.15.121:0/1663106883 wait complete. 2024-04-05T21:58:00.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.614+0000 7f1496b04700 1 -- 172.21.15.121:0/3324978325 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14900ffbd0 msgr2=0x7f1490100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:00.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.614+0000 7f1496b04700 1 --2- 172.21.15.121:0/3324978325 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14900ffbd0 0x7f1490100050 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f147c009910 tx=0x7f147c009c20 comp rx=0 tx=0).stop 2024-04-05T21:58:00.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.614+0000 7f1496b04700 1 -- 172.21.15.121:0/3324978325 shutdown_connections 2024-04-05T21:58:00.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.614+0000 7f1496b04700 1 --2- 172.21.15.121:0/3324978325 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14900ffbd0 0x7f1490100050 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.614+0000 7f1496b04700 1 --2- 172.21.15.121:0/3324978325 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1490105a40 0x7f1490105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.614+0000 7f1496b04700 1 -- 172.21.15.121:0/3324978325 >> 172.21.15.121:0/3324978325 conn(0x7f14900755a0 msgr2=0x7f14900759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:00.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.614+0000 7f1496b04700 1 -- 172.21.15.121:0/3324978325 shutdown_connections 2024-04-05T21:58:00.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.614+0000 7f1496b04700 1 -- 172.21.15.121:0/3324978325 wait complete. 2024-04-05T21:58:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f1496b04700 1 Processor -- start 2024-04-05T21:58:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f1496b04700 1 -- start start 2024-04-05T21:58:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f1496b04700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14900ffbd0 0x7f149019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f1496b04700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1490105a40 0x7f149019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f1496b04700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149019f1e0 con 0x7f1490105a40 2024-04-05T21:58:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f1496b04700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1490198690 con 0x7f14900ffbd0 2024-04-05T21:58:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f148ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1490105a40 0x7f149019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f148ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1490105a40 0x7f149019eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52114/0 (socket says 172.21.15.121:52114) 2024-04-05T21:58:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f148ffff700 1 -- 172.21.15.121:0/90014494 learned_addr learned my addr 172.21.15.121:0/90014494 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:58:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f14948a0700 1 --2- 172.21.15.121:0/90014494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14900ffbd0 0x7f149019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f148ffff700 1 -- 172.21.15.121:0/90014494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14900ffbd0 msgr2=0x7f149019e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f148ffff700 1 --2- 172.21.15.121:0/90014494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14900ffbd0 0x7f149019e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f148ffff700 1 -- 172.21.15.121:0/90014494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147c0095f0 con 0x7f1490105a40 2024-04-05T21:58:00.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.615+0000 7f148ffff700 1 --2- 172.21.15.121:0/90014494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1490105a40 0x7f149019eb50 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f147c006010 tx=0x7f147c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:00.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.616+0000 7f148dffb700 1 -- 172.21.15.121:0/90014494 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f147c01d070 con 0x7f1490105a40 2024-04-05T21:58:00.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.616+0000 7f148dffb700 1 -- 172.21.15.121:0/90014494 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f147c005940 con 0x7f1490105a40 2024-04-05T21:58:00.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.616+0000 7f1496b04700 1 -- 172.21.15.121:0/90014494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1490198910 con 0x7f1490105a40 2024-04-05T21:58:00.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.616+0000 7f148dffb700 1 -- 172.21.15.121:0/90014494 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f147c021410 con 0x7f1490105a40 2024-04-05T21:58:00.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.616+0000 7f1496b04700 1 -- 172.21.15.121:0/90014494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1490198e00 con 0x7f1490105a40 2024-04-05T21:58:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.617+0000 7f1496b04700 1 -- 172.21.15.121:0/90014494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f149004ea90 con 0x7f1490105a40 2024-04-05T21:58:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.617+0000 7f148dffb700 1 -- 172.21.15.121:0/90014494 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f147c003a60 con 0x7f1490105a40 2024-04-05T21:58:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.618+0000 7f148dffb700 1 --2- 172.21.15.121:0/90014494 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f148006cce0 0x7f148006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.618+0000 7f148dffb700 1 -- 172.21.15.121:0/90014494 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f147c08de90 con 0x7f1490105a40 2024-04-05T21:58:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.618+0000 7f14948a0700 1 --2- 172.21.15.121:0/90014494 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f148006cce0 0x7f148006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.618+0000 7f14948a0700 1 --2- 172.21.15.121:0/90014494 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f148006cce0 0x7f148006f1a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1484005950 tx=0x7f148400a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:00.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.622+0000 7f148dffb700 1 -- 172.21.15.121:0/90014494 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f147c05ba10 con 0x7f1490105a40 2024-04-05T21:58:00.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.889+0000 7f1496b04700 1 -- 172.21.15.121:0/90014494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1490199ba0 con 0x7f1490105a40 2024-04-05T21:58:00.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.889+0000 7f148dffb700 1 -- 172.21.15.121:0/90014494 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f147c026e10 con 0x7f1490105a40 2024-04-05T21:58:00.891 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:58:00.891 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:58:00.891 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:58:00.891 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:58:00.891 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:58:00.891 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:58:00.891 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:58:00.891 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:58:00.891 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:58:00.891 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:58:00.891 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:58:00.891 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:58:00.892 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:58:00.892 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:58:00.892 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:58:00.892 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:58:00.892 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:58:00.892 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:58:00.892 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:58:00.892 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:58:00.892 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:58:00.892 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:58:00.892 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:58:00.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.891+0000 7f1496b04700 1 -- 172.21.15.121:0/90014494 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f148006cce0 msgr2=0x7f148006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:00.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.891+0000 7f1496b04700 1 --2- 172.21.15.121:0/90014494 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f148006cce0 0x7f148006f1a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1484005950 tx=0x7f148400a300 comp rx=0 tx=0).stop 2024-04-05T21:58:00.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.891+0000 7f1496b04700 1 -- 172.21.15.121:0/90014494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1490105a40 msgr2=0x7f149019eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:00.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.891+0000 7f1496b04700 1 --2- 172.21.15.121:0/90014494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1490105a40 0x7f149019eb50 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f147c006010 tx=0x7f147c00bd70 comp rx=0 tx=0).stop 2024-04-05T21:58:00.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.891+0000 7f1496b04700 1 -- 172.21.15.121:0/90014494 shutdown_connections 2024-04-05T21:58:00.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.891+0000 7f1496b04700 1 --2- 172.21.15.121:0/90014494 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f148006cce0 0x7f148006f1a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.891+0000 7f1496b04700 1 --2- 172.21.15.121:0/90014494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f14900ffbd0 0x7f149019e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.891+0000 7f1496b04700 1 --2- 172.21.15.121:0/90014494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1490105a40 0x7f149019eb50 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:00.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.891+0000 7f1496b04700 1 -- 172.21.15.121:0/90014494 >> 172.21.15.121:0/90014494 conn(0x7f14900755a0 msgr2=0x7f14900fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:00.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.891+0000 7f1496b04700 1 -- 172.21.15.121:0/90014494 shutdown_connections 2024-04-05T21:58:00.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:00.891+0000 7f1496b04700 1 -- 172.21.15.121:0/90014494 wait complete. 2024-04-05T21:58:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:00 smithi121 ceph-mon[145859]: from='client.35554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:00 smithi121 ceph-mon[145859]: from='client.35558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:00 smithi121 ceph-mon[145859]: pgmap v954: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:00 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3385555911' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:58:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:58:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:00 smithi188 ceph-mon[107793]: from='client.35554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:00 smithi188 ceph-mon[107793]: from='client.35558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:00 smithi188 ceph-mon[107793]: pgmap v954: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:00 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3385555911' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:58:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:58:02.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:01 smithi121 ceph-mon[145859]: from='client.35570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:02.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:01 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/90014494' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:58:02.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:01 smithi188 ceph-mon[107793]: from='client.35570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:02.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:01 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/90014494' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:58:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:02 smithi121 ceph-mon[145859]: pgmap v955: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:02 smithi188 ceph-mon[107793]: pgmap v955: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:05.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:04 smithi121 ceph-mon[145859]: pgmap v956: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:04 smithi188 ceph-mon[107793]: pgmap v956: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:07.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:06 smithi121 ceph-mon[145859]: pgmap v957: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:06 smithi188 ceph-mon[107793]: pgmap v957: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:09.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:08 smithi121 ceph-mon[145859]: pgmap v958: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:09.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:08 smithi188 ceph-mon[107793]: pgmap v958: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:11.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:10 smithi121 ceph-mon[145859]: pgmap v959: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:10 smithi188 ceph-mon[107793]: pgmap v959: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:13.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:12 smithi121 ceph-mon[145859]: pgmap v960: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:12 smithi188 ceph-mon[107793]: pgmap v960: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:15.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:14 smithi121 ceph-mon[145859]: pgmap v961: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:14 smithi188 ceph-mon[107793]: pgmap v961: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:58:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:58:17.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:16 smithi121 ceph-mon[145859]: pgmap v962: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:17.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:16 smithi188 ceph-mon[107793]: pgmap v962: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:19.146 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:18 smithi121 ceph-mon[145859]: pgmap v963: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:19.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:18 smithi188 ceph-mon[107793]: pgmap v963: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:21.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:20 smithi121 ceph-mon[145859]: pgmap v964: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:20 smithi188 ceph-mon[107793]: pgmap v964: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:23.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:22 smithi121 ceph-mon[145859]: pgmap v965: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:23.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:22 smithi188 ceph-mon[107793]: pgmap v965: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:25.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:24 smithi121 ceph-mon[145859]: pgmap v966: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:24 smithi188 ceph-mon[107793]: pgmap v966: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:27.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:26 smithi121 ceph-mon[145859]: pgmap v967: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:27.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:26 smithi188 ceph-mon[107793]: pgmap v967: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:29.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:28 smithi121 ceph-mon[145859]: pgmap v968: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:28 smithi188 ceph-mon[107793]: pgmap v968: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:31.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.008+0000 7f4d07e07700 1 -- 172.21.15.121:0/3020455400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d000ffa40 msgr2=0x7f4d000ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.008+0000 7f4d07e07700 1 --2- 172.21.15.121:0/3020455400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d000ffa40 0x7f4d000ffec0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0009910 tx=0x7f4cf0009c20 comp rx=0 tx=0).stop 2024-04-05T21:58:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.008+0000 7f4d07e07700 1 -- 172.21.15.121:0/3020455400 shutdown_connections 2024-04-05T21:58:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.008+0000 7f4d07e07700 1 --2- 172.21.15.121:0/3020455400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d000ffa40 0x7f4d000ffec0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.008+0000 7f4d07e07700 1 --2- 172.21.15.121:0/3020455400 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d00105a40 0x7f4d00105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.008+0000 7f4d07e07700 1 -- 172.21.15.121:0/3020455400 >> 172.21.15.121:0/3020455400 conn(0x7f4d000fb570 msgr2=0x7f4d000fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.009+0000 7f4d07e07700 1 -- 172.21.15.121:0/3020455400 shutdown_connections 2024-04-05T21:58:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.009+0000 7f4d07e07700 1 -- 172.21.15.121:0/3020455400 wait complete. 2024-04-05T21:58:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.009+0000 7f4d07e07700 1 Processor -- start 2024-04-05T21:58:31.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.009+0000 7f4d07e07700 1 -- start start 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.009+0000 7f4d07e07700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d000ffa40 0x7f4d00195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.009+0000 7f4d07e07700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d00105a40 0x7f4d00195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.009+0000 7f4d07e07700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d001960d0 con 0x7f4d000ffa40 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.009+0000 7f4d07e07700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d00196210 con 0x7f4d00105a40 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.010+0000 7f4d05ba3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d000ffa40 0x7f4d00195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.010+0000 7f4d05ba3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d000ffa40 0x7f4d00195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57770/0 (socket says 172.21.15.121:57770) 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.010+0000 7f4d05ba3700 1 -- 172.21.15.121:0/3406696846 learned_addr learned my addr 172.21.15.121:0/3406696846 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.010+0000 7f4d053a2700 1 --2- 172.21.15.121:0/3406696846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d00105a40 0x7f4d00195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.010+0000 7f4d05ba3700 1 -- 172.21.15.121:0/3406696846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d00105a40 msgr2=0x7f4d00195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.010+0000 7f4d05ba3700 1 --2- 172.21.15.121:0/3406696846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d00105a40 0x7f4d00195a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.010+0000 7f4d05ba3700 1 -- 172.21.15.121:0/3406696846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf00095f0 con 0x7f4d000ffa40 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.010+0000 7f4d05ba3700 1 --2- 172.21.15.121:0/3406696846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d000ffa40 0x7f4d00195540 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc007b50 tx=0x7f4cfc007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.010+0000 7f4cf6ffd700 1 -- 172.21.15.121:0/3406696846 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cfc011070 con 0x7f4d000ffa40 2024-04-05T21:58:31.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.010+0000 7f4cf6ffd700 1 -- 172.21.15.121:0/3406696846 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4cfc00c2e0 con 0x7f4d000ffa40 2024-04-05T21:58:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.010+0000 7f4d07e07700 1 -- 172.21.15.121:0/3406696846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d0019a060 con 0x7f4d000ffa40 2024-04-05T21:58:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.010+0000 7f4cf6ffd700 1 -- 172.21.15.121:0/3406696846 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cfc00ab50 con 0x7f4d000ffa40 2024-04-05T21:58:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.010+0000 7f4d07e07700 1 -- 172.21.15.121:0/3406696846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d0019a5b0 con 0x7f4d000ffa40 2024-04-05T21:58:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.011+0000 7f4d07e07700 1 -- 172.21.15.121:0/3406696846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d0004ea90 con 0x7f4d000ffa40 2024-04-05T21:58:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.012+0000 7f4cf6ffd700 1 -- 172.21.15.121:0/3406696846 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4cfc005070 con 0x7f4d000ffa40 2024-04-05T21:58:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.013+0000 7f4cf6ffd700 1 --2- 172.21.15.121:0/3406696846 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4cec06cce0 0x7f4cec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.013+0000 7f4cf6ffd700 1 -- 172.21.15.121:0/3406696846 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4cfc08dec0 con 0x7f4d000ffa40 2024-04-05T21:58:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.013+0000 7f4d053a2700 1 --2- 172.21.15.121:0/3406696846 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4cec06cce0 0x7f4cec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:31.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.013+0000 7f4d053a2700 1 --2- 172.21.15.121:0/3406696846 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4cec06cce0 0x7f4cec06f1a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4d00196b60 tx=0x7f4cf001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:31.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.017+0000 7f4cf6ffd700 1 -- 172.21.15.121:0/3406696846 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4cfc057fb0 con 0x7f4d000ffa40 2024-04-05T21:58:31.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.227+0000 7f4d07e07700 1 -- 172.21.15.121:0/3406696846 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d0019aa90 con 0x7f4cec06cce0 2024-04-05T21:58:31.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.228+0000 7f4cf6ffd700 1 -- 172.21.15.121:0/3406696846 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4d0019aa90 con 0x7f4cec06cce0 2024-04-05T21:58:31.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.230+0000 7f4d07e07700 1 -- 172.21.15.121:0/3406696846 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4cec06cce0 msgr2=0x7f4cec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.230+0000 7f4d07e07700 1 --2- 172.21.15.121:0/3406696846 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4cec06cce0 0x7f4cec06f1a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4d00196b60 tx=0x7f4cf001a040 comp rx=0 tx=0).stop 2024-04-05T21:58:31.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.230+0000 7f4d07e07700 1 -- 172.21.15.121:0/3406696846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d000ffa40 msgr2=0x7f4d00195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.230+0000 7f4d07e07700 1 --2- 172.21.15.121:0/3406696846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d000ffa40 0x7f4d00195540 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc007b50 tx=0x7f4cfc007e60 comp rx=0 tx=0).stop 2024-04-05T21:58:31.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.230+0000 7f4d07e07700 1 -- 172.21.15.121:0/3406696846 shutdown_connections 2024-04-05T21:58:31.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.230+0000 7f4d07e07700 1 --2- 172.21.15.121:0/3406696846 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4cec06cce0 0x7f4cec06f1a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.230+0000 7f4d07e07700 1 --2- 172.21.15.121:0/3406696846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d000ffa40 0x7f4d00195540 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.230+0000 7f4d07e07700 1 --2- 172.21.15.121:0/3406696846 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4d00105a40 0x7f4d00195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.230+0000 7f4d07e07700 1 -- 172.21.15.121:0/3406696846 >> 172.21.15.121:0/3406696846 conn(0x7f4d000fb570 msgr2=0x7f4d000fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:31.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.230+0000 7f4d07e07700 1 -- 172.21.15.121:0/3406696846 shutdown_connections 2024-04-05T21:58:31.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.230+0000 7f4d07e07700 1 -- 172.21.15.121:0/3406696846 wait complete. 2024-04-05T21:58:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:30 smithi121 ceph-mon[145859]: pgmap v969: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:58:31.241 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:58:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:30 smithi188 ceph-mon[107793]: pgmap v969: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:58:31.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.333+0000 7f2c2d823700 1 -- 172.21.15.121:0/511502553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c28101d60 msgr2=0x7f2c28102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.333+0000 7f2c2d823700 1 --2- 172.21.15.121:0/511502553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c28101d60 0x7f2c28102140 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2c18009940 tx=0x7f2c18009c50 comp rx=0 tx=0).stop 2024-04-05T21:58:31.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.333+0000 7f2c2d823700 1 -- 172.21.15.121:0/511502553 shutdown_connections 2024-04-05T21:58:31.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.333+0000 7f2c2d823700 1 --2- 172.21.15.121:0/511502553 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c28102680 0x7f2c2810ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.333+0000 7f2c2d823700 1 --2- 172.21.15.121:0/511502553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c28101d60 0x7f2c28102140 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.333+0000 7f2c2d823700 1 -- 172.21.15.121:0/511502553 >> 172.21.15.121:0/511502553 conn(0x7f2c280fb550 msgr2=0x7f2c280fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:31.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.333+0000 7f2c2d823700 1 -- 172.21.15.121:0/511502553 shutdown_connections 2024-04-05T21:58:31.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.333+0000 7f2c2d823700 1 -- 172.21.15.121:0/511502553 wait complete. 2024-04-05T21:58:31.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.334+0000 7f2c2d823700 1 Processor -- start 2024-04-05T21:58:31.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.334+0000 7f2c2d823700 1 -- start start 2024-04-05T21:58:31.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.334+0000 7f2c2d823700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c28101d60 0x7f2c2819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:31.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.334+0000 7f2c2d823700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c28102680 0x7f2c2819ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:31.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.334+0000 7f2c2d823700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2819d100 con 0x7f2c28102680 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.334+0000 7f2c2d823700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c28196600 con 0x7f2c28101d60 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.335+0000 7f2c267fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c28102680 0x7f2c2819ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.335+0000 7f2c267fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c28102680 0x7f2c2819ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57790/0 (socket says 172.21.15.121:57790) 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.335+0000 7f2c267fc700 1 -- 172.21.15.121:0/1829519527 learned_addr learned my addr 172.21.15.121:0/1829519527 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.335+0000 7f2c267fc700 1 -- 172.21.15.121:0/1829519527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c28101d60 msgr2=0x7f2c2819c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.335+0000 7f2c26ffd700 1 --2- 172.21.15.121:0/1829519527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c28101d60 0x7f2c2819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.335+0000 7f2c267fc700 1 --2- 172.21.15.121:0/1829519527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c28101d60 0x7f2c2819c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.335+0000 7f2c267fc700 1 -- 172.21.15.121:0/1829519527 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c180095f0 con 0x7f2c28102680 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.335+0000 7f2c267fc700 1 --2- 172.21.15.121:0/1829519527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c28102680 0x7f2c2819ca70 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00b4d0 tx=0x7f2c1c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.335+0000 7f2c2c821700 1 -- 172.21.15.121:0/1829519527 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c1c011800 con 0x7f2c28102680 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.335+0000 7f2c2c821700 1 -- 172.21.15.121:0/1829519527 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c1c011e00 con 0x7f2c28102680 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.335+0000 7f2c2d823700 1 -- 172.21.15.121:0/1829519527 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c281968e0 con 0x7f2c28102680 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.335+0000 7f2c2c821700 1 -- 172.21.15.121:0/1829519527 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c1c00f2e0 con 0x7f2c28102680 2024-04-05T21:58:31.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.335+0000 7f2c2d823700 1 -- 172.21.15.121:0/1829519527 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c28196e30 con 0x7f2c28102680 2024-04-05T21:58:31.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.336+0000 7f2c2d823700 1 -- 172.21.15.121:0/1829519527 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c28108270 con 0x7f2c28102680 2024-04-05T21:58:31.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.337+0000 7f2c2c821700 1 -- 172.21.15.121:0/1829519527 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2c1c010380 con 0x7f2c28102680 2024-04-05T21:58:31.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.337+0000 7f2c2c821700 1 --2- 172.21.15.121:0/1829519527 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2c1006cce0 0x7f2c1006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:31.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.337+0000 7f2c2c821700 1 -- 172.21.15.121:0/1829519527 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2c1c08c140 con 0x7f2c28102680 2024-04-05T21:58:31.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.337+0000 7f2c26ffd700 1 --2- 172.21.15.121:0/1829519527 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2c1006cce0 0x7f2c1006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:31.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.337+0000 7f2c26ffd700 1 --2- 172.21.15.121:0/1829519527 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2c1006cce0 0x7f2c1006f1a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800b580 tx=0x7f2c1800b790 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:31.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.342+0000 7f2c2c821700 1 -- 172.21.15.121:0/1829519527 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2c1c059c40 con 0x7f2c28102680 2024-04-05T21:58:31.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.554+0000 7f2c2d823700 1 -- 172.21.15.121:0/1829519527 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c28197130 con 0x7f2c1006cce0 2024-04-05T21:58:31.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.555+0000 7f2c2c821700 1 -- 172.21.15.121:0/1829519527 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2c28197130 con 0x7f2c1006cce0 2024-04-05T21:58:31.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.557+0000 7f2c2d823700 1 -- 172.21.15.121:0/1829519527 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2c1006cce0 msgr2=0x7f2c1006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.557+0000 7f2c2d823700 1 --2- 172.21.15.121:0/1829519527 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2c1006cce0 0x7f2c1006f1a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2c1800b580 tx=0x7f2c1800b790 comp rx=0 tx=0).stop 2024-04-05T21:58:31.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.557+0000 7f2c2d823700 1 -- 172.21.15.121:0/1829519527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c28102680 msgr2=0x7f2c2819ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.557+0000 7f2c2d823700 1 --2- 172.21.15.121:0/1829519527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c28102680 0x7f2c2819ca70 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00b4d0 tx=0x7f2c1c00b890 comp rx=0 tx=0).stop 2024-04-05T21:58:31.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.557+0000 7f2c2d823700 1 -- 172.21.15.121:0/1829519527 shutdown_connections 2024-04-05T21:58:31.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.557+0000 7f2c2d823700 1 --2- 172.21.15.121:0/1829519527 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2c1006cce0 0x7f2c1006f1a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.557+0000 7f2c2d823700 1 --2- 172.21.15.121:0/1829519527 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2c28101d60 0x7f2c2819c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.557+0000 7f2c2d823700 1 --2- 172.21.15.121:0/1829519527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c28102680 0x7f2c2819ca70 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.557+0000 7f2c2d823700 1 -- 172.21.15.121:0/1829519527 >> 172.21.15.121:0/1829519527 conn(0x7f2c280fb550 msgr2=0x7f2c281053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:31.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.557+0000 7f2c2d823700 1 -- 172.21.15.121:0/1829519527 shutdown_connections 2024-04-05T21:58:31.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.557+0000 7f2c2d823700 1 -- 172.21.15.121:0/1829519527 wait complete. 2024-04-05T21:58:31.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.660+0000 7f067607a700 1 -- 172.21.15.121:0/3376702968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0670100dc0 msgr2=0x7f06701011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.660+0000 7f067607a700 1 --2- 172.21.15.121:0/3376702968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0670100dc0 0x7f06701011a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f0664009910 tx=0x7f0664009c20 comp rx=0 tx=0).stop 2024-04-05T21:58:31.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.660+0000 7f067607a700 1 -- 172.21.15.121:0/3376702968 shutdown_connections 2024-04-05T21:58:31.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.660+0000 7f067607a700 1 --2- 172.21.15.121:0/3376702968 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0670101770 0x7f06701057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.660+0000 7f067607a700 1 --2- 172.21.15.121:0/3376702968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0670100dc0 0x7f06701011a0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.660+0000 7f067607a700 1 -- 172.21.15.121:0/3376702968 >> 172.21.15.121:0/3376702968 conn(0x7f06700fc670 msgr2=0x7f06700fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:31.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.660+0000 7f067607a700 1 -- 172.21.15.121:0/3376702968 shutdown_connections 2024-04-05T21:58:31.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.660+0000 7f067607a700 1 -- 172.21.15.121:0/3376702968 wait complete. 2024-04-05T21:58:31.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.660+0000 7f067607a700 1 Processor -- start 2024-04-05T21:58:31.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f067607a700 1 -- start start 2024-04-05T21:58:31.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f067607a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0670100dc0 0x7f0670196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:31.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f067607a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0670101770 0x7f0670196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:31.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f067607a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06701972e0 con 0x7f0670101770 2024-04-05T21:58:31.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f067607a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067019a010 con 0x7f0670100dc0 2024-04-05T21:58:31.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f066effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0670101770 0x7f0670196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:31.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f066effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0670101770 0x7f0670196c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57808/0 (socket says 172.21.15.121:57808) 2024-04-05T21:58:31.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f066effd700 1 -- 172.21.15.121:0/54333723 learned_addr learned my addr 172.21.15.121:0/54333723 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:58:31.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f066f7fe700 1 --2- 172.21.15.121:0/54333723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0670100dc0 0x7f0670196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:31.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f066effd700 1 -- 172.21.15.121:0/54333723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0670100dc0 msgr2=0x7f0670196710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f066effd700 1 --2- 172.21.15.121:0/54333723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0670100dc0 0x7f0670196710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f066effd700 1 -- 172.21.15.121:0/54333723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06640095f0 con 0x7f0670101770 2024-04-05T21:58:31.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f066f7fe700 1 --2- 172.21.15.121:0/54333723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0670100dc0 0x7f0670196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:58:31.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.661+0000 7f066effd700 1 --2- 172.21.15.121:0/54333723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0670101770 0x7f0670196c50 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f066000d900 tx=0x7f066000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:31.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.662+0000 7f066cff9700 1 -- 172.21.15.121:0/54333723 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0660009900 con 0x7f0670101770 2024-04-05T21:58:31.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.662+0000 7f066cff9700 1 -- 172.21.15.121:0/54333723 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0660010460 con 0x7f0670101770 2024-04-05T21:58:31.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.662+0000 7f067607a700 1 -- 172.21.15.121:0/54333723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067019a2f0 con 0x7f0670101770 2024-04-05T21:58:31.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.662+0000 7f066cff9700 1 -- 172.21.15.121:0/54333723 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f066000f5f0 con 0x7f0670101770 2024-04-05T21:58:31.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.662+0000 7f067607a700 1 -- 172.21.15.121:0/54333723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067019a840 con 0x7f0670101770 2024-04-05T21:58:31.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.663+0000 7f067607a700 1 -- 172.21.15.121:0/54333723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0670109160 con 0x7f0670101770 2024-04-05T21:58:31.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.663+0000 7f066cff9700 1 -- 172.21.15.121:0/54333723 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f06600105d0 con 0x7f0670101770 2024-04-05T21:58:31.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.663+0000 7f066cff9700 1 --2- 172.21.15.121:0/54333723 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f065806cce0 0x7f065806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:31.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.663+0000 7f066cff9700 1 -- 172.21.15.121:0/54333723 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f066008c3c0 con 0x7f0670101770 2024-04-05T21:58:31.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.664+0000 7f066f7fe700 1 --2- 172.21.15.121:0/54333723 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f065806cce0 0x7f065806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:31.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.664+0000 7f066f7fe700 1 --2- 172.21.15.121:0/54333723 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f065806cce0 0x7f065806f1a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f066400b5c0 tx=0x7f066401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:31.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.668+0000 7f066cff9700 1 -- 172.21.15.121:0/54333723 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0660056430 con 0x7f0670101770 2024-04-05T21:58:31.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.871+0000 7f067607a700 1 -- 172.21.15.121:0/54333723 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f067019aa60 con 0x7f065806cce0 2024-04-05T21:58:31.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.881+0000 7f066cff9700 1 -- 172.21.15.121:0/54333723 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f067019aa60 con 0x7f065806cce0 2024-04-05T21:58:31.881 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:58:31.881 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (29m) 7m ago 40m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:58:31.881 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (41m) 7m ago 41m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (39m) 7m ago 39m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (27m) 7m ago 41m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (27m) 7m ago 39m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (29m) 7m ago 40m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (23m) 7m ago 34m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (23m) 7m ago 34m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (23m) 7m ago 34m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (7m) 7m ago 34m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (31m) 7m ago 42m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (30m) 7m ago 39m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (28m) 7m ago 42m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (27m) 7m ago 39m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (30m) 7m ago 40m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:58:31.882 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (30m) 7m ago 39m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:58:31.883 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (27m) 7m ago 38m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:58:31.883 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (26m) 7m ago 38m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:58:31.883 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (25m) 7m ago 37m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:58:31.883 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (25m) 7m ago 37m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:58:31.883 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (24m) 7m ago 36m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:58:31.883 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (24m) 7m ago 36m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:58:31.883 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (29m) 7m ago 40m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:58:31.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.883+0000 7f067607a700 1 -- 172.21.15.121:0/54333723 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f065806cce0 msgr2=0x7f065806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.883+0000 7f067607a700 1 --2- 172.21.15.121:0/54333723 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f065806cce0 0x7f065806f1a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f066400b5c0 tx=0x7f066401a040 comp rx=0 tx=0).stop 2024-04-05T21:58:31.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.883+0000 7f067607a700 1 -- 172.21.15.121:0/54333723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0670101770 msgr2=0x7f0670196c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.883+0000 7f067607a700 1 --2- 172.21.15.121:0/54333723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0670101770 0x7f0670196c50 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f066000d900 tx=0x7f066000dcc0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.883+0000 7f067607a700 1 -- 172.21.15.121:0/54333723 shutdown_connections 2024-04-05T21:58:31.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.883+0000 7f067607a700 1 --2- 172.21.15.121:0/54333723 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f065806cce0 0x7f065806f1a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.883+0000 7f067607a700 1 --2- 172.21.15.121:0/54333723 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0670100dc0 0x7f0670196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.883+0000 7f067607a700 1 --2- 172.21.15.121:0/54333723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0670101770 0x7f0670196c50 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.883+0000 7f067607a700 1 -- 172.21.15.121:0/54333723 >> 172.21.15.121:0/54333723 conn(0x7f06700fc670 msgr2=0x7f06700fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:31.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.883+0000 7f067607a700 1 -- 172.21.15.121:0/54333723 shutdown_connections 2024-04-05T21:58:31.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.883+0000 7f067607a700 1 -- 172.21.15.121:0/54333723 wait complete. 2024-04-05T21:58:31.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.984+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3159073758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b3c105a40 msgr2=0x7f0b3c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.984+0000 7f0b42a4c700 1 --2- 172.21.15.121:0/3159073758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b3c105a40 0x7f0b3c105e20 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009910 tx=0x7f0b2c009c20 comp rx=0 tx=0).stop 2024-04-05T21:58:31.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.984+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3159073758 shutdown_connections 2024-04-05T21:58:31.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.984+0000 7f0b42a4c700 1 --2- 172.21.15.121:0/3159073758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b3c0ffa40 0x7f0b3c0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.984+0000 7f0b42a4c700 1 --2- 172.21.15.121:0/3159073758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b3c105a40 0x7f0b3c105e20 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.984+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3159073758 >> 172.21.15.121:0/3159073758 conn(0x7f0b3c0fb570 msgr2=0x7f0b3c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:31.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.984+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3159073758 shutdown_connections 2024-04-05T21:58:31.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.984+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3159073758 wait complete. 2024-04-05T21:58:31.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b42a4c700 1 Processor -- start 2024-04-05T21:58:31.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b42a4c700 1 -- start start 2024-04-05T21:58:31.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b42a4c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b3c0ffa40 0x7f0b3c195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:31.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b42a4c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b3c105a40 0x7f0b3c195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b42a4c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c196170 con 0x7f0b3c105a40 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b42a4c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c199f00 con 0x7f0b3c0ffa40 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b3b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b3c105a40 0x7f0b3c195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b3b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b3c105a40 0x7f0b3c195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57822/0 (socket says 172.21.15.121:57822) 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b3b7fe700 1 -- 172.21.15.121:0/3568643820 learned_addr learned my addr 172.21.15.121:0/3568643820 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b3bfff700 1 --2- 172.21.15.121:0/3568643820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b3c0ffa40 0x7f0b3c195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b3b7fe700 1 -- 172.21.15.121:0/3568643820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b3c0ffa40 msgr2=0x7f0b3c195550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b3b7fe700 1 --2- 172.21.15.121:0/3568643820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b3c0ffa40 0x7f0b3c195550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b3b7fe700 1 -- 172.21.15.121:0/3568643820 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c0095f0 con 0x7f0b3c105a40 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.985+0000 7f0b3bfff700 1 --2- 172.21.15.121:0/3568643820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b3c0ffa40 0x7f0b3c195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.986+0000 7f0b3b7fe700 1 --2- 172.21.15.121:0/3568643820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b3c105a40 0x7f0b3c195a90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f0b24007b50 tx=0x7f0b24007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.986+0000 7f0b397fa700 1 -- 172.21.15.121:0/3568643820 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b24011070 con 0x7f0b3c105a40 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.986+0000 7f0b397fa700 1 -- 172.21.15.121:0/3568643820 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b2400c2e0 con 0x7f0b3c105a40 2024-04-05T21:58:31.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.986+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3568643820 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3c19a180 con 0x7f0b3c105a40 2024-04-05T21:58:31.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.986+0000 7f0b397fa700 1 -- 172.21.15.121:0/3568643820 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b2400e770 con 0x7f0b3c105a40 2024-04-05T21:58:31.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.986+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3568643820 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3c19a6d0 con 0x7f0b3c105a40 2024-04-05T21:58:31.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.987+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3568643820 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3c04ea90 con 0x7f0b3c105a40 2024-04-05T21:58:31.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.987+0000 7f0b397fa700 1 -- 172.21.15.121:0/3568643820 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0b24004a00 con 0x7f0b3c105a40 2024-04-05T21:58:31.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.987+0000 7f0b397fa700 1 --2- 172.21.15.121:0/3568643820 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0b2806cce0 0x7f0b2806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:31.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.988+0000 7f0b397fa700 1 -- 172.21.15.121:0/3568643820 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0b2408dd10 con 0x7f0b3c105a40 2024-04-05T21:58:31.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.988+0000 7f0b3bfff700 1 --2- 172.21.15.121:0/3568643820 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0b2806cce0 0x7f0b2806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:31.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.988+0000 7f0b3bfff700 1 --2- 172.21.15.121:0/3568643820 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0b2806cce0 0x7f0b2806f1a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00b5c0 tx=0x7f0b2c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:31.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:31.992+0000 7f0b397fa700 1 -- 172.21.15.121:0/3568643820 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0b24057d80 con 0x7f0b3c105a40 2024-04-05T21:58:32.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.264+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3568643820 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0b3c1968b0 con 0x7f0b3c105a40 2024-04-05T21:58:32.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.265+0000 7f0b397fa700 1 -- 172.21.15.121:0/3568643820 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f0b2405b3a0 con 0x7f0b3c105a40 2024-04-05T21:58:32.265 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:58:32.265 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:58:32.265 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:58:32.266 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:58:32.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.267+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3568643820 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0b2806cce0 msgr2=0x7f0b2806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:32.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.267+0000 7f0b42a4c700 1 --2- 172.21.15.121:0/3568643820 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0b2806cce0 0x7f0b2806f1a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00b5c0 tx=0x7f0b2c01a040 comp rx=0 tx=0).stop 2024-04-05T21:58:32.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.267+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3568643820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b3c105a40 msgr2=0x7f0b3c195a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:32.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.267+0000 7f0b42a4c700 1 --2- 172.21.15.121:0/3568643820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b3c105a40 0x7f0b3c195a90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f0b24007b50 tx=0x7f0b24007f10 comp rx=0 tx=0).stop 2024-04-05T21:58:32.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.267+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3568643820 shutdown_connections 2024-04-05T21:58:32.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.267+0000 7f0b42a4c700 1 --2- 172.21.15.121:0/3568643820 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0b2806cce0 0x7f0b2806f1a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.267+0000 7f0b42a4c700 1 --2- 172.21.15.121:0/3568643820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0b3c0ffa40 0x7f0b3c195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.267+0000 7f0b42a4c700 1 --2- 172.21.15.121:0/3568643820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b3c105a40 0x7f0b3c195a90 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.267+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3568643820 >> 172.21.15.121:0/3568643820 conn(0x7f0b3c0fb570 msgr2=0x7f0b3c0fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:32.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.267+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3568643820 shutdown_connections 2024-04-05T21:58:32.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.267+0000 7f0b42a4c700 1 -- 172.21.15.121:0/3568643820 wait complete. 2024-04-05T21:58:32.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.369+0000 7f636fa03700 1 -- 172.21.15.121:0/491496055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63680ffec0 msgr2=0x7f6368105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:32.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.369+0000 7f636fa03700 1 --2- 172.21.15.121:0/491496055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63680ffec0 0x7f6368105850 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f6364009910 tx=0x7f6364009c20 comp rx=0 tx=0).stop 2024-04-05T21:58:32.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.369+0000 7f636fa03700 1 -- 172.21.15.121:0/491496055 shutdown_connections 2024-04-05T21:58:32.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.369+0000 7f636fa03700 1 --2- 172.21.15.121:0/491496055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63680ffec0 0x7f6368105850 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.369+0000 7f636fa03700 1 --2- 172.21.15.121:0/491496055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63680ff510 0x7f63680ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.369+0000 7f636fa03700 1 -- 172.21.15.121:0/491496055 >> 172.21.15.121:0/491496055 conn(0x7f6368074d80 msgr2=0x7f6368075190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:32.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.369+0000 7f636fa03700 1 -- 172.21.15.121:0/491496055 shutdown_connections 2024-04-05T21:58:32.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.369+0000 7f636fa03700 1 -- 172.21.15.121:0/491496055 wait complete. 2024-04-05T21:58:32.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.370+0000 7f636fa03700 1 Processor -- start 2024-04-05T21:58:32.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.370+0000 7f636fa03700 1 -- start start 2024-04-05T21:58:32.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.370+0000 7f636fa03700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63680ff510 0x7f6368196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:32.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.370+0000 7f636fa03700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63680ffec0 0x7f6368196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:32.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.370+0000 7f636fa03700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63681972d0 con 0x7f63680ffec0 2024-04-05T21:58:32.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.370+0000 7f636fa03700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636819a000 con 0x7f63680ff510 2024-04-05T21:58:32.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.370+0000 7f636cf9e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63680ffec0 0x7f6368196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:32.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.370+0000 7f636cf9e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63680ffec0 0x7f6368196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57842/0 (socket says 172.21.15.121:57842) 2024-04-05T21:58:32.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.370+0000 7f636cf9e700 1 -- 172.21.15.121:0/3568651688 learned_addr learned my addr 172.21.15.121:0/3568651688 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:58:32.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.371+0000 7f636d79f700 1 --2- 172.21.15.121:0/3568651688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63680ff510 0x7f6368196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:32.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.371+0000 7f636cf9e700 1 -- 172.21.15.121:0/3568651688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63680ff510 msgr2=0x7f6368196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:32.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.371+0000 7f636cf9e700 1 --2- 172.21.15.121:0/3568651688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63680ff510 0x7f6368196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.371+0000 7f636cf9e700 1 -- 172.21.15.121:0/3568651688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63640095f0 con 0x7f63680ffec0 2024-04-05T21:58:32.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.371+0000 7f636cf9e700 1 --2- 172.21.15.121:0/3568651688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63680ffec0 0x7f6368196c40 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f636400bd70 tx=0x7f636400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:32.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.371+0000 7f635e7fc700 1 -- 172.21.15.121:0/3568651688 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f636401d070 con 0x7f63680ffec0 2024-04-05T21:58:32.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.371+0000 7f635e7fc700 1 -- 172.21.15.121:0/3568651688 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f636400f460 con 0x7f63680ffec0 2024-04-05T21:58:32.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.371+0000 7f636fa03700 1 -- 172.21.15.121:0/3568651688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636819a280 con 0x7f63680ffec0 2024-04-05T21:58:32.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.371+0000 7f635e7fc700 1 -- 172.21.15.121:0/3568651688 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6364021640 con 0x7f63680ffec0 2024-04-05T21:58:32.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.371+0000 7f636fa03700 1 -- 172.21.15.121:0/3568651688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636819a770 con 0x7f63680ffec0 2024-04-05T21:58:32.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.372+0000 7f636fa03700 1 -- 172.21.15.121:0/3568651688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6368109180 con 0x7f63680ffec0 2024-04-05T21:58:32.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.372+0000 7f635e7fc700 1 -- 172.21.15.121:0/3568651688 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6364003a60 con 0x7f63680ffec0 2024-04-05T21:58:32.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.373+0000 7f635e7fc700 1 --2- 172.21.15.121:0/3568651688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f635406cce0 0x7f635406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:32.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.373+0000 7f635e7fc700 1 -- 172.21.15.121:0/3568651688 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f636408d670 con 0x7f63680ffec0 2024-04-05T21:58:32.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.373+0000 7f636d79f700 1 --2- 172.21.15.121:0/3568651688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f635406cce0 0x7f635406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:32.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.373+0000 7f636d79f700 1 --2- 172.21.15.121:0/3568651688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f635406cce0 0x7f635406f1a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6358005950 tx=0x7f63580058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:32.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.377+0000 7f635e7fc700 1 -- 172.21.15.121:0/3568651688 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6364057760 con 0x7f63680ffec0 2024-04-05T21:58:32.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.612+0000 7f636fa03700 1 -- 172.21.15.121:0/3568651688 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f636804ea90 con 0x7f63680ffec0 2024-04-05T21:58:32.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.612+0000 7f635e7fc700 1 -- 172.21.15.121:0/3568651688 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f6364061070 con 0x7f63680ffec0 2024-04-05T21:58:32.612 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:58:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:58:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:58:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:58:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:58:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:58:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:58:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:58:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:58:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:58:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:58:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:58:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:58:32.613 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:58:32.614 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:58:32.615 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:58:32.615 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:58:32.615 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:58:32.615 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:58:32.615 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:58:32.615 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:58:32.615 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:58:32.615 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:58:32.615 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:58:32.615 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:58:32.615 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:58:32.615 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:58:32.615 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:58:32.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.614+0000 7f636fa03700 1 -- 172.21.15.121:0/3568651688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f635406cce0 msgr2=0x7f635406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:32.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.614+0000 7f636fa03700 1 --2- 172.21.15.121:0/3568651688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f635406cce0 0x7f635406f1a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6358005950 tx=0x7f63580058e0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.614+0000 7f636fa03700 1 -- 172.21.15.121:0/3568651688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63680ffec0 msgr2=0x7f6368196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:32.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.614+0000 7f636fa03700 1 --2- 172.21.15.121:0/3568651688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63680ffec0 0x7f6368196c40 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f636400bd70 tx=0x7f636400be50 comp rx=0 tx=0).stop 2024-04-05T21:58:32.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.614+0000 7f636fa03700 1 -- 172.21.15.121:0/3568651688 shutdown_connections 2024-04-05T21:58:32.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.614+0000 7f636fa03700 1 --2- 172.21.15.121:0/3568651688 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f635406cce0 0x7f635406f1a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.614+0000 7f636fa03700 1 --2- 172.21.15.121:0/3568651688 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f63680ff510 0x7f6368196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.614+0000 7f636fa03700 1 --2- 172.21.15.121:0/3568651688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63680ffec0 0x7f6368196c40 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.614+0000 7f636fa03700 1 -- 172.21.15.121:0/3568651688 >> 172.21.15.121:0/3568651688 conn(0x7f6368074d80 msgr2=0x7f63680fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:32.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.614+0000 7f636fa03700 1 -- 172.21.15.121:0/3568651688 shutdown_connections 2024-04-05T21:58:32.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.614+0000 7f636fa03700 1 -- 172.21.15.121:0/3568651688 wait complete. 2024-04-05T21:58:32.617 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:58:32.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.717+0000 7f8738251700 1 -- 172.21.15.121:0/3741737619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87300ffee0 msgr2=0x7f8730105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:32.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.717+0000 7f8738251700 1 --2- 172.21.15.121:0/3741737619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87300ffee0 0x7f8730105870 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f872c009910 tx=0x7f872c009c20 comp rx=0 tx=0).stop 2024-04-05T21:58:32.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.717+0000 7f8738251700 1 -- 172.21.15.121:0/3741737619 shutdown_connections 2024-04-05T21:58:32.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.717+0000 7f8738251700 1 --2- 172.21.15.121:0/3741737619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87300ffee0 0x7f8730105870 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.717+0000 7f8738251700 1 --2- 172.21.15.121:0/3741737619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87300ff530 0x7f87300ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.717+0000 7f8738251700 1 -- 172.21.15.121:0/3741737619 >> 172.21.15.121:0/3741737619 conn(0x7f8730074df0 msgr2=0x7f8730075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:32.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.717+0000 7f8738251700 1 -- 172.21.15.121:0/3741737619 shutdown_connections 2024-04-05T21:58:32.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.717+0000 7f8738251700 1 -- 172.21.15.121:0/3741737619 wait complete. 2024-04-05T21:58:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.718+0000 7f8738251700 1 Processor -- start 2024-04-05T21:58:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.718+0000 7f8738251700 1 -- start start 2024-04-05T21:58:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.718+0000 7f8738251700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87300ff530 0x7f873019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.718+0000 7f8738251700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87300ffee0 0x7f873019ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.718+0000 7f8738251700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873019d0c0 con 0x7f87300ffee0 2024-04-05T21:58:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.718+0000 7f8738251700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8730196570 con 0x7f87300ff530 2024-04-05T21:58:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.718+0000 7f87357ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87300ffee0 0x7f873019ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.718+0000 7f87357ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87300ffee0 0x7f873019ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57854/0 (socket says 172.21.15.121:57854) 2024-04-05T21:58:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.718+0000 7f87357ec700 1 -- 172.21.15.121:0/4047006435 learned_addr learned my addr 172.21.15.121:0/4047006435 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:58:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.719+0000 7f8735fed700 1 --2- 172.21.15.121:0/4047006435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87300ff530 0x7f873019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.719+0000 7f87357ec700 1 -- 172.21.15.121:0/4047006435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87300ff530 msgr2=0x7f873019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.719+0000 7f87357ec700 1 --2- 172.21.15.121:0/4047006435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87300ff530 0x7f873019c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.719+0000 7f87357ec700 1 -- 172.21.15.121:0/4047006435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872c0095f0 con 0x7f87300ffee0 2024-04-05T21:58:32.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.719+0000 7f87357ec700 1 --2- 172.21.15.121:0/4047006435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87300ffee0 0x7f873019ca30 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f872c000c00 tx=0x7f872c00ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:32.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.719+0000 7f8726ffd700 1 -- 172.21.15.121:0/4047006435 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f872c01d070 con 0x7f87300ffee0 2024-04-05T21:58:32.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.719+0000 7f8726ffd700 1 -- 172.21.15.121:0/4047006435 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f872c00bed0 con 0x7f87300ffee0 2024-04-05T21:58:32.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.719+0000 7f8738251700 1 -- 172.21.15.121:0/4047006435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87301967f0 con 0x7f87300ffee0 2024-04-05T21:58:32.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.719+0000 7f8726ffd700 1 -- 172.21.15.121:0/4047006435 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f872c00fd90 con 0x7f87300ffee0 2024-04-05T21:58:32.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.719+0000 7f8738251700 1 -- 172.21.15.121:0/4047006435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8730196c80 con 0x7f87300ffee0 2024-04-05T21:58:32.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.720+0000 7f8738251700 1 -- 172.21.15.121:0/4047006435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8730109150 con 0x7f87300ffee0 2024-04-05T21:58:32.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.720+0000 7f8726ffd700 1 -- 172.21.15.121:0/4047006435 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f872c021410 con 0x7f87300ffee0 2024-04-05T21:58:32.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.721+0000 7f8726ffd700 1 --2- 172.21.15.121:0/4047006435 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f871c06cce0 0x7f871c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:32.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.721+0000 7f8726ffd700 1 -- 172.21.15.121:0/4047006435 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f872c08d6b0 con 0x7f87300ffee0 2024-04-05T21:58:32.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.721+0000 7f8735fed700 1 --2- 172.21.15.121:0/4047006435 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f871c06cce0 0x7f871c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:32.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.721+0000 7f8735fed700 1 --2- 172.21.15.121:0/4047006435 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f871c06cce0 0x7f871c06f1a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f8720009c60 tx=0x7f8720009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:32.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.725+0000 7f8726ffd700 1 -- 172.21.15.121:0/4047006435 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f872c05b230 con 0x7f87300ffee0 2024-04-05T21:58:32.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.937+0000 7f8738251700 1 -- 172.21.15.121:0/4047006435 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f873002ccc0 con 0x7f871c06cce0 2024-04-05T21:58:32.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.938+0000 7f8726ffd700 1 -- 172.21.15.121:0/4047006435 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f873002ccc0 con 0x7f871c06cce0 2024-04-05T21:58:32.939 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:58:32.939 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:58:32.939 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:58:32.939 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:58:32.939 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:58:32.939 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:58:32.939 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:58:32.939 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:58:32.939 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:58:32.939 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:58:32.939 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:58:32.940 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:58:32.940 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:58:32.940 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:58:32.940 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:58:32.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.940+0000 7f8738251700 1 -- 172.21.15.121:0/4047006435 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f871c06cce0 msgr2=0x7f871c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:32.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.940+0000 7f8738251700 1 --2- 172.21.15.121:0/4047006435 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f871c06cce0 0x7f871c06f1a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f8720009c60 tx=0x7f8720009260 comp rx=0 tx=0).stop 2024-04-05T21:58:32.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.941+0000 7f8738251700 1 -- 172.21.15.121:0/4047006435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87300ffee0 msgr2=0x7f873019ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:32.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.941+0000 7f8738251700 1 --2- 172.21.15.121:0/4047006435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87300ffee0 0x7f873019ca30 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f872c000c00 tx=0x7f872c00ba70 comp rx=0 tx=0).stop 2024-04-05T21:58:32.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.941+0000 7f8738251700 1 -- 172.21.15.121:0/4047006435 shutdown_connections 2024-04-05T21:58:32.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.941+0000 7f8738251700 1 --2- 172.21.15.121:0/4047006435 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f871c06cce0 0x7f871c06f1a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.941+0000 7f8738251700 1 --2- 172.21.15.121:0/4047006435 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f87300ff530 0x7f873019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.941+0000 7f8738251700 1 --2- 172.21.15.121:0/4047006435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87300ffee0 0x7f873019ca30 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:32.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.941+0000 7f8738251700 1 -- 172.21.15.121:0/4047006435 >> 172.21.15.121:0/4047006435 conn(0x7f8730074df0 msgr2=0x7f87301050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:32.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.941+0000 7f8738251700 1 -- 172.21.15.121:0/4047006435 shutdown_connections 2024-04-05T21:58:32.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:32.941+0000 7f8738251700 1 -- 172.21.15.121:0/4047006435 wait complete. 2024-04-05T21:58:33.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:32 smithi188 ceph-mon[107793]: from='client.35578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:33.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:32 smithi188 ceph-mon[107793]: from='client.35582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:33.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:32 smithi188 ceph-mon[107793]: pgmap v970: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:33.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:32 smithi188 ceph-mon[107793]: from='client.35586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:33.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:32 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3568643820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:58:33.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:32 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3568651688' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:58:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.043+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/335096243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb64101750 msgr2=0x7fdb641057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:33.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.043+0000 7fdb6a6ae700 1 --2- 172.21.15.121:0/335096243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb64101750 0x7fdb641057a0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009910 tx=0x7fdb54009c20 comp rx=0 tx=0).stop 2024-04-05T21:58:33.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.043+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/335096243 shutdown_connections 2024-04-05T21:58:33.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.043+0000 7fdb6a6ae700 1 --2- 172.21.15.121:0/335096243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb64101750 0x7fdb641057a0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:33.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.043+0000 7fdb6a6ae700 1 --2- 172.21.15.121:0/335096243 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb64100da0 0x7fdb64101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:33.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.043+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/335096243 >> 172.21.15.121:0/335096243 conn(0x7fdb640fc630 msgr2=0x7fdb640fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:33.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.043+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/335096243 shutdown_connections 2024-04-05T21:58:33.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.043+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/335096243 wait complete. 2024-04-05T21:58:33.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.043+0000 7fdb6a6ae700 1 Processor -- start 2024-04-05T21:58:33.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.043+0000 7fdb6a6ae700 1 -- start start 2024-04-05T21:58:33.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb6a6ae700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb64100da0 0x7fdb64073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:33.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb6a6ae700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb64101750 0x7fdb640735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:33.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb6a6ae700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb64073b10 con 0x7fdb64101750 2024-04-05T21:58:33.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb6a6ae700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb64073c50 con 0x7fdb64100da0 2024-04-05T21:58:33.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb637fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb64101750 0x7fdb640735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:33.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb637fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb64101750 0x7fdb640735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57862/0 (socket says 172.21.15.121:57862) 2024-04-05T21:58:33.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb637fe700 1 -- 172.21.15.121:0/2319036752 learned_addr learned my addr 172.21.15.121:0/2319036752 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:58:33.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb63fff700 1 --2- 172.21.15.121:0/2319036752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb64100da0 0x7fdb64073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:33.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb637fe700 1 -- 172.21.15.121:0/2319036752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb64100da0 msgr2=0x7fdb64073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:33.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb637fe700 1 --2- 172.21.15.121:0/2319036752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb64100da0 0x7fdb64073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:33.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb637fe700 1 -- 172.21.15.121:0/2319036752 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb540095f0 con 0x7fdb64101750 2024-04-05T21:58:33.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb63fff700 1 --2- 172.21.15.121:0/2319036752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb64100da0 0x7fdb64073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:58:33.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb637fe700 1 --2- 172.21.15.121:0/2319036752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb64101750 0x7fdb640735d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fdb540092e0 tx=0x7fdb5400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:33.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb617fa700 1 -- 172.21.15.121:0/2319036752 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb5401d070 con 0x7fdb64101750 2024-04-05T21:58:33.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb617fa700 1 -- 172.21.15.121:0/2319036752 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb5400f460 con 0x7fdb64101750 2024-04-05T21:58:33.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/2319036752 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb64073ed0 con 0x7fdb64101750 2024-04-05T21:58:33.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb617fa700 1 -- 172.21.15.121:0/2319036752 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb54021600 con 0x7fdb64101750 2024-04-05T21:58:33.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.044+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/2319036752 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb641a2ca0 con 0x7fdb64101750 2024-04-05T21:58:33.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.045+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/2319036752 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb641090f0 con 0x7fdb64101750 2024-04-05T21:58:33.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.046+0000 7fdb617fa700 1 -- 172.21.15.121:0/2319036752 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdb5400f5d0 con 0x7fdb64101750 2024-04-05T21:58:33.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.046+0000 7fdb617fa700 1 --2- 172.21.15.121:0/2319036752 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb4c06cce0 0x7fdb4c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:58:33.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.046+0000 7fdb63fff700 1 --2- 172.21.15.121:0/2319036752 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb4c06cce0 0x7fdb4c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:58:33.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.046+0000 7fdb617fa700 1 -- 172.21.15.121:0/2319036752 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdb5408d860 con 0x7fdb64101750 2024-04-05T21:58:33.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.047+0000 7fdb63fff700 1 --2- 172.21.15.121:0/2319036752 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb4c06cce0 0x7fdb4c06f1a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdb580099d0 tx=0x7fdb58008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:58:33.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.051+0000 7fdb617fa700 1 -- 172.21.15.121:0/2319036752 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdb540578d0 con 0x7fdb64101750 2024-04-05T21:58:33.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:32 smithi121 ceph-mon[145859]: from='client.35578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:33.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:32 smithi121 ceph-mon[145859]: from='client.35582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:33.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:32 smithi121 ceph-mon[145859]: pgmap v970: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:33.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:32 smithi121 ceph-mon[145859]: from='client.35586 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:33.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:32 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3568643820' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:58:33.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:32 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3568651688' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:58:33.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.321+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/2319036752 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdb6402cc70 con 0x7fdb64101750 2024-04-05T21:58:33.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.321+0000 7fdb617fa700 1 -- 172.21.15.121:0/2319036752 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fdb54026e10 con 0x7fdb64101750 2024-04-05T21:58:33.322 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:58:33.322 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:58:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:58:33.322 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:58:33.322 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:58:33.322 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:58:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:58:33.322 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:58:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:58:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:58:33.323 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:58:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:58:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:58:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:58:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:58:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:58:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:58:33.325 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:58:33.326 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:58:33.326 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:58:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:58:33.326 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:58:33.326 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:58:33.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.323+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/2319036752 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb4c06cce0 msgr2=0x7fdb4c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:33.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.323+0000 7fdb6a6ae700 1 --2- 172.21.15.121:0/2319036752 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb4c06cce0 0x7fdb4c06f1a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdb580099d0 tx=0x7fdb58008040 comp rx=0 tx=0).stop 2024-04-05T21:58:33.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.323+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/2319036752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb64101750 msgr2=0x7fdb640735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:58:33.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.323+0000 7fdb6a6ae700 1 --2- 172.21.15.121:0/2319036752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb64101750 0x7fdb640735d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fdb540092e0 tx=0x7fdb5400be20 comp rx=0 tx=0).stop 2024-04-05T21:58:33.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.323+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/2319036752 shutdown_connections 2024-04-05T21:58:33.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.323+0000 7fdb6a6ae700 1 --2- 172.21.15.121:0/2319036752 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdb4c06cce0 0x7fdb4c06f1a0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:33.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.323+0000 7fdb6a6ae700 1 --2- 172.21.15.121:0/2319036752 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdb64100da0 0x7fdb64073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:33.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.323+0000 7fdb6a6ae700 1 --2- 172.21.15.121:0/2319036752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb64101750 0x7fdb640735d0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:58:33.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.323+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/2319036752 >> 172.21.15.121:0/2319036752 conn(0x7fdb640fc630 msgr2=0x7fdb640fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:58:33.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.323+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/2319036752 shutdown_connections 2024-04-05T21:58:33.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:58:33.323+0000 7fdb6a6ae700 1 -- 172.21.15.121:0/2319036752 wait complete. 2024-04-05T21:58:34.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:33 smithi121 ceph-mon[145859]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:34.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:33 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2319036752' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:58:34.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:33 smithi188 ceph-mon[107793]: from='client.35598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:58:34.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:33 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2319036752' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:58:35.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:34 smithi121 ceph-mon[145859]: pgmap v971: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:35.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:34 smithi188 ceph-mon[107793]: pgmap v971: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:37.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:36 smithi121 ceph-mon[145859]: pgmap v972: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:36 smithi188 ceph-mon[107793]: pgmap v972: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:39.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:38 smithi121 ceph-mon[145859]: pgmap v973: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:38 smithi188 ceph-mon[107793]: pgmap v973: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:40 smithi121 ceph-mon[145859]: pgmap v974: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:40 smithi188 ceph-mon[107793]: pgmap v974: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:43.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:42 smithi121 ceph-mon[145859]: pgmap v975: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:42 smithi188 ceph-mon[107793]: pgmap v975: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:45.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:44 smithi121 ceph-mon[145859]: pgmap v976: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:44 smithi188 ceph-mon[107793]: pgmap v976: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:58:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:58:47.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:46 smithi121 ceph-mon[145859]: pgmap v977: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:46 smithi188 ceph-mon[107793]: pgmap v977: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:49.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:48 smithi121 ceph-mon[145859]: pgmap v978: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:48 smithi188 ceph-mon[107793]: pgmap v978: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:51.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:50 smithi121 ceph-mon[145859]: pgmap v979: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:50 smithi188 ceph-mon[107793]: pgmap v979: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:53.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:52 smithi121 ceph-mon[145859]: pgmap v980: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:52 smithi188 ceph-mon[107793]: pgmap v980: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:55.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:54 smithi121 ceph-mon[145859]: pgmap v981: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:54 smithi188 ceph-mon[107793]: pgmap v981: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:56 smithi121 ceph-mon[145859]: pgmap v982: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:56 smithi188 ceph-mon[107793]: pgmap v982: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:59.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:58:58 smithi121 ceph-mon[145859]: pgmap v983: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:58:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:58:58 smithi188 ceph-mon[107793]: pgmap v983: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:00 smithi121 ceph-mon[145859]: pgmap v984: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:59:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:00 smithi188 ceph-mon[107793]: pgmap v984: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:59:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:02 smithi121 ceph-mon[145859]: pgmap v985: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:02 smithi188 ceph-mon[107793]: pgmap v985: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.439+0000 7ff9d14e0700 1 -- 172.21.15.121:0/104960055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9cc1026d0 msgr2=0x7ff9cc10abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.439+0000 7ff9d14e0700 1 --2- 172.21.15.121:0/104960055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9cc1026d0 0x7ff9cc10abc0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc009910 tx=0x7ff9bc009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.439+0000 7ff9d14e0700 1 -- 172.21.15.121:0/104960055 shutdown_connections 2024-04-05T21:59:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.439+0000 7ff9d14e0700 1 --2- 172.21.15.121:0/104960055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9cc1026d0 0x7ff9cc10abc0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:03.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.439+0000 7ff9d14e0700 1 --2- 172.21.15.121:0/104960055 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9cc101db0 0x7ff9cc102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.439+0000 7ff9d14e0700 1 -- 172.21.15.121:0/104960055 >> 172.21.15.121:0/104960055 conn(0x7ff9cc0fb560 msgr2=0x7ff9cc0fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.439+0000 7ff9d14e0700 1 -- 172.21.15.121:0/104960055 shutdown_connections 2024-04-05T21:59:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.439+0000 7ff9d14e0700 1 -- 172.21.15.121:0/104960055 wait complete. 2024-04-05T21:59:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.440+0000 7ff9d14e0700 1 Processor -- start 2024-04-05T21:59:03.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.440+0000 7ff9d14e0700 1 -- start start 2024-04-05T21:59:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.440+0000 7ff9d14e0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9cc101db0 0x7ff9cc0ffa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.440+0000 7ff9d14e0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9cc1026d0 0x7ff9cc0fff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.440+0000 7ff9d14e0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc100490 con 0x7ff9cc1026d0 2024-04-05T21:59:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.440+0000 7ff9d14e0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9cc1005d0 con 0x7ff9cc101db0 2024-04-05T21:59:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.440+0000 7ff9ca7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9cc1026d0 0x7ff9cc0fff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.440+0000 7ff9ca7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9cc1026d0 0x7ff9cc0fff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41652/0 (socket says 172.21.15.121:41652) 2024-04-05T21:59:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.440+0000 7ff9ca7fc700 1 -- 172.21.15.121:0/3001874617 learned_addr learned my addr 172.21.15.121:0/3001874617 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.440+0000 7ff9caffd700 1 --2- 172.21.15.121:0/3001874617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9cc101db0 0x7ff9cc0ffa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:03.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.441+0000 7ff9ca7fc700 1 -- 172.21.15.121:0/3001874617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9cc101db0 msgr2=0x7ff9cc0ffa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.441+0000 7ff9ca7fc700 1 --2- 172.21.15.121:0/3001874617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9cc101db0 0x7ff9cc0ffa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.441+0000 7ff9ca7fc700 1 -- 172.21.15.121:0/3001874617 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9bc0095f0 con 0x7ff9cc1026d0 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.441+0000 7ff9caffd700 1 --2- 172.21.15.121:0/3001874617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9cc101db0 0x7ff9cc0ffa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.441+0000 7ff9ca7fc700 1 --2- 172.21.15.121:0/3001874617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9cc1026d0 0x7ff9cc0fff50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc000c00 tx=0x7ff9bc00bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.441+0000 7ff9b3fff700 1 -- 172.21.15.121:0/3001874617 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9bc01d070 con 0x7ff9cc1026d0 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.441+0000 7ff9b3fff700 1 -- 172.21.15.121:0/3001874617 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9bc003cf0 con 0x7ff9cc1026d0 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.441+0000 7ff9d14e0700 1 -- 172.21.15.121:0/3001874617 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9cc100850 con 0x7ff9cc1026d0 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.441+0000 7ff9b3fff700 1 -- 172.21.15.121:0/3001874617 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9bc021410 con 0x7ff9cc1026d0 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.441+0000 7ff9d14e0700 1 -- 172.21.15.121:0/3001874617 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9cc1a2b90 con 0x7ff9cc1026d0 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.442+0000 7ff9d14e0700 1 -- 172.21.15.121:0/3001874617 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9cc108250 con 0x7ff9cc1026d0 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.442+0000 7ff9b3fff700 1 -- 172.21.15.121:0/3001874617 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff9bc003850 con 0x7ff9cc1026d0 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.443+0000 7ff9b3fff700 1 --2- 172.21.15.121:0/3001874617 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff9b406cce0 0x7ff9b406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.443+0000 7ff9b3fff700 1 -- 172.21.15.121:0/3001874617 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff9bc08d730 con 0x7ff9cc1026d0 2024-04-05T21:59:03.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.443+0000 7ff9caffd700 1 --2- 172.21.15.121:0/3001874617 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff9b406cce0 0x7ff9b406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:03.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.443+0000 7ff9caffd700 1 --2- 172.21.15.121:0/3001874617 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff9b406cce0 0x7ff9b406f1a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009be0 tx=0x7ff9c0009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:03.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.447+0000 7ff9b3fff700 1 -- 172.21.15.121:0/3001874617 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff9bc057820 con 0x7ff9cc1026d0 2024-04-05T21:59:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.661+0000 7ff9d14e0700 1 -- 172.21.15.121:0/3001874617 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9cc0611d0 con 0x7ff9b406cce0 2024-04-05T21:59:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.662+0000 7ff9b3fff700 1 -- 172.21.15.121:0/3001874617 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff9cc0611d0 con 0x7ff9b406cce0 2024-04-05T21:59:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.664+0000 7ff9d14e0700 1 -- 172.21.15.121:0/3001874617 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff9b406cce0 msgr2=0x7ff9b406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.664+0000 7ff9d14e0700 1 --2- 172.21.15.121:0/3001874617 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff9b406cce0 0x7ff9b406f1a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009be0 tx=0x7ff9c0009260 comp rx=0 tx=0).stop 2024-04-05T21:59:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.664+0000 7ff9d14e0700 1 -- 172.21.15.121:0/3001874617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9cc1026d0 msgr2=0x7ff9cc0fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.664+0000 7ff9d14e0700 1 --2- 172.21.15.121:0/3001874617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9cc1026d0 0x7ff9cc0fff50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc000c00 tx=0x7ff9bc00bb20 comp rx=0 tx=0).stop 2024-04-05T21:59:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.664+0000 7ff9d14e0700 1 -- 172.21.15.121:0/3001874617 shutdown_connections 2024-04-05T21:59:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.664+0000 7ff9d14e0700 1 --2- 172.21.15.121:0/3001874617 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff9b406cce0 0x7ff9b406f1a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.664+0000 7ff9d14e0700 1 --2- 172.21.15.121:0/3001874617 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff9cc101db0 0x7ff9cc0ffa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.665+0000 7ff9d14e0700 1 --2- 172.21.15.121:0/3001874617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9cc1026d0 0x7ff9cc0fff50 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.665+0000 7ff9d14e0700 1 -- 172.21.15.121:0/3001874617 >> 172.21.15.121:0/3001874617 conn(0x7ff9cc0fb560 msgr2=0x7ff9cc105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.665+0000 7ff9d14e0700 1 -- 172.21.15.121:0/3001874617 shutdown_connections 2024-04-05T21:59:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.665+0000 7ff9d14e0700 1 -- 172.21.15.121:0/3001874617 wait complete. 2024-04-05T21:59:03.675 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:59:03.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.768+0000 7f6bb602c700 1 -- 172.21.15.121:0/1418547724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bb0102630 msgr2=0x7f6bb010ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:03.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.768+0000 7f6bb602c700 1 --2- 172.21.15.121:0/1418547724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bb0102630 0x7f6bb010ab20 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009910 tx=0x7f6ba0009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:03.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.768+0000 7f6bb602c700 1 -- 172.21.15.121:0/1418547724 shutdown_connections 2024-04-05T21:59:03.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.768+0000 7f6bb602c700 1 --2- 172.21.15.121:0/1418547724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bb0102630 0x7f6bb010ab20 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:03.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.768+0000 7f6bb602c700 1 --2- 172.21.15.121:0/1418547724 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bb0101d10 0x7f6bb01020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:03.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.768+0000 7f6bb602c700 1 -- 172.21.15.121:0/1418547724 >> 172.21.15.121:0/1418547724 conn(0x7f6bb00fb540 msgr2=0x7f6bb00fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:03.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.768+0000 7f6bb602c700 1 -- 172.21.15.121:0/1418547724 shutdown_connections 2024-04-05T21:59:03.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.768+0000 7f6bb602c700 1 -- 172.21.15.121:0/1418547724 wait complete. 2024-04-05T21:59:03.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.769+0000 7f6bb602c700 1 Processor -- start 2024-04-05T21:59:03.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.769+0000 7f6bb602c700 1 -- start start 2024-04-05T21:59:03.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6bb602c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bb0101d10 0x7f6bb00ff970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:03.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6bb602c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bb0102630 0x7f6bb00ffeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:03.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6bb602c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb01003f0 con 0x7f6bb0102630 2024-04-05T21:59:03.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6bb602c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb0100530 con 0x7f6bb0101d10 2024-04-05T21:59:03.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6baeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bb0102630 0x7f6bb00ffeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:03.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6baeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bb0102630 0x7f6bb00ffeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41668/0 (socket says 172.21.15.121:41668) 2024-04-05T21:59:03.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6baeffd700 1 -- 172.21.15.121:0/4267620872 learned_addr learned my addr 172.21.15.121:0/4267620872 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:03.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6baf7fe700 1 --2- 172.21.15.121:0/4267620872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bb0101d10 0x7f6bb00ff970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:03.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6baeffd700 1 -- 172.21.15.121:0/4267620872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bb0101d10 msgr2=0x7f6bb00ff970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:03.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6baeffd700 1 --2- 172.21.15.121:0/4267620872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bb0101d10 0x7f6bb00ff970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:03.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6baeffd700 1 -- 172.21.15.121:0/4267620872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba00095f0 con 0x7f6bb0102630 2024-04-05T21:59:03.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6baeffd700 1 --2- 172.21.15.121:0/4267620872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bb0102630 0x7f6bb00ffeb0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6ba000bd70 tx=0x7f6ba000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:03.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6bacff9700 1 -- 172.21.15.121:0/4267620872 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ba001d070 con 0x7f6bb0102630 2024-04-05T21:59:03.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6bb602c700 1 -- 172.21.15.121:0/4267620872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb01007b0 con 0x7f6bb0102630 2024-04-05T21:59:03.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6bacff9700 1 -- 172.21.15.121:0/4267620872 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ba000f460 con 0x7f6bb0102630 2024-04-05T21:59:03.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6bacff9700 1 -- 172.21.15.121:0/4267620872 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ba0021640 con 0x7f6bb0102630 2024-04-05T21:59:03.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.770+0000 7f6bb602c700 1 -- 172.21.15.121:0/4267620872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb01a2bc0 con 0x7f6bb0102630 2024-04-05T21:59:03.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.772+0000 7f6bacff9700 1 -- 172.21.15.121:0/4267620872 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6ba0003a60 con 0x7f6bb0102630 2024-04-05T21:59:03.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.772+0000 7f6bb602c700 1 -- 172.21.15.121:0/4267620872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb0108220 con 0x7f6bb0102630 2024-04-05T21:59:03.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.772+0000 7f6bacff9700 1 --2- 172.21.15.121:0/4267620872 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6b9806cce0 0x7f6b9806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:03.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.772+0000 7f6bacff9700 1 -- 172.21.15.121:0/4267620872 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6ba008d740 con 0x7f6bb0102630 2024-04-05T21:59:03.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.772+0000 7f6baf7fe700 1 --2- 172.21.15.121:0/4267620872 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6b9806cce0 0x7f6b9806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:03.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.773+0000 7f6baf7fe700 1 --2- 172.21.15.121:0/4267620872 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6b9806cce0 0x7f6b9806f1a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4005950 tx=0x7f6ba40058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:03.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.777+0000 7f6bacff9700 1 -- 172.21.15.121:0/4267620872 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6ba00577b0 con 0x7f6bb0102630 2024-04-05T21:59:03.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.990+0000 7f6bb602c700 1 -- 172.21.15.121:0/4267620872 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bb002cce0 con 0x7f6b9806cce0 2024-04-05T21:59:03.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.991+0000 7f6bacff9700 1 -- 172.21.15.121:0/4267620872 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6bb002cce0 con 0x7f6b9806cce0 2024-04-05T21:59:03.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.993+0000 7f6bb602c700 1 -- 172.21.15.121:0/4267620872 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6b9806cce0 msgr2=0x7f6b9806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:03.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.993+0000 7f6bb602c700 1 --2- 172.21.15.121:0/4267620872 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6b9806cce0 0x7f6b9806f1a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4005950 tx=0x7f6ba40058e0 comp rx=0 tx=0).stop 2024-04-05T21:59:03.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.993+0000 7f6bb602c700 1 -- 172.21.15.121:0/4267620872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bb0102630 msgr2=0x7f6bb00ffeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:03.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.993+0000 7f6bb602c700 1 --2- 172.21.15.121:0/4267620872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bb0102630 0x7f6bb00ffeb0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6ba000bd70 tx=0x7f6ba000be50 comp rx=0 tx=0).stop 2024-04-05T21:59:03.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.993+0000 7f6bb602c700 1 -- 172.21.15.121:0/4267620872 shutdown_connections 2024-04-05T21:59:03.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.993+0000 7f6bb602c700 1 --2- 172.21.15.121:0/4267620872 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6b9806cce0 0x7f6b9806f1a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:03.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.993+0000 7f6bb602c700 1 --2- 172.21.15.121:0/4267620872 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6bb0101d10 0x7f6bb00ff970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:03.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.993+0000 7f6bb602c700 1 --2- 172.21.15.121:0/4267620872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bb0102630 0x7f6bb00ffeb0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:03.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.993+0000 7f6bb602c700 1 -- 172.21.15.121:0/4267620872 >> 172.21.15.121:0/4267620872 conn(0x7f6bb00fb540 msgr2=0x7f6bb0105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:03.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.993+0000 7f6bb602c700 1 -- 172.21.15.121:0/4267620872 shutdown_connections 2024-04-05T21:59:03.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:03.993+0000 7f6bb602c700 1 -- 172.21.15.121:0/4267620872 wait complete. 2024-04-05T21:59:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.096+0000 7f279d681700 1 -- 172.21.15.121:0/2714020032 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2798100de0 msgr2=0x7f27981011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.096+0000 7f279d681700 1 --2- 172.21.15.121:0/2714020032 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2798100de0 0x7f27981011c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f278c009910 tx=0x7f278c009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.096+0000 7f279d681700 1 -- 172.21.15.121:0/2714020032 shutdown_connections 2024-04-05T21:59:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.096+0000 7f279d681700 1 --2- 172.21.15.121:0/2714020032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2798101790 0x7f27981057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.096+0000 7f279d681700 1 --2- 172.21.15.121:0/2714020032 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2798100de0 0x7f27981011c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.096+0000 7f279d681700 1 -- 172.21.15.121:0/2714020032 >> 172.21.15.121:0/2714020032 conn(0x7f27980fc6b0 msgr2=0x7f27980fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:04.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.096+0000 7f279d681700 1 -- 172.21.15.121:0/2714020032 shutdown_connections 2024-04-05T21:59:04.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.096+0000 7f279d681700 1 -- 172.21.15.121:0/2714020032 wait complete. 2024-04-05T21:59:04.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.096+0000 7f279d681700 1 Processor -- start 2024-04-05T21:59:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f279d681700 1 -- start start 2024-04-05T21:59:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f279d681700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2798100de0 0x7f27981966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f279d681700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2798101790 0x7f2798196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f279d681700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2798197280 con 0x7f2798101790 2024-04-05T21:59:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f279d681700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2798199fb0 con 0x7f2798100de0 2024-04-05T21:59:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f27967fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2798101790 0x7f2798196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f27967fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2798101790 0x7f2798196bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41698/0 (socket says 172.21.15.121:41698) 2024-04-05T21:59:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f27967fc700 1 -- 172.21.15.121:0/835488312 learned_addr learned my addr 172.21.15.121:0/835488312 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f2796ffd700 1 --2- 172.21.15.121:0/835488312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2798100de0 0x7f27981966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f27967fc700 1 -- 172.21.15.121:0/835488312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2798100de0 msgr2=0x7f27981966b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f27967fc700 1 --2- 172.21.15.121:0/835488312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2798100de0 0x7f27981966b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f27967fc700 1 -- 172.21.15.121:0/835488312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c0095f0 con 0x7f2798101790 2024-04-05T21:59:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f2796ffd700 1 --2- 172.21.15.121:0/835488312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2798100de0 0x7f27981966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:59:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f27967fc700 1 --2- 172.21.15.121:0/835488312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2798101790 0x7f2798196bf0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f278800e8f0 tx=0x7f278800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:04.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f277ffff700 1 -- 172.21.15.121:0/835488312 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2788009790 con 0x7f2798101790 2024-04-05T21:59:04.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f279d681700 1 -- 172.21.15.121:0/835488312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f279819a290 con 0x7f2798101790 2024-04-05T21:59:04.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f277ffff700 1 -- 172.21.15.121:0/835488312 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2788004cd0 con 0x7f2798101790 2024-04-05T21:59:04.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f277ffff700 1 -- 172.21.15.121:0/835488312 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2788005580 con 0x7f2798101790 2024-04-05T21:59:04.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.097+0000 7f279d681700 1 -- 172.21.15.121:0/835488312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279819a7e0 con 0x7f2798101790 2024-04-05T21:59:04.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.098+0000 7f279d681700 1 -- 172.21.15.121:0/835488312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2798109180 con 0x7f2798101790 2024-04-05T21:59:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.099+0000 7f277ffff700 1 -- 172.21.15.121:0/835488312 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2788004750 con 0x7f2798101790 2024-04-05T21:59:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.099+0000 7f277ffff700 1 --2- 172.21.15.121:0/835488312 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f278006cce0 0x7f278006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.099+0000 7f277ffff700 1 -- 172.21.15.121:0/835488312 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2788014070 con 0x7f2798101790 2024-04-05T21:59:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.099+0000 7f2796ffd700 1 --2- 172.21.15.121:0/835488312 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f278006cce0 0x7f278006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.100+0000 7f2796ffd700 1 --2- 172.21.15.121:0/835488312 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f278006cce0 0x7f278006f1a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f278c00b6e0 tx=0x7f278c009f70 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.104+0000 7f277ffff700 1 -- 172.21.15.121:0/835488312 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f27880569c0 con 0x7f2798101790 2024-04-05T21:59:04.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.304+0000 7f279d681700 1 -- 172.21.15.121:0/835488312 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f279819ab20 con 0x7f278006cce0 2024-04-05T21:59:04.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.314+0000 7f277ffff700 1 -- 172.21.15.121:0/835488312 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f279819ab20 con 0x7f278006cce0 2024-04-05T21:59:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:59:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (29m) 8m ago 41m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:59:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (41m) 8m ago 41m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:59:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (40m) 8m ago 40m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:59:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (28m) 8m ago 41m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:59:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (27m) 8m ago 40m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:59:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (29m) 8m ago 41m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:59:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (24m) 8m ago 35m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:59:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (23m) 8m ago 35m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:59:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (23m) 8m ago 35m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:59:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (8m) 8m ago 35m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:59:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (31m) 8m ago 42m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:59:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (31m) 8m ago 40m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:59:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (28m) 8m ago 43m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:59:04.316 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (28m) 8m ago 40m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:59:04.316 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (30m) 8m ago 41m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:59:04.316 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (30m) 8m ago 40m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:59:04.316 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (27m) 8m ago 39m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:59:04.316 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (26m) 8m ago 38m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:59:04.316 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (26m) 8m ago 38m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:59:04.316 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (25m) 8m ago 37m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:59:04.316 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (25m) 8m ago 37m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:59:04.316 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (24m) 8m ago 36m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:59:04.316 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (30m) 8m ago 40m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:59:04.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.316+0000 7f279d681700 1 -- 172.21.15.121:0/835488312 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f278006cce0 msgr2=0x7f278006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:04.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.316+0000 7f279d681700 1 --2- 172.21.15.121:0/835488312 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f278006cce0 0x7f278006f1a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f278c00b6e0 tx=0x7f278c009f70 comp rx=0 tx=0).stop 2024-04-05T21:59:04.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.316+0000 7f279d681700 1 -- 172.21.15.121:0/835488312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2798101790 msgr2=0x7f2798196bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:04.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.316+0000 7f279d681700 1 --2- 172.21.15.121:0/835488312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2798101790 0x7f2798196bf0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f278800e8f0 tx=0x7f278800ecb0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.316+0000 7f279d681700 1 -- 172.21.15.121:0/835488312 shutdown_connections 2024-04-05T21:59:04.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.316+0000 7f279d681700 1 --2- 172.21.15.121:0/835488312 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f278006cce0 0x7f278006f1a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.316+0000 7f279d681700 1 --2- 172.21.15.121:0/835488312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2798100de0 0x7f27981966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.316+0000 7f279d681700 1 --2- 172.21.15.121:0/835488312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2798101790 0x7f2798196bf0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.316+0000 7f279d681700 1 -- 172.21.15.121:0/835488312 >> 172.21.15.121:0/835488312 conn(0x7f27980fc6b0 msgr2=0x7f27980fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:04.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.316+0000 7f279d681700 1 -- 172.21.15.121:0/835488312 shutdown_connections 2024-04-05T21:59:04.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.316+0000 7f279d681700 1 -- 172.21.15.121:0/835488312 wait complete. 2024-04-05T21:59:04.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.417+0000 7f3bba894700 1 -- 172.21.15.121:0/1461056430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bb4105a40 msgr2=0x7f3bb4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:04.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.417+0000 7f3bba894700 1 --2- 172.21.15.121:0/1461056430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bb4105a40 0x7f3bb4105e20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009910 tx=0x7f3ba4009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:04.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.417+0000 7f3bba894700 1 -- 172.21.15.121:0/1461056430 shutdown_connections 2024-04-05T21:59:04.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.417+0000 7f3bba894700 1 --2- 172.21.15.121:0/1461056430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bb40ffbd0 0x7f3bb4100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.417+0000 7f3bba894700 1 --2- 172.21.15.121:0/1461056430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bb4105a40 0x7f3bb4105e20 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.417+0000 7f3bba894700 1 -- 172.21.15.121:0/1461056430 >> 172.21.15.121:0/1461056430 conn(0x7f3bb40755a0 msgr2=0x7f3bb40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:04.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.417+0000 7f3bba894700 1 -- 172.21.15.121:0/1461056430 shutdown_connections 2024-04-05T21:59:04.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.417+0000 7f3bba894700 1 -- 172.21.15.121:0/1461056430 wait complete. 2024-04-05T21:59:04.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.418+0000 7f3bba894700 1 Processor -- start 2024-04-05T21:59:04.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.418+0000 7f3bba894700 1 -- start start 2024-04-05T21:59:04.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.418+0000 7f3bba894700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bb40ffbd0 0x7f3bb41974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:04.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.418+0000 7f3bba894700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bb4105a40 0x7f3bb4197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:04.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.418+0000 7f3bba894700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4198020 con 0x7f3bb4105a40 2024-04-05T21:59:04.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.418+0000 7f3bba894700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4198160 con 0x7f3bb40ffbd0 2024-04-05T21:59:04.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bb37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bb4105a40 0x7f3bb4197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:04.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bb37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bb4105a40 0x7f3bb4197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60622/0 (socket says 172.21.15.121:60622) 2024-04-05T21:59:04.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bb37fe700 1 -- 172.21.15.121:0/2723995060 learned_addr learned my addr 172.21.15.121:0/2723995060 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:04.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bb3fff700 1 --2- 172.21.15.121:0/2723995060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bb40ffbd0 0x7f3bb41974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:04.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bb37fe700 1 -- 172.21.15.121:0/2723995060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bb40ffbd0 msgr2=0x7f3bb41974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:04.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bb37fe700 1 --2- 172.21.15.121:0/2723995060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bb40ffbd0 0x7f3bb41974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bb37fe700 1 -- 172.21.15.121:0/2723995060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba40095f0 con 0x7f3bb4105a40 2024-04-05T21:59:04.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bb3fff700 1 --2- 172.21.15.121:0/2723995060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bb40ffbd0 0x7f3bb41974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T21:59:04.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bb37fe700 1 --2- 172.21.15.121:0/2723995060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bb4105a40 0x7f3bb4197a00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f3ba800e920 tx=0x7f3ba800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:04.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bb17fa700 1 -- 172.21.15.121:0/2723995060 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ba8009790 con 0x7f3bb4105a40 2024-04-05T21:59:04.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bb17fa700 1 -- 172.21.15.121:0/2723995060 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ba800cba0 con 0x7f3bb4105a40 2024-04-05T21:59:04.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bba894700 1 -- 172.21.15.121:0/2723995060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb41a0530 con 0x7f3bb4105a40 2024-04-05T21:59:04.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bb17fa700 1 -- 172.21.15.121:0/2723995060 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ba8018650 con 0x7f3bb4105a40 2024-04-05T21:59:04.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.419+0000 7f3bba894700 1 -- 172.21.15.121:0/2723995060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb41a09f0 con 0x7f3bb4105a40 2024-04-05T21:59:04.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.420+0000 7f3bba894700 1 -- 172.21.15.121:0/2723995060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb4108040 con 0x7f3bb4105a40 2024-04-05T21:59:04.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.421+0000 7f3bb17fa700 1 -- 172.21.15.121:0/2723995060 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3ba800cd10 con 0x7f3bb4105a40 2024-04-05T21:59:04.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.421+0000 7f3bb17fa700 1 --2- 172.21.15.121:0/2723995060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3b9c06cce0 0x7f3b9c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:04.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.421+0000 7f3bb17fa700 1 -- 172.21.15.121:0/2723995060 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3ba8014070 con 0x7f3bb4105a40 2024-04-05T21:59:04.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.421+0000 7f3bb3fff700 1 --2- 172.21.15.121:0/2723995060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3b9c06cce0 0x7f3b9c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:04.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.421+0000 7f3bb3fff700 1 --2- 172.21.15.121:0/2723995060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3b9c06cce0 0x7f3b9c06f1a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40059d0 tx=0x7f3ba4005940 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:04.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.425+0000 7f3bb17fa700 1 -- 172.21.15.121:0/2723995060 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3ba8052ab0 con 0x7f3bb4105a40 2024-04-05T21:59:04.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.704+0000 7f3bba894700 1 -- 172.21.15.121:0/2723995060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3bb404ea90 con 0x7f3bb4105a40 2024-04-05T21:59:04.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.704+0000 7f3bb17fa700 1 -- 172.21.15.121:0/2723995060 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f3ba805a3f0 con 0x7f3bb4105a40 2024-04-05T21:59:04.705 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:59:04.705 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:59:04.705 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:59:04.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:59:04.705 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:59:04.705 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:59:04.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:59:04.705 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:59:04.705 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:59:04.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:59:04.705 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:59:04.706 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:59:04.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:59:04.706 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:59:04.706 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:59:04.706 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:59:04.706 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:59:04.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.706+0000 7f3bba894700 1 -- 172.21.15.121:0/2723995060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3b9c06cce0 msgr2=0x7f3b9c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:04.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.706+0000 7f3bba894700 1 --2- 172.21.15.121:0/2723995060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3b9c06cce0 0x7f3b9c06f1a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40059d0 tx=0x7f3ba4005940 comp rx=0 tx=0).stop 2024-04-05T21:59:04.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.706+0000 7f3bba894700 1 -- 172.21.15.121:0/2723995060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bb4105a40 msgr2=0x7f3bb4197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:04.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.706+0000 7f3bba894700 1 --2- 172.21.15.121:0/2723995060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bb4105a40 0x7f3bb4197a00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f3ba800e920 tx=0x7f3ba800ece0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.706+0000 7f3bba894700 1 -- 172.21.15.121:0/2723995060 shutdown_connections 2024-04-05T21:59:04.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.706+0000 7f3bba894700 1 --2- 172.21.15.121:0/2723995060 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3b9c06cce0 0x7f3b9c06f1a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.706+0000 7f3bba894700 1 --2- 172.21.15.121:0/2723995060 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3bb40ffbd0 0x7f3bb41974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.706+0000 7f3bba894700 1 --2- 172.21.15.121:0/2723995060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bb4105a40 0x7f3bb4197a00 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.706+0000 7f3bba894700 1 -- 172.21.15.121:0/2723995060 >> 172.21.15.121:0/2723995060 conn(0x7f3bb40755a0 msgr2=0x7f3bb40fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:04.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.707+0000 7f3bba894700 1 -- 172.21.15.121:0/2723995060 shutdown_connections 2024-04-05T21:59:04.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.707+0000 7f3bba894700 1 -- 172.21.15.121:0/2723995060 wait complete. 2024-04-05T21:59:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.809+0000 7f1347d75700 1 -- 172.21.15.121:0/1195587149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1340105a40 msgr2=0x7f1340105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.809+0000 7f1347d75700 1 --2- 172.21.15.121:0/1195587149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1340105a40 0x7f1340105e20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f133c009910 tx=0x7f133c009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.810+0000 7f1347d75700 1 -- 172.21.15.121:0/1195587149 shutdown_connections 2024-04-05T21:59:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.810+0000 7f1347d75700 1 --2- 172.21.15.121:0/1195587149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f13400ffbd0 0x7f1340100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.810+0000 7f1347d75700 1 --2- 172.21.15.121:0/1195587149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1340105a40 0x7f1340105e20 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.810+0000 7f1347d75700 1 -- 172.21.15.121:0/1195587149 >> 172.21.15.121:0/1195587149 conn(0x7f13400755a0 msgr2=0x7f13400759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.810+0000 7f1347d75700 1 -- 172.21.15.121:0/1195587149 shutdown_connections 2024-04-05T21:59:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.810+0000 7f1347d75700 1 -- 172.21.15.121:0/1195587149 wait complete. 2024-04-05T21:59:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.810+0000 7f1347d75700 1 Processor -- start 2024-04-05T21:59:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.810+0000 7f1347d75700 1 -- start start 2024-04-05T21:59:04.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.810+0000 7f1347d75700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13400ffbd0 0x7f13401974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.810+0000 7f1347d75700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1340105a40 0x7f13401979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.810+0000 7f1347d75700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1340198000 con 0x7f13400ffbd0 2024-04-05T21:59:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.810+0000 7f1347d75700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1340198170 con 0x7f1340105a40 2024-04-05T21:59:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.811+0000 7f1345b11700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13400ffbd0 0x7f13401974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.811+0000 7f1345b11700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13400ffbd0 0x7f13401974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60646/0 (socket says 172.21.15.121:60646) 2024-04-05T21:59:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.811+0000 7f1345b11700 1 -- 172.21.15.121:0/3480486149 learned_addr learned my addr 172.21.15.121:0/3480486149 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.811+0000 7f1345310700 1 --2- 172.21.15.121:0/3480486149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1340105a40 0x7f13401979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.811+0000 7f1345b11700 1 -- 172.21.15.121:0/3480486149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1340105a40 msgr2=0x7f13401979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.811+0000 7f1345b11700 1 --2- 172.21.15.121:0/3480486149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1340105a40 0x7f13401979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.811+0000 7f1345b11700 1 -- 172.21.15.121:0/3480486149 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c0095f0 con 0x7f13400ffbd0 2024-04-05T21:59:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.811+0000 7f1345b11700 1 --2- 172.21.15.121:0/3480486149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13400ffbd0 0x7f13401974a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f133c006010 tx=0x7f133c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.811+0000 7f1336ffd700 1 -- 172.21.15.121:0/3480486149 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f133c01d070 con 0x7f13400ffbd0 2024-04-05T21:59:04.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.811+0000 7f1347d75700 1 -- 172.21.15.121:0/3480486149 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13401a0510 con 0x7f13400ffbd0 2024-04-05T21:59:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.811+0000 7f1336ffd700 1 -- 172.21.15.121:0/3480486149 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f133c00f460 con 0x7f13400ffbd0 2024-04-05T21:59:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.811+0000 7f1336ffd700 1 -- 172.21.15.121:0/3480486149 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f133c021620 con 0x7f13400ffbd0 2024-04-05T21:59:04.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.811+0000 7f1347d75700 1 -- 172.21.15.121:0/3480486149 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13401a0980 con 0x7f13400ffbd0 2024-04-05T21:59:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.812+0000 7f1347d75700 1 -- 172.21.15.121:0/3480486149 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1340108040 con 0x7f13400ffbd0 2024-04-05T21:59:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.813+0000 7f1336ffd700 1 -- 172.21.15.121:0/3480486149 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f133c00f5d0 con 0x7f13400ffbd0 2024-04-05T21:59:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.813+0000 7f1336ffd700 1 --2- 172.21.15.121:0/3480486149 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f132c06cce0 0x7f132c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.813+0000 7f1336ffd700 1 -- 172.21.15.121:0/3480486149 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f133c08d850 con 0x7f13400ffbd0 2024-04-05T21:59:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.813+0000 7f1345310700 1 --2- 172.21.15.121:0/3480486149 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f132c06cce0 0x7f132c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:04.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.813+0000 7f1345310700 1 --2- 172.21.15.121:0/3480486149 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f132c06cce0 0x7f132c06f1a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f1340068ce0 tx=0x7f133000a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:04.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:04.817+0000 7f1336ffd700 1 -- 172.21.15.121:0/3480486149 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f133c0578c0 con 0x7f13400ffbd0 2024-04-05T21:59:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.052+0000 7f1347d75700 1 -- 172.21.15.121:0/3480486149 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f134004ea90 con 0x7f13400ffbd0 2024-04-05T21:59:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.052+0000 7f1336ffd700 1 -- 172.21.15.121:0/3480486149 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f133c05aee0 con 0x7f13400ffbd0 2024-04-05T21:59:05.053 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:59:05.053 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:59:05.053 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:59:05.053 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:59:05.053 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:59:05.053 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:59:05.053 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:59:05.053 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:59:05.053 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:59:05.053 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:59:05.053 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:59:05.053 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:59:05.054 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:59:05.055 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:59:05.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.054+0000 7f1347d75700 1 -- 172.21.15.121:0/3480486149 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f132c06cce0 msgr2=0x7f132c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:05.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.054+0000 7f1347d75700 1 --2- 172.21.15.121:0/3480486149 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f132c06cce0 0x7f132c06f1a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f1340068ce0 tx=0x7f133000a300 comp rx=0 tx=0).stop 2024-04-05T21:59:05.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.054+0000 7f1347d75700 1 -- 172.21.15.121:0/3480486149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13400ffbd0 msgr2=0x7f13401974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:05.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.054+0000 7f1347d75700 1 --2- 172.21.15.121:0/3480486149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13400ffbd0 0x7f13401974a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f133c006010 tx=0x7f133c00bd70 comp rx=0 tx=0).stop 2024-04-05T21:59:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.054+0000 7f1347d75700 1 -- 172.21.15.121:0/3480486149 shutdown_connections 2024-04-05T21:59:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.054+0000 7f1347d75700 1 --2- 172.21.15.121:0/3480486149 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f132c06cce0 0x7f132c06f1a0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.054+0000 7f1347d75700 1 --2- 172.21.15.121:0/3480486149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13400ffbd0 0x7f13401974a0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.054+0000 7f1347d75700 1 --2- 172.21.15.121:0/3480486149 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1340105a40 0x7f13401979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.054+0000 7f1347d75700 1 -- 172.21.15.121:0/3480486149 >> 172.21.15.121:0/3480486149 conn(0x7f13400755a0 msgr2=0x7f13400fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.054+0000 7f1347d75700 1 -- 172.21.15.121:0/3480486149 shutdown_connections 2024-04-05T21:59:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.054+0000 7f1347d75700 1 -- 172.21.15.121:0/3480486149 wait complete. 2024-04-05T21:59:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:59:05.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.158+0000 7f93776d7700 1 -- 172.21.15.121:0/3209344317 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9370105a40 msgr2=0x7f9370105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:05.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.158+0000 7f93776d7700 1 --2- 172.21.15.121:0/3209344317 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9370105a40 0x7f9370105e20 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f9360009910 tx=0x7f9360009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:05.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.158+0000 7f93776d7700 1 -- 172.21.15.121:0/3209344317 shutdown_connections 2024-04-05T21:59:05.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.158+0000 7f93776d7700 1 --2- 172.21.15.121:0/3209344317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93700ffa40 0x7f93700ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.158+0000 7f93776d7700 1 --2- 172.21.15.121:0/3209344317 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9370105a40 0x7f9370105e20 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.158+0000 7f93776d7700 1 -- 172.21.15.121:0/3209344317 >> 172.21.15.121:0/3209344317 conn(0x7f93700fb570 msgr2=0x7f93700fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:05.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.158+0000 7f93776d7700 1 -- 172.21.15.121:0/3209344317 shutdown_connections 2024-04-05T21:59:05.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.158+0000 7f93776d7700 1 -- 172.21.15.121:0/3209344317 wait complete. 2024-04-05T21:59:05.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f93776d7700 1 Processor -- start 2024-04-05T21:59:05.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f93776d7700 1 -- start start 2024-04-05T21:59:05.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f93776d7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93700ffa40 0x7f9370196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:05.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f93776d7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9370105a40 0x7f9370196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:05.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f93776d7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9370197290 con 0x7f9370105a40 2024-04-05T21:59:05.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f93776d7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937019b020 con 0x7f93700ffa40 2024-04-05T21:59:05.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f9374c72700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9370105a40 0x7f9370196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:05.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f9374c72700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9370105a40 0x7f9370196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60660/0 (socket says 172.21.15.121:60660) 2024-04-05T21:59:05.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f9374c72700 1 -- 172.21.15.121:0/2417058636 learned_addr learned my addr 172.21.15.121:0/2417058636 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:05.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f9375473700 1 --2- 172.21.15.121:0/2417058636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93700ffa40 0x7f9370196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:05.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f9374c72700 1 -- 172.21.15.121:0/2417058636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93700ffa40 msgr2=0x7f9370196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:05.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f9374c72700 1 --2- 172.21.15.121:0/2417058636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93700ffa40 0x7f9370196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f9374c72700 1 -- 172.21.15.121:0/2417058636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93600095f0 con 0x7f9370105a40 2024-04-05T21:59:05.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.159+0000 7f9374c72700 1 --2- 172.21.15.121:0/2417058636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9370105a40 0x7f9370196bb0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f936c00d8d0 tx=0x7f936c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:05.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.160+0000 7f93667fc700 1 -- 172.21.15.121:0/2417058636 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f936c0098a0 con 0x7f9370105a40 2024-04-05T21:59:05.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.160+0000 7f93776d7700 1 -- 172.21.15.121:0/2417058636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f937019b300 con 0x7f9370105a40 2024-04-05T21:59:05.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.160+0000 7f93667fc700 1 -- 172.21.15.121:0/2417058636 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f936c010460 con 0x7f9370105a40 2024-04-05T21:59:05.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.160+0000 7f93667fc700 1 -- 172.21.15.121:0/2417058636 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f936c009ea0 con 0x7f9370105a40 2024-04-05T21:59:05.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.160+0000 7f93776d7700 1 -- 172.21.15.121:0/2417058636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f937019b850 con 0x7f9370105a40 2024-04-05T21:59:05.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.161+0000 7f93776d7700 1 -- 172.21.15.121:0/2417058636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9370109050 con 0x7f9370105a40 2024-04-05T21:59:05.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.161+0000 7f93667fc700 1 -- 172.21.15.121:0/2417058636 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f936c0105d0 con 0x7f9370105a40 2024-04-05T21:59:05.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.162+0000 7f93667fc700 1 --2- 172.21.15.121:0/2417058636 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f935c06cce0 0x7f935c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:05.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.162+0000 7f93667fc700 1 -- 172.21.15.121:0/2417058636 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f936c08c480 con 0x7f9370105a40 2024-04-05T21:59:05.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.162+0000 7f9375473700 1 --2- 172.21.15.121:0/2417058636 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f935c06cce0 0x7f935c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:05.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.162+0000 7f9375473700 1 --2- 172.21.15.121:0/2417058636 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f935c06cce0 0x7f935c06f1a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9360006010 tx=0x7f936001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:05.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.166+0000 7f93667fc700 1 -- 172.21.15.121:0/2417058636 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f936c0564f0 con 0x7f9370105a40 2024-04-05T21:59:05.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:04 smithi121 ceph-mon[145859]: from='client.35606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:05.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:04 smithi121 ceph-mon[145859]: pgmap v986: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:05.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:04 smithi121 ceph-mon[145859]: from='client.35610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:05.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:04 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2723995060' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:59:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:04 smithi188 ceph-mon[107793]: from='client.35606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:04 smithi188 ceph-mon[107793]: pgmap v986: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:04 smithi188 ceph-mon[107793]: from='client.35610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:04 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2723995060' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:59:05.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.375+0000 7f93776d7700 1 -- 172.21.15.121:0/2417058636 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f937019bbb0 con 0x7f935c06cce0 2024-04-05T21:59:05.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.376+0000 7f93667fc700 1 -- 172.21.15.121:0/2417058636 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f937019bbb0 con 0x7f935c06cce0 2024-04-05T21:59:05.377 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:59:05.377 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:59:05.377 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:59:05.377 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:59:05.377 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:59:05.377 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:59:05.377 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:59:05.377 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:59:05.377 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:59:05.377 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:59:05.377 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:59:05.377 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:59:05.378 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:59:05.378 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:59:05.378 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:59:05.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.378+0000 7f93776d7700 1 -- 172.21.15.121:0/2417058636 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f935c06cce0 msgr2=0x7f935c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:05.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.378+0000 7f93776d7700 1 --2- 172.21.15.121:0/2417058636 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f935c06cce0 0x7f935c06f1a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9360006010 tx=0x7f936001a040 comp rx=0 tx=0).stop 2024-04-05T21:59:05.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.378+0000 7f93776d7700 1 -- 172.21.15.121:0/2417058636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9370105a40 msgr2=0x7f9370196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:05.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.378+0000 7f93776d7700 1 --2- 172.21.15.121:0/2417058636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9370105a40 0x7f9370196bb0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f936c00d8d0 tx=0x7f936c00dbe0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.378+0000 7f93776d7700 1 -- 172.21.15.121:0/2417058636 shutdown_connections 2024-04-05T21:59:05.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.378+0000 7f93776d7700 1 --2- 172.21.15.121:0/2417058636 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f935c06cce0 0x7f935c06f1a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.378+0000 7f93776d7700 1 --2- 172.21.15.121:0/2417058636 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f93700ffa40 0x7f9370196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.378+0000 7f93776d7700 1 --2- 172.21.15.121:0/2417058636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9370105a40 0x7f9370196bb0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.378+0000 7f93776d7700 1 -- 172.21.15.121:0/2417058636 >> 172.21.15.121:0/2417058636 conn(0x7f93700fb570 msgr2=0x7f9370068f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:05.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.378+0000 7f93776d7700 1 -- 172.21.15.121:0/2417058636 shutdown_connections 2024-04-05T21:59:05.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.378+0000 7f93776d7700 1 -- 172.21.15.121:0/2417058636 wait complete. 2024-04-05T21:59:05.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.482+0000 7fe47d290700 1 -- 172.21.15.121:0/3536213763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4780ffbd0 msgr2=0x7fe478100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.482+0000 7fe47d290700 1 --2- 172.21.15.121:0/3536213763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4780ffbd0 0x7fe478100050 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe460009910 tx=0x7fe460009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.482+0000 7fe47d290700 1 -- 172.21.15.121:0/3536213763 shutdown_connections 2024-04-05T21:59:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.482+0000 7fe47d290700 1 --2- 172.21.15.121:0/3536213763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4780ffbd0 0x7fe478100050 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.482+0000 7fe47d290700 1 --2- 172.21.15.121:0/3536213763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe478105a40 0x7fe478105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.482+0000 7fe47d290700 1 -- 172.21.15.121:0/3536213763 >> 172.21.15.121:0/3536213763 conn(0x7fe4780755a0 msgr2=0x7fe4780759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.482+0000 7fe47d290700 1 -- 172.21.15.121:0/3536213763 shutdown_connections 2024-04-05T21:59:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.482+0000 7fe47d290700 1 -- 172.21.15.121:0/3536213763 wait complete. 2024-04-05T21:59:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.482+0000 7fe47d290700 1 Processor -- start 2024-04-05T21:59:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.482+0000 7fe47d290700 1 -- start start 2024-04-05T21:59:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe47d290700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4780ffbd0 0x7fe4781974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe47d290700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe478105a40 0x7fe4781979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe47d290700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe478197f80 con 0x7fe478105a40 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe47d290700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4781980c0 con 0x7fe4780ffbd0 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe4767fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe478105a40 0x7fe4781979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe4767fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe478105a40 0x7fe4781979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60664/0 (socket says 172.21.15.121:60664) 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe4767fc700 1 -- 172.21.15.121:0/3614280792 learned_addr learned my addr 172.21.15.121:0/3614280792 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe476ffd700 1 --2- 172.21.15.121:0/3614280792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4780ffbd0 0x7fe4781974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe4767fc700 1 -- 172.21.15.121:0/3614280792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4780ffbd0 msgr2=0x7fe4781974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe4767fc700 1 --2- 172.21.15.121:0/3614280792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4780ffbd0 0x7fe4781974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe4767fc700 1 -- 172.21.15.121:0/3614280792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4600095f0 con 0x7fe478105a40 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe4767fc700 1 --2- 172.21.15.121:0/3614280792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe478105a40 0x7fe4781979f0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fe460006010 tx=0x7fe46000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe46ffff700 1 -- 172.21.15.121:0/3614280792 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe46001d070 con 0x7fe478105a40 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe47d290700 1 -- 172.21.15.121:0/3614280792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4781a0520 con 0x7fe478105a40 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe46ffff700 1 -- 172.21.15.121:0/3614280792 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe46000f460 con 0x7fe478105a40 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe46ffff700 1 -- 172.21.15.121:0/3614280792 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe460021640 con 0x7fe478105a40 2024-04-05T21:59:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.483+0000 7fe47d290700 1 -- 172.21.15.121:0/3614280792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4781a08d0 con 0x7fe478105a40 2024-04-05T21:59:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.484+0000 7fe47d290700 1 -- 172.21.15.121:0/3614280792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe478107fd0 con 0x7fe478105a40 2024-04-05T21:59:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.485+0000 7fe46ffff700 1 -- 172.21.15.121:0/3614280792 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe46000fa20 con 0x7fe478105a40 2024-04-05T21:59:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.485+0000 7fe46ffff700 1 --2- 172.21.15.121:0/3614280792 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe46406cce0 0x7fe46406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.485+0000 7fe46ffff700 1 -- 172.21.15.121:0/3614280792 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe46008d8c0 con 0x7fe478105a40 2024-04-05T21:59:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.485+0000 7fe476ffd700 1 --2- 172.21.15.121:0/3614280792 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe46406cce0 0x7fe46406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.486+0000 7fe476ffd700 1 --2- 172.21.15.121:0/3614280792 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe46406cce0 0x7fe46406f1a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe47000e270 tx=0x7fe470009040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.490+0000 7fe46ffff700 1 -- 172.21.15.121:0/3614280792 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe460057930 con 0x7fe478105a40 2024-04-05T21:59:05.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.760+0000 7fe47d290700 1 -- 172.21.15.121:0/3614280792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe47804ea90 con 0x7fe478105a40 2024-04-05T21:59:05.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.761+0000 7fe46ffff700 1 -- 172.21.15.121:0/3614280792 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fe460026e10 con 0x7fe478105a40 2024-04-05T21:59:05.761 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:59:05.761 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:59:05.761 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:59:05.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:59:05.761 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:59:05.761 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:59:05.762 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:59:05.762 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:59:05.762 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:59:05.762 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:59:05.762 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:59:05.762 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:59:05.762 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:59:05.762 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:59:05.762 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:59:05.762 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:59:05.762 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:59:05.762 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:59:05.762 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:59:05.763 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:59:05.763 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:59:05.763 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:59:05.763 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:59:05.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.763+0000 7fe47d290700 1 -- 172.21.15.121:0/3614280792 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe46406cce0 msgr2=0x7fe46406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:05.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.763+0000 7fe47d290700 1 --2- 172.21.15.121:0/3614280792 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe46406cce0 0x7fe46406f1a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe47000e270 tx=0x7fe470009040 comp rx=0 tx=0).stop 2024-04-05T21:59:05.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.763+0000 7fe47d290700 1 -- 172.21.15.121:0/3614280792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe478105a40 msgr2=0x7fe4781979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:05.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.763+0000 7fe47d290700 1 --2- 172.21.15.121:0/3614280792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe478105a40 0x7fe4781979f0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fe460006010 tx=0x7fe46000be20 comp rx=0 tx=0).stop 2024-04-05T21:59:05.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.763+0000 7fe47d290700 1 -- 172.21.15.121:0/3614280792 shutdown_connections 2024-04-05T21:59:05.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.763+0000 7fe47d290700 1 --2- 172.21.15.121:0/3614280792 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe46406cce0 0x7fe46406f1a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.763+0000 7fe47d290700 1 --2- 172.21.15.121:0/3614280792 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe4780ffbd0 0x7fe4781974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.763+0000 7fe47d290700 1 --2- 172.21.15.121:0/3614280792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe478105a40 0x7fe4781979f0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:05.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.763+0000 7fe47d290700 1 -- 172.21.15.121:0/3614280792 >> 172.21.15.121:0/3614280792 conn(0x7fe4780755a0 msgr2=0x7fe4780fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:05.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.763+0000 7fe47d290700 1 -- 172.21.15.121:0/3614280792 shutdown_connections 2024-04-05T21:59:05.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:05.763+0000 7fe47d290700 1 -- 172.21.15.121:0/3614280792 wait complete. 2024-04-05T21:59:06.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:05 smithi121 ceph-mon[145859]: from='client.35614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:06.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:05 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3480486149' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:59:06.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:05 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3614280792' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:59:06.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:05 smithi188 ceph-mon[107793]: from='client.35614 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:06.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:05 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3480486149' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:59:06.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:05 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3614280792' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:59:07.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:06 smithi121 ceph-mon[145859]: from='client.35626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:07.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:06 smithi121 ceph-mon[145859]: pgmap v987: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:06 smithi188 ceph-mon[107793]: from='client.35626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:06 smithi188 ceph-mon[107793]: pgmap v987: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:09.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:08 smithi121 ceph-mon[145859]: pgmap v988: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:09.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:08 smithi188 ceph-mon[107793]: pgmap v988: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:10 smithi188 ceph-mon[107793]: pgmap v989: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:11.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:10 smithi121 ceph-mon[145859]: pgmap v989: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:12 smithi188 ceph-mon[107793]: pgmap v990: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:13.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:12 smithi121 ceph-mon[145859]: pgmap v990: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:14 smithi188 ceph-mon[107793]: pgmap v991: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:15.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:14 smithi121 ceph-mon[145859]: pgmap v991: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:59:16.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:59:17.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:17 smithi188 ceph-mon[107793]: pgmap v992: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:17.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:17 smithi121 ceph-mon[145859]: pgmap v992: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:18.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:18 smithi188 ceph-mon[107793]: pgmap v993: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:18.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:18 smithi121 ceph-mon[145859]: pgmap v993: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:21.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:20 smithi121 ceph-mon[145859]: pgmap v994: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:20 smithi188 ceph-mon[107793]: pgmap v994: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:23.166 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:22 smithi188 ceph-mon[107793]: pgmap v995: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:23.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:22 smithi121 ceph-mon[145859]: pgmap v995: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:25.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:24 smithi121 ceph-mon[145859]: pgmap v996: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:24 smithi188 ceph-mon[107793]: pgmap v996: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:27.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:26 smithi121 ceph-mon[145859]: pgmap v997: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:27.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:26 smithi188 ceph-mon[107793]: pgmap v997: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:29.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:28 smithi121 ceph-mon[145859]: pgmap v998: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:29.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:28 smithi188 ceph-mon[107793]: pgmap v998: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:30 smithi121 ceph-mon[145859]: pgmap v999: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:31.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:59:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:30 smithi188 ceph-mon[107793]: pgmap v999: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:59:33.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:32 smithi121 ceph-mon[145859]: pgmap v1000: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:33.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:32 smithi188 ceph-mon[107793]: pgmap v1000: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:35.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:34 smithi121 ceph-mon[145859]: pgmap v1001: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:35.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:34 smithi188 ceph-mon[107793]: pgmap v1001: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:35.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.879+0000 7f8c79f48700 1 -- 172.21.15.121:0/3348157771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740ffa40 msgr2=0x7f8c740ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:35.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.879+0000 7f8c79f48700 1 --2- 172.21.15.121:0/3348157771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740ffa40 0x7f8c740ffec0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009910 tx=0x7f8c64009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:35.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.879+0000 7f8c79f48700 1 -- 172.21.15.121:0/3348157771 shutdown_connections 2024-04-05T21:59:35.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.879+0000 7f8c79f48700 1 --2- 172.21.15.121:0/3348157771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740ffa40 0x7f8c740ffec0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:35.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.879+0000 7f8c79f48700 1 --2- 172.21.15.121:0/3348157771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c74105a40 0x7f8c74105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:35.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.879+0000 7f8c79f48700 1 -- 172.21.15.121:0/3348157771 >> 172.21.15.121:0/3348157771 conn(0x7f8c740fb570 msgr2=0x7f8c740fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:35.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.879+0000 7f8c79f48700 1 -- 172.21.15.121:0/3348157771 shutdown_connections 2024-04-05T21:59:35.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.879+0000 7f8c79f48700 1 -- 172.21.15.121:0/3348157771 wait complete. 2024-04-05T21:59:35.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.880+0000 7f8c79f48700 1 Processor -- start 2024-04-05T21:59:35.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.880+0000 7f8c79f48700 1 -- start start 2024-04-05T21:59:35.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.880+0000 7f8c79f48700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740ffa40 0x7f8c74195560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.880+0000 7f8c79f48700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c74105a40 0x7f8c74195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.880+0000 7f8c79f48700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c74196180 con 0x7f8c740ffa40 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.880+0000 7f8c79f48700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c74199f10 con 0x7f8c74105a40 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.880+0000 7f8c737fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740ffa40 0x7f8c74195560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.881+0000 7f8c737fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740ffa40 0x7f8c74195560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60020/0 (socket says 172.21.15.121:60020) 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.881+0000 7f8c737fe700 1 -- 172.21.15.121:0/2328404312 learned_addr learned my addr 172.21.15.121:0/2328404312 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.881+0000 7f8c72ffd700 1 --2- 172.21.15.121:0/2328404312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c74105a40 0x7f8c74195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.881+0000 7f8c737fe700 1 -- 172.21.15.121:0/2328404312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c74105a40 msgr2=0x7f8c74195aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.881+0000 7f8c737fe700 1 --2- 172.21.15.121:0/2328404312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c74105a40 0x7f8c74195aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.881+0000 7f8c737fe700 1 -- 172.21.15.121:0/2328404312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c640095f0 con 0x7f8c740ffa40 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.881+0000 7f8c72ffd700 1 --2- 172.21.15.121:0/2328404312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c74105a40 0x7f8c74195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.881+0000 7f8c737fe700 1 --2- 172.21.15.121:0/2328404312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740ffa40 0x7f8c74195560 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f8c680098f0 tx=0x7f8c68009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.881+0000 7f8c70ff9700 1 -- 172.21.15.121:0/2328404312 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c68012070 con 0x7f8c740ffa40 2024-04-05T21:59:35.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.881+0000 7f8c79f48700 1 -- 172.21.15.121:0/2328404312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7419a190 con 0x7f8c740ffa40 2024-04-05T21:59:35.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.881+0000 7f8c70ff9700 1 -- 172.21.15.121:0/2328404312 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c6800d850 con 0x7f8c740ffa40 2024-04-05T21:59:35.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.881+0000 7f8c70ff9700 1 -- 172.21.15.121:0/2328404312 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c68005700 con 0x7f8c740ffa40 2024-04-05T21:59:35.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.881+0000 7f8c79f48700 1 -- 172.21.15.121:0/2328404312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7419a6e0 con 0x7f8c740ffa40 2024-04-05T21:59:35.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.882+0000 7f8c79f48700 1 -- 172.21.15.121:0/2328404312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7404ea90 con 0x7f8c740ffa40 2024-04-05T21:59:35.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.883+0000 7f8c70ff9700 1 -- 172.21.15.121:0/2328404312 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8c68016070 con 0x7f8c740ffa40 2024-04-05T21:59:35.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.883+0000 7f8c70ff9700 1 --2- 172.21.15.121:0/2328404312 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8c5c06cce0 0x7f8c5c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:35.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.883+0000 7f8c70ff9700 1 -- 172.21.15.121:0/2328404312 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8c6808e290 con 0x7f8c740ffa40 2024-04-05T21:59:35.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.884+0000 7f8c72ffd700 1 --2- 172.21.15.121:0/2328404312 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8c5c06cce0 0x7f8c5c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:35.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.884+0000 7f8c72ffd700 1 --2- 172.21.15.121:0/2328404312 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8c5c06cce0 0x7f8c5c06f1a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400b5c0 tx=0x7f8c64005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:35.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:35.888+0000 7f8c70ff9700 1 -- 172.21.15.121:0/2328404312 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8c68058380 con 0x7f8c740ffa40 2024-04-05T21:59:36.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.097+0000 7f8c79f48700 1 -- 172.21.15.121:0/2328404312 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c7419abc0 con 0x7f8c5c06cce0 2024-04-05T21:59:36.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.098+0000 7f8c70ff9700 1 -- 172.21.15.121:0/2328404312 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8c7419abc0 con 0x7f8c5c06cce0 2024-04-05T21:59:36.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.100+0000 7f8c79f48700 1 -- 172.21.15.121:0/2328404312 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8c5c06cce0 msgr2=0x7f8c5c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:36.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.100+0000 7f8c79f48700 1 --2- 172.21.15.121:0/2328404312 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8c5c06cce0 0x7f8c5c06f1a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400b5c0 tx=0x7f8c64005960 comp rx=0 tx=0).stop 2024-04-05T21:59:36.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.100+0000 7f8c79f48700 1 -- 172.21.15.121:0/2328404312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740ffa40 msgr2=0x7f8c74195560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:36.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.100+0000 7f8c79f48700 1 --2- 172.21.15.121:0/2328404312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740ffa40 0x7f8c74195560 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f8c680098f0 tx=0x7f8c68009cb0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.100+0000 7f8c79f48700 1 -- 172.21.15.121:0/2328404312 shutdown_connections 2024-04-05T21:59:36.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.100+0000 7f8c79f48700 1 --2- 172.21.15.121:0/2328404312 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8c5c06cce0 0x7f8c5c06f1a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.100+0000 7f8c79f48700 1 --2- 172.21.15.121:0/2328404312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740ffa40 0x7f8c74195560 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.100+0000 7f8c79f48700 1 --2- 172.21.15.121:0/2328404312 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8c74105a40 0x7f8c74195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.100+0000 7f8c79f48700 1 -- 172.21.15.121:0/2328404312 >> 172.21.15.121:0/2328404312 conn(0x7f8c740fb570 msgr2=0x7f8c740fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:36.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.100+0000 7f8c79f48700 1 -- 172.21.15.121:0/2328404312 shutdown_connections 2024-04-05T21:59:36.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.100+0000 7f8c79f48700 1 -- 172.21.15.121:0/2328404312 wait complete. 2024-04-05T21:59:36.111 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T21:59:36.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.204+0000 7fceeb02a700 1 -- 172.21.15.121:0/1568736825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100de0 msgr2=0x7fcee41011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:36.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.204+0000 7fceeb02a700 1 --2- 172.21.15.121:0/1568736825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100de0 0x7fcee41011c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fced8009910 tx=0x7fced8009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:36.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.204+0000 7fceeb02a700 1 -- 172.21.15.121:0/1568736825 shutdown_connections 2024-04-05T21:59:36.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.204+0000 7fceeb02a700 1 --2- 172.21.15.121:0/1568736825 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcee4101790 0x7fcee41057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.204+0000 7fceeb02a700 1 --2- 172.21.15.121:0/1568736825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100de0 0x7fcee41011c0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.204+0000 7fceeb02a700 1 -- 172.21.15.121:0/1568736825 >> 172.21.15.121:0/1568736825 conn(0x7fcee40fc670 msgr2=0x7fcee40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.204+0000 7fceeb02a700 1 -- 172.21.15.121:0/1568736825 shutdown_connections 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.204+0000 7fceeb02a700 1 -- 172.21.15.121:0/1568736825 wait complete. 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fceeb02a700 1 Processor -- start 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fceeb02a700 1 -- start start 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fceeb02a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100de0 0x7fcee4196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fceeb02a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcee4101790 0x7fcee4196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fceeb02a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee4197290 con 0x7fcee4100de0 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fceeb02a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee419b020 con 0x7fcee4101790 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fcee8dc6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100de0 0x7fcee4196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fcee8dc6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100de0 0x7fcee4196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60048/0 (socket says 172.21.15.121:60048) 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fcee8dc6700 1 -- 172.21.15.121:0/1105661909 learned_addr learned my addr 172.21.15.121:0/1105661909 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fcee3fff700 1 --2- 172.21.15.121:0/1105661909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcee4101790 0x7fcee4196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fcee8dc6700 1 -- 172.21.15.121:0/1105661909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcee4101790 msgr2=0x7fcee4196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:36.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fcee8dc6700 1 --2- 172.21.15.121:0/1105661909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcee4101790 0x7fcee4196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fcee8dc6700 1 -- 172.21.15.121:0/1105661909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced80095f0 con 0x7fcee4100de0 2024-04-05T21:59:36.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fcee8dc6700 1 --2- 172.21.15.121:0/1105661909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100de0 0x7fcee4196670 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fced8006010 tx=0x7fced800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:36.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fcee1ffb700 1 -- 172.21.15.121:0/1105661909 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fced801d070 con 0x7fcee4100de0 2024-04-05T21:59:36.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fcee1ffb700 1 -- 172.21.15.121:0/1105661909 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fced800f460 con 0x7fcee4100de0 2024-04-05T21:59:36.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fceeb02a700 1 -- 172.21.15.121:0/1105661909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee419b2a0 con 0x7fcee4100de0 2024-04-05T21:59:36.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.205+0000 7fcee1ffb700 1 -- 172.21.15.121:0/1105661909 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fced8021640 con 0x7fcee4100de0 2024-04-05T21:59:36.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.206+0000 7fceeb02a700 1 -- 172.21.15.121:0/1105661909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee419b790 con 0x7fcee4100de0 2024-04-05T21:59:36.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.206+0000 7fceeb02a700 1 -- 172.21.15.121:0/1105661909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee4109180 con 0x7fcee4100de0 2024-04-05T21:59:36.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.207+0000 7fcee1ffb700 1 -- 172.21.15.121:0/1105661909 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fced800fa60 con 0x7fcee4100de0 2024-04-05T21:59:36.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.207+0000 7fcee1ffb700 1 --2- 172.21.15.121:0/1105661909 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fced406cd30 0x7fced406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:36.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.208+0000 7fcee1ffb700 1 -- 172.21.15.121:0/1105661909 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fced808d880 con 0x7fcee4100de0 2024-04-05T21:59:36.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.208+0000 7fcee3fff700 1 --2- 172.21.15.121:0/1105661909 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fced406cd30 0x7fced406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:36.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.208+0000 7fcee3fff700 1 --2- 172.21.15.121:0/1105661909 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fced406cd30 0x7fced406f1f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fced0005fd0 tx=0x7fced000a560 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:36.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.212+0000 7fcee1ffb700 1 -- 172.21.15.121:0/1105661909 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fced80578d0 con 0x7fcee4100de0 2024-04-05T21:59:36.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.418+0000 7fceeb02a700 1 -- 172.21.15.121:0/1105661909 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcee419bb30 con 0x7fced406cd30 2024-04-05T21:59:36.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.420+0000 7fcee1ffb700 1 -- 172.21.15.121:0/1105661909 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcee419bb30 con 0x7fced406cd30 2024-04-05T21:59:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.422+0000 7fceeb02a700 1 -- 172.21.15.121:0/1105661909 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fced406cd30 msgr2=0x7fced406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.422+0000 7fceeb02a700 1 --2- 172.21.15.121:0/1105661909 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fced406cd30 0x7fced406f1f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fced0005fd0 tx=0x7fced000a560 comp rx=0 tx=0).stop 2024-04-05T21:59:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.422+0000 7fceeb02a700 1 -- 172.21.15.121:0/1105661909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100de0 msgr2=0x7fcee4196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.422+0000 7fceeb02a700 1 --2- 172.21.15.121:0/1105661909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100de0 0x7fcee4196670 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fced8006010 tx=0x7fced800be20 comp rx=0 tx=0).stop 2024-04-05T21:59:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.422+0000 7fceeb02a700 1 -- 172.21.15.121:0/1105661909 shutdown_connections 2024-04-05T21:59:36.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.422+0000 7fceeb02a700 1 --2- 172.21.15.121:0/1105661909 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fced406cd30 0x7fced406f1f0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.422+0000 7fceeb02a700 1 --2- 172.21.15.121:0/1105661909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee4100de0 0x7fcee4196670 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.422+0000 7fceeb02a700 1 --2- 172.21.15.121:0/1105661909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcee4101790 0x7fcee4196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.422+0000 7fceeb02a700 1 -- 172.21.15.121:0/1105661909 >> 172.21.15.121:0/1105661909 conn(0x7fcee40fc670 msgr2=0x7fcee40fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:36.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.422+0000 7fceeb02a700 1 -- 172.21.15.121:0/1105661909 shutdown_connections 2024-04-05T21:59:36.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.422+0000 7fceeb02a700 1 -- 172.21.15.121:0/1105661909 wait complete. 2024-04-05T21:59:36.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.524+0000 7fb96f436700 1 -- 172.21.15.121:0/597067960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb968104a50 msgr2=0x7fb968104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:36.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.524+0000 7fb96f436700 1 --2- 172.21.15.121:0/597067960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb968104a50 0x7fb968104e30 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb958009910 tx=0x7fb958009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:36.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.524+0000 7fb96f436700 1 -- 172.21.15.121:0/597067960 shutdown_connections 2024-04-05T21:59:36.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.524+0000 7fb96f436700 1 --2- 172.21.15.121:0/597067960 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb968100180 0x7fb968100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.524+0000 7fb96f436700 1 --2- 172.21.15.121:0/597067960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb968104a50 0x7fb968104e30 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.524+0000 7fb96f436700 1 -- 172.21.15.121:0/597067960 >> 172.21.15.121:0/597067960 conn(0x7fb968075e30 msgr2=0x7fb9680fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:36.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.524+0000 7fb96f436700 1 -- 172.21.15.121:0/597067960 shutdown_connections 2024-04-05T21:59:36.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.524+0000 7fb96f436700 1 -- 172.21.15.121:0/597067960 wait complete. 2024-04-05T21:59:36.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.524+0000 7fb96f436700 1 Processor -- start 2024-04-05T21:59:36.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.524+0000 7fb96f436700 1 -- start start 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96f436700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb968100180 0x7fb9680fb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96f436700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb968104a50 0x7fb9680fb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96f436700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9680fbd20 con 0x7fb968100180 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96f436700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9680fbe60 con 0x7fb968104a50 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96d1d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb968100180 0x7fb9680fb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96d1d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb968100180 0x7fb9680fb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60068/0 (socket says 172.21.15.121:60068) 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96d1d2700 1 -- 172.21.15.121:0/4196293388 learned_addr learned my addr 172.21.15.121:0/4196293388 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96c9d1700 1 --2- 172.21.15.121:0/4196293388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb968104a50 0x7fb9680fb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96d1d2700 1 -- 172.21.15.121:0/4196293388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb968104a50 msgr2=0x7fb9680fb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96d1d2700 1 --2- 172.21.15.121:0/4196293388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb968104a50 0x7fb9680fb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96d1d2700 1 -- 172.21.15.121:0/4196293388 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9580095f0 con 0x7fb968100180 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96c9d1700 1 --2- 172.21.15.121:0/4196293388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb968104a50 0x7fb9680fb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96d1d2700 1 --2- 172.21.15.121:0/4196293388 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb968100180 0x7fb9680fb2a0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fb9580092e0 tx=0x7fb95800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:36.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb95e7fc700 1 -- 172.21.15.121:0/4196293388 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95801d070 con 0x7fb968100180 2024-04-05T21:59:36.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb95e7fc700 1 -- 172.21.15.121:0/4196293388 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb958005e00 con 0x7fb968100180 2024-04-05T21:59:36.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96f436700 1 -- 172.21.15.121:0/4196293388 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9680fc0e0 con 0x7fb968100180 2024-04-05T21:59:36.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb95e7fc700 1 -- 172.21.15.121:0/4196293388 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb95800fe60 con 0x7fb968100180 2024-04-05T21:59:36.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.525+0000 7fb96f436700 1 -- 172.21.15.121:0/4196293388 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9681a2b40 con 0x7fb968100180 2024-04-05T21:59:36.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.526+0000 7fb96f436700 1 -- 172.21.15.121:0/4196293388 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96804ea90 con 0x7fb968100180 2024-04-05T21:59:36.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.527+0000 7fb95e7fc700 1 -- 172.21.15.121:0/4196293388 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb95800f6d0 con 0x7fb968100180 2024-04-05T21:59:36.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.527+0000 7fb95e7fc700 1 --2- 172.21.15.121:0/4196293388 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb95406cce0 0x7fb95406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:36.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.527+0000 7fb96c9d1700 1 --2- 172.21.15.121:0/4196293388 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb95406cce0 0x7fb95406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:36.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.527+0000 7fb95e7fc700 1 -- 172.21.15.121:0/4196293388 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb95808d7e0 con 0x7fb968100180 2024-04-05T21:59:36.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.528+0000 7fb96c9d1700 1 --2- 172.21.15.121:0/4196293388 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb95406cce0 0x7fb95406f1a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb964005fd0 tx=0x7fb964005ee0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:36.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.532+0000 7fb95e7fc700 1 -- 172.21.15.121:0/4196293388 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb958057850 con 0x7fb968100180 2024-04-05T21:59:36.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.730+0000 7fb96f436700 1 -- 172.21.15.121:0/4196293388 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb968102b30 con 0x7fb95406cce0 2024-04-05T21:59:36.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.740+0000 7fb95e7fc700 1 -- 172.21.15.121:0/4196293388 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fb968102b30 con 0x7fb95406cce0 2024-04-05T21:59:36.740 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T21:59:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (30m) 8m ago 41m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T21:59:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (42m) 8m ago 42m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T21:59:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (41m) 8m ago 41m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T21:59:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (28m) 8m ago 42m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T21:59:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (28m) 8m ago 40m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T21:59:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (30m) 8m ago 41m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T21:59:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (24m) 8m ago 35m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T21:59:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (24m) 8m ago 35m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T21:59:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (24m) 8m ago 35m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T21:59:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (9m) 8m ago 35m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T21:59:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (32m) 8m ago 43m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T21:59:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (31m) 8m ago 40m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T21:59:36.741 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (29m) 8m ago 43m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T21:59:36.742 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (28m) 8m ago 40m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T21:59:36.742 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (31m) 8m ago 42m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T21:59:36.742 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (31m) 8m ago 40m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T21:59:36.742 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (28m) 8m ago 39m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T21:59:36.742 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (27m) 8m ago 39m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T21:59:36.742 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (26m) 8m ago 38m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T21:59:36.742 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (26m) 8m ago 38m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T21:59:36.742 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (25m) 8m ago 37m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T21:59:36.742 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (25m) 8m ago 37m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T21:59:36.742 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (30m) 8m ago 41m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T21:59:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.742+0000 7fb96f436700 1 -- 172.21.15.121:0/4196293388 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb95406cce0 msgr2=0x7fb95406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.742+0000 7fb96f436700 1 --2- 172.21.15.121:0/4196293388 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb95406cce0 0x7fb95406f1a0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb964005fd0 tx=0x7fb964005ee0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.742+0000 7fb96f436700 1 -- 172.21.15.121:0/4196293388 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb968100180 msgr2=0x7fb9680fb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.742+0000 7fb96f436700 1 --2- 172.21.15.121:0/4196293388 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb968100180 0x7fb9680fb2a0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fb9580092e0 tx=0x7fb95800be20 comp rx=0 tx=0).stop 2024-04-05T21:59:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.742+0000 7fb96f436700 1 -- 172.21.15.121:0/4196293388 shutdown_connections 2024-04-05T21:59:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.742+0000 7fb96f436700 1 --2- 172.21.15.121:0/4196293388 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb95406cce0 0x7fb95406f1a0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.742+0000 7fb96f436700 1 --2- 172.21.15.121:0/4196293388 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb968100180 0x7fb9680fb2a0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.742+0000 7fb96f436700 1 --2- 172.21.15.121:0/4196293388 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb968104a50 0x7fb9680fb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.742+0000 7fb96f436700 1 -- 172.21.15.121:0/4196293388 >> 172.21.15.121:0/4196293388 conn(0x7fb968075e30 msgr2=0x7fb9680fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:36.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.742+0000 7fb96f436700 1 -- 172.21.15.121:0/4196293388 shutdown_connections 2024-04-05T21:59:36.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.742+0000 7fb96f436700 1 -- 172.21.15.121:0/4196293388 wait complete. 2024-04-05T21:59:36.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.843+0000 7fecbfdee700 1 -- 172.21.15.121:0/4242962828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb80ffbd0 msgr2=0x7fecb8100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:36.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.843+0000 7fecbfdee700 1 --2- 172.21.15.121:0/4242962828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb80ffbd0 0x7fecb8100050 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7feca8009910 tx=0x7feca8009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:36.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.843+0000 7fecbfdee700 1 -- 172.21.15.121:0/4242962828 shutdown_connections 2024-04-05T21:59:36.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.843+0000 7fecbfdee700 1 --2- 172.21.15.121:0/4242962828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb80ffbd0 0x7fecb8100050 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.843+0000 7fecbfdee700 1 --2- 172.21.15.121:0/4242962828 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fecb8105a40 0x7fecb8105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.843+0000 7fecbfdee700 1 -- 172.21.15.121:0/4242962828 >> 172.21.15.121:0/4242962828 conn(0x7fecb80755a0 msgr2=0x7fecb80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.843+0000 7fecbfdee700 1 -- 172.21.15.121:0/4242962828 shutdown_connections 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.843+0000 7fecbfdee700 1 -- 172.21.15.121:0/4242962828 wait complete. 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbfdee700 1 Processor -- start 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbfdee700 1 -- start start 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbfdee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb80ffbd0 0x7fecb8197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbfdee700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fecb8105a40 0x7fecb81979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbfdee700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb8197ff0 con 0x7fecb80ffbd0 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbfdee700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb8198160 con 0x7fecb8105a40 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbdb8a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb80ffbd0 0x7fecb8197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbdb8a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb80ffbd0 0x7fecb8197490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60086/0 (socket says 172.21.15.121:60086) 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbdb8a700 1 -- 172.21.15.121:0/3629393525 learned_addr learned my addr 172.21.15.121:0/3629393525 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:36.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbd389700 1 --2- 172.21.15.121:0/3629393525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fecb8105a40 0x7fecb81979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:36.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbdb8a700 1 -- 172.21.15.121:0/3629393525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fecb8105a40 msgr2=0x7fecb81979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:36.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbdb8a700 1 --2- 172.21.15.121:0/3629393525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fecb8105a40 0x7fecb81979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:36.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbdb8a700 1 -- 172.21.15.121:0/3629393525 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca80095f0 con 0x7fecb80ffbd0 2024-04-05T21:59:36.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.844+0000 7fecbd389700 1 --2- 172.21.15.121:0/3629393525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fecb8105a40 0x7fecb81979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T21:59:36.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.845+0000 7fecbdb8a700 1 --2- 172.21.15.121:0/3629393525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb80ffbd0 0x7fecb8197490 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fecb400e920 tx=0x7fecb400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:36.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.845+0000 7fecaeffd700 1 -- 172.21.15.121:0/3629393525 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecb4009790 con 0x7fecb80ffbd0 2024-04-05T21:59:36.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.845+0000 7fecaeffd700 1 -- 172.21.15.121:0/3629393525 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecb400cba0 con 0x7fecb80ffbd0 2024-04-05T21:59:36.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.845+0000 7fecbfdee700 1 -- 172.21.15.121:0/3629393525 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb81a0500 con 0x7fecb80ffbd0 2024-04-05T21:59:36.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.845+0000 7fecaeffd700 1 -- 172.21.15.121:0/3629393525 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecb4018650 con 0x7fecb80ffbd0 2024-04-05T21:59:36.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.845+0000 7fecbfdee700 1 -- 172.21.15.121:0/3629393525 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb81a09f0 con 0x7fecb80ffbd0 2024-04-05T21:59:36.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.846+0000 7fecbfdee700 1 -- 172.21.15.121:0/3629393525 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb8107fd0 con 0x7fecb80ffbd0 2024-04-05T21:59:36.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.846+0000 7fecaeffd700 1 -- 172.21.15.121:0/3629393525 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fecb400cd10 con 0x7fecb80ffbd0 2024-04-05T21:59:36.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.847+0000 7fecaeffd700 1 --2- 172.21.15.121:0/3629393525 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feca406cce0 0x7feca406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:36.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.847+0000 7fecbd389700 1 --2- 172.21.15.121:0/3629393525 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feca406cce0 0x7feca406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:36.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.847+0000 7fecaeffd700 1 -- 172.21.15.121:0/3629393525 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fecb4014070 con 0x7fecb80ffbd0 2024-04-05T21:59:36.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.847+0000 7fecbd389700 1 --2- 172.21.15.121:0/3629393525 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feca406cce0 0x7feca406f1a0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fecb8068ce0 tx=0x7feca8009f70 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:36.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:36.851+0000 7fecaeffd700 1 -- 172.21.15.121:0/3629393525 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fecb405a6c0 con 0x7fecb80ffbd0 2024-04-05T21:59:37.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.124+0000 7fecbfdee700 1 -- 172.21.15.121:0/3629393525 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fecb802cd70 con 0x7fecb80ffbd0 2024-04-05T21:59:37.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.124+0000 7fecaeffd700 1 -- 172.21.15.121:0/3629393525 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fecb405a250 con 0x7fecb80ffbd0 2024-04-05T21:59:37.125 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:59:37.125 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T21:59:37.125 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:59:37.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:59:37.125 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T21:59:37.125 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T21:59:37.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:59:37.126 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T21:59:37.126 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T21:59:37.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:59:37.126 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T21:59:37.126 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T21:59:37.126 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T21:59:37.126 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T21:59:37.126 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T21:59:37.126 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T21:59:37.126 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:59:37.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.126+0000 7fecbfdee700 1 -- 172.21.15.121:0/3629393525 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feca406cce0 msgr2=0x7feca406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:37.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.126+0000 7fecbfdee700 1 --2- 172.21.15.121:0/3629393525 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feca406cce0 0x7feca406f1a0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fecb8068ce0 tx=0x7feca8009f70 comp rx=0 tx=0).stop 2024-04-05T21:59:37.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.126+0000 7fecbfdee700 1 -- 172.21.15.121:0/3629393525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb80ffbd0 msgr2=0x7fecb8197490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:37.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.126+0000 7fecbfdee700 1 --2- 172.21.15.121:0/3629393525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb80ffbd0 0x7fecb8197490 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fecb400e920 tx=0x7fecb400ece0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.126+0000 7fecbfdee700 1 -- 172.21.15.121:0/3629393525 shutdown_connections 2024-04-05T21:59:37.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.126+0000 7fecbfdee700 1 --2- 172.21.15.121:0/3629393525 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feca406cce0 0x7feca406f1a0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.127+0000 7fecbfdee700 1 --2- 172.21.15.121:0/3629393525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecb80ffbd0 0x7fecb8197490 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.127+0000 7fecbfdee700 1 --2- 172.21.15.121:0/3629393525 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fecb8105a40 0x7fecb81979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.127+0000 7fecbfdee700 1 -- 172.21.15.121:0/3629393525 >> 172.21.15.121:0/3629393525 conn(0x7fecb80755a0 msgr2=0x7fecb80fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:37.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.127+0000 7fecbfdee700 1 -- 172.21.15.121:0/3629393525 shutdown_connections 2024-04-05T21:59:37.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.127+0000 7fecbfdee700 1 -- 172.21.15.121:0/3629393525 wait complete. 2024-04-05T21:59:37.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.229+0000 7febc03aa700 1 -- 172.21.15.121:0/2889163110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb80fef70 msgr2=0x7febb80ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:37.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.229+0000 7febc03aa700 1 --2- 172.21.15.121:0/2889163110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb80fef70 0x7febb80ff3f0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7febac009910 tx=0x7febac009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:37.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.229+0000 7febc03aa700 1 -- 172.21.15.121:0/2889163110 shutdown_connections 2024-04-05T21:59:37.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.229+0000 7febc03aa700 1 --2- 172.21.15.121:0/2889163110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb80fef70 0x7febb80ff3f0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.229+0000 7febc03aa700 1 --2- 172.21.15.121:0/2889163110 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febb8105b50 0x7febb8105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.229+0000 7febc03aa700 1 -- 172.21.15.121:0/2889163110 >> 172.21.15.121:0/2889163110 conn(0x7febb8075e30 msgr2=0x7febb80fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:37.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.229+0000 7febc03aa700 1 -- 172.21.15.121:0/2889163110 shutdown_connections 2024-04-05T21:59:37.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.229+0000 7febc03aa700 1 -- 172.21.15.121:0/2889163110 wait complete. 2024-04-05T21:59:37.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febc03aa700 1 Processor -- start 2024-04-05T21:59:37.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febc03aa700 1 -- start start 2024-04-05T21:59:37.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febc03aa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb80fef70 0x7febb819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:37.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febc03aa700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febb8105b50 0x7febb819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:37.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febc03aa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb819d0e0 con 0x7febb80fef70 2024-04-05T21:59:37.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febc03aa700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb8196590 con 0x7febb8105b50 2024-04-05T21:59:37.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febbe146700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb80fef70 0x7febb819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:37.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febbe146700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb80fef70 0x7febb819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60104/0 (socket says 172.21.15.121:60104) 2024-04-05T21:59:37.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febbe146700 1 -- 172.21.15.121:0/2874918524 learned_addr learned my addr 172.21.15.121:0/2874918524 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:37.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febbd945700 1 --2- 172.21.15.121:0/2874918524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febb8105b50 0x7febb819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:37.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febbe146700 1 -- 172.21.15.121:0/2874918524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febb8105b50 msgr2=0x7febb819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:37.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febbe146700 1 --2- 172.21.15.121:0/2874918524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febb8105b50 0x7febb819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febbe146700 1 -- 172.21.15.121:0/2874918524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac0095f0 con 0x7febb80fef70 2024-04-05T21:59:37.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.230+0000 7febbe146700 1 --2- 172.21.15.121:0/2874918524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb80fef70 0x7febb819c510 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7febb400b4b0 tx=0x7febb400b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:37.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.231+0000 7febab7fe700 1 -- 172.21.15.121:0/2874918524 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febb4011800 con 0x7febb80fef70 2024-04-05T21:59:37.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.231+0000 7febab7fe700 1 -- 172.21.15.121:0/2874918524 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7febb4011e00 con 0x7febb80fef70 2024-04-05T21:59:37.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.231+0000 7febc03aa700 1 -- 172.21.15.121:0/2874918524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb8196870 con 0x7febb80fef70 2024-04-05T21:59:37.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.231+0000 7febab7fe700 1 -- 172.21.15.121:0/2874918524 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febb4010e60 con 0x7febb80fef70 2024-04-05T21:59:37.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.231+0000 7febc03aa700 1 -- 172.21.15.121:0/2874918524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb8196dc0 con 0x7febb80fef70 2024-04-05T21:59:37.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.231+0000 7febc03aa700 1 -- 172.21.15.121:0/2874918524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb8109120 con 0x7febb80fef70 2024-04-05T21:59:37.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.232+0000 7febab7fe700 1 -- 172.21.15.121:0/2874918524 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7febb4011960 con 0x7febb80fef70 2024-04-05T21:59:37.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.232+0000 7febab7fe700 1 --2- 172.21.15.121:0/2874918524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feba406cce0 0x7feba406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:37.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.232+0000 7febab7fe700 1 -- 172.21.15.121:0/2874918524 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7febb408c0d0 con 0x7febb80fef70 2024-04-05T21:59:37.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.232+0000 7febbd945700 1 --2- 172.21.15.121:0/2874918524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feba406cce0 0x7feba406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:37.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.233+0000 7febbd945700 1 --2- 172.21.15.121:0/2874918524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feba406cce0 0x7feba406f1a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7febb8107250 tx=0x7febac01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:37.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.237+0000 7febab7fe700 1 -- 172.21.15.121:0/2874918524 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7febb4059bd0 con 0x7febb80fef70 2024-04-05T21:59:37.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:36 smithi121 ceph-mon[145859]: pgmap v1002: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:36 smithi188 ceph-mon[107793]: pgmap v1002: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:37.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.469+0000 7febc03aa700 1 -- 172.21.15.121:0/2874918524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7febb8197040 con 0x7febb80fef70 2024-04-05T21:59:37.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.470+0000 7febab7fe700 1 -- 172.21.15.121:0/2874918524 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7febb4059760 con 0x7febb80fef70 2024-04-05T21:59:37.471 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T21:59:37.471 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T21:59:37.471 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:59:37.471 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T21:59:37.471 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:59:37.471 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T21:59:37.471 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T21:59:37.472 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T21:59:37.472 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T21:59:37.472 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T21:59:37.472 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T21:59:37.472 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T21:59:37.472 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T21:59:37.472 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T21:59:37.472 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T21:59:37.472 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T21:59:37.472 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T21:59:37.472 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T21:59:37.472 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T21:59:37.473 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T21:59:37.473 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T21:59:37.473 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T21:59:37.473 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T21:59:37.473 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T21:59:37.473 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T21:59:37.474 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T21:59:37.474 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T21:59:37.474 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T21:59:37.474 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T21:59:37.474 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T21:59:37.474 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T21:59:37.474 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T21:59:37.475 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T21:59:37.475 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:59:37.475 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:59:37.475 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T21:59:37.475 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T21:59:37.475 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:59:37.475 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:59:37.475 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:59:37.475 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T21:59:37.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.472+0000 7febc03aa700 1 -- 172.21.15.121:0/2874918524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feba406cce0 msgr2=0x7feba406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:37.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.472+0000 7febc03aa700 1 --2- 172.21.15.121:0/2874918524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feba406cce0 0x7feba406f1a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7febb8107250 tx=0x7febac01a040 comp rx=0 tx=0).stop 2024-04-05T21:59:37.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.472+0000 7febc03aa700 1 -- 172.21.15.121:0/2874918524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb80fef70 msgr2=0x7febb819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:37.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.472+0000 7febc03aa700 1 --2- 172.21.15.121:0/2874918524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb80fef70 0x7febb819c510 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7febb400b4b0 tx=0x7febb400b7c0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.472+0000 7febc03aa700 1 -- 172.21.15.121:0/2874918524 shutdown_connections 2024-04-05T21:59:37.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.472+0000 7febc03aa700 1 --2- 172.21.15.121:0/2874918524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7feba406cce0 0x7feba406f1a0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.472+0000 7febc03aa700 1 --2- 172.21.15.121:0/2874918524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb80fef70 0x7febb819c510 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.472+0000 7febc03aa700 1 --2- 172.21.15.121:0/2874918524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7febb8105b50 0x7febb819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.472+0000 7febc03aa700 1 -- 172.21.15.121:0/2874918524 >> 172.21.15.121:0/2874918524 conn(0x7febb8075e30 msgr2=0x7febb8104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:37.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.472+0000 7febc03aa700 1 -- 172.21.15.121:0/2874918524 shutdown_connections 2024-04-05T21:59:37.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.472+0000 7febc03aa700 1 -- 172.21.15.121:0/2874918524 wait complete. 2024-04-05T21:59:37.477 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T21:59:37.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.575+0000 7fa209335700 1 -- 172.21.15.121:0/2820108540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204100da0 msgr2=0x7fa204101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:37.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.575+0000 7fa209335700 1 --2- 172.21.15.121:0/2820108540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204100da0 0x7fa204101180 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009910 tx=0x7fa1f4009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:37.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.575+0000 7fa209335700 1 -- 172.21.15.121:0/2820108540 shutdown_connections 2024-04-05T21:59:37.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.575+0000 7fa209335700 1 --2- 172.21.15.121:0/2820108540 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa204101750 0x7fa2041057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.575+0000 7fa209335700 1 --2- 172.21.15.121:0/2820108540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204100da0 0x7fa204101180 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.575+0000 7fa209335700 1 -- 172.21.15.121:0/2820108540 >> 172.21.15.121:0/2820108540 conn(0x7fa2040fc630 msgr2=0x7fa2040fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:37.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.575+0000 7fa209335700 1 -- 172.21.15.121:0/2820108540 shutdown_connections 2024-04-05T21:59:37.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.576+0000 7fa209335700 1 -- 172.21.15.121:0/2820108540 wait complete. 2024-04-05T21:59:37.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.576+0000 7fa209335700 1 Processor -- start 2024-04-05T21:59:37.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.576+0000 7fa209335700 1 -- start start 2024-04-05T21:59:37.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.576+0000 7fa209335700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa204100da0 0x7fa204196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:37.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.576+0000 7fa209335700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204101750 0x7fa204196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:37.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.576+0000 7fa209335700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa204197260 con 0x7fa204101750 2024-04-05T21:59:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.576+0000 7fa209335700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa204199f90 con 0x7fa204100da0 2024-04-05T21:59:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.577+0000 7fa2027fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204101750 0x7fa204196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.577+0000 7fa2027fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204101750 0x7fa204196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60132/0 (socket says 172.21.15.121:60132) 2024-04-05T21:59:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.577+0000 7fa2027fc700 1 -- 172.21.15.121:0/2853780844 learned_addr learned my addr 172.21.15.121:0/2853780844 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.577+0000 7fa202ffd700 1 --2- 172.21.15.121:0/2853780844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa204100da0 0x7fa204196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.577+0000 7fa2027fc700 1 -- 172.21.15.121:0/2853780844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa204100da0 msgr2=0x7fa204196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.577+0000 7fa2027fc700 1 --2- 172.21.15.121:0/2853780844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa204100da0 0x7fa204196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.577+0000 7fa2027fc700 1 -- 172.21.15.121:0/2853780844 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f40095f0 con 0x7fa204101750 2024-04-05T21:59:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.577+0000 7fa2027fc700 1 --2- 172.21.15.121:0/2853780844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204101750 0x7fa204196bd0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800e8f0 tx=0x7fa1f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.577+0000 7fa1ebfff700 1 -- 172.21.15.121:0/2853780844 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1f8009790 con 0x7fa204101750 2024-04-05T21:59:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.577+0000 7fa1ebfff700 1 -- 172.21.15.121:0/2853780844 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1f8004cd0 con 0x7fa204101750 2024-04-05T21:59:37.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.577+0000 7fa209335700 1 -- 172.21.15.121:0/2853780844 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa20419a270 con 0x7fa204101750 2024-04-05T21:59:37.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.577+0000 7fa1ebfff700 1 -- 172.21.15.121:0/2853780844 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1f8005580 con 0x7fa204101750 2024-04-05T21:59:37.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.577+0000 7fa209335700 1 -- 172.21.15.121:0/2853780844 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa20419a7c0 con 0x7fa204101750 2024-04-05T21:59:37.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.578+0000 7fa209335700 1 -- 172.21.15.121:0/2853780844 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa204109140 con 0x7fa204101750 2024-04-05T21:59:37.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.579+0000 7fa1ebfff700 1 -- 172.21.15.121:0/2853780844 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa1f8004750 con 0x7fa204101750 2024-04-05T21:59:37.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.579+0000 7fa1ebfff700 1 --2- 172.21.15.121:0/2853780844 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1ec06cce0 0x7fa1ec06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:37.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.579+0000 7fa1ebfff700 1 -- 172.21.15.121:0/2853780844 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa1f8014070 con 0x7fa204101750 2024-04-05T21:59:37.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.579+0000 7fa202ffd700 1 --2- 172.21.15.121:0/2853780844 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1ec06cce0 0x7fa1ec06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:37.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.580+0000 7fa202ffd700 1 --2- 172.21.15.121:0/2853780844 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1ec06cce0 0x7fa1ec06f1a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4000c00 tx=0x7fa1f4009f70 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:37.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.584+0000 7fa1ebfff700 1 -- 172.21.15.121:0/2853780844 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa1f80569c0 con 0x7fa204101750 2024-04-05T21:59:37.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.792+0000 7fa209335700 1 -- 172.21.15.121:0/2853780844 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa20419aab0 con 0x7fa1ec06cce0 2024-04-05T21:59:37.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.794+0000 7fa1ebfff700 1 -- 172.21.15.121:0/2853780844 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa20419aab0 con 0x7fa1ec06cce0 2024-04-05T21:59:37.794 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T21:59:37.794 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T21:59:37.794 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T21:59:37.794 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T21:59:37.794 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T21:59:37.794 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T21:59:37.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T21:59:37.795 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T21:59:37.795 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T21:59:37.795 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T21:59:37.795 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T21:59:37.795 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T21:59:37.795 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T21:59:37.795 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T21:59:37.795 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T21:59:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.796+0000 7fa209335700 1 -- 172.21.15.121:0/2853780844 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1ec06cce0 msgr2=0x7fa1ec06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.796+0000 7fa209335700 1 --2- 172.21.15.121:0/2853780844 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1ec06cce0 0x7fa1ec06f1a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4000c00 tx=0x7fa1f4009f70 comp rx=0 tx=0).stop 2024-04-05T21:59:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.796+0000 7fa209335700 1 -- 172.21.15.121:0/2853780844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204101750 msgr2=0x7fa204196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.796+0000 7fa209335700 1 --2- 172.21.15.121:0/2853780844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204101750 0x7fa204196bd0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800e8f0 tx=0x7fa1f800ecb0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.796+0000 7fa209335700 1 -- 172.21.15.121:0/2853780844 shutdown_connections 2024-04-05T21:59:37.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.796+0000 7fa209335700 1 --2- 172.21.15.121:0/2853780844 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa1ec06cce0 0x7fa1ec06f1a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.796+0000 7fa209335700 1 --2- 172.21.15.121:0/2853780844 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa204100da0 0x7fa204196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.796+0000 7fa209335700 1 --2- 172.21.15.121:0/2853780844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa204101750 0x7fa204196bd0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.796+0000 7fa209335700 1 -- 172.21.15.121:0/2853780844 >> 172.21.15.121:0/2853780844 conn(0x7fa2040fc630 msgr2=0x7fa2040fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:37.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.796+0000 7fa209335700 1 -- 172.21.15.121:0/2853780844 shutdown_connections 2024-04-05T21:59:37.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.796+0000 7fa209335700 1 -- 172.21.15.121:0/2853780844 wait complete. 2024-04-05T21:59:37.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.899+0000 7fab0e658700 1 -- 172.21.15.121:0/171684672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab08105a40 msgr2=0x7fab08105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:37.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.899+0000 7fab0e658700 1 --2- 172.21.15.121:0/171684672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab08105a40 0x7fab08105e20 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7faaf8009910 tx=0x7faaf8009c20 comp rx=0 tx=0).stop 2024-04-05T21:59:37.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.899+0000 7fab0e658700 1 -- 172.21.15.121:0/171684672 shutdown_connections 2024-04-05T21:59:37.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.899+0000 7fab0e658700 1 --2- 172.21.15.121:0/171684672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab080ffa40 0x7fab080ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.899+0000 7fab0e658700 1 --2- 172.21.15.121:0/171684672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab08105a40 0x7fab08105e20 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.899+0000 7fab0e658700 1 -- 172.21.15.121:0/171684672 >> 172.21.15.121:0/171684672 conn(0x7fab080fb570 msgr2=0x7fab080fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:37.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.899+0000 7fab0e658700 1 -- 172.21.15.121:0/171684672 shutdown_connections 2024-04-05T21:59:37.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.899+0000 7fab0e658700 1 -- 172.21.15.121:0/171684672 wait complete. 2024-04-05T21:59:37.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.900+0000 7fab0e658700 1 Processor -- start 2024-04-05T21:59:37.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.900+0000 7fab0e658700 1 -- start start 2024-04-05T21:59:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.900+0000 7fab0e658700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab080ffa40 0x7fab08196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.900+0000 7fab0e658700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab08105a40 0x7fab08196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.900+0000 7fab0e658700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab081971a0 con 0x7fab08105a40 2024-04-05T21:59:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.900+0000 7fab0e658700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab081972e0 con 0x7fab080ffa40 2024-04-05T21:59:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.900+0000 7fab077fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab08105a40 0x7fab08196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.900+0000 7fab077fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab08105a40 0x7fab08196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60160/0 (socket says 172.21.15.121:60160) 2024-04-05T21:59:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.900+0000 7fab077fe700 1 -- 172.21.15.121:0/2366657333 learned_addr learned my addr 172.21.15.121:0/2366657333 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T21:59:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.900+0000 7fab07fff700 1 --2- 172.21.15.121:0/2366657333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab080ffa40 0x7fab08196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.901+0000 7fab077fe700 1 -- 172.21.15.121:0/2366657333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab080ffa40 msgr2=0x7fab08196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.901+0000 7fab077fe700 1 --2- 172.21.15.121:0/2366657333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab080ffa40 0x7fab08196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.901+0000 7fab077fe700 1 -- 172.21.15.121:0/2366657333 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf80095f0 con 0x7fab08105a40 2024-04-05T21:59:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.901+0000 7fab077fe700 1 --2- 172.21.15.121:0/2366657333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab08105a40 0x7fab08196b50 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7faafc00c930 tx=0x7faafc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.901+0000 7fab057fa700 1 -- 172.21.15.121:0/2366657333 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faafc007ad0 con 0x7fab08105a40 2024-04-05T21:59:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.901+0000 7fab057fa700 1 -- 172.21.15.121:0/2366657333 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faafc00ce80 con 0x7fab08105a40 2024-04-05T21:59:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.901+0000 7fab0e658700 1 -- 172.21.15.121:0/2366657333 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0819b130 con 0x7fab08105a40 2024-04-05T21:59:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.901+0000 7fab057fa700 1 -- 172.21.15.121:0/2366657333 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faafc018630 con 0x7fab08105a40 2024-04-05T21:59:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.901+0000 7fab0e658700 1 -- 172.21.15.121:0/2366657333 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0819b680 con 0x7fab08105a40 2024-04-05T21:59:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.902+0000 7fab0e658700 1 -- 172.21.15.121:0/2366657333 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab0804ea90 con 0x7fab08105a40 2024-04-05T21:59:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.903+0000 7fab057fa700 1 -- 172.21.15.121:0/2366657333 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7faafc007c30 con 0x7fab08105a40 2024-04-05T21:59:37.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.903+0000 7fab057fa700 1 --2- 172.21.15.121:0/2366657333 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faaf006cce0 0x7faaf006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T21:59:37.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.903+0000 7fab057fa700 1 -- 172.21.15.121:0/2366657333 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7faafc08c4c0 con 0x7fab08105a40 2024-04-05T21:59:37.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.903+0000 7fab07fff700 1 --2- 172.21.15.121:0/2366657333 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faaf006cce0 0x7faaf006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T21:59:37.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.903+0000 7fab07fff700 1 --2- 172.21.15.121:0/2366657333 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faaf006cce0 0x7faaf006f1a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7faaf8006010 tx=0x7faaf801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T21:59:37.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:37.907+0000 7fab057fa700 1 -- 172.21.15.121:0/2366657333 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7faafc0565b0 con 0x7fab08105a40 2024-04-05T21:59:38.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:38.182+0000 7fab0e658700 1 -- 172.21.15.121:0/2366657333 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fab08197970 con 0x7fab08105a40 2024-04-05T21:59:38.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:38.182+0000 7fab057fa700 1 -- 172.21.15.121:0/2366657333 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7faafc059bd0 con 0x7fab08105a40 2024-04-05T21:59:38.183 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T21:59:38.183 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T21:59:38.183 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T21:59:38.183 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T21:59:38.184 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:59:38.184 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T21:59:38.184 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T21:59:38.184 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T21:59:38.184 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T21:59:38.184 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T21:59:38.184 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T21:59:38.184 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T21:59:38.184 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T21:59:38.184 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T21:59:38.184 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T21:59:38.184 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T21:59:38.184 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T21:59:38.185 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T21:59:38.185 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T21:59:38.185 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T21:59:38.185 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T21:59:38.185 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T21:59:38.185 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T21:59:38.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:38.185+0000 7fab0e658700 1 -- 172.21.15.121:0/2366657333 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faaf006cce0 msgr2=0x7faaf006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:38.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:38.185+0000 7fab0e658700 1 --2- 172.21.15.121:0/2366657333 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faaf006cce0 0x7faaf006f1a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7faaf8006010 tx=0x7faaf801a040 comp rx=0 tx=0).stop 2024-04-05T21:59:38.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:38.185+0000 7fab0e658700 1 -- 172.21.15.121:0/2366657333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab08105a40 msgr2=0x7fab08196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T21:59:38.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:38.185+0000 7fab0e658700 1 --2- 172.21.15.121:0/2366657333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab08105a40 0x7fab08196b50 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7faafc00c930 tx=0x7faafc00cc40 comp rx=0 tx=0).stop 2024-04-05T21:59:38.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:38.185+0000 7fab0e658700 1 -- 172.21.15.121:0/2366657333 shutdown_connections 2024-04-05T21:59:38.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:38.185+0000 7fab0e658700 1 --2- 172.21.15.121:0/2366657333 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7faaf006cce0 0x7faaf006f1a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:38.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:38.185+0000 7fab0e658700 1 --2- 172.21.15.121:0/2366657333 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fab080ffa40 0x7fab08196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:38.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:38.185+0000 7fab0e658700 1 --2- 172.21.15.121:0/2366657333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab08105a40 0x7fab08196b50 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T21:59:38.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:38.185+0000 7fab0e658700 1 -- 172.21.15.121:0/2366657333 >> 172.21.15.121:0/2366657333 conn(0x7fab080fb570 msgr2=0x7fab081037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T21:59:38.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:38.185+0000 7fab0e658700 1 -- 172.21.15.121:0/2366657333 shutdown_connections 2024-04-05T21:59:38.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T21:59:38.185+0000 7fab0e658700 1 -- 172.21.15.121:0/2366657333 wait complete. 2024-04-05T21:59:38.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:37 smithi121 ceph-mon[145859]: from='client.35634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:38.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:37 smithi121 ceph-mon[145859]: from='client.35638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:38.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:37 smithi121 ceph-mon[145859]: from='client.35642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:38.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:37 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3629393525' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:59:38.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:37 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2874918524' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:59:38.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:37 smithi188 ceph-mon[107793]: from='client.35634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:38.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:37 smithi188 ceph-mon[107793]: from='client.35638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:38.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:37 smithi188 ceph-mon[107793]: from='client.35642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:38.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:37 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3629393525' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T21:59:38.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:37 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2874918524' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T21:59:39.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:38 smithi121 ceph-mon[145859]: from='client.35654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:39.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:38 smithi121 ceph-mon[145859]: pgmap v1003: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:39.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:38 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2366657333' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:59:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:38 smithi188 ceph-mon[107793]: from='client.35654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T21:59:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:38 smithi188 ceph-mon[107793]: pgmap v1003: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:38 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2366657333' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T21:59:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:40 smithi121 ceph-mon[145859]: pgmap v1004: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:40 smithi188 ceph-mon[107793]: pgmap v1004: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:43.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:42 smithi121 ceph-mon[145859]: pgmap v1005: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:42 smithi188 ceph-mon[107793]: pgmap v1005: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:45.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:44 smithi121 ceph-mon[145859]: pgmap v1006: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:44 smithi188 ceph-mon[107793]: pgmap v1006: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:59:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T21:59:47.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:46 smithi121 ceph-mon[145859]: pgmap v1007: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:46 smithi188 ceph-mon[107793]: pgmap v1007: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:49.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:48 smithi121 ceph-mon[145859]: pgmap v1008: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:48 smithi188 ceph-mon[107793]: pgmap v1008: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:51.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:50 smithi121 ceph-mon[145859]: pgmap v1009: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:50 smithi188 ceph-mon[107793]: pgmap v1009: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:53.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:52 smithi121 ceph-mon[145859]: pgmap v1010: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:52 smithi188 ceph-mon[107793]: pgmap v1010: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:55.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:54 smithi121 ceph-mon[145859]: pgmap v1011: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:54 smithi188 ceph-mon[107793]: pgmap v1011: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:56 smithi121 ceph-mon[145859]: pgmap v1012: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:56 smithi188 ceph-mon[107793]: pgmap v1012: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:59.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 21:59:58 smithi121 ceph-mon[145859]: pgmap v1013: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T21:59:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 21:59:58 smithi188 ceph-mon[107793]: pgmap v1013: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:00:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:00:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:00:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : /usr/bin/podman: timeout after 895 seconds 2024-04-05T22:00:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:00:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : Traceback (most recent call last): 2024-04-05T22:00:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:00:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : "__main__", mod_spec) 2024-04-05T22:00:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:00:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : exec(code, run_globals) 2024-04-05T22:00:00.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:00:00.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:00:00.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:00:00.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:00:00.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:00:00.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:00:00.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:00:00.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:00:00.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:00:00.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:00:00.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-04-05T22:00:00.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:00:00.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T21:59:59.999+0000 7f91506ab700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-04-05T22:00:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: pgmap v1014: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:00:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:00:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:00:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: /usr/bin/podman: timeout after 895 seconds 2024-04-05T22:00:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:00:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: Traceback (most recent call last): 2024-04-05T22:00:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:00:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: "__main__", mod_spec) 2024-04-05T22:00:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:00:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: exec(code, run_globals) 2024-04-05T22:00:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:00:01.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:00:01.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:00:01.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:00:01.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:00:01.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:00:01.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:00:01.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:00:01.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:00:01.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:00:01.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: fs cephfs is degraded 2024-04-05T22:00:01.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:00:01.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: fs cephfs is offline because no MDS is active for it. 2024-04-05T22:00:01.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:00:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: pgmap v1014: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: Health detail: HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:00:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:00:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:00:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: /usr/bin/podman: timeout after 895 seconds 2024-04-05T22:00:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:00:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: Traceback (most recent call last): 2024-04-05T22:00:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:00:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: "__main__", mod_spec) 2024-04-05T22:00:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:00:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: exec(code, run_globals) 2024-04-05T22:00:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:00:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:00:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:00:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:00:01.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:00:01.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:00:01.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:00:01.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:00:01.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:00:01.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:00:01.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: fs cephfs is degraded 2024-04-05T22:00:01.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:00:01.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: fs cephfs is offline because no MDS is active for it. 2024-04-05T22:00:01.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:00:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:02 smithi121 ceph-mon[145859]: pgmap v1015: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:02 smithi188 ceph-mon[107793]: pgmap v1015: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:05.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:04 smithi121 ceph-mon[145859]: pgmap v1016: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:04 smithi188 ceph-mon[107793]: pgmap v1016: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:07.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:06 smithi121 ceph-mon[145859]: pgmap v1017: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:06 smithi188 ceph-mon[107793]: pgmap v1017: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:08.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.302+0000 7f3b5ee43700 1 -- 172.21.15.121:0/2982617625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b58105a40 msgr2=0x7f3b58105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:08.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.302+0000 7f3b5ee43700 1 --2- 172.21.15.121:0/2982617625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b58105a40 0x7f3b58105e20 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009910 tx=0x7f3b4c009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:08.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.302+0000 7f3b5ee43700 1 -- 172.21.15.121:0/2982617625 shutdown_connections 2024-04-05T22:00:08.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.302+0000 7f3b5ee43700 1 --2- 172.21.15.121:0/2982617625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b580ffbd0 0x7f3b58100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.302+0000 7f3b5ee43700 1 --2- 172.21.15.121:0/2982617625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b58105a40 0x7f3b58105e20 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.302+0000 7f3b5ee43700 1 -- 172.21.15.121:0/2982617625 >> 172.21.15.121:0/2982617625 conn(0x7f3b580755a0 msgr2=0x7f3b580759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.302+0000 7f3b5ee43700 1 -- 172.21.15.121:0/2982617625 shutdown_connections 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.302+0000 7f3b5ee43700 1 -- 172.21.15.121:0/2982617625 wait complete. 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.302+0000 7f3b5ee43700 1 Processor -- start 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.302+0000 7f3b5ee43700 1 -- start start 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.303+0000 7f3b5ee43700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b580ffbd0 0x7f3b58101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.303+0000 7f3b5ee43700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b58105a40 0x7f3b58101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.303+0000 7f3b5ee43700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b58102120 con 0x7f3b58105a40 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.303+0000 7f3b5ee43700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b58102290 con 0x7f3b580ffbd0 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.303+0000 7f3b57fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b58105a40 0x7f3b58101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.303+0000 7f3b57fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b58105a40 0x7f3b58101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32816/0 (socket says 172.21.15.121:32816) 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.303+0000 7f3b57fff700 1 -- 172.21.15.121:0/929913330 learned_addr learned my addr 172.21.15.121:0/929913330 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.303+0000 7f3b5cbdf700 1 --2- 172.21.15.121:0/929913330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b580ffbd0 0x7f3b58101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.303+0000 7f3b57fff700 1 -- 172.21.15.121:0/929913330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b580ffbd0 msgr2=0x7f3b58101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.303+0000 7f3b57fff700 1 --2- 172.21.15.121:0/929913330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b580ffbd0 0x7f3b58101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.303+0000 7f3b57fff700 1 -- 172.21.15.121:0/929913330 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c0095f0 con 0x7f3b58105a40 2024-04-05T22:00:08.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.303+0000 7f3b57fff700 1 --2- 172.21.15.121:0/929913330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b58105a40 0x7f3b58101b50 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400c8a0 tx=0x7f3b4400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:08.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.303+0000 7f3b55ffb700 1 -- 172.21.15.121:0/929913330 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b4400cea0 con 0x7f3b58105a40 2024-04-05T22:00:08.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.304+0000 7f3b55ffb700 1 -- 172.21.15.121:0/929913330 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3b44004830 con 0x7f3b58105a40 2024-04-05T22:00:08.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.304+0000 7f3b5ee43700 1 -- 172.21.15.121:0/929913330 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b581a2810 con 0x7f3b58105a40 2024-04-05T22:00:08.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.304+0000 7f3b55ffb700 1 -- 172.21.15.121:0/929913330 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b44005500 con 0x7f3b58105a40 2024-04-05T22:00:08.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.304+0000 7f3b5ee43700 1 -- 172.21.15.121:0/929913330 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b581a2c80 con 0x7f3b58105a40 2024-04-05T22:00:08.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.305+0000 7f3b5ee43700 1 -- 172.21.15.121:0/929913330 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b58109050 con 0x7f3b58105a40 2024-04-05T22:00:08.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.305+0000 7f3b55ffb700 1 -- 172.21.15.121:0/929913330 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3b440049a0 con 0x7f3b58105a40 2024-04-05T22:00:08.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.305+0000 7f3b55ffb700 1 --2- 172.21.15.121:0/929913330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3b4806cce0 0x7f3b4806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:08.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.306+0000 7f3b55ffb700 1 -- 172.21.15.121:0/929913330 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3b4408c2b0 con 0x7f3b58105a40 2024-04-05T22:00:08.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.306+0000 7f3b5cbdf700 1 --2- 172.21.15.121:0/929913330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3b4806cce0 0x7f3b4806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:08.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.306+0000 7f3b5cbdf700 1 --2- 172.21.15.121:0/929913330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3b4806cce0 0x7f3b4806f1a0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c006010 tx=0x7f3b4c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:08.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.310+0000 7f3b55ffb700 1 -- 172.21.15.121:0/929913330 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3b44056320 con 0x7f3b58105a40 2024-04-05T22:00:08.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.523+0000 7f3b5ee43700 1 -- 172.21.15.121:0/929913330 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b5802cce0 con 0x7f3b4806cce0 2024-04-05T22:00:08.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.524+0000 7f3b55ffb700 1 -- 172.21.15.121:0/929913330 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3b5802cce0 con 0x7f3b4806cce0 2024-04-05T22:00:08.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.526+0000 7f3b5ee43700 1 -- 172.21.15.121:0/929913330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3b4806cce0 msgr2=0x7f3b4806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:08.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.526+0000 7f3b5ee43700 1 --2- 172.21.15.121:0/929913330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3b4806cce0 0x7f3b4806f1a0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c006010 tx=0x7f3b4c01a040 comp rx=0 tx=0).stop 2024-04-05T22:00:08.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.526+0000 7f3b5ee43700 1 -- 172.21.15.121:0/929913330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b58105a40 msgr2=0x7f3b58101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:08.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.526+0000 7f3b5ee43700 1 --2- 172.21.15.121:0/929913330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b58105a40 0x7f3b58101b50 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400c8a0 tx=0x7f3b4400cc60 comp rx=0 tx=0).stop 2024-04-05T22:00:08.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.526+0000 7f3b5ee43700 1 -- 172.21.15.121:0/929913330 shutdown_connections 2024-04-05T22:00:08.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.526+0000 7f3b5ee43700 1 --2- 172.21.15.121:0/929913330 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3b4806cce0 0x7f3b4806f1a0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.526+0000 7f3b5ee43700 1 --2- 172.21.15.121:0/929913330 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3b580ffbd0 0x7f3b58101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.526+0000 7f3b5ee43700 1 --2- 172.21.15.121:0/929913330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3b58105a40 0x7f3b58101b50 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.526+0000 7f3b5ee43700 1 -- 172.21.15.121:0/929913330 >> 172.21.15.121:0/929913330 conn(0x7f3b580755a0 msgr2=0x7f3b58068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:08.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.526+0000 7f3b5ee43700 1 -- 172.21.15.121:0/929913330 shutdown_connections 2024-04-05T22:00:08.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.526+0000 7f3b5ee43700 1 -- 172.21.15.121:0/929913330 wait complete. 2024-04-05T22:00:08.537 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T22:00:08.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.630+0000 7fc26f454700 1 -- 172.21.15.121:0/3756083581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc268069020 msgr2=0x7fc268105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:08.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.630+0000 7fc26f454700 1 --2- 172.21.15.121:0/3756083581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc268069020 0x7fc268105790 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fc264009910 tx=0x7fc264009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:08.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.630+0000 7fc26f454700 1 -- 172.21.15.121:0/3756083581 shutdown_connections 2024-04-05T22:00:08.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.630+0000 7fc26f454700 1 --2- 172.21.15.121:0/3756083581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc268069020 0x7fc268105790 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.630+0000 7fc26f454700 1 --2- 172.21.15.121:0/3756083581 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc268068670 0x7fc268068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.630+0000 7fc26f454700 1 -- 172.21.15.121:0/3756083581 >> 172.21.15.121:0/3756083581 conn(0x7fc2680755a0 msgr2=0x7fc2680759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:08.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.630+0000 7fc26f454700 1 -- 172.21.15.121:0/3756083581 shutdown_connections 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.630+0000 7fc26f454700 1 -- 172.21.15.121:0/3756083581 wait complete. 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.631+0000 7fc26f454700 1 Processor -- start 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.631+0000 7fc26f454700 1 -- start start 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.631+0000 7fc26f454700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc268068670 0x7fc2680ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.631+0000 7fc26f454700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc268069020 0x7fc2681001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.631+0000 7fc26f454700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc268100720 con 0x7fc268068670 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.631+0000 7fc26f454700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc268100860 con 0x7fc268069020 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.631+0000 7fc26d1f0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc268068670 0x7fc2680ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.631+0000 7fc26d1f0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc268068670 0x7fc2680ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32822/0 (socket says 172.21.15.121:32822) 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.631+0000 7fc26d1f0700 1 -- 172.21.15.121:0/217584419 learned_addr learned my addr 172.21.15.121:0/217584419 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.631+0000 7fc26c9ef700 1 --2- 172.21.15.121:0/217584419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc268069020 0x7fc2681001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.631+0000 7fc26d1f0700 1 -- 172.21.15.121:0/217584419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc268069020 msgr2=0x7fc2681001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.631+0000 7fc26d1f0700 1 --2- 172.21.15.121:0/217584419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc268069020 0x7fc2681001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.631+0000 7fc26d1f0700 1 -- 172.21.15.121:0/217584419 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2640095f0 con 0x7fc268068670 2024-04-05T22:00:08.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.632+0000 7fc26d1f0700 1 --2- 172.21.15.121:0/217584419 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc268068670 0x7fc2680ffca0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc25800d900 tx=0x7fc25800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:08.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.632+0000 7fc25e7fc700 1 -- 172.21.15.121:0/217584419 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2580041f0 con 0x7fc268068670 2024-04-05T22:00:08.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.632+0000 7fc25e7fc700 1 -- 172.21.15.121:0/217584419 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc258004cd0 con 0x7fc268068670 2024-04-05T22:00:08.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.632+0000 7fc26f454700 1 -- 172.21.15.121:0/217584419 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2681a2810 con 0x7fc268068670 2024-04-05T22:00:08.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.632+0000 7fc25e7fc700 1 -- 172.21.15.121:0/217584419 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc258005660 con 0x7fc268068670 2024-04-05T22:00:08.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.632+0000 7fc26f454700 1 -- 172.21.15.121:0/217584419 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2681a2c20 con 0x7fc268068670 2024-04-05T22:00:08.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.633+0000 7fc26f454700 1 -- 172.21.15.121:0/217584419 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2681090a0 con 0x7fc268068670 2024-04-05T22:00:08.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.633+0000 7fc25e7fc700 1 -- 172.21.15.121:0/217584419 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc258004e40 con 0x7fc268068670 2024-04-05T22:00:08.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.634+0000 7fc25e7fc700 1 --2- 172.21.15.121:0/217584419 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc25406cce0 0x7fc25406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:08.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.634+0000 7fc25e7fc700 1 -- 172.21.15.121:0/217584419 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc25808c1c0 con 0x7fc268068670 2024-04-05T22:00:08.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.634+0000 7fc26c9ef700 1 --2- 172.21.15.121:0/217584419 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc25406cce0 0x7fc25406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:08.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.634+0000 7fc26c9ef700 1 --2- 172.21.15.121:0/217584419 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc25406cce0 0x7fc25406f1a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc268101490 tx=0x7fc26401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:08.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.638+0000 7fc25e7fc700 1 -- 172.21.15.121:0/217584419 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc258056230 con 0x7fc268068670 2024-04-05T22:00:08.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.850+0000 7fc26f454700 1 -- 172.21.15.121:0/217584419 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc26802cdd0 con 0x7fc25406cce0 2024-04-05T22:00:08.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.851+0000 7fc25e7fc700 1 -- 172.21.15.121:0/217584419 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc26802cdd0 con 0x7fc25406cce0 2024-04-05T22:00:08.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.853+0000 7fc26f454700 1 -- 172.21.15.121:0/217584419 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc25406cce0 msgr2=0x7fc25406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.853+0000 7fc26f454700 1 --2- 172.21.15.121:0/217584419 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc25406cce0 0x7fc25406f1a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc268101490 tx=0x7fc26401a040 comp rx=0 tx=0).stop 2024-04-05T22:00:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.853+0000 7fc26f454700 1 -- 172.21.15.121:0/217584419 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc268068670 msgr2=0x7fc2680ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.853+0000 7fc26f454700 1 --2- 172.21.15.121:0/217584419 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc268068670 0x7fc2680ffca0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc25800d900 tx=0x7fc25800dc10 comp rx=0 tx=0).stop 2024-04-05T22:00:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.853+0000 7fc26f454700 1 -- 172.21.15.121:0/217584419 shutdown_connections 2024-04-05T22:00:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.853+0000 7fc26f454700 1 --2- 172.21.15.121:0/217584419 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc25406cce0 0x7fc25406f1a0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.853+0000 7fc26f454700 1 --2- 172.21.15.121:0/217584419 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc268068670 0x7fc2680ffca0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.853+0000 7fc26f454700 1 --2- 172.21.15.121:0/217584419 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc268069020 0x7fc2681001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.853+0000 7fc26f454700 1 -- 172.21.15.121:0/217584419 >> 172.21.15.121:0/217584419 conn(0x7fc2680755a0 msgr2=0x7fc268105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.853+0000 7fc26f454700 1 -- 172.21.15.121:0/217584419 shutdown_connections 2024-04-05T22:00:08.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.854+0000 7fc26f454700 1 -- 172.21.15.121:0/217584419 wait complete. 2024-04-05T22:00:08.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.955+0000 7f181f6d7700 1 -- 172.21.15.121:0/2862771788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1818105a40 msgr2=0x7f1818105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:08.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.955+0000 7f181f6d7700 1 --2- 172.21.15.121:0/2862771788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1818105a40 0x7f1818105e20 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1814009910 tx=0x7f1814009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:08.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.955+0000 7f181f6d7700 1 -- 172.21.15.121:0/2862771788 shutdown_connections 2024-04-05T22:00:08.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181f6d7700 1 --2- 172.21.15.121:0/2862771788 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f18180ffa40 0x7f18180ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181f6d7700 1 --2- 172.21.15.121:0/2862771788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1818105a40 0x7f1818105e20 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181f6d7700 1 -- 172.21.15.121:0/2862771788 >> 172.21.15.121:0/2862771788 conn(0x7f18180fb570 msgr2=0x7f18180fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:08.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181f6d7700 1 -- 172.21.15.121:0/2862771788 shutdown_connections 2024-04-05T22:00:08.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181f6d7700 1 -- 172.21.15.121:0/2862771788 wait complete. 2024-04-05T22:00:08.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181f6d7700 1 Processor -- start 2024-04-05T22:00:08.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181f6d7700 1 -- start start 2024-04-05T22:00:08.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181f6d7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18180ffa40 0x7f1818195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:08.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181f6d7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1818105a40 0x7f1818195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:08.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181f6d7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1818196170 con 0x7f18180ffa40 2024-04-05T22:00:08.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181f6d7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1818199f00 con 0x7f1818105a40 2024-04-05T22:00:08.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181d473700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18180ffa40 0x7f1818195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:08.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181d473700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18180ffa40 0x7f1818195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32852/0 (socket says 172.21.15.121:32852) 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.956+0000 7f181d473700 1 -- 172.21.15.121:0/806412003 learned_addr learned my addr 172.21.15.121:0/806412003 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.957+0000 7f181d473700 1 -- 172.21.15.121:0/806412003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1818105a40 msgr2=0x7f1818195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.957+0000 7f181cc72700 1 --2- 172.21.15.121:0/806412003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1818105a40 0x7f1818195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.957+0000 7f181d473700 1 --2- 172.21.15.121:0/806412003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1818105a40 0x7f1818195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.957+0000 7f181d473700 1 -- 172.21.15.121:0/806412003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18140095f0 con 0x7f18180ffa40 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.957+0000 7f181cc72700 1 --2- 172.21.15.121:0/806412003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1818105a40 0x7f1818195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.957+0000 7f181d473700 1 --2- 172.21.15.121:0/806412003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18180ffa40 0x7f1818195550 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f1814006010 tx=0x7f181400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.957+0000 7f180e7fc700 1 -- 172.21.15.121:0/806412003 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f181401f070 con 0x7f18180ffa40 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.957+0000 7f180e7fc700 1 -- 172.21.15.121:0/806412003 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1814019610 con 0x7f18180ffa40 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.957+0000 7f181f6d7700 1 -- 172.21.15.121:0/806412003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181819a180 con 0x7f18180ffa40 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.957+0000 7f180e7fc700 1 -- 172.21.15.121:0/806412003 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f181400f670 con 0x7f18180ffa40 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.957+0000 7f181f6d7700 1 -- 172.21.15.121:0/806412003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181819a670 con 0x7f18180ffa40 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.958+0000 7f181f6d7700 1 -- 172.21.15.121:0/806412003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181804ea90 con 0x7f18180ffa40 2024-04-05T22:00:08.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.958+0000 7f180e7fc700 1 -- 172.21.15.121:0/806412003 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1814019c10 con 0x7f18180ffa40 2024-04-05T22:00:08.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.959+0000 7f180e7fc700 1 --2- 172.21.15.121:0/806412003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f180406cce0 0x7f180406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:08.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.959+0000 7f180e7fc700 1 -- 172.21.15.121:0/806412003 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f181408faf0 con 0x7f18180ffa40 2024-04-05T22:00:08.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.959+0000 7f181cc72700 1 --2- 172.21.15.121:0/806412003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f180406cce0 0x7f180406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:08.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.959+0000 7f181cc72700 1 --2- 172.21.15.121:0/806412003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f180406cce0 0x7f180406f1a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1808005fd0 tx=0x7f1808005dc0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:08.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:08.963+0000 7f180e7fc700 1 -- 172.21.15.121:0/806412003 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1814059b60 con 0x7f18180ffa40 2024-04-05T22:00:09.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.164+0000 7f181f6d7700 1 -- 172.21.15.121:0/806412003 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f181819ab70 con 0x7f180406cce0 2024-04-05T22:00:09.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.175+0000 7f180e7fc700 1 -- 172.21.15.121:0/806412003 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f181819ab70 con 0x7f180406cce0 2024-04-05T22:00:09.175 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T22:00:09.175 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (31m) 9m ago 42m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T22:00:09.175 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (42m) 9m ago 42m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T22:00:09.175 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (41m) 9m ago 41m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T22:00:09.175 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (29m) 9m ago 42m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T22:00:09.175 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (29m) 9m ago 41m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (30m) 9m ago 42m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (25m) 9m ago 36m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (25m) 9m ago 36m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (24m) 9m ago 36m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (9m) 9m ago 36m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (32m) 9m ago 44m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (32m) 9m ago 41m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (29m) 9m ago 44m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (29m) 9m ago 41m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (31m) 9m ago 42m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (31m) 9m ago 41m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (28m) 9m ago 40m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (28m) 9m ago 39m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T22:00:09.176 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (27m) 9m ago 39m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T22:00:09.177 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (26m) 9m ago 38m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T22:00:09.177 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (26m) 9m ago 38m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T22:00:09.177 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (25m) 9m ago 37m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T22:00:09.177 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (31m) 9m ago 41m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T22:00:09.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.177+0000 7f181f6d7700 1 -- 172.21.15.121:0/806412003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f180406cce0 msgr2=0x7f180406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:09.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.177+0000 7f181f6d7700 1 --2- 172.21.15.121:0/806412003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f180406cce0 0x7f180406f1a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1808005fd0 tx=0x7f1808005dc0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.177+0000 7f181f6d7700 1 -- 172.21.15.121:0/806412003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18180ffa40 msgr2=0x7f1818195550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.177+0000 7f181f6d7700 1 --2- 172.21.15.121:0/806412003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18180ffa40 0x7f1818195550 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f1814006010 tx=0x7f181400be20 comp rx=0 tx=0).stop 2024-04-05T22:00:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.177+0000 7f181f6d7700 1 -- 172.21.15.121:0/806412003 shutdown_connections 2024-04-05T22:00:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.177+0000 7f181f6d7700 1 --2- 172.21.15.121:0/806412003 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f180406cce0 0x7f180406f1a0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.177+0000 7f181f6d7700 1 --2- 172.21.15.121:0/806412003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18180ffa40 0x7f1818195550 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.177+0000 7f181f6d7700 1 --2- 172.21.15.121:0/806412003 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1818105a40 0x7f1818195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.177+0000 7f181f6d7700 1 -- 172.21.15.121:0/806412003 >> 172.21.15.121:0/806412003 conn(0x7f18180fb570 msgr2=0x7f18180fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.177+0000 7f181f6d7700 1 -- 172.21.15.121:0/806412003 shutdown_connections 2024-04-05T22:00:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.177+0000 7f181f6d7700 1 -- 172.21.15.121:0/806412003 wait complete. 2024-04-05T22:00:09.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:08 smithi121 ceph-mon[145859]: pgmap v1018: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:09.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.278+0000 7f75ff9cb700 1 -- 172.21.15.121:0/688220996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f81002a0 msgr2=0x7f75f810cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:09.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.278+0000 7f75ff9cb700 1 --2- 172.21.15.121:0/688220996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f81002a0 0x7f75f810cf60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009910 tx=0x7f75f4009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:09.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.278+0000 7f75ff9cb700 1 -- 172.21.15.121:0/688220996 shutdown_connections 2024-04-05T22:00:09.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.278+0000 7f75ff9cb700 1 --2- 172.21.15.121:0/688220996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f81002a0 0x7f75f810cf60 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.278+0000 7f75ff9cb700 1 --2- 172.21.15.121:0/688220996 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75f80ff8f0 0x7f75f80ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.278+0000 7f75ff9cb700 1 -- 172.21.15.121:0/688220996 >> 172.21.15.121:0/688220996 conn(0x7f75f80fb570 msgr2=0x7f75f80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:09.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.278+0000 7f75ff9cb700 1 -- 172.21.15.121:0/688220996 shutdown_connections 2024-04-05T22:00:09.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.278+0000 7f75ff9cb700 1 -- 172.21.15.121:0/688220996 wait complete. 2024-04-05T22:00:09.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:08 smithi188 ceph-mon[107793]: pgmap v1018: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:09.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.279+0000 7f75ff9cb700 1 Processor -- start 2024-04-05T22:00:09.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.279+0000 7f75ff9cb700 1 -- start start 2024-04-05T22:00:09.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.279+0000 7f75ff9cb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f80ff8f0 0x7f75f819e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:09.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.279+0000 7f75ff9cb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75f81002a0 0x7f75f819eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:09.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.279+0000 7f75ff9cb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f819f1b0 con 0x7f75f80ff8f0 2024-04-05T22:00:09.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.279+0000 7f75ff9cb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f8198660 con 0x7f75f81002a0 2024-04-05T22:00:09.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.279+0000 7f75fd767700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f80ff8f0 0x7f75f819e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.279+0000 7f75fd767700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f80ff8f0 0x7f75f819e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32880/0 (socket says 172.21.15.121:32880) 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.279+0000 7f75fd767700 1 -- 172.21.15.121:0/1863829950 learned_addr learned my addr 172.21.15.121:0/1863829950 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.280+0000 7f75fcf66700 1 --2- 172.21.15.121:0/1863829950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75f81002a0 0x7f75f819eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.280+0000 7f75fd767700 1 -- 172.21.15.121:0/1863829950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75f81002a0 msgr2=0x7f75f819eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.280+0000 7f75fd767700 1 --2- 172.21.15.121:0/1863829950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75f81002a0 0x7f75f819eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.280+0000 7f75fd767700 1 -- 172.21.15.121:0/1863829950 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f40095f0 con 0x7f75f80ff8f0 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.280+0000 7f75fcf66700 1 --2- 172.21.15.121:0/1863829950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75f81002a0 0x7f75f819eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.280+0000 7f75fd767700 1 --2- 172.21.15.121:0/1863829950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f80ff8f0 0x7f75f819e5e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f75e800e920 tx=0x7f75e800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.280+0000 7f75ee7fc700 1 -- 172.21.15.121:0/1863829950 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75e8009790 con 0x7f75f80ff8f0 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.280+0000 7f75ee7fc700 1 -- 172.21.15.121:0/1863829950 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75e800cba0 con 0x7f75f80ff8f0 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.280+0000 7f75ff9cb700 1 -- 172.21.15.121:0/1863829950 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f8198940 con 0x7f75f80ff8f0 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.280+0000 7f75ee7fc700 1 -- 172.21.15.121:0/1863829950 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75e8018650 con 0x7f75f80ff8f0 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.280+0000 7f75ff9cb700 1 -- 172.21.15.121:0/1863829950 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f8198e90 con 0x7f75f80ff8f0 2024-04-05T22:00:09.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.281+0000 7f75ff9cb700 1 -- 172.21.15.121:0/1863829950 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f810a6d0 con 0x7f75f80ff8f0 2024-04-05T22:00:09.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.281+0000 7f75ee7fc700 1 -- 172.21.15.121:0/1863829950 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f75e800cd10 con 0x7f75f80ff8f0 2024-04-05T22:00:09.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.282+0000 7f75ee7fc700 1 --2- 172.21.15.121:0/1863829950 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f75e406cce0 0x7f75e406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:09.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.282+0000 7f75fcf66700 1 --2- 172.21.15.121:0/1863829950 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f75e406cce0 0x7f75e406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:09.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.282+0000 7f75ee7fc700 1 -- 172.21.15.121:0/1863829950 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f75e8014070 con 0x7f75f80ff8f0 2024-04-05T22:00:09.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.282+0000 7f75fcf66700 1 --2- 172.21.15.121:0/1863829950 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f75e406cce0 0x7f75e406f1a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f75f80fbd00 tx=0x7f75f4009f70 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:09.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.286+0000 7f75ee7fc700 1 -- 172.21.15.121:0/1863829950 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f75e8052ab0 con 0x7f75f80ff8f0 2024-04-05T22:00:09.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.562+0000 7f75ff9cb700 1 -- 172.21.15.121:0/1863829950 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f75f81990b0 con 0x7f75f80ff8f0 2024-04-05T22:00:09.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.563+0000 7f75ee7fc700 1 -- 172.21.15.121:0/1863829950 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f75e805a1c0 con 0x7f75f80ff8f0 2024-04-05T22:00:09.563 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T22:00:09.564 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T22:00:09.565 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T22:00:09.565 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:00:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.565+0000 7f75ff9cb700 1 -- 172.21.15.121:0/1863829950 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f75e406cce0 msgr2=0x7f75e406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.565+0000 7f75ff9cb700 1 --2- 172.21.15.121:0/1863829950 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f75e406cce0 0x7f75e406f1a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f75f80fbd00 tx=0x7f75f4009f70 comp rx=0 tx=0).stop 2024-04-05T22:00:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.565+0000 7f75ff9cb700 1 -- 172.21.15.121:0/1863829950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f80ff8f0 msgr2=0x7f75f819e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.565+0000 7f75ff9cb700 1 --2- 172.21.15.121:0/1863829950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f80ff8f0 0x7f75f819e5e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f75e800e920 tx=0x7f75e800ece0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.565+0000 7f75ff9cb700 1 -- 172.21.15.121:0/1863829950 shutdown_connections 2024-04-05T22:00:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.565+0000 7f75ff9cb700 1 --2- 172.21.15.121:0/1863829950 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f75e406cce0 0x7f75e406f1a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.565+0000 7f75ff9cb700 1 --2- 172.21.15.121:0/1863829950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75f80ff8f0 0x7f75f819e5e0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.565+0000 7f75ff9cb700 1 --2- 172.21.15.121:0/1863829950 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75f81002a0 0x7f75f819eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.565+0000 7f75ff9cb700 1 -- 172.21.15.121:0/1863829950 >> 172.21.15.121:0/1863829950 conn(0x7f75f80fb570 msgr2=0x7f75f80fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.565+0000 7f75ff9cb700 1 -- 172.21.15.121:0/1863829950 shutdown_connections 2024-04-05T22:00:09.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.565+0000 7f75ff9cb700 1 -- 172.21.15.121:0/1863829950 wait complete. 2024-04-05T22:00:09.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.667+0000 7f1bb5546700 1 -- 172.21.15.121:0/1904895958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb0105a50 msgr2=0x7f1bb0105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:09.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.667+0000 7f1bb5546700 1 --2- 172.21.15.121:0/1904895958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb0105a50 0x7f1bb0105e30 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0009910 tx=0x7f1ba0009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:09.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.667+0000 7f1bb5546700 1 -- 172.21.15.121:0/1904895958 shutdown_connections 2024-04-05T22:00:09.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.667+0000 7f1bb5546700 1 --2- 172.21.15.121:0/1904895958 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bb00ffa50 0x7f1bb00ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.667+0000 7f1bb5546700 1 --2- 172.21.15.121:0/1904895958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb0105a50 0x7f1bb0105e30 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.667+0000 7f1bb5546700 1 -- 172.21.15.121:0/1904895958 >> 172.21.15.121:0/1904895958 conn(0x7f1bb00fb540 msgr2=0x7f1bb00fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:09.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.667+0000 7f1bb5546700 1 -- 172.21.15.121:0/1904895958 shutdown_connections 2024-04-05T22:00:09.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.667+0000 7f1bb5546700 1 -- 172.21.15.121:0/1904895958 wait complete. 2024-04-05T22:00:09.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.668+0000 7f1bb5546700 1 Processor -- start 2024-04-05T22:00:09.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.668+0000 7f1bb5546700 1 -- start start 2024-04-05T22:00:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.668+0000 7f1bb5546700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bb00ffa50 0x7f1bb0195590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.668+0000 7f1bb5546700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb0105a50 0x7f1bb0195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.668+0000 7f1bb5546700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb01961b0 con 0x7f1bb0105a50 2024-04-05T22:00:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.668+0000 7f1bb5546700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb0199f40 con 0x7f1bb00ffa50 2024-04-05T22:00:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.668+0000 7f1bae7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb0105a50 0x7f1bb0195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.669+0000 7f1bae7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb0105a50 0x7f1bb0195ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32896/0 (socket says 172.21.15.121:32896) 2024-04-05T22:00:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.669+0000 7f1bae7fc700 1 -- 172.21.15.121:0/3824556577 learned_addr learned my addr 172.21.15.121:0/3824556577 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.669+0000 7f1baeffd700 1 --2- 172.21.15.121:0/3824556577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bb00ffa50 0x7f1bb0195590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.669+0000 7f1bae7fc700 1 -- 172.21.15.121:0/3824556577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bb00ffa50 msgr2=0x7f1bb0195590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.669+0000 7f1bae7fc700 1 --2- 172.21.15.121:0/3824556577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bb00ffa50 0x7f1bb0195590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.669+0000 7f1bae7fc700 1 -- 172.21.15.121:0/3824556577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba00095f0 con 0x7f1bb0105a50 2024-04-05T22:00:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.669+0000 7f1baeffd700 1 --2- 172.21.15.121:0/3824556577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bb00ffa50 0x7f1bb0195590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:00:09.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.669+0000 7f1bae7fc700 1 --2- 172.21.15.121:0/3824556577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb0105a50 0x7f1bb0195ad0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400d8d0 tx=0x7f1ba400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:09.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.669+0000 7f1b97fff700 1 -- 172.21.15.121:0/3824556577 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ba4009940 con 0x7f1bb0105a50 2024-04-05T22:00:09.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.669+0000 7f1b97fff700 1 -- 172.21.15.121:0/3824556577 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ba4010460 con 0x7f1bb0105a50 2024-04-05T22:00:09.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.669+0000 7f1bb5546700 1 -- 172.21.15.121:0/3824556577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb019a220 con 0x7f1bb0105a50 2024-04-05T22:00:09.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.669+0000 7f1b97fff700 1 -- 172.21.15.121:0/3824556577 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ba400f5f0 con 0x7f1bb0105a50 2024-04-05T22:00:09.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.669+0000 7f1bb5546700 1 -- 172.21.15.121:0/3824556577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb019a770 con 0x7f1bb0105a50 2024-04-05T22:00:09.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.670+0000 7f1bb5546700 1 -- 172.21.15.121:0/3824556577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb0108050 con 0x7f1bb0105a50 2024-04-05T22:00:09.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.671+0000 7f1b97fff700 1 -- 172.21.15.121:0/3824556577 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1ba40105d0 con 0x7f1bb0105a50 2024-04-05T22:00:09.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.672+0000 7f1b97fff700 1 --2- 172.21.15.121:0/3824556577 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1b9806cce0 0x7f1b9806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:09.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.672+0000 7f1b97fff700 1 -- 172.21.15.121:0/3824556577 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1ba408c5f0 con 0x7f1bb0105a50 2024-04-05T22:00:09.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.672+0000 7f1baeffd700 1 --2- 172.21.15.121:0/3824556577 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1b9806cce0 0x7f1b9806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:09.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.672+0000 7f1baeffd700 1 --2- 172.21.15.121:0/3824556577 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1b9806cce0 0x7f1b9806f1a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000b5c0 tx=0x7f1ba0005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:09.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.676+0000 7f1b97fff700 1 -- 172.21.15.121:0/3824556577 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1ba40566f0 con 0x7f1bb0105a50 2024-04-05T22:00:09.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.911+0000 7f1bb5546700 1 -- 172.21.15.121:0/3824556577 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1bb004ea90 con 0x7f1bb0105a50 2024-04-05T22:00:09.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.911+0000 7f1b97fff700 1 -- 172.21.15.121:0/3824556577 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f1ba4016750 con 0x7f1bb0105a50 2024-04-05T22:00:09.911 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T22:00:09.911 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T22:00:09.912 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T22:00:09.913 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T22:00:09.914 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T22:00:09.914 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T22:00:09.914 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T22:00:09.914 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:00:09.914 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:00:09.914 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T22:00:09.914 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:00:09.914 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:00:09.914 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:00:09.914 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:00:09.914 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:00:09.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.913+0000 7f1bb5546700 1 -- 172.21.15.121:0/3824556577 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1b9806cce0 msgr2=0x7f1b9806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:09.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.913+0000 7f1bb5546700 1 --2- 172.21.15.121:0/3824556577 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1b9806cce0 0x7f1b9806f1a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000b5c0 tx=0x7f1ba0005960 comp rx=0 tx=0).stop 2024-04-05T22:00:09.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.913+0000 7f1bb5546700 1 -- 172.21.15.121:0/3824556577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb0105a50 msgr2=0x7f1bb0195ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:09.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.913+0000 7f1bb5546700 1 --2- 172.21.15.121:0/3824556577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb0105a50 0x7f1bb0195ad0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400d8d0 tx=0x7f1ba400dc90 comp rx=0 tx=0).stop 2024-04-05T22:00:09.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.913+0000 7f1bb5546700 1 -- 172.21.15.121:0/3824556577 shutdown_connections 2024-04-05T22:00:09.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.913+0000 7f1bb5546700 1 --2- 172.21.15.121:0/3824556577 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f1b9806cce0 0x7f1b9806f1a0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.913+0000 7f1bb5546700 1 --2- 172.21.15.121:0/3824556577 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1bb00ffa50 0x7f1bb0195590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.913+0000 7f1bb5546700 1 --2- 172.21.15.121:0/3824556577 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb0105a50 0x7f1bb0195ad0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:09.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.913+0000 7f1bb5546700 1 -- 172.21.15.121:0/3824556577 >> 172.21.15.121:0/3824556577 conn(0x7f1bb00fb540 msgr2=0x7f1bb00fc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:09.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.913+0000 7f1bb5546700 1 -- 172.21.15.121:0/3824556577 shutdown_connections 2024-04-05T22:00:09.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:09.913+0000 7f1bb5546700 1 -- 172.21.15.121:0/3824556577 wait complete. 2024-04-05T22:00:09.916 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T22:00:10.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.018+0000 7f607ebb6700 1 -- 172.21.15.121:0/427144125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6078105b50 msgr2=0x7f6078105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:10.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.018+0000 7f607ebb6700 1 --2- 172.21.15.121:0/427144125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6078105b50 0x7f6078105f30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f606c009910 tx=0x7f606c009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:10.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.018+0000 7f607ebb6700 1 -- 172.21.15.121:0/427144125 shutdown_connections 2024-04-05T22:00:10.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.018+0000 7f607ebb6700 1 --2- 172.21.15.121:0/427144125 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f60780fef70 0x7f60780ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:10.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.018+0000 7f607ebb6700 1 --2- 172.21.15.121:0/427144125 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6078105b50 0x7f6078105f30 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.018+0000 7f607ebb6700 1 -- 172.21.15.121:0/427144125 >> 172.21.15.121:0/427144125 conn(0x7f6078075e30 msgr2=0x7f60780fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.018+0000 7f607ebb6700 1 -- 172.21.15.121:0/427144125 shutdown_connections 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.018+0000 7f607ebb6700 1 -- 172.21.15.121:0/427144125 wait complete. 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.019+0000 7f607ebb6700 1 Processor -- start 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.019+0000 7f607ebb6700 1 -- start start 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.019+0000 7f607ebb6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60780fef70 0x7f6078100150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.019+0000 7f607ebb6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6078105b50 0x7f6078100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.019+0000 7f607ebb6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6078100bd0 con 0x7f60780fef70 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.019+0000 7f607ebb6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6078100d10 con 0x7f6078105b50 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.019+0000 7f607c952700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60780fef70 0x7f6078100150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.019+0000 7f607c952700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60780fef70 0x7f6078100150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32922/0 (socket says 172.21.15.121:32922) 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.019+0000 7f607c952700 1 -- 172.21.15.121:0/1131969134 learned_addr learned my addr 172.21.15.121:0/1131969134 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.019+0000 7f6077fff700 1 --2- 172.21.15.121:0/1131969134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6078105b50 0x7f6078100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:10.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.020+0000 7f607c952700 1 -- 172.21.15.121:0/1131969134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6078105b50 msgr2=0x7f6078100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:10.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.020+0000 7f607c952700 1 --2- 172.21.15.121:0/1131969134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6078105b50 0x7f6078100690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:10.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.020+0000 7f607c952700 1 -- 172.21.15.121:0/1131969134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606c0095f0 con 0x7f60780fef70 2024-04-05T22:00:10.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.020+0000 7f607c952700 1 --2- 172.21.15.121:0/1131969134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60780fef70 0x7f6078100150 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f606c006010 tx=0x7f606c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:10.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.020+0000 7f6075ffb700 1 -- 172.21.15.121:0/1131969134 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f606c01d070 con 0x7f60780fef70 2024-04-05T22:00:10.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.020+0000 7f6075ffb700 1 -- 172.21.15.121:0/1131969134 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f606c005710 con 0x7f60780fef70 2024-04-05T22:00:10.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.020+0000 7f607ebb6700 1 -- 172.21.15.121:0/1131969134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6078100f90 con 0x7f60780fef70 2024-04-05T22:00:10.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.020+0000 7f6075ffb700 1 -- 172.21.15.121:0/1131969134 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f606c00fe20 con 0x7f60780fef70 2024-04-05T22:00:10.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.020+0000 7f607ebb6700 1 -- 172.21.15.121:0/1131969134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60781a2c40 con 0x7f60780fef70 2024-04-05T22:00:10.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.021+0000 7f607ebb6700 1 -- 172.21.15.121:0/1131969134 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6078109120 con 0x7f60780fef70 2024-04-05T22:00:10.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.021+0000 7f6075ffb700 1 -- 172.21.15.121:0/1131969134 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f606c003a60 con 0x7f60780fef70 2024-04-05T22:00:10.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.022+0000 7f6075ffb700 1 --2- 172.21.15.121:0/1131969134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f606806cce0 0x7f606806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:10.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.022+0000 7f6075ffb700 1 -- 172.21.15.121:0/1131969134 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f606c08d6d0 con 0x7f60780fef70 2024-04-05T22:00:10.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.022+0000 7f6077fff700 1 --2- 172.21.15.121:0/1131969134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f606806cce0 0x7f606806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:10.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.022+0000 7f6077fff700 1 --2- 172.21.15.121:0/1131969134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f606806cce0 0x7f606806f1a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6078101940 tx=0x7f606400b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:10.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.026+0000 7f6075ffb700 1 -- 172.21.15.121:0/1131969134 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f606c057740 con 0x7f60780fef70 2024-04-05T22:00:10.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:09 smithi121 ceph-mon[145859]: from='client.35662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:10.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:09 smithi121 ceph-mon[145859]: from='client.35666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:10.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:09 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1863829950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:00:10.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:09 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3824556577' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:00:10.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.254+0000 7f607ebb6700 1 -- 172.21.15.121:0/1131969134 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f60780611d0 con 0x7f606806cce0 2024-04-05T22:00:10.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.255+0000 7f6075ffb700 1 -- 172.21.15.121:0/1131969134 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f60780611d0 con 0x7f606806cce0 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T22:00:10.256 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:00:10.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.257+0000 7f607ebb6700 1 -- 172.21.15.121:0/1131969134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f606806cce0 msgr2=0x7f606806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:10.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.257+0000 7f607ebb6700 1 --2- 172.21.15.121:0/1131969134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f606806cce0 0x7f606806f1a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6078101940 tx=0x7f606400b410 comp rx=0 tx=0).stop 2024-04-05T22:00:10.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.257+0000 7f607ebb6700 1 -- 172.21.15.121:0/1131969134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60780fef70 msgr2=0x7f6078100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:10.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.257+0000 7f607ebb6700 1 --2- 172.21.15.121:0/1131969134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60780fef70 0x7f6078100150 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f606c006010 tx=0x7f606c00bd70 comp rx=0 tx=0).stop 2024-04-05T22:00:10.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.257+0000 7f607ebb6700 1 -- 172.21.15.121:0/1131969134 shutdown_connections 2024-04-05T22:00:10.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.257+0000 7f607ebb6700 1 --2- 172.21.15.121:0/1131969134 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f606806cce0 0x7f606806f1a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:10.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.257+0000 7f607ebb6700 1 --2- 172.21.15.121:0/1131969134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60780fef70 0x7f6078100150 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:10.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.257+0000 7f607ebb6700 1 --2- 172.21.15.121:0/1131969134 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6078105b50 0x7f6078100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:10.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.257+0000 7f607ebb6700 1 -- 172.21.15.121:0/1131969134 >> 172.21.15.121:0/1131969134 conn(0x7f6078075e30 msgr2=0x7f6078104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:10.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.258+0000 7f607ebb6700 1 -- 172.21.15.121:0/1131969134 shutdown_connections 2024-04-05T22:00:10.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.258+0000 7f607ebb6700 1 -- 172.21.15.121:0/1131969134 wait complete. 2024-04-05T22:00:10.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:09 smithi188 ceph-mon[107793]: from='client.35662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:10.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:09 smithi188 ceph-mon[107793]: from='client.35666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:10.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:09 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1863829950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:00:10.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:09 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3824556577' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:00:10.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.360+0000 7f5f88a30700 1 -- 172.21.15.121:0/665187653 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f84100de0 msgr2=0x7f5f841011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:10.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.360+0000 7f5f88a30700 1 --2- 172.21.15.121:0/665187653 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f84100de0 0x7f5f841011c0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009910 tx=0x7f5f78009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:10.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.360+0000 7f5f88a30700 1 -- 172.21.15.121:0/665187653 shutdown_connections 2024-04-05T22:00:10.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.360+0000 7f5f88a30700 1 --2- 172.21.15.121:0/665187653 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f84101790 0x7f5f841057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:10.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.360+0000 7f5f88a30700 1 --2- 172.21.15.121:0/665187653 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f84100de0 0x7f5f841011c0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:10.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.360+0000 7f5f88a30700 1 -- 172.21.15.121:0/665187653 >> 172.21.15.121:0/665187653 conn(0x7f5f840fc670 msgr2=0x7f5f840fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:10.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.360+0000 7f5f88a30700 1 -- 172.21.15.121:0/665187653 shutdown_connections 2024-04-05T22:00:10.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.360+0000 7f5f88a30700 1 -- 172.21.15.121:0/665187653 wait complete. 2024-04-05T22:00:10.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.361+0000 7f5f88a30700 1 Processor -- start 2024-04-05T22:00:10.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.361+0000 7f5f88a30700 1 -- start start 2024-04-05T22:00:10.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.361+0000 7f5f88a30700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f84100de0 0x7f5f84196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:10.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.361+0000 7f5f88a30700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f84101790 0x7f5f84196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:10.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.361+0000 7f5f88a30700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f84197290 con 0x7f5f84101790 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.361+0000 7f5f88a30700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8419b020 con 0x7f5f84100de0 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.361+0000 7f5f81d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f84101790 0x7f5f84196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.361+0000 7f5f81d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f84101790 0x7f5f84196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32938/0 (socket says 172.21.15.121:32938) 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.361+0000 7f5f81d9b700 1 -- 172.21.15.121:0/2859569963 learned_addr learned my addr 172.21.15.121:0/2859569963 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.361+0000 7f5f8259c700 1 --2- 172.21.15.121:0/2859569963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f84100de0 0x7f5f84196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.361+0000 7f5f81d9b700 1 -- 172.21.15.121:0/2859569963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f84100de0 msgr2=0x7f5f84196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.361+0000 7f5f81d9b700 1 --2- 172.21.15.121:0/2859569963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f84100de0 0x7f5f84196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.361+0000 7f5f81d9b700 1 -- 172.21.15.121:0/2859569963 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f780095f0 con 0x7f5f84101790 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.362+0000 7f5f81d9b700 1 --2- 172.21.15.121:0/2859569963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f84101790 0x7f5f84196bb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5f7400e8f0 tx=0x7f5f7400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.362+0000 7f5f737fe700 1 -- 172.21.15.121:0/2859569963 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f74009790 con 0x7f5f84101790 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.362+0000 7f5f737fe700 1 -- 172.21.15.121:0/2859569963 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f7400c810 con 0x7f5f84101790 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.362+0000 7f5f88a30700 1 -- 172.21.15.121:0/2859569963 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8419b300 con 0x7f5f84101790 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.362+0000 7f5f737fe700 1 -- 172.21.15.121:0/2859569963 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f74005520 con 0x7f5f84101790 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.362+0000 7f5f88a30700 1 -- 172.21.15.121:0/2859569963 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8419b850 con 0x7f5f84101790 2024-04-05T22:00:10.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.363+0000 7f5f88a30700 1 -- 172.21.15.121:0/2859569963 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f84109180 con 0x7f5f84101790 2024-04-05T22:00:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.363+0000 7f5f737fe700 1 -- 172.21.15.121:0/2859569963 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5f7400c980 con 0x7f5f84101790 2024-04-05T22:00:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.364+0000 7f5f737fe700 1 --2- 172.21.15.121:0/2859569963 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5f6c06cce0 0x7f5f6c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.364+0000 7f5f8259c700 1 --2- 172.21.15.121:0/2859569963 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5f6c06cce0 0x7f5f6c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.364+0000 7f5f737fe700 1 -- 172.21.15.121:0/2859569963 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5f74014070 con 0x7f5f84101790 2024-04-05T22:00:10.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.364+0000 7f5f8259c700 1 --2- 172.21.15.121:0/2859569963 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5f6c06cce0 0x7f5f6c06f1a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009fd0 tx=0x7f5f78005940 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:10.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.368+0000 7f5f737fe700 1 -- 172.21.15.121:0/2859569963 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5f74056920 con 0x7f5f84101790 2024-04-05T22:00:10.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.643+0000 7f5f88a30700 1 -- 172.21.15.121:0/2859569963 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5f8404ea90 con 0x7f5f84101790 2024-04-05T22:00:10.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.644+0000 7f5f737fe700 1 -- 172.21.15.121:0/2859569963 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f5f7400cc30 con 0x7f5f84101790 2024-04-05T22:00:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:00:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:00:10.644 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:00:10.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:00:10.645 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:00:10.646 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:00:10.646 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T22:00:10.646 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:00:10.646 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T22:00:10.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.646+0000 7f5f88a30700 1 -- 172.21.15.121:0/2859569963 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5f6c06cce0 msgr2=0x7f5f6c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:10.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.646+0000 7f5f88a30700 1 --2- 172.21.15.121:0/2859569963 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5f6c06cce0 0x7f5f6c06f1a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009fd0 tx=0x7f5f78005940 comp rx=0 tx=0).stop 2024-04-05T22:00:10.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.646+0000 7f5f88a30700 1 -- 172.21.15.121:0/2859569963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f84101790 msgr2=0x7f5f84196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:10.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.646+0000 7f5f88a30700 1 --2- 172.21.15.121:0/2859569963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f84101790 0x7f5f84196bb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f5f7400e8f0 tx=0x7f5f7400ec00 comp rx=0 tx=0).stop 2024-04-05T22:00:10.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.646+0000 7f5f88a30700 1 -- 172.21.15.121:0/2859569963 shutdown_connections 2024-04-05T22:00:10.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.646+0000 7f5f88a30700 1 --2- 172.21.15.121:0/2859569963 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5f6c06cce0 0x7f5f6c06f1a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:10.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.646+0000 7f5f88a30700 1 --2- 172.21.15.121:0/2859569963 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5f84100de0 0x7f5f84196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:10.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.646+0000 7f5f88a30700 1 --2- 172.21.15.121:0/2859569963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f84101790 0x7f5f84196bb0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:10.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.646+0000 7f5f88a30700 1 -- 172.21.15.121:0/2859569963 >> 172.21.15.121:0/2859569963 conn(0x7f5f840fc670 msgr2=0x7f5f840fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:10.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.646+0000 7f5f88a30700 1 -- 172.21.15.121:0/2859569963 shutdown_connections 2024-04-05T22:00:10.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:10.646+0000 7f5f88a30700 1 -- 172.21.15.121:0/2859569963 wait complete. 2024-04-05T22:00:11.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:10 smithi121 ceph-mon[145859]: from='client.35670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:11.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:10 smithi121 ceph-mon[145859]: pgmap v1019: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:11.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:10 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2859569963' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:00:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:10 smithi188 ceph-mon[107793]: from='client.35670 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:10 smithi188 ceph-mon[107793]: pgmap v1019: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:10 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2859569963' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:00:12.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:11 smithi121 ceph-mon[145859]: from='client.35682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:12.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:11 smithi188 ceph-mon[107793]: from='client.35682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:13.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:12 smithi121 ceph-mon[145859]: pgmap v1020: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:12 smithi188 ceph-mon[107793]: pgmap v1020: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:15.221 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:14 smithi121 ceph-mon[145859]: pgmap v1021: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:14 smithi188 ceph-mon[107793]: pgmap v1021: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:00:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:00:17.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:16 smithi121 ceph-mon[145859]: pgmap v1022: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:17.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:16 smithi188 ceph-mon[107793]: pgmap v1022: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:19.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:18 smithi121 ceph-mon[145859]: pgmap v1023: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:19.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:18 smithi188 ceph-mon[107793]: pgmap v1023: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:21.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:20 smithi121 ceph-mon[145859]: pgmap v1024: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:20 smithi188 ceph-mon[107793]: pgmap v1024: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:23.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:22 smithi121 ceph-mon[145859]: pgmap v1025: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:23.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:22 smithi188 ceph-mon[107793]: pgmap v1025: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:25.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:24 smithi121 ceph-mon[145859]: pgmap v1026: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:24 smithi188 ceph-mon[107793]: pgmap v1026: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:27.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:26 smithi121 ceph-mon[145859]: pgmap v1027: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:27.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:26 smithi188 ceph-mon[107793]: pgmap v1027: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:29.030 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:28 smithi188 ceph-mon[107793]: pgmap v1028: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:29.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:28 smithi121 ceph-mon[145859]: pgmap v1028: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:30 smithi188 ceph-mon[107793]: pgmap v1029: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:00:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:30 smithi121 ceph-mon[145859]: pgmap v1029: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:00:33.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:32 smithi188 ceph-mon[107793]: pgmap v1030: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:33.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:32 smithi121 ceph-mon[145859]: pgmap v1030: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:35.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:34 smithi188 ceph-mon[107793]: pgmap v1031: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:35.409 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:34 smithi121 ceph-mon[145859]: pgmap v1031: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:37 smithi188 ceph-mon[107793]: pgmap v1032: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:37.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:37 smithi121 ceph-mon[145859]: pgmap v1032: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:39 smithi188 ceph-mon[107793]: pgmap v1033: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:39.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:39 smithi121 ceph-mon[145859]: pgmap v1033: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:40.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.762+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/1265070795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb680ffbd0 msgr2=0x7fbb68100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:40.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.762+0000 7fbb6e4f3700 1 --2- 172.21.15.121:0/1265070795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb680ffbd0 0x7fbb68100050 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009910 tx=0x7fbb50009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:40.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.762+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/1265070795 shutdown_connections 2024-04-05T22:00:40.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.762+0000 7fbb6e4f3700 1 --2- 172.21.15.121:0/1265070795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb680ffbd0 0x7fbb68100050 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:40.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.762+0000 7fbb6e4f3700 1 --2- 172.21.15.121:0/1265070795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb68105a40 0x7fbb68105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:40.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.762+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/1265070795 >> 172.21.15.121:0/1265070795 conn(0x7fbb680755a0 msgr2=0x7fbb680759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:40.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.762+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/1265070795 shutdown_connections 2024-04-05T22:00:40.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.762+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/1265070795 wait complete. 2024-04-05T22:00:40.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.763+0000 7fbb6e4f3700 1 Processor -- start 2024-04-05T22:00:40.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.763+0000 7fbb6e4f3700 1 -- start start 2024-04-05T22:00:40.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.763+0000 7fbb6e4f3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb680ffbd0 0x7fbb6819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:40.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.763+0000 7fbb6e4f3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb68105a40 0x7fbb6819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:40.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.763+0000 7fbb6e4f3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6819d0e0 con 0x7fbb680ffbd0 2024-04-05T22:00:40.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.763+0000 7fbb6e4f3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb68196590 con 0x7fbb68105a40 2024-04-05T22:00:40.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.763+0000 7fbb67fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb680ffbd0 0x7fbb6819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:40.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.763+0000 7fbb67fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb680ffbd0 0x7fbb6819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45570/0 (socket says 172.21.15.121:45570) 2024-04-05T22:00:40.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.763+0000 7fbb67fff700 1 -- 172.21.15.121:0/4031847346 learned_addr learned my addr 172.21.15.121:0/4031847346 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:40.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.763+0000 7fbb677fe700 1 --2- 172.21.15.121:0/4031847346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb68105a40 0x7fbb6819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:40.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.764+0000 7fbb67fff700 1 -- 172.21.15.121:0/4031847346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb68105a40 msgr2=0x7fbb6819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:40.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.764+0000 7fbb67fff700 1 --2- 172.21.15.121:0/4031847346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb68105a40 0x7fbb6819ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:40.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.764+0000 7fbb67fff700 1 -- 172.21.15.121:0/4031847346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb500095f0 con 0x7fbb680ffbd0 2024-04-05T22:00:40.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.764+0000 7fbb677fe700 1 --2- 172.21.15.121:0/4031847346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb68105a40 0x7fbb6819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:00:40.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.764+0000 7fbb67fff700 1 --2- 172.21.15.121:0/4031847346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb680ffbd0 0x7fbb6819c510 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800d900 tx=0x7fbb5800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:40.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.764+0000 7fbb657fa700 1 -- 172.21.15.121:0/4031847346 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb580041f0 con 0x7fbb680ffbd0 2024-04-05T22:00:40.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.764+0000 7fbb657fa700 1 -- 172.21.15.121:0/4031847346 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb58004cd0 con 0x7fbb680ffbd0 2024-04-05T22:00:40.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.764+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/4031847346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb68196870 con 0x7fbb680ffbd0 2024-04-05T22:00:40.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.764+0000 7fbb657fa700 1 -- 172.21.15.121:0/4031847346 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb580056d0 con 0x7fbb680ffbd0 2024-04-05T22:00:40.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.764+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/4031847346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb68196dc0 con 0x7fbb680ffbd0 2024-04-05T22:00:40.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.765+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/4031847346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb68108fe0 con 0x7fbb680ffbd0 2024-04-05T22:00:40.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.765+0000 7fbb657fa700 1 -- 172.21.15.121:0/4031847346 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbb58004e40 con 0x7fbb680ffbd0 2024-04-05T22:00:40.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.766+0000 7fbb657fa700 1 --2- 172.21.15.121:0/4031847346 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbb5406cc90 0x7fbb5406f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:40.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.766+0000 7fbb657fa700 1 -- 172.21.15.121:0/4031847346 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbb5808c710 con 0x7fbb680ffbd0 2024-04-05T22:00:40.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.766+0000 7fbb677fe700 1 --2- 172.21.15.121:0/4031847346 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbb5406cc90 0x7fbb5406f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:40.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.766+0000 7fbb677fe700 1 --2- 172.21.15.121:0/4031847346 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbb5406cc90 0x7fbb5406f150 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000b5c0 tx=0x7fbb50005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:40.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.770+0000 7fbb657fa700 1 -- 172.21.15.121:0/4031847346 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbb5805a210 con 0x7fbb680ffbd0 2024-04-05T22:00:40.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.982+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/4031847346 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb68196fe0 con 0x7fbb5406cc90 2024-04-05T22:00:40.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.983+0000 7fbb657fa700 1 -- 172.21.15.121:0/4031847346 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbb68196fe0 con 0x7fbb5406cc90 2024-04-05T22:00:40.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.985+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/4031847346 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbb5406cc90 msgr2=0x7fbb5406f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:40.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.985+0000 7fbb6e4f3700 1 --2- 172.21.15.121:0/4031847346 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbb5406cc90 0x7fbb5406f150 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000b5c0 tx=0x7fbb50005960 comp rx=0 tx=0).stop 2024-04-05T22:00:40.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.985+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/4031847346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb680ffbd0 msgr2=0x7fbb6819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:40.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.985+0000 7fbb6e4f3700 1 --2- 172.21.15.121:0/4031847346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb680ffbd0 0x7fbb6819c510 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800d900 tx=0x7fbb5800dcc0 comp rx=0 tx=0).stop 2024-04-05T22:00:40.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.985+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/4031847346 shutdown_connections 2024-04-05T22:00:40.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.985+0000 7fbb6e4f3700 1 --2- 172.21.15.121:0/4031847346 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbb5406cc90 0x7fbb5406f150 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:40.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.985+0000 7fbb6e4f3700 1 --2- 172.21.15.121:0/4031847346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbb680ffbd0 0x7fbb6819c510 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:40.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.985+0000 7fbb6e4f3700 1 --2- 172.21.15.121:0/4031847346 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbb68105a40 0x7fbb6819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:40.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.985+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/4031847346 >> 172.21.15.121:0/4031847346 conn(0x7fbb680755a0 msgr2=0x7fbb68103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:40.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.985+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/4031847346 shutdown_connections 2024-04-05T22:00:40.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:40.985+0000 7fbb6e4f3700 1 -- 172.21.15.121:0/4031847346 wait complete. 2024-04-05T22:00:40.996 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T22:00:41.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.088+0000 7fdc74143700 1 -- 172.21.15.121:0/469312136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc6c069020 msgr2=0x7fdc6c105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:41.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.088+0000 7fdc74143700 1 --2- 172.21.15.121:0/469312136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc6c069020 0x7fdc6c105790 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009910 tx=0x7fdc5c009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:41.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.089+0000 7fdc74143700 1 -- 172.21.15.121:0/469312136 shutdown_connections 2024-04-05T22:00:41.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.089+0000 7fdc74143700 1 --2- 172.21.15.121:0/469312136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc6c069020 0x7fdc6c105790 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.089+0000 7fdc74143700 1 --2- 172.21.15.121:0/469312136 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc6c068670 0x7fdc6c068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.089+0000 7fdc74143700 1 -- 172.21.15.121:0/469312136 >> 172.21.15.121:0/469312136 conn(0x7fdc6c0755a0 msgr2=0x7fdc6c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:41.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.089+0000 7fdc74143700 1 -- 172.21.15.121:0/469312136 shutdown_connections 2024-04-05T22:00:41.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.089+0000 7fdc74143700 1 -- 172.21.15.121:0/469312136 wait complete. 2024-04-05T22:00:41.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.089+0000 7fdc74143700 1 Processor -- start 2024-04-05T22:00:41.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.089+0000 7fdc74143700 1 -- start start 2024-04-05T22:00:41.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.089+0000 7fdc74143700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc6c068670 0x7fdc6c196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:41.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.089+0000 7fdc74143700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc6c069020 0x7fdc6c196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:41.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.089+0000 7fdc74143700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c197260 con 0x7fdc6c069020 2024-04-05T22:00:41.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.089+0000 7fdc74143700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c19aff0 con 0x7fdc6c068670 2024-04-05T22:00:41.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.090+0000 7fdc716de700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc6c069020 0x7fdc6c196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:41.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.090+0000 7fdc716de700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc6c069020 0x7fdc6c196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45600/0 (socket says 172.21.15.121:45600) 2024-04-05T22:00:41.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.090+0000 7fdc716de700 1 -- 172.21.15.121:0/330379069 learned_addr learned my addr 172.21.15.121:0/330379069 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:41.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.090+0000 7fdc716de700 1 -- 172.21.15.121:0/330379069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc6c068670 msgr2=0x7fdc6c196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:41.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.090+0000 7fdc71edf700 1 --2- 172.21.15.121:0/330379069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc6c068670 0x7fdc6c196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:41.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.090+0000 7fdc716de700 1 --2- 172.21.15.121:0/330379069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc6c068670 0x7fdc6c196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.090+0000 7fdc716de700 1 -- 172.21.15.121:0/330379069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c0095f0 con 0x7fdc6c069020 2024-04-05T22:00:41.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.090+0000 7fdc716de700 1 --2- 172.21.15.121:0/330379069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc6c069020 0x7fdc6c196b80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c0092e0 tx=0x7fdc5c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:41.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.090+0000 7fdc62ffd700 1 -- 172.21.15.121:0/330379069 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc5c01d070 con 0x7fdc6c069020 2024-04-05T22:00:41.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.090+0000 7fdc62ffd700 1 -- 172.21.15.121:0/330379069 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc5c00f460 con 0x7fdc6c069020 2024-04-05T22:00:41.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.090+0000 7fdc74143700 1 -- 172.21.15.121:0/330379069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6c19b270 con 0x7fdc6c069020 2024-04-05T22:00:41.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.090+0000 7fdc62ffd700 1 -- 172.21.15.121:0/330379069 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc5c021600 con 0x7fdc6c069020 2024-04-05T22:00:41.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.090+0000 7fdc74143700 1 -- 172.21.15.121:0/330379069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c19b760 con 0x7fdc6c069020 2024-04-05T22:00:41.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.091+0000 7fdc74143700 1 -- 172.21.15.121:0/330379069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6c1090a0 con 0x7fdc6c069020 2024-04-05T22:00:41.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.092+0000 7fdc62ffd700 1 -- 172.21.15.121:0/330379069 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdc5c00f5d0 con 0x7fdc6c069020 2024-04-05T22:00:41.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.092+0000 7fdc62ffd700 1 --2- 172.21.15.121:0/330379069 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdc5806cce0 0x7fdc5806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:41.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.092+0000 7fdc62ffd700 1 -- 172.21.15.121:0/330379069 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdc5c08d840 con 0x7fdc6c069020 2024-04-05T22:00:41.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.092+0000 7fdc71edf700 1 --2- 172.21.15.121:0/330379069 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdc5806cce0 0x7fdc5806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:41.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.092+0000 7fdc71edf700 1 --2- 172.21.15.121:0/330379069 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdc5806cce0 0x7fdc5806f1a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fdc68005950 tx=0x7fdc68005cf0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:41.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.096+0000 7fdc62ffd700 1 -- 172.21.15.121:0/330379069 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdc5c0578b0 con 0x7fdc6c069020 2024-04-05T22:00:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:41 smithi188 ceph-mon[107793]: pgmap v1034: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:41.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.307+0000 7fdc74143700 1 -- 172.21.15.121:0/330379069 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc6c02cd60 con 0x7fdc5806cce0 2024-04-05T22:00:41.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.308+0000 7fdc62ffd700 1 -- 172.21.15.121:0/330379069 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fdc6c02cd60 con 0x7fdc5806cce0 2024-04-05T22:00:41.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.311+0000 7fdc74143700 1 -- 172.21.15.121:0/330379069 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdc5806cce0 msgr2=0x7fdc5806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:41.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.311+0000 7fdc74143700 1 --2- 172.21.15.121:0/330379069 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdc5806cce0 0x7fdc5806f1a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fdc68005950 tx=0x7fdc68005cf0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.311+0000 7fdc74143700 1 -- 172.21.15.121:0/330379069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc6c069020 msgr2=0x7fdc6c196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:41.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.311+0000 7fdc74143700 1 --2- 172.21.15.121:0/330379069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc6c069020 0x7fdc6c196b80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c0092e0 tx=0x7fdc5c00be20 comp rx=0 tx=0).stop 2024-04-05T22:00:41.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.311+0000 7fdc74143700 1 -- 172.21.15.121:0/330379069 shutdown_connections 2024-04-05T22:00:41.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.311+0000 7fdc74143700 1 --2- 172.21.15.121:0/330379069 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdc5806cce0 0x7fdc5806f1a0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.311+0000 7fdc74143700 1 --2- 172.21.15.121:0/330379069 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdc6c068670 0x7fdc6c196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.311+0000 7fdc74143700 1 --2- 172.21.15.121:0/330379069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc6c069020 0x7fdc6c196b80 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.311+0000 7fdc74143700 1 -- 172.21.15.121:0/330379069 >> 172.21.15.121:0/330379069 conn(0x7fdc6c0755a0 msgr2=0x7fdc6c105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:41.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.311+0000 7fdc74143700 1 -- 172.21.15.121:0/330379069 shutdown_connections 2024-04-05T22:00:41.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.311+0000 7fdc74143700 1 -- 172.21.15.121:0/330379069 wait complete. 2024-04-05T22:00:41.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.412+0000 7f6679da9700 1 -- 172.21.15.121:0/2023456746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6674100de0 msgr2=0x7f66741011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:41.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.412+0000 7f6679da9700 1 --2- 172.21.15.121:0/2023456746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6674100de0 0x7f66741011c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f6668009910 tx=0x7f6668009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:41.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.412+0000 7f6679da9700 1 -- 172.21.15.121:0/2023456746 shutdown_connections 2024-04-05T22:00:41.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.412+0000 7f6679da9700 1 --2- 172.21.15.121:0/2023456746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6674101790 0x7f66741057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.412+0000 7f6679da9700 1 --2- 172.21.15.121:0/2023456746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6674100de0 0x7f66741011c0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.412+0000 7f6679da9700 1 -- 172.21.15.121:0/2023456746 >> 172.21.15.121:0/2023456746 conn(0x7f66740fc6b0 msgr2=0x7f66740fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:41.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.413+0000 7f6679da9700 1 -- 172.21.15.121:0/2023456746 shutdown_connections 2024-04-05T22:00:41.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.413+0000 7f6679da9700 1 -- 172.21.15.121:0/2023456746 wait complete. 2024-04-05T22:00:41.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.413+0000 7f6679da9700 1 Processor -- start 2024-04-05T22:00:41.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.413+0000 7f6679da9700 1 -- start start 2024-04-05T22:00:41.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.413+0000 7f6679da9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6674100de0 0x7f66741966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:41.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.413+0000 7f6679da9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6674101790 0x7f6674196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:41.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.413+0000 7f6679da9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6674197280 con 0x7f6674100de0 2024-04-05T22:00:41.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.413+0000 7f6679da9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6674199fb0 con 0x7f6674101790 2024-04-05T22:00:41.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f66737fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6674100de0 0x7f66741966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:41.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f66737fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6674100de0 0x7f66741966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45622/0 (socket says 172.21.15.121:45622) 2024-04-05T22:00:41.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f66737fe700 1 -- 172.21.15.121:0/4242164068 learned_addr learned my addr 172.21.15.121:0/4242164068 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:41.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f6672ffd700 1 --2- 172.21.15.121:0/4242164068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6674101790 0x7f6674196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:41.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f66737fe700 1 -- 172.21.15.121:0/4242164068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6674101790 msgr2=0x7f6674196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:41.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f66737fe700 1 --2- 172.21.15.121:0/4242164068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6674101790 0x7f6674196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f66737fe700 1 -- 172.21.15.121:0/4242164068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66680095f0 con 0x7f6674100de0 2024-04-05T22:00:41.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f6672ffd700 1 --2- 172.21.15.121:0/4242164068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6674101790 0x7f6674196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:00:41.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f66737fe700 1 --2- 172.21.15.121:0/4242164068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6674100de0 0x7f66741966b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6668006010 tx=0x7f666800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:41.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f6670ff9700 1 -- 172.21.15.121:0/4242164068 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f666801d070 con 0x7f6674100de0 2024-04-05T22:00:41.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f6670ff9700 1 -- 172.21.15.121:0/4242164068 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f666800f460 con 0x7f6674100de0 2024-04-05T22:00:41.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f6679da9700 1 -- 172.21.15.121:0/4242164068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667419a230 con 0x7f6674100de0 2024-04-05T22:00:41.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f6670ff9700 1 -- 172.21.15.121:0/4242164068 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6668021640 con 0x7f6674100de0 2024-04-05T22:00:41.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.414+0000 7f6679da9700 1 -- 172.21.15.121:0/4242164068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667419a720 con 0x7f6674100de0 2024-04-05T22:00:41.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.415+0000 7f6679da9700 1 -- 172.21.15.121:0/4242164068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6674109180 con 0x7f6674100de0 2024-04-05T22:00:41.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.416+0000 7f6670ff9700 1 -- 172.21.15.121:0/4242164068 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f666800fa60 con 0x7f6674100de0 2024-04-05T22:00:41.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.416+0000 7f6670ff9700 1 --2- 172.21.15.121:0/4242164068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f665c06cce0 0x7f665c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:41.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.416+0000 7f6670ff9700 1 -- 172.21.15.121:0/4242164068 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f666808d880 con 0x7f6674100de0 2024-04-05T22:00:41.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.416+0000 7f6672ffd700 1 --2- 172.21.15.121:0/4242164068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f665c06cce0 0x7f665c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:41.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.417+0000 7f6672ffd700 1 --2- 172.21.15.121:0/4242164068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f665c06cce0 0x7f665c06f1a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6674197cd0 tx=0x7f6664005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:41.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.421+0000 7f6670ff9700 1 -- 172.21.15.121:0/4242164068 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f66680578f0 con 0x7f6674100de0 2024-04-05T22:00:41.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:41 smithi121 ceph-mon[145859]: pgmap v1034: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:41.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.622+0000 7f6679da9700 1 -- 172.21.15.121:0/4242164068 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f667419a940 con 0x7f665c06cce0 2024-04-05T22:00:41.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.632+0000 7f6670ff9700 1 -- 172.21.15.121:0/4242164068 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f667419a940 con 0x7f665c06cce0 2024-04-05T22:00:41.633 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T22:00:41.633 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (31m) 10m ago 42m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T22:00:41.633 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (43m) 10m ago 43m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T22:00:41.633 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (42m) 10m ago 42m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T22:00:41.633 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (29m) 10m ago 43m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T22:00:41.633 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (29m) 10m ago 42m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T22:00:41.633 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (31m) 10m ago 42m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T22:00:41.633 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (25m) 10m ago 36m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T22:00:41.633 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (25m) 10m ago 36m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T22:00:41.633 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (25m) 10m ago 36m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T22:00:41.633 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (10m) 10m ago 36m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T22:00:41.633 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (33m) 10m ago 44m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T22:00:41.633 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (32m) 10m ago 41m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T22:00:41.634 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (30m) 10m ago 44m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T22:00:41.634 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (29m) 10m ago 41m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T22:00:41.634 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (32m) 10m ago 43m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T22:00:41.634 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (32m) 10m ago 41m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T22:00:41.634 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (29m) 10m ago 40m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T22:00:41.634 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (28m) 10m ago 40m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T22:00:41.634 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (28m) 10m ago 39m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T22:00:41.634 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (27m) 10m ago 39m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T22:00:41.634 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (26m) 10m ago 38m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T22:00:41.634 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (26m) 10m ago 38m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T22:00:41.634 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (31m) 10m ago 42m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T22:00:41.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.634+0000 7f6679da9700 1 -- 172.21.15.121:0/4242164068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f665c06cce0 msgr2=0x7f665c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:41.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.634+0000 7f6679da9700 1 --2- 172.21.15.121:0/4242164068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f665c06cce0 0x7f665c06f1a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6674197cd0 tx=0x7f6664005f50 comp rx=0 tx=0).stop 2024-04-05T22:00:41.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.634+0000 7f6679da9700 1 -- 172.21.15.121:0/4242164068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6674100de0 msgr2=0x7f66741966b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:41.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.634+0000 7f6679da9700 1 --2- 172.21.15.121:0/4242164068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6674100de0 0x7f66741966b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f6668006010 tx=0x7f666800be20 comp rx=0 tx=0).stop 2024-04-05T22:00:41.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.634+0000 7f6679da9700 1 -- 172.21.15.121:0/4242164068 shutdown_connections 2024-04-05T22:00:41.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.634+0000 7f6679da9700 1 --2- 172.21.15.121:0/4242164068 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f665c06cce0 0x7f665c06f1a0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.634+0000 7f6679da9700 1 --2- 172.21.15.121:0/4242164068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6674100de0 0x7f66741966b0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.635+0000 7f6679da9700 1 --2- 172.21.15.121:0/4242164068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6674101790 0x7f6674196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.635+0000 7f6679da9700 1 -- 172.21.15.121:0/4242164068 >> 172.21.15.121:0/4242164068 conn(0x7f66740fc6b0 msgr2=0x7f66740fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:41.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.635+0000 7f6679da9700 1 -- 172.21.15.121:0/4242164068 shutdown_connections 2024-04-05T22:00:41.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.635+0000 7f6679da9700 1 -- 172.21.15.121:0/4242164068 wait complete. 2024-04-05T22:00:41.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.735+0000 7f6c45498700 1 -- 172.21.15.121:0/2738319000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c40104ae0 msgr2=0x7f6c40104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:41.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.735+0000 7f6c45498700 1 --2- 172.21.15.121:0/2738319000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c40104ae0 0x7f6c40104ec0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009910 tx=0x7f6c34009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:41.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.736+0000 7f6c45498700 1 -- 172.21.15.121:0/2738319000 shutdown_connections 2024-04-05T22:00:41.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.736+0000 7f6c45498700 1 --2- 172.21.15.121:0/2738319000 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c401003e0 0x7f6c40100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.736+0000 7f6c45498700 1 --2- 172.21.15.121:0/2738319000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c40104ae0 0x7f6c40104ec0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.736+0000 7f6c45498700 1 -- 172.21.15.121:0/2738319000 >> 172.21.15.121:0/2738319000 conn(0x7f6c40074df0 msgr2=0x7f6c40075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:41.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.736+0000 7f6c45498700 1 -- 172.21.15.121:0/2738319000 shutdown_connections 2024-04-05T22:00:41.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.736+0000 7f6c45498700 1 -- 172.21.15.121:0/2738319000 wait complete. 2024-04-05T22:00:41.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.736+0000 7f6c45498700 1 Processor -- start 2024-04-05T22:00:41.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.736+0000 7f6c45498700 1 -- start start 2024-04-05T22:00:41.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c45498700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c401003e0 0x7f6c4019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:41.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c45498700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c40104ae0 0x7f6c4019ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:41.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c45498700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4019d0c0 con 0x7f6c40104ae0 2024-04-05T22:00:41.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c45498700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c40196570 con 0x7f6c401003e0 2024-04-05T22:00:41.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c3e7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c40104ae0 0x7f6c4019ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:41.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c3e7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c40104ae0 0x7f6c4019ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45632/0 (socket says 172.21.15.121:45632) 2024-04-05T22:00:41.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c3e7fc700 1 -- 172.21.15.121:0/2005108402 learned_addr learned my addr 172.21.15.121:0/2005108402 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:41.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c3effd700 1 --2- 172.21.15.121:0/2005108402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c401003e0 0x7f6c4019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:41.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c3e7fc700 1 -- 172.21.15.121:0/2005108402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c401003e0 msgr2=0x7f6c4019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:41.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c3e7fc700 1 --2- 172.21.15.121:0/2005108402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c401003e0 0x7f6c4019c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:41.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c3e7fc700 1 -- 172.21.15.121:0/2005108402 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c340095f0 con 0x7f6c40104ae0 2024-04-05T22:00:41.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c3e7fc700 1 --2- 172.21.15.121:0/2005108402 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c40104ae0 0x7f6c4019ca30 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000c8f0 tx=0x7f6c3000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:41.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c27fff700 1 -- 172.21.15.121:0/2005108402 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c30007a70 con 0x7f6c40104ae0 2024-04-05T22:00:41.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c45498700 1 -- 172.21.15.121:0/2005108402 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c40196850 con 0x7f6c40104ae0 2024-04-05T22:00:41.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c27fff700 1 -- 172.21.15.121:0/2005108402 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c30004cd0 con 0x7f6c40104ae0 2024-04-05T22:00:41.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c27fff700 1 -- 172.21.15.121:0/2005108402 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c30005580 con 0x7f6c40104ae0 2024-04-05T22:00:41.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.737+0000 7f6c45498700 1 -- 172.21.15.121:0/2005108402 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c40196da0 con 0x7f6c40104ae0 2024-04-05T22:00:41.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.738+0000 7f6c45498700 1 -- 172.21.15.121:0/2005108402 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c401090e0 con 0x7f6c40104ae0 2024-04-05T22:00:41.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.739+0000 7f6c27fff700 1 -- 172.21.15.121:0/2005108402 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6c30004750 con 0x7f6c40104ae0 2024-04-05T22:00:41.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.739+0000 7f6c27fff700 1 --2- 172.21.15.121:0/2005108402 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6c2806cce0 0x7f6c2806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:41.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.739+0000 7f6c27fff700 1 -- 172.21.15.121:0/2005108402 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6c30013070 con 0x7f6c40104ae0 2024-04-05T22:00:41.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.739+0000 7f6c3effd700 1 --2- 172.21.15.121:0/2005108402 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6c2806cce0 0x7f6c2806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:41.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.740+0000 7f6c3effd700 1 --2- 172.21.15.121:0/2005108402 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6c2806cce0 0x7f6c2806f1a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400b5c0 tx=0x7f6c34005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:41.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:41.744+0000 7f6c27fff700 1 -- 172.21.15.121:0/2005108402 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6c30059f70 con 0x7f6c40104ae0 2024-04-05T22:00:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.018+0000 7f6c45498700 1 -- 172.21.15.121:0/2005108402 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6c4004ea90 con 0x7f6c40104ae0 2024-04-05T22:00:42.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.018+0000 7f6c27fff700 1 -- 172.21.15.121:0/2005108402 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f6c30059b00 con 0x7f6c40104ae0 2024-04-05T22:00:42.019 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:00:42.019 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T22:00:42.019 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:00:42.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:00:42.019 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T22:00:42.019 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:00:42.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:00:42.019 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T22:00:42.019 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T22:00:42.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:00:42.019 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T22:00:42.020 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T22:00:42.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:00:42.020 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T22:00:42.020 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T22:00:42.020 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T22:00:42.020 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:00:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.020+0000 7f6c45498700 1 -- 172.21.15.121:0/2005108402 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6c2806cce0 msgr2=0x7f6c2806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.020+0000 7f6c45498700 1 --2- 172.21.15.121:0/2005108402 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6c2806cce0 0x7f6c2806f1a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6c3400b5c0 tx=0x7f6c34005fb0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.020+0000 7f6c45498700 1 -- 172.21.15.121:0/2005108402 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c40104ae0 msgr2=0x7f6c4019ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.020+0000 7f6c45498700 1 --2- 172.21.15.121:0/2005108402 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c40104ae0 0x7f6c4019ca30 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000c8f0 tx=0x7f6c3000ccb0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.020+0000 7f6c45498700 1 -- 172.21.15.121:0/2005108402 shutdown_connections 2024-04-05T22:00:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.020+0000 7f6c45498700 1 --2- 172.21.15.121:0/2005108402 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6c2806cce0 0x7f6c2806f1a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.020+0000 7f6c45498700 1 --2- 172.21.15.121:0/2005108402 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6c401003e0 0x7f6c4019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.020+0000 7f6c45498700 1 --2- 172.21.15.121:0/2005108402 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c40104ae0 0x7f6c4019ca30 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.020+0000 7f6c45498700 1 -- 172.21.15.121:0/2005108402 >> 172.21.15.121:0/2005108402 conn(0x7f6c40074df0 msgr2=0x7f6c400ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.021+0000 7f6c45498700 1 -- 172.21.15.121:0/2005108402 shutdown_connections 2024-04-05T22:00:42.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.021+0000 7f6c45498700 1 -- 172.21.15.121:0/2005108402 wait complete. 2024-04-05T22:00:42.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.123+0000 7f7587e3a700 1 -- 172.21.15.121:0/3983968621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7580100d70 msgr2=0x7f75801048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:42.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.123+0000 7f7587e3a700 1 --2- 172.21.15.121:0/3983968621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7580100d70 0x7f75801048a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f757c009910 tx=0x7f757c009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:42.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.123+0000 7f7587e3a700 1 -- 172.21.15.121:0/3983968621 shutdown_connections 2024-04-05T22:00:42.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.123+0000 7f7587e3a700 1 --2- 172.21.15.121:0/3983968621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7580100d70 0x7f75801048a0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.123+0000 7f7587e3a700 1 --2- 172.21.15.121:0/3983968621 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f75801003c0 0x7f75801007a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.123+0000 7f7587e3a700 1 -- 172.21.15.121:0/3983968621 >> 172.21.15.121:0/3983968621 conn(0x7f7580075e30 msgr2=0x7f75800fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:42.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.123+0000 7f7587e3a700 1 -- 172.21.15.121:0/3983968621 shutdown_connections 2024-04-05T22:00:42.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.123+0000 7f7587e3a700 1 -- 172.21.15.121:0/3983968621 wait complete. 2024-04-05T22:00:42.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7587e3a700 1 Processor -- start 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7587e3a700 1 -- start start 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7587e3a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75801003c0 0x7f75801955f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7587e3a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7580100d70 0x7f7580195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7587e3a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7580196070 con 0x7f75801003c0 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7587e3a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75801961e0 con 0x7f7580100d70 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7585bd6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75801003c0 0x7f75801955f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7585bd6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75801003c0 0x7f75801955f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45654/0 (socket says 172.21.15.121:45654) 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7585bd6700 1 -- 172.21.15.121:0/2617334286 learned_addr learned my addr 172.21.15.121:0/2617334286 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f75853d5700 1 --2- 172.21.15.121:0/2617334286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7580100d70 0x7f7580195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7585bd6700 1 -- 172.21.15.121:0/2617334286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7580100d70 msgr2=0x7f7580195b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7585bd6700 1 --2- 172.21.15.121:0/2617334286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7580100d70 0x7f7580195b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7585bd6700 1 -- 172.21.15.121:0/2617334286 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c0095f0 con 0x7f75801003c0 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f75853d5700 1 --2- 172.21.15.121:0/2617334286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7580100d70 0x7f7580195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:00:42.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7585bd6700 1 --2- 172.21.15.121:0/2617334286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75801003c0 0x7f75801955f0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f757000b4b0 tx=0x7f757000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:42.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7576ffd700 1 -- 172.21.15.121:0/2617334286 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7570011800 con 0x7f75801003c0 2024-04-05T22:00:42.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.124+0000 7f7576ffd700 1 -- 172.21.15.121:0/2617334286 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7570011e00 con 0x7f75801003c0 2024-04-05T22:00:42.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.125+0000 7f7587e3a700 1 -- 172.21.15.121:0/2617334286 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75801a2c60 con 0x7f75801003c0 2024-04-05T22:00:42.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.125+0000 7f7576ffd700 1 -- 172.21.15.121:0/2617334286 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7570010e50 con 0x7f75801003c0 2024-04-05T22:00:42.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.125+0000 7f7587e3a700 1 -- 172.21.15.121:0/2617334286 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75801a2fc0 con 0x7f75801003c0 2024-04-05T22:00:42.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.126+0000 7f7587e3a700 1 -- 172.21.15.121:0/2617334286 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75801081d0 con 0x7f75801003c0 2024-04-05T22:00:42.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.126+0000 7f7576ffd700 1 -- 172.21.15.121:0/2617334286 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7570011960 con 0x7f75801003c0 2024-04-05T22:00:42.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.126+0000 7f7576ffd700 1 --2- 172.21.15.121:0/2617334286 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f756c06cce0 0x7f756c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:42.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.126+0000 7f7576ffd700 1 -- 172.21.15.121:0/2617334286 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f757008c0e0 con 0x7f75801003c0 2024-04-05T22:00:42.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.126+0000 7f75853d5700 1 --2- 172.21.15.121:0/2617334286 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f756c06cce0 0x7f756c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:42.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.127+0000 7f75853d5700 1 --2- 172.21.15.121:0/2617334286 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f756c06cce0 0x7f756c06f1a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f75800ff9b0 tx=0x7f757c00b540 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:42.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.131+0000 7f7576ffd700 1 -- 172.21.15.121:0/2617334286 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7570059be0 con 0x7f75801003c0 2024-04-05T22:00:42.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:42 smithi188 ceph-mon[107793]: from='client.35690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:42.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.369+0000 7f7587e3a700 1 -- 172.21.15.121:0/2617334286 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f75800ff730 con 0x7f75801003c0 2024-04-05T22:00:42.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.370+0000 7f7576ffd700 1 -- 172.21.15.121:0/2617334286 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f7570059770 con 0x7f75801003c0 2024-04-05T22:00:42.370 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T22:00:42.370 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T22:00:42.370 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:00:42.370 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T22:00:42.370 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:00:42.370 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T22:00:42.370 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:00:42.371 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T22:00:42.372 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:00:42.373 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:00:42.373 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:00:42.373 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:00:42.373 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:00:42.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.372+0000 7f7587e3a700 1 -- 172.21.15.121:0/2617334286 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f756c06cce0 msgr2=0x7f756c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:42.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.372+0000 7f7587e3a700 1 --2- 172.21.15.121:0/2617334286 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f756c06cce0 0x7f756c06f1a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f75800ff9b0 tx=0x7f757c00b540 comp rx=0 tx=0).stop 2024-04-05T22:00:42.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.372+0000 7f7587e3a700 1 -- 172.21.15.121:0/2617334286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75801003c0 msgr2=0x7f75801955f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:42.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.372+0000 7f7587e3a700 1 --2- 172.21.15.121:0/2617334286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75801003c0 0x7f75801955f0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f757000b4b0 tx=0x7f757000b870 comp rx=0 tx=0).stop 2024-04-05T22:00:42.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.372+0000 7f7587e3a700 1 -- 172.21.15.121:0/2617334286 shutdown_connections 2024-04-05T22:00:42.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.372+0000 7f7587e3a700 1 --2- 172.21.15.121:0/2617334286 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f756c06cce0 0x7f756c06f1a0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.372+0000 7f7587e3a700 1 --2- 172.21.15.121:0/2617334286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75801003c0 0x7f75801955f0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.372+0000 7f7587e3a700 1 --2- 172.21.15.121:0/2617334286 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7580100d70 0x7f7580195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.372+0000 7f7587e3a700 1 -- 172.21.15.121:0/2617334286 >> 172.21.15.121:0/2617334286 conn(0x7f7580075e30 msgr2=0x7f75800fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:42.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.372+0000 7f7587e3a700 1 -- 172.21.15.121:0/2617334286 shutdown_connections 2024-04-05T22:00:42.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.372+0000 7f7587e3a700 1 -- 172.21.15.121:0/2617334286 wait complete. 2024-04-05T22:00:42.375 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T22:00:42.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.475+0000 7f6887c9f700 1 -- 172.21.15.121:0/672966706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6880101790 msgr2=0x7f68801057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:42.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.475+0000 7f6887c9f700 1 --2- 172.21.15.121:0/672966706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6880101790 0x7f68801057e0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f6870009910 tx=0x7f6870009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:42.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.475+0000 7f6887c9f700 1 -- 172.21.15.121:0/672966706 shutdown_connections 2024-04-05T22:00:42.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.475+0000 7f6887c9f700 1 --2- 172.21.15.121:0/672966706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6880101790 0x7f68801057e0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.475+0000 7f6887c9f700 1 --2- 172.21.15.121:0/672966706 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6880100de0 0x7f68801011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.475+0000 7f6887c9f700 1 -- 172.21.15.121:0/672966706 >> 172.21.15.121:0/672966706 conn(0x7f68800fc670 msgr2=0x7f68800fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:42.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.475+0000 7f6887c9f700 1 -- 172.21.15.121:0/672966706 shutdown_connections 2024-04-05T22:00:42.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.475+0000 7f6887c9f700 1 -- 172.21.15.121:0/672966706 wait complete. 2024-04-05T22:00:42.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f6887c9f700 1 Processor -- start 2024-04-05T22:00:42.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f6887c9f700 1 -- start start 2024-04-05T22:00:42.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f6887c9f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6880100de0 0x7f68801966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:42.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f6887c9f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6880101790 0x7f6880196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f6887c9f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68801972a0 con 0x7f6880101790 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f6887c9f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6880199fd0 con 0x7f6880100de0 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f688523a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6880101790 0x7f6880196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f688523a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6880101790 0x7f6880196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45676/0 (socket says 172.21.15.121:45676) 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f688523a700 1 -- 172.21.15.121:0/788655131 learned_addr learned my addr 172.21.15.121:0/788655131 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f6885a3b700 1 --2- 172.21.15.121:0/788655131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6880100de0 0x7f68801966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f688523a700 1 -- 172.21.15.121:0/788655131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6880100de0 msgr2=0x7f68801966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f688523a700 1 --2- 172.21.15.121:0/788655131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6880100de0 0x7f68801966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f688523a700 1 -- 172.21.15.121:0/788655131 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68700095f0 con 0x7f6880101790 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.476+0000 7f688523a700 1 --2- 172.21.15.121:0/788655131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6880101790 0x7f6880196c10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f68700092e0 tx=0x7f687000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.477+0000 7f6876ffd700 1 -- 172.21.15.121:0/788655131 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f687001d070 con 0x7f6880101790 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.477+0000 7f6876ffd700 1 -- 172.21.15.121:0/788655131 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6870005940 con 0x7f6880101790 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.477+0000 7f6887c9f700 1 -- 172.21.15.121:0/788655131 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688019a250 con 0x7f6880101790 2024-04-05T22:00:42.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.477+0000 7f6876ffd700 1 -- 172.21.15.121:0/788655131 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f687000fe50 con 0x7f6880101790 2024-04-05T22:00:42.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.477+0000 7f6887c9f700 1 -- 172.21.15.121:0/788655131 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688019a740 con 0x7f6880101790 2024-04-05T22:00:42.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.478+0000 7f6887c9f700 1 -- 172.21.15.121:0/788655131 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68801091e0 con 0x7f6880101790 2024-04-05T22:00:42.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.478+0000 7f6876ffd700 1 -- 172.21.15.121:0/788655131 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6870003a60 con 0x7f6880101790 2024-04-05T22:00:42.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.479+0000 7f6876ffd700 1 --2- 172.21.15.121:0/788655131 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f686c06cce0 0x7f686c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:42.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.479+0000 7f6876ffd700 1 -- 172.21.15.121:0/788655131 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f687008d740 con 0x7f6880101790 2024-04-05T22:00:42.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.479+0000 7f6885a3b700 1 --2- 172.21.15.121:0/788655131 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f686c06cce0 0x7f686c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:42.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.479+0000 7f6885a3b700 1 --2- 172.21.15.121:0/788655131 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f686c06cce0 0x7f686c06f1a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f687c005950 tx=0x7f687c0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:42.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.483+0000 7f6876ffd700 1 -- 172.21.15.121:0/788655131 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f68700577b0 con 0x7f6880101790 2024-04-05T22:00:42.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:42 smithi121 ceph-mon[145859]: from='client.35690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:42.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.693+0000 7f6887c9f700 1 -- 172.21.15.121:0/788655131 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f688002cce0 con 0x7f686c06cce0 2024-04-05T22:00:42.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.694+0000 7f6876ffd700 1 -- 172.21.15.121:0/788655131 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f688002cce0 con 0x7f686c06cce0 2024-04-05T22:00:42.694 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:00:42.694 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T22:00:42.695 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T22:00:42.695 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T22:00:42.695 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T22:00:42.695 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T22:00:42.695 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T22:00:42.695 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T22:00:42.695 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T22:00:42.695 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T22:00:42.695 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T22:00:42.695 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T22:00:42.695 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T22:00:42.695 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T22:00:42.695 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:00:42.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.696+0000 7f6887c9f700 1 -- 172.21.15.121:0/788655131 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f686c06cce0 msgr2=0x7f686c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:42.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.696+0000 7f6887c9f700 1 --2- 172.21.15.121:0/788655131 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f686c06cce0 0x7f686c06f1a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f687c005950 tx=0x7f687c0058e0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.696+0000 7f6887c9f700 1 -- 172.21.15.121:0/788655131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6880101790 msgr2=0x7f6880196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:42.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.696+0000 7f6887c9f700 1 --2- 172.21.15.121:0/788655131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6880101790 0x7f6880196c10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f68700092e0 tx=0x7f687000be20 comp rx=0 tx=0).stop 2024-04-05T22:00:42.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.696+0000 7f6887c9f700 1 -- 172.21.15.121:0/788655131 shutdown_connections 2024-04-05T22:00:42.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.696+0000 7f6887c9f700 1 --2- 172.21.15.121:0/788655131 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f686c06cce0 0x7f686c06f1a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.696+0000 7f6887c9f700 1 --2- 172.21.15.121:0/788655131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6880100de0 0x7f68801966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.696+0000 7f6887c9f700 1 --2- 172.21.15.121:0/788655131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6880101790 0x7f6880196c10 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.696+0000 7f6887c9f700 1 -- 172.21.15.121:0/788655131 >> 172.21.15.121:0/788655131 conn(0x7f68800fc670 msgr2=0x7f68800fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:42.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.696+0000 7f6887c9f700 1 -- 172.21.15.121:0/788655131 shutdown_connections 2024-04-05T22:00:42.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.696+0000 7f6887c9f700 1 -- 172.21.15.121:0/788655131 wait complete. 2024-04-05T22:00:42.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.798+0000 7fbd8e38a700 1 -- 172.21.15.121:0/3438437774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd88102620 msgr2=0x7fbd8810ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:42.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.798+0000 7fbd8e38a700 1 --2- 172.21.15.121:0/3438437774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd88102620 0x7fbd8810ab10 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009910 tx=0x7fbd7c009c20 comp rx=0 tx=0).stop 2024-04-05T22:00:42.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.798+0000 7fbd8e38a700 1 -- 172.21.15.121:0/3438437774 shutdown_connections 2024-04-05T22:00:42.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.798+0000 7fbd8e38a700 1 --2- 172.21.15.121:0/3438437774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd88102620 0x7fbd8810ab10 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.798+0000 7fbd8e38a700 1 --2- 172.21.15.121:0/3438437774 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd88101d00 0x7fbd881020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.798+0000 7fbd8e38a700 1 -- 172.21.15.121:0/3438437774 >> 172.21.15.121:0/3438437774 conn(0x7fbd880fb570 msgr2=0x7fbd880fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:42.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.799+0000 7fbd8e38a700 1 -- 172.21.15.121:0/3438437774 shutdown_connections 2024-04-05T22:00:42.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.799+0000 7fbd8e38a700 1 -- 172.21.15.121:0/3438437774 wait complete. 2024-04-05T22:00:42.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.799+0000 7fbd8e38a700 1 Processor -- start 2024-04-05T22:00:42.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.799+0000 7fbd8e38a700 1 -- start start 2024-04-05T22:00:42.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.799+0000 7fbd8e38a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd88101d00 0x7fbd881966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:42.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.799+0000 7fbd8e38a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd88102620 0x7fbd88196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:42.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.799+0000 7fbd8e38a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd881972a0 con 0x7fbd88101d00 2024-04-05T22:00:42.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.799+0000 7fbd8e38a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd88199fd0 con 0x7fbd88102620 2024-04-05T22:00:42.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.800+0000 7fbd87fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd88101d00 0x7fbd881966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:42.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.800+0000 7fbd87fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd88101d00 0x7fbd881966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45702/0 (socket says 172.21.15.121:45702) 2024-04-05T22:00:42.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.800+0000 7fbd87fff700 1 -- 172.21.15.121:0/1678485229 learned_addr learned my addr 172.21.15.121:0/1678485229 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:00:42.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.800+0000 7fbd877fe700 1 --2- 172.21.15.121:0/1678485229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd88102620 0x7fbd88196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:42.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.800+0000 7fbd87fff700 1 -- 172.21.15.121:0/1678485229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd88102620 msgr2=0x7fbd88196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:42.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.800+0000 7fbd87fff700 1 --2- 172.21.15.121:0/1678485229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd88102620 0x7fbd88196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:42.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.800+0000 7fbd87fff700 1 -- 172.21.15.121:0/1678485229 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7c0095f0 con 0x7fbd88101d00 2024-04-05T22:00:42.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.800+0000 7fbd87fff700 1 --2- 172.21.15.121:0/1678485229 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd88101d00 0x7fbd881966d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800d8d0 tx=0x7fbd7800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:42.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.800+0000 7fbd857fa700 1 -- 172.21.15.121:0/1678485229 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd780098a0 con 0x7fbd88101d00 2024-04-05T22:00:42.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.800+0000 7fbd857fa700 1 -- 172.21.15.121:0/1678485229 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd78010460 con 0x7fbd88101d00 2024-04-05T22:00:42.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.800+0000 7fbd8e38a700 1 -- 172.21.15.121:0/1678485229 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd8819a2b0 con 0x7fbd88101d00 2024-04-05T22:00:42.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.800+0000 7fbd857fa700 1 -- 172.21.15.121:0/1678485229 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd78009ea0 con 0x7fbd88101d00 2024-04-05T22:00:42.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.800+0000 7fbd8e38a700 1 -- 172.21.15.121:0/1678485229 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd8819a7d0 con 0x7fbd88101d00 2024-04-05T22:00:42.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.801+0000 7fbd8e38a700 1 -- 172.21.15.121:0/1678485229 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd88108210 con 0x7fbd88101d00 2024-04-05T22:00:42.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.802+0000 7fbd857fa700 1 -- 172.21.15.121:0/1678485229 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbd780105d0 con 0x7fbd88101d00 2024-04-05T22:00:42.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.802+0000 7fbd857fa700 1 --2- 172.21.15.121:0/1678485229 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd7006cce0 0x7fbd7006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:00:42.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.802+0000 7fbd857fa700 1 -- 172.21.15.121:0/1678485229 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbd7808c4a0 con 0x7fbd88101d00 2024-04-05T22:00:42.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.802+0000 7fbd877fe700 1 --2- 172.21.15.121:0/1678485229 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd7006cce0 0x7fbd7006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:00:42.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.802+0000 7fbd877fe700 1 --2- 172.21.15.121:0/1678485229 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd7006cce0 0x7fbd7006f1a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fbd88197df0 tx=0x7fbd7c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:00:42.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:42.806+0000 7fbd857fa700 1 -- 172.21.15.121:0/1678485229 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbd78056510 con 0x7fbd88101d00 2024-04-05T22:00:43.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:43.077+0000 7fbd8e38a700 1 -- 172.21.15.121:0/1678485229 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbd8804ea90 con 0x7fbd88101d00 2024-04-05T22:00:43.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:43.078+0000 7fbd857fa700 1 -- 172.21.15.121:0/1678485229 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fbd78016e10 con 0x7fbd88101d00 2024-04-05T22:00:43.078 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:00:43.078 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:00:43.078 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:00:43.078 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:00:43.079 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:00:43.080 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:00:43.080 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:00:43.080 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T22:00:43.080 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:00:43.080 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T22:00:43.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:43.080+0000 7fbd8e38a700 1 -- 172.21.15.121:0/1678485229 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd7006cce0 msgr2=0x7fbd7006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:43.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:43.080+0000 7fbd8e38a700 1 --2- 172.21.15.121:0/1678485229 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd7006cce0 0x7fbd7006f1a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fbd88197df0 tx=0x7fbd7c01a040 comp rx=0 tx=0).stop 2024-04-05T22:00:43.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:43.080+0000 7fbd8e38a700 1 -- 172.21.15.121:0/1678485229 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd88101d00 msgr2=0x7fbd881966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:00:43.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:43.080+0000 7fbd8e38a700 1 --2- 172.21.15.121:0/1678485229 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd88101d00 0x7fbd881966d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800d8d0 tx=0x7fbd7800dbe0 comp rx=0 tx=0).stop 2024-04-05T22:00:43.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:43.080+0000 7fbd8e38a700 1 -- 172.21.15.121:0/1678485229 shutdown_connections 2024-04-05T22:00:43.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:43.080+0000 7fbd8e38a700 1 --2- 172.21.15.121:0/1678485229 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd7006cce0 0x7fbd7006f1a0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:43.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:43.080+0000 7fbd8e38a700 1 --2- 172.21.15.121:0/1678485229 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbd88101d00 0x7fbd881966d0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:43.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:43.080+0000 7fbd8e38a700 1 --2- 172.21.15.121:0/1678485229 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbd88102620 0x7fbd88196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:00:43.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:43.080+0000 7fbd8e38a700 1 -- 172.21.15.121:0/1678485229 >> 172.21.15.121:0/1678485229 conn(0x7fbd880fb570 msgr2=0x7fbd88105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:00:43.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:43.080+0000 7fbd8e38a700 1 -- 172.21.15.121:0/1678485229 shutdown_connections 2024-04-05T22:00:43.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:00:43.080+0000 7fbd8e38a700 1 -- 172.21.15.121:0/1678485229 wait complete. 2024-04-05T22:00:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:43 smithi188 ceph-mon[107793]: from='client.35694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:43 smithi188 ceph-mon[107793]: from='client.35698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:43 smithi188 ceph-mon[107793]: pgmap v1035: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:43 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2005108402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:00:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:43 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2617334286' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:00:43.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:43 smithi121 ceph-mon[145859]: from='client.35694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:43.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:43 smithi121 ceph-mon[145859]: from='client.35698 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:43.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:43 smithi121 ceph-mon[145859]: pgmap v1035: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:43.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:43 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2005108402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:00:43.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:43 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2617334286' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:00:44.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:44 smithi188 ceph-mon[107793]: from='client.35710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:44.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:44 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1678485229' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:00:44.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:44 smithi121 ceph-mon[145859]: from='client.35710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:00:44.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:44 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1678485229' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:00:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:45 smithi188 ceph-mon[107793]: pgmap v1036: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:45.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:45 smithi121 ceph-mon[145859]: pgmap v1036: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:46 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:00:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:46 smithi188 ceph-mon[107793]: pgmap v1037: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:46.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:46 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:00:46.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:46 smithi121 ceph-mon[145859]: pgmap v1037: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:49.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:48 smithi121 ceph-mon[145859]: pgmap v1038: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:48 smithi188 ceph-mon[107793]: pgmap v1038: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:51.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:50 smithi121 ceph-mon[145859]: pgmap v1039: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:50 smithi188 ceph-mon[107793]: pgmap v1039: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:53.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:52 smithi121 ceph-mon[145859]: pgmap v1040: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:52 smithi188 ceph-mon[107793]: pgmap v1040: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:55.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:54 smithi121 ceph-mon[145859]: pgmap v1041: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:54 smithi188 ceph-mon[107793]: pgmap v1041: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:56 smithi121 ceph-mon[145859]: pgmap v1042: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:56 smithi188 ceph-mon[107793]: pgmap v1042: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:59.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:00:58 smithi121 ceph-mon[145859]: pgmap v1043: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:00:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:00:58 smithi188 ceph-mon[107793]: pgmap v1043: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:00 smithi121 ceph-mon[145859]: pgmap v1044: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:01:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:00 smithi188 ceph-mon[107793]: pgmap v1044: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:01:03.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:02 smithi121 ceph-mon[145859]: pgmap v1045: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:02 smithi188 ceph-mon[107793]: pgmap v1045: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:05.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:04 smithi121 ceph-mon[145859]: pgmap v1046: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:04 smithi188 ceph-mon[107793]: pgmap v1046: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:07.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:06 smithi121 ceph-mon[145859]: pgmap v1047: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:06 smithi188 ceph-mon[107793]: pgmap v1047: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:09.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:08 smithi121 ceph-mon[145859]: pgmap v1048: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:09.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:08 smithi188 ceph-mon[107793]: pgmap v1048: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:10.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:10 smithi121 ceph-mon[145859]: pgmap v1049: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:10 smithi188 ceph-mon[107793]: pgmap v1049: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:13.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.195+0000 7f68aa271700 1 -- 172.21.15.121:0/2059876598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a4105b50 msgr2=0x7f68a4105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:13.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.195+0000 7f68aa271700 1 --2- 172.21.15.121:0/2059876598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a4105b50 0x7f68a4105f30 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f6898009910 tx=0x7f6898009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:13.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.195+0000 7f68aa271700 1 -- 172.21.15.121:0/2059876598 shutdown_connections 2024-04-05T22:01:13.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.195+0000 7f68aa271700 1 --2- 172.21.15.121:0/2059876598 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68a40fef70 0x7f68a40ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.195+0000 7f68aa271700 1 --2- 172.21.15.121:0/2059876598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a4105b50 0x7f68a4105f30 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.195+0000 7f68aa271700 1 -- 172.21.15.121:0/2059876598 >> 172.21.15.121:0/2059876598 conn(0x7f68a4075e30 msgr2=0x7f68a40fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.195+0000 7f68aa271700 1 -- 172.21.15.121:0/2059876598 shutdown_connections 2024-04-05T22:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.195+0000 7f68aa271700 1 -- 172.21.15.121:0/2059876598 wait complete. 2024-04-05T22:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.196+0000 7f68aa271700 1 Processor -- start 2024-04-05T22:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.196+0000 7f68aa271700 1 -- start start 2024-04-05T22:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68aa271700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68a40fef70 0x7f68a419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68aa271700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a4105b50 0x7f68a419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68aa271700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a419d070 con 0x7f68a4105b50 2024-04-05T22:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68aa271700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a4196590 con 0x7f68a40fef70 2024-04-05T22:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68a37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a4105b50 0x7f68a419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68a37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a4105b50 0x7f68a419ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56122/0 (socket says 172.21.15.121:56122) 2024-04-05T22:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68a37fe700 1 -- 172.21.15.121:0/2104892625 learned_addr learned my addr 172.21.15.121:0/2104892625 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:13.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68a3fff700 1 --2- 172.21.15.121:0/2104892625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68a40fef70 0x7f68a419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:13.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68a37fe700 1 -- 172.21.15.121:0/2104892625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68a40fef70 msgr2=0x7f68a419c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:13.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68a37fe700 1 --2- 172.21.15.121:0/2104892625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68a40fef70 0x7f68a419c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68a37fe700 1 -- 172.21.15.121:0/2104892625 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68980095f0 con 0x7f68a4105b50 2024-04-05T22:01:13.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68a37fe700 1 --2- 172.21.15.121:0/2104892625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a4105b50 0x7f68a419ca50 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f689400e920 tx=0x7f689400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:13.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68a17fa700 1 -- 172.21.15.121:0/2104892625 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f689400c9d0 con 0x7f68a4105b50 2024-04-05T22:01:13.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68aa271700 1 -- 172.21.15.121:0/2104892625 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a4196870 con 0x7f68a4105b50 2024-04-05T22:01:13.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68a17fa700 1 -- 172.21.15.121:0/2104892625 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6894010430 con 0x7f68a4105b50 2024-04-05T22:01:13.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68a17fa700 1 -- 172.21.15.121:0/2104892625 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6894018650 con 0x7f68a4105b50 2024-04-05T22:01:13.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.197+0000 7f68aa271700 1 -- 172.21.15.121:0/2104892625 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a4196dc0 con 0x7f68a4105b50 2024-04-05T22:01:13.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.198+0000 7f68aa271700 1 -- 172.21.15.121:0/2104892625 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a4109120 con 0x7f68a4105b50 2024-04-05T22:01:13.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.199+0000 7f68a17fa700 1 -- 172.21.15.121:0/2104892625 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f68940105a0 con 0x7f68a4105b50 2024-04-05T22:01:13.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.199+0000 7f68a17fa700 1 --2- 172.21.15.121:0/2104892625 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f688c06cce0 0x7f688c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:13.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.199+0000 7f68a17fa700 1 -- 172.21.15.121:0/2104892625 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6894014070 con 0x7f68a4105b50 2024-04-05T22:01:13.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.199+0000 7f68a3fff700 1 --2- 172.21.15.121:0/2104892625 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f688c06cce0 0x7f688c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:13.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.200+0000 7f68a3fff700 1 --2- 172.21.15.121:0/2104892625 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f688c06cce0 0x7f688c06f1a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6898009fd0 tx=0x7f6898005940 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:13.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.204+0000 7f68a17fa700 1 -- 172.21.15.121:0/2104892625 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6894052ab0 con 0x7f68a4105b50 2024-04-05T22:01:13.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:12 smithi121 ceph-mon[145859]: pgmap v1050: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:12 smithi188 ceph-mon[107793]: pgmap v1050: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:13.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.414+0000 7f68aa271700 1 -- 172.21.15.121:0/2104892625 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f68a4196fe0 con 0x7f688c06cce0 2024-04-05T22:01:13.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.416+0000 7f68a17fa700 1 -- 172.21.15.121:0/2104892625 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f68a4196fe0 con 0x7f688c06cce0 2024-04-05T22:01:13.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.418+0000 7f68aa271700 1 -- 172.21.15.121:0/2104892625 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f688c06cce0 msgr2=0x7f688c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:13.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.418+0000 7f68aa271700 1 --2- 172.21.15.121:0/2104892625 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f688c06cce0 0x7f688c06f1a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6898009fd0 tx=0x7f6898005940 comp rx=0 tx=0).stop 2024-04-05T22:01:13.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.418+0000 7f68aa271700 1 -- 172.21.15.121:0/2104892625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a4105b50 msgr2=0x7f68a419ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:13.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.418+0000 7f68aa271700 1 --2- 172.21.15.121:0/2104892625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a4105b50 0x7f68a419ca50 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f689400e920 tx=0x7f689400ec30 comp rx=0 tx=0).stop 2024-04-05T22:01:13.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.418+0000 7f68aa271700 1 -- 172.21.15.121:0/2104892625 shutdown_connections 2024-04-05T22:01:13.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.418+0000 7f68aa271700 1 --2- 172.21.15.121:0/2104892625 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f688c06cce0 0x7f688c06f1a0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.418+0000 7f68aa271700 1 --2- 172.21.15.121:0/2104892625 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68a40fef70 0x7f68a419c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.418+0000 7f68aa271700 1 --2- 172.21.15.121:0/2104892625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a4105b50 0x7f68a419ca50 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.418+0000 7f68aa271700 1 -- 172.21.15.121:0/2104892625 >> 172.21.15.121:0/2104892625 conn(0x7f68a4075e30 msgr2=0x7f68a4104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:13.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.418+0000 7f68aa271700 1 -- 172.21.15.121:0/2104892625 shutdown_connections 2024-04-05T22:01:13.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.418+0000 7f68aa271700 1 -- 172.21.15.121:0/2104892625 wait complete. 2024-04-05T22:01:13.428 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T22:01:13.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.523+0000 7fd8812c1700 1 -- 172.21.15.121:0/1158201747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd87c0ffbd0 msgr2=0x7fd87c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:13.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.523+0000 7fd8812c1700 1 --2- 172.21.15.121:0/1158201747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd87c0ffbd0 0x7fd87c100050 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fd864009910 tx=0x7fd864009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:13.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.523+0000 7fd8812c1700 1 -- 172.21.15.121:0/1158201747 shutdown_connections 2024-04-05T22:01:13.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.523+0000 7fd8812c1700 1 --2- 172.21.15.121:0/1158201747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd87c0ffbd0 0x7fd87c100050 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.523+0000 7fd8812c1700 1 --2- 172.21.15.121:0/1158201747 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd87c105a40 0x7fd87c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.523+0000 7fd8812c1700 1 -- 172.21.15.121:0/1158201747 >> 172.21.15.121:0/1158201747 conn(0x7fd87c0755a0 msgr2=0x7fd87c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:13.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.523+0000 7fd8812c1700 1 -- 172.21.15.121:0/1158201747 shutdown_connections 2024-04-05T22:01:13.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.523+0000 7fd8812c1700 1 -- 172.21.15.121:0/1158201747 wait complete. 2024-04-05T22:01:13.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.524+0000 7fd8812c1700 1 Processor -- start 2024-04-05T22:01:13.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.524+0000 7fd8812c1700 1 -- start start 2024-04-05T22:01:13.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.524+0000 7fd8812c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd87c0ffbd0 0x7fd87c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:13.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.524+0000 7fd8812c1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd87c105a40 0x7fd87c19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:13.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.524+0000 7fd8812c1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c19d040 con 0x7fd87c0ffbd0 2024-04-05T22:01:13.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.524+0000 7fd8812c1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87c196570 con 0x7fd87c105a40 2024-04-05T22:01:13.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.524+0000 7fd87affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd87c0ffbd0 0x7fd87c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:13.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.524+0000 7fd87affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd87c0ffbd0 0x7fd87c19c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56130/0 (socket says 172.21.15.121:56130) 2024-04-05T22:01:13.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.524+0000 7fd87affd700 1 -- 172.21.15.121:0/4170347524 learned_addr learned my addr 172.21.15.121:0/4170347524 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:13.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.524+0000 7fd87a7fc700 1 --2- 172.21.15.121:0/4170347524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd87c105a40 0x7fd87c19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:13.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.524+0000 7fd87affd700 1 -- 172.21.15.121:0/4170347524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd87c105a40 msgr2=0x7fd87c19c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:13.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.524+0000 7fd87affd700 1 --2- 172.21.15.121:0/4170347524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd87c105a40 0x7fd87c19c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.524+0000 7fd87affd700 1 -- 172.21.15.121:0/4170347524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8640095f0 con 0x7fd87c0ffbd0 2024-04-05T22:01:13.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.525+0000 7fd87affd700 1 --2- 172.21.15.121:0/4170347524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd87c0ffbd0 0x7fd87c19c420 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0060b0 tx=0x7fd86c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:13.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.525+0000 7fd873fff700 1 -- 172.21.15.121:0/4170347524 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd86c00dda0 con 0x7fd87c0ffbd0 2024-04-05T22:01:13.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.525+0000 7fd873fff700 1 -- 172.21.15.121:0/4170347524 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd86c00f040 con 0x7fd87c0ffbd0 2024-04-05T22:01:13.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.525+0000 7fd8812c1700 1 -- 172.21.15.121:0/4170347524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87c196850 con 0x7fd87c0ffbd0 2024-04-05T22:01:13.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.525+0000 7fd873fff700 1 -- 172.21.15.121:0/4170347524 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd86c0145d0 con 0x7fd87c0ffbd0 2024-04-05T22:01:13.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.525+0000 7fd8812c1700 1 -- 172.21.15.121:0/4170347524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87c196da0 con 0x7fd87c0ffbd0 2024-04-05T22:01:13.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.526+0000 7fd8812c1700 1 -- 172.21.15.121:0/4170347524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87c108fe0 con 0x7fd87c0ffbd0 2024-04-05T22:01:13.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.526+0000 7fd873fff700 1 -- 172.21.15.121:0/4170347524 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd86c015400 con 0x7fd87c0ffbd0 2024-04-05T22:01:13.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.526+0000 7fd873fff700 1 --2- 172.21.15.121:0/4170347524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd86806cce0 0x7fd86806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:13.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.527+0000 7fd873fff700 1 -- 172.21.15.121:0/4170347524 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd86c08bf30 con 0x7fd87c0ffbd0 2024-04-05T22:01:13.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.527+0000 7fd87a7fc700 1 --2- 172.21.15.121:0/4170347524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd86806cce0 0x7fd86806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:13.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.527+0000 7fd87a7fc700 1 --2- 172.21.15.121:0/4170347524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd86806cce0 0x7fd86806f1a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd87c197cf0 tx=0x7fd86401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:13.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.531+0000 7fd873fff700 1 -- 172.21.15.121:0/4170347524 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd86c0bfab0 con 0x7fd87c0ffbd0 2024-04-05T22:01:13.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.740+0000 7fd8812c1700 1 -- 172.21.15.121:0/4170347524 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd87c197070 con 0x7fd86806cce0 2024-04-05T22:01:13.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.742+0000 7fd873fff700 1 -- 172.21.15.121:0/4170347524 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd87c197070 con 0x7fd86806cce0 2024-04-05T22:01:13.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.744+0000 7fd8812c1700 1 -- 172.21.15.121:0/4170347524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd86806cce0 msgr2=0x7fd86806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:13.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.744+0000 7fd8812c1700 1 --2- 172.21.15.121:0/4170347524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd86806cce0 0x7fd86806f1a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd87c197cf0 tx=0x7fd86401a040 comp rx=0 tx=0).stop 2024-04-05T22:01:13.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.744+0000 7fd8812c1700 1 -- 172.21.15.121:0/4170347524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd87c0ffbd0 msgr2=0x7fd87c19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:13.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.744+0000 7fd8812c1700 1 --2- 172.21.15.121:0/4170347524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd87c0ffbd0 0x7fd87c19c420 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0060b0 tx=0x7fd86c00d3b0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.744+0000 7fd8812c1700 1 -- 172.21.15.121:0/4170347524 shutdown_connections 2024-04-05T22:01:13.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.744+0000 7fd8812c1700 1 --2- 172.21.15.121:0/4170347524 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd86806cce0 0x7fd86806f1a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.744+0000 7fd8812c1700 1 --2- 172.21.15.121:0/4170347524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd87c0ffbd0 0x7fd87c19c420 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.744+0000 7fd8812c1700 1 --2- 172.21.15.121:0/4170347524 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd87c105a40 0x7fd87c19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.744+0000 7fd8812c1700 1 -- 172.21.15.121:0/4170347524 >> 172.21.15.121:0/4170347524 conn(0x7fd87c0755a0 msgr2=0x7fd87c103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:13.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.744+0000 7fd8812c1700 1 -- 172.21.15.121:0/4170347524 shutdown_connections 2024-04-05T22:01:13.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.744+0000 7fd8812c1700 1 -- 172.21.15.121:0/4170347524 wait complete. 2024-04-05T22:01:13.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.848+0000 7f0bc5305700 1 -- 172.21.15.121:0/3517927317 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc0069020 msgr2=0x7f0bc0105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:13.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.848+0000 7f0bc5305700 1 --2- 172.21.15.121:0/3517927317 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc0069020 0x7f0bc0105790 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0009910 tx=0x7f0bb0009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:13.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.848+0000 7f0bc5305700 1 -- 172.21.15.121:0/3517927317 shutdown_connections 2024-04-05T22:01:13.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.848+0000 7f0bc5305700 1 --2- 172.21.15.121:0/3517927317 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc0069020 0x7f0bc0105790 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.848+0000 7f0bc5305700 1 --2- 172.21.15.121:0/3517927317 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bc0068670 0x7f0bc0068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.848+0000 7f0bc5305700 1 -- 172.21.15.121:0/3517927317 >> 172.21.15.121:0/3517927317 conn(0x7f0bc00755a0 msgr2=0x7f0bc00759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:13.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.848+0000 7f0bc5305700 1 -- 172.21.15.121:0/3517927317 shutdown_connections 2024-04-05T22:01:13.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.848+0000 7f0bc5305700 1 -- 172.21.15.121:0/3517927317 wait complete. 2024-04-05T22:01:13.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.849+0000 7f0bc5305700 1 Processor -- start 2024-04-05T22:01:13.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.849+0000 7f0bc5305700 1 -- start start 2024-04-05T22:01:13.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.849+0000 7f0bc5305700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc0068670 0x7f0bc0196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:13.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.849+0000 7f0bc5305700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bc0069020 0x7f0bc0196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:13.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.849+0000 7f0bc5305700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc0197260 con 0x7f0bc0068670 2024-04-05T22:01:13.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.849+0000 7f0bc5305700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc019aff0 con 0x7f0bc0069020 2024-04-05T22:01:13.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.849+0000 7f0bbeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc0068670 0x7f0bc0196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:13.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.849+0000 7f0bbeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc0068670 0x7f0bc0196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56150/0 (socket says 172.21.15.121:56150) 2024-04-05T22:01:13.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.849+0000 7f0bbeffd700 1 -- 172.21.15.121:0/3336497728 learned_addr learned my addr 172.21.15.121:0/3336497728 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:13.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.849+0000 7f0bbe7fc700 1 --2- 172.21.15.121:0/3336497728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bc0069020 0x7f0bc0196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:13.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.850+0000 7f0bbeffd700 1 -- 172.21.15.121:0/3336497728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bc0069020 msgr2=0x7f0bc0196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:13.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.850+0000 7f0bbeffd700 1 --2- 172.21.15.121:0/3336497728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bc0069020 0x7f0bc0196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.850+0000 7f0bbeffd700 1 -- 172.21.15.121:0/3336497728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb00095f0 con 0x7f0bc0068670 2024-04-05T22:01:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.850+0000 7f0bbeffd700 1 --2- 172.21.15.121:0/3336497728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc0068670 0x7f0bc0196640 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400d8d0 tx=0x7f0bb400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.850+0000 7f0ba7fff700 1 -- 172.21.15.121:0/3336497728 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bb4009940 con 0x7f0bc0068670 2024-04-05T22:01:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.850+0000 7f0ba7fff700 1 -- 172.21.15.121:0/3336497728 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bb4010460 con 0x7f0bc0068670 2024-04-05T22:01:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.850+0000 7f0bc5305700 1 -- 172.21.15.121:0/3336497728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc019b2d0 con 0x7f0bc0068670 2024-04-05T22:01:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.850+0000 7f0ba7fff700 1 -- 172.21.15.121:0/3336497728 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bb400f5f0 con 0x7f0bc0068670 2024-04-05T22:01:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.850+0000 7f0bc5305700 1 -- 172.21.15.121:0/3336497728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc019b820 con 0x7f0bc0068670 2024-04-05T22:01:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.851+0000 7f0bc5305700 1 -- 172.21.15.121:0/3336497728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc01090a0 con 0x7f0bc0068670 2024-04-05T22:01:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.852+0000 7f0ba7fff700 1 -- 172.21.15.121:0/3336497728 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0bb40105d0 con 0x7f0bc0068670 2024-04-05T22:01:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.852+0000 7f0ba7fff700 1 --2- 172.21.15.121:0/3336497728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ba806cce0 0x7f0ba806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.852+0000 7f0ba7fff700 1 -- 172.21.15.121:0/3336497728 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0bb408c5e0 con 0x7f0bc0068670 2024-04-05T22:01:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.852+0000 7f0bbe7fc700 1 --2- 172.21.15.121:0/3336497728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ba806cce0 0x7f0ba806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.852+0000 7f0bbe7fc700 1 --2- 172.21.15.121:0/3336497728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ba806cce0 0x7f0ba806f1a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0197c60 tx=0x7f0bb001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:13.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:13.856+0000 7f0ba7fff700 1 -- 172.21.15.121:0/3336497728 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0bb4056650 con 0x7f0bc0068670 2024-04-05T22:01:14.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.059+0000 7f0bc5305700 1 -- 172.21.15.121:0/3336497728 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bc019bb30 con 0x7f0ba806cce0 2024-04-05T22:01:14.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.069+0000 7f0ba7fff700 1 -- 172.21.15.121:0/3336497728 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f0bc019bb30 con 0x7f0ba806cce0 2024-04-05T22:01:14.069 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T22:01:14.069 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (32m) 10m ago 43m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T22:01:14.069 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (43m) 10m ago 43m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T22:01:14.069 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (42m) 10m ago 42m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T22:01:14.069 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (30m) 10m ago 43m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T22:01:14.070 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (30m) 10m ago 42m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T22:01:14.070 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (31m) 10m ago 43m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T22:01:14.070 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (26m) 10m ago 37m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T22:01:14.070 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (26m) 10m ago 37m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T22:01:14.070 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (25m) 10m ago 37m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T22:01:14.070 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (10m) 10m ago 37m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T22:01:14.070 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (34m) 10m ago 45m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T22:01:14.070 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (33m) 10m ago 42m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T22:01:14.070 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (30m) 10m ago 45m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T22:01:14.070 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (30m) 10m ago 42m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T22:01:14.070 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (32m) 10m ago 43m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T22:01:14.070 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (32m) 10m ago 42m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T22:01:14.071 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (29m) 10m ago 41m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T22:01:14.071 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (29m) 10m ago 40m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T22:01:14.071 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (28m) 10m ago 40m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T22:01:14.071 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (27m) 10m ago 40m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T22:01:14.071 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (27m) 10m ago 39m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T22:01:14.071 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (26m) 10m ago 39m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T22:01:14.071 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (32m) 10m ago 42m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T22:01:14.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.071+0000 7f0bc5305700 1 -- 172.21.15.121:0/3336497728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ba806cce0 msgr2=0x7f0ba806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:14.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.071+0000 7f0bc5305700 1 --2- 172.21.15.121:0/3336497728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ba806cce0 0x7f0ba806f1a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0197c60 tx=0x7f0bb001a040 comp rx=0 tx=0).stop 2024-04-05T22:01:14.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.071+0000 7f0bc5305700 1 -- 172.21.15.121:0/3336497728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc0068670 msgr2=0x7f0bc0196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:14.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.071+0000 7f0bc5305700 1 --2- 172.21.15.121:0/3336497728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc0068670 0x7f0bc0196640 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0bb400d8d0 tx=0x7f0bb400dc90 comp rx=0 tx=0).stop 2024-04-05T22:01:14.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.071+0000 7f0bc5305700 1 -- 172.21.15.121:0/3336497728 shutdown_connections 2024-04-05T22:01:14.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.071+0000 7f0bc5305700 1 --2- 172.21.15.121:0/3336497728 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ba806cce0 0x7f0ba806f1a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.071+0000 7f0bc5305700 1 --2- 172.21.15.121:0/3336497728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bc0068670 0x7f0bc0196640 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.071+0000 7f0bc5305700 1 --2- 172.21.15.121:0/3336497728 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0bc0069020 0x7f0bc0196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.071+0000 7f0bc5305700 1 -- 172.21.15.121:0/3336497728 >> 172.21.15.121:0/3336497728 conn(0x7f0bc00755a0 msgr2=0x7f0bc0105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:14.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.071+0000 7f0bc5305700 1 -- 172.21.15.121:0/3336497728 shutdown_connections 2024-04-05T22:01:14.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.071+0000 7f0bc5305700 1 -- 172.21.15.121:0/3336497728 wait complete. 2024-04-05T22:01:14.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.172+0000 7f1946105700 1 -- 172.21.15.121:0/1794904479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1940068670 msgr2=0x7f1940068a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:14.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.172+0000 7f1946105700 1 --2- 172.21.15.121:0/1794904479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1940068670 0x7f1940068a50 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1938009910 tx=0x7f1938009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:14.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.172+0000 7f1946105700 1 -- 172.21.15.121:0/1794904479 shutdown_connections 2024-04-05T22:01:14.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.172+0000 7f1946105700 1 --2- 172.21.15.121:0/1794904479 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1940069020 0x7f1940105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.172+0000 7f1946105700 1 --2- 172.21.15.121:0/1794904479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1940068670 0x7f1940068a50 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.172+0000 7f1946105700 1 -- 172.21.15.121:0/1794904479 >> 172.21.15.121:0/1794904479 conn(0x7f19400755a0 msgr2=0x7f19400759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:14.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.172+0000 7f1946105700 1 -- 172.21.15.121:0/1794904479 shutdown_connections 2024-04-05T22:01:14.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.172+0000 7f1946105700 1 -- 172.21.15.121:0/1794904479 wait complete. 2024-04-05T22:01:14.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.172+0000 7f1946105700 1 Processor -- start 2024-04-05T22:01:14.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f1946105700 1 -- start start 2024-04-05T22:01:14.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f1946105700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1940068670 0x7f19400ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:14.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f1946105700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1940069020 0x7f19401001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:14.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f1946105700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1940100720 con 0x7f1940068670 2024-04-05T22:01:14.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f1946105700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1940100860 con 0x7f1940069020 2024-04-05T22:01:14.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f193f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1940068670 0x7f19400ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:14.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f193f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1940068670 0x7f19400ffca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56172/0 (socket says 172.21.15.121:56172) 2024-04-05T22:01:14.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f193f7fe700 1 -- 172.21.15.121:0/3978094731 learned_addr learned my addr 172.21.15.121:0/3978094731 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:14.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f193effd700 1 --2- 172.21.15.121:0/3978094731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1940069020 0x7f19401001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:14.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f193f7fe700 1 -- 172.21.15.121:0/3978094731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1940069020 msgr2=0x7f19401001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:14.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f193f7fe700 1 --2- 172.21.15.121:0/3978094731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1940069020 0x7f19401001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f193f7fe700 1 -- 172.21.15.121:0/3978094731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19380095f0 con 0x7f1940068670 2024-04-05T22:01:14.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f193f7fe700 1 --2- 172.21.15.121:0/3978094731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1940068670 0x7f19400ffca0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1938006010 tx=0x7f193800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:14.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.173+0000 7f193cff9700 1 -- 172.21.15.121:0/3978094731 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f193801d070 con 0x7f1940068670 2024-04-05T22:01:14.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.174+0000 7f193cff9700 1 -- 172.21.15.121:0/3978094731 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1938005710 con 0x7f1940068670 2024-04-05T22:01:14.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.174+0000 7f1946105700 1 -- 172.21.15.121:0/3978094731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1940100ae0 con 0x7f1940068670 2024-04-05T22:01:14.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.174+0000 7f193cff9700 1 -- 172.21.15.121:0/3978094731 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f193800fe20 con 0x7f1940068670 2024-04-05T22:01:14.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.174+0000 7f1946105700 1 -- 172.21.15.121:0/3978094731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19401a2b70 con 0x7f1940068670 2024-04-05T22:01:14.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.175+0000 7f1946105700 1 -- 172.21.15.121:0/3978094731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1940109130 con 0x7f1940068670 2024-04-05T22:01:14.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.175+0000 7f193cff9700 1 -- 172.21.15.121:0/3978094731 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f1938003a60 con 0x7f1940068670 2024-04-05T22:01:14.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.175+0000 7f193cff9700 1 --2- 172.21.15.121:0/3978094731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f192c06cce0 0x7f192c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:14.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.175+0000 7f193cff9700 1 -- 172.21.15.121:0/3978094731 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f193808d790 con 0x7f1940068670 2024-04-05T22:01:14.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.176+0000 7f193effd700 1 --2- 172.21.15.121:0/3978094731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f192c06cce0 0x7f192c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:14.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.176+0000 7f193effd700 1 --2- 172.21.15.121:0/3978094731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f192c06cce0 0x7f192c06f1a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1940101490 tx=0x7f192800b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:14.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.180+0000 7f193cff9700 1 -- 172.21.15.121:0/3978094731 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1938057800 con 0x7f1940068670 2024-04-05T22:01:14.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.458+0000 7f1946105700 1 -- 172.21.15.121:0/3978094731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f194004ea90 con 0x7f1940068670 2024-04-05T22:01:14.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.459+0000 7f193cff9700 1 -- 172.21.15.121:0/3978094731 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f1938003d10 con 0x7f1940068670 2024-04-05T22:01:14.459 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:01:14.459 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T22:01:14.459 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:01:14.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:01:14.460 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T22:01:14.460 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:01:14.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:01:14.460 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T22:01:14.460 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T22:01:14.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:01:14.460 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T22:01:14.460 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T22:01:14.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:01:14.460 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T22:01:14.460 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T22:01:14.460 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T22:01:14.460 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:01:14.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.461+0000 7f1946105700 1 -- 172.21.15.121:0/3978094731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f192c06cce0 msgr2=0x7f192c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:14.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.461+0000 7f1946105700 1 --2- 172.21.15.121:0/3978094731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f192c06cce0 0x7f192c06f1a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1940101490 tx=0x7f192800b410 comp rx=0 tx=0).stop 2024-04-05T22:01:14.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.461+0000 7f1946105700 1 -- 172.21.15.121:0/3978094731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1940068670 msgr2=0x7f19400ffca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:14.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.461+0000 7f1946105700 1 --2- 172.21.15.121:0/3978094731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1940068670 0x7f19400ffca0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f1938006010 tx=0x7f193800bd70 comp rx=0 tx=0).stop 2024-04-05T22:01:14.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.461+0000 7f1946105700 1 -- 172.21.15.121:0/3978094731 shutdown_connections 2024-04-05T22:01:14.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.461+0000 7f1946105700 1 --2- 172.21.15.121:0/3978094731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f192c06cce0 0x7f192c06f1a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.461+0000 7f1946105700 1 --2- 172.21.15.121:0/3978094731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1940068670 0x7f19400ffca0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.461+0000 7f1946105700 1 --2- 172.21.15.121:0/3978094731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1940069020 0x7f19401001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.461+0000 7f1946105700 1 -- 172.21.15.121:0/3978094731 >> 172.21.15.121:0/3978094731 conn(0x7f19400755a0 msgr2=0x7f19400fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:14.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.461+0000 7f1946105700 1 -- 172.21.15.121:0/3978094731 shutdown_connections 2024-04-05T22:01:14.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.461+0000 7f1946105700 1 -- 172.21.15.121:0/3978094731 wait complete. 2024-04-05T22:01:14.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.563+0000 7f07b5623700 1 -- 172.21.15.121:0/4152895292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07b00ffa70 msgr2=0x7f07b00ffef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:14.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.563+0000 7f07b5623700 1 --2- 172.21.15.121:0/4152895292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07b00ffa70 0x7f07b00ffef0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f07a0009910 tx=0x7f07a0009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:14.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.563+0000 7f07b5623700 1 -- 172.21.15.121:0/4152895292 shutdown_connections 2024-04-05T22:01:14.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.563+0000 7f07b5623700 1 --2- 172.21.15.121:0/4152895292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07b00ffa70 0x7f07b00ffef0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.563+0000 7f07b5623700 1 --2- 172.21.15.121:0/4152895292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07b0105a70 0x7f07b0105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.563+0000 7f07b5623700 1 -- 172.21.15.121:0/4152895292 >> 172.21.15.121:0/4152895292 conn(0x7f07b00fb560 msgr2=0x7f07b00fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:14.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.563+0000 7f07b5623700 1 -- 172.21.15.121:0/4152895292 shutdown_connections 2024-04-05T22:01:14.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.563+0000 7f07b5623700 1 -- 172.21.15.121:0/4152895292 wait complete. 2024-04-05T22:01:14.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.564+0000 7f07b5623700 1 Processor -- start 2024-04-05T22:01:14.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.564+0000 7f07b5623700 1 -- start start 2024-04-05T22:01:14.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.564+0000 7f07b5623700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07b00ffa70 0x7f07b019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:14.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.564+0000 7f07b5623700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07b0105a70 0x7f07b019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:14.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.564+0000 7f07b5623700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b019f1d0 con 0x7f07b0105a70 2024-04-05T22:01:14.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.564+0000 7f07b5623700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b0198680 con 0x7f07b00ffa70 2024-04-05T22:01:14.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.564+0000 7f07ae7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07b0105a70 0x7f07b019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:14.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.564+0000 7f07ae7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07b0105a70 0x7f07b019eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42438/0 (socket says 172.21.15.121:42438) 2024-04-05T22:01:14.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.564+0000 7f07ae7fc700 1 -- 172.21.15.121:0/2124581300 learned_addr learned my addr 172.21.15.121:0/2124581300 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:14.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.565+0000 7f07aeffd700 1 --2- 172.21.15.121:0/2124581300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07b00ffa70 0x7f07b019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:14.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.565+0000 7f07ae7fc700 1 -- 172.21.15.121:0/2124581300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07b00ffa70 msgr2=0x7f07b019e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:14.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.565+0000 7f07ae7fc700 1 --2- 172.21.15.121:0/2124581300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07b00ffa70 0x7f07b019e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.565+0000 7f07ae7fc700 1 -- 172.21.15.121:0/2124581300 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a00095f0 con 0x7f07b0105a70 2024-04-05T22:01:14.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.565+0000 7f07ae7fc700 1 --2- 172.21.15.121:0/2124581300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07b0105a70 0x7f07b019eb40 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f07a0006010 tx=0x7f07a000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:14.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.565+0000 7f0797fff700 1 -- 172.21.15.121:0/2124581300 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07a001d070 con 0x7f07b0105a70 2024-04-05T22:01:14.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.565+0000 7f0797fff700 1 -- 172.21.15.121:0/2124581300 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07a000f460 con 0x7f07b0105a70 2024-04-05T22:01:14.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.565+0000 7f07b5623700 1 -- 172.21.15.121:0/2124581300 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b0198900 con 0x7f07b0105a70 2024-04-05T22:01:14.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.565+0000 7f0797fff700 1 -- 172.21.15.121:0/2124581300 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07a0021640 con 0x7f07b0105a70 2024-04-05T22:01:14.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.565+0000 7f07b5623700 1 -- 172.21.15.121:0/2124581300 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b0198df0 con 0x7f07b0105a70 2024-04-05T22:01:14.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.566+0000 7f07b5623700 1 -- 172.21.15.121:0/2124581300 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b004ea90 con 0x7f07b0105a70 2024-04-05T22:01:14.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.567+0000 7f0797fff700 1 -- 172.21.15.121:0/2124581300 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f07a000fa60 con 0x7f07b0105a70 2024-04-05T22:01:14.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.567+0000 7f0797fff700 1 --2- 172.21.15.121:0/2124581300 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f079806cce0 0x7f079806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:14.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.567+0000 7f0797fff700 1 -- 172.21.15.121:0/2124581300 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f07a008dce0 con 0x7f07b0105a70 2024-04-05T22:01:14.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.567+0000 7f07aeffd700 1 --2- 172.21.15.121:0/2124581300 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f079806cce0 0x7f079806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:14.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.568+0000 7f07aeffd700 1 --2- 172.21.15.121:0/2124581300 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f079806cce0 0x7f079806f1a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f07a4005fd0 tx=0x7f07a400a560 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:14.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.571+0000 7f0797fff700 1 -- 172.21.15.121:0/2124581300 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f07a005b7c0 con 0x7f07b0105a70 2024-04-05T22:01:14.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.804+0000 7f07b5623700 1 -- 172.21.15.121:0/2124581300 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f07b0199b90 con 0x7f07b0105a70 2024-04-05T22:01:14.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.804+0000 7f0797fff700 1 -- 172.21.15.121:0/2124581300 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f07a005b350 con 0x7f07b0105a70 2024-04-05T22:01:14.805 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T22:01:14.805 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T22:01:14.805 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:01:14.805 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T22:01:14.805 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:01:14.805 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T22:01:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T22:01:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T22:01:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T22:01:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T22:01:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T22:01:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T22:01:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T22:01:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T22:01:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T22:01:14.806 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T22:01:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T22:01:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T22:01:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T22:01:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T22:01:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:01:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T22:01:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T22:01:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T22:01:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T22:01:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T22:01:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T22:01:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T22:01:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T22:01:14.808 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T22:01:14.808 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T22:01:14.808 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T22:01:14.808 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T22:01:14.808 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:01:14.808 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:01:14.808 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T22:01:14.808 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:01:14.808 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:01:14.808 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:01:14.808 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:01:14.808 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:01:14.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.806+0000 7f07b5623700 1 -- 172.21.15.121:0/2124581300 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f079806cce0 msgr2=0x7f079806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:14.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.806+0000 7f07b5623700 1 --2- 172.21.15.121:0/2124581300 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f079806cce0 0x7f079806f1a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f07a4005fd0 tx=0x7f07a400a560 comp rx=0 tx=0).stop 2024-04-05T22:01:14.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.807+0000 7f07b5623700 1 -- 172.21.15.121:0/2124581300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07b0105a70 msgr2=0x7f07b019eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:14.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.807+0000 7f07b5623700 1 --2- 172.21.15.121:0/2124581300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07b0105a70 0x7f07b019eb40 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f07a0006010 tx=0x7f07a000be20 comp rx=0 tx=0).stop 2024-04-05T22:01:14.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.807+0000 7f07b5623700 1 -- 172.21.15.121:0/2124581300 shutdown_connections 2024-04-05T22:01:14.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.807+0000 7f07b5623700 1 --2- 172.21.15.121:0/2124581300 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f079806cce0 0x7f079806f1a0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.807+0000 7f07b5623700 1 --2- 172.21.15.121:0/2124581300 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f07b00ffa70 0x7f07b019e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.807+0000 7f07b5623700 1 --2- 172.21.15.121:0/2124581300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f07b0105a70 0x7f07b019eb40 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.807+0000 7f07b5623700 1 -- 172.21.15.121:0/2124581300 >> 172.21.15.121:0/2124581300 conn(0x7f07b00fb560 msgr2=0x7f07b0068dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:14.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.807+0000 7f07b5623700 1 -- 172.21.15.121:0/2124581300 shutdown_connections 2024-04-05T22:01:14.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.807+0000 7f07b5623700 1 -- 172.21.15.121:0/2124581300 wait complete. 2024-04-05T22:01:14.810 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T22:01:14.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.910+0000 7fb7a9754700 1 -- 172.21.15.121:0/1392541820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7a4105a40 msgr2=0x7fb7a4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:14.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.910+0000 7fb7a9754700 1 --2- 172.21.15.121:0/1392541820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7a4105a40 0x7fb7a4105e20 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fb798009910 tx=0x7fb798009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:14.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.910+0000 7fb7a9754700 1 -- 172.21.15.121:0/1392541820 shutdown_connections 2024-04-05T22:01:14.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.910+0000 7fb7a9754700 1 --2- 172.21.15.121:0/1392541820 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7a40ffbd0 0x7fb7a4100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.910+0000 7fb7a9754700 1 --2- 172.21.15.121:0/1392541820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7a4105a40 0x7fb7a4105e20 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.910+0000 7fb7a9754700 1 -- 172.21.15.121:0/1392541820 >> 172.21.15.121:0/1392541820 conn(0x7fb7a40755a0 msgr2=0x7fb7a40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:14.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.910+0000 7fb7a9754700 1 -- 172.21.15.121:0/1392541820 shutdown_connections 2024-04-05T22:01:14.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.910+0000 7fb7a9754700 1 -- 172.21.15.121:0/1392541820 wait complete. 2024-04-05T22:01:14.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.910+0000 7fb7a9754700 1 Processor -- start 2024-04-05T22:01:14.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb7a9754700 1 -- start start 2024-04-05T22:01:14.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb7a9754700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7a40ffbd0 0x7fb7a41975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:14.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb7a9754700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7a4105a40 0x7fb7a4197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:14.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb7a9754700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4198100 con 0x7fb7a40ffbd0 2024-04-05T22:01:14.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb7a9754700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4198270 con 0x7fb7a4105a40 2024-04-05T22:01:14.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb7a2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7a40ffbd0 0x7fb7a41975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:14.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb7a2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7a40ffbd0 0x7fb7a41975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42458/0 (socket says 172.21.15.121:42458) 2024-04-05T22:01:14.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb7a2ffd700 1 -- 172.21.15.121:0/2704754421 learned_addr learned my addr 172.21.15.121:0/2704754421 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:14.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb7a27fc700 1 --2- 172.21.15.121:0/2704754421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7a4105a40 0x7fb7a4197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:14.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb7a2ffd700 1 -- 172.21.15.121:0/2704754421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7a4105a40 msgr2=0x7fb7a4197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:14.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb7a2ffd700 1 --2- 172.21.15.121:0/2704754421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7a4105a40 0x7fb7a4197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:14.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb7a2ffd700 1 -- 172.21.15.121:0/2704754421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7980095f0 con 0x7fb7a40ffbd0 2024-04-05T22:01:14.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb7a2ffd700 1 --2- 172.21.15.121:0/2704754421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7a40ffbd0 0x7fb7a41975a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb798006010 tx=0x7fb79800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:14.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.911+0000 7fb78bfff700 1 -- 172.21.15.121:0/2704754421 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb79801d070 con 0x7fb7a40ffbd0 2024-04-05T22:01:14.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.912+0000 7fb78bfff700 1 -- 172.21.15.121:0/2704754421 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb79800f460 con 0x7fb7a40ffbd0 2024-04-05T22:01:14.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.912+0000 7fb7a9754700 1 -- 172.21.15.121:0/2704754421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a41a0610 con 0x7fb7a40ffbd0 2024-04-05T22:01:14.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.912+0000 7fb78bfff700 1 -- 172.21.15.121:0/2704754421 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb798021620 con 0x7fb7a40ffbd0 2024-04-05T22:01:14.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.912+0000 7fb7a9754700 1 -- 172.21.15.121:0/2704754421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a41a0a80 con 0x7fb7a40ffbd0 2024-04-05T22:01:14.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.913+0000 7fb7a9754700 1 -- 172.21.15.121:0/2704754421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a4108040 con 0x7fb7a40ffbd0 2024-04-05T22:01:14.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.913+0000 7fb78bfff700 1 -- 172.21.15.121:0/2704754421 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb79800f5d0 con 0x7fb7a40ffbd0 2024-04-05T22:01:14.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.913+0000 7fb78bfff700 1 --2- 172.21.15.121:0/2704754421 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb78c06cd30 0x7fb78c06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:14.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.914+0000 7fb78bfff700 1 -- 172.21.15.121:0/2704754421 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb79808d8f0 con 0x7fb7a40ffbd0 2024-04-05T22:01:14.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.914+0000 7fb7a27fc700 1 --2- 172.21.15.121:0/2704754421 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb78c06cd30 0x7fb78c06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:14.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.914+0000 7fb7a27fc700 1 --2- 172.21.15.121:0/2704754421 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb78c06cd30 0x7fb78c06f1f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4068ce0 tx=0x7fb794009190 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:14.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:14.918+0000 7fb78bfff700 1 -- 172.21.15.121:0/2704754421 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb7980578c0 con 0x7fb7a40ffbd0 2024-04-05T22:01:15.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.131+0000 7fb7a9754700 1 -- 172.21.15.121:0/2704754421 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7a402cd30 con 0x7fb78c06cd30 2024-04-05T22:01:15.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.133+0000 7fb78bfff700 1 -- 172.21.15.121:0/2704754421 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb7a402cd30 con 0x7fb78c06cd30 2024-04-05T22:01:15.133 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:01:15.133 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T22:01:15.133 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T22:01:15.133 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T22:01:15.133 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T22:01:15.133 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T22:01:15.133 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T22:01:15.133 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T22:01:15.133 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T22:01:15.133 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T22:01:15.134 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T22:01:15.134 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T22:01:15.134 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T22:01:15.134 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T22:01:15.134 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:01:15.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.135+0000 7fb7a9754700 1 -- 172.21.15.121:0/2704754421 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb78c06cd30 msgr2=0x7fb78c06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:15.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.135+0000 7fb7a9754700 1 --2- 172.21.15.121:0/2704754421 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb78c06cd30 0x7fb78c06f1f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4068ce0 tx=0x7fb794009190 comp rx=0 tx=0).stop 2024-04-05T22:01:15.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.135+0000 7fb7a9754700 1 -- 172.21.15.121:0/2704754421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7a40ffbd0 msgr2=0x7fb7a41975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:15.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.135+0000 7fb7a9754700 1 --2- 172.21.15.121:0/2704754421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7a40ffbd0 0x7fb7a41975a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb798006010 tx=0x7fb79800bd70 comp rx=0 tx=0).stop 2024-04-05T22:01:15.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.135+0000 7fb7a9754700 1 -- 172.21.15.121:0/2704754421 shutdown_connections 2024-04-05T22:01:15.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.135+0000 7fb7a9754700 1 --2- 172.21.15.121:0/2704754421 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb78c06cd30 0x7fb78c06f1f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:15.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.135+0000 7fb7a9754700 1 --2- 172.21.15.121:0/2704754421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7a40ffbd0 0x7fb7a41975a0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:15.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.135+0000 7fb7a9754700 1 --2- 172.21.15.121:0/2704754421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb7a4105a40 0x7fb7a4197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:15.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.135+0000 7fb7a9754700 1 -- 172.21.15.121:0/2704754421 >> 172.21.15.121:0/2704754421 conn(0x7fb7a40755a0 msgr2=0x7fb7a40fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:15.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.135+0000 7fb7a9754700 1 -- 172.21.15.121:0/2704754421 shutdown_connections 2024-04-05T22:01:15.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.135+0000 7fb7a9754700 1 -- 172.21.15.121:0/2704754421 wait complete. 2024-04-05T22:01:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.238+0000 7f7eafd01700 1 -- 172.21.15.121:0/2618664753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ea8105a40 msgr2=0x7f7ea8105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:15.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.238+0000 7f7eafd01700 1 --2- 172.21.15.121:0/2618664753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ea8105a40 0x7f7ea8105e20 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c009910 tx=0x7f7e9c009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:15.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.238+0000 7f7eafd01700 1 -- 172.21.15.121:0/2618664753 shutdown_connections 2024-04-05T22:01:15.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.238+0000 7f7eafd01700 1 --2- 172.21.15.121:0/2618664753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ea80ffbd0 0x7f7ea8100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:15.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.238+0000 7f7eafd01700 1 --2- 172.21.15.121:0/2618664753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ea8105a40 0x7f7ea8105e20 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:15.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.238+0000 7f7eafd01700 1 -- 172.21.15.121:0/2618664753 >> 172.21.15.121:0/2618664753 conn(0x7f7ea80755a0 msgr2=0x7f7ea80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:15.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:14 smithi121 ceph-mon[145859]: from='client.35718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:15.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:14 smithi121 ceph-mon[145859]: from='client.35722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:15.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:14 smithi121 ceph-mon[145859]: pgmap v1051: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:15.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:14 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3978094731' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:01:15.240 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:14 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2124581300' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:01:15.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.238+0000 7f7eafd01700 1 -- 172.21.15.121:0/2618664753 shutdown_connections 2024-04-05T22:01:15.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.238+0000 7f7eafd01700 1 -- 172.21.15.121:0/2618664753 wait complete. 2024-04-05T22:01:15.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.238+0000 7f7eafd01700 1 Processor -- start 2024-04-05T22:01:15.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7eafd01700 1 -- start start 2024-04-05T22:01:15.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7eafd01700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ea80ffbd0 0x7f7ea8195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:15.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7eafd01700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ea8105a40 0x7f7ea8195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:15.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7eafd01700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea8196270 con 0x7f7ea80ffbd0 2024-04-05T22:01:15.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7eafd01700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea819a000 con 0x7f7ea8105a40 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7eada9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ea80ffbd0 0x7f7ea8195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7eada9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ea80ffbd0 0x7f7ea8195650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42472/0 (socket says 172.21.15.121:42472) 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7eada9d700 1 -- 172.21.15.121:0/787051753 learned_addr learned my addr 172.21.15.121:0/787051753 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7ead29c700 1 --2- 172.21.15.121:0/787051753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ea8105a40 0x7f7ea8195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7eada9d700 1 -- 172.21.15.121:0/787051753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ea8105a40 msgr2=0x7f7ea8195b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7eada9d700 1 --2- 172.21.15.121:0/787051753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ea8105a40 0x7f7ea8195b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7eada9d700 1 -- 172.21.15.121:0/787051753 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c0095f0 con 0x7f7ea80ffbd0 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7ead29c700 1 --2- 172.21.15.121:0/787051753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ea8105a40 0x7f7ea8195b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7eada9d700 1 --2- 172.21.15.121:0/787051753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ea80ffbd0 0x7f7ea8195650 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00b5c0 tx=0x7f7e9c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.239+0000 7f7e9affd700 1 -- 172.21.15.121:0/787051753 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e9c01f070 con 0x7f7ea80ffbd0 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.240+0000 7f7e9affd700 1 -- 172.21.15.121:0/787051753 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e9c019610 con 0x7f7ea80ffbd0 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.240+0000 7f7eafd01700 1 -- 172.21.15.121:0/787051753 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea819a280 con 0x7f7ea80ffbd0 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.240+0000 7f7e9affd700 1 -- 172.21.15.121:0/787051753 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e9c00f690 con 0x7f7ea80ffbd0 2024-04-05T22:01:15.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.240+0000 7f7eafd01700 1 -- 172.21.15.121:0/787051753 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea819a770 con 0x7f7ea80ffbd0 2024-04-05T22:01:15.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.240+0000 7f7eafd01700 1 -- 172.21.15.121:0/787051753 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea804ea90 con 0x7f7ea80ffbd0 2024-04-05T22:01:15.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.241+0000 7f7e9affd700 1 -- 172.21.15.121:0/787051753 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7e9c019780 con 0x7f7ea80ffbd0 2024-04-05T22:01:15.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.242+0000 7f7e9affd700 1 --2- 172.21.15.121:0/787051753 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7e9406cce0 0x7f7e9406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:15.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.242+0000 7f7e9affd700 1 -- 172.21.15.121:0/787051753 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7e9c08fd90 con 0x7f7ea80ffbd0 2024-04-05T22:01:15.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.242+0000 7f7ead29c700 1 --2- 172.21.15.121:0/787051753 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7e9406cce0 0x7f7e9406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:15.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.242+0000 7f7ead29c700 1 --2- 172.21.15.121:0/787051753 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7e9406cce0 0x7f7e9406f1a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8196c70 tx=0x7f7ea4008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:15.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.245+0000 7f7e9affd700 1 -- 172.21.15.121:0/787051753 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7e9c059e00 con 0x7f7ea80ffbd0 2024-04-05T22:01:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:14 smithi188 ceph-mon[107793]: from='client.35718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:14 smithi188 ceph-mon[107793]: from='client.35722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:14 smithi188 ceph-mon[107793]: pgmap v1051: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:14 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3978094731' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:01:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:14 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2124581300' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:01:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.518+0000 7f7eafd01700 1 -- 172.21.15.121:0/787051753 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7ea81969b0 con 0x7f7ea80ffbd0 2024-04-05T22:01:15.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.519+0000 7f7e9affd700 1 -- 172.21.15.121:0/787051753 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f7e9c028090 con 0x7f7ea80ffbd0 2024-04-05T22:01:15.519 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:01:15.519 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:01:15.519 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:01:15.519 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T22:01:15.519 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:01:15.520 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T22:01:15.520 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:01:15.520 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T22:01:15.520 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:01:15.520 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T22:01:15.520 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:01:15.520 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:01:15.520 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:01:15.520 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:01:15.520 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:01:15.520 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:01:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:01:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:01:15.521 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:01:15.521 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:01:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T22:01:15.521 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:01:15.521 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T22:01:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.521+0000 7f7eafd01700 1 -- 172.21.15.121:0/787051753 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7e9406cce0 msgr2=0x7f7e9406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.521+0000 7f7eafd01700 1 --2- 172.21.15.121:0/787051753 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7e9406cce0 0x7f7e9406f1a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8196c70 tx=0x7f7ea4008040 comp rx=0 tx=0).stop 2024-04-05T22:01:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.521+0000 7f7eafd01700 1 -- 172.21.15.121:0/787051753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ea80ffbd0 msgr2=0x7f7ea8195650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.521+0000 7f7eafd01700 1 --2- 172.21.15.121:0/787051753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ea80ffbd0 0x7f7ea8195650 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c00b5c0 tx=0x7f7e9c00be20 comp rx=0 tx=0).stop 2024-04-05T22:01:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.521+0000 7f7eafd01700 1 -- 172.21.15.121:0/787051753 shutdown_connections 2024-04-05T22:01:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.521+0000 7f7eafd01700 1 --2- 172.21.15.121:0/787051753 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7e9406cce0 0x7f7e9406f1a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.521+0000 7f7eafd01700 1 --2- 172.21.15.121:0/787051753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ea80ffbd0 0x7f7ea8195650 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.521+0000 7f7eafd01700 1 --2- 172.21.15.121:0/787051753 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ea8105a40 0x7f7ea8195b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.521+0000 7f7eafd01700 1 -- 172.21.15.121:0/787051753 >> 172.21.15.121:0/787051753 conn(0x7f7ea80755a0 msgr2=0x7f7ea80fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.521+0000 7f7eafd01700 1 -- 172.21.15.121:0/787051753 shutdown_connections 2024-04-05T22:01:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:15.521+0000 7f7eafd01700 1 -- 172.21.15.121:0/787051753 wait complete. 2024-04-05T22:01:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:15 smithi121 ceph-mon[145859]: from='client.35726 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:01:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:15 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/787051753' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:01:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:15 smithi188 ceph-mon[107793]: from='client.35726 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:01:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:15 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/787051753' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:01:17.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:16 smithi121 ceph-mon[145859]: from='client.35738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:17.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:16 smithi121 ceph-mon[145859]: pgmap v1052: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:17.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:16 smithi188 ceph-mon[107793]: from='client.35738 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:17.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:16 smithi188 ceph-mon[107793]: pgmap v1052: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:19.232 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:18 smithi188 ceph-mon[107793]: pgmap v1053: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:19.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:18 smithi121 ceph-mon[145859]: pgmap v1053: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:21.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:20 smithi121 ceph-mon[145859]: pgmap v1054: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:20 smithi188 ceph-mon[107793]: pgmap v1054: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:23.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:22 smithi121 ceph-mon[145859]: pgmap v1055: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:23.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:22 smithi188 ceph-mon[107793]: pgmap v1055: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:25.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:24 smithi121 ceph-mon[145859]: pgmap v1056: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:24 smithi188 ceph-mon[107793]: pgmap v1056: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:27.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:26 smithi121 ceph-mon[145859]: pgmap v1057: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:27.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:26 smithi188 ceph-mon[107793]: pgmap v1057: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:29.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:28 smithi121 ceph-mon[145859]: pgmap v1058: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:28 smithi188 ceph-mon[107793]: pgmap v1058: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:30 smithi121 ceph-mon[145859]: pgmap v1059: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:31.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:01:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:30 smithi188 ceph-mon[107793]: pgmap v1059: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:01:33.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:32 smithi121 ceph-mon[145859]: pgmap v1060: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:33.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:32 smithi188 ceph-mon[107793]: pgmap v1060: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:35.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:34 smithi121 ceph-mon[145859]: pgmap v1061: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:35.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:34 smithi188 ceph-mon[107793]: pgmap v1061: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:37.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:36 smithi121 ceph-mon[145859]: pgmap v1062: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:36 smithi188 ceph-mon[107793]: pgmap v1062: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:39.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:38 smithi121 ceph-mon[145859]: pgmap v1063: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:38 smithi188 ceph-mon[107793]: pgmap v1063: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:40 smithi121 ceph-mon[145859]: pgmap v1064: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:40 smithi188 ceph-mon[107793]: pgmap v1064: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:43.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:42 smithi121 ceph-mon[145859]: pgmap v1065: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:42 smithi188 ceph-mon[107793]: pgmap v1065: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:45.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:44 smithi121 ceph-mon[145859]: pgmap v1066: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:44 smithi188 ceph-mon[107793]: pgmap v1066: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:45.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.635+0000 7f83ba460700 1 -- 172.21.15.121:0/3513031713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83b4101750 msgr2=0x7f83b41057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:45.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.635+0000 7f83ba460700 1 --2- 172.21.15.121:0/3513031713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83b4101750 0x7f83b41057a0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f839c009910 tx=0x7f839c009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:45.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.635+0000 7f83ba460700 1 -- 172.21.15.121:0/3513031713 shutdown_connections 2024-04-05T22:01:45.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.635+0000 7f83ba460700 1 --2- 172.21.15.121:0/3513031713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83b4101750 0x7f83b41057a0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:45.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.635+0000 7f83ba460700 1 --2- 172.21.15.121:0/3513031713 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83b4100da0 0x7f83b4101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:45.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.635+0000 7f83ba460700 1 -- 172.21.15.121:0/3513031713 >> 172.21.15.121:0/3513031713 conn(0x7f83b40fc630 msgr2=0x7f83b40fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:45.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.635+0000 7f83ba460700 1 -- 172.21.15.121:0/3513031713 shutdown_connections 2024-04-05T22:01:45.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.635+0000 7f83ba460700 1 -- 172.21.15.121:0/3513031713 wait complete. 2024-04-05T22:01:45.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.636+0000 7f83ba460700 1 Processor -- start 2024-04-05T22:01:45.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.636+0000 7f83ba460700 1 -- start start 2024-04-05T22:01:45.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.636+0000 7f83ba460700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83b4100da0 0x7f83b419c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:45.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.636+0000 7f83ba460700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83b4101750 0x7f83b419cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:45.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.636+0000 7f83ba460700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b419d140 con 0x7f83b4100da0 2024-04-05T22:01:45.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.636+0000 7f83ba460700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b41965f0 con 0x7f83b4101750 2024-04-05T22:01:45.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.637+0000 7f83b3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83b4100da0 0x7f83b419c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:45.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.637+0000 7f83b3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83b4100da0 0x7f83b419c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44502/0 (socket says 172.21.15.121:44502) 2024-04-05T22:01:45.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.637+0000 7f83b3fff700 1 -- 172.21.15.121:0/629318678 learned_addr learned my addr 172.21.15.121:0/629318678 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:45.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.637+0000 7f83b37fe700 1 --2- 172.21.15.121:0/629318678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83b4101750 0x7f83b419cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:45.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.637+0000 7f83b3fff700 1 -- 172.21.15.121:0/629318678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83b4101750 msgr2=0x7f83b419cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:45.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.637+0000 7f83b3fff700 1 --2- 172.21.15.121:0/629318678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83b4101750 0x7f83b419cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:45.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.637+0000 7f83b3fff700 1 -- 172.21.15.121:0/629318678 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c0095f0 con 0x7f83b4100da0 2024-04-05T22:01:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.637+0000 7f83b3fff700 1 --2- 172.21.15.121:0/629318678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83b4100da0 0x7f83b419c570 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f83a400b4b0 tx=0x7f83a400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.637+0000 7f83b17fa700 1 -- 172.21.15.121:0/629318678 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83a4011800 con 0x7f83b4100da0 2024-04-05T22:01:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.637+0000 7f83ba460700 1 -- 172.21.15.121:0/629318678 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b41968d0 con 0x7f83b4100da0 2024-04-05T22:01:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.637+0000 7f83b17fa700 1 -- 172.21.15.121:0/629318678 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83a4011e00 con 0x7f83b4100da0 2024-04-05T22:01:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.637+0000 7f83b17fa700 1 -- 172.21.15.121:0/629318678 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83a4010e50 con 0x7f83b4100da0 2024-04-05T22:01:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.637+0000 7f83ba460700 1 -- 172.21.15.121:0/629318678 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b4196e20 con 0x7f83b4100da0 2024-04-05T22:01:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.638+0000 7f83ba460700 1 -- 172.21.15.121:0/629318678 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b41090d0 con 0x7f83b4100da0 2024-04-05T22:01:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.639+0000 7f83b17fa700 1 -- 172.21.15.121:0/629318678 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f83a4010380 con 0x7f83b4100da0 2024-04-05T22:01:45.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.639+0000 7f83b17fa700 1 --2- 172.21.15.121:0/629318678 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f83a006cce0 0x7f83a006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:45.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.639+0000 7f83b37fe700 1 --2- 172.21.15.121:0/629318678 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f83a006cce0 0x7f83a006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:45.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.639+0000 7f83b17fa700 1 -- 172.21.15.121:0/629318678 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f83a408c150 con 0x7f83b4100da0 2024-04-05T22:01:45.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.640+0000 7f83b37fe700 1 --2- 172.21.15.121:0/629318678 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f83a006cce0 0x7f83a006f1a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f839c00b5c0 tx=0x7f839c005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:45.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.644+0000 7f83b17fa700 1 -- 172.21.15.121:0/629318678 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f83a4059c50 con 0x7f83b4100da0 2024-04-05T22:01:45.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.855+0000 7f83ba460700 1 -- 172.21.15.121:0/629318678 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f83b41970f0 con 0x7f83a006cce0 2024-04-05T22:01:45.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.856+0000 7f83b17fa700 1 -- 172.21.15.121:0/629318678 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f83b41970f0 con 0x7f83a006cce0 2024-04-05T22:01:45.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.858+0000 7f83ba460700 1 -- 172.21.15.121:0/629318678 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f83a006cce0 msgr2=0x7f83a006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:45.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.858+0000 7f83ba460700 1 --2- 172.21.15.121:0/629318678 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f83a006cce0 0x7f83a006f1a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f839c00b5c0 tx=0x7f839c005fb0 comp rx=0 tx=0).stop 2024-04-05T22:01:45.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.858+0000 7f83ba460700 1 -- 172.21.15.121:0/629318678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83b4100da0 msgr2=0x7f83b419c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:45.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.858+0000 7f83ba460700 1 --2- 172.21.15.121:0/629318678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83b4100da0 0x7f83b419c570 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f83a400b4b0 tx=0x7f83a400b870 comp rx=0 tx=0).stop 2024-04-05T22:01:45.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.858+0000 7f83ba460700 1 -- 172.21.15.121:0/629318678 shutdown_connections 2024-04-05T22:01:45.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.858+0000 7f83ba460700 1 --2- 172.21.15.121:0/629318678 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f83a006cce0 0x7f83a006f1a0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:45.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.858+0000 7f83ba460700 1 --2- 172.21.15.121:0/629318678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83b4100da0 0x7f83b419c570 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:45.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.858+0000 7f83ba460700 1 --2- 172.21.15.121:0/629318678 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83b4101750 0x7f83b419cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:45.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.858+0000 7f83ba460700 1 -- 172.21.15.121:0/629318678 >> 172.21.15.121:0/629318678 conn(0x7f83b40fc630 msgr2=0x7f83b40fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:45.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.858+0000 7f83ba460700 1 -- 172.21.15.121:0/629318678 shutdown_connections 2024-04-05T22:01:45.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.858+0000 7f83ba460700 1 -- 172.21.15.121:0/629318678 wait complete. 2024-04-05T22:01:45.870 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T22:01:45.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.962+0000 7f8dcd44a700 1 -- 172.21.15.121:0/3133748289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8dc80ffee0 msgr2=0x7f8dc8105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:45.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.962+0000 7f8dcd44a700 1 --2- 172.21.15.121:0/3133748289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8dc80ffee0 0x7f8dc8105870 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009910 tx=0x7f8db8009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:45.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.962+0000 7f8dcd44a700 1 -- 172.21.15.121:0/3133748289 shutdown_connections 2024-04-05T22:01:45.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.962+0000 7f8dcd44a700 1 --2- 172.21.15.121:0/3133748289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8dc80ffee0 0x7f8dc8105870 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:45.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.962+0000 7f8dcd44a700 1 --2- 172.21.15.121:0/3133748289 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dc80ff530 0x7f8dc80ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:45.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.962+0000 7f8dcd44a700 1 -- 172.21.15.121:0/3133748289 >> 172.21.15.121:0/3133748289 conn(0x7f8dc8074df0 msgr2=0x7f8dc8075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:45.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.962+0000 7f8dcd44a700 1 -- 172.21.15.121:0/3133748289 shutdown_connections 2024-04-05T22:01:45.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.962+0000 7f8dcd44a700 1 -- 172.21.15.121:0/3133748289 wait complete. 2024-04-05T22:01:45.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.962+0000 7f8dcd44a700 1 Processor -- start 2024-04-05T22:01:45.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.962+0000 7f8dcd44a700 1 -- start start 2024-04-05T22:01:45.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dcd44a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8dc80ff530 0x7f8dc819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:45.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dcd44a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dc80ffee0 0x7f8dc819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:45.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dcd44a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc819d0c0 con 0x7f8dc80ff530 2024-04-05T22:01:45.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dcd44a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc8196570 con 0x7f8dc80ffee0 2024-04-05T22:01:45.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dc6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8dc80ff530 0x7f8dc819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:45.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dc6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8dc80ff530 0x7f8dc819c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44532/0 (socket says 172.21.15.121:44532) 2024-04-05T22:01:45.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dc6ffd700 1 -- 172.21.15.121:0/1750633983 learned_addr learned my addr 172.21.15.121:0/1750633983 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:45.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dc67fc700 1 --2- 172.21.15.121:0/1750633983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dc80ffee0 0x7f8dc819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:45.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dc6ffd700 1 -- 172.21.15.121:0/1750633983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dc80ffee0 msgr2=0x7f8dc819ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:45.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dc6ffd700 1 --2- 172.21.15.121:0/1750633983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dc80ffee0 0x7f8dc819ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:45.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dc6ffd700 1 -- 172.21.15.121:0/1750633983 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db80095f0 con 0x7f8dc80ff530 2024-04-05T22:01:45.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dc6ffd700 1 --2- 172.21.15.121:0/1750633983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8dc80ff530 0x7f8dc819c4f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00d8d0 tx=0x7f8dbc00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:45.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8daffff700 1 -- 172.21.15.121:0/1750633983 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dbc0098a0 con 0x7f8dc80ff530 2024-04-05T22:01:45.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8daffff700 1 -- 172.21.15.121:0/1750633983 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8dbc010460 con 0x7f8dc80ff530 2024-04-05T22:01:45.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8daffff700 1 -- 172.21.15.121:0/1750633983 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dbc009ea0 con 0x7f8dc80ff530 2024-04-05T22:01:45.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dcd44a700 1 -- 172.21.15.121:0/1750633983 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc8196850 con 0x7f8dc80ff530 2024-04-05T22:01:45.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.963+0000 7f8dcd44a700 1 -- 172.21.15.121:0/1750633983 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc8196d20 con 0x7f8dc80ff530 2024-04-05T22:01:45.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.964+0000 7f8dcd44a700 1 -- 172.21.15.121:0/1750633983 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc8109150 con 0x7f8dc80ff530 2024-04-05T22:01:45.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.965+0000 7f8daffff700 1 -- 172.21.15.121:0/1750633983 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8dbc009a00 con 0x7f8dc80ff530 2024-04-05T22:01:45.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.965+0000 7f8daffff700 1 --2- 172.21.15.121:0/1750633983 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8db006cce0 0x7f8db006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:45.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.965+0000 7f8daffff700 1 -- 172.21.15.121:0/1750633983 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8dbc08c450 con 0x7f8dc80ff530 2024-04-05T22:01:45.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.965+0000 7f8dc67fc700 1 --2- 172.21.15.121:0/1750633983 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8db006cce0 0x7f8db006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:45.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.966+0000 7f8dc67fc700 1 --2- 172.21.15.121:0/1750633983 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8db006cce0 0x7f8db006f1a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8db8006010 tx=0x7f8db801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:45.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:45.970+0000 7f8daffff700 1 -- 172.21.15.121:0/1750633983 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8dbc059f50 con 0x7f8dc80ff530 2024-04-05T22:01:46.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.202+0000 7f8dcd44a700 1 -- 172.21.15.121:0/1750633983 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8dc8196f90 con 0x7f8db006cce0 2024-04-05T22:01:46.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.203+0000 7f8daffff700 1 -- 172.21.15.121:0/1750633983 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8dc8196f90 con 0x7f8db006cce0 2024-04-05T22:01:46.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.205+0000 7f8dcd44a700 1 -- 172.21.15.121:0/1750633983 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8db006cce0 msgr2=0x7f8db006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:46.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.205+0000 7f8dcd44a700 1 --2- 172.21.15.121:0/1750633983 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8db006cce0 0x7f8db006f1a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8db8006010 tx=0x7f8db801a040 comp rx=0 tx=0).stop 2024-04-05T22:01:46.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.205+0000 7f8dcd44a700 1 -- 172.21.15.121:0/1750633983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8dc80ff530 msgr2=0x7f8dc819c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:46.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.205+0000 7f8dcd44a700 1 --2- 172.21.15.121:0/1750633983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8dc80ff530 0x7f8dc819c4f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00d8d0 tx=0x7f8dbc00dbe0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.205+0000 7f8dcd44a700 1 -- 172.21.15.121:0/1750633983 shutdown_connections 2024-04-05T22:01:46.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.205+0000 7f8dcd44a700 1 --2- 172.21.15.121:0/1750633983 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8db006cce0 0x7f8db006f1a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.205+0000 7f8dcd44a700 1 --2- 172.21.15.121:0/1750633983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8dc80ff530 0x7f8dc819c4f0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.205+0000 7f8dcd44a700 1 --2- 172.21.15.121:0/1750633983 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8dc80ffee0 0x7f8dc819ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.206+0000 7f8dcd44a700 1 -- 172.21.15.121:0/1750633983 >> 172.21.15.121:0/1750633983 conn(0x7f8dc8074df0 msgr2=0x7f8dc81050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:46.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.206+0000 7f8dcd44a700 1 -- 172.21.15.121:0/1750633983 shutdown_connections 2024-04-05T22:01:46.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.206+0000 7f8dcd44a700 1 -- 172.21.15.121:0/1750633983 wait complete. 2024-04-05T22:01:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:01:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.308+0000 7ff24787e700 1 -- 172.21.15.121:0/3346966742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff240105a40 msgr2=0x7ff240105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.308+0000 7ff24787e700 1 --2- 172.21.15.121:0/3346966742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff240105a40 0x7ff240105e20 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff230009910 tx=0x7ff230009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.308+0000 7ff24787e700 1 -- 172.21.15.121:0/3346966742 shutdown_connections 2024-04-05T22:01:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.308+0000 7ff24787e700 1 --2- 172.21.15.121:0/3346966742 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff2400ffbd0 0x7ff240100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.308+0000 7ff24787e700 1 --2- 172.21.15.121:0/3346966742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff240105a40 0x7ff240105e20 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.308+0000 7ff24787e700 1 -- 172.21.15.121:0/3346966742 >> 172.21.15.121:0/3346966742 conn(0x7ff2400755a0 msgr2=0x7ff2400759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.308+0000 7ff24787e700 1 -- 172.21.15.121:0/3346966742 shutdown_connections 2024-04-05T22:01:46.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.308+0000 7ff24787e700 1 -- 172.21.15.121:0/3346966742 wait complete. 2024-04-05T22:01:46.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.308+0000 7ff24787e700 1 Processor -- start 2024-04-05T22:01:46.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff24787e700 1 -- start start 2024-04-05T22:01:46.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff24787e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2400ffbd0 0x7ff2401974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:46.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff24787e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240105a40 0x7ff240197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:46.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff24787e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff240198020 con 0x7ff2400ffbd0 2024-04-05T22:01:46.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff24787e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff240198160 con 0x7ff240105a40 2024-04-05T22:01:46.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff24561a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2400ffbd0 0x7ff2401974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:46.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff24561a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2400ffbd0 0x7ff2401974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44540/0 (socket says 172.21.15.121:44540) 2024-04-05T22:01:46.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff24561a700 1 -- 172.21.15.121:0/2115934795 learned_addr learned my addr 172.21.15.121:0/2115934795 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:46.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff244e19700 1 --2- 172.21.15.121:0/2115934795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240105a40 0x7ff240197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:46.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff24561a700 1 -- 172.21.15.121:0/2115934795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240105a40 msgr2=0x7ff240197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:46.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff24561a700 1 --2- 172.21.15.121:0/2115934795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240105a40 0x7ff240197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff24561a700 1 -- 172.21.15.121:0/2115934795 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2300095f0 con 0x7ff2400ffbd0 2024-04-05T22:01:46.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff24561a700 1 --2- 172.21.15.121:0/2115934795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2400ffbd0 0x7ff2401974c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff230006010 tx=0x7ff23000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:46.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff2367fc700 1 -- 172.21.15.121:0/2115934795 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff23001d070 con 0x7ff2400ffbd0 2024-04-05T22:01:46.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff24787e700 1 -- 172.21.15.121:0/2115934795 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2401a0530 con 0x7ff2400ffbd0 2024-04-05T22:01:46.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff2367fc700 1 -- 172.21.15.121:0/2115934795 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff23000f460 con 0x7ff2400ffbd0 2024-04-05T22:01:46.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.309+0000 7ff2367fc700 1 -- 172.21.15.121:0/2115934795 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff230021620 con 0x7ff2400ffbd0 2024-04-05T22:01:46.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.310+0000 7ff24787e700 1 -- 172.21.15.121:0/2115934795 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2401a0940 con 0x7ff2400ffbd0 2024-04-05T22:01:46.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.310+0000 7ff24787e700 1 -- 172.21.15.121:0/2115934795 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff240108040 con 0x7ff2400ffbd0 2024-04-05T22:01:46.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.311+0000 7ff2367fc700 1 -- 172.21.15.121:0/2115934795 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff23000f5d0 con 0x7ff2400ffbd0 2024-04-05T22:01:46.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.311+0000 7ff2367fc700 1 --2- 172.21.15.121:0/2115934795 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff22c06cce0 0x7ff22c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:46.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.311+0000 7ff2367fc700 1 -- 172.21.15.121:0/2115934795 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff23008d850 con 0x7ff2400ffbd0 2024-04-05T22:01:46.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.311+0000 7ff244e19700 1 --2- 172.21.15.121:0/2115934795 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff22c06cce0 0x7ff22c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:46.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.312+0000 7ff244e19700 1 --2- 172.21.15.121:0/2115934795 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff22c06cce0 0x7ff22c06f1a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff240068cc0 tx=0x7ff23c00a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:46.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.316+0000 7ff2367fc700 1 -- 172.21.15.121:0/2115934795 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff2300578c0 con 0x7ff2400ffbd0 2024-04-05T22:01:46.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:01:46.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.517+0000 7ff24787e700 1 -- 172.21.15.121:0/2115934795 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff24002cd30 con 0x7ff22c06cce0 2024-04-05T22:01:46.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.527+0000 7ff2367fc700 1 -- 172.21.15.121:0/2115934795 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff24002cd30 con 0x7ff22c06cce0 2024-04-05T22:01:46.529 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T22:01:46.529 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (32m) 11m ago 44m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T22:01:46.529 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (44m) 11m ago 44m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T22:01:46.529 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (43m) 11m ago 43m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T22:01:46.529 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (30m) 11m ago 44m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T22:01:46.529 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (30m) 11m ago 43m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T22:01:46.529 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (32m) 11m ago 43m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T22:01:46.529 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (27m) 11m ago 37m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T22:01:46.529 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (26m) 11m ago 37m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T22:01:46.529 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (26m) 11m ago 37m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T22:01:46.529 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (11m) 11m ago 37m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T22:01:46.529 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (34m) 11m ago 45m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T22:01:46.529 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (33m) 11m ago 42m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T22:01:46.530 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (31m) 11m ago 45m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T22:01:46.530 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (31m) 11m ago 42m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T22:01:46.530 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (33m) 11m ago 44m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T22:01:46.530 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (33m) 11m ago 43m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T22:01:46.530 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (30m) 11m ago 42m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T22:01:46.530 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (29m) 11m ago 41m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T22:01:46.530 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (29m) 11m ago 41m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T22:01:46.530 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (28m) 11m ago 40m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T22:01:46.530 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (27m) 11m ago 40m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T22:01:46.530 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (27m) 11m ago 39m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T22:01:46.530 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (32m) 11m ago 43m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T22:01:46.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.530+0000 7ff24787e700 1 -- 172.21.15.121:0/2115934795 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff22c06cce0 msgr2=0x7ff22c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:46.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.530+0000 7ff24787e700 1 --2- 172.21.15.121:0/2115934795 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff22c06cce0 0x7ff22c06f1a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff240068cc0 tx=0x7ff23c00a300 comp rx=0 tx=0).stop 2024-04-05T22:01:46.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.530+0000 7ff24787e700 1 -- 172.21.15.121:0/2115934795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2400ffbd0 msgr2=0x7ff2401974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:46.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.530+0000 7ff24787e700 1 --2- 172.21.15.121:0/2115934795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2400ffbd0 0x7ff2401974c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff230006010 tx=0x7ff23000bd70 comp rx=0 tx=0).stop 2024-04-05T22:01:46.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.530+0000 7ff24787e700 1 -- 172.21.15.121:0/2115934795 shutdown_connections 2024-04-05T22:01:46.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.530+0000 7ff24787e700 1 --2- 172.21.15.121:0/2115934795 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff22c06cce0 0x7ff22c06f1a0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.530+0000 7ff24787e700 1 --2- 172.21.15.121:0/2115934795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2400ffbd0 0x7ff2401974c0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.530+0000 7ff24787e700 1 --2- 172.21.15.121:0/2115934795 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff240105a40 0x7ff240197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.530+0000 7ff24787e700 1 -- 172.21.15.121:0/2115934795 >> 172.21.15.121:0/2115934795 conn(0x7ff2400755a0 msgr2=0x7ff2400fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:46.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.530+0000 7ff24787e700 1 -- 172.21.15.121:0/2115934795 shutdown_connections 2024-04-05T22:01:46.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.530+0000 7ff24787e700 1 -- 172.21.15.121:0/2115934795 wait complete. 2024-04-05T22:01:46.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.631+0000 7f27725e5700 1 -- 172.21.15.121:0/737263462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f276c105a40 msgr2=0x7f276c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:46.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.631+0000 7f27725e5700 1 --2- 172.21.15.121:0/737263462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f276c105a40 0x7f276c105e20 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f275c009910 tx=0x7f275c009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:46.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.631+0000 7f27725e5700 1 -- 172.21.15.121:0/737263462 shutdown_connections 2024-04-05T22:01:46.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.631+0000 7f27725e5700 1 --2- 172.21.15.121:0/737263462 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f276c0ffbd0 0x7f276c100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.631+0000 7f27725e5700 1 --2- 172.21.15.121:0/737263462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f276c105a40 0x7f276c105e20 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.631+0000 7f27725e5700 1 -- 172.21.15.121:0/737263462 >> 172.21.15.121:0/737263462 conn(0x7f276c0755a0 msgr2=0x7f276c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:46.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.631+0000 7f27725e5700 1 -- 172.21.15.121:0/737263462 shutdown_connections 2024-04-05T22:01:46.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.631+0000 7f27725e5700 1 -- 172.21.15.121:0/737263462 wait complete. 2024-04-05T22:01:46.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.632+0000 7f27725e5700 1 Processor -- start 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.632+0000 7f27725e5700 1 -- start start 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.632+0000 7f27725e5700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f276c0ffbd0 0x7f276c195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.632+0000 7f27725e5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f276c105a40 0x7f276c195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.632+0000 7f27725e5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276c196170 con 0x7f276c105a40 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.632+0000 7f27725e5700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276c199f00 con 0x7f276c0ffbd0 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.632+0000 7f276b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f276c105a40 0x7f276c195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.632+0000 7f276b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f276c105a40 0x7f276c195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44556/0 (socket says 172.21.15.121:44556) 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.632+0000 7f276b7fe700 1 -- 172.21.15.121:0/983329292 learned_addr learned my addr 172.21.15.121:0/983329292 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.632+0000 7f276bfff700 1 --2- 172.21.15.121:0/983329292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f276c0ffbd0 0x7f276c195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.633+0000 7f276b7fe700 1 -- 172.21.15.121:0/983329292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f276c0ffbd0 msgr2=0x7f276c195550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.633+0000 7f276b7fe700 1 --2- 172.21.15.121:0/983329292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f276c0ffbd0 0x7f276c195550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.633+0000 7f276b7fe700 1 -- 172.21.15.121:0/983329292 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275c0095f0 con 0x7f276c105a40 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.633+0000 7f276b7fe700 1 --2- 172.21.15.121:0/983329292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f276c105a40 0x7f276c195a90 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f275400d8d0 tx=0x7f275400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:46.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.633+0000 7f27697fa700 1 -- 172.21.15.121:0/983329292 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27540098a0 con 0x7f276c105a40 2024-04-05T22:01:46.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.633+0000 7f27725e5700 1 -- 172.21.15.121:0/983329292 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276c19a1e0 con 0x7f276c105a40 2024-04-05T22:01:46.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.633+0000 7f27697fa700 1 -- 172.21.15.121:0/983329292 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2754010460 con 0x7f276c105a40 2024-04-05T22:01:46.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.633+0000 7f27697fa700 1 -- 172.21.15.121:0/983329292 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2754009ea0 con 0x7f276c105a40 2024-04-05T22:01:46.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.633+0000 7f27725e5700 1 -- 172.21.15.121:0/983329292 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f276c19a730 con 0x7f276c105a40 2024-04-05T22:01:46.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.634+0000 7f27725e5700 1 -- 172.21.15.121:0/983329292 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f276c108040 con 0x7f276c105a40 2024-04-05T22:01:46.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.634+0000 7f27697fa700 1 -- 172.21.15.121:0/983329292 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f27540105d0 con 0x7f276c105a40 2024-04-05T22:01:46.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.635+0000 7f27697fa700 1 --2- 172.21.15.121:0/983329292 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f275806cce0 0x7f275806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:46.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.635+0000 7f27697fa700 1 -- 172.21.15.121:0/983329292 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f275408c480 con 0x7f276c105a40 2024-04-05T22:01:46.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.635+0000 7f276bfff700 1 --2- 172.21.15.121:0/983329292 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f275806cce0 0x7f275806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:46.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.635+0000 7f276bfff700 1 --2- 172.21.15.121:0/983329292 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f275806cce0 0x7f275806f1a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f275c006010 tx=0x7f275c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:46.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.639+0000 7f27697fa700 1 -- 172.21.15.121:0/983329292 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f27540564f0 con 0x7f276c105a40 2024-04-05T22:01:46.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.912+0000 7f27725e5700 1 -- 172.21.15.121:0/983329292 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f276c04ea90 con 0x7f276c105a40 2024-04-05T22:01:46.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.912+0000 7f27697fa700 1 -- 172.21.15.121:0/983329292 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f2754010880 con 0x7f276c105a40 2024-04-05T22:01:46.914 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:01:46.914 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T22:01:46.914 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:01:46.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:01:46.914 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T22:01:46.914 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:01:46.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:01:46.914 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T22:01:46.914 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T22:01:46.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:01:46.914 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T22:01:46.914 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T22:01:46.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:01:46.915 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T22:01:46.915 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T22:01:46.915 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T22:01:46.915 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:01:46.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.914+0000 7f27725e5700 1 -- 172.21.15.121:0/983329292 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f275806cce0 msgr2=0x7f275806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:46.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.914+0000 7f27725e5700 1 --2- 172.21.15.121:0/983329292 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f275806cce0 0x7f275806f1a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f275c006010 tx=0x7f275c01a040 comp rx=0 tx=0).stop 2024-04-05T22:01:46.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.914+0000 7f27725e5700 1 -- 172.21.15.121:0/983329292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f276c105a40 msgr2=0x7f276c195a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:46.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.914+0000 7f27725e5700 1 --2- 172.21.15.121:0/983329292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f276c105a40 0x7f276c195a90 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f275400d8d0 tx=0x7f275400dbe0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.914+0000 7f27725e5700 1 -- 172.21.15.121:0/983329292 shutdown_connections 2024-04-05T22:01:46.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.914+0000 7f27725e5700 1 --2- 172.21.15.121:0/983329292 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f275806cce0 0x7f275806f1a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.914+0000 7f27725e5700 1 --2- 172.21.15.121:0/983329292 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f276c0ffbd0 0x7f276c195550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.914+0000 7f27725e5700 1 --2- 172.21.15.121:0/983329292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f276c105a40 0x7f276c195a90 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:46.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.914+0000 7f27725e5700 1 -- 172.21.15.121:0/983329292 >> 172.21.15.121:0/983329292 conn(0x7f276c0755a0 msgr2=0x7f276c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:46.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.914+0000 7f27725e5700 1 -- 172.21.15.121:0/983329292 shutdown_connections 2024-04-05T22:01:46.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:46.914+0000 7f27725e5700 1 -- 172.21.15.121:0/983329292 wait complete. 2024-04-05T22:01:47.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.018+0000 7fd8013a7700 1 -- 172.21.15.121:0/2087941005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7fc101c70 msgr2=0x7fd7fc102050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:47.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.018+0000 7fd8013a7700 1 --2- 172.21.15.121:0/2087941005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7fc101c70 0x7fd7fc102050 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0009910 tx=0x7fd7f0009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:47.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.019+0000 7fd8013a7700 1 -- 172.21.15.121:0/2087941005 shutdown_connections 2024-04-05T22:01:47.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.019+0000 7fd8013a7700 1 --2- 172.21.15.121:0/2087941005 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7fc102620 0x7fd7fc10ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.019+0000 7fd8013a7700 1 --2- 172.21.15.121:0/2087941005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7fc101c70 0x7fd7fc102050 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.019+0000 7fd8013a7700 1 -- 172.21.15.121:0/2087941005 >> 172.21.15.121:0/2087941005 conn(0x7fd7fc0fb550 msgr2=0x7fd7fc0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:47.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.019+0000 7fd8013a7700 1 -- 172.21.15.121:0/2087941005 shutdown_connections 2024-04-05T22:01:47.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.019+0000 7fd8013a7700 1 -- 172.21.15.121:0/2087941005 wait complete. 2024-04-05T22:01:47.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.019+0000 7fd8013a7700 1 Processor -- start 2024-04-05T22:01:47.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.019+0000 7fd8013a7700 1 -- start start 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd8013a7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7fc101c70 0x7fd7fc0ff960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd8013a7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7fc102620 0x7fd7fc0ffea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd8013a7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc1003e0 con 0x7fd7fc101c70 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd8013a7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc100520 con 0x7fd7fc102620 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd7faffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7fc101c70 0x7fd7fc0ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd7faffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7fc101c70 0x7fd7fc0ff960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44572/0 (socket says 172.21.15.121:44572) 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd7faffd700 1 -- 172.21.15.121:0/2700449257 learned_addr learned my addr 172.21.15.121:0/2700449257 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd7fa7fc700 1 --2- 172.21.15.121:0/2700449257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7fc102620 0x7fd7fc0ffea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd7faffd700 1 -- 172.21.15.121:0/2700449257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7fc102620 msgr2=0x7fd7fc0ffea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd7faffd700 1 --2- 172.21.15.121:0/2700449257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7fc102620 0x7fd7fc0ffea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd7faffd700 1 -- 172.21.15.121:0/2700449257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f00095f0 con 0x7fd7fc101c70 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd7faffd700 1 --2- 172.21.15.121:0/2700449257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7fc101c70 0x7fd7fc0ff960 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0006010 tx=0x7fd7f000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd7e3fff700 1 -- 172.21.15.121:0/2700449257 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7f001d070 con 0x7fd7fc101c70 2024-04-05T22:01:47.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd7e3fff700 1 -- 172.21.15.121:0/2700449257 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7f0005710 con 0x7fd7fc101c70 2024-04-05T22:01:47.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd8013a7700 1 -- 172.21.15.121:0/2700449257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7fc1007a0 con 0x7fd7fc101c70 2024-04-05T22:01:47.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd7e3fff700 1 -- 172.21.15.121:0/2700449257 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7f000fe20 con 0x7fd7fc101c70 2024-04-05T22:01:47.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.020+0000 7fd8013a7700 1 -- 172.21.15.121:0/2700449257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7fc1a2ba0 con 0x7fd7fc101c70 2024-04-05T22:01:47.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.021+0000 7fd8013a7700 1 -- 172.21.15.121:0/2700449257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7fc108210 con 0x7fd7fc101c70 2024-04-05T22:01:47.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.022+0000 7fd7e3fff700 1 -- 172.21.15.121:0/2700449257 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd7f0003a60 con 0x7fd7fc101c70 2024-04-05T22:01:47.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.022+0000 7fd7e3fff700 1 --2- 172.21.15.121:0/2700449257 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7e406cce0 0x7fd7e406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:47.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.022+0000 7fd7e3fff700 1 -- 172.21.15.121:0/2700449257 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd7f008d790 con 0x7fd7fc101c70 2024-04-05T22:01:47.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.022+0000 7fd7fa7fc700 1 --2- 172.21.15.121:0/2700449257 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7e406cce0 0x7fd7e406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:47.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.023+0000 7fd7fa7fc700 1 --2- 172.21.15.121:0/2700449257 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7e406cce0 0x7fd7e406f1a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc101150 tx=0x7fd7ec00b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:47.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.027+0000 7fd7e3fff700 1 -- 172.21.15.121:0/2700449257 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd7f0057800 con 0x7fd7fc101c70 2024-04-05T22:01:47.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.258+0000 7fd8013a7700 1 -- 172.21.15.121:0/2700449257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd7fc04ea90 con 0x7fd7fc101c70 2024-04-05T22:01:47.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.258+0000 7fd7e3fff700 1 -- 172.21.15.121:0/2700449257 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fd7f0003d10 con 0x7fd7fc101c70 2024-04-05T22:01:47.259 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T22:01:47.259 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T22:01:47.260 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T22:01:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T22:01:47.262 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T22:01:47.262 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:01:47.262 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:01:47.262 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T22:01:47.262 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:01:47.262 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:01:47.262 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:01:47.262 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:01:47.262 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:01:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.260+0000 7fd8013a7700 1 -- 172.21.15.121:0/2700449257 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7e406cce0 msgr2=0x7fd7e406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.260+0000 7fd8013a7700 1 --2- 172.21.15.121:0/2700449257 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7e406cce0 0x7fd7e406f1a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc101150 tx=0x7fd7ec00b410 comp rx=0 tx=0).stop 2024-04-05T22:01:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.260+0000 7fd8013a7700 1 -- 172.21.15.121:0/2700449257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7fc101c70 msgr2=0x7fd7fc0ff960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.260+0000 7fd8013a7700 1 --2- 172.21.15.121:0/2700449257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7fc101c70 0x7fd7fc0ff960 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0006010 tx=0x7fd7f000bd70 comp rx=0 tx=0).stop 2024-04-05T22:01:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.260+0000 7fd8013a7700 1 -- 172.21.15.121:0/2700449257 shutdown_connections 2024-04-05T22:01:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.260+0000 7fd8013a7700 1 --2- 172.21.15.121:0/2700449257 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7e406cce0 0x7fd7e406f1a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.260+0000 7fd8013a7700 1 --2- 172.21.15.121:0/2700449257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7fc101c70 0x7fd7fc0ff960 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.260+0000 7fd8013a7700 1 --2- 172.21.15.121:0/2700449257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7fc102620 0x7fd7fc0ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.260+0000 7fd8013a7700 1 -- 172.21.15.121:0/2700449257 >> 172.21.15.121:0/2700449257 conn(0x7fd7fc0fb550 msgr2=0x7fd7fc105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.260+0000 7fd8013a7700 1 -- 172.21.15.121:0/2700449257 shutdown_connections 2024-04-05T22:01:47.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.260+0000 7fd8013a7700 1 -- 172.21.15.121:0/2700449257 wait complete. 2024-04-05T22:01:47.264 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T22:01:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:46 smithi188 ceph-mon[107793]: from='client.35746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:46 smithi188 ceph-mon[107793]: pgmap v1067: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:46 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/983329292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:01:47.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.365+0000 7fd58e36e700 1 -- 172.21.15.121:0/1675199584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5880ffbd0 msgr2=0x7fd588100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:47.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.365+0000 7fd58e36e700 1 --2- 172.21.15.121:0/1675199584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5880ffbd0 0x7fd588100050 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fd570009910 tx=0x7fd570009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:47.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.365+0000 7fd58e36e700 1 -- 172.21.15.121:0/1675199584 shutdown_connections 2024-04-05T22:01:47.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.365+0000 7fd58e36e700 1 --2- 172.21.15.121:0/1675199584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5880ffbd0 0x7fd588100050 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.365+0000 7fd58e36e700 1 --2- 172.21.15.121:0/1675199584 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd588105a40 0x7fd588105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.365+0000 7fd58e36e700 1 -- 172.21.15.121:0/1675199584 >> 172.21.15.121:0/1675199584 conn(0x7fd5880755a0 msgr2=0x7fd5880759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:47.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.365+0000 7fd58e36e700 1 -- 172.21.15.121:0/1675199584 shutdown_connections 2024-04-05T22:01:47.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.365+0000 7fd58e36e700 1 -- 172.21.15.121:0/1675199584 wait complete. 2024-04-05T22:01:47.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.365+0000 7fd58e36e700 1 Processor -- start 2024-04-05T22:01:47.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.366+0000 7fd58e36e700 1 -- start start 2024-04-05T22:01:47.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.366+0000 7fd58e36e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5880ffbd0 0x7fd5881956b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:47.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.366+0000 7fd58e36e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd588105a40 0x7fd588195bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:47.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.366+0000 7fd58e36e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd588196280 con 0x7fd588105a40 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.366+0000 7fd58e36e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd588198fb0 con 0x7fd5880ffbd0 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.366+0000 7fd5877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd588105a40 0x7fd588195bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.366+0000 7fd5877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd588105a40 0x7fd588195bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44588/0 (socket says 172.21.15.121:44588) 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.366+0000 7fd5877fe700 1 -- 172.21.15.121:0/4076127952 learned_addr learned my addr 172.21.15.121:0/4076127952 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.366+0000 7fd587fff700 1 --2- 172.21.15.121:0/4076127952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5880ffbd0 0x7fd5881956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.366+0000 7fd5877fe700 1 -- 172.21.15.121:0/4076127952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5880ffbd0 msgr2=0x7fd5881956b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.366+0000 7fd5877fe700 1 --2- 172.21.15.121:0/4076127952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5880ffbd0 0x7fd5881956b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.366+0000 7fd5877fe700 1 -- 172.21.15.121:0/4076127952 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5700095f0 con 0x7fd588105a40 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.367+0000 7fd5877fe700 1 --2- 172.21.15.121:0/4076127952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd588105a40 0x7fd588195bf0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fd570006010 tx=0x7fd57000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.367+0000 7fd5857fa700 1 -- 172.21.15.121:0/4076127952 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd57001d070 con 0x7fd588105a40 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.367+0000 7fd5857fa700 1 -- 172.21.15.121:0/4076127952 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd57000f460 con 0x7fd588105a40 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.367+0000 7fd5857fa700 1 -- 172.21.15.121:0/4076127952 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd570021620 con 0x7fd588105a40 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.367+0000 7fd58e36e700 1 -- 172.21.15.121:0/4076127952 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd588199230 con 0x7fd588105a40 2024-04-05T22:01:47.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.367+0000 7fd58e36e700 1 -- 172.21.15.121:0/4076127952 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd588199720 con 0x7fd588105a40 2024-04-05T22:01:47.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.368+0000 7fd58e36e700 1 -- 172.21.15.121:0/4076127952 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd588108060 con 0x7fd588105a40 2024-04-05T22:01:47.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.368+0000 7fd5857fa700 1 -- 172.21.15.121:0/4076127952 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd57000f5d0 con 0x7fd588105a40 2024-04-05T22:01:47.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.368+0000 7fd5857fa700 1 --2- 172.21.15.121:0/4076127952 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd57406cd30 0x7fd57406f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:47.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.369+0000 7fd5857fa700 1 -- 172.21.15.121:0/4076127952 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd57008d8f0 con 0x7fd588105a40 2024-04-05T22:01:47.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.369+0000 7fd587fff700 1 --2- 172.21.15.121:0/4076127952 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd57406cd30 0x7fd57406f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:47.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.369+0000 7fd587fff700 1 --2- 172.21.15.121:0/4076127952 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd57406cd30 0x7fd57406f1f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd578009a10 tx=0x7fd578009190 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:47.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.373+0000 7fd5857fa700 1 -- 172.21.15.121:0/4076127952 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd5700578c0 con 0x7fd588105a40 2024-04-05T22:01:47.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:46 smithi121 ceph-mon[145859]: from='client.35746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:47.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:46 smithi121 ceph-mon[145859]: pgmap v1067: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:47.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:46 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/983329292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:01:47.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.584+0000 7fd58e36e700 1 -- 172.21.15.121:0/4076127952 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd588199940 con 0x7fd57406cd30 2024-04-05T22:01:47.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.585+0000 7fd5857fa700 1 -- 172.21.15.121:0/4076127952 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd588199940 con 0x7fd57406cd30 2024-04-05T22:01:47.587 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:01:47.587 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T22:01:47.587 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T22:01:47.587 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T22:01:47.587 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T22:01:47.587 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T22:01:47.587 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T22:01:47.587 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T22:01:47.587 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T22:01:47.587 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T22:01:47.588 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T22:01:47.588 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T22:01:47.588 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T22:01:47.588 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T22:01:47.588 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:01:47.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.587+0000 7fd58e36e700 1 -- 172.21.15.121:0/4076127952 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd57406cd30 msgr2=0x7fd57406f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:47.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.587+0000 7fd58e36e700 1 --2- 172.21.15.121:0/4076127952 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd57406cd30 0x7fd57406f1f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd578009a10 tx=0x7fd578009190 comp rx=0 tx=0).stop 2024-04-05T22:01:47.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.587+0000 7fd58e36e700 1 -- 172.21.15.121:0/4076127952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd588105a40 msgr2=0x7fd588195bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:47.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.587+0000 7fd58e36e700 1 --2- 172.21.15.121:0/4076127952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd588105a40 0x7fd588195bf0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fd570006010 tx=0x7fd57000bd70 comp rx=0 tx=0).stop 2024-04-05T22:01:47.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.588+0000 7fd58e36e700 1 -- 172.21.15.121:0/4076127952 shutdown_connections 2024-04-05T22:01:47.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.588+0000 7fd58e36e700 1 --2- 172.21.15.121:0/4076127952 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd57406cd30 0x7fd57406f1f0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.588+0000 7fd58e36e700 1 --2- 172.21.15.121:0/4076127952 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd5880ffbd0 0x7fd5881956b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.588+0000 7fd58e36e700 1 --2- 172.21.15.121:0/4076127952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd588105a40 0x7fd588195bf0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.588+0000 7fd58e36e700 1 -- 172.21.15.121:0/4076127952 >> 172.21.15.121:0/4076127952 conn(0x7fd5880755a0 msgr2=0x7fd5880fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:47.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.588+0000 7fd58e36e700 1 -- 172.21.15.121:0/4076127952 shutdown_connections 2024-04-05T22:01:47.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.588+0000 7fd58e36e700 1 -- 172.21.15.121:0/4076127952 wait complete. 2024-04-05T22:01:47.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.689+0000 7fc07d197700 1 -- 172.21.15.121:0/1117231700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc078101790 msgr2=0x7fc0781057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:47.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.689+0000 7fc07d197700 1 --2- 172.21.15.121:0/1117231700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc078101790 0x7fc0781057e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fc06c009910 tx=0x7fc06c009c20 comp rx=0 tx=0).stop 2024-04-05T22:01:47.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.689+0000 7fc07d197700 1 -- 172.21.15.121:0/1117231700 shutdown_connections 2024-04-05T22:01:47.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.689+0000 7fc07d197700 1 --2- 172.21.15.121:0/1117231700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc078101790 0x7fc0781057e0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.689+0000 7fc07d197700 1 --2- 172.21.15.121:0/1117231700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc078100de0 0x7fc0781011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.689+0000 7fc07d197700 1 -- 172.21.15.121:0/1117231700 >> 172.21.15.121:0/1117231700 conn(0x7fc0780fc670 msgr2=0x7fc0780fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:47.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.689+0000 7fc07d197700 1 -- 172.21.15.121:0/1117231700 shutdown_connections 2024-04-05T22:01:47.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.689+0000 7fc07d197700 1 -- 172.21.15.121:0/1117231700 wait complete. 2024-04-05T22:01:47.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.690+0000 7fc07d197700 1 Processor -- start 2024-04-05T22:01:47.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.690+0000 7fc07d197700 1 -- start start 2024-04-05T22:01:47.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.690+0000 7fc07d197700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc078100de0 0x7fc0781966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:47.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.690+0000 7fc07d197700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc078101790 0x7fc078196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:47.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.690+0000 7fc07d197700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0781972a0 con 0x7fc078101790 2024-04-05T22:01:47.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.690+0000 7fc07d197700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc078199fd0 con 0x7fc078100de0 2024-04-05T22:01:47.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.691+0000 7fc07659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc078101790 0x7fc078196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:47.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.691+0000 7fc07659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc078101790 0x7fc078196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44600/0 (socket says 172.21.15.121:44600) 2024-04-05T22:01:47.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.691+0000 7fc07659c700 1 -- 172.21.15.121:0/1491352731 learned_addr learned my addr 172.21.15.121:0/1491352731 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:01:47.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.691+0000 7fc076d9d700 1 --2- 172.21.15.121:0/1491352731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc078100de0 0x7fc0781966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:47.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.691+0000 7fc07659c700 1 -- 172.21.15.121:0/1491352731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc078100de0 msgr2=0x7fc0781966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:47.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.691+0000 7fc07659c700 1 --2- 172.21.15.121:0/1491352731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc078100de0 0x7fc0781966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.691+0000 7fc07659c700 1 -- 172.21.15.121:0/1491352731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06c0095f0 con 0x7fc078101790 2024-04-05T22:01:47.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.691+0000 7fc07659c700 1 --2- 172.21.15.121:0/1491352731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc078101790 0x7fc078196c10 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0092e0 tx=0x7fc06c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:47.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.691+0000 7fc05ffff700 1 -- 172.21.15.121:0/1491352731 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc06c01d070 con 0x7fc078101790 2024-04-05T22:01:47.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.691+0000 7fc05ffff700 1 -- 172.21.15.121:0/1491352731 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc06c005940 con 0x7fc078101790 2024-04-05T22:01:47.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.691+0000 7fc07d197700 1 -- 172.21.15.121:0/1491352731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07819a250 con 0x7fc078101790 2024-04-05T22:01:47.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.691+0000 7fc05ffff700 1 -- 172.21.15.121:0/1491352731 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc06c00fe50 con 0x7fc078101790 2024-04-05T22:01:47.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.691+0000 7fc07d197700 1 -- 172.21.15.121:0/1491352731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07819a740 con 0x7fc078101790 2024-04-05T22:01:47.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.692+0000 7fc07d197700 1 -- 172.21.15.121:0/1491352731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0781091e0 con 0x7fc078101790 2024-04-05T22:01:47.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.693+0000 7fc05ffff700 1 -- 172.21.15.121:0/1491352731 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc06c003a60 con 0x7fc078101790 2024-04-05T22:01:47.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.693+0000 7fc05ffff700 1 --2- 172.21.15.121:0/1491352731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc06006cce0 0x7fc06006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:01:47.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.693+0000 7fc076d9d700 1 --2- 172.21.15.121:0/1491352731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc06006cce0 0x7fc06006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:01:47.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.693+0000 7fc05ffff700 1 -- 172.21.15.121:0/1491352731 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc06c08d7e0 con 0x7fc078101790 2024-04-05T22:01:47.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.694+0000 7fc076d9d700 1 --2- 172.21.15.121:0/1491352731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc06006cce0 0x7fc06006f1a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc068005950 tx=0x7fc0680058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:01:47.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.698+0000 7fc05ffff700 1 -- 172.21.15.121:0/1491352731 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc06c057850 con 0x7fc078101790 2024-04-05T22:01:47.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.966+0000 7fc07d197700 1 -- 172.21.15.121:0/1491352731 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc07804ea90 con 0x7fc078101790 2024-04-05T22:01:47.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.966+0000 7fc05ffff700 1 -- 172.21.15.121:0/1491352731 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fc06c003d10 con 0x7fc078101790 2024-04-05T22:01:47.968 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:01:47.968 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:01:47.968 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:01:47.968 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T22:01:47.968 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:01:47.968 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T22:01:47.968 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:01:47.969 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T22:01:47.970 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:01:47.970 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T22:01:47.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.968+0000 7fc07d197700 1 -- 172.21.15.121:0/1491352731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc06006cce0 msgr2=0x7fc06006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:47.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.968+0000 7fc07d197700 1 --2- 172.21.15.121:0/1491352731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc06006cce0 0x7fc06006f1a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc068005950 tx=0x7fc0680058e0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.968+0000 7fc07d197700 1 -- 172.21.15.121:0/1491352731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc078101790 msgr2=0x7fc078196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:01:47.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.968+0000 7fc07d197700 1 --2- 172.21.15.121:0/1491352731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc078101790 0x7fc078196c10 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0092e0 tx=0x7fc06c00be20 comp rx=0 tx=0).stop 2024-04-05T22:01:47.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.968+0000 7fc07d197700 1 -- 172.21.15.121:0/1491352731 shutdown_connections 2024-04-05T22:01:47.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.968+0000 7fc07d197700 1 --2- 172.21.15.121:0/1491352731 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc06006cce0 0x7fc06006f1a0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.968+0000 7fc07d197700 1 --2- 172.21.15.121:0/1491352731 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc078100de0 0x7fc0781966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.968+0000 7fc07d197700 1 --2- 172.21.15.121:0/1491352731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc078101790 0x7fc078196c10 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:01:47.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.969+0000 7fc07d197700 1 -- 172.21.15.121:0/1491352731 >> 172.21.15.121:0/1491352731 conn(0x7fc0780fc670 msgr2=0x7fc0780fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:01:47.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.969+0000 7fc07d197700 1 -- 172.21.15.121:0/1491352731 shutdown_connections 2024-04-05T22:01:47.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:01:47.969+0000 7fc07d197700 1 -- 172.21.15.121:0/1491352731 wait complete. 2024-04-05T22:01:48.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:47 smithi188 ceph-mon[107793]: from='client.35750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:48.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:47 smithi188 ceph-mon[107793]: from='client.35754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:48.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:47 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2700449257' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:01:48.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:47 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1491352731' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:01:48.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:47 smithi121 ceph-mon[145859]: from='client.35750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:48.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:47 smithi121 ceph-mon[145859]: from='client.35754 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:48.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:47 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2700449257' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:01:48.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:47 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1491352731' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:01:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:48 smithi188 ceph-mon[107793]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:48 smithi188 ceph-mon[107793]: pgmap v1068: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:49.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:48 smithi121 ceph-mon[145859]: from='client.35766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:01:49.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:48 smithi121 ceph-mon[145859]: pgmap v1068: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:51 smithi188 ceph-mon[107793]: pgmap v1069: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:51.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:51 smithi121 ceph-mon[145859]: pgmap v1069: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:53 smithi188 ceph-mon[107793]: pgmap v1070: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:53.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:53 smithi121 ceph-mon[145859]: pgmap v1070: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:55 smithi188 ceph-mon[107793]: pgmap v1071: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:55.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:55 smithi121 ceph-mon[145859]: pgmap v1071: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:57 smithi188 ceph-mon[107793]: pgmap v1072: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:57.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:57 smithi121 ceph-mon[145859]: pgmap v1072: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:01:59 smithi188 ceph-mon[107793]: pgmap v1073: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:01:59.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:01:59 smithi121 ceph-mon[145859]: pgmap v1073: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:01 smithi188 ceph-mon[107793]: pgmap v1074: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:02:01.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:01 smithi121 ceph-mon[145859]: pgmap v1074: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:01.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:02:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:03 smithi188 ceph-mon[107793]: pgmap v1075: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:03.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:03 smithi121 ceph-mon[145859]: pgmap v1075: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:05 smithi188 ceph-mon[107793]: pgmap v1076: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:05 smithi121 ceph-mon[145859]: pgmap v1076: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:07 smithi188 ceph-mon[107793]: pgmap v1077: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:07.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:07 smithi121 ceph-mon[145859]: pgmap v1077: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:08.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:08 smithi121 ceph-mon[145859]: pgmap v1078: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:08.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:08 smithi188 ceph-mon[107793]: pgmap v1078: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:11.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:10 smithi121 ceph-mon[145859]: pgmap v1079: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:10 smithi188 ceph-mon[107793]: pgmap v1079: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:13.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:12 smithi121 ceph-mon[145859]: pgmap v1080: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:12 smithi188 ceph-mon[107793]: pgmap v1080: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:15.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:14 smithi188 ceph-mon[107793]: pgmap v1081: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:15.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:14 smithi121 ceph-mon[145859]: pgmap v1081: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:02:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:02:17.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:16 smithi121 ceph-mon[145859]: pgmap v1082: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:17.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:16 smithi188 ceph-mon[107793]: pgmap v1082: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.086+0000 7f021b8c6700 1 -- 172.21.15.121:0/2530740982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02140ffa50 msgr2=0x7f02140ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.086+0000 7f021b8c6700 1 --2- 172.21.15.121:0/2530740982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02140ffa50 0x7f02140ffed0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f0210009910 tx=0x7f0210009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.086+0000 7f021b8c6700 1 -- 172.21.15.121:0/2530740982 shutdown_connections 2024-04-05T22:02:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.086+0000 7f021b8c6700 1 --2- 172.21.15.121:0/2530740982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02140ffa50 0x7f02140ffed0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.086+0000 7f021b8c6700 1 --2- 172.21.15.121:0/2530740982 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0214105a50 0x7f0214105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.086+0000 7f021b8c6700 1 -- 172.21.15.121:0/2530740982 >> 172.21.15.121:0/2530740982 conn(0x7f02140fb540 msgr2=0x7f02140fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.086+0000 7f021b8c6700 1 -- 172.21.15.121:0/2530740982 shutdown_connections 2024-04-05T22:02:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.086+0000 7f021b8c6700 1 -- 172.21.15.121:0/2530740982 wait complete. 2024-04-05T22:02:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.086+0000 7f021b8c6700 1 Processor -- start 2024-04-05T22:02:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.086+0000 7f021b8c6700 1 -- start start 2024-04-05T22:02:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.087+0000 7f021b8c6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02140ffa50 0x7f02141966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:18.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.087+0000 7f021b8c6700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0214105a50 0x7f0214196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.087+0000 7f021b8c6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02141972b0 con 0x7f02140ffa50 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.087+0000 7f021b8c6700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0214199fe0 con 0x7f0214105a50 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.087+0000 7f0219662700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02140ffa50 0x7f02141966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.087+0000 7f0219662700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02140ffa50 0x7f02141966e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50252/0 (socket says 172.21.15.121:50252) 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.087+0000 7f0219662700 1 -- 172.21.15.121:0/2120851763 learned_addr learned my addr 172.21.15.121:0/2120851763 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.087+0000 7f0218e61700 1 --2- 172.21.15.121:0/2120851763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0214105a50 0x7f0214196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.087+0000 7f0219662700 1 -- 172.21.15.121:0/2120851763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0214105a50 msgr2=0x7f0214196c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.087+0000 7f0219662700 1 --2- 172.21.15.121:0/2120851763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0214105a50 0x7f0214196c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.087+0000 7f0219662700 1 -- 172.21.15.121:0/2120851763 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02100095f0 con 0x7f02140ffa50 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.087+0000 7f0219662700 1 --2- 172.21.15.121:0/2120851763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02140ffa50 0x7f02141966e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f020400d8d0 tx=0x7f020400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.087+0000 7f020a7fc700 1 -- 172.21.15.121:0/2120851763 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02040098a0 con 0x7f02140ffa50 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.088+0000 7f020a7fc700 1 -- 172.21.15.121:0/2120851763 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0204010460 con 0x7f02140ffa50 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.088+0000 7f021b8c6700 1 -- 172.21.15.121:0/2120851763 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021419a2c0 con 0x7f02140ffa50 2024-04-05T22:02:18.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.088+0000 7f020a7fc700 1 -- 172.21.15.121:0/2120851763 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0204009ea0 con 0x7f02140ffa50 2024-04-05T22:02:18.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.088+0000 7f021b8c6700 1 -- 172.21.15.121:0/2120851763 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021419a810 con 0x7f02140ffa50 2024-04-05T22:02:18.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.088+0000 7f021b8c6700 1 -- 172.21.15.121:0/2120851763 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0214108ff0 con 0x7f02140ffa50 2024-04-05T22:02:18.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.089+0000 7f020a7fc700 1 -- 172.21.15.121:0/2120851763 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f02040105d0 con 0x7f02140ffa50 2024-04-05T22:02:18.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.089+0000 7f020a7fc700 1 --2- 172.21.15.121:0/2120851763 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f020006cce0 0x7f020006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:18.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.090+0000 7f020a7fc700 1 -- 172.21.15.121:0/2120851763 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f020408c480 con 0x7f02140ffa50 2024-04-05T22:02:18.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.090+0000 7f0218e61700 1 --2- 172.21.15.121:0/2120851763 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f020006cce0 0x7f020006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:18.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.090+0000 7f0218e61700 1 --2- 172.21.15.121:0/2120851763 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f020006cce0 0x7f020006f1a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0214197d00 tx=0x7f021001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:18.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.094+0000 7f020a7fc700 1 -- 172.21.15.121:0/2120851763 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f02040564f0 con 0x7f02140ffa50 2024-04-05T22:02:18.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.303+0000 7f021b8c6700 1 -- 172.21.15.121:0/2120851763 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f021419aa30 con 0x7f020006cce0 2024-04-05T22:02:18.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.305+0000 7f020a7fc700 1 -- 172.21.15.121:0/2120851763 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f021419aa30 con 0x7f020006cce0 2024-04-05T22:02:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.307+0000 7f021b8c6700 1 -- 172.21.15.121:0/2120851763 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f020006cce0 msgr2=0x7f020006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.307+0000 7f021b8c6700 1 --2- 172.21.15.121:0/2120851763 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f020006cce0 0x7f020006f1a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0214197d00 tx=0x7f021001a040 comp rx=0 tx=0).stop 2024-04-05T22:02:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.307+0000 7f021b8c6700 1 -- 172.21.15.121:0/2120851763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02140ffa50 msgr2=0x7f02141966e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.307+0000 7f021b8c6700 1 --2- 172.21.15.121:0/2120851763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02140ffa50 0x7f02141966e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f020400d8d0 tx=0x7f020400dbe0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.307+0000 7f021b8c6700 1 -- 172.21.15.121:0/2120851763 shutdown_connections 2024-04-05T22:02:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.307+0000 7f021b8c6700 1 --2- 172.21.15.121:0/2120851763 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f020006cce0 0x7f020006f1a0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.307+0000 7f021b8c6700 1 --2- 172.21.15.121:0/2120851763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02140ffa50 0x7f02141966e0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.307+0000 7f021b8c6700 1 --2- 172.21.15.121:0/2120851763 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0214105a50 0x7f0214196c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.307+0000 7f021b8c6700 1 -- 172.21.15.121:0/2120851763 >> 172.21.15.121:0/2120851763 conn(0x7f02140fb540 msgr2=0x7f0214069010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.307+0000 7f021b8c6700 1 -- 172.21.15.121:0/2120851763 shutdown_connections 2024-04-05T22:02:18.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.307+0000 7f021b8c6700 1 -- 172.21.15.121:0/2120851763 wait complete. 2024-04-05T22:02:18.317 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T22:02:18.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.410+0000 7fd63c314700 1 -- 172.21.15.121:0/3028357305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd634105a40 msgr2=0x7fd634105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:18.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.410+0000 7fd63c314700 1 --2- 172.21.15.121:0/3028357305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd634105a40 0x7fd634105e20 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd630009910 tx=0x7fd630009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:18.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.410+0000 7fd63c314700 1 -- 172.21.15.121:0/3028357305 shutdown_connections 2024-04-05T22:02:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.410+0000 7fd63c314700 1 --2- 172.21.15.121:0/3028357305 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6340ffbd0 0x7fd634100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.410+0000 7fd63c314700 1 --2- 172.21.15.121:0/3028357305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd634105a40 0x7fd634105e20 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.410+0000 7fd63c314700 1 -- 172.21.15.121:0/3028357305 >> 172.21.15.121:0/3028357305 conn(0x7fd6340755a0 msgr2=0x7fd6340759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.411+0000 7fd63c314700 1 -- 172.21.15.121:0/3028357305 shutdown_connections 2024-04-05T22:02:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.411+0000 7fd63c314700 1 -- 172.21.15.121:0/3028357305 wait complete. 2024-04-05T22:02:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.411+0000 7fd63c314700 1 Processor -- start 2024-04-05T22:02:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.411+0000 7fd63c314700 1 -- start start 2024-04-05T22:02:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.411+0000 7fd63c314700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6340ffbd0 0x7fd63419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.411+0000 7fd63c314700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd634105a40 0x7fd63419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.411+0000 7fd63c314700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63419f220 con 0x7fd634105a40 2024-04-05T22:02:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.411+0000 7fd63c314700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6341986d0 con 0x7fd6340ffbd0 2024-04-05T22:02:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.411+0000 7fd6398af700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd634105a40 0x7fd63419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:18.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.412+0000 7fd6398af700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd634105a40 0x7fd63419eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50278/0 (socket says 172.21.15.121:50278) 2024-04-05T22:02:18.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.412+0000 7fd6398af700 1 -- 172.21.15.121:0/1835129941 learned_addr learned my addr 172.21.15.121:0/1835129941 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:18.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.412+0000 7fd63a0b0700 1 --2- 172.21.15.121:0/1835129941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6340ffbd0 0x7fd63419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:18.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.412+0000 7fd6398af700 1 -- 172.21.15.121:0/1835129941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6340ffbd0 msgr2=0x7fd63419e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:18.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.412+0000 7fd6398af700 1 --2- 172.21.15.121:0/1835129941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6340ffbd0 0x7fd63419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.412+0000 7fd6398af700 1 -- 172.21.15.121:0/1835129941 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6300095f0 con 0x7fd634105a40 2024-04-05T22:02:18.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.412+0000 7fd6398af700 1 --2- 172.21.15.121:0/1835129941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd634105a40 0x7fd63419eb90 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd62400e920 tx=0x7fd62400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:18.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.412+0000 7fd62b7fe700 1 -- 172.21.15.121:0/1835129941 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd624009790 con 0x7fd634105a40 2024-04-05T22:02:18.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.412+0000 7fd62b7fe700 1 -- 172.21.15.121:0/1835129941 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd62400cba0 con 0x7fd634105a40 2024-04-05T22:02:18.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.412+0000 7fd63c314700 1 -- 172.21.15.121:0/1835129941 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6341989b0 con 0x7fd634105a40 2024-04-05T22:02:18.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.412+0000 7fd62b7fe700 1 -- 172.21.15.121:0/1835129941 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd624018650 con 0x7fd634105a40 2024-04-05T22:02:18.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.412+0000 7fd63c314700 1 -- 172.21.15.121:0/1835129941 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd634198f00 con 0x7fd634105a40 2024-04-05T22:02:18.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.413+0000 7fd63c314700 1 -- 172.21.15.121:0/1835129941 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63404ea90 con 0x7fd634105a40 2024-04-05T22:02:18.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.413+0000 7fd62b7fe700 1 -- 172.21.15.121:0/1835129941 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd62400cd10 con 0x7fd634105a40 2024-04-05T22:02:18.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.414+0000 7fd62b7fe700 1 --2- 172.21.15.121:0/1835129941 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd62006cce0 0x7fd62006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:18.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.414+0000 7fd62b7fe700 1 -- 172.21.15.121:0/1835129941 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd624014070 con 0x7fd634105a40 2024-04-05T22:02:18.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.414+0000 7fd63a0b0700 1 --2- 172.21.15.121:0/1835129941 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd62006cce0 0x7fd62006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:18.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.414+0000 7fd63a0b0700 1 --2- 172.21.15.121:0/1835129941 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd62006cce0 0x7fd62006f1a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd630006010 tx=0x7fd63000b5a0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:18.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.418+0000 7fd62b7fe700 1 -- 172.21.15.121:0/1835129941 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd624052ab0 con 0x7fd634105a40 2024-04-05T22:02:18.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.629+0000 7fd63c314700 1 -- 172.21.15.121:0/1835129941 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd634199430 con 0x7fd62006cce0 2024-04-05T22:02:18.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.630+0000 7fd62b7fe700 1 -- 172.21.15.121:0/1835129941 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd634199430 con 0x7fd62006cce0 2024-04-05T22:02:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.632+0000 7fd63c314700 1 -- 172.21.15.121:0/1835129941 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd62006cce0 msgr2=0x7fd62006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.632+0000 7fd63c314700 1 --2- 172.21.15.121:0/1835129941 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd62006cce0 0x7fd62006f1a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd630006010 tx=0x7fd63000b5a0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.632+0000 7fd63c314700 1 -- 172.21.15.121:0/1835129941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd634105a40 msgr2=0x7fd63419eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.632+0000 7fd63c314700 1 --2- 172.21.15.121:0/1835129941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd634105a40 0x7fd63419eb90 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd62400e920 tx=0x7fd62400ece0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.632+0000 7fd63c314700 1 -- 172.21.15.121:0/1835129941 shutdown_connections 2024-04-05T22:02:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.632+0000 7fd63c314700 1 --2- 172.21.15.121:0/1835129941 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd62006cce0 0x7fd62006f1a0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.632+0000 7fd63c314700 1 --2- 172.21.15.121:0/1835129941 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd6340ffbd0 0x7fd63419e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.632+0000 7fd63c314700 1 --2- 172.21.15.121:0/1835129941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd634105a40 0x7fd63419eb90 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.632+0000 7fd63c314700 1 -- 172.21.15.121:0/1835129941 >> 172.21.15.121:0/1835129941 conn(0x7fd6340755a0 msgr2=0x7fd6340fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.633+0000 7fd63c314700 1 -- 172.21.15.121:0/1835129941 shutdown_connections 2024-04-05T22:02:18.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.633+0000 7fd63c314700 1 -- 172.21.15.121:0/1835129941 wait complete. 2024-04-05T22:02:18.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.736+0000 7ff6de87a700 1 -- 172.21.15.121:0/4222737068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80ff050 msgr2=0x7ff6d80ff4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:18.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.736+0000 7ff6de87a700 1 --2- 172.21.15.121:0/4222737068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80ff050 0x7ff6d80ff4d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc009910 tx=0x7ff6cc009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:18.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.736+0000 7ff6de87a700 1 -- 172.21.15.121:0/4222737068 shutdown_connections 2024-04-05T22:02:18.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.736+0000 7ff6de87a700 1 --2- 172.21.15.121:0/4222737068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80ff050 0x7ff6d80ff4d0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.736+0000 7ff6de87a700 1 --2- 172.21.15.121:0/4222737068 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d8105b20 0x7ff6d8105f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.736+0000 7ff6de87a700 1 -- 172.21.15.121:0/4222737068 >> 172.21.15.121:0/4222737068 conn(0x7ff6d8075e30 msgr2=0x7ff6d80fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:18.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.736+0000 7ff6de87a700 1 -- 172.21.15.121:0/4222737068 shutdown_connections 2024-04-05T22:02:18.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.736+0000 7ff6de87a700 1 -- 172.21.15.121:0/4222737068 wait complete. 2024-04-05T22:02:18.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.737+0000 7ff6de87a700 1 Processor -- start 2024-04-05T22:02:18.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.737+0000 7ff6de87a700 1 -- start start 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.737+0000 7ff6de87a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80ff050 0x7ff6d81966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.737+0000 7ff6de87a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d8105b20 0x7ff6d8196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.737+0000 7ff6de87a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d8197290 con 0x7ff6d80ff050 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.737+0000 7ff6de87a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d8199fc0 con 0x7ff6d8105b20 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.737+0000 7ff6d7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80ff050 0x7ff6d81966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.737+0000 7ff6d7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80ff050 0x7ff6d81966c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50296/0 (socket says 172.21.15.121:50296) 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.737+0000 7ff6d7fff700 1 -- 172.21.15.121:0/3171769146 learned_addr learned my addr 172.21.15.121:0/3171769146 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.737+0000 7ff6d77fe700 1 --2- 172.21.15.121:0/3171769146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d8105b20 0x7ff6d8196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.738+0000 7ff6d7fff700 1 -- 172.21.15.121:0/3171769146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d8105b20 msgr2=0x7ff6d8196c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.738+0000 7ff6d7fff700 1 --2- 172.21.15.121:0/3171769146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d8105b20 0x7ff6d8196c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.738+0000 7ff6d7fff700 1 -- 172.21.15.121:0/3171769146 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc0095f0 con 0x7ff6d80ff050 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.738+0000 7ff6d7fff700 1 --2- 172.21.15.121:0/3171769146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80ff050 0x7ff6d81966c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800e920 tx=0x7ff6c800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.738+0000 7ff6d57fa700 1 -- 172.21.15.121:0/3171769146 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6c800c9d0 con 0x7ff6d80ff050 2024-04-05T22:02:18.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.738+0000 7ff6de87a700 1 -- 172.21.15.121:0/3171769146 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d819a2a0 con 0x7ff6d80ff050 2024-04-05T22:02:18.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.738+0000 7ff6d57fa700 1 -- 172.21.15.121:0/3171769146 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6c8010430 con 0x7ff6d80ff050 2024-04-05T22:02:18.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.738+0000 7ff6d57fa700 1 -- 172.21.15.121:0/3171769146 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6c8018650 con 0x7ff6d80ff050 2024-04-05T22:02:18.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.738+0000 7ff6de87a700 1 -- 172.21.15.121:0/3171769146 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d819a7f0 con 0x7ff6d80ff050 2024-04-05T22:02:18.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.739+0000 7ff6de87a700 1 -- 172.21.15.121:0/3171769146 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d81090c0 con 0x7ff6d80ff050 2024-04-05T22:02:18.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.739+0000 7ff6d57fa700 1 -- 172.21.15.121:0/3171769146 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff6c80105a0 con 0x7ff6d80ff050 2024-04-05T22:02:18.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.740+0000 7ff6d57fa700 1 --2- 172.21.15.121:0/3171769146 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff6c006cce0 0x7ff6c006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:18.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.740+0000 7ff6d57fa700 1 -- 172.21.15.121:0/3171769146 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff6c8014070 con 0x7ff6d80ff050 2024-04-05T22:02:18.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.740+0000 7ff6d77fe700 1 --2- 172.21.15.121:0/3171769146 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff6c006cce0 0x7ff6c006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:18.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.740+0000 7ff6d77fe700 1 --2- 172.21.15.121:0/3171769146 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff6c006cce0 0x7ff6c006f1a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8197ce0 tx=0x7ff6cc009f70 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:18.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.744+0000 7ff6d57fa700 1 -- 172.21.15.121:0/3171769146 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff6c8052ab0 con 0x7ff6d80ff050 2024-04-05T22:02:18.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.945+0000 7ff6de87a700 1 -- 172.21.15.121:0/3171769146 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6d819aa10 con 0x7ff6c006cce0 2024-04-05T22:02:18.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.955+0000 7ff6d57fa700 1 -- 172.21.15.121:0/3171769146 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7ff6d819aa10 con 0x7ff6c006cce0 2024-04-05T22:02:18.956 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T22:02:18.956 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (33m) 11m ago 44m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T22:02:18.956 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (44m) 11m ago 44m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T22:02:18.956 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (43m) 11m ago 43m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T22:02:18.956 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (31m) 11m ago 44m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T22:02:18.956 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (31m) 11m ago 43m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T22:02:18.956 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (32m) 11m ago 44m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T22:02:18.956 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (27m) 11m ago 38m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T22:02:18.956 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (27m) 11m ago 38m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T22:02:18.956 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (26m) 11m ago 38m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T22:02:18.956 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (11m) 11m ago 38m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T22:02:18.956 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (35m) 11m ago 46m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T22:02:18.956 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (34m) 11m ago 43m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T22:02:18.957 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (31m) 11m ago 46m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T22:02:18.957 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (31m) 11m ago 43m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T22:02:18.957 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (33m) 11m ago 44m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T22:02:18.957 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (33m) 11m ago 43m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T22:02:18.957 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (30m) 11m ago 42m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T22:02:18.957 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (30m) 11m ago 42m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T22:02:18.957 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (29m) 11m ago 41m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T22:02:18.957 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (29m) 11m ago 41m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T22:02:18.957 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (28m) 11m ago 40m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T22:02:18.957 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (27m) 11m ago 40m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T22:02:18.957 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (33m) 11m ago 44m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T22:02:18.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.958+0000 7ff6de87a700 1 -- 172.21.15.121:0/3171769146 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff6c006cce0 msgr2=0x7ff6c006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:18.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.958+0000 7ff6de87a700 1 --2- 172.21.15.121:0/3171769146 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff6c006cce0 0x7ff6c006f1a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8197ce0 tx=0x7ff6cc009f70 comp rx=0 tx=0).stop 2024-04-05T22:02:18.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.958+0000 7ff6de87a700 1 -- 172.21.15.121:0/3171769146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80ff050 msgr2=0x7ff6d81966c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:18.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.958+0000 7ff6de87a700 1 --2- 172.21.15.121:0/3171769146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80ff050 0x7ff6d81966c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800e920 tx=0x7ff6c800ec30 comp rx=0 tx=0).stop 2024-04-05T22:02:18.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.958+0000 7ff6de87a700 1 -- 172.21.15.121:0/3171769146 shutdown_connections 2024-04-05T22:02:18.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.958+0000 7ff6de87a700 1 --2- 172.21.15.121:0/3171769146 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff6c006cce0 0x7ff6c006f1a0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.958+0000 7ff6de87a700 1 --2- 172.21.15.121:0/3171769146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80ff050 0x7ff6d81966c0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.958+0000 7ff6de87a700 1 --2- 172.21.15.121:0/3171769146 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff6d8105b20 0x7ff6d8196c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:18.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.958+0000 7ff6de87a700 1 -- 172.21.15.121:0/3171769146 >> 172.21.15.121:0/3171769146 conn(0x7ff6d8075e30 msgr2=0x7ff6d8104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:18.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.958+0000 7ff6de87a700 1 -- 172.21.15.121:0/3171769146 shutdown_connections 2024-04-05T22:02:18.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:18.958+0000 7ff6de87a700 1 -- 172.21.15.121:0/3171769146 wait complete. 2024-04-05T22:02:19.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.061+0000 7f803b5e1700 1 -- 172.21.15.121:0/2831277431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80340ffdb0 msgr2=0x7f8034103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:19.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.061+0000 7f803b5e1700 1 --2- 172.21.15.121:0/2831277431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80340ffdb0 0x7f8034103e00 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f8024009910 tx=0x7f8024009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:19.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.061+0000 7f803b5e1700 1 -- 172.21.15.121:0/2831277431 shutdown_connections 2024-04-05T22:02:19.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.061+0000 7f803b5e1700 1 --2- 172.21.15.121:0/2831277431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80340ffdb0 0x7f8034103e00 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.061+0000 7f803b5e1700 1 --2- 172.21.15.121:0/2831277431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80340ff400 0x7f80340ff7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.061+0000 7f803b5e1700 1 -- 172.21.15.121:0/2831277431 >> 172.21.15.121:0/2831277431 conn(0x7f8034075e30 msgr2=0x7f80340fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:19.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.061+0000 7f803b5e1700 1 -- 172.21.15.121:0/2831277431 shutdown_connections 2024-04-05T22:02:19.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.061+0000 7f803b5e1700 1 -- 172.21.15.121:0/2831277431 wait complete. 2024-04-05T22:02:19.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.062+0000 7f803b5e1700 1 Processor -- start 2024-04-05T22:02:19.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.062+0000 7f803b5e1700 1 -- start start 2024-04-05T22:02:19.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.062+0000 7f803b5e1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80340ff400 0x7f8034194440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:19.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.062+0000 7f803b5e1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80340ffdb0 0x7f8034194980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:19.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.062+0000 7f803b5e1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8034195060 con 0x7f80340ff400 2024-04-05T22:02:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.062+0000 7f803b5e1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8034198df0 con 0x7f80340ffdb0 2024-04-05T22:02:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.062+0000 7f803937d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80340ff400 0x7f8034194440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.062+0000 7f803937d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80340ff400 0x7f8034194440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50316/0 (socket says 172.21.15.121:50316) 2024-04-05T22:02:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.062+0000 7f803937d700 1 -- 172.21.15.121:0/2020028405 learned_addr learned my addr 172.21.15.121:0/2020028405 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.063+0000 7f803937d700 1 -- 172.21.15.121:0/2020028405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80340ffdb0 msgr2=0x7f8034194980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.063+0000 7f803937d700 1 --2- 172.21.15.121:0/2020028405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80340ffdb0 0x7f8034194980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.063+0000 7f803937d700 1 -- 172.21.15.121:0/2020028405 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80240095f0 con 0x7f80340ff400 2024-04-05T22:02:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.063+0000 7f803937d700 1 --2- 172.21.15.121:0/2020028405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80340ff400 0x7f8034194440 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8030010c40 tx=0x7f803000c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.063+0000 7f802a7fc700 1 -- 172.21.15.121:0/2020028405 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8030018410 con 0x7f80340ff400 2024-04-05T22:02:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.063+0000 7f802a7fc700 1 -- 172.21.15.121:0/2020028405 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8030018a10 con 0x7f80340ff400 2024-04-05T22:02:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.063+0000 7f803b5e1700 1 -- 172.21.15.121:0/2020028405 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80341990d0 con 0x7f80340ff400 2024-04-05T22:02:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.063+0000 7f802a7fc700 1 -- 172.21.15.121:0/2020028405 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f803000b930 con 0x7f80340ff400 2024-04-05T22:02:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.063+0000 7f803b5e1700 1 -- 172.21.15.121:0/2020028405 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8034199620 con 0x7f80340ff400 2024-04-05T22:02:19.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.064+0000 7f803b5e1700 1 -- 172.21.15.121:0/2020028405 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8034107800 con 0x7f80340ff400 2024-04-05T22:02:19.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.064+0000 7f802a7fc700 1 -- 172.21.15.121:0/2020028405 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8030018b80 con 0x7f80340ff400 2024-04-05T22:02:19.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.065+0000 7f802a7fc700 1 --2- 172.21.15.121:0/2020028405 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f802006cce0 0x7f802006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:19.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.065+0000 7f802a7fc700 1 -- 172.21.15.121:0/2020028405 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f803008e8d0 con 0x7f80340ff400 2024-04-05T22:02:19.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.065+0000 7f8038b7c700 1 --2- 172.21.15.121:0/2020028405 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f802006cce0 0x7f802006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:19.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.065+0000 7f8038b7c700 1 --2- 172.21.15.121:0/2020028405 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f802006cce0 0x7f802006f1a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f802400b6e0 tx=0x7f8024000bc0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:19.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.069+0000 7f802a7fc700 1 -- 172.21.15.121:0/2020028405 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8030058950 con 0x7f80340ff400 2024-04-05T22:02:19.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:18 smithi121 ceph-mon[145859]: pgmap v1083: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:19.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:18 smithi188 ceph-mon[107793]: pgmap v1083: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:19.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.345+0000 7f803b5e1700 1 -- 172.21.15.121:0/2020028405 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f80341998a0 con 0x7f80340ff400 2024-04-05T22:02:19.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.346+0000 7f802a7fc700 1 -- 172.21.15.121:0/2020028405 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f803005bf70 con 0x7f80340ff400 2024-04-05T22:02:19.346 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:02:19.346 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T22:02:19.346 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:02:19.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:02:19.346 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T22:02:19.346 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:02:19.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:02:19.347 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T22:02:19.347 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T22:02:19.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:02:19.347 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T22:02:19.347 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T22:02:19.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:02:19.347 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T22:02:19.347 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T22:02:19.347 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T22:02:19.347 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:02:19.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.348+0000 7f803b5e1700 1 -- 172.21.15.121:0/2020028405 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f802006cce0 msgr2=0x7f802006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:19.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.348+0000 7f803b5e1700 1 --2- 172.21.15.121:0/2020028405 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f802006cce0 0x7f802006f1a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f802400b6e0 tx=0x7f8024000bc0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.348+0000 7f803b5e1700 1 -- 172.21.15.121:0/2020028405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80340ff400 msgr2=0x7f8034194440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:19.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.348+0000 7f803b5e1700 1 --2- 172.21.15.121:0/2020028405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80340ff400 0x7f8034194440 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f8030010c40 tx=0x7f803000c3d0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.348+0000 7f803b5e1700 1 -- 172.21.15.121:0/2020028405 shutdown_connections 2024-04-05T22:02:19.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.348+0000 7f803b5e1700 1 --2- 172.21.15.121:0/2020028405 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f802006cce0 0x7f802006f1a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.348+0000 7f803b5e1700 1 --2- 172.21.15.121:0/2020028405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80340ff400 0x7f8034194440 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.348+0000 7f803b5e1700 1 --2- 172.21.15.121:0/2020028405 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f80340ffdb0 0x7f8034194980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.348+0000 7f803b5e1700 1 -- 172.21.15.121:0/2020028405 >> 172.21.15.121:0/2020028405 conn(0x7f8034075e30 msgr2=0x7f8034103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:19.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.348+0000 7f803b5e1700 1 -- 172.21.15.121:0/2020028405 shutdown_connections 2024-04-05T22:02:19.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.348+0000 7f803b5e1700 1 -- 172.21.15.121:0/2020028405 wait complete. 2024-04-05T22:02:19.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.450+0000 7ff31c108700 1 -- 172.21.15.121:0/89207569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3140ffa40 msgr2=0x7ff3140ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:19.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.450+0000 7ff31c108700 1 --2- 172.21.15.121:0/89207569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3140ffa40 0x7ff3140ffec0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7ff304009910 tx=0x7ff304009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:19.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.450+0000 7ff31c108700 1 -- 172.21.15.121:0/89207569 shutdown_connections 2024-04-05T22:02:19.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.450+0000 7ff31c108700 1 --2- 172.21.15.121:0/89207569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3140ffa40 0x7ff3140ffec0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.450+0000 7ff31c108700 1 --2- 172.21.15.121:0/89207569 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff314105a40 0x7ff314105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.450+0000 7ff31c108700 1 -- 172.21.15.121:0/89207569 >> 172.21.15.121:0/89207569 conn(0x7ff3140fb570 msgr2=0x7ff3140fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:19.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.451+0000 7ff31c108700 1 -- 172.21.15.121:0/89207569 shutdown_connections 2024-04-05T22:02:19.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.451+0000 7ff31c108700 1 -- 172.21.15.121:0/89207569 wait complete. 2024-04-05T22:02:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.451+0000 7ff31c108700 1 Processor -- start 2024-04-05T22:02:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.451+0000 7ff31c108700 1 -- start start 2024-04-05T22:02:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.451+0000 7ff31c108700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3140ffa40 0x7ff31419c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.451+0000 7ff31c108700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff314105a40 0x7ff31419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.451+0000 7ff31c108700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31419d110 con 0x7ff314105a40 2024-04-05T22:02:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.451+0000 7ff31c108700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3141965c0 con 0x7ff3140ffa40 2024-04-05T22:02:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.451+0000 7ff3196a3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff314105a40 0x7ff31419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.452+0000 7ff3196a3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff314105a40 0x7ff31419ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50336/0 (socket says 172.21.15.121:50336) 2024-04-05T22:02:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.452+0000 7ff3196a3700 1 -- 172.21.15.121:0/2873346327 learned_addr learned my addr 172.21.15.121:0/2873346327 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:19.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.452+0000 7ff319ea4700 1 --2- 172.21.15.121:0/2873346327 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3140ffa40 0x7ff31419c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.452+0000 7ff3196a3700 1 -- 172.21.15.121:0/2873346327 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3140ffa40 msgr2=0x7ff31419c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.452+0000 7ff3196a3700 1 --2- 172.21.15.121:0/2873346327 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3140ffa40 0x7ff31419c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.452+0000 7ff3196a3700 1 -- 172.21.15.121:0/2873346327 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3040095f0 con 0x7ff314105a40 2024-04-05T22:02:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.452+0000 7ff3196a3700 1 --2- 172.21.15.121:0/2873346327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff314105a40 0x7ff31419ca80 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ff304006010 tx=0x7ff30400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.452+0000 7ff30affd700 1 -- 172.21.15.121:0/2873346327 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff30401d070 con 0x7ff314105a40 2024-04-05T22:02:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.452+0000 7ff31c108700 1 -- 172.21.15.121:0/2873346327 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff314196840 con 0x7ff314105a40 2024-04-05T22:02:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.452+0000 7ff30affd700 1 -- 172.21.15.121:0/2873346327 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff30400f460 con 0x7ff314105a40 2024-04-05T22:02:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.452+0000 7ff30affd700 1 -- 172.21.15.121:0/2873346327 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff304021640 con 0x7ff314105a40 2024-04-05T22:02:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.452+0000 7ff31c108700 1 -- 172.21.15.121:0/2873346327 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff314196d30 con 0x7ff314105a40 2024-04-05T22:02:19.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.453+0000 7ff31c108700 1 -- 172.21.15.121:0/2873346327 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff31404ea90 con 0x7ff314105a40 2024-04-05T22:02:19.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.454+0000 7ff30affd700 1 -- 172.21.15.121:0/2873346327 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff30400fa20 con 0x7ff314105a40 2024-04-05T22:02:19.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.454+0000 7ff30affd700 1 --2- 172.21.15.121:0/2873346327 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff30006cce0 0x7ff30006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:19.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.454+0000 7ff30affd700 1 -- 172.21.15.121:0/2873346327 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff30408dc80 con 0x7ff314105a40 2024-04-05T22:02:19.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.454+0000 7ff319ea4700 1 --2- 172.21.15.121:0/2873346327 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff30006cce0 0x7ff30006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:19.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.455+0000 7ff319ea4700 1 --2- 172.21.15.121:0/2873346327 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff30006cce0 0x7ff30006f1a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff310006fd0 tx=0x7ff310008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:19.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.458+0000 7ff30affd700 1 -- 172.21.15.121:0/2873346327 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff30405b920 con 0x7ff314105a40 2024-04-05T22:02:19.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.692+0000 7ff31c108700 1 -- 172.21.15.121:0/2873346327 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff314197ad0 con 0x7ff314105a40 2024-04-05T22:02:19.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.693+0000 7ff30affd700 1 -- 172.21.15.121:0/2873346327 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7ff304026750 con 0x7ff314105a40 2024-04-05T22:02:19.693 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T22:02:19.693 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T22:02:19.693 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:02:19.693 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T22:02:19.693 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T22:02:19.694 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:02:19.695 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T22:02:19.696 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:02:19.696 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:02:19.696 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:02:19.696 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:02:19.696 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:02:19.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.695+0000 7ff31c108700 1 -- 172.21.15.121:0/2873346327 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff30006cce0 msgr2=0x7ff30006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:19.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.695+0000 7ff31c108700 1 --2- 172.21.15.121:0/2873346327 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff30006cce0 0x7ff30006f1a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff310006fd0 tx=0x7ff310008040 comp rx=0 tx=0).stop 2024-04-05T22:02:19.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.695+0000 7ff31c108700 1 -- 172.21.15.121:0/2873346327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff314105a40 msgr2=0x7ff31419ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:19.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.695+0000 7ff31c108700 1 --2- 172.21.15.121:0/2873346327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff314105a40 0x7ff31419ca80 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7ff304006010 tx=0x7ff30400be20 comp rx=0 tx=0).stop 2024-04-05T22:02:19.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.695+0000 7ff31c108700 1 -- 172.21.15.121:0/2873346327 shutdown_connections 2024-04-05T22:02:19.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.695+0000 7ff31c108700 1 --2- 172.21.15.121:0/2873346327 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff30006cce0 0x7ff30006f1a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.695+0000 7ff31c108700 1 --2- 172.21.15.121:0/2873346327 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff3140ffa40 0x7ff31419c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.695+0000 7ff31c108700 1 --2- 172.21.15.121:0/2873346327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff314105a40 0x7ff31419ca80 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.695+0000 7ff31c108700 1 -- 172.21.15.121:0/2873346327 >> 172.21.15.121:0/2873346327 conn(0x7ff3140fb570 msgr2=0x7ff3141038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:19.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.695+0000 7ff31c108700 1 -- 172.21.15.121:0/2873346327 shutdown_connections 2024-04-05T22:02:19.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.695+0000 7ff31c108700 1 -- 172.21.15.121:0/2873346327 wait complete. 2024-04-05T22:02:19.697 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T22:02:19.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.798+0000 7f83edf1f700 1 -- 172.21.15.121:0/1186649260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e8100de0 msgr2=0x7f83e81011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:19.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.798+0000 7f83edf1f700 1 --2- 172.21.15.121:0/1186649260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e8100de0 0x7f83e81011c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f83dc009910 tx=0x7f83dc009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:19.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.798+0000 7f83edf1f700 1 -- 172.21.15.121:0/1186649260 shutdown_connections 2024-04-05T22:02:19.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.798+0000 7f83edf1f700 1 --2- 172.21.15.121:0/1186649260 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83e8101790 0x7f83e81057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.798+0000 7f83edf1f700 1 --2- 172.21.15.121:0/1186649260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e8100de0 0x7f83e81011c0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.798+0000 7f83edf1f700 1 -- 172.21.15.121:0/1186649260 >> 172.21.15.121:0/1186649260 conn(0x7f83e80fc670 msgr2=0x7f83e80fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:19.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.798+0000 7f83edf1f700 1 -- 172.21.15.121:0/1186649260 shutdown_connections 2024-04-05T22:02:19.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.798+0000 7f83edf1f700 1 -- 172.21.15.121:0/1186649260 wait complete. 2024-04-05T22:02:19.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.798+0000 7f83edf1f700 1 Processor -- start 2024-04-05T22:02:19.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83edf1f700 1 -- start start 2024-04-05T22:02:19.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83edf1f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83e8100de0 0x7f83e8073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:19.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83edf1f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e8101790 0x7f83e80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:19.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83edf1f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e8073b10 con 0x7f83e8101790 2024-04-05T22:02:19.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83edf1f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e8073c50 con 0x7f83e8100de0 2024-04-05T22:02:19.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83e6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e8101790 0x7f83e80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:19.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83e6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e8101790 0x7f83e80735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50364/0 (socket says 172.21.15.121:50364) 2024-04-05T22:02:19.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83e6ffd700 1 -- 172.21.15.121:0/1935238867 learned_addr learned my addr 172.21.15.121:0/1935238867 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:19.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83e77fe700 1 --2- 172.21.15.121:0/1935238867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83e8100de0 0x7f83e8073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:19.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83e6ffd700 1 -- 172.21.15.121:0/1935238867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83e8100de0 msgr2=0x7f83e8073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:19.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83e6ffd700 1 --2- 172.21.15.121:0/1935238867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83e8100de0 0x7f83e8073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:19.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83e6ffd700 1 -- 172.21.15.121:0/1935238867 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83dc0095f0 con 0x7f83e8101790 2024-04-05T22:02:19.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83e6ffd700 1 --2- 172.21.15.121:0/1935238867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e8101790 0x7f83e80735d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f83d800e8f0 tx=0x7f83d800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:19.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83e4ff9700 1 -- 172.21.15.121:0/1935238867 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83d8009790 con 0x7f83e8101790 2024-04-05T22:02:19.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83e4ff9700 1 -- 172.21.15.121:0/1935238867 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83d800c810 con 0x7f83e8101790 2024-04-05T22:02:19.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83edf1f700 1 -- 172.21.15.121:0/1935238867 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e81a2840 con 0x7f83e8101790 2024-04-05T22:02:19.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.799+0000 7f83e4ff9700 1 -- 172.21.15.121:0/1935238867 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83d8005520 con 0x7f83e8101790 2024-04-05T22:02:19.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.800+0000 7f83edf1f700 1 -- 172.21.15.121:0/1935238867 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e81a2c50 con 0x7f83e8101790 2024-04-05T22:02:19.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.800+0000 7f83edf1f700 1 -- 172.21.15.121:0/1935238867 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e8109180 con 0x7f83e8101790 2024-04-05T22:02:19.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.801+0000 7f83e4ff9700 1 -- 172.21.15.121:0/1935238867 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f83d800c980 con 0x7f83e8101790 2024-04-05T22:02:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.801+0000 7f83e4ff9700 1 --2- 172.21.15.121:0/1935238867 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f83d006cce0 0x7f83d006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.801+0000 7f83e4ff9700 1 -- 172.21.15.121:0/1935238867 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f83d8014070 con 0x7f83e8101790 2024-04-05T22:02:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.801+0000 7f83e77fe700 1 --2- 172.21.15.121:0/1935238867 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f83d006cce0 0x7f83d006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:19.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.802+0000 7f83e77fe700 1 --2- 172.21.15.121:0/1935238867 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f83d006cce0 0x7f83d006f1a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f83dc009fd0 tx=0x7f83dc005940 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:19.806+0000 7f83e4ff9700 1 -- 172.21.15.121:0/1935238867 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f83d8056920 con 0x7f83e8101790 2024-04-05T22:02:20.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.018+0000 7f83edf1f700 1 -- 172.21.15.121:0/1935238867 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f83e802cda0 con 0x7f83d006cce0 2024-04-05T22:02:20.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.020+0000 7f83e4ff9700 1 -- 172.21.15.121:0/1935238867 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f83e802cda0 con 0x7f83d006cce0 2024-04-05T22:02:20.020 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:02:20.020 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T22:02:20.020 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T22:02:20.020 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T22:02:20.020 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T22:02:20.020 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T22:02:20.020 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T22:02:20.020 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T22:02:20.020 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T22:02:20.021 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T22:02:20.021 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T22:02:20.021 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T22:02:20.021 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T22:02:20.021 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T22:02:20.021 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:02:20.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.022+0000 7f83edf1f700 1 -- 172.21.15.121:0/1935238867 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f83d006cce0 msgr2=0x7f83d006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:20.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.022+0000 7f83edf1f700 1 --2- 172.21.15.121:0/1935238867 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f83d006cce0 0x7f83d006f1a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f83dc009fd0 tx=0x7f83dc005940 comp rx=0 tx=0).stop 2024-04-05T22:02:20.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.022+0000 7f83edf1f700 1 -- 172.21.15.121:0/1935238867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e8101790 msgr2=0x7f83e80735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:20.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.022+0000 7f83edf1f700 1 --2- 172.21.15.121:0/1935238867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e8101790 0x7f83e80735d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f83d800e8f0 tx=0x7f83d800ec00 comp rx=0 tx=0).stop 2024-04-05T22:02:20.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.022+0000 7f83edf1f700 1 -- 172.21.15.121:0/1935238867 shutdown_connections 2024-04-05T22:02:20.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.022+0000 7f83edf1f700 1 --2- 172.21.15.121:0/1935238867 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f83d006cce0 0x7f83d006f1a0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:20.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.022+0000 7f83edf1f700 1 --2- 172.21.15.121:0/1935238867 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83e8100de0 0x7f83e8073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:20.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.022+0000 7f83edf1f700 1 --2- 172.21.15.121:0/1935238867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e8101790 0x7f83e80735d0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:20.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.022+0000 7f83edf1f700 1 -- 172.21.15.121:0/1935238867 >> 172.21.15.121:0/1935238867 conn(0x7f83e80fc670 msgr2=0x7f83e80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:20.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.022+0000 7f83edf1f700 1 -- 172.21.15.121:0/1935238867 shutdown_connections 2024-04-05T22:02:20.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.022+0000 7f83edf1f700 1 -- 172.21.15.121:0/1935238867 wait complete. 2024-04-05T22:02:20.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.124+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1094700237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad480ffa40 msgr2=0x7fad480ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:20.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.124+0000 7fad4f4fb700 1 --2- 172.21.15.121:0/1094700237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad480ffa40 0x7fad480ffec0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fad44009910 tx=0x7fad44009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:20.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.124+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1094700237 shutdown_connections 2024-04-05T22:02:20.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.124+0000 7fad4f4fb700 1 --2- 172.21.15.121:0/1094700237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad480ffa40 0x7fad480ffec0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:20.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.124+0000 7fad4f4fb700 1 --2- 172.21.15.121:0/1094700237 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad48105a40 0x7fad48105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:20.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.124+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1094700237 >> 172.21.15.121:0/1094700237 conn(0x7fad480fb570 msgr2=0x7fad480fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:20.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.124+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1094700237 shutdown_connections 2024-04-05T22:02:20.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.124+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1094700237 wait complete. 2024-04-05T22:02:20.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.125+0000 7fad4f4fb700 1 Processor -- start 2024-04-05T22:02:20.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.125+0000 7fad4f4fb700 1 -- start start 2024-04-05T22:02:20.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.125+0000 7fad4f4fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad480ffa40 0x7fad480fbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:20.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.125+0000 7fad4f4fb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad48105a40 0x7fad480fc480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:20.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.125+0000 7fad4f4fb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad480fc9c0 con 0x7fad480ffa40 2024-04-05T22:02:20.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.125+0000 7fad4f4fb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad480fcb00 con 0x7fad48105a40 2024-04-05T22:02:20.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.125+0000 7fad4d297700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad480ffa40 0x7fad480fbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:20.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.125+0000 7fad4d297700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad480ffa40 0x7fad480fbf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50386/0 (socket says 172.21.15.121:50386) 2024-04-05T22:02:20.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.125+0000 7fad4d297700 1 -- 172.21.15.121:0/1293205185 learned_addr learned my addr 172.21.15.121:0/1293205185 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:20.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.126+0000 7fad4ca96700 1 --2- 172.21.15.121:0/1293205185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad48105a40 0x7fad480fc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:20.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.126+0000 7fad4d297700 1 -- 172.21.15.121:0/1293205185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad48105a40 msgr2=0x7fad480fc480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:20.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.126+0000 7fad4d297700 1 --2- 172.21.15.121:0/1293205185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad48105a40 0x7fad480fc480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:20.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.126+0000 7fad4d297700 1 -- 172.21.15.121:0/1293205185 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad440095f0 con 0x7fad480ffa40 2024-04-05T22:02:20.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.126+0000 7fad4ca96700 1 --2- 172.21.15.121:0/1293205185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad48105a40 0x7fad480fc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:02:20.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.126+0000 7fad4d297700 1 --2- 172.21.15.121:0/1293205185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad480ffa40 0x7fad480fbf40 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fad3800d900 tx=0x7fad3800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:20.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.126+0000 7fad3e7fc700 1 -- 172.21.15.121:0/1293205185 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad38004a00 con 0x7fad480ffa40 2024-04-05T22:02:20.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.126+0000 7fad3e7fc700 1 -- 172.21.15.121:0/1293205185 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad38005020 con 0x7fad480ffa40 2024-04-05T22:02:20.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.126+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1293205185 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad481a2800 con 0x7fad480ffa40 2024-04-05T22:02:20.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.126+0000 7fad3e7fc700 1 -- 172.21.15.121:0/1293205185 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad38009c00 con 0x7fad480ffa40 2024-04-05T22:02:20.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.126+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1293205185 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad481a2c10 con 0x7fad480ffa40 2024-04-05T22:02:20.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.127+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1293205185 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad48108fe0 con 0x7fad480ffa40 2024-04-05T22:02:20.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.128+0000 7fad3e7fc700 1 -- 172.21.15.121:0/1293205185 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fad38005190 con 0x7fad480ffa40 2024-04-05T22:02:20.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.128+0000 7fad3e7fc700 1 --2- 172.21.15.121:0/1293205185 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fad3406cce0 0x7fad3406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:20.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.128+0000 7fad3e7fc700 1 -- 172.21.15.121:0/1293205185 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fad3808cc70 con 0x7fad480ffa40 2024-04-05T22:02:20.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.128+0000 7fad4ca96700 1 --2- 172.21.15.121:0/1293205185 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fad3406cce0 0x7fad3406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:20.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.129+0000 7fad4ca96700 1 --2- 172.21.15.121:0/1293205185 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fad3406cce0 0x7fad3406f1a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fad44006010 tx=0x7fad4400b540 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:20.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.132+0000 7fad3e7fc700 1 -- 172.21.15.121:0/1293205185 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fad38056d40 con 0x7fad480ffa40 2024-04-05T22:02:20.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:19 smithi121 ceph-mon[145859]: from='client.35774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:20.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:19 smithi121 ceph-mon[145859]: from='client.35778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:20.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:19 smithi121 ceph-mon[145859]: from='client.35782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:20.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:19 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2020028405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:02:20.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:19 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2873346327' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:02:20.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:19 smithi188 ceph-mon[107793]: from='client.35774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:20.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:19 smithi188 ceph-mon[107793]: from='client.35778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:20.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:19 smithi188 ceph-mon[107793]: from='client.35782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:20.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:19 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2020028405' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:02:20.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:19 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2873346327' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:02:20.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.404+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1293205185 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fad480fd470 con 0x7fad480ffa40 2024-04-05T22:02:20.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.404+0000 7fad3e7fc700 1 -- 172.21.15.121:0/1293205185 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fad480fd470 con 0x7fad480ffa40 2024-04-05T22:02:20.405 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:02:20.405 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:02:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:02:20.405 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T22:02:20.405 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:02:20.405 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T22:02:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:02:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T22:02:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:02:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T22:02:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:02:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:02:20.405 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:02:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:02:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:02:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:02:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:02:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:02:20.406 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:02:20.406 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:02:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T22:02:20.406 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:02:20.406 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T22:02:20.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.406+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1293205185 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fad3406cce0 msgr2=0x7fad3406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:20.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.406+0000 7fad4f4fb700 1 --2- 172.21.15.121:0/1293205185 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fad3406cce0 0x7fad3406f1a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fad44006010 tx=0x7fad4400b540 comp rx=0 tx=0).stop 2024-04-05T22:02:20.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.406+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1293205185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad480ffa40 msgr2=0x7fad480fbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:20.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.406+0000 7fad4f4fb700 1 --2- 172.21.15.121:0/1293205185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad480ffa40 0x7fad480fbf40 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fad3800d900 tx=0x7fad3800dcc0 comp rx=0 tx=0).stop 2024-04-05T22:02:20.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.406+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1293205185 shutdown_connections 2024-04-05T22:02:20.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.406+0000 7fad4f4fb700 1 --2- 172.21.15.121:0/1293205185 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fad3406cce0 0x7fad3406f1a0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:20.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.406+0000 7fad4f4fb700 1 --2- 172.21.15.121:0/1293205185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad480ffa40 0x7fad480fbf40 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:20.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.406+0000 7fad4f4fb700 1 --2- 172.21.15.121:0/1293205185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fad48105a40 0x7fad480fc480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:20.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.406+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1293205185 >> 172.21.15.121:0/1293205185 conn(0x7fad480fb570 msgr2=0x7fad48068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:20.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.406+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1293205185 shutdown_connections 2024-04-05T22:02:20.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:20.406+0000 7fad4f4fb700 1 -- 172.21.15.121:0/1293205185 wait complete. 2024-04-05T22:02:21.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:20 smithi121 ceph-mon[145859]: pgmap v1084: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:21.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:20 smithi121 ceph-mon[145859]: from='client.35794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:21.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:20 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1293205185' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:02:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:20 smithi188 ceph-mon[107793]: pgmap v1084: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:20 smithi188 ceph-mon[107793]: from='client.35794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:20 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1293205185' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:02:23.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:22 smithi121 ceph-mon[145859]: pgmap v1085: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:23.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:22 smithi188 ceph-mon[107793]: pgmap v1085: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:25.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:24 smithi121 ceph-mon[145859]: pgmap v1086: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:24 smithi188 ceph-mon[107793]: pgmap v1086: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:27.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:26 smithi121 ceph-mon[145859]: pgmap v1087: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:27.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:26 smithi188 ceph-mon[107793]: pgmap v1087: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:29.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:28 smithi121 ceph-mon[145859]: pgmap v1088: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:28 smithi188 ceph-mon[107793]: pgmap v1088: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:30 smithi121 ceph-mon[145859]: pgmap v1089: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:02:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:30 smithi188 ceph-mon[107793]: pgmap v1089: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:02:33.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:32 smithi121 ceph-mon[145859]: pgmap v1090: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:33.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:32 smithi188 ceph-mon[107793]: pgmap v1090: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:35.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:34 smithi121 ceph-mon[145859]: pgmap v1091: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:35.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:34 smithi188 ceph-mon[107793]: pgmap v1091: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:37.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:36 smithi121 ceph-mon[145859]: pgmap v1092: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:36 smithi188 ceph-mon[107793]: pgmap v1092: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:39.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:38 smithi121 ceph-mon[145859]: pgmap v1093: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:39.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:38 smithi188 ceph-mon[107793]: pgmap v1093: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:40 smithi121 ceph-mon[145859]: pgmap v1094: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:40 smithi188 ceph-mon[107793]: pgmap v1094: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:43.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:42 smithi121 ceph-mon[145859]: pgmap v1095: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:43.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:42 smithi188 ceph-mon[107793]: pgmap v1095: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:45.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:44 smithi121 ceph-mon[145859]: pgmap v1096: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:44 smithi188 ceph-mon[107793]: pgmap v1096: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:02:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:02:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:46 smithi188 ceph-mon[107793]: pgmap v1097: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:47.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:46 smithi121 ceph-mon[145859]: pgmap v1097: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:48 smithi188 ceph-mon[107793]: pgmap v1098: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:49.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:48 smithi121 ceph-mon[145859]: pgmap v1098: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:50.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.522+0000 7f4e9d41b700 1 -- 172.21.15.121:0/3417932294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e980ffa40 msgr2=0x7f4e980ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:50.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.522+0000 7f4e9d41b700 1 --2- 172.21.15.121:0/3417932294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e980ffa40 0x7f4e980ffec0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f4e88009910 tx=0x7f4e88009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:50.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.522+0000 7f4e9d41b700 1 -- 172.21.15.121:0/3417932294 shutdown_connections 2024-04-05T22:02:50.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.522+0000 7f4e9d41b700 1 --2- 172.21.15.121:0/3417932294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e980ffa40 0x7f4e980ffec0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:50.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.522+0000 7f4e9d41b700 1 --2- 172.21.15.121:0/3417932294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e98105a40 0x7f4e98105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:50.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.522+0000 7f4e9d41b700 1 -- 172.21.15.121:0/3417932294 >> 172.21.15.121:0/3417932294 conn(0x7f4e980fb570 msgr2=0x7f4e980fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:50.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.522+0000 7f4e9d41b700 1 -- 172.21.15.121:0/3417932294 shutdown_connections 2024-04-05T22:02:50.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.522+0000 7f4e9d41b700 1 -- 172.21.15.121:0/3417932294 wait complete. 2024-04-05T22:02:50.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.523+0000 7f4e9d41b700 1 Processor -- start 2024-04-05T22:02:50.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.523+0000 7f4e9d41b700 1 -- start start 2024-04-05T22:02:50.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.523+0000 7f4e9d41b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e980ffa40 0x7f4e98196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:50.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.523+0000 7f4e9d41b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e98105a40 0x7f4e98196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:50.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.523+0000 7f4e9d41b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98197230 con 0x7f4e98105a40 2024-04-05T22:02:50.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.523+0000 7f4e9d41b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9819afc0 con 0x7f4e980ffa40 2024-04-05T22:02:50.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.523+0000 7f4e967fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e98105a40 0x7f4e98196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:50.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.523+0000 7f4e967fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e98105a40 0x7f4e98196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58440/0 (socket says 172.21.15.121:58440) 2024-04-05T22:02:50.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.523+0000 7f4e967fc700 1 -- 172.21.15.121:0/1984787990 learned_addr learned my addr 172.21.15.121:0/1984787990 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:50.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.523+0000 7f4e96ffd700 1 --2- 172.21.15.121:0/1984787990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e980ffa40 0x7f4e98196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:50.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.524+0000 7f4e967fc700 1 -- 172.21.15.121:0/1984787990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e980ffa40 msgr2=0x7f4e98196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:50.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.524+0000 7f4e967fc700 1 --2- 172.21.15.121:0/1984787990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e980ffa40 0x7f4e98196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:50.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.524+0000 7f4e967fc700 1 -- 172.21.15.121:0/1984787990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e880095f0 con 0x7f4e98105a40 2024-04-05T22:02:50.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.524+0000 7f4e967fc700 1 --2- 172.21.15.121:0/1984787990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e98105a40 0x7f4e98196b50 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f4e880092e0 tx=0x7f4e8800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:50.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.524+0000 7f4e7ffff700 1 -- 172.21.15.121:0/1984787990 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e8801d070 con 0x7f4e98105a40 2024-04-05T22:02:50.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.524+0000 7f4e7ffff700 1 -- 172.21.15.121:0/1984787990 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e88005e00 con 0x7f4e98105a40 2024-04-05T22:02:50.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.524+0000 7f4e9d41b700 1 -- 172.21.15.121:0/1984787990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9819b240 con 0x7f4e98105a40 2024-04-05T22:02:50.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.524+0000 7f4e7ffff700 1 -- 172.21.15.121:0/1984787990 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e8800fe60 con 0x7f4e98105a40 2024-04-05T22:02:50.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.524+0000 7f4e9d41b700 1 -- 172.21.15.121:0/1984787990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9819b730 con 0x7f4e98105a40 2024-04-05T22:02:50.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.525+0000 7f4e9d41b700 1 -- 172.21.15.121:0/1984787990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e98108fe0 con 0x7f4e98105a40 2024-04-05T22:02:50.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.525+0000 7f4e7ffff700 1 -- 172.21.15.121:0/1984787990 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4e8800f6d0 con 0x7f4e98105a40 2024-04-05T22:02:50.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.526+0000 7f4e7ffff700 1 --2- 172.21.15.121:0/1984787990 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4e8006cce0 0x7f4e8006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:50.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.526+0000 7f4e7ffff700 1 -- 172.21.15.121:0/1984787990 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4e8808d8b0 con 0x7f4e98105a40 2024-04-05T22:02:50.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.526+0000 7f4e96ffd700 1 --2- 172.21.15.121:0/1984787990 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4e8006cce0 0x7f4e8006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:50.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.527+0000 7f4e96ffd700 1 --2- 172.21.15.121:0/1984787990 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4e8006cce0 0x7f4e8006f1a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c005950 tx=0x7f4e8c0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:50.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.531+0000 7f4e7ffff700 1 -- 172.21.15.121:0/1984787990 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4e88057920 con 0x7f4e98105a40 2024-04-05T22:02:50.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.740+0000 7f4e9d41b700 1 -- 172.21.15.121:0/1984787990 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e9802cd60 con 0x7f4e8006cce0 2024-04-05T22:02:50.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.741+0000 7f4e7ffff700 1 -- 172.21.15.121:0/1984787990 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4e9802cd60 con 0x7f4e8006cce0 2024-04-05T22:02:50.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.743+0000 7f4e9d41b700 1 -- 172.21.15.121:0/1984787990 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4e8006cce0 msgr2=0x7f4e8006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:50.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.743+0000 7f4e9d41b700 1 --2- 172.21.15.121:0/1984787990 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4e8006cce0 0x7f4e8006f1a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c005950 tx=0x7f4e8c0058e0 comp rx=0 tx=0).stop 2024-04-05T22:02:50.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.743+0000 7f4e9d41b700 1 -- 172.21.15.121:0/1984787990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e98105a40 msgr2=0x7f4e98196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:50.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.743+0000 7f4e9d41b700 1 --2- 172.21.15.121:0/1984787990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e98105a40 0x7f4e98196b50 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f4e880092e0 tx=0x7f4e8800be20 comp rx=0 tx=0).stop 2024-04-05T22:02:50.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.744+0000 7f4e9d41b700 1 -- 172.21.15.121:0/1984787990 shutdown_connections 2024-04-05T22:02:50.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.744+0000 7f4e9d41b700 1 --2- 172.21.15.121:0/1984787990 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4e8006cce0 0x7f4e8006f1a0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:50.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.744+0000 7f4e9d41b700 1 --2- 172.21.15.121:0/1984787990 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e980ffa40 0x7f4e98196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:50.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.744+0000 7f4e9d41b700 1 --2- 172.21.15.121:0/1984787990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e98105a40 0x7f4e98196b50 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:50.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.744+0000 7f4e9d41b700 1 -- 172.21.15.121:0/1984787990 >> 172.21.15.121:0/1984787990 conn(0x7f4e980fb570 msgr2=0x7f4e98068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:50.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.744+0000 7f4e9d41b700 1 -- 172.21.15.121:0/1984787990 shutdown_connections 2024-04-05T22:02:50.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.744+0000 7f4e9d41b700 1 -- 172.21.15.121:0/1984787990 wait complete. 2024-04-05T22:02:50.754 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T22:02:50.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.847+0000 7f41c27e0700 1 -- 172.21.15.121:0/1189577037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41bc101770 msgr2=0x7f41bc1057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:50.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.847+0000 7f41c27e0700 1 --2- 172.21.15.121:0/1189577037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41bc101770 0x7f41bc1057c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009910 tx=0x7f41a4009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:50.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.847+0000 7f41c27e0700 1 -- 172.21.15.121:0/1189577037 shutdown_connections 2024-04-05T22:02:50.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.847+0000 7f41c27e0700 1 --2- 172.21.15.121:0/1189577037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41bc101770 0x7f41bc1057c0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:50.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.847+0000 7f41c27e0700 1 --2- 172.21.15.121:0/1189577037 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41bc100dc0 0x7f41bc1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:50.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.847+0000 7f41c27e0700 1 -- 172.21.15.121:0/1189577037 >> 172.21.15.121:0/1189577037 conn(0x7f41bc0fc650 msgr2=0x7f41bc0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.847+0000 7f41c27e0700 1 -- 172.21.15.121:0/1189577037 shutdown_connections 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.847+0000 7f41c27e0700 1 -- 172.21.15.121:0/1189577037 wait complete. 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.848+0000 7f41c27e0700 1 Processor -- start 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.848+0000 7f41c27e0700 1 -- start start 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.848+0000 7f41c27e0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41bc100dc0 0x7f41bc1966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.848+0000 7f41c27e0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41bc101770 0x7f41bc196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.848+0000 7f41c27e0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc1972a0 con 0x7f41bc100dc0 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.848+0000 7f41c27e0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41bc199fd0 con 0x7f41bc101770 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.848+0000 7f41bbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41bc100dc0 0x7f41bc1966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.848+0000 7f41bbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41bc100dc0 0x7f41bc1966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58466/0 (socket says 172.21.15.121:58466) 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.848+0000 7f41bbfff700 1 -- 172.21.15.121:0/3491613722 learned_addr learned my addr 172.21.15.121:0/3491613722 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.848+0000 7f41bb7fe700 1 --2- 172.21.15.121:0/3491613722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41bc101770 0x7f41bc196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.848+0000 7f41bbfff700 1 -- 172.21.15.121:0/3491613722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41bc101770 msgr2=0x7f41bc196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:50.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.848+0000 7f41bbfff700 1 --2- 172.21.15.121:0/3491613722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41bc101770 0x7f41bc196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:50.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.848+0000 7f41bbfff700 1 -- 172.21.15.121:0/3491613722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a40095f0 con 0x7f41bc100dc0 2024-04-05T22:02:50.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.849+0000 7f41bbfff700 1 --2- 172.21.15.121:0/3491613722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41bc100dc0 0x7f41bc1966d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00d8d0 tx=0x7f41ac00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:50.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.849+0000 7f41b97fa700 1 -- 172.21.15.121:0/3491613722 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41ac0098a0 con 0x7f41bc100dc0 2024-04-05T22:02:50.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.849+0000 7f41b97fa700 1 -- 172.21.15.121:0/3491613722 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41ac010460 con 0x7f41bc100dc0 2024-04-05T22:02:50.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.849+0000 7f41c27e0700 1 -- 172.21.15.121:0/3491613722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41bc19a2b0 con 0x7f41bc100dc0 2024-04-05T22:02:50.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.849+0000 7f41b97fa700 1 -- 172.21.15.121:0/3491613722 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41ac009ea0 con 0x7f41bc100dc0 2024-04-05T22:02:50.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.849+0000 7f41c27e0700 1 -- 172.21.15.121:0/3491613722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41bc19a7d0 con 0x7f41bc100dc0 2024-04-05T22:02:50.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.850+0000 7f41c27e0700 1 -- 172.21.15.121:0/3491613722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41bc1091c0 con 0x7f41bc100dc0 2024-04-05T22:02:50.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.850+0000 7f41b97fa700 1 -- 172.21.15.121:0/3491613722 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f41ac0105d0 con 0x7f41bc100dc0 2024-04-05T22:02:50.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.850+0000 7f41b97fa700 1 --2- 172.21.15.121:0/3491613722 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f41a806cce0 0x7f41a806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:50.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.851+0000 7f41b97fa700 1 -- 172.21.15.121:0/3491613722 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f41ac08c480 con 0x7f41bc100dc0 2024-04-05T22:02:50.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.851+0000 7f41bb7fe700 1 --2- 172.21.15.121:0/3491613722 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f41a806cce0 0x7f41a806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:50.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.851+0000 7f41bb7fe700 1 --2- 172.21.15.121:0/3491613722 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f41a806cce0 0x7f41a806f1a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f41bc197df0 tx=0x7f41a401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:50.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:50.855+0000 7f41b97fa700 1 -- 172.21.15.121:0/3491613722 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f41ac0564f0 con 0x7f41bc100dc0 2024-04-05T22:02:51.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.065+0000 7f41c27e0700 1 -- 172.21.15.121:0/3491613722 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f41bc19a9f0 con 0x7f41a806cce0 2024-04-05T22:02:51.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.067+0000 7f41b97fa700 1 -- 172.21.15.121:0/3491613722 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f41bc19a9f0 con 0x7f41a806cce0 2024-04-05T22:02:51.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.069+0000 7f41c27e0700 1 -- 172.21.15.121:0/3491613722 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f41a806cce0 msgr2=0x7f41a806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:51.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.069+0000 7f41c27e0700 1 --2- 172.21.15.121:0/3491613722 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f41a806cce0 0x7f41a806f1a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f41bc197df0 tx=0x7f41a401a040 comp rx=0 tx=0).stop 2024-04-05T22:02:51.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.069+0000 7f41c27e0700 1 -- 172.21.15.121:0/3491613722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41bc100dc0 msgr2=0x7f41bc1966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:51.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.069+0000 7f41c27e0700 1 --2- 172.21.15.121:0/3491613722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41bc100dc0 0x7f41bc1966d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00d8d0 tx=0x7f41ac00dbe0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.069+0000 7f41c27e0700 1 -- 172.21.15.121:0/3491613722 shutdown_connections 2024-04-05T22:02:51.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.069+0000 7f41c27e0700 1 --2- 172.21.15.121:0/3491613722 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f41a806cce0 0x7f41a806f1a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.069+0000 7f41c27e0700 1 --2- 172.21.15.121:0/3491613722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41bc100dc0 0x7f41bc1966d0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.069+0000 7f41c27e0700 1 --2- 172.21.15.121:0/3491613722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f41bc101770 0x7f41bc196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.069+0000 7f41c27e0700 1 -- 172.21.15.121:0/3491613722 >> 172.21.15.121:0/3491613722 conn(0x7f41bc0fc650 msgr2=0x7f41bc0fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:51.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.069+0000 7f41c27e0700 1 -- 172.21.15.121:0/3491613722 shutdown_connections 2024-04-05T22:02:51.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.069+0000 7f41c27e0700 1 -- 172.21.15.121:0/3491613722 wait complete. 2024-04-05T22:02:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.170+0000 7fae2797f700 1 -- 172.21.15.121:0/721185218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae20101c80 msgr2=0x7fae20102060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.170+0000 7fae2797f700 1 --2- 172.21.15.121:0/721185218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae20101c80 0x7fae20102060 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fae1c009910 tx=0x7fae1c009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:51.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.170+0000 7fae2797f700 1 -- 172.21.15.121:0/721185218 shutdown_connections 2024-04-05T22:02:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.170+0000 7fae2797f700 1 --2- 172.21.15.121:0/721185218 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae20102630 0x7fae2010ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.170+0000 7fae2797f700 1 --2- 172.21.15.121:0/721185218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae20101c80 0x7fae20102060 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.170+0000 7fae2797f700 1 -- 172.21.15.121:0/721185218 >> 172.21.15.121:0/721185218 conn(0x7fae200fb540 msgr2=0x7fae200fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.170+0000 7fae2797f700 1 -- 172.21.15.121:0/721185218 shutdown_connections 2024-04-05T22:02:51.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.170+0000 7fae2797f700 1 -- 172.21.15.121:0/721185218 wait complete. 2024-04-05T22:02:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.171+0000 7fae2797f700 1 Processor -- start 2024-04-05T22:02:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.171+0000 7fae2797f700 1 -- start start 2024-04-05T22:02:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.171+0000 7fae2797f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae20101c80 0x7fae20196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.171+0000 7fae2797f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae20102630 0x7fae20196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:51.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.171+0000 7fae2797f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae201971d0 con 0x7fae20102630 2024-04-05T22:02:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.171+0000 7fae2797f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae20197310 con 0x7fae20101c80 2024-04-05T22:02:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.172+0000 7fae24f1a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae20102630 0x7fae20196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.172+0000 7fae24f1a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae20102630 0x7fae20196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58476/0 (socket says 172.21.15.121:58476) 2024-04-05T22:02:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.172+0000 7fae24f1a700 1 -- 172.21.15.121:0/1546076607 learned_addr learned my addr 172.21.15.121:0/1546076607 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.172+0000 7fae2571b700 1 --2- 172.21.15.121:0/1546076607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae20101c80 0x7fae20196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.172+0000 7fae24f1a700 1 -- 172.21.15.121:0/1546076607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae20101c80 msgr2=0x7fae20196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.172+0000 7fae24f1a700 1 --2- 172.21.15.121:0/1546076607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae20101c80 0x7fae20196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.172+0000 7fae24f1a700 1 -- 172.21.15.121:0/1546076607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c0095f0 con 0x7fae20102630 2024-04-05T22:02:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.172+0000 7fae24f1a700 1 --2- 172.21.15.121:0/1546076607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae20102630 0x7fae20196b80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fae1000c960 tx=0x7fae1000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.172+0000 7fae167fc700 1 -- 172.21.15.121:0/1546076607 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae10007a70 con 0x7fae20102630 2024-04-05T22:02:51.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.172+0000 7fae2797f700 1 -- 172.21.15.121:0/1546076607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2019b160 con 0x7fae20102630 2024-04-05T22:02:51.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.172+0000 7fae167fc700 1 -- 172.21.15.121:0/1546076607 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae10007bd0 con 0x7fae20102630 2024-04-05T22:02:51.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.172+0000 7fae167fc700 1 -- 172.21.15.121:0/1546076607 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae10018630 con 0x7fae20102630 2024-04-05T22:02:51.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.172+0000 7fae2797f700 1 -- 172.21.15.121:0/1546076607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2019b6b0 con 0x7fae20102630 2024-04-05T22:02:51.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.173+0000 7fae2797f700 1 -- 172.21.15.121:0/1546076607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae20108220 con 0x7fae20102630 2024-04-05T22:02:51.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.174+0000 7fae167fc700 1 -- 172.21.15.121:0/1546076607 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fae1000f450 con 0x7fae20102630 2024-04-05T22:02:51.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.174+0000 7fae167fc700 1 --2- 172.21.15.121:0/1546076607 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fae0c06cce0 0x7fae0c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:51.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.174+0000 7fae167fc700 1 -- 172.21.15.121:0/1546076607 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fae1008c560 con 0x7fae20102630 2024-04-05T22:02:51.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.174+0000 7fae2571b700 1 --2- 172.21.15.121:0/1546076607 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fae0c06cce0 0x7fae0c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:51.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.174+0000 7fae2571b700 1 --2- 172.21.15.121:0/1546076607 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fae0c06cce0 0x7fae0c06f1a0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fae1c006010 tx=0x7fae1c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:51.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.179+0000 7fae167fc700 1 -- 172.21.15.121:0/1546076607 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fae100565d0 con 0x7fae20102630 2024-04-05T22:02:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:50 smithi188 ceph-mon[107793]: pgmap v1099: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:51.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.382+0000 7fae2797f700 1 -- 172.21.15.121:0/1546076607 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fae2002cd60 con 0x7fae0c06cce0 2024-04-05T22:02:51.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.392+0000 7fae167fc700 1 -- 172.21.15.121:0/1546076607 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fae2002cd60 con 0x7fae0c06cce0 2024-04-05T22:02:51.393 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T22:02:51.393 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (33m) 12m ago 45m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T22:02:51.393 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (45m) 12m ago 45m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T22:02:51.393 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (44m) 12m ago 44m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T22:02:51.393 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (31m) 12m ago 45m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T22:02:51.393 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (31m) 12m ago 44m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T22:02:51.393 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (33m) 12m ago 44m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T22:02:51.393 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (28m) 12m ago 39m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T22:02:51.394 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (27m) 12m ago 38m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T22:02:51.394 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (27m) 12m ago 39m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T22:02:51.394 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (12m) 12m ago 38m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T22:02:51.394 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (35m) 12m ago 46m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T22:02:51.394 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (34m) 12m ago 44m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T22:02:51.394 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (32m) 12m ago 46m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T22:02:51.394 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (32m) 12m ago 43m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T22:02:51.394 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (34m) 12m ago 45m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T22:02:51.394 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (34m) 12m ago 44m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T22:02:51.394 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (31m) 12m ago 43m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T22:02:51.395 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (30m) 12m ago 42m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T22:02:51.395 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (30m) 12m ago 42m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T22:02:51.395 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (29m) 12m ago 41m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T22:02:51.395 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (28m) 12m ago 41m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T22:02:51.395 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (28m) 12m ago 40m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T22:02:51.395 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (34m) 12m ago 44m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T22:02:51.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.395+0000 7fae2797f700 1 -- 172.21.15.121:0/1546076607 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fae0c06cce0 msgr2=0x7fae0c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:51.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.395+0000 7fae2797f700 1 --2- 172.21.15.121:0/1546076607 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fae0c06cce0 0x7fae0c06f1a0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fae1c006010 tx=0x7fae1c01a040 comp rx=0 tx=0).stop 2024-04-05T22:02:51.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.395+0000 7fae2797f700 1 -- 172.21.15.121:0/1546076607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae20102630 msgr2=0x7fae20196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:51.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.395+0000 7fae2797f700 1 --2- 172.21.15.121:0/1546076607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae20102630 0x7fae20196b80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fae1000c960 tx=0x7fae1000cc70 comp rx=0 tx=0).stop 2024-04-05T22:02:51.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.395+0000 7fae2797f700 1 -- 172.21.15.121:0/1546076607 shutdown_connections 2024-04-05T22:02:51.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.395+0000 7fae2797f700 1 --2- 172.21.15.121:0/1546076607 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fae0c06cce0 0x7fae0c06f1a0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.395+0000 7fae2797f700 1 --2- 172.21.15.121:0/1546076607 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fae20101c80 0x7fae20196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.395+0000 7fae2797f700 1 --2- 172.21.15.121:0/1546076607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae20102630 0x7fae20196b80 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.395+0000 7fae2797f700 1 -- 172.21.15.121:0/1546076607 >> 172.21.15.121:0/1546076607 conn(0x7fae200fb540 msgr2=0x7fae20105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:51.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.395+0000 7fae2797f700 1 -- 172.21.15.121:0/1546076607 shutdown_connections 2024-04-05T22:02:51.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.395+0000 7fae2797f700 1 -- 172.21.15.121:0/1546076607 wait complete. 2024-04-05T22:02:51.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:50 smithi121 ceph-mon[145859]: pgmap v1099: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:51.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.496+0000 7f5f019cd700 1 -- 172.21.15.121:0/3775796418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5efc1017c0 msgr2=0x7f5efc105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:51.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.496+0000 7f5f019cd700 1 --2- 172.21.15.121:0/3775796418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5efc1017c0 0x7f5efc105810 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009910 tx=0x7f5eec009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:51.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.496+0000 7f5f019cd700 1 -- 172.21.15.121:0/3775796418 shutdown_connections 2024-04-05T22:02:51.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.496+0000 7f5f019cd700 1 --2- 172.21.15.121:0/3775796418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5efc1017c0 0x7f5efc105810 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.496+0000 7f5f019cd700 1 --2- 172.21.15.121:0/3775796418 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5efc100e10 0x7f5efc1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.496+0000 7f5f019cd700 1 -- 172.21.15.121:0/3775796418 >> 172.21.15.121:0/3775796418 conn(0x7f5efc0fc6c0 msgr2=0x7f5efc0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:51.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.496+0000 7f5f019cd700 1 -- 172.21.15.121:0/3775796418 shutdown_connections 2024-04-05T22:02:51.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.496+0000 7f5f019cd700 1 -- 172.21.15.121:0/3775796418 wait complete. 2024-04-05T22:02:51.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.497+0000 7f5f019cd700 1 Processor -- start 2024-04-05T22:02:51.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.497+0000 7f5f019cd700 1 -- start start 2024-04-05T22:02:51.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.497+0000 7f5f019cd700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5efc100e10 0x7f5efc196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:51.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.497+0000 7f5f019cd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5efc1017c0 0x7f5efc196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:51.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.497+0000 7f5f019cd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc197260 con 0x7f5efc1017c0 2024-04-05T22:02:51.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.497+0000 7f5f019cd700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5efc199f90 con 0x7f5efc100e10 2024-04-05T22:02:51.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.497+0000 7f5efa7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5efc1017c0 0x7f5efc196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:51.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.497+0000 7f5efa7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5efc1017c0 0x7f5efc196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58506/0 (socket says 172.21.15.121:58506) 2024-04-05T22:02:51.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.497+0000 7f5efa7fc700 1 -- 172.21.15.121:0/1977617231 learned_addr learned my addr 172.21.15.121:0/1977617231 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:51.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.497+0000 7f5efaffd700 1 --2- 172.21.15.121:0/1977617231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5efc100e10 0x7f5efc196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:51.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.498+0000 7f5efa7fc700 1 -- 172.21.15.121:0/1977617231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5efc100e10 msgr2=0x7f5efc196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:51.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.498+0000 7f5efa7fc700 1 --2- 172.21.15.121:0/1977617231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5efc100e10 0x7f5efc196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.498+0000 7f5efa7fc700 1 -- 172.21.15.121:0/1977617231 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec0095f0 con 0x7f5efc1017c0 2024-04-05T22:02:51.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.498+0000 7f5efa7fc700 1 --2- 172.21.15.121:0/1977617231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5efc1017c0 0x7f5efc196bd0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f5eec000c00 tx=0x7f5eec00bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:51.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.498+0000 7f5f009cb700 1 -- 172.21.15.121:0/1977617231 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5eec01d070 con 0x7f5efc1017c0 2024-04-05T22:02:51.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.498+0000 7f5f009cb700 1 -- 172.21.15.121:0/1977617231 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5eec003cf0 con 0x7f5efc1017c0 2024-04-05T22:02:51.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.498+0000 7f5f019cd700 1 -- 172.21.15.121:0/1977617231 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5efc19a210 con 0x7f5efc1017c0 2024-04-05T22:02:51.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.498+0000 7f5f009cb700 1 -- 172.21.15.121:0/1977617231 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5eec021410 con 0x7f5efc1017c0 2024-04-05T22:02:51.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.498+0000 7f5f019cd700 1 -- 172.21.15.121:0/1977617231 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5efc19a6a0 con 0x7f5efc1017c0 2024-04-05T22:02:51.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.499+0000 7f5f019cd700 1 -- 172.21.15.121:0/1977617231 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5efc109210 con 0x7f5efc1017c0 2024-04-05T22:02:51.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.499+0000 7f5f009cb700 1 -- 172.21.15.121:0/1977617231 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5eec003850 con 0x7f5efc1017c0 2024-04-05T22:02:51.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.500+0000 7f5f009cb700 1 --2- 172.21.15.121:0/1977617231 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5ee406cce0 0x7f5ee406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:51.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.500+0000 7f5efaffd700 1 --2- 172.21.15.121:0/1977617231 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5ee406cce0 0x7f5ee406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:51.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.500+0000 7f5f009cb700 1 -- 172.21.15.121:0/1977617231 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5eec08d720 con 0x7f5efc1017c0 2024-04-05T22:02:51.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.500+0000 7f5efaffd700 1 --2- 172.21.15.121:0/1977617231 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5ee406cce0 0x7f5ee406f1a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0009bf0 tx=0x7f5ef0009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:51.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.504+0000 7f5f009cb700 1 -- 172.21.15.121:0/1977617231 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5eec057810 con 0x7f5efc1017c0 2024-04-05T22:02:51.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.781+0000 7f5f019cd700 1 -- 172.21.15.121:0/1977617231 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5efc04ea90 con 0x7f5efc1017c0 2024-04-05T22:02:51.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.782+0000 7f5f009cb700 1 -- 172.21.15.121:0/1977617231 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f5eec003e60 con 0x7f5efc1017c0 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:02:51.783 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T22:02:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T22:02:51.784 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T22:02:51.784 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:02:51.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.784+0000 7f5f019cd700 1 -- 172.21.15.121:0/1977617231 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5ee406cce0 msgr2=0x7f5ee406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:51.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.784+0000 7f5f019cd700 1 --2- 172.21.15.121:0/1977617231 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5ee406cce0 0x7f5ee406f1a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0009bf0 tx=0x7f5ef0009260 comp rx=0 tx=0).stop 2024-04-05T22:02:51.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.784+0000 7f5f019cd700 1 -- 172.21.15.121:0/1977617231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5efc1017c0 msgr2=0x7f5efc196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:51.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.784+0000 7f5f019cd700 1 --2- 172.21.15.121:0/1977617231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5efc1017c0 0x7f5efc196bd0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f5eec000c00 tx=0x7f5eec00bb20 comp rx=0 tx=0).stop 2024-04-05T22:02:51.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.784+0000 7f5f019cd700 1 -- 172.21.15.121:0/1977617231 shutdown_connections 2024-04-05T22:02:51.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.784+0000 7f5f019cd700 1 --2- 172.21.15.121:0/1977617231 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5ee406cce0 0x7f5ee406f1a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.784+0000 7f5f019cd700 1 --2- 172.21.15.121:0/1977617231 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5efc100e10 0x7f5efc196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.784+0000 7f5f019cd700 1 --2- 172.21.15.121:0/1977617231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5efc1017c0 0x7f5efc196bd0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.784+0000 7f5f019cd700 1 -- 172.21.15.121:0/1977617231 >> 172.21.15.121:0/1977617231 conn(0x7f5efc0fc6c0 msgr2=0x7f5efc0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:51.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.784+0000 7f5f019cd700 1 -- 172.21.15.121:0/1977617231 shutdown_connections 2024-04-05T22:02:51.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.784+0000 7f5f019cd700 1 -- 172.21.15.121:0/1977617231 wait complete. 2024-04-05T22:02:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.886+0000 7f91f8f4d700 1 -- 172.21.15.121:0/815747978 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f4101750 msgr2=0x7f91f41057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.886+0000 7f91f8f4d700 1 --2- 172.21.15.121:0/815747978 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f4101750 0x7f91f41057a0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f91e8009910 tx=0x7f91e8009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.886+0000 7f91f8f4d700 1 -- 172.21.15.121:0/815747978 shutdown_connections 2024-04-05T22:02:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.886+0000 7f91f8f4d700 1 --2- 172.21.15.121:0/815747978 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f4101750 0x7f91f41057a0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.886+0000 7f91f8f4d700 1 --2- 172.21.15.121:0/815747978 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f4100da0 0x7f91f4101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.886+0000 7f91f8f4d700 1 -- 172.21.15.121:0/815747978 >> 172.21.15.121:0/815747978 conn(0x7f91f40fc630 msgr2=0x7f91f40fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.886+0000 7f91f8f4d700 1 -- 172.21.15.121:0/815747978 shutdown_connections 2024-04-05T22:02:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.886+0000 7f91f8f4d700 1 -- 172.21.15.121:0/815747978 wait complete. 2024-04-05T22:02:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.887+0000 7f91f8f4d700 1 Processor -- start 2024-04-05T22:02:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.887+0000 7f91f8f4d700 1 -- start start 2024-04-05T22:02:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.887+0000 7f91f8f4d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f4100da0 0x7f91f4196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:51.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.887+0000 7f91f8f4d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f4101750 0x7f91f4196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:51.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.887+0000 7f91f8f4d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f4197270 con 0x7f91f4101750 2024-04-05T22:02:51.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.887+0000 7f91f8f4d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f419b000 con 0x7f91f4100da0 2024-04-05T22:02:51.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.887+0000 7f91f1d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f4101750 0x7f91f4196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:51.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.887+0000 7f91f1d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f4101750 0x7f91f4196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58518/0 (socket says 172.21.15.121:58518) 2024-04-05T22:02:51.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.887+0000 7f91f1d9b700 1 -- 172.21.15.121:0/457424646 learned_addr learned my addr 172.21.15.121:0/457424646 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:51.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.888+0000 7f91f259c700 1 --2- 172.21.15.121:0/457424646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f4100da0 0x7f91f4196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:51.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.888+0000 7f91f1d9b700 1 -- 172.21.15.121:0/457424646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f4100da0 msgr2=0x7f91f4196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:51.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.888+0000 7f91f1d9b700 1 --2- 172.21.15.121:0/457424646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f4100da0 0x7f91f4196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:51.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.888+0000 7f91f1d9b700 1 -- 172.21.15.121:0/457424646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e80095f0 con 0x7f91f4101750 2024-04-05T22:02:51.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.888+0000 7f91f259c700 1 --2- 172.21.15.121:0/457424646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f4100da0 0x7f91f4196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:02:51.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.888+0000 7f91f1d9b700 1 --2- 172.21.15.121:0/457424646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f4101750 0x7f91f4196b90 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f91e80092e0 tx=0x7f91e800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:51.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.888+0000 7f91e37fe700 1 -- 172.21.15.121:0/457424646 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91e801d070 con 0x7f91f4101750 2024-04-05T22:02:51.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.888+0000 7f91e37fe700 1 -- 172.21.15.121:0/457424646 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91e8005940 con 0x7f91f4101750 2024-04-05T22:02:51.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.888+0000 7f91e37fe700 1 -- 172.21.15.121:0/457424646 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91e800fe50 con 0x7f91f4101750 2024-04-05T22:02:51.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.888+0000 7f91f8f4d700 1 -- 172.21.15.121:0/457424646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f419b280 con 0x7f91f4101750 2024-04-05T22:02:51.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.888+0000 7f91f8f4d700 1 -- 172.21.15.121:0/457424646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f419b770 con 0x7f91f4101750 2024-04-05T22:02:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.889+0000 7f91f8f4d700 1 -- 172.21.15.121:0/457424646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f41091a0 con 0x7f91f4101750 2024-04-05T22:02:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.889+0000 7f91e37fe700 1 -- 172.21.15.121:0/457424646 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f91e8003a60 con 0x7f91f4101750 2024-04-05T22:02:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.890+0000 7f91e37fe700 1 --2- 172.21.15.121:0/457424646 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f91dc06cce0 0x7f91dc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.890+0000 7f91e37fe700 1 -- 172.21.15.121:0/457424646 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f91e808d7c0 con 0x7f91f4101750 2024-04-05T22:02:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.890+0000 7f91f259c700 1 --2- 172.21.15.121:0/457424646 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f91dc06cce0 0x7f91dc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.890+0000 7f91f259c700 1 --2- 172.21.15.121:0/457424646 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f91dc06cce0 0x7f91dc06f1a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f91e4005fd0 tx=0x7f91e4005ee0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:51.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:51.894+0000 7f91e37fe700 1 -- 172.21.15.121:0/457424646 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f91e8057830 con 0x7f91f4101750 2024-04-05T22:02:52.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.126+0000 7f91f8f4d700 1 -- 172.21.15.121:0/457424646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f91f404ea90 con 0x7f91f4101750 2024-04-05T22:02:52.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.126+0000 7f91e37fe700 1 -- 172.21.15.121:0/457424646 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f91e805ae50 con 0x7f91f4101750 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T22:02:52.127 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T22:02:52.128 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T22:02:52.128 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T22:02:52.128 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T22:02:52.128 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T22:02:52.128 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T22:02:52.128 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:02:52.128 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T22:02:52.128 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T22:02:52.128 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T22:02:52.128 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T22:02:52.128 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T22:02:52.128 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:02:52.129 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:02:52.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.128+0000 7f91f8f4d700 1 -- 172.21.15.121:0/457424646 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f91dc06cce0 msgr2=0x7f91dc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:52.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.128+0000 7f91f8f4d700 1 --2- 172.21.15.121:0/457424646 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f91dc06cce0 0x7f91dc06f1a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f91e4005fd0 tx=0x7f91e4005ee0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.128+0000 7f91f8f4d700 1 -- 172.21.15.121:0/457424646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f4101750 msgr2=0x7f91f4196b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:52.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.128+0000 7f91f8f4d700 1 --2- 172.21.15.121:0/457424646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f4101750 0x7f91f4196b90 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f91e80092e0 tx=0x7f91e800be20 comp rx=0 tx=0).stop 2024-04-05T22:02:52.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.128+0000 7f91f8f4d700 1 -- 172.21.15.121:0/457424646 shutdown_connections 2024-04-05T22:02:52.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.128+0000 7f91f8f4d700 1 --2- 172.21.15.121:0/457424646 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f91dc06cce0 0x7f91dc06f1a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.128+0000 7f91f8f4d700 1 --2- 172.21.15.121:0/457424646 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f91f4100da0 0x7f91f4196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.128+0000 7f91f8f4d700 1 --2- 172.21.15.121:0/457424646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f4101750 0x7f91f4196b90 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.128+0000 7f91f8f4d700 1 -- 172.21.15.121:0/457424646 >> 172.21.15.121:0/457424646 conn(0x7f91f40fc630 msgr2=0x7f91f40fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:52.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.128+0000 7f91f8f4d700 1 -- 172.21.15.121:0/457424646 shutdown_connections 2024-04-05T22:02:52.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.128+0000 7f91f8f4d700 1 -- 172.21.15.121:0/457424646 wait complete. 2024-04-05T22:02:52.131 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T22:02:52.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.230+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/1194035221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae4105a40 msgr2=0x7f7ae4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:52.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.230+0000 7f7aeaf1b700 1 --2- 172.21.15.121:0/1194035221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae4105a40 0x7f7ae4105e20 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8009910 tx=0x7f7ad8009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.231+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/1194035221 shutdown_connections 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.231+0000 7f7aeaf1b700 1 --2- 172.21.15.121:0/1194035221 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ae40ffa40 0x7f7ae40ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.231+0000 7f7aeaf1b700 1 --2- 172.21.15.121:0/1194035221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae4105a40 0x7f7ae4105e20 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.231+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/1194035221 >> 172.21.15.121:0/1194035221 conn(0x7f7ae40fb570 msgr2=0x7f7ae40fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.231+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/1194035221 shutdown_connections 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.231+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/1194035221 wait complete. 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.231+0000 7f7aeaf1b700 1 Processor -- start 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.231+0000 7f7aeaf1b700 1 -- start start 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.231+0000 7f7aeaf1b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ae40ffa40 0x7f7ae41974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.231+0000 7f7aeaf1b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae4105a40 0x7f7ae4197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.231+0000 7f7aeaf1b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae4198020 con 0x7f7ae4105a40 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.231+0000 7f7aeaf1b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae4198160 con 0x7f7ae40ffa40 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.232+0000 7f7ae3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae4105a40 0x7f7ae4197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.232+0000 7f7ae3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae4105a40 0x7f7ae4197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58542/0 (socket says 172.21.15.121:58542) 2024-04-05T22:02:52.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.232+0000 7f7ae3fff700 1 -- 172.21.15.121:0/3449411987 learned_addr learned my addr 172.21.15.121:0/3449411987 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:52.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.232+0000 7f7ae8cb7700 1 --2- 172.21.15.121:0/3449411987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ae40ffa40 0x7f7ae41974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:52.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.232+0000 7f7ae3fff700 1 -- 172.21.15.121:0/3449411987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ae40ffa40 msgr2=0x7f7ae41974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:52.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.232+0000 7f7ae3fff700 1 --2- 172.21.15.121:0/3449411987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ae40ffa40 0x7f7ae41974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.232+0000 7f7ae3fff700 1 -- 172.21.15.121:0/3449411987 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad80095f0 con 0x7f7ae4105a40 2024-04-05T22:02:52.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.232+0000 7f7ae3fff700 1 --2- 172.21.15.121:0/3449411987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae4105a40 0x7f7ae4197a00 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000d8d0 tx=0x7f7ad000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:52.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.232+0000 7f7ae1ffb700 1 -- 172.21.15.121:0/3449411987 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ad00098a0 con 0x7f7ae4105a40 2024-04-05T22:02:52.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.232+0000 7f7ae1ffb700 1 -- 172.21.15.121:0/3449411987 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ad0010460 con 0x7f7ae4105a40 2024-04-05T22:02:52.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.232+0000 7f7ae1ffb700 1 -- 172.21.15.121:0/3449411987 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ad0009ea0 con 0x7f7ae4105a40 2024-04-05T22:02:52.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.232+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/3449411987 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae41a0530 con 0x7f7ae4105a40 2024-04-05T22:02:52.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.232+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/3449411987 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae41a09f0 con 0x7f7ae4105a40 2024-04-05T22:02:52.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.233+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/3449411987 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae404ea90 con 0x7f7ae4105a40 2024-04-05T22:02:52.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.234+0000 7f7ae1ffb700 1 -- 172.21.15.121:0/3449411987 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7ad00105d0 con 0x7f7ae4105a40 2024-04-05T22:02:52.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.234+0000 7f7ae1ffb700 1 --2- 172.21.15.121:0/3449411987 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ad406cce0 0x7f7ad406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:52.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.234+0000 7f7ae1ffb700 1 -- 172.21.15.121:0/3449411987 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7ad008c480 con 0x7f7ae4105a40 2024-04-05T22:02:52.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.234+0000 7f7ae8cb7700 1 --2- 172.21.15.121:0/3449411987 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ad406cce0 0x7f7ad406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:52.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.234+0000 7f7ae8cb7700 1 --2- 172.21.15.121:0/3449411987 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ad406cce0 0x7f7ad406f1a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800b5c0 tx=0x7f7ad801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:52.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.239+0000 7f7ae1ffb700 1 -- 172.21.15.121:0/3449411987 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7ad00564f0 con 0x7f7ae4105a40 2024-04-05T22:02:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:52 smithi188 ceph-mon[107793]: from='client.35802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:52 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1977617231' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:02:52.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.450+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/3449411987 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ae40fc6f0 con 0x7f7ad406cce0 2024-04-05T22:02:52.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.451+0000 7f7ae1ffb700 1 -- 172.21.15.121:0/3449411987 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7ae40fc6f0 con 0x7f7ad406cce0 2024-04-05T22:02:52.452 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:02:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T22:02:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T22:02:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T22:02:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T22:02:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T22:02:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T22:02:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T22:02:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T22:02:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T22:02:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T22:02:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T22:02:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T22:02:52.453 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T22:02:52.453 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:02:52.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.453+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/3449411987 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ad406cce0 msgr2=0x7f7ad406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:52.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.454+0000 7f7aeaf1b700 1 --2- 172.21.15.121:0/3449411987 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ad406cce0 0x7f7ad406f1a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800b5c0 tx=0x7f7ad801a040 comp rx=0 tx=0).stop 2024-04-05T22:02:52.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.454+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/3449411987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae4105a40 msgr2=0x7f7ae4197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:52.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.454+0000 7f7aeaf1b700 1 --2- 172.21.15.121:0/3449411987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae4105a40 0x7f7ae4197a00 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000d8d0 tx=0x7f7ad000dbe0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.454+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/3449411987 shutdown_connections 2024-04-05T22:02:52.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.454+0000 7f7aeaf1b700 1 --2- 172.21.15.121:0/3449411987 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7ad406cce0 0x7f7ad406f1a0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.454+0000 7f7aeaf1b700 1 --2- 172.21.15.121:0/3449411987 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7ae40ffa40 0x7f7ae41974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.454+0000 7f7aeaf1b700 1 --2- 172.21.15.121:0/3449411987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae4105a40 0x7f7ae4197a00 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.454+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/3449411987 >> 172.21.15.121:0/3449411987 conn(0x7f7ae40fb570 msgr2=0x7f7ae40fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:52.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.454+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/3449411987 shutdown_connections 2024-04-05T22:02:52.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.454+0000 7f7aeaf1b700 1 -- 172.21.15.121:0/3449411987 wait complete. 2024-04-05T22:02:52.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:52 smithi121 ceph-mon[145859]: from='client.35802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:52.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:52 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1977617231' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:02:52.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.559+0000 7f68c7c57700 1 -- 172.21.15.121:0/2131828368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68c0101770 msgr2=0x7f68c01057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:52.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.559+0000 7f68c7c57700 1 --2- 172.21.15.121:0/2131828368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68c0101770 0x7f68c01057c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f68bc009910 tx=0x7f68bc009c20 comp rx=0 tx=0).stop 2024-04-05T22:02:52.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.559+0000 7f68c7c57700 1 -- 172.21.15.121:0/2131828368 shutdown_connections 2024-04-05T22:02:52.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.559+0000 7f68c7c57700 1 --2- 172.21.15.121:0/2131828368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68c0101770 0x7f68c01057c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.560+0000 7f68c7c57700 1 --2- 172.21.15.121:0/2131828368 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68c0100dc0 0x7f68c01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.560+0000 7f68c7c57700 1 -- 172.21.15.121:0/2131828368 >> 172.21.15.121:0/2131828368 conn(0x7f68c00fc650 msgr2=0x7f68c00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:52.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.560+0000 7f68c7c57700 1 -- 172.21.15.121:0/2131828368 shutdown_connections 2024-04-05T22:02:52.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.560+0000 7f68c7c57700 1 -- 172.21.15.121:0/2131828368 wait complete. 2024-04-05T22:02:52.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.560+0000 7f68c7c57700 1 Processor -- start 2024-04-05T22:02:52.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.560+0000 7f68c7c57700 1 -- start start 2024-04-05T22:02:52.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.560+0000 7f68c7c57700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68c0100dc0 0x7f68c0196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:52.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.560+0000 7f68c7c57700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68c0101770 0x7f68c0196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:52.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.560+0000 7f68c7c57700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c01972e0 con 0x7f68c0100dc0 2024-04-05T22:02:52.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.560+0000 7f68c7c57700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c019a010 con 0x7f68c0101770 2024-04-05T22:02:52.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.560+0000 7f68c59f3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68c0100dc0 0x7f68c0196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.561+0000 7f68c59f3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68c0100dc0 0x7f68c0196710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58556/0 (socket says 172.21.15.121:58556) 2024-04-05T22:02:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.561+0000 7f68c59f3700 1 -- 172.21.15.121:0/619929901 learned_addr learned my addr 172.21.15.121:0/619929901 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:02:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.561+0000 7f68c51f2700 1 --2- 172.21.15.121:0/619929901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68c0101770 0x7f68c0196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.561+0000 7f68c59f3700 1 -- 172.21.15.121:0/619929901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68c0101770 msgr2=0x7f68c0196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.561+0000 7f68c59f3700 1 --2- 172.21.15.121:0/619929901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68c0101770 0x7f68c0196c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.561+0000 7f68c59f3700 1 -- 172.21.15.121:0/619929901 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68bc0095f0 con 0x7f68c0100dc0 2024-04-05T22:02:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.561+0000 7f68c51f2700 1 --2- 172.21.15.121:0/619929901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68c0101770 0x7f68c0196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:02:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.561+0000 7f68c59f3700 1 --2- 172.21.15.121:0/619929901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68c0100dc0 0x7f68c0196710 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f68b000d8d0 tx=0x7f68b000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.561+0000 7f68b6ffd700 1 -- 172.21.15.121:0/619929901 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68b0009940 con 0x7f68c0100dc0 2024-04-05T22:02:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.561+0000 7f68b6ffd700 1 -- 172.21.15.121:0/619929901 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68b0010460 con 0x7f68c0100dc0 2024-04-05T22:02:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.561+0000 7f68c7c57700 1 -- 172.21.15.121:0/619929901 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c019a2f0 con 0x7f68c0100dc0 2024-04-05T22:02:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.561+0000 7f68b6ffd700 1 -- 172.21.15.121:0/619929901 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68b000f5f0 con 0x7f68c0100dc0 2024-04-05T22:02:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.561+0000 7f68c7c57700 1 -- 172.21.15.121:0/619929901 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c019a810 con 0x7f68c0100dc0 2024-04-05T22:02:52.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.562+0000 7f68c7c57700 1 -- 172.21.15.121:0/619929901 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c01091c0 con 0x7f68c0100dc0 2024-04-05T22:02:52.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.563+0000 7f68b6ffd700 1 -- 172.21.15.121:0/619929901 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f68b00105d0 con 0x7f68c0100dc0 2024-04-05T22:02:52.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.563+0000 7f68b6ffd700 1 --2- 172.21.15.121:0/619929901 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f68ac06cce0 0x7f68ac06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:02:52.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.563+0000 7f68b6ffd700 1 -- 172.21.15.121:0/619929901 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f68b008c5e0 con 0x7f68c0100dc0 2024-04-05T22:02:52.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.563+0000 7f68c51f2700 1 --2- 172.21.15.121:0/619929901 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f68ac06cce0 0x7f68ac06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:02:52.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.563+0000 7f68c51f2700 1 --2- 172.21.15.121:0/619929901 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f68ac06cce0 0x7f68ac06f1a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f68c0197e30 tx=0x7f68bc01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:02:52.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.568+0000 7f68b6ffd700 1 -- 172.21.15.121:0/619929901 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f68b0056650 con 0x7f68c0100dc0 2024-04-05T22:02:52.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.838+0000 7f68c7c57700 1 -- 172.21.15.121:0/619929901 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f68c004ea90 con 0x7f68c0100dc0 2024-04-05T22:02:52.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.838+0000 7f68b6ffd700 1 -- 172.21.15.121:0/619929901 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f68b0020e10 con 0x7f68c0100dc0 2024-04-05T22:02:52.839 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:02:52.839 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:02:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:02:52.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T22:02:52.839 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:02:52.839 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T22:02:52.839 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:02:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T22:02:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:02:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T22:02:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:02:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:02:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:02:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:02:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:02:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:02:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:02:52.840 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:02:52.840 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:02:52.841 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:02:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T22:02:52.841 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:02:52.841 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T22:02:52.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.840+0000 7f68c7c57700 1 -- 172.21.15.121:0/619929901 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f68ac06cce0 msgr2=0x7f68ac06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:52.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.840+0000 7f68c7c57700 1 --2- 172.21.15.121:0/619929901 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f68ac06cce0 0x7f68ac06f1a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f68c0197e30 tx=0x7f68bc01a040 comp rx=0 tx=0).stop 2024-04-05T22:02:52.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.840+0000 7f68c7c57700 1 -- 172.21.15.121:0/619929901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68c0100dc0 msgr2=0x7f68c0196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:02:52.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.840+0000 7f68c7c57700 1 --2- 172.21.15.121:0/619929901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68c0100dc0 0x7f68c0196710 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f68b000d8d0 tx=0x7f68b000dc90 comp rx=0 tx=0).stop 2024-04-05T22:02:52.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.840+0000 7f68c7c57700 1 -- 172.21.15.121:0/619929901 shutdown_connections 2024-04-05T22:02:52.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.840+0000 7f68c7c57700 1 --2- 172.21.15.121:0/619929901 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f68ac06cce0 0x7f68ac06f1a0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.840+0000 7f68c7c57700 1 --2- 172.21.15.121:0/619929901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68c0100dc0 0x7f68c0196710 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.840+0000 7f68c7c57700 1 --2- 172.21.15.121:0/619929901 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f68c0101770 0x7f68c0196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:02:52.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.840+0000 7f68c7c57700 1 -- 172.21.15.121:0/619929901 >> 172.21.15.121:0/619929901 conn(0x7f68c00fc650 msgr2=0x7f68c00fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:02:52.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.841+0000 7f68c7c57700 1 -- 172.21.15.121:0/619929901 shutdown_connections 2024-04-05T22:02:52.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:02:52.841+0000 7f68c7c57700 1 -- 172.21.15.121:0/619929901 wait complete. 2024-04-05T22:02:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:53 smithi188 ceph-mon[107793]: from='client.35806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:53 smithi188 ceph-mon[107793]: from='client.35810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:53 smithi188 ceph-mon[107793]: pgmap v1100: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:53 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/457424646' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:02:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:53 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/619929901' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:02:53.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:53 smithi121 ceph-mon[145859]: from='client.35806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:53.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:53 smithi121 ceph-mon[145859]: from='client.35810 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:53.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:53 smithi121 ceph-mon[145859]: pgmap v1100: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:53.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:53 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/457424646' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:02:53.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:53 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/619929901' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:02:54.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:54 smithi188 ceph-mon[107793]: from='client.35822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:54.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:54 smithi121 ceph-mon[145859]: from='client.35822 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:02:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:55 smithi188 ceph-mon[107793]: pgmap v1101: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:55.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:55 smithi121 ceph-mon[145859]: pgmap v1101: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:57 smithi188 ceph-mon[107793]: pgmap v1102: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:57.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:57 smithi121 ceph-mon[145859]: pgmap v1102: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:02:59 smithi188 ceph-mon[107793]: pgmap v1103: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:02:59.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:02:59 smithi121 ceph-mon[145859]: pgmap v1103: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:01 smithi188 ceph-mon[107793]: pgmap v1104: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:03:01.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:01 smithi121 ceph-mon[145859]: pgmap v1104: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:01.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:03:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:03 smithi188 ceph-mon[107793]: pgmap v1105: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:03.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:03 smithi121 ceph-mon[145859]: pgmap v1105: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:05 smithi188 ceph-mon[107793]: pgmap v1106: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:05 smithi121 ceph-mon[145859]: pgmap v1106: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:07.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:07 smithi121 ceph-mon[145859]: pgmap v1107: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:07.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:07 smithi188 ceph-mon[107793]: pgmap v1107: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:09.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:09 smithi121 ceph-mon[145859]: pgmap v1108: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:09.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:09 smithi188 ceph-mon[107793]: pgmap v1108: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:11.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:11 smithi121 ceph-mon[145859]: pgmap v1109: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:11.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:11 smithi188 ceph-mon[107793]: pgmap v1109: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:12.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:12 smithi121 ceph-mon[145859]: pgmap v1110: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:12.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:12 smithi188 ceph-mon[107793]: pgmap v1110: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:15.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:14 smithi121 ceph-mon[145859]: pgmap v1111: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:14 smithi188 ceph-mon[107793]: pgmap v1111: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:03:16.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:03:17.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:16 smithi121 ceph-mon[145859]: pgmap v1112: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:17.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:16 smithi188 ceph-mon[107793]: pgmap v1112: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:19.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:18 smithi121 ceph-mon[145859]: pgmap v1113: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:19.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:18 smithi188 ceph-mon[107793]: pgmap v1113: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:21.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:20 smithi121 ceph-mon[145859]: pgmap v1114: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:20 smithi188 ceph-mon[107793]: pgmap v1114: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:22.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.957+0000 7f8d56406700 1 -- 172.21.15.121:0/3916962715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50101d00 msgr2=0x7f8d501020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:22.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.957+0000 7f8d56406700 1 --2- 172.21.15.121:0/3916962715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50101d00 0x7f8d501020e0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009910 tx=0x7f8d40009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:22.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.958+0000 7f8d56406700 1 -- 172.21.15.121:0/3916962715 shutdown_connections 2024-04-05T22:03:22.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.958+0000 7f8d56406700 1 --2- 172.21.15.121:0/3916962715 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d50102620 0x7f8d5010ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:22.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.958+0000 7f8d56406700 1 --2- 172.21.15.121:0/3916962715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50101d00 0x7f8d501020e0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:22.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.958+0000 7f8d56406700 1 -- 172.21.15.121:0/3916962715 >> 172.21.15.121:0/3916962715 conn(0x7f8d500fb570 msgr2=0x7f8d500fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:22.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.958+0000 7f8d56406700 1 -- 172.21.15.121:0/3916962715 shutdown_connections 2024-04-05T22:03:22.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.958+0000 7f8d56406700 1 -- 172.21.15.121:0/3916962715 wait complete. 2024-04-05T22:03:22.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.958+0000 7f8d56406700 1 Processor -- start 2024-04-05T22:03:22.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.958+0000 7f8d56406700 1 -- start start 2024-04-05T22:03:22.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d56406700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d50101d00 0x7f8d50196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:22.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d56406700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50102620 0x7f8d50196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:22.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d56406700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d50197250 con 0x7f8d50102620 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d56406700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d50199f80 con 0x7f8d50101d00 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d4f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50102620 0x7f8d50196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d4f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50102620 0x7f8d50196bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54352/0 (socket says 172.21.15.121:54352) 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d4f7fe700 1 -- 172.21.15.121:0/4207838048 learned_addr learned my addr 172.21.15.121:0/4207838048 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d4ffff700 1 --2- 172.21.15.121:0/4207838048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d50101d00 0x7f8d50196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d4f7fe700 1 -- 172.21.15.121:0/4207838048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d50101d00 msgr2=0x7f8d50196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d4f7fe700 1 --2- 172.21.15.121:0/4207838048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d50101d00 0x7f8d50196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d4f7fe700 1 -- 172.21.15.121:0/4207838048 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d400095f0 con 0x7f8d50102620 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d4f7fe700 1 --2- 172.21.15.121:0/4207838048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50102620 0x7f8d50196bc0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400d6d0 tx=0x7f8d4400d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d4d7fa700 1 -- 172.21.15.121:0/4207838048 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d4400f860 con 0x7f8d50102620 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d4d7fa700 1 -- 172.21.15.121:0/4207838048 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d4400fe60 con 0x7f8d50102620 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d56406700 1 -- 172.21.15.121:0/4207838048 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5019a260 con 0x7f8d50102620 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d4d7fa700 1 -- 172.21.15.121:0/4207838048 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d4400e3b0 con 0x7f8d50102620 2024-04-05T22:03:22.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.959+0000 7f8d56406700 1 -- 172.21.15.121:0/4207838048 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5019a7b0 con 0x7f8d50102620 2024-04-05T22:03:22.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.960+0000 7f8d56406700 1 -- 172.21.15.121:0/4207838048 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d50108210 con 0x7f8d50102620 2024-04-05T22:03:22.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.961+0000 7f8d4d7fa700 1 -- 172.21.15.121:0/4207838048 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8d44010460 con 0x7f8d50102620 2024-04-05T22:03:22.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.961+0000 7f8d4d7fa700 1 --2- 172.21.15.121:0/4207838048 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d3806cce0 0x7f8d3806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:22.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.961+0000 7f8d4d7fa700 1 -- 172.21.15.121:0/4207838048 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8d4408c1f0 con 0x7f8d50102620 2024-04-05T22:03:22.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.961+0000 7f8d4ffff700 1 --2- 172.21.15.121:0/4207838048 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d3806cce0 0x7f8d3806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:22.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.962+0000 7f8d4ffff700 1 --2- 172.21.15.121:0/4207838048 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d3806cce0 0x7f8d3806f1a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8d4000b5c0 tx=0x7f8d4001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:22.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:22.966+0000 7f8d4d7fa700 1 -- 172.21.15.121:0/4207838048 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8d440562e0 con 0x7f8d50102620 2024-04-05T22:03:23.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.174+0000 7f8d56406700 1 -- 172.21.15.121:0/4207838048 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d5019ab40 con 0x7f8d3806cce0 2024-04-05T22:03:23.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.175+0000 7f8d4d7fa700 1 -- 172.21.15.121:0/4207838048 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8d5019ab40 con 0x7f8d3806cce0 2024-04-05T22:03:23.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.177+0000 7f8d56406700 1 -- 172.21.15.121:0/4207838048 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d3806cce0 msgr2=0x7f8d3806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:23.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.177+0000 7f8d56406700 1 --2- 172.21.15.121:0/4207838048 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d3806cce0 0x7f8d3806f1a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8d4000b5c0 tx=0x7f8d4001a040 comp rx=0 tx=0).stop 2024-04-05T22:03:23.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.177+0000 7f8d56406700 1 -- 172.21.15.121:0/4207838048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50102620 msgr2=0x7f8d50196bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:23.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.177+0000 7f8d56406700 1 --2- 172.21.15.121:0/4207838048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50102620 0x7f8d50196bc0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400d6d0 tx=0x7f8d4400d9e0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.178+0000 7f8d56406700 1 -- 172.21.15.121:0/4207838048 shutdown_connections 2024-04-05T22:03:23.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.178+0000 7f8d56406700 1 --2- 172.21.15.121:0/4207838048 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d3806cce0 0x7f8d3806f1a0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.178+0000 7f8d56406700 1 --2- 172.21.15.121:0/4207838048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d50101d00 0x7f8d50196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.178+0000 7f8d56406700 1 --2- 172.21.15.121:0/4207838048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50102620 0x7f8d50196bc0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.178+0000 7f8d56406700 1 -- 172.21.15.121:0/4207838048 >> 172.21.15.121:0/4207838048 conn(0x7f8d500fb570 msgr2=0x7f8d50105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:23.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.178+0000 7f8d56406700 1 -- 172.21.15.121:0/4207838048 shutdown_connections 2024-04-05T22:03:23.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.178+0000 7f8d56406700 1 -- 172.21.15.121:0/4207838048 wait complete. 2024-04-05T22:03:23.188 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T22:03:23.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:22 smithi121 ceph-mon[145859]: pgmap v1115: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:23.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:22 smithi188 ceph-mon[107793]: pgmap v1115: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:23.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.281+0000 7f947f9c0700 1 -- 172.21.15.121:0/2998143597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9478105b50 msgr2=0x7f9478105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:23.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.281+0000 7f947f9c0700 1 --2- 172.21.15.121:0/2998143597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9478105b50 0x7f9478105f30 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f9474009910 tx=0x7f9474009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:23.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.281+0000 7f947f9c0700 1 -- 172.21.15.121:0/2998143597 shutdown_connections 2024-04-05T22:03:23.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.281+0000 7f947f9c0700 1 --2- 172.21.15.121:0/2998143597 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f94780fef70 0x7f94780ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.281+0000 7f947f9c0700 1 --2- 172.21.15.121:0/2998143597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9478105b50 0x7f9478105f30 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.281+0000 7f947f9c0700 1 -- 172.21.15.121:0/2998143597 >> 172.21.15.121:0/2998143597 conn(0x7f9478075e30 msgr2=0x7f94780fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:23.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.281+0000 7f947f9c0700 1 -- 172.21.15.121:0/2998143597 shutdown_connections 2024-04-05T22:03:23.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.281+0000 7f947f9c0700 1 -- 172.21.15.121:0/2998143597 wait complete. 2024-04-05T22:03:23.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.282+0000 7f947f9c0700 1 Processor -- start 2024-04-05T22:03:23.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.282+0000 7f947f9c0700 1 -- start start 2024-04-05T22:03:23.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.282+0000 7f947f9c0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94780fef70 0x7f947819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.282+0000 7f947f9c0700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9478105b50 0x7f947819ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.282+0000 7f947f9c0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947819cfe0 con 0x7f94780fef70 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.282+0000 7f947f9c0700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9478196590 con 0x7f9478105b50 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.282+0000 7f947d75c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94780fef70 0x7f947819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.282+0000 7f947d75c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94780fef70 0x7f947819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54370/0 (socket says 172.21.15.121:54370) 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.282+0000 7f947d75c700 1 -- 172.21.15.121:0/2859041384 learned_addr learned my addr 172.21.15.121:0/2859041384 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.282+0000 7f947d75c700 1 -- 172.21.15.121:0/2859041384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9478105b50 msgr2=0x7f947819ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.282+0000 7f947d75c700 1 --2- 172.21.15.121:0/2859041384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9478105b50 0x7f947819ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.282+0000 7f947d75c700 1 -- 172.21.15.121:0/2859041384 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94740095f0 con 0x7f94780fef70 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.283+0000 7f947d75c700 1 --2- 172.21.15.121:0/2859041384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94780fef70 0x7f947819c510 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f9474006010 tx=0x7f947400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.283+0000 7f946e7fc700 1 -- 172.21.15.121:0/2859041384 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f947401d070 con 0x7f94780fef70 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.283+0000 7f946e7fc700 1 -- 172.21.15.121:0/2859041384 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f947400f460 con 0x7f94780fef70 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.283+0000 7f947f9c0700 1 -- 172.21.15.121:0/2859041384 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9478196810 con 0x7f94780fef70 2024-04-05T22:03:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.283+0000 7f946e7fc700 1 -- 172.21.15.121:0/2859041384 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9474021640 con 0x7f94780fef70 2024-04-05T22:03:23.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.283+0000 7f947f9c0700 1 -- 172.21.15.121:0/2859041384 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9478196d00 con 0x7f94780fef70 2024-04-05T22:03:23.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.284+0000 7f947f9c0700 1 -- 172.21.15.121:0/2859041384 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9478109120 con 0x7f94780fef70 2024-04-05T22:03:23.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.284+0000 7f946e7fc700 1 -- 172.21.15.121:0/2859041384 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f947400fa20 con 0x7f94780fef70 2024-04-05T22:03:23.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.285+0000 7f946e7fc700 1 --2- 172.21.15.121:0/2859041384 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f946406cce0 0x7f946406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:23.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.285+0000 7f946e7fc700 1 -- 172.21.15.121:0/2859041384 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f947408d8c0 con 0x7f94780fef70 2024-04-05T22:03:23.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.285+0000 7f947cf5b700 1 --2- 172.21.15.121:0/2859041384 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f946406cce0 0x7f946406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:23.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.286+0000 7f947cf5b700 1 --2- 172.21.15.121:0/2859041384 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f946406cce0 0x7f946406f1a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9468005fd0 tx=0x7f9468005d00 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:23.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.290+0000 7f946e7fc700 1 -- 172.21.15.121:0/2859041384 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f947405b3c0 con 0x7f94780fef70 2024-04-05T22:03:23.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.499+0000 7f947f9c0700 1 -- 172.21.15.121:0/2859041384 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9478196f20 con 0x7f946406cce0 2024-04-05T22:03:23.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.500+0000 7f946e7fc700 1 -- 172.21.15.121:0/2859041384 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9478196f20 con 0x7f946406cce0 2024-04-05T22:03:23.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.502+0000 7f947f9c0700 1 -- 172.21.15.121:0/2859041384 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f946406cce0 msgr2=0x7f946406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:23.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.502+0000 7f947f9c0700 1 --2- 172.21.15.121:0/2859041384 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f946406cce0 0x7f946406f1a0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9468005fd0 tx=0x7f9468005d00 comp rx=0 tx=0).stop 2024-04-05T22:03:23.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.502+0000 7f947f9c0700 1 -- 172.21.15.121:0/2859041384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94780fef70 msgr2=0x7f947819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:23.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.502+0000 7f947f9c0700 1 --2- 172.21.15.121:0/2859041384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94780fef70 0x7f947819c510 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f9474006010 tx=0x7f947400be20 comp rx=0 tx=0).stop 2024-04-05T22:03:23.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.503+0000 7f947f9c0700 1 -- 172.21.15.121:0/2859041384 shutdown_connections 2024-04-05T22:03:23.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.503+0000 7f947f9c0700 1 --2- 172.21.15.121:0/2859041384 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f946406cce0 0x7f946406f1a0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.503+0000 7f947f9c0700 1 --2- 172.21.15.121:0/2859041384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94780fef70 0x7f947819c510 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.503+0000 7f947f9c0700 1 --2- 172.21.15.121:0/2859041384 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9478105b50 0x7f947819ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.503+0000 7f947f9c0700 1 -- 172.21.15.121:0/2859041384 >> 172.21.15.121:0/2859041384 conn(0x7f9478075e30 msgr2=0x7f9478104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:23.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.503+0000 7f947f9c0700 1 -- 172.21.15.121:0/2859041384 shutdown_connections 2024-04-05T22:03:23.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.503+0000 7f947f9c0700 1 -- 172.21.15.121:0/2859041384 wait complete. 2024-04-05T22:03:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.605+0000 7f38f5fa2700 1 -- 172.21.15.121:0/3587614316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38f0105880 msgr2=0x7f38f0105c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.605+0000 7f38f5fa2700 1 --2- 172.21.15.121:0/3587614316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38f0105880 0x7f38f0105c60 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f38e4009910 tx=0x7f38e4009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.605+0000 7f38f5fa2700 1 -- 172.21.15.121:0/3587614316 shutdown_connections 2024-04-05T22:03:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.605+0000 7f38f5fa2700 1 --2- 172.21.15.121:0/3587614316 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38f00ffa10 0x7f38f00ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.605+0000 7f38f5fa2700 1 --2- 172.21.15.121:0/3587614316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38f0105880 0x7f38f0105c60 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.605+0000 7f38f5fa2700 1 -- 172.21.15.121:0/3587614316 >> 172.21.15.121:0/3587614316 conn(0x7f38f0075330 msgr2=0x7f38f0075740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.605+0000 7f38f5fa2700 1 -- 172.21.15.121:0/3587614316 shutdown_connections 2024-04-05T22:03:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.605+0000 7f38f5fa2700 1 -- 172.21.15.121:0/3587614316 wait complete. 2024-04-05T22:03:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.606+0000 7f38f5fa2700 1 Processor -- start 2024-04-05T22:03:23.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.606+0000 7f38f5fa2700 1 -- start start 2024-04-05T22:03:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.606+0000 7f38f5fa2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38f00ffa10 0x7f38f01964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.606+0000 7f38f5fa2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38f0105880 0x7f38f0196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.606+0000 7f38f5fa2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f01970f0 con 0x7f38f00ffa10 2024-04-05T22:03:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.606+0000 7f38f5fa2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f019ae80 con 0x7f38f0105880 2024-04-05T22:03:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.606+0000 7f38f4fa0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38f00ffa10 0x7f38f01964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.606+0000 7f38f4fa0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38f00ffa10 0x7f38f01964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54380/0 (socket says 172.21.15.121:54380) 2024-04-05T22:03:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.606+0000 7f38f4fa0700 1 -- 172.21.15.121:0/4049891704 learned_addr learned my addr 172.21.15.121:0/4049891704 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.606+0000 7f38effff700 1 --2- 172.21.15.121:0/4049891704 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38f0105880 0x7f38f0196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.606+0000 7f38f4fa0700 1 -- 172.21.15.121:0/4049891704 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38f0105880 msgr2=0x7f38f0196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.606+0000 7f38f4fa0700 1 --2- 172.21.15.121:0/4049891704 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38f0105880 0x7f38f0196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.606+0000 7f38f4fa0700 1 -- 172.21.15.121:0/4049891704 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e40095f0 con 0x7f38f00ffa10 2024-04-05T22:03:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.607+0000 7f38f4fa0700 1 --2- 172.21.15.121:0/4049891704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38f00ffa10 0x7f38f01964d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f38e4006010 tx=0x7f38e400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.607+0000 7f38edffb700 1 -- 172.21.15.121:0/4049891704 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38e401d070 con 0x7f38f00ffa10 2024-04-05T22:03:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.607+0000 7f38edffb700 1 -- 172.21.15.121:0/4049891704 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38e4005940 con 0x7f38f00ffa10 2024-04-05T22:03:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.607+0000 7f38f5fa2700 1 -- 172.21.15.121:0/4049891704 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f019b100 con 0x7f38f00ffa10 2024-04-05T22:03:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.607+0000 7f38edffb700 1 -- 172.21.15.121:0/4049891704 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38e400fd90 con 0x7f38f00ffa10 2024-04-05T22:03:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.607+0000 7f38f5fa2700 1 -- 172.21.15.121:0/4049891704 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f019b5f0 con 0x7f38f00ffa10 2024-04-05T22:03:23.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.608+0000 7f38f5fa2700 1 -- 172.21.15.121:0/4049891704 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f0108e90 con 0x7f38f00ffa10 2024-04-05T22:03:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.608+0000 7f38edffb700 1 -- 172.21.15.121:0/4049891704 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f38e4003a60 con 0x7f38f00ffa10 2024-04-05T22:03:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.609+0000 7f38edffb700 1 --2- 172.21.15.121:0/4049891704 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f38e006cce0 0x7f38e006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.609+0000 7f38edffb700 1 -- 172.21.15.121:0/4049891704 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f38e408d770 con 0x7f38f00ffa10 2024-04-05T22:03:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.609+0000 7f38effff700 1 --2- 172.21.15.121:0/4049891704 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f38e006cce0 0x7f38e006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:23.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.609+0000 7f38effff700 1 --2- 172.21.15.121:0/4049891704 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f38e006cce0 0x7f38e006f1a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f38f0197af0 tx=0x7f38dc00b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:23.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.613+0000 7f38edffb700 1 -- 172.21.15.121:0/4049891704 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f38e40577e0 con 0x7f38f00ffa10 2024-04-05T22:03:23.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.814+0000 7f38f5fa2700 1 -- 172.21.15.121:0/4049891704 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f38f002cd30 con 0x7f38e006cce0 2024-04-05T22:03:23.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.825+0000 7f38edffb700 1 -- 172.21.15.121:0/4049891704 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f38f002cd30 con 0x7f38e006cce0 2024-04-05T22:03:23.825 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T22:03:23.825 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (34m) 12m ago 45m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T22:03:23.825 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (45m) 12m ago 45m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T22:03:23.825 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (44m) 12m ago 44m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T22:03:23.825 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (32m) 12m ago 45m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T22:03:23.825 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (32m) 12m ago 44m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T22:03:23.825 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (33m) 12m ago 45m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T22:03:23.825 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (28m) 12m ago 39m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T22:03:23.825 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (28m) 12m ago 39m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T22:03:23.825 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (27m) 12m ago 39m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T22:03:23.825 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (12m) 12m ago 39m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T22:03:23.825 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (36m) 12m ago 47m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T22:03:23.826 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (35m) 12m ago 44m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T22:03:23.826 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (32m) 12m ago 47m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T22:03:23.826 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (32m) 12m ago 44m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T22:03:23.826 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (35m) 12m ago 45m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T22:03:23.826 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (34m) 12m ago 44m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T22:03:23.826 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (31m) 12m ago 43m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T22:03:23.826 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (31m) 12m ago 43m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T22:03:23.826 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (30m) 12m ago 42m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T22:03:23.826 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (30m) 12m ago 42m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T22:03:23.826 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (29m) 12m ago 41m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T22:03:23.826 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (28m) 12m ago 41m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T22:03:23.826 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (34m) 12m ago 45m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T22:03:23.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.827+0000 7f38f5fa2700 1 -- 172.21.15.121:0/4049891704 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f38e006cce0 msgr2=0x7f38e006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:23.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.827+0000 7f38f5fa2700 1 --2- 172.21.15.121:0/4049891704 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f38e006cce0 0x7f38e006f1a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f38f0197af0 tx=0x7f38dc00b410 comp rx=0 tx=0).stop 2024-04-05T22:03:23.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.827+0000 7f38f5fa2700 1 -- 172.21.15.121:0/4049891704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38f00ffa10 msgr2=0x7f38f01964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:23.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.827+0000 7f38f5fa2700 1 --2- 172.21.15.121:0/4049891704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38f00ffa10 0x7f38f01964d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f38e4006010 tx=0x7f38e400bd70 comp rx=0 tx=0).stop 2024-04-05T22:03:23.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.827+0000 7f38f5fa2700 1 -- 172.21.15.121:0/4049891704 shutdown_connections 2024-04-05T22:03:23.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.827+0000 7f38f5fa2700 1 --2- 172.21.15.121:0/4049891704 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f38e006cce0 0x7f38e006f1a0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.827+0000 7f38f5fa2700 1 --2- 172.21.15.121:0/4049891704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38f00ffa10 0x7f38f01964d0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.827+0000 7f38f5fa2700 1 --2- 172.21.15.121:0/4049891704 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38f0105880 0x7f38f0196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.827+0000 7f38f5fa2700 1 -- 172.21.15.121:0/4049891704 >> 172.21.15.121:0/4049891704 conn(0x7f38f0075330 msgr2=0x7f38f0068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:23.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.827+0000 7f38f5fa2700 1 -- 172.21.15.121:0/4049891704 shutdown_connections 2024-04-05T22:03:23.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.827+0000 7f38f5fa2700 1 -- 172.21.15.121:0/4049891704 wait complete. 2024-04-05T22:03:23.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.928+0000 7fd101b2d700 1 -- 172.21.15.121:0/2824798131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc0ffbd0 msgr2=0x7fd0fc100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:23.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.928+0000 7fd101b2d700 1 --2- 172.21.15.121:0/2824798131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc0ffbd0 0x7fd0fc100050 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009910 tx=0x7fd0ec009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.928+0000 7fd101b2d700 1 -- 172.21.15.121:0/2824798131 shutdown_connections 2024-04-05T22:03:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.928+0000 7fd101b2d700 1 --2- 172.21.15.121:0/2824798131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc0ffbd0 0x7fd0fc100050 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.928+0000 7fd101b2d700 1 --2- 172.21.15.121:0/2824798131 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0fc105a40 0x7fd0fc105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.928+0000 7fd101b2d700 1 -- 172.21.15.121:0/2824798131 >> 172.21.15.121:0/2824798131 conn(0x7fd0fc0755a0 msgr2=0x7fd0fc0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.928+0000 7fd101b2d700 1 -- 172.21.15.121:0/2824798131 shutdown_connections 2024-04-05T22:03:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.928+0000 7fd101b2d700 1 -- 172.21.15.121:0/2824798131 wait complete. 2024-04-05T22:03:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.928+0000 7fd101b2d700 1 Processor -- start 2024-04-05T22:03:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.929+0000 7fd101b2d700 1 -- start start 2024-04-05T22:03:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.929+0000 7fd101b2d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc0ffbd0 0x7fd0fc195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.929+0000 7fd101b2d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0fc105a40 0x7fd0fc195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.929+0000 7fd101b2d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc196160 con 0x7fd0fc0ffbd0 2024-04-05T22:03:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.929+0000 7fd101b2d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc199ef0 con 0x7fd0fc105a40 2024-04-05T22:03:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.929+0000 7fd0fb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc0ffbd0 0x7fd0fc195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.929+0000 7fd0fb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc0ffbd0 0x7fd0fc195540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54392/0 (socket says 172.21.15.121:54392) 2024-04-05T22:03:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.929+0000 7fd0fb7fe700 1 -- 172.21.15.121:0/1858303716 learned_addr learned my addr 172.21.15.121:0/1858303716 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.929+0000 7fd0fb7fe700 1 -- 172.21.15.121:0/1858303716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0fc105a40 msgr2=0x7fd0fc195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.929+0000 7fd0fb7fe700 1 --2- 172.21.15.121:0/1858303716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0fc105a40 0x7fd0fc195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.929+0000 7fd0fb7fe700 1 -- 172.21.15.121:0/1858303716 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec0095f0 con 0x7fd0fc0ffbd0 2024-04-05T22:03:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.929+0000 7fd0fb7fe700 1 --2- 172.21.15.121:0/1858303716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc0ffbd0 0x7fd0fc195540 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400d8d0 tx=0x7fd0e400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.929+0000 7fd0f8ff9700 1 -- 172.21.15.121:0/1858303716 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0e4009940 con 0x7fd0fc0ffbd0 2024-04-05T22:03:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.930+0000 7fd0f8ff9700 1 -- 172.21.15.121:0/1858303716 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0e4010460 con 0x7fd0fc0ffbd0 2024-04-05T22:03:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.930+0000 7fd101b2d700 1 -- 172.21.15.121:0/1858303716 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc19a1d0 con 0x7fd0fc0ffbd0 2024-04-05T22:03:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.930+0000 7fd0f8ff9700 1 -- 172.21.15.121:0/1858303716 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0e400f5f0 con 0x7fd0fc0ffbd0 2024-04-05T22:03:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.930+0000 7fd101b2d700 1 -- 172.21.15.121:0/1858303716 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0fc19a720 con 0x7fd0fc0ffbd0 2024-04-05T22:03:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.930+0000 7fd101b2d700 1 -- 172.21.15.121:0/1858303716 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0fc107fd0 con 0x7fd0fc0ffbd0 2024-04-05T22:03:23.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.931+0000 7fd0f8ff9700 1 -- 172.21.15.121:0/1858303716 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd0e40105d0 con 0x7fd0fc0ffbd0 2024-04-05T22:03:23.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.931+0000 7fd0f8ff9700 1 --2- 172.21.15.121:0/1858303716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd0e806cce0 0x7fd0e806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:23.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.931+0000 7fd0f8ff9700 1 -- 172.21.15.121:0/1858303716 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd0e408c4f0 con 0x7fd0fc0ffbd0 2024-04-05T22:03:23.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.931+0000 7fd0faffd700 1 --2- 172.21.15.121:0/1858303716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd0e806cce0 0x7fd0e806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:23.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.932+0000 7fd0faffd700 1 --2- 172.21.15.121:0/1858303716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd0e806cce0 0x7fd0e806f1a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00b5c0 tx=0x7fd0ec005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:23.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:23.936+0000 7fd0f8ff9700 1 -- 172.21.15.121:0/1858303716 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd0e40565e0 con 0x7fd0fc0ffbd0 2024-04-05T22:03:24.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.208+0000 7fd101b2d700 1 -- 172.21.15.121:0/1858303716 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd0fc19a990 con 0x7fd0fc0ffbd0 2024-04-05T22:03:24.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.208+0000 7fd0f8ff9700 1 -- 172.21.15.121:0/1858303716 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fd0e4010880 con 0x7fd0fc0ffbd0 2024-04-05T22:03:24.209 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:03:24.209 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T22:03:24.209 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:03:24.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:03:24.209 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T22:03:24.209 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:03:24.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:03:24.209 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T22:03:24.209 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T22:03:24.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:03:24.209 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T22:03:24.210 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T22:03:24.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:03:24.210 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T22:03:24.210 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T22:03:24.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T22:03:24.210 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:03:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.210+0000 7fd101b2d700 1 -- 172.21.15.121:0/1858303716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd0e806cce0 msgr2=0x7fd0e806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.210+0000 7fd101b2d700 1 --2- 172.21.15.121:0/1858303716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd0e806cce0 0x7fd0e806f1a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00b5c0 tx=0x7fd0ec005960 comp rx=0 tx=0).stop 2024-04-05T22:03:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.210+0000 7fd101b2d700 1 -- 172.21.15.121:0/1858303716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc0ffbd0 msgr2=0x7fd0fc195540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.210+0000 7fd101b2d700 1 --2- 172.21.15.121:0/1858303716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc0ffbd0 0x7fd0fc195540 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400d8d0 tx=0x7fd0e400dc90 comp rx=0 tx=0).stop 2024-04-05T22:03:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.210+0000 7fd101b2d700 1 -- 172.21.15.121:0/1858303716 shutdown_connections 2024-04-05T22:03:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.210+0000 7fd101b2d700 1 --2- 172.21.15.121:0/1858303716 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd0e806cce0 0x7fd0e806f1a0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.210+0000 7fd101b2d700 1 --2- 172.21.15.121:0/1858303716 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc0ffbd0 0x7fd0fc195540 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.210+0000 7fd101b2d700 1 --2- 172.21.15.121:0/1858303716 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd0fc105a40 0x7fd0fc195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.210+0000 7fd101b2d700 1 -- 172.21.15.121:0/1858303716 >> 172.21.15.121:0/1858303716 conn(0x7fd0fc0755a0 msgr2=0x7fd0fc0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.210+0000 7fd101b2d700 1 -- 172.21.15.121:0/1858303716 shutdown_connections 2024-04-05T22:03:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.210+0000 7fd101b2d700 1 -- 172.21.15.121:0/1858303716 wait complete. 2024-04-05T22:03:24.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 -- 172.21.15.121:0/4195731659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce10105a50 msgr2=0x7fce10105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:24.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 --2- 172.21.15.121:0/4195731659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce10105a50 0x7fce10105e30 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fce04009910 tx=0x7fce04009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:24.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 -- 172.21.15.121:0/4195731659 shutdown_connections 2024-04-05T22:03:24.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 --2- 172.21.15.121:0/4195731659 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce100ffa50 0x7fce100ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 --2- 172.21.15.121:0/4195731659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce10105a50 0x7fce10105e30 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 -- 172.21.15.121:0/4195731659 >> 172.21.15.121:0/4195731659 conn(0x7fce100fb540 msgr2=0x7fce100fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:24.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 -- 172.21.15.121:0/4195731659 shutdown_connections 2024-04-05T22:03:24.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 -- 172.21.15.121:0/4195731659 wait complete. 2024-04-05T22:03:24.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 Processor -- start 2024-04-05T22:03:24.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 -- start start 2024-04-05T22:03:24.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce100ffa50 0x7fce10197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:24.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce10105a50 0x7fce10197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:24.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce10197fc0 con 0x7fce100ffa50 2024-04-05T22:03:24.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.313+0000 7fce171ec700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce10198100 con 0x7fce10105a50 2024-04-05T22:03:24.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce14f88700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce100ffa50 0x7fce10197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:24.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce14f88700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce100ffa50 0x7fce10197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48762/0 (socket says 172.21.15.121:48762) 2024-04-05T22:03:24.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce14f88700 1 -- 172.21.15.121:0/3717160494 learned_addr learned my addr 172.21.15.121:0/3717160494 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce0ffff700 1 --2- 172.21.15.121:0/3717160494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce10105a50 0x7fce10197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce14f88700 1 -- 172.21.15.121:0/3717160494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce10105a50 msgr2=0x7fce10197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce14f88700 1 --2- 172.21.15.121:0/3717160494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce10105a50 0x7fce10197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce14f88700 1 -- 172.21.15.121:0/3717160494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce040095f0 con 0x7fce100ffa50 2024-04-05T22:03:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce0ffff700 1 --2- 172.21.15.121:0/3717160494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce10105a50 0x7fce10197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:03:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce14f88700 1 --2- 172.21.15.121:0/3717160494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce100ffa50 0x7fce10197430 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fce04006010 tx=0x7fce0400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce0dffb700 1 -- 172.21.15.121:0/3717160494 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce0401d070 con 0x7fce100ffa50 2024-04-05T22:03:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce0dffb700 1 -- 172.21.15.121:0/3717160494 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce0400f460 con 0x7fce100ffa50 2024-04-05T22:03:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce0dffb700 1 -- 172.21.15.121:0/3717160494 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce04021640 con 0x7fce100ffa50 2024-04-05T22:03:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce171ec700 1 -- 172.21.15.121:0/3717160494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce101a04a0 con 0x7fce100ffa50 2024-04-05T22:03:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.314+0000 7fce171ec700 1 -- 172.21.15.121:0/3717160494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce101a08b0 con 0x7fce100ffa50 2024-04-05T22:03:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.315+0000 7fce171ec700 1 -- 172.21.15.121:0/3717160494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce10108050 con 0x7fce100ffa50 2024-04-05T22:03:24.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.316+0000 7fce0dffb700 1 -- 172.21.15.121:0/3717160494 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fce0400fa20 con 0x7fce100ffa50 2024-04-05T22:03:24.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.316+0000 7fce0dffb700 1 --2- 172.21.15.121:0/3717160494 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce0006cce0 0x7fce0006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:24.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.316+0000 7fce0ffff700 1 --2- 172.21.15.121:0/3717160494 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce0006cce0 0x7fce0006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:24.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.316+0000 7fce0dffb700 1 -- 172.21.15.121:0/3717160494 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fce0408d7b0 con 0x7fce100ffa50 2024-04-05T22:03:24.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.317+0000 7fce0ffff700 1 --2- 172.21.15.121:0/3717160494 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce0006cce0 0x7fce0006f1a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc005fd0 tx=0x7fcdfc005e20 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:24.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.321+0000 7fce0dffb700 1 -- 172.21.15.121:0/3717160494 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fce040578a0 con 0x7fce100ffa50 2024-04-05T22:03:24.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.555+0000 7fce171ec700 1 -- 172.21.15.121:0/3717160494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fce1002cd30 con 0x7fce100ffa50 2024-04-05T22:03:24.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.555+0000 7fce0dffb700 1 -- 172.21.15.121:0/3717160494 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fce04057020 con 0x7fce100ffa50 2024-04-05T22:03:24.555 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T22:03:24.556 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T22:03:24.557 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T22:03:24.558 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T22:03:24.558 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T22:03:24.558 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T22:03:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:03:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:03:24.558 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T22:03:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:03:24.558 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:03:24.558 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:03:24.558 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:03:24.558 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:03:24.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.557+0000 7fce171ec700 1 -- 172.21.15.121:0/3717160494 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce0006cce0 msgr2=0x7fce0006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:24.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.557+0000 7fce171ec700 1 --2- 172.21.15.121:0/3717160494 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce0006cce0 0x7fce0006f1a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc005fd0 tx=0x7fcdfc005e20 comp rx=0 tx=0).stop 2024-04-05T22:03:24.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.557+0000 7fce171ec700 1 -- 172.21.15.121:0/3717160494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce100ffa50 msgr2=0x7fce10197430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:24.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.557+0000 7fce171ec700 1 --2- 172.21.15.121:0/3717160494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce100ffa50 0x7fce10197430 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fce04006010 tx=0x7fce0400be20 comp rx=0 tx=0).stop 2024-04-05T22:03:24.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.557+0000 7fce171ec700 1 -- 172.21.15.121:0/3717160494 shutdown_connections 2024-04-05T22:03:24.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.557+0000 7fce171ec700 1 --2- 172.21.15.121:0/3717160494 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fce0006cce0 0x7fce0006f1a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.557+0000 7fce171ec700 1 --2- 172.21.15.121:0/3717160494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce100ffa50 0x7fce10197430 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.557+0000 7fce171ec700 1 --2- 172.21.15.121:0/3717160494 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce10105a50 0x7fce10197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.557+0000 7fce171ec700 1 -- 172.21.15.121:0/3717160494 >> 172.21.15.121:0/3717160494 conn(0x7fce100fb540 msgr2=0x7fce100fbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:24.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.557+0000 7fce171ec700 1 -- 172.21.15.121:0/3717160494 shutdown_connections 2024-04-05T22:03:24.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.557+0000 7fce171ec700 1 -- 172.21.15.121:0/3717160494 wait complete. 2024-04-05T22:03:24.560 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T22:03:24.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.660+0000 7fe9c923c700 1 -- 172.21.15.121:0/3303845619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c40ff2f0 msgr2=0x7fe9c40ff6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:24.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.660+0000 7fe9c923c700 1 --2- 172.21.15.121:0/3303845619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c40ff2f0 0x7fe9c40ff6d0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009910 tx=0x7fe9b4009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:24.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.660+0000 7fe9c923c700 1 -- 172.21.15.121:0/3303845619 shutdown_connections 2024-04-05T22:03:24.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.660+0000 7fe9c923c700 1 --2- 172.21.15.121:0/3303845619 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9c40ffca0 0x7fe9c41057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.660+0000 7fe9c923c700 1 --2- 172.21.15.121:0/3303845619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c40ff2f0 0x7fe9c40ff6d0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.660+0000 7fe9c923c700 1 -- 172.21.15.121:0/3303845619 >> 172.21.15.121:0/3303845619 conn(0x7fe9c4075e30 msgr2=0x7fe9c40fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:24.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.660+0000 7fe9c923c700 1 -- 172.21.15.121:0/3303845619 shutdown_connections 2024-04-05T22:03:24.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.660+0000 7fe9c923c700 1 -- 172.21.15.121:0/3303845619 wait complete. 2024-04-05T22:03:24.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.661+0000 7fe9c923c700 1 Processor -- start 2024-04-05T22:03:24.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.661+0000 7fe9c923c700 1 -- start start 2024-04-05T22:03:24.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9c923c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9c40ff2f0 0x7fe9c4196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:24.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9c923c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c40ffca0 0x7fe9c4196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9c923c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c41971a0 con 0x7fe9c40ffca0 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9c923c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c419af30 con 0x7fe9c40ff2f0 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9c259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c40ffca0 0x7fe9c4196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9c259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c40ffca0 0x7fe9c4196ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48778/0 (socket says 172.21.15.121:48778) 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9c259c700 1 -- 172.21.15.121:0/3969987486 learned_addr learned my addr 172.21.15.121:0/3969987486 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9c2d9d700 1 --2- 172.21.15.121:0/3969987486 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9c40ff2f0 0x7fe9c4196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9c259c700 1 -- 172.21.15.121:0/3969987486 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9c40ff2f0 msgr2=0x7fe9c4196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9c259c700 1 --2- 172.21.15.121:0/3969987486 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9c40ff2f0 0x7fe9c4196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9c259c700 1 -- 172.21.15.121:0/3969987486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b40095f0 con 0x7fe9c40ffca0 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9c259c700 1 --2- 172.21.15.121:0/3969987486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c40ffca0 0x7fe9c4196ac0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800d8d0 tx=0x7fe9b800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9abfff700 1 -- 172.21.15.121:0/3969987486 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9b80098a0 con 0x7fe9c40ffca0 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9abfff700 1 -- 172.21.15.121:0/3969987486 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9b8010460 con 0x7fe9c40ffca0 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9c923c700 1 -- 172.21.15.121:0/3969987486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c419b210 con 0x7fe9c40ffca0 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.662+0000 7fe9abfff700 1 -- 172.21.15.121:0/3969987486 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9b8009ea0 con 0x7fe9c40ffca0 2024-04-05T22:03:24.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.663+0000 7fe9c923c700 1 -- 172.21.15.121:0/3969987486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c419b7e0 con 0x7fe9c40ffca0 2024-04-05T22:03:24.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.664+0000 7fe9c923c700 1 -- 172.21.15.121:0/3969987486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c4109110 con 0x7fe9c40ffca0 2024-04-05T22:03:24.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.664+0000 7fe9abfff700 1 -- 172.21.15.121:0/3969987486 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe9b80105d0 con 0x7fe9c40ffca0 2024-04-05T22:03:24.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.664+0000 7fe9abfff700 1 --2- 172.21.15.121:0/3969987486 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe9ac06cce0 0x7fe9ac06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:24.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.665+0000 7fe9abfff700 1 -- 172.21.15.121:0/3969987486 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe9b808c480 con 0x7fe9c40ffca0 2024-04-05T22:03:24.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.665+0000 7fe9c2d9d700 1 --2- 172.21.15.121:0/3969987486 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe9ac06cce0 0x7fe9ac06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:24.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.665+0000 7fe9c2d9d700 1 --2- 172.21.15.121:0/3969987486 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe9ac06cce0 0x7fe9ac06f1a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4006010 tx=0x7fe9b401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:24.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.669+0000 7fe9abfff700 1 -- 172.21.15.121:0/3969987486 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe9b80564f0 con 0x7fe9c40ffca0 2024-04-05T22:03:24.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.878+0000 7fe9c923c700 1 -- 172.21.15.121:0/3969987486 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9c419bac0 con 0x7fe9ac06cce0 2024-04-05T22:03:24.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.879+0000 7fe9abfff700 1 -- 172.21.15.121:0/3969987486 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe9c419bac0 con 0x7fe9ac06cce0 2024-04-05T22:03:24.879 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T22:03:24.880 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:03:24.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.881+0000 7fe9c923c700 1 -- 172.21.15.121:0/3969987486 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe9ac06cce0 msgr2=0x7fe9ac06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:24.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.881+0000 7fe9c923c700 1 --2- 172.21.15.121:0/3969987486 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe9ac06cce0 0x7fe9ac06f1a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4006010 tx=0x7fe9b401a040 comp rx=0 tx=0).stop 2024-04-05T22:03:24.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.881+0000 7fe9c923c700 1 -- 172.21.15.121:0/3969987486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c40ffca0 msgr2=0x7fe9c4196ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:24.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.881+0000 7fe9c923c700 1 --2- 172.21.15.121:0/3969987486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c40ffca0 0x7fe9c4196ac0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800d8d0 tx=0x7fe9b800dbe0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.881+0000 7fe9c923c700 1 -- 172.21.15.121:0/3969987486 shutdown_connections 2024-04-05T22:03:24.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.881+0000 7fe9c923c700 1 --2- 172.21.15.121:0/3969987486 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe9ac06cce0 0x7fe9ac06f1a0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.881+0000 7fe9c923c700 1 --2- 172.21.15.121:0/3969987486 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe9c40ff2f0 0x7fe9c4196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.881+0000 7fe9c923c700 1 --2- 172.21.15.121:0/3969987486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c40ffca0 0x7fe9c4196ac0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.881+0000 7fe9c923c700 1 -- 172.21.15.121:0/3969987486 >> 172.21.15.121:0/3969987486 conn(0x7fe9c4075e30 msgr2=0x7fe9c40fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:24.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.882+0000 7fe9c923c700 1 -- 172.21.15.121:0/3969987486 shutdown_connections 2024-04-05T22:03:24.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.882+0000 7fe9c923c700 1 -- 172.21.15.121:0/3969987486 wait complete. 2024-04-05T22:03:24.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.985+0000 7fb16a331700 1 -- 172.21.15.121:0/2913062359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1640ffa50 msgr2=0x7fb1640ffed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:24.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.985+0000 7fb16a331700 1 --2- 172.21.15.121:0/2913062359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1640ffa50 0x7fb1640ffed0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009910 tx=0x7fb14c009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:24.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.985+0000 7fb16a331700 1 -- 172.21.15.121:0/2913062359 shutdown_connections 2024-04-05T22:03:24.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.985+0000 7fb16a331700 1 --2- 172.21.15.121:0/2913062359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1640ffa50 0x7fb1640ffed0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.985+0000 7fb16a331700 1 --2- 172.21.15.121:0/2913062359 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb164105a50 0x7fb164105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.985+0000 7fb16a331700 1 -- 172.21.15.121:0/2913062359 >> 172.21.15.121:0/2913062359 conn(0x7fb1640fb540 msgr2=0x7fb1640fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:24.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.985+0000 7fb16a331700 1 -- 172.21.15.121:0/2913062359 shutdown_connections 2024-04-05T22:03:24.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.985+0000 7fb16a331700 1 -- 172.21.15.121:0/2913062359 wait complete. 2024-04-05T22:03:24.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.986+0000 7fb16a331700 1 Processor -- start 2024-04-05T22:03:24.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.986+0000 7fb16a331700 1 -- start start 2024-04-05T22:03:24.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.986+0000 7fb16a331700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1640ffa50 0x7fb164197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.986+0000 7fb16a331700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb164105a50 0x7fb164197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.986+0000 7fb16a331700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb164198000 con 0x7fb1640ffa50 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.986+0000 7fb16a331700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb164198170 con 0x7fb164105a50 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.986+0000 7fb163fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1640ffa50 0x7fb164197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.986+0000 7fb163fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1640ffa50 0x7fb164197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48788/0 (socket says 172.21.15.121:48788) 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.986+0000 7fb163fff700 1 -- 172.21.15.121:0/74646556 learned_addr learned my addr 172.21.15.121:0/74646556 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.986+0000 7fb1637fe700 1 --2- 172.21.15.121:0/74646556 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb164105a50 0x7fb164197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.986+0000 7fb163fff700 1 -- 172.21.15.121:0/74646556 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb164105a50 msgr2=0x7fb164197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.986+0000 7fb163fff700 1 --2- 172.21.15.121:0/74646556 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb164105a50 0x7fb164197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.986+0000 7fb163fff700 1 -- 172.21.15.121:0/74646556 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14c0095f0 con 0x7fb1640ffa50 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.987+0000 7fb163fff700 1 --2- 172.21.15.121:0/74646556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1640ffa50 0x7fb164197530 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fb15400c930 tx=0x7fb15400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.987+0000 7fb1617fa700 1 -- 172.21.15.121:0/74646556 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb154007ad0 con 0x7fb1640ffa50 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.987+0000 7fb16a331700 1 -- 172.21.15.121:0/74646556 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1641a05a0 con 0x7fb1640ffa50 2024-04-05T22:03:24.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.987+0000 7fb1617fa700 1 -- 172.21.15.121:0/74646556 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb154007c30 con 0x7fb1640ffa50 2024-04-05T22:03:24.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.987+0000 7fb1617fa700 1 -- 172.21.15.121:0/74646556 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb154018630 con 0x7fb1640ffa50 2024-04-05T22:03:24.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.987+0000 7fb16a331700 1 -- 172.21.15.121:0/74646556 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1641a0a10 con 0x7fb1640ffa50 2024-04-05T22:03:24.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.988+0000 7fb16a331700 1 -- 172.21.15.121:0/74646556 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb164107fe0 con 0x7fb1640ffa50 2024-04-05T22:03:24.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.988+0000 7fb1617fa700 1 -- 172.21.15.121:0/74646556 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb15400f450 con 0x7fb1640ffa50 2024-04-05T22:03:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.989+0000 7fb1617fa700 1 --2- 172.21.15.121:0/74646556 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb15006cce0 0x7fb15006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.989+0000 7fb1617fa700 1 -- 172.21.15.121:0/74646556 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb15408c7b0 con 0x7fb1640ffa50 2024-04-05T22:03:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.989+0000 7fb1637fe700 1 --2- 172.21.15.121:0/74646556 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb15006cce0 0x7fb15006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.989+0000 7fb1637fe700 1 --2- 172.21.15.121:0/74646556 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb15006cce0 0x7fb15006f1a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb14c00b5c0 tx=0x7fb14c005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:24.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:24.993+0000 7fb1617fa700 1 -- 172.21.15.121:0/74646556 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb1540568d0 con 0x7fb1640ffa50 2024-04-05T22:03:25.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:24 smithi121 ceph-mon[145859]: from='client.35830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:25.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:24 smithi121 ceph-mon[145859]: from='client.35834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:25.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:24 smithi121 ceph-mon[145859]: from='client.35838 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:25.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:24 smithi121 ceph-mon[145859]: pgmap v1116: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:25.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:24 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1858303716' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:03:25.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:24 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3717160494' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:03:25.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:25.262+0000 7fb16a331700 1 -- 172.21.15.121:0/74646556 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fb16404ea90 con 0x7fb1640ffa50 2024-04-05T22:03:25.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:25.263+0000 7fb1617fa700 1 -- 172.21.15.121:0/74646556 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fb154059ef0 con 0x7fb1640ffa50 2024-04-05T22:03:25.263 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:03:25.263 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:03:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:03:25.263 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T22:03:25.263 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:03:25.263 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T22:03:25.263 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T22:03:25.264 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:03:25.265 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T22:03:25.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:25.265+0000 7fb16a331700 1 -- 172.21.15.121:0/74646556 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb15006cce0 msgr2=0x7fb15006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:25.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:25.265+0000 7fb16a331700 1 --2- 172.21.15.121:0/74646556 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb15006cce0 0x7fb15006f1a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb14c00b5c0 tx=0x7fb14c005fd0 comp rx=0 tx=0).stop 2024-04-05T22:03:25.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:25.265+0000 7fb16a331700 1 -- 172.21.15.121:0/74646556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1640ffa50 msgr2=0x7fb164197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:25.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:25.265+0000 7fb16a331700 1 --2- 172.21.15.121:0/74646556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1640ffa50 0x7fb164197530 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fb15400c930 tx=0x7fb15400ccf0 comp rx=0 tx=0).stop 2024-04-05T22:03:25.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:25.265+0000 7fb16a331700 1 -- 172.21.15.121:0/74646556 shutdown_connections 2024-04-05T22:03:25.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:25.265+0000 7fb16a331700 1 --2- 172.21.15.121:0/74646556 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb15006cce0 0x7fb15006f1a0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:25.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:25.265+0000 7fb16a331700 1 --2- 172.21.15.121:0/74646556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1640ffa50 0x7fb164197530 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:25.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:25.265+0000 7fb16a331700 1 --2- 172.21.15.121:0/74646556 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb164105a50 0x7fb164197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:25.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:25.265+0000 7fb16a331700 1 -- 172.21.15.121:0/74646556 >> 172.21.15.121:0/74646556 conn(0x7fb1640fb540 msgr2=0x7fb1640fc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:25.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:25.265+0000 7fb16a331700 1 -- 172.21.15.121:0/74646556 shutdown_connections 2024-04-05T22:03:25.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:25.265+0000 7fb16a331700 1 -- 172.21.15.121:0/74646556 wait complete. 2024-04-05T22:03:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:24 smithi188 ceph-mon[107793]: from='client.35830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:24 smithi188 ceph-mon[107793]: from='client.35834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:24 smithi188 ceph-mon[107793]: from='client.35838 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:24 smithi188 ceph-mon[107793]: pgmap v1116: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:24 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1858303716' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:03:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:24 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3717160494' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:03:26.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:25 smithi121 ceph-mon[145859]: from='client.35850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:26.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:25 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/74646556' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:03:26.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:25 smithi188 ceph-mon[107793]: from='client.35850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:26.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:25 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/74646556' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:03:27.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:26 smithi121 ceph-mon[145859]: pgmap v1117: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:27.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:26 smithi188 ceph-mon[107793]: pgmap v1117: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:29.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:28 smithi121 ceph-mon[145859]: pgmap v1118: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:28 smithi188 ceph-mon[107793]: pgmap v1118: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:30 smithi121 ceph-mon[145859]: pgmap v1119: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:31.242 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:03:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:30 smithi188 ceph-mon[107793]: pgmap v1119: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:03:33.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:32 smithi121 ceph-mon[145859]: pgmap v1120: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:33.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:32 smithi188 ceph-mon[107793]: pgmap v1120: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:35.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:34 smithi121 ceph-mon[145859]: pgmap v1121: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:35.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:34 smithi188 ceph-mon[107793]: pgmap v1121: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:37.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:36 smithi121 ceph-mon[145859]: pgmap v1122: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:36 smithi188 ceph-mon[107793]: pgmap v1122: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:38.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:38 smithi188 ceph-mon[107793]: pgmap v1123: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:38.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:38 smithi121 ceph-mon[145859]: pgmap v1123: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:41.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:40 smithi121 ceph-mon[145859]: pgmap v1124: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:40 smithi188 ceph-mon[107793]: pgmap v1124: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:43.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:42 smithi121 ceph-mon[145859]: pgmap v1125: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:43.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:42 smithi188 ceph-mon[107793]: pgmap v1125: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:45.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:44 smithi121 ceph-mon[145859]: pgmap v1126: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:44 smithi188 ceph-mon[107793]: pgmap v1126: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:03:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:03:47.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:46 smithi121 ceph-mon[145859]: pgmap v1127: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:47.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:46 smithi188 ceph-mon[107793]: pgmap v1127: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:49.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:48 smithi121 ceph-mon[145859]: pgmap v1128: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:48 smithi188 ceph-mon[107793]: pgmap v1128: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:51.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:50 smithi121 ceph-mon[145859]: pgmap v1129: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:50 smithi188 ceph-mon[107793]: pgmap v1129: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:53.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:52 smithi121 ceph-mon[145859]: pgmap v1130: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:52 smithi188 ceph-mon[107793]: pgmap v1130: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:55.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:54 smithi121 ceph-mon[145859]: pgmap v1131: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:55.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:54 smithi188 ceph-mon[107793]: pgmap v1131: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:55.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.382+0000 7f4453ca3700 1 -- 172.21.15.121:0/1467248710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f444c105a40 msgr2=0x7f444c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:55.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.382+0000 7f4453ca3700 1 --2- 172.21.15.121:0/1467248710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f444c105a40 0x7f444c105e20 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f443c009910 tx=0x7f443c009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:55.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.382+0000 7f4453ca3700 1 -- 172.21.15.121:0/1467248710 shutdown_connections 2024-04-05T22:03:55.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.382+0000 7f4453ca3700 1 --2- 172.21.15.121:0/1467248710 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f444c0ffbd0 0x7f444c100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:55.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.382+0000 7f4453ca3700 1 --2- 172.21.15.121:0/1467248710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f444c105a40 0x7f444c105e20 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:55.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.382+0000 7f4453ca3700 1 -- 172.21.15.121:0/1467248710 >> 172.21.15.121:0/1467248710 conn(0x7f444c0755a0 msgr2=0x7f444c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:55.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.382+0000 7f4453ca3700 1 -- 172.21.15.121:0/1467248710 shutdown_connections 2024-04-05T22:03:55.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.382+0000 7f4453ca3700 1 -- 172.21.15.121:0/1467248710 wait complete. 2024-04-05T22:03:55.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f4453ca3700 1 Processor -- start 2024-04-05T22:03:55.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f4453ca3700 1 -- start start 2024-04-05T22:03:55.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f4453ca3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f444c0ffbd0 0x7f444c101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:55.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f4453ca3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f444c105a40 0x7f444c101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f4453ca3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c102090 con 0x7f444c0ffbd0 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f4453ca3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c1021d0 con 0x7f444c105a40 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f4451a3f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f444c0ffbd0 0x7f444c101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f4451a3f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f444c0ffbd0 0x7f444c101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60488/0 (socket says 172.21.15.121:60488) 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f4451a3f700 1 -- 172.21.15.121:0/1082743120 learned_addr learned my addr 172.21.15.121:0/1082743120 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f445123e700 1 --2- 172.21.15.121:0/1082743120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f444c105a40 0x7f444c101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f4451a3f700 1 -- 172.21.15.121:0/1082743120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f444c105a40 msgr2=0x7f444c101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f4451a3f700 1 --2- 172.21.15.121:0/1082743120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f444c105a40 0x7f444c101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f4451a3f700 1 -- 172.21.15.121:0/1082743120 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c0095f0 con 0x7f444c0ffbd0 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.383+0000 7f4451a3f700 1 --2- 172.21.15.121:0/1082743120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f444c0ffbd0 0x7f444c101610 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f443c006010 tx=0x7f443c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.384+0000 7f4442ffd700 1 -- 172.21.15.121:0/1082743120 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f443c01d070 con 0x7f444c0ffbd0 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.384+0000 7f4442ffd700 1 -- 172.21.15.121:0/1082743120 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f443c00f460 con 0x7f444c0ffbd0 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.384+0000 7f4453ca3700 1 -- 172.21.15.121:0/1082743120 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f444c102450 con 0x7f444c0ffbd0 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.384+0000 7f4442ffd700 1 -- 172.21.15.121:0/1082743120 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f443c021640 con 0x7f444c0ffbd0 2024-04-05T22:03:55.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.384+0000 7f4453ca3700 1 -- 172.21.15.121:0/1082743120 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f444c1a2c30 con 0x7f444c0ffbd0 2024-04-05T22:03:55.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.385+0000 7f4453ca3700 1 -- 172.21.15.121:0/1082743120 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f444c109050 con 0x7f444c0ffbd0 2024-04-05T22:03:55.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.385+0000 7f4442ffd700 1 -- 172.21.15.121:0/1082743120 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f443c00fa20 con 0x7f444c0ffbd0 2024-04-05T22:03:55.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.386+0000 7f4442ffd700 1 --2- 172.21.15.121:0/1082743120 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f443806cce0 0x7f443806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:55.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.386+0000 7f4442ffd700 1 -- 172.21.15.121:0/1082743120 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f443c08d7b0 con 0x7f444c0ffbd0 2024-04-05T22:03:55.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.386+0000 7f445123e700 1 --2- 172.21.15.121:0/1082743120 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f443806cce0 0x7f443806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:55.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.386+0000 7f445123e700 1 --2- 172.21.15.121:0/1082743120 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f443806cce0 0x7f443806f1a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4448006fd0 tx=0x7f4448008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:55.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.390+0000 7f4442ffd700 1 -- 172.21.15.121:0/1082743120 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f443c0578a0 con 0x7f444c0ffbd0 2024-04-05T22:03:55.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.602+0000 7f4453ca3700 1 -- 172.21.15.121:0/1082743120 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f444c0611d0 con 0x7f443806cce0 2024-04-05T22:03:55.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.603+0000 7f4442ffd700 1 -- 172.21.15.121:0/1082743120 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f444c0611d0 con 0x7f443806cce0 2024-04-05T22:03:55.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.605+0000 7f4453ca3700 1 -- 172.21.15.121:0/1082743120 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f443806cce0 msgr2=0x7f443806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:55.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.605+0000 7f4453ca3700 1 --2- 172.21.15.121:0/1082743120 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f443806cce0 0x7f443806f1a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4448006fd0 tx=0x7f4448008040 comp rx=0 tx=0).stop 2024-04-05T22:03:55.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.605+0000 7f4453ca3700 1 -- 172.21.15.121:0/1082743120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f444c0ffbd0 msgr2=0x7f444c101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:55.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.605+0000 7f4453ca3700 1 --2- 172.21.15.121:0/1082743120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f444c0ffbd0 0x7f444c101610 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f443c006010 tx=0x7f443c00be20 comp rx=0 tx=0).stop 2024-04-05T22:03:55.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.605+0000 7f4453ca3700 1 -- 172.21.15.121:0/1082743120 shutdown_connections 2024-04-05T22:03:55.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.605+0000 7f4453ca3700 1 --2- 172.21.15.121:0/1082743120 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f443806cce0 0x7f443806f1a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:55.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.605+0000 7f4453ca3700 1 --2- 172.21.15.121:0/1082743120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f444c0ffbd0 0x7f444c101610 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:55.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.605+0000 7f4453ca3700 1 --2- 172.21.15.121:0/1082743120 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f444c105a40 0x7f444c101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:55.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.605+0000 7f4453ca3700 1 -- 172.21.15.121:0/1082743120 >> 172.21.15.121:0/1082743120 conn(0x7f444c0755a0 msgr2=0x7f444c103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:55.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.605+0000 7f4453ca3700 1 -- 172.21.15.121:0/1082743120 shutdown_connections 2024-04-05T22:03:55.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.605+0000 7f4453ca3700 1 -- 172.21.15.121:0/1082743120 wait complete. 2024-04-05T22:03:55.616 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T22:03:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.709+0000 7f83a90ba700 1 -- 172.21.15.121:0/2383092142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a40ffa40 msgr2=0x7f83a40ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.709+0000 7f83a90ba700 1 --2- 172.21.15.121:0/2383092142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a40ffa40 0x7f83a40ffec0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f838c009910 tx=0x7f838c009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.709+0000 7f83a90ba700 1 -- 172.21.15.121:0/2383092142 shutdown_connections 2024-04-05T22:03:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.709+0000 7f83a90ba700 1 --2- 172.21.15.121:0/2383092142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a40ffa40 0x7f83a40ffec0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.709+0000 7f83a90ba700 1 --2- 172.21.15.121:0/2383092142 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83a4105a40 0x7f83a4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.709+0000 7f83a90ba700 1 -- 172.21.15.121:0/2383092142 >> 172.21.15.121:0/2383092142 conn(0x7f83a40fb570 msgr2=0x7f83a40fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.709+0000 7f83a90ba700 1 -- 172.21.15.121:0/2383092142 shutdown_connections 2024-04-05T22:03:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.709+0000 7f83a90ba700 1 -- 172.21.15.121:0/2383092142 wait complete. 2024-04-05T22:03:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.709+0000 7f83a90ba700 1 Processor -- start 2024-04-05T22:03:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.710+0000 7f83a90ba700 1 -- start start 2024-04-05T22:03:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.710+0000 7f83a90ba700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83a40ffa40 0x7f83a4101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.710+0000 7f83a90ba700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a4105a40 0x7f83a4101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.710+0000 7f83a90ba700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a4102090 con 0x7f83a4105a40 2024-04-05T22:03:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.710+0000 7f83a90ba700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a41021d0 con 0x7f83a40ffa40 2024-04-05T22:03:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.710+0000 7f83a259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a4105a40 0x7f83a4101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.710+0000 7f83a259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a4105a40 0x7f83a4101b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60502/0 (socket says 172.21.15.121:60502) 2024-04-05T22:03:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.710+0000 7f83a259c700 1 -- 172.21.15.121:0/3241725165 learned_addr learned my addr 172.21.15.121:0/3241725165 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.710+0000 7f83a2d9d700 1 --2- 172.21.15.121:0/3241725165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83a40ffa40 0x7f83a4101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.710+0000 7f83a259c700 1 -- 172.21.15.121:0/3241725165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83a40ffa40 msgr2=0x7f83a4101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.710+0000 7f83a259c700 1 --2- 172.21.15.121:0/3241725165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83a40ffa40 0x7f83a4101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.710+0000 7f83a259c700 1 -- 172.21.15.121:0/3241725165 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c0095f0 con 0x7f83a4105a40 2024-04-05T22:03:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.711+0000 7f83a259c700 1 --2- 172.21.15.121:0/3241725165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a4105a40 0x7f83a4101b50 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f838c006010 tx=0x7f838c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.711+0000 7f839bfff700 1 -- 172.21.15.121:0/3241725165 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f838c01d070 con 0x7f83a4105a40 2024-04-05T22:03:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.711+0000 7f83a90ba700 1 -- 172.21.15.121:0/3241725165 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a4102450 con 0x7f83a4105a40 2024-04-05T22:03:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.711+0000 7f839bfff700 1 -- 172.21.15.121:0/3241725165 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f838c005940 con 0x7f83a4105a40 2024-04-05T22:03:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.711+0000 7f839bfff700 1 -- 172.21.15.121:0/3241725165 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f838c00fd90 con 0x7f83a4105a40 2024-04-05T22:03:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.711+0000 7f83a90ba700 1 -- 172.21.15.121:0/3241725165 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a41a2b10 con 0x7f83a4105a40 2024-04-05T22:03:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.712+0000 7f83a90ba700 1 -- 172.21.15.121:0/3241725165 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a404ea90 con 0x7f83a4105a40 2024-04-05T22:03:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.712+0000 7f839bfff700 1 -- 172.21.15.121:0/3241725165 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f838c003a60 con 0x7f83a4105a40 2024-04-05T22:03:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.713+0000 7f839bfff700 1 --2- 172.21.15.121:0/3241725165 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f839006cce0 0x7f839006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.713+0000 7f83a2d9d700 1 --2- 172.21.15.121:0/3241725165 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f839006cce0 0x7f839006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:55.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.713+0000 7f839bfff700 1 -- 172.21.15.121:0/3241725165 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f838c08d740 con 0x7f83a4105a40 2024-04-05T22:03:55.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.713+0000 7f83a2d9d700 1 --2- 172.21.15.121:0/3241725165 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f839006cce0 0x7f839006f1a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f839c009550 tx=0x7f839c0091f0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:55.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.717+0000 7f839bfff700 1 -- 172.21.15.121:0/3241725165 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f838c0577b0 con 0x7f83a4105a40 2024-04-05T22:03:55.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.931+0000 7f83a90ba700 1 -- 172.21.15.121:0/3241725165 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f83a4068ca0 con 0x7f839006cce0 2024-04-05T22:03:55.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.932+0000 7f839bfff700 1 -- 172.21.15.121:0/3241725165 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f83a4068ca0 con 0x7f839006cce0 2024-04-05T22:03:55.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.934+0000 7f83a90ba700 1 -- 172.21.15.121:0/3241725165 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f839006cce0 msgr2=0x7f839006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:55.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.934+0000 7f83a90ba700 1 --2- 172.21.15.121:0/3241725165 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f839006cce0 0x7f839006f1a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f839c009550 tx=0x7f839c0091f0 comp rx=0 tx=0).stop 2024-04-05T22:03:55.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.934+0000 7f83a90ba700 1 -- 172.21.15.121:0/3241725165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a4105a40 msgr2=0x7f83a4101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:55.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.934+0000 7f83a90ba700 1 --2- 172.21.15.121:0/3241725165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a4105a40 0x7f83a4101b50 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f838c006010 tx=0x7f838c00bd70 comp rx=0 tx=0).stop 2024-04-05T22:03:55.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.934+0000 7f83a90ba700 1 -- 172.21.15.121:0/3241725165 shutdown_connections 2024-04-05T22:03:55.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.934+0000 7f83a90ba700 1 --2- 172.21.15.121:0/3241725165 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f839006cce0 0x7f839006f1a0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:55.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.934+0000 7f83a90ba700 1 --2- 172.21.15.121:0/3241725165 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f83a40ffa40 0x7f83a4101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:55.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.934+0000 7f83a90ba700 1 --2- 172.21.15.121:0/3241725165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a4105a40 0x7f83a4101b50 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:55.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.934+0000 7f83a90ba700 1 -- 172.21.15.121:0/3241725165 >> 172.21.15.121:0/3241725165 conn(0x7f83a40fb570 msgr2=0x7f83a41037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:55.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.934+0000 7f83a90ba700 1 -- 172.21.15.121:0/3241725165 shutdown_connections 2024-04-05T22:03:55.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:55.934+0000 7f83a90ba700 1 -- 172.21.15.121:0/3241725165 wait complete. 2024-04-05T22:03:56.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.037+0000 7f2f91b24700 1 -- 172.21.15.121:0/406793446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c069020 msgr2=0x7f2f8c105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:56.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.037+0000 7f2f91b24700 1 --2- 172.21.15.121:0/406793446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c069020 0x7f2f8c105790 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2f74009910 tx=0x7f2f74009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:56.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.037+0000 7f2f91b24700 1 -- 172.21.15.121:0/406793446 shutdown_connections 2024-04-05T22:03:56.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.037+0000 7f2f91b24700 1 --2- 172.21.15.121:0/406793446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c069020 0x7f2f8c105790 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.037+0000 7f2f91b24700 1 --2- 172.21.15.121:0/406793446 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f8c068670 0x7f2f8c068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.037+0000 7f2f91b24700 1 -- 172.21.15.121:0/406793446 >> 172.21.15.121:0/406793446 conn(0x7f2f8c0755a0 msgr2=0x7f2f8c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.037+0000 7f2f91b24700 1 -- 172.21.15.121:0/406793446 shutdown_connections 2024-04-05T22:03:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.037+0000 7f2f91b24700 1 -- 172.21.15.121:0/406793446 wait complete. 2024-04-05T22:03:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f91b24700 1 Processor -- start 2024-04-05T22:03:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f91b24700 1 -- start start 2024-04-05T22:03:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f91b24700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f8c068670 0x7f2f8c0ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f91b24700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c069020 0x7f2f8c1001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f91b24700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c100720 con 0x7f2f8c069020 2024-04-05T22:03:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f91b24700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c100860 con 0x7f2f8c068670 2024-04-05T22:03:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f8affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c069020 0x7f2f8c1001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f8affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c069020 0x7f2f8c1001e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60524/0 (socket says 172.21.15.121:60524) 2024-04-05T22:03:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f8affd700 1 -- 172.21.15.121:0/1116690010 learned_addr learned my addr 172.21.15.121:0/1116690010 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f8b7fe700 1 --2- 172.21.15.121:0/1116690010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f8c068670 0x7f2f8c0ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f8affd700 1 -- 172.21.15.121:0/1116690010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f8c068670 msgr2=0x7f2f8c0ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f8affd700 1 --2- 172.21.15.121:0/1116690010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f8c068670 0x7f2f8c0ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f8affd700 1 -- 172.21.15.121:0/1116690010 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f740095f0 con 0x7f2f8c069020 2024-04-05T22:03:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.038+0000 7f2f8b7fe700 1 --2- 172.21.15.121:0/1116690010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f8c068670 0x7f2f8c0ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:03:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.039+0000 7f2f8affd700 1 --2- 172.21.15.121:0/1116690010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c069020 0x7f2f8c1001e0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f2f740092e0 tx=0x7f2f7400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.039+0000 7f2f88ff9700 1 -- 172.21.15.121:0/1116690010 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f7401d070 con 0x7f2f8c069020 2024-04-05T22:03:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.039+0000 7f2f88ff9700 1 -- 172.21.15.121:0/1116690010 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f7400f460 con 0x7f2f8c069020 2024-04-05T22:03:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.039+0000 7f2f91b24700 1 -- 172.21.15.121:0/1116690010 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8c100ae0 con 0x7f2f8c069020 2024-04-05T22:03:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.039+0000 7f2f88ff9700 1 -- 172.21.15.121:0/1116690010 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f74021600 con 0x7f2f8c069020 2024-04-05T22:03:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.039+0000 7f2f91b24700 1 -- 172.21.15.121:0/1116690010 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8c1a2b40 con 0x7f2f8c069020 2024-04-05T22:03:56.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.040+0000 7f2f91b24700 1 -- 172.21.15.121:0/1116690010 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8c1090a0 con 0x7f2f8c069020 2024-04-05T22:03:56.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.040+0000 7f2f88ff9700 1 -- 172.21.15.121:0/1116690010 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2f7400f5d0 con 0x7f2f8c069020 2024-04-05T22:03:56.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.041+0000 7f2f88ff9700 1 --2- 172.21.15.121:0/1116690010 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2f7806cce0 0x7f2f7806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:56.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.041+0000 7f2f88ff9700 1 -- 172.21.15.121:0/1116690010 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2f7408d900 con 0x7f2f8c069020 2024-04-05T22:03:56.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.041+0000 7f2f8b7fe700 1 --2- 172.21.15.121:0/1116690010 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2f7806cce0 0x7f2f7806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:56.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.041+0000 7f2f8b7fe700 1 --2- 172.21.15.121:0/1116690010 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2f7806cce0 0x7f2f7806f1a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c005950 tx=0x7f2f7c005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:56.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.045+0000 7f2f88ff9700 1 -- 172.21.15.121:0/1116690010 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2f740578d0 con 0x7f2f8c069020 2024-04-05T22:03:56.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.243+0000 7f2f91b24700 1 -- 172.21.15.121:0/1116690010 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f8c02cd50 con 0x7f2f7806cce0 2024-04-05T22:03:56.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.254+0000 7f2f88ff9700 1 -- 172.21.15.121:0/1116690010 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f2f8c02cd50 con 0x7f2f7806cce0 2024-04-05T22:03:56.254 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T22:03:56.254 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (34m) 13m ago 46m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T22:03:56.254 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (46m) 13m ago 46m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T22:03:56.254 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (45m) 13m ago 45m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T22:03:56.254 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (32m) 13m ago 46m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T22:03:56.254 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (32m) 13m ago 45m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T22:03:56.255 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (34m) 13m ago 45m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T22:03:56.255 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (29m) 13m ago 40m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T22:03:56.255 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (28m) 13m ago 39m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T22:03:56.255 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (28m) 13m ago 40m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T22:03:56.255 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (13m) 13m ago 39m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T22:03:56.255 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (36m) 13m ago 47m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T22:03:56.255 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (35m) 13m ago 45m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T22:03:56.255 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (33m) 13m ago 47m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T22:03:56.255 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (33m) 13m ago 45m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T22:03:56.255 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (35m) 13m ago 46m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T22:03:56.256 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (35m) 13m ago 45m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T22:03:56.256 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (32m) 13m ago 44m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T22:03:56.256 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (31m) 13m ago 43m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T22:03:56.256 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (31m) 13m ago 43m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T22:03:56.256 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (30m) 13m ago 42m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T22:03:56.256 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (30m) 13m ago 42m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T22:03:56.256 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (29m) 13m ago 41m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T22:03:56.256 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (35m) 13m ago 45m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T22:03:56.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.256+0000 7f2f91b24700 1 -- 172.21.15.121:0/1116690010 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2f7806cce0 msgr2=0x7f2f7806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:56.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.256+0000 7f2f91b24700 1 --2- 172.21.15.121:0/1116690010 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2f7806cce0 0x7f2f7806f1a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c005950 tx=0x7f2f7c005f50 comp rx=0 tx=0).stop 2024-04-05T22:03:56.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.256+0000 7f2f91b24700 1 -- 172.21.15.121:0/1116690010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c069020 msgr2=0x7f2f8c1001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:56.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.256+0000 7f2f91b24700 1 --2- 172.21.15.121:0/1116690010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c069020 0x7f2f8c1001e0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f2f740092e0 tx=0x7f2f7400be20 comp rx=0 tx=0).stop 2024-04-05T22:03:56.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.256+0000 7f2f91b24700 1 -- 172.21.15.121:0/1116690010 shutdown_connections 2024-04-05T22:03:56.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.256+0000 7f2f91b24700 1 --2- 172.21.15.121:0/1116690010 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2f7806cce0 0x7f2f7806f1a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.256+0000 7f2f91b24700 1 --2- 172.21.15.121:0/1116690010 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2f8c068670 0x7f2f8c0ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.256+0000 7f2f91b24700 1 --2- 172.21.15.121:0/1116690010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c069020 0x7f2f8c1001e0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.256+0000 7f2f91b24700 1 -- 172.21.15.121:0/1116690010 >> 172.21.15.121:0/1116690010 conn(0x7f2f8c0755a0 msgr2=0x7f2f8c105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:56.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.256+0000 7f2f91b24700 1 -- 172.21.15.121:0/1116690010 shutdown_connections 2024-04-05T22:03:56.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.256+0000 7f2f91b24700 1 -- 172.21.15.121:0/1116690010 wait complete. 2024-04-05T22:03:56.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.356+0000 7fc46762b700 1 -- 172.21.15.121:0/1819690561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc460100de0 msgr2=0x7fc4601011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:56.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.356+0000 7fc46762b700 1 --2- 172.21.15.121:0/1819690561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc460100de0 0x7fc4601011c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fc45c009910 tx=0x7fc45c009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:56.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.357+0000 7fc46762b700 1 -- 172.21.15.121:0/1819690561 shutdown_connections 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.357+0000 7fc46762b700 1 --2- 172.21.15.121:0/1819690561 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc460101790 0x7fc4601057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.357+0000 7fc46762b700 1 --2- 172.21.15.121:0/1819690561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc460100de0 0x7fc4601011c0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.357+0000 7fc46762b700 1 -- 172.21.15.121:0/1819690561 >> 172.21.15.121:0/1819690561 conn(0x7fc4600fc670 msgr2=0x7fc4600fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.357+0000 7fc46762b700 1 -- 172.21.15.121:0/1819690561 shutdown_connections 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.357+0000 7fc46762b700 1 -- 172.21.15.121:0/1819690561 wait complete. 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.357+0000 7fc46762b700 1 Processor -- start 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.357+0000 7fc46762b700 1 -- start start 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.357+0000 7fc46762b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc460100de0 0x7fc46019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.357+0000 7fc46762b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc460101790 0x7fc46019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.357+0000 7fc46762b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46019f1f0 con 0x7fc460101790 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.357+0000 7fc46762b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4601986a0 con 0x7fc460100de0 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.358+0000 7fc464bc6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc460101790 0x7fc46019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.358+0000 7fc464bc6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc460101790 0x7fc46019eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60556/0 (socket says 172.21.15.121:60556) 2024-04-05T22:03:56.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.358+0000 7fc464bc6700 1 -- 172.21.15.121:0/275855279 learned_addr learned my addr 172.21.15.121:0/275855279 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:56.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.358+0000 7fc464bc6700 1 -- 172.21.15.121:0/275855279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc460100de0 msgr2=0x7fc46019e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:56.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.358+0000 7fc464bc6700 1 --2- 172.21.15.121:0/275855279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc460100de0 0x7fc46019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.358+0000 7fc464bc6700 1 -- 172.21.15.121:0/275855279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c0095f0 con 0x7fc460101790 2024-04-05T22:03:56.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.358+0000 7fc464bc6700 1 --2- 172.21.15.121:0/275855279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc460101790 0x7fc46019eb60 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc45000d900 tx=0x7fc45000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:56.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.358+0000 7fc4567fc700 1 -- 172.21.15.121:0/275855279 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc450009900 con 0x7fc460101790 2024-04-05T22:03:56.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.358+0000 7fc4567fc700 1 -- 172.21.15.121:0/275855279 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc450010460 con 0x7fc460101790 2024-04-05T22:03:56.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.358+0000 7fc46762b700 1 -- 172.21.15.121:0/275855279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc460198980 con 0x7fc460101790 2024-04-05T22:03:56.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.358+0000 7fc4567fc700 1 -- 172.21.15.121:0/275855279 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc45000f5f0 con 0x7fc460101790 2024-04-05T22:03:56.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.358+0000 7fc46762b700 1 -- 172.21.15.121:0/275855279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc460198ed0 con 0x7fc460101790 2024-04-05T22:03:56.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.359+0000 7fc46762b700 1 -- 172.21.15.121:0/275855279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc460104f60 con 0x7fc460101790 2024-04-05T22:03:56.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.360+0000 7fc4567fc700 1 -- 172.21.15.121:0/275855279 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc4500105d0 con 0x7fc460101790 2024-04-05T22:03:56.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.360+0000 7fc4567fc700 1 --2- 172.21.15.121:0/275855279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc44c06cce0 0x7fc44c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:56.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.360+0000 7fc4567fc700 1 -- 172.21.15.121:0/275855279 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc45008c3c0 con 0x7fc460101790 2024-04-05T22:03:56.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.360+0000 7fc4653c7700 1 --2- 172.21.15.121:0/275855279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc44c06cce0 0x7fc44c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:56.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.360+0000 7fc4653c7700 1 --2- 172.21.15.121:0/275855279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc44c06cce0 0x7fc44c06f1a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00b5c0 tx=0x7fc45c005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:56.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.365+0000 7fc4567fc700 1 -- 172.21.15.121:0/275855279 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc450059ec0 con 0x7fc460101790 2024-04-05T22:03:56.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.640+0000 7fc46762b700 1 -- 172.21.15.121:0/275855279 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc46004ea90 con 0x7fc460101790 2024-04-05T22:03:56.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.640+0000 7fc4567fc700 1 -- 172.21.15.121:0/275855279 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fc450059a50 con 0x7fc460101790 2024-04-05T22:03:56.640 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T22:03:56.641 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T22:03:56.642 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T22:03:56.642 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:03:56.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.642+0000 7fc46762b700 1 -- 172.21.15.121:0/275855279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc44c06cce0 msgr2=0x7fc44c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:56.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.642+0000 7fc46762b700 1 --2- 172.21.15.121:0/275855279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc44c06cce0 0x7fc44c06f1a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00b5c0 tx=0x7fc45c005960 comp rx=0 tx=0).stop 2024-04-05T22:03:56.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.642+0000 7fc46762b700 1 -- 172.21.15.121:0/275855279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc460101790 msgr2=0x7fc46019eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:56.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.642+0000 7fc46762b700 1 --2- 172.21.15.121:0/275855279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc460101790 0x7fc46019eb60 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fc45000d900 tx=0x7fc45000dcc0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.642+0000 7fc46762b700 1 -- 172.21.15.121:0/275855279 shutdown_connections 2024-04-05T22:03:56.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.642+0000 7fc46762b700 1 --2- 172.21.15.121:0/275855279 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc44c06cce0 0x7fc44c06f1a0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.642+0000 7fc46762b700 1 --2- 172.21.15.121:0/275855279 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc460100de0 0x7fc46019e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.642+0000 7fc46762b700 1 --2- 172.21.15.121:0/275855279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc460101790 0x7fc46019eb60 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.642+0000 7fc46762b700 1 -- 172.21.15.121:0/275855279 >> 172.21.15.121:0/275855279 conn(0x7fc4600fc670 msgr2=0x7fc4600fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:56.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.642+0000 7fc46762b700 1 -- 172.21.15.121:0/275855279 shutdown_connections 2024-04-05T22:03:56.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.642+0000 7fc46762b700 1 -- 172.21.15.121:0/275855279 wait complete. 2024-04-05T22:03:56.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 -- 172.21.15.121:0/1359413961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff59c105a40 msgr2=0x7ff59c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:56.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 --2- 172.21.15.121:0/1359413961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff59c105a40 0x7ff59c105e20 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff590009910 tx=0x7ff590009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 -- 172.21.15.121:0/1359413961 shutdown_connections 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 --2- 172.21.15.121:0/1359413961 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff59c0ffbd0 0x7ff59c100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 --2- 172.21.15.121:0/1359413961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff59c105a40 0x7ff59c105e20 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 -- 172.21.15.121:0/1359413961 >> 172.21.15.121:0/1359413961 conn(0x7ff59c0755a0 msgr2=0x7ff59c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 -- 172.21.15.121:0/1359413961 shutdown_connections 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 -- 172.21.15.121:0/1359413961 wait complete. 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 Processor -- start 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 -- start start 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff59c0ffbd0 0x7ff59c196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff59c105a40 0x7ff59c196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c1971d0 con 0x7ff59c105a40 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.745+0000 7ff5a2a64700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c197310 con 0x7ff59c0ffbd0 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.746+0000 7ff59bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff59c105a40 0x7ff59c196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.746+0000 7ff59bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff59c105a40 0x7ff59c196b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60576/0 (socket says 172.21.15.121:60576) 2024-04-05T22:03:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.746+0000 7ff59bfff700 1 -- 172.21.15.121:0/520176413 learned_addr learned my addr 172.21.15.121:0/520176413 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.746+0000 7ff5a0800700 1 --2- 172.21.15.121:0/520176413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff59c0ffbd0 0x7ff59c196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.746+0000 7ff59bfff700 1 -- 172.21.15.121:0/520176413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff59c0ffbd0 msgr2=0x7ff59c196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.746+0000 7ff59bfff700 1 --2- 172.21.15.121:0/520176413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff59c0ffbd0 0x7ff59c196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.746+0000 7ff59bfff700 1 -- 172.21.15.121:0/520176413 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5900095f0 con 0x7ff59c105a40 2024-04-05T22:03:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.746+0000 7ff59bfff700 1 --2- 172.21.15.121:0/520176413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff59c105a40 0x7ff59c196b80 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ff58800c930 tx=0x7ff58800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.746+0000 7ff599ffb700 1 -- 172.21.15.121:0/520176413 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff588007ad0 con 0x7ff59c105a40 2024-04-05T22:03:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.746+0000 7ff599ffb700 1 -- 172.21.15.121:0/520176413 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff58800ce80 con 0x7ff59c105a40 2024-04-05T22:03:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.746+0000 7ff5a2a64700 1 -- 172.21.15.121:0/520176413 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c19b160 con 0x7ff59c105a40 2024-04-05T22:03:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.746+0000 7ff599ffb700 1 -- 172.21.15.121:0/520176413 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff588018630 con 0x7ff59c105a40 2024-04-05T22:03:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.746+0000 7ff5a2a64700 1 -- 172.21.15.121:0/520176413 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c19b6b0 con 0x7ff59c105a40 2024-04-05T22:03:56.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.747+0000 7ff5a2a64700 1 -- 172.21.15.121:0/520176413 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59c109050 con 0x7ff59c105a40 2024-04-05T22:03:56.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.748+0000 7ff599ffb700 1 -- 172.21.15.121:0/520176413 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7ff588007c30 con 0x7ff59c105a40 2024-04-05T22:03:56.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.748+0000 7ff599ffb700 1 --2- 172.21.15.121:0/520176413 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff58c06cce0 0x7ff58c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:56.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.748+0000 7ff599ffb700 1 -- 172.21.15.121:0/520176413 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7ff58808c5a0 con 0x7ff59c105a40 2024-04-05T22:03:56.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.748+0000 7ff5a0800700 1 --2- 172.21.15.121:0/520176413 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff58c06cce0 0x7ff58c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:56.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.748+0000 7ff5a0800700 1 --2- 172.21.15.121:0/520176413 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff58c06cce0 0x7ff58c06f1a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff590006010 tx=0x7ff59001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:56.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.752+0000 7ff599ffb700 1 -- 172.21.15.121:0/520176413 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7ff588056610 con 0x7ff59c105a40 2024-04-05T22:03:56.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.987+0000 7ff5a2a64700 1 -- 172.21.15.121:0/520176413 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff59c04ea90 con 0x7ff59c105a40 2024-04-05T22:03:56.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.988+0000 7ff599ffb700 1 -- 172.21.15.121:0/520176413 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7ff588059c30 con 0x7ff59c105a40 2024-04-05T22:03:56.988 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T22:03:56.988 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T22:03:56.988 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:03:56.988 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T22:03:56.988 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T22:03:56.989 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T22:03:56.990 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:03:56.990 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T22:03:56.990 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T22:03:56.990 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T22:03:56.990 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T22:03:56.990 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T22:03:56.990 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T22:03:56.990 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T22:03:56.990 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T22:03:56.990 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T22:03:56.990 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T22:03:56.990 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T22:03:56.990 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T22:03:56.991 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:03:56.991 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:03:56.991 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T22:03:56.991 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:03:56.991 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:03:56.991 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:03:56.991 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:03:56.991 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:03:56.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.990+0000 7ff5a2a64700 1 -- 172.21.15.121:0/520176413 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff58c06cce0 msgr2=0x7ff58c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:56.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.990+0000 7ff5a2a64700 1 --2- 172.21.15.121:0/520176413 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff58c06cce0 0x7ff58c06f1a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff590006010 tx=0x7ff59001a040 comp rx=0 tx=0).stop 2024-04-05T22:03:56.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.990+0000 7ff5a2a64700 1 -- 172.21.15.121:0/520176413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff59c105a40 msgr2=0x7ff59c196b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:56.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.990+0000 7ff5a2a64700 1 --2- 172.21.15.121:0/520176413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff59c105a40 0x7ff59c196b80 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7ff58800c930 tx=0x7ff58800cc40 comp rx=0 tx=0).stop 2024-04-05T22:03:56.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.990+0000 7ff5a2a64700 1 -- 172.21.15.121:0/520176413 shutdown_connections 2024-04-05T22:03:56.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.990+0000 7ff5a2a64700 1 --2- 172.21.15.121:0/520176413 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7ff58c06cce0 0x7ff58c06f1a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.990+0000 7ff5a2a64700 1 --2- 172.21.15.121:0/520176413 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7ff59c0ffbd0 0x7ff59c196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.990+0000 7ff5a2a64700 1 --2- 172.21.15.121:0/520176413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff59c105a40 0x7ff59c196b80 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:56.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.990+0000 7ff5a2a64700 1 -- 172.21.15.121:0/520176413 >> 172.21.15.121:0/520176413 conn(0x7ff59c0755a0 msgr2=0x7ff59c068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:56.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.990+0000 7ff5a2a64700 1 -- 172.21.15.121:0/520176413 shutdown_connections 2024-04-05T22:03:56.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:56.990+0000 7ff5a2a64700 1 -- 172.21.15.121:0/520176413 wait complete. 2024-04-05T22:03:56.993 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T22:03:57.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.094+0000 7f9ab0193700 1 -- 172.21.15.121:0/762708421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa8069020 msgr2=0x7f9aa8105790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:57.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.094+0000 7f9ab0193700 1 --2- 172.21.15.121:0/762708421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa8069020 0x7f9aa8105790 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009910 tx=0x7f9aa4009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:57.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.094+0000 7f9ab0193700 1 -- 172.21.15.121:0/762708421 shutdown_connections 2024-04-05T22:03:57.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.094+0000 7f9ab0193700 1 --2- 172.21.15.121:0/762708421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa8069020 0x7f9aa8105790 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.094+0000 7f9ab0193700 1 --2- 172.21.15.121:0/762708421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9aa8068670 0x7f9aa8068a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.094+0000 7f9ab0193700 1 -- 172.21.15.121:0/762708421 >> 172.21.15.121:0/762708421 conn(0x7f9aa80755a0 msgr2=0x7f9aa80759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:57.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.094+0000 7f9ab0193700 1 -- 172.21.15.121:0/762708421 shutdown_connections 2024-04-05T22:03:57.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.094+0000 7f9ab0193700 1 -- 172.21.15.121:0/762708421 wait complete. 2024-04-05T22:03:57.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9ab0193700 1 Processor -- start 2024-04-05T22:03:57.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9ab0193700 1 -- start start 2024-04-05T22:03:57.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9ab0193700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa8068670 0x7f9aa819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:57.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9ab0193700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9aa8069020 0x7f9aa819ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9ab0193700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa819f2e0 con 0x7f9aa8068670 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9ab0193700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa8198790 con 0x7f9aa8069020 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9aadf2f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa8068670 0x7f9aa819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9aadf2f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa8068670 0x7f9aa819e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60602/0 (socket says 172.21.15.121:60602) 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9aadf2f700 1 -- 172.21.15.121:0/3081563294 learned_addr learned my addr 172.21.15.121:0/3081563294 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9aad72e700 1 --2- 172.21.15.121:0/3081563294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9aa8069020 0x7f9aa819ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9aadf2f700 1 -- 172.21.15.121:0/3081563294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9aa8069020 msgr2=0x7f9aa819ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9aadf2f700 1 --2- 172.21.15.121:0/3081563294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9aa8069020 0x7f9aa819ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9aadf2f700 1 -- 172.21.15.121:0/3081563294 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa40095f0 con 0x7f9aa8068670 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.095+0000 7f9aadf2f700 1 --2- 172.21.15.121:0/3081563294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa8068670 0x7f9aa819e710 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800d8d0 tx=0x7f9a9800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.096+0000 7f9a9effd700 1 -- 172.21.15.121:0/3081563294 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a980098a0 con 0x7f9aa8068670 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.096+0000 7f9a9effd700 1 -- 172.21.15.121:0/3081563294 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a98010460 con 0x7f9aa8068670 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.096+0000 7f9ab0193700 1 -- 172.21.15.121:0/3081563294 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa8198a70 con 0x7f9aa8068670 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.096+0000 7f9a9effd700 1 -- 172.21.15.121:0/3081563294 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a98009ea0 con 0x7f9aa8068670 2024-04-05T22:03:57.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.096+0000 7f9ab0193700 1 -- 172.21.15.121:0/3081563294 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa8198f90 con 0x7f9aa8068670 2024-04-05T22:03:57.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.097+0000 7f9ab0193700 1 -- 172.21.15.121:0/3081563294 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa804ea90 con 0x7f9aa8068670 2024-04-05T22:03:57.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.097+0000 7f9a9effd700 1 -- 172.21.15.121:0/3081563294 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9a980105d0 con 0x7f9aa8068670 2024-04-05T22:03:57.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.098+0000 7f9a9effd700 1 --2- 172.21.15.121:0/3081563294 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a9406cce0 0x7f9a9406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:57.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.098+0000 7f9a9effd700 1 -- 172.21.15.121:0/3081563294 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9a9808c430 con 0x7f9aa8068670 2024-04-05T22:03:57.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.098+0000 7f9aad72e700 1 --2- 172.21.15.121:0/3081563294 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a9406cce0 0x7f9a9406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:57.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.098+0000 7f9aad72e700 1 --2- 172.21.15.121:0/3081563294 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a9406cce0 0x7f9a9406f1a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006010 tx=0x7f9aa401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:57.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.102+0000 7f9a9effd700 1 -- 172.21.15.121:0/3081563294 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9a9805a040 con 0x7f9aa8068670 2024-04-05T22:03:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:56 smithi121 ceph-mon[145859]: from='client.35858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:56 smithi121 ceph-mon[145859]: pgmap v1132: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:56 smithi121 ceph-mon[145859]: from='client.35862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:56 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/275855279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:03:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:56 smithi188 ceph-mon[107793]: from='client.35858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:56 smithi188 ceph-mon[107793]: pgmap v1132: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:56 smithi188 ceph-mon[107793]: from='client.35862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:56 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/275855279' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:03:57.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.311+0000 7f9ab0193700 1 -- 172.21.15.121:0/3081563294 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9aa81993f0 con 0x7f9a9406cce0 2024-04-05T22:03:57.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.313+0000 7f9a9effd700 1 -- 172.21.15.121:0/3081563294 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9aa81993f0 con 0x7f9a9406cce0 2024-04-05T22:03:57.313 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:03:57.313 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T22:03:57.313 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T22:03:57.313 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T22:03:57.313 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T22:03:57.313 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T22:03:57.313 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T22:03:57.313 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T22:03:57.313 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T22:03:57.313 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T22:03:57.313 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T22:03:57.313 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T22:03:57.314 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T22:03:57.314 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T22:03:57.314 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:03:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.315+0000 7f9ab0193700 1 -- 172.21.15.121:0/3081563294 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a9406cce0 msgr2=0x7f9a9406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.315+0000 7f9ab0193700 1 --2- 172.21.15.121:0/3081563294 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a9406cce0 0x7f9a9406f1a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4006010 tx=0x7f9aa401a040 comp rx=0 tx=0).stop 2024-04-05T22:03:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.315+0000 7f9ab0193700 1 -- 172.21.15.121:0/3081563294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa8068670 msgr2=0x7f9aa819e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.315+0000 7f9ab0193700 1 --2- 172.21.15.121:0/3081563294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa8068670 0x7f9aa819e710 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800d8d0 tx=0x7f9a9800dbe0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.315+0000 7f9ab0193700 1 -- 172.21.15.121:0/3081563294 shutdown_connections 2024-04-05T22:03:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.315+0000 7f9ab0193700 1 --2- 172.21.15.121:0/3081563294 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9a9406cce0 0x7f9a9406f1a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.315+0000 7f9ab0193700 1 --2- 172.21.15.121:0/3081563294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aa8068670 0x7f9aa819e710 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.315+0000 7f9ab0193700 1 --2- 172.21.15.121:0/3081563294 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9aa8069020 0x7f9aa819ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.315+0000 7f9ab0193700 1 -- 172.21.15.121:0/3081563294 >> 172.21.15.121:0/3081563294 conn(0x7f9aa80755a0 msgr2=0x7f9aa8102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:57.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.315+0000 7f9ab0193700 1 -- 172.21.15.121:0/3081563294 shutdown_connections 2024-04-05T22:03:57.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.315+0000 7f9ab0193700 1 -- 172.21.15.121:0/3081563294 wait complete. 2024-04-05T22:03:57.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.421+0000 7f674f7a8700 1 -- 172.21.15.121:0/16240740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6748105a40 msgr2=0x7f6748105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:57.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.421+0000 7f674f7a8700 1 --2- 172.21.15.121:0/16240740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6748105a40 0x7f6748105e20 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f6744009910 tx=0x7f6744009c20 comp rx=0 tx=0).stop 2024-04-05T22:03:57.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.421+0000 7f674f7a8700 1 -- 172.21.15.121:0/16240740 shutdown_connections 2024-04-05T22:03:57.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.421+0000 7f674f7a8700 1 --2- 172.21.15.121:0/16240740 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f67480ffbd0 0x7f6748100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.421+0000 7f674f7a8700 1 --2- 172.21.15.121:0/16240740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6748105a40 0x7f6748105e20 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.421+0000 7f674f7a8700 1 -- 172.21.15.121:0/16240740 >> 172.21.15.121:0/16240740 conn(0x7f67480755a0 msgr2=0x7f67480759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:57.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.421+0000 7f674f7a8700 1 -- 172.21.15.121:0/16240740 shutdown_connections 2024-04-05T22:03:57.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.421+0000 7f674f7a8700 1 -- 172.21.15.121:0/16240740 wait complete. 2024-04-05T22:03:57.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674f7a8700 1 Processor -- start 2024-04-05T22:03:57.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674f7a8700 1 -- start start 2024-04-05T22:03:57.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674f7a8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67480ffbd0 0x7f674819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:57.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674f7a8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6748105a40 0x7f674819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:57.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674f7a8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674819f220 con 0x7f67480ffbd0 2024-04-05T22:03:57.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674f7a8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67481986d0 con 0x7f6748105a40 2024-04-05T22:03:57.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674d544700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67480ffbd0 0x7f674819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:57.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674d544700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67480ffbd0 0x7f674819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60632/0 (socket says 172.21.15.121:60632) 2024-04-05T22:03:57.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674d544700 1 -- 172.21.15.121:0/2151588042 learned_addr learned my addr 172.21.15.121:0/2151588042 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:03:57.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674cd43700 1 --2- 172.21.15.121:0/2151588042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6748105a40 0x7f674819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:57.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674d544700 1 -- 172.21.15.121:0/2151588042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6748105a40 msgr2=0x7f674819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:57.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674d544700 1 --2- 172.21.15.121:0/2151588042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6748105a40 0x7f674819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674d544700 1 -- 172.21.15.121:0/2151588042 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67440095f0 con 0x7f67480ffbd0 2024-04-05T22:03:57.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.422+0000 7f674d544700 1 --2- 172.21.15.121:0/2151588042 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67480ffbd0 0x7f674819e650 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f67440092e0 tx=0x7f674400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:57.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.423+0000 7f673e7fc700 1 -- 172.21.15.121:0/2151588042 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f674401d070 con 0x7f67480ffbd0 2024-04-05T22:03:57.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.423+0000 7f673e7fc700 1 -- 172.21.15.121:0/2151588042 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6744005e00 con 0x7f67480ffbd0 2024-04-05T22:03:57.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.423+0000 7f674f7a8700 1 -- 172.21.15.121:0/2151588042 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6748198950 con 0x7f67480ffbd0 2024-04-05T22:03:57.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.423+0000 7f673e7fc700 1 -- 172.21.15.121:0/2151588042 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f674400fe90 con 0x7f67480ffbd0 2024-04-05T22:03:57.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.423+0000 7f674f7a8700 1 -- 172.21.15.121:0/2151588042 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6748198e40 con 0x7f67480ffbd0 2024-04-05T22:03:57.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.424+0000 7f674f7a8700 1 -- 172.21.15.121:0/2151588042 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674804ea90 con 0x7f67480ffbd0 2024-04-05T22:03:57.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.424+0000 7f673e7fc700 1 -- 172.21.15.121:0/2151588042 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f674400f6d0 con 0x7f67480ffbd0 2024-04-05T22:03:57.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.424+0000 7f673e7fc700 1 --2- 172.21.15.121:0/2151588042 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f673406cce0 0x7f673406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:03:57.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.425+0000 7f673e7fc700 1 -- 172.21.15.121:0/2151588042 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f674408d760 con 0x7f67480ffbd0 2024-04-05T22:03:57.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.425+0000 7f674cd43700 1 --2- 172.21.15.121:0/2151588042 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f673406cce0 0x7f673406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:03:57.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.425+0000 7f674cd43700 1 --2- 172.21.15.121:0/2151588042 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f673406cce0 0x7f673406f1a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6748199b40 tx=0x7f67380073d0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:03:57.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.429+0000 7f673e7fc700 1 -- 172.21.15.121:0/2151588042 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f674405b2e0 con 0x7f67480ffbd0 2024-04-05T22:03:57.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.702+0000 7f674f7a8700 1 -- 172.21.15.121:0/2151588042 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f67481991b0 con 0x7f67480ffbd0 2024-04-05T22:03:57.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.703+0000 7f673e7fc700 1 -- 172.21.15.121:0/2151588042 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f6744026e10 con 0x7f67480ffbd0 2024-04-05T22:03:57.703 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:03:57.703 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:03:57.703 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:03:57.703 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T22:03:57.703 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:03:57.703 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T22:03:57.703 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:03:57.703 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T22:03:57.703 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:03:57.703 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T22:03:57.704 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:03:57.704 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:03:57.704 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:03:57.704 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:03:57.704 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:03:57.704 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:03:57.704 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:03:57.704 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:03:57.704 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:03:57.704 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:03:57.704 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T22:03:57.704 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:03:57.704 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T22:03:57.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.705+0000 7f674f7a8700 1 -- 172.21.15.121:0/2151588042 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f673406cce0 msgr2=0x7f673406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:57.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.705+0000 7f674f7a8700 1 --2- 172.21.15.121:0/2151588042 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f673406cce0 0x7f673406f1a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6748199b40 tx=0x7f67380073d0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.705+0000 7f674f7a8700 1 -- 172.21.15.121:0/2151588042 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67480ffbd0 msgr2=0x7f674819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:03:57.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.705+0000 7f674f7a8700 1 --2- 172.21.15.121:0/2151588042 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67480ffbd0 0x7f674819e650 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f67440092e0 tx=0x7f674400bd70 comp rx=0 tx=0).stop 2024-04-05T22:03:57.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.705+0000 7f674f7a8700 1 -- 172.21.15.121:0/2151588042 shutdown_connections 2024-04-05T22:03:57.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.705+0000 7f674f7a8700 1 --2- 172.21.15.121:0/2151588042 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f673406cce0 0x7f673406f1a0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.705+0000 7f674f7a8700 1 --2- 172.21.15.121:0/2151588042 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67480ffbd0 0x7f674819e650 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.705+0000 7f674f7a8700 1 --2- 172.21.15.121:0/2151588042 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6748105a40 0x7f674819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:03:57.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.705+0000 7f674f7a8700 1 -- 172.21.15.121:0/2151588042 >> 172.21.15.121:0/2151588042 conn(0x7f67480755a0 msgr2=0x7f67480fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:03:57.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.705+0000 7f674f7a8700 1 -- 172.21.15.121:0/2151588042 shutdown_connections 2024-04-05T22:03:57.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:03:57.705+0000 7f674f7a8700 1 -- 172.21.15.121:0/2151588042 wait complete. 2024-04-05T22:03:58.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:57 smithi121 ceph-mon[145859]: from='client.35866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:58.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:57 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/520176413' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:03:58.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:57 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2151588042' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:03:58.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:57 smithi188 ceph-mon[107793]: from='client.35866 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:58.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:57 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/520176413' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:03:58.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:57 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2151588042' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:03:59.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:58 smithi121 ceph-mon[145859]: from='client.35878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:59.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:03:58 smithi121 ceph-mon[145859]: pgmap v1133: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:03:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:58 smithi188 ceph-mon[107793]: from='client.35878 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:03:59.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:03:58 smithi188 ceph-mon[107793]: pgmap v1133: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:00 smithi121 ceph-mon[145859]: pgmap v1134: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:04:01.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:00 smithi188 ceph-mon[107793]: pgmap v1134: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:04:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:02 smithi121 ceph-mon[145859]: pgmap v1135: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:03.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:02 smithi188 ceph-mon[107793]: pgmap v1135: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:04 smithi188 ceph-mon[107793]: pgmap v1136: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:04 smithi121 ceph-mon[145859]: pgmap v1136: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:06 smithi188 ceph-mon[107793]: pgmap v1137: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:07.376 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:06 smithi121 ceph-mon[145859]: pgmap v1137: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:09.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:08 smithi188 ceph-mon[107793]: pgmap v1138: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:09.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:08 smithi121 ceph-mon[145859]: pgmap v1138: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:10 smithi188 ceph-mon[107793]: pgmap v1139: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:11.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:10 smithi121 ceph-mon[145859]: pgmap v1139: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:13 smithi188 ceph-mon[107793]: pgmap v1140: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:13.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:13 smithi121 ceph-mon[145859]: pgmap v1140: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:15 smithi188 ceph-mon[107793]: pgmap v1141: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:15.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:15 smithi121 ceph-mon[145859]: pgmap v1141: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:04:16.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:04:17.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:17 smithi188 ceph-mon[107793]: pgmap v1142: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:17.469 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:17 smithi121 ceph-mon[145859]: pgmap v1142: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:19.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:19 smithi188 ceph-mon[107793]: pgmap v1143: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:19.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:19 smithi121 ceph-mon[145859]: pgmap v1143: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:21.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:21 smithi188 ceph-mon[107793]: pgmap v1144: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:21.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:21 smithi121 ceph-mon[145859]: pgmap v1144: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:23.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:23 smithi188 ceph-mon[107793]: pgmap v1145: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:23.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:23 smithi121 ceph-mon[145859]: pgmap v1145: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:25.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:25 smithi188 ceph-mon[107793]: pgmap v1146: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:25.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:25 smithi121 ceph-mon[145859]: pgmap v1146: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:27.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:27 smithi188 ceph-mon[107793]: pgmap v1147: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:27.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:27 smithi121 ceph-mon[145859]: pgmap v1147: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:27.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.817+0000 7f42ab5ad700 1 -- 172.21.15.121:0/593538959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42a4105a40 msgr2=0x7f42a4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:27.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.817+0000 7f42ab5ad700 1 --2- 172.21.15.121:0/593538959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42a4105a40 0x7f42a4105e20 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f4294009910 tx=0x7f4294009c20 comp rx=0 tx=0).stop 2024-04-05T22:04:27.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.817+0000 7f42ab5ad700 1 -- 172.21.15.121:0/593538959 shutdown_connections 2024-04-05T22:04:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.817+0000 7f42ab5ad700 1 --2- 172.21.15.121:0/593538959 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42a40ffbd0 0x7f42a4100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.817+0000 7f42ab5ad700 1 --2- 172.21.15.121:0/593538959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42a4105a40 0x7f42a4105e20 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.817+0000 7f42ab5ad700 1 -- 172.21.15.121:0/593538959 >> 172.21.15.121:0/593538959 conn(0x7f42a40755a0 msgr2=0x7f42a40759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.817+0000 7f42ab5ad700 1 -- 172.21.15.121:0/593538959 shutdown_connections 2024-04-05T22:04:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.817+0000 7f42ab5ad700 1 -- 172.21.15.121:0/593538959 wait complete. 2024-04-05T22:04:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.817+0000 7f42ab5ad700 1 Processor -- start 2024-04-05T22:04:27.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42ab5ad700 1 -- start start 2024-04-05T22:04:27.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42ab5ad700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42a40ffbd0 0x7f42a41974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:27.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42ab5ad700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42a4105a40 0x7f42a41979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:27.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42ab5ad700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a4198000 con 0x7f42a40ffbd0 2024-04-05T22:04:27.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42ab5ad700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a4198170 con 0x7f42a4105a40 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42a9349700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42a40ffbd0 0x7f42a41974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42a9349700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42a40ffbd0 0x7f42a41974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36908/0 (socket says 172.21.15.121:36908) 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42a9349700 1 -- 172.21.15.121:0/1277525772 learned_addr learned my addr 172.21.15.121:0/1277525772 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42a8b48700 1 --2- 172.21.15.121:0/1277525772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42a4105a40 0x7f42a41979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42a9349700 1 -- 172.21.15.121:0/1277525772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42a4105a40 msgr2=0x7f42a41979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42a9349700 1 --2- 172.21.15.121:0/1277525772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42a4105a40 0x7f42a41979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42a9349700 1 -- 172.21.15.121:0/1277525772 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42940095f0 con 0x7f42a40ffbd0 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42a9349700 1 --2- 172.21.15.121:0/1277525772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42a40ffbd0 0x7f42a41974a0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f4294000c00 tx=0x7f429400bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f429a7fc700 1 -- 172.21.15.121:0/1277525772 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f429401d070 con 0x7f42a40ffbd0 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f429a7fc700 1 -- 172.21.15.121:0/1277525772 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4294005dc0 con 0x7f42a40ffbd0 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f42ab5ad700 1 -- 172.21.15.121:0/1277525772 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a41a0510 con 0x7f42a40ffbd0 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.818+0000 7f429a7fc700 1 -- 172.21.15.121:0/1277525772 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4294003940 con 0x7f42a40ffbd0 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.819+0000 7f42ab5ad700 1 -- 172.21.15.121:0/1277525772 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a41a0920 con 0x7f42a40ffbd0 2024-04-05T22:04:27.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.819+0000 7f42ab5ad700 1 -- 172.21.15.121:0/1277525772 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a4108040 con 0x7f42a40ffbd0 2024-04-05T22:04:27.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.820+0000 7f429a7fc700 1 -- 172.21.15.121:0/1277525772 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4294004ae0 con 0x7f42a40ffbd0 2024-04-05T22:04:27.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.820+0000 7f429a7fc700 1 --2- 172.21.15.121:0/1277525772 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f429006cce0 0x7f429006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:27.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.820+0000 7f429a7fc700 1 -- 172.21.15.121:0/1277525772 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f429408db90 con 0x7f42a40ffbd0 2024-04-05T22:04:27.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.820+0000 7f42a8b48700 1 --2- 172.21.15.121:0/1277525772 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f429006cce0 0x7f429006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:27.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.821+0000 7f42a8b48700 1 --2- 172.21.15.121:0/1277525772 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f429006cce0 0x7f429006f1a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f42a4068ce0 tx=0x7f42a0009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:27.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:27.825+0000 7f429a7fc700 1 -- 172.21.15.121:0/1277525772 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4294057c00 con 0x7f42a40ffbd0 2024-04-05T22:04:28.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.038+0000 7f42ab5ad700 1 -- 172.21.15.121:0/1277525772 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f42a40611d0 con 0x7f429006cce0 2024-04-05T22:04:28.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.039+0000 7f429a7fc700 1 -- 172.21.15.121:0/1277525772 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f42a40611d0 con 0x7f429006cce0 2024-04-05T22:04:28.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.041+0000 7f42ab5ad700 1 -- 172.21.15.121:0/1277525772 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f429006cce0 msgr2=0x7f429006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:28.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.041+0000 7f42ab5ad700 1 --2- 172.21.15.121:0/1277525772 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f429006cce0 0x7f429006f1a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f42a4068ce0 tx=0x7f42a0009260 comp rx=0 tx=0).stop 2024-04-05T22:04:28.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.041+0000 7f42ab5ad700 1 -- 172.21.15.121:0/1277525772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42a40ffbd0 msgr2=0x7f42a41974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:28.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.041+0000 7f42ab5ad700 1 --2- 172.21.15.121:0/1277525772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42a40ffbd0 0x7f42a41974a0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f4294000c00 tx=0x7f429400bde0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.041+0000 7f42ab5ad700 1 -- 172.21.15.121:0/1277525772 shutdown_connections 2024-04-05T22:04:28.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.041+0000 7f42ab5ad700 1 --2- 172.21.15.121:0/1277525772 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f429006cce0 0x7f429006f1a0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.041+0000 7f42ab5ad700 1 --2- 172.21.15.121:0/1277525772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42a40ffbd0 0x7f42a41974a0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.041+0000 7f42ab5ad700 1 --2- 172.21.15.121:0/1277525772 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f42a4105a40 0x7f42a41979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.041+0000 7f42ab5ad700 1 -- 172.21.15.121:0/1277525772 >> 172.21.15.121:0/1277525772 conn(0x7f42a40755a0 msgr2=0x7f42a40fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:28.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.041+0000 7f42ab5ad700 1 -- 172.21.15.121:0/1277525772 shutdown_connections 2024-04-05T22:04:28.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.041+0000 7f42ab5ad700 1 -- 172.21.15.121:0/1277525772 wait complete. 2024-04-05T22:04:28.051 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T22:04:28.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.145+0000 7f82d8278700 1 -- 172.21.15.121:0/7248083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82d0100de0 msgr2=0x7f82d01011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:28.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.145+0000 7f82d8278700 1 --2- 172.21.15.121:0/7248083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82d0100de0 0x7f82d01011c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f82c0009910 tx=0x7f82c0009c20 comp rx=0 tx=0).stop 2024-04-05T22:04:28.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.145+0000 7f82d8278700 1 -- 172.21.15.121:0/7248083 shutdown_connections 2024-04-05T22:04:28.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.145+0000 7f82d8278700 1 --2- 172.21.15.121:0/7248083 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82d0101790 0x7f82d01057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.145+0000 7f82d8278700 1 --2- 172.21.15.121:0/7248083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82d0100de0 0x7f82d01011c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.145+0000 7f82d8278700 1 -- 172.21.15.121:0/7248083 >> 172.21.15.121:0/7248083 conn(0x7f82d00fc670 msgr2=0x7f82d00fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:28.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.145+0000 7f82d8278700 1 -- 172.21.15.121:0/7248083 shutdown_connections 2024-04-05T22:04:28.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.145+0000 7f82d8278700 1 -- 172.21.15.121:0/7248083 wait complete. 2024-04-05T22:04:28.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.145+0000 7f82d8278700 1 Processor -- start 2024-04-05T22:04:28.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.145+0000 7f82d8278700 1 -- start start 2024-04-05T22:04:28.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82d8278700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82d0100de0 0x7f82d0196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:28.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82d8278700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82d0101790 0x7f82d0196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:28.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82d8278700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d0197260 con 0x7f82d0100de0 2024-04-05T22:04:28.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82d8278700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d0199f90 con 0x7f82d0101790 2024-04-05T22:04:28.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82d6014700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82d0100de0 0x7f82d0196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:28.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82d6014700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82d0100de0 0x7f82d0196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36912/0 (socket says 172.21.15.121:36912) 2024-04-05T22:04:28.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82d6014700 1 -- 172.21.15.121:0/3486819543 learned_addr learned my addr 172.21.15.121:0/3486819543 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:04:28.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82d6014700 1 -- 172.21.15.121:0/3486819543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82d0101790 msgr2=0x7f82d0196bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:28.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82d6014700 1 --2- 172.21.15.121:0/3486819543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82d0101790 0x7f82d0196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82d6014700 1 -- 172.21.15.121:0/3486819543 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c00095f0 con 0x7f82d0100de0 2024-04-05T22:04:28.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82d6014700 1 --2- 172.21.15.121:0/3486819543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82d0100de0 0x7f82d0196690 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f82c0006010 tx=0x7f82c000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:28.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82c77fe700 1 -- 172.21.15.121:0/3486819543 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82c001d070 con 0x7f82d0100de0 2024-04-05T22:04:28.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82c77fe700 1 -- 172.21.15.121:0/3486819543 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82c000f460 con 0x7f82d0100de0 2024-04-05T22:04:28.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82d8278700 1 -- 172.21.15.121:0/3486819543 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d019a210 con 0x7f82d0100de0 2024-04-05T22:04:28.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82c77fe700 1 -- 172.21.15.121:0/3486819543 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82c0021620 con 0x7f82d0100de0 2024-04-05T22:04:28.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.146+0000 7f82d8278700 1 -- 172.21.15.121:0/3486819543 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d019a700 con 0x7f82d0100de0 2024-04-05T22:04:28.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.147+0000 7f82d8278700 1 -- 172.21.15.121:0/3486819543 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d0109180 con 0x7f82d0100de0 2024-04-05T22:04:28.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.148+0000 7f82c77fe700 1 -- 172.21.15.121:0/3486819543 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f82c000fa40 con 0x7f82d0100de0 2024-04-05T22:04:28.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.148+0000 7f82c77fe700 1 --2- 172.21.15.121:0/3486819543 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82bc06cce0 0x7f82bc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:28.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.148+0000 7f82c77fe700 1 -- 172.21.15.121:0/3486819543 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f82c008d6c0 con 0x7f82d0100de0 2024-04-05T22:04:28.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.148+0000 7f82d5813700 1 --2- 172.21.15.121:0/3486819543 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82bc06cce0 0x7f82bc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:28.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.148+0000 7f82d5813700 1 --2- 172.21.15.121:0/3486819543 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82bc06cce0 0x7f82bc06f1a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f82cc005950 tx=0x7f82cc0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:28.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.152+0000 7f82c77fe700 1 -- 172.21.15.121:0/3486819543 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f82c00577b0 con 0x7f82d0100de0 2024-04-05T22:04:28.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.362+0000 7f82d8278700 1 -- 172.21.15.121:0/3486819543 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f82d002cce0 con 0x7f82bc06cce0 2024-04-05T22:04:28.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.363+0000 7f82c77fe700 1 -- 172.21.15.121:0/3486819543 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f82d002cce0 con 0x7f82bc06cce0 2024-04-05T22:04:28.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.365+0000 7f82d8278700 1 -- 172.21.15.121:0/3486819543 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82bc06cce0 msgr2=0x7f82bc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.365+0000 7f82d8278700 1 --2- 172.21.15.121:0/3486819543 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82bc06cce0 0x7f82bc06f1a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f82cc005950 tx=0x7f82cc0058e0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.365+0000 7f82d8278700 1 -- 172.21.15.121:0/3486819543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82d0100de0 msgr2=0x7f82d0196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.365+0000 7f82d8278700 1 --2- 172.21.15.121:0/3486819543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82d0100de0 0x7f82d0196690 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f82c0006010 tx=0x7f82c000be20 comp rx=0 tx=0).stop 2024-04-05T22:04:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.365+0000 7f82d8278700 1 -- 172.21.15.121:0/3486819543 shutdown_connections 2024-04-05T22:04:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.365+0000 7f82d8278700 1 --2- 172.21.15.121:0/3486819543 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f82bc06cce0 0x7f82bc06f1a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.365+0000 7f82d8278700 1 --2- 172.21.15.121:0/3486819543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82d0100de0 0x7f82d0196690 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.365+0000 7f82d8278700 1 --2- 172.21.15.121:0/3486819543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f82d0101790 0x7f82d0196bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.365+0000 7f82d8278700 1 -- 172.21.15.121:0/3486819543 >> 172.21.15.121:0/3486819543 conn(0x7f82d00fc670 msgr2=0x7f82d00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.366+0000 7f82d8278700 1 -- 172.21.15.121:0/3486819543 shutdown_connections 2024-04-05T22:04:28.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.366+0000 7f82d8278700 1 -- 172.21.15.121:0/3486819543 wait complete. 2024-04-05T22:04:28.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.468+0000 7f55951dc700 1 -- 172.21.15.121:0/3825610032 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55900ffdb0 msgr2=0x7f5590103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:28.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.468+0000 7f55951dc700 1 --2- 172.21.15.121:0/3825610032 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55900ffdb0 0x7f5590103e00 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f5580009910 tx=0x7f5580009c20 comp rx=0 tx=0).stop 2024-04-05T22:04:28.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.468+0000 7f55951dc700 1 -- 172.21.15.121:0/3825610032 shutdown_connections 2024-04-05T22:04:28.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.468+0000 7f55951dc700 1 --2- 172.21.15.121:0/3825610032 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55900ffdb0 0x7f5590103e00 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.468+0000 7f55951dc700 1 --2- 172.21.15.121:0/3825610032 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55900ff400 0x7f55900ff7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.468+0000 7f55951dc700 1 -- 172.21.15.121:0/3825610032 >> 172.21.15.121:0/3825610032 conn(0x7f5590075e30 msgr2=0x7f55900fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:28.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.468+0000 7f55951dc700 1 -- 172.21.15.121:0/3825610032 shutdown_connections 2024-04-05T22:04:28.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.468+0000 7f55951dc700 1 -- 172.21.15.121:0/3825610032 wait complete. 2024-04-05T22:04:28.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.469+0000 7f55951dc700 1 Processor -- start 2024-04-05T22:04:28.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.469+0000 7f55951dc700 1 -- start start 2024-04-05T22:04:28.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f55951dc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55900ff400 0x7f55901974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:28.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f55951dc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55900ffdb0 0x7f55901979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:28.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f55951dc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5590198000 con 0x7f55900ff400 2024-04-05T22:04:28.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f55951dc700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5590198170 con 0x7f55900ffdb0 2024-04-05T22:04:28.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f558ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55900ff400 0x7f55901974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:28.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f558ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55900ff400 0x7f55901974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36930/0 (socket says 172.21.15.121:36930) 2024-04-05T22:04:28.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f558ed9d700 1 -- 172.21.15.121:0/1620593185 learned_addr learned my addr 172.21.15.121:0/1620593185 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:04:28.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f558e59c700 1 --2- 172.21.15.121:0/1620593185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55900ffdb0 0x7f55901979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:28.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f558ed9d700 1 -- 172.21.15.121:0/1620593185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55900ffdb0 msgr2=0x7f55901979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:28.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f558ed9d700 1 --2- 172.21.15.121:0/1620593185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55900ffdb0 0x7f55901979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f558ed9d700 1 -- 172.21.15.121:0/1620593185 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55800095f0 con 0x7f55900ff400 2024-04-05T22:04:28.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f558ed9d700 1 --2- 172.21.15.121:0/1620593185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55900ff400 0x7f55901974a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f557800d8d0 tx=0x7f557800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:28.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f5587fff700 1 -- 172.21.15.121:0/1620593185 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55780098a0 con 0x7f55900ff400 2024-04-05T22:04:28.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f55951dc700 1 -- 172.21.15.121:0/1620593185 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55901a0510 con 0x7f55900ff400 2024-04-05T22:04:28.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f5587fff700 1 -- 172.21.15.121:0/1620593185 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5578010460 con 0x7f55900ff400 2024-04-05T22:04:28.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f5587fff700 1 -- 172.21.15.121:0/1620593185 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5578009ea0 con 0x7f55900ff400 2024-04-05T22:04:28.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.470+0000 7f55951dc700 1 -- 172.21.15.121:0/1620593185 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55901a09d0 con 0x7f55900ff400 2024-04-05T22:04:28.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.472+0000 7f55951dc700 1 -- 172.21.15.121:0/1620593185 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5590107800 con 0x7f55900ff400 2024-04-05T22:04:28.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.472+0000 7f5587fff700 1 -- 172.21.15.121:0/1620593185 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f557800f6c0 con 0x7f55900ff400 2024-04-05T22:04:28.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.472+0000 7f5587fff700 1 --2- 172.21.15.121:0/1620593185 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f557c06cce0 0x7f557c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:28.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.472+0000 7f558e59c700 1 --2- 172.21.15.121:0/1620593185 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f557c06cce0 0x7f557c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:28.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.472+0000 7f5587fff700 1 -- 172.21.15.121:0/1620593185 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f557808c470 con 0x7f55900ff400 2024-04-05T22:04:28.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.473+0000 7f558e59c700 1 --2- 172.21.15.121:0/1620593185 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f557c06cce0 0x7f557c06f1a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5580006010 tx=0x7f558001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:28.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.479+0000 7f5587fff700 1 -- 172.21.15.121:0/1620593185 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f55780564e0 con 0x7f55900ff400 2024-04-05T22:04:28.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.680+0000 7f55951dc700 1 -- 172.21.15.121:0/1620593185 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f559002cd10 con 0x7f557c06cce0 2024-04-05T22:04:28.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.690+0000 7f5587fff700 1 -- 172.21.15.121:0/1620593185 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f559002cd10 con 0x7f557c06cce0 2024-04-05T22:04:28.690 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (35m) 13m ago 46m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (47m) 13m ago 47m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (45m) 13m ago 45m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (33m) 13m ago 46m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (33m) 13m ago 45m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (35m) 13m ago 46m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (29m) 13m ago 40m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (29m) 13m ago 40m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (29m) 13m ago 40m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (13m) 13m ago 40m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (37m) 13m ago 48m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (36m) 13m ago 45m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (34m) 13m ago 48m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (33m) 13m ago 45m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T22:04:28.691 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (36m) 13m ago 46m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T22:04:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (36m) 13m ago 45m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T22:04:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (33m) 13m ago 44m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T22:04:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (32m) 13m ago 44m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T22:04:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (31m) 13m ago 43m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T22:04:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (31m) 13m ago 43m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T22:04:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (30m) 13m ago 42m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T22:04:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (30m) 13m ago 42m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T22:04:28.692 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (35m) 13m ago 46m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T22:04:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.692+0000 7f55951dc700 1 -- 172.21.15.121:0/1620593185 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f557c06cce0 msgr2=0x7f557c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.692+0000 7f55951dc700 1 --2- 172.21.15.121:0/1620593185 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f557c06cce0 0x7f557c06f1a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5580006010 tx=0x7f558001a040 comp rx=0 tx=0).stop 2024-04-05T22:04:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.692+0000 7f55951dc700 1 -- 172.21.15.121:0/1620593185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55900ff400 msgr2=0x7f55901974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.692+0000 7f55951dc700 1 --2- 172.21.15.121:0/1620593185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55900ff400 0x7f55901974a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f557800d8d0 tx=0x7f557800dbe0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.693+0000 7f55951dc700 1 -- 172.21.15.121:0/1620593185 shutdown_connections 2024-04-05T22:04:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.693+0000 7f55951dc700 1 --2- 172.21.15.121:0/1620593185 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f557c06cce0 0x7f557c06f1a0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.693+0000 7f55951dc700 1 --2- 172.21.15.121:0/1620593185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55900ff400 0x7f55901974a0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.693+0000 7f55951dc700 1 --2- 172.21.15.121:0/1620593185 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f55900ffdb0 0x7f55901979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.693+0000 7f55951dc700 1 -- 172.21.15.121:0/1620593185 >> 172.21.15.121:0/1620593185 conn(0x7f5590075e30 msgr2=0x7f5590103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.693+0000 7f55951dc700 1 -- 172.21.15.121:0/1620593185 shutdown_connections 2024-04-05T22:04:28.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.693+0000 7f55951dc700 1 -- 172.21.15.121:0/1620593185 wait complete. 2024-04-05T22:04:28.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.794+0000 7f3811029700 1 -- 172.21.15.121:0/421865116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f380c105a50 msgr2=0x7f380c105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:28.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.794+0000 7f3811029700 1 --2- 172.21.15.121:0/421865116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f380c105a50 0x7f380c105e30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f3800009910 tx=0x7f3800009c20 comp rx=0 tx=0).stop 2024-04-05T22:04:28.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.794+0000 7f3811029700 1 -- 172.21.15.121:0/421865116 shutdown_connections 2024-04-05T22:04:28.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.794+0000 7f3811029700 1 --2- 172.21.15.121:0/421865116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f380c0ffa50 0x7f380c0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.794+0000 7f3811029700 1 --2- 172.21.15.121:0/421865116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f380c105a50 0x7f380c105e30 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.794+0000 7f3811029700 1 -- 172.21.15.121:0/421865116 >> 172.21.15.121:0/421865116 conn(0x7f380c0fb540 msgr2=0x7f380c0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:28.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.794+0000 7f3811029700 1 -- 172.21.15.121:0/421865116 shutdown_connections 2024-04-05T22:04:28.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.794+0000 7f3811029700 1 -- 172.21.15.121:0/421865116 wait complete. 2024-04-05T22:04:28.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.795+0000 7f3811029700 1 Processor -- start 2024-04-05T22:04:28.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.795+0000 7f3811029700 1 -- start start 2024-04-05T22:04:28.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.795+0000 7f3811029700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f380c0ffa50 0x7f380c196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:28.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.795+0000 7f3811029700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f380c105a50 0x7f380c196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:28.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.795+0000 7f3811029700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c197260 con 0x7f380c0ffa50 2024-04-05T22:04:28.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.795+0000 7f3811029700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c19aff0 con 0x7f380c105a50 2024-04-05T22:04:28.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.795+0000 7f380ad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f380c0ffa50 0x7f380c196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:28.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.795+0000 7f380ad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f380c0ffa50 0x7f380c196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36948/0 (socket says 172.21.15.121:36948) 2024-04-05T22:04:28.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.795+0000 7f380ad9d700 1 -- 172.21.15.121:0/686229599 learned_addr learned my addr 172.21.15.121:0/686229599 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:04:28.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.795+0000 7f380a59c700 1 --2- 172.21.15.121:0/686229599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f380c105a50 0x7f380c196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:28.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.796+0000 7f380ad9d700 1 -- 172.21.15.121:0/686229599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f380c105a50 msgr2=0x7f380c196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:28.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.796+0000 7f380ad9d700 1 --2- 172.21.15.121:0/686229599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f380c105a50 0x7f380c196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:28.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.796+0000 7f380ad9d700 1 -- 172.21.15.121:0/686229599 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38000095f0 con 0x7f380c0ffa50 2024-04-05T22:04:28.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.796+0000 7f380ad9d700 1 --2- 172.21.15.121:0/686229599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f380c0ffa50 0x7f380c196640 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3800006010 tx=0x7f380000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:28.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.796+0000 7f37f3fff700 1 -- 172.21.15.121:0/686229599 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f380001d070 con 0x7f380c0ffa50 2024-04-05T22:04:28.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.796+0000 7f3811029700 1 -- 172.21.15.121:0/686229599 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380c19b270 con 0x7f380c0ffa50 2024-04-05T22:04:28.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.796+0000 7f37f3fff700 1 -- 172.21.15.121:0/686229599 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3800005940 con 0x7f380c0ffa50 2024-04-05T22:04:28.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.796+0000 7f37f3fff700 1 -- 172.21.15.121:0/686229599 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3800021410 con 0x7f380c0ffa50 2024-04-05T22:04:28.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.796+0000 7f3811029700 1 -- 172.21.15.121:0/686229599 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f380c19b760 con 0x7f380c0ffa50 2024-04-05T22:04:28.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.797+0000 7f3811029700 1 -- 172.21.15.121:0/686229599 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380c109060 con 0x7f380c0ffa50 2024-04-05T22:04:28.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.797+0000 7f37f3fff700 1 -- 172.21.15.121:0/686229599 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3800003a60 con 0x7f380c0ffa50 2024-04-05T22:04:28.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.798+0000 7f37f3fff700 1 --2- 172.21.15.121:0/686229599 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37f406cce0 0x7f37f406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:28.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.798+0000 7f37f3fff700 1 -- 172.21.15.121:0/686229599 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f380008d810 con 0x7f380c0ffa50 2024-04-05T22:04:28.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.798+0000 7f380a59c700 1 --2- 172.21.15.121:0/686229599 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37f406cce0 0x7f37f406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:28.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.798+0000 7f380a59c700 1 --2- 172.21.15.121:0/686229599 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37f406cce0 0x7f37f406f1a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f380c197c60 tx=0x7f37fc00a300 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:28.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:28.802+0000 7f37f3fff700 1 -- 172.21.15.121:0/686229599 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3800057880 con 0x7f380c0ffa50 2024-04-05T22:04:29.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.077+0000 7f3811029700 1 -- 172.21.15.121:0/686229599 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f380c04ea90 con 0x7f380c0ffa50 2024-04-05T22:04:29.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.078+0000 7f37f3fff700 1 -- 172.21.15.121:0/686229599 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f380005aea0 con 0x7f380c0ffa50 2024-04-05T22:04:29.078 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:04:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T22:04:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:04:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:04:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T22:04:29.078 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:04:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:04:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T22:04:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T22:04:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:04:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T22:04:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T22:04:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:04:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T22:04:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T22:04:29.079 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T22:04:29.079 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:04:29.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.080+0000 7f3811029700 1 -- 172.21.15.121:0/686229599 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37f406cce0 msgr2=0x7f37f406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:29.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.080+0000 7f3811029700 1 --2- 172.21.15.121:0/686229599 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37f406cce0 0x7f37f406f1a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f380c197c60 tx=0x7f37fc00a300 comp rx=0 tx=0).stop 2024-04-05T22:04:29.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.080+0000 7f3811029700 1 -- 172.21.15.121:0/686229599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f380c0ffa50 msgr2=0x7f380c196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:29.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.080+0000 7f3811029700 1 --2- 172.21.15.121:0/686229599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f380c0ffa50 0x7f380c196640 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3800006010 tx=0x7f380000bd70 comp rx=0 tx=0).stop 2024-04-05T22:04:29.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.080+0000 7f3811029700 1 -- 172.21.15.121:0/686229599 shutdown_connections 2024-04-05T22:04:29.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.080+0000 7f3811029700 1 --2- 172.21.15.121:0/686229599 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f37f406cce0 0x7f37f406f1a0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.080+0000 7f3811029700 1 --2- 172.21.15.121:0/686229599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f380c0ffa50 0x7f380c196640 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.080+0000 7f3811029700 1 --2- 172.21.15.121:0/686229599 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f380c105a50 0x7f380c196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.080+0000 7f3811029700 1 -- 172.21.15.121:0/686229599 >> 172.21.15.121:0/686229599 conn(0x7f380c0fb540 msgr2=0x7f380c068f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:29.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.080+0000 7f3811029700 1 -- 172.21.15.121:0/686229599 shutdown_connections 2024-04-05T22:04:29.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.080+0000 7f3811029700 1 -- 172.21.15.121:0/686229599 wait complete. 2024-04-05T22:04:29.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.182+0000 7f397902c700 1 -- 172.21.15.121:0/3048412973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3974101770 msgr2=0x7f39741057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:29.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.182+0000 7f397902c700 1 --2- 172.21.15.121:0/3048412973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3974101770 0x7f39741057c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f3964009910 tx=0x7f3964009c20 comp rx=0 tx=0).stop 2024-04-05T22:04:29.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.182+0000 7f397902c700 1 -- 172.21.15.121:0/3048412973 shutdown_connections 2024-04-05T22:04:29.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.182+0000 7f397902c700 1 --2- 172.21.15.121:0/3048412973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3974101770 0x7f39741057c0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.182+0000 7f397902c700 1 --2- 172.21.15.121:0/3048412973 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3974100dc0 0x7f39741011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.182+0000 7f397902c700 1 -- 172.21.15.121:0/3048412973 >> 172.21.15.121:0/3048412973 conn(0x7f39740fc670 msgr2=0x7f39740fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:29.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.182+0000 7f397902c700 1 -- 172.21.15.121:0/3048412973 shutdown_connections 2024-04-05T22:04:29.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.182+0000 7f397902c700 1 -- 172.21.15.121:0/3048412973 wait complete. 2024-04-05T22:04:29.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.182+0000 7f397902c700 1 Processor -- start 2024-04-05T22:04:29.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.183+0000 7f397902c700 1 -- start start 2024-04-05T22:04:29.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.183+0000 7f397902c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3974100dc0 0x7f3974073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:29.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.183+0000 7f397902c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3974101770 0x7f39740735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:29.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.183+0000 7f397902c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3974073b10 con 0x7f3974100dc0 2024-04-05T22:04:29.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.183+0000 7f397902c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3974073c50 con 0x7f3974101770 2024-04-05T22:04:29.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.183+0000 7f3972d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3974100dc0 0x7f3974073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:29.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.183+0000 7f3972d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3974100dc0 0x7f3974073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36970/0 (socket says 172.21.15.121:36970) 2024-04-05T22:04:29.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.183+0000 7f3972d9d700 1 -- 172.21.15.121:0/1293518631 learned_addr learned my addr 172.21.15.121:0/1293518631 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:04:29.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.183+0000 7f397259c700 1 --2- 172.21.15.121:0/1293518631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3974101770 0x7f39740735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:29.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.183+0000 7f3972d9d700 1 -- 172.21.15.121:0/1293518631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3974101770 msgr2=0x7f39740735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:29.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.183+0000 7f3972d9d700 1 --2- 172.21.15.121:0/1293518631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3974101770 0x7f39740735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.183+0000 7f3972d9d700 1 -- 172.21.15.121:0/1293518631 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39640095f0 con 0x7f3974100dc0 2024-04-05T22:04:29.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.183+0000 7f3972d9d700 1 --2- 172.21.15.121:0/1293518631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3974100dc0 0x7f3974073090 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f396800d8d0 tx=0x7f396800dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:29.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.184+0000 7f395bfff700 1 -- 172.21.15.121:0/1293518631 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39680098a0 con 0x7f3974100dc0 2024-04-05T22:04:29.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.184+0000 7f397902c700 1 -- 172.21.15.121:0/1293518631 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39741a2820 con 0x7f3974100dc0 2024-04-05T22:04:29.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.184+0000 7f395bfff700 1 -- 172.21.15.121:0/1293518631 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3968010460 con 0x7f3974100dc0 2024-04-05T22:04:29.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.184+0000 7f395bfff700 1 -- 172.21.15.121:0/1293518631 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3968009ea0 con 0x7f3974100dc0 2024-04-05T22:04:29.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.184+0000 7f397902c700 1 -- 172.21.15.121:0/1293518631 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39741a2c30 con 0x7f3974100dc0 2024-04-05T22:04:29.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.185+0000 7f395bfff700 1 -- 172.21.15.121:0/1293518631 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f39680105d0 con 0x7f3974100dc0 2024-04-05T22:04:29.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.185+0000 7f397902c700 1 -- 172.21.15.121:0/1293518631 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39741091c0 con 0x7f3974100dc0 2024-04-05T22:04:29.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.186+0000 7f395bfff700 1 --2- 172.21.15.121:0/1293518631 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f395c06cce0 0x7f395c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:29.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.186+0000 7f395bfff700 1 -- 172.21.15.121:0/1293518631 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f396808c480 con 0x7f3974100dc0 2024-04-05T22:04:29.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.186+0000 7f397259c700 1 --2- 172.21.15.121:0/1293518631 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f395c06cce0 0x7f395c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:29.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.186+0000 7f397259c700 1 --2- 172.21.15.121:0/1293518631 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f395c06cce0 0x7f395c06f1a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3974074880 tx=0x7f396401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:29.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.190+0000 7f395bfff700 1 -- 172.21.15.121:0/1293518631 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f39680564f0 con 0x7f3974100dc0 2024-04-05T22:04:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:29 smithi188 ceph-mon[107793]: pgmap v1148: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:29.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:29 smithi188 ceph-mon[107793]: from='client.35886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:04:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.425+0000 7f397902c700 1 -- 172.21.15.121:0/1293518631 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f397404ea90 con 0x7f3974100dc0 2024-04-05T22:04:29.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.425+0000 7f395bfff700 1 -- 172.21.15.121:0/1293518631 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f3968010880 con 0x7f3974100dc0 2024-04-05T22:04:29.425 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T22:04:29.425 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T22:04:29.425 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:04:29.425 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T22:04:29.426 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:04:29.426 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T22:04:29.426 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T22:04:29.426 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T22:04:29.426 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T22:04:29.426 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T22:04:29.426 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T22:04:29.426 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T22:04:29.426 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T22:04:29.426 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T22:04:29.426 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T22:04:29.426 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T22:04:29.426 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T22:04:29.427 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T22:04:29.428 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T22:04:29.428 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T22:04:29.428 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:04:29.428 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:04:29.428 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T22:04:29.428 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:04:29.428 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:04:29.428 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:04:29.428 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:04:29.428 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:04:29.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.427+0000 7f397902c700 1 -- 172.21.15.121:0/1293518631 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f395c06cce0 msgr2=0x7f395c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:29.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.427+0000 7f397902c700 1 --2- 172.21.15.121:0/1293518631 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f395c06cce0 0x7f395c06f1a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3974074880 tx=0x7f396401a040 comp rx=0 tx=0).stop 2024-04-05T22:04:29.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.427+0000 7f397902c700 1 -- 172.21.15.121:0/1293518631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3974100dc0 msgr2=0x7f3974073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:29.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.427+0000 7f397902c700 1 --2- 172.21.15.121:0/1293518631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3974100dc0 0x7f3974073090 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f396800d8d0 tx=0x7f396800dbe0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.427+0000 7f397902c700 1 -- 172.21.15.121:0/1293518631 shutdown_connections 2024-04-05T22:04:29.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.427+0000 7f397902c700 1 --2- 172.21.15.121:0/1293518631 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f395c06cce0 0x7f395c06f1a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.427+0000 7f397902c700 1 --2- 172.21.15.121:0/1293518631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3974100dc0 0x7f3974073090 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.427+0000 7f397902c700 1 --2- 172.21.15.121:0/1293518631 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3974101770 0x7f39740735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.427+0000 7f397902c700 1 -- 172.21.15.121:0/1293518631 >> 172.21.15.121:0/1293518631 conn(0x7f39740fc670 msgr2=0x7f39740fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:29.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.427+0000 7f397902c700 1 -- 172.21.15.121:0/1293518631 shutdown_connections 2024-04-05T22:04:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.427+0000 7f397902c700 1 -- 172.21.15.121:0/1293518631 wait complete. 2024-04-05T22:04:29.430 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T22:04:29.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:29 smithi121 ceph-mon[145859]: pgmap v1148: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:29.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:29 smithi121 ceph-mon[145859]: from='client.35886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:04:29.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.531+0000 7f212048c700 1 -- 172.21.15.121:0/2260927672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2118100de0 msgr2=0x7f21181011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:29.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.531+0000 7f212048c700 1 --2- 172.21.15.121:0/2260927672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2118100de0 0x7f21181011c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2108009910 tx=0x7f2108009c20 comp rx=0 tx=0).stop 2024-04-05T22:04:29.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.531+0000 7f212048c700 1 -- 172.21.15.121:0/2260927672 shutdown_connections 2024-04-05T22:04:29.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.531+0000 7f212048c700 1 --2- 172.21.15.121:0/2260927672 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2118101790 0x7f21181057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.531+0000 7f212048c700 1 --2- 172.21.15.121:0/2260927672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2118100de0 0x7f21181011c0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.531+0000 7f212048c700 1 -- 172.21.15.121:0/2260927672 >> 172.21.15.121:0/2260927672 conn(0x7f21180fc670 msgr2=0x7f21180fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:29.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.531+0000 7f212048c700 1 -- 172.21.15.121:0/2260927672 shutdown_connections 2024-04-05T22:04:29.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.531+0000 7f212048c700 1 -- 172.21.15.121:0/2260927672 wait complete. 2024-04-05T22:04:29.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f212048c700 1 Processor -- start 2024-04-05T22:04:29.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f212048c700 1 -- start start 2024-04-05T22:04:29.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f212048c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2118100de0 0x7f2118196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:29.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f212048c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2118101790 0x7f2118196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:29.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f212048c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2118197290 con 0x7f2118100de0 2024-04-05T22:04:29.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f212048c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211819b020 con 0x7f2118101790 2024-04-05T22:04:29.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f211e228700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2118100de0 0x7f2118196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f211e228700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2118100de0 0x7f2118196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36986/0 (socket says 172.21.15.121:36986) 2024-04-05T22:04:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f211e228700 1 -- 172.21.15.121:0/3184185447 learned_addr learned my addr 172.21.15.121:0/3184185447 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:04:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f211da27700 1 --2- 172.21.15.121:0/3184185447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2118101790 0x7f2118196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f211e228700 1 -- 172.21.15.121:0/3184185447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2118101790 msgr2=0x7f2118196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f211e228700 1 --2- 172.21.15.121:0/3184185447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2118101790 0x7f2118196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f211e228700 1 -- 172.21.15.121:0/3184185447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21080095f0 con 0x7f2118100de0 2024-04-05T22:04:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.532+0000 7f211da27700 1 --2- 172.21.15.121:0/3184185447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2118101790 0x7f2118196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:04:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.533+0000 7f211e228700 1 --2- 172.21.15.121:0/3184185447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2118100de0 0x7f2118196670 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2108006010 tx=0x7f210800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.533+0000 7f210f7fe700 1 -- 172.21.15.121:0/3184185447 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f210801d070 con 0x7f2118100de0 2024-04-05T22:04:29.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.533+0000 7f210f7fe700 1 -- 172.21.15.121:0/3184185447 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f210800f460 con 0x7f2118100de0 2024-04-05T22:04:29.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.533+0000 7f210f7fe700 1 -- 172.21.15.121:0/3184185447 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2108021640 con 0x7f2118100de0 2024-04-05T22:04:29.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.533+0000 7f212048c700 1 -- 172.21.15.121:0/3184185447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211819b2a0 con 0x7f2118100de0 2024-04-05T22:04:29.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.533+0000 7f212048c700 1 -- 172.21.15.121:0/3184185447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f211819b790 con 0x7f2118100de0 2024-04-05T22:04:29.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.534+0000 7f212048c700 1 -- 172.21.15.121:0/3184185447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2118109180 con 0x7f2118100de0 2024-04-05T22:04:29.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.534+0000 7f210f7fe700 1 -- 172.21.15.121:0/3184185447 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f210800fa60 con 0x7f2118100de0 2024-04-05T22:04:29.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.534+0000 7f210f7fe700 1 --2- 172.21.15.121:0/3184185447 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f210406cce0 0x7f210406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:29.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.535+0000 7f210f7fe700 1 -- 172.21.15.121:0/3184185447 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f210808d840 con 0x7f2118100de0 2024-04-05T22:04:29.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.535+0000 7f211da27700 1 --2- 172.21.15.121:0/3184185447 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f210406cce0 0x7f210406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:29.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.535+0000 7f211da27700 1 --2- 172.21.15.121:0/3184185447 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f210406cce0 0x7f210406f1a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2118197c90 tx=0x7f2114005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:29.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.539+0000 7f210f7fe700 1 -- 172.21.15.121:0/3184185447 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f21080578d0 con 0x7f2118100de0 2024-04-05T22:04:29.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.748+0000 7f212048c700 1 -- 172.21.15.121:0/3184185447 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f211802cd30 con 0x7f210406cce0 2024-04-05T22:04:29.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.749+0000 7f210f7fe700 1 -- 172.21.15.121:0/3184185447 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f211802cd30 con 0x7f210406cce0 2024-04-05T22:04:29.749 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:04:29.749 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T22:04:29.749 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T22:04:29.749 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T22:04:29.750 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T22:04:29.750 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T22:04:29.750 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T22:04:29.750 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T22:04:29.750 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T22:04:29.750 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T22:04:29.750 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T22:04:29.750 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T22:04:29.750 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T22:04:29.750 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T22:04:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:04:29.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.751+0000 7f212048c700 1 -- 172.21.15.121:0/3184185447 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f210406cce0 msgr2=0x7f210406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:29.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.751+0000 7f212048c700 1 --2- 172.21.15.121:0/3184185447 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f210406cce0 0x7f210406f1a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2118197c90 tx=0x7f2114005f50 comp rx=0 tx=0).stop 2024-04-05T22:04:29.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.751+0000 7f212048c700 1 -- 172.21.15.121:0/3184185447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2118100de0 msgr2=0x7f2118196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:29.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.751+0000 7f212048c700 1 --2- 172.21.15.121:0/3184185447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2118100de0 0x7f2118196670 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2108006010 tx=0x7f210800be20 comp rx=0 tx=0).stop 2024-04-05T22:04:29.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.751+0000 7f212048c700 1 -- 172.21.15.121:0/3184185447 shutdown_connections 2024-04-05T22:04:29.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.751+0000 7f212048c700 1 --2- 172.21.15.121:0/3184185447 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f210406cce0 0x7f210406f1a0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.751+0000 7f212048c700 1 --2- 172.21.15.121:0/3184185447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2118100de0 0x7f2118196670 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.751+0000 7f212048c700 1 --2- 172.21.15.121:0/3184185447 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2118101790 0x7f2118196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.751+0000 7f212048c700 1 -- 172.21.15.121:0/3184185447 >> 172.21.15.121:0/3184185447 conn(0x7f21180fc670 msgr2=0x7f21180fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:29.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.751+0000 7f212048c700 1 -- 172.21.15.121:0/3184185447 shutdown_connections 2024-04-05T22:04:29.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.751+0000 7f212048c700 1 -- 172.21.15.121:0/3184185447 wait complete. 2024-04-05T22:04:29.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.854+0000 7fc2f585f700 1 -- 172.21.15.121:0/2610398272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f0100dc0 msgr2=0x7fc2f01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.854+0000 7fc2f585f700 1 --2- 172.21.15.121:0/2610398272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f0100dc0 0x7fc2f01011a0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009910 tx=0x7fc2e4009c20 comp rx=0 tx=0).stop 2024-04-05T22:04:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.854+0000 7fc2f585f700 1 -- 172.21.15.121:0/2610398272 shutdown_connections 2024-04-05T22:04:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.854+0000 7fc2f585f700 1 --2- 172.21.15.121:0/2610398272 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2f0101770 0x7fc2f01057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.854+0000 7fc2f585f700 1 --2- 172.21.15.121:0/2610398272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f0100dc0 0x7fc2f01011a0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.854+0000 7fc2f585f700 1 -- 172.21.15.121:0/2610398272 >> 172.21.15.121:0/2610398272 conn(0x7fc2f00fc650 msgr2=0x7fc2f00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.854+0000 7fc2f585f700 1 -- 172.21.15.121:0/2610398272 shutdown_connections 2024-04-05T22:04:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.854+0000 7fc2f585f700 1 -- 172.21.15.121:0/2610398272 wait complete. 2024-04-05T22:04:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.854+0000 7fc2f585f700 1 Processor -- start 2024-04-05T22:04:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.854+0000 7fc2f585f700 1 -- start start 2024-04-05T22:04:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2f585f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2f0100dc0 0x7fc2f0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2f585f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f0101770 0x7fc2f00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2f585f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0073b10 con 0x7fc2f0101770 2024-04-05T22:04:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2f585f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0073c50 con 0x7fc2f0100dc0 2024-04-05T22:04:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2ee7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f0101770 0x7fc2f00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2ee7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f0101770 0x7fc2f00735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37008/0 (socket says 172.21.15.121:37008) 2024-04-05T22:04:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2ee7fc700 1 -- 172.21.15.121:0/1471067758 learned_addr learned my addr 172.21.15.121:0/1471067758 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:04:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2eeffd700 1 --2- 172.21.15.121:0/1471067758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2f0100dc0 0x7fc2f0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2ee7fc700 1 -- 172.21.15.121:0/1471067758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2f0100dc0 msgr2=0x7fc2f0073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2ee7fc700 1 --2- 172.21.15.121:0/1471067758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2f0100dc0 0x7fc2f0073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2ee7fc700 1 -- 172.21.15.121:0/1471067758 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e40095f0 con 0x7fc2f0101770 2024-04-05T22:04:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2ee7fc700 1 --2- 172.21.15.121:0/1471067758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f0101770 0x7fc2f00735d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000c8f0 tx=0x7fc2e000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:29.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2f485d700 1 -- 172.21.15.121:0/1471067758 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e00078a0 con 0x7fc2f0101770 2024-04-05T22:04:29.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2f485d700 1 -- 172.21.15.121:0/1471067758 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2e000f450 con 0x7fc2f0101770 2024-04-05T22:04:29.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2f585f700 1 -- 172.21.15.121:0/1471067758 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f01a2810 con 0x7fc2f0101770 2024-04-05T22:04:29.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2f485d700 1 -- 172.21.15.121:0/1471067758 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2e0007ea0 con 0x7fc2f0101770 2024-04-05T22:04:29.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.855+0000 7fc2f585f700 1 -- 172.21.15.121:0/1471067758 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f01a2c20 con 0x7fc2f0101770 2024-04-05T22:04:29.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.856+0000 7fc2f585f700 1 -- 172.21.15.121:0/1471067758 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f0109160 con 0x7fc2f0101770 2024-04-05T22:04:29.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.857+0000 7fc2f485d700 1 -- 172.21.15.121:0/1471067758 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc2e000f5c0 con 0x7fc2f0101770 2024-04-05T22:04:29.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.857+0000 7fc2f485d700 1 --2- 172.21.15.121:0/1471067758 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc2d806cce0 0x7fc2d806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:04:29.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.857+0000 7fc2f485d700 1 -- 172.21.15.121:0/1471067758 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc2e008c4d0 con 0x7fc2f0101770 2024-04-05T22:04:29.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.857+0000 7fc2eeffd700 1 --2- 172.21.15.121:0/1471067758 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc2d806cce0 0x7fc2d806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:04:29.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.858+0000 7fc2eeffd700 1 --2- 172.21.15.121:0/1471067758 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc2d806cce0 0x7fc2d806f1a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4006010 tx=0x7fc2e401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:04:29.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:29.861+0000 7fc2f485d700 1 -- 172.21.15.121:0/1471067758 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc2e0056540 con 0x7fc2f0101770 2024-04-05T22:04:30.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:30.134+0000 7fc2f585f700 1 -- 172.21.15.121:0/1471067758 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc2f004ea90 con 0x7fc2f0101770 2024-04-05T22:04:30.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:30.134+0000 7fc2f485d700 1 -- 172.21.15.121:0/1471067758 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fc2e0059b60 con 0x7fc2f0101770 2024-04-05T22:04:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:04:30.135 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:04:30.136 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:04:30.137 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:04:30.137 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:04:30.137 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:04:30.137 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:04:30.137 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T22:04:30.137 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:04:30.137 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T22:04:30.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:30.136+0000 7fc2f585f700 1 -- 172.21.15.121:0/1471067758 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc2d806cce0 msgr2=0x7fc2d806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:30.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:30.136+0000 7fc2f585f700 1 --2- 172.21.15.121:0/1471067758 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc2d806cce0 0x7fc2d806f1a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4006010 tx=0x7fc2e401a040 comp rx=0 tx=0).stop 2024-04-05T22:04:30.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:30.136+0000 7fc2f585f700 1 -- 172.21.15.121:0/1471067758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f0101770 msgr2=0x7fc2f00735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:04:30.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:30.136+0000 7fc2f585f700 1 --2- 172.21.15.121:0/1471067758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f0101770 0x7fc2f00735d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000c8f0 tx=0x7fc2e000cc00 comp rx=0 tx=0).stop 2024-04-05T22:04:30.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:30.136+0000 7fc2f585f700 1 -- 172.21.15.121:0/1471067758 shutdown_connections 2024-04-05T22:04:30.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:30.136+0000 7fc2f585f700 1 --2- 172.21.15.121:0/1471067758 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc2d806cce0 0x7fc2d806f1a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:30.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:30.136+0000 7fc2f585f700 1 --2- 172.21.15.121:0/1471067758 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc2f0100dc0 0x7fc2f0073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:30.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:30.136+0000 7fc2f585f700 1 --2- 172.21.15.121:0/1471067758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2f0101770 0x7fc2f00735d0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:04:30.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:30.136+0000 7fc2f585f700 1 -- 172.21.15.121:0/1471067758 >> 172.21.15.121:0/1471067758 conn(0x7fc2f00fc650 msgr2=0x7fc2f00fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:04:30.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:30.136+0000 7fc2f585f700 1 -- 172.21.15.121:0/1471067758 shutdown_connections 2024-04-05T22:04:30.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:04:30.137+0000 7fc2f585f700 1 -- 172.21.15.121:0/1471067758 wait complete. 2024-04-05T22:04:30.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:30 smithi121 ceph-mon[145859]: from='client.35890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:04:30.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:30 smithi121 ceph-mon[145859]: from='client.35894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:04:30.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:30 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/686229599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:04:30.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:30 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1293518631' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:04:30.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:30 smithi188 ceph-mon[107793]: from='client.35890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:04:30.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:30 smithi188 ceph-mon[107793]: from='client.35894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:04:30.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:30 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/686229599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:04:30.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:30 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1293518631' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:04:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:31 smithi188 ceph-mon[107793]: from='client.35906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:04:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:31 smithi188 ceph-mon[107793]: pgmap v1149: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:31 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1471067758' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:04:31.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:04:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:31 smithi121 ceph-mon[145859]: from='client.35906 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:04:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:31 smithi121 ceph-mon[145859]: pgmap v1149: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:31 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1471067758' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:04:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:04:33.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:33 smithi121 ceph-mon[145859]: pgmap v1150: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:33.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:33 smithi188 ceph-mon[107793]: pgmap v1150: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:35.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:35 smithi121 ceph-mon[145859]: pgmap v1151: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:35 smithi188 ceph-mon[107793]: pgmap v1151: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:37.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:37 smithi121 ceph-mon[145859]: pgmap v1152: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:37.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:37 smithi188 ceph-mon[107793]: pgmap v1152: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:39.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:39 smithi121 ceph-mon[145859]: pgmap v1153: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:39.528 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:39 smithi188 ceph-mon[107793]: pgmap v1153: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:41.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:41 smithi121 ceph-mon[145859]: pgmap v1154: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:41.528 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:41 smithi188 ceph-mon[107793]: pgmap v1154: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:42.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:42 smithi121 ceph-mon[145859]: pgmap v1155: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:42.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:42 smithi188 ceph-mon[107793]: pgmap v1155: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:45.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:44 smithi121 ceph-mon[145859]: pgmap v1156: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:44 smithi188 ceph-mon[107793]: pgmap v1156: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:04:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:04:47.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:46 smithi121 ceph-mon[145859]: pgmap v1157: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:46 smithi188 ceph-mon[107793]: pgmap v1157: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:49.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:48 smithi121 ceph-mon[145859]: pgmap v1158: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:48 smithi188 ceph-mon[107793]: pgmap v1158: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:51.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:50 smithi121 ceph-mon[145859]: pgmap v1159: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:50 smithi188 ceph-mon[107793]: pgmap v1159: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:53.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:52 smithi121 ceph-mon[145859]: pgmap v1160: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:52 smithi188 ceph-mon[107793]: pgmap v1160: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:55.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:54 smithi121 ceph-mon[145859]: pgmap v1161: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:55.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:54 smithi188 ceph-mon[107793]: pgmap v1161: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:57.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:56 smithi121 ceph-mon[145859]: pgmap v1162: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:57.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:56 smithi188 ceph-mon[107793]: pgmap v1162: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:59.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:04:58 smithi188 ceph-mon[107793]: pgmap v1163: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:04:59.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:04:58 smithi121 ceph-mon[145859]: pgmap v1163: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.252+0000 7f2276577700 1 -- 172.21.15.121:0/3075870543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2270105a40 msgr2=0x7f2270105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.252+0000 7f2276577700 1 --2- 172.21.15.121:0/3075870543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2270105a40 0x7f2270105e20 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2264009910 tx=0x7f2264009c20 comp rx=0 tx=0).stop 2024-04-05T22:05:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.252+0000 7f2276577700 1 -- 172.21.15.121:0/3075870543 shutdown_connections 2024-04-05T22:05:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.252+0000 7f2276577700 1 --2- 172.21.15.121:0/3075870543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f22700ffbd0 0x7f2270100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.252+0000 7f2276577700 1 --2- 172.21.15.121:0/3075870543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2270105a40 0x7f2270105e20 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.252+0000 7f2276577700 1 -- 172.21.15.121:0/3075870543 >> 172.21.15.121:0/3075870543 conn(0x7f22700755a0 msgr2=0x7f22700759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:00.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.252+0000 7f2276577700 1 -- 172.21.15.121:0/3075870543 shutdown_connections 2024-04-05T22:05:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.252+0000 7f2276577700 1 -- 172.21.15.121:0/3075870543 wait complete. 2024-04-05T22:05:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.252+0000 7f2276577700 1 Processor -- start 2024-04-05T22:05:00.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.252+0000 7f2276577700 1 -- start start 2024-04-05T22:05:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f2276577700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22700ffbd0 0x7f2270196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f2276577700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2270105a40 0x7f2270196b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:00.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f2276577700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2270197260 con 0x7f22700ffbd0 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f2276577700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227019aff0 con 0x7f2270105a40 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f226ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22700ffbd0 0x7f2270196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f226ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22700ffbd0 0x7f2270196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43740/0 (socket says 172.21.15.121:43740) 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f226ffff700 1 -- 172.21.15.121:0/2448335107 learned_addr learned my addr 172.21.15.121:0/2448335107 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f226f7fe700 1 --2- 172.21.15.121:0/2448335107 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2270105a40 0x7f2270196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f226ffff700 1 -- 172.21.15.121:0/2448335107 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2270105a40 msgr2=0x7f2270196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f226ffff700 1 --2- 172.21.15.121:0/2448335107 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2270105a40 0x7f2270196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f226ffff700 1 -- 172.21.15.121:0/2448335107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22640095f0 con 0x7f22700ffbd0 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f226f7fe700 1 --2- 172.21.15.121:0/2448335107 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2270105a40 0x7f2270196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f226ffff700 1 --2- 172.21.15.121:0/2448335107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22700ffbd0 0x7f2270196640 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f22640092e0 tx=0x7f226400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f226d7fa700 1 -- 172.21.15.121:0/2448335107 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f226401d070 con 0x7f22700ffbd0 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f226d7fa700 1 -- 172.21.15.121:0/2448335107 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2264005e00 con 0x7f22700ffbd0 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f2276577700 1 -- 172.21.15.121:0/2448335107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227019b270 con 0x7f22700ffbd0 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.253+0000 7f226d7fa700 1 -- 172.21.15.121:0/2448335107 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f226400fe60 con 0x7f22700ffbd0 2024-04-05T22:05:00.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.254+0000 7f2276577700 1 -- 172.21.15.121:0/2448335107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f227019b760 con 0x7f22700ffbd0 2024-04-05T22:05:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.254+0000 7f2276577700 1 -- 172.21.15.121:0/2448335107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2270109050 con 0x7f22700ffbd0 2024-04-05T22:05:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.255+0000 7f226d7fa700 1 -- 172.21.15.121:0/2448335107 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f226400f6d0 con 0x7f22700ffbd0 2024-04-05T22:05:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.255+0000 7f226d7fa700 1 --2- 172.21.15.121:0/2448335107 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f225806cce0 0x7f225806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.255+0000 7f226d7fa700 1 -- 172.21.15.121:0/2448335107 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f226408d8b0 con 0x7f22700ffbd0 2024-04-05T22:05:00.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.256+0000 7f226f7fe700 1 --2- 172.21.15.121:0/2448335107 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f225806cce0 0x7f225806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:00.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.256+0000 7f226f7fe700 1 --2- 172.21.15.121:0/2448335107 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f225806cce0 0x7f225806f1a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2260005950 tx=0x7f22600058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.260+0000 7f226d7fa700 1 -- 172.21.15.121:0/2448335107 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2264057920 con 0x7f22700ffbd0 2024-04-05T22:05:00.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.469+0000 7f2276577700 1 -- 172.21.15.121:0/2448335107 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f227019bab0 con 0x7f225806cce0 2024-04-05T22:05:00.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.470+0000 7f226d7fa700 1 -- 172.21.15.121:0/2448335107 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f227019bab0 con 0x7f225806cce0 2024-04-05T22:05:00.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.473+0000 7f2276577700 1 -- 172.21.15.121:0/2448335107 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f225806cce0 msgr2=0x7f225806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:00.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.473+0000 7f2276577700 1 --2- 172.21.15.121:0/2448335107 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f225806cce0 0x7f225806f1a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2260005950 tx=0x7f22600058e0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.473+0000 7f2276577700 1 -- 172.21.15.121:0/2448335107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22700ffbd0 msgr2=0x7f2270196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:00.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.473+0000 7f2276577700 1 --2- 172.21.15.121:0/2448335107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22700ffbd0 0x7f2270196640 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f22640092e0 tx=0x7f226400be20 comp rx=0 tx=0).stop 2024-04-05T22:05:00.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.473+0000 7f2276577700 1 -- 172.21.15.121:0/2448335107 shutdown_connections 2024-04-05T22:05:00.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.473+0000 7f2276577700 1 --2- 172.21.15.121:0/2448335107 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f225806cce0 0x7f225806f1a0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.473+0000 7f2276577700 1 --2- 172.21.15.121:0/2448335107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22700ffbd0 0x7f2270196640 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.473+0000 7f2276577700 1 --2- 172.21.15.121:0/2448335107 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2270105a40 0x7f2270196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.473+0000 7f2276577700 1 -- 172.21.15.121:0/2448335107 >> 172.21.15.121:0/2448335107 conn(0x7f22700755a0 msgr2=0x7f2270068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:00.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.473+0000 7f2276577700 1 -- 172.21.15.121:0/2448335107 shutdown_connections 2024-04-05T22:05:00.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.473+0000 7f2276577700 1 -- 172.21.15.121:0/2448335107 wait complete. 2024-04-05T22:05:00.483 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T22:05:00.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/3550896420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e4100140 msgr2=0x7fd7e41005c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:00.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 --2- 172.21.15.121:0/3550896420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e4100140 0x7fd7e41005c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009910 tx=0x7fd7d4009c20 comp rx=0 tx=0).stop 2024-04-05T22:05:00.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/3550896420 shutdown_connections 2024-04-05T22:05:00.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 --2- 172.21.15.121:0/3550896420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e4100140 0x7fd7e41005c0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 --2- 172.21.15.121:0/3550896420 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7e4104a50 0x7fd7e4104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/3550896420 >> 172.21.15.121:0/3550896420 conn(0x7fd7e4075e30 msgr2=0x7fd7e40fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:00.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/3550896420 shutdown_connections 2024-04-05T22:05:00.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/3550896420 wait complete. 2024-04-05T22:05:00.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 Processor -- start 2024-04-05T22:05:00.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 -- start start 2024-04-05T22:05:00.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e4100140 0x7fd7e41954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:00.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7e4104a50 0x7fd7e41959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:00.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e41960d0 con 0x7fd7e4100140 2024-04-05T22:05:00.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.577+0000 7fd7ebf1a700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e4199e60 con 0x7fd7e4104a50 2024-04-05T22:05:00.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7e9cb6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e4100140 0x7fd7e41954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:00.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7e9cb6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e4100140 0x7fd7e41954b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43752/0 (socket says 172.21.15.121:43752) 2024-04-05T22:05:00.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7e9cb6700 1 -- 172.21.15.121:0/4197530116 learned_addr learned my addr 172.21.15.121:0/4197530116 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:00.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7e94b5700 1 --2- 172.21.15.121:0/4197530116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7e4104a50 0x7fd7e41959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:00.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7e9cb6700 1 -- 172.21.15.121:0/4197530116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7e4104a50 msgr2=0x7fd7e41959f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:00.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7e9cb6700 1 --2- 172.21.15.121:0/4197530116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7e4104a50 0x7fd7e41959f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7e9cb6700 1 -- 172.21.15.121:0/4197530116 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d40095f0 con 0x7fd7e4100140 2024-04-05T22:05:00.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7e94b5700 1 --2- 172.21.15.121:0/4197530116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7e4104a50 0x7fd7e41959f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:05:00.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7e9cb6700 1 --2- 172.21.15.121:0/4197530116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e4100140 0x7fd7e41954b0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000e920 tx=0x7fd7e000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:00.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7daffd700 1 -- 172.21.15.121:0/4197530116 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7e0009790 con 0x7fd7e4100140 2024-04-05T22:05:00.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7daffd700 1 -- 172.21.15.121:0/4197530116 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7e000cba0 con 0x7fd7e4100140 2024-04-05T22:05:00.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/4197530116 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e419a140 con 0x7fd7e4100140 2024-04-05T22:05:00.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7daffd700 1 -- 172.21.15.121:0/4197530116 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7e0018650 con 0x7fd7e4100140 2024-04-05T22:05:00.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.578+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/4197530116 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e419a710 con 0x7fd7e4100140 2024-04-05T22:05:00.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.579+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/4197530116 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e4107f70 con 0x7fd7e4100140 2024-04-05T22:05:00.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.580+0000 7fd7daffd700 1 -- 172.21.15.121:0/4197530116 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fd7e000cd10 con 0x7fd7e4100140 2024-04-05T22:05:00.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.580+0000 7fd7daffd700 1 --2- 172.21.15.121:0/4197530116 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d006cce0 0x7fd7d006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:00.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.580+0000 7fd7daffd700 1 -- 172.21.15.121:0/4197530116 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fd7e0014070 con 0x7fd7e4100140 2024-04-05T22:05:00.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.580+0000 7fd7e94b5700 1 --2- 172.21.15.121:0/4197530116 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d006cce0 0x7fd7d006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:00.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.580+0000 7fd7e94b5700 1 --2- 172.21.15.121:0/4197530116 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d006cce0 0x7fd7d006f1a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40059d0 tx=0x7fd7d4005940 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:00.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.584+0000 7fd7daffd700 1 -- 172.21.15.121:0/4197530116 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fd7e0052ab0 con 0x7fd7e4100140 2024-04-05T22:05:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.791+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/4197530116 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7e419aa40 con 0x7fd7d006cce0 2024-04-05T22:05:00.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.792+0000 7fd7daffd700 1 -- 172.21.15.121:0/4197530116 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd7e419aa40 con 0x7fd7d006cce0 2024-04-05T22:05:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.794+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/4197530116 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d006cce0 msgr2=0x7fd7d006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.794+0000 7fd7ebf1a700 1 --2- 172.21.15.121:0/4197530116 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d006cce0 0x7fd7d006f1a0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40059d0 tx=0x7fd7d4005940 comp rx=0 tx=0).stop 2024-04-05T22:05:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.794+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/4197530116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e4100140 msgr2=0x7fd7e41954b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:00.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.794+0000 7fd7ebf1a700 1 --2- 172.21.15.121:0/4197530116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e4100140 0x7fd7e41954b0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000e920 tx=0x7fd7e000ece0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.794+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/4197530116 shutdown_connections 2024-04-05T22:05:00.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.794+0000 7fd7ebf1a700 1 --2- 172.21.15.121:0/4197530116 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fd7d006cce0 0x7fd7d006f1a0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.794+0000 7fd7ebf1a700 1 --2- 172.21.15.121:0/4197530116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e4100140 0x7fd7e41954b0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.794+0000 7fd7ebf1a700 1 --2- 172.21.15.121:0/4197530116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fd7e4104a50 0x7fd7e41959f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.794+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/4197530116 >> 172.21.15.121:0/4197530116 conn(0x7fd7e4075e30 msgr2=0x7fd7e4103440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:00.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.794+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/4197530116 shutdown_connections 2024-04-05T22:05:00.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.794+0000 7fd7ebf1a700 1 -- 172.21.15.121:0/4197530116 wait complete. 2024-04-05T22:05:00.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.896+0000 7fcf72195700 1 -- 172.21.15.121:0/3319510902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf6c0ffbd0 msgr2=0x7fcf6c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:00.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.896+0000 7fcf72195700 1 --2- 172.21.15.121:0/3319510902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf6c0ffbd0 0x7fcf6c100050 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009910 tx=0x7fcf60009c20 comp rx=0 tx=0).stop 2024-04-05T22:05:00.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.896+0000 7fcf72195700 1 -- 172.21.15.121:0/3319510902 shutdown_connections 2024-04-05T22:05:00.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.896+0000 7fcf72195700 1 --2- 172.21.15.121:0/3319510902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf6c0ffbd0 0x7fcf6c100050 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.896+0000 7fcf72195700 1 --2- 172.21.15.121:0/3319510902 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf6c105a40 0x7fcf6c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.896+0000 7fcf72195700 1 -- 172.21.15.121:0/3319510902 >> 172.21.15.121:0/3319510902 conn(0x7fcf6c0755a0 msgr2=0x7fcf6c0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:00.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.896+0000 7fcf72195700 1 -- 172.21.15.121:0/3319510902 shutdown_connections 2024-04-05T22:05:00.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.896+0000 7fcf72195700 1 -- 172.21.15.121:0/3319510902 wait complete. 2024-04-05T22:05:00.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.897+0000 7fcf72195700 1 Processor -- start 2024-04-05T22:05:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.897+0000 7fcf72195700 1 -- start start 2024-04-05T22:05:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.897+0000 7fcf72195700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf6c0ffbd0 0x7fcf6c101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.897+0000 7fcf72195700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf6c105a40 0x7fcf6c101b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.897+0000 7fcf72195700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c102090 con 0x7fcf6c0ffbd0 2024-04-05T22:05:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.897+0000 7fcf72195700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c1021d0 con 0x7fcf6c105a40 2024-04-05T22:05:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.898+0000 7fcf6b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf6c0ffbd0 0x7fcf6c101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.898+0000 7fcf6b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf6c0ffbd0 0x7fcf6c101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43760/0 (socket says 172.21.15.121:43760) 2024-04-05T22:05:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.898+0000 7fcf6b7fe700 1 -- 172.21.15.121:0/3922613897 learned_addr learned my addr 172.21.15.121:0/3922613897 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.898+0000 7fcf6affd700 1 --2- 172.21.15.121:0/3922613897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf6c105a40 0x7fcf6c101b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.898+0000 7fcf6b7fe700 1 -- 172.21.15.121:0/3922613897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf6c105a40 msgr2=0x7fcf6c101b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.898+0000 7fcf6b7fe700 1 --2- 172.21.15.121:0/3922613897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf6c105a40 0x7fcf6c101b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.898+0000 7fcf6b7fe700 1 -- 172.21.15.121:0/3922613897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf600095f0 con 0x7fcf6c0ffbd0 2024-04-05T22:05:00.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.898+0000 7fcf6b7fe700 1 --2- 172.21.15.121:0/3922613897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf6c0ffbd0 0x7fcf6c101610 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00d900 tx=0x7fcf5c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:00.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.898+0000 7fcf68ff9700 1 -- 172.21.15.121:0/3922613897 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf5c0041f0 con 0x7fcf6c0ffbd0 2024-04-05T22:05:00.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.898+0000 7fcf68ff9700 1 -- 172.21.15.121:0/3922613897 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf5c004cd0 con 0x7fcf6c0ffbd0 2024-04-05T22:05:00.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.898+0000 7fcf72195700 1 -- 172.21.15.121:0/3922613897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6c1a2910 con 0x7fcf6c0ffbd0 2024-04-05T22:05:00.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.898+0000 7fcf68ff9700 1 -- 172.21.15.121:0/3922613897 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf5c005660 con 0x7fcf6c0ffbd0 2024-04-05T22:05:00.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.898+0000 7fcf72195700 1 -- 172.21.15.121:0/3922613897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6c1a2d20 con 0x7fcf6c0ffbd0 2024-04-05T22:05:00.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.899+0000 7fcf72195700 1 -- 172.21.15.121:0/3922613897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6c108fe0 con 0x7fcf6c0ffbd0 2024-04-05T22:05:00.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.900+0000 7fcf68ff9700 1 -- 172.21.15.121:0/3922613897 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fcf5c004e40 con 0x7fcf6c0ffbd0 2024-04-05T22:05:00.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.900+0000 7fcf68ff9700 1 --2- 172.21.15.121:0/3922613897 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcf5406cce0 0x7fcf5406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:00.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.900+0000 7fcf68ff9700 1 -- 172.21.15.121:0/3922613897 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fcf5c08c320 con 0x7fcf6c0ffbd0 2024-04-05T22:05:00.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.900+0000 7fcf6affd700 1 --2- 172.21.15.121:0/3922613897 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcf5406cce0 0x7fcf5406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:00.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.900+0000 7fcf6affd700 1 --2- 172.21.15.121:0/3922613897 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcf5406cce0 0x7fcf5406f1a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c102e00 tx=0x7fcf6001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:00.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:00.905+0000 7fcf68ff9700 1 -- 172.21.15.121:0/3922613897 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fcf5c056410 con 0x7fcf6c0ffbd0 2024-04-05T22:05:01.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.106+0000 7fcf72195700 1 -- 172.21.15.121:0/3922613897 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf6c0611d0 con 0x7fcf5406cce0 2024-04-05T22:05:01.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.116+0000 7fcf68ff9700 1 -- 172.21.15.121:0/3922613897 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7fcf6c0611d0 con 0x7fcf5406cce0 2024-04-05T22:05:01.116 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T22:05:01.116 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (35m) 14m ago 47m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T22:05:01.116 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (47m) 14m ago 47m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T22:05:01.116 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (46m) 14m ago 46m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T22:05:01.117 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (34m) 14m ago 47m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T22:05:01.117 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (33m) 14m ago 46m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T22:05:01.117 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (35m) 14m ago 47m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T22:05:01.117 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (30m) 14m ago 41m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T22:05:01.117 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (29m) 14m ago 41m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T22:05:01.117 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (29m) 14m ago 41m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T22:05:01.117 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (14m) 14m ago 41m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T22:05:01.117 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (37m) 14m ago 48m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T22:05:01.117 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (36m) 14m ago 46m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T22:05:01.117 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (34m) 14m ago 49m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T22:05:01.117 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (34m) 14m ago 46m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T22:05:01.117 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (36m) 14m ago 47m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T22:05:01.118 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (36m) 14m ago 46m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T22:05:01.118 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (33m) 14m ago 45m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T22:05:01.118 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (32m) 14m ago 44m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T22:05:01.118 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (32m) 14m ago 44m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T22:05:01.118 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (31m) 14m ago 43m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T22:05:01.118 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (31m) 14m ago 43m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T22:05:01.118 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (30m) 14m ago 42m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T22:05:01.118 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (36m) 14m ago 46m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T22:05:01.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.118+0000 7fcf72195700 1 -- 172.21.15.121:0/3922613897 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcf5406cce0 msgr2=0x7fcf5406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:01.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.118+0000 7fcf72195700 1 --2- 172.21.15.121:0/3922613897 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcf5406cce0 0x7fcf5406f1a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c102e00 tx=0x7fcf6001a040 comp rx=0 tx=0).stop 2024-04-05T22:05:01.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.118+0000 7fcf72195700 1 -- 172.21.15.121:0/3922613897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf6c0ffbd0 msgr2=0x7fcf6c101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:01.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.118+0000 7fcf72195700 1 --2- 172.21.15.121:0/3922613897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf6c0ffbd0 0x7fcf6c101610 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00d900 tx=0x7fcf5c00dc10 comp rx=0 tx=0).stop 2024-04-05T22:05:01.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.118+0000 7fcf72195700 1 -- 172.21.15.121:0/3922613897 shutdown_connections 2024-04-05T22:05:01.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.118+0000 7fcf72195700 1 --2- 172.21.15.121:0/3922613897 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcf5406cce0 0x7fcf5406f1a0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.118+0000 7fcf72195700 1 --2- 172.21.15.121:0/3922613897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf6c0ffbd0 0x7fcf6c101610 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.118+0000 7fcf72195700 1 --2- 172.21.15.121:0/3922613897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fcf6c105a40 0x7fcf6c101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.118+0000 7fcf72195700 1 -- 172.21.15.121:0/3922613897 >> 172.21.15.121:0/3922613897 conn(0x7fcf6c0755a0 msgr2=0x7fcf6c103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:01.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.118+0000 7fcf72195700 1 -- 172.21.15.121:0/3922613897 shutdown_connections 2024-04-05T22:05:01.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.118+0000 7fcf72195700 1 -- 172.21.15.121:0/3922613897 wait complete. 2024-04-05T22:05:01.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.219+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2426564407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f64105a40 msgr2=0x7f3f64105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.219+0000 7f3f6bfa3700 1 --2- 172.21.15.121:0/2426564407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f64105a40 0x7f3f64105e20 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f3f60009910 tx=0x7f3f60009c20 comp rx=0 tx=0).stop 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.219+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2426564407 shutdown_connections 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.219+0000 7f3f6bfa3700 1 --2- 172.21.15.121:0/2426564407 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f640ffa40 0x7f3f640ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.219+0000 7f3f6bfa3700 1 --2- 172.21.15.121:0/2426564407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f64105a40 0x7f3f64105e20 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.219+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2426564407 >> 172.21.15.121:0/2426564407 conn(0x7f3f640fb570 msgr2=0x7f3f640fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.219+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2426564407 shutdown_connections 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.220+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2426564407 wait complete. 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.220+0000 7f3f6bfa3700 1 Processor -- start 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.220+0000 7f3f6bfa3700 1 -- start start 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.220+0000 7f3f6bfa3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f640ffa40 0x7f3f641952b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.220+0000 7f3f6bfa3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f64105a40 0x7f3f641957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.220+0000 7f3f6bfa3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64195d30 con 0x7f3f640ffa40 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.220+0000 7f3f6bfa3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64195ea0 con 0x7f3f64105a40 2024-04-05T22:05:01.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.220+0000 7f3f69d3f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f640ffa40 0x7f3f641952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.220+0000 7f3f69d3f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f640ffa40 0x7f3f641952b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43774/0 (socket says 172.21.15.121:43774) 2024-04-05T22:05:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.220+0000 7f3f69d3f700 1 -- 172.21.15.121:0/2816250377 learned_addr learned my addr 172.21.15.121:0/2816250377 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.221+0000 7f3f6953e700 1 --2- 172.21.15.121:0/2816250377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f64105a40 0x7f3f641957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.221+0000 7f3f69d3f700 1 -- 172.21.15.121:0/2816250377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f64105a40 msgr2=0x7f3f641957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.221+0000 7f3f69d3f700 1 --2- 172.21.15.121:0/2816250377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f64105a40 0x7f3f641957f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.221+0000 7f3f69d3f700 1 -- 172.21.15.121:0/2816250377 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f600095f0 con 0x7f3f640ffa40 2024-04-05T22:05:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.221+0000 7f3f6953e700 1 --2- 172.21.15.121:0/2816250377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f64105a40 0x7f3f641957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:05:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.221+0000 7f3f69d3f700 1 --2- 172.21.15.121:0/2816250377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f640ffa40 0x7f3f641952b0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f3f60006010 tx=0x7f3f6000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.221+0000 7f3f5affd700 1 -- 172.21.15.121:0/2816250377 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f6001d070 con 0x7f3f640ffa40 2024-04-05T22:05:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.221+0000 7f3f5affd700 1 -- 172.21.15.121:0/2816250377 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f6000f460 con 0x7f3f640ffa40 2024-04-05T22:05:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.221+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2816250377 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f64196120 con 0x7f3f640ffa40 2024-04-05T22:05:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.221+0000 7f3f5affd700 1 -- 172.21.15.121:0/2816250377 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f60021640 con 0x7f3f640ffa40 2024-04-05T22:05:01.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.221+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2816250377 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f641a2ba0 con 0x7f3f640ffa40 2024-04-05T22:05:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.222+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2816250377 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6404ea90 con 0x7f3f640ffa40 2024-04-05T22:05:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.223+0000 7f3f5affd700 1 -- 172.21.15.121:0/2816250377 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3f6000fa20 con 0x7f3f640ffa40 2024-04-05T22:05:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.223+0000 7f3f5affd700 1 --2- 172.21.15.121:0/2816250377 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3f5006cce0 0x7f3f5006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:01.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.223+0000 7f3f5affd700 1 -- 172.21.15.121:0/2816250377 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3f6008d630 con 0x7f3f640ffa40 2024-04-05T22:05:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.223+0000 7f3f6953e700 1 --2- 172.21.15.121:0/2816250377 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3f5006cce0 0x7f3f5006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:01.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.223+0000 7f3f6953e700 1 --2- 172.21.15.121:0/2816250377 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3f5006cce0 0x7f3f5006f1a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3f64069600 tx=0x7f3f54005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:01.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.228+0000 7f3f5affd700 1 -- 172.21.15.121:0/2816250377 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3f6005b1b0 con 0x7f3f640ffa40 2024-04-05T22:05:01.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:00 smithi188 ceph-mon[107793]: pgmap v1164: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:01.275 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:05:01.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:00 smithi121 ceph-mon[145859]: pgmap v1164: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:01.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:05:01.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.501+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2816250377 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3f64196120 con 0x7f3f640ffa40 2024-04-05T22:05:01.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.501+0000 7f3f5affd700 1 -- 172.21.15.121:0/2816250377 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f3f64196120 con 0x7f3f640ffa40 2024-04-05T22:05:01.502 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:05:01.502 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T22:05:01.502 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:05:01.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:05:01.502 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T22:05:01.502 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:05:01.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:05:01.502 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T22:05:01.502 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T22:05:01.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:05:01.502 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T22:05:01.503 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T22:05:01.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:05:01.503 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T22:05:01.503 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T22:05:01.503 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T22:05:01.503 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:05:01.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.503+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2816250377 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3f5006cce0 msgr2=0x7f3f5006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:01.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.503+0000 7f3f6bfa3700 1 --2- 172.21.15.121:0/2816250377 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3f5006cce0 0x7f3f5006f1a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3f64069600 tx=0x7f3f54005f50 comp rx=0 tx=0).stop 2024-04-05T22:05:01.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.504+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2816250377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f640ffa40 msgr2=0x7f3f641952b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:01.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.504+0000 7f3f6bfa3700 1 --2- 172.21.15.121:0/2816250377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f640ffa40 0x7f3f641952b0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f3f60006010 tx=0x7f3f6000be20 comp rx=0 tx=0).stop 2024-04-05T22:05:01.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.504+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2816250377 shutdown_connections 2024-04-05T22:05:01.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.504+0000 7f3f6bfa3700 1 --2- 172.21.15.121:0/2816250377 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3f5006cce0 0x7f3f5006f1a0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.504+0000 7f3f6bfa3700 1 --2- 172.21.15.121:0/2816250377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f640ffa40 0x7f3f641952b0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.504+0000 7f3f6bfa3700 1 --2- 172.21.15.121:0/2816250377 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f3f64105a40 0x7f3f641957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.504+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2816250377 >> 172.21.15.121:0/2816250377 conn(0x7f3f640fb570 msgr2=0x7f3f640fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:01.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.504+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2816250377 shutdown_connections 2024-04-05T22:05:01.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.504+0000 7f3f6bfa3700 1 -- 172.21.15.121:0/2816250377 wait complete. 2024-04-05T22:05:01.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.606+0000 7f6df19da700 1 -- 172.21.15.121:0/1061184021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dec100620 msgr2=0x7f6dec100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:01.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.606+0000 7f6df19da700 1 --2- 172.21.15.121:0/1061184021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dec100620 0x7f6dec100a00 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009910 tx=0x7f6dd4009c20 comp rx=0 tx=0).stop 2024-04-05T22:05:01.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.606+0000 7f6df19da700 1 -- 172.21.15.121:0/1061184021 shutdown_connections 2024-04-05T22:05:01.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.606+0000 7f6df19da700 1 --2- 172.21.15.121:0/1061184021 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6dec100fd0 0x7f6dec104720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.606+0000 7f6df19da700 1 --2- 172.21.15.121:0/1061184021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dec100620 0x7f6dec100a00 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.606+0000 7f6df19da700 1 -- 172.21.15.121:0/1061184021 >> 172.21.15.121:0/1061184021 conn(0x7f6dec074e00 msgr2=0x7f6dec075210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:01.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.606+0000 7f6df19da700 1 -- 172.21.15.121:0/1061184021 shutdown_connections 2024-04-05T22:05:01.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.606+0000 7f6df19da700 1 -- 172.21.15.121:0/1061184021 wait complete. 2024-04-05T22:05:01.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6df19da700 1 Processor -- start 2024-04-05T22:05:01.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6df19da700 1 -- start start 2024-04-05T22:05:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6df19da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dec100620 0x7f6dec19c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6df19da700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6dec100fd0 0x7f6dec19ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6df19da700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec19d0f0 con 0x7f6dec100620 2024-04-05T22:05:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6df19da700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dec1965a0 con 0x7f6dec100fd0 2024-04-05T22:05:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6deaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dec100620 0x7f6dec19c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6deaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dec100620 0x7f6dec19c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43792/0 (socket says 172.21.15.121:43792) 2024-04-05T22:05:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6deaffd700 1 -- 172.21.15.121:0/2830167196 learned_addr learned my addr 172.21.15.121:0/2830167196 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6dea7fc700 1 --2- 172.21.15.121:0/2830167196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6dec100fd0 0x7f6dec19ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6deaffd700 1 -- 172.21.15.121:0/2830167196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6dec100fd0 msgr2=0x7f6dec19ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6deaffd700 1 --2- 172.21.15.121:0/2830167196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6dec100fd0 0x7f6dec19ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6deaffd700 1 -- 172.21.15.121:0/2830167196 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd40095f0 con 0x7f6dec100620 2024-04-05T22:05:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6dea7fc700 1 --2- 172.21.15.121:0/2830167196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6dec100fd0 0x7f6dec19ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:05:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.607+0000 7f6deaffd700 1 --2- 172.21.15.121:0/2830167196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dec100620 0x7f6dec19c520 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4006010 tx=0x7f6dd400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.608+0000 7f6df09d8700 1 -- 172.21.15.121:0/2830167196 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd401d070 con 0x7f6dec100620 2024-04-05T22:05:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.608+0000 7f6df19da700 1 -- 172.21.15.121:0/2830167196 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dec196820 con 0x7f6dec100620 2024-04-05T22:05:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.608+0000 7f6df09d8700 1 -- 172.21.15.121:0/2830167196 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6dd400f460 con 0x7f6dec100620 2024-04-05T22:05:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.608+0000 7f6df09d8700 1 -- 172.21.15.121:0/2830167196 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd4021640 con 0x7f6dec100620 2024-04-05T22:05:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.608+0000 7f6df19da700 1 -- 172.21.15.121:0/2830167196 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dec196c90 con 0x7f6dec100620 2024-04-05T22:05:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.609+0000 7f6df19da700 1 -- 172.21.15.121:0/2830167196 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dec04ea90 con 0x7f6dec100620 2024-04-05T22:05:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.609+0000 7f6df09d8700 1 -- 172.21.15.121:0/2830167196 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6dd400f9e0 con 0x7f6dec100620 2024-04-05T22:05:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.609+0000 7f6df09d8700 1 --2- 172.21.15.121:0/2830167196 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6dd806cce0 0x7f6dd806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.610+0000 7f6df09d8700 1 -- 172.21.15.121:0/2830167196 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6dd408d900 con 0x7f6dec100620 2024-04-05T22:05:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.610+0000 7f6dea7fc700 1 --2- 172.21.15.121:0/2830167196 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6dd806cce0 0x7f6dd806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.610+0000 7f6dea7fc700 1 --2- 172.21.15.121:0/2830167196 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6dd806cce0 0x7f6dd806f1a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6dec106130 tx=0x7f6ddc008040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.614+0000 7f6df09d8700 1 -- 172.21.15.121:0/2830167196 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6dd405b400 con 0x7f6dec100620 2024-04-05T22:05:01.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.847+0000 7f6df19da700 1 -- 172.21.15.121:0/2830167196 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6dec197070 con 0x7f6dec100620 2024-04-05T22:05:01.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.847+0000 7f6df09d8700 1 -- 172.21.15.121:0/2830167196 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f6dd405af90 con 0x7f6dec100620 2024-04-05T22:05:01.848 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T22:05:01.848 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T22:05:01.848 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:05:01.848 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T22:05:01.848 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:05:01.848 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T22:05:01.848 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T22:05:01.848 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T22:05:01.848 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T22:05:01.848 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T22:05:01.848 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T22:05:01.848 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T22:05:01.849 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:05:01.850 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:05:01.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.849+0000 7f6df19da700 1 -- 172.21.15.121:0/2830167196 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6dd806cce0 msgr2=0x7f6dd806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:01.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.849+0000 7f6df19da700 1 --2- 172.21.15.121:0/2830167196 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6dd806cce0 0x7f6dd806f1a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6dec106130 tx=0x7f6ddc008040 comp rx=0 tx=0).stop 2024-04-05T22:05:01.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.849+0000 7f6df19da700 1 -- 172.21.15.121:0/2830167196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dec100620 msgr2=0x7f6dec19c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:01.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.849+0000 7f6df19da700 1 --2- 172.21.15.121:0/2830167196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dec100620 0x7f6dec19c520 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4006010 tx=0x7f6dd400be20 comp rx=0 tx=0).stop 2024-04-05T22:05:01.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.849+0000 7f6df19da700 1 -- 172.21.15.121:0/2830167196 shutdown_connections 2024-04-05T22:05:01.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.849+0000 7f6df19da700 1 --2- 172.21.15.121:0/2830167196 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6dd806cce0 0x7f6dd806f1a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.849+0000 7f6df19da700 1 --2- 172.21.15.121:0/2830167196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6dec100620 0x7f6dec19c520 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.849+0000 7f6df19da700 1 --2- 172.21.15.121:0/2830167196 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6dec100fd0 0x7f6dec19ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.849+0000 7f6df19da700 1 -- 172.21.15.121:0/2830167196 >> 172.21.15.121:0/2830167196 conn(0x7f6dec074e00 msgr2=0x7f6dec0fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:01.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.850+0000 7f6df19da700 1 -- 172.21.15.121:0/2830167196 shutdown_connections 2024-04-05T22:05:01.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.850+0000 7f6df19da700 1 -- 172.21.15.121:0/2830167196 wait complete. 2024-04-05T22:05:01.852 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T22:05:01.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.953+0000 7fb09c85b700 1 -- 172.21.15.121:0/2507021743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0980ffa40 msgr2=0x7fb0980ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:01.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.953+0000 7fb09c85b700 1 --2- 172.21.15.121:0/2507021743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0980ffa40 0x7fb0980ffec0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fb08c009910 tx=0x7fb08c009c20 comp rx=0 tx=0).stop 2024-04-05T22:05:01.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.953+0000 7fb09c85b700 1 -- 172.21.15.121:0/2507021743 shutdown_connections 2024-04-05T22:05:01.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.953+0000 7fb09c85b700 1 --2- 172.21.15.121:0/2507021743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0980ffa40 0x7fb0980ffec0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.953+0000 7fb09c85b700 1 --2- 172.21.15.121:0/2507021743 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb098105a40 0x7fb098105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.953+0000 7fb09c85b700 1 -- 172.21.15.121:0/2507021743 >> 172.21.15.121:0/2507021743 conn(0x7fb0980fb570 msgr2=0x7fb0980fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:01.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.953+0000 7fb09c85b700 1 -- 172.21.15.121:0/2507021743 shutdown_connections 2024-04-05T22:05:01.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.953+0000 7fb09c85b700 1 -- 172.21.15.121:0/2507021743 wait complete. 2024-04-05T22:05:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.954+0000 7fb09c85b700 1 Processor -- start 2024-04-05T22:05:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.954+0000 7fb09c85b700 1 -- start start 2024-04-05T22:05:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb09c85b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0980ffa40 0x7fb098195660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb09c85b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb098105a40 0x7fb098195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb09c85b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb098196280 con 0x7fb0980ffa40 2024-04-05T22:05:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb09c85b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09819a010 con 0x7fb098105a40 2024-04-05T22:05:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb09659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0980ffa40 0x7fb098195660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb09659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0980ffa40 0x7fb098195660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43798/0 (socket says 172.21.15.121:43798) 2024-04-05T22:05:01.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb09659c700 1 -- 172.21.15.121:0/2772389099 learned_addr learned my addr 172.21.15.121:0/2772389099 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb095d9b700 1 --2- 172.21.15.121:0/2772389099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb098105a40 0x7fb098195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb09659c700 1 -- 172.21.15.121:0/2772389099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb098105a40 msgr2=0x7fb098195ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb09659c700 1 --2- 172.21.15.121:0/2772389099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb098105a40 0x7fb098195ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb09659c700 1 -- 172.21.15.121:0/2772389099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08c0095f0 con 0x7fb0980ffa40 2024-04-05T22:05:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb09659c700 1 --2- 172.21.15.121:0/2772389099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0980ffa40 0x7fb098195660 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fb08800d960 tx=0x7fb08800dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb0877fe700 1 -- 172.21.15.121:0/2772389099 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb08800eba0 con 0x7fb0980ffa40 2024-04-05T22:05:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.955+0000 7fb0877fe700 1 -- 172.21.15.121:0/2772389099 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb08800ed00 con 0x7fb0980ffa40 2024-04-05T22:05:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.956+0000 7fb09c85b700 1 -- 172.21.15.121:0/2772389099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb09819a290 con 0x7fb0980ffa40 2024-04-05T22:05:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.956+0000 7fb0877fe700 1 -- 172.21.15.121:0/2772389099 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb088014730 con 0x7fb0980ffa40 2024-04-05T22:05:01.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.956+0000 7fb09c85b700 1 -- 172.21.15.121:0/2772389099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09819a7e0 con 0x7fb0980ffa40 2024-04-05T22:05:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.957+0000 7fb09c85b700 1 -- 172.21.15.121:0/2772389099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb09804ea90 con 0x7fb0980ffa40 2024-04-05T22:05:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.957+0000 7fb0877fe700 1 -- 172.21.15.121:0/2772389099 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb088016440 con 0x7fb0980ffa40 2024-04-05T22:05:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.957+0000 7fb0877fe700 1 --2- 172.21.15.121:0/2772389099 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb08006cce0 0x7fb08006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.957+0000 7fb0877fe700 1 -- 172.21.15.121:0/2772389099 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb08808dfd0 con 0x7fb0980ffa40 2024-04-05T22:05:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.957+0000 7fb095d9b700 1 --2- 172.21.15.121:0/2772389099 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb08006cce0 0x7fb08006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:01.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.958+0000 7fb095d9b700 1 --2- 172.21.15.121:0/2772389099 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb08006cce0 0x7fb08006f1a0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb098196c80 tx=0x7fb08c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:01.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:01.962+0000 7fb0877fe700 1 -- 172.21.15.121:0/2772389099 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb088058040 con 0x7fb0980ffa40 2024-04-05T22:05:02.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.173+0000 7fb09c85b700 1 -- 172.21.15.121:0/2772389099 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb09819ac20 con 0x7fb08006cce0 2024-04-05T22:05:02.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.174+0000 7fb0877fe700 1 -- 172.21.15.121:0/2772389099 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb09819ac20 con 0x7fb08006cce0 2024-04-05T22:05:02.174 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:05:02.174 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T22:05:02.174 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T22:05:02.174 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T22:05:02.175 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T22:05:02.175 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T22:05:02.175 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T22:05:02.175 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T22:05:02.175 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T22:05:02.175 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T22:05:02.175 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T22:05:02.175 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T22:05:02.175 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T22:05:02.175 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T22:05:02.175 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:05:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.177+0000 7fb09c85b700 1 -- 172.21.15.121:0/2772389099 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb08006cce0 msgr2=0x7fb08006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.177+0000 7fb09c85b700 1 --2- 172.21.15.121:0/2772389099 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb08006cce0 0x7fb08006f1a0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb098196c80 tx=0x7fb08c01a040 comp rx=0 tx=0).stop 2024-04-05T22:05:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.177+0000 7fb09c85b700 1 -- 172.21.15.121:0/2772389099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0980ffa40 msgr2=0x7fb098195660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.177+0000 7fb09c85b700 1 --2- 172.21.15.121:0/2772389099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0980ffa40 0x7fb098195660 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fb08800d960 tx=0x7fb08800dc70 comp rx=0 tx=0).stop 2024-04-05T22:05:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.177+0000 7fb09c85b700 1 -- 172.21.15.121:0/2772389099 shutdown_connections 2024-04-05T22:05:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.177+0000 7fb09c85b700 1 --2- 172.21.15.121:0/2772389099 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb08006cce0 0x7fb08006f1a0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.177+0000 7fb09c85b700 1 --2- 172.21.15.121:0/2772389099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0980ffa40 0x7fb098195660 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.177+0000 7fb09c85b700 1 --2- 172.21.15.121:0/2772389099 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb098105a40 0x7fb098195ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.177+0000 7fb09c85b700 1 -- 172.21.15.121:0/2772389099 >> 172.21.15.121:0/2772389099 conn(0x7fb0980fb570 msgr2=0x7fb0980fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.177+0000 7fb09c85b700 1 -- 172.21.15.121:0/2772389099 shutdown_connections 2024-04-05T22:05:02.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.177+0000 7fb09c85b700 1 -- 172.21.15.121:0/2772389099 wait complete. 2024-04-05T22:05:02.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.278+0000 7fe0177c4700 1 -- 172.21.15.121:0/3176647296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0101026a0 msgr2=0x7fe01010ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:02.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.278+0000 7fe0177c4700 1 --2- 172.21.15.121:0/3176647296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0101026a0 0x7fe01010ab90 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fe000009940 tx=0x7fe000009c50 comp rx=0 tx=0).stop 2024-04-05T22:05:02.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.278+0000 7fe0177c4700 1 -- 172.21.15.121:0/3176647296 shutdown_connections 2024-04-05T22:05:02.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.278+0000 7fe0177c4700 1 --2- 172.21.15.121:0/3176647296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0101026a0 0x7fe01010ab90 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:02.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.278+0000 7fe0177c4700 1 --2- 172.21.15.121:0/3176647296 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe010101d80 0x7fe010102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:02.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.278+0000 7fe0177c4700 1 -- 172.21.15.121:0/3176647296 >> 172.21.15.121:0/3176647296 conn(0x7fe0100fb570 msgr2=0x7fe0100fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:02.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.279+0000 7fe0177c4700 1 -- 172.21.15.121:0/3176647296 shutdown_connections 2024-04-05T22:05:02.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.279+0000 7fe0177c4700 1 -- 172.21.15.121:0/3176647296 wait complete. 2024-04-05T22:05:02.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:01 smithi188 ceph-mon[107793]: from='client.35914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:02.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:01 smithi188 ceph-mon[107793]: from='client.35918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:02.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:01 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2816250377' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:02.280 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:01 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2830167196' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:05:02.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.279+0000 7fe0177c4700 1 Processor -- start 2024-04-05T22:05:02.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.279+0000 7fe0177c4700 1 -- start start 2024-04-05T22:05:02.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.279+0000 7fe0177c4700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe010101d80 0x7fe0101966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:02.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.279+0000 7fe0177c4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0101026a0 0x7fe010196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:02.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.279+0000 7fe0177c4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe010197280 con 0x7fe0101026a0 2024-04-05T22:05:02.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.279+0000 7fe0177c4700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe010199fb0 con 0x7fe010101d80 2024-04-05T22:05:02.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.279+0000 7fe014d5f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0101026a0 0x7fe010196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:02.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.279+0000 7fe014d5f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0101026a0 0x7fe010196bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43818/0 (socket says 172.21.15.121:43818) 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.279+0000 7fe014d5f700 1 -- 172.21.15.121:0/3230622720 learned_addr learned my addr 172.21.15.121:0/3230622720 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.280+0000 7fe015560700 1 --2- 172.21.15.121:0/3230622720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe010101d80 0x7fe0101966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.280+0000 7fe014d5f700 1 -- 172.21.15.121:0/3230622720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe010101d80 msgr2=0x7fe0101966b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.280+0000 7fe014d5f700 1 --2- 172.21.15.121:0/3230622720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe010101d80 0x7fe0101966b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.280+0000 7fe014d5f700 1 -- 172.21.15.121:0/3230622720 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0000095f0 con 0x7fe0101026a0 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.280+0000 7fe015560700 1 --2- 172.21.15.121:0/3230622720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe010101d80 0x7fe0101966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.280+0000 7fe014d5f700 1 --2- 172.21.15.121:0/3230622720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0101026a0 0x7fe010196bf0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fe000006010 tx=0x7fe0000056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.280+0000 7fe0067fc700 1 -- 172.21.15.121:0/3230622720 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe00001d070 con 0x7fe0101026a0 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.280+0000 7fe0067fc700 1 -- 172.21.15.121:0/3230622720 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe00000f460 con 0x7fe0101026a0 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.280+0000 7fe0177c4700 1 -- 172.21.15.121:0/3230622720 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01019a230 con 0x7fe0101026a0 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.280+0000 7fe0067fc700 1 -- 172.21.15.121:0/3230622720 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe000021640 con 0x7fe0101026a0 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.280+0000 7fe0177c4700 1 -- 172.21.15.121:0/3230622720 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01019a720 con 0x7fe0101026a0 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.281+0000 7fe0177c4700 1 -- 172.21.15.121:0/3230622720 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe010108290 con 0x7fe0101026a0 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.281+0000 7fe0067fc700 1 -- 172.21.15.121:0/3230622720 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe00000fa40 con 0x7fe0101026a0 2024-04-05T22:05:02.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.282+0000 7fe0067fc700 1 --2- 172.21.15.121:0/3230622720 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdffc06cce0 0x7fdffc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:02.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.282+0000 7fe0067fc700 1 -- 172.21.15.121:0/3230622720 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe00008d950 con 0x7fe0101026a0 2024-04-05T22:05:02.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.282+0000 7fe015560700 1 --2- 172.21.15.121:0/3230622720 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdffc06cce0 0x7fdffc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:02.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.282+0000 7fe015560700 1 --2- 172.21.15.121:0/3230622720 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdffc06cce0 0x7fdffc06f1a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe00c005950 tx=0x7fe00c005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:02.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.286+0000 7fe0067fc700 1 -- 172.21.15.121:0/3230622720 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe0000579c0 con 0x7fe0101026a0 2024-04-05T22:05:02.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:01 smithi121 ceph-mon[145859]: from='client.35914 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:02.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:01 smithi121 ceph-mon[145859]: from='client.35918 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:02.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:01 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2816250377' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:02.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:01 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2830167196' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:05:02.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.557+0000 7fe0177c4700 1 -- 172.21.15.121:0/3230622720 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe010107c70 con 0x7fe0101026a0 2024-04-05T22:05:02.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.558+0000 7fe0067fc700 1 -- 172.21.15.121:0/3230622720 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7fe000026e10 con 0x7fe0101026a0 2024-04-05T22:05:02.558 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:05:02.558 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:05:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:05:02.558 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T22:05:02.558 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:05:02.558 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T22:05:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:05:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T22:05:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:05:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T22:05:02.558 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:05:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:05:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:05:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:05:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:05:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:05:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:05:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:05:02.559 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:05:02.559 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:05:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T22:05:02.559 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:05:02.559 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T22:05:02.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.560+0000 7fe0177c4700 1 -- 172.21.15.121:0/3230622720 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdffc06cce0 msgr2=0x7fdffc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:02.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.560+0000 7fe0177c4700 1 --2- 172.21.15.121:0/3230622720 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdffc06cce0 0x7fdffc06f1a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe00c005950 tx=0x7fe00c005f50 comp rx=0 tx=0).stop 2024-04-05T22:05:02.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.560+0000 7fe0177c4700 1 -- 172.21.15.121:0/3230622720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0101026a0 msgr2=0x7fe010196bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:02.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.560+0000 7fe0177c4700 1 --2- 172.21.15.121:0/3230622720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0101026a0 0x7fe010196bf0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fe000006010 tx=0x7fe0000056a0 comp rx=0 tx=0).stop 2024-04-05T22:05:02.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.560+0000 7fe0177c4700 1 -- 172.21.15.121:0/3230622720 shutdown_connections 2024-04-05T22:05:02.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.560+0000 7fe0177c4700 1 --2- 172.21.15.121:0/3230622720 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fdffc06cce0 0x7fdffc06f1a0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:02.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.560+0000 7fe0177c4700 1 --2- 172.21.15.121:0/3230622720 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe010101d80 0x7fe0101966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:02.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.560+0000 7fe0177c4700 1 --2- 172.21.15.121:0/3230622720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0101026a0 0x7fe010196bf0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:02.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.560+0000 7fe0177c4700 1 -- 172.21.15.121:0/3230622720 >> 172.21.15.121:0/3230622720 conn(0x7fe0100fb570 msgr2=0x7fe0101053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:02.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.560+0000 7fe0177c4700 1 -- 172.21.15.121:0/3230622720 shutdown_connections 2024-04-05T22:05:02.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:02.560+0000 7fe0177c4700 1 -- 172.21.15.121:0/3230622720 wait complete. 2024-04-05T22:05:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:03 smithi121 ceph-mon[145859]: from='client.35922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:03 smithi121 ceph-mon[145859]: pgmap v1165: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:03 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3230622720' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:05:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:03 smithi188 ceph-mon[107793]: from='client.35922 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:03 smithi188 ceph-mon[107793]: pgmap v1165: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:03.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:03 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3230622720' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:05:04.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:04 smithi188 ceph-mon[107793]: from='client.35934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:04.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:04 smithi121 ceph-mon[145859]: from='client.35934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:05 smithi188 ceph-mon[107793]: pgmap v1166: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:05.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:05 smithi121 ceph-mon[145859]: pgmap v1166: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:07.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:07 smithi188 ceph-mon[107793]: pgmap v1167: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:07.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:07 smithi121 ceph-mon[145859]: pgmap v1167: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:09.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:09 smithi188 ceph-mon[107793]: pgmap v1168: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:09.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:09 smithi121 ceph-mon[145859]: pgmap v1168: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:11.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:11 smithi188 ceph-mon[107793]: pgmap v1169: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:11.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:11 smithi121 ceph-mon[145859]: pgmap v1169: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:13.239 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:13 smithi121 ceph-mon[145859]: pgmap v1170: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:13.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:13 smithi188 ceph-mon[107793]: pgmap v1170: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:15.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:15 smithi188 ceph-mon[107793]: pgmap v1171: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:15.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:15 smithi121 ceph-mon[145859]: pgmap v1171: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:16.415 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:16 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:05:16.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:16 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:05:17.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:17 smithi121 ceph-mon[145859]: pgmap v1172: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:17.528 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:17 smithi188 ceph-mon[107793]: pgmap v1172: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:19.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:19 smithi121 ceph-mon[145859]: pgmap v1173: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:19.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:19 smithi188 ceph-mon[107793]: pgmap v1173: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:21.461 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:21 smithi188 ceph-mon[107793]: pgmap v1174: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:21.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:21 smithi121 ceph-mon[145859]: pgmap v1174: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:23.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:23 smithi121 ceph-mon[145859]: pgmap v1175: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:23.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:23 smithi188 ceph-mon[107793]: pgmap v1175: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:25.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:25 smithi121 ceph-mon[145859]: pgmap v1176: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:25.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:25 smithi188 ceph-mon[107793]: pgmap v1176: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:27.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:27 smithi121 ceph-mon[145859]: pgmap v1177: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:27.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:27 smithi188 ceph-mon[107793]: pgmap v1177: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:29.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:29 smithi121 ceph-mon[145859]: pgmap v1178: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:29.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:29 smithi188 ceph-mon[107793]: pgmap v1178: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:31 smithi121 ceph-mon[145859]: pgmap v1179: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:31.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:31 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:05:31.528 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:31 smithi188 ceph-mon[107793]: pgmap v1179: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:31.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:31 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:05:32.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.675+0000 7f6fa0952700 1 -- 172.21.15.121:0/1843664830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f9c101db0 msgr2=0x7f6f9c102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:32.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.675+0000 7f6fa0952700 1 --2- 172.21.15.121:0/1843664830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f9c101db0 0x7f6f9c102190 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f6f90009910 tx=0x7f6f90009c20 comp rx=0 tx=0).stop 2024-04-05T22:05:32.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.675+0000 7f6fa0952700 1 -- 172.21.15.121:0/1843664830 shutdown_connections 2024-04-05T22:05:32.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.675+0000 7f6fa0952700 1 --2- 172.21.15.121:0/1843664830 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f9c1026d0 0x7f6f9c10abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:32.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.675+0000 7f6fa0952700 1 --2- 172.21.15.121:0/1843664830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f9c101db0 0x7f6f9c102190 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:32.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.675+0000 7f6fa0952700 1 -- 172.21.15.121:0/1843664830 >> 172.21.15.121:0/1843664830 conn(0x7f6f9c0fb540 msgr2=0x7f6f9c0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:32.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.676+0000 7f6fa0952700 1 -- 172.21.15.121:0/1843664830 shutdown_connections 2024-04-05T22:05:32.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.676+0000 7f6fa0952700 1 -- 172.21.15.121:0/1843664830 wait complete. 2024-04-05T22:05:32.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.676+0000 7f6fa0952700 1 Processor -- start 2024-04-05T22:05:32.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.676+0000 7f6fa0952700 1 -- start start 2024-04-05T22:05:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6fa0952700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f9c101db0 0x7f6f9c196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6fa0952700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f9c1026d0 0x7f6f9c196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6fa0952700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c197270 con 0x7f6f9c101db0 2024-04-05T22:05:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6fa0952700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c19b000 con 0x7f6f9c1026d0 2024-04-05T22:05:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6f9a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f9c101db0 0x7f6f9c196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6f9a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f9c101db0 0x7f6f9c196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32970/0 (socket says 172.21.15.121:32970) 2024-04-05T22:05:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6f9a59c700 1 -- 172.21.15.121:0/1293996141 learned_addr learned my addr 172.21.15.121:0/1293996141 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6f99d9b700 1 --2- 172.21.15.121:0/1293996141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f9c1026d0 0x7f6f9c196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6f9a59c700 1 -- 172.21.15.121:0/1293996141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f9c1026d0 msgr2=0x7f6f9c196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6f9a59c700 1 --2- 172.21.15.121:0/1293996141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f9c1026d0 0x7f6f9c196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6f9a59c700 1 -- 172.21.15.121:0/1293996141 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f900095f0 con 0x7f6f9c101db0 2024-04-05T22:05:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6f9a59c700 1 --2- 172.21.15.121:0/1293996141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f9c101db0 0x7f6f9c196650 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f6f90006010 tx=0x7f6f9000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:32.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6f8b7fe700 1 -- 172.21.15.121:0/1293996141 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f9001d070 con 0x7f6f9c101db0 2024-04-05T22:05:32.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6f8b7fe700 1 -- 172.21.15.121:0/1293996141 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f9000f460 con 0x7f6f9c101db0 2024-04-05T22:05:32.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6fa0952700 1 -- 172.21.15.121:0/1293996141 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9c19b280 con 0x7f6f9c101db0 2024-04-05T22:05:32.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6f8b7fe700 1 -- 172.21.15.121:0/1293996141 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f90021620 con 0x7f6f9c101db0 2024-04-05T22:05:32.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.677+0000 7f6fa0952700 1 -- 172.21.15.121:0/1293996141 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9c19b770 con 0x7f6f9c101db0 2024-04-05T22:05:32.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.678+0000 7f6fa0952700 1 -- 172.21.15.121:0/1293996141 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9c1082c0 con 0x7f6f9c101db0 2024-04-05T22:05:32.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.679+0000 7f6f8b7fe700 1 -- 172.21.15.121:0/1293996141 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6f9000f5d0 con 0x7f6f9c101db0 2024-04-05T22:05:32.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.679+0000 7f6f8b7fe700 1 --2- 172.21.15.121:0/1293996141 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6f8406cce0 0x7f6f8406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:32.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.680+0000 7f6f99d9b700 1 --2- 172.21.15.121:0/1293996141 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6f8406cce0 0x7f6f8406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:32.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.680+0000 7f6f8b7fe700 1 -- 172.21.15.121:0/1293996141 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6f9008d8a0 con 0x7f6f9c101db0 2024-04-05T22:05:32.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.680+0000 7f6f99d9b700 1 --2- 172.21.15.121:0/1293996141 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6f8406cce0 0x7f6f8406f1a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c197c70 tx=0x7f6f8c005cf0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:32.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.684+0000 7f6f8b7fe700 1 -- 172.21.15.121:0/1293996141 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6f90057910 con 0x7f6f9c101db0 2024-04-05T22:05:32.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.893+0000 7f6fa0952700 1 -- 172.21.15.121:0/1293996141 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f9c02cd60 con 0x7f6f8406cce0 2024-04-05T22:05:32.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.894+0000 7f6f8b7fe700 1 -- 172.21.15.121:0/1293996141 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6f9c02cd60 con 0x7f6f8406cce0 2024-04-05T22:05:32.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.896+0000 7f6fa0952700 1 -- 172.21.15.121:0/1293996141 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6f8406cce0 msgr2=0x7f6f8406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:32.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.896+0000 7f6fa0952700 1 --2- 172.21.15.121:0/1293996141 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6f8406cce0 0x7f6f8406f1a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c197c70 tx=0x7f6f8c005cf0 comp rx=0 tx=0).stop 2024-04-05T22:05:32.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.896+0000 7f6fa0952700 1 -- 172.21.15.121:0/1293996141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f9c101db0 msgr2=0x7f6f9c196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:32.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.896+0000 7f6fa0952700 1 --2- 172.21.15.121:0/1293996141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f9c101db0 0x7f6f9c196650 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f6f90006010 tx=0x7f6f9000be20 comp rx=0 tx=0).stop 2024-04-05T22:05:32.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.896+0000 7f6fa0952700 1 -- 172.21.15.121:0/1293996141 shutdown_connections 2024-04-05T22:05:32.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.896+0000 7f6fa0952700 1 --2- 172.21.15.121:0/1293996141 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6f8406cce0 0x7f6f8406f1a0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:32.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.896+0000 7f6fa0952700 1 --2- 172.21.15.121:0/1293996141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f9c101db0 0x7f6f9c196650 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:32.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.896+0000 7f6fa0952700 1 --2- 172.21.15.121:0/1293996141 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f9c1026d0 0x7f6f9c196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:32.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.896+0000 7f6fa0952700 1 -- 172.21.15.121:0/1293996141 >> 172.21.15.121:0/1293996141 conn(0x7f6f9c0fb540 msgr2=0x7f6f9c105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:32.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.896+0000 7f6fa0952700 1 -- 172.21.15.121:0/1293996141 shutdown_connections 2024-04-05T22:05:32.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.896+0000 7f6fa0952700 1 -- 172.21.15.121:0/1293996141 wait complete. 2024-04-05T22:05:32.906 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T22:05:33.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.999+0000 7f2125a1e700 1 -- 172.21.15.121:0/1212311024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2120105a70 msgr2=0x7f2120105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:33.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.999+0000 7f2125a1e700 1 --2- 172.21.15.121:0/1212311024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2120105a70 0x7f2120105e50 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2110009910 tx=0x7f2110009c20 comp rx=0 tx=0).stop 2024-04-05T22:05:33.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.999+0000 7f2125a1e700 1 -- 172.21.15.121:0/1212311024 shutdown_connections 2024-04-05T22:05:33.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.999+0000 7f2125a1e700 1 --2- 172.21.15.121:0/1212311024 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f21200ffa70 0x7f21200ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.999+0000 7f2125a1e700 1 --2- 172.21.15.121:0/1212311024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2120105a70 0x7f2120105e50 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.999+0000 7f2125a1e700 1 -- 172.21.15.121:0/1212311024 >> 172.21.15.121:0/1212311024 conn(0x7f21200fb560 msgr2=0x7f21200fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:33.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:32.999+0000 7f2125a1e700 1 -- 172.21.15.121:0/1212311024 shutdown_connections 2024-04-05T22:05:33.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.000+0000 7f2125a1e700 1 -- 172.21.15.121:0/1212311024 wait complete. 2024-04-05T22:05:33.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.000+0000 7f2125a1e700 1 Processor -- start 2024-04-05T22:05:33.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.000+0000 7f2125a1e700 1 -- start start 2024-04-05T22:05:33.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.000+0000 7f2125a1e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21200ffa70 0x7f212019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:33.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.000+0000 7f2125a1e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2120105a70 0x7f212019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:33.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.000+0000 7f2125a1e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212019f1f0 con 0x7f21200ffa70 2024-04-05T22:05:33.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.000+0000 7f2125a1e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21201986a0 con 0x7f2120105a70 2024-04-05T22:05:33.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.001+0000 7f211effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21200ffa70 0x7f212019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:33.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.001+0000 7f211effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21200ffa70 0x7f212019e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32988/0 (socket says 172.21.15.121:32988) 2024-04-05T22:05:33.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.001+0000 7f211effd700 1 -- 172.21.15.121:0/240698666 learned_addr learned my addr 172.21.15.121:0/240698666 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:33.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.001+0000 7f211e7fc700 1 --2- 172.21.15.121:0/240698666 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2120105a70 0x7f212019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:33.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.001+0000 7f211effd700 1 -- 172.21.15.121:0/240698666 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2120105a70 msgr2=0x7f212019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:33.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.001+0000 7f211effd700 1 --2- 172.21.15.121:0/240698666 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2120105a70 0x7f212019eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.001+0000 7f211effd700 1 -- 172.21.15.121:0/240698666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21100095f0 con 0x7f21200ffa70 2024-04-05T22:05:33.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.001+0000 7f211effd700 1 --2- 172.21.15.121:0/240698666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21200ffa70 0x7f212019e620 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2110000c00 tx=0x7f211000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:33.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.001+0000 7f2124a1c700 1 -- 172.21.15.121:0/240698666 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f211001d070 con 0x7f21200ffa70 2024-04-05T22:05:33.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.001+0000 7f2125a1e700 1 -- 172.21.15.121:0/240698666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2120198920 con 0x7f21200ffa70 2024-04-05T22:05:33.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.001+0000 7f2124a1c700 1 -- 172.21.15.121:0/240698666 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21100044c0 con 0x7f21200ffa70 2024-04-05T22:05:33.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.001+0000 7f2124a1c700 1 -- 172.21.15.121:0/240698666 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21100039d0 con 0x7f21200ffa70 2024-04-05T22:05:33.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.001+0000 7f2125a1e700 1 -- 172.21.15.121:0/240698666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2120198db0 con 0x7f21200ffa70 2024-04-05T22:05:33.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.002+0000 7f2125a1e700 1 -- 172.21.15.121:0/240698666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f212004ea90 con 0x7f21200ffa70 2024-04-05T22:05:33.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.003+0000 7f2124a1c700 1 -- 172.21.15.121:0/240698666 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f21100048e0 con 0x7f21200ffa70 2024-04-05T22:05:33.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.003+0000 7f2124a1c700 1 --2- 172.21.15.121:0/240698666 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f210806cce0 0x7f210806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:33.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.003+0000 7f2124a1c700 1 -- 172.21.15.121:0/240698666 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f211008dc60 con 0x7f21200ffa70 2024-04-05T22:05:33.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.003+0000 7f211e7fc700 1 --2- 172.21.15.121:0/240698666 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f210806cce0 0x7f210806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:33.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.004+0000 7f211e7fc700 1 --2- 172.21.15.121:0/240698666 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f210806cce0 0x7f210806f1a0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2120104e30 tx=0x7f2114009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:33.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.008+0000 7f2124a1c700 1 -- 172.21.15.121:0/240698666 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f211005b760 con 0x7f21200ffa70 2024-04-05T22:05:33.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.228+0000 7f2125a1e700 1 -- 172.21.15.121:0/240698666 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f21201992d0 con 0x7f210806cce0 2024-04-05T22:05:33.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.229+0000 7f2124a1c700 1 -- 172.21.15.121:0/240698666 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f21201992d0 con 0x7f210806cce0 2024-04-05T22:05:33.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.231+0000 7f2125a1e700 1 -- 172.21.15.121:0/240698666 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f210806cce0 msgr2=0x7f210806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:33.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.231+0000 7f2125a1e700 1 --2- 172.21.15.121:0/240698666 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f210806cce0 0x7f210806f1a0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2120104e30 tx=0x7f2114009260 comp rx=0 tx=0).stop 2024-04-05T22:05:33.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.231+0000 7f2125a1e700 1 -- 172.21.15.121:0/240698666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21200ffa70 msgr2=0x7f212019e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:33.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.231+0000 7f2125a1e700 1 --2- 172.21.15.121:0/240698666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21200ffa70 0x7f212019e620 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2110000c00 tx=0x7f211000be90 comp rx=0 tx=0).stop 2024-04-05T22:05:33.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.231+0000 7f2125a1e700 1 -- 172.21.15.121:0/240698666 shutdown_connections 2024-04-05T22:05:33.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.231+0000 7f2125a1e700 1 --2- 172.21.15.121:0/240698666 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f210806cce0 0x7f210806f1a0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.231+0000 7f2125a1e700 1 --2- 172.21.15.121:0/240698666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21200ffa70 0x7f212019e620 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.231+0000 7f2125a1e700 1 --2- 172.21.15.121:0/240698666 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2120105a70 0x7f212019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.231+0000 7f2125a1e700 1 -- 172.21.15.121:0/240698666 >> 172.21.15.121:0/240698666 conn(0x7f21200fb560 msgr2=0x7f2120103580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:33.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.231+0000 7f2125a1e700 1 -- 172.21.15.121:0/240698666 shutdown_connections 2024-04-05T22:05:33.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.231+0000 7f2125a1e700 1 -- 172.21.15.121:0/240698666 wait complete. 2024-04-05T22:05:33.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.333+0000 7f6f756fb700 1 -- 172.21.15.121:0/436984285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f70100dd0 msgr2=0x7f6f701011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:33.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.333+0000 7f6f756fb700 1 --2- 172.21.15.121:0/436984285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f70100dd0 0x7f6f701011b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f6f64009910 tx=0x7f6f64009c20 comp rx=0 tx=0).stop 2024-04-05T22:05:33.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.334+0000 7f6f756fb700 1 -- 172.21.15.121:0/436984285 shutdown_connections 2024-04-05T22:05:33.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.334+0000 7f6f756fb700 1 --2- 172.21.15.121:0/436984285 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f70101780 0x7f6f701057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.334+0000 7f6f756fb700 1 --2- 172.21.15.121:0/436984285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f70100dd0 0x7f6f701011b0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.334+0000 7f6f756fb700 1 -- 172.21.15.121:0/436984285 >> 172.21.15.121:0/436984285 conn(0x7f6f700fc640 msgr2=0x7f6f700fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:33.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.334+0000 7f6f756fb700 1 -- 172.21.15.121:0/436984285 shutdown_connections 2024-04-05T22:05:33.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.334+0000 7f6f756fb700 1 -- 172.21.15.121:0/436984285 wait complete. 2024-04-05T22:05:33.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.334+0000 7f6f756fb700 1 Processor -- start 2024-04-05T22:05:33.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.334+0000 7f6f756fb700 1 -- start start 2024-04-05T22:05:33.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f756fb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f70100dd0 0x7f6f7019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:33.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f756fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f70101780 0x7f6f7019ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:33.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f756fb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7019d0c0 con 0x7f6f70101780 2024-04-05T22:05:33.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f756fb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70196570 con 0x7f6f70100dd0 2024-04-05T22:05:33.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f6e7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f70101780 0x7f6f7019ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:33.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f6e7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f70101780 0x7f6f7019ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33008/0 (socket says 172.21.15.121:33008) 2024-04-05T22:05:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f6e7fc700 1 -- 172.21.15.121:0/3539366234 learned_addr learned my addr 172.21.15.121:0/3539366234 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f6effd700 1 --2- 172.21.15.121:0/3539366234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f70100dd0 0x7f6f7019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f6e7fc700 1 -- 172.21.15.121:0/3539366234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f70100dd0 msgr2=0x7f6f7019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f6e7fc700 1 --2- 172.21.15.121:0/3539366234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f70100dd0 0x7f6f7019c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f6e7fc700 1 -- 172.21.15.121:0/3539366234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f640095f0 con 0x7f6f70101780 2024-04-05T22:05:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f6e7fc700 1 --2- 172.21.15.121:0/3539366234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f70101780 0x7f6f7019ca30 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000d900 tx=0x7f6f6000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f57fff700 1 -- 172.21.15.121:0/3539366234 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f60009900 con 0x7f6f70101780 2024-04-05T22:05:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f756fb700 1 -- 172.21.15.121:0/3539366234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f70196850 con 0x7f6f70101780 2024-04-05T22:05:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f57fff700 1 -- 172.21.15.121:0/3539366234 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f60010460 con 0x7f6f70101780 2024-04-05T22:05:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f57fff700 1 -- 172.21.15.121:0/3539366234 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f6000f5f0 con 0x7f6f70101780 2024-04-05T22:05:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.335+0000 7f6f756fb700 1 -- 172.21.15.121:0/3539366234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f70196da0 con 0x7f6f70101780 2024-04-05T22:05:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.336+0000 7f6f756fb700 1 -- 172.21.15.121:0/3539366234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f70109170 con 0x7f6f70101780 2024-04-05T22:05:33.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.337+0000 7f6f57fff700 1 -- 172.21.15.121:0/3539366234 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6f600105d0 con 0x7f6f70101780 2024-04-05T22:05:33.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.337+0000 7f6f57fff700 1 --2- 172.21.15.121:0/3539366234 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6f5806cce0 0x7f6f5806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:33.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.337+0000 7f6f57fff700 1 -- 172.21.15.121:0/3539366234 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6f6008c0e0 con 0x7f6f70101780 2024-04-05T22:05:33.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.337+0000 7f6f6effd700 1 --2- 172.21.15.121:0/3539366234 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6f5806cce0 0x7f6f5806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:33.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.338+0000 7f6f6effd700 1 --2- 172.21.15.121:0/3539366234 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6f5806cce0 0x7f6f5806f1a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f6f64006010 tx=0x7f6f6401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:33.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.342+0000 7f6f57fff700 1 -- 172.21.15.121:0/3539366234 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6f60059be0 con 0x7f6f70101780 2024-04-05T22:05:33.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:33 smithi121 ceph-mon[145859]: pgmap v1180: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:33.528 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:33 smithi188 ceph-mon[107793]: pgmap v1180: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:33.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.542+0000 7f6f756fb700 1 -- 172.21.15.121:0/3539366234 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f70196fc0 con 0x7f6f5806cce0 2024-04-05T22:05:33.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.553+0000 7f6f57fff700 1 -- 172.21.15.121:0/3539366234 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f6f70196fc0 con 0x7f6f5806cce0 2024-04-05T22:05:33.553 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T22:05:33.553 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (36m) 14m ago 47m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T22:05:33.553 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (48m) 14m ago 48m 9701k - 18.2.1 5be31c24972a e4ccec086b90 2024-04-05T22:05:33.553 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (46m) 14m ago 46m 9613k - 18.2.1 5be31c24972a db63d44687f1 2024-04-05T22:05:33.553 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (34m) 14m ago 48m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T22:05:33.553 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (34m) 14m ago 46m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T22:05:33.553 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (36m) 14m ago 47m 88.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T22:05:33.553 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (30m) 14m ago 41m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T22:05:33.554 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (30m) 14m ago 41m 21.6M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T22:05:33.554 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (30m) 14m ago 41m 22.3M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T22:05:33.554 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (14m) 14m ago 41m 14.9M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T22:05:33.554 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (38m) 14m ago 49m 514M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T22:05:33.554 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (37m) 14m ago 46m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T22:05:33.554 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (35m) 14m ago 49m 67.1M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T22:05:33.554 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (34m) 14m ago 46m 48.7M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T22:05:33.554 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (37m) 14m ago 47m 18.5M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T22:05:33.554 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (37m) 14m ago 46m 17.7M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T22:05:33.554 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (34m) 14m ago 45m 97.4M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T22:05:33.554 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (33m) 14m ago 45m 106M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T22:05:33.554 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (32m) 14m ago 44m 92.1M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T22:05:33.555 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (32m) 14m ago 44m 120M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T22:05:33.555 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (31m) 14m ago 43m 99.4M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T22:05:33.555 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (31m) 14m ago 43m 89.9M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T22:05:33.555 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (36m) 14m ago 47m 87.0M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T22:05:33.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.555+0000 7f6f756fb700 1 -- 172.21.15.121:0/3539366234 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6f5806cce0 msgr2=0x7f6f5806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:33.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.555+0000 7f6f756fb700 1 --2- 172.21.15.121:0/3539366234 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6f5806cce0 0x7f6f5806f1a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f6f64006010 tx=0x7f6f6401a040 comp rx=0 tx=0).stop 2024-04-05T22:05:33.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.555+0000 7f6f756fb700 1 -- 172.21.15.121:0/3539366234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f70101780 msgr2=0x7f6f7019ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:33.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.555+0000 7f6f756fb700 1 --2- 172.21.15.121:0/3539366234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f70101780 0x7f6f7019ca30 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000d900 tx=0x7f6f6000dc10 comp rx=0 tx=0).stop 2024-04-05T22:05:33.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.555+0000 7f6f756fb700 1 -- 172.21.15.121:0/3539366234 shutdown_connections 2024-04-05T22:05:33.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.555+0000 7f6f756fb700 1 --2- 172.21.15.121:0/3539366234 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6f5806cce0 0x7f6f5806f1a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.555+0000 7f6f756fb700 1 --2- 172.21.15.121:0/3539366234 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6f70100dd0 0x7f6f7019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.555+0000 7f6f756fb700 1 --2- 172.21.15.121:0/3539366234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f70101780 0x7f6f7019ca30 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.555+0000 7f6f756fb700 1 -- 172.21.15.121:0/3539366234 >> 172.21.15.121:0/3539366234 conn(0x7f6f700fc640 msgr2=0x7f6f700fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:33.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.555+0000 7f6f756fb700 1 -- 172.21.15.121:0/3539366234 shutdown_connections 2024-04-05T22:05:33.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.555+0000 7f6f756fb700 1 -- 172.21.15.121:0/3539366234 wait complete. 2024-04-05T22:05:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.657+0000 7f6cbd85c700 1 -- 172.21.15.121:0/3050128835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb8100dc0 msgr2=0x7f6cb81011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.657+0000 7f6cbd85c700 1 --2- 172.21.15.121:0/3050128835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb8100dc0 0x7f6cb81011a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f6cac009910 tx=0x7f6cac009c20 comp rx=0 tx=0).stop 2024-04-05T22:05:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.657+0000 7f6cbd85c700 1 -- 172.21.15.121:0/3050128835 shutdown_connections 2024-04-05T22:05:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.657+0000 7f6cbd85c700 1 --2- 172.21.15.121:0/3050128835 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cb8101770 0x7f6cb81057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.657+0000 7f6cbd85c700 1 --2- 172.21.15.121:0/3050128835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb8100dc0 0x7f6cb81011a0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.657+0000 7f6cbd85c700 1 -- 172.21.15.121:0/3050128835 >> 172.21.15.121:0/3050128835 conn(0x7f6cb80fc650 msgr2=0x7f6cb80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.657+0000 7f6cbd85c700 1 -- 172.21.15.121:0/3050128835 shutdown_connections 2024-04-05T22:05:33.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.657+0000 7f6cbd85c700 1 -- 172.21.15.121:0/3050128835 wait complete. 2024-04-05T22:05:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.657+0000 7f6cbd85c700 1 Processor -- start 2024-04-05T22:05:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cbd85c700 1 -- start start 2024-04-05T22:05:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cbd85c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb8100dc0 0x7f6cb819e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cbd85c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cb8101770 0x7f6cb819eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cbd85c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb819f1c0 con 0x7f6cb8100dc0 2024-04-05T22:05:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cbd85c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb8198670 con 0x7f6cb8101770 2024-04-05T22:05:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cb6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb8100dc0 0x7f6cb819e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cb6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb8100dc0 0x7f6cb819e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33028/0 (socket says 172.21.15.121:33028) 2024-04-05T22:05:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cb6ffd700 1 -- 172.21.15.121:0/1658837722 learned_addr learned my addr 172.21.15.121:0/1658837722 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cb67fc700 1 --2- 172.21.15.121:0/1658837722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cb8101770 0x7f6cb819eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:33.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cb6ffd700 1 -- 172.21.15.121:0/1658837722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cb8101770 msgr2=0x7f6cb819eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cb6ffd700 1 --2- 172.21.15.121:0/1658837722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cb8101770 0x7f6cb819eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cb6ffd700 1 -- 172.21.15.121:0/1658837722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cac0095f0 con 0x7f6cb8100dc0 2024-04-05T22:05:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cb67fc700 1 --2- 172.21.15.121:0/1658837722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cb8101770 0x7f6cb819eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:05:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cb6ffd700 1 --2- 172.21.15.121:0/1658837722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb8100dc0 0x7f6cb819e5f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f6cac006010 tx=0x7f6cac00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cbc85a700 1 -- 172.21.15.121:0/1658837722 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cac01d070 con 0x7f6cb8100dc0 2024-04-05T22:05:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cbd85c700 1 -- 172.21.15.121:0/1658837722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb81988f0 con 0x7f6cb8100dc0 2024-04-05T22:05:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cbc85a700 1 -- 172.21.15.121:0/1658837722 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6cac00f460 con 0x7f6cb8100dc0 2024-04-05T22:05:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cbc85a700 1 -- 172.21.15.121:0/1658837722 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cac021640 con 0x7f6cb8100dc0 2024-04-05T22:05:33.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.658+0000 7f6cbd85c700 1 -- 172.21.15.121:0/1658837722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb8198de0 con 0x7f6cb8100dc0 2024-04-05T22:05:33.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.660+0000 7f6cbd85c700 1 -- 172.21.15.121:0/1658837722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb8104f40 con 0x7f6cb8100dc0 2024-04-05T22:05:33.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.660+0000 7f6cbc85a700 1 -- 172.21.15.121:0/1658837722 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6cac00fa20 con 0x7f6cb8100dc0 2024-04-05T22:05:33.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.660+0000 7f6cbc85a700 1 --2- 172.21.15.121:0/1658837722 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6ca006cce0 0x7f6ca006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:33.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.660+0000 7f6cbc85a700 1 -- 172.21.15.121:0/1658837722 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6cac08d830 con 0x7f6cb8100dc0 2024-04-05T22:05:33.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.660+0000 7f6cb67fc700 1 --2- 172.21.15.121:0/1658837722 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6ca006cce0 0x7f6ca006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:33.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.661+0000 7f6cb67fc700 1 --2- 172.21.15.121:0/1658837722 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6ca006cce0 0x7f6ca006f1a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f6cb80fcdc0 tx=0x7f6ca800b5f0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:33.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.665+0000 7f6cbc85a700 1 -- 172.21.15.121:0/1658837722 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6cac05b330 con 0x7f6cb8100dc0 2024-04-05T22:05:33.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.940+0000 7f6cbd85c700 1 -- 172.21.15.121:0/1658837722 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6cb802cc70 con 0x7f6cb8100dc0 2024-04-05T22:05:33.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.940+0000 7f6cbc85a700 1 -- 172.21.15.121:0/1658837722 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f6cac057070 con 0x7f6cb8100dc0 2024-04-05T22:05:33.940 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:05:33.940 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T22:05:33.941 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:05:33.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.942+0000 7f6cbd85c700 1 -- 172.21.15.121:0/1658837722 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6ca006cce0 msgr2=0x7f6ca006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:33.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.942+0000 7f6cbd85c700 1 --2- 172.21.15.121:0/1658837722 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6ca006cce0 0x7f6ca006f1a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f6cb80fcdc0 tx=0x7f6ca800b5f0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.942+0000 7f6cbd85c700 1 -- 172.21.15.121:0/1658837722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb8100dc0 msgr2=0x7f6cb819e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:33.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.942+0000 7f6cbd85c700 1 --2- 172.21.15.121:0/1658837722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb8100dc0 0x7f6cb819e5f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f6cac006010 tx=0x7f6cac00be20 comp rx=0 tx=0).stop 2024-04-05T22:05:33.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.942+0000 7f6cbd85c700 1 -- 172.21.15.121:0/1658837722 shutdown_connections 2024-04-05T22:05:33.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.942+0000 7f6cbd85c700 1 --2- 172.21.15.121:0/1658837722 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6ca006cce0 0x7f6ca006f1a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.942+0000 7f6cbd85c700 1 --2- 172.21.15.121:0/1658837722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cb8100dc0 0x7f6cb819e5f0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.942+0000 7f6cbd85c700 1 --2- 172.21.15.121:0/1658837722 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6cb8101770 0x7f6cb819eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:33.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.942+0000 7f6cbd85c700 1 -- 172.21.15.121:0/1658837722 >> 172.21.15.121:0/1658837722 conn(0x7f6cb80fc650 msgr2=0x7f6cb80fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:33.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.942+0000 7f6cbd85c700 1 -- 172.21.15.121:0/1658837722 shutdown_connections 2024-04-05T22:05:33.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:33.943+0000 7f6cbd85c700 1 -- 172.21.15.121:0/1658837722 wait complete. 2024-04-05T22:05:34.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.045+0000 7f11618d8700 1 -- 172.21.15.121:0/1857602171 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f115c0ff400 msgr2=0x7f115c0ff7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:34.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.045+0000 7f11618d8700 1 --2- 172.21.15.121:0/1857602171 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f115c0ff400 0x7f115c0ff7e0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f1144009910 tx=0x7f1144009c20 comp rx=0 tx=0).stop 2024-04-05T22:05:34.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.045+0000 7f11618d8700 1 -- 172.21.15.121:0/1857602171 shutdown_connections 2024-04-05T22:05:34.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.045+0000 7f11618d8700 1 --2- 172.21.15.121:0/1857602171 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f115c0ffdb0 0x7f115c103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.045+0000 7f11618d8700 1 --2- 172.21.15.121:0/1857602171 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f115c0ff400 0x7f115c0ff7e0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.045+0000 7f11618d8700 1 -- 172.21.15.121:0/1857602171 >> 172.21.15.121:0/1857602171 conn(0x7f115c075e30 msgr2=0x7f115c0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:34.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.045+0000 7f11618d8700 1 -- 172.21.15.121:0/1857602171 shutdown_connections 2024-04-05T22:05:34.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.045+0000 7f11618d8700 1 -- 172.21.15.121:0/1857602171 wait complete. 2024-04-05T22:05:34.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.045+0000 7f11618d8700 1 Processor -- start 2024-04-05T22:05:34.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f11618d8700 1 -- start start 2024-04-05T22:05:34.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f11618d8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f115c0ff400 0x7f115c1955f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:34.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f11618d8700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f115c0ffdb0 0x7f115c195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:34.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f11618d8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c196210 con 0x7f115c0ff400 2024-04-05T22:05:34.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f11618d8700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c199fa0 con 0x7f115c0ffdb0 2024-04-05T22:05:34.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f115affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f115c0ff400 0x7f115c1955f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:34.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f115affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f115c0ff400 0x7f115c1955f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33042/0 (socket says 172.21.15.121:33042) 2024-04-05T22:05:34.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f115affd700 1 -- 172.21.15.121:0/2899475951 learned_addr learned my addr 172.21.15.121:0/2899475951 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:34.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f115a7fc700 1 --2- 172.21.15.121:0/2899475951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f115c0ffdb0 0x7f115c195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:34.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f115affd700 1 -- 172.21.15.121:0/2899475951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f115c0ffdb0 msgr2=0x7f115c195b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:34.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f115affd700 1 --2- 172.21.15.121:0/2899475951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f115c0ffdb0 0x7f115c195b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f115affd700 1 -- 172.21.15.121:0/2899475951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11440095f0 con 0x7f115c0ff400 2024-04-05T22:05:34.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f115a7fc700 1 --2- 172.21.15.121:0/2899475951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f115c0ffdb0 0x7f115c195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:05:34.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f115affd700 1 --2- 172.21.15.121:0/2899475951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f115c0ff400 0x7f115c1955f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f1144006010 tx=0x7f114400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:34.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.046+0000 7f11608d6700 1 -- 172.21.15.121:0/2899475951 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f114401d070 con 0x7f115c0ff400 2024-04-05T22:05:34.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.047+0000 7f11608d6700 1 -- 172.21.15.121:0/2899475951 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f114400f460 con 0x7f115c0ff400 2024-04-05T22:05:34.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.047+0000 7f11608d6700 1 -- 172.21.15.121:0/2899475951 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1144021620 con 0x7f115c0ff400 2024-04-05T22:05:34.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.047+0000 7f11618d8700 1 -- 172.21.15.121:0/2899475951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f115c19a220 con 0x7f115c0ff400 2024-04-05T22:05:34.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.047+0000 7f11618d8700 1 -- 172.21.15.121:0/2899475951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f115c19a6c0 con 0x7f115c0ff400 2024-04-05T22:05:34.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.047+0000 7f11618d8700 1 -- 172.21.15.121:0/2899475951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f115c1077a0 con 0x7f115c0ff400 2024-04-05T22:05:34.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.048+0000 7f11608d6700 1 -- 172.21.15.121:0/2899475951 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f114400fa40 con 0x7f115c0ff400 2024-04-05T22:05:34.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.048+0000 7f11608d6700 1 --2- 172.21.15.121:0/2899475951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f114806cce0 0x7f114806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:34.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.048+0000 7f11608d6700 1 -- 172.21.15.121:0/2899475951 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f114408d790 con 0x7f115c0ff400 2024-04-05T22:05:34.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.048+0000 7f115a7fc700 1 --2- 172.21.15.121:0/2899475951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f114806cce0 0x7f114806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:34.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.049+0000 7f115a7fc700 1 --2- 172.21.15.121:0/2899475951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f114806cce0 0x7f114806f1a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f115c196c10 tx=0x7f114c00b2c0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:34.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.053+0000 7f11608d6700 1 -- 172.21.15.121:0/2899475951 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1144057800 con 0x7f115c0ff400 2024-04-05T22:05:34.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.287+0000 7f11618d8700 1 -- 172.21.15.121:0/2899475951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f115c02cc70 con 0x7f115c0ff400 2024-04-05T22:05:34.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.288+0000 7f11608d6700 1 -- 172.21.15.121:0/2899475951 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 32 v32) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f114405ae20 con 0x7f115c0ff400 2024-04-05T22:05:34.288 INFO:teuthology.orchestra.run.smithi121.stdout:e32 2024-04-05T22:05:34.288 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T22:05:34.288 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:05:34.288 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T22:05:34.288 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:05:34.288 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T22:05:34.288 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T22:05:34.288 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 23 2024-04-05T22:05:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-04-05T22:05:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T22:05:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T21:34:37.155847+0000 2024-04-05T22:05:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T22:05:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T22:05:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T22:05:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T22:05:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T22:05:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T22:05:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T22:05:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T22:05:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T22:05:34.289 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout:up {} 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout:failed 0,1 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:05:34.290 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:05:34.291 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-04-05T22:05:34.291 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:05:34.291 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{-1:34360} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:05:34.291 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{-1:34364} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:05:34.291 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{-1:44297} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:05:34.291 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{-1:44827} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:05:34.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.290+0000 7f11618d8700 1 -- 172.21.15.121:0/2899475951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f114806cce0 msgr2=0x7f114806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:34.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.290+0000 7f11618d8700 1 --2- 172.21.15.121:0/2899475951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f114806cce0 0x7f114806f1a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f115c196c10 tx=0x7f114c00b2c0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.290+0000 7f11618d8700 1 -- 172.21.15.121:0/2899475951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f115c0ff400 msgr2=0x7f115c1955f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:34.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.290+0000 7f11618d8700 1 --2- 172.21.15.121:0/2899475951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f115c0ff400 0x7f115c1955f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f1144006010 tx=0x7f114400be20 comp rx=0 tx=0).stop 2024-04-05T22:05:34.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.290+0000 7f11618d8700 1 -- 172.21.15.121:0/2899475951 shutdown_connections 2024-04-05T22:05:34.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.290+0000 7f11618d8700 1 --2- 172.21.15.121:0/2899475951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f114806cce0 0x7f114806f1a0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.290+0000 7f11618d8700 1 --2- 172.21.15.121:0/2899475951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f115c0ff400 0x7f115c1955f0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.290+0000 7f11618d8700 1 --2- 172.21.15.121:0/2899475951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f115c0ffdb0 0x7f115c195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.290+0000 7f11618d8700 1 -- 172.21.15.121:0/2899475951 >> 172.21.15.121:0/2899475951 conn(0x7f115c075e30 msgr2=0x7f115c103170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:34.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.290+0000 7f11618d8700 1 -- 172.21.15.121:0/2899475951 shutdown_connections 2024-04-05T22:05:34.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.290+0000 7f11618d8700 1 -- 172.21.15.121:0/2899475951 wait complete. 2024-04-05T22:05:34.293 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T22:05:34.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.394+0000 7f4ea106f700 1 -- 172.21.15.121:0/282506668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9c101790 msgr2=0x7f4e9c1057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:34.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.394+0000 7f4ea106f700 1 --2- 172.21.15.121:0/282506668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9c101790 0x7f4e9c1057e0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c009910 tx=0x7f4e8c009c20 comp rx=0 tx=0).stop 2024-04-05T22:05:34.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.394+0000 7f4ea106f700 1 -- 172.21.15.121:0/282506668 shutdown_connections 2024-04-05T22:05:34.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.394+0000 7f4ea106f700 1 --2- 172.21.15.121:0/282506668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9c101790 0x7f4e9c1057e0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.394+0000 7f4ea106f700 1 --2- 172.21.15.121:0/282506668 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e9c100de0 0x7f4e9c1011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.394+0000 7f4ea106f700 1 -- 172.21.15.121:0/282506668 >> 172.21.15.121:0/282506668 conn(0x7f4e9c0fc6b0 msgr2=0x7f4e9c0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:34.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.394+0000 7f4ea106f700 1 -- 172.21.15.121:0/282506668 shutdown_connections 2024-04-05T22:05:34.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.394+0000 7f4ea106f700 1 -- 172.21.15.121:0/282506668 wait complete. 2024-04-05T22:05:34.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4ea106f700 1 Processor -- start 2024-04-05T22:05:34.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4ea106f700 1 -- start start 2024-04-05T22:05:34.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4ea106f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e9c100de0 0x7f4e9c196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:34.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4ea106f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9c101790 0x7f4e9c196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4ea106f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c197290 con 0x7f4e9c101790 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4ea106f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c19b020 con 0x7f4e9c100de0 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4e9a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9c101790 0x7f4e9c196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4e9a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9c101790 0x7f4e9c196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45200/0 (socket says 172.21.15.121:45200) 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4e9a59c700 1 -- 172.21.15.121:0/779006897 learned_addr learned my addr 172.21.15.121:0/779006897 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4e9ad9d700 1 --2- 172.21.15.121:0/779006897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e9c100de0 0x7f4e9c196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4e9a59c700 1 -- 172.21.15.121:0/779006897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e9c100de0 msgr2=0x7f4e9c196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4e9a59c700 1 --2- 172.21.15.121:0/779006897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e9c100de0 0x7f4e9c196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4e9a59c700 1 -- 172.21.15.121:0/779006897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c0095f0 con 0x7f4e9c101790 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.395+0000 7f4e9a59c700 1 --2- 172.21.15.121:0/779006897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9c101790 0x7f4e9c196bb0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c00bd70 tx=0x7f4e8c00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.396+0000 7f4e93fff700 1 -- 172.21.15.121:0/779006897 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e8c01d070 con 0x7f4e9c101790 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.396+0000 7f4ea106f700 1 -- 172.21.15.121:0/779006897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9c19b2a0 con 0x7f4e9c101790 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.396+0000 7f4e93fff700 1 -- 172.21.15.121:0/779006897 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e8c00f460 con 0x7f4e9c101790 2024-04-05T22:05:34.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.396+0000 7f4e93fff700 1 -- 172.21.15.121:0/779006897 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e8c021640 con 0x7f4e9c101790 2024-04-05T22:05:34.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.396+0000 7f4ea106f700 1 -- 172.21.15.121:0/779006897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9c19b790 con 0x7f4e9c101790 2024-04-05T22:05:34.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.397+0000 7f4ea106f700 1 -- 172.21.15.121:0/779006897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e9c1091e0 con 0x7f4e9c101790 2024-04-05T22:05:34.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.397+0000 7f4e93fff700 1 -- 172.21.15.121:0/779006897 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4e8c00f9e0 con 0x7f4e9c101790 2024-04-05T22:05:34.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.398+0000 7f4e93fff700 1 --2- 172.21.15.121:0/779006897 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4e8806cce0 0x7f4e8806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:34.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.398+0000 7f4e93fff700 1 -- 172.21.15.121:0/779006897 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4e8c08db70 con 0x7f4e9c101790 2024-04-05T22:05:34.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.398+0000 7f4e9ad9d700 1 --2- 172.21.15.121:0/779006897 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4e8806cce0 0x7f4e8806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:34.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.398+0000 7f4e9ad9d700 1 --2- 172.21.15.121:0/779006897 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4e8806cce0 0x7f4e8806f1a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4e84005950 tx=0x7f4e8400b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:34.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.402+0000 7f4e93fff700 1 -- 172.21.15.121:0/779006897 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4e8c057bc0 con 0x7f4e9c101790 2024-04-05T22:05:34.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:34 smithi121 ceph-mon[145859]: from='client.35942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:34.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:34 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1658837722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:34.528 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:34 smithi188 ceph-mon[107793]: from='client.35942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:34.528 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:34 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1658837722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:34.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.614+0000 7f4ea106f700 1 -- 172.21.15.121:0/779006897 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e9c02cdb0 con 0x7f4e8806cce0 2024-04-05T22:05:34.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.615+0000 7f4e93fff700 1 -- 172.21.15.121:0/779006897 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4e9c02cdb0 con 0x7f4e8806cce0 2024-04-05T22:05:34.615 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:05:34.615 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da", 2024-04-05T22:05:34.615 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": true, 2024-04-05T22:05:34.615 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-05T22:05:34.615 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [ 2024-04-05T22:05:34.615 INFO:teuthology.orchestra.run.smithi121.stdout: "mon", 2024-04-05T22:05:34.615 INFO:teuthology.orchestra.run.smithi121.stdout: "mds", 2024-04-05T22:05:34.615 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-04-05T22:05:34.615 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr", 2024-04-05T22:05:34.615 INFO:teuthology.orchestra.run.smithi121.stdout: "osd" 2024-04-05T22:05:34.615 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-04-05T22:05:34.615 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": "16/23 daemons upgraded", 2024-04-05T22:05:34.616 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "Currently upgrading mds daemons", 2024-04-05T22:05:34.616 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T22:05:34.616 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:05:34.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.617+0000 7f4ea106f700 1 -- 172.21.15.121:0/779006897 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4e8806cce0 msgr2=0x7f4e8806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:34.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.617+0000 7f4ea106f700 1 --2- 172.21.15.121:0/779006897 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4e8806cce0 0x7f4e8806f1a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4e84005950 tx=0x7f4e8400b410 comp rx=0 tx=0).stop 2024-04-05T22:05:34.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.617+0000 7f4ea106f700 1 -- 172.21.15.121:0/779006897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9c101790 msgr2=0x7f4e9c196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:34.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.617+0000 7f4ea106f700 1 --2- 172.21.15.121:0/779006897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9c101790 0x7f4e9c196bb0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c00bd70 tx=0x7f4e8c00bda0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.617+0000 7f4ea106f700 1 -- 172.21.15.121:0/779006897 shutdown_connections 2024-04-05T22:05:34.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.617+0000 7f4ea106f700 1 --2- 172.21.15.121:0/779006897 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4e8806cce0 0x7f4e8806f1a0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.617+0000 7f4ea106f700 1 --2- 172.21.15.121:0/779006897 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4e9c100de0 0x7f4e9c196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.617+0000 7f4ea106f700 1 --2- 172.21.15.121:0/779006897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9c101790 0x7f4e9c196bb0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.617+0000 7f4ea106f700 1 -- 172.21.15.121:0/779006897 >> 172.21.15.121:0/779006897 conn(0x7f4e9c0fc6b0 msgr2=0x7f4e9c0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:34.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.617+0000 7f4ea106f700 1 -- 172.21.15.121:0/779006897 shutdown_connections 2024-04-05T22:05:34.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.617+0000 7f4ea106f700 1 -- 172.21.15.121:0/779006897 wait complete. 2024-04-05T22:05:34.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.719+0000 7f168e786700 1 -- 172.21.15.121:0/2642098810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1688101790 msgr2=0x7f16881057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:34.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.719+0000 7f168e786700 1 --2- 172.21.15.121:0/2642098810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1688101790 0x7f16881057e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f167c009910 tx=0x7f167c009c20 comp rx=0 tx=0).stop 2024-04-05T22:05:34.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.719+0000 7f168e786700 1 -- 172.21.15.121:0/2642098810 shutdown_connections 2024-04-05T22:05:34.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.719+0000 7f168e786700 1 --2- 172.21.15.121:0/2642098810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1688101790 0x7f16881057e0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.719+0000 7f168e786700 1 --2- 172.21.15.121:0/2642098810 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1688100de0 0x7f16881011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.719+0000 7f168e786700 1 -- 172.21.15.121:0/2642098810 >> 172.21.15.121:0/2642098810 conn(0x7f16880fc670 msgr2=0x7f16880fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:34.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.719+0000 7f168e786700 1 -- 172.21.15.121:0/2642098810 shutdown_connections 2024-04-05T22:05:34.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.719+0000 7f168e786700 1 -- 172.21.15.121:0/2642098810 wait complete. 2024-04-05T22:05:34.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.720+0000 7f168e786700 1 Processor -- start 2024-04-05T22:05:34.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.720+0000 7f168e786700 1 -- start start 2024-04-05T22:05:34.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.720+0000 7f168e786700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1688100de0 0x7f168819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:34.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.720+0000 7f168e786700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1688101790 0x7f168819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:34.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.720+0000 7f168e786700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168819f230 con 0x7f1688101790 2024-04-05T22:05:34.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.720+0000 7f168e786700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16881986e0 con 0x7f1688100de0 2024-04-05T22:05:34.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.720+0000 7f16877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1688101790 0x7f168819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:34.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.720+0000 7f16877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1688101790 0x7f168819eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45226/0 (socket says 172.21.15.121:45226) 2024-04-05T22:05:34.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.720+0000 7f16877fe700 1 -- 172.21.15.121:0/3294025202 learned_addr learned my addr 172.21.15.121:0/3294025202 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:05:34.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.720+0000 7f1687fff700 1 --2- 172.21.15.121:0/3294025202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1688100de0 0x7f168819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:34.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.720+0000 7f16877fe700 1 -- 172.21.15.121:0/3294025202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1688100de0 msgr2=0x7f168819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:34.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.720+0000 7f16877fe700 1 --2- 172.21.15.121:0/3294025202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1688100de0 0x7f168819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:34.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.720+0000 7f16877fe700 1 -- 172.21.15.121:0/3294025202 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c0095f0 con 0x7f1688101790 2024-04-05T22:05:34.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.721+0000 7f16877fe700 1 --2- 172.21.15.121:0/3294025202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1688101790 0x7f168819eba0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f167c000c00 tx=0x7f167c00ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:34.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.721+0000 7f16857fa700 1 -- 172.21.15.121:0/3294025202 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f167c01d070 con 0x7f1688101790 2024-04-05T22:05:34.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.721+0000 7f16857fa700 1 -- 172.21.15.121:0/3294025202 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f167c00bed0 con 0x7f1688101790 2024-04-05T22:05:34.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.721+0000 7f168e786700 1 -- 172.21.15.121:0/3294025202 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1688198960 con 0x7f1688101790 2024-04-05T22:05:34.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.721+0000 7f16857fa700 1 -- 172.21.15.121:0/3294025202 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f167c00fd90 con 0x7f1688101790 2024-04-05T22:05:34.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.721+0000 7f168e786700 1 -- 172.21.15.121:0/3294025202 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1688198df0 con 0x7f1688101790 2024-04-05T22:05:34.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.722+0000 7f168e786700 1 -- 172.21.15.121:0/3294025202 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1688104f60 con 0x7f1688101790 2024-04-05T22:05:34.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.722+0000 7f16857fa700 1 -- 172.21.15.121:0/3294025202 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f167c021410 con 0x7f1688101790 2024-04-05T22:05:34.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.722+0000 7f16857fa700 1 --2- 172.21.15.121:0/3294025202 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f167006cce0 0x7f167006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:05:34.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.723+0000 7f16857fa700 1 -- 172.21.15.121:0/3294025202 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f167c08d780 con 0x7f1688101790 2024-04-05T22:05:34.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.723+0000 7f1687fff700 1 --2- 172.21.15.121:0/3294025202 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f167006cce0 0x7f167006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:05:34.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.723+0000 7f1687fff700 1 --2- 172.21.15.121:0/3294025202 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f167006cce0 0x7f167006f1a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1678009c60 tx=0x7f1678009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:05:34.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.727+0000 7f16857fa700 1 -- 172.21.15.121:0/3294025202 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f167c05b280 con 0x7f1688101790 2024-04-05T22:05:34.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.996+0000 7f168e786700 1 -- 172.21.15.121:0/3294025202 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f168804ea90 con 0x7f1688101790 2024-04-05T22:05:34.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.997+0000 7f16857fa700 1 -- 172.21.15.121:0/3294025202 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f167c08be60 con 0x7f1688101790 2024-04-05T22:05:34.997 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-04-05T22:05:34.997 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-04-05T22:05:34.997 INFO:teuthology.orchestra.run.smithi121.stdout: host smithi188 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi188/config 2024-04-05T22:05:34.997 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/podman: timeout after 895 seconds 2024-04-05T22:05:34.997 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:05:34.997 INFO:teuthology.orchestra.run.smithi121.stdout:Traceback (most recent call last): 2024-04-05T22:05:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-04-05T22:05:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: "__main__", mod_spec) 2024-04-05T22:05:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-04-05T22:05:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: exec(code, run_globals) 2024-04-05T22:05:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10889, in 2024-04-05T22:05:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 10877, in main 2024-04-05T22:05:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2576, in _infer_config 2024-04-05T22:05:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2492, in _infer_fsid 2024-04-05T22:05:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2604, in _infer_image 2024-04-05T22:05:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2479, in _validate_fsid 2024-04-05T22:05:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 7145, in command_ceph_volume 2024-04-05T22:05:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: File "/tmp/tmp8bkopsn3.cephadm.build/__main__.py", line 2267, in call_throws 2024-04-05T22:05:34.998 INFO:teuthology.orchestra.run.smithi121.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da -e NODE_NAME=smithi188 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/run/ceph:z -v /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4:/var/log/ceph:z -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp4hqzyppq:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:e639f54c391e09c31343b40aff3a6933ac9093b0263868a228dffb17dcb264da inventory --format=json-pretty --filter-for-batch 2024-04-05T22:05:34.999 INFO:teuthology.orchestra.run.smithi121.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-05T22:05:34.999 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is degraded 2024-04-05T22:05:34.999 INFO:teuthology.orchestra.run.smithi121.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-04-05T22:05:34.999 INFO:teuthology.orchestra.run.smithi121.stdout: fs cephfs is offline because no MDS is active for it. 2024-04-05T22:05:34.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.999+0000 7f168e786700 1 -- 172.21.15.121:0/3294025202 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f167006cce0 msgr2=0x7f167006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.999+0000 7f168e786700 1 --2- 172.21.15.121:0/3294025202 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f167006cce0 0x7f167006f1a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1678009c60 tx=0x7f1678009260 comp rx=0 tx=0).stop 2024-04-05T22:05:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.999+0000 7f168e786700 1 -- 172.21.15.121:0/3294025202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1688101790 msgr2=0x7f168819eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:05:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.999+0000 7f168e786700 1 --2- 172.21.15.121:0/3294025202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1688101790 0x7f168819eba0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f167c000c00 tx=0x7f167c00ba70 comp rx=0 tx=0).stop 2024-04-05T22:05:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.999+0000 7f168e786700 1 -- 172.21.15.121:0/3294025202 shutdown_connections 2024-04-05T22:05:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.999+0000 7f168e786700 1 --2- 172.21.15.121:0/3294025202 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f167006cce0 0x7f167006f1a0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.999+0000 7f168e786700 1 --2- 172.21.15.121:0/3294025202 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1688100de0 0x7f168819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.999+0000 7f168e786700 1 --2- 172.21.15.121:0/3294025202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1688101790 0x7f168819eba0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:05:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.999+0000 7f168e786700 1 -- 172.21.15.121:0/3294025202 >> 172.21.15.121:0/3294025202 conn(0x7f16880fc670 msgr2=0x7f16880fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:05:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.999+0000 7f168e786700 1 -- 172.21.15.121:0/3294025202 shutdown_connections 2024-04-05T22:05:35.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:05:34.999+0000 7f168e786700 1 -- 172.21.15.121:0/3294025202 wait complete. 2024-04-05T22:05:35.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:35 smithi121 ceph-mon[145859]: from='client.35946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:35.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:35 smithi121 ceph-mon[145859]: from='client.35950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:35.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:35 smithi121 ceph-mon[145859]: pgmap v1181: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:35.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:35 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2899475951' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:05:35.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:35 smithi121 ceph-mon[145859]: from='client.35962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:35.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:35 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3294025202' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:05:35.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:05:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:05:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:35 smithi121 ceph-mon[145859]: pgmap v1182: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:35 smithi121 ceph-mon[145859]: pgmap v1183: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:35.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:35 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:35 smithi188 ceph-mon[107793]: from='client.35946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:35 smithi188 ceph-mon[107793]: from='client.35950 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:35 smithi188 ceph-mon[107793]: pgmap v1181: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:35 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2899475951' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:05:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:35 smithi188 ceph-mon[107793]: from='client.35962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:05:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:35 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3294025202' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:05:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:05:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:05:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:35 smithi188 ceph-mon[107793]: pgmap v1182: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:35 smithi188 ceph-mon[107793]: pgmap v1183: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail 2024-04-05T22:05:35.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:35 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:36.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:05:36.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:36.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:36.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:36.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: Upgrade: Setting container_image for all mds 2024-04-05T22:05:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T22:05:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi121.gxheqt"}]': finished 2024-04-05T22:05:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T22:05:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi121.ifoxmw"}]': finished 2024-04-05T22:05:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T22:05:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi188.bjmisu"}]': finished 2024-04-05T22:05:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi188.latnft"}]: dispatch 2024-04-05T22:05:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi188.latnft"}]': finished 2024-04-05T22:05:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: Upgrade: Setting filesystem cephfs Joinable 2024-04-05T22:05:36.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:36 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-05T22:05:36.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:05:36.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:36.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:36.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:36.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:36.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:36.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: Upgrade: Setting container_image for all mds 2024-04-05T22:05:36.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:36.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi121.gxheqt"}]: dispatch 2024-04-05T22:05:36.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi121.gxheqt"}]': finished 2024-04-05T22:05:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi121.ifoxmw"}]: dispatch 2024-04-05T22:05:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi121.ifoxmw"}]': finished 2024-04-05T22:05:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi188.bjmisu"}]: dispatch 2024-04-05T22:05:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi188.bjmisu"}]': finished 2024-04-05T22:05:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi188.latnft"}]: dispatch 2024-04-05T22:05:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi188.latnft"}]': finished 2024-04-05T22:05:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: Upgrade: Setting filesystem cephfs Joinable 2024-04-05T22:05:36.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:36 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-05T22:05:37.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-05T22:05:37.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:37 smithi121 ceph-mon[145859]: Standby daemon mds.cephfs.smithi121.gxheqt assigned to filesystem cephfs as rank 0 2024-04-05T22:05:37.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:37 smithi121 ceph-mon[145859]: Standby daemon mds.cephfs.smithi121.ifoxmw assigned to filesystem cephfs as rank 1 2024-04-05T22:05:37.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:37 smithi121 ceph-mon[145859]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-05T22:05:37.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:37 smithi121 ceph-mon[145859]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T22:05:37.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:37 smithi121 ceph-mon[145859]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-05T22:05:37.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:37 smithi121 ceph-mon[145859]: fsmap cephfs:2/2 {0=cephfs.smithi121.gxheqt=up:replay,1=cephfs.smithi121.ifoxmw=up:replay} 2 up:standby 2024-04-05T22:05:37.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:37 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-05T22:05:37.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:37 smithi121 ceph-mon[145859]: pgmap v1184: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1 op/s 2024-04-05T22:05:37.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-05T22:05:37.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:37 smithi188 ceph-mon[107793]: Standby daemon mds.cephfs.smithi121.gxheqt assigned to filesystem cephfs as rank 0 2024-04-05T22:05:37.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:37 smithi188 ceph-mon[107793]: Standby daemon mds.cephfs.smithi121.ifoxmw assigned to filesystem cephfs as rank 1 2024-04-05T22:05:37.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:37 smithi188 ceph-mon[107793]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-05T22:05:37.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:37 smithi188 ceph-mon[107793]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-04-05T22:05:37.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:37 smithi188 ceph-mon[107793]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-04-05T22:05:37.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:37 smithi188 ceph-mon[107793]: fsmap cephfs:2/2 {0=cephfs.smithi121.gxheqt=up:replay,1=cephfs.smithi121.ifoxmw=up:replay} 2 up:standby 2024-04-05T22:05:37.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:37 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-04-05T22:05:37.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:37 smithi188 ceph-mon[107793]: pgmap v1184: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.8 MiB/s rd, 1 op/s 2024-04-05T22:05:38.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:38 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-05T22:05:38.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:38 smithi121 ceph-mon[145859]: fsmap cephfs:2/2 {0=cephfs.smithi121.gxheqt=up:replay,1=cephfs.smithi121.ifoxmw=up:replay} 2 up:standby 2024-04-05T22:05:38.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:38 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:38.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:38 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:38.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:38 smithi121 ceph-mon[145859]: Upgrade: Setting container_image for all rgw 2024-04-05T22:05:38.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:38 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:38.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:38 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:38.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:38 smithi121 ceph-mon[145859]: Upgrade: Setting container_image for all rbd-mirror 2024-04-05T22:05:38.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:38 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:38.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:38 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:38.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:38 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-04-05T22:05:38.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:38 smithi188 ceph-mon[107793]: fsmap cephfs:2/2 {0=cephfs.smithi121.gxheqt=up:replay,1=cephfs.smithi121.ifoxmw=up:replay} 2 up:standby 2024-04-05T22:05:38.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:38 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:38.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:38 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:38.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:38 smithi188 ceph-mon[107793]: Upgrade: Setting container_image for all rgw 2024-04-05T22:05:38.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:38 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:38.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:38 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:38.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:38 smithi188 ceph-mon[107793]: Upgrade: Setting container_image for all rbd-mirror 2024-04-05T22:05:38.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:38 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:38.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:38 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:05:40.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:40 smithi121 ceph-mon[145859]: pgmap v1185: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 7.9 MiB/s rd, 3 op/s 2024-04-05T22:05:40.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:40 smithi121 ceph-mon[145859]: Upgrade: Updating ceph-exporter.smithi121 (1/2) 2024-04-05T22:05:40.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:40 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:40.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:40 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi121", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-05T22:05:40.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:40 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:05:40.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:40 smithi121 ceph-mon[145859]: Deploying daemon ceph-exporter.smithi121 on smithi121 2024-04-05T22:05:40.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:40 smithi188 ceph-mon[107793]: pgmap v1185: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 7.9 MiB/s rd, 3 op/s 2024-04-05T22:05:40.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:40 smithi188 ceph-mon[107793]: Upgrade: Updating ceph-exporter.smithi121 (1/2) 2024-04-05T22:05:40.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:40 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:40.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:40 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi121", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-05T22:05:40.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:40 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:05:40.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:40 smithi188 ceph-mon[107793]: Deploying daemon ceph-exporter.smithi121 on smithi121 2024-04-05T22:05:42.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:42 smithi121 ceph-mon[145859]: pgmap v1186: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 7 op/s 2024-04-05T22:05:42.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:42 smithi188 ceph-mon[107793]: pgmap v1186: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 7 op/s 2024-04-05T22:05:43.364 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:43 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] up:resolve 2024-04-05T22:05:43.364 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:43 smithi121 ceph-mon[145859]: fsmap cephfs:2/2 {0=cephfs.smithi121.gxheqt=up:replay,1=cephfs.smithi121.ifoxmw=up:resolve} 2 up:standby 2024-04-05T22:05:43.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:43 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] up:resolve 2024-04-05T22:05:43.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:43 smithi188 ceph-mon[107793]: fsmap cephfs:2/2 {0=cephfs.smithi121.gxheqt=up:replay,1=cephfs.smithi121.ifoxmw=up:resolve} 2 up:standby 2024-04-05T22:05:44.413 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:44 smithi121 ceph-mon[145859]: pgmap v1187: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 9 op/s 2024-04-05T22:05:44.528 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:44 smithi188 ceph-mon[107793]: pgmap v1187: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 9 op/s 2024-04-05T22:05:46.257 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:46 smithi188 ceph-mon[107793]: pgmap v1188: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 9 op/s 2024-04-05T22:05:46.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:46 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] up:resolve 2024-04-05T22:05:46.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:46 smithi188 ceph-mon[107793]: fsmap cephfs:2/2 {0=cephfs.smithi121.gxheqt=up:resolve,1=cephfs.smithi121.ifoxmw=up:resolve} 2 up:standby 2024-04-05T22:05:46.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:46 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:46.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:46 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:05:46.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:46 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:46.258 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:46 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:46.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:46 smithi121 ceph-mon[145859]: pgmap v1188: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 26 MiB/s rd, 9 op/s 2024-04-05T22:05:46.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:46 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] up:resolve 2024-04-05T22:05:46.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:46 smithi121 ceph-mon[145859]: fsmap cephfs:2/2 {0=cephfs.smithi121.gxheqt=up:resolve,1=cephfs.smithi121.ifoxmw=up:resolve} 2 up:standby 2024-04-05T22:05:46.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:46 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:46.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:46 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:05:46.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:46 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:46.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:46 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:47.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:47 smithi188 ceph-mon[107793]: reconnect by client.14528 192.168.0.1:0/1232807809 after 0.000999985 2024-04-05T22:05:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:47 smithi188 ceph-mon[107793]: reconnect by client.14528 192.168.0.1:0/1232807809 after 0.000999985 2024-04-05T22:05:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:47 smithi188 ceph-mon[107793]: reconnect by client.24293 192.168.0.1:0/220997234 after 0.000999985 2024-04-05T22:05:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:47 smithi188 ceph-mon[107793]: reconnect by client.24293 192.168.0.1:0/220997234 after 0.000999985 2024-04-05T22:05:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:47 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] up:reconnect 2024-04-05T22:05:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:47 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] up:reconnect 2024-04-05T22:05:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:47 smithi188 ceph-mon[107793]: fsmap cephfs:2/2 {0=cephfs.smithi121.gxheqt=up:reconnect,1=cephfs.smithi121.ifoxmw=up:reconnect} 2 up:standby 2024-04-05T22:05:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:47 smithi188 ceph-mon[107793]: Upgrade: Updating ceph-exporter.smithi188 (2/2) 2024-04-05T22:05:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:47 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:47 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi188", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-05T22:05:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:47 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:05:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:47 smithi188 ceph-mon[107793]: Deploying daemon ceph-exporter.smithi188 on smithi188 2024-04-05T22:05:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:47 smithi188 ceph-mon[107793]: pgmap v1189: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 7 op/s 2024-04-05T22:05:47.339 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:47 smithi121 ceph-mon[145859]: reconnect by client.14528 192.168.0.1:0/1232807809 after 0.000999985 2024-04-05T22:05:47.339 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:47 smithi121 ceph-mon[145859]: reconnect by client.14528 192.168.0.1:0/1232807809 after 0.000999985 2024-04-05T22:05:47.339 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:47 smithi121 ceph-mon[145859]: reconnect by client.24293 192.168.0.1:0/220997234 after 0.000999985 2024-04-05T22:05:47.339 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:47 smithi121 ceph-mon[145859]: reconnect by client.24293 192.168.0.1:0/220997234 after 0.000999985 2024-04-05T22:05:47.339 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:47 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] up:reconnect 2024-04-05T22:05:47.339 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:47 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] up:reconnect 2024-04-05T22:05:47.339 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:47 smithi121 ceph-mon[145859]: fsmap cephfs:2/2 {0=cephfs.smithi121.gxheqt=up:reconnect,1=cephfs.smithi121.ifoxmw=up:reconnect} 2 up:standby 2024-04-05T22:05:47.340 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:47 smithi121 ceph-mon[145859]: Upgrade: Updating ceph-exporter.smithi188 (2/2) 2024-04-05T22:05:47.340 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:47 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:47.340 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:47 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi188", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-04-05T22:05:47.340 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:47 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:05:47.340 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:47 smithi121 ceph-mon[145859]: Deploying daemon ceph-exporter.smithi188 on smithi188 2024-04-05T22:05:47.340 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:47 smithi121 ceph-mon[145859]: pgmap v1189: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 7 op/s 2024-04-05T22:05:48.378 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:48 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] up:rejoin 2024-04-05T22:05:48.379 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:48 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] up:rejoin 2024-04-05T22:05:48.379 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:48 smithi188 ceph-mon[107793]: fsmap cephfs:2/2 {0=cephfs.smithi121.gxheqt=up:rejoin,1=cephfs.smithi121.ifoxmw=up:rejoin} 2 up:standby 2024-04-05T22:05:48.379 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:48 smithi188 ceph-mon[107793]: daemon mds.cephfs.smithi121.gxheqt is now active in filesystem cephfs as rank 0 2024-04-05T22:05:48.379 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:48 smithi188 ceph-mon[107793]: daemon mds.cephfs.smithi121.ifoxmw is now active in filesystem cephfs as rank 1 2024-04-05T22:05:48.379 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:48 smithi188 ceph-mon[107793]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-05T22:05:48.379 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:48 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] up:active 2024-04-05T22:05:48.379 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:48 smithi188 ceph-mon[107793]: mds.? [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] up:active 2024-04-05T22:05:48.379 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:48 smithi188 ceph-mon[107793]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi121.ifoxmw=up:active} 2 up:standby-replay 2024-04-05T22:05:48.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:48 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] up:rejoin 2024-04-05T22:05:48.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:48 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] up:rejoin 2024-04-05T22:05:48.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:48 smithi121 ceph-mon[145859]: fsmap cephfs:2/2 {0=cephfs.smithi121.gxheqt=up:rejoin,1=cephfs.smithi121.ifoxmw=up:rejoin} 2 up:standby 2024-04-05T22:05:48.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:48 smithi121 ceph-mon[145859]: daemon mds.cephfs.smithi121.gxheqt is now active in filesystem cephfs as rank 0 2024-04-05T22:05:48.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:48 smithi121 ceph-mon[145859]: daemon mds.cephfs.smithi121.ifoxmw is now active in filesystem cephfs as rank 1 2024-04-05T22:05:48.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:48 smithi121 ceph-mon[145859]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-05T22:05:48.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:48 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] up:active 2024-04-05T22:05:48.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:48 smithi121 ceph-mon[145859]: mds.? [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] up:active 2024-04-05T22:05:48.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:48 smithi121 ceph-mon[145859]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi121.ifoxmw=up:active} 2 up:standby-replay 2024-04-05T22:05:49.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:49 smithi121 ceph-mon[145859]: pgmap v1190: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 682 B/s wr, 9 op/s 2024-04-05T22:05:49.517 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:49 smithi188 ceph-mon[107793]: pgmap v1190: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 682 B/s wr, 9 op/s 2024-04-05T22:05:51.115 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:51 smithi188 ceph-mon[107793]: pgmap v1191: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 853 B/s wr, 12 op/s 2024-04-05T22:05:51.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:51 smithi121 ceph-mon[145859]: pgmap v1191: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 853 B/s wr, 12 op/s 2024-04-05T22:05:52.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:52 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:52.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:52 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:52.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:52 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:05:52.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:52 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:52.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:52 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:52.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:52 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:52.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:52 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:53.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:52 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:53.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:52 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:53.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:52 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:05:53.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:52 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:53.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:52 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:53.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:52 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:53.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:52 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:53.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:53 smithi121 ceph-mon[145859]: pgmap v1192: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7.8 KiB/s wr, 16 op/s 2024-04-05T22:05:53.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:53 smithi188 ceph-mon[107793]: pgmap v1192: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7.8 KiB/s wr, 16 op/s 2024-04-05T22:05:55.582 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:55.582 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:55.582 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:55.582 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:55 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:55.582 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:55 smithi188 ceph-mon[107793]: pgmap v1193: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 12 KiB/s wr, 18 op/s 2024-04-05T22:05:55.637 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:55.638 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:55.638 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:55.638 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:55 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:55.638 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:55 smithi121 ceph-mon[145859]: pgmap v1193: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 12 KiB/s wr, 18 op/s 2024-04-05T22:05:57.702 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:57.702 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:57.702 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:57.702 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:57 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:57.702 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:57 smithi188 ceph-mon[107793]: pgmap v1194: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 12 KiB/s wr, 18 op/s 2024-04-05T22:05:57.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:57.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:57.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:57.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:57 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:57.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:57 smithi121 ceph-mon[145859]: pgmap v1194: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 12 KiB/s wr, 18 op/s 2024-04-05T22:05:59.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:59 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:59.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:59 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:59.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:05:59 smithi121 ceph-mon[145859]: pgmap v1195: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 12 KiB/s wr, 18 op/s 2024-04-05T22:05:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:59 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:59 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:05:59.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:05:59 smithi188 ceph-mon[107793]: pgmap v1195: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 12 KiB/s wr, 18 op/s 2024-04-05T22:06:01.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:01.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: pgmap v1196: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 12 KiB/s wr, 17 op/s 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: Upgrade: Setting filesystem cephfs Joinable 2024-04-05T22:06:01.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:01 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-05T22:06:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:06:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:06:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:06:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: pgmap v1196: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 12 KiB/s wr, 17 op/s 2024-04-05T22:06:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:06:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:01.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: Upgrade: Setting filesystem cephfs Joinable 2024-04-05T22:06:01.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:01 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-04-05T22:06:02.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-05T22:06:02.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: Cluster is now healthy 2024-04-05T22:06:02.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-05T22:06:02.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi121.ifoxmw=up:active} 2 up:standby-replay 2024-04-05T22:06:02.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: Upgrade: Setting container_image for all ceph-exporter 2024-04-05T22:06:02.489 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi121"}]: dispatch 2024-04-05T22:06:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi121"}]': finished 2024-04-05T22:06:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi188"}]: dispatch 2024-04-05T22:06:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi188"}]': finished 2024-04-05T22:06:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: Upgrade: Setting container_image for all iscsi 2024-04-05T22:06:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: Upgrade: Setting container_image for all nfs 2024-04-05T22:06:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: Upgrade: Setting container_image for all nvmeof 2024-04-05T22:06:02.490 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: Upgrade: Finalizing container_image settings 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-05T22:06:02.491 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-05T22:06:02.492 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: Upgrade: Complete! 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:06:02.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:06:02.494 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:06:02.494 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:02.494 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:06:02.494 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:06:02.494 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:06:02.494 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:02.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-04-05T22:06:02.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: Cluster is now healthy 2024-04-05T22:06:02.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-04-05T22:06:02.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: fsmap cephfs:2 {0=cephfs.smithi121.gxheqt=up:active,1=cephfs.smithi121.ifoxmw=up:active} 2 up:standby-replay 2024-04-05T22:06:02.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: Upgrade: Setting container_image for all ceph-exporter 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi121"}]: dispatch 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi121"}]': finished 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi188"}]: dispatch 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter.smithi188"}]': finished 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: Upgrade: Setting container_image for all iscsi 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: Upgrade: Setting container_image for all nfs 2024-04-05T22:06:02.530 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: Upgrade: Setting container_image for all nvmeof 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: Upgrade: Finalizing container_image settings 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.531 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-05T22:06:02.532 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: Upgrade: Complete! 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-05T22:06:02.533 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:06:02.534 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:06:02.534 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:06:02.534 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:02.534 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:06:02.534 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:06:02.534 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:06:02.534 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:03.488 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:03 smithi121 ceph-mon[145859]: pgmap v1197: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 12 KiB/s wr, 12 op/s 2024-04-05T22:06:03.529 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:03 smithi188 ceph-mon[107793]: pgmap v1197: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 12 MiB/s rd, 12 KiB/s wr, 12 op/s 2024-04-05T22:06:05.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.116+0000 7f0a7854f700 1 -- 172.21.15.121:0/95639379 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a70101750 msgr2=0x7f0a701057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.116+0000 7f0a7854f700 1 --2- 172.21.15.121:0/95639379 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a70101750 0x7f0a701057a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009910 tx=0x7f0a60009c20 comp rx=0 tx=0).stop 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.117+0000 7f0a7854f700 1 -- 172.21.15.121:0/95639379 shutdown_connections 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.117+0000 7f0a7854f700 1 --2- 172.21.15.121:0/95639379 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a70101750 0x7f0a701057a0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.117+0000 7f0a7854f700 1 --2- 172.21.15.121:0/95639379 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a70100da0 0x7f0a70101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.117+0000 7f0a7854f700 1 -- 172.21.15.121:0/95639379 >> 172.21.15.121:0/95639379 conn(0x7f0a700fc630 msgr2=0x7f0a700fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.117+0000 7f0a7854f700 1 -- 172.21.15.121:0/95639379 shutdown_connections 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.117+0000 7f0a7854f700 1 -- 172.21.15.121:0/95639379 wait complete. 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.117+0000 7f0a7854f700 1 Processor -- start 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.117+0000 7f0a7854f700 1 -- start start 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.117+0000 7f0a7854f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a70100da0 0x7f0a70073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.117+0000 7f0a7854f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a70101750 0x7f0a700735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.117+0000 7f0a7854f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a70073b10 con 0x7f0a70100da0 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.117+0000 7f0a7854f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a70073c50 con 0x7f0a70101750 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.118+0000 7f0a762eb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a70100da0 0x7f0a70073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:05.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.118+0000 7f0a762eb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a70100da0 0x7f0a70073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51840/0 (socket says 172.21.15.121:51840) 2024-04-05T22:06:05.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.118+0000 7f0a762eb700 1 -- 172.21.15.121:0/4211126761 learned_addr learned my addr 172.21.15.121:0/4211126761 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:06:05.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.118+0000 7f0a75aea700 1 --2- 172.21.15.121:0/4211126761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a70101750 0x7f0a700735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:05.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.118+0000 7f0a762eb700 1 -- 172.21.15.121:0/4211126761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a70101750 msgr2=0x7f0a700735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:05.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.118+0000 7f0a762eb700 1 --2- 172.21.15.121:0/4211126761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a70101750 0x7f0a700735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:05.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.118+0000 7f0a762eb700 1 -- 172.21.15.121:0/4211126761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a600095f0 con 0x7f0a70100da0 2024-04-05T22:06:05.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.118+0000 7f0a762eb700 1 --2- 172.21.15.121:0/4211126761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a70100da0 0x7f0a70073090 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00d8d0 tx=0x7f0a6c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:06:05.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.118+0000 7f0a677fe700 1 -- 172.21.15.121:0/4211126761 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a6c009940 con 0x7f0a70100da0 2024-04-05T22:06:05.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.118+0000 7f0a677fe700 1 -- 172.21.15.121:0/4211126761 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a6c010460 con 0x7f0a70100da0 2024-04-05T22:06:05.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.118+0000 7f0a7854f700 1 -- 172.21.15.121:0/4211126761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a701a2820 con 0x7f0a70100da0 2024-04-05T22:06:05.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.118+0000 7f0a677fe700 1 -- 172.21.15.121:0/4211126761 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a6c00f5f0 con 0x7f0a70100da0 2024-04-05T22:06:05.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.118+0000 7f0a7854f700 1 -- 172.21.15.121:0/4211126761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a701a2b80 con 0x7f0a70100da0 2024-04-05T22:06:05.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.119+0000 7f0a7854f700 1 -- 172.21.15.121:0/4211126761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a701091a0 con 0x7f0a70100da0 2024-04-05T22:06:05.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.120+0000 7f0a677fe700 1 -- 172.21.15.121:0/4211126761 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0a6c0105d0 con 0x7f0a70100da0 2024-04-05T22:06:05.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.120+0000 7f0a677fe700 1 --2- 172.21.15.121:0/4211126761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0a5c06cce0 0x7f0a5c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:05.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.120+0000 7f0a677fe700 1 -- 172.21.15.121:0/4211126761 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0a6c08c5e0 con 0x7f0a70100da0 2024-04-05T22:06:05.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.120+0000 7f0a75aea700 1 --2- 172.21.15.121:0/4211126761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0a5c06cce0 0x7f0a5c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:05.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.121+0000 7f0a75aea700 1 --2- 172.21.15.121:0/4211126761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0a5c06cce0 0x7f0a5c06f1a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0a70074810 tx=0x7f0a6001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:06:05.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.125+0000 7f0a677fe700 1 -- 172.21.15.121:0/4211126761 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0a6c056650 con 0x7f0a70100da0 2024-04-05T22:06:05.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.335+0000 7f0a7854f700 1 -- 172.21.15.121:0/4211126761 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a7002cd60 con 0x7f0a5c06cce0 2024-04-05T22:06:05.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.336+0000 7f0a677fe700 1 -- 172.21.15.121:0/4211126761 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f0a7002cd60 con 0x7f0a5c06cce0 2024-04-05T22:06:05.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.338+0000 7f0a7854f700 1 -- 172.21.15.121:0/4211126761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0a5c06cce0 msgr2=0x7f0a5c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:05.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.338+0000 7f0a7854f700 1 --2- 172.21.15.121:0/4211126761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0a5c06cce0 0x7f0a5c06f1a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0a70074810 tx=0x7f0a6001a040 comp rx=0 tx=0).stop 2024-04-05T22:06:05.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.338+0000 7f0a7854f700 1 -- 172.21.15.121:0/4211126761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a70100da0 msgr2=0x7f0a70073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:05.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.338+0000 7f0a7854f700 1 --2- 172.21.15.121:0/4211126761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a70100da0 0x7f0a70073090 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00d8d0 tx=0x7f0a6c00dc90 comp rx=0 tx=0).stop 2024-04-05T22:06:05.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.338+0000 7f0a7854f700 1 -- 172.21.15.121:0/4211126761 shutdown_connections 2024-04-05T22:06:05.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.338+0000 7f0a7854f700 1 --2- 172.21.15.121:0/4211126761 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0a5c06cce0 0x7f0a5c06f1a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:05.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.338+0000 7f0a7854f700 1 --2- 172.21.15.121:0/4211126761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a70100da0 0x7f0a70073090 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:05.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.338+0000 7f0a7854f700 1 --2- 172.21.15.121:0/4211126761 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0a70101750 0x7f0a700735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:05.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.338+0000 7f0a7854f700 1 -- 172.21.15.121:0/4211126761 >> 172.21.15.121:0/4211126761 conn(0x7f0a700fc630 msgr2=0x7f0a700fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:06:05.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.338+0000 7f0a7854f700 1 -- 172.21.15.121:0/4211126761 shutdown_connections 2024-04-05T22:06:05.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:05.338+0000 7f0a7854f700 1 -- 172.21.15.121:0/4211126761 wait complete. 2024-04-05T22:06:05.983 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:05 smithi121 ceph-mon[145859]: pgmap v1198: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 6.4 MiB/s rd, 4.5 KiB/s wr, 6 op/s 2024-04-05T22:06:05.984 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:05 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:06.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph orch ps' 2024-04-05T22:06:06.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:05 smithi188 ceph-mon[107793]: pgmap v1198: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 6.4 MiB/s rd, 4.5 KiB/s wr, 6 op/s 2024-04-05T22:06:06.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:05 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T22:06:06.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:06 smithi121 ceph-mon[145859]: from='client.35974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:06:07.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:06 smithi188 ceph-mon[107793]: from='client.35974 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:06:07.817 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:07 smithi121 ceph-mon[145859]: pgmap v1199: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 719 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-05T22:06:08.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:07 smithi188 ceph-mon[107793]: pgmap v1199: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 719 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-05T22:06:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.073+0000 7f2377d65700 1 -- 172.21.15.121:0/2645337257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23700ffbd0 msgr2=0x7f2370100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:08.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.073+0000 7f2377d65700 1 --2- 172.21.15.121:0/2645337257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23700ffbd0 0x7f2370100050 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f2360009910 tx=0x7f2360009c20 comp rx=0 tx=0).stop 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.073+0000 7f2377d65700 1 -- 172.21.15.121:0/2645337257 shutdown_connections 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.073+0000 7f2377d65700 1 --2- 172.21.15.121:0/2645337257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23700ffbd0 0x7f2370100050 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.073+0000 7f2377d65700 1 --2- 172.21.15.121:0/2645337257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2370105a40 0x7f2370105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.073+0000 7f2377d65700 1 -- 172.21.15.121:0/2645337257 >> 172.21.15.121:0/2645337257 conn(0x7f23700755a0 msgr2=0x7f23700759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.073+0000 7f2377d65700 1 -- 172.21.15.121:0/2645337257 shutdown_connections 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.073+0000 7f2377d65700 1 -- 172.21.15.121:0/2645337257 wait complete. 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.073+0000 7f2377d65700 1 Processor -- start 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.073+0000 7f2377d65700 1 -- start start 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2377d65700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23700ffbd0 0x7f2370196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2377d65700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2370105a40 0x7f2370196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2377d65700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2370197290 con 0x7f2370105a40 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2377d65700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237019b020 con 0x7f23700ffbd0 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2375300700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2370105a40 0x7f2370196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:08.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2375300700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2370105a40 0x7f2370196bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51866/0 (socket says 172.21.15.121:51866) 2024-04-05T22:06:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2375300700 1 -- 172.21.15.121:0/2097305942 learned_addr learned my addr 172.21.15.121:0/2097305942 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:06:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2375b01700 1 --2- 172.21.15.121:0/2097305942 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23700ffbd0 0x7f2370196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2375300700 1 -- 172.21.15.121:0/2097305942 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23700ffbd0 msgr2=0x7f2370196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2375300700 1 --2- 172.21.15.121:0/2097305942 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23700ffbd0 0x7f2370196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2375300700 1 -- 172.21.15.121:0/2097305942 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23600095f0 con 0x7f2370105a40 2024-04-05T22:06:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2375300700 1 --2- 172.21.15.121:0/2097305942 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2370105a40 0x7f2370196bb0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f2360006010 tx=0x7f236000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:06:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2366ffd700 1 -- 172.21.15.121:0/2097305942 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f236001d070 con 0x7f2370105a40 2024-04-05T22:06:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2377d65700 1 -- 172.21.15.121:0/2097305942 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237019b2a0 con 0x7f2370105a40 2024-04-05T22:06:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2366ffd700 1 -- 172.21.15.121:0/2097305942 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f236000f460 con 0x7f2370105a40 2024-04-05T22:06:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2366ffd700 1 -- 172.21.15.121:0/2097305942 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2360021620 con 0x7f2370105a40 2024-04-05T22:06:08.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.074+0000 7f2377d65700 1 -- 172.21.15.121:0/2097305942 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237019b790 con 0x7f2370105a40 2024-04-05T22:06:08.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.075+0000 7f2377d65700 1 -- 172.21.15.121:0/2097305942 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237004ea90 con 0x7f2370105a40 2024-04-05T22:06:08.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.076+0000 7f2366ffd700 1 -- 172.21.15.121:0/2097305942 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f236000fa40 con 0x7f2370105a40 2024-04-05T22:06:08.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.077+0000 7f2366ffd700 1 --2- 172.21.15.121:0/2097305942 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f235c06cce0 0x7f235c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:08.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.077+0000 7f2366ffd700 1 -- 172.21.15.121:0/2097305942 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f236008dbf0 con 0x7f2370105a40 2024-04-05T22:06:08.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.077+0000 7f2375b01700 1 --2- 172.21.15.121:0/2097305942 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f235c06cce0 0x7f235c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:08.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.077+0000 7f2375b01700 1 --2- 172.21.15.121:0/2097305942 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f235c06cce0 0x7f235c06f1a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f236c005950 tx=0x7f236c0058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:06:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.081+0000 7f2366ffd700 1 -- 172.21.15.121:0/2097305942 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2360057ce0 con 0x7f2370105a40 2024-04-05T22:06:08.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.283+0000 7f2377d65700 1 -- 172.21.15.121:0/2097305942 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f237019bca0 con 0x7f235c06cce0 2024-04-05T22:06:08.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.294+0000 7f2366ffd700 1 -- 172.21.15.121:0/2097305942 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3816 (secure 0 0 0) 0x7f237019bca0 con 0x7f235c06cce0 2024-04-05T22:06:08.294 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T22:06:08.294 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (36m) 11s ago 48m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T22:06:08.294 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (23s) 11s ago 48m 7340k - 18.2.2-1173-g8919605b 067bdb249792 476a804f366b 2024-04-05T22:06:08.294 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (17s) 13s ago 47m 6883k - 18.2.2-1173-g8919605b 067bdb249792 c07757ef70c2 2024-04-05T22:06:08.294 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (35m) 11s ago 48m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T22:06:08.294 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (35m) 13s ago 47m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T22:06:08.294 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (36m) 11s ago 48m 89.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (31m) 11s ago 42m 69.9M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (31m) 11s ago 42m 65.7M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (30m) 13s ago 42m 127M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (15m) 13s ago 42m 148M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (38m) 11s ago 50m 516M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (38m) 13s ago 47m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (35m) 11s ago 50m 82.2M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (35m) 13s ago 47m 56.4M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (37m) 11s ago 48m 21.4M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (37m) 13s ago 47m 17.8M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (34m) 11s ago 46m 103M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (34m) 11s ago 45m 118M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (33m) 11s ago 45m 98.9M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (32m) 13s ago 44m 133M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T22:06:08.295 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (32m) 13s ago 44m 107M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T22:06:08.296 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (31m) 13s ago 43m 98.5M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T22:06:08.296 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (37m) 11s ago 47m 89.4M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T22:06:08.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.296+0000 7f2377d65700 1 -- 172.21.15.121:0/2097305942 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f235c06cce0 msgr2=0x7f235c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:08.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.296+0000 7f2377d65700 1 --2- 172.21.15.121:0/2097305942 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f235c06cce0 0x7f235c06f1a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f236c005950 tx=0x7f236c0058e0 comp rx=0 tx=0).stop 2024-04-05T22:06:08.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.296+0000 7f2377d65700 1 -- 172.21.15.121:0/2097305942 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2370105a40 msgr2=0x7f2370196bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:08.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.296+0000 7f2377d65700 1 --2- 172.21.15.121:0/2097305942 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2370105a40 0x7f2370196bb0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f2360006010 tx=0x7f236000be20 comp rx=0 tx=0).stop 2024-04-05T22:06:08.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.296+0000 7f2377d65700 1 -- 172.21.15.121:0/2097305942 shutdown_connections 2024-04-05T22:06:08.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.296+0000 7f2377d65700 1 --2- 172.21.15.121:0/2097305942 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f235c06cce0 0x7f235c06f1a0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:08.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.296+0000 7f2377d65700 1 --2- 172.21.15.121:0/2097305942 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f23700ffbd0 0x7f2370196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:08.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.296+0000 7f2377d65700 1 --2- 172.21.15.121:0/2097305942 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2370105a40 0x7f2370196bb0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:08.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.296+0000 7f2377d65700 1 -- 172.21.15.121:0/2097305942 >> 172.21.15.121:0/2097305942 conn(0x7f23700755a0 msgr2=0x7f23701037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:06:08.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.296+0000 7f2377d65700 1 -- 172.21.15.121:0/2097305942 shutdown_connections 2024-04-05T22:06:08.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:08.296+0000 7f2377d65700 1 -- 172.21.15.121:0/2097305942 wait complete. 2024-04-05T22:06:09.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph orch upgrade status' 2024-04-05T22:06:09.282 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T22:06:09.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:09 smithi121 ceph-mon[145859]: from='client.35978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:06:09.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:09 smithi121 ceph-mon[145859]: pgmap v1200: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 11 KiB/s rd, 4 op/s 2024-04-05T22:06:10.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:09 smithi188 ceph-mon[107793]: from='client.35978 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:06:10.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:09 smithi188 ceph-mon[107793]: pgmap v1200: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 11 KiB/s rd, 4 op/s 2024-04-05T22:06:11.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:11 smithi121 ceph-mon[145859]: pgmap v1201: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 5.4 KiB/s rd, 4 op/s 2024-04-05T22:06:12.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:11 smithi188 ceph-mon[107793]: pgmap v1201: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 5.4 KiB/s rd, 4 op/s 2024-04-05T22:06:13.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.079+0000 7f696ece1700 1 -- 172.21.15.121:0/2751505036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6968101d00 msgr2=0x7f69681020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:13.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.079+0000 7f696ece1700 1 --2- 172.21.15.121:0/2751505036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6968101d00 0x7f69681020e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f695c009910 tx=0x7f695c009c20 comp rx=0 tx=0).stop 2024-04-05T22:06:13.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.079+0000 7f696ece1700 1 -- 172.21.15.121:0/2751505036 shutdown_connections 2024-04-05T22:06:13.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.079+0000 7f696ece1700 1 --2- 172.21.15.121:0/2751505036 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6968102620 0x7f696810ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:13.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.079+0000 7f696ece1700 1 --2- 172.21.15.121:0/2751505036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6968101d00 0x7f69681020e0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:13.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.079+0000 7f696ece1700 1 -- 172.21.15.121:0/2751505036 >> 172.21.15.121:0/2751505036 conn(0x7f69680fb570 msgr2=0x7f69680fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:06:13.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.079+0000 7f696ece1700 1 -- 172.21.15.121:0/2751505036 shutdown_connections 2024-04-05T22:06:13.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.079+0000 7f696ece1700 1 -- 172.21.15.121:0/2751505036 wait complete. 2024-04-05T22:06:13.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.080+0000 7f696ece1700 1 Processor -- start 2024-04-05T22:06:13.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.080+0000 7f696ece1700 1 -- start start 2024-04-05T22:06:13.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.080+0000 7f696ece1700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6968101d00 0x7f696819c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.080+0000 7f696ece1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6968102620 0x7f696819cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.080+0000 7f696ece1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696819d150 con 0x7f6968102620 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.080+0000 7f696ece1700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6968196600 con 0x7f6968101d00 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.080+0000 7f6967fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6968102620 0x7f696819cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.080+0000 7f6967fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6968102620 0x7f696819cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51878/0 (socket says 172.21.15.121:51878) 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.080+0000 7f6967fff700 1 -- 172.21.15.121:0/2010856347 learned_addr learned my addr 172.21.15.121:0/2010856347 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.081+0000 7f696ca7d700 1 --2- 172.21.15.121:0/2010856347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6968101d00 0x7f696819c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.081+0000 7f6967fff700 1 -- 172.21.15.121:0/2010856347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6968101d00 msgr2=0x7f696819c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.081+0000 7f6967fff700 1 --2- 172.21.15.121:0/2010856347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6968101d00 0x7f696819c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.081+0000 7f6967fff700 1 -- 172.21.15.121:0/2010856347 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c0095f0 con 0x7f6968102620 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.081+0000 7f6967fff700 1 --2- 172.21.15.121:0/2010856347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6968102620 0x7f696819cac0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f695400b3d0 tx=0x7f695400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.081+0000 7f6965ffb700 1 -- 172.21.15.121:0/2010856347 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6954011800 con 0x7f6968102620 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.081+0000 7f6965ffb700 1 -- 172.21.15.121:0/2010856347 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6954011e00 con 0x7f6968102620 2024-04-05T22:06:13.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.081+0000 7f696ece1700 1 -- 172.21.15.121:0/2010856347 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69681968e0 con 0x7f6968102620 2024-04-05T22:06:13.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.081+0000 7f6965ffb700 1 -- 172.21.15.121:0/2010856347 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6954010e30 con 0x7f6968102620 2024-04-05T22:06:13.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.081+0000 7f696ece1700 1 -- 172.21.15.121:0/2010856347 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6968196e30 con 0x7f6968102620 2024-04-05T22:06:13.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.082+0000 7f696ece1700 1 -- 172.21.15.121:0/2010856347 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6968108210 con 0x7f6968102620 2024-04-05T22:06:13.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.082+0000 7f6965ffb700 1 -- 172.21.15.121:0/2010856347 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6954011960 con 0x7f6968102620 2024-04-05T22:06:13.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.083+0000 7f6965ffb700 1 --2- 172.21.15.121:0/2010856347 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f695806cce0 0x7f695806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:13.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.083+0000 7f6965ffb700 1 -- 172.21.15.121:0/2010856347 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f695408bff0 con 0x7f6968102620 2024-04-05T22:06:13.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.083+0000 7f696ca7d700 1 --2- 172.21.15.121:0/2010856347 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f695806cce0 0x7f695806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:13.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.083+0000 7f696ca7d700 1 --2- 172.21.15.121:0/2010856347 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f695806cce0 0x7f695806f1a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f695c00b5c0 tx=0x7f695c005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:06:13.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.087+0000 7f6965ffb700 1 -- 172.21.15.121:0/2010856347 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6954059b70 con 0x7f6968102620 2024-04-05T22:06:13.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.301+0000 7f696ece1700 1 -- 172.21.15.121:0/2010856347 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f69681970b0 con 0x7f695806cce0 2024-04-05T22:06:13.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.302+0000 7f6965ffb700 1 -- 172.21.15.121:0/2010856347 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f69681970b0 con 0x7f695806cce0 2024-04-05T22:06:13.302 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:06:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "target_image": null, 2024-04-05T22:06:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "in_progress": false, 2024-04-05T22:06:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "which": "", 2024-04-05T22:06:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "services_complete": [], 2024-04-05T22:06:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "progress": null, 2024-04-05T22:06:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "message": "", 2024-04-05T22:06:13.302 INFO:teuthology.orchestra.run.smithi121.stdout: "is_paused": false 2024-04-05T22:06:13.303 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:06:13.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.304+0000 7f696ece1700 1 -- 172.21.15.121:0/2010856347 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f695806cce0 msgr2=0x7f695806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:13.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.304+0000 7f696ece1700 1 --2- 172.21.15.121:0/2010856347 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f695806cce0 0x7f695806f1a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f695c00b5c0 tx=0x7f695c005fd0 comp rx=0 tx=0).stop 2024-04-05T22:06:13.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.304+0000 7f696ece1700 1 -- 172.21.15.121:0/2010856347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6968102620 msgr2=0x7f696819cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:13.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.304+0000 7f696ece1700 1 --2- 172.21.15.121:0/2010856347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6968102620 0x7f696819cac0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f695400b3d0 tx=0x7f695400b790 comp rx=0 tx=0).stop 2024-04-05T22:06:13.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.304+0000 7f696ece1700 1 -- 172.21.15.121:0/2010856347 shutdown_connections 2024-04-05T22:06:13.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.304+0000 7f696ece1700 1 --2- 172.21.15.121:0/2010856347 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f695806cce0 0x7f695806f1a0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:13.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.304+0000 7f696ece1700 1 --2- 172.21.15.121:0/2010856347 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6968101d00 0x7f696819c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:13.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.304+0000 7f696ece1700 1 --2- 172.21.15.121:0/2010856347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6968102620 0x7f696819cac0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:13.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.304+0000 7f696ece1700 1 -- 172.21.15.121:0/2010856347 >> 172.21.15.121:0/2010856347 conn(0x7f69680fb570 msgr2=0x7f6968105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:06:13.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.304+0000 7f696ece1700 1 -- 172.21.15.121:0/2010856347 shutdown_connections 2024-04-05T22:06:13.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:13.304+0000 7f696ece1700 1 -- 172.21.15.121:0/2010856347 wait complete. 2024-04-05T22:06:13.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph health detail' 2024-04-05T22:06:13.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:13 smithi121 ceph-mon[145859]: pgmap v1202: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:06:14.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:13 smithi188 ceph-mon[107793]: pgmap v1202: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:06:14.084 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T22:06:14.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:14 smithi121 ceph-mon[145859]: from='client.35982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:06:15.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:14 smithi188 ceph-mon[107793]: from='client.35982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:06:15.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.938+0000 7f29222b7700 1 -- 172.21.15.121:0/904227431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f291c105a40 msgr2=0x7f291c105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:15.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.938+0000 7f29222b7700 1 --2- 172.21.15.121:0/904227431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f291c105a40 0x7f291c105e20 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f290c009910 tx=0x7f290c009c20 comp rx=0 tx=0).stop 2024-04-05T22:06:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.938+0000 7f29222b7700 1 -- 172.21.15.121:0/904227431 shutdown_connections 2024-04-05T22:06:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.938+0000 7f29222b7700 1 --2- 172.21.15.121:0/904227431 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f291c0ffa40 0x7f291c0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.938+0000 7f29222b7700 1 --2- 172.21.15.121:0/904227431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f291c105a40 0x7f291c105e20 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.938+0000 7f29222b7700 1 -- 172.21.15.121:0/904227431 >> 172.21.15.121:0/904227431 conn(0x7f291c0fb570 msgr2=0x7f291c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:06:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.939+0000 7f29222b7700 1 -- 172.21.15.121:0/904227431 shutdown_connections 2024-04-05T22:06:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.939+0000 7f29222b7700 1 -- 172.21.15.121:0/904227431 wait complete. 2024-04-05T22:06:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.939+0000 7f29222b7700 1 Processor -- start 2024-04-05T22:06:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.939+0000 7f29222b7700 1 -- start start 2024-04-05T22:06:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.939+0000 7f29222b7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f291c0ffa40 0x7f291c1974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f29222b7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f291c105a40 0x7f291c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f29222b7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c197fa0 con 0x7f291c0ffa40 2024-04-05T22:06:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f29222b7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c1980e0 con 0x7f291c105a40 2024-04-05T22:06:15.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f291bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f291c0ffa40 0x7f291c1974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f291bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f291c0ffa40 0x7f291c1974d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55232/0 (socket says 172.21.15.121:55232) 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f291bfff700 1 -- 172.21.15.121:0/2301085015 learned_addr learned my addr 172.21.15.121:0/2301085015 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f291b7fe700 1 --2- 172.21.15.121:0/2301085015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f291c105a40 0x7f291c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f291bfff700 1 -- 172.21.15.121:0/2301085015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f291c105a40 msgr2=0x7f291c197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f291bfff700 1 --2- 172.21.15.121:0/2301085015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f291c105a40 0x7f291c197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f291bfff700 1 -- 172.21.15.121:0/2301085015 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c0095f0 con 0x7f291c0ffa40 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f291bfff700 1 --2- 172.21.15.121:0/2301085015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f291c0ffa40 0x7f291c1974d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f290c00b5c0 tx=0x7f290c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f29197fa700 1 -- 172.21.15.121:0/2301085015 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f290c01d070 con 0x7f291c0ffa40 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f29197fa700 1 -- 172.21.15.121:0/2301085015 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f290c00f460 con 0x7f291c0ffa40 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f29197fa700 1 -- 172.21.15.121:0/2301085015 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f290c021640 con 0x7f291c0ffa40 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f29222b7700 1 -- 172.21.15.121:0/2301085015 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291c1a0540 con 0x7f291c0ffa40 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.940+0000 7f29222b7700 1 -- 172.21.15.121:0/2301085015 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291c1a08f0 con 0x7f291c0ffa40 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.941+0000 7f29222b7700 1 -- 172.21.15.121:0/2301085015 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291c108040 con 0x7f291c0ffa40 2024-04-05T22:06:15.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.942+0000 7f29197fa700 1 -- 172.21.15.121:0/2301085015 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f290c00f9b0 con 0x7f291c0ffa40 2024-04-05T22:06:15.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.942+0000 7f29197fa700 1 --2- 172.21.15.121:0/2301085015 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f290406cce0 0x7f290406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:15.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.942+0000 7f29197fa700 1 -- 172.21.15.121:0/2301085015 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f290c08d8c0 con 0x7f291c0ffa40 2024-04-05T22:06:15.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.942+0000 7f291b7fe700 1 --2- 172.21.15.121:0/2301085015 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f290406cce0 0x7f290406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:15.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.942+0000 7f291b7fe700 1 --2- 172.21.15.121:0/2301085015 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f290406cce0 0x7f290406f1a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2910005ea0 tx=0x7f2910005e30 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:06:15.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:15.947+0000 7f29197fa700 1 -- 172.21.15.121:0/2301085015 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f290c057930 con 0x7f291c0ffa40 2024-04-05T22:06:15.987 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:15 smithi121 ceph-mon[145859]: pgmap v1203: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:06:15.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:15.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:06:16.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:15 smithi188 ceph-mon[107793]: pgmap v1203: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:06:16.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:06:16.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:06:16.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:16.218+0000 7f29222b7700 1 -- 172.21.15.121:0/2301085015 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f291c04ea90 con 0x7f291c0ffa40 2024-04-05T22:06:16.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:16.219+0000 7f29197fa700 1 -- 172.21.15.121:0/2301085015 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f290c05af50 con 0x7f291c0ffa40 2024-04-05T22:06:16.219 INFO:teuthology.orchestra.run.smithi121.stdout:HEALTH_OK 2024-04-05T22:06:16.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:16.221+0000 7f29222b7700 1 -- 172.21.15.121:0/2301085015 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f290406cce0 msgr2=0x7f290406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:16.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:16.221+0000 7f29222b7700 1 --2- 172.21.15.121:0/2301085015 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f290406cce0 0x7f290406f1a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2910005ea0 tx=0x7f2910005e30 comp rx=0 tx=0).stop 2024-04-05T22:06:16.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:16.221+0000 7f29222b7700 1 -- 172.21.15.121:0/2301085015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f291c0ffa40 msgr2=0x7f291c1974d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:16.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:16.221+0000 7f29222b7700 1 --2- 172.21.15.121:0/2301085015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f291c0ffa40 0x7f291c1974d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f290c00b5c0 tx=0x7f290c00be20 comp rx=0 tx=0).stop 2024-04-05T22:06:16.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:16.221+0000 7f29222b7700 1 -- 172.21.15.121:0/2301085015 shutdown_connections 2024-04-05T22:06:16.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:16.221+0000 7f29222b7700 1 --2- 172.21.15.121:0/2301085015 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f290406cce0 0x7f290406f1a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:16.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:16.221+0000 7f29222b7700 1 --2- 172.21.15.121:0/2301085015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f291c0ffa40 0x7f291c1974d0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:16.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:16.221+0000 7f29222b7700 1 --2- 172.21.15.121:0/2301085015 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f291c105a40 0x7f291c197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:16.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:16.221+0000 7f29222b7700 1 -- 172.21.15.121:0/2301085015 >> 172.21.15.121:0/2301085015 conn(0x7f291c0fb570 msgr2=0x7f291c0fbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:06:16.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:16.221+0000 7f29222b7700 1 -- 172.21.15.121:0/2301085015 shutdown_connections 2024-04-05T22:06:16.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:16.221+0000 7f29222b7700 1 -- 172.21.15.121:0/2301085015 wait complete. 2024-04-05T22:06:16.895 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:16 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2301085015' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:06:16.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph versions' 2024-04-05T22:06:17.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:16 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2301085015' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-05T22:06:17.143 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T22:06:17.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:17 smithi121 ceph-mon[145859]: pgmap v1204: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:18.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:17 smithi188 ceph-mon[107793]: pgmap v1204: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.846+0000 7f3762ffb700 1 -- 172.21.15.121:0/2230449620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c0ffa40 msgr2=0x7f375c0ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.846+0000 7f3762ffb700 1 --2- 172.21.15.121:0/2230449620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c0ffa40 0x7f375c0ffec0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f3748009910 tx=0x7f3748009c20 comp rx=0 tx=0).stop 2024-04-05T22:06:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.847+0000 7f3762ffb700 1 -- 172.21.15.121:0/2230449620 shutdown_connections 2024-04-05T22:06:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.847+0000 7f3762ffb700 1 --2- 172.21.15.121:0/2230449620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c0ffa40 0x7f375c0ffec0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.847+0000 7f3762ffb700 1 --2- 172.21.15.121:0/2230449620 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f375c105a40 0x7f375c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.847+0000 7f3762ffb700 1 -- 172.21.15.121:0/2230449620 >> 172.21.15.121:0/2230449620 conn(0x7f375c0fb570 msgr2=0x7f375c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:06:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.847+0000 7f3762ffb700 1 -- 172.21.15.121:0/2230449620 shutdown_connections 2024-04-05T22:06:18.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.847+0000 7f3762ffb700 1 -- 172.21.15.121:0/2230449620 wait complete. 2024-04-05T22:06:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.847+0000 7f3762ffb700 1 Processor -- start 2024-04-05T22:06:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.847+0000 7f3762ffb700 1 -- start start 2024-04-05T22:06:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.847+0000 7f3762ffb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c0ffa40 0x7f375c0fbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.847+0000 7f3762ffb700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f375c105a40 0x7f375c0fc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.847+0000 7f3762ffb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0fc960 con 0x7f375c0ffa40 2024-04-05T22:06:18.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.847+0000 7f3762ffb700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375c0fcaa0 con 0x7f375c105a40 2024-04-05T22:06:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f3760d97700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c0ffa40 0x7f375c0fbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f3760d97700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c0ffa40 0x7f375c0fbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55246/0 (socket says 172.21.15.121:55246) 2024-04-05T22:06:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f3760d97700 1 -- 172.21.15.121:0/962568506 learned_addr learned my addr 172.21.15.121:0/962568506 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:06:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f375bfff700 1 --2- 172.21.15.121:0/962568506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f375c105a40 0x7f375c0fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f3760d97700 1 -- 172.21.15.121:0/962568506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f375c105a40 msgr2=0x7f375c0fc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f3760d97700 1 --2- 172.21.15.121:0/962568506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f375c105a40 0x7f375c0fc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f3760d97700 1 -- 172.21.15.121:0/962568506 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37480095f0 con 0x7f375c0ffa40 2024-04-05T22:06:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f375bfff700 1 --2- 172.21.15.121:0/962568506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f375c105a40 0x7f375c0fc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:06:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f3760d97700 1 --2- 172.21.15.121:0/962568506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c0ffa40 0x7f375c0fbee0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f375000d8d0 tx=0x7f375000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:06:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f3759ffb700 1 -- 172.21.15.121:0/962568506 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3750009940 con 0x7f375c0ffa40 2024-04-05T22:06:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f3762ffb700 1 -- 172.21.15.121:0/962568506 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375c1a27e0 con 0x7f375c0ffa40 2024-04-05T22:06:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f3759ffb700 1 -- 172.21.15.121:0/962568506 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3750010460 con 0x7f375c0ffa40 2024-04-05T22:06:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f3759ffb700 1 -- 172.21.15.121:0/962568506 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f375000f5f0 con 0x7f375c0ffa40 2024-04-05T22:06:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.848+0000 7f3762ffb700 1 -- 172.21.15.121:0/962568506 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375c1a2bf0 con 0x7f375c0ffa40 2024-04-05T22:06:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.849+0000 7f3762ffb700 1 -- 172.21.15.121:0/962568506 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375c108fe0 con 0x7f375c0ffa40 2024-04-05T22:06:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.850+0000 7f3759ffb700 1 -- 172.21.15.121:0/962568506 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f37500105d0 con 0x7f375c0ffa40 2024-04-05T22:06:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.850+0000 7f3759ffb700 1 --2- 172.21.15.121:0/962568506 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f374c06cce0 0x7f374c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:06:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.850+0000 7f3759ffb700 1 -- 172.21.15.121:0/962568506 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f375008c5e0 con 0x7f375c0ffa40 2024-04-05T22:06:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.850+0000 7f375bfff700 1 --2- 172.21.15.121:0/962568506 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f374c06cce0 0x7f374c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:06:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.851+0000 7f375bfff700 1 --2- 172.21.15.121:0/962568506 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f374c06cce0 0x7f374c06f1a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f374800b5c0 tx=0x7f3748005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:06:18.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:18.855+0000 7f3759ffb700 1 -- 172.21.15.121:0/962568506 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3750056650 con 0x7f375c0ffa40 2024-04-05T22:06:19.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:19.130+0000 7f3762ffb700 1 -- 172.21.15.121:0/962568506 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f375c02cdd0 con 0x7f375c0ffa40 2024-04-05T22:06:19.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:19.130+0000 7f3759ffb700 1 -- 172.21.15.121:0/962568506 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7f3750010880 con 0x7f375c0ffa40 2024-04-05T22:06:19.130 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:06:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T22:06:19.130 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T22:06:19.131 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:06:19.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:19.132+0000 7f3762ffb700 1 -- 172.21.15.121:0/962568506 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f374c06cce0 msgr2=0x7f374c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:19.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:19.132+0000 7f3762ffb700 1 --2- 172.21.15.121:0/962568506 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f374c06cce0 0x7f374c06f1a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f374800b5c0 tx=0x7f3748005960 comp rx=0 tx=0).stop 2024-04-05T22:06:19.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:19.132+0000 7f3762ffb700 1 -- 172.21.15.121:0/962568506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c0ffa40 msgr2=0x7f375c0fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:06:19.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:19.132+0000 7f3762ffb700 1 --2- 172.21.15.121:0/962568506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c0ffa40 0x7f375c0fbee0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f375000d8d0 tx=0x7f375000dc90 comp rx=0 tx=0).stop 2024-04-05T22:06:19.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:19.132+0000 7f3762ffb700 1 -- 172.21.15.121:0/962568506 shutdown_connections 2024-04-05T22:06:19.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:19.132+0000 7f3762ffb700 1 --2- 172.21.15.121:0/962568506 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f374c06cce0 0x7f374c06f1a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:19.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:19.132+0000 7f3762ffb700 1 --2- 172.21.15.121:0/962568506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f375c0ffa40 0x7f375c0fbee0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:19.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:19.132+0000 7f3762ffb700 1 --2- 172.21.15.121:0/962568506 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f375c105a40 0x7f375c0fc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:06:19.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:19.132+0000 7f3762ffb700 1 -- 172.21.15.121:0/962568506 >> 172.21.15.121:0/962568506 conn(0x7f375c0fb570 msgr2=0x7f375c068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:06:19.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:19.132+0000 7f3762ffb700 1 -- 172.21.15.121:0/962568506 shutdown_connections 2024-04-05T22:06:19.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:06:19.133+0000 7f3762ffb700 1 -- 172.21.15.121:0/962568506 wait complete. 2024-04-05T22:06:19.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:19 smithi121 ceph-mon[145859]: pgmap v1205: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:19.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:19 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/962568506' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:19.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-05T22:06:20.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:19 smithi188 ceph-mon[107793]: pgmap v1205: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:20.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:19 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/962568506' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:06:20.037 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T22:06:21.713 INFO:teuthology.orchestra.run.smithi121.stdout:wait for servicemap items w/ changing names to refresh 2024-04-05T22:06:21.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:21 smithi121 ceph-mon[145859]: pgmap v1206: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:22.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:21 smithi188 ceph-mon[107793]: pgmap v1206: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:22.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'sleep 60' 2024-04-05T22:06:22.709 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T22:06:23.872 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:23 smithi121 ceph-mon[145859]: pgmap v1207: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:06:24.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:23 smithi188 ceph-mon[107793]: pgmap v1207: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:06:25.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:25 smithi121 ceph-mon[145859]: pgmap v1208: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:06:26.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:25 smithi188 ceph-mon[107793]: pgmap v1208: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:06:27.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:27 smithi121 ceph-mon[145859]: pgmap v1209: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:28.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:27 smithi188 ceph-mon[107793]: pgmap v1209: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:29.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:29 smithi121 ceph-mon[145859]: pgmap v1210: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:30.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:29 smithi188 ceph-mon[107793]: pgmap v1210: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:30.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:06:31.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:06:31.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:31 smithi121 ceph-mon[145859]: pgmap v1211: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:32.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:31 smithi188 ceph-mon[107793]: pgmap v1211: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:33.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:33 smithi121 ceph-mon[145859]: pgmap v1212: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:06:34.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:33 smithi188 ceph-mon[107793]: pgmap v1212: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:06:35.990 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:35 smithi121 ceph-mon[145859]: pgmap v1213: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:06:36.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:35 smithi188 ceph-mon[107793]: pgmap v1213: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:06:37.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:37 smithi121 ceph-mon[145859]: pgmap v1214: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:38.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:37 smithi188 ceph-mon[107793]: pgmap v1214: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:39.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:39 smithi121 ceph-mon[145859]: pgmap v1215: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:40.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:39 smithi188 ceph-mon[107793]: pgmap v1215: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:41.991 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:41 smithi121 ceph-mon[145859]: pgmap v1216: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:42.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:41 smithi188 ceph-mon[107793]: pgmap v1216: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:43.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:43 smithi121 ceph-mon[145859]: pgmap v1217: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:06:44.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:43 smithi188 ceph-mon[107793]: pgmap v1217: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:06:45.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:45 smithi121 ceph-mon[145859]: pgmap v1218: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:06:45.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:06:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:45 smithi188 ceph-mon[107793]: pgmap v1218: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:06:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:06:47.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:47 smithi121 ceph-mon[145859]: pgmap v1219: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:48.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:47 smithi188 ceph-mon[107793]: pgmap v1219: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:49.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:49 smithi121 ceph-mon[145859]: pgmap v1220: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:50.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:49 smithi188 ceph-mon[107793]: pgmap v1220: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:51.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:51 smithi121 ceph-mon[145859]: pgmap v1221: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:52.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:51 smithi188 ceph-mon[107793]: pgmap v1221: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:54.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:53 smithi188 ceph-mon[107793]: pgmap v1222: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:06:54.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:53 smithi121 ceph-mon[145859]: pgmap v1222: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:06:56.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:55 smithi188 ceph-mon[107793]: pgmap v1223: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:06:56.033 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:55 smithi121 ceph-mon[145859]: pgmap v1223: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:06:58.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:57 smithi188 ceph-mon[107793]: pgmap v1224: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:06:58.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:57 smithi121 ceph-mon[145859]: pgmap v1224: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:00.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:06:59 smithi188 ceph-mon[107793]: pgmap v1225: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:00.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:06:59 smithi121 ceph-mon[145859]: pgmap v1225: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:01.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:07:01.088 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:07:02.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:01 smithi188 ceph-mon[107793]: pgmap v1226: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:02.250 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:01 smithi121 ceph-mon[145859]: pgmap v1226: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:03.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:07:03.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:07:03.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:07:03.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:02 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:07:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:07:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:07:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:07:03.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:02 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:07:04.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:03 smithi188 ceph-mon[107793]: pgmap v1227: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:07:04.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:03 smithi121 ceph-mon[145859]: pgmap v1227: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:07:06.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:05 smithi188 ceph-mon[107793]: pgmap v1228: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:07:06.144 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:05 smithi121 ceph-mon[145859]: pgmap v1228: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:07:08.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:07 smithi188 ceph-mon[107793]: pgmap v1229: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:08.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:07 smithi121 ceph-mon[145859]: pgmap v1229: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:10.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:09 smithi188 ceph-mon[107793]: pgmap v1230: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:10.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:09 smithi121 ceph-mon[145859]: pgmap v1230: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:12.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:11 smithi188 ceph-mon[107793]: pgmap v1231: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:12.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:11 smithi121 ceph-mon[145859]: pgmap v1231: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:14.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:13 smithi121 ceph-mon[145859]: pgmap v1232: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:07:14.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:13 smithi188 ceph-mon[107793]: pgmap v1232: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:07:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:15 smithi121 ceph-mon[145859]: pgmap v1233: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:07:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:07:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:15 smithi188 ceph-mon[107793]: pgmap v1233: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:07:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:07:18.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:17 smithi121 ceph-mon[145859]: pgmap v1234: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:18.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:17 smithi188 ceph-mon[107793]: pgmap v1234: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:20.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:19 smithi121 ceph-mon[145859]: pgmap v1235: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:20.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:19 smithi188 ceph-mon[107793]: pgmap v1235: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:22.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:21 smithi121 ceph-mon[145859]: pgmap v1236: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:22.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:21 smithi188 ceph-mon[107793]: pgmap v1236: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:24.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:23 smithi121 ceph-mon[145859]: pgmap v1237: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:07:24.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:23 smithi188 ceph-mon[107793]: pgmap v1237: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:07:25.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph orch ps' 2024-04-05T22:07:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T22:07:26.155 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:25 smithi121 ceph-mon[145859]: pgmap v1238: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:07:26.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:25 smithi188 ceph-mon[107793]: pgmap v1238: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:07:27.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.056+0000 7efcf0dc3700 1 -- 172.21.15.121:0/2972246222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec105a40 msgr2=0x7efcec105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:27.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.056+0000 7efcf0dc3700 1 --2- 172.21.15.121:0/2972246222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec105a40 0x7efcec105e20 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009910 tx=0x7efcdc009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:27.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.056+0000 7efcf0dc3700 1 -- 172.21.15.121:0/2972246222 shutdown_connections 2024-04-05T22:07:27.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.056+0000 7efcf0dc3700 1 --2- 172.21.15.121:0/2972246222 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efcec0ffbd0 0x7efcec100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:27.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.056+0000 7efcf0dc3700 1 --2- 172.21.15.121:0/2972246222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec105a40 0x7efcec105e20 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:27.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.056+0000 7efcf0dc3700 1 -- 172.21.15.121:0/2972246222 >> 172.21.15.121:0/2972246222 conn(0x7efcec0755a0 msgr2=0x7efcec0759b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:27.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.056+0000 7efcf0dc3700 1 -- 172.21.15.121:0/2972246222 shutdown_connections 2024-04-05T22:07:27.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.056+0000 7efcf0dc3700 1 -- 172.21.15.121:0/2972246222 wait complete. 2024-04-05T22:07:27.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.057+0000 7efcf0dc3700 1 Processor -- start 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.057+0000 7efcf0dc3700 1 -- start start 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.057+0000 7efcf0dc3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efcec0ffbd0 0x7efcec1975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.057+0000 7efcf0dc3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec105a40 0x7efcec197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.057+0000 7efcf0dc3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec198100 con 0x7efcec105a40 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.057+0000 7efcf0dc3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec198270 con 0x7efcec0ffbd0 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efce9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec105a40 0x7efcec197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efce9d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec105a40 0x7efcec197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34530/0 (socket says 172.21.15.121:34530) 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efce9d9b700 1 -- 172.21.15.121:0/4044142124 learned_addr learned my addr 172.21.15.121:0/4044142124 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efcea59c700 1 --2- 172.21.15.121:0/4044142124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efcec0ffbd0 0x7efcec1975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efce9d9b700 1 -- 172.21.15.121:0/4044142124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efcec0ffbd0 msgr2=0x7efcec1975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efce9d9b700 1 --2- 172.21.15.121:0/4044142124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efcec0ffbd0 0x7efcec1975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efce9d9b700 1 -- 172.21.15.121:0/4044142124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc0095f0 con 0x7efcec105a40 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efcea59c700 1 --2- 172.21.15.121:0/4044142124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efcec0ffbd0 0x7efcec1975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:07:27.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efce9d9b700 1 --2- 172.21.15.121:0/4044142124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec105a40 0x7efcec197ae0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7efcd400b4b0 tx=0x7efcd400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efce37fe700 1 -- 172.21.15.121:0/4044142124 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcd4011800 con 0x7efcec105a40 2024-04-05T22:07:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efce37fe700 1 -- 172.21.15.121:0/4044142124 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efcd4011e00 con 0x7efcec105a40 2024-04-05T22:07:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efce37fe700 1 -- 172.21.15.121:0/4044142124 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efcd4010e50 con 0x7efcec105a40 2024-04-05T22:07:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efcf0dc3700 1 -- 172.21.15.121:0/4044142124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcec1a0610 con 0x7efcec105a40 2024-04-05T22:07:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.058+0000 7efcf0dc3700 1 -- 172.21.15.121:0/4044142124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec1a0b00 con 0x7efcec105a40 2024-04-05T22:07:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.059+0000 7efcf0dc3700 1 -- 172.21.15.121:0/4044142124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcec108040 con 0x7efcec105a40 2024-04-05T22:07:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.059+0000 7efce37fe700 1 -- 172.21.15.121:0/4044142124 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7efcd4010380 con 0x7efcec105a40 2024-04-05T22:07:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.060+0000 7efce37fe700 1 --2- 172.21.15.121:0/4044142124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efcd806cce0 0x7efcd806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.060+0000 7efce37fe700 1 -- 172.21.15.121:0/4044142124 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7efcd408c130 con 0x7efcec105a40 2024-04-05T22:07:27.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.060+0000 7efcea59c700 1 --2- 172.21.15.121:0/4044142124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efcd806cce0 0x7efcd806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:27.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.060+0000 7efcea59c700 1 --2- 172.21.15.121:0/4044142124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efcd806cce0 0x7efcd806f1a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00b5c0 tx=0x7efcdc01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:27.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.065+0000 7efce37fe700 1 -- 172.21.15.121:0/4044142124 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7efcd4056220 con 0x7efcec105a40 2024-04-05T22:07:27.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.268+0000 7efcf0dc3700 1 -- 172.21.15.121:0/4044142124 --> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7efcec02cce0 con 0x7efcd806cce0 2024-04-05T22:07:27.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.279+0000 7efce37fe700 1 -- 172.21.15.121:0/4044142124 <== mgr.14656 v2:172.21.15.121:6800/2306077913 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7efcec02cce0 con 0x7efcd806cce0 2024-04-05T22:07:27.279 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-05T22:07:27.279 INFO:teuthology.orchestra.run.smithi121.stdout:alertmanager.smithi121 smithi121 *:9093,9094 running (38m) 90s ago 49m 26.8M - 0.25.0 c8568f914cd2 be860acfe6aa 2024-04-05T22:07:27.279 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi121 smithi121 running (102s) 90s ago 50m 7340k - 18.2.2-1173-g8919605b 067bdb249792 476a804f366b 2024-04-05T22:07:27.279 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-exporter.smithi188 smithi188 running (96s) 92s ago 48m 6883k - 18.2.2-1173-g8919605b 067bdb249792 c07757ef70c2 2024-04-05T22:07:27.279 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi121 smithi121 running (36m) 90s ago 49m 6647k - 18.2.2-1173-g8919605b 067bdb249792 efab5ba4135b 2024-04-05T22:07:27.279 INFO:teuthology.orchestra.run.smithi121.stdout:crash.smithi188 smithi188 running (36m) 92s ago 48m 6651k - 18.2.2-1173-g8919605b 067bdb249792 fabb358c5883 2024-04-05T22:07:27.279 INFO:teuthology.orchestra.run.smithi121.stdout:grafana.smithi121 smithi121 *:3000 running (38m) 90s ago 49m 89.0M - 9.4.7 954c08fa6188 ceea4140d5d9 2024-04-05T22:07:27.279 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.gxheqt smithi121 running (32m) 90s ago 43m 69.9M - 18.2.2-1173-g8919605b 067bdb249792 3cbd88412b67 2024-04-05T22:07:27.279 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi121.ifoxmw smithi121 running (32m) 90s ago 43m 65.7M - 18.2.2-1173-g8919605b 067bdb249792 afe05e8aeef4 2024-04-05T22:07:27.279 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.bjmisu smithi188 running (32m) 92s ago 43m 127M - 18.2.2-1173-g8919605b 067bdb249792 8780efa9f829 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:mds.cephfs.smithi188.latnft smithi188 running (16m) 92s ago 43m 148M - 18.2.2-1173-g8919605b 067bdb249792 c6a2530d6d9f 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi121.etbquh smithi121 *:8443,9283,8765 running (40m) 90s ago 51m 516M - 18.2.2-1173-g8919605b 067bdb249792 c63a5f19bbf9 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.smithi188.tgcuir smithi188 *:8443,9283,8765 running (39m) 92s ago 48m 457M - 18.2.2-1173-g8919605b 067bdb249792 0158380ba48e 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi121 smithi121 running (37m) 90s ago 51m 82.2M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a9b277ce0e7 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:mon.smithi188 smithi188 running (36m) 92s ago 48m 56.4M 2048M 18.2.2-1173-g8919605b 067bdb249792 1a1944725b7d 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi121 smithi121 *:9100 running (39m) 90s ago 49m 21.4M - 1.5.0 0da6a335fe13 0384e495ed7d 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:node-exporter.smithi188 smithi188 *:9100 running (39m) 92s ago 48m 17.8M - 1.5.0 0da6a335fe13 f8eec6af8f6d 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (35m) 90s ago 47m 103M 1227M 18.2.2-1173-g8919605b 067bdb249792 698c6ac23041 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi121 running (35m) 90s ago 47m 118M 1227M 18.2.2-1173-g8919605b 067bdb249792 0fd0bdd316f7 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi121 running (34m) 90s ago 46m 98.9M 1227M 18.2.2-1173-g8919605b 067bdb249792 6b8b7f7862d3 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi188 running (34m) 92s ago 46m 133M 2251M 18.2.2-1173-g8919605b 067bdb249792 8607be213a4a 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi188 running (33m) 92s ago 45m 107M 2251M 18.2.2-1173-g8919605b 067bdb249792 e7985264d99e 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi188 running (33m) 92s ago 45m 98.5M 2251M 18.2.2-1173-g8919605b 067bdb249792 62b26c18a78c 2024-04-05T22:07:27.280 INFO:teuthology.orchestra.run.smithi121.stdout:prometheus.smithi121 smithi121 *:9095 running (38m) 90s ago 49m 89.4M - 2.43.0 a07b618ecd1d 31262beab782 2024-04-05T22:07:27.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.281+0000 7efcf0dc3700 1 -- 172.21.15.121:0/4044142124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efcd806cce0 msgr2=0x7efcd806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:27.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.281+0000 7efcf0dc3700 1 --2- 172.21.15.121:0/4044142124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efcd806cce0 0x7efcd806f1a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00b5c0 tx=0x7efcdc01a040 comp rx=0 tx=0).stop 2024-04-05T22:07:27.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.281+0000 7efcf0dc3700 1 -- 172.21.15.121:0/4044142124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec105a40 msgr2=0x7efcec197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:27.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.281+0000 7efcf0dc3700 1 --2- 172.21.15.121:0/4044142124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec105a40 0x7efcec197ae0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7efcd400b4b0 tx=0x7efcd400b870 comp rx=0 tx=0).stop 2024-04-05T22:07:27.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.281+0000 7efcf0dc3700 1 -- 172.21.15.121:0/4044142124 shutdown_connections 2024-04-05T22:07:27.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.281+0000 7efcf0dc3700 1 --2- 172.21.15.121:0/4044142124 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7efcd806cce0 0x7efcd806f1a0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:27.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.281+0000 7efcf0dc3700 1 --2- 172.21.15.121:0/4044142124 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7efcec0ffbd0 0x7efcec1975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:27.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.281+0000 7efcf0dc3700 1 --2- 172.21.15.121:0/4044142124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec105a40 0x7efcec197ae0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:27.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.281+0000 7efcf0dc3700 1 -- 172.21.15.121:0/4044142124 >> 172.21.15.121:0/4044142124 conn(0x7efcec0755a0 msgr2=0x7efcec0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:27.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.281+0000 7efcf0dc3700 1 -- 172.21.15.121:0/4044142124 shutdown_connections 2024-04-05T22:07:27.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:27.281+0000 7efcf0dc3700 1 -- 172.21.15.121:0/4044142124 wait complete. 2024-04-05T22:07:27.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph versions' 2024-04-05T22:07:27.972 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:27 smithi121 ceph-mon[145859]: pgmap v1239: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:28.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:27 smithi188 ceph-mon[107793]: pgmap v1239: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:28.174 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T22:07:29.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:28 smithi121 ceph-mon[145859]: from='client.35994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:07:29.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:28 smithi188 ceph-mon[107793]: from='client.35994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-05T22:07:29.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.901+0000 7eff4a156700 1 -- 172.21.15.121:0/774546048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff44100dc0 msgr2=0x7eff441011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:29.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.901+0000 7eff4a156700 1 --2- 172.21.15.121:0/774546048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff44100dc0 0x7eff441011a0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009910 tx=0x7eff2c009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.901+0000 7eff4a156700 1 -- 172.21.15.121:0/774546048 shutdown_connections 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.901+0000 7eff4a156700 1 --2- 172.21.15.121:0/774546048 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff44101770 0x7eff441057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.901+0000 7eff4a156700 1 --2- 172.21.15.121:0/774546048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff44100dc0 0x7eff441011a0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.901+0000 7eff4a156700 1 -- 172.21.15.121:0/774546048 >> 172.21.15.121:0/774546048 conn(0x7eff440fc650 msgr2=0x7eff440fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.901+0000 7eff4a156700 1 -- 172.21.15.121:0/774546048 shutdown_connections 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.901+0000 7eff4a156700 1 -- 172.21.15.121:0/774546048 wait complete. 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.902+0000 7eff4a156700 1 Processor -- start 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.902+0000 7eff4a156700 1 -- start start 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff4a156700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff44100dc0 0x7eff4419c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff4a156700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff44101770 0x7eff4419cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff4a156700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4419d190 con 0x7eff44101770 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff4a156700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff44196640 con 0x7eff44100dc0 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff42ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff44101770 0x7eff4419cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:29.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff42ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff44101770 0x7eff4419cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34546/0 (socket says 172.21.15.121:34546) 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff42ffd700 1 -- 172.21.15.121:0/1283929909 learned_addr learned my addr 172.21.15.121:0/1283929909 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff437fe700 1 --2- 172.21.15.121:0/1283929909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff44100dc0 0x7eff4419c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff42ffd700 1 -- 172.21.15.121:0/1283929909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff44100dc0 msgr2=0x7eff4419c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff42ffd700 1 --2- 172.21.15.121:0/1283929909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff44100dc0 0x7eff4419c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff42ffd700 1 -- 172.21.15.121:0/1283929909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c0095f0 con 0x7eff44101770 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff437fe700 1 --2- 172.21.15.121:0/1283929909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff44100dc0 0x7eff4419c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff42ffd700 1 --2- 172.21.15.121:0/1283929909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff44101770 0x7eff4419cb00 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7eff3400b480 tx=0x7eff3400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff40ff9700 1 -- 172.21.15.121:0/1283929909 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff340118c0 con 0x7eff44101770 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff4a156700 1 -- 172.21.15.121:0/1283929909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff44196920 con 0x7eff44101770 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff40ff9700 1 -- 172.21.15.121:0/1283929909 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff34010380 con 0x7eff44101770 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.903+0000 7eff40ff9700 1 -- 172.21.15.121:0/1283929909 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff3400f510 con 0x7eff44101770 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.904+0000 7eff4a156700 1 -- 172.21.15.121:0/1283929909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff44196e70 con 0x7eff44101770 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.904+0000 7eff4a156700 1 -- 172.21.15.121:0/1283929909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff44109160 con 0x7eff44101770 2024-04-05T22:07:29.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.905+0000 7eff40ff9700 1 -- 172.21.15.121:0/1283929909 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7eff340104f0 con 0x7eff44101770 2024-04-05T22:07:29.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.905+0000 7eff40ff9700 1 --2- 172.21.15.121:0/1283929909 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7eff3006cce0 0x7eff3006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:29.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.905+0000 7eff40ff9700 1 -- 172.21.15.121:0/1283929909 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7eff3408c210 con 0x7eff44101770 2024-04-05T22:07:29.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.905+0000 7eff437fe700 1 --2- 172.21.15.121:0/1283929909 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7eff3006cce0 0x7eff3006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:29.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.906+0000 7eff437fe700 1 --2- 172.21.15.121:0/1283929909 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7eff3006cce0 0x7eff3006f1a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00b5c0 tx=0x7eff2c005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:29.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:29.910+0000 7eff40ff9700 1 -- 172.21.15.121:0/1283929909 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7eff34059d10 con 0x7eff44101770 2024-04-05T22:07:29.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:29 smithi121 ceph-mon[145859]: pgmap v1240: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:30.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:30.184+0000 7eff4a156700 1 -- 172.21.15.121:0/1283929909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7eff441970f0 con 0x7eff44101770 2024-04-05T22:07:30.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:30.184+0000 7eff40ff9700 1 -- 172.21.15.121:0/1283929909 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7eff340107a0 con 0x7eff44101770 2024-04-05T22:07:30.184 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-05T22:07:30.184 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": { 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": { 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 2 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": { 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 6 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 4 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: "overall": { 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)": 14 2024-04-05T22:07:30.185 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-05T22:07:30.186 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-05T22:07:30.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:30.186+0000 7eff4a156700 1 -- 172.21.15.121:0/1283929909 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7eff3006cce0 msgr2=0x7eff3006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:30.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:30.186+0000 7eff4a156700 1 --2- 172.21.15.121:0/1283929909 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7eff3006cce0 0x7eff3006f1a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00b5c0 tx=0x7eff2c005960 comp rx=0 tx=0).stop 2024-04-05T22:07:30.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:30.186+0000 7eff4a156700 1 -- 172.21.15.121:0/1283929909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff44101770 msgr2=0x7eff4419cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:30.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:30.186+0000 7eff4a156700 1 --2- 172.21.15.121:0/1283929909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff44101770 0x7eff4419cb00 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7eff3400b480 tx=0x7eff3400b840 comp rx=0 tx=0).stop 2024-04-05T22:07:30.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:30.186+0000 7eff4a156700 1 -- 172.21.15.121:0/1283929909 shutdown_connections 2024-04-05T22:07:30.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:30.186+0000 7eff4a156700 1 --2- 172.21.15.121:0/1283929909 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7eff3006cce0 0x7eff3006f1a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:30.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:30.186+0000 7eff4a156700 1 --2- 172.21.15.121:0/1283929909 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff44100dc0 0x7eff4419c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:30.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:30.186+0000 7eff4a156700 1 --2- 172.21.15.121:0/1283929909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff44101770 0x7eff4419cb00 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:30.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:30.186+0000 7eff4a156700 1 -- 172.21.15.121:0/1283929909 >> 172.21.15.121:0/1283929909 conn(0x7eff440fc650 msgr2=0x7eff440fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:30.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:30.187+0000 7eff4a156700 1 -- 172.21.15.121:0/1283929909 shutdown_connections 2024-04-05T22:07:30.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:30.187+0000 7eff4a156700 1 -- 172.21.15.121:0/1283929909 wait complete. 2024-04-05T22:07:30.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:29 smithi188 ceph-mon[107793]: pgmap v1240: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:30.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-05T22:07:31.100 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T22:07:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:30 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1283929909' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:07:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:07:31.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:30 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1283929909' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:07:31.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:07:32.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:31 smithi121 ceph-mon[145859]: pgmap v1241: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:32.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:31 smithi188 ceph-mon[107793]: pgmap v1241: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:32.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.861+0000 7fc33a445700 1 -- 172.21.15.121:0/2861835050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3340730b0 msgr2=0x7fc334073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.861+0000 7fc33a445700 1 --2- 172.21.15.121:0/2861835050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3340730b0 0x7fc334073490 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fc31c009910 tx=0x7fc31c009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.861+0000 7fc33a445700 1 -- 172.21.15.121:0/2861835050 shutdown_connections 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.861+0000 7fc33a445700 1 --2- 172.21.15.121:0/2861835050 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3340739d0 0x7fc33410cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.861+0000 7fc33a445700 1 --2- 172.21.15.121:0/2861835050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3340730b0 0x7fc334073490 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.861+0000 7fc33a445700 1 -- 172.21.15.121:0/2861835050 >> 172.21.15.121:0/2861835050 conn(0x7fc3340fc630 msgr2=0x7fc3340fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.861+0000 7fc33a445700 1 -- 172.21.15.121:0/2861835050 shutdown_connections 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.861+0000 7fc33a445700 1 -- 172.21.15.121:0/2861835050 wait complete. 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc33a445700 1 Processor -- start 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc33a445700 1 -- start start 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc33a445700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3340730b0 0x7fc33419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc33a445700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3340739d0 0x7fc33419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc33a445700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33419f1f0 con 0x7fc3340730b0 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc33a445700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3341986a0 con 0x7fc3340739d0 2024-04-05T22:07:32.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc333fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3340730b0 0x7fc33419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:32.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc333fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3340730b0 0x7fc33419e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34550/0 (socket says 172.21.15.121:34550) 2024-04-05T22:07:32.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc333fff700 1 -- 172.21.15.121:0/1530362307 learned_addr learned my addr 172.21.15.121:0/1530362307 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:32.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc3337fe700 1 --2- 172.21.15.121:0/1530362307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3340739d0 0x7fc33419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:32.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc333fff700 1 -- 172.21.15.121:0/1530362307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3340739d0 msgr2=0x7fc33419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:32.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc333fff700 1 --2- 172.21.15.121:0/1530362307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3340739d0 0x7fc33419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:32.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc333fff700 1 -- 172.21.15.121:0/1530362307 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31c0095f0 con 0x7fc3340730b0 2024-04-05T22:07:32.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc333fff700 1 --2- 172.21.15.121:0/1530362307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3340730b0 0x7fc33419e620 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc31c006010 tx=0x7fc31c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:32.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.862+0000 7fc3317fa700 1 -- 172.21.15.121:0/1530362307 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc31c01d070 con 0x7fc3340730b0 2024-04-05T22:07:32.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.863+0000 7fc3317fa700 1 -- 172.21.15.121:0/1530362307 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc31c00f460 con 0x7fc3340730b0 2024-04-05T22:07:32.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.863+0000 7fc33a445700 1 -- 172.21.15.121:0/1530362307 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc334198920 con 0x7fc3340730b0 2024-04-05T22:07:32.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.863+0000 7fc3317fa700 1 -- 172.21.15.121:0/1530362307 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc31c021620 con 0x7fc3340730b0 2024-04-05T22:07:32.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.863+0000 7fc33a445700 1 -- 172.21.15.121:0/1530362307 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc334198e10 con 0x7fc3340730b0 2024-04-05T22:07:32.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.864+0000 7fc33a445700 1 -- 172.21.15.121:0/1530362307 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33410a5c0 con 0x7fc3340730b0 2024-04-05T22:07:32.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.864+0000 7fc3317fa700 1 -- 172.21.15.121:0/1530362307 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc31c02b440 con 0x7fc3340730b0 2024-04-05T22:07:32.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.864+0000 7fc3317fa700 1 --2- 172.21.15.121:0/1530362307 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc32006cce0 0x7fc32006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:32.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.865+0000 7fc3317fa700 1 -- 172.21.15.121:0/1530362307 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc31c08d7c0 con 0x7fc3340730b0 2024-04-05T22:07:32.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.865+0000 7fc3337fe700 1 --2- 172.21.15.121:0/1530362307 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc32006cce0 0x7fc32006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:32.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.865+0000 7fc3337fe700 1 --2- 172.21.15.121:0/1530362307 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc32006cce0 0x7fc32006f1a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc324005950 tx=0x7fc3240058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:32.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:32.869+0000 7fc3317fa700 1 -- 172.21.15.121:0/1530362307 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc31c05b2c0 con 0x7fc3340730b0 2024-04-05T22:07:33.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:33.144+0000 7fc33a445700 1 -- 172.21.15.121:0/1530362307 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc33402cc70 con 0x7fc3340730b0 2024-04-05T22:07:33.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:33.144+0000 7fc3317fa700 1 -- 172.21.15.121:0/1530362307 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fc31c05ae50 con 0x7fc3340730b0 2024-04-05T22:07:33.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:33.146+0000 7fc33a445700 1 -- 172.21.15.121:0/1530362307 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc32006cce0 msgr2=0x7fc32006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:33.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:33.146+0000 7fc33a445700 1 --2- 172.21.15.121:0/1530362307 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc32006cce0 0x7fc32006f1a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc324005950 tx=0x7fc3240058e0 comp rx=0 tx=0).stop 2024-04-05T22:07:33.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:33.146+0000 7fc33a445700 1 -- 172.21.15.121:0/1530362307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3340730b0 msgr2=0x7fc33419e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:33.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:33.146+0000 7fc33a445700 1 --2- 172.21.15.121:0/1530362307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3340730b0 0x7fc33419e620 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc31c006010 tx=0x7fc31c00be20 comp rx=0 tx=0).stop 2024-04-05T22:07:33.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:33.146+0000 7fc33a445700 1 -- 172.21.15.121:0/1530362307 shutdown_connections 2024-04-05T22:07:33.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:33.146+0000 7fc33a445700 1 --2- 172.21.15.121:0/1530362307 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc32006cce0 0x7fc32006f1a0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:33.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:33.146+0000 7fc33a445700 1 --2- 172.21.15.121:0/1530362307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3340730b0 0x7fc33419e620 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:33.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:33.146+0000 7fc33a445700 1 --2- 172.21.15.121:0/1530362307 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc3340739d0 0x7fc33419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:33.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:33.146+0000 7fc33a445700 1 -- 172.21.15.121:0/1530362307 >> 172.21.15.121:0/1530362307 conn(0x7fc3340fc630 msgr2=0x7fc334107700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:33.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:33.146+0000 7fc33a445700 1 -- 172.21.15.121:0/1530362307 shutdown_connections 2024-04-05T22:07:33.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:33.146+0000 7fc33a445700 1 -- 172.21.15.121:0/1530362307 wait complete. 2024-04-05T22:07:33.156 INFO:teuthology.orchestra.run.smithi121.stdout:true 2024-04-05T22:07:33.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -e sha1=8919605b0a5dc1f1650c8aec3051c77c0e3e240e -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-05T22:07:34.072 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T22:07:34.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:33 smithi121 ceph-mon[145859]: pgmap v1242: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:07:34.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:33 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1530362307' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:07:34.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:33 smithi188 ceph-mon[107793]: pgmap v1242: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:07:34.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:33 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1530362307' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:07:35.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.790+0000 7fbc7777b700 1 -- 172.21.15.121:0/3165741812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc700ff400 msgr2=0x7fbc700ff7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:35.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.790+0000 7fbc7777b700 1 --2- 172.21.15.121:0/3165741812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc700ff400 0x7fbc700ff7e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fbc60009910 tx=0x7fbc60009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:35.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.790+0000 7fbc7777b700 1 -- 172.21.15.121:0/3165741812 shutdown_connections 2024-04-05T22:07:35.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.790+0000 7fbc7777b700 1 --2- 172.21.15.121:0/3165741812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc700ffdb0 0x7fbc70103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.790+0000 7fbc7777b700 1 --2- 172.21.15.121:0/3165741812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc700ff400 0x7fbc700ff7e0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.790+0000 7fbc7777b700 1 -- 172.21.15.121:0/3165741812 >> 172.21.15.121:0/3165741812 conn(0x7fbc70075e30 msgr2=0x7fbc700fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.790+0000 7fbc7777b700 1 -- 172.21.15.121:0/3165741812 shutdown_connections 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.790+0000 7fbc7777b700 1 -- 172.21.15.121:0/3165741812 wait complete. 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.791+0000 7fbc7777b700 1 Processor -- start 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.791+0000 7fbc7777b700 1 -- start start 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.791+0000 7fbc7777b700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc700ff400 0x7fbc7019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.791+0000 7fbc7777b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc700ffdb0 0x7fbc7019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.791+0000 7fbc7777b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7019f270 con 0x7fbc700ffdb0 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.791+0000 7fbc7777b700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc70198720 con 0x7fbc700ff400 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.791+0000 7fbc74d16700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc700ffdb0 0x7fbc7019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.791+0000 7fbc74d16700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc700ffdb0 0x7fbc7019ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48364/0 (socket says 172.21.15.121:48364) 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.791+0000 7fbc74d16700 1 -- 172.21.15.121:0/2714577806 learned_addr learned my addr 172.21.15.121:0/2714577806 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:35.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.792+0000 7fbc75517700 1 --2- 172.21.15.121:0/2714577806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc700ff400 0x7fbc7019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:35.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.792+0000 7fbc74d16700 1 -- 172.21.15.121:0/2714577806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc700ff400 msgr2=0x7fbc7019e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:35.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.792+0000 7fbc74d16700 1 --2- 172.21.15.121:0/2714577806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc700ff400 0x7fbc7019e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:35.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.792+0000 7fbc74d16700 1 -- 172.21.15.121:0/2714577806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc600095f0 con 0x7fbc700ffdb0 2024-04-05T22:07:35.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.792+0000 7fbc75517700 1 --2- 172.21.15.121:0/2714577806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc700ff400 0x7fbc7019e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:07:35.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.792+0000 7fbc74d16700 1 --2- 172.21.15.121:0/2714577806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc700ffdb0 0x7fbc7019ebe0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00d900 tx=0x7fbc6c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:35.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.792+0000 7fbc667fc700 1 -- 172.21.15.121:0/2714577806 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc6c009900 con 0x7fbc700ffdb0 2024-04-05T22:07:35.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.792+0000 7fbc667fc700 1 -- 172.21.15.121:0/2714577806 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc6c010460 con 0x7fbc700ffdb0 2024-04-05T22:07:35.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.792+0000 7fbc7777b700 1 -- 172.21.15.121:0/2714577806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc70198a00 con 0x7fbc700ffdb0 2024-04-05T22:07:35.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.792+0000 7fbc667fc700 1 -- 172.21.15.121:0/2714577806 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc6c00f5f0 con 0x7fbc700ffdb0 2024-04-05T22:07:35.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.792+0000 7fbc7777b700 1 -- 172.21.15.121:0/2714577806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc70198f50 con 0x7fbc700ffdb0 2024-04-05T22:07:35.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.793+0000 7fbc7777b700 1 -- 172.21.15.121:0/2714577806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc7004ea90 con 0x7fbc700ffdb0 2024-04-05T22:07:35.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.793+0000 7fbc667fc700 1 -- 172.21.15.121:0/2714577806 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbc6c0105d0 con 0x7fbc700ffdb0 2024-04-05T22:07:35.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.794+0000 7fbc667fc700 1 --2- 172.21.15.121:0/2714577806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbc5c06cce0 0x7fbc5c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:35.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.794+0000 7fbc667fc700 1 -- 172.21.15.121:0/2714577806 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbc6c08c3c0 con 0x7fbc700ffdb0 2024-04-05T22:07:35.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.794+0000 7fbc75517700 1 --2- 172.21.15.121:0/2714577806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbc5c06cce0 0x7fbc5c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:35.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.794+0000 7fbc75517700 1 --2- 172.21.15.121:0/2714577806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbc5c06cce0 0x7fbc5c06f1a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000b5c0 tx=0x7fbc60005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:35.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:35.798+0000 7fbc667fc700 1 -- 172.21.15.121:0/2714577806 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbc6c059ec0 con 0x7fbc700ffdb0 2024-04-05T22:07:36.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:36.073+0000 7fbc7777b700 1 -- 172.21.15.121:0/2714577806 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbc70199c30 con 0x7fbc700ffdb0 2024-04-05T22:07:36.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:36.074+0000 7fbc667fc700 1 -- 172.21.15.121:0/2714577806 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+633 (secure 0 0 0) 0x7fbc6c059a50 con 0x7fbc700ffdb0 2024-04-05T22:07:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:36.076+0000 7fbc7777b700 1 -- 172.21.15.121:0/2714577806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbc5c06cce0 msgr2=0x7fbc5c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:36.076+0000 7fbc7777b700 1 --2- 172.21.15.121:0/2714577806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbc5c06cce0 0x7fbc5c06f1a0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000b5c0 tx=0x7fbc60005960 comp rx=0 tx=0).stop 2024-04-05T22:07:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:36.076+0000 7fbc7777b700 1 -- 172.21.15.121:0/2714577806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc700ffdb0 msgr2=0x7fbc7019ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:36.076+0000 7fbc7777b700 1 --2- 172.21.15.121:0/2714577806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc700ffdb0 0x7fbc7019ebe0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00d900 tx=0x7fbc6c00dcc0 comp rx=0 tx=0).stop 2024-04-05T22:07:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:36.076+0000 7fbc7777b700 1 -- 172.21.15.121:0/2714577806 shutdown_connections 2024-04-05T22:07:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:36.076+0000 7fbc7777b700 1 --2- 172.21.15.121:0/2714577806 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbc5c06cce0 0x7fbc5c06f1a0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:36.076+0000 7fbc7777b700 1 --2- 172.21.15.121:0/2714577806 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbc700ff400 0x7fbc7019e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:36.076+0000 7fbc7777b700 1 --2- 172.21.15.121:0/2714577806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc700ffdb0 0x7fbc7019ebe0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:36.076+0000 7fbc7777b700 1 -- 172.21.15.121:0/2714577806 >> 172.21.15.121:0/2714577806 conn(0x7fbc70075e30 msgr2=0x7fbc701027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:36.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:36.076+0000 7fbc7777b700 1 -- 172.21.15.121:0/2714577806 shutdown_connections 2024-04-05T22:07:36.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:36.076+0000 7fbc7777b700 1 -- 172.21.15.121:0/2714577806 wait complete. 2024-04-05T22:07:36.086 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph version 18.2.2-1173-g8919605b (8919605b0a5dc1f1650c8aec3051c77c0e3e240e) reef (stable)" 2024-04-05T22:07:36.103 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:35 smithi121 ceph-mon[145859]: pgmap v1243: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:07:36.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:35 smithi188 ceph-mon[107793]: pgmap v1243: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:07:36.747 DEBUG:teuthology.parallel:result is None 2024-04-05T22:07:36.747 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-05T22:07:36.760 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T22:07:36.760 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-05T22:07:36.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:36 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2714577806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:07:36.988 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/mon.smithi121/config 2024-04-05T22:07:37.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:36 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2714577806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-05T22:07:38.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:37 smithi188 ceph-mon[107793]: pgmap v1244: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:38.126 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:37 smithi121 ceph-mon[145859]: pgmap v1244: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:38.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.663+0000 7f28e1700700 1 -- 172.21.15.121:0/1151581583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28dc0ffee0 msgr2=0x7f28dc105870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:38.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.664+0000 7f28e1700700 1 --2- 172.21.15.121:0/1151581583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28dc0ffee0 0x7f28dc105870 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f28d0009910 tx=0x7f28d0009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:38.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.664+0000 7f28e1700700 1 -- 172.21.15.121:0/1151581583 shutdown_connections 2024-04-05T22:07:38.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.664+0000 7f28e1700700 1 --2- 172.21.15.121:0/1151581583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28dc0ffee0 0x7f28dc105870 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.664+0000 7f28e1700700 1 --2- 172.21.15.121:0/1151581583 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28dc0ff530 0x7f28dc0ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.664+0000 7f28e1700700 1 -- 172.21.15.121:0/1151581583 >> 172.21.15.121:0/1151581583 conn(0x7f28dc074df0 msgr2=0x7f28dc075200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.664+0000 7f28e1700700 1 -- 172.21.15.121:0/1151581583 shutdown_connections 2024-04-05T22:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.664+0000 7f28e1700700 1 -- 172.21.15.121:0/1151581583 wait complete. 2024-04-05T22:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.664+0000 7f28e1700700 1 Processor -- start 2024-04-05T22:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.664+0000 7f28e1700700 1 -- start start 2024-04-05T22:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28e1700700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28dc0ff530 0x7f28dc19c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28e1700700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28dc0ffee0 0x7f28dc19ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28e1700700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc19d0c0 con 0x7f28dc0ffee0 2024-04-05T22:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28e1700700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc196570 con 0x7f28dc0ff530 2024-04-05T22:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28da7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28dc0ffee0 0x7f28dc19ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28da7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28dc0ffee0 0x7f28dc19ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48384/0 (socket says 172.21.15.121:48384) 2024-04-05T22:07:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28da7fc700 1 -- 172.21.15.121:0/3318658430 learned_addr learned my addr 172.21.15.121:0/3318658430 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28daffd700 1 --2- 172.21.15.121:0/3318658430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28dc0ff530 0x7f28dc19c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28da7fc700 1 -- 172.21.15.121:0/3318658430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28dc0ff530 msgr2=0x7f28dc19c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28da7fc700 1 --2- 172.21.15.121:0/3318658430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28dc0ff530 0x7f28dc19c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28da7fc700 1 -- 172.21.15.121:0/3318658430 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d00095f0 con 0x7f28dc0ffee0 2024-04-05T22:07:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28da7fc700 1 --2- 172.21.15.121:0/3318658430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28dc0ffee0 0x7f28dc19ca30 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f28d000bd70 tx=0x7f28d000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28c3fff700 1 -- 172.21.15.121:0/3318658430 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28d001d070 con 0x7f28dc0ffee0 2024-04-05T22:07:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28c3fff700 1 -- 172.21.15.121:0/3318658430 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28d000f460 con 0x7f28dc0ffee0 2024-04-05T22:07:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28e1700700 1 -- 172.21.15.121:0/3318658430 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28dc1967f0 con 0x7f28dc0ffee0 2024-04-05T22:07:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28c3fff700 1 -- 172.21.15.121:0/3318658430 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28d0021640 con 0x7f28dc0ffee0 2024-04-05T22:07:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.665+0000 7f28e1700700 1 -- 172.21.15.121:0/3318658430 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28dc196ce0 con 0x7f28dc0ffee0 2024-04-05T22:07:38.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.666+0000 7f28e1700700 1 -- 172.21.15.121:0/3318658430 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28dc109150 con 0x7f28dc0ffee0 2024-04-05T22:07:38.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.667+0000 7f28c3fff700 1 -- 172.21.15.121:0/3318658430 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f28d000f9e0 con 0x7f28dc0ffee0 2024-04-05T22:07:38.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.667+0000 7f28c3fff700 1 --2- 172.21.15.121:0/3318658430 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28c406cce0 0x7f28c406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:38.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.667+0000 7f28c3fff700 1 -- 172.21.15.121:0/3318658430 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f28d008d6b0 con 0x7f28dc0ffee0 2024-04-05T22:07:38.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.667+0000 7f28daffd700 1 --2- 172.21.15.121:0/3318658430 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28c406cce0 0x7f28c406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:38.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.668+0000 7f28daffd700 1 --2- 172.21.15.121:0/3318658430 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28c406cce0 0x7f28c406f1a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f28cc005950 tx=0x7f28cc00b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:38.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.672+0000 7f28c3fff700 1 -- 172.21.15.121:0/3318658430 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f28d005b1b0 con 0x7f28dc0ffee0 2024-04-05T22:07:38.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.909+0000 7f28e1700700 1 -- 172.21.15.121:0/3318658430 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f28dc108b30 con 0x7f28dc0ffee0 2024-04-05T22:07:38.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.909+0000 7f28c3fff700 1 -- 172.21.15.121:0/3318658430 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 41 v41) v1 ==== 76+0+1880 (secure 0 0 0) 0x7f28d005ad40 con 0x7f28dc0ffee0 2024-04-05T22:07:38.909 INFO:teuthology.orchestra.run.smithi121.stdout:e41 2024-04-05T22:07:38.910 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-05T22:07:38.910 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:07:38.910 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-04-05T22:07:38.910 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:38.910 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-04-05T22:07:38.910 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-04-05T22:07:38.910 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 41 2024-04-05T22:07:38.910 INFO:teuthology.orchestra.run.smithi121.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-05T22:07:38.910 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-04-05T21:23:43.437872+0000 2024-04-05T22:07:38.910 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-04-05T22:06:01.604327+0000 2024-04-05T22:07:38.911 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-04-05T22:07:38.911 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-04-05T22:07:38.911 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 60 2024-04-05T22:07:38.911 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-04-05T22:07:38.911 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-04-05T22:07:38.911 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-04-05T22:07:38.911 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-04-05T22:07:38.911 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-04-05T22:07:38.911 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 80 2024-04-05T22:07:38.911 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-05T22:07:38.911 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 2 2024-04-05T22:07:38.911 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1 2024-04-05T22:07:38.912 INFO:teuthology.orchestra.run.smithi121.stdout:up {0=34360,1=34364} 2024-04-05T22:07:38.912 INFO:teuthology.orchestra.run.smithi121.stdout:failed 2024-04-05T22:07:38.912 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-04-05T22:07:38.912 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-04-05T22:07:38.912 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3] 2024-04-05T22:07:38.912 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-04-05T22:07:38.912 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-04-05T22:07:38.912 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-04-05T22:07:38.912 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-04-05T22:07:38.912 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 1 2024-04-05T22:07:38.912 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.gxheqt{0:34360} state up:active seq 469 join_fscid=1 addr [v2:172.21.15.121:6826/3939200727,v1:172.21.15.121:6827/3939200727] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:07:38.912 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.bjmisu{0:44297} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.188:6824/2880188624,v1:172.21.15.188:6825/2880188624] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:07:38.913 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi121.ifoxmw{1:34364} state up:active seq 463 join_fscid=1 addr [v2:172.21.15.121:6828/3468502927,v1:172.21.15.121:6829/3468502927] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:07:38.913 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.cephfs.smithi188.latnft{1:44827} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.188:6826/3202979994,v1:172.21.15.188:6827/3202979994] compat {c=[1],r=[1],i=[7ff]}] 2024-04-05T22:07:38.913 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:38.913 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:38.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.911+0000 7f28e1700700 1 -- 172.21.15.121:0/3318658430 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28c406cce0 msgr2=0x7f28c406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:38.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.911+0000 7f28e1700700 1 --2- 172.21.15.121:0/3318658430 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28c406cce0 0x7f28c406f1a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f28cc005950 tx=0x7f28cc00b410 comp rx=0 tx=0).stop 2024-04-05T22:07:38.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.911+0000 7f28e1700700 1 -- 172.21.15.121:0/3318658430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28dc0ffee0 msgr2=0x7f28dc19ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:38.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.911+0000 7f28e1700700 1 --2- 172.21.15.121:0/3318658430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28dc0ffee0 0x7f28dc19ca30 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f28d000bd70 tx=0x7f28d000bda0 comp rx=0 tx=0).stop 2024-04-05T22:07:38.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.911+0000 7f28e1700700 1 -- 172.21.15.121:0/3318658430 shutdown_connections 2024-04-05T22:07:38.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.911+0000 7f28e1700700 1 --2- 172.21.15.121:0/3318658430 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f28c406cce0 0x7f28c406f1a0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:38.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.911+0000 7f28e1700700 1 --2- 172.21.15.121:0/3318658430 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f28dc0ff530 0x7f28dc19c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:38.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.911+0000 7f28e1700700 1 --2- 172.21.15.121:0/3318658430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28dc0ffee0 0x7f28dc19ca30 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:38.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.911+0000 7f28e1700700 1 -- 172.21.15.121:0/3318658430 >> 172.21.15.121:0/3318658430 conn(0x7f28dc074df0 msgr2=0x7f28dc1050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:38.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.911+0000 7f28e1700700 1 -- 172.21.15.121:0/3318658430 shutdown_connections 2024-04-05T22:07:38.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:38.912+0000 7f28e1700700 1 -- 172.21.15.121:0/3318658430 wait complete. 2024-04-05T22:07:38.915 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 41 2024-04-05T22:07:39.546 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-04-05T22:07:39.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T22:07:39.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.688+0000 7fabcb74d700 1 -- 172.21.15.121:0/303660254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabc40ff420 msgr2=0x7fabc40ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:39.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.688+0000 7fabcb74d700 1 --2- 172.21.15.121:0/303660254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabc40ff420 0x7fabc40ff800 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009910 tx=0x7fabb4009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:39.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.688+0000 7fabcb74d700 1 -- 172.21.15.121:0/303660254 shutdown_connections 2024-04-05T22:07:39.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.688+0000 7fabcb74d700 1 --2- 172.21.15.121:0/303660254 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabc40ffdd0 0x7fabc4105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:39.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.688+0000 7fabcb74d700 1 --2- 172.21.15.121:0/303660254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabc40ff420 0x7fabc40ff800 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:39.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.688+0000 7fabcb74d700 1 -- 172.21.15.121:0/303660254 >> 172.21.15.121:0/303660254 conn(0x7fabc4074c20 msgr2=0x7fabc4075030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:39.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.688+0000 7fabcb74d700 1 -- 172.21.15.121:0/303660254 shutdown_connections 2024-04-05T22:07:39.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.688+0000 7fabcb74d700 1 -- 172.21.15.121:0/303660254 wait complete. 2024-04-05T22:07:39.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.689+0000 7fabcb74d700 1 Processor -- start 2024-04-05T22:07:39.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.689+0000 7fabcb74d700 1 -- start start 2024-04-05T22:07:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.689+0000 7fabcb74d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabc40ff420 0x7fabc4196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.689+0000 7fabcb74d700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabc4196a50 0x7fabc419aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.689+0000 7fabcb74d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc4197070 con 0x7fabc40ff420 2024-04-05T22:07:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.689+0000 7fabcb74d700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc41971e0 con 0x7fabc4196a50 2024-04-05T22:07:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.689+0000 7fabc94e9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabc40ff420 0x7fabc4196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.689+0000 7fabc94e9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabc40ff420 0x7fabc4196510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48400/0 (socket says 172.21.15.121:48400) 2024-04-05T22:07:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.689+0000 7fabc94e9700 1 -- 172.21.15.121:0/2708079375 learned_addr learned my addr 172.21.15.121:0/2708079375 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.690+0000 7fabc94e9700 1 -- 172.21.15.121:0/2708079375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabc4196a50 msgr2=0x7fabc419aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.690+0000 7fabc8ce8700 1 --2- 172.21.15.121:0/2708079375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabc4196a50 0x7fabc419aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.690+0000 7fabc94e9700 1 --2- 172.21.15.121:0/2708079375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabc4196a50 0x7fabc419aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.690+0000 7fabc94e9700 1 -- 172.21.15.121:0/2708079375 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb40095f0 con 0x7fabc40ff420 2024-04-05T22:07:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.690+0000 7fabc8ce8700 1 --2- 172.21.15.121:0/2708079375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabc4196a50 0x7fabc419aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:07:39.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.690+0000 7fabc94e9700 1 --2- 172.21.15.121:0/2708079375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabc40ff420 0x7fabc4196510 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fabb4006010 tx=0x7fabb400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:39.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.690+0000 7fabba7fc700 1 -- 172.21.15.121:0/2708079375 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabb401d070 con 0x7fabc40ff420 2024-04-05T22:07:39.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.690+0000 7fabba7fc700 1 -- 172.21.15.121:0/2708079375 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabb400f460 con 0x7fabc40ff420 2024-04-05T22:07:39.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.690+0000 7fabcb74d700 1 -- 172.21.15.121:0/2708079375 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc419b4b0 con 0x7fabc40ff420 2024-04-05T22:07:39.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.690+0000 7fabba7fc700 1 -- 172.21.15.121:0/2708079375 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabb4021640 con 0x7fabc40ff420 2024-04-05T22:07:39.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.690+0000 7fabcb74d700 1 -- 172.21.15.121:0/2708079375 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc419b9b0 con 0x7fabc40ff420 2024-04-05T22:07:39.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.691+0000 7fabcb74d700 1 -- 172.21.15.121:0/2708079375 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc41090a0 con 0x7fabc40ff420 2024-04-05T22:07:39.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.691+0000 7fabba7fc700 1 -- 172.21.15.121:0/2708079375 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fabb400fa60 con 0x7fabc40ff420 2024-04-05T22:07:39.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.692+0000 7fabba7fc700 1 --2- 172.21.15.121:0/2708079375 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fabb006cda0 0x7fabb006f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:39.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.692+0000 7fabba7fc700 1 -- 172.21.15.121:0/2708079375 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fabb408d7e0 con 0x7fabc40ff420 2024-04-05T22:07:39.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.692+0000 7fabc8ce8700 1 --2- 172.21.15.121:0/2708079375 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fabb006cda0 0x7fabb006f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:39.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.692+0000 7fabc8ce8700 1 --2- 172.21.15.121:0/2708079375 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fabb006cda0 0x7fabb006f260 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fabc0005fd0 tx=0x7fabc0005dc0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:39.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.696+0000 7fabba7fc700 1 -- 172.21.15.121:0/2708079375 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fabb40578d0 con 0x7fabc40ff420 2024-04-05T22:07:39.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.891+0000 7fabcb74d700 1 -- 172.21.15.121:0/2708079375 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabc404ea90 con 0x7fabc40ff420 2024-04-05T22:07:39.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.892+0000 7fabba7fc700 1 -- 172.21.15.121:0/2708079375 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16341 (secure 0 0 0) 0x7fabb405aef0 con 0x7fabc40ff420 2024-04-05T22:07:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:39.892 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":84,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","created":"2024-04-05T21:15:58.797109+0000","modified":"2024-04-05T21:50:31.003759+0000","last_up_change":"2024-04-05T21:34:34.199575+0000","last_in_change":"2024-04-05T21:21:56.806570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T21:20:53.291734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-05T21:23:42.376376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-05T21:23:42.658469+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"133652f8-9fe2-45e9-90b6-36f89191114b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2472269034},{"type":"v1","addr":"172.21.15.121:6803","nonce":2472269034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2472269034},{"type":"v1","addr":"172.21.15.121:6805","nonce":2472269034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":2472269034},{"type":"v1","addr":"172.21.15.121:6809","nonce":2472269034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2472269034},{"type":"v1","addr":"172.21.15.121:6807","nonce":2472269034}]},"public_addr":"172.21.15.121:6803/2472269034","cluster_addr":"172.21.15.121:6805/2472269034","heartbeat_back_addr":"172.21.15.121:6809/2472269034","heartbeat_front_addr":"172.21.15.121:6807/2472269034","state":["exists","up"]},{"osd":1,"uuid":"6695518f-c4a5-416a-8af1-2899fdebefd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":627590186},{"type":"v1","addr":"172.21.15.121:6811","nonce":627590186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":627590186},{"type":"v1","addr":"172.21.15.121:6813","nonce":627590186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":627590186},{"type":"v1","addr":"172.21.15.121:6817","nonce":627590186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":627590186},{"type":"v1","addr":"172.21.15.121:6815","nonce":627590186}]},"public_addr":"172.21.15.121:6811/627590186","cluster_addr":"172.21.15.121:6813/627590186","heartbeat_back_addr":"172.21.15.121:6817/627590186","heartbeat_front_addr":"172.21.15.121:6815/627590186","state":["exists","up"]},{"osd":2,"uuid":"e3b6290b-ef26-4cda-96e2-8bb5649219a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2092662684},{"type":"v1","addr":"172.21.15.121:6819","nonce":2092662684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2092662684},{"type":"v1","addr":"172.21.15.121:6821","nonce":2092662684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2092662684},{"type":"v1","addr":"172.21.15.121:6825","nonce":2092662684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2092662684},{"type":"v1","addr":"172.21.15.121:6823","nonce":2092662684}]},"public_addr":"172.21.15.121:6819/2092662684","cluster_addr":"172.21.15.121:6821/2092662684","heartbeat_back_addr":"172.21.15.121:6825/2092662684","heartbeat_front_addr":"172.21.15.121:6823/2092662684","state":["exists","up"]},{"osd":3,"uuid":"77fa8da5-d202-45f1-b6fa-5af969e9300c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":83582134},{"type":"v1","addr":"172.21.15.188:6801","nonce":83582134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":83582134},{"type":"v1","addr":"172.21.15.188:6803","nonce":83582134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":83582134},{"type":"v1","addr":"172.21.15.188:6807","nonce":83582134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":83582134},{"type":"v1","addr":"172.21.15.188:6805","nonce":83582134}]},"public_addr":"172.21.15.188:6801/83582134","cluster_addr":"172.21.15.188:6803/83582134","heartbeat_back_addr":"172.21.15.188:6807/83582134","heartbeat_front_addr":"172.21.15.188:6805/83582134","state":["exists","up"]},{"osd":4,"uuid":"abf81f28-7473-45ef-a716-5ef16e038b6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3966070828},{"type":"v1","addr":"172.21.15.188:6809","nonce":3966070828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3966070828},{"type":"v1","addr":"172.21.15.188:6811","nonce":3966070828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3966070828},{"type":"v1","addr":"172.21.15.188:6815","nonce":3966070828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3966070828},{"type":"v1","addr":"172.21.15.188:6813","nonce":3966070828}]},"public_addr":"172.21.15.188:6809/3966070828","cluster_addr":"172.21.15.188:6811/3966070828","heartbeat_back_addr":"172.21.15.188:6815/3966070828","heartbeat_front_addr":"172.21.15.188:6813/3966070828","state":["exists","up"]},{"osd":5,"uuid":"700a347b-e303-447d-bf03-d6677955d140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3805501159},{"type":"v1","addr":"172.21.15.188:6817","nonce":3805501159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3805501159},{"type":"v1","addr":"172.21.15.188:6819","nonce":3805501159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3805501159},{"type":"v1","addr":"172.21.15.188:6823","nonce":3805501159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3805501159},{"type":"v1","addr":"172.21.15.188:6821","nonce":3805501159}]},"public_addr":"172.21.15.188:6817/3805501159","cluster_addr":"172.21.15.188:6819/3805501159","heartbeat_back_addr":"172.21.15.188:6823/3805501159","heartbeat_front_addr":"172.21.15.188:6821/3805501159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:19:48.038490+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:19.269394+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:51.162437+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:18.203341+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:46.805653+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:22:15.296548+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.188:6825/213881583":"2024-04-06T21:34:37.155434+0000","172.21.15.188:6824/213881583":"2024-04-06T21:34:37.155434+0000","172.21.15.121:6829/3760584056":"2024-04-06T21:34:37.155420+0000","172.21.15.121:6828/3760584056":"2024-04-06T21:34:37.155420+0000","172.21.15.121:0/566857212":"2024-04-06T21:26:08.566250+0000","172.21.15.121:6801/1237774071":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6800/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/1364585923":"2024-04-06T21:16:32.951887+0000","172.21.15.121:6801/2735531747":"2024-04-06T21:26:08.566250+0000","172.21.15.121:0/4016372676":"2024-04-06T21:26:08.566250+0000","172.21.15.188:6829/2212394428":"2024-04-06T21:27:30.099010+0000","172.21.15.188:6828/2212394428":"2024-04-06T21:27:30.099010+0000","172.21.15.121:0/42058318":"2024-04-06T21:17:08.626094+0000","172.21.15.188:0/2907182826":"2024-04-06T21:27:30.099010+0000","172.21.15.121:0/341396693":"2024-04-06T21:17:08.626094+0000","172.21.15.121:6827/1615448980":"2024-04-06T21:24:00.251505+0000","172.21.15.188:0/3491833383":"2024-04-06T21:27:30.099010+0000","172.21.15.121:6800/1237774071":"2024-04-06T21:18:24.954707+0000","172.21.15.121:0/1060423936":"2024-04-06T21:26:08.566250+0000","172.21.15.121:6800/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/3762940011":"2024-04-06T21:18:24.954707+0000","172.21.15.188:0/1842750729":"2024-04-06T21:27:30.099010+0000","172.21.15.121:6800/2735531747":"2024-04-06T21:26:08.566250+0000","172.21.15.121:0/2214582362":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/2536941662":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/2383002272":"2024-04-06T21:16:32.951887+0000","172.21.15.121:6801/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.188:0/6988375":"2024-04-06T21:27:30.099010+0000","172.21.15.121:0/3247654758":"2024-04-06T21:18:24.954707+0000","172.21.15.188:0/1621914365":"2024-04-06T21:27:30.099010+0000","172.21.15.121:6826/1615448980":"2024-04-06T21:24:00.251505+0000","172.21.15.188:0/4116965307":"2024-04-06T21:27:30.099010+0000","172.21.15.121:6801/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/3235459786":"2024-04-06T21:18:24.954707+0000","172.21.15.121:0/846460013":"2024-04-06T21:26:08.566250+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T22:07:39.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.895+0000 7fabcb74d700 1 -- 172.21.15.121:0/2708079375 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fabb006cda0 msgr2=0x7fabb006f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:39.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.895+0000 7fabcb74d700 1 --2- 172.21.15.121:0/2708079375 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fabb006cda0 0x7fabb006f260 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fabc0005fd0 tx=0x7fabc0005dc0 comp rx=0 tx=0).stop 2024-04-05T22:07:39.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.895+0000 7fabcb74d700 1 -- 172.21.15.121:0/2708079375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabc40ff420 msgr2=0x7fabc4196510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:39.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.895+0000 7fabcb74d700 1 --2- 172.21.15.121:0/2708079375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabc40ff420 0x7fabc4196510 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fabb4006010 tx=0x7fabb400be20 comp rx=0 tx=0).stop 2024-04-05T22:07:39.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.895+0000 7fabcb74d700 1 -- 172.21.15.121:0/2708079375 shutdown_connections 2024-04-05T22:07:39.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.895+0000 7fabcb74d700 1 --2- 172.21.15.121:0/2708079375 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fabb006cda0 0x7fabb006f260 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:39.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.895+0000 7fabcb74d700 1 --2- 172.21.15.121:0/2708079375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabc40ff420 0x7fabc4196510 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:39.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.895+0000 7fabcb74d700 1 --2- 172.21.15.121:0/2708079375 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fabc4196a50 0x7fabc419aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:39.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.895+0000 7fabcb74d700 1 -- 172.21.15.121:0/2708079375 >> 172.21.15.121:0/2708079375 conn(0x7fabc4074c20 msgr2=0x7fabc40feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:39.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.895+0000 7fabcb74d700 1 -- 172.21.15.121:0/2708079375 shutdown_connections 2024-04-05T22:07:39.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:39.895+0000 7fabcb74d700 1 -- 172.21.15.121:0/2708079375 wait complete. 2024-04-05T22:07:39.907 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T21:20:53.291734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-05T21:23:42.376376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-05T21:23:42.658469+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T22:07:39.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T22:07:40.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.033+0000 7fb9e23e2700 1 -- 172.21.15.121:0/2562290109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9dc105970 msgr2=0x7fb9dc105d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.033+0000 7fb9e23e2700 1 --2- 172.21.15.121:0/2562290109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9dc105970 0x7fb9dc105d50 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009910 tx=0x7fb9cc009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.033+0000 7fb9e23e2700 1 -- 172.21.15.121:0/2562290109 shutdown_connections 2024-04-05T22:07:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.033+0000 7fb9e23e2700 1 --2- 172.21.15.121:0/2562290109 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9dc0ffae0 0x7fb9dc0fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.033+0000 7fb9e23e2700 1 --2- 172.21.15.121:0/2562290109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9dc105970 0x7fb9dc105d50 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.033+0000 7fb9e23e2700 1 -- 172.21.15.121:0/2562290109 >> 172.21.15.121:0/2562290109 conn(0x7fb9dc0753d0 msgr2=0x7fb9dc0757e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.033+0000 7fb9e23e2700 1 -- 172.21.15.121:0/2562290109 shutdown_connections 2024-04-05T22:07:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.033+0000 7fb9e23e2700 1 -- 172.21.15.121:0/2562290109 wait complete. 2024-04-05T22:07:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.034+0000 7fb9e23e2700 1 Processor -- start 2024-04-05T22:07:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.034+0000 7fb9e23e2700 1 -- start start 2024-04-05T22:07:40.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.034+0000 7fb9e23e2700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9dc0ffae0 0x7fb9dc195510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:40.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.034+0000 7fb9e23e2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9dc105970 0x7fb9dc195a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:40.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.034+0000 7fb9e23e2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc196130 con 0x7fb9dc105970 2024-04-05T22:07:40.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.034+0000 7fb9e23e2700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc199eb0 con 0x7fb9dc0ffae0 2024-04-05T22:07:40.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.035+0000 7fb9db7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9dc105970 0x7fb9dc195a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:40.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.035+0000 7fb9db7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9dc105970 0x7fb9dc195a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48416/0 (socket says 172.21.15.121:48416) 2024-04-05T22:07:40.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.035+0000 7fb9db7fe700 1 -- 172.21.15.121:0/3487196746 learned_addr learned my addr 172.21.15.121:0/3487196746 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:40.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.035+0000 7fb9dbfff700 1 --2- 172.21.15.121:0/3487196746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9dc0ffae0 0x7fb9dc195510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.035+0000 7fb9db7fe700 1 -- 172.21.15.121:0/3487196746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9dc0ffae0 msgr2=0x7fb9dc195510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.035+0000 7fb9db7fe700 1 --2- 172.21.15.121:0/3487196746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9dc0ffae0 0x7fb9dc195510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.035+0000 7fb9db7fe700 1 -- 172.21.15.121:0/3487196746 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc0095f0 con 0x7fb9dc105970 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.035+0000 7fb9db7fe700 1 --2- 172.21.15.121:0/3487196746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9dc105970 0x7fb9dc195a50 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000d8d0 tx=0x7fb9d000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.035+0000 7fb9d97fa700 1 -- 172.21.15.121:0/3487196746 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9d0009940 con 0x7fb9dc105970 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.035+0000 7fb9e23e2700 1 -- 172.21.15.121:0/3487196746 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc19a190 con 0x7fb9dc105970 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.035+0000 7fb9d97fa700 1 -- 172.21.15.121:0/3487196746 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9d0010460 con 0x7fb9dc105970 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.035+0000 7fb9d97fa700 1 -- 172.21.15.121:0/3487196746 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9d000f5f0 con 0x7fb9dc105970 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.035+0000 7fb9e23e2700 1 -- 172.21.15.121:0/3487196746 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc19a6f0 con 0x7fb9dc105970 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.036+0000 7fb9e23e2700 1 -- 172.21.15.121:0/3487196746 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9dc107f60 con 0x7fb9dc105970 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.036+0000 7fb9d97fa700 1 -- 172.21.15.121:0/3487196746 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb9d00105d0 con 0x7fb9dc105970 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.037+0000 7fb9d97fa700 1 --2- 172.21.15.121:0/3487196746 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb9c406cce0 0x7fb9c406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.037+0000 7fb9d97fa700 1 -- 172.21.15.121:0/3487196746 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb9d008c5e0 con 0x7fb9dc105970 2024-04-05T22:07:40.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.037+0000 7fb9dbfff700 1 --2- 172.21.15.121:0/3487196746 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb9c406cce0 0x7fb9c406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:40.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.037+0000 7fb9dbfff700 1 --2- 172.21.15.121:0/3487196746 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb9c406cce0 0x7fb9c406f1a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00b5c0 tx=0x7fb9cc005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.041+0000 7fb9d97fa700 1 -- 172.21.15.121:0/3487196746 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb9d0056650 con 0x7fb9dc105970 2024-04-05T22:07:40.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.233+0000 7fb9e23e2700 1 -- 172.21.15.121:0/3487196746 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb9dc19a910 con 0x7fb9dc105970 2024-04-05T22:07:40.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.233+0000 7fb9d97fa700 1 -- 172.21.15.121:0/3487196746 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb9d0010880 con 0x7fb9dc105970 2024-04-05T22:07:40.233 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2024-04-05T22:07:40.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.235+0000 7fb9e23e2700 1 -- 172.21.15.121:0/3487196746 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb9c406cce0 msgr2=0x7fb9c406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:40.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.235+0000 7fb9e23e2700 1 --2- 172.21.15.121:0/3487196746 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb9c406cce0 0x7fb9c406f1a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00b5c0 tx=0x7fb9cc005fd0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.235+0000 7fb9e23e2700 1 -- 172.21.15.121:0/3487196746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9dc105970 msgr2=0x7fb9dc195a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:40.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.235+0000 7fb9e23e2700 1 --2- 172.21.15.121:0/3487196746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9dc105970 0x7fb9dc195a50 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000d8d0 tx=0x7fb9d000dc90 comp rx=0 tx=0).stop 2024-04-05T22:07:40.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.236+0000 7fb9e23e2700 1 -- 172.21.15.121:0/3487196746 shutdown_connections 2024-04-05T22:07:40.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.236+0000 7fb9e23e2700 1 --2- 172.21.15.121:0/3487196746 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb9c406cce0 0x7fb9c406f1a0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.236+0000 7fb9e23e2700 1 --2- 172.21.15.121:0/3487196746 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb9dc0ffae0 0x7fb9dc195510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.236+0000 7fb9e23e2700 1 --2- 172.21.15.121:0/3487196746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9dc105970 0x7fb9dc195a50 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.236+0000 7fb9e23e2700 1 -- 172.21.15.121:0/3487196746 >> 172.21.15.121:0/3487196746 conn(0x7fb9dc0753d0 msgr2=0x7fb9dc0fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:40.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.236+0000 7fb9e23e2700 1 -- 172.21.15.121:0/3487196746 shutdown_connections 2024-04-05T22:07:40.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.236+0000 7fb9e23e2700 1 -- 172.21.15.121:0/3487196746 wait complete. 2024-04-05T22:07:40.237 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:39 smithi121 ceph-mon[145859]: pgmap v1245: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:40.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:39 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3318658430' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:07:40.248 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-05T22:07:40.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:39 smithi188 ceph-mon[107793]: pgmap v1245: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:40.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:39 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3318658430' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-05T22:07:40.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.372+0000 7eff1c844700 1 -- 172.21.15.121:0/312000257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff18100a50 msgr2=0x7eff18100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:40.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.372+0000 7eff1c844700 1 --2- 172.21.15.121:0/312000257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff18100a50 0x7eff18100ed0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7eff08009910 tx=0x7eff08009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:40.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.372+0000 7eff1c844700 1 -- 172.21.15.121:0/312000257 shutdown_connections 2024-04-05T22:07:40.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.372+0000 7eff1c844700 1 --2- 172.21.15.121:0/312000257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff18100a50 0x7eff18100ed0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.372+0000 7eff1c844700 1 --2- 172.21.15.121:0/312000257 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff18106a60 0x7eff18106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.372+0000 7eff1c844700 1 -- 172.21.15.121:0/312000257 >> 172.21.15.121:0/312000257 conn(0x7eff180fc4e0 msgr2=0x7eff180fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:40.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.372+0000 7eff1c844700 1 -- 172.21.15.121:0/312000257 shutdown_connections 2024-04-05T22:07:40.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.372+0000 7eff1c844700 1 -- 172.21.15.121:0/312000257 wait complete. 2024-04-05T22:07:40.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.373+0000 7eff1c844700 1 Processor -- start 2024-04-05T22:07:40.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.373+0000 7eff1c844700 1 -- start start 2024-04-05T22:07:40.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.373+0000 7eff1c844700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff18100a50 0x7eff18073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:40.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.373+0000 7eff1c844700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff18106a60 0x7eff180735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:40.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.373+0000 7eff1c844700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff18073b10 con 0x7eff18100a50 2024-04-05T22:07:40.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.373+0000 7eff1c844700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff18073c50 con 0x7eff18106a60 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.373+0000 7eff1659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff18100a50 0x7eff18073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.373+0000 7eff1659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff18100a50 0x7eff18073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48436/0 (socket says 172.21.15.121:48436) 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.373+0000 7eff1659c700 1 -- 172.21.15.121:0/1711916818 learned_addr learned my addr 172.21.15.121:0/1711916818 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.373+0000 7eff15d9b700 1 --2- 172.21.15.121:0/1711916818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff18106a60 0x7eff180735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.373+0000 7eff1659c700 1 -- 172.21.15.121:0/1711916818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff18106a60 msgr2=0x7eff180735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.373+0000 7eff1659c700 1 --2- 172.21.15.121:0/1711916818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff18106a60 0x7eff180735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.373+0000 7eff1659c700 1 -- 172.21.15.121:0/1711916818 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff080095f0 con 0x7eff18100a50 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.374+0000 7eff1659c700 1 --2- 172.21.15.121:0/1711916818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff18100a50 0x7eff18073090 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00d8d0 tx=0x7eff0c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.374+0000 7eff077fe700 1 -- 172.21.15.121:0/1711916818 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff0c009940 con 0x7eff18100a50 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.374+0000 7eff077fe700 1 -- 172.21.15.121:0/1711916818 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff0c010460 con 0x7eff18100a50 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.374+0000 7eff1c844700 1 -- 172.21.15.121:0/1711916818 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff181a2760 con 0x7eff18100a50 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.374+0000 7eff077fe700 1 -- 172.21.15.121:0/1711916818 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff0c00f5f0 con 0x7eff18100a50 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.374+0000 7eff1c844700 1 -- 172.21.15.121:0/1711916818 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff181a2b80 con 0x7eff18100a50 2024-04-05T22:07:40.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.375+0000 7eff1c844700 1 -- 172.21.15.121:0/1711916818 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff18108fe0 con 0x7eff18100a50 2024-04-05T22:07:40.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.375+0000 7eff077fe700 1 -- 172.21.15.121:0/1711916818 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7eff0c0105d0 con 0x7eff18100a50 2024-04-05T22:07:40.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.375+0000 7eff077fe700 1 --2- 172.21.15.121:0/1711916818 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7eff0006cce0 0x7eff0006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:40.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.375+0000 7eff077fe700 1 -- 172.21.15.121:0/1711916818 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7eff0c08c670 con 0x7eff18100a50 2024-04-05T22:07:40.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.376+0000 7eff15d9b700 1 --2- 172.21.15.121:0/1711916818 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7eff0006cce0 0x7eff0006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:40.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.376+0000 7eff15d9b700 1 --2- 172.21.15.121:0/1711916818 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7eff0006cce0 0x7eff0006f1a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7eff18074880 tx=0x7eff0801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:40.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.380+0000 7eff077fe700 1 -- 172.21.15.121:0/1711916818 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7eff0c056650 con 0x7eff18100a50 2024-04-05T22:07:40.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.576+0000 7eff1c844700 1 -- 172.21.15.121:0/1711916818 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7eff1804ea90 con 0x7eff18100a50 2024-04-05T22:07:40.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.576+0000 7eff077fe700 1 -- 172.21.15.121:0/1711916818 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7eff0c010880 con 0x7eff18100a50 2024-04-05T22:07:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 32 2024-04-05T22:07:40.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.578+0000 7eff1c844700 1 -- 172.21.15.121:0/1711916818 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7eff0006cce0 msgr2=0x7eff0006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:40.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.578+0000 7eff1c844700 1 --2- 172.21.15.121:0/1711916818 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7eff0006cce0 0x7eff0006f1a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7eff18074880 tx=0x7eff0801a040 comp rx=0 tx=0).stop 2024-04-05T22:07:40.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.578+0000 7eff1c844700 1 -- 172.21.15.121:0/1711916818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff18100a50 msgr2=0x7eff18073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:40.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.578+0000 7eff1c844700 1 --2- 172.21.15.121:0/1711916818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff18100a50 0x7eff18073090 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00d8d0 tx=0x7eff0c00dc90 comp rx=0 tx=0).stop 2024-04-05T22:07:40.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.578+0000 7eff1c844700 1 -- 172.21.15.121:0/1711916818 shutdown_connections 2024-04-05T22:07:40.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.578+0000 7eff1c844700 1 --2- 172.21.15.121:0/1711916818 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7eff0006cce0 0x7eff0006f1a0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.578+0000 7eff1c844700 1 --2- 172.21.15.121:0/1711916818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff18100a50 0x7eff18073090 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.578+0000 7eff1c844700 1 --2- 172.21.15.121:0/1711916818 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7eff18106a60 0x7eff180735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.578+0000 7eff1c844700 1 -- 172.21.15.121:0/1711916818 >> 172.21.15.121:0/1711916818 conn(0x7eff180fc4e0 msgr2=0x7eff180fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:40.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.579+0000 7eff1c844700 1 -- 172.21.15.121:0/1711916818 shutdown_connections 2024-04-05T22:07:40.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.579+0000 7eff1c844700 1 -- 172.21.15.121:0/1711916818 wait complete. 2024-04-05T22:07:40.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-05T22:07:40.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.716+0000 7fbaf8061700 1 -- 172.21.15.121:0/3897443626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaf00ff420 msgr2=0x7fbaf00ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:40.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.716+0000 7fbaf8061700 1 --2- 172.21.15.121:0/3897443626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaf00ff420 0x7fbaf00ff800 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fbae4009910 tx=0x7fbae4009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:40.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.716+0000 7fbaf8061700 1 -- 172.21.15.121:0/3897443626 shutdown_connections 2024-04-05T22:07:40.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.716+0000 7fbaf8061700 1 --2- 172.21.15.121:0/3897443626 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbaf00ffdd0 0x7fbaf0105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.716+0000 7fbaf8061700 1 --2- 172.21.15.121:0/3897443626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaf00ff420 0x7fbaf00ff800 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.716+0000 7fbaf8061700 1 -- 172.21.15.121:0/3897443626 >> 172.21.15.121:0/3897443626 conn(0x7fbaf0074ca0 msgr2=0x7fbaf00750b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:40.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.716+0000 7fbaf8061700 1 -- 172.21.15.121:0/3897443626 shutdown_connections 2024-04-05T22:07:40.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.716+0000 7fbaf8061700 1 -- 172.21.15.121:0/3897443626 wait complete. 2024-04-05T22:07:40.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.717+0000 7fbaf8061700 1 Processor -- start 2024-04-05T22:07:40.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.717+0000 7fbaf8061700 1 -- start start 2024-04-05T22:07:40.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.717+0000 7fbaf8061700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaf00ff420 0x7fbaf019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:40.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.717+0000 7fbaf8061700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbaf019eb60 0x7fbaf01986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:40.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.717+0000 7fbaf8061700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf019f180 con 0x7fbaf00ff420 2024-04-05T22:07:40.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.717+0000 7fbaf8061700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf0198bf0 con 0x7fbaf019eb60 2024-04-05T22:07:40.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.717+0000 7fbaf5dfd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaf00ff420 0x7fbaf019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:40.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.717+0000 7fbaf5dfd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaf00ff420 0x7fbaf019e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48448/0 (socket says 172.21.15.121:48448) 2024-04-05T22:07:40.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.717+0000 7fbaf5dfd700 1 -- 172.21.15.121:0/3111811259 learned_addr learned my addr 172.21.15.121:0/3111811259 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:40.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.717+0000 7fbaf55fc700 1 --2- 172.21.15.121:0/3111811259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbaf019eb60 0x7fbaf01986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:40.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.717+0000 7fbaf5dfd700 1 -- 172.21.15.121:0/3111811259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbaf019eb60 msgr2=0x7fbaf01986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:40.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.717+0000 7fbaf5dfd700 1 --2- 172.21.15.121:0/3111811259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbaf019eb60 0x7fbaf01986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.717+0000 7fbaf5dfd700 1 -- 172.21.15.121:0/3111811259 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae40095f0 con 0x7fbaf00ff420 2024-04-05T22:07:40.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.718+0000 7fbaf5dfd700 1 --2- 172.21.15.121:0/3111811259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaf00ff420 0x7fbaf019e620 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fbae4005f50 tx=0x7fbae400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:40.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.718+0000 7fbae2ffd700 1 -- 172.21.15.121:0/3111811259 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbae401d070 con 0x7fbaf00ff420 2024-04-05T22:07:40.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.718+0000 7fbaf8061700 1 -- 172.21.15.121:0/3111811259 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf0198e70 con 0x7fbaf00ff420 2024-04-05T22:07:40.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.718+0000 7fbae2ffd700 1 -- 172.21.15.121:0/3111811259 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbae400f460 con 0x7fbaf00ff420 2024-04-05T22:07:40.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.718+0000 7fbae2ffd700 1 -- 172.21.15.121:0/3111811259 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbae4021640 con 0x7fbaf00ff420 2024-04-05T22:07:40.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.718+0000 7fbaf8061700 1 -- 172.21.15.121:0/3111811259 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf0199370 con 0x7fbaf00ff420 2024-04-05T22:07:40.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.719+0000 7fbaf8061700 1 -- 172.21.15.121:0/3111811259 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf0104ed0 con 0x7fbaf00ff420 2024-04-05T22:07:40.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.719+0000 7fbae2ffd700 1 -- 172.21.15.121:0/3111811259 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbae400f9e0 con 0x7fbaf00ff420 2024-04-05T22:07:40.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.720+0000 7fbae2ffd700 1 --2- 172.21.15.121:0/3111811259 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbadc06cce0 0x7fbadc06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:40.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.720+0000 7fbae2ffd700 1 -- 172.21.15.121:0/3111811259 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbae408d8b0 con 0x7fbaf00ff420 2024-04-05T22:07:40.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.720+0000 7fbaf55fc700 1 --2- 172.21.15.121:0/3111811259 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbadc06cce0 0x7fbadc06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:40.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.720+0000 7fbaf55fc700 1 --2- 172.21.15.121:0/3111811259 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbadc06cce0 0x7fbadc06f1a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fbaec005fd0 tx=0x7fbaec00a560 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:40.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.724+0000 7fbae2ffd700 1 -- 172.21.15.121:0/3111811259 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbae405b3b0 con 0x7fbaf00ff420 2024-04-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.916+0000 7fbaf8061700 1 -- 172.21.15.121:0/3111811259 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fbaf0199be0 con 0x7fbaf00ff420 2024-04-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.916+0000 7fbae2ffd700 1 -- 172.21.15.121:0/3111811259 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fbae405af40 con 0x7fbaf00ff420 2024-04-05T22:07:40.916 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 32 2024-04-05T22:07:40.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.918+0000 7fbaf8061700 1 -- 172.21.15.121:0/3111811259 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbadc06cce0 msgr2=0x7fbadc06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:40.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.918+0000 7fbaf8061700 1 --2- 172.21.15.121:0/3111811259 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbadc06cce0 0x7fbadc06f1a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fbaec005fd0 tx=0x7fbaec00a560 comp rx=0 tx=0).stop 2024-04-05T22:07:40.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.918+0000 7fbaf8061700 1 -- 172.21.15.121:0/3111811259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaf00ff420 msgr2=0x7fbaf019e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:40.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.918+0000 7fbaf8061700 1 --2- 172.21.15.121:0/3111811259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaf00ff420 0x7fbaf019e620 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fbae4005f50 tx=0x7fbae400be20 comp rx=0 tx=0).stop 2024-04-05T22:07:40.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.918+0000 7fbaf8061700 1 -- 172.21.15.121:0/3111811259 shutdown_connections 2024-04-05T22:07:40.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.918+0000 7fbaf8061700 1 --2- 172.21.15.121:0/3111811259 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbadc06cce0 0x7fbadc06f1a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.918+0000 7fbaf8061700 1 --2- 172.21.15.121:0/3111811259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaf00ff420 0x7fbaf019e620 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.918+0000 7fbaf8061700 1 --2- 172.21.15.121:0/3111811259 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbaf019eb60 0x7fbaf01986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:40.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.918+0000 7fbaf8061700 1 -- 172.21.15.121:0/3111811259 >> 172.21.15.121:0/3111811259 conn(0x7fbaf0074ca0 msgr2=0x7fbaf00fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:40.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.918+0000 7fbaf8061700 1 -- 172.21.15.121:0/3111811259 shutdown_connections 2024-04-05T22:07:40.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:40.919+0000 7fbaf8061700 1 -- 172.21.15.121:0/3111811259 wait complete. 2024-04-05T22:07:40.929 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:40 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2708079375' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T22:07:40.929 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:40 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3487196746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T22:07:40.929 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:40 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1711916818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T22:07:40.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-05T22:07:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.052+0000 7fa5f8606700 1 -- 172.21.15.121:0/3711480750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f0105990 msgr2=0x7fa5f0105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.052+0000 7fa5f8606700 1 --2- 172.21.15.121:0/3711480750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f0105990 0x7fa5f0105d70 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0009910 tx=0x7fa5e0009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.052+0000 7fa5f8606700 1 -- 172.21.15.121:0/3711480750 shutdown_connections 2024-04-05T22:07:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.052+0000 7fa5f8606700 1 --2- 172.21.15.121:0/3711480750 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f00ffac0 0x7fa5f00fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.052+0000 7fa5f8606700 1 --2- 172.21.15.121:0/3711480750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f0105990 0x7fa5f0105d70 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.052+0000 7fa5f8606700 1 -- 172.21.15.121:0/3711480750 >> 172.21.15.121:0/3711480750 conn(0x7fa5f0075400 msgr2=0x7fa5f0075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.052+0000 7fa5f8606700 1 -- 172.21.15.121:0/3711480750 shutdown_connections 2024-04-05T22:07:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.052+0000 7fa5f8606700 1 -- 172.21.15.121:0/3711480750 wait complete. 2024-04-05T22:07:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.052+0000 7fa5f8606700 1 Processor -- start 2024-04-05T22:07:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.052+0000 7fa5f8606700 1 -- start start 2024-04-05T22:07:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.052+0000 7fa5f8606700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f00ffac0 0x7fa5f0195410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5f8606700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f0105990 0x7fa5f0195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5f8606700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f0196030 con 0x7fa5f0105990 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5f8606700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f0199db0 con 0x7fa5f00ffac0 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5f5ba1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f0105990 0x7fa5f0195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5f5ba1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f0105990 0x7fa5f0195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48468/0 (socket says 172.21.15.121:48468) 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5f5ba1700 1 -- 172.21.15.121:0/1412180103 learned_addr learned my addr 172.21.15.121:0/1412180103 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5f5ba1700 1 -- 172.21.15.121:0/1412180103 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f00ffac0 msgr2=0x7fa5f0195410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5f63a2700 1 --2- 172.21.15.121:0/1412180103 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f00ffac0 0x7fa5f0195410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5f5ba1700 1 --2- 172.21.15.121:0/1412180103 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f00ffac0 0x7fa5f0195410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5f5ba1700 1 -- 172.21.15.121:0/1412180103 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e00095f0 con 0x7fa5f0105990 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5f5ba1700 1 --2- 172.21.15.121:0/1412180103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f0105990 0x7fa5f0195950 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00e920 tx=0x7fa5ec00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5e77fe700 1 -- 172.21.15.121:0/1412180103 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5ec009790 con 0x7fa5f0105990 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5f8606700 1 -- 172.21.15.121:0/1412180103 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f019a090 con 0x7fa5f0105990 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5e77fe700 1 -- 172.21.15.121:0/1412180103 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5ec00cba0 con 0x7fa5f0105990 2024-04-05T22:07:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5e77fe700 1 -- 172.21.15.121:0/1412180103 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa5ec018650 con 0x7fa5f0105990 2024-04-05T22:07:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.053+0000 7fa5f8606700 1 -- 172.21.15.121:0/1412180103 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f019a5f0 con 0x7fa5f0105990 2024-04-05T22:07:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.054+0000 7fa5f8606700 1 -- 172.21.15.121:0/1412180103 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f004ea90 con 0x7fa5f0105990 2024-04-05T22:07:41.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.055+0000 7fa5e77fe700 1 -- 172.21.15.121:0/1412180103 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fa5ec00cd10 con 0x7fa5f0105990 2024-04-05T22:07:41.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.055+0000 7fa5e77fe700 1 --2- 172.21.15.121:0/1412180103 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5dc06cda0 0x7fa5dc06f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:41.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.056+0000 7fa5f63a2700 1 --2- 172.21.15.121:0/1412180103 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5dc06cda0 0x7fa5dc06f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:41.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.056+0000 7fa5e77fe700 1 -- 172.21.15.121:0/1412180103 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fa5ec014070 con 0x7fa5f0105990 2024-04-05T22:07:41.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.056+0000 7fa5f63a2700 1 --2- 172.21.15.121:0/1412180103 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5dc06cda0 0x7fa5dc06f260 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0006010 tx=0x7fa5e000b640 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:41.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.060+0000 7fa5e77fe700 1 -- 172.21.15.121:0/1412180103 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fa5ec052ab0 con 0x7fa5f0105990 2024-04-05T22:07:41.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:40 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2708079375' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T22:07:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:40 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3487196746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T22:07:41.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:40 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1711916818' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T22:07:41.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.298+0000 7fa5f8606700 1 -- 172.21.15.121:0/1412180103 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5f0196760 con 0x7fa5f0105990 2024-04-05T22:07:41.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.299+0000 7fa5e77fe700 1 -- 172.21.15.121:0/1412180103 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4873 (secure 0 0 0) 0x7fa5ec05a450 con 0x7fa5f0105990 2024-04-05T22:07:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:41.299 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T22:06:01.604327+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34360,"mds_1":34364},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34360":{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":0,"incarnation":34,"state":"up:active","state_seq":469,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34364":{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":1,"incarnation":34,"state":"up:active","state_seq":463,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_44297":{"gid":44297,"name":"cephfs.smithi188.bjmisu","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6825/2880188624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2880188624},{"type":"v1","addr":"172.21.15.188:6825","nonce":2880188624}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_44827":{"gid":44827,"name":"cephfs.smithi188.latnft","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6827/3202979994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3202979994},{"type":"v1","addr":"172.21.15.188:6827","nonce":3202979994}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:41.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.301+0000 7fa5f8606700 1 -- 172.21.15.121:0/1412180103 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5dc06cda0 msgr2=0x7fa5dc06f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:41.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.301+0000 7fa5f8606700 1 --2- 172.21.15.121:0/1412180103 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5dc06cda0 0x7fa5dc06f260 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0006010 tx=0x7fa5e000b640 comp rx=0 tx=0).stop 2024-04-05T22:07:41.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.301+0000 7fa5f8606700 1 -- 172.21.15.121:0/1412180103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f0105990 msgr2=0x7fa5f0195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:41.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.301+0000 7fa5f8606700 1 --2- 172.21.15.121:0/1412180103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f0105990 0x7fa5f0195950 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00e920 tx=0x7fa5ec00ece0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.301+0000 7fa5f8606700 1 -- 172.21.15.121:0/1412180103 shutdown_connections 2024-04-05T22:07:41.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.301+0000 7fa5f8606700 1 --2- 172.21.15.121:0/1412180103 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fa5dc06cda0 0x7fa5dc06f260 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.301+0000 7fa5f8606700 1 --2- 172.21.15.121:0/1412180103 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fa5f00ffac0 0x7fa5f0195410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.301+0000 7fa5f8606700 1 --2- 172.21.15.121:0/1412180103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5f0105990 0x7fa5f0195950 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.301+0000 7fa5f8606700 1 -- 172.21.15.121:0/1412180103 >> 172.21.15.121:0/1412180103 conn(0x7fa5f0075400 msgr2=0x7fa5f00fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:41.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.301+0000 7fa5f8606700 1 -- 172.21.15.121:0/1412180103 shutdown_connections 2024-04-05T22:07:41.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.301+0000 7fa5f8606700 1 -- 172.21.15.121:0/1412180103 wait complete. 2024-04-05T22:07:41.302 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 41 2024-04-05T22:07:41.313 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 50} 2024-04-05T22:07:41.313 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-04-05T22:07:41.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.436+0000 7fb42ec50700 1 -- 172.21.15.121:0/446072905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb428105990 msgr2=0x7fb428105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:41.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.436+0000 7fb42ec50700 1 --2- 172.21.15.121:0/446072905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb428105990 0x7fb428105d70 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009910 tx=0x7fb41c009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:41.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.436+0000 7fb42ec50700 1 -- 172.21.15.121:0/446072905 shutdown_connections 2024-04-05T22:07:41.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.436+0000 7fb42ec50700 1 --2- 172.21.15.121:0/446072905 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb4280ff980 0x7fb4280ffe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.436+0000 7fb42ec50700 1 --2- 172.21.15.121:0/446072905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb428105990 0x7fb428105d70 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.436+0000 7fb42ec50700 1 -- 172.21.15.121:0/446072905 >> 172.21.15.121:0/446072905 conn(0x7fb4280fb430 msgr2=0x7fb4280fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:41.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.436+0000 7fb42ec50700 1 -- 172.21.15.121:0/446072905 shutdown_connections 2024-04-05T22:07:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.436+0000 7fb42ec50700 1 -- 172.21.15.121:0/446072905 wait complete. 2024-04-05T22:07:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.436+0000 7fb42ec50700 1 Processor -- start 2024-04-05T22:07:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.436+0000 7fb42ec50700 1 -- start start 2024-04-05T22:07:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb42ec50700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4280ff980 0x7fb4280fbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb42ec50700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb428105990 0x7fb4280fc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb42ec50700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4280fc8d0 con 0x7fb4280ff980 2024-04-05T22:07:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb42ec50700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4280fca40 con 0x7fb428105990 2024-04-05T22:07:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb42c9ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4280ff980 0x7fb4280fbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb42c9ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4280ff980 0x7fb4280fbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48492/0 (socket says 172.21.15.121:48492) 2024-04-05T22:07:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb42c9ec700 1 -- 172.21.15.121:0/1130703698 learned_addr learned my addr 172.21.15.121:0/1130703698 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb427fff700 1 --2- 172.21.15.121:0/1130703698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb428105990 0x7fb4280fc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb42c9ec700 1 -- 172.21.15.121:0/1130703698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb428105990 msgr2=0x7fb4280fc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:41.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb42c9ec700 1 --2- 172.21.15.121:0/1130703698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb428105990 0x7fb4280fc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb42c9ec700 1 -- 172.21.15.121:0/1130703698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c0095f0 con 0x7fb4280ff980 2024-04-05T22:07:41.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb427fff700 1 --2- 172.21.15.121:0/1130703698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb428105990 0x7fb4280fc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:07:41.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb42c9ec700 1 --2- 172.21.15.121:0/1130703698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4280ff980 0x7fb4280fbdc0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0092e0 tx=0x7fb41c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:41.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb425ffb700 1 -- 172.21.15.121:0/1130703698 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb41c01d070 con 0x7fb4280ff980 2024-04-05T22:07:41.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb425ffb700 1 -- 172.21.15.121:0/1130703698 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb41c005e00 con 0x7fb4280ff980 2024-04-05T22:07:41.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb42ec50700 1 -- 172.21.15.121:0/1130703698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4281a26f0 con 0x7fb4280ff980 2024-04-05T22:07:41.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb425ffb700 1 -- 172.21.15.121:0/1130703698 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb41c00fe60 con 0x7fb4280ff980 2024-04-05T22:07:41.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.437+0000 7fb42ec50700 1 -- 172.21.15.121:0/1130703698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4281a2ab0 con 0x7fb4280ff980 2024-04-05T22:07:41.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.438+0000 7fb42ec50700 1 -- 172.21.15.121:0/1130703698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb428108f90 con 0x7fb4280ff980 2024-04-05T22:07:41.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.439+0000 7fb425ffb700 1 -- 172.21.15.121:0/1130703698 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb41c00f6d0 con 0x7fb4280ff980 2024-04-05T22:07:41.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.439+0000 7fb425ffb700 1 --2- 172.21.15.121:0/1130703698 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb41806cda0 0x7fb41806f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:41.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.439+0000 7fb425ffb700 1 -- 172.21.15.121:0/1130703698 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb41c08d8b0 con 0x7fb4280ff980 2024-04-05T22:07:41.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.439+0000 7fb427fff700 1 --2- 172.21.15.121:0/1130703698 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb41806cda0 0x7fb41806f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:41.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.440+0000 7fb427fff700 1 --2- 172.21.15.121:0/1130703698 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb41806cda0 0x7fb41806f260 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb414005fd0 tx=0x7fb414005ee0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:41.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.444+0000 7fb425ffb700 1 -- 172.21.15.121:0/1130703698 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb41c057920 con 0x7fb4280ff980 2024-04-05T22:07:41.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.685+0000 7fb42ec50700 1 -- 172.21.15.121:0/1130703698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7fb42802cdd0 con 0x7fb4280ff980 2024-04-05T22:07:41.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.685+0000 7fb425ffb700 1 -- 172.21.15.121:0/1130703698 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v41) v1 ==== 107+0+4866 (secure 0 0 0) 0x7fb41c05af40 con 0x7fb4280ff980 2024-04-05T22:07:41.685 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:41.685 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:25:26.273340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14476,"mds_1":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14476":{"gid":14476,"name":"cephfs.smithi121.ifoxmw","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.121:6829/3760584056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3760584056},{"type":"v1","addr":"172.21.15.121:6829","nonce":3760584056}]},"join_fscid":1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14488":{"gid":14488,"name":"cephfs.smithi121.gxheqt","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6827/4279084130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":4279084130},{"type":"v1","addr":"172.21.15.121:6827","nonce":4279084130}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24255":{"gid":24255,"name":"cephfs.smithi188.bjmisu","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.188:6825/213881583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":213881583},{"type":"v1","addr":"172.21.15.188:6825","nonce":213881583}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24265":{"gid":24265,"name":"cephfs.smithi188.latnft","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.188:6827/3730442150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3730442150},{"type":"v1","addr":"172.21.15.188:6827","nonce":3730442150}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:41.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.687+0000 7fb42ec50700 1 -- 172.21.15.121:0/1130703698 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb41806cda0 msgr2=0x7fb41806f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:41.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.687+0000 7fb42ec50700 1 --2- 172.21.15.121:0/1130703698 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb41806cda0 0x7fb41806f260 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb414005fd0 tx=0x7fb414005ee0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.687+0000 7fb42ec50700 1 -- 172.21.15.121:0/1130703698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4280ff980 msgr2=0x7fb4280fbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:41.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.687+0000 7fb42ec50700 1 --2- 172.21.15.121:0/1130703698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4280ff980 0x7fb4280fbdc0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0092e0 tx=0x7fb41c00be20 comp rx=0 tx=0).stop 2024-04-05T22:07:41.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.687+0000 7fb42ec50700 1 -- 172.21.15.121:0/1130703698 shutdown_connections 2024-04-05T22:07:41.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.687+0000 7fb42ec50700 1 --2- 172.21.15.121:0/1130703698 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb41806cda0 0x7fb41806f260 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.687+0000 7fb42ec50700 1 --2- 172.21.15.121:0/1130703698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4280ff980 0x7fb4280fbdc0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.687+0000 7fb42ec50700 1 --2- 172.21.15.121:0/1130703698 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb428105990 0x7fb4280fc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.687+0000 7fb42ec50700 1 -- 172.21.15.121:0/1130703698 >> 172.21.15.121:0/1130703698 conn(0x7fb4280fb430 msgr2=0x7fb428068b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:41.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.688+0000 7fb42ec50700 1 -- 172.21.15.121:0/1130703698 shutdown_connections 2024-04-05T22:07:41.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.688+0000 7fb42ec50700 1 -- 172.21.15.121:0/1130703698 wait complete. 2024-04-05T22:07:41.688 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 19 2024-04-05T22:07:41.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-04-05T22:07:41.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.822+0000 7fe2561c9700 1 -- 172.21.15.121:0/3594491796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2500ffac0 msgr2=0x7fe2500fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.822+0000 7fe2561c9700 1 --2- 172.21.15.121:0/3594491796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2500ffac0 0x7fe2500fff20 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe240009910 tx=0x7fe240009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.822+0000 7fe2561c9700 1 -- 172.21.15.121:0/3594491796 shutdown_connections 2024-04-05T22:07:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.822+0000 7fe2561c9700 1 --2- 172.21.15.121:0/3594491796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2500ffac0 0x7fe2500fff20 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.822+0000 7fe2561c9700 1 --2- 172.21.15.121:0/3594491796 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe250105990 0x7fe250105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.822+0000 7fe2561c9700 1 -- 172.21.15.121:0/3594491796 >> 172.21.15.121:0/3594491796 conn(0x7fe250075400 msgr2=0x7fe250075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.822+0000 7fe2561c9700 1 -- 172.21.15.121:0/3594491796 shutdown_connections 2024-04-05T22:07:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.822+0000 7fe2561c9700 1 -- 172.21.15.121:0/3594491796 wait complete. 2024-04-05T22:07:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.822+0000 7fe2561c9700 1 Processor -- start 2024-04-05T22:07:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.822+0000 7fe2561c9700 1 -- start start 2024-04-05T22:07:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe2561c9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2500ffac0 0x7fe250197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe2561c9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe250105990 0x7fe250197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe2561c9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe250197e80 con 0x7fe250105990 2024-04-05T22:07:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe2561c9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe250197fc0 con 0x7fe2500ffac0 2024-04-05T22:07:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe24effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe250105990 0x7fe250197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe24effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe250105990 0x7fe250197860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48508/0 (socket says 172.21.15.121:48508) 2024-04-05T22:07:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe24effd700 1 -- 172.21.15.121:0/761812090 learned_addr learned my addr 172.21.15.121:0/761812090 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe24f7fe700 1 --2- 172.21.15.121:0/761812090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2500ffac0 0x7fe250197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe24effd700 1 -- 172.21.15.121:0/761812090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2500ffac0 msgr2=0x7fe250197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe24effd700 1 --2- 172.21.15.121:0/761812090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2500ffac0 0x7fe250197320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe24effd700 1 -- 172.21.15.121:0/761812090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2400095f0 con 0x7fe250105990 2024-04-05T22:07:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe24effd700 1 --2- 172.21.15.121:0/761812090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe250105990 0x7fe250197860 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe240006010 tx=0x7fe24000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe24cff9700 1 -- 172.21.15.121:0/761812090 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe24001d070 con 0x7fe250105990 2024-04-05T22:07:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe24cff9700 1 -- 172.21.15.121:0/761812090 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe240005710 con 0x7fe250105990 2024-04-05T22:07:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe2561c9700 1 -- 172.21.15.121:0/761812090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2501a0390 con 0x7fe250105990 2024-04-05T22:07:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe24cff9700 1 -- 172.21.15.121:0/761812090 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe24000fe20 con 0x7fe250105990 2024-04-05T22:07:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.823+0000 7fe2561c9700 1 -- 172.21.15.121:0/761812090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2501a07b0 con 0x7fe250105990 2024-04-05T22:07:41.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.824+0000 7fe2561c9700 1 -- 172.21.15.121:0/761812090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe250107f10 con 0x7fe250105990 2024-04-05T22:07:41.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.825+0000 7fe24cff9700 1 -- 172.21.15.121:0/761812090 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe240003a60 con 0x7fe250105990 2024-04-05T22:07:41.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.825+0000 7fe24cff9700 1 --2- 172.21.15.121:0/761812090 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe23806cdf0 0x7fe23806f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:41.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.825+0000 7fe24cff9700 1 -- 172.21.15.121:0/761812090 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe24008d7f0 con 0x7fe250105990 2024-04-05T22:07:41.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.825+0000 7fe24f7fe700 1 --2- 172.21.15.121:0/761812090 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe23806cdf0 0x7fe23806f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:41.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.826+0000 7fe24f7fe700 1 --2- 172.21.15.121:0/761812090 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe23806cdf0 0x7fe23806f2b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe244005950 tx=0x7fe24400b410 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:41.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:41.830+0000 7fe24cff9700 1 -- 172.21.15.121:0/761812090 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe2400577c0 con 0x7fe250105990 2024-04-05T22:07:41.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:41 smithi121 ceph-mon[145859]: pgmap v1246: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:41.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:41 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3111811259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T22:07:41.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:41 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1412180103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T22:07:41.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:41 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1130703698' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-05T22:07:42.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.071+0000 7fe2561c9700 1 -- 172.21.15.121:0/761812090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7fe25002cdd0 con 0x7fe250105990 2024-04-05T22:07:42.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.071+0000 7fe24cff9700 1 -- 172.21.15.121:0/761812090 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v41) v1 ==== 107+0+4867 (secure 0 0 0) 0x7fe24005ade0 con 0x7fe250105990 2024-04-05T22:07:42.071 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:42.072 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:27:26.656630+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14476,"mds_1":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14476":{"gid":14476,"name":"cephfs.smithi121.ifoxmw","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.121:6829/3760584056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3760584056},{"type":"v1","addr":"172.21.15.121:6829","nonce":3760584056}]},"join_fscid":1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14488":{"gid":14488,"name":"cephfs.smithi121.gxheqt","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6827/4279084130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":4279084130},{"type":"v1","addr":"172.21.15.121:6827","nonce":4279084130}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24255":{"gid":24255,"name":"cephfs.smithi188.bjmisu","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.188:6825/213881583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":213881583},{"type":"v1","addr":"172.21.15.188:6825","nonce":213881583}]},"join_fscid":1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24265":{"gid":24265,"name":"cephfs.smithi188.latnft","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.188:6827/3730442150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3730442150},{"type":"v1","addr":"172.21.15.188:6827","nonce":3730442150}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:42.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.073+0000 7fe2561c9700 1 -- 172.21.15.121:0/761812090 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe23806cdf0 msgr2=0x7fe23806f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:42.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.074+0000 7fe2561c9700 1 --2- 172.21.15.121:0/761812090 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe23806cdf0 0x7fe23806f2b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe244005950 tx=0x7fe24400b410 comp rx=0 tx=0).stop 2024-04-05T22:07:42.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.074+0000 7fe2561c9700 1 -- 172.21.15.121:0/761812090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe250105990 msgr2=0x7fe250197860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:42.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.074+0000 7fe2561c9700 1 --2- 172.21.15.121:0/761812090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe250105990 0x7fe250197860 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe240006010 tx=0x7fe24000bd70 comp rx=0 tx=0).stop 2024-04-05T22:07:42.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.074+0000 7fe2561c9700 1 -- 172.21.15.121:0/761812090 shutdown_connections 2024-04-05T22:07:42.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.074+0000 7fe2561c9700 1 --2- 172.21.15.121:0/761812090 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe23806cdf0 0x7fe23806f2b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.074+0000 7fe2561c9700 1 --2- 172.21.15.121:0/761812090 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe2500ffac0 0x7fe250197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.074+0000 7fe2561c9700 1 --2- 172.21.15.121:0/761812090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe250105990 0x7fe250197860 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.074+0000 7fe2561c9700 1 -- 172.21.15.121:0/761812090 >> 172.21.15.121:0/761812090 conn(0x7fe250075400 msgr2=0x7fe2500fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:42.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.074+0000 7fe2561c9700 1 -- 172.21.15.121:0/761812090 shutdown_connections 2024-04-05T22:07:42.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.074+0000 7fe2561c9700 1 -- 172.21.15.121:0/761812090 wait complete. 2024-04-05T22:07:42.075 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 20 2024-04-05T22:07:42.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-04-05T22:07:42.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.208+0000 7f16b3091700 1 -- 172.21.15.121:0/4114030488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16ac105990 msgr2=0x7f16ac105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.208+0000 7f16b3091700 1 --2- 172.21.15.121:0/4114030488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16ac105990 0x7f16ac105d70 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009910 tx=0x7f16a0009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.208+0000 7f16b3091700 1 -- 172.21.15.121:0/4114030488 shutdown_connections 2024-04-05T22:07:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.208+0000 7f16b3091700 1 --2- 172.21.15.121:0/4114030488 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16ac0ffac0 0x7f16ac0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.208+0000 7f16b3091700 1 --2- 172.21.15.121:0/4114030488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16ac105990 0x7f16ac105d70 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.208+0000 7f16b3091700 1 -- 172.21.15.121:0/4114030488 >> 172.21.15.121:0/4114030488 conn(0x7f16ac075400 msgr2=0x7f16ac075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.208+0000 7f16b3091700 1 -- 172.21.15.121:0/4114030488 shutdown_connections 2024-04-05T22:07:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.208+0000 7f16b3091700 1 -- 172.21.15.121:0/4114030488 wait complete. 2024-04-05T22:07:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.209+0000 7f16b3091700 1 Processor -- start 2024-04-05T22:07:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.209+0000 7f16b3091700 1 -- start start 2024-04-05T22:07:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.209+0000 7f16b3091700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16ac0ffac0 0x7f16ac19e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:42.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.209+0000 7f16b3091700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16ac105990 0x7f16ac19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:42.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.209+0000 7f16b3091700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac19f190 con 0x7f16ac0ffac0 2024-04-05T22:07:42.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.209+0000 7f16b3091700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac198630 con 0x7f16ac105990 2024-04-05T22:07:42.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.209+0000 7f16b0e2d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16ac0ffac0 0x7f16ac19e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:42.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.209+0000 7f16b0e2d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16ac0ffac0 0x7f16ac19e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48522/0 (socket says 172.21.15.121:48522) 2024-04-05T22:07:42.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.209+0000 7f16b0e2d700 1 -- 172.21.15.121:0/2300706491 learned_addr learned my addr 172.21.15.121:0/2300706491 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:42.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.209+0000 7f16b0e2d700 1 -- 172.21.15.121:0/2300706491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16ac105990 msgr2=0x7f16ac19eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:42.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.209+0000 7f16b0e2d700 1 --2- 172.21.15.121:0/2300706491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16ac105990 0x7f16ac19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.209+0000 7f16b0e2d700 1 -- 172.21.15.121:0/2300706491 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a00095f0 con 0x7f16ac0ffac0 2024-04-05T22:07:42.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.210+0000 7f16b0e2d700 1 --2- 172.21.15.121:0/2300706491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16ac0ffac0 0x7f16ac19e5c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f16a0006010 tx=0x7f16a000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:42.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.210+0000 7f16a9ffb700 1 -- 172.21.15.121:0/2300706491 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16a001d070 con 0x7f16ac0ffac0 2024-04-05T22:07:42.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.210+0000 7f16a9ffb700 1 -- 172.21.15.121:0/2300706491 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16a000f460 con 0x7f16ac0ffac0 2024-04-05T22:07:42.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.210+0000 7f16b3091700 1 -- 172.21.15.121:0/2300706491 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac1988b0 con 0x7f16ac0ffac0 2024-04-05T22:07:42.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.210+0000 7f16a9ffb700 1 -- 172.21.15.121:0/2300706491 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16a0021640 con 0x7f16ac0ffac0 2024-04-05T22:07:42.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.210+0000 7f16b3091700 1 -- 172.21.15.121:0/2300706491 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac198db0 con 0x7f16ac0ffac0 2024-04-05T22:07:42.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.211+0000 7f16b3091700 1 -- 172.21.15.121:0/2300706491 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac04ea90 con 0x7f16ac0ffac0 2024-04-05T22:07:42.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.211+0000 7f16a9ffb700 1 -- 172.21.15.121:0/2300706491 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f16a000fa60 con 0x7f16ac0ffac0 2024-04-05T22:07:42.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.212+0000 7f16a9ffb700 1 --2- 172.21.15.121:0/2300706491 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f169c06cda0 0x7f169c06f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:42.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.212+0000 7f16a9ffb700 1 -- 172.21.15.121:0/2300706491 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f16a008d920 con 0x7f16ac0ffac0 2024-04-05T22:07:42.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.212+0000 7f16abfff700 1 --2- 172.21.15.121:0/2300706491 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f169c06cda0 0x7f169c06f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:42.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.212+0000 7f16abfff700 1 --2- 172.21.15.121:0/2300706491 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f169c06cda0 0x7f169c06f260 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f16ac075ac0 tx=0x7f1698006d20 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:42.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.216+0000 7f16a9ffb700 1 -- 172.21.15.121:0/2300706491 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f16a005b380 con 0x7f16ac0ffac0 2024-04-05T22:07:42.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:41 smithi188 ceph-mon[107793]: pgmap v1246: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:42.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:41 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3111811259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T22:07:42.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:41 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1412180103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-05T22:07:42.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:41 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1130703698' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-05T22:07:42.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.456+0000 7f16b3091700 1 -- 172.21.15.121:0/2300706491 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7f16ac199190 con 0x7f16ac0ffac0 2024-04-05T22:07:42.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.457+0000 7f16a9ffb700 1 -- 172.21.15.121:0/2300706491 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v41) v1 ==== 107+0+4865 (secure 0 0 0) 0x7f16a005af10 con 0x7f16ac0ffac0 2024-04-05T22:07:42.456 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:42.457 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:30:52.246236+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14476,"mds_1":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14476":{"gid":14476,"name":"cephfs.smithi121.ifoxmw","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.121:6829/3760584056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3760584056},{"type":"v1","addr":"172.21.15.121:6829","nonce":3760584056}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14488":{"gid":14488,"name":"cephfs.smithi121.gxheqt","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6827/4279084130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":4279084130},{"type":"v1","addr":"172.21.15.121:6827","nonce":4279084130}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24255":{"gid":24255,"name":"cephfs.smithi188.bjmisu","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.188:6825/213881583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":213881583},{"type":"v1","addr":"172.21.15.188:6825","nonce":213881583}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24265":{"gid":24265,"name":"cephfs.smithi188.latnft","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.188:6827/3730442150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3730442150},{"type":"v1","addr":"172.21.15.188:6827","nonce":3730442150}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:42.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.459+0000 7f16b3091700 1 -- 172.21.15.121:0/2300706491 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f169c06cda0 msgr2=0x7f169c06f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:42.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.459+0000 7f16b3091700 1 --2- 172.21.15.121:0/2300706491 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f169c06cda0 0x7f169c06f260 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f16ac075ac0 tx=0x7f1698006d20 comp rx=0 tx=0).stop 2024-04-05T22:07:42.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.459+0000 7f16b3091700 1 -- 172.21.15.121:0/2300706491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16ac0ffac0 msgr2=0x7f16ac19e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:42.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.459+0000 7f16b3091700 1 --2- 172.21.15.121:0/2300706491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16ac0ffac0 0x7f16ac19e5c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f16a0006010 tx=0x7f16a000be20 comp rx=0 tx=0).stop 2024-04-05T22:07:42.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.459+0000 7f16b3091700 1 -- 172.21.15.121:0/2300706491 shutdown_connections 2024-04-05T22:07:42.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.459+0000 7f16b3091700 1 --2- 172.21.15.121:0/2300706491 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f169c06cda0 0x7f169c06f260 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.459+0000 7f16b3091700 1 --2- 172.21.15.121:0/2300706491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16ac0ffac0 0x7f16ac19e5c0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.459+0000 7f16b3091700 1 --2- 172.21.15.121:0/2300706491 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f16ac105990 0x7f16ac19eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.459+0000 7f16b3091700 1 -- 172.21.15.121:0/2300706491 >> 172.21.15.121:0/2300706491 conn(0x7f16ac075400 msgr2=0x7f16ac067ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:42.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.459+0000 7f16b3091700 1 -- 172.21.15.121:0/2300706491 shutdown_connections 2024-04-05T22:07:42.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.459+0000 7f16b3091700 1 -- 172.21.15.121:0/2300706491 wait complete. 2024-04-05T22:07:42.460 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 21 2024-04-05T22:07:42.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-04-05T22:07:42.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.594+0000 7fafcab38700 1 -- 172.21.15.121:0/2142396478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc40ffac0 msgr2=0x7fafc40fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:42.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.594+0000 7fafcab38700 1 --2- 172.21.15.121:0/2142396478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc40ffac0 0x7fafc40fff20 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009910 tx=0x7fafb0009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:42.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.594+0000 7fafcab38700 1 -- 172.21.15.121:0/2142396478 shutdown_connections 2024-04-05T22:07:42.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.594+0000 7fafcab38700 1 --2- 172.21.15.121:0/2142396478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc40ffac0 0x7fafc40fff20 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.594+0000 7fafcab38700 1 --2- 172.21.15.121:0/2142396478 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafc4105990 0x7fafc4105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.594+0000 7fafcab38700 1 -- 172.21.15.121:0/2142396478 >> 172.21.15.121:0/2142396478 conn(0x7fafc4075400 msgr2=0x7fafc4075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:42.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.594+0000 7fafcab38700 1 -- 172.21.15.121:0/2142396478 shutdown_connections 2024-04-05T22:07:42.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.594+0000 7fafcab38700 1 -- 172.21.15.121:0/2142396478 wait complete. 2024-04-05T22:07:42.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafcab38700 1 Processor -- start 2024-04-05T22:07:42.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafcab38700 1 -- start start 2024-04-05T22:07:42.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafcab38700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafc40ffac0 0x7fafc4196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:42.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafcab38700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc4105990 0x7fafc4196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafcab38700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc4197130 con 0x7fafc4105990 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafcab38700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc419aeb0 con 0x7fafc40ffac0 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafc3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc4105990 0x7fafc4196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafc3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc4105990 0x7fafc4196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48534/0 (socket says 172.21.15.121:48534) 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafc3fff700 1 -- 172.21.15.121:0/1636121523 learned_addr learned my addr 172.21.15.121:0/1636121523 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafc88d4700 1 --2- 172.21.15.121:0/1636121523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafc40ffac0 0x7fafc4196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafc3fff700 1 -- 172.21.15.121:0/1636121523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafc40ffac0 msgr2=0x7fafc4196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafc3fff700 1 --2- 172.21.15.121:0/1636121523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafc40ffac0 0x7fafc4196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafc3fff700 1 -- 172.21.15.121:0/1636121523 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb00095f0 con 0x7fafc4105990 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafc88d4700 1 --2- 172.21.15.121:0/1636121523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafc40ffac0 0x7fafc4196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafc3fff700 1 --2- 172.21.15.121:0/1636121523 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc4105990 0x7fafc4196a50 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fafb0006010 tx=0x7fafb000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.595+0000 7fafc1ffb700 1 -- 172.21.15.121:0/1636121523 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafb001d070 con 0x7fafc4105990 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.596+0000 7fafc1ffb700 1 -- 172.21.15.121:0/1636121523 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafb000f460 con 0x7fafc4105990 2024-04-05T22:07:42.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.596+0000 7fafcab38700 1 -- 172.21.15.121:0/1636121523 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc419b130 con 0x7fafc4105990 2024-04-05T22:07:42.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.596+0000 7fafc1ffb700 1 -- 172.21.15.121:0/1636121523 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafb0021640 con 0x7fafc4105990 2024-04-05T22:07:42.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.596+0000 7fafcab38700 1 -- 172.21.15.121:0/1636121523 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc419b630 con 0x7fafc4105990 2024-04-05T22:07:42.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.597+0000 7fafcab38700 1 -- 172.21.15.121:0/1636121523 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc4108f20 con 0x7fafc4105990 2024-04-05T22:07:42.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.597+0000 7fafc1ffb700 1 -- 172.21.15.121:0/1636121523 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fafb000fa60 con 0x7fafc4105990 2024-04-05T22:07:42.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.597+0000 7fafc1ffb700 1 --2- 172.21.15.121:0/1636121523 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fafb406cda0 0x7fafb406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:42.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.598+0000 7fafc1ffb700 1 -- 172.21.15.121:0/1636121523 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fafb008d880 con 0x7fafc4105990 2024-04-05T22:07:42.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.598+0000 7fafc88d4700 1 --2- 172.21.15.121:0/1636121523 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fafb406cda0 0x7fafb406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:42.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.598+0000 7fafc88d4700 1 --2- 172.21.15.121:0/1636121523 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fafb406cda0 0x7fafb406f260 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fafb8005950 tx=0x7fafb8005f50 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:42.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.602+0000 7fafc1ffb700 1 -- 172.21.15.121:0/1636121523 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fafb00578f0 con 0x7fafc4105990 2024-04-05T22:07:42.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.844+0000 7fafcab38700 1 -- 172.21.15.121:0/1636121523 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7fafc402cd30 con 0x7fafc4105990 2024-04-05T22:07:42.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.844+0000 7fafc1ffb700 1 -- 172.21.15.121:0/1636121523 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v41) v1 ==== 107+0+3287 (secure 0 0 0) 0x7fafb0026cf0 con 0x7fafc4105990 2024-04-05T22:07:42.844 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:42.844 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:34:36.984648+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14476,"mds_1":24255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14476":{"gid":14476,"name":"cephfs.smithi121.ifoxmw","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.121:6829/3760584056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3760584056},{"type":"v1","addr":"172.21.15.121:6829","nonce":3760584056}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24255":{"gid":24255,"name":"cephfs.smithi188.bjmisu","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.188:6825/213881583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":213881583},{"type":"v1","addr":"172.21.15.188:6825","nonce":213881583}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:42.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.846+0000 7fafcab38700 1 -- 172.21.15.121:0/1636121523 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fafb406cda0 msgr2=0x7fafb406f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:42.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.846+0000 7fafcab38700 1 --2- 172.21.15.121:0/1636121523 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fafb406cda0 0x7fafb406f260 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fafb8005950 tx=0x7fafb8005f50 comp rx=0 tx=0).stop 2024-04-05T22:07:42.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.846+0000 7fafcab38700 1 -- 172.21.15.121:0/1636121523 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc4105990 msgr2=0x7fafc4196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:42.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.846+0000 7fafcab38700 1 --2- 172.21.15.121:0/1636121523 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc4105990 0x7fafc4196a50 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fafb0006010 tx=0x7fafb000be20 comp rx=0 tx=0).stop 2024-04-05T22:07:42.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.846+0000 7fafcab38700 1 -- 172.21.15.121:0/1636121523 shutdown_connections 2024-04-05T22:07:42.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.846+0000 7fafcab38700 1 --2- 172.21.15.121:0/1636121523 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fafb406cda0 0x7fafb406f260 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.846+0000 7fafcab38700 1 --2- 172.21.15.121:0/1636121523 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fafc40ffac0 0x7fafc4196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.846+0000 7fafcab38700 1 --2- 172.21.15.121:0/1636121523 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc4105990 0x7fafc4196a50 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.846+0000 7fafcab38700 1 -- 172.21.15.121:0/1636121523 >> 172.21.15.121:0/1636121523 conn(0x7fafc4075400 msgr2=0x7fafc4067fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:42.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.847+0000 7fafcab38700 1 -- 172.21.15.121:0/1636121523 shutdown_connections 2024-04-05T22:07:42.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.847+0000 7fafcab38700 1 -- 172.21.15.121:0/1636121523 wait complete. 2024-04-05T22:07:42.848 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 22 2024-04-05T22:07:42.859 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 22 2024-04-05T22:07:42.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-04-05T22:07:42.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.981+0000 7fc1aec19700 1 -- 172.21.15.121:0/2969833189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1a80ffac0 msgr2=0x7fc1a80fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:42.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.981+0000 7fc1aec19700 1 --2- 172.21.15.121:0/2969833189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1a80ffac0 0x7fc1a80fff20 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fc194009910 tx=0x7fc194009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:42.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.981+0000 7fc1aec19700 1 -- 172.21.15.121:0/2969833189 shutdown_connections 2024-04-05T22:07:42.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.981+0000 7fc1aec19700 1 --2- 172.21.15.121:0/2969833189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1a80ffac0 0x7fc1a80fff20 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.981+0000 7fc1aec19700 1 --2- 172.21.15.121:0/2969833189 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1a8105990 0x7fc1a8105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.981+0000 7fc1aec19700 1 -- 172.21.15.121:0/2969833189 >> 172.21.15.121:0/2969833189 conn(0x7fc1a8075400 msgr2=0x7fc1a8075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:42.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.981+0000 7fc1aec19700 1 -- 172.21.15.121:0/2969833189 shutdown_connections 2024-04-05T22:07:42.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.981+0000 7fc1aec19700 1 -- 172.21.15.121:0/2969833189 wait complete. 2024-04-05T22:07:42.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1aec19700 1 Processor -- start 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1aec19700 1 -- start start 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1aec19700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1a80ffac0 0x7fc1a8197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1aec19700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1a8105990 0x7fc1a8197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1aec19700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a8197e80 con 0x7fc1a80ffac0 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1aec19700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a8197fc0 con 0x7fc1a8105990 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1ac9b5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1a80ffac0 0x7fc1a8197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1ac9b5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1a80ffac0 0x7fc1a8197320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48562/0 (socket says 172.21.15.121:48562) 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1ac9b5700 1 -- 172.21.15.121:0/2375580940 learned_addr learned my addr 172.21.15.121:0/2375580940 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1a7fff700 1 --2- 172.21.15.121:0/2375580940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1a8105990 0x7fc1a8197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1ac9b5700 1 -- 172.21.15.121:0/2375580940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1a8105990 msgr2=0x7fc1a8197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1ac9b5700 1 --2- 172.21.15.121:0/2375580940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1a8105990 0x7fc1a8197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1ac9b5700 1 -- 172.21.15.121:0/2375580940 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1940095f0 con 0x7fc1a80ffac0 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1a7fff700 1 --2- 172.21.15.121:0/2375580940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1a8105990 0x7fc1a8197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:07:42.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1ac9b5700 1 --2- 172.21.15.121:0/2375580940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1a80ffac0 0x7fc1a8197320 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00e920 tx=0x7fc19c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:42.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.982+0000 7fc1a5ffb700 1 -- 172.21.15.121:0/2375580940 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc19c009790 con 0x7fc1a80ffac0 2024-04-05T22:07:42.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.983+0000 7fc1aec19700 1 -- 172.21.15.121:0/2375580940 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a81a0390 con 0x7fc1a80ffac0 2024-04-05T22:07:42.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.983+0000 7fc1a5ffb700 1 -- 172.21.15.121:0/2375580940 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc19c00cba0 con 0x7fc1a80ffac0 2024-04-05T22:07:42.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.983+0000 7fc1a5ffb700 1 -- 172.21.15.121:0/2375580940 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc19c018650 con 0x7fc1a80ffac0 2024-04-05T22:07:42.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.983+0000 7fc1aec19700 1 -- 172.21.15.121:0/2375580940 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a81a0860 con 0x7fc1a80ffac0 2024-04-05T22:07:42.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.984+0000 7fc1aec19700 1 -- 172.21.15.121:0/2375580940 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a8107f10 con 0x7fc1a80ffac0 2024-04-05T22:07:42.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.984+0000 7fc1a5ffb700 1 -- 172.21.15.121:0/2375580940 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc19c00cd10 con 0x7fc1a80ffac0 2024-04-05T22:07:42.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.985+0000 7fc1a5ffb700 1 --2- 172.21.15.121:0/2375580940 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc19806cda0 0x7fc19806f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:42.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.985+0000 7fc1a5ffb700 1 -- 172.21.15.121:0/2375580940 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc19c014070 con 0x7fc1a80ffac0 2024-04-05T22:07:42.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.985+0000 7fc1a7fff700 1 --2- 172.21.15.121:0/2375580940 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc19806cda0 0x7fc19806f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:42.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.985+0000 7fc1a7fff700 1 --2- 172.21.15.121:0/2375580940 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc19806cda0 0x7fc19806f260 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc19400b6e0 tx=0x7fc194009f70 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:42.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:42.989+0000 7fc1a5ffb700 1 -- 172.21.15.121:0/2375580940 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc19c052ab0 con 0x7fc1a80ffac0 2024-04-05T22:07:43.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:42 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/761812090' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-05T22:07:43.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:42 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2300706491' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-05T22:07:43.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.225+0000 7fc1aec19700 1 -- 172.21.15.121:0/2375580940 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7fc1a804ea90 con 0x7fc1a80ffac0 2024-04-05T22:07:43.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.225+0000 7fc1a5ffb700 1 -- 172.21.15.121:0/2375580940 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v41) v1 ==== 107+0+3266 (secure 0 0 0) 0x7fc19c05a450 con 0x7fc1a80ffac0 2024-04-05T22:07:43.225 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:43.225 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34320,"name":"cephfs.smithi121.gxheqt","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6827/1474300111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1474300111},{"type":"v1","addr":"172.21.15.121:6827","nonce":1474300111}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":44245,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/963169033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":963169033},{"type":"v1","addr":"172.21.15.188:6827","nonce":963169033}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:34:37.155847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.227+0000 7fc1aec19700 1 -- 172.21.15.121:0/2375580940 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc19806cda0 msgr2=0x7fc19806f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.228+0000 7fc1aec19700 1 --2- 172.21.15.121:0/2375580940 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc19806cda0 0x7fc19806f260 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc19400b6e0 tx=0x7fc194009f70 comp rx=0 tx=0).stop 2024-04-05T22:07:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.228+0000 7fc1aec19700 1 -- 172.21.15.121:0/2375580940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1a80ffac0 msgr2=0x7fc1a8197320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.228+0000 7fc1aec19700 1 --2- 172.21.15.121:0/2375580940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1a80ffac0 0x7fc1a8197320 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00e920 tx=0x7fc19c00ece0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.228+0000 7fc1aec19700 1 -- 172.21.15.121:0/2375580940 shutdown_connections 2024-04-05T22:07:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.228+0000 7fc1aec19700 1 --2- 172.21.15.121:0/2375580940 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc19806cda0 0x7fc19806f260 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.228+0000 7fc1aec19700 1 --2- 172.21.15.121:0/2375580940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1a80ffac0 0x7fc1a8197320 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.228+0000 7fc1aec19700 1 --2- 172.21.15.121:0/2375580940 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc1a8105990 0x7fc1a8197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.228+0000 7fc1aec19700 1 -- 172.21.15.121:0/2375580940 >> 172.21.15.121:0/2375580940 conn(0x7fc1a8075400 msgr2=0x7fc1a80fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.228+0000 7fc1aec19700 1 -- 172.21.15.121:0/2375580940 shutdown_connections 2024-04-05T22:07:43.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.228+0000 7fc1aec19700 1 -- 172.21.15.121:0/2375580940 wait complete. 2024-04-05T22:07:43.229 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 23 2024-04-05T22:07:43.237 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:42 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/761812090' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-05T22:07:43.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:42 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2300706491' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-05T22:07:43.240 DEBUG:tasks.fs:max_mds not reduced in epoch 23 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T22:07:43.240 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 23 2024-04-05T22:07:43.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-04-05T22:07:43.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.362+0000 7f6aeb03f700 1 -- 172.21.15.121:0/4014784383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ae40ff420 msgr2=0x7f6ae40ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.362+0000 7f6aeb03f700 1 --2- 172.21.15.121:0/4014784383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ae40ff420 0x7f6ae40ff800 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009910 tx=0x7f6ad8009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.363+0000 7f6aeb03f700 1 -- 172.21.15.121:0/4014784383 shutdown_connections 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.363+0000 7f6aeb03f700 1 --2- 172.21.15.121:0/4014784383 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ae40ffdd0 0x7f6ae4103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.363+0000 7f6aeb03f700 1 --2- 172.21.15.121:0/4014784383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ae40ff420 0x7f6ae40ff800 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.363+0000 7f6aeb03f700 1 -- 172.21.15.121:0/4014784383 >> 172.21.15.121:0/4014784383 conn(0x7f6ae4074c20 msgr2=0x7f6ae4075030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.363+0000 7f6aeb03f700 1 -- 172.21.15.121:0/4014784383 shutdown_connections 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.363+0000 7f6aeb03f700 1 -- 172.21.15.121:0/4014784383 wait complete. 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.363+0000 7f6aeb03f700 1 Processor -- start 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.363+0000 7f6aeb03f700 1 -- start start 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.363+0000 7f6aeb03f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ae40ff420 0x7f6ae4100e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.363+0000 7f6aeb03f700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ae4101340 0x7f6ae41017c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.363+0000 7f6aeb03f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae41a2710 con 0x7f6ae40ff420 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.363+0000 7f6aeb03f700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae41a2850 con 0x7f6ae4101340 2024-04-05T22:07:43.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6ae8ddb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ae40ff420 0x7f6ae4100e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6ae8ddb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ae40ff420 0x7f6ae4100e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48574/0 (socket says 172.21.15.121:48574) 2024-04-05T22:07:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6ae8ddb700 1 -- 172.21.15.121:0/366398355 learned_addr learned my addr 172.21.15.121:0/366398355 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6ae3fff700 1 --2- 172.21.15.121:0/366398355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ae4101340 0x7f6ae41017c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6ae8ddb700 1 -- 172.21.15.121:0/366398355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ae4101340 msgr2=0x7f6ae41017c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6ae8ddb700 1 --2- 172.21.15.121:0/366398355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ae4101340 0x7f6ae41017c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6ae8ddb700 1 -- 172.21.15.121:0/366398355 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad80095f0 con 0x7f6ae40ff420 2024-04-05T22:07:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6ae3fff700 1 --2- 172.21.15.121:0/366398355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ae4101340 0x7f6ae41017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:07:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6ae8ddb700 1 --2- 172.21.15.121:0/366398355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ae40ff420 0x7f6ae4100e00 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8006010 tx=0x7f6ad800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6ae1ffb700 1 -- 172.21.15.121:0/366398355 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ad801d070 con 0x7f6ae40ff420 2024-04-05T22:07:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6ae1ffb700 1 -- 172.21.15.121:0/366398355 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ad800f460 con 0x7f6ae40ff420 2024-04-05T22:07:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6aeb03f700 1 -- 172.21.15.121:0/366398355 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae41a2990 con 0x7f6ae40ff420 2024-04-05T22:07:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6ae1ffb700 1 -- 172.21.15.121:0/366398355 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ad8017630 con 0x7f6ae40ff420 2024-04-05T22:07:43.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.364+0000 7f6aeb03f700 1 -- 172.21.15.121:0/366398355 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae41a2e40 con 0x7f6ae40ff420 2024-04-05T22:07:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.365+0000 7f6aeb03f700 1 -- 172.21.15.121:0/366398355 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae41096e0 con 0x7f6ae40ff420 2024-04-05T22:07:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.366+0000 7f6ae1ffb700 1 -- 172.21.15.121:0/366398355 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6ad800f9b0 con 0x7f6ae40ff420 2024-04-05T22:07:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.366+0000 7f6ae1ffb700 1 --2- 172.21.15.121:0/366398355 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6ad406cda0 0x7f6ad406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.366+0000 7f6ae3fff700 1 --2- 172.21.15.121:0/366398355 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6ad406cda0 0x7f6ad406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.366+0000 7f6ae1ffb700 1 -- 172.21.15.121:0/366398355 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6ad808d930 con 0x7f6ae40ff420 2024-04-05T22:07:43.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.367+0000 7f6ae3fff700 1 --2- 172.21.15.121:0/366398355 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6ad406cda0 0x7f6ad406f260 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0005fd0 tx=0x7f6ad0005ee0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:43.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.371+0000 7f6ae1ffb700 1 -- 172.21.15.121:0/366398355 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6ad8057a20 con 0x7f6ae40ff420 2024-04-05T22:07:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.611+0000 7f6aeb03f700 1 -- 172.21.15.121:0/366398355 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7f6ae404ea90 con 0x7f6ae40ff420 2024-04-05T22:07:43.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.611+0000 7f6ae1ffb700 1 -- 172.21.15.121:0/366398355 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v41) v1 ==== 107+0+4830 (secure 0 0 0) 0x7f6ad805b040 con 0x7f6ae40ff420 2024-04-05T22:07:43.611 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:43.611 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34320,"name":"cephfs.smithi121.gxheqt","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6827/1474300111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1474300111},{"type":"v1","addr":"172.21.15.121:6827","nonce":1474300111}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":34328,"name":"cephfs.smithi121.ifoxmw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6829/1553603485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1553603485},{"type":"v1","addr":"172.21.15.121:6829","nonce":1553603485}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44245,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/963169033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":963169033},{"type":"v1","addr":"172.21.15.188:6827","nonce":963169033}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":44255,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2257429686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2257429686},{"type":"v1","addr":"172.21.15.188:6825","nonce":2257429686}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:34:37.155847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:43.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.613+0000 7f6aeb03f700 1 -- 172.21.15.121:0/366398355 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6ad406cda0 msgr2=0x7f6ad406f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:43.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.613+0000 7f6aeb03f700 1 --2- 172.21.15.121:0/366398355 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6ad406cda0 0x7f6ad406f260 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0005fd0 tx=0x7f6ad0005ee0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.613+0000 7f6aeb03f700 1 -- 172.21.15.121:0/366398355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ae40ff420 msgr2=0x7f6ae4100e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:43.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.613+0000 7f6aeb03f700 1 --2- 172.21.15.121:0/366398355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ae40ff420 0x7f6ae4100e00 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8006010 tx=0x7f6ad800be20 comp rx=0 tx=0).stop 2024-04-05T22:07:43.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.613+0000 7f6aeb03f700 1 -- 172.21.15.121:0/366398355 shutdown_connections 2024-04-05T22:07:43.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.613+0000 7f6aeb03f700 1 --2- 172.21.15.121:0/366398355 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6ad406cda0 0x7f6ad406f260 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.613+0000 7f6aeb03f700 1 --2- 172.21.15.121:0/366398355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ae40ff420 0x7f6ae4100e00 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.613+0000 7f6aeb03f700 1 --2- 172.21.15.121:0/366398355 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6ae4101340 0x7f6ae41017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.613+0000 7f6aeb03f700 1 -- 172.21.15.121:0/366398355 >> 172.21.15.121:0/366398355 conn(0x7f6ae4074c20 msgr2=0x7f6ae40fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:43.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.614+0000 7f6aeb03f700 1 -- 172.21.15.121:0/366398355 shutdown_connections 2024-04-05T22:07:43.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.614+0000 7f6aeb03f700 1 -- 172.21.15.121:0/366398355 wait complete. 2024-04-05T22:07:43.615 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 24 2024-04-05T22:07:43.626 DEBUG:tasks.fs:max_mds not reduced in epoch 24 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T22:07:43.626 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 24 2024-04-05T22:07:43.626 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-04-05T22:07:43.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 -- 172.21.15.121:0/229814274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fa8106a80 msgr2=0x7f0fa8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:43.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 --2- 172.21.15.121:0/229814274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fa8106a80 0x7f0fa8106e60 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f0f98009910 tx=0x7f0f98009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:43.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 -- 172.21.15.121:0/229814274 shutdown_connections 2024-04-05T22:07:43.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 --2- 172.21.15.121:0/229814274 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fa8067da0 0x7f0fa8068220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 --2- 172.21.15.121:0/229814274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fa8106a80 0x7f0fa8106e60 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 -- 172.21.15.121:0/229814274 >> 172.21.15.121:0/229814274 conn(0x7f0fa8075400 msgr2=0x7f0fa8075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:43.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 -- 172.21.15.121:0/229814274 shutdown_connections 2024-04-05T22:07:43.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 -- 172.21.15.121:0/229814274 wait complete. 2024-04-05T22:07:43.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 Processor -- start 2024-04-05T22:07:43.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 -- start start 2024-04-05T22:07:43.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fa8067da0 0x7f0fa81964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:43.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fa8106a80 0x7f0fa81969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:43.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa81970d0 con 0x7f0fa8106a80 2024-04-05T22:07:43.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.748+0000 7f0fb0076700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa819ae50 con 0x7f0fa8067da0 2024-04-05T22:07:43.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0fad611700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fa8106a80 0x7f0fa81969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:43.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0fad611700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fa8106a80 0x7f0fa81969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48600/0 (socket says 172.21.15.121:48600) 2024-04-05T22:07:43.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0fad611700 1 -- 172.21.15.121:0/1659491692 learned_addr learned my addr 172.21.15.121:0/1659491692 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:43.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0fade12700 1 --2- 172.21.15.121:0/1659491692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fa8067da0 0x7f0fa81964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:43.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0fad611700 1 -- 172.21.15.121:0/1659491692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fa8067da0 msgr2=0x7f0fa81964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:43.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0fad611700 1 --2- 172.21.15.121:0/1659491692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fa8067da0 0x7f0fa81964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0fad611700 1 -- 172.21.15.121:0/1659491692 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f980095f0 con 0x7f0fa8106a80 2024-04-05T22:07:43.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0fade12700 1 --2- 172.21.15.121:0/1659491692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fa8067da0 0x7f0fa81964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:07:43.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0fad611700 1 --2- 172.21.15.121:0/1659491692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fa8106a80 0x7f0fa81969f0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400d8d0 tx=0x7f0fa400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:43.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0f9effd700 1 -- 172.21.15.121:0/1659491692 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fa4009940 con 0x7f0fa8106a80 2024-04-05T22:07:43.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0fb0076700 1 -- 172.21.15.121:0/1659491692 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa819b130 con 0x7f0fa8106a80 2024-04-05T22:07:43.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0f9effd700 1 -- 172.21.15.121:0/1659491692 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fa4010460 con 0x7f0fa8106a80 2024-04-05T22:07:43.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0f9effd700 1 -- 172.21.15.121:0/1659491692 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fa400f5f0 con 0x7f0fa8106a80 2024-04-05T22:07:43.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.749+0000 7f0fb0076700 1 -- 172.21.15.121:0/1659491692 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa819b690 con 0x7f0fa8106a80 2024-04-05T22:07:43.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.750+0000 7f0fb0076700 1 -- 172.21.15.121:0/1659491692 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa804ea90 con 0x7f0fa8106a80 2024-04-05T22:07:43.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.750+0000 7f0f9effd700 1 -- 172.21.15.121:0/1659491692 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0fa40105d0 con 0x7f0fa8106a80 2024-04-05T22:07:43.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.751+0000 7f0f9effd700 1 --2- 172.21.15.121:0/1659491692 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0f9406cda0 0x7f0f9406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:43.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.751+0000 7f0f9effd700 1 -- 172.21.15.121:0/1659491692 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0fa408c5e0 con 0x7f0fa8106a80 2024-04-05T22:07:43.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.751+0000 7f0fade12700 1 --2- 172.21.15.121:0/1659491692 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0f9406cda0 0x7f0f9406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:43.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.751+0000 7f0fade12700 1 --2- 172.21.15.121:0/1659491692 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0f9406cda0 0x7f0f9406f260 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0f9800b5c0 tx=0x7f0f9801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:43.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.755+0000 7f0f9effd700 1 -- 172.21.15.121:0/1659491692 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0fa4056650 con 0x7f0fa8106a80 2024-04-05T22:07:43.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:43 smithi121 ceph-mon[145859]: pgmap v1247: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:07:43.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:43 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1636121523' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-05T22:07:43.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:43 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2375580940' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-05T22:07:43.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:43 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/366398355' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-05T22:07:43.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.994+0000 7f0fb0076700 1 -- 172.21.15.121:0/1659491692 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7f0fa819b950 con 0x7f0fa8106a80 2024-04-05T22:07:43.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.995+0000 7f0f9effd700 1 -- 172.21.15.121:0/1659491692 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v41) v1 ==== 107+0+4048 (secure 0 0 0) 0x7f0fa4010880 con 0x7f0fa8106a80 2024-04-05T22:07:43.995 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:43.995 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34328,"name":"cephfs.smithi121.ifoxmw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6829/1553603485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1553603485},{"type":"v1","addr":"172.21.15.121:6829","nonce":1553603485}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44245,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/963169033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":963169033},{"type":"v1","addr":"172.21.15.188:6827","nonce":963169033}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":44255,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2257429686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2257429686},{"type":"v1","addr":"172.21.15.188:6825","nonce":2257429686}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:34:37.155847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:43.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.997+0000 7f0fb0076700 1 -- 172.21.15.121:0/1659491692 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0f9406cda0 msgr2=0x7f0f9406f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:43.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.997+0000 7f0fb0076700 1 --2- 172.21.15.121:0/1659491692 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0f9406cda0 0x7f0f9406f260 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0f9800b5c0 tx=0x7f0f9801a040 comp rx=0 tx=0).stop 2024-04-05T22:07:43.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.997+0000 7f0fb0076700 1 -- 172.21.15.121:0/1659491692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fa8106a80 msgr2=0x7f0fa81969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:43.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.997+0000 7f0fb0076700 1 --2- 172.21.15.121:0/1659491692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fa8106a80 0x7f0fa81969f0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400d8d0 tx=0x7f0fa400dc90 comp rx=0 tx=0).stop 2024-04-05T22:07:43.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.997+0000 7f0fb0076700 1 -- 172.21.15.121:0/1659491692 shutdown_connections 2024-04-05T22:07:43.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.997+0000 7f0fb0076700 1 --2- 172.21.15.121:0/1659491692 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0f9406cda0 0x7f0f9406f260 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.997+0000 7f0fb0076700 1 --2- 172.21.15.121:0/1659491692 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0fa8067da0 0x7f0fa81964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.997+0000 7f0fb0076700 1 --2- 172.21.15.121:0/1659491692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fa8106a80 0x7f0fa81969f0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:43.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.997+0000 7f0fb0076700 1 -- 172.21.15.121:0/1659491692 >> 172.21.15.121:0/1659491692 conn(0x7f0fa8075400 msgr2=0x7f0fa80fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:43.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.997+0000 7f0fb0076700 1 -- 172.21.15.121:0/1659491692 shutdown_connections 2024-04-05T22:07:43.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:43.997+0000 7f0fb0076700 1 -- 172.21.15.121:0/1659491692 wait complete. 2024-04-05T22:07:43.998 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 25 2024-04-05T22:07:44.009 DEBUG:tasks.fs:max_mds not reduced in epoch 25 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T22:07:44.009 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 25 2024-04-05T22:07:44.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-04-05T22:07:44.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.131+0000 7f5480008700 1 -- 172.21.15.121:0/16279224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54780ff990 msgr2=0x7f54780ffe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:44.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.131+0000 7f5480008700 1 --2- 172.21.15.121:0/16279224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54780ff990 0x7f54780ffe10 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f5468009910 tx=0x7f5468009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:44.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.131+0000 7f5480008700 1 -- 172.21.15.121:0/16279224 shutdown_connections 2024-04-05T22:07:44.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.131+0000 7f5480008700 1 --2- 172.21.15.121:0/16279224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54780ff990 0x7f54780ffe10 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.131+0000 7f5480008700 1 --2- 172.21.15.121:0/16279224 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54781059a0 0x7f5478105d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.131+0000 7f5480008700 1 -- 172.21.15.121:0/16279224 >> 172.21.15.121:0/16279224 conn(0x7f54780fb4c0 msgr2=0x7f54780fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.132+0000 7f5480008700 1 -- 172.21.15.121:0/16279224 shutdown_connections 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.132+0000 7f5480008700 1 -- 172.21.15.121:0/16279224 wait complete. 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.132+0000 7f5480008700 1 Processor -- start 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.132+0000 7f5480008700 1 -- start start 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.132+0000 7f5480008700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54780ff990 0x7f5478196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.132+0000 7f5480008700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54781059a0 0x7f5478196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.132+0000 7f5480008700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54781970b0 con 0x7f54780ff990 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.132+0000 7f5480008700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54781971f0 con 0x7f54781059a0 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.132+0000 7f547dda4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54780ff990 0x7f5478196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.132+0000 7f547dda4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54780ff990 0x7f5478196520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48610/0 (socket says 172.21.15.121:48610) 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.132+0000 7f547dda4700 1 -- 172.21.15.121:0/3687509904 learned_addr learned my addr 172.21.15.121:0/3687509904 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:44.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.133+0000 7f547d5a3700 1 --2- 172.21.15.121:0/3687509904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54781059a0 0x7f5478196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:44.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.133+0000 7f547dda4700 1 -- 172.21.15.121:0/3687509904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54781059a0 msgr2=0x7f5478196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:44.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.133+0000 7f547dda4700 1 --2- 172.21.15.121:0/3687509904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54781059a0 0x7f5478196a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.133+0000 7f547dda4700 1 -- 172.21.15.121:0/3687509904 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54680095f0 con 0x7f54780ff990 2024-04-05T22:07:44.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.133+0000 7f547dda4700 1 --2- 172.21.15.121:0/3687509904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54780ff990 0x7f5478196520 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f547400c930 tx=0x7f547400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:44.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.133+0000 7f546effd700 1 -- 172.21.15.121:0/3687509904 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5474007ad0 con 0x7f54780ff990 2024-04-05T22:07:44.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.133+0000 7f546effd700 1 -- 172.21.15.121:0/3687509904 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5474007c30 con 0x7f54780ff990 2024-04-05T22:07:44.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.133+0000 7f5480008700 1 -- 172.21.15.121:0/3687509904 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547819b060 con 0x7f54780ff990 2024-04-05T22:07:44.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.133+0000 7f546effd700 1 -- 172.21.15.121:0/3687509904 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5474018630 con 0x7f54780ff990 2024-04-05T22:07:44.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.133+0000 7f5480008700 1 -- 172.21.15.121:0/3687509904 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547819b5c0 con 0x7f54780ff990 2024-04-05T22:07:44.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.134+0000 7f5480008700 1 -- 172.21.15.121:0/3687509904 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5478108f30 con 0x7f54780ff990 2024-04-05T22:07:44.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.134+0000 7f546effd700 1 -- 172.21.15.121:0/3687509904 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f547400f450 con 0x7f54780ff990 2024-04-05T22:07:44.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.135+0000 7f546effd700 1 --2- 172.21.15.121:0/3687509904 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f546406cda0 0x7f546406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:44.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.135+0000 7f546effd700 1 -- 172.21.15.121:0/3687509904 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f547408c780 con 0x7f54780ff990 2024-04-05T22:07:44.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.135+0000 7f547d5a3700 1 --2- 172.21.15.121:0/3687509904 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f546406cda0 0x7f546406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:44.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.135+0000 7f547d5a3700 1 --2- 172.21.15.121:0/3687509904 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f546406cda0 0x7f546406f260 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f546800b5c0 tx=0x7f5468005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:44.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.139+0000 7f546effd700 1 -- 172.21.15.121:0/3687509904 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5474056920 con 0x7f54780ff990 2024-04-05T22:07:44.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:43 smithi188 ceph-mon[107793]: pgmap v1247: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:07:44.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:43 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1636121523' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-05T22:07:44.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:43 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2375580940' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-05T22:07:44.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:43 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/366398355' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-05T22:07:44.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.377+0000 7f5480008700 1 -- 172.21.15.121:0/3687509904 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7f547804ea90 con 0x7f54780ff990 2024-04-05T22:07:44.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.378+0000 7f546effd700 1 -- 172.21.15.121:0/3687509904 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v41) v1 ==== 107+0+4830 (secure 0 0 0) 0x7f547408ae60 con 0x7f54780ff990 2024-04-05T22:07:44.377 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:44.378 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34328,"name":"cephfs.smithi121.ifoxmw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6829/1553603485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1553603485},{"type":"v1","addr":"172.21.15.121:6829","nonce":1553603485}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44245,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/963169033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":963169033},{"type":"v1","addr":"172.21.15.188:6827","nonce":963169033}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":44255,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2257429686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2257429686},{"type":"v1","addr":"172.21.15.188:6825","nonce":2257429686}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:34:37.155847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:44.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.379+0000 7f5480008700 1 -- 172.21.15.121:0/3687509904 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f546406cda0 msgr2=0x7f546406f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:44.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.379+0000 7f5480008700 1 --2- 172.21.15.121:0/3687509904 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f546406cda0 0x7f546406f260 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f546800b5c0 tx=0x7f5468005fb0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.379+0000 7f5480008700 1 -- 172.21.15.121:0/3687509904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54780ff990 msgr2=0x7f5478196520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:44.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.379+0000 7f5480008700 1 --2- 172.21.15.121:0/3687509904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54780ff990 0x7f5478196520 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f547400c930 tx=0x7f547400ccf0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.380+0000 7f5480008700 1 -- 172.21.15.121:0/3687509904 shutdown_connections 2024-04-05T22:07:44.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.380+0000 7f5480008700 1 --2- 172.21.15.121:0/3687509904 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f546406cda0 0x7f546406f260 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.380+0000 7f5480008700 1 --2- 172.21.15.121:0/3687509904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54780ff990 0x7f5478196520 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.380+0000 7f5480008700 1 --2- 172.21.15.121:0/3687509904 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f54781059a0 0x7f5478196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.380+0000 7f5480008700 1 -- 172.21.15.121:0/3687509904 >> 172.21.15.121:0/3687509904 conn(0x7f54780fb4c0 msgr2=0x7f5478068fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:44.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.380+0000 7f5480008700 1 -- 172.21.15.121:0/3687509904 shutdown_connections 2024-04-05T22:07:44.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.380+0000 7f5480008700 1 -- 172.21.15.121:0/3687509904 wait complete. 2024-04-05T22:07:44.381 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 26 2024-04-05T22:07:44.392 DEBUG:tasks.fs:max_mds not reduced in epoch 26 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T22:07:44.392 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 26 2024-04-05T22:07:44.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-04-05T22:07:44.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.514+0000 7f6de81dc700 1 -- 172.21.15.121:0/2012704776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0102610 msgr2=0x7f6de010aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:44.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.514+0000 7f6de81dc700 1 --2- 172.21.15.121:0/2012704776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0102610 0x7f6de010aaf0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009910 tx=0x7f6dd0009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:44.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.515+0000 7f6de81dc700 1 -- 172.21.15.121:0/2012704776 shutdown_connections 2024-04-05T22:07:44.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.515+0000 7f6de81dc700 1 --2- 172.21.15.121:0/2012704776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0102610 0x7f6de010aaf0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.515+0000 7f6de81dc700 1 --2- 172.21.15.121:0/2012704776 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de0101cf0 0x7f6de01020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.515+0000 7f6de81dc700 1 -- 172.21.15.121:0/2012704776 >> 172.21.15.121:0/2012704776 conn(0x7f6de00fb430 msgr2=0x7f6de00fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:44.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.515+0000 7f6de81dc700 1 -- 172.21.15.121:0/2012704776 shutdown_connections 2024-04-05T22:07:44.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.515+0000 7f6de81dc700 1 -- 172.21.15.121:0/2012704776 wait complete. 2024-04-05T22:07:44.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.515+0000 7f6de81dc700 1 Processor -- start 2024-04-05T22:07:44.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.515+0000 7f6de81dc700 1 -- start start 2024-04-05T22:07:44.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.515+0000 7f6de81dc700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de0101cf0 0x7f6de00ff940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:44.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.515+0000 7f6de81dc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0102610 0x7f6de00ffe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:44.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.515+0000 7f6de81dc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de01003c0 con 0x7f6de0102610 2024-04-05T22:07:44.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.515+0000 7f6de81dc700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de0100500 con 0x7f6de0101cf0 2024-04-05T22:07:44.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6de5777700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0102610 0x7f6de00ffe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:44.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6de5777700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0102610 0x7f6de00ffe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36822/0 (socket says 172.21.15.121:36822) 2024-04-05T22:07:44.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6de5777700 1 -- 172.21.15.121:0/3191474700 learned_addr learned my addr 172.21.15.121:0/3191474700 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:44.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6de5777700 1 -- 172.21.15.121:0/3191474700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de0101cf0 msgr2=0x7f6de00ff940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:44.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6de5f78700 1 --2- 172.21.15.121:0/3191474700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de0101cf0 0x7f6de00ff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:44.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6de5777700 1 --2- 172.21.15.121:0/3191474700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de0101cf0 0x7f6de00ff940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6de5777700 1 -- 172.21.15.121:0/3191474700 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd00095f0 con 0x7f6de0102610 2024-04-05T22:07:44.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6de5f78700 1 --2- 172.21.15.121:0/3191474700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de0101cf0 0x7f6de00ff940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:07:44.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6de5777700 1 --2- 172.21.15.121:0/3191474700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0102610 0x7f6de00ffe80 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0006010 tx=0x7f6dd000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:44.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6dd6ffd700 1 -- 172.21.15.121:0/3191474700 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd001d070 con 0x7f6de0102610 2024-04-05T22:07:44.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6de81dc700 1 -- 172.21.15.121:0/3191474700 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de0100780 con 0x7f6de0102610 2024-04-05T22:07:44.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6dd6ffd700 1 -- 172.21.15.121:0/3191474700 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6dd000f460 con 0x7f6de0102610 2024-04-05T22:07:44.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6dd6ffd700 1 -- 172.21.15.121:0/3191474700 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd0021640 con 0x7f6de0102610 2024-04-05T22:07:44.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.516+0000 7f6de81dc700 1 -- 172.21.15.121:0/3191474700 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de01a2a20 con 0x7f6de0102610 2024-04-05T22:07:44.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.517+0000 7f6de81dc700 1 -- 172.21.15.121:0/3191474700 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de0108190 con 0x7f6de0102610 2024-04-05T22:07:44.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.517+0000 7f6dd6ffd700 1 -- 172.21.15.121:0/3191474700 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f6dd000f5d0 con 0x7f6de0102610 2024-04-05T22:07:44.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.518+0000 7f6dd6ffd700 1 --2- 172.21.15.121:0/3191474700 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6dcc06cda0 0x7f6dcc06f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:44.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.518+0000 7f6dd6ffd700 1 -- 172.21.15.121:0/3191474700 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f6dd008d7e0 con 0x7f6de0102610 2024-04-05T22:07:44.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.518+0000 7f6de5f78700 1 --2- 172.21.15.121:0/3191474700 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6dcc06cda0 0x7f6dcc06f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:44.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.518+0000 7f6de5f78700 1 --2- 172.21.15.121:0/3191474700 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6dcc06cda0 0x7f6dcc06f260 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc005fd0 tx=0x7f6ddc005ee0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:44.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.522+0000 7f6dd6ffd700 1 -- 172.21.15.121:0/3191474700 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f6dd0057850 con 0x7f6de0102610 2024-04-05T22:07:44.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.762+0000 7f6de81dc700 1 -- 172.21.15.121:0/3191474700 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7f6de004ea90 con 0x7f6de0102610 2024-04-05T22:07:44.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.763+0000 7f6dd6ffd700 1 -- 172.21.15.121:0/3191474700 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v41) v1 ==== 107+0+4048 (secure 0 0 0) 0x7f6dd000f880 con 0x7f6de0102610 2024-04-05T22:07:44.763 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:44.763 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":44245,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/963169033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":963169033},{"type":"v1","addr":"172.21.15.188:6827","nonce":963169033}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":44255,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2257429686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2257429686},{"type":"v1","addr":"172.21.15.188:6825","nonce":2257429686}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:34:37.155847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:44.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.765+0000 7f6de81dc700 1 -- 172.21.15.121:0/3191474700 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6dcc06cda0 msgr2=0x7f6dcc06f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:44.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.765+0000 7f6de81dc700 1 --2- 172.21.15.121:0/3191474700 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6dcc06cda0 0x7f6dcc06f260 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc005fd0 tx=0x7f6ddc005ee0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.765+0000 7f6de81dc700 1 -- 172.21.15.121:0/3191474700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0102610 msgr2=0x7f6de00ffe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:44.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.765+0000 7f6de81dc700 1 --2- 172.21.15.121:0/3191474700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0102610 0x7f6de00ffe80 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0006010 tx=0x7f6dd000be20 comp rx=0 tx=0).stop 2024-04-05T22:07:44.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.765+0000 7f6de81dc700 1 -- 172.21.15.121:0/3191474700 shutdown_connections 2024-04-05T22:07:44.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.765+0000 7f6de81dc700 1 --2- 172.21.15.121:0/3191474700 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f6dcc06cda0 0x7f6dcc06f260 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.765+0000 7f6de81dc700 1 --2- 172.21.15.121:0/3191474700 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f6de0101cf0 0x7f6de00ff940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.765+0000 7f6de81dc700 1 --2- 172.21.15.121:0/3191474700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6de0102610 0x7f6de00ffe80 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.765+0000 7f6de81dc700 1 -- 172.21.15.121:0/3191474700 >> 172.21.15.121:0/3191474700 conn(0x7f6de00fb430 msgr2=0x7f6de01052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:44.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.765+0000 7f6de81dc700 1 -- 172.21.15.121:0/3191474700 shutdown_connections 2024-04-05T22:07:44.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.765+0000 7f6de81dc700 1 -- 172.21.15.121:0/3191474700 wait complete. 2024-04-05T22:07:44.766 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 27 2024-04-05T22:07:44.777 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T22:07:44.777 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 27 2024-04-05T22:07:44.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-04-05T22:07:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.900+0000 7fc305954700 1 -- 172.21.15.121:0/3873996920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc300100530 msgr2=0x7fc300100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.900+0000 7fc305954700 1 --2- 172.21.15.121:0/3873996920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc300100530 0x7fc300100910 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4009910 tx=0x7fc2f4009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.900+0000 7fc305954700 1 -- 172.21.15.121:0/3873996920 shutdown_connections 2024-04-05T22:07:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.900+0000 7fc305954700 1 --2- 172.21.15.121:0/3873996920 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc300100ee0 0x7fc3001045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.900+0000 7fc305954700 1 --2- 172.21.15.121:0/3873996920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc300100530 0x7fc300100910 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.900+0000 7fc305954700 1 -- 172.21.15.121:0/3873996920 >> 172.21.15.121:0/3873996920 conn(0x7fc300074c20 msgr2=0x7fc300075030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.900+0000 7fc305954700 1 -- 172.21.15.121:0/3873996920 shutdown_connections 2024-04-05T22:07:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.900+0000 7fc305954700 1 -- 172.21.15.121:0/3873996920 wait complete. 2024-04-05T22:07:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.900+0000 7fc305954700 1 Processor -- start 2024-04-05T22:07:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.900+0000 7fc305954700 1 -- start start 2024-04-05T22:07:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc305954700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc300100530 0x7fc30019e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc305954700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc30019eac0 0x7fc3001985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc305954700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30019f000 con 0x7fc30019eac0 2024-04-05T22:07:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc305954700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30019f170 con 0x7fc300100530 2024-04-05T22:07:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc2fe7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc30019eac0 0x7fc3001985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc2fe7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc30019eac0 0x7fc3001985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36836/0 (socket says 172.21.15.121:36836) 2024-04-05T22:07:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc2fe7fc700 1 -- 172.21.15.121:0/1304613116 learned_addr learned my addr 172.21.15.121:0/1304613116 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc2feffd700 1 --2- 172.21.15.121:0/1304613116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc300100530 0x7fc30019e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc2fe7fc700 1 -- 172.21.15.121:0/1304613116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc300100530 msgr2=0x7fc30019e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc2fe7fc700 1 --2- 172.21.15.121:0/1304613116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc300100530 0x7fc30019e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc2fe7fc700 1 -- 172.21.15.121:0/1304613116 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f40095f0 con 0x7fc30019eac0 2024-04-05T22:07:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc2fe7fc700 1 --2- 172.21.15.121:0/1304613116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc30019eac0 0x7fc3001985f0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000c990 tx=0x7fc2f000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc304952700 1 -- 172.21.15.121:0/1304613116 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2f0007a30 con 0x7fc30019eac0 2024-04-05T22:07:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc304952700 1 -- 172.21.15.121:0/1304613116 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2f0007b90 con 0x7fc30019eac0 2024-04-05T22:07:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc305954700 1 -- 172.21.15.121:0/1304613116 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc300198bf0 con 0x7fc30019eac0 2024-04-05T22:07:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc304952700 1 -- 172.21.15.121:0/1304613116 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2f0018630 con 0x7fc30019eac0 2024-04-05T22:07:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.901+0000 7fc305954700 1 -- 172.21.15.121:0/1304613116 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc300199150 con 0x7fc30019eac0 2024-04-05T22:07:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.902+0000 7fc305954700 1 -- 172.21.15.121:0/1304613116 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc300199430 con 0x7fc30019eac0 2024-04-05T22:07:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.903+0000 7fc304952700 1 -- 172.21.15.121:0/1304613116 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc2f0007d00 con 0x7fc30019eac0 2024-04-05T22:07:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.903+0000 7fc304952700 1 --2- 172.21.15.121:0/1304613116 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc2e806cda0 0x7fc2e806f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.903+0000 7fc304952700 1 -- 172.21.15.121:0/1304613116 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc2f008c660 con 0x7fc30019eac0 2024-04-05T22:07:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.903+0000 7fc2feffd700 1 --2- 172.21.15.121:0/1304613116 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc2e806cda0 0x7fc2e806f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.904+0000 7fc2feffd700 1 --2- 172.21.15.121:0/1304613116 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc2e806cda0 0x7fc2e806f260 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4006010 tx=0x7fc2f401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:44.908+0000 7fc304952700 1 -- 172.21.15.121:0/1304613116 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc2f005a160 con 0x7fc30019eac0 2024-04-05T22:07:45.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:44 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1659491692' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-05T22:07:45.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:44 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3687509904' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-05T22:07:45.122 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:44 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3191474700' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-05T22:07:45.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.149+0000 7fc305954700 1 -- 172.21.15.121:0/1304613116 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7fc30004ea90 con 0x7fc30019eac0 2024-04-05T22:07:45.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.149+0000 7fc304952700 1 -- 172.21.15.121:0/1304613116 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v41) v1 ==== 107+0+4830 (secure 0 0 0) 0x7fc2f0059cf0 con 0x7fc30019eac0 2024-04-05T22:07:45.150 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:45.150 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44245,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/963169033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":963169033},{"type":"v1","addr":"172.21.15.188:6827","nonce":963169033}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":44255,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2257429686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2257429686},{"type":"v1","addr":"172.21.15.188:6825","nonce":2257429686}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:34:37.155847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:45.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.152+0000 7fc305954700 1 -- 172.21.15.121:0/1304613116 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc2e806cda0 msgr2=0x7fc2e806f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:45.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.152+0000 7fc305954700 1 --2- 172.21.15.121:0/1304613116 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc2e806cda0 0x7fc2e806f260 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4006010 tx=0x7fc2f401a040 comp rx=0 tx=0).stop 2024-04-05T22:07:45.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.152+0000 7fc305954700 1 -- 172.21.15.121:0/1304613116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc30019eac0 msgr2=0x7fc3001985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:45.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.152+0000 7fc305954700 1 --2- 172.21.15.121:0/1304613116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc30019eac0 0x7fc3001985f0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000c990 tx=0x7fc2f000cd50 comp rx=0 tx=0).stop 2024-04-05T22:07:45.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.152+0000 7fc305954700 1 -- 172.21.15.121:0/1304613116 shutdown_connections 2024-04-05T22:07:45.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.152+0000 7fc305954700 1 --2- 172.21.15.121:0/1304613116 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc2e806cda0 0x7fc2e806f260 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.152+0000 7fc305954700 1 --2- 172.21.15.121:0/1304613116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc300100530 0x7fc30019e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.152+0000 7fc305954700 1 --2- 172.21.15.121:0/1304613116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc30019eac0 0x7fc3001985f0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.152+0000 7fc305954700 1 -- 172.21.15.121:0/1304613116 >> 172.21.15.121:0/1304613116 conn(0x7fc300074c20 msgr2=0x7fc3001035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:45.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.152+0000 7fc305954700 1 -- 172.21.15.121:0/1304613116 shutdown_connections 2024-04-05T22:07:45.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.152+0000 7fc305954700 1 -- 172.21.15.121:0/1304613116 wait complete. 2024-04-05T22:07:45.154 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 28 2024-04-05T22:07:45.165 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T22:07:45.165 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 28 2024-04-05T22:07:45.165 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-04-05T22:07:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:44 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1659491692' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-05T22:07:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:44 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3687509904' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-05T22:07:45.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:44 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3191474700' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-05T22:07:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.288+0000 7f7c0dde3700 1 -- 172.21.15.121:0/1148022563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c08106a80 msgr2=0x7f7c08106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.288+0000 7f7c0dde3700 1 --2- 172.21.15.121:0/1148022563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c08106a80 0x7f7c08106e60 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc009910 tx=0x7f7bfc009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.288+0000 7f7c0dde3700 1 -- 172.21.15.121:0/1148022563 shutdown_connections 2024-04-05T22:07:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.288+0000 7f7c0dde3700 1 --2- 172.21.15.121:0/1148022563 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c08067da0 0x7f7c08068220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.288+0000 7f7c0dde3700 1 --2- 172.21.15.121:0/1148022563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c08106a80 0x7f7c08106e60 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.288+0000 7f7c0dde3700 1 -- 172.21.15.121:0/1148022563 >> 172.21.15.121:0/1148022563 conn(0x7f7c08075400 msgr2=0x7f7c08075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.288+0000 7f7c0dde3700 1 -- 172.21.15.121:0/1148022563 shutdown_connections 2024-04-05T22:07:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.288+0000 7f7c0dde3700 1 -- 172.21.15.121:0/1148022563 wait complete. 2024-04-05T22:07:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.288+0000 7f7c0dde3700 1 Processor -- start 2024-04-05T22:07:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c0dde3700 1 -- start start 2024-04-05T22:07:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c0dde3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c08067da0 0x7f7c081964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c0dde3700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c08106a80 0x7f7c081969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c0dde3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c08197040 con 0x7f7c08067da0 2024-04-05T22:07:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c0dde3700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c08197180 con 0x7f7c08106a80 2024-04-05T22:07:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c077fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c08067da0 0x7f7c081964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c077fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c08067da0 0x7f7c081964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36868/0 (socket says 172.21.15.121:36868) 2024-04-05T22:07:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c077fe700 1 -- 172.21.15.121:0/148500424 learned_addr learned my addr 172.21.15.121:0/148500424 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c06ffd700 1 --2- 172.21.15.121:0/148500424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c08106a80 0x7f7c081969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c077fe700 1 -- 172.21.15.121:0/148500424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c08106a80 msgr2=0x7f7c081969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c077fe700 1 --2- 172.21.15.121:0/148500424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c08106a80 0x7f7c081969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c077fe700 1 -- 172.21.15.121:0/148500424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bfc0095f0 con 0x7f7c08067da0 2024-04-05T22:07:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c06ffd700 1 --2- 172.21.15.121:0/148500424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c08106a80 0x7f7c081969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:07:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c077fe700 1 --2- 172.21.15.121:0/148500424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c08067da0 0x7f7c081964b0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc006010 tx=0x7f7bfc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c04ff9700 1 -- 172.21.15.121:0/148500424 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bfc01d070 con 0x7f7c08067da0 2024-04-05T22:07:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c04ff9700 1 -- 172.21.15.121:0/148500424 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bfc00f460 con 0x7f7c08067da0 2024-04-05T22:07:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.289+0000 7f7c0dde3700 1 -- 172.21.15.121:0/148500424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0819af90 con 0x7f7c08067da0 2024-04-05T22:07:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.290+0000 7f7c04ff9700 1 -- 172.21.15.121:0/148500424 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bfc021640 con 0x7f7c08067da0 2024-04-05T22:07:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.290+0000 7f7c0dde3700 1 -- 172.21.15.121:0/148500424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0819b490 con 0x7f7c08067da0 2024-04-05T22:07:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.290+0000 7f7c0dde3700 1 -- 172.21.15.121:0/148500424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0804ea90 con 0x7f7c08067da0 2024-04-05T22:07:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.291+0000 7f7c04ff9700 1 -- 172.21.15.121:0/148500424 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f7bfc00fa20 con 0x7f7c08067da0 2024-04-05T22:07:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.291+0000 7f7c04ff9700 1 --2- 172.21.15.121:0/148500424 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7bf006cda0 0x7f7bf006f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.291+0000 7f7c04ff9700 1 -- 172.21.15.121:0/148500424 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f7bfc08d960 con 0x7f7c08067da0 2024-04-05T22:07:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.291+0000 7f7c06ffd700 1 --2- 172.21.15.121:0/148500424 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7bf006cda0 0x7f7bf006f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.292+0000 7f7c06ffd700 1 --2- 172.21.15.121:0/148500424 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7bf006cda0 0x7f7bf006f260 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8005fd0 tx=0x7f7bf8005e30 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:45.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.296+0000 7f7c04ff9700 1 -- 172.21.15.121:0/148500424 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f7bfc057930 con 0x7f7c08067da0 2024-04-05T22:07:45.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.533+0000 7f7c0dde3700 1 -- 172.21.15.121:0/148500424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7f7c08197800 con 0x7f7c08067da0 2024-04-05T22:07:45.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.533+0000 7f7c04ff9700 1 -- 172.21.15.121:0/148500424 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v41) v1 ==== 107+0+4048 (secure 0 0 0) 0x7f7bfc05af50 con 0x7f7c08067da0 2024-04-05T22:07:45.535 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:45.535 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44245,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/963169033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":963169033},{"type":"v1","addr":"172.21.15.188:6827","nonce":963169033}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:34:37.155847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:45.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.536+0000 7f7c0dde3700 1 -- 172.21.15.121:0/148500424 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7bf006cda0 msgr2=0x7f7bf006f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:45.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.536+0000 7f7c0dde3700 1 --2- 172.21.15.121:0/148500424 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7bf006cda0 0x7f7bf006f260 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8005fd0 tx=0x7f7bf8005e30 comp rx=0 tx=0).stop 2024-04-05T22:07:45.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.536+0000 7f7c0dde3700 1 -- 172.21.15.121:0/148500424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c08067da0 msgr2=0x7f7c081964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:45.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.536+0000 7f7c0dde3700 1 --2- 172.21.15.121:0/148500424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c08067da0 0x7f7c081964b0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc006010 tx=0x7f7bfc00be20 comp rx=0 tx=0).stop 2024-04-05T22:07:45.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.536+0000 7f7c0dde3700 1 -- 172.21.15.121:0/148500424 shutdown_connections 2024-04-05T22:07:45.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.536+0000 7f7c0dde3700 1 --2- 172.21.15.121:0/148500424 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f7bf006cda0 0x7f7bf006f260 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.536+0000 7f7c0dde3700 1 --2- 172.21.15.121:0/148500424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c08067da0 0x7f7c081964b0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.536+0000 7f7c0dde3700 1 --2- 172.21.15.121:0/148500424 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f7c08106a80 0x7f7c081969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.536+0000 7f7c0dde3700 1 -- 172.21.15.121:0/148500424 >> 172.21.15.121:0/148500424 conn(0x7f7c08075400 msgr2=0x7f7c080fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:45.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.536+0000 7f7c0dde3700 1 -- 172.21.15.121:0/148500424 shutdown_connections 2024-04-05T22:07:45.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.536+0000 7f7c0dde3700 1 -- 172.21.15.121:0/148500424 wait complete. 2024-04-05T22:07:45.538 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 29 2024-04-05T22:07:45.549 DEBUG:tasks.fs:max_mds not reduced in epoch 29 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T22:07:45.549 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 29 2024-04-05T22:07:45.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-04-05T22:07:45.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 -- 172.21.15.121:0/2961105342 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17440ffac0 msgr2=0x7f17440fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 --2- 172.21.15.121:0/2961105342 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17440ffac0 0x7f17440fff20 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f1730009910 tx=0x7f1730009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 -- 172.21.15.121:0/2961105342 shutdown_connections 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 --2- 172.21.15.121:0/2961105342 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17440ffac0 0x7f17440fff20 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 --2- 172.21.15.121:0/2961105342 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1744105990 0x7f1744105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 -- 172.21.15.121:0/2961105342 >> 172.21.15.121:0/2961105342 conn(0x7f1744075400 msgr2=0x7f1744075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 -- 172.21.15.121:0/2961105342 shutdown_connections 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 -- 172.21.15.121:0/2961105342 wait complete. 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 Processor -- start 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 -- start start 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17440ffac0 0x7f17441953c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1744105990 0x7f1744195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1744195fe0 con 0x7f17440ffac0 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.670+0000 7f174abe9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1744199d60 con 0x7f1744105990 2024-04-05T22:07:45.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f1748985700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17440ffac0 0x7f17441953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:45.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f1748985700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17440ffac0 0x7f17441953c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36890/0 (socket says 172.21.15.121:36890) 2024-04-05T22:07:45.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f1748985700 1 -- 172.21.15.121:0/1863079306 learned_addr learned my addr 172.21.15.121:0/1863079306 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:45.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f1743fff700 1 --2- 172.21.15.121:0/1863079306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1744105990 0x7f1744195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:45.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f1748985700 1 -- 172.21.15.121:0/1863079306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1744105990 msgr2=0x7f1744195900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:45.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f1748985700 1 --2- 172.21.15.121:0/1863079306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1744105990 0x7f1744195900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f1748985700 1 -- 172.21.15.121:0/1863079306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17300095f0 con 0x7f17440ffac0 2024-04-05T22:07:45.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f1743fff700 1 --2- 172.21.15.121:0/1863079306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1744105990 0x7f1744195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:07:45.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f1748985700 1 --2- 172.21.15.121:0/1863079306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17440ffac0 0x7f17441953c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f173800e920 tx=0x7f173800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:45.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f1741ffb700 1 -- 172.21.15.121:0/1863079306 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1738009790 con 0x7f17440ffac0 2024-04-05T22:07:45.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f1741ffb700 1 -- 172.21.15.121:0/1863079306 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f173800cba0 con 0x7f17440ffac0 2024-04-05T22:07:45.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f174abe9700 1 -- 172.21.15.121:0/1863079306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f174419a040 con 0x7f17440ffac0 2024-04-05T22:07:45.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f1741ffb700 1 -- 172.21.15.121:0/1863079306 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1738018650 con 0x7f17440ffac0 2024-04-05T22:07:45.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.671+0000 7f174abe9700 1 -- 172.21.15.121:0/1863079306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174419a5a0 con 0x7f17440ffac0 2024-04-05T22:07:45.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.672+0000 7f174abe9700 1 -- 172.21.15.121:0/1863079306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1744107f10 con 0x7f17440ffac0 2024-04-05T22:07:45.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.673+0000 7f1741ffb700 1 -- 172.21.15.121:0/1863079306 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f173800cd10 con 0x7f17440ffac0 2024-04-05T22:07:45.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.673+0000 7f1741ffb700 1 --2- 172.21.15.121:0/1863079306 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f173406cda0 0x7f173406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:45.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.673+0000 7f1743fff700 1 --2- 172.21.15.121:0/1863079306 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f173406cda0 0x7f173406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:45.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.673+0000 7f1741ffb700 1 -- 172.21.15.121:0/1863079306 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f1738014070 con 0x7f17440ffac0 2024-04-05T22:07:45.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.674+0000 7f1743fff700 1 --2- 172.21.15.121:0/1863079306 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f173406cda0 0x7f173406f260 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f17441969d0 tx=0x7f1730009f90 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:45.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.678+0000 7f1741ffb700 1 -- 172.21.15.121:0/1863079306 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f1738052ab0 con 0x7f17440ffac0 2024-04-05T22:07:45.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.917+0000 7f174abe9700 1 -- 172.21.15.121:0/1863079306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7f174419a8d0 con 0x7f17440ffac0 2024-04-05T22:07:45.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.918+0000 7f1741ffb700 1 -- 172.21.15.121:0/1863079306 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v41) v1 ==== 107+0+4830 (secure 0 0 0) 0x7f173805a3f0 con 0x7f17440ffac0 2024-04-05T22:07:45.919 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:45.920 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44245,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/963169033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":963169033},{"type":"v1","addr":"172.21.15.188:6827","nonce":963169033}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23},{"gid":44297,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2880188624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2880188624},{"type":"v1","addr":"172.21.15.188:6825","nonce":2880188624}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:34:37.155847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:45.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.920+0000 7f174abe9700 1 -- 172.21.15.121:0/1863079306 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f173406cda0 msgr2=0x7f173406f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:45.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.920+0000 7f174abe9700 1 --2- 172.21.15.121:0/1863079306 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f173406cda0 0x7f173406f260 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f17441969d0 tx=0x7f1730009f90 comp rx=0 tx=0).stop 2024-04-05T22:07:45.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.920+0000 7f174abe9700 1 -- 172.21.15.121:0/1863079306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17440ffac0 msgr2=0x7f17441953c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:45.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.920+0000 7f174abe9700 1 --2- 172.21.15.121:0/1863079306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17440ffac0 0x7f17441953c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f173800e920 tx=0x7f173800ece0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.920+0000 7f174abe9700 1 -- 172.21.15.121:0/1863079306 shutdown_connections 2024-04-05T22:07:45.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.920+0000 7f174abe9700 1 --2- 172.21.15.121:0/1863079306 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f173406cda0 0x7f173406f260 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.920+0000 7f174abe9700 1 --2- 172.21.15.121:0/1863079306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17440ffac0 0x7f17441953c0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.920+0000 7f174abe9700 1 --2- 172.21.15.121:0/1863079306 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f1744105990 0x7f1744195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:45.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.920+0000 7f174abe9700 1 -- 172.21.15.121:0/1863079306 >> 172.21.15.121:0/1863079306 conn(0x7f1744075400 msgr2=0x7f17440fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:45.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.920+0000 7f174abe9700 1 -- 172.21.15.121:0/1863079306 shutdown_connections 2024-04-05T22:07:45.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:45.920+0000 7f174abe9700 1 -- 172.21.15.121:0/1863079306 wait complete. 2024-04-05T22:07:45.922 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-04-05T22:07:45.933 DEBUG:tasks.fs:max_mds not reduced in epoch 30 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T22:07:45.933 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 30 2024-04-05T22:07:45.934 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-04-05T22:07:46.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.055+0000 7f8d66d52700 1 -- 172.21.15.121:0/450768735 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d600ffac0 msgr2=0x7f8d600fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:46.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.055+0000 7f8d66d52700 1 --2- 172.21.15.121:0/450768735 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d600ffac0 0x7f8d600fff20 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c009910 tx=0x7f8d4c009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:46.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.055+0000 7f8d66d52700 1 -- 172.21.15.121:0/450768735 shutdown_connections 2024-04-05T22:07:46.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.055+0000 7f8d66d52700 1 --2- 172.21.15.121:0/450768735 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d600ffac0 0x7f8d600fff20 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.055+0000 7f8d66d52700 1 --2- 172.21.15.121:0/450768735 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d60105990 0x7f8d60105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.055+0000 7f8d66d52700 1 -- 172.21.15.121:0/450768735 >> 172.21.15.121:0/450768735 conn(0x7f8d60075400 msgr2=0x7f8d60075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.055+0000 7f8d66d52700 1 -- 172.21.15.121:0/450768735 shutdown_connections 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.055+0000 7f8d66d52700 1 -- 172.21.15.121:0/450768735 wait complete. 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.056+0000 7f8d66d52700 1 Processor -- start 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.056+0000 7f8d66d52700 1 -- start start 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.056+0000 7f8d66d52700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d600ffac0 0x7f8d601972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.056+0000 7f8d66d52700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d60105990 0x7f8d60197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.056+0000 7f8d66d52700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d60197ee0 con 0x7f8d600ffac0 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.056+0000 7f8d66d52700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d60198020 con 0x7f8d60105990 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.056+0000 7f8d64aee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d600ffac0 0x7f8d601972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.056+0000 7f8d64aee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d600ffac0 0x7f8d601972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36912/0 (socket says 172.21.15.121:36912) 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.056+0000 7f8d64aee700 1 -- 172.21.15.121:0/3627248771 learned_addr learned my addr 172.21.15.121:0/3627248771 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.056+0000 7f8d64aee700 1 -- 172.21.15.121:0/3627248771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d60105990 msgr2=0x7f8d60197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.056+0000 7f8d64aee700 1 --2- 172.21.15.121:0/3627248771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d60105990 0x7f8d60197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.056+0000 7f8d64aee700 1 -- 172.21.15.121:0/3627248771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c0095f0 con 0x7f8d600ffac0 2024-04-05T22:07:46.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.057+0000 7f8d64aee700 1 --2- 172.21.15.121:0/3627248771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d600ffac0 0x7f8d601972c0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400d8d0 tx=0x7f8d5400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:46.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.057+0000 7f8d5dffb700 1 -- 172.21.15.121:0/3627248771 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d54009940 con 0x7f8d600ffac0 2024-04-05T22:07:46.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.057+0000 7f8d5dffb700 1 -- 172.21.15.121:0/3627248771 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d54010460 con 0x7f8d600ffac0 2024-04-05T22:07:46.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.057+0000 7f8d66d52700 1 -- 172.21.15.121:0/3627248771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d601a0330 con 0x7f8d600ffac0 2024-04-05T22:07:46.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.057+0000 7f8d5dffb700 1 -- 172.21.15.121:0/3627248771 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d5400f5f0 con 0x7f8d600ffac0 2024-04-05T22:07:46.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.057+0000 7f8d66d52700 1 -- 172.21.15.121:0/3627248771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d601a0860 con 0x7f8d600ffac0 2024-04-05T22:07:46.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.058+0000 7f8d66d52700 1 -- 172.21.15.121:0/3627248771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d60107f10 con 0x7f8d600ffac0 2024-04-05T22:07:46.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.058+0000 7f8d5dffb700 1 -- 172.21.15.121:0/3627248771 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f8d540105d0 con 0x7f8d600ffac0 2024-04-05T22:07:46.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.058+0000 7f8d5dffb700 1 --2- 172.21.15.121:0/3627248771 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d5006cda0 0x7f8d5006f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:46.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.059+0000 7f8d5dffb700 1 -- 172.21.15.121:0/3627248771 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f8d5408c4f0 con 0x7f8d600ffac0 2024-04-05T22:07:46.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.059+0000 7f8d5ffff700 1 --2- 172.21.15.121:0/3627248771 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d5006cda0 0x7f8d5006f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:46.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.059+0000 7f8d5ffff700 1 --2- 172.21.15.121:0/3627248771 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d5006cda0 0x7f8d5006f260 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00b5c0 tx=0x7f8d4c005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:46.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.063+0000 7f8d5dffb700 1 -- 172.21.15.121:0/3627248771 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f8d540565e0 con 0x7f8d600ffac0 2024-04-05T22:07:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:45 smithi121 ceph-mon[145859]: pgmap v1248: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:07:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:45 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1304613116' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-05T22:07:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:07:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:45 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/148500424' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-05T22:07:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:45 smithi188 ceph-mon[107793]: pgmap v1248: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:07:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:45 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1304613116' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-05T22:07:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:07:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:45 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/148500424' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-05T22:07:46.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.306+0000 7f8d66d52700 1 -- 172.21.15.121:0/3627248771 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7f8d6004ea90 con 0x7f8d600ffac0 2024-04-05T22:07:46.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.307+0000 7f8d5dffb700 1 -- 172.21.15.121:0/3627248771 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v41) v1 ==== 107+0+4051 (secure 0 0 0) 0x7f8d54010880 con 0x7f8d600ffac0 2024-04-05T22:07:46.308 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:46.308 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44297,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2880188624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2880188624},{"type":"v1","addr":"172.21.15.188:6825","nonce":2880188624}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:34:37.155847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:46.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.309+0000 7f8d66d52700 1 -- 172.21.15.121:0/3627248771 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d5006cda0 msgr2=0x7f8d5006f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:46.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.309+0000 7f8d66d52700 1 --2- 172.21.15.121:0/3627248771 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d5006cda0 0x7f8d5006f260 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00b5c0 tx=0x7f8d4c005960 comp rx=0 tx=0).stop 2024-04-05T22:07:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.309+0000 7f8d66d52700 1 -- 172.21.15.121:0/3627248771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d600ffac0 msgr2=0x7f8d601972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.309+0000 7f8d66d52700 1 --2- 172.21.15.121:0/3627248771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d600ffac0 0x7f8d601972c0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400d8d0 tx=0x7f8d5400dc90 comp rx=0 tx=0).stop 2024-04-05T22:07:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.309+0000 7f8d66d52700 1 -- 172.21.15.121:0/3627248771 shutdown_connections 2024-04-05T22:07:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.309+0000 7f8d66d52700 1 --2- 172.21.15.121:0/3627248771 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f8d5006cda0 0x7f8d5006f260 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.309+0000 7f8d66d52700 1 --2- 172.21.15.121:0/3627248771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d600ffac0 0x7f8d601972c0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.309+0000 7f8d66d52700 1 --2- 172.21.15.121:0/3627248771 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f8d60105990 0x7f8d60197800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.309+0000 7f8d66d52700 1 -- 172.21.15.121:0/3627248771 >> 172.21.15.121:0/3627248771 conn(0x7f8d60075400 msgr2=0x7f8d600fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.309+0000 7f8d66d52700 1 -- 172.21.15.121:0/3627248771 shutdown_connections 2024-04-05T22:07:46.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.309+0000 7f8d66d52700 1 -- 172.21.15.121:0/3627248771 wait complete. 2024-04-05T22:07:46.314 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 31 2024-04-05T22:07:46.322 DEBUG:tasks.fs:max_mds not reduced in epoch 31 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T22:07:46.323 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 31 2024-04-05T22:07:46.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-04-05T22:07:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.444+0000 7f8070b83700 1 -- 172.21.15.121:0/3299814676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c105990 msgr2=0x7f806c105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.444+0000 7f8070b83700 1 --2- 172.21.15.121:0/3299814676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c105990 0x7f806c105d70 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f805c009910 tx=0x7f805c009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.444+0000 7f8070b83700 1 -- 172.21.15.121:0/3299814676 shutdown_connections 2024-04-05T22:07:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.444+0000 7f8070b83700 1 --2- 172.21.15.121:0/3299814676 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f806c0ffac0 0x7f806c0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.444+0000 7f8070b83700 1 --2- 172.21.15.121:0/3299814676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c105990 0x7f806c105d70 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.444+0000 7f8070b83700 1 -- 172.21.15.121:0/3299814676 >> 172.21.15.121:0/3299814676 conn(0x7f806c075400 msgr2=0x7f806c075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.444+0000 7f8070b83700 1 -- 172.21.15.121:0/3299814676 shutdown_connections 2024-04-05T22:07:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.444+0000 7f8070b83700 1 -- 172.21.15.121:0/3299814676 wait complete. 2024-04-05T22:07:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f8070b83700 1 Processor -- start 2024-04-05T22:07:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f8070b83700 1 -- start start 2024-04-05T22:07:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f8070b83700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f806c0ffac0 0x7f806c197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f8070b83700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c105990 0x7f806c197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f8070b83700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c197f90 con 0x7f806c105990 2024-04-05T22:07:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f8070b83700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c198100 con 0x7f806c0ffac0 2024-04-05T22:07:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f8069d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c105990 0x7f806c197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f8069d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c105990 0x7f806c197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36926/0 (socket says 172.21.15.121:36926) 2024-04-05T22:07:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f8069d9b700 1 -- 172.21.15.121:0/1960961156 learned_addr learned my addr 172.21.15.121:0/1960961156 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f806a59c700 1 --2- 172.21.15.121:0/1960961156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f806c0ffac0 0x7f806c197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f8069d9b700 1 -- 172.21.15.121:0/1960961156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f806c0ffac0 msgr2=0x7f806c197430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f8069d9b700 1 --2- 172.21.15.121:0/1960961156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f806c0ffac0 0x7f806c197430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f8069d9b700 1 -- 172.21.15.121:0/1960961156 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c0095f0 con 0x7f806c105990 2024-04-05T22:07:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.445+0000 7f8069d9b700 1 --2- 172.21.15.121:0/1960961156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c105990 0x7f806c197970 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f806000d8d0 tx=0x7f806000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.446+0000 7f805b7fe700 1 -- 172.21.15.121:0/1960961156 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80600098a0 con 0x7f806c105990 2024-04-05T22:07:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.446+0000 7f805b7fe700 1 -- 172.21.15.121:0/1960961156 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8060010460 con 0x7f806c105990 2024-04-05T22:07:46.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.446+0000 7f8070b83700 1 -- 172.21.15.121:0/1960961156 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f806c1a04a0 con 0x7f806c105990 2024-04-05T22:07:46.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.446+0000 7f805b7fe700 1 -- 172.21.15.121:0/1960961156 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8060009ea0 con 0x7f806c105990 2024-04-05T22:07:46.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.446+0000 7f8070b83700 1 -- 172.21.15.121:0/1960961156 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806c1a09a0 con 0x7f806c105990 2024-04-05T22:07:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.447+0000 7f8070b83700 1 -- 172.21.15.121:0/1960961156 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806c107f30 con 0x7f806c105990 2024-04-05T22:07:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.447+0000 7f805b7fe700 1 -- 172.21.15.121:0/1960961156 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f80600105d0 con 0x7f806c105990 2024-04-05T22:07:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.447+0000 7f805b7fe700 1 --2- 172.21.15.121:0/1960961156 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f805406cda0 0x7f805406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.448+0000 7f805b7fe700 1 -- 172.21.15.121:0/1960961156 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f806008c520 con 0x7f806c105990 2024-04-05T22:07:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.448+0000 7f806a59c700 1 --2- 172.21.15.121:0/1960961156 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f805406cda0 0x7f805406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.448+0000 7f806a59c700 1 --2- 172.21.15.121:0/1960961156 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f805406cda0 0x7f805406f260 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f805c006010 tx=0x7f805c01a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:46.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.452+0000 7f805b7fe700 1 -- 172.21.15.121:0/1960961156 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f80600564f0 con 0x7f806c105990 2024-04-05T22:07:46.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.703+0000 7f8070b83700 1 -- 172.21.15.121:0/1960961156 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f806c04ea90 con 0x7f806c105990 2024-04-05T22:07:46.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.704+0000 7f805b7fe700 1 -- 172.21.15.121:0/1960961156 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v41) v1 ==== 107+0+4833 (secure 0 0 0) 0x7f8060010880 con 0x7f806c105990 2024-04-05T22:07:46.705 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:46.705 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44297,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2880188624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2880188624},{"type":"v1","addr":"172.21.15.188:6825","nonce":2880188624}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44827,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/3202979994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3202979994},{"type":"v1","addr":"172.21.15.188:6827","nonce":3202979994}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":23,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T21:34:37.155847+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:46.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.706+0000 7f8070b83700 1 -- 172.21.15.121:0/1960961156 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f805406cda0 msgr2=0x7f805406f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:46.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.706+0000 7f8070b83700 1 --2- 172.21.15.121:0/1960961156 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f805406cda0 0x7f805406f260 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f805c006010 tx=0x7f805c01a040 comp rx=0 tx=0).stop 2024-04-05T22:07:46.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.706+0000 7f8070b83700 1 -- 172.21.15.121:0/1960961156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c105990 msgr2=0x7f806c197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:46.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.706+0000 7f8070b83700 1 --2- 172.21.15.121:0/1960961156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c105990 0x7f806c197970 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f806000d8d0 tx=0x7f806000dbe0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.706+0000 7f8070b83700 1 -- 172.21.15.121:0/1960961156 shutdown_connections 2024-04-05T22:07:46.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.706+0000 7f8070b83700 1 --2- 172.21.15.121:0/1960961156 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f805406cda0 0x7f805406f260 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.706+0000 7f8070b83700 1 --2- 172.21.15.121:0/1960961156 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f806c0ffac0 0x7f806c197430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.706+0000 7f8070b83700 1 --2- 172.21.15.121:0/1960961156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c105990 0x7f806c197970 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.706+0000 7f8070b83700 1 -- 172.21.15.121:0/1960961156 >> 172.21.15.121:0/1960961156 conn(0x7f806c075400 msgr2=0x7f806c0fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:46.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.706+0000 7f8070b83700 1 -- 172.21.15.121:0/1960961156 shutdown_connections 2024-04-05T22:07:46.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.706+0000 7f8070b83700 1 -- 172.21.15.121:0/1960961156 wait complete. 2024-04-05T22:07:46.708 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-04-05T22:07:46.719 DEBUG:tasks.fs:max_mds not reduced in epoch 32 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-04-05T22:07:46.719 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 32 2024-04-05T22:07:46.720 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-04-05T22:07:46.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 -- 172.21.15.121:0/3844852211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe64105960 msgr2=0x7fbe64105d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:46.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 --2- 172.21.15.121:0/3844852211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe64105960 0x7fbe64105d40 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009910 tx=0x7fbe54009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:46.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 -- 172.21.15.121:0/3844852211 shutdown_connections 2024-04-05T22:07:46.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 --2- 172.21.15.121:0/3844852211 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe640ff950 0x7fbe640ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 --2- 172.21.15.121:0/3844852211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe64105960 0x7fbe64105d40 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 -- 172.21.15.121:0/3844852211 >> 172.21.15.121:0/3844852211 conn(0x7fbe640fb420 msgr2=0x7fbe640fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:46.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 -- 172.21.15.121:0/3844852211 shutdown_connections 2024-04-05T22:07:46.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 -- 172.21.15.121:0/3844852211 wait complete. 2024-04-05T22:07:46.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 Processor -- start 2024-04-05T22:07:46.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 -- start start 2024-04-05T22:07:46.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe640ff950 0x7fbe64197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:46.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe64105960 0x7fbe64197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:46.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64197ea0 con 0x7fbe64105960 2024-04-05T22:07:46.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.841+0000 7fbe6c136700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64197fe0 con 0x7fbe640ff950 2024-04-05T22:07:46.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.842+0000 7fbe696d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe64105960 0x7fbe64197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:46.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.842+0000 7fbe696d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe64105960 0x7fbe64197880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36944/0 (socket says 172.21.15.121:36944) 2024-04-05T22:07:46.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.842+0000 7fbe696d1700 1 -- 172.21.15.121:0/3045400064 learned_addr learned my addr 172.21.15.121:0/3045400064 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:46.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.842+0000 7fbe69ed2700 1 --2- 172.21.15.121:0/3045400064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe640ff950 0x7fbe64197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:46.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.842+0000 7fbe696d1700 1 -- 172.21.15.121:0/3045400064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe640ff950 msgr2=0x7fbe64197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:46.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.842+0000 7fbe696d1700 1 --2- 172.21.15.121:0/3045400064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe640ff950 0x7fbe64197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:46.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.842+0000 7fbe696d1700 1 -- 172.21.15.121:0/3045400064 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe540095f0 con 0x7fbe64105960 2024-04-05T22:07:46.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.842+0000 7fbe696d1700 1 --2- 172.21.15.121:0/3045400064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe64105960 0x7fbe64197880 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000c8a0 tx=0x7fbe6000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:46.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.842+0000 7fbe5affd700 1 -- 172.21.15.121:0/3045400064 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe6000cea0 con 0x7fbe64105960 2024-04-05T22:07:46.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.842+0000 7fbe6c136700 1 -- 172.21.15.121:0/3045400064 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe641a03b0 con 0x7fbe64105960 2024-04-05T22:07:46.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.842+0000 7fbe5affd700 1 -- 172.21.15.121:0/3045400064 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe60004830 con 0x7fbe64105960 2024-04-05T22:07:46.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.842+0000 7fbe5affd700 1 -- 172.21.15.121:0/3045400064 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe60005500 con 0x7fbe64105960 2024-04-05T22:07:46.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.842+0000 7fbe6c136700 1 -- 172.21.15.121:0/3045400064 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe641a0880 con 0x7fbe64105960 2024-04-05T22:07:46.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.843+0000 7fbe6c136700 1 -- 172.21.15.121:0/3045400064 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6404ea90 con 0x7fbe64105960 2024-04-05T22:07:46.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.844+0000 7fbe5affd700 1 -- 172.21.15.121:0/3045400064 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbe600049a0 con 0x7fbe64105960 2024-04-05T22:07:46.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.844+0000 7fbe5affd700 1 --2- 172.21.15.121:0/3045400064 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbe5006cda0 0x7fbe5006f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:46.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.844+0000 7fbe5affd700 1 -- 172.21.15.121:0/3045400064 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbe6008c2b0 con 0x7fbe64105960 2024-04-05T22:07:46.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.844+0000 7fbe69ed2700 1 --2- 172.21.15.121:0/3045400064 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbe5006cda0 0x7fbe5006f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:46.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.845+0000 7fbe69ed2700 1 --2- 172.21.15.121:0/3045400064 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbe5006cda0 0x7fbe5006f260 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbe54006010 tx=0x7fbe5401a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:46.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:46.849+0000 7fbe5affd700 1 -- 172.21.15.121:0/3045400064 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbe60056320 con 0x7fbe64105960 2024-04-05T22:07:47.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.085+0000 7fbe6c136700 1 -- 172.21.15.121:0/3045400064 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7fbe64069470 con 0x7fbe64105960 2024-04-05T22:07:47.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.085+0000 7fbe5affd700 1 -- 172.21.15.121:0/3045400064 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v41) v1 ==== 107+0+4832 (secure 0 0 0) 0x7fbe60059940 con 0x7fbe64105960 2024-04-05T22:07:47.086 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:47.086 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":44297,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2880188624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2880188624},{"type":"v1","addr":"172.21.15.188:6825","nonce":2880188624}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44827,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/3202979994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3202979994},{"type":"v1","addr":"172.21.15.188:6827","nonce":3202979994}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T22:05:36.198281+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:47.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.088+0000 7fbe6c136700 1 -- 172.21.15.121:0/3045400064 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbe5006cda0 msgr2=0x7fbe5006f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:47.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.088+0000 7fbe6c136700 1 --2- 172.21.15.121:0/3045400064 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbe5006cda0 0x7fbe5006f260 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbe54006010 tx=0x7fbe5401a040 comp rx=0 tx=0).stop 2024-04-05T22:07:47.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.088+0000 7fbe6c136700 1 -- 172.21.15.121:0/3045400064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe64105960 msgr2=0x7fbe64197880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:47.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.088+0000 7fbe6c136700 1 --2- 172.21.15.121:0/3045400064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe64105960 0x7fbe64197880 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000c8a0 tx=0x7fbe6000cc60 comp rx=0 tx=0).stop 2024-04-05T22:07:47.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.088+0000 7fbe6c136700 1 -- 172.21.15.121:0/3045400064 shutdown_connections 2024-04-05T22:07:47.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.088+0000 7fbe6c136700 1 --2- 172.21.15.121:0/3045400064 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbe5006cda0 0x7fbe5006f260 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.088+0000 7fbe6c136700 1 --2- 172.21.15.121:0/3045400064 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbe640ff950 0x7fbe64197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.088+0000 7fbe6c136700 1 --2- 172.21.15.121:0/3045400064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe64105960 0x7fbe64197880 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.088+0000 7fbe6c136700 1 -- 172.21.15.121:0/3045400064 >> 172.21.15.121:0/3045400064 conn(0x7fbe640fb420 msgr2=0x7fbe640fbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:47.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.088+0000 7fbe6c136700 1 -- 172.21.15.121:0/3045400064 shutdown_connections 2024-04-05T22:07:47.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.088+0000 7fbe6c136700 1 -- 172.21.15.121:0/3045400064 wait complete. 2024-04-05T22:07:47.090 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 33 2024-04-05T22:07:47.101 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 33 2024-04-05T22:07:47.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-04-05T22:07:47.125 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:46 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1863079306' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-05T22:07:47.125 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:46 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3627248771' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-05T22:07:47.125 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:46 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1960961156' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-05T22:07:47.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.222+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1664097543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81d0106a90 msgr2=0x7f81d0106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.222+0000 7f81d6a6e700 1 --2- 172.21.15.121:0/1664097543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81d0106a90 0x7f81d0106e70 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f81c800b0a0 tx=0x7f81c800b3b0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.222+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1664097543 shutdown_connections 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.222+0000 7f81d6a6e700 1 --2- 172.21.15.121:0/1664097543 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81d0100a80 0x7f81d0100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.222+0000 7f81d6a6e700 1 --2- 172.21.15.121:0/1664097543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81d0106a90 0x7f81d0106e70 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.222+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1664097543 >> 172.21.15.121:0/1664097543 conn(0x7f81d00fc570 msgr2=0x7f81d00fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.222+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1664097543 shutdown_connections 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.222+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1664097543 wait complete. 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.222+0000 7f81d6a6e700 1 Processor -- start 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.222+0000 7f81d6a6e700 1 -- start start 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81d6a6e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81d0100a80 0x7f81d0073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81d6a6e700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81d0106a90 0x7f81d00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81d6a6e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d0073ba0 con 0x7f81d0100a80 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81d6a6e700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d0073d10 con 0x7f81d0106a90 2024-04-05T22:07:47.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81d480a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81d0100a80 0x7f81d0073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:47.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81d480a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81d0100a80 0x7f81d0073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36960/0 (socket says 172.21.15.121:36960) 2024-04-05T22:07:47.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81d480a700 1 -- 172.21.15.121:0/1784171544 learned_addr learned my addr 172.21.15.121:0/1784171544 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:47.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81d480a700 1 -- 172.21.15.121:0/1784171544 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81d0106a90 msgr2=0x7f81d00735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:47.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81d480a700 1 --2- 172.21.15.121:0/1784171544 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81d0106a90 0x7f81d00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81d480a700 1 -- 172.21.15.121:0/1784171544 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c8009d00 con 0x7f81d0100a80 2024-04-05T22:07:47.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81d480a700 1 --2- 172.21.15.121:0/1784171544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81d0100a80 0x7f81d0073090 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f81c800d010 tx=0x7f81c8007be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:47.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81cdffb700 1 -- 172.21.15.121:0/1784171544 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81c800e040 con 0x7f81d0100a80 2024-04-05T22:07:47.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81cdffb700 1 -- 172.21.15.121:0/1784171544 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81c8003e60 con 0x7f81d0100a80 2024-04-05T22:07:47.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1784171544 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d01a2750 con 0x7f81d0100a80 2024-04-05T22:07:47.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81cdffb700 1 -- 172.21.15.121:0/1784171544 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81c801d7c0 con 0x7f81d0100a80 2024-04-05T22:07:47.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.223+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1784171544 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d01a2b10 con 0x7f81d0100a80 2024-04-05T22:07:47.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.224+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1784171544 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d004ea90 con 0x7f81d0100a80 2024-04-05T22:07:47.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.225+0000 7f81cdffb700 1 -- 172.21.15.121:0/1784171544 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f81c80046e0 con 0x7f81d0100a80 2024-04-05T22:07:47.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.225+0000 7f81cdffb700 1 --2- 172.21.15.121:0/1784171544 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f81bc06cda0 0x7f81bc06f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:47.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.226+0000 7f81cffff700 1 --2- 172.21.15.121:0/1784171544 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f81bc06cda0 0x7f81bc06f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:47.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.226+0000 7f81cdffb700 1 -- 172.21.15.121:0/1784171544 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f81c808df60 con 0x7f81d0100a80 2024-04-05T22:07:47.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.226+0000 7f81cffff700 1 --2- 172.21.15.121:0/1784171544 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f81bc06cda0 0x7f81bc06f260 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f81b8005950 tx=0x7f81b80058e0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:47.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.230+0000 7f81cdffb700 1 -- 172.21.15.121:0/1784171544 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f81c8057f30 con 0x7f81d0100a80 2024-04-05T22:07:47.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:46 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1863079306' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-05T22:07:47.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:46 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3627248771' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-05T22:07:47.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:46 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1960961156' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-05T22:07:47.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.467+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1784171544 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7f81d01a2f50 con 0x7f81d0100a80 2024-04-05T22:07:47.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.468+0000 7f81cdffb700 1 -- 172.21.15.121:0/1784171544 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v41) v1 ==== 107+0+4855 (secure 0 0 0) 0x7f81c805b550 con 0x7f81d0100a80 2024-04-05T22:07:47.469 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:47.469 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44297,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2880188624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2880188624},{"type":"v1","addr":"172.21.15.188:6825","nonce":2880188624}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44827,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/3202979994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3202979994},{"type":"v1","addr":"172.21.15.188:6827","nonce":3202979994}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T22:05:36.202119+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34360,"mds_1":34364},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34360":{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":0,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34364":{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":1,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:47.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.470+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1784171544 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f81bc06cda0 msgr2=0x7f81bc06f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:47.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.470+0000 7f81d6a6e700 1 --2- 172.21.15.121:0/1784171544 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f81bc06cda0 0x7f81bc06f260 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f81b8005950 tx=0x7f81b80058e0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.470+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1784171544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81d0100a80 msgr2=0x7f81d0073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:47.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.470+0000 7f81d6a6e700 1 --2- 172.21.15.121:0/1784171544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81d0100a80 0x7f81d0073090 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f81c800d010 tx=0x7f81c8007be0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.470+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1784171544 shutdown_connections 2024-04-05T22:07:47.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.470+0000 7f81d6a6e700 1 --2- 172.21.15.121:0/1784171544 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f81bc06cda0 0x7f81bc06f260 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.470+0000 7f81d6a6e700 1 --2- 172.21.15.121:0/1784171544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81d0100a80 0x7f81d0073090 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.470+0000 7f81d6a6e700 1 --2- 172.21.15.121:0/1784171544 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f81d0106a90 0x7f81d00735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.470+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1784171544 >> 172.21.15.121:0/1784171544 conn(0x7f81d00fc570 msgr2=0x7f81d00fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:47.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.470+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1784171544 shutdown_connections 2024-04-05T22:07:47.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.470+0000 7f81d6a6e700 1 -- 172.21.15.121:0/1784171544 wait complete. 2024-04-05T22:07:47.472 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 34 2024-04-05T22:07:47.483 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 34 2024-04-05T22:07:47.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-04-05T22:07:47.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.645+0000 7f2d8bd34700 1 -- 172.21.15.121:0/312218265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d840ffac0 msgr2=0x7f2d840fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:47.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.645+0000 7f2d8bd34700 1 --2- 172.21.15.121:0/312218265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d840ffac0 0x7f2d840fff20 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f2d80009910 tx=0x7f2d80009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:47.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.645+0000 7f2d8bd34700 1 -- 172.21.15.121:0/312218265 shutdown_connections 2024-04-05T22:07:47.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.645+0000 7f2d8bd34700 1 --2- 172.21.15.121:0/312218265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d840ffac0 0x7f2d840fff20 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.645+0000 7f2d8bd34700 1 --2- 172.21.15.121:0/312218265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d84105990 0x7f2d84105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.645+0000 7f2d8bd34700 1 -- 172.21.15.121:0/312218265 >> 172.21.15.121:0/312218265 conn(0x7f2d84075400 msgr2=0x7f2d84075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.645+0000 7f2d8bd34700 1 -- 172.21.15.121:0/312218265 shutdown_connections 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.645+0000 7f2d8bd34700 1 -- 172.21.15.121:0/312218265 wait complete. 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d8bd34700 1 Processor -- start 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d8bd34700 1 -- start start 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d8bd34700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d840ffac0 0x7f2d84197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d8bd34700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d84105990 0x7f2d84197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d8bd34700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d84197e60 con 0x7f2d840ffac0 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d8bd34700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d84197fd0 con 0x7f2d84105990 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d89ad0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d840ffac0 0x7f2d84197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d89ad0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d840ffac0 0x7f2d84197300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36972/0 (socket says 172.21.15.121:36972) 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d89ad0700 1 -- 172.21.15.121:0/4075129566 learned_addr learned my addr 172.21.15.121:0/4075129566 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d892cf700 1 --2- 172.21.15.121:0/4075129566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d84105990 0x7f2d84197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d89ad0700 1 -- 172.21.15.121:0/4075129566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d84105990 msgr2=0x7f2d84197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d89ad0700 1 --2- 172.21.15.121:0/4075129566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d84105990 0x7f2d84197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d89ad0700 1 -- 172.21.15.121:0/4075129566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d800095f0 con 0x7f2d840ffac0 2024-04-05T22:07:47.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d892cf700 1 --2- 172.21.15.121:0/4075129566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d84105990 0x7f2d84197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:07:47.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d89ad0700 1 --2- 172.21.15.121:0/4075129566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d840ffac0 0x7f2d84197300 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f2d7400d8d0 tx=0x7f2d7400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:47.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d7affd700 1 -- 172.21.15.121:0/4075129566 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d74009940 con 0x7f2d840ffac0 2024-04-05T22:07:47.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d7affd700 1 -- 172.21.15.121:0/4075129566 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d74010460 con 0x7f2d840ffac0 2024-04-05T22:07:47.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d8bd34700 1 -- 172.21.15.121:0/4075129566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d841a0370 con 0x7f2d840ffac0 2024-04-05T22:07:47.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.646+0000 7f2d7affd700 1 -- 172.21.15.121:0/4075129566 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d7400f5f0 con 0x7f2d840ffac0 2024-04-05T22:07:47.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.647+0000 7f2d8bd34700 1 -- 172.21.15.121:0/4075129566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d841a0870 con 0x7f2d840ffac0 2024-04-05T22:07:47.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.647+0000 7f2d8bd34700 1 -- 172.21.15.121:0/4075129566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d84107f10 con 0x7f2d840ffac0 2024-04-05T22:07:47.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.648+0000 7f2d7affd700 1 -- 172.21.15.121:0/4075129566 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f2d740105d0 con 0x7f2d840ffac0 2024-04-05T22:07:47.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.648+0000 7f2d7affd700 1 --2- 172.21.15.121:0/4075129566 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2d7006cda0 0x7f2d7006f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:47.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.648+0000 7f2d7affd700 1 -- 172.21.15.121:0/4075129566 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f2d7408c4f0 con 0x7f2d840ffac0 2024-04-05T22:07:47.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.648+0000 7f2d892cf700 1 --2- 172.21.15.121:0/4075129566 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2d7006cda0 0x7f2d7006f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:47.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.649+0000 7f2d892cf700 1 --2- 172.21.15.121:0/4075129566 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2d7006cda0 0x7f2d7006f260 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2d84068670 tx=0x7f2d8001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:47.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.653+0000 7f2d7affd700 1 -- 172.21.15.121:0/4075129566 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f2d740565e0 con 0x7f2d840ffac0 2024-04-05T22:07:47.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.893+0000 7f2d8bd34700 1 -- 172.21.15.121:0/4075129566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f2d8404ea90 con 0x7f2d840ffac0 2024-04-05T22:07:47.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.893+0000 7f2d7affd700 1 -- 172.21.15.121:0/4075129566 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v41) v1 ==== 107+0+4854 (secure 0 0 0) 0x7f2d74010880 con 0x7f2d840ffac0 2024-04-05T22:07:47.894 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:47.894 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44297,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2880188624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2880188624},{"type":"v1","addr":"172.21.15.188:6825","nonce":2880188624}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44827,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/3202979994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3202979994},{"type":"v1","addr":"172.21.15.188:6827","nonce":3202979994}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T22:05:37.204038+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34360,"mds_1":34364},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34360":{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":0,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34364":{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":1,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.895+0000 7f2d8bd34700 1 -- 172.21.15.121:0/4075129566 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2d7006cda0 msgr2=0x7f2d7006f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.895+0000 7f2d8bd34700 1 --2- 172.21.15.121:0/4075129566 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2d7006cda0 0x7f2d7006f260 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2d84068670 tx=0x7f2d8001a040 comp rx=0 tx=0).stop 2024-04-05T22:07:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.895+0000 7f2d8bd34700 1 -- 172.21.15.121:0/4075129566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d840ffac0 msgr2=0x7f2d84197300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.895+0000 7f2d8bd34700 1 --2- 172.21.15.121:0/4075129566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d840ffac0 0x7f2d84197300 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f2d7400d8d0 tx=0x7f2d7400dc90 comp rx=0 tx=0).stop 2024-04-05T22:07:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.895+0000 7f2d8bd34700 1 -- 172.21.15.121:0/4075129566 shutdown_connections 2024-04-05T22:07:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.895+0000 7f2d8bd34700 1 --2- 172.21.15.121:0/4075129566 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f2d7006cda0 0x7f2d7006f260 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.895+0000 7f2d8bd34700 1 --2- 172.21.15.121:0/4075129566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d840ffac0 0x7f2d84197300 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.895+0000 7f2d8bd34700 1 --2- 172.21.15.121:0/4075129566 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f2d84105990 0x7f2d84197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.895+0000 7f2d8bd34700 1 -- 172.21.15.121:0/4075129566 >> 172.21.15.121:0/4075129566 conn(0x7f2d84075400 msgr2=0x7f2d840fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.896+0000 7f2d8bd34700 1 -- 172.21.15.121:0/4075129566 shutdown_connections 2024-04-05T22:07:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:47.896+0000 7f2d8bd34700 1 -- 172.21.15.121:0/4075129566 wait complete. 2024-04-05T22:07:47.898 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 35 2024-04-05T22:07:47.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-04-05T22:07:48.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:47 smithi188 ceph-mon[107793]: pgmap v1249: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:48.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:47 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3045400064' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-05T22:07:48.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:47 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1784171544' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-05T22:07:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.030+0000 7f5ad086c700 1 -- 172.21.15.121:0/4269999170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5acc100540 msgr2=0x7f5acc100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.030+0000 7f5ad086c700 1 --2- 172.21.15.121:0/4269999170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5acc100540 0x7f5acc100920 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009910 tx=0x7f5ac0009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.030+0000 7f5ad086c700 1 -- 172.21.15.121:0/4269999170 shutdown_connections 2024-04-05T22:07:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.030+0000 7f5ad086c700 1 --2- 172.21.15.121:0/4269999170 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5acc100ef0 0x7f5acc1045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.030+0000 7f5ad086c700 1 --2- 172.21.15.121:0/4269999170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5acc100540 0x7f5acc100920 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.030+0000 7f5ad086c700 1 -- 172.21.15.121:0/4269999170 >> 172.21.15.121:0/4269999170 conn(0x7f5acc075d00 msgr2=0x7f5acc076110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.030+0000 7f5ad086c700 1 -- 172.21.15.121:0/4269999170 shutdown_connections 2024-04-05T22:07:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.030+0000 7f5ad086c700 1 -- 172.21.15.121:0/4269999170 wait complete. 2024-04-05T22:07:48.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.031+0000 7f5ad086c700 1 Processor -- start 2024-04-05T22:07:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.031+0000 7f5ad086c700 1 -- start start 2024-04-05T22:07:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.031+0000 7f5ad086c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5acc100540 0x7f5acc195440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.031+0000 7f5ad086c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5acc195980 0x7f5acc199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.031+0000 7f5ad086c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc195fa0 con 0x7f5acc100540 2024-04-05T22:07:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.031+0000 7f5ad086c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc196110 con 0x7f5acc195980 2024-04-05T22:07:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.031+0000 7f5aca59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5acc100540 0x7f5acc195440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.031+0000 7f5aca59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5acc100540 0x7f5acc195440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36980/0 (socket says 172.21.15.121:36980) 2024-04-05T22:07:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.031+0000 7f5aca59c700 1 -- 172.21.15.121:0/2490970812 learned_addr learned my addr 172.21.15.121:0/2490970812 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.031+0000 7f5ac9d9b700 1 --2- 172.21.15.121:0/2490970812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5acc195980 0x7f5acc199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.031+0000 7f5aca59c700 1 -- 172.21.15.121:0/2490970812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5acc195980 msgr2=0x7f5acc199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.031+0000 7f5aca59c700 1 --2- 172.21.15.121:0/2490970812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5acc195980 0x7f5acc199de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.031+0000 7f5aca59c700 1 -- 172.21.15.121:0/2490970812 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac00095f0 con 0x7f5acc100540 2024-04-05T22:07:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.032+0000 7f5aca59c700 1 --2- 172.21.15.121:0/2490970812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5acc100540 0x7f5acc195440 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0000c00 tx=0x7f5ac000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.032+0000 7f5abb7fe700 1 -- 172.21.15.121:0/2490970812 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ac001d070 con 0x7f5acc100540 2024-04-05T22:07:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.032+0000 7f5ad086c700 1 -- 172.21.15.121:0/2490970812 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5acc19a3e0 con 0x7f5acc100540 2024-04-05T22:07:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.032+0000 7f5abb7fe700 1 -- 172.21.15.121:0/2490970812 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ac00044c0 con 0x7f5acc100540 2024-04-05T22:07:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.032+0000 7f5abb7fe700 1 -- 172.21.15.121:0/2490970812 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ac00039d0 con 0x7f5acc100540 2024-04-05T22:07:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.032+0000 7f5ad086c700 1 -- 172.21.15.121:0/2490970812 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5acc19a880 con 0x7f5acc100540 2024-04-05T22:07:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.033+0000 7f5ad086c700 1 -- 172.21.15.121:0/2490970812 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5acc107f60 con 0x7f5acc100540 2024-04-05T22:07:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.033+0000 7f5abb7fe700 1 -- 172.21.15.121:0/2490970812 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f5ac00048e0 con 0x7f5acc100540 2024-04-05T22:07:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.033+0000 7f5abb7fe700 1 --2- 172.21.15.121:0/2490970812 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5ab406cda0 0x7f5ab406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.034+0000 7f5abb7fe700 1 -- 172.21.15.121:0/2490970812 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f5ac008dc60 con 0x7f5acc100540 2024-04-05T22:07:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.034+0000 7f5ac9d9b700 1 --2- 172.21.15.121:0/2490970812 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5ab406cda0 0x7f5ab406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.034+0000 7f5ac9d9b700 1 --2- 172.21.15.121:0/2490970812 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5ab406cda0 0x7f5ab406f260 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5acc073000 tx=0x7f5abc009260 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.038+0000 7f5abb7fe700 1 -- 172.21.15.121:0/2490970812 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f5ac0057cd0 con 0x7f5acc100540 2024-04-05T22:07:48.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:47 smithi121 ceph-mon[145859]: pgmap v1249: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:48.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:47 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3045400064' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-05T22:07:48.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:47 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1784171544' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-05T22:07:48.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.273+0000 7f5ad086c700 1 -- 172.21.15.121:0/2490970812 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7f5acc04ea90 con 0x7f5acc100540 2024-04-05T22:07:48.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.274+0000 7f5abb7fe700 1 -- 172.21.15.121:0/2490970812 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v41) v1 ==== 107+0+4857 (secure 0 0 0) 0x7f5ac005b2f0 con 0x7f5acc100540 2024-04-05T22:07:48.274 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:48.275 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44297,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2880188624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2880188624},{"type":"v1","addr":"172.21.15.188:6825","nonce":2880188624}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44827,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/3202979994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3202979994},{"type":"v1","addr":"172.21.15.188:6827","nonce":3202979994}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T22:05:42.047263+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34360,"mds_1":34364},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34360":{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":0,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34364":{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":1,"incarnation":34,"state":"up:resolve","state_seq":460,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:48.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.276+0000 7f5ad086c700 1 -- 172.21.15.121:0/2490970812 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5ab406cda0 msgr2=0x7f5ab406f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:48.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.276+0000 7f5ad086c700 1 --2- 172.21.15.121:0/2490970812 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5ab406cda0 0x7f5ab406f260 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5acc073000 tx=0x7f5abc009260 comp rx=0 tx=0).stop 2024-04-05T22:07:48.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.276+0000 7f5ad086c700 1 -- 172.21.15.121:0/2490970812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5acc100540 msgr2=0x7f5acc195440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:48.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.276+0000 7f5ad086c700 1 --2- 172.21.15.121:0/2490970812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5acc100540 0x7f5acc195440 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0000c00 tx=0x7f5ac000be90 comp rx=0 tx=0).stop 2024-04-05T22:07:48.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.276+0000 7f5ad086c700 1 -- 172.21.15.121:0/2490970812 shutdown_connections 2024-04-05T22:07:48.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.276+0000 7f5ad086c700 1 --2- 172.21.15.121:0/2490970812 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f5ab406cda0 0x7f5ab406f260 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.276+0000 7f5ad086c700 1 --2- 172.21.15.121:0/2490970812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5acc100540 0x7f5acc195440 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.276+0000 7f5ad086c700 1 --2- 172.21.15.121:0/2490970812 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f5acc195980 0x7f5acc199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.276+0000 7f5ad086c700 1 -- 172.21.15.121:0/2490970812 >> 172.21.15.121:0/2490970812 conn(0x7f5acc075d00 msgr2=0x7f5acc0fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:48.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.276+0000 7f5ad086c700 1 -- 172.21.15.121:0/2490970812 shutdown_connections 2024-04-05T22:07:48.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.276+0000 7f5ad086c700 1 -- 172.21.15.121:0/2490970812 wait complete. 2024-04-05T22:07:48.278 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 36 2024-04-05T22:07:48.289 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 37 2024-04-05T22:07:48.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.410+0000 7f4c23904700 1 -- 172.21.15.121:0/2763150207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c1c0ff970 msgr2=0x7f4c1c0ffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:48.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.410+0000 7f4c23904700 1 --2- 172.21.15.121:0/2763150207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c1c0ff970 0x7f4c1c0ffdf0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c009910 tx=0x7f4c0c009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:48.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.410+0000 7f4c23904700 1 -- 172.21.15.121:0/2763150207 shutdown_connections 2024-04-05T22:07:48.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.410+0000 7f4c23904700 1 --2- 172.21.15.121:0/2763150207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c1c0ff970 0x7f4c1c0ffdf0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.410+0000 7f4c23904700 1 --2- 172.21.15.121:0/2763150207 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c1c105980 0x7f4c1c105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.410+0000 7f4c23904700 1 -- 172.21.15.121:0/2763150207 >> 172.21.15.121:0/2763150207 conn(0x7f4c1c0fb480 msgr2=0x7f4c1c0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.411+0000 7f4c23904700 1 -- 172.21.15.121:0/2763150207 shutdown_connections 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.411+0000 7f4c23904700 1 -- 172.21.15.121:0/2763150207 wait complete. 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.411+0000 7f4c23904700 1 Processor -- start 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.411+0000 7f4c23904700 1 -- start start 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.411+0000 7f4c23904700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c1c0ff970 0x7f4c1c197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.411+0000 7f4c23904700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c1c105980 0x7f4c1c1978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.411+0000 7f4c23904700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c197ed0 con 0x7f4c1c0ff970 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.411+0000 7f4c23904700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c198040 con 0x7f4c1c105980 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.411+0000 7f4c216a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c1c0ff970 0x7f4c1c197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.411+0000 7f4c216a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c1c0ff970 0x7f4c1c197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37002/0 (socket says 172.21.15.121:37002) 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.411+0000 7f4c216a0700 1 -- 172.21.15.121:0/4079952268 learned_addr learned my addr 172.21.15.121:0/4079952268 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.412+0000 7f4c20e9f700 1 --2- 172.21.15.121:0/4079952268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c1c105980 0x7f4c1c1978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.412+0000 7f4c216a0700 1 -- 172.21.15.121:0/4079952268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c1c105980 msgr2=0x7f4c1c1978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.412+0000 7f4c216a0700 1 --2- 172.21.15.121:0/4079952268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c1c105980 0x7f4c1c1978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.412+0000 7f4c216a0700 1 -- 172.21.15.121:0/4079952268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c0095f0 con 0x7f4c1c0ff970 2024-04-05T22:07:48.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.412+0000 7f4c216a0700 1 --2- 172.21.15.121:0/4079952268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c1c0ff970 0x7f4c1c197370 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800d8d0 tx=0x7f4c1800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:48.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.412+0000 7f4c127fc700 1 -- 172.21.15.121:0/4079952268 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c18009940 con 0x7f4c1c0ff970 2024-04-05T22:07:48.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.412+0000 7f4c23904700 1 -- 172.21.15.121:0/4079952268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1c1a03e0 con 0x7f4c1c0ff970 2024-04-05T22:07:48.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.412+0000 7f4c127fc700 1 -- 172.21.15.121:0/4079952268 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c18010460 con 0x7f4c1c0ff970 2024-04-05T22:07:48.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.412+0000 7f4c127fc700 1 -- 172.21.15.121:0/4079952268 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c1800f5f0 con 0x7f4c1c0ff970 2024-04-05T22:07:48.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.412+0000 7f4c23904700 1 -- 172.21.15.121:0/4079952268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1c1a08e0 con 0x7f4c1c0ff970 2024-04-05T22:07:48.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.413+0000 7f4c23904700 1 -- 172.21.15.121:0/4079952268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1c107f00 con 0x7f4c1c0ff970 2024-04-05T22:07:48.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.413+0000 7f4c127fc700 1 -- 172.21.15.121:0/4079952268 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f4c180105d0 con 0x7f4c1c0ff970 2024-04-05T22:07:48.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.414+0000 7f4c127fc700 1 --2- 172.21.15.121:0/4079952268 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4c0806cda0 0x7f4c0806f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:48.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.414+0000 7f4c127fc700 1 -- 172.21.15.121:0/4079952268 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f4c1808c5e0 con 0x7f4c1c0ff970 2024-04-05T22:07:48.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.414+0000 7f4c20e9f700 1 --2- 172.21.15.121:0/4079952268 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4c0806cda0 0x7f4c0806f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:48.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.414+0000 7f4c20e9f700 1 --2- 172.21.15.121:0/4079952268 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4c0806cda0 0x7f4c0806f260 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c006010 tx=0x7f4c0c005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:48.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.418+0000 7f4c127fc700 1 -- 172.21.15.121:0/4079952268 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f4c18056650 con 0x7f4c1c0ff970 2024-04-05T22:07:48.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.656+0000 7f4c23904700 1 -- 172.21.15.121:0/4079952268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 37, "format": "json"} v 0) v1 -- 0x7f4c1c04ea90 con 0x7f4c1c0ff970 2024-04-05T22:07:48.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.656+0000 7f4c127fc700 1 -- 172.21.15.121:0/4079952268 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 37, "format": "json"}]=0 dumped fsmap epoch 37 v41) v1 ==== 107+0+4860 (secure 0 0 0) 0x7f4c18010880 con 0x7f4c1c0ff970 2024-04-05T22:07:48.657 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:48.657 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44297,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2880188624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2880188624},{"type":"v1","addr":"172.21.15.188:6825","nonce":2880188624}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44827,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/3202979994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3202979994},{"type":"v1","addr":"172.21.15.188:6827","nonce":3202979994}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T22:05:45.059571+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34360,"mds_1":34364},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34360":{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":0,"incarnation":34,"state":"up:resolve","state_seq":466,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34364":{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":1,"incarnation":34,"state":"up:resolve","state_seq":460,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:48.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.658+0000 7f4c23904700 1 -- 172.21.15.121:0/4079952268 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4c0806cda0 msgr2=0x7f4c0806f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:48.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.658+0000 7f4c23904700 1 --2- 172.21.15.121:0/4079952268 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4c0806cda0 0x7f4c0806f260 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c006010 tx=0x7f4c0c005960 comp rx=0 tx=0).stop 2024-04-05T22:07:48.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.658+0000 7f4c23904700 1 -- 172.21.15.121:0/4079952268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c1c0ff970 msgr2=0x7f4c1c197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:48.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.658+0000 7f4c23904700 1 --2- 172.21.15.121:0/4079952268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c1c0ff970 0x7f4c1c197370 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800d8d0 tx=0x7f4c1800dc90 comp rx=0 tx=0).stop 2024-04-05T22:07:48.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.658+0000 7f4c23904700 1 -- 172.21.15.121:0/4079952268 shutdown_connections 2024-04-05T22:07:48.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.658+0000 7f4c23904700 1 --2- 172.21.15.121:0/4079952268 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f4c0806cda0 0x7f4c0806f260 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.658+0000 7f4c23904700 1 --2- 172.21.15.121:0/4079952268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c1c0ff970 0x7f4c1c197370 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.658+0000 7f4c23904700 1 --2- 172.21.15.121:0/4079952268 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4c1c105980 0x7f4c1c1978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.658+0000 7f4c23904700 1 -- 172.21.15.121:0/4079952268 >> 172.21.15.121:0/4079952268 conn(0x7f4c1c0fb480 msgr2=0x7f4c1c0fbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:48.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.658+0000 7f4c23904700 1 -- 172.21.15.121:0/4079952268 shutdown_connections 2024-04-05T22:07:48.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.658+0000 7f4c23904700 1 -- 172.21.15.121:0/4079952268 wait complete. 2024-04-05T22:07:48.661 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 37 2024-04-05T22:07:48.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 38 2024-04-05T22:07:48.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.793+0000 7f9bfcb25700 1 -- 172.21.15.121:0/345158663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bf8101d00 msgr2=0x7f9bf81020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:48.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.793+0000 7f9bfcb25700 1 --2- 172.21.15.121:0/345158663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bf8101d00 0x7f9bf81020e0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f9bec009910 tx=0x7f9bec009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:48.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.793+0000 7f9bfcb25700 1 -- 172.21.15.121:0/345158663 shutdown_connections 2024-04-05T22:07:48.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.793+0000 7f9bfcb25700 1 --2- 172.21.15.121:0/345158663 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf8102620 0x7f9bf810ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.793+0000 7f9bfcb25700 1 --2- 172.21.15.121:0/345158663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bf8101d00 0x7f9bf81020e0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.793+0000 7f9bfcb25700 1 -- 172.21.15.121:0/345158663 >> 172.21.15.121:0/345158663 conn(0x7f9bf80fb4c0 msgr2=0x7f9bf80fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:48.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.793+0000 7f9bfcb25700 1 -- 172.21.15.121:0/345158663 shutdown_connections 2024-04-05T22:07:48.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.793+0000 7f9bfcb25700 1 -- 172.21.15.121:0/345158663 wait complete. 2024-04-05T22:07:48.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.794+0000 7f9bfcb25700 1 Processor -- start 2024-04-05T22:07:48.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.794+0000 7f9bfcb25700 1 -- start start 2024-04-05T22:07:48.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.794+0000 7f9bfcb25700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bf8101d00 0x7f9bf819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.794+0000 7f9bfcb25700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf8102620 0x7f9bf819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.794+0000 7f9bfcb25700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf8198570 con 0x7f9bf8101d00 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.794+0000 7f9bfcb25700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf81986e0 con 0x7f9bf8102620 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.794+0000 7f9bf659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bf8101d00 0x7f9bf819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.795+0000 7f9bf659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bf8101d00 0x7f9bf819e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37006/0 (socket says 172.21.15.121:37006) 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.795+0000 7f9bf659c700 1 -- 172.21.15.121:0/2643316072 learned_addr learned my addr 172.21.15.121:0/2643316072 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.795+0000 7f9bf5d9b700 1 --2- 172.21.15.121:0/2643316072 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf8102620 0x7f9bf819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.795+0000 7f9bf659c700 1 -- 172.21.15.121:0/2643316072 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf8102620 msgr2=0x7f9bf819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.795+0000 7f9bf659c700 1 --2- 172.21.15.121:0/2643316072 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf8102620 0x7f9bf819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.795+0000 7f9bf659c700 1 -- 172.21.15.121:0/2643316072 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec0095f0 con 0x7f9bf8101d00 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.795+0000 7f9bf5d9b700 1 --2- 172.21.15.121:0/2643316072 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf8102620 0x7f9bf819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.795+0000 7f9bf659c700 1 --2- 172.21.15.121:0/2643316072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bf8101d00 0x7f9bf819e590 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f9bec006010 tx=0x7f9bec00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.795+0000 7f9be77fe700 1 -- 172.21.15.121:0/2643316072 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bec01d070 con 0x7f9bf8101d00 2024-04-05T22:07:48.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.795+0000 7f9bfcb25700 1 -- 172.21.15.121:0/2643316072 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf8198960 con 0x7f9bf8101d00 2024-04-05T22:07:48.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.795+0000 7f9be77fe700 1 -- 172.21.15.121:0/2643316072 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bec00f460 con 0x7f9bf8101d00 2024-04-05T22:07:48.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.795+0000 7f9be77fe700 1 -- 172.21.15.121:0/2643316072 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bec021640 con 0x7f9bf8101d00 2024-04-05T22:07:48.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.795+0000 7f9bfcb25700 1 -- 172.21.15.121:0/2643316072 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf8198e60 con 0x7f9bf8101d00 2024-04-05T22:07:48.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.796+0000 7f9bfcb25700 1 -- 172.21.15.121:0/2643316072 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf810a660 con 0x7f9bf8101d00 2024-04-05T22:07:48.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.796+0000 7f9be77fe700 1 -- 172.21.15.121:0/2643316072 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f9bec00f5d0 con 0x7f9bf8101d00 2024-04-05T22:07:48.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.797+0000 7f9be77fe700 1 --2- 172.21.15.121:0/2643316072 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9be006cda0 0x7f9be006f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:48.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.797+0000 7f9be77fe700 1 -- 172.21.15.121:0/2643316072 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f9bec08d810 con 0x7f9bf8101d00 2024-04-05T22:07:48.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.797+0000 7f9bf5d9b700 1 --2- 172.21.15.121:0/2643316072 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9be006cda0 0x7f9be006f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:48.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.797+0000 7f9bf5d9b700 1 --2- 172.21.15.121:0/2643316072 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9be006cda0 0x7f9be006f260 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9be8005fd0 tx=0x7f9be8005ee0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:48.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:48.802+0000 7f9be77fe700 1 -- 172.21.15.121:0/2643316072 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9bec057880 con 0x7f9bf8101d00 2024-04-05T22:07:48.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:48 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4075129566' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-05T22:07:48.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:48 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2490970812' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-05T22:07:48.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:48 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/4079952268' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-05T22:07:49.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.040+0000 7f9bfcb25700 1 -- 172.21.15.121:0/2643316072 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 38, "format": "json"} v 0) v1 -- 0x7f9bf8199bc0 con 0x7f9bf8101d00 2024-04-05T22:07:49.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.040+0000 7f9be77fe700 1 -- 172.21.15.121:0/2643316072 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 38, "format": "json"}]=0 dumped fsmap epoch 38 v41) v1 ==== 107+0+4864 (secure 0 0 0) 0x7f9bec005740 con 0x7f9bf8101d00 2024-04-05T22:07:49.041 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:49.041 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44297,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2880188624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2880188624},{"type":"v1","addr":"172.21.15.188:6825","nonce":2880188624}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44827,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/3202979994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3202979994},{"type":"v1","addr":"172.21.15.188:6827","nonce":3202979994}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T22:05:46.066773+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34360,"mds_1":34364},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34360":{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":0,"incarnation":34,"state":"up:reconnect","state_seq":467,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34364":{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":1,"incarnation":34,"state":"up:reconnect","state_seq":461,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:49.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.042+0000 7f9bfcb25700 1 -- 172.21.15.121:0/2643316072 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9be006cda0 msgr2=0x7f9be006f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:49.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.043+0000 7f9bfcb25700 1 --2- 172.21.15.121:0/2643316072 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9be006cda0 0x7f9be006f260 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9be8005fd0 tx=0x7f9be8005ee0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.043+0000 7f9bfcb25700 1 -- 172.21.15.121:0/2643316072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bf8101d00 msgr2=0x7f9bf819e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:49.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.043+0000 7f9bfcb25700 1 --2- 172.21.15.121:0/2643316072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bf8101d00 0x7f9bf819e590 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f9bec006010 tx=0x7f9bec00be20 comp rx=0 tx=0).stop 2024-04-05T22:07:49.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.043+0000 7f9bfcb25700 1 -- 172.21.15.121:0/2643316072 shutdown_connections 2024-04-05T22:07:49.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.043+0000 7f9bfcb25700 1 --2- 172.21.15.121:0/2643316072 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f9be006cda0 0x7f9be006f260 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.043+0000 7f9bfcb25700 1 --2- 172.21.15.121:0/2643316072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9bf8101d00 0x7f9bf819e590 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.043+0000 7f9bfcb25700 1 --2- 172.21.15.121:0/2643316072 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9bf8102620 0x7f9bf819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.043+0000 7f9bfcb25700 1 -- 172.21.15.121:0/2643316072 >> 172.21.15.121:0/2643316072 conn(0x7f9bf80fb4c0 msgr2=0x7f9bf81000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:49.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.043+0000 7f9bfcb25700 1 -- 172.21.15.121:0/2643316072 shutdown_connections 2024-04-05T22:07:49.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.043+0000 7f9bfcb25700 1 -- 172.21.15.121:0/2643316072 wait complete. 2024-04-05T22:07:49.045 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 38 2024-04-05T22:07:49.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 39 2024-04-05T22:07:49.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.177+0000 7fe61abb9700 1 -- 172.21.15.121:0/1441023741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614105990 msgr2=0x7fe614105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:49.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.177+0000 7fe61abb9700 1 --2- 172.21.15.121:0/1441023741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614105990 0x7fe614105d70 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fe608009910 tx=0x7fe608009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:49.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.177+0000 7fe61abb9700 1 -- 172.21.15.121:0/1441023741 shutdown_connections 2024-04-05T22:07:49.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.177+0000 7fe61abb9700 1 --2- 172.21.15.121:0/1441023741 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6140ffac0 0x7fe6140fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.177+0000 7fe61abb9700 1 --2- 172.21.15.121:0/1441023741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614105990 0x7fe614105d70 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.177+0000 7fe61abb9700 1 -- 172.21.15.121:0/1441023741 >> 172.21.15.121:0/1441023741 conn(0x7fe614075400 msgr2=0x7fe614075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:49.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.177+0000 7fe61abb9700 1 -- 172.21.15.121:0/1441023741 shutdown_connections 2024-04-05T22:07:49.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.177+0000 7fe61abb9700 1 -- 172.21.15.121:0/1441023741 wait complete. 2024-04-05T22:07:49.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.177+0000 7fe61abb9700 1 Processor -- start 2024-04-05T22:07:49.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.177+0000 7fe61abb9700 1 -- start start 2024-04-05T22:07:49.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe61abb9700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6140ffac0 0x7fe614195230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:49.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe61abb9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614105990 0x7fe614195770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:49.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe61abb9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe614195cb0 con 0x7fe614105990 2024-04-05T22:07:49.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe61abb9700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe614195e20 con 0x7fe6140ffac0 2024-04-05T22:07:49.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe613fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614105990 0x7fe614195770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:49.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe613fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614105990 0x7fe614195770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37014/0 (socket says 172.21.15.121:37014) 2024-04-05T22:07:49.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe613fff700 1 -- 172.21.15.121:0/3286772270 learned_addr learned my addr 172.21.15.121:0/3286772270 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:49.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe618955700 1 --2- 172.21.15.121:0/3286772270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6140ffac0 0x7fe614195230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:49.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe613fff700 1 -- 172.21.15.121:0/3286772270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6140ffac0 msgr2=0x7fe614195230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:49.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe613fff700 1 --2- 172.21.15.121:0/3286772270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6140ffac0 0x7fe614195230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe613fff700 1 -- 172.21.15.121:0/3286772270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6080095f0 con 0x7fe614105990 2024-04-05T22:07:49.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe613fff700 1 --2- 172.21.15.121:0/3286772270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614105990 0x7fe614195770 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fe60000b4b0 tx=0x7fe60000b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:49.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe611ffb700 1 -- 172.21.15.121:0/3286772270 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe600011800 con 0x7fe614105990 2024-04-05T22:07:49.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe611ffb700 1 -- 172.21.15.121:0/3286772270 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe600011e00 con 0x7fe614105990 2024-04-05T22:07:49.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe61abb9700 1 -- 172.21.15.121:0/3286772270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6141a28a0 con 0x7fe614105990 2024-04-05T22:07:49.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.178+0000 7fe611ffb700 1 -- 172.21.15.121:0/3286772270 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe600010e50 con 0x7fe614105990 2024-04-05T22:07:49.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.179+0000 7fe61abb9700 1 -- 172.21.15.121:0/3286772270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6141a2c10 con 0x7fe614105990 2024-04-05T22:07:49.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.179+0000 7fe61abb9700 1 -- 172.21.15.121:0/3286772270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe614107f30 con 0x7fe614105990 2024-04-05T22:07:49.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.180+0000 7fe611ffb700 1 -- 172.21.15.121:0/3286772270 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fe600010380 con 0x7fe614105990 2024-04-05T22:07:49.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.180+0000 7fe611ffb700 1 --2- 172.21.15.121:0/3286772270 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe60406cda0 0x7fe60406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:49.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.181+0000 7fe618955700 1 --2- 172.21.15.121:0/3286772270 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe60406cda0 0x7fe60406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:49.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.181+0000 7fe611ffb700 1 -- 172.21.15.121:0/3286772270 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fe60008c110 con 0x7fe614105990 2024-04-05T22:07:49.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.181+0000 7fe618955700 1 --2- 172.21.15.121:0/3286772270 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe60406cda0 0x7fe60406f260 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe608000c00 tx=0x7fe608005fb0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:49.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.185+0000 7fe611ffb700 1 -- 172.21.15.121:0/3286772270 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fe600059bf0 con 0x7fe614105990 2024-04-05T22:07:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:48 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4075129566' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-04-05T22:07:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:48 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2490970812' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-04-05T22:07:49.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:48 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/4079952268' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-04-05T22:07:49.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.424+0000 7fe61abb9700 1 -- 172.21.15.121:0/3286772270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 39, "format": "json"} v 0) v1 -- 0x7fe614195fe0 con 0x7fe614105990 2024-04-05T22:07:49.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.424+0000 7fe611ffb700 1 -- 172.21.15.121:0/3286772270 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 39, "format": "json"}]=0 dumped fsmap epoch 39 v41) v1 ==== 107+0+4858 (secure 0 0 0) 0x7fe600059780 con 0x7fe614105990 2024-04-05T22:07:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:49.426 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44297,"name":"cephfs.smithi188.bjmisu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6825/2880188624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2880188624},{"type":"v1","addr":"172.21.15.188:6825","nonce":2880188624}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":44827,"name":"cephfs.smithi188.latnft","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.188:6827/3202979994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3202979994},{"type":"v1","addr":"172.21.15.188:6827","nonce":3202979994}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T22:05:47.068504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34360,"mds_1":34364},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34360":{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":0,"incarnation":34,"state":"up:rejoin","state_seq":468,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34364":{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":1,"incarnation":34,"state":"up:rejoin","state_seq":462,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:49.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.426+0000 7fe61abb9700 1 -- 172.21.15.121:0/3286772270 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe60406cda0 msgr2=0x7fe60406f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:49.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.426+0000 7fe61abb9700 1 --2- 172.21.15.121:0/3286772270 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe60406cda0 0x7fe60406f260 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe608000c00 tx=0x7fe608005fb0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.426+0000 7fe61abb9700 1 -- 172.21.15.121:0/3286772270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614105990 msgr2=0x7fe614195770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:49.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.426+0000 7fe61abb9700 1 --2- 172.21.15.121:0/3286772270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614105990 0x7fe614195770 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fe60000b4b0 tx=0x7fe60000b870 comp rx=0 tx=0).stop 2024-04-05T22:07:49.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.427+0000 7fe61abb9700 1 -- 172.21.15.121:0/3286772270 shutdown_connections 2024-04-05T22:07:49.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.427+0000 7fe61abb9700 1 --2- 172.21.15.121:0/3286772270 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fe60406cda0 0x7fe60406f260 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.427+0000 7fe61abb9700 1 --2- 172.21.15.121:0/3286772270 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fe6140ffac0 0x7fe614195230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.427+0000 7fe61abb9700 1 --2- 172.21.15.121:0/3286772270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614105990 0x7fe614195770 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.427+0000 7fe61abb9700 1 -- 172.21.15.121:0/3286772270 >> 172.21.15.121:0/3286772270 conn(0x7fe614075400 msgr2=0x7fe6140fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:49.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.427+0000 7fe61abb9700 1 -- 172.21.15.121:0/3286772270 shutdown_connections 2024-04-05T22:07:49.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.427+0000 7fe61abb9700 1 -- 172.21.15.121:0/3286772270 wait complete. 2024-04-05T22:07:49.429 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 39 2024-04-05T22:07:49.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 40 2024-04-05T22:07:49.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.563+0000 7fddef3aa700 1 -- 172.21.15.121:0/834293774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdde80ffdd0 msgr2=0x7fdde8105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:49.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.563+0000 7fddef3aa700 1 --2- 172.21.15.121:0/834293774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdde80ffdd0 0x7fdde8105750 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009910 tx=0x7fdde4009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:49.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.563+0000 7fddef3aa700 1 -- 172.21.15.121:0/834293774 shutdown_connections 2024-04-05T22:07:49.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.563+0000 7fddef3aa700 1 --2- 172.21.15.121:0/834293774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdde80ffdd0 0x7fdde8105750 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.563+0000 7fddef3aa700 1 --2- 172.21.15.121:0/834293774 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdde80ff420 0x7fdde80ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.563+0000 7fddef3aa700 1 -- 172.21.15.121:0/834293774 >> 172.21.15.121:0/834293774 conn(0x7fdde8074c20 msgr2=0x7fdde8075030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:49.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.563+0000 7fddef3aa700 1 -- 172.21.15.121:0/834293774 shutdown_connections 2024-04-05T22:07:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.563+0000 7fddef3aa700 1 -- 172.21.15.121:0/834293774 wait complete. 2024-04-05T22:07:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.563+0000 7fddef3aa700 1 Processor -- start 2024-04-05T22:07:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddef3aa700 1 -- start start 2024-04-05T22:07:49.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddef3aa700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdde80ff420 0x7fdde8196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:49.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddef3aa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdde8196a50 0x7fdde819aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:49.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddef3aa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde8196fe0 con 0x7fdde8196a50 2024-04-05T22:07:49.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddef3aa700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde8197150 con 0x7fdde80ff420 2024-04-05T22:07:49.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddec945700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdde8196a50 0x7fdde819aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:49.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddec945700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdde8196a50 0x7fdde819aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37028/0 (socket says 172.21.15.121:37028) 2024-04-05T22:07:49.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddec945700 1 -- 172.21.15.121:0/2475994786 learned_addr learned my addr 172.21.15.121:0/2475994786 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:49.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fdded146700 1 --2- 172.21.15.121:0/2475994786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdde80ff420 0x7fdde8196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:49.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddec945700 1 -- 172.21.15.121:0/2475994786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdde80ff420 msgr2=0x7fdde8196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:49.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddec945700 1 --2- 172.21.15.121:0/2475994786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdde80ff420 0x7fdde8196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddec945700 1 -- 172.21.15.121:0/2475994786 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde40095f0 con 0x7fdde8196a50 2024-04-05T22:07:49.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fdded146700 1 --2- 172.21.15.121:0/2475994786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdde80ff420 0x7fdde8196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-05T22:07:49.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddec945700 1 --2- 172.21.15.121:0/2475994786 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdde8196a50 0x7fdde819aeb0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fdde400bd70 tx=0x7fdde400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:49.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddde7fc700 1 -- 172.21.15.121:0/2475994786 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdde401d070 con 0x7fdde8196a50 2024-04-05T22:07:49.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddde7fc700 1 -- 172.21.15.121:0/2475994786 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdde400f460 con 0x7fdde8196a50 2024-04-05T22:07:49.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddde7fc700 1 -- 172.21.15.121:0/2475994786 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdde4021640 con 0x7fdde8196a50 2024-04-05T22:07:49.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.564+0000 7fddef3aa700 1 -- 172.21.15.121:0/2475994786 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde819b450 con 0x7fdde8196a50 2024-04-05T22:07:49.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.565+0000 7fddef3aa700 1 -- 172.21.15.121:0/2475994786 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde819b980 con 0x7fdde8196a50 2024-04-05T22:07:49.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.565+0000 7fddef3aa700 1 -- 172.21.15.121:0/2475994786 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde81090a0 con 0x7fdde8196a50 2024-04-05T22:07:49.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.566+0000 7fddde7fc700 1 -- 172.21.15.121:0/2475994786 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fdde400f9f0 con 0x7fdde8196a50 2024-04-05T22:07:49.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.566+0000 7fddde7fc700 1 --2- 172.21.15.121:0/2475994786 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fddd406cda0 0x7fddd406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:49.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.566+0000 7fddde7fc700 1 -- 172.21.15.121:0/2475994786 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fdde408d7d0 con 0x7fdde8196a50 2024-04-05T22:07:49.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.566+0000 7fdded146700 1 --2- 172.21.15.121:0/2475994786 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fddd406cda0 0x7fddd406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:49.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.567+0000 7fdded146700 1 --2- 172.21.15.121:0/2475994786 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fddd406cda0 0x7fddd406f260 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fddd8005fd0 tx=0x7fddd8005e30 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:49.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.571+0000 7fddde7fc700 1 -- 172.21.15.121:0/2475994786 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fdde4057820 con 0x7fdde8196a50 2024-04-05T22:07:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.807+0000 7fddef3aa700 1 -- 172.21.15.121:0/2475994786 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 40, "format": "json"} v 0) v1 -- 0x7fdde8197860 con 0x7fdde8196a50 2024-04-05T22:07:49.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.808+0000 7fddde7fc700 1 -- 172.21.15.121:0/2475994786 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 40, "format": "json"}]=0 dumped fsmap epoch 40 v41) v1 ==== 107+0+4873 (secure 0 0 0) 0x7fdde405ae40 con 0x7fdde8196a50 2024-04-05T22:07:49.809 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:49.809 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-05T21:23:43.437872+0000","modified":"2024-04-05T22:05:47.287336+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34360,"mds_1":34364},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34360":{"gid":34360,"name":"cephfs.smithi121.gxheqt","rank":0,"incarnation":34,"state":"up:active","state_seq":469,"addr":"172.21.15.121:6827/3939200727","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3939200727},{"type":"v1","addr":"172.21.15.121:6827","nonce":3939200727}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34364":{"gid":34364,"name":"cephfs.smithi121.ifoxmw","rank":1,"incarnation":34,"state":"up:active","state_seq":463,"addr":"172.21.15.121:6829/3468502927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3468502927},{"type":"v1","addr":"172.21.15.121:6829","nonce":3468502927}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_44297":{"gid":44297,"name":"cephfs.smithi188.bjmisu","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6825/2880188624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":2880188624},{"type":"v1","addr":"172.21.15.188:6825","nonce":2880188624}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_44827":{"gid":44827,"name":"cephfs.smithi188.latnft","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.188:6827/3202979994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6826","nonce":3202979994},{"type":"v1","addr":"172.21.15.188:6827","nonce":3202979994}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-05T22:07:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.811+0000 7fddef3aa700 1 -- 172.21.15.121:0/2475994786 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fddd406cda0 msgr2=0x7fddd406f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.811+0000 7fddef3aa700 1 --2- 172.21.15.121:0/2475994786 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fddd406cda0 0x7fddd406f260 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fddd8005fd0 tx=0x7fddd8005e30 comp rx=0 tx=0).stop 2024-04-05T22:07:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.811+0000 7fddef3aa700 1 -- 172.21.15.121:0/2475994786 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdde8196a50 msgr2=0x7fdde819aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.811+0000 7fddef3aa700 1 --2- 172.21.15.121:0/2475994786 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdde8196a50 0x7fdde819aeb0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fdde400bd70 tx=0x7fdde400be50 comp rx=0 tx=0).stop 2024-04-05T22:07:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.811+0000 7fddef3aa700 1 -- 172.21.15.121:0/2475994786 shutdown_connections 2024-04-05T22:07:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.811+0000 7fddef3aa700 1 --2- 172.21.15.121:0/2475994786 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fddd406cda0 0x7fddd406f260 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.811+0000 7fddef3aa700 1 --2- 172.21.15.121:0/2475994786 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fdde80ff420 0x7fdde8196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.811+0000 7fddef3aa700 1 --2- 172.21.15.121:0/2475994786 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdde8196a50 0x7fdde819aeb0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.811+0000 7fddef3aa700 1 -- 172.21.15.121:0/2475994786 >> 172.21.15.121:0/2475994786 conn(0x7fdde8074c20 msgr2=0x7fdde80fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:49.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.811+0000 7fddef3aa700 1 -- 172.21.15.121:0/2475994786 shutdown_connections 2024-04-05T22:07:49.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:49.811+0000 7fddef3aa700 1 -- 172.21.15.121:0/2475994786 wait complete. 2024-04-05T22:07:49.812 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 40 2024-04-05T22:07:49.823 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-05T22:07:49.836 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-05T22:07:49.836 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T22:07:49.837 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-05T22:07:49.856 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T22:07:49.856 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-05T22:07:49.916 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T22:07:50.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.076+0000 7fb6a4314700 1 -- 172.21.15.121:0/3731107946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb69c105990 msgr2=0x7fb69c105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.076+0000 7fb6a4314700 1 --2- 172.21.15.121:0/3731107946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb69c105990 0x7fb69c105d70 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009910 tx=0x7fb68c009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.076+0000 7fb6a4314700 1 -- 172.21.15.121:0/3731107946 shutdown_connections 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.076+0000 7fb6a4314700 1 --2- 172.21.15.121:0/3731107946 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb69c0ffac0 0x7fb69c0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.076+0000 7fb6a4314700 1 --2- 172.21.15.121:0/3731107946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb69c105990 0x7fb69c105d70 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.076+0000 7fb6a4314700 1 -- 172.21.15.121:0/3731107946 >> 172.21.15.121:0/3731107946 conn(0x7fb69c075400 msgr2=0x7fb69c075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.076+0000 7fb6a4314700 1 -- 172.21.15.121:0/3731107946 shutdown_connections 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.076+0000 7fb6a4314700 1 -- 172.21.15.121:0/3731107946 wait complete. 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.076+0000 7fb6a4314700 1 Processor -- start 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.076+0000 7fb6a4314700 1 -- start start 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a4314700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb69c0ffac0 0x7fb69c101550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a4314700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb69c105990 0x7fb69c101a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a4314700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c101fd0 con 0x7fb69c105990 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a4314700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c102110 con 0x7fb69c0ffac0 2024-04-05T22:07:50.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a18af700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb69c105990 0x7fb69c101a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:50.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a18af700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb69c105990 0x7fb69c101a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37050/0 (socket says 172.21.15.121:37050) 2024-04-05T22:07:50.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a18af700 1 -- 172.21.15.121:0/842976200 learned_addr learned my addr 172.21.15.121:0/842976200 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:50.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a18af700 1 -- 172.21.15.121:0/842976200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb69c0ffac0 msgr2=0x7fb69c101550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:50.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a20b0700 1 --2- 172.21.15.121:0/842976200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb69c0ffac0 0x7fb69c101550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:50.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a18af700 1 --2- 172.21.15.121:0/842976200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb69c0ffac0 0x7fb69c101550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a18af700 1 -- 172.21.15.121:0/842976200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c0095f0 con 0x7fb69c105990 2024-04-05T22:07:50.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a18af700 1 --2- 172.21.15.121:0/842976200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb69c105990 0x7fb69c101a90 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fb69800d8d0 tx=0x7fb69800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:50.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6937fe700 1 -- 172.21.15.121:0/842976200 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb698009940 con 0x7fb69c105990 2024-04-05T22:07:50.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a4314700 1 -- 172.21.15.121:0/842976200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69c1a2760 con 0x7fb69c105990 2024-04-05T22:07:50.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6937fe700 1 -- 172.21.15.121:0/842976200 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb698010460 con 0x7fb69c105990 2024-04-05T22:07:50.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6937fe700 1 -- 172.21.15.121:0/842976200 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb69800f5f0 con 0x7fb69c105990 2024-04-05T22:07:50.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.077+0000 7fb6a4314700 1 -- 172.21.15.121:0/842976200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69c1a2b80 con 0x7fb69c105990 2024-04-05T22:07:50.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.078+0000 7fb6a4314700 1 -- 172.21.15.121:0/842976200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69c04ea90 con 0x7fb69c105990 2024-04-05T22:07:50.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.079+0000 7fb6937fe700 1 -- 172.21.15.121:0/842976200 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fb6980105d0 con 0x7fb69c105990 2024-04-05T22:07:50.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.079+0000 7fb6937fe700 1 --2- 172.21.15.121:0/842976200 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb68806cda0 0x7fb68806f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:50.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.079+0000 7fb6937fe700 1 -- 172.21.15.121:0/842976200 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fb69808c670 con 0x7fb69c105990 2024-04-05T22:07:50.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.079+0000 7fb6a20b0700 1 --2- 172.21.15.121:0/842976200 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb68806cda0 0x7fb68806f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:50.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.080+0000 7fb6a20b0700 1 --2- 172.21.15.121:0/842976200 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb68806cda0 0x7fb68806f260 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00b5c0 tx=0x7fb68c005fd0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:50.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.083+0000 7fb6937fe700 1 -- 172.21.15.121:0/842976200 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fb698056650 con 0x7fb69c105990 2024-04-05T22:07:50.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:49 smithi121 ceph-mon[145859]: pgmap v1250: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:50.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:49 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2643316072' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-05T22:07:50.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:49 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3286772270' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-04-05T22:07:50.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:49 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2475994786' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-04-05T22:07:50.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:49 smithi188 ceph-mon[107793]: pgmap v1250: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:50.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:49 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2643316072' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-04-05T22:07:50.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:49 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3286772270' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-04-05T22:07:50.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:49 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2475994786' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-04-05T22:07:50.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.280+0000 7fb6a4314700 1 -- 172.21.15.121:0/842976200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb69c1a2ed0 con 0x7fb69c105990 2024-04-05T22:07:50.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.281+0000 7fb6937fe700 1 -- 172.21.15.121:0/842976200 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16341 (secure 0 0 0) 0x7fb698010880 con 0x7fb69c105990 2024-04-05T22:07:50.282 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:50.282 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":84,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","created":"2024-04-05T21:15:58.797109+0000","modified":"2024-04-05T21:50:31.003759+0000","last_up_change":"2024-04-05T21:34:34.199575+0000","last_in_change":"2024-04-05T21:21:56.806570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T21:20:53.291734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-05T21:23:42.376376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-05T21:23:42.658469+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"133652f8-9fe2-45e9-90b6-36f89191114b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2472269034},{"type":"v1","addr":"172.21.15.121:6803","nonce":2472269034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2472269034},{"type":"v1","addr":"172.21.15.121:6805","nonce":2472269034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":2472269034},{"type":"v1","addr":"172.21.15.121:6809","nonce":2472269034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2472269034},{"type":"v1","addr":"172.21.15.121:6807","nonce":2472269034}]},"public_addr":"172.21.15.121:6803/2472269034","cluster_addr":"172.21.15.121:6805/2472269034","heartbeat_back_addr":"172.21.15.121:6809/2472269034","heartbeat_front_addr":"172.21.15.121:6807/2472269034","state":["exists","up"]},{"osd":1,"uuid":"6695518f-c4a5-416a-8af1-2899fdebefd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":627590186},{"type":"v1","addr":"172.21.15.121:6811","nonce":627590186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":627590186},{"type":"v1","addr":"172.21.15.121:6813","nonce":627590186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":627590186},{"type":"v1","addr":"172.21.15.121:6817","nonce":627590186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":627590186},{"type":"v1","addr":"172.21.15.121:6815","nonce":627590186}]},"public_addr":"172.21.15.121:6811/627590186","cluster_addr":"172.21.15.121:6813/627590186","heartbeat_back_addr":"172.21.15.121:6817/627590186","heartbeat_front_addr":"172.21.15.121:6815/627590186","state":["exists","up"]},{"osd":2,"uuid":"e3b6290b-ef26-4cda-96e2-8bb5649219a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2092662684},{"type":"v1","addr":"172.21.15.121:6819","nonce":2092662684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2092662684},{"type":"v1","addr":"172.21.15.121:6821","nonce":2092662684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2092662684},{"type":"v1","addr":"172.21.15.121:6825","nonce":2092662684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2092662684},{"type":"v1","addr":"172.21.15.121:6823","nonce":2092662684}]},"public_addr":"172.21.15.121:6819/2092662684","cluster_addr":"172.21.15.121:6821/2092662684","heartbeat_back_addr":"172.21.15.121:6825/2092662684","heartbeat_front_addr":"172.21.15.121:6823/2092662684","state":["exists","up"]},{"osd":3,"uuid":"77fa8da5-d202-45f1-b6fa-5af969e9300c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":83582134},{"type":"v1","addr":"172.21.15.188:6801","nonce":83582134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":83582134},{"type":"v1","addr":"172.21.15.188:6803","nonce":83582134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":83582134},{"type":"v1","addr":"172.21.15.188:6807","nonce":83582134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":83582134},{"type":"v1","addr":"172.21.15.188:6805","nonce":83582134}]},"public_addr":"172.21.15.188:6801/83582134","cluster_addr":"172.21.15.188:6803/83582134","heartbeat_back_addr":"172.21.15.188:6807/83582134","heartbeat_front_addr":"172.21.15.188:6805/83582134","state":["exists","up"]},{"osd":4,"uuid":"abf81f28-7473-45ef-a716-5ef16e038b6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3966070828},{"type":"v1","addr":"172.21.15.188:6809","nonce":3966070828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3966070828},{"type":"v1","addr":"172.21.15.188:6811","nonce":3966070828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3966070828},{"type":"v1","addr":"172.21.15.188:6815","nonce":3966070828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3966070828},{"type":"v1","addr":"172.21.15.188:6813","nonce":3966070828}]},"public_addr":"172.21.15.188:6809/3966070828","cluster_addr":"172.21.15.188:6811/3966070828","heartbeat_back_addr":"172.21.15.188:6815/3966070828","heartbeat_front_addr":"172.21.15.188:6813/3966070828","state":["exists","up"]},{"osd":5,"uuid":"700a347b-e303-447d-bf03-d6677955d140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3805501159},{"type":"v1","addr":"172.21.15.188:6817","nonce":3805501159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3805501159},{"type":"v1","addr":"172.21.15.188:6819","nonce":3805501159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3805501159},{"type":"v1","addr":"172.21.15.188:6823","nonce":3805501159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3805501159},{"type":"v1","addr":"172.21.15.188:6821","nonce":3805501159}]},"public_addr":"172.21.15.188:6817/3805501159","cluster_addr":"172.21.15.188:6819/3805501159","heartbeat_back_addr":"172.21.15.188:6823/3805501159","heartbeat_front_addr":"172.21.15.188:6821/3805501159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:19:48.038490+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:19.269394+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:51.162437+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:18.203341+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:46.805653+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:22:15.296548+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.188:6825/213881583":"2024-04-06T21:34:37.155434+0000","172.21.15.188:6824/213881583":"2024-04-06T21:34:37.155434+0000","172.21.15.121:6829/3760584056":"2024-04-06T21:34:37.155420+0000","172.21.15.121:6828/3760584056":"2024-04-06T21:34:37.155420+0000","172.21.15.121:0/566857212":"2024-04-06T21:26:08.566250+0000","172.21.15.121:6801/1237774071":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6800/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/1364585923":"2024-04-06T21:16:32.951887+0000","172.21.15.121:6801/2735531747":"2024-04-06T21:26:08.566250+0000","172.21.15.121:0/4016372676":"2024-04-06T21:26:08.566250+0000","172.21.15.188:6829/2212394428":"2024-04-06T21:27:30.099010+0000","172.21.15.188:6828/2212394428":"2024-04-06T21:27:30.099010+0000","172.21.15.121:0/42058318":"2024-04-06T21:17:08.626094+0000","172.21.15.188:0/2907182826":"2024-04-06T21:27:30.099010+0000","172.21.15.121:0/341396693":"2024-04-06T21:17:08.626094+0000","172.21.15.121:6827/1615448980":"2024-04-06T21:24:00.251505+0000","172.21.15.188:0/3491833383":"2024-04-06T21:27:30.099010+0000","172.21.15.121:6800/1237774071":"2024-04-06T21:18:24.954707+0000","172.21.15.121:0/1060423936":"2024-04-06T21:26:08.566250+0000","172.21.15.121:6800/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/3762940011":"2024-04-06T21:18:24.954707+0000","172.21.15.188:0/1842750729":"2024-04-06T21:27:30.099010+0000","172.21.15.121:6800/2735531747":"2024-04-06T21:26:08.566250+0000","172.21.15.121:0/2214582362":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/2536941662":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/2383002272":"2024-04-06T21:16:32.951887+0000","172.21.15.121:6801/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.188:0/6988375":"2024-04-06T21:27:30.099010+0000","172.21.15.121:0/3247654758":"2024-04-06T21:18:24.954707+0000","172.21.15.188:0/1621914365":"2024-04-06T21:27:30.099010+0000","172.21.15.121:6826/1615448980":"2024-04-06T21:24:00.251505+0000","172.21.15.188:0/4116965307":"2024-04-06T21:27:30.099010+0000","172.21.15.121:6801/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/3235459786":"2024-04-06T21:18:24.954707+0000","172.21.15.121:0/846460013":"2024-04-06T21:26:08.566250+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T22:07:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.284+0000 7fb6a4314700 1 -- 172.21.15.121:0/842976200 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb68806cda0 msgr2=0x7fb68806f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.284+0000 7fb6a4314700 1 --2- 172.21.15.121:0/842976200 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb68806cda0 0x7fb68806f260 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00b5c0 tx=0x7fb68c005fd0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.284+0000 7fb6a4314700 1 -- 172.21.15.121:0/842976200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb69c105990 msgr2=0x7fb69c101a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.284+0000 7fb6a4314700 1 --2- 172.21.15.121:0/842976200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb69c105990 0x7fb69c101a90 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fb69800d8d0 tx=0x7fb69800dc90 comp rx=0 tx=0).stop 2024-04-05T22:07:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.284+0000 7fb6a4314700 1 -- 172.21.15.121:0/842976200 shutdown_connections 2024-04-05T22:07:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.284+0000 7fb6a4314700 1 --2- 172.21.15.121:0/842976200 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fb68806cda0 0x7fb68806f260 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.284+0000 7fb6a4314700 1 --2- 172.21.15.121:0/842976200 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fb69c0ffac0 0x7fb69c101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.284+0000 7fb6a4314700 1 --2- 172.21.15.121:0/842976200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb69c105990 0x7fb69c101a90 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.284+0000 7fb6a4314700 1 -- 172.21.15.121:0/842976200 >> 172.21.15.121:0/842976200 conn(0x7fb69c075400 msgr2=0x7fb69c067da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.284+0000 7fb6a4314700 1 -- 172.21.15.121:0/842976200 shutdown_connections 2024-04-05T22:07:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.284+0000 7fb6a4314700 1 -- 172.21.15.121:0/842976200 wait complete. 2024-04-05T22:07:50.296 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T21:20:53.291734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-05T21:23:42.376376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-05T21:23:42.658469+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T22:07:50.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T22:07:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.421+0000 7fbdaec77700 1 -- 172.21.15.121:0/557797703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbda8067da0 msgr2=0x7fbda8068220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.421+0000 7fbdaec77700 1 --2- 172.21.15.121:0/557797703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbda8067da0 0x7fbda8068220 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009910 tx=0x7fbd94009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.421+0000 7fbdaec77700 1 -- 172.21.15.121:0/557797703 shutdown_connections 2024-04-05T22:07:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.421+0000 7fbdaec77700 1 --2- 172.21.15.121:0/557797703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbda8067da0 0x7fbda8068220 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.421+0000 7fbdaec77700 1 --2- 172.21.15.121:0/557797703 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbda8106b90 0x7fbda8106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.421+0000 7fbdaec77700 1 -- 172.21.15.121:0/557797703 >> 172.21.15.121:0/557797703 conn(0x7fbda80753d0 msgr2=0x7fbda80757e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.421+0000 7fbdaec77700 1 -- 172.21.15.121:0/557797703 shutdown_connections 2024-04-05T22:07:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.421+0000 7fbdaec77700 1 -- 172.21.15.121:0/557797703 wait complete. 2024-04-05T22:07:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.421+0000 7fbdaec77700 1 Processor -- start 2024-04-05T22:07:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.421+0000 7fbdaec77700 1 -- start start 2024-04-05T22:07:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbdaec77700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbda8067da0 0x7fbda8196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbdaec77700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbda8106b90 0x7fbda8196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbdaec77700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda8197140 con 0x7fbda8067da0 2024-04-05T22:07:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbdaec77700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda819aec0 con 0x7fbda8106b90 2024-04-05T22:07:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbdaca13700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbda8067da0 0x7fbda8196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbdaca13700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbda8067da0 0x7fbda8196520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37068/0 (socket says 172.21.15.121:37068) 2024-04-05T22:07:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbdaca13700 1 -- 172.21.15.121:0/3352936063 learned_addr learned my addr 172.21.15.121:0/3352936063 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbda7fff700 1 --2- 172.21.15.121:0/3352936063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbda8106b90 0x7fbda8196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbdaca13700 1 -- 172.21.15.121:0/3352936063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbda8106b90 msgr2=0x7fbda8196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbdaca13700 1 --2- 172.21.15.121:0/3352936063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbda8106b90 0x7fbda8196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbdaca13700 1 -- 172.21.15.121:0/3352936063 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd940095f0 con 0x7fbda8067da0 2024-04-05T22:07:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbda7fff700 1 --2- 172.21.15.121:0/3352936063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbda8106b90 0x7fbda8196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:07:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbdaca13700 1 --2- 172.21.15.121:0/3352936063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbda8067da0 0x7fbda8196520 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00d8d0 tx=0x7fbd9c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbda5ffb700 1 -- 172.21.15.121:0/3352936063 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd9c009940 con 0x7fbda8067da0 2024-04-05T22:07:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbda5ffb700 1 -- 172.21.15.121:0/3352936063 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd9c010460 con 0x7fbda8067da0 2024-04-05T22:07:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbdaec77700 1 -- 172.21.15.121:0/3352936063 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda819b1a0 con 0x7fbda8067da0 2024-04-05T22:07:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.422+0000 7fbda5ffb700 1 -- 172.21.15.121:0/3352936063 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd9c00f5f0 con 0x7fbda8067da0 2024-04-05T22:07:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.423+0000 7fbdaec77700 1 -- 172.21.15.121:0/3352936063 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda819b780 con 0x7fbda8067da0 2024-04-05T22:07:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.423+0000 7fbdaec77700 1 -- 172.21.15.121:0/3352936063 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda8108f70 con 0x7fbda8067da0 2024-04-05T22:07:50.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.424+0000 7fbda5ffb700 1 -- 172.21.15.121:0/3352936063 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fbd9c0105d0 con 0x7fbda8067da0 2024-04-05T22:07:50.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.424+0000 7fbda5ffb700 1 --2- 172.21.15.121:0/3352936063 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd9806cce0 0x7fbd9806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:50.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.424+0000 7fbda5ffb700 1 -- 172.21.15.121:0/3352936063 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fbd9c08c5e0 con 0x7fbda8067da0 2024-04-05T22:07:50.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.424+0000 7fbda7fff700 1 --2- 172.21.15.121:0/3352936063 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd9806cce0 0x7fbd9806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:50.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.425+0000 7fbda7fff700 1 --2- 172.21.15.121:0/3352936063 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd9806cce0 0x7fbd9806f1a0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbd94006010 tx=0x7fbd9400b540 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:50.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.430+0000 7fbda5ffb700 1 -- 172.21.15.121:0/3352936063 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fbd9c056650 con 0x7fbda8067da0 2024-04-05T22:07:50.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.623+0000 7fbdaec77700 1 -- 172.21.15.121:0/3352936063 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbda819b9a0 con 0x7fbda8067da0 2024-04-05T22:07:50.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.623+0000 7fbda5ffb700 1 -- 172.21.15.121:0/3352936063 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbd9c010880 con 0x7fbda8067da0 2024-04-05T22:07:50.623 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2024-04-05T22:07:50.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.625+0000 7fbdaec77700 1 -- 172.21.15.121:0/3352936063 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd9806cce0 msgr2=0x7fbd9806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:50.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.625+0000 7fbdaec77700 1 --2- 172.21.15.121:0/3352936063 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd9806cce0 0x7fbd9806f1a0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbd94006010 tx=0x7fbd9400b540 comp rx=0 tx=0).stop 2024-04-05T22:07:50.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.625+0000 7fbdaec77700 1 -- 172.21.15.121:0/3352936063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbda8067da0 msgr2=0x7fbda8196520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:50.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.625+0000 7fbdaec77700 1 --2- 172.21.15.121:0/3352936063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbda8067da0 0x7fbda8196520 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00d8d0 tx=0x7fbd9c00dc90 comp rx=0 tx=0).stop 2024-04-05T22:07:50.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.625+0000 7fbdaec77700 1 -- 172.21.15.121:0/3352936063 shutdown_connections 2024-04-05T22:07:50.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.625+0000 7fbdaec77700 1 --2- 172.21.15.121:0/3352936063 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fbd9806cce0 0x7fbd9806f1a0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.625+0000 7fbdaec77700 1 --2- 172.21.15.121:0/3352936063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbda8067da0 0x7fbda8196520 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.625+0000 7fbdaec77700 1 --2- 172.21.15.121:0/3352936063 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fbda8106b90 0x7fbda8196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.625+0000 7fbdaec77700 1 -- 172.21.15.121:0/3352936063 >> 172.21.15.121:0/3352936063 conn(0x7fbda80753d0 msgr2=0x7fbda80feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:50.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.625+0000 7fbdaec77700 1 -- 172.21.15.121:0/3352936063 shutdown_connections 2024-04-05T22:07:50.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.625+0000 7fbdaec77700 1 -- 172.21.15.121:0/3352936063 wait complete. 2024-04-05T22:07:50.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-05T22:07:50.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.763+0000 7f915b2da700 1 -- 172.21.15.121:0/2351306573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9154100f00 msgr2=0x7f9154104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:50.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.763+0000 7f915b2da700 1 --2- 172.21.15.121:0/2351306573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9154100f00 0x7f9154104600 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f9150009910 tx=0x7f9150009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:50.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.763+0000 7f915b2da700 1 -- 172.21.15.121:0/2351306573 shutdown_connections 2024-04-05T22:07:50.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.763+0000 7f915b2da700 1 --2- 172.21.15.121:0/2351306573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9154100f00 0x7f9154104600 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.763+0000 7f915b2da700 1 --2- 172.21.15.121:0/2351306573 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9154100550 0x7f9154100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.763+0000 7f915b2da700 1 -- 172.21.15.121:0/2351306573 >> 172.21.15.121:0/2351306573 conn(0x7f9154074c10 msgr2=0x7f9154075020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:50.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.763+0000 7f915b2da700 1 -- 172.21.15.121:0/2351306573 shutdown_connections 2024-04-05T22:07:50.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.763+0000 7f915b2da700 1 -- 172.21.15.121:0/2351306573 wait complete. 2024-04-05T22:07:50.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f915b2da700 1 Processor -- start 2024-04-05T22:07:50.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f915b2da700 1 -- start start 2024-04-05T22:07:50.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f915b2da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9154100550 0x7f915419b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:50.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f915b2da700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f915419bac0 0x7f91541955f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:50.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f915b2da700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915419c090 con 0x7f9154100550 2024-04-05T22:07:50.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f915b2da700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9154195b30 con 0x7f915419bac0 2024-04-05T22:07:50.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f9159076700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9154100550 0x7f915419b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:50.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f9159076700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9154100550 0x7f915419b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37078/0 (socket says 172.21.15.121:37078) 2024-04-05T22:07:50.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f9159076700 1 -- 172.21.15.121:0/294919265 learned_addr learned my addr 172.21.15.121:0/294919265 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:50.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f9158875700 1 --2- 172.21.15.121:0/294919265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f915419bac0 0x7f91541955f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:50.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f9159076700 1 -- 172.21.15.121:0/294919265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f915419bac0 msgr2=0x7f91541955f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:50.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f9159076700 1 --2- 172.21.15.121:0/294919265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f915419bac0 0x7f91541955f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f9159076700 1 -- 172.21.15.121:0/294919265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91500095f0 con 0x7f9154100550 2024-04-05T22:07:50.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.764+0000 7f9159076700 1 --2- 172.21.15.121:0/294919265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9154100550 0x7f915419b580 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f914400c8a0 tx=0x7f914400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:50.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.765+0000 7f914a7fc700 1 -- 172.21.15.121:0/294919265 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f914400cea0 con 0x7f9154100550 2024-04-05T22:07:50.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.765+0000 7f914a7fc700 1 -- 172.21.15.121:0/294919265 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9144004830 con 0x7f9154100550 2024-04-05T22:07:50.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.765+0000 7f915b2da700 1 -- 172.21.15.121:0/294919265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9154195e10 con 0x7f9154100550 2024-04-05T22:07:50.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.765+0000 7f914a7fc700 1 -- 172.21.15.121:0/294919265 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9144005500 con 0x7f9154100550 2024-04-05T22:07:50.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.765+0000 7f915b2da700 1 -- 172.21.15.121:0/294919265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9154196370 con 0x7f9154100550 2024-04-05T22:07:50.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.765+0000 7f915b2da700 1 -- 172.21.15.121:0/294919265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9154108040 con 0x7f9154100550 2024-04-05T22:07:50.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.766+0000 7f914a7fc700 1 -- 172.21.15.121:0/294919265 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f91440049a0 con 0x7f9154100550 2024-04-05T22:07:50.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.766+0000 7f914a7fc700 1 --2- 172.21.15.121:0/294919265 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f914006cce0 0x7f914006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:50.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.766+0000 7f914a7fc700 1 -- 172.21.15.121:0/294919265 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f914408c1d0 con 0x7f9154100550 2024-04-05T22:07:50.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.766+0000 7f9158875700 1 --2- 172.21.15.121:0/294919265 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f914006cce0 0x7f914006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:50.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.767+0000 7f9158875700 1 --2- 172.21.15.121:0/294919265 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f914006cce0 0x7f914006f1a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9150000c00 tx=0x7f915001a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:50.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.771+0000 7f914a7fc700 1 -- 172.21.15.121:0/294919265 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9144059d50 con 0x7f9154100550 2024-04-05T22:07:50.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.966+0000 7f915b2da700 1 -- 172.21.15.121:0/294919265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f915404ea90 con 0x7f9154100550 2024-04-05T22:07:50.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.966+0000 7f914a7fc700 1 -- 172.21.15.121:0/294919265 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f91440598e0 con 0x7f9154100550 2024-04-05T22:07:50.966 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 32 2024-04-05T22:07:50.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.969+0000 7f915b2da700 1 -- 172.21.15.121:0/294919265 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f914006cce0 msgr2=0x7f914006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:50.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.969+0000 7f915b2da700 1 --2- 172.21.15.121:0/294919265 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f914006cce0 0x7f914006f1a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9150000c00 tx=0x7f915001a040 comp rx=0 tx=0).stop 2024-04-05T22:07:50.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.969+0000 7f915b2da700 1 -- 172.21.15.121:0/294919265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9154100550 msgr2=0x7f915419b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:50.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.969+0000 7f915b2da700 1 --2- 172.21.15.121:0/294919265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9154100550 0x7f915419b580 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f914400c8a0 tx=0x7f914400cc60 comp rx=0 tx=0).stop 2024-04-05T22:07:50.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.969+0000 7f915b2da700 1 -- 172.21.15.121:0/294919265 shutdown_connections 2024-04-05T22:07:50.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.969+0000 7f915b2da700 1 --2- 172.21.15.121:0/294919265 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f914006cce0 0x7f914006f1a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.969+0000 7f915b2da700 1 --2- 172.21.15.121:0/294919265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9154100550 0x7f915419b580 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.969+0000 7f915b2da700 1 --2- 172.21.15.121:0/294919265 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f915419bac0 0x7f91541955f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:50.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.969+0000 7f915b2da700 1 -- 172.21.15.121:0/294919265 >> 172.21.15.121:0/294919265 conn(0x7f9154074c10 msgr2=0x7f91540fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:50.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.969+0000 7f915b2da700 1 -- 172.21.15.121:0/294919265 shutdown_connections 2024-04-05T22:07:50.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:50.969+0000 7f915b2da700 1 -- 172.21.15.121:0/294919265 wait complete. 2024-04-05T22:07:50.979 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:50 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/842976200' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T22:07:50.979 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:50 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/3352936063' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T22:07:50.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-05T22:07:51.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.105+0000 7f9528263700 1 -- 172.21.15.121:0/3431374821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95200ff960 msgr2=0x7f95200ffde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:51.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.105+0000 7f9528263700 1 --2- 172.21.15.121:0/3431374821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95200ff960 0x7f95200ffde0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f9510009910 tx=0x7f9510009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:51.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.106+0000 7f9528263700 1 -- 172.21.15.121:0/3431374821 shutdown_connections 2024-04-05T22:07:51.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.106+0000 7f9528263700 1 --2- 172.21.15.121:0/3431374821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95200ff960 0x7f95200ffde0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.106+0000 7f9528263700 1 --2- 172.21.15.121:0/3431374821 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f9520105970 0x7f9520105d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.106+0000 7f9528263700 1 -- 172.21.15.121:0/3431374821 >> 172.21.15.121:0/3431374821 conn(0x7f95200fb450 msgr2=0x7f95200fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:51.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.106+0000 7f9528263700 1 -- 172.21.15.121:0/3431374821 shutdown_connections 2024-04-05T22:07:51.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.106+0000 7f9528263700 1 -- 172.21.15.121:0/3431374821 wait complete. 2024-04-05T22:07:51.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.106+0000 7f9528263700 1 Processor -- start 2024-04-05T22:07:51.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.106+0000 7f9528263700 1 -- start start 2024-04-05T22:07:51.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f9528263700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95200ff960 0x7f9520197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f9528263700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9520105970 0x7f9520197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f9528263700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9520197ee0 con 0x7f9520105970 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f9528263700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9520198020 con 0x7f95200ff960 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f95257fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9520105970 0x7f9520197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f95257fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9520105970 0x7f9520197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37084/0 (socket says 172.21.15.121:37084) 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f95257fe700 1 -- 172.21.15.121:0/2515150421 learned_addr learned my addr 172.21.15.121:0/2515150421 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f9525fff700 1 --2- 172.21.15.121:0/2515150421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95200ff960 0x7f9520197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f95257fe700 1 -- 172.21.15.121:0/2515150421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95200ff960 msgr2=0x7f9520197410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f95257fe700 1 --2- 172.21.15.121:0/2515150421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95200ff960 0x7f9520197410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f95257fe700 1 -- 172.21.15.121:0/2515150421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95100095f0 con 0x7f9520105970 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f95257fe700 1 --2- 172.21.15.121:0/2515150421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9520105970 0x7f9520197950 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f9510006010 tx=0x7f951000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f9516ffd700 1 -- 172.21.15.121:0/2515150421 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f951001d070 con 0x7f9520105970 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f9516ffd700 1 -- 172.21.15.121:0/2515150421 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f951000f460 con 0x7f9520105970 2024-04-05T22:07:51.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f9528263700 1 -- 172.21.15.121:0/2515150421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95201a0480 con 0x7f9520105970 2024-04-05T22:07:51.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f9516ffd700 1 -- 172.21.15.121:0/2515150421 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9510021640 con 0x7f9520105970 2024-04-05T22:07:51.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.107+0000 7f9528263700 1 -- 172.21.15.121:0/2515150421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95201a0840 con 0x7f9520105970 2024-04-05T22:07:51.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.108+0000 7f9528263700 1 -- 172.21.15.121:0/2515150421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9520107ef0 con 0x7f9520105970 2024-04-05T22:07:51.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.109+0000 7f9516ffd700 1 -- 172.21.15.121:0/2515150421 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f951000fa20 con 0x7f9520105970 2024-04-05T22:07:51.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.109+0000 7f9516ffd700 1 --2- 172.21.15.121:0/2515150421 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f950c06cce0 0x7f950c06f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:51.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.109+0000 7f9516ffd700 1 -- 172.21.15.121:0/2515150421 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f951008dc60 con 0x7f9520105970 2024-04-05T22:07:51.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.109+0000 7f9525fff700 1 --2- 172.21.15.121:0/2515150421 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f950c06cce0 0x7f950c06f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:51.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.110+0000 7f9525fff700 1 --2- 172.21.15.121:0/2515150421 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f950c06cce0 0x7f950c06f1a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f951c005ea0 tx=0x7f951c005e30 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:51.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.113+0000 7f9516ffd700 1 -- 172.21.15.121:0/2515150421 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f9510057d50 con 0x7f9520105970 2024-04-05T22:07:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:50 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/842976200' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T22:07:51.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:50 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/3352936063' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T22:07:51.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.307+0000 7f9528263700 1 -- 172.21.15.121:0/2515150421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f9520069360 con 0x7f9520105970 2024-04-05T22:07:51.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.308+0000 7f9516ffd700 1 -- 172.21.15.121:0/2515150421 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f951005b370 con 0x7f9520105970 2024-04-05T22:07:51.307 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 32 2024-04-05T22:07:51.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.310+0000 7f9528263700 1 -- 172.21.15.121:0/2515150421 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f950c06cce0 msgr2=0x7f950c06f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:51.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.310+0000 7f9528263700 1 --2- 172.21.15.121:0/2515150421 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f950c06cce0 0x7f950c06f1a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f951c005ea0 tx=0x7f951c005e30 comp rx=0 tx=0).stop 2024-04-05T22:07:51.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.310+0000 7f9528263700 1 -- 172.21.15.121:0/2515150421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9520105970 msgr2=0x7f9520197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:51.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.310+0000 7f9528263700 1 --2- 172.21.15.121:0/2515150421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9520105970 0x7f9520197950 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f9510006010 tx=0x7f951000be20 comp rx=0 tx=0).stop 2024-04-05T22:07:51.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.310+0000 7f9528263700 1 -- 172.21.15.121:0/2515150421 shutdown_connections 2024-04-05T22:07:51.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.310+0000 7f9528263700 1 --2- 172.21.15.121:0/2515150421 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f950c06cce0 0x7f950c06f1a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.310+0000 7f9528263700 1 --2- 172.21.15.121:0/2515150421 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f95200ff960 0x7f9520197410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.310+0000 7f9528263700 1 --2- 172.21.15.121:0/2515150421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9520105970 0x7f9520197950 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.310+0000 7f9528263700 1 -- 172.21.15.121:0/2515150421 >> 172.21.15.121:0/2515150421 conn(0x7f95200fb450 msgr2=0x7f95200fc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:51.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.310+0000 7f9528263700 1 -- 172.21.15.121:0/2515150421 shutdown_connections 2024-04-05T22:07:51.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.310+0000 7f9528263700 1 -- 172.21.15.121:0/2515150421 wait complete. 2024-04-05T22:07:51.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-05T22:07:51.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.446+0000 7f38707c7700 1 -- 172.21.15.121:0/3904349220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3868105990 msgr2=0x7f3868105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:51.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.446+0000 7f38707c7700 1 --2- 172.21.15.121:0/3904349220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3868105990 0x7f3868105d70 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f3858009910 tx=0x7f3858009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:51.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.446+0000 7f38707c7700 1 -- 172.21.15.121:0/3904349220 shutdown_connections 2024-04-05T22:07:51.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.446+0000 7f38707c7700 1 --2- 172.21.15.121:0/3904349220 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38680ffac0 0x7f38680fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.446+0000 7f38707c7700 1 --2- 172.21.15.121:0/3904349220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3868105990 0x7f3868105d70 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.446+0000 7f38707c7700 1 -- 172.21.15.121:0/3904349220 >> 172.21.15.121:0/3904349220 conn(0x7f3868075400 msgr2=0x7f3868075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:51.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.446+0000 7f38707c7700 1 -- 172.21.15.121:0/3904349220 shutdown_connections 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.446+0000 7f38707c7700 1 -- 172.21.15.121:0/3904349220 wait complete. 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f38707c7700 1 Processor -- start 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f38707c7700 1 -- start start 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f38707c7700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38680ffac0 0x7f3868197450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f38707c7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3868105990 0x7f3868197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f38707c7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3868197fb0 con 0x7f3868105990 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f38707c7700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38681980f0 con 0x7f38680ffac0 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f386dd62700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3868105990 0x7f3868197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f386dd62700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3868105990 0x7f3868197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37104/0 (socket says 172.21.15.121:37104) 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f386dd62700 1 -- 172.21.15.121:0/2256553096 learned_addr learned my addr 172.21.15.121:0/2256553096 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f386e563700 1 --2- 172.21.15.121:0/2256553096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38680ffac0 0x7f3868197450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f386dd62700 1 -- 172.21.15.121:0/2256553096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38680ffac0 msgr2=0x7f3868197450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f386dd62700 1 --2- 172.21.15.121:0/2256553096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38680ffac0 0x7f3868197450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f386dd62700 1 -- 172.21.15.121:0/2256553096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38580095f0 con 0x7f3868105990 2024-04-05T22:07:51.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f386e563700 1 --2- 172.21.15.121:0/2256553096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38680ffac0 0x7f3868197450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:07:51.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f386dd62700 1 --2- 172.21.15.121:0/2256553096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3868105990 0x7f3868197990 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f386400d8d0 tx=0x7f386400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:51.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f385f7fe700 1 -- 172.21.15.121:0/2256553096 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3864009940 con 0x7f3868105990 2024-04-05T22:07:51.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f385f7fe700 1 -- 172.21.15.121:0/2256553096 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3864010460 con 0x7f3868105990 2024-04-05T22:07:51.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f38707c7700 1 -- 172.21.15.121:0/2256553096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38681a04c0 con 0x7f3868105990 2024-04-05T22:07:51.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f385f7fe700 1 -- 172.21.15.121:0/2256553096 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f386400f5f0 con 0x7f3868105990 2024-04-05T22:07:51.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.447+0000 7f38707c7700 1 -- 172.21.15.121:0/2256553096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38681a0990 con 0x7f3868105990 2024-04-05T22:07:51.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.448+0000 7f38707c7700 1 -- 172.21.15.121:0/2256553096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3868107f30 con 0x7f3868105990 2024-04-05T22:07:51.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.449+0000 7f385f7fe700 1 -- 172.21.15.121:0/2256553096 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f38640105d0 con 0x7f3868105990 2024-04-05T22:07:51.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.449+0000 7f385f7fe700 1 --2- 172.21.15.121:0/2256553096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f385406cda0 0x7f385406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:51.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.449+0000 7f385f7fe700 1 -- 172.21.15.121:0/2256553096 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f386408c5e0 con 0x7f3868105990 2024-04-05T22:07:51.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.449+0000 7f386e563700 1 --2- 172.21.15.121:0/2256553096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f385406cda0 0x7f385406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:51.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.450+0000 7f386e563700 1 --2- 172.21.15.121:0/2256553096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f385406cda0 0x7f385406f260 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f385800b5c0 tx=0x7f3858005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:51.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.454+0000 7f385f7fe700 1 -- 172.21.15.121:0/2256553096 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3864056650 con 0x7f3868105990 2024-04-05T22:07:51.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.670+0000 7f38707c7700 1 -- 172.21.15.121:0/2256553096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f386804ea90 con 0x7f3868105990 2024-04-05T22:07:51.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.670+0000 7f385f7fe700 1 -- 172.21.15.121:0/2256553096 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 35 entries v84) v1 ==== 81+0+2101 (secure 0 0 0) 0x7f3864010880 con 0x7f3868105990 2024-04-05T22:07:51.670 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:6825/213881583 2024-04-06T21:34:37.155434+0000 2024-04-05T22:07:51.670 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:6824/213881583 2024-04-06T21:34:37.155434+0000 2024-04-05T22:07:51.670 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6829/3760584056 2024-04-06T21:34:37.155420+0000 2024-04-05T22:07:51.670 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6828/3760584056 2024-04-06T21:34:37.155420+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/566857212 2024-04-06T21:26:08.566250+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/1237774071 2024-04-06T21:18:24.954707+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/1004235250 2024-04-06T21:17:08.626094+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1364585923 2024-04-06T21:16:32.951887+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/2735531747 2024-04-06T21:26:08.566250+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/4016372676 2024-04-06T21:26:08.566250+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:6829/2212394428 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:6828/2212394428 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/42058318 2024-04-06T21:17:08.626094+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:0/2907182826 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/341396693 2024-04-06T21:17:08.626094+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6827/1615448980 2024-04-06T21:24:00.251505+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:0/3491833383 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:51.671 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/1237774071 2024-04-06T21:18:24.954707+0000 2024-04-05T22:07:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1060423936 2024-04-06T21:26:08.566250+0000 2024-04-05T22:07:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/169707344 2024-04-06T21:16:32.951887+0000 2024-04-05T22:07:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/3762940011 2024-04-06T21:18:24.954707+0000 2024-04-05T22:07:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:0/1842750729 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/2735531747 2024-04-06T21:26:08.566250+0000 2024-04-05T22:07:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/2214582362 2024-04-06T21:17:08.626094+0000 2024-04-05T22:07:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/2536941662 2024-04-06T21:16:32.951887+0000 2024-04-05T22:07:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/2383002272 2024-04-06T21:16:32.951887+0000 2024-04-05T22:07:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/169707344 2024-04-06T21:16:32.951887+0000 2024-04-05T22:07:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:0/6988375 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/3247654758 2024-04-06T21:18:24.954707+0000 2024-04-05T22:07:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:0/1621914365 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:51.672 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6826/1615448980 2024-04-06T21:24:00.251505+0000 2024-04-05T22:07:51.673 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:0/4116965307 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:51.673 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/1004235250 2024-04-06T21:17:08.626094+0000 2024-04-05T22:07:51.673 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/3235459786 2024-04-06T21:18:24.954707+0000 2024-04-05T22:07:51.673 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/846460013 2024-04-06T21:26:08.566250+0000 2024-04-05T22:07:51.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.672+0000 7f38707c7700 1 -- 172.21.15.121:0/2256553096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f385406cda0 msgr2=0x7f385406f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:51.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.672+0000 7f38707c7700 1 --2- 172.21.15.121:0/2256553096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f385406cda0 0x7f385406f260 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f385800b5c0 tx=0x7f3858005960 comp rx=0 tx=0).stop 2024-04-05T22:07:51.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.672+0000 7f38707c7700 1 -- 172.21.15.121:0/2256553096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3868105990 msgr2=0x7f3868197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:51.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.672+0000 7f38707c7700 1 --2- 172.21.15.121:0/2256553096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3868105990 0x7f3868197990 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f386400d8d0 tx=0x7f386400dc90 comp rx=0 tx=0).stop 2024-04-05T22:07:51.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.672+0000 7f38707c7700 1 -- 172.21.15.121:0/2256553096 shutdown_connections 2024-04-05T22:07:51.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.672+0000 7f38707c7700 1 --2- 172.21.15.121:0/2256553096 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f385406cda0 0x7f385406f260 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.672+0000 7f38707c7700 1 --2- 172.21.15.121:0/2256553096 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f38680ffac0 0x7f3868197450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.672+0000 7f38707c7700 1 --2- 172.21.15.121:0/2256553096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3868105990 0x7f3868197990 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.672+0000 7f38707c7700 1 -- 172.21.15.121:0/2256553096 >> 172.21.15.121:0/2256553096 conn(0x7f3868075400 msgr2=0x7f38680fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:51.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.673+0000 7f38707c7700 1 -- 172.21.15.121:0/2256553096 shutdown_connections 2024-04-05T22:07:51.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.673+0000 7f38707c7700 1 -- 172.21.15.121:0/2256553096 wait complete. 2024-04-05T22:07:51.674 INFO:teuthology.orchestra.run.smithi121.stderr:listed 35 entries 2024-04-05T22:07:51.685 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-05T22:07:51.685 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-05T22:07:51.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-05T22:07:51.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.866+0000 7fc639499700 1 -- 172.21.15.121:0/576977116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6341059a0 msgr2=0x7fc634105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.866+0000 7fc639499700 1 --2- 172.21.15.121:0/576977116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6341059a0 0x7fc634105d80 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fc628009910 tx=0x7fc628009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.867+0000 7fc639499700 1 -- 172.21.15.121:0/576977116 shutdown_connections 2024-04-05T22:07:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.867+0000 7fc639499700 1 --2- 172.21.15.121:0/576977116 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6340ff990 0x7fc6340ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.867+0000 7fc639499700 1 --2- 172.21.15.121:0/576977116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6341059a0 0x7fc634105d80 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.867+0000 7fc639499700 1 -- 172.21.15.121:0/576977116 >> 172.21.15.121:0/576977116 conn(0x7fc6340fb480 msgr2=0x7fc6340fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.867+0000 7fc639499700 1 -- 172.21.15.121:0/576977116 shutdown_connections 2024-04-05T22:07:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.867+0000 7fc639499700 1 -- 172.21.15.121:0/576977116 wait complete. 2024-04-05T22:07:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.867+0000 7fc639499700 1 Processor -- start 2024-04-05T22:07:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.867+0000 7fc639499700 1 -- start start 2024-04-05T22:07:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.867+0000 7fc639499700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6340ff990 0x7fc634196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.867+0000 7fc639499700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6341059a0 0x7fc634196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.867+0000 7fc639499700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc634197250 con 0x7fc6341059a0 2024-04-05T22:07:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.867+0000 7fc639499700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63419afd0 con 0x7fc6340ff990 2024-04-05T22:07:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.868+0000 7fc6327fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6341059a0 0x7fc634196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.868+0000 7fc6327fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6341059a0 0x7fc634196b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37128/0 (socket says 172.21.15.121:37128) 2024-04-05T22:07:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.868+0000 7fc6327fc700 1 -- 172.21.15.121:0/216271216 learned_addr learned my addr 172.21.15.121:0/216271216 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.868+0000 7fc632ffd700 1 --2- 172.21.15.121:0/216271216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6340ff990 0x7fc634196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.868+0000 7fc6327fc700 1 -- 172.21.15.121:0/216271216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6340ff990 msgr2=0x7fc634196630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.868+0000 7fc6327fc700 1 --2- 172.21.15.121:0/216271216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6340ff990 0x7fc634196630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.868+0000 7fc6327fc700 1 -- 172.21.15.121:0/216271216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6280095f0 con 0x7fc6341059a0 2024-04-05T22:07:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.868+0000 7fc6327fc700 1 --2- 172.21.15.121:0/216271216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6341059a0 0x7fc634196b70 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fc62400d8d0 tx=0x7fc62400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.868+0000 7fc61bfff700 1 -- 172.21.15.121:0/216271216 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6240098a0 con 0x7fc6341059a0 2024-04-05T22:07:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.868+0000 7fc61bfff700 1 -- 172.21.15.121:0/216271216 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc624010460 con 0x7fc6341059a0 2024-04-05T22:07:51.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.868+0000 7fc639499700 1 -- 172.21.15.121:0/216271216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63419b2b0 con 0x7fc6341059a0 2024-04-05T22:07:51.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.868+0000 7fc61bfff700 1 -- 172.21.15.121:0/216271216 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc624009ea0 con 0x7fc6341059a0 2024-04-05T22:07:51.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.868+0000 7fc639499700 1 -- 172.21.15.121:0/216271216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc63419b810 con 0x7fc6341059a0 2024-04-05T22:07:51.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.869+0000 7fc639499700 1 -- 172.21.15.121:0/216271216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc634108fa0 con 0x7fc6341059a0 2024-04-05T22:07:51.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.870+0000 7fc61bfff700 1 -- 172.21.15.121:0/216271216 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fc6240105d0 con 0x7fc6341059a0 2024-04-05T22:07:51.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.870+0000 7fc61bfff700 1 --2- 172.21.15.121:0/216271216 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc61c06cda0 0x7fc61c06f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:51.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.870+0000 7fc61bfff700 1 -- 172.21.15.121:0/216271216 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fc62408c480 con 0x7fc6341059a0 2024-04-05T22:07:51.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.870+0000 7fc632ffd700 1 --2- 172.21.15.121:0/216271216 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc61c06cda0 0x7fc61c06f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:51.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.870+0000 7fc632ffd700 1 --2- 172.21.15.121:0/216271216 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc61c06cda0 0x7fc61c06f260 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc628006010 tx=0x7fc62801a040 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:51.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:51.874+0000 7fc61bfff700 1 -- 172.21.15.121:0/216271216 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fc6240564f0 con 0x7fc6341059a0 2024-04-05T22:07:51.875 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:51 smithi121 ceph-mon[145859]: pgmap v1251: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:51.876 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:51 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/294919265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T22:07:52.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.067+0000 7fc639499700 1 -- 172.21.15.121:0/216271216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc63404ea90 con 0x7fc6341059a0 2024-04-05T22:07:52.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.068+0000 7fc61bfff700 1 -- 172.21.15.121:0/216271216 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16341 (secure 0 0 0) 0x7fc624010880 con 0x7fc6341059a0 2024-04-05T22:07:52.068 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-05T22:07:52.069 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":84,"fsid":"9e6dd7f6-f391-11ee-b649-cb9ed24678a4","created":"2024-04-05T21:15:58.797109+0000","modified":"2024-04-05T21:50:31.003759+0000","last_up_change":"2024-04-05T21:34:34.199575+0000","last_in_change":"2024-04-05T21:21:56.806570+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-05T21:20:53.291734+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-05T21:23:42.376376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-05T21:23:42.658469+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"133652f8-9fe2-45e9-90b6-36f89191114b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":48,"up_from":52,"up_thru":77,"down_at":49,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2472269034},{"type":"v1","addr":"172.21.15.121:6803","nonce":2472269034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2472269034},{"type":"v1","addr":"172.21.15.121:6805","nonce":2472269034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":2472269034},{"type":"v1","addr":"172.21.15.121:6809","nonce":2472269034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2472269034},{"type":"v1","addr":"172.21.15.121:6807","nonce":2472269034}]},"public_addr":"172.21.15.121:6803/2472269034","cluster_addr":"172.21.15.121:6805/2472269034","heartbeat_back_addr":"172.21.15.121:6809/2472269034","heartbeat_front_addr":"172.21.15.121:6807/2472269034","state":["exists","up"]},{"osd":1,"uuid":"6695518f-c4a5-416a-8af1-2899fdebefd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":53,"up_from":57,"up_thru":77,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":627590186},{"type":"v1","addr":"172.21.15.121:6811","nonce":627590186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":627590186},{"type":"v1","addr":"172.21.15.121:6813","nonce":627590186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":627590186},{"type":"v1","addr":"172.21.15.121:6817","nonce":627590186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":627590186},{"type":"v1","addr":"172.21.15.121:6815","nonce":627590186}]},"public_addr":"172.21.15.121:6811/627590186","cluster_addr":"172.21.15.121:6813/627590186","heartbeat_back_addr":"172.21.15.121:6817/627590186","heartbeat_front_addr":"172.21.15.121:6815/627590186","state":["exists","up"]},{"osd":2,"uuid":"e3b6290b-ef26-4cda-96e2-8bb5649219a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":58,"up_from":62,"up_thru":77,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2092662684},{"type":"v1","addr":"172.21.15.121:6819","nonce":2092662684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2092662684},{"type":"v1","addr":"172.21.15.121:6821","nonce":2092662684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2092662684},{"type":"v1","addr":"172.21.15.121:6825","nonce":2092662684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2092662684},{"type":"v1","addr":"172.21.15.121:6823","nonce":2092662684}]},"public_addr":"172.21.15.121:6819/2092662684","cluster_addr":"172.21.15.121:6821/2092662684","heartbeat_back_addr":"172.21.15.121:6825/2092662684","heartbeat_front_addr":"172.21.15.121:6823/2092662684","state":["exists","up"]},{"osd":3,"uuid":"77fa8da5-d202-45f1-b6fa-5af969e9300c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":63,"up_from":67,"up_thru":77,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":83582134},{"type":"v1","addr":"172.21.15.188:6801","nonce":83582134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":83582134},{"type":"v1","addr":"172.21.15.188:6803","nonce":83582134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":83582134},{"type":"v1","addr":"172.21.15.188:6807","nonce":83582134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":83582134},{"type":"v1","addr":"172.21.15.188:6805","nonce":83582134}]},"public_addr":"172.21.15.188:6801/83582134","cluster_addr":"172.21.15.188:6803/83582134","heartbeat_back_addr":"172.21.15.188:6807/83582134","heartbeat_front_addr":"172.21.15.188:6805/83582134","state":["exists","up"]},{"osd":4,"uuid":"abf81f28-7473-45ef-a716-5ef16e038b6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":68,"up_from":72,"up_thru":77,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3966070828},{"type":"v1","addr":"172.21.15.188:6809","nonce":3966070828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3966070828},{"type":"v1","addr":"172.21.15.188:6811","nonce":3966070828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3966070828},{"type":"v1","addr":"172.21.15.188:6815","nonce":3966070828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3966070828},{"type":"v1","addr":"172.21.15.188:6813","nonce":3966070828}]},"public_addr":"172.21.15.188:6809/3966070828","cluster_addr":"172.21.15.188:6811/3966070828","heartbeat_back_addr":"172.21.15.188:6815/3966070828","heartbeat_front_addr":"172.21.15.188:6813/3966070828","state":["exists","up"]},{"osd":5,"uuid":"700a347b-e303-447d-bf03-d6677955d140","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":73,"up_from":77,"up_thru":77,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3805501159},{"type":"v1","addr":"172.21.15.188:6817","nonce":3805501159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3805501159},{"type":"v1","addr":"172.21.15.188:6819","nonce":3805501159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3805501159},{"type":"v1","addr":"172.21.15.188:6823","nonce":3805501159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3805501159},{"type":"v1","addr":"172.21.15.188:6821","nonce":3805501159}]},"public_addr":"172.21.15.188:6817/3805501159","cluster_addr":"172.21.15.188:6819/3805501159","heartbeat_back_addr":"172.21.15.188:6823/3805501159","heartbeat_front_addr":"172.21.15.188:6821/3805501159","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:19:48.038490+0000","dead_epoch":48},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:19.269394+0000","dead_epoch":53},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:20:51.162437+0000","dead_epoch":58},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:18.203341+0000","dead_epoch":63},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:21:46.805653+0000","dead_epoch":68},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-04-05T21:22:15.296548+0000","dead_epoch":73}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.188:6825/213881583":"2024-04-06T21:34:37.155434+0000","172.21.15.188:6824/213881583":"2024-04-06T21:34:37.155434+0000","172.21.15.121:6829/3760584056":"2024-04-06T21:34:37.155420+0000","172.21.15.121:6828/3760584056":"2024-04-06T21:34:37.155420+0000","172.21.15.121:0/566857212":"2024-04-06T21:26:08.566250+0000","172.21.15.121:6801/1237774071":"2024-04-06T21:18:24.954707+0000","172.21.15.121:6800/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/1364585923":"2024-04-06T21:16:32.951887+0000","172.21.15.121:6801/2735531747":"2024-04-06T21:26:08.566250+0000","172.21.15.121:0/4016372676":"2024-04-06T21:26:08.566250+0000","172.21.15.188:6829/2212394428":"2024-04-06T21:27:30.099010+0000","172.21.15.188:6828/2212394428":"2024-04-06T21:27:30.099010+0000","172.21.15.121:0/42058318":"2024-04-06T21:17:08.626094+0000","172.21.15.188:0/2907182826":"2024-04-06T21:27:30.099010+0000","172.21.15.121:0/341396693":"2024-04-06T21:17:08.626094+0000","172.21.15.121:6827/1615448980":"2024-04-06T21:24:00.251505+0000","172.21.15.188:0/3491833383":"2024-04-06T21:27:30.099010+0000","172.21.15.121:6800/1237774071":"2024-04-06T21:18:24.954707+0000","172.21.15.121:0/1060423936":"2024-04-06T21:26:08.566250+0000","172.21.15.121:6800/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/3762940011":"2024-04-06T21:18:24.954707+0000","172.21.15.188:0/1842750729":"2024-04-06T21:27:30.099010+0000","172.21.15.121:6800/2735531747":"2024-04-06T21:26:08.566250+0000","172.21.15.121:0/2214582362":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/2536941662":"2024-04-06T21:16:32.951887+0000","172.21.15.121:0/2383002272":"2024-04-06T21:16:32.951887+0000","172.21.15.121:6801/169707344":"2024-04-06T21:16:32.951887+0000","172.21.15.188:0/6988375":"2024-04-06T21:27:30.099010+0000","172.21.15.121:0/3247654758":"2024-04-06T21:18:24.954707+0000","172.21.15.188:0/1621914365":"2024-04-06T21:27:30.099010+0000","172.21.15.121:6826/1615448980":"2024-04-06T21:24:00.251505+0000","172.21.15.188:0/4116965307":"2024-04-06T21:27:30.099010+0000","172.21.15.121:6801/1004235250":"2024-04-06T21:17:08.626094+0000","172.21.15.121:0/3235459786":"2024-04-06T21:18:24.954707+0000","172.21.15.121:0/846460013":"2024-04-06T21:26:08.566250+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-05T22:07:52.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.070+0000 7fc639499700 1 -- 172.21.15.121:0/216271216 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc61c06cda0 msgr2=0x7fc61c06f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:52.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.070+0000 7fc639499700 1 --2- 172.21.15.121:0/216271216 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc61c06cda0 0x7fc61c06f260 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc628006010 tx=0x7fc62801a040 comp rx=0 tx=0).stop 2024-04-05T22:07:52.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.070+0000 7fc639499700 1 -- 172.21.15.121:0/216271216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6341059a0 msgr2=0x7fc634196b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:52.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.070+0000 7fc639499700 1 --2- 172.21.15.121:0/216271216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6341059a0 0x7fc634196b70 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fc62400d8d0 tx=0x7fc62400dbe0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.071+0000 7fc639499700 1 -- 172.21.15.121:0/216271216 shutdown_connections 2024-04-05T22:07:52.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.071+0000 7fc639499700 1 --2- 172.21.15.121:0/216271216 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fc61c06cda0 0x7fc61c06f260 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.071+0000 7fc639499700 1 --2- 172.21.15.121:0/216271216 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fc6340ff990 0x7fc634196630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.071+0000 7fc639499700 1 --2- 172.21.15.121:0/216271216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6341059a0 0x7fc634196b70 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.071+0000 7fc639499700 1 -- 172.21.15.121:0/216271216 >> 172.21.15.121:0/216271216 conn(0x7fc6340fb480 msgr2=0x7fc634069060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:52.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.071+0000 7fc639499700 1 -- 172.21.15.121:0/216271216 shutdown_connections 2024-04-05T22:07:52.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.071+0000 7fc639499700 1 -- 172.21.15.121:0/216271216 wait complete. 2024-04-05T22:07:52.083 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-05T21:20:53.291734+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-05T21:23:42.376376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-05T21:23:42.658469+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-05T22:07:52.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-05T22:07:52.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.207+0000 7f0ee0551700 1 -- 172.21.15.121:0/1413089167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8106b90 msgr2=0x7f0ed8106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:52.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.207+0000 7f0ee0551700 1 --2- 172.21.15.121:0/1413089167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8106b90 0x7f0ed8106f70 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8009910 tx=0x7f0ec8009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:52.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.207+0000 7f0ee0551700 1 -- 172.21.15.121:0/1413089167 shutdown_connections 2024-04-05T22:07:52.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.207+0000 7f0ee0551700 1 --2- 172.21.15.121:0/1413089167 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ed8067da0 0x7f0ed8068220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.207+0000 7f0ee0551700 1 --2- 172.21.15.121:0/1413089167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8106b90 0x7f0ed8106f70 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.207+0000 7f0ee0551700 1 -- 172.21.15.121:0/1413089167 >> 172.21.15.121:0/1413089167 conn(0x7f0ed80753d0 msgr2=0x7f0ed80757e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:52.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.207+0000 7f0ee0551700 1 -- 172.21.15.121:0/1413089167 shutdown_connections 2024-04-05T22:07:52.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.207+0000 7f0ee0551700 1 -- 172.21.15.121:0/1413089167 wait complete. 2024-04-05T22:07:52.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.208+0000 7f0ee0551700 1 Processor -- start 2024-04-05T22:07:52.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.208+0000 7f0ee0551700 1 -- start start 2024-04-05T22:07:52.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.208+0000 7f0ee0551700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8067da0 0x7f0ed8196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:52.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.208+0000 7f0ee0551700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ed8106b90 0x7f0ed8196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:52.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.208+0000 7f0ee0551700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed8197280 con 0x7f0ed8067da0 2024-04-05T22:07:52.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.208+0000 7f0ee0551700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed819b000 con 0x7f0ed8106b90 2024-04-05T22:07:52.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.208+0000 7f0ede2ed700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8067da0 0x7f0ed8196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:52.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.208+0000 7f0ede2ed700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8067da0 0x7f0ed8196660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37140/0 (socket says 172.21.15.121:37140) 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.208+0000 7f0ede2ed700 1 -- 172.21.15.121:0/1691603329 learned_addr learned my addr 172.21.15.121:0/1691603329 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.208+0000 7f0ede2ed700 1 -- 172.21.15.121:0/1691603329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ed8106b90 msgr2=0x7f0ed8196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.208+0000 7f0ede2ed700 1 --2- 172.21.15.121:0/1691603329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ed8106b90 0x7f0ed8196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.208+0000 7f0ede2ed700 1 -- 172.21.15.121:0/1691603329 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec80095f0 con 0x7f0ed8067da0 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.208+0000 7f0ede2ed700 1 --2- 172.21.15.121:0/1691603329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8067da0 0x7f0ed8196660 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8006010 tx=0x7f0ec800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.209+0000 7f0ecf7fe700 1 -- 172.21.15.121:0/1691603329 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ec801d070 con 0x7f0ed8067da0 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.209+0000 7f0ecf7fe700 1 -- 172.21.15.121:0/1691603329 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ec800f460 con 0x7f0ed8067da0 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.209+0000 7f0ee0551700 1 -- 172.21.15.121:0/1691603329 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed819b280 con 0x7f0ed8067da0 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.209+0000 7f0ecf7fe700 1 -- 172.21.15.121:0/1691603329 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ec8021640 con 0x7f0ed8067da0 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.209+0000 7f0ee0551700 1 -- 172.21.15.121:0/1691603329 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed819b800 con 0x7f0ed8067da0 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.210+0000 7f0ee0551700 1 -- 172.21.15.121:0/1691603329 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed8109040 con 0x7f0ed8067da0 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.210+0000 7f0ecf7fe700 1 -- 172.21.15.121:0/1691603329 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f0ec800f9e0 con 0x7f0ed8067da0 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.210+0000 7f0ecf7fe700 1 --2- 172.21.15.121:0/1691603329 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ec406cce0 0x7f0ec406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:52.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.211+0000 7f0eddaec700 1 --2- 172.21.15.121:0/1691603329 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ec406cce0 0x7f0ec406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:52.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.211+0000 7f0ecf7fe700 1 -- 172.21.15.121:0/1691603329 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f0ec808d830 con 0x7f0ed8067da0 2024-04-05T22:07:52.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.211+0000 7f0eddaec700 1 --2- 172.21.15.121:0/1691603329 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ec406cce0 0x7f0ec406f1a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8197c70 tx=0x7f0ed4006d20 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:52.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.215+0000 7f0ecf7fe700 1 -- 172.21.15.121:0/1691603329 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f0ec80578a0 con 0x7f0ed8067da0 2024-04-05T22:07:52.237 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:51 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2515150421' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T22:07:52.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:51 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2256553096' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-05T22:07:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:51 smithi188 ceph-mon[107793]: pgmap v1251: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:51 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/294919265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T22:07:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:51 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2515150421' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T22:07:52.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:51 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2256553096' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-05T22:07:52.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.410+0000 7f0ee0551700 1 -- 172.21.15.121:0/1691603329 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0ed819ba20 con 0x7f0ed8067da0 2024-04-05T22:07:52.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.411+0000 7f0ecf7fe700 1 -- 172.21.15.121:0/1691603329 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ec8057020 con 0x7f0ed8067da0 2024-04-05T22:07:52.411 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2024-04-05T22:07:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.413+0000 7f0ee0551700 1 -- 172.21.15.121:0/1691603329 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ec406cce0 msgr2=0x7f0ec406f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.413+0000 7f0ee0551700 1 --2- 172.21.15.121:0/1691603329 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ec406cce0 0x7f0ec406f1a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8197c70 tx=0x7f0ed4006d20 comp rx=0 tx=0).stop 2024-04-05T22:07:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.413+0000 7f0ee0551700 1 -- 172.21.15.121:0/1691603329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8067da0 msgr2=0x7f0ed8196660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.413+0000 7f0ee0551700 1 --2- 172.21.15.121:0/1691603329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8067da0 0x7f0ed8196660 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8006010 tx=0x7f0ec800be20 comp rx=0 tx=0).stop 2024-04-05T22:07:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.413+0000 7f0ee0551700 1 -- 172.21.15.121:0/1691603329 shutdown_connections 2024-04-05T22:07:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.413+0000 7f0ee0551700 1 --2- 172.21.15.121:0/1691603329 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f0ec406cce0 0x7f0ec406f1a0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.413+0000 7f0ee0551700 1 --2- 172.21.15.121:0/1691603329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ed8067da0 0x7f0ed8196660 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.413+0000 7f0ee0551700 1 --2- 172.21.15.121:0/1691603329 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f0ed8106b90 0x7f0ed8196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.413+0000 7f0ee0551700 1 -- 172.21.15.121:0/1691603329 >> 172.21.15.121:0/1691603329 conn(0x7f0ed80753d0 msgr2=0x7f0ed80feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.413+0000 7f0ee0551700 1 -- 172.21.15.121:0/1691603329 shutdown_connections 2024-04-05T22:07:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.413+0000 7f0ee0551700 1 -- 172.21.15.121:0/1691603329 wait complete. 2024-04-05T22:07:52.425 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-05T22:07:52.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.555+0000 7fce1384c700 1 -- 172.21.15.121:0/954061201 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce0c100f00 msgr2=0x7fce0c104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:52.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.555+0000 7fce1384c700 1 --2- 172.21.15.121:0/954061201 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce0c100f00 0x7fce0c104600 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fce08009910 tx=0x7fce08009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:52.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.555+0000 7fce1384c700 1 -- 172.21.15.121:0/954061201 shutdown_connections 2024-04-05T22:07:52.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.555+0000 7fce1384c700 1 --2- 172.21.15.121:0/954061201 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce0c100f00 0x7fce0c104600 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.555+0000 7fce1384c700 1 --2- 172.21.15.121:0/954061201 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce0c100550 0x7fce0c100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.555+0000 7fce1384c700 1 -- 172.21.15.121:0/954061201 >> 172.21.15.121:0/954061201 conn(0x7fce0c074c10 msgr2=0x7fce0c075020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:52.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.556+0000 7fce1384c700 1 -- 172.21.15.121:0/954061201 shutdown_connections 2024-04-05T22:07:52.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.556+0000 7fce1384c700 1 -- 172.21.15.121:0/954061201 wait complete. 2024-04-05T22:07:52.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.556+0000 7fce1384c700 1 Processor -- start 2024-04-05T22:07:52.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.556+0000 7fce1384c700 1 -- start start 2024-04-05T22:07:52.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.556+0000 7fce1384c700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce0c100550 0x7fce0c197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:52.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.556+0000 7fce1384c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce0c197aa0 0x7fce0c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:52.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.556+0000 7fce1384c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c1a0660 con 0x7fce0c197aa0 2024-04-05T22:07:52.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.556+0000 7fce1384c700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c1a07a0 con 0x7fce0c100550 2024-04-05T22:07:52.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.556+0000 7fce10de7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce0c197aa0 0x7fce0c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:52.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.556+0000 7fce10de7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce0c197aa0 0x7fce0c197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37166/0 (socket says 172.21.15.121:37166) 2024-04-05T22:07:52.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.556+0000 7fce10de7700 1 -- 172.21.15.121:0/2904506362 learned_addr learned my addr 172.21.15.121:0/2904506362 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:52.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.557+0000 7fce115e8700 1 --2- 172.21.15.121:0/2904506362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce0c100550 0x7fce0c197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:52.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.557+0000 7fce10de7700 1 -- 172.21.15.121:0/2904506362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce0c100550 msgr2=0x7fce0c197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:52.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.557+0000 7fce10de7700 1 --2- 172.21.15.121:0/2904506362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce0c100550 0x7fce0c197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.557+0000 7fce10de7700 1 -- 172.21.15.121:0/2904506362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce080095f0 con 0x7fce0c197aa0 2024-04-05T22:07:52.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.557+0000 7fce10de7700 1 --2- 172.21.15.121:0/2904506362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce0c197aa0 0x7fce0c197f20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fce0800bd70 tx=0x7fce0800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:52.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.557+0000 7fce027fc700 1 -- 172.21.15.121:0/2904506362 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce0801d070 con 0x7fce0c197aa0 2024-04-05T22:07:52.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.557+0000 7fce027fc700 1 -- 172.21.15.121:0/2904506362 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce0800f460 con 0x7fce0c197aa0 2024-04-05T22:07:52.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.557+0000 7fce1384c700 1 -- 172.21.15.121:0/2904506362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0c1a09a0 con 0x7fce0c197aa0 2024-04-05T22:07:52.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.557+0000 7fce027fc700 1 -- 172.21.15.121:0/2904506362 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce08021640 con 0x7fce0c197aa0 2024-04-05T22:07:52.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.557+0000 7fce1384c700 1 -- 172.21.15.121:0/2904506362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0c1a0e70 con 0x7fce0c197aa0 2024-04-05T22:07:52.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.558+0000 7fce1384c700 1 -- 172.21.15.121:0/2904506362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0c108040 con 0x7fce0c197aa0 2024-04-05T22:07:52.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.558+0000 7fce027fc700 1 -- 172.21.15.121:0/2904506362 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7fce0800f9e0 con 0x7fce0c197aa0 2024-04-05T22:07:52.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.559+0000 7fce027fc700 1 --2- 172.21.15.121:0/2904506362 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcdf806cce0 0x7fcdf806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:52.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.559+0000 7fce027fc700 1 -- 172.21.15.121:0/2904506362 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7fce0808d6b0 con 0x7fce0c197aa0 2024-04-05T22:07:52.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.559+0000 7fce115e8700 1 --2- 172.21.15.121:0/2904506362 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcdf806cce0 0x7fcdf806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:52.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.559+0000 7fce115e8700 1 --2- 172.21.15.121:0/2904506362 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcdf806cce0 0x7fcdf806f1a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc004630 tx=0x7fcdfc00b3c0 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:52.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.564+0000 7fce027fc700 1 -- 172.21.15.121:0/2904506362 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7fce08057720 con 0x7fce0c197aa0 2024-04-05T22:07:52.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.757+0000 7fce1384c700 1 -- 172.21.15.121:0/2904506362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fce0c04ea90 con 0x7fce0c197aa0 2024-04-05T22:07:52.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.757+0000 7fce027fc700 1 -- 172.21.15.121:0/2904506362 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7fce0805ad40 con 0x7fce0c197aa0 2024-04-05T22:07:52.758 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 32 2024-04-05T22:07:52.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.759+0000 7fce1384c700 1 -- 172.21.15.121:0/2904506362 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcdf806cce0 msgr2=0x7fcdf806f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:52.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.759+0000 7fce1384c700 1 --2- 172.21.15.121:0/2904506362 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcdf806cce0 0x7fcdf806f1a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc004630 tx=0x7fcdfc00b3c0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.759+0000 7fce1384c700 1 -- 172.21.15.121:0/2904506362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce0c197aa0 msgr2=0x7fce0c197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:52.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.759+0000 7fce1384c700 1 --2- 172.21.15.121:0/2904506362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce0c197aa0 0x7fce0c197f20 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fce0800bd70 tx=0x7fce0800bda0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.760+0000 7fce1384c700 1 -- 172.21.15.121:0/2904506362 shutdown_connections 2024-04-05T22:07:52.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.760+0000 7fce1384c700 1 --2- 172.21.15.121:0/2904506362 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7fcdf806cce0 0x7fcdf806f1a0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.760+0000 7fce1384c700 1 --2- 172.21.15.121:0/2904506362 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7fce0c100550 0x7fce0c197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.760+0000 7fce1384c700 1 --2- 172.21.15.121:0/2904506362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce0c197aa0 0x7fce0c197f20 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.760+0000 7fce1384c700 1 -- 172.21.15.121:0/2904506362 >> 172.21.15.121:0/2904506362 conn(0x7fce0c074c10 msgr2=0x7fce0c0fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:52.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.760+0000 7fce1384c700 1 -- 172.21.15.121:0/2904506362 shutdown_connections 2024-04-05T22:07:52.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.760+0000 7fce1384c700 1 -- 172.21.15.121:0/2904506362 wait complete. 2024-04-05T22:07:52.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-05T22:07:52.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.897+0000 7f400e9ad700 1 -- 172.21.15.121:0/4255664162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4008105980 msgr2=0x7f4008105d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:52.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.897+0000 7f400e9ad700 1 --2- 172.21.15.121:0/4255664162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4008105980 0x7f4008105d60 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009910 tx=0x7f3ffc009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:52.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.898+0000 7f400e9ad700 1 -- 172.21.15.121:0/4255664162 shutdown_connections 2024-04-05T22:07:52.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.898+0000 7f400e9ad700 1 --2- 172.21.15.121:0/4255664162 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f40080ff970 0x7f40080ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.898+0000 7f400e9ad700 1 --2- 172.21.15.121:0/4255664162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4008105980 0x7f4008105d60 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.898+0000 7f400e9ad700 1 -- 172.21.15.121:0/4255664162 >> 172.21.15.121:0/4255664162 conn(0x7f40080fb420 msgr2=0x7f40080fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:52.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.898+0000 7f400e9ad700 1 -- 172.21.15.121:0/4255664162 shutdown_connections 2024-04-05T22:07:52.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.898+0000 7f400e9ad700 1 -- 172.21.15.121:0/4255664162 wait complete. 2024-04-05T22:07:52.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.898+0000 7f400e9ad700 1 Processor -- start 2024-04-05T22:07:52.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.898+0000 7f400e9ad700 1 -- start start 2024-04-05T22:07:52.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f400e9ad700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40080ff970 0x7f4008195590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:52.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f400e9ad700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4008105980 0x7f4008195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:52.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f400e9ad700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40081961b0 con 0x7f40080ff970 2024-04-05T22:07:52.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f400e9ad700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4008199f30 con 0x7f4008105980 2024-04-05T22:07:52.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f4007fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40080ff970 0x7f4008195590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:52.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f4007fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40080ff970 0x7f4008195590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37192/0 (socket says 172.21.15.121:37192) 2024-04-05T22:07:52.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f4007fff700 1 -- 172.21.15.121:0/412705393 learned_addr learned my addr 172.21.15.121:0/412705393 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:52.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f40077fe700 1 --2- 172.21.15.121:0/412705393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4008105980 0x7f4008195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:52.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f4007fff700 1 -- 172.21.15.121:0/412705393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4008105980 msgr2=0x7f4008195ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:52.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f4007fff700 1 --2- 172.21.15.121:0/412705393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4008105980 0x7f4008195ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:52.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f4007fff700 1 -- 172.21.15.121:0/412705393 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc0095f0 con 0x7f40080ff970 2024-04-05T22:07:52.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f40077fe700 1 --2- 172.21.15.121:0/412705393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4008105980 0x7f4008195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:07:52.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f4007fff700 1 --2- 172.21.15.121:0/412705393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40080ff970 0x7f4008195590 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc006010 tx=0x7f3ffc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:52.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f40057fa700 1 -- 172.21.15.121:0/412705393 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ffc01d070 con 0x7f40080ff970 2024-04-05T22:07:52.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f40057fa700 1 -- 172.21.15.121:0/412705393 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ffc00f460 con 0x7f40080ff970 2024-04-05T22:07:52.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f400e9ad700 1 -- 172.21.15.121:0/412705393 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400819a1b0 con 0x7f40080ff970 2024-04-05T22:07:52.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f40057fa700 1 -- 172.21.15.121:0/412705393 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ffc021620 con 0x7f40080ff970 2024-04-05T22:07:52.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.899+0000 7f400e9ad700 1 -- 172.21.15.121:0/412705393 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400819a6b0 con 0x7f40080ff970 2024-04-05T22:07:52.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.900+0000 7f400e9ad700 1 -- 172.21.15.121:0/412705393 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4008107f70 con 0x7f40080ff970 2024-04-05T22:07:52.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.901+0000 7f40057fa700 1 -- 172.21.15.121:0/412705393 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f3ffc00fa40 con 0x7f40080ff970 2024-04-05T22:07:52.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.901+0000 7f40057fa700 1 --2- 172.21.15.121:0/412705393 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ff006cce0 0x7f3ff006f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:52.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.902+0000 7f40057fa700 1 -- 172.21.15.121:0/412705393 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f3ffc08dc90 con 0x7f40080ff970 2024-04-05T22:07:52.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.902+0000 7f40077fe700 1 --2- 172.21.15.121:0/412705393 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ff006cce0 0x7f3ff006f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:52.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.902+0000 7f40077fe700 1 --2- 172.21.15.121:0/412705393 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ff006cce0 0x7f3ff006f1a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8005d90 tx=0x7f3ff8005d00 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:52.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:52.905+0000 7f40057fa700 1 -- 172.21.15.121:0/412705393 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f3ffc057ce0 con 0x7f40080ff970 2024-04-05T22:07:53.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.097+0000 7f400e9ad700 1 -- 172.21.15.121:0/412705393 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f400804ea90 con 0x7f40080ff970 2024-04-05T22:07:53.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.097+0000 7f40057fa700 1 -- 172.21.15.121:0/412705393 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v84) v1 ==== 107+0+11 (secure 0 0 0) 0x7f3ffc05b300 con 0x7f40080ff970 2024-04-05T22:07:53.097 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 32 2024-04-05T22:07:53.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.099+0000 7f400e9ad700 1 -- 172.21.15.121:0/412705393 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ff006cce0 msgr2=0x7f3ff006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:53.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.099+0000 7f400e9ad700 1 --2- 172.21.15.121:0/412705393 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ff006cce0 0x7f3ff006f1a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8005d90 tx=0x7f3ff8005d00 comp rx=0 tx=0).stop 2024-04-05T22:07:53.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.099+0000 7f400e9ad700 1 -- 172.21.15.121:0/412705393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40080ff970 msgr2=0x7f4008195590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:53.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.099+0000 7f400e9ad700 1 --2- 172.21.15.121:0/412705393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40080ff970 0x7f4008195590 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc006010 tx=0x7f3ffc00be20 comp rx=0 tx=0).stop 2024-04-05T22:07:53.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.099+0000 7f400e9ad700 1 -- 172.21.15.121:0/412705393 shutdown_connections 2024-04-05T22:07:53.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.099+0000 7f400e9ad700 1 --2- 172.21.15.121:0/412705393 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f3ff006cce0 0x7f3ff006f1a0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:53.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.099+0000 7f400e9ad700 1 --2- 172.21.15.121:0/412705393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40080ff970 0x7f4008195590 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:53.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.099+0000 7f400e9ad700 1 --2- 172.21.15.121:0/412705393 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f4008105980 0x7f4008195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:53.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.099+0000 7f400e9ad700 1 -- 172.21.15.121:0/412705393 >> 172.21.15.121:0/412705393 conn(0x7f40080fb420 msgr2=0x7f40080fc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:53.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.100+0000 7f400e9ad700 1 -- 172.21.15.121:0/412705393 shutdown_connections 2024-04-05T22:07:53.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.100+0000 7f400e9ad700 1 -- 172.21.15.121:0/412705393 wait complete. 2024-04-05T22:07:53.110 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:52 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/216271216' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T22:07:53.110 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:52 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/1691603329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T22:07:53.110 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:52 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/2904506362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T22:07:53.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-05T22:07:53.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.234+0000 7f17ff514700 1 -- 172.21.15.121:0/3319883396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f8105990 msgr2=0x7f17f8105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:53.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.234+0000 7f17ff514700 1 --2- 172.21.15.121:0/3319883396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f8105990 0x7f17f8105d70 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009910 tx=0x7f17e8009c20 comp rx=0 tx=0).stop 2024-04-05T22:07:53.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.234+0000 7f17ff514700 1 -- 172.21.15.121:0/3319883396 shutdown_connections 2024-04-05T22:07:53.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.234+0000 7f17ff514700 1 --2- 172.21.15.121:0/3319883396 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17f80ffac0 0x7f17f80fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:53.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.234+0000 7f17ff514700 1 --2- 172.21.15.121:0/3319883396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f8105990 0x7f17f8105d70 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:53.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.234+0000 7f17ff514700 1 -- 172.21.15.121:0/3319883396 >> 172.21.15.121:0/3319883396 conn(0x7f17f8075400 msgr2=0x7f17f8075810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:53.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.234+0000 7f17ff514700 1 -- 172.21.15.121:0/3319883396 shutdown_connections 2024-04-05T22:07:53.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.234+0000 7f17ff514700 1 -- 172.21.15.121:0/3319883396 wait complete. 2024-04-05T22:07:53.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.234+0000 7f17ff514700 1 Processor -- start 2024-04-05T22:07:53.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.234+0000 7f17ff514700 1 -- start start 2024-04-05T22:07:53.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17ff514700 1 --2- >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17f80ffac0 0x7f17f81954f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:53.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17ff514700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f8105990 0x7f17f8195a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:53.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17ff514700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8196110 con 0x7f17f8105990 2024-04-05T22:07:53.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17ff514700 1 -- --> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8199e90 con 0x7f17f80ffac0 2024-04-05T22:07:53.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17fcaaf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f8105990 0x7f17f8195a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:53.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17fcaaf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f8105990 0x7f17f8195a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37204/0 (socket says 172.21.15.121:37204) 2024-04-05T22:07:53.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17fcaaf700 1 -- 172.21.15.121:0/85208951 learned_addr learned my addr 172.21.15.121:0/85208951 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-05T22:07:53.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17fd2b0700 1 --2- 172.21.15.121:0/85208951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17f80ffac0 0x7f17f81954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:53.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17fcaaf700 1 -- 172.21.15.121:0/85208951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17f80ffac0 msgr2=0x7f17f81954f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:53.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17fcaaf700 1 --2- 172.21.15.121:0/85208951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17f80ffac0 0x7f17f81954f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:53.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17fcaaf700 1 -- 172.21.15.121:0/85208951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e80095f0 con 0x7f17f8105990 2024-04-05T22:07:53.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17fd2b0700 1 --2- 172.21.15.121:0/85208951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17f80ffac0 0x7f17f81954f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-05T22:07:53.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17fcaaf700 1 --2- 172.21.15.121:0/85208951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f8105990 0x7f17f8195a30 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f17f400c8a0 tx=0x7f17f400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:53.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17ee7fc700 1 -- 172.21.15.121:0/85208951 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17f400cea0 con 0x7f17f8105990 2024-04-05T22:07:53.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17ee7fc700 1 -- 172.21.15.121:0/85208951 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17f4004830 con 0x7f17f8105990 2024-04-05T22:07:53.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17ff514700 1 -- 172.21.15.121:0/85208951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f819a170 con 0x7f17f8105990 2024-04-05T22:07:53.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17ee7fc700 1 -- 172.21.15.121:0/85208951 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17f4005500 con 0x7f17f8105990 2024-04-05T22:07:53.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.235+0000 7f17ff514700 1 -- 172.21.15.121:0/85208951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f819a6d0 con 0x7f17f8105990 2024-04-05T22:07:53.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.236+0000 7f17ff514700 1 -- 172.21.15.121:0/85208951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f8107f30 con 0x7f17f8105990 2024-04-05T22:07:53.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.237+0000 7f17ee7fc700 1 -- 172.21.15.121:0/85208951 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 33) v1 ==== 90852+0+0 (secure 0 0 0) 0x7f17f40049a0 con 0x7f17f8105990 2024-04-05T22:07:53.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.237+0000 7f17ee7fc700 1 --2- 172.21.15.121:0/85208951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f17e406cda0 0x7f17e406f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-05T22:07:53.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.237+0000 7f17ee7fc700 1 -- 172.21.15.121:0/85208951 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6328+0+0 (secure 0 0 0) 0x7f17f408c2b0 con 0x7f17f8105990 2024-04-05T22:07:53.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.237+0000 7f17fd2b0700 1 --2- 172.21.15.121:0/85208951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f17e406cda0 0x7f17e406f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-05T22:07:53.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.238+0000 7f17fd2b0700 1 --2- 172.21.15.121:0/85208951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f17e406cda0 0x7f17e406f260 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f17e800b5c0 tx=0x7f17e8005960 comp rx=0 tx=0).ready entity=mgr.14656 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-05T22:07:53.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.242+0000 7f17ee7fc700 1 -- 172.21.15.121:0/85208951 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182936 (secure 0 0 0) 0x7f17f4056320 con 0x7f17f8105990 2024-04-05T22:07:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:52 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/216271216' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-05T22:07:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:52 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/1691603329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-05T22:07:53.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:52 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/2904506362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-05T22:07:53.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.449+0000 7f17ff514700 1 -- 172.21.15.121:0/85208951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f17f804ea90 con 0x7f17f8105990 2024-04-05T22:07:53.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.450+0000 7f17ee7fc700 1 -- 172.21.15.121:0/85208951 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 35 entries v84) v1 ==== 81+0+2101 (secure 0 0 0) 0x7f17f4059940 con 0x7f17f8105990 2024-04-05T22:07:53.450 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:6825/213881583 2024-04-06T21:34:37.155434+0000 2024-04-05T22:07:53.450 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:6824/213881583 2024-04-06T21:34:37.155434+0000 2024-04-05T22:07:53.450 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6829/3760584056 2024-04-06T21:34:37.155420+0000 2024-04-05T22:07:53.450 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6828/3760584056 2024-04-06T21:34:37.155420+0000 2024-04-05T22:07:53.450 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/566857212 2024-04-06T21:26:08.566250+0000 2024-04-05T22:07:53.450 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/1237774071 2024-04-06T21:18:24.954707+0000 2024-04-05T22:07:53.450 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/1004235250 2024-04-06T21:17:08.626094+0000 2024-04-05T22:07:53.450 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1364585923 2024-04-06T21:16:32.951887+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/2735531747 2024-04-06T21:26:08.566250+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/4016372676 2024-04-06T21:26:08.566250+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:6829/2212394428 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:6828/2212394428 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/42058318 2024-04-06T21:17:08.626094+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:0/2907182826 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/341396693 2024-04-06T21:17:08.626094+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6827/1615448980 2024-04-06T21:24:00.251505+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:0/3491833383 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/1237774071 2024-04-06T21:18:24.954707+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1060423936 2024-04-06T21:26:08.566250+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/169707344 2024-04-06T21:16:32.951887+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/3762940011 2024-04-06T21:18:24.954707+0000 2024-04-05T22:07:53.451 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:0/1842750729 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:53.452 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/2735531747 2024-04-06T21:26:08.566250+0000 2024-04-05T22:07:53.452 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/2214582362 2024-04-06T21:17:08.626094+0000 2024-04-05T22:07:53.452 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/2536941662 2024-04-06T21:16:32.951887+0000 2024-04-05T22:07:53.452 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/2383002272 2024-04-06T21:16:32.951887+0000 2024-04-05T22:07:53.452 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/169707344 2024-04-06T21:16:32.951887+0000 2024-04-05T22:07:53.452 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:0/6988375 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:53.452 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/3247654758 2024-04-06T21:18:24.954707+0000 2024-04-05T22:07:53.452 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:0/1621914365 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:53.452 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6826/1615448980 2024-04-06T21:24:00.251505+0000 2024-04-05T22:07:53.452 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.188:0/4116965307 2024-04-06T21:27:30.099010+0000 2024-04-05T22:07:53.452 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/1004235250 2024-04-06T21:17:08.626094+0000 2024-04-05T22:07:53.452 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/3235459786 2024-04-06T21:18:24.954707+0000 2024-04-05T22:07:53.452 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/846460013 2024-04-06T21:26:08.566250+0000 2024-04-05T22:07:53.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.452+0000 7f17ff514700 1 -- 172.21.15.121:0/85208951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f17e406cda0 msgr2=0x7f17e406f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:53.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.452+0000 7f17ff514700 1 --2- 172.21.15.121:0/85208951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f17e406cda0 0x7f17e406f260 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f17e800b5c0 tx=0x7f17e8005960 comp rx=0 tx=0).stop 2024-04-05T22:07:53.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.452+0000 7f17ff514700 1 -- 172.21.15.121:0/85208951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f8105990 msgr2=0x7f17f8195a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-05T22:07:53.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.452+0000 7f17ff514700 1 --2- 172.21.15.121:0/85208951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f8105990 0x7f17f8195a30 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f17f400c8a0 tx=0x7f17f400cc60 comp rx=0 tx=0).stop 2024-04-05T22:07:53.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.452+0000 7f17ff514700 1 -- 172.21.15.121:0/85208951 shutdown_connections 2024-04-05T22:07:53.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.452+0000 7f17ff514700 1 --2- 172.21.15.121:0/85208951 >> [v2:172.21.15.121:6800/2306077913,v1:172.21.15.121:6801/2306077913] conn(0x7f17e406cda0 0x7f17e406f260 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:53.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.452+0000 7f17ff514700 1 --2- 172.21.15.121:0/85208951 >> [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] conn(0x7f17f80ffac0 0x7f17f81954f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:53.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.452+0000 7f17ff514700 1 --2- 172.21.15.121:0/85208951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f8105990 0x7f17f8195a30 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-05T22:07:53.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.452+0000 7f17ff514700 1 -- 172.21.15.121:0/85208951 >> 172.21.15.121:0/85208951 conn(0x7f17f8075400 msgr2=0x7f17f80fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-05T22:07:53.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.452+0000 7f17ff514700 1 -- 172.21.15.121:0/85208951 shutdown_connections 2024-04-05T22:07:53.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-05T22:07:53.452+0000 7f17ff514700 1 -- 172.21.15.121:0/85208951 wait complete. 2024-04-05T22:07:53.454 INFO:teuthology.orchestra.run.smithi121.stderr:listed 35 entries 2024-04-05T22:07:53.464 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2024-04-05T22:07:53.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-05T22:07:53.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-05T22:07:53.511 INFO:teuthology.orchestra.run:waiting for 300 2024-04-05T22:07:54.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:53 smithi121 ceph-mon[145859]: pgmap v1252: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:07:54.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:53 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/412705393' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T22:07:54.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:53 smithi121 ceph-mon[145859]: from='client.? 172.21.15.121:0/85208951' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-05T22:07:54.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:53 smithi188 ceph-mon[107793]: pgmap v1252: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:07:54.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:53 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/412705393' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-05T22:07:54.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:53 smithi188 ceph-mon[107793]: from='client.? 172.21.15.121:0/85208951' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-05T22:07:56.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:55 smithi121 ceph-mon[145859]: pgmap v1253: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:07:56.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:55 smithi188 ceph-mon[107793]: pgmap v1253: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:07:58.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:57 smithi121 ceph-mon[145859]: pgmap v1254: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:07:58.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:57 smithi188 ceph-mon[107793]: pgmap v1254: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:00.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:07:59 smithi121 ceph-mon[145859]: pgmap v1255: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:00.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:07:59 smithi188 ceph-mon[107793]: pgmap v1255: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:08:01.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:08:01.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:01 smithi121 ceph-mon[145859]: pgmap v1256: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:02.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:01 smithi188 ceph-mon[107793]: pgmap v1256: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:03.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:03 smithi121 ceph-mon[145859]: pgmap v1257: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:08:03.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:03 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:08:03.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:03 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:08:03.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:03 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:08:03.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:03 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:08:03.778 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:03 smithi188 ceph-mon[107793]: pgmap v1257: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:08:03.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:03 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:08:03.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:03 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:08:03.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:03 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:08:03.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:03 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:08:05.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:05 smithi121 ceph-mon[145859]: pgmap v1258: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:08:06.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:05 smithi188 ceph-mon[107793]: pgmap v1258: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:08:07.986 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:07 smithi188 ceph-mon[107793]: pgmap v1259: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:07.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:07 smithi121 ceph-mon[145859]: pgmap v1259: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:09.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:09 smithi121 ceph-mon[145859]: pgmap v1260: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:10.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:09 smithi188 ceph-mon[107793]: pgmap v1260: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:11.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:11 smithi121 ceph-mon[145859]: pgmap v1261: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:12.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:11 smithi188 ceph-mon[107793]: pgmap v1261: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:13.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:13 smithi121 ceph-mon[145859]: pgmap v1262: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:08:14.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:13 smithi188 ceph-mon[107793]: pgmap v1262: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:08:15.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:15 smithi121 ceph-mon[145859]: pgmap v1263: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:08:15.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:08:16.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:15 smithi188 ceph-mon[107793]: pgmap v1263: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:08:16.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:08:17.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:17 smithi121 ceph-mon[145859]: pgmap v1264: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:18.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:17 smithi188 ceph-mon[107793]: pgmap v1264: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:19.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:19 smithi121 ceph-mon[145859]: pgmap v1265: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:20.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:19 smithi188 ceph-mon[107793]: pgmap v1265: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:21.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:21 smithi121 ceph-mon[145859]: pgmap v1266: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:22.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:21 smithi188 ceph-mon[107793]: pgmap v1266: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:23.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:23 smithi121 ceph-mon[145859]: pgmap v1267: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:08:24.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:23 smithi188 ceph-mon[107793]: pgmap v1267: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:08:25.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:25 smithi121 ceph-mon[145859]: pgmap v1268: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:08:26.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:25 smithi188 ceph-mon[107793]: pgmap v1268: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:08:27.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:27 smithi121 ceph-mon[145859]: pgmap v1269: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:28.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:27 smithi188 ceph-mon[107793]: pgmap v1269: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:29.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:29 smithi121 ceph-mon[145859]: pgmap v1270: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:30.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:29 smithi188 ceph-mon[107793]: pgmap v1270: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:30.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:08:31.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:08:31.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:31 smithi121 ceph-mon[145859]: pgmap v1271: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:32.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:31 smithi188 ceph-mon[107793]: pgmap v1271: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:33.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:33 smithi121 ceph-mon[145859]: pgmap v1272: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:08:34.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:33 smithi188 ceph-mon[107793]: pgmap v1272: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:08:36.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:35 smithi188 ceph-mon[107793]: pgmap v1273: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:08:36.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:35 smithi121 ceph-mon[145859]: pgmap v1273: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:08:38.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:37 smithi188 ceph-mon[107793]: pgmap v1274: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:38.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:37 smithi121 ceph-mon[145859]: pgmap v1274: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:40.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:39 smithi188 ceph-mon[107793]: pgmap v1275: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:40.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:39 smithi121 ceph-mon[145859]: pgmap v1275: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:42.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:41 smithi188 ceph-mon[107793]: pgmap v1276: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:42.178 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:41 smithi121 ceph-mon[145859]: pgmap v1276: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:44.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:43 smithi188 ceph-mon[107793]: pgmap v1277: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:08:44.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:43 smithi121 ceph-mon[145859]: pgmap v1277: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:08:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:45 smithi188 ceph-mon[107793]: pgmap v1278: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:08:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:08:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:45 smithi121 ceph-mon[145859]: pgmap v1278: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:08:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:08:48.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:47 smithi188 ceph-mon[107793]: pgmap v1279: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:48.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:47 smithi121 ceph-mon[145859]: pgmap v1279: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:50.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:49 smithi188 ceph-mon[107793]: pgmap v1280: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:50.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:49 smithi121 ceph-mon[145859]: pgmap v1280: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:52.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:51 smithi188 ceph-mon[107793]: pgmap v1281: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:52.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:51 smithi121 ceph-mon[145859]: pgmap v1281: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:54.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:53 smithi188 ceph-mon[107793]: pgmap v1282: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:08:54.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:53 smithi121 ceph-mon[145859]: pgmap v1282: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:08:56.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:55 smithi188 ceph-mon[107793]: pgmap v1283: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:08:56.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:55 smithi121 ceph-mon[145859]: pgmap v1283: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:08:58.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:57 smithi188 ceph-mon[107793]: pgmap v1284: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:08:58.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:57 smithi121 ceph-mon[145859]: pgmap v1284: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:00.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:08:59 smithi188 ceph-mon[107793]: pgmap v1285: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:00.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:08:59 smithi121 ceph-mon[145859]: pgmap v1285: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:09:01.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:09:02.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:01 smithi121 ceph-mon[145859]: pgmap v1286: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:02.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:01 smithi188 ceph-mon[107793]: pgmap v1286: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:04.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:03 smithi188 ceph-mon[107793]: pgmap v1287: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:09:04.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:03 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:09:04.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:03 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:09:04.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:03 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:09:04.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:03 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:09:04.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:03 smithi121 ceph-mon[145859]: pgmap v1287: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:09:04.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:03 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:09:04.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:03 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:09:04.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:03 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:09:04.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:03 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:09:06.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:05 smithi121 ceph-mon[145859]: pgmap v1288: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:09:06.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:05 smithi188 ceph-mon[107793]: pgmap v1288: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:09:08.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:07 smithi121 ceph-mon[145859]: pgmap v1289: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:08.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:07 smithi188 ceph-mon[107793]: pgmap v1289: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:10.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:09 smithi121 ceph-mon[145859]: pgmap v1290: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:10.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:09 smithi188 ceph-mon[107793]: pgmap v1290: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:12.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:11 smithi121 ceph-mon[145859]: pgmap v1291: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:12.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:11 smithi188 ceph-mon[107793]: pgmap v1291: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:14.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:13 smithi188 ceph-mon[107793]: pgmap v1292: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:09:14.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:13 smithi121 ceph-mon[145859]: pgmap v1292: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:09:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:15 smithi121 ceph-mon[145859]: pgmap v1293: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:09:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:09:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:15 smithi188 ceph-mon[107793]: pgmap v1293: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:09:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:09:17.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:17 smithi121 ceph-mon[145859]: pgmap v1294: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:18.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:17 smithi188 ceph-mon[107793]: pgmap v1294: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:20.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:19 smithi121 ceph-mon[145859]: pgmap v1295: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:20.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:19 smithi188 ceph-mon[107793]: pgmap v1295: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:22.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:21 smithi121 ceph-mon[145859]: pgmap v1296: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:22.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:21 smithi188 ceph-mon[107793]: pgmap v1296: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:24.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:23 smithi121 ceph-mon[145859]: pgmap v1297: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:09:24.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:23 smithi188 ceph-mon[107793]: pgmap v1297: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:09:26.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:25 smithi121 ceph-mon[145859]: pgmap v1298: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:09:26.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:25 smithi188 ceph-mon[107793]: pgmap v1298: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:09:27.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:27 smithi121 ceph-mon[145859]: pgmap v1299: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:28.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:27 smithi188 ceph-mon[107793]: pgmap v1299: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:30.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:29 smithi121 ceph-mon[145859]: pgmap v1300: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:30.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:29 smithi188 ceph-mon[107793]: pgmap v1300: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:09:31.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:09:32.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:31 smithi121 ceph-mon[145859]: pgmap v1301: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:32.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:31 smithi188 ceph-mon[107793]: pgmap v1301: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:34.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:33 smithi121 ceph-mon[145859]: pgmap v1302: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:09:34.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:33 smithi188 ceph-mon[107793]: pgmap v1302: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:09:36.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:35 smithi121 ceph-mon[145859]: pgmap v1303: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:09:36.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:35 smithi188 ceph-mon[107793]: pgmap v1303: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:09:38.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:37 smithi121 ceph-mon[145859]: pgmap v1304: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:38.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:37 smithi188 ceph-mon[107793]: pgmap v1304: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:40.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:39 smithi121 ceph-mon[145859]: pgmap v1305: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:40.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:39 smithi188 ceph-mon[107793]: pgmap v1305: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:42.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:41 smithi121 ceph-mon[145859]: pgmap v1306: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:42.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:41 smithi188 ceph-mon[107793]: pgmap v1306: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:44.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:43 smithi121 ceph-mon[145859]: pgmap v1307: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:09:44.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:43 smithi188 ceph-mon[107793]: pgmap v1307: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:09:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:45 smithi121 ceph-mon[145859]: pgmap v1308: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:09:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:09:46.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:45 smithi188 ceph-mon[107793]: pgmap v1308: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:09:46.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:09:48.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:47 smithi121 ceph-mon[145859]: pgmap v1309: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:48.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:47 smithi188 ceph-mon[107793]: pgmap v1309: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:50.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:49 smithi121 ceph-mon[145859]: pgmap v1310: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:50.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:49 smithi188 ceph-mon[107793]: pgmap v1310: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:52.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:51 smithi121 ceph-mon[145859]: pgmap v1311: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:51 smithi188 ceph-mon[107793]: pgmap v1311: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:54.237 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:53 smithi121 ceph-mon[145859]: pgmap v1312: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:09:54.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:53 smithi188 ceph-mon[107793]: pgmap v1312: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:09:56.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:55 smithi121 ceph-mon[145859]: pgmap v1313: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:09:56.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:55 smithi188 ceph-mon[107793]: pgmap v1313: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:09:58.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:57 smithi121 ceph-mon[145859]: pgmap v1314: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:09:58.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:57 smithi188 ceph-mon[107793]: pgmap v1314: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:00.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:09:59 smithi121 ceph-mon[145859]: pgmap v1315: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:00.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:09:59 smithi188 ceph-mon[107793]: pgmap v1315: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:00 smithi121 ceph-mon[145859]: overall HEALTH_OK 2024-04-05T22:10:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:10:01.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:00 smithi188 ceph-mon[107793]: overall HEALTH_OK 2024-04-05T22:10:01.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:10:02.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:01 smithi121 ceph-mon[145859]: pgmap v1316: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:02.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:01 smithi188 ceph-mon[107793]: pgmap v1316: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:04.223 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:03 smithi188 ceph-mon[107793]: pgmap v1317: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:10:04.223 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:03 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:10:04.237 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:03 smithi121 ceph-mon[145859]: pgmap v1317: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:10:04.237 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:03 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:10:05.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:04 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:10:05.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:04 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:10:05.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:04 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:10:05.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:04 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:10:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:04 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:10:05.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:04 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:10:06.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:05 smithi121 ceph-mon[145859]: pgmap v1318: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:10:06.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:05 smithi188 ceph-mon[107793]: pgmap v1318: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:10:08.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:07 smithi121 ceph-mon[145859]: pgmap v1319: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:08.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:07 smithi188 ceph-mon[107793]: pgmap v1319: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:10.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:09 smithi121 ceph-mon[145859]: pgmap v1320: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:10.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:09 smithi188 ceph-mon[107793]: pgmap v1320: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:12.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:11 smithi121 ceph-mon[145859]: pgmap v1321: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:12.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:11 smithi188 ceph-mon[107793]: pgmap v1321: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:14.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:13 smithi121 ceph-mon[145859]: pgmap v1322: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:10:14.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:13 smithi188 ceph-mon[107793]: pgmap v1322: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:10:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:15 smithi121 ceph-mon[145859]: pgmap v1323: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:10:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:10:16.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:15 smithi188 ceph-mon[107793]: pgmap v1323: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:10:16.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:10:18.233 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:17 smithi121 ceph-mon[145859]: pgmap v1324: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:18.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:17 smithi188 ceph-mon[107793]: pgmap v1324: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:20.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:19 smithi121 ceph-mon[145859]: pgmap v1325: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:20.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:19 smithi188 ceph-mon[107793]: pgmap v1325: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:22.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:21 smithi121 ceph-mon[145859]: pgmap v1326: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:22.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:21 smithi188 ceph-mon[107793]: pgmap v1326: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:24.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:23 smithi121 ceph-mon[145859]: pgmap v1327: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:10:24.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:23 smithi188 ceph-mon[107793]: pgmap v1327: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:10:26.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:25 smithi121 ceph-mon[145859]: pgmap v1328: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:10:26.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:25 smithi188 ceph-mon[107793]: pgmap v1328: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:10:28.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:27 smithi121 ceph-mon[145859]: pgmap v1329: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:28.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:27 smithi188 ceph-mon[107793]: pgmap v1329: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:30.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:29 smithi121 ceph-mon[145859]: pgmap v1330: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:30.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:29 smithi188 ceph-mon[107793]: pgmap v1330: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:10:31.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:10:32.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:31 smithi121 ceph-mon[145859]: pgmap v1331: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:32.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:31 smithi188 ceph-mon[107793]: pgmap v1331: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:34.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:33 smithi121 ceph-mon[145859]: pgmap v1332: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:10:34.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:33 smithi188 ceph-mon[107793]: pgmap v1332: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:10:36.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:35 smithi121 ceph-mon[145859]: pgmap v1333: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:10:36.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:35 smithi188 ceph-mon[107793]: pgmap v1333: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:10:38.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:37 smithi121 ceph-mon[145859]: pgmap v1334: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:38.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:37 smithi188 ceph-mon[107793]: pgmap v1334: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:40.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:39 smithi121 ceph-mon[145859]: pgmap v1335: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:40.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:39 smithi188 ceph-mon[107793]: pgmap v1335: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:42.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:41 smithi121 ceph-mon[145859]: pgmap v1336: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:42.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:41 smithi188 ceph-mon[107793]: pgmap v1336: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:43.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:43 smithi121 ceph-mon[145859]: pgmap v1337: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:10:43.778 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:43 smithi188 ceph-mon[107793]: pgmap v1337: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:10:45.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:45 smithi121 ceph-mon[145859]: pgmap v1338: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:10:45.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:10:46.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:45 smithi188 ceph-mon[107793]: pgmap v1338: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:10:46.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:10:47.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:47 smithi121 ceph-mon[145859]: pgmap v1339: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:48.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:47 smithi188 ceph-mon[107793]: pgmap v1339: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:49.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:49 smithi121 ceph-mon[145859]: pgmap v1340: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:50.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:49 smithi188 ceph-mon[107793]: pgmap v1340: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:51.987 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:51 smithi121 ceph-mon[145859]: pgmap v1341: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:52.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:51 smithi188 ceph-mon[107793]: pgmap v1341: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:53.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:53 smithi121 ceph-mon[145859]: pgmap v1342: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:10:54.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:53 smithi188 ceph-mon[107793]: pgmap v1342: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:10:56.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:55 smithi188 ceph-mon[107793]: pgmap v1343: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:10:56.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:55 smithi121 ceph-mon[145859]: pgmap v1343: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:10:58.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:57 smithi188 ceph-mon[107793]: pgmap v1344: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:10:58.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:57 smithi121 ceph-mon[145859]: pgmap v1344: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:00.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:10:59 smithi188 ceph-mon[107793]: pgmap v1345: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:00.237 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:10:59 smithi121 ceph-mon[145859]: pgmap v1345: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:01.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:11:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:11:02.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:01 smithi188 ceph-mon[107793]: pgmap v1346: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:02.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:01 smithi121 ceph-mon[145859]: pgmap v1346: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:03.733 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:03 smithi121 ceph-mon[145859]: pgmap v1347: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:11:03.778 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:03 smithi188 ceph-mon[107793]: pgmap v1347: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:11:04.737 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:04 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:11:04.778 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:04 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:11:05.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:05 smithi121 ceph-mon[145859]: pgmap v1348: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:11:05.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:05 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:11:05.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:05 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:11:05.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:05 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:11:05.778 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:05 smithi188 ceph-mon[107793]: pgmap v1348: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:11:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:05 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:11:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:05 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:11:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:05 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:11:07.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:07 smithi121 ceph-mon[145859]: pgmap v1349: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:08.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:07 smithi188 ceph-mon[107793]: pgmap v1349: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:09.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:09 smithi121 ceph-mon[145859]: pgmap v1350: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:10.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:09 smithi188 ceph-mon[107793]: pgmap v1350: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:12.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:11 smithi188 ceph-mon[107793]: pgmap v1351: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:12.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:11 smithi121 ceph-mon[145859]: pgmap v1351: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:14.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:13 smithi188 ceph-mon[107793]: pgmap v1352: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:11:14.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:13 smithi121 ceph-mon[145859]: pgmap v1352: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:11:16.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:15 smithi188 ceph-mon[107793]: pgmap v1353: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:11:16.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:11:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:15 smithi121 ceph-mon[145859]: pgmap v1353: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:11:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:11:18.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:17 smithi188 ceph-mon[107793]: pgmap v1354: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:18.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:17 smithi121 ceph-mon[145859]: pgmap v1354: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:20.025 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:19 smithi188 ceph-mon[107793]: pgmap v1355: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:20.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:19 smithi121 ceph-mon[145859]: pgmap v1355: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:22.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:21 smithi188 ceph-mon[107793]: pgmap v1356: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:22.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:21 smithi121 ceph-mon[145859]: pgmap v1356: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:24.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:23 smithi188 ceph-mon[107793]: pgmap v1357: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:11:24.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:23 smithi121 ceph-mon[145859]: pgmap v1357: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:11:26.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:25 smithi188 ceph-mon[107793]: pgmap v1358: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:11:26.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:25 smithi121 ceph-mon[145859]: pgmap v1358: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:11:28.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:27 smithi188 ceph-mon[107793]: pgmap v1359: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:28.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:27 smithi121 ceph-mon[145859]: pgmap v1359: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:30.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:29 smithi188 ceph-mon[107793]: pgmap v1360: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:30.237 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:29 smithi121 ceph-mon[145859]: pgmap v1360: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:31.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:11:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:11:32.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:31 smithi188 ceph-mon[107793]: pgmap v1361: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:32.237 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:31 smithi121 ceph-mon[145859]: pgmap v1361: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:34.064 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:33 smithi121 ceph-mon[145859]: pgmap v1362: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:11:34.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:33 smithi188 ceph-mon[107793]: pgmap v1362: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:11:36.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:35 smithi121 ceph-mon[145859]: pgmap v1363: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T22:11:36.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:35 smithi188 ceph-mon[107793]: pgmap v1363: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T22:11:38.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:37 smithi121 ceph-mon[145859]: pgmap v1364: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T22:11:38.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:37 smithi188 ceph-mon[107793]: pgmap v1364: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T22:11:40.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:39 smithi121 ceph-mon[145859]: pgmap v1365: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T22:11:40.239 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:39 smithi188 ceph-mon[107793]: pgmap v1365: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T22:11:42.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:41 smithi121 ceph-mon[145859]: pgmap v1366: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T22:11:42.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:41 smithi188 ceph-mon[107793]: pgmap v1366: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-04-05T22:11:44.173 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:43 smithi121 ceph-mon[145859]: pgmap v1367: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T22:11:44.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:43 smithi188 ceph-mon[107793]: pgmap v1367: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-04-05T22:11:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:45 smithi121 ceph-mon[145859]: pgmap v1368: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T22:11:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:11:46.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:45 smithi188 ceph-mon[107793]: pgmap v1368: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-04-05T22:11:46.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:11:48.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:47 smithi121 ceph-mon[145859]: pgmap v1369: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:48.279 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:47 smithi188 ceph-mon[107793]: pgmap v1369: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:50.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:49 smithi121 ceph-mon[145859]: pgmap v1370: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:50.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:49 smithi188 ceph-mon[107793]: pgmap v1370: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:52.237 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:51 smithi121 ceph-mon[145859]: pgmap v1371: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:51 smithi188 ceph-mon[107793]: pgmap v1371: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:54.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:53 smithi121 ceph-mon[145859]: pgmap v1372: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:11:54.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:53 smithi188 ceph-mon[107793]: pgmap v1372: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:11:56.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:55 smithi121 ceph-mon[145859]: pgmap v1373: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:11:56.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:55 smithi188 ceph-mon[107793]: pgmap v1373: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:11:58.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:57 smithi121 ceph-mon[145859]: pgmap v1374: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:11:58.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:57 smithi188 ceph-mon[107793]: pgmap v1374: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:00.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:11:59 smithi121 ceph-mon[145859]: pgmap v1375: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:00.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:11:59 smithi188 ceph-mon[107793]: pgmap v1375: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:01.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:00 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:12:01.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:00 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:12:02.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:01 smithi121 ceph-mon[145859]: pgmap v1376: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:02.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:01 smithi188 ceph-mon[107793]: pgmap v1376: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:03.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:03 smithi121 ceph-mon[145859]: pgmap v1377: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:12:03.778 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:03 smithi188 ceph-mon[107793]: pgmap v1377: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:12:05.778 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:05 smithi188 ceph-mon[107793]: pgmap v1378: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:12:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:05 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:12:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:05 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:12:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:05 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:12:05.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:05 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:12:06.237 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:05 smithi121 ceph-mon[145859]: pgmap v1378: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:12:06.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:05 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-05T22:12:06.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:05 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-05T22:12:06.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:05 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-05T22:12:06.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:05 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' 2024-04-05T22:12:08.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:07 smithi188 ceph-mon[107793]: pgmap v1379: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:08.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:07 smithi121 ceph-mon[145859]: pgmap v1379: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:10.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:09 smithi188 ceph-mon[107793]: pgmap v1380: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:10.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:09 smithi121 ceph-mon[145859]: pgmap v1380: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:12.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:11 smithi188 ceph-mon[107793]: pgmap v1381: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:12.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:11 smithi121 ceph-mon[145859]: pgmap v1381: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:14.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:13 smithi188 ceph-mon[107793]: pgmap v1382: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:12:14.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:13 smithi121 ceph-mon[145859]: pgmap v1382: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:12:16.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:15 smithi188 ceph-mon[107793]: pgmap v1383: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:12:16.029 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:15 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:12:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:15 smithi121 ceph-mon[145859]: pgmap v1383: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:12:16.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:15 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:12:18.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:17 smithi188 ceph-mon[107793]: pgmap v1384: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:18.237 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:17 smithi121 ceph-mon[145859]: pgmap v1384: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:19.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:19 smithi121 ceph-mon[145859]: pgmap v1385: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:20.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:19 smithi188 ceph-mon[107793]: pgmap v1385: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:22.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:21 smithi188 ceph-mon[107793]: pgmap v1386: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:22.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:21 smithi121 ceph-mon[145859]: pgmap v1386: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:24.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:23 smithi121 ceph-mon[145859]: pgmap v1387: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:12:24.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:23 smithi188 ceph-mon[107793]: pgmap v1387: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:12:26.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:25 smithi188 ceph-mon[107793]: pgmap v1388: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:12:26.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:25 smithi121 ceph-mon[145859]: pgmap v1388: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:12:28.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:27 smithi121 ceph-mon[145859]: pgmap v1389: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:28.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:27 smithi188 ceph-mon[107793]: pgmap v1389: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:29.988 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:29 smithi121 ceph-mon[145859]: pgmap v1390: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:30.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:29 smithi188 ceph-mon[107793]: pgmap v1390: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:31.028 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:30 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:12:31.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:30 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:12:32.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:31 smithi121 ceph-mon[145859]: pgmap v1391: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:32.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:31 smithi188 ceph-mon[107793]: pgmap v1391: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:34.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:33 smithi121 ceph-mon[145859]: pgmap v1392: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:12:34.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:33 smithi188 ceph-mon[107793]: pgmap v1392: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:12:36.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:35 smithi121 ceph-mon[145859]: pgmap v1393: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:12:36.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:35 smithi188 ceph-mon[107793]: pgmap v1393: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:12:38.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:37 smithi121 ceph-mon[145859]: pgmap v1394: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:38.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:37 smithi188 ceph-mon[107793]: pgmap v1394: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:40.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:39 smithi121 ceph-mon[145859]: pgmap v1395: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:40.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:39 smithi188 ceph-mon[107793]: pgmap v1395: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:42.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:41 smithi121 ceph-mon[145859]: pgmap v1396: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:42.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:41 smithi188 ceph-mon[107793]: pgmap v1396: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:44.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:43 smithi121 ceph-mon[145859]: pgmap v1397: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:12:44.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:43 smithi188 ceph-mon[107793]: pgmap v1397: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-04-05T22:12:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:45 smithi121 ceph-mon[145859]: pgmap v1398: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:12:46.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:45 smithi121 ceph-mon[145859]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:12:46.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:45 smithi188 ceph-mon[107793]: pgmap v1398: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-04-05T22:12:46.281 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:45 smithi188 ceph-mon[107793]: from='mgr.14656 172.21.15.121:0/3534878301' entity='mgr.smithi121.etbquh' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-05T22:12:48.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:47 smithi121 ceph-mon[145859]: pgmap v1399: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:48.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:47 smithi188 ceph-mon[107793]: pgmap v1399: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:50.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:49 smithi121 ceph-mon[145859]: pgmap v1400: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:50.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:49 smithi188 ceph-mon[107793]: pgmap v1400: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:52.238 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:51 smithi121 ceph-mon[145859]: pgmap v1401: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:52.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:51 smithi188 ceph-mon[107793]: pgmap v1401: 65 pgs: 65 active+clean; 270 MiB data, 1.0 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-05T22:12:52.587 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-04-05T22:12:52.588 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-05T22:12:52.590 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-05T22:12:52.602 INFO:tasks.cephadm:Teardown begin 2024-04-05T22:12:52.603 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/cephadm.py", line 1750, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-05T22:12:52.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T22:12:52.641 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T22:12:52.679 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-05T22:12:52.679 DEBUG:teuthology.orchestra.run.smithi121:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-05T22:12:52.702 DEBUG:teuthology.orchestra.run.smithi188:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-05T22:12:52.740 INFO:tasks.cephadm:Stopping all daemons... 2024-04-05T22:12:52.740 INFO:tasks.cephadm.mon.smithi121:Stopping mon.smithi121... 2024-04-05T22:12:52.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mon.smithi121 2024-04-05T22:12:52.987 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:52 smithi121 systemd[1]: Stopping Ceph mon.smithi121 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T22:12:53.443 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:53 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T22:12:53.123+0000 7f91536b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi121 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T22:12:53.443 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:53 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121[145835]: 2024-04-05T22:12:53.123+0000 7f91536b1700 -1 mon.smithi121@0(leader) e2 *** Got Signal Terminated *** 2024-04-05T22:12:53.738 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Apr 05 22:12:53 smithi121 bash[213045]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi121 2024-04-05T22:12:54.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mon.smithi121.service' 2024-04-05T22:12:54.068 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T22:12:54.068 INFO:tasks.cephadm.mon.smithi121:Stopped mon.smithi121 2024-04-05T22:12:54.069 INFO:tasks.cephadm.mon.smithi188:Stopping mon.smithi188... 2024-04-05T22:12:54.069 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mon.smithi188 2024-04-05T22:12:54.357 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:54 smithi188 systemd[1]: Stopping Ceph mon.smithi188 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T22:12:54.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:54 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi188[107770]: 2024-04-05T22:12:54.356+0000 7f9cda932700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi188 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T22:12:54.779 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:54 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi188[107770]: 2024-04-05T22:12:54.356+0000 7f9cda932700 -1 mon.smithi188@1(peon) e2 *** Got Signal Terminated *** 2024-04-05T22:12:55.278 INFO:journalctl@ceph.mon.smithi188.smithi188.stdout:Apr 05 22:12:55 smithi188 bash[153698]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-mon-smithi188 2024-04-05T22:12:55.451 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@mon.smithi188.service' 2024-04-05T22:12:55.502 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T22:12:55.503 INFO:tasks.cephadm.mon.smithi188:Stopped mon.smithi188 2024-04-05T22:12:55.503 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-05T22:12:55.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.0 2024-04-05T22:12:55.787 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 22:12:55 smithi121 systemd[1]: Stopping Ceph osd.0 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T22:12:56.238 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 22:12:55 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-0[157629]: 2024-04-05T22:12:55.787+0000 7fd0dfea8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T22:12:56.238 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 22:12:55 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-0[157629]: 2024-04-05T22:12:55.787+0000 7fd0dfea8700 -1 osd.0 84 *** Got signal Terminated *** 2024-04-05T22:12:56.238 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 22:12:55 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-0[157629]: 2024-04-05T22:12:55.787+0000 7fd0dfea8700 -1 osd.0 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T22:13:01.988 INFO:journalctl@ceph.osd.0.smithi121.stdout:Apr 05 22:13:01 smithi121 bash[213204]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-0 2024-04-05T22:13:03.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.0.service' 2024-04-05T22:13:03.776 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T22:13:03.777 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-05T22:13:03.777 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-05T22:13:03.777 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.1 2024-04-05T22:13:03.987 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 22:13:03 smithi121 systemd[1]: Stopping Ceph osd.1 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T22:13:04.487 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 22:13:04 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1[164473]: 2024-04-05T22:13:04.063+0000 7f637d0b0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T22:13:04.488 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 22:13:04 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1[164473]: 2024-04-05T22:13:04.063+0000 7f637d0b0700 -1 osd.1 84 *** Got signal Terminated *** 2024-04-05T22:13:04.488 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 22:13:04 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1[164473]: 2024-04-05T22:13:04.063+0000 7f637d0b0700 -1 osd.1 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T22:13:10.087 INFO:journalctl@ceph.osd.1.smithi121.stdout:Apr 05 22:13:09 smithi121 bash[213478]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-1 2024-04-05T22:13:11.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.1.service' 2024-04-05T22:13:11.657 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T22:13:11.657 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-05T22:13:11.657 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-05T22:13:11.657 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.2 2024-04-05T22:13:11.966 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 22:13:11 smithi121 systemd[1]: Stopping Ceph osd.2 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T22:13:12.237 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 22:13:11 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-2[171794]: 2024-04-05T22:13:11.966+0000 7f663fb0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T22:13:12.238 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 22:13:11 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-2[171794]: 2024-04-05T22:13:11.966+0000 7f663fb0d700 -1 osd.2 84 *** Got signal Terminated *** 2024-04-05T22:13:12.238 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 22:13:11 smithi121 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-2[171794]: 2024-04-05T22:13:11.966+0000 7f663fb0d700 -1 osd.2 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T22:13:17.987 INFO:journalctl@ceph.osd.2.smithi121.stdout:Apr 05 22:13:17 smithi121 bash[213749]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-2 2024-04-05T22:13:19.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.2.service' 2024-04-05T22:13:19.484 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T22:13:19.484 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-05T22:13:19.484 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-05T22:13:19.484 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.3 2024-04-05T22:13:19.778 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 22:13:19 smithi188 systemd[1]: Stopping Ceph osd.3 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T22:13:20.278 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 22:13:19 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-3[118320]: 2024-04-05T22:13:19.790+0000 7f3af495a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T22:13:20.279 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 22:13:19 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-3[118320]: 2024-04-05T22:13:19.790+0000 7f3af495a700 -1 osd.3 84 *** Got signal Terminated *** 2024-04-05T22:13:20.279 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 22:13:19 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-3[118320]: 2024-04-05T22:13:19.790+0000 7f3af495a700 -1 osd.3 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T22:13:22.028 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:21 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:21.711+0000 7f64ee101700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.003094+0000 front 2024-04-05T22:13:00.003041+0000 (oldest deadline 2024-04-05T22:13:21.603357+0000) 2024-04-05T22:13:23.028 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:22 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:22.686+0000 7f64ee101700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.003094+0000 front 2024-04-05T22:13:00.003041+0000 (oldest deadline 2024-04-05T22:13:21.603357+0000) 2024-04-05T22:13:24.028 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 22:13:23 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-3[118320]: 2024-04-05T22:13:23.931+0000 7f3aed749700 -1 osd.3 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:12:57.591265+0000 front 2024-04-05T22:12:57.591347+0000 (oldest deadline 2024-04-05T22:13:23.491305+0000) 2024-04-05T22:13:24.029 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:23 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:23.732+0000 7f64ee101700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.003094+0000 front 2024-04-05T22:13:00.003041+0000 (oldest deadline 2024-04-05T22:13:21.603357+0000) 2024-04-05T22:13:24.029 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:23 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:23.827+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:25.028 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:24 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:24.767+0000 7f64ee101700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.003094+0000 front 2024-04-05T22:13:00.003041+0000 (oldest deadline 2024-04-05T22:13:21.603357+0000) 2024-04-05T22:13:25.029 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:24 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:24.841+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:25.737 INFO:journalctl@ceph.osd.3.smithi188.stdout:Apr 05 22:13:25 smithi188 bash[153853]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-3 2024-04-05T22:13:26.028 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:25 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:25.737+0000 7f64ee101700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.003094+0000 front 2024-04-05T22:13:00.003041+0000 (oldest deadline 2024-04-05T22:13:21.603357+0000) 2024-04-05T22:13:26.029 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:25 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:25.876+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:26.848 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:26 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:26.706+0000 7f64ee101700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.003094+0000 front 2024-04-05T22:13:00.003041+0000 (oldest deadline 2024-04-05T22:13:21.603357+0000) 2024-04-05T22:13:27.151 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:26 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:26.847+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:27.159 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.3.service' 2024-04-05T22:13:27.201 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T22:13:27.201 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-05T22:13:27.201 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-05T22:13:27.201 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.4 2024-04-05T22:13:27.506 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:27 smithi188 systemd[1]: Stopping Ceph osd.4 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T22:13:27.506 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:27 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:27.505+0000 7f64f130a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T22:13:27.778 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:27 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:27.505+0000 7f64f130a700 -1 osd.4 84 *** Got signal Terminated *** 2024-04-05T22:13:27.778 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:27 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:27.505+0000 7f64f130a700 -1 osd.4 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T22:13:27.779 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:27 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:27.658+0000 7f64ee101700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.003094+0000 front 2024-04-05T22:13:00.003041+0000 (oldest deadline 2024-04-05T22:13:21.603357+0000) 2024-04-05T22:13:28.278 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:27 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:27.838+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:29.028 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:28 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:28.885+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:29.029 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:28 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:28.636+0000 7f64ee101700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.003094+0000 front 2024-04-05T22:13:00.003041+0000 (oldest deadline 2024-04-05T22:13:21.603357+0000) 2024-04-05T22:13:29.930 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:29 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:29.670+0000 7f64ee101700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.003094+0000 front 2024-04-05T22:13:00.003041+0000 (oldest deadline 2024-04-05T22:13:21.603357+0000) 2024-04-05T22:13:30.278 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:29 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:29.930+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:30.969 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:30 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:30.713+0000 7f64ee101700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.003094+0000 front 2024-04-05T22:13:00.003041+0000 (oldest deadline 2024-04-05T22:13:21.603357+0000) 2024-04-05T22:13:30.970 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:30 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:30.713+0000 7f64ee101700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-04-05T22:13:06.204803+0000 front 2024-04-05T22:13:06.204790+0000 (oldest deadline 2024-04-05T22:13:30.304622+0000) 2024-04-05T22:13:31.278 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:30 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:30.970+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:32.011 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:31 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:31.713+0000 7f64ee101700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.003094+0000 front 2024-04-05T22:13:00.003041+0000 (oldest deadline 2024-04-05T22:13:21.603357+0000) 2024-04-05T22:13:32.011 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:31 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4[125147]: 2024-04-05T22:13:31.713+0000 7f64ee101700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-04-05T22:13:06.204803+0000 front 2024-04-05T22:13:06.204790+0000 (oldest deadline 2024-04-05T22:13:30.304622+0000) 2024-04-05T22:13:32.278 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:32 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:32.012+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:32.278 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:32 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:32.012+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-04-05T22:13:08.992501+0000 front 2024-04-05T22:13:08.992518+0000 (oldest deadline 2024-04-05T22:13:31.292388+0000) 2024-04-05T22:13:33.279 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:32 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:32.990+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:33.279 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:32 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:32.990+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-04-05T22:13:08.992501+0000 front 2024-04-05T22:13:08.992518+0000 (oldest deadline 2024-04-05T22:13:31.292388+0000) 2024-04-05T22:13:33.778 INFO:journalctl@ceph.osd.4.smithi188.stdout:Apr 05 22:13:33 smithi188 bash[154119]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-4 2024-04-05T22:13:34.278 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:34 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:34.028+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:34.278 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:34 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:34.028+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-04-05T22:13:08.992501+0000 front 2024-04-05T22:13:08.992518+0000 (oldest deadline 2024-04-05T22:13:31.292388+0000) 2024-04-05T22:13:35.204 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:35 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:35.065+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:35.204 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:35 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:35.065+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-04-05T22:13:08.992501+0000 front 2024-04-05T22:13:08.992518+0000 (oldest deadline 2024-04-05T22:13:31.292388+0000) 2024-04-05T22:13:35.211 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.4.service' 2024-04-05T22:13:35.253 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T22:13:35.254 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-05T22:13:35.254 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-05T22:13:35.254 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.5 2024-04-05T22:13:35.528 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:35 smithi188 systemd[1]: Stopping Ceph osd.5 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4... 2024-04-05T22:13:36.025 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:35 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:35.545+0000 7f5769af9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-05T22:13:36.026 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:35 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:35.545+0000 7f5769af9700 -1 osd.5 84 *** Got signal Terminated *** 2024-04-05T22:13:36.026 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:35 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:35.545+0000 7f5769af9700 -1 osd.5 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-05T22:13:36.026 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:36 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:36.025+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:36.026 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:36 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:36.025+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-04-05T22:13:08.992501+0000 front 2024-04-05T22:13:08.992518+0000 (oldest deadline 2024-04-05T22:13:31.292388+0000) 2024-04-05T22:13:37.278 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:37 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:36.999+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:37.279 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:37 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:36.999+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-04-05T22:13:08.992501+0000 front 2024-04-05T22:13:08.992518+0000 (oldest deadline 2024-04-05T22:13:31.292388+0000) 2024-04-05T22:13:38.279 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:37 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:37.987+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:38.279 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:37 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:37.987+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-04-05T22:13:08.992501+0000 front 2024-04-05T22:13:08.992518+0000 (oldest deadline 2024-04-05T22:13:31.292388+0000) 2024-04-05T22:13:38.279 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:37 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:37.987+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-04-05T22:13:14.193153+0000 front 2024-04-05T22:13:14.193118+0000 (oldest deadline 2024-04-05T22:13:37.693028+0000) 2024-04-05T22:13:39.278 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:38 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:38.998+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:39.279 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:38 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:38.998+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-04-05T22:13:08.992501+0000 front 2024-04-05T22:13:08.992518+0000 (oldest deadline 2024-04-05T22:13:31.292388+0000) 2024-04-05T22:13:39.279 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:38 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:38.998+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-04-05T22:13:14.193153+0000 front 2024-04-05T22:13:14.193118+0000 (oldest deadline 2024-04-05T22:13:37.693028+0000) 2024-04-05T22:13:40.278 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:40 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:40.021+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-04-05T22:13:00.191626+0000 front 2024-04-05T22:13:00.191578+0000 (oldest deadline 2024-04-05T22:13:23.691595+0000) 2024-04-05T22:13:40.279 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:40 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:40.021+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-04-05T22:13:08.992501+0000 front 2024-04-05T22:13:08.992518+0000 (oldest deadline 2024-04-05T22:13:31.292388+0000) 2024-04-05T22:13:40.279 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:40 smithi188 ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5[131441]: 2024-04-05T22:13:40.021+0000 7f57668f0700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-04-05T22:13:14.193153+0000 front 2024-04-05T22:13:14.193118+0000 (oldest deadline 2024-04-05T22:13:37.693028+0000) 2024-04-05T22:13:41.528 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:41 smithi188 bash[154381]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4-osd-5 2024-04-05T22:13:42.988 DEBUG:teuthology.orchestra.run.smithi188:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.5.service' 2024-04-05T22:13:43.019 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:42 smithi188 systemd[1]: ceph-9e6dd7f6-f391-11ee-b649-cb9ed24678a4@osd.5.service: Succeeded. 2024-04-05T22:13:43.019 INFO:journalctl@ceph.osd.5.smithi188.stdout:Apr 05 22:13:42 smithi188 systemd[1]: Stopped Ceph osd.5 for 9e6dd7f6-f391-11ee-b649-cb9ed24678a4. 2024-04-05T22:13:43.030 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-05T22:13:43.030 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-05T22:13:43.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-05T22:13:43.208 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T22:13:46.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[102066]: fuse finished with error 0 and tester_r 0 2024-04-05T22:14:04.528 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-05T22:14:04.708 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T22:14:16.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T22:14:16.134 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-05T22:14:16.164 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-05T22:14:16.166 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641605/remote/smithi121/crash 2024-04-05T22:14:16.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash -- . 2024-04-05T22:14:16.205 INFO:teuthology.orchestra.run.smithi121.stderr:tar: /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-05T22:14:16.205 INFO:teuthology.orchestra.run.smithi121.stderr:tar: Error is not recoverable: exiting now 2024-04-05T22:14:16.207 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641605/remote/smithi188/crash 2024-04-05T22:14:16.208 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash -- . 2024-04-05T22:14:16.235 INFO:teuthology.orchestra.run.smithi188.stderr:tar: /var/lib/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-05T22:14:16.235 INFO:teuthology.orchestra.run.smithi188.stderr:tar: Error is not recoverable: exiting now 2024-04-05T22:14:16.236 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-05T22:14:16.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-04-05T22:14:16.278 INFO:teuthology.orchestra.run.smithi121.stderr:grep: /var/log/ceph/9e6dd7f6-f391-11ee-b649-cb9ed24678a4/ceph.log: No such file or directory 2024-04-05T22:14:16.280 INFO:tasks.cephadm:Compressing logs... 2024-04-05T22:14:16.280 DEBUG:teuthology.orchestra.run.smithi121:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-05T22:14:16.323 DEBUG:teuthology.orchestra.run.smithi188:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-05T22:14:16.348 INFO:teuthology.orchestra.run.smithi121.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-05T22:14:16.349 INFO:teuthology.orchestra.run.smithi188.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-05T22:15:46.816 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-04-05T22:16:07.990 INFO:tasks.cephadm:Archiving logs... 2024-04-05T22:16:07.991 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641605/remote/smithi121/log 2024-04-05T22:16:07.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-05T22:16:21.453 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641605/remote/smithi188/log 2024-04-05T22:16:21.471 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-05T22:16:32.235 INFO:tasks.cephadm:Removing cluster... 2024-04-05T22:16:32.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 --force 2024-04-05T22:16:32.419 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T22:16:32.748 DEBUG:teuthology.orchestra.run.smithi188:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 --force 2024-04-05T22:16:32.935 INFO:teuthology.orchestra.run.smithi188.stdout:Deleting cluster with fsid: 9e6dd7f6-f391-11ee-b649-cb9ed24678a4 2024-04-05T22:16:33.228 INFO:tasks.cephadm:Removing cephadm ... 2024-04-05T22:16:33.229 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-05T22:16:33.249 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-05T22:16:33.266 INFO:tasks.cephadm:Teardown complete 2024-04-05T22:16:33.267 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-05T22:16:33.280 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-05T22:16:33.281 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-05T22:16:33.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-05T22:16:33.291 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-05T22:16:33.380 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-05T22:16:33.392 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-05T22:16:33.393 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T22:16:33.396 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-05T22:16:33.410 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2024-04-05T22:16:33.412 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2024-04-05T22:16:33.414 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T22:16:33.414 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-04-05T22:16:33.414 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 40 -1074us[-1074us] +/- 88ms 2024-04-05T22:16:33.414 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv02.front.sepia.ceph.com 3 7 377 43 +12us[ +34us] +/- 63ms 2024-04-05T22:16:33.415 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +1513us[+1513us] +/- 111ms 2024-04-05T22:16:33.415 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T22:16:33.416 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-05T22:16:33.416 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-05T22:16:33.416 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 -1129us[-1144us] +/- 87ms 2024-04-05T22:16:33.416 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -25us[ -41us] +/- 63ms 2024-04-05T22:16:33.417 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +1493us[+1493us] +/- 111ms 2024-04-05T22:16:33.417 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-05T22:16:33.417 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-05T22:16:33.431 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-05T22:16:33.432 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-05T22:16:33.443 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-05T22:16:33.530 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-05T22:16:33.602 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-05T22:16:33.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T22:16:33.633 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-05T22:16:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T22:16:33.663 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-05T22:16:33.686 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-05T22:16:33.715 INFO:teuthology.task.internal:Duration was 4219.312736 seconds 2024-04-05T22:16:33.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-05T22:16:33.728 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-05T22:16:33.728 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-05T22:16:33.731 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-05T22:16:33.774 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T22:16:33.776 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-05T22:16:34.000 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-05T22:16:34.000 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-04-05T22:16:34.001 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-05T22:16:34.022 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2024-04-05T22:16:34.022 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-05T22:16:34.044 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-05T22:16:34.044 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-05T22:16:34.064 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-05T22:16:34.142 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-05T22:16:34.143 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-05T22:16:34.145 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-05T22:16:34.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-05T22:16:34.330 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-05T22:16:34.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-05T22:16:34.358 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-05T22:16:34.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-05T22:16:34.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-05T22:16:34.402 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-05T22:16:34.426 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-04-05T22:16:34.453 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2024-04-05T22:16:34.483 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-05T22:16:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T22:16:34.540 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-05T22:16:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T22:16:34.558 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-05T22:16:34.572 INFO:teuthology.task.internal:Transferring archived files... 2024-04-05T22:16:34.573 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641605/remote/smithi121 2024-04-05T22:16:34.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-05T22:16:34.628 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641605/remote/smithi188 2024-04-05T22:16:34.628 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-05T22:16:34.672 INFO:teuthology.task.internal:Removing archive directory... 2024-04-05T22:16:34.672 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-05T22:16:34.674 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-05T22:16:34.722 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-05T22:16:34.759 INFO:teuthology.task.internal:Not uploading archives. 2024-04-05T22:16:34.759 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-05T22:16:34.772 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-05T22:16:34.772 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-05T22:16:34.774 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-05T22:16:34.789 INFO:teuthology.orchestra.run.smithi121.stdout: 525417 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 5 22:16 /home/ubuntu/cephtest 2024-04-05T22:16:34.790 INFO:teuthology.orchestra.run.smithi121.stdout: 920269 4 d--------- 2 ubuntu ubuntu 4096 Apr 5 21:24 /home/ubuntu/cephtest/mnt.0 2024-04-05T22:16:34.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-05T22:16:34.791 INFO:teuthology.orchestra.run.smithi121.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-04-05T22:16:34.791 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-05T22:16:34.791 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_8919605b0a5dc1f1650c8aec3051c77c0e3e240e/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi121 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-05T22:16:34.792 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-05T22:16:34.805 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-05T22:16:34.818 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-05T22:16:34.818 INFO:teuthology.nuke:Checking targets against current locks 2024-04-05T22:16:34.844 DEBUG:teuthology.nuke:shortname: smithi121 2024-04-05T22:16:34.844 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-05T22:16:34.869 DEBUG:teuthology.nuke:shortname: smithi188 2024-04-05T22:16:34.869 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-05T22:16:34.873 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641605', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 20:58:00.114737', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO58RlMZFByVT5ZRBanllmMbC9CSawxqmhUpRvHjVWyUXy5xZEoiBB5gELQFV7WNcoMcdiVG87UfgWfN2t6CfHk='} 2024-04-05T22:16:34.896 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641605', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-05 20:58:00.112903', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEIbE86l8eTwgRWHd5p9CW3jojPfoughoMo3/8pqxotWgoG4si8oBR41hhjMMH+sWlEujqcu+S9EdAxqzfUZ0pU='} 2024-04-05T22:16:34.920 INFO:teuthology.orchestra.console.smithi121:Power off 2024-04-05T22:16:34.921 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-05T22:16:34.952 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2024-04-05T22:16:34.953 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T22:16:34.968 INFO:teuthology.orchestra.console.smithi188:Power off 2024-04-05T22:16:34.968 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-05T22:16:34.978 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-04-05T22:16:34.991 DEBUG:teuthology.orchestra.console.smithi188:power off output: Chassis Power Control: Down/Off 2024-04-05T22:16:34.991 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T22:16:35.015 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-05T22:16:38.979 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T22:16:39.015 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T22:16:39.089 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-04-05T22:16:39.126 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-04-05T22:16:43.090 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T22:16:43.127 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-05T22:16:43.200 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2024-04-05T22:16:43.237 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2024-04-05T22:16:43.302 INFO:teuthology.orchestra.console.smithi121:Power off completed 2024-04-05T22:16:43.338 INFO:teuthology.orchestra.console.smithi188:Power off completed 2024-04-05T22:16:43.443 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 4219.312736034393 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-04-05T22:16:43.444 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-05T22:16:43.527 INFO:teuthology.run:FAIL